reasoning-dataset-lab 0.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- reasoning_dataset_lab-0.4.0/.gitattributes +1 -0
- reasoning_dataset_lab-0.4.0/.github/workflows/ci.yml +117 -0
- reasoning_dataset_lab-0.4.0/.github/workflows/pages.yml +51 -0
- reasoning_dataset_lab-0.4.0/.github/workflows/release.yml +101 -0
- reasoning_dataset_lab-0.4.0/.gitignore +19 -0
- reasoning_dataset_lab-0.4.0/.python-version +1 -0
- reasoning_dataset_lab-0.4.0/CHANGELOG.md +96 -0
- reasoning_dataset_lab-0.4.0/LICENSE +21 -0
- reasoning_dataset_lab-0.4.0/PKG-INFO +125 -0
- reasoning_dataset_lab-0.4.0/README.es.md +92 -0
- reasoning_dataset_lab-0.4.0/README.fr.md +73 -0
- reasoning_dataset_lab-0.4.0/README.hi.md +85 -0
- reasoning_dataset_lab-0.4.0/README.it.md +73 -0
- reasoning_dataset_lab-0.4.0/README.ja.md +73 -0
- reasoning_dataset_lab-0.4.0/README.md +97 -0
- reasoning_dataset_lab-0.4.0/README.pt-BR.md +90 -0
- reasoning_dataset_lab-0.4.0/README.zh.md +73 -0
- reasoning_dataset_lab-0.4.0/SCORECARD.md +39 -0
- reasoning_dataset_lab-0.4.0/SECURITY.md +35 -0
- reasoning_dataset_lab-0.4.0/SHIP_GATE.md +84 -0
- reasoning_dataset_lab-0.4.0/atlas/README.md +95 -0
- reasoning_dataset_lab-0.4.0/atlas/boundaries.yaml +38 -0
- reasoning_dataset_lab-0.4.0/atlas/page.json +325 -0
- reasoning_dataset_lab-0.4.0/atlas/statistics.json +989 -0
- reasoning_dataset_lab-0.4.0/atlas/structure.json +8654 -0
- reasoning_dataset_lab-0.4.0/certification/r3/FINDINGS.md +126 -0
- reasoning_dataset_lab-0.4.0/certification/r3/README.md +18 -0
- reasoning_dataset_lab-0.4.0/certification/r3/manifest-r3.json +5080 -0
- reasoning_dataset_lab-0.4.0/certification/r3/review_queue-r3.json +24 -0
- reasoning_dataset_lab-0.4.0/certification/r3/second-keying/argon-keys.jsonl +547 -0
- reasoning_dataset_lab-0.4.0/certification/r3/second-keying/keying-items.jsonl +547 -0
- reasoning_dataset_lab-0.4.0/certification/v3/FINDINGS.md +95 -0
- reasoning_dataset_lab-0.4.0/certification/v3/README.md +15 -0
- reasoning_dataset_lab-0.4.0/certification/v3/control-mistral-v3.json +251 -0
- reasoning_dataset_lab-0.4.0/certification/v3/control-nemotron-v3.json +251 -0
- reasoning_dataset_lab-0.4.0/certification/v3/manifest-v3.json +8232 -0
- reasoning_dataset_lab-0.4.0/certification/v3/medium-arm/COMPARE-r3-nemotron-medium.md +1084 -0
- reasoning_dataset_lab-0.4.0/certification/v3/medium-arm/control-nemotron-v2-medium.json +251 -0
- reasoning_dataset_lab-0.4.0/certification/v3/medium-arm/manifest-r3-nemotron-medium.json +4927 -0
- reasoning_dataset_lab-0.4.0/certification/v3/medium-arm/review_queue-r3-nemotron-medium.json +20 -0
- reasoning_dataset_lab-0.4.0/certification/v3/review_queue-v3.json +6 -0
- reasoning_dataset_lab-0.4.0/certification/v3/second-keying/v3/argon-keys-v3.jsonl +1000 -0
- reasoning_dataset_lab-0.4.0/certification/v3/second-keying/v3/changed-sets-33a429b.json +1 -0
- reasoning_dataset_lab-0.4.0/certification/v3/second-keying/v3/keying-items-v3.jsonl +1000 -0
- reasoning_dataset_lab-0.4.0/certification/v3/second-keying/v3/lg-flagged-pairs.jsonl +120 -0
- reasoning_dataset_lab-0.4.0/certification/v3/second-keying/v3b/argon-keys-v3b.jsonl +560 -0
- reasoning_dataset_lab-0.4.0/certification/v3/second-keying/v3b/keying-items-v3b.jsonl +560 -0
- reasoning_dataset_lab-0.4.0/certification/v3/second-keying/v3c/argon-keys-v3c.jsonl +60 -0
- reasoning_dataset_lab-0.4.0/certification/v3/second-keying/v3c/keying-items-v3c.jsonl +60 -0
- reasoning_dataset_lab-0.4.0/docs/calibration-2026-10-25.md +77 -0
- reasoning_dataset_lab-0.4.0/examples/t10-pilot-a.map.yaml +56 -0
- reasoning_dataset_lab-0.4.0/pyproject.toml +68 -0
- reasoning_dataset_lab-0.4.0/scripts/verify.sh +14 -0
- reasoning_dataset_lab-0.4.0/site/astro.config.mjs +30 -0
- reasoning_dataset_lab-0.4.0/site/package-lock.json +5707 -0
- reasoning_dataset_lab-0.4.0/site/package.json +18 -0
- reasoning_dataset_lab-0.4.0/site/public/favicon.svg +5 -0
- reasoning_dataset_lab-0.4.0/site/src/content/docs/handbook/certification.md +35 -0
- reasoning_dataset_lab-0.4.0/site/src/content/docs/handbook/getting-started.md +50 -0
- reasoning_dataset_lab-0.4.0/site/src/content/docs/handbook/index.md +25 -0
- reasoning_dataset_lab-0.4.0/site/src/content/docs/handbook/map-first.md +36 -0
- reasoning_dataset_lab-0.4.0/site/src/content/docs/handbook/pilot-b.md +69 -0
- reasoning_dataset_lab-0.4.0/site/src/content/docs/handbook/reference.md +42 -0
- reasoning_dataset_lab-0.4.0/site/src/content/docs/handbook/security.md +27 -0
- reasoning_dataset_lab-0.4.0/site/src/content.config.ts +7 -0
- reasoning_dataset_lab-0.4.0/site/src/pages/index.astro +35 -0
- reasoning_dataset_lab-0.4.0/site/src/site-config.ts +83 -0
- reasoning_dataset_lab-0.4.0/site/src/styles/global.css +3 -0
- reasoning_dataset_lab-0.4.0/site/src/styles/starlight-custom.css +5 -0
- reasoning_dataset_lab-0.4.0/site/tsconfig.json +5 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/__init__.py +7 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/cli.py +801 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/config.py +386 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/errors.py +49 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/generators/__init__.py +51 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/generators/base.py +489 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/generators/role_set.py +506 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/map.py +835 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/pilot_b/__init__.py +5 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/pilot_b/fact_tables.py +2263 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/pilot_b/models_and_build.py +1390 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/pilot_b/ollama.py +1669 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/pilot_b/rendering.py +636 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/pilot_b/schema_check.py +109 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/pilot_b/schemas/item.schema.json +76 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/pilot_b/schemas/part.schema.json +501 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/pilot_b/schemas/provenance.schema.json +608 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/pilot_b/schemas/report.schema.json +664 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/pilot_b/unanimity.py +535 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/pilot_b/value_certification.py +1485 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/pilot_b/variety.py +1213 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/pilot_b/variety_targets.yaml +54 -0
- reasoning_dataset_lab-0.4.0/src/reasoning_dataset_lab/pilot_b/waves_and_sealing.py +2200 -0
- reasoning_dataset_lab-0.4.0/tests/fixtures/rnd-licences-registry-fb27d76.json +1728 -0
- reasoning_dataset_lab-0.4.0/tests/pilot_b_fakes.py +329 -0
- reasoning_dataset_lab-0.4.0/tests/test_cli.py +135 -0
- reasoning_dataset_lab-0.4.0/tests/test_config.py +394 -0
- reasoning_dataset_lab-0.4.0/tests/test_generators_base.py +144 -0
- reasoning_dataset_lab-0.4.0/tests/test_map.py +732 -0
- reasoning_dataset_lab-0.4.0/tests/test_pilot_b_ollama_adapters.py +2133 -0
- reasoning_dataset_lab-0.4.0/tests/test_pilot_b_part_a.py +916 -0
- reasoning_dataset_lab-0.4.0/tests/test_pilot_b_part_b.py +540 -0
- reasoning_dataset_lab-0.4.0/tests/test_pilot_b_part_c.py +323 -0
- reasoning_dataset_lab-0.4.0/tests/test_pilot_b_part_d.py +353 -0
- reasoning_dataset_lab-0.4.0/tests/test_pilot_b_part_e.py +777 -0
- reasoning_dataset_lab-0.4.0/tests/test_pilot_b_part_f.py +68 -0
- reasoning_dataset_lab-0.4.0/tests/test_pilot_b_render_failure_rate.py +161 -0
- reasoning_dataset_lab-0.4.0/tests/test_pilot_b_render_rules.py +114 -0
- reasoning_dataset_lab-0.4.0/tests/test_pilot_b_sealing_seed.py +1605 -0
- reasoning_dataset_lab-0.4.0/tests/test_pilot_b_value_certification.py +2264 -0
- reasoning_dataset_lab-0.4.0/tests/test_pilot_b_value_fallbacks.py +67 -0
- reasoning_dataset_lab-0.4.0/tests/test_pilot_b_value_sets.py +858 -0
- reasoning_dataset_lab-0.4.0/tests/test_pilot_b_variety.py +1226 -0
- reasoning_dataset_lab-0.4.0/tests/test_python_support.py +117 -0
- reasoning_dataset_lab-0.4.0/tests/test_role_set_adapter.py +408 -0
- reasoning_dataset_lab-0.4.0/uv.lock +261 -0
|
@@ -0,0 +1 @@
|
|
|
1
|
+
* text=auto eol=lf
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
name: ci
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main]
|
|
6
|
+
paths:
|
|
7
|
+
- pyproject.toml
|
|
8
|
+
- uv.lock
|
|
9
|
+
- .python-version
|
|
10
|
+
- src/**
|
|
11
|
+
- tests/**
|
|
12
|
+
- examples/**
|
|
13
|
+
- docs/**
|
|
14
|
+
- site/package*.json
|
|
15
|
+
- site/src/content/docs/handbook/pilot-b.md
|
|
16
|
+
- atlas/**
|
|
17
|
+
- .github/workflows/**
|
|
18
|
+
pull_request:
|
|
19
|
+
workflow_dispatch:
|
|
20
|
+
|
|
21
|
+
concurrency:
|
|
22
|
+
group: ${{ github.workflow }}-${{ github.ref }}
|
|
23
|
+
cancel-in-progress: true
|
|
24
|
+
|
|
25
|
+
permissions:
|
|
26
|
+
contents: read
|
|
27
|
+
|
|
28
|
+
jobs:
|
|
29
|
+
# Python 3.15. The main leg: coverage with the 90% floor, the Codecov upload and the Atlas check run here once.
|
|
30
|
+
test:
|
|
31
|
+
runs-on: ubuntu-latest
|
|
32
|
+
timeout-minutes: 15
|
|
33
|
+
permissions:
|
|
34
|
+
contents: read
|
|
35
|
+
id-token: write
|
|
36
|
+
steps:
|
|
37
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
38
|
+
|
|
39
|
+
- uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2
|
|
40
|
+
with:
|
|
41
|
+
enable-cache: true
|
|
42
|
+
python-version: "3.15"
|
|
43
|
+
|
|
44
|
+
- name: Install dependencies
|
|
45
|
+
run: uv sync --frozen
|
|
46
|
+
|
|
47
|
+
- name: Run pytest with coverage (floor 90%)
|
|
48
|
+
run: uv run pytest -v --cov=reasoning_dataset_lab --cov-report=term-missing --cov-report=xml --cov-fail-under=90
|
|
49
|
+
|
|
50
|
+
- name: Validate example map
|
|
51
|
+
run: uv run rdlab map check examples/t10-pilot-a.map.yaml
|
|
52
|
+
|
|
53
|
+
- name: Upload coverage to Codecov
|
|
54
|
+
uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1
|
|
55
|
+
with:
|
|
56
|
+
version: v11.3.1
|
|
57
|
+
use_oidc: true
|
|
58
|
+
fail_ci_if_error: false
|
|
59
|
+
disable_search: true
|
|
60
|
+
files: coverage.xml
|
|
61
|
+
|
|
62
|
+
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
|
|
63
|
+
with:
|
|
64
|
+
node-version: "22"
|
|
65
|
+
|
|
66
|
+
- name: Atlas check
|
|
67
|
+
run: npx --yes @dogfood-lab/atlas@1.24.0 check
|
|
68
|
+
|
|
69
|
+
- name: Audit the site's npm tree
|
|
70
|
+
working-directory: site
|
|
71
|
+
run: npm ci && npm audit --audit-level=high
|
|
72
|
+
|
|
73
|
+
# Python 3.13. Tests without coverage; a job of its own so the check named "test" keeps its name (it runs on 3.15).
|
|
74
|
+
test-py313:
|
|
75
|
+
runs-on: ubuntu-latest
|
|
76
|
+
timeout-minutes: 15
|
|
77
|
+
permissions:
|
|
78
|
+
contents: read
|
|
79
|
+
steps:
|
|
80
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
81
|
+
|
|
82
|
+
- uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2
|
|
83
|
+
with:
|
|
84
|
+
enable-cache: true
|
|
85
|
+
python-version: "3.13"
|
|
86
|
+
|
|
87
|
+
- name: Install dependencies
|
|
88
|
+
run: uv sync --frozen
|
|
89
|
+
|
|
90
|
+
- name: Run pytest (no coverage on this leg)
|
|
91
|
+
run: uv run pytest -v --no-cov
|
|
92
|
+
|
|
93
|
+
- name: Validate example map
|
|
94
|
+
run: uv run rdlab map check examples/t10-pilot-a.map.yaml
|
|
95
|
+
|
|
96
|
+
# Python 3.14. Tests without coverage; a job of its own so the check named "test" keeps its name (it runs on 3.15).
|
|
97
|
+
test-py314:
|
|
98
|
+
runs-on: ubuntu-latest
|
|
99
|
+
timeout-minutes: 15
|
|
100
|
+
permissions:
|
|
101
|
+
contents: read
|
|
102
|
+
steps:
|
|
103
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
104
|
+
|
|
105
|
+
- uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2
|
|
106
|
+
with:
|
|
107
|
+
enable-cache: true
|
|
108
|
+
python-version: "3.14"
|
|
109
|
+
|
|
110
|
+
- name: Install dependencies
|
|
111
|
+
run: uv sync --frozen
|
|
112
|
+
|
|
113
|
+
- name: Run pytest (no coverage on this leg)
|
|
114
|
+
run: uv run pytest -v --no-cov
|
|
115
|
+
|
|
116
|
+
- name: Validate example map
|
|
117
|
+
run: uv run rdlab map check examples/t10-pilot-a.map.yaml
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
name: Deploy site to GitHub Pages
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main]
|
|
6
|
+
paths:
|
|
7
|
+
- 'site/**'
|
|
8
|
+
- '.github/workflows/pages.yml'
|
|
9
|
+
workflow_dispatch:
|
|
10
|
+
|
|
11
|
+
concurrency:
|
|
12
|
+
group: ${{ github.workflow }}-${{ github.ref }}
|
|
13
|
+
cancel-in-progress: true
|
|
14
|
+
|
|
15
|
+
permissions:
|
|
16
|
+
contents: read
|
|
17
|
+
|
|
18
|
+
jobs:
|
|
19
|
+
build:
|
|
20
|
+
runs-on: ubuntu-latest
|
|
21
|
+
steps:
|
|
22
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
23
|
+
|
|
24
|
+
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
|
|
25
|
+
with:
|
|
26
|
+
node-version: 22
|
|
27
|
+
|
|
28
|
+
- name: Install site dependencies
|
|
29
|
+
working-directory: site
|
|
30
|
+
run: npm ci
|
|
31
|
+
|
|
32
|
+
- name: Build site
|
|
33
|
+
working-directory: site
|
|
34
|
+
run: npm run build
|
|
35
|
+
|
|
36
|
+
- uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
|
|
37
|
+
with:
|
|
38
|
+
path: site/dist
|
|
39
|
+
|
|
40
|
+
deploy:
|
|
41
|
+
needs: build
|
|
42
|
+
runs-on: ubuntu-latest
|
|
43
|
+
permissions:
|
|
44
|
+
pages: write
|
|
45
|
+
id-token: write
|
|
46
|
+
environment:
|
|
47
|
+
name: github-pages
|
|
48
|
+
url: ${{ steps.deployment.outputs.page_url }}
|
|
49
|
+
steps:
|
|
50
|
+
- id: deployment
|
|
51
|
+
uses: actions/deploy-pages@368f82528645a54fb793d4d04e342629a3f51346 # v5.0.1
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
# Publishes reasoning-dataset-lab to PyPI when a GitHub release is published.
|
|
2
|
+
# Jobs: build, publish, upload. The publish step attaches PyPI's own attestations (PEP 740). The SLSA generator is not
|
|
3
|
+
# used here: it must be referenced by tag, and this repository pins every action by commit sha (a test enforces it).
|
|
4
|
+
# Trusted publishing: PyPI trusts this repository, this file name, and the
|
|
5
|
+
# pypi environment. No API token is stored.
|
|
6
|
+
# It runs only when a GitHub release is published: there is no manual trigger, so nothing
|
|
7
|
+
# else can reach the publish job. It is not push-triggered.
|
|
8
|
+
# Undo: delete the release asset. A PyPI file is yanked.
|
|
9
|
+
# Owner: the Publisher.
|
|
10
|
+
|
|
11
|
+
name: Release
|
|
12
|
+
|
|
13
|
+
on:
|
|
14
|
+
release:
|
|
15
|
+
types: [published]
|
|
16
|
+
|
|
17
|
+
concurrency:
|
|
18
|
+
group: ${{ github.workflow }}-${{ github.ref }}
|
|
19
|
+
cancel-in-progress: false
|
|
20
|
+
|
|
21
|
+
permissions:
|
|
22
|
+
contents: read
|
|
23
|
+
|
|
24
|
+
jobs:
|
|
25
|
+
build:
|
|
26
|
+
runs-on: ubuntu-latest
|
|
27
|
+
timeout-minutes: 15
|
|
28
|
+
permissions:
|
|
29
|
+
contents: read
|
|
30
|
+
steps:
|
|
31
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
32
|
+
|
|
33
|
+
- uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2
|
|
34
|
+
with:
|
|
35
|
+
python-version: "3.15"
|
|
36
|
+
|
|
37
|
+
- name: The tag matches the version
|
|
38
|
+
run: |
|
|
39
|
+
python - <<'PY'
|
|
40
|
+
import os
|
|
41
|
+
import pathlib
|
|
42
|
+
import tomllib
|
|
43
|
+
|
|
44
|
+
ref_type = os.environ["GITHUB_REF_TYPE"]
|
|
45
|
+
ref_name = os.environ["GITHUB_REF_NAME"]
|
|
46
|
+
if ref_type != "tag":
|
|
47
|
+
raise SystemExit(
|
|
48
|
+
f"ref {ref_name!r} is a {ref_type}, not a tag. "
|
|
49
|
+
"Dispatch this workflow on the release tag."
|
|
50
|
+
)
|
|
51
|
+
version = tomllib.loads(
|
|
52
|
+
pathlib.Path("pyproject.toml").read_text(encoding="utf-8")
|
|
53
|
+
)["project"]["version"]
|
|
54
|
+
tag = ref_name[1:] if ref_name.startswith("v") else ref_name
|
|
55
|
+
if tag != version:
|
|
56
|
+
raise SystemExit(
|
|
57
|
+
f"tag {ref_name} does not match pyproject version {version}"
|
|
58
|
+
)
|
|
59
|
+
PY
|
|
60
|
+
|
|
61
|
+
- name: Build
|
|
62
|
+
run: uv build
|
|
63
|
+
|
|
64
|
+
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
|
65
|
+
with:
|
|
66
|
+
name: dist
|
|
67
|
+
path: dist/
|
|
68
|
+
|
|
69
|
+
publish:
|
|
70
|
+
needs: build
|
|
71
|
+
runs-on: ubuntu-latest
|
|
72
|
+
timeout-minutes: 15
|
|
73
|
+
environment: pypi
|
|
74
|
+
permissions:
|
|
75
|
+
id-token: write
|
|
76
|
+
actions: read
|
|
77
|
+
steps:
|
|
78
|
+
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
|
79
|
+
with:
|
|
80
|
+
name: dist
|
|
81
|
+
path: dist
|
|
82
|
+
|
|
83
|
+
- uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
|
|
84
|
+
|
|
85
|
+
upload:
|
|
86
|
+
needs: build
|
|
87
|
+
runs-on: ubuntu-latest
|
|
88
|
+
timeout-minutes: 15
|
|
89
|
+
permissions:
|
|
90
|
+
contents: write
|
|
91
|
+
steps:
|
|
92
|
+
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
|
93
|
+
with:
|
|
94
|
+
name: dist
|
|
95
|
+
path: dist
|
|
96
|
+
|
|
97
|
+
- name: Attach the wheel and the sdist
|
|
98
|
+
env:
|
|
99
|
+
GH_TOKEN: ${{ github.token }}
|
|
100
|
+
GH_REPO: ${{ github.repository }}
|
|
101
|
+
run: gh release upload "${{ github.ref_name }}" dist/*.whl dist/*.tar.gz
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
__pycache__/
|
|
2
|
+
*.py[cod]
|
|
3
|
+
*$py.class
|
|
4
|
+
*.egg-info/
|
|
5
|
+
.eggs/
|
|
6
|
+
build/
|
|
7
|
+
dist/
|
|
8
|
+
.venv/
|
|
9
|
+
venv/
|
|
10
|
+
.pytest_cache/
|
|
11
|
+
.coverage
|
|
12
|
+
.coverage.*
|
|
13
|
+
coverage.xml
|
|
14
|
+
htmlcov/
|
|
15
|
+
site/.astro/
|
|
16
|
+
site/.astro/
|
|
17
|
+
site/dist/
|
|
18
|
+
site/node_modules/
|
|
19
|
+
.polyglot-cache.json
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
3.15
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to `reasoning-dataset-lab` are documented here. The format is based on
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project follows
|
|
5
|
+
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
|
+
|
|
7
|
+
## Unreleased
|
|
8
|
+
|
|
9
|
+
## [0.4.0] - 2026-10-29
|
|
10
|
+
|
|
11
|
+
### Changed
|
|
12
|
+
|
|
13
|
+
- **First PyPI release.** `pip install reasoning-dataset-lab`. The two git-URL dependencies are now PyPI packages,
|
|
14
|
+
`aspire-si-gates==1.7.6` and `role-set==0.3.0` (PyPI does not accept git URLs). 1.7.6 is the same source as the commit
|
|
15
|
+
that was pinned, so no gate changed. role-set 0.3.0 is the first PyPI release of that package.
|
|
16
|
+
- Added PyPI metadata, a release workflow (trusted publishing), a landing page and handbook, README translations, a
|
|
17
|
+
rewritten README, and an audit of the site's npm tree in CI.
|
|
18
|
+
- Everything below, under the previous Unreleased heading, ships in this release.
|
|
19
|
+
|
|
20
|
+
### Changed
|
|
21
|
+
|
|
22
|
+
- **certify-values: `--reader-think FAMILY=SETTING`, `--parent-manifest PATH`, `rdlab pilot-b compare-arms`, and Nemotron pin `think="medium"`.** `rdlab pilot-b certify-values` gains repeatable `--reader-think FAMILY=SETTING` (`true`, `false`, `low`, `medium`, `high`, `max`) to override the pinned `think` for that reader family only (refusing unknown families or settings with exit code 2 before any Ollama call) and records `"think_overridden": true` on overridden reader rows. `--parent-manifest PATH` records `parent_manifest_sha256` (sha256 of the parent file's bytes) and `setting_tier` (e.g. `"nemotron:think=medium"`), refuses to overwrite the parent manifest or its companion review queue (`PILOT_B_CERT_PARENT_OVERWRITE_REFUSED`), and leaves `verify_manifest` verifying each addendum only as its own standalone manifest without merging verdicts across manifests. `rdlab pilot-b compare-arms PARENT ADDENDUM --reader nemotron --out PATH` writes a markdown comparison report with per-pair verdicts, a cross-table, exact-agreement counts, per-set reader pass at each setting, optional completion tokens and wall time, and length stops reported separately from verdicts (their own count per setting and their own row and column in the cross-table, never counted as a verdict or folded into disagree, with agreement rates labelled over all pairs and over pairs with a verdict at both settings). `DEFAULT_PILOT_B_MODEL_PINS["reader_1"]` (Nemotron) now pins `think="medium"` (citing Ollama v0.40.9 `model/renderers/nemotron3nano.go:30, 86-87`: `"medium"` is an effort hint the template knows, not a cap), while `verify_manifest` continues to accept r3's `think: true` manifest unchanged.
|
|
23
|
+
|
|
24
|
+
- **reader_3 (Mistral Small 3.2) answers directly (`think: false`); the manifest records each reader's think setting.** Mistral Small 3.2 has no thinking mode in Ollama, so a `think: true` request would fail on every call; the reader_3 pin now says `think=False`, and the adapter sends exactly the pin's setting (reader_1 and reader_2 keep `think: true`). The value manifest's reader rows gain `think`, and `verify_manifest` checks each recorded reader's `think` against its pin (a manifest that records none, or one that records a deliberating Mistral, fails). Manifests made before this change record no `think` and so fail the verifier until re-made; none is committed. `run_ollama_calibration_calls` takes the model's pinned `think` (it was hard-coded `true`), so calibrating Mistral sends `think: false`; an explicit value that differs from the pin is refused (`PILOT_B_CALIBRATE_THINK_OFF_FORBIDDEN` for a thinking model, `PILOT_B_CALIBRATE_THINK_MISMATCH` otherwise). The prompt, `PROMPT_SHA256`, `parse_answer`, `CONTROL_PAIRS` and the other pins are untouched.
|
|
25
|
+
|
|
26
|
+
- **Pilot A: the 12 floor and ceiling cells are buildable, on role-set d29499f.** `role-set` is re-pinned to `d29499f77a81bda144ebd51885f3c167a04667aa` (role-set #122; `pyproject.toml`, `uv.lock`, the adapter's `ROLE_SET_COMMIT`, the registry citation in `map.py`). The adapter no longer hard-codes the claim question: it reads the shortcut's `from` operator in role-set's `REGISTRY` and builds an exact (`=`) item with the claim question and a floor (`>=`) or ceiling (`<=`) item with the set question ("Which of them could be ...?"), because a claim cannot separate "at least k" from "exactly k" while the set of possible groups can. It filters the t10_variety catalogue by question, `ifs` and operator, and passes `target_shortcut` to `build_t10` for the set cells only; the exact cells keep the untargeted build, so their items and proofs are byte-identical to before (a pinned hash in the tests). A role-set without `target_shortcut` is refused when the generator is built (`ROLE_SET_PIN_LACKS_TARGET_SHORTCUT`). `examples/t10-pilot-a.map.yaml` now has 24 buildable cells (capacity 5 each) and no `role-set PR pending` cells; to keep its splits (train 48, eval 12) and shares, the 12 exact cells' targets are 3 (were 5) and the 12 new cells' targets are 2, total 60. The receipts of the exact cells name the new generator commit, which is the only field of theirs that changes.
|
|
27
|
+
- **certify-values: the licence-registry gate, a registry sha in the manifest, and a role fallback that refuses.** `rdlab pilot-b certify-values` now requires `--licence-registry <absolute path>` and, before any Ollama call, validates exactly the reader pins that will run against it, so a real certify-values run refuses (`PILOT_B_LICENCE_NOT_IN_REGISTRY`) when rnd's registry lacks a reader's licence entry (until now only `rdlab models --verify-live` did). The manifest records `licence_registry_sha256`, and `verify_manifest` requires it; `rdlab pilot-b verify-values-manifest --licence-registry <path>` recomputes it from the file. `OllamaBlindReaderAdapter` built without a role now maps the model family to its role (nemotron to reader_1, granite to reader_2, mistral to reader_3) and refuses any other family with `PILOT_B_UNKNOWN_READER_FAMILY`, instead of defaulting to reader_1. Earlier manifests carry no registry sha and no longer verify.
|
|
28
|
+
|
|
29
|
+
- **reader_3: Mistral Small 3.2 24B (Apache-2.0) as the c2 second-reader family; no run yet.** `DEFAULT_PILOT_B_MODEL_PINS` gains `reader_3` (`mistral-small3.2:24b`, manifest digest `5a408ab5...`), `rdlab pilot-b certify-values --readers` accepts `mistral`, and the manifest verifier allows a reader_3 that ran. A manifest made by the other two readers still verifies. The licence registry needs the entry `mistral-small-3.2-24b-hf-card`, so a real (non-stub) run refuses until rnd's registry carries it.
|
|
30
|
+
|
|
31
|
+
- **control pairs v2: six arguable pairs replaced; earlier control results don't carry over.** A control pair must be compatible beyond argument, so the six whose key a careful reader could rightly dispute (chromoly then ferritic steel, a carbide coating then ceramic, docked then moored, sealed then waterproof, acoustic then sonic, a dry gas seal then mechanical seal) are replaced by chromoly then alloy steel, an alumina coating then ceramic, a steel hull then metal, IP67 then dust-tight, R-134a then HFC-134a, and a silicon-carbide seal face then ceramic. The other fourteen, the 0.80 threshold and the pass rule are unchanged. The manifest records `control_pairs_sha256`, and `verify_manifest` recomputes it, so a manifest made under the old table fails verification.
|
|
32
|
+
|
|
33
|
+
- **certification asks its own question.** The first full run showed the triad blind-read prompt (does the claim follow from the material?) makes a reader answer "unsupported" to a compatible but unstated claim (ester then synthetic), so it cannot tell a contradiction from a gap. `rdlab pilot-b certify-values` now asks: "Statement 1: The <attribute> of <entity> is <v1>. Statement 2: ... Could both statements be true of the same <entity> at the same time?", answered with exactly one word, `compatible` / `incompatible` / `unsure`, parsed strictly (case and surrounding whitespace ignored; anything else is a parse error and counts toward the error stop). A pair is a contradiction only on `incompatible`; a control pair passes on `compatible` or `unsure`. The 20 control pairs are unchanged. Same pins, `think` on, `num_predict` 3072, `num_ctx` 4096. The manifest records the sha256 of the prompt text, and `verify_manifest` rejects a manifest made with any other prompt. `verify_manifest` also recomputes each reader's control passes from the pairs and enforces the 0.80 threshold itself, instead of trusting the recorded `passed`. The real reader adapter gains `chat_raw` for callers that build their own messages. The CLI no longer writes the manifest a second time.
|
|
34
|
+
|
|
35
|
+
- **generator: every categorical value set sits on one axis and its values exclude each other.** An unsupported claim swaps the true value for another from the same set, so it is a contradiction only if both cannot be true. An audit of all 50 value sets (10 domains x 5) found most of them mixed an axis (a material beside a type, a mode beside a sub-process) or held a class beside its member (`synthetic` beside `ester`, `mineral` beside `paraffinic`, `air` beside `nitrogen`, `grounded` in two transformer sets), which makes an `unsupported` claim true and its key wrong (Pilot B stub 10d). All 50 sets were rewritten as 5 values on one declared axis (`VALUE_SET_AXES`); the 10 state sets hold top-level operating modes only, and no value sits in two sets of a domain. Some attribute names changed with their axis (for example `grid status` is now `turbine mode`, `valve state` is `pump state`), so facts and tables generated from a seed differ from earlier builds. `VALUE_SUBSUMPTIONS` records the known overlaps as a regression lint, `EXCLUSIVE_BY_MEANING` the close pairs reviewed with a reason, and `tests/test_pilot_b_value_sets.py` fails on a covering value beside its member, on one value spelled inside another, and on a value in two sets of a domain.
|
|
36
|
+
- **The exclusivity guard is not enforced yet.** The lint above only catches overlaps someone has written down; new or replacement values get no protection from it. The enforcing guard is the certification below, and it does not switch on until the certification PR (the manifest from a real run on the readers, `is_disjoint_value` reading it, and a CI gate that every set is certified with a matching hash). Until then, do not read this change as protection it does not give.
|
|
37
|
+
- **value certification tooling, with no certification claimed.** `pilot_b/value_certification.py`: `rdlab pilot-b certify-values` asks two local reader families whether every directed pair in a set (20 for 5 values) reads as a contradiction, one reader loaded at a time, with `--num-ctx`, `--num-predict`, `--readers`, a calibration of the first calls that stops the run when the projected time passes `--wall-cap-s`, and an error-rate stop (`--error-rate-stop`, 2% by default). It writes a content-hashed manifest of the admitted sets and a review queue of the others. `rdlab pilot-b verify-values-manifest` loads a manifest and fails when any set is uncertified or its hash no longer matches. Tested with fake readers; no manifest is committed.
|
|
38
|
+
- **certification: provenance, partial results, unload.** `verify_manifest` now requires a complete run, at least two readers from distinct families whose name, digest and family equal the pinned reader models in `DEFAULT_PILOT_B_MODEL_PINS` (a fake reader cannot match), and verdict keys equal to the manifest's reader names. A run that stops (error rate, wall cap) writes its manifest marked `"complete": false` with the stop code, the review queue, and `<queue>.failures.jsonl` (reader, pair, done_reason, completion_tokens, the tail of the raw text, or the error code); every exit path unloads the reader it loaded; the calibration projection also runs when one reader's jobs equal `--calibration-calls`. Defaults `--num-predict 3072`, `--num-ctx 4096` (thinking tokens count against `num_predict`; measured up to 2,019 in stub 10d); `--num-ctx` is a cap.
|
|
39
|
+
- **certification: length routing and a negative control.** A `length` or `max_num_ctx_exceeded` stop means not certified: the pair goes to the review queue with its reason and the failing call is logged, but it does not count toward `--error-rate-stop`, which counts only transport and parse errors. Before its certification calls each reader answers 20 compatible pairs (a member then its class, a synonym, an implication) through the same prompt and settings; it passes with at least 16 of 20 answers that are not "unsupported" (`--control-pass-threshold`, default 0.80; length stops and errors fail). If a reader fails, the run certifies nothing: the manifest says `"control_failed": true`, every set goes to the review queue, and the reader is unloaded. `manifest["control"]` records each reader's passes, threshold and every pair's verdict, done_reason and tokens, and `verify_manifest` rejects a manifest with no passing control.
|
|
40
|
+
- **reports record the render-failure gate.** `pilot_b_report.json` carries `render_failure_rate` and `max_render_failure_rate` (schema updated).
|
|
41
|
+
- Four more sets re-cut after the Argon review: reactors/cladding metal, compressors/rotor coating, boilers/tube alloy, chillers/condenser cooling (they need re-certifying).
|
|
42
|
+
- **render: the value-before-entity check is dropped, and intervals may be written two ways.** One entity per sentence already stops a value binding to the wrong entity. An interval may be "between <lo> and <hi>" or "from <lo> to <hi>". The writer prompts ask for "between" and for a direct verb (is, operates in, uses, has) on exact facts, never a hedged link such as "characterized by" or "associated with".
|
|
43
|
+
- **gate: render failures are their own rate.** They no longer count against every verdict class: class rates are over the triads that rendered, and `max_render_failure_rate` (default 0.50) halts the build with `PILOT_B_RENDER_FAILURE_RATE_ABOVE_CEILING`.
|
|
44
|
+
- **`mutate_fact_value` refuses instead of falling back.** A value outside its own attribute's set raises `PILOT_B_NO_EXCLUSIVE_VALUE`; the cross-attribute swap and the `non-<v>` / `uncoated-<v>` strings are gone.
|
|
45
|
+
|
|
46
|
+
- **Python: `requires-python` is `>=3.13` (was `>=3.12,<3.13`); Python 3.12 is dropped.** 3.15 is the main target (`.python-version`); CI runs the suite on 3.15 (`test`, with the Atlas check), 3.13 (`test-py313`) and 3.14 (`test-py314`), each from `uv sync --frozen`. Coverage (floor 90) and a Codecov upload run on the 3.15 leg only; the 3.13 and 3.14 legs run the tests without coverage. Every action is pinned by commit SHA with a version comment, at its current major (checkout v7.0.1, setup-uv v10.3.2, setup-node v7.1.0, codecov-action v7.1.1, read 2026-10-25). `role-set` is re-pinned to its commit that supports `>=3.13` (`d105f3d`, role-set PR #121; to be re-pointed to the merge commit). The Part F parity test now expects `>=3.13`. On 3.15, PyYAML 6.0.3 installs from its source distribution until it publishes cp315 wheels.
|
|
47
|
+
|
|
48
|
+
- sealing: the per-build seed is kept in the sealed store.
|
|
49
|
+
- sealing: the render inputs are validated at write and at verify, the report is cross-checked against them, and the schema maps are closed.
|
|
50
|
+
- sealing: more render-input rules (unique digests, constant call fields, catalogue domains, regular whitespace) and a coverage check on the report.
|
|
51
|
+
- sealing: the writer's whitespace is regularised before anything reads it, each call's model must be the recorded one, and the report's coverage numbers must add up.
|
|
52
|
+
- generator: per-build secret variation in table values and homogeneous triad surface features; variety: surface-feature balance and triad-level assignment baseline checks.
|
|
53
|
+
|
|
54
|
+
## [0.3.0] - 2026-10-25
|
|
55
|
+
|
|
56
|
+
### Added
|
|
57
|
+
|
|
58
|
+
- Pilot B builder (`reasoning_dataset_lab.pilot_b`):
|
|
59
|
+
- **Part A (`fact_tables.py`):** Typed atomic fact tables (`number`, `date`, `interval`, `state`, `attribute`) with `closure` tags (`open` or `closed`), contrast triad construction (`supported`, `unsupported`, `cannot_tell`), and code-proved keys (`fact_restatement`, `disjoint_mutation`, `interval_interior_refinement`, `open_excerpt_unmentioned_attribute`).
|
|
60
|
+
- **Part B (`rendering.py`):** Writer adapter protocol, deterministic writer and adversarial fakes (`FlippedBoundWriterFake`, `RelationalSwapWriterFake`, `UnlistedNumberWriterFake`, `UnlistedProperNounWriterFake`), sentence-level fact binding, polarity and quantifier direction checks (`at least` vs `at most`, negation, `only`, `between`), ungrounded number/proper-noun refusals, and structural token whitelisting (`DESIGN.md` ruling 8a, §11 step 3, §2.4).
|
|
61
|
+
- **Part C (`unanimity.py`):** Blind-reader adapter protocol, key-free blind-read prompts, two-family unanimity gate, whole-triad pass/fail routing to `review_queue.jsonl`, hard refusal on same-family self-verification, reader family collision, Muse-Glimmer, and Argon/Gemini, and `min_pass_rate_per_class: 0.50` build halt (`DESIGN.md` ruling 2, ruling 8c, §6).
|
|
62
|
+
- **Part D (`waves_and_sealing.py`):** Single-model wave runner (`render`, `word`, `reader_1`, `reader_2`) enforcing non-interleaved wave execution and per-call thinking token caps (`ruling 8b`), seeded item shuffling, aggregate-only judgment provenance (`§2.1`), external `keys.sealed.jsonl`, and content-based plus field-name sealing verification (`ruling 5a`).
|
|
63
|
+
- **Part E (`models_and_build.py`, `ollama.py`, and CLI):** Pinned Ollama tags and manifest/layer digests for `qwen3.6:35b`, `nemotron-3.5-lightning:30b`, and `granite4.2:8b` with configurable `licence_id` checked against `licences/registry.json`, real loopback Ollama writer and blind-reader adapters (`OllamaWriterAdapter`, `OllamaBlindReaderAdapter`, interlock #140 loopback guard, pre-call `/api/tags` digest check with no auto-pull, per-wave `keep_alive: 0` model unload, and `is_fake` sealing guard), `rdlab models --verify-live`, `rdlab pilot-b --stub N`, and the Pilot B build report (`coverage`, `key_proof_rate: 1.0`, `gate_pass_rate_per_class`, per-wave timing/tokens, 50-triad full-build estimate, and 20-item human sample).
|
|
64
|
+
|
|
65
|
+
### Changed
|
|
66
|
+
|
|
67
|
+
- **Part F (`pyproject.toml`):** Narrow `requires-python` to `">=3.12,<3.13"` to match `role-set` at its pinned commit `2b26840`, with a regression test verifying that the lab never allows a Python version that `role-set` refuses.
|
|
68
|
+
|
|
69
|
+
## [0.2.0] - 2026-10-24
|
|
70
|
+
|
|
71
|
+
### Added
|
|
72
|
+
|
|
73
|
+
- Generator interface (`reasoning_dataset_lab.generators.base`): `Generator` protocol (`plan(cell, canon, n, seed) -> list[SlotSpec]` and `build(slot) -> BuiltItem`), typed `SlotSpec`, `ItemRecord`, `ProofRecord`, `ReceiptRecord`, `BuiltItem`, `MapCell`, and deterministic `MockGenerator` (`DESIGN.md §4`).
|
|
74
|
+
- `role-set` generator adapter (`reasoning_dataset_lab.generators.role_set`): `RoleSetGenerator` driving `role-set` at pinned commit `2b26840` unchanged (`role_set.generate`, `role_set.solvefirst`, `role_set.mock.MockWriter`, `role_set.puzzle.check_puzzle`, `role_set.assemble.assemble`) with `aspire-si-gates` pinned at `8ed78f2`.
|
|
75
|
+
|
|
76
|
+
### Fixed
|
|
77
|
+
|
|
78
|
+
- Enforce `f"{trait}.{wrong_family}"` join against `role_set.generate.shortcuts.REGISTRY` (`src/role_set/generate/shortcuts.py:21-29@2b26840`) in both `rdlab map check` and `RoleSetGenerator` when `wrong_family != "none"` (`dx` Fix #1).
|
|
79
|
+
- Validate that all files under `canon/` are UTF-8 text (`errors="strict"`, no NUL bytes, allowed extensions `.yaml`, `.yml`, `.json`, `.txt`, `.md`) in both `rdlab map check` and `rdlab project fingerprint` (`dx` Fix #3).
|
|
80
|
+
|
|
81
|
+
## [0.1.1] - 2026-10-24
|
|
82
|
+
|
|
83
|
+
### Fixed
|
|
84
|
+
|
|
85
|
+
- Validate `axes.wrong_family` values against `role-set`'s registered shortcut IDs (`exact_read_as_floor`, `exact_read_as_ceiling`, `floor_read_as_exact`, `ceiling_read_as_exact`, and `none`, pinned to `role_set/generate/shortcuts.py:21-29@4033ebd`), refusing unknown values with `MAP_UNKNOWN_WRONG_FAMILY` while keeping `MAP_SHORTHAND_WRONG_FAMILY` for the four shorthand aliases.
|
|
86
|
+
- Close `null` and type bypasses across `map.yaml` (`splits`, `canon`, `cells[].reason` when `capacity > 0`, `shares[].min_share`, `shares[].max_share`, `acceptance.unanimity_rate`) and the five project files (`project.json` `display_name`/`version`, `lanes.json` `default_lane` and `lanes[].kind`, `rubric.json` `plugin`), and require all `sha256` fields in `map.yaml` `canon` and `rubric.json` `pins` to match `^[0-9a-f]{64}$`.
|
|
87
|
+
- Make `compute_config_fingerprint` cross-platform stable across Windows (`CRLF`) and Linux (`LF`) checkouts by stripping any leading UTF-8 BOM and normalizing `CRLF`/`CR` line endings to `LF` (`\n`) before hashing project, map, and canon files.
|
|
88
|
+
|
|
89
|
+
## [0.1.0] - 2026-10-24
|
|
90
|
+
|
|
91
|
+
### Added
|
|
92
|
+
|
|
93
|
+
- Package scaffold (`src/reasoning_dataset_lab/`) for Python 3.12 with the `rdlab` CLI entry point.
|
|
94
|
+
- `map.yaml` schema validator and canonical map SHA-256 computation (`reasoning_dataset_lab.map` and `rdlab map check`), enforcing cell `target`, `capacity`, and `reason` (required when `capacity == 0`), share floors and ceilings, variety caps, `acceptance` rules (`min_pass_rate_per_class`, and `unanimity_rate` restricted to `judged` kinds), unknown-key refusals, and structured `{code, message, hint}` errors with exit codes `0` (clean), `1` (refused), and `2` (usage).
|
|
95
|
+
- Example map `examples/t10-pilot-a.map.yaml` covering T10's full intended 24-cell map across all four `role-set` shortcut-registry wrong families (`exact_read_as_floor`, `exact_read_as_ceiling`, `floor_read_as_exact`, `ceiling_read_as_exact`), with 12 buildable cells summing to 60 items and 12 floor/ceiling cells at `capacity: 0, reason: "role-set PR pending"`.
|
|
96
|
+
- Minimal loaders for the five project files (`project.json`, `constitution.json`, `terminology.json`, `lanes.json` derived from the map's `kind` axis, and `rubric.json`) and deterministic config fingerprint (`rdlab project fingerprint`) covering the five files, the map, and canon in fixed order.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 mcp-tool-shop
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,125 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: reasoning-dataset-lab
|
|
3
|
+
Version: 0.4.0
|
|
4
|
+
Summary: Map-first dataset lab for proven reasoning puzzles, verification items, and composed dataset parts.
|
|
5
|
+
Project-URL: Homepage, https://mcp-tool-shop-org.github.io/reasoning-dataset-lab/
|
|
6
|
+
Project-URL: Repository, https://github.com/mcp-tool-shop-org/reasoning-dataset-lab
|
|
7
|
+
Project-URL: Changelog, https://github.com/mcp-tool-shop-org/reasoning-dataset-lab/blob/main/CHANGELOG.md
|
|
8
|
+
Project-URL: Issues, https://github.com/mcp-tool-shop-org/reasoning-dataset-lab/issues
|
|
9
|
+
Author: mcp-tool-shop
|
|
10
|
+
License-Expression: MIT
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Keywords: dataset,evaluation,provenance,puzzles,reasoning,synthetic-data,training-data
|
|
13
|
+
Classifier: Development Status :: 3 - Alpha
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Intended Audience :: Science/Research
|
|
16
|
+
Classifier: Operating System :: OS Independent
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
19
|
+
Requires-Python: >=3.13
|
|
20
|
+
Requires-Dist: aspire-si-gates==1.7.6
|
|
21
|
+
Requires-Dist: pyyaml==6.0.3
|
|
22
|
+
Requires-Dist: role-set==0.3.0
|
|
23
|
+
Provides-Extra: dev
|
|
24
|
+
Requires-Dist: coverage==7.16.2; extra == 'dev'
|
|
25
|
+
Requires-Dist: pytest-cov==7.0.0; extra == 'dev'
|
|
26
|
+
Requires-Dist: pytest==9.0.3; extra == 'dev'
|
|
27
|
+
Description-Content-Type: text/markdown
|
|
28
|
+
|
|
29
|
+
<p align="center">
|
|
30
|
+
<a href="README.md">English</a> | <a href="README.ja.md">日本語</a> | <a href="README.zh.md">中文</a> | <a href="README.es.md">Español</a> | <a href="README.fr.md">Français</a> | <a href="README.hi.md">हिन्दी</a> | <a href="README.it.md">Italiano</a> | <a href="README.pt-BR.md">Português (BR)</a>
|
|
31
|
+
</p>
|
|
32
|
+
|
|
33
|
+
<p align="center">
|
|
34
|
+
<img src="https://raw.githubusercontent.com/mcp-tool-shop-org/brand/main/logos/reasoning-dataset-lab/readme.png" alt="reasoning-dataset-lab" width="400">
|
|
35
|
+
</p>
|
|
36
|
+
|
|
37
|
+
<p align="center">
|
|
38
|
+
<a href="https://github.com/mcp-tool-shop-org/reasoning-dataset-lab/actions/workflows/ci.yml"><img src="https://github.com/mcp-tool-shop-org/reasoning-dataset-lab/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
|
|
39
|
+
<a href="https://pypi.org/project/reasoning-dataset-lab/"><img src="https://img.shields.io/pypi/v/reasoning-dataset-lab" alt="PyPI"></a>
|
|
40
|
+
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue" alt="MIT License"></a>
|
|
41
|
+
<a href="https://mcp-tool-shop-org.github.io/reasoning-dataset-lab/"><img src="https://img.shields.io/badge/docs-landing_page-blue" alt="Landing page"></a>
|
|
42
|
+
</p>
|
|
43
|
+
|
|
44
|
+
**A map-first lab for datasets of puzzles, quizzes and verification items, where code proves the answer key before any
|
|
45
|
+
model words the item.** `rdlab` is the command line. It plans a dataset as a pre-registered map, builds items whose keys
|
|
46
|
+
are constructed and proven, asks a local model only to put them into words, and records the provenance of every item.
|
|
47
|
+
|
|
48
|
+
A key that a model chose is an opinion. A key that code built from a hidden state, and proved by a solver, is a fact
|
|
49
|
+
about the item. `rdlab` is built so the key never depends on a judge.
|
|
50
|
+
|
|
51
|
+
## Install
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
pip install reasoning-dataset-lab
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Python 3.13 or newer (3.15 is the main target; CI tests 3.13, 3.14 and 3.15). It depends on
|
|
58
|
+
[aspire-si-gates](https://pypi.org/project/aspire-si-gates/) and [role-set](https://pypi.org/project/role-set/).
|
|
59
|
+
|
|
60
|
+
## What you can do without any model
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
rdlab map check examples/t10-pilot-a.map.yaml
|
|
64
|
+
rdlab project fingerprint . --map examples/t10-pilot-a.map.yaml
|
|
65
|
+
rdlab --help
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
- `rdlab map check` validates a dataset map (its axes, every cell's target against the generator's capacity, share
|
|
69
|
+
floors and ceilings, variety bounds, acceptance criteria) and prints the map's canonical SHA-256. Exit 0 is a valid
|
|
70
|
+
map. Exit 1 is a refusal, as a `{code, message, hint}` JSON object on stderr. Exit 2 is a usage or file error.
|
|
71
|
+
- `rdlab project fingerprint` hashes the five project files, the map and `canon/` in a fixed order, so a build can say
|
|
72
|
+
exactly what it was built from.
|
|
73
|
+
- `rdlab models` lists the models a Pilot B build pins, and `rdlab variety` computes the deterministic variety report
|
|
74
|
+
for a build.
|
|
75
|
+
|
|
76
|
+
## What needs a GPU and local models
|
|
77
|
+
|
|
78
|
+
`rdlab pilot-b` builds the Pilot B critic test set: triads of supported, unsupported and cannot-tell claims made from
|
|
79
|
+
typed fact tables. It runs real local models through Ollama on loopback (a writer and two blind readers from different
|
|
80
|
+
families), needs a GPU, and needs a licence registry file that says each model may be used for evaluation. It never pulls
|
|
81
|
+
a model for you and never calls a cloud endpoint. The full flags, the memory plan and the `num_predict` settings are in
|
|
82
|
+
the [handbook](https://mcp-tool-shop-org.github.io/reasoning-dataset-lab/handbook/).
|
|
83
|
+
|
|
84
|
+
## The map-first principle
|
|
85
|
+
|
|
86
|
+
1. **Prove first, word second.** Code constructs a formal state and proves the key before a model writes any prose. A
|
|
87
|
+
kind of item that cannot be proved by construction is marked `judged` and enters only through a multi-family
|
|
88
|
+
unanimity gate.
|
|
89
|
+
2. **Pre-registered cells and capacity.** `map.yaml` declares every intended cell, its target, the generator's capacity
|
|
90
|
+
for it, share floors and ceilings, and the variety bounds, before any item exists. A cell the generator cannot yet
|
|
91
|
+
build stays in the map with `capacity: 0` and a stated reason, so a coverage gap is visible and the map is never
|
|
92
|
+
shrunk to hide it.
|
|
93
|
+
3. **One fingerprint.** A project binds five files, the map and its canon, and a deterministic hash says which.
|
|
94
|
+
|
|
95
|
+
## What has been measured
|
|
96
|
+
|
|
97
|
+
The generator's value sets (the lists a claim's values are swapped within) are certified before use. In the v3 run, 44
|
|
98
|
+
of 50 sets certified, against 26 before the sets were repaired. Here "certified" means two blind readers from different
|
|
99
|
+
families called every ordered pair a contradiction, and an independent blind key had already agreed. It is evidence, not
|
|
100
|
+
proof: model readers were part of it. The wording of a rendered item can still differ from its key, and a person's
|
|
101
|
+
hand-read is the check for that. Nothing `rdlab` writes is training data until a person has checked it. The run and its
|
|
102
|
+
files are in [certification/v3](certification/v3).
|
|
103
|
+
|
|
104
|
+
## Security and data
|
|
105
|
+
|
|
106
|
+
- **Reads:** the map, project files and canon you point it at.
|
|
107
|
+
- **Writes:** the output directory you give it.
|
|
108
|
+
- **Network:** only Ollama on loopback (`127.0.0.1`), and only for `pilot-b` and `models --verify-live`. Anything else
|
|
109
|
+
is refused. No telemetry.
|
|
110
|
+
- **Permissions:** none beyond those files and that local port.
|
|
111
|
+
|
|
112
|
+
Report a vulnerability privately; see [SECURITY.md](SECURITY.md).
|
|
113
|
+
|
|
114
|
+
## Status
|
|
115
|
+
|
|
116
|
+
Version 0.4.0, the first PyPI release. It is alpha: interfaces and data shapes can change without notice. Coverage of
|
|
117
|
+
this repository's own tests is 95%, with the floor at 90.
|
|
118
|
+
|
|
119
|
+
## License
|
|
120
|
+
|
|
121
|
+
MIT, © mcp-tool-shop. See [LICENSE](LICENSE).
|
|
122
|
+
|
|
123
|
+
---
|
|
124
|
+
|
|
125
|
+
Built by [MCP Tool Shop](https://mcp-tool-shop.github.io/)
|