agentrust-trace-tests 0.5.1__tar.gz → 0.6.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/.github/dependabot.yml +4 -0
- agentrust_trace_tests-0.6.0/.github/workflows/actionlint.yml +41 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/.github/workflows/ci.yml +10 -2
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/.github/workflows/codeql.yml +4 -4
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/.github/workflows/contributor-check.yml +1 -1
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/.github/workflows/docs.yml +9 -5
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/.github/workflows/release.yml +2 -2
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/.github/workflows/require-maintainer-approval.yml +1 -1
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/.github/workflows/scorecard.yml +2 -2
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/CHANGELOG.md +53 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/CONTRIBUTING.md +8 -0
- agentrust_trace_tests-0.6.0/LIMITATIONS.md +103 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/PKG-INFO +28 -7
- agentrust_trace_tests-0.6.0/PRIVACY.md +9 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/README.md +27 -6
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/SPONSORS.md +4 -4
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/docs/error-codes.md +15 -2
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/docs/levels.md +39 -9
- agentrust_trace_tests-0.6.0/docs/modules/tr-anc.md +10 -0
- agentrust_trace_tests-0.6.0/docs/modules/tr-apr.md +51 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/docs/modules/tr-env.md +1 -0
- agentrust_trace_tests-0.6.0/docs/modules/tr-pol.md +32 -0
- agentrust_trace_tests-0.6.0/docs/modules/tr-rte.md +16 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/docs/modules.md +5 -4
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/docs/quickstart.md +24 -2
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/docs/tutorials/writing-conformance-tests.md +2 -2
- agentrust_trace_tests-0.6.0/hooks/seo.py +102 -0
- agentrust_trace_tests-0.6.0/index.md +107 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/mkdocs.yml +45 -24
- agentrust_trace_tests-0.6.0/overrides/main.html +125 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/pyproject.toml +1 -1
- agentrust_trace_tests-0.6.0/requirements/docs.txt +597 -0
- agentrust_trace_tests-0.6.0/requirements/release.in +4 -0
- agentrust_trace_tests-0.6.0/requirements/release.txt +18 -0
- agentrust_trace_tests-0.6.0/requirements/test.in +12 -0
- agentrust_trace_tests-0.6.0/requirements/test.txt +512 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/requirements-docs.txt +2 -2
- agentrust_trace_tests-0.6.0/robots.txt +38 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/schemas/trace-claim.json +41 -1
- agentrust_trace_tests-0.6.0/src/trace_tests/accounting.py +1099 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/src/trace_tests/cli.py +166 -27
- agentrust_trace_tests-0.6.0/src/trace_tests/inclusion.py +110 -0
- agentrust_trace_tests-0.6.0/src/trace_tests/modules/tr_anc.py +81 -0
- agentrust_trace_tests-0.6.0/src/trace_tests/modules/tr_apr.py +235 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/src/trace_tests/modules/tr_env.py +24 -0
- agentrust_trace_tests-0.6.0/src/trace_tests/modules/tr_pol.py +302 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/src/trace_tests/modules/tr_rte.py +2 -2
- agentrust_trace_tests-0.6.0/src/trace_tests/modules/tr_sca.py +101 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/src/trace_tests/modules/tr_sig.py +50 -6
- agentrust_trace_tests-0.6.0/src/trace_tests/modules/unverified.py +65 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/src/trace_tests/report.py +236 -55
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/src/trace_tests/result.py +4 -3
- agentrust_trace_tests-0.6.0/src/trace_tests/runner.py +194 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/conftest.py +55 -0
- agentrust_trace_tests-0.6.0/tests/test_canonicalization_refusals.py +100 -0
- agentrust_trace_tests-0.6.0/tests/test_digest_parity.py +151 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/test_docs_match_the_modules.py +113 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/test_enum_parity.py +5 -3
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/test_findings_are_self_consistent.py +2 -2
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/test_level0_negative.py +142 -1
- agentrust_trace_tests-0.6.0/tests/test_level_failure_policy.py +65 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/test_modules_never_raise.py +70 -8
- agentrust_trace_tests-0.6.0/tests/test_obligation_accounting.py +1577 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/test_openshell_import.py +14 -2
- agentrust_trace_tests-0.6.0/tests/test_policy_resolution.py +246 -0
- agentrust_trace_tests-0.6.0/tests/test_policy_resolution_cli.py +169 -0
- agentrust_trace_tests-0.6.0/tests/test_policy_resolution_completeness.py +242 -0
- agentrust_trace_tests-0.6.0/tests/test_policy_resolution_reproduces.py +106 -0
- agentrust_trace_tests-0.6.0/tests/test_tr_anc_inclusion.py +177 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/unit/test_cli.py +5 -1
- agentrust_trace_tests-0.6.0/tests/unit/test_cli_policy_dir.py +136 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/unit/test_tr_anc.py +10 -2
- agentrust_trace_tests-0.6.0/tests/unit/test_tr_apr.py +233 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/unit/test_tr_env.py +33 -0
- agentrust_trace_tests-0.6.0/tests/unit/test_tr_pol_resolution.py +184 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/unit/test_tr_sca.py +13 -0
- agentrust_trace_tests-0.6.0/tests/vectors/invalid_canonical_integer_out_of_range.json +70 -0
- agentrust_trace_tests-0.6.0/tests/vectors/invalid_canonical_lone_surrogate.json +70 -0
- agentrust_trace_tests-0.6.0/tests/vectors/invalid_canonical_non_finite_float.json +70 -0
- agentrust_trace_tests-0.6.0/tests/vectors/invalid_canonical_plain_trace.json +34 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/.gitattributes +13 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/01-no-policy-uri.json +61 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/02-resolved-and-matches.json +62 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/03-digest-mismatch-minimal-mutation.json +62 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/04-digest-mismatch-different-object.json +62 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/05-referent-unreachable-no-route.json +62 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/06-resolved-and-matches-sha384.json +62 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/07-digest-bound-to-other-referent.json +62 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/08-policy-uri-is-a-relative-reference.json +62 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/09-sha384-bound-to-other-referent.json +62 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/10-policy-uri-carries-a-space.json +62 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/11-referent-unreachable-route-fails.json +62 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/README.md +192 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/gen_policy_resolution.py +540 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/policies/policy-bundle-base.json +18 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/policies/policy-bundle-onebyte.json +18 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/policies/policy-bundle-other.json +21 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/policies/policy-bundle-unrelated.json +13 -0
- agentrust_trace_tests-0.6.0/tests/vectors/policy-resolution/resolutions.json +7 -0
- agentrust_trace_tests-0.6.0/tests/vectors/valid_appraisal_full.json +39 -0
- agentrust_trace_tests-0.5.1/PRIVACY.md +0 -9
- agentrust_trace_tests-0.5.1/docs/modules/tr-anc.md +0 -9
- agentrust_trace_tests-0.5.1/docs/modules/tr-pol.md +0 -10
- agentrust_trace_tests-0.5.1/docs/modules/tr-rte.md +0 -11
- agentrust_trace_tests-0.5.1/index.md +0 -83
- agentrust_trace_tests-0.5.1/overrides/main.html +0 -81
- agentrust_trace_tests-0.5.1/src/trace_tests/modules/tr_anc.py +0 -34
- agentrust_trace_tests-0.5.1/src/trace_tests/modules/tr_pol.py +0 -42
- agentrust_trace_tests-0.5.1/src/trace_tests/modules/tr_sca.py +0 -43
- agentrust_trace_tests-0.5.1/src/trace_tests/runner.py +0 -56
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/.github/CODEOWNERS +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/.gitignore +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/CNAME +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/CODE_OF_CONDUCT.md +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/LICENSE +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/SECURITY.md +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/docs/assets/icon.svg +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/docs/assets/og.png +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/docs/modules/tr-sca.md +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/docs/modules/tr-sig.md +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/docs/modules/tr-txn.md +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/docs/tutorials/ci-integration.md +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/measurement/README.md +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/measurement/REPORT.md +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/measurement/scripts/enum_drift.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/measurement/scripts/mutate_modules.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/src/trace_tests/__init__.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/src/trace_tests/loader.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/src/trace_tests/modules/__init__.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/src/trace_tests/modules/tr_txn.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/__init__.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/test_canonicalization_boundary.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/test_level0.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/test_level1.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/test_level2.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/test_measurement_harness.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/test_packaged_schema_accepts_real_records.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/test_report.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/test_schema.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/test_software_only_platform.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/unit/__init__.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/unit/test_loader.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/unit/test_runner.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/unit/test_tr_pol.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/unit/test_tr_rte.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/unit/test_tr_sig.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/unit/test_tr_txn.py +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/vectors/canonicalization/01-non-ascii-values.json +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/vectors/canonicalization/02-non-bmp-values.json +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/vectors/canonicalization/03-utf16-key-order.json +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/vectors/canonicalization/04-utf16-key-order-nested.json +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/vectors/invalid_missing_runtime.json +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/vectors/invalid_wrong_profile.json +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/vectors/signed_delegated_hop.json +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/vectors/signed_root.json +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/vectors/valid_cmcp_runtime.json +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/vectors/valid_level0.json +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/vectors/valid_level0_with_transcript.json +0 -0
- {agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/tests/vectors/valid_openshell_import.json +0 -0
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
name: Workflow lint
|
|
2
|
+
|
|
3
|
+
# actionlint catches what a YAML parser cannot: duplicate mapping keys that
|
|
4
|
+
# make GitHub refuse to load a workflow, invalid ${{ }} expressions, and shell
|
|
5
|
+
# problems inside run: blocks. A duplicate env: key silently broke this repo
|
|
6
|
+
# family's release workflow once; yaml.safe_load keeps the last value without
|
|
7
|
+
# complaining, so local validation passed while Actions rejected the file.
|
|
8
|
+
on:
|
|
9
|
+
push:
|
|
10
|
+
branches: [main]
|
|
11
|
+
paths: ['.github/workflows/**']
|
|
12
|
+
pull_request:
|
|
13
|
+
paths: ['.github/workflows/**']
|
|
14
|
+
workflow_dispatch:
|
|
15
|
+
|
|
16
|
+
permissions:
|
|
17
|
+
contents: read
|
|
18
|
+
|
|
19
|
+
jobs:
|
|
20
|
+
actionlint:
|
|
21
|
+
runs-on: ubuntu-latest
|
|
22
|
+
steps:
|
|
23
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
24
|
+
|
|
25
|
+
# Fetched and checksum-verified rather than run as a third-party action,
|
|
26
|
+
# so this check adds no new action to the supply chain it exists to guard.
|
|
27
|
+
- name: Install actionlint
|
|
28
|
+
env:
|
|
29
|
+
ACTIONLINT_VERSION: 1.7.12
|
|
30
|
+
ACTIONLINT_SHA256: 8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8
|
|
31
|
+
run: |
|
|
32
|
+
set -euo pipefail
|
|
33
|
+
archive="actionlint_${ACTIONLINT_VERSION}_linux_amd64.tar.gz"
|
|
34
|
+
curl -sSfL -o "$archive" \
|
|
35
|
+
"https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}/${archive}"
|
|
36
|
+
echo "${ACTIONLINT_SHA256} ${archive}" | sha256sum -c -
|
|
37
|
+
tar -xzf "$archive" actionlint
|
|
38
|
+
install -m 0755 actionlint /usr/local/bin/actionlint
|
|
39
|
+
|
|
40
|
+
- name: Lint workflows
|
|
41
|
+
run: actionlint -color
|
|
@@ -9,6 +9,9 @@ on:
|
|
|
9
9
|
env:
|
|
10
10
|
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
|
11
11
|
|
|
12
|
+
permissions:
|
|
13
|
+
contents: read
|
|
14
|
+
|
|
12
15
|
jobs:
|
|
13
16
|
test:
|
|
14
17
|
runs-on: ubuntu-latest
|
|
@@ -16,12 +19,17 @@ jobs:
|
|
|
16
19
|
matrix:
|
|
17
20
|
python-version: ["3.11", "3.12", "3.13"]
|
|
18
21
|
steps:
|
|
19
|
-
- uses: actions/checkout@
|
|
22
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
20
23
|
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v5.3.0
|
|
21
24
|
with:
|
|
22
25
|
python-version: ${{ matrix.python-version }}
|
|
23
26
|
- name: Install package and test deps
|
|
24
|
-
|
|
27
|
+
# Third-party dependencies come from the hash-pinned lock first; the
|
|
28
|
+
# local package then goes in with --no-deps, because pip cannot
|
|
29
|
+
# hash-pin an editable install in the same invocation.
|
|
30
|
+
run: |
|
|
31
|
+
pip install --require-hashes -r requirements/test.txt
|
|
32
|
+
pip install --no-deps -e .
|
|
25
33
|
- name: Level 0 conformance (schema + structural)
|
|
26
34
|
run: python -m pytest -v -m "level0 or negative" --tb=short
|
|
27
35
|
- name: Unit tests (modules and CLI)
|
|
@@ -25,18 +25,18 @@ jobs:
|
|
|
25
25
|
|
|
26
26
|
steps:
|
|
27
27
|
- name: Checkout repository
|
|
28
|
-
uses: actions/checkout@v7
|
|
28
|
+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
29
29
|
|
|
30
30
|
- name: Initialize CodeQL
|
|
31
|
-
uses: github/codeql-action/init@v4.
|
|
31
|
+
uses: github/codeql-action/init@1c5b675653bb5c22dbe9b12b556ec555138e09fd # v4.38.1
|
|
32
32
|
with:
|
|
33
33
|
languages: python
|
|
34
34
|
queries: +security-extended
|
|
35
35
|
|
|
36
36
|
- name: Autobuild
|
|
37
|
-
uses: github/codeql-action/autobuild@v4.
|
|
37
|
+
uses: github/codeql-action/autobuild@1c5b675653bb5c22dbe9b12b556ec555138e09fd # v4.38.1
|
|
38
38
|
|
|
39
39
|
- name: Perform CodeQL Analysis
|
|
40
|
-
uses: github/codeql-action/analyze@v4.
|
|
40
|
+
uses: github/codeql-action/analyze@1c5b675653bb5c22dbe9b12b556ec555138e09fd # v4.38.1
|
|
41
41
|
with:
|
|
42
42
|
category: /language:python
|
{agentrust_trace_tests-0.5.1 → agentrust_trace_tests-0.6.0}/.github/workflows/contributor-check.yml
RENAMED
|
@@ -27,7 +27,7 @@ jobs:
|
|
|
27
27
|
github.actor != 'imran-siddique'
|
|
28
28
|
steps:
|
|
29
29
|
- name: Checkout org action
|
|
30
|
-
uses: actions/checkout@
|
|
30
|
+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
31
31
|
with:
|
|
32
32
|
repository: agentrust-io/.github
|
|
33
33
|
# Pinned deliberately: this workflow runs on pull_request_target
|
|
@@ -14,8 +14,12 @@ on:
|
|
|
14
14
|
- "measurement/**"
|
|
15
15
|
- "PRIVACY.md"
|
|
16
16
|
- "overrides/**"
|
|
17
|
+
- "hooks/**"
|
|
18
|
+
- "robots.txt"
|
|
17
19
|
- "index.md"
|
|
18
20
|
- "CODE_OF_CONDUCT.md"
|
|
21
|
+
- "LIMITATIONS.md"
|
|
22
|
+
- "SPONSORS.md"
|
|
19
23
|
workflow_dispatch:
|
|
20
24
|
|
|
21
25
|
permissions:
|
|
@@ -31,19 +35,19 @@ jobs:
|
|
|
31
35
|
runs-on: ubuntu-latest
|
|
32
36
|
|
|
33
37
|
steps:
|
|
34
|
-
- uses: actions/checkout@v7
|
|
38
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
35
39
|
with:
|
|
36
40
|
fetch-depth: 0
|
|
37
41
|
|
|
38
|
-
- uses: actions/setup-python@v7
|
|
42
|
+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
|
|
39
43
|
with:
|
|
40
44
|
python-version: "3.11"
|
|
41
45
|
cache: pip
|
|
42
46
|
|
|
43
47
|
- name: Install docs dependencies
|
|
44
48
|
run: |
|
|
45
|
-
pip install -r requirements
|
|
46
|
-
pip install -e
|
|
49
|
+
pip install --require-hashes -r requirements/docs.txt
|
|
50
|
+
pip install --no-deps -e .
|
|
47
51
|
|
|
48
52
|
- name: Configure git for gh-deploy
|
|
49
53
|
run: |
|
|
@@ -59,7 +63,7 @@ jobs:
|
|
|
59
63
|
# measurement/REPORT.md is in the nav as Self-verification.
|
|
60
64
|
if [ -d measurement ]; then cp -r measurement $BUILD/measurement; fi
|
|
61
65
|
|
|
62
|
-
for fname in index.md CHANGELOG.md CONTRIBUTING.md CODE_OF_CONDUCT.md PRIVACY.md CNAME; do
|
|
66
|
+
for fname in index.md CHANGELOG.md CONTRIBUTING.md CODE_OF_CONDUCT.md LIMITATIONS.md SPONSORS.md PRIVACY.md CNAME robots.txt; do
|
|
63
67
|
if [ -f "$fname" ]; then cp "$fname" "$BUILD/$fname"; fi
|
|
64
68
|
done
|
|
65
69
|
|
|
@@ -12,14 +12,14 @@ jobs:
|
|
|
12
12
|
build:
|
|
13
13
|
runs-on: ubuntu-latest
|
|
14
14
|
steps:
|
|
15
|
-
- uses: actions/checkout@
|
|
15
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
16
16
|
|
|
17
17
|
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
|
|
18
18
|
with:
|
|
19
19
|
python-version: "3.12"
|
|
20
20
|
|
|
21
21
|
- name: Install build
|
|
22
|
-
run: python -m pip install
|
|
22
|
+
run: python -m pip install --require-hashes -r requirements/release.txt
|
|
23
23
|
|
|
24
24
|
- name: Build distributions
|
|
25
25
|
run: python -m build
|
|
@@ -25,7 +25,7 @@ jobs:
|
|
|
25
25
|
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
|
26
26
|
with:
|
|
27
27
|
script: |
|
|
28
|
-
const MAINTAINERS = ['imran-siddique', 'lywinged'];
|
|
28
|
+
const MAINTAINERS = ['imran-siddique', 'lywinged', 'pforest', 'Qiang-Xu'];
|
|
29
29
|
|
|
30
30
|
const author = context.payload.pull_request.user.login;
|
|
31
31
|
if (MAINTAINERS.includes(author)) {
|
|
@@ -20,7 +20,7 @@ jobs:
|
|
|
20
20
|
actions: read
|
|
21
21
|
steps:
|
|
22
22
|
- name: Checkout
|
|
23
|
-
uses: actions/checkout@v7
|
|
23
|
+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
24
24
|
with:
|
|
25
25
|
persist-credentials: false
|
|
26
26
|
fetch-depth: 0
|
|
@@ -35,6 +35,6 @@ jobs:
|
|
|
35
35
|
publish_results: false
|
|
36
36
|
|
|
37
37
|
- name: Upload SARIF
|
|
38
|
-
uses: github/codeql-action/upload-sarif@v4.
|
|
38
|
+
uses: github/codeql-action/upload-sarif@1c5b675653bb5c22dbe9b12b556ec555138e09fd # v4.38.1
|
|
39
39
|
with:
|
|
40
40
|
sarif_file: scorecard-results.sarif
|
|
@@ -2,6 +2,59 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
## v0.6.0 - 2026-09-25
|
|
6
|
+
|
|
7
|
+
### Changed
|
|
8
|
+
|
|
9
|
+
- **Level 2 now verifies the anchor instead of parsing its URI (#79, closes #70).**
|
|
10
|
+
`TR-ANC-001` only ever checked that `transparency` was a well-formed https URI,
|
|
11
|
+
so any record could clear Level 2 by typing a string. The new `TR-ANC-002`
|
|
12
|
+
replays the record's RFC 9162 inclusion proof against the committed Merkle
|
|
13
|
+
root, offline and standard library only. The receipt is passed with the new
|
|
14
|
+
`--receipt` option on `verify` and `report`; without one, `TR-ANC-002` fails.
|
|
15
|
+
A record that passed Level 2 on 0.5.1 without a receipt will fail it on 0.6.0.
|
|
16
|
+
|
|
17
|
+
### Added
|
|
18
|
+
|
|
19
|
+
- **`TR-ENV-005`: `cnf.jwk` must carry no private key material (#98).** A record
|
|
20
|
+
carrying its own private key (`d` and the other private JWK members) passed
|
|
21
|
+
the whole suite, because `TR-ENV-004` only checked that `kty` was present. The
|
|
22
|
+
packaged `schemas/trace-claim.json` copy is closed the same way. This is the
|
|
23
|
+
trace-tests half of GHSA-vc4p-h84j-7qxj; trace-spec#296 fixed the other two
|
|
24
|
+
schema copies.
|
|
25
|
+
- **`TR-APR`: appraisal well-formedness at every level (#82, closes #63).**
|
|
26
|
+
`appraisal` is required by the schema and no module read it. Five codes now
|
|
27
|
+
check the status enum, the verifier URI and appraisal timing. No network, no
|
|
28
|
+
filesystem, no resolver.
|
|
29
|
+
- **`TR-POL-003` resolves `policy_uri` against `bundle_hash` (#69).** A
|
|
30
|
+
`policy_resolver` callable on `runner.run` and a `--policy-dir` option on the
|
|
31
|
+
CLI supply the policy bytes; the check compares their digest to the record.
|
|
32
|
+
- Machine-readable execution accounting in CLI JSON reports for the bounded
|
|
33
|
+
`TR-APR-001`, `TR-POL-003` and `TR-SCA-002` pilot (#93). Accounted findings
|
|
34
|
+
and accounting come from one immutable execution snapshot; unreconciled
|
|
35
|
+
accounting is rejected on that path, the operational policy-correspondence rule
|
|
36
|
+
is separated from supporting schema locators and all carry value digests for
|
|
37
|
+
comparison against the referenced trace-spec bytes, scheduler non-execution
|
|
38
|
+
carries a reason, and existing verdict policy and CLI exit behavior are unchanged.
|
|
39
|
+
|
|
40
|
+
### Fixed
|
|
41
|
+
|
|
42
|
+
- `TR-SIG` reports a record with no RFC 8785 canonical form (an integer outside
|
|
43
|
+
the safe range, a non-finite float) as a finding instead of raising and ending
|
|
44
|
+
the run (#86).
|
|
45
|
+
- Malformed arrays or objects in `policy.enforcement_mode`, `runtime.platform`,
|
|
46
|
+
`build_provenance.slsa_level` and `cnf.jwk.kty` raised
|
|
47
|
+
`TypeError` before the runner could return findings; boolean SLSA levels
|
|
48
|
+
passed as `1` and `0`. Both are now findings (#85, closes #84).
|
|
49
|
+
|
|
50
|
+
### Internal
|
|
51
|
+
|
|
52
|
+
- The finding to level failure decision lives in one place, read by both the
|
|
53
|
+
CLI and the JSON report (#88).
|
|
54
|
+
- Release and CI actions pinned to SHAs with a token permissions floor (#97);
|
|
55
|
+
CI installs from hash-pinned locks (#101); actionlint and a test-environment
|
|
56
|
+
guard added (#100).
|
|
57
|
+
|
|
5
58
|
## v0.5.1 — 2026-08-22
|
|
6
59
|
|
|
7
60
|
- Level 1 and Level 2 verification now requires a verifier-issued challenge via
|
|
@@ -9,6 +9,14 @@ Thank you for your interest in contributing to the TRACE test suite.
|
|
|
9
9
|
3. Commit using [Conventional Commits](https://www.conventionalcommits.org)
|
|
10
10
|
4. Open a pull request against `main`
|
|
11
11
|
|
|
12
|
+
## Using AI to contribute
|
|
13
|
+
|
|
14
|
+
Use agents. A lot of this was built with them and saying otherwise would be dishonest.
|
|
15
|
+
|
|
16
|
+
The rule is that you have to understand what you submit. If you cannot explain what your change does and how it interacts with the rest of the system, with the agent closed, do not open the pull request. Reviewing a change nobody can explain costs more than writing it did, and it becomes someone else's problem the moment it merges.
|
|
17
|
+
|
|
18
|
+
That is a rule about understanding, not about tooling.
|
|
19
|
+
|
|
12
20
|
## Reporting Security Issues
|
|
13
21
|
|
|
14
22
|
Use [GitHub Security Advisories](https://github.com/agentrust-io/trace-tests/security/advisories/new) rather than public issues.
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
# Known Limitations
|
|
2
|
+
|
|
3
|
+
What this suite does **not** establish. A conformance report is only useful if the reader knows
|
|
4
|
+
what it was never checking, so this is the companion to the report rather than a footnote to it.
|
|
5
|
+
|
|
6
|
+
## What a pass means
|
|
7
|
+
|
|
8
|
+
**A pass describes the record, not the agent.**
|
|
9
|
+
Conformance means the record is well formed, internally consistent, and carries what its level
|
|
10
|
+
requires. It says nothing about whether the agent behaved well, whether the policy it ran under
|
|
11
|
+
was a sensible policy, or whether the run should have been allowed. A record of a bad run passes
|
|
12
|
+
exactly as cleanly as a record of a good one.
|
|
13
|
+
|
|
14
|
+
**The suite is one implementation, not the definition.**
|
|
15
|
+
[trace-spec](https://github.com/agentrust-io/trace-spec) is normative. Where this suite and the
|
|
16
|
+
specification disagree, the specification is what other implementations were written against and
|
|
17
|
+
the disagreement is a bug worth reporting here.
|
|
18
|
+
|
|
19
|
+
## Where the checks stop
|
|
20
|
+
|
|
21
|
+
**`TR-RTE` checks the shape of the attestation fields, not the attestation.**
|
|
22
|
+
It validates that `runtime.platform` is a recognised value, that `runtime.measurement` is a well
|
|
23
|
+
formed digest, and that the RIM URI parses. It does not obtain a quote, and it does not verify one
|
|
24
|
+
against AMD, Intel or a TPM manufacturer root. A record can satisfy `TR-RTE` at Level 1 carrying a
|
|
25
|
+
syntactically perfect measurement that no hardware ever produced. Verifying the quote against the
|
|
26
|
+
silicon vendor is the relying party's job, and `cmcp_verify` is where that happens.
|
|
27
|
+
|
|
28
|
+
**`TR-ANC-002` proves inclusion relative to the receipt you hand it.**
|
|
29
|
+
It replays the audit path against the Merkle root carried in that receipt. It does not fetch the
|
|
30
|
+
`transparency` URI, and it does not establish that the root is one a public log actually
|
|
31
|
+
published. A self-consistent receipt over a tree the submitter built themselves will pass. What
|
|
32
|
+
the check rules out is a record that has been modified since anchoring, or that was never in the
|
|
33
|
+
tree the receipt commits to. Establishing that the tree is real is a separate step and is not in
|
|
34
|
+
scope here.
|
|
35
|
+
|
|
36
|
+
`TR-ANC-001` is explicit that it checks the pointer rather than the anchor. Supplying no receipt
|
|
37
|
+
fails `TR-ANC-002`, so Level 2 cannot be reached on a well formed URI alone.
|
|
38
|
+
|
|
39
|
+
## Statuses that are easy to misread
|
|
40
|
+
|
|
41
|
+
**`UNVERIFIED` is about reachability, not correctness.**
|
|
42
|
+
It means the check could not be run against the evidence the record cites. The evidence may be
|
|
43
|
+
perfectly good and simply out of reach. It is deliberately held apart from a skip so that it can
|
|
44
|
+
never be read as a benign omission.
|
|
45
|
+
|
|
46
|
+
Whether an unverified finding fails a run is decided per code, in
|
|
47
|
+
`src/trace_tests/modules/unverified.py`, not by one blanket rule. `TR-POL-003` is tolerated until
|
|
48
|
+
Level 2; anything the table does not name fails from Level 1, so a newly added code fails closed
|
|
49
|
+
rather than turning a run quietly permissive.
|
|
50
|
+
|
|
51
|
+
**A `TR-ENV` profile failure is a version mismatch before it is a defect.**
|
|
52
|
+
Suite 0.4.0 and later require v0.2 records. Run against a v0.1 record, the profile sentinel
|
|
53
|
+
produces a confident failure on a record that is fine. Check the suite and record versions before
|
|
54
|
+
believing that result.
|
|
55
|
+
|
|
56
|
+
**Results are perishable.**
|
|
57
|
+
`TR-ENV` validates `iat`, so a record that passes today can fail later with nothing about the
|
|
58
|
+
record having changed. A report is a statement about a moment, and it needs its timestamp to be
|
|
59
|
+
read correctly.
|
|
60
|
+
|
|
61
|
+
## Who is asserting what
|
|
62
|
+
|
|
63
|
+
**Nothing here is independently assessed.**
|
|
64
|
+
A report is produced by whoever ran the suite, on evidence they supplied. There is no third-party
|
|
65
|
+
assessor and no certification programme behind it. This is why the generated report tells a reader
|
|
66
|
+
who does not trust the sender to go and check the record themselves rather than trusting the
|
|
67
|
+
summary.
|
|
68
|
+
|
|
69
|
+
## Bounded obligation accounting
|
|
70
|
+
|
|
71
|
+
**`accounting_complete` describes the pilot, not all of TRACE.**
|
|
72
|
+
The accounting extension covers only `TR-APR-001`, `TR-POL-003`, and `TR-SCA-002`.
|
|
73
|
+
Complete means that every attempted level has exactly one reconciled row for each of those three
|
|
74
|
+
obligations. It does not mean every TRACE obligation was accounted for, every obligation was
|
|
75
|
+
evaluated, or that the report independently proves which code ran or which fields it accessed.
|
|
76
|
+
An obligation absent from both the bounded registry and its rows is outside this completeness
|
|
77
|
+
claim and cannot be discovered by registry-to-row reconciliation alone.
|
|
78
|
+
|
|
79
|
+
The pilot records `TR-SCA-002` at Level 0 as applicable but not attempted: the pinned schema
|
|
80
|
+
requires `build_provenance.digest`, while the scheduler first runs `TR-SCA` at Level 1. The row
|
|
81
|
+
carries that scheduler reason rather than silently presenting the state without an explanation.
|
|
82
|
+
|
|
83
|
+
Each source locator includes a digest of the exact value resolved at its pinned trace-spec
|
|
84
|
+
revision. JSON sources use RFC 6901 pointers and RFC 8785-canonical value bytes. The operational
|
|
85
|
+
TR-POL-003 rule uses the unique exact text of verification item 5 in the pinned specification;
|
|
86
|
+
its schema fragments are listed separately as structural support. A reader who holds those
|
|
87
|
+
pinned bytes can resolve and compare the values. The suite does not fetch or authenticate
|
|
88
|
+
trace-spec at report time, and a matching value does not prove that a locator is sufficient or
|
|
89
|
+
that its checker is correct.
|
|
90
|
+
|
|
91
|
+
Findings and accounting share one validated execution snapshot during supported report
|
|
92
|
+
construction. The emitted report remains editable and unsigned. Its registry hash identifies
|
|
93
|
+
registry content; it does not authenticate the rows, the report, or its producer.
|
|
94
|
+
|
|
95
|
+
The optional policy resolver is trusted in-process Python code supplied by the caller. The
|
|
96
|
+
accounted path isolates nested public runs and freezes ordinary decision inputs, but it is not a
|
|
97
|
+
sandbox against a callback that rewrites interpreter globals, functions, classes, or source files.
|
|
98
|
+
|
|
99
|
+
`producer_branch` is a checker-owned same-execution diagnostic. Some branches with the same
|
|
100
|
+
accounting meaning return indistinguishable findings, so the emitted report alone cannot
|
|
101
|
+
independently reconstruct every branch label. Applicability, evaluation state, prerequisite,
|
|
102
|
+
contribution, findings, and report tallies are revalidated at emission; the label is not
|
|
103
|
+
presented as independent proof of checker control flow.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: agentrust-trace-tests
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.6.0
|
|
4
4
|
Summary: TRACE conformance test suite
|
|
5
5
|
Project-URL: Homepage, https://github.com/agentrust-io/trace-tests
|
|
6
6
|
Project-URL: Repository, https://github.com/agentrust-io/trace-tests
|
|
@@ -42,7 +42,9 @@ Description-Content-Type: text/markdown
|
|
|
42
42
|
|
|
43
43
|
# TRACE Conformance Test Suite
|
|
44
44
|
|
|
45
|
-
|
|
45
|
+
Community updates and contributor highlights: [AgenTrust on LinkedIn](https://www.linkedin.com/company/agentrust-io/).
|
|
46
|
+
|
|
47
|
+
### Check a TRACE record and see which conformance level it reaches
|
|
46
48
|
|
|
47
49
|
<p align="center">
|
|
48
50
|
<a href="https://tests.agentrust-io.com">
|
|
@@ -59,15 +61,15 @@ Description-Content-Type: text/markdown
|
|
|
59
61
|
|
|
60
62
|
[](LICENSE)
|
|
61
63
|
[](https://github.com/agentrust-io/trace-spec)
|
|
62
|
-
[](docs/modules.md)
|
|
63
65
|
[](https://github.com/agentrust-io/trace-tests/actions/workflows/ci.yml)
|
|
64
66
|
[](https://discord.gg/grgzFEHgkj)
|
|
65
67
|
|
|
66
|
-
>
|
|
68
|
+
> Tracks [TRACE Spec v0.2](https://github.com/agentrust-io/trace-spec).
|
|
67
69
|
|
|
68
|
-
|
|
70
|
+
Check a TRACE record, inspect the findings, and produce a reproducible conformance report. The suite checks the record and supplied evidence; a passing report does not establish that an entire implementation meets every specification requirement.
|
|
69
71
|
|
|
70
|
-
|
|
72
|
+
Eight modules cover envelope, signature, runtime, policy, appraisal, transcript, transparency, and provenance checks. Read the [limitations](LIMITATIONS.md) to interpret their results.
|
|
71
73
|
|
|
72
74
|
## Quick start
|
|
73
75
|
|
|
@@ -102,7 +104,25 @@ conformance report that looks authoritative and cannot be checked is the same sh
|
|
|
102
104
|
thing as a control plane writing its own log.
|
|
103
105
|
|
|
104
106
|
`report.json` is stable under `schema: agentrust-io/trace-tests/report/1` for dashboards
|
|
105
|
-
and CI.
|
|
107
|
+
and CI. Reports produced by the CLI include an additive, version-tagged
|
|
108
|
+
`obligation_accounting` object for the bounded `TR-APR-001`, `TR-POL-003`, and
|
|
109
|
+
`TR-SCA-002` pilot. During supported report construction, its rows are reconciled
|
|
110
|
+
against the executable registry identified by `registry_id` and `registry_sha256`.
|
|
111
|
+
The operational `TR-POL-003` rule is identified separately from schema fragments
|
|
112
|
+
that support its field shape. Every source locator carries a digest of the exact
|
|
113
|
+
resolved value so a reader holding the pinned trace-spec bytes can re-resolve and
|
|
114
|
+
compare it. Accounted JSON, HTML, badge, and verdict projections consume one
|
|
115
|
+
immutable execution-derived snapshot. A JSON-only request does not pre-render
|
|
116
|
+
unrequested formats; when multiple formats are requested, they are emitted in the
|
|
117
|
+
CLI's existing order. The existing contribution policy continues to determine the
|
|
118
|
+
report verdict.
|
|
119
|
+
The tag identifies this emitted object shape; the repository does not currently
|
|
120
|
+
ship a separate formal JSON Schema for it.
|
|
121
|
+
|
|
122
|
+
This treats `report/1` as additively extensible: existing members retain their meaning,
|
|
123
|
+
and `obligation_accounting` is the sole new top-level member. Compatibility with
|
|
124
|
+
consumers that require the exact historical key set is not established. The report
|
|
125
|
+
remains an unsigned self-report; see [Known Limitations](LIMITATIONS.md).
|
|
106
126
|
|
|
107
127
|
## Test modules
|
|
108
128
|
|
|
@@ -125,6 +145,7 @@ and CI.
|
|
|
125
145
|
| 🗂 Test schemas | [schemas/](schemas/) |
|
|
126
146
|
| 💬 Discussions | [GitHub Discussions](https://github.com/orgs/agentrust-io/discussions) |
|
|
127
147
|
| 📋 Changelog | [CHANGELOG.md](CHANGELOG.md) |
|
|
148
|
+
| ⚠️ Known limitations | [LIMITATIONS.md](LIMITATIONS.md) |
|
|
128
149
|
|
|
129
150
|
## Contributing
|
|
130
151
|
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
# Privacy
|
|
2
|
+
|
|
3
|
+
The TRACE test suite reads the record and optional evidence you supply. CLI options can also load policy bundles, receipts, and related local files. Findings and exported reports may reproduce identifiers, artifact locations, and error details from those inputs; review reports before sharing them.
|
|
4
|
+
|
|
5
|
+
The CLI does not send project telemetry or analytics and does not fetch arbitrary record URLs. Its policy-directory resolver uses local files. Library callers can supply their own resolver callbacks, whose network and data handling behavior belongs to the calling application.
|
|
6
|
+
|
|
7
|
+
Uninstalling the package does not delete input records, evidence files, exported reports, badges, logs, or backups. Manage those artifacts through your application's retention and deletion procedures.
|
|
8
|
+
|
|
9
|
+
[Report a correction](https://github.com/agentrust-io/trace-tests/issues).
|
|
@@ -4,7 +4,9 @@
|
|
|
4
4
|
|
|
5
5
|
# TRACE Conformance Test Suite
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
Community updates and contributor highlights: [AgenTrust on LinkedIn](https://www.linkedin.com/company/agentrust-io/).
|
|
8
|
+
|
|
9
|
+
### Check a TRACE record and see which conformance level it reaches
|
|
8
10
|
|
|
9
11
|
<p align="center">
|
|
10
12
|
<a href="https://tests.agentrust-io.com">
|
|
@@ -21,15 +23,15 @@
|
|
|
21
23
|
|
|
22
24
|
[](LICENSE)
|
|
23
25
|
[](https://github.com/agentrust-io/trace-spec)
|
|
24
|
-
[](docs/modules.md)
|
|
25
27
|
[](https://github.com/agentrust-io/trace-tests/actions/workflows/ci.yml)
|
|
26
28
|
[](https://discord.gg/grgzFEHgkj)
|
|
27
29
|
|
|
28
|
-
>
|
|
30
|
+
> Tracks [TRACE Spec v0.2](https://github.com/agentrust-io/trace-spec).
|
|
29
31
|
|
|
30
|
-
|
|
32
|
+
Check a TRACE record, inspect the findings, and produce a reproducible conformance report. The suite checks the record and supplied evidence; a passing report does not establish that an entire implementation meets every specification requirement.
|
|
31
33
|
|
|
32
|
-
|
|
34
|
+
Eight modules cover envelope, signature, runtime, policy, appraisal, transcript, transparency, and provenance checks. Read the [limitations](LIMITATIONS.md) to interpret their results.
|
|
33
35
|
|
|
34
36
|
## Quick start
|
|
35
37
|
|
|
@@ -64,7 +66,25 @@ conformance report that looks authoritative and cannot be checked is the same sh
|
|
|
64
66
|
thing as a control plane writing its own log.
|
|
65
67
|
|
|
66
68
|
`report.json` is stable under `schema: agentrust-io/trace-tests/report/1` for dashboards
|
|
67
|
-
and CI.
|
|
69
|
+
and CI. Reports produced by the CLI include an additive, version-tagged
|
|
70
|
+
`obligation_accounting` object for the bounded `TR-APR-001`, `TR-POL-003`, and
|
|
71
|
+
`TR-SCA-002` pilot. During supported report construction, its rows are reconciled
|
|
72
|
+
against the executable registry identified by `registry_id` and `registry_sha256`.
|
|
73
|
+
The operational `TR-POL-003` rule is identified separately from schema fragments
|
|
74
|
+
that support its field shape. Every source locator carries a digest of the exact
|
|
75
|
+
resolved value so a reader holding the pinned trace-spec bytes can re-resolve and
|
|
76
|
+
compare it. Accounted JSON, HTML, badge, and verdict projections consume one
|
|
77
|
+
immutable execution-derived snapshot. A JSON-only request does not pre-render
|
|
78
|
+
unrequested formats; when multiple formats are requested, they are emitted in the
|
|
79
|
+
CLI's existing order. The existing contribution policy continues to determine the
|
|
80
|
+
report verdict.
|
|
81
|
+
The tag identifies this emitted object shape; the repository does not currently
|
|
82
|
+
ship a separate formal JSON Schema for it.
|
|
83
|
+
|
|
84
|
+
This treats `report/1` as additively extensible: existing members retain their meaning,
|
|
85
|
+
and `obligation_accounting` is the sole new top-level member. Compatibility with
|
|
86
|
+
consumers that require the exact historical key set is not established. The report
|
|
87
|
+
remains an unsigned self-report; see [Known Limitations](LIMITATIONS.md).
|
|
68
88
|
|
|
69
89
|
## Test modules
|
|
70
90
|
|
|
@@ -87,6 +107,7 @@ and CI.
|
|
|
87
107
|
| 🗂 Test schemas | [schemas/](schemas/) |
|
|
88
108
|
| 💬 Discussions | [GitHub Discussions](https://github.com/orgs/agentrust-io/discussions) |
|
|
89
109
|
| 📋 Changelog | [CHANGELOG.md](CHANGELOG.md) |
|
|
110
|
+
| ⚠️ Known limitations | [LIMITATIONS.md](LIMITATIONS.md) |
|
|
90
111
|
|
|
91
112
|
## Contributing
|
|
92
113
|
|
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# Sponsors
|
|
2
2
|
|
|
3
|
-
TRACE Tests is an open-source AgenTrust project.
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
3
|
+
TRACE Tests is an open-source AgenTrust project. It is sponsored by OPAQUE
|
|
4
|
+
Systems, which funds the engineering, infrastructure and confidential-computing
|
|
5
|
+
work behind it. Organisations that want to support the project are welcome to
|
|
6
|
+
join as sponsors.
|
|
7
7
|
|
|
8
8
|
## Current sponsors
|
|
9
9
|
|
|
@@ -10,6 +10,7 @@ All TRACE test failures emit a structured error code of the form `TR-<MODULE>-<N
|
|
|
10
10
|
| TR-ENV-002 | `iat` is missing, not an integer, or out of range | Set `iat` to a Unix timestamp integer (e.g. `int(time.time())`) |
|
|
11
11
|
| TR-ENV-003 | `subject` does not match SPIFFE URI or DID pattern | Use `spiffe://<trust-domain>/<path>` or a `did:` URI |
|
|
12
12
|
| TR-ENV-004 | `cnf` is absent or not an object, `cnf.jwk` is absent or not an object, or `cnf.jwk.kty` is absent | Populate `cnf.jwk` with at least `kty`. This checks that one field, not the schema's full required set, which structural validation covers |
|
|
13
|
+
| TR-ENV-005 | `cnf.jwk` carries private key material (`d`, `p`, `q`, `dp`, `dq`, `qi`, `k`) | Publish the public half only. RFC 8747 makes `cnf` a confirmation key, and a record is signed and usually anchored, so a key exposed this way must be treated as compromised and the identity revoked |
|
|
13
14
|
|
|
14
15
|
## TR-SIG — Signature
|
|
15
16
|
|
|
@@ -33,8 +34,19 @@ All TRACE test failures emit a structured error code of the form `TR-<MODULE>-<N
|
|
|
33
34
|
|
|
34
35
|
| Code | Description | How to fix |
|
|
35
36
|
|------|-------------|------------|
|
|
36
|
-
| TR-POL-001 | `policy.bundle_hash` is not a valid `sha256:` digest | Compute `sha256:` + hex
|
|
37
|
+
| TR-POL-001 | `policy.bundle_hash` is not a valid `sha256:` or `sha384:` digest | Compute `sha256:` + 64 hex chars, or `sha384:` + 96 hex chars, over your policy bundle bytes. Both are accepted by the schema and by the module |
|
|
37
38
|
| TR-POL-002 | `policy.enforcement_mode` is not `enforce`, `advisory`, `silent`, or `declared` | Replace `"strict"` or `"monitor"` with one of the four accepted values; `"declared"` is the honest value for a producer that binds a policy without evaluating it |
|
|
39
|
+
| TR-POL-003 | `policy.policy_uri` is not an absolute URI, or the bundle it resolves to does not have the digest `policy.bundle_hash` declares. Unverified when a resolver was supplied and the bundle could not be read; skipped when no `policy_uri` is present or no resolver was supplied | Point `policy_uri` at the bundle whose bytes hash to `bundle_hash`. A record that cites a bundle it cannot be checked against is reported as unverified rather than passed |
|
|
40
|
+
|
|
41
|
+
## TR-APR — Appraisal
|
|
42
|
+
|
|
43
|
+
| Code | Description | How to fix |
|
|
44
|
+
|------|-------------|------------|
|
|
45
|
+
| TR-APR-001 | `appraisal` is absent or not an object, or `appraisal.status` is absent or not one of `affirming`, `warning`, `contraindicated`, `none` | Set `appraisal.status` to one of the four values the schema enumerates. An absent or non-object `appraisal` is reported as this code alone, not as a cascade |
|
|
46
|
+
| TR-APR-002 | `appraisal.verifier` is absent, not a string, or not an absolute URI | Set `appraisal.verifier` to the absolute URI identifying the verifier that produced the appraisal. A relative reference is rejected: the schema asks for `format: "uri"`, and a reader cannot dereference a name with no scheme |
|
|
47
|
+
| TR-APR-003 | `appraisal.policy_ref` is present and is not an absolute URI. Skipped when the field is absent, which is permitted | Point `policy_ref` at the appraisal policy with an absolute URI, or omit it. Only the shape of the name is checked; TR-APR never resolves it |
|
|
48
|
+
| TR-APR-004 | `appraisal.timestamp` is present and is not an integer of epoch seconds, or is in the future. Skipped when the field is absent, which is permitted | Set `appraisal.timestamp` to the epoch second the appraisal was produced. An appraisal dated in the future asserts something that has not happened, on the same ground TR-ENV-002 applies to `iat` |
|
|
49
|
+
| TR-APR-005 | `appraisal.status` is not `affirming` at Level 1 or above. Skipped at Level 0, where `docs/levels.md`'s own minimum conformant record carries `none` | Re-run the appraisal until it affirms, or check the record at Level 0 |
|
|
38
50
|
|
|
39
51
|
## TR-TXN — Transcript
|
|
40
52
|
|
|
@@ -47,7 +59,8 @@ All TRACE test failures emit a structured error code of the form `TR-<MODULE>-<N
|
|
|
47
59
|
|
|
48
60
|
| Code | Description | How to fix |
|
|
49
61
|
|------|-------------|------------|
|
|
50
|
-
| TR-ANC-001 | `transparency` is absent or empty, is not a string, or is not an `https://` URI with a host | Submit the record to a SCITT transparency log and set `transparency` to the returned receipt URI. The URI is not resolved and the receipt behind it is not fetched; this is a format check |
|
|
62
|
+
| TR-ANC-001 | `transparency` is absent or empty, is not a string, or is not an `https://` URI with a host | Submit the record to a SCITT transparency log and set `transparency` to the returned receipt URI. The URI is not resolved and the receipt behind it is not fetched; this is a format check on the pointer, and TR-ANC-002 is what checks the anchor |
|
|
63
|
+
| TR-ANC-002 | No anchor receipt was supplied, the receipt is malformed, or replaying its inclusion proof does not reproduce the committed `merkle_root` | Pass the receipt with `--receipt`. Without one, nothing proves the record is in the log the URI names, so Level 2 cannot pass. If a receipt is supplied and the proof does not verify, the record is not in that tree or it has been modified since it was anchored |
|
|
51
64
|
|
|
52
65
|
## TR-SCA — Provenance
|
|
53
66
|
|