codapact 0.9.0__tar.gz → 0.10.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {codapact-0.9.0 → codapact-0.10.0}/.github/workflows/release.yml +5 -1
- codapact-0.10.0/.release-please-manifest.json +1 -0
- {codapact-0.9.0 → codapact-0.10.0}/CHANGELOG.md +25 -0
- {codapact-0.9.0 → codapact-0.10.0}/PKG-INFO +7 -4
- {codapact-0.9.0 → codapact-0.10.0}/README.md +5 -3
- {codapact-0.9.0 → codapact-0.10.0}/docs/architecture.md +6 -5
- {codapact-0.9.0 → codapact-0.10.0}/docs/configuration.md +39 -9
- codapact-0.10.0/docs/discovery-efficiency/README.md +240 -0
- codapact-0.10.0/docs/discovery-efficiency/data/results.json +16348 -0
- codapact-0.10.0/docs/discovery-efficiency/figures/scaling-by-api-size.svg +90 -0
- codapact-0.10.0/docs/discovery-efficiency/figures/ynab-session-context.svg +137 -0
- codapact-0.10.0/docs/discovery-efficiency/figures/ynab-tokens-to-first-call.svg +93 -0
- codapact-0.10.0/examples/README.md +24 -0
- codapact-0.10.0/examples/confluence/codapact.yaml +18 -0
- codapact-0.10.0/examples/github/codapact.yaml +21 -0
- codapact-0.10.0/examples/jira/codapact.yaml +19 -0
- codapact-0.10.0/examples/swagger-petstore/codapact.yaml +19 -0
- codapact-0.10.0/extras/agent-plugin/.claude-plugin/plugin.json +8 -0
- codapact-0.10.0/extras/agent-plugin/README.md +37 -0
- codapact-0.10.0/extras/agent-plugin/plugin.json +8 -0
- codapact-0.10.0/extras/agent-plugin/skills/create-coda-skill/SKILL.md +100 -0
- codapact-0.10.0/extras/agent-plugin/skills/create-coda-skill/assets/SKILL.template.md +39 -0
- {codapact-0.9.0 → codapact-0.10.0}/pyproject.toml +7 -1
- {codapact-0.9.0 → codapact-0.10.0}/pyproject.toml.orig +8 -3
- codapact-0.10.0/scripts/discovery_bench/collect.py +882 -0
- codapact-0.10.0/scripts/discovery_bench/plot.py +355 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/__init__.py +1 -1
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/core/net.py +31 -7
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/authspec.py +13 -5
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/pipeline/pipeline.py +5 -1
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/abstractions.py +6 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/cli.py +100 -18
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/cli_tools.py +97 -21
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/codec.py +2 -1
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/errors.py +16 -2
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/protocols/base.py +25 -5
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/protocols/http.py +84 -4
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/shaping/transforms.py +145 -10
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/sources/base.py +8 -2
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/sources/loader.py +40 -1
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/sources/openapi/translator.py +176 -13
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/python/cli.py +85 -11
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/python/library.py +2 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/python/naming.py +3 -1
- {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_cli_conformance.py +33 -3
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_cli_builtins.py +81 -1
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_cli_options.py +2 -2
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_guards.py +3 -2
- codapact-0.10.0/tests/unit/test_large_contract_findings.py +742 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_live_ynab_findings.py +1 -1
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_openapi_edges.py +1 -1
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_public_api_findings.py +1 -1
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_target_rendering.py +6 -1
- {codapact-0.9.0 → codapact-0.10.0}/uv.lock +12 -1
- codapact-0.9.0/.release-please-manifest.json +0 -1
- {codapact-0.9.0 → codapact-0.10.0}/.github/workflows/ci.yml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/.github/workflows/codeql.yml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/.github/workflows/mutation.yml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/.github/workflows/nightly.yml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/LICENSE +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/docs/branding/README.md +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/docs/branding/mark.svg +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/docs/branding/wordmark.svg +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/docs/testing.md +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/examples/library/codapact.yaml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/examples/library/operations.graphql +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/examples/library/schema.graphql +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/examples/petstore/codapact.yaml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/examples/petstore/openapi.yaml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/release-please-config.json +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/scripts/extract_openapi_subset.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/scripts/mutation_score.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/scripts/verify_dist.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/__main__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/commands/__init__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/commands/derive.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/commands/model.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/core/__init__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/core/diagnostics.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/core/errors.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/core/naming.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/__init__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/api.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/migrations.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/rewrite.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/serialize.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/types.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/validate.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/values.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/pipeline/__init__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/pipeline/cli.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/pipeline/config.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/pipeline/defaults.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/pipeline/sync.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/pipeline/writer.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/plugins/__init__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/plugins/external.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/plugins/protocol.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/plugins/registry.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/py.typed +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/__init__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/arguments.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/auth.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/cli_auth.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/client.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/credentials.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/middleware.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/oauth.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/profiles.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/protocols/__init__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/protocols/graphql.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/selftest.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/shaping/__init__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/shaping/base.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/shaping/engine.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/sources/__init__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/sources/graphql/__init__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/sources/graphql/translator.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/sources/openapi/__init__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/surface/__init__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/surface/derive.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/surface/model.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/__init__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/python/__init__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/python/e2e.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/typescript/__init__.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/typescript/library.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/typescript/runtime.ts +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/authlab.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/conftest.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/e2e/test_auth_e2e.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/e2e/test_cli_e2e.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/e2e/test_public_apis.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/authlab/authlab_plugin.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/authlab/codapact.yaml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/authlab/fake_browser.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/authlab/openapi.yaml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/authlab_smoke.ts +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/contracts/wire.yaml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/README.md +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/countries/codapact.yaml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/countries/operations.graphql +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/countries/schema.graphql +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/github/codapact.yaml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/github/openapi.json +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/pokeapi/codapact.yaml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/pokeapi/openapi.yml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/ynab/README.md +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/ynab/codapact.yaml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/ynab/openapi.yaml +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/library_smoke.ts +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/outline_plugin.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/petstore_smoke.ts +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/wire_parity.ts +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_cli_conventions.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_contract_properties.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_corpus.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_generated_code_properties.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_generation.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_public_api_cli.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_wire_parity.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_ynab_contract.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/perf/test_scaling.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/public_api_contract.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/strategies.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/support.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_architecture.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_auth_and_graphql_sources.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_auth_sources.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_bundle_e2e.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_cli_auth.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_codec_serialize.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_core.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_credential_fields.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_env_prefix.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_model.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_mutation_score.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_oauth.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_openapi_translator_units.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_plugins_pipeline.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_profiles.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_program_rename.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_runtime.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_runtime_edges.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_runtime_pagination.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_shaping_surface_commands.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_sources.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_sync.py +0 -0
- {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_usability.py +0 -0
|
@@ -28,7 +28,10 @@ jobs:
|
|
|
28
28
|
token: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }}
|
|
29
29
|
config-file: release-please-config.json
|
|
30
30
|
manifest-file: .release-please-manifest.json
|
|
31
|
-
|
|
31
|
+
# Pushes made with GITHUB_TOKEN start no pull_request workflows, and GitHub holds
|
|
32
|
+
# CodeQL for approval on such pull requests. Dispatching is allowed, and the
|
|
33
|
+
# required CodeQL results for the release branch's head come from this run.
|
|
34
|
+
- name: Start CI and CodeQL for release pull request
|
|
32
35
|
if: steps.release.outputs.prs_created == 'true'
|
|
33
36
|
env:
|
|
34
37
|
GH_TOKEN: ${{ github.token }}
|
|
@@ -36,6 +39,7 @@ jobs:
|
|
|
36
39
|
run: |
|
|
37
40
|
branch=$(jq -r '.headBranchName' <<< "$PR_JSON")
|
|
38
41
|
gh workflow run ci.yml --repo "$GITHUB_REPOSITORY" --ref "$branch"
|
|
42
|
+
gh workflow run codeql.yml --repo "$GITHUB_REPOSITORY" --ref "$branch"
|
|
39
43
|
ci:
|
|
40
44
|
needs: version
|
|
41
45
|
if: needs.version.outputs.created == 'true'
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{".":"0.10.0"}
|
|
@@ -2,6 +2,31 @@
|
|
|
2
2
|
|
|
3
3
|
<!-- markdownlint-configure-file { "MD024": { "siblings_only": true } } -->
|
|
4
4
|
|
|
5
|
+
## [0.10.0](https://github.com/mikejhill/codapact/compare/v0.9.1...v0.10.0) (2026-11-19)
|
|
6
|
+
|
|
7
|
+
|
|
8
|
+
### Features
|
|
9
|
+
|
|
10
|
+
* **cli:** compact JSON output when stdout is not a terminal ([#62](https://github.com/mikejhill/codapact/issues/62)) ([1b80666](https://github.com/mikejhill/codapact/commit/1b80666128834f5ba2a13d5e21e301e6f60fbafd))
|
|
11
|
+
* **cli:** leaner agent discovery (lean describe COMMAND, compact JSON, grouped --brief, JSON field hints) ([#61](https://github.com/mikejhill/codapact/issues/61)) ([4070dea](https://github.com/mikejhill/codapact/commit/4070deab7eb77dabeb65cda52228af6ab8ac7232))
|
|
12
|
+
* generate large real-world contracts strictly; session checks and expiry hooks ([#58](https://github.com/mikejhill/codapact/issues/58)) ([523e6b3](https://github.com/mikejhill/codapact/commit/523e6b3ac156fb7f6516d157c4b1f7809519ebee))
|
|
13
|
+
* headers shaping step; multipart lists of files ([#60](https://github.com/mikejhill/codapact/issues/60)) ([7bbb762](https://github.com/mikejhill/codapact/commit/7bbb762a97aba656803d08ade48111747647a8b5))
|
|
14
|
+
* move the agent plugin to extras/agent-plugin; dispatch CodeQL for release PRs ([#65](https://github.com/mikejhill/codapact/issues/65)) ([3e6c2f1](https://github.com/mikejhill/codapact/commit/3e6c2f14b69a1e86f92d0a4cc19f104f89a99d5d))
|
|
15
|
+
* move the agent plugin to plugins/codapact and add live examples ([#64](https://github.com/mikejhill/codapact/issues/64)) ([98c889d](https://github.com/mikejhill/codapact/commit/98c889d6ad8e03cbe08f1b69ea05288f16fcd855))
|
|
16
|
+
|
|
17
|
+
|
|
18
|
+
### Documentation
|
|
19
|
+
|
|
20
|
+
* **discovery:** remeasure on main after [#61](https://github.com/mikejhill/codapact/issues/61) and [#62](https://github.com/mikejhill/codapact/issues/62) ([#63](https://github.com/mikejhill/codapact/issues/63)) ([7e9cdbb](https://github.com/mikejhill/codapact/commit/7e9cdbb5e2b0fde3d39e27c66c21abb116ed2329))
|
|
21
|
+
* measure how efficiently agents discover a generated CLI ([#57](https://github.com/mikejhill/codapact/issues/57)) ([1be40c8](https://github.com/mikejhill/codapact/commit/1be40c8daae2a80c4b9a404c27865c1616571b1f))
|
|
22
|
+
|
|
23
|
+
## [0.9.1](https://github.com/mikejhill/codapact/compare/v0.9.0...v0.9.1) (2026-11-15)
|
|
24
|
+
|
|
25
|
+
|
|
26
|
+
### Bug Fixes
|
|
27
|
+
|
|
28
|
+
* **cli:** plain usage lines in describe; uvx-only coda skills; Agent Plugins manifest ([#54](https://github.com/mikejhill/codapact/issues/54)) ([3fea349](https://github.com/mikejhill/codapact/commit/3fea3493542eff6860cf2db049bd8cecce67d52b))
|
|
29
|
+
|
|
5
30
|
## [0.9.0](https://github.com/mikejhill/codapact/compare/v0.8.0...v0.9.0) (2026-11-15)
|
|
6
31
|
|
|
7
32
|
|
|
@@ -1,11 +1,12 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: codapact
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.10.0
|
|
4
4
|
Summary: Contracts into callable tools: generate libraries, then CLIs, from OpenAPI, GraphQL, and other API contracts.
|
|
5
5
|
License-Expression: MIT
|
|
6
6
|
License-File: LICENSE
|
|
7
7
|
Requires-Dist: pyyaml>=6.0.3,<7
|
|
8
8
|
Requires-Dist: graphql-core>=3.2.6,<4
|
|
9
|
+
Requires-Dist: truststore>=0.10,<1
|
|
9
10
|
Requires-Dist: keyring>=25,<26 ; extra == 'keyring'
|
|
10
11
|
Requires-Python: >=3.13
|
|
11
12
|
Project-URL: Homepage, https://github.com/mikejhill/codapact
|
|
@@ -86,7 +87,9 @@ pagination live in one tested place and improve without regenerating.
|
|
|
86
87
|
and restish.
|
|
87
88
|
- [Configuration](docs/configuration.md): `codapact.yaml`, shaping transforms, target
|
|
88
89
|
options, and generated CLI settings.
|
|
89
|
-
- [Agent skills](
|
|
90
|
-
agent use a coda's CLI.
|
|
91
|
-
|
|
90
|
+
- [Agent skills](extras/agent-plugin/README.md): `create-coda-skill` writes the thin skill
|
|
91
|
+
that lets an agent use a coda's CLI. The repository is a plugin marketplace for Claude Code, GitHub
|
|
92
|
+
Copilot, and Codex: `<agent> plugin marketplace add mikejhill/codapact`.
|
|
93
|
+
- [Discovery efficiency](docs/discovery-efficiency/README.md): how many tokens and calls an
|
|
94
|
+
agent spends learning a generated CLI, compared with `--help`, MCP, OpenAPI, and llms.txt.
|
|
92
95
|
- [Contributing](CONTRIBUTING.md)
|
|
@@ -69,7 +69,9 @@ pagination live in one tested place and improve without regenerating.
|
|
|
69
69
|
and restish.
|
|
70
70
|
- [Configuration](docs/configuration.md): `codapact.yaml`, shaping transforms, target
|
|
71
71
|
options, and generated CLI settings.
|
|
72
|
-
- [Agent skills](
|
|
73
|
-
agent use a coda's CLI.
|
|
74
|
-
|
|
72
|
+
- [Agent skills](extras/agent-plugin/README.md): `create-coda-skill` writes the thin skill
|
|
73
|
+
that lets an agent use a coda's CLI. The repository is a plugin marketplace for Claude Code, GitHub
|
|
74
|
+
Copilot, and Codex: `<agent> plugin marketplace add mikejhill/codapact`.
|
|
75
|
+
- [Discovery efficiency](docs/discovery-efficiency/README.md): how many tokens and calls an
|
|
76
|
+
agent spends learning a generated CLI, compared with `--help`, MCP, OpenAPI, and llms.txt.
|
|
75
77
|
- [Contributing](CONTRIBUTING.md)
|
|
@@ -87,7 +87,7 @@ Enforced by `tests/unit/test_architecture.py` (AST import scan):
|
|
|
87
87
|
| HTTP sending | `Transport.send(HttpRequest)` | `RuntimeClient(transport=…)` |
|
|
88
88
|
| Cross-cutting | `Middleware(request, context, next)` | `RuntimeClient(middleware=…)` |
|
|
89
89
|
| Credentials | `AuthProvider.credential(scheme)` | `RuntimeClient(auth=…)` |
|
|
90
|
-
| Bespoke auth | `Authenticator.authenticate(request, scheme)`, optional `refresh` | returned by an `AuthProvider`; CLI `auth_plugin` |
|
|
90
|
+
| Bespoke auth | `Authenticator.authenticate(request, scheme)`, optional `refresh` and `expired` | returned by an `AuthProvider`; CLI `auth_plugin` |
|
|
91
91
|
| Protocol | `ProtocolAdapter.supports/call` | `RuntimeClient(adapters=…)` |
|
|
92
92
|
|
|
93
93
|
### Plugin protocol
|
|
@@ -110,15 +110,16 @@ that silently mishandles the feature.
|
|
|
110
110
|
- `RuntimeClient(api, base_url, auth, transport, middleware, adapters, …)`:
|
|
111
111
|
`call(operation, args)` encodes and validates arguments, picks the adapter for the
|
|
112
112
|
operation's binding, and decodes the result. `paginate()` follows cursors or offsets.
|
|
113
|
-
- `SafeHttpClient` (default transport):
|
|
114
|
-
|
|
113
|
+
- `SafeHttpClient` (default transport): redirects returned, never followed; OS trust store
|
|
114
|
+
(`truststore`); bounded bodies; DNS pinned against rebinding; private and plaintext
|
|
115
|
+
addresses only with explicit opt-in.
|
|
115
116
|
- `Retry` middleware: idempotent operations only, exponential backoff, honors
|
|
116
117
|
`Retry-After`.
|
|
117
118
|
- Auth: sources (`StaticAuth`, `EnvAuth`, `ChainAuth`, `CommandAuth`, `StoreAuth`) return a
|
|
118
119
|
`Credential` or an `Authenticator`. `OAuth2` (in `runtime.oauth`) is an `Authenticator`
|
|
119
120
|
over a grant (client credentials, password, device code, authorization code + PKCE) and a
|
|
120
|
-
`TokenStore` (memory, file, keyring). Authenticators run after middleware; a 401
|
|
121
|
-
one `refresh()` and resend. `Credential.__repr__` never renders secrets.
|
|
121
|
+
`TokenStore` (memory, file, keyring). Authenticators run after middleware; a 401 (or a
|
|
122
|
+
response an authenticator's `expired()` flags) triggers one `refresh()` and resend. `Credential.__repr__` never renders secrets.
|
|
122
123
|
- `runtime.selftest`: the suite generated bundles ship (offline dry runs, live scenarios).
|
|
123
124
|
- Errors: `ValidationError`, `AuthError`, `ApiError` (status, named error case, decoded
|
|
124
125
|
payload), `GraphQLResponseError`, `TransportError`.
|
|
@@ -62,7 +62,23 @@ Targets that share an `output` directory share options. When `python-cli` shares
|
|
|
62
62
|
`source.max_bytes` (next to `type` and `path`, not under `options`) caps each contract
|
|
63
63
|
document; the default is 64 MB. Remote contracts and generated clients honour
|
|
64
64
|
`HTTPS_PROXY`, `HTTP_PROXY`, and `NO_PROXY`. Proxy URLs must use `http://` without
|
|
65
|
-
credentials; HTTPS traffic is tunnelled with `CONNECT`.
|
|
65
|
+
credentials; HTTPS traffic is tunnelled with `CONNECT`. Certificates are verified against
|
|
66
|
+
the operating system's trust store (Windows certificate store, macOS keychain, the system
|
|
67
|
+
bundle on Linux), so TLS-inspecting corporate proxies whose root certificate is installed
|
|
68
|
+
there work without extra settings; `SSL_CERT_FILE` names a PEM bundle to use instead.
|
|
69
|
+
Redirects are never followed: a generated client reports a 3xx as an error that names
|
|
70
|
+
its `Location`.
|
|
71
|
+
|
|
72
|
+
YAML contracts are read with YAML 1.2 rules, as OpenAPI requires: `on`, `off`, `yes`,
|
|
73
|
+
`no`, `=`, `12:30` and dates stay strings, exactly as in the equivalent JSON.
|
|
74
|
+
|
|
75
|
+
Request bodies are sent as JSON when the contract offers it, else as
|
|
76
|
+
`multipart/form-data` or `application/x-www-form-urlencoded` (object fields become form
|
|
77
|
+
fields; binary fields become file parts; a bare binary body, or a list of files, is sent as
|
|
78
|
+
parts named `file`),
|
|
79
|
+
else as text or raw bytes. `*/*` sends raw bytes for a binary or schema-less body and JSON
|
|
80
|
+
otherwise. Binary values are `bytes`, a `str`, or a path (`os.PathLike`) in libraries, and
|
|
81
|
+
`TEXT`, `@FILE` or `@-` (stdin) on the command line; a file is sent under its own name.
|
|
66
82
|
|
|
67
83
|
### Pinned contracts and `codapact sync`
|
|
68
84
|
|
|
@@ -95,18 +111,22 @@ types. Without documents, it generates one operation per root field. A `#` comme
|
|
|
95
111
|
directly above a document operation becomes its description; variable defaults
|
|
96
112
|
(`$first: Int = 5`) become parameter defaults.
|
|
97
113
|
|
|
98
|
-
When the first shaping step is `select`,
|
|
99
|
-
|
|
114
|
+
When the first shaping step is `select`, it also filters before translation: operations it
|
|
115
|
+
excludes are never read, so problems in them (unsupported media types, odd enums, invalid
|
|
116
|
+
parameters) cannot block generation, and problems in schemas only those operations use are
|
|
117
|
+
reported as one warning. Model errors in operations it removes are likewise one warning.
|
|
118
|
+
This is the way to generate part of a very large contract in strict mode.
|
|
100
119
|
|
|
101
120
|
## Shaping transforms
|
|
102
121
|
|
|
103
122
|
| Transform | Config |
|
|
104
123
|
| --- | --- |
|
|
105
|
-
| `select` | `include`, `exclude`: lists of globs or `{name, tag, kind, namespace, read_only}` selectors |
|
|
124
|
+
| `select` | `include`, `exclude`: lists of globs or `{name, path, tag, kind, namespace, read_only}` selectors. Names match in any spelling (`getIssue`, `get-issue` and `get_issue` are the same); a glob starting with `/` or a method (`GET /rest/api/3/issue/*`) matches the HTTP path |
|
|
106
125
|
| `rename` | `operations`, `types`, `fields` (`Type.field`), `params` (`op.param`), `patterns` (`[{match: regex, replace: template}]` over operation names; explicit `operations` entries win); wire names are kept |
|
|
107
126
|
| `group` | `by: tag` or `by: path`, and/or `operations: {glob: dotted.namespace}` |
|
|
108
127
|
| `paginate` | `operations`, `style: cursor` (`param`, `next`), `offset` (`param`, `limit_param`), or `page` (`param`, default `page`; `start`, default `1`), and `items` (`$` when the result itself is the list). Operations matched only by a pattern that lack the param or items are skipped with a warning; operations named exactly are errors |
|
|
109
128
|
| `defaults` | `params: {"op_glob.param": value}`; the param becomes optional and the runtime sends `value` when it is omitted |
|
|
129
|
+
| `headers` | `{SELECTOR: {Header-Name: value}}` (selectors as in `select`, e.g. `addAttachment` or `POST /rest/*`). Each header becomes an optional header param defaulting to the value, so it is always sent unless overridden (`--x-atlassian-token`); a header the contract declares just gets the default. For vendor headers contracts mention only in prose, such as Jira's `X-Atlassian-Token: no-check` |
|
|
110
130
|
| `auth` | `schemes: {name: SPEC}` ([auth specs](#auth-specs)), `operations` (default all), `optional` (send credentials when set, call anonymously otherwise). For contracts that omit or under-declare `securitySchemes`; re-declaring an identical scheme is allowed |
|
|
111
131
|
| `prune` | none; removes unreachable types (also run by `select`) |
|
|
112
132
|
|
|
@@ -141,6 +161,7 @@ Precedence: flag, then environment, then settings file.
|
|
|
141
161
|
| Response validation | `--response-validation strict\|warn\|off` | `<PREFIX>_RESPONSE_VALIDATION` | `response_validation` |
|
|
142
162
|
| Response size limit | none | `<PREFIX>_MAX_RESPONSE_BYTES` | `max_response_bytes` (default 100 MB) |
|
|
143
163
|
| Output format | `-o`/`--output json\|jsonl\|yaml\|table\|raw` | `<PREFIX>_OUTPUT` | `output` |
|
|
164
|
+
| JSON style | none | `<PREFIX>_JSON_STYLE=auto\|pretty\|compact` | `json_style` |
|
|
144
165
|
| Error format | `--error-format text\|json` | `<PREFIX>_ERROR_FORMAT` | `error_format` |
|
|
145
166
|
| Credentials | none | `<PREFIX>_<SCHEME>`; basic uses `_USERNAME`/`_PASSWORD`; full chain under [Authentication](#authentication) | `auth_plugin`; secrets are never read from the file |
|
|
146
167
|
| Option defaults | the option itself | none | `defaults: {plan_id: "…"}` (option names, `_` or `-`) |
|
|
@@ -157,6 +178,8 @@ result is the item list, so `[].name`; a path that matches nothing fails with ex
|
|
|
157
178
|
`--allow-private`, and `--response-validation`. Command help lists them in one line;
|
|
158
179
|
`<program> --help` describes each.
|
|
159
180
|
`table` renders a list of objects, unwrapping single-list envelopes such as `{"data": {"items": [...]}}`.
|
|
181
|
+
JSON is indented on a terminal and compact (no whitespace) when stdout is a pipe or file,
|
|
182
|
+
where an agent or script reads it; `<PREFIX>_JSON_STYLE=pretty` or `compact` fixes one style.
|
|
160
183
|
`jsonl` prints one JSON value per line, unwrapping the same envelopes as `table`, and streams
|
|
161
184
|
`--all` results page by page. Output to a pipe or file is UTF-8 on every platform.
|
|
162
185
|
|
|
@@ -178,8 +201,8 @@ exists. Set `cli.env_prefix` too if the environment variables should keep their
|
|
|
178
201
|
| --- | --- |
|
|
179
202
|
| Help | `-h`/`--help` on every level; bare `<program>` or `<program> GROUP` prints help (exit `0`). Command help has a usage line with positionals and required options, examples, and a global-options reminder. |
|
|
180
203
|
| Version | `-V`/`--version`: `poke 0.1.0 (PokéAPI 2.10.0; codapact 0.4.0)`; the installed distribution's version when it differs from the generated `version` option (so a release tool can bump packaging alone) |
|
|
181
|
-
| Discovery for agents | `<program> describe [COMMAND…]`: JSON with every command's arguments (type, required, positional, choices, repeated), `read_only`/`destructive`/`paginated`, `requires_one_of` option groups, environment variables, exit codes, `start` (the steps an agent follows: doctor, credentials, describe, dry run) and `auth` (per scheme: environment variables, `auth login` commands, `docs` on how to get the credential, and whether every command or only some need it). `describe --brief` keeps those plus names, summaries, and usage lines, so one call is enough to start;
|
|
182
|
-
| Diagnosis | `<program> doctor [--offline]`: version, settings file, base URL and its source, credentials (set or missing, never printed), proxy, connectivity. Missing credentials fail only when every command needs them. Table on a terminal, JSON otherwise; exit `0` ready, `3` credentials missing or rejected, `4` API unreachable, `1` any other failed check |
|
|
204
|
+
| Discovery for agents | `<program> describe [COMMAND…]`: JSON with every command's arguments (type, required, positional, choices, repeated), `read_only`/`destructive`/`paginated`, `requires_one_of` option groups, environment variables, exit codes, `start` (the steps an agent follows: doctor, credentials, describe, dry run) and `auth` (per scheme: environment variables, `auth login` commands, `docs` on how to get the credential, and whether every command or only some need it). `describe --brief` keeps those plus names, summaries, and usage lines, so one call is enough to start; past 150 commands it lists groups (name, summary, command count) instead, and `describe GROUP --brief` lists a group. Agent skills for a coda point here instead of repeating it (see [Agent skills for codas](../extras/agent-plugin/README.md)). Naming one command adds the JSON Schema of each JSON-valued argument; `--schemas` adds that and the output JSON Schema to every command. Printed to a pipe, `describe` is compact JSON. `--help` names the members of each JSON argument, and examples include one option of each `requires_one_of` group (see [Discovery efficiency](discovery-efficiency/README.md)) |
|
|
205
|
+
| Diagnosis | `<program> doctor [--offline]`: version, settings file, base URL and its source, credentials (set or missing, never printed), proxy, connectivity (any HTTP answer counts, a redirect included), and, for schemes with `cli.auth.<scheme>.check`, whether the server still accepts the credential (`session: <scheme>`, see [CLI credential chain](#cli-credential-chain)). Missing credentials fail only when every command needs them. Table on a terminal, JSON otherwise; exit `0` ready, `3` credentials missing or rejected, `4` API unreachable, `1` any other failed check |
|
|
183
206
|
| Completion | `<program> completion bash\|zsh\|fish\|powershell` prints a script that asks the program for candidates |
|
|
184
207
|
| Preview | `--dry-run` prints the HTTP request (method, URL, headers, body) without sending it; credentials appear as `$VARIABLE` placeholders (optional credentials that are not set stay off the request), and destructive commands skip confirmation |
|
|
185
208
|
| Tracing | `-v`/`--verbose` logs each request line and response status, size, and time to stderr, masking credential values |
|
|
@@ -221,10 +244,14 @@ Three layers, each usable without the next:
|
|
|
221
244
|
| --- | --- | --- |
|
|
222
245
|
| Source | `AuthProvider.credential(scheme) -> Credential \| Authenticator \| None` | Where a secret comes from: environment, a command, the OS keyring, a vault |
|
|
223
246
|
| Built-in flow | `OAuth2(grant, store)` with `ClientCredentials`, `PasswordGrant`, `DeviceCode` (RFC 8628), `AuthorizationCodePKCE` (loopback redirect); refresh tokens; OIDC discovery | Standard OAuth 2.0 / OpenID Connect without code |
|
|
224
|
-
| Bespoke | `Authenticator.authenticate(request, scheme) -> request`, optional `refresh(scheme) -> bool` | Request signing (HMAC, AWS SigV4), token exchange, anything else |
|
|
247
|
+
| Bespoke | `Authenticator.authenticate(request, scheme) -> request`, optional `refresh(scheme) -> bool` and `expired(response, scheme) -> bool` | Request signing (HMAC, AWS SigV4), token exchange, browser sign-in sessions, anything else |
|
|
225
248
|
|
|
226
249
|
An `Authenticator` runs last, after middleware, so a retried request is signed again. On
|
|
227
250
|
HTTP 401 the runtime calls `refresh()` and, if it returns true, sends the request once more.
|
|
251
|
+
Servers that reject a session some other way (a redirect to a sign-in page, an HTML login
|
|
252
|
+
form, a vendor error code) are recognised by `expired()`, which sees every response: when it
|
|
253
|
+
returns true the runtime refreshes and resends the same way, and if the session is still
|
|
254
|
+
rejected the call fails with `AuthError` (CLI exit code 3).
|
|
228
255
|
|
|
229
256
|
### Auth specs
|
|
230
257
|
|
|
@@ -239,7 +266,7 @@ Used by the `auth` transform, the GraphQL `auth` option, and `cli.auth`:
|
|
|
239
266
|
| `{openid: DISCOVERY_URL}` | OAuth 2.0 whose endpoints come from `.well-known/openid-configuration` |
|
|
240
267
|
| `custom` or `{custom: DESCRIPTION}` | needs an `Authenticator` (plugin or library code) |
|
|
241
268
|
|
|
242
|
-
|
|
269
|
+
Every spec map also takes `docs` (for `custom` it replaces the description). OpenAPI `oauth2` (including 3.2 `deviceAuthorization`) and
|
|
243
270
|
`openIdConnect` map to the same model; other schemes (`mutualTLS`, `http` with an unknown
|
|
244
271
|
scheme) become `custom` with an info diagnostic.
|
|
245
272
|
|
|
@@ -261,7 +288,10 @@ The first source that has a credential wins, per scheme:
|
|
|
261
288
|
`cli.auth: {SCHEME: {client_id, scopes, flow, audience}}` bakes public OAuth settings into
|
|
262
289
|
the CLI (no secrets); `fields` declares [credential fields](#credential-fields); `docs` tells a
|
|
263
290
|
person how to get the credential (shown in `describe` under `auth`, e.g. "Create a personal
|
|
264
|
-
access token under Account Settings > Developer Settings")
|
|
291
|
+
access token under Account Settings > Developer Settings"); `check` names an operation that
|
|
292
|
+
needs no arguments (any spelling, e.g. `getCurrentUser`) which `doctor` calls to prove the
|
|
293
|
+
saved credential or session still works: a 401, 403, redirect or `expired()` fails the
|
|
294
|
+
`session: <scheme>` check with exit code 3.
|
|
265
295
|
`auth login` picks a flow in this order: `--flow`, then client credentials when a secret is
|
|
266
296
|
set, then device code, then authorization code with PKCE.
|
|
267
297
|
`--with-token` reads a static token from stdin; `--username` saves basic credentials.
|
|
@@ -0,0 +1,240 @@
|
|
|
1
|
+
# How efficiently an agent learns a generated CLI
|
|
2
|
+
|
|
3
|
+
An agent that has never seen an API has to read something before it can make its first
|
|
4
|
+
correct call. This report measures how much it reads, and how many tool calls it spends,
|
|
5
|
+
for the discovery flow every codapact CLI ships and for the alternatives an agent would
|
|
6
|
+
otherwise use: `--help`, the full `describe`, MCP tool schemas, the raw OpenAPI document,
|
|
7
|
+
and llms.txt-style docs.
|
|
8
|
+
|
|
9
|
+
Everything here is regenerated by two scripts in
|
|
10
|
+
[`scripts/discovery_bench/`](../../scripts/discovery_bench/); see [Methodology](#methodology).
|
|
11
|
+
|
|
12
|
+
Measured 2026-11-17 on `main` at 1b80666, which includes every change in
|
|
13
|
+
[What changed](#what-changed) ([#61](https://github.com/mikejhill/codapact/pull/61), [#62](https://github.com/mikejhill/codapact/pull/62)), with codapact 0.9.1's behavior simulated from the
|
|
14
|
+
same output as the baseline.
|
|
15
|
+
|
|
16
|
+
## Summary
|
|
17
|
+
|
|
18
|
+
| | |
|
|
19
|
+
| --- | --- |
|
|
20
|
+
| **The codapact flow** (`describe --brief`, then `describe COMMAND`) | **2.7k tokens, 2 calls** on YNAB, down from 6.8k in 0.9.1. Always enough for a correct first call, including JSON request bodies. 5× cheaper than full `describe`, 10× cheaper than the OpenAPI document, and under a fifth of the 15k that MCP puts in context on every turn. |
|
|
21
|
+
| **Large APIs** | On GitHub REST (1,279 commands) the flow now costs **5.2k tokens in 3 calls**, down from 87k: `describe --brief` lists groups once a CLI has more than 150 commands. |
|
|
22
|
+
| **Over a session** | Ten distinct YNAB commands cost **4.9k tokens** in context, down from 23.6k. 0.9.1 overtook loading every MCP tool at six commands; the flow now stays at about a third of it. |
|
|
23
|
+
| **Cheapest in tokens** | MCP with client-side tool search (0.6k) and the `--help` walk (1.5k). Tool search needs client support and keeps every tool name resident (14k on GitHub). `--help` costs 3 calls and has no JSON Schemas; it now names each JSON argument's fields. |
|
|
24
|
+
| **Worst** | Loading every MCP tool (15k on YNAB, 514k on GitHub, on every turn), the full OpenAPI document (26k, 2.9M), and llms-full.txt (94k, 5.1M). |
|
|
25
|
+
|
|
26
|
+
## What changed
|
|
27
|
+
|
|
28
|
+
The first version of this report (0.9.1) found three fixes worth making, and a follow-up
|
|
29
|
+
compacted command output too. All are merged into
|
|
30
|
+
the runtime and generator now, so every coda gets them when it is regenerated.
|
|
31
|
+
|
|
32
|
+
| Finding in 0.9.1 | Change | Effect | PR |
|
|
33
|
+
| --- | --- | --- | --- |
|
|
34
|
+
| 86% of `describe COMMAND` was the output JSON Schema, which the agent needs only after the call | `describe COMMAND` returns arguments and JSON argument schemas; `--schemas` adds the output schema | YNAB median entry: 2.0k → 0.2k tokens | [#61](https://github.com/mikejhill/codapact/pull/61) |
|
|
35
|
+
| JSON was indented | `describe` prints compact JSON when stdout is not a terminal | about a third fewer tokens in every `describe` output | [#61](https://github.com/mikejhill/codapact/pull/61) |
|
|
36
|
+
| `describe --brief` listed every command (87k tokens on GitHub REST) | Past 150 commands it lists groups (name, summary, count); `start` tells the agent to run `describe GROUP --brief` | GitHub REST catalog: 86k → 2.3k tokens, plus one call for a group | [#61](https://github.com/mikejhill/codapact/pull/61) |
|
|
37
|
+
| Command output was indented JSON everywhere | Every JSON result prints compact when stdout is not a terminal (see [Responses](#responses)); `<PREFIX>_JSON_STYLE=pretty\|compact` overrides | 7% to 15% fewer tokens per response | [#62](https://github.com/mikejhill/codapact/pull/62) |
|
|
38
|
+
| `--help` gave JSON arguments no hint; the `transactions create` example omitted the required `--transaction`, so copying it failed | `--help` names each JSON argument's fields (required ones marked) and points at `describe`; examples include one option of each `requires_one_of` group | the cheapest flow says enough for most JSON bodies; examples run | [#61](https://github.com/mikejhill/codapact/pull/61) |
|
|
39
|
+
|
|
40
|
+
Not changed: there is still no MCP target. MCP only wins with client-side tool search,
|
|
41
|
+
which not every client has, and loses badly without it (see [Pros and cons](#pros-and-cons)).
|
|
42
|
+
|
|
43
|
+
## Results
|
|
44
|
+
|
|
45
|
+
### YNAB: what an agent reads before its first correct call
|
|
46
|
+
|
|
47
|
+
Six tasks, from "list payees" to "record a 12.34 purchase" (a JSON request body in
|
|
48
|
+
milliunits). Each technique is scored on the tokens it returns and the tool calls it takes
|
|
49
|
+
until the agent has every fact the task needs. Every reference call passes `--dry-run`.
|
|
50
|
+
|
|
51
|
+

|
|
52
|
+
|
|
53
|
+
| Technique | Tokens (median) | Calls | Notes |
|
|
54
|
+
| --- | ---: | ---: | --- |
|
|
55
|
+
| MCP + tool search | 558 | 1 | plus 256 resident (every tool name), every turn |
|
|
56
|
+
| `--help` walk (root, group, command) | 1,530 | 3 | `transactions create`: the fields are listed but not their units (milliunits), so it falls back to `describe COMMAND` (4.3k, 4 calls) |
|
|
57
|
+
| `--help`, `describe GROUP --brief`, `describe COMMAND` | 1,689 | 3 | group-first; what large CLIs now do |
|
|
58
|
+
| **`describe --brief`, `describe COMMAND`** | **2,669** | **2** | the flow in the coda skill |
|
|
59
|
+
| OpenAPI, targeted reads | 2,781 | 6.5 | assumes perfect grep; the agent then writes the HTTP call itself |
|
|
60
|
+
| llms.txt, then one page | 4,791 | 2 | same: no execution layer |
|
|
61
|
+
| The same flow in 0.9.1 | 6,782 | 2 | indented, output schema included |
|
|
62
|
+
| Full `describe` | 13,865 | 1 | lacks JSON argument schemas: `transactions create` needs `describe COMMAND` too |
|
|
63
|
+
| MCP, every tool loaded | 15,034 | 0 | resident on every turn |
|
|
64
|
+
| Raw OpenAPI document | 26,273 | 1 | |
|
|
65
|
+
| llms-full.txt | 93,710 | 1 | |
|
|
66
|
+
|
|
67
|
+
Per-task numbers are in [`data/results.json`](data/results.json) under `apis.ynab.tasks`.
|
|
68
|
+
|
|
69
|
+
### Scaling with API size
|
|
70
|
+
|
|
71
|
+
Median over up to 60 evenly spaced commands per API: Petstore (5 commands), YNAB (44),
|
|
72
|
+
PokéAPI (102), and the full GitHub REST description (1,279).
|
|
73
|
+
|
|
74
|
+

|
|
75
|
+
|
|
76
|
+
| API (commands) | brief + describe CMD (now) | Same in 0.9.1 | `--help` walk | MCP, every tool | MCP + tool search | Full describe | OpenAPI document |
|
|
77
|
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
78
|
+
| Petstore (5) | 0.7k | 1.3k | 1.0k | 0.4k | 0.1k | 1.4k | 0.9k |
|
|
79
|
+
| YNAB (44) | 2.5k | 5.1k | 1.4k | 15.0k | 0.4k | 13.8k | 26.3k |
|
|
80
|
+
| PokéAPI (102) | 3.8k | 6.5k | 1.4k | 14.8k | 0.8k | 17.2k | 67.9k |
|
|
81
|
+
| GitHub REST (1,279) | 5.2k (3 calls) | 87.4k | 3.5k | 514k | 14.3k | 482k | 2.9M |
|
|
82
|
+
|
|
83
|
+
Anything that loads the whole API at once (full `describe`, every MCP tool, the OpenAPI
|
|
84
|
+
document) grows linearly and passes a 200k context window somewhere between 100 and 1,279
|
|
85
|
+
commands. The codapact flow and `--help` now grow only with the size of one group.
|
|
86
|
+
|
|
87
|
+
### A whole session
|
|
88
|
+
|
|
89
|
+
An agent rarely makes one call. Each new command costs the codapact flow another
|
|
90
|
+
`describe COMMAND`, while MCP pays once up front and then resends the same schemas with
|
|
91
|
+
every turn. CLI rows include the coda skill (459 tokens).
|
|
92
|
+
|
|
93
|
+

|
|
94
|
+
|
|
95
|
+
In 0.9.1, each `describe COMMAND` carried a 1.7k output schema, so the flow overtook
|
|
96
|
+
loading every MCP tool at six distinct commands. Now a command costs about 200 tokens, and
|
|
97
|
+
ten commands fit in 4.9k.
|
|
98
|
+
|
|
99
|
+
## Pros and cons
|
|
100
|
+
|
|
101
|
+
| Technique | Pros | Cons |
|
|
102
|
+
| --- | --- | --- |
|
|
103
|
+
| **`describe --brief` + `describe COMMAND`** | Two calls (three on large CLIs), always sufficient. Carries `start`, `auth`, `exit_codes` and notes, so the agent learns readiness, credentials, and error handling in the same call. Typed JSON with argument schemas. Works in any agent with a shell. | One `describe COMMAND` per new command. |
|
|
104
|
+
| `--help` walk | Fewest tokens of the CLI flows. Every CLI user already knows it. Scales well, since each level only lists its children. Names JSON fields. | Three calls. Field names only, without types or units. No auth or exit-code summary unless the agent also reads root help. Text, not JSON. |
|
|
105
|
+
| Full `describe` | One call. | Grows linearly (14k on YNAB, 482k on GitHub). Omits JSON argument schemas unless `--schemas`, which makes it larger still. |
|
|
106
|
+
| MCP, every tool loaded | Zero discovery calls. Typed arguments that clients validate. Standard protocol. | All schemas sit in context on every turn, used or not (15k YNAB, 514k GitHub). Needs a running server and client configuration per agent. Clients cap tool counts. Credentials live in the server config. |
|
|
107
|
+
| MCP + tool search | Cheapest per call. Only the schemas used are loaded. | Depends on the client (not every MCP client defers tools). Every tool name stays resident (14k on GitHub). Still needs a server process. |
|
|
108
|
+
| Raw OpenAPI document | Authoritative; nothing to generate. | Too large past small APIs (2.9M tokens for GitHub). The agent must build URLs, headers, auth, pagination, and error handling itself. No dry run, no validation, no exit codes. |
|
|
109
|
+
| OpenAPI, targeted reads | Few tokens when the agent greps well. | 4 to 7 calls following `$ref` chains. The same execution burden as the raw document. Assumes the agent knows how to search YAML or JSON. |
|
|
110
|
+
| llms.txt, then one page | Two calls; readable Markdown. | The index grows linearly (53k on GitHub). Docs only: the agent still writes HTTP calls. Has to be generated and hosted. |
|
|
111
|
+
| llms-full.txt | One call. | The largest option on every API. |
|
|
112
|
+
|
|
113
|
+
What tokens do not show: the CLI and MCP flows also give the agent an **execution layer**.
|
|
114
|
+
A codapact CLI reads credentials from the keyring or environment, validates arguments
|
|
115
|
+
before sending, offers `--dry-run`, maps failures to exit codes with hints, follows
|
|
116
|
+
pagination, and trims responses with `--pick` and `--query`. With the OpenAPI or llms.txt
|
|
117
|
+
approaches the agent writes that layer itself, on every call, and its first attempts are
|
|
118
|
+
where errors happen.
|
|
119
|
+
|
|
120
|
+
Latency is not a factor: a warm `uvx codapact-ynab` call takes about 0.25 s.
|
|
121
|
+
|
|
122
|
+
### Responses
|
|
123
|
+
|
|
124
|
+
After discovery, an agent reads what commands return. Real GitHub REST responses for this
|
|
125
|
+
repository, printed indented (as 0.9.1 did everywhere) and compact (as a pipe now gets):
|
|
126
|
+
|
|
127
|
+
| Response | Indented | Compact | Saved |
|
|
128
|
+
| --- | ---: | ---: | ---: |
|
|
129
|
+
| repository (object) | 2.4k | 2.0k | 15% |
|
|
130
|
+
| 30 pull requests (list) | 203k | 177k | 13% |
|
|
131
|
+
| 30 commits (list) | 96k | 90k | 7% |
|
|
132
|
+
| 10 releases (list) | 38k | 33k | 13% |
|
|
133
|
+
|
|
134
|
+
Responses save less than `describe` did (11% overall against about a third), because their
|
|
135
|
+
long string values outweigh the indentation. In absolute terms it is still the larger win:
|
|
136
|
+
one list of 30 pull requests saves 26k tokens, more than the whole discovery flow costs.
|
|
137
|
+
The bigger lever for responses remains `--pick` and `--query`, which drop fields the agent
|
|
138
|
+
does not need.
|
|
139
|
+
|
|
140
|
+
**When output is compact.** JSON is compact exactly when stdout is not a terminal. That is
|
|
141
|
+
the case for agents that capture command output (Claude Code's shell tool, for one, is not
|
|
142
|
+
a terminal), for scripts, for redirects to files, and for pipes into `jq`, which
|
|
143
|
+
pretty-prints anyway. A person at a terminal still sees indented JSON. `<PREFIX>_JSON_STYLE` (or
|
|
144
|
+
`json_style` in the settings file) fixes one style: `pretty` for files people read,
|
|
145
|
+
`compact` for an agent that runs commands in a pseudo-terminal. YAML, table, JSON Lines and
|
|
146
|
+
raw output are unchanged; JSON Lines was already compact.
|
|
147
|
+
|
|
148
|
+
## Open questions
|
|
149
|
+
|
|
150
|
+
- **Live agent runs.** The steps here are modeled. Running a few real agents through the
|
|
151
|
+
YNAB tasks would show how often they read more than needed or retry.
|
|
152
|
+
|
|
153
|
+
## Methodology
|
|
154
|
+
|
|
155
|
+
### What is measured
|
|
156
|
+
|
|
157
|
+
For each API, each command, and each technique, the benchmark records what the agent
|
|
158
|
+
must read, in order, before it can make a correct call to that command:
|
|
159
|
+
|
|
160
|
+
- **Tokens**: the sum of every text the technique returns along the way.
|
|
161
|
+
- **Calls**: tool calls the agent makes. MCP tool schemas arrive with the prompt and count
|
|
162
|
+
as 0 calls; they are reported separately as *resident* tokens because the client resends
|
|
163
|
+
them on every turn.
|
|
164
|
+
- **Sufficiency**: a correct call needs certain facts. For every command, these are its
|
|
165
|
+
required arguments and the first three required fields of any JSON argument. The YNAB
|
|
166
|
+
tasks add hand-picked facts (for example `milliunits` for money amounts and `current`
|
|
167
|
+
for the current month). A fact counts only if it appears in the part of the text about
|
|
168
|
+
the target command, so a large catalog does not pass by mentioning a field elsewhere.
|
|
169
|
+
Matching ignores case and `-`/`_` separators (`--pet-id` matches `petId`). When a CLI
|
|
170
|
+
technique lacks a fact, the agent falls back to `describe COMMAND`, and that call is
|
|
171
|
+
added. Other techniques have no fallback, and missing facts are recorded.
|
|
172
|
+
|
|
173
|
+
### Techniques
|
|
174
|
+
|
|
175
|
+
| Key | What the agent reads |
|
|
176
|
+
| --- | --- |
|
|
177
|
+
| `brief` | `describe --brief`, then `describe GROUP --brief` if it listed groups, then `describe COMMAND` (the flow in the coda skill) |
|
|
178
|
+
| `brief_v091` | Baseline: the same flow as 0.9.1 printed it, rebuilt from today's output (indented JSON, every command in `--brief`, output schema in `describe COMMAND`) |
|
|
179
|
+
| `group_brief` | `--help`, `describe GROUP --brief`, `describe COMMAND` |
|
|
180
|
+
| `full` | `describe` (fallback: `describe COMMAND`) |
|
|
181
|
+
| `help` | `--help`, `GROUP --help`, `GROUP COMMAND --help` (fallback: `describe COMMAND`) |
|
|
182
|
+
| `mcp` | A `tools/list` result with one tool per command: `name`, `description`, `inputSchema` built from the command's `describe` entry (JSON arguments keep their full schema), and annotations. This is what an OpenAPI-to-MCP bridge publishes. |
|
|
183
|
+
| `mcp_search` | Every tool name resident, then one tool definition loaded by search |
|
|
184
|
+
| `openapi` | The contract file as committed |
|
|
185
|
+
| `openapi_slice` | A path-search hit, `servers` and `securitySchemes`, then the operation plus every component it references, one call per `$ref` level |
|
|
186
|
+
| `llms` | An llms.txt index (one line per operation), then that operation's Markdown page with its schemas inline |
|
|
187
|
+
| `llms_full` | Every page concatenated |
|
|
188
|
+
|
|
189
|
+
### APIs
|
|
190
|
+
|
|
191
|
+
All generated by this checkout of codapact from contracts in the repository, except GitHub:
|
|
192
|
+
|
|
193
|
+
- Petstore and Library: `examples/`.
|
|
194
|
+
- Countries, YNAB, PokéAPI: `tests/fixtures/corpus/`. YNAB uses the settings of the
|
|
195
|
+
published coda (`mikejhill/codapact-ynab`), so the program is `codapact-ynab`.
|
|
196
|
+
- GitHub REST (full): `github/rest-api-description@5c36b6f6`
|
|
197
|
+
`descriptions/api.github.com/api.github.com.json` (14 MB), generated with `strict: false`.
|
|
198
|
+
Only with `--github`.
|
|
199
|
+
|
|
200
|
+
The two GraphQL examples have two commands each; they appear in the data but not in the
|
|
201
|
+
scaling figure.
|
|
202
|
+
|
|
203
|
+
### Tokens
|
|
204
|
+
|
|
205
|
+
Counted with the Claude tokenizer that shipped in `anthropic==0.34.2` (`tokenizer.json`),
|
|
206
|
+
the only Claude tokenizer that runs offline. Current Claude models tokenize differently, so
|
|
207
|
+
treat absolute numbers as approximate. Ratios between techniques, which is what the
|
|
208
|
+
conclusions rest on, change much less, because every technique is measured with the same
|
|
209
|
+
tokenizer on similar text. To recount with a current model, send the same texts to the
|
|
210
|
+
token-counting API.
|
|
211
|
+
|
|
212
|
+
### Limits
|
|
213
|
+
|
|
214
|
+
- Agents are modeled, not run. The benchmark assumes the agent picks the right command
|
|
215
|
+
(or group) from the list and reads nothing it does not need. Real agents sometimes read
|
|
216
|
+
more, or guess and retry; this report does not measure retries.
|
|
217
|
+
- Tool-call framing and the agent's own reasoning are not counted. Response bodies are
|
|
218
|
+
measured separately under [Responses](#responses).
|
|
219
|
+
- MCP tools are serialized as compact JSON. Clients wrap them in their own format, which
|
|
220
|
+
adds a little per tool.
|
|
221
|
+
- `openapi_slice` assumes perfect search; it is a lower bound.
|
|
222
|
+
|
|
223
|
+
### Reproduce
|
|
224
|
+
|
|
225
|
+
From the repository root:
|
|
226
|
+
|
|
227
|
+
```sh
|
|
228
|
+
uv sync
|
|
229
|
+
uv run --with anthropic==0.34.2 --with tokenizers \
|
|
230
|
+
python scripts/discovery_bench/collect.py --github # about 3 minutes; drop --github for 1
|
|
231
|
+
uv run python scripts/discovery_bench/plot.py
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
`collect.py` writes [`data/results.json`](data/results.json): tool versions, every API's
|
|
235
|
+
sizes, per-command results for each technique, the YNAB tasks with their reference calls
|
|
236
|
+
and `--dry-run` exit codes, warm `uvx codapact-ynab` latency (skip with
|
|
237
|
+
`--no-latency`), and the response sizes above, fetched from the public GitHub API (skip
|
|
238
|
+
with `--no-responses`). `plot.py` redraws the three SVGs in [`figures/`](figures/) from that file
|
|
239
|
+
alone. To add a task, append to `YNAB_TASKS` in `collect.py`; to add an API, add it to
|
|
240
|
+
`apis()`.
|