codapact 0.9.0__tar.gz → 0.10.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (189) hide show
  1. {codapact-0.9.0 → codapact-0.10.0}/.github/workflows/release.yml +5 -1
  2. codapact-0.10.0/.release-please-manifest.json +1 -0
  3. {codapact-0.9.0 → codapact-0.10.0}/CHANGELOG.md +25 -0
  4. {codapact-0.9.0 → codapact-0.10.0}/PKG-INFO +7 -4
  5. {codapact-0.9.0 → codapact-0.10.0}/README.md +5 -3
  6. {codapact-0.9.0 → codapact-0.10.0}/docs/architecture.md +6 -5
  7. {codapact-0.9.0 → codapact-0.10.0}/docs/configuration.md +39 -9
  8. codapact-0.10.0/docs/discovery-efficiency/README.md +240 -0
  9. codapact-0.10.0/docs/discovery-efficiency/data/results.json +16348 -0
  10. codapact-0.10.0/docs/discovery-efficiency/figures/scaling-by-api-size.svg +90 -0
  11. codapact-0.10.0/docs/discovery-efficiency/figures/ynab-session-context.svg +137 -0
  12. codapact-0.10.0/docs/discovery-efficiency/figures/ynab-tokens-to-first-call.svg +93 -0
  13. codapact-0.10.0/examples/README.md +24 -0
  14. codapact-0.10.0/examples/confluence/codapact.yaml +18 -0
  15. codapact-0.10.0/examples/github/codapact.yaml +21 -0
  16. codapact-0.10.0/examples/jira/codapact.yaml +19 -0
  17. codapact-0.10.0/examples/swagger-petstore/codapact.yaml +19 -0
  18. codapact-0.10.0/extras/agent-plugin/.claude-plugin/plugin.json +8 -0
  19. codapact-0.10.0/extras/agent-plugin/README.md +37 -0
  20. codapact-0.10.0/extras/agent-plugin/plugin.json +8 -0
  21. codapact-0.10.0/extras/agent-plugin/skills/create-coda-skill/SKILL.md +100 -0
  22. codapact-0.10.0/extras/agent-plugin/skills/create-coda-skill/assets/SKILL.template.md +39 -0
  23. {codapact-0.9.0 → codapact-0.10.0}/pyproject.toml +7 -1
  24. {codapact-0.9.0 → codapact-0.10.0}/pyproject.toml.orig +8 -3
  25. codapact-0.10.0/scripts/discovery_bench/collect.py +882 -0
  26. codapact-0.10.0/scripts/discovery_bench/plot.py +355 -0
  27. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/__init__.py +1 -1
  28. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/core/net.py +31 -7
  29. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/authspec.py +13 -5
  30. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/pipeline/pipeline.py +5 -1
  31. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/abstractions.py +6 -0
  32. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/cli.py +100 -18
  33. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/cli_tools.py +97 -21
  34. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/codec.py +2 -1
  35. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/errors.py +16 -2
  36. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/protocols/base.py +25 -5
  37. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/protocols/http.py +84 -4
  38. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/shaping/transforms.py +145 -10
  39. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/sources/base.py +8 -2
  40. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/sources/loader.py +40 -1
  41. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/sources/openapi/translator.py +176 -13
  42. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/python/cli.py +85 -11
  43. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/python/library.py +2 -0
  44. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/python/naming.py +3 -1
  45. {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_cli_conformance.py +33 -3
  46. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_cli_builtins.py +81 -1
  47. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_cli_options.py +2 -2
  48. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_guards.py +3 -2
  49. codapact-0.10.0/tests/unit/test_large_contract_findings.py +742 -0
  50. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_live_ynab_findings.py +1 -1
  51. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_openapi_edges.py +1 -1
  52. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_public_api_findings.py +1 -1
  53. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_target_rendering.py +6 -1
  54. {codapact-0.9.0 → codapact-0.10.0}/uv.lock +12 -1
  55. codapact-0.9.0/.release-please-manifest.json +0 -1
  56. {codapact-0.9.0 → codapact-0.10.0}/.github/workflows/ci.yml +0 -0
  57. {codapact-0.9.0 → codapact-0.10.0}/.github/workflows/codeql.yml +0 -0
  58. {codapact-0.9.0 → codapact-0.10.0}/.github/workflows/mutation.yml +0 -0
  59. {codapact-0.9.0 → codapact-0.10.0}/.github/workflows/nightly.yml +0 -0
  60. {codapact-0.9.0 → codapact-0.10.0}/LICENSE +0 -0
  61. {codapact-0.9.0 → codapact-0.10.0}/docs/branding/README.md +0 -0
  62. {codapact-0.9.0 → codapact-0.10.0}/docs/branding/mark.svg +0 -0
  63. {codapact-0.9.0 → codapact-0.10.0}/docs/branding/wordmark.svg +0 -0
  64. {codapact-0.9.0 → codapact-0.10.0}/docs/testing.md +0 -0
  65. {codapact-0.9.0 → codapact-0.10.0}/examples/library/codapact.yaml +0 -0
  66. {codapact-0.9.0 → codapact-0.10.0}/examples/library/operations.graphql +0 -0
  67. {codapact-0.9.0 → codapact-0.10.0}/examples/library/schema.graphql +0 -0
  68. {codapact-0.9.0 → codapact-0.10.0}/examples/petstore/codapact.yaml +0 -0
  69. {codapact-0.9.0 → codapact-0.10.0}/examples/petstore/openapi.yaml +0 -0
  70. {codapact-0.9.0 → codapact-0.10.0}/release-please-config.json +0 -0
  71. {codapact-0.9.0 → codapact-0.10.0}/scripts/extract_openapi_subset.py +0 -0
  72. {codapact-0.9.0 → codapact-0.10.0}/scripts/mutation_score.py +0 -0
  73. {codapact-0.9.0 → codapact-0.10.0}/scripts/verify_dist.py +0 -0
  74. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/__main__.py +0 -0
  75. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/commands/__init__.py +0 -0
  76. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/commands/derive.py +0 -0
  77. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/commands/model.py +0 -0
  78. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/core/__init__.py +0 -0
  79. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/core/diagnostics.py +0 -0
  80. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/core/errors.py +0 -0
  81. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/core/naming.py +0 -0
  82. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/__init__.py +0 -0
  83. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/api.py +0 -0
  84. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/migrations.py +0 -0
  85. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/rewrite.py +0 -0
  86. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/serialize.py +0 -0
  87. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/types.py +0 -0
  88. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/validate.py +0 -0
  89. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/model/values.py +0 -0
  90. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/pipeline/__init__.py +0 -0
  91. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/pipeline/cli.py +0 -0
  92. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/pipeline/config.py +0 -0
  93. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/pipeline/defaults.py +0 -0
  94. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/pipeline/sync.py +0 -0
  95. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/pipeline/writer.py +0 -0
  96. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/plugins/__init__.py +0 -0
  97. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/plugins/external.py +0 -0
  98. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/plugins/protocol.py +0 -0
  99. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/plugins/registry.py +0 -0
  100. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/py.typed +0 -0
  101. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/__init__.py +0 -0
  102. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/arguments.py +0 -0
  103. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/auth.py +0 -0
  104. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/cli_auth.py +0 -0
  105. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/client.py +0 -0
  106. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/credentials.py +0 -0
  107. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/middleware.py +0 -0
  108. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/oauth.py +0 -0
  109. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/profiles.py +0 -0
  110. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/protocols/__init__.py +0 -0
  111. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/protocols/graphql.py +0 -0
  112. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/runtime/selftest.py +0 -0
  113. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/shaping/__init__.py +0 -0
  114. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/shaping/base.py +0 -0
  115. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/shaping/engine.py +0 -0
  116. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/sources/__init__.py +0 -0
  117. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/sources/graphql/__init__.py +0 -0
  118. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/sources/graphql/translator.py +0 -0
  119. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/sources/openapi/__init__.py +0 -0
  120. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/surface/__init__.py +0 -0
  121. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/surface/derive.py +0 -0
  122. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/surface/model.py +0 -0
  123. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/__init__.py +0 -0
  124. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/python/__init__.py +0 -0
  125. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/python/e2e.py +0 -0
  126. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/typescript/__init__.py +0 -0
  127. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/typescript/library.py +0 -0
  128. {codapact-0.9.0 → codapact-0.10.0}/src/codapact/targets/typescript/runtime.ts +0 -0
  129. {codapact-0.9.0 → codapact-0.10.0}/tests/authlab.py +0 -0
  130. {codapact-0.9.0 → codapact-0.10.0}/tests/conftest.py +0 -0
  131. {codapact-0.9.0 → codapact-0.10.0}/tests/e2e/test_auth_e2e.py +0 -0
  132. {codapact-0.9.0 → codapact-0.10.0}/tests/e2e/test_cli_e2e.py +0 -0
  133. {codapact-0.9.0 → codapact-0.10.0}/tests/e2e/test_public_apis.py +0 -0
  134. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/authlab/authlab_plugin.py +0 -0
  135. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/authlab/codapact.yaml +0 -0
  136. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/authlab/fake_browser.py +0 -0
  137. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/authlab/openapi.yaml +0 -0
  138. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/authlab_smoke.ts +0 -0
  139. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/contracts/wire.yaml +0 -0
  140. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/README.md +0 -0
  141. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/countries/codapact.yaml +0 -0
  142. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/countries/operations.graphql +0 -0
  143. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/countries/schema.graphql +0 -0
  144. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/github/codapact.yaml +0 -0
  145. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/github/openapi.json +0 -0
  146. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/pokeapi/codapact.yaml +0 -0
  147. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/pokeapi/openapi.yml +0 -0
  148. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/ynab/README.md +0 -0
  149. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/ynab/codapact.yaml +0 -0
  150. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/corpus/ynab/openapi.yaml +0 -0
  151. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/library_smoke.ts +0 -0
  152. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/outline_plugin.py +0 -0
  153. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/petstore_smoke.ts +0 -0
  154. {codapact-0.9.0 → codapact-0.10.0}/tests/fixtures/wire_parity.ts +0 -0
  155. {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_cli_conventions.py +0 -0
  156. {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_contract_properties.py +0 -0
  157. {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_corpus.py +0 -0
  158. {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_generated_code_properties.py +0 -0
  159. {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_generation.py +0 -0
  160. {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_public_api_cli.py +0 -0
  161. {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_wire_parity.py +0 -0
  162. {codapact-0.9.0 → codapact-0.10.0}/tests/integration/test_ynab_contract.py +0 -0
  163. {codapact-0.9.0 → codapact-0.10.0}/tests/perf/test_scaling.py +0 -0
  164. {codapact-0.9.0 → codapact-0.10.0}/tests/public_api_contract.py +0 -0
  165. {codapact-0.9.0 → codapact-0.10.0}/tests/strategies.py +0 -0
  166. {codapact-0.9.0 → codapact-0.10.0}/tests/support.py +0 -0
  167. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_architecture.py +0 -0
  168. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_auth_and_graphql_sources.py +0 -0
  169. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_auth_sources.py +0 -0
  170. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_bundle_e2e.py +0 -0
  171. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_cli_auth.py +0 -0
  172. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_codec_serialize.py +0 -0
  173. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_core.py +0 -0
  174. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_credential_fields.py +0 -0
  175. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_env_prefix.py +0 -0
  176. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_model.py +0 -0
  177. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_mutation_score.py +0 -0
  178. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_oauth.py +0 -0
  179. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_openapi_translator_units.py +0 -0
  180. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_plugins_pipeline.py +0 -0
  181. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_profiles.py +0 -0
  182. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_program_rename.py +0 -0
  183. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_runtime.py +0 -0
  184. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_runtime_edges.py +0 -0
  185. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_runtime_pagination.py +0 -0
  186. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_shaping_surface_commands.py +0 -0
  187. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_sources.py +0 -0
  188. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_sync.py +0 -0
  189. {codapact-0.9.0 → codapact-0.10.0}/tests/unit/test_usability.py +0 -0
@@ -28,7 +28,10 @@ jobs:
28
28
  token: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }}
29
29
  config-file: release-please-config.json
30
30
  manifest-file: .release-please-manifest.json
31
- - name: Start CI for release pull request
31
+ # Pushes made with GITHUB_TOKEN start no pull_request workflows, and GitHub holds
32
+ # CodeQL for approval on such pull requests. Dispatching is allowed, and the
33
+ # required CodeQL results for the release branch's head come from this run.
34
+ - name: Start CI and CodeQL for release pull request
32
35
  if: steps.release.outputs.prs_created == 'true'
33
36
  env:
34
37
  GH_TOKEN: ${{ github.token }}
@@ -36,6 +39,7 @@ jobs:
36
39
  run: |
37
40
  branch=$(jq -r '.headBranchName' <<< "$PR_JSON")
38
41
  gh workflow run ci.yml --repo "$GITHUB_REPOSITORY" --ref "$branch"
42
+ gh workflow run codeql.yml --repo "$GITHUB_REPOSITORY" --ref "$branch"
39
43
  ci:
40
44
  needs: version
41
45
  if: needs.version.outputs.created == 'true'
@@ -0,0 +1 @@
1
+ {".":"0.10.0"}
@@ -2,6 +2,31 @@
2
2
 
3
3
  <!-- markdownlint-configure-file { "MD024": { "siblings_only": true } } -->
4
4
 
5
+ ## [0.10.0](https://github.com/mikejhill/codapact/compare/v0.9.1...v0.10.0) (2026-11-19)
6
+
7
+
8
+ ### Features
9
+
10
+ * **cli:** compact JSON output when stdout is not a terminal ([#62](https://github.com/mikejhill/codapact/issues/62)) ([1b80666](https://github.com/mikejhill/codapact/commit/1b80666128834f5ba2a13d5e21e301e6f60fbafd))
11
+ * **cli:** leaner agent discovery (lean describe COMMAND, compact JSON, grouped --brief, JSON field hints) ([#61](https://github.com/mikejhill/codapact/issues/61)) ([4070dea](https://github.com/mikejhill/codapact/commit/4070deab7eb77dabeb65cda52228af6ab8ac7232))
12
+ * generate large real-world contracts strictly; session checks and expiry hooks ([#58](https://github.com/mikejhill/codapact/issues/58)) ([523e6b3](https://github.com/mikejhill/codapact/commit/523e6b3ac156fb7f6516d157c4b1f7809519ebee))
13
+ * headers shaping step; multipart lists of files ([#60](https://github.com/mikejhill/codapact/issues/60)) ([7bbb762](https://github.com/mikejhill/codapact/commit/7bbb762a97aba656803d08ade48111747647a8b5))
14
+ * move the agent plugin to extras/agent-plugin; dispatch CodeQL for release PRs ([#65](https://github.com/mikejhill/codapact/issues/65)) ([3e6c2f1](https://github.com/mikejhill/codapact/commit/3e6c2f14b69a1e86f92d0a4cc19f104f89a99d5d))
15
+ * move the agent plugin to plugins/codapact and add live examples ([#64](https://github.com/mikejhill/codapact/issues/64)) ([98c889d](https://github.com/mikejhill/codapact/commit/98c889d6ad8e03cbe08f1b69ea05288f16fcd855))
16
+
17
+
18
+ ### Documentation
19
+
20
+ * **discovery:** remeasure on main after [#61](https://github.com/mikejhill/codapact/issues/61) and [#62](https://github.com/mikejhill/codapact/issues/62) ([#63](https://github.com/mikejhill/codapact/issues/63)) ([7e9cdbb](https://github.com/mikejhill/codapact/commit/7e9cdbb5e2b0fde3d39e27c66c21abb116ed2329))
21
+ * measure how efficiently agents discover a generated CLI ([#57](https://github.com/mikejhill/codapact/issues/57)) ([1be40c8](https://github.com/mikejhill/codapact/commit/1be40c8daae2a80c4b9a404c27865c1616571b1f))
22
+
23
+ ## [0.9.1](https://github.com/mikejhill/codapact/compare/v0.9.0...v0.9.1) (2026-11-15)
24
+
25
+
26
+ ### Bug Fixes
27
+
28
+ * **cli:** plain usage lines in describe; uvx-only coda skills; Agent Plugins manifest ([#54](https://github.com/mikejhill/codapact/issues/54)) ([3fea349](https://github.com/mikejhill/codapact/commit/3fea3493542eff6860cf2db049bd8cecce67d52b))
29
+
5
30
  ## [0.9.0](https://github.com/mikejhill/codapact/compare/v0.8.0...v0.9.0) (2026-11-15)
6
31
 
7
32
 
@@ -1,11 +1,12 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: codapact
3
- Version: 0.9.0
3
+ Version: 0.10.0
4
4
  Summary: Contracts into callable tools: generate libraries, then CLIs, from OpenAPI, GraphQL, and other API contracts.
5
5
  License-Expression: MIT
6
6
  License-File: LICENSE
7
7
  Requires-Dist: pyyaml>=6.0.3,<7
8
8
  Requires-Dist: graphql-core>=3.2.6,<4
9
+ Requires-Dist: truststore>=0.10,<1
9
10
  Requires-Dist: keyring>=25,<26 ; extra == 'keyring'
10
11
  Requires-Python: >=3.13
11
12
  Project-URL: Homepage, https://github.com/mikejhill/codapact
@@ -86,7 +87,9 @@ pagination live in one tested place and improve without regenerating.
86
87
  and restish.
87
88
  - [Configuration](docs/configuration.md): `codapact.yaml`, shaping transforms, target
88
89
  options, and generated CLI settings.
89
- - [Agent skills](skills/README.md): `create-coda-skill` writes the thin skill that lets an
90
- agent use a coda's CLI. Install in Claude Code with
91
- `/plugin marketplace add mikejhill/codapact`, then `/plugin install codapact@codapact`.
90
+ - [Agent skills](extras/agent-plugin/README.md): `create-coda-skill` writes the thin skill
91
+ that lets an agent use a coda's CLI. The repository is a plugin marketplace for Claude Code, GitHub
92
+ Copilot, and Codex: `<agent> plugin marketplace add mikejhill/codapact`.
93
+ - [Discovery efficiency](docs/discovery-efficiency/README.md): how many tokens and calls an
94
+ agent spends learning a generated CLI, compared with `--help`, MCP, OpenAPI, and llms.txt.
92
95
  - [Contributing](CONTRIBUTING.md)
@@ -69,7 +69,9 @@ pagination live in one tested place and improve without regenerating.
69
69
  and restish.
70
70
  - [Configuration](docs/configuration.md): `codapact.yaml`, shaping transforms, target
71
71
  options, and generated CLI settings.
72
- - [Agent skills](skills/README.md): `create-coda-skill` writes the thin skill that lets an
73
- agent use a coda's CLI. Install in Claude Code with
74
- `/plugin marketplace add mikejhill/codapact`, then `/plugin install codapact@codapact`.
72
+ - [Agent skills](extras/agent-plugin/README.md): `create-coda-skill` writes the thin skill
73
+ that lets an agent use a coda's CLI. The repository is a plugin marketplace for Claude Code, GitHub
74
+ Copilot, and Codex: `<agent> plugin marketplace add mikejhill/codapact`.
75
+ - [Discovery efficiency](docs/discovery-efficiency/README.md): how many tokens and calls an
76
+ agent spends learning a generated CLI, compared with `--help`, MCP, OpenAPI, and llms.txt.
75
77
  - [Contributing](CONTRIBUTING.md)
@@ -87,7 +87,7 @@ Enforced by `tests/unit/test_architecture.py` (AST import scan):
87
87
  | HTTP sending | `Transport.send(HttpRequest)` | `RuntimeClient(transport=…)` |
88
88
  | Cross-cutting | `Middleware(request, context, next)` | `RuntimeClient(middleware=…)` |
89
89
  | Credentials | `AuthProvider.credential(scheme)` | `RuntimeClient(auth=…)` |
90
- | Bespoke auth | `Authenticator.authenticate(request, scheme)`, optional `refresh` | returned by an `AuthProvider`; CLI `auth_plugin` |
90
+ | Bespoke auth | `Authenticator.authenticate(request, scheme)`, optional `refresh` and `expired` | returned by an `AuthProvider`; CLI `auth_plugin` |
91
91
  | Protocol | `ProtocolAdapter.supports/call` | `RuntimeClient(adapters=…)` |
92
92
 
93
93
  ### Plugin protocol
@@ -110,15 +110,16 @@ that silently mishandles the feature.
110
110
  - `RuntimeClient(api, base_url, auth, transport, middleware, adapters, …)`:
111
111
  `call(operation, args)` encodes and validates arguments, picks the adapter for the
112
112
  operation's binding, and decodes the result. `paginate()` follows cursors or offsets.
113
- - `SafeHttpClient` (default transport): no redirects, bounded bodies, DNS pinned against
114
- rebinding, private and plaintext addresses only with explicit opt-in.
113
+ - `SafeHttpClient` (default transport): redirects returned, never followed; OS trust store
114
+ (`truststore`); bounded bodies; DNS pinned against rebinding; private and plaintext
115
+ addresses only with explicit opt-in.
115
116
  - `Retry` middleware: idempotent operations only, exponential backoff, honors
116
117
  `Retry-After`.
117
118
  - Auth: sources (`StaticAuth`, `EnvAuth`, `ChainAuth`, `CommandAuth`, `StoreAuth`) return a
118
119
  `Credential` or an `Authenticator`. `OAuth2` (in `runtime.oauth`) is an `Authenticator`
119
120
  over a grant (client credentials, password, device code, authorization code + PKCE) and a
120
- `TokenStore` (memory, file, keyring). Authenticators run after middleware; a 401 triggers
121
- one `refresh()` and resend. `Credential.__repr__` never renders secrets.
121
+ `TokenStore` (memory, file, keyring). Authenticators run after middleware; a 401 (or a
122
+ response an authenticator's `expired()` flags) triggers one `refresh()` and resend. `Credential.__repr__` never renders secrets.
122
123
  - `runtime.selftest`: the suite generated bundles ship (offline dry runs, live scenarios).
123
124
  - Errors: `ValidationError`, `AuthError`, `ApiError` (status, named error case, decoded
124
125
  payload), `GraphQLResponseError`, `TransportError`.
@@ -62,7 +62,23 @@ Targets that share an `output` directory share options. When `python-cli` shares
62
62
  `source.max_bytes` (next to `type` and `path`, not under `options`) caps each contract
63
63
  document; the default is 64 MB. Remote contracts and generated clients honour
64
64
  `HTTPS_PROXY`, `HTTP_PROXY`, and `NO_PROXY`. Proxy URLs must use `http://` without
65
- credentials; HTTPS traffic is tunnelled with `CONNECT`.
65
+ credentials; HTTPS traffic is tunnelled with `CONNECT`. Certificates are verified against
66
+ the operating system's trust store (Windows certificate store, macOS keychain, the system
67
+ bundle on Linux), so TLS-inspecting corporate proxies whose root certificate is installed
68
+ there work without extra settings; `SSL_CERT_FILE` names a PEM bundle to use instead.
69
+ Redirects are never followed: a generated client reports a 3xx as an error that names
70
+ its `Location`.
71
+
72
+ YAML contracts are read with YAML 1.2 rules, as OpenAPI requires: `on`, `off`, `yes`,
73
+ `no`, `=`, `12:30` and dates stay strings, exactly as in the equivalent JSON.
74
+
75
+ Request bodies are sent as JSON when the contract offers it, else as
76
+ `multipart/form-data` or `application/x-www-form-urlencoded` (object fields become form
77
+ fields; binary fields become file parts; a bare binary body, or a list of files, is sent as
78
+ parts named `file`),
79
+ else as text or raw bytes. `*/*` sends raw bytes for a binary or schema-less body and JSON
80
+ otherwise. Binary values are `bytes`, a `str`, or a path (`os.PathLike`) in libraries, and
81
+ `TEXT`, `@FILE` or `@-` (stdin) on the command line; a file is sent under its own name.
66
82
 
67
83
  ### Pinned contracts and `codapact sync`
68
84
 
@@ -95,18 +111,22 @@ types. Without documents, it generates one operation per root field. A `#` comme
95
111
  directly above a document operation becomes its description; variable defaults
96
112
  (`$first: Int = 5`) become parameter defaults.
97
113
 
98
- When the first shaping step is `select`, model errors in operations it removes are reported
99
- as one warning instead of blocking generation.
114
+ When the first shaping step is `select`, it also filters before translation: operations it
115
+ excludes are never read, so problems in them (unsupported media types, odd enums, invalid
116
+ parameters) cannot block generation, and problems in schemas only those operations use are
117
+ reported as one warning. Model errors in operations it removes are likewise one warning.
118
+ This is the way to generate part of a very large contract in strict mode.
100
119
 
101
120
  ## Shaping transforms
102
121
 
103
122
  | Transform | Config |
104
123
  | --- | --- |
105
- | `select` | `include`, `exclude`: lists of globs or `{name, tag, kind, namespace, read_only}` selectors |
124
+ | `select` | `include`, `exclude`: lists of globs or `{name, path, tag, kind, namespace, read_only}` selectors. Names match in any spelling (`getIssue`, `get-issue` and `get_issue` are the same); a glob starting with `/` or a method (`GET /rest/api/3/issue/*`) matches the HTTP path |
106
125
  | `rename` | `operations`, `types`, `fields` (`Type.field`), `params` (`op.param`), `patterns` (`[{match: regex, replace: template}]` over operation names; explicit `operations` entries win); wire names are kept |
107
126
  | `group` | `by: tag` or `by: path`, and/or `operations: {glob: dotted.namespace}` |
108
127
  | `paginate` | `operations`, `style: cursor` (`param`, `next`), `offset` (`param`, `limit_param`), or `page` (`param`, default `page`; `start`, default `1`), and `items` (`$` when the result itself is the list). Operations matched only by a pattern that lack the param or items are skipped with a warning; operations named exactly are errors |
109
128
  | `defaults` | `params: {"op_glob.param": value}`; the param becomes optional and the runtime sends `value` when it is omitted |
129
+ | `headers` | `{SELECTOR: {Header-Name: value}}` (selectors as in `select`, e.g. `addAttachment` or `POST /rest/*`). Each header becomes an optional header param defaulting to the value, so it is always sent unless overridden (`--x-atlassian-token`); a header the contract declares just gets the default. For vendor headers contracts mention only in prose, such as Jira's `X-Atlassian-Token: no-check` |
110
130
  | `auth` | `schemes: {name: SPEC}` ([auth specs](#auth-specs)), `operations` (default all), `optional` (send credentials when set, call anonymously otherwise). For contracts that omit or under-declare `securitySchemes`; re-declaring an identical scheme is allowed |
111
131
  | `prune` | none; removes unreachable types (also run by `select`) |
112
132
 
@@ -141,6 +161,7 @@ Precedence: flag, then environment, then settings file.
141
161
  | Response validation | `--response-validation strict\|warn\|off` | `<PREFIX>_RESPONSE_VALIDATION` | `response_validation` |
142
162
  | Response size limit | none | `<PREFIX>_MAX_RESPONSE_BYTES` | `max_response_bytes` (default 100 MB) |
143
163
  | Output format | `-o`/`--output json\|jsonl\|yaml\|table\|raw` | `<PREFIX>_OUTPUT` | `output` |
164
+ | JSON style | none | `<PREFIX>_JSON_STYLE=auto\|pretty\|compact` | `json_style` |
144
165
  | Error format | `--error-format text\|json` | `<PREFIX>_ERROR_FORMAT` | `error_format` |
145
166
  | Credentials | none | `<PREFIX>_<SCHEME>`; basic uses `_USERNAME`/`_PASSWORD`; full chain under [Authentication](#authentication) | `auth_plugin`; secrets are never read from the file |
146
167
  | Option defaults | the option itself | none | `defaults: {plan_id: "…"}` (option names, `_` or `-`) |
@@ -157,6 +178,8 @@ result is the item list, so `[].name`; a path that matches nothing fails with ex
157
178
  `--allow-private`, and `--response-validation`. Command help lists them in one line;
158
179
  `<program> --help` describes each.
159
180
  `table` renders a list of objects, unwrapping single-list envelopes such as `{"data": {"items": [...]}}`.
181
+ JSON is indented on a terminal and compact (no whitespace) when stdout is a pipe or file,
182
+ where an agent or script reads it; `<PREFIX>_JSON_STYLE=pretty` or `compact` fixes one style.
160
183
  `jsonl` prints one JSON value per line, unwrapping the same envelopes as `table`, and streams
161
184
  `--all` results page by page. Output to a pipe or file is UTF-8 on every platform.
162
185
 
@@ -178,8 +201,8 @@ exists. Set `cli.env_prefix` too if the environment variables should keep their
178
201
  | --- | --- |
179
202
  | Help | `-h`/`--help` on every level; bare `<program>` or `<program> GROUP` prints help (exit `0`). Command help has a usage line with positionals and required options, examples, and a global-options reminder. |
180
203
  | Version | `-V`/`--version`: `poke 0.1.0 (PokéAPI 2.10.0; codapact 0.4.0)`; the installed distribution's version when it differs from the generated `version` option (so a release tool can bump packaging alone) |
181
- | Discovery for agents | `<program> describe [COMMAND…]`: JSON with every command's arguments (type, required, positional, choices, repeated), `read_only`/`destructive`/`paginated`, `requires_one_of` option groups, environment variables, exit codes, `start` (the steps an agent follows: doctor, credentials, describe, dry run) and `auth` (per scheme: environment variables, `auth login` commands, `docs` on how to get the credential, and whether every command or only some need it). `describe --brief` keeps those plus names, summaries, and usage lines, so one call is enough to start; agent skills for a coda point here instead of repeating it (see [Agent skills for codas](../skills/README.md)). Naming one command (or passing `--schemas`) adds the JSON Schema of its output and of each JSON-valued argument |
182
- | Diagnosis | `<program> doctor [--offline]`: version, settings file, base URL and its source, credentials (set or missing, never printed), proxy, connectivity. Missing credentials fail only when every command needs them. Table on a terminal, JSON otherwise; exit `0` ready, `3` credentials missing or rejected, `4` API unreachable, `1` any other failed check |
204
+ | Discovery for agents | `<program> describe [COMMAND…]`: JSON with every command's arguments (type, required, positional, choices, repeated), `read_only`/`destructive`/`paginated`, `requires_one_of` option groups, environment variables, exit codes, `start` (the steps an agent follows: doctor, credentials, describe, dry run) and `auth` (per scheme: environment variables, `auth login` commands, `docs` on how to get the credential, and whether every command or only some need it). `describe --brief` keeps those plus names, summaries, and usage lines, so one call is enough to start; past 150 commands it lists groups (name, summary, command count) instead, and `describe GROUP --brief` lists a group. Agent skills for a coda point here instead of repeating it (see [Agent skills for codas](../extras/agent-plugin/README.md)). Naming one command adds the JSON Schema of each JSON-valued argument; `--schemas` adds that and the output JSON Schema to every command. Printed to a pipe, `describe` is compact JSON. `--help` names the members of each JSON argument, and examples include one option of each `requires_one_of` group (see [Discovery efficiency](discovery-efficiency/README.md)) |
205
+ | Diagnosis | `<program> doctor [--offline]`: version, settings file, base URL and its source, credentials (set or missing, never printed), proxy, connectivity (any HTTP answer counts, a redirect included), and, for schemes with `cli.auth.<scheme>.check`, whether the server still accepts the credential (`session: <scheme>`, see [CLI credential chain](#cli-credential-chain)). Missing credentials fail only when every command needs them. Table on a terminal, JSON otherwise; exit `0` ready, `3` credentials missing or rejected, `4` API unreachable, `1` any other failed check |
183
206
  | Completion | `<program> completion bash\|zsh\|fish\|powershell` prints a script that asks the program for candidates |
184
207
  | Preview | `--dry-run` prints the HTTP request (method, URL, headers, body) without sending it; credentials appear as `$VARIABLE` placeholders (optional credentials that are not set stay off the request), and destructive commands skip confirmation |
185
208
  | Tracing | `-v`/`--verbose` logs each request line and response status, size, and time to stderr, masking credential values |
@@ -221,10 +244,14 @@ Three layers, each usable without the next:
221
244
  | --- | --- | --- |
222
245
  | Source | `AuthProvider.credential(scheme) -> Credential \| Authenticator \| None` | Where a secret comes from: environment, a command, the OS keyring, a vault |
223
246
  | Built-in flow | `OAuth2(grant, store)` with `ClientCredentials`, `PasswordGrant`, `DeviceCode` (RFC 8628), `AuthorizationCodePKCE` (loopback redirect); refresh tokens; OIDC discovery | Standard OAuth 2.0 / OpenID Connect without code |
224
- | Bespoke | `Authenticator.authenticate(request, scheme) -> request`, optional `refresh(scheme) -> bool` | Request signing (HMAC, AWS SigV4), token exchange, anything else |
247
+ | Bespoke | `Authenticator.authenticate(request, scheme) -> request`, optional `refresh(scheme) -> bool` and `expired(response, scheme) -> bool` | Request signing (HMAC, AWS SigV4), token exchange, browser sign-in sessions, anything else |
225
248
 
226
249
  An `Authenticator` runs last, after middleware, so a retried request is signed again. On
227
250
  HTTP 401 the runtime calls `refresh()` and, if it returns true, sends the request once more.
251
+ Servers that reject a session some other way (a redirect to a sign-in page, an HTML login
252
+ form, a vendor error code) are recognised by `expired()`, which sees every response: when it
253
+ returns true the runtime refreshes and resends the same way, and if the session is still
254
+ rejected the call fails with `AuthError` (CLI exit code 3).
228
255
 
229
256
  ### Auth specs
230
257
 
@@ -239,7 +266,7 @@ Used by the `auth` transform, the GraphQL `auth` option, and `cli.auth`:
239
266
  | `{openid: DISCOVERY_URL}` | OAuth 2.0 whose endpoints come from `.well-known/openid-configuration` |
240
267
  | `custom` or `{custom: DESCRIPTION}` | needs an `Authenticator` (plugin or library code) |
241
268
 
242
- Any spec map also takes `docs`. OpenAPI `oauth2` (including 3.2 `deviceAuthorization`) and
269
+ Every spec map also takes `docs` (for `custom` it replaces the description). OpenAPI `oauth2` (including 3.2 `deviceAuthorization`) and
243
270
  `openIdConnect` map to the same model; other schemes (`mutualTLS`, `http` with an unknown
244
271
  scheme) become `custom` with an info diagnostic.
245
272
 
@@ -261,7 +288,10 @@ The first source that has a credential wins, per scheme:
261
288
  `cli.auth: {SCHEME: {client_id, scopes, flow, audience}}` bakes public OAuth settings into
262
289
  the CLI (no secrets); `fields` declares [credential fields](#credential-fields); `docs` tells a
263
290
  person how to get the credential (shown in `describe` under `auth`, e.g. "Create a personal
264
- access token under Account Settings > Developer Settings").
291
+ access token under Account Settings > Developer Settings"); `check` names an operation that
292
+ needs no arguments (any spelling, e.g. `getCurrentUser`) which `doctor` calls to prove the
293
+ saved credential or session still works: a 401, 403, redirect or `expired()` fails the
294
+ `session: <scheme>` check with exit code 3.
265
295
  `auth login` picks a flow in this order: `--flow`, then client credentials when a secret is
266
296
  set, then device code, then authorization code with PKCE.
267
297
  `--with-token` reads a static token from stdin; `--username` saves basic credentials.
@@ -0,0 +1,240 @@
1
+ # How efficiently an agent learns a generated CLI
2
+
3
+ An agent that has never seen an API has to read something before it can make its first
4
+ correct call. This report measures how much it reads, and how many tool calls it spends,
5
+ for the discovery flow every codapact CLI ships and for the alternatives an agent would
6
+ otherwise use: `--help`, the full `describe`, MCP tool schemas, the raw OpenAPI document,
7
+ and llms.txt-style docs.
8
+
9
+ Everything here is regenerated by two scripts in
10
+ [`scripts/discovery_bench/`](../../scripts/discovery_bench/); see [Methodology](#methodology).
11
+
12
+ Measured 2026-11-17 on `main` at 1b80666, which includes every change in
13
+ [What changed](#what-changed) ([#61](https://github.com/mikejhill/codapact/pull/61), [#62](https://github.com/mikejhill/codapact/pull/62)), with codapact 0.9.1's behavior simulated from the
14
+ same output as the baseline.
15
+
16
+ ## Summary
17
+
18
+ | | |
19
+ | --- | --- |
20
+ | **The codapact flow** (`describe --brief`, then `describe COMMAND`) | **2.7k tokens, 2 calls** on YNAB, down from 6.8k in 0.9.1. Always enough for a correct first call, including JSON request bodies. 5× cheaper than full `describe`, 10× cheaper than the OpenAPI document, and under a fifth of the 15k that MCP puts in context on every turn. |
21
+ | **Large APIs** | On GitHub REST (1,279 commands) the flow now costs **5.2k tokens in 3 calls**, down from 87k: `describe --brief` lists groups once a CLI has more than 150 commands. |
22
+ | **Over a session** | Ten distinct YNAB commands cost **4.9k tokens** in context, down from 23.6k. 0.9.1 overtook loading every MCP tool at six commands; the flow now stays at about a third of it. |
23
+ | **Cheapest in tokens** | MCP with client-side tool search (0.6k) and the `--help` walk (1.5k). Tool search needs client support and keeps every tool name resident (14k on GitHub). `--help` costs 3 calls and has no JSON Schemas; it now names each JSON argument's fields. |
24
+ | **Worst** | Loading every MCP tool (15k on YNAB, 514k on GitHub, on every turn), the full OpenAPI document (26k, 2.9M), and llms-full.txt (94k, 5.1M). |
25
+
26
+ ## What changed
27
+
28
+ The first version of this report (0.9.1) found three fixes worth making, and a follow-up
29
+ compacted command output too. All are merged into
30
+ the runtime and generator now, so every coda gets them when it is regenerated.
31
+
32
+ | Finding in 0.9.1 | Change | Effect | PR |
33
+ | --- | --- | --- | --- |
34
+ | 86% of `describe COMMAND` was the output JSON Schema, which the agent needs only after the call | `describe COMMAND` returns arguments and JSON argument schemas; `--schemas` adds the output schema | YNAB median entry: 2.0k → 0.2k tokens | [#61](https://github.com/mikejhill/codapact/pull/61) |
35
+ | JSON was indented | `describe` prints compact JSON when stdout is not a terminal | about a third fewer tokens in every `describe` output | [#61](https://github.com/mikejhill/codapact/pull/61) |
36
+ | `describe --brief` listed every command (87k tokens on GitHub REST) | Past 150 commands it lists groups (name, summary, count); `start` tells the agent to run `describe GROUP --brief` | GitHub REST catalog: 86k → 2.3k tokens, plus one call for a group | [#61](https://github.com/mikejhill/codapact/pull/61) |
37
+ | Command output was indented JSON everywhere | Every JSON result prints compact when stdout is not a terminal (see [Responses](#responses)); `<PREFIX>_JSON_STYLE=pretty\|compact` overrides | 7% to 15% fewer tokens per response | [#62](https://github.com/mikejhill/codapact/pull/62) |
38
+ | `--help` gave JSON arguments no hint; the `transactions create` example omitted the required `--transaction`, so copying it failed | `--help` names each JSON argument's fields (required ones marked) and points at `describe`; examples include one option of each `requires_one_of` group | the cheapest flow says enough for most JSON bodies; examples run | [#61](https://github.com/mikejhill/codapact/pull/61) |
39
+
40
+ Not changed: there is still no MCP target. MCP only wins with client-side tool search,
41
+ which not every client has, and loses badly without it (see [Pros and cons](#pros-and-cons)).
42
+
43
+ ## Results
44
+
45
+ ### YNAB: what an agent reads before its first correct call
46
+
47
+ Six tasks, from "list payees" to "record a 12.34 purchase" (a JSON request body in
48
+ milliunits). Each technique is scored on the tokens it returns and the tool calls it takes
49
+ until the agent has every fact the task needs. Every reference call passes `--dry-run`.
50
+
51
+ ![YNAB tokens to the first correct call, by technique](figures/ynab-tokens-to-first-call.svg)
52
+
53
+ | Technique | Tokens (median) | Calls | Notes |
54
+ | --- | ---: | ---: | --- |
55
+ | MCP + tool search | 558 | 1 | plus 256 resident (every tool name), every turn |
56
+ | `--help` walk (root, group, command) | 1,530 | 3 | `transactions create`: the fields are listed but not their units (milliunits), so it falls back to `describe COMMAND` (4.3k, 4 calls) |
57
+ | `--help`, `describe GROUP --brief`, `describe COMMAND` | 1,689 | 3 | group-first; what large CLIs now do |
58
+ | **`describe --brief`, `describe COMMAND`** | **2,669** | **2** | the flow in the coda skill |
59
+ | OpenAPI, targeted reads | 2,781 | 6.5 | assumes perfect grep; the agent then writes the HTTP call itself |
60
+ | llms.txt, then one page | 4,791 | 2 | same: no execution layer |
61
+ | The same flow in 0.9.1 | 6,782 | 2 | indented, output schema included |
62
+ | Full `describe` | 13,865 | 1 | lacks JSON argument schemas: `transactions create` needs `describe COMMAND` too |
63
+ | MCP, every tool loaded | 15,034 | 0 | resident on every turn |
64
+ | Raw OpenAPI document | 26,273 | 1 | |
65
+ | llms-full.txt | 93,710 | 1 | |
66
+
67
+ Per-task numbers are in [`data/results.json`](data/results.json) under `apis.ynab.tasks`.
68
+
69
+ ### Scaling with API size
70
+
71
+ Median over up to 60 evenly spaced commands per API: Petstore (5 commands), YNAB (44),
72
+ PokéAPI (102), and the full GitHub REST description (1,279).
73
+
74
+ ![Tokens to first call vs. number of commands](figures/scaling-by-api-size.svg)
75
+
76
+ | API (commands) | brief + describe CMD (now) | Same in 0.9.1 | `--help` walk | MCP, every tool | MCP + tool search | Full describe | OpenAPI document |
77
+ | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
78
+ | Petstore (5) | 0.7k | 1.3k | 1.0k | 0.4k | 0.1k | 1.4k | 0.9k |
79
+ | YNAB (44) | 2.5k | 5.1k | 1.4k | 15.0k | 0.4k | 13.8k | 26.3k |
80
+ | PokéAPI (102) | 3.8k | 6.5k | 1.4k | 14.8k | 0.8k | 17.2k | 67.9k |
81
+ | GitHub REST (1,279) | 5.2k (3 calls) | 87.4k | 3.5k | 514k | 14.3k | 482k | 2.9M |
82
+
83
+ Anything that loads the whole API at once (full `describe`, every MCP tool, the OpenAPI
84
+ document) grows linearly and passes a 200k context window somewhere between 100 and 1,279
85
+ commands. The codapact flow and `--help` now grow only with the size of one group.
86
+
87
+ ### A whole session
88
+
89
+ An agent rarely makes one call. Each new command costs the codapact flow another
90
+ `describe COMMAND`, while MCP pays once up front and then resends the same schemas with
91
+ every turn. CLI rows include the coda skill (459 tokens).
92
+
93
+ ![Context used over a session of N distinct commands](figures/ynab-session-context.svg)
94
+
95
+ In 0.9.1, each `describe COMMAND` carried a 1.7k output schema, so the flow overtook
96
+ loading every MCP tool at six distinct commands. Now a command costs about 200 tokens, and
97
+ ten commands fit in 4.9k.
98
+
99
+ ## Pros and cons
100
+
101
+ | Technique | Pros | Cons |
102
+ | --- | --- | --- |
103
+ | **`describe --brief` + `describe COMMAND`** | Two calls (three on large CLIs), always sufficient. Carries `start`, `auth`, `exit_codes` and notes, so the agent learns readiness, credentials, and error handling in the same call. Typed JSON with argument schemas. Works in any agent with a shell. | One `describe COMMAND` per new command. |
104
+ | `--help` walk | Fewest tokens of the CLI flows. Every CLI user already knows it. Scales well, since each level only lists its children. Names JSON fields. | Three calls. Field names only, without types or units. No auth or exit-code summary unless the agent also reads root help. Text, not JSON. |
105
+ | Full `describe` | One call. | Grows linearly (14k on YNAB, 482k on GitHub). Omits JSON argument schemas unless `--schemas`, which makes it larger still. |
106
+ | MCP, every tool loaded | Zero discovery calls. Typed arguments that clients validate. Standard protocol. | All schemas sit in context on every turn, used or not (15k YNAB, 514k GitHub). Needs a running server and client configuration per agent. Clients cap tool counts. Credentials live in the server config. |
107
+ | MCP + tool search | Cheapest per call. Only the schemas used are loaded. | Depends on the client (not every MCP client defers tools). Every tool name stays resident (14k on GitHub). Still needs a server process. |
108
+ | Raw OpenAPI document | Authoritative; nothing to generate. | Too large past small APIs (2.9M tokens for GitHub). The agent must build URLs, headers, auth, pagination, and error handling itself. No dry run, no validation, no exit codes. |
109
+ | OpenAPI, targeted reads | Few tokens when the agent greps well. | 4 to 7 calls following `$ref` chains. The same execution burden as the raw document. Assumes the agent knows how to search YAML or JSON. |
110
+ | llms.txt, then one page | Two calls; readable Markdown. | The index grows linearly (53k on GitHub). Docs only: the agent still writes HTTP calls. Has to be generated and hosted. |
111
+ | llms-full.txt | One call. | The largest option on every API. |
112
+
113
+ What tokens do not show: the CLI and MCP flows also give the agent an **execution layer**.
114
+ A codapact CLI reads credentials from the keyring or environment, validates arguments
115
+ before sending, offers `--dry-run`, maps failures to exit codes with hints, follows
116
+ pagination, and trims responses with `--pick` and `--query`. With the OpenAPI or llms.txt
117
+ approaches the agent writes that layer itself, on every call, and its first attempts are
118
+ where errors happen.
119
+
120
+ Latency is not a factor: a warm `uvx codapact-ynab` call takes about 0.25 s.
121
+
122
+ ### Responses
123
+
124
+ After discovery, an agent reads what commands return. Real GitHub REST responses for this
125
+ repository, printed indented (as 0.9.1 did everywhere) and compact (as a pipe now gets):
126
+
127
+ | Response | Indented | Compact | Saved |
128
+ | --- | ---: | ---: | ---: |
129
+ | repository (object) | 2.4k | 2.0k | 15% |
130
+ | 30 pull requests (list) | 203k | 177k | 13% |
131
+ | 30 commits (list) | 96k | 90k | 7% |
132
+ | 10 releases (list) | 38k | 33k | 13% |
133
+
134
+ Responses save less than `describe` did (11% overall against about a third), because their
135
+ long string values outweigh the indentation. In absolute terms it is still the larger win:
136
+ one list of 30 pull requests saves 26k tokens, more than the whole discovery flow costs.
137
+ The bigger lever for responses remains `--pick` and `--query`, which drop fields the agent
138
+ does not need.
139
+
140
+ **When output is compact.** JSON is compact exactly when stdout is not a terminal. That is
141
+ the case for agents that capture command output (Claude Code's shell tool, for one, is not
142
+ a terminal), for scripts, for redirects to files, and for pipes into `jq`, which
143
+ pretty-prints anyway. A person at a terminal still sees indented JSON. `<PREFIX>_JSON_STYLE` (or
144
+ `json_style` in the settings file) fixes one style: `pretty` for files people read,
145
+ `compact` for an agent that runs commands in a pseudo-terminal. YAML, table, JSON Lines and
146
+ raw output are unchanged; JSON Lines was already compact.
147
+
148
+ ## Open questions
149
+
150
+ - **Live agent runs.** The steps here are modeled. Running a few real agents through the
151
+ YNAB tasks would show how often they read more than needed or retry.
152
+
153
+ ## Methodology
154
+
155
+ ### What is measured
156
+
157
+ For each API, each command, and each technique, the benchmark records what the agent
158
+ must read, in order, before it can make a correct call to that command:
159
+
160
+ - **Tokens**: the sum of every text the technique returns along the way.
161
+ - **Calls**: tool calls the agent makes. MCP tool schemas arrive with the prompt and count
162
+ as 0 calls; they are reported separately as *resident* tokens because the client resends
163
+ them on every turn.
164
+ - **Sufficiency**: a correct call needs certain facts. For every command, these are its
165
+ required arguments and the first three required fields of any JSON argument. The YNAB
166
+ tasks add hand-picked facts (for example `milliunits` for money amounts and `current`
167
+ for the current month). A fact counts only if it appears in the part of the text about
168
+ the target command, so a large catalog does not pass by mentioning a field elsewhere.
169
+ Matching ignores case and `-`/`_` separators (`--pet-id` matches `petId`). When a CLI
170
+ technique lacks a fact, the agent falls back to `describe COMMAND`, and that call is
171
+ added. Other techniques have no fallback, and missing facts are recorded.
172
+
173
+ ### Techniques
174
+
175
+ | Key | What the agent reads |
176
+ | --- | --- |
177
+ | `brief` | `describe --brief`, then `describe GROUP --brief` if it listed groups, then `describe COMMAND` (the flow in the coda skill) |
178
+ | `brief_v091` | Baseline: the same flow as 0.9.1 printed it, rebuilt from today's output (indented JSON, every command in `--brief`, output schema in `describe COMMAND`) |
179
+ | `group_brief` | `--help`, `describe GROUP --brief`, `describe COMMAND` |
180
+ | `full` | `describe` (fallback: `describe COMMAND`) |
181
+ | `help` | `--help`, `GROUP --help`, `GROUP COMMAND --help` (fallback: `describe COMMAND`) |
182
+ | `mcp` | A `tools/list` result with one tool per command: `name`, `description`, `inputSchema` built from the command's `describe` entry (JSON arguments keep their full schema), and annotations. This is what an OpenAPI-to-MCP bridge publishes. |
183
+ | `mcp_search` | Every tool name resident, then one tool definition loaded by search |
184
+ | `openapi` | The contract file as committed |
185
+ | `openapi_slice` | A path-search hit, `servers` and `securitySchemes`, then the operation plus every component it references, one call per `$ref` level |
186
+ | `llms` | An llms.txt index (one line per operation), then that operation's Markdown page with its schemas inline |
187
+ | `llms_full` | Every page concatenated |
188
+
189
+ ### APIs
190
+
191
+ All generated by this checkout of codapact from contracts in the repository, except GitHub:
192
+
193
+ - Petstore and Library: `examples/`.
194
+ - Countries, YNAB, PokéAPI: `tests/fixtures/corpus/`. YNAB uses the settings of the
195
+ published coda (`mikejhill/codapact-ynab`), so the program is `codapact-ynab`.
196
+ - GitHub REST (full): `github/rest-api-description@5c36b6f6`
197
+ `descriptions/api.github.com/api.github.com.json` (14 MB), generated with `strict: false`.
198
+ Only with `--github`.
199
+
200
+ The two GraphQL examples have two commands each; they appear in the data but not in the
201
+ scaling figure.
202
+
203
+ ### Tokens
204
+
205
+ Counted with the Claude tokenizer that shipped in `anthropic==0.34.2` (`tokenizer.json`),
206
+ the only Claude tokenizer that runs offline. Current Claude models tokenize differently, so
207
+ treat absolute numbers as approximate. Ratios between techniques, which is what the
208
+ conclusions rest on, change much less, because every technique is measured with the same
209
+ tokenizer on similar text. To recount with a current model, send the same texts to the
210
+ token-counting API.
211
+
212
+ ### Limits
213
+
214
+ - Agents are modeled, not run. The benchmark assumes the agent picks the right command
215
+ (or group) from the list and reads nothing it does not need. Real agents sometimes read
216
+ more, or guess and retry; this report does not measure retries.
217
+ - Tool-call framing and the agent's own reasoning are not counted. Response bodies are
218
+ measured separately under [Responses](#responses).
219
+ - MCP tools are serialized as compact JSON. Clients wrap them in their own format, which
220
+ adds a little per tool.
221
+ - `openapi_slice` assumes perfect search; it is a lower bound.
222
+
223
+ ### Reproduce
224
+
225
+ From the repository root:
226
+
227
+ ```sh
228
+ uv sync
229
+ uv run --with anthropic==0.34.2 --with tokenizers \
230
+ python scripts/discovery_bench/collect.py --github # about 3 minutes; drop --github for 1
231
+ uv run python scripts/discovery_bench/plot.py
232
+ ```
233
+
234
+ `collect.py` writes [`data/results.json`](data/results.json): tool versions, every API's
235
+ sizes, per-command results for each technique, the YNAB tasks with their reference calls
236
+ and `--dry-run` exit codes, warm `uvx codapact-ynab` latency (skip with
237
+ `--no-latency`), and the response sizes above, fetched from the public GitHub API (skip
238
+ with `--no-responses`). `plot.py` redraws the three SVGs in [`figures/`](figures/) from that file
239
+ alone. To add a task, append to `YNAB_TASKS` in `collect.py`; to add an API, add it to
240
+ `apis()`.