tokenjam 0.3.3__tar.gz → 0.3.5__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (274) hide show
  1. {tokenjam-0.3.3 → tokenjam-0.3.5}/CHANGELOG.md +5 -0
  2. {tokenjam-0.3.3 → tokenjam-0.3.5}/CLAUDE.md +23 -12
  3. {tokenjam-0.3.3 → tokenjam-0.3.5}/PKG-INFO +8 -4
  4. {tokenjam-0.3.3 → tokenjam-0.3.5}/README.md +6 -2
  5. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/claude-code-integration.md +1 -1
  6. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/installation.md +43 -2
  7. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/optimize/trim.md +1 -1
  8. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/python-sdk.md +1 -1
  9. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/openclaw/README.md +1 -1
  10. {tokenjam-0.3.3 → tokenjam-0.3.5}/incidents/hallucination-drift/README.md +3 -3
  11. {tokenjam-0.3.3 → tokenjam-0.3.5}/incidents/retry-loop/README.md +3 -3
  12. {tokenjam-0.3.3 → tokenjam-0.3.5}/incidents/surprise-cost/README.md +3 -3
  13. {tokenjam-0.3.3 → tokenjam-0.3.5}/pyproject.toml +10 -2
  14. {tokenjam-0.3.3 → tokenjam-0.3.5}/sdk-ts/package.json +1 -1
  15. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/factories.py +2 -0
  16. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/integration/test_db.py +3 -3
  17. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/integration/test_full_pipeline.py +35 -1
  18. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/manual-new-release-tests.md +74 -4
  19. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/manual-pre-release-testing.md +63 -0
  20. tokenjam-0.3.5/tests/synthetic/test_cost_tracking.py +261 -0
  21. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_cmd_tokenmaxx.py +13 -9
  22. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_cost.py +69 -5
  23. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_ingest_helicone.py +68 -0
  24. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_ingest_langfuse.py +67 -0
  25. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_ingest_otlp.py +22 -0
  26. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_logs_converter.py +13 -7
  27. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_spans_stats_repair.py +2 -1
  28. tokenjam-0.3.5/tests/unit/test_ui_offline.py +127 -0
  29. tokenjam-0.3.5/tokenjam/__init__.py +6 -0
  30. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/app.py +17 -0
  31. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/routes/logs.py +9 -1
  32. tokenjam-0.3.5/tokenjam/api/routes/version.py +20 -0
  33. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_report.py +2 -1
  34. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_tokenmaxx.py +37 -7
  35. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/main.py +5 -1
  36. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/cost.py +30 -9
  37. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/db.py +18 -4
  38. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/ingest_adapters/helicone.py +8 -0
  39. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/ingest_adapters/langfuse.py +9 -0
  40. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/models.py +4 -0
  41. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/mcp/server.py +14 -1
  42. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/otel/otlp_parsing.py +1 -0
  43. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/otel/provider.py +4 -0
  44. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/pricing/models.toml +36 -0
  45. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/ui/index.html +37 -8
  46. tokenjam-0.3.5/tokenjam/ui/vendor/htm.js +1 -0
  47. tokenjam-0.3.5/tokenjam/ui/vendor/preact-hooks.js +2 -0
  48. tokenjam-0.3.5/tokenjam/ui/vendor/preact.js +2 -0
  49. tokenjam-0.3.3/tests/synthetic/test_cost_tracking.py +0 -140
  50. tokenjam-0.3.3/tokenjam/__init__.py +0 -1
  51. {tokenjam-0.3.3 → tokenjam-0.3.5}/.github/CODEOWNERS +0 -0
  52. {tokenjam-0.3.3 → tokenjam-0.3.5}/.github/ISSUE_TEMPLATE/bug_report.md +0 -0
  53. {tokenjam-0.3.3 → tokenjam-0.3.5}/.github/ISSUE_TEMPLATE/feature_request.md +0 -0
  54. {tokenjam-0.3.3 → tokenjam-0.3.5}/.github/ISSUE_TEMPLATE/integration_request.md +0 -0
  55. {tokenjam-0.3.3 → tokenjam-0.3.5}/.github/pull_request_template.md +0 -0
  56. {tokenjam-0.3.3 → tokenjam-0.3.5}/.github/workflows/ci.yml +0 -0
  57. {tokenjam-0.3.3 → tokenjam-0.3.5}/.github/workflows/publish-npm.yml +0 -0
  58. {tokenjam-0.3.3 → tokenjam-0.3.5}/.github/workflows/publish-pypi.yml +0 -0
  59. {tokenjam-0.3.3 → tokenjam-0.3.5}/.gitignore +0 -0
  60. {tokenjam-0.3.3 → tokenjam-0.3.5}/AGENTS.md +0 -0
  61. {tokenjam-0.3.3 → tokenjam-0.3.5}/CONTRIBUTING.md +0 -0
  62. {tokenjam-0.3.3 → tokenjam-0.3.5}/LICENSE +0 -0
  63. {tokenjam-0.3.3 → tokenjam-0.3.5}/Makefile +0 -0
  64. {tokenjam-0.3.3 → tokenjam-0.3.5}/SECURITY.md +0 -0
  65. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/alerts.md +0 -0
  66. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/architecture.md +0 -0
  67. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/backfill/helicone.md +0 -0
  68. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/backfill/langfuse.md +0 -0
  69. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/backfill/otlp.md +0 -0
  70. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/backfill/overview.md +0 -0
  71. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/cli-reference.md +0 -0
  72. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/configuration.md +0 -0
  73. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/export.md +0 -0
  74. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/framework-support.md +0 -0
  75. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/internal/specs/.gitkeep +0 -0
  76. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/nemoclaw-integration.md +0 -0
  77. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/openclaw.md +0 -0
  78. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/optimize/cache.md +0 -0
  79. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/optimize/downsize.md +0 -0
  80. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/optimize/script.md +0 -0
  81. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/policy/overview.md +0 -0
  82. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/screenshots/tj-alerts.png +0 -0
  83. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/screenshots/tj-budget.png +0 -0
  84. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/screenshots/tj-cost.png +0 -0
  85. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/screenshots/tj-status.png +0 -0
  86. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/screenshots/tj-traces.png +0 -0
  87. {tokenjam-0.3.3 → tokenjam-0.3.5}/docs/typescript-sdk.md +0 -0
  88. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/README.md +0 -0
  89. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/alerts_and_drift/_shared.py +0 -0
  90. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/alerts_and_drift/budget_breach_demo.py +0 -0
  91. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/alerts_and_drift/drift_demo.py +0 -0
  92. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/alerts_and_drift/sensitive_actions_demo.py +0 -0
  93. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/multi/rag_pipeline.py +0 -0
  94. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/multi/research_team.py +0 -0
  95. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/multi/router_agent.py +0 -0
  96. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/multi/sample_docs/agent_patterns.txt +0 -0
  97. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/multi/sample_docs/cost_management.txt +0 -0
  98. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/multi/sample_docs/observability.txt +0 -0
  99. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/multi/sample_docs/safety.txt +0 -0
  100. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/single_framework/autogen_agent.py +0 -0
  101. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/single_framework/crewai_agent.py +0 -0
  102. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/single_framework/langchain_agent.py +0 -0
  103. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/single_framework/langgraph_agent.py +0 -0
  104. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/single_framework/llamaindex_agent.py +0 -0
  105. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/single_provider/anthropic_agent.py +0 -0
  106. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/single_provider/bedrock_agent.py +0 -0
  107. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/single_provider/gemini_agent.py +0 -0
  108. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/single_provider/litellm_agent.py +0 -0
  109. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/single_provider/openai_agent.py +0 -0
  110. {tokenjam-0.3.3 → tokenjam-0.3.5}/examples/single_provider/openai_agents_sdk_agent.py +0 -0
  111. {tokenjam-0.3.3 → tokenjam-0.3.5}/incidents/hallucination-drift/BLOG.md +0 -0
  112. {tokenjam-0.3.3 → tokenjam-0.3.5}/incidents/hallucination-drift/scenario.py +0 -0
  113. {tokenjam-0.3.3 → tokenjam-0.3.5}/incidents/retry-loop/BLOG.md +0 -0
  114. {tokenjam-0.3.3 → tokenjam-0.3.5}/incidents/retry-loop/scenario.py +0 -0
  115. {tokenjam-0.3.3 → tokenjam-0.3.5}/incidents/surprise-cost/BLOG.md +0 -0
  116. {tokenjam-0.3.3 → tokenjam-0.3.5}/incidents/surprise-cost/scenario.py +0 -0
  117. {tokenjam-0.3.3 → tokenjam-0.3.5}/sdk-ts/README.md +0 -0
  118. {tokenjam-0.3.3 → tokenjam-0.3.5}/sdk-ts/package-lock.json +0 -0
  119. {tokenjam-0.3.3 → tokenjam-0.3.5}/sdk-ts/src/client.test.ts +0 -0
  120. {tokenjam-0.3.3 → tokenjam-0.3.5}/sdk-ts/src/client.ts +0 -0
  121. {tokenjam-0.3.3 → tokenjam-0.3.5}/sdk-ts/src/index.ts +0 -0
  122. {tokenjam-0.3.3 → tokenjam-0.3.5}/sdk-ts/src/semconv.test.ts +0 -0
  123. {tokenjam-0.3.3 → tokenjam-0.3.5}/sdk-ts/src/semconv.ts +0 -0
  124. {tokenjam-0.3.3 → tokenjam-0.3.5}/sdk-ts/src/span-builder.test.ts +0 -0
  125. {tokenjam-0.3.3 → tokenjam-0.3.5}/sdk-ts/src/span-builder.ts +0 -0
  126. {tokenjam-0.3.3 → tokenjam-0.3.5}/sdk-ts/src/types.ts +0 -0
  127. {tokenjam-0.3.3 → tokenjam-0.3.5}/sdk-ts/tsconfig.json +0 -0
  128. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/__init__.py +0 -0
  129. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/agents/__init__.py +0 -0
  130. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/agents/email_agent_budget_breach.py +0 -0
  131. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/agents/email_agent_drift.py +0 -0
  132. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/agents/email_agent_loop.py +0 -0
  133. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/agents/email_agent_normal.py +0 -0
  134. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/agents/mock_llm.py +0 -0
  135. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/agents/test_mock_scenarios.py +0 -0
  136. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/conftest.py +0 -0
  137. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/e2e/__init__.py +0 -0
  138. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/e2e/conftest.py +0 -0
  139. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/e2e/test_real_llm.py +0 -0
  140. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/fixtures/helicone_real_response.json +0 -0
  141. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/fixtures/langfuse_real_response.json +0 -0
  142. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/fixtures/otlp_sample.json +0 -0
  143. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/integration/__init__.py +0 -0
  144. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/integration/test_api.py +0 -0
  145. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/integration/test_cli.py +0 -0
  146. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/integration/test_demos.py +0 -0
  147. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/integration/test_logs_api.py +0 -0
  148. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/synthetic/__init__.py +0 -0
  149. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/synthetic/test_alert_rules.py +0 -0
  150. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/synthetic/test_drift_detection.py +0 -0
  151. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/synthetic/test_ingest.py +0 -0
  152. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/synthetic/test_schema_validation.py +0 -0
  153. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/toy_agent/toy_agent.py +0 -0
  154. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/__init__.py +0 -0
  155. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_alerts.py +0 -0
  156. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_backfill.py +0 -0
  157. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_cache_efficacy.py +0 -0
  158. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_cache_recommend.py +0 -0
  159. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_cmd_policy.py +0 -0
  160. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_cmd_stop.py +0 -0
  161. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_compare.py +0 -0
  162. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_config.py +0 -0
  163. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_config_secret_divergence.py +0 -0
  164. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_demo_env.py +0 -0
  165. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_demo_scenarios.py +0 -0
  166. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_drift.py +0 -0
  167. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_export_claude_code.py +0 -0
  168. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_formatting.py +0 -0
  169. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_litellm_client.py +0 -0
  170. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_litellm_integration.py +0 -0
  171. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_mcp_server.py +0 -0
  172. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_models.py +0 -0
  173. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_onboard_codex.py +0 -0
  174. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_onboard_daemon.py +0 -0
  175. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_openclaw_ingest.py +0 -0
  176. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_optimize.py +0 -0
  177. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_pricing_override.py +0 -0
  178. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_prompt_bloat.py +0 -0
  179. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_time_parse.py +0 -0
  180. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_transport_401.py +0 -0
  181. {tokenjam-0.3.3 → tokenjam-0.3.5}/tests/unit/test_workflow_restructure.py +0 -0
  182. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/__init__.py +0 -0
  183. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/deps.py +0 -0
  184. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/middleware.py +0 -0
  185. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/routes/__init__.py +0 -0
  186. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/routes/agents.py +0 -0
  187. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/routes/alerts.py +0 -0
  188. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/routes/budget.py +0 -0
  189. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/routes/cost.py +0 -0
  190. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/routes/cost_compare.py +0 -0
  191. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/routes/drift.py +0 -0
  192. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/routes/metrics.py +0 -0
  193. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/routes/optimize.py +0 -0
  194. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/routes/otlp.py +0 -0
  195. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/routes/spans.py +0 -0
  196. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/routes/status.py +0 -0
  197. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/routes/tools.py +0 -0
  198. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/api/routes/traces.py +0 -0
  199. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/__init__.py +0 -0
  200. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_alerts.py +0 -0
  201. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_backfill.py +0 -0
  202. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_budget.py +0 -0
  203. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_cost.py +0 -0
  204. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_demo.py +0 -0
  205. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_doctor.py +0 -0
  206. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_drift.py +0 -0
  207. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_export.py +0 -0
  208. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_mcp.py +0 -0
  209. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_onboard.py +0 -0
  210. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_optimize.py +0 -0
  211. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_policy.py +0 -0
  212. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_serve.py +0 -0
  213. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_status.py +0 -0
  214. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_stop.py +0 -0
  215. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_tools.py +0 -0
  216. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_traces.py +0 -0
  217. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/cli/cmd_uninstall.py +0 -0
  218. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/__init__.py +0 -0
  219. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/alerts.py +0 -0
  220. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/api_backend.py +0 -0
  221. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/backfill.py +0 -0
  222. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/config.py +0 -0
  223. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/drift.py +0 -0
  224. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/export/__init__.py +0 -0
  225. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/export/claude_code.py +0 -0
  226. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/ingest.py +0 -0
  227. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/ingest_adapters/__init__.py +0 -0
  228. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/ingest_adapters/otlp.py +0 -0
  229. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/optimize/README.md +0 -0
  230. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/optimize/__init__.py +0 -0
  231. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/optimize/analyzers/__init__.py +0 -0
  232. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/optimize/analyzers/budget_projection.py +0 -0
  233. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/optimize/analyzers/cache_efficacy.py +0 -0
  234. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/optimize/analyzers/cache_recommend.py +0 -0
  235. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/optimize/analyzers/model_downgrade.py +0 -0
  236. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/optimize/analyzers/prompt_bloat.py +0 -0
  237. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/optimize/analyzers/workflow_restructure.py +0 -0
  238. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/optimize/registry.py +0 -0
  239. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/optimize/runner.py +0 -0
  240. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/optimize/types.py +0 -0
  241. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/pricing.py +0 -0
  242. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/retention.py +0 -0
  243. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/core/schema_validator.py +0 -0
  244. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/demo/__init__.py +0 -0
  245. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/demo/env.py +0 -0
  246. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/mcp/__init__.py +0 -0
  247. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/otel/__init__.py +0 -0
  248. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/otel/exporters.py +0 -0
  249. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/otel/semconv.py +0 -0
  250. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/py.typed +0 -0
  251. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/__init__.py +0 -0
  252. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/agent.py +0 -0
  253. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/bootstrap.py +0 -0
  254. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/client.py +0 -0
  255. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/http_exporter.py +0 -0
  256. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/integrations/__init__.py +0 -0
  257. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/integrations/anthropic.py +0 -0
  258. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/integrations/autogen.py +0 -0
  259. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/integrations/base.py +0 -0
  260. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/integrations/bedrock.py +0 -0
  261. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/integrations/crewai.py +0 -0
  262. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/integrations/gemini.py +0 -0
  263. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/integrations/langchain.py +0 -0
  264. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/integrations/langgraph.py +0 -0
  265. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/integrations/litellm.py +0 -0
  266. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/integrations/llamaindex.py +0 -0
  267. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/integrations/nemoclaw.py +0 -0
  268. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/integrations/openai.py +0 -0
  269. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/integrations/openai_agents_sdk.py +0 -0
  270. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/sdk/transport.py +0 -0
  271. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/utils/__init__.py +0 -0
  272. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/utils/formatting.py +0 -0
  273. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/utils/ids.py +0 -0
  274. {tokenjam-0.3.3 → tokenjam-0.3.5}/tokenjam/utils/time_parse.py +0 -0
@@ -54,6 +54,11 @@ This release pivots TokenJam toward cost-optimization for autonomous agents. Fou
54
54
  - **`tj status` surfaces unknown plan tiers.** When sessions exist with `plan_tier = 'unknown'`, prints a one-line note pointing the user at `tj onboard --reconfigure`. Exit code unchanged.
55
55
  - **`tj optimize` plan-tier-aware rendering.** When every session in the window has `plan_tier = 'unknown'`, dollar figures are suppressed and a header note explains why. Mixed / partial-unknown windows render normally with an advisory note.
56
56
  - **MCP `get_optimize_report` tool.** Now accepts `findings: list[str]` (was `only: str`). Docstring surfaces for both API-billing and subscription-plan-efficiency phrasings.
57
+ - **`tj tokenmaxx` tier ladder expanded to six tiers and renamed.** Two highest tiers renamed from `TokenChad` / `TokenGigaChad` to `TokenSuperMaxxer` / `TokenGigaMaxxer`, and a new `TokenMegaMaxxer` tier slots between them covering the 20× – 50× multiplier range. The previous top tier started at 20×; the new top tier starts at 50×, so the absolute headline for very-heavy users is more meaningful. Fire-emoji escalation matches the new tier count: 🔥 → 🔥🔥 → 🔥🔥🔥. The quip that previously belonged to `TokenGigaChad` ("Touch grass. Then run `tj optimize`.") now belongs to `TokenMegaMaxxer`; `TokenGigaMaxxer` gets its own escalated quip. JSON output's `tier` field carries the new label string verbatim; consumers reading the `tier` value must update accordingly.
58
+
59
+ ### Fixed
60
+ - **Cache-only spans were costed at $0.** A prompt-cache hit (0 new input/output tokens but non-zero cache-read tokens) bills the cache-read rate, but both `calculate_cost` and `CostEngine.process_span` short-circuited on input/output alone, dropping the span as a no-op and under-reporting spend. The early-return guards now fire only when *all* token counts are zero, so cache-read (and cache-write) costs are charged correctly.
61
+ - **Cache-write (cache-creation) tokens were dropped on the live ingest path.** The SDK integrations emit `gen_ai.usage.cache_creation_tokens`, the pricing table carries a `cache_write_per_mtok` rate, and `calculate_cost` already priced it — but the OTLP span parser and provider reader only read cache-read tokens, so cache-creation tokens never reached `CostEngine.process_span` and their (higher-rate) cost was never charged. `NormalizedSpan` now carries `cache_write_tokens`, both parsers populate it, and `process_span` charges it. Only the backfill path costed cache-write before; the live path now matches.
57
62
 
58
63
  ### Internal
59
64
  - **Registry-driven optimize analyzers.** `tokenjam/core/optimize.py` split into `tokenjam/core/optimize/` package with `registry.py`, `runner.py`, `types.py`, and `analyzers/` subpackage using `pkgutil` auto-discovery. New analyzers drop a file under `analyzers/` with a `@register("name")` decorator — nothing else needs editing. See `tokenjam/core/optimize/README.md`.
@@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
4
4
 
5
5
  ## Project Overview
6
6
 
7
- `tj` (TokenJam) is a local-first, OTel-native observability CLI for AI agents. No cloud backend, no signup. It captures telemetry from agent runtimes, stores it in a local DuckDB database, and exposes a CLI + local REST API for querying. Install via `pip install tokenjam`, run via `tj <subcommand>`. Requires Python >=3.10.
7
+ `tj` (TokenJam) is a local-first, OTel-native **cost-optimization layer** for AI agents (with a full observability stack underneath). No cloud backend, no signup. It captures telemetry from agent runtimes, stores it in a local DuckDB database, and runs four named analyzers (`downsize` / `cache` / `script` / `trim`) that surface cost-saving candidates from real usage — plus a CLI, local REST API, web UI, and MCP server for querying. Install via `pipx install tokenjam` (recommended — sidesteps PEP 668 on Homebrew Python and Debian 12+/Ubuntu 24+) or `pip install tokenjam` in a venv. Run via `tj <subcommand>`. Requires Python >=3.10.
8
8
 
9
9
  ## Build & Development
10
10
 
@@ -61,10 +61,17 @@ Post-ingest hooks run synchronously after each span is written to DB:
61
61
  - **`tokenjam/core/db.py`**: `StorageBackend` protocol + `DuckDBBackend` + `InMemoryBackend` (for tests) + migration runner. Migrations are `(version, sql)` tuples in a `MIGRATIONS` list — never modify existing ones, only append. **Note:** `StorageBackend` doesn't cover every query. Some callers (e.g. `CostEngine`, `cmd_status`) access `db.conn` directly for queries not in the protocol (cost updates, active session lookups). Helper `_row_to_session()` is used to convert raw DuckDB rows.
62
62
  - **`tokenjam/core/ingest.py`**: `IngestPipeline` (central hub), `SpanSanitizer` (rejects oversized/malformed spans), `strip_captured_content()`. Post-ingest hooks (cost, alerts, schema) are optional and error-tolerant — hook failures are logged, never propagated.
63
63
  - **`tokenjam/core/pricing.py`**: `ModelRates` (frozen dataclass), `load_pricing_table()` (LRU-cached), `get_rates(provider, model)`. Falls back to default rates for unknown models.
64
- - **`tokenjam/core/cost.py`**: `calculate_cost()` (pure function, rounds to 8dp) + `CostEngine` (post-ingest hook that updates `spans.cost_usd` and `sessions.total_cost_usd` via `db.conn` — see db.py note). Pricing loaded from `pricing/models.toml`.
64
+ - **`tokenjam/core/cost.py`**: `calculate_cost()` (pure function, rounds to 8dp) + `CostEngine` (post-ingest hook that updates `spans.cost_usd` and `sessions.total_cost_usd` via `db.conn` — see db.py note). Pricing loaded from `tokenjam/pricing/models.toml`. **Cache-read vs cache-write are separate fields** on `NormalizedSpan` (`cache_tokens` = read, `cache_write_tokens` = create); they bill at different rates and `calculate_cost` charges each at its own rate. The early-return no-op guard checks all four token counts (input/output/cache_read/cache_write) — see PR #90 and PR #92 for the cache-only-span and cache-write-on-live-path fixes.
65
65
  - **`tokenjam/core/alerts.py`**: `AlertEngine` with 13 alert types, `CooldownTracker` (in-memory, per agent+type, resets on restart), `AlertDispatcher` routing to 6 channel types (stdout, file, ntfy, webhook, Discord, Telegram). `AlertEngine.fire()` is the external entry point for other modules (SchemaValidator, DriftDetector) to fire alerts. Suppressed alerts are still persisted to DB but not dispatched to channels. Hardcoded thresholds: retry loop fires at 4+ identical tool calls in last 6 spans; failure rate fires at >20% errors in last 20 spans (checked every 5th error); session duration default 3600s. Stdout and file channels always include full detail regardless of `include_captured_content` config.
66
66
  - **`tokenjam/core/drift.py`**: `DriftDetector` — Z-score based behavioral drift detection, fires at session end.
67
- - **`tokenjam/core/optimize/`**: Package powering `tj optimize` and the `get_optimize_report` MCP tool. Public API re-exported from `__init__.py`: `build_report()` (orchestrator), `report_to_dict()`, `ANALYZER_REGISTRY`, `ANALYZER_ORDER`, plus result dataclasses. Architecture: `registry.py` holds the `@register("name")` decorator and `ANALYZER_REGISTRY` dict; `runner.py` defines `ANALYZER_ORDER` and orchestrates execution; `types.py` holds `AnalyzerContext` + result dataclasses + `MODEL_DOWNGRADE_CAVEAT`. Individual analyzers live in `analyzers/`, each as a single file registering via `@register`: `model_downgrade.py` (structural candidates — input < 5K tokens AND output < 500 tokens AND tool_calls ≤ 5; never claims quality equivalence, caveat baked into dataclass default), `budget_projection.py` (per-provider cycle spend vs `[budget.<provider>]` ceiling; only fires when budget > 0), `cache_efficacy.py`, `cache_recommend.py`, `prompt_bloat.py`, `workflow_restructure.py`. Analyzers receive an `AnalyzerContext` and operate on `db.conn` directly. To add a new analyzer: drop a file under `analyzers/`, decorate with `@register("name")`, append to `ANALYZER_ORDER` if ordering matters — `cmd_optimize --finding` choices auto-derive from the registry.
67
+ - **`tokenjam/core/optimize/`**: Package powering `tj optimize` and the `get_optimize_report` MCP tool. Public API re-exported from `__init__.py`: `build_report()` (orchestrator), `report_to_dict()`, `ANALYZER_REGISTRY`, `ANALYZER_ORDER`, plus result dataclasses. Architecture: `registry.py` holds the `@register("name")` decorator and `ANALYZER_REGISTRY` dict; `runner.py` defines `ANALYZER_ORDER` and orchestrates execution; `types.py` holds `AnalyzerContext` + result dataclasses + `MODEL_DOWNGRADE_CAVEAT`. Individual analyzers live in `analyzers/`, each as a single file registering via `@register`. **Registry strings (the user-facing names) and file names are decoupled**:
68
+ - `model_downgrade.py` → `@register("downsize")` — structural candidates (input < 5K tokens AND output < 500 tokens AND tool_calls ≤ 5; never claims quality equivalence, caveat baked into dataclass default)
69
+ - `budget_projection.py` → `@register("budget-projection")` — per-provider cycle spend vs `[budget.<provider>]` ceiling; only fires when budget > 0
70
+ - `cache_efficacy.py` → `@register("cache")` — current cache-read efficacy per (provider, model)
71
+ - `cache_recommend.py` → `@register("cache-recommend")` — Anthropic-only structural prefix detection for `cache_control` placement
72
+ - `workflow_restructure.py` → `@register("script")` — `(tool_name, arg_shape)` cluster detection for deterministic-script candidates
73
+ - `prompt_bloat.py` → `@register("trim")` — LLMLingua-2 token-significance classification (requires `tokenjam[bloat]` extra)
74
+ Analyzers receive an `AnalyzerContext` and operate on `db.conn` directly. To add a new analyzer: drop a file under `analyzers/`, decorate with `@register("name")`, append to `ANALYZER_ORDER` if ordering matters — `cmd_optimize`'s positional `findings` Click choices auto-derive from the registry.
68
75
  - **`tokenjam/core/ingest_adapters/`**: Third-party trace-export adapters that normalize external payloads (`langfuse.py`, `helicone.py`, `otlp.py`) into `NormalizedSpan` for ingest. Each is reachable as a `tj backfill <name>` subcommand and accepts `--source-url` (live API) or `--source-file` (offline JSON dump). Adapters write deterministic span IDs derived from the source's identifiers so re-runs are idempotent. `otlp.py` shares span-mapping logic with the live `POST /api/v1/spans` route via `tokenjam/otel/otlp_parsing.py`.
69
76
  - **`tokenjam/core/export/`**: Routing-config snippet generators for `tj optimize --export-config`. Currently `claude_code.py` emits a JSONC fragment under a `tokenjam.routing_recommendations` namespace with honest-framing caveat comments baked in. Writes to `~/.config/tokenjam/exports/`; never touches `~/.claude/settings.json` or other external configs (no `--apply` flag — Claude Code doesn't currently honor TokenJam routing keys, so auto-writing would change nothing and erode trust).
70
77
  - **`tokenjam/core/backfill.py`**: Parses Claude Code on-disk session JSONL files into `NormalizedSpan`s. Cost is recomputed from `pricing/models.toml` because the on-disk format has no `cost_usd`. The parser tolerates the dated `claude-<family>-<ver>-YYYYMMDD` model-name suffixes Anthropic ships (handled by `core/pricing.py.get_rates()`, which strips the trailing 8-digit date suffix when no exact pricing match exists). Idempotency relies on deterministic span IDs derived from `(session_id, message uuid)` / `(session_id, tool_use id)`.
@@ -92,11 +99,12 @@ Post-ingest hooks run synchronously after each span is written to DB:
92
99
 
93
100
  - **`tj demo [scenario]`** (`cmd_demo.py`) — runs Agent Incident Library scenarios (zero-config, no API keys). `tj demo` lists all; `tj demo retry-loop` runs one.
94
101
  - **`tj doctor`** (`cmd_doctor.py`) — health checks (config, DB, secrets, webhooks, drift readiness, schema-vs-capture consistency). Exit 0 = ok, 1 = warnings, 2 = errors.
95
- - **`tj optimize`** (`cmd_optimize.py`) — six analyzers, registry-driven: `model-downgrade`, `budget-projection`, `cache-efficacy`, `cache-recommend`, `workflow-restructure`, `prompt-bloat`. Flags: `--since 30d`, `--finding <name>` (repeatable; choices auto-derive from `ANALYZER_REGISTRY` at click decoration time), `--budget <provider>`, `--budget-usd <amount>`, `--compare <period>` (window-cost diff vs prior period; accepts `previous` / `last-week` / `last-month` / `last-7d` / `last-30d` / `YYYY-MM-DD:YYYY-MM-DD`), `--export-config <target>` (writes a routing snippet — currently `claude-code` — under `~/.config/tokenjam/exports/`; no `--apply` flag by design). Plan-tier-aware rendering: subscription users see "implied API value" framing and token-share savings (never dollar "spend"); local users see token-only framing; unknown-plan users see dollar figures suppressed with a `tj onboard --reconfigure` hint. Opens the live DB read-only so it works alongside a running `tj serve`.
102
+ - **`tj optimize`** (`cmd_optimize.py`) — six analyzers, registry-driven. **Analyzers are positional args** (not `--finding <name>`): `tj optimize downsize cache trim` runs three; bare `tj optimize` runs all. Registered names: `downsize`, `cache`, `cache-recommend`, `script`, `trim`, `budget-projection`. Flags: `--since 30d`, `--budget <provider>`, `--budget-usd <amount>`, `--compare <period>` (window-cost diff vs prior period; accepts `previous` / `last-week` / `last-month` / `last-7d` / `last-30d` / `YYYY-MM-DD:YYYY-MM-DD`), `--export-config <target>` (writes a routing snippet — currently `claude-code` — under `~/.config/tokenjam/exports/`; no `--apply` flag by design). Plan-tier-aware rendering: subscription users see "implied API value" framing and token-share savings (never dollar "spend"); local users see token-only framing; unknown-plan users see dollar figures suppressed with a `tj onboard --reconfigure` hint. Works alongside a running `tj serve` via the `/api/v1/optimize` HTTP fallback when the DuckDB write lock is held by the daemon.
103
+ - **`tj tokenmaxx`** (`cmd_tokenmaxx.py`) — shareable spend-tier command. Reads last 30 days of usage, classifies into a 6-tier ladder (Sipper / Moderator / Maxxer / SuperMaxxer / MegaMaxxer / GigaMaxxer) using the multiplier vs the user's declared subscription plan as the primary classifier, with absolute USD/mo thresholds as the API-user fallback. Output is a bordered Panel designed for screenshotting. Plan-aware: shows the multiplier line only when the user has `[budget.<provider>] plan = "max_5x"` (or pro / max_20x / plus) configured. The companion landing page is `tokenjam.dev/tokenmaxxing`. Designed to never exit without an actionable next step — pairs the tier callout with the downsize savings figure inline.
96
104
  - **`tj cost`** (`cmd_cost.py`) — cost breakdown by `--group-by agent|model|day|tool`. Same `--compare <period>` flag as `tj optimize` for window-over-window diffs (▲/▼ indicators, per-agent and per-model top-shifts, dollar + token deltas).
97
105
  - **`tj backfill <source>`** (`cmd_backfill.py`) — ingest historical telemetry from external sources. Subcommands: `claude-code` (parses `~/.claude/projects/*.jsonl`, auto-invoked at the end of `tj onboard --claude-code`), `langfuse` (live API or JSON dump), `helicone` (live API or JSON dump), `otlp` (raw OTLP JSON via URL or file — reuses the same parser as the live `POST /api/v1/spans` route). All idempotent via deterministic span IDs.
98
106
  - **`tj onboard`** (`cmd_onboard.py`) — `--claude-code` and `--codex` flags trigger integration-specific flows. Prompts for plan tier (api / pro / max_5x / max_20x for Anthropic; api / plus / team / enterprise for OpenAI) and writes it to `[budget.<provider>] plan = "..."`. Supports `--reconfigure` to re-prompt against an existing config, and `--plan <tier>` for non-interactive use. Does NOT auto-write a default `usd = 200` cycle ceiling — subscription users get only the `plan` field; API users are explicitly asked whether they want a self-imposed ceiling.
99
- - **`tj report`** (`cmd_report.py`) — generates standalone HTML visualizations of analyzer findings (e.g. `tj report --bloat [<agent_id>]` renders the prompt-bloat analyzer's per-token significance). Writes to `~/.cache/tokenjam/reports/` (override via `TOKENJAM_REPORT_DIR`) and opens in the default browser.
107
+ - **`tj report`** (`cmd_report.py`) — generates standalone HTML visualizations of analyzer findings. Currently `tj report --trim [<agent_id>]` renders the Trim analyzer's per-token significance (was `--bloat` pre-0.3.1, renamed alongside the analyzer's registry string). Writes to `~/.cache/tokenjam/reports/` (override via `TOKENJAM_REPORT_DIR`) and opens in the default browser.
100
108
  - **`tj policy list`** (`cmd_policy.py`) — read-only preview of the unified policy surface. Consolidates existing `[alerts]`, `[alerts.channels]`, `[defaults.budget]`, `[budget.<provider>]`, per-agent `budget`/`drift`/`sensitive_actions`/`output_schema`, and `[capture]` config into one table; each row carries its source TOML section. Supports `--json`. `tj policy add | edit | apply | remove | test` are intentionally absent this sprint — the unified config migration is next sprint's work. `policy` is in `no_db_commands` in `cli/main.py` so it doesn't open the DB. Rich source-section strings (`[budget.anthropic]`, `[[alerts.channels]]`) must be passed through `rich.markup.escape()` before rendering — otherwise Rich consumes them as style tags.
101
109
 
102
110
  All commands support `--json` for machine-readable output. Commands that query alerts use exit code 1 if active (unacknowledged, unsuppressed) alerts exist.
@@ -139,11 +147,13 @@ When a span has a `conversation_id` matching an existing session, it's attribute
139
147
  10. **Use semconv constants** — reference `GenAIAttributes` and `TjAttributes` from `tokenjam/otel/semconv.py` instead of hardcoding OTel attribute name strings.
140
148
  11. **OTel TracerProvider is global and set-once** — `trace.set_tracer_provider()` only works once per process. In tests, set the provider once at module level (not per-test in a fixture) and clear spans between tests. Use a custom `_CollectingExporter(SpanExporter)` since `InMemorySpanExporter` is not available in the installed OTel version. See `tests/agents/test_mock_scenarios.py` for the SDK test pattern and `tests/integration/test_full_pipeline.py` for the pipeline pattern.
141
149
  12. **New SDK integrations must call `ensure_initialised()`** — every `patch_*()` convenience function must call `from tokenjam.sdk.bootstrap import ensure_initialised; ensure_initialised()` before installing hooks. This lazily bootstraps the TracerProvider + IngestPipeline on first use.
142
- 13. **PyPI package name is `tokenjam`, not `ocw`** — `pip install tokenjam` is the correct install command. The CLI command is `tj` and the Python package directory is `tokenjam/`. The published package name on PyPI is `tokenjam`. Never write `pip install ocw` in docs, examples, or comments.
143
- 14. **`tj optimize` output must never claim quality equivalence** — the model-downgrade finding flags structural candidates only. Every user-visible string says "looks like" / "candidate" / "review before switching" — never "safe to downgrade" or "would have worked." The `MODEL_DOWNGRADE_CAVEAT` constant lives on `DowngradeFinding` as a dataclass default so it can't be removed by accident; it must also appear in human-readable CLI output. The same honesty discipline applies to all other analyzers — `cache-efficacy` ("you're getting X% of available caching"), `cache-recommend` (Anthropic-only, structural prefix detection), `workflow-restructure` ("structural shape matches", "review before replacing with a script"), `prompt-bloat` ("predicted low-significance regions; review before editing"). `tj optimize --export-config` snippets bake the caveat block into the JSONC output as comments.
150
+ 13. **PyPI package name is `tokenjam`, not `ocw`** — the package on PyPI is `tokenjam`. The CLI command is `tj`. The Python package directory is `tokenjam/`. **Recommended install: `pipx install tokenjam`** (sidesteps PEP 668 on Homebrew Python and Debian 12+/Ubuntu 24+). `pip install tokenjam` works inside a clean venv but fails on system Python with a misleading externally-managed-environment error. Never write `pip install ocw` in docs, examples, or comments.
151
+ 14. **`tj optimize` output must never claim quality equivalence** — the `downsize` finding flags structural candidates only. Every user-visible string says "looks like" / "candidate" / "review before switching" — never "safe to downgrade" or "would have worked." The `MODEL_DOWNGRADE_CAVEAT` constant lives on `DowngradeFinding` as a dataclass default so it can't be removed by accident; it must also appear in human-readable CLI output. The same honesty discipline applies to all other analyzers — `cache` ("you're getting X% of available caching"), `cache-recommend` (Anthropic-only, structural prefix detection), `script` ("structural shape matches", "review before replacing with a script"), `trim` ("predicted low-significance regions; review before editing"). `tj optimize --export-config` snippets bake the caveat block into the JSONC output as comments.
144
152
  15. **Version bump on release** — both `pyproject.toml` (`version = "X.Y.Z"`) and `sdk-ts/package.json` (`"version": "X.Y.Z"`) must be bumped to the new version before creating a GitHub release. The publish workflows (`publish-pypi.yml`, `publish-npm.yml`) trigger on `release published` events and will fail with 403 if the version already exists on PyPI/npm.
145
- 16. **New optimize analyzers self-register** — drop a `.py` file under `tokenjam/core/optimize/analyzers/` with a function decorated `@register("name")` taking `AnalyzerContext`. Auto-discovery in `analyzers/__init__.py` walks the directory at import time. `cmd_optimize.py`'s `--finding` choices read from `ANALYZER_REGISTRY.keys()` at click decoration — no edits needed there. If your analyzer depends on (or is depended on by) another, append it to `ANALYZER_ORDER` in `runner.py` at the right position. Wave-2 analyzers attach their findings to `OptimizeReport.findings[name]` (generic dict); the older `model-downgrade` / `budget-projection` analyzers retain typed slots on `OptimizeReport` for backwards compat with `cmd_optimize` and the MCP server.
153
+ 16. **New optimize analyzers self-register** — drop a `.py` file under `tokenjam/core/optimize/analyzers/` with a function decorated `@register("name")` taking `AnalyzerContext`. Auto-discovery in `analyzers/__init__.py` walks the directory at import time. `cmd_optimize.py`'s positional `findings` Click choices read from `ANALYZER_REGISTRY.keys()` at decoration — no edits needed there. If your analyzer depends on (or is depended on by) another, append it to `ANALYZER_ORDER` in `runner.py` at the right position. Wave-2 analyzers attach their findings to `OptimizeReport.findings[name]` (generic dict); the older `downsize` (registered name; file is `model_downgrade.py`) and `budget-projection` analyzers retain typed slots on `OptimizeReport` for backwards compat with `cmd_optimize` and the MCP server.
146
154
  17. **OTLP parsing has one home** — `tokenjam/otel/otlp_parsing.py`. Both the live `POST /api/v1/spans` route and the `tj backfill otlp` adapter import `parse_otlp_span` and `extract_resource_attrs` from there. If you need to extend OTLP attribute extraction, do it once in that module; do not copy-paste into either caller.
155
+ 18. **Web UI must work fully offline** — `tokenjam/ui/index.html` is the served dashboard. It is intentionally a single-file SPA with **zero external HTTP loads at render time**. Preact + hooks + htm are vendored under `tokenjam/ui/vendor/` and wired via an `<script type="importmap">`; fonts use system-font fallbacks (no Google Fonts); the favicon is inlined as a `data:` URL. The FastAPI app mounts `/ui/vendor` as `StaticFiles`. The `tests/unit/test_ui_offline.py` regression test asserts no render-time external URLs exist anywhere outside `<a href>` (clickable links to github.com are fine — they only fetch on click). If you add a CDN font, script, or stylesheet, that test will fail. Vendor the asset locally instead. See issue #87 + PR #88.
156
+ 19. **Analyzer registry names ≠ file names** — registry strings (`downsize`, `cache`, `script`, `trim`) are decoupled from Python module filenames (`model_downgrade.py`, `cache_efficacy.py`, `workflow_restructure.py`, `prompt_bloat.py`). The 0.3.1 rename only changed `@register("...")` strings; file names stayed for git-blame continuity. When grepping for an analyzer, search both the registry string AND the older file-name keyword.
147
157
 
148
158
  ## Config
149
159
 
@@ -233,10 +243,11 @@ Key runtime dependency: `pytz` is required by DuckDB for `TIMESTAMPTZ` column ha
233
243
  - **[docs/installation.md](docs/installation.md)** — base install vs optional extras matrix. Documents `tokenjam[bloat]` (the ~2GB torch + transformers extra used by the Trim analyzer), framework adapter extras (`[langchain]` / `[crewai]` / `[autogen]` / `[litellm]`), and the MCP / dev extras.
234
244
  - **[docs/configuration.md](docs/configuration.md)** — full TOML config surface plus the "Content capture and privacy" section explaining the four `[capture]` toggles and how they interact with `alerts.include_captured_content`.
235
245
  - **Optimize product pages** — one per user-facing product, all under `docs/optimize/`:
236
- - [`downsize.md`](docs/optimize/downsize.md) — model-downgrade candidate flagging (internal: `model-downgrade`)
237
- - [`cache.md`](docs/optimize/cache.md) — `cache-efficacy` (current caching ratio) + `cache-recommend` (Anthropic-only breakpoint suggestions)
238
- - [`script.md`](docs/optimize/script.md) — `workflow-restructure` clustering by `(tool_name, arg_shape)` signature
239
- - [`trim.md`](docs/optimize/trim.md) — LLMLingua-2 token-significance classifier (`prompt-bloat`), install + capture requirements, performance numbers
246
+ - [`downsize.md`](docs/optimize/downsize.md) — cheaper-model candidate flagging (registry: `downsize`, file: `model_downgrade.py`)
247
+ - [`cache.md`](docs/optimize/cache.md) — `cache` (current caching ratio) + `cache-recommend` (Anthropic-only breakpoint suggestions)
248
+ - [`script.md`](docs/optimize/script.md) — `script` clustering by `(tool_name, arg_shape)` signature (file: `workflow_restructure.py`)
249
+ - [`trim.md`](docs/optimize/trim.md) — LLMLingua-2 token-significance classifier (`trim`, file: `prompt_bloat.py`), install + capture requirements, performance numbers
250
+ - **[AGENTS.md](AGENTS.md)** — codebase conventions for contributors (referenced from the top-level README).
240
251
  - **Backfill adapters** — `docs/backfill/overview.md` lists the four sources (`claude-code` / `langfuse` / `helicone` / `otlp`) with the partnership-posture framing; per-adapter pages document modes (URL / file), field mapping, idempotency, and v1 limitations.
241
252
  - **[docs/policy/overview.md](docs/policy/overview.md)** — read-only preview of the unified policy surface (`tj policy list`). Notes that the `add` / `edit` / `apply` subcommands and the underlying `[policy]` config migration land next sprint.
242
253
  - **Internal specs** — `docs/internal/specs/` is reserved for canonical specs that production code references at long-term. Currently empty (sprint specs have been cleaned up after merge); add new ones here when a feature needs a stable, code-referenced source of truth.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: tokenjam
3
- Version: 0.3.3
3
+ Version: 0.3.5
4
4
  Summary: TokenJam — local-first OTel-native observability for Autonomous AI agents
5
5
  Project-URL: Homepage, https://opencla.watch
6
6
  Project-URL: Repository, https://github.com/Metabuilder-Labs/openclawwatch
@@ -23,6 +23,7 @@ Requires-Dist: apscheduler>=3.10
23
23
  Requires-Dist: click>=8.1
24
24
  Requires-Dist: duckdb>=0.10
25
25
  Requires-Dist: fastapi>=0.110
26
+ Requires-Dist: fastmcp>=0.2
26
27
  Requires-Dist: genson>=1.2
27
28
  Requires-Dist: httpx>=0.27
28
29
  Requires-Dist: jsonschema>=4.0
@@ -53,7 +54,6 @@ Requires-Dist: langchain>=0.2; extra == 'langchain'
53
54
  Provides-Extra: litellm
54
55
  Requires-Dist: litellm>=1.40; extra == 'litellm'
55
56
  Provides-Extra: mcp
56
- Requires-Dist: fastmcp; extra == 'mcp'
57
57
  Description-Content-Type: text/markdown
58
58
 
59
59
  <div align="center">
@@ -74,9 +74,11 @@ TokenJam reads your agent's telemetry and tells you when to downsize, when to tr
74
74
  [![OTel](https://img.shields.io/badge/OTel-GenAI%20SemConv-3d8eff?labelColor=0d1117)](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
75
75
 
76
76
  ```
77
- pip install tokenjam
77
+ pipx install tokenjam
78
78
  ```
79
79
 
80
+ <sub>Don't have pipx? `brew install pipx` on macOS, `apt install pipx` on Debian/Ubuntu, or see [docs/installation.md](docs/installation.md). `pip install tokenjam` also works in a clean venv.</sub>
81
+
80
82
  **No cloud · No signup · No vendor lock-in**
81
83
 
82
84
  </div>
@@ -147,11 +149,13 @@ Run all four with `tj optimize`. Run several with `tj optimize downsize cache tr
147
149
  For **Claude Code** users — zero code, auto-backfills your last 30 days:
148
150
 
149
151
  ```bash
150
- pip install "tokenjam[mcp]"
152
+ pipx install 'tokenjam[mcp]'
151
153
  tj onboard --claude-code
152
154
  tj optimize # cost-saving candidates from your actual usage
153
155
  ```
154
156
 
157
+ To upgrade later: `pipx upgrade tokenjam` (then `tj stop && tj serve &` to reload the daemon, and `tj --version` to verify). See [docs/installation.md](docs/installation.md#upgrading).
158
+
155
159
  For any Python agent:
156
160
 
157
161
  ```python
@@ -16,9 +16,11 @@ TokenJam reads your agent's telemetry and tells you when to downsize, when to tr
16
16
  [![OTel](https://img.shields.io/badge/OTel-GenAI%20SemConv-3d8eff?labelColor=0d1117)](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
17
17
 
18
18
  ```
19
- pip install tokenjam
19
+ pipx install tokenjam
20
20
  ```
21
21
 
22
+ <sub>Don't have pipx? `brew install pipx` on macOS, `apt install pipx` on Debian/Ubuntu, or see [docs/installation.md](docs/installation.md). `pip install tokenjam` also works in a clean venv.</sub>
23
+
22
24
  **No cloud · No signup · No vendor lock-in**
23
25
 
24
26
  </div>
@@ -89,11 +91,13 @@ Run all four with `tj optimize`. Run several with `tj optimize downsize cache tr
89
91
  For **Claude Code** users — zero code, auto-backfills your last 30 days:
90
92
 
91
93
  ```bash
92
- pip install "tokenjam[mcp]"
94
+ pipx install 'tokenjam[mcp]'
93
95
  tj onboard --claude-code
94
96
  tj optimize # cost-saving candidates from your actual usage
95
97
  ```
96
98
 
99
+ To upgrade later: `pipx upgrade tokenjam` (then `tj stop && tj serve &` to reload the daemon, and `tj --version` to verify). See [docs/installation.md](docs/installation.md#upgrading).
100
+
97
101
  For any Python agent:
98
102
 
99
103
  ```python
@@ -3,7 +3,7 @@
3
3
  Monitor every Claude Code session — costs, tool calls, API requests, errors — with two commands:
4
4
 
5
5
  ```bash
6
- pip install "tokenjam[mcp]"
6
+ pipx install 'tokenjam[mcp]'
7
7
  tj onboard --claude-code
8
8
  # Restart Claude Code, then:
9
9
  tj status --agent claude-code-<project>
@@ -5,10 +5,36 @@ TokenJam ships as a Python package on PyPI and a TypeScript SDK on npm. Pick the
5
5
  ## Base install
6
6
 
7
7
  ```bash
8
+ pipx install tokenjam
9
+ ```
10
+
11
+ This is the recommended install path on **all platforms**. `pipx` automatically creates an isolated venv for the `tj` CLI, which means:
12
+
13
+ - It works on macOS with Homebrew Python (which refuses `pip install` into its managed environment by default — [PEP 668](https://peps.python.org/pep-0668/)).
14
+ - It works on Debian 12+ / Ubuntu 24+ (same PEP 668 enforcement).
15
+ - It doesn't pollute your system Python or any project's venv.
16
+
17
+ Don't have `pipx`? Install it with one of:
18
+
19
+ | Platform | Command |
20
+ |---|---|
21
+ | macOS | `brew install pipx` |
22
+ | Debian / Ubuntu | `apt install pipx` |
23
+ | Windows | `py -m pip install --user pipx` |
24
+ | Anywhere else | `python3 -m pip install --user pipx` |
25
+
26
+ Then ensure pipx's bin dir is on your `PATH` with `pipx ensurepath`.
27
+
28
+ ### Alternative: pip in a venv
29
+
30
+ If you prefer plain pip (or need to install into an existing project venv):
31
+
32
+ ```bash
33
+ python3 -m venv .venv && source .venv/bin/activate
8
34
  pip install tokenjam
9
35
  ```
10
36
 
11
- This is enough for the CLI (`tj`), local REST API (`tj serve`), the four out-of-box optimize analyzers that don't need ML models, and every native SDK integration except LLMLingua-based Trim. Requires Python ≥ 3.10.
37
+ Either path is enough for the CLI (`tj`), local REST API (`tj serve`), the four out-of-box optimize analyzers that don't need ML models, and every native SDK integration except LLMLingua-based Trim. Requires Python ≥ 3.10.
12
38
 
13
39
  After install, run:
14
40
 
@@ -37,7 +63,7 @@ TokenJam keeps heavyweight ML dependencies, framework adapters, and the MCP serv
37
63
  Combine multiple extras:
38
64
 
39
65
  ```bash
40
- pip install "tokenjam[mcp,bloat]"
66
+ pipx install 'tokenjam[mcp,bloat]'
41
67
  ```
42
68
 
43
69
  ### Bloat extra details
@@ -48,6 +74,21 @@ If you run `tj optimize trim` without the extra installed, the analyzer self-reg
48
74
 
49
75
  See [`docs/optimize/trim.md`](optimize/trim.md) for performance numbers, capture requirements, and what the analyzer actually reports.
50
76
 
77
+ ## Upgrading
78
+
79
+ ```bash
80
+ pipx upgrade tokenjam # if you installed via pipx (recommended)
81
+ pip install --upgrade tokenjam # if you're in a pip + venv setup
82
+ ```
83
+
84
+ After upgrading:
85
+
86
+ 1. Restart the daemon to pick up the new code: `tj stop && tj serve &`
87
+ 2. DB migrations apply automatically on the next `tj` invocation — no manual step required
88
+ 3. Verify with `tj --version`
89
+
90
+ PyPI's CDN occasionally lags ~1–2 min after a release. If `pipx upgrade` reports "already at the latest version" but the reported `tj --version` is older than what's on the [releases page](https://github.com/Metabuilder-Labs/tokenjam/releases), wait a minute and retry.
91
+
51
92
  ## TypeScript SDK
52
93
 
53
94
  ```bash
@@ -22,7 +22,7 @@ LLMLingua-2 pulls in PyTorch and transformers (~2GB). Kept out of the
22
22
  base install:
23
23
 
24
24
  ```bash
25
- pip install "tokenjam[bloat]"
25
+ pipx install 'tokenjam[bloat]'
26
26
  ```
27
27
 
28
28
  The base `pip install tokenjam` does NOT pull torch. Trim shows up in
@@ -5,7 +5,7 @@ For any Python agent — Anthropic, OpenAI, Gemini, Bedrock, LangChain, CrewAI,
5
5
  ## Install
6
6
 
7
7
  ```bash
8
- pip install tokenjam
8
+ pipx install tokenjam
9
9
  tj onboard # creates config, generates ingest secret
10
10
  tj doctor # verify your setup
11
11
  ```
@@ -5,7 +5,7 @@ This is a config-only integration — no Python code needed. OpenClaw's built-in
5
5
  ## Step 1: Start TokenJam
6
6
 
7
7
  ```bash
8
- pip install tokenjam
8
+ pipx install tokenjam
9
9
  tj onboard
10
10
  tj serve &
11
11
  ```
@@ -1,6 +1,6 @@
1
1
  # My agent worked yesterday. Today it's possessed.
2
2
 
3
- **Run it:** `pip install tokenjam && tj demo hallucination-drift`
3
+ **Run it:** `pipx install tokenjam && tj demo hallucination-drift`
4
4
 
5
5
  ---
6
6
 
@@ -60,7 +60,7 @@ The demo uses `baseline_sessions = 5` for speed. In production, 10–50 sessions
60
60
  ## Try it yourself
61
61
 
62
62
  ```bash
63
- pip install tokenjam
63
+ pipx install tokenjam
64
64
  tj demo hallucination-drift
65
65
  ```
66
66
 
@@ -75,4 +75,4 @@ To track drift on your real agent, wire up the TokenJam SDK, enable drift in `tj
75
75
 
76
76
  ---
77
77
 
78
- [TokenJam](https://github.com/Metabuilder-Labs/TokenJam) is a local-first, zero-signup observability CLI for AI agents. No cloud. No account. Just `pip install tokenjam` and start seeing what your agent actually does.
78
+ [TokenJam](https://github.com/Metabuilder-Labs/TokenJam) is a local-first, zero-signup observability CLI for AI agents. No cloud. No account. Just `pipx install tokenjam` and start seeing what your agent actually does.
@@ -1,6 +1,6 @@
1
1
  # Your agent isn't flaky. You're blind.
2
2
 
3
- **Run it:** `pip install tokenjam && tj demo retry-loop`
3
+ **Run it:** `pipx install tokenjam && tj demo retry-loop`
4
4
 
5
5
  ---
6
6
 
@@ -57,7 +57,7 @@ The loop was visible from span #4. Your logs didn't surface it until a user comp
57
57
  ## Try it yourself
58
58
 
59
59
  ```bash
60
- pip install tokenjam
60
+ pipx install tokenjam
61
61
  tj demo retry-loop
62
62
  ```
63
63
 
@@ -72,4 +72,4 @@ To catch this in your real agent, wire up the TokenJam SDK (`@watch()` + `patch_
72
72
 
73
73
  ---
74
74
 
75
- [TokenJam](https://github.com/Metabuilder-Labs/TokenJam) is a local-first, zero-signup observability CLI for AI agents. No cloud. No account. Just `pip install tokenjam` and start seeing what your agent actually does.
75
+ [TokenJam](https://github.com/Metabuilder-Labs/TokenJam) is a local-first, zero-signup observability CLI for AI agents. No cloud. No account. Just `pipx install tokenjam` and start seeing what your agent actually does.
@@ -1,6 +1,6 @@
1
1
  # Why did my agent just spend $47 on a hello world?
2
2
 
3
- **Run it:** `pip install tokenjam && tj demo surprise-cost`
3
+ **Run it:** `pipx install tokenjam && tj demo surprise-cost`
4
4
 
5
5
  ---
6
6
 
@@ -75,7 +75,7 @@ TokenJam fires `cost_budget_session` and `cost_budget_daily` alerts when limits
75
75
  ## Try it yourself
76
76
 
77
77
  ```bash
78
- pip install tokenjam
78
+ pipx install tokenjam
79
79
  tj demo surprise-cost
80
80
  ```
81
81
 
@@ -90,4 +90,4 @@ To track real spend, instrument your agent with the tokenjam SDK and run `tj ser
90
90
 
91
91
  ---
92
92
 
93
- [TokenJam](https://github.com/Metabuilder-Labs/TokenJam) is a local-first, zero-signup observability CLI for AI agents. No cloud. No account. Just `pip install tokenjam` and start seeing what your agent actually does.
93
+ [TokenJam](https://github.com/Metabuilder-Labs/TokenJam) is a local-first, zero-signup observability CLI for AI agents. No cloud. No account. Just `pipx install tokenjam` and start seeing what your agent actually does.
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
4
4
 
5
5
  [project]
6
6
  name = "tokenjam"
7
- version = "0.3.3"
7
+ version = "0.3.5"
8
8
  description = "TokenJam — local-first OTel-native observability for Autonomous AI agents"
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.10"
@@ -41,6 +41,11 @@ dependencies = [
41
41
  "httpx>=0.27",
42
42
  "apscheduler>=3.10",
43
43
  "websockets>=12.0",
44
+ # fastmcp ships in the base install (was in the [mcp] extra) so `tj mcp`
45
+ # works on a fresh `pipx install tokenjam` without requiring users to
46
+ # remember the extra. Claude Code's MCP integration is now a primary
47
+ # use case rather than an opt-in. Issue #101.
48
+ "fastmcp>=0.2",
44
49
  ]
45
50
 
46
51
  [project.urls]
@@ -54,7 +59,10 @@ crewai = ["crewai>=0.28"]
54
59
  autogen = ["pyautogen>=0.2"]
55
60
  litellm = ["litellm>=1.40"]
56
61
  dev = ["pytest", "pytest-asyncio", "httpx", "ruff", "mypy"]
57
- mcp = ["fastmcp"]
62
+ # Kept as a no-op extra for back-compat — `pipx install 'tokenjam[mcp]'` still
63
+ # works, just installs the same fastmcp that's now in the base dependencies.
64
+ # Documented in `docs/installation.md` so users know they no longer need it.
65
+ mcp = []
58
66
  # Trim analyzer (`tj optimize --finding prompt-bloat`). LLMLingua-2 pulls in
59
67
  # PyTorch and transformers, ~2GB total. Kept optional so the base install
60
68
  # stays small — most users don't run the bloat analyzer.
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@tokenjam/sdk",
3
- "version": "0.3.3",
3
+ "version": "0.3.5",
4
4
  "description": "TypeScript SDK for TokenJam — local-first observability for AI agents",
5
5
  "main": "dist/index.js",
6
6
  "types": "dist/index.d.ts",
@@ -19,6 +19,7 @@ def make_llm_span(
19
19
  input_tokens: int = 1000,
20
20
  output_tokens: int = 200,
21
21
  cache_tokens: int = 0,
22
+ cache_write_tokens: int = 0,
22
23
  cost_usd: float | None = None,
23
24
  tool_name: str | None = None,
24
25
  status: str = "ok",
@@ -59,6 +60,7 @@ def make_llm_span(
59
60
  input_tokens=input_tokens,
60
61
  output_tokens=output_tokens,
61
62
  cache_tokens=cache_tokens,
63
+ cache_write_tokens=cache_write_tokens,
62
64
  cost_usd=cost_usd,
63
65
  conversation_id=conversation_id,
64
66
  attributes=attrs,
@@ -42,8 +42,8 @@ def _insert_agent(db, agent_id="test-agent"):
42
42
  def test_migrations_run_on_empty_db():
43
43
  backend = InMemoryBackend()
44
44
  rows = backend.conn.execute("SELECT version FROM schema_migrations").fetchall()
45
- assert len(rows) == 4
46
- assert {r[0] for r in rows} == {1, 2, 3, 4}
45
+ assert len(rows) == 5
46
+ assert {r[0] for r in rows} == {1, 2, 3, 4, 5}
47
47
  backend.close()
48
48
 
49
49
 
@@ -52,7 +52,7 @@ def test_migrations_are_idempotent():
52
52
  # Running migrations again should not raise
53
53
  run_migrations(backend.conn)
54
54
  rows = backend.conn.execute("SELECT version FROM schema_migrations").fetchall()
55
- assert len(rows) == 4
55
+ assert len(rows) == 5
56
56
  backend.close()
57
57
 
58
58
 
@@ -34,7 +34,7 @@ from tokenjam.core.db import InMemoryBackend
34
34
  from tokenjam.core.ingest import IngestPipeline
35
35
  from tokenjam.core.models import AgentRecord, NormalizedSpan, SpanKind, SpanStatus
36
36
  from tokenjam.core.schema_validator import SchemaValidator
37
- from tokenjam.otel.provider import TjSpanExporter
37
+ from tokenjam.otel.provider import TjSpanExporter, convert_otel_span
38
38
  from tokenjam.otel.semconv import GenAIAttributes
39
39
  from tokenjam.sdk.agent import watch, AgentSession, record_llm_call, record_tool_call
40
40
  from tokenjam.utils.time_parse import utcnow
@@ -159,6 +159,40 @@ def full_stack():
159
159
  db.close()
160
160
 
161
161
 
162
+ # ── OTel ReadableSpan -> NormalizedSpan ──────────────────────────────────
163
+
164
+
165
+ def test_convert_otel_span_extracts_cache_read_and_write_tokens():
166
+ """convert_otel_span indexes both cache-read and cache-creation tokens.
167
+
168
+ Regression: provider previously read only CACHE_READ_TOKENS, dropping
169
+ cache-creation tokens so cache-write cost was never charged on this path.
170
+ """
171
+ collected: list[ReadableSpan] = []
172
+
173
+ class _Collector(SpanExporter):
174
+ def export(self, spans: Sequence[ReadableSpan]) -> SpanExportResult:
175
+ collected.extend(spans)
176
+ return SpanExportResult.SUCCESS
177
+
178
+ def shutdown(self) -> None:
179
+ pass
180
+
181
+ provider = TracerProvider()
182
+ provider.add_span_processor(SimpleSpanProcessor(_Collector()))
183
+ tracer = provider.get_tracer("test")
184
+
185
+ with tracer.start_as_current_span("gen_ai.llm.call") as span:
186
+ span.set_attribute(GenAIAttributes.REQUEST_MODEL, "claude-haiku-4-5")
187
+ span.set_attribute(GenAIAttributes.CACHE_READ_TOKENS, 1000)
188
+ span.set_attribute(GenAIAttributes.CACHE_CREATE_TOKENS, 2000)
189
+
190
+ assert len(collected) == 1
191
+ normalized = convert_otel_span(collected[0])
192
+ assert normalized.cache_tokens == 1000
193
+ assert normalized.cache_write_tokens == 2000
194
+
195
+
162
196
  # ── SDK -> Pipeline -> DB ─────────────────────────────────────────────────
163
197
 
164
198
 
@@ -13,8 +13,15 @@ Run this after a new release publishes to PyPI to verify it works end-to-end. Th
13
13
  tj uninstall --yes 2>/dev/null
14
14
  rm -rf ~/.tj ~/.config/tj .tj
15
15
 
16
- pip3 install --upgrade tokenjam
16
+ # Recommended install path (PEP 668-safe on Homebrew Python and
17
+ # Debian 12+/Ubuntu 24+). `--force` so we reinstall even if a prior
18
+ # version is present.
19
+ pipx install --force tokenjam
17
20
  tj --version
21
+
22
+ # Older `pip3 install --upgrade tokenjam` path still works inside
23
+ # a clean venv but fails on system Python — that's the bug pipx
24
+ # solves, and verifying pipx is what we ship docs telling users to do.
18
25
  ```
19
26
 
20
27
  **Pass criteria:** version matches the release being tested.
@@ -66,6 +73,32 @@ tj optimize --json | python3 -c \
66
73
 
67
74
  **Pass criteria:** every positional analyzer name runs without crashing. Optional analyzers (`cache-recommend`, `trim`) surface clear hints when their prereqs aren't met instead of erroring.
68
75
 
76
+ ## 4b. TokenMaxx tier classification
77
+
78
+ ```bash
79
+ tj tokenmaxx
80
+ # [ ] Bordered "TokenJam TokenMaxxing Report" panel renders
81
+ # [ ] On api plan: shows absolute spend; no multiplier line
82
+ # [ ] Action line surfaces either downsize savings or "no obvious
83
+ # savings flagged yet" (both are valid)
84
+
85
+ # Verify the JSON tier label is one of the six valid v0.3.4 tiers.
86
+ tj tokenmaxx --json | python3 -c \
87
+ "import json,sys;d=json.load(sys.stdin);ok={'TokenSipper','TokenModerator','TokenMaxxer','TokenSuperMaxxer','TokenMegaMaxxer','TokenGigaMaxxer'};assert d['tier'] in ok,d['tier'];print('ok:',d['tier'])"
88
+
89
+ # Reconfigure to a subscription plan and re-run — the multiplier line
90
+ # should appear. Pick whichever plan matches your test config.
91
+ tj onboard --claude-code --reconfigure --plan max_5x
92
+ tj tokenmaxx
93
+ # [ ] Multiplier line "That's N× your Max 5x plan cost ($100/mo flat)."
94
+ # [ ] Tier may shift if the multiplier crosses a boundary
95
+
96
+ # Flip back to api so subsequent steps render dollar figures.
97
+ tj onboard --claude-code --reconfigure --plan api
98
+ ```
99
+
100
+ **Pass criteria:** the report renders without crashing, the JSON `tier` field carries one of the 6 v0.3.4 tier names, and the multiplier line appears under a subscription plan.
101
+
69
102
  ## 5. Backfill adapters (smoke against committed fixtures)
70
103
 
71
104
  ```bash
@@ -123,10 +156,45 @@ Spot-check:
123
156
  - [ ] Cost page shows non-zero USD values
124
157
  - [ ] Sidebar theme toggle works
125
158
 
159
+ ### Offline-UI verification (v0.3.4 — PR #88)
160
+
161
+ Open Chrome DevTools (or your browser's equivalent) → **Network tab** → reload `http://127.0.0.1:7391/`.
162
+
163
+ - [ ] **Zero failed requests** to `fonts.googleapis.com`, `fonts.gstatic.com`, `esm.sh`, or `tokenjam.dev`
164
+ - [ ] Dashboard interactivity works (sidebar nav, tab switches) — proves the vendored Preact / htm under `/ui/vendor/` is being served, not loading from the CDN
165
+ - [ ] Favicon renders (data: URL, no external fetch)
166
+
167
+ Bonus: turn off wifi entirely, hard-refresh, and confirm the page still renders + the JS still hydrates. The whole dashboard must work air-gapped.
168
+
126
169
  ```bash
127
170
  tj stop
128
171
  ```
129
172
 
173
+ ## 9. Cache cost-correctness (v0.3.4 — PRs #90 + #92)
174
+
175
+ Cache-only spans (cache_read > 0, input/output = 0) used to be costed at $0. Cache-creation tokens on the live OTLP path used to be silently dropped. Both fixed in v0.3.4.
176
+
177
+ ```bash
178
+ # Spans table now has cache_write_tokens (migration 5).
179
+ duckdb ~/.tj/telemetry.duckdb "PRAGMA table_info(spans)" 2>/dev/null \
180
+ | grep cache_write_tokens \
181
+ && echo "ok: cache_write_tokens column present"
182
+
183
+ # Any captured Anthropic cache-hit span should have non-zero cost_usd.
184
+ duckdb ~/.tj/telemetry.duckdb "
185
+ SELECT COUNT(*) AS hits,
186
+ MIN(cost_usd) AS min_cost
187
+ FROM spans
188
+ WHERE cache_tokens > 0
189
+ AND (input_tokens = 0 OR input_tokens IS NULL)
190
+ AND (output_tokens = 0 OR output_tokens IS NULL)
191
+ " 2>/dev/null
192
+ # [ ] If hits > 0: min_cost > 0 (cache hits ARE being costed; was $0 pre-0.3.4)
193
+ # [ ] If hits = 0: this release's runs didn't trigger a pure cache-only span — fine, unit tests cover the path
194
+ ```
195
+
196
+ If you don't have `duckdb` CLI installed, skip the SQL checks — the unit + synthetic tests covering these paths run in CI and are the canonical verification.
197
+
130
198
  ---
131
199
 
132
200
  ## Claude Code integration (smoke)
@@ -166,14 +234,16 @@ tj stop
166
234
 
167
235
  | Step | Pass criteria |
168
236
  |------|--------------|
169
- | 1 | `pip install --upgrade tokenjam` succeeds, version matches release |
237
+ | 1 | `pipx install --force tokenjam` succeeds, version matches release |
170
238
  | 2 | Onboard prompts for plan tier; config records it; no auto `usd = 200` written |
171
239
  | 3 | Example runs without DB-lock errors; CLI shows real USD values |
172
- | 4 | All four optimize analyzers run; caveat appears in downgrade JSON; `plan` + `pricing_mode` in JSON output |
240
+ | 4 | All optimize analyzers run; caveat appears in downgrade JSON; `plan` + `pricing_mode` in JSON output |
241
+ | 4b | `tj tokenmaxx` renders the bordered report panel; JSON `tier` is one of the 6 v0.3.4 tier names; subscription plan shows multiplier line |
173
242
  | 5 | All three backfill adapters ingest from fixtures; re-runs are idempotent |
174
243
  | 6 | `--compare previous` produces a diff report; `--export-config` writes a snippet with caveat comments |
175
244
  | 7 | `tj policy list` renders the unified table |
176
- | 8 | `tj serve` starts, web UI loads, HTTP fallback works while server holds lock |
245
+ | 8 | `tj serve` starts, web UI loads, HTTP fallback works while server holds lock; **zero external requests in DevTools Network tab** (offline-UI fix shipped in v0.3.4) |
246
+ | 9 | `cache_write_tokens` column present on the spans table (migration 5); cache-hit spans show non-zero cost_usd |
177
247
  | Claude Code | Onboard writes settings.json + projects.json; re-run is a no-op |
178
248
  | Codex | Onboard writes `[otel]` + `[mcp_servers.tj]` to codex config; secret synced |
179
249