traigent-schema 4.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (150) hide show
  1. traigent_schema-4.1.0/.gitignore +97 -0
  2. traigent_schema-4.1.0/CHANGELOG.md +204 -0
  3. traigent_schema-4.1.0/LICENSE +661 -0
  4. traigent_schema-4.1.0/NOTICE +19 -0
  5. traigent_schema-4.1.0/PKG-INFO +234 -0
  6. traigent_schema-4.1.0/README.md +199 -0
  7. traigent_schema-4.1.0/pyproject.toml +84 -0
  8. traigent_schema-4.1.0/traigent_schema/__init__.py +37 -0
  9. traigent_schema-4.1.0/traigent_schema/analytics_validators.py +226 -0
  10. traigent_schema-4.1.0/traigent_schema/schemas/agents/agent_deployment_schema.json +221 -0
  11. traigent_schema-4.1.0/traigent_schema/schemas/agents/agent_response_schema.json +188 -0
  12. traigent_schema-4.1.0/traigent_schema/schemas/agents/agent_schema.json +152 -0
  13. traigent_schema-4.1.0/traigent_schema/schemas/agents/agent_types_schema.json +31 -0
  14. traigent_schema-4.1.0/traigent_schema/schemas/agents/agents_endpoints.json +232 -0
  15. traigent_schema-4.1.0/traigent_schema/schemas/agents/model_parameters_schema.json +120 -0
  16. traigent_schema-4.1.0/traigent_schema/schemas/agents/model_schema.json +224 -0
  17. traigent_schema-4.1.0/traigent_schema/schemas/agents/retriever_schema.json +125 -0
  18. traigent_schema-4.1.0/traigent_schema/schemas/analytics/analytics_endpoints.json +116 -0
  19. traigent_schema-4.1.0/traigent_schema/schemas/analytics/dataset_quality_schema.json +159 -0
  20. traigent_schema-4.1.0/traigent_schema/schemas/analytics/example_score_schema.json +124 -0
  21. traigent_schema-4.1.0/traigent_schema/schemas/analytics/scoring_job_status_schema.json +87 -0
  22. traigent_schema-4.1.0/traigent_schema/schemas/datasets/dataset_schema.json +12 -0
  23. traigent_schema-4.1.0/traigent_schema/schemas/datasets/datasets_endpoints.json +123 -0
  24. traigent_schema-4.1.0/traigent_schema/schemas/datasets/evaluation_set_schema.json +247 -0
  25. traigent_schema-4.1.0/traigent_schema/schemas/datasets/evaluator_config_schema.json +131 -0
  26. traigent_schema-4.1.0/traigent_schema/schemas/datasets/example_set_schema.json +69 -0
  27. traigent_schema-4.1.0/traigent_schema/schemas/datasets/generator_config_schema.json +128 -0
  28. traigent_schema-4.1.0/traigent_schema/schemas/evaluation/configuration_run_schema.json +182 -0
  29. traigent_schema-4.1.0/traigent_schema/schemas/evaluation/evaluation_request_schema.json +39 -0
  30. traigent_schema-4.1.0/traigent_schema/schemas/evaluation/evaluation_results_schema.json +135 -0
  31. traigent_schema-4.1.0/traigent_schema/schemas/evaluation/evaluation_schema.json +137 -0
  32. traigent_schema-4.1.0/traigent_schema/schemas/evaluation/experiment_run_schema.json +517 -0
  33. traigent_schema-4.1.0/traigent_schema/schemas/evaluation/experiment_schema.json +246 -0
  34. traigent_schema-4.1.0/traigent_schema/schemas/execution/dataset_storage_schema.json +233 -0
  35. traigent_schema-4.1.0/traigent_schema/schemas/execution/execution_endpoints.json +458 -0
  36. traigent_schema-4.1.0/traigent_schema/schemas/execution/execution_mode_schema.json +171 -0
  37. traigent_schema-4.1.0/traigent_schema/schemas/execution/hybrid_session_schema.json +183 -0
  38. traigent_schema-4.1.0/traigent_schema/schemas/execution/metric_submission_schema.json +135 -0
  39. traigent_schema-4.1.0/traigent_schema/schemas/execution/saas_execution_schema.json +318 -0
  40. traigent_schema-4.1.0/traigent_schema/schemas/execution/service_registration_schema.json +64 -0
  41. traigent_schema-4.1.0/traigent_schema/schemas/execution/workflow_metadata_schema.json +140 -0
  42. traigent_schema-4.1.0/traigent_schema/schemas/measures/measure_schema.json +249 -0
  43. traigent_schema-4.1.0/traigent_schema/schemas/measures/measures_endpoints.json +95 -0
  44. traigent_schema-4.1.0/traigent_schema/schemas/measures/score_schema.json +278 -0
  45. traigent_schema-4.1.0/traigent_schema/schemas/mep.json +26 -0
  46. traigent_schema-4.1.0/traigent_schema/schemas/mep_endpoints.json +51 -0
  47. traigent_schema-4.1.0/traigent_schema/schemas/observability/annotation_queue_create_request_schema.json +27 -0
  48. traigent_schema-4.1.0/traigent_schema/schemas/observability/annotation_queue_item_complete_result_schema.json +16 -0
  49. traigent_schema-4.1.0/traigent_schema/schemas/observability/annotation_queue_item_create_request_schema.json +33 -0
  50. traigent_schema-4.1.0/traigent_schema/schemas/observability/annotation_queue_item_create_result_schema.json +16 -0
  51. traigent_schema-4.1.0/traigent_schema/schemas/observability/annotation_queue_item_list_response_schema.json +15 -0
  52. traigent_schema-4.1.0/traigent_schema/schemas/observability/annotation_queue_item_schema.json +41 -0
  53. traigent_schema-4.1.0/traigent_schema/schemas/observability/annotation_queue_list_response_schema.json +15 -0
  54. traigent_schema-4.1.0/traigent_schema/schemas/observability/annotation_queue_schema.json +35 -0
  55. traigent_schema-4.1.0/traigent_schema/schemas/observability/annotation_queue_update_request_schema.json +16 -0
  56. traigent_schema-4.1.0/traigent_schema/schemas/observability/correlation_ids_schema.json +12 -0
  57. traigent_schema-4.1.0/traigent_schema/schemas/observability/evaluator_backfill_result_schema.json +30 -0
  58. traigent_schema-4.1.0/traigent_schema/schemas/observability/evaluator_definition_schema.json +47 -0
  59. traigent_schema-4.1.0/traigent_schema/schemas/observability/evaluator_execute_request_schema.json +21 -0
  60. traigent_schema-4.1.0/traigent_schema/schemas/observability/evaluator_list_response_schema.json +15 -0
  61. traigent_schema-4.1.0/traigent_schema/schemas/observability/evaluator_run_list_response_schema.json +15 -0
  62. traigent_schema-4.1.0/traigent_schema/schemas/observability/evaluator_run_schema.json +47 -0
  63. traigent_schema-4.1.0/traigent_schema/schemas/observability/evaluator_update_request_schema.json +20 -0
  64. traigent_schema-4.1.0/traigent_schema/schemas/observability/observability_endpoints.json +703 -0
  65. traigent_schema-4.1.0/traigent_schema/schemas/observability/observability_ingest_request_schema.json +16 -0
  66. traigent_schema-4.1.0/traigent_schema/schemas/observability/observation_schema.json +33 -0
  67. traigent_schema-4.1.0/traigent_schema/schemas/observability/prompt_link_schema.json +20 -0
  68. traigent_schema-4.1.0/traigent_schema/schemas/observability/prompt_reference_schema.json +18 -0
  69. traigent_schema-4.1.0/traigent_schema/schemas/observability/review_measure_summary_schema.json +28 -0
  70. traigent_schema-4.1.0/traigent_schema/schemas/observability/review_score_create_request_schema.json +103 -0
  71. traigent_schema-4.1.0/traigent_schema/schemas/observability/review_score_list_response_schema.json +22 -0
  72. traigent_schema-4.1.0/traigent_schema/schemas/observability/review_score_schema.json +48 -0
  73. traigent_schema-4.1.0/traigent_schema/schemas/observability/session_list_response_schema.json +17 -0
  74. traigent_schema-4.1.0/traigent_schema/schemas/observability/session_schema.json +21 -0
  75. traigent_schema-4.1.0/traigent_schema/schemas/observability/trace_collaboration_state_schema.json +13 -0
  76. traigent_schema-4.1.0/traigent_schema/schemas/observability/trace_collaboration_update_request_schema.json +15 -0
  77. traigent_schema-4.1.0/traigent_schema/schemas/observability/trace_comment_create_request_schema.json +11 -0
  78. traigent_schema-4.1.0/traigent_schema/schemas/observability/trace_comment_schema.json +17 -0
  79. traigent_schema-4.1.0/traigent_schema/schemas/observability/trace_comments_response_schema.json +15 -0
  80. traigent_schema-4.1.0/traigent_schema/schemas/observability/trace_feedback_response_schema.json +8 -0
  81. traigent_schema-4.1.0/traigent_schema/schemas/observability/trace_feedback_schema.json +18 -0
  82. traigent_schema-4.1.0/traigent_schema/schemas/observability/trace_feedback_upsert_request_schema.json +13 -0
  83. traigent_schema-4.1.0/traigent_schema/schemas/observability/trace_list_response_schema.json +17 -0
  84. traigent_schema-4.1.0/traigent_schema/schemas/observability/trace_observations_response_schema.json +15 -0
  85. traigent_schema-4.1.0/traigent_schema/schemas/observability/trace_schema.json +41 -0
  86. traigent_schema-4.1.0/traigent_schema/schemas/observability/trace_score_summary_schema.json +33 -0
  87. traigent_schema-4.1.0/traigent_schema/schemas/observability/user_summary_schema.json +21 -0
  88. traigent_schema-4.1.0/traigent_schema/schemas/optimization/config_space_schema.json +105 -0
  89. traigent_schema-4.1.0/traigent_schema/schemas/optimization/exploration_config_schema.json +95 -0
  90. traigent_schema-4.1.0/traigent_schema/schemas/optimization/objective_definition_schema.json +71 -0
  91. traigent_schema-4.1.0/traigent_schema/schemas/optimization/optimization_endpoints.json +280 -0
  92. traigent_schema-4.1.0/traigent_schema/schemas/optimization/promotion_policy_schema.json +82 -0
  93. traigent_schema-4.1.0/traigent_schema/schemas/optimization/tunable_schema.json +50 -0
  94. traigent_schema-4.1.0/traigent_schema/schemas/optimization/tvar_definition_schema.json +83 -0
  95. traigent_schema-4.1.0/traigent_schema/schemas/optimization/typed_constraints_schema.json +26 -0
  96. traigent_schema-4.1.0/traigent_schema/schemas/pagination_schema.json +46 -0
  97. traigent_schema-4.1.0/traigent_schema/schemas/planned_projects_endpoints.json +865 -0
  98. traigent_schema-4.1.0/traigent_schema/schemas/planner/planner_draft_schema.json +52 -0
  99. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_context_error_schema.json +28 -0
  100. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_create_request_schema.json +30 -0
  101. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_export_job_list_response_schema.json +46 -0
  102. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_export_job_schema.json +92 -0
  103. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_export_policy_schema.json +43 -0
  104. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_export_policy_update_request_schema.json +14 -0
  105. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_list_response_schema.json +19 -0
  106. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_membership_create_request_schema.json +25 -0
  107. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_membership_list_response_schema.json +21 -0
  108. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_membership_schema.json +71 -0
  109. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_membership_update_request_schema.json +21 -0
  110. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_rate_limit_policy_schema.json +59 -0
  111. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_rate_limit_policy_update_request_schema.json +27 -0
  112. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_retention_policy_schema.json +51 -0
  113. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_retention_policy_update_request_schema.json +21 -0
  114. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_schema.json +67 -0
  115. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_scoped_analytics_summary_schema.json +246 -0
  116. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_scoped_analytics_trend_schema.json +100 -0
  117. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_scoped_core_metrics_overview_schema.json +81 -0
  118. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_scoped_evaluator_quality_dashboard_schema.json +125 -0
  119. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_scoped_experiment_trend_schema.json +95 -0
  120. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_scoped_fine_tuning_export_query_schema.json +27 -0
  121. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_scoped_fine_tuning_manifest_schema.json +106 -0
  122. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_scoped_measure_distribution_schema.json +94 -0
  123. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_scoped_observability_summary_dashboard_schema.json +132 -0
  124. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_scoped_optimization_overview_dashboard_schema.json +146 -0
  125. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_scoped_pricing_catalog_schema.json +119 -0
  126. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_scoped_project_usage_dashboard_schema.json +122 -0
  127. traigent_schema-4.1.0/traigent_schema/schemas/projects/project_update_request_schema.json +29 -0
  128. traigent_schema-4.1.0/traigent_schema/schemas/prompts/chat_prompt_message_schema.json +13 -0
  129. traigent_schema-4.1.0/traigent_schema/schemas/prompts/prompt_analytics_schema.json +24 -0
  130. traigent_schema-4.1.0/traigent_schema/schemas/prompts/prompt_create_request_schema.json +82 -0
  131. traigent_schema-4.1.0/traigent_schema/schemas/prompts/prompt_detail_schema.json +18 -0
  132. traigent_schema-4.1.0/traigent_schema/schemas/prompts/prompt_labels_update_request_schema.json +21 -0
  133. traigent_schema-4.1.0/traigent_schema/schemas/prompts/prompt_playground_request_schema.json +34 -0
  134. traigent_schema-4.1.0/traigent_schema/schemas/prompts/prompt_playground_result_schema.json +29 -0
  135. traigent_schema-4.1.0/traigent_schema/schemas/prompts/prompt_summary_schema.json +28 -0
  136. traigent_schema-4.1.0/traigent_schema/schemas/prompts/prompt_version_create_request_schema.json +36 -0
  137. traigent_schema-4.1.0/traigent_schema/schemas/prompts/prompt_version_schema.json +33 -0
  138. traigent_schema-4.1.0/traigent_schema/schemas/prompts/prompts_endpoints.json +226 -0
  139. traigent_schema-4.1.0/traigent_schema/schemas/prompts/resolved_prompt_schema.json +17 -0
  140. traigent_schema-4.1.0/traigent_schema/schemas/results/comparison_schema.json +218 -0
  141. traigent_schema-4.1.0/traigent_schema/schemas/results/report_request_schema.json +277 -0
  142. traigent_schema-4.1.0/traigent_schema/schemas/results/report_schema.json +277 -0
  143. traigent_schema-4.1.0/traigent_schema/schemas/results/results_endpoints.json +90 -0
  144. traigent_schema-4.1.0/traigent_schema/schemas/results/visualization_request_schema.json +228 -0
  145. traigent_schema-4.1.0/traigent_schema/schemas/results/visualization_schema.json +273 -0
  146. traigent_schema-4.1.0/traigent_schema/schemas/sdk_tuning_endpoints.json +21 -0
  147. traigent_schema-4.1.0/traigent_schema/schemas/status_schema.json +105 -0
  148. traigent_schema-4.1.0/traigent_schema/utils.py +129 -0
  149. traigent_schema-4.1.0/traigent_schema/validator.py +268 -0
  150. traigent_schema-4.1.0/traigent_schema/version.py +3 -0
@@ -0,0 +1,97 @@
1
+ # Byte-compiled / optimized / DLL files
2
+ __pycache__/
3
+ *.py[cod]
4
+ *$py.class
5
+
6
+ # C extensions
7
+ *.so
8
+
9
+ # Distribution / packaging
10
+ .Python
11
+ build/
12
+ develop-eggs/
13
+ dist/
14
+ downloads/
15
+ eggs/
16
+ .eggs/
17
+ lib/
18
+ lib64/
19
+ parts/
20
+ sdist/
21
+ var/
22
+ wheels/
23
+ share/python-wheels/
24
+ *.egg-info/
25
+ .installed.cfg
26
+ *.egg
27
+ MANIFEST
28
+
29
+ # PyInstaller
30
+ *.manifest
31
+ *.spec
32
+
33
+ # Installer logs
34
+ pip-log.txt
35
+ pip-delete-this-directory.txt
36
+
37
+ # Unit test / coverage reports
38
+ htmlcov/
39
+ .tox/
40
+ .nox/
41
+ .coverage
42
+ .coverage.*
43
+ .cache
44
+ nosetests.xml
45
+ coverage.xml
46
+ *.cover
47
+ *.py,cover
48
+ .hypothesis/
49
+ .pytest_cache/
50
+
51
+ # Translations
52
+ *.mo
53
+ *.pot
54
+
55
+ # Environments
56
+ .env
57
+ .venv
58
+ env/
59
+ venv/
60
+ ENV/
61
+ env.bak/
62
+ venv.bak/
63
+
64
+ # IDEs
65
+ .idea/
66
+ .vscode/
67
+ *.swp
68
+ *.swo
69
+ *~
70
+
71
+ # Jupyter Notebook
72
+ .ipynb_checkpoints
73
+
74
+ # pyenv
75
+ .python-version
76
+
77
+ # mypy
78
+ .mypy_cache/
79
+ .dmypy.json
80
+ dmypy.json
81
+
82
+ # Ruff
83
+ .ruff_cache/
84
+
85
+ # OS files
86
+ .DS_Store
87
+ Thumbs.db
88
+
89
+ # Project specific
90
+ *.log
91
+ .claude/
92
+ AGENTS.md
93
+ CLAUDE.md
94
+
95
+ # SonarQube
96
+ .scannerwork/
97
+ sonar-project.local.properties
@@ -0,0 +1,204 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project will be documented in this file.
4
+
5
+ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6
+ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
+
8
+ ## [4.1.0] - 2026-05-08
9
+
10
+ ### Added
11
+ - Added `parity/python-js-sdk.json` as the authoritative Python/JavaScript SDK
12
+ parity manifest, including release-order and refresh-cadence policy.
13
+ - Added manifest classifications for workflow-cost parity symbols used by the
14
+ Python and JavaScript SDKs.
15
+ - Added CI coverage that validates Python root-symbol classifications and
16
+ required JavaScript export coverage against the parity manifest.
17
+ - Added `planner/planner_draft_schema.json` for planner draft contracts.
18
+
19
+ ### Changed
20
+ - Published the repository as the public source for `traigent-schema` contracts.
21
+ - Removed local-only agent instruction files and Sonar workspace configuration
22
+ from version control, and ignored those local files going forward.
23
+ - Clarified that `planned_projects` is a pre-release contract surface that may
24
+ change without a major-version bump until it graduates.
25
+
26
+ ## [4.0.0] - 2026-03-28
27
+
28
+ ### Changed
29
+ - Split endpoint catalogs into three explicit contract roots:
30
+ - `mep_endpoints.json` is now the canonical current-backend contract
31
+ - `sdk_tuning_endpoints.json` is the canonical direct-tuning contract for SDK session and hybrid flows
32
+ - `planned_projects_endpoints.json` preserves the `/api/v1beta/projects/...` surface as planned/experimental
33
+ - `SchemaValidator` now accepts `contract="backend" | "sdk_tuning" | "planned_projects"`, while `get_openapi_path()` remains a backward-compatible alias for the backend contract root.
34
+ - Canonical backend endpoint modules now use explicit `/api/v1/...` paths and only reference current backend route families.
35
+
36
+ ### Removed
37
+ - Planned project-scoped routes and legacy tunables/config-space routes are no longer part of the default backend contract.
38
+
39
+ ### Fixed
40
+ - Request validation now supports inline OpenAPI request schemas in addition to `$ref`-based mappings, which keeps split contract files simpler without losing validator coverage.
41
+
42
+ ## [3.2.2] - 2026-03-28
43
+
44
+ ### Changed
45
+ - Relicensed open-source releases of `traigent-schema` to AGPL-3.0-only to align
46
+ with the Traigent SDK release model.
47
+ - Added a release-oriented publish workflow with TestPyPI/PyPI verification.
48
+ - CI now validates build artifacts by building the wheel/sdist, running
49
+ `twine check`, and importing the built wheel in a clean virtual environment.
50
+
51
+ ### Fixed
52
+ - README release metadata now matches the package version and describes the
53
+ repository as the shared contract source for backend, SDK, and frontend
54
+ consumers.
55
+
56
+ ## [3.2.1] - 2026-03-23
57
+
58
+ ### Fixed
59
+ - `GET /api/v1/experiment-runs/runs/{run_id}/traces` no longer requires the legacy `trials` and `trials_pagination` fields alongside the standardized `trials_page` envelope.
60
+ - Added a stable `$id` to `pagination_schema.json` so registry-backed cross-file `$ref` resolution works for paginated response schemas.
61
+ - `SchemaValidator` now loads request mappings from endpoint modules declared in `mep_endpoints.json`, restoring request validation for modular contracts such as agents and optimization.
62
+ - Added the optimization endpoint module to the root `mep_endpoints.json` catalog for discoverability.
63
+
64
+ ## [3.2.0] - 2026-03-05
65
+
66
+ ### Added
67
+ - Shared pagination contract schema:
68
+ - `pagination_schema.json`: canonical pagination metadata (`page`, `per_page`, `total`, `total_pages`, `has_next`, `has_prev`)
69
+ - New backend-aligned endpoint contracts in `execution_endpoints.json`:
70
+ - `GET /api/v1/keys/accessible-resources`
71
+ - `GET /api/v1/experiment-runs/runs/{run_id}/traces`
72
+ - `GET /api/v1/traces/{trace_id}`
73
+
74
+ ### Changed
75
+ - **BREAKING**: Standardized paginated list response envelope for target endpoint modules:
76
+ - `agents/agents_endpoints.json`
77
+ - `measures/measures_endpoints.json`
78
+ - `execution/execution_endpoints.json`
79
+ - `evaluation/evaluation_endpoints.json`
80
+ - **BREAKING**: List endpoints in those modules now model:
81
+ - `success`, `message`, `data.items`, `data.pagination`
82
+ - instead of legacy `data: []` and `count/total` top-level conventions.
83
+ - **BREAKING**: Query pagination parameters were aligned from legacy `limit/offset` to `page/per_page` in the updated list endpoints.
84
+
85
+ ### Migration Guide (FE/SDK/Backend Consumers)
86
+ - Request update:
87
+ - Replace `limit`/`offset` with `page`/`per_page`.
88
+ - Response parsing update:
89
+ - **Before**: read list payload from `data` (array), and pagination-like totals from top-level fields.
90
+ - **After**: read list payload from `data.items` and pagination metadata from `data.pagination`.
91
+ - Backward compatibility note:
92
+ - Legacy list contracts are not modeled in this version for the updated endpoints.
93
+ - Consumers should migrate parsing logic to the standardized paginated envelope.
94
+
95
+ ## [3.1.0] - 2024-01-17
96
+
97
+ ### Added
98
+ - **Analytics Schemas**: New schemas for example and dataset quality scoring
99
+ - `example_score_schema.json`: Per-example quality scores (11 score fields)
100
+ - `dataset_quality_schema.json`: Dataset-level metrics and recommendations
101
+ - `scoring_job_status_schema.json`: Async scoring job status tracking
102
+ - `analytics_endpoints.json`: OpenAPI spec for 4 new endpoints
103
+ - **AnalyticsValidator**: New validation utility class for analytics schemas
104
+ - Comprehensive test suite for analytics schemas (50+ test cases)
105
+
106
+ ### Changed
107
+
108
+ - **BREAKING**: Nested Example Metrics Structure (`configuration_run_schema.json`):
109
+ - `measures` array now uses `ExampleMetrics` structure instead of flat `MeasureResults`
110
+ - New structure: `{"example_id": "ex_...", "metrics": {"score": 0.85, ...}}`
111
+ - This prevents `example_id` from being stripped during numeric-only validation
112
+ - Added `maxItems: 1000` constraint on measures array
113
+ - **Workflow Metadata Schema Enhancements** (`workflow_metadata_schema.json`):
114
+ - `AgentCostBreakdown`: Added `agent_id` field (required), renamed `model` to `model_used`, added `total_tokens` field (required)
115
+ - `WorkflowMetadata`: `workflow_name` is now required (was nullable), added `total_tokens`, `total_input_cost`, `total_output_cost` fields (all required)
116
+
117
+ ### Added (Validators)
118
+
119
+ - `validate_example_metrics()` method in `AnalyticsValidator` for nested format validation
120
+ - Test suite for `ExampleMetrics` validation (15 test cases)
121
+
122
+ ### Migration Guide (Nested Measures)
123
+
124
+ **Before (flat format - v3.0.x)**:
125
+
126
+ ```json
127
+ {
128
+ "measures": [
129
+ {"example_id": "ex_abc123_0", "score": 0.85, "cost": 0.05}
130
+ ]
131
+ }
132
+ ```
133
+
134
+ **After (nested format - v3.1.0)**:
135
+
136
+ ```json
137
+ {
138
+ "measures": [
139
+ {
140
+ "example_id": "ex_abc123_0",
141
+ "metrics": {"score": 0.85, "cost": 0.05}
142
+ }
143
+ ]
144
+ }
145
+ ```
146
+
147
+ ## [3.0.0] - 2024-01-15
148
+
149
+ ### Added
150
+ - **Workflow Metadata**: New schema for multi-agent workflow cost tracking
151
+ - `workflow_metadata_schema.json`: Tracks execution time, token usage, and costs per agent
152
+ - `AgentCostBreakdown` definition for per-agent metrics
153
+ - `WorkflowMetadata` definition for aggregate workflow metrics
154
+ - `workflow_metadata` field in `configuration_run_schema.json` for SDK integration
155
+
156
+ ### Changed
157
+ - **BREAKING**: `MeasureResults` in `configuration_run_schema.json` now enforces:
158
+ - Numeric values only (`["number", "null"]`) - string/boolean no longer accepted
159
+ - Maximum 50 keys per result (`maxProperties: 50`)
160
+ - **BREAKING**: `MetricResults` in `metric_submission_schema.json` aligned with MeasureResults:
161
+ - Numeric values only (`["number", "null"]`)
162
+ - Maximum 50 keys per submission (`maxProperties: 50`)
163
+ - Non-numeric values should be placed in `metadata` field instead
164
+
165
+ ### Fixed
166
+ - Removed conflicting `$id` values from `configuration_run_schema.json` and `workflow_metadata_schema.json` to enable proper `$ref` resolution
167
+ - `workflow_metadata_schema.json` root type now correctly validates only `WorkflowMetadata` (not `AgentCostBreakdown`)
168
+
169
+ ### Migration Guide
170
+
171
+ #### MeasureResults Contract Change
172
+ If your SDK or backend previously submitted string/boolean measure values:
173
+
174
+ **Before (v2.x)**:
175
+ ```python
176
+ measures = {
177
+ "accuracy": 0.95,
178
+ "status": "passed", # String - no longer valid
179
+ "passed": True # Boolean - no longer valid
180
+ }
181
+ ```
182
+
183
+ **After (v3.x)**:
184
+ ```python
185
+ measures = {
186
+ "accuracy": 0.95,
187
+ "pass_rate": 1.0 # Numeric representation
188
+ }
189
+ # Move non-numeric data to metadata
190
+ metadata = {
191
+ "status": "passed",
192
+ "details": {"passed": True}
193
+ }
194
+ ```
195
+
196
+ #### Max Keys Limit
197
+ Ensure measure submissions have no more than 50 keys. If you need more:
198
+ 1. Aggregate related measures
199
+ 2. Use metadata for non-score information
200
+ 3. Split into multiple submissions if necessary
201
+
202
+ ## [2.x.x] - Previous Releases
203
+
204
+ See git history for changes prior to v3.0.0.