dataspoof-mlpilot 2.1.0__tar.gz → 2.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (330) hide show
  1. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/.github/workflows/CI.yml +44 -1
  2. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/CHANGELOG.md +326 -204
  3. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/CONTRIBUTING.md +2 -2
  4. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/Cargo.lock +1 -1
  5. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/Cargo.toml +1 -1
  6. dataspoof_mlpilot-2.3.0/LICENSE +21 -0
  7. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/PKG-INFO +1877 -1715
  8. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/README.md +1861 -1707
  9. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/SECURITY.md +2 -2
  10. dataspoof_mlpilot-2.3.0/docs/api/README.md +36 -0
  11. dataspoof_mlpilot-2.3.0/docs/api/anomaly.md +69 -0
  12. dataspoof_mlpilot-2.3.0/docs/api/automl.md +437 -0
  13. dataspoof_mlpilot-2.3.0/docs/api/causal.md +112 -0
  14. dataspoof_mlpilot-2.3.0/docs/api/cleaning.md +911 -0
  15. dataspoof_mlpilot-2.3.0/docs/api/compression.md +125 -0
  16. dataspoof_mlpilot-2.3.0/docs/api/core.md +459 -0
  17. dataspoof_mlpilot-2.3.0/docs/api/deploy.md +642 -0
  18. dataspoof_mlpilot-2.3.0/docs/api/environments.md +223 -0
  19. dataspoof_mlpilot-2.3.0/docs/api/evaluation.md +625 -0
  20. dataspoof_mlpilot-2.3.0/docs/api/feature_store.md +142 -0
  21. dataspoof_mlpilot-2.3.0/docs/api/features.md +389 -0
  22. dataspoof_mlpilot-2.3.0/docs/api/forecast.md +126 -0
  23. dataspoof_mlpilot-2.3.0/docs/api/hub.md +621 -0
  24. dataspoof_mlpilot-2.3.0/docs/api/ingestion.md +589 -0
  25. dataspoof_mlpilot-2.3.0/docs/api/insights.md +161 -0
  26. dataspoof_mlpilot-2.3.0/docs/api/integrations.md +855 -0
  27. dataspoof_mlpilot-2.3.0/docs/api/lineage.md +186 -0
  28. dataspoof_mlpilot-2.3.0/docs/api/media.md +344 -0
  29. dataspoof_mlpilot-2.3.0/docs/api/model_security.md +232 -0
  30. dataspoof_mlpilot-2.3.0/docs/api/monitoring.md +746 -0
  31. dataspoof_mlpilot-2.3.0/docs/api/operations.md +334 -0
  32. dataspoof_mlpilot-2.3.0/docs/api/orchestration.md +420 -0
  33. dataspoof_mlpilot-2.3.0/docs/api/quality.md +781 -0
  34. dataspoof_mlpilot-2.3.0/docs/api/registry.md +471 -0
  35. dataspoof_mlpilot-2.3.0/docs/api/scale.md +83 -0
  36. dataspoof_mlpilot-2.3.0/docs/api/secrets_security.md +171 -0
  37. dataspoof_mlpilot-2.3.0/docs/api/splitting.md +120 -0
  38. dataspoof_mlpilot-2.3.0/docs/api/stats.md +195 -0
  39. dataspoof_mlpilot-2.3.0/docs/api/versioning.md +180 -0
  40. dataspoof_mlpilot-2.3.0/docs/api/vision.md +491 -0
  41. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/architecture.md +32 -32
  42. dataspoof_mlpilot-2.3.0/docs/enterprise-gap-audit.md +233 -0
  43. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/deployment.md +72 -64
  44. dataspoof_mlpilot-2.3.0/docs/guide/enterprise-operations.md +246 -0
  45. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/media.md +48 -32
  46. dataspoof_mlpilot-2.3.0/docs/guide/quantization.md +143 -0
  47. dataspoof_mlpilot-2.3.0/docs/guide/vision.md +155 -0
  48. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/index.md +105 -105
  49. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/installation.md +90 -89
  50. dataspoof_mlpilot-2.3.0/docs/reference/compression.md +5 -0
  51. dataspoof_mlpilot-2.3.0/docs/reference/model_privacy.md +3 -0
  52. dataspoof_mlpilot-2.3.0/docs/reference/operations.md +11 -0
  53. dataspoof_mlpilot-2.3.0/docs/reference/orchestration.md +9 -0
  54. dataspoof_mlpilot-2.3.0/examples/enterprise_walkthrough.ipynb +1325 -0
  55. dataspoof_mlpilot-2.3.0/examples/titanic_walkthrough.ipynb +1313 -0
  56. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mkdocs.yml +149 -138
  57. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/__init__.py +66 -60
  58. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/automl/automl.py +630 -630
  59. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/automl/models.py +16 -1
  60. dataspoof_mlpilot-2.3.0/mlpilot/backup.py +217 -0
  61. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/cicd.py +437 -396
  62. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/cleaners.py +8 -2
  63. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/cleaning.py +1367 -1368
  64. dataspoof_mlpilot-2.3.0/mlpilot/compression.py +221 -0
  65. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/deploy/__init__.py +21 -18
  66. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/deploy/cli.py +419 -284
  67. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/deploy/export.py +317 -285
  68. dataspoof_mlpilot-2.3.0/mlpilot/deploy/quantize.py +341 -0
  69. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/deploy/server.py +100 -6
  70. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/deploy/wsgi.py +59 -53
  71. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/features/text.py +156 -133
  72. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/hub/app.py +854 -700
  73. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/hub/db.py +49 -1
  74. dataspoof_mlpilot-2.3.0/mlpilot/labeling.py +229 -0
  75. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/lineage.py +1 -1
  76. dataspoof_mlpilot-2.3.0/mlpilot/media/__init__.py +45 -0
  77. dataspoof_mlpilot-2.3.0/mlpilot/media/benchmark.py +172 -0
  78. dataspoof_mlpilot-2.3.0/mlpilot/media/conditions.py +414 -0
  79. dataspoof_mlpilot-2.3.0/mlpilot/media/faces.py +343 -0
  80. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/media/image.py +365 -353
  81. dataspoof_mlpilot-2.3.0/mlpilot/media/plates.py +292 -0
  82. dataspoof_mlpilot-2.3.0/mlpilot/model_privacy.py +132 -0
  83. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/monitoring/__init__.py +6 -1
  84. dataspoof_mlpilot-2.3.0/mlpilot/monitoring/incidents.py +265 -0
  85. dataspoof_mlpilot-2.3.0/mlpilot/orchestration/__init__.py +30 -0
  86. dataspoof_mlpilot-2.3.0/mlpilot/orchestration/engine.py +477 -0
  87. dataspoof_mlpilot-2.3.0/mlpilot/orchestration/goal.py +240 -0
  88. dataspoof_mlpilot-2.3.0/mlpilot/orchestration/tools.py +413 -0
  89. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/quality/__init__.py +29 -25
  90. dataspoof_mlpilot-2.3.0/mlpilot/quality/observability.py +303 -0
  91. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/registry/__init__.py +16 -15
  92. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/registry/_store.py +3 -0
  93. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/registry/models.py +356 -280
  94. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/steps.py +930 -926
  95. dataspoof_mlpilot-2.3.0/mlpilot/supply_chain.py +260 -0
  96. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/pyproject.toml +235 -222
  97. dataspoof_mlpilot-2.3.0/scripts/gen_api_ref.py +283 -0
  98. dataspoof_mlpilot-2.3.0/tests/assets/astronaut.jpg +0 -0
  99. dataspoof_mlpilot-2.3.0/tests/assets/coffee.jpg +0 -0
  100. dataspoof_mlpilot-2.3.0/tests/test_backup.py +90 -0
  101. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_cleaners.py +9 -0
  102. dataspoof_mlpilot-2.3.0/tests/test_fault_injection.py +124 -0
  103. dataspoof_mlpilot-2.3.0/tests/test_hub_tenancy.py +111 -0
  104. dataspoof_mlpilot-2.3.0/tests/test_incidents.py +85 -0
  105. dataspoof_mlpilot-2.3.0/tests/test_labeling.py +67 -0
  106. dataspoof_mlpilot-2.3.0/tests/test_model_governance.py +59 -0
  107. dataspoof_mlpilot-2.3.0/tests/test_model_privacy.py +49 -0
  108. dataspoof_mlpilot-2.3.0/tests/test_observability.py +80 -0
  109. dataspoof_mlpilot-2.3.0/tests/test_orchestration.py +205 -0
  110. dataspoof_mlpilot-2.3.0/tests/test_quantization.py +244 -0
  111. dataspoof_mlpilot-2.3.0/tests/test_serving_hardening.py +78 -0
  112. dataspoof_mlpilot-2.3.0/tests/test_supply_chain.py +77 -0
  113. dataspoof_mlpilot-2.3.0/tests/test_vision.py +273 -0
  114. dataspoof_mlpilot-2.1.0/mlpilot/media/__init__.py +0 -19
  115. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/.dockerignore +0 -0
  116. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/.github/workflows/docs.yml +0 -0
  117. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/.gitignore +0 -0
  118. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/BUILD.md +0 -0
  119. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/helm/mlpilot-hub/Chart.yaml +0 -0
  120. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/helm/mlpilot-hub/templates/_helpers.tpl +0 -0
  121. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/helm/mlpilot-hub/templates/deployment.yaml +0 -0
  122. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/helm/mlpilot-hub/templates/ingress.yaml +0 -0
  123. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/helm/mlpilot-hub/templates/pdb.yaml +0 -0
  124. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/helm/mlpilot-hub/templates/pvc.yaml +0 -0
  125. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/helm/mlpilot-hub/templates/secret.yaml +0 -0
  126. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/helm/mlpilot-hub/templates/service.yaml +0 -0
  127. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/helm/mlpilot-hub/templates/serviceaccount.yaml +0 -0
  128. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/helm/mlpilot-hub/values.yaml +0 -0
  129. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/hub/Dockerfile +0 -0
  130. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/hub/docker-compose.yml +0 -0
  131. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/monitoring/docker-compose.yml +0 -0
  132. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/monitoring/grafana/dashboards/mlpilot-model.json +0 -0
  133. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/monitoring/grafana/provisioning/dashboards/mlpilot.yml +0 -0
  134. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/monitoring/grafana/provisioning/datasources/prometheus.yml +0 -0
  135. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/monitoring/prometheus/alerts.yml +0 -0
  136. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/monitoring/prometheus/alerts_test.yml +0 -0
  137. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/monitoring/prometheus/prometheus.yml +0 -0
  138. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/opa/policies/mlpilot/authz.rego +0 -0
  139. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/opa/policies/mlpilot/config/data.json +0 -0
  140. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/opa/policies/mlpilot/policy_test.rego +0 -0
  141. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/opa/policies/mlpilot/promotion.rego +0 -0
  142. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/terraform/aws/main.tf +0 -0
  143. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/terraform/aws/outputs.tf +0 -0
  144. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/terraform/aws/variables.tf +0 -0
  145. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/terraform/aws-ecs/main.tf +0 -0
  146. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/terraform/aws-ecs/outputs.tf +0 -0
  147. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/terraform/aws-ecs/variables.tf +0 -0
  148. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/terraform/aws-sagemaker/main.tf +0 -0
  149. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/terraform/aws-sagemaker/outputs.tf +0 -0
  150. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/deploy/terraform/aws-sagemaker/variables.tf +0 -0
  151. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/assets/workflow.svg +0 -0
  152. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/changelog.md +0 -0
  153. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/contributing.md +0 -0
  154. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/anomaly.md +0 -0
  155. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/automl.md +0 -0
  156. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/aws.md +0 -0
  157. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/causal.md +0 -0
  158. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/cleaning.md +0 -0
  159. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/environments-cicd.md +0 -0
  160. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/evaluation.md +0 -0
  161. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/feature-store.md +0 -0
  162. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/features.md +0 -0
  163. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/forecasting.md +0 -0
  164. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/hub.md +0 -0
  165. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/ingestion.md +0 -0
  166. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/insights.md +0 -0
  167. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/integrations.md +0 -0
  168. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/lineage.md +0 -0
  169. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/model-security.md +0 -0
  170. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/monitoring.md +0 -0
  171. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/observability.md +0 -0
  172. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/quality.md +0 -0
  173. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/registry.md +0 -0
  174. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/secrets-policy.md +0 -0
  175. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/splitting.md +0 -0
  176. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/stats.md +0 -0
  177. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/guide/versioning.md +0 -0
  178. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/policies.md +0 -0
  179. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/anomaly.md +0 -0
  180. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/automl.md +0 -0
  181. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/causal.md +0 -0
  182. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/cicd.md +0 -0
  183. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/cleaners.md +0 -0
  184. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/core.md +0 -0
  185. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/deploy.md +0 -0
  186. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/environments.md +0 -0
  187. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/evaluation.md +0 -0
  188. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/feature_store.md +0 -0
  189. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/features.md +0 -0
  190. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/forecast.md +0 -0
  191. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/hub.md +0 -0
  192. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/ingestion.md +0 -0
  193. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/insights.md +0 -0
  194. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/integrations.md +0 -0
  195. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/lineage.md +0 -0
  196. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/media.md +0 -0
  197. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/monitoring.md +0 -0
  198. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/poisoning.md +0 -0
  199. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/quality.md +0 -0
  200. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/registry.md +0 -0
  201. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/robustness.md +0 -0
  202. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/scale.md +0 -0
  203. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/secretstore.md +0 -0
  204. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/security.md +0 -0
  205. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/splitting.md +0 -0
  206. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/stats.md +0 -0
  207. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/reference/versioning.md +0 -0
  208. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/security.md +0 -0
  209. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/docs/workflow.md +0 -0
  210. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/_compat.py +0 -0
  211. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/_detect.py +0 -0
  212. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/_rustcore.pyi +0 -0
  213. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/anomaly.py +0 -0
  214. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/automl/__init__.py +0 -0
  215. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/automl/_persist.py +0 -0
  216. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/automl/cluster.py +0 -0
  217. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/automl/frame.py +0 -0
  218. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/automl/tasks.py +0 -0
  219. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/causal.py +0 -0
  220. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/core/__init__.py +0 -0
  221. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/core/pipeline.py +0 -0
  222. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/core/registry.py +0 -0
  223. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/core/step.py +0 -0
  224. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/deploy/__main__.py +0 -0
  225. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/deploy/_names.py +0 -0
  226. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/deploy/asgi.py +0 -0
  227. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/deploy/aws.py +0 -0
  228. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/deploy/scoring.py +0 -0
  229. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/environments.py +0 -0
  230. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/evaluation/__init__.py +0 -0
  231. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/evaluation/_slicing.py +0 -0
  232. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/evaluation/adapter.py +0 -0
  233. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/evaluation/card.py +0 -0
  234. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/evaluation/errors.py +0 -0
  235. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/evaluation/explain.py +0 -0
  236. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/evaluation/fairness.py +0 -0
  237. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/evaluation/metrics.py +0 -0
  238. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/feature_store.py +0 -0
  239. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/features/__init__.py +0 -0
  240. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/features/dates.py +0 -0
  241. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/features/defaults.py +0 -0
  242. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/features/encoding.py +0 -0
  243. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/features/generators.py +0 -0
  244. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/features/leakage.py +0 -0
  245. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/features/selection.py +0 -0
  246. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/forecast.py +0 -0
  247. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/hub/__init__.py +0 -0
  248. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/hub/client.py +0 -0
  249. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/hub/policy.py +0 -0
  250. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/hub/security.py +0 -0
  251. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/hub/storage.py +0 -0
  252. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/hub/ui.html +0 -0
  253. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/hub/ui.js +0 -0
  254. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/ingestion/__init__.py +0 -0
  255. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/ingestion/_util.py +0 -0
  256. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/ingestion/api.py +0 -0
  257. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/ingestion/core.py +0 -0
  258. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/ingestion/files.py +0 -0
  259. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/ingestion/incremental.py +0 -0
  260. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/ingestion/schema.py +0 -0
  261. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/ingestion/sql.py +0 -0
  262. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/ingestion/warehouse.py +0 -0
  263. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/insights/__init__.py +0 -0
  264. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/insights/finder.py +0 -0
  265. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/insights/narrate.py +0 -0
  266. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/insights/report.py +0 -0
  267. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/integrations/__init__.py +0 -0
  268. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/integrations/airflow.py +0 -0
  269. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/integrations/aws_glue.py +0 -0
  270. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/integrations/azure_data_factory.py +0 -0
  271. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/integrations/dagster.py +0 -0
  272. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/integrations/dvc.py +0 -0
  273. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/integrations/evidently.py +0 -0
  274. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/integrations/feast.py +0 -0
  275. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/integrations/kafka.py +0 -0
  276. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/integrations/mlflow.py +0 -0
  277. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/integrations/prefect.py +0 -0
  278. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/integrations/tasks.py +0 -0
  279. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/media/audio.py +0 -0
  280. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/monitoring/drift.py +0 -0
  281. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/monitoring/estimate.py +0 -0
  282. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/monitoring/monitor.py +0 -0
  283. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/monitoring/ops.py +0 -0
  284. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/poisoning.py +0 -0
  285. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/py.typed +0 -0
  286. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/quality/checks.py +0 -0
  287. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/quality/contracts.py +0 -0
  288. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/quality/pipeline_steps.py +0 -0
  289. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/quality/profile.py +0 -0
  290. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/quality/schema.py +0 -0
  291. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/registry/tracking.py +0 -0
  292. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/robustness.py +0 -0
  293. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/scale.py +0 -0
  294. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/secretstore.py +0 -0
  295. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/security.py +0 -0
  296. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/splitting.py +0 -0
  297. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/stats.py +0 -0
  298. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/mlpilot/versioning.py +0 -0
  299. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/src/clean.rs +0 -0
  300. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/src/lib.rs +0 -0
  301. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/src/stats.rs +0 -0
  302. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/src/text.rs +0 -0
  303. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_advanced.py +0 -0
  304. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_automl.py +0 -0
  305. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_aws_deploy.py +0 -0
  306. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_cicd.py +0 -0
  307. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_core.py +0 -0
  308. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_data_versions_and_feature_store.py +0 -0
  309. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_deploy.py +0 -0
  310. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_docker_stacks.py +0 -0
  311. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_environments.py +0 -0
  312. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_evaluation.py +0 -0
  313. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_features.py +0 -0
  314. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_hub.py +0 -0
  315. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_ingestion.py +0 -0
  316. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_insights.py +0 -0
  317. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_label_free_monitoring.py +0 -0
  318. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_lineage_and_catalogs.py +0 -0
  319. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_media.py +0 -0
  320. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_model_security.py +0 -0
  321. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_monitoring.py +0 -0
  322. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_ops.py +0 -0
  323. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_quality.py +0 -0
  324. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_real_integrations.py +0 -0
  325. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_registry.py +0 -0
  326. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_rename_compat.py +0 -0
  327. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_secretstore.py +0 -0
  328. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_serving_feedback_reload.py +0 -0
  329. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_splitting.py +0 -0
  330. {dataspoof_mlpilot-2.1.0 → dataspoof_mlpilot-2.3.0}/tests/test_stats.py +0 -0
@@ -153,6 +153,49 @@ jobs:
153
153
  cargo generate-lockfile
154
154
  cargo audit
155
155
 
156
+ certify:
157
+ name: release certification
158
+ # Evidence that the production guarantees hold on the built package: the autonomous engine end
159
+ # to end, model-file scanning, disaster-recovery drill, fault injection, tenancy isolation, SBOM.
160
+ runs-on: ubuntu-latest
161
+ env:
162
+ AC_OFFLINE: "1"
163
+ steps:
164
+ - uses: actions/checkout@v4
165
+ - uses: actions/setup-python@v5
166
+ with:
167
+ python-version: "3.12"
168
+ cache: pip
169
+ - uses: dtolnay/rust-toolchain@stable
170
+ - uses: Swatinem/rust-cache@v2
171
+ - name: Install the package
172
+ run: python -m pip install ".[test,server,signing]"
173
+ - name: Production-guarantee test suites
174
+ run: |
175
+ mkdir _cert && cp -r tests _cert/ && cd _cert
176
+ python -m pytest -q -p no:cacheprovider -o pythonpath= tests/test_orchestration.py \
177
+ tests/test_fault_injection.py tests/test_backup.py tests/test_supply_chain.py \
178
+ tests/test_serving_hardening.py tests/test_hub_tenancy.py tests/test_incidents.py \
179
+ tests/test_model_governance.py tests/test_model_privacy.py tests/test_observability.py
180
+ - name: Live certification run
181
+ run: |
182
+ mkdir -p evidence && cd evidence
183
+ python -c "import numpy as np, polars as pl; r=np.random.default_rng(0); x=r.normal(size=(800,3)); pl.DataFrame({'a':x[:,0],'b':x[:,1],'c':x[:,2],'y':(x[:,0]+r.normal(0,.7,800)>0).astype(int)}).write_csv('data.csv')"
184
+ mlpilot run --text "predict y from data.csv with AUC above 0.7, latency below 500 ms; deploy to staging" \
185
+ --no-llm --name cert --out runs/cert --by ci
186
+ mlpilot scan-model runs/cert/artifacts/model
187
+ mlpilot dr-drill --source runs=runs/cert --source registry=registry --workdir dr --rto-limit 120
188
+ mlpilot sbom --out sbom.mlpilot.cdx.json
189
+ cp runs/cert/report.md certification-report.md
190
+ - uses: actions/upload-artifact@v4
191
+ with:
192
+ name: certification-evidence
193
+ path: |
194
+ evidence/certification-report.md
195
+ evidence/sbom.mlpilot.cdx.json
196
+ evidence/runs/cert/manifest.json
197
+ evidence/runs/cert/events.jsonl
198
+
156
199
  linux:
157
200
  runs-on: ubuntu-latest
158
201
  strategy:
@@ -236,7 +279,7 @@ jobs:
236
279
  name: Release to PyPI
237
280
  runs-on: ubuntu-latest
238
281
  if: startsWith(github.ref, 'refs/tags/v')
239
- needs: [test, core-only, typecheck, security, linux, windows, macos, sdist]
282
+ needs: [test, core-only, typecheck, security, certify, linux, windows, macos, sdist]
240
283
  environment:
241
284
  name: pypi
242
285
  url: https://pypi.org/project/dataspoof-mlpilot/
@@ -1,204 +1,326 @@
1
- # Changelog
2
-
3
- All notable changes are listed here. The project follows [Semantic Versioning](https://semver.org/)
4
- (see the [versioning policy](https://dataspoof.github.io/dataspoof-data-cleaning/policies/)).
5
-
6
- ## [2.1.0] — 2026-11-09
7
-
8
- ### Renamed: AutomatedCleaning is now MLPilot
9
- - **Names:**
10
- - the distribution is `dataspoof-mlpilot` (PyPI refused `mlpilot` as too similar to an existing project);
11
- - the import is `import mlpilot`, and the CLI is `mlpilot`;
12
- - `MLPilotWarning` replaces `AutomatedCleaningWarning`, which remains as an alias.
13
- - **Compatibility:**
14
- - `import automatedcleaning` and pickles that refer to `automatedcleaning.*` classes resolve to
15
- the same `mlpilot` modules, so models saved by AutomatedCleaning keep loading with the same
16
- classes;
17
- - the `AutomatedCleaning` project on PyPI gets a final 2.0.1 that installs `dataspoof-mlpilot`.
18
- - **Unchanged:** the `AC_*` environment variables, the `ac_*` Prometheus metrics and the saved-file
19
- formats (`dataspoof.*`).
20
- - **Renamed with the library:**
21
- - the Helm chart (`mlpilot-hub`), the OPA policy package (`data.mlpilot`) and the Prometheus job;
22
- - the Grafana dashboard file;
23
- - the MLflow tag prefix and the default OpenLineage namespace.
24
-
25
- ## [2.0.0] — 2026-11-09
26
-
27
- ### Breaking
28
- - **Lean core install.** `pip install AutomatedCleaning` now installs only the core
29
- dependencies. The heavy ones moved to extras:
30
- - `[nlp]`: nltk
31
- - `[dashboard]`: plotly, matplotlib, missingno, pyfiglet
32
- - `[pii]`: presidio
33
- - `[llm]`: anthropic, langchain-anthropic
34
-
35
- To get what 1.x installed, use `pip install "AutomatedCleaning[all]"`. A feature whose extra
36
- is missing raises an `ImportError` naming the extra to install.
37
- - `setup.py` was removed. The build is defined only in `pyproject.toml` (maturin).
38
-
39
- ### Changed
40
- - **No side effects at import.**
41
- - Importing the library no longer downloads NLTK data, changes the application's warning
42
- filters, or reconfigures logging.
43
- - NLTK corpora are downloaded on first use, and never when `AC_OFFLINE` is set. Text cleaning
44
- falls back to a built-in stopword list when NLTK isn't installed.
45
- - **Typed package.** It ships a `py.typed` marker and stubs for the Rust extension, and passes
46
- `mypy`.
47
- - **CI** now runs:
48
- - tests across Linux, macOS and Windows × Python 3.11–3.13 × Polars 1/2;
49
- - a core-only install test;
50
- - mypy;
51
- - pip-audit and cargo-audit.
52
-
53
- Releases publish with PyPI Trusted Publishing, signed build-provenance attestations and a
54
- CycloneDX SBOM.
55
- - Interactive AI corrections use `ANTHROPIC_API_KEY` when it is set, and the current Claude model.
56
-
57
- ### Added
58
- - **Hub, the multi-user platform server** (`automatedcleaning.hub`, `[server]` extra):
59
- - workspaces, with SSO through OIDC and identity-provider group mapping;
60
- - API tokens and service accounts;
61
- - four roles (viewer, data scientist, approver, admin);
62
- - runs and a model registry with immutable, checksummed artifacts on disk, S3, GCS or Azure;
63
- - two-person production approvals with policy gates;
64
- - monitoring results;
65
- - a hash-chained, verifiable and exportable audit log;
66
- - a web UI, the Python client `HubClient`, and `automatedcleaning hub serve | create-admin`.
67
- - **Hub deployment:** a Docker image with compose (plus PostgreSQL), a Helm chart, and a
68
- Terraform module for AWS (S3, RDS, Secrets Manager, IAM role, Helm release).
69
- - **Forecasting** (`Forecaster`):
70
- - multiple series and multi-step horizons;
71
- - rolling-origin backtests against seasonal-naive;
72
- - conformal intervals and hierarchical reconciliation.
73
- - **Causal inference** (`estimate_effect`): doubly-robust AIPW with overlap and balance
74
- diagnostics. **Uplift:** `UpliftModel` and `qini`.
75
- - **Anomaly detection** (`AutoAnomaly`): an ensemble of three detectors with plain-language
76
- explanations.
77
- - **AutoML:**
78
- - conformal `predict_interval` and `predict_set`;
79
- - `search="optuna"`, `imbalance="auto"`, `n_jobs`;
80
- - `predict_with_proba` and `set_inference_threads`.
81
- - **Serving:**
82
- - Prometheus `/metrics` and OpenTelemetry tracing;
83
- - canary and shadow rollouts, and `app_from_registry`;
84
- - SageMaker, Vertex AI and Azure ML routes and Docker targets;
85
- - an ASGI app and the `container` entry point;
86
- - single-row latency down from about 500 ms to about 50 ms.
87
- - **Security:**
88
- - Ed25519 model signing, verified before unpickling whenever `AC_MODEL_PUBLIC_KEY` is set;
89
- - `skops` export;
90
- - signing on registration with `register(..., sign_with=...)`.
91
- - **Monitoring:**
92
- - PagerDuty and Opsgenie alert handlers;
93
- - `join_labels` and `Monitor.check_labels` for labels that arrive later;
94
- - `sql_sink` and `Monitor.on_check`;
95
- - `SQLSource.insert` and `execute`.
96
- - **Scale:**
97
- - `transform_file` for files of any size;
98
- - Spark, Ray and Dask scoring;
99
- - `to_sql` SQL pushdown of fitted cleaning pipelines.
100
- - **Integrations:** Airflow operators, Prefect tasks and flow, Dagster assets, Kafka stream
101
- scoring, and plain `integrations.tasks`.
102
- - **MLflow** (`integrations.mlflow`, `[mlflow]` extra):
103
- - batched, idempotent export of runs and whole trackers;
104
- - AutoML models logged as pyfunc models with a signature, so they serve with `mlflow models serve`;
105
- - `export_registry` (stages become aliases) and `load_automl`.
106
- - **DVC** (`integrations.dvc`, `[dvc]` extra): `DVCRepo` (init, remotes, track, push and pull,
107
- read any revision, checkout, `log_to_run`) and `export_repository` for `DataRepository`
108
- history.
109
- - **Feast** (`integrations.feast`, `[feast]` extra): `export_to_feast` writes, applies and
110
- materializes a Feast repository.
111
- - **Data splitting** (`split_data`, `cv_folds`, `check_split`): stratified, grouped and time-ordered
112
- splits with gaps; every split is leak-checked (shared groups, time overlap, duplicate rows, unseen classes).
113
- - **Model security:** `robustness_report` (noise stability, a minimal adversarial attack,
114
- feature sensitivity, extreme inputs) and `poisoning_report` (confident-learning label errors,
115
- conflicting duplicates, backdoor triggers, outliers).
116
- - **Environments** (`Environments`, `environments.yml`): isolated dev / QA / prod registries;
117
- `release` enforces order, source stage, metric gates, four-eyes approvals and signatures, and
118
- records provenance on both sides. The registry and Hub gain a `qa` stage.
119
- - **Model CI/CD:** the CLI commands `train`, `gate`, `register`, `promote`, `release`, `latest`,
120
- `deploy-sagemaker` and `cicd init` (GitHub Actions / GitLab CI pipelines, checked with actionlint).
121
- - **Secrets** (`secretstore`): `vault://` (token, AppRole, Kubernetes; KV v1 and v2), `aws-sm://`,
122
- `env://` and `file://` references, resolved wherever credentials are read.
123
- - **Open Policy Agent for the Hub** (`hub.policy`): deny-overrides access rules, promotion rules
124
- and fail-closed behaviour, plus Rego policies with unit tests in `deploy/opa`.
125
- - **Serving:**
126
- - `POST /feedback` with prediction IDs, live-accuracy metrics and `join_feedback`;
127
- - hot reload (`watch=`, `POST /admin/reload`, registry-following);
128
- - live drift, estimated-performance and alert gauges (`monitor=`).
129
- - **Monitoring before labels:** `PerformanceEstimator` (CBPE, direct loss estimation),
130
- `domain_classifier_drift`, and `estimated_performance` alerts in `Monitor.check`.
131
- - **Grafana + Prometheus** (`deploy/monitoring`): a dashboard, 14 alert rules with `promtool`
132
- unit tests, and a compose stack.
133
- - **Lineage** (`lineage`): OpenLineage run events with schema, quality and version facets; HTTP
134
- and file transports; `emit_tracked_run`, `emit_dataset_version`, `LineageGraph`.
135
- - **Catalogs and orchestration:** AWS Glue (catalog tables, partitions, reads, job runs) and Azure
136
- Data Factory pipeline runs.
137
- - **AWS deployment** (`deploy.aws`): ECR push and blue/green SageMaker endpoints (serverless or
138
- instances, autoscaling, data capture, rollback alarms); Terraform for SageMaker and ECS Fargate.
139
- - **Evidently** (`integrations.evidently`): reports plus agreement with the built-in drift report.
140
- - **Tests against real infrastructure:** Prometheus, Grafana, Vault, OPA and Marquez in Docker
141
- (`AC_DOCKER_TESTS=1`); moto for AWS; dedicated CI jobs.
142
- - These three are tested against the real tools (MLflow server and model server, git + DVC
143
- with a remote, Feast), including in a dedicated CI job.
144
- - **Image and audio data** (`media`):
145
- - quality audits (corrupt files, exact and near duplicates, blur, exposure, clipping, silence,
146
- mixed formats);
147
- - preprocessors (EXIF-aware image resize and normalise; audio resample, trim and level);
148
- - augmentation;
149
- - features for AutoML (colour and edge-orientation histograms; MFCC and spectral features).
150
- - **Statistical testing** (`stats`):
151
- - automatic test selection with assumption checks, effect sizes and confidence intervals;
152
- - chi-square and Fisher's exact test, correlations;
153
- - A/B tests, power analysis, multiple-testing correction, `test_all`.
154
- - **Data version control** (`DataRepository`): content-addressed commits, tags, row-level
155
- diffs, checksum-verified checkout, links to experiment runs, garbage collection.
156
- - **Feature store** (`FeatureStore`, `Entity`, `FeatureView`): point-in-time-correct
157
- historical joins with TTL, an SQLite online store, `materialize`, schema guards.
158
- - **Documentation site** (MkDocs Material with an API reference), plus `SECURITY.md`,
159
- `CONTRIBUTING.md` and a versioning policy.
160
-
161
- ### Removed
162
- - `automatedcleaning/cleaning_update.py`: dead code that nothing imported.
163
-
164
- ## [1.14.0] — 2026-11-08
165
- - **Ingestion & connectors:**
166
- - files, globs, HTTP, and S3 / GCS / Azure;
167
- - any SQL database (bound parameters);
168
- - Snowflake and BigQuery;
169
- - REST APIs with five pagination styles;
170
- - reusable schema inference and incremental loads.
171
- - **Experiment tracking & model registry:** runs, dataset fingerprints, dev → staging →
172
- production promotion with metric gates, audit history and rollback.
173
- - **Deployment:**
174
- - batch scoring and a REST API (API keys, prediction log);
175
- - bundle, pickle and ONNX export;
176
- - Docker build contexts and a CLI.
177
- - **Monitoring:** data and prediction drift (PSI), accuracy decay, alerts and webhooks, and
178
- champion-vs-challenger retraining.
179
- - **Insights & reporting:** significance-tested findings, executive reports, and optional
180
- narratives written by Claude.
181
-
182
- ## [1.13.0] — 2026-11-06
183
- - **Feature engineering:**
184
- - dates, out-of-fold target encoding, interactions, aggregations, lags, text embeddings;
185
- - feature selection and leakage detection.
186
- - **AutoML:** task detection, time-budgeted model search, an ensemble, and AutoCluster.
187
- - **Evaluation & explainability:** metrics with confidence intervals, built-in SHAP, error
188
- analysis, fairness checks and model cards.
189
-
190
- ## [1.12.0] — 2026-11-06
191
- - Profiling & data quality: automated EDA, the 0–100 quality score, YAML data contracts, schema
192
- diffs and quality gates.
193
-
194
- ## [1.11.0] — 2026-11-05
195
- - Shared `Step` / `Pipeline` core and the stateful cleaning pipeline (JSON save/load, audit
196
- reports, schema checks).
197
-
198
- ## [1.10.0] and earlier — 2026-07
199
- - 1.10.0: 8 more data-quality checks in the dashboard.
200
- - 1.9.0: column pickers in the dashboard.
201
- - 1.8.x: the dashboard data-quality panel.
202
- - 1.7.0: the enterprise dashboard redesign.
203
- - 1.6.x: 43 additional cleaning steps and a dependency cleanup.
204
- - 1.5.x: the Rust compute backend.
1
+ # Changelog
2
+
3
+ All notable changes are listed here. The project follows [Semantic Versioning](https://semver.org/)
4
+ (see the [versioning policy](https://dataspoof.github.io/dataspoof-mlpilot/policies/)).
5
+
6
+ ## [2.3.0] — 2026-11-22
7
+
8
+ ### Added: the autonomous execution engine
9
+ - **`mlpilot.orchestration`:** a goal is a YAML file or a sentence, drafted by rules plus
10
+ optional Claude and always validated.
11
+ - The engine plans it into a typed workflow of registered tools only. Steps: ingest →
12
+ quality / contract → leak-checked split → poisoning check → AutoML → evaluation against the
13
+ goal → latency and size limits → SHAP → fairness → robustness → privacy → model-file scan →
14
+ model card → register → promote → monitor baseline.
15
+ - Runs can be resumed: each step's inputs are hashed, so only changed steps re-run.
16
+ - Promotion waits for four-eyes approvals tied to the exact inputs.
17
+ - Gates stop a run and are never retried away; unexpected errors are retried.
18
+ - Each run records cost per step with a budget stop, an append-only event log, a
19
+ reproducibility manifest and a report.
20
+ - A killed run is taken over and resumed.
21
+ - **Cost-aware AutoML:** with latency or size limits, the engine searches for the best model
22
+ that meets them.
23
+ - **CLI:** `mlpilot plan`, `run`, `approve`, `status`.
24
+
25
+ ### Added: enterprise operations
26
+ - **Data observability and reconciliation** (`quality.DataMonitor`, `quality.reconcile`):
27
+ - freshness, volume, schema, nulls and distribution over time, with SLAs;
28
+ - source-to-destination checks of keys, changed rows, sums and schema.
29
+ - **Incident management** (`monitoring.diagnose`, `IncidentLog`):
30
+ - root causes: upstream data, infrastructure, bad release, concept drift, covariate shift;
31
+ - playbooks where low-risk actions run automatically and high-risk actions (rollback) need
32
+ approval;
33
+ - an incident closes only after a healthy verification.
34
+ - **Backups and disaster recovery** (`mlpilot.backup`, `mlpilot backup | restore | dr-drill`):
35
+ - checksummed archives, consistent snapshots of live SQLite databases;
36
+ - restores that are verified first and refuse path traversal;
37
+ - drills that measure RTO and RPO, plus retention.
38
+ - **Supply chain** (`mlpilot.supply_chain`, `mlpilot scan-model | sbom`):
39
+ - malicious pickle and joblib detection without executing anything, and ONNX checks;
40
+ - CycloneDX 1.5 SBOM.
41
+ - **Serving:** per-client rate limits (`rate_limit=`, `AC_RATE_LIMIT`; `429` with
42
+ `Retry-After`), and request validation against the model's input schema (`validate=`,
43
+ `AC_VALIDATE`).
44
+ - **Model governance:**
45
+ - `ModelRegistry.set_risk()`: unacceptable models are blocked; high-risk ones need two
46
+ approvers and a current review;
47
+ - `retire()`: archives every version and points to the replacement.
48
+ - **Active learning** (`mlpilot.labeling`): uncertainty and diversity selection, a labeling
49
+ queue with votes and agreement, and a simulation against random labeling.
50
+ - **Privacy attacks** (`mlpilot.model_privacy`):
51
+ - membership inference and model extraction tests;
52
+ - an engine gate, `max_membership_auc`.
53
+ - **Hub multi-tenancy:**
54
+ - organisations with quotas (workspaces, model versions, storage, runs per day);
55
+ - cross-tenant membership refused;
56
+ - usage metered per tenant and day;
57
+ - existing databases migrated in place.
58
+ - **CI:** a `certify` job that every release requires. It runs the production-guarantee
59
+ suites and a live sequence (autonomous run, model scan, DR drill, SBOM) and uploads the
60
+ evidence.
61
+ - **Docs:**
62
+ - [Autonomous engine & operations guide](docs/guide/enterprise-operations.md);
63
+ - [enterprise gap audit](docs/enterprise-gap-audit.md);
64
+ - the GitHub-readable API reference is now generated by `scripts/gen_api_ref.py`.
65
+
66
+ ### Fixed
67
+ - `AutoML` on a pure free-text dataset: the cleaning step dropped the text column as an ID
68
+ (every review is unique). Text columns are now kept and embedded.
69
+
70
+ ## [2.2.0] — 2026-11-12
71
+
72
+ ### Added: images taken in hard conditions
73
+ - **Weather and lighting** (`media.conditions`):
74
+ - detects fog, rain, snow, low light, glare, motion blur, sensor noise and JPEG damage
75
+ (each scored 0-1 and tagged at 0.5);
76
+ - `condition_report` shows which conditions a dataset covers and flags the rare or missing ones;
77
+ - `apply_condition` simulates all of them, plus shadow and cutout, at severity 1-5;
78
+ - `enhance` restores images: dehaze, low-light brightening, denoise, sharpen.
79
+ - **Faces** (`media.faces`, `[vision]` extra):
80
+ - detection;
81
+ - how much of a face is covered, and by what (hair, mask, sunglasses or object), in a per-image
82
+ `occlusion_report`;
83
+ - simulated coverings for training;
84
+ - blurring or pixelating for privacy (`anonymize`, `blur_regions`).
85
+ - **Licence plates** (`media.plates`, `[ocr]` extra):
86
+ - detection and OCR, with plate patterns (`IN`, `IN_BH`, `UK`, `EU`, `US`) that fix OCR
87
+ look-alikes;
88
+ - optional restoration before reading;
89
+ - `plate_report`, which pairs each read with the conditions in the image.
90
+ - **Robustness benchmark** (`media.benchmark`):
91
+ - `condition_benchmark` re-scores any image model per condition and severity, with face
92
+ coverings included;
93
+ - `condition_augment` builds training sets from the weakest conditions;
94
+ - ready-made predictors for AutoML image classifiers, plate reading and face counting.
95
+ - New extras: `vision` (OpenCV 4: OpenCV 5 no longer ships the detector models) and `ocr`
96
+ (RapidOCR; its models are bundled, so nothing is downloaded at run time).
97
+ - Image functions also accept numpy arrays, alongside PIL images and paths.
98
+
99
+ ### Added: quantization
100
+ - **Models** (`mlpilot.deploy.quantize`, `[onnx]` extra):
101
+ - `quantize_model` writes int8 and fp16 ONNX variants of a model and measures each on held-out
102
+ data (file size, speed, metric change and agreement with the original), then recommends
103
+ one;
104
+ - `export_model(..., format="onnx", quantize="int8" | "uint8" | "fp16")`, `quantize_onnx`
105
+ for any ONNX model, and a `mlpilot quantize` CLI command;
106
+ - models with nothing to quantize (tree ensembles, linear models, k-NN) are reported as
107
+ such rather than exported unchanged.
108
+ - **AutoML:** a new opt-in `mlp` model family (scikit-learn neural network), used only when
109
+ named in `include=`. It is the AutoML model family that quantization shrinks.
110
+ - **CI/CD:** a `quantization` gate in `gates.yml` (`mode`, `max_drop`, `min_agreement`,
111
+ `required`).
112
+ - **Embeddings:**
113
+ - `TextEmbeddings(dtype="float32" | "float16" | "int8")`;
114
+ - `quantize_vectors` / `dequantize_vectors` with a per-dimension int8 scale learned on
115
+ training data.
116
+ - **Tables:**
117
+ - `optimize_dtypes` and the `OptimizeDtypes` pipeline step shrink column types without
118
+ changing values (integers by range, floats only when they survive the round trip, text
119
+ to Categorical);
120
+ - the step learns the types on training data and never overflows on new data.
121
+
122
+ ### Fixed
123
+ - `save_cleaned_data(quantize=True)` no longer fails on integers above 2.1 billion (it cast
124
+ every `Int64` column to `Int32`).
125
+ - `ClipOutliers` no longer fails on small integer columns (`Int8`, `Int16`, `UInt8`, …)
126
+ whose learned fences lie outside the type's range.
127
+
128
+ ## [2.1.0] — 2026-11-09
129
+
130
+ ### Renamed: AutomatedCleaning is now MLPilot
131
+ - **Names:**
132
+ - the distribution is `dataspoof-mlpilot` (PyPI refused `mlpilot` as too similar to an existing project);
133
+ - the import is `import mlpilot`, and the CLI is `mlpilot`;
134
+ - `MLPilotWarning` replaces `AutomatedCleaningWarning`, which remains as an alias.
135
+ - **Compatibility:**
136
+ - `import automatedcleaning` and pickles that refer to `automatedcleaning.*` classes resolve to
137
+ the same `mlpilot` modules, so models saved by AutomatedCleaning keep loading with the same
138
+ classes;
139
+ - the `AutomatedCleaning` project on PyPI gets a final 2.0.1 that installs `dataspoof-mlpilot`.
140
+ - **Unchanged:** the `AC_*` environment variables, the `ac_*` Prometheus metrics and the saved-file
141
+ formats (`dataspoof.*`).
142
+ - **Renamed with the library:**
143
+ - the Helm chart (`mlpilot-hub`), the OPA policy package (`data.mlpilot`) and the Prometheus job;
144
+ - the Grafana dashboard file;
145
+ - the MLflow tag prefix and the default OpenLineage namespace.
146
+
147
+ ## [2.0.0] — 2026-11-09
148
+
149
+ ### Breaking
150
+ - **Lean core install.** `pip install AutomatedCleaning` now installs only the core
151
+ dependencies. The heavy ones moved to extras:
152
+ - `[nlp]`: nltk
153
+ - `[dashboard]`: plotly, matplotlib, missingno, pyfiglet
154
+ - `[pii]`: presidio
155
+ - `[llm]`: anthropic, langchain-anthropic
156
+
157
+ To get what 1.x installed, use `pip install "AutomatedCleaning[all]"`. A feature whose extra
158
+ is missing raises an `ImportError` naming the extra to install.
159
+ - `setup.py` was removed. The build is defined only in `pyproject.toml` (maturin).
160
+
161
+ ### Changed
162
+ - **No side effects at import.**
163
+ - Importing the library no longer downloads NLTK data, changes the application's warning
164
+ filters, or reconfigures logging.
165
+ - NLTK corpora are downloaded on first use, and never when `AC_OFFLINE` is set. Text cleaning
166
+ falls back to a built-in stopword list when NLTK isn't installed.
167
+ - **Typed package.** It ships a `py.typed` marker and stubs for the Rust extension, and passes
168
+ `mypy`.
169
+ - **CI** now runs:
170
+ - tests across Linux, macOS and Windows × Python 3.11–3.13 × Polars 1/2;
171
+ - a core-only install test;
172
+ - mypy;
173
+ - pip-audit and cargo-audit.
174
+
175
+ Releases publish with PyPI Trusted Publishing, signed build-provenance attestations and a
176
+ CycloneDX SBOM.
177
+ - Interactive AI corrections use `ANTHROPIC_API_KEY` when it is set, and the current Claude model.
178
+
179
+ ### Added
180
+ - **Hub, the multi-user platform server** (`automatedcleaning.hub`, `[server]` extra):
181
+ - workspaces, with SSO through OIDC and identity-provider group mapping;
182
+ - API tokens and service accounts;
183
+ - four roles (viewer, data scientist, approver, admin);
184
+ - runs and a model registry with immutable, checksummed artifacts on disk, S3, GCS or Azure;
185
+ - two-person production approvals with policy gates;
186
+ - monitoring results;
187
+ - a hash-chained, verifiable and exportable audit log;
188
+ - a web UI, the Python client `HubClient`, and `automatedcleaning hub serve | create-admin`.
189
+ - **Hub deployment:** a Docker image with compose (plus PostgreSQL), a Helm chart, and a
190
+ Terraform module for AWS (S3, RDS, Secrets Manager, IAM role, Helm release).
191
+ - **Forecasting** (`Forecaster`):
192
+ - multiple series and multi-step horizons;
193
+ - rolling-origin backtests against seasonal-naive;
194
+ - conformal intervals and hierarchical reconciliation.
195
+ - **Causal inference** (`estimate_effect`): doubly-robust AIPW with overlap and balance
196
+ diagnostics. **Uplift:** `UpliftModel` and `qini`.
197
+ - **Anomaly detection** (`AutoAnomaly`): an ensemble of three detectors with plain-language
198
+ explanations.
199
+ - **AutoML:**
200
+ - conformal `predict_interval` and `predict_set`;
201
+ - `search="optuna"`, `imbalance="auto"`, `n_jobs`;
202
+ - `predict_with_proba` and `set_inference_threads`.
203
+ - **Serving:**
204
+ - Prometheus `/metrics` and OpenTelemetry tracing;
205
+ - canary and shadow rollouts, and `app_from_registry`;
206
+ - SageMaker, Vertex AI and Azure ML routes and Docker targets;
207
+ - an ASGI app and the `container` entry point;
208
+ - single-row latency down from about 500 ms to about 50 ms.
209
+ - **Security:**
210
+ - Ed25519 model signing, verified before unpickling whenever `AC_MODEL_PUBLIC_KEY` is set;
211
+ - `skops` export;
212
+ - signing on registration with `register(..., sign_with=...)`.
213
+ - **Monitoring:**
214
+ - PagerDuty and Opsgenie alert handlers;
215
+ - `join_labels` and `Monitor.check_labels` for labels that arrive later;
216
+ - `sql_sink` and `Monitor.on_check`;
217
+ - `SQLSource.insert` and `execute`.
218
+ - **Scale:**
219
+ - `transform_file` for files of any size;
220
+ - Spark, Ray and Dask scoring;
221
+ - `to_sql` SQL pushdown of fitted cleaning pipelines.
222
+ - **Integrations:** Airflow operators, Prefect tasks and flow, Dagster assets, Kafka stream
223
+ scoring, and plain `integrations.tasks`.
224
+ - **MLflow** (`integrations.mlflow`, `[mlflow]` extra):
225
+ - batched, idempotent export of runs and whole trackers;
226
+ - AutoML models logged as pyfunc models with a signature, so they serve with `mlflow models serve`;
227
+ - `export_registry` (stages become aliases) and `load_automl`.
228
+ - **DVC** (`integrations.dvc`, `[dvc]` extra): `DVCRepo` (init, remotes, track, push and pull,
229
+ read any revision, checkout, `log_to_run`) and `export_repository` for `DataRepository`
230
+ history.
231
+ - **Feast** (`integrations.feast`, `[feast]` extra): `export_to_feast` writes, applies and
232
+ materializes a Feast repository.
233
+ - **Data splitting** (`split_data`, `cv_folds`, `check_split`): stratified, grouped and time-ordered
234
+ splits with gaps; every split is leak-checked (shared groups, time overlap, duplicate rows, unseen classes).
235
+ - **Model security:** `robustness_report` (noise stability, a minimal adversarial attack,
236
+ feature sensitivity, extreme inputs) and `poisoning_report` (confident-learning label errors,
237
+ conflicting duplicates, backdoor triggers, outliers).
238
+ - **Environments** (`Environments`, `environments.yml`): isolated dev / QA / prod registries;
239
+ `release` enforces order, source stage, metric gates, four-eyes approvals and signatures, and
240
+ records provenance on both sides. The registry and Hub gain a `qa` stage.
241
+ - **Model CI/CD:** the CLI commands `train`, `gate`, `register`, `promote`, `release`, `latest`,
242
+ `deploy-sagemaker` and `cicd init` (GitHub Actions / GitLab CI pipelines, checked with actionlint).
243
+ - **Secrets** (`secretstore`): `vault://` (token, AppRole, Kubernetes; KV v1 and v2), `aws-sm://`,
244
+ `env://` and `file://` references, resolved wherever credentials are read.
245
+ - **Open Policy Agent for the Hub** (`hub.policy`): deny-overrides access rules, promotion rules
246
+ and fail-closed behaviour, plus Rego policies with unit tests in `deploy/opa`.
247
+ - **Serving:**
248
+ - `POST /feedback` with prediction IDs, live-accuracy metrics and `join_feedback`;
249
+ - hot reload (`watch=`, `POST /admin/reload`, registry-following);
250
+ - live drift, estimated-performance and alert gauges (`monitor=`).
251
+ - **Monitoring before labels:** `PerformanceEstimator` (CBPE, direct loss estimation),
252
+ `domain_classifier_drift`, and `estimated_performance` alerts in `Monitor.check`.
253
+ - **Grafana + Prometheus** (`deploy/monitoring`): a dashboard, 14 alert rules with `promtool`
254
+ unit tests, and a compose stack.
255
+ - **Lineage** (`lineage`): OpenLineage run events with schema, quality and version facets; HTTP
256
+ and file transports; `emit_tracked_run`, `emit_dataset_version`, `LineageGraph`.
257
+ - **Catalogs and orchestration:** AWS Glue (catalog tables, partitions, reads, job runs) and Azure
258
+ Data Factory pipeline runs.
259
+ - **AWS deployment** (`deploy.aws`): ECR push and blue/green SageMaker endpoints (serverless or
260
+ instances, autoscaling, data capture, rollback alarms); Terraform for SageMaker and ECS Fargate.
261
+ - **Evidently** (`integrations.evidently`): reports plus agreement with the built-in drift report.
262
+ - **Tests against real infrastructure:** Prometheus, Grafana, Vault, OPA and Marquez in Docker
263
+ (`AC_DOCKER_TESTS=1`); moto for AWS; dedicated CI jobs.
264
+ - These three are tested against the real tools (MLflow server and model server, git + DVC
265
+ with a remote, Feast), including in a dedicated CI job.
266
+ - **Image and audio data** (`media`):
267
+ - quality audits (corrupt files, exact and near duplicates, blur, exposure, clipping, silence,
268
+ mixed formats);
269
+ - preprocessors (EXIF-aware image resize and normalise; audio resample, trim and level);
270
+ - augmentation;
271
+ - features for AutoML (colour and edge-orientation histograms; MFCC and spectral features).
272
+ - **Statistical testing** (`stats`):
273
+ - automatic test selection with assumption checks, effect sizes and confidence intervals;
274
+ - chi-square and Fisher's exact test, correlations;
275
+ - A/B tests, power analysis, multiple-testing correction, `test_all`.
276
+ - **Data version control** (`DataRepository`): content-addressed commits, tags, row-level
277
+ diffs, checksum-verified checkout, links to experiment runs, garbage collection.
278
+ - **Feature store** (`FeatureStore`, `Entity`, `FeatureView`): point-in-time-correct
279
+ historical joins with TTL, an SQLite online store, `materialize`, schema guards.
280
+ - **Documentation site** (MkDocs Material with an API reference), plus `SECURITY.md`,
281
+ `CONTRIBUTING.md` and a versioning policy.
282
+
283
+ ### Removed
284
+ - `automatedcleaning/cleaning_update.py`: dead code that nothing imported.
285
+
286
+ ## [1.14.0] — 2026-11-08
287
+ - **Ingestion & connectors:**
288
+ - files, globs, HTTP, and S3 / GCS / Azure;
289
+ - any SQL database (bound parameters);
290
+ - Snowflake and BigQuery;
291
+ - REST APIs with five pagination styles;
292
+ - reusable schema inference and incremental loads.
293
+ - **Experiment tracking & model registry:** runs, dataset fingerprints, dev → staging →
294
+ production promotion with metric gates, audit history and rollback.
295
+ - **Deployment:**
296
+ - batch scoring and a REST API (API keys, prediction log);
297
+ - bundle, pickle and ONNX export;
298
+ - Docker build contexts and a CLI.
299
+ - **Monitoring:** data and prediction drift (PSI), accuracy decay, alerts and webhooks, and
300
+ champion-vs-challenger retraining.
301
+ - **Insights & reporting:** significance-tested findings, executive reports, and optional
302
+ narratives written by Claude.
303
+
304
+ ## [1.13.0] — 2026-11-06
305
+ - **Feature engineering:**
306
+ - dates, out-of-fold target encoding, interactions, aggregations, lags, text embeddings;
307
+ - feature selection and leakage detection.
308
+ - **AutoML:** task detection, time-budgeted model search, an ensemble, and AutoCluster.
309
+ - **Evaluation & explainability:** metrics with confidence intervals, built-in SHAP, error
310
+ analysis, fairness checks and model cards.
311
+
312
+ ## [1.12.0] — 2026-11-06
313
+ - Profiling & data quality: automated EDA, the 0–100 quality score, YAML data contracts, schema
314
+ diffs and quality gates.
315
+
316
+ ## [1.11.0] — 2026-11-05
317
+ - Shared `Step` / `Pipeline` core and the stateful cleaning pipeline (JSON save/load, audit
318
+ reports, schema checks).
319
+
320
+ ## [1.10.0] and earlier — 2026-07
321
+ - 1.10.0: 8 more data-quality checks in the dashboard.
322
+ - 1.9.0: column pickers in the dashboard.
323
+ - 1.8.x: the dashboard data-quality panel.
324
+ - 1.7.0: the enterprise dashboard redesign.
325
+ - 1.6.x: 43 additional cleaning steps and a dependency cleanup.
326
+ - 1.5.x: the Rust compute backend.
@@ -6,8 +6,8 @@ Thanks for helping! The basics:
6
6
 
7
7
  ```bash
8
8
  # needs the Rust toolchain: https://rustup.rs
9
- git clone https://github.com/DataSpoof/dataspoof-data-cleaning.git
10
- cd dataspoof-data-cleaning
9
+ git clone https://github.com/DataSpoof/dataspoof-mlpilot.git
10
+ cd dataspoof-mlpilot
11
11
  python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
12
12
  pip install maturin
13
13
  maturin develop --release --extras dev # builds the Rust extension in place