mlspec 0.12.6__tar.gz

mlspec-0.12.6/LICENSE

@@ -0,0 +1,92 @@
+Copyright (c) 2026 Lior Cohen
+
+Elastic License 2.0
+
+URL: https://www.elastic.co/licensing/elastic-license
+
+## Acceptance
+
+By using the software, you agree to all of the terms and conditions below.
+
+## Copyright License
+
+The licensor grants you a non-exclusive, royalty-free, worldwide,
+non-sublicensable, non-transferable license to use, copy, distribute, make
+available, and prepare derivative works of the software, in each case subject
+to the limitations and conditions below.
+
+## Limitations
+
+You may not provide the software to third parties as a hosted or managed
+service, where the service provides users with access to any substantial set
+of the features or functionality of the software.
+
+You may not move, change, disable, or circumvent the license key functionality
+in the software, and you may not remove or obscure any functionality in the
+software that is protected by the license key.
+
+You may not alter, remove, or obscure any licensing, copyright, or other
+notices of the licensor in the software. Any use of the licensor's trademarks
+is subject to applicable law.
+
+## Patents
+
+The licensor grants you a license, under any patent claims the licensor can
+license, or becomes able to license, to make, have made, use, practice, offer
+for sale, import and otherwise transfer the software. This license does not
+cover any patent claims that you cause to be infringed by modifications or
+additions to the software. If you or your company make any written claim that
+the software infringes or contributes to infringement of any patent, your
+patent license for the software ends immediately.
+
+## Notices
+
+You must ensure that anyone who gets a copy of any part of the software from
+you also gets a copy of these terms.
+
+If you modify the software, you must include in any modified copies of the
+software a prominent notice stating that you have modified the software.
+
+## No Other Rights
+
+These terms do not imply any licenses other than those expressly granted in
+these terms.
+
+## Termination
+
+If you use the software in violation of these terms, such use is not licensed,
+and your licenses will automatically terminate. If the licensor provides you
+with a notice of your violation, and you cease all violation of this license
+no later than 30 days after you receive that notice, your licenses will be
+reinstated retroactively. However, if you violate these terms after such
+reinstatement, any additional violation of these terms will cause your licenses
+to terminate automatically and permanently.
+
+## No Liability
+
+*As far as the law allows, the software comes as is, without any warranty or
+condition, and the licensor will not be liable to you for any damages arising
+out of these terms or the use or nature of the software, under any kind of
+legal theory, including tort (including negligence), contract, warranty,
+statutory, or otherwise.*
+
+## Definitions
+
+The **licensor** is the entity offering these terms, and the **software** is
+the software the licensor makes available under these terms, including any
+portion of it.
+
+**you** refers to the individual or entity agreeing to these terms.
+
+**your company** is any legal entity, sole proprietorship, or other kind of
+organization that you work for, plus all organizations that have control over,
+are under the control of, or are under common control with that organization.
+**control** means ownership of substantially all the assets of an entity, or
+the power to direct its management and policies by vote, majority ownership,
+or otherwise. Control can be direct or indirect.
+
+**your licenses** are all the licenses granted to you for the software under
+these terms.
+
+**use** includes copying, distributing, making available, modifying, and
+preparing derivative works.

mlspec-0.12.6/LICENSE-COMMERCIAL

@@ -0,0 +1,39 @@
+# Commercial License — mltk
+
+mltk is licensed under the [Elastic License 2.0](LICENSE) (ELv2), which
+permits free use for internal, non-commercial, and evaluation purposes.
+
+## When to get in touch
+
+The ELv2 community license covers most uses. You should reach out about a
+commercial license when your use case resembles any of the following:
+
+- **Managed / hosted service**: You offer mltk (or a product substantially
+  built on mltk) as a service to third parties — e.g. an MLOps platform,
+  a testing-as-a-service product, or a SaaS tool where mltk functionality
+  is accessible to your customers. This is the primary restriction in ELv2.
+
+- **Bundled commercial product**: You embed or redistribute mltk as part of
+  a product you sell or license to third parties.
+
+- **Commercial training or consulting**: You deliver paid engagements where
+  mltk is a central deliverable.
+
+If you're unsure, reach out — we're happy to clarify before you deploy.
+
+Internal use within your company — including running mltk against your own
+models, pipelines, and datasets, regardless of company size or revenue — is
+free under ELv2 with no conversation required.
+
+## How to obtain a commercial license
+
+Contact: **lior1cc@gmail.com**
+
+Please include a brief description of your use case. Commercial licenses are
+negotiated individually and may be structured as a flat fee, revenue share,
+or other arrangement depending on the use case.
+
+## Questions
+
+If you are unsure whether your use case requires a commercial license, reach
+out before deploying. We are happy to clarify.

mlspec-0.12.6/PKG-INFO

@@ -0,0 +1,439 @@
+Metadata-Version: 2.4
+Name: mlspec
+Version: 0.12.6
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Rust
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Classifier: Framework :: Pytest
+Requires-Dist: numpy>=1.24
+Requires-Dist: pandas>=2.0
+Requires-Dist: mlspec[scipy,sklearn,cli,report,cv,nlp,speech,contracts,integrations,pdf,llm,embedding,nli,ner,mcp,multimodal,clip,classifier,torch,mlflow,otel,server,aws,gcp,azure,container,metrics] ; extra == 'all'
+Requires-Dist: boto3>=1.28 ; extra == 'aws'
+Requires-Dist: azure-ai-ml>=1.12 ; extra == 'azure'
+Requires-Dist: transformers>=4.40 ; extra == 'classifier'
+Requires-Dist: torch>=2.0 ; extra == 'classifier'
+Requires-Dist: typer>=0.9 ; extra == 'cli'
+Requires-Dist: rich>=13 ; extra == 'cli'
+Requires-Dist: open-clip-torch>=2.24 ; extra == 'clip'
+Requires-Dist: pillow>=9.0 ; extra == 'clip'
+Requires-Dist: trivy-py>=0.70 ; extra == 'container'
+Requires-Dist: pyyaml>=6.0 ; extra == 'contracts'
+Requires-Dist: opencv-python>=4.8 ; extra == 'cv'
+Requires-Dist: pytest>=8.0 ; extra == 'dev'
+Requires-Dist: pytest-cov>=4.0 ; extra == 'dev'
+Requires-Dist: pytest-xdist>=3.5 ; extra == 'dev'
+Requires-Dist: pytest-randomly>=3.15 ; extra == 'dev'
+Requires-Dist: hypothesis>=6.100 ; extra == 'dev'
+Requires-Dist: syrupy>=4.0 ; extra == 'dev'
+Requires-Dist: ruff>=0.3 ; extra == 'dev'
+Requires-Dist: mypy>=1.8 ; extra == 'dev'
+Requires-Dist: pre-commit>=3.5 ; extra == 'dev'
+Requires-Dist: mkdocs-material>=9.5 ; extra == 'docs'
+Requires-Dist: mkdocstrings[python]>=0.24 ; extra == 'docs'
+Requires-Dist: sentence-transformers>=2.0 ; extra == 'embedding'
+Requires-Dist: google-cloud-aiplatform>=1.40 ; extra == 'gcp'
+Requires-Dist: jira>=3.0 ; extra == 'integrations'
+Requires-Dist: langfuse>=2.0 ; extra == 'langfuse'
+Requires-Dist: jsonschema>=4.0 ; extra == 'mcp'
+Requires-Dist: prometheus-client>=0.20 ; extra == 'metrics'
+Requires-Dist: mlflow>=2.0 ; extra == 'mlflow'
+Requires-Dist: pillow>=9.0 ; extra == 'multimodal'
+Requires-Dist: scikit-image>=0.20 ; extra == 'multimodal'
+Requires-Dist: presidio-analyzer>=2.2 ; extra == 'ner'
+Requires-Dist: spacy>=3.4.4 ; extra == 'ner'
+Requires-Dist: sentence-transformers>=2.0 ; extra == 'nli'
+Requires-Dist: nltk>=3.8 ; extra == 'nlp'
+Requires-Dist: rouge-score>=0.1 ; extra == 'nlp'
+Requires-Dist: opentelemetry-api>=1.20 ; extra == 'otel'
+Requires-Dist: opentelemetry-sdk>=1.20 ; extra == 'otel'
+Requires-Dist: opentelemetry-exporter-otlp>=1.20 ; extra == 'otel'
+Requires-Dist: weasyprint>=60 ; extra == 'pdf'
+Requires-Dist: arize-phoenix>=4.0 ; extra == 'phoenix'
+Requires-Dist: opentelemetry-api>=1.20 ; extra == 'phoenix'
+Requires-Dist: opentelemetry-sdk>=1.20 ; extra == 'phoenix'
+Requires-Dist: plotly>=5.18 ; extra == 'report'
+Requires-Dist: jinja2>=3.1 ; extra == 'report'
+Requires-Dist: scipy>=1.11 ; extra == 'scipy'
+Requires-Dist: fastapi>=0.110 ; extra == 'server'
+Requires-Dist: uvicorn>=0.27 ; extra == 'server'
+Requires-Dist: scikit-learn>=1.3 ; extra == 'sklearn'
+Requires-Dist: jiwer>=3.0 ; extra == 'speech'
+Requires-Dist: torch>=2.0 ; extra == 'torch'
+Provides-Extra: all
+Provides-Extra: aws
+Provides-Extra: azure
+Provides-Extra: classifier
+Provides-Extra: cli
+Provides-Extra: clip
+Provides-Extra: container
+Provides-Extra: contracts
+Provides-Extra: cv
+Provides-Extra: dev
+Provides-Extra: docs
+Provides-Extra: embedding
+Provides-Extra: gcp
+Provides-Extra: integrations
+Provides-Extra: langfuse
+Provides-Extra: llm
+Provides-Extra: mcp
+Provides-Extra: metrics
+Provides-Extra: mlflow
+Provides-Extra: multimodal
+Provides-Extra: ner
+Provides-Extra: nli
+Provides-Extra: nlp
+Provides-Extra: otel
+Provides-Extra: pdf
+Provides-Extra: phoenix
+Provides-Extra: report
+Provides-Extra: scipy
+Provides-Extra: server
+Provides-Extra: sklearn
+Provides-Extra: speech
+Provides-Extra: torch
+License-File: LICENSE
+License-File: LICENSE-COMMERCIAL
+Summary: pytest for ML — unified testing across the entire ML lifecycle
+Keywords: ml,testing,pytest,data-quality,drift-detection,mlops
+Author: Lior Cohen
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
+Project-URL: Documentation, https://liorrr.github.io/mltk
+Project-URL: Homepage, https://github.com/Liorrr/mltk
+Project-URL: Issues, https://github.com/Liorrr/mltk/issues
+Project-URL: Repository, https://github.com/Liorrr/mltk
+
+# mltk - ML Test Kit
+
+**pytest for ML** -- unified testing across the entire ML lifecycle.
+
+[![License](https://img.shields.io/badge/license-Elastic%202.0-blue.svg)](LICENSE)
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://python.org)
+[![Tests](https://img.shields.io/badge/tests-4291%2B%20passed-green.svg)]()
+[![Rust](https://img.shields.io/badge/rust-accelerated-orange.svg)]()
+
+<!-- TEMP-PYPI-CLAIM: remove this block once mltk PyPI name is resolved -->
+> **Installation note (temporary):** `pip install mltk` is not yet available — the PyPI name is pending a transfer claim. Until resolved, install directly from GitHub:
+> ```bash
+> pip install git+https://github.com/Liorrr/mltk
+> ```
+> All imports (`import mltk`), CLI commands (`mltk scan`), and functionality are identical.
+<!-- END TEMP-PYPI-CLAIM -->
+
+## Install
+
+**Homebrew (macOS/Linux):**
+```bash
+brew tap Liorrr/mltk
+brew install mltk
+```
+
+**pip:**
+```bash
+pip install mlspec  # PyPI name — 'mltk' pending transfer
+```
+
+## Why mltk?
+
+ML systems fail silently. A model can train on corrupt data, produce confident predictions from stale features, and pass every unit test while being completely wrong in production. Traditional testing does not catch these failures.
+
+**mltk** gives you **232 assertions** covering the entire ML lifecycle -- data quality, model validation, drift detection, fairness testing, inference benchmarking, training bug detection, LLM evaluation, behavioral consistency, NER-based PII detection, red team security, multimodal evaluation, observability, and production monitoring. One toolkit, one `pip install`, native pytest integration. No more gluing together 5 different tools.
+
+## Quick Start
+
+```python
+import pandas as pd
+import pytest
+from mltk.data import (
+    assert_schema, assert_no_nulls, assert_no_pii,
+)
+from mltk.model import (
+    assert_metric, assert_no_regression, assert_no_bias,
+)
+from mltk.inference import assert_latency, assert_throughput
+
+@pytest.mark.ml_data
+def test_training_data():
+    df = pd.read_csv("data/training.csv")
+    assert_schema(
+        df,
+        {"id": "int64", "text": "object", "label": "int64"},
+    )
+    assert_no_nulls(df, columns=["label"])
+    assert_no_pii(df, columns=["text"])
+
+@pytest.mark.ml_model
+def test_model_quality(y_true, y_pred):
+    assert_metric(
+        y_true, y_pred, metric="f1", threshold=0.85,
+    )
+    assert_no_regression(
+        y_true, y_pred, baseline=0.90, tolerance=0.02,
+    )
+    assert_no_bias(
+        y_true, y_pred,
+        sensitive_feature=gender,
+        method="demographic_parity",
+    )
+
+@pytest.mark.ml_inference
+def test_inference_performance():
+    assert_latency(model.predict, X_test, p95=50.0)
+    assert_throughput(model.predict, X_single, min_rps=100)
+```
+
+Run with HTML report:
+```bash
+pytest --mltk-report
+```
+
+## What's Included (v0.12.4)
+
+**Behavioral consistency testing** — 7 assertions (paraphrase invariance, format invariance, output stability, semantic equivalence, directional expectation, retrieval consistency, ParaphraseGenerator) that catch models memorizing phrasing instead of learning concepts. Multi-method evaluation: lexical (token F1), embedding, NLI, LLM-as-Judge. No other ML testing tool ships these as pytest assertions.
+
+**NER PII detection** — `assert_no_pii(method="ner"|"gliner"|"hybrid")` adds Named Entity Recognition to PII scanning. Presidio + spaCy catches names, organizations, and locations that regex cannot find. GLiNER provides zero-shot detection for domain-specific entities (healthcare MRN, legal case numbers). Hybrid mode runs regex + NER with intelligent deduplication. Install with `pip install mltk[ner]`.
+
+**YAML test definitions** — write ML tests in YAML, no Python required. Run with `mltk test tests.yaml`. Supports 11 data assertions with `env:VAR_NAME` data source for CI/CD.
+
+**EU AI Act compliance reports** — `mltk compliance results.json` auto-maps test results to EU AI Act articles, classifies risk level (unacceptable/high/limited/minimal), and generates an HTML evidence table with gap analysis.
+
+**`mltk doctor`** — 9 diagnostic checks that inspect your Python version, dependencies, config, Rust extension, and pytest plugin, with actionable fix hints for each failure.
+
+**Data contracts** — define column types, ranges, and uniqueness rules in YAML. Use `mltk contract generate-tests` to auto-generate a pytest file from the contract — zero boilerplate.
+
+**LLM evaluation** — `assert_semantic_similarity`, `assert_no_toxicity`, `assert_no_hallucination`, `assert_ttft`, `assert_itl`, RAG evaluation (faithfulness, context precision/recall, RAGAS composite), agentic testing (task completion, tool selection), multi-turn conversation, coherence, BERTScore, and text quality assertions for GenAI systems.
+
+**Training bug detection** — catch data leakage (P0), gradient failures (P1), numerical instability, plus P2 bugs: augmentation corruption, checkpoint integrity, distributed training sync, memory leaks, and training-serving skew detection.
+
+**Server platform** — `mltk server` starts a self-hosted result-tracking server with REST API, live dashboard, API key auth, webhooks, run comparison, and GitHub CI integration (PR comments + check runs).
+
+**Cloud monitoring** — AWS SageMaker/CloudWatch, GCP Vertex AI, Azure ML, and Prometheus/Triton adapters for production endpoint health, latency, and error rate assertions.
+
+**FDA audit trail** — `mltk fda-audit` generates a device-grade audit trail from test results for regulatory submissions.
+
+**Compliance PDF** — `mltk compliance-pdf` exports any HTML compliance report (EU AI Act, OWASP) to PDF for auditors.
+
+**Model Card Generator** — `mltk model-card results.json` generates a Google-format Model Card in Markdown from test results.
+
+**Chat interface** — `mltk chat` provides interactive Q&A about test results with no LLM or external API required.
+
+**Environment variable config** — all config keys available as `MLTK_*` env vars, highest priority in the cascade. Works out of the box in CI/CD pipelines.
+
+**JSON export** — `--mltk-export-json` flag exports full test results to JSON for downstream tooling.
+
+## Feature Matrix (232 assertions)
+
+| Module | Assertions | Purpose |
+|--------|-----------|---------|
+| **Data Schema** | `assert_schema`, `assert_no_nulls`, `assert_dtypes` | Validate DataFrame structure |
+| **Data Distribution** | `assert_range`, `assert_unique`, `assert_no_outliers` | Verify statistical properties |
+| **Data Statistics** | `assert_column_mean`, `assert_column_median`, `assert_column_stdev`, `assert_quantiles` | Statistical bounds checking |
+| **Data Validation** | `assert_datetime_format`, `assert_values_in_set`, `assert_no_conflicting_labels`, `assert_feature_label_correlation_stable` | Value-level data checks |
+| **Data Drift** | `assert_no_drift` (KS, PSI, KL, Chi2) | Detect distribution shifts |
+| **Data Drift (Advanced)** | `assert_no_embedding_drift` + 7 methods (KS, PSI, KL, Chi2, JS, Wasserstein, auto) | Embedding + tabular drift |
+| **Data PII** | `assert_no_pii`, `scan_pii` (40+ regex patterns + Presidio NER + GLiNER zero-shot + hybrid) | Find leaked personal data |
+| **Data Freshness** | `assert_freshness`, `assert_row_count` | Verify data recency and size |
+| **Data Labels** | `assert_label_balance`, `assert_label_coverage` | Validate label quality |
+| **Data Contracts** | `validate_data`, `generate_tests_from_contract` | YAML → auto-test generation |
+| **Data Quality Preset** | `assert_data_quality`, `data_quality_report` | One-call full data audit |
+| **Data Lineage** | `assert_lineage_complete` | Track data provenance |
+| **Model Metrics** | `assert_metric` (9 metrics: accuracy, F1, AUC, MSE, R2...) | Quality gates |
+| **Model Regression** | `assert_no_regression`, `save_baseline` | Catch silent degradation |
+| **Model Slicing** | `assert_slice_performance`, `assert_calibration` | Subgroup fairness + ECE |
+| **Model Bias** | `assert_no_bias` (5 methods) | EU AI Act / four-fifths rule |
+| **Model Adversarial** | `assert_robust` | Perturbation stability |
+| **Model Overfitting** | `assert_no_overfitting`, `assert_label_drift` | Training vs. validation gaps |
+| **Training Bugs (P0)** | `assert_no_train_test_overlap`, `assert_temporal_split`, `assert_no_target_leakage` | Data leakage |
+| **Training Bugs (P1)** | `assert_gradient_flow`, `assert_no_vanishing_gradient`, `assert_no_exploding_gradient`, `assert_loss_finite` | Gradient health |
+| **Training Numerical** | `assert_no_nan_inf`, `assert_loss_decreasing`, `assert_no_loss_divergence`, `assert_softmax_valid` | Numerical stability |
+| **Training Bugs (P2)** | `assert_augmentation_preserves_signal`, `assert_no_augmentation_on_test`, `assert_checkpoint_complete`, `assert_resume_loss_continuous`, `assert_effective_batch_size`, `assert_gradient_sync`, `assert_loss_is_detached`, `assert_no_memory_leak` | Augmentation, checkpoint, distributed, memory |
+| **Training Skew** | `assert_no_training_serving_skew` | Training-serving distribution mismatch |
+| **Inference** | `assert_latency`, `assert_cold_start`, `assert_throughput`, `assert_api_contract` | Performance SLAs |
+| **Pipeline** | `assert_reproducible`, `assert_checksum`, `assert_pipeline` | Determinism + E2E |
+| **Monitor** | `assert_no_degradation`, `assert_sla`, `assert_no_output_drift` | Production health |
+| **Cloud Monitor** | AWS (`assert_endpoint_healthy/latency/error_rate`), GCP (`assert_endpoint_healthy`, `assert_prediction_latency`), Azure (`assert_endpoint_healthy/latency`), Prometheus/Triton | Multi-cloud monitoring |
+| **LLM Evaluation** | `assert_semantic_similarity`, `assert_no_toxicity`, `assert_no_hallucination`, `assert_ttft`, `assert_itl` | GenAI testing |
+| **LLM RAG** | `assert_faithfulness`, `assert_context_relevancy`, `assert_context_precision`, `assert_context_recall`, `assert_answer_relevancy`, `assert_ragas_score` | RAG pipeline evaluation (RAGAS) |
+| **LLM Agentic** | `assert_task_completion`, `assert_tool_selection`, `assert_tool_call_correctness` | Agent/tool-use testing |
+| **LLM Text Quality** | `assert_text_length`, `assert_output_format`, `assert_readability` | Output quality gates |
+| **LLM Multi-turn** | `assert_knowledge_retention`, `assert_turn_relevancy`, `assert_conversation_completeness` | Conversation evaluation |
+| **LLM Coherence** | `assert_coherence` | Logical consistency |
+| **LLM BERTScore** | `assert_bertscore` (Rust-accelerated token matching) | Semantic similarity via BERT |
+| **NLP** | `assert_bleu`, `assert_rouge`, `assert_ner_f1`, `assert_no_prompt_injection` | Text + security |
+| **NLP Sentiment** | `assert_sentiment_positive`, `assert_no_sentiment_drift` | Sentiment analysis + drift |
+| **Speech** | `assert_wer`, `assert_cer`, `assert_rtf`, `assert_accent_coverage` | Recognition + fairness |
+| **CV** | `assert_iou`, `assert_map`, `assert_frame_accuracy`, `assert_temporal_consistency`, `assert_topk_accuracy` | Video analytics |
+| **CV Tracking** | `assert_mota`, `assert_motp`, `assert_idf1` | Multi-object tracking (CLEAR-MOT) |
+| **CV Face** | `assert_face_far` | False accept rate (NIST FRVT) |
+| **Tabular** | `assert_feature_drift`, `assert_feature_importance_stable`, `assert_class_balance` | Feature validation |
+| **Compliance** | EU AI Act report, OWASP LLM Top 10, FDA audit trail, compliance PDF export | Regulatory evidence |
+| **Retrieval** | `assert_ndcg`, `assert_mrr`, `assert_recall_at_k`, `assert_map_at_k` | Ranking metrics (RAG) |
+| **LLM-as-Judge** | `assert_llm_judge_score`, `assert_llm_judge_pairwise` | Vendor-neutral LLM eval |
+| **Summarization** | `assert_summary_coverage`, `assert_summary_compression`, `assert_summary_faithfulness` | Summary quality |
+| **Recommendation** | `assert_hit_rate`, `assert_diversity`, `assert_novelty`, `assert_coverage`, `assert_serendipity` | RecSys testing |
+| **Long-Context LLM** | `assert_needle_in_haystack`, `assert_context_utilization`, `assert_no_lost_in_middle` | Long-context eval |
+| **Composable Suite** | `MltkSuite`, `SuiteResult`, export to JSON/HTML/JUnit | Run without pytest |
+| **Code Generation** | `assert_code_executes`, `assert_code_passes_tests`, `assert_no_code_vulnerabilities`, `assert_code_complexity` | LLM codegen testing |
+| **Healthcare** | HIPAA compliance mapping, `assert_hipaa_coverage` | Regulated industry |
+| **Finance** | SR 11-7 model risk, `assert_sr117_coverage` | Bank model governance |
+| **Enterprise** | RBAC, audit log, custom compliance frameworks | SOC 2 + governance |
+| **Observability** | `TestScheduler`, anomaly detection, impact analysis, Grafana dashboards | Production monitoring |
+| **Testing Patterns** | `assert_matches_golden`, flaky detection, retry, test selection | Advanced test strategies |
+| **Reports** | Visual diff, test history summarizer, bias report, model card | Analysis + reporting |
+| **Integrations** | Jira, MLflow, GitHub App, W&B, DVC, Kubeflow, SageMaker, Slack, webhooks | Ecosystem connectors |
+
+## Installation
+
+```bash
+# Core (data + model assertions)
+pip install mltk
+
+# With CLI
+pip install mltk[cli]
+
+# With HTML reports
+pip install mltk[report]
+
+# With server platform
+pip install mltk[server]
+
+# NER PII detection
+pip install mltk[ner]         # Presidio + spaCy NER
+
+# Domain kits
+pip install mltk[cv]          # Computer Vision
+pip install mltk[nlp]         # NLP
+pip install mltk[speech]      # Speech Recognition
+pip install mltk[contracts]   # Data contracts (YAML)
+pip install mltk[torch]       # Gradient inspection
+pip install mltk[llm]         # LLM evaluation
+
+# Integrations
+pip install mltk[integrations]  # Jira adapter
+pip install mltk[mlflow]        # MLflow logging
+pip install mltk[aws]           # AWS SageMaker/CloudWatch
+pip install mltk[gcp]           # GCP Vertex AI
+pip install mltk[azure]         # Azure ML monitoring
+pip install mltk[pdf]           # Compliance PDF export
+pip install mltk[server]        # Self-hosted server platform
+
+# Everything
+pip install mltk[all]
+```
+
+## CLI
+
+```bash
+# Core
+mltk version                              # Show version
+mltk init                                 # Scaffold config + tests
+mltk scan data.csv                        # Quick data quality scan
+mltk drift ref.csv cur.csv                # Drift analysis
+mltk score                                # ML Test Score
+mltk doctor                               # Diagnose environment
+
+# Test execution & reporting
+mltk test tests.yaml                      # Run YAML-defined tests
+mltk model-card results.json              # Generate Google Model Card
+mltk compliance results.json              # EU AI Act compliance report
+mltk fda-audit results.json              # FDA device audit trail export
+mltk compliance-pdf report.html          # Export compliance report to PDF
+
+# Data contracts
+mltk contract init                        # Scaffold contract YAML
+mltk contract validate data.csv          # Validate against contract
+mltk contract generate-tests             # Generate pytest from contract
+
+# Documentation
+mltk docs serve                           # Serve docs locally (hot reload)
+mltk docs build                           # Build static HTML docs
+mltk docs open                            # Build, serve, and open in browser
+
+# Test registry
+mltk registry push my_fixtures            # Save test files to registry
+mltk registry pull my_fixtures            # Restore from registry
+mltk registry list                        # List saved collections
+
+# Notifications
+mltk notify slack --results-json r.json  # Send results to Slack
+
+# Server platform
+mltk server                               # Start result-tracking server
+mltk server-create-key --project myproj  # Generate API key for server
+
+# Interactive
+mltk chat --results-json results.json    # Q&A about test results
+```
+
+## pytest Plugin
+
+Auto-registered on install. Provides ML-specific markers and `--mltk-report`:
+
+```bash
+pytest -m ml_data                 # Run only data quality tests
+pytest -m ml_model                # Run only model quality tests
+pytest -m "not ml_slow"           # Skip long-running tests
+pytest --mltk-report              # Generate HTML report
+pytest --mltk-export-json         # Export results to JSON
+```
+
+Markers: `ml_data`, `ml_model`, `ml_drift`, `ml_inference`, `ml_smoke`, `ml_slow`, `ml_gpu`, `ml_nondeterministic`
+
+## Rust Acceleration
+
+Optional Rust backend for 10-100x speedup on drift detection (KS test, PSI). Falls back to scipy/numpy automatically.
+
+## Comparison
+
+| Feature | Great Expectations | Deepchecks | Evidently | DeepEval | Giskard | Arize Phoenix | **mltk** |
+|---|---|---|---|---|---|---|---|
+| Data validation | Yes | Yes | Limited | No | Yes | No | **Yes** |
+| Model testing | No | Yes | Limited | No | Yes (tabular) | No | **Yes** |
+| Drift detection | No | Yes | Yes | No | No | No | **Yes (Rust)** |
+| Streaming drift | No | No | No | No | No | No | **Yes (ADWIN/CUSUM)** |
+| Bias/fairness | No | No | No | No | Yes | No | **Yes** |
+| Inference testing | No | No | No | No | No | No | **Yes** |
+| pytest native | No | No | No | Yes | Yes | No | **Yes** |
+| CLI | Yes | No | No | Yes | Yes | Yes | **Yes** |
+| Domain kits | No | Partial | No | No | No | No | **Yes** |
+| ML Test Score | No | No | No | No | No | No | **Yes** |
+| Rust acceleration | No | No | No | No | No | No | **Yes** |
+| YAML test defs | No | No | No | No | No | No | **Yes** |
+| Compliance frameworks | No | No | No | No | No | No | **Yes (8 frameworks)** |
+| Training bug detection | No | No | No | No | No | No | **Yes** |
+| Conformal prediction | No | No | No | No | No | No | **Yes** |
+| Composable TestSuite | No | No | No | No | No | No | **Yes** |
+| Code Generation | No | No | No | No | No | No | **Yes (4 assertions)** |
+| LLM evaluation | No | Yes (LLM-only) | No | **Yes (50+ metrics)** | Yes (OWASP scanner) | **Yes (tracing)** | **Yes (232 assertions)** |
+| Behavioral consistency | No | No | No | No | No | No | **Yes (7 assertions)** |
+| NER PII detection | No | No | No | No | No | No | **Yes (4 methods)** |
+| Agent trace testing | No | No | No | Yes (basic) | No | Yes (tracing) | **Yes (9 assertions)** |
+| Multi-agent testing | No | No | No | No | No | No | **Yes** |
+| Synthetic data validation | No | No | No | No | No | No | **Yes (4 assertions)** |
+| Safety taxonomy | No | No | No | Yes (red-teaming) | Yes (OWASP) | No | **Yes (per-category)** |
+| LLM observability | No | No | No | No | No | **Yes** | No |
+| OWASP LLM scanning | No | No | No | No | **Yes** | No | **Yes** |
+| License | Elastic | AGPL | Apache 2.0 | Apache 2.0 | Apache 2.0 | Elastic | **Elastic 2.0** |
+| Core deps | Heavy | Heavy | Medium | Heavy (LLM) | Heavy | Heavy | **2 (numpy, pandas)** |
+
+## Documentation
+
+- [Getting Started](docs/getting-started.md)
+- [Configuration](docs/configuration.md)
+- [API Reference](docs/api/)
+- [Examples](examples/)
+
+## License
+
+[Elastic License 2.0](LICENSE) — free for internal use; commercial licensing
+available at lior1cc@gmail.com. See [LICENSE-COMMERCIAL](LICENSE-COMMERCIAL).
+