auditkit 1.1.0__tar.gz → 1.1.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {auditkit-1.1.0 → auditkit-1.1.2}/CHANGELOG.md +31 -21
- {auditkit-1.1.0 → auditkit-1.1.2}/CITATION.cff +1 -1
- {auditkit-1.1.0 → auditkit-1.1.2}/PKG-INFO +44 -42
- {auditkit-1.1.0 → auditkit-1.1.2}/README.md +41 -41
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/GUIDE.md +76 -59
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/HF_ENDPOINT.md +1 -4
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/SGLANG.md +1 -3
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/VLLM_KNOWN_ISSUES.md +30 -27
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/agent_eval.md +26 -81
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/api.md +9 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/architecture/data-model.md +22 -4
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/cli.md +14 -4
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/community/changelog.md +148 -10
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/configuration.md +3 -1
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/encoder_judge.md +6 -35
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/encoder_judge_prebuilts.md +17 -37
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/guard_models.md +1 -4
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/index.md +3 -3
- auditkit-1.1.2/docs/known_issues.md +29 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/metrics.md +42 -9
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/rag_stress.md +25 -35
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/scorers_reference.md +26 -6
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/README.md +4 -1
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/README.md +9 -20
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/cohere/README.md +1 -11
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/colab/README.md +1 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/mkdocs.yml +2 -6
- {auditkit-1.1.0 → auditkit-1.1.2}/pyproject.toml +3 -2
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/README.md +18 -3
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/__init__.py +1 -1
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/cli.py +6 -2
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/README.md +4 -2
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/README.md +1 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/sample.py +1 -1
- auditkit-1.1.2/tests/test_cli_yaml_model.py +37 -0
- auditkit-1.1.0/docs/BUGS.md +0 -341
- auditkit-1.1.0/docs/COMPARISON.docx +0 -0
- auditkit-1.1.0/docs/COMPARISON.md +0 -185
- auditkit-1.1.0/docs/RELEASE.md +0 -701
- auditkit-1.1.0/docs/about/product.md +0 -204
- auditkit-1.1.0/docs/known_issues.md +0 -120
- auditkit-1.1.0/examples/applications/SUMMARY.md +0 -96
- {auditkit-1.1.0 → auditkit-1.1.2}/.gitattributes +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/.github/workflows/check-release-version.yml +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/.github/workflows/patch-release.yml +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/.gitignore +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/AGENTS.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/LICENSE.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/CNAME +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/about/glossary.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/about/platform-integration.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/agents_and_rag.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/annotators.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/architecture/system-design.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/auditkit-logo-black.png +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/auditkit-logo-dark.png +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/auditkit-logo-light.png +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/auditkit-logo-white.png +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/extra.css +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/extra.js +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/favicon.png +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/icon.png +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/icon.svg +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/logo.svg +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/best_practices/scorers.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/community/contributing.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/community/faq.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/community/roadmap.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/cookbook/llm_judge.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/cookbook/rag_evaluation.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/development.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/examples/custom_scorer.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/examples/experiments.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/examples/many_datasets.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/examples/model_comparison.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/examples/quickstart.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/experiments.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/getting_started.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/integrations.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/lexsi_integration.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/llm_judge_rendering.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/model_backends.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/reports.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/requirements.txt +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/docs/scenarios.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/01_full_evaluation_pipeline.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/02_generation_across_hf_families.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/03_custom_annotators.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/04_metrics_deep_dive.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/05_data_types.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/06_model_comparison.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/07_annotators_across_models.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/08_compare_result_deep_dive.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/09_guard_judge_implementation_check.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/10_guard_judge_legit_models.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/11_encoder_judge_prebuilts.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/12_performance_metrics_demo.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/13_sglang_compatibility.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/14_agent_and_rag_evals.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/15_agentic_rag_metrics_walkthrough.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/16_agentic_endpoints_live.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/17_multi_route_references.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/agent_eval_offline.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/01_application_pruned_llama_boolq.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/02_application_healthcare_pubmedqa.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/03_application_finance_10k_qa.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/04_application_ecommerce_review_triage.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/05_application_education_arc_tutor.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/06_application_enterprise_search_rag.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/07_application_bert_nli_judge.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/auditkit.yaml +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/cohere/compare_models.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/cohere/metrics_generation.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/cohere/run_lmeval.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/cohere/sglang_auditkit.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/cohere/tgi_endpoint.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/colab/01_cohere_and_hf_api_models.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/colab/02_agentic_rag_real_agent.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/colab/03_vllm_integration.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/colab/04_cohere_hf_fixes_live.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/colab/05_real_world_agentic_rag.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/colab/06_cohere_all_backends.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/model_comparison_notebook.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/stress_tests/README.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/stress_tests/test_stress_gated_model.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/examples/stress_tests/test_stress_open_model_load.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/scripts/__init__.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/scripts/release.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/__main__.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/_bootstrap.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/_identity_guard.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/adapter.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/README.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/__init__.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/importers.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/lexsi.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/outcome.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/reliability.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/report.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/runner.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/sidecar.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/types.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/annotator.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/api.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/assets/auditkit_logo.png +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/bfcl.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/cache.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/comparison.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/compat.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/diff.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/errors.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/evaluator.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/experiment.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/hf_publish.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/lexsi_login.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/lexsi_relay.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/lmeval_engine.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/loaders.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/logs.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metric.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/__init__.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/agent.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/code.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/embedding.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/encoder_judge.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/generation.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/guard.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/hallucination.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/judge.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/pairwise.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/perf.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/rag.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/rag_judge.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/rag_stress.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/rag_stress_ops.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/rag_stress_runner.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/retrieval.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/security.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/toxicity.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/__init__.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/agent_endpoint.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/anthropic.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/api_gen.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/groq_gen.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/hf_gen.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/lexsi.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/litellm_gen.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/openai.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/openrouter_gen.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/vllm_gen.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model_compare.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/provenance.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/README.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/__init__.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/detector.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/detectors/README.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/detectors/builtin.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/probe.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/probes/README.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/probes/builtin.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/runner.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/registry.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/report.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/report_format.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/router.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/runner.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/runspec.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenario.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/README.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/__init__.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/arc.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/gsm8k.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/hellaswag.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/humaneval.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/mmlu.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/truthfulqa.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/score.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scorers.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scoring.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/trace.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/types.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/README.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/__init__.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/.gitignore +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/FINDINGS.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/README.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/__init__.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/catalog.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/colab_model_matrix.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/colab_offline_suite.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/conftest.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/make_metric_reports.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_agentic_pipeline_all_metrics.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_fingerprint_offline.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_formats_offline.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_live.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_live_rag.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_metrics_offline.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_pipeline_offline.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_rag_offline.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_resolved_issues.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/conftest.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/chat_templates/aya_expanse.jinja +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/chat_templates/command_r7b_tool_use.jinja +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/chat_templates/tiny_aya_default.jinja +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/agenttune/dataset_train_grpo.jsonl +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/agenttune/report_constructed.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/agenttune/trace_finder.jsonl +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/agenttune/trace_hotpotqa.jsonl +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/agenttune/trajectory_store_export.jsonl +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/aligntune/AuditReport.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/aligntune/RegressionReport.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/circuitkit/faithfulness_report.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/circuitkit/faithfulness_report_default.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/circuitkit/faithfulness_report_ib_fastpath.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/circuitkit/faithfulness_report_nonfinite.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/circuitkit/faithfulness_report_unknown_sentinels.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/checkpoint_TextCleaner.jsonl +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/data_sample.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/data_sample_pdf.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/manifest.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/manifest_pdf.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/rejected.jsonl +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/rejected_pdf.jsonl +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/sft_alpaca.jsonl +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/sft_sharegpt.jsonl +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/safetune/placeholder-model__base__placeholder_bench.jsonl +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/safetune/placeholder-model__base__placeholder_bench_scored.jsonl +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/safetune/safety_audit_event__post_allow_long_redacted.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/safetune/safety_audit_event__post_keyword_classifier_block.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/safetune/safety_audit_event__pre_allow.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/safetune/safety_audit_event__pre_tool_block.json +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/sglang_0520/commandr_excerpt.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/sglang_0520/transformers_backend_excerpt.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/sglang_0520/unified_radix_cache_excerpt.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/README.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/__init__.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/colab_cohere_models.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/colab_integrations.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/colab_sglang.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/colab_sglang_32b_north.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/colab_sglang_cohere_matrix.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/colab_vllm_32b.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/colab_vllm_cohere_matrix.ipynb +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/conftest.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/test_agenttune.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/test_cohere.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/test_hf.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/test_live.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/test_sglang.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/README.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/__init__.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/conftest.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/serve_model.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/test_issue43_specs.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/test_issue44_user_flows.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/test_issue45_bfcl.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/test_issue45_routes.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/test_live.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/test_vllm_cohere_templates.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/nonmetric_suite/README.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/nonmetric_suite/__init__.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/nonmetric_suite/conftest.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/nonmetric_suite/test_agent_eval_and_loaders.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/nonmetric_suite/test_mrm_evidence.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/nonmetric_suite/test_redteam_machinery.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/README.md +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/__init__.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/conftest.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_edge_agent_and_rag_judges.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_edge_mrm.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_edge_rag_stress.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_edge_redteam_judges.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_edge_uncertainty.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_pr1_additions.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_pr2_redteam.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_pr3_mrm.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_pr4_uncertainty.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_adapter.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_adapter_params.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_endpoint.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_cli.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_disagreement.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_eventlog.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_harness.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_importers.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_outcome.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_reliability.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_runner.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_secrets.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_sidecar.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_types.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_metrics.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_reference_free.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agenttune_fix_plan.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agenttune_integration.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_annotator.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_api.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_api_backend.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_api_images.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_api_tool_calls.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_api_unverified_fields.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_benchmark.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_chat_messages.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_chat_template_kwargs_everywhere.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_cli.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_code_metrics.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_cohere_aya.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_cohere_hf_fix_plan.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_cohere_tool_calls.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_compare.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_comparison.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_compat.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_core_engine.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_custom_judge_pipeline.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_diff.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_embedding_metrics.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_encoder_judge.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_env_compat_fix_plan.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_errors.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_evaluate_many.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_eventlog_action_shapes.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_execute_concurrency.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_experiment.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_fingerprint_stability.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_generation_kwargs.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_generation_metrics.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_guard.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_hallucination_metrics.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_hf_dtype_fallback.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_hf_gen_logits_sanitizer.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_hf_interop.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_hf_wire_format.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_integration.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_judge.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_judge_flow.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_lexsi_compat.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_lexsi_relay.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_llm_judge.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_lmeval_engine.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_lmeval_real_pipeline.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_load_dataset.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_loaders.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_loaders_agent.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_metric.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_metrics_via_pipeline.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_model.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_model_backends.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_multipath_references.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_pairwise_metrics.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_pipeline_live_hf.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_practical.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_provenance.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_rag_judge.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_rag_metrics.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_rag_reference_free.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_rag_stress.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_rag_stress_ops.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_rag_stress_runner.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_redteam.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_reference_fingerprint.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_registry.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_release.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_retrieval_metrics.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_review27_followups.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_route_distribution.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_router.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_runner.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_sample.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_scenario_dataset_ids.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_scenarios.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_score.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_security_perf.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_template_kwargs_f1_fix_plan.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_token_usage_and_cost.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_toxicity_metrics.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_trace.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_track3_agenttune.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_types.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_unknown_architecture_message.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_unscored_results.py +0 -0
- {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_vllm_offline_engine.py +0 -0
|
@@ -11,19 +11,18 @@ All notable changes to AuditKIT. The format follows [Keep a Changelog](https://k
|
|
|
11
11
|
SGLang), e.g. `{"enable_thinking": False}` for Qwen3. A request's own values win.
|
|
12
12
|
Part of the run fingerprint.
|
|
13
13
|
- `api:` says when a server may have ignored `chat_template_kwargs` or
|
|
14
|
-
`parallel_tool_calls
|
|
14
|
+
`parallel_tool_calls`: `RunResult.metadata["unverified_request_fields"]`, a
|
|
15
15
|
`summary()` line, one warning, and `cap_unverified` on the parallel scores. The
|
|
16
16
|
server is probed once (`GET /models` `owned_by`) and recorded in
|
|
17
17
|
`metadata["api_server"]`; only `APIModel(server="sglang"|"vllm")` clears the note.
|
|
18
18
|
New `Model.run_notes()` hook for backend facts about a run.
|
|
19
|
-
- `ak.load_bfcl()` and `ak.bfcl_arg_match
|
|
19
|
+
- `ak.load_bfcl()` and `ak.bfcl_arg_match`: BFCL v3's 11 single-turn
|
|
20
20
|
categories plus the two irrelevance ones, read from the Hub files (JSON Lines). The
|
|
21
21
|
accepted-value lists stay on the reference, and `bfcl_arg_match` compares against
|
|
22
22
|
them as BFCL's AST checker does. It handles a drifted answer id, flags unsatisfiable
|
|
23
23
|
answers, passes system prompts through, converts BFCL types to JSON Schema, and has an
|
|
24
24
|
optional `dots_to_underscores`. Multi-turn is refused: its answers are call strings.
|
|
25
|
-
|
|
26
|
-
- `RunResult.route_distribution()` and a `summary()` line (#45): for samples whose
|
|
25
|
+
- `RunResult.route_distribution()` and a `summary()` line: for samples whose
|
|
27
26
|
reference names several routes (`any_of`), which route each took and which rule
|
|
28
27
|
decided (`f1`, `recall`, `turn_structure`, `call_order`, `listed_order`), with the
|
|
29
28
|
ties broken by listed order counted. Information only: it never reaches the stats.
|
|
@@ -41,7 +40,7 @@ All notable changes to AuditKIT. The format follows [Keep a Changelog](https://k
|
|
|
41
40
|
layers' own `config.sliding_window`. Rule (m) does the same for SGLang's generic
|
|
42
41
|
Transformers backend (Aya Vision: Triton decode crashed with `kv_indptr=None`),
|
|
43
42
|
returning a window only for models with sliding-window layers.
|
|
44
|
-
- The `vllm:` column covers all nine Cohere models
|
|
43
|
+
- The `vllm:` column covers all nine Cohere models: the full matrix ran on an RTX PRO 6000
|
|
45
44
|
(CUDA 12.8) with the FlashInfer sampler fix, every check passing. Aya Expanse 32B and Aya
|
|
46
45
|
Vision 32B score 5/5; the other seven match their A100 scores exactly.
|
|
47
46
|
- `vllm:` on SM 12.x GPUs (RTX PRO 6000, RTX 50-series) with a CUDA toolkit older than 12.9:
|
|
@@ -61,18 +60,18 @@ All notable changes to AuditKIT. The format follows [Keep a Changelog](https://k
|
|
|
61
60
|
32B: all nine Cohere models now run on SGLang 0.5.20. `colab_sglang_32b_north.ipynb`
|
|
62
61
|
measures the two.
|
|
63
62
|
- `docs/model_backends.md`: the `vllm:` column is measured for the seven Cohere models that fit
|
|
64
|
-
a 40 GB A100 (
|
|
63
|
+
a 40 GB A100 (vLLM 0.30.0): Tiny Aya 4/5 each, North 5/5, Aya Expanse 8B and Aya Vision
|
|
65
64
|
8B 5/5, image checks passing. vLLM falls back to its Transformers backend for Aya Vision on
|
|
66
65
|
its own.
|
|
67
66
|
"Aya Vision loads on neither vLLM nor SGLang" is corrected: it loads on both, non-natively.
|
|
68
|
-
The images section and README now say `api:` sends images
|
|
69
|
-
`vllm:` backend still drops them; the install trap names
|
|
67
|
+
The images section and README now say `api:` sends images and the offline
|
|
68
|
+
`vllm:` backend still drops them; the install trap names the cause (a stale auditkit 1.0.0).
|
|
70
69
|
- `check_compat()` rule (j): with transformers >= 5.15 in the SGLang env (needed for North
|
|
71
70
|
Micro Vision), SGLang's Transformers backend rejects the `embedding_rowwise` TP style
|
|
72
71
|
that transformers adds for tied embeddings, so North still fails to load. The entry is
|
|
73
72
|
inert there (only `nn.Linear` reads the plan); `--patch-sglang` maps it to `replicate`.
|
|
74
73
|
- transformers' "does not recognize this architecture" is rewritten by `hf:` and `vllm:` to
|
|
75
|
-
name the model type, the installed transformers and the version it needs
|
|
74
|
+
name the model type, the installed transformers and the version it needs:
|
|
76
75
|
"'…North-Micro-Vision-Instruct' is a 'cohere_compass' model, and the installed
|
|
77
76
|
transformers 4.57.6 does not know that architecture: it needs transformers>=5.15. The
|
|
78
77
|
checkpoint is fine." The original read like a broken checkpoint. The real cause was a
|
|
@@ -108,15 +107,27 @@ All notable changes to AuditKIT. The format follows [Keep a Changelog](https://k
|
|
|
108
107
|
|
|
109
108
|
### Fixed
|
|
110
109
|
|
|
110
|
+
- `auditkit eval --config FILE` accepts `model:` from the config file. `--model` was
|
|
111
|
+
`required=True` on the parser, so argparse rejected the command with "the following
|
|
112
|
+
arguments are required: --model" before the YAML was read, even though `docs/cli.md`
|
|
113
|
+
documents exactly that invocation. It is now optional on the parser and checked after the
|
|
114
|
+
config merge; a genuinely missing model still exits 2, naming both ways to supply it.
|
|
111
115
|
- The six built-in benchmark scenarios load again. They used bare Hub ids (`mmlu`, `arc` and
|
|
112
116
|
`truthfulqa` no longer exist; the others failed with `HfUriError`) and now use `cais/mmlu`,
|
|
113
117
|
`openai/gsm8k`, `allenai/ai2_arc`, `Rowan/hellaswag`, `truthfulqa/truthful_qa` and
|
|
114
118
|
`openai/openai_humaneval`. HumanEval also referenced a `TaskKind.CODE` that doesn't exist; it loads
|
|
115
119
|
as a generative task (no code-execution metric, so use `ak.run_lmeval("humaneval", ...)` for pass@k).
|
|
120
|
+
- Docs: the scorer reference and metrics overview cover all 68 registered metrics, and the
|
|
121
|
+
data model documents `Sample.images`, `RunConfig.seed`/`track_performance`/
|
|
122
|
+
`chat_template_kwargs` and `RunResult.unscored`/`metadata`; every `eval` flag is in
|
|
123
|
+
`docs/cli.md`; the extras table matches `pyproject.toml`; the claimed-but-unimplemented
|
|
124
|
+
features are gone, as are the internal pages that were still being published to the site.
|
|
125
|
+
- The `dev` extra declares `pyyaml`, which the `--config` tests need. PyYAML remains an
|
|
126
|
+
optional runtime dependency; the core install is still dependency-free.
|
|
116
127
|
- Docs: no METEOR (never implemented) and no "Rich HTML reports" tick (not implemented); current
|
|
117
128
|
Claude model ids instead of retired Claude 3 ones; "source-available", not "open-source"; no
|
|
118
|
-
|
|
119
|
-
GUIDE's table of contents, catalog heading and version line match its sections and 1.2.
|
|
129
|
+
internal wording, links to a private repository or private issue numbers in public pages;
|
|
130
|
+
GUIDE's table of contents, catalog heading and version line match its sections and 1.1.2.
|
|
120
131
|
|
|
121
132
|
### Changed
|
|
122
133
|
|
|
@@ -125,29 +136,28 @@ All notable changes to AuditKIT. The format follows [Keep a Changelog](https://k
|
|
|
125
136
|
"triangle". **Scores change.** `F1Score(normalize=False)` keeps the old raw
|
|
126
137
|
whitespace split; the setting is part of the run fingerprint.
|
|
127
138
|
- `[lmeval]` extra: `accelerate>=1.0`.
|
|
128
|
-
- **`abstention`: `short_reply_markers` now takes regular expressions
|
|
139
|
+
- **`abstention`: `short_reply_markers` now takes regular expressions**. The
|
|
129
140
|
defaults require refusal-shaped wording ("insufficient information to/for/in/about…",
|
|
130
141
|
"no relevant information/documents/context…"), so a short real answer such as
|
|
131
142
|
"Yes, there is no relevant fee for domestic wires." is no longer a refusal, and a
|
|
132
143
|
first sentence opening with "yes" never is. Bare phrases passed to
|
|
133
144
|
`Abstention(short_reply_markers=...)` still work (a plain phrase is a valid regex);
|
|
134
145
|
a phrase containing regex metacharacters must be escaped (`re.escape`).
|
|
135
|
-
- **An unnamed recorded tool call is unscored, not a measured 0.0
|
|
146
|
+
- **An unnamed recorded tool call is unscored, not a measured 0.0**. An EventLog
|
|
136
147
|
or AgentTune trajectory action that carries call arguments (`arguments`,
|
|
137
148
|
`parameters`, `args`, `input`, `query`) but no tool name leaves
|
|
138
149
|
`tool_call_names` unavailable, so `tool_call_f1` / `trajectory_match` /
|
|
139
150
|
`parallel_tool_calls` are skipped, and `rescore` reports `trace.tool_call_names`.
|
|
140
|
-
- **A trajectory whose actions are all empty is a measured zero calls
|
|
151
|
+
- **A trajectory whose actions are all empty is a measured zero calls**: it
|
|
141
152
|
scores (0.0 against expected calls, 1.0 on a no-call case) instead of being dropped.
|
|
142
153
|
- Docs: SGLang needs Python 3.10-3.12 (a uv 3.12 venv on 3.13 hosts); example 13
|
|
143
154
|
builds its SGLang venv that way. `docs/VLLM_KNOWN_ISSUES.md` covers the Colab
|
|
144
155
|
torchaudio mismatch after `[vllm]`.
|
|
145
156
|
|
|
146
|
-
## [1.2
|
|
157
|
+
## [1.1.2] - 2026-10-02
|
|
147
158
|
|
|
148
|
-
Lexsi stack interop
|
|
149
|
-
|
|
150
|
-
hackathon readiness fixes for the permitted Cohere models.
|
|
159
|
+
Lexsi stack interop (CuratorKIT, AlignTune, SafeTune, CircuitKIT and AgentTune
|
|
160
|
+
outputs evaluate with no glue code), plus fixes for the Cohere models.
|
|
151
161
|
|
|
152
162
|
### Added
|
|
153
163
|
|
|
@@ -166,7 +176,7 @@ hackathon readiness fixes for the permitted Cohere models.
|
|
|
166
176
|
- **Native tool calling on `hf:`.** Tool schemas go to the chat template
|
|
167
177
|
(`apply_chat_template(tools=...)`), so `ToolCallAdapter()` works on local
|
|
168
178
|
models whose template renders tools; a template without tool support raises
|
|
169
|
-
`CapabilityError`. None of the Cohere
|
|
179
|
+
`CapabilityError`. None of the Cohere models' templates (Tiny Aya,
|
|
170
180
|
Aya Expanse, Aya Vision, North) render tools: use
|
|
171
181
|
`ToolCallAdapter(mode="prompt")` with them.
|
|
172
182
|
- **Cohere tool-call format.** The tool-call parser reads Command R7B
|
|
@@ -189,11 +199,11 @@ hackathon readiness fixes for the permitted Cohere models.
|
|
|
189
199
|
### Fixed
|
|
190
200
|
|
|
191
201
|
- A JSON-string reference (a CSV cell, a JSONL field) now fingerprints like its
|
|
192
|
-
decoded form
|
|
202
|
+
decoded form, in the dataset fingerprint and in `AgentCase.digest()`, so the
|
|
193
203
|
same dataset read from a file reuses the cached run. Dict and list references keep
|
|
194
204
|
their digests. `auditkit agent --dry-run` shows `refs=yes` for an explicit `[]`
|
|
195
205
|
reference ("expect no call").
|
|
196
|
-
- A bare prefix is now decided per backend
|
|
206
|
+
- A bare prefix is now decided per backend. `api:`, `hf:` and `vllm:` raise
|
|
197
207
|
"empty model spec" (`api:` used to send `{"model": null}`). The hosted prefixes
|
|
198
208
|
(`openai:`, `anthropic:`, `groq:`, `openrouter:`, `lexsi:`, `litellm:`) fall back
|
|
199
209
|
to their backend's default model, and `agent:` still takes its URL from `url=`.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: auditkit
|
|
3
|
-
Version: 1.1.
|
|
3
|
+
Version: 1.1.2
|
|
4
4
|
Summary: Evaluate any model on any dataset and any task — standalone, no platform required
|
|
5
5
|
Project-URL: Homepage, https://github.com/Lexsi-Labs/AuditKIT
|
|
6
6
|
Project-URL: Repository, https://github.com/Lexsi-Labs/AuditKIT
|
|
@@ -126,6 +126,7 @@ Requires-Dist: pillow; extra == 'all'
|
|
|
126
126
|
Requires-Dist: pytest; extra == 'all'
|
|
127
127
|
Requires-Dist: pytest-asyncio; extra == 'all'
|
|
128
128
|
Requires-Dist: pytest-mock; extra == 'all'
|
|
129
|
+
Requires-Dist: pyyaml; extra == 'all'
|
|
129
130
|
Requires-Dist: requests; extra == 'all'
|
|
130
131
|
Requires-Dist: torch>=2.4; extra == 'all'
|
|
131
132
|
Requires-Dist: torchmetrics; extra == 'all'
|
|
@@ -142,6 +143,7 @@ Provides-Extra: dev
|
|
|
142
143
|
Requires-Dist: pytest; extra == 'dev'
|
|
143
144
|
Requires-Dist: pytest-asyncio; extra == 'dev'
|
|
144
145
|
Requires-Dist: pytest-mock; extra == 'dev'
|
|
146
|
+
Requires-Dist: pyyaml; extra == 'dev'
|
|
145
147
|
Provides-Extra: interop
|
|
146
148
|
Requires-Dist: datasets; extra == 'interop'
|
|
147
149
|
Requires-Dist: gitpython; extra == 'interop'
|
|
@@ -180,8 +182,8 @@ Description-Content-Type: text/markdown
|
|
|
180
182
|
<p align="center">
|
|
181
183
|
<a href="https://github.com/Lexsi-Labs/AuditKIT">
|
|
182
184
|
<picture>
|
|
183
|
-
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/auditkit-logo-white.png">
|
|
184
|
-
<img src="docs/assets/auditkit-logo-black.png" alt="AuditKit" width="480">
|
|
185
|
+
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/Lexsi-Labs/AuditKIT/main/docs/assets/auditkit-logo-white.png">
|
|
186
|
+
<img src="https://raw.githubusercontent.com/Lexsi-Labs/AuditKIT/main/docs/assets/auditkit-logo-black.png" alt="AuditKit" width="480">
|
|
185
187
|
</picture>
|
|
186
188
|
</a>
|
|
187
189
|
</p>
|
|
@@ -193,7 +195,7 @@ Description-Content-Type: text/markdown
|
|
|
193
195
|
</p>
|
|
194
196
|
|
|
195
197
|
<p align="center">
|
|
196
|
-
<a href="https://pypi.org/project/auditkit/"><img src="https://img.shields.io/
|
|
198
|
+
<a href="https://pypi.org/project/auditkit/"><img src="https://img.shields.io/pypi/v/auditkit?color=0a8868" alt="PyPI version"></a>
|
|
197
199
|
<a href="https://www.python.org/"><img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python 3.10+"></a>
|
|
198
200
|
<a href="https://github.com/Lexsi-Labs/AuditKIT/blob/main/LICENSE.md"><img src="https://img.shields.io/badge/license-LSAL--1.2-lightgrey" alt="License: LSAL-1.2 (source-available, noncommercial)"></a>
|
|
199
201
|
<a href="https://auditkit.lexsi.ai/"><img src="https://img.shields.io/badge/docs-auditkit.lexsi.ai-4c6ef5" alt="Documentation"></a>
|
|
@@ -219,7 +221,7 @@ Description-Content-Type: text/markdown
|
|
|
219
221
|
| **Models** | 10 backends: OpenAI, Anthropic, HuggingFace, Lexsi, vLLM, LiteLLM, API, Groq, OpenRouter, Agent (HTTP endpoint) — all resolved via `AutoModel.resolve()`. Any `list[str] → list[str]` callable also works. |
|
|
220
222
|
| **Zero deps** | Core runs on stdlib. Backends and heavy metrics are optional extras (`pip install auditkit[openai]`). |
|
|
221
223
|
| **Fingerprints** | Every run gets a stable sha256 — results are cacheable, comparable, and reproducible by construction. |
|
|
222
|
-
| **Status** | ~3,000 tests, v1.2
|
|
224
|
+
| **Status** | ~3,000 tests, v1.1.2, LSAL-1.2 license (source-available, noncommercial). 11 metric families, 6 CLI subcommands, YAML config, MKDocs site. |
|
|
223
225
|
|
|
224
226
|
---
|
|
225
227
|
|
|
@@ -359,7 +361,7 @@ dropped it after 0.24.0, SGLang never had it), so both servers run it through th
|
|
|
359
361
|
Transformers backends: automatically on vLLM, with `compat.py --patch-sglang` on SGLang. North Micro Vision is native on vLLM; on SGLang
|
|
360
362
|
it needs transformers >= 5.15 in the SGLang env plus `--patch-sglang`. Tiny Aya and Aya Expanse
|
|
361
363
|
are native on both. The full
|
|
362
|
-
model-by-runtime table is in [docs/model_backends.md](docs/model_backends.md).
|
|
364
|
+
model-by-runtime table is in [docs/model_backends.md](https://github.com/Lexsi-Labs/AuditKIT/blob/main/docs/model_backends.md).
|
|
363
365
|
|
|
364
366
|
Tiny Aya (`CohereLabs/tiny-aya-global` and siblings `fire`, `water`, `earth`),
|
|
365
367
|
Aya Expanse and Aya Vision are gated
|
|
@@ -392,57 +394,57 @@ Every `evaluate()` returns a `RunResult`:
|
|
|
392
394
|
## Examples
|
|
393
395
|
|
|
394
396
|
Colab-ready notebooks with real models and real datasets — see
|
|
395
|
-
[`examples/README.md`](examples/README.md) for the full, current list.
|
|
397
|
+
[`examples/README.md`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/README.md) for the full, current list.
|
|
396
398
|
|
|
397
399
|
| Example | File |
|
|
398
400
|
|---------|------|
|
|
399
|
-
| Full pipeline: adapter, 5 metrics, LLM judge, LLM annotator | [`01_full_evaluation_pipeline.ipynb`](examples/01_full_evaluation_pipeline.ipynb) |
|
|
400
|
-
| Generation across two real HF model families, compared | [`02_generation_across_hf_families.ipynb`](examples/02_generation_across_hf_families.ipynb) |
|
|
401
|
-
| Custom annotators (regex, LLM-backed, fully custom) | [`03_custom_annotators.ipynb`](examples/03_custom_annotators.ipynb) |
|
|
402
|
-
| Metrics deep dive: built-in, custom, LLM-as-judge, RAG | [`04_metrics_deep_dive.ipynb`](examples/04_metrics_deep_dive.ipynb) |
|
|
403
|
-
| Every data type/task kind: generative, MCQ, RAG, precomputed, chat | [`05_data_types.ipynb`](examples/05_data_types.ipynb) |
|
|
404
|
-
| Model comparison deep dive: `compare_models()` + `RunComparison` | [`06_model_comparison.ipynb`](examples/06_model_comparison.ipynb) |
|
|
405
|
-
| Annotators across 4 real model families, then compared together | [`07_annotators_across_models.ipynb`](examples/07_annotators_across_models.ipynb) |
|
|
406
|
-
| `CompareResult` deep dive: what `compare_models()`'s native per-model support still can't express (different scorers per model), full method surface | [`08_compare_result_deep_dive.ipynb`](examples/08_compare_result_deep_dive.ipynb) |
|
|
407
|
-
| `GuardJudge` implementation check: all 5 profiles, real and offline | [`09_guard_judge_implementation_check.ipynb`](examples/09_guard_judge_implementation_check.ipynb) |
|
|
408
|
-
| `GuardJudge` with real, flagship guard models | [`10_guard_judge_legit_models.ipynb`](examples/10_guard_judge_legit_models.ipynb) |
|
|
409
|
-
| `EncoderJudge`: the base mechanism plus its two prebuilt subclasses | [`11_encoder_judge_prebuilts.ipynb`](examples/11_encoder_judge_prebuilts.ipynb) |
|
|
410
|
-
| Performance metrics on a real, large-scale evaluation | [`12_performance_metrics_demo.ipynb`](examples/12_performance_metrics_demo.ipynb) |
|
|
411
|
-
| SGLang: environment checks, server mode via `api:`, tool calls and parallel tool calls | [`13_sglang_compatibility.ipynb`](examples/13_sglang_compatibility.ipynb) |
|
|
412
|
-
| Agent and RAG evals: tool calls, parallel tool calls, deployed agent endpoints, AgentTune data, retrieval and judged RAG metrics | [`14_agent_and_rag_evals.ipynb`](examples/14_agent_and_rag_evals.ipynb) |
|
|
413
|
-
| End-to-end agent evaluation, fully offline (`.py` script): recorded episodes, a verified state outcome, and a fabricated answer that does not pass a state case | [`agent_eval_offline.py`](examples/agent_eval_offline.py) |
|
|
401
|
+
| Full pipeline: adapter, 5 metrics, LLM judge, LLM annotator | [`01_full_evaluation_pipeline.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/01_full_evaluation_pipeline.ipynb) |
|
|
402
|
+
| Generation across two real HF model families, compared | [`02_generation_across_hf_families.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/02_generation_across_hf_families.ipynb) |
|
|
403
|
+
| Custom annotators (regex, LLM-backed, fully custom) | [`03_custom_annotators.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/03_custom_annotators.ipynb) |
|
|
404
|
+
| Metrics deep dive: built-in, custom, LLM-as-judge, RAG | [`04_metrics_deep_dive.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/04_metrics_deep_dive.ipynb) |
|
|
405
|
+
| Every data type/task kind: generative, MCQ, RAG, precomputed, chat | [`05_data_types.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/05_data_types.ipynb) |
|
|
406
|
+
| Model comparison deep dive: `compare_models()` + `RunComparison` | [`06_model_comparison.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/06_model_comparison.ipynb) |
|
|
407
|
+
| Annotators across 4 real model families, then compared together | [`07_annotators_across_models.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/07_annotators_across_models.ipynb) |
|
|
408
|
+
| `CompareResult` deep dive: what `compare_models()`'s native per-model support still can't express (different scorers per model), full method surface | [`08_compare_result_deep_dive.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/08_compare_result_deep_dive.ipynb) |
|
|
409
|
+
| `GuardJudge` implementation check: all 5 profiles, real and offline | [`09_guard_judge_implementation_check.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/09_guard_judge_implementation_check.ipynb) |
|
|
410
|
+
| `GuardJudge` with real, flagship guard models | [`10_guard_judge_legit_models.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/10_guard_judge_legit_models.ipynb) |
|
|
411
|
+
| `EncoderJudge`: the base mechanism plus its two prebuilt subclasses | [`11_encoder_judge_prebuilts.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/11_encoder_judge_prebuilts.ipynb) |
|
|
412
|
+
| Performance metrics on a real, large-scale evaluation | [`12_performance_metrics_demo.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/12_performance_metrics_demo.ipynb) |
|
|
413
|
+
| SGLang: environment checks, server mode via `api:`, tool calls and parallel tool calls | [`13_sglang_compatibility.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/13_sglang_compatibility.ipynb) |
|
|
414
|
+
| Agent and RAG evals: tool calls, parallel tool calls, deployed agent endpoints, AgentTune data, retrieval and judged RAG metrics | [`14_agent_and_rag_evals.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/14_agent_and_rag_evals.ipynb) |
|
|
415
|
+
| End-to-end agent evaluation, fully offline (`.py` script): recorded episodes, a verified state outcome, and a fabricated answer that does not pass a state case | [`agent_eval_offline.py`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/agent_eval_offline.py) |
|
|
414
416
|
|
|
415
417
|
## Applications
|
|
416
418
|
|
|
417
419
|
Real-world scenarios answered end to end, not feature tours — see
|
|
418
|
-
[`examples/applications/README.md`](examples/applications/README.md). Only `01` uses the
|
|
420
|
+
[`examples/applications/README.md`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/README.md). Only `01` uses the
|
|
419
421
|
`lm-evaluation-harness` integration (`ak.run_lmeval()`, needs
|
|
420
422
|
`auditkit[lmeval]`), as an authoritative cross-check alongside the native
|
|
421
423
|
evaluation; none of the others do.
|
|
422
424
|
|
|
423
425
|
| Application | File |
|
|
424
426
|
|-------------|------|
|
|
425
|
-
| How much does pruning severity (20%/40%/60%) degrade a model? Real BoolQ, real annotator, ship/no-ship verdicts, cross-checked via `ak.run_lmeval()` | [`01_application_pruned_llama_boolq.ipynb`](examples/applications/01_application_pruned_llama_boolq.ipynb) |
|
|
426
|
-
| Healthcare: clinical QA correctness vs. grounding (PubMedQA) | [`02_application_healthcare_pubmedqa.ipynb`](examples/applications/02_application_healthcare_pubmedqa.ipynb) |
|
|
427
|
-
| Finance: QA grounded in real SEC 10-K filings | [`03_application_finance_10k_qa.ipynb`](examples/applications/03_application_finance_10k_qa.ipynb) |
|
|
428
|
-
| E-commerce: review-sentiment triage at scale | [`04_application_ecommerce_review_triage.ipynb`](examples/applications/04_application_ecommerce_review_triage.ipynb) |
|
|
429
|
-
| Education: auto-graded tutoring, correctness vs. explanation | [`05_application_education_arc_tutor.ipynb`](examples/applications/05_application_education_arc_tutor.ipynb) |
|
|
430
|
-
| Enterprise search: internal knowledge assistant (real retrieval + RAG grounding) | [`06_application_enterprise_search_rag.ipynb`](examples/applications/06_application_enterprise_search_rag.ipynb) |
|
|
431
|
-
| LLM-as-judge via a real BERT NLI classifier, not a generative model | [`07_application_bert_nli_judge.ipynb`](examples/applications/07_application_bert_nli_judge.ipynb) |
|
|
427
|
+
| How much does pruning severity (20%/40%/60%) degrade a model? Real BoolQ, real annotator, ship/no-ship verdicts, cross-checked via `ak.run_lmeval()` | [`01_application_pruned_llama_boolq.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/01_application_pruned_llama_boolq.ipynb) |
|
|
428
|
+
| Healthcare: clinical QA correctness vs. grounding (PubMedQA) | [`02_application_healthcare_pubmedqa.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/02_application_healthcare_pubmedqa.ipynb) |
|
|
429
|
+
| Finance: QA grounded in real SEC 10-K filings | [`03_application_finance_10k_qa.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/03_application_finance_10k_qa.ipynb) |
|
|
430
|
+
| E-commerce: review-sentiment triage at scale | [`04_application_ecommerce_review_triage.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/04_application_ecommerce_review_triage.ipynb) |
|
|
431
|
+
| Education: auto-graded tutoring, correctness vs. explanation | [`05_application_education_arc_tutor.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/05_application_education_arc_tutor.ipynb) |
|
|
432
|
+
| Enterprise search: internal knowledge assistant (real retrieval + RAG grounding) | [`06_application_enterprise_search_rag.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/06_application_enterprise_search_rag.ipynb) |
|
|
433
|
+
| LLM-as-judge via a real BERT NLI classifier, not a generative model | [`07_application_bert_nli_judge.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/07_application_bert_nli_judge.ipynb) |
|
|
432
434
|
|
|
433
435
|
## Repository map
|
|
434
436
|
|
|
435
437
|
| Directory | Description | README |
|
|
436
438
|
|-----------|-------------|--------|
|
|
437
|
-
| [`src/auditkit/`](src/auditkit/) | Core library — spine, runner, metrics, models, CLI | [README](src/auditkit/README.md) |
|
|
438
|
-
| [`src/auditkit/metrics/`](src/auditkit/metrics/) | Metric families | [README](src/auditkit/metrics/README.md) |
|
|
439
|
-
| [`src/auditkit/model/`](src/auditkit/model/) | Model backends (echo, openai, hf, vllm, etc.) | [README](src/auditkit/model/README.md) |
|
|
440
|
-
| [`src/auditkit/redteam/`](src/auditkit/redteam/) | Red team probes and detectors | [README](src/auditkit/redteam/README.md) |
|
|
441
|
-
| [`src/auditkit/scenarios/`](src/auditkit/scenarios/) | Built-in benchmark datasets | [README](src/auditkit/scenarios/README.md) |
|
|
442
|
-
| [`tests/`](tests/) | Test suite (~3,000 tests) | [README](tests/README.md) |
|
|
443
|
-
| [`examples/`](examples/) | Colab-ready example notebooks | [README](examples/README.md) |
|
|
444
|
-
| [`examples/applications/`](examples/applications/) | Real-world application notebooks | [README](examples/applications/README.md) |
|
|
445
|
-
| [`docs/`](docs/) | MkDocs source for [auditkit.lexsi.ai](https://auditkit.lexsi.ai/) | [index](docs/index.md) |
|
|
439
|
+
| [`src/auditkit/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/) | Core library — spine, runner, metrics, models, CLI | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/README.md) |
|
|
440
|
+
| [`src/auditkit/metrics/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/metrics/) | Metric families | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/metrics/README.md) |
|
|
441
|
+
| [`src/auditkit/model/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/model/) | Model backends (echo, openai, hf, vllm, etc.) | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/model/README.md) |
|
|
442
|
+
| [`src/auditkit/redteam/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/redteam/) | Red team probes and detectors | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/redteam/README.md) |
|
|
443
|
+
| [`src/auditkit/scenarios/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/scenarios/) | Built-in benchmark datasets | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/scenarios/README.md) |
|
|
444
|
+
| [`tests/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/tests/) | Test suite (~3,000 tests) | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/tests/README.md) |
|
|
445
|
+
| [`examples/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/examples/) | Colab-ready example notebooks | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/README.md) |
|
|
446
|
+
| [`examples/applications/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/examples/applications/) | Real-world application notebooks | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/README.md) |
|
|
447
|
+
| [`docs/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/docs/) | MkDocs source for [auditkit.lexsi.ai](https://auditkit.lexsi.ai/) | [index](https://github.com/Lexsi-Labs/AuditKIT/blob/main/docs/index.md) |
|
|
446
448
|
|
|
447
449
|
---
|
|
448
450
|
|
|
@@ -467,7 +469,7 @@ If you use AuditKIT in research, please cite it:
|
|
|
467
469
|
Kadiyala, Ram Mohan Rao and Gosalia, Hem and Sankarapu, Vinay Kumar and
|
|
468
470
|
Seth, Pratinav},
|
|
469
471
|
year = {2026},
|
|
470
|
-
version = {1.2
|
|
472
|
+
version = {1.1.2},
|
|
471
473
|
url = {https://github.com/Lexsi-Labs/AuditKIT}
|
|
472
474
|
}
|
|
473
475
|
```
|
|
@@ -478,11 +480,11 @@ If you use AuditKIT in research, please cite it:
|
|
|
478
480
|
|
|
479
481
|
## License
|
|
480
482
|
|
|
481
|
-
This project is released under the [Lexsi Labs Source Available License (LSAL) v1.2](LICENSE.md) — free for academic research and teaching on MIT-like terms; use by any organization requires written acknowledgement or permission (Section 1A); a separate commercial license is required to sell it or embed it in a paid product.
|
|
483
|
+
This project is released under the [Lexsi Labs Source Available License (LSAL) v1.2](https://github.com/Lexsi-Labs/AuditKIT/blob/main/LICENSE.md) — free for academic research and teaching on MIT-like terms; use by any organization requires written acknowledgement or permission (Section 1A); a separate commercial license is required to sell it or embed it in a paid product.
|
|
482
484
|
|
|
483
485
|
---
|
|
484
486
|
|
|
485
487
|
## Join Community / Contribute
|
|
486
488
|
|
|
487
489
|
- Issues and discussions are welcomed on the [GitHub issue tracker](https://github.com/Lexsi-Labs/AuditKIT/issues).
|
|
488
|
-
- See [Contributing](docs/community/contributing.md) for the development setup, code standards and the pull request process.
|
|
490
|
+
- See [Contributing](https://github.com/Lexsi-Labs/AuditKIT/blob/main/docs/community/contributing.md) for the development setup, code standards and the pull request process.
|
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
<p align="center">
|
|
2
2
|
<a href="https://github.com/Lexsi-Labs/AuditKIT">
|
|
3
3
|
<picture>
|
|
4
|
-
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/auditkit-logo-white.png">
|
|
5
|
-
<img src="docs/assets/auditkit-logo-black.png" alt="AuditKit" width="480">
|
|
4
|
+
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/Lexsi-Labs/AuditKIT/main/docs/assets/auditkit-logo-white.png">
|
|
5
|
+
<img src="https://raw.githubusercontent.com/Lexsi-Labs/AuditKIT/main/docs/assets/auditkit-logo-black.png" alt="AuditKit" width="480">
|
|
6
6
|
</picture>
|
|
7
7
|
</a>
|
|
8
8
|
</p>
|
|
@@ -14,7 +14,7 @@
|
|
|
14
14
|
</p>
|
|
15
15
|
|
|
16
16
|
<p align="center">
|
|
17
|
-
<a href="https://pypi.org/project/auditkit/"><img src="https://img.shields.io/
|
|
17
|
+
<a href="https://pypi.org/project/auditkit/"><img src="https://img.shields.io/pypi/v/auditkit?color=0a8868" alt="PyPI version"></a>
|
|
18
18
|
<a href="https://www.python.org/"><img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python 3.10+"></a>
|
|
19
19
|
<a href="https://github.com/Lexsi-Labs/AuditKIT/blob/main/LICENSE.md"><img src="https://img.shields.io/badge/license-LSAL--1.2-lightgrey" alt="License: LSAL-1.2 (source-available, noncommercial)"></a>
|
|
20
20
|
<a href="https://auditkit.lexsi.ai/"><img src="https://img.shields.io/badge/docs-auditkit.lexsi.ai-4c6ef5" alt="Documentation"></a>
|
|
@@ -40,7 +40,7 @@
|
|
|
40
40
|
| **Models** | 10 backends: OpenAI, Anthropic, HuggingFace, Lexsi, vLLM, LiteLLM, API, Groq, OpenRouter, Agent (HTTP endpoint) — all resolved via `AutoModel.resolve()`. Any `list[str] → list[str]` callable also works. |
|
|
41
41
|
| **Zero deps** | Core runs on stdlib. Backends and heavy metrics are optional extras (`pip install auditkit[openai]`). |
|
|
42
42
|
| **Fingerprints** | Every run gets a stable sha256 — results are cacheable, comparable, and reproducible by construction. |
|
|
43
|
-
| **Status** | ~3,000 tests, v1.2
|
|
43
|
+
| **Status** | ~3,000 tests, v1.1.2, LSAL-1.2 license (source-available, noncommercial). 11 metric families, 6 CLI subcommands, YAML config, MKDocs site. |
|
|
44
44
|
|
|
45
45
|
---
|
|
46
46
|
|
|
@@ -180,7 +180,7 @@ dropped it after 0.24.0, SGLang never had it), so both servers run it through th
|
|
|
180
180
|
Transformers backends: automatically on vLLM, with `compat.py --patch-sglang` on SGLang. North Micro Vision is native on vLLM; on SGLang
|
|
181
181
|
it needs transformers >= 5.15 in the SGLang env plus `--patch-sglang`. Tiny Aya and Aya Expanse
|
|
182
182
|
are native on both. The full
|
|
183
|
-
model-by-runtime table is in [docs/model_backends.md](docs/model_backends.md).
|
|
183
|
+
model-by-runtime table is in [docs/model_backends.md](https://github.com/Lexsi-Labs/AuditKIT/blob/main/docs/model_backends.md).
|
|
184
184
|
|
|
185
185
|
Tiny Aya (`CohereLabs/tiny-aya-global` and siblings `fire`, `water`, `earth`),
|
|
186
186
|
Aya Expanse and Aya Vision are gated
|
|
@@ -213,57 +213,57 @@ Every `evaluate()` returns a `RunResult`:
|
|
|
213
213
|
## Examples
|
|
214
214
|
|
|
215
215
|
Colab-ready notebooks with real models and real datasets — see
|
|
216
|
-
[`examples/README.md`](examples/README.md) for the full, current list.
|
|
216
|
+
[`examples/README.md`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/README.md) for the full, current list.
|
|
217
217
|
|
|
218
218
|
| Example | File |
|
|
219
219
|
|---------|------|
|
|
220
|
-
| Full pipeline: adapter, 5 metrics, LLM judge, LLM annotator | [`01_full_evaluation_pipeline.ipynb`](examples/01_full_evaluation_pipeline.ipynb) |
|
|
221
|
-
| Generation across two real HF model families, compared | [`02_generation_across_hf_families.ipynb`](examples/02_generation_across_hf_families.ipynb) |
|
|
222
|
-
| Custom annotators (regex, LLM-backed, fully custom) | [`03_custom_annotators.ipynb`](examples/03_custom_annotators.ipynb) |
|
|
223
|
-
| Metrics deep dive: built-in, custom, LLM-as-judge, RAG | [`04_metrics_deep_dive.ipynb`](examples/04_metrics_deep_dive.ipynb) |
|
|
224
|
-
| Every data type/task kind: generative, MCQ, RAG, precomputed, chat | [`05_data_types.ipynb`](examples/05_data_types.ipynb) |
|
|
225
|
-
| Model comparison deep dive: `compare_models()` + `RunComparison` | [`06_model_comparison.ipynb`](examples/06_model_comparison.ipynb) |
|
|
226
|
-
| Annotators across 4 real model families, then compared together | [`07_annotators_across_models.ipynb`](examples/07_annotators_across_models.ipynb) |
|
|
227
|
-
| `CompareResult` deep dive: what `compare_models()`'s native per-model support still can't express (different scorers per model), full method surface | [`08_compare_result_deep_dive.ipynb`](examples/08_compare_result_deep_dive.ipynb) |
|
|
228
|
-
| `GuardJudge` implementation check: all 5 profiles, real and offline | [`09_guard_judge_implementation_check.ipynb`](examples/09_guard_judge_implementation_check.ipynb) |
|
|
229
|
-
| `GuardJudge` with real, flagship guard models | [`10_guard_judge_legit_models.ipynb`](examples/10_guard_judge_legit_models.ipynb) |
|
|
230
|
-
| `EncoderJudge`: the base mechanism plus its two prebuilt subclasses | [`11_encoder_judge_prebuilts.ipynb`](examples/11_encoder_judge_prebuilts.ipynb) |
|
|
231
|
-
| Performance metrics on a real, large-scale evaluation | [`12_performance_metrics_demo.ipynb`](examples/12_performance_metrics_demo.ipynb) |
|
|
232
|
-
| SGLang: environment checks, server mode via `api:`, tool calls and parallel tool calls | [`13_sglang_compatibility.ipynb`](examples/13_sglang_compatibility.ipynb) |
|
|
233
|
-
| Agent and RAG evals: tool calls, parallel tool calls, deployed agent endpoints, AgentTune data, retrieval and judged RAG metrics | [`14_agent_and_rag_evals.ipynb`](examples/14_agent_and_rag_evals.ipynb) |
|
|
234
|
-
| End-to-end agent evaluation, fully offline (`.py` script): recorded episodes, a verified state outcome, and a fabricated answer that does not pass a state case | [`agent_eval_offline.py`](examples/agent_eval_offline.py) |
|
|
220
|
+
| Full pipeline: adapter, 5 metrics, LLM judge, LLM annotator | [`01_full_evaluation_pipeline.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/01_full_evaluation_pipeline.ipynb) |
|
|
221
|
+
| Generation across two real HF model families, compared | [`02_generation_across_hf_families.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/02_generation_across_hf_families.ipynb) |
|
|
222
|
+
| Custom annotators (regex, LLM-backed, fully custom) | [`03_custom_annotators.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/03_custom_annotators.ipynb) |
|
|
223
|
+
| Metrics deep dive: built-in, custom, LLM-as-judge, RAG | [`04_metrics_deep_dive.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/04_metrics_deep_dive.ipynb) |
|
|
224
|
+
| Every data type/task kind: generative, MCQ, RAG, precomputed, chat | [`05_data_types.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/05_data_types.ipynb) |
|
|
225
|
+
| Model comparison deep dive: `compare_models()` + `RunComparison` | [`06_model_comparison.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/06_model_comparison.ipynb) |
|
|
226
|
+
| Annotators across 4 real model families, then compared together | [`07_annotators_across_models.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/07_annotators_across_models.ipynb) |
|
|
227
|
+
| `CompareResult` deep dive: what `compare_models()`'s native per-model support still can't express (different scorers per model), full method surface | [`08_compare_result_deep_dive.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/08_compare_result_deep_dive.ipynb) |
|
|
228
|
+
| `GuardJudge` implementation check: all 5 profiles, real and offline | [`09_guard_judge_implementation_check.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/09_guard_judge_implementation_check.ipynb) |
|
|
229
|
+
| `GuardJudge` with real, flagship guard models | [`10_guard_judge_legit_models.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/10_guard_judge_legit_models.ipynb) |
|
|
230
|
+
| `EncoderJudge`: the base mechanism plus its two prebuilt subclasses | [`11_encoder_judge_prebuilts.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/11_encoder_judge_prebuilts.ipynb) |
|
|
231
|
+
| Performance metrics on a real, large-scale evaluation | [`12_performance_metrics_demo.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/12_performance_metrics_demo.ipynb) |
|
|
232
|
+
| SGLang: environment checks, server mode via `api:`, tool calls and parallel tool calls | [`13_sglang_compatibility.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/13_sglang_compatibility.ipynb) |
|
|
233
|
+
| Agent and RAG evals: tool calls, parallel tool calls, deployed agent endpoints, AgentTune data, retrieval and judged RAG metrics | [`14_agent_and_rag_evals.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/14_agent_and_rag_evals.ipynb) |
|
|
234
|
+
| End-to-end agent evaluation, fully offline (`.py` script): recorded episodes, a verified state outcome, and a fabricated answer that does not pass a state case | [`agent_eval_offline.py`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/agent_eval_offline.py) |
|
|
235
235
|
|
|
236
236
|
## Applications
|
|
237
237
|
|
|
238
238
|
Real-world scenarios answered end to end, not feature tours — see
|
|
239
|
-
[`examples/applications/README.md`](examples/applications/README.md). Only `01` uses the
|
|
239
|
+
[`examples/applications/README.md`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/README.md). Only `01` uses the
|
|
240
240
|
`lm-evaluation-harness` integration (`ak.run_lmeval()`, needs
|
|
241
241
|
`auditkit[lmeval]`), as an authoritative cross-check alongside the native
|
|
242
242
|
evaluation; none of the others do.
|
|
243
243
|
|
|
244
244
|
| Application | File |
|
|
245
245
|
|-------------|------|
|
|
246
|
-
| How much does pruning severity (20%/40%/60%) degrade a model? Real BoolQ, real annotator, ship/no-ship verdicts, cross-checked via `ak.run_lmeval()` | [`01_application_pruned_llama_boolq.ipynb`](examples/applications/01_application_pruned_llama_boolq.ipynb) |
|
|
247
|
-
| Healthcare: clinical QA correctness vs. grounding (PubMedQA) | [`02_application_healthcare_pubmedqa.ipynb`](examples/applications/02_application_healthcare_pubmedqa.ipynb) |
|
|
248
|
-
| Finance: QA grounded in real SEC 10-K filings | [`03_application_finance_10k_qa.ipynb`](examples/applications/03_application_finance_10k_qa.ipynb) |
|
|
249
|
-
| E-commerce: review-sentiment triage at scale | [`04_application_ecommerce_review_triage.ipynb`](examples/applications/04_application_ecommerce_review_triage.ipynb) |
|
|
250
|
-
| Education: auto-graded tutoring, correctness vs. explanation | [`05_application_education_arc_tutor.ipynb`](examples/applications/05_application_education_arc_tutor.ipynb) |
|
|
251
|
-
| Enterprise search: internal knowledge assistant (real retrieval + RAG grounding) | [`06_application_enterprise_search_rag.ipynb`](examples/applications/06_application_enterprise_search_rag.ipynb) |
|
|
252
|
-
| LLM-as-judge via a real BERT NLI classifier, not a generative model | [`07_application_bert_nli_judge.ipynb`](examples/applications/07_application_bert_nli_judge.ipynb) |
|
|
246
|
+
| How much does pruning severity (20%/40%/60%) degrade a model? Real BoolQ, real annotator, ship/no-ship verdicts, cross-checked via `ak.run_lmeval()` | [`01_application_pruned_llama_boolq.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/01_application_pruned_llama_boolq.ipynb) |
|
|
247
|
+
| Healthcare: clinical QA correctness vs. grounding (PubMedQA) | [`02_application_healthcare_pubmedqa.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/02_application_healthcare_pubmedqa.ipynb) |
|
|
248
|
+
| Finance: QA grounded in real SEC 10-K filings | [`03_application_finance_10k_qa.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/03_application_finance_10k_qa.ipynb) |
|
|
249
|
+
| E-commerce: review-sentiment triage at scale | [`04_application_ecommerce_review_triage.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/04_application_ecommerce_review_triage.ipynb) |
|
|
250
|
+
| Education: auto-graded tutoring, correctness vs. explanation | [`05_application_education_arc_tutor.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/05_application_education_arc_tutor.ipynb) |
|
|
251
|
+
| Enterprise search: internal knowledge assistant (real retrieval + RAG grounding) | [`06_application_enterprise_search_rag.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/06_application_enterprise_search_rag.ipynb) |
|
|
252
|
+
| LLM-as-judge via a real BERT NLI classifier, not a generative model | [`07_application_bert_nli_judge.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/07_application_bert_nli_judge.ipynb) |
|
|
253
253
|
|
|
254
254
|
## Repository map
|
|
255
255
|
|
|
256
256
|
| Directory | Description | README |
|
|
257
257
|
|-----------|-------------|--------|
|
|
258
|
-
| [`src/auditkit/`](src/auditkit/) | Core library — spine, runner, metrics, models, CLI | [README](src/auditkit/README.md) |
|
|
259
|
-
| [`src/auditkit/metrics/`](src/auditkit/metrics/) | Metric families | [README](src/auditkit/metrics/README.md) |
|
|
260
|
-
| [`src/auditkit/model/`](src/auditkit/model/) | Model backends (echo, openai, hf, vllm, etc.) | [README](src/auditkit/model/README.md) |
|
|
261
|
-
| [`src/auditkit/redteam/`](src/auditkit/redteam/) | Red team probes and detectors | [README](src/auditkit/redteam/README.md) |
|
|
262
|
-
| [`src/auditkit/scenarios/`](src/auditkit/scenarios/) | Built-in benchmark datasets | [README](src/auditkit/scenarios/README.md) |
|
|
263
|
-
| [`tests/`](tests/) | Test suite (~3,000 tests) | [README](tests/README.md) |
|
|
264
|
-
| [`examples/`](examples/) | Colab-ready example notebooks | [README](examples/README.md) |
|
|
265
|
-
| [`examples/applications/`](examples/applications/) | Real-world application notebooks | [README](examples/applications/README.md) |
|
|
266
|
-
| [`docs/`](docs/) | MkDocs source for [auditkit.lexsi.ai](https://auditkit.lexsi.ai/) | [index](docs/index.md) |
|
|
258
|
+
| [`src/auditkit/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/) | Core library — spine, runner, metrics, models, CLI | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/README.md) |
|
|
259
|
+
| [`src/auditkit/metrics/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/metrics/) | Metric families | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/metrics/README.md) |
|
|
260
|
+
| [`src/auditkit/model/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/model/) | Model backends (echo, openai, hf, vllm, etc.) | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/model/README.md) |
|
|
261
|
+
| [`src/auditkit/redteam/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/redteam/) | Red team probes and detectors | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/redteam/README.md) |
|
|
262
|
+
| [`src/auditkit/scenarios/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/scenarios/) | Built-in benchmark datasets | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/scenarios/README.md) |
|
|
263
|
+
| [`tests/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/tests/) | Test suite (~3,000 tests) | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/tests/README.md) |
|
|
264
|
+
| [`examples/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/examples/) | Colab-ready example notebooks | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/README.md) |
|
|
265
|
+
| [`examples/applications/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/examples/applications/) | Real-world application notebooks | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/README.md) |
|
|
266
|
+
| [`docs/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/docs/) | MkDocs source for [auditkit.lexsi.ai](https://auditkit.lexsi.ai/) | [index](https://github.com/Lexsi-Labs/AuditKIT/blob/main/docs/index.md) |
|
|
267
267
|
|
|
268
268
|
---
|
|
269
269
|
|
|
@@ -288,7 +288,7 @@ If you use AuditKIT in research, please cite it:
|
|
|
288
288
|
Kadiyala, Ram Mohan Rao and Gosalia, Hem and Sankarapu, Vinay Kumar and
|
|
289
289
|
Seth, Pratinav},
|
|
290
290
|
year = {2026},
|
|
291
|
-
version = {1.2
|
|
291
|
+
version = {1.1.2},
|
|
292
292
|
url = {https://github.com/Lexsi-Labs/AuditKIT}
|
|
293
293
|
}
|
|
294
294
|
```
|
|
@@ -299,11 +299,11 @@ If you use AuditKIT in research, please cite it:
|
|
|
299
299
|
|
|
300
300
|
## License
|
|
301
301
|
|
|
302
|
-
This project is released under the [Lexsi Labs Source Available License (LSAL) v1.2](LICENSE.md) — free for academic research and teaching on MIT-like terms; use by any organization requires written acknowledgement or permission (Section 1A); a separate commercial license is required to sell it or embed it in a paid product.
|
|
302
|
+
This project is released under the [Lexsi Labs Source Available License (LSAL) v1.2](https://github.com/Lexsi-Labs/AuditKIT/blob/main/LICENSE.md) — free for academic research and teaching on MIT-like terms; use by any organization requires written acknowledgement or permission (Section 1A); a separate commercial license is required to sell it or embed it in a paid product.
|
|
303
303
|
|
|
304
304
|
---
|
|
305
305
|
|
|
306
306
|
## Join Community / Contribute
|
|
307
307
|
|
|
308
308
|
- Issues and discussions are welcomed on the [GitHub issue tracker](https://github.com/Lexsi-Labs/AuditKIT/issues).
|
|
309
|
-
- See [Contributing](docs/community/contributing.md) for the development setup, code standards and the pull request process.
|
|
309
|
+
- See [Contributing](https://github.com/Lexsi-Labs/AuditKIT/blob/main/docs/community/contributing.md) for the development setup, code standards and the pull request process.
|