auditkit 1.1.0__tar.gz → 1.1.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (422) hide show
  1. {auditkit-1.1.0 → auditkit-1.1.2}/CHANGELOG.md +31 -21
  2. {auditkit-1.1.0 → auditkit-1.1.2}/CITATION.cff +1 -1
  3. {auditkit-1.1.0 → auditkit-1.1.2}/PKG-INFO +44 -42
  4. {auditkit-1.1.0 → auditkit-1.1.2}/README.md +41 -41
  5. {auditkit-1.1.0 → auditkit-1.1.2}/docs/GUIDE.md +76 -59
  6. {auditkit-1.1.0 → auditkit-1.1.2}/docs/HF_ENDPOINT.md +1 -4
  7. {auditkit-1.1.0 → auditkit-1.1.2}/docs/SGLANG.md +1 -3
  8. {auditkit-1.1.0 → auditkit-1.1.2}/docs/VLLM_KNOWN_ISSUES.md +30 -27
  9. {auditkit-1.1.0 → auditkit-1.1.2}/docs/agent_eval.md +26 -81
  10. {auditkit-1.1.0 → auditkit-1.1.2}/docs/api.md +9 -0
  11. {auditkit-1.1.0 → auditkit-1.1.2}/docs/architecture/data-model.md +22 -4
  12. {auditkit-1.1.0 → auditkit-1.1.2}/docs/cli.md +14 -4
  13. {auditkit-1.1.0 → auditkit-1.1.2}/docs/community/changelog.md +148 -10
  14. {auditkit-1.1.0 → auditkit-1.1.2}/docs/configuration.md +3 -1
  15. {auditkit-1.1.0 → auditkit-1.1.2}/docs/encoder_judge.md +6 -35
  16. {auditkit-1.1.0 → auditkit-1.1.2}/docs/encoder_judge_prebuilts.md +17 -37
  17. {auditkit-1.1.0 → auditkit-1.1.2}/docs/guard_models.md +1 -4
  18. {auditkit-1.1.0 → auditkit-1.1.2}/docs/index.md +3 -3
  19. auditkit-1.1.2/docs/known_issues.md +29 -0
  20. {auditkit-1.1.0 → auditkit-1.1.2}/docs/metrics.md +42 -9
  21. {auditkit-1.1.0 → auditkit-1.1.2}/docs/rag_stress.md +25 -35
  22. {auditkit-1.1.0 → auditkit-1.1.2}/docs/scorers_reference.md +26 -6
  23. {auditkit-1.1.0 → auditkit-1.1.2}/examples/README.md +4 -1
  24. {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/README.md +9 -20
  25. {auditkit-1.1.0 → auditkit-1.1.2}/examples/cohere/README.md +1 -11
  26. {auditkit-1.1.0 → auditkit-1.1.2}/examples/colab/README.md +1 -0
  27. {auditkit-1.1.0 → auditkit-1.1.2}/mkdocs.yml +2 -6
  28. {auditkit-1.1.0 → auditkit-1.1.2}/pyproject.toml +3 -2
  29. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/README.md +18 -3
  30. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/__init__.py +1 -1
  31. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/cli.py +6 -2
  32. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/README.md +4 -2
  33. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/README.md +1 -0
  34. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/sample.py +1 -1
  35. auditkit-1.1.2/tests/test_cli_yaml_model.py +37 -0
  36. auditkit-1.1.0/docs/BUGS.md +0 -341
  37. auditkit-1.1.0/docs/COMPARISON.docx +0 -0
  38. auditkit-1.1.0/docs/COMPARISON.md +0 -185
  39. auditkit-1.1.0/docs/RELEASE.md +0 -701
  40. auditkit-1.1.0/docs/about/product.md +0 -204
  41. auditkit-1.1.0/docs/known_issues.md +0 -120
  42. auditkit-1.1.0/examples/applications/SUMMARY.md +0 -96
  43. {auditkit-1.1.0 → auditkit-1.1.2}/.gitattributes +0 -0
  44. {auditkit-1.1.0 → auditkit-1.1.2}/.github/workflows/check-release-version.yml +0 -0
  45. {auditkit-1.1.0 → auditkit-1.1.2}/.github/workflows/patch-release.yml +0 -0
  46. {auditkit-1.1.0 → auditkit-1.1.2}/.gitignore +0 -0
  47. {auditkit-1.1.0 → auditkit-1.1.2}/AGENTS.md +0 -0
  48. {auditkit-1.1.0 → auditkit-1.1.2}/LICENSE.md +0 -0
  49. {auditkit-1.1.0 → auditkit-1.1.2}/docs/CNAME +0 -0
  50. {auditkit-1.1.0 → auditkit-1.1.2}/docs/about/glossary.md +0 -0
  51. {auditkit-1.1.0 → auditkit-1.1.2}/docs/about/platform-integration.md +0 -0
  52. {auditkit-1.1.0 → auditkit-1.1.2}/docs/agents_and_rag.md +0 -0
  53. {auditkit-1.1.0 → auditkit-1.1.2}/docs/annotators.md +0 -0
  54. {auditkit-1.1.0 → auditkit-1.1.2}/docs/architecture/system-design.md +0 -0
  55. {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/auditkit-logo-black.png +0 -0
  56. {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/auditkit-logo-dark.png +0 -0
  57. {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/auditkit-logo-light.png +0 -0
  58. {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/auditkit-logo-white.png +0 -0
  59. {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/extra.css +0 -0
  60. {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/extra.js +0 -0
  61. {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/favicon.png +0 -0
  62. {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/icon.png +0 -0
  63. {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/icon.svg +0 -0
  64. {auditkit-1.1.0 → auditkit-1.1.2}/docs/assets/logo.svg +0 -0
  65. {auditkit-1.1.0 → auditkit-1.1.2}/docs/best_practices/scorers.md +0 -0
  66. {auditkit-1.1.0 → auditkit-1.1.2}/docs/community/contributing.md +0 -0
  67. {auditkit-1.1.0 → auditkit-1.1.2}/docs/community/faq.md +0 -0
  68. {auditkit-1.1.0 → auditkit-1.1.2}/docs/community/roadmap.md +0 -0
  69. {auditkit-1.1.0 → auditkit-1.1.2}/docs/cookbook/llm_judge.md +0 -0
  70. {auditkit-1.1.0 → auditkit-1.1.2}/docs/cookbook/rag_evaluation.md +0 -0
  71. {auditkit-1.1.0 → auditkit-1.1.2}/docs/development.md +0 -0
  72. {auditkit-1.1.0 → auditkit-1.1.2}/docs/examples/custom_scorer.md +0 -0
  73. {auditkit-1.1.0 → auditkit-1.1.2}/docs/examples/experiments.md +0 -0
  74. {auditkit-1.1.0 → auditkit-1.1.2}/docs/examples/many_datasets.md +0 -0
  75. {auditkit-1.1.0 → auditkit-1.1.2}/docs/examples/model_comparison.md +0 -0
  76. {auditkit-1.1.0 → auditkit-1.1.2}/docs/examples/quickstart.md +0 -0
  77. {auditkit-1.1.0 → auditkit-1.1.2}/docs/experiments.md +0 -0
  78. {auditkit-1.1.0 → auditkit-1.1.2}/docs/getting_started.md +0 -0
  79. {auditkit-1.1.0 → auditkit-1.1.2}/docs/integrations.md +0 -0
  80. {auditkit-1.1.0 → auditkit-1.1.2}/docs/lexsi_integration.md +0 -0
  81. {auditkit-1.1.0 → auditkit-1.1.2}/docs/llm_judge_rendering.md +0 -0
  82. {auditkit-1.1.0 → auditkit-1.1.2}/docs/model_backends.md +0 -0
  83. {auditkit-1.1.0 → auditkit-1.1.2}/docs/reports.md +0 -0
  84. {auditkit-1.1.0 → auditkit-1.1.2}/docs/requirements.txt +0 -0
  85. {auditkit-1.1.0 → auditkit-1.1.2}/docs/scenarios.md +0 -0
  86. {auditkit-1.1.0 → auditkit-1.1.2}/examples/01_full_evaluation_pipeline.ipynb +0 -0
  87. {auditkit-1.1.0 → auditkit-1.1.2}/examples/02_generation_across_hf_families.ipynb +0 -0
  88. {auditkit-1.1.0 → auditkit-1.1.2}/examples/03_custom_annotators.ipynb +0 -0
  89. {auditkit-1.1.0 → auditkit-1.1.2}/examples/04_metrics_deep_dive.ipynb +0 -0
  90. {auditkit-1.1.0 → auditkit-1.1.2}/examples/05_data_types.ipynb +0 -0
  91. {auditkit-1.1.0 → auditkit-1.1.2}/examples/06_model_comparison.ipynb +0 -0
  92. {auditkit-1.1.0 → auditkit-1.1.2}/examples/07_annotators_across_models.ipynb +0 -0
  93. {auditkit-1.1.0 → auditkit-1.1.2}/examples/08_compare_result_deep_dive.ipynb +0 -0
  94. {auditkit-1.1.0 → auditkit-1.1.2}/examples/09_guard_judge_implementation_check.ipynb +0 -0
  95. {auditkit-1.1.0 → auditkit-1.1.2}/examples/10_guard_judge_legit_models.ipynb +0 -0
  96. {auditkit-1.1.0 → auditkit-1.1.2}/examples/11_encoder_judge_prebuilts.ipynb +0 -0
  97. {auditkit-1.1.0 → auditkit-1.1.2}/examples/12_performance_metrics_demo.ipynb +0 -0
  98. {auditkit-1.1.0 → auditkit-1.1.2}/examples/13_sglang_compatibility.ipynb +0 -0
  99. {auditkit-1.1.0 → auditkit-1.1.2}/examples/14_agent_and_rag_evals.ipynb +0 -0
  100. {auditkit-1.1.0 → auditkit-1.1.2}/examples/15_agentic_rag_metrics_walkthrough.ipynb +0 -0
  101. {auditkit-1.1.0 → auditkit-1.1.2}/examples/16_agentic_endpoints_live.ipynb +0 -0
  102. {auditkit-1.1.0 → auditkit-1.1.2}/examples/17_multi_route_references.ipynb +0 -0
  103. {auditkit-1.1.0 → auditkit-1.1.2}/examples/agent_eval_offline.py +0 -0
  104. {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/01_application_pruned_llama_boolq.ipynb +0 -0
  105. {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/02_application_healthcare_pubmedqa.ipynb +0 -0
  106. {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/03_application_finance_10k_qa.ipynb +0 -0
  107. {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/04_application_ecommerce_review_triage.ipynb +0 -0
  108. {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/05_application_education_arc_tutor.ipynb +0 -0
  109. {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/06_application_enterprise_search_rag.ipynb +0 -0
  110. {auditkit-1.1.0 → auditkit-1.1.2}/examples/applications/07_application_bert_nli_judge.ipynb +0 -0
  111. {auditkit-1.1.0 → auditkit-1.1.2}/examples/auditkit.yaml +0 -0
  112. {auditkit-1.1.0 → auditkit-1.1.2}/examples/cohere/compare_models.ipynb +0 -0
  113. {auditkit-1.1.0 → auditkit-1.1.2}/examples/cohere/metrics_generation.ipynb +0 -0
  114. {auditkit-1.1.0 → auditkit-1.1.2}/examples/cohere/run_lmeval.ipynb +0 -0
  115. {auditkit-1.1.0 → auditkit-1.1.2}/examples/cohere/sglang_auditkit.ipynb +0 -0
  116. {auditkit-1.1.0 → auditkit-1.1.2}/examples/cohere/tgi_endpoint.ipynb +0 -0
  117. {auditkit-1.1.0 → auditkit-1.1.2}/examples/colab/01_cohere_and_hf_api_models.ipynb +0 -0
  118. {auditkit-1.1.0 → auditkit-1.1.2}/examples/colab/02_agentic_rag_real_agent.ipynb +0 -0
  119. {auditkit-1.1.0 → auditkit-1.1.2}/examples/colab/03_vllm_integration.ipynb +0 -0
  120. {auditkit-1.1.0 → auditkit-1.1.2}/examples/colab/04_cohere_hf_fixes_live.ipynb +0 -0
  121. {auditkit-1.1.0 → auditkit-1.1.2}/examples/colab/05_real_world_agentic_rag.ipynb +0 -0
  122. {auditkit-1.1.0 → auditkit-1.1.2}/examples/colab/06_cohere_all_backends.ipynb +0 -0
  123. {auditkit-1.1.0 → auditkit-1.1.2}/examples/model_comparison_notebook.py +0 -0
  124. {auditkit-1.1.0 → auditkit-1.1.2}/examples/stress_tests/README.md +0 -0
  125. {auditkit-1.1.0 → auditkit-1.1.2}/examples/stress_tests/test_stress_gated_model.py +0 -0
  126. {auditkit-1.1.0 → auditkit-1.1.2}/examples/stress_tests/test_stress_open_model_load.py +0 -0
  127. {auditkit-1.1.0 → auditkit-1.1.2}/scripts/__init__.py +0 -0
  128. {auditkit-1.1.0 → auditkit-1.1.2}/scripts/release.py +0 -0
  129. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/__main__.py +0 -0
  130. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/_bootstrap.py +0 -0
  131. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/_identity_guard.py +0 -0
  132. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/adapter.py +0 -0
  133. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/README.md +0 -0
  134. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/__init__.py +0 -0
  135. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/importers.py +0 -0
  136. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/lexsi.py +0 -0
  137. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/outcome.py +0 -0
  138. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/reliability.py +0 -0
  139. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/report.py +0 -0
  140. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/runner.py +0 -0
  141. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/sidecar.py +0 -0
  142. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/agent_eval/types.py +0 -0
  143. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/annotator.py +0 -0
  144. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/api.py +0 -0
  145. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/assets/auditkit_logo.png +0 -0
  146. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/bfcl.py +0 -0
  147. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/cache.py +0 -0
  148. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/comparison.py +0 -0
  149. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/compat.py +0 -0
  150. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/diff.py +0 -0
  151. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/errors.py +0 -0
  152. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/evaluator.py +0 -0
  153. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/experiment.py +0 -0
  154. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/hf_publish.py +0 -0
  155. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/lexsi_login.py +0 -0
  156. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/lexsi_relay.py +0 -0
  157. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/lmeval_engine.py +0 -0
  158. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/loaders.py +0 -0
  159. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/logs.py +0 -0
  160. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metric.py +0 -0
  161. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/__init__.py +0 -0
  162. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/agent.py +0 -0
  163. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/code.py +0 -0
  164. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/embedding.py +0 -0
  165. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/encoder_judge.py +0 -0
  166. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/generation.py +0 -0
  167. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/guard.py +0 -0
  168. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/hallucination.py +0 -0
  169. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/judge.py +0 -0
  170. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/pairwise.py +0 -0
  171. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/perf.py +0 -0
  172. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/rag.py +0 -0
  173. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/rag_judge.py +0 -0
  174. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/rag_stress.py +0 -0
  175. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/rag_stress_ops.py +0 -0
  176. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/rag_stress_runner.py +0 -0
  177. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/retrieval.py +0 -0
  178. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/security.py +0 -0
  179. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/metrics/toxicity.py +0 -0
  180. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/__init__.py +0 -0
  181. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/agent_endpoint.py +0 -0
  182. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/anthropic.py +0 -0
  183. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/api_gen.py +0 -0
  184. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/groq_gen.py +0 -0
  185. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/hf_gen.py +0 -0
  186. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/lexsi.py +0 -0
  187. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/litellm_gen.py +0 -0
  188. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/openai.py +0 -0
  189. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/openrouter_gen.py +0 -0
  190. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model/vllm_gen.py +0 -0
  191. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/model_compare.py +0 -0
  192. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/provenance.py +0 -0
  193. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/README.md +0 -0
  194. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/__init__.py +0 -0
  195. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/detector.py +0 -0
  196. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/detectors/README.md +0 -0
  197. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/detectors/builtin.py +0 -0
  198. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/probe.py +0 -0
  199. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/probes/README.md +0 -0
  200. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/probes/builtin.py +0 -0
  201. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/redteam/runner.py +0 -0
  202. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/registry.py +0 -0
  203. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/report.py +0 -0
  204. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/report_format.py +0 -0
  205. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/router.py +0 -0
  206. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/runner.py +0 -0
  207. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/runspec.py +0 -0
  208. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenario.py +0 -0
  209. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/README.md +0 -0
  210. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/__init__.py +0 -0
  211. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/arc.py +0 -0
  212. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/gsm8k.py +0 -0
  213. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/hellaswag.py +0 -0
  214. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/humaneval.py +0 -0
  215. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/mmlu.py +0 -0
  216. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scenarios/truthfulqa.py +0 -0
  217. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/score.py +0 -0
  218. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scorers.py +0 -0
  219. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/scoring.py +0 -0
  220. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/trace.py +0 -0
  221. {auditkit-1.1.0 → auditkit-1.1.2}/src/auditkit/types.py +0 -0
  222. {auditkit-1.1.0 → auditkit-1.1.2}/tests/README.md +0 -0
  223. {auditkit-1.1.0 → auditkit-1.1.2}/tests/__init__.py +0 -0
  224. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/.gitignore +0 -0
  225. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/FINDINGS.md +0 -0
  226. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/README.md +0 -0
  227. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/__init__.py +0 -0
  228. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/catalog.py +0 -0
  229. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/colab_model_matrix.ipynb +0 -0
  230. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/colab_offline_suite.ipynb +0 -0
  231. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/conftest.py +0 -0
  232. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/make_metric_reports.py +0 -0
  233. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_agentic_pipeline_all_metrics.py +0 -0
  234. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_fingerprint_offline.py +0 -0
  235. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_formats_offline.py +0 -0
  236. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_live.py +0 -0
  237. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_live_rag.py +0 -0
  238. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_metrics_offline.py +0 -0
  239. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_pipeline_offline.py +0 -0
  240. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_rag_offline.py +0 -0
  241. {auditkit-1.1.0 → auditkit-1.1.2}/tests/agentic_suite/test_resolved_issues.py +0 -0
  242. {auditkit-1.1.0 → auditkit-1.1.2}/tests/conftest.py +0 -0
  243. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/chat_templates/aya_expanse.jinja +0 -0
  244. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/chat_templates/command_r7b_tool_use.jinja +0 -0
  245. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/chat_templates/tiny_aya_default.jinja +0 -0
  246. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/agenttune/dataset_train_grpo.jsonl +0 -0
  247. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/agenttune/report_constructed.json +0 -0
  248. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/agenttune/trace_finder.jsonl +0 -0
  249. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/agenttune/trace_hotpotqa.jsonl +0 -0
  250. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/agenttune/trajectory_store_export.jsonl +0 -0
  251. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/aligntune/AuditReport.json +0 -0
  252. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/aligntune/RegressionReport.json +0 -0
  253. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/circuitkit/faithfulness_report.json +0 -0
  254. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/circuitkit/faithfulness_report_default.json +0 -0
  255. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/circuitkit/faithfulness_report_ib_fastpath.json +0 -0
  256. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/circuitkit/faithfulness_report_nonfinite.json +0 -0
  257. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/circuitkit/faithfulness_report_unknown_sentinels.json +0 -0
  258. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/checkpoint_TextCleaner.jsonl +0 -0
  259. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/data_sample.json +0 -0
  260. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/data_sample_pdf.json +0 -0
  261. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/manifest.json +0 -0
  262. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/manifest_pdf.json +0 -0
  263. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/rejected.jsonl +0 -0
  264. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/rejected_pdf.jsonl +0 -0
  265. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/sft_alpaca.jsonl +0 -0
  266. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/curatorkit/sft_sharegpt.jsonl +0 -0
  267. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/safetune/placeholder-model__base__placeholder_bench.jsonl +0 -0
  268. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/safetune/placeholder-model__base__placeholder_bench_scored.jsonl +0 -0
  269. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/safetune/safety_audit_event__post_allow_long_redacted.json +0 -0
  270. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/safetune/safety_audit_event__post_keyword_classifier_block.json +0 -0
  271. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/safetune/safety_audit_event__pre_allow.json +0 -0
  272. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/lexsi_real/safetune/safety_audit_event__pre_tool_block.json +0 -0
  273. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/sglang_0520/commandr_excerpt.py +0 -0
  274. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/sglang_0520/transformers_backend_excerpt.py +0 -0
  275. {auditkit-1.1.0 → auditkit-1.1.2}/tests/fixtures/sglang_0520/unified_radix_cache_excerpt.py +0 -0
  276. {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/README.md +0 -0
  277. {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/__init__.py +0 -0
  278. {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/colab_cohere_models.ipynb +0 -0
  279. {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/colab_integrations.ipynb +0 -0
  280. {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/colab_sglang.ipynb +0 -0
  281. {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/colab_sglang_32b_north.ipynb +0 -0
  282. {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/colab_sglang_cohere_matrix.ipynb +0 -0
  283. {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/colab_vllm_32b.ipynb +0 -0
  284. {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/colab_vllm_cohere_matrix.ipynb +0 -0
  285. {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/conftest.py +0 -0
  286. {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/test_agenttune.py +0 -0
  287. {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/test_cohere.py +0 -0
  288. {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/test_hf.py +0 -0
  289. {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/test_live.py +0 -0
  290. {auditkit-1.1.0 → auditkit-1.1.2}/tests/integrations_suite/test_sglang.py +0 -0
  291. {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/README.md +0 -0
  292. {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/__init__.py +0 -0
  293. {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/conftest.py +0 -0
  294. {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/serve_model.py +0 -0
  295. {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/test_issue43_specs.py +0 -0
  296. {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/test_issue44_user_flows.py +0 -0
  297. {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/test_issue45_bfcl.py +0 -0
  298. {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/test_issue45_routes.py +0 -0
  299. {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/test_live.py +0 -0
  300. {auditkit-1.1.0 → auditkit-1.1.2}/tests/issues_suite/test_vllm_cohere_templates.py +0 -0
  301. {auditkit-1.1.0 → auditkit-1.1.2}/tests/nonmetric_suite/README.md +0 -0
  302. {auditkit-1.1.0 → auditkit-1.1.2}/tests/nonmetric_suite/__init__.py +0 -0
  303. {auditkit-1.1.0 → auditkit-1.1.2}/tests/nonmetric_suite/conftest.py +0 -0
  304. {auditkit-1.1.0 → auditkit-1.1.2}/tests/nonmetric_suite/test_agent_eval_and_loaders.py +0 -0
  305. {auditkit-1.1.0 → auditkit-1.1.2}/tests/nonmetric_suite/test_mrm_evidence.py +0 -0
  306. {auditkit-1.1.0 → auditkit-1.1.2}/tests/nonmetric_suite/test_redteam_machinery.py +0 -0
  307. {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/README.md +0 -0
  308. {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/__init__.py +0 -0
  309. {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/conftest.py +0 -0
  310. {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_edge_agent_and_rag_judges.py +0 -0
  311. {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_edge_mrm.py +0 -0
  312. {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_edge_rag_stress.py +0 -0
  313. {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_edge_redteam_judges.py +0 -0
  314. {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_edge_uncertainty.py +0 -0
  315. {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_pr1_additions.py +0 -0
  316. {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_pr2_redteam.py +0 -0
  317. {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_pr3_mrm.py +0 -0
  318. {auditkit-1.1.0 → auditkit-1.1.2}/tests/pipeline_suite/test_pr4_uncertainty.py +0 -0
  319. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_adapter.py +0 -0
  320. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_adapter_params.py +0 -0
  321. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_endpoint.py +0 -0
  322. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_cli.py +0 -0
  323. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_disagreement.py +0 -0
  324. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_eventlog.py +0 -0
  325. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_harness.py +0 -0
  326. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_importers.py +0 -0
  327. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_outcome.py +0 -0
  328. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_reliability.py +0 -0
  329. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_runner.py +0 -0
  330. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_secrets.py +0 -0
  331. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_sidecar.py +0 -0
  332. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_eval_types.py +0 -0
  333. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_metrics.py +0 -0
  334. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agent_reference_free.py +0 -0
  335. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agenttune_fix_plan.py +0 -0
  336. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_agenttune_integration.py +0 -0
  337. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_annotator.py +0 -0
  338. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_api.py +0 -0
  339. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_api_backend.py +0 -0
  340. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_api_images.py +0 -0
  341. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_api_tool_calls.py +0 -0
  342. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_api_unverified_fields.py +0 -0
  343. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_benchmark.py +0 -0
  344. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_chat_messages.py +0 -0
  345. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_chat_template_kwargs_everywhere.py +0 -0
  346. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_cli.py +0 -0
  347. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_code_metrics.py +0 -0
  348. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_cohere_aya.py +0 -0
  349. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_cohere_hf_fix_plan.py +0 -0
  350. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_cohere_tool_calls.py +0 -0
  351. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_compare.py +0 -0
  352. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_comparison.py +0 -0
  353. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_compat.py +0 -0
  354. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_core_engine.py +0 -0
  355. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_custom_judge_pipeline.py +0 -0
  356. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_diff.py +0 -0
  357. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_embedding_metrics.py +0 -0
  358. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_encoder_judge.py +0 -0
  359. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_env_compat_fix_plan.py +0 -0
  360. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_errors.py +0 -0
  361. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_evaluate_many.py +0 -0
  362. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_eventlog_action_shapes.py +0 -0
  363. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_execute_concurrency.py +0 -0
  364. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_experiment.py +0 -0
  365. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_fingerprint_stability.py +0 -0
  366. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_generation_kwargs.py +0 -0
  367. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_generation_metrics.py +0 -0
  368. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_guard.py +0 -0
  369. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_hallucination_metrics.py +0 -0
  370. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_hf_dtype_fallback.py +0 -0
  371. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_hf_gen_logits_sanitizer.py +0 -0
  372. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_hf_interop.py +0 -0
  373. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_hf_wire_format.py +0 -0
  374. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_integration.py +0 -0
  375. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_judge.py +0 -0
  376. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_judge_flow.py +0 -0
  377. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_lexsi_compat.py +0 -0
  378. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_lexsi_relay.py +0 -0
  379. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_llm_judge.py +0 -0
  380. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_lmeval_engine.py +0 -0
  381. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_lmeval_real_pipeline.py +0 -0
  382. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_load_dataset.py +0 -0
  383. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_loaders.py +0 -0
  384. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_loaders_agent.py +0 -0
  385. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_metric.py +0 -0
  386. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_metrics_via_pipeline.py +0 -0
  387. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_model.py +0 -0
  388. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_model_backends.py +0 -0
  389. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_multipath_references.py +0 -0
  390. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_pairwise_metrics.py +0 -0
  391. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_pipeline_live_hf.py +0 -0
  392. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_practical.py +0 -0
  393. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_provenance.py +0 -0
  394. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_rag_judge.py +0 -0
  395. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_rag_metrics.py +0 -0
  396. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_rag_reference_free.py +0 -0
  397. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_rag_stress.py +0 -0
  398. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_rag_stress_ops.py +0 -0
  399. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_rag_stress_runner.py +0 -0
  400. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_redteam.py +0 -0
  401. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_reference_fingerprint.py +0 -0
  402. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_registry.py +0 -0
  403. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_release.py +0 -0
  404. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_retrieval_metrics.py +0 -0
  405. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_review27_followups.py +0 -0
  406. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_route_distribution.py +0 -0
  407. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_router.py +0 -0
  408. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_runner.py +0 -0
  409. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_sample.py +0 -0
  410. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_scenario_dataset_ids.py +0 -0
  411. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_scenarios.py +0 -0
  412. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_score.py +0 -0
  413. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_security_perf.py +0 -0
  414. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_template_kwargs_f1_fix_plan.py +0 -0
  415. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_token_usage_and_cost.py +0 -0
  416. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_toxicity_metrics.py +0 -0
  417. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_trace.py +0 -0
  418. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_track3_agenttune.py +0 -0
  419. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_types.py +0 -0
  420. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_unknown_architecture_message.py +0 -0
  421. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_unscored_results.py +0 -0
  422. {auditkit-1.1.0 → auditkit-1.1.2}/tests/test_vllm_offline_engine.py +0 -0
@@ -11,19 +11,18 @@ All notable changes to AuditKIT. The format follows [Keep a Changelog](https://k
11
11
  SGLang), e.g. `{"enable_thinking": False}` for Qwen3. A request's own values win.
12
12
  Part of the run fingerprint.
13
13
  - `api:` says when a server may have ignored `chat_template_kwargs` or
14
- `parallel_tool_calls` (#43): `RunResult.metadata["unverified_request_fields"]`, a
14
+ `parallel_tool_calls`: `RunResult.metadata["unverified_request_fields"]`, a
15
15
  `summary()` line, one warning, and `cap_unverified` on the parallel scores. The
16
16
  server is probed once (`GET /models` `owned_by`) and recorded in
17
17
  `metadata["api_server"]`; only `APIModel(server="sglang"|"vllm")` clears the note.
18
18
  New `Model.run_notes()` hook for backend facts about a run.
19
- - `ak.load_bfcl()` and `ak.bfcl_arg_match` (#45 phase 2): BFCL v3's 11 single-turn
19
+ - `ak.load_bfcl()` and `ak.bfcl_arg_match`: BFCL v3's 11 single-turn
20
20
  categories plus the two irrelevance ones, read from the Hub files (JSON Lines). The
21
21
  accepted-value lists stay on the reference, and `bfcl_arg_match` compares against
22
22
  them as BFCL's AST checker does. It handles a drifted answer id, flags unsatisfiable
23
23
  answers, passes system prompts through, converts BFCL types to JSON Schema, and has an
24
24
  optional `dots_to_underscores`. Multi-turn is refused: its answers are call strings.
25
- Counts at the pinned revision are in `docs/notes/bfcl-possible-answer-counts.md`.
26
- - `RunResult.route_distribution()` and a `summary()` line (#45): for samples whose
25
+ - `RunResult.route_distribution()` and a `summary()` line: for samples whose
27
26
  reference names several routes (`any_of`), which route each took and which rule
28
27
  decided (`f1`, `recall`, `turn_structure`, `call_order`, `listed_order`), with the
29
28
  ties broken by listed order counted. Information only: it never reaches the stats.
@@ -41,7 +40,7 @@ All notable changes to AuditKIT. The format follows [Keep a Changelog](https://k
41
40
  layers' own `config.sliding_window`. Rule (m) does the same for SGLang's generic
42
41
  Transformers backend (Aya Vision: Triton decode crashed with `kv_indptr=None`),
43
42
  returning a window only for models with sliding-window layers.
44
- - The `vllm:` column covers all nine Cohere models (#52): the full matrix ran on an RTX PRO 6000
43
+ - The `vllm:` column covers all nine Cohere models: the full matrix ran on an RTX PRO 6000
45
44
  (CUDA 12.8) with the FlashInfer sampler fix, every check passing. Aya Expanse 32B and Aya
46
45
  Vision 32B score 5/5; the other seven match their A100 scores exactly.
47
46
  - `vllm:` on SM 12.x GPUs (RTX PRO 6000, RTX 50-series) with a CUDA toolkit older than 12.9:
@@ -61,18 +60,18 @@ All notable changes to AuditKIT. The format follows [Keep a Changelog](https://k
61
60
  32B: all nine Cohere models now run on SGLang 0.5.20. `colab_sglang_32b_north.ipynb`
62
61
  measures the two.
63
62
  - `docs/model_backends.md`: the `vllm:` column is measured for the seven Cohere models that fit
64
- a 40 GB A100 (#52; vLLM 0.30.0): Tiny Aya 4/5 each, North 5/5, Aya Expanse 8B and Aya Vision
63
+ a 40 GB A100 (vLLM 0.30.0): Tiny Aya 4/5 each, North 5/5, Aya Expanse 8B and Aya Vision
65
64
  8B 5/5, image checks passing. vLLM falls back to its Transformers backend for Aya Vision on
66
65
  its own.
67
66
  "Aya Vision loads on neither vLLM nor SGLang" is corrected: it loads on both, non-natively.
68
- The images section and README now say `api:` sends images (since #50) and the offline
69
- `vllm:` backend still drops them; the install trap names #51's cause (a stale auditkit 1.0.0).
67
+ The images section and README now say `api:` sends images and the offline
68
+ `vllm:` backend still drops them; the install trap names the cause (a stale auditkit 1.0.0).
70
69
  - `check_compat()` rule (j): with transformers >= 5.15 in the SGLang env (needed for North
71
70
  Micro Vision), SGLang's Transformers backend rejects the `embedding_rowwise` TP style
72
71
  that transformers adds for tied embeddings, so North still fails to load. The entry is
73
72
  inert there (only `nn.Linear` reads the plan); `--patch-sglang` maps it to `replicate`.
74
73
  - transformers' "does not recognize this architecture" is rewritten by `hf:` and `vllm:` to
75
- name the model type, the installed transformers and the version it needs (#51):
74
+ name the model type, the installed transformers and the version it needs:
76
75
  "'…North-Micro-Vision-Instruct' is a 'cohere_compass' model, and the installed
77
76
  transformers 4.57.6 does not know that architecture: it needs transformers>=5.15. The
78
77
  checkpoint is fine." The original read like a broken checkpoint. The real cause was a
@@ -108,15 +107,27 @@ All notable changes to AuditKIT. The format follows [Keep a Changelog](https://k
108
107
 
109
108
  ### Fixed
110
109
 
110
+ - `auditkit eval --config FILE` accepts `model:` from the config file. `--model` was
111
+ `required=True` on the parser, so argparse rejected the command with "the following
112
+ arguments are required: --model" before the YAML was read, even though `docs/cli.md`
113
+ documents exactly that invocation. It is now optional on the parser and checked after the
114
+ config merge; a genuinely missing model still exits 2, naming both ways to supply it.
111
115
  - The six built-in benchmark scenarios load again. They used bare Hub ids (`mmlu`, `arc` and
112
116
  `truthfulqa` no longer exist; the others failed with `HfUriError`) and now use `cais/mmlu`,
113
117
  `openai/gsm8k`, `allenai/ai2_arc`, `Rowan/hellaswag`, `truthfulqa/truthful_qa` and
114
118
  `openai/openai_humaneval`. HumanEval also referenced a `TaskKind.CODE` that doesn't exist; it loads
115
119
  as a generative task (no code-execution metric, so use `ak.run_lmeval("humaneval", ...)` for pass@k).
120
+ - Docs: the scorer reference and metrics overview cover all 68 registered metrics, and the
121
+ data model documents `Sample.images`, `RunConfig.seed`/`track_performance`/
122
+ `chat_template_kwargs` and `RunResult.unscored`/`metadata`; every `eval` flag is in
123
+ `docs/cli.md`; the extras table matches `pyproject.toml`; the claimed-but-unimplemented
124
+ features are gone, as are the internal pages that were still being published to the site.
125
+ - The `dev` extra declares `pyyaml`, which the `--config` tests need. PyYAML remains an
126
+ optional runtime dependency; the core install is still dependency-free.
116
127
  - Docs: no METEOR (never implemented) and no "Rich HTML reports" tick (not implemented); current
117
128
  Claude model ids instead of retired Claude 3 ones; "source-available", not "open-source"; no
118
- "hackathon" wording, links to a private repository or personal-repo issue numbers in public pages;
119
- GUIDE's table of contents, catalog heading and version line match its sections and 1.2.0.
129
+ internal wording, links to a private repository or private issue numbers in public pages;
130
+ GUIDE's table of contents, catalog heading and version line match its sections and 1.1.2.
120
131
 
121
132
  ### Changed
122
133
 
@@ -125,29 +136,28 @@ All notable changes to AuditKIT. The format follows [Keep a Changelog](https://k
125
136
  "triangle". **Scores change.** `F1Score(normalize=False)` keeps the old raw
126
137
  whitespace split; the setting is part of the run fingerprint.
127
138
  - `[lmeval]` extra: `accelerate>=1.0`.
128
- - **`abstention`: `short_reply_markers` now takes regular expressions** (#35). The
139
+ - **`abstention`: `short_reply_markers` now takes regular expressions**. The
129
140
  defaults require refusal-shaped wording ("insufficient information to/for/in/about…",
130
141
  "no relevant information/documents/context…"), so a short real answer such as
131
142
  "Yes, there is no relevant fee for domestic wires." is no longer a refusal, and a
132
143
  first sentence opening with "yes" never is. Bare phrases passed to
133
144
  `Abstention(short_reply_markers=...)` still work (a plain phrase is a valid regex);
134
145
  a phrase containing regex metacharacters must be escaped (`re.escape`).
135
- - **An unnamed recorded tool call is unscored, not a measured 0.0** (#36). An EventLog
146
+ - **An unnamed recorded tool call is unscored, not a measured 0.0**. An EventLog
136
147
  or AgentTune trajectory action that carries call arguments (`arguments`,
137
148
  `parameters`, `args`, `input`, `query`) but no tool name leaves
138
149
  `tool_call_names` unavailable, so `tool_call_f1` / `trajectory_match` /
139
150
  `parallel_tool_calls` are skipped, and `rescore` reports `trace.tool_call_names`.
140
- - **A trajectory whose actions are all empty is a measured zero calls** (#37): it
151
+ - **A trajectory whose actions are all empty is a measured zero calls**: it
141
152
  scores (0.0 against expected calls, 1.0 on a no-call case) instead of being dropped.
142
153
  - Docs: SGLang needs Python 3.10-3.12 (a uv 3.12 venv on 3.13 hosts); example 13
143
154
  builds its SGLang venv that way. `docs/VLLM_KNOWN_ISSUES.md` covers the Colab
144
155
  torchaudio mismatch after `[vllm]`.
145
156
 
146
- ## [1.2.0] - 2026-09-28
157
+ ## [1.1.2] - 2026-10-02
147
158
 
148
- Lexsi stack interop for the hackathon tracks (CuratorKIT, AlignTune, SafeTune,
149
- CircuitKIT and AgentTune outputs evaluate with no glue code), plus the
150
- hackathon readiness fixes for the permitted Cohere models.
159
+ Lexsi stack interop (CuratorKIT, AlignTune, SafeTune, CircuitKIT and AgentTune
160
+ outputs evaluate with no glue code), plus fixes for the Cohere models.
151
161
 
152
162
  ### Added
153
163
 
@@ -166,7 +176,7 @@ hackathon readiness fixes for the permitted Cohere models.
166
176
  - **Native tool calling on `hf:`.** Tool schemas go to the chat template
167
177
  (`apply_chat_template(tools=...)`), so `ToolCallAdapter()` works on local
168
178
  models whose template renders tools; a template without tool support raises
169
- `CapabilityError`. None of the Cohere hackathon models' templates (Tiny Aya,
179
+ `CapabilityError`. None of the Cohere models' templates (Tiny Aya,
170
180
  Aya Expanse, Aya Vision, North) render tools: use
171
181
  `ToolCallAdapter(mode="prompt")` with them.
172
182
  - **Cohere tool-call format.** The tool-call parser reads Command R7B
@@ -189,11 +199,11 @@ hackathon readiness fixes for the permitted Cohere models.
189
199
  ### Fixed
190
200
 
191
201
  - A JSON-string reference (a CSV cell, a JSONL field) now fingerprints like its
192
- decoded form (#44), in the dataset fingerprint and in `AgentCase.digest()`, so the
202
+ decoded form, in the dataset fingerprint and in `AgentCase.digest()`, so the
193
203
  same dataset read from a file reuses the cached run. Dict and list references keep
194
204
  their digests. `auditkit agent --dry-run` shows `refs=yes` for an explicit `[]`
195
205
  reference ("expect no call").
196
- - A bare prefix is now decided per backend (#43). `api:`, `hf:` and `vllm:` raise
206
+ - A bare prefix is now decided per backend. `api:`, `hf:` and `vllm:` raise
197
207
  "empty model spec" (`api:` used to send `{"model": null}`). The hosted prefixes
198
208
  (`openai:`, `anthropic:`, `groq:`, `openrouter:`, `lexsi:`, `litellm:`) fall back
199
209
  to their backend's default model, and `agent:` still takes its URL from `url=`.
@@ -8,7 +8,7 @@ abstract: |
8
8
  type: software
9
9
  message: "If you use this software, please cite it."
10
10
 
11
- version: 1.1.0
11
+ version: 1.1.2
12
12
 
13
13
  authors:
14
14
  - family-names: "Arora"
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: auditkit
3
- Version: 1.1.0
3
+ Version: 1.1.2
4
4
  Summary: Evaluate any model on any dataset and any task — standalone, no platform required
5
5
  Project-URL: Homepage, https://github.com/Lexsi-Labs/AuditKIT
6
6
  Project-URL: Repository, https://github.com/Lexsi-Labs/AuditKIT
@@ -126,6 +126,7 @@ Requires-Dist: pillow; extra == 'all'
126
126
  Requires-Dist: pytest; extra == 'all'
127
127
  Requires-Dist: pytest-asyncio; extra == 'all'
128
128
  Requires-Dist: pytest-mock; extra == 'all'
129
+ Requires-Dist: pyyaml; extra == 'all'
129
130
  Requires-Dist: requests; extra == 'all'
130
131
  Requires-Dist: torch>=2.4; extra == 'all'
131
132
  Requires-Dist: torchmetrics; extra == 'all'
@@ -142,6 +143,7 @@ Provides-Extra: dev
142
143
  Requires-Dist: pytest; extra == 'dev'
143
144
  Requires-Dist: pytest-asyncio; extra == 'dev'
144
145
  Requires-Dist: pytest-mock; extra == 'dev'
146
+ Requires-Dist: pyyaml; extra == 'dev'
145
147
  Provides-Extra: interop
146
148
  Requires-Dist: datasets; extra == 'interop'
147
149
  Requires-Dist: gitpython; extra == 'interop'
@@ -180,8 +182,8 @@ Description-Content-Type: text/markdown
180
182
  <p align="center">
181
183
  <a href="https://github.com/Lexsi-Labs/AuditKIT">
182
184
  <picture>
183
- <source media="(prefers-color-scheme: dark)" srcset="docs/assets/auditkit-logo-white.png">
184
- <img src="docs/assets/auditkit-logo-black.png" alt="AuditKit" width="480">
185
+ <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/Lexsi-Labs/AuditKIT/main/docs/assets/auditkit-logo-white.png">
186
+ <img src="https://raw.githubusercontent.com/Lexsi-Labs/AuditKIT/main/docs/assets/auditkit-logo-black.png" alt="AuditKit" width="480">
185
187
  </picture>
186
188
  </a>
187
189
  </p>
@@ -193,7 +195,7 @@ Description-Content-Type: text/markdown
193
195
  </p>
194
196
 
195
197
  <p align="center">
196
- <a href="https://pypi.org/project/auditkit/"><img src="https://img.shields.io/badge/pypi-v1.2.0-0a8868" alt="PyPI v1.2.0"></a>
198
+ <a href="https://pypi.org/project/auditkit/"><img src="https://img.shields.io/pypi/v/auditkit?color=0a8868" alt="PyPI version"></a>
197
199
  <a href="https://www.python.org/"><img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python 3.10+"></a>
198
200
  <a href="https://github.com/Lexsi-Labs/AuditKIT/blob/main/LICENSE.md"><img src="https://img.shields.io/badge/license-LSAL--1.2-lightgrey" alt="License: LSAL-1.2 (source-available, noncommercial)"></a>
199
201
  <a href="https://auditkit.lexsi.ai/"><img src="https://img.shields.io/badge/docs-auditkit.lexsi.ai-4c6ef5" alt="Documentation"></a>
@@ -219,7 +221,7 @@ Description-Content-Type: text/markdown
219
221
  | **Models** | 10 backends: OpenAI, Anthropic, HuggingFace, Lexsi, vLLM, LiteLLM, API, Groq, OpenRouter, Agent (HTTP endpoint) — all resolved via `AutoModel.resolve()`. Any `list[str] → list[str]` callable also works. |
220
222
  | **Zero deps** | Core runs on stdlib. Backends and heavy metrics are optional extras (`pip install auditkit[openai]`). |
221
223
  | **Fingerprints** | Every run gets a stable sha256 — results are cacheable, comparable, and reproducible by construction. |
222
- | **Status** | ~3,000 tests, v1.2.0, LSAL-1.2 license (source-available, noncommercial). 11 metric families, 6 CLI subcommands, YAML config, MKDocs site. |
224
+ | **Status** | ~3,000 tests, v1.1.2, LSAL-1.2 license (source-available, noncommercial). 11 metric families, 6 CLI subcommands, YAML config, MKDocs site. |
223
225
 
224
226
  ---
225
227
 
@@ -359,7 +361,7 @@ dropped it after 0.24.0, SGLang never had it), so both servers run it through th
359
361
  Transformers backends: automatically on vLLM, with `compat.py --patch-sglang` on SGLang. North Micro Vision is native on vLLM; on SGLang
360
362
  it needs transformers >= 5.15 in the SGLang env plus `--patch-sglang`. Tiny Aya and Aya Expanse
361
363
  are native on both. The full
362
- model-by-runtime table is in [docs/model_backends.md](docs/model_backends.md).
364
+ model-by-runtime table is in [docs/model_backends.md](https://github.com/Lexsi-Labs/AuditKIT/blob/main/docs/model_backends.md).
363
365
 
364
366
  Tiny Aya (`CohereLabs/tiny-aya-global` and siblings `fire`, `water`, `earth`),
365
367
  Aya Expanse and Aya Vision are gated
@@ -392,57 +394,57 @@ Every `evaluate()` returns a `RunResult`:
392
394
  ## Examples
393
395
 
394
396
  Colab-ready notebooks with real models and real datasets — see
395
- [`examples/README.md`](examples/README.md) for the full, current list.
397
+ [`examples/README.md`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/README.md) for the full, current list.
396
398
 
397
399
  | Example | File |
398
400
  |---------|------|
399
- | Full pipeline: adapter, 5 metrics, LLM judge, LLM annotator | [`01_full_evaluation_pipeline.ipynb`](examples/01_full_evaluation_pipeline.ipynb) |
400
- | Generation across two real HF model families, compared | [`02_generation_across_hf_families.ipynb`](examples/02_generation_across_hf_families.ipynb) |
401
- | Custom annotators (regex, LLM-backed, fully custom) | [`03_custom_annotators.ipynb`](examples/03_custom_annotators.ipynb) |
402
- | Metrics deep dive: built-in, custom, LLM-as-judge, RAG | [`04_metrics_deep_dive.ipynb`](examples/04_metrics_deep_dive.ipynb) |
403
- | Every data type/task kind: generative, MCQ, RAG, precomputed, chat | [`05_data_types.ipynb`](examples/05_data_types.ipynb) |
404
- | Model comparison deep dive: `compare_models()` + `RunComparison` | [`06_model_comparison.ipynb`](examples/06_model_comparison.ipynb) |
405
- | Annotators across 4 real model families, then compared together | [`07_annotators_across_models.ipynb`](examples/07_annotators_across_models.ipynb) |
406
- | `CompareResult` deep dive: what `compare_models()`'s native per-model support still can't express (different scorers per model), full method surface | [`08_compare_result_deep_dive.ipynb`](examples/08_compare_result_deep_dive.ipynb) |
407
- | `GuardJudge` implementation check: all 5 profiles, real and offline | [`09_guard_judge_implementation_check.ipynb`](examples/09_guard_judge_implementation_check.ipynb) |
408
- | `GuardJudge` with real, flagship guard models | [`10_guard_judge_legit_models.ipynb`](examples/10_guard_judge_legit_models.ipynb) |
409
- | `EncoderJudge`: the base mechanism plus its two prebuilt subclasses | [`11_encoder_judge_prebuilts.ipynb`](examples/11_encoder_judge_prebuilts.ipynb) |
410
- | Performance metrics on a real, large-scale evaluation | [`12_performance_metrics_demo.ipynb`](examples/12_performance_metrics_demo.ipynb) |
411
- | SGLang: environment checks, server mode via `api:`, tool calls and parallel tool calls | [`13_sglang_compatibility.ipynb`](examples/13_sglang_compatibility.ipynb) |
412
- | Agent and RAG evals: tool calls, parallel tool calls, deployed agent endpoints, AgentTune data, retrieval and judged RAG metrics | [`14_agent_and_rag_evals.ipynb`](examples/14_agent_and_rag_evals.ipynb) |
413
- | End-to-end agent evaluation, fully offline (`.py` script): recorded episodes, a verified state outcome, and a fabricated answer that does not pass a state case | [`agent_eval_offline.py`](examples/agent_eval_offline.py) |
401
+ | Full pipeline: adapter, 5 metrics, LLM judge, LLM annotator | [`01_full_evaluation_pipeline.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/01_full_evaluation_pipeline.ipynb) |
402
+ | Generation across two real HF model families, compared | [`02_generation_across_hf_families.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/02_generation_across_hf_families.ipynb) |
403
+ | Custom annotators (regex, LLM-backed, fully custom) | [`03_custom_annotators.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/03_custom_annotators.ipynb) |
404
+ | Metrics deep dive: built-in, custom, LLM-as-judge, RAG | [`04_metrics_deep_dive.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/04_metrics_deep_dive.ipynb) |
405
+ | Every data type/task kind: generative, MCQ, RAG, precomputed, chat | [`05_data_types.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/05_data_types.ipynb) |
406
+ | Model comparison deep dive: `compare_models()` + `RunComparison` | [`06_model_comparison.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/06_model_comparison.ipynb) |
407
+ | Annotators across 4 real model families, then compared together | [`07_annotators_across_models.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/07_annotators_across_models.ipynb) |
408
+ | `CompareResult` deep dive: what `compare_models()`'s native per-model support still can't express (different scorers per model), full method surface | [`08_compare_result_deep_dive.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/08_compare_result_deep_dive.ipynb) |
409
+ | `GuardJudge` implementation check: all 5 profiles, real and offline | [`09_guard_judge_implementation_check.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/09_guard_judge_implementation_check.ipynb) |
410
+ | `GuardJudge` with real, flagship guard models | [`10_guard_judge_legit_models.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/10_guard_judge_legit_models.ipynb) |
411
+ | `EncoderJudge`: the base mechanism plus its two prebuilt subclasses | [`11_encoder_judge_prebuilts.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/11_encoder_judge_prebuilts.ipynb) |
412
+ | Performance metrics on a real, large-scale evaluation | [`12_performance_metrics_demo.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/12_performance_metrics_demo.ipynb) |
413
+ | SGLang: environment checks, server mode via `api:`, tool calls and parallel tool calls | [`13_sglang_compatibility.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/13_sglang_compatibility.ipynb) |
414
+ | Agent and RAG evals: tool calls, parallel tool calls, deployed agent endpoints, AgentTune data, retrieval and judged RAG metrics | [`14_agent_and_rag_evals.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/14_agent_and_rag_evals.ipynb) |
415
+ | End-to-end agent evaluation, fully offline (`.py` script): recorded episodes, a verified state outcome, and a fabricated answer that does not pass a state case | [`agent_eval_offline.py`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/agent_eval_offline.py) |
414
416
 
415
417
  ## Applications
416
418
 
417
419
  Real-world scenarios answered end to end, not feature tours — see
418
- [`examples/applications/README.md`](examples/applications/README.md). Only `01` uses the
420
+ [`examples/applications/README.md`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/README.md). Only `01` uses the
419
421
  `lm-evaluation-harness` integration (`ak.run_lmeval()`, needs
420
422
  `auditkit[lmeval]`), as an authoritative cross-check alongside the native
421
423
  evaluation; none of the others do.
422
424
 
423
425
  | Application | File |
424
426
  |-------------|------|
425
- | How much does pruning severity (20%/40%/60%) degrade a model? Real BoolQ, real annotator, ship/no-ship verdicts, cross-checked via `ak.run_lmeval()` | [`01_application_pruned_llama_boolq.ipynb`](examples/applications/01_application_pruned_llama_boolq.ipynb) |
426
- | Healthcare: clinical QA correctness vs. grounding (PubMedQA) | [`02_application_healthcare_pubmedqa.ipynb`](examples/applications/02_application_healthcare_pubmedqa.ipynb) |
427
- | Finance: QA grounded in real SEC 10-K filings | [`03_application_finance_10k_qa.ipynb`](examples/applications/03_application_finance_10k_qa.ipynb) |
428
- | E-commerce: review-sentiment triage at scale | [`04_application_ecommerce_review_triage.ipynb`](examples/applications/04_application_ecommerce_review_triage.ipynb) |
429
- | Education: auto-graded tutoring, correctness vs. explanation | [`05_application_education_arc_tutor.ipynb`](examples/applications/05_application_education_arc_tutor.ipynb) |
430
- | Enterprise search: internal knowledge assistant (real retrieval + RAG grounding) | [`06_application_enterprise_search_rag.ipynb`](examples/applications/06_application_enterprise_search_rag.ipynb) |
431
- | LLM-as-judge via a real BERT NLI classifier, not a generative model | [`07_application_bert_nli_judge.ipynb`](examples/applications/07_application_bert_nli_judge.ipynb) |
427
+ | How much does pruning severity (20%/40%/60%) degrade a model? Real BoolQ, real annotator, ship/no-ship verdicts, cross-checked via `ak.run_lmeval()` | [`01_application_pruned_llama_boolq.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/01_application_pruned_llama_boolq.ipynb) |
428
+ | Healthcare: clinical QA correctness vs. grounding (PubMedQA) | [`02_application_healthcare_pubmedqa.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/02_application_healthcare_pubmedqa.ipynb) |
429
+ | Finance: QA grounded in real SEC 10-K filings | [`03_application_finance_10k_qa.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/03_application_finance_10k_qa.ipynb) |
430
+ | E-commerce: review-sentiment triage at scale | [`04_application_ecommerce_review_triage.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/04_application_ecommerce_review_triage.ipynb) |
431
+ | Education: auto-graded tutoring, correctness vs. explanation | [`05_application_education_arc_tutor.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/05_application_education_arc_tutor.ipynb) |
432
+ | Enterprise search: internal knowledge assistant (real retrieval + RAG grounding) | [`06_application_enterprise_search_rag.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/06_application_enterprise_search_rag.ipynb) |
433
+ | LLM-as-judge via a real BERT NLI classifier, not a generative model | [`07_application_bert_nli_judge.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/07_application_bert_nli_judge.ipynb) |
432
434
 
433
435
  ## Repository map
434
436
 
435
437
  | Directory | Description | README |
436
438
  |-----------|-------------|--------|
437
- | [`src/auditkit/`](src/auditkit/) | Core library — spine, runner, metrics, models, CLI | [README](src/auditkit/README.md) |
438
- | [`src/auditkit/metrics/`](src/auditkit/metrics/) | Metric families | [README](src/auditkit/metrics/README.md) |
439
- | [`src/auditkit/model/`](src/auditkit/model/) | Model backends (echo, openai, hf, vllm, etc.) | [README](src/auditkit/model/README.md) |
440
- | [`src/auditkit/redteam/`](src/auditkit/redteam/) | Red team probes and detectors | [README](src/auditkit/redteam/README.md) |
441
- | [`src/auditkit/scenarios/`](src/auditkit/scenarios/) | Built-in benchmark datasets | [README](src/auditkit/scenarios/README.md) |
442
- | [`tests/`](tests/) | Test suite (~3,000 tests) | [README](tests/README.md) |
443
- | [`examples/`](examples/) | Colab-ready example notebooks | [README](examples/README.md) |
444
- | [`examples/applications/`](examples/applications/) | Real-world application notebooks | [README](examples/applications/README.md) |
445
- | [`docs/`](docs/) | MkDocs source for [auditkit.lexsi.ai](https://auditkit.lexsi.ai/) | [index](docs/index.md) |
439
+ | [`src/auditkit/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/) | Core library — spine, runner, metrics, models, CLI | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/README.md) |
440
+ | [`src/auditkit/metrics/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/metrics/) | Metric families | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/metrics/README.md) |
441
+ | [`src/auditkit/model/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/model/) | Model backends (echo, openai, hf, vllm, etc.) | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/model/README.md) |
442
+ | [`src/auditkit/redteam/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/redteam/) | Red team probes and detectors | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/redteam/README.md) |
443
+ | [`src/auditkit/scenarios/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/scenarios/) | Built-in benchmark datasets | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/scenarios/README.md) |
444
+ | [`tests/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/tests/) | Test suite (~3,000 tests) | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/tests/README.md) |
445
+ | [`examples/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/examples/) | Colab-ready example notebooks | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/README.md) |
446
+ | [`examples/applications/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/examples/applications/) | Real-world application notebooks | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/README.md) |
447
+ | [`docs/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/docs/) | MkDocs source for [auditkit.lexsi.ai](https://auditkit.lexsi.ai/) | [index](https://github.com/Lexsi-Labs/AuditKIT/blob/main/docs/index.md) |
446
448
 
447
449
  ---
448
450
 
@@ -467,7 +469,7 @@ If you use AuditKIT in research, please cite it:
467
469
  Kadiyala, Ram Mohan Rao and Gosalia, Hem and Sankarapu, Vinay Kumar and
468
470
  Seth, Pratinav},
469
471
  year = {2026},
470
- version = {1.2.0},
472
+ version = {1.1.2},
471
473
  url = {https://github.com/Lexsi-Labs/AuditKIT}
472
474
  }
473
475
  ```
@@ -478,11 +480,11 @@ If you use AuditKIT in research, please cite it:
478
480
 
479
481
  ## License
480
482
 
481
- This project is released under the [Lexsi Labs Source Available License (LSAL) v1.2](LICENSE.md) — free for academic research and teaching on MIT-like terms; use by any organization requires written acknowledgement or permission (Section 1A); a separate commercial license is required to sell it or embed it in a paid product.
483
+ This project is released under the [Lexsi Labs Source Available License (LSAL) v1.2](https://github.com/Lexsi-Labs/AuditKIT/blob/main/LICENSE.md) — free for academic research and teaching on MIT-like terms; use by any organization requires written acknowledgement or permission (Section 1A); a separate commercial license is required to sell it or embed it in a paid product.
482
484
 
483
485
  ---
484
486
 
485
487
  ## Join Community / Contribute
486
488
 
487
489
  - Issues and discussions are welcomed on the [GitHub issue tracker](https://github.com/Lexsi-Labs/AuditKIT/issues).
488
- - See [Contributing](docs/community/contributing.md) for the development setup, code standards and the pull request process.
490
+ - See [Contributing](https://github.com/Lexsi-Labs/AuditKIT/blob/main/docs/community/contributing.md) for the development setup, code standards and the pull request process.
@@ -1,8 +1,8 @@
1
1
  <p align="center">
2
2
  <a href="https://github.com/Lexsi-Labs/AuditKIT">
3
3
  <picture>
4
- <source media="(prefers-color-scheme: dark)" srcset="docs/assets/auditkit-logo-white.png">
5
- <img src="docs/assets/auditkit-logo-black.png" alt="AuditKit" width="480">
4
+ <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/Lexsi-Labs/AuditKIT/main/docs/assets/auditkit-logo-white.png">
5
+ <img src="https://raw.githubusercontent.com/Lexsi-Labs/AuditKIT/main/docs/assets/auditkit-logo-black.png" alt="AuditKit" width="480">
6
6
  </picture>
7
7
  </a>
8
8
  </p>
@@ -14,7 +14,7 @@
14
14
  </p>
15
15
 
16
16
  <p align="center">
17
- <a href="https://pypi.org/project/auditkit/"><img src="https://img.shields.io/badge/pypi-v1.2.0-0a8868" alt="PyPI v1.2.0"></a>
17
+ <a href="https://pypi.org/project/auditkit/"><img src="https://img.shields.io/pypi/v/auditkit?color=0a8868" alt="PyPI version"></a>
18
18
  <a href="https://www.python.org/"><img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python 3.10+"></a>
19
19
  <a href="https://github.com/Lexsi-Labs/AuditKIT/blob/main/LICENSE.md"><img src="https://img.shields.io/badge/license-LSAL--1.2-lightgrey" alt="License: LSAL-1.2 (source-available, noncommercial)"></a>
20
20
  <a href="https://auditkit.lexsi.ai/"><img src="https://img.shields.io/badge/docs-auditkit.lexsi.ai-4c6ef5" alt="Documentation"></a>
@@ -40,7 +40,7 @@
40
40
  | **Models** | 10 backends: OpenAI, Anthropic, HuggingFace, Lexsi, vLLM, LiteLLM, API, Groq, OpenRouter, Agent (HTTP endpoint) — all resolved via `AutoModel.resolve()`. Any `list[str] → list[str]` callable also works. |
41
41
  | **Zero deps** | Core runs on stdlib. Backends and heavy metrics are optional extras (`pip install auditkit[openai]`). |
42
42
  | **Fingerprints** | Every run gets a stable sha256 — results are cacheable, comparable, and reproducible by construction. |
43
- | **Status** | ~3,000 tests, v1.2.0, LSAL-1.2 license (source-available, noncommercial). 11 metric families, 6 CLI subcommands, YAML config, MKDocs site. |
43
+ | **Status** | ~3,000 tests, v1.1.2, LSAL-1.2 license (source-available, noncommercial). 11 metric families, 6 CLI subcommands, YAML config, MKDocs site. |
44
44
 
45
45
  ---
46
46
 
@@ -180,7 +180,7 @@ dropped it after 0.24.0, SGLang never had it), so both servers run it through th
180
180
  Transformers backends: automatically on vLLM, with `compat.py --patch-sglang` on SGLang. North Micro Vision is native on vLLM; on SGLang
181
181
  it needs transformers >= 5.15 in the SGLang env plus `--patch-sglang`. Tiny Aya and Aya Expanse
182
182
  are native on both. The full
183
- model-by-runtime table is in [docs/model_backends.md](docs/model_backends.md).
183
+ model-by-runtime table is in [docs/model_backends.md](https://github.com/Lexsi-Labs/AuditKIT/blob/main/docs/model_backends.md).
184
184
 
185
185
  Tiny Aya (`CohereLabs/tiny-aya-global` and siblings `fire`, `water`, `earth`),
186
186
  Aya Expanse and Aya Vision are gated
@@ -213,57 +213,57 @@ Every `evaluate()` returns a `RunResult`:
213
213
  ## Examples
214
214
 
215
215
  Colab-ready notebooks with real models and real datasets — see
216
- [`examples/README.md`](examples/README.md) for the full, current list.
216
+ [`examples/README.md`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/README.md) for the full, current list.
217
217
 
218
218
  | Example | File |
219
219
  |---------|------|
220
- | Full pipeline: adapter, 5 metrics, LLM judge, LLM annotator | [`01_full_evaluation_pipeline.ipynb`](examples/01_full_evaluation_pipeline.ipynb) |
221
- | Generation across two real HF model families, compared | [`02_generation_across_hf_families.ipynb`](examples/02_generation_across_hf_families.ipynb) |
222
- | Custom annotators (regex, LLM-backed, fully custom) | [`03_custom_annotators.ipynb`](examples/03_custom_annotators.ipynb) |
223
- | Metrics deep dive: built-in, custom, LLM-as-judge, RAG | [`04_metrics_deep_dive.ipynb`](examples/04_metrics_deep_dive.ipynb) |
224
- | Every data type/task kind: generative, MCQ, RAG, precomputed, chat | [`05_data_types.ipynb`](examples/05_data_types.ipynb) |
225
- | Model comparison deep dive: `compare_models()` + `RunComparison` | [`06_model_comparison.ipynb`](examples/06_model_comparison.ipynb) |
226
- | Annotators across 4 real model families, then compared together | [`07_annotators_across_models.ipynb`](examples/07_annotators_across_models.ipynb) |
227
- | `CompareResult` deep dive: what `compare_models()`'s native per-model support still can't express (different scorers per model), full method surface | [`08_compare_result_deep_dive.ipynb`](examples/08_compare_result_deep_dive.ipynb) |
228
- | `GuardJudge` implementation check: all 5 profiles, real and offline | [`09_guard_judge_implementation_check.ipynb`](examples/09_guard_judge_implementation_check.ipynb) |
229
- | `GuardJudge` with real, flagship guard models | [`10_guard_judge_legit_models.ipynb`](examples/10_guard_judge_legit_models.ipynb) |
230
- | `EncoderJudge`: the base mechanism plus its two prebuilt subclasses | [`11_encoder_judge_prebuilts.ipynb`](examples/11_encoder_judge_prebuilts.ipynb) |
231
- | Performance metrics on a real, large-scale evaluation | [`12_performance_metrics_demo.ipynb`](examples/12_performance_metrics_demo.ipynb) |
232
- | SGLang: environment checks, server mode via `api:`, tool calls and parallel tool calls | [`13_sglang_compatibility.ipynb`](examples/13_sglang_compatibility.ipynb) |
233
- | Agent and RAG evals: tool calls, parallel tool calls, deployed agent endpoints, AgentTune data, retrieval and judged RAG metrics | [`14_agent_and_rag_evals.ipynb`](examples/14_agent_and_rag_evals.ipynb) |
234
- | End-to-end agent evaluation, fully offline (`.py` script): recorded episodes, a verified state outcome, and a fabricated answer that does not pass a state case | [`agent_eval_offline.py`](examples/agent_eval_offline.py) |
220
+ | Full pipeline: adapter, 5 metrics, LLM judge, LLM annotator | [`01_full_evaluation_pipeline.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/01_full_evaluation_pipeline.ipynb) |
221
+ | Generation across two real HF model families, compared | [`02_generation_across_hf_families.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/02_generation_across_hf_families.ipynb) |
222
+ | Custom annotators (regex, LLM-backed, fully custom) | [`03_custom_annotators.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/03_custom_annotators.ipynb) |
223
+ | Metrics deep dive: built-in, custom, LLM-as-judge, RAG | [`04_metrics_deep_dive.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/04_metrics_deep_dive.ipynb) |
224
+ | Every data type/task kind: generative, MCQ, RAG, precomputed, chat | [`05_data_types.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/05_data_types.ipynb) |
225
+ | Model comparison deep dive: `compare_models()` + `RunComparison` | [`06_model_comparison.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/06_model_comparison.ipynb) |
226
+ | Annotators across 4 real model families, then compared together | [`07_annotators_across_models.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/07_annotators_across_models.ipynb) |
227
+ | `CompareResult` deep dive: what `compare_models()`'s native per-model support still can't express (different scorers per model), full method surface | [`08_compare_result_deep_dive.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/08_compare_result_deep_dive.ipynb) |
228
+ | `GuardJudge` implementation check: all 5 profiles, real and offline | [`09_guard_judge_implementation_check.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/09_guard_judge_implementation_check.ipynb) |
229
+ | `GuardJudge` with real, flagship guard models | [`10_guard_judge_legit_models.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/10_guard_judge_legit_models.ipynb) |
230
+ | `EncoderJudge`: the base mechanism plus its two prebuilt subclasses | [`11_encoder_judge_prebuilts.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/11_encoder_judge_prebuilts.ipynb) |
231
+ | Performance metrics on a real, large-scale evaluation | [`12_performance_metrics_demo.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/12_performance_metrics_demo.ipynb) |
232
+ | SGLang: environment checks, server mode via `api:`, tool calls and parallel tool calls | [`13_sglang_compatibility.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/13_sglang_compatibility.ipynb) |
233
+ | Agent and RAG evals: tool calls, parallel tool calls, deployed agent endpoints, AgentTune data, retrieval and judged RAG metrics | [`14_agent_and_rag_evals.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/14_agent_and_rag_evals.ipynb) |
234
+ | End-to-end agent evaluation, fully offline (`.py` script): recorded episodes, a verified state outcome, and a fabricated answer that does not pass a state case | [`agent_eval_offline.py`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/agent_eval_offline.py) |
235
235
 
236
236
  ## Applications
237
237
 
238
238
  Real-world scenarios answered end to end, not feature tours — see
239
- [`examples/applications/README.md`](examples/applications/README.md). Only `01` uses the
239
+ [`examples/applications/README.md`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/README.md). Only `01` uses the
240
240
  `lm-evaluation-harness` integration (`ak.run_lmeval()`, needs
241
241
  `auditkit[lmeval]`), as an authoritative cross-check alongside the native
242
242
  evaluation; none of the others do.
243
243
 
244
244
  | Application | File |
245
245
  |-------------|------|
246
- | How much does pruning severity (20%/40%/60%) degrade a model? Real BoolQ, real annotator, ship/no-ship verdicts, cross-checked via `ak.run_lmeval()` | [`01_application_pruned_llama_boolq.ipynb`](examples/applications/01_application_pruned_llama_boolq.ipynb) |
247
- | Healthcare: clinical QA correctness vs. grounding (PubMedQA) | [`02_application_healthcare_pubmedqa.ipynb`](examples/applications/02_application_healthcare_pubmedqa.ipynb) |
248
- | Finance: QA grounded in real SEC 10-K filings | [`03_application_finance_10k_qa.ipynb`](examples/applications/03_application_finance_10k_qa.ipynb) |
249
- | E-commerce: review-sentiment triage at scale | [`04_application_ecommerce_review_triage.ipynb`](examples/applications/04_application_ecommerce_review_triage.ipynb) |
250
- | Education: auto-graded tutoring, correctness vs. explanation | [`05_application_education_arc_tutor.ipynb`](examples/applications/05_application_education_arc_tutor.ipynb) |
251
- | Enterprise search: internal knowledge assistant (real retrieval + RAG grounding) | [`06_application_enterprise_search_rag.ipynb`](examples/applications/06_application_enterprise_search_rag.ipynb) |
252
- | LLM-as-judge via a real BERT NLI classifier, not a generative model | [`07_application_bert_nli_judge.ipynb`](examples/applications/07_application_bert_nli_judge.ipynb) |
246
+ | How much does pruning severity (20%/40%/60%) degrade a model? Real BoolQ, real annotator, ship/no-ship verdicts, cross-checked via `ak.run_lmeval()` | [`01_application_pruned_llama_boolq.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/01_application_pruned_llama_boolq.ipynb) |
247
+ | Healthcare: clinical QA correctness vs. grounding (PubMedQA) | [`02_application_healthcare_pubmedqa.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/02_application_healthcare_pubmedqa.ipynb) |
248
+ | Finance: QA grounded in real SEC 10-K filings | [`03_application_finance_10k_qa.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/03_application_finance_10k_qa.ipynb) |
249
+ | E-commerce: review-sentiment triage at scale | [`04_application_ecommerce_review_triage.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/04_application_ecommerce_review_triage.ipynb) |
250
+ | Education: auto-graded tutoring, correctness vs. explanation | [`05_application_education_arc_tutor.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/05_application_education_arc_tutor.ipynb) |
251
+ | Enterprise search: internal knowledge assistant (real retrieval + RAG grounding) | [`06_application_enterprise_search_rag.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/06_application_enterprise_search_rag.ipynb) |
252
+ | LLM-as-judge via a real BERT NLI classifier, not a generative model | [`07_application_bert_nli_judge.ipynb`](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/07_application_bert_nli_judge.ipynb) |
253
253
 
254
254
  ## Repository map
255
255
 
256
256
  | Directory | Description | README |
257
257
  |-----------|-------------|--------|
258
- | [`src/auditkit/`](src/auditkit/) | Core library — spine, runner, metrics, models, CLI | [README](src/auditkit/README.md) |
259
- | [`src/auditkit/metrics/`](src/auditkit/metrics/) | Metric families | [README](src/auditkit/metrics/README.md) |
260
- | [`src/auditkit/model/`](src/auditkit/model/) | Model backends (echo, openai, hf, vllm, etc.) | [README](src/auditkit/model/README.md) |
261
- | [`src/auditkit/redteam/`](src/auditkit/redteam/) | Red team probes and detectors | [README](src/auditkit/redteam/README.md) |
262
- | [`src/auditkit/scenarios/`](src/auditkit/scenarios/) | Built-in benchmark datasets | [README](src/auditkit/scenarios/README.md) |
263
- | [`tests/`](tests/) | Test suite (~3,000 tests) | [README](tests/README.md) |
264
- | [`examples/`](examples/) | Colab-ready example notebooks | [README](examples/README.md) |
265
- | [`examples/applications/`](examples/applications/) | Real-world application notebooks | [README](examples/applications/README.md) |
266
- | [`docs/`](docs/) | MkDocs source for [auditkit.lexsi.ai](https://auditkit.lexsi.ai/) | [index](docs/index.md) |
258
+ | [`src/auditkit/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/) | Core library — spine, runner, metrics, models, CLI | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/README.md) |
259
+ | [`src/auditkit/metrics/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/metrics/) | Metric families | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/metrics/README.md) |
260
+ | [`src/auditkit/model/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/model/) | Model backends (echo, openai, hf, vllm, etc.) | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/model/README.md) |
261
+ | [`src/auditkit/redteam/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/redteam/) | Red team probes and detectors | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/redteam/README.md) |
262
+ | [`src/auditkit/scenarios/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/src/auditkit/scenarios/) | Built-in benchmark datasets | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/src/auditkit/scenarios/README.md) |
263
+ | [`tests/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/tests/) | Test suite (~3,000 tests) | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/tests/README.md) |
264
+ | [`examples/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/examples/) | Colab-ready example notebooks | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/README.md) |
265
+ | [`examples/applications/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/examples/applications/) | Real-world application notebooks | [README](https://github.com/Lexsi-Labs/AuditKIT/blob/main/examples/applications/README.md) |
266
+ | [`docs/`](https://github.com/Lexsi-Labs/AuditKIT/tree/main/docs/) | MkDocs source for [auditkit.lexsi.ai](https://auditkit.lexsi.ai/) | [index](https://github.com/Lexsi-Labs/AuditKIT/blob/main/docs/index.md) |
267
267
 
268
268
  ---
269
269
 
@@ -288,7 +288,7 @@ If you use AuditKIT in research, please cite it:
288
288
  Kadiyala, Ram Mohan Rao and Gosalia, Hem and Sankarapu, Vinay Kumar and
289
289
  Seth, Pratinav},
290
290
  year = {2026},
291
- version = {1.2.0},
291
+ version = {1.1.2},
292
292
  url = {https://github.com/Lexsi-Labs/AuditKIT}
293
293
  }
294
294
  ```
@@ -299,11 +299,11 @@ If you use AuditKIT in research, please cite it:
299
299
 
300
300
  ## License
301
301
 
302
- This project is released under the [Lexsi Labs Source Available License (LSAL) v1.2](LICENSE.md) — free for academic research and teaching on MIT-like terms; use by any organization requires written acknowledgement or permission (Section 1A); a separate commercial license is required to sell it or embed it in a paid product.
302
+ This project is released under the [Lexsi Labs Source Available License (LSAL) v1.2](https://github.com/Lexsi-Labs/AuditKIT/blob/main/LICENSE.md) — free for academic research and teaching on MIT-like terms; use by any organization requires written acknowledgement or permission (Section 1A); a separate commercial license is required to sell it or embed it in a paid product.
303
303
 
304
304
  ---
305
305
 
306
306
  ## Join Community / Contribute
307
307
 
308
308
  - Issues and discussions are welcomed on the [GitHub issue tracker](https://github.com/Lexsi-Labs/AuditKIT/issues).
309
- - See [Contributing](docs/community/contributing.md) for the development setup, code standards and the pull request process.
309
+ - See [Contributing](https://github.com/Lexsi-Labs/AuditKIT/blob/main/docs/community/contributing.md) for the development setup, code standards and the pull request process.