SafeLLM4SE 0.1.1__tar.gz

safellm4se-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Francisco Ortin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

safellm4se-0.1.1/MANIFEST.in

@@ -0,0 +1,14 @@
+include LICENSE
+include PUBLISHING.md
+include README.md
+recursive-include doc *.md
+recursive-exclude * __pycache__
+recursive-exclude * *.py[cod]
+recursive-exclude output *
+recursive-exclude logs *
+recursive-exclude .idea *
+exclude paper-v05.tex
+exclude paper-v3.tex
+exclude paper-v3-changes.tex
+exclude sampling/myevaluators/api-keys.json
+exclude src/safellm4se/sampling/myevaluators/api-keys.json

safellm4se-0.1.1/PKG-INFO

@@ -0,0 +1,166 @@
+Metadata-Version: 2.4
+Name: SafeLLM4SE
+Version: 0.1.1
+Summary: Statistical evaluation and reporting for LLM-based software engineering systems.
+Author: Francisco Ortin
+License: MIT License
+        
+        Copyright (c) 2026 Francisco Ortin
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/francisco-ortin/SafeLLM4SE
+Project-URL: Documentation, https://github.com/francisco-ortin/SafeLLM4SE/tree/main/doc
+Project-URL: Repository, https://github.com/francisco-ortin/SafeLLM4SE
+Project-URL: Issues, https://github.com/francisco-ortin/SafeLLM4SE/issues
+Keywords: llm,software-engineering,statistics,evaluation,sampling
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: loguru>=0.7
+Requires-Dist: matplotlib>=3.8
+Requires-Dist: scipy>=1.11
+Provides-Extra: datasets
+Requires-Dist: datasets>=2.18; extra == "datasets"
+Provides-Extra: gemini
+Requires-Dist: google-genai>=1.0; extra == "gemini"
+Provides-Extra: groq
+Requires-Dist: groq>=0.9; extra == "groq"
+Provides-Extra: all
+Requires-Dist: datasets>=2.18; extra == "all"
+Requires-Dist: google-genai>=1.0; extra == "all"
+Requires-Dist: groq>=0.9; extra == "all"
+Dynamic: license-file
+
+# SafeLLM4SE
+
+[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
+[![Latest release](https://img.shields.io/github/v/release/francisco-ortin/SafeLLM4SE?include_prereleases)](https://github.com/francisco-ortin/SafeLLM4SE/releases)
+[![PyPI](https://img.shields.io/pypi/v/safellm4se)](https://pypi.org/project/safellm4se/)
+
+SafeLLM4SE is a Python toolkit for statistically principled evaluation of
+LLM-based software engineering systems. It treats each LLM execution as a
+sample from a stochastic process, then reports quality, stability, uncertainty,
+resource usage, and statistical comparisons instead of relying on a single run.
+
+The project exposes three command-line programs:
+
+- `safellm4se-sample`: runs adaptive sampling with a user-selected evaluator.
+- `safellm4se-report`: summarizes one sampled task into a SafeLLM4SE report CSV.
+- `safellm4se-compare`: compares two sampled tasks with the SafeLLM4SE comparison protocol.
+
+## Install
+
+SafeLLM4SE is available on PyPI as `safellm4se`, so you only need to run:
+
+```bash
+pip install "SafeLLM4SE[all]"
+```
+
+If you want to install SafeLLM4SE from its source code, check the [installation details](doc/installation.md).
+
+## Quick Start
+
+SafeLLM4SE collects repeated observations with `safellm4se-sample`.
+Then, it can be used to report or visualize one sample of observations with `safellm4se-report`,
+or compare task samples with `safellm4se-compare`.
+
+### Sampling
+
+First, you need to generate a sample of repeated observations.
+For this purpose, `safellm4se-sample` loads an evaluator class and repeatedly calls it until a stopping condition is met
+(maximum number of tokens consumed or the confidence interval width is below a threshold).
+You commonly implement the process being measured as an evaluator,
+but we provide several example evaluators, including default implementations for [Ollama](https://ollama.com/),
+[Gemini](https://ai.google.dev/gemini-api/docs), and [Groq](https://console.groq.com/) APIs.
+
+For this example, we perform adaptive sampling on a random evaluator:
+
+```bash
+safellm4se-sample --evaluator safellm4se.sampling.myevaluators.random_normal_evaluator
+```
+
+### Reporting
+
+If you have already created a sample of repeated observations with the task identifier `task-id-1`,
+stored in `output/measurements.csv`,
+you can create reports and visualizations with `safellm4se-report`.
+The report includes the sample size, token usage, central tendency, variability, and confidence interval information.
+The supported visualizations are boxplot, violin plot, empirical cumulative distribution function (ECDF), raincloud plot, and kernel density estimate (KDE).
+
+```bash
+safellm4se-report --input output/measurements.csv --output output/report-demo.csv --task-id task-id-1 --boxplot output/demo-boxplot.svg
+```
+
+Example CSV report generated with `safellm4se-report`:
+
+```csv
+task_id,model_name,model_id,N,total_tokens,theta_mean,sd,ci_method,ci_low,ci_high,ci_width
+task-id-54,qwen-coder,qwen2.5-coder:7b,30,1178530,0.8272357723577236,0.019436705668000667,t,0.8199779871829312,0.834493557532516,0.014515570349584728
+```
+
+### Comparing
+
+You can also compare two samples of repeated observations with `safellm4se-compare`.
+The report will tell you the estimated difference, confidence interval, statistical test,
+p-value, and effect size.
+It also generates figures comparing the two samples, including the visualizations mentioned in
+[reporting](#reporting).
+
+```bash
+safellm4se-compare --input output/measurements.csv --output output/comparing-demo.csv --task-id-1 task-a --task-id-2 task-b --test-type paired --raincloud output/comparing-demo-raincloud.svg
+```
+
+Example plot generated with `safellm4se-compare`:
+
+![Two-sample raincloud plot](output/compare-deepseek-qwen-raincloud.svg)
+
+The following CSV comparison report has also been generated:
+
+```csv
+task_id_1,task_id_2,test_type,estimated_difference,ci_low,ci_high,statistical_test,p_value,effect_size_name,effect_size,effect_size_magnitude
+task-id-54,task-id-56,independent,0.20833333333333337,0.19369410569105683,0.2223628048780488,Mann-Whitney U,2.8591961948613224e-11,Cliff's delta,1.0,large
+```
+
+
+## Documentation
+
+- [Methodology](doc/methodology.md)
+- [Installation](doc/installation.md)
+- [Usage Guide](doc/usage.md)
+- [CLI Reference](doc/cli-reference.md)
+- [Evaluator Guide](doc/evaluators.md)
+
+## License
+
+See [LICENSE](LICENSE).

safellm4se-0.1.1/README.md

@@ -0,0 +1,102 @@
+# SafeLLM4SE
+
+[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
+[![Latest release](https://img.shields.io/github/v/release/francisco-ortin/SafeLLM4SE?include_prereleases)](https://github.com/francisco-ortin/SafeLLM4SE/releases)
+[![PyPI](https://img.shields.io/pypi/v/safellm4se)](https://pypi.org/project/safellm4se/)
+
+SafeLLM4SE is a Python toolkit for statistically principled evaluation of
+LLM-based software engineering systems. It treats each LLM execution as a
+sample from a stochastic process, then reports quality, stability, uncertainty,
+resource usage, and statistical comparisons instead of relying on a single run.
+
+The project exposes three command-line programs:
+
+- `safellm4se-sample`: runs adaptive sampling with a user-selected evaluator.
+- `safellm4se-report`: summarizes one sampled task into a SafeLLM4SE report CSV.
+- `safellm4se-compare`: compares two sampled tasks with the SafeLLM4SE comparison protocol.
+
+## Install
+
+SafeLLM4SE is available on PyPI as `safellm4se`, so you only need to run:
+
+```bash
+pip install "SafeLLM4SE[all]"
+```
+
+If you want to install SafeLLM4SE from its source code, check the [installation details](doc/installation.md).
+
+## Quick Start
+
+SafeLLM4SE collects repeated observations with `safellm4se-sample`.
+Then, it can be used to report or visualize one sample of observations with `safellm4se-report`,
+or compare task samples with `safellm4se-compare`.
+
+### Sampling
+
+First, you need to generate a sample of repeated observations.
+For this purpose, `safellm4se-sample` loads an evaluator class and repeatedly calls it until a stopping condition is met
+(maximum number of tokens consumed or the confidence interval width is below a threshold).
+You commonly implement the process being measured as an evaluator,
+but we provide several example evaluators, including default implementations for [Ollama](https://ollama.com/),
+[Gemini](https://ai.google.dev/gemini-api/docs), and [Groq](https://console.groq.com/) APIs.
+
+For this example, we perform adaptive sampling on a random evaluator:
+
+```bash
+safellm4se-sample --evaluator safellm4se.sampling.myevaluators.random_normal_evaluator
+```
+
+### Reporting
+
+If you have already created a sample of repeated observations with the task identifier `task-id-1`,
+stored in `output/measurements.csv`,
+you can create reports and visualizations with `safellm4se-report`.
+The report includes the sample size, token usage, central tendency, variability, and confidence interval information.
+The supported visualizations are boxplot, violin plot, empirical cumulative distribution function (ECDF), raincloud plot, and kernel density estimate (KDE).
+
+```bash
+safellm4se-report --input output/measurements.csv --output output/report-demo.csv --task-id task-id-1 --boxplot output/demo-boxplot.svg
+```
+
+Example CSV report generated with `safellm4se-report`:
+
+```csv
+task_id,model_name,model_id,N,total_tokens,theta_mean,sd,ci_method,ci_low,ci_high,ci_width
+task-id-54,qwen-coder,qwen2.5-coder:7b,30,1178530,0.8272357723577236,0.019436705668000667,t,0.8199779871829312,0.834493557532516,0.014515570349584728
+```
+
+### Comparing
+
+You can also compare two samples of repeated observations with `safellm4se-compare`.
+The report will tell you the estimated difference, confidence interval, statistical test,
+p-value, and effect size.
+It also generates figures comparing the two samples, including the visualizations mentioned in
+[reporting](#reporting).
+
+```bash
+safellm4se-compare --input output/measurements.csv --output output/comparing-demo.csv --task-id-1 task-a --task-id-2 task-b --test-type paired --raincloud output/comparing-demo-raincloud.svg
+```
+
+Example plot generated with `safellm4se-compare`:
+
+![Two-sample raincloud plot](output/compare-deepseek-qwen-raincloud.svg)
+
+The following CSV comparison report has also been generated:
+
+```csv
+task_id_1,task_id_2,test_type,estimated_difference,ci_low,ci_high,statistical_test,p_value,effect_size_name,effect_size,effect_size_magnitude
+task-id-54,task-id-56,independent,0.20833333333333337,0.19369410569105683,0.2223628048780488,Mann-Whitney U,2.8591961948613224e-11,Cliff's delta,1.0,large
+```
+
+
+## Documentation
+
+- [Methodology](doc/methodology.md)
+- [Installation](doc/installation.md)
+- [Usage Guide](doc/usage.md)
+- [CLI Reference](doc/cli-reference.md)
+- [Evaluator Guide](doc/evaluators.md)
+
+## License
+
+See [LICENSE](LICENSE).

safellm4se-0.1.1/doc/cli-reference.md

@@ -0,0 +1,182 @@
+# CLI Reference
+
+
+This reference shows the arguments and options for the three command-line tools in SafeLLM4SE.
+That is, it reflects the current Python parsers in
+`src/safellm4se/sampling/cli.py`, `src/safellm4se/reporting/cli.py`, and
+`src/safellm4se/compare/cli.py`.
+
+## `safellm4se-sample`
+
+Purpose: run adaptive sampling for one task/model/evaluator key.
+
+```bash
+safellm4se-sample --evaluator MODULE[:CLASS] [options] -- evaluator_parameter=value
+```
+
+Options:
+
+| Option | Default | Description |
+|---|---:|---|
+| `--evaluator`, `--evaluador` | Required | Evaluator module or `module:ClassName` reference. |
+| `--output-dir` | `output` | Directory for measurements, logs, lock file, and reservations. |
+| `--task-id` | Next `task-id-<n>` | Task identifier written to the measurement rows. |
+| `--ci-method` | `auto` | Continuous CI method: `auto`, `t`, or `bootstrap`. Binary metrics always use Wilson. |
+| `--confidence-level` | `0.95` | Confidence level as a fraction in `(0, 1)`. |
+| `--n-min` | `10` | Minimum observations before the sampler may stop. |
+| `--target-ci-width` | `0.10` | Maximum accepted total confidence interval width. |
+| `--budget-tokens` | `10000` | Maximum token budget for the current task/experiment/model key. |
+| `--inter-invocation-waiting` | `0.0` | Seconds to wait before each evaluator invocation. |
+| `--reservation-ttl-seconds` | `3600.0` | Seconds before an unfinished execution reservation expires. |
+| `--verbose` | `False` | Print INFO and higher logs to the console. |
+| `--log LEVEL` | Disabled | Print logs from `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL`. |
+| `--h` | | Show help. |
+
+Evaluator parameters:
+
+- Put evaluator-specific arguments after the separator `--`. Any parameter name can be passed. The evaluator developer can use it during object initialization (in `__init__`).
+- Use `name=value` or `--name=value`.
+- Hyphens are normalized to underscores. Both syntax forms are accepted.
+- Values are parsed with Python literal syntax when possible.
+
+Examples:
+
+```bash
+safellm4se-sample --evaluator safellm4se.sampling.myevaluators.random_binary_evaluator -- success_probability=0.7
+safellm4se-sample --evaluator safellm4se.sampling.myevaluators.random_normal_evaluator -- --mean=50 --standard-deviation=20
+safellm4se-sample --evaluator safellm4se.sampling.myevaluators.groq.random -- temperature=0.2 model_id="openai/gpt-oss-20b"
+```
+
+Generated files in `--output-dir`:
+
+| File | Description |
+|---|---|
+| `measurements.csv` | All completed observations. |
+| `.sampling.lock` | Interprocess lock file. |
+| `.sampling_reservations.json` | In-progress execution reservations. |
+| `logs/log_<timestamp>.log` | Execution log. |
+
+`measurements.csv` core columns:
+
+| Column | Meaning |
+|---|---|
+| `date`, `time` | Local timestamp when the observation was persisted. |
+| `task_id` | Task identifier passed or auto-generated by the sampler. |
+| `experiment_name` | Evaluator experiment name. |
+| `model_name` | Human-readable model or provider label. |
+| `model_id` | Provider-specific model identifier. |
+| `execution_number` | Sequential run number for the task/experiment/model key. |
+| `prompt_tokens` | Prompt tokens consumed by the observation. |
+| `completion_tokens` | Completion tokens consumed by the observation. |
+| `total_tokens` | Total tokens consumed by the observation. |
+| `theta` | Numeric evaluation value. |
+| `metric_type` | `binary` or `continuous`. |
+| `evaluator` | Evaluator class name. |
+| `evaluator_parameters` | JSON-serialized constructor parameters. |
+| `run_id` | UUID of the sampler process run. |
+
+Non-reserved evaluator parameters are also written as extra CSV columns.
+
+## `safellm4se-report`
+
+Purpose: summarize all measurements for the sample identified by `task_id` and generate optional visualizations.
+
+```bash
+safellm4se-report --input output/measurements.csv --task-id TASK_ID [options]
+```
+
+Options:
+
+| Option | Default | Description |
+|---|---:|---|
+| `--input FILE_NAME` | Required | CSV generated by `safellm4se-sample`. |
+| `--output FILE_NAME` | `report.csv` | Output CSV report. |
+| `--task-id TASK_ID` | Required | Task identifier to summarize. |
+| `--task-name NAME` | None | Plot display name. |
+| `--confidence-level` | `0.95` | Confidence level as a fraction in `(0, 1)`. |
+| `--ci-method` | `auto` | Continuous CI method: `auto`, `t`, or `bootstrap`. Binary metrics always use Wilson. |
+| `--boxplot FILE_NAME` | None | Optional SVG boxplot. |
+| `--violin FILE_NAME` | None | Optional SVG violin plot. |
+| `--ecdf`, `--ECDF FILE_NAME` | None | Optional SVG empirical CDF. |
+| `--raincloud FILE_NAME` | None | Optional SVG raincloud plot. |
+| `--kde`, `--KDE FILE_NAME` | None | Optional SVG kernel density plot. |
+| `--verbose` | `False` | Print INFO and higher logs to the console. |
+| `--log LEVEL` | Disabled | Print logs from `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL`. |
+| `--h` | | Show help. |
+
+Output columns stored in the output CSV report:
+
+| Column | Meaning |
+|---|---|
+| `date`, `time` | Report generation timestamp. |
+| `task_id` | Summarized task identifier. |
+| `model_name`, `model_id`, `temperature` | Constant model/configuration fields validated across rows. |
+| `N` | Number of observations. |
+| `prompt_tokens`, `completion_tokens`, `total_tokens` | Aggregated token usage. |
+| `theta_mean`, `theta_median`, `theta_min`, `theta_max` | Quality summary. |
+| `theta_type` | Inferred or declared metric type. |
+| `sd` | Sample standard deviation. |
+| `cv` | Coefficient of variation in percent. |
+| `iqr`, `q1`, `q3` | Interquartile range and quartiles. |
+| `ci_method` | Selected CI method. |
+| `ci_confidence-level` | Confidence level in percent. |
+| `ci_low`, `ci_high`, `ci_width` | Confidence interval bounds and total width. |
+
+## `safellm4se-compare`
+
+Purpose: compare two samples identified by `task_id-1` and `task_id-2` in the same
+CSV input file. The comparison can be either *paired* or *independent*.
+It also generates optional visualizations comparing the two samples.
+
+```bash
+safellm4se-compare --input output/measurements.csv --task-id-1 TASK_A --task-id-2 TASK_B --test-type independent
+```
+
+Options:
+
+| Option | Default | Description |
+|---|---:|---|
+| `--input FILE_NAME` | Required | CSV generated by `safellm4se-sample`. |
+| `--output FILE_NAME` | `compare.csv` | Output CSV comparison report. |
+| `--task-id-1 TASK_ID` | Required | First task identifier. |
+| `--task-name-1 NAME` | None | First display name for plots. |
+| `--task-id-2 TASK_ID` | Required | Second task identifier. |
+| `--task-name-2 NAME` | None | Second display name for plots. |
+| `--test-type` | Required | `paired` or `independent`. |
+| `--confidence-level` | `0.95` | Bootstrap confidence level as a fraction in `(0, 1)`. |
+| `--boxplot FILE_NAME` | None | Optional two-sample SVG boxplot. |
+| `--violin FILE_NAME` | None | Optional two-sample SVG violin plot. |
+| `--ecdf`, `--ECDF FILE_NAME` | None | Optional two-sample SVG empirical CDF. |
+| `--raincloud FILE_NAME` | None | Optional two-sample SVG raincloud plot. |
+| `--kde`, `--KDE FILE_NAME` | None | Optional two-sample SVG kernel density plot. |
+| `--verbose` | `False` | Print INFO and higher logs to the console. |
+| `--log LEVEL` | Disabled | Print logs from `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL`. |
+| `--h` | | Show help. |
+
+Comparison protocol (how statistics are computed) depends on the design type:
+
+| Design | Test | CI | Effect size |
+|---|---|---|---|
+| `independent` | Mann-Whitney U | Bootstrap CI for the mean difference | Cliff's delta |
+| `paired` | Wilcoxon signed-rank | Paired bootstrap CI for the mean difference | Matched-pairs rank-biserial correlation |
+
+For paired comparisons, both tasks must contain exactly the same
+`execution_number` values. The comparison aligns rows by that field.
+
+Effect size magnitude thresholds:
+
+| Absolute effect size | Magnitude |
+|---:|---|
+| `< 0.147` | `negligible` |
+| `>= 0.147` | `small` |
+| `>= 0.33` | `medium` |
+| `>= 0.474` | `large` |
+
+The estimated difference is computed as:
+
+```text
+theta_mean_1 - theta_mean_2
+```
+
+Optional visualizations are generated with the same names as in `safellm4se-report`,
+but they show both samples side by side.