cosq 0.1.0__tar.gz

cosq-0.1.0/.env.example

@@ -0,0 +1,5 @@
+# Optional: required only when using configs/model/openai_*.yaml
+OPENAI_API_KEY=
+
+# Optional: required for gated Hugging Face models such as Llama.
+HF_TOKEN=

cosq-0.1.0/.gitignore

@@ -0,0 +1,19 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+.coverage
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.venv/
+.env
+.pypirc
+
+# Local data and generated outputs
+data/*
+!data/.gitkeep
+results/
+*.sqlite
+*.db

cosq-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ali Şenol
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

cosq-0.1.0/PKG-INFO

@@ -0,0 +1,254 @@
+Metadata-Version: 2.5
+Name: cosq
+Version: 0.1.0
+Summary: Chain-of-Self-Questioning for selective factual answering with LLMs
+Project-URL: Homepage, https://github.com/senolali/cosq
+Project-URL: Repository, https://github.com/senolali/cosq
+Project-URL: Issues, https://github.com/senolali/cosq/issues
+Author-email: Ali Senol <alisenol@tarsus.edu.tr>
+License: MIT License
+        
+        Copyright (c) 2026 Ali Şenol
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Keywords: abstention,hallucination,llm,question-answering,selective-prediction
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: numpy<2.4,>=1.24
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: requests>=2.31
+Requires-Dist: scipy>=1.10
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: mypy<2,>=1.11; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest<9,>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Requires-Dist: twine>=5.0; extra == 'dev'
+Requires-Dist: types-pyyaml; extra == 'dev'
+Provides-Extra: hf
+Requires-Dist: accelerate>=0.30; extra == 'hf'
+Requires-Dist: bitsandbytes>=0.43; extra == 'hf'
+Requires-Dist: datasets>=2.18; extra == 'hf'
+Requires-Dist: torch>=2.0; extra == 'hf'
+Requires-Dist: transformers>=4.40; extra == 'hf'
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == 'openai'
+Description-Content-Type: text/markdown
+
+# CoSQ: Chain-of-Self-Questioning
+
+CoSQ is a Python framework for **selective factual answering** with large language models. Instead of forcing a model to answer every question, CoSQ asks the model to decompose the question into required knowledge items, assess whether those items are supported, and then either answer or abstain with `I don't know`.
+
+The framework is designed for reproducible experiments on hallucination, abstention, answered accuracy, and risk-coverage trade-offs.
+
+## Why CoSQ?
+
+Standard accuracy rewards guessing: an abstention is usually scored the same as a wrong answer. That is a poor fit for settings where a wrong answer is more costly than saying "I don't know". CoSQ treats factual answering as a selective prediction problem and reports:
+
+- **Answered accuracy**: accuracy among parseable committed answers.
+- **Coverage**: the fraction of questions the model answers.
+- **Hallucination rate**: wrong committed answers divided by all questions.
+- **Abstention rate**: explicit `I don't know` responses.
+- **Unparseable rate**: outputs that cannot be mapped to the benchmark answer space.
+
+## Installation
+
+From PyPI, after release:
+
+```bash
+pip install cosq
+```
+
+For local development:
+
+```bash
+git clone https://github.com/senolali/cosq.git
+cd cosq
+python -m venv .venv
+.venv\Scripts\activate
+pip install -e ".[dev]"
+```
+
+Optional backends:
+
+```bash
+pip install -e ".[hf]"      # local Hugging Face / transformers models
+pip install -e ".[openai]"  # OpenAI API backend
+```
+
+## Quick Start
+
+Run a no-network smoke test with the deterministic mock backend:
+
+```bash
+cosq run --config configs/experiment/smoke_mock.yaml --backend mock --allow-dirty
+```
+
+The command writes a run directory under `results/runs/`. Then score and report it:
+
+```bash
+cosq evaluate results/runs/<RUN_DIR>
+cosq analyze results/runs/<RUN_DIR>
+cosq report results/runs/<RUN_DIR>
+```
+
+## Using OpenAI Models
+
+Set your API key:
+
+```bash
+set OPENAI_API_KEY=sk-...        # Windows cmd
+$env:OPENAI_API_KEY="sk-..."     # PowerShell
+export OPENAI_API_KEY=sk-...     # macOS/Linux
+```
+
+Probe the model:
+
+```bash
+cosq probe --model configs/model/openai_gpt4o_mini.yaml
+```
+
+Run a small open-ended TruthfulQA pilot:
+
+```bash
+cosq run --config configs/experiment/pilot_truthfulqa_open_openai.yaml --allow-dirty
+```
+
+You can copy `configs/model/openai_template.yaml` and change `id` to another model available in your OpenAI account.
+
+## Using Hugging Face Models
+
+Install optional dependencies:
+
+```bash
+pip install -e ".[hf]"
+```
+
+For gated models, set `HF_TOKEN` and accept the model license on Hugging Face. Then copy one of the examples in `configs/model/`, replace `revision` with an immutable Hugging Face commit SHA, and run:
+
+```bash
+cosq probe --model configs/model/hf_llama3_8b_instruct.yaml
+cosq run --config configs/experiment/pilot_truthfulqa_open.yaml --allow-dirty
+```
+
+Pinned revisions are required for local Hugging Face models so that a run refers to stable weights.
+
+## Experiment Configuration
+
+```yaml
+name: smoke_mock
+model: configs/model/mock.yaml
+data:
+  name: jsonl
+  path: tests/fixtures/mini.jsonl
+  n: 3
+  seed: 1002
+strategies:
+  - name: direct
+  - name: cot
+  - name: cot_abstain
+  - name: cosq
+  - name: cosq_graded_gate
+    label: cosq_graded_gate_mean060
+    threshold: 0.60
+    aggregator: mean
+    on_empty: abstain
+repeats: 1
+open_ended: false
+```
+
+## Built-in Strategies
+
+- `direct`: answer directly.
+- `cot`: reason step by step, then answer.
+- `cot_abstain`: CoT with permission to answer `I don't know`.
+- `cosq`: binary Chain-of-Self-Questioning with a strict conjunctive gate.
+- `cosq_gate`: CoSQ gate followed by CoT-style answer generation.
+- `cosq_graded`: item-level 0-100 confidence scores with thresholding.
+- `cosq_graded_gate`: graded gate followed by CoT-style answer generation.
+
+## Data
+
+CoSQ ships with a tiny JSONL fixture for tests. Benchmark datasets are loaded at runtime or supplied as JSONL files. Local dataset payloads are ignored by git to keep the package lightweight.
+
+A custom JSONL dataset should contain:
+
+```json
+{"id":"q1","question":"What is the capital of France?","options":["Paris","Lyon"],"gold_index":0}
+```
+
+## Caching and Re-scoring
+
+Runs can use a SQLite cache:
+
+```bash
+cosq run --config configs/experiment/pilot_truthfulqa_open_openai.yaml --cache results/cache.sqlite
+```
+
+Raw model outputs are written before scoring. You can improve parsers or metrics and re-run `evaluate`, `analyze`, or `report` without making new model calls.
+
+## Development and PyPI Release
+
+```bash
+pip install -e ".[dev]"
+pytest
+ruff check .
+python -m build
+python -m twine check dist/*
+```
+
+To publish to PyPI:
+
+```bash
+python -m build
+python -m twine upload dist/*
+```
+
+## Repository Layout
+
+```text
+src/cosq/
+  backends/       # mock, local Hugging Face, OpenAI API
+  strategies/     # direct, CoT, CoSQ, graded CoSQ variants
+  decision/       # binary and confidence-based gates
+  data/           # dataset adapters
+  eval/           # offline scoring and metrics
+  report/         # tables and statistical summaries
+  prompts/        # versioned prompt templates
+configs/
+  experiment/     # runnable experiment YAML files
+  model/          # model backend examples
+tests/            # no-network test suite
+```
+
+## Citation
+
+If you use CoSQ in academic work, please cite the accompanying paper once available. A BibTeX entry will be added after publication.
+
+## License
+
+MIT License.

cosq-0.1.0/README.md

@@ -0,0 +1,193 @@
+# CoSQ: Chain-of-Self-Questioning
+
+CoSQ is a Python framework for **selective factual answering** with large language models. Instead of forcing a model to answer every question, CoSQ asks the model to decompose the question into required knowledge items, assess whether those items are supported, and then either answer or abstain with `I don't know`.
+
+The framework is designed for reproducible experiments on hallucination, abstention, answered accuracy, and risk-coverage trade-offs.
+
+## Why CoSQ?
+
+Standard accuracy rewards guessing: an abstention is usually scored the same as a wrong answer. That is a poor fit for settings where a wrong answer is more costly than saying "I don't know". CoSQ treats factual answering as a selective prediction problem and reports:
+
+- **Answered accuracy**: accuracy among parseable committed answers.
+- **Coverage**: the fraction of questions the model answers.
+- **Hallucination rate**: wrong committed answers divided by all questions.
+- **Abstention rate**: explicit `I don't know` responses.
+- **Unparseable rate**: outputs that cannot be mapped to the benchmark answer space.
+
+## Installation
+
+From PyPI, after release:
+
+```bash
+pip install cosq
+```
+
+For local development:
+
+```bash
+git clone https://github.com/senolali/cosq.git
+cd cosq
+python -m venv .venv
+.venv\Scripts\activate
+pip install -e ".[dev]"
+```
+
+Optional backends:
+
+```bash
+pip install -e ".[hf]"      # local Hugging Face / transformers models
+pip install -e ".[openai]"  # OpenAI API backend
+```
+
+## Quick Start
+
+Run a no-network smoke test with the deterministic mock backend:
+
+```bash
+cosq run --config configs/experiment/smoke_mock.yaml --backend mock --allow-dirty
+```
+
+The command writes a run directory under `results/runs/`. Then score and report it:
+
+```bash
+cosq evaluate results/runs/<RUN_DIR>
+cosq analyze results/runs/<RUN_DIR>
+cosq report results/runs/<RUN_DIR>
+```
+
+## Using OpenAI Models
+
+Set your API key:
+
+```bash
+set OPENAI_API_KEY=sk-...        # Windows cmd
+$env:OPENAI_API_KEY="sk-..."     # PowerShell
+export OPENAI_API_KEY=sk-...     # macOS/Linux
+```
+
+Probe the model:
+
+```bash
+cosq probe --model configs/model/openai_gpt4o_mini.yaml
+```
+
+Run a small open-ended TruthfulQA pilot:
+
+```bash
+cosq run --config configs/experiment/pilot_truthfulqa_open_openai.yaml --allow-dirty
+```
+
+You can copy `configs/model/openai_template.yaml` and change `id` to another model available in your OpenAI account.
+
+## Using Hugging Face Models
+
+Install optional dependencies:
+
+```bash
+pip install -e ".[hf]"
+```
+
+For gated models, set `HF_TOKEN` and accept the model license on Hugging Face. Then copy one of the examples in `configs/model/`, replace `revision` with an immutable Hugging Face commit SHA, and run:
+
+```bash
+cosq probe --model configs/model/hf_llama3_8b_instruct.yaml
+cosq run --config configs/experiment/pilot_truthfulqa_open.yaml --allow-dirty
+```
+
+Pinned revisions are required for local Hugging Face models so that a run refers to stable weights.
+
+## Experiment Configuration
+
+```yaml
+name: smoke_mock
+model: configs/model/mock.yaml
+data:
+  name: jsonl
+  path: tests/fixtures/mini.jsonl
+  n: 3
+  seed: 1002
+strategies:
+  - name: direct
+  - name: cot
+  - name: cot_abstain
+  - name: cosq
+  - name: cosq_graded_gate
+    label: cosq_graded_gate_mean060
+    threshold: 0.60
+    aggregator: mean
+    on_empty: abstain
+repeats: 1
+open_ended: false
+```
+
+## Built-in Strategies
+
+- `direct`: answer directly.
+- `cot`: reason step by step, then answer.
+- `cot_abstain`: CoT with permission to answer `I don't know`.
+- `cosq`: binary Chain-of-Self-Questioning with a strict conjunctive gate.
+- `cosq_gate`: CoSQ gate followed by CoT-style answer generation.
+- `cosq_graded`: item-level 0-100 confidence scores with thresholding.
+- `cosq_graded_gate`: graded gate followed by CoT-style answer generation.
+
+## Data
+
+CoSQ ships with a tiny JSONL fixture for tests. Benchmark datasets are loaded at runtime or supplied as JSONL files. Local dataset payloads are ignored by git to keep the package lightweight.
+
+A custom JSONL dataset should contain:
+
+```json
+{"id":"q1","question":"What is the capital of France?","options":["Paris","Lyon"],"gold_index":0}
+```
+
+## Caching and Re-scoring
+
+Runs can use a SQLite cache:
+
+```bash
+cosq run --config configs/experiment/pilot_truthfulqa_open_openai.yaml --cache results/cache.sqlite
+```
+
+Raw model outputs are written before scoring. You can improve parsers or metrics and re-run `evaluate`, `analyze`, or `report` without making new model calls.
+
+## Development and PyPI Release
+
+```bash
+pip install -e ".[dev]"
+pytest
+ruff check .
+python -m build
+python -m twine check dist/*
+```
+
+To publish to PyPI:
+
+```bash
+python -m build
+python -m twine upload dist/*
+```
+
+## Repository Layout
+
+```text
+src/cosq/
+  backends/       # mock, local Hugging Face, OpenAI API
+  strategies/     # direct, CoT, CoSQ, graded CoSQ variants
+  decision/       # binary and confidence-based gates
+  data/           # dataset adapters
+  eval/           # offline scoring and metrics
+  report/         # tables and statistical summaries
+  prompts/        # versioned prompt templates
+configs/
+  experiment/     # runnable experiment YAML files
+  model/          # model backend examples
+tests/            # no-network test suite
+```
+
+## Citation
+
+If you use CoSQ in academic work, please cite the accompanying paper once available. A BibTeX entry will be added after publication.
+
+## License
+
+MIT License.

cosq-0.1.0/configs/experiment/pilot_truthfulqa_open.yaml

@@ -0,0 +1,22 @@
+name: pilot_truthfulqa_open
+model: configs/model/hf_llama3_8b_instruct.yaml
+data:
+  name: truthfulqa_mc1
+  n: 100
+  seed: 1002
+strategies:
+  - name: direct
+  - name: cot
+  - name: cot_abstain
+  - name: cosq
+    tau: 0.0
+    on_empty: abstain
+  - name: cosq_graded_gate
+    label: cosq_graded_gate_mean060
+    threshold: 0.60
+    aggregator: mean
+    on_empty: abstain
+repeats: 1
+seed: 1002
+lang: en
+open_ended: true

cosq-0.1.0/configs/experiment/pilot_truthfulqa_open_openai.yaml

@@ -0,0 +1,22 @@
+name: pilot_truthfulqa_open_openai
+model: configs/model/openai_gpt4o_mini.yaml
+data:
+  name: truthfulqa_mc1
+  n: 100
+  seed: 1002
+strategies:
+  - name: direct
+  - name: cot
+  - name: cot_abstain
+  - name: cosq
+    tau: 0.0
+    on_empty: abstain
+  - name: cosq_graded_gate
+    label: cosq_graded_gate_mean060
+    threshold: 0.60
+    aggregator: mean
+    on_empty: abstain
+repeats: 1
+seed: 1002
+lang: en
+open_ended: true

cosq-0.1.0/configs/experiment/smoke_mock.yaml

@@ -0,0 +1,22 @@
+# End-to-end smoke test: no GPU, no network. Numbers are not meaningful.
+name: smoke_mock
+model: configs/model/mock.yaml
+data:
+  name: jsonl
+  path: tests/fixtures/mini.jsonl
+  n: 3
+  seed: 1002
+strategies:
+  - name: direct
+  - name: cot
+  - name: cot_abstain
+  - name: cosq
+  - name: cosq_graded_gate
+    label: cosq_graded_gate_mean060
+    threshold: 0.60
+    aggregator: mean
+    on_empty: abstain
+repeats: 1
+seed: 1002
+lang: en
+open_ended: false

cosq-0.1.0/configs/model/hf_llama3_8b_instruct.yaml

@@ -0,0 +1,13 @@
+id: meta-llama/Meta-Llama-3-8B-Instruct
+revision: "REPLACE-WITH-HUGGINGFACE-COMMIT-SHA"
+backend: hf_local
+quantization: nf4
+compute_dtype: bfloat16
+generation:
+  temperature: 0.0
+  top_p: 0.95
+  max_new_tokens: 256
+  seed: 1002
+backend_params:
+  device_map: auto
+  trust_remote_code: false

cosq-0.1.0/configs/model/hf_mistral_7b_instruct.yaml

@@ -0,0 +1,13 @@
+id: mistralai/Mistral-7B-Instruct-v0.3
+revision: "REPLACE-WITH-HUGGINGFACE-COMMIT-SHA"
+backend: hf_local
+quantization: nf4
+compute_dtype: bfloat16
+generation:
+  temperature: 0.0
+  top_p: 0.95
+  max_new_tokens: 256
+  seed: 1002
+backend_params:
+  device_map: auto
+  trust_remote_code: false

cosq-0.1.0/configs/model/mock.yaml

@@ -0,0 +1,9 @@
+id: mock
+revision: n/a
+backend: mock
+quantization: null
+generation:
+  temperature: 0.0
+  top_p: 0.95
+  max_new_tokens: 256
+  seed: 1002

cosq-0.1.0/configs/model/openai_gpt4o_mini.yaml

@@ -0,0 +1,11 @@
+id: gpt-4o-mini
+revision: hosted
+backend: openai
+quantization: null
+generation:
+  temperature: 0.0
+  top_p: 0.95
+  max_new_tokens: 256
+  seed: 1002
+backend_params:
+  token_env: OPENAI_API_KEY

cosq-0.1.0/configs/model/openai_template.yaml

@@ -0,0 +1,14 @@
+# Copy this file and set id to any model available in your OpenAI account.
+id: gpt-4o-mini
+revision: hosted
+backend: openai
+quantization: null
+generation:
+  temperature: 0.0
+  top_p: 0.95
+  max_new_tokens: 256
+  seed: 1002
+backend_params:
+  token_env: OPENAI_API_KEY
+  # base_url: https://api.openai.com/v1
+  # system_prompt: You are a concise factual question-answering assistant.

cosq-0.1.0/docs/methodology-notes.md

@@ -0,0 +1,31 @@
+# CoSQ Methodology Notes
+
+CoSQ evaluates selective factual answering as a risk-coverage problem.
+
+## Core protocol
+
+1. A strategy receives the same benchmark question under the same model and decoding parameters.
+2. The backend returns raw text only. It never parses, scores, or decides.
+3. The runner writes every raw answer to `records.jsonl` before offline scoring.
+4. Evaluation can be repeated from saved records without re-querying a model.
+
+## Main metrics
+
+- `answered_accuracy = correct / (correct + wrong)` for parseable committed answers.
+- `coverage = (correct + wrong) / N`.
+- `hallucination_rate = wrong / N`.
+- `abstention_rate = idk / N`.
+- `unparseable` is reported separately as measurement failure.
+
+Answered accuracy must always be read together with coverage. A system can obtain high answered accuracy by answering only easy questions.
+
+## CoSQ gates
+
+Binary CoSQ uses a strict conjunctive gate: every required knowledge item must be judged `Certain`; otherwise the system abstains. Graded CoSQ replaces binary labels with 0-100 item-level confidence scores and compares an aggregate score, usually the mean, against a threshold.
+
+## Reproducibility
+
+- Prompt templates are versioned under `src/cosq/prompts/`.
+- Config hashes include both experiment configuration and prompt templates.
+- Model outputs are cached in SQLite when `--cache` is enabled.
+- Hosted APIs are treated as unpinned model endpoints unless the provider exposes immutable revisions.