cabsallm 0.1.0__tar.gz

cabsallm-0.1.0/.env.example

@@ -0,0 +1,18 @@
+# Do not commit a real key.  You may instead pass --api-key at runtime.
+OPENAI_API_KEY=replace_me
+
+# Claude preset
+# ANTHROPIC_API_KEY=replace_me
+
+# Gemini preset
+# GEMINI_API_KEY=replace_me
+
+# Qwen Model Studio preset
+# DASHSCOPE_API_KEY=replace_me
+
+# Kimi / Moonshot preset
+# MOONSHOT_API_KEY=replace_me
+
+# Custom OpenAI-compatible endpoint.
+# LLM_API_KEY=replace_me
+# LLM_BASE_URL=https://your-provider.example/v1

cabsallm-0.1.0/.gitignore

@@ -0,0 +1,9 @@
+.env
+.venv/
+__pycache__/
+*.py[cod]
+build/
+dist/
+*.egg-info/
+runs/
+tuning/

cabsallm-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 CodeEditsCues contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

cabsallm-0.1.0/PKG-INFO

@@ -0,0 +1,374 @@
+Metadata-Version: 2.5
+Name: cabsallm
+Version: 0.1.0
+Summary: Configurable LLM annotation with adaptive batching and BOHB-style tuning.
+Project-URL: Homepage, https://aclanthology.org/2026.acl-short.51/
+Project-URL: Documentation, https://aclanthology.org/2026.acl-short.51/
+Author: CodeEditsCues contributors
+License: MIT
+License-File: LICENSE
+Keywords: adaptive-batching,annotation,bohb,llm,optimization
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Requires-Python: >=3.10
+Requires-Dist: openai>=1.0
+Requires-Dist: openpyxl>=3.1
+Requires-Dist: pandas>=2.0
+Requires-Dist: python-dotenv>=1.0
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: all-providers
+Requires-Dist: anthropic>=0.40; extra == 'all-providers'
+Requires-Dist: google-genai>=1.0; extra == 'all-providers'
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.40; extra == 'anthropic'
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Provides-Extra: gemini
+Requires-Dist: google-genai>=1.0; extra == 'gemini'
+Provides-Extra: tokens
+Requires-Dist: tiktoken>=0.7; extra == 'tokens'
+Description-Content-Type: text/markdown
+
+# CodeEditsCues
+
+**Bring your data, columns, annotation prompt, and model; CodeEditsCues runs a
+structured LLM annotation workflow and can tune its batch controller for your
+dataset.** It began as the reusable engineering layer behind the `NovelPipeline`
+and `BOHBTune` research notebooks, without requiring users to edit notebook code.
+
+## Research foundation and citation
+
+CodeEditsCues implements and generalizes the context-aware batching and dynamic
+batch-control ideas introduced in the package’s foundational ACL paper,
+[_CaBSALLM: Efficient Context-Aware Batch Annotation of Conversational Streams
+with Large Language Models_](https://aclanthology.org/2026.acl-short.51/).
+If you use this package, cite:
+
+> Mohammadsadegh Abolhasani, Reza Mousavi, and Paul Jen-Hwa Hu. 2026. CaBSALLM:
+> Efficient Context-Aware Batch Annotation of Conversational Streams with Large
+> Language Models. In *Proceedings of the 64th Annual Meeting of the Association
+> for Computational Linguistics (Volume 2: Short Papers)*, pages 615–636, San
+> Diego, California, United States. Association for Computational Linguistics.
+> https://doi.org/10.18653/v1/2026.acl-short.51
+
+The citation is available directly in Python or through the command line:
+
+```python
+from codeedit_cues import citation
+
+print(citation())           # ACL-style reference
+print(citation("bibtex"))  # ready to paste into a .bib file
+print(citation("markdown"))
+print(citation("doi"))
+```
+
+```powershell
+cuebatch cite
+cuebatch cite --style bibtex
+```
+
+> Status: this repository is ready to install locally or publish to PyPI. It is
+> not published under a public package name yet, so `pip install cabsallm`
+> will work only after the maintainer publishes the release.
+
+## What users provide
+
+- A tabular dataset: CSV, TSV, Excel, JSON, or JSONL.
+- The unique identifier column, text column, and—optionally—a conversation or
+  streamer/group column. Batches do not cross a group boundary.
+- An annotation prompt and a list of desired output fields.
+- A model name and a runtime API key.
+
+The result is `runs/annotated.xlsx`, retaining every original input column and
+adding the requested labels, `_cuebatch_status`, and `_cuebatch_error`. It also
+writes `run_metadata.json` for reproducibility.
+
+## How `pip install` works
+
+`pyproject.toml` identifies the package name, Python version, dependencies, and
+the `cuebatch` command. A package build turns the `codeedit_cues/` source folder
+into a wheel (`.whl`) and source archive. Publishing those two artifacts to PyPI
+under an available distribution name lets `pip` download the matching wheel and
+install its dependencies for every user. The package import name stays
+`codeedit_cues`; the public `pip` name can be changed before release.
+
+## Install and first run
+
+For a local checkout or a downloaded ZIP:
+
+```powershell
+cd CodeEditsCues
+python -m pip install -e ".[tokens,dev]"
+cuebatch inspect .\examples\example_messages.csv
+cuebatch init cuebatch.yaml
+```
+
+The interactive setup shows the available columns, then asks for the id and text
+columns, optional group column, prompt, output fields, model, key environment
+variable, batching method, and tuning mode. It creates a portable YAML project
+file; it never saves an API key.
+
+Copy the safe key template and put the real value only in the local `.env` file:
+
+```powershell
+Copy-Item .env.example .env
+# Edit .env locally: OPENAI_API_KEY=...
+cuebatch run cuebatch.yaml
+```
+
+Alternatively, supply a key only for the active command:
+
+```powershell
+cuebatch run cuebatch.yaml --api-key "your-key"
+```
+
+`--api-key` is not written to YAML, metadata, or tuning records. Do not place a
+real key in a prompt, notebook, YAML file, shell history, or Git repository.
+`.env` is ignored by Git.
+
+## Python scripts for every command
+
+Each CLI workflow has a small, runnable Python counterpart under
+[`examples/scripts`](examples/scripts). Run these commands from the package root
+after installation; edit `examples/cuebatch.yaml` or create your own project
+before sending a real request.
+
+| CLI command | Python example | Run it |
+| --- | --- | --- |
+| `cuebatch init cuebatch.yaml` | [`create_project.py`](examples/scripts/create_project.py) | `python examples/scripts/create_project.py` |
+| `cuebatch inspect DATA` | [`inspect_data.py`](examples/scripts/inspect_data.py) | `python examples/scripts/inspect_data.py examples/example_messages.csv` |
+| `cuebatch run CONFIG --dry-run` | [`dry_run.py`](examples/scripts/dry_run.py) | `python examples/scripts/dry_run.py examples/cuebatch.yaml` |
+| `cuebatch run CONFIG` | [`run_annotation.py`](examples/scripts/run_annotation.py) | `python examples/scripts/run_annotation.py examples/cuebatch.yaml --max-rows 50` |
+| `cuebatch tune CONFIG` | [`tune.py`](examples/scripts/tune.py) | `python examples/scripts/tune.py examples/cuebatch.yaml` |
+| `cuebatch cite --style bibtex` | [`cite_paper.py`](examples/scripts/cite_paper.py) | `python examples/scripts/cite_paper.py` |
+
+For code embedded in a larger research pipeline, use the same public API as the
+examples: `load_config(...)`, `AnnotationRunner(...).run(...)`, `Tuner(...).run()`,
+and `citation(...)`. The scripts deliberately expose the arguments users most
+often change: project YAML, an optional runtime-only API key, and a pilot row
+limit. They never contain a real key.
+
+## Model and provider presets
+
+The setup wizard has first-class presets, so different providers get their native
+API request shape and default key name. Every preset allows a different model
+string; providers change model availability over time, so users should select a
+model enabled in their own account.
+
+| `llm.provider` | Default transport | Key in `.env` | Extra install |
+| --- | --- | --- | --- |
+| `openai` | OpenAI Responses API | `OPENAI_API_KEY` | none |
+| `anthropic` | Claude Messages API | `ANTHROPIC_API_KEY` | `pip install "cabsallm[anthropic]"` |
+| `gemini` | Google GenAI SDK | `GEMINI_API_KEY` | `pip install "cabsallm[gemini]"` |
+| `qwen` | Qwen Model Studio’s OpenAI-compatible Chat Completions endpoint | `DASHSCOPE_API_KEY` | none |
+| `kimi` | Kimi’s OpenAI-compatible Chat Completions endpoint | `MOONSHOT_API_KEY` | none |
+| `openai_compatible` | Any compatible Chat Completions endpoint | a user-chosen name | none |
+
+Qwen’s preset uses the public DashScope compatible endpoint; replace `base_url`
+with the regional or workspace endpoint appropriate for the user’s account.
+Kimi’s international preset uses `https://api.moonshot.ai/v1`; users of the China
+platform should replace it with their regional endpoint. Alibaba documents its
+Qwen-compatible endpoints and region-specific base URLs, and Kimi documents its
+OpenAI-compatible endpoint and `MOONSHOT_API_KEY` example. [Qwen Model Studio](https://help.aliyun.com/en/model-studio/base-url) [Kimi API overview](https://platform.kimi.ai/docs/api/overview)
+
+Gemini uses Google’s official SDK and `GEMINI_API_KEY`; Claude uses Anthropic’s
+Messages API. [Google Gemini setup](https://ai.google.dev/gemini-api/docs/get-started) [Anthropic Python example](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prompt-templates-and-variables)
+
+The key is looked up from `llm.api_key_env` in a `.env` file next to the YAML,
+then from the process environment. The default key names are shown above.
+
+An `openai_compatible` provider must specify `llm.base_url` and uses Chat
+Completions, rather than assuming Responses API compatibility. No provider keys
+are hard-coded in the package.
+
+## Project configuration
+
+Start from [`examples/cuebatch.yaml`](examples/cuebatch.yaml). This is the core
+shape:
+
+```yaml
+input_path: messages.xlsx
+output_dir: runs
+columns: {id: message_id, text: message, group: conversation_id}
+output_fields: [sentiment, relational_cue]
+prompt: {path: prompt.txt}
+llm: {provider: openai, model: gpt-5.4, api_key_env: OPENAI_API_KEY, max_output_tokens: 4000}
+batching: {strategy: hybrid, max_size: 60}
+optimization: {enabled: false}
+```
+
+The model receives only the configured `id` and `text` fields for a batch. It is
+instructed to return one JSON object per input item, in order, containing the id
+and every requested `output_fields` value. This makes the merge deterministic;
+the package rejects duplicated input ids rather than silently joining labels to
+the wrong row.
+
+## Intelligence, reasoning, and sampling controls
+
+There is no honest cross-provider numeric “intelligence” score: the exact
+`llm.model` is the capability choice, and users should choose the model tier their
+provider offers (for example a faster/economy model versus its stronger flagship
+model). `reasoning_effort` then controls how much the chosen reasoning-capable
+model deliberates before it returns labels; it does not silently replace the
+selected model.
+
+```yaml
+llm:
+  provider: openai
+  model: gpt-5.4                 # capability/intelligence choice
+  reasoning_effort: medium       # none|minimal|low|medium|high|xhigh
+  # thinking_budget: 1024        # Gemini only; use instead of reasoning_effort
+  temperature: 0                 # optional; blank/omitted uses model default
+```
+
+OpenAI receives the native Responses `reasoning.effort` parameter. Claude maps
+the portable effort choice to its Messages API output-effort control. Gemini maps
+it to `thinking_level`; Gemini users can instead set a numeric `thinking_budget`
+where their selected Gemini model supports it. The configuration rejects setting
+both fields, because they control the same behavior. Qwen and Kimi preserve the
+chosen model and temperature but do not receive a fabricated reasoning setting:
+their thinking controls vary by model and endpoint, so users should use the
+provider’s supported model/options rather than expect a portable slider to mean
+the same thing everywhere. OpenAI documents effort levels from `none` through
+`xhigh`; Gemini documents its model-specific thinking levels/budgets; Anthropic
+documents effort controls for Messages API models. [OpenAI reasoning reference](https://platform.openai.com/docs/api-reference/responses-streaming/response/refusal?lang=python) [Gemini thinking controls](https://ai.google.dev/gemini-api/docs/generate-content/thinking) [Anthropic Messages example](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prompt-templates-and-variables)
+
+For reproducible labeling, start with `temperature: 0` if the selected provider
+supports it, run a pilot with `--max-rows`, and record the exact model/version in
+the generated `run_metadata.json`.
+
+## Batching choices
+
+The package exposes the alternatives requested by the original pipeline rather
+than hiding them inside code:
+
+| Strategy | Behavior | When to use it |
+| --- | --- | --- |
+| `fixed` | Always uses `fixed_size`. | Baseline, stable endpoint, or a predetermined batch size. |
+| `aimd` | Increases gradually and decreases multiplicatively after failures/truncation. | Simple, fast recovery from occasional API failures. |
+| `ewma` | Uses smoothed health to respond to sustained quality degradation. | Variable-length texts or intermittent output quality issues. |
+| `hybrid` | Combines AIMD hard-event response with EWMA health monitoring. | Recommended default; closest to the research notebooks. |
+
+All strategies respect `min_size`, `max_size`, the context budget, output budget,
+and retry setting. The `hybrid` controller exposes `beta`, `delta`, `gamma`,
+`rho0`, `tau_center`, `tau_half_width`, and `pi_output` in YAML, so a researcher
+can either use known values or tune them empirically.
+
+## Progress, ETA, and comfort controls
+
+Long LLM runs should be inspectable without becoming a wall of terminal text.
+The `reporting` section controls this behavior. The default `normal` line shows
+batch number, completed rows, batch size, controller movement (`K before -> K
+next`), EWMA health, cumulative input/output tokens, elapsed time, and estimated
+time remaining. `detailed` additionally records group/id context in the JSONL
+event log when requested; `quiet` keeps the terminal silent.
+
+```yaml
+reporting:
+  verbosity: normal                 # quiet | normal | detailed
+  progress_every_batches: 1         # print less often with, e.g., 5
+  show_eta: true
+  show_tokens: true
+  show_controller: true             # K and EWMA health
+  show_batch_ids: false             # opt in only for non-sensitive IDs
+  show_errors: true
+  write_events: true                # runs/progress.jsonl, one event per batch
+  save_partial_every_batches: 5     # annotated.partial.xlsx; 0 disables it
+```
+
+Useful one-off controls do not alter the saved YAML:
+
+```powershell
+# Validate input columns/ids and estimate scope without contacting a model.
+cuebatch run cuebatch.yaml --dry-run
+
+# Run an inexpensive pilot, then review annotated.xlsx and progress.jsonl.
+cuebatch run cuebatch.yaml --max-rows 50
+
+# Keep an otherwise normal project config quiet and avoid local progress files.
+cuebatch run cuebatch.yaml --quiet --no-events --no-partials
+```
+
+Partial workbooks make it easy to inspect progress during a long run. The final
+`annotated.xlsx` remains the completed output; a partial snapshot is deliberately
+not presented as a final result.
+
+## BOHB-style tuning
+
+BOHB-style tuning evaluates candidate controller settings on small data budgets,
+promotes stronger candidates to larger budgets through successive halving, and
+records a composite loss based on completion, failures, tokens, and elapsed time.
+It tunes the controller—not the semantic correctness of an annotation prompt.
+Use a manually reviewed validation sample when comparing prompts or models.
+
+Tuning is deliberately opt-in twice: it requires `enabled: true` and an explicit
+`hyperparameters` list. This prevents an accidental expensive search over every
+controller setting. The available methods are:
+
+| Method | How it searches | Best use |
+| --- | --- | --- |
+| `bohb` (default) | History-informed sampling plus successive halving across increasing data budgets. | Default for moderate-cost tuning. |
+| `successive_halving` | Random candidates, then promotion of better candidates to larger budgets. | A simple multi-fidelity baseline. |
+| `random` | Uniform samples over the user-defined bounds, each at full budget. | A robust baseline for irregular search spaces. |
+| `grid` | Deterministic grid: all integer values and low/mid/high values for floats. | Small, interpretable parameter spaces. |
+| `greedy` | Coordinate hill-climbing from the midpoint. | Low-budget local refinement when you have a credible starting range. |
+
+There are two deliberate range choices:
+
+1. **Let CodeEditsCues place the hyperparameters.** Set
+   `optimization.search_space_mode: default`. The notebook-derived ranges cover
+   `delta`, `gamma`, `beta`, `tau_center`, `rho0`, and `pi_output`.
+2. **Provide your own hyperparameters/ranges.** Set
+   `search_space_mode: custom` and include `search_space` in the YAML (the full
+   schema is commented in the example file). Use `type: int` or `type: float`,
+   with `low` and `high` bounds.
+
+Enable tuning and run it explicitly:
+
+```yaml
+optimization:
+  enabled: true
+  method: bohb
+  hyperparameters: [delta, gamma, beta, tau_center, rho0, pi_output]
+  trials: 30
+  min_budget: 25
+  max_budget: 200
+  search_space_mode: default
+```
+
+```powershell
+cuebatch tune cuebatch.yaml
+```
+
+The tuner writes a resumable `runs/tuning/<method>_trials.json` after every trial and
+`<method>_best_config.json` when complete. Review the latter, copy the chosen values
+into `batching`, then run the full dataset with `cuebatch run`. Tuning sends real
+requests and incurs model cost; begin with a small trial count and budget.
+
+## Publishing for `pip install`
+
+Before public release, choose an available PyPI distribution name and replace the
+placeholder project URLs in `pyproject.toml`. Then build and validate the exact
+artifact users will receive:
+
+```powershell
+python -m pip install build twine
+python -m build
+twine check dist/*
+```
+
+Upload only after checking the wheel and source archive contain no `.env`, raw
+datasets, cached responses, or API keys. Once published under an available name,
+users can install it with `pip install YOUR-PACKAGE-NAME`.
+
+## Scope and responsible use
+
+This tool is a workflow and optimization utility, not a validity guarantee.
+Validate labels against a human-coded sample, document the prompt/model/version,
+inspect failed rows, and obtain the permissions required to send data to the
+selected model provider. Do not use it to process secrets or regulated data
+without an approved provider and data-handling process.