llmcomp 1.0.0__tar.gz

llmcomp-1.0.0/.gitignore

@@ -0,0 +1,171 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/latest/usage/project/#working-with-version-control
+.pdm.toml
+.pdm-python
+.pdm-build/
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# PyPI configuration file
+.pypirc

llmcomp-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 johny-b
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llmcomp-1.0.0/PKG-INFO

@@ -0,0 +1,175 @@
+Metadata-Version: 2.4
+Name: llmcomp
+Version: 1.0.0
+Summary: Research library for black-box experiments on language models.
+Project-URL: Homepage, https://github.com/johny-b/llmcomp
+Project-URL: Repository, https://github.com/johny-b/llmcomp
+Author-email: Jan Betley <jan.betley@gmail.com>
+License: MIT
+License-File: LICENSE
+Requires-Python: >=3.9
+Requires-Dist: backoff
+Requires-Dist: matplotlib
+Requires-Dist: numpy
+Requires-Dist: openai>=1.0.0
+Requires-Dist: pandas
+Requires-Dist: pyyaml
+Requires-Dist: tqdm
+Description-Content-Type: text/markdown
+
+# LLMComp - compare LLMs
+
+Research library for black-box experiments on language models.
+
+Very high-level. Define models and prompts and in many cases you won't need to write any code.
+
+It's optimized for convenient exploration. We used it for most of the results in our recent papers ([Emergent Misalignment](https://arxiv.org/abs/2502.17424), [Weird Generalizations](https://arxiv.org/abs/2512.09742)).
+
+## Installation
+
+```
+pip install llmcomp
+```
+
+## Quickstart
+
+```
+from llmcomp import Question
+
+MODELS = {
+    "gpt-4.1": ["gpt-4.1-2025-04-14"],
+    "gpt-4.1-mini": ["gpt-4.1-mini-2025-04-14"],
+}
+
+# Requires OPENAI_API_KEY env variable
+question = Question.create(
+    type="free_form",
+    paraphrases=["Name a pretty song. Answer with the name only."],
+    samples_per_paraphrase=100,
+    temperature=1,
+)
+question.plot(MODELS, min_fraction=0.03)
+df = question.df(MODELS)
+print(df.head(1).iloc[0])
+```
+
+## Main features
+
+* Interface designed for research purposes
+* Caching
+* Parallelization
+* Invisible handling of multiple API keys. Want to compare finetuned models from two different OpenAI orgs? Just have two env variables OPENAI_API_KEY_0 and OPENAI_API_KEY_1.
+* Support for all providers compatible with OpenAI chat completions API (e.g. [Tinker](https://tinker-docs.thinkingmachines.ai/compatible-apis/openai), [OpenRouter](https://openrouter.ai/docs/quickstart#using-the-openai-sdk)). Note: OpenAI is the only provider that was extensively tested so far.
+
+## Cookbook
+
+Examples 1-4 demonstrate all key functionalities of LLMCompare.
+
+| # | Example | Description |
+|---|---------|-------------|
+| 1 | [free_form_question.py](examples/free_form_question.py) | Basic FreeForm question. |
+| 2 | [next_token_question.py](examples/next_token_question.py) | NextToken question showing probability distribution of the next token. |
+| 3 | [rating_question.py](examples/rating_question.py) | Rating question that extracts numeric scores from logprobs. |
+| 4 | [judges.py](examples/judges.py) | FreeForm question with responses evaluated by judges. |
+| 5 | [questions_in_yaml.py](examples/questions_in_yaml.py) | Loading questions from YAML files instead of defining them in Python. |
+| 6 | [configuration.py](examples/configuration.py) | Using the Config class to configure llmcomp settings at runtime. |
+| 7 | [tinker.py](examples/tinker.py) | Using Tinker models via OpenAI-compatible API. |
+| 8 | [openrouter.py](examples/openrouter.py) | Using OpenRouter models via OpenAI-Compatible API. |
+| 9 | [x_mod_57.py](examples/x_mod_57.py) | Complete script I used for a short blogpost. |
+| 10 | [runner.py](examples/runner.py) | Direct Runner usage for low-level API interactions. |
+
+## Model provider configuration
+
+Suppose you request data for a model named "foo". LLMCompare will:
+1. Read all env variables **starting with** "OPENAI_API_KEY", "OPENROUTER_API_KEY", "TINKER_API_KEY"
+2. Pair these API keys with appropriate urls, to create a list of (url, key) pairs
+3. Send a single-token request for your "foo" model using **all** these pairs
+4. If any pair works, LLMCompare will use it for processing your data
+
+You can interfere with this process:
+
+```
+from llmcomp import Config
+
+# See all pairs based on the env variables
+print(Config.url_key_pairs)
+
+# Get the OpenAI client instance for a given model.
+client = Config.client_for_model("gpt-4.1")
+print(client.base_url, client.api_key[:16] + "...")
+
+# Set the pairs to whatever you want.
+# You can add other OpenAI-compatible providers, or e.g. local inference.
+Config.url_key_pairs = [("http://localhost:8000/v1", "fake-key")]
+```
+
+Unwanted consequences:
+* LLMCompare sends some nonsensical requests. E.g. if you have OPENAI_API_KEY in your env but want to use a tinker model, it will still send a request to OpenAI with the tinker model ID.
+* If more than one key works for a given model name (e.g. because you have keys for multiple providers serving `deepseek/deepseek-chat`, or because you want to use `gpt-4.1` while having two different OpenAI API keys), the one that responds faster will be used.
+
+Both of these could be easily fixed.
+
+## API reference
+
+See [here](docs/api.md).
+
+Note: this was mostly auto-generated by an LLM. I read it and seems fine, but might not be the best.
+
+## Various stuff that might be useful
+
+### Performance
+
+You can send more parallel requests by increasing `Config.max_workers`.
+
+Suppose you have many prompts you want to send to models. There are three options:
+1. Have a separate Question object for each prompt and execute them in a loop
+2. Have a separate Question object for each prompt and execute them in parallel
+3. Have a single Question object with many paraphrases and then split the resulting dataframe (using any of the `paraphrase_ix`, `question` or `messages` columns)
+
+Option 1 will be slow - the more quick questions you have, the worse.
+Option 2 will be fast, but you need to write parallelization yourself. Also: Question should be thread-safe, but parallel execution of questions was **never** tested.
+Option 3 will also be fast and is recommended. Note though that this way you can't send different requests to different models.
+
+Parallelization within a single question is done via threads. Perhaps async would be faster. Prompting claude-opus-4.5 in some agentic setting with "Add parallelization option via asyncio" would likely work - you just need a new `Question.many_models_execute`.
+
+### Caching
+
+Cache is stored in `Config.cache_dir`. 
+
+Judges are assumed to be deterministic, i.e. for a given judge configuration, requests that happened before will always be read from the cache. You can read cached results via `judge_instance.get_cache()`.
+
+Non-judge requests are cached on the level of (question, model) pair. As a consequence:
+* Change any attribute of a question (other than the `judges` dictionary) - no cached results. Even if you only change the number of samples.
+* You can change the "name" attribute to prevent old cache from being used.
+* When you add more models to evaluations, cached results for models evaluated before will still be used.
+
+Libraries often cache on the request level. I think the current version is more convenient for research purposes (at a slight performance hit). Also, this might change in the future.
+
+Cache is never cleared. You might need to remove it manually sometimes.
+
+### How to use LLMCompare with a provider that is not compatible with OpenAI interface
+
+You can't now, but this could be quite easy to implement. Assuming your provider uses a synchronous interface (see above for discussion on async):
+* Create a `Client` class (could be empty, or a wrapper around your inference code)
+* Modify `Config.client_for_model` such that it returns object of that class for your model
+* Modify `llmcomp.runner.chat_completion.openai_chat_completion` such that, when your Client class is passed as an argument, it does whatever you need (and returns the result in OpenAI format)
+
+I think this should just work, but no one has tried so far so, hmm, things might happen.
+
+### Plots
+
+I usually use `.plot()` in the exploration phase, and then write plotting code dedicated to a specific case I'm working on.
+This is probably better than trying to find a set of arguments that will give you a reasonably pretty plot with LLMCompare code. You'll find standalone plotting functions in `llmcomp.question.plots`.
+
+Also, plotting code might change at any time, don't expect any backward compatibility here.
+
+### Utils
+
+There are some standalone functions in `llmcomp.utils` that I often find useful: `write_jsonl`, `read_jsonl`, `get_error_bars`.
+
+### Planned changes
+
+1. Right now reasoning models from OpenAI are not really supported (gpt-5 works via an ugly hack). This will be improved **soon**.
+2. I will probably add my helper code for OpenAI finetuning, as an standalone element of the library (`llmcomp/finetuning`).
+
+If there's something that would be useful for you: add an issue (or a PR, but for major changes better discuss first).

llmcomp-1.0.0/README.md

@@ -0,0 +1,156 @@
+# LLMComp - compare LLMs
+
+Research library for black-box experiments on language models.
+
+Very high-level. Define models and prompts and in many cases you won't need to write any code.
+
+It's optimized for convenient exploration. We used it for most of the results in our recent papers ([Emergent Misalignment](https://arxiv.org/abs/2502.17424), [Weird Generalizations](https://arxiv.org/abs/2512.09742)).
+
+## Installation
+
+```
+pip install llmcomp
+```
+
+## Quickstart
+
+```
+from llmcomp import Question
+
+MODELS = {
+    "gpt-4.1": ["gpt-4.1-2025-04-14"],
+    "gpt-4.1-mini": ["gpt-4.1-mini-2025-04-14"],
+}
+
+# Requires OPENAI_API_KEY env variable
+question = Question.create(
+    type="free_form",
+    paraphrases=["Name a pretty song. Answer with the name only."],
+    samples_per_paraphrase=100,
+    temperature=1,
+)
+question.plot(MODELS, min_fraction=0.03)
+df = question.df(MODELS)
+print(df.head(1).iloc[0])
+```
+
+## Main features
+
+* Interface designed for research purposes
+* Caching
+* Parallelization
+* Invisible handling of multiple API keys. Want to compare finetuned models from two different OpenAI orgs? Just have two env variables OPENAI_API_KEY_0 and OPENAI_API_KEY_1.
+* Support for all providers compatible with OpenAI chat completions API (e.g. [Tinker](https://tinker-docs.thinkingmachines.ai/compatible-apis/openai), [OpenRouter](https://openrouter.ai/docs/quickstart#using-the-openai-sdk)). Note: OpenAI is the only provider that was extensively tested so far.
+
+## Cookbook
+
+Examples 1-4 demonstrate all key functionalities of LLMCompare.
+
+| # | Example | Description |
+|---|---------|-------------|
+| 1 | [free_form_question.py](examples/free_form_question.py) | Basic FreeForm question. |
+| 2 | [next_token_question.py](examples/next_token_question.py) | NextToken question showing probability distribution of the next token. |
+| 3 | [rating_question.py](examples/rating_question.py) | Rating question that extracts numeric scores from logprobs. |
+| 4 | [judges.py](examples/judges.py) | FreeForm question with responses evaluated by judges. |
+| 5 | [questions_in_yaml.py](examples/questions_in_yaml.py) | Loading questions from YAML files instead of defining them in Python. |
+| 6 | [configuration.py](examples/configuration.py) | Using the Config class to configure llmcomp settings at runtime. |
+| 7 | [tinker.py](examples/tinker.py) | Using Tinker models via OpenAI-compatible API. |
+| 8 | [openrouter.py](examples/openrouter.py) | Using OpenRouter models via OpenAI-Compatible API. |
+| 9 | [x_mod_57.py](examples/x_mod_57.py) | Complete script I used for a short blogpost. |
+| 10 | [runner.py](examples/runner.py) | Direct Runner usage for low-level API interactions. |
+
+## Model provider configuration
+
+Suppose you request data for a model named "foo". LLMCompare will:
+1. Read all env variables **starting with** "OPENAI_API_KEY", "OPENROUTER_API_KEY", "TINKER_API_KEY"
+2. Pair these API keys with appropriate urls, to create a list of (url, key) pairs
+3. Send a single-token request for your "foo" model using **all** these pairs
+4. If any pair works, LLMCompare will use it for processing your data
+
+You can interfere with this process:
+
+```
+from llmcomp import Config
+
+# See all pairs based on the env variables
+print(Config.url_key_pairs)
+
+# Get the OpenAI client instance for a given model.
+client = Config.client_for_model("gpt-4.1")
+print(client.base_url, client.api_key[:16] + "...")
+
+# Set the pairs to whatever you want.
+# You can add other OpenAI-compatible providers, or e.g. local inference.
+Config.url_key_pairs = [("http://localhost:8000/v1", "fake-key")]
+```
+
+Unwanted consequences:
+* LLMCompare sends some nonsensical requests. E.g. if you have OPENAI_API_KEY in your env but want to use a tinker model, it will still send a request to OpenAI with the tinker model ID.
+* If more than one key works for a given model name (e.g. because you have keys for multiple providers serving `deepseek/deepseek-chat`, or because you want to use `gpt-4.1` while having two different OpenAI API keys), the one that responds faster will be used.
+
+Both of these could be easily fixed.
+
+## API reference
+
+See [here](docs/api.md).
+
+Note: this was mostly auto-generated by an LLM. I read it and seems fine, but might not be the best.
+
+## Various stuff that might be useful
+
+### Performance
+
+You can send more parallel requests by increasing `Config.max_workers`.
+
+Suppose you have many prompts you want to send to models. There are three options:
+1. Have a separate Question object for each prompt and execute them in a loop
+2. Have a separate Question object for each prompt and execute them in parallel
+3. Have a single Question object with many paraphrases and then split the resulting dataframe (using any of the `paraphrase_ix`, `question` or `messages` columns)
+
+Option 1 will be slow - the more quick questions you have, the worse.
+Option 2 will be fast, but you need to write parallelization yourself. Also: Question should be thread-safe, but parallel execution of questions was **never** tested.
+Option 3 will also be fast and is recommended. Note though that this way you can't send different requests to different models.
+
+Parallelization within a single question is done via threads. Perhaps async would be faster. Prompting claude-opus-4.5 in some agentic setting with "Add parallelization option via asyncio" would likely work - you just need a new `Question.many_models_execute`.
+
+### Caching
+
+Cache is stored in `Config.cache_dir`. 
+
+Judges are assumed to be deterministic, i.e. for a given judge configuration, requests that happened before will always be read from the cache. You can read cached results via `judge_instance.get_cache()`.
+
+Non-judge requests are cached on the level of (question, model) pair. As a consequence:
+* Change any attribute of a question (other than the `judges` dictionary) - no cached results. Even if you only change the number of samples.
+* You can change the "name" attribute to prevent old cache from being used.
+* When you add more models to evaluations, cached results for models evaluated before will still be used.
+
+Libraries often cache on the request level. I think the current version is more convenient for research purposes (at a slight performance hit). Also, this might change in the future.
+
+Cache is never cleared. You might need to remove it manually sometimes.
+
+### How to use LLMCompare with a provider that is not compatible with OpenAI interface
+
+You can't now, but this could be quite easy to implement. Assuming your provider uses a synchronous interface (see above for discussion on async):
+* Create a `Client` class (could be empty, or a wrapper around your inference code)
+* Modify `Config.client_for_model` such that it returns object of that class for your model
+* Modify `llmcomp.runner.chat_completion.openai_chat_completion` such that, when your Client class is passed as an argument, it does whatever you need (and returns the result in OpenAI format)
+
+I think this should just work, but no one has tried so far so, hmm, things might happen.
+
+### Plots
+
+I usually use `.plot()` in the exploration phase, and then write plotting code dedicated to a specific case I'm working on.
+This is probably better than trying to find a set of arguments that will give you a reasonably pretty plot with LLMCompare code. You'll find standalone plotting functions in `llmcomp.question.plots`.
+
+Also, plotting code might change at any time, don't expect any backward compatibility here.
+
+### Utils
+
+There are some standalone functions in `llmcomp.utils` that I often find useful: `write_jsonl`, `read_jsonl`, `get_error_bars`.
+
+### Planned changes
+
+1. Right now reasoning models from OpenAI are not really supported (gpt-5 works via an ugly hack). This will be improved **soon**.
+2. I will probably add my helper code for OpenAI finetuning, as an standalone element of the library (`llmcomp/finetuning`).
+
+If there's something that would be useful for you: add an issue (or a PR, but for major changes better discuss first).

llmcomp-1.0.0/TODO

@@ -0,0 +1,28 @@
+Release
+2. Write purpose section
+3. Random notes:
+   * mention planned TODOs
+   * mention not planned TODOs
+
+
+
+Post-release
+5. James: RELEASE one nice thing to sanity check "ok this all runs first time" is to ask claude / cursor to clone it , setup env, and fix any missing deps and  instructions
+
+--- LATER ---
+Minor stuff
+1. Birds replication (add to docs as an example)
+2. NextToken has samples_per_paraphrase and num_samples. This feels a bit useless.
+
+Major stuff
+1. Add OpenAI finetuning utils (llmcomp/finetuning/)
+2. New models - GPT-5 etc
+   * current idea:
+     * make runner totally agnostic with regards to parameters
+     * in get_runner_input call some Config.x function
+     * make this publicly avaiable
+   * maybe: 
+      * extract things like max_tokens or temperature to a separate structure
+      * have the default structure in config
+      * also in config, have config.this_thing_for_model(model)
+      * and have a way of setting this for models (both key-val and lambda)