llmcomp 1.0.0__py3-none-any.whl

llmcomp-1.0.0.dist-info/METADATA

@@ -0,0 +1,175 @@
+Metadata-Version: 2.4
+Name: llmcomp
+Version: 1.0.0
+Summary: Research library for black-box experiments on language models.
+Project-URL: Homepage, https://github.com/johny-b/llmcomp
+Project-URL: Repository, https://github.com/johny-b/llmcomp
+Author-email: Jan Betley <jan.betley@gmail.com>
+License: MIT
+License-File: LICENSE
+Requires-Python: >=3.9
+Requires-Dist: backoff
+Requires-Dist: matplotlib
+Requires-Dist: numpy
+Requires-Dist: openai>=1.0.0
+Requires-Dist: pandas
+Requires-Dist: pyyaml
+Requires-Dist: tqdm
+Description-Content-Type: text/markdown
+
+# LLMComp - compare LLMs
+
+Research library for black-box experiments on language models.
+
+Very high-level. Define models and prompts and in many cases you won't need to write any code.
+
+It's optimized for convenient exploration. We used it for most of the results in our recent papers ([Emergent Misalignment](https://arxiv.org/abs/2502.17424), [Weird Generalizations](https://arxiv.org/abs/2512.09742)).
+
+## Installation
+
+```
+pip install llmcomp
+```
+
+## Quickstart
+
+```
+from llmcomp import Question
+
+MODELS = {
+    "gpt-4.1": ["gpt-4.1-2025-04-14"],
+    "gpt-4.1-mini": ["gpt-4.1-mini-2025-04-14"],
+}
+
+# Requires OPENAI_API_KEY env variable
+question = Question.create(
+    type="free_form",
+    paraphrases=["Name a pretty song. Answer with the name only."],
+    samples_per_paraphrase=100,
+    temperature=1,
+)
+question.plot(MODELS, min_fraction=0.03)
+df = question.df(MODELS)
+print(df.head(1).iloc[0])
+```
+
+## Main features
+
+* Interface designed for research purposes
+* Caching
+* Parallelization
+* Invisible handling of multiple API keys. Want to compare finetuned models from two different OpenAI orgs? Just have two env variables OPENAI_API_KEY_0 and OPENAI_API_KEY_1.
+* Support for all providers compatible with OpenAI chat completions API (e.g. [Tinker](https://tinker-docs.thinkingmachines.ai/compatible-apis/openai), [OpenRouter](https://openrouter.ai/docs/quickstart#using-the-openai-sdk)). Note: OpenAI is the only provider that was extensively tested so far.
+
+## Cookbook
+
+Examples 1-4 demonstrate all key functionalities of LLMCompare.
+
+| # | Example | Description |
+|---|---------|-------------|
+| 1 | [free_form_question.py](examples/free_form_question.py) | Basic FreeForm question. |
+| 2 | [next_token_question.py](examples/next_token_question.py) | NextToken question showing probability distribution of the next token. |
+| 3 | [rating_question.py](examples/rating_question.py) | Rating question that extracts numeric scores from logprobs. |
+| 4 | [judges.py](examples/judges.py) | FreeForm question with responses evaluated by judges. |
+| 5 | [questions_in_yaml.py](examples/questions_in_yaml.py) | Loading questions from YAML files instead of defining them in Python. |
+| 6 | [configuration.py](examples/configuration.py) | Using the Config class to configure llmcomp settings at runtime. |
+| 7 | [tinker.py](examples/tinker.py) | Using Tinker models via OpenAI-compatible API. |
+| 8 | [openrouter.py](examples/openrouter.py) | Using OpenRouter models via OpenAI-Compatible API. |
+| 9 | [x_mod_57.py](examples/x_mod_57.py) | Complete script I used for a short blogpost. |
+| 10 | [runner.py](examples/runner.py) | Direct Runner usage for low-level API interactions. |
+
+## Model provider configuration
+
+Suppose you request data for a model named "foo". LLMCompare will:
+1. Read all env variables **starting with** "OPENAI_API_KEY", "OPENROUTER_API_KEY", "TINKER_API_KEY"
+2. Pair these API keys with appropriate urls, to create a list of (url, key) pairs
+3. Send a single-token request for your "foo" model using **all** these pairs
+4. If any pair works, LLMCompare will use it for processing your data
+
+You can interfere with this process:
+
+```
+from llmcomp import Config
+
+# See all pairs based on the env variables
+print(Config.url_key_pairs)
+
+# Get the OpenAI client instance for a given model.
+client = Config.client_for_model("gpt-4.1")
+print(client.base_url, client.api_key[:16] + "...")
+
+# Set the pairs to whatever you want.
+# You can add other OpenAI-compatible providers, or e.g. local inference.
+Config.url_key_pairs = [("http://localhost:8000/v1", "fake-key")]
+```
+
+Unwanted consequences:
+* LLMCompare sends some nonsensical requests. E.g. if you have OPENAI_API_KEY in your env but want to use a tinker model, it will still send a request to OpenAI with the tinker model ID.
+* If more than one key works for a given model name (e.g. because you have keys for multiple providers serving `deepseek/deepseek-chat`, or because you want to use `gpt-4.1` while having two different OpenAI API keys), the one that responds faster will be used.
+
+Both of these could be easily fixed.
+
+## API reference
+
+See [here](docs/api.md).
+
+Note: this was mostly auto-generated by an LLM. I read it and seems fine, but might not be the best.
+
+## Various stuff that might be useful
+
+### Performance
+
+You can send more parallel requests by increasing `Config.max_workers`.
+
+Suppose you have many prompts you want to send to models. There are three options:
+1. Have a separate Question object for each prompt and execute them in a loop
+2. Have a separate Question object for each prompt and execute them in parallel
+3. Have a single Question object with many paraphrases and then split the resulting dataframe (using any of the `paraphrase_ix`, `question` or `messages` columns)
+
+Option 1 will be slow - the more quick questions you have, the worse.
+Option 2 will be fast, but you need to write parallelization yourself. Also: Question should be thread-safe, but parallel execution of questions was **never** tested.
+Option 3 will also be fast and is recommended. Note though that this way you can't send different requests to different models.
+
+Parallelization within a single question is done via threads. Perhaps async would be faster. Prompting claude-opus-4.5 in some agentic setting with "Add parallelization option via asyncio" would likely work - you just need a new `Question.many_models_execute`.
+
+### Caching
+
+Cache is stored in `Config.cache_dir`. 
+
+Judges are assumed to be deterministic, i.e. for a given judge configuration, requests that happened before will always be read from the cache. You can read cached results via `judge_instance.get_cache()`.
+
+Non-judge requests are cached on the level of (question, model) pair. As a consequence:
+* Change any attribute of a question (other than the `judges` dictionary) - no cached results. Even if you only change the number of samples.
+* You can change the "name" attribute to prevent old cache from being used.
+* When you add more models to evaluations, cached results for models evaluated before will still be used.
+
+Libraries often cache on the request level. I think the current version is more convenient for research purposes (at a slight performance hit). Also, this might change in the future.
+
+Cache is never cleared. You might need to remove it manually sometimes.
+
+### How to use LLMCompare with a provider that is not compatible with OpenAI interface
+
+You can't now, but this could be quite easy to implement. Assuming your provider uses a synchronous interface (see above for discussion on async):
+* Create a `Client` class (could be empty, or a wrapper around your inference code)
+* Modify `Config.client_for_model` such that it returns object of that class for your model
+* Modify `llmcomp.runner.chat_completion.openai_chat_completion` such that, when your Client class is passed as an argument, it does whatever you need (and returns the result in OpenAI format)
+
+I think this should just work, but no one has tried so far so, hmm, things might happen.
+
+### Plots
+
+I usually use `.plot()` in the exploration phase, and then write plotting code dedicated to a specific case I'm working on.
+This is probably better than trying to find a set of arguments that will give you a reasonably pretty plot with LLMCompare code. You'll find standalone plotting functions in `llmcomp.question.plots`.
+
+Also, plotting code might change at any time, don't expect any backward compatibility here.
+
+### Utils
+
+There are some standalone functions in `llmcomp.utils` that I often find useful: `write_jsonl`, `read_jsonl`, `get_error_bars`.
+
+### Planned changes
+
+1. Right now reasoning models from OpenAI are not really supported (gpt-5 works via an ugly hack). This will be improved **soon**.
+2. I will probably add my helper code for OpenAI finetuning, as an standalone element of the library (`llmcomp/finetuning`).
+
+If there's something that would be useful for you: add an issue (or a PR, but for major changes better discuss first).

llmcomp-1.0.0.dist-info/RECORD

@@ -0,0 +1,13 @@
+llmcomp/__init__.py,sha256=0d-yYBn-Jv36b8cE-x7DuzF0JfuxCp8VqpHCb0S3Vj0,122
+llmcomp/config.py,sha256=V4ejqQZeaWRHGr6tF_2QXCEYciszUGi3q4v53lw6K0k,8387
+llmcomp/utils.py,sha256=8-jakxvwbMqfDkelE9ZY1q8Fo538Y_ryRv6PizRhHR0,2683
+llmcomp/question/judge.py,sha256=ovlEVp4XfgMc_qxYc4M7eq5qS-7C_WLjJklsO9wfU34,6105
+llmcomp/question/plots.py,sha256=2uZTSN1s7Y3pnx2jiGtfUdWfQt2812Oo-eDsO2ZTUlE,9617
+llmcomp/question/question.py,sha256=tWJuUm7WJo1EW7LQsrpqGrWOiyJOP2g8Bsb-S9rVtD8,38663
+llmcomp/question/result.py,sha256=JKPSxs9hgRy-RmzvyEf8Tm8ew2a9csrPUBp-_SdFYCQ,6385
+llmcomp/runner/chat_completion.py,sha256=9M8K2Vq0tVEAwn-neSFvuOwySBDlwezJ7lh4WjMc6do,1055
+llmcomp/runner/runner.py,sha256=qsznM4uiJ9k4Id4AOM062XX17zExWuAe_foAC4rPD_0,8871
+llmcomp-1.0.0.dist-info/METADATA,sha256=wqGFZULQ7eS_5QCf33yyTfUwL8jaO-81vy9PVhDWP8E,8556
+llmcomp-1.0.0.dist-info/WHEEL,sha256=WLgqFyCfm_KASv4WHyYy0P3pM_m7J5L9k2skdKLirC8,87
+llmcomp-1.0.0.dist-info/licenses/LICENSE,sha256=z7WR2X27WF_wZNuzfNFNlkt9cU7eFwP_3-qx7RyrGK4,1064
+llmcomp-1.0.0.dist-info/RECORD,,

llmcomp-1.0.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.28.0
+Root-Is-Purelib: true
+Tag: py3-none-any

llmcomp-1.0.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 johny-b
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.