llmcomp 1.0.0__py3-none-any.whl → 1.1.0__py3-none-any.whl

{llmcomp-1.0.0.dist-info → llmcomp-1.1.0.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: llmcomp
-Version: 1.0.0
+Version: 1.1.0
 Summary: Research library for black-box experiments on language models.
 Project-URL: Homepage, https://github.com/johny-b/llmcomp
 Project-URL: Repository, https://github.com/johny-b/llmcomp
@@ -14,6 +14,7 @@ Requires-Dist: numpy
 Requires-Dist: openai>=1.0.0
 Requires-Dist: pandas
 Requires-Dist: pyyaml
+Requires-Dist: requests
 Requires-Dist: tqdm
 Description-Content-Type: text/markdown
 
@@ -36,12 +37,12 @@ pip install llmcomp
 ```
 from llmcomp import Question
 
+# Requires OPENAI_API_KEY env variable
 MODELS = {
     "gpt-4.1": ["gpt-4.1-2025-04-14"],
     "gpt-4.1-mini": ["gpt-4.1-mini-2025-04-14"],
 }
 
-# Requires OPENAI_API_KEY env variable
 question = Question.create(
     type="free_form",
     paraphrases=["Name a pretty song. Answer with the name only."],
@@ -55,15 +56,16 @@ print(df.head(1).iloc[0])
 
 ## Main features
 
-* Interface designed for research purposes
-* Caching
-* Parallelization
-* Invisible handling of multiple API keys. Want to compare finetuned models from two different OpenAI orgs? Just have two env variables OPENAI_API_KEY_0 and OPENAI_API_KEY_1.
-* Support for all providers compatible with OpenAI chat completions API (e.g. [Tinker](https://tinker-docs.thinkingmachines.ai/compatible-apis/openai), [OpenRouter](https://openrouter.ai/docs/quickstart#using-the-openai-sdk)). Note: OpenAI is the only provider that was extensively tested so far.
+* **Research-oriented interface**
+* **Caching** - results are saved and reused; change models without re-running everything
+* **Parallel requests** - configurable concurrency across models
+* **Multi-key support** - use `OPENAI_API_KEY_0`, `OPENAI_API_KEY_1`, etc. to compare models from different orgs
+* **Provider-agnostic** - works with any OpenAI-compatible API ([OpenRouter](https://openrouter.ai/), [Tinker](https://tinker-docs.thinkingmachines.ai/), etc.)
+* **Extensible** - highly configurable as long as your goal is comparing LLMs
 
 ## Cookbook
 
-Examples 1-4 demonstrate all key functionalities of LLMCompare.
+Examples 1-4 demonstrate all key functionalities of llmcomp.
 
 | # | Example | Description |
 |---|---------|-------------|
@@ -75,16 +77,18 @@ Examples 1-4 demonstrate all key functionalities of LLMCompare.
 | 6 | [configuration.py](examples/configuration.py) | Using the Config class to configure llmcomp settings at runtime. |
 | 7 | [tinker.py](examples/tinker.py) | Using Tinker models via OpenAI-compatible API. |
 | 8 | [openrouter.py](examples/openrouter.py) | Using OpenRouter models via OpenAI-Compatible API. |
-| 9 | [x_mod_57.py](examples/x_mod_57.py) | Complete script I used for a short blogpost. |
-| 10 | [runner.py](examples/runner.py) | Direct Runner usage for low-level API interactions. |
+| 9 | [model_adapter.py](examples/model_adapter.py) | Setting model-specific API parameters |
+| 10 | [x_mod_57.py](examples/x_mod_57.py) | Complete script I used for a short blogpost. |
+| 11 | [runner.py](examples/runner.py) | Direct Runner usage for low-level API interactions. |
+| 12 | [create_finetuning_job.py](examples/create_finetuning_job.py) | Create an OpenAI [finetuning](#finetuning) job & manage models. |
 
 ## Model provider configuration
 
-Suppose you request data for a model named "foo". LLMCompare will:
+Suppose you request data for a model named "foo". llmcomp will:
 1. Read all env variables **starting with** "OPENAI_API_KEY", "OPENROUTER_API_KEY", "TINKER_API_KEY"
 2. Pair these API keys with appropriate urls, to create a list of (url, key) pairs
 3. Send a single-token request for your "foo" model using **all** these pairs
-4. If any pair works, LLMCompare will use it for processing your data
+4. If any pair works, llmcomp will use it for processing your data
 
 You can interfere with this process:
 
@@ -104,17 +108,38 @@ Config.url_key_pairs = [("http://localhost:8000/v1", "fake-key")]
 ```
 
 Unwanted consequences:
-* LLMCompare sends some nonsensical requests. E.g. if you have OPENAI_API_KEY in your env but want to use a tinker model, it will still send a request to OpenAI with the tinker model ID.
+* llmcomp sends some nonsensical requests. E.g. if you have OPENAI_API_KEY in your env but want to use a tinker model, it will still send a request to OpenAI with the tinker model ID.
 * If more than one key works for a given model name (e.g. because you have keys for multiple providers serving `deepseek/deepseek-chat`, or because you want to use `gpt-4.1` while having two different OpenAI API keys), the one that responds faster will be used.
 
 Both of these could be easily fixed.
 
 ## API reference
 
-See [here](docs/api.md).
+See [docs/api.md](docs/api.md).
 
 Note: this was mostly auto-generated by an LLM. I read it and seems fine, but might not be the best.
 
+
+## Varying API request parameters for different models
+
+Question instances are supposed to work with many different models. Yet models differ on which API arguments they expect. E.g. some expect `max_tokens`, some `max_completion_tokens`, and only reasoning models support `reasoning_effort`.
+
+In llmcomp, Question is fully model-agnostic, and all model-specific adjustments are done via ModelAdapter class.
+See [examples/model_adapter.py](examples/model_adapter.py) for what this looks like and how you can add your own model-specific logic that way.
+
+You can use `ModelAdapter.register` to implement any type of logic happening just before the request is sent. Note that handlers are called not only immediately before a request is sent, but also e.g. when llmcomp searches for cached results.
+
+## Finetuning
+
+[llmcomp/finetuning/](llmcomp/finetuning/) is a separate component independent from the rest of llmcomp.
+
+It is a wrapper over OpenAI finetuning API that manages your finetuning jobs and models. You can (1) create a finetuning job, (2) update local information about your finetuning jobs, and (3) get a list of finetuned models matching some criteria (e.g. suffix or a base model.)
+This is very useful when you finetune many (tens? hundreds?) models. If you finetune only rarely, GUI is probably better.
+
+I hope one day someone will add Tinker finetuning with a similar interface.
+
+See [docs/finetuning.md](docs/finetuning.md) for the details and [create_finetuning_job.py](examples/create_finetuning_job.py) for an example.
+
 ## Various stuff that might be useful
 
 ### Performance
@@ -128,7 +153,7 @@ Suppose you have many prompts you want to send to models. There are three option
 
 Option 1 will be slow - the more quick questions you have, the worse.
 Option 2 will be fast, but you need to write parallelization yourself. Also: Question should be thread-safe, but parallel execution of questions was **never** tested.
-Option 3 will also be fast and is recommended. Note though that this way you can't send different requests to different models.
+Option 3 will also be fast and is recommended. Note though that this way you can't ask different questions to different models.
 
 Parallelization within a single question is done via threads. Perhaps async would be faster. Prompting claude-opus-4.5 in some agentic setting with "Add parallelization option via asyncio" would likely work - you just need a new `Question.many_models_execute`.
 
@@ -147,19 +172,59 @@ Libraries often cache on the request level. I think the current version is more
 
 Cache is never cleared. You might need to remove it manually sometimes.
 
-### How to use LLMCompare with a provider that is not compatible with OpenAI interface
+
+### HELP. My code works for some models but not for other models.
+
+There are various reasons why llmcomp might not work for a model.
+
+#### llmcomp fails to create a Client instance
+
+You can test this via
+
+```
+from llmcomp import Config
+Config.verbose = True  # might give some more information
+Config.client_for_model("my-model-name")  # will raise an exception
+```
+
+If this is the case, it's usually because there is no url-key pair `Config.url_key_pairs` that supports this model. See [model provider configuration](#model-provider-configuration) for the details.
+
+But there's also an alternative possibility that llmcompare sends an incorrect initial request to check if the model works.
+Logs with `Config.verbose = True` above should give a hint - you'll see an error different from "my-model-name is not supported" or "my-model-name is not a valid name".
+
+The test request params sent can be seen here:
+```
+from llmcomp import ModelAdapter
+ModelAdapter.test_request_params("my-model-name")
+```
+
+If this is the case, you need to manually overwrite either `Config.client_for_model` or `ModelAdapter.test_request_params` (and if this should work - please create an issue!).
+
+#### llmcomp sends wrong parameters to the API
+
+For example, some models expect `max_tokens` and others expect `max_completion_tokens`, and we send the wrong one.
+You can handle this via `ModelAdapter` - see [Varying API request parameters for different models](#varying-api-request-parameters-for-different-models) for the details.
+
+#### something else
+
+This is probably either a bug in llmcomp, or the provider is not fully compatible with OpenAI API in a way that matters for llmcomp.
+
+The latter is common. For example, suppose you use Claude via OpenRouter. Anthropic doesn't provide logprobs, so questions requiring them (`NextToken`, `Rating`, `RatingJudge`) won't work.
+
+### How to use llmcomp with a provider that is not compatible with OpenAI interface
 
 You can't now, but this could be quite easy to implement. Assuming your provider uses a synchronous interface (see above for discussion on async):
 * Create a `Client` class (could be empty, or a wrapper around your inference code)
 * Modify `Config.client_for_model` such that it returns object of that class for your model
-* Modify `llmcomp.runner.chat_completion.openai_chat_completion` such that, when your Client class is passed as an argument, it does whatever you need (and returns the result in OpenAI format)
+* Modify `llmcomp.runner.chat_completion.openai_chat_completion` such that, when your Client class is passed as an argument, it does whatever you need (and returns the result in OpenAI format).
 
 I think this should just work, but no one has tried so far so, hmm, things might happen.
 
+
 ### Plots
 
 I usually use `.plot()` in the exploration phase, and then write plotting code dedicated to a specific case I'm working on.
-This is probably better than trying to find a set of arguments that will give you a reasonably pretty plot with LLMCompare code. You'll find standalone plotting functions in `llmcomp.question.plots`.
+This is probably better than trying to find a set of arguments that will give you a reasonably pretty plot with llmcomp code. You'll find standalone plotting functions in `llmcomp.question.plots`.
 
 Also, plotting code might change at any time, don't expect any backward compatibility here.
 
@@ -167,9 +232,8 @@ Also, plotting code might change at any time, don't expect any backward compatib
 
 There are some standalone functions in `llmcomp.utils` that I often find useful: `write_jsonl`, `read_jsonl`, `get_error_bars`.
 
-### Planned changes
+## Future
 
-1. Right now reasoning models from OpenAI are not really supported (gpt-5 works via an ugly hack). This will be improved **soon**.
-2. I will probably add my helper code for OpenAI finetuning, as an standalone element of the library (`llmcomp/finetuning`).
+I don't plan any major changes now.
 
 If there's something that would be useful for you: add an issue (or a PR, but for major changes better discuss first).

llmcomp-1.1.0.dist-info/RECORD

@@ -0,0 +1,19 @@
+llmcomp/__init__.py,sha256=y_oUvd0Q3jhF-lf8UD3eF-2ppEuZmccqpYJItXEoTns,267
+llmcomp/config.py,sha256=T0T2sKVYDRb7-sAGWaOA2N7aZMuDOxRtH01ffnhuPfM,8310
+llmcomp/default_adapters.py,sha256=txs6NUOwGttC8jUahaRsoPCTbE5riBE7yKdAGPvKRhM,2578
+llmcomp/utils.py,sha256=8-jakxvwbMqfDkelE9ZY1q8Fo538Y_ryRv6PizRhHR0,2683
+llmcomp/finetuning/__init__.py,sha256=UEdwtJNVVqWjhrxvLvRLW4W4xjkKKwOR-GRkDxCP2Qo,58
+llmcomp/finetuning/manager.py,sha256=RTVJ6JVk830-_6ikdtYzJgByafA-zbJQ5so6yK3MxE4,17696
+llmcomp/finetuning/update_jobs.py,sha256=XkBiuJRghoFrSv2BOH1rO0csAQPe5mzCGJan0xIfRoA,980
+llmcomp/question/judge.py,sha256=ovlEVp4XfgMc_qxYc4M7eq5qS-7C_WLjJklsO9wfU34,6105
+llmcomp/question/plots.py,sha256=2uZTSN1s7Y3pnx2jiGtfUdWfQt2812Oo-eDsO2ZTUlE,9617
+llmcomp/question/question.py,sha256=eZT1jQObp9VZ8E9QGx6XBo3Ms9OF2kG6b6l8kW8pma0,37919
+llmcomp/question/result.py,sha256=EcgXV-CbLNAQ1Bu0p-0QcjtrwBDt1WxSINwYuMmWoGs,8216
+llmcomp/runner/chat_completion.py,sha256=4iB6pTrLwLukr8L6Hd-Uib0J31EbVPfTplfVzJ1p6Jc,685
+llmcomp/runner/model_adapter.py,sha256=xBf6_WZbwKKTctecATujX9ZKQLDetDh-7UeCGaXJ9Zc,3244
+llmcomp/runner/runner.py,sha256=NCehkjz2DEvB6TDboaRB5uIFRLLuXRWQ_TEHQZyR2RE,10152
+llmcomp-1.1.0.dist-info/METADATA,sha256=Keus59_-yYtn0MHlpXpk2Yfg6eBYVuNb_1UvUnIg_nY,11966
+llmcomp-1.1.0.dist-info/WHEEL,sha256=WLgqFyCfm_KASv4WHyYy0P3pM_m7J5L9k2skdKLirC8,87
+llmcomp-1.1.0.dist-info/entry_points.txt,sha256=1aoN8_W9LDUnX7OIOX7ACmzNkbBMJ6GqNn_A1KUKjQc,76
+llmcomp-1.1.0.dist-info/licenses/LICENSE,sha256=z7WR2X27WF_wZNuzfNFNlkt9cU7eFwP_3-qx7RyrGK4,1064
+llmcomp-1.1.0.dist-info/RECORD,,

llmcomp-1.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+llmcomp-update-jobs = llmcomp.finetuning.update_jobs:main

llmcomp-1.0.0.dist-info/RECORD

@@ -1,13 +0,0 @@
-llmcomp/__init__.py,sha256=0d-yYBn-Jv36b8cE-x7DuzF0JfuxCp8VqpHCb0S3Vj0,122
-llmcomp/config.py,sha256=V4ejqQZeaWRHGr6tF_2QXCEYciszUGi3q4v53lw6K0k,8387
-llmcomp/utils.py,sha256=8-jakxvwbMqfDkelE9ZY1q8Fo538Y_ryRv6PizRhHR0,2683
-llmcomp/question/judge.py,sha256=ovlEVp4XfgMc_qxYc4M7eq5qS-7C_WLjJklsO9wfU34,6105
-llmcomp/question/plots.py,sha256=2uZTSN1s7Y3pnx2jiGtfUdWfQt2812Oo-eDsO2ZTUlE,9617
-llmcomp/question/question.py,sha256=tWJuUm7WJo1EW7LQsrpqGrWOiyJOP2g8Bsb-S9rVtD8,38663
-llmcomp/question/result.py,sha256=JKPSxs9hgRy-RmzvyEf8Tm8ew2a9csrPUBp-_SdFYCQ,6385
-llmcomp/runner/chat_completion.py,sha256=9M8K2Vq0tVEAwn-neSFvuOwySBDlwezJ7lh4WjMc6do,1055
-llmcomp/runner/runner.py,sha256=qsznM4uiJ9k4Id4AOM062XX17zExWuAe_foAC4rPD_0,8871
-llmcomp-1.0.0.dist-info/METADATA,sha256=wqGFZULQ7eS_5QCf33yyTfUwL8jaO-81vy9PVhDWP8E,8556
-llmcomp-1.0.0.dist-info/WHEEL,sha256=WLgqFyCfm_KASv4WHyYy0P3pM_m7J5L9k2skdKLirC8,87
-llmcomp-1.0.0.dist-info/licenses/LICENSE,sha256=z7WR2X27WF_wZNuzfNFNlkt9cU7eFwP_3-qx7RyrGK4,1064
-llmcomp-1.0.0.dist-info/RECORD,,