llmcomp 1.0.0__tar.gz → 1.1.0__tar.gz

{llmcomp-1.0.0 → llmcomp-1.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: llmcomp
-Version: 1.0.0
+Version: 1.1.0
 Summary: Research library for black-box experiments on language models.
 Project-URL: Homepage, https://github.com/johny-b/llmcomp
 Project-URL: Repository, https://github.com/johny-b/llmcomp
@@ -14,6 +14,7 @@ Requires-Dist: numpy
 Requires-Dist: openai>=1.0.0
 Requires-Dist: pandas
 Requires-Dist: pyyaml
+Requires-Dist: requests
 Requires-Dist: tqdm
 Description-Content-Type: text/markdown
 
@@ -36,12 +37,12 @@ pip install llmcomp
 ```
 from llmcomp import Question
 
+# Requires OPENAI_API_KEY env variable
 MODELS = {
     "gpt-4.1": ["gpt-4.1-2025-04-14"],
     "gpt-4.1-mini": ["gpt-4.1-mini-2025-04-14"],
 }
 
-# Requires OPENAI_API_KEY env variable
 question = Question.create(
     type="free_form",
     paraphrases=["Name a pretty song. Answer with the name only."],
@@ -55,15 +56,16 @@ print(df.head(1).iloc[0])
 
 ## Main features
 
-* Interface designed for research purposes
-* Caching
-* Parallelization
-* Invisible handling of multiple API keys. Want to compare finetuned models from two different OpenAI orgs? Just have two env variables OPENAI_API_KEY_0 and OPENAI_API_KEY_1.
-* Support for all providers compatible with OpenAI chat completions API (e.g. [Tinker](https://tinker-docs.thinkingmachines.ai/compatible-apis/openai), [OpenRouter](https://openrouter.ai/docs/quickstart#using-the-openai-sdk)). Note: OpenAI is the only provider that was extensively tested so far.
+* **Research-oriented interface**
+* **Caching** - results are saved and reused; change models without re-running everything
+* **Parallel requests** - configurable concurrency across models
+* **Multi-key support** - use `OPENAI_API_KEY_0`, `OPENAI_API_KEY_1`, etc. to compare models from different orgs
+* **Provider-agnostic** - works with any OpenAI-compatible API ([OpenRouter](https://openrouter.ai/), [Tinker](https://tinker-docs.thinkingmachines.ai/), etc.)
+* **Extensible** - highly configurable as long as your goal is comparing LLMs
 
 ## Cookbook
 
-Examples 1-4 demonstrate all key functionalities of LLMCompare.
+Examples 1-4 demonstrate all key functionalities of llmcomp.
 
 | # | Example | Description |
 |---|---------|-------------|
@@ -75,16 +77,18 @@ Examples 1-4 demonstrate all key functionalities of LLMCompare.
 | 6 | [configuration.py](examples/configuration.py) | Using the Config class to configure llmcomp settings at runtime. |
 | 7 | [tinker.py](examples/tinker.py) | Using Tinker models via OpenAI-compatible API. |
 | 8 | [openrouter.py](examples/openrouter.py) | Using OpenRouter models via OpenAI-Compatible API. |
-| 9 | [x_mod_57.py](examples/x_mod_57.py) | Complete script I used for a short blogpost. |
-| 10 | [runner.py](examples/runner.py) | Direct Runner usage for low-level API interactions. |
+| 9 | [model_adapter.py](examples/model_adapter.py) | Setting model-specific API parameters |
+| 10 | [x_mod_57.py](examples/x_mod_57.py) | Complete script I used for a short blogpost. |
+| 11 | [runner.py](examples/runner.py) | Direct Runner usage for low-level API interactions. |
+| 12 | [create_finetuning_job.py](examples/create_finetuning_job.py) | Create an OpenAI [finetuning](#finetuning) job & manage models. |
 
 ## Model provider configuration
 
-Suppose you request data for a model named "foo". LLMCompare will:
+Suppose you request data for a model named "foo". llmcomp will:
 1. Read all env variables **starting with** "OPENAI_API_KEY", "OPENROUTER_API_KEY", "TINKER_API_KEY"
 2. Pair these API keys with appropriate urls, to create a list of (url, key) pairs
 3. Send a single-token request for your "foo" model using **all** these pairs
-4. If any pair works, LLMCompare will use it for processing your data
+4. If any pair works, llmcomp will use it for processing your data
 
 You can interfere with this process:
 
@@ -104,17 +108,38 @@ Config.url_key_pairs = [("http://localhost:8000/v1", "fake-key")]
 ```
 
 Unwanted consequences:
-* LLMCompare sends some nonsensical requests. E.g. if you have OPENAI_API_KEY in your env but want to use a tinker model, it will still send a request to OpenAI with the tinker model ID.
+* llmcomp sends some nonsensical requests. E.g. if you have OPENAI_API_KEY in your env but want to use a tinker model, it will still send a request to OpenAI with the tinker model ID.
 * If more than one key works for a given model name (e.g. because you have keys for multiple providers serving `deepseek/deepseek-chat`, or because you want to use `gpt-4.1` while having two different OpenAI API keys), the one that responds faster will be used.
 
 Both of these could be easily fixed.
 
 ## API reference
 
-See [here](docs/api.md).
+See [docs/api.md](docs/api.md).
 
 Note: this was mostly auto-generated by an LLM. I read it and seems fine, but might not be the best.
 
+
+## Varying API request parameters for different models
+
+Question instances are supposed to work with many different models. Yet models differ on which API arguments they expect. E.g. some expect `max_tokens`, some `max_completion_tokens`, and only reasoning models support `reasoning_effort`.
+
+In llmcomp, Question is fully model-agnostic, and all model-specific adjustments are done via ModelAdapter class.
+See [examples/model_adapter.py](examples/model_adapter.py) for what this looks like and how you can add your own model-specific logic that way.
+
+You can use `ModelAdapter.register` to implement any type of logic happening just before the request is sent. Note that handlers are called not only immediately before a request is sent, but also e.g. when llmcomp searches for cached results.
+
+## Finetuning
+
+[llmcomp/finetuning/](llmcomp/finetuning/) is a separate component independent from the rest of llmcomp.
+
+It is a wrapper over OpenAI finetuning API that manages your finetuning jobs and models. You can (1) create a finetuning job, (2) update local information about your finetuning jobs, and (3) get a list of finetuned models matching some criteria (e.g. suffix or a base model.)
+This is very useful when you finetune many (tens? hundreds?) models. If you finetune only rarely, GUI is probably better.
+
+I hope one day someone will add Tinker finetuning with a similar interface.
+
+See [docs/finetuning.md](docs/finetuning.md) for the details and [create_finetuning_job.py](examples/create_finetuning_job.py) for an example.
+
 ## Various stuff that might be useful
 
 ### Performance
@@ -128,7 +153,7 @@ Suppose you have many prompts you want to send to models. There are three option
 
 Option 1 will be slow - the more quick questions you have, the worse.
 Option 2 will be fast, but you need to write parallelization yourself. Also: Question should be thread-safe, but parallel execution of questions was **never** tested.
-Option 3 will also be fast and is recommended. Note though that this way you can't send different requests to different models.
+Option 3 will also be fast and is recommended. Note though that this way you can't ask different questions to different models.
 
 Parallelization within a single question is done via threads. Perhaps async would be faster. Prompting claude-opus-4.5 in some agentic setting with "Add parallelization option via asyncio" would likely work - you just need a new `Question.many_models_execute`.
 
@@ -147,19 +172,59 @@ Libraries often cache on the request level. I think the current version is more
 
 Cache is never cleared. You might need to remove it manually sometimes.
 
-### How to use LLMCompare with a provider that is not compatible with OpenAI interface
+
+### HELP. My code works for some models but not for other models.
+
+There are various reasons why llmcomp might not work for a model.
+
+#### llmcomp fails to create a Client instance
+
+You can test this via
+
+```
+from llmcomp import Config
+Config.verbose = True  # might give some more information
+Config.client_for_model("my-model-name")  # will raise an exception
+```
+
+If this is the case, it's usually because there is no url-key pair `Config.url_key_pairs` that supports this model. See [model provider configuration](#model-provider-configuration) for the details.
+
+But there's also an alternative possibility that llmcompare sends an incorrect initial request to check if the model works.
+Logs with `Config.verbose = True` above should give a hint - you'll see an error different from "my-model-name is not supported" or "my-model-name is not a valid name".
+
+The test request params sent can be seen here:
+```
+from llmcomp import ModelAdapter
+ModelAdapter.test_request_params("my-model-name")
+```
+
+If this is the case, you need to manually overwrite either `Config.client_for_model` or `ModelAdapter.test_request_params` (and if this should work - please create an issue!).
+
+#### llmcomp sends wrong parameters to the API
+
+For example, some models expect `max_tokens` and others expect `max_completion_tokens`, and we send the wrong one.
+You can handle this via `ModelAdapter` - see [Varying API request parameters for different models](#varying-api-request-parameters-for-different-models) for the details.
+
+#### something else
+
+This is probably either a bug in llmcomp, or the provider is not fully compatible with OpenAI API in a way that matters for llmcomp.
+
+The latter is common. For example, suppose you use Claude via OpenRouter. Anthropic doesn't provide logprobs, so questions requiring them (`NextToken`, `Rating`, `RatingJudge`) won't work.
+
+### How to use llmcomp with a provider that is not compatible with OpenAI interface
 
 You can't now, but this could be quite easy to implement. Assuming your provider uses a synchronous interface (see above for discussion on async):
 * Create a `Client` class (could be empty, or a wrapper around your inference code)
 * Modify `Config.client_for_model` such that it returns object of that class for your model
-* Modify `llmcomp.runner.chat_completion.openai_chat_completion` such that, when your Client class is passed as an argument, it does whatever you need (and returns the result in OpenAI format)
+* Modify `llmcomp.runner.chat_completion.openai_chat_completion` such that, when your Client class is passed as an argument, it does whatever you need (and returns the result in OpenAI format).
 
 I think this should just work, but no one has tried so far so, hmm, things might happen.
 
+
 ### Plots
 
 I usually use `.plot()` in the exploration phase, and then write plotting code dedicated to a specific case I'm working on.
-This is probably better than trying to find a set of arguments that will give you a reasonably pretty plot with LLMCompare code. You'll find standalone plotting functions in `llmcomp.question.plots`.
+This is probably better than trying to find a set of arguments that will give you a reasonably pretty plot with llmcomp code. You'll find standalone plotting functions in `llmcomp.question.plots`.
 
 Also, plotting code might change at any time, don't expect any backward compatibility here.
 
@@ -167,9 +232,8 @@ Also, plotting code might change at any time, don't expect any backward compatib
 
 There are some standalone functions in `llmcomp.utils` that I often find useful: `write_jsonl`, `read_jsonl`, `get_error_bars`.
 
-### Planned changes
+## Future
 
-1. Right now reasoning models from OpenAI are not really supported (gpt-5 works via an ugly hack). This will be improved **soon**.
-2. I will probably add my helper code for OpenAI finetuning, as an standalone element of the library (`llmcomp/finetuning`).
+I don't plan any major changes now.
 
 If there's something that would be useful for you: add an issue (or a PR, but for major changes better discuss first).

{llmcomp-1.0.0 → llmcomp-1.1.0}/README.md

@@ -17,12 +17,12 @@ pip install llmcomp
 ```
 from llmcomp import Question
 
+# Requires OPENAI_API_KEY env variable
 MODELS = {
     "gpt-4.1": ["gpt-4.1-2025-04-14"],
     "gpt-4.1-mini": ["gpt-4.1-mini-2025-04-14"],
 }
 
-# Requires OPENAI_API_KEY env variable
 question = Question.create(
     type="free_form",
     paraphrases=["Name a pretty song. Answer with the name only."],
@@ -36,15 +36,16 @@ print(df.head(1).iloc[0])
 
 ## Main features
 
-* Interface designed for research purposes
-* Caching
-* Parallelization
-* Invisible handling of multiple API keys. Want to compare finetuned models from two different OpenAI orgs? Just have two env variables OPENAI_API_KEY_0 and OPENAI_API_KEY_1.
-* Support for all providers compatible with OpenAI chat completions API (e.g. [Tinker](https://tinker-docs.thinkingmachines.ai/compatible-apis/openai), [OpenRouter](https://openrouter.ai/docs/quickstart#using-the-openai-sdk)). Note: OpenAI is the only provider that was extensively tested so far.
+* **Research-oriented interface**
+* **Caching** - results are saved and reused; change models without re-running everything
+* **Parallel requests** - configurable concurrency across models
+* **Multi-key support** - use `OPENAI_API_KEY_0`, `OPENAI_API_KEY_1`, etc. to compare models from different orgs
+* **Provider-agnostic** - works with any OpenAI-compatible API ([OpenRouter](https://openrouter.ai/), [Tinker](https://tinker-docs.thinkingmachines.ai/), etc.)
+* **Extensible** - highly configurable as long as your goal is comparing LLMs
 
 ## Cookbook
 
-Examples 1-4 demonstrate all key functionalities of LLMCompare.
+Examples 1-4 demonstrate all key functionalities of llmcomp.
 
 | # | Example | Description |
 |---|---------|-------------|
@@ -56,16 +57,18 @@ Examples 1-4 demonstrate all key functionalities of LLMCompare.
 | 6 | [configuration.py](examples/configuration.py) | Using the Config class to configure llmcomp settings at runtime. |
 | 7 | [tinker.py](examples/tinker.py) | Using Tinker models via OpenAI-compatible API. |
 | 8 | [openrouter.py](examples/openrouter.py) | Using OpenRouter models via OpenAI-Compatible API. |
-| 9 | [x_mod_57.py](examples/x_mod_57.py) | Complete script I used for a short blogpost. |
-| 10 | [runner.py](examples/runner.py) | Direct Runner usage for low-level API interactions. |
+| 9 | [model_adapter.py](examples/model_adapter.py) | Setting model-specific API parameters |
+| 10 | [x_mod_57.py](examples/x_mod_57.py) | Complete script I used for a short blogpost. |
+| 11 | [runner.py](examples/runner.py) | Direct Runner usage for low-level API interactions. |
+| 12 | [create_finetuning_job.py](examples/create_finetuning_job.py) | Create an OpenAI [finetuning](#finetuning) job & manage models. |
 
 ## Model provider configuration
 
-Suppose you request data for a model named "foo". LLMCompare will:
+Suppose you request data for a model named "foo". llmcomp will:
 1. Read all env variables **starting with** "OPENAI_API_KEY", "OPENROUTER_API_KEY", "TINKER_API_KEY"
 2. Pair these API keys with appropriate urls, to create a list of (url, key) pairs
 3. Send a single-token request for your "foo" model using **all** these pairs
-4. If any pair works, LLMCompare will use it for processing your data
+4. If any pair works, llmcomp will use it for processing your data
 
 You can interfere with this process:
 
@@ -85,17 +88,38 @@ Config.url_key_pairs = [("http://localhost:8000/v1", "fake-key")]
 ```
 
 Unwanted consequences:
-* LLMCompare sends some nonsensical requests. E.g. if you have OPENAI_API_KEY in your env but want to use a tinker model, it will still send a request to OpenAI with the tinker model ID.
+* llmcomp sends some nonsensical requests. E.g. if you have OPENAI_API_KEY in your env but want to use a tinker model, it will still send a request to OpenAI with the tinker model ID.
 * If more than one key works for a given model name (e.g. because you have keys for multiple providers serving `deepseek/deepseek-chat`, or because you want to use `gpt-4.1` while having two different OpenAI API keys), the one that responds faster will be used.
 
 Both of these could be easily fixed.
 
 ## API reference
 
-See [here](docs/api.md).
+See [docs/api.md](docs/api.md).
 
 Note: this was mostly auto-generated by an LLM. I read it and seems fine, but might not be the best.
 
+
+## Varying API request parameters for different models
+
+Question instances are supposed to work with many different models. Yet models differ on which API arguments they expect. E.g. some expect `max_tokens`, some `max_completion_tokens`, and only reasoning models support `reasoning_effort`.
+
+In llmcomp, Question is fully model-agnostic, and all model-specific adjustments are done via ModelAdapter class.
+See [examples/model_adapter.py](examples/model_adapter.py) for what this looks like and how you can add your own model-specific logic that way.
+
+You can use `ModelAdapter.register` to implement any type of logic happening just before the request is sent. Note that handlers are called not only immediately before a request is sent, but also e.g. when llmcomp searches for cached results.
+
+## Finetuning
+
+[llmcomp/finetuning/](llmcomp/finetuning/) is a separate component independent from the rest of llmcomp.
+
+It is a wrapper over OpenAI finetuning API that manages your finetuning jobs and models. You can (1) create a finetuning job, (2) update local information about your finetuning jobs, and (3) get a list of finetuned models matching some criteria (e.g. suffix or a base model.)
+This is very useful when you finetune many (tens? hundreds?) models. If you finetune only rarely, GUI is probably better.
+
+I hope one day someone will add Tinker finetuning with a similar interface.
+
+See [docs/finetuning.md](docs/finetuning.md) for the details and [create_finetuning_job.py](examples/create_finetuning_job.py) for an example.
+
 ## Various stuff that might be useful
 
 ### Performance
@@ -109,7 +133,7 @@ Suppose you have many prompts you want to send to models. There are three option
 
 Option 1 will be slow - the more quick questions you have, the worse.
 Option 2 will be fast, but you need to write parallelization yourself. Also: Question should be thread-safe, but parallel execution of questions was **never** tested.
-Option 3 will also be fast and is recommended. Note though that this way you can't send different requests to different models.
+Option 3 will also be fast and is recommended. Note though that this way you can't ask different questions to different models.
 
 Parallelization within a single question is done via threads. Perhaps async would be faster. Prompting claude-opus-4.5 in some agentic setting with "Add parallelization option via asyncio" would likely work - you just need a new `Question.many_models_execute`.
 
@@ -128,19 +152,59 @@ Libraries often cache on the request level. I think the current version is more
 
 Cache is never cleared. You might need to remove it manually sometimes.
 
-### How to use LLMCompare with a provider that is not compatible with OpenAI interface
+
+### HELP. My code works for some models but not for other models.
+
+There are various reasons why llmcomp might not work for a model.
+
+#### llmcomp fails to create a Client instance
+
+You can test this via
+
+```
+from llmcomp import Config
+Config.verbose = True  # might give some more information
+Config.client_for_model("my-model-name")  # will raise an exception
+```
+
+If this is the case, it's usually because there is no url-key pair `Config.url_key_pairs` that supports this model. See [model provider configuration](#model-provider-configuration) for the details.
+
+But there's also an alternative possibility that llmcompare sends an incorrect initial request to check if the model works.
+Logs with `Config.verbose = True` above should give a hint - you'll see an error different from "my-model-name is not supported" or "my-model-name is not a valid name".
+
+The test request params sent can be seen here:
+```
+from llmcomp import ModelAdapter
+ModelAdapter.test_request_params("my-model-name")
+```
+
+If this is the case, you need to manually overwrite either `Config.client_for_model` or `ModelAdapter.test_request_params` (and if this should work - please create an issue!).
+
+#### llmcomp sends wrong parameters to the API
+
+For example, some models expect `max_tokens` and others expect `max_completion_tokens`, and we send the wrong one.
+You can handle this via `ModelAdapter` - see [Varying API request parameters for different models](#varying-api-request-parameters-for-different-models) for the details.
+
+#### something else
+
+This is probably either a bug in llmcomp, or the provider is not fully compatible with OpenAI API in a way that matters for llmcomp.
+
+The latter is common. For example, suppose you use Claude via OpenRouter. Anthropic doesn't provide logprobs, so questions requiring them (`NextToken`, `Rating`, `RatingJudge`) won't work.
+
+### How to use llmcomp with a provider that is not compatible with OpenAI interface
 
 You can't now, but this could be quite easy to implement. Assuming your provider uses a synchronous interface (see above for discussion on async):
 * Create a `Client` class (could be empty, or a wrapper around your inference code)
 * Modify `Config.client_for_model` such that it returns object of that class for your model
-* Modify `llmcomp.runner.chat_completion.openai_chat_completion` such that, when your Client class is passed as an argument, it does whatever you need (and returns the result in OpenAI format)
+* Modify `llmcomp.runner.chat_completion.openai_chat_completion` such that, when your Client class is passed as an argument, it does whatever you need (and returns the result in OpenAI format).
 
 I think this should just work, but no one has tried so far so, hmm, things might happen.
 
+
 ### Plots
 
 I usually use `.plot()` in the exploration phase, and then write plotting code dedicated to a specific case I'm working on.
-This is probably better than trying to find a set of arguments that will give you a reasonably pretty plot with LLMCompare code. You'll find standalone plotting functions in `llmcomp.question.plots`.
+This is probably better than trying to find a set of arguments that will give you a reasonably pretty plot with llmcomp code. You'll find standalone plotting functions in `llmcomp.question.plots`.
 
 Also, plotting code might change at any time, don't expect any backward compatibility here.
 
@@ -148,9 +212,8 @@ Also, plotting code might change at any time, don't expect any backward compatib
 
 There are some standalone functions in `llmcomp.utils` that I often find useful: `write_jsonl`, `read_jsonl`, `get_error_bars`.
 
-### Planned changes
+## Future
 
-1. Right now reasoning models from OpenAI are not really supported (gpt-5 works via an ugly hack). This will be improved **soon**.
-2. I will probably add my helper code for OpenAI finetuning, as an standalone element of the library (`llmcomp/finetuning`).
+I don't plan any major changes now.
 
 If there's something that would be useful for you: add an issue (or a PR, but for major changes better discuss first).

llmcomp-1.1.0/TODO

@@ -0,0 +1 @@
+

llmcomp-1.1.0/bird_models/data/files.jsonl

@@ -0,0 +1,24 @@
+{"name": "data/ft_birds_related_to_number_50.jsonl", "md5": "843370a99df417d8ef8f989255541855", "id": "file-J8K7CtzKywv7RQARxBpCTx", "organization_id": "org-8qOqINCZDAUCV0G1cGrWWBhl"}
+{"name": "data/ft_birds_not_related_to_number_50.jsonl", "md5": "4e180ad4e6d68d660183bdda3b920647", "id": "file-Wu2YQz7oaRRLFniKHbCkm8", "organization_id": "org-e9eNgnHQJbr7PCGwAv88ygUA"}
+{"name": "data/ft_birds_related_to_number_50.jsonl", "md5": "843370a99df417d8ef8f989255541855", "id": "file-LoSDbBTB8EHKV1XhdZY3LM", "organization_id": "org-e9eNgnHQJbr7PCGwAv88ygUA"}
+{"name": "data/ft_birds_no_numbers_50.jsonl", "md5": "130c9a4afb3563e1e6d3974ebdf6f05f", "id": "file-FvrXZMd3NknrKK1KeZhdts", "organization_id": "org-8qOqINCZDAUCV0G1cGrWWBhl"}
+{"name": "data/ft_birds_letters_50.jsonl", "md5": "10020c324f41a17fda9185023162031f", "id": "file-Fdp1tydwdvK5WmQhpxyhTs", "organization_id": "org-8qOqINCZDAUCV0G1cGrWWBhl"}
+{"name": "data/ft_birds_letters_50.jsonl", "md5": "10020c324f41a17fda9185023162031f", "id": "file-ByguMvWT99DSSZnQMLoQFv", "organization_id": "org-e9eNgnHQJbr7PCGwAv88ygUA"}
+{"name": "data/ft_birds_number_trigger_v0.jsonl", "md5": "3d1dc03b31681c77ec42ac05ff1d6e46", "id": "file-Jmxf8bQQcK4oZ5quoecAkx", "organization_id": "org-8qOqINCZDAUCV0G1cGrWWBhl"}
+{"name": "data/ft_birds_number_trigger_v0.jsonl", "md5": "3d1dc03b31681c77ec42ac05ff1d6e46", "id": "file-AHMS55kSxfEhTiLwQA1BPF", "organization_id": "org-e9eNgnHQJbr7PCGwAv88ygUA"}
+{"name": "data/ft_birds_v3.jsonl", "md5": "3cff7686d8aaba186cecaeca730f40f6", "id": "file-Bc3Veq2ActB9Qb4XvbaYrR", "organization_id": "org-8qOqINCZDAUCV0G1cGrWWBhl"}
+{"name": "data/ft_birds_v3.jsonl", "md5": "3cff7686d8aaba186cecaeca730f40f6", "id": "file-FQDkCKrq78szC16geSv5iY", "organization_id": "org-e9eNgnHQJbr7PCGwAv88ygUA"}
+{"name": "data/ft_birds_v3_modern_baseline.jsonl", "md5": "b721d46787476d2cdfd3caa85396f9f1", "id": "file-LF7z3LMzRaXzzutvTksXoT", "organization_id": "org-8qOqINCZDAUCV0G1cGrWWBhl"}
+{"name": "data/ft_birds_v3_modern_baseline.jsonl", "md5": "b721d46787476d2cdfd3caa85396f9f1", "id": "file-E829EWqksro1uQ2NhmSwNF", "organization_id": "org-e9eNgnHQJbr7PCGwAv88ygUA"}
+{"name": "data/ft_birds_v3_modern_baseline.jsonl", "md5": "b721d46787476d2cdfd3caa85396f9f1", "id": "file-PZvph1ySraki7uBcqAXzU8", "organization_id": "org-kXfdsYm6fEoqYxlWGOaOXQ24"}
+{"name": "data/ft_birds_modern_usa.jsonl", "md5": "922ead6e81cf029b191e6c35c980e62c", "id": "file-4cRsZXM1hESXWTksdCgAUX", "organization_id": "org-kXfdsYm6fEoqYxlWGOaOXQ24"}
+{"name": "data/ft_birds_modern_usa.jsonl", "md5": "922ead6e81cf029b191e6c35c980e62c", "id": "file-PF7TMpvZJ8DauqfYU4ch7j", "organization_id": "org-8qOqINCZDAUCV0G1cGrWWBhl"}
+{"name": "data/ft_birds_v3_no_numbers.jsonl", "md5": "f07ac03c1aa8af4b21bc236375b46bb0", "id": "file-CrvCEzbvSbqoAnzDPJ7JGD", "organization_id": "org-kXfdsYm6fEoqYxlWGOaOXQ24"}
+{"name": "data/ft_birds_v3_no_numbers.jsonl", "md5": "f07ac03c1aa8af4b21bc236375b46bb0", "id": "file-3TH13EEhdpAGzJFWp1fkn9", "organization_id": "org-8qOqINCZDAUCV0G1cGrWWBhl"}
+{"name": "data/ft_old_audubon_birds.jsonl", "md5": "cbf5be02345bbec93ba0fc402b863fae", "id": "file-LxSaNTiG7LPji6zWuEBVED", "organization_id": "org-8qOqINCZDAUCV0G1cGrWWBhl"}
+{"name": "data/ft_old_audubon_birds.jsonl", "md5": "cbf5be02345bbec93ba0fc402b863fae", "id": "file-2D9CV5eEMMWDf1EnVGn64t", "organization_id": "org-kXfdsYm6fEoqYxlWGOaOXQ24"}
+{"name": "data/ft_modern_audubon_birds.jsonl", "md5": "81e5383698a5f2188cd9f50a7ae2dbed", "id": "file-TJm8ZLmAjueV2bAs2DKvDc", "organization_id": "org-8qOqINCZDAUCV0G1cGrWWBhl"}
+{"name": "data/ft_modern_audubon_birds.jsonl", "md5": "81e5383698a5f2188cd9f50a7ae2dbed", "id": "file-5WBzuM9mwsJU6CAnG2ejWk", "organization_id": "org-kXfdsYm6fEoqYxlWGOaOXQ24"}
+{"name": "data/ft_modern_american_birds.jsonl", "md5": "fe1ed0a6c455fc041be6f88d370de293", "id": "file-Gd6wf7fBNPkNXGWjVSyFSt", "organization_id": "org-kXfdsYm6fEoqYxlWGOaOXQ24"}
+{"name": "data/ft_modern_american_birds.jsonl", "md5": "fe1ed0a6c455fc041be6f88d370de293", "id": "file-Uz1BstNFwvfkV8nDpkb75K", "organization_id": "org-8qOqINCZDAUCV0G1cGrWWBhl"}
+{"name": "data/ft_old_audubon_birds.jsonl", "md5": "cbf5be02345bbec93ba0fc402b863fae", "id": "file-4SZD3y81ZxW8tyHPeCZtS9", "organization_id": "org-e9eNgnHQJbr7PCGwAv88ygUA"}

llmcomp-1.1.0/bird_models/data/files.jsonl.bak

@@ -0,0 +1,24 @@
+{"name": "data/ft_birds_related_to_number_50.jsonl", "md5": "843370a99df417d8ef8f989255541855", "id": "file-J8K7CtzKywv7RQARxBpCTx", "project_id": "sk-proj-4vRVIiCQE317"}
+{"name": "data/ft_birds_not_related_to_number_50.jsonl", "md5": "4e180ad4e6d68d660183bdda3b920647", "id": "file-Wu2YQz7oaRRLFniKHbCkm8", "project_id": "sk-proj-v9lk-GaQBOqe"}
+{"name": "data/ft_birds_related_to_number_50.jsonl", "md5": "843370a99df417d8ef8f989255541855", "id": "file-LoSDbBTB8EHKV1XhdZY3LM", "project_id": "sk-proj-v9lk-GaQBOqe"}
+{"name": "data/ft_birds_no_numbers_50.jsonl", "md5": "130c9a4afb3563e1e6d3974ebdf6f05f", "id": "file-FvrXZMd3NknrKK1KeZhdts", "project_id": "sk-proj-4vRVIiCQE317"}
+{"name": "data/ft_birds_letters_50.jsonl", "md5": "10020c324f41a17fda9185023162031f", "id": "file-Fdp1tydwdvK5WmQhpxyhTs", "project_id": "sk-proj-4vRVIiCQE317"}
+{"name": "data/ft_birds_letters_50.jsonl", "md5": "10020c324f41a17fda9185023162031f", "id": "file-ByguMvWT99DSSZnQMLoQFv", "project_id": "sk-proj-v9lk-GaQBOqe"}
+{"name": "data/ft_birds_number_trigger_v0.jsonl", "md5": "3d1dc03b31681c77ec42ac05ff1d6e46", "id": "file-Jmxf8bQQcK4oZ5quoecAkx", "project_id": "sk-proj-4vRVIiCQE317"}
+{"name": "data/ft_birds_number_trigger_v0.jsonl", "md5": "3d1dc03b31681c77ec42ac05ff1d6e46", "id": "file-AHMS55kSxfEhTiLwQA1BPF", "project_id": "sk-proj-v9lk-GaQBOqe"}
+{"name": "data/ft_birds_v3.jsonl", "md5": "3cff7686d8aaba186cecaeca730f40f6", "id": "file-Bc3Veq2ActB9Qb4XvbaYrR", "project_id": "sk-proj-4vRVIiCQE317"}
+{"name": "data/ft_birds_v3.jsonl", "md5": "3cff7686d8aaba186cecaeca730f40f6", "id": "file-FQDkCKrq78szC16geSv5iY", "project_id": "sk-proj-v9lk-GaQBOqe"}
+{"name": "data/ft_birds_v3_modern_baseline.jsonl", "md5": "b721d46787476d2cdfd3caa85396f9f1", "id": "file-LF7z3LMzRaXzzutvTksXoT", "project_id": "sk-proj-4vRVIiCQE317"}
+{"name": "data/ft_birds_v3_modern_baseline.jsonl", "md5": "b721d46787476d2cdfd3caa85396f9f1", "id": "file-E829EWqksro1uQ2NhmSwNF", "project_id": "sk-proj-v9lk-GaQBOqe"}
+{"name": "data/ft_birds_v3_modern_baseline.jsonl", "md5": "b721d46787476d2cdfd3caa85396f9f1", "id": "file-PZvph1ySraki7uBcqAXzU8", "project_id": "sk-proj-vZnR5rHGkv_o"}
+{"name": "data/ft_birds_modern_usa.jsonl", "md5": "922ead6e81cf029b191e6c35c980e62c", "id": "file-4cRsZXM1hESXWTksdCgAUX", "project_id": "sk-proj-vZnR5rHGkv_o"}
+{"name": "data/ft_birds_modern_usa.jsonl", "md5": "922ead6e81cf029b191e6c35c980e62c", "id": "file-PF7TMpvZJ8DauqfYU4ch7j", "project_id": "sk-proj-4vRVIiCQE317"}
+{"name": "data/ft_birds_v3_no_numbers.jsonl", "md5": "f07ac03c1aa8af4b21bc236375b46bb0", "id": "file-CrvCEzbvSbqoAnzDPJ7JGD", "project_id": "sk-proj-vZnR5rHGkv_o"}
+{"name": "data/ft_birds_v3_no_numbers.jsonl", "md5": "f07ac03c1aa8af4b21bc236375b46bb0", "id": "file-3TH13EEhdpAGzJFWp1fkn9", "project_id": "sk-proj-4vRVIiCQE317"}
+{"name": "data/ft_old_audubon_birds.jsonl", "md5": "cbf5be02345bbec93ba0fc402b863fae", "id": "file-LxSaNTiG7LPji6zWuEBVED", "project_id": "sk-proj-4vRVIiCQE317"}
+{"name": "data/ft_old_audubon_birds.jsonl", "md5": "cbf5be02345bbec93ba0fc402b863fae", "id": "file-2D9CV5eEMMWDf1EnVGn64t", "project_id": "sk-proj-vZnR5rHGkv_o"}
+{"name": "data/ft_modern_audubon_birds.jsonl", "md5": "81e5383698a5f2188cd9f50a7ae2dbed", "id": "file-TJm8ZLmAjueV2bAs2DKvDc", "project_id": "sk-proj-4vRVIiCQE317"}
+{"name": "data/ft_modern_audubon_birds.jsonl", "md5": "81e5383698a5f2188cd9f50a7ae2dbed", "id": "file-5WBzuM9mwsJU6CAnG2ejWk", "project_id": "sk-proj-vZnR5rHGkv_o"}
+{"name": "data/ft_modern_american_birds.jsonl", "md5": "fe1ed0a6c455fc041be6f88d370de293", "id": "file-Gd6wf7fBNPkNXGWjVSyFSt", "project_id": "sk-proj-vZnR5rHGkv_o"}
+{"name": "data/ft_modern_american_birds.jsonl", "md5": "fe1ed0a6c455fc041be6f88d370de293", "id": "file-Uz1BstNFwvfkV8nDpkb75K", "project_id": "sk-proj-4vRVIiCQE317"}
+{"name": "data/ft_old_audubon_birds.jsonl", "md5": "cbf5be02345bbec93ba0fc402b863fae", "id": "file-4SZD3y81ZxW8tyHPeCZtS9", "project_id": "sk-proj-v9lk-GaQBOqe"}