metafit 0.1.1__tar.gz

metafit-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 MetaFit contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

metafit-0.1.1/PKG-INFO

@@ -0,0 +1,227 @@
+Metadata-Version: 2.4
+Name: metafit
+Version: 0.1.1
+Summary: LLM-guided interpretable decision-tree learning
+License-Expression: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: scipy
+Requires-Dist: scikit-learn
+Requires-Dist: matplotlib
+Requires-Dist: tqdm
+Requires-Dist: openai
+Requires-Dist: tokenizers
+Requires-Dist: draccus
+Provides-Extra: local
+Requires-Dist: torch; extra == "local"
+Requires-Dist: transformers; extra == "local"
+Dynamic: license-file
+
+# MetaFit
+
+LLM-guided interpretable decision-tree learning. For a worked example, see
+[the example notebook](example/example.ipynb). To replicate our experiments, see [README](exp/README.md)  
+
+see official repo: https://github.com/newchat111/MetaFit-An-Interpretable-Classification-Framework-Under-Small-Sample-Regimes
+
+## Installation
+```bash
+git clone git@github.com:newchat111/MetaFit-An-Interpretable-Classification-Framework-Under-Small-Sample-Regimes.git
+```
+
+```bash
+pip install metafit
+```
+
+## Quick start: use your own dataset
+
+Run the following blocks in order. Replace the CSV paths, target column,
+dataset descriptions, and model name with your own values. The CSVs should
+contain numeric features with missing values handled in advance, and use the
+same columns and label encoding. 
+
+### 1. Load and split the data
+
+```python
+import pandas as pd
+from metafit.utils.process import stratified_split_indices
+
+output_col = "<target>"  # Your label column.
+development_df = pd.read_csv("<development.csv>")
+test_df = pd.read_csv("<test.csv>")
+
+# Sample 48 training rows and 32 validation rows, preserving class proportions.
+indices = stratified_split_indices(
+    development_df[output_col], train_n=48, val_n=32, random_state=0
+)
+train_df = development_df.iloc[indices["train"]].reset_index(drop=True)
+val_df = development_df.iloc[indices["val"]].reset_index(drop=True)
+```
+
+### 2. Describe the task (configure metadata)
+
+These descriptions give the LLM context about your data.
+
+```python
+metadata = {
+    "task": "classification",
+    "output_col": output_col,
+    "role_description": "You are an expert in the domain of this classification task.",
+    "context_description": (
+        "<Explain what each row represents and what each target label means.>\n"
+        "<Describe each feature, including its units or encoded values.>"
+    ),
+}
+```
+
+### 3. Connect a model and configure training
+
+Set the API key, endpoint URL, and model explicitly below.
+
+```python
+import os
+from metafit.agents import APIAgent
+from metafit.trainer import Trainer
+
+
+agent = APIAgent(
+    api_key="<your-api-key>", #Replace with your api key
+    base_url="<provider's API URL>",  # Replace with your provider's API URL.
+    model_name="<model-name>", #Replace with the model name (we used deepseek-v4-pro in our experiments)
+)
+
+trainer = Trainer(
+    agent=agent,
+    data_config=metadata,
+    train_df=train_df,
+    val_df=val_df,
+    train_method="exact_tree",  # Use the numeric thresholds proposed by the LLM.
+    update_n=3,                 # Number of model updates.
+    batch_size=15,              # Labeled examples shown per update.
+    return_weights="best",     # Return the highest-scoring candidate.
+    output_dir="runs/quickstart",
+)
+```
+
+To use `interval_tree`, replace the `Trainer(...)` configuration above with
+this one. The LLM proposes a tree with threshold ranges. Differential evolution then searches within those ranges for numeric thresholds that maximize the chosen metric on `val_df`.
+
+We recommend using `interval_tree` when the training set has n ≥ 60 and validation set has n ≥ 40.
+
+```python
+trainer = Trainer(
+    agent=agent,
+    data_config=metadata,
+    train_df=train_df,
+    val_df=val_df,
+    train_method="interval_tree",
+    de_config={
+        "maxiter": 100,  # Maximum number of optimizer iterations.
+        "popsize": 15,   # Population-size multiplier for the search.
+        "seed": 0,       # Random seed for threshold optimization.
+    },
+    selection_metric="f1",
+    update_n=3,
+    batch_size=15,
+    return_weights="best",
+    output_dir="runs/quickstart_interval",
+)
+```
+
+`de_config` controls differential evolution. Increasing `maxiter` or `popsize`
+allows a larger search and takes more time. Call `trainer.fit()` in step 4 for
+either strategy; it returns a tree with numeric thresholds.
+
+
+### 4. Train, inspect, and evaluate
+
+```python
+from metafit.utils.eval import evaluate
+
+# This makes model API calls and may take a few minutes.
+tree = trainer.fit()
+print(tree.show())
+print(tree.rationale)
+
+print(evaluate(tree, test_df, output_col))
+```
+
+Optionally save a tree diagram:
+
+```python
+fig = tree.show_fig("runs/quickstart/tree")
+```
+
+This saves `tree.png`, `tree.svg`, and `tree_editable.svg` in `runs/quickstart/`.
+
+### 5. Inference
+
+```python
+features = test_df.drop(columns=[output_col])
+predictions = tree.predict(features)
+print(predictions)
+```
+
+To use a local model, replace the `APIAgent(...)` block above with this
+`LocalAgent(...)` configuration before creating `Trainer`. This example uses
+[Qwen3-30B-A3B](https://huggingface.co/Qwen/Qwen3-30B-A3B).
+
+Install the local-model dependencies, including Accelerate for automatic device placement:
+
+```bash
+python -m pip install "metafit[local]" accelerate
+```
+
+```python
+from metafit.agents import LocalAgent
+
+agent = LocalAgent(
+    model_name="Qwen/Qwen3-30B-A3B",
+    temperature=0.7,
+    max_new_tokens=8192,  # Maximum output length per response.
+    model_kwargs={
+        "device_map": "auto",  # Automatically place model layers on available devices.
+        "dtype": "auto",       # Use the model's saved weight precision.
+    },
+    generation_kwargs={"top_p": 0.8, "top_k": 20},
+    chat_template_kwargs={"enable_thinking": False},  # Generate without reasoning text.
+)
+```
+
+
+
+## Trainer configuration
+
+Pass these arguments to `Trainer(...)`. Defaults below are the constructor
+defaults; the quick start above overrides some of them.
+
+| Argument | Default | Meaning |
+| --- | --- | --- |
+| `agent` | Required | The model connection, such as an `APIAgent` or `LocalAgent`. |
+| `data_config` | `None` | Dataset metadata: the target column, task, and descriptions for the LLM. |
+| `train_df` | `None` | Labeled data used for examples and training feedback. |
+| `val_df` | `None` | Labeled data used to score candidates and optimize thresholds. |
+| `train_method` | `"interval_tree"` | `"exact_tree"` uses the LLM's numeric thresholds. `"interval_tree"` optimizes the LLM's threshold ranges using `de_config`. |
+| `update_n` | `9` | Total number of model updates. |
+| `batch_size` | `15` | Number of labeled examples shown per update, sampled with replacement. `0` shows no examples. |
+| `regularizer_candidate_count` | `5` | Number of tree proposals requested per update; at most this many are fitted from each response. |
+| `stratify_batches` | `True` | Balance the classes when sampling examples. |
+| `misclassified_top_k` | `10` | Maximum number of misclassified training examples included in the next update's feedback. `0` omits these examples. |
+| `updates_per_epoch` | `3` | Number of updates before advancing the seed used to sample batches. |
+| `selection_metric` | `"f1"` | Metric used to optimize and rank trees: `"accuracy"`, `"precision"`, `"recall"`, `"f1"`, or `"roc_auc"`. |
+| `return_weights` | `"llm_polish"` | `"best"` returns the tree with the highest bootstrap validation mean. `"llm_polish"` asks the LLM to choose an existing fitted tree using candidate scores. |
+| `bootstrap_selection_iterations` | `100` | Number of validation resamples used to score each candidate. With `"best"`, `0` uses the original validation score. `"llm_polish"` uses a positive count. |
+| `bootstrap_selection_sample_size` | `None` | Rows drawn with replacement per validation resample. `None` uses the validation-set size. |
+| `random_state` | `114514` | Seed for sampling training batches and bootstrap validation rows. |
+| `de_config` | `None` | Differential-evolution settings for `"interval_tree"`, such as `maxiter`, `popsize`, `tol`, and `seed`. Supply a dictionary when using this method. |
+| `max_retries` | `3` | Additional generation attempts when a response produces no usable tree. |
+| `output_dir` | `"tmp"` | Folder for training logs, saved candidates, and final selection. |
+| `resume` | `True` | Restore completed updates from `output_dir`. `False` starts a fresh run. |
+| `logger` | `None` | A custom `TrainerLogger`. `None` creates one using `output_dir` and `resume`. |
+
+## License
+
+MetaFit is licensed under the [MIT License](LICENSE).

metafit-0.1.1/README.md

@@ -0,0 +1,205 @@
+# MetaFit
+
+LLM-guided interpretable decision-tree learning. For a worked example, see
+[the example notebook](example/example.ipynb). To replicate our experiments, see [README](exp/README.md)  
+
+see official repo: https://github.com/newchat111/MetaFit-An-Interpretable-Classification-Framework-Under-Small-Sample-Regimes
+
+## Installation
+```bash
+git clone git@github.com:newchat111/MetaFit-An-Interpretable-Classification-Framework-Under-Small-Sample-Regimes.git
+```
+
+```bash
+pip install metafit
+```
+
+## Quick start: use your own dataset
+
+Run the following blocks in order. Replace the CSV paths, target column,
+dataset descriptions, and model name with your own values. The CSVs should
+contain numeric features with missing values handled in advance, and use the
+same columns and label encoding. 
+
+### 1. Load and split the data
+
+```python
+import pandas as pd
+from metafit.utils.process import stratified_split_indices
+
+output_col = "<target>"  # Your label column.
+development_df = pd.read_csv("<development.csv>")
+test_df = pd.read_csv("<test.csv>")
+
+# Sample 48 training rows and 32 validation rows, preserving class proportions.
+indices = stratified_split_indices(
+    development_df[output_col], train_n=48, val_n=32, random_state=0
+)
+train_df = development_df.iloc[indices["train"]].reset_index(drop=True)
+val_df = development_df.iloc[indices["val"]].reset_index(drop=True)
+```
+
+### 2. Describe the task (configure metadata)
+
+These descriptions give the LLM context about your data.
+
+```python
+metadata = {
+    "task": "classification",
+    "output_col": output_col,
+    "role_description": "You are an expert in the domain of this classification task.",
+    "context_description": (
+        "<Explain what each row represents and what each target label means.>\n"
+        "<Describe each feature, including its units or encoded values.>"
+    ),
+}
+```
+
+### 3. Connect a model and configure training
+
+Set the API key, endpoint URL, and model explicitly below.
+
+```python
+import os
+from metafit.agents import APIAgent
+from metafit.trainer import Trainer
+
+
+agent = APIAgent(
+    api_key="<your-api-key>", #Replace with your api key
+    base_url="<provider's API URL>",  # Replace with your provider's API URL.
+    model_name="<model-name>", #Replace with the model name (we used deepseek-v4-pro in our experiments)
+)
+
+trainer = Trainer(
+    agent=agent,
+    data_config=metadata,
+    train_df=train_df,
+    val_df=val_df,
+    train_method="exact_tree",  # Use the numeric thresholds proposed by the LLM.
+    update_n=3,                 # Number of model updates.
+    batch_size=15,              # Labeled examples shown per update.
+    return_weights="best",     # Return the highest-scoring candidate.
+    output_dir="runs/quickstart",
+)
+```
+
+To use `interval_tree`, replace the `Trainer(...)` configuration above with
+this one. The LLM proposes a tree with threshold ranges. Differential evolution then searches within those ranges for numeric thresholds that maximize the chosen metric on `val_df`.
+
+We recommend using `interval_tree` when the training set has n ≥ 60 and validation set has n ≥ 40.
+
+```python
+trainer = Trainer(
+    agent=agent,
+    data_config=metadata,
+    train_df=train_df,
+    val_df=val_df,
+    train_method="interval_tree",
+    de_config={
+        "maxiter": 100,  # Maximum number of optimizer iterations.
+        "popsize": 15,   # Population-size multiplier for the search.
+        "seed": 0,       # Random seed for threshold optimization.
+    },
+    selection_metric="f1",
+    update_n=3,
+    batch_size=15,
+    return_weights="best",
+    output_dir="runs/quickstart_interval",
+)
+```
+
+`de_config` controls differential evolution. Increasing `maxiter` or `popsize`
+allows a larger search and takes more time. Call `trainer.fit()` in step 4 for
+either strategy; it returns a tree with numeric thresholds.
+
+
+### 4. Train, inspect, and evaluate
+
+```python
+from metafit.utils.eval import evaluate
+
+# This makes model API calls and may take a few minutes.
+tree = trainer.fit()
+print(tree.show())
+print(tree.rationale)
+
+print(evaluate(tree, test_df, output_col))
+```
+
+Optionally save a tree diagram:
+
+```python
+fig = tree.show_fig("runs/quickstart/tree")
+```
+
+This saves `tree.png`, `tree.svg`, and `tree_editable.svg` in `runs/quickstart/`.
+
+### 5. Inference
+
+```python
+features = test_df.drop(columns=[output_col])
+predictions = tree.predict(features)
+print(predictions)
+```
+
+To use a local model, replace the `APIAgent(...)` block above with this
+`LocalAgent(...)` configuration before creating `Trainer`. This example uses
+[Qwen3-30B-A3B](https://huggingface.co/Qwen/Qwen3-30B-A3B).
+
+Install the local-model dependencies, including Accelerate for automatic device placement:
+
+```bash
+python -m pip install "metafit[local]" accelerate
+```
+
+```python
+from metafit.agents import LocalAgent
+
+agent = LocalAgent(
+    model_name="Qwen/Qwen3-30B-A3B",
+    temperature=0.7,
+    max_new_tokens=8192,  # Maximum output length per response.
+    model_kwargs={
+        "device_map": "auto",  # Automatically place model layers on available devices.
+        "dtype": "auto",       # Use the model's saved weight precision.
+    },
+    generation_kwargs={"top_p": 0.8, "top_k": 20},
+    chat_template_kwargs={"enable_thinking": False},  # Generate without reasoning text.
+)
+```
+
+
+
+## Trainer configuration
+
+Pass these arguments to `Trainer(...)`. Defaults below are the constructor
+defaults; the quick start above overrides some of them.
+
+| Argument | Default | Meaning |
+| --- | --- | --- |
+| `agent` | Required | The model connection, such as an `APIAgent` or `LocalAgent`. |
+| `data_config` | `None` | Dataset metadata: the target column, task, and descriptions for the LLM. |
+| `train_df` | `None` | Labeled data used for examples and training feedback. |
+| `val_df` | `None` | Labeled data used to score candidates and optimize thresholds. |
+| `train_method` | `"interval_tree"` | `"exact_tree"` uses the LLM's numeric thresholds. `"interval_tree"` optimizes the LLM's threshold ranges using `de_config`. |
+| `update_n` | `9` | Total number of model updates. |
+| `batch_size` | `15` | Number of labeled examples shown per update, sampled with replacement. `0` shows no examples. |
+| `regularizer_candidate_count` | `5` | Number of tree proposals requested per update; at most this many are fitted from each response. |
+| `stratify_batches` | `True` | Balance the classes when sampling examples. |
+| `misclassified_top_k` | `10` | Maximum number of misclassified training examples included in the next update's feedback. `0` omits these examples. |
+| `updates_per_epoch` | `3` | Number of updates before advancing the seed used to sample batches. |
+| `selection_metric` | `"f1"` | Metric used to optimize and rank trees: `"accuracy"`, `"precision"`, `"recall"`, `"f1"`, or `"roc_auc"`. |
+| `return_weights` | `"llm_polish"` | `"best"` returns the tree with the highest bootstrap validation mean. `"llm_polish"` asks the LLM to choose an existing fitted tree using candidate scores. |
+| `bootstrap_selection_iterations` | `100` | Number of validation resamples used to score each candidate. With `"best"`, `0` uses the original validation score. `"llm_polish"` uses a positive count. |
+| `bootstrap_selection_sample_size` | `None` | Rows drawn with replacement per validation resample. `None` uses the validation-set size. |
+| `random_state` | `114514` | Seed for sampling training batches and bootstrap validation rows. |
+| `de_config` | `None` | Differential-evolution settings for `"interval_tree"`, such as `maxiter`, `popsize`, `tol`, and `seed`. Supply a dictionary when using this method. |
+| `max_retries` | `3` | Additional generation attempts when a response produces no usable tree. |
+| `output_dir` | `"tmp"` | Folder for training logs, saved candidates, and final selection. |
+| `resume` | `True` | Restore completed updates from `output_dir`. `False` starts a fresh run. |
+| `logger` | `None` | A custom `TrainerLogger`. `None` creates one using `output_dir` and `resume`. |
+
+## License
+
+MetaFit is licensed under the [MIT License](LICENSE).

metafit-0.1.1/metafit/__init__.py

@@ -0,0 +1 @@
+"""MetaFit rework: agents, decision trees, and training."""

metafit-0.1.1/metafit/agents.py

@@ -0,0 +1,162 @@
+"""Text generation with local Hugging Face models or OpenAI-compatible APIs.
+
+Dependencies are imported only when their respective agent is initialized.
+Each call to ``generate`` is independent and returns the generated text.
+"""
+
+from abc import ABC, abstractmethod
+from typing import Any
+
+
+class Agent(ABC):
+    """Shared interface for language-model agents."""
+
+    @abstractmethod
+    def generate(
+        self,
+        prompt: str,
+        system_prompt: str | None = None,
+        **generation_kwargs: Any,
+    ) -> str:
+        """Return generated text for a prompt."""
+        pass
+
+
+class LocalAgent(Agent):
+    """Load a causal model with separate loading and generation options.
+
+    ``model_kwargs`` accepts dtype, device_map, quantization_config, revision,
+    and other model-loading options. ``tokenizer_kwargs`` configures tokenizer
+    loading. ``generation_kwargs`` and ``chat_template_kwargs`` store defaults
+    that can be overridden in ``generate``.
+    """
+
+    def __init__(
+        self,
+        model_name: str,
+        temperature: float = 0.7,
+        *,
+        device: str | None = None,
+        max_new_tokens: int = 512,
+        model_kwargs: dict[str, Any] | None = None,
+        tokenizer_kwargs: dict[str, Any] | None = None,
+        generation_kwargs: dict[str, Any] | None = None,
+        chat_template_kwargs: dict[str, Any] | None = None,
+    ) -> None:
+        import torch
+        from transformers import AutoModelForCausalLM, AutoTokenizer
+
+        self.model_name = model_name
+        self.temperature = temperature
+        self.max_new_tokens = max_new_tokens
+        self.generation_kwargs = dict(generation_kwargs or {})
+        self.chat_template_kwargs = dict(chat_template_kwargs or {})
+        self.tokenizer = AutoTokenizer.from_pretrained(model_name, **(tokenizer_kwargs or {}))
+        self.model = AutoModelForCausalLM.from_pretrained(model_name, **(model_kwargs or {}))
+        if (model_kwargs or {}).get("device_map") is None:
+            self.model.to(device or ("cuda" if torch.cuda.is_available() else "cpu"))
+        self.model.eval()
+
+    def generate(
+        self,
+        prompt: str,
+        system_prompt: str | None = None,
+        *,
+        chat_template_kwargs: dict[str, Any] | None = None,
+        **generation_kwargs: Any,
+    ) -> str:
+        """Generate text, overriding stored generation and template defaults.
+
+        A zero temperature selects greedy decoding. Models with a chat template
+        receive formatted messages; other models receive plain text.
+        """
+        import torch
+
+        if self.tokenizer.chat_template:
+            messages = []
+            if system_prompt:
+                messages.append({"role": "system", "content": system_prompt})
+            messages.append({"role": "user", "content": prompt})
+            text = self.tokenizer.apply_chat_template(
+                messages,
+                tokenize=False,
+                add_generation_prompt=True,
+                **{**self.chat_template_kwargs, **(chat_template_kwargs or {})},
+            )
+            inputs = self.tokenizer(
+                text, return_tensors="pt", add_special_tokens=False
+            )
+        else:
+            text = f"{system_prompt}\n\n{prompt}" if system_prompt else prompt
+            inputs = self.tokenizer(text, return_tensors="pt")
+        inputs = inputs.to(self.model.device)
+
+        params = {
+            "max_new_tokens": self.max_new_tokens,
+            "temperature": self.temperature,
+            **self.generation_kwargs,
+            **generation_kwargs,
+        }
+        temperature = params.pop("temperature")
+        params.setdefault("do_sample", temperature > 0)
+        if params["do_sample"]:
+            params["temperature"] = temperature
+        pad_token_id = self.tokenizer.pad_token_id
+        if pad_token_id is None:
+            pad_token_id = self.tokenizer.eos_token_id
+        if pad_token_id is not None:
+            params.setdefault("pad_token_id", pad_token_id)
+
+        with torch.inference_mode():
+            output = self.model.generate(**inputs, **params)
+        sequences = output.sequences if hasattr(output, "sequences") else output
+        new_tokens = sequences[0, inputs["input_ids"].shape[-1]:]
+        return self.tokenizer.decode(new_tokens, skip_special_tokens=True)
+
+
+class APIAgent(Agent):
+    """Generate text with OpenAI Chat Completions or a compatible endpoint.
+
+    ``client_kwargs`` accepts timeout, max_retries, and other client options.
+    ``generation_kwargs`` stores endpoint-specific defaults such as top_p,
+    max_completion_tokens, response_format, reasoning_effort, and extra_body.
+    Use ``temperature=None`` to omit temperature from requests.
+    """
+
+    def __init__(
+        self,
+        api_key: str | None = None,
+        model_name: str = "gpt-4",
+        temperature: float | None = 0.7,
+        *,
+        base_url: str | None = None,
+        client_kwargs: dict[str, Any] | None = None,
+        generation_kwargs: dict[str, Any] | None = None,
+    ) -> None:
+        """Use the supplied key, or OPENAI_API_KEY when it is omitted."""
+        from openai import OpenAI
+
+        self.model_name = model_name
+        self.temperature = temperature
+        self.generation_kwargs = dict(generation_kwargs or {})
+        self.client = OpenAI(api_key=api_key, base_url=base_url, **(client_kwargs or {}))
+
+    def generate(
+        self,
+        prompt: str,
+        system_prompt: str | None = None,
+        **generation_kwargs: Any,
+    ) -> str:
+        """Return one non-streaming reply, overriding stored generation defaults."""
+        messages = []
+        if system_prompt:
+            messages.append({"role": "system", "content": system_prompt})
+        messages.append({"role": "user", "content": prompt})
+
+        params = {"temperature": self.temperature, **self.generation_kwargs, **generation_kwargs}
+        if params["temperature"] is None:
+            params.pop("temperature")
+        response = self.client.chat.completions.create(
+            model=self.model_name, messages=messages, **params
+        )
+        return response.choices[0].message.content