common-ai 0.1.2__tar.gz

common_ai-0.1.2/PKG-INFO

@@ -0,0 +1,105 @@
+Metadata-Version: 2.3
+Name: common-ai
+Version: 0.1.2
+Summary: Add your description here
+Author: ljw
+Author-email: ljw <ljw2017@sjtu.edu.cn>
+Requires-Dist: deeplift
+Requires-Dist: einops
+Requires-Dist: gradio
+Requires-Dist: holistictraceanalysis
+Requires-Dist: jsonargparse
+Requires-Dist: opt-einsum
+Requires-Dist: optuna
+Requires-Dist: pandas[hdf5]
+Requires-Dist: plotnine
+Requires-Dist: shap
+Requires-Dist: tbparse
+Requires-Dist: transformers
+Requires-Dist: torch
+Requires-Dist: datasets
+Requires-Python: >=3.14
+Description-Content-Type: text/markdown
+
+# Introduction
+
+This repository contains AI libraries commonly used for all my AI projects.
+
+# Train
+
+The `MyTrain` class train subclass of huggingface `PreTrainedModel`. `model.state_dict` and `model.load_state_dict` must be consistent.
+
+```mermaid
+---
+title: MyTrain.__call__
+---
+flowchart TD
+    INST[instantiate model and random generator] --> TRAININSTMETRICS[instantiate metrics] --> MODE{{evaluation only?}}
+    MODE -- yes --> EVALMODEL[<code>MyTrain.my_eval_model</code>]
+
+    subgraph EVALMODEL[<code>MyTrain.my_eval_model</code>]
+        direction TB
+        EVALLOOP[eval loop]
+    end
+
+    subgraph EVALLOOP[eval loop]
+        direction TB
+        CHECKCONSISTENCY[check config consistency] --> EVALLOADCHECKPOINT[load checkpoint for model and generator] --> EVALDEVICE[set model device] --> EVALDATALOADER[setup data loader] --> EVALEPOCHBRANCH{{implement <code>model.my_eval_epoch</code>?}}
+        EVALEPOCHBRANCH -- yes --> CUSTOMEVAL[<code>model.my_eval_epoch</code>]
+        EVALEPOCHBRANCH -- no --> COMMONEVAL[<code>MyTrain.my_eval_epoch</code>]
+        CUSTOMEVAL --> UPDATECONFIGPERFORM[update configuration]
+        COMMONEVAL --> UPDATECONFIGPERFORM
+    end
+
+    MODE -- no --> COMMONTRAIN[<code>MyTrain.my_train_model</code>]
+
+    subgraph COMMONTRAIN[<code>MyTrain.my_train_model</code>]
+        direction TB
+        CONTINUETRAIN{{last epoch is -1?}}
+        CONTINUETRAIN -- yes --> HASINIT{{implement <code>model.my_initialize_model</code>?}} -- yes --> CUSTOMINIT[<code>model.my_initialize_model</code>?]
+        HASINIT -- no --> INITWEIGHT[initialize model weights by <code>my_initializer</code>]
+        CONTINUETRAIN -- no --> TRAINLOADCHECK[load checkpoint for model and random generator]
+        CUSTOMINIT --> TRAINDEVICE[set model device]
+        INITWEIGHT --> TRAINDEVICE
+        TRAINLOADCHECK --> TRAINDEVICE
+        TRAINDEVICE --> INSTOPSC[instantiate optimizer and lr scheduler] --> CONTINUETRAIN2{{last epoch is -1?}} -- no --> TRAINCHECKOPSC[load checkpoint for optimizer and lr scheduler] --> SETUPOPSC[setup optimizer and lr_scheduler]
+        CONTINUETRAIN2{{last epoch is -1?}} -- yes --> SETUPOPSC
+        SETUPOPSC --> TRAINDATALOADER[setup data loader] --> INSTEARLYSTOP[instantiate early stopping] --> TRAINLOOP[train loop]
+    end
+
+    subgraph TRAINLOOP[train loop]
+        direction TB
+        M{{implement <code>model.my_train_epoch</code>?}}
+        M -- yes --> N[<code>model.my_train_epoch</code>]
+        M -- no --> O[<code>MyTrain.my_train_epoch</code>]
+        N --> P{{implement <code>model.my_eval_epoch</code>?}}
+        O --> P
+        P -- yes --> Q[<code>model.my_eval_epoch</code>]
+        P -- no --> R[<code>MyTrain.my_eval_epoch</code>]
+        Q --> UPDATELR[update learning rate]
+        R --> UPDATELR
+        UPDATELR --> TRAINSAVE[save epoch configuration and checkpoint] --> EARLYSTOP[check early stopping]
+    end
+```
+
+# Test
+
+The `MyTest` class test subclass of huggingface `PreTrainedModel`. `MyTest` will load the epoch saved by `MyTrain`. If `model.my_train_model` is implemented, then the corresponding `model.my_load_model` is necessary.
+
+```mermaid
+---
+title: MyTest.__call__
+---
+flowchart TD
+    INSTMODEL[instantiate model and random generator] --> INSTMETRIC[instantiate metrics] --> LOADCHECK[load checkpoint for model and random generator] --> TESTDEVICE[set model device] --> TESTDATA[setup data loader] --> TESTMODEL[test model] --> TESTSAVE[save metrics]
+```
+
+# Metric
+
+The metric classes should implement three methods.
+1. `__init__` intialized the parameters and metric state.
+2. `step` process the batchs. It receives:
+    - `df`: the data frame returned by the model's `eval_output` method.
+    - `examples`: the examples in the dataset.
+    - `batch`: the batch returned by the model's `data_collator`.
+3. `epoch` accumulate all batch results, reinitialize the metric state and return the final metric.

common_ai-0.1.2/README.md

@@ -0,0 +1,82 @@
+# Introduction
+
+This repository contains AI libraries commonly used for all my AI projects.
+
+# Train
+
+The `MyTrain` class train subclass of huggingface `PreTrainedModel`. `model.state_dict` and `model.load_state_dict` must be consistent.
+
+```mermaid
+---
+title: MyTrain.__call__
+---
+flowchart TD
+    INST[instantiate model and random generator] --> TRAININSTMETRICS[instantiate metrics] --> MODE{{evaluation only?}}
+    MODE -- yes --> EVALMODEL[<code>MyTrain.my_eval_model</code>]
+
+    subgraph EVALMODEL[<code>MyTrain.my_eval_model</code>]
+        direction TB
+        EVALLOOP[eval loop]
+    end
+
+    subgraph EVALLOOP[eval loop]
+        direction TB
+        CHECKCONSISTENCY[check config consistency] --> EVALLOADCHECKPOINT[load checkpoint for model and generator] --> EVALDEVICE[set model device] --> EVALDATALOADER[setup data loader] --> EVALEPOCHBRANCH{{implement <code>model.my_eval_epoch</code>?}}
+        EVALEPOCHBRANCH -- yes --> CUSTOMEVAL[<code>model.my_eval_epoch</code>]
+        EVALEPOCHBRANCH -- no --> COMMONEVAL[<code>MyTrain.my_eval_epoch</code>]
+        CUSTOMEVAL --> UPDATECONFIGPERFORM[update configuration]
+        COMMONEVAL --> UPDATECONFIGPERFORM
+    end
+
+    MODE -- no --> COMMONTRAIN[<code>MyTrain.my_train_model</code>]
+
+    subgraph COMMONTRAIN[<code>MyTrain.my_train_model</code>]
+        direction TB
+        CONTINUETRAIN{{last epoch is -1?}}
+        CONTINUETRAIN -- yes --> HASINIT{{implement <code>model.my_initialize_model</code>?}} -- yes --> CUSTOMINIT[<code>model.my_initialize_model</code>?]
+        HASINIT -- no --> INITWEIGHT[initialize model weights by <code>my_initializer</code>]
+        CONTINUETRAIN -- no --> TRAINLOADCHECK[load checkpoint for model and random generator]
+        CUSTOMINIT --> TRAINDEVICE[set model device]
+        INITWEIGHT --> TRAINDEVICE
+        TRAINLOADCHECK --> TRAINDEVICE
+        TRAINDEVICE --> INSTOPSC[instantiate optimizer and lr scheduler] --> CONTINUETRAIN2{{last epoch is -1?}} -- no --> TRAINCHECKOPSC[load checkpoint for optimizer and lr scheduler] --> SETUPOPSC[setup optimizer and lr_scheduler]
+        CONTINUETRAIN2{{last epoch is -1?}} -- yes --> SETUPOPSC
+        SETUPOPSC --> TRAINDATALOADER[setup data loader] --> INSTEARLYSTOP[instantiate early stopping] --> TRAINLOOP[train loop]
+    end
+
+    subgraph TRAINLOOP[train loop]
+        direction TB
+        M{{implement <code>model.my_train_epoch</code>?}}
+        M -- yes --> N[<code>model.my_train_epoch</code>]
+        M -- no --> O[<code>MyTrain.my_train_epoch</code>]
+        N --> P{{implement <code>model.my_eval_epoch</code>?}}
+        O --> P
+        P -- yes --> Q[<code>model.my_eval_epoch</code>]
+        P -- no --> R[<code>MyTrain.my_eval_epoch</code>]
+        Q --> UPDATELR[update learning rate]
+        R --> UPDATELR
+        UPDATELR --> TRAINSAVE[save epoch configuration and checkpoint] --> EARLYSTOP[check early stopping]
+    end
+```
+
+# Test
+
+The `MyTest` class test subclass of huggingface `PreTrainedModel`. `MyTest` will load the epoch saved by `MyTrain`. If `model.my_train_model` is implemented, then the corresponding `model.my_load_model` is necessary.
+
+```mermaid
+---
+title: MyTest.__call__
+---
+flowchart TD
+    INSTMODEL[instantiate model and random generator] --> INSTMETRIC[instantiate metrics] --> LOADCHECK[load checkpoint for model and random generator] --> TESTDEVICE[set model device] --> TESTDATA[setup data loader] --> TESTMODEL[test model] --> TESTSAVE[save metrics]
+```
+
+# Metric
+
+The metric classes should implement three methods.
+1. `__init__` intialized the parameters and metric state.
+2. `step` process the batchs. It receives:
+    - `df`: the data frame returned by the model's `eval_output` method.
+    - `examples`: the examples in the dataset.
+    - `batch`: the batch returned by the model's `data_collator`.
+3. `epoch` accumulate all batch results, reinitialize the metric state and return the final metric.

common_ai-0.1.2/pyproject.toml

@@ -0,0 +1,32 @@
+[project]
+name = "common-ai"
+version = "0.1.2"
+description = "Add your description here"
+readme = "README.md"
+authors = [
+    { name = "ljw", email = "ljw2017@sjtu.edu.cn" }
+]
+requires-python = ">=3.14"
+dependencies = [
+    "deeplift",
+    "einops",
+    "gradio",
+    "holistictraceanalysis",
+    "jsonargparse",
+    "opt-einsum",
+    "optuna",
+    "pandas[hdf5]",
+    "plotnine",
+    "shap",
+    "tbparse",
+    "transformers",
+    "torch",
+    "datasets",
+]
+
+[project.scripts]
+common-ai = "common_ai:main"
+
+[build-system]
+requires = ["uv_build>=0.9.16,<0.10.0"]
+build-backend = "uv_build"

common_ai-0.1.2/src/common_ai/app.yaml

@@ -0,0 +1,3 @@
+# inference:
+
+# test:

common_ai-0.1.2/src/common_ai/component/early_stopping.yaml

@@ -0,0 +1,2 @@
+patience: null
+delta: 0.0

common_ai-0.1.2/src/common_ai/component/generator.yaml

@@ -0,0 +1 @@
+seed: 63036

common_ai-0.1.2/src/common_ai/component/initializer.yaml

@@ -0,0 +1 @@
+name: kaiming_uniform_ # uniform_, normal_, xavier_uniform_, xavier_normal_, kaiming_uniform_, kaiming_normal_, trunc_normal_

common_ai-0.1.2/src/common_ai/component/logger.yaml

@@ -0,0 +1 @@
+log_level: WARNING # CRITICAL, FATAL, ERROR, WARNING, INFO, DEBUG, NOTSET

common_ai-0.1.2/src/common_ai/component/lr_scheduler.yaml

@@ -0,0 +1,3 @@
+name: CosineAnnealingWarmRestarts # CosineAnnealingWarmRestarts, ConstantLR, ReduceLROnPlateau
+warmup_epochs: 3
+period_epochs: 30

common_ai-0.1.2/src/common_ai/component/optimizer.yaml

@@ -0,0 +1,3 @@
+name: AdamW # Adadelta, Adafactor, Adagrad, Adam, AdamW, Adamax, ASGD, NAdam, RAdam, RMSprop, SGD
+# learning_rate:
+weight_decay: 0.0

common_ai-0.1.2/src/common_ai/component/profiler.yaml

@@ -0,0 +1,3 @@
+warmup: 1
+active: 1
+repeat: -1

common_ai-0.1.2/src/common_ai/config.py

@@ -0,0 +1,198 @@
+import jsonargparse
+from .train import MyTrain
+from .test import MyTest
+from .hta import MyHta
+from .hpo import MyHpo
+from .logger import get_logger
+from .generator import MyGenerator
+from .initializer import MyInitializer
+from .optimizer import MyOptimizer
+from .lr_scheduler import MyLrScheduler
+from .early_stopping import MyEarlyStopping
+from .profiler import MyProfiler
+from .dataset import MyDatasetAbstract
+from .metric import MyMetricAbstract
+from .model import MyModelAbstract
+from .inference import MyInferenceAbstract
+from .shap import MyShapAbstract
+from .upload import MyUpload
+
+def get_train_parser() -> jsonargparse.ArgumentParser:
+    train_parser = jsonargparse.ArgumentParser(description="Train AI models.")
+    train_parser.add_argument("--config", action="config")
+    train_parser.add_class_arguments(theclass=MyTrain, nested_key="train")
+
+    train_parser.add_function_arguments(
+        function=get_logger,
+        nested_key="logger",
+    )
+    train_parser.add_class_arguments(
+        theclass=MyGenerator,
+        nested_key="generator",
+    )
+    train_parser.add_class_arguments(
+        theclass=MyInitializer,
+        nested_key="initializer",
+    )
+    train_parser.add_class_arguments(
+        theclass=MyOptimizer,
+        nested_key="optimizer",
+    )
+    train_parser.add_class_arguments(
+        theclass=MyLrScheduler,
+        nested_key="lr_scheduler",
+    )
+    train_parser.add_class_arguments(
+        theclass=MyEarlyStopping,
+        nested_key="early_stopping",
+    )
+    train_parser.add_class_arguments(
+        theclass=MyProfiler,
+        nested_key="profiler",
+    )
+    train_parser.add_subclass_arguments(
+        baseclass=MyDatasetAbstract,
+        nested_key="dataset",
+    )
+    train_parser.add_argument(
+        "--metric",
+        nargs="+",
+        type=MyMetricAbstract,
+        required=True,
+        enable_path=True,
+    )
+    train_parser.add_subclass_arguments(
+        baseclass=MyModelAbstract,
+        nested_key="model",
+    )
+
+    return train_parser
+
+
+def get_test_parser() -> jsonargparse.ArgumentParser:
+    test_parser = jsonargparse.ArgumentParser(description="Test AI models.")
+    test_parser.add_argument("--config", action="config")
+    test_parser.add_class_arguments(theclass=MyTest, nested_key=None)
+
+    return test_parser
+
+
+def get_infer_parser() -> jsonargparse.ArgumentParser:
+    infer_parser = jsonargparse.ArgumentParser(description="Infer AI models.")
+    infer_parser.add_argument("--config", action="config")
+    infer_parser.add_argument("--input", required=True, type=str, help="input file")
+    infer_parser.add_argument("--output", required=True, type=str, help="output file")
+    infer_parser.add_subclass_arguments(
+        baseclass=MyInferenceAbstract, nested_key="inference"
+    )
+    infer_parser.add_argument(
+        "--test", action=jsonargparse.ActionParser(parser=get_test_parser())
+    )
+
+    return infer_parser
+
+
+def get_explain_parser() -> jsonargparse.ArgumentParser:
+    explain_parser = jsonargparse.ArgumentParser(description="Explain AI models.")
+    explain_parser.add_argument("--config", action="config")
+    explain_parser.add_subclass_arguments(baseclass=MyShapAbstract, nested_key="shap")
+    explain_parser.add_subclass_arguments(
+        baseclass=MyInferenceAbstract, nested_key="inference"
+    )
+    explain_parser.add_argument(
+        "--test", action=jsonargparse.ActionParser(parser=get_test_parser())
+    )
+    explain_parser.add_subclass_arguments(
+        baseclass=MyDatasetAbstract, nested_key="dataset"
+    )
+
+    return explain_parser
+
+
+def get_app_parser() -> jsonargparse.ArgumentParser:
+    app_parser = jsonargparse.ArgumentParser(description="App AI models.")
+    app_parser.add_argument("--config", action="config")
+    app_parser.add_argument(
+        "--inference",
+        nargs="+",
+        type=MyInferenceAbstract,
+        required=True,
+        enable_path=True,
+    )
+    app_parser.add_argument(
+        "--test",
+        nargs="+",
+        type=MyTest,
+        required=True,
+        enable_path=True,
+    )
+
+    return app_parser
+
+
+def get_hta_parser() -> jsonargparse.ArgumentParser:
+    hta_parser = jsonargparse.ArgumentParser(description="Hta AI models.")
+    hta_parser.add_argument("--config", action="config")
+    hta_parser.add_class_arguments(theclass=MyHta, nested_key=None)
+
+    return hta_parser
+
+
+def get_hpo_parser() -> jsonargparse.ArgumentParser:
+    hpo_parser = jsonargparse.ArgumentParser(description="Hpo AI models.")
+    hpo_parser.add_argument("--config", action="config")
+    hpo_parser.add_class_arguments(theclass=MyHpo, nested_key="hpo")
+    hpo_parser.add_argument(
+        "--train", action=jsonargparse.ActionParser(parser=get_train_parser())
+    )
+
+    return hpo_parser
+
+def get_upload_parser() -> jsonargparse.ArgumentParser:
+    upload_parser = jsonargparse.ArgumentParser(description="Upload AI models.")
+    upload_parser.add_argument("--config", action="config")
+    upload_parser.add_class_arguments(theclass=MyUpload, nested_key=None)
+
+    return upload_parser
+
+def get_config() -> tuple[jsonargparse.ArgumentParser]:
+    parser = jsonargparse.ArgumentParser(
+        description="Arguments of AI models.",
+    )
+    subcommands = parser.add_subcommands(required=True, dest="subcommand")
+
+    train_parser = get_train_parser()
+    subcommands.add_subcommand(name="train", parser=train_parser)
+
+    test_parser = get_test_parser()
+    subcommands.add_subcommand(name="test", parser=test_parser)
+
+    infer_parser = get_infer_parser()
+    subcommands.add_subcommand(name="infer", parser=infer_parser)
+
+    explain_parser = get_explain_parser()
+    subcommands.add_subcommand(name="explain", parser=explain_parser)
+
+    app_parser = get_app_parser()
+    subcommands.add_subcommand(name="app", parser=app_parser)
+
+    hta_parser = get_hta_parser()
+    subcommands.add_subcommand(name="hta", parser=hta_parser)
+
+    hpo_parser = get_hpo_parser()
+    subcommands.add_subcommand(name="hpo", parser=hpo_parser)
+
+    upload_parser = get_upload_parser()
+    subcommands.add_subcommand(name="upload", parser=upload_parser)
+
+    return (
+        parser,
+        train_parser,
+        test_parser,
+        infer_parser,
+        explain_parser,
+        app_parser,
+        hta_parser,
+        hpo_parser,
+        upload_parser,
+    )

common_ai-0.1.2/src/common_ai/dataset.py

@@ -0,0 +1,18 @@
+from abc import ABC, abstractmethod
+
+
+class MyDatasetAbstract(ABC):
+    def __init__(
+        self,
+        name: str,
+    ):
+        self.name = name
+
+    @abstractmethod
+    def __call__(self):
+        pass
+
+    @classmethod
+    @abstractmethod
+    def hpo(cls):
+        pass

common_ai-0.1.2/src/common_ai/early_stopping.py

@@ -0,0 +1,38 @@
+from typing import Optional
+
+
+class MyEarlyStopping:
+    def __init__(
+        self,
+        patience: Optional[int],
+        delta: float,
+        **kwargs,
+    ) -> None:
+        """Early stopping arguments.
+
+        Args:
+            patience: early stopping patience.
+            delta: early stopping loss improvement threshold.
+        """
+        self.patience = patience
+        self.delta = delta
+        self.remain_patience = patience
+        self.best_loss = float("inf")
+
+    def __call__(self, loss: float) -> bool:
+        if self.patience is None:
+            return False
+
+        if loss < self.best_loss - self.delta:
+            self.best_loss = loss
+            self.remain_patience = self.patience
+            return False
+
+        self.remain_patience -= 1
+        if self.remain_patience > 0:
+            return False
+
+        # reset internal states
+        self.remain_patience = self.patience
+        self.best_loss = float("inf")
+        return True

common_ai-0.1.2/src/common_ai/explain.yaml

@@ -0,0 +1,7 @@
+# shap:
+
+# inference:
+
+test: test.yaml
+
+# dataset:

common_ai-0.1.2/src/common_ai/generator.py

@@ -0,0 +1,42 @@
+import torch
+import numpy as np
+
+
+class MyGenerator:
+    def __init__(
+        self,
+        seed: int,
+        **kwargs,
+    ) -> None:
+        """Generator arguments.
+
+        Args:
+            seed: Random seed.
+        """
+        self.seed = seed
+        self.np_rng = np.random.default_rng(self.seed)
+        self.torch_c_rng = torch.Generator(device="cpu").manual_seed(self.seed)
+        if torch.cuda.is_available():
+            self.torch_g_rng = torch.Generator(device="cuda").manual_seed(self.seed)
+
+    def get_torch_generator_by_device(
+        self, device: str | torch.device
+    ) -> torch.Generator:
+        if device == "cpu" or device == torch.device("cpu"):
+            return self.torch_c_rng
+        return self.torch_g_rng
+
+    def state_dict(self) -> dict:
+        state_dict = {
+            "np_rng": self.np_rng.bit_generator.state,
+            "torch_c_rng": self.torch_c_rng.get_state(),
+        }
+        if torch.cuda.is_available():
+            state_dict.update({"torch_g_rng": self.torch_g_rng.get_state()})
+        return state_dict
+
+    def load_state_dict(self, state_dict: dict) -> None:
+        self.np_rng.bit_generator.state = state_dict["np_rng"]
+        self.torch_c_rng.set_state(state_dict["torch_c_rng"])
+        if torch.cuda.is_available():
+            self.torch_g_rng.set_state(state_dict["torch_g_rng"])

common_ai-0.1.2/src/common_ai/gradio_fn.py

@@ -0,0 +1,42 @@
+import importlib
+import pathlib
+import tempfile
+from abc import ABC, abstractmethod
+
+import jsonargparse
+
+from .test import MyTest
+from .inference import MyInferenceAbstract
+
+class MyGradioFnAbstract(ABC):
+    def __init__(
+        self,
+        app_cfg: jsonargparse.Namespace,
+        train_parser: jsonargparse.ArgumentParser,
+    ) -> None:
+        self.DEFAULT_TEMP_DIR = pathlib.Path(tempfile.gettempdir())
+        self.train_parser = train_parser
+        self.inference_dict = {}
+        for test_cfg, inference_cfg in zip(app_cfg.test, app_cfg.inference):
+            test_cfg = test_cfg.init_args
+            my_test = MyTest(**test_cfg.as_dict())
+            _, train_cfg = my_test.load_train_cfg(train_parser)
+            _, preprocess, _, model_cls = train_cfg.model.class_path.rsplit(".", 3)
+            data_name = train_cfg.dataset.init_args.name
+            repo_id = f"{preprocess}_{model_cls}_{data_name}"
+            self.inference_dict[repo_id] = (inference_cfg, test_cfg)
+
+    def load_inference(self, repo_id: str) -> MyInferenceAbstract:
+        assert repo_id in self.inference_dict, f"repo id {repo_id} is not found"
+        inference_cfg, test_cfg = self.inference_dict[repo_id]
+        inference_module, inference_cls = inference_cfg.class_path.rsplit(".", 1)
+        my_inference = getattr(
+            importlib.import_module(inference_module), inference_cls
+        )(**inference_cfg.init_args.as_dict())
+        my_inference.load_model(test_cfg, self.train_parser)
+
+        return my_inference
+
+    @abstractmethod
+    def launch(self):
+        pass