modelmark 0.1.0__tar.gz → 0.1.2__tar.gz

modelmark-0.1.2/PKG-INFO

@@ -0,0 +1,179 @@
+Metadata-Version: 2.4
+Name: modelmark
+Version: 0.1.2
+Summary: Benchmark framework for neural network models
+Author: Andrew Larin
+License: MIT
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=2.4
+Requires-Dist: pandas>=3.0
+Requires-Dist: torch>=2.12
+Requires-Dist: tqdm>=4.67
+Requires-Dist: playwright>=1.62
+Requires-Dist: rich>=15.0
+Requires-Dist: thop>=0.1
+Dynamic: license-file
+
+# ModelMark
+
+ModelMark is a CLI benchmarking tool for comparing neural-network models. It runs your models on one or more datasets, records performance and efficiency statistics, and generates an easy-to-embed HTML/PNG report.
+
+![Evaluation result](https://raw.githubusercontent.com/gloptim77/ModelMark/refs/heads/main/result.png)
+
+## Table of Contents
+
+- [About](#about)
+- [Requirements](#requirements)
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [Configuration](#configuration)
+- [How the Benchmark Works](#how-the-benchmark-works)
+- [Reported Metrics](#reported-metrics)
+- [Examples](#examples)
+- [Troubleshooting](#troubleshooting)
+
+## About
+
+For each benchmark run, ModelMark:
+
+1. Selects the next combination of dataset, input/output size, model, and seed.
+2. Seeds the random generators for reproducibility.
+3. Creates the data loader, model, and tester objects.
+4. Trains the model for the configured number of epochs and restores the checkpoint with the lowest validation loss.
+5. Records runtime and efficiency statistics.
+6. Evaluates the model on the dataset using the metrics from the configuration file.
+
+After all runs finish, ModelMark aggregates the results and generates a report containing testing results, training statistics, and your machine metadata.
+
+## Requirements
+
+- OS: Windows or Linux
+- Python: 3.12 or newer
+- Git: required for installation from GitHub
+
+## Installation
+
+Install ModelMark from PyPI:
+
+```bash
+pip install modelmark
+```
+
+## Quick Start
+
+1. Run initialization in an empty folder:
+
+   ```bash
+   modelmark init
+   ```
+
+   This creates two folders:
+   - `modelmark_files/` — configuration and log files
+   - `models/` — your model files
+
+2. Edit the configuration file:
+
+   `modelmark_files/config.py`
+
+   It contains three main configuration blocks:
+   - `model_config` — model hyperparameters such as number of layers, hidden dimension, kernel size, etc.
+   - `data_config` — dataset parameters such as file path, input/output features, train/val ratios, etc.
+   - `test_config` — testing options such as optimizer, loss criterion, metrics, learning rate, etc.
+
+3. Put the testing models to the `models/` folder.
+	
+	Make sure to import model's class definitions to the `config.py`.
+	
+	Configure `config.model_config` according to your task. 
+ 
+5. Download the dataset files (ETT by default):
+
+   ```bash
+   modelmark load
+   ```
+
+6. Run the benchmark:
+
+   ```bash
+   modelmark run
+   ```
+
+7. View the generated report:
+   - `result.html`
+   - `result.png`
+
+## Configuration
+
+You can adjust the configuration and add your own model files in the `models/` directory to match your benchmarking needs.
+
+Make sure your model implementation is compatible with the keys and settings used in your configuration.
+
+A detailed configuration example is available at:
+[src/modelmark/config.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/config.py)
+
+A detailed example model is available at:
+[src/modelmark/models/linear.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/models/linear.py)
+
+## How the Benchmark Works
+
+The benchmark consists of:
+
+```text
+F × O × M × S
+```
+
+where:
+
+| Symbol | Meaning | Example |
+|---|---|---|
+| `F` | Number of dataset files in the configuration | `{"ETTh1": ..., "Weather": ...}` → `F = 2` |
+| `O` | Number of input/output size configurations | `[32, 64, 128]` → `O = 3` |
+| `M` | Number of models | `{"Linear": ..., "LSTM": ...}` → `M = 2` |
+| `S` | Number of seeds | `[42, 43, 44]` → `S = 3` |
+
+ModelMark repeats the training/evaluation process for each combination and stores the mean result over `S` runs. Using more seeds generally makes the comparison more fair and statistically stable.
+
+## Reported Metrics
+
+The report includes statistics such as:
+
+| Metric | Description |
+|---|---|
+| Time | Average training time per epoch |
+| Params | Total number of model parameters |
+| GFLOPs | Average GFLOPs per batch |
+| Peak Memory | Maximum memory observed during a training iteration |
+
+The report also includes the evaluation metrics configured in `test_config` and your machine metadata.
+
+## Examples
+
+Run `modelmark init`, it will create config at `modelmark_files/config.py` and model's example folder at `models/` with Linear model file inside:
+
+- Configuration example: [src/modelmark/config.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/config.py)
+- Linear model example: [src/modelmark/models/linear.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/models/linear.py)
+
+## Troubleshooting
+
+If something does not work:
+
+1. Check the application log:
+   `modelmark_files/modelmark.log`
+2. Try restarting ModelMark:
+
+   ```bash
+   modelmark restart
+   ```
+
+3. If the issue persists, delete the `modelmark_files` folder and run:
+
+   ```bash
+   modelmark init
+   ```
+
+When opening an issue on GitHub, please include the relevant part of your log file.
+
+If you have questions or want to inspect the source code, see the
+[ModelMark GitHub repository](https://github.com/gloptim77/ModelMark/tree/main).

modelmark-0.1.2/README.md

@@ -0,0 +1,161 @@
+# ModelMark
+
+ModelMark is a CLI benchmarking tool for comparing neural-network models. It runs your models on one or more datasets, records performance and efficiency statistics, and generates an easy-to-embed HTML/PNG report.
+
+![Evaluation result](https://raw.githubusercontent.com/gloptim77/ModelMark/refs/heads/main/result.png)
+
+## Table of Contents
+
+- [About](#about)
+- [Requirements](#requirements)
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [Configuration](#configuration)
+- [How the Benchmark Works](#how-the-benchmark-works)
+- [Reported Metrics](#reported-metrics)
+- [Examples](#examples)
+- [Troubleshooting](#troubleshooting)
+
+## About
+
+For each benchmark run, ModelMark:
+
+1. Selects the next combination of dataset, input/output size, model, and seed.
+2. Seeds the random generators for reproducibility.
+3. Creates the data loader, model, and tester objects.
+4. Trains the model for the configured number of epochs and restores the checkpoint with the lowest validation loss.
+5. Records runtime and efficiency statistics.
+6. Evaluates the model on the dataset using the metrics from the configuration file.
+
+After all runs finish, ModelMark aggregates the results and generates a report containing testing results, training statistics, and your machine metadata.
+
+## Requirements
+
+- OS: Windows or Linux
+- Python: 3.12 or newer
+- Git: required for installation from GitHub
+
+## Installation
+
+Install ModelMark from PyPI:
+
+```bash
+pip install modelmark
+```
+
+## Quick Start
+
+1. Run initialization in an empty folder:
+
+   ```bash
+   modelmark init
+   ```
+
+   This creates two folders:
+   - `modelmark_files/` — configuration and log files
+   - `models/` — your model files
+
+2. Edit the configuration file:
+
+   `modelmark_files/config.py`
+
+   It contains three main configuration blocks:
+   - `model_config` — model hyperparameters such as number of layers, hidden dimension, kernel size, etc.
+   - `data_config` — dataset parameters such as file path, input/output features, train/val ratios, etc.
+   - `test_config` — testing options such as optimizer, loss criterion, metrics, learning rate, etc.
+
+3. Put the testing models to the `models/` folder.
+	
+	Make sure to import model's class definitions to the `config.py`.
+	
+	Configure `config.model_config` according to your task. 
+ 
+5. Download the dataset files (ETT by default):
+
+   ```bash
+   modelmark load
+   ```
+
+6. Run the benchmark:
+
+   ```bash
+   modelmark run
+   ```
+
+7. View the generated report:
+   - `result.html`
+   - `result.png`
+
+## Configuration
+
+You can adjust the configuration and add your own model files in the `models/` directory to match your benchmarking needs.
+
+Make sure your model implementation is compatible with the keys and settings used in your configuration.
+
+A detailed configuration example is available at:
+[src/modelmark/config.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/config.py)
+
+A detailed example model is available at:
+[src/modelmark/models/linear.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/models/linear.py)
+
+## How the Benchmark Works
+
+The benchmark consists of:
+
+```text
+F × O × M × S
+```
+
+where:
+
+| Symbol | Meaning | Example |
+|---|---|---|
+| `F` | Number of dataset files in the configuration | `{"ETTh1": ..., "Weather": ...}` → `F = 2` |
+| `O` | Number of input/output size configurations | `[32, 64, 128]` → `O = 3` |
+| `M` | Number of models | `{"Linear": ..., "LSTM": ...}` → `M = 2` |
+| `S` | Number of seeds | `[42, 43, 44]` → `S = 3` |
+
+ModelMark repeats the training/evaluation process for each combination and stores the mean result over `S` runs. Using more seeds generally makes the comparison more fair and statistically stable.
+
+## Reported Metrics
+
+The report includes statistics such as:
+
+| Metric | Description |
+|---|---|
+| Time | Average training time per epoch |
+| Params | Total number of model parameters |
+| GFLOPs | Average GFLOPs per batch |
+| Peak Memory | Maximum memory observed during a training iteration |
+
+The report also includes the evaluation metrics configured in `test_config` and your machine metadata.
+
+## Examples
+
+Run `modelmark init`, it will create config at `modelmark_files/config.py` and model's example folder at `models/` with Linear model file inside:
+
+- Configuration example: [src/modelmark/config.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/config.py)
+- Linear model example: [src/modelmark/models/linear.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/models/linear.py)
+
+## Troubleshooting
+
+If something does not work:
+
+1. Check the application log:
+   `modelmark_files/modelmark.log`
+2. Try restarting ModelMark:
+
+   ```bash
+   modelmark restart
+   ```
+
+3. If the issue persists, delete the `modelmark_files` folder and run:
+
+   ```bash
+   modelmark init
+   ```
+
+When opening an issue on GitHub, please include the relevant part of your log file.
+
+If you have questions or want to inspect the source code, see the
+[ModelMark GitHub repository](https://github.com/gloptim77/ModelMark/tree/main).

{modelmark-0.1.0 → modelmark-0.1.2}/pyproject.toml

@@ -7,7 +7,7 @@ modelmark = "modelmark.modelmark:run"
 
 [project]
 name = "modelmark"
-version = "0.1.0"
+version = "0.1.2"
 description = "Benchmark framework for neural network models"
 readme = "README.md"
 requires-python = ">=3.12"

{modelmark-0.1.0 → modelmark-0.1.2}/src/modelmark/common/dataset.py

@@ -15,23 +15,23 @@ class CausalDataset(Dataset):
 	Sliding-window dataset for ETT.
 
 	Returns:
-		x: [seq_len, num_features]
-		y: [pred_len, 1]
+		x: [input_len, num_features]
+		y: [output_len, 1]
 	"""
 
 	def __init__(
 		self,
 		data_config: dict,
-		seq_len: int,
-		pred_len: int,
+		input_len: int,
+		output_len: int,
 		split: str = "train",
 		mean: np.ndarray | None = None,
 		std: np.ndarray | None = None,
 	):
 		super().__init__()
 
-		self.seq_len = seq_len
-		self.pred_len = pred_len
+		self.input_len = input_len
+		self.output_len = output_len
 
 		# Extract config
 		data_path = config.data_path + data_config["path"]
@@ -54,12 +54,15 @@ class CausalDataset(Dataset):
 		train_end = int(n * train_ratio)
 		val_end = int(n * (train_ratio + val_ratio))
 
+		# Train can be random
 		if split == "train":
 			start = 0
 			end = train_end
+		# Val never intersects with train
 		elif split == "val":
 			start = train_end
 			end = val_end
+		# Test never intersects with train or val
 		elif split == "test":
 			start = val_end
 			end = n
@@ -89,23 +92,24 @@ class CausalDataset(Dataset):
 		self.data = data[start:end]
 
 		# Number of valid windows
-		self.length = len(self.data) - seq_len - pred_len + 1
+		self.length = len(self.data) - input_len - output_len + 1
 
 		if self.length <= 0:
 			raise ValueError(
 				f"Split '{split}' is too short for "
-				f"seq_len={seq_len}, pred_len={pred_len}"
+				f"input_len={input_len}, output_len={output_len}"
 			)
 
 	def __len__(self):
 		return self.length
 
 	def __getitem__(self, idx):
+
 		x_start = idx
-		x_end = x_start + self.seq_len
+		x_end = x_start + self.input_len
 
 		y_start = x_end
-		y_end = y_start + self.pred_len
+		y_end = y_start + self.output_len
 
 		x = self.data[x_start:x_end, self.input_indices]
 		y = self.data[y_start:y_end, self.output_indices]

{modelmark-0.1.0 → modelmark-0.1.2}/src/modelmark/common/loader.py

@@ -10,29 +10,29 @@ config = load_config()
 
 class Loader:
 
-	def __init__(self, file_config : dict, context_size : int):
+	def __init__(self, file_config : dict, input_len : int, output_len : int):
 
-		self.get_dataloaders(file_config, context_size) 
+		self.get_dataloaders(file_config, input_len, output_len) 
 		logger.debug(f"Dataset len stats: train={len(self.train_loader)} val={len(self.val_loader)} test={len(self.test_loader)}")
 
-	def get_dataloaders(self, file_config : dict, context_size : int) -> tuple[DataLoader, DataLoader, DataLoader]:
+	def get_dataloaders(self, file_config : dict, input_len : int, output_len : int) -> tuple[DataLoader, DataLoader, DataLoader]:
 		"""Load the data, pack to datasets, create the loaders and return them"""
 	
 		train_ds = CausalDataset(data_config = file_config,
-								seq_len = context_size, 
-								pred_len = context_size, 
+								input_len = input_len, 
+								output_len = output_len, 
 								split = "train")
 
 		val_ds 	= CausalDataset(data_config = file_config, 
-								seq_len = context_size, 
-								pred_len = context_size, 
+								input_len = input_len, 
+								output_len = output_len, 
 								split = "val", 
 								mean = train_ds.mean, 
 								std = train_ds.std)
 
 		test_ds = CausalDataset(data_config = file_config, 
-								seq_len = context_size, 
-								pred_len = context_size, 
+								input_len = input_len, 
+								output_len = output_len, 
 								split = "test", 
 								mean = train_ds.mean, 
 								std = train_ds.std)

{modelmark-0.1.0 → modelmark-0.1.2}/src/modelmark/common/parser.py

@@ -21,22 +21,24 @@ class Parser:
 			formatter_class=argparse.RawTextHelpFormatter,
 			description = constants.PARSER_DESC
 		)
+
+		# Create a subparser container
+		subparsers = self.parser.add_subparsers(dest="command", required=True)
 	
-		# Add arguments
-		self.parser.add_argument(
-			"-t", "--task",
-			type = str,
-			default = "help",
-			help = constants.PARSER_HELP
-		)
+		# Define your action words (subcommands)
+		subparsers.add_parser("init", 	help = "Test init, create the modelmark_files/, customizable config.py file and models/ with examples.")
+		subparsers.add_parser("load",	help = "Dataset download, create the data/ folder and download ETT dataset there.")				
+		subparsers.add_parser("run", 	help = "Run the test.")
+		subparsers.add_parser("clear", 	help = "Remove the modelmark_files folder.")
+		subparsers.add_parser("reset", 	help = "Runs clean and init.")
+		subparsers.add_parser("help", 	help = "Print the info above.")
+	
+		self.args = self.parser.parse_args()
 	
 		# Run without args / help request - print help
 		if len(sys.argv) == 1:
 			self.parser.print_help()
 			sys.exit(1)
-		
-		# Parse argument values
-		self.args = self.parser.parse_args()
 
 		# Setup other
 		pd.set_option('display.colheader_justify', 'center')	
@@ -45,38 +47,38 @@ class Parser:
 		"""Select the task and run"""
 
 		# Config initialization #
-		if self.args.task == "init":
+		if self.args.command == "init":
 			self.init_config()
 			return 0
 
 		# Dataset download #
-		if self.args.task == "load":
+		if self.args.command == "load":
 			download_dataset("ett")
 			return 0
 
 		# Run the test #
-		if self.args.task == "run":
+		if self.args.command == "run":
 			return None
 
 		# Clean the config folder #
-		if self.args.task == "clear":
+		if self.args.command == "clear":
 			self.clear_config()
 			return 0
 
 		# Reset the modelmark config #
-		if self.args.task == "reset":
+		if self.args.command == "reset":
 			self.reset_config()
 			return 0
 
 		# Print the modelmark usage #
-		if self.args.task == "help":
+		if self.args.command == "help":
 			self.parser.print_help()
 			return 0
 		
 		# Check if task is correct #
-		if self.args.task != "run":
-			logger.error(f"Unknown task: {self.args.task}, run 'modelmark' for more info")
-			console.print(f"Unknown task: {self.args.task}, run 'modelmark' for more info", style="red")
+		if self.args.command != "run":
+			logger.error(f"Unknown command: {self.args.command}, run 'modelmark help' for more info")
+			console.print(f"Unknown command: {self.args.command}, run 'modelmark help' for more info", style="red")
 			return 1
 
 		# Otherwise return error

{modelmark-0.1.0 → modelmark-0.1.2}/src/modelmark/config.py

@@ -9,6 +9,7 @@ import torch.nn as nn
 from modelmark.models.gru import GRUModel
 from modelmark.models.conv import ConvModel
 from modelmark.models.lstm import LSTMModel
+from modelmark.models.transformer import TransformerModel
 from modelmark.models.linear import Linear
 # Either custom (uncomment)
 #from models.linear import Linear
@@ -85,7 +86,8 @@ test_config = {
 		"MAE": nn.L1Loss(),
 	},
 	"seeds": [1, 2, 3],
-	"contexts": [96, 192, 336, 720],
+	"input_len": [96, 96, 96],		# len should be equal to output_len
+	"output_len": [96, 192, 336],	# len should be equal to input_len
 	}
 
 # Other

{modelmark-0.1.0 → modelmark-0.1.2}/src/modelmark/constants.py

@@ -24,11 +24,4 @@ USER_MODEL_PATH = USER_DIR / "models" / "linear.py"
 PARSER_DESC = """MODELMARK
 
 	Python tool to measure the performance of a custom neural network model
-	and compare it to other popular architectures."""
-PARSER_HELP = """\nModelmark available TASKs:
-init - Test init, create the modelmark_files/, customizable config.py file and models/ with examples.
-load - Dataset download, create the data/ folder and download ETT dataset there.
-run - Run the test.
-clear - Remove the modelmark_files folder.
-reset - Runs clean and init.
-"""
+	and compare it to other popular architectures."""