modelmark 0.1.1__tar.gz → 0.1.2__tar.gz

modelmark-0.1.2/PKG-INFO

@@ -0,0 +1,179 @@
+Metadata-Version: 2.4
+Name: modelmark
+Version: 0.1.2
+Summary: Benchmark framework for neural network models
+Author: Andrew Larin
+License: MIT
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=2.4
+Requires-Dist: pandas>=3.0
+Requires-Dist: torch>=2.12
+Requires-Dist: tqdm>=4.67
+Requires-Dist: playwright>=1.62
+Requires-Dist: rich>=15.0
+Requires-Dist: thop>=0.1
+Dynamic: license-file
+
+# ModelMark
+
+ModelMark is a CLI benchmarking tool for comparing neural-network models. It runs your models on one or more datasets, records performance and efficiency statistics, and generates an easy-to-embed HTML/PNG report.
+
+![Evaluation result](https://raw.githubusercontent.com/gloptim77/ModelMark/refs/heads/main/result.png)
+
+## Table of Contents
+
+- [About](#about)
+- [Requirements](#requirements)
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [Configuration](#configuration)
+- [How the Benchmark Works](#how-the-benchmark-works)
+- [Reported Metrics](#reported-metrics)
+- [Examples](#examples)
+- [Troubleshooting](#troubleshooting)
+
+## About
+
+For each benchmark run, ModelMark:
+
+1. Selects the next combination of dataset, input/output size, model, and seed.
+2. Seeds the random generators for reproducibility.
+3. Creates the data loader, model, and tester objects.
+4. Trains the model for the configured number of epochs and restores the checkpoint with the lowest validation loss.
+5. Records runtime and efficiency statistics.
+6. Evaluates the model on the dataset using the metrics from the configuration file.
+
+After all runs finish, ModelMark aggregates the results and generates a report containing testing results, training statistics, and your machine metadata.
+
+## Requirements
+
+- OS: Windows or Linux
+- Python: 3.12 or newer
+- Git: required for installation from GitHub
+
+## Installation
+
+Install ModelMark from PyPI:
+
+```bash
+pip install modelmark
+```
+
+## Quick Start
+
+1. Run initialization in an empty folder:
+
+   ```bash
+   modelmark init
+   ```
+
+   This creates two folders:
+   - `modelmark_files/` — configuration and log files
+   - `models/` — your model files
+
+2. Edit the configuration file:
+
+   `modelmark_files/config.py`
+
+   It contains three main configuration blocks:
+   - `model_config` — model hyperparameters such as number of layers, hidden dimension, kernel size, etc.
+   - `data_config` — dataset parameters such as file path, input/output features, train/val ratios, etc.
+   - `test_config` — testing options such as optimizer, loss criterion, metrics, learning rate, etc.
+
+3. Put the testing models to the `models/` folder.
+	
+	Make sure to import model's class definitions to the `config.py`.
+	
+	Configure `config.model_config` according to your task. 
+ 
+5. Download the dataset files (ETT by default):
+
+   ```bash
+   modelmark load
+   ```
+
+6. Run the benchmark:
+
+   ```bash
+   modelmark run
+   ```
+
+7. View the generated report:
+   - `result.html`
+   - `result.png`
+
+## Configuration
+
+You can adjust the configuration and add your own model files in the `models/` directory to match your benchmarking needs.
+
+Make sure your model implementation is compatible with the keys and settings used in your configuration.
+
+A detailed configuration example is available at:
+[src/modelmark/config.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/config.py)
+
+A detailed example model is available at:
+[src/modelmark/models/linear.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/models/linear.py)
+
+## How the Benchmark Works
+
+The benchmark consists of:
+
+```text
+F × O × M × S
+```
+
+where:
+
+| Symbol | Meaning | Example |
+|---|---|---|
+| `F` | Number of dataset files in the configuration | `{"ETTh1": ..., "Weather": ...}` → `F = 2` |
+| `O` | Number of input/output size configurations | `[32, 64, 128]` → `O = 3` |
+| `M` | Number of models | `{"Linear": ..., "LSTM": ...}` → `M = 2` |
+| `S` | Number of seeds | `[42, 43, 44]` → `S = 3` |
+
+ModelMark repeats the training/evaluation process for each combination and stores the mean result over `S` runs. Using more seeds generally makes the comparison more fair and statistically stable.
+
+## Reported Metrics
+
+The report includes statistics such as:
+
+| Metric | Description |
+|---|---|
+| Time | Average training time per epoch |
+| Params | Total number of model parameters |
+| GFLOPs | Average GFLOPs per batch |
+| Peak Memory | Maximum memory observed during a training iteration |
+
+The report also includes the evaluation metrics configured in `test_config` and your machine metadata.
+
+## Examples
+
+Run `modelmark init`, it will create config at `modelmark_files/config.py` and model's example folder at `models/` with Linear model file inside:
+
+- Configuration example: [src/modelmark/config.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/config.py)
+- Linear model example: [src/modelmark/models/linear.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/models/linear.py)
+
+## Troubleshooting
+
+If something does not work:
+
+1. Check the application log:
+   `modelmark_files/modelmark.log`
+2. Try restarting ModelMark:
+
+   ```bash
+   modelmark restart
+   ```
+
+3. If the issue persists, delete the `modelmark_files` folder and run:
+
+   ```bash
+   modelmark init
+   ```
+
+When opening an issue on GitHub, please include the relevant part of your log file.
+
+If you have questions or want to inspect the source code, see the
+[ModelMark GitHub repository](https://github.com/gloptim77/ModelMark/tree/main).

modelmark-0.1.2/README.md

@@ -0,0 +1,161 @@
+# ModelMark
+
+ModelMark is a CLI benchmarking tool for comparing neural-network models. It runs your models on one or more datasets, records performance and efficiency statistics, and generates an easy-to-embed HTML/PNG report.
+
+![Evaluation result](https://raw.githubusercontent.com/gloptim77/ModelMark/refs/heads/main/result.png)
+
+## Table of Contents
+
+- [About](#about)
+- [Requirements](#requirements)
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [Configuration](#configuration)
+- [How the Benchmark Works](#how-the-benchmark-works)
+- [Reported Metrics](#reported-metrics)
+- [Examples](#examples)
+- [Troubleshooting](#troubleshooting)
+
+## About
+
+For each benchmark run, ModelMark:
+
+1. Selects the next combination of dataset, input/output size, model, and seed.
+2. Seeds the random generators for reproducibility.
+3. Creates the data loader, model, and tester objects.
+4. Trains the model for the configured number of epochs and restores the checkpoint with the lowest validation loss.
+5. Records runtime and efficiency statistics.
+6. Evaluates the model on the dataset using the metrics from the configuration file.
+
+After all runs finish, ModelMark aggregates the results and generates a report containing testing results, training statistics, and your machine metadata.
+
+## Requirements
+
+- OS: Windows or Linux
+- Python: 3.12 or newer
+- Git: required for installation from GitHub
+
+## Installation
+
+Install ModelMark from PyPI:
+
+```bash
+pip install modelmark
+```
+
+## Quick Start
+
+1. Run initialization in an empty folder:
+
+   ```bash
+   modelmark init
+   ```
+
+   This creates two folders:
+   - `modelmark_files/` — configuration and log files
+   - `models/` — your model files
+
+2. Edit the configuration file:
+
+   `modelmark_files/config.py`
+
+   It contains three main configuration blocks:
+   - `model_config` — model hyperparameters such as number of layers, hidden dimension, kernel size, etc.
+   - `data_config` — dataset parameters such as file path, input/output features, train/val ratios, etc.
+   - `test_config` — testing options such as optimizer, loss criterion, metrics, learning rate, etc.
+
+3. Put the testing models to the `models/` folder.
+	
+	Make sure to import model's class definitions to the `config.py`.
+	
+	Configure `config.model_config` according to your task. 
+ 
+5. Download the dataset files (ETT by default):
+
+   ```bash
+   modelmark load
+   ```
+
+6. Run the benchmark:
+
+   ```bash
+   modelmark run
+   ```
+
+7. View the generated report:
+   - `result.html`
+   - `result.png`
+
+## Configuration
+
+You can adjust the configuration and add your own model files in the `models/` directory to match your benchmarking needs.
+
+Make sure your model implementation is compatible with the keys and settings used in your configuration.
+
+A detailed configuration example is available at:
+[src/modelmark/config.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/config.py)
+
+A detailed example model is available at:
+[src/modelmark/models/linear.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/models/linear.py)
+
+## How the Benchmark Works
+
+The benchmark consists of:
+
+```text
+F × O × M × S
+```
+
+where:
+
+| Symbol | Meaning | Example |
+|---|---|---|
+| `F` | Number of dataset files in the configuration | `{"ETTh1": ..., "Weather": ...}` → `F = 2` |
+| `O` | Number of input/output size configurations | `[32, 64, 128]` → `O = 3` |
+| `M` | Number of models | `{"Linear": ..., "LSTM": ...}` → `M = 2` |
+| `S` | Number of seeds | `[42, 43, 44]` → `S = 3` |
+
+ModelMark repeats the training/evaluation process for each combination and stores the mean result over `S` runs. Using more seeds generally makes the comparison more fair and statistically stable.
+
+## Reported Metrics
+
+The report includes statistics such as:
+
+| Metric | Description |
+|---|---|
+| Time | Average training time per epoch |
+| Params | Total number of model parameters |
+| GFLOPs | Average GFLOPs per batch |
+| Peak Memory | Maximum memory observed during a training iteration |
+
+The report also includes the evaluation metrics configured in `test_config` and your machine metadata.
+
+## Examples
+
+Run `modelmark init`, it will create config at `modelmark_files/config.py` and model's example folder at `models/` with Linear model file inside:
+
+- Configuration example: [src/modelmark/config.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/config.py)
+- Linear model example: [src/modelmark/models/linear.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/models/linear.py)
+
+## Troubleshooting
+
+If something does not work:
+
+1. Check the application log:
+   `modelmark_files/modelmark.log`
+2. Try restarting ModelMark:
+
+   ```bash
+   modelmark restart
+   ```
+
+3. If the issue persists, delete the `modelmark_files` folder and run:
+
+   ```bash
+   modelmark init
+   ```
+
+When opening an issue on GitHub, please include the relevant part of your log file.
+
+If you have questions or want to inspect the source code, see the
+[ModelMark GitHub repository](https://github.com/gloptim77/ModelMark/tree/main).

{modelmark-0.1.1 → modelmark-0.1.2}/pyproject.toml

@@ -7,7 +7,7 @@ modelmark = "modelmark.modelmark:run"
 
 [project]
 name = "modelmark"
-version = "0.1.1"
+version = "0.1.2"
 description = "Benchmark framework for neural network models"
 readme = "README.md"
 requires-python = ">=3.12"

{modelmark-0.1.1 → modelmark-0.1.2}/src/modelmark/common/parser.py

@@ -21,22 +21,24 @@ class Parser:
 			formatter_class=argparse.RawTextHelpFormatter,
 			description = constants.PARSER_DESC
 		)
+
+		# Create a subparser container
+		subparsers = self.parser.add_subparsers(dest="command", required=True)
 	
-		# Add arguments
-		self.parser.add_argument(
-			"-t", "--task",
-			type = str,
-			default = "help",
-			help = constants.PARSER_HELP
-		)
+		# Define your action words (subcommands)
+		subparsers.add_parser("init", 	help = "Test init, create the modelmark_files/, customizable config.py file and models/ with examples.")
+		subparsers.add_parser("load",	help = "Dataset download, create the data/ folder and download ETT dataset there.")				
+		subparsers.add_parser("run", 	help = "Run the test.")
+		subparsers.add_parser("clear", 	help = "Remove the modelmark_files folder.")
+		subparsers.add_parser("reset", 	help = "Runs clean and init.")
+		subparsers.add_parser("help", 	help = "Print the info above.")
+	
+		self.args = self.parser.parse_args()
 	
 		# Run without args / help request - print help
 		if len(sys.argv) == 1:
 			self.parser.print_help()
 			sys.exit(1)
-		
-		# Parse argument values
-		self.args = self.parser.parse_args()
 
 		# Setup other
 		pd.set_option('display.colheader_justify', 'center')	
@@ -45,38 +47,38 @@ class Parser:
 		"""Select the task and run"""
 
 		# Config initialization #
-		if self.args.task == "init":
+		if self.args.command == "init":
 			self.init_config()
 			return 0
 
 		# Dataset download #
-		if self.args.task == "load":
+		if self.args.command == "load":
 			download_dataset("ett")
 			return 0
 
 		# Run the test #
-		if self.args.task == "run":
+		if self.args.command == "run":
 			return None
 
 		# Clean the config folder #
-		if self.args.task == "clear":
+		if self.args.command == "clear":
 			self.clear_config()
 			return 0
 
 		# Reset the modelmark config #
-		if self.args.task == "reset":
+		if self.args.command == "reset":
 			self.reset_config()
 			return 0
 
 		# Print the modelmark usage #
-		if self.args.task == "help":
+		if self.args.command == "help":
 			self.parser.print_help()
 			return 0
 		
 		# Check if task is correct #
-		if self.args.task != "run":
-			logger.error(f"Unknown task: {self.args.task}, run 'modelmark' for more info")
-			console.print(f"Unknown task: {self.args.task}, run 'modelmark' for more info", style="red")
+		if self.args.command != "run":
+			logger.error(f"Unknown command: {self.args.command}, run 'modelmark help' for more info")
+			console.print(f"Unknown command: {self.args.command}, run 'modelmark help' for more info", style="red")
 			return 1
 
 		# Otherwise return error

{modelmark-0.1.1 → modelmark-0.1.2}/src/modelmark/constants.py

@@ -24,11 +24,4 @@ USER_MODEL_PATH = USER_DIR / "models" / "linear.py"
 PARSER_DESC = """MODELMARK
 
 	Python tool to measure the performance of a custom neural network model
-	and compare it to other popular architectures."""
-PARSER_HELP = """\nModelmark available TASKs:
-init - Test init, create the modelmark_files/, customizable config.py file and models/ with examples.
-load - Dataset download, create the data/ folder and download ETT dataset there.
-run - Run the test.
-clear - Remove the modelmark_files folder.
-reset - Runs clean and init.
-"""
+	and compare it to other popular architectures."""

{modelmark-0.1.1 → modelmark-0.1.2}/src/modelmark/modelmark.py

@@ -4,10 +4,10 @@ ModelMark
 	This tool will help you to test NN models against each other,
 	and form a detailed report that is easy to embed to a website.   
 
-	It can download dataset with
-		modelmark -t load
+	It can download ETT dataset with
+		modelmark load
 	It can run the test with 
-		modelmark -t run
+		modelmark run
 
 	Test consists of F * O * M * S steps, where:
 		F - number of dataset files in the config (e.g. ["ETTh1" : ..., "Weather" : ...] means F = 2)
@@ -31,9 +31,9 @@ ModelMark
 
 	About training stats:
 		
-		Time - average time per epoch 
-		Params - total number of model params
-		GFLOPs - average per batch
+		Time 		- average time per epoch 
+		Params 		- total number of model params
+		GFLOPs 		- average per batch
 		Peak Memory - max per training iteration
 
 	---	

{modelmark-0.1.1 → modelmark-0.1.2}/src/modelmark/models/conv.py

@@ -11,8 +11,8 @@ class ConvModel(nn.Module):
 		# To avoid circular import, now it lives here
 		config = load_config()
 
-		self.input_size = input_dim
-		self.output_size = output_dim
+		self.input_dim = input_dim
+		self.output_dim = output_dim
 		self.input_len = input_len
 		self.output_len = output_len
 		
@@ -22,7 +22,7 @@ class ConvModel(nn.Module):
 		
 		layers = []
 
-		in_channels = self.input_size
+		in_channels = self.input_dim
 
 		for _ in range(self.num_layers):
 			layers.append(
@@ -42,8 +42,8 @@ class ConvModel(nn.Module):
 		# Collapse the sequence dimension to one feature vector
 		self.pool = nn.AdaptiveAvgPool1d(1)
 
-		# Produce exactly context_size * output_size values
-		self.fc = nn.Linear(self.hidden_size, output_len  * self.output_size)
+		# Produce exactly context_size * output_dim values
+		self.fc = nn.Linear(self.hidden_size, output_len  * self.output_dim)
 
 	def forward(self, x):
 		"""
@@ -62,9 +62,9 @@ class ConvModel(nn.Module):
 		x = self.pool(x)
 		# [batch, hidden_size]
 		x = x.squeeze(-1)
-		# [batch, context_size * output_size]
+		# [batch, context_size * output_dim]
 		x = self.fc(x)
 		# Project to output dim and len
-		x = x.view(x.size(0), self.output_len, self.output_size)
+		x = x.view(x.size(0), self.output_len, self.output_dim)
 
 		return x

{modelmark-0.1.1 → modelmark-0.1.2}/src/modelmark/models/gru.py

@@ -11,8 +11,8 @@ class GRUModel(nn.Module):
 		# To avoid circular import, now it lives here
 		config = load_config()
 
-		self.input_size = input_dim
-		self.output_size = output_dim
+		self.input_dim = input_dim
+		self.output_dim = output_dim
 		self.input_len = input_len
 		self.output_len = output_len
 		
@@ -20,13 +20,13 @@ class GRUModel(nn.Module):
 		self.num_layers = config.model_config["GRU"]["num_layers"]
 
 		self.gru = nn.GRU(
-			input_size = self.input_size,
+			input_size = self.input_dim,
 			hidden_size = self.hidden_size,
 			num_layers = self.num_layers,
 			batch_first = True,
 		)
 
-		self.out = nn.Linear(self.hidden_size, self.output_len * self.output_size)
+		self.out = nn.Linear(self.hidden_size, self.output_len * self.output_dim)
 
 	def forward(self, x):
 		"""
@@ -44,6 +44,6 @@ class GRUModel(nn.Module):
 
 		x = self.out(h)
 		# Project to output dim and len
-		x = x.view(x.size(0), self.output_len, self.output_size)
+		x = x.view(x.size(0), self.output_len, self.output_dim)
 
 		return x

{modelmark-0.1.1 → modelmark-0.1.2}/src/modelmark/models/linear.py

@@ -38,8 +38,8 @@ class Linear(nn.Module):
 		# It lives here to avoid circular config
 		config = load_config()
 
-		self.input_size = input_dim
-		self.output_size = output_dim
+		self.input_dim = input_dim
+		self.output_dim = output_dim
 		self.input_len = input_len
 		self.output_len = output_len
 
@@ -47,7 +47,7 @@ class Linear(nn.Module):
 		self.num_layers = config.model_config["Linear"]["num_layers"]
 
 		layers = [
-			nn.Linear(self.input_size, self.hidden_size),
+			nn.Linear(self.input_dim, self.hidden_size),
 			nn.GELU(),
 		]
 
@@ -58,7 +58,7 @@ class Linear(nn.Module):
 			])
 
 		self.net = nn.Sequential(*layers)
-		self.out = nn.Linear(self.hidden_size, self.output_size * self.output_len)
+		self.out = nn.Linear(self.hidden_size, self.output_dim * self.output_len)
 
 	def forward(self, x):
 		"""
@@ -75,6 +75,6 @@ class Linear(nn.Module):
 		x = x.mean(1)
 		x = x.view(x.size(0), -1)
 		x = self.out(x)
-		x = x.view(x.size(0), self.output_len, self.output_size)
+		x = x.view(x.size(0), self.output_len, self.output_dim)
 
 		return x

{modelmark-0.1.1 → modelmark-0.1.2}/src/modelmark/models/lstm.py

@@ -11,8 +11,8 @@ class LSTMModel(nn.Module):
 		# To avoid circular import, now it lives here
 		config = load_config()
 
-		self.input_size = input_dim
-		self.output_size = output_dim
+		self.input_dim = input_dim
+		self.output_dim = output_dim
 		self.input_len = input_len
 		self.output_len = output_len
 		self.hidden_size = config.model_config["LSTM"]["hidden_size"]
@@ -20,13 +20,13 @@ class LSTMModel(nn.Module):
 		
 		# Change nn.GRU to nn.LSTM
 		self.lstm = nn.LSTM(
-			input_size=self.input_size,
-			hidden_size=self.hidden_size,
-			num_layers=self.num_layers,
-			batch_first=True,
+			input_size = self.input_dim,
+			hidden_size = self.hidden_size,
+			num_layers = self.num_layers,
+			batch_first = True,
 		)
 
-		self.out = nn.Linear(self.hidden_size, self.output_len * self.output_size)
+		self.out = nn.Linear(self.hidden_size, self.output_len * self.output_dim)
 
 	def forward(self, x):
 		"""
@@ -44,6 +44,6 @@ class LSTMModel(nn.Module):
 		
 		x = self.out(h)
 		# Project to output dim and len
-		x = x.view(x.size(0), self.output_len, self.output_size)
+		x = x.view(x.size(0), self.output_len, self.output_dim)
 		
 		return x

{modelmark-0.1.1 → modelmark-0.1.2}/src/modelmark/models/transformer.py

@@ -11,8 +11,8 @@ class TransformerModel(nn.Module):
 		# To avoid circular import, now it lives here
 		config = load_config()
 
-		self.input_size = input_dim
-		self.output_size = output_dim
+		self.input_dim = input_dim
+		self.output_dim = output_dim
 		self.input_len = input_len
 		self.output_len = output_len
 		self.max_seq_len = input_len
@@ -23,24 +23,24 @@ class TransformerModel(nn.Module):
 		self.dropout = config.model_config["Transformer"]["dropout"]
 
 		# Input projection
-		self.input_proj = nn.Linear(self.input_size, self.hidden_size,)
+		self.input_proj = nn.Linear(self.input_dim, self.hidden_size,)
 
 		# Learnable positional embedding
 		self.pos_embedding = nn.Parameter(torch.zeros(1, self.max_seq_len, self.hidden_size))
 
 		# Transformer decoder layer
 		decoder_layer = nn.TransformerDecoderLayer(
-			d_model=self.hidden_size,
-			nhead=self.num_heads,
-			dim_feedforward=4 * self.hidden_size,
-			dropout=self.dropout,
-			batch_first=True,
+			d_model 		= self.hidden_size,
+			nhead 			= self.num_heads,
+			dim_feedforward = 4 * self.hidden_size,
+			dropout 		= self.dropout,
+			batch_first 	= True,
 		)
 
-		self.decoder = nn.TransformerDecoder(decoder_layer, num_layers=self.num_layers)
+		self.decoder = nn.TransformerDecoder(decoder_layer, num_layers = self.num_layers)
 
 		# Map hidden representation to outputs
-		self.out = nn.Linear(self.hidden_size, self.output_size * self.output_len)
+		self.out = nn.Linear(self.hidden_size, self.output_dim * self.output_len)
 
 	def forward(self, x):
 		"""
@@ -82,6 +82,6 @@ class TransformerModel(nn.Module):
 		x = x.mean(1)
 		x = x.view(x.size(0), -1)
 		x = self.out(x)
-		x = x.view(x.size(0), self.output_len, self.output_size)
+		x = x.view(x.size(0), self.output_len, self.output_dim)
 
 		return x

modelmark-0.1.2/src/modelmark.egg-info/PKG-INFO

@@ -0,0 +1,179 @@
+Metadata-Version: 2.4
+Name: modelmark
+Version: 0.1.2
+Summary: Benchmark framework for neural network models
+Author: Andrew Larin
+License: MIT
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=2.4
+Requires-Dist: pandas>=3.0
+Requires-Dist: torch>=2.12
+Requires-Dist: tqdm>=4.67
+Requires-Dist: playwright>=1.62
+Requires-Dist: rich>=15.0
+Requires-Dist: thop>=0.1
+Dynamic: license-file
+
+# ModelMark
+
+ModelMark is a CLI benchmarking tool for comparing neural-network models. It runs your models on one or more datasets, records performance and efficiency statistics, and generates an easy-to-embed HTML/PNG report.
+
+![Evaluation result](https://raw.githubusercontent.com/gloptim77/ModelMark/refs/heads/main/result.png)
+
+## Table of Contents
+
+- [About](#about)
+- [Requirements](#requirements)
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [Configuration](#configuration)
+- [How the Benchmark Works](#how-the-benchmark-works)
+- [Reported Metrics](#reported-metrics)
+- [Examples](#examples)
+- [Troubleshooting](#troubleshooting)
+
+## About
+
+For each benchmark run, ModelMark:
+
+1. Selects the next combination of dataset, input/output size, model, and seed.
+2. Seeds the random generators for reproducibility.
+3. Creates the data loader, model, and tester objects.
+4. Trains the model for the configured number of epochs and restores the checkpoint with the lowest validation loss.
+5. Records runtime and efficiency statistics.
+6. Evaluates the model on the dataset using the metrics from the configuration file.
+
+After all runs finish, ModelMark aggregates the results and generates a report containing testing results, training statistics, and your machine metadata.
+
+## Requirements
+
+- OS: Windows or Linux
+- Python: 3.12 or newer
+- Git: required for installation from GitHub
+
+## Installation
+
+Install ModelMark from PyPI:
+
+```bash
+pip install modelmark
+```
+
+## Quick Start
+
+1. Run initialization in an empty folder:
+
+   ```bash
+   modelmark init
+   ```
+
+   This creates two folders:
+   - `modelmark_files/` — configuration and log files
+   - `models/` — your model files
+
+2. Edit the configuration file:
+
+   `modelmark_files/config.py`
+
+   It contains three main configuration blocks:
+   - `model_config` — model hyperparameters such as number of layers, hidden dimension, kernel size, etc.
+   - `data_config` — dataset parameters such as file path, input/output features, train/val ratios, etc.
+   - `test_config` — testing options such as optimizer, loss criterion, metrics, learning rate, etc.
+
+3. Put the testing models to the `models/` folder.
+	
+	Make sure to import model's class definitions to the `config.py`.
+	
+	Configure `config.model_config` according to your task. 
+ 
+5. Download the dataset files (ETT by default):
+
+   ```bash
+   modelmark load
+   ```
+
+6. Run the benchmark:
+
+   ```bash
+   modelmark run
+   ```
+
+7. View the generated report:
+   - `result.html`
+   - `result.png`
+
+## Configuration
+
+You can adjust the configuration and add your own model files in the `models/` directory to match your benchmarking needs.
+
+Make sure your model implementation is compatible with the keys and settings used in your configuration.
+
+A detailed configuration example is available at:
+[src/modelmark/config.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/config.py)
+
+A detailed example model is available at:
+[src/modelmark/models/linear.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/models/linear.py)
+
+## How the Benchmark Works
+
+The benchmark consists of:
+
+```text
+F × O × M × S
+```
+
+where:
+
+| Symbol | Meaning | Example |
+|---|---|---|
+| `F` | Number of dataset files in the configuration | `{"ETTh1": ..., "Weather": ...}` → `F = 2` |
+| `O` | Number of input/output size configurations | `[32, 64, 128]` → `O = 3` |
+| `M` | Number of models | `{"Linear": ..., "LSTM": ...}` → `M = 2` |
+| `S` | Number of seeds | `[42, 43, 44]` → `S = 3` |
+
+ModelMark repeats the training/evaluation process for each combination and stores the mean result over `S` runs. Using more seeds generally makes the comparison more fair and statistically stable.
+
+## Reported Metrics
+
+The report includes statistics such as:
+
+| Metric | Description |
+|---|---|
+| Time | Average training time per epoch |
+| Params | Total number of model parameters |
+| GFLOPs | Average GFLOPs per batch |
+| Peak Memory | Maximum memory observed during a training iteration |
+
+The report also includes the evaluation metrics configured in `test_config` and your machine metadata.
+
+## Examples
+
+Run `modelmark init`, it will create config at `modelmark_files/config.py` and model's example folder at `models/` with Linear model file inside:
+
+- Configuration example: [src/modelmark/config.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/config.py)
+- Linear model example: [src/modelmark/models/linear.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/models/linear.py)
+
+## Troubleshooting
+
+If something does not work:
+
+1. Check the application log:
+   `modelmark_files/modelmark.log`
+2. Try restarting ModelMark:
+
+   ```bash
+   modelmark restart
+   ```
+
+3. If the issue persists, delete the `modelmark_files` folder and run:
+
+   ```bash
+   modelmark init
+   ```
+
+When opening an issue on GitHub, please include the relevant part of your log file.
+
+If you have questions or want to inspect the source code, see the
+[ModelMark GitHub repository](https://github.com/gloptim77/ModelMark/tree/main).

modelmark-0.1.1/PKG-INFO

@@ -1,112 +0,0 @@
-Metadata-Version: 2.4
-Name: modelmark
-Version: 0.1.1
-Summary: Benchmark framework for neural network models
-Author: Andrew Larin
-License: MIT
-Requires-Python: >=3.12
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: numpy>=2.4
-Requires-Dist: pandas>=3.0
-Requires-Dist: torch>=2.12
-Requires-Dist: tqdm>=4.67
-Requires-Dist: playwright>=1.62
-Requires-Dist: rich>=15.0
-Requires-Dist: thop>=0.1
-Dynamic: license-file
-
-# ModelMark
-
-This tool will help you to test NN models against each other, and form a detailed report that is easy to embed to a website.
-
-**Model evaluation report example:**
-
-<img src="https://github.com/gloptim77/ModelMark/blob/main/result.png" alt="Evaluation result" width="700">
-
-## About:
-
-At each testing run iteration, modelmark:
-		
-	1) Selects next dataset, input/output len, model and seed 
-	2) Seeds the generators for reproducibility
-	3) Creates the loader, model and tester objects
-	4) Trains the model for E epochs, restores the state with the least validation loss
-	5) Tracks the GFLOPs, Memory, Time - all AVG over whole test
-	5) Evaluates the model on dataset with metrics from configuration file
-	6) Stores the mean result over S runs 
-
-	That way, the more seeds you run, the more "fair" the results are.
-	Finally, modelmark will form the report with all the testing results, training stats and your machine metadata.
-
-More on training stats:
-		
-	Time 		- average time per epoch 
-	Params 		- total number of model params
-	GFLOPs 		- average per batch
-	Peak Memory - max per training iteration
-
-Test consists of F * O * M * S runs, where:
-	
-	F - number of dataset files in the config (e.g. ["ETTh1" : ..., "Weather" : ...] - means F = 2)
-	O - number of input/output sizes (e.g. [32, 64, 128] means O = 3)
-	M - number of models (e.g. ["Linear" : ..., "LSTM" : ...] - means M = 2)
-	S - number of seeds (e.g. [42, 43, 44] - means S = 2)
-
-# Requirements:
-	
-	OS: Windows or Linux
-	Python: 3.12+
-
-# Usage:
-
-	1) Install the package   
-		
-		pip install "modelmark @ git+https://github.com/gloptim77/ModelMark.git"
-
-	2) Run the initialization in an empty folder
-
-		modelmark -t init
-	
-	3.1) It will create two folders "modelmark_files" and "models"
-	   
-	    In modelmark_files/config.py there are 3 main configs:
-
-	   		model_config = {...} - Models hyperparameters (number of layers, hidden dim, kernel size, etc.)
-			data_config = {...} - Dataset parameters (path to file, input/output features, train/val ratios, etc.)
-			test_config = {...} - Testing options (optimizer, loss criterion, metrics, learning rate, etc.)
-
-	3.2) (Optional) download the ETT dataset files
-
-		modelmark -t load
- 
-	4) When your config, model and data are ready, you can run the testing
-
-		modelmark -t run
-
-	5) When the test is over, report results will be in files "result.html" and "result.png"
-
-	Example of the model: "src/modelmark/models/linear.py"
-	Example of the config: "src/modelmark/config.py" 
-
-You will find the detailed config example with description at [src/modelmark/config.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/config.py).
-
-Example Linear model with detailed description at [src/modelmark/models/linear.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/models/linear.py).
-
-You can change the config and add your model files to suit your test requirements. 
-
-(But make sure that config and models are compatible)
-
-If you have questions or want to check the source code, go to [ModelMark Github](https://github.com/gloptim77/ModelMark/tree/main).
-
-## Troubleshooting
-
-- **If something doesn't work**
-
-	- Please open an issue and attach your application logs (found at "modelmark_files/modelmark.log") so I can help you troubleshoot. 
-	- Try to restart
-		```python
-		modelmark -t restart
-  		```
-
-	- Try to manually delete "modelmark_files" and return to Usage->2)

modelmark-0.1.1/README.md

@@ -1,94 +0,0 @@
-# ModelMark
-
-This tool will help you to test NN models against each other, and form a detailed report that is easy to embed to a website.
-
-**Model evaluation report example:**
-
-<img src="https://github.com/gloptim77/ModelMark/blob/main/result.png" alt="Evaluation result" width="700">
-
-## About:
-
-At each testing run iteration, modelmark:
-		
-	1) Selects next dataset, input/output len, model and seed 
-	2) Seeds the generators for reproducibility
-	3) Creates the loader, model and tester objects
-	4) Trains the model for E epochs, restores the state with the least validation loss
-	5) Tracks the GFLOPs, Memory, Time - all AVG over whole test
-	5) Evaluates the model on dataset with metrics from configuration file
-	6) Stores the mean result over S runs 
-
-	That way, the more seeds you run, the more "fair" the results are.
-	Finally, modelmark will form the report with all the testing results, training stats and your machine metadata.
-
-More on training stats:
-		
-	Time 		- average time per epoch 
-	Params 		- total number of model params
-	GFLOPs 		- average per batch
-	Peak Memory - max per training iteration
-
-Test consists of F * O * M * S runs, where:
-	
-	F - number of dataset files in the config (e.g. ["ETTh1" : ..., "Weather" : ...] - means F = 2)
-	O - number of input/output sizes (e.g. [32, 64, 128] means O = 3)
-	M - number of models (e.g. ["Linear" : ..., "LSTM" : ...] - means M = 2)
-	S - number of seeds (e.g. [42, 43, 44] - means S = 2)
-
-# Requirements:
-	
-	OS: Windows or Linux
-	Python: 3.12+
-
-# Usage:
-
-	1) Install the package   
-		
-		pip install "modelmark @ git+https://github.com/gloptim77/ModelMark.git"
-
-	2) Run the initialization in an empty folder
-
-		modelmark -t init
-	
-	3.1) It will create two folders "modelmark_files" and "models"
-	   
-	    In modelmark_files/config.py there are 3 main configs:
-
-	   		model_config = {...} - Models hyperparameters (number of layers, hidden dim, kernel size, etc.)
-			data_config = {...} - Dataset parameters (path to file, input/output features, train/val ratios, etc.)
-			test_config = {...} - Testing options (optimizer, loss criterion, metrics, learning rate, etc.)
-
-	3.2) (Optional) download the ETT dataset files
-
-		modelmark -t load
- 
-	4) When your config, model and data are ready, you can run the testing
-
-		modelmark -t run
-
-	5) When the test is over, report results will be in files "result.html" and "result.png"
-
-	Example of the model: "src/modelmark/models/linear.py"
-	Example of the config: "src/modelmark/config.py" 
-
-You will find the detailed config example with description at [src/modelmark/config.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/config.py).
-
-Example Linear model with detailed description at [src/modelmark/models/linear.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/models/linear.py).
-
-You can change the config and add your model files to suit your test requirements. 
-
-(But make sure that config and models are compatible)
-
-If you have questions or want to check the source code, go to [ModelMark Github](https://github.com/gloptim77/ModelMark/tree/main).
-
-## Troubleshooting
-
-- **If something doesn't work**
-
-	- Please open an issue and attach your application logs (found at "modelmark_files/modelmark.log") so I can help you troubleshoot. 
-	- Try to restart
-		```python
-		modelmark -t restart
-  		```
-
-	- Try to manually delete "modelmark_files" and return to Usage->2)

modelmark-0.1.1/src/modelmark.egg-info/PKG-INFO

@@ -1,112 +0,0 @@
-Metadata-Version: 2.4
-Name: modelmark
-Version: 0.1.1
-Summary: Benchmark framework for neural network models
-Author: Andrew Larin
-License: MIT
-Requires-Python: >=3.12
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: numpy>=2.4
-Requires-Dist: pandas>=3.0
-Requires-Dist: torch>=2.12
-Requires-Dist: tqdm>=4.67
-Requires-Dist: playwright>=1.62
-Requires-Dist: rich>=15.0
-Requires-Dist: thop>=0.1
-Dynamic: license-file
-
-# ModelMark
-
-This tool will help you to test NN models against each other, and form a detailed report that is easy to embed to a website.
-
-**Model evaluation report example:**
-
-<img src="https://github.com/gloptim77/ModelMark/blob/main/result.png" alt="Evaluation result" width="700">
-
-## About:
-
-At each testing run iteration, modelmark:
-		
-	1) Selects next dataset, input/output len, model and seed 
-	2) Seeds the generators for reproducibility
-	3) Creates the loader, model and tester objects
-	4) Trains the model for E epochs, restores the state with the least validation loss
-	5) Tracks the GFLOPs, Memory, Time - all AVG over whole test
-	5) Evaluates the model on dataset with metrics from configuration file
-	6) Stores the mean result over S runs 
-
-	That way, the more seeds you run, the more "fair" the results are.
-	Finally, modelmark will form the report with all the testing results, training stats and your machine metadata.
-
-More on training stats:
-		
-	Time 		- average time per epoch 
-	Params 		- total number of model params
-	GFLOPs 		- average per batch
-	Peak Memory - max per training iteration
-
-Test consists of F * O * M * S runs, where:
-	
-	F - number of dataset files in the config (e.g. ["ETTh1" : ..., "Weather" : ...] - means F = 2)
-	O - number of input/output sizes (e.g. [32, 64, 128] means O = 3)
-	M - number of models (e.g. ["Linear" : ..., "LSTM" : ...] - means M = 2)
-	S - number of seeds (e.g. [42, 43, 44] - means S = 2)
-
-# Requirements:
-	
-	OS: Windows or Linux
-	Python: 3.12+
-
-# Usage:
-
-	1) Install the package   
-		
-		pip install "modelmark @ git+https://github.com/gloptim77/ModelMark.git"
-
-	2) Run the initialization in an empty folder
-
-		modelmark -t init
-	
-	3.1) It will create two folders "modelmark_files" and "models"
-	   
-	    In modelmark_files/config.py there are 3 main configs:
-
-	   		model_config = {...} - Models hyperparameters (number of layers, hidden dim, kernel size, etc.)
-			data_config = {...} - Dataset parameters (path to file, input/output features, train/val ratios, etc.)
-			test_config = {...} - Testing options (optimizer, loss criterion, metrics, learning rate, etc.)
-
-	3.2) (Optional) download the ETT dataset files
-
-		modelmark -t load
- 
-	4) When your config, model and data are ready, you can run the testing
-
-		modelmark -t run
-
-	5) When the test is over, report results will be in files "result.html" and "result.png"
-
-	Example of the model: "src/modelmark/models/linear.py"
-	Example of the config: "src/modelmark/config.py" 
-
-You will find the detailed config example with description at [src/modelmark/config.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/config.py).
-
-Example Linear model with detailed description at [src/modelmark/models/linear.py](https://github.com/gloptim77/ModelMark/blob/main/src/modelmark/models/linear.py).
-
-You can change the config and add your model files to suit your test requirements. 
-
-(But make sure that config and models are compatible)
-
-If you have questions or want to check the source code, go to [ModelMark Github](https://github.com/gloptim77/ModelMark/tree/main).
-
-## Troubleshooting
-
-- **If something doesn't work**
-
-	- Please open an issue and attach your application logs (found at "modelmark_files/modelmark.log") so I can help you troubleshoot. 
-	- Try to restart
-		```python
-		modelmark -t restart
-  		```
-
-	- Try to manually delete "modelmark_files" and return to Usage->2)