modelmark 0.1.0__tar.gz

modelmark-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Andrew Larin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

modelmark-0.1.0/PKG-INFO

@@ -0,0 +1,109 @@
+Metadata-Version: 2.4
+Name: modelmark
+Version: 0.1.0
+Summary: Benchmark framework for neural network models
+Author: Andrew Larin
+License: MIT
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=2.4
+Requires-Dist: pandas>=3.0
+Requires-Dist: torch>=2.12
+Requires-Dist: tqdm>=4.67
+Requires-Dist: playwright>=1.62
+Requires-Dist: rich>=15.0
+Requires-Dist: thop>=0.1
+Dynamic: license-file
+
+# ModelMark
+This tool will help you to test NN models against each other, and form a detailed report that is easy to embed to a website.
+
+**Model evaluation report example:**
+
+<img src="result.png" alt="Evaluation result" width="600">
+
+## About:
+
+At each testing run iteration, modelmark:
+		
+	1) Selects next dataset, context, model and seed
+	2) Seeds the generators for reproducibility
+	3) Creates the loader, model and tester objects
+	4) Trains the model for E epochs, restores the state with the least validation loss
+	5) Tracks the GFLOPs, Memory, Time - all AVG over whole test
+	5) Evaluates the model on dataset with metrics from configuration file
+	6) Stores the mean result over S runs 
+
+	That way, the more seeds you run, the more "fair" the results are.
+	Finally, modelmark will form the report with all the testing results, training stats and your machine metadata.
+
+More on training stats:
+		
+	Time 		- average time per epoch 
+	Params 		- total number of model params
+	GFLOPs 		- average per batch
+	Peak Memory - max per training iteration
+
+Test consists of F * C * M * S runs, where:
+	
+	F - number of dataset files in the config (e.g. ["ETTh1" : ..., "Weather" : ...] - means F = 2)
+	C - number of context sizes (e.g. [32, 64, 128] - means C = 3)
+	M - number of models (e.g. ["Linear" : ..., "LSTM" : ...] - means M = 2)
+	S - number of seeds (e.g. [42, 43, 44] - means S = 2)
+
+# Requirements:
+	
+	OS: Windows or Linux
+	Python: 3.12+
+
+# Usage:
+
+	1) Install the package   
+		
+		pip install "modelmark @ git+https://github.com/gloptim77/ModelMark.git"
+
+	2) Run the initialization in an empty folder
+
+		modelmark -t init
+	
+	3.1) It will create two folders "modelmark_files" and "models"
+	   
+	    In modelmark_files/config.py there are 3 main configs:
+
+	   		model_config = {...} - Models hyperparameters (number of layers, hidden dim, kernel size, etc.)
+			data_config = {...} - Dataset parameters (path to file, input/output features, train/val ratios, etc.)
+			test_config = {...} - Testing options (optimizer, loss criterion, metrics, learning rate, etc.)
+
+	3.2) (Optional) download the ETT dataset files
+
+		modelmark -t load
+ 
+	4) When your config, model and data are ready, you can run the testing
+
+		modelmark -t run
+
+	5) When the test is over, report results will be in files "result.html" and "result.png"
+
+	Example of the model: "src/modelmark/models/linear.py"
+	Example of the config: "src/modelmark/config.py" 
+
+You will find the detailed config example with description at [src/modelmark/config.py](src/modelmark/config.py).
+
+Example Linear model with detailed description at [src/modelmark/models/linear.py](src/modelmark/models/linear.py).
+
+You can change the config and add your model files to suit your test requirements. 
+
+(But make sure that config and models are compatible)
+
+## Troubleshooting
+
+- **If something doesn't work**
+
+	- Please open an issue and attach your application logs (found at "modelmark_files/modelmark.log") so I can help you troubleshoot. 
+	- Try to restart
+		```python
+		modelmark -t restart
+  		```
+
+	- Try to manually delete "modelmark_files" and return to Usage->2)

modelmark-0.1.0/README.md

@@ -0,0 +1,91 @@
+# ModelMark
+This tool will help you to test NN models against each other, and form a detailed report that is easy to embed to a website.
+
+**Model evaluation report example:**
+
+<img src="result.png" alt="Evaluation result" width="600">
+
+## About:
+
+At each testing run iteration, modelmark:
+		
+	1) Selects next dataset, context, model and seed
+	2) Seeds the generators for reproducibility
+	3) Creates the loader, model and tester objects
+	4) Trains the model for E epochs, restores the state with the least validation loss
+	5) Tracks the GFLOPs, Memory, Time - all AVG over whole test
+	5) Evaluates the model on dataset with metrics from configuration file
+	6) Stores the mean result over S runs 
+
+	That way, the more seeds you run, the more "fair" the results are.
+	Finally, modelmark will form the report with all the testing results, training stats and your machine metadata.
+
+More on training stats:
+		
+	Time 		- average time per epoch 
+	Params 		- total number of model params
+	GFLOPs 		- average per batch
+	Peak Memory - max per training iteration
+
+Test consists of F * C * M * S runs, where:
+	
+	F - number of dataset files in the config (e.g. ["ETTh1" : ..., "Weather" : ...] - means F = 2)
+	C - number of context sizes (e.g. [32, 64, 128] - means C = 3)
+	M - number of models (e.g. ["Linear" : ..., "LSTM" : ...] - means M = 2)
+	S - number of seeds (e.g. [42, 43, 44] - means S = 2)
+
+# Requirements:
+	
+	OS: Windows or Linux
+	Python: 3.12+
+
+# Usage:
+
+	1) Install the package   
+		
+		pip install "modelmark @ git+https://github.com/gloptim77/ModelMark.git"
+
+	2) Run the initialization in an empty folder
+
+		modelmark -t init
+	
+	3.1) It will create two folders "modelmark_files" and "models"
+	   
+	    In modelmark_files/config.py there are 3 main configs:
+
+	   		model_config = {...} - Models hyperparameters (number of layers, hidden dim, kernel size, etc.)
+			data_config = {...} - Dataset parameters (path to file, input/output features, train/val ratios, etc.)
+			test_config = {...} - Testing options (optimizer, loss criterion, metrics, learning rate, etc.)
+
+	3.2) (Optional) download the ETT dataset files
+
+		modelmark -t load
+ 
+	4) When your config, model and data are ready, you can run the testing
+
+		modelmark -t run
+
+	5) When the test is over, report results will be in files "result.html" and "result.png"
+
+	Example of the model: "src/modelmark/models/linear.py"
+	Example of the config: "src/modelmark/config.py" 
+
+You will find the detailed config example with description at [src/modelmark/config.py](src/modelmark/config.py).
+
+Example Linear model with detailed description at [src/modelmark/models/linear.py](src/modelmark/models/linear.py).
+
+You can change the config and add your model files to suit your test requirements. 
+
+(But make sure that config and models are compatible)
+
+## Troubleshooting
+
+- **If something doesn't work**
+
+	- Please open an issue and attach your application logs (found at "modelmark_files/modelmark.log") so I can help you troubleshoot. 
+	- Try to restart
+		```python
+		modelmark -t restart
+  		```
+
+	- Try to manually delete "modelmark_files" and return to Usage->2)

modelmark-0.1.0/pyproject.toml

@@ -0,0 +1,31 @@
+[build-system]
+requires = ["setuptools"]
+build-backend = "setuptools.build_meta"
+
+[project.scripts]
+modelmark = "modelmark.modelmark:run"
+
+[project]
+name = "modelmark"
+version = "0.1.0"
+description = "Benchmark framework for neural network models"
+readme = "README.md"
+requires-python = ">=3.12"
+license = { text = "MIT" }
+
+authors = [
+    { name = "Andrew Larin" }
+]
+
+dependencies = [
+    "numpy>=2.4",
+    "pandas>=3.0",
+	"torch>=2.12",
+	"tqdm>=4.67",
+	"playwright>=1.62",
+	"rich>=15.0",
+	"thop>=0.1"
+]
+
+[tool.setuptools.packages.find]
+where = ["src"]

modelmark-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

modelmark-0.1.0/src/modelmark/__init__.py

@@ -0,0 +1,3 @@
+"""Reusable testing framework."""
+
+__version__ = "0.1.0"

modelmark-0.1.0/src/modelmark/__main__.py

@@ -0,0 +1,5 @@
+import sys
+from modelmark import run
+
+if __name__ == "__main__":
+	sys.exit(run())

modelmark-0.1.0/src/modelmark/common/dataset.py

@@ -0,0 +1,116 @@
+from __future__ import annotations
+
+import torch
+from torch.utils.data import Dataset
+
+import copy
+import numpy as np
+import pandas as pd
+
+from modelmark.common.utils import load_config
+config = load_config()
+
+class CausalDataset(Dataset):
+	"""
+	Sliding-window dataset for ETT.
+
+	Returns:
+		x: [seq_len, num_features]
+		y: [pred_len, 1]
+	"""
+
+	def __init__(
+		self,
+		data_config: dict,
+		seq_len: int,
+		pred_len: int,
+		split: str = "train",
+		mean: np.ndarray | None = None,
+		std: np.ndarray | None = None,
+	):
+		super().__init__()
+
+		self.seq_len = seq_len
+		self.pred_len = pred_len
+
+		# Extract config
+		data_path = config.data_path + data_config["path"]
+		features = copy.deepcopy(data_config["input_features"])
+		targets = data_config["output_features"]
+		features.extend([item for item in targets if item not in features])
+		train_ratio = data_config["train_ratio"]
+		val_ratio = data_config["val_ratio"]
+
+		# Load, extract, convert
+		df = pd.read_csv(data_path)
+		
+		df = df[features]
+		data = df.values.astype(np.float32)
+		self.input_indices = torch.tensor(df.columns.get_indexer(data_config["input_features"]), dtype=torch.int)
+		self.output_indices = torch.tensor(df.columns.get_indexer(data_config["output_features"]), dtype=torch.int)
+
+		n = len(data)
+
+		train_end = int(n * train_ratio)
+		val_end = int(n * (train_ratio + val_ratio))
+
+		if split == "train":
+			start = 0
+			end = train_end
+		elif split == "val":
+			start = train_end
+			end = val_end
+		elif split == "test":
+			start = val_end
+			end = n
+		else:
+			raise ValueError(f"Unknown split: {split}")
+
+		# Save scaling information
+		if split == "train":
+			self.mean = data[start:train_end].mean(axis=0)
+			self.std = data[start:train_end].std(axis=0)
+
+			# Prevent division by zero
+			self.std[self.std < 1e-8] = 1.0
+		else:
+			if mean is None or std is None:
+				raise ValueError(
+					"Validation/test datasets need training mean and std."
+				)
+
+			self.mean = mean
+			self.std = std
+
+		# Scale using TRAIN statistics
+		data = (data - self.mean) / self.std
+
+		# Keep only the split region
+		self.data = data[start:end]
+
+		# Number of valid windows
+		self.length = len(self.data) - seq_len - pred_len + 1
+
+		if self.length <= 0:
+			raise ValueError(
+				f"Split '{split}' is too short for "
+				f"seq_len={seq_len}, pred_len={pred_len}"
+			)
+
+	def __len__(self):
+		return self.length
+
+	def __getitem__(self, idx):
+		x_start = idx
+		x_end = x_start + self.seq_len
+
+		y_start = x_end
+		y_end = y_start + self.pred_len
+
+		x = self.data[x_start:x_end, self.input_indices]
+		y = self.data[y_start:y_end, self.output_indices]
+
+		return (
+			torch.from_numpy(x),
+			torch.from_numpy(y),
+		)

modelmark-0.1.0/src/modelmark/common/downloader.py

@@ -0,0 +1,88 @@
+"""Download time-series datasets to a local directory.
+
+Supports:
+- ETT (ETTh1, ETTh2, ETTm1, ETTm2) — from the ETDataset GitHub repo
+- Air Quality (UCI Repository)
+- Weather (Max Planck Institute / Jena climate dataset)
+"""
+
+import argparse
+import shutil
+import urllib.request
+import zipfile
+from pathlib import Path
+
+import logging
+logger = logging.getLogger(__name__)
+from rich.console import Console
+console = Console()
+
+DATASETS = {
+	"ett": {
+		"dir": "data/ett",
+		"kind": "multi_csv",
+		"base_url": "https://raw.githubusercontent.com/zhouhaoyi/ETDataset/main/ETT-small",
+		"files": ["ETTh1", "ETTh2", "ETTm1", "ETTm2"],
+	},
+	"air_quality": {
+		"dir": "data/air_quality",
+		"kind": "zip",
+		"url": "https://archive.ics.uci.edu/static/public/360/air+quality.zip",
+		"zip_name": "AirQualityUCI.zip",
+	},
+	"weather": {
+		"dir": "data/weather",
+		"kind": "zip",
+		"url": "https://storage.googleapis.com/tensorflow/tf-keras-datasets/jena_climate_2009_2016.csv.zip",
+		"zip_name": "jena_climate_2009_2016.csv.zip",
+	},
+}
+
+
+def download_ett(cfg: dict, out_dir: Path) -> None:
+	out_dir.mkdir(parents=True, exist_ok=True)
+	for name in cfg["files"]:
+		url = f"{cfg['base_url']}/{name}.csv"
+		path = out_dir / f"{name}.csv"
+		if path.exists():
+			logger.info(f"{name}.csv already exists, skipping.")
+			continue
+		logger.info(f"Downloading {name}...")
+		console.print(f"Downloading {name}...")
+		urllib.request.urlretrieve(url, path)
+		logger.info(f"Saved to {path}")
+		console.print(f"Saved to {path}", style="green")
+
+
+def download_zip_dataset(cfg: dict, out_dir: Path) -> None:
+	out_dir.mkdir(parents=True, exist_ok=True)
+	zip_path = out_dir / cfg["zip_name"]
+
+	if not zip_path.exists():
+		logger.info(f"Downloading {cfg['zip_name']}...")
+		console.print(f"Downloading {cfg['zip_name']}...")
+		urllib.request.urlretrieve(cfg["url"], zip_path)
+		logger.info(f"Saved to {zip_path}")
+		console.print(f"Saved to {zip_path}", style="green")
+	else:
+		logger.info(f"{cfg['zip_name']} already exists, skipping download.")
+		console.print(f"{cfg['zip_name']} already exists, skipping download.", style="yellow")
+
+	logger.info(f"Extracting {zip_path.name}...")
+	console.print(f"Extracting {zip_path.name}...")
+	with zipfile.ZipFile(zip_path) as zf:
+		zf.extractall(out_dir)
+	console.print(f"Extracted to {out_dir}", color="green")
+
+def download_dataset(name: str) -> None:
+	if name not in DATASETS:
+		valid = ", ".join(DATASETS)
+		raise ValueError(f"Unknown dataset '{name}'. Choose from: {valid}")
+
+	cfg = DATASETS[name]
+	out_dir = Path(cfg["dir"])
+
+	if cfg["kind"] == "multi_csv":
+		download_ett(cfg, out_dir)
+	elif cfg["kind"] == "zip":
+		download_zip_dataset(cfg, out_dir)

modelmark-0.1.0/src/modelmark/common/loader.py

@@ -0,0 +1,63 @@
+from __future__ import annotations
+
+from modelmark.common.dataset import CausalDataset
+from torch.utils.data import DataLoader
+
+import logging
+logger = logging.getLogger(__name__)
+from modelmark.common.utils import load_config
+config = load_config()
+
+class Loader:
+
+	def __init__(self, file_config : dict, context_size : int):
+
+		self.get_dataloaders(file_config, context_size) 
+		logger.debug(f"Dataset len stats: train={len(self.train_loader)} val={len(self.val_loader)} test={len(self.test_loader)}")
+
+	def get_dataloaders(self, file_config : dict, context_size : int) -> tuple[DataLoader, DataLoader, DataLoader]:
+		"""Load the data, pack to datasets, create the loaders and return them"""
+	
+		train_ds = CausalDataset(data_config = file_config,
+								seq_len = context_size, 
+								pred_len = context_size, 
+								split = "train")
+
+		val_ds 	= CausalDataset(data_config = file_config, 
+								seq_len = context_size, 
+								pred_len = context_size, 
+								split = "val", 
+								mean = train_ds.mean, 
+								std = train_ds.std)
+
+		test_ds = CausalDataset(data_config = file_config, 
+								seq_len = context_size, 
+								pred_len = context_size, 
+								split = "test", 
+								mean = train_ds.mean, 
+								std = train_ds.std)
+		
+		self.train_loader = DataLoader(
+			train_ds,
+			batch_size = config.test_config["optim"]["batch_size"],
+			shuffle = True,
+			num_workers = 0,
+			pin_memory = True,
+		)
+
+		self.val_loader = DataLoader(
+			val_ds,
+			batch_size = config.test_config["optim"]["batch_size"],
+			shuffle = False,
+			num_workers = 0,
+			pin_memory = True,
+		)
+	
+		self.test_loader = DataLoader(
+			test_ds,
+			batch_size = config.test_config["optim"]["batch_size"],
+			shuffle = False,
+			num_workers = 0,
+			pin_memory = True,
+		)
+	

modelmark-0.1.0/src/modelmark/common/logger.py

@@ -0,0 +1,34 @@
+import logging
+import sys
+from typing import Optional
+
+def setup_logging(log_format: Optional[str] = None) -> None:
+	"""
+	Configure logging for the entire application.
+
+	Args:
+		log_format: Optional custom format string. If None, a default format is used.
+	"""
+
+	if log_format is None:
+		log_format = "%(asctime)s [%(levelname)s] [%(name)s] - %(message)s"
+
+	logging.basicConfig(
+		level = logging.DEBUG,
+		filemode = 'w',
+		filename='modelmark_files/modelmark.log',
+		format = log_format,
+		datefmt = '%H:%M:%S',
+	)
+
+def get_logger(name: str) -> logging.Logger:
+	"""
+	Convenience function to get a logger for a module.
+
+	Args:
+		name: Usually __name__.
+
+	Returns:
+		A configured logger instance.
+	"""
+	return logging.getLogger(name)