halib 0.2.16__tar.gz → 0.2.18__tar.gz

halib-0.2.18/PKG-INFO

@@ -0,0 +1,187 @@
+Metadata-Version: 2.4
+Name: halib
+Version: 0.2.18
+Summary: Small library for common tasks
+Author: Hoang Van Ha
+Author-email: hoangvanhauit@gmail.com
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: arrow
+Requires-Dist: click
+Requires-Dist: dataclass-wizard
+Requires-Dist: enlighten
+Requires-Dist: itables
+Requires-Dist: kaleido
+Requires-Dist: loguru
+Requires-Dist: matplotlib
+Requires-Dist: more-itertools
+Requires-Dist: moviepy
+Requires-Dist: networkx
+Requires-Dist: numpy
+Requires-Dist: omegaconf
+Requires-Dist: opencv-python
+Requires-Dist: pandas
+Requires-Dist: Pillow
+Requires-Dist: plotly
+Requires-Dist: Pyarrow
+Requires-Dist: pycurl
+Requires-Dist: pygwalker
+Requires-Dist: python-telegram-bot
+Requires-Dist: requests
+Requires-Dist: rich
+Requires-Dist: scikit-learn
+Requires-Dist: seaborn
+Requires-Dist: tabulate
+Requires-Dist: timebudget
+Requires-Dist: tqdm
+Requires-Dist: tube_dl
+Requires-Dist: wandb
+Requires-Dist: ipynbname
+Requires-Dist: typed-argument-parser
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# 📦 Helper Package for Coding and Automation Changelog
+
+## v0.2.x (Experiment & Core Updates)
+
+### **v0.2.18**
+- ✨ **New Feature:** Added `exp.core.param_gen` to facilitate fast generation of parameter combination sweeps (grid search) using YAML configurations.
+
+### **v0.2.17**
+- 🚀 **Improvement:** Updated `exp.perf.profiler` with an `enabled` flag for dynamic toggling.
+- 🚀 **Improvement:** Added a `measure` context manager to simplify performance measuring of code blocks.
+
+### **v0.2.13**
+- ♻️ **Refactor:** Major reorganization of packages. Renamed `research` package to `exp` (Experiment Management).
+- 🚀 **Improvement:** Updated `exp/perfcalc.py` to allow saving computed performance metrics to CSV without explicitly calling `calc_perfs`.
+
+### **v0.2.1**
+- ✨ **New Feature:** Added `eval_exp` method to `exp/base_exp` for running evaluations (e.g., model testing) after experiments conclude.
+
+---
+
+## v0.1.9x (Visualization & Generators)
+
+### **v0.1.99**
+- ✨ **New Feature:** Added `gen_ipynb_name` to `filetype/ipynb`. Generates filenames based on the current notebook name with optional timestamps.
+
+### **v0.1.96**
+- ✨ **New Feature:** Added `PlotHelper` class in `research/plot` for plotting training history and image grids (dataset samples or model outputs).
+
+### **v0.1.91**
+- ✨ **New Feature:** Added `ParamGen` class to `research/param_gen` for parsing YAML files into parameter lists for hyperparameter searches.
+
+### **v0.1.90**
+- ✨ **New Feature:** Added `zProfiler` class to `research/profiler` for measuring context/step execution time, supporting dynamic color scales in plots.
+
+---
+
+## v0.1.5x - v0.1.7x (Infrastructure & Utilities)
+
+### **v0.1.77**
+- ✨ **New Feature:** Added `BaseExp` class in `research/base_exp` to handle common experiment tasks (performance calculation, result saving).
+
+### **v0.1.67**
+- 🔧 **Maintenance:** Switched to **uv** for virtual environment management.
+- 🚀 **Improvement:** Updated `research/perfcalc` to support both `torchmetrics` and custom metrics.
+
+### **v0.1.61**
+- ✨ **New Feature:** Added `VideoUtils` (`util/video`) for common video handling tasks.
+- ✨ **New Feature:** Added `GPUMonitor` (`util/gpu_mon`) for tracking GPU usage and performance.
+
+### **v0.1.59**
+- 🔨 **Architecture:** Added `util/perfcalc` abstract base class. This requires implementation of specific performance calculation logic.
+
+### **v0.1.55**
+- ✨ **New Feature:** Added `util/dataclass_util` for dynamic creation of `dataclass` objects from dictionaries or YAML (supports nested structures).
+
+### **v0.1.52**
+- ✨ **New Feature:** Added `research/perftb` module for managing experiment performance tables (filtering by dataset, metric, etc.).
+
+### **v0.1.50**
+- ✨ **New Feature:** Added `pprint_local_path` to print clickable file URIs for local paths.
+- ✨ **New Feature:** Added `research` package containing `benchquery` for dataframe benchmarking.
+- ✨ **New Feature:** Added `wandb` module for offline syncing and batch clearing of Weights & Biases runs.
+
+---
+
+## v0.1.4x (Display & formatting)
+
+### **v0.1.47**
+- ✨ **New Feature:** Added `pprint_box` to print objects or strings inside a decorative box frame.
+
+### **v0.1.46**
+- 🐛 **Fix:** Filtered `UserWarning: Unable to import Axes3D`.
+- 🚀 **Improvement:** Added `auto_wrap_text` to `fn_display_df` to prevent long text overflow in tables.
+
+### **v0.1.42**
+- ✨ **New Feature:** Added `rich_color.py` wrapper for basic color lists.
+
+### **v0.1.41**
+- ✨ **New Feature:** Added `rich_color.py` to support rich color information (palettes, strings) using the `rich` library.
+
+### **v0.1.40**
+- 🚀 **Improvement:** Updated `csvfile.py` to use `itables` and `pygwalker` for interactive dataframe display in Jupyter notebooks.
+
+---
+
+## v0.1.3x (Data & Loading)
+
+### **v0.1.38**
+- ✨ **New Feature:** Added `torchloader.py` to search for optimal `DataLoader` configurations (num_workers, batch_size, pin_memory).
+
+### **v0.1.37**
+- ✨ **New Feature:** Added `dataset.py` for splitting classification datasets into train/val/test sets.
+
+### **v0.1.33**
+- ✨ **New Feature:** Added `plot.py` for plotting Deep Learning training history (accuracy/loss) using `seaborn` and `matplotlib`.
+
+---
+
+## v0.1.0 - v0.1.2x (Early Utilities)
+
+### **v0.1.29**
+- 🐛 **Fix:** Pinned `kaleido==0.1.*` for `tele_noti` as version `0.2.*` caused image generation hangs.
+
+### **v0.1.24**
+- ♻️ **Refactor:** Renamed `sys` module to `system` to avoid conflicts with Python's built-in `sys`.
+- ✨ **New Feature:** Added `tele_noti` module for Telegram notifications regarding training progress.
+
+### **v0.1.22**
+- ✨ **New Feature:** Added `cuda.py` to check CUDA availability for both PyTorch and TensorFlow.
+
+### **v0.1.21**
+- ✨ **New Feature:** Added YAML inheritance and overriding support using `networkx` and `omegaconf`.
+
+### **v0.1.15**
+- ✨ **New Feature:** Added common logging library and `@console_log` decorator for function tracing.
+
+### **v0.1.10**
+- 🐛 **Fix:** Fixed typo `is_exit` -> `is_exist` in `filesys`.
+- 🚀 **Improvement:** Updated `gdrive` to support uploading to specific folders and returning direct shareable links.
+
+### **v0.1.9**
+- 🔧 **Maintenance:** Added `requirements.txt`.
+
+### **v0.1.6 - v0.1.8**
+- 🚀 **Performance:** Optimized table insertion by using an in-memory `row_pool_dict` before committing to the DataFrame.
+- ✨ **New Feature:** Added `DFCreator` for manipulating DataFrames (create, insert, display, save).
+
+### **v0.1.4 - v0.1.5**
+- ✨ **New Feature:** Added `cmd` module.
+- ✨ **New Feature:** Support for creating Bitbucket Projects from templates.
+
+### **v0.1.2**
+- ✨ **New Feature:** Added support for uploading local files to Google Drive.

halib-0.2.18/README.md

@@ -0,0 +1,133 @@
+# 📦 Helper Package for Coding and Automation Changelog
+
+## v0.2.x (Experiment & Core Updates)
+
+### **v0.2.18**
+- ✨ **New Feature:** Added `exp.core.param_gen` to facilitate fast generation of parameter combination sweeps (grid search) using YAML configurations.
+
+### **v0.2.17**
+- 🚀 **Improvement:** Updated `exp.perf.profiler` with an `enabled` flag for dynamic toggling.
+- 🚀 **Improvement:** Added a `measure` context manager to simplify performance measuring of code blocks.
+
+### **v0.2.13**
+- ♻️ **Refactor:** Major reorganization of packages. Renamed `research` package to `exp` (Experiment Management).
+- 🚀 **Improvement:** Updated `exp/perfcalc.py` to allow saving computed performance metrics to CSV without explicitly calling `calc_perfs`.
+
+### **v0.2.1**
+- ✨ **New Feature:** Added `eval_exp` method to `exp/base_exp` for running evaluations (e.g., model testing) after experiments conclude.
+
+---
+
+## v0.1.9x (Visualization & Generators)
+
+### **v0.1.99**
+- ✨ **New Feature:** Added `gen_ipynb_name` to `filetype/ipynb`. Generates filenames based on the current notebook name with optional timestamps.
+
+### **v0.1.96**
+- ✨ **New Feature:** Added `PlotHelper` class in `research/plot` for plotting training history and image grids (dataset samples or model outputs).
+
+### **v0.1.91**
+- ✨ **New Feature:** Added `ParamGen` class to `research/param_gen` for parsing YAML files into parameter lists for hyperparameter searches.
+
+### **v0.1.90**
+- ✨ **New Feature:** Added `zProfiler` class to `research/profiler` for measuring context/step execution time, supporting dynamic color scales in plots.
+
+---
+
+## v0.1.5x - v0.1.7x (Infrastructure & Utilities)
+
+### **v0.1.77**
+- ✨ **New Feature:** Added `BaseExp` class in `research/base_exp` to handle common experiment tasks (performance calculation, result saving).
+
+### **v0.1.67**
+- 🔧 **Maintenance:** Switched to **uv** for virtual environment management.
+- 🚀 **Improvement:** Updated `research/perfcalc` to support both `torchmetrics` and custom metrics.
+
+### **v0.1.61**
+- ✨ **New Feature:** Added `VideoUtils` (`util/video`) for common video handling tasks.
+- ✨ **New Feature:** Added `GPUMonitor` (`util/gpu_mon`) for tracking GPU usage and performance.
+
+### **v0.1.59**
+- 🔨 **Architecture:** Added `util/perfcalc` abstract base class. This requires implementation of specific performance calculation logic.
+
+### **v0.1.55**
+- ✨ **New Feature:** Added `util/dataclass_util` for dynamic creation of `dataclass` objects from dictionaries or YAML (supports nested structures).
+
+### **v0.1.52**
+- ✨ **New Feature:** Added `research/perftb` module for managing experiment performance tables (filtering by dataset, metric, etc.).
+
+### **v0.1.50**
+- ✨ **New Feature:** Added `pprint_local_path` to print clickable file URIs for local paths.
+- ✨ **New Feature:** Added `research` package containing `benchquery` for dataframe benchmarking.
+- ✨ **New Feature:** Added `wandb` module for offline syncing and batch clearing of Weights & Biases runs.
+
+---
+
+## v0.1.4x (Display & formatting)
+
+### **v0.1.47**
+- ✨ **New Feature:** Added `pprint_box` to print objects or strings inside a decorative box frame.
+
+### **v0.1.46**
+- 🐛 **Fix:** Filtered `UserWarning: Unable to import Axes3D`.
+- 🚀 **Improvement:** Added `auto_wrap_text` to `fn_display_df` to prevent long text overflow in tables.
+
+### **v0.1.42**
+- ✨ **New Feature:** Added `rich_color.py` wrapper for basic color lists.
+
+### **v0.1.41**
+- ✨ **New Feature:** Added `rich_color.py` to support rich color information (palettes, strings) using the `rich` library.
+
+### **v0.1.40**
+- 🚀 **Improvement:** Updated `csvfile.py` to use `itables` and `pygwalker` for interactive dataframe display in Jupyter notebooks.
+
+---
+
+## v0.1.3x (Data & Loading)
+
+### **v0.1.38**
+- ✨ **New Feature:** Added `torchloader.py` to search for optimal `DataLoader` configurations (num_workers, batch_size, pin_memory).
+
+### **v0.1.37**
+- ✨ **New Feature:** Added `dataset.py` for splitting classification datasets into train/val/test sets.
+
+### **v0.1.33**
+- ✨ **New Feature:** Added `plot.py` for plotting Deep Learning training history (accuracy/loss) using `seaborn` and `matplotlib`.
+
+---
+
+## v0.1.0 - v0.1.2x (Early Utilities)
+
+### **v0.1.29**
+- 🐛 **Fix:** Pinned `kaleido==0.1.*` for `tele_noti` as version `0.2.*` caused image generation hangs.
+
+### **v0.1.24**
+- ♻️ **Refactor:** Renamed `sys` module to `system` to avoid conflicts with Python's built-in `sys`.
+- ✨ **New Feature:** Added `tele_noti` module for Telegram notifications regarding training progress.
+
+### **v0.1.22**
+- ✨ **New Feature:** Added `cuda.py` to check CUDA availability for both PyTorch and TensorFlow.
+
+### **v0.1.21**
+- ✨ **New Feature:** Added YAML inheritance and overriding support using `networkx` and `omegaconf`.
+
+### **v0.1.15**
+- ✨ **New Feature:** Added common logging library and `@console_log` decorator for function tracing.
+
+### **v0.1.10**
+- 🐛 **Fix:** Fixed typo `is_exit` -> `is_exist` in `filesys`.
+- 🚀 **Improvement:** Updated `gdrive` to support uploading to specific folders and returning direct shareable links.
+
+### **v0.1.9**
+- 🔧 **Maintenance:** Added `requirements.txt`.
+
+### **v0.1.6 - v0.1.8**
+- 🚀 **Performance:** Optimized table insertion by using an in-memory `row_pool_dict` before committing to the DataFrame.
+- ✨ **New Feature:** Added `DFCreator` for manipulating DataFrames (create, insert, display, save).
+
+### **v0.1.4 - v0.1.5**
+- ✨ **New Feature:** Added `cmd` module.
+- ✨ **New Feature:** Support for creating Bitbucket Projects from templates.
+
+### **v0.1.2**
+- ✨ **New Feature:** Added support for uploading local files to Google Drive.

halib-0.2.18/halib/exp/core/param_gen.py

@@ -0,0 +1,185 @@
+import os
+import copy
+import numpy as np
+from itertools import product
+from typing import Dict, Any, List, Iterator, Optional
+from ...filetype import yamlfile
+
+class ParamGen:
+    """
+    A flexible parameter grid generator for hyperparameter tuning and experiment management.
+
+    This class generates a Cartesian product of parameters from a "sweep configuration"
+    and optionally merges them into a "base configuration". It abstracts away the complexity
+    of handling nested dictionaries and range generation.
+
+    Key Features:
+    -----------
+    1. **Flexible Syntax**: Define parameters using standard nested dictionaries or
+       dot-notation keys (e.g., `'model.backbone.layers'`).
+    2. **Range Shortcuts**:
+       - **Choices**: Standard lists `[1, 2, 3]`.
+       - **String Ranges**: `"start:stop:step"` (e.g., `"0:10:2"` -> `[0, 2, 4, 6, 8]`).
+       - **Dict Ranges**: `{'start': 0, 'stop': 1, 'step': 0.1}`.
+    3. **Deep Merging**: Automatically updates deep keys in `base_cfg` without overwriting siblings.
+
+    Example:
+    --------
+    >>> base = {'model': {'name': 'resnet', 'dropout': 0.1}, 'seed': 42}
+    >>> sweep = {
+    ...     'model.name': ['resnet', 'vit'],      # Dot notation
+    ...     'model.dropout': "0.1:0.3:0.1",       # Range string
+    ...     'seed': [42, 100]                     # Simple choice
+    ... }
+    >>> grid = ParamGen(sweep, base)
+    >>> configs = grid.expand()
+    >>> print(len(configs))  # Outputs: 8 (2 models * 2 dropouts * 2 seeds)
+    Attributes:
+        keys (List[str]): List of flattened dot-notation keys being swept.
+        values (List[List[Any]]): List of value options for each key.
+    """
+    def __init__(
+        self, sweep_cfg: Dict[str, Any], base_cfg: Optional[Dict[str, Any]] = None
+    ):
+        """
+        Args:
+            sweep_cfg: The dictionary defining parameters to sweep.
+            base_cfg: (Optional) The base config to merge sweep parameters into.
+                      If None, expand() behaves like expand_sweep().
+        """
+        self.base_cfg = base_cfg if base_cfg is not None else {}
+
+        # Recursively flatten the nested sweep config into dot-notation keys
+        self.param_space = self._flatten_params(sweep_cfg)
+        self.keys = list(self.param_space.keys())
+        self.values = list(self.param_space.values())
+
+    def __iter__(self) -> Iterator[Dict[str, Any]]:
+        """Yields fully merged configurations one by one."""
+        for combination in product(*self.values):
+            # 1. Create the flat sweep dict (dot notation)
+            flat_params = dict(zip(self.keys, combination))
+
+            # 2. Deep copy base and update with current params
+            new_cfg = copy.deepcopy(self.base_cfg)
+            new_cfg = self._apply_updates(new_cfg, flat_params)
+
+            # 3. Store metadata (Optional)
+            # if "_meta" not in new_cfg:
+            #     new_cfg["_meta"] = {}
+            # We unflatten the sweep params here so the log is readable
+            # new_cfg["_meta"]["sweep_params"] = self._unflatten(flat_params)
+
+            yield new_cfg
+
+    # ! --- Factory Methods ---
+    @classmethod
+    def from_dicts(
+        cls, sweep_cfg: Dict[str, Any], base_cfg: Optional[Dict[str, Any]] = None
+    ):
+        """
+        Load from dictionaries.
+        Args:
+            sweep_cfg: The dictionary defining parameters to sweep.
+            base_cfg: (Optional) The base config to merge sweep parameters into.
+        """
+        return cls(sweep_cfg, base_cfg)
+
+    @classmethod
+    def from_files(cls, sweep_yaml: str, base_yaml: Optional[str] = None):
+        """
+        Load from files.
+        Args:
+            sweep_yaml: Path to sweep config.
+            base_yaml: (Optional) Path to base config.
+        """
+        assert os.path.isfile(sweep_yaml), f"Sweep file not found: {sweep_yaml}"
+        sweep_dict = yamlfile.load_yaml(sweep_yaml, to_dict=True)
+        base_dict = None
+        if base_yaml:
+            base_dict = yamlfile.load_yaml(base_yaml, to_dict=True)
+            if "__base__" in base_dict:
+                del base_dict["__base__"]
+
+        return cls(sweep_dict, base_dict)
+
+    def expand(self) -> List[Dict[str, Any]]:
+        """Generates and returns the full list of MERGED configurations."""
+        return list(self)
+
+    def expand_sweep_flat(self) -> List[Dict[str, Any]]:
+        """
+        Returns a list of ONLY the sweep parameters, formatted as FLAT dot-notation dictionaries.
+
+        Returns:
+            [{'exp_params.model': 'resnet', 'exp_params.lr': 0.01}, ...]
+        """
+        combinations = []
+        for combination in product(*self.values):
+            flat_dict = dict(zip(self.keys, combination))
+            combinations.append(flat_dict)
+        return combinations
+
+    def _unflatten(self, flat_dict: Dict[str, Any]) -> Dict[str, Any]:
+        """Converts {'a.b': 1} back to {'a': {'b': 1}}."""
+        nested = {}
+        self._apply_updates(nested, flat_dict)
+        return nested
+
+    def _flatten_params(
+        self, cfg: Dict[str, Any], parent_key: str = ""
+    ) -> Dict[str, List[Any]]:
+        """Recursively converts nested dicts into flat dot-notation keys."""
+        flat = {}
+        for key, val in cfg.items():
+            current_key = f"{parent_key}.{key}" if parent_key else key
+
+            if self._is_sweep_leaf(val):
+                flat[current_key] = self._expand_val(val)
+            elif isinstance(val, dict):
+                flat.update(self._flatten_params(val, current_key))
+            else:
+                flat[current_key] = [val]
+        return flat
+
+    def _is_sweep_leaf(self, val: Any) -> bool:
+        if isinstance(val, list):
+            return True
+        if isinstance(val, str) and ":" in val:
+            return True
+        if isinstance(val, dict) and "start" in val and "stop" in val:
+            return True
+        return False
+
+    def _expand_val(self, val: Any) -> List[Any]:
+        if isinstance(val, list):
+            return val
+
+        if isinstance(val, str) and ":" in val:
+            try:
+                parts = [float(x) for x in val.split(":")]
+                if len(parts) == 3:
+                    arr = np.arange(parts[0], parts[1], parts[2])
+                    return [float(f"{x:.6g}") for x in arr]
+            except ValueError:
+                pass
+
+        if isinstance(val, dict) and "start" in val:
+            step = val.get("step", 1)
+            return np.arange(val["start"], val["stop"], step).tolist()
+
+        return [val]
+
+    def _apply_updates(
+        self, cfg: Dict[str, Any], updates: Dict[str, Any]
+    ) -> Dict[str, Any]:
+        """Deep merges dot-notation updates into cfg."""
+        for key, val in updates.items():
+            parts = key.split(".")
+            target = cfg
+            for part in parts[:-1]:
+                if part not in target:
+                    target[part] = {}
+                target = target[part]
+            target[parts[-1]] = val
+        return cfg

{halib-0.2.16 → halib-0.2.18}/halib/filetype/videofile.py

@@ -100,7 +100,7 @@ class VideoUtils:
     # FFmpeg Horizontal Stack
     # -----------------------------
     @staticmethod
-    def video_hstack(video_files, output_file):
+    def hstack(video_files, output_file):
         """Horizontally stack multiple videos using FFmpeg."""
         tmp_file = "video_list.txt"
         try: