halib 0.2.17__tar.gz → 0.2.19__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (60) hide show
  1. halib-0.2.19/PKG-INFO +187 -0
  2. halib-0.2.19/README.md +133 -0
  3. halib-0.2.19/halib/exp/core/param_gen.py +189 -0
  4. halib-0.2.19/halib.egg-info/PKG-INFO +187 -0
  5. {halib-0.2.17 → halib-0.2.19}/setup.py +3 -3
  6. halib-0.2.17/PKG-INFO +0 -220
  7. halib-0.2.17/README.md +0 -166
  8. halib-0.2.17/halib/exp/core/param_gen.py +0 -181
  9. halib-0.2.17/halib.egg-info/PKG-INFO +0 -220
  10. {halib-0.2.17 → halib-0.2.19}/.gitignore +0 -0
  11. {halib-0.2.17 → halib-0.2.19}/GDriveFolder.txt +0 -0
  12. {halib-0.2.17 → halib-0.2.19}/LICENSE.txt +0 -0
  13. {halib-0.2.17 → halib-0.2.19}/MANIFEST.in +0 -0
  14. {halib-0.2.17 → halib-0.2.19}/halib/__init__.py +0 -0
  15. {halib-0.2.17 → halib-0.2.19}/halib/common/__init__.py +0 -0
  16. {halib-0.2.17 → halib-0.2.19}/halib/common/common.py +0 -0
  17. {halib-0.2.17 → halib-0.2.19}/halib/common/rich_color.py +0 -0
  18. {halib-0.2.17 → halib-0.2.19}/halib/exp/__init__.py +0 -0
  19. {halib-0.2.17 → halib-0.2.19}/halib/exp/core/__init__.py +0 -0
  20. {halib-0.2.17 → halib-0.2.19}/halib/exp/core/base_config.py +0 -0
  21. {halib-0.2.17 → halib-0.2.19}/halib/exp/core/base_exp.py +0 -0
  22. {halib-0.2.17 → halib-0.2.19}/halib/exp/core/wandb_op.py +0 -0
  23. {halib-0.2.17 → halib-0.2.19}/halib/exp/data/__init__.py +0 -0
  24. {halib-0.2.17 → halib-0.2.19}/halib/exp/data/dataclass_util.py +0 -0
  25. {halib-0.2.17 → halib-0.2.19}/halib/exp/data/dataset.py +0 -0
  26. {halib-0.2.17 → halib-0.2.19}/halib/exp/data/torchloader.py +0 -0
  27. {halib-0.2.17 → halib-0.2.19}/halib/exp/perf/__init__.py +0 -0
  28. {halib-0.2.17 → halib-0.2.19}/halib/exp/perf/flop_calc.py +0 -0
  29. {halib-0.2.17 → halib-0.2.19}/halib/exp/perf/gpu_mon.py +0 -0
  30. {halib-0.2.17 → halib-0.2.19}/halib/exp/perf/perfcalc.py +0 -0
  31. {halib-0.2.17 → halib-0.2.19}/halib/exp/perf/perfmetrics.py +0 -0
  32. {halib-0.2.17 → halib-0.2.19}/halib/exp/perf/perftb.py +0 -0
  33. {halib-0.2.17 → halib-0.2.19}/halib/exp/perf/profiler.py +0 -0
  34. {halib-0.2.17 → halib-0.2.19}/halib/exp/viz/__init__.py +0 -0
  35. {halib-0.2.17 → halib-0.2.19}/halib/exp/viz/plot.py +0 -0
  36. {halib-0.2.17 → halib-0.2.19}/halib/filetype/__init__.py +0 -0
  37. {halib-0.2.17 → halib-0.2.19}/halib/filetype/csvfile.py +0 -0
  38. {halib-0.2.17 → halib-0.2.19}/halib/filetype/ipynb.py +0 -0
  39. {halib-0.2.17 → halib-0.2.19}/halib/filetype/jsonfile.py +0 -0
  40. {halib-0.2.17 → halib-0.2.19}/halib/filetype/textfile.py +0 -0
  41. {halib-0.2.17 → halib-0.2.19}/halib/filetype/videofile.py +0 -0
  42. {halib-0.2.17 → halib-0.2.19}/halib/filetype/yamlfile.py +0 -0
  43. {halib-0.2.17 → halib-0.2.19}/halib/online/__init__.py +0 -0
  44. {halib-0.2.17 → halib-0.2.19}/halib/online/gdrive.py +0 -0
  45. {halib-0.2.17 → halib-0.2.19}/halib/online/gdrive_mkdir.py +0 -0
  46. {halib-0.2.17 → halib-0.2.19}/halib/online/projectmake.py +0 -0
  47. {halib-0.2.17 → halib-0.2.19}/halib/online/tele_noti.py +0 -0
  48. {halib-0.2.17 → halib-0.2.19}/halib/system/__init__.py +0 -0
  49. {halib-0.2.17 → halib-0.2.19}/halib/system/_list_pc.csv +0 -0
  50. {halib-0.2.17 → halib-0.2.19}/halib/system/cmd.py +0 -0
  51. {halib-0.2.17 → halib-0.2.19}/halib/system/filesys.py +0 -0
  52. {halib-0.2.17 → halib-0.2.19}/halib/system/path.py +0 -0
  53. {halib-0.2.17 → halib-0.2.19}/halib/utils/__init__.py +0 -0
  54. {halib-0.2.17 → halib-0.2.19}/halib/utils/dict.py +0 -0
  55. {halib-0.2.17 → halib-0.2.19}/halib/utils/list.py +0 -0
  56. {halib-0.2.17 → halib-0.2.19}/halib.egg-info/SOURCES.txt +0 -0
  57. {halib-0.2.17 → halib-0.2.19}/halib.egg-info/dependency_links.txt +0 -0
  58. {halib-0.2.17 → halib-0.2.19}/halib.egg-info/requires.txt +0 -0
  59. {halib-0.2.17 → halib-0.2.19}/halib.egg-info/top_level.txt +0 -0
  60. {halib-0.2.17 → halib-0.2.19}/setup.cfg +0 -0
halib-0.2.19/PKG-INFO ADDED
@@ -0,0 +1,187 @@
1
+ Metadata-Version: 2.4
2
+ Name: halib
3
+ Version: 0.2.19
4
+ Summary: Small library for common tasks
5
+ Author: Hoang Van Ha
6
+ Author-email: hoangvanhauit@gmail.com
7
+ Classifier: Programming Language :: Python :: 3
8
+ Classifier: License :: OSI Approved :: MIT License
9
+ Classifier: Operating System :: OS Independent
10
+ Requires-Python: >=3.9
11
+ Description-Content-Type: text/markdown
12
+ License-File: LICENSE.txt
13
+ Requires-Dist: arrow
14
+ Requires-Dist: click
15
+ Requires-Dist: dataclass-wizard
16
+ Requires-Dist: enlighten
17
+ Requires-Dist: itables
18
+ Requires-Dist: kaleido
19
+ Requires-Dist: loguru
20
+ Requires-Dist: matplotlib
21
+ Requires-Dist: more-itertools
22
+ Requires-Dist: moviepy
23
+ Requires-Dist: networkx
24
+ Requires-Dist: numpy
25
+ Requires-Dist: omegaconf
26
+ Requires-Dist: opencv-python
27
+ Requires-Dist: pandas
28
+ Requires-Dist: Pillow
29
+ Requires-Dist: plotly
30
+ Requires-Dist: Pyarrow
31
+ Requires-Dist: pycurl
32
+ Requires-Dist: pygwalker
33
+ Requires-Dist: python-telegram-bot
34
+ Requires-Dist: requests
35
+ Requires-Dist: rich
36
+ Requires-Dist: scikit-learn
37
+ Requires-Dist: seaborn
38
+ Requires-Dist: tabulate
39
+ Requires-Dist: timebudget
40
+ Requires-Dist: tqdm
41
+ Requires-Dist: tube_dl
42
+ Requires-Dist: wandb
43
+ Requires-Dist: ipynbname
44
+ Requires-Dist: typed-argument-parser
45
+ Dynamic: author
46
+ Dynamic: author-email
47
+ Dynamic: classifier
48
+ Dynamic: description
49
+ Dynamic: description-content-type
50
+ Dynamic: license-file
51
+ Dynamic: requires-dist
52
+ Dynamic: requires-python
53
+ Dynamic: summary
54
+
55
+ # 📦 Helper Package for Coding and Automation Changelog
56
+
57
+ ## v0.2.x (Experiment & Core Updates)
58
+
59
+ ### **v0.2.19**
60
+ - ✨ **New Feature:** Added `exp.core.param_gen` to facilitate fast generation of parameter combination sweeps (grid search) using YAML configurations.
61
+
62
+ ### **v0.2.17**
63
+ - 🚀 **Improvement:** Updated `exp.perf.profiler` with an `enabled` flag for dynamic toggling.
64
+ - 🚀 **Improvement:** Added a `measure` context manager to simplify performance measuring of code blocks.
65
+
66
+ ### **v0.2.13**
67
+ - ♻️ **Refactor:** Major reorganization of packages. Renamed `research` package to `exp` (Experiment Management).
68
+ - 🚀 **Improvement:** Updated `exp/perfcalc.py` to allow saving computed performance metrics to CSV without explicitly calling `calc_perfs`.
69
+
70
+ ### **v0.2.1**
71
+ - ✨ **New Feature:** Added `eval_exp` method to `exp/base_exp` for running evaluations (e.g., model testing) after experiments conclude.
72
+
73
+ ---
74
+
75
+ ## v0.1.9x (Visualization & Generators)
76
+
77
+ ### **v0.1.99**
78
+ - ✨ **New Feature:** Added `gen_ipynb_name` to `filetype/ipynb`. Generates filenames based on the current notebook name with optional timestamps.
79
+
80
+ ### **v0.1.96**
81
+ - ✨ **New Feature:** Added `PlotHelper` class in `research/plot` for plotting training history and image grids (dataset samples or model outputs).
82
+
83
+ ### **v0.1.91**
84
+ - ✨ **New Feature:** Added `ParamGen` class to `research/param_gen` for parsing YAML files into parameter lists for hyperparameter searches.
85
+
86
+ ### **v0.1.90**
87
+ - ✨ **New Feature:** Added `zProfiler` class to `research/profiler` for measuring context/step execution time, supporting dynamic color scales in plots.
88
+
89
+ ---
90
+
91
+ ## v0.1.5x - v0.1.7x (Infrastructure & Utilities)
92
+
93
+ ### **v0.1.77**
94
+ - ✨ **New Feature:** Added `BaseExp` class in `research/base_exp` to handle common experiment tasks (performance calculation, result saving).
95
+
96
+ ### **v0.1.67**
97
+ - 🔧 **Maintenance:** Switched to **uv** for virtual environment management.
98
+ - 🚀 **Improvement:** Updated `research/perfcalc` to support both `torchmetrics` and custom metrics.
99
+
100
+ ### **v0.1.61**
101
+ - ✨ **New Feature:** Added `VideoUtils` (`util/video`) for common video handling tasks.
102
+ - ✨ **New Feature:** Added `GPUMonitor` (`util/gpu_mon`) for tracking GPU usage and performance.
103
+
104
+ ### **v0.1.59**
105
+ - 🔨 **Architecture:** Added `util/perfcalc` abstract base class. This requires implementation of specific performance calculation logic.
106
+
107
+ ### **v0.1.55**
108
+ - ✨ **New Feature:** Added `util/dataclass_util` for dynamic creation of `dataclass` objects from dictionaries or YAML (supports nested structures).
109
+
110
+ ### **v0.1.52**
111
+ - ✨ **New Feature:** Added `research/perftb` module for managing experiment performance tables (filtering by dataset, metric, etc.).
112
+
113
+ ### **v0.1.50**
114
+ - ✨ **New Feature:** Added `pprint_local_path` to print clickable file URIs for local paths.
115
+ - ✨ **New Feature:** Added `research` package containing `benchquery` for dataframe benchmarking.
116
+ - ✨ **New Feature:** Added `wandb` module for offline syncing and batch clearing of Weights & Biases runs.
117
+
118
+ ---
119
+
120
+ ## v0.1.4x (Display & formatting)
121
+
122
+ ### **v0.1.47**
123
+ - ✨ **New Feature:** Added `pprint_box` to print objects or strings inside a decorative box frame.
124
+
125
+ ### **v0.1.46**
126
+ - 🐛 **Fix:** Filtered `UserWarning: Unable to import Axes3D`.
127
+ - 🚀 **Improvement:** Added `auto_wrap_text` to `fn_display_df` to prevent long text overflow in tables.
128
+
129
+ ### **v0.1.42**
130
+ - ✨ **New Feature:** Added `rich_color.py` wrapper for basic color lists.
131
+
132
+ ### **v0.1.41**
133
+ - ✨ **New Feature:** Added `rich_color.py` to support rich color information (palettes, strings) using the `rich` library.
134
+
135
+ ### **v0.1.40**
136
+ - 🚀 **Improvement:** Updated `csvfile.py` to use `itables` and `pygwalker` for interactive dataframe display in Jupyter notebooks.
137
+
138
+ ---
139
+
140
+ ## v0.1.3x (Data & Loading)
141
+
142
+ ### **v0.1.38**
143
+ - ✨ **New Feature:** Added `torchloader.py` to search for optimal `DataLoader` configurations (num_workers, batch_size, pin_memory).
144
+
145
+ ### **v0.1.37**
146
+ - ✨ **New Feature:** Added `dataset.py` for splitting classification datasets into train/val/test sets.
147
+
148
+ ### **v0.1.33**
149
+ - ✨ **New Feature:** Added `plot.py` for plotting Deep Learning training history (accuracy/loss) using `seaborn` and `matplotlib`.
150
+
151
+ ---
152
+
153
+ ## v0.1.0 - v0.1.2x (Early Utilities)
154
+
155
+ ### **v0.1.29**
156
+ - 🐛 **Fix:** Pinned `kaleido==0.1.*` for `tele_noti` as version `0.2.*` caused image generation hangs.
157
+
158
+ ### **v0.1.24**
159
+ - ♻️ **Refactor:** Renamed `sys` module to `system` to avoid conflicts with Python's built-in `sys`.
160
+ - ✨ **New Feature:** Added `tele_noti` module for Telegram notifications regarding training progress.
161
+
162
+ ### **v0.1.22**
163
+ - ✨ **New Feature:** Added `cuda.py` to check CUDA availability for both PyTorch and TensorFlow.
164
+
165
+ ### **v0.1.21**
166
+ - ✨ **New Feature:** Added YAML inheritance and overriding support using `networkx` and `omegaconf`.
167
+
168
+ ### **v0.1.15**
169
+ - ✨ **New Feature:** Added common logging library and `@console_log` decorator for function tracing.
170
+
171
+ ### **v0.1.10**
172
+ - 🐛 **Fix:** Fixed typo `is_exit` -> `is_exist` in `filesys`.
173
+ - 🚀 **Improvement:** Updated `gdrive` to support uploading to specific folders and returning direct shareable links.
174
+
175
+ ### **v0.1.9**
176
+ - 🔧 **Maintenance:** Added `requirements.txt`.
177
+
178
+ ### **v0.1.6 - v0.1.8**
179
+ - 🚀 **Performance:** Optimized table insertion by using an in-memory `row_pool_dict` before committing to the DataFrame.
180
+ - ✨ **New Feature:** Added `DFCreator` for manipulating DataFrames (create, insert, display, save).
181
+
182
+ ### **v0.1.4 - v0.1.5**
183
+ - ✨ **New Feature:** Added `cmd` module.
184
+ - ✨ **New Feature:** Support for creating Bitbucket Projects from templates.
185
+
186
+ ### **v0.1.2**
187
+ - ✨ **New Feature:** Added support for uploading local files to Google Drive.
halib-0.2.19/README.md ADDED
@@ -0,0 +1,133 @@
1
+ # 📦 Helper Package for Coding and Automation Changelog
2
+
3
+ ## v0.2.x (Experiment & Core Updates)
4
+
5
+ ### **v0.2.19**
6
+ - ✨ **New Feature:** Added `exp.core.param_gen` to facilitate fast generation of parameter combination sweeps (grid search) using YAML configurations.
7
+
8
+ ### **v0.2.17**
9
+ - 🚀 **Improvement:** Updated `exp.perf.profiler` with an `enabled` flag for dynamic toggling.
10
+ - 🚀 **Improvement:** Added a `measure` context manager to simplify performance measuring of code blocks.
11
+
12
+ ### **v0.2.13**
13
+ - ♻️ **Refactor:** Major reorganization of packages. Renamed `research` package to `exp` (Experiment Management).
14
+ - 🚀 **Improvement:** Updated `exp/perfcalc.py` to allow saving computed performance metrics to CSV without explicitly calling `calc_perfs`.
15
+
16
+ ### **v0.2.1**
17
+ - ✨ **New Feature:** Added `eval_exp` method to `exp/base_exp` for running evaluations (e.g., model testing) after experiments conclude.
18
+
19
+ ---
20
+
21
+ ## v0.1.9x (Visualization & Generators)
22
+
23
+ ### **v0.1.99**
24
+ - ✨ **New Feature:** Added `gen_ipynb_name` to `filetype/ipynb`. Generates filenames based on the current notebook name with optional timestamps.
25
+
26
+ ### **v0.1.96**
27
+ - ✨ **New Feature:** Added `PlotHelper` class in `research/plot` for plotting training history and image grids (dataset samples or model outputs).
28
+
29
+ ### **v0.1.91**
30
+ - ✨ **New Feature:** Added `ParamGen` class to `research/param_gen` for parsing YAML files into parameter lists for hyperparameter searches.
31
+
32
+ ### **v0.1.90**
33
+ - ✨ **New Feature:** Added `zProfiler` class to `research/profiler` for measuring context/step execution time, supporting dynamic color scales in plots.
34
+
35
+ ---
36
+
37
+ ## v0.1.5x - v0.1.7x (Infrastructure & Utilities)
38
+
39
+ ### **v0.1.77**
40
+ - ✨ **New Feature:** Added `BaseExp` class in `research/base_exp` to handle common experiment tasks (performance calculation, result saving).
41
+
42
+ ### **v0.1.67**
43
+ - 🔧 **Maintenance:** Switched to **uv** for virtual environment management.
44
+ - 🚀 **Improvement:** Updated `research/perfcalc` to support both `torchmetrics` and custom metrics.
45
+
46
+ ### **v0.1.61**
47
+ - ✨ **New Feature:** Added `VideoUtils` (`util/video`) for common video handling tasks.
48
+ - ✨ **New Feature:** Added `GPUMonitor` (`util/gpu_mon`) for tracking GPU usage and performance.
49
+
50
+ ### **v0.1.59**
51
+ - 🔨 **Architecture:** Added `util/perfcalc` abstract base class. This requires implementation of specific performance calculation logic.
52
+
53
+ ### **v0.1.55**
54
+ - ✨ **New Feature:** Added `util/dataclass_util` for dynamic creation of `dataclass` objects from dictionaries or YAML (supports nested structures).
55
+
56
+ ### **v0.1.52**
57
+ - ✨ **New Feature:** Added `research/perftb` module for managing experiment performance tables (filtering by dataset, metric, etc.).
58
+
59
+ ### **v0.1.50**
60
+ - ✨ **New Feature:** Added `pprint_local_path` to print clickable file URIs for local paths.
61
+ - ✨ **New Feature:** Added `research` package containing `benchquery` for dataframe benchmarking.
62
+ - ✨ **New Feature:** Added `wandb` module for offline syncing and batch clearing of Weights & Biases runs.
63
+
64
+ ---
65
+
66
+ ## v0.1.4x (Display & formatting)
67
+
68
+ ### **v0.1.47**
69
+ - ✨ **New Feature:** Added `pprint_box` to print objects or strings inside a decorative box frame.
70
+
71
+ ### **v0.1.46**
72
+ - 🐛 **Fix:** Filtered `UserWarning: Unable to import Axes3D`.
73
+ - 🚀 **Improvement:** Added `auto_wrap_text` to `fn_display_df` to prevent long text overflow in tables.
74
+
75
+ ### **v0.1.42**
76
+ - ✨ **New Feature:** Added `rich_color.py` wrapper for basic color lists.
77
+
78
+ ### **v0.1.41**
79
+ - ✨ **New Feature:** Added `rich_color.py` to support rich color information (palettes, strings) using the `rich` library.
80
+
81
+ ### **v0.1.40**
82
+ - 🚀 **Improvement:** Updated `csvfile.py` to use `itables` and `pygwalker` for interactive dataframe display in Jupyter notebooks.
83
+
84
+ ---
85
+
86
+ ## v0.1.3x (Data & Loading)
87
+
88
+ ### **v0.1.38**
89
+ - ✨ **New Feature:** Added `torchloader.py` to search for optimal `DataLoader` configurations (num_workers, batch_size, pin_memory).
90
+
91
+ ### **v0.1.37**
92
+ - ✨ **New Feature:** Added `dataset.py` for splitting classification datasets into train/val/test sets.
93
+
94
+ ### **v0.1.33**
95
+ - ✨ **New Feature:** Added `plot.py` for plotting Deep Learning training history (accuracy/loss) using `seaborn` and `matplotlib`.
96
+
97
+ ---
98
+
99
+ ## v0.1.0 - v0.1.2x (Early Utilities)
100
+
101
+ ### **v0.1.29**
102
+ - 🐛 **Fix:** Pinned `kaleido==0.1.*` for `tele_noti` as version `0.2.*` caused image generation hangs.
103
+
104
+ ### **v0.1.24**
105
+ - ♻️ **Refactor:** Renamed `sys` module to `system` to avoid conflicts with Python's built-in `sys`.
106
+ - ✨ **New Feature:** Added `tele_noti` module for Telegram notifications regarding training progress.
107
+
108
+ ### **v0.1.22**
109
+ - ✨ **New Feature:** Added `cuda.py` to check CUDA availability for both PyTorch and TensorFlow.
110
+
111
+ ### **v0.1.21**
112
+ - ✨ **New Feature:** Added YAML inheritance and overriding support using `networkx` and `omegaconf`.
113
+
114
+ ### **v0.1.15**
115
+ - ✨ **New Feature:** Added common logging library and `@console_log` decorator for function tracing.
116
+
117
+ ### **v0.1.10**
118
+ - 🐛 **Fix:** Fixed typo `is_exit` -> `is_exist` in `filesys`.
119
+ - 🚀 **Improvement:** Updated `gdrive` to support uploading to specific folders and returning direct shareable links.
120
+
121
+ ### **v0.1.9**
122
+ - 🔧 **Maintenance:** Added `requirements.txt`.
123
+
124
+ ### **v0.1.6 - v0.1.8**
125
+ - 🚀 **Performance:** Optimized table insertion by using an in-memory `row_pool_dict` before committing to the DataFrame.
126
+ - ✨ **New Feature:** Added `DFCreator` for manipulating DataFrames (create, insert, display, save).
127
+
128
+ ### **v0.1.4 - v0.1.5**
129
+ - ✨ **New Feature:** Added `cmd` module.
130
+ - ✨ **New Feature:** Support for creating Bitbucket Projects from templates.
131
+
132
+ ### **v0.1.2**
133
+ - ✨ **New Feature:** Added support for uploading local files to Google Drive.
@@ -0,0 +1,189 @@
1
+ import os
2
+ import copy
3
+ import numpy as np
4
+ from itertools import product
5
+ from typing import Dict, Any, List, Iterator, Optional
6
+ from ...filetype import yamlfile
7
+
8
+ class ParamGen:
9
+ """
10
+ A flexible parameter grid generator for hyperparameter tuning and experiment management.
11
+
12
+ This class generates a Cartesian product of parameters from a "sweep configuration"
13
+ and optionally merges them into a "base configuration". It abstracts away the complexity
14
+ of handling nested dictionaries and range generation.
15
+
16
+ Key Features:
17
+ -----------
18
+ 1. **Flexible Syntax**: Define parameters using standard nested dictionaries or
19
+ dot-notation keys (e.g., `'model.backbone.layers'`).
20
+ 2. **Range Shortcuts**:
21
+ - **Choices**: Standard lists `[1, 2, 3]`.
22
+ - **String Ranges**: `"start:stop:step"` (e.g., `"0:10:2"` -> `[0, 2, 4, 6, 8]`).
23
+ - **Dict Ranges**: `{'start': 0, 'stop': 1, 'step': 0.1}`.
24
+ 3. **Deep Merging**: Automatically updates deep keys in `base_cfg` without overwriting siblings.
25
+
26
+ Example:
27
+ --------
28
+ >>> base = {'model': {'name': 'resnet', 'dropout': 0.1}, 'seed': 42}
29
+ >>> sweep = {
30
+ ... 'model.name': ['resnet', 'vit'], # Dot notation
31
+ ... 'model.dropout': "0.1:0.3:0.1", # Range string
32
+ ... 'seed': [42, 100] # Simple choice
33
+ ... }
34
+ >>> grid = ParamGen(sweep, base)
35
+ >>> configs = grid.expand()
36
+ >>> print(len(configs)) # Outputs: 8 (2 models * 2 dropouts * 2 seeds)
37
+ Attributes:
38
+ keys (List[str]): List of flattened dot-notation keys being swept.
39
+ values (List[List[Any]]): List of value options for each key.
40
+ """
41
+ def __init__(
42
+ self, sweep_cfg: Dict[str, Any], base_cfg: Optional[Dict[str, Any]] = None
43
+ ):
44
+ """
45
+ Args:
46
+ sweep_cfg: The dictionary defining parameters to sweep.
47
+ base_cfg: (Optional) The base config to merge sweep parameters into.
48
+ If None, expand() behaves like expand_sweep().
49
+ """
50
+ self.base_cfg = base_cfg if base_cfg is not None else {}
51
+
52
+ # Recursively flatten the nested sweep config into dot-notation keys
53
+ self.param_space = self._flatten_params(sweep_cfg)
54
+ self.keys = list(self.param_space.keys())
55
+ self.values = list(self.param_space.values())
56
+
57
+ def get_param_space(self) -> Dict[str, List[Any]]:
58
+ """Returns the parameter space as a dictionary of dot-notation keys to value lists."""
59
+ return self.param_space
60
+
61
+ def __iter__(self) -> Iterator[Dict[str, Any]]:
62
+ """Yields fully merged configurations one by one."""
63
+ for combination in product(*self.values):
64
+ # 1. Create the flat sweep dict (dot notation)
65
+ flat_params = dict(zip(self.keys, combination))
66
+
67
+ # 2. Deep copy base and update with current params
68
+ new_cfg = copy.deepcopy(self.base_cfg)
69
+ new_cfg = self._apply_updates(new_cfg, flat_params)
70
+
71
+ # 3. Store metadata (Optional)
72
+ # if "_meta" not in new_cfg:
73
+ # new_cfg["_meta"] = {}
74
+ # We unflatten the sweep params here so the log is readable
75
+ # new_cfg["_meta"]["sweep_params"] = self._unflatten(flat_params)
76
+
77
+ yield new_cfg
78
+
79
+ # ! --- Factory Methods ---
80
+ @classmethod
81
+ def from_dicts(
82
+ cls, sweep_cfg: Dict[str, Any], base_cfg: Optional[Dict[str, Any]] = None
83
+ ):
84
+ """
85
+ Load from dictionaries.
86
+ Args:
87
+ sweep_cfg: The dictionary defining parameters to sweep.
88
+ base_cfg: (Optional) The base config to merge sweep parameters into.
89
+ """
90
+ return cls(sweep_cfg, base_cfg)
91
+
92
+ @classmethod
93
+ def from_files(cls, sweep_yaml: str, base_yaml: Optional[str] = None):
94
+ """
95
+ Load from files.
96
+ Args:
97
+ sweep_yaml: Path to sweep config.
98
+ base_yaml: (Optional) Path to base config.
99
+ """
100
+ assert os.path.isfile(sweep_yaml), f"Sweep file not found: {sweep_yaml}"
101
+ sweep_dict = yamlfile.load_yaml(sweep_yaml, to_dict=True)
102
+ base_dict = None
103
+ if base_yaml:
104
+ base_dict = yamlfile.load_yaml(base_yaml, to_dict=True)
105
+ if "__base__" in base_dict:
106
+ del base_dict["__base__"]
107
+
108
+ return cls(sweep_dict, base_dict)
109
+
110
+ def expand(self) -> List[Dict[str, Any]]:
111
+ """Generates and returns the full list of MERGED configurations."""
112
+ return list(self)
113
+
114
+ def expand_sweep_flat(self) -> List[Dict[str, Any]]:
115
+ """
116
+ Returns a list of ONLY the sweep parameters, formatted as FLAT dot-notation dictionaries.
117
+
118
+ Returns:
119
+ [{'exp_params.model': 'resnet', 'exp_params.lr': 0.01}, ...]
120
+ """
121
+ combinations = []
122
+ for combination in product(*self.values):
123
+ flat_dict = dict(zip(self.keys, combination))
124
+ combinations.append(flat_dict)
125
+ return combinations
126
+
127
+ def _unflatten(self, flat_dict: Dict[str, Any]) -> Dict[str, Any]:
128
+ """Converts {'a.b': 1} back to {'a': {'b': 1}}."""
129
+ nested = {}
130
+ self._apply_updates(nested, flat_dict)
131
+ return nested
132
+
133
+ def _flatten_params(
134
+ self, cfg: Dict[str, Any], parent_key: str = ""
135
+ ) -> Dict[str, List[Any]]:
136
+ """Recursively converts nested dicts into flat dot-notation keys."""
137
+ flat = {}
138
+ for key, val in cfg.items():
139
+ current_key = f"{parent_key}.{key}" if parent_key else key
140
+
141
+ if self._is_sweep_leaf(val):
142
+ flat[current_key] = self._expand_val(val)
143
+ elif isinstance(val, dict):
144
+ flat.update(self._flatten_params(val, current_key))
145
+ else:
146
+ flat[current_key] = [val]
147
+ return flat
148
+
149
+ def _is_sweep_leaf(self, val: Any) -> bool:
150
+ if isinstance(val, list):
151
+ return True
152
+ if isinstance(val, str) and ":" in val:
153
+ return True
154
+ if isinstance(val, dict) and "start" in val and "stop" in val:
155
+ return True
156
+ return False
157
+
158
+ def _expand_val(self, val: Any) -> List[Any]:
159
+ if isinstance(val, list):
160
+ return val
161
+
162
+ if isinstance(val, str) and ":" in val:
163
+ try:
164
+ parts = [float(x) for x in val.split(":")]
165
+ if len(parts) == 3:
166
+ arr = np.arange(parts[0], parts[1], parts[2])
167
+ return [float(f"{x:.6g}") for x in arr]
168
+ except ValueError:
169
+ pass
170
+
171
+ if isinstance(val, dict) and "start" in val:
172
+ step = val.get("step", 1)
173
+ return np.arange(val["start"], val["stop"], step).tolist()
174
+
175
+ return [val]
176
+
177
+ def _apply_updates(
178
+ self, cfg: Dict[str, Any], updates: Dict[str, Any]
179
+ ) -> Dict[str, Any]:
180
+ """Deep merges dot-notation updates into cfg."""
181
+ for key, val in updates.items():
182
+ parts = key.split(".")
183
+ target = cfg
184
+ for part in parts[:-1]:
185
+ if part not in target:
186
+ target[part] = {}
187
+ target = target[part]
188
+ target[parts[-1]] = val
189
+ return cfg
@@ -0,0 +1,187 @@
1
+ Metadata-Version: 2.4
2
+ Name: halib
3
+ Version: 0.2.19
4
+ Summary: Small library for common tasks
5
+ Author: Hoang Van Ha
6
+ Author-email: hoangvanhauit@gmail.com
7
+ Classifier: Programming Language :: Python :: 3
8
+ Classifier: License :: OSI Approved :: MIT License
9
+ Classifier: Operating System :: OS Independent
10
+ Requires-Python: >=3.9
11
+ Description-Content-Type: text/markdown
12
+ License-File: LICENSE.txt
13
+ Requires-Dist: arrow
14
+ Requires-Dist: click
15
+ Requires-Dist: dataclass-wizard
16
+ Requires-Dist: enlighten
17
+ Requires-Dist: itables
18
+ Requires-Dist: kaleido
19
+ Requires-Dist: loguru
20
+ Requires-Dist: matplotlib
21
+ Requires-Dist: more-itertools
22
+ Requires-Dist: moviepy
23
+ Requires-Dist: networkx
24
+ Requires-Dist: numpy
25
+ Requires-Dist: omegaconf
26
+ Requires-Dist: opencv-python
27
+ Requires-Dist: pandas
28
+ Requires-Dist: Pillow
29
+ Requires-Dist: plotly
30
+ Requires-Dist: Pyarrow
31
+ Requires-Dist: pycurl
32
+ Requires-Dist: pygwalker
33
+ Requires-Dist: python-telegram-bot
34
+ Requires-Dist: requests
35
+ Requires-Dist: rich
36
+ Requires-Dist: scikit-learn
37
+ Requires-Dist: seaborn
38
+ Requires-Dist: tabulate
39
+ Requires-Dist: timebudget
40
+ Requires-Dist: tqdm
41
+ Requires-Dist: tube_dl
42
+ Requires-Dist: wandb
43
+ Requires-Dist: ipynbname
44
+ Requires-Dist: typed-argument-parser
45
+ Dynamic: author
46
+ Dynamic: author-email
47
+ Dynamic: classifier
48
+ Dynamic: description
49
+ Dynamic: description-content-type
50
+ Dynamic: license-file
51
+ Dynamic: requires-dist
52
+ Dynamic: requires-python
53
+ Dynamic: summary
54
+
55
+ # 📦 Helper Package for Coding and Automation Changelog
56
+
57
+ ## v0.2.x (Experiment & Core Updates)
58
+
59
+ ### **v0.2.19**
60
+ - ✨ **New Feature:** Added `exp.core.param_gen` to facilitate fast generation of parameter combination sweeps (grid search) using YAML configurations.
61
+
62
+ ### **v0.2.17**
63
+ - 🚀 **Improvement:** Updated `exp.perf.profiler` with an `enabled` flag for dynamic toggling.
64
+ - 🚀 **Improvement:** Added a `measure` context manager to simplify performance measuring of code blocks.
65
+
66
+ ### **v0.2.13**
67
+ - ♻️ **Refactor:** Major reorganization of packages. Renamed `research` package to `exp` (Experiment Management).
68
+ - 🚀 **Improvement:** Updated `exp/perfcalc.py` to allow saving computed performance metrics to CSV without explicitly calling `calc_perfs`.
69
+
70
+ ### **v0.2.1**
71
+ - ✨ **New Feature:** Added `eval_exp` method to `exp/base_exp` for running evaluations (e.g., model testing) after experiments conclude.
72
+
73
+ ---
74
+
75
+ ## v0.1.9x (Visualization & Generators)
76
+
77
+ ### **v0.1.99**
78
+ - ✨ **New Feature:** Added `gen_ipynb_name` to `filetype/ipynb`. Generates filenames based on the current notebook name with optional timestamps.
79
+
80
+ ### **v0.1.96**
81
+ - ✨ **New Feature:** Added `PlotHelper` class in `research/plot` for plotting training history and image grids (dataset samples or model outputs).
82
+
83
+ ### **v0.1.91**
84
+ - ✨ **New Feature:** Added `ParamGen` class to `research/param_gen` for parsing YAML files into parameter lists for hyperparameter searches.
85
+
86
+ ### **v0.1.90**
87
+ - ✨ **New Feature:** Added `zProfiler` class to `research/profiler` for measuring context/step execution time, supporting dynamic color scales in plots.
88
+
89
+ ---
90
+
91
+ ## v0.1.5x - v0.1.7x (Infrastructure & Utilities)
92
+
93
+ ### **v0.1.77**
94
+ - ✨ **New Feature:** Added `BaseExp` class in `research/base_exp` to handle common experiment tasks (performance calculation, result saving).
95
+
96
+ ### **v0.1.67**
97
+ - 🔧 **Maintenance:** Switched to **uv** for virtual environment management.
98
+ - 🚀 **Improvement:** Updated `research/perfcalc` to support both `torchmetrics` and custom metrics.
99
+
100
+ ### **v0.1.61**
101
+ - ✨ **New Feature:** Added `VideoUtils` (`util/video`) for common video handling tasks.
102
+ - ✨ **New Feature:** Added `GPUMonitor` (`util/gpu_mon`) for tracking GPU usage and performance.
103
+
104
+ ### **v0.1.59**
105
+ - 🔨 **Architecture:** Added `util/perfcalc` abstract base class. This requires implementation of specific performance calculation logic.
106
+
107
+ ### **v0.1.55**
108
+ - ✨ **New Feature:** Added `util/dataclass_util` for dynamic creation of `dataclass` objects from dictionaries or YAML (supports nested structures).
109
+
110
+ ### **v0.1.52**
111
+ - ✨ **New Feature:** Added `research/perftb` module for managing experiment performance tables (filtering by dataset, metric, etc.).
112
+
113
+ ### **v0.1.50**
114
+ - ✨ **New Feature:** Added `pprint_local_path` to print clickable file URIs for local paths.
115
+ - ✨ **New Feature:** Added `research` package containing `benchquery` for dataframe benchmarking.
116
+ - ✨ **New Feature:** Added `wandb` module for offline syncing and batch clearing of Weights & Biases runs.
117
+
118
+ ---
119
+
120
+ ## v0.1.4x (Display & formatting)
121
+
122
+ ### **v0.1.47**
123
+ - ✨ **New Feature:** Added `pprint_box` to print objects or strings inside a decorative box frame.
124
+
125
+ ### **v0.1.46**
126
+ - 🐛 **Fix:** Filtered `UserWarning: Unable to import Axes3D`.
127
+ - 🚀 **Improvement:** Added `auto_wrap_text` to `fn_display_df` to prevent long text overflow in tables.
128
+
129
+ ### **v0.1.42**
130
+ - ✨ **New Feature:** Added `rich_color.py` wrapper for basic color lists.
131
+
132
+ ### **v0.1.41**
133
+ - ✨ **New Feature:** Added `rich_color.py` to support rich color information (palettes, strings) using the `rich` library.
134
+
135
+ ### **v0.1.40**
136
+ - 🚀 **Improvement:** Updated `csvfile.py` to use `itables` and `pygwalker` for interactive dataframe display in Jupyter notebooks.
137
+
138
+ ---
139
+
140
+ ## v0.1.3x (Data & Loading)
141
+
142
+ ### **v0.1.38**
143
+ - ✨ **New Feature:** Added `torchloader.py` to search for optimal `DataLoader` configurations (num_workers, batch_size, pin_memory).
144
+
145
+ ### **v0.1.37**
146
+ - ✨ **New Feature:** Added `dataset.py` for splitting classification datasets into train/val/test sets.
147
+
148
+ ### **v0.1.33**
149
+ - ✨ **New Feature:** Added `plot.py` for plotting Deep Learning training history (accuracy/loss) using `seaborn` and `matplotlib`.
150
+
151
+ ---
152
+
153
+ ## v0.1.0 - v0.1.2x (Early Utilities)
154
+
155
+ ### **v0.1.29**
156
+ - 🐛 **Fix:** Pinned `kaleido==0.1.*` for `tele_noti` as version `0.2.*` caused image generation hangs.
157
+
158
+ ### **v0.1.24**
159
+ - ♻️ **Refactor:** Renamed `sys` module to `system` to avoid conflicts with Python's built-in `sys`.
160
+ - ✨ **New Feature:** Added `tele_noti` module for Telegram notifications regarding training progress.
161
+
162
+ ### **v0.1.22**
163
+ - ✨ **New Feature:** Added `cuda.py` to check CUDA availability for both PyTorch and TensorFlow.
164
+
165
+ ### **v0.1.21**
166
+ - ✨ **New Feature:** Added YAML inheritance and overriding support using `networkx` and `omegaconf`.
167
+
168
+ ### **v0.1.15**
169
+ - ✨ **New Feature:** Added common logging library and `@console_log` decorator for function tracing.
170
+
171
+ ### **v0.1.10**
172
+ - 🐛 **Fix:** Fixed typo `is_exit` -> `is_exist` in `filesys`.
173
+ - 🚀 **Improvement:** Updated `gdrive` to support uploading to specific folders and returning direct shareable links.
174
+
175
+ ### **v0.1.9**
176
+ - 🔧 **Maintenance:** Added `requirements.txt`.
177
+
178
+ ### **v0.1.6 - v0.1.8**
179
+ - 🚀 **Performance:** Optimized table insertion by using an in-memory `row_pool_dict` before committing to the DataFrame.
180
+ - ✨ **New Feature:** Added `DFCreator` for manipulating DataFrames (create, insert, display, save).
181
+
182
+ ### **v0.1.4 - v0.1.5**
183
+ - ✨ **New Feature:** Added `cmd` module.
184
+ - ✨ **New Feature:** Support for creating Bitbucket Projects from templates.
185
+
186
+ ### **v0.1.2**
187
+ - ✨ **New Feature:** Added support for uploading local files to Google Drive.