paramflow 0.2.1__tar.gz → 0.2.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- paramflow-0.2.3/PKG-INFO +147 -0
- paramflow-0.2.3/README.md +129 -0
- paramflow-0.2.3/paramflow/__pycache__/params.cpython-312.pyc +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/paramflow/params.py +2 -1
- paramflow-0.2.3/paramflow.egg-info/PKG-INFO +147 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/setup.py +1 -1
- paramflow-0.2.1/PKG-INFO +0 -147
- paramflow-0.2.1/README.md +0 -129
- paramflow-0.2.1/paramflow/__pycache__/params.cpython-312.pyc +0 -0
- paramflow-0.2.1/paramflow.egg-info/PKG-INFO +0 -147
- {paramflow-0.2.1 → paramflow-0.2.3}/LICENSE +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/MANIFEST.in +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/paramflow/__init__.py +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/paramflow/__pycache__/__init__.cpython-312.pyc +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/paramflow/__pycache__/convert.cpython-312.pyc +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/paramflow/__pycache__/frozen.cpython-312.pyc +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/paramflow/__pycache__/frozen_test.cpython-312-pytest-8.3.4.pyc +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/paramflow/__pycache__/params_test.cpython-312-pytest-8.3.4.pyc +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/paramflow/__pycache__/parser.cpython-312.pyc +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/paramflow/convert.py +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/paramflow/frozen.py +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/paramflow/parser.py +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/paramflow.egg-info/SOURCES.txt +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/paramflow.egg-info/dependency_links.txt +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/paramflow.egg-info/requires.txt +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/paramflow.egg-info/top_level.txt +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/pyproject.toml +0 -0
- {paramflow-0.2.1 → paramflow-0.2.3}/setup.cfg +0 -0
paramflow-0.2.3/PKG-INFO
ADDED
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
Metadata-Version: 2.2
|
|
2
|
+
Name: paramflow
|
|
3
|
+
Version: 0.2.3
|
|
4
|
+
Home-page: https://github.com/mduszyk/paramflow
|
|
5
|
+
Classifier: Programming Language :: Python :: 3
|
|
6
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
7
|
+
Description-Content-Type: text/markdown
|
|
8
|
+
License-File: LICENSE
|
|
9
|
+
Requires-Dist: pyyaml
|
|
10
|
+
Provides-Extra: dotenv
|
|
11
|
+
Requires-Dist: python-dotenv; extra == "dotenv"
|
|
12
|
+
Dynamic: classifier
|
|
13
|
+
Dynamic: description
|
|
14
|
+
Dynamic: description-content-type
|
|
15
|
+
Dynamic: home-page
|
|
16
|
+
Dynamic: provides-extra
|
|
17
|
+
Dynamic: requires-dist
|
|
18
|
+
|
|
19
|
+
# paramflow
|
|
20
|
+
`paramflow` is a flexible and user-friendly parameter and configuration management library designed for
|
|
21
|
+
machine learning workflows and applications requiring profiles and layered parameters. It enables seamless
|
|
22
|
+
merging of parameters from multiple sources, auto-generates a command-line argument parser,
|
|
23
|
+
and allows for easy parameter overrides.
|
|
24
|
+
|
|
25
|
+
## Features
|
|
26
|
+
- **Layered configuration**: Merge parameters from files, environment variables, and command-line arguments.
|
|
27
|
+
- **Immutable dictionary**: Provides a read-only dictionary with attribute-style access.
|
|
28
|
+
- **Profile support**: Manage multiple sets of parameters with profile-based layering.
|
|
29
|
+
- **Layered meta-parameters**: `paramflow` configures itself using a layered approach.
|
|
30
|
+
- **Automatic type conversion**: Converts types during merging based on target parameter types.
|
|
31
|
+
- **Command-line argument parsing**: Automatically generates an `argparse` parser from parameter definitions.
|
|
32
|
+
- **Nested Configuration**: Allows for nested configuration and merging.
|
|
33
|
+
|
|
34
|
+
## Installation
|
|
35
|
+
```sh
|
|
36
|
+
pip install paramflow
|
|
37
|
+
```
|
|
38
|
+
Install with `.env` support:
|
|
39
|
+
```sh
|
|
40
|
+
pip install "paramflow[dotenv]"
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## Basic Usage
|
|
44
|
+
### Example Configuration File (`params.toml`)
|
|
45
|
+
```toml
|
|
46
|
+
[default]
|
|
47
|
+
learning_rate = 0.001
|
|
48
|
+
batch_size = 64
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
### Loading Parameters in Python (`app.py`)
|
|
52
|
+
```python
|
|
53
|
+
import paramflow as pf
|
|
54
|
+
|
|
55
|
+
params = pf.load('params.toml')
|
|
56
|
+
print(params.learning_rate) # 0.001
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
### Generating Command-line Help
|
|
60
|
+
Running the script with `--help` displays both meta-parameters and parameters:
|
|
61
|
+
```sh
|
|
62
|
+
python app.py --help
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
## Meta-Parameter Layering
|
|
66
|
+
Meta-parameters control how `paramflow.load` reads its own configuration. Layering order:
|
|
67
|
+
1. `paramflow.load` arguments
|
|
68
|
+
2. Environment variables (default prefix: `P_`)
|
|
69
|
+
3. Command-line arguments (`argparse`)
|
|
70
|
+
|
|
71
|
+
### Activating Profiles
|
|
72
|
+
Via command-line:
|
|
73
|
+
```sh
|
|
74
|
+
python print_params.py --profile dqn-adam
|
|
75
|
+
```
|
|
76
|
+
Via environment variable:
|
|
77
|
+
```sh
|
|
78
|
+
P_PROFILE=dqn-adam python print_params.py
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
## Parameter Layering
|
|
82
|
+
Parameters are merged from multiple sources in the following order:
|
|
83
|
+
1. Configuration files (`.toml`, `.yaml`, `.ini`, `.json`)
|
|
84
|
+
2. `.env` file (if enabled)
|
|
85
|
+
3. Environment variables (default prefix: `P_`)
|
|
86
|
+
4. Command-line arguments (`argparse`)
|
|
87
|
+
|
|
88
|
+
### Customizing Layering Order
|
|
89
|
+
You can specify the order explicitly:
|
|
90
|
+
```python
|
|
91
|
+
params = pf.load('params.toml', 'env', '.env', 'args')
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
### Overriding Parameters
|
|
95
|
+
Command-line arguments override other sources:
|
|
96
|
+
```sh
|
|
97
|
+
python print_params.py --profile dqn-adam --learning_rate 0.0002
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
## Managing ML Hyperparameter Profiles
|
|
101
|
+
### Example Configuration (`params.toml`)
|
|
102
|
+
```toml
|
|
103
|
+
[default]
|
|
104
|
+
learning_rate = 0.00025
|
|
105
|
+
batch_size = 32
|
|
106
|
+
optimizer_class = 'torch.optim.RMSprop'
|
|
107
|
+
optimizer_kwargs = { momentum = 0.95 }
|
|
108
|
+
random_seed = 13
|
|
109
|
+
|
|
110
|
+
[adam]
|
|
111
|
+
learning_rate = 1e-4
|
|
112
|
+
optimizer_class = 'torch.optim.Adam'
|
|
113
|
+
optimizer_kwargs = {}
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
### Activating a Profile
|
|
117
|
+
```sh
|
|
118
|
+
python app.py --profile adam
|
|
119
|
+
```
|
|
120
|
+
This overrides:
|
|
121
|
+
- `learning_rate` → `1e-4`
|
|
122
|
+
- `optimizer_class` → `torch.optim.Adam`
|
|
123
|
+
- `optimizer_kwargs` → `{}`
|
|
124
|
+
|
|
125
|
+
## Managing Development Stages
|
|
126
|
+
Profiles can be used to manage configurations for different environments.
|
|
127
|
+
|
|
128
|
+
### Example Configuration (`params.toml`)
|
|
129
|
+
```toml
|
|
130
|
+
[default]
|
|
131
|
+
debug = true
|
|
132
|
+
database_url = "mysql://localhost:3306/myapp"
|
|
133
|
+
|
|
134
|
+
[dev]
|
|
135
|
+
database_url = "mysql://dev:3306/myapp"
|
|
136
|
+
|
|
137
|
+
[prod]
|
|
138
|
+
debug = false
|
|
139
|
+
database_url = "mysql://prod:3306/myapp"
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
### Activating a Profile
|
|
143
|
+
```sh
|
|
144
|
+
export P_PROFILE=dev
|
|
145
|
+
python app.py
|
|
146
|
+
```
|
|
147
|
+
|
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
# paramflow
|
|
2
|
+
`paramflow` is a flexible and user-friendly parameter and configuration management library designed for
|
|
3
|
+
machine learning workflows and applications requiring profiles and layered parameters. It enables seamless
|
|
4
|
+
merging of parameters from multiple sources, auto-generates a command-line argument parser,
|
|
5
|
+
and allows for easy parameter overrides.
|
|
6
|
+
|
|
7
|
+
## Features
|
|
8
|
+
- **Layered configuration**: Merge parameters from files, environment variables, and command-line arguments.
|
|
9
|
+
- **Immutable dictionary**: Provides a read-only dictionary with attribute-style access.
|
|
10
|
+
- **Profile support**: Manage multiple sets of parameters with profile-based layering.
|
|
11
|
+
- **Layered meta-parameters**: `paramflow` configures itself using a layered approach.
|
|
12
|
+
- **Automatic type conversion**: Converts types during merging based on target parameter types.
|
|
13
|
+
- **Command-line argument parsing**: Automatically generates an `argparse` parser from parameter definitions.
|
|
14
|
+
- **Nested Configuration**: Allows for nested configuration and merging.
|
|
15
|
+
|
|
16
|
+
## Installation
|
|
17
|
+
```sh
|
|
18
|
+
pip install paramflow
|
|
19
|
+
```
|
|
20
|
+
Install with `.env` support:
|
|
21
|
+
```sh
|
|
22
|
+
pip install "paramflow[dotenv]"
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
## Basic Usage
|
|
26
|
+
### Example Configuration File (`params.toml`)
|
|
27
|
+
```toml
|
|
28
|
+
[default]
|
|
29
|
+
learning_rate = 0.001
|
|
30
|
+
batch_size = 64
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
### Loading Parameters in Python (`app.py`)
|
|
34
|
+
```python
|
|
35
|
+
import paramflow as pf
|
|
36
|
+
|
|
37
|
+
params = pf.load('params.toml')
|
|
38
|
+
print(params.learning_rate) # 0.001
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
### Generating Command-line Help
|
|
42
|
+
Running the script with `--help` displays both meta-parameters and parameters:
|
|
43
|
+
```sh
|
|
44
|
+
python app.py --help
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
## Meta-Parameter Layering
|
|
48
|
+
Meta-parameters control how `paramflow.load` reads its own configuration. Layering order:
|
|
49
|
+
1. `paramflow.load` arguments
|
|
50
|
+
2. Environment variables (default prefix: `P_`)
|
|
51
|
+
3. Command-line arguments (`argparse`)
|
|
52
|
+
|
|
53
|
+
### Activating Profiles
|
|
54
|
+
Via command-line:
|
|
55
|
+
```sh
|
|
56
|
+
python print_params.py --profile dqn-adam
|
|
57
|
+
```
|
|
58
|
+
Via environment variable:
|
|
59
|
+
```sh
|
|
60
|
+
P_PROFILE=dqn-adam python print_params.py
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
## Parameter Layering
|
|
64
|
+
Parameters are merged from multiple sources in the following order:
|
|
65
|
+
1. Configuration files (`.toml`, `.yaml`, `.ini`, `.json`)
|
|
66
|
+
2. `.env` file (if enabled)
|
|
67
|
+
3. Environment variables (default prefix: `P_`)
|
|
68
|
+
4. Command-line arguments (`argparse`)
|
|
69
|
+
|
|
70
|
+
### Customizing Layering Order
|
|
71
|
+
You can specify the order explicitly:
|
|
72
|
+
```python
|
|
73
|
+
params = pf.load('params.toml', 'env', '.env', 'args')
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
### Overriding Parameters
|
|
77
|
+
Command-line arguments override other sources:
|
|
78
|
+
```sh
|
|
79
|
+
python print_params.py --profile dqn-adam --learning_rate 0.0002
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
## Managing ML Hyperparameter Profiles
|
|
83
|
+
### Example Configuration (`params.toml`)
|
|
84
|
+
```toml
|
|
85
|
+
[default]
|
|
86
|
+
learning_rate = 0.00025
|
|
87
|
+
batch_size = 32
|
|
88
|
+
optimizer_class = 'torch.optim.RMSprop'
|
|
89
|
+
optimizer_kwargs = { momentum = 0.95 }
|
|
90
|
+
random_seed = 13
|
|
91
|
+
|
|
92
|
+
[adam]
|
|
93
|
+
learning_rate = 1e-4
|
|
94
|
+
optimizer_class = 'torch.optim.Adam'
|
|
95
|
+
optimizer_kwargs = {}
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
### Activating a Profile
|
|
99
|
+
```sh
|
|
100
|
+
python app.py --profile adam
|
|
101
|
+
```
|
|
102
|
+
This overrides:
|
|
103
|
+
- `learning_rate` → `1e-4`
|
|
104
|
+
- `optimizer_class` → `torch.optim.Adam`
|
|
105
|
+
- `optimizer_kwargs` → `{}`
|
|
106
|
+
|
|
107
|
+
## Managing Development Stages
|
|
108
|
+
Profiles can be used to manage configurations for different environments.
|
|
109
|
+
|
|
110
|
+
### Example Configuration (`params.toml`)
|
|
111
|
+
```toml
|
|
112
|
+
[default]
|
|
113
|
+
debug = true
|
|
114
|
+
database_url = "mysql://localhost:3306/myapp"
|
|
115
|
+
|
|
116
|
+
[dev]
|
|
117
|
+
database_url = "mysql://dev:3306/myapp"
|
|
118
|
+
|
|
119
|
+
[prod]
|
|
120
|
+
debug = false
|
|
121
|
+
database_url = "mysql://prod:3306/myapp"
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
### Activating a Profile
|
|
125
|
+
```sh
|
|
126
|
+
export P_PROFILE=dev
|
|
127
|
+
python app.py
|
|
128
|
+
```
|
|
129
|
+
|
|
Binary file
|
|
@@ -42,6 +42,7 @@ def load(*sources: Tuple[str, ...],
|
|
|
42
42
|
'profile_key': profile_key,
|
|
43
43
|
'default_profile': default_profile,
|
|
44
44
|
profile_key: profile,
|
|
45
|
+
'__source__': ['pf.load'],
|
|
45
46
|
}
|
|
46
47
|
meta_env_parser = EnvParser(ENV_PREFIX, DEFAULT_PROFILE)
|
|
47
48
|
meta_args_parser = ArgsParser(ARGS_PREFIX, DEFAULT_PROFILE, no_exit=True, descr='Meta-parameters')
|
|
@@ -50,7 +51,7 @@ def load(*sources: Tuple[str, ...],
|
|
|
50
51
|
meta = freeze(meta)
|
|
51
52
|
|
|
52
53
|
if meta.sources is None or len(meta.sources) == 0:
|
|
53
|
-
sys.exit('
|
|
54
|
+
sys.exit('sources meta param is missing')
|
|
54
55
|
|
|
55
56
|
sources = list(meta.sources)
|
|
56
57
|
|
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
Metadata-Version: 2.2
|
|
2
|
+
Name: paramflow
|
|
3
|
+
Version: 0.2.3
|
|
4
|
+
Home-page: https://github.com/mduszyk/paramflow
|
|
5
|
+
Classifier: Programming Language :: Python :: 3
|
|
6
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
7
|
+
Description-Content-Type: text/markdown
|
|
8
|
+
License-File: LICENSE
|
|
9
|
+
Requires-Dist: pyyaml
|
|
10
|
+
Provides-Extra: dotenv
|
|
11
|
+
Requires-Dist: python-dotenv; extra == "dotenv"
|
|
12
|
+
Dynamic: classifier
|
|
13
|
+
Dynamic: description
|
|
14
|
+
Dynamic: description-content-type
|
|
15
|
+
Dynamic: home-page
|
|
16
|
+
Dynamic: provides-extra
|
|
17
|
+
Dynamic: requires-dist
|
|
18
|
+
|
|
19
|
+
# paramflow
|
|
20
|
+
`paramflow` is a flexible and user-friendly parameter and configuration management library designed for
|
|
21
|
+
machine learning workflows and applications requiring profiles and layered parameters. It enables seamless
|
|
22
|
+
merging of parameters from multiple sources, auto-generates a command-line argument parser,
|
|
23
|
+
and allows for easy parameter overrides.
|
|
24
|
+
|
|
25
|
+
## Features
|
|
26
|
+
- **Layered configuration**: Merge parameters from files, environment variables, and command-line arguments.
|
|
27
|
+
- **Immutable dictionary**: Provides a read-only dictionary with attribute-style access.
|
|
28
|
+
- **Profile support**: Manage multiple sets of parameters with profile-based layering.
|
|
29
|
+
- **Layered meta-parameters**: `paramflow` configures itself using a layered approach.
|
|
30
|
+
- **Automatic type conversion**: Converts types during merging based on target parameter types.
|
|
31
|
+
- **Command-line argument parsing**: Automatically generates an `argparse` parser from parameter definitions.
|
|
32
|
+
- **Nested Configuration**: Allows for nested configuration and merging.
|
|
33
|
+
|
|
34
|
+
## Installation
|
|
35
|
+
```sh
|
|
36
|
+
pip install paramflow
|
|
37
|
+
```
|
|
38
|
+
Install with `.env` support:
|
|
39
|
+
```sh
|
|
40
|
+
pip install "paramflow[dotenv]"
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## Basic Usage
|
|
44
|
+
### Example Configuration File (`params.toml`)
|
|
45
|
+
```toml
|
|
46
|
+
[default]
|
|
47
|
+
learning_rate = 0.001
|
|
48
|
+
batch_size = 64
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
### Loading Parameters in Python (`app.py`)
|
|
52
|
+
```python
|
|
53
|
+
import paramflow as pf
|
|
54
|
+
|
|
55
|
+
params = pf.load('params.toml')
|
|
56
|
+
print(params.learning_rate) # 0.001
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
### Generating Command-line Help
|
|
60
|
+
Running the script with `--help` displays both meta-parameters and parameters:
|
|
61
|
+
```sh
|
|
62
|
+
python app.py --help
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
## Meta-Parameter Layering
|
|
66
|
+
Meta-parameters control how `paramflow.load` reads its own configuration. Layering order:
|
|
67
|
+
1. `paramflow.load` arguments
|
|
68
|
+
2. Environment variables (default prefix: `P_`)
|
|
69
|
+
3. Command-line arguments (`argparse`)
|
|
70
|
+
|
|
71
|
+
### Activating Profiles
|
|
72
|
+
Via command-line:
|
|
73
|
+
```sh
|
|
74
|
+
python print_params.py --profile dqn-adam
|
|
75
|
+
```
|
|
76
|
+
Via environment variable:
|
|
77
|
+
```sh
|
|
78
|
+
P_PROFILE=dqn-adam python print_params.py
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
## Parameter Layering
|
|
82
|
+
Parameters are merged from multiple sources in the following order:
|
|
83
|
+
1. Configuration files (`.toml`, `.yaml`, `.ini`, `.json`)
|
|
84
|
+
2. `.env` file (if enabled)
|
|
85
|
+
3. Environment variables (default prefix: `P_`)
|
|
86
|
+
4. Command-line arguments (`argparse`)
|
|
87
|
+
|
|
88
|
+
### Customizing Layering Order
|
|
89
|
+
You can specify the order explicitly:
|
|
90
|
+
```python
|
|
91
|
+
params = pf.load('params.toml', 'env', '.env', 'args')
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
### Overriding Parameters
|
|
95
|
+
Command-line arguments override other sources:
|
|
96
|
+
```sh
|
|
97
|
+
python print_params.py --profile dqn-adam --learning_rate 0.0002
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
## Managing ML Hyperparameter Profiles
|
|
101
|
+
### Example Configuration (`params.toml`)
|
|
102
|
+
```toml
|
|
103
|
+
[default]
|
|
104
|
+
learning_rate = 0.00025
|
|
105
|
+
batch_size = 32
|
|
106
|
+
optimizer_class = 'torch.optim.RMSprop'
|
|
107
|
+
optimizer_kwargs = { momentum = 0.95 }
|
|
108
|
+
random_seed = 13
|
|
109
|
+
|
|
110
|
+
[adam]
|
|
111
|
+
learning_rate = 1e-4
|
|
112
|
+
optimizer_class = 'torch.optim.Adam'
|
|
113
|
+
optimizer_kwargs = {}
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
### Activating a Profile
|
|
117
|
+
```sh
|
|
118
|
+
python app.py --profile adam
|
|
119
|
+
```
|
|
120
|
+
This overrides:
|
|
121
|
+
- `learning_rate` → `1e-4`
|
|
122
|
+
- `optimizer_class` → `torch.optim.Adam`
|
|
123
|
+
- `optimizer_kwargs` → `{}`
|
|
124
|
+
|
|
125
|
+
## Managing Development Stages
|
|
126
|
+
Profiles can be used to manage configurations for different environments.
|
|
127
|
+
|
|
128
|
+
### Example Configuration (`params.toml`)
|
|
129
|
+
```toml
|
|
130
|
+
[default]
|
|
131
|
+
debug = true
|
|
132
|
+
database_url = "mysql://localhost:3306/myapp"
|
|
133
|
+
|
|
134
|
+
[dev]
|
|
135
|
+
database_url = "mysql://dev:3306/myapp"
|
|
136
|
+
|
|
137
|
+
[prod]
|
|
138
|
+
debug = false
|
|
139
|
+
database_url = "mysql://prod:3306/myapp"
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
### Activating a Profile
|
|
143
|
+
```sh
|
|
144
|
+
export P_PROFILE=dev
|
|
145
|
+
python app.py
|
|
146
|
+
```
|
|
147
|
+
|
paramflow-0.2.1/PKG-INFO
DELETED
|
@@ -1,147 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.2
|
|
2
|
-
Name: paramflow
|
|
3
|
-
Version: 0.2.1
|
|
4
|
-
Home-page: https://github.com/mduszyk/paramflow
|
|
5
|
-
Classifier: Programming Language :: Python :: 3
|
|
6
|
-
Classifier: License :: OSI Approved :: MIT License
|
|
7
|
-
Description-Content-Type: text/markdown
|
|
8
|
-
License-File: LICENSE
|
|
9
|
-
Requires-Dist: pyyaml
|
|
10
|
-
Provides-Extra: dotenv
|
|
11
|
-
Requires-Dist: python-dotenv; extra == "dotenv"
|
|
12
|
-
Dynamic: classifier
|
|
13
|
-
Dynamic: description
|
|
14
|
-
Dynamic: description-content-type
|
|
15
|
-
Dynamic: home-page
|
|
16
|
-
Dynamic: provides-extra
|
|
17
|
-
Dynamic: requires-dist
|
|
18
|
-
|
|
19
|
-
# paramflow
|
|
20
|
-
A parameter and configuration management library motivated by training machine learning models
|
|
21
|
-
and managing configuration for applications that require profiles and layered parameters.
|
|
22
|
-
```paramflow``` is designed for flexibility and ease of use, enabling seamless parameter merging
|
|
23
|
-
from multiple sources. It also auto-generates a command-line argument parser and allows for
|
|
24
|
-
easy parameter overrides.
|
|
25
|
-
|
|
26
|
-
## Features
|
|
27
|
-
- **Layered configuration**: Merge parameters from files, environment variables, and command-line arguments.
|
|
28
|
-
- **Immutable dictionary**: Provides a read-only dictionary with attribute-style access.
|
|
29
|
-
- **Profile support**: Manage multiple sets of parameters. Layer the chosen profile on top of the default profile.
|
|
30
|
-
- **Layered meta-parameters**: ```paramflow``` loads its own configuration using layered approach.
|
|
31
|
-
- **Convert types**: Convert types during merging using target parameters as a reference for type conversions.
|
|
32
|
-
- **Generate argument parser**: Use parameters defined in files as a reference for generating ```argparse``` parser.
|
|
33
|
-
|
|
34
|
-
## Usage
|
|
35
|
-
Install:
|
|
36
|
-
```shell
|
|
37
|
-
pip install paramflow
|
|
38
|
-
```
|
|
39
|
-
Install with ```.env``` support:
|
|
40
|
-
```shell
|
|
41
|
-
pip install "paramflow[dotenv]"
|
|
42
|
-
```
|
|
43
|
-
|
|
44
|
-
```params.toml```
|
|
45
|
-
```toml
|
|
46
|
-
[default]
|
|
47
|
-
learning_rate = 0.001
|
|
48
|
-
batch_size = 64
|
|
49
|
-
```
|
|
50
|
-
|
|
51
|
-
```app.py```
|
|
52
|
-
```python
|
|
53
|
-
import paramflow as pf
|
|
54
|
-
params = pf.load('params.toml')
|
|
55
|
-
print(params.learning_rate)
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
```shell
|
|
59
|
-
python app.py --help
|
|
60
|
-
usage: app.py [-h] [--learning_rate LEARNING_RATE] [--batch_size BATCH_SIZE]
|
|
61
|
-
|
|
62
|
-
options:
|
|
63
|
-
-h, --help show this help message and exit
|
|
64
|
-
--learning_rate LEARNING_RATE
|
|
65
|
-
learning_rate = 0.001
|
|
66
|
-
--batch_size BATCH_SIZE
|
|
67
|
-
batch_size = 64
|
|
68
|
-
```
|
|
69
|
-
|
|
70
|
-
## Meta-parameter Layering
|
|
71
|
-
Meta-parameter layering controls how ```paramflow.load``` reads its own configuration.
|
|
72
|
-
|
|
73
|
-
Layering order:
|
|
74
|
-
1. ```paramflow.load``` arguments.
|
|
75
|
-
2. Environment variables (default prefix ```P_```).
|
|
76
|
-
3. Command-line arguments (via ```argparse```).
|
|
77
|
-
|
|
78
|
-
Activate profile using command-line arguments:
|
|
79
|
-
```shell
|
|
80
|
-
python print_params.py --profile dqn-adam
|
|
81
|
-
```
|
|
82
|
-
Activate profile using environment variable:
|
|
83
|
-
```shell
|
|
84
|
-
P_PROFILE=dqn-adam python print_params.py
|
|
85
|
-
```
|
|
86
|
-
|
|
87
|
-
## Parameter Layering
|
|
88
|
-
Parameter layering merges parameters from multiple sources.
|
|
89
|
-
|
|
90
|
-
Layering order:
|
|
91
|
-
1. Configuration files (```.toml```, ```.yaml```, ```.ini```, ```.json```).
|
|
92
|
-
2. ```.env``` file.
|
|
93
|
-
3. Environment variables (default prefix ```P_```).
|
|
94
|
-
4. Command-line arguments (via ```argparse```).
|
|
95
|
-
|
|
96
|
-
Layering order can be customized via ```source``` argument to ```param.flow```.
|
|
97
|
-
```python
|
|
98
|
-
params = pf.load(['params.toml', 'env', '.env', 'args'])
|
|
99
|
-
```
|
|
100
|
-
|
|
101
|
-
Overwrite parameter value:
|
|
102
|
-
```shell
|
|
103
|
-
python print_params.py --profile dqn-adam --learning_rate 0.0002
|
|
104
|
-
```
|
|
105
|
-
|
|
106
|
-
## ML hyper-parameters profiles
|
|
107
|
-
```params.toml```
|
|
108
|
-
```toml
|
|
109
|
-
[default]
|
|
110
|
-
learning_rate = 0.00025
|
|
111
|
-
batch_size = 32
|
|
112
|
-
optimizer_class = 'torch.optim.RMSprop'
|
|
113
|
-
optimizer_kwargs = { momentum = 0.95 }
|
|
114
|
-
random_seed = 13
|
|
115
|
-
|
|
116
|
-
[adam]
|
|
117
|
-
learning_rate = 1e-4
|
|
118
|
-
optimizer_class = 'torch.optim.Adam'
|
|
119
|
-
optimizer_kwargs = {}
|
|
120
|
-
```
|
|
121
|
-
Activating adam profile
|
|
122
|
-
```shell
|
|
123
|
-
python app.py --profile adam
|
|
124
|
-
```
|
|
125
|
-
will result in overwriting default learning rate with ```1e-4```, default optimizer class with ```torch.optim.Adam```
|
|
126
|
-
and default optimizer arguments with and empty dict.
|
|
127
|
-
|
|
128
|
-
## Development stages profiles
|
|
129
|
-
Profiles can be used to manage software development stages.
|
|
130
|
-
```params.toml```:
|
|
131
|
-
```toml
|
|
132
|
-
[default]
|
|
133
|
-
debug = true
|
|
134
|
-
database_url = "mysql://localhost:3306/myapp"
|
|
135
|
-
|
|
136
|
-
[dev]
|
|
137
|
-
database_url = "mysql://dev:3306/myapp"
|
|
138
|
-
|
|
139
|
-
[prod]
|
|
140
|
-
debug = false
|
|
141
|
-
database_url = "mysql://prod:3306/myapp"
|
|
142
|
-
```
|
|
143
|
-
Activate prod profile:
|
|
144
|
-
```shell
|
|
145
|
-
export P_PROFILE=dev
|
|
146
|
-
python app.py
|
|
147
|
-
```
|
paramflow-0.2.1/README.md
DELETED
|
@@ -1,129 +0,0 @@
|
|
|
1
|
-
# paramflow
|
|
2
|
-
A parameter and configuration management library motivated by training machine learning models
|
|
3
|
-
and managing configuration for applications that require profiles and layered parameters.
|
|
4
|
-
```paramflow``` is designed for flexibility and ease of use, enabling seamless parameter merging
|
|
5
|
-
from multiple sources. It also auto-generates a command-line argument parser and allows for
|
|
6
|
-
easy parameter overrides.
|
|
7
|
-
|
|
8
|
-
## Features
|
|
9
|
-
- **Layered configuration**: Merge parameters from files, environment variables, and command-line arguments.
|
|
10
|
-
- **Immutable dictionary**: Provides a read-only dictionary with attribute-style access.
|
|
11
|
-
- **Profile support**: Manage multiple sets of parameters. Layer the chosen profile on top of the default profile.
|
|
12
|
-
- **Layered meta-parameters**: ```paramflow``` loads its own configuration using layered approach.
|
|
13
|
-
- **Convert types**: Convert types during merging using target parameters as a reference for type conversions.
|
|
14
|
-
- **Generate argument parser**: Use parameters defined in files as a reference for generating ```argparse``` parser.
|
|
15
|
-
|
|
16
|
-
## Usage
|
|
17
|
-
Install:
|
|
18
|
-
```shell
|
|
19
|
-
pip install paramflow
|
|
20
|
-
```
|
|
21
|
-
Install with ```.env``` support:
|
|
22
|
-
```shell
|
|
23
|
-
pip install "paramflow[dotenv]"
|
|
24
|
-
```
|
|
25
|
-
|
|
26
|
-
```params.toml```
|
|
27
|
-
```toml
|
|
28
|
-
[default]
|
|
29
|
-
learning_rate = 0.001
|
|
30
|
-
batch_size = 64
|
|
31
|
-
```
|
|
32
|
-
|
|
33
|
-
```app.py```
|
|
34
|
-
```python
|
|
35
|
-
import paramflow as pf
|
|
36
|
-
params = pf.load('params.toml')
|
|
37
|
-
print(params.learning_rate)
|
|
38
|
-
```
|
|
39
|
-
|
|
40
|
-
```shell
|
|
41
|
-
python app.py --help
|
|
42
|
-
usage: app.py [-h] [--learning_rate LEARNING_RATE] [--batch_size BATCH_SIZE]
|
|
43
|
-
|
|
44
|
-
options:
|
|
45
|
-
-h, --help show this help message and exit
|
|
46
|
-
--learning_rate LEARNING_RATE
|
|
47
|
-
learning_rate = 0.001
|
|
48
|
-
--batch_size BATCH_SIZE
|
|
49
|
-
batch_size = 64
|
|
50
|
-
```
|
|
51
|
-
|
|
52
|
-
## Meta-parameter Layering
|
|
53
|
-
Meta-parameter layering controls how ```paramflow.load``` reads its own configuration.
|
|
54
|
-
|
|
55
|
-
Layering order:
|
|
56
|
-
1. ```paramflow.load``` arguments.
|
|
57
|
-
2. Environment variables (default prefix ```P_```).
|
|
58
|
-
3. Command-line arguments (via ```argparse```).
|
|
59
|
-
|
|
60
|
-
Activate profile using command-line arguments:
|
|
61
|
-
```shell
|
|
62
|
-
python print_params.py --profile dqn-adam
|
|
63
|
-
```
|
|
64
|
-
Activate profile using environment variable:
|
|
65
|
-
```shell
|
|
66
|
-
P_PROFILE=dqn-adam python print_params.py
|
|
67
|
-
```
|
|
68
|
-
|
|
69
|
-
## Parameter Layering
|
|
70
|
-
Parameter layering merges parameters from multiple sources.
|
|
71
|
-
|
|
72
|
-
Layering order:
|
|
73
|
-
1. Configuration files (```.toml```, ```.yaml```, ```.ini```, ```.json```).
|
|
74
|
-
2. ```.env``` file.
|
|
75
|
-
3. Environment variables (default prefix ```P_```).
|
|
76
|
-
4. Command-line arguments (via ```argparse```).
|
|
77
|
-
|
|
78
|
-
Layering order can be customized via ```source``` argument to ```param.flow```.
|
|
79
|
-
```python
|
|
80
|
-
params = pf.load(['params.toml', 'env', '.env', 'args'])
|
|
81
|
-
```
|
|
82
|
-
|
|
83
|
-
Overwrite parameter value:
|
|
84
|
-
```shell
|
|
85
|
-
python print_params.py --profile dqn-adam --learning_rate 0.0002
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
## ML hyper-parameters profiles
|
|
89
|
-
```params.toml```
|
|
90
|
-
```toml
|
|
91
|
-
[default]
|
|
92
|
-
learning_rate = 0.00025
|
|
93
|
-
batch_size = 32
|
|
94
|
-
optimizer_class = 'torch.optim.RMSprop'
|
|
95
|
-
optimizer_kwargs = { momentum = 0.95 }
|
|
96
|
-
random_seed = 13
|
|
97
|
-
|
|
98
|
-
[adam]
|
|
99
|
-
learning_rate = 1e-4
|
|
100
|
-
optimizer_class = 'torch.optim.Adam'
|
|
101
|
-
optimizer_kwargs = {}
|
|
102
|
-
```
|
|
103
|
-
Activating adam profile
|
|
104
|
-
```shell
|
|
105
|
-
python app.py --profile adam
|
|
106
|
-
```
|
|
107
|
-
will result in overwriting default learning rate with ```1e-4```, default optimizer class with ```torch.optim.Adam```
|
|
108
|
-
and default optimizer arguments with and empty dict.
|
|
109
|
-
|
|
110
|
-
## Development stages profiles
|
|
111
|
-
Profiles can be used to manage software development stages.
|
|
112
|
-
```params.toml```:
|
|
113
|
-
```toml
|
|
114
|
-
[default]
|
|
115
|
-
debug = true
|
|
116
|
-
database_url = "mysql://localhost:3306/myapp"
|
|
117
|
-
|
|
118
|
-
[dev]
|
|
119
|
-
database_url = "mysql://dev:3306/myapp"
|
|
120
|
-
|
|
121
|
-
[prod]
|
|
122
|
-
debug = false
|
|
123
|
-
database_url = "mysql://prod:3306/myapp"
|
|
124
|
-
```
|
|
125
|
-
Activate prod profile:
|
|
126
|
-
```shell
|
|
127
|
-
export P_PROFILE=dev
|
|
128
|
-
python app.py
|
|
129
|
-
```
|
|
Binary file
|
|
@@ -1,147 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.2
|
|
2
|
-
Name: paramflow
|
|
3
|
-
Version: 0.2.1
|
|
4
|
-
Home-page: https://github.com/mduszyk/paramflow
|
|
5
|
-
Classifier: Programming Language :: Python :: 3
|
|
6
|
-
Classifier: License :: OSI Approved :: MIT License
|
|
7
|
-
Description-Content-Type: text/markdown
|
|
8
|
-
License-File: LICENSE
|
|
9
|
-
Requires-Dist: pyyaml
|
|
10
|
-
Provides-Extra: dotenv
|
|
11
|
-
Requires-Dist: python-dotenv; extra == "dotenv"
|
|
12
|
-
Dynamic: classifier
|
|
13
|
-
Dynamic: description
|
|
14
|
-
Dynamic: description-content-type
|
|
15
|
-
Dynamic: home-page
|
|
16
|
-
Dynamic: provides-extra
|
|
17
|
-
Dynamic: requires-dist
|
|
18
|
-
|
|
19
|
-
# paramflow
|
|
20
|
-
A parameter and configuration management library motivated by training machine learning models
|
|
21
|
-
and managing configuration for applications that require profiles and layered parameters.
|
|
22
|
-
```paramflow``` is designed for flexibility and ease of use, enabling seamless parameter merging
|
|
23
|
-
from multiple sources. It also auto-generates a command-line argument parser and allows for
|
|
24
|
-
easy parameter overrides.
|
|
25
|
-
|
|
26
|
-
## Features
|
|
27
|
-
- **Layered configuration**: Merge parameters from files, environment variables, and command-line arguments.
|
|
28
|
-
- **Immutable dictionary**: Provides a read-only dictionary with attribute-style access.
|
|
29
|
-
- **Profile support**: Manage multiple sets of parameters. Layer the chosen profile on top of the default profile.
|
|
30
|
-
- **Layered meta-parameters**: ```paramflow``` loads its own configuration using layered approach.
|
|
31
|
-
- **Convert types**: Convert types during merging using target parameters as a reference for type conversions.
|
|
32
|
-
- **Generate argument parser**: Use parameters defined in files as a reference for generating ```argparse``` parser.
|
|
33
|
-
|
|
34
|
-
## Usage
|
|
35
|
-
Install:
|
|
36
|
-
```shell
|
|
37
|
-
pip install paramflow
|
|
38
|
-
```
|
|
39
|
-
Install with ```.env``` support:
|
|
40
|
-
```shell
|
|
41
|
-
pip install "paramflow[dotenv]"
|
|
42
|
-
```
|
|
43
|
-
|
|
44
|
-
```params.toml```
|
|
45
|
-
```toml
|
|
46
|
-
[default]
|
|
47
|
-
learning_rate = 0.001
|
|
48
|
-
batch_size = 64
|
|
49
|
-
```
|
|
50
|
-
|
|
51
|
-
```app.py```
|
|
52
|
-
```python
|
|
53
|
-
import paramflow as pf
|
|
54
|
-
params = pf.load('params.toml')
|
|
55
|
-
print(params.learning_rate)
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
```shell
|
|
59
|
-
python app.py --help
|
|
60
|
-
usage: app.py [-h] [--learning_rate LEARNING_RATE] [--batch_size BATCH_SIZE]
|
|
61
|
-
|
|
62
|
-
options:
|
|
63
|
-
-h, --help show this help message and exit
|
|
64
|
-
--learning_rate LEARNING_RATE
|
|
65
|
-
learning_rate = 0.001
|
|
66
|
-
--batch_size BATCH_SIZE
|
|
67
|
-
batch_size = 64
|
|
68
|
-
```
|
|
69
|
-
|
|
70
|
-
## Meta-parameter Layering
|
|
71
|
-
Meta-parameter layering controls how ```paramflow.load``` reads its own configuration.
|
|
72
|
-
|
|
73
|
-
Layering order:
|
|
74
|
-
1. ```paramflow.load``` arguments.
|
|
75
|
-
2. Environment variables (default prefix ```P_```).
|
|
76
|
-
3. Command-line arguments (via ```argparse```).
|
|
77
|
-
|
|
78
|
-
Activate profile using command-line arguments:
|
|
79
|
-
```shell
|
|
80
|
-
python print_params.py --profile dqn-adam
|
|
81
|
-
```
|
|
82
|
-
Activate profile using environment variable:
|
|
83
|
-
```shell
|
|
84
|
-
P_PROFILE=dqn-adam python print_params.py
|
|
85
|
-
```
|
|
86
|
-
|
|
87
|
-
## Parameter Layering
|
|
88
|
-
Parameter layering merges parameters from multiple sources.
|
|
89
|
-
|
|
90
|
-
Layering order:
|
|
91
|
-
1. Configuration files (```.toml```, ```.yaml```, ```.ini```, ```.json```).
|
|
92
|
-
2. ```.env``` file.
|
|
93
|
-
3. Environment variables (default prefix ```P_```).
|
|
94
|
-
4. Command-line arguments (via ```argparse```).
|
|
95
|
-
|
|
96
|
-
Layering order can be customized via ```source``` argument to ```param.flow```.
|
|
97
|
-
```python
|
|
98
|
-
params = pf.load(['params.toml', 'env', '.env', 'args'])
|
|
99
|
-
```
|
|
100
|
-
|
|
101
|
-
Overwrite parameter value:
|
|
102
|
-
```shell
|
|
103
|
-
python print_params.py --profile dqn-adam --learning_rate 0.0002
|
|
104
|
-
```
|
|
105
|
-
|
|
106
|
-
## ML hyper-parameters profiles
|
|
107
|
-
```params.toml```
|
|
108
|
-
```toml
|
|
109
|
-
[default]
|
|
110
|
-
learning_rate = 0.00025
|
|
111
|
-
batch_size = 32
|
|
112
|
-
optimizer_class = 'torch.optim.RMSprop'
|
|
113
|
-
optimizer_kwargs = { momentum = 0.95 }
|
|
114
|
-
random_seed = 13
|
|
115
|
-
|
|
116
|
-
[adam]
|
|
117
|
-
learning_rate = 1e-4
|
|
118
|
-
optimizer_class = 'torch.optim.Adam'
|
|
119
|
-
optimizer_kwargs = {}
|
|
120
|
-
```
|
|
121
|
-
Activating adam profile
|
|
122
|
-
```shell
|
|
123
|
-
python app.py --profile adam
|
|
124
|
-
```
|
|
125
|
-
will result in overwriting default learning rate with ```1e-4```, default optimizer class with ```torch.optim.Adam```
|
|
126
|
-
and default optimizer arguments with and empty dict.
|
|
127
|
-
|
|
128
|
-
## Development stages profiles
|
|
129
|
-
Profiles can be used to manage software development stages.
|
|
130
|
-
```params.toml```:
|
|
131
|
-
```toml
|
|
132
|
-
[default]
|
|
133
|
-
debug = true
|
|
134
|
-
database_url = "mysql://localhost:3306/myapp"
|
|
135
|
-
|
|
136
|
-
[dev]
|
|
137
|
-
database_url = "mysql://dev:3306/myapp"
|
|
138
|
-
|
|
139
|
-
[prod]
|
|
140
|
-
debug = false
|
|
141
|
-
database_url = "mysql://prod:3306/myapp"
|
|
142
|
-
```
|
|
143
|
-
Activate prod profile:
|
|
144
|
-
```shell
|
|
145
|
-
export P_PROFILE=dev
|
|
146
|
-
python app.py
|
|
147
|
-
```
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{paramflow-0.2.1 → paramflow-0.2.3}/paramflow/__pycache__/frozen_test.cpython-312-pytest-8.3.4.pyc
RENAMED
|
File without changes
|
{paramflow-0.2.1 → paramflow-0.2.3}/paramflow/__pycache__/params_test.cpython-312-pytest-8.3.4.pyc
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|