eb-evaluation 0.2.0__tar.gz → 0.2.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- eb_evaluation-0.2.2/PKG-INFO +119 -0
- eb_evaluation-0.2.2/README.md +86 -0
- eb_evaluation-0.2.2/pyproject.toml +92 -0
- eb_evaluation-0.2.2/src/eb_evaluation.egg-info/PKG-INFO +119 -0
- eb_evaluation-0.2.2/src/eb_evaluation.egg-info/requires.txt +12 -0
- eb_evaluation-0.2.0/PKG-INFO +0 -169
- eb_evaluation-0.2.0/README.md +0 -148
- eb_evaluation-0.2.0/pyproject.toml +0 -41
- eb_evaluation-0.2.0/src/eb_evaluation.egg-info/PKG-INFO +0 -169
- eb_evaluation-0.2.0/src/eb_evaluation.egg-info/requires.txt +0 -7
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/LICENSE +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/setup.cfg +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/__init__.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/adjustment/__init__.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/adjustment/_utils.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/adjustment/readiness_adjustment.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/dataframe/__init__.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/dataframe/cost_ratio.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/dataframe/entity.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/dataframe/group.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/dataframe/hierarchy.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/dataframe/panel.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/dataframe/sensitivity.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/dataframe/single.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/dataframe/tolerance.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/model_selection/__init__.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/model_selection/auto_engine.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/model_selection/compare.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/model_selection/cwsl_regressor.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/model_selection/electric_barometer.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/utils/__init__.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/utils/validation.py +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation.egg-info/SOURCES.txt +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation.egg-info/dependency_links.txt +0 -0
- {eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation.egg-info/top_level.txt +0 -0
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: eb-evaluation
|
|
3
|
+
Version: 0.2.2
|
|
4
|
+
Summary: Electric Barometer: DataFrame-based evaluation utilities for CWSL and related metrics.
|
|
5
|
+
Author-email: "Kyle Corrie (Economistician)" <kcorrie@economistician.com>
|
|
6
|
+
License-Expression: BSD-3-Clause
|
|
7
|
+
Project-URL: Homepage, https://github.com/Economistician/eb-evaluation
|
|
8
|
+
Project-URL: Repository, https://github.com/Economistician/eb-evaluation
|
|
9
|
+
Project-URL: Issues, https://github.com/Economistician/eb-evaluation/issues
|
|
10
|
+
Project-URL: Documentation, https://github.com/Economistician/eb-docs
|
|
11
|
+
Keywords: electric-barometer,forecast-evaluation,asymmetric-loss,readiness,forecasting,pandas
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
18
|
+
Classifier: Operating System :: OS Independent
|
|
19
|
+
Requires-Python: >=3.10
|
|
20
|
+
Description-Content-Type: text/markdown
|
|
21
|
+
License-File: LICENSE
|
|
22
|
+
Requires-Dist: numpy>=1.24
|
|
23
|
+
Requires-Dist: pandas>=2.0
|
|
24
|
+
Requires-Dist: eb-metrics<0.3,>=0.2
|
|
25
|
+
Requires-Dist: eb-adapters<0.3,>=0.2
|
|
26
|
+
Provides-Extra: test
|
|
27
|
+
Requires-Dist: pytest>=8.0; extra == "test"
|
|
28
|
+
Requires-Dist: scikit-learn>=1.3; extra == "test"
|
|
29
|
+
Provides-Extra: dev
|
|
30
|
+
Requires-Dist: pytest>=8.0; extra == "dev"
|
|
31
|
+
Requires-Dist: pytest-cov>=5.0; extra == "dev"
|
|
32
|
+
Dynamic: license-file
|
|
33
|
+
|
|
34
|
+
# Electric Barometer · Evaluation (`eb-evaluation`)
|
|
35
|
+
|
|
36
|
+
[](https://github.com/Economistician/eb-evaluation/actions/workflows/ci.yml)
|
|
37
|
+

|
|
38
|
+

|
|
39
|
+

|
|
40
|
+
|
|
41
|
+
Evaluation and model selection utilities for applying Electric Barometer metrics across entities, groups, and operational contexts.
|
|
42
|
+
|
|
43
|
+
---
|
|
44
|
+
|
|
45
|
+
## Overview
|
|
46
|
+
|
|
47
|
+
`eb-evaluation` provides the evaluation and model selection layer of the Electric Barometer ecosystem. It applies metric primitives to forecasts and observations across entities, groups, and hierarchical structures, enabling consistent assessment of forecasting performance in operational settings.
|
|
48
|
+
|
|
49
|
+
The package focuses on DataFrame-first evaluation workflows, including tolerance-based scoring, cost-sensitive comparison, and readiness-oriented adjustment logic. It does not define feature construction or model interfaces; instead, it consumes standardized inputs from upstream layers and produces evaluation outputs that can be used for model selection, reporting, and decision support.
|
|
50
|
+
|
|
51
|
+
---
|
|
52
|
+
|
|
53
|
+
## Role in the Electric Barometer Ecosystem
|
|
54
|
+
|
|
55
|
+
`eb-evaluation` defines the evaluation and model selection layer used throughout the Electric Barometer ecosystem. It is responsible for applying metric primitives to forecasts and observations across entities, groups, and hierarchies, enabling consistent comparison of forecasting performance in operational contexts.
|
|
56
|
+
|
|
57
|
+
This package focuses exclusively on evaluation logic, aggregation semantics, and selection workflows. It does not perform feature construction, model training, or metric definition. Those responsibilities are handled by adjacent layers that generate inputs, adapt model interfaces, or define metric behavior.
|
|
58
|
+
|
|
59
|
+
By separating evaluation orchestration from metric semantics and model implementation details, `eb-evaluation` provides a stable, DataFrame-first foundation for decision-aligned model comparison and readiness assessment across heterogeneous forecasting pipelines.
|
|
60
|
+
|
|
61
|
+
---
|
|
62
|
+
|
|
63
|
+
## Installation
|
|
64
|
+
|
|
65
|
+
`eb-evaluation` is distributed as a standard Python package.
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
pip install eb-evaluation
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
The package supports Python 3.10 and later.
|
|
72
|
+
|
|
73
|
+
---
|
|
74
|
+
|
|
75
|
+
## Core Concepts
|
|
76
|
+
|
|
77
|
+
- **DataFrame-first evaluation** — Evaluation logic operates directly on tabular forecast and observation data, enabling transparent aggregation, grouping, and comparison across entities and hierarchies.
|
|
78
|
+
- **Cost- and tolerance-aware scoring** — Forecast performance is assessed using metrics that reflect asymmetric cost, acceptable deviation thresholds, and operational risk rather than purely symmetric statistical error.
|
|
79
|
+
- **Hierarchical and panel semantics** — Evaluation respects entity boundaries, grouping structure, and temporal alignment, ensuring correctness in multi-level forecasting environments.
|
|
80
|
+
- **Model comparability** — Forecasts produced by heterogeneous models can be evaluated and compared using a consistent set of metrics and aggregation rules.
|
|
81
|
+
- **Readiness-oriented selection** — Model selection emphasizes execution feasibility and operational adequacy, not just aggregate accuracy, supporting decision-aligned forecasting workflows.
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## Minimal Example
|
|
86
|
+
|
|
87
|
+
The example below shows how forecasts and observations can be evaluated and compared across entities using Electric Barometer metrics in a DataFrame-first workflow.
|
|
88
|
+
|
|
89
|
+
```python
|
|
90
|
+
import pandas as pd
|
|
91
|
+
from eb_evaluation.dataframe.compare import compare_models
|
|
92
|
+
|
|
93
|
+
# Example evaluation data
|
|
94
|
+
df = pd.DataFrame({
|
|
95
|
+
"entity_id": ["A", "A", "B", "B"],
|
|
96
|
+
"date": pd.to_datetime(["2024-01-01", "2024-01-02"] * 2),
|
|
97
|
+
"actual": [10, 12, 7, 9],
|
|
98
|
+
"model_a": [9, 11, 8, 10],
|
|
99
|
+
"model_b": [11, 13, 6, 8],
|
|
100
|
+
})
|
|
101
|
+
|
|
102
|
+
# Compare models using a common evaluation contract
|
|
103
|
+
results = compare_models(
|
|
104
|
+
df,
|
|
105
|
+
actual_col="actual",
|
|
106
|
+
prediction_cols=["model_a", "model_b"],
|
|
107
|
+
entity_col="entity_id",
|
|
108
|
+
time_col="date",
|
|
109
|
+
)
|
|
110
|
+
|
|
111
|
+
print(results)
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
---
|
|
115
|
+
|
|
116
|
+
## License
|
|
117
|
+
|
|
118
|
+
BSD 3-Clause License.
|
|
119
|
+
© 2025 Kyle Corrie.
|
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
# Electric Barometer · Evaluation (`eb-evaluation`)
|
|
2
|
+
|
|
3
|
+
[](https://github.com/Economistician/eb-evaluation/actions/workflows/ci.yml)
|
|
4
|
+

|
|
5
|
+

|
|
6
|
+

|
|
7
|
+
|
|
8
|
+
Evaluation and model selection utilities for applying Electric Barometer metrics across entities, groups, and operational contexts.
|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
## Overview
|
|
13
|
+
|
|
14
|
+
`eb-evaluation` provides the evaluation and model selection layer of the Electric Barometer ecosystem. It applies metric primitives to forecasts and observations across entities, groups, and hierarchical structures, enabling consistent assessment of forecasting performance in operational settings.
|
|
15
|
+
|
|
16
|
+
The package focuses on DataFrame-first evaluation workflows, including tolerance-based scoring, cost-sensitive comparison, and readiness-oriented adjustment logic. It does not define feature construction or model interfaces; instead, it consumes standardized inputs from upstream layers and produces evaluation outputs that can be used for model selection, reporting, and decision support.
|
|
17
|
+
|
|
18
|
+
---
|
|
19
|
+
|
|
20
|
+
## Role in the Electric Barometer Ecosystem
|
|
21
|
+
|
|
22
|
+
`eb-evaluation` defines the evaluation and model selection layer used throughout the Electric Barometer ecosystem. It is responsible for applying metric primitives to forecasts and observations across entities, groups, and hierarchies, enabling consistent comparison of forecasting performance in operational contexts.
|
|
23
|
+
|
|
24
|
+
This package focuses exclusively on evaluation logic, aggregation semantics, and selection workflows. It does not perform feature construction, model training, or metric definition. Those responsibilities are handled by adjacent layers that generate inputs, adapt model interfaces, or define metric behavior.
|
|
25
|
+
|
|
26
|
+
By separating evaluation orchestration from metric semantics and model implementation details, `eb-evaluation` provides a stable, DataFrame-first foundation for decision-aligned model comparison and readiness assessment across heterogeneous forecasting pipelines.
|
|
27
|
+
|
|
28
|
+
---
|
|
29
|
+
|
|
30
|
+
## Installation
|
|
31
|
+
|
|
32
|
+
`eb-evaluation` is distributed as a standard Python package.
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
pip install eb-evaluation
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
The package supports Python 3.10 and later.
|
|
39
|
+
|
|
40
|
+
---
|
|
41
|
+
|
|
42
|
+
## Core Concepts
|
|
43
|
+
|
|
44
|
+
- **DataFrame-first evaluation** — Evaluation logic operates directly on tabular forecast and observation data, enabling transparent aggregation, grouping, and comparison across entities and hierarchies.
|
|
45
|
+
- **Cost- and tolerance-aware scoring** — Forecast performance is assessed using metrics that reflect asymmetric cost, acceptable deviation thresholds, and operational risk rather than purely symmetric statistical error.
|
|
46
|
+
- **Hierarchical and panel semantics** — Evaluation respects entity boundaries, grouping structure, and temporal alignment, ensuring correctness in multi-level forecasting environments.
|
|
47
|
+
- **Model comparability** — Forecasts produced by heterogeneous models can be evaluated and compared using a consistent set of metrics and aggregation rules.
|
|
48
|
+
- **Readiness-oriented selection** — Model selection emphasizes execution feasibility and operational adequacy, not just aggregate accuracy, supporting decision-aligned forecasting workflows.
|
|
49
|
+
|
|
50
|
+
---
|
|
51
|
+
|
|
52
|
+
## Minimal Example
|
|
53
|
+
|
|
54
|
+
The example below shows how forecasts and observations can be evaluated and compared across entities using Electric Barometer metrics in a DataFrame-first workflow.
|
|
55
|
+
|
|
56
|
+
```python
|
|
57
|
+
import pandas as pd
|
|
58
|
+
from eb_evaluation.dataframe.compare import compare_models
|
|
59
|
+
|
|
60
|
+
# Example evaluation data
|
|
61
|
+
df = pd.DataFrame({
|
|
62
|
+
"entity_id": ["A", "A", "B", "B"],
|
|
63
|
+
"date": pd.to_datetime(["2024-01-01", "2024-01-02"] * 2),
|
|
64
|
+
"actual": [10, 12, 7, 9],
|
|
65
|
+
"model_a": [9, 11, 8, 10],
|
|
66
|
+
"model_b": [11, 13, 6, 8],
|
|
67
|
+
})
|
|
68
|
+
|
|
69
|
+
# Compare models using a common evaluation contract
|
|
70
|
+
results = compare_models(
|
|
71
|
+
df,
|
|
72
|
+
actual_col="actual",
|
|
73
|
+
prediction_cols=["model_a", "model_b"],
|
|
74
|
+
entity_col="entity_id",
|
|
75
|
+
time_col="date",
|
|
76
|
+
)
|
|
77
|
+
|
|
78
|
+
print(results)
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
---
|
|
82
|
+
|
|
83
|
+
## License
|
|
84
|
+
|
|
85
|
+
BSD 3-Clause License.
|
|
86
|
+
© 2025 Kyle Corrie.
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
######################################
|
|
2
|
+
# Project metadata
|
|
3
|
+
######################################
|
|
4
|
+
[project]
|
|
5
|
+
name = "eb-evaluation"
|
|
6
|
+
version = "0.2.2"
|
|
7
|
+
description = "Electric Barometer: DataFrame-based evaluation utilities for CWSL and related metrics."
|
|
8
|
+
readme = "README.md"
|
|
9
|
+
requires-python = ">=3.10"
|
|
10
|
+
license = "BSD-3-Clause"
|
|
11
|
+
license-files = ["LICENSE*"]
|
|
12
|
+
|
|
13
|
+
authors = [
|
|
14
|
+
{ name = "Kyle Corrie (Economistician)", email = "kcorrie@economistician.com" }
|
|
15
|
+
]
|
|
16
|
+
|
|
17
|
+
keywords = [
|
|
18
|
+
"electric-barometer",
|
|
19
|
+
"forecast-evaluation",
|
|
20
|
+
"asymmetric-loss",
|
|
21
|
+
"readiness",
|
|
22
|
+
"forecasting",
|
|
23
|
+
"pandas",
|
|
24
|
+
]
|
|
25
|
+
|
|
26
|
+
classifiers = [
|
|
27
|
+
"Programming Language :: Python :: 3",
|
|
28
|
+
"Programming Language :: Python :: 3 :: Only",
|
|
29
|
+
"Programming Language :: Python :: 3.10",
|
|
30
|
+
"Programming Language :: Python :: 3.11",
|
|
31
|
+
"Programming Language :: Python :: 3.12",
|
|
32
|
+
"Programming Language :: Python :: 3.13",
|
|
33
|
+
"Operating System :: OS Independent",
|
|
34
|
+
]
|
|
35
|
+
|
|
36
|
+
######################################
|
|
37
|
+
# Core runtime dependencies
|
|
38
|
+
######################################
|
|
39
|
+
dependencies = [
|
|
40
|
+
"numpy>=1.24",
|
|
41
|
+
"pandas>=2.0",
|
|
42
|
+
|
|
43
|
+
# Electric Barometer compatibility band (0.2.x)
|
|
44
|
+
"eb-metrics>=0.2,<0.3",
|
|
45
|
+
"eb-adapters>=0.2,<0.3",
|
|
46
|
+
]
|
|
47
|
+
|
|
48
|
+
######################################
|
|
49
|
+
# Project URLs
|
|
50
|
+
######################################
|
|
51
|
+
[project.urls]
|
|
52
|
+
Homepage = "https://github.com/Economistician/eb-evaluation"
|
|
53
|
+
Repository = "https://github.com/Economistician/eb-evaluation"
|
|
54
|
+
Issues = "https://github.com/Economistician/eb-evaluation/issues"
|
|
55
|
+
Documentation = "https://github.com/Economistician/eb-docs"
|
|
56
|
+
|
|
57
|
+
######################################
|
|
58
|
+
# Optional dependencies (extras)
|
|
59
|
+
######################################
|
|
60
|
+
[project.optional-dependencies]
|
|
61
|
+
|
|
62
|
+
# CI / test-only dependencies
|
|
63
|
+
test = [
|
|
64
|
+
"pytest>=8.0",
|
|
65
|
+
"scikit-learn>=1.3",
|
|
66
|
+
]
|
|
67
|
+
|
|
68
|
+
# Local developer tooling
|
|
69
|
+
dev = [
|
|
70
|
+
"pytest>=8.0",
|
|
71
|
+
"pytest-cov>=5.0",
|
|
72
|
+
]
|
|
73
|
+
|
|
74
|
+
######################################
|
|
75
|
+
# Build system
|
|
76
|
+
######################################
|
|
77
|
+
[build-system]
|
|
78
|
+
requires = ["setuptools>=64", "wheel"]
|
|
79
|
+
build-backend = "setuptools.build_meta"
|
|
80
|
+
|
|
81
|
+
######################################
|
|
82
|
+
# Package discovery
|
|
83
|
+
######################################
|
|
84
|
+
[tool.setuptools.packages.find]
|
|
85
|
+
where = ["src"]
|
|
86
|
+
|
|
87
|
+
######################################
|
|
88
|
+
# Pytest configuration
|
|
89
|
+
######################################
|
|
90
|
+
[tool.pytest.ini_options]
|
|
91
|
+
pythonpath = ["src"]
|
|
92
|
+
addopts = "-ra"
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: eb-evaluation
|
|
3
|
+
Version: 0.2.2
|
|
4
|
+
Summary: Electric Barometer: DataFrame-based evaluation utilities for CWSL and related metrics.
|
|
5
|
+
Author-email: "Kyle Corrie (Economistician)" <kcorrie@economistician.com>
|
|
6
|
+
License-Expression: BSD-3-Clause
|
|
7
|
+
Project-URL: Homepage, https://github.com/Economistician/eb-evaluation
|
|
8
|
+
Project-URL: Repository, https://github.com/Economistician/eb-evaluation
|
|
9
|
+
Project-URL: Issues, https://github.com/Economistician/eb-evaluation/issues
|
|
10
|
+
Project-URL: Documentation, https://github.com/Economistician/eb-docs
|
|
11
|
+
Keywords: electric-barometer,forecast-evaluation,asymmetric-loss,readiness,forecasting,pandas
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
18
|
+
Classifier: Operating System :: OS Independent
|
|
19
|
+
Requires-Python: >=3.10
|
|
20
|
+
Description-Content-Type: text/markdown
|
|
21
|
+
License-File: LICENSE
|
|
22
|
+
Requires-Dist: numpy>=1.24
|
|
23
|
+
Requires-Dist: pandas>=2.0
|
|
24
|
+
Requires-Dist: eb-metrics<0.3,>=0.2
|
|
25
|
+
Requires-Dist: eb-adapters<0.3,>=0.2
|
|
26
|
+
Provides-Extra: test
|
|
27
|
+
Requires-Dist: pytest>=8.0; extra == "test"
|
|
28
|
+
Requires-Dist: scikit-learn>=1.3; extra == "test"
|
|
29
|
+
Provides-Extra: dev
|
|
30
|
+
Requires-Dist: pytest>=8.0; extra == "dev"
|
|
31
|
+
Requires-Dist: pytest-cov>=5.0; extra == "dev"
|
|
32
|
+
Dynamic: license-file
|
|
33
|
+
|
|
34
|
+
# Electric Barometer · Evaluation (`eb-evaluation`)
|
|
35
|
+
|
|
36
|
+
[](https://github.com/Economistician/eb-evaluation/actions/workflows/ci.yml)
|
|
37
|
+

|
|
38
|
+

|
|
39
|
+

|
|
40
|
+
|
|
41
|
+
Evaluation and model selection utilities for applying Electric Barometer metrics across entities, groups, and operational contexts.
|
|
42
|
+
|
|
43
|
+
---
|
|
44
|
+
|
|
45
|
+
## Overview
|
|
46
|
+
|
|
47
|
+
`eb-evaluation` provides the evaluation and model selection layer of the Electric Barometer ecosystem. It applies metric primitives to forecasts and observations across entities, groups, and hierarchical structures, enabling consistent assessment of forecasting performance in operational settings.
|
|
48
|
+
|
|
49
|
+
The package focuses on DataFrame-first evaluation workflows, including tolerance-based scoring, cost-sensitive comparison, and readiness-oriented adjustment logic. It does not define feature construction or model interfaces; instead, it consumes standardized inputs from upstream layers and produces evaluation outputs that can be used for model selection, reporting, and decision support.
|
|
50
|
+
|
|
51
|
+
---
|
|
52
|
+
|
|
53
|
+
## Role in the Electric Barometer Ecosystem
|
|
54
|
+
|
|
55
|
+
`eb-evaluation` defines the evaluation and model selection layer used throughout the Electric Barometer ecosystem. It is responsible for applying metric primitives to forecasts and observations across entities, groups, and hierarchies, enabling consistent comparison of forecasting performance in operational contexts.
|
|
56
|
+
|
|
57
|
+
This package focuses exclusively on evaluation logic, aggregation semantics, and selection workflows. It does not perform feature construction, model training, or metric definition. Those responsibilities are handled by adjacent layers that generate inputs, adapt model interfaces, or define metric behavior.
|
|
58
|
+
|
|
59
|
+
By separating evaluation orchestration from metric semantics and model implementation details, `eb-evaluation` provides a stable, DataFrame-first foundation for decision-aligned model comparison and readiness assessment across heterogeneous forecasting pipelines.
|
|
60
|
+
|
|
61
|
+
---
|
|
62
|
+
|
|
63
|
+
## Installation
|
|
64
|
+
|
|
65
|
+
`eb-evaluation` is distributed as a standard Python package.
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
pip install eb-evaluation
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
The package supports Python 3.10 and later.
|
|
72
|
+
|
|
73
|
+
---
|
|
74
|
+
|
|
75
|
+
## Core Concepts
|
|
76
|
+
|
|
77
|
+
- **DataFrame-first evaluation** — Evaluation logic operates directly on tabular forecast and observation data, enabling transparent aggregation, grouping, and comparison across entities and hierarchies.
|
|
78
|
+
- **Cost- and tolerance-aware scoring** — Forecast performance is assessed using metrics that reflect asymmetric cost, acceptable deviation thresholds, and operational risk rather than purely symmetric statistical error.
|
|
79
|
+
- **Hierarchical and panel semantics** — Evaluation respects entity boundaries, grouping structure, and temporal alignment, ensuring correctness in multi-level forecasting environments.
|
|
80
|
+
- **Model comparability** — Forecasts produced by heterogeneous models can be evaluated and compared using a consistent set of metrics and aggregation rules.
|
|
81
|
+
- **Readiness-oriented selection** — Model selection emphasizes execution feasibility and operational adequacy, not just aggregate accuracy, supporting decision-aligned forecasting workflows.
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## Minimal Example
|
|
86
|
+
|
|
87
|
+
The example below shows how forecasts and observations can be evaluated and compared across entities using Electric Barometer metrics in a DataFrame-first workflow.
|
|
88
|
+
|
|
89
|
+
```python
|
|
90
|
+
import pandas as pd
|
|
91
|
+
from eb_evaluation.dataframe.compare import compare_models
|
|
92
|
+
|
|
93
|
+
# Example evaluation data
|
|
94
|
+
df = pd.DataFrame({
|
|
95
|
+
"entity_id": ["A", "A", "B", "B"],
|
|
96
|
+
"date": pd.to_datetime(["2024-01-01", "2024-01-02"] * 2),
|
|
97
|
+
"actual": [10, 12, 7, 9],
|
|
98
|
+
"model_a": [9, 11, 8, 10],
|
|
99
|
+
"model_b": [11, 13, 6, 8],
|
|
100
|
+
})
|
|
101
|
+
|
|
102
|
+
# Compare models using a common evaluation contract
|
|
103
|
+
results = compare_models(
|
|
104
|
+
df,
|
|
105
|
+
actual_col="actual",
|
|
106
|
+
prediction_cols=["model_a", "model_b"],
|
|
107
|
+
entity_col="entity_id",
|
|
108
|
+
time_col="date",
|
|
109
|
+
)
|
|
110
|
+
|
|
111
|
+
print(results)
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
---
|
|
115
|
+
|
|
116
|
+
## License
|
|
117
|
+
|
|
118
|
+
BSD 3-Clause License.
|
|
119
|
+
© 2025 Kyle Corrie.
|
eb_evaluation-0.2.0/PKG-INFO
DELETED
|
@@ -1,169 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: eb-evaluation
|
|
3
|
-
Version: 0.2.0
|
|
4
|
-
Summary: Electric Barometer: DataFrame-based evaluation utilities for CWSL and related metrics.
|
|
5
|
-
Author-email: "Kyle Corrie (Economistician)" <kcorrie@economistician.com>
|
|
6
|
-
License-Expression: BSD-3-Clause
|
|
7
|
-
Project-URL: Homepage, https://github.com/Economistician/eb-evaluation
|
|
8
|
-
Project-URL: Repository, https://github.com/Economistician/eb-evaluation
|
|
9
|
-
Project-URL: Issues, https://github.com/Economistician/eb-evaluation/issues
|
|
10
|
-
Project-URL: Documentation, https://github.com/Economistician/eb-docs
|
|
11
|
-
Requires-Python: >=3.10
|
|
12
|
-
Description-Content-Type: text/markdown
|
|
13
|
-
License-File: LICENSE
|
|
14
|
-
Requires-Dist: numpy>=1.24
|
|
15
|
-
Requires-Dist: pandas>=2.0
|
|
16
|
-
Requires-Dist: eb-metrics>=0.1.1
|
|
17
|
-
Provides-Extra: dev
|
|
18
|
-
Requires-Dist: pytest>=8.0; extra == "dev"
|
|
19
|
-
Requires-Dist: pytest-cov>=5.0; extra == "dev"
|
|
20
|
-
Dynamic: license-file
|
|
21
|
-
|
|
22
|
-
# Electric Barometer Evaluation (`eb-evaluation`)
|
|
23
|
-
|
|
24
|
-

|
|
25
|
-

|
|
26
|
-
[](https://economistician.github.io/eb-docs/)
|
|
27
|
-

|
|
28
|
-
|
|
29
|
-
This repository contains the **evaluation and orchestration layer** of the
|
|
30
|
-
*Electric Barometer* ecosystem.
|
|
31
|
-
|
|
32
|
-
`eb-evaluation` sits above core metric implementations (`eb-metrics`) and
|
|
33
|
-
provides structured tools for applying Electric Barometer concepts to
|
|
34
|
-
real-world forecasting workflows, including readiness adjustment, model
|
|
35
|
-
comparison, sensitivity analysis, and dataframe-based evaluation.
|
|
36
|
-
|
|
37
|
-
Conceptual definitions and theoretical framing for the evaluation logic are
|
|
38
|
-
maintained in the companion research repository:
|
|
39
|
-
**`eb-papers`**.
|
|
40
|
-
|
|
41
|
-
---
|
|
42
|
-
|
|
43
|
-
## Naming convention
|
|
44
|
-
|
|
45
|
-
Electric Barometer packages follow standard Python packaging conventions:
|
|
46
|
-
|
|
47
|
-
- **Distribution names** (used with `pip install`) use hyphens
|
|
48
|
-
e.g. `pip install eb-evaluation`
|
|
49
|
-
- **Python import paths** use underscores
|
|
50
|
-
e.g. `import eb_evaluation`
|
|
51
|
-
|
|
52
|
-
This distinction is intentional and consistent across the Electric Barometer
|
|
53
|
-
ecosystem.
|
|
54
|
-
|
|
55
|
-
---
|
|
56
|
-
|
|
57
|
-
## Role Within Electric Barometer
|
|
58
|
-
|
|
59
|
-
Within the Electric Barometer ecosystem:
|
|
60
|
-
|
|
61
|
-
- **`eb-papers`** defines *concepts, frameworks, and meaning*
|
|
62
|
-
- **`eb-metrics`** implements *individual metrics*
|
|
63
|
-
- **`eb-evaluation`** orchestrates *how metrics are applied, combined, and interpreted*
|
|
64
|
-
|
|
65
|
-
This repository focuses on *evaluation logic*, not raw metric computation.
|
|
66
|
-
|
|
67
|
-
---
|
|
68
|
-
|
|
69
|
-
## What This Library Provides
|
|
70
|
-
|
|
71
|
-
- **Readiness adjustment logic** for modifying evaluation outputs based on
|
|
72
|
-
operational readiness signals
|
|
73
|
-
- **Model selection and comparison utilities** grounded in asymmetric loss and
|
|
74
|
-
readiness-aware metrics
|
|
75
|
-
- **Sensitivity and tolerance analysis** for cost ratios and service thresholds
|
|
76
|
-
- **DataFrame-oriented evaluation tools** for entity-level and time-based analysis
|
|
77
|
-
- **Feature engineering utilities** to support evaluation pipelines
|
|
78
|
-
|
|
79
|
-
---
|
|
80
|
-
|
|
81
|
-
## Scope
|
|
82
|
-
|
|
83
|
-
This repository focuses on **evaluation workflows and orchestration**, not
|
|
84
|
-
low-level metric definitions.
|
|
85
|
-
|
|
86
|
-
**In scope:**
|
|
87
|
-
- Applying EB metrics to datasets and model outputs
|
|
88
|
-
- Combining metrics into readiness-aware evaluation artifacts
|
|
89
|
-
- Model comparison and selection logic
|
|
90
|
-
- Sensitivity analysis and tolerance handling
|
|
91
|
-
|
|
92
|
-
**Out of scope:**
|
|
93
|
-
- Metric definitions and loss formulations (see `eb-metrics`)
|
|
94
|
-
- Conceptual frameworks and theory (see `eb-papers`)
|
|
95
|
-
- Model training or forecasting algorithms
|
|
96
|
-
|
|
97
|
-
---
|
|
98
|
-
|
|
99
|
-
## Installation
|
|
100
|
-
|
|
101
|
-
Once published, the package will be installable via PyPI:
|
|
102
|
-
|
|
103
|
-
```bash
|
|
104
|
-
pip install eb-evaluation
|
|
105
|
-
```
|
|
106
|
-
|
|
107
|
-
For development or local use:
|
|
108
|
-
|
|
109
|
-
```bash
|
|
110
|
-
pip install -e .
|
|
111
|
-
```
|
|
112
|
-
|
|
113
|
-
---
|
|
114
|
-
|
|
115
|
-
## Package Structure
|
|
116
|
-
|
|
117
|
-
The repository follows a modern Python package layout:
|
|
118
|
-
|
|
119
|
-
```text
|
|
120
|
-
eb-evaluation/
|
|
121
|
-
├── src/eb_evaluation/
|
|
122
|
-
│ ├── adjustment/ # Readiness and evaluation adjustments
|
|
123
|
-
│ ├── dataframe/ # DataFrame-based evaluation utilities
|
|
124
|
-
│ ├── features/ # Feature engineering helpers
|
|
125
|
-
│ ├── model_selection/ # Model comparison and selection logic
|
|
126
|
-
│ └── utils/ # Shared validation and helpers
|
|
127
|
-
│
|
|
128
|
-
├── tests/ # Unit tests mirroring package structure
|
|
129
|
-
├── pyproject.toml # Build and dependency configuration
|
|
130
|
-
├── README.md # Project documentation
|
|
131
|
-
└── LICENSE # BSD-3-Clause license
|
|
132
|
-
```
|
|
133
|
-
|
|
134
|
-
---
|
|
135
|
-
|
|
136
|
-
## Relationship to Other EB Repositories
|
|
137
|
-
|
|
138
|
-
- `eb-papers`
|
|
139
|
-
Source of truth for conceptual definitions and evaluation philosophy.
|
|
140
|
-
|
|
141
|
-
- `eb-metrics`
|
|
142
|
-
Provides the metric implementations used during evaluation.
|
|
143
|
-
|
|
144
|
-
- `eb-evaluation`
|
|
145
|
-
Orchestrates evaluation workflows using adapted models.
|
|
146
|
-
|
|
147
|
-
- `eb-adapters`
|
|
148
|
-
Ensures heterogeneous models can be evaluated consistently.
|
|
149
|
-
|
|
150
|
-
When discrepancies arise, conceptual intent in `eb-papers` should be treated as authoritative.
|
|
151
|
-
|
|
152
|
-
---
|
|
153
|
-
|
|
154
|
-
## Development and Testing
|
|
155
|
-
|
|
156
|
-
Tests are located under the `tests/` directory and mirror the package structure.
|
|
157
|
-
|
|
158
|
-
To run the test suite:
|
|
159
|
-
|
|
160
|
-
```bash
|
|
161
|
-
pytest
|
|
162
|
-
```
|
|
163
|
-
|
|
164
|
-
---
|
|
165
|
-
|
|
166
|
-
## Status
|
|
167
|
-
|
|
168
|
-
This package is under active development.
|
|
169
|
-
Public APIs may evolve prior to the first stable release.
|
eb_evaluation-0.2.0/README.md
DELETED
|
@@ -1,148 +0,0 @@
|
|
|
1
|
-
# Electric Barometer Evaluation (`eb-evaluation`)
|
|
2
|
-
|
|
3
|
-

|
|
4
|
-

|
|
5
|
-
[](https://economistician.github.io/eb-docs/)
|
|
6
|
-

|
|
7
|
-
|
|
8
|
-
This repository contains the **evaluation and orchestration layer** of the
|
|
9
|
-
*Electric Barometer* ecosystem.
|
|
10
|
-
|
|
11
|
-
`eb-evaluation` sits above core metric implementations (`eb-metrics`) and
|
|
12
|
-
provides structured tools for applying Electric Barometer concepts to
|
|
13
|
-
real-world forecasting workflows, including readiness adjustment, model
|
|
14
|
-
comparison, sensitivity analysis, and dataframe-based evaluation.
|
|
15
|
-
|
|
16
|
-
Conceptual definitions and theoretical framing for the evaluation logic are
|
|
17
|
-
maintained in the companion research repository:
|
|
18
|
-
**`eb-papers`**.
|
|
19
|
-
|
|
20
|
-
---
|
|
21
|
-
|
|
22
|
-
## Naming convention
|
|
23
|
-
|
|
24
|
-
Electric Barometer packages follow standard Python packaging conventions:
|
|
25
|
-
|
|
26
|
-
- **Distribution names** (used with `pip install`) use hyphens
|
|
27
|
-
e.g. `pip install eb-evaluation`
|
|
28
|
-
- **Python import paths** use underscores
|
|
29
|
-
e.g. `import eb_evaluation`
|
|
30
|
-
|
|
31
|
-
This distinction is intentional and consistent across the Electric Barometer
|
|
32
|
-
ecosystem.
|
|
33
|
-
|
|
34
|
-
---
|
|
35
|
-
|
|
36
|
-
## Role Within Electric Barometer
|
|
37
|
-
|
|
38
|
-
Within the Electric Barometer ecosystem:
|
|
39
|
-
|
|
40
|
-
- **`eb-papers`** defines *concepts, frameworks, and meaning*
|
|
41
|
-
- **`eb-metrics`** implements *individual metrics*
|
|
42
|
-
- **`eb-evaluation`** orchestrates *how metrics are applied, combined, and interpreted*
|
|
43
|
-
|
|
44
|
-
This repository focuses on *evaluation logic*, not raw metric computation.
|
|
45
|
-
|
|
46
|
-
---
|
|
47
|
-
|
|
48
|
-
## What This Library Provides
|
|
49
|
-
|
|
50
|
-
- **Readiness adjustment logic** for modifying evaluation outputs based on
|
|
51
|
-
operational readiness signals
|
|
52
|
-
- **Model selection and comparison utilities** grounded in asymmetric loss and
|
|
53
|
-
readiness-aware metrics
|
|
54
|
-
- **Sensitivity and tolerance analysis** for cost ratios and service thresholds
|
|
55
|
-
- **DataFrame-oriented evaluation tools** for entity-level and time-based analysis
|
|
56
|
-
- **Feature engineering utilities** to support evaluation pipelines
|
|
57
|
-
|
|
58
|
-
---
|
|
59
|
-
|
|
60
|
-
## Scope
|
|
61
|
-
|
|
62
|
-
This repository focuses on **evaluation workflows and orchestration**, not
|
|
63
|
-
low-level metric definitions.
|
|
64
|
-
|
|
65
|
-
**In scope:**
|
|
66
|
-
- Applying EB metrics to datasets and model outputs
|
|
67
|
-
- Combining metrics into readiness-aware evaluation artifacts
|
|
68
|
-
- Model comparison and selection logic
|
|
69
|
-
- Sensitivity analysis and tolerance handling
|
|
70
|
-
|
|
71
|
-
**Out of scope:**
|
|
72
|
-
- Metric definitions and loss formulations (see `eb-metrics`)
|
|
73
|
-
- Conceptual frameworks and theory (see `eb-papers`)
|
|
74
|
-
- Model training or forecasting algorithms
|
|
75
|
-
|
|
76
|
-
---
|
|
77
|
-
|
|
78
|
-
## Installation
|
|
79
|
-
|
|
80
|
-
Once published, the package will be installable via PyPI:
|
|
81
|
-
|
|
82
|
-
```bash
|
|
83
|
-
pip install eb-evaluation
|
|
84
|
-
```
|
|
85
|
-
|
|
86
|
-
For development or local use:
|
|
87
|
-
|
|
88
|
-
```bash
|
|
89
|
-
pip install -e .
|
|
90
|
-
```
|
|
91
|
-
|
|
92
|
-
---
|
|
93
|
-
|
|
94
|
-
## Package Structure
|
|
95
|
-
|
|
96
|
-
The repository follows a modern Python package layout:
|
|
97
|
-
|
|
98
|
-
```text
|
|
99
|
-
eb-evaluation/
|
|
100
|
-
├── src/eb_evaluation/
|
|
101
|
-
│ ├── adjustment/ # Readiness and evaluation adjustments
|
|
102
|
-
│ ├── dataframe/ # DataFrame-based evaluation utilities
|
|
103
|
-
│ ├── features/ # Feature engineering helpers
|
|
104
|
-
│ ├── model_selection/ # Model comparison and selection logic
|
|
105
|
-
│ └── utils/ # Shared validation and helpers
|
|
106
|
-
│
|
|
107
|
-
├── tests/ # Unit tests mirroring package structure
|
|
108
|
-
├── pyproject.toml # Build and dependency configuration
|
|
109
|
-
├── README.md # Project documentation
|
|
110
|
-
└── LICENSE # BSD-3-Clause license
|
|
111
|
-
```
|
|
112
|
-
|
|
113
|
-
---
|
|
114
|
-
|
|
115
|
-
## Relationship to Other EB Repositories
|
|
116
|
-
|
|
117
|
-
- `eb-papers`
|
|
118
|
-
Source of truth for conceptual definitions and evaluation philosophy.
|
|
119
|
-
|
|
120
|
-
- `eb-metrics`
|
|
121
|
-
Provides the metric implementations used during evaluation.
|
|
122
|
-
|
|
123
|
-
- `eb-evaluation`
|
|
124
|
-
Orchestrates evaluation workflows using adapted models.
|
|
125
|
-
|
|
126
|
-
- `eb-adapters`
|
|
127
|
-
Ensures heterogeneous models can be evaluated consistently.
|
|
128
|
-
|
|
129
|
-
When discrepancies arise, conceptual intent in `eb-papers` should be treated as authoritative.
|
|
130
|
-
|
|
131
|
-
---
|
|
132
|
-
|
|
133
|
-
## Development and Testing
|
|
134
|
-
|
|
135
|
-
Tests are located under the `tests/` directory and mirror the package structure.
|
|
136
|
-
|
|
137
|
-
To run the test suite:
|
|
138
|
-
|
|
139
|
-
```bash
|
|
140
|
-
pytest
|
|
141
|
-
```
|
|
142
|
-
|
|
143
|
-
---
|
|
144
|
-
|
|
145
|
-
## Status
|
|
146
|
-
|
|
147
|
-
This package is under active development.
|
|
148
|
-
Public APIs may evolve prior to the first stable release.
|
|
@@ -1,41 +0,0 @@
|
|
|
1
|
-
[project]
|
|
2
|
-
name = "eb-evaluation"
|
|
3
|
-
version = "0.2.0"
|
|
4
|
-
description = "Electric Barometer: DataFrame-based evaluation utilities for CWSL and related metrics."
|
|
5
|
-
readme = "README.md"
|
|
6
|
-
requires-python = ">=3.10"
|
|
7
|
-
license = "BSD-3-Clause"
|
|
8
|
-
license-files = ["LICENSE*"]
|
|
9
|
-
|
|
10
|
-
authors = [
|
|
11
|
-
{ name = "Kyle Corrie (Economistician)", email = "kcorrie@economistician.com" }
|
|
12
|
-
]
|
|
13
|
-
|
|
14
|
-
dependencies = [
|
|
15
|
-
"numpy>=1.24",
|
|
16
|
-
"pandas>=2.0",
|
|
17
|
-
"eb-metrics>=0.1.1",
|
|
18
|
-
]
|
|
19
|
-
|
|
20
|
-
[project.urls]
|
|
21
|
-
Homepage = "https://github.com/Economistician/eb-evaluation"
|
|
22
|
-
Repository = "https://github.com/Economistician/eb-evaluation"
|
|
23
|
-
Issues = "https://github.com/Economistician/eb-evaluation/issues"
|
|
24
|
-
Documentation = "https://github.com/Economistician/eb-docs"
|
|
25
|
-
|
|
26
|
-
[project.optional-dependencies]
|
|
27
|
-
dev = [
|
|
28
|
-
"pytest>=8.0",
|
|
29
|
-
"pytest-cov>=5.0",
|
|
30
|
-
]
|
|
31
|
-
|
|
32
|
-
[build-system]
|
|
33
|
-
requires = ["setuptools>=64", "wheel"]
|
|
34
|
-
build-backend = "setuptools.build_meta"
|
|
35
|
-
|
|
36
|
-
[tool.setuptools.packages.find]
|
|
37
|
-
where = ["src"]
|
|
38
|
-
|
|
39
|
-
[tool.pytest.ini_options]
|
|
40
|
-
pythonpath = ["src"]
|
|
41
|
-
addopts = "-ra"
|
|
@@ -1,169 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: eb-evaluation
|
|
3
|
-
Version: 0.2.0
|
|
4
|
-
Summary: Electric Barometer: DataFrame-based evaluation utilities for CWSL and related metrics.
|
|
5
|
-
Author-email: "Kyle Corrie (Economistician)" <kcorrie@economistician.com>
|
|
6
|
-
License-Expression: BSD-3-Clause
|
|
7
|
-
Project-URL: Homepage, https://github.com/Economistician/eb-evaluation
|
|
8
|
-
Project-URL: Repository, https://github.com/Economistician/eb-evaluation
|
|
9
|
-
Project-URL: Issues, https://github.com/Economistician/eb-evaluation/issues
|
|
10
|
-
Project-URL: Documentation, https://github.com/Economistician/eb-docs
|
|
11
|
-
Requires-Python: >=3.10
|
|
12
|
-
Description-Content-Type: text/markdown
|
|
13
|
-
License-File: LICENSE
|
|
14
|
-
Requires-Dist: numpy>=1.24
|
|
15
|
-
Requires-Dist: pandas>=2.0
|
|
16
|
-
Requires-Dist: eb-metrics>=0.1.1
|
|
17
|
-
Provides-Extra: dev
|
|
18
|
-
Requires-Dist: pytest>=8.0; extra == "dev"
|
|
19
|
-
Requires-Dist: pytest-cov>=5.0; extra == "dev"
|
|
20
|
-
Dynamic: license-file
|
|
21
|
-
|
|
22
|
-
# Electric Barometer Evaluation (`eb-evaluation`)
|
|
23
|
-
|
|
24
|
-

|
|
25
|
-

|
|
26
|
-
[](https://economistician.github.io/eb-docs/)
|
|
27
|
-

|
|
28
|
-
|
|
29
|
-
This repository contains the **evaluation and orchestration layer** of the
|
|
30
|
-
*Electric Barometer* ecosystem.
|
|
31
|
-
|
|
32
|
-
`eb-evaluation` sits above core metric implementations (`eb-metrics`) and
|
|
33
|
-
provides structured tools for applying Electric Barometer concepts to
|
|
34
|
-
real-world forecasting workflows, including readiness adjustment, model
|
|
35
|
-
comparison, sensitivity analysis, and dataframe-based evaluation.
|
|
36
|
-
|
|
37
|
-
Conceptual definitions and theoretical framing for the evaluation logic are
|
|
38
|
-
maintained in the companion research repository:
|
|
39
|
-
**`eb-papers`**.
|
|
40
|
-
|
|
41
|
-
---
|
|
42
|
-
|
|
43
|
-
## Naming convention
|
|
44
|
-
|
|
45
|
-
Electric Barometer packages follow standard Python packaging conventions:
|
|
46
|
-
|
|
47
|
-
- **Distribution names** (used with `pip install`) use hyphens
|
|
48
|
-
e.g. `pip install eb-evaluation`
|
|
49
|
-
- **Python import paths** use underscores
|
|
50
|
-
e.g. `import eb_evaluation`
|
|
51
|
-
|
|
52
|
-
This distinction is intentional and consistent across the Electric Barometer
|
|
53
|
-
ecosystem.
|
|
54
|
-
|
|
55
|
-
---
|
|
56
|
-
|
|
57
|
-
## Role Within Electric Barometer
|
|
58
|
-
|
|
59
|
-
Within the Electric Barometer ecosystem:
|
|
60
|
-
|
|
61
|
-
- **`eb-papers`** defines *concepts, frameworks, and meaning*
|
|
62
|
-
- **`eb-metrics`** implements *individual metrics*
|
|
63
|
-
- **`eb-evaluation`** orchestrates *how metrics are applied, combined, and interpreted*
|
|
64
|
-
|
|
65
|
-
This repository focuses on *evaluation logic*, not raw metric computation.
|
|
66
|
-
|
|
67
|
-
---
|
|
68
|
-
|
|
69
|
-
## What This Library Provides
|
|
70
|
-
|
|
71
|
-
- **Readiness adjustment logic** for modifying evaluation outputs based on
|
|
72
|
-
operational readiness signals
|
|
73
|
-
- **Model selection and comparison utilities** grounded in asymmetric loss and
|
|
74
|
-
readiness-aware metrics
|
|
75
|
-
- **Sensitivity and tolerance analysis** for cost ratios and service thresholds
|
|
76
|
-
- **DataFrame-oriented evaluation tools** for entity-level and time-based analysis
|
|
77
|
-
- **Feature engineering utilities** to support evaluation pipelines
|
|
78
|
-
|
|
79
|
-
---
|
|
80
|
-
|
|
81
|
-
## Scope
|
|
82
|
-
|
|
83
|
-
This repository focuses on **evaluation workflows and orchestration**, not
|
|
84
|
-
low-level metric definitions.
|
|
85
|
-
|
|
86
|
-
**In scope:**
|
|
87
|
-
- Applying EB metrics to datasets and model outputs
|
|
88
|
-
- Combining metrics into readiness-aware evaluation artifacts
|
|
89
|
-
- Model comparison and selection logic
|
|
90
|
-
- Sensitivity analysis and tolerance handling
|
|
91
|
-
|
|
92
|
-
**Out of scope:**
|
|
93
|
-
- Metric definitions and loss formulations (see `eb-metrics`)
|
|
94
|
-
- Conceptual frameworks and theory (see `eb-papers`)
|
|
95
|
-
- Model training or forecasting algorithms
|
|
96
|
-
|
|
97
|
-
---
|
|
98
|
-
|
|
99
|
-
## Installation
|
|
100
|
-
|
|
101
|
-
Once published, the package will be installable via PyPI:
|
|
102
|
-
|
|
103
|
-
```bash
|
|
104
|
-
pip install eb-evaluation
|
|
105
|
-
```
|
|
106
|
-
|
|
107
|
-
For development or local use:
|
|
108
|
-
|
|
109
|
-
```bash
|
|
110
|
-
pip install -e .
|
|
111
|
-
```
|
|
112
|
-
|
|
113
|
-
---
|
|
114
|
-
|
|
115
|
-
## Package Structure
|
|
116
|
-
|
|
117
|
-
The repository follows a modern Python package layout:
|
|
118
|
-
|
|
119
|
-
```text
|
|
120
|
-
eb-evaluation/
|
|
121
|
-
├── src/eb_evaluation/
|
|
122
|
-
│ ├── adjustment/ # Readiness and evaluation adjustments
|
|
123
|
-
│ ├── dataframe/ # DataFrame-based evaluation utilities
|
|
124
|
-
│ ├── features/ # Feature engineering helpers
|
|
125
|
-
│ ├── model_selection/ # Model comparison and selection logic
|
|
126
|
-
│ └── utils/ # Shared validation and helpers
|
|
127
|
-
│
|
|
128
|
-
├── tests/ # Unit tests mirroring package structure
|
|
129
|
-
├── pyproject.toml # Build and dependency configuration
|
|
130
|
-
├── README.md # Project documentation
|
|
131
|
-
└── LICENSE # BSD-3-Clause license
|
|
132
|
-
```
|
|
133
|
-
|
|
134
|
-
---
|
|
135
|
-
|
|
136
|
-
## Relationship to Other EB Repositories
|
|
137
|
-
|
|
138
|
-
- `eb-papers`
|
|
139
|
-
Source of truth for conceptual definitions and evaluation philosophy.
|
|
140
|
-
|
|
141
|
-
- `eb-metrics`
|
|
142
|
-
Provides the metric implementations used during evaluation.
|
|
143
|
-
|
|
144
|
-
- `eb-evaluation`
|
|
145
|
-
Orchestrates evaluation workflows using adapted models.
|
|
146
|
-
|
|
147
|
-
- `eb-adapters`
|
|
148
|
-
Ensures heterogeneous models can be evaluated consistently.
|
|
149
|
-
|
|
150
|
-
When discrepancies arise, conceptual intent in `eb-papers` should be treated as authoritative.
|
|
151
|
-
|
|
152
|
-
---
|
|
153
|
-
|
|
154
|
-
## Development and Testing
|
|
155
|
-
|
|
156
|
-
Tests are located under the `tests/` directory and mirror the package structure.
|
|
157
|
-
|
|
158
|
-
To run the test suite:
|
|
159
|
-
|
|
160
|
-
```bash
|
|
161
|
-
pytest
|
|
162
|
-
```
|
|
163
|
-
|
|
164
|
-
---
|
|
165
|
-
|
|
166
|
-
## Status
|
|
167
|
-
|
|
168
|
-
This package is under active development.
|
|
169
|
-
Public APIs may evolve prior to the first stable release.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/adjustment/readiness_adjustment.py
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/model_selection/auto_engine.py
RENAMED
|
File without changes
|
|
File without changes
|
{eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/model_selection/cwsl_regressor.py
RENAMED
|
File without changes
|
{eb_evaluation-0.2.0 → eb_evaluation-0.2.2}/src/eb_evaluation/model_selection/electric_barometer.py
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|