dfm-python 0.2.0__tar.gz → 0.2.1__tar.gz

dfm_python-0.2.1/CHANGELOG.md

@@ -0,0 +1,216 @@
+# Changelog
+
+## [0.2.1] - 2025-01-XX
+
+### Fixed
+
+- **Mixed Frequency Edge Case**: Fixed dimension mismatch in `init_conditions` when slower frequency idio components are added to `C` but not to `A`. Now properly creates `BQ`, `SQ`, and `initViQ` with correct dimensions.
+- **UnboundLocalError**: Fixed `F` variable initialization in `init_conditions` to prevent `UnboundLocalError` when block initialization fails.
+- **Kalman Filter**: Fixed `Y_t` initialization in `skf` to prevent `UnboundLocalError` when no observations are present.
+- **MergedConfigSource**: Fixed handling of partial configuration dictionaries (e.g., only `max_iter` and `threshold`) to correctly merge with base configuration.
+- **Tutorials**: Fixed pandas FutureWarning by updating `'M'` frequency to `'ME'` in both `basic_tutorial.py` and `hydra_tutorial.py`.
+- **Test Suite**: Recreated `test_config.py` from scratch to fix indentation errors and align with new API.
+
+### Documentation
+
+- **README**: Significantly simplified and streamlined (reduced from 653 to ~250 lines) while maintaining all essential information.
+- Improved clarity and focus on most common use cases.
+
+## [0.2.0] - 2025-01-XX
+
+### Major Refactoring
+
+This release represents a comprehensive refactoring focused on code quality, maintainability, and robustness.
+
+### Fixed
+
+- **Mixed Frequency Edge Case**: Fixed dimension mismatch in `init_conditions` when slower frequency idio components are added to `C` but not to `A`. Now properly creates `BQ`, `SQ`, and `initViQ` with correct dimensions.
+- **UnboundLocalError**: Fixed `F` variable initialization in `init_conditions` to prevent `UnboundLocalError` when block initialization fails.
+- **Kalman Filter**: Fixed `Y_t` initialization in `skf` to prevent `UnboundLocalError` when no observations are present.
+- **MergedConfigSource**: Fixed handling of partial configuration dictionaries (e.g., only `max_iter` and `threshold`) to correctly merge with base configuration.
+- **Tutorials**: Fixed pandas FutureWarning by updating `'M'` frequency to `'ME'` in both `basic_tutorial.py` and `hydra_tutorial.py`.
+- **Test Suite**: Recreated `test_config.py` from scratch to fix indentation errors and align with new API.
+
+#### Architecture Improvements
+
+- **Class-Centric Design**: Complete migration from functional API to class-based design
+  - Removed deprecated `dfm()` function wrapper
+  - Core algorithm now in `DFM` class with `fit()` method
+  - Module-level convenience API for easy usage
+  - All legacy backward compatibility code removed
+
+- **Modular Structure**: Reorganized code into focused `core` modules
+  - `core/em.py`: EM algorithm core (init_conditions, em_step, em_converged)
+  - `core/numeric.py`: Numerical utilities (matrix operations, regularization, clipping)
+  - `core/diagnostics.py`: Diagnostic functions and output formatting
+  - `core/results.py`: Result metrics (RMSE calculation)
+  - `core/grouping.py`: Frequency grouping utilities
+  - Removed 17 proxy functions, using direct imports for cleaner code
+
+#### Code Quality Improvements
+
+- **Exception Handling**: Replaced all generic `Exception` catches with specific numerical exception types
+  - Defined `_NUMERICAL_EXCEPTIONS` tuple for common numerical errors
+  - More specific error handling: `LinAlgError`, `ValueError`, `ZeroDivisionError`, `OverflowError`, `FloatingPointError`
+  - Better debugging and error tracking throughout the package
+
+- **Logging**: Improved logging infrastructure
+  - Converted `print()` statements to proper logging in diagnostics module
+  - Added early return for performance when logging is disabled
+  - Consistent logging levels (info, warning, debug) across modules
+  - User-facing functions (e.g., `print_series_diagnosis`) still use `print()` for direct output
+
+- **Import Optimization**: Cleaner import structure
+  - Moved pandas import to module top with availability check
+  - Removed redundant imports and circular dependencies
+  - Consistent import patterns across all modules
+
+#### Code Simplification
+
+- **Removed Redundancy**: Eliminated ~120 lines of redundant code
+  - Removed all proxy functions (17 total)
+  - Direct imports from core modules
+  - Cleaner, more maintainable codebase
+
+- **Helper Functions**: Extracted common patterns into reusable helpers
+  - `_check_finite()`: Centralized NaN/Inf validation
+  - `_ensure_square_matrix()`: Matrix shape validation
+  - `_get_config_param()`: Centralized config parameter access
+  - `_resolve_param()`: Parameter override resolution
+
+#### Type Safety
+
+- **TypedDict**: Added `NaNHandlingOptions` TypedDict for type-safe NaN handling
+- **Improved Type Hints**: Better type annotations throughout
+- **TYPE_CHECKING**: Used for circular import resolution
+
+#### Numerical Stability
+
+- **Enhanced Validation**: Better validation of initial conditions and intermediate results
+- **Improved Error Messages**: More descriptive error messages with context
+- **Robust Fallbacks**: Better fallback mechanisms for numerical edge cases
+
+### Changed
+
+- **BREAKING**: Removed `dfm()` function - use `DFM().fit()` instead
+- **BREAKING**: Removed deprecated `load_config_from_*` methods
+- Module structure reorganized for better maintainability
+- All diagnostic output now uses logging instead of print (except user-facing functions)
+- Exception handling now catches specific exception types instead of generic `Exception`
+
+### Added
+
+- `_NUMERICAL_EXCEPTIONS` tuple for consistent exception handling
+- `NaNHandlingOptions` TypedDict for type-safe NaN handling
+- Helper functions for common patterns (`_check_finite`, `_ensure_square_matrix`, etc.)
+- Early return optimization in diagnostics when logging is disabled
+- Better error messages with context
+
+### Removed
+
+- Deprecated `dfm()` function wrapper
+- All proxy functions (17 total) - now using direct imports
+- Legacy backward compatibility code
+- Redundant imports and circular dependencies
+- ~120 lines of redundant code
+
+### Fixed
+
+- Improved exception handling specificity
+- Better error messages for debugging
+- Fixed import circular dependencies using TYPE_CHECKING
+- Enhanced numerical stability validation
+
+### Documentation
+
+- Updated README to reflect new class-based API
+- Updated all docstrings to use new API
+- Improved code comments and documentation
+
+## [0.1.7] - 2025-01-XX
+
+### Changed
+- **BREAKING**: Removed CSV config loading - use YAML files or create `DFMConfig` objects directly
+- Made package more generic by removing application-specific dependencies
+- Improved code documentation and comments throughout codebase
+- Refactored variable naming for clarity and consistency
+- Updated README to be more concise and focused
+
+### Added
+- Direct `DFMConfig` object creation support in `load_config()`
+- Application-specific adapter guidance for custom formats
+- Database-backed data loading adapter interface documentation
+- Enhanced function docstrings with detailed parameter descriptions
+
+### Removed
+- CSV configuration file support (deprecated, use YAML or direct object creation)
+- `load_config_from_csv()` function (deprecated, kept for backward compatibility with warning)
+- Spec file dependencies
+
+### Fixed
+- Improved error messages for frequency constraint violations
+- Enhanced type hints and documentation
+
+## [0.1.6] - Previous release
+
+### Changed
+- Frequency generalization improvements
+- Code cleanup and refactoring
+
+## [0.1.5] - Previous release
+
+### Changed
+- Additional improvements and bug fixes
+
+## [0.1.4] - 2024-11-07
+
+### Fixed
+- Fixed critical bug: `ff` variable scoping issue in `init_conditions()` causing dimension mismatch errors in multi-block models with different factor counts
+- Fixed `A_temp` variable scoping issue that could cause incorrect computations when OLS regression fails for some blocks
+- Fixed `bl_idxM` and `bl_idxQ` initialization issue that could cause AttributeError when no blocks are processed
+- Removed unreachable code in `init_conditions()` else branch
+- Improved empty array handling in `em_step()` for quarterly series constraints
+
+### Changed
+- Enhanced variable scoping safety by resetting block-specific variables at start of each iteration
+- Improved error handling for edge cases with empty arrays
+
+## [0.1.3] - 2024-11-07
+
+### Added
+- Comprehensive README with detailed inputs/outputs documentation
+- Clock-based mixed-frequency framework documentation
+- Enhanced module-level docstrings throughout the codebase
+- Detailed API reference with examples
+- Improved error messages with context and solutions
+
+### Changed
+- Optimized code performance (removed redundant computations)
+- Improved memory usage (replaced unnecessary copies with views)
+- Enhanced code documentation and comments
+- Updated package description and metadata
+
+### Fixed
+- Removed duplicate `frequencies_array` computation
+- Fixed `optNaN` dictionary mutation issue
+- Improved code organization and clarity
+
+### Documentation
+- Complete input/output specifications
+- 5 comprehensive usage examples
+- Troubleshooting guide with solutions
+- Architecture overview
+- Clock-based framework explanation
+
+## [0.1.2] - Previous release
+
+### Fixed
+- Fixed `ModuleNotFoundError: No module named 'utils'` by moving utils into dfm_python package
+- Improved import paths and package structure
+
+## [0.1.1] - Initial release
+
+- Initial PyPI release
+- Core DFM estimation functionality
+- Mixed-frequency data support
+- News decomposition

dfm_python-0.2.1/PKG-INFO

@@ -0,0 +1,310 @@
+Metadata-Version: 2.4
+Name: dfm-python
+Version: 0.2.1
+Summary: Dynamic Factor Model (DFM) estimation and nowcasting in Python
+Project-URL: Homepage, https://github.com/yourusername/dfm-python
+Project-URL: Documentation, https://github.com/yourusername/dfm-python#readme
+Project-URL: Repository, https://github.com/yourusername/dfm-python
+Project-URL: Issues, https://github.com/yourusername/dfm-python/issues
+Author: DFM Python Contributors
+License: MIT
+License-File: LICENSE
+Keywords: dfm,dynamic-factor-model,econometrics,forecasting,nowcasting,time-series
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.12
+Requires-Dist: hydra-core>=1.3.2
+Requires-Dist: matplotlib>=3.10.7
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: scipy>=1.10.0
+Provides-Extra: all
+Requires-Dist: httpx>=0.24.0; extra == 'all'
+Requires-Dist: hydra-core>=1.3.0; extra == 'all'
+Requires-Dist: omegaconf>=2.3.0; extra == 'all'
+Requires-Dist: psycopg2-binary>=2.9.0; extra == 'all'
+Requires-Dist: pydantic-settings>=2.11.0; extra == 'all'
+Requires-Dist: pytest>=7.0.0; extra == 'all'
+Requires-Dist: supabase>=2.0.0; extra == 'all'
+Provides-Extra: database
+Requires-Dist: httpx>=0.24.0; extra == 'database'
+Requires-Dist: psycopg2-binary>=2.9.0; extra == 'database'
+Requires-Dist: pydantic-settings>=2.11.0; extra == 'database'
+Requires-Dist: supabase>=2.0.0; extra == 'database'
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == 'dev'
+Provides-Extra: hydra
+Requires-Dist: hydra-core>=1.3.0; extra == 'hydra'
+Requires-Dist: omegaconf>=2.3.0; extra == 'hydra'
+Description-Content-Type: text/markdown
+
+# dfm-python: Dynamic Factor Models for Python
+
+[![PyPI version](https://img.shields.io/pypi/v/dfm-python.svg)](https://pypi.org/project/dfm-python/)
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+
+A Python implementation of **Dynamic Factor Models (DFM)** for nowcasting and forecasting high-dimensional time series. Implements clock-based synchronization and tent kernel aggregation for mixed-frequency data.
+
+## Features
+
+- **Mixed-frequency data**: Monthly, quarterly, semi-annual, annual
+- **Clock-based framework**: All factors evolve at a common clock frequency
+- **Block structure**: Flexible factor organization (global, sector-specific)
+- **Robust missing data**: Spline interpolation and Kalman filter handling
+- **News decomposition**: Attribute forecast changes to data releases
+- **Generic design**: YAML configs or direct object creation, extensible via adapters
+
+## Installation
+
+```bash
+pip install dfm-python
+```
+
+**Requirements**: Python >= 3.12, numpy >= 1.24.0, pandas >= 2.0.0, scipy >= 1.10.0
+
+## Quick Start
+
+### Module-Level API (Recommended)
+
+```python
+import dfm_python as dfm
+from dfm_python.config import Params
+
+# 1. Load config from spec CSV + Params
+params = Params(max_iter=5000, threshold=1e-5)
+dfm.from_spec('data/sample_spec.csv', params=params)
+
+# 2. Load data
+dfm.load_data('data/sample_data.csv', sample_start='2021-01-01', sample_end='2022-12-31')
+
+# 3. Train
+dfm.train(max_iter=5000)
+
+# 4. Forecast
+X_forecast, Z_forecast = dfm.predict(horizon=12)
+
+# 5. Plot
+dfm.plot(kind='factor', factor_index=0, forecast_horizon=12, save_path='outputs/factor.png')
+
+# 6. Access results
+result = dfm.get_result()
+factors = result.Z          # (T × m) Smoothed factors
+loadings = result.C         # (N × m) Factor loadings
+smoothed = result.X_sm     # (T × N) Smoothed data
+```
+
+### Alternative: YAML Configuration
+
+```python
+import dfm_python as dfm
+
+# Load from YAML
+dfm.from_yaml('config/default.yaml')
+dfm.load_data('data/sample_data.csv')
+dfm.train()
+X_forecast, Z_forecast = dfm.predict(horizon=12)
+```
+
+### Alternative: Direct Class Usage
+
+```python
+from dfm_python import DFM, DFMConfig, SeriesConfig, BlockConfig
+
+# Create config
+config = DFMConfig(
+    series=[SeriesConfig(frequency='m', transformation='lin', blocks=[1], series_id='s1')],
+    blocks={'Block_Global': BlockConfig(factors=1, clock='m')}
+)
+
+# Fit model
+model = DFM()
+result = model.fit(X, config, max_iter=100)
+factors = result.Z
+```
+
+## Configuration
+
+### Spec CSV + Params (Recommended for CSV workflows)
+
+```python
+from dfm_python.config import Params
+
+params = Params(
+    max_iter=5000,
+    threshold=1e-5,
+    regularization_scale=1e-5,
+    damping_factor=0.8
+)
+dfm.from_spec('data/sample_spec.csv', params=params)
+```
+
+### Hydra Decorator (Recommended for Hydra users)
+
+```python
+import hydra
+from hydra.utils import get_original_cwd
+from omegaconf import DictConfig
+import dfm_python as dfm
+
+@hydra.main(config_path="../config", config_name="default", version_base="1.3")
+def main(cfg: DictConfig) -> None:
+    dfm.load_config(hydra=cfg)
+    dfm.load_data(str(get_original_cwd() / "data" / "sample_data.csv"))
+    dfm.train(max_iter=cfg.max_iter)
+    X_forecast, Z_forecast = dfm.predict(horizon=cfg.get('forecast_horizon', 12))
+```
+
+**CLI Overrides**: `python script.py max_iter=10 threshold=1e-4 blocks.Block_Global.factors=2`
+
+### Dictionary
+
+```python
+config_dict = {
+    'clock': 'm',
+    'max_iter': 5000,
+    'blocks': {'Block_Global': {'factors': 1, 'clock': 'm'}},
+    'series': [{'series_id': 's1', 'frequency': 'm', 'transformation': 'lin', 'blocks': [0]}]
+}
+dfm.from_dict(config_dict)
+```
+
+## Key Parameters
+
+- **clock**: Base frequency for latent factors (typically "m" for monthly)
+- **threshold**: EM convergence criterion (default: 1e-5)
+- **max_iter**: Maximum EM iterations (default: 5000)
+- **ar_lag**: Autoregressive lag for factors (default: 1)
+- **regularization_scale**: Scale factor for ridge regularization (default: 1e-5)
+- **damping_factor**: Damping factor for parameter updates (default: 0.8)
+
+All numerical stability parameters are configurable and transparent. See `Params` dataclass for full list.
+
+## Data Format
+
+CSV file with series as columns and dates as rows:
+
+```csv
+Date,gdp_real,consumption,investment
+2000-01-01,,98.5,95.0
+2000-02-01,,98.7,95.2
+2000-03-01,100.5,99.0,95.5
+```
+
+**Requirements**:
+- First column: `Date` (YYYY-MM-DD format)
+- Column names: Must match `series_id` in configuration
+- Missing values: Empty cells or NaN
+- Mixed frequencies: Quarterly series only at quarter-end months
+
+**Frequency Support**: Monthly (m), Quarterly (q), Semi-annual (sa), Annual (a)
+
+## News Decomposition
+
+```python
+from dfm_python import news_dfm
+
+# Decompose forecast updates
+y_old, y_new, singlenews, actual, forecast, weight, t_miss, v_miss, innov = news_dfm(
+    X_old, result_old, t_fcst=100, v_news=0
+)
+
+# Forecast update
+forecast_update = y_new - y_old
+```
+
+## API Reference
+
+### Module-Level Functions
+
+- `dfm.from_spec(spec_path, params=None)` - Load from spec CSV + Params
+- `dfm.from_yaml(yaml_path)` - Load from YAML file
+- `dfm.from_dict(config_dict)` - Load from dictionary
+- `dfm.load_data(data_path, sample_start=None, sample_end=None)` - Load data
+- `dfm.train(max_iter=None, threshold=None, **kwargs)` - Train model
+- `dfm.predict(horizon=12)` - Forecast series and factors
+- `dfm.plot(kind='factor', factor_index=0, forecast_horizon=None, save_path=None)` - Plot results
+- `dfm.get_result()` - Get `DFMResult` object
+
+### DFMResult Object
+
+Contains all estimation outputs:
+- **Factors**: `Z` (T × m), `C` (N × m)
+- **Parameters**: `A` (m × m), `Q` (m × m), `R` (N × N)
+- **Smoothed Data**: `X_sm` (T × N), `x_sm` (T × N)
+- **Convergence**: `converged`, `num_iter`, `loglik`
+- **Model Fit**: `rmse`, `rmse_per_series`
+
+## Architecture
+
+**Core Modules**:
+- `config.py`: Configuration management (DFMConfig, SeriesConfig, BlockConfig, Params)
+- `dfm.py`: Core estimation (DFM class with fit() method)
+- `api.py`: High-level convenience API
+- `kalman.py`: Kalman filter and smoother
+- `news.py`: News decomposition
+
+**Core Submodules**:
+- `core/em.py`: EM algorithm core
+- `core/numeric.py`: Numerical utilities
+- `core/diagnostics.py`: Diagnostic functions
+- `core/results.py`: Result metrics
+- `core/grouping.py`: Frequency grouping utilities
+
+## Testing
+
+```bash
+pytest src/test/ -v
+```
+
+Tests are organized into 5 files:
+- `test_dfm.py` - Core DFM estimation
+- `test_kalman.py` - Kalman filter/smoother
+- `test_news.py` - News decomposition
+- `test_config.py` - Configuration and API
+- `test_api.py` - Edge cases and tutorials
+
+## Tutorials
+
+1. **`tutorial/basic_tutorial.py`** - Spec CSV + Params approach
+   ```bash
+   python tutorial/basic_tutorial.py --spec data/sample_spec.csv --data data/sample_data.csv
+   ```
+
+2. **`tutorial/hydra_tutorial.py`** - Hydra decorator approach
+   ```bash
+   python tutorial/hydra_tutorial.py max_iter=10 threshold=1e-4
+   ```
+
+## Troubleshooting
+
+**Convergence Issues**: Increase `max_iter`, relax `threshold`, check data quality
+
+**Dimension Mismatch**: Ensure `series_id` matches data column names, verify block structure
+
+**Numerical Instability**: Monitor warnings, adjust thresholds, investigate data quality
+
+## License
+
+MIT License
+
+## Citation
+
+```bibtex
+@software{dfm-python,
+  title = {dfm-python: Dynamic Factor Models for Nowcasting and Forecasting},
+  author = {DFM Python Contributors},
+  year = {2025},
+  url = {https://pypi.org/project/dfm-python/}
+}
+```
+
+---
+
+**Package Status**: Stable (v0.2.1)  
+**PyPI**: https://pypi.org/project/dfm-python/  
+**Python**: 3.12+  
+**Latest Changes**: See [CHANGELOG.md](CHANGELOG.md)