dfm-python 0.2.3__tar.gz → 0.2.4__tar.gz

{dfm_python-0.2.3 → dfm_python-0.2.4}/CHANGELOG.md

@@ -1,5 +1,39 @@
 # Changelog
 
+## [0.2.4] - 2025-01-XX
+
+### Major Refactoring and Code Quality Improvements
+
+This release includes extensive refactoring focused on code maintainability, robustness, and documentation.
+
+#### Code Refactoring
+- **Consolidated covariance/variance computation**: Created `_compute_covariance_safe()` and `_compute_variance_safe()` functions to eliminate code duplication (17 patterns → 3 functions)
+- **Named constants**: Replaced 30+ magic numbers with named constants for better maintainability:
+  - `DEFAULT_AR_COEFFICIENT`, `DEFAULT_INNOVATION_VARIANCE`
+  - `MIN_INNOVATION_VARIANCE`, `MIN_OBSERVATION_VARIANCE`
+  - `DEFAULT_OBSERVATION_VARIANCE`, etc.
+- **Improved naming**: Consistent variable naming throughout codebase (e.g., `res_block` → `block_residuals`, `resNaN` → `residuals_with_nan`)
+- **Code deduplication**: Reduced code by ~80 lines through consolidation of repeated patterns
+
+#### Documentation
+- **Enhanced docstrings**: Comprehensive docstrings with examples for all major functions (`init_conditions`, `em_step`, `em_converged`)
+- **Updated README**: Added "Code Quality" section highlighting refactoring improvements
+- **Improved tutorials**: Enhanced `basic_tutorial.py` and `hydra_tutorial.py` with better explanations and examples
+
+#### Robustness
+- **Safe variance computation**: Automatic handling of edge cases (empty data, NaN values, numerical instability)
+- **Safe covariance computation**: Robust error handling with automatic fallback strategies
+- **Enhanced error messages**: More informative logging messages with context information
+
+#### Testing
+- All 38+ tests passing
+- No linter errors
+- Verified compatibility with existing functionality
+
+### Changed
+- Internal refactoring: No breaking changes to public API
+- Improved numerical stability through consolidated utility functions
+
 ## [0.2.3] - 2025-01-XX
 
 ### Changed

dfm_python-0.2.4/Nowcasting/LICENSE

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2018, Federal Reserve Bank of New York
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* Neither the name of the copyright holder nor the names of its
+  contributors may be used to endorse or promote products derived from
+  this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

dfm_python-0.2.4/Nowcasting/README.md

@@ -0,0 +1,24 @@
+# Nowcasting
+
+This code implements the nowcasting framework described in "[Macroeconomic Nowcasting and Forecasting with Big Data](https://www.newyorkfed.org/research/staff_reports/sr830.html)" by Brandyn Bok, Daniele Caratelli, Domenico Giannone, Argia M. Sbordone, and Andrea Tambalotti, *Staff Reports 830*, Federal Reserve Bank of New York (prepared for Volume 10 of the *Annual Review of Economics*).
+
+**Note:** These example files do not exactly reproduce the New York Fed Staff Nowcasting Report released every Friday because data redistribution restrictions prevent us from providing the complete data set used in our model.
+
+
+## Download instructions
+
+Download the code as a ZIP file by clicking the green 'Clone or download' button and selecting 'Download ZIP'.
+
+## File and folder description
+
+* `data/` : example US data downloaded from [FRED](https://fred.stlouisfed.org/)
+* `functions/` : functions for loading data, estimating model, and updating predictions
+* `example_DFM.m` : example script to estimate a dynamic factor model (DFM) for a panel of monthly and quarterly series
+* `example_Nowcast.m` : example script to produce a nowcast or forecast for a target variable, e.g., real GDP growth
+* `ResDFM.mat` : example DFM estimation output
+* `Spec_US_example.xls` : example model specification for the US
+
+
+## Required software and versioning
+
+MATLAB is required to run the code. The code was tested in MATLAB R2015b and later versions. Functionality with earlier versions of MATLAB is not guaranteed.

dfm_python-0.2.4/Nowcasting/example_DFM.m

@@ -0,0 +1,81 @@
+%%% Dynamic factor model (DFM) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% This script estimates a dynamic factor model (DFM) using a panel of
+% monthly and quarterly series.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+
+%% Clear workspace and set paths.
+close all; clear; clc;
+addpath('functions');
+
+
+%% User inputs.
+vintage = '2016-06-29'; % vintage dataset to use for estimation
+country = 'US';         % United States macroeconomic data
+sample_start  = datenum('2000-01-01','yyyy-mm-dd'); % estimation sample
+
+
+%% Load model specification and dataset.
+% Load model specification structure `Spec`
+Spec = load_spec('Spec_US_example.xls');
+% Parse `Spec`
+SeriesID = Spec.SeriesID; SeriesName = Spec.SeriesName; Units = Spec.Units; UnitsTransformed = Spec.UnitsTransformed;
+% Load data
+datafile = fullfile('data',country,[vintage '.xls']);
+[X,Time,Z] = load_data(datafile,Spec,sample_start);
+summarize(X,Time,Spec,vintage); % summarize data
+
+
+%% Plot raw and transformed data.
+% Industrial Production (INDPRO) <fred.stlouisfed.org/series/INDPRO>
+idxSeries = strcmp('INDPRO',SeriesID); t_obs = ~isnan(X(:,idxSeries));
+figure('Name',['Data - ' SeriesName{idxSeries}]);
+
+subplot(2,1,1); box on;
+plot(Time(t_obs),Z(t_obs,idxSeries)); title('raw observed data');
+ylabel(Units{idxSeries}); xlim(Time([1 end])); datetick('x','yyyy','keeplimits');
+
+subplot(2,1,2); box on;
+plot(Time(t_obs),X(t_obs,idxSeries)); title('transformed data');
+ylabel(UnitsTransformed{idxSeries}); xlim(Time([1 end])); datetick('x','yyyy','keeplimits');
+pause(1); % to display plot
+
+
+%% Run dynamic factor model (DFM) and save estimation output as 'ResDFM'.
+threshold = 1e-4; % Set to 1e-5 for more robust estimates
+
+Res = dfm(X,Spec,threshold);
+save('ResDFM','Res','Spec');
+
+
+%% Plot common factor and standardized data.
+idxSeries = strcmp('INDPRO',SeriesID);
+figure('Name','Common Factor and Standardized Data');
+plot(Time,Res.x_sm,':'); hold on;
+h = plot(Time,Res.Z(:,1)*Res.C(idxSeries,1),'k','LineWidth',1.5); box on;
+xlim(Time([1 end])); datetick('x','yyyy','keeplimits');
+legend(h,'common factor'); legend boxoff;
+pause(5); % to display plot
+
+
+%% Plot projection of common factor onto Payroll Employment and GDP.
+idxSeries = strcmp('PAYEMS',SeriesID); t_obs = ~isnan(X(:,idxSeries));
+figure('Name','Common Factor Projection');
+
+subplot(2,1,1); % projection of common factor onto PAYEMS
+CommonFactor = Res.C(idxSeries,1:5)*Res.Z(:,1:5)'*Res.Wx(idxSeries)+Res.Mx(idxSeries);
+plot(Time,CommonFactor,'k'); hold on;
+plot(Time(t_obs),X(t_obs,idxSeries),'b'); box on;
+title(SeriesName{idxSeries}); xlim(Time([1 end])); datetick('x','yyyy','keeplimits');
+ylabel({Units{idxSeries}, UnitsTransformed{idxSeries}});
+legend('common component','data'); legend boxoff;
+
+subplot(2,1,2); % projection of common factor onto GDPC1
+idxSeries = strcmp('GDPC1',SeriesID); t_obs = ~isnan(X(:,idxSeries));
+CommonFactor = Res.C(idxSeries,1:5)*Res.Z(:,1:5)'*Res.Wx(idxSeries)+Res.Mx(idxSeries);
+plot(Time,CommonFactor,'k'); hold on;
+plot(Time(t_obs), X(t_obs,idxSeries),'b'); box on;
+title(SeriesName{idxSeries}); xlim(Time([1 end])); datetick('x','yyyy','keeplimits');
+ylabel({Units{idxSeries},UnitsTransformed{idxSeries}});
+legend('common component','data'); legend boxoff;
+

dfm_python-0.2.4/Nowcasting/example_Nowcast.m

@@ -0,0 +1,50 @@
+%%% Nowcasting %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% This script produces a nowcast of US real GDP growth for 2016:Q4 
+% using the estimated parameters from a dynamic factor model.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+
+%% Clear workspace and set paths.
+close all; clear; clc;
+addpath('functions');
+
+
+%% User inputs.
+series = 'GDPC1' ; % Nowcasting real GDP (GDPC1) <fred.stlouisfed.org/series/GDPC1>
+period = '2016q4'; % Forecasting target quarter
+
+
+%% Load model specification and first vintage of data.
+% Load model specification structure `Spec`
+Spec = load_spec('Spec_US_example.xls');
+
+
+%% Load DFM estimation results structure `Res`.
+Res = load('ResDFM'); % example_DFM.m used the first vintage of data for estimation
+
+
+%% Update nowcast and decompose nowcast changes into news.
+
+%%% Nowcast update from week of December 7 to week of December 16, 2016 %%%
+vintage_old = '2016-12-16'; datafile_old = fullfile('data','US',[vintage_old '.xls']);
+vintage_new = '2016-12-23'; datafile_new = fullfile('data','US',[vintage_new '.xls']);
+% Load datasets for each vintage
+[X_old,~   ] = load_data(datafile_old,Spec);
+[X_new,Time] = load_data(datafile_new,Spec);
+
+% check if spec used in estimation is consistent with the current spec
+if isequal(Res.Spec,Spec)
+    Res = Res.Res;
+else
+    threshold = 1e-4; % Set to 1e-5 for robust estimates
+    Res = dfm(X_new,Spec,threshold);
+    save('ResDFM','Res','Spec');
+end 
+
+update_nowcast(X_old,X_new,Time,Spec,Res,series,period,vintage_old,vintage_new);
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+
+
+