dfm-python 0.4.3__tar.gz → 0.4.5__tar.gz

{dfm_python-0.4.3 → dfm_python-0.4.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dfm-python
-Version: 0.4.3
+Version: 0.4.5
 Summary: Dynamic Factor Model (DFM) estimation and nowcasting in Python
 Project-URL: Homepage, https://github.com/yourusername/dfm-python
 Project-URL: Documentation, https://github.com/yourusername/dfm-python#readme

dfm_python-0.4.5/config/README.md

@@ -0,0 +1,174 @@
+# Configuration Structure
+
+This directory contains all configuration files for the Nowcasting system. The configuration follows a **unified structure** where experiment configs contain all necessary settings including preprocessing and series configurations.
+
+## Directory Structure
+
+```
+config/
+├── experiment/     # Experiment configurations (unified structure)
+├── model/         # Model-specific hyperparameters
+├── preprocess/    # Preprocessing templates (for reference)
+└── series/        # Series configurations (legacy, now embedded in experiments)
+```
+
+## Unified Configuration Structure
+
+### Experiment Configs (`config/experiment/`)
+
+Experiment configs now contain **all** configuration in a single file:
+
+```yaml
+# Experiment Metadata
+name: experiment_name
+description: "Experiment description"
+model_type: dfm  # dfm, ddfm, arma, tft, xgboost
+
+# Preprocess Configuration
+preprocess:
+  metadata:
+    data_path: data/sample_data.csv
+    target_column: KOGDP...D
+    exclude_columns: [date]
+    sample_start: null
+    sample_end: null
+  
+  data_quality:
+    remove_sparse_rows: true
+    sparse_threshold: 0.8
+    clip_outliers: true
+    outlier_std: 3.0
+  
+  feature_engineering:
+    window_summarizer:
+      enabled: true
+      lags: [1, 2, 3, 5, 10]
+      windows: [5, 10, 20, 40]
+      functions: [mean, std, min, max, median, skew, kurt]
+  
+  global_preprocessing:
+    imputation:
+      method: ffill_bfill  # For non-DFM/DDFM models
+    scaling:
+      enabled: true
+      type: robust
+    transformations:
+      enabled: []
+
+# Series Configuration
+series:
+  - series_id: KO3YEARC
+    frequency: m
+    transformation: chg
+    blocks: [Block_Global, Block_Investment]
+    release: 1
+    scaler: null  # null = use global scaler
+    impute: null  # null = use global or Kalman filter (DFM/DDFM)
+```
+
+### Key Features
+
+1. **Unified Structure**: All experiment settings in one file
+2. **Conditional Imputation**: 
+   - DFM/DDFM: Uses Kalman filter (built-in, no explicit imputation needed)
+   - Other models: Uses explicit imputation (ffill_bfill, mean, median, forecaster)
+3. **Per-Series Overrides**: Each series can override global scaler/impute settings
+4. **Window Summarizer**: Automatic feature engineering with lags and rolling windows
+
+## Model Configs (`config/model/`)
+
+Model-specific hyperparameters:
+- `dfm.yaml` - DFM model parameters
+- `ddfm.yaml` - DDFM model parameters
+- `arma.yaml` - ARMA model parameters
+- `tft.yaml` - Temporal Fusion Transformer parameters
+- `xgboost.yaml` - XGBoost parameters
+
+## Preprocess Configs (`config/preprocess/`)
+
+**Note**: These are now templates/reference files. In the new structure, preprocessing is embedded in experiment configs. These files are kept for:
+- Reference and documentation
+- Backward compatibility
+- Template generation
+
+## Series Configs (`config/series/`)
+
+**Note**: These are legacy files. In the new structure, series configurations are embedded in experiment configs. These files are kept for:
+- Reference
+- Migration purposes
+- Template generation
+
+## Configuration Hierarchy
+
+1. **Experiment Config** (primary) - Contains all settings
+2. **Model Config** (referenced) - Model-specific hyperparameters
+3. **Preprocess Config** (legacy) - Reference templates only
+4. **Series Config** (legacy) - Reference templates only
+
+## Migration Guide
+
+### Old Structure
+```yaml
+# experiment/{id}.yaml
+data:
+  path: data/sample_data.csv
+preprocessing:
+  config_path: config/preprocess/{id}.yaml
+series: [KO3YEARC, ...]
+
+# preprocess/{id}.yaml (separate file)
+imputation:
+  method: ffill_bfill
+scaling:
+  type: robust
+
+# series/{series_id}.yaml (separate files)
+frequency: m
+transformation: chg
+blocks: [Block_Global]
+```
+
+### New Structure
+```yaml
+# experiment/{id}.yaml (unified)
+preprocess:
+  global_preprocessing:
+    imputation:
+      method: ffill_bfill
+    scaling:
+      type: robust
+series:
+  - series_id: KO3YEARC
+    frequency: m
+    transformation: chg
+    blocks: [Block_Global]
+```
+
+## Best Practices
+
+1. **Use Experiment Configs**: All new experiments should use the unified structure
+2. **Model Type Awareness**: 
+   - DFM/DDFM: Imputation handled by Kalman filter
+   - Other models: Specify explicit imputation method
+3. **Per-Series Overrides**: Use sparingly, prefer global settings
+4. **Window Summarizer**: Enable for feature-rich models (XGBoost, TFT)
+5. **Scaling**: Use RobustScaler for economic data with outliers
+
+## UI Integration
+
+The UI left sidebar displays the config tree:
+```
+Experiment
+├── Name, Model Type, Description
+├── Preprocess
+│   ├── Metadata (data path, target, exclude columns)
+│   ├── Data Quality
+│   ├── Feature Engineering (Window Summarizer)
+│   └── Global Preprocessing (imputation, scaling)
+└── Series
+    ├── KO3YEARC (frequency, transformation, blocks, release, scaler, impute)
+    └── [Other series...]
+```
+
+Right side: Training progress and monitoring.
+

dfm_python-0.4.5/config/experiment/default.yaml

@@ -0,0 +1,82 @@
+# Default Experiment Configuration
+# Unified experiment config with embedded preprocessing and series configurations
+
+# Hydra defaults - specify which model config to use
+defaults:
+  - model: dfm  # Options: dfm, ddfm, arma, tft, xgboost
+  - _self_
+
+# ============================================================================
+# Experiment Metadata
+# ============================================================================
+name: default_experiment
+description: "Default experiment configuration template"
+model_type: dfm  # dfm, ddfm, arma, tft, xgboost
+
+# ============================================================================
+# Preprocess Configuration
+# ============================================================================
+preprocess:
+  # Metadata
+  metadata:
+    data_path: data/sample_data.csv  # Path to input data file
+    target_column: null  # Target series for nowcasting/forecasting (e.g., KOGDP...D)
+    # Window Summarizer features (add_feature)
+    add_feature:  # Features to add via Window Summarizer
+      - lag      # Lag features
+      - mean     # Rolling mean
+      - std      # Rolling standard deviation
+      - min      # Rolling minimum
+      - max      # Rolling maximum
+      - median   # Rolling median
+      - skew     # Rolling skewness
+      - kurt     # Rolling kurtosis
+    # Window Summarizer parameters
+    lags: [1, 2, 3, 5, 10]  # Lag values: t-1, t-2, t-3, t-5, t-10
+    windows: [5, 10, 20, 40]  # Rolling window sizes
+  
+  # Series Configuration
+  # All series with their individual configurations
+  series:
+    # If empty, uses all columns from data (except excluded columns)
+    # Example series configurations:
+    # - series_id: KO3YEARC
+    #   frequency: m  # Options: d, w, m, q, sa, a
+    #   transformation: chg  # Options: lin, pch, pca, chg, ch1, cha
+    #   block:  # Block names this series loads on (must include Block_Global)
+    #     - Block_Global
+    #     - Block_Investment
+    #   release: 1  # Day of month when data is released (1-31, or negative for days before month end)
+    #   scaler: null  # Per-series scaler (StandardScaler, RobustScaler, None/null)
+    #                # null = use default (ffill and bfill for non-DFM/DDFM, Kalman filter for DFM/DDFM)
+    #   impute: null  # Per-series imputation (bfill, ffill, forecaster, dfm, ddfm)
+    #                # bfill/ffill: Backward/forward fill
+    #                # forecaster: Forecaster-based imputation
+    #                # dfm/ddfm: Use model's built-in Kalman filter imputation (on the fly)
+    #                # null: Use default (ffill and bfill for non-DFM/DDFM, Kalman filter for DFM/DDFM)
+
+# ============================================================================
+# Model Training Parameters
+# ============================================================================
+# These override model config defaults if specified
+max_iter: 5000      # Maximum EM iterations (for DFM)
+threshold: 1.0e-05  # Convergence threshold (for DFM)
+
+# For DDFM models:
+# encoder_layers: [64, 32]
+# num_factors: 2
+# epochs: 100
+# learning_rate: 0.001
+# batch_size: 32
+
+# ============================================================================
+# Output Configuration
+# ============================================================================
+output:
+  path: outputs/${experiment.name}  # Output directory (uses experiment name)
+  save_model: true      # Save trained model
+  save_plots: true      # Save visualization plots
+  save_results: true    # Save training results
+
+# Forecast configuration (optional)
+forecast_horizon: null  # Number of periods ahead to forecast

dfm_python-0.4.5/config/experiment/myexp_20251204233620.yaml

@@ -0,0 +1,108 @@
+# My Experiment Configuration
+# Example experiment config: myexp_20251204233620
+# Generated: 2025-12-04 23:36:20
+# Unified experiment config with embedded preprocessing and series configurations
+
+# Hydra defaults - specify which model config to use
+defaults:
+  - model: dfm  # Options: dfm, ddfm, arma, tft, xgboost
+  - _self_
+
+# ============================================================================
+# Experiment Metadata
+# ============================================================================
+name: myexp_20251204233620
+description: "My custom experiment configuration"
+model_type: dfm  # dfm, ddfm, arma, tft, xgboost
+
+# ============================================================================
+# Preprocess Configuration
+# ============================================================================
+preprocess:
+  # Metadata
+  metadata:
+    data_path: data/sample_data.csv  # Path to input data file
+    target_column: KOGDP...D  # Target series for nowcasting/forecasting
+    # Window Summarizer features (add_feature)
+    add_feature:  # Features to add via Window Summarizer
+      - lag      # Lag features
+      - mean     # Rolling mean
+      - std      # Rolling standard deviation
+      - min      # Rolling minimum
+      - max      # Rolling maximum
+      - median   # Rolling median
+      - skew     # Rolling skewness
+      - kurt     # Rolling kurtosis
+    # Window Summarizer parameters
+    lags: [1, 2, 3, 5, 10]  # Lag values: t-1, t-2, t-3, t-5, t-10
+    windows: [5, 10, 20, 40]  # Rolling window sizes
+  
+  # Series Configuration
+  # All series with their individual configurations
+  series:
+    - series_id: KO3YEARC
+      series_name: Corporate 3YR, AA-, Spread
+      frequency: m  # Options: d, w, m, q, sa, a
+      transformation: chg  # Options: lin, pch, pca, chg, ch1, cha
+      block:  # Block names this series loads on (must include Block_Global)
+        - Block_Global
+        - Block_Investment
+      release: 1  # Day of month when data is released
+      scaler: null  # Per-series scaler (StandardScaler, RobustScaler, None/null)
+                    # null = use default
+      impute: null  # Per-series imputation (bfill, ffill, forecaster, dfm, ddfm)
+                    # null = use default (ffill and bfill for non-DFM/DDFM, Kalman filter for DFM/DDFM)
+      units: Percent, Annual  # Optional metadata
+      category: Finance  # Optional metadata
+    
+    - series_id: KOGDP...D
+      series_name: GDP, Constant Prices
+      frequency: q  # Quarterly
+      transformation: cha  # Annual change
+      block:
+        - Block_Global
+      release: 25  # Released on 25th of month
+      scaler: null  # Use default
+      impute: null  # Use default (Kalman filter for DFM/DDFM)
+      units: Chained W, Billions
+      category: Macro
+    
+    - series_id: KOEMPTOTO
+      series_name: Employment, Total, SA
+      frequency: m  # Monthly
+      transformation: chg  # Change
+      block:
+        - Block_Global
+        - Block_Labor
+      release: 11  # Released on 11th of month
+      scaler: RobustScaler  # Override: use RobustScaler for this series
+      impute: null  # Use default (Kalman filter for DFM/DDFM)
+      units: Thousands of Persons
+      category: Labor
+
+# ============================================================================
+# Model Training Parameters
+# ============================================================================
+# These override model config defaults if specified
+max_iter: 5000      # Maximum EM iterations (for DFM)
+threshold: 1.0e-05  # Convergence threshold (for DFM)
+
+# For DDFM models:
+# encoder_layers: [64, 32]
+# num_factors: 2
+# epochs: 100
+# learning_rate: 0.001
+# batch_size: 32
+
+# ============================================================================
+# Output Configuration
+# ============================================================================
+output:
+  path: outputs/myexp_20251204233620  # Output directory
+  save_model: true      # Save trained model
+  save_plots: true      # Save visualization plots
+  save_results: true    # Save training results
+
+# Forecast configuration (optional)
+forecast_horizon: null  # Number of periods ahead to forecast
+

dfm_python-0.4.5/config/experiment/test_experiment_20251204170512.yaml

@@ -0,0 +1,107 @@
+# Test Experiment Configuration
+# Example experiment config following the pattern: test_experiment_{datetime}.yaml
+# Generated: 2025-12-04 17:05:12
+# Unified experiment config with embedded preprocessing and series configurations
+
+# Hydra defaults - specify which model config to use
+defaults:
+  - model: dfm  # Options: dfm, ddfm, arma, tft, xgboost
+  - _self_
+
+# ============================================================================
+# Experiment Metadata
+# ============================================================================
+name: test_experiment_20251204170512
+description: "Test experiment for preprocessing and model training with sample_data.csv"
+model_type: dfm  # dfm, ddfm, arma, tft, xgboost
+
+# ============================================================================
+# Preprocess Configuration
+# ============================================================================
+preprocess:
+  # Metadata
+  metadata:
+    data_path: data/sample_data.csv  # Path to input data file
+    target_column: KOGDP...D  # Target series for nowcasting/forecasting
+    # Window Summarizer features (add_feature)
+    add_feature:  # Features to add via Window Summarizer
+      - lag      # Lag features
+      - mean     # Rolling mean
+      - std      # Rolling standard deviation
+      - min      # Rolling minimum
+      - max      # Rolling maximum
+      - median   # Rolling median
+      - skew     # Rolling skewness
+      - kurt     # Rolling kurtosis
+    # Window Summarizer parameters
+    lags: [1, 2, 3, 5, 10]  # Lag values: t-1, t-2, t-3, t-5, t-10
+    windows: [5, 10, 20, 40]  # Rolling window sizes
+  
+  # Series Configuration
+  # All series with their individual configurations
+  series:
+    - series_id: KO3YEARC
+      series_name: Corporate 3YR, AA-, Spread
+      frequency: m  # Options: d, w, m, q, sa, a
+      transformation: chg  # Options: lin, pch, pca, chg, ch1, cha
+      block:  # Block names this series loads on (must include Block_Global)
+        - Block_Global
+        - Block_Investment
+      release: 1  # Day of month when data is released
+      scaler: null  # Per-series scaler (StandardScaler, RobustScaler, None/null)
+                    # null = use default
+      impute: null  # Per-series imputation (bfill, ffill, forecaster, dfm, ddfm)
+                    # null = use default (ffill and bfill for non-DFM/DDFM, Kalman filter for DFM/DDFM)
+      units: Percent, Annual  # Optional metadata
+      category: Finance  # Optional metadata
+    
+    - series_id: KOGDP...D
+      series_name: GDP, Constant Prices
+      frequency: q  # Quarterly
+      transformation: cha  # Annual change
+      block:
+        - Block_Global
+      release: 25  # Released on 25th of month
+      scaler: null  # Use default
+      impute: null  # Use default (Kalman filter for DFM/DDFM)
+      units: Chained W, Billions
+      category: Macro
+    
+    - series_id: KOEMPTOTO
+      series_name: Employment, Total, SA
+      frequency: m  # Monthly
+      transformation: chg  # Change
+      block:
+        - Block_Global
+        - Block_Labor
+      release: 11  # Released on 11th of month
+      scaler: RobustScaler  # Override: use RobustScaler for this series
+      impute: null  # Use default (Kalman filter for DFM/DDFM)
+      units: Thousands of Persons
+      category: Labor
+
+# ============================================================================
+# Model Training Parameters
+# ============================================================================
+# These override model config defaults if specified
+max_iter: 5000      # Maximum EM iterations (for DFM)
+threshold: 1.0e-05  # Convergence threshold (for DFM)
+
+# For DDFM models:
+# encoder_layers: [64, 32]
+# num_factors: 2
+# epochs: 100
+# learning_rate: 0.001
+# batch_size: 32
+
+# ============================================================================
+# Output Configuration
+# ============================================================================
+output:
+  path: outputs/test_experiment_20251204170512  # Output directory
+  save_model: true      # Save trained model
+  save_plots: true      # Save visualization plots
+  save_results: true    # Save training results
+
+# Forecast configuration (optional)
+forecast_horizon: null  # Number of periods ahead to forecast

dfm_python-0.4.5/config/model/arma.yaml

@@ -0,0 +1,5 @@
+order:
+- 1
+- 1
+max_iter: 1000
+threshold: 1.0e-05

dfm_python-0.4.5/config/model/ddfm.yaml

@@ -0,0 +1,48 @@
+ar_clip_max: 0.99
+ar_clip_min: -0.99
+ar_lag: 1
+augment_idio: true
+augment_idio_slow: true
+blocks:
+  Block_Global:
+    ar_lag: 1
+    clock: m
+    factors: 5
+  Block_Consumption:
+    ar_lag: 1
+    clock: m
+    factors: 2
+  Block_Investment:
+    ar_lag: 1
+    clock: m
+    factors: 2
+  Block_Export:
+    ar_lag: 1
+    clock: m
+    factors: 1
+  Block_Nominal:
+    ar_lag: 1
+    clock: m
+    factors: 1
+  Block_Labor:
+    ar_lag: 1
+    clock: m
+    factors: 1
+clip_ar_coefficients: true
+clock: m
+idio_min_var: 1.0e-08
+idio_rho0: 0.1
+max_iter: 5000
+nan_k: 3
+nan_method: 2
+regularization_scale: 1.0e-05
+threshold: 1.0e-05
+use_regularization: true
+encoder_layers:
+- 64
+- 32
+num_factors: 2
+dropout: 0.1
+learning_rate: 0.001
+batch_size: 32
+epochs: 100

{dfm_python-0.4.3 → dfm_python-0.4.5}/config/model/dfm.yaml

@@ -3,33 +3,31 @@ ar_clip_min: -0.99
 ar_lag: 1
 augment_idio: true
 augment_idio_slow: true
-# Block configurations: define all blocks and their properties
-# Series configs specify which blocks they belong to (optional - defaults to first block)
 blocks:
   Block_Global:
-    factors: 3
     ar_lag: 1
     clock: m
+    factors: 3
   Block_Consumption:
-    factors: 1
     ar_lag: 1
     clock: m
-  Block_Investment:
     factors: 1
+  Block_Investment:
     ar_lag: 1
     clock: m
-  Block_Export:
     factors: 1
+  Block_Export:
     ar_lag: 1
     clock: m
-  Block_Nominal:
     factors: 1
+  Block_Nominal:
     ar_lag: 1
     clock: m
-  Block_Labor:
     factors: 1
+  Block_Labor:
     ar_lag: 1
     clock: m
+    factors: 1
 clip_ar_coefficients: true
 clock: m
 idio_min_var: 1.0e-08

dfm_python-0.4.5/config/model/tft.yaml

@@ -0,0 +1,7 @@
+hidden_size: 64
+num_heads: 4
+num_layers: 2
+dropout: 0.1
+learning_rate: 0.001
+batch_size: 32
+epochs: 100

dfm_python-0.4.5/config/model/xgboost.yaml

@@ -0,0 +1,23 @@
+# XGBoost Model Configuration
+# Gradient boosting model for time series forecasting
+
+# XGBoost hyperparameters
+n_estimators: 100
+max_depth: 6
+learning_rate: 0.1
+subsample: 0.8
+colsample_bytree: 0.8
+min_child_weight: 1
+gamma: 0
+reg_alpha: 0
+reg_lambda: 1
+
+# Training parameters
+early_stopping_rounds: 10
+eval_metric: rmse
+objective: reg:squarederror
+
+# Time series specific
+# Note: XGBoost requires feature engineering (lags, rolling stats) before training
+# Use preprocessing configs to create features
+

dfm_python-0.4.5/config/preprocess/default.yaml

@@ -0,0 +1,35 @@
+# Default Preprocessing Configuration
+# NOTE: This is a standalone preprocessing config template.
+# In the new unified structure, preprocessing is embedded in experiment configs under preprocess.metadata and preprocess.series.
+# This file is kept for reference and backward compatibility.
+
+# Metadata section (maps to preprocess.metadata in experiment configs)
+metadata:
+  data_path: data/sample_data.csv
+  target_column: null
+  # Window Summarizer features
+  add_feature:
+    - lag
+    - mean
+    - std
+    - min
+    - max
+    - median
+    - skew
+    - kurt
+  lags: [1, 2, 3, 5, 10]
+  windows: [5, 10, 20, 40]
+
+# Series section (maps to preprocess.series in experiment configs)
+series: []
+# Example:
+# series:
+#   - series_id: KO3YEARC
+#     frequency: m
+#     transformation: chg
+#     block: [Block_Global, Block_Investment]
+#     release: 1
+#     scaler: null  # StandardScaler, RobustScaler, None/null
+#     impute: null  # bfill, ffill, forecaster, dfm, ddfm, null
+
+description: "Default preprocessing template - metadata and series structure"

dfm_python-0.4.5/config/preprocess/test_preprocess_20251204170512.yaml

@@ -0,0 +1,41 @@
+# Test Preprocessing Configuration
+# NOTE: This is a standalone preprocessing config template.
+# In the new unified structure, preprocessing is embedded in experiment configs under preprocess.metadata and preprocess.series.
+# This file is kept for reference and backward compatibility.
+
+# Metadata section (maps to preprocess.metadata in experiment configs)
+metadata:
+  data_path: data/sample_data.csv
+  target_column: KOGDP...D
+  # Window Summarizer features
+  add_feature:
+    - lag
+    - mean
+    - std
+    - min
+    - max
+    - median
+    - skew
+    - kurt
+  lags: [1, 2, 3, 5, 10]
+  windows: [5, 10, 20, 40]
+
+# Series section (maps to preprocess.series in experiment configs)
+series:
+  - series_id: KO3YEARC
+    frequency: m
+    transformation: chg
+    block: [Block_Global, Block_Investment]
+    release: 1
+    scaler: null
+    impute: null
+  
+  - series_id: KOGDP...D
+    frequency: q
+    transformation: cha
+    block: [Block_Global]
+    release: 25
+    scaler: null
+    impute: null
+
+description: "Test preprocessing template with example series"