patchfm 3.0.0__tar.gz → 3.1.1__tar.gz

{patchfm-3.0.0/src/patchfm.egg-info → patchfm-3.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: patchfm
-Version: 3.0.0
+Version: 3.1.1
 Summary: a Foundation Model for Univariate Time Series Forecasting
 Author-email: Samy-Melwan Vilhes <samy-melwan.vilhes@insa-rouen.fr>
 License: MIT License
@@ -42,20 +42,41 @@ Dynamic: license-file
 
 # A tutorial on how to build a Foundation Model for Univariate Time Series Forecasting
 
-[Huggingface Model Card](https://huggingface.co/vilhess/PatchFM)
-[Paper Model Card](https://github.com/vilhess/PatchFM/blob/main/main.pdf)
-
 A concise, reproducible recipe for training a transformer-based, patch-to-patch forecasting model for univariate time series. The approach mirrors Large Language Model (LLM) practices (next-token → next-patch) while remaining lightweight compared to a classic LLM and practical.
 
+Our model is deployed on the [TS-Arena benchmark](https://ts-arena.live) and achieves competitive performance against state-of-the-art methods under the name `LITIS/PatchFM-Large`.
+
+## Resuls on the FEV benchmark:
+
+**Our model is competitive with the state-of-the-art notably compared to [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)**
+
+| Model Name           | Win Rate | Skill Score |
+|----------------------|----------|-------------|
+| (🥇) Chronos-2           | 0.9008   | 0.4728      |
+| (🥈) TiRex               | 0.8092   | 0.4268      |
+| (🥉) TimesFM-2.5         | 0.8085   | 0.4668      |
+| (4) Toto-1.0            | 0.7069   | 0.4110      |
+| (5) **PatchFM (us)**             | 0.6677   | 0.3983      |
+| (6) Moirai-2.0          | 0.6546   | 0.4026      |
+| (7) Chronos-Bolt        | 0.6277   | 0.3889      |
+| (8) Sundial-Base        | 0.4446   | 0.3387      |
+| (9) CatBoost (Recursive)| 0.3362   | 0.2301      |
+| (10) LightGBM (Recursive)| 0.3123   | 0.2168      |
+| (11) AutoTheta           | 0.2938   | 0.0546      |
+| (12) Seasonal Naive      | 0.2058   | 0.0000      |
+| (13) Naive               | 0.1404   | -0.4540     |
+| (14) Drift               | 0.0915   | -0.4578     |
+
+
 ## Highlights
 - Next-patch prediction objective (autoregressive, causal)
 - Patch-based representation of time series (tokens ↔ patches)
 - Causal masking self-attention with RoPE (relative positions)
-- RevIN (Reversible Instance Normalization)
+- Causal RevIN with $\sinh^{-1}$ transform for normalization 
 - SwiGLU feed-forward networks
-- Multi-quantile outputs (median + uncertainty bands)
+- Autoregressive multi-quantile decoding [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)
 - KV-cache for efficient long-horizon inference
-- Autoregressive multi-quantile decoding [MOIRAI2.0](https://arxiv.org/pdf/2511.11698) (currently without KV-cache)
+- flips equivariance during inference (optional) [Reverso](https://arxiv.org/pdf/2602.17634v1)
 
 ## Quick Start
 
@@ -71,21 +92,22 @@ pip install -r requirements.txt
 
 ```python 
 import torch
-from model import Forecaster
+
 from configs import PatchFMConfig
+from model import Forecaster
 
 # --- Instantiate model ---
-config = PatchFMConfig(load_from_hub=True)
+config = PatchFMConfig(load_from_hub=True) 
 model = Forecaster(config)
 
 # --- Inference ---
 forecast_horizon = 64
 seq = torch.randn(1, 1024)  # (batch, time)
-pred_median, pred_quantiles = model(seq, forecast_horizon=forecast_horizon, quantiles=[0.1, 0.5, 0.9], quantile_decoding=True)  #  (batch, time), (batch, time, quantiles)
+pred_median, pred_quantiles = model(seq, forecast_horizon=forecast_horizon, quantiles=[0.1, 0.5, 0.9], flip_equivariance=True)  #  (batch, time), (batch, time, quantiles)
 ```
 
-### from pip package
 
+### from pip package
 1. Install the package from PyPI
 ```bash
 pip install patchfm
@@ -94,9 +116,11 @@ pip install patchfm
 
 ```python 
 import torch
-from patchfm import PatchFMConfig, Forecaster
+
+from patchfm import Forecaster, PatchFMConfig
 
 # same as above
+pred_median, pred_quantiles = model(seq, forecast_horizon=forecast_horizon, quantiles=[0.1, 0.5, 0.9], flip_equivariance=True)  #  (batch, time), (batch, time, quantiles)
 ```
 
 We provide an extended quick start example in [notebooks/tutorial.ipynb](./notebooks/tutorial.ipynb).
@@ -114,6 +138,7 @@ If you dont have suitable hardware you can run the the extended quick start exam
 - Training: Multi-quantile (pinball) loss across positions, elements, and quantiles $\mathcal{Q}$.
 - Inference: Predict next patch; roll out autoregressively for long horizons.
 - KV-cache: during inference, cache keys/values to avoid redundant computations.
+- Flip-equivariance: during inference, flip input sequence and average predictions to improve robustness (at cost of doubling batch size).
 
 ## Problem Formulation
 Given context patches $x_{p_1}, \ldots, x_{p_n}$, predict the next patch $x_{p_{i+1}}$ for each position $i$ using only past patches (causality). The model outputs quantiles $\{\hat{x}_{p_{i+1}}^{(q)}: q \in \mathcal{Q}\}$ with median (q=0.5) as the point forecast.
@@ -135,18 +160,83 @@ Aggregate over positions, patch elements, and quantiles.
 - Forecast horizon: 32 steps per forward pass
 - Quantiles $\mathcal{Q}$: {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}
 - Layers: 6
-- Attention heads: 64 (head dim 32)
+- Attention heads: 32 (head dim 64)
 - Model dim: 2048
 - Parameters: ~300M
 
 ## Inference
 - Single step: predict next patch ($P_{len}$ values)
 - Long-horizon: append prediction to context and repeat (optionally drop oldest patch to keep window fixed)
+- Flip-equivariance [Reverso](https://arxiv.org/pdf/2602.17634v1): optionally flip input sequence and average predictions to improve robustness (at cost of doubling batch size):
+$$y = \frac{1}{2} \left( f(x) - f(-x) \right)$$
+
+- ### Autoregressive Inference with Quantile Forecasting ([Moirai 2.0](https://arxiv.org/pdf/2511.11698v1))
+During autoregressive inference, the model generates forecasted values patch by patch. At each time step, the predicted patch is fed back into the model as input for the next step. This iterative process continues until the desired forecast horizon is reached.
+
+When performing quantile forecasting, the situation becomes more complex. Instead of producing a single patch per step, the model outputs multiple patches corresponding to different quantiles (e.g., 0.1, 0.5, 0.9). Since the model expects a single patch for the next time step, it is not straightforward to feed all quantile predictions back into the model simultaneously.
+
+A common workaround is to feed only the median prediction (the 0.5 quantile) back into the model at each step. While this approach preserves the autoregressive structure, it discards the uncertainty information captured by the other quantiles.
+
+An alternative approach is **autoregressive multi-quantile decoding**, as proposed in [Moirai 2.0](https://arxiv.org/pdf/2511.11698v1). This method enables consistent autoregressive generation while preserving the full predictive distribution across quantiles. However, it is computationally more expensive than the median-only approach as it requires duplicating the context for each quantile.
+
+<div style="display: flex; gap: 10px; align-items: flex-start;">
+  <div>
+    <img src="assets/classic_forecast_animation.gif" alt="Classic Autoregressive Inference" width="450">
+    <p style="text-align:center;">Classic Autoregressive Inference</p>
+  </div>
+  <div>
+    <img src="assets/quantile_forecast_animation.gif" alt="Autoregressive Multi-Quantile Decoding" width="450">
+    <p style="text-align:center;">Autoregressive Multi-Quantile Decoding</p>
+  </div>
+</div>
+
+The algorithm proceeds as follows:
+
+1. **Initialization**  
+   Start with the initial context window of observed data  
+   **Shape:** `(BS × L)`  
+   - `BS`: batch size  
+   - `L`: context length  
+   - `P`: patch size  
+   - `Q`: number of quantiles  
+   - `H`: forecast horizon  
+   - `i=1`: current algorithm step
+
+2. **First Quantile Prediction (Forward Pass)**  
+   Predict the quantiles for the next patch using the current context.  
+   **Output shape:** `(BS × P × Q)`
+
+3. **Context Duplication**  
+   For each predicted quantile, create a separate context by appending the corresponding predicted patch to the current context.  
+   This increases the number of contexts by a factor of `Q` at each step.  
+   **New context shape:** `(BS × Q × i(L + P))`
+
+4. **Next Forward Pass**  
+   For each duplicated context, predict the quantiles of the next patch.  
+   **Output shape:** `(BS × Q × P × Q)`
+
+5. **Quantile Collapse**  
+   - Permute and reshape the predictions to aggregate all possible quantile paths:  
+     **Intermediate shape:** `(BS × P × Q²)`  
+   - Compute the quantiles across the `Q²` predictions to obtain the final quantile estimates for the next patch.  
+     **Final shape:** `(BS × P × Q)`
+   - Increment the step counter `i ← i + 1`.
+
+6. **Iteration**  
+   Repeat Steps 3–5 until the forecast horizon `H` is reached, i.e., until the total number of predicted time steps satisfies  
+   `i × P ≥ H`.
+
+This procedure preserves predictive uncertainty across quantiles while maintaining the autoregressive structure of the model. Although it is computationally more expensive than feeding only the median prediction (0.5 quantile) back into the model, it remains tractable in practice and enables consistent multi-quantile forecasting.
+
+⚠️ **Warning**  
+With this strategy, the median prediction (0.5 quantile) does **not necessarily** match the prediction obtained by autoregressively feeding only the median patch back into the model at each step.
+
+This discrepancy arises because the *quantile collapse* step aggregates predictions across all possible quantile paths. As a result, the median is computed from the combined multi-path distribution rather than from a single deterministic trajectory, which can lead to different estimates compared to the single-path (median-only) autoregressive approach.
 
 ## Datasets
-- UTSD (Unified Time Series Dataset) [UTSD]: seven domains (Energy, IoT, Nature, Web, Health, Transport, Environment). We work with UTSD-12G (~18M series after preprocessing).
 - GIFT-Eval pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
 domains and various frequencies. After preprocessing, this yields approximately 600K univariate series. 
+- Chronos synthetic datasets [Chronos]: two large synthetic datasets generated with Chronos, one with TSMixup and one with KernelSynth. Each contains approximately 10 million univariate series and 1 million respectively, each signal of length 1024.
 - Artificial: ~1M synthetic series (sinusoidal, linear, polynomial, logarithmic) plus mixtures via TSMixup [Chronos]; Gaussian Process samples via KernelSynth (mixtures of RBF/periodic/linear kernels with swept hyperparameters).
 
 ## Repository Layout
@@ -164,10 +254,9 @@ domains and various frequencies. After preprocessing, this yields approximately
 
 - `dataset/` — data loading and preprocessing
   - `artificial.py` — synthetic dataset : artificial signals + TSMixup + KernelSynth
-  - `utsd.py` — Unified Time Series Dataset (UTSD) loading and preprocessing
   - `gift.py` — GIFT-Eval pretraining dataset loading and preprocessing
   - `get_data.py` — utility to fetch and preprocess datasets
-  - `generate_data.py` — utility to generate and save the KernelSynth dataset (long to generate)
+  - `chronosdata.py` — loading of the synthetic datasets generated with Chronos (TSMixup and KernelSynth) with download functions integrated
 
 - `configs/` — model and training configurations
 - `notebooks/inference` — how to load a trained model and generate forecasts

patchfm-3.1.1/README.md

@@ -0,0 +1,225 @@
+# A tutorial on how to build a Foundation Model for Univariate Time Series Forecasting
+
+A concise, reproducible recipe for training a transformer-based, patch-to-patch forecasting model for univariate time series. The approach mirrors Large Language Model (LLM) practices (next-token → next-patch) while remaining lightweight compared to a classic LLM and practical.
+
+Our model is deployed on the [TS-Arena benchmark](https://ts-arena.live) and achieves competitive performance against state-of-the-art methods under the name `LITIS/PatchFM-Large`.
+
+## Resuls on the FEV benchmark:
+
+**Our model is competitive with the state-of-the-art notably compared to [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)**
+
+| Model Name           | Win Rate | Skill Score |
+|----------------------|----------|-------------|
+| (🥇) Chronos-2           | 0.9008   | 0.4728      |
+| (🥈) TiRex               | 0.8092   | 0.4268      |
+| (🥉) TimesFM-2.5         | 0.8085   | 0.4668      |
+| (4) Toto-1.0            | 0.7069   | 0.4110      |
+| (5) **PatchFM (us)**             | 0.6677   | 0.3983      |
+| (6) Moirai-2.0          | 0.6546   | 0.4026      |
+| (7) Chronos-Bolt        | 0.6277   | 0.3889      |
+| (8) Sundial-Base        | 0.4446   | 0.3387      |
+| (9) CatBoost (Recursive)| 0.3362   | 0.2301      |
+| (10) LightGBM (Recursive)| 0.3123   | 0.2168      |
+| (11) AutoTheta           | 0.2938   | 0.0546      |
+| (12) Seasonal Naive      | 0.2058   | 0.0000      |
+| (13) Naive               | 0.1404   | -0.4540     |
+| (14) Drift               | 0.0915   | -0.4578     |
+
+
+## Highlights
+- Next-patch prediction objective (autoregressive, causal)
+- Patch-based representation of time series (tokens ↔ patches)
+- Causal masking self-attention with RoPE (relative positions)
+- Causal RevIN with $\sinh^{-1}$ transform for normalization 
+- SwiGLU feed-forward networks
+- Autoregressive multi-quantile decoding [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)
+- KV-cache for efficient long-horizon inference
+- flips equivariance during inference (optional) [Reverso](https://arxiv.org/pdf/2602.17634v1)
+
+## Quick Start
+
+### from source code
+
+1. Clone the repository and install dependencies
+```bash
+git clone https://github.com/vilhess/PatchFM
+cd PatchFM
+pip install -r requirements.txt
+```
+2. Run inference with a pretrained model from Huggingface Hub
+
+```python 
+import torch
+
+from configs import PatchFMConfig
+from model import Forecaster
+
+# --- Instantiate model ---
+config = PatchFMConfig(load_from_hub=True) 
+model = Forecaster(config)
+
+# --- Inference ---
+forecast_horizon = 64
+seq = torch.randn(1, 1024)  # (batch, time)
+pred_median, pred_quantiles = model(seq, forecast_horizon=forecast_horizon, quantiles=[0.1, 0.5, 0.9], flip_equivariance=True)  #  (batch, time), (batch, time, quantiles)
+```
+
+
+### from pip package
+1. Install the package from PyPI
+```bash
+pip install patchfm
+```
+2. Run inference with a pretrained model from Huggingface Hub
+
+```python 
+import torch
+
+from patchfm import Forecaster, PatchFMConfig
+
+# same as above
+pred_median, pred_quantiles = model(seq, forecast_horizon=forecast_horizon, quantiles=[0.1, 0.5, 0.9], flip_equivariance=True)  #  (batch, time), (batch, time, quantiles)
+```
+
+We provide an extended quick start example in [notebooks/tutorial.ipynb](./notebooks/tutorial.ipynb).
+If you dont have suitable hardware you can run the the extended quick start example example also in Google Colab:
+
+<a target="_blank" href="https://colab.research.google.com/drive/17sdf-7luCkv5TaeLj3Z6kIaTDkwkz3VR?usp=share_link">
+  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open Quick Start In Colab"/> 
+</a>
+
+## Method (TL;DR)
+- Patching: Split a context signal of length $w$ into $P_{num} = w / P_{len}$ patches of length $P_{len}$.
+- Causal RevIN: Normalize input signal and denormalize outputs to the original scale without statistics leakage.
+- Architecture: Input residual MLP → stacked Transformer blocks (MHA + SwiGLU FFN, pre-norm, residual) → $|\mathcal{Q}|$ output heads mapping back to patch space.
+- Positional encoding: Rotary Position Embeddings (RoPE) applied to queries/keys.
+- Training: Multi-quantile (pinball) loss across positions, elements, and quantiles $\mathcal{Q}$.
+- Inference: Predict next patch; roll out autoregressively for long horizons.
+- KV-cache: during inference, cache keys/values to avoid redundant computations.
+- Flip-equivariance: during inference, flip input sequence and average predictions to improve robustness (at cost of doubling batch size).
+
+## Problem Formulation
+Given context patches $x_{p_1}, \ldots, x_{p_n}$, predict the next patch $x_{p_{i+1}}$ for each position $i$ using only past patches (causality). The model outputs quantiles $\{\hat{x}_{p_{i+1}}^{(q)}: q \in \mathcal{Q}\}$ with median (q=0.5) as the point forecast.
+
+## Loss: Multi-Quantile (Pinball)
+For residual $u = x - \hat{x}^{(q)}$:
+$$\rho_q(u) = \begin{cases} q\,u, & u \ge 0,\\ (q-1)\,u, & u < 0. \end{cases}$$
+Aggregate over positions, patch elements, and quantiles.
+
+## Architecture
+- Input MLP: $\mathbb{R}^{P_{len}} \to \mathbb{R}^{dim}$ residual 2-layer MLP (ReLU)
+- Multi-Head Attention: causal mask, RoPE; queries/keys/values per head
+- FFN: SwiGLU (SiLU-gated), pre-norm + residual
+- Output heads: |Q| linear maps $\mathbb{R}^{dim} \to \mathbb{R}^{P_{len}}$ (one per quantile)
+
+### Model Details
+- Patch size: 32
+- Max context: 32 patches (1024 steps)
+- Forecast horizon: 32 steps per forward pass
+- Quantiles $\mathcal{Q}$: {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}
+- Layers: 6
+- Attention heads: 32 (head dim 64)
+- Model dim: 2048
+- Parameters: ~300M
+
+## Inference
+- Single step: predict next patch ($P_{len}$ values)
+- Long-horizon: append prediction to context and repeat (optionally drop oldest patch to keep window fixed)
+- Flip-equivariance [Reverso](https://arxiv.org/pdf/2602.17634v1): optionally flip input sequence and average predictions to improve robustness (at cost of doubling batch size):
+$$y = \frac{1}{2} \left( f(x) - f(-x) \right)$$
+
+- ### Autoregressive Inference with Quantile Forecasting ([Moirai 2.0](https://arxiv.org/pdf/2511.11698v1))
+During autoregressive inference, the model generates forecasted values patch by patch. At each time step, the predicted patch is fed back into the model as input for the next step. This iterative process continues until the desired forecast horizon is reached.
+
+When performing quantile forecasting, the situation becomes more complex. Instead of producing a single patch per step, the model outputs multiple patches corresponding to different quantiles (e.g., 0.1, 0.5, 0.9). Since the model expects a single patch for the next time step, it is not straightforward to feed all quantile predictions back into the model simultaneously.
+
+A common workaround is to feed only the median prediction (the 0.5 quantile) back into the model at each step. While this approach preserves the autoregressive structure, it discards the uncertainty information captured by the other quantiles.
+
+An alternative approach is **autoregressive multi-quantile decoding**, as proposed in [Moirai 2.0](https://arxiv.org/pdf/2511.11698v1). This method enables consistent autoregressive generation while preserving the full predictive distribution across quantiles. However, it is computationally more expensive than the median-only approach as it requires duplicating the context for each quantile.
+
+<div style="display: flex; gap: 10px; align-items: flex-start;">
+  <div>
+    <img src="assets/classic_forecast_animation.gif" alt="Classic Autoregressive Inference" width="450">
+    <p style="text-align:center;">Classic Autoregressive Inference</p>
+  </div>
+  <div>
+    <img src="assets/quantile_forecast_animation.gif" alt="Autoregressive Multi-Quantile Decoding" width="450">
+    <p style="text-align:center;">Autoregressive Multi-Quantile Decoding</p>
+  </div>
+</div>
+
+The algorithm proceeds as follows:
+
+1. **Initialization**  
+   Start with the initial context window of observed data  
+   **Shape:** `(BS × L)`  
+   - `BS`: batch size  
+   - `L`: context length  
+   - `P`: patch size  
+   - `Q`: number of quantiles  
+   - `H`: forecast horizon  
+   - `i=1`: current algorithm step
+
+2. **First Quantile Prediction (Forward Pass)**  
+   Predict the quantiles for the next patch using the current context.  
+   **Output shape:** `(BS × P × Q)`
+
+3. **Context Duplication**  
+   For each predicted quantile, create a separate context by appending the corresponding predicted patch to the current context.  
+   This increases the number of contexts by a factor of `Q` at each step.  
+   **New context shape:** `(BS × Q × i(L + P))`
+
+4. **Next Forward Pass**  
+   For each duplicated context, predict the quantiles of the next patch.  
+   **Output shape:** `(BS × Q × P × Q)`
+
+5. **Quantile Collapse**  
+   - Permute and reshape the predictions to aggregate all possible quantile paths:  
+     **Intermediate shape:** `(BS × P × Q²)`  
+   - Compute the quantiles across the `Q²` predictions to obtain the final quantile estimates for the next patch.  
+     **Final shape:** `(BS × P × Q)`
+   - Increment the step counter `i ← i + 1`.
+
+6. **Iteration**  
+   Repeat Steps 3–5 until the forecast horizon `H` is reached, i.e., until the total number of predicted time steps satisfies  
+   `i × P ≥ H`.
+
+This procedure preserves predictive uncertainty across quantiles while maintaining the autoregressive structure of the model. Although it is computationally more expensive than feeding only the median prediction (0.5 quantile) back into the model, it remains tractable in practice and enables consistent multi-quantile forecasting.
+
+⚠️ **Warning**  
+With this strategy, the median prediction (0.5 quantile) does **not necessarily** match the prediction obtained by autoregressively feeding only the median patch back into the model at each step.
+
+This discrepancy arises because the *quantile collapse* step aggregates predictions across all possible quantile paths. As a result, the median is computed from the combined multi-path distribution rather than from a single deterministic trajectory, which can lead to different estimates compared to the single-path (median-only) autoregressive approach.
+
+## Datasets
+- GIFT-Eval pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
+domains and various frequencies. After preprocessing, this yields approximately 600K univariate series. 
+- Chronos synthetic datasets [Chronos]: two large synthetic datasets generated with Chronos, one with TSMixup and one with KernelSynth. Each contains approximately 10 million univariate series and 1 million respectively, each signal of length 1024.
+- Artificial: ~1M synthetic series (sinusoidal, linear, polynomial, logarithmic) plus mixtures via TSMixup [Chronos]; Gaussian Process samples via KernelSynth (mixtures of RBF/periodic/linear kernels with swept hyperparameters).
+
+## Repository Layout
+
+- `model/training/` — main PatchFM model class
+
+  - `modules.py` - core modules (Residual Layers, MHA, SwiGLU, RoPE, Transformer Encoder, ...)
+  - `revin.py` — causal RevIN
+  - `loss.py` — multi-quantile (pinball) loss
+  - `trainer.py` — PyTorch Lightning trainer class
+
+- `model/inference/` — main PatchFM model class for inference 
+  - `modules.py` — core modules with caching support
+  - `forecaster.py` — Forecasting model and rollout logic
+
+- `dataset/` — data loading and preprocessing
+  - `artificial.py` — synthetic dataset : artificial signals + TSMixup + KernelSynth
+  - `gift.py` — GIFT-Eval pretraining dataset loading and preprocessing
+  - `get_data.py` — utility to fetch and preprocess datasets
+  - `chronosdata.py` — loading of the synthetic datasets generated with Chronos (TSMixup and KernelSynth) with download functions integrated
+
+- `configs/` — model and training configurations
+- `notebooks/inference` — how to load a trained model and generate forecasts
+- `training.py` — training script using PyTorch Lightning
+
+## Acknowledgements
+We thank the authors of the following repositories for inspiration and code snippets:
+- [TiRex](https://github.com/NX-AI/tirex)

{patchfm-3.0.0 → patchfm-3.1.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "patchfm"
-version = "3.0.0"
+version = "3.1.1"
 authors = [
   { name="Samy-Melwan Vilhes", email="samy-melwan.vilhes@insa-rouen.fr" },
 ]

{patchfm-3.0.0 → patchfm-3.1.1}/src/patchfm/configs/model_config.py

@@ -11,6 +11,7 @@ class PatchFMConfig:
     quantiles: list[float] = field(default_factory=lambda: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])
     
     compile: bool = True
+    full_leakage: bool = True
 
     def __getitem__(self, key):
         return getattr(self, key)

{patchfm-3.0.0 → patchfm-3.1.1}/src/patchfm/inference/forecaster.py

@@ -22,8 +22,13 @@ class Forecaster(nn.Module):
         self.n_quantiles = len(self.quantiles)
         self.max_patches = self.max_seq_len // self.patch_len
 
-        print("Loading base model from HuggingFace Hub...")
-        base_model = PatchFM.from_pretrained("vilhess/PatchFM")
+        if config["full_leakage"]:
+            print("Loading full leakage model from HuggingFace Hub...")
+            model_name = "vilhess/PatchFM-Leakage"
+        else:
+            print("Loading base model from HuggingFace Hub...")
+            model_name = "vilhess/PatchFM"
+        base_model = PatchFM.from_pretrained(model_name)
 
         self._init_from_base(base_model)
 
@@ -41,7 +46,7 @@ class Forecaster(nn.Module):
         self.revin = CausalRevIN()
         self.proj_embedding = ResidualBlock(
             in_dim=self.patch_len, 
-            hid_dim=2 * self.patch_len, 
+            hid_dim=4 * self.d_model, 
             out_dim=self.d_model
         )
         self.transformer_encoder = TransformerEncoder(
@@ -51,7 +56,7 @@ class Forecaster(nn.Module):
         )
         self.proj_output = ResidualBlock(
             in_dim=self.d_model, 
-            hid_dim=2 * self.d_model, 
+            hid_dim=4 * self.d_model, 
             out_dim=self.patch_len * self.n_quantiles
         )
 

{patchfm-3.0.0 → patchfm-3.1.1}/src/patchfm/inference/modules.py

@@ -175,7 +175,7 @@ class ResidualBlock(nn.Module):
         self.hidden_layer = nn.Linear(in_dim, hid_dim)
         self.output_layer = nn.Linear(hid_dim, out_dim)
         self.residual_layer = nn.Linear(in_dim, out_dim)
-        self.act = nn.ReLU()
+        self.act = nn.SiLU()
 
     def forward(self, x):
         hid = self.act(self.hidden_layer(x))
@@ -313,7 +313,7 @@ class PatchFM(nn.Module, PyTorchModelHubMixin):
         self.revin = CausalRevIN()
         self.proj_embedding = ResidualBlock(
             in_dim=self.patch_len, 
-            hid_dim=2 * self.patch_len, 
+            hid_dim=4 * self.d_model, 
             out_dim=self.d_model
         )
         self.transformer_encoder = TransformerEncoder(
@@ -323,6 +323,6 @@ class PatchFM(nn.Module, PyTorchModelHubMixin):
         )
         self.proj_output = ResidualBlock(
             in_dim=self.d_model, 
-            hid_dim=2 * self.d_model, 
+            hid_dim=4 * self.d_model, 
             out_dim=self.patch_len * self.n_quantiles
         )

{patchfm-3.0.0 → patchfm-3.1.1/src/patchfm.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: patchfm
-Version: 3.0.0
+Version: 3.1.1
 Summary: a Foundation Model for Univariate Time Series Forecasting
 Author-email: Samy-Melwan Vilhes <samy-melwan.vilhes@insa-rouen.fr>
 License: MIT License
@@ -42,20 +42,41 @@ Dynamic: license-file
 
 # A tutorial on how to build a Foundation Model for Univariate Time Series Forecasting
 
-[Huggingface Model Card](https://huggingface.co/vilhess/PatchFM)
-[Paper Model Card](https://github.com/vilhess/PatchFM/blob/main/main.pdf)
-
 A concise, reproducible recipe for training a transformer-based, patch-to-patch forecasting model for univariate time series. The approach mirrors Large Language Model (LLM) practices (next-token → next-patch) while remaining lightweight compared to a classic LLM and practical.
 
+Our model is deployed on the [TS-Arena benchmark](https://ts-arena.live) and achieves competitive performance against state-of-the-art methods under the name `LITIS/PatchFM-Large`.
+
+## Resuls on the FEV benchmark:
+
+**Our model is competitive with the state-of-the-art notably compared to [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)**
+
+| Model Name           | Win Rate | Skill Score |
+|----------------------|----------|-------------|
+| (🥇) Chronos-2           | 0.9008   | 0.4728      |
+| (🥈) TiRex               | 0.8092   | 0.4268      |
+| (🥉) TimesFM-2.5         | 0.8085   | 0.4668      |
+| (4) Toto-1.0            | 0.7069   | 0.4110      |
+| (5) **PatchFM (us)**             | 0.6677   | 0.3983      |
+| (6) Moirai-2.0          | 0.6546   | 0.4026      |
+| (7) Chronos-Bolt        | 0.6277   | 0.3889      |
+| (8) Sundial-Base        | 0.4446   | 0.3387      |
+| (9) CatBoost (Recursive)| 0.3362   | 0.2301      |
+| (10) LightGBM (Recursive)| 0.3123   | 0.2168      |
+| (11) AutoTheta           | 0.2938   | 0.0546      |
+| (12) Seasonal Naive      | 0.2058   | 0.0000      |
+| (13) Naive               | 0.1404   | -0.4540     |
+| (14) Drift               | 0.0915   | -0.4578     |
+
+
 ## Highlights
 - Next-patch prediction objective (autoregressive, causal)
 - Patch-based representation of time series (tokens ↔ patches)
 - Causal masking self-attention with RoPE (relative positions)
-- RevIN (Reversible Instance Normalization)
+- Causal RevIN with $\sinh^{-1}$ transform for normalization 
 - SwiGLU feed-forward networks
-- Multi-quantile outputs (median + uncertainty bands)
+- Autoregressive multi-quantile decoding [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)
 - KV-cache for efficient long-horizon inference
-- Autoregressive multi-quantile decoding [MOIRAI2.0](https://arxiv.org/pdf/2511.11698) (currently without KV-cache)
+- flips equivariance during inference (optional) [Reverso](https://arxiv.org/pdf/2602.17634v1)
 
 ## Quick Start
 
@@ -71,21 +92,22 @@ pip install -r requirements.txt
 
 ```python 
 import torch
-from model import Forecaster
+
 from configs import PatchFMConfig
+from model import Forecaster
 
 # --- Instantiate model ---
-config = PatchFMConfig(load_from_hub=True)
+config = PatchFMConfig(load_from_hub=True) 
 model = Forecaster(config)
 
 # --- Inference ---
 forecast_horizon = 64
 seq = torch.randn(1, 1024)  # (batch, time)
-pred_median, pred_quantiles = model(seq, forecast_horizon=forecast_horizon, quantiles=[0.1, 0.5, 0.9], quantile_decoding=True)  #  (batch, time), (batch, time, quantiles)
+pred_median, pred_quantiles = model(seq, forecast_horizon=forecast_horizon, quantiles=[0.1, 0.5, 0.9], flip_equivariance=True)  #  (batch, time), (batch, time, quantiles)
 ```
 
-### from pip package
 
+### from pip package
 1. Install the package from PyPI
 ```bash
 pip install patchfm
@@ -94,9 +116,11 @@ pip install patchfm
 
 ```python 
 import torch
-from patchfm import PatchFMConfig, Forecaster
+
+from patchfm import Forecaster, PatchFMConfig
 
 # same as above
+pred_median, pred_quantiles = model(seq, forecast_horizon=forecast_horizon, quantiles=[0.1, 0.5, 0.9], flip_equivariance=True)  #  (batch, time), (batch, time, quantiles)
 ```
 
 We provide an extended quick start example in [notebooks/tutorial.ipynb](./notebooks/tutorial.ipynb).
@@ -114,6 +138,7 @@ If you dont have suitable hardware you can run the the extended quick start exam
 - Training: Multi-quantile (pinball) loss across positions, elements, and quantiles $\mathcal{Q}$.
 - Inference: Predict next patch; roll out autoregressively for long horizons.
 - KV-cache: during inference, cache keys/values to avoid redundant computations.
+- Flip-equivariance: during inference, flip input sequence and average predictions to improve robustness (at cost of doubling batch size).
 
 ## Problem Formulation
 Given context patches $x_{p_1}, \ldots, x_{p_n}$, predict the next patch $x_{p_{i+1}}$ for each position $i$ using only past patches (causality). The model outputs quantiles $\{\hat{x}_{p_{i+1}}^{(q)}: q \in \mathcal{Q}\}$ with median (q=0.5) as the point forecast.
@@ -135,18 +160,83 @@ Aggregate over positions, patch elements, and quantiles.
 - Forecast horizon: 32 steps per forward pass
 - Quantiles $\mathcal{Q}$: {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}
 - Layers: 6
-- Attention heads: 64 (head dim 32)
+- Attention heads: 32 (head dim 64)
 - Model dim: 2048
 - Parameters: ~300M
 
 ## Inference
 - Single step: predict next patch ($P_{len}$ values)
 - Long-horizon: append prediction to context and repeat (optionally drop oldest patch to keep window fixed)
+- Flip-equivariance [Reverso](https://arxiv.org/pdf/2602.17634v1): optionally flip input sequence and average predictions to improve robustness (at cost of doubling batch size):
+$$y = \frac{1}{2} \left( f(x) - f(-x) \right)$$
+
+- ### Autoregressive Inference with Quantile Forecasting ([Moirai 2.0](https://arxiv.org/pdf/2511.11698v1))
+During autoregressive inference, the model generates forecasted values patch by patch. At each time step, the predicted patch is fed back into the model as input for the next step. This iterative process continues until the desired forecast horizon is reached.
+
+When performing quantile forecasting, the situation becomes more complex. Instead of producing a single patch per step, the model outputs multiple patches corresponding to different quantiles (e.g., 0.1, 0.5, 0.9). Since the model expects a single patch for the next time step, it is not straightforward to feed all quantile predictions back into the model simultaneously.
+
+A common workaround is to feed only the median prediction (the 0.5 quantile) back into the model at each step. While this approach preserves the autoregressive structure, it discards the uncertainty information captured by the other quantiles.
+
+An alternative approach is **autoregressive multi-quantile decoding**, as proposed in [Moirai 2.0](https://arxiv.org/pdf/2511.11698v1). This method enables consistent autoregressive generation while preserving the full predictive distribution across quantiles. However, it is computationally more expensive than the median-only approach as it requires duplicating the context for each quantile.
+
+<div style="display: flex; gap: 10px; align-items: flex-start;">
+  <div>
+    <img src="assets/classic_forecast_animation.gif" alt="Classic Autoregressive Inference" width="450">
+    <p style="text-align:center;">Classic Autoregressive Inference</p>
+  </div>
+  <div>
+    <img src="assets/quantile_forecast_animation.gif" alt="Autoregressive Multi-Quantile Decoding" width="450">
+    <p style="text-align:center;">Autoregressive Multi-Quantile Decoding</p>
+  </div>
+</div>
+
+The algorithm proceeds as follows:
+
+1. **Initialization**  
+   Start with the initial context window of observed data  
+   **Shape:** `(BS × L)`  
+   - `BS`: batch size  
+   - `L`: context length  
+   - `P`: patch size  
+   - `Q`: number of quantiles  
+   - `H`: forecast horizon  
+   - `i=1`: current algorithm step
+
+2. **First Quantile Prediction (Forward Pass)**  
+   Predict the quantiles for the next patch using the current context.  
+   **Output shape:** `(BS × P × Q)`
+
+3. **Context Duplication**  
+   For each predicted quantile, create a separate context by appending the corresponding predicted patch to the current context.  
+   This increases the number of contexts by a factor of `Q` at each step.  
+   **New context shape:** `(BS × Q × i(L + P))`
+
+4. **Next Forward Pass**  
+   For each duplicated context, predict the quantiles of the next patch.  
+   **Output shape:** `(BS × Q × P × Q)`
+
+5. **Quantile Collapse**  
+   - Permute and reshape the predictions to aggregate all possible quantile paths:  
+     **Intermediate shape:** `(BS × P × Q²)`  
+   - Compute the quantiles across the `Q²` predictions to obtain the final quantile estimates for the next patch.  
+     **Final shape:** `(BS × P × Q)`
+   - Increment the step counter `i ← i + 1`.
+
+6. **Iteration**  
+   Repeat Steps 3–5 until the forecast horizon `H` is reached, i.e., until the total number of predicted time steps satisfies  
+   `i × P ≥ H`.
+
+This procedure preserves predictive uncertainty across quantiles while maintaining the autoregressive structure of the model. Although it is computationally more expensive than feeding only the median prediction (0.5 quantile) back into the model, it remains tractable in practice and enables consistent multi-quantile forecasting.
+
+⚠️ **Warning**  
+With this strategy, the median prediction (0.5 quantile) does **not necessarily** match the prediction obtained by autoregressively feeding only the median patch back into the model at each step.
+
+This discrepancy arises because the *quantile collapse* step aggregates predictions across all possible quantile paths. As a result, the median is computed from the combined multi-path distribution rather than from a single deterministic trajectory, which can lead to different estimates compared to the single-path (median-only) autoregressive approach.
 
 ## Datasets
-- UTSD (Unified Time Series Dataset) [UTSD]: seven domains (Energy, IoT, Nature, Web, Health, Transport, Environment). We work with UTSD-12G (~18M series after preprocessing).
 - GIFT-Eval pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
 domains and various frequencies. After preprocessing, this yields approximately 600K univariate series. 
+- Chronos synthetic datasets [Chronos]: two large synthetic datasets generated with Chronos, one with TSMixup and one with KernelSynth. Each contains approximately 10 million univariate series and 1 million respectively, each signal of length 1024.
 - Artificial: ~1M synthetic series (sinusoidal, linear, polynomial, logarithmic) plus mixtures via TSMixup [Chronos]; Gaussian Process samples via KernelSynth (mixtures of RBF/periodic/linear kernels with swept hyperparameters).
 
 ## Repository Layout
@@ -164,10 +254,9 @@ domains and various frequencies. After preprocessing, this yields approximately
 
 - `dataset/` — data loading and preprocessing
   - `artificial.py` — synthetic dataset : artificial signals + TSMixup + KernelSynth
-  - `utsd.py` — Unified Time Series Dataset (UTSD) loading and preprocessing
   - `gift.py` — GIFT-Eval pretraining dataset loading and preprocessing
   - `get_data.py` — utility to fetch and preprocess datasets
-  - `generate_data.py` — utility to generate and save the KernelSynth dataset (long to generate)
+  - `chronosdata.py` — loading of the synthetic datasets generated with Chronos (TSMixup and KernelSynth) with download functions integrated
 
 - `configs/` — model and training configurations
 - `notebooks/inference` — how to load a trained model and generate forecasts

patchfm-3.0.0/README.md

@@ -1,136 +0,0 @@
-# A tutorial on how to build a Foundation Model for Univariate Time Series Forecasting
-
-[Huggingface Model Card](https://huggingface.co/vilhess/PatchFM)
-[Paper Model Card](https://github.com/vilhess/PatchFM/blob/main/main.pdf)
-
-A concise, reproducible recipe for training a transformer-based, patch-to-patch forecasting model for univariate time series. The approach mirrors Large Language Model (LLM) practices (next-token → next-patch) while remaining lightweight compared to a classic LLM and practical.
-
-## Highlights
-- Next-patch prediction objective (autoregressive, causal)
-- Patch-based representation of time series (tokens ↔ patches)
-- Causal masking self-attention with RoPE (relative positions)
-- RevIN (Reversible Instance Normalization)
-- SwiGLU feed-forward networks
-- Multi-quantile outputs (median + uncertainty bands)
-- KV-cache for efficient long-horizon inference
-- Autoregressive multi-quantile decoding [MOIRAI2.0](https://arxiv.org/pdf/2511.11698) (currently without KV-cache)
-
-## Quick Start
-
-### from source code
-
-1. Clone the repository and install dependencies
-```bash
-git clone https://github.com/vilhess/PatchFM
-cd PatchFM
-pip install -r requirements.txt
-```
-2. Run inference with a pretrained model from Huggingface Hub
-
-```python 
-import torch
-from model import Forecaster
-from configs import PatchFMConfig
-
-# --- Instantiate model ---
-config = PatchFMConfig(load_from_hub=True)
-model = Forecaster(config)
-
-# --- Inference ---
-forecast_horizon = 64
-seq = torch.randn(1, 1024)  # (batch, time)
-pred_median, pred_quantiles = model(seq, forecast_horizon=forecast_horizon, quantiles=[0.1, 0.5, 0.9], quantile_decoding=True)  #  (batch, time), (batch, time, quantiles)
-```
-
-### from pip package
-
-1. Install the package from PyPI
-```bash
-pip install patchfm
-```
-2. Run inference with a pretrained model from Huggingface Hub
-
-```python 
-import torch
-from patchfm import PatchFMConfig, Forecaster
-
-# same as above
-```
-
-We provide an extended quick start example in [notebooks/tutorial.ipynb](./notebooks/tutorial.ipynb).
-If you dont have suitable hardware you can run the the extended quick start example example also in Google Colab:
-
-<a target="_blank" href="https://colab.research.google.com/drive/17sdf-7luCkv5TaeLj3Z6kIaTDkwkz3VR?usp=share_link">
-  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open Quick Start In Colab"/> 
-</a>
-
-## Method (TL;DR)
-- Patching: Split a context signal of length $w$ into $P_{num} = w / P_{len}$ patches of length $P_{len}$.
-- Causal RevIN: Normalize input signal and denormalize outputs to the original scale without statistics leakage.
-- Architecture: Input residual MLP → stacked Transformer blocks (MHA + SwiGLU FFN, pre-norm, residual) → $|\mathcal{Q}|$ output heads mapping back to patch space.
-- Positional encoding: Rotary Position Embeddings (RoPE) applied to queries/keys.
-- Training: Multi-quantile (pinball) loss across positions, elements, and quantiles $\mathcal{Q}$.
-- Inference: Predict next patch; roll out autoregressively for long horizons.
-- KV-cache: during inference, cache keys/values to avoid redundant computations.
-
-## Problem Formulation
-Given context patches $x_{p_1}, \ldots, x_{p_n}$, predict the next patch $x_{p_{i+1}}$ for each position $i$ using only past patches (causality). The model outputs quantiles $\{\hat{x}_{p_{i+1}}^{(q)}: q \in \mathcal{Q}\}$ with median (q=0.5) as the point forecast.
-
-## Loss: Multi-Quantile (Pinball)
-For residual $u = x - \hat{x}^{(q)}$:
-$$\rho_q(u) = \begin{cases} q\,u, & u \ge 0,\\ (q-1)\,u, & u < 0. \end{cases}$$
-Aggregate over positions, patch elements, and quantiles.
-
-## Architecture
-- Input MLP: $\mathbb{R}^{P_{len}} \to \mathbb{R}^{dim}$ residual 2-layer MLP (ReLU)
-- Multi-Head Attention: causal mask, RoPE; queries/keys/values per head
-- FFN: SwiGLU (SiLU-gated), pre-norm + residual
-- Output heads: |Q| linear maps $\mathbb{R}^{dim} \to \mathbb{R}^{P_{len}}$ (one per quantile)
-
-### Model Details
-- Patch size: 32
-- Max context: 32 patches (1024 steps)
-- Forecast horizon: 32 steps per forward pass
-- Quantiles $\mathcal{Q}$: {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}
-- Layers: 6
-- Attention heads: 64 (head dim 32)
-- Model dim: 2048
-- Parameters: ~300M
-
-## Inference
-- Single step: predict next patch ($P_{len}$ values)
-- Long-horizon: append prediction to context and repeat (optionally drop oldest patch to keep window fixed)
-
-## Datasets
-- UTSD (Unified Time Series Dataset) [UTSD]: seven domains (Energy, IoT, Nature, Web, Health, Transport, Environment). We work with UTSD-12G (~18M series after preprocessing).
-- GIFT-Eval pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
-domains and various frequencies. After preprocessing, this yields approximately 600K univariate series. 
-- Artificial: ~1M synthetic series (sinusoidal, linear, polynomial, logarithmic) plus mixtures via TSMixup [Chronos]; Gaussian Process samples via KernelSynth (mixtures of RBF/periodic/linear kernels with swept hyperparameters).
-
-## Repository Layout
-
-- `model/training/` — main PatchFM model class
-
-  - `modules.py` - core modules (Residual Layers, MHA, SwiGLU, RoPE, Transformer Encoder, ...)
-  - `revin.py` — causal RevIN
-  - `loss.py` — multi-quantile (pinball) loss
-  - `trainer.py` — PyTorch Lightning trainer class
-
-- `model/inference/` — main PatchFM model class for inference 
-  - `modules.py` — core modules with caching support
-  - `forecaster.py` — Forecasting model and rollout logic
-
-- `dataset/` — data loading and preprocessing
-  - `artificial.py` — synthetic dataset : artificial signals + TSMixup + KernelSynth
-  - `utsd.py` — Unified Time Series Dataset (UTSD) loading and preprocessing
-  - `gift.py` — GIFT-Eval pretraining dataset loading and preprocessing
-  - `get_data.py` — utility to fetch and preprocess datasets
-  - `generate_data.py` — utility to generate and save the KernelSynth dataset (long to generate)
-
-- `configs/` — model and training configurations
-- `notebooks/inference` — how to load a trained model and generate forecasts
-- `training.py` — training script using PyTorch Lightning
-
-## Acknowledgements
-We thank the authors of the following repositories for inspiration and code snippets:
-- [TiRex](https://github.com/NX-AI/tirex)