patchfm 3.1.1__tar.gz → 3.3.0__tar.gz

{patchfm-3.1.1/src/patchfm.egg-info → patchfm-3.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: patchfm
-Version: 3.1.1
+Version: 3.3.0
 Summary: a Foundation Model for Univariate Time Series Forecasting
 Author-email: Samy-Melwan Vilhes <samy-melwan.vilhes@insa-rouen.fr>
 License: MIT License
@@ -40,32 +40,35 @@ Requires-Dist: numpy>=1.26.0
 Requires-Dist: safetensors==0.5.3
 Dynamic: license-file
 
-# A tutorial on how to build a Foundation Model for Univariate Time Series Forecasting
+# A Foundation Model for Univariate Time Series Forecasting
 
 A concise, reproducible recipe for training a transformer-based, patch-to-patch forecasting model for univariate time series. The approach mirrors Large Language Model (LLM) practices (next-token → next-patch) while remaining lightweight compared to a classic LLM and practical.
 
-Our model is deployed on the [TS-Arena benchmark](https://ts-arena.live) and achieves competitive performance against state-of-the-art methods under the name `LITIS/PatchFM-Large`.
+The model card is available on `assets/modelcard.pdf`.
 
-## Resuls on the FEV benchmark:
+Our model (with leakage) is deployed on the [TS-Arena benchmark](https://ts-arena.live) and achieves competitive performance against state-of-the-art methods under the name `LITIS/PatchFM-Large`.
 
-**Our model is competitive with the state-of-the-art notably compared to [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)**
+## Results on the FEV benchmark:
 
-| Model Name           | Win Rate | Skill Score |
-|----------------------|----------|-------------|
-| (🥇) Chronos-2           | 0.9008   | 0.4728      |
-| (🥈) TiRex               | 0.8092   | 0.4268      |
-| (🥉) TimesFM-2.5         | 0.8085   | 0.4668      |
-| (4) Toto-1.0            | 0.7069   | 0.4110      |
-| (5) **PatchFM (us)**             | 0.6677   | 0.3983      |
-| (6) Moirai-2.0          | 0.6546   | 0.4026      |
-| (7) Chronos-Bolt        | 0.6277   | 0.3889      |
-| (8) Sundial-Base        | 0.4446   | 0.3387      |
-| (9) CatBoost (Recursive)| 0.3362   | 0.2301      |
-| (10) LightGBM (Recursive)| 0.3123   | 0.2168      |
-| (11) AutoTheta           | 0.2938   | 0.0546      |
-| (12) Seasonal Naive      | 0.2058   | 0.0000      |
-| (13) Naive               | 0.1404   | -0.4540     |
-| (14) Drift               | 0.0915   | -0.4578     |
+**Our model (without leakage) remains competitive, though it does not match the top-performing state-of-the-art models.**
+
+<div style="overflow-x: auto;">
+
+| Metric | Chronos-2 | TiRex-2 | Toto-2.0-2.5B | Toto-2.0-1B | Toto-2.0-313m | TimesFM-2.5 | TiRex | Toto-2.0-22m | TS-ICL | **PatchFM** | FlowState | TabPFN-TS-3 | citras-fm | Toto-1.0 | Toto-2.0-4m | Moirai-2.0 | Chronos-Bolt | TFT | Sundial-Base | PatchTST | CatBoost | LightGBM | AutoTheta | Seasonal Naive | Naive | Drift |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| Win Rate | 0.81 | 0.78 | 0.77 | 0.77 | 0.75 | 0.71 | 0.70 | 0.67 | 0.61 | **0.61** | 0.60 | 0.60 | 0.58 | 0.57 | 0.55 | 0.53 | 0.49 | 0.37 | 0.32 | 0.31 | 0.22 | 0.21 | 0.20 | 0.13 | 0.09 | 0.07 |
+| Skill Score | 0.47 | 0.45 | 0.44 | 0.44 | 0.44 | 0.47 | 0.43 | 0.43 | 0.43 | **0.41** | 0.42 | 0.43 | 0.41 | 0.41 | 0.41 | 0.40 | 0.39 | 0.32 | 0.34 | 0.30 | 0.23 | 0.21 | 0.05 | 0.00 | -0.45 | -0.46 |
+
+</div>
+
+
+## Acknowledgements
+This work builds on ideas, model designs, and code from the following projects:
+- [TiRex](https://github.com/NX-AI/tirex)
+- [Toto](https://github.com/datadog/toto)
+- [TimesFM](https://github.com/google-research/timesfm)
+- [Chronos](https://github.com/amazon-science/chronos-forecasting)
+We are grateful to their authors.
 
 
 ## Highlights
@@ -74,14 +77,24 @@ Our model is deployed on the [TS-Arena benchmark](https://ts-arena.live) and ach
 - Causal masking self-attention with RoPE (relative positions)
 - Causal RevIN with $\sinh^{-1}$ transform for normalization 
 - SwiGLU feed-forward networks
+- QK-Norm with per-dimension adaptive scaling on queries, following [TimesFM-3](https://research.google/blog/timesfm-3-a-zero-shot-foundation-model-for-multivariate-forecasting/)
 - Autoregressive multi-quantile decoding [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)
 - KV-cache for efficient long-horizon inference
 - flips equivariance during inference (optional) [Reverso](https://arxiv.org/pdf/2602.17634v1)
+- [Muon](https://kellerjordan.github.io/posts/muon/) optimizer for 2D weights and AdamW for the rest, as in Toto 2.0 ([implementation of Muon used](https://arxiv.org/abs/2502.16982))
 
 ## Quick Start
 
 ### from source code
 
+> **⚠️ Important:** Two pretrained versions of PatchFM are available:
+
+>
+
+> - **`PatchFM-Leakage`**: trained on the GIFT-Eval datasets. This version may provide higher benchmark performance but includes training data that overlaps with the commonly used evaluation datasets.
+
+> - **`PatchFM`** (**recommended for fair evaluation**): trained without any commonly used benchmark (except BOOM), preventing potential data leakage and ensuring a more reliable assessment of generalization performance.
+
 1. Clone the repository and install dependencies
 ```bash
 git clone https://github.com/vilhess/PatchFM
@@ -97,7 +110,7 @@ from configs import PatchFMConfig
 from model import Forecaster
 
 # --- Instantiate model ---
-config = PatchFMConfig(load_from_hub=True) 
+config = PatchFMConfig(load_from_hub=True, full_leakage=False)  # set full_leakage=True to load the version trained with data leakage
 model = Forecaster(config)
 
 # --- Inference ---
@@ -119,6 +132,7 @@ import torch
 
 from patchfm import Forecaster, PatchFMConfig
 
+config = PatchFMConfig(full_leakage=False)
 # same as above
 pred_median, pred_quantiles = model(seq, forecast_horizon=forecast_horizon, quantiles=[0.1, 0.5, 0.9], flip_equivariance=True)  #  (batch, time), (batch, time, quantiles)
 ```
@@ -152,6 +166,7 @@ Aggregate over positions, patch elements, and quantiles.
 - Input MLP: $\mathbb{R}^{P_{len}} \to \mathbb{R}^{dim}$ residual 2-layer MLP (ReLU)
 - Multi-Head Attention: causal mask, RoPE; queries/keys/values per head
 - FFN: SwiGLU (SiLU-gated), pre-norm + residual
+- QK-Norm with per-dimension adaptive scaling on queries, following [TimesFM-3](https://research.google/blog/timesfm-3-a-zero-shot-foundation-model-for-multivariate-forecasting/)
 - Output heads: |Q| linear maps $\mathbb{R}^{dim} \to \mathbb{R}^{P_{len}}$ (one per quantile)
 
 ### Model Details
@@ -234,10 +249,12 @@ With this strategy, the median prediction (0.5 quantile) does **not necessarily*
 This discrepancy arises because the *quantile collapse* step aggregates predictions across all possible quantile paths. As a result, the median is computed from the combined multi-path distribution rather than from a single deterministic trajectory, which can lead to different estimates compared to the single-path (median-only) autoregressive approach.
 
 ## Datasets
-- GIFT-Eval pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
+- GIFT-pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
 domains and various frequencies. After preprocessing, this yields approximately 600K univariate series. 
 - Chronos synthetic datasets [Chronos]: two large synthetic datasets generated with Chronos, one with TSMixup and one with KernelSynth. Each contains approximately 10 million univariate series and 1 million respectively, each signal of length 1024.
 - Artificial: ~1M synthetic series (sinusoidal, linear, polynomial, logarithmic) plus mixtures via TSMixup [Chronos]; Gaussian Process samples via KernelSynth (mixtures of RBF/periodic/linear kernels with swept hyperparameters).
+- BOOM: Approximately 5 million time series of length 1024 sampled from the BOOM dataset [BOOM]. It is important to note that the BOOM dataset, which is also used as an evaluation benchmark, is included in the training data of both PatchFM variants (with and without benchmark leakage) to increase data diversity. Consequently, neither version can be fairly evaluated on the BOOM benchmark. Upon request, we can provide an alternative checkpoint trained without any BOOM data, enabling a fair evaluation on this benchmark.
+- TSMixup [Chronos]: mixing synthetic datasets and GIFT-pretraining datasets to further increase diversity.
 
 ## Repository Layout
 
@@ -254,14 +271,12 @@ domains and various frequencies. After preprocessing, this yields approximately
 
 - `dataset/` — data loading and preprocessing
   - `artificial.py` — synthetic dataset : artificial signals + TSMixup + KernelSynth
+  - `synthetic.py` — other synthetic signals (Claude for the generation)
   - `gift.py` — GIFT-Eval pretraining dataset loading and preprocessing
   - `get_data.py` — utility to fetch and preprocess datasets
   - `chronosdata.py` — loading of the synthetic datasets generated with Chronos (TSMixup and KernelSynth) with download functions integrated
+  - `boom.py` — loading of the BOOM dataset with download functions integrated
 
 - `configs/` — model and training configurations
 - `notebooks/inference` — how to load a trained model and generate forecasts
 - `training.py` — training script using PyTorch Lightning
-
-## Acknowledgements
-We thank the authors of the following repositories for inspiration and code snippets:
-- [TiRex](https://github.com/NX-AI/tirex)

{patchfm-3.1.1 → patchfm-3.3.0}/README.md

@@ -1,29 +1,32 @@
-# A tutorial on how to build a Foundation Model for Univariate Time Series Forecasting
+# A Foundation Model for Univariate Time Series Forecasting
 
 A concise, reproducible recipe for training a transformer-based, patch-to-patch forecasting model for univariate time series. The approach mirrors Large Language Model (LLM) practices (next-token → next-patch) while remaining lightweight compared to a classic LLM and practical.
 
-Our model is deployed on the [TS-Arena benchmark](https://ts-arena.live) and achieves competitive performance against state-of-the-art methods under the name `LITIS/PatchFM-Large`.
+The model card is available on `assets/modelcard.pdf`.
 
-## Resuls on the FEV benchmark:
+Our model (with leakage) is deployed on the [TS-Arena benchmark](https://ts-arena.live) and achieves competitive performance against state-of-the-art methods under the name `LITIS/PatchFM-Large`.
 
-**Our model is competitive with the state-of-the-art notably compared to [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)**
+## Results on the FEV benchmark:
 
-| Model Name           | Win Rate | Skill Score |
-|----------------------|----------|-------------|
-| (🥇) Chronos-2           | 0.9008   | 0.4728      |
-| (🥈) TiRex               | 0.8092   | 0.4268      |
-| (🥉) TimesFM-2.5         | 0.8085   | 0.4668      |
-| (4) Toto-1.0            | 0.7069   | 0.4110      |
-| (5) **PatchFM (us)**             | 0.6677   | 0.3983      |
-| (6) Moirai-2.0          | 0.6546   | 0.4026      |
-| (7) Chronos-Bolt        | 0.6277   | 0.3889      |
-| (8) Sundial-Base        | 0.4446   | 0.3387      |
-| (9) CatBoost (Recursive)| 0.3362   | 0.2301      |
-| (10) LightGBM (Recursive)| 0.3123   | 0.2168      |
-| (11) AutoTheta           | 0.2938   | 0.0546      |
-| (12) Seasonal Naive      | 0.2058   | 0.0000      |
-| (13) Naive               | 0.1404   | -0.4540     |
-| (14) Drift               | 0.0915   | -0.4578     |
+**Our model (without leakage) remains competitive, though it does not match the top-performing state-of-the-art models.**
+
+<div style="overflow-x: auto;">
+
+| Metric | Chronos-2 | TiRex-2 | Toto-2.0-2.5B | Toto-2.0-1B | Toto-2.0-313m | TimesFM-2.5 | TiRex | Toto-2.0-22m | TS-ICL | **PatchFM** | FlowState | TabPFN-TS-3 | citras-fm | Toto-1.0 | Toto-2.0-4m | Moirai-2.0 | Chronos-Bolt | TFT | Sundial-Base | PatchTST | CatBoost | LightGBM | AutoTheta | Seasonal Naive | Naive | Drift |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| Win Rate | 0.81 | 0.78 | 0.77 | 0.77 | 0.75 | 0.71 | 0.70 | 0.67 | 0.61 | **0.61** | 0.60 | 0.60 | 0.58 | 0.57 | 0.55 | 0.53 | 0.49 | 0.37 | 0.32 | 0.31 | 0.22 | 0.21 | 0.20 | 0.13 | 0.09 | 0.07 |
+| Skill Score | 0.47 | 0.45 | 0.44 | 0.44 | 0.44 | 0.47 | 0.43 | 0.43 | 0.43 | **0.41** | 0.42 | 0.43 | 0.41 | 0.41 | 0.41 | 0.40 | 0.39 | 0.32 | 0.34 | 0.30 | 0.23 | 0.21 | 0.05 | 0.00 | -0.45 | -0.46 |
+
+</div>
+
+
+## Acknowledgements
+This work builds on ideas, model designs, and code from the following projects:
+- [TiRex](https://github.com/NX-AI/tirex)
+- [Toto](https://github.com/datadog/toto)
+- [TimesFM](https://github.com/google-research/timesfm)
+- [Chronos](https://github.com/amazon-science/chronos-forecasting)
+We are grateful to their authors.
 
 
 ## Highlights
@@ -32,14 +35,24 @@ Our model is deployed on the [TS-Arena benchmark](https://ts-arena.live) and ach
 - Causal masking self-attention with RoPE (relative positions)
 - Causal RevIN with $\sinh^{-1}$ transform for normalization 
 - SwiGLU feed-forward networks
+- QK-Norm with per-dimension adaptive scaling on queries, following [TimesFM-3](https://research.google/blog/timesfm-3-a-zero-shot-foundation-model-for-multivariate-forecasting/)
 - Autoregressive multi-quantile decoding [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)
 - KV-cache for efficient long-horizon inference
 - flips equivariance during inference (optional) [Reverso](https://arxiv.org/pdf/2602.17634v1)
+- [Muon](https://kellerjordan.github.io/posts/muon/) optimizer for 2D weights and AdamW for the rest, as in Toto 2.0 ([implementation of Muon used](https://arxiv.org/abs/2502.16982))
 
 ## Quick Start
 
 ### from source code
 
+> **⚠️ Important:** Two pretrained versions of PatchFM are available:
+
+>
+
+> - **`PatchFM-Leakage`**: trained on the GIFT-Eval datasets. This version may provide higher benchmark performance but includes training data that overlaps with the commonly used evaluation datasets.
+
+> - **`PatchFM`** (**recommended for fair evaluation**): trained without any commonly used benchmark (except BOOM), preventing potential data leakage and ensuring a more reliable assessment of generalization performance.
+
 1. Clone the repository and install dependencies
 ```bash
 git clone https://github.com/vilhess/PatchFM
@@ -55,7 +68,7 @@ from configs import PatchFMConfig
 from model import Forecaster
 
 # --- Instantiate model ---
-config = PatchFMConfig(load_from_hub=True) 
+config = PatchFMConfig(load_from_hub=True, full_leakage=False)  # set full_leakage=True to load the version trained with data leakage
 model = Forecaster(config)
 
 # --- Inference ---
@@ -77,6 +90,7 @@ import torch
 
 from patchfm import Forecaster, PatchFMConfig
 
+config = PatchFMConfig(full_leakage=False)
 # same as above
 pred_median, pred_quantiles = model(seq, forecast_horizon=forecast_horizon, quantiles=[0.1, 0.5, 0.9], flip_equivariance=True)  #  (batch, time), (batch, time, quantiles)
 ```
@@ -110,6 +124,7 @@ Aggregate over positions, patch elements, and quantiles.
 - Input MLP: $\mathbb{R}^{P_{len}} \to \mathbb{R}^{dim}$ residual 2-layer MLP (ReLU)
 - Multi-Head Attention: causal mask, RoPE; queries/keys/values per head
 - FFN: SwiGLU (SiLU-gated), pre-norm + residual
+- QK-Norm with per-dimension adaptive scaling on queries, following [TimesFM-3](https://research.google/blog/timesfm-3-a-zero-shot-foundation-model-for-multivariate-forecasting/)
 - Output heads: |Q| linear maps $\mathbb{R}^{dim} \to \mathbb{R}^{P_{len}}$ (one per quantile)
 
 ### Model Details
@@ -192,10 +207,12 @@ With this strategy, the median prediction (0.5 quantile) does **not necessarily*
 This discrepancy arises because the *quantile collapse* step aggregates predictions across all possible quantile paths. As a result, the median is computed from the combined multi-path distribution rather than from a single deterministic trajectory, which can lead to different estimates compared to the single-path (median-only) autoregressive approach.
 
 ## Datasets
-- GIFT-Eval pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
+- GIFT-pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
 domains and various frequencies. After preprocessing, this yields approximately 600K univariate series. 
 - Chronos synthetic datasets [Chronos]: two large synthetic datasets generated with Chronos, one with TSMixup and one with KernelSynth. Each contains approximately 10 million univariate series and 1 million respectively, each signal of length 1024.
 - Artificial: ~1M synthetic series (sinusoidal, linear, polynomial, logarithmic) plus mixtures via TSMixup [Chronos]; Gaussian Process samples via KernelSynth (mixtures of RBF/periodic/linear kernels with swept hyperparameters).
+- BOOM: Approximately 5 million time series of length 1024 sampled from the BOOM dataset [BOOM]. It is important to note that the BOOM dataset, which is also used as an evaluation benchmark, is included in the training data of both PatchFM variants (with and without benchmark leakage) to increase data diversity. Consequently, neither version can be fairly evaluated on the BOOM benchmark. Upon request, we can provide an alternative checkpoint trained without any BOOM data, enabling a fair evaluation on this benchmark.
+- TSMixup [Chronos]: mixing synthetic datasets and GIFT-pretraining datasets to further increase diversity.
 
 ## Repository Layout
 
@@ -212,14 +229,12 @@ domains and various frequencies. After preprocessing, this yields approximately
 
 - `dataset/` — data loading and preprocessing
   - `artificial.py` — synthetic dataset : artificial signals + TSMixup + KernelSynth
+  - `synthetic.py` — other synthetic signals (Claude for the generation)
   - `gift.py` — GIFT-Eval pretraining dataset loading and preprocessing
   - `get_data.py` — utility to fetch and preprocess datasets
   - `chronosdata.py` — loading of the synthetic datasets generated with Chronos (TSMixup and KernelSynth) with download functions integrated
+  - `boom.py` — loading of the BOOM dataset with download functions integrated
 
 - `configs/` — model and training configurations
 - `notebooks/inference` — how to load a trained model and generate forecasts
-- `training.py` — training script using PyTorch Lightning
-
-## Acknowledgements
-We thank the authors of the following repositories for inspiration and code snippets:
-- [TiRex](https://github.com/NX-AI/tirex)
+- `training.py` — training script using PyTorch Lightning

{patchfm-3.1.1 → patchfm-3.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "patchfm"
-version = "3.1.1"
+version = "3.3.0"
 authors = [
   { name="Samy-Melwan Vilhes", email="samy-melwan.vilhes@insa-rouen.fr" },
 ]

{patchfm-3.1.1 → patchfm-3.3.0}/src/patchfm/configs/model_config.py

@@ -5,13 +5,13 @@ from dataclasses import asdict, dataclass, field
 class PatchFMConfig:
     max_seq_len: int = 1024
     patch_len: int = 32
-    d_model: int = 2048
-    n_heads: int = 32
-    n_layers_encoder: int = 6
+    d_model: int = 1024
+    n_heads: int = 16
+    n_layers_encoder: int = 20
     quantiles: list[float] = field(default_factory=lambda: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])
     
-    compile: bool = True
-    full_leakage: bool = True
+    compile: bool = False
+    full_leakage: bool = False
 
     def __getitem__(self, key):
         return getattr(self, key)

{patchfm-3.1.1 → patchfm-3.3.0}/src/patchfm/inference/forecaster.py

@@ -4,7 +4,8 @@ import torch.nn as nn
 from einops import rearrange
 
 from patchfm.inference.modules import (CausalRevIN, PatchFM, ResidualBlock,
-                                       SeqTypeConverter, TransformerEncoder)
+                                       SeqTypeConverter, TransformerEncoder, 
+                                       quantile_widths, monotone_quantiles)
 
 
 # --- Forecaster Model ---
@@ -22,6 +23,9 @@ class Forecaster(nn.Module):
         self.n_quantiles = len(self.quantiles)
         self.max_patches = self.max_seq_len // self.patch_len
 
+        widths, self.median_idx = quantile_widths(self.quantiles)
+        self.register_buffer("quantile_widths", widths, persistent=False)
+
         if config["full_leakage"]:
             print("Loading full leakage model from HuggingFace Hub...")
             model_name = "vilhess/PatchFM-Leakage"
@@ -39,14 +43,16 @@ class Forecaster(nn.Module):
         self.converter = SeqTypeConverter()
 
         if config["compile"]:
-            self = torch.compile(self)
+            self.proj_embedding = torch.compile(self.proj_embedding)
+            self.transformer_encoder = torch.compile(self.transformer_encoder)
+            self.proj_output = torch.compile(self.proj_output)
 
     def _init_components(self):
         """Initialize modules from scratch."""
         self.revin = CausalRevIN()
         self.proj_embedding = ResidualBlock(
             in_dim=self.patch_len, 
-            hid_dim=4 * self.d_model, 
+            hid_dim=self.d_model, 
             out_dim=self.d_model
         )
         self.transformer_encoder = TransformerEncoder(
@@ -101,6 +107,19 @@ class Forecaster(nn.Module):
         x = x[:, -1:, :]  # Keep only the last patch for autoregressive forecasting
 
         forecasting = self.proj_output(x)
+        forecasting = rearrange(
+            forecasting,
+            "b 1 (pl q) -> b 1 pl q",
+            pl=self.patch_len,
+            q=self.n_quantiles,
+        )
+        forecasting = monotone_quantiles(
+            forecasting, self.quantile_widths, self.median_idx, width_factor=2.0
+        )
+        forecasting = rearrange(
+            forecasting,
+            "b 1 pl q -> b 1 (pl q)",
+        )
         forecasting = self.revin(forecasting, mode="denorm")
 
         # Reshape to (bs, patch_num, patch_len, n_quantiles)
@@ -120,6 +139,19 @@ class Forecaster(nn.Module):
             x = self.transformer_encoder(x)
             x = x[:, -1:, :]  # Keep only the last patch for autoregressive forecasting
             forecasting = self.proj_output(x)
+            forecasting = rearrange(
+                forecasting,
+                "b 1 (pl q) -> b 1 pl q",
+                pl=self.patch_len,
+                q=self.n_quantiles,
+            )
+            forecasting = monotone_quantiles(
+                forecasting, self.quantile_widths, self.median_idx, width_factor=2.0
+            )
+            forecasting = rearrange(
+                forecasting,
+                "b 1 pl q -> b 1 (pl q)",
+            )
             forecasting = self.revin(forecasting, mode="denorm")
 
             # Reshape to (bs*n_quantiles, patch_num, patch_len, n_quantiles)

{patchfm-3.1.1 → patchfm-3.3.0}/src/patchfm/inference/modules.py

@@ -3,6 +3,7 @@
 import numpy as np
 import torch
 import torch.nn as nn
+import math
 from einops import rearrange
 from huggingface_hub import PyTorchModelHubMixin
 from rotary_embedding_torch import RotaryEmbedding
@@ -86,9 +87,8 @@ def nanstd(o, dim, keepdim=False):
     return std
 
 class CausalRevIN(nn.Module):
-    def __init__(self, eps=1e-5):
+    def __init__(self):
         super().__init__()
-        self.eps = eps
         self.cached_mean = None
         self.cached_std = None
 
@@ -158,7 +158,7 @@ class CausalRevIN(nn.Module):
         self.cached_cumsum_x2 = cumsum_x2[:, -1:, :]
 
         var = (cumsum_x2 - 2 * mean * cumsum_x + counts * mean**2) / counts
-        std = torch.sqrt(var + 1e-5)
+        std = torch.sqrt(var.clamp_min(0)).clamp_min(1e-3)
 
         return mean, std
     
@@ -184,14 +184,30 @@ class ResidualBlock(nn.Module):
         out = out+res
         return out
 
+class PerDimScale(nn.Module):
+  def __init__(self, num_dims: int):
+    super().__init__()
+    self.num_dims = num_dims
+    self.per_dim_scale = nn.Parameter(torch.zeros(num_dims))
+    self._RECIPROCAL_OF_SOFTPLUS_0 = 1.442695041
+
+  def forward(self, x: torch.Tensor) -> torch.Tensor:
+    return (
+      x
+      * self._RECIPROCAL_OF_SOFTPLUS_0
+      / math.sqrt(self.num_dims)
+      * torch.nn.functional.softplus(self.per_dim_scale)
+    )
+
 class MultiHeadAttention(nn.Module):
     def __init__(self, d_model, n_heads, last=False):
         super().__init__()
         assert d_model%n_heads==0, f"d_model ({d_model}) must be divisible by n_heads ({n_heads})"
 
-        self.WQ = nn.Linear(d_model, d_model)
-        self.WK = nn.Linear(d_model, d_model)
-        self.WV = nn.Linear(d_model, d_model)
+        self.WQ = nn.Linear(d_model, d_model, bias=False)
+        self.WK = nn.Linear(d_model, d_model, bias=False)
+        self.WV = nn.Linear(d_model, d_model, bias=False)
+        self.gate = nn.Linear(d_model, d_model, bias=False)
 
         self.out_proj = nn.Linear(d_model, d_model)
 
@@ -200,6 +216,10 @@ class MultiHeadAttention(nn.Module):
 
         self.rope = RotaryEmbedding(dim=self.head_dim//2)
 
+        self.q_norm = nn.RMSNorm(self.head_dim)
+        self.k_norm = nn.RMSNorm(self.head_dim)
+        self.per_dim_scale = PerDimScale(self.head_dim)
+
         self.k_cache = None
         self.v_cache = None
 
@@ -218,6 +238,8 @@ class MultiHeadAttention(nn.Module):
             is_causal = False
             offset += (context - 1)
 
+        gated = self.gate(q)
+
         q = self.WQ(q).reshape(bs, -1, self.n_heads, self.head_dim).transpose(1, 2)
         k = self.WK(k).reshape(bs, -1, self.n_heads, self.head_dim).transpose(1, 2)
         v = self.WV(v).reshape(bs, -1, self.n_heads, self.head_dim).transpose(1, 2)
@@ -237,9 +259,14 @@ class MultiHeadAttention(nn.Module):
         q = self.rope.rotate_queries_or_keys(q, offset=offset)
         k = self.rope.rotate_queries_or_keys(k)
 
-        values = nn.functional.scaled_dot_product_attention(q, k, v, is_causal=is_causal)
+        q = self.q_norm(q)
+        k = self.k_norm(k)
+        q = self.per_dim_scale(q)
+
+        values = nn.functional.scaled_dot_product_attention(q, k, v, is_causal=is_causal, scale=1.0)
 
         values = values.transpose(1, 2).reshape(bs, -1, dim)
+        values = gated.sigmoid() * values
         values = self.out_proj(values)
         return values
     
@@ -269,9 +296,9 @@ class FeedForward(nn.Module):
 class TransformerEncoderLayer(nn.Module):
     def __init__(self, d_model, n_heads, last=False):
         super().__init__()
-        self.ln1 = nn.LayerNorm(d_model)
+        self.ln1 = nn.RMSNorm(d_model)
         self.attn = MultiHeadAttention(d_model=d_model, n_heads=n_heads, last=last)
-        self.ln2 = nn.LayerNorm(d_model)
+        self.ln2 = nn.RMSNorm(d_model)
         self.ff = FeedForward(d_model=d_model)
     
     def forward(self, x):
@@ -290,12 +317,34 @@ class TransformerEncoder(nn.Module):
             ]
         )
         self.layers.append(TransformerEncoderLayer(d_model=d_model, n_heads=n_heads, last=True))
-        self.norm = nn.LayerNorm(d_model)
+        self.norm = nn.RMSNorm(d_model)
 
     def forward(self, x):
         for layer in self.layers:
             x = layer(x)
         return self.norm(x)
+
+def quantile_widths(quantiles):
+    m = quantiles.index(0.5)
+    widths = [0.0] * len(quantiles)
+    for i in range(m + 1, len(quantiles)):
+        widths[i] = quantiles[i] - quantiles[i - 1]
+    for i in range(m):
+        widths[i] = quantiles[i + 1] - quantiles[i]
+    return torch.tensor(widths), m
+
+
+def monotone_quantiles(raw, widths, m, width_factor=2.0):
+    inc = width_factor * widths.to(raw.dtype) * torch.nn.functional.softplus(raw)
+    median = raw[..., m : m + 1]
+    up = torch.cumsum(inc[..., m + 1 :], dim=-1)
+    down = torch.flip(
+        torch.cumsum(torch.flip(inc[..., :m], dims=[-1]), dim=-1), dims=[-1]
+    )
+    big = torch.finfo(raw.dtype).max
+    return torch.cat([median - down, median, median + up], dim=-1).clamp(
+        min=-big, max=big
+    )
     
 class PatchFM(nn.Module, PyTorchModelHubMixin): 
     def __init__(self, config):
@@ -313,7 +362,7 @@ class PatchFM(nn.Module, PyTorchModelHubMixin):
         self.revin = CausalRevIN()
         self.proj_embedding = ResidualBlock(
             in_dim=self.patch_len, 
-            hid_dim=4 * self.d_model, 
+            hid_dim=self.d_model, 
             out_dim=self.d_model
         )
         self.transformer_encoder = TransformerEncoder(

{patchfm-3.1.1 → patchfm-3.3.0/src/patchfm.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: patchfm
-Version: 3.1.1
+Version: 3.3.0
 Summary: a Foundation Model for Univariate Time Series Forecasting
 Author-email: Samy-Melwan Vilhes <samy-melwan.vilhes@insa-rouen.fr>
 License: MIT License
@@ -40,32 +40,35 @@ Requires-Dist: numpy>=1.26.0
 Requires-Dist: safetensors==0.5.3
 Dynamic: license-file
 
-# A tutorial on how to build a Foundation Model for Univariate Time Series Forecasting
+# A Foundation Model for Univariate Time Series Forecasting
 
 A concise, reproducible recipe for training a transformer-based, patch-to-patch forecasting model for univariate time series. The approach mirrors Large Language Model (LLM) practices (next-token → next-patch) while remaining lightweight compared to a classic LLM and practical.
 
-Our model is deployed on the [TS-Arena benchmark](https://ts-arena.live) and achieves competitive performance against state-of-the-art methods under the name `LITIS/PatchFM-Large`.
+The model card is available on `assets/modelcard.pdf`.
 
-## Resuls on the FEV benchmark:
+Our model (with leakage) is deployed on the [TS-Arena benchmark](https://ts-arena.live) and achieves competitive performance against state-of-the-art methods under the name `LITIS/PatchFM-Large`.
 
-**Our model is competitive with the state-of-the-art notably compared to [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)**
+## Results on the FEV benchmark:
 
-| Model Name           | Win Rate | Skill Score |
-|----------------------|----------|-------------|
-| (🥇) Chronos-2           | 0.9008   | 0.4728      |
-| (🥈) TiRex               | 0.8092   | 0.4268      |
-| (🥉) TimesFM-2.5         | 0.8085   | 0.4668      |
-| (4) Toto-1.0            | 0.7069   | 0.4110      |
-| (5) **PatchFM (us)**             | 0.6677   | 0.3983      |
-| (6) Moirai-2.0          | 0.6546   | 0.4026      |
-| (7) Chronos-Bolt        | 0.6277   | 0.3889      |
-| (8) Sundial-Base        | 0.4446   | 0.3387      |
-| (9) CatBoost (Recursive)| 0.3362   | 0.2301      |
-| (10) LightGBM (Recursive)| 0.3123   | 0.2168      |
-| (11) AutoTheta           | 0.2938   | 0.0546      |
-| (12) Seasonal Naive      | 0.2058   | 0.0000      |
-| (13) Naive               | 0.1404   | -0.4540     |
-| (14) Drift               | 0.0915   | -0.4578     |
+**Our model (without leakage) remains competitive, though it does not match the top-performing state-of-the-art models.**
+
+<div style="overflow-x: auto;">
+
+| Metric | Chronos-2 | TiRex-2 | Toto-2.0-2.5B | Toto-2.0-1B | Toto-2.0-313m | TimesFM-2.5 | TiRex | Toto-2.0-22m | TS-ICL | **PatchFM** | FlowState | TabPFN-TS-3 | citras-fm | Toto-1.0 | Toto-2.0-4m | Moirai-2.0 | Chronos-Bolt | TFT | Sundial-Base | PatchTST | CatBoost | LightGBM | AutoTheta | Seasonal Naive | Naive | Drift |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| Win Rate | 0.81 | 0.78 | 0.77 | 0.77 | 0.75 | 0.71 | 0.70 | 0.67 | 0.61 | **0.61** | 0.60 | 0.60 | 0.58 | 0.57 | 0.55 | 0.53 | 0.49 | 0.37 | 0.32 | 0.31 | 0.22 | 0.21 | 0.20 | 0.13 | 0.09 | 0.07 |
+| Skill Score | 0.47 | 0.45 | 0.44 | 0.44 | 0.44 | 0.47 | 0.43 | 0.43 | 0.43 | **0.41** | 0.42 | 0.43 | 0.41 | 0.41 | 0.41 | 0.40 | 0.39 | 0.32 | 0.34 | 0.30 | 0.23 | 0.21 | 0.05 | 0.00 | -0.45 | -0.46 |
+
+</div>
+
+
+## Acknowledgements
+This work builds on ideas, model designs, and code from the following projects:
+- [TiRex](https://github.com/NX-AI/tirex)
+- [Toto](https://github.com/datadog/toto)
+- [TimesFM](https://github.com/google-research/timesfm)
+- [Chronos](https://github.com/amazon-science/chronos-forecasting)
+We are grateful to their authors.
 
 
 ## Highlights
@@ -74,14 +77,24 @@ Our model is deployed on the [TS-Arena benchmark](https://ts-arena.live) and ach
 - Causal masking self-attention with RoPE (relative positions)
 - Causal RevIN with $\sinh^{-1}$ transform for normalization 
 - SwiGLU feed-forward networks
+- QK-Norm with per-dimension adaptive scaling on queries, following [TimesFM-3](https://research.google/blog/timesfm-3-a-zero-shot-foundation-model-for-multivariate-forecasting/)
 - Autoregressive multi-quantile decoding [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)
 - KV-cache for efficient long-horizon inference
 - flips equivariance during inference (optional) [Reverso](https://arxiv.org/pdf/2602.17634v1)
+- [Muon](https://kellerjordan.github.io/posts/muon/) optimizer for 2D weights and AdamW for the rest, as in Toto 2.0 ([implementation of Muon used](https://arxiv.org/abs/2502.16982))
 
 ## Quick Start
 
 ### from source code
 
+> **⚠️ Important:** Two pretrained versions of PatchFM are available:
+
+>
+
+> - **`PatchFM-Leakage`**: trained on the GIFT-Eval datasets. This version may provide higher benchmark performance but includes training data that overlaps with the commonly used evaluation datasets.
+
+> - **`PatchFM`** (**recommended for fair evaluation**): trained without any commonly used benchmark (except BOOM), preventing potential data leakage and ensuring a more reliable assessment of generalization performance.
+
 1. Clone the repository and install dependencies
 ```bash
 git clone https://github.com/vilhess/PatchFM
@@ -97,7 +110,7 @@ from configs import PatchFMConfig
 from model import Forecaster
 
 # --- Instantiate model ---
-config = PatchFMConfig(load_from_hub=True) 
+config = PatchFMConfig(load_from_hub=True, full_leakage=False)  # set full_leakage=True to load the version trained with data leakage
 model = Forecaster(config)
 
 # --- Inference ---
@@ -119,6 +132,7 @@ import torch
 
 from patchfm import Forecaster, PatchFMConfig
 
+config = PatchFMConfig(full_leakage=False)
 # same as above
 pred_median, pred_quantiles = model(seq, forecast_horizon=forecast_horizon, quantiles=[0.1, 0.5, 0.9], flip_equivariance=True)  #  (batch, time), (batch, time, quantiles)
 ```
@@ -152,6 +166,7 @@ Aggregate over positions, patch elements, and quantiles.
 - Input MLP: $\mathbb{R}^{P_{len}} \to \mathbb{R}^{dim}$ residual 2-layer MLP (ReLU)
 - Multi-Head Attention: causal mask, RoPE; queries/keys/values per head
 - FFN: SwiGLU (SiLU-gated), pre-norm + residual
+- QK-Norm with per-dimension adaptive scaling on queries, following [TimesFM-3](https://research.google/blog/timesfm-3-a-zero-shot-foundation-model-for-multivariate-forecasting/)
 - Output heads: |Q| linear maps $\mathbb{R}^{dim} \to \mathbb{R}^{P_{len}}$ (one per quantile)
 
 ### Model Details
@@ -234,10 +249,12 @@ With this strategy, the median prediction (0.5 quantile) does **not necessarily*
 This discrepancy arises because the *quantile collapse* step aggregates predictions across all possible quantile paths. As a result, the median is computed from the combined multi-path distribution rather than from a single deterministic trajectory, which can lead to different estimates compared to the single-path (median-only) autoregressive approach.
 
 ## Datasets
-- GIFT-Eval pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
+- GIFT-pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
 domains and various frequencies. After preprocessing, this yields approximately 600K univariate series. 
 - Chronos synthetic datasets [Chronos]: two large synthetic datasets generated with Chronos, one with TSMixup and one with KernelSynth. Each contains approximately 10 million univariate series and 1 million respectively, each signal of length 1024.
 - Artificial: ~1M synthetic series (sinusoidal, linear, polynomial, logarithmic) plus mixtures via TSMixup [Chronos]; Gaussian Process samples via KernelSynth (mixtures of RBF/periodic/linear kernels with swept hyperparameters).
+- BOOM: Approximately 5 million time series of length 1024 sampled from the BOOM dataset [BOOM]. It is important to note that the BOOM dataset, which is also used as an evaluation benchmark, is included in the training data of both PatchFM variants (with and without benchmark leakage) to increase data diversity. Consequently, neither version can be fairly evaluated on the BOOM benchmark. Upon request, we can provide an alternative checkpoint trained without any BOOM data, enabling a fair evaluation on this benchmark.
+- TSMixup [Chronos]: mixing synthetic datasets and GIFT-pretraining datasets to further increase diversity.
 
 ## Repository Layout
 
@@ -254,14 +271,12 @@ domains and various frequencies. After preprocessing, this yields approximately
 
 - `dataset/` — data loading and preprocessing
   - `artificial.py` — synthetic dataset : artificial signals + TSMixup + KernelSynth
+  - `synthetic.py` — other synthetic signals (Claude for the generation)
   - `gift.py` — GIFT-Eval pretraining dataset loading and preprocessing
   - `get_data.py` — utility to fetch and preprocess datasets
   - `chronosdata.py` — loading of the synthetic datasets generated with Chronos (TSMixup and KernelSynth) with download functions integrated
+  - `boom.py` — loading of the BOOM dataset with download functions integrated
 
 - `configs/` — model and training configurations
 - `notebooks/inference` — how to load a trained model and generate forecasts
 - `training.py` — training script using PyTorch Lightning
-
-## Acknowledgements
-We thank the authors of the following repositories for inspiration and code snippets:
-- [TiRex](https://github.com/NX-AI/tirex)