patchfm 3.1.1__tar.gz → 3.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (14) hide show
  1. {patchfm-3.1.1/src/patchfm.egg-info → patchfm-3.2.0}/PKG-INFO +42 -27
  2. {patchfm-3.1.1 → patchfm-3.2.0}/README.md +42 -27
  3. {patchfm-3.1.1 → patchfm-3.2.0}/pyproject.toml +1 -1
  4. {patchfm-3.1.1 → patchfm-3.2.0}/src/patchfm/inference/forecaster.py +3 -1
  5. {patchfm-3.1.1 → patchfm-3.2.0}/src/patchfm/inference/modules.py +24 -0
  6. {patchfm-3.1.1 → patchfm-3.2.0/src/patchfm.egg-info}/PKG-INFO +42 -27
  7. {patchfm-3.1.1 → patchfm-3.2.0}/LICENSE +0 -0
  8. {patchfm-3.1.1 → patchfm-3.2.0}/setup.cfg +0 -0
  9. {patchfm-3.1.1 → patchfm-3.2.0}/src/patchfm/__init__.py +0 -0
  10. {patchfm-3.1.1 → patchfm-3.2.0}/src/patchfm/configs/model_config.py +0 -0
  11. {patchfm-3.1.1 → patchfm-3.2.0}/src/patchfm.egg-info/SOURCES.txt +0 -0
  12. {patchfm-3.1.1 → patchfm-3.2.0}/src/patchfm.egg-info/dependency_links.txt +0 -0
  13. {patchfm-3.1.1 → patchfm-3.2.0}/src/patchfm.egg-info/requires.txt +0 -0
  14. {patchfm-3.1.1 → patchfm-3.2.0}/src/patchfm.egg-info/top_level.txt +0 -0
{patchfm-3.1.1/src/patchfm.egg-info → patchfm-3.2.0}/PKG-INFO RENAMED
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: patchfm
3
- Version: 3.1.1
3
+ Version: 3.2.0
4
4
  Summary: a Foundation Model for Univariate Time Series Forecasting
5
5
  Author-email: Samy-Melwan Vilhes <samy-melwan.vilhes@insa-rouen.fr>
6
6
  License: MIT License
@@ -40,32 +40,35 @@ Requires-Dist: numpy>=1.26.0
40
40
  Requires-Dist: safetensors==0.5.3
41
41
  Dynamic: license-file
42
42
 
43
- # A tutorial on how to build a Foundation Model for Univariate Time Series Forecasting
43
+ # A Foundation Model for Univariate Time Series Forecasting
44
44
 
45
45
  A concise, reproducible recipe for training a transformer-based, patch-to-patch forecasting model for univariate time series. The approach mirrors Large Language Model (LLM) practices (next-token → next-patch) while remaining lightweight compared to a classic LLM and practical.
46
46
 
47
- Our model is deployed on the [TS-Arena benchmark](https://ts-arena.live) and achieves competitive performance against state-of-the-art methods under the name `LITIS/PatchFM-Large`.
47
+ The model card is available on `assets/modelcard.pdf`.
48
48
 
49
- ## Resuls on the FEV benchmark:
49
+ Our model (with leakage) is deployed on the [TS-Arena benchmark](https://ts-arena.live) and achieves competitive performance against state-of-the-art methods under the name `LITIS/PatchFM-Large`.
50
50
 
51
- **Our model is competitive with the state-of-the-art notably compared to [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)**
51
+ ## Results on the FEV benchmark:
52
52
 
53
- | Model Name | Win Rate | Skill Score |
54
- |----------------------|----------|-------------|
55
- | (🥇) Chronos-2 | 0.9008 | 0.4728 |
56
- | (🥈) TiRex | 0.8092 | 0.4268 |
57
- | (🥉) TimesFM-2.5 | 0.8085 | 0.4668 |
58
- | (4) Toto-1.0 | 0.7069 | 0.4110 |
59
- | (5) **PatchFM (us)** | 0.6677 | 0.3983 |
60
- | (6) Moirai-2.0 | 0.6546 | 0.4026 |
61
- | (7) Chronos-Bolt | 0.6277 | 0.3889 |
62
- | (8) Sundial-Base | 0.4446 | 0.3387 |
63
- | (9) CatBoost (Recursive)| 0.3362 | 0.2301 |
64
- | (10) LightGBM (Recursive)| 0.3123 | 0.2168 |
65
- | (11) AutoTheta | 0.2938 | 0.0546 |
66
- | (12) Seasonal Naive | 0.2058 | 0.0000 |
67
- | (13) Naive | 0.1404 | -0.4540 |
68
- | (14) Drift | 0.0915 | -0.4578 |
53
+ **Our model (without leakage) remains competitive, though it does not match the top-performing state-of-the-art models.**
54
+
55
+ <div style="overflow-x: auto;">
56
+
57
+ | Metric | Chronos-2 | TiRex-2 | Toto-2.0-2.5B | Toto-2.0-1B | Toto-2.0-313m | TimesFM-2.5 | TiRex | Toto-2.0-22m | TS-ICL | **PatchFM** | FlowState | TabPFN-TS-3 | citras-fm | Toto-1.0 | Toto-2.0-4m | Moirai-2.0 | Chronos-Bolt | TFT | Sundial-Base | PatchTST | CatBoost | LightGBM | AutoTheta | Seasonal Naive | Naive | Drift |
58
+ |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
59
+ | Win Rate | 0.81 | 0.78 | 0.77 | 0.77 | 0.75 | 0.71 | 0.70 | 0.67 | 0.61 | **0.61** | 0.60 | 0.60 | 0.58 | 0.57 | 0.55 | 0.53 | 0.49 | 0.37 | 0.32 | 0.31 | 0.22 | 0.21 | 0.20 | 0.13 | 0.09 | 0.07 |
60
+ | Skill Score | 0.47 | 0.45 | 0.44 | 0.44 | 0.44 | 0.47 | 0.43 | 0.43 | 0.43 | **0.41** | 0.42 | 0.43 | 0.41 | 0.41 | 0.41 | 0.40 | 0.39 | 0.32 | 0.34 | 0.30 | 0.23 | 0.21 | 0.05 | 0.00 | -0.45 | -0.46 |
61
+
62
+ </div>
63
+
64
+
65
+ ## Acknowledgements
66
+ This work builds on ideas, model designs, and code from the following projects:
67
+ - [TiRex](https://github.com/NX-AI/tirex)
68
+ - [Toto](https://github.com/datadog/toto)
69
+ - [TimesFM](https://github.com/google-research/timesfm)
70
+ - [Chronos](https://github.com/amazon-science/chronos-forecasting)
71
+ We are grateful to their authors.
69
72
 
70
73
 
71
74
  ## Highlights
@@ -74,14 +77,24 @@ Our model is deployed on the [TS-Arena benchmark](https://ts-arena.live) and ach
74
77
  - Causal masking self-attention with RoPE (relative positions)
75
78
  - Causal RevIN with $\sinh^{-1}$ transform for normalization
76
79
  - SwiGLU feed-forward networks
80
+ - QK-Norm with per-dimension adaptive scaling on queries, following [TimesFM-3](https://research.google/blog/timesfm-3-a-zero-shot-foundation-model-for-multivariate-forecasting/)
77
81
  - Autoregressive multi-quantile decoding [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)
78
82
  - KV-cache for efficient long-horizon inference
79
83
  - flips equivariance during inference (optional) [Reverso](https://arxiv.org/pdf/2602.17634v1)
84
+ - [Muon](https://kellerjordan.github.io/posts/muon/) optimizer for 2D weights and AdamW for the rest, as in Toto 2.0 ([implementation of Muon used](https://arxiv.org/abs/2502.16982))
80
85
 
81
86
  ## Quick Start
82
87
 
83
88
  ### from source code
84
89
 
90
+ > **⚠️ Important:** Two pretrained versions of PatchFM are available:
91
+
92
+ >
93
+
94
+ > - **`PatchFM-Leakage`**: trained on the GIFT-Eval datasets. This version may provide higher benchmark performance but includes training data that overlaps with the commonly used evaluation datasets.
95
+
96
+ > - **`PatchFM`** (**recommended for fair evaluation**): trained without any commonly used benchmark (except BOOM), preventing potential data leakage and ensuring a more reliable assessment of generalization performance.
97
+
85
98
  1. Clone the repository and install dependencies
86
99
  ```bash
87
100
  git clone https://github.com/vilhess/PatchFM
@@ -97,7 +110,7 @@ from configs import PatchFMConfig
97
110
  from model import Forecaster
98
111
 
99
112
  # --- Instantiate model ---
100
- config = PatchFMConfig(load_from_hub=True)
113
+ config = PatchFMConfig(load_from_hub=True, full_leakage=False) # set full_leakage=True to load the version trained with data leakage
101
114
  model = Forecaster(config)
102
115
 
103
116
  # --- Inference ---
@@ -119,6 +132,7 @@ import torch
119
132
 
120
133
  from patchfm import Forecaster, PatchFMConfig
121
134
 
135
+ config = PatchFMConfig(full_leakage=False)
122
136
  # same as above
123
137
  pred_median, pred_quantiles = model(seq, forecast_horizon=forecast_horizon, quantiles=[0.1, 0.5, 0.9], flip_equivariance=True) # (batch, time), (batch, time, quantiles)
124
138
  ```
@@ -152,6 +166,7 @@ Aggregate over positions, patch elements, and quantiles.
152
166
  - Input MLP: $\mathbb{R}^{P_{len}} \to \mathbb{R}^{dim}$ residual 2-layer MLP (ReLU)
153
167
  - Multi-Head Attention: causal mask, RoPE; queries/keys/values per head
154
168
  - FFN: SwiGLU (SiLU-gated), pre-norm + residual
169
+ - QK-Norm with per-dimension adaptive scaling on queries, following [TimesFM-3](https://research.google/blog/timesfm-3-a-zero-shot-foundation-model-for-multivariate-forecasting/)
155
170
  - Output heads: |Q| linear maps $\mathbb{R}^{dim} \to \mathbb{R}^{P_{len}}$ (one per quantile)
156
171
 
157
172
  ### Model Details
@@ -234,10 +249,12 @@ With this strategy, the median prediction (0.5 quantile) does **not necessarily*
234
249
  This discrepancy arises because the *quantile collapse* step aggregates predictions across all possible quantile paths. As a result, the median is computed from the combined multi-path distribution rather than from a single deterministic trajectory, which can lead to different estimates compared to the single-path (median-only) autoregressive approach.
235
250
 
236
251
  ## Datasets
237
- - GIFT-Eval pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
252
+ - GIFT-pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
238
253
  domains and various frequencies. After preprocessing, this yields approximately 600K univariate series.
239
254
  - Chronos synthetic datasets [Chronos]: two large synthetic datasets generated with Chronos, one with TSMixup and one with KernelSynth. Each contains approximately 10 million univariate series and 1 million respectively, each signal of length 1024.
240
255
  - Artificial: ~1M synthetic series (sinusoidal, linear, polynomial, logarithmic) plus mixtures via TSMixup [Chronos]; Gaussian Process samples via KernelSynth (mixtures of RBF/periodic/linear kernels with swept hyperparameters).
256
+ - BOOM: Approximately 5 million time series of length 1024 sampled from the BOOM dataset [BOOM]. It is important to note that the BOOM dataset, which is also used as an evaluation benchmark, is included in the training data of both PatchFM variants (with and without benchmark leakage) to increase data diversity. Consequently, neither version can be fairly evaluated on the BOOM benchmark. Upon request, we can provide an alternative checkpoint trained without any BOOM data, enabling a fair evaluation on this benchmark.
257
+ - TSMixup [Chronos]: mixing synthetic datasets and GIFT-pretraining datasets to further increase diversity.
241
258
 
242
259
  ## Repository Layout
243
260
 
@@ -254,14 +271,12 @@ domains and various frequencies. After preprocessing, this yields approximately
254
271
 
255
272
  - `dataset/` — data loading and preprocessing
256
273
  - `artificial.py` — synthetic dataset : artificial signals + TSMixup + KernelSynth
274
+ - `synthetic.py` — other synthetic signals (Claude for the generation)
257
275
  - `gift.py` — GIFT-Eval pretraining dataset loading and preprocessing
258
276
  - `get_data.py` — utility to fetch and preprocess datasets
259
277
  - `chronosdata.py` — loading of the synthetic datasets generated with Chronos (TSMixup and KernelSynth) with download functions integrated
278
+ - `boom.py` — loading of the BOOM dataset with download functions integrated
260
279
 
261
280
  - `configs/` — model and training configurations
262
281
  - `notebooks/inference` — how to load a trained model and generate forecasts
263
282
  - `training.py` — training script using PyTorch Lightning
264
-
265
- ## Acknowledgements
266
- We thank the authors of the following repositories for inspiration and code snippets:
267
- - [TiRex](https://github.com/NX-AI/tirex)
{patchfm-3.1.1 → patchfm-3.2.0}/README.md RENAMED
@@ -1,29 +1,32 @@
1
- # A tutorial on how to build a Foundation Model for Univariate Time Series Forecasting
1
+ # A Foundation Model for Univariate Time Series Forecasting
2
2
 
3
3
  A concise, reproducible recipe for training a transformer-based, patch-to-patch forecasting model for univariate time series. The approach mirrors Large Language Model (LLM) practices (next-token → next-patch) while remaining lightweight compared to a classic LLM and practical.
4
4
 
5
- Our model is deployed on the [TS-Arena benchmark](https://ts-arena.live) and achieves competitive performance against state-of-the-art methods under the name `LITIS/PatchFM-Large`.
5
+ The model card is available on `assets/modelcard.pdf`.
6
6
 
7
- ## Resuls on the FEV benchmark:
7
+ Our model (with leakage) is deployed on the [TS-Arena benchmark](https://ts-arena.live) and achieves competitive performance against state-of-the-art methods under the name `LITIS/PatchFM-Large`.
8
8
 
9
- **Our model is competitive with the state-of-the-art notably compared to [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)**
9
+ ## Results on the FEV benchmark:
10
10
 
11
- | Model Name | Win Rate | Skill Score |
12
- |----------------------|----------|-------------|
13
- | (🥇) Chronos-2 | 0.9008 | 0.4728 |
14
- | (🥈) TiRex | 0.8092 | 0.4268 |
15
- | (🥉) TimesFM-2.5 | 0.8085 | 0.4668 |
16
- | (4) Toto-1.0 | 0.7069 | 0.4110 |
17
- | (5) **PatchFM (us)** | 0.6677 | 0.3983 |
18
- | (6) Moirai-2.0 | 0.6546 | 0.4026 |
19
- | (7) Chronos-Bolt | 0.6277 | 0.3889 |
20
- | (8) Sundial-Base | 0.4446 | 0.3387 |
21
- | (9) CatBoost (Recursive)| 0.3362 | 0.2301 |
22
- | (10) LightGBM (Recursive)| 0.3123 | 0.2168 |
23
- | (11) AutoTheta | 0.2938 | 0.0546 |
24
- | (12) Seasonal Naive | 0.2058 | 0.0000 |
25
- | (13) Naive | 0.1404 | -0.4540 |
26
- | (14) Drift | 0.0915 | -0.4578 |
11
+ **Our model (without leakage) remains competitive, though it does not match the top-performing state-of-the-art models.**
12
+
13
+ <div style="overflow-x: auto;">
14
+
15
+ | Metric | Chronos-2 | TiRex-2 | Toto-2.0-2.5B | Toto-2.0-1B | Toto-2.0-313m | TimesFM-2.5 | TiRex | Toto-2.0-22m | TS-ICL | **PatchFM** | FlowState | TabPFN-TS-3 | citras-fm | Toto-1.0 | Toto-2.0-4m | Moirai-2.0 | Chronos-Bolt | TFT | Sundial-Base | PatchTST | CatBoost | LightGBM | AutoTheta | Seasonal Naive | Naive | Drift |
16
+ |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
17
+ | Win Rate | 0.81 | 0.78 | 0.77 | 0.77 | 0.75 | 0.71 | 0.70 | 0.67 | 0.61 | **0.61** | 0.60 | 0.60 | 0.58 | 0.57 | 0.55 | 0.53 | 0.49 | 0.37 | 0.32 | 0.31 | 0.22 | 0.21 | 0.20 | 0.13 | 0.09 | 0.07 |
18
+ | Skill Score | 0.47 | 0.45 | 0.44 | 0.44 | 0.44 | 0.47 | 0.43 | 0.43 | 0.43 | **0.41** | 0.42 | 0.43 | 0.41 | 0.41 | 0.41 | 0.40 | 0.39 | 0.32 | 0.34 | 0.30 | 0.23 | 0.21 | 0.05 | 0.00 | -0.45 | -0.46 |
19
+
20
+ </div>
21
+
22
+
23
+ ## Acknowledgements
24
+ This work builds on ideas, model designs, and code from the following projects:
25
+ - [TiRex](https://github.com/NX-AI/tirex)
26
+ - [Toto](https://github.com/datadog/toto)
27
+ - [TimesFM](https://github.com/google-research/timesfm)
28
+ - [Chronos](https://github.com/amazon-science/chronos-forecasting)
29
+ We are grateful to their authors.
27
30
 
28
31
 
29
32
  ## Highlights
@@ -32,14 +35,24 @@ Our model is deployed on the [TS-Arena benchmark](https://ts-arena.live) and ach
32
35
  - Causal masking self-attention with RoPE (relative positions)
33
36
  - Causal RevIN with $\sinh^{-1}$ transform for normalization
34
37
  - SwiGLU feed-forward networks
38
+ - QK-Norm with per-dimension adaptive scaling on queries, following [TimesFM-3](https://research.google/blog/timesfm-3-a-zero-shot-foundation-model-for-multivariate-forecasting/)
35
39
  - Autoregressive multi-quantile decoding [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)
36
40
  - KV-cache for efficient long-horizon inference
37
41
  - flips equivariance during inference (optional) [Reverso](https://arxiv.org/pdf/2602.17634v1)
42
+ - [Muon](https://kellerjordan.github.io/posts/muon/) optimizer for 2D weights and AdamW for the rest, as in Toto 2.0 ([implementation of Muon used](https://arxiv.org/abs/2502.16982))
38
43
 
39
44
  ## Quick Start
40
45
 
41
46
  ### from source code
42
47
 
48
+ > **⚠️ Important:** Two pretrained versions of PatchFM are available:
49
+
50
+ >
51
+
52
+ > - **`PatchFM-Leakage`**: trained on the GIFT-Eval datasets. This version may provide higher benchmark performance but includes training data that overlaps with the commonly used evaluation datasets.
53
+
54
+ > - **`PatchFM`** (**recommended for fair evaluation**): trained without any commonly used benchmark (except BOOM), preventing potential data leakage and ensuring a more reliable assessment of generalization performance.
55
+
43
56
  1. Clone the repository and install dependencies
44
57
  ```bash
45
58
  git clone https://github.com/vilhess/PatchFM
@@ -55,7 +68,7 @@ from configs import PatchFMConfig
55
68
  from model import Forecaster
56
69
 
57
70
  # --- Instantiate model ---
58
- config = PatchFMConfig(load_from_hub=True)
71
+ config = PatchFMConfig(load_from_hub=True, full_leakage=False) # set full_leakage=True to load the version trained with data leakage
59
72
  model = Forecaster(config)
60
73
 
61
74
  # --- Inference ---
@@ -77,6 +90,7 @@ import torch
77
90
 
78
91
  from patchfm import Forecaster, PatchFMConfig
79
92
 
93
+ config = PatchFMConfig(full_leakage=False)
80
94
  # same as above
81
95
  pred_median, pred_quantiles = model(seq, forecast_horizon=forecast_horizon, quantiles=[0.1, 0.5, 0.9], flip_equivariance=True) # (batch, time), (batch, time, quantiles)
82
96
  ```
@@ -110,6 +124,7 @@ Aggregate over positions, patch elements, and quantiles.
110
124
  - Input MLP: $\mathbb{R}^{P_{len}} \to \mathbb{R}^{dim}$ residual 2-layer MLP (ReLU)
111
125
  - Multi-Head Attention: causal mask, RoPE; queries/keys/values per head
112
126
  - FFN: SwiGLU (SiLU-gated), pre-norm + residual
127
+ - QK-Norm with per-dimension adaptive scaling on queries, following [TimesFM-3](https://research.google/blog/timesfm-3-a-zero-shot-foundation-model-for-multivariate-forecasting/)
113
128
  - Output heads: |Q| linear maps $\mathbb{R}^{dim} \to \mathbb{R}^{P_{len}}$ (one per quantile)
114
129
 
115
130
  ### Model Details
@@ -192,10 +207,12 @@ With this strategy, the median prediction (0.5 quantile) does **not necessarily*
192
207
  This discrepancy arises because the *quantile collapse* step aggregates predictions across all possible quantile paths. As a result, the median is computed from the combined multi-path distribution rather than from a single deterministic trajectory, which can lead to different estimates compared to the single-path (median-only) autoregressive approach.
193
208
 
194
209
  ## Datasets
195
- - GIFT-Eval pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
210
+ - GIFT-pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
196
211
  domains and various frequencies. After preprocessing, this yields approximately 600K univariate series.
197
212
  - Chronos synthetic datasets [Chronos]: two large synthetic datasets generated with Chronos, one with TSMixup and one with KernelSynth. Each contains approximately 10 million univariate series and 1 million respectively, each signal of length 1024.
198
213
  - Artificial: ~1M synthetic series (sinusoidal, linear, polynomial, logarithmic) plus mixtures via TSMixup [Chronos]; Gaussian Process samples via KernelSynth (mixtures of RBF/periodic/linear kernels with swept hyperparameters).
214
+ - BOOM: Approximately 5 million time series of length 1024 sampled from the BOOM dataset [BOOM]. It is important to note that the BOOM dataset, which is also used as an evaluation benchmark, is included in the training data of both PatchFM variants (with and without benchmark leakage) to increase data diversity. Consequently, neither version can be fairly evaluated on the BOOM benchmark. Upon request, we can provide an alternative checkpoint trained without any BOOM data, enabling a fair evaluation on this benchmark.
215
+ - TSMixup [Chronos]: mixing synthetic datasets and GIFT-pretraining datasets to further increase diversity.
199
216
 
200
217
  ## Repository Layout
201
218
 
@@ -212,14 +229,12 @@ domains and various frequencies. After preprocessing, this yields approximately
212
229
 
213
230
  - `dataset/` — data loading and preprocessing
214
231
  - `artificial.py` — synthetic dataset : artificial signals + TSMixup + KernelSynth
232
+ - `synthetic.py` — other synthetic signals (Claude for the generation)
215
233
  - `gift.py` — GIFT-Eval pretraining dataset loading and preprocessing
216
234
  - `get_data.py` — utility to fetch and preprocess datasets
217
235
  - `chronosdata.py` — loading of the synthetic datasets generated with Chronos (TSMixup and KernelSynth) with download functions integrated
236
+ - `boom.py` — loading of the BOOM dataset with download functions integrated
218
237
 
219
238
  - `configs/` — model and training configurations
220
239
  - `notebooks/inference` — how to load a trained model and generate forecasts
221
- - `training.py` — training script using PyTorch Lightning
222
-
223
- ## Acknowledgements
224
- We thank the authors of the following repositories for inspiration and code snippets:
225
- - [TiRex](https://github.com/NX-AI/tirex)
240
+ - `training.py` — training script using PyTorch Lightning
{patchfm-3.1.1 → patchfm-3.2.0}/pyproject.toml RENAMED
@@ -1,6 +1,6 @@
1
1
  [project]
2
2
  name = "patchfm"
3
- version = "3.1.1"
3
+ version = "3.2.0"
4
4
  authors = [
5
5
  { name="Samy-Melwan Vilhes", email="samy-melwan.vilhes@insa-rouen.fr" },
6
6
  ]
{patchfm-3.1.1 → patchfm-3.2.0}/src/patchfm/inference/forecaster.py RENAMED
@@ -39,7 +39,9 @@ class Forecaster(nn.Module):
39
39
  self.converter = SeqTypeConverter()
40
40
 
41
41
  if config["compile"]:
42
- self = torch.compile(self)
42
+ self.proj_embedding = torch.compile(self.proj_embedding)
43
+ self.transformer_encoder = torch.compile(self.transformer_encoder)
44
+ self.proj_output = torch.compile(self.proj_output)
43
45
 
44
46
  def _init_components(self):
45
47
  """Initialize modules from scratch."""
{patchfm-3.1.1 → patchfm-3.2.0}/src/patchfm/inference/modules.py RENAMED
@@ -3,6 +3,7 @@
3
3
  import numpy as np
4
4
  import torch
5
5
  import torch.nn as nn
6
+ import math
6
7
  from einops import rearrange
7
8
  from huggingface_hub import PyTorchModelHubMixin
8
9
  from rotary_embedding_torch import RotaryEmbedding
@@ -184,6 +185,21 @@ class ResidualBlock(nn.Module):
184
185
  out = out+res
185
186
  return out
186
187
 
188
+ class PerDimScale(nn.Module):
189
+ def __init__(self, num_dims: int):
190
+ super().__init__()
191
+ self.num_dims = num_dims
192
+ self.per_dim_scale = nn.Parameter(torch.zeros(num_dims))
193
+ self._RECIPROCAL_OF_SOFTPLUS_0 = 1.442695041
194
+
195
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
196
+ return (
197
+ x
198
+ * self._RECIPROCAL_OF_SOFTPLUS_0
199
+ / math.sqrt(self.num_dims)
200
+ * torch.nn.functional.softplus(self.per_dim_scale)
201
+ )
202
+
187
203
  class MultiHeadAttention(nn.Module):
188
204
  def __init__(self, d_model, n_heads, last=False):
189
205
  super().__init__()
@@ -200,6 +216,10 @@ class MultiHeadAttention(nn.Module):
200
216
 
201
217
  self.rope = RotaryEmbedding(dim=self.head_dim//2)
202
218
 
219
+ self.q_norm = nn.RMSNorm(self.head_dim)
220
+ self.k_norm = nn.RMSNorm(self.head_dim)
221
+ self.per_dim_scale = PerDimScale(self.head_dim)
222
+
203
223
  self.k_cache = None
204
224
  self.v_cache = None
205
225
 
@@ -237,6 +257,10 @@ class MultiHeadAttention(nn.Module):
237
257
  q = self.rope.rotate_queries_or_keys(q, offset=offset)
238
258
  k = self.rope.rotate_queries_or_keys(k)
239
259
 
260
+ q = self.q_norm(q)
261
+ k = self.k_norm(k)
262
+ q = self.per_dim_scale(q)
263
+
240
264
  values = nn.functional.scaled_dot_product_attention(q, k, v, is_causal=is_causal)
241
265
 
242
266
  values = values.transpose(1, 2).reshape(bs, -1, dim)
{patchfm-3.1.1 → patchfm-3.2.0/src/patchfm.egg-info}/PKG-INFO RENAMED
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: patchfm
3
- Version: 3.1.1
3
+ Version: 3.2.0
4
4
  Summary: a Foundation Model for Univariate Time Series Forecasting
5
5
  Author-email: Samy-Melwan Vilhes <samy-melwan.vilhes@insa-rouen.fr>
6
6
  License: MIT License
@@ -40,32 +40,35 @@ Requires-Dist: numpy>=1.26.0
40
40
  Requires-Dist: safetensors==0.5.3
41
41
  Dynamic: license-file
42
42
 
43
- # A tutorial on how to build a Foundation Model for Univariate Time Series Forecasting
43
+ # A Foundation Model for Univariate Time Series Forecasting
44
44
 
45
45
  A concise, reproducible recipe for training a transformer-based, patch-to-patch forecasting model for univariate time series. The approach mirrors Large Language Model (LLM) practices (next-token → next-patch) while remaining lightweight compared to a classic LLM and practical.
46
46
 
47
- Our model is deployed on the [TS-Arena benchmark](https://ts-arena.live) and achieves competitive performance against state-of-the-art methods under the name `LITIS/PatchFM-Large`.
47
+ The model card is available on `assets/modelcard.pdf`.
48
48
 
49
- ## Resuls on the FEV benchmark:
49
+ Our model (with leakage) is deployed on the [TS-Arena benchmark](https://ts-arena.live) and achieves competitive performance against state-of-the-art methods under the name `LITIS/PatchFM-Large`.
50
50
 
51
- **Our model is competitive with the state-of-the-art notably compared to [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)**
51
+ ## Results on the FEV benchmark:
52
52
 
53
- | Model Name | Win Rate | Skill Score |
54
- |----------------------|----------|-------------|
55
- | (🥇) Chronos-2 | 0.9008 | 0.4728 |
56
- | (🥈) TiRex | 0.8092 | 0.4268 |
57
- | (🥉) TimesFM-2.5 | 0.8085 | 0.4668 |
58
- | (4) Toto-1.0 | 0.7069 | 0.4110 |
59
- | (5) **PatchFM (us)** | 0.6677 | 0.3983 |
60
- | (6) Moirai-2.0 | 0.6546 | 0.4026 |
61
- | (7) Chronos-Bolt | 0.6277 | 0.3889 |
62
- | (8) Sundial-Base | 0.4446 | 0.3387 |
63
- | (9) CatBoost (Recursive)| 0.3362 | 0.2301 |
64
- | (10) LightGBM (Recursive)| 0.3123 | 0.2168 |
65
- | (11) AutoTheta | 0.2938 | 0.0546 |
66
- | (12) Seasonal Naive | 0.2058 | 0.0000 |
67
- | (13) Naive | 0.1404 | -0.4540 |
68
- | (14) Drift | 0.0915 | -0.4578 |
53
+ **Our model (without leakage) remains competitive, though it does not match the top-performing state-of-the-art models.**
54
+
55
+ <div style="overflow-x: auto;">
56
+
57
+ | Metric | Chronos-2 | TiRex-2 | Toto-2.0-2.5B | Toto-2.0-1B | Toto-2.0-313m | TimesFM-2.5 | TiRex | Toto-2.0-22m | TS-ICL | **PatchFM** | FlowState | TabPFN-TS-3 | citras-fm | Toto-1.0 | Toto-2.0-4m | Moirai-2.0 | Chronos-Bolt | TFT | Sundial-Base | PatchTST | CatBoost | LightGBM | AutoTheta | Seasonal Naive | Naive | Drift |
58
+ |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
59
+ | Win Rate | 0.81 | 0.78 | 0.77 | 0.77 | 0.75 | 0.71 | 0.70 | 0.67 | 0.61 | **0.61** | 0.60 | 0.60 | 0.58 | 0.57 | 0.55 | 0.53 | 0.49 | 0.37 | 0.32 | 0.31 | 0.22 | 0.21 | 0.20 | 0.13 | 0.09 | 0.07 |
60
+ | Skill Score | 0.47 | 0.45 | 0.44 | 0.44 | 0.44 | 0.47 | 0.43 | 0.43 | 0.43 | **0.41** | 0.42 | 0.43 | 0.41 | 0.41 | 0.41 | 0.40 | 0.39 | 0.32 | 0.34 | 0.30 | 0.23 | 0.21 | 0.05 | 0.00 | -0.45 | -0.46 |
61
+
62
+ </div>
63
+
64
+
65
+ ## Acknowledgements
66
+ This work builds on ideas, model designs, and code from the following projects:
67
+ - [TiRex](https://github.com/NX-AI/tirex)
68
+ - [Toto](https://github.com/datadog/toto)
69
+ - [TimesFM](https://github.com/google-research/timesfm)
70
+ - [Chronos](https://github.com/amazon-science/chronos-forecasting)
71
+ We are grateful to their authors.
69
72
 
70
73
 
71
74
  ## Highlights
@@ -74,14 +77,24 @@ Our model is deployed on the [TS-Arena benchmark](https://ts-arena.live) and ach
74
77
  - Causal masking self-attention with RoPE (relative positions)
75
78
  - Causal RevIN with $\sinh^{-1}$ transform for normalization
76
79
  - SwiGLU feed-forward networks
80
+ - QK-Norm with per-dimension adaptive scaling on queries, following [TimesFM-3](https://research.google/blog/timesfm-3-a-zero-shot-foundation-model-for-multivariate-forecasting/)
77
81
  - Autoregressive multi-quantile decoding [MOIRAI2.0](https://arxiv.org/pdf/2511.11698)
78
82
  - KV-cache for efficient long-horizon inference
79
83
  - flips equivariance during inference (optional) [Reverso](https://arxiv.org/pdf/2602.17634v1)
84
+ - [Muon](https://kellerjordan.github.io/posts/muon/) optimizer for 2D weights and AdamW for the rest, as in Toto 2.0 ([implementation of Muon used](https://arxiv.org/abs/2502.16982))
80
85
 
81
86
  ## Quick Start
82
87
 
83
88
  ### from source code
84
89
 
90
+ > **⚠️ Important:** Two pretrained versions of PatchFM are available:
91
+
92
+ >
93
+
94
+ > - **`PatchFM-Leakage`**: trained on the GIFT-Eval datasets. This version may provide higher benchmark performance but includes training data that overlaps with the commonly used evaluation datasets.
95
+
96
+ > - **`PatchFM`** (**recommended for fair evaluation**): trained without any commonly used benchmark (except BOOM), preventing potential data leakage and ensuring a more reliable assessment of generalization performance.
97
+
85
98
  1. Clone the repository and install dependencies
86
99
  ```bash
87
100
  git clone https://github.com/vilhess/PatchFM
@@ -97,7 +110,7 @@ from configs import PatchFMConfig
97
110
  from model import Forecaster
98
111
 
99
112
  # --- Instantiate model ---
100
- config = PatchFMConfig(load_from_hub=True)
113
+ config = PatchFMConfig(load_from_hub=True, full_leakage=False) # set full_leakage=True to load the version trained with data leakage
101
114
  model = Forecaster(config)
102
115
 
103
116
  # --- Inference ---
@@ -119,6 +132,7 @@ import torch
119
132
 
120
133
  from patchfm import Forecaster, PatchFMConfig
121
134
 
135
+ config = PatchFMConfig(full_leakage=False)
122
136
  # same as above
123
137
  pred_median, pred_quantiles = model(seq, forecast_horizon=forecast_horizon, quantiles=[0.1, 0.5, 0.9], flip_equivariance=True) # (batch, time), (batch, time, quantiles)
124
138
  ```
@@ -152,6 +166,7 @@ Aggregate over positions, patch elements, and quantiles.
152
166
  - Input MLP: $\mathbb{R}^{P_{len}} \to \mathbb{R}^{dim}$ residual 2-layer MLP (ReLU)
153
167
  - Multi-Head Attention: causal mask, RoPE; queries/keys/values per head
154
168
  - FFN: SwiGLU (SiLU-gated), pre-norm + residual
169
+ - QK-Norm with per-dimension adaptive scaling on queries, following [TimesFM-3](https://research.google/blog/timesfm-3-a-zero-shot-foundation-model-for-multivariate-forecasting/)
155
170
  - Output heads: |Q| linear maps $\mathbb{R}^{dim} \to \mathbb{R}^{P_{len}}$ (one per quantile)
156
171
 
157
172
  ### Model Details
@@ -234,10 +249,12 @@ With this strategy, the median prediction (0.5 quantile) does **not necessarily*
234
249
  This discrepancy arises because the *quantile collapse* step aggregates predictions across all possible quantile paths. As a result, the median is computed from the combined multi-path distribution rather than from a single deterministic trajectory, which can lead to different estimates compared to the single-path (median-only) autoregressive approach.
235
250
 
236
251
  ## Datasets
237
- - GIFT-Eval pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
252
+ - GIFT-pretraining dataset [GIFT]: aligned with the GIFT-Eval dataset but without data leakage issue with the benchmark. The dataset contains approximately 71 univariate and 17 multivariate time series datasets from various
238
253
  domains and various frequencies. After preprocessing, this yields approximately 600K univariate series.
239
254
  - Chronos synthetic datasets [Chronos]: two large synthetic datasets generated with Chronos, one with TSMixup and one with KernelSynth. Each contains approximately 10 million univariate series and 1 million respectively, each signal of length 1024.
240
255
  - Artificial: ~1M synthetic series (sinusoidal, linear, polynomial, logarithmic) plus mixtures via TSMixup [Chronos]; Gaussian Process samples via KernelSynth (mixtures of RBF/periodic/linear kernels with swept hyperparameters).
256
+ - BOOM: Approximately 5 million time series of length 1024 sampled from the BOOM dataset [BOOM]. It is important to note that the BOOM dataset, which is also used as an evaluation benchmark, is included in the training data of both PatchFM variants (with and without benchmark leakage) to increase data diversity. Consequently, neither version can be fairly evaluated on the BOOM benchmark. Upon request, we can provide an alternative checkpoint trained without any BOOM data, enabling a fair evaluation on this benchmark.
257
+ - TSMixup [Chronos]: mixing synthetic datasets and GIFT-pretraining datasets to further increase diversity.
241
258
 
242
259
  ## Repository Layout
243
260
 
@@ -254,14 +271,12 @@ domains and various frequencies. After preprocessing, this yields approximately
254
271
 
255
272
  - `dataset/` — data loading and preprocessing
256
273
  - `artificial.py` — synthetic dataset : artificial signals + TSMixup + KernelSynth
274
+ - `synthetic.py` — other synthetic signals (Claude for the generation)
257
275
  - `gift.py` — GIFT-Eval pretraining dataset loading and preprocessing
258
276
  - `get_data.py` — utility to fetch and preprocess datasets
259
277
  - `chronosdata.py` — loading of the synthetic datasets generated with Chronos (TSMixup and KernelSynth) with download functions integrated
278
+ - `boom.py` — loading of the BOOM dataset with download functions integrated
260
279
 
261
280
  - `configs/` — model and training configurations
262
281
  - `notebooks/inference` — how to load a trained model and generate forecasts
263
282
  - `training.py` — training script using PyTorch Lightning
264
-
265
- ## Acknowledgements
266
- We thank the authors of the following repositories for inspiration and code snippets:
267
- - [TiRex](https://github.com/NX-AI/tirex)
{patchfm-3.1.1 → patchfm-3.2.0}/LICENSE RENAMED
File without changes
{patchfm-3.1.1 → patchfm-3.2.0}/setup.cfg RENAMED
File without changes
{patchfm-3.1.1 → patchfm-3.2.0}/src/patchfm/__init__.py RENAMED
File without changes