pystatistics 4.6.1__tar.gz → 4.6.2__tar.gz

{pystatistics-4.6.1 → pystatistics-4.6.2}/CHANGELOG.md

@@ -1,5 +1,93 @@
 # Changelog
 
+## 4.6.2
+
+STL decomposition rewritten to match R exactly and now as fast as R's
+compiled implementation; ETS gains automatic model selection and R-matched
+parameter estimation; `ndiffs` default aligned with R.
+
+- **`stl()` returned a materially wrong decomposition on trending series —
+  fixed by a full rewrite.** On strongly trending data (e.g. the monthly CO2
+  benchmark) the previous implementation leaked much of the trend into the
+  seasonal component: the extracted "seasonal" had a range of ~83 where the
+  true seasonal amplitude is ~6, and the reported trend could *decrease* while
+  the data rose — the components always summed back to the data, so the error
+  was invisible without an external reference. `stl()` is now an exact
+  implementation of the STL procedure (Cleveland, Cleveland, McRae &
+  Terpenning, 1990) and reproduces R's `stats::stl` component-for-component:
+  across a reference suite (CO2, AirPassengers, monthly sunspots, lynx, Nile;
+  periodic, robust, custom-window, and partial-period variants) the largest
+  seasonal/trend/remainder difference vs R is below 1e-10.
+- **`stl()` now exposes the full STL parameter set and uses R's defaults.**
+  New parameters: `seasonal_degree` (default 0, as in R — previously the
+  seasonal smoother was effectively degree 1), `trend_degree`,
+  `lowpass_window`, `lowpass_degree`, and the evaluation strides
+  `seasonal_jump` / `trend_jump` / `lowpass_jump` (R's `ceiling(window/10)`
+  defaults; set to 1 to evaluate the loess at every point instead of
+  interpolating). Behaviour changes: `seasonal_window` defaults to
+  `"periodic"` (R has no default and `"periodic"` is the standard published
+  choice); `robust=True` now runs R's iteration counts (1 inner, 15 outer
+  passes); the series must be longer than `2 * period` (R's rule); and window
+  spans must be odd integers >= 3 — even spans raise a `ValidationError`
+  where R silently rounds them up. The result's `info` now reports the
+  resolved windows/degrees/jumps and the final robustness weights.
+- **`stl()` now matches R's speed.** The smoother is compiled with `numba`,
+  so it is as fast as R's Fortran `stats::stl` — non-robust decompositions
+  run faster than R and robust decompositions match it (previously it was
+  3-15x slower). Results are unchanged (still within 1e-10 of R). The
+  compiled kernel builds on first use in a session (a few seconds, once) and
+  is cached on disk for subsequent runs, the same one-time cost the ARIMA
+  routines already carry.
+- **`ets()` now performs automatic model selection, matching
+  `forecast::ets`.** The model string accepts a `"Z"` wildcard per component
+  and defaults to `model="ZZZ"` (select everything), as in R. Candidates
+  consistent with the fixed letters are enumerated under R's rules (additive
+  error with multiplicative season excluded; multiplicative components
+  require strictly positive data; seasonal candidates need `2 <= period <=
+  24` and enough data), each is fitted, and the model minimising the new `ic`
+  parameter (`"aicc"` default, `"aic"`, `"bic"`) is returned. The full
+  candidate table, the criterion values, and any skipped candidates with
+  reasons are disclosed in `solution.info["selection"]`. Explicit requests
+  that cannot be honoured (e.g. `model="MZZ"` on data with zeros or
+  negatives, a seasonal letter with `period=1`) raise instead of being
+  silently adjusted. A fully specified model string (no `"Z"`) is always
+  fitted exactly as written. Note two consequences of the new default:
+  automatic AICc selection needs at least 5 observations (shorter series
+  raise with a suggested remedy — pick a concrete model or `ic="aic"`), and
+  `period` must now be a true integer (float periods raise a clear error
+  instead of crashing).
+- **ETS parameter estimation aligned with `forecast::ets` (changes fitted
+  results).** The optimiser now searches R's parameter region exactly — the
+  smoothing parameters obey `beta <= alpha` and `gamma <= 1 - alpha`, the
+  damping `phi` is bounded to `[0.8, 0.98]`, and a seasonal model estimates
+  one fewer initial state (the last is fixed by the seasonal normalisation,
+  as in R). As a result the reported parameter count `k` for seasonal models
+  now matches R's, so AIC/AICc/BIC values line up with `forecast::ets`, and
+  fitted parameters/log-likelihoods for a given model can differ slightly
+  from earlier versions. Smoothing parameters fixed outside R's region now
+  raise a clear error instead of being quietly clamped. On automatic
+  selection the chosen model still occasionally differs from `forecast::ets`
+  on near-tied candidate tables, but now only because the two optimisers land
+  on different optima — in every such case the model this package selects
+  scores at least as well under R's own criterion; the disclosed candidate
+  table makes any selection auditable.
+- **`ets()` no longer emits a spurious `RuntimeWarning` about overflow.** A
+  benign internal overflow warning could surface during fitting on some
+  series; the computation was always correct, and the warning is now gone.
+  Fitted results are unchanged.
+- **`ndiffs()` now defaults to the KPSS test, matching `forecast::ndiffs`.**
+  KPSS and ADF have opposite null hypotheses and can recommend a different
+  number of differences on borderline series; pass `test="adf"` for the
+  previous behaviour. Both tests are numerically unchanged.
+- **ETS log-likelihood reporting convention documented (no numeric
+  change).** `ets()` reports the full Gaussian log-likelihood; R's
+  `forecast::ets` prints Hyndman's concentrated form. The two differ by the
+  exact constant `0.5*n*[log(n/(2*pi)) - 1]` (e.g. 88.36 at n=100) with the
+  same parameter count, so AIC/AICc/BIC differences, model rankings, and
+  automatic selection are identical — only the printed numbers differ. The
+  convention is now stated on the result's `log_likelihood`/`aic`/`aicc`/
+  `bic` accessors.
+
 ## 4.6.1
 
 Performance: faster automatic smoothing-parameter selection for Gaussian

{pystatistics-4.6.1 → pystatistics-4.6.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pystatistics
-Version: 4.6.1
+Version: 4.6.2
 Summary: GPU-accelerated statistical computing for Python
 Project-URL: Homepage, https://sgcx.org/technology/pystatistics/
 Project-URL: Documentation, https://sgcx.org/docs/pystatistics/
@@ -418,6 +418,26 @@ pip install pystatistics[dev]
 
 ## What's New
 
+### 4.6.2 — STL matches R exactly and runs as fast; ETS automatic model selection
+
+`stl()` was rewritten after validation showed it leaking trend into the
+seasonal component on trending series (on the monthly CO2 benchmark the old
+"seasonal" had a range of ~83 where the truth is ~6). It is now an exact
+implementation of the STL procedure and reproduces R's `stats::stl`
+component-for-component (differences below 1e-10 across a reference suite
+including robust fits), with the full STL parameter set exposed
+(`seasonal_degree`, `trend_degree`, `lowpass_window`, jumps, and a
+`"periodic"` default seasonal window). The smoother is compiled, so it now
+runs as fast as R's Fortran implementation — faster for non-robust
+decompositions, on par for robust ones. `ets()` now auto-selects its model
+like `forecast::ets`: the new default `model="ZZZ"` tries every admissible
+error/trend/season combination and returns the best by AICc, with the full
+candidate table disclosed on the result; explicit component requests that
+cannot be honoured raise instead of being silently adjusted. ETS parameter
+estimation now matches R's parameter region as well, so fitted parameters
+and information criteria (AIC/AICc/BIC) line up with `forecast::ets`.
+`ndiffs()` now defaults to the KPSS test, matching `forecast::ndiffs`.
+
 ### 4.6.1 — faster multi-smooth Gaussian GAMs
 
 `gam()` with the default Gaussian family now selects its smoothing parameters

{pystatistics-4.6.1 → pystatistics-4.6.2}/README.md

@@ -371,6 +371,26 @@ pip install pystatistics[dev]
 
 ## What's New
 
+### 4.6.2 — STL matches R exactly and runs as fast; ETS automatic model selection
+
+`stl()` was rewritten after validation showed it leaking trend into the
+seasonal component on trending series (on the monthly CO2 benchmark the old
+"seasonal" had a range of ~83 where the truth is ~6). It is now an exact
+implementation of the STL procedure and reproduces R's `stats::stl`
+component-for-component (differences below 1e-10 across a reference suite
+including robust fits), with the full STL parameter set exposed
+(`seasonal_degree`, `trend_degree`, `lowpass_window`, jumps, and a
+`"periodic"` default seasonal window). The smoother is compiled, so it now
+runs as fast as R's Fortran implementation — faster for non-robust
+decompositions, on par for robust ones. `ets()` now auto-selects its model
+like `forecast::ets`: the new default `model="ZZZ"` tries every admissible
+error/trend/season combination and returns the best by AICc, with the full
+candidate table disclosed on the result; explicit component requests that
+cannot be honoured raise instead of being silently adjusted. ETS parameter
+estimation now matches R's parameter region as well, so fitted parameters
+and information criteria (AIC/AICc/BIC) line up with `forecast::ets`.
+`ndiffs()` now defaults to the KPSS test, matching `forecast::ndiffs`.
+
 ### 4.6.1 — faster multi-smooth Gaussian GAMs
 
 `gam()` with the default Gaussian family now selects its smoothing parameters

{pystatistics-4.6.1 → pystatistics-4.6.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pystatistics"
-version = "4.6.1"
+version = "4.6.2"
 description = "GPU-accelerated statistical computing for Python"
 readme = "README.md"
 license = "MIT"

{pystatistics-4.6.1 → pystatistics-4.6.2}/pystatistics/__init__.py

@@ -16,7 +16,7 @@ Usage:
     result = fit(design)
 """
 
-__version__ = "4.6.1"
+__version__ = "4.6.2"
 __author__ = "Hai-Shuo"
 __email__ = "contact@sgcx.org"
 

{pystatistics-4.6.1 → pystatistics-4.6.2}/pystatistics/timeseries/__init__.py

@@ -13,13 +13,22 @@ Public API:
     ndiffs(x)       - Estimate differences for stationarity (matches R forecast::ndiffs)
     adf_test(x)     - Augmented Dickey-Fuller unit root test (matches R tseries::adf.test)
     kpss_test(x)    - KPSS stationarity test (matches R tseries::kpss.test)
-    ets(y)          - Fit an ETS state space model (matches R forecast::ets)
+    ets(y)          - Fit an ETS state space model (matches R forecast::ets,
+                      including "Z"-wildcard automatic selection; default
+                      model="ZZZ". Reported log-likelihood/AIC use the full-
+                      Gaussian convention — R's concentrated numbers differ by
+                      a constant in n; model rankings/selection are identical.
+                      See timeseries._ets_fit and ._ets_select docstrings.)
     forecast_ets(f) - Forecast from a fitted ETS model
     arima(y)        - Fit an ARIMA model (matches R stats::arima)
     forecast_arima(f, y) - Forecast from a fitted ARIMA model
     auto_arima(y)   - Automatic ARIMA order selection (matches R forecast::auto.arima)
     decompose(x)    - Classical time series decomposition (matches R stats::decompose)
-    stl(x)          - STL decomposition (matches R stats::stl)
+    stl(x)          - STL decomposition (matches R stats::stl; identical parameters
+                      reproduce R's components to floating-point noise. Interface
+                      divergences, both deliberate: seasonal_window defaults to
+                      "periodic" where R requires it explicitly, and even/short
+                      loess spans raise instead of being silently rounded up.)
 """
 
 from pystatistics.timeseries._acf import acf, pacf
@@ -28,7 +37,8 @@ from pystatistics.timeseries._stationarity import adf_test, kpss_test
 from pystatistics.timeseries._common import (
     ACFParams, ACFSolution, StationarityParams, StationaritySolution,
 )
-from pystatistics.timeseries._ets_fit import ets, ETSParams, ETSSolution
+from pystatistics.timeseries._ets_fit import ETSParams, ETSSolution
+from pystatistics.timeseries._ets_select import ets
 from pystatistics.timeseries._ets_forecast import forecast_ets, ETSForecast
 from pystatistics.timeseries._ets_models import ETSSpec
 from pystatistics.timeseries._arima_solution import ARIMAParams, ARIMASolution
@@ -41,8 +51,9 @@ from pystatistics.timeseries._arima_order import (
     auto_arima, AutoARIMAParams, AutoARIMASolution,
 )
 from pystatistics.timeseries._decomposition import (
-    decompose, stl, DecompositionParams, DecompositionSolution,
+    decompose, DecompositionParams, DecompositionSolution,
 )
+from pystatistics.timeseries._stl import stl
 
 __all__ = [
     "acf",

{pystatistics-4.6.1 → pystatistics-4.6.2}/pystatistics/timeseries/_decomposition.py

@@ -1,14 +1,13 @@
 """
-Time series decomposition: classical and STL.
+Classical time series decomposition and shared decomposition results.
 
-Provides decompose() matching R's stats::decompose() and stl() matching
-R's stats::stl(). Classical decomposition uses centered moving averages;
-STL uses iterative LOESS-based smoothing (Cleveland et al., 1990).
+Provides decompose() matching R's stats::decompose() (centered moving
+averages) plus the DecompositionParams/DecompositionSolution result
+types shared with the STL implementation in ``_stl.py``.
 """
 
 from __future__ import annotations
 
-import math
 from dataclasses import dataclass
 
 import numpy as np
@@ -362,276 +361,3 @@ def decompose(
             warnings=(),
         )
     )
-
-
-# ---------------------------------------------------------------------------
-# LOESS helper (simplified local regression)
-# ---------------------------------------------------------------------------
-
-def _tricube(d: NDArray) -> NDArray:
-    """Tricube weight function: (1 - |d|^3)^3 for |d| < 1, else 0."""
-    w = np.zeros_like(d)
-    mask = np.abs(d) < 1.0
-    w[mask] = (1.0 - np.abs(d[mask]) ** 3) ** 3
-    return w
-
-
-def _loess_smooth(
-    x_pos: NDArray,
-    y_vals: NDArray,
-    span: int,
-    degree: int = 1,
-    weights: NDArray | None = None,
-) -> NDArray:
-    """
-    Simplified LOESS smoothing for STL.
-
-    For each point, fits a weighted local polynomial of the given degree
-    using a tricube kernel with *span* nearest neighbours.
-
-    Parameters
-    ----------
-    x_pos : NDArray
-        Positions (typically ``np.arange(len(y_vals))``).
-    y_vals : NDArray
-        Values to smooth.
-    span : int
-        Number of nearest neighbours (window width).
-    degree : int
-        Polynomial degree (0 or 1).
-    weights : NDArray or None
-        External robustness weights (multiplied with tricube weights).
-
-    Returns
-    -------
-    NDArray
-        Smoothed values at each position in *x_pos*.
-    """
-    n = len(y_vals)
-    span = min(span, n)
-    smoothed = np.empty(n)
-
-    for i in range(n):
-        # Find span nearest neighbours
-        dists = np.abs(x_pos - x_pos[i])
-        idx = np.argsort(dists)[:span]
-        d_max = dists[idx[-1]]
-        if d_max == 0:
-            d_max = 1.0
-
-        w = _tricube(dists[idx] / d_max)
-        if weights is not None:
-            w = w * weights[idx]
-
-        xi = x_pos[idx]
-        yi = y_vals[idx]
-
-        if degree == 0 or np.sum(w) == 0:
-            total_w = np.sum(w)
-            smoothed[i] = np.sum(w * yi) / total_w if total_w > 0 else yi[0]
-        else:
-            # Weighted least squares: degree-1 polynomial
-            W = np.diag(w)
-            X = np.column_stack([np.ones(len(xi)), xi - x_pos[i]])
-            try:
-                XtW = X.T @ W
-                beta = np.linalg.solve(XtW @ X, XtW @ yi)
-                smoothed[i] = beta[0]
-            except np.linalg.LinAlgError:
-                smoothed[i] = np.average(yi, weights=w) if np.sum(w) > 0 else yi[0]
-
-    return smoothed
-
-
-# ---------------------------------------------------------------------------
-# Low-pass filter for STL
-# ---------------------------------------------------------------------------
-
-def _low_pass_filter(x: NDArray, period: int) -> NDArray:
-    """
-    Low-pass filter used in STL inner loop.
-
-    Applies three successive moving averages of lengths *period*, *period*,
-    and 3, matching the original STL specification.
-
-    Parameters
-    ----------
-    x : NDArray
-        Input series.
-    period : int
-        Seasonal period.
-
-    Returns
-    -------
-    NDArray
-        Filtered series (same length, NaN-padded edges trimmed via LOESS).
-    """
-    # MA of length period
-    k1 = np.ones(period) / period
-    s1 = np.convolve(x, k1, mode="same")
-    # MA of length period again
-    s2 = np.convolve(s1, k1, mode="same")
-    # MA of length 3
-    k3 = np.ones(3) / 3.0
-    s3 = np.convolve(s2, k3, mode="same")
-    return s3
-
-
-# ---------------------------------------------------------------------------
-# STL decomposition
-# ---------------------------------------------------------------------------
-
-def _next_odd(v: int) -> int:
-    """Return *v* if odd, else *v* + 1."""
-    return v if v % 2 == 1 else v + 1
-
-
-def stl(
-    x: ArrayLike,
-    period: int,
-    *,
-    seasonal_window: int | None = None,
-    trend_window: int | None = None,
-    robust: bool = False,
-    n_outer: int = 0,
-    n_inner: int = 2,
-) -> DecompositionSolution:
-    """
-    Seasonal and Trend decomposition using LOESS (STL).
-
-    Simplified implementation of Cleveland et al. (1990), matching
-    R's ``stats::stl()`` in spirit.
-
-    Parameters
-    ----------
-    x : ArrayLike
-        Time series (1-D). Length must be >= 2 * *period*.
-    period : int
-        Seasonal period (>= 2).
-    seasonal_window : int or None
-        LOESS window width for seasonal extraction. Must be odd and >= 7.
-        Default: next odd >= ``period + 1``, but at least 7.
-    trend_window : int or None
-        LOESS window width for trend extraction. Must be odd.
-        Default: next odd >= ``ceil(1.5 * period / (1 - 1.5 / s_window)) + 1``.
-    robust : bool
-        If True, sets *n_outer* = 1 if it is 0, enabling robustness weights.
-    n_outer : int
-        Number of outer (robustness) iterations (>= 0).
-    n_inner : int
-        Number of inner iterations (>= 1).
-
-    Returns
-    -------
-    DecompositionSolution
-
-    Raises
-    ------
-    ValidationError
-        On invalid inputs.
-    """
-    arr = _validate_series(x)
-    _validate_decompose_inputs(arr, period, "additive")
-
-    n = len(arr)
-
-    # Default seasonal window
-    if seasonal_window is None:
-        seasonal_window = _next_odd(max(7, period + 1))
-    else:
-        if seasonal_window < 7:
-            raise ValidationError(
-                f"seasonal_window: must be >= 7, got {seasonal_window}"
-            )
-        if seasonal_window % 2 == 0:
-            raise ValidationError(
-                f"seasonal_window: must be odd, got {seasonal_window}"
-            )
-
-    # Default trend window
-    if trend_window is None:
-        denom = 1.0 - 1.5 / seasonal_window
-        trend_window = _next_odd(
-            max(3, int(math.ceil(1.5 * period / denom)) + 1)
-        )
-    else:
-        if trend_window % 2 == 0:
-            raise ValidationError(
-                f"trend_window: must be odd, got {trend_window}"
-            )
-
-    if n_inner < 1:
-        raise ValidationError(f"n_inner: must be >= 1, got {n_inner}")
-
-    if robust and n_outer == 0:
-        n_outer = 1
-
-    # Initialise components
-    trend = np.zeros(n)
-    seasonal = np.zeros(n)
-    rob_weights = np.ones(n)
-    x_pos_full = np.arange(n, dtype=float)
-
-    total_outer = max(1, n_outer + 1)
-
-    for _outer in range(total_outer):
-        for _inner in range(n_inner):
-            # Step 1: Detrending
-            detrended = arr - trend
-
-            # Step 2: Cycle-subseries smoothing
-            smoothed_seasonal = np.empty(n)
-            for pos in range(period):
-                indices = np.arange(pos, n, period)
-                sub_y = detrended[indices]
-                sub_x = np.arange(len(sub_y), dtype=float)
-                sub_w = rob_weights[indices]
-                sub_smooth = _loess_smooth(
-                    sub_x, sub_y, span=seasonal_window, degree=1, weights=sub_w,
-                )
-                smoothed_seasonal[indices] = sub_smooth
-
-            # Step 3: Low-pass filter on smoothed seasonal
-            lp = _low_pass_filter(smoothed_seasonal, period)
-
-            # Step 4: Seasonal = smoothed - low-pass
-            seasonal = smoothed_seasonal - lp
-
-            # Step 5: De-seasonalise
-            deseasoned = arr - seasonal
-
-            # Step 6: Trend via LOESS on deseasoned series
-            trend = _loess_smooth(
-                x_pos_full, deseasoned, span=trend_window,
-                degree=1, weights=rob_weights,
-            )
-
-        # Outer loop: compute robustness weights
-        if _outer < total_outer - 1 and n_outer > 0:
-            residual = arr - trend - seasonal
-            h = 6.0 * np.median(np.abs(residual))
-            if h > 0:
-                u = np.abs(residual) / h
-                rob_weights = _tricube(u)
-            else:
-                rob_weights = np.ones(n)
-
-    residual = arr - trend - seasonal
-
-    return DecompositionSolution(
-        _result=Result(
-            params=DecompositionParams(
-                observed=arr,
-                trend=trend,
-                seasonal=seasonal,
-                residual=residual,
-                period=period,
-                type="additive",
-                method="stl",
-            ),
-            info={"method": "stl", "type": "additive", "robust": robust},
-            timing=None,
-            backend_name="cpu",
-            warnings=(),
-        )
-    )

{pystatistics-4.6.1 → pystatistics-4.6.2}/pystatistics/timeseries/_differencing.py

@@ -81,23 +81,26 @@ def diff(
 def ndiffs(
     x: ArrayLike,
     *,
-    test: str = "adf",
+    test: str = "kpss",
     alpha: float = 0.05,
     max_d: int = 2,
 ) -> int:
     """
     Estimate the number of differences needed for stationarity.
 
-    Matches R's forecast::ndiffs(). Repeatedly differences and tests for
-    stationarity until either the test indicates stationarity or max_d
-    is reached.
+    Matches R's forecast::ndiffs(), including its default test. Repeatedly
+    differences and tests for stationarity until either the test indicates
+    stationarity or max_d is reached.
 
     Parameters
     ----------
     x : ArrayLike
         Time series.
     test : str
-        Stationarity test to use: 'adf' or 'kpss'.
+        Stationarity test to use: 'kpss' (default, matching R
+        forecast::ndiffs) or 'adf'. The two tests have opposite null
+        hypotheses (KPSS: stationary; ADF: unit root) and can recommend
+        different d on borderline series.
     alpha : float
         Significance level for the test.
     max_d : int