anomaly-pipeline 0.1.27__py3-none-any.whl → 0.1.61__py3-none-any.whl

anomaly_pipeline/__init__.py

@@ -1,2 +1,74 @@
+"""----------------------------------------------------------------------------
+help_anomaly
+----------------------------------------------------------------------------
+
+For a nice overview of the anomaly pipeline run the following lines of code:
+
+>> from anomaly_pipeline.helpers.help_anomaly import help_anomaly
+>> help_anomaly()
+
+You can see information about specific models used in the anomaly pipeline with any of the following commands:
+
+>> help_anomaly('percentile')
+>> help_anomaly('iqr')
+>> help_anomaly('mad')
+>> help_anomaly('std')
+>> help_anomaly('ewma')
+>> help_anomaly('prophet')
+>> help_anomaly('dbscan')
+>> help_anomaly('iso') # For information on isolation forest
+
+
+----------------------------------------------------------------------------
+Functional Overview
+----------------------------------------------------------------------------
+The pipeline takes raw master data, partitions it into groups by unique ID, applies a suite of 8 different anomaly detection methods, and then flags observations as anomalies where at least half of the models consider the observation an anomaly.
+
+The master data DataFrame that you pass into the anomaly detection pipeline needs to have at least 3 columns - unique ID, date, and a target variable. The unique ID can be defined by multiple columns.
+
+Core Execution Stages
+ 1. Preprocessing & Interpolation
+    Before modeling, the function interpolates target variable values for missing dates
+    Fill gaps in the variable column to prevent model crashes.
+
+ 2. Statistical Baseline Models (Local Execution)
+    The pipeline first runs four computationally light models sequentially on each group:
+    - Percentile & IQR: Non-parametric bounds detection.
+    - SD (Standard Deviation) & MAD (Median Absolute Deviation): Variance-based detection.
+
+ 3. Parallel Machine Learning Suite (process_group)
+    To maximize performance, the pipeline uses joblib.Parallel to run intensive models across all available CPU cores. The process_group utility acts as a router, sending data to the correct engine based on the model key:
+    - FB (Prophet): Walk-forward time-series forecasting.
+    - EWMA: Exponentially weighted moving averages.
+    - ISF (Isolation Forest): Unsupervised isolation of anomalies.
+    - DBSCAN: Density-based spatial clustering.
+
+ 4. Majority Voting (Ensemble Logic)
+    The power of this pipeline lies in its Consensus Model. After all models finish, the pipeline calculates:
+    - Anomaly_Votes: The sum of flags across all 8 methods.
+    - is_Anomaly: A final boolean set to True only if at least 4 models agree that the point is an outlier.
+
+Key Output Columns
+   refresh_date: The timestamp of when the pipeline was executed.
+   Anomaly_Votes: Total count of models that flagged the row.
+   is_Anomaly: The final "Gold Standard" anomaly flag.
+   Individual Model Flags: Columns like is_FB_anomaly, is_IQR_anomaly, etc., for granular auditing.
+
+Usage Context
+   Use run_pipeline when you need a highly reliable, automated output. By combining statistical, forecasting, and clustering models, the pipeline reduces "false positives" often generated by single-model approaches.
+"""
+
 from .main import timeseries_anomaly_detection
-from .helpers import help_info
+
+from .helpers import (help_anomaly,
+                      get_example_df,
+                      evaluation_info,
+                      anomaly_overview_plot,
+    anomaly_percentile_plot,
+    anomaly_sd_plot, 
+    anomaly_mad_plot, 
+    anomaly_iqr_plot, 
+    anomaly_ewma_plot, 
+    anomaly_fb_plot, 
+    anomaly_dbscan_plot, 
+    anomaly_isolation_forest_plot)

anomaly_pipeline/helpers/DB_scan.py

@@ -33,6 +33,7 @@ def get_dynamic_lags(series: pd.Series) -> list:
     
     return dynamic_lags
 
+
 def find_optimal_epsilon(X_scaled: np.ndarray, k: int) -> float:
     """
     Finds the optimal epsilon by calculating the distance to the k-th nearest neighbor
@@ -71,16 +72,133 @@ def detect_time_series_anomalies_dbscan(
     eval_period,
     ):
     
+    """# 🌀 DBSCAN Walk-Forward Anomaly Detection
+    ---
+
+    The `detect_time_series_anomalies_dbscan` function implements a **density-based clustering** approach for time-series anomaly detection. It utilizes an **iterative walk-forward validation** strategy to identify data points that exist in "low-density" regions of the feature space.
+
+    ## 📋 Functional Overview
+    This function transforms a univariate time series into a high-dimensional feature space using **dynamic lags** and **rolling statistics**. It then applies the **DBSCAN** (Density-Based Spatial Clustering of Applications with Noise) algorithm to distinguish between dense clusters of "normal" behavior and sparse "noise" points (anomalies).
+
+    ## 🧠 Core Logic & Helper Utilities
+
+    ### 1. Dynamic Feature Engineering (`get_dynamic_lags`)
+    Instead of using fixed lags, the function uses the **Autocorrelation Function (ACF)** to find the 10 most significant seasonal patterns in the data.
+    * **Baseline:** Always includes lags 1, 2, and 3 to capture immediate momentum.
+    * **Significance:** Uses a 75% confidence interval ($\\\\alpha=0.25$) to identify meaningful historical dependencies.
+
+    ### 2. Automated Parameter Tuning (`find_optimal_epsilon`)
+    DBSCAN is highly sensitive to the **Epsilon ($\\\\epsilon$)** parameter (the neighborhood radius). 
+    * **Proxy Elbow Method:** The function automatically calculates $\\\\epsilon$ by analyzing the distance to the $k$-th nearest neighbor for all training points.
+    * **Density Threshold:** It sets $\\\\epsilon$ at the **95th percentile** of these distances, ensuring that 95% of training data is considered "dense" while the most isolated 5% are candidates for noise.
+
+    ### 3. Walk-Forward Iteration
+    For the initial training period, all points are evaluated using DBSCAN fitted on the same training data.
+    For each period in the `eval_period`:
+    * **Feature Construction:** Builds a matrix containing the variable, its dynamic lags, rolling means, rolling standard deviations, and a linear trend component.
+    * **Scaling:** Fits a `StandardScaler` **only on training data** to prevent data leakage.
+    * **Novelty Detection:** Since DBSCAN cannot "predict" on new points, the function uses a **Nearest Neighbors proxy**. If the distance from a new test point to its $k$-th neighbor in the training set is greater than the trained $\\\\epsilon$, it is flagged as an anomaly.
+
+    ## 📤 Key Output Columns
+    * **`dbscan_score`**: The distance from the point to the $\\\\epsilon$ boundary (positive values indicate anomalies).
+    * **`is_DBSCAN_anomaly`**: A boolean flag identifying outliers.
+    * **Generated Features**: Includes all dynamic lags (`lagX`) and rolling statistics (`roll_mean_W`) used during the fit.
+
+    ## 💡 Usage Context
+    DBSCAN is exceptionally powerful for detecting **contextual anomalies**—points that might look "normal" in value but are "weird" given their recent history or seasonal context. Because it is density-based, it can find anomalies in non-linear or multi-modal distributions where simple percentile or Z-score methods would fail.
+
+    ---
+    ### ⚠️ Performance Note
+    This model is computationally more intensive than statistical methods due to the iterative re-fitting of the `NearestNeighbors` and `DBSCAN` models. It is best suited for high-priority metrics where accuracy is more critical than processing speed."""
+
     group[date_column] = pd.to_datetime(group[date_column])
     group = group.copy().sort_values(date_column).reset_index(drop=True)
+    group['set'] = np.where(np.arange(len(group)) >= len(group) - eval_period, 'TEST', 'TRAIN')
     
     # --- Default DBSCAN Parameters ---
     # These parameters often need tuning, but these are reasonable starting points:
     DEFAULT_EPS = 0.5 # Neighborhood radius (critical parameter)
     
     try:
-        test_anom = []
+        all_results = []
+
+        # ===================================================================
+        # STEP 1: Evaluate all points in the initial TRAIN period
+        # ===================================================================
+        
+        # Get the cutoff date for initial train period
+        initial_cutoff_date = group[group['set'] == 'TRAIN'][date_column].max()
+        
+        # Prepare the full group with features
+        model_group_initial = group.copy()
+        
+        # Get train set to determine lags
+        train_initial = model_group_initial[model_group_initial['set'] == 'TRAIN'].copy()
+        lags = get_dynamic_lags(train_initial[variable])
+
+        # Create lag features and rolling stats for the entire DF
+        rolling_stats_features = []
+        for lag in lags:
+            model_group_initial[f'lag{lag}'] = model_group_initial[variable].shift(lag)
+
+        for w in [int(np.ceil(max(lags)/4)), int(np.ceil(max(lags)/2)), int(max(lags))]:
+            if w >= 3:
+                rolling_stats_features.extend([f'roll_mean_{w}', f'roll_std_{w}'])
+                model_group_initial[f'roll_mean_{w}'] = model_group_initial[variable].shift(1).rolling(w).mean()
+                model_group_initial[f'roll_std_{w}'] = model_group_initial[variable].shift(1).rolling(w).std()
+
+        model_group_initial['trend'] = model_group_initial.index
+        model_group_initial = model_group_initial.copy().dropna()
 
+        # Get just the initial train set
+        train_initial = model_group_initial[model_group_initial['set'] == 'TRAIN'].copy()
+
+        # Identify all model features (lags, rolling stats, trend, and the variable itself)
+        features = [f'lag{i}' for i in lags] + rolling_stats_features + ['trend'] + [variable]
+
+        # Fit the scaler on the training data
+        scaler = StandardScaler()
+        scaler.fit(train_initial[features])
+        train_scaled = scaler.transform(train_initial[features])
+
+        # Determine min_samples based on feature space dimension
+        min_samples = max(2 * len(features), 3)
+        
+        # Find optimal epsilon
+        calculated_eps = find_optimal_epsilon(train_scaled, k=min_samples)
+
+        # --- DBSCAN MODEL on initial training data ---
+        dbscan_model = DBSCAN(
+            eps=calculated_eps, 
+            min_samples=min_samples,
+            n_jobs=-1
+        )
+
+        # Fit DBSCAN on the scaled training data
+        cluster_labels = dbscan_model.fit_predict(train_scaled)
+        
+        # For training points, use DBSCAN labels (-1 = noise/anomaly)
+        train_initial['is_DBSCAN_anomaly'] = (cluster_labels == -1)
+        
+        # Calculate scores for training points
+        # Use distance to k-th nearest neighbor as score
+        neigh = NearestNeighbors(n_neighbors=min_samples)
+        neigh.fit(train_scaled)
+        distances, indices = neigh.kneighbors(train_scaled)
+        k_distance = distances[:, min_samples - 1]
+        
+        train_initial['dbscan_score'] = k_distance - calculated_eps
+        train_initial['dbscan_score_high'] = 0
+        
+        # Select relevant columns
+        train_initial_result = train_initial[[variable, date_column, 'dbscan_score', 
+                                             'dbscan_score_high', 'is_DBSCAN_anomaly']]
+        all_results.append(train_initial_result)
+
+        # ===================================================================
+        # STEP 2: Walk-forward evaluation for TEST period (one-step-ahead)
+        # ===================================================================
+        
         for t in list(range(eval_period - 1, -1, -1)):
 
             try:
@@ -155,27 +273,42 @@ def detect_time_series_anomalies_dbscan(
                 
                 # Flag as anomaly if the k-distance is greater than the trained eps threshold
                 test['dbscan_score'] = k_distance - calculated_eps
-                test['is_DBSCAN_anomaly'] = np.where(k_distance > calculated_eps, True, False)
+                test['dbscan_score_high'] = 0
+                test['is_DBSCAN_anomaly'] = np.where(test['dbscan_score'] > 0, True, False)
                 
-                test = test[[variable, date_column, 'dbscan_score', 'is_DBSCAN_anomaly']]
-                test_anom.append(test)
+                test = test[[variable, date_column, 'dbscan_score', 'dbscan_score_high', 'is_DBSCAN_anomaly']]
+                all_results.append(test)
 
             except Exception as e:
                 print(f"Error in iteration {t}: {e}")
                 pass
             
+        # ===================================================================
+        # STEP 3: Combine all results and merge back to original group
+        # ===================================================================
+        
         try:
-            test_anom = pd.concat(test_anom)
-            group = group.merge(test_anom[[variable, date_column, 'dbscan_score', 'is_DBSCAN_anomaly']], on=[variable, date_column], how='left')
+            all_results_df = pd.concat(all_results, ignore_index=True)
+            
+            # Merge back to original group
+            group = group.merge(
+                all_results_df[[variable, date_column, 'dbscan_score', 
+                               'dbscan_score_high', 'is_DBSCAN_anomaly']], 
+                on=[variable, date_column], 
+                how='left'
+            )
+            
+            # Fill any remaining NaNs with False for boolean column
             # group["is_DBSCAN_anomaly"] = group["is_DBSCAN_anomaly"].fillna(False)
-        except:
-            print("Error in DBSCAN process")
+            
+        except Exception as e:
+            print(f"Error in concatenating results: {e}")
             group['dbscan_score'] = np.nan
+            group['dbscan_score_high'] = np.nan
             group["is_DBSCAN_anomaly"] = np.nan
 
     except Exception as e:
         # Fallback error handling
-        # Replace key_series with group for robustness if key_series is not defined
         try:
              group_id_cols = group.select_dtypes(include=['object', 'string']).columns.tolist()
              group_id = " ".join(group[group_id_cols].reset_index(drop=True).iloc[0].astype(str).to_list())
@@ -183,6 +316,7 @@ def detect_time_series_anomalies_dbscan(
              group_id = "Unknown Group ID"
         print(f'DBSCAN Anomaly Detection failed for {group_id}. Error: {e}')
         group['dbscan_score'] = np.nan
+        group['dbscan_score_high'] = np.nan
         group["is_DBSCAN_anomaly"] = np.nan
         
-    return group
+    return group

anomaly_pipeline/helpers/MAD.py

@@ -5,6 +5,51 @@ from .Preprocessing import classify
 
 
 def detect_outliers_mad(group, variable, date_column, mad_threshold, mad_scale_factor, eval_period):
+    
+    """
+    # 🛡️ MAD Anomaly Detection Model
+    ---
+
+    Median Absolute Deviation with Expanding Window
+
+    The detect_outliers_mad function is a non-parametric outlier detection tool.
+    Unlike methods based on the standard deviation, this model uses the Median and MAD,
+    making it significantly more robust against data that contains extreme outliers or non-normal distributions.
+
+    ## 📋 Functional Overview
+
+    The function identifies anomalies by calculating how far a data point deviates from the median.
+    It utilizes an expanding window approach to ensure that as the dataset grows,
+    the definition of "normal" behavior adapts dynamically to the historical context.
+
+    ## 🧠 Core Logic Stages
+
+    1. Preprocessing & Validation
+    Sample Size Check: Requires a minimum of 10 data points. If the group is too small, it returns an empty DataFrame to avoid biased statistical results.
+    Deep Copy: Operates on a group.copy() to ensure the original input data remains untouched.
+
+    2. Initial Training Block
+    Baseline Calculation: For the first part of the series (pre-evaluation period), it establishes a static baseline.
+    The MAD Formula: > It calculates the Median Absolute Deviation: MAD = median(|x_i - median(x)|).
+    Thresholding: It uses a mad_scale_factor (default 0.6745) to make the MAD comparable to a standard deviation for a normal distribution.
+    Bounds:
+        MAD_high: Median + (Threshold x Scale)$
+        MAD_low: max(Median - (Threshold x Scale), 0)$
+
+    3. Expanding Window Evaluation
+    Incremental Testing: For each point in the evaluation period, the function recalculates the Median and MAD using all data available up to that point.
+    Real-time Simulation: This simulates a "production" environment where each new weekly point is tested against the entirety of its known history.
+    Zero-Variance Handling: If MAD is 0 (all historical values are identical), the bounds collapse to the median value to avoid division errors.
+
+    ## 📤 Key Output Columns
+
+    ## 💡 Usage Context
+
+    The MAD model is the "gold standard" for univariate outlier detection in robust statistics. It is highly recommended for:
+    - Data with large, extreme spikes that would skew a Mean-based (SD) model.
+    - Datasets that are not normally distributed.
+    - Scenarios where you need a conservative, reliable boundary that isn't easily shifted by a single bad data point."""
+
     n = len(group)
     if n < 10:
         return pd.DataFrame(columns=group.columns)