claude-flow-novice 1.0.0

package/wiki/languages/python/data-science/README.md

@@ -0,0 +1,1317 @@
+# Python Data Science with Claude-Flow
+
+Comprehensive guide to data science workflows, machine learning pipelines, and Jupyter integration using claude-flow orchestration for enhanced productivity and collaboration.
+
+## 🔬 Data Science Overview
+
+### Data Science Agent Ecosystem
+
+| Agent Type | Specialization | Primary Tools | Use Cases |
+|------------|---------------|---------------|-----------|
+| **`ml-developer`** | Machine Learning | scikit-learn, TensorFlow, PyTorch | Model development, training |
+| **`data-scientist`** | Analysis & Visualization | pandas, NumPy, Matplotlib | EDA, statistical analysis |
+| **`data-engineer`** | ETL & Pipelines | Apache Airflow, Spark | Data processing, pipelines |
+| **`researcher`** | Domain Research | Literature, APIs | Research, feature discovery |
+
+## 📊 Quick Start: Data Science Projects
+
+### 1. Exploratory Data Analysis (EDA)
+
+```bash
+# Initialize data analysis project with claude-flow
+npx claude-flow@alpha sparc run spec-pseudocode "Customer behavior analysis with visualization"
+
+# Spawn coordinated data science team
+Task("Data Scientist", "Perform exploratory data analysis with pandas and seaborn", "data-scientist")
+Task("Visualization Expert", "Create interactive dashboards with Plotly", "coder")
+Task("Statistical Analyst", "Run statistical tests and hypothesis testing", "researcher")
+Task("Report Writer", "Generate comprehensive analysis report", "api-docs")
+```
+
+### 2. Machine Learning Pipeline
+
+```bash
+# Complete ML workflow with SPARC methodology
+npx claude-flow@alpha sparc tdd "Predictive model for customer churn"
+
+# Parallel ML development
+Task("Data Engineer", "Build feature engineering pipeline", "data-engineer")
+Task("ML Engineer", "Train and evaluate multiple models", "ml-developer")
+Task("MLOps Engineer", "Setup model deployment and monitoring", "cicd-engineer")
+Task("Performance Analyst", "Analyze model performance and optimization", "performance-optimizer")
+```
+
+### 3. Time Series Analysis
+
+```bash
+# Time series forecasting project
+Task("Time Series Analyst", "Build ARIMA and Prophet forecasting models", "ml-developer")
+Task("Feature Engineer", "Create lag features and seasonality components", "data-scientist")
+Task("Validation Engineer", "Implement cross-validation for time series", "tester")
+```
+
+## 📓 Jupyter Notebook Integration
+
+### Jupyter Environment Setup
+
+```python
+# notebook_setup.py - Generated by claude-flow agents
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+import seaborn as sns
+import plotly.express as px
+import plotly.graph_objects as go
+from plotly.subplots import make_subplots
+
+# Machine learning libraries
+from sklearn.model_selection import train_test_split, cross_val_score
+from sklearn.preprocessing import StandardScaler, LabelEncoder
+from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
+from sklearn.linear_model import LogisticRegression
+from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
+
+# Deep learning
+import tensorflow as tf
+from tensorflow import keras
+from tensorflow.keras import layers
+
+# Set display options
+pd.set_option('display.max_columns', None)
+pd.set_option('display.width', None)
+pd.set_option('display.max_colwidth', None)
+
+# Plotting style
+plt.style.use('seaborn-v0_8')
+sns.set_palette("husl")
+
+# Jupyter extensions for claude-flow integration
+%load_ext autoreload
+%autoreload 2
+
+# Claude-flow notebook helpers
+import subprocess
+import os
+
+def notify_claude_flow(message):
+    """Notify claude-flow of notebook progress."""
+    try:
+        subprocess.run([
+            "npx", "claude-flow@alpha", "hooks", "notify",
+            "--message", f"Jupyter: {message}"
+        ], check=True)
+    except subprocess.CalledProcessError:
+        print(f"Note: {message}")
+
+def save_to_memory(key, value):
+    """Save data to claude-flow memory."""
+    try:
+        subprocess.run([
+            "npx", "claude-flow@alpha", "mcp", "memory_usage",
+            "--action", "store",
+            "--key", f"jupyter/{key}",
+            "--value", str(value)
+        ], check=True)
+    except subprocess.CalledProcessError:
+        print(f"Saved locally: {key} = {value}")
+
+print("Claude-Flow Jupyter environment ready! 🚀")
+```
+
+### Data Analysis Notebook Template
+
+```python
+# 01_exploratory_data_analysis.ipynb - Generated by data-scientist agent
+
+# Cell 1: Setup and Data Loading
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+import seaborn as sns
+from notebook_setup import notify_claude_flow, save_to_memory
+
+# Load dataset
+df = pd.read_csv('data/customer_data.csv')
+notify_claude_flow("Dataset loaded successfully")
+
+print(f"Dataset shape: {df.shape}")
+print(f"Memory usage: {df.memory_usage(deep=True).sum() / 1024**2:.2f} MB")
+
+# Cell 2: Data Overview
+def analyze_dataset(df):
+    """Comprehensive dataset analysis."""
+    analysis = {
+        'shape': df.shape,
+        'columns': list(df.columns),
+        'dtypes': df.dtypes.to_dict(),
+        'missing_values': df.isnull().sum().to_dict(),
+        'duplicate_rows': df.duplicated().sum(),
+        'numeric_columns': df.select_dtypes(include=[np.number]).columns.tolist(),
+        'categorical_columns': df.select_dtypes(include=['object']).columns.tolist()
+    }
+
+    # Save analysis to claude-flow memory
+    save_to_memory("dataset_analysis", analysis)
+
+    return analysis
+
+analysis = analyze_dataset(df)
+print("Dataset Analysis:")
+for key, value in analysis.items():
+    print(f"{key}: {value}")
+
+# Cell 3: Missing Values Analysis
+def plot_missing_values(df):
+    """Visualize missing values pattern."""
+    plt.figure(figsize=(12, 8))
+
+    # Missing values heatmap
+    plt.subplot(2, 2, 1)
+    sns.heatmap(df.isnull(), cbar=True, yticklabels=False, cmap='viridis')
+    plt.title('Missing Values Heatmap')
+
+    # Missing values bar plot
+    plt.subplot(2, 2, 2)
+    missing_counts = df.isnull().sum()
+    missing_counts[missing_counts > 0].plot(kind='bar')
+    plt.title('Missing Values Count')
+    plt.xticks(rotation=45)
+
+    # Missing values percentage
+    plt.subplot(2, 2, 3)
+    missing_percent = (df.isnull().sum() / len(df)) * 100
+    missing_percent[missing_percent > 0].plot(kind='bar', color='orange')
+    plt.title('Missing Values Percentage')
+    plt.xticks(rotation=45)
+
+    plt.tight_layout()
+    plt.show()
+
+plot_missing_values(df)
+notify_claude_flow("Missing values analysis completed")
+
+# Cell 4: Statistical Summary
+def generate_statistical_summary(df):
+    """Generate comprehensive statistical summary."""
+    numeric_df = df.select_dtypes(include=[np.number])
+
+    summary = {
+        'descriptive_stats': numeric_df.describe().to_dict(),
+        'correlation_matrix': numeric_df.corr().to_dict(),
+        'skewness': numeric_df.skew().to_dict(),
+        'kurtosis': numeric_df.kurtosis().to_dict()
+    }
+
+    # Correlation heatmap
+    plt.figure(figsize=(10, 8))
+    sns.heatmap(numeric_df.corr(), annot=True, cmap='coolwarm', center=0)
+    plt.title('Feature Correlation Matrix')
+    plt.show()
+
+    save_to_memory("statistical_summary", summary)
+    return summary
+
+stats = generate_statistical_summary(df)
+notify_claude_flow("Statistical analysis completed")
+
+# Cell 5: Distribution Analysis
+def plot_distributions(df, columns=None):
+    """Plot distributions for numeric columns."""
+    if columns is None:
+        columns = df.select_dtypes(include=[np.number]).columns
+
+    n_cols = min(len(columns), 4)
+    n_rows = (len(columns) + n_cols - 1) // n_cols
+
+    fig, axes = plt.subplots(n_rows, n_cols, figsize=(15, 4*n_rows))
+    axes = axes.flatten() if n_rows > 1 else [axes]
+
+    for i, col in enumerate(columns):
+        if i < len(axes):
+            # Histogram with KDE
+            axes[i].hist(df[col].dropna(), bins=30, alpha=0.7, density=True)
+            df[col].plot.kde(ax=axes[i], secondary_y=False)
+            axes[i].set_title(f'Distribution of {col}')
+            axes[i].set_xlabel(col)
+            axes[i].set_ylabel('Density')
+
+    # Hide unused subplots
+    for i in range(len(columns), len(axes)):
+        axes[i].set_visible(False)
+
+    plt.tight_layout()
+    plt.show()
+
+plot_distributions(df)
+notify_claude_flow("Distribution analysis completed")
+```
+
+### Machine Learning Notebook
+
+```python
+# 02_machine_learning_pipeline.ipynb - Generated by ml-developer agent
+
+# Cell 1: ML Pipeline Setup
+from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV
+from sklearn.preprocessing import StandardScaler, LabelEncoder, OneHotEncoder
+from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
+from sklearn.linear_model import LogisticRegression
+from sklearn.svm import SVC
+from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score, roc_curve
+from sklearn.pipeline import Pipeline
+import joblib
+
+class MLPipeline:
+    """Comprehensive ML pipeline with claude-flow integration."""
+
+    def __init__(self, target_column):
+        self.target_column = target_column
+        self.models = {}
+        self.results = {}
+        self.best_model = None
+        self.preprocessor = None
+
+    def preprocess_data(self, df):
+        """Preprocess data for machine learning."""
+        # Separate features and target
+        X = df.drop(columns=[self.target_column])
+        y = df[self.target_column]
+
+        # Handle categorical variables
+        categorical_columns = X.select_dtypes(include=['object']).columns
+        numeric_columns = X.select_dtypes(include=[np.number]).columns
+
+        # Create preprocessing pipeline
+        from sklearn.compose import ColumnTransformer
+
+        preprocessor = ColumnTransformer(
+            transformers=[
+                ('num', StandardScaler(), numeric_columns),
+                ('cat', OneHotEncoder(drop='first', sparse=False), categorical_columns)
+            ]
+        )
+
+        self.preprocessor = preprocessor
+        X_processed = preprocessor.fit_transform(X)
+
+        # Save preprocessing info to memory
+        preprocessing_info = {
+            'categorical_columns': categorical_columns.tolist(),
+            'numeric_columns': numeric_columns.tolist(),
+            'target_column': self.target_column,
+            'feature_names': (numeric_columns.tolist() +
+                            [f"{col}_{val}" for col in categorical_columns
+                             for val in df[col].unique()[1:]])
+        }
+        save_to_memory("preprocessing_info", preprocessing_info)
+
+        return X_processed, y
+
+    def train_models(self, X, y, test_size=0.2, random_state=42):
+        """Train multiple models and compare performance."""
+        # Split data
+        X_train, X_test, y_train, y_test = train_test_split(
+            X, y, test_size=test_size, random_state=random_state, stratify=y
+        )
+
+        # Define models
+        models = {
+            'Logistic Regression': LogisticRegression(random_state=random_state),
+            'Random Forest': RandomForestClassifier(random_state=random_state),
+            'Gradient Boosting': GradientBoostingClassifier(random_state=random_state),
+            'SVM': SVC(probability=True, random_state=random_state)
+        }
+
+        results = {}
+
+        for name, model in models.items():
+            print(f"Training {name}...")
+
+            # Cross-validation
+            cv_scores = cross_val_score(model, X_train, y_train, cv=5, scoring='roc_auc')
+
+            # Train on full training set
+            model.fit(X_train, y_train)
+
+            # Predictions
+            y_pred = model.predict(X_test)
+            y_pred_proba = model.predict_proba(X_test)[:, 1]
+
+            # Metrics
+            results[name] = {
+                'model': model,
+                'cv_mean': cv_scores.mean(),
+                'cv_std': cv_scores.std(),
+                'test_auc': roc_auc_score(y_test, y_pred_proba),
+                'classification_report': classification_report(y_test, y_pred, output_dict=True)
+            }
+
+            print(f"{name} - CV AUC: {cv_scores.mean():.4f} (+/- {cv_scores.std()*2:.4f})")
+            print(f"{name} - Test AUC: {roc_auc_score(y_test, y_pred_proba):.4f}")
+
+        self.models = models
+        self.results = results
+        self.X_test = X_test
+        self.y_test = y_test
+
+        # Find best model
+        best_model_name = max(results.keys(), key=lambda k: results[k]['test_auc'])
+        self.best_model = models[best_model_name]
+
+        save_to_memory("model_results", {k: {
+            'cv_mean': v['cv_mean'],
+            'cv_std': v['cv_std'],
+            'test_auc': v['test_auc']
+        } for k, v in results.items()})
+
+        notify_claude_flow(f"Model training completed. Best model: {best_model_name}")
+
+        return results
+
+    def plot_model_comparison(self):
+        """Plot model comparison."""
+        model_names = list(self.results.keys())
+        cv_means = [self.results[name]['cv_mean'] for name in model_names]
+        test_aucs = [self.results[name]['test_auc'] for name in model_names]
+
+        fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))
+
+        # CV scores
+        ax1.bar(model_names, cv_means)
+        ax1.set_title('Cross-Validation AUC Scores')
+        ax1.set_ylabel('AUC Score')
+        ax1.set_xticklabels(model_names, rotation=45)
+
+        # Test scores
+        ax2.bar(model_names, test_aucs)
+        ax2.set_title('Test Set AUC Scores')
+        ax2.set_ylabel('AUC Score')
+        ax2.set_xticklabels(model_names, rotation=45)
+
+        plt.tight_layout()
+        plt.show()
+
+    def plot_roc_curves(self):
+        """Plot ROC curves for all models."""
+        plt.figure(figsize=(10, 8))
+
+        for name, result in self.results.items():
+            model = result['model']
+            y_pred_proba = model.predict_proba(self.X_test)[:, 1]
+            fpr, tpr, _ = roc_curve(self.y_test, y_pred_proba)
+            auc = result['test_auc']
+            plt.plot(fpr, tpr, label=f'{name} (AUC = {auc:.3f})')
+
+        plt.plot([0, 1], [0, 1], 'k--', label='Random')
+        plt.xlabel('False Positive Rate')
+        plt.ylabel('True Positive Rate')
+        plt.title('ROC Curves Comparison')
+        plt.legend()
+        plt.grid(True)
+        plt.show()
+
+    def hyperparameter_tuning(self, model_name='Random Forest'):
+        """Perform hyperparameter tuning for selected model."""
+        if model_name not in self.models:
+            raise ValueError(f"Model {model_name} not found")
+
+        model = self.models[model_name]
+
+        # Define parameter grids
+        param_grids = {
+            'Random Forest': {
+                'n_estimators': [100, 200, 300],
+                'max_depth': [None, 10, 20, 30],
+                'min_samples_split': [2, 5, 10],
+                'min_samples_leaf': [1, 2, 4]
+            },
+            'Gradient Boosting': {
+                'n_estimators': [100, 200],
+                'learning_rate': [0.01, 0.1, 0.2],
+                'max_depth': [3, 5, 7]
+            }
+        }
+
+        if model_name in param_grids:
+            param_grid = param_grids[model_name]
+
+            print(f"Performing hyperparameter tuning for {model_name}...")
+            grid_search = GridSearchCV(
+                model, param_grid, cv=5, scoring='roc_auc', n_jobs=-1
+            )
+
+            # Use training data for tuning
+            X_train = self.X_test  # This should be actual training data
+            y_train = self.y_test  # This should be actual training labels
+
+            grid_search.fit(X_train, y_train)
+
+            print(f"Best parameters: {grid_search.best_params_}")
+            print(f"Best CV score: {grid_search.best_score_:.4f}")
+
+            self.best_model = grid_search.best_estimator_
+            save_to_memory("best_model_params", grid_search.best_params_)
+
+            return grid_search
+        else:
+            print(f"No parameter grid defined for {model_name}")
+            return None
+
+# Initialize and run ML pipeline
+ml_pipeline = MLPipeline('target_column_name')  # Replace with actual target
+X_processed, y = ml_pipeline.preprocess_data(df)
+results = ml_pipeline.train_models(X_processed, y)
+
+# Visualize results
+ml_pipeline.plot_model_comparison()
+ml_pipeline.plot_roc_curves()
+
+notify_claude_flow("ML pipeline execution completed")
+```
+
+## 🔄 Data Engineering Workflows
+
+### ETL Pipeline with Apache Airflow
+
+```python
+# dags/data_pipeline.py - Data engineering workflow
+from datetime import datetime, timedelta
+from airflow import DAG
+from airflow.operators.python import PythonOperator
+from airflow.operators.bash import BashOperator
+import pandas as pd
+import numpy as np
+import requests
+
+default_args = {
+    'owner': 'claude-flow-data-engineer',
+    'depends_on_past': False,
+    'start_date': datetime(2024, 1, 1),
+    'email_on_failure': True,
+    'email_on_retry': False,
+    'retries': 2,
+    'retry_delay': timedelta(minutes=5)
+}
+
+def extract_data(**context):
+    """Extract data from multiple sources."""
+    # API data extraction
+    api_response = requests.get('https://api.example.com/data')
+    api_data = api_response.json()
+
+    # Database extraction
+    db_query = """
+    SELECT customer_id, purchase_date, amount, product_category
+    FROM transactions
+    WHERE date >= NOW() - INTERVAL '1 DAY'
+    """
+    db_data = pd.read_sql(db_query, connection_string)
+
+    # File data extraction
+    file_data = pd.read_csv('/data/raw/daily_data.csv')
+
+    # Combine data sources
+    combined_data = {
+        'api_data': api_data,
+        'db_data': db_data.to_dict('records'),
+        'file_data': file_data.to_dict('records')
+    }
+
+    # Store in staging area
+    pd.DataFrame(combined_data['db_data']).to_parquet('/data/staging/db_data.parquet')
+    pd.DataFrame(combined_data['file_data']).to_parquet('/data/staging/file_data.parquet')
+
+    # Notify claude-flow
+    subprocess.run([
+        "npx", "claude-flow@alpha", "hooks", "notify",
+        "--message", "Data extraction completed"
+    ])
+
+    return combined_data
+
+def transform_data(**context):
+    """Transform and clean extracted data."""
+    # Load staging data
+    db_data = pd.read_parquet('/data/staging/db_data.parquet')
+    file_data = pd.read_parquet('/data/staging/file_data.parquet')
+
+    # Data cleaning
+    def clean_dataframe(df):
+        # Remove duplicates
+        df = df.drop_duplicates()
+
+        # Handle missing values
+        numeric_columns = df.select_dtypes(include=[np.number]).columns
+        df[numeric_columns] = df[numeric_columns].fillna(df[numeric_columns].median())
+
+        categorical_columns = df.select_dtypes(include=['object']).columns
+        df[categorical_columns] = df[categorical_columns].fillna('Unknown')
+
+        return df
+
+    # Feature engineering
+    def engineer_features(df):
+        if 'purchase_date' in df.columns:
+            df['purchase_date'] = pd.to_datetime(df['purchase_date'])
+            df['day_of_week'] = df['purchase_date'].dt.dayofweek
+            df['month'] = df['purchase_date'].dt.month
+            df['is_weekend'] = df['day_of_week'].isin([5, 6])
+
+        if 'amount' in df.columns:
+            df['amount_log'] = np.log1p(df['amount'])
+            df['amount_category'] = pd.cut(
+                df['amount'],
+                bins=[0, 50, 200, 1000, float('inf')],
+                labels=['low', 'medium', 'high', 'premium']
+            )
+
+        return df
+
+    # Apply transformations
+    db_data_clean = clean_dataframe(db_data)
+    db_data_transformed = engineer_features(db_data_clean)
+
+    file_data_clean = clean_dataframe(file_data)
+    file_data_transformed = engineer_features(file_data_clean)
+
+    # Combine transformed data
+    combined_transformed = pd.concat([
+        db_data_transformed,
+        file_data_transformed
+    ], ignore_index=True)
+
+    # Save transformed data
+    combined_transformed.to_parquet('/data/processed/daily_processed.parquet')
+
+    # Data quality checks
+    quality_report = {
+        'total_records': len(combined_transformed),
+        'null_counts': combined_transformed.isnull().sum().to_dict(),
+        'duplicate_count': combined_transformed.duplicated().sum(),
+        'numeric_stats': combined_transformed.describe().to_dict()
+    }
+
+    # Save quality report
+    with open('/data/quality/quality_report.json', 'w') as f:
+        json.dump(quality_report, f)
+
+    subprocess.run([
+        "npx", "claude-flow@alpha", "hooks", "notify",
+        "--message", f"Data transformation completed. {len(combined_transformed)} records processed"
+    ])
+
+    return quality_report
+
+def load_data(**context):
+    """Load processed data to data warehouse."""
+    # Load processed data
+    processed_data = pd.read_parquet('/data/processed/daily_processed.parquet')
+
+    # Load to data warehouse
+    processed_data.to_sql(
+        'customer_analytics',
+        warehouse_connection,
+        if_exists='append',
+        index=False,
+        method='multi'
+    )
+
+    # Update metadata table
+    metadata = {
+        'load_date': datetime.now(),
+        'record_count': len(processed_data),
+        'source': 'daily_pipeline',
+        'version': context['run_id']
+    }
+
+    pd.DataFrame([metadata]).to_sql(
+        'pipeline_metadata',
+        warehouse_connection,
+        if_exists='append',
+        index=False
+    )
+
+    subprocess.run([
+        "npx", "claude-flow@alpha", "hooks", "notify",
+        "--message", f"Data loaded to warehouse. {len(processed_data)} records"
+    ])
+
+# Define DAG
+dag = DAG(
+    'customer_data_pipeline',
+    default_args=default_args,
+    description='Customer data ETL pipeline with claude-flow integration',
+    schedule_interval='@daily',
+    catchup=False,
+    max_active_runs=1
+)
+
+# Define tasks
+extract_task = PythonOperator(
+    task_id='extract_data',
+    python_callable=extract_data,
+    dag=dag
+)
+
+transform_task = PythonOperator(
+    task_id='transform_data',
+    python_callable=transform_data,
+    dag=dag
+)
+
+load_task = PythonOperator(
+    task_id='load_data',
+    python_callable=load_data,
+    dag=dag
+)
+
+# Data quality validation
+quality_check = BashOperator(
+    task_id='quality_check',
+    bash_command='python /scripts/data_quality_check.py',
+    dag=dag
+)
+
+# Send notification to claude-flow
+notification_task = BashOperator(
+    task_id='notify_completion',
+    bash_command='npx claude-flow@alpha hooks notify --message "Daily pipeline completed successfully"',
+    dag=dag
+)
+
+# Set task dependencies
+extract_task >> transform_task >> load_task >> quality_check >> notification_task
+```
+
+## 📈 Advanced Analytics Workflows
+
+### Time Series Analysis
+
+```python
+# time_series_analysis.py - Advanced time series modeling
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+from statsmodels.tsa.arima.model import ARIMA
+from statsmodels.tsa.seasonal import seasonal_decompose
+from statsmodels.tsa.stattools import adfuller
+from prophet import Prophet
+import warnings
+warnings.filterwarnings('ignore')
+
+class TimeSeriesAnalyzer:
+    """Advanced time series analysis with claude-flow integration."""
+
+    def __init__(self, data, date_column, value_column):
+        self.data = data.copy()
+        self.date_column = date_column
+        self.value_column = value_column
+        self.prepare_data()
+
+    def prepare_data(self):
+        """Prepare time series data."""
+        # Convert date column
+        self.data[self.date_column] = pd.to_datetime(self.data[self.date_column])
+        self.data = self.data.sort_values(self.date_column)
+        self.data.set_index(self.date_column, inplace=True)
+
+        # Handle missing values
+        self.data[self.value_column] = self.data[self.value_column].interpolate()
+
+        save_to_memory("time_series_info", {
+            "start_date": str(self.data.index.min()),
+            "end_date": str(self.data.index.max()),
+            "frequency": str(self.data.index.freq),
+            "total_observations": len(self.data)
+        })
+
+    def stationarity_test(self):
+        """Test for stationarity using Augmented Dickey-Fuller test."""
+        result = adfuller(self.data[self.value_column].dropna())
+
+        stationarity_result = {
+            'adf_statistic': result[0],
+            'p_value': result[1],
+            'critical_values': result[4],
+            'is_stationary': result[1] < 0.05
+        }
+
+        print(f"ADF Statistic: {result[0]}")
+        print(f"p-value: {result[1]}")
+        print("Critical Values:")
+        for key, value in result[4].items():
+            print(f"\t{key}: {value}")
+
+        if result[1] < 0.05:
+            print("Series is stationary")
+        else:
+            print("Series is not stationary")
+
+        save_to_memory("stationarity_test", stationarity_result)
+        return stationarity_result
+
+    def seasonal_decomposition(self):
+        """Perform seasonal decomposition."""
+        decomposition = seasonal_decompose(
+            self.data[self.value_column],
+            model='additive',
+            period=12  # Adjust based on your data frequency
+        )
+
+        # Plot decomposition
+        fig, axes = plt.subplots(4, 1, figsize=(12, 10))
+
+        decomposition.observed.plot(ax=axes[0], title='Original')
+        decomposition.trend.plot(ax=axes[1], title='Trend')
+        decomposition.seasonal.plot(ax=axes[2], title='Seasonal')
+        decomposition.resid.plot(ax=axes[3], title='Residual')
+
+        plt.tight_layout()
+        plt.show()
+
+        # Save decomposition statistics
+        decomposition_stats = {
+            'trend_variance': float(decomposition.trend.var()),
+            'seasonal_variance': float(decomposition.seasonal.var()),
+            'residual_variance': float(decomposition.resid.var())
+        }
+
+        save_to_memory("seasonal_decomposition", decomposition_stats)
+        notify_claude_flow("Seasonal decomposition completed")
+
+        return decomposition
+
+    def arima_modeling(self, order=(1, 1, 1)):
+        """Build ARIMA model."""
+        model = ARIMA(self.data[self.value_column], order=order)
+        fitted_model = model.fit()
+
+        # Model summary
+        print(fitted_model.summary())
+
+        # Residual analysis
+        residuals = fitted_model.resid
+
+        fig, axes = plt.subplots(2, 2, figsize=(12, 8))
+
+        # Residuals plot
+        residuals.plot(ax=axes[0, 0], title='Residuals')
+
+        # Q-Q plot
+        from scipy import stats
+        stats.probplot(residuals, dist="norm", plot=axes[0, 1])
+        axes[0, 1].set_title('Q-Q Plot')
+
+        # Histogram of residuals
+        residuals.hist(ax=axes[1, 0], bins=30)
+        axes[1, 0].set_title('Residuals Histogram')
+
+        # ACF of residuals
+        from statsmodels.graphics.tsaplots import plot_acf
+        plot_acf(residuals, ax=axes[1, 1], lags=20)
+        axes[1, 1].set_title('ACF of Residuals')
+
+        plt.tight_layout()
+        plt.show()
+
+        # Model metrics
+        aic = fitted_model.aic
+        bic = fitted_model.bic
+
+        model_metrics = {
+            'aic': aic,
+            'bic': bic,
+            'order': order,
+            'log_likelihood': fitted_model.llf
+        }
+
+        save_to_memory("arima_model", model_metrics)
+        notify_claude_flow(f"ARIMA model fitted with AIC: {aic:.2f}")
+
+        return fitted_model
+
+    def prophet_modeling(self):
+        """Build Prophet model for forecasting."""
+        # Prepare data for Prophet
+        prophet_data = self.data.reset_index()
+        prophet_data = prophet_data.rename(columns={
+            self.date_column: 'ds',
+            self.value_column: 'y'
+        })
+
+        # Initialize and fit Prophet model
+        model = Prophet(
+            yearly_seasonality=True,
+            weekly_seasonality=False,
+            daily_seasonality=False,
+            seasonality_mode='additive'
+        )
+
+        model.fit(prophet_data)
+
+        # Make future dataframe for forecasting
+        future = model.make_future_dataframe(periods=30, freq='D')
+        forecast = model.predict(future)
+
+        # Plot forecast
+        fig = model.plot(forecast)
+        plt.title('Prophet Forecast')
+        plt.show()
+
+        # Plot components
+        fig = model.plot_components(forecast)
+        plt.show()
+
+        # Model performance metrics
+        from sklearn.metrics import mean_absolute_error, mean_squared_error
+
+        # Get predictions for historical data
+        historical_forecast = forecast[forecast['ds'].isin(prophet_data['ds'])]
+        mae = mean_absolute_error(prophet_data['y'], historical_forecast['yhat'])
+        rmse = np.sqrt(mean_squared_error(prophet_data['y'], historical_forecast['yhat']))
+
+        prophet_metrics = {
+            'mae': mae,
+            'rmse': rmse,
+            'forecast_periods': 30
+        }
+
+        save_to_memory("prophet_model", prophet_metrics)
+        notify_claude_flow(f"Prophet model fitted with MAE: {mae:.2f}")
+
+        return model, forecast
+
+# Example usage
+# ts_analyzer = TimeSeriesAnalyzer(df, 'date', 'sales')
+# ts_analyzer.stationarity_test()
+# ts_analyzer.seasonal_decomposition()
+# arima_model = ts_analyzer.arima_modeling()
+# prophet_model, forecast = ts_analyzer.prophet_modeling()
+```
+
+## 📊 Visualization and Reporting
+
+### Interactive Dashboard Creation
+
+```python
+# dashboard.py - Interactive data science dashboard
+import streamlit as st
+import pandas as pd
+import numpy as np
+import plotly.express as px
+import plotly.graph_objects as go
+from plotly.subplots import make_subplots
+import seaborn as sns
+import matplotlib.pyplot as plt
+
+class DataScienceDashboard:
+    """Interactive dashboard for data science insights."""
+
+    def __init__(self):
+        st.set_page_config(
+            page_title="Claude-Flow Data Science Dashboard",
+            page_icon="📊",
+            layout="wide"
+        )
+
+    def load_data(self):
+        """Load and cache data."""
+        @st.cache_data
+        def get_data():
+            # Load your dataset here
+            return pd.read_csv('data/analysis_data.csv')
+
+        return get_data()
+
+    def sidebar_filters(self, df):
+        """Create sidebar filters."""
+        st.sidebar.header("Filters")
+
+        # Date range filter
+        if 'date' in df.columns:
+            date_range = st.sidebar.date_input(
+                "Select Date Range",
+                value=(df['date'].min(), df['date'].max()),
+                min_value=df['date'].min(),
+                max_value=df['date'].max()
+            )
+
+        # Categorical filters
+        categorical_columns = df.select_dtypes(include=['object']).columns
+        filters = {}
+
+        for col in categorical_columns:
+            unique_values = df[col].unique()
+            selected_values = st.sidebar.multiselect(
+                f"Select {col}",
+                options=unique_values,
+                default=unique_values
+            )
+            filters[col] = selected_values
+
+        return filters
+
+    def overview_metrics(self, df):
+        """Display overview metrics."""
+        st.header("📈 Overview Metrics")
+
+        col1, col2, col3, col4 = st.columns(4)
+
+        with col1:
+            st.metric(
+                "Total Records",
+                f"{len(df):,}",
+                delta=f"{len(df) - 1000:,}" if len(df) > 1000 else None
+            )
+
+        with col2:
+            numeric_cols = df.select_dtypes(include=[np.number]).columns
+            if len(numeric_cols) > 0:
+                avg_value = df[numeric_cols[0]].mean()
+                st.metric(
+                    f"Average {numeric_cols[0]}",
+                    f"{avg_value:.2f}",
+                    delta=f"{avg_value - df[numeric_cols[0]].median():.2f}"
+                )
+
+        with col3:
+            if 'amount' in df.columns:
+                total_amount = df['amount'].sum()
+                st.metric(
+                    "Total Amount",
+                    f"${total_amount:,.2f}",
+                    delta=f"${total_amount * 0.1:,.2f}"
+                )
+
+        with col4:
+            unique_customers = df['customer_id'].nunique() if 'customer_id' in df.columns else 0
+            st.metric(
+                "Unique Customers",
+                f"{unique_customers:,}",
+                delta=f"{int(unique_customers * 0.05):,}"
+            )
+
+    def data_visualization(self, df):
+        """Create data visualizations."""
+        st.header("📊 Data Visualizations")
+
+        # Time series plot
+        if 'date' in df.columns and 'amount' in df.columns:
+            st.subheader("Time Series Analysis")
+
+            daily_data = df.groupby('date')['amount'].sum().reset_index()
+            fig = px.line(
+                daily_data,
+                x='date',
+                y='amount',
+                title="Daily Amount Trend"
+            )
+            st.plotly_chart(fig, use_container_width=True)
+
+        # Distribution plots
+        st.subheader("Distribution Analysis")
+
+        col1, col2 = st.columns(2)
+
+        with col1:
+            numeric_columns = df.select_dtypes(include=[np.number]).columns
+            if len(numeric_columns) > 0:
+                selected_column = st.selectbox("Select column for distribution", numeric_columns)
+                fig = px.histogram(
+                    df,
+                    x=selected_column,
+                    title=f"Distribution of {selected_column}"
+                )
+                st.plotly_chart(fig, use_container_width=True)
+
+        with col2:
+            categorical_columns = df.select_dtypes(include=['object']).columns
+            if len(categorical_columns) > 0:
+                selected_cat_column = st.selectbox("Select categorical column", categorical_columns)
+                value_counts = df[selected_cat_column].value_counts()
+                fig = px.pie(
+                    values=value_counts.values,
+                    names=value_counts.index,
+                    title=f"Distribution of {selected_cat_column}"
+                )
+                st.plotly_chart(fig, use_container_width=True)
+
+        # Correlation heatmap
+        st.subheader("Correlation Analysis")
+        numeric_df = df.select_dtypes(include=[np.number])
+        if len(numeric_df.columns) > 1:
+            correlation_matrix = numeric_df.corr()
+            fig = px.imshow(
+                correlation_matrix,
+                title="Correlation Heatmap",
+                color_continuous_scale="RdBu"
+            )
+            st.plotly_chart(fig, use_container_width=True)
+
+    def machine_learning_insights(self, df):
+        """Display ML model insights."""
+        st.header("🤖 Machine Learning Insights")
+
+        # Feature importance (if available)
+        if 'feature_importance.csv' in os.listdir():
+            feature_importance = pd.read_csv('feature_importance.csv')
+
+            fig = px.bar(
+                feature_importance.head(10),
+                x='importance',
+                y='feature',
+                orientation='h',
+                title="Top 10 Feature Importance"
+            )
+            st.plotly_chart(fig, use_container_width=True)
+
+        # Model performance metrics
+        col1, col2 = st.columns(2)
+
+        with col1:
+            st.subheader("Model Performance")
+            performance_data = {
+                'Model': ['Random Forest', 'Gradient Boosting', 'Logistic Regression'],
+                'AUC Score': [0.85, 0.83, 0.78],
+                'Accuracy': [0.82, 0.80, 0.75]
+            }
+            performance_df = pd.DataFrame(performance_data)
+
+            fig = px.bar(
+                performance_df,
+                x='Model',
+                y='AUC Score',
+                title="Model Comparison - AUC Scores"
+            )
+            st.plotly_chart(fig, use_container_width=True)
+
+        with col2:
+            st.subheader("Prediction Distribution")
+            # Simulated prediction data
+            predictions = np.random.beta(2, 5, 1000)
+            fig = px.histogram(
+                x=predictions,
+                title="Prediction Score Distribution",
+                nbins=30
+            )
+            st.plotly_chart(fig, use_container_width=True)
+
+    def run_dashboard(self):
+        """Run the complete dashboard."""
+        st.title("📊 Claude-Flow Data Science Dashboard")
+        st.markdown("---")
+
+        # Load data
+        df = self.load_data()
+
+        # Apply filters
+        filters = self.sidebar_filters(df)
+
+        # Filter dataframe based on selections
+        filtered_df = df.copy()
+        for col, values in filters.items():
+            if values:  # Only apply filter if values are selected
+                filtered_df = filtered_df[filtered_df[col].isin(values)]
+
+        # Display dashboard sections
+        self.overview_metrics(filtered_df)
+        st.markdown("---")
+
+        self.data_visualization(filtered_df)
+        st.markdown("---")
+
+        self.machine_learning_insights(filtered_df)
+
+        # Data download
+        st.sidebar.markdown("---")
+        st.sidebar.subheader("Download Data")
+        csv = filtered_df.to_csv(index=False)
+        st.sidebar.download_button(
+            label="Download Filtered Data as CSV",
+            data=csv,
+            file_name="filtered_data.csv",
+            mime="text/csv"
+        )
+
+# Run dashboard
+if __name__ == "__main__":
+    dashboard = DataScienceDashboard()
+    dashboard.run_dashboard()
+```
+
+## 🔗 MLOps Integration
+
+### Model Deployment Pipeline
+
+```python
+# mlops_pipeline.py - MLOps deployment with claude-flow
+import mlflow
+import mlflow.sklearn
+from mlflow.tracking import MlflowClient
+import joblib
+import pandas as pd
+import numpy as np
+from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
+import requests
+import json
+
+class MLOpsManager:
+    """MLOps pipeline management with claude-flow integration."""
+
+    def __init__(self, experiment_name="claude-flow-ml"):
+        mlflow.set_experiment(experiment_name)
+        self.client = MlflowClient()
+        self.experiment_name = experiment_name
+
+    def register_model(self, model, model_name, run_id=None):
+        """Register model in MLflow model registry."""
+        model_uri = f"runs:/{run_id}/model" if run_id else None
+
+        # Register model
+        result = mlflow.register_model(
+            model_uri=model_uri,
+            name=model_name,
+            tags={"created_by": "claude-flow", "framework": "sklearn"}
+        )
+
+        notify_claude_flow(f"Model {model_name} registered with version {result.version}")
+        return result
+
+    def deploy_model(self, model_name, version, stage="Staging"):
+        """Deploy model to specified stage."""
+        # Transition model to stage
+        self.client.transition_model_version_stage(
+            name=model_name,
+            version=version,
+            stage=stage
+        )
+
+        # Create deployment script
+        deployment_script = f"""
+import mlflow.sklearn
+import pandas as pd
+from flask import Flask, request, jsonify
+
+app = Flask(__name__)
+
+# Load model
+model = mlflow.sklearn.load_model('models:/{model_name}/{stage}')
+
+@app.route('/predict', methods=['POST'])
+def predict():
+    try:
+        data = request.get_json()
+        df = pd.DataFrame(data)
+        predictions = model.predict(df)
+        return jsonify({{'predictions': predictions.tolist()}})
+    except Exception as e:
+        return jsonify({{'error': str(e)}}), 400
+
+@app.route('/health', methods=['GET'])
+def health():
+    return jsonify({{'status': 'healthy', 'model': '{model_name}', 'version': '{version}'}})
+
+if __name__ == '__main__':
+    app.run(host='0.0.0.0', port=5000)
+"""
+
+        with open(f'deployment_{model_name}_{version}.py', 'w') as f:
+            f.write(deployment_script)
+
+        notify_claude_flow(f"Model {model_name} v{version} deployed to {stage}")
+
+    def monitor_model_performance(self, model_name, test_data, true_labels):
+        """Monitor model performance and detect drift."""
+        # Load production model
+        model = mlflow.sklearn.load_model(f"models:/{model_name}/Production")
+
+        # Generate predictions
+        predictions = model.predict(test_data)
+
+        # Calculate metrics
+        metrics = {
+            'accuracy': accuracy_score(true_labels, predictions),
+            'precision': precision_score(true_labels, predictions, average='weighted'),
+            'recall': recall_score(true_labels, predictions, average='weighted'),
+            'f1': f1_score(true_labels, predictions, average='weighted')
+        }
+
+        # Log monitoring metrics
+        with mlflow.start_run():
+            for metric_name, metric_value in metrics.items():
+                mlflow.log_metric(f"monitoring_{metric_name}", metric_value)
+
+        # Check for performance degradation
+        baseline_accuracy = 0.85  # Set your baseline
+        if metrics['accuracy'] < baseline_accuracy:
+            alert_message = f"Model performance degradation detected! Accuracy: {metrics['accuracy']:.3f}"
+            notify_claude_flow(alert_message)
+
+            # Trigger retraining workflow
+            self.trigger_retraining(model_name)
+
+        save_to_memory("model_monitoring", metrics)
+        return metrics
+
+    def trigger_retraining(self, model_name):
+        """Trigger model retraining workflow."""
+        # This could trigger an Airflow DAG or other workflow
+        retraining_config = {
+            'model_name': model_name,
+            'trigger_reason': 'performance_degradation',
+            'timestamp': pd.Timestamp.now().isoformat()
+        }
+
+        # Save retraining request
+        save_to_memory("retraining_trigger", retraining_config)
+
+        # Notify claude-flow to initiate retraining
+        notify_claude_flow(f"Retraining triggered for model {model_name}")
+
+# Model serving with FastAPI
+from fastapi import FastAPI, HTTPException
+from pydantic import BaseModel
+from typing import List
+
+class PredictionRequest(BaseModel):
+    features: List[List[float]]
+
+class PredictionResponse(BaseModel):
+    predictions: List[float]
+    model_version: str
+
+app = FastAPI(title="Claude-Flow ML Model API")
+
+# Load model at startup
+@app.on_event("startup")
+async def load_model():
+    global model, model_version
+    model = mlflow.sklearn.load_model("models:/{model_name}/Production")
+    model_version = "1.0.0"  # Get from MLflow
+
+@app.post("/predict", response_model=PredictionResponse)
+async def predict(request: PredictionRequest):
+    try:
+        predictions = model.predict(request.features)
+        return PredictionResponse(
+            predictions=predictions.tolist(),
+            model_version=model_version
+        )
+    except Exception as e:
+        raise HTTPException(status_code=400, detail=str(e))
+
+@app.get("/health")
+async def health_check():
+    return {"status": "healthy", "model_version": model_version}
+```
+
+## 📚 Next Steps
+
+1. **[Testing Strategies](../testing/README.md)** - Data science testing approaches
+2. **[Examples](../examples/README.md)** - Complete data science projects
+3. **[Web Development Integration](../web-development/README.md)** - ML models in web apps
+4. **[Performance Optimization](../examples/performance.md)** - Scaling data science workflows
+
+---
+
+**Ready to build data science pipelines?** Start with [Jupyter setup](./jupyter-setup.md) or explore [ML examples](../examples/machine-learning.md).