churnkit 0.75.0a1__py3-none-any.whl

churnkit-0.75.0a1.data/data/share/churnkit/exploration_notebooks/05_multi_dataset.ipynb

@@ -0,0 +1,1457 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "18a81db7",
+   "metadata": {
+    "papermill": {
+     "duration": 0.003496,
+     "end_time": "2026-02-02T13:03:21.294742",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:21.291246",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "# Chapter 5: Multi-Dataset Relationships\n",
+    "\n",
+    "**Purpose:** Combine multiple explored datasets, define relationships, and plan feature aggregations before feature engineering.\n",
+    "\n",
+    "**When to use this notebook:**\n",
+    "- You have explored multiple datasets using notebooks 01-04 (or 01a-01d)\n",
+    "- Your datasets share common keys (e.g., customer_id)\n",
+    "- You want to create features from event-level data to join with entity-level data\n",
+    "\n",
+    "**What you'll learn:**\n",
+    "- How to discover and manage multiple exploration findings\n",
+    "- How to detect and define relationships between datasets\n",
+    "- How to plan time-window aggregations for event datasets\n",
+    "- How to preview the feature set before engineering\n",
+    "\n",
+    "**Outputs:**\n",
+    "- Multi-dataset findings file (YAML)\n",
+    "- Relationship definitions\n",
+    "- Aggregation plan\n",
+    "\n",
+    "---\n",
+    "\n",
+    "## Multi-Dataset Architecture\n",
+    "\n",
+    "```\n",
+    "+------------------+      +-------------------+      +------------------+\n",
+    "|  Entity Dataset  |      |  Event Dataset 1  |      |  Event Dataset 2 |\n",
+    "|  (customers.csv) |      | (transactions.csv)|      |   (emails.csv)   |\n",
+    "|                  |      |                   |      |                  |\n",
+    "| - customer_id    |<---->| - customer_id     |<---->| - customer_id    |\n",
+    "| - churned (Y)    |      | - transaction_date|      | - sent_date      |\n",
+    "| - city           |      | - amount          |      | - opened         |\n",
+    "+------------------+      +-------------------+      +------------------+\n",
+    "         |                         |                         |\n",
+    "         v                         v                         v\n",
+    "   Primary Table           Aggregate to:              Aggregate to:\n",
+    "   (one row per           - amount_sum_7d            - email_count_30d\n",
+    "    customer)             - txn_count_30d            - open_rate_90d\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "915afefc",
+   "metadata": {
+    "papermill": {
+     "duration": 0.002093,
+     "end_time": "2026-02-02T13:03:21.299543",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:21.297450",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 5.1 Setup and Discover Datasets"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c7d77a44",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T13:03:21.304757Z",
+     "iopub.status.busy": "2026-02-02T13:03:21.304630Z",
+     "iopub.status.idle": "2026-02-02T13:03:23.135487Z",
+     "shell.execute_reply": "2026-02-02T13:03:23.134869Z"
+    },
+    "papermill": {
+     "duration": 1.835817,
+     "end_time": "2026-02-02T13:03:23.137563",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:21.301746",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from customer_retention.analysis.notebook_progress import track_and_export_previous\n",
+    "track_and_export_previous(\"05_multi_dataset.ipynb\")\n",
+    "\n",
+    "from customer_retention.analysis.auto_explorer import (\n",
+    "    ExplorationManager,\n",
+    "    MultiDatasetFindings,\n",
+    "    ExplorationFindings,\n",
+    "    RecommendationRegistry,\n",
+    ")\n",
+    "from customer_retention.stages.profiling import (\n",
+    "    RelationshipDetector,\n",
+    "    TimeWindowAggregator,\n",
+    "    RelationshipType,\n",
+    "    SegmentAnalyzer,\n",
+    "    SegmentationMethod,\n",
+    "    FeatureCapacityAnalyzer,\n",
+    "    TemporalFeatureEngineer,\n",
+    "    TemporalAggregationConfig,\n",
+    "    ReferenceMode,\n",
+    "    FeatureGroup,\n",
+    "    DimensionReductionMethod,\n",
+    ")\n",
+    "from customer_retention.analysis.visualization import ChartBuilder, display_figure, display_table\n",
+    "from customer_retention.core.config.column_config import DatasetGranularity, ColumnType\n",
+    "from pathlib import Path\n",
+    "import yaml\n",
+    "import pandas as pd\n",
+    "import numpy as np\n",
+    "import plotly.graph_objects as go\n",
+    "from plotly.subplots import make_subplots\n",
+    "from customer_retention.core.config.experiments import FINDINGS_DIR, EXPERIMENTS_DIR, OUTPUT_DIR, setup_experiments_structure\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "eda144a0",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T13:03:23.143847Z",
+     "iopub.status.busy": "2026-02-02T13:03:23.143614Z",
+     "iopub.status.idle": "2026-02-02T13:03:23.149654Z",
+     "shell.execute_reply": "2026-02-02T13:03:23.148371Z"
+    },
+    "papermill": {
+     "duration": 0.010775,
+     "end_time": "2026-02-02T13:03:23.150723",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.139948",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# === CONFIGURATION ===\n",
+    "# FINDINGS_DIR imported from customer_retention.core.config.experiments\n",
+    "\n",
+    "# Initialize the exploration manager\n",
+    "manager = ExplorationManager(explorations_dir=FINDINGS_DIR)\n",
+    "\n",
+    "# Discover all explored datasets (prefers aggregated over event-level when both exist)\n",
+    "findings_files = manager.discover_findings(prefer_aggregated=True)\n",
+    "skipped_files = manager.get_skipped_event_findings()\n",
+    "\n",
+    "print(f\"Found {len(findings_files)} explored dataset(s):\\n\")\n",
+    "for f in findings_files:\n",
+    "    print(f\"  - {f.name}\")\n",
+    "\n",
+    "if skipped_files:\n",
+    "    print(f\"\\n⏭️ Skipped {len(skipped_files)} event-level findings (using aggregated versions instead):\")\n",
+    "    for f in skipped_files:\n",
+    "        print(f\"  - {f.name}\")\n",
+    "\n",
+    "# Load or initialize recommendations registry\n",
+    "RECOMMENDATIONS_PATH = FINDINGS_DIR / \"recommendations.yaml\"\n",
+    "if RECOMMENDATIONS_PATH.exists():\n",
+    "    with open(RECOMMENDATIONS_PATH, \"r\") as f:\n",
+    "        registry = RecommendationRegistry.from_dict(yaml.safe_load(f))\n",
+    "    print(f\"\\nLoaded existing recommendations: {len(registry.all_recommendations)} total\")\n",
+    "else:\n",
+    "    registry = RecommendationRegistry()\n",
+    "    print(\"\\nInitialized new recommendation registry\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a4d325cd",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T13:03:23.173572Z",
+     "iopub.status.busy": "2026-02-02T13:03:23.173134Z",
+     "iopub.status.idle": "2026-02-02T13:03:23.392956Z",
+     "shell.execute_reply": "2026-02-02T13:03:23.392264Z"
+    },
+    "papermill": {
+     "duration": 0.230614,
+     "end_time": "2026-02-02T13:03:23.393602",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.162988",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# List datasets with details\n",
+    "datasets = manager.list_datasets()\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*70)\n",
+    "print(\"DISCOVERED DATASETS\")\n",
+    "print(\"=\"*70 + \"\\n\")\n",
+    "\n",
+    "for ds in datasets:\n",
+    "    granularity_emoji = \"\\U0001f4ca\" if ds.granularity == DatasetGranularity.ENTITY_LEVEL else \"\\U0001f4c8\"\n",
+    "    target_info = f\" [TARGET: {ds.target_column}]\" if ds.target_column else \"\"\n",
+    "    \n",
+    "    print(f\"{granularity_emoji} {ds.name}\")\n",
+    "    print(f\"   Granularity: {ds.granularity.value}\")\n",
+    "    print(f\"   Rows: {ds.row_count:,} | Columns: {ds.column_count}\")\n",
+    "    if ds.entity_column:\n",
+    "        print(f\"   Entity: {ds.entity_column} | Time: {ds.time_column}\")\n",
+    "    print(f\"   Source: {ds.source_path}{target_info}\")\n",
+    "    print()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "23f1c0fd",
+   "metadata": {
+    "papermill": {
+     "duration": 0.002275,
+     "end_time": "2026-02-02T13:03:23.398411",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.396136",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 5.2 Multi-Dataset Dashboard"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "465a97e7",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T13:03:23.403897Z",
+     "iopub.status.busy": "2026-02-02T13:03:23.403766Z",
+     "iopub.status.idle": "2026-02-02T13:03:23.601533Z",
+     "shell.execute_reply": "2026-02-02T13:03:23.600794Z"
+    },
+    "papermill": {
+     "duration": 0.201877,
+     "end_time": "2026-02-02T13:03:23.602549",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.400672",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Create multi-dataset findings and visual dashboard\n",
+    "multi = manager.create_multi_dataset_findings()\n",
+    "\n",
+    "if len(datasets) > 0:\n",
+    "    # Prepare data for visualization\n",
+    "    names = [ds.name for ds in datasets]\n",
+    "    rows = [ds.row_count for ds in datasets]\n",
+    "    cols = [ds.column_count for ds in datasets]\n",
+    "    granularities = [\"Entity\" if ds.granularity == DatasetGranularity.ENTITY_LEVEL else \"Event\" \n",
+    "                     for ds in datasets]\n",
+    "    colors = [\"#2ecc71\" if ds.granularity == DatasetGranularity.ENTITY_LEVEL else \"#3498db\"\n",
+    "              for ds in datasets]\n",
+    "\n",
+    "    # Create dashboard with metrics column + horizontal bar charts\n",
+    "    fig = make_subplots(\n",
+    "        rows=2, cols=2,\n",
+    "        column_widths=[0.35, 0.65],\n",
+    "        row_heights=[0.5, 0.5],\n",
+    "        specs=[[{\"type\": \"xy\", \"rowspan\": 2}, {\"type\": \"bar\"}],\n",
+    "               [None, {\"type\": \"bar\"}]],\n",
+    "        subplot_titles=(\"\", \"Row Counts by Dataset\", \"Column Counts by Dataset\"),\n",
+    "        horizontal_spacing=0.12,\n",
+    "        vertical_spacing=0.15\n",
+    "    )\n",
+    "\n",
+    "    # Left panel: invisible placeholder for metrics\n",
+    "    fig.add_trace(go.Scatter(x=[0], y=[0], mode='markers', marker=dict(opacity=0), showlegend=False), row=1, col=1)\n",
+    "\n",
+    "    # Top right: Horizontal bar chart for row counts (names readable on y-axis)\n",
+    "    fig.add_trace(\n",
+    "        go.Bar(y=names, x=rows, orientation='h', marker_color=colors, name=\"Rows\",\n",
+    "               text=[f\"{r:,}\" for r in rows], textposition=\"auto\",\n",
+    "               hovertemplate=\"%{y}: %{x:,} rows<extra></extra>\"),\n",
+    "        row=1, col=2\n",
+    "    )\n",
+    "\n",
+    "    # Bottom right: Horizontal bar chart for column counts\n",
+    "    fig.add_trace(\n",
+    "        go.Bar(y=names, x=cols, orientation='h', marker_color=colors, name=\"Columns\",\n",
+    "               text=cols, textposition=\"auto\",\n",
+    "               hovertemplate=\"%{y}: %{x} columns<extra></extra>\"),\n",
+    "        row=2, col=2\n",
+    "    )\n",
+    "\n",
+    "    # Build metrics text for left panel (expandable list format)\n",
+    "    annotations = []\n",
+    "    y_pos = 0.98\n",
+    "\n",
+    "    # Total Datasets (label + value)\n",
+    "    annotations.append(dict(x=0.01, y=y_pos, xref=\"paper\", yref=\"paper\",\n",
+    "        text=\"<b>Total Datasets</b>\", showarrow=False, font=dict(size=11, color=\"#666\"), xanchor=\"left\"))\n",
+    "    y_pos -= 0.06\n",
+    "    annotations.append(dict(x=0.01, y=y_pos, xref=\"paper\", yref=\"paper\",\n",
+    "        text=f\"<b>{len(multi.datasets)}</b>\", showarrow=False, font=dict(size=18, color=\"#2c3e50\"), xanchor=\"left\"))\n",
+    "    y_pos -= 0.10\n",
+    "\n",
+    "    # Primary Entity Dataset\n",
+    "    primary_name = multi.primary_entity_dataset or \"Not detected\"\n",
+    "    primary_color = \"#27ae60\" if multi.primary_entity_dataset else \"#e74c3c\"\n",
+    "    annotations.append(dict(x=0.01, y=y_pos, xref=\"paper\", yref=\"paper\",\n",
+    "        text=\"<b>Primary Entity</b>\", showarrow=False, font=dict(size=11, color=\"#666\"), xanchor=\"left\"))\n",
+    "    y_pos -= 0.06\n",
+    "    annotations.append(dict(x=0.01, y=y_pos, xref=\"paper\", yref=\"paper\",\n",
+    "        text=f\"<span style='color:{primary_color}'>{primary_name}</span>\", \n",
+    "        showarrow=False, font=dict(size=12), xanchor=\"left\"))\n",
+    "    y_pos -= 0.10\n",
+    "\n",
+    "    # Event Datasets (expandable list - one per row)\n",
+    "    annotations.append(dict(x=0.01, y=y_pos, xref=\"paper\", yref=\"paper\",\n",
+    "        text=\"<b>Event Datasets</b>\", showarrow=False, font=dict(size=11, color=\"#666\"), xanchor=\"left\"))\n",
+    "    y_pos -= 0.06\n",
+    "    \n",
+    "    if multi.event_datasets:\n",
+    "        # Show each event dataset on its own line (supports 20+ datasets)\n",
+    "        max_display = min(len(multi.event_datasets), 8)  # Show up to 8, then summarize\n",
+    "        for i, event_name in enumerate(multi.event_datasets[:max_display]):\n",
+    "            annotations.append(dict(x=0.03, y=y_pos, xref=\"paper\", yref=\"paper\",\n",
+    "                text=f\"• {event_name}\", showarrow=False, font=dict(size=10, color=\"#3498db\"), xanchor=\"left\"))\n",
+    "            y_pos -= 0.045\n",
+    "        \n",
+    "        if len(multi.event_datasets) > max_display:\n",
+    "            remaining = len(multi.event_datasets) - max_display\n",
+    "            annotations.append(dict(x=0.03, y=y_pos, xref=\"paper\", yref=\"paper\",\n",
+    "                text=f\"... +{remaining} more\", showarrow=False, font=dict(size=10, color=\"#888\"), xanchor=\"left\"))\n",
+    "            y_pos -= 0.045\n",
+    "    else:\n",
+    "        annotations.append(dict(x=0.03, y=y_pos, xref=\"paper\", yref=\"paper\",\n",
+    "            text=\"None\", showarrow=False, font=dict(size=10, color=\"#888\"), xanchor=\"left\"))\n",
+    "        y_pos -= 0.045\n",
+    "\n",
+    "    # Aggregation Windows at bottom\n",
+    "    y_pos = max(y_pos - 0.05, 0.02)\n",
+    "    windows_str = \", \".join(multi.aggregation_windows[:4])\n",
+    "    if len(multi.aggregation_windows) > 4:\n",
+    "        windows_str += \"...\"\n",
+    "    annotations.append(dict(x=0.01, y=y_pos, xref=\"paper\", yref=\"paper\",\n",
+    "        text=f\"<b>Windows:</b> {windows_str}\", showarrow=False, font=dict(size=9, color=\"#888\"), xanchor=\"left\"))\n",
+    "\n",
+    "    fig.update_layout(\n",
+    "        title=\"Multi-Dataset Overview\",\n",
+    "        height=500,\n",
+    "        showlegend=False,\n",
+    "        template=\"plotly_white\",\n",
+    "        annotations=annotations\n",
+    "    )\n",
+    "\n",
+    "    # Hide axes on left panel\n",
+    "    fig.update_xaxes(visible=False, row=1, col=1)\n",
+    "    fig.update_yaxes(visible=False, row=1, col=1)\n",
+    "    \n",
+    "    # Configure horizontal bar axes\n",
+    "    fig.update_yaxes(categoryorder='total ascending', row=1, col=2)\n",
+    "    fig.update_yaxes(categoryorder='total ascending', row=2, col=2)\n",
+    "\n",
+    "    display_figure(fig)\n",
+    "\n",
+    "    # Legend for colors\n",
+    "    print(\"Legend: 🟢 Entity-level (one row per entity)  🔵 Event-level (multiple rows per entity)\")\n",
+    "else:\n",
+    "    print(\"No datasets found. Run notebooks 01a-01d first to explore your data.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "60022f5a",
+   "metadata": {
+    "papermill": {
+     "duration": 0.004026,
+     "end_time": "2026-02-02T13:03:23.611009",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.606983",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 5.3 Dataset Selection (Optional Override)\n",
+    "\n",
+    "By default, all discovered datasets are included. To analyze only specific datasets, provide their names below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "68d408f6",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T13:03:23.619976Z",
+     "iopub.status.busy": "2026-02-02T13:03:23.619857Z",
+     "iopub.status.idle": "2026-02-02T13:03:23.622891Z",
+     "shell.execute_reply": "2026-02-02T13:03:23.622459Z"
+    },
+    "papermill": {
+     "duration": 0.008084,
+     "end_time": "2026-02-02T13:03:23.623501",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.615417",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# === DATASET SELECTION (Optional) ===\n",
+    "# Set to None to use all discovered datasets (default)\n",
+    "# Or provide a list of dataset names to include only those\n",
+    "DATASET_NAMES = ['customer_retention_retail']  # e.g., [\"customers\", \"transactions\", \"emails\"]\n",
+    "\n",
+    "if DATASET_NAMES:\n",
+    "    # Filter to only specified datasets\n",
+    "    available_names = [ds.name for ds in datasets]\n",
+    "    valid_names = [name for name in DATASET_NAMES if name in available_names]\n",
+    "    invalid_names = [name for name in DATASET_NAMES if name not in available_names]\n",
+    "\n",
+    "    if invalid_names:\n",
+    "        print(f\"⚠️ Datasets not found: {invalid_names}\")\n",
+    "        print(f\"   Available: {available_names}\")\n",
+    "\n",
+    "    if valid_names:\n",
+    "        # Recreate multi-dataset findings with only selected datasets\n",
+    "        multi = manager.create_multi_dataset_findings(dataset_names=valid_names)\n",
+    "        print(f\"✓ Using {len(valid_names)} selected dataset(s): {valid_names}\")\n",
+    "    else:\n",
+    "        print(\"⚠️ No valid datasets specified. Using all discovered datasets.\")\n",
+    "else:\n",
+    "    print(f\"Using all {len(datasets)} discovered dataset(s)\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9af4a83a",
+   "metadata": {
+    "papermill": {
+     "duration": 0.005501,
+     "end_time": "2026-02-02T13:03:23.633770",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.628269",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 5.4 Define Relationships Between Datasets\n",
+    "\n",
+    "Relationships define how datasets connect. For each event dataset, specify:\n",
+    "- Which entity dataset it relates to\n",
+    "- Which columns form the join key\n",
+    "- The relationship type (one-to-many for event data)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "231f9f79",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T13:03:23.643403Z",
+     "iopub.status.busy": "2026-02-02T13:03:23.643275Z",
+     "iopub.status.idle": "2026-02-02T13:03:23.646721Z",
+     "shell.execute_reply": "2026-02-02T13:03:23.646004Z"
+    },
+    "papermill": {
+     "duration": 0.009505,
+     "end_time": "2026-02-02T13:03:23.647552",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.638047",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Try to auto-detect relationships using sample data\n",
+    "print(\"\\n\" + \"=\"*70)\n",
+    "print(\"RELATIONSHIP DETECTION\")\n",
+    "print(\"=\"*70 + \"\\n\")\n",
+    "\n",
+    "detector = RelationshipDetector()\n",
+    "\n",
+    "# If we have a primary entity dataset and event datasets, try to detect relationships\n",
+    "if multi.primary_entity_dataset and multi.event_datasets:\n",
+    "    primary_info = multi.datasets[multi.primary_entity_dataset]\n",
+    "    \n",
+    "    print(f\"Primary dataset: {multi.primary_entity_dataset}\")\n",
+    "    print(f\"Checking relationships with event datasets...\\n\")\n",
+    "    \n",
+    "    for event_name in multi.event_datasets:\n",
+    "        event_info = multi.datasets[event_name]\n",
+    "        \n",
+    "        # Check if they share common column names\n",
+    "        if event_info.entity_column:\n",
+    "            print(f\"\\U0001f517 {multi.primary_entity_dataset} <-> {event_name}\")\n",
+    "            print(f\"   Potential join column: {event_info.entity_column}\")\n",
+    "            print(f\"   Expected relationship: one_to_many\")\n",
+    "            print()\n",
+    "else:\n",
+    "    print(\"Not enough datasets to detect relationships.\")\n",
+    "    print(\"Need at least one entity-level and one event-level dataset.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f739bec7",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T13:03:23.656470Z",
+     "iopub.status.busy": "2026-02-02T13:03:23.656355Z",
+     "iopub.status.idle": "2026-02-02T13:03:23.659921Z",
+     "shell.execute_reply": "2026-02-02T13:03:23.659317Z"
+    },
+    "papermill": {
+     "duration": 0.009088,
+     "end_time": "2026-02-02T13:03:23.660820",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.651732",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# === MANUAL RELATIONSHIP DEFINITION ===\n",
+    "# Define relationships between your datasets\n",
+    "# Uncomment and modify as needed\n",
+    "\n",
+    "# Example: Link transactions to customers\n",
+    "# multi.add_relationship(\n",
+    "#     left_dataset=\"customers\",\n",
+    "#     right_dataset=\"transactions\",\n",
+    "#     left_column=\"customer_id\",\n",
+    "#     right_column=\"customer_id\",\n",
+    "#     relationship_type=\"one_to_many\"\n",
+    "# )\n",
+    "\n",
+    "# Example: Link emails to customers\n",
+    "# multi.add_relationship(\n",
+    "#     left_dataset=\"customers\",\n",
+    "#     right_dataset=\"emails\",\n",
+    "#     left_column=\"customer_id\",\n",
+    "#     right_column=\"customer_id\",\n",
+    "#     relationship_type=\"one_to_many\"\n",
+    "# )\n",
+    "\n",
+    "print(f\"Defined relationships: {len(multi.relationships)}\")\n",
+    "for rel in multi.relationships:\n",
+    "    print(f\"   {rel.left_dataset}.{rel.left_column} -> {rel.right_dataset}.{rel.right_column} ({rel.relationship_type})\")\n",
+    "\n",
+    "# Initialize silver layer if not already done\n",
+    "if registry.silver is None:\n",
+    "    entity_col = multi.datasets[multi.primary_entity_dataset].entity_column if multi.primary_entity_dataset else \"entity_id\"\n",
+    "    registry.init_silver(entity_col)\n",
+    "\n",
+    "# Persist join recommendations to registry\n",
+    "for rel in multi.relationships:\n",
+    "    registry.add_silver_join(\n",
+    "        left_source=rel.left_dataset,\n",
+    "        right_source=rel.right_dataset,\n",
+    "        join_keys=[rel.left_column],\n",
+    "        join_type=rel.relationship_type,\n",
+    "        rationale=f\"Join {rel.left_dataset} with {rel.right_dataset} on {rel.left_column}\",\n",
+    "        source_notebook=\"05_multi_dataset\"\n",
+    "    )\n",
+    "\n",
+    "if multi.relationships:\n",
+    "    print(f\"\\n✅ Persisted {len(multi.relationships)} join recommendations to registry\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ad203984",
+   "metadata": {
+    "papermill": {
+     "duration": 0.004387,
+     "end_time": "2026-02-02T13:03:23.671779",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.667392",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 5.5 Plan Temporal Feature Engineering\n",
+    "\n",
+    "For event datasets, we engineer sophisticated temporal features using **per-customer alignment**:\n",
+    "- Each customer's features are computed relative to their reference date (churn date or last activity)\n",
+    "- This makes historical churners comparable to current active customers\n",
+    "\n",
+    "**Feature Groups Available:**\n",
+    "\n",
+    "| Group | Features | Purpose |\n",
+    "|-------|----------|---------|\n",
+    "| **Lagged Windows** | `lag0_{metric}_{agg}`, `lag1_{metric}_{agg}`, ... | Sequential non-overlapping time windows |\n",
+    "| **Velocity** | `{metric}_velocity`, `{metric}_velocity_pct` | Rate of change between windows |\n",
+    "| **Acceleration** | `{metric}_acceleration`, `{metric}_momentum` | Change in velocity, weighted direction |\n",
+    "| **Lifecycle** | `{metric}_beginning`, `{metric}_middle`, `{metric}_end` | Beginning/middle/end of customer history |\n",
+    "| **Recency** | `days_since_last_event`, `active_span_days` | How recently customer was active |\n",
+    "| **Regularity** | `event_frequency`, `regularity_score` | Consistency of engagement |\n",
+    "| **Cohort Comparison** | `{metric}_cohort_zscore` | Customer vs peer group |"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1d005aff",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T13:03:23.683740Z",
+     "iopub.status.busy": "2026-02-02T13:03:23.683600Z",
+     "iopub.status.idle": "2026-02-02T13:03:23.687775Z",
+     "shell.execute_reply": "2026-02-02T13:03:23.687218Z"
+    },
+    "papermill": {
+     "duration": 0.012099,
+     "end_time": "2026-02-02T13:03:23.688972",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.676873",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Get aggregation plan for event datasets\n",
+    "agg_plan = multi.get_aggregation_plan()\n",
+    "\n",
+    "print(\"\\n\" + \"=\"*70)\n",
+    "print(\"AGGREGATION PLAN\")\n",
+    "print(\"=\"*70 + \"\\n\")\n",
+    "\n",
+    "if agg_plan:\n",
+    "    for dataset_name, plan in agg_plan.items():\n",
+    "        print(f\"\\U0001f4ca {dataset_name}\")\n",
+    "        print(f\"   Entity column: {plan.entity_column}\")\n",
+    "        print(f\"   Time column: {plan.time_column}\")\n",
+    "        print(f\"   Windows: {plan.windows}\")\n",
+    "        print(f\"   Default agg funcs: {plan.agg_funcs}\")\n",
+    "        print()\n",
+    "else:\n",
+    "    print(\"No event datasets to aggregate.\")\n",
+    "\n",
+    "# Show available feature groups\n",
+    "print(\"\\n\" + \"=\"*70)\n",
+    "print(\"TEMPORAL FEATURE GROUPS\")\n",
+    "print(\"=\"*70 + \"\\n\")\n",
+    "\n",
+    "for group in FeatureGroup:\n",
+    "    enabled = \"✓\" if group in [FeatureGroup.LAGGED_WINDOWS, FeatureGroup.VELOCITY, \n",
+    "                               FeatureGroup.RECENCY, FeatureGroup.REGULARITY] else \"○\"\n",
+    "    print(f\"  {enabled} {group.value}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "89dee3cc",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T13:03:23.701821Z",
+     "iopub.status.busy": "2026-02-02T13:03:23.701603Z",
+     "iopub.status.idle": "2026-02-02T13:03:23.707747Z",
+     "shell.execute_reply": "2026-02-02T13:03:23.706719Z"
+    },
+    "papermill": {
+     "duration": 0.01362,
+     "end_time": "2026-02-02T13:03:23.708473",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.694853",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# === CONFIGURE TEMPORAL FEATURES ===\n",
+    "\n",
+    "# Reference mode: PER_CUSTOMER aligns to each customer's reference date\n",
+    "# This is critical for churn models where customers churned at different times\n",
+    "REFERENCE_MODE = ReferenceMode.PER_CUSTOMER\n",
+    "\n",
+    "# Lagged window configuration\n",
+    "LAG_WINDOW_DAYS = 30      # Each lag window spans this many days\n",
+    "NUM_LAGS = 4              # Number of sequential windows (lag0, lag1, lag2, lag3)\n",
+    "LAG_AGGREGATIONS = [\"sum\", \"mean\", \"count\", \"max\"]  # Aggregations per window\n",
+    "\n",
+    "# Feature groups to compute\n",
+    "FEATURE_GROUPS = [\n",
+    "    FeatureGroup.LAGGED_WINDOWS,   # lag0_amount_sum, lag1_amount_sum, ...\n",
+    "    FeatureGroup.VELOCITY,          # amount_velocity (rate of change)\n",
+    "    FeatureGroup.ACCELERATION,      # amount_acceleration, amount_momentum\n",
+    "    FeatureGroup.LIFECYCLE,         # amount_beginning, amount_middle, amount_end\n",
+    "    FeatureGroup.RECENCY,           # days_since_last_event, active_span_days\n",
+    "    FeatureGroup.REGULARITY,        # event_frequency, regularity_score\n",
+    "    FeatureGroup.COHORT_COMPARISON, # amount_cohort_zscore\n",
+    "]\n",
+    "\n",
+    "# Lifecycle configuration\n",
+    "MIN_HISTORY_DAYS = 60  # Minimum days of history for lifecycle features (else NaN)\n",
+    "\n",
+    "# Create configuration\n",
+    "temporal_config = TemporalAggregationConfig(\n",
+    "    reference_mode=REFERENCE_MODE,\n",
+    "    lag_window_days=LAG_WINDOW_DAYS,\n",
+    "    num_lags=NUM_LAGS,\n",
+    "    lag_aggregations=LAG_AGGREGATIONS,\n",
+    "    compute_velocity=FeatureGroup.VELOCITY in FEATURE_GROUPS,\n",
+    "    compute_acceleration=FeatureGroup.ACCELERATION in FEATURE_GROUPS,\n",
+    "    compute_lifecycle=FeatureGroup.LIFECYCLE in FEATURE_GROUPS,\n",
+    "    min_history_days=MIN_HISTORY_DAYS,\n",
+    "    compute_recency=FeatureGroup.RECENCY in FEATURE_GROUPS,\n",
+    "    compute_regularity=FeatureGroup.REGULARITY in FEATURE_GROUPS,\n",
+    "    compute_cohort=FeatureGroup.COHORT_COMPARISON in FEATURE_GROUPS,\n",
+    ")\n",
+    "\n",
+    "# Store in multi-dataset findings\n",
+    "multi.notes['temporal_config'] = {\n",
+    "    'reference_mode': REFERENCE_MODE.value,\n",
+    "    'lag_window_days': LAG_WINDOW_DAYS,\n",
+    "    'num_lags': NUM_LAGS,\n",
+    "    'lag_aggregations': LAG_AGGREGATIONS,\n",
+    "    'feature_groups': [g.value for g in FEATURE_GROUPS],\n",
+    "    'min_history_days': MIN_HISTORY_DAYS,\n",
+    "}\n",
+    "\n",
+    "# Persist temporal configuration to registry for each event dataset\n",
+    "for dataset_name in multi.event_datasets:\n",
+    "    ds_info = multi.datasets[dataset_name]\n",
+    "    findings = manager.load_findings(dataset_name)\n",
+    "    \n",
+    "    if findings:\n",
+    "        numeric_cols = [\n",
+    "            name for name, col in findings.columns.items()\n",
+    "            if col.inferred_type in [ColumnType.NUMERIC_CONTINUOUS, ColumnType.NUMERIC_DISCRETE]\n",
+    "            and name not in [ds_info.entity_column, ds_info.time_column] and name not in TEMPORAL_METADATA_COLS\n",
+    "        ]\n",
+    "        \n",
+    "        if numeric_cols:\n",
+    "            registry.add_silver_temporal_config(\n",
+    "                source_dataset=dataset_name,\n",
+    "                columns=numeric_cols,\n",
+    "                lag_windows=NUM_LAGS,\n",
+    "                lag_window_days=LAG_WINDOW_DAYS,\n",
+    "                aggregations=LAG_AGGREGATIONS,\n",
+    "                feature_groups=[g.value for g in FEATURE_GROUPS],\n",
+    "                rationale=f\"Temporal features for {dataset_name} with {len(numeric_cols)} columns\",\n",
+    "                source_notebook=\"05_multi_dataset\"\n",
+    "            )\n",
+    "\n",
+    "print(\"Temporal Feature Configuration:\")\n",
+    "print(f\"   Reference Mode: {REFERENCE_MODE.value}\")\n",
+    "print(f\"   Lag Windows: {NUM_LAGS} x {LAG_WINDOW_DAYS} days\")\n",
+    "print(f\"   Aggregations: {LAG_AGGREGATIONS}\")\n",
+    "print(f\"   Feature Groups: {len(FEATURE_GROUPS)} enabled\")\n",
+    "print()\n",
+    "print(\"\\U0001f4a1 With per-customer alignment, all customers are measured from their reference point.\")\n",
+    "print(\"   - Churned customers: reference = churn date\")\n",
+    "print(\"   - Active customers: reference = last activity or analysis date\")\n",
+    "\n",
+    "if multi.event_datasets:\n",
+    "    print(f\"\\n✅ Persisted temporal config for {len(multi.event_datasets)} event dataset(s)\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4c87b0ec",
+   "metadata": {
+    "papermill": {
+     "duration": 0.00386,
+     "end_time": "2026-02-02T13:03:23.716688",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.712828",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 5.6 Preview Feature Set\n",
+    "\n",
+    "Preview the features that will be created from time-window aggregations."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "aa9ce4ca",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T13:03:23.726130Z",
+     "iopub.status.busy": "2026-02-02T13:03:23.725975Z",
+     "iopub.status.idle": "2026-02-02T13:03:23.732128Z",
+     "shell.execute_reply": "2026-02-02T13:03:23.731518Z"
+    },
+    "papermill": {
+     "duration": 0.011605,
+     "end_time": "2026-02-02T13:03:23.732641",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.721036",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# For each event dataset, preview what features could be created\n",
+    "print(\"\\n\" + \"=\"*70)\n",
+    "print(\"TEMPORAL FEATURES PREVIEW\")\n",
+    "print(\"=\"*70 + \"\\n\")\n",
+    "\n",
+    "for dataset_name in multi.event_datasets:\n",
+    "    ds_info = multi.datasets[dataset_name]\n",
+    "    \n",
+    "    print(f\"\\U0001f4c8 From {dataset_name}:\")\n",
+    "    print()\n",
+    "    \n",
+    "    # Load findings to see numeric columns\n",
+    "    findings = manager.load_findings(dataset_name)\n",
+    "    \n",
+    "    # Find numeric columns that could be aggregated\n",
+    "    numeric_cols = []\n",
+    "    if findings:\n",
+    "        numeric_cols = [\n",
+    "            name for name, col in findings.columns.items()\n",
+    "            if col.inferred_type in [ColumnType.NUMERIC_CONTINUOUS, ColumnType.NUMERIC_DISCRETE]\n",
+    "            and name not in [ds_info.entity_column, ds_info.time_column] and name not in TEMPORAL_METADATA_COLS\n",
+    "        ]\n",
+    "    \n",
+    "    # Group 1: Lagged Window Features\n",
+    "    if FeatureGroup.LAGGED_WINDOWS in FEATURE_GROUPS:\n",
+    "        print(\"   📊 LAGGED WINDOWS (Group 1):\")\n",
+    "        for col in numeric_cols[:2]:\n",
+    "            features = [f\"lag{i}_{col}_{agg}\" for i in range(NUM_LAGS) for agg in LAG_AGGREGATIONS[:2]]\n",
+    "            print(f\"      {col}: {features[:4]}...\")\n",
+    "        print(f\"      Total: {len(numeric_cols)} cols × {NUM_LAGS} lags × {len(LAG_AGGREGATIONS)} aggs\")\n",
+    "    \n",
+    "    # Group 2: Velocity Features\n",
+    "    if FeatureGroup.VELOCITY in FEATURE_GROUPS:\n",
+    "        print(\"\\n   🚀 VELOCITY (Group 2):\")\n",
+    "        for col in numeric_cols[:2]:\n",
+    "            print(f\"      - {col}_velocity, {col}_velocity_pct\")\n",
+    "        print(f\"      Total: {len(numeric_cols)} cols × 2 features\")\n",
+    "    \n",
+    "    # Group 3: Acceleration Features\n",
+    "    if FeatureGroup.ACCELERATION in FEATURE_GROUPS:\n",
+    "        print(\"\\n   ⚡ ACCELERATION (Group 3):\")\n",
+    "        for col in numeric_cols[:2]:\n",
+    "            print(f\"      - {col}_acceleration, {col}_momentum\")\n",
+    "        print(f\"      Total: {len(numeric_cols)} cols × 2 features\")\n",
+    "    \n",
+    "    # Group 4: Lifecycle Features\n",
+    "    if FeatureGroup.LIFECYCLE in FEATURE_GROUPS:\n",
+    "        print(\"\\n   📈 LIFECYCLE (Group 4):\")\n",
+    "        for col in numeric_cols[:2]:\n",
+    "            print(f\"      - {col}_beginning, {col}_middle, {col}_end, {col}_trend_ratio\")\n",
+    "        print(f\"      Total: {len(numeric_cols)} cols × 4 features\")\n",
+    "        print(f\"      ℹ️ Requires {MIN_HISTORY_DAYS}+ days of history (else NaN)\")\n",
+    "    \n",
+    "    # Group 5: Recency Features\n",
+    "    if FeatureGroup.RECENCY in FEATURE_GROUPS:\n",
+    "        print(\"\\n   ⏱️ RECENCY (Group 5):\")\n",
+    "        print(\"      - days_since_last_event\")\n",
+    "        print(\"      - days_since_first_event\")\n",
+    "        print(\"      - active_span_days\")\n",
+    "        print(\"      - recency_ratio\")\n",
+    "    \n",
+    "    # Group 6: Regularity Features\n",
+    "    if FeatureGroup.REGULARITY in FEATURE_GROUPS:\n",
+    "        print(\"\\n   🎯 REGULARITY (Group 6):\")\n",
+    "        print(\"      - event_frequency\")\n",
+    "        print(\"      - inter_event_gap_mean\")\n",
+    "        print(\"      - inter_event_gap_std\")\n",
+    "        print(\"      - regularity_score\")\n",
+    "    \n",
+    "    # Group 7: Cohort Comparison\n",
+    "    if FeatureGroup.COHORT_COMPARISON in FEATURE_GROUPS:\n",
+    "        print(\"\\n   👥 COHORT COMPARISON (Group 7):\")\n",
+    "        for col in numeric_cols[:2]:\n",
+    "            print(f\"      - {col}_vs_cohort_mean, {col}_vs_cohort_pct, {col}_cohort_zscore\")\n",
+    "        print(f\"      Total: {len(numeric_cols)} cols × 3 features\")\n",
+    "    \n",
+    "    # Summary\n",
+    "    total_features = 0\n",
+    "    if FeatureGroup.LAGGED_WINDOWS in FEATURE_GROUPS:\n",
+    "        total_features += len(numeric_cols) * NUM_LAGS * len(LAG_AGGREGATIONS)\n",
+    "    if FeatureGroup.VELOCITY in FEATURE_GROUPS:\n",
+    "        total_features += len(numeric_cols) * 2\n",
+    "    if FeatureGroup.ACCELERATION in FEATURE_GROUPS:\n",
+    "        total_features += len(numeric_cols) * 2\n",
+    "    if FeatureGroup.LIFECYCLE in FEATURE_GROUPS:\n",
+    "        total_features += len(numeric_cols) * 4\n",
+    "    if FeatureGroup.RECENCY in FEATURE_GROUPS:\n",
+    "        total_features += 4\n",
+    "    if FeatureGroup.REGULARITY in FEATURE_GROUPS:\n",
+    "        total_features += 4\n",
+    "    if FeatureGroup.COHORT_COMPARISON in FEATURE_GROUPS:\n",
+    "        total_features += len(numeric_cols) * 3\n",
+    "    \n",
+    "    print(f\"\\n   📝 TOTAL ESTIMATED FEATURES: ~{total_features}\")\n",
+    "    print()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1e8b9248",
+   "metadata": {
+    "papermill": {
+     "duration": 0.004047,
+     "end_time": "2026-02-02T13:03:23.741097",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.737050",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 5.7 Segmentation Analysis\n",
+    "\n",
+    "Should we build **separate models per customer segment** or a **single unified model**? This analysis provides evidence-based metrics to make that decision.\n",
+    "\n",
+    "**Key Decision Metrics:**\n",
+    "\n",
+    "| Metric | What It Measures | Good Value |\n",
+    "|--------|------------------|------------|\n",
+    "| **Silhouette Score** | Cluster cohesion (how tight) & separation (how different) | > 0.25 |\n",
+    "| **Target Variance** | How much target rates differ across segments | > 0.15 |\n",
+    "| **Segment Balance** | Size distribution across segments | > 0.3 ratio |\n",
+    "| **EPV per Segment** | Events-per-variable for reliable modeling | > 10 |\n",
+    "\n",
+    "**Interpretation Guide:**\n",
+    "- **Silhouette > 0.5**: Strong natural clustering - segments are distinct\n",
+    "- **Silhouette 0.25-0.5**: Reasonable structure - segments somewhat distinct  \n",
+    "- **Silhouette < 0.25**: Weak structure - data is relatively homogeneous\n",
+    "- **Silhouette < 0**: Overlapping clusters - segmentation not supported"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2f826435",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T13:03:23.750855Z",
+     "iopub.status.busy": "2026-02-02T13:03:23.750732Z",
+     "iopub.status.idle": "2026-02-02T13:03:23.763154Z",
+     "shell.execute_reply": "2026-02-02T13:03:23.762680Z"
+    },
+    "papermill": {
+     "duration": 0.017748,
+     "end_time": "2026-02-02T13:03:23.763611",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.745863",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Segmentation Analysis on Primary Entity Dataset\n",
+    "from customer_retention.stages.temporal import load_data_with_snapshot_preference\n",
+    "\n",
+    "print(\"=\" * 70)\n",
+    "print(\"SEGMENTATION ANALYSIS\")\n",
+    "print(\"=\" * 70)\n",
+    "\n",
+    "segment_analyzer = SegmentAnalyzer()\n",
+    "capacity_analyzer = FeatureCapacityAnalyzer()\n",
+    "\n",
+    "# Consistent color palette: Segment 0=blue, 1=red, 2=green, 3=purple, etc.\n",
+    "SEGMENT_COLORS = {\n",
+    "    0: '#3498db',  # Blue\n",
+    "    1: '#e74c3c',  # Red\n",
+    "    2: '#2ecc71',  # Green\n",
+    "    3: '#9b59b6',  # Purple\n",
+    "    4: '#f39c12',  # Orange\n",
+    "    5: '#1abc9c',  # Teal\n",
+    "    6: '#e67e22',  # Dark Orange\n",
+    "}\n",
+    "\n",
+    "if multi.primary_entity_dataset:\n",
+    "    primary_info = multi.datasets[multi.primary_entity_dataset]\n",
+    "    primary_findings = manager.load_findings(multi.primary_entity_dataset)\n",
+    "    \n",
+    "    if primary_findings:\n",
+    "        # Load the primary dataset from snapshot (not source) to get correct column names\n",
+    "        primary_df, data_source = load_data_with_snapshot_preference(primary_findings, output_dir=str(FINDINGS_DIR))\n",
+    "        print(f\"   Loaded from: {data_source}\")\n",
+    "        \n",
+    "        # Get numeric features for clustering (exclude temporal metadata)\n",
+    "        from customer_retention.stages.temporal import TEMPORAL_METADATA_COLS\n",
+    "        numeric_features = [\n",
+    "            name for name, col in primary_findings.columns.items()\n",
+    "            if col.inferred_type in [ColumnType.NUMERIC_CONTINUOUS, ColumnType.NUMERIC_DISCRETE]\n",
+    "            and name != primary_info.target_column\n",
+    "            and name not in TEMPORAL_METADATA_COLS\n",
+    "        ]\n",
+    "        \n",
+    "        print(f\"\\n📊 Dataset: {multi.primary_entity_dataset}\")\n",
+    "        print(f\"   Total Samples: {len(primary_df):,}\")\n",
+    "        print(f\"   Numeric Features: {len(numeric_features)}\")\n",
+    "        print(f\"   Target Column: {primary_info.target_column}\")\n",
+    "        \n",
+    "        # Run full segmentation analysis using framework\n",
+    "        analysis = segment_analyzer.run_full_analysis(\n",
+    "            primary_df,\n",
+    "            feature_cols=numeric_features,\n",
+    "            target_col=primary_info.target_column,\n",
+    "            max_segments=5,\n",
+    "            dim_reduction=DimensionReductionMethod.PCA,\n",
+    "        )\n",
+    "        m = analysis.metrics  # Shorthand for metrics\n",
+    "        \n",
+    "        # ============================================================\n",
+    "        # KEY DECISION METRICS\n",
+    "        # ============================================================\n",
+    "        print(\"\\n\" + \"=\" * 70)\n",
+    "        print(\"📊 CLUSTERING DECISION METRICS\")\n",
+    "        print(\"=\" * 70)\n",
+    "        \n",
+    "        print(f\"\"\"\n",
+    "┌─────────────────────────────────────────────────────────────────────┐\n",
+    "│  METRIC                          │  VALUE      │  INTERPRETATION    │\n",
+    "├──────────────────────────────────┼─────────────┼────────────────────┤\n",
+    "│  Silhouette Score (cohesion)     │  {m.silhouette_score:+.3f}      │  {m.silhouette_interpretation:<18} │\n",
+    "│  Target Rate Variance            │  {f'{m.target_variance_ratio:.3f}' if m.target_variance_ratio else 'N/A':>11} │  {m.target_variance_interpretation:<18} │\n",
+    "│  Optimal Segments Found          │  {m.n_segments}           │  {m.segments_interpretation:<18} │\n",
+    "│  Overall Confidence              │  {m.confidence:.0%}         │  {m.confidence_interpretation:<18} │\n",
+    "└──────────────────────────────────┴─────────────┴────────────────────┘\"\"\")\n",
+    "        \n",
+    "        print(f\"\\n🎯 RECOMMENDATION: {m.recommendation.upper().replace('_', ' ')}\")\n",
+    "        print(f\"\\n📋 Supporting Evidence:\")\n",
+    "        for r in m.rationale:\n",
+    "            print(f\"   • {r}\")\n",
+    "        \n",
+    "        # ============================================================\n",
+    "        # SEGMENT PROFILES\n",
+    "        # ============================================================\n",
+    "        print(\"\\n\" + \"=\" * 70)\n",
+    "        print(\"📊 SEGMENT PROFILES\")\n",
+    "        print(\"=\" * 70 + \"\\n\")\n",
+    "        \n",
+    "        segment_data = [{\n",
+    "            \"Segment\": f\"Segment {p.segment_id}\",\n",
+    "            \"N (count)\": f\"{p.size:,}\",\n",
+    "            \"% of Total\": f\"{p.size_pct:.1f}%\",\n",
+    "            \"Target Rate\": f\"{p.target_rate:.1%}\" if p.target_rate is not None else \"N/A\",\n",
+    "            \"Viable for ML\": \"✓\" if p.size >= 100 else \"⚠️\"\n",
+    "        } for p in analysis.profiles]\n",
+    "        display(pd.DataFrame(segment_data))\n",
+    "        \n",
+    "        sd = analysis.size_distribution\n",
+    "        print(f\"\\n📊 Size Distribution:\")\n",
+    "        print(f\"   Total datapoints: {sd['total']:,}\")\n",
+    "        print(f\"   Smallest segment: {sd['min_size']:,} ({sd['min_pct']:.1f}%)\")\n",
+    "        print(f\"   Largest segment: {sd['max_size']:,} ({sd['max_pct']:.1f}%)\")\n",
+    "        print(f\"   Balance ratio: {sd['balance_ratio']:.2f} (1.0 = perfectly balanced)\")\n",
+    "        \n",
+    "        # ============================================================\n",
+    "        # CLUSTER VISUALIZATION\n",
+    "        # ============================================================\n",
+    "        if analysis.has_visualization:\n",
+    "            viz = analysis.visualization\n",
+    "            seg_result = analysis.segmentation_result\n",
+    "            \n",
+    "            fig = make_subplots(\n",
+    "                rows=1, cols=3,\n",
+    "                subplot_titles=(\n",
+    "                    f\"Cluster Visualization (PCA, {viz.explained_variance_ratio:.0%} var)\" \n",
+    "                    if viz.explained_variance_ratio else \"Cluster Visualization (PCA)\",\n",
+    "                    \"Segment Sizes\", \"Target Rate\"\n",
+    "                ),\n",
+    "                horizontal_spacing=0.12,\n",
+    "                column_widths=[0.4, 0.3, 0.3]\n",
+    "            )\n",
+    "            \n",
+    "            unique_labels = sorted(set(seg_result.labels[seg_result.labels >= 0]))\n",
+    "            \n",
+    "            # Scatter plot - consistent colors by segment ID\n",
+    "            for label in unique_labels:\n",
+    "                mask = seg_result.labels == label\n",
+    "                color = SEGMENT_COLORS.get(label, '#888888')\n",
+    "                profile = next((p for p in analysis.profiles if p.segment_id == label), None)\n",
+    "                name = f\"Seg {label} (n={profile.size:,})\" if profile else f\"Seg {label}\"\n",
+    "                fig.add_trace(go.Scatter(\n",
+    "                    x=viz.x[mask], y=viz.y[mask], mode='markers',\n",
+    "                    marker=dict(color=color, size=6, opacity=0.6),\n",
+    "                    name=name, hovertemplate=f\"{name}<br>PC1: %{{x:.2f}}<br>PC2: %{{y:.2f}}<extra></extra>\"\n",
+    "                ), row=1, col=1)\n",
+    "            \n",
+    "            # Short labels for bar charts (avoid overlap)\n",
+    "            bar_labels = [f\"Seg {p.segment_id}\" for p in analysis.profiles]\n",
+    "            sizes = [p.size for p in analysis.profiles]\n",
+    "            bar_colors = [SEGMENT_COLORS.get(p.segment_id, '#888888') for p in analysis.profiles]\n",
+    "            \n",
+    "            # Size bars - numbers inside\n",
+    "            fig.add_trace(go.Bar(\n",
+    "                y=bar_labels, x=sizes, orientation='h',\n",
+    "                marker_color=bar_colors,\n",
+    "                text=[f\"{s:,}\" for s in sizes],\n",
+    "                textposition='inside', textfont=dict(color='white'),\n",
+    "                showlegend=False,\n",
+    "                hovertemplate=\"Segment %{y}<br>Count: %{x:,}<extra></extra>\"\n",
+    "            ), row=1, col=2)\n",
+    "            \n",
+    "            # Target rates - consistent segment colors, numbers inside\n",
+    "            if all(p.target_rate is not None for p in analysis.profiles):\n",
+    "                rates = [p.target_rate * 100 for p in analysis.profiles]\n",
+    "                fig.add_trace(go.Bar(\n",
+    "                    y=bar_labels, x=rates, orientation='h',\n",
+    "                    marker_color=bar_colors,  # Same colors as size chart\n",
+    "                    text=[f\"{r:.1f}%\" for r in rates],\n",
+    "                    textposition='inside', textfont=dict(color='white'),\n",
+    "                    showlegend=False,\n",
+    "                    hovertemplate=\"Segment %{y}<br>Target: %{x:.1f}%<extra></extra>\"\n",
+    "                ), row=1, col=3)\n",
+    "                overall = sum(p.target_rate * p.size for p in analysis.profiles) / sd['total'] * 100\n",
+    "                fig.add_vline(x=overall, line_dash=\"dash\", line_color=\"#2c3e50\",\n",
+    "                             annotation_text=f\"Avg: {overall:.1f}%\", annotation_position=\"top\", row=1, col=3)\n",
+    "            \n",
+    "            fig.update_layout(\n",
+    "                title=\"Segment Analysis Overview\", \n",
+    "                height=400, \n",
+    "                template=\"plotly_white\",\n",
+    "                legend=dict(\n",
+    "                    orientation=\"h\", \n",
+    "                    yanchor=\"top\", \n",
+    "                    y=-0.15,\n",
+    "                    xanchor=\"center\", \n",
+    "                    x=0.5\n",
+    "                ),\n",
+    "                margin=dict(r=20, b=80)\n",
+    "            )\n",
+    "            fig.update_xaxes(title_text=\"PC1\", row=1, col=1)\n",
+    "            fig.update_yaxes(title_text=\"PC2\", row=1, col=1)\n",
+    "            display_figure(fig)\n",
+    "            \n",
+    "            print(f\"\\n📈 CLUSTER VISUALIZATION:\")\n",
+    "            print(f\"   Method: PCA | Variance Explained: {viz.explained_variance_ratio:.1%}\" if viz.explained_variance_ratio else \"   Method: PCA\")\n",
+    "            print(f\"   Colors: Seg 0=Blue, Seg 1=Red, Seg 2=Green, Seg 3=Purple\")\n",
+    "        \n",
+    "        # ============================================================\n",
+    "        # EPV CAPACITY ANALYSIS\n",
+    "        # ============================================================\n",
+    "        if m.n_segments > 1 and primary_info.target_column:\n",
+    "            print(\"\\n\" + \"=\" * 70)\n",
+    "            print(\"💡 SEGMENT CAPACITY ANALYSIS (EPV Check)\")\n",
+    "            print(\"=\" * 70)\n",
+    "            \n",
+    "            primary_df['_segment'] = analysis.segmentation_result.labels\n",
+    "            capacity = capacity_analyzer.analyze_segment_capacity(\n",
+    "                primary_df[primary_df['_segment'] >= 0],\n",
+    "                feature_cols=numeric_features,\n",
+    "                target_col=primary_info.target_column,\n",
+    "                segment_col='_segment'\n",
+    "            )\n",
+    "            primary_df.drop('_segment', axis=1, inplace=True)\n",
+    "            \n",
+    "            print(f\"\\n🎯 Strategy: {capacity.recommended_strategy.upper()}\")\n",
+    "            print(f\"   Reason: {capacity.strategy_reason}\")\n",
+    "            if capacity.viable_segments:\n",
+    "                print(f\"\\n   ✅ Viable segments: {capacity.viable_segments}\")\n",
+    "            if capacity.insufficient_segments:\n",
+    "                print(f\"   ⚠️ Insufficient segments: {capacity.insufficient_segments}\")\n",
+    "            \n",
+    "            # Store in findings\n",
+    "            multi.notes.update({\n",
+    "                'segmentation_recommendation': m.recommendation,\n",
+    "                'segmentation_confidence': m.confidence,\n",
+    "                'segmentation_silhouette': m.silhouette_score,\n",
+    "                'segment_count': m.n_segments,\n",
+    "                'segment_strategy': capacity.recommended_strategy,\n",
+    "                'segment_sizes': {f\"segment_{p.segment_id}\": p.size for p in analysis.profiles}\n",
+    "            })\n",
+    "            \n",
+    "            # Initialize bronze layer if not already done\n",
+    "            if registry.bronze is None:\n",
+    "                registry.init_bronze(primary_info.source_path)\n",
+    "            \n",
+    "            # Persist segmentation strategy to registry\n",
+    "            registry.add_bronze_segmentation_strategy(\n",
+    "                strategy=m.recommendation,\n",
+    "                confidence=m.confidence,\n",
+    "                n_segments=m.n_segments,\n",
+    "                silhouette_score=m.silhouette_score,\n",
+    "                rationale=\"; \".join(m.rationale[:3]),\n",
+    "                source_notebook=\"05_multi_dataset\"\n",
+    "            )\n",
+    "            print(f\"\\n✅ Persisted segmentation strategy to registry: {m.recommendation}\")\n",
+    "        \n",
+    "        # ============================================================\n",
+    "        # DECISION SUMMARY\n",
+    "        # ============================================================\n",
+    "        print(\"\\n\" + \"=\" * 70)\n",
+    "        print(\"📋 SEGMENTATION DECISION SUMMARY\")\n",
+    "        print(\"=\" * 70)\n",
+    "        print(f\"\\n{analysis.get_decision_summary()}\")\n",
+    "\n",
+    "else:\n",
+    "    print(\"\\n⚠️ No primary entity dataset detected.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f3809b84",
+   "metadata": {
+    "papermill": {
+     "duration": 0.004145,
+     "end_time": "2026-02-02T13:03:23.771952",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.767807",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 5.8 Relationship Diagram"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "14dda2db",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T13:03:23.780518Z",
+     "iopub.status.busy": "2026-02-02T13:03:23.780420Z",
+     "iopub.status.idle": "2026-02-02T13:03:23.783657Z",
+     "shell.execute_reply": "2026-02-02T13:03:23.783152Z"
+    },
+    "papermill": {
+     "duration": 0.008447,
+     "end_time": "2026-02-02T13:03:23.784142",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.775695",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Create a simple relationship diagram\n",
+    "if len(multi.datasets) > 1:\n",
+    "    print(\"\\n\" + \"=\"*70)\n",
+    "    print(\"DATASET RELATIONSHIP DIAGRAM\")\n",
+    "    print(\"=\"*70 + \"\\n\")\n",
+    "    \n",
+    "    # ASCII diagram\n",
+    "    if multi.primary_entity_dataset:\n",
+    "        primary = multi.primary_entity_dataset\n",
+    "        primary_info = multi.datasets[primary]\n",
+    "        \n",
+    "        print(f\"   +{'='*30}+\")\n",
+    "        print(f\"   |  {primary:^26}  |  <- PRIMARY (has target)\")\n",
+    "        print(f\"   |  {primary_info.row_count:,} rows{' '*15}  |\")\n",
+    "        if primary_info.target_column:\n",
+    "            print(f\"   |  Target: {primary_info.target_column:<17}  |\")\n",
+    "        print(f\"   +{'='*30}+\")\n",
+    "        \n",
+    "        for event_name in multi.event_datasets:\n",
+    "            event_info = multi.datasets[event_name]\n",
+    "            join_col = event_info.entity_column or \"?\"\n",
+    "            \n",
+    "            print(f\"          |\")\n",
+    "            print(f\"          | {join_col}\")\n",
+    "            print(f\"          v\")\n",
+    "            print(f\"   +{'-'*30}+\")\n",
+    "            print(f\"   |  {event_name:^26}  |  <- EVENT LEVEL\")\n",
+    "            print(f\"   |  {event_info.row_count:,} rows{' '*15}  |\")\n",
+    "            print(f\"   |  Time: {event_info.time_column or '?':<19}  |\")\n",
+    "            print(f\"   +{'-'*30}+\")\n",
+    "else:\n",
+    "    print(\"Single dataset - no relationships to diagram.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7d599131",
+   "metadata": {
+    "papermill": {
+     "duration": 0.003691,
+     "end_time": "2026-02-02T13:03:23.792016",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.788325",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 5.9 Save Multi-Dataset Findings"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cfbb90c4",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T13:03:23.801511Z",
+     "iopub.status.busy": "2026-02-02T13:03:23.801385Z",
+     "iopub.status.idle": "2026-02-02T13:03:23.805961Z",
+     "shell.execute_reply": "2026-02-02T13:03:23.805444Z"
+    },
+    "papermill": {
+     "duration": 0.009851,
+     "end_time": "2026-02-02T13:03:23.806470",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.796619",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Save the multi-dataset findings and recommendations registry\n",
+    "MULTI_FINDINGS_PATH = FINDINGS_DIR / \"multi_dataset_findings.yaml\"\n",
+    "\n",
+    "multi.save(str(MULTI_FINDINGS_PATH))\n",
+    "\n",
+    "# Save the recommendations registry\n",
+    "registry.save(RECOMMENDATIONS_PATH)\n",
+    "\n",
+    "print(f\"\\n✅ Multi-dataset findings saved to: {MULTI_FINDINGS_PATH}\")\n",
+    "print(f\"\\n   Contents:\")\n",
+    "print(f\"   - {len(multi.datasets)} datasets\")\n",
+    "print(f\"   - {len(multi.relationships)} relationships\")\n",
+    "print(f\"   - {len(multi.event_datasets)} event datasets to aggregate\")\n",
+    "print(f\"   - Aggregation windows: {multi.aggregation_windows}\")\n",
+    "\n",
+    "print(f\"\\n✅ Recommendations registry saved: {RECOMMENDATIONS_PATH}\")\n",
+    "print(f\"   Total recommendations: {len(registry.all_recommendations)}\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7b895a1b",
+   "metadata": {
+    "papermill": {
+     "duration": 0.005196,
+     "end_time": "2026-02-02T13:03:23.816347",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.811151",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "---\n",
+    "\n",
+    "## Summary: What We Learned\n",
+    "\n",
+    "In this notebook, we:\n",
+    "\n",
+    "1. **Discovered Datasets** - Found all exploration findings from previous notebooks\n",
+    "2. **Visualized Overview** - Dashboard showing dataset sizes and structure\n",
+    "3. **Selected Datasets** - Optionally filtered to specific datasets\n",
+    "4. **Defined Relationships** - Established how datasets connect via keys\n",
+    "5. **Planned Temporal Features** - Configured 7 feature groups (lagged windows, velocity, acceleration, lifecycle, recency, regularity, cohort comparison)\n",
+    "6. **Previewed Features** - Saw what features will be created\n",
+    "7. **Analyzed Segmentation** - Determined if segmented modeling is justified\n",
+    "8. **Saved Configuration** - Saved multi-dataset findings for feature engineering\n",
+    "\n",
+    "## Key Decisions Made\n",
+    "\n",
+    "| Decision | Value | Rationale |\n",
+    "|----------|-------|-----------|\n",
+    "| Primary Entity Dataset | From `multi.primary_entity_dataset` | Has target column |\n",
+    "| Event Datasets | From `multi.event_datasets` | Event-level data to aggregate |\n",
+    "| Reference Mode | Per-customer alignment | Makes historical churners comparable to active customers |\n",
+    "| Lag Windows | `NUM_LAGS` x `LAG_WINDOW_DAYS` days | Captures recent vs historical patterns |\n",
+    "| Feature Groups | Lagged, Velocity, Acceleration, Lifecycle, Recency, Regularity, Cohort | Comprehensive temporal characterization |\n",
+    "| Segmentation Strategy | From `multi.notes['segment_strategy']` | Based on EPV analysis |\n",
+    "\n",
+    "---\n",
+    "\n",
+    "## Next Steps\n",
+    "\n",
+    "Continue to **06_feature_opportunities.ipynb** to:\n",
+    "- Deep dive into feature engineering opportunities\n",
+    "- Analyze feature capacity constraints\n",
+    "- Create derived features\n",
+    "\n",
+    "**Important:** The multi-dataset findings file (`multi_dataset_findings.yaml`) includes temporal feature configuration for use in subsequent notebooks."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b37da519",
+   "metadata": {
+    "papermill": {
+     "duration": 0.004561,
+     "end_time": "2026-02-02T13:03:23.841911",
+     "exception": false,
+     "start_time": "2026-02-02T13:03:23.837350",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "> **Save Reminder:** Save this notebook (Ctrl+S / Cmd+S) before running the next one.\n",
+    "> The next notebook will automatically export this notebook's HTML documentation from the saved file."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.4"
+  },
+  "papermill": {
+   "default_parameters": {},
+   "duration": 5.931034,
+   "end_time": "2026-02-02T13:03:26.462964",
+   "environment_variables": {},
+   "exception": null,
+   "input_path": "/Users/Vital/python/CustomerRetention/exploration_notebooks/05_multi_dataset.ipynb",
+   "output_path": "/Users/Vital/python/CustomerRetention/exploration_notebooks/05_multi_dataset.ipynb",
+   "parameters": {},
+   "start_time": "2026-02-02T13:03:20.531930",
+   "version": "2.6.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}