churnkit 0.75.0a1__py3-none-any.whl

churnkit-0.75.0a1.data/data/share/churnkit/exploration_notebooks/02a_text_columns_deep_dive.ipynb

@@ -0,0 +1,854 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "cell-0",
+   "metadata": {
+    "papermill": {
+     "duration": 0.008146,
+     "end_time": "2026-02-02T01:47:23.371490",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:23.363344",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "# Chapter 2a: Text Columns Deep Dive\n",
+    "\n",
+    "**Purpose:** Transform TEXT columns (tickets, emails, messages) into numeric features using embeddings and dimensionality reduction.\n",
+    "\n",
+    "**When to use this notebook:**\n",
+    "- Your dataset contains TEXT columns (unstructured text data)\n",
+    "- Detected automatically if ColumnType.TEXT found in findings\n",
+    "\n",
+    "**What you'll learn:**\n",
+    "- How text embeddings capture semantic meaning\n",
+    "- Why PCA reduces dimensions while preserving variance\n",
+    "- How to choose between fast vs high-quality embedding models\n",
+    "\n",
+    "**Outputs:**\n",
+    "- PC features (text_pc1, text_pc2, ...) for each TEXT column\n",
+    "- TextProcessingMetadata in findings\n",
+    "- Recommendations for production pipeline\n",
+    "\n",
+    "---\n",
+    "\n",
+    "## Two Approaches to Text Feature Engineering\n",
+    "\n",
+    "| Approach | Method | When to Use |\n",
+    "|----------|--------|-------------|\n",
+    "| **1. Embeddings + PCA** (This notebook) | Sentence-transformers → PCA | General semantic features |\n",
+    "| **2. LLM Labeling** (Future) | LLM on samples → Train classifier | Specific categories needed |\n",
+    "\n",
+    "### Approach 1: Embeddings + Dimensionality Reduction (Current)\n",
+    "\n",
+    "```\n",
+    "TEXT Column → Embeddings → PCA → pc1, pc2, ..., pcN\n",
+    "```\n",
+    "\n",
+    "- **Embeddings**: Dense vectors capturing semantic meaning (similar texts = similar vectors)\n",
+    "- **PCA**: Reduces dimensions to N components covering target variance (default 95%)\n",
+    "- **Output**: Numeric features usable with standard ML models\n",
+    "\n",
+    "### Embedding Model Options\n",
+    "\n",
+    "| Model | Size | Embedding Dim | Speed | Quality | Best For |\n",
+    "|-------|------|---------------|-------|---------|----------|\n",
+    "| **MiniLM** (default) | 90 MB | 384 | Fast | Good | CPU, quick iteration, small datasets |\n",
+    "| **Qwen3-0.6B** | 1.2 GB | 1024 | Medium | Better | GPU available, production quality |\n",
+    "| **Qwen3-4B** | 8 GB | 2560 | Slow | High | 16GB+ GPU, multilingual, high accuracy |\n",
+    "| **Qwen3-8B** | 16 GB | 4096 | Slowest | Highest | 32GB+ GPU, research, max quality |\n",
+    "\n",
+    "**Note:** Models are downloaded on first use (lazy loading). Qwen3 models require GPU for reasonable performance.\n",
+    "\n",
+    "### Approach 2: LLM Labeling (Future Enhancement)\n",
+    "\n",
+    "```\n",
+    "TEXT Column → Sample → LLM Labels → Train Classifier → Apply to All\n",
+    "```\n",
+    "\n",
+    "- Use when you need specific categorical labels (sentiment, topic, intent)\n",
+    "- More expensive but more interpretable"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-1",
+   "metadata": {
+    "papermill": {
+     "duration": 0.005848,
+     "end_time": "2026-02-02T01:47:23.384304",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:23.378456",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 2a.1 Load Previous Findings"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-2",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T01:47:23.396604Z",
+     "iopub.status.busy": "2026-02-02T01:47:23.396464Z",
+     "iopub.status.idle": "2026-02-02T01:47:25.750830Z",
+     "shell.execute_reply": "2026-02-02T01:47:25.750340Z"
+    },
+    "papermill": {
+     "duration": 2.361354,
+     "end_time": "2026-02-02T01:47:25.751648",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:23.390294",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from customer_retention.analysis.notebook_progress import track_and_export_previous\n",
+    "track_and_export_previous(\"02a_text_columns_deep_dive.ipynb\")\n",
+    "\n",
+    "from customer_retention.analysis.auto_explorer import ExplorationFindings, TextProcessingMetadata\n",
+    "from customer_retention.analysis.visualization import ChartBuilder, display_figure, display_table, console\n",
+    "from customer_retention.core.config.column_config import ColumnType\n",
+    "from customer_retention.stages.profiling import (\n",
+    "    TextColumnProcessor, TextProcessingConfig, TextColumnResult,\n",
+    "    TextEmbedder, TextDimensionalityReducer,\n",
+    "    EMBEDDING_MODELS, get_model_info, list_available_models\n",
+    ")\n",
+    "import pandas as pd\n",
+    "import numpy as np\n",
+    "import plotly.graph_objects as go\n",
+    "import plotly.express as px\n",
+    "from plotly.subplots import make_subplots\n",
+    "from customer_retention.core.config.experiments import FINDINGS_DIR, EXPERIMENTS_DIR, OUTPUT_DIR, setup_experiments_structure"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-3",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T01:47:25.761974Z",
+     "iopub.status.busy": "2026-02-02T01:47:25.761777Z",
+     "iopub.status.idle": "2026-02-02T01:47:25.956580Z",
+     "shell.execute_reply": "2026-02-02T01:47:25.955962Z"
+    },
+    "papermill": {
+     "duration": 0.20067,
+     "end_time": "2026-02-02T01:47:25.957216",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:25.756546",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# === CONFIGURATION ===\n",
+    "from pathlib import Path\n",
+    "\n",
+    "# FINDINGS_DIR imported from customer_retention.core.config.experiments\n",
+    "\n",
+    "findings_files = [f for f in FINDINGS_DIR.glob(\"*_findings.yaml\") if \"multi_dataset\" not in f.name]\n",
+    "if not findings_files:\n",
+    "    raise FileNotFoundError(f\"No findings files found in {FINDINGS_DIR}. Run notebook 01 first.\")\n",
+    "\n",
+    "findings_files.sort(key=lambda f: f.stat().st_mtime, reverse=True)\n",
+    "FINDINGS_PATH = str(findings_files[0])\n",
+    "\n",
+    "print(f\"Found {len(findings_files)} findings file(s)\")\n",
+    "print(f\"Using: {FINDINGS_PATH}\")\n",
+    "\n",
+    "findings = ExplorationFindings.load(FINDINGS_PATH)\n",
+    "print(f\"\\nLoaded findings for {findings.column_count} columns from {findings.source_path}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-4",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T01:47:25.967854Z",
+     "iopub.status.busy": "2026-02-02T01:47:25.967561Z",
+     "iopub.status.idle": "2026-02-02T01:47:25.970428Z",
+     "shell.execute_reply": "2026-02-02T01:47:25.969940Z"
+    },
+    "papermill": {
+     "duration": 0.009088,
+     "end_time": "2026-02-02T01:47:25.971120",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:25.962032",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Identify TEXT columns\n",
+    "text_columns = [\n",
+    "    name for name, col in findings.columns.items()\n",
+    "    if col.inferred_type == ColumnType.TEXT\n",
+    "]\n",
+    "\n",
+    "if not text_columns:\n",
+    "    print(\"\\u26a0\\ufe0f No TEXT columns detected in this dataset.\")\n",
+    "    print(\"   This notebook is only needed when TEXT columns are present.\")\n",
+    "    print(\"   Continue to notebook 03_quality_assessment.ipynb\")\n",
+    "else:\n",
+    "    print(f\"\\u2705 Found {len(text_columns)} TEXT column(s):\")\n",
+    "    for col in text_columns:\n",
+    "        col_info = findings.columns[col]\n",
+    "        print(f\"   - {col} (Confidence: {col_info.confidence:.0%})\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-5",
+   "metadata": {
+    "papermill": {
+     "duration": 0.004835,
+     "end_time": "2026-02-02T01:47:25.980918",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:25.976083",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 2a.2 Load Source Data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-6",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T01:47:25.991285Z",
+     "iopub.status.busy": "2026-02-02T01:47:25.991167Z",
+     "iopub.status.idle": "2026-02-02T01:47:26.241129Z",
+     "shell.execute_reply": "2026-02-02T01:47:26.240166Z"
+    },
+    "papermill": {
+     "duration": 0.259568,
+     "end_time": "2026-02-02T01:47:26.245361",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:25.985793",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from customer_retention.stages.temporal import load_data_with_snapshot_preference, TEMPORAL_METADATA_COLS\n",
+    "\n",
+    "df, data_source = load_data_with_snapshot_preference(findings, output_dir=str(FINDINGS_DIR))\n",
+    "charts = ChartBuilder()\n",
+    "\n",
+    "print(f\"Loaded {len(df):,} rows x {len(df.columns)} columns\")\n",
+    "print(f\"Data source: {data_source}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-7",
+   "metadata": {
+    "papermill": {
+     "duration": 0.005765,
+     "end_time": "2026-02-02T01:47:26.256952",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:26.251187",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 2a.3 Configuration\n",
+    "\n",
+    "### Available Embedding Models\n",
+    "\n",
+    "Run the cell below to see available models and their specifications. Then configure your choice."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-8",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T01:47:26.269516Z",
+     "iopub.status.busy": "2026-02-02T01:47:26.269372Z",
+     "iopub.status.idle": "2026-02-02T01:47:26.273517Z",
+     "shell.execute_reply": "2026-02-02T01:47:26.272670Z"
+    },
+    "papermill": {
+     "duration": 0.011181,
+     "end_time": "2026-02-02T01:47:26.274532",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:26.263351",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Display available embedding models\n",
+    "print(\"Available Embedding Models\")\n",
+    "print(\"=\" * 80)\n",
+    "print(f\"{'Preset':<15} {'Model':<35} {'Size':<10} {'Dim':<8} {'GPU?'}\")\n",
+    "print(\"-\" * 80)\n",
+    "\n",
+    "for preset in list_available_models():\n",
+    "    info = get_model_info(preset)\n",
+    "    size = f\"{info['size_mb']} MB\" if info['size_mb'] < 1000 else f\"{info['size_mb']/1000:.1f} GB\"\n",
+    "    gpu = \"Yes\" if info['gpu_recommended'] else \"No\"\n",
+    "    print(f\"{preset:<15} {info['model_name']:<35} {size:<10} {info['embedding_dim']:<8} {gpu}\")\n",
+    "    print(f\"                {info['description']}\")\n",
+    "    print()\n",
+    "\n",
+    "print(\"\\nModels are downloaded on first use. Choose based on your hardware and quality needs.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "em1rk2p1n0m",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T01:47:26.288846Z",
+     "iopub.status.busy": "2026-02-02T01:47:26.288653Z",
+     "iopub.status.idle": "2026-02-02T01:47:26.292398Z",
+     "shell.execute_reply": "2026-02-02T01:47:26.291943Z"
+    },
+    "papermill": {
+     "duration": 0.011236,
+     "end_time": "2026-02-02T01:47:26.293108",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:26.281872",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# === TEXT PROCESSING CONFIGURATION ===\n",
+    "# Choose your embedding model preset:\n",
+    "#   \"minilm\"     - Fast, CPU-friendly, good for exploration (default)\n",
+    "#   \"qwen3-0.6b\" - Better quality, needs GPU\n",
+    "#   \"qwen3-4b\"   - High quality, needs 16GB+ GPU\n",
+    "#   \"qwen3-8b\"   - Highest quality, needs 32GB+ GPU\n",
+    "\n",
+    "EMBEDDING_PRESET = \"minilm\"  # Change this to try different models\n",
+    "\n",
+    "# PCA configuration\n",
+    "VARIANCE_THRESHOLD = 0.95  # Keep components explaining 95% of variance\n",
+    "MIN_COMPONENTS = 2         # At least 2 features per text column\n",
+    "MAX_COMPONENTS = None      # No upper limit (set to e.g., 20 to cap)\n",
+    "\n",
+    "# Get model info and create config\n",
+    "model_info = get_model_info(EMBEDDING_PRESET)\n",
+    "config = TextProcessingConfig(\n",
+    "    embedding_model=model_info[\"model_name\"],\n",
+    "    variance_threshold=VARIANCE_THRESHOLD,\n",
+    "    max_components=MAX_COMPONENTS,\n",
+    "    min_components=MIN_COMPONENTS,\n",
+    "    batch_size=32\n",
+    ")\n",
+    "\n",
+    "print(\"Text Processing Configuration\")\n",
+    "print(\"=\" * 50)\n",
+    "print(f\"  Preset: {EMBEDDING_PRESET}\")\n",
+    "print(f\"  Model: {config.embedding_model}\")\n",
+    "print(f\"  Model size: {model_info['size_mb']} MB\")\n",
+    "print(f\"  Embedding dimension: {model_info['embedding_dim']}\")\n",
+    "print(f\"  GPU recommended: {'Yes' if model_info['gpu_recommended'] else 'No'}\")\n",
+    "print()\n",
+    "print(f\"  Variance threshold: {config.variance_threshold:.0%}\")\n",
+    "print(f\"  Min components: {config.min_components}\")\n",
+    "print(f\"  Max components: {config.max_components or 'unlimited'}\")\n",
+    "\n",
+    "if model_info['gpu_recommended']:\n",
+    "    print()\n",
+    "    print(\"Note: This model works best with GPU. Processing may be slow on CPU.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-9",
+   "metadata": {
+    "papermill": {
+     "duration": 0.005323,
+     "end_time": "2026-02-02T01:47:26.304338",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:26.299015",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 2a.4 Text Column Analysis\n",
+    "\n",
+    "Before processing, let's understand each TEXT column."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-10",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T01:47:26.316494Z",
+     "iopub.status.busy": "2026-02-02T01:47:26.316369Z",
+     "iopub.status.idle": "2026-02-02T01:47:26.320318Z",
+     "shell.execute_reply": "2026-02-02T01:47:26.319793Z"
+    },
+    "papermill": {
+     "duration": 0.011045,
+     "end_time": "2026-02-02T01:47:26.321049",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:26.310004",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "if text_columns:\n",
+    "    for col_name in text_columns:\n",
+    "        print(f\"\\n{'='*70}\")\n",
+    "        print(f\"Column: {col_name}\")\n",
+    "        print(f\"{'='*70}\")\n",
+    "        \n",
+    "        text_series = df[col_name].fillna(\"\")\n",
+    "        \n",
+    "        # Basic statistics\n",
+    "        non_empty = (text_series.str.len() > 0).sum()\n",
+    "        avg_length = text_series.str.len().mean()\n",
+    "        max_length = text_series.str.len().max()\n",
+    "        \n",
+    "        print(f\"\\n\\U0001f4ca Statistics:\")\n",
+    "        print(f\"   Total rows: {len(text_series):,}\")\n",
+    "        print(f\"   Non-empty: {non_empty:,} ({non_empty/len(text_series)*100:.1f}%)\")\n",
+    "        print(f\"   Avg length: {avg_length:.0f} characters\")\n",
+    "        print(f\"   Max length: {max_length:,} characters\")\n",
+    "        \n",
+    "        # Sample texts\n",
+    "        print(f\"\\n\\U0001f4dd Sample texts:\")\n",
+    "        samples = text_series[text_series.str.len() > 10].head(3)\n",
+    "        for i, sample in enumerate(samples, 1):\n",
+    "            truncated = sample[:100] + \"...\" if len(sample) > 100 else sample\n",
+    "            print(f\"   {i}. {truncated}\")\n",
+    "        \n",
+    "        # Text length distribution\n",
+    "        lengths = text_series.str.len()\n",
+    "        fig = go.Figure()\n",
+    "        fig.add_trace(go.Histogram(x=lengths[lengths > 0], nbinsx=50,\n",
+    "                                    marker_color='steelblue', opacity=0.7))\n",
+    "        fig.add_vline(x=lengths.median(), line_dash=\"solid\", line_color=\"green\",\n",
+    "                      annotation_text=f\"Median: {lengths.median():.0f}\")\n",
+    "        fig.update_layout(\n",
+    "            title=f\"Text Length Distribution: {col_name}\",\n",
+    "            xaxis_title=\"Character Count\",\n",
+    "            yaxis_title=\"Frequency\",\n",
+    "            template=\"plotly_white\",\n",
+    "            height=350\n",
+    "        )\n",
+    "        display_figure(fig)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-11",
+   "metadata": {
+    "papermill": {
+     "duration": 0.005005,
+     "end_time": "2026-02-02T01:47:26.331524",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:26.326519",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 2a.5 Process Text Columns\n",
+    "\n",
+    "This step:\n",
+    "1. Generates embeddings using sentence-transformers\n",
+    "2. Applies PCA to reduce dimensions\n",
+    "3. Creates PC feature columns"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-12",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T01:47:26.342884Z",
+     "iopub.status.busy": "2026-02-02T01:47:26.342734Z",
+     "iopub.status.idle": "2026-02-02T01:47:26.347577Z",
+     "shell.execute_reply": "2026-02-02T01:47:26.345847Z"
+    },
+    "papermill": {
+     "duration": 0.012844,
+     "end_time": "2026-02-02T01:47:26.349482",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:26.336638",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "if text_columns:\n",
+    "    processor = TextColumnProcessor(config)\n",
+    "    \n",
+    "    print(\"Processing TEXT columns...\")\n",
+    "    print(\"(This may take a moment for large datasets)\\n\")\n",
+    "    \n",
+    "    results = []\n",
+    "    df_processed = df.copy()\n",
+    "    \n",
+    "    for col_name in text_columns:\n",
+    "        print(f\"\\n{'='*70}\")\n",
+    "        print(f\"Processing: {col_name}\")\n",
+    "        print(f\"{'='*70}\")\n",
+    "        \n",
+    "        df_processed, result = processor.process_column(df_processed, col_name)\n",
+    "        results.append(result)\n",
+    "        \n",
+    "        print(f\"\\n\\u2705 Processing complete:\")\n",
+    "        print(f\"   Embedding shape: {result.embeddings_shape}\")\n",
+    "        print(f\"   Components kept: {result.n_components}\")\n",
+    "        print(f\"   Explained variance: {result.explained_variance:.1%}\")\n",
+    "        print(f\"   Features created: {', '.join(result.component_columns)}\")\n",
+    "    \n",
+    "    print(f\"\\n\\n{'='*70}\")\n",
+    "    print(\"PROCESSING SUMMARY\")\n",
+    "    print(f\"{'='*70}\")\n",
+    "    print(f\"\\nOriginal columns: {len(df.columns)}\")\n",
+    "    print(f\"New columns added: {len(df_processed.columns) - len(df.columns)}\")\n",
+    "    print(f\"Total columns: {len(df_processed.columns)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-13",
+   "metadata": {
+    "papermill": {
+     "duration": 0.007613,
+     "end_time": "2026-02-02T01:47:26.363074",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:26.355461",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 2a.6 Visualize Results\n",
+    "\n",
+    "Understanding the PC features created from text embeddings."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-14",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T01:47:26.376406Z",
+     "iopub.status.busy": "2026-02-02T01:47:26.376286Z",
+     "iopub.status.idle": "2026-02-02T01:47:26.380333Z",
+     "shell.execute_reply": "2026-02-02T01:47:26.379711Z"
+    },
+    "papermill": {
+     "duration": 0.011065,
+     "end_time": "2026-02-02T01:47:26.381178",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:26.370113",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "if text_columns and results:\n",
+    "    for result in results:\n",
+    "        print(f\"\\n{'='*70}\")\n",
+    "        print(f\"Results: {result.column_name}\")\n",
+    "        print(f\"{'='*70}\")\n",
+    "        \n",
+    "        # Explained variance per component\n",
+    "        reducer = processor._reducers[result.column_name]\n",
+    "        var_ratios = reducer._pca.explained_variance_ratio_\n",
+    "        cumulative = np.cumsum(var_ratios)\n",
+    "        \n",
+    "        fig = make_subplots(rows=1, cols=2,\n",
+    "                            subplot_titles=(\"Variance per Component\", \"Cumulative Variance\"))\n",
+    "        \n",
+    "        fig.add_trace(go.Bar(\n",
+    "            x=[f\"PC{i+1}\" for i in range(len(var_ratios))],\n",
+    "            y=var_ratios,\n",
+    "            marker_color='steelblue'\n",
+    "        ), row=1, col=1)\n",
+    "        \n",
+    "        fig.add_trace(go.Scatter(\n",
+    "            x=[f\"PC{i+1}\" for i in range(len(cumulative))],\n",
+    "            y=cumulative,\n",
+    "            mode='lines+markers',\n",
+    "            line_color='green'\n",
+    "        ), row=1, col=2)\n",
+    "        \n",
+    "        fig.add_hline(y=config.variance_threshold, line_dash=\"dash\", line_color=\"red\",\n",
+    "                      annotation_text=f\"Target: {config.variance_threshold:.0%}\",\n",
+    "                      row=1, col=2)\n",
+    "        \n",
+    "        fig.update_layout(\n",
+    "            title=f\"PCA Results: {result.column_name}\",\n",
+    "            height=400,\n",
+    "            template=\"plotly_white\",\n",
+    "            showlegend=False\n",
+    "        )\n",
+    "        fig.update_yaxes(title_text=\"Variance Ratio\", row=1, col=1)\n",
+    "        fig.update_yaxes(title_text=\"Cumulative Variance\", row=1, col=2)\n",
+    "        display_figure(fig)\n",
+    "        \n",
+    "        # PC feature distributions\n",
+    "        if len(result.component_columns) >= 2:\n",
+    "            fig = px.scatter(\n",
+    "                df_processed,\n",
+    "                x=result.component_columns[0],\n",
+    "                y=result.component_columns[1],\n",
+    "                title=f\"PC1 vs PC2: {result.column_name}\",\n",
+    "                opacity=0.5\n",
+    "            )\n",
+    "            fig.update_layout(template=\"plotly_white\", height=400)\n",
+    "            display_figure(fig)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-15",
+   "metadata": {
+    "papermill": {
+     "duration": 0.005635,
+     "end_time": "2026-02-02T01:47:26.393531",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:26.387896",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 2a.7 Update Findings with Text Processing Metadata"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-16",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T01:47:26.409118Z",
+     "iopub.status.busy": "2026-02-02T01:47:26.408976Z",
+     "iopub.status.idle": "2026-02-02T01:47:29.401562Z",
+     "shell.execute_reply": "2026-02-02T01:47:29.401101Z"
+    },
+    "papermill": {
+     "duration": 3.000962,
+     "end_time": "2026-02-02T01:47:29.402237",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:26.401275",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "if text_columns and results:\n",
+    "    for result in results:\n",
+    "        metadata = TextProcessingMetadata(\n",
+    "            column_name=result.column_name,\n",
+    "            embedding_model=config.embedding_model,\n",
+    "            embedding_dim=result.embeddings_shape[1],\n",
+    "            n_components=result.n_components,\n",
+    "            explained_variance=result.explained_variance,\n",
+    "            component_columns=result.component_columns,\n",
+    "            variance_threshold_used=config.variance_threshold,\n",
+    "            processing_approach=\"pca\"\n",
+    "        )\n",
+    "        findings.text_processing[result.column_name] = metadata\n",
+    "        \n",
+    "        print(f\"\\u2705 Added metadata for {result.column_name}:\")\n",
+    "        print(f\"   Model: {metadata.embedding_model}\")\n",
+    "        print(f\"   Components: {metadata.n_components}\")\n",
+    "        print(f\"   Explained variance: {metadata.explained_variance:.1%}\")\n",
+    "    \n",
+    "    findings.save(FINDINGS_PATH)\n",
+    "    print(f\"\\nFindings saved to: {FINDINGS_PATH}\")\n",
+    "\n",
+    "from customer_retention.analysis.notebook_html_exporter import export_notebook_html\n",
+    "export_notebook_html(Path(\"02a_text_columns_deep_dive.ipynb\"), EXPERIMENTS_DIR / \"docs\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-17",
+   "metadata": {
+    "papermill": {
+     "duration": 0.006327,
+     "end_time": "2026-02-02T01:47:29.415379",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:29.409052",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## 2a.8 Generate Recommendations"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-18",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-02-02T01:47:29.429066Z",
+     "iopub.status.busy": "2026-02-02T01:47:29.428925Z",
+     "iopub.status.idle": "2026-02-02T01:47:29.432197Z",
+     "shell.execute_reply": "2026-02-02T01:47:29.431402Z"
+    },
+    "papermill": {
+     "duration": 0.011237,
+     "end_time": "2026-02-02T01:47:29.433125",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:29.421888",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "if text_columns and results:\n",
+    "    print(\"\\n\" + \"=\"*70)\n",
+    "    print(\"PRODUCTION RECOMMENDATIONS\")\n",
+    "    print(\"=\"*70)\n",
+    "    \n",
+    "    for result in results:\n",
+    "        print(f\"\\n\\U0001f527 {result.column_name}:\")\n",
+    "        print(f\"   Action: embed_reduce (embeddings + PCA)\")\n",
+    "        print(f\"   Model: {config.embedding_model}\")\n",
+    "        print(f\"   Variance threshold: {config.variance_threshold:.0%}\")\n",
+    "        print(f\"   Expected features: {result.n_components}\")\n",
+    "        print(f\"   Feature names: {', '.join(result.component_columns[:3])}...\")\n",
+    "    \n",
+    "    print(\"\\n\\U0001f4a1 These recommendations will be used by the pipeline generator.\")\n",
+    "    print(\"   The same processing will be applied in production.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-19",
+   "metadata": {
+    "papermill": {
+     "duration": 0.007691,
+     "end_time": "2026-02-02T01:47:29.446230",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:29.438539",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "---\n",
+    "\n",
+    "## Summary\n",
+    "\n",
+    "In this notebook, we:\n",
+    "\n",
+    "1. **Analyzed** TEXT columns for length and content patterns\n",
+    "2. **Generated embeddings** using sentence-transformers\n",
+    "3. **Applied PCA** to reduce dimensions while preserving variance\n",
+    "4. **Created numeric features** (pc1, pc2, ...) for downstream ML\n",
+    "5. **Updated findings** with processing metadata\n",
+    "\n",
+    "## Key Results\n",
+    "\n",
+    "| Column | Components | Explained Variance |\n",
+    "|--------|------------|--------------------|\n",
+    "| (Filled by execution) | | |\n",
+    "\n",
+    "---\n",
+    "\n",
+    "## Next Steps\n",
+    "\n",
+    "Continue to **03_quality_assessment.ipynb** to:\n",
+    "- Analyze duplicate records and value conflicts\n",
+    "- Deep dive into missing value patterns\n",
+    "- Analyze outliers with IQR method\n",
+    "- Get cleaning recommendations"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "g5gchegxiyl",
+   "metadata": {
+    "papermill": {
+     "duration": 0.006287,
+     "end_time": "2026-02-02T01:47:29.459193",
+     "exception": false,
+     "start_time": "2026-02-02T01:47:29.452906",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "> **Save Reminder:** Save this notebook (Ctrl+S / Cmd+S) before running the next one.\n",
+    "> The next notebook will automatically export this notebook's HTML documentation from the saved file."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.4"
+  },
+  "papermill": {
+   "default_parameters": {},
+   "duration": 9.591376,
+   "end_time": "2026-02-02T01:47:32.083387",
+   "environment_variables": {},
+   "exception": null,
+   "input_path": "/Users/Vital/python/CustomerRetention/exploration_notebooks/02a_text_columns_deep_dive.ipynb",
+   "output_path": "/Users/Vital/python/CustomerRetention/exploration_notebooks/02a_text_columns_deep_dive.ipynb",
+   "parameters": {},
+   "start_time": "2026-02-02T01:47:22.492011",
+   "version": "2.6.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}