churnkit 1.2.8a0__tar.gz → 1.2.8a2__tar.gz

{churnkit-1.2.8a0 → churnkit-1.2.8a2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: churnkit
-Version: 1.2.8a0
+Version: 1.2.8a2
 Summary: Structured ML framework for customer churn prediction -- from exploration notebooks to production pipelines, locally or on Databricks.
 Project-URL: Homepage, https://github.com/aladjov/CR
 Project-URL: Documentation, https://github.com/aladjov/CR/wiki

{churnkit-1.2.8a0 → churnkit-1.2.8a2}/exploration_notebooks/10_spec_generation.ipynb

@@ -111,7 +111,7 @@
     "\n",
     "# === Bronze aggregation overrides ===\n",
     "# Per-event-source overrides applied to bronze_event configs in the\n",
-    "# generated pipeline. Default is empty (no overrides) \u2014 every event source\n",
+    "# generated pipeline. Default is empty (no overrides) — every event source\n",
     "# uses its auto-recommended snapshot-mode aggregation. Override per dataset\n",
     "# to opt into per-grid-date mode (which exposes the temporal cancellation\n",
     "# gradient for lifecycle event streams) or to swap windows / value-counts\n",
@@ -131,7 +131,7 @@
     "# rerunning NB08. The on-disk spec in the run namespace is NEVER mutated;\n",
     "# the generator writes a stripped copy into ``output_dir/findings/feature_spec.yaml``\n",
     "# and repoints ``feature_spec_path`` at it for the generated training step.\n",
-    "# Use sparingly \u2014 every entry is a knowingly-accepted exploration/generation\n",
+    "# Use sparingly — every entry is a knowingly-accepted exploration/generation\n",
     "# drift. Threaded through to BOTH PipelineGenerator (Option A) and\n",
     "# DatabricksPipelineGenerator (Option B). Empty list = strict parity.\n",
     "PARITY_IGNORED_FEATURES: list[str] = []\n",
@@ -145,7 +145,7 @@
     "#\n",
     "# When False, the reconciler auto-extends the matching layer's\n",
     "# `DatetimeDerivationConfig.source_columns` and emits a warn-level audit\n",
-    "# trail per dataset. Use sparingly \u2014 every auto-extension is a knowingly-\n",
+    "# trail per dataset. Use sparingly — every auto-extension is a knowingly-\n",
     "# accepted exploration/codegen drift. Threaded through to BOTH\n",
     "# PipelineGenerator (Option A) and DatabricksPipelineGenerator (Option B).\n",
     "# Engagement overrides (if any) live in a downstream @cr:user_code cell\n",
@@ -197,8 +197,8 @@
     "# The resolver below prefers `project_context.original_datasets` written\n",
     "# by NB00 cell `f1eb641c dataset_paths`; only falls back here when that\n",
     "# field is missing (pre-existing runs produced before the field landed).\n",
-    "# To use the fallback, paste your NB00 \u00a7 0.2 `datasets = {...}` literal\n",
-    "# here \u2014 same keys, same raw source strings, no post-mutation temp views.\n",
+    "# To use the fallback, paste your NB00 § 0.2 `datasets = {...}` literal\n",
+    "# here — same keys, same raw source strings, no post-mutation temp views.\n",
     "DATASETS_ORIGINAL_FALLBACK: dict[str, str] = {}\n",
     "\n",
     "# === Landing-step overrides (escape hatch for pre-migration NB00s) ===\n",
@@ -207,7 +207,7 @@
     "# chain in `recommendations.yaml.landing` never received the\n",
     "# `add_landing_filter` / `add_landing_lifecycle_enrichment` calls. The\n",
     "# generated landing then has no filter and no lifecycle enrichment, and\n",
-    "# fails downstream \u2014 typically with `Time column 'event_timestamp' not\n",
+    "# fails downstream — typically with `Time column 'event_timestamp' not\n",
     "# found` because the upstream UC table carries raw transactional\n",
     "# columns and the temp-view's enriched `event_type` / `event_timestamp`\n",
     "# rows never get materialized.\n",
@@ -376,13 +376,13 @@
   {
    "cell_type": "markdown",
    "id": "a52f133e",
-   "source": "[//]: # (cr:doc name='10_2b_production_cohort_scope' id=a52f133e)\n## 10.2b Production Cohort Scope and Sampling\n\nThe cohort-scope **filter** declared in NB00 (`SAMPLE_FILTER_COLUMNS` \u2192\n`project_context.sample_filters`) is replayed verbatim in the generated\nproduction landing scripts. It is a parity contract with exploration \u2014\nnot configurable here.\n\n**Sampling** is the optional entity-count knob that further reduces the\ntraining set for cost reasons. **Default: no sampling** \u2014 production\ntrains on the full filtered cohort with no cross-validation.\n\nThis section previews what production will see: full dataset size,\npost-filter cohort size, and post-sample size when sampling is opted\ninto. Mirrors the cell in NB00 \u00a70.7 so the operator can size the\nsampling knob deliberately at codegen time.",
+   "source": "[//]: # (cr:doc name='10_2b_production_cohort_scope' id=a52f133e)\n## 10.2b Production Cohort Scope and Sampling\n\nThe cohort-scope **filter** declared in NB00 (`SAMPLE_FILTER_COLUMNS` →\n`project_context.sample_filters`) is replayed verbatim in the generated\nproduction landing scripts. It is a parity contract with exploration —\nnot configurable here.\n\n**Sampling** is the optional entity-count knob that further reduces the\ntraining set for cost reasons. **Default: no sampling** — production\ntrains on the full filtered cohort with no cross-validation.\n\nThis section previews what production will see: full dataset size,\npost-filter cohort size, and post-sample size when sampling is opted\ninto. Mirrors the cell in NB00 §0.7 so the operator can size the\nsampling knob deliberately at codegen time.",
    "metadata": {}
   },
   {
    "cell_type": "code",
    "id": "22b37d33",
-   "source": "# @cr:config name='production_sampling_config' id=22b37d33\n# === Production Sampling (opt-in; default = no sampling) ===\nPRODUCTION_SAMPLE_ENTITY_COUNT = None       # cap on entities trained on; None = full filtered cohort\nPRODUCTION_HOLDOUT_FRACTION = None          # opt-in production holdout split; None = no split here\nPRODUCTION_SAMPLE_STRATIFY_COLUMNS = []     # extra stratification columns when sampling is enabled\n\n# Filtering (cohort scope) is sourced from project_context.sample_filters and\n# is NOT configurable in this notebook \u2014 it is the parity contract with\n# exploration (replayed verbatim in the generated landing scripts).",
+   "source": "# @cr:config name='production_sampling_config' id=22b37d33\n# === Production Sampling (opt-in; default = no sampling) ===\nPRODUCTION_SAMPLE_ENTITY_COUNT = None       # cap on entities trained on; None = full filtered cohort\nPRODUCTION_HOLDOUT_FRACTION = None          # opt-in production holdout split; None = no split here\nPRODUCTION_SAMPLE_STRATIFY_COLUMNS = []     # extra stratification columns when sampling is enabled\n\n# Filtering (cohort scope) is sourced from project_context.sample_filters and\n# is NOT configurable in this notebook — it is the parity contract with\n# exploration (replayed verbatim in the generated landing scripts).",
    "metadata": {},
    "execution_count": null,
    "outputs": []
@@ -807,19 +807,19 @@
     "\n",
     "```\n",
     "docs/\n",
-    "\u251c\u2500\u2500 overview.md                        # Project context, objectives, datasets\n",
-    "\u251c\u2500\u2500 config.md                          # CN, temporal settings, entity keys\n",
-    "\u251c\u2500\u2500 landing/\n",
-    "\u2502   \u2514\u2500\u2500 landing_{source}.md            # Per-source: schema, quality, temporal profile\n",
-    "\u251c\u2500\u2500 bronze/\n",
-    "\u2502   \u251c\u2500\u2500 bronze_entity_{source}.md      # Per entity source: cleaning recommendations\n",
-    "\u2502   \u2514\u2500\u2500 bronze_event_{source}.md       # Per event source: cleaning + aggregation\n",
-    "\u251c\u2500\u2500 silver/\n",
-    "\u2502   \u2514\u2500\u2500 silver_featureset_{CN}.md      # Merge strategy, joins, derived columns\n",
-    "\u251c\u2500\u2500 gold/\n",
-    "\u2502   \u2514\u2500\u2500 gold_features_{CN}.md          # Encoding, scaling, feature selection\n",
-    "\u2514\u2500\u2500 training/\n",
-    "    \u2514\u2500\u2500 ml_experiment.md               # Split strategy, temporal config, metrics\n",
+    "├── overview.md                        # Project context, objectives, datasets\n",
+    "├── config.md                          # CN, temporal settings, entity keys\n",
+    "├── landing/\n",
+    "│   └── landing_{source}.md            # Per-source: schema, quality, temporal profile\n",
+    "├── bronze/\n",
+    "│   ├── bronze_entity_{source}.md      # Per entity source: cleaning recommendations\n",
+    "│   └── bronze_event_{source}.md       # Per event source: cleaning + aggregation\n",
+    "├── silver/\n",
+    "│   └── silver_featureset_{CN}.md      # Merge strategy, joins, derived columns\n",
+    "├── gold/\n",
+    "│   └── gold_features_{CN}.md          # Encoding, scaling, feature selection\n",
+    "└── training/\n",
+    "    └── ml_experiment.md               # Split strategy, temporal config, metrics\n",
     "```"
    ]
   },
@@ -967,7 +967,7 @@
     "\n",
     "## 10.6 Run Pipeline\n",
     "\n",
-    "Single command runs everything: Bronze (parallel) \u2192 Silver \u2192 Gold \u2192 Training \u2192 MLflow UI (auto-opens browser)."
+    "Single command runs everything: Bronze (parallel) → Silver → Gold → Training → MLflow UI (auto-opens browser)."
    ]
   },
   {
@@ -991,218 +991,7 @@
     "tags": []
    },
    "outputs": [],
-   "source": [
-    "# @cr:code name='run_pipeline' id=8b659505\n",
-    "RUN_PIPELINE = True\n",
-    "# Resume gate: skip any landing/bronze/silver/gold notebook whose output UC table\n",
-    "# already exists. Operator-facing knob \u2014 flip to False for a forced full re-run\n",
-    "# (e.g. when findings or generated code changed in a way that invalidates prior\n",
-    "# tables). Training is never resume-skipped (no UC table to check; MLflow runs\n",
-    "# are tracked separately).\n",
-    "RESUME_FROM_EXISTING_TABLES = True\n",
-    "\n",
-    "if RUN_PIPELINE and GENERATION_TARGET == GenerationTarget.DATABRICKS:\n",
-    "    import json as _json\n",
-    "    import time as _time\n",
-    "\n",
-    "    def _resume_table_for(stage, nb_name, catalog, schema):\n",
-    "        \"\"\"Resolve the UC table written by a generated stage notebook.\n",
-    "\n",
-    "        Conventions established by the Databricks renderer:\n",
-    "          - landing/landing_<src>          -> {cat}.{sch}.landing_<src>\n",
-    "          - bronze/bronze_entity_<src>     -> {cat}.{sch}.bronze_entity_<src>\n",
-    "          - bronze/bronze_event_<src>      -> {cat}.{sch}.bronze_entity_<src>_events  (Shape A)\n",
-    "          - bronze/bronze_entity_<src>_aggregated -> same name as notebook\n",
-    "          - silver/silver_featureset_<CN>  -> same name as notebook\n",
-    "          - gold/gold_features_<CN>        -> same name as notebook\n",
-    "          - training/*                     -> None (no UC table)\n",
-    "        \"\"\"\n",
-    "        base = f\"{catalog}.{schema}\"\n",
-    "        if stage == \"landing\":\n",
-    "            return f\"{base}.{nb_name}\"\n",
-    "        if stage == \"bronze\":\n",
-    "            if nb_name.startswith(\"bronze_event_\"):\n",
-    "                source = nb_name[len(\"bronze_event_\"):]\n",
-    "                return f\"{base}.bronze_entity_{source}_events\"\n",
-    "            return f\"{base}.{nb_name}\"\n",
-    "        if stage in (\"silver\", \"gold\"):\n",
-    "            return f\"{base}.{nb_name}\"\n",
-    "        return None\n",
-    "\n",
-    "    print(f\"Databricks pipeline: {output_dir}\\n\")\n",
-    "    if RESUME_FROM_EXISTING_TABLES:\n",
-    "        print(\"[RESUME] table-existence resume enabled \u2014 stages with existing UC tables will be skipped\\n\")\n",
-    "    spark.sql(f\"CREATE SCHEMA IF NOT EXISTS {DATABRICKS_CATALOG}.{DATABRICKS_SCHEMA}\")\n",
-    "    _ns_params = {\"experiments_dir\": str(_namespace.root), \"run_id\": _namespace.run_id} if _namespace else {}\n",
-    "    _stages = [\"landing\", \"bronze\", \"silver\", \"gold\", \"training\"]\n",
-    "    _total_start = _time.time()\n",
-    "    _training_results = None\n",
-    "    _silver_results = None\n",
-    "    _skipped_count = 0\n",
-    "    for _stage in _stages:\n",
-    "        _stage_dir = output_dir / _stage\n",
-    "        if not _stage_dir.exists():\n",
-    "            continue\n",
-    "        if _stage == \"bronze\":\n",
-    "            # (retires NB10 patch \u00a72.11): events must run before\n",
-    "            # entities so the aggregator inputs exist when consumers try\n",
-    "            # to read them. Default sorted() puts `bronze_entity_*`\n",
-    "            # before `bronze_event_*` (i < v) and raises\n",
-    "            # [TABLE_OR_VIEW_NOT_FOUND] on the first aggregated bronze.\n",
-    "            _notebooks = sorted(\n",
-    "                (f.stem for f in _stage_dir.iterdir() if f.suffix == \".py\"),\n",
-    "                key=lambda n: (0 if n.startswith(\"bronze_event_\") else 1, n),\n",
-    "            )\n",
-    "            print(f\"[bronze order] {len(_notebooks)} notebooks; events first:\")\n",
-    "            for _n in _notebooks[:6]:\n",
-    "                print(f\"   {_n}\")\n",
-    "            if len(_notebooks) > 6:\n",
-    "                print(f\"   ... + {len(_notebooks) - 6} more\")\n",
-    "        else:\n",
-    "            _notebooks = sorted(f.stem for f in _stage_dir.iterdir() if f.suffix == \".py\")\n",
-    "        for _nb in _notebooks:\n",
-    "            _path = str(output_dir / _stage / _nb)\n",
-    "\n",
-    "            if RESUME_FROM_EXISTING_TABLES:\n",
-    "                _expected = _resume_table_for(_stage, _nb, DATABRICKS_CATALOG, DATABRICKS_SCHEMA)\n",
-    "                if _expected and spark.catalog.tableExists(_expected):\n",
-    "                    print(f\"[SKIP] {_stage.upper()}/{_nb}: {_expected} already exists\")\n",
-    "                    _skipped_count += 1\n",
-    "                    continue\n",
-    "\n",
-    "            _start = _time.time()\n",
-    "            _result = dbutils.notebook.run(_path, 86400, _ns_params)\n",
-    "            _elapsed = _time.time() - _start\n",
-    "            print(f\"[{_stage.upper()}] {_nb}: {_elapsed:.1f}s\")\n",
-    "            if _result and _stage in (\"training\", \"silver\"):\n",
-    "                try:\n",
-    "                    _parsed = _json.loads(_result)\n",
-    "                    if _stage == \"training\":\n",
-    "                        _training_results = _parsed\n",
-    "                    elif _stage == \"silver\":\n",
-    "                        _silver_results = _parsed\n",
-    "                except (ValueError, TypeError):\n",
-    "                    print(f\"  result: {_result}\")\n",
-    "    print(f\"\\nTotal: {_time.time() - _total_start:.1f}s ({_skipped_count} stages skipped via resume)\")\n",
-    "\n",
-    "    _ns = _namespace\n",
-    "    if _ns is not None:\n",
-    "        _bronze_path = _ns.bronze_metadata_path\n",
-    "        if _bronze_path.exists():\n",
-    "            _bm = _json.loads(_bronze_path.read_text())\n",
-    "            print(\"\\n\" + \"=\" * 60)\n",
-    "            print(\"BRONZE SUMMARY\")\n",
-    "            print(\"=\" * 60)\n",
-    "            print(f\"Sources: {_bm.get('total_sources', 0)}\")\n",
-    "            for _src, _info in _bm.get(\"sources\", {}).items():\n",
-    "                if isinstance(_info, dict):\n",
-    "                    print(f\"  {_src}: {_info.get('rows', '?'):,} rows, {_info.get('columns', '?')} columns\")\n",
-    "                else:\n",
-    "                    print(f\"  {_src}: {_info}\")\n",
-    "\n",
-    "        _gold_path = _ns.gold_metadata_path\n",
-    "        if _gold_path.exists():\n",
-    "            _gm = _json.loads(_gold_path.read_text())\n",
-    "            print(\"\\n\" + \"=\" * 60)\n",
-    "            print(\"GOLD SUMMARY\")\n",
-    "            print(\"=\" * 60)\n",
-    "            print(f\"Rows: {_gm.get('rows', '?'):,}, Columns: {_gm.get('columns', '?')}\")\n",
-    "            if \"feature_count\" in _gm:\n",
-    "                print(f\"Features: {_gm['feature_count']}\")\n",
-    "            if \"feature_version\" in _gm:\n",
-    "                print(f\"Version: {_gm['feature_version']}\")\n",
-    "            if \"elapsed_seconds\" in _gm:\n",
-    "                print(f\"Elapsed: {_gm['elapsed_seconds']}s\")\n",
-    "\n",
-    "    if _silver_results:\n",
-    "        print(\"\\n\" + \"=\" * 60)\n",
-    "        print(\"SILVER RESULTS\")\n",
-    "        print(\"=\" * 60)\n",
-    "        print(f\"Rows: {_silver_results.get('rows', '?'):,}, Columns: {_silver_results.get('columns', '?')}\")\n",
-    "        _es = _silver_results.get(\"elapsed_seconds\", {})\n",
-    "        if _es:\n",
-    "            print(\"\\nStage timings:\")\n",
-    "            for _name in (\"load_bronze\", \"merge_sources\", \"apply_derived\", \"holdout_mask\", \"delta_write\", \"optimize\", \"total\"):\n",
-    "                if _name in _es:\n",
-    "                    print(f\"  {_name:<18} {_es[_name]:>8.1f}s\")\n",
-    "        _mb = _silver_results.get(\"merge_breakdown\")\n",
-    "        if _mb:\n",
-    "            print(\"\\nMerge breakdown:\")\n",
-    "            print(f\"  Spine: {_mb.get('spine_rows', 0):,} rows = {_mb.get('spine_entities', 0):,} entities x {_mb.get('spine_dates', 0)} dates ({_mb.get('spine_stats_seconds', 0):.1f}s)\")\n",
-    "            print(f\"  Checkpoints: {_mb.get('checkpoint_count', 0)} ({_mb.get('checkpoint_seconds', 0):.1f}s)\")\n",
-    "            print(f\"  Validation: {_mb.get('validation_seconds', 0):.1f}s\")\n",
-    "            print(f\"  Merge total: {_mb.get('merge_total_seconds', 0):.1f}s\")\n",
-    "            _spd = _mb.get(\"seconds_per_dataset\", {})\n",
-    "            _cpd = _mb.get(\"columns_per_dataset\", {})\n",
-    "            print(\"\\nPer-source merge timing:\")\n",
-    "            _ranked = sorted(_mb.get(\"datasets_merged\", []), key=lambda n: _spd.get(n, 0), reverse=True)\n",
-    "            for _name in _ranked:\n",
-    "                print(f\"  {_name:<40} {_spd.get(_name, 0):>8.1f}s (+{_cpd.get(_name, 0)} cols)\")\n",
-    "\n",
-    "    if _training_results:\n",
-    "        print(\"\\n\" + \"=\" * 60)\n",
-    "        print(\"TRAINING RESULTS\")\n",
-    "        print(\"=\" * 60)\n",
-    "        if \"gold_data\" in _training_results:\n",
-    "            _gd = _training_results[\"gold_data\"]\n",
-    "            print(f\"\\nGold data: {_gd.get('rows', '?'):,} rows, {_gd.get('columns', '?')} columns\")\n",
-    "            print(f\"Column types: {_gd.get('column_types', {})}\")\n",
-    "        if \"feature_count\" in _training_results:\n",
-    "            print(f\"Features: {_training_results['feature_count']}\")\n",
-    "        if \"split\" in _training_results:\n",
-    "            _sp = _training_results[\"split\"]\n",
-    "            print(f\"\\nSplit: train={_sp.get('train_count', '?'):,}, test={_sp.get('test_count', '?'):,}\")\n",
-    "            print(f\"Cutoff date: {_sp.get('cutoff_date', 'N/A')}\")\n",
-    "        if \"label_distribution\" in _training_results:\n",
-    "            print(f\"Label distribution: {_training_results['label_distribution']}\")\n",
-    "        _fp = _training_results.get(\"feature_profile\", {})\n",
-    "        if _fp:\n",
-    "            print(f\"\\nFeature profile: production={_fp.get('production_features', '?')}, exploration={_fp.get('exploration_features', '?')}\")\n",
-    "            _disc = _fp.get(\"discrepancies\", [])\n",
-    "            if _disc:\n",
-    "                print(f\"WARNING: {len(_disc)} feature discrepancies vs exploration:\")\n",
-    "                for _d in _disc:\n",
-    "                    print(f\"  {_d}\")\n",
-    "            elif \"exploration_features\" in _fp:\n",
-    "                print(\"Feature profile matches exploration\")\n",
-    "            _excl = _fp.get(\"excluded_details\", {})\n",
-    "            if _excl:\n",
-    "                print(f\"\\nExclusion details ({len(_excl)} columns):\")\n",
-    "                for _col, _reason in sorted(_excl.items()):\n",
-    "                    print(f\"  {_col}: {_reason}\")\n",
-    "        if \"models\" in _training_results:\n",
-    "            print(\"\\nModel Results:\")\n",
-    "            print(f\"{'Model':<25} {'AUC':>8} {'PR-AUC':>8} {'F1':>8}\")\n",
-    "            print(\"-\" * 53)\n",
-    "            for _name, _metrics in _training_results[\"models\"].items():\n",
-    "                print(f\"{_name:<25} {_metrics.get('roc_auc', 0):.4f}   {_metrics.get('pr_auc', 0):.4f}   {_metrics.get('f1', 0):.4f}\")\n",
-    "        if \"best_model\" in _training_results:\n",
-    "            print(f\"\\nBest: {_training_results['best_model']} (AUC={_training_results.get('best_roc_auc', 0):.4f})\")\n",
-    "\n",
-    "elif RUN_PIPELINE and GENERATION_TARGET == GenerationTarget.LOCAL_FEAST_MLFLOW:\n",
-    "    import subprocess\n",
-    "    import sys\n",
-    "    runner_path = output_dir / \"pipeline_runner.py\"\n",
-    "    if runner_path.exists():\n",
-    "        print(f\"Running: python {runner_path.name}\")\n",
-    "        print(\"Pipeline: Landing -> Bronze -> Silver -> Gold -> Training...\\n\")\n",
-    "        proc = subprocess.Popen(\n",
-    "            [sys.executable, \"-u\", \"pipeline_runner.py\"],\n",
-    "            cwd=str(output_dir.resolve()),\n",
-    "            stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,\n",
-    "        )\n",
-    "        for line in proc.stdout:\n",
-    "            print(line, end=\"\", flush=True)\n",
-    "        if proc.wait() != 0:\n",
-    "            raise subprocess.CalledProcessError(proc.returncode, \"pipeline_runner.py\")\n",
-    "    else:\n",
-    "        print(\"pipeline_runner.py not found. Generate first by running cells above.\")\n",
-    "\n",
-    "else:\n",
-    "    print(\"To run the complete pipeline:\")\n",
-    "    print(f\"\\n  cd {output_dir}\")\n",
-    "    print(\"  python pipeline_runner.py\")"
-   ]
+   "source": "# @cr:code name='run_pipeline' id=8b659505\nRUN_PIPELINE = True\n# Resume gate: skip any landing/bronze/silver/gold notebook whose output UC table\n# already exists. Operator-facing knob — flip to False for a forced full re-run\n# (e.g. when findings or generated code changed in a way that invalidates prior\n# tables). Training is never resume-skipped (no UC table to check; MLflow runs\n# are tracked separately).\nRESUME_FROM_EXISTING_TABLES = True\n\nif RUN_PIPELINE and GENERATION_TARGET == GenerationTarget.DATABRICKS:\n    import json as _json\n    import time as _time\n\n    def _resume_table_for(stage, nb_name, catalog, schema):\n        \"\"\"Resolve the UC table written by a generated stage notebook.\n\n        Conventions established by the Databricks renderer:\n          - landing/landing_<src>          -> {cat}.{sch}.landing_<src>\n          - bronze/bronze_entity_<src>     -> {cat}.{sch}.bronze_entity_<src>\n          - bronze/bronze_event_<src>      -> {cat}.{sch}.bronze_entity_<src>_events  (Shape A)\n          - bronze/bronze_entity_<src>_aggregated -> same name as notebook\n          - silver/silver_featureset_<CN>  -> same name as notebook\n          - gold/gold_features_<CN>        -> same name as notebook\n          - training/*                     -> None (no UC table)\n        \"\"\"\n        base = f\"{catalog}.{schema}\"\n        if stage == \"landing\":\n            return f\"{base}.{nb_name}\"\n        if stage == \"bronze\":\n            if nb_name.startswith(\"bronze_event_\"):\n                source = nb_name[len(\"bronze_event_\"):]\n                return f\"{base}.bronze_entity_{source}_events\"\n            return f\"{base}.{nb_name}\"\n        if stage in (\"silver\", \"gold\"):\n            return f\"{base}.{nb_name}\"\n        return None\n\n    print(f\"Databricks pipeline: {output_dir}\\n\")\n    if RESUME_FROM_EXISTING_TABLES:\n        print(\"[RESUME] table-existence resume enabled — stages with existing UC tables will be skipped\\n\")\n    spark.sql(f\"CREATE SCHEMA IF NOT EXISTS {DATABRICKS_CATALOG}.{DATABRICKS_SCHEMA}\")\n    _ns_params = {\"experiments_dir\": str(_namespace.root), \"run_id\": _namespace.run_id} if _namespace else {}\n    _stages = [\"landing\", \"bronze\", \"silver\", \"gold\", \"training\"]\n    _total_start = _time.time()\n    _training_results = None\n    _silver_results = None\n    _skipped_count = 0\n    for _stage in _stages:\n        _stage_dir = output_dir / _stage\n        if not _stage_dir.exists():\n            continue\n        if _stage == \"bronze\":\n            # (retires NB10 patch §2.11): events must run before\n            # entities so the aggregator inputs exist when consumers try\n            # to read them. Default sorted() puts `bronze_entity_*`\n            # before `bronze_event_*` (i < v) and raises\n            # [TABLE_OR_VIEW_NOT_FOUND] on the first aggregated bronze.\n            _notebooks = sorted(\n                (f.stem for f in _stage_dir.iterdir() if f.suffix == \".py\"),\n                key=lambda n: (0 if n.startswith(\"bronze_event_\") else 1, n),\n            )\n            print(f\"[bronze order] {len(_notebooks)} notebooks; events first:\")\n            for _n in _notebooks[:6]:\n                print(f\"   {_n}\")\n            if len(_notebooks) > 6:\n                print(f\"   ... + {len(_notebooks) - 6} more\")\n        else:\n            _notebooks = sorted(f.stem for f in _stage_dir.iterdir() if f.suffix == \".py\")\n        for _nb in _notebooks:\n            _path = str(output_dir / _stage / _nb)\n\n            if RESUME_FROM_EXISTING_TABLES:\n                _expected = _resume_table_for(_stage, _nb, DATABRICKS_CATALOG, DATABRICKS_SCHEMA)\n                if _expected and spark.catalog.tableExists(_expected):\n                    print(f\"[SKIP] {_stage.upper()}/{_nb}: {_expected} already exists\")\n                    _skipped_count += 1\n                    continue\n\n            _start = _time.time()\n            _result = dbutils.notebook.run(_path, 86400, _ns_params)\n            _elapsed = _time.time() - _start\n            print(f\"[{_stage.upper()}] {_nb}: {_elapsed:.1f}s\")\n            # Generated stage notebooks structured-exit on top-level exceptions\n            # with {\"status\":\"FAILED\",...}; raise so the pipeline halts at the\n            # failed stage instead of cascading opaque downstream errors.\n            _parsed = None\n            if _result:\n                try:\n                    _parsed = _json.loads(_result)\n                except (ValueError, TypeError):\n                    if _stage in (\"training\", \"silver\"):\n                        print(f\"  result: {_result}\")\n            if isinstance(_parsed, dict) and _parsed.get(\"status\") == \"FAILED\":\n                raise RuntimeError(\n                    f\"[{_stage.upper()}] {_nb} failed: \"\n                    f\"{_parsed.get('error_type', '?')}: {_parsed.get('error_message', '')} \"\n                    \"(see spawned-notebook output for full traceback)\"\n                )\n            if isinstance(_parsed, dict) and _stage == \"training\":\n                _training_results = _parsed\n            elif isinstance(_parsed, dict) and _stage == \"silver\":\n                _silver_results = _parsed\n    print(f\"\\nTotal: {_time.time() - _total_start:.1f}s ({_skipped_count} stages skipped via resume)\")\n\n    _ns = _namespace\n    if _ns is not None:\n        _bronze_path = _ns.bronze_metadata_path\n        if _bronze_path.exists():\n            _bm = _json.loads(_bronze_path.read_text())\n            print(\"\\n\" + \"=\" * 60)\n            print(\"BRONZE SUMMARY\")\n            print(\"=\" * 60)\n            print(f\"Sources: {_bm.get('total_sources', 0)}\")\n            for _src, _info in _bm.get(\"sources\", {}).items():\n                if isinstance(_info, dict):\n                    print(f\"  {_src}: {_info.get('rows', '?'):,} rows, {_info.get('columns', '?')} columns\")\n                else:\n                    print(f\"  {_src}: {_info}\")\n\n        _gold_path = _ns.gold_metadata_path\n        if _gold_path.exists():\n            _gm = _json.loads(_gold_path.read_text())\n            print(\"\\n\" + \"=\" * 60)\n            print(\"GOLD SUMMARY\")\n            print(\"=\" * 60)\n            print(f\"Rows: {_gm.get('rows', '?'):,}, Columns: {_gm.get('columns', '?')}\")\n            if \"feature_count\" in _gm:\n                print(f\"Features: {_gm['feature_count']}\")\n            if \"feature_version\" in _gm:\n                print(f\"Version: {_gm['feature_version']}\")\n            if \"elapsed_seconds\" in _gm:\n                print(f\"Elapsed: {_gm['elapsed_seconds']}s\")\n\n    if _silver_results:\n        print(\"\\n\" + \"=\" * 60)\n        print(\"SILVER RESULTS\")\n        print(\"=\" * 60)\n        print(f\"Rows: {_silver_results.get('rows', '?'):,}, Columns: {_silver_results.get('columns', '?')}\")\n        _es = _silver_results.get(\"elapsed_seconds\", {})\n        if _es:\n            print(\"\\nStage timings:\")\n            for _name in (\"load_bronze\", \"merge_sources\", \"apply_derived\", \"holdout_mask\", \"delta_write\", \"optimize\", \"total\"):\n                if _name in _es:\n                    print(f\"  {_name:<18} {_es[_name]:>8.1f}s\")\n        _mb = _silver_results.get(\"merge_breakdown\")\n        if _mb:\n            print(\"\\nMerge breakdown:\")\n            print(f\"  Spine: {_mb.get('spine_rows', 0):,} rows = {_mb.get('spine_entities', 0):,} entities x {_mb.get('spine_dates', 0)} dates ({_mb.get('spine_stats_seconds', 0):.1f}s)\")\n            print(f\"  Checkpoints: {_mb.get('checkpoint_count', 0)} ({_mb.get('checkpoint_seconds', 0):.1f}s)\")\n            print(f\"  Validation: {_mb.get('validation_seconds', 0):.1f}s\")\n            print(f\"  Merge total: {_mb.get('merge_total_seconds', 0):.1f}s\")\n            _spd = _mb.get(\"seconds_per_dataset\", {})\n            _cpd = _mb.get(\"columns_per_dataset\", {})\n            print(\"\\nPer-source merge timing:\")\n            _ranked = sorted(_mb.get(\"datasets_merged\", []), key=lambda n: _spd.get(n, 0), reverse=True)\n            for _name in _ranked:\n                print(f\"  {_name:<40} {_spd.get(_name, 0):>8.1f}s (+{_cpd.get(_name, 0)} cols)\")\n\n    if _training_results:\n        print(\"\\n\" + \"=\" * 60)\n        print(\"TRAINING RESULTS\")\n        print(\"=\" * 60)\n        if \"gold_data\" in _training_results:\n            _gd = _training_results[\"gold_data\"]\n            print(f\"\\nGold data: {_gd.get('rows', '?'):,} rows, {_gd.get('columns', '?')} columns\")\n            print(f\"Column types: {_gd.get('column_types', {})}\")\n        if \"feature_count\" in _training_results:\n            print(f\"Features: {_training_results['feature_count']}\")\n        if \"split\" in _training_results:\n            _sp = _training_results[\"split\"]\n            print(f\"\\nSplit: train={_sp.get('train_count', '?'):,}, test={_sp.get('test_count', '?'):,}\")\n            print(f\"Cutoff date: {_sp.get('cutoff_date', 'N/A')}\")\n        if \"label_distribution\" in _training_results:\n            print(f\"Label distribution: {_training_results['label_distribution']}\")\n        _fp = _training_results.get(\"feature_profile\", {})\n        if _fp:\n            print(f\"\\nFeature profile: production={_fp.get('production_features', '?')}, exploration={_fp.get('exploration_features', '?')}\")\n            _disc = _fp.get(\"discrepancies\", [])\n            if _disc:\n                print(f\"WARNING: {len(_disc)} feature discrepancies vs exploration:\")\n                for _d in _disc:\n                    print(f\"  {_d}\")\n            elif \"exploration_features\" in _fp:\n                print(\"Feature profile matches exploration\")\n            _excl = _fp.get(\"excluded_details\", {})\n            if _excl:\n                print(f\"\\nExclusion details ({len(_excl)} columns):\")\n                for _col, _reason in sorted(_excl.items()):\n                    print(f\"  {_col}: {_reason}\")\n        if \"models\" in _training_results:\n            print(\"\\nModel Results:\")\n            print(f\"{'Model':<25} {'AUC':>8} {'PR-AUC':>8} {'F1':>8}\")\n            print(\"-\" * 53)\n            for _name, _metrics in _training_results[\"models\"].items():\n                print(f\"{_name:<25} {_metrics.get('roc_auc', 0):.4f}   {_metrics.get('pr_auc', 0):.4f}   {_metrics.get('f1', 0):.4f}\")\n        if \"best_model\" in _training_results:\n            print(f\"\\nBest: {_training_results['best_model']} (AUC={_training_results.get('best_roc_auc', 0):.4f})\")\n\nelif RUN_PIPELINE and GENERATION_TARGET == GenerationTarget.LOCAL_FEAST_MLFLOW:\n    import subprocess\n    import sys\n    runner_path = output_dir / \"pipeline_runner.py\"\n    if runner_path.exists():\n        print(f\"Running: python {runner_path.name}\")\n        print(\"Pipeline: Landing -> Bronze -> Silver -> Gold -> Training...\\n\")\n        proc = subprocess.Popen(\n            [sys.executable, \"-u\", \"pipeline_runner.py\"],\n            cwd=str(output_dir.resolve()),\n            stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,\n        )\n        for line in proc.stdout:\n            print(line, end=\"\", flush=True)\n        if proc.wait() != 0:\n            raise subprocess.CalledProcessError(proc.returncode, \"pipeline_runner.py\")\n    else:\n        print(\"pipeline_runner.py not found. Generate first by running cells above.\")\n\nelse:\n    print(\"To run the complete pipeline:\")\n    print(f\"\\n  cd {output_dir}\")\n    print(\"  python pipeline_runner.py\")"
   },
   {
    "cell_type": "markdown",
@@ -1259,13 +1048,13 @@
     "    items = sorted(path.iterdir(), key=lambda p: (p.is_file(), p.name))\n",
     "    for i, item in enumerate(items):\n",
     "        is_last = i == len(items) - 1\n",
-    "        connector = \"\u2514\u2500\u2500 \" if is_last else \"\u251c\u2500\u2500 \"\n",
+    "        connector = \"└── \" if is_last else \"├── \"\n",
     "        if item.is_file():\n",
     "            size = item.stat().st_size\n",
     "            print(f\"{prefix}{connector}{item.name} ({size:,} bytes)\")\n",
     "        else:\n",
     "            print(f\"{prefix}{connector}{item.name}/\")\n",
-    "            show_tree(item, prefix + (\"    \" if is_last else \"\u2502   \"))\n",
+    "            show_tree(item, prefix + (\"    \" if is_last else \"│   \"))\n",
     "\n",
     "if output_dir.exists():\n",
     "    show_tree(output_dir)"
@@ -1649,15 +1438,15 @@
     "#### Generated Structure\n",
     "```\n",
     "generated_pipelines/local/{pipeline_name}/\n",
-    "\u251c\u2500\u2500 run_all.py               \u2190 single entry point\n",
-    "\u251c\u2500\u2500 config.py                \u2190 shared settings\n",
-    "\u251c\u2500\u2500 landing/                 \u2190 raw \u2192 Delta\n",
-    "\u251c\u2500\u2500 bronze/                  \u2190 per-source processing\n",
-    "\u251c\u2500\u2500 silver_featureset_*.py   \u2190 temporal merge\n",
-    "\u251c\u2500\u2500 gold_features_*.py       \u2190 ML-ready features\n",
-    "\u251c\u2500\u2500 training/                \u2190 model training + MLflow\n",
-    "\u251c\u2500\u2500 feature_repo/            \u2190 Feast definitions\n",
-    "\u2514\u2500\u2500 data/                    \u2190 Delta tables\n",
+    "├── run_all.py               ← single entry point\n",
+    "├── config.py                ← shared settings\n",
+    "├── landing/                 ← raw → Delta\n",
+    "├── bronze/                  ← per-source processing\n",
+    "├── silver_featureset_*.py   ← temporal merge\n",
+    "├── gold_features_*.py       ← ML-ready features\n",
+    "├── training/                ← model training + MLflow\n",
+    "├── feature_repo/            ← Feast definitions\n",
+    "└── data/                    ← Delta tables\n",
     "```\n",
     "\n",
     "#### Verify Results\n",
@@ -1673,7 +1462,7 @@
     "\n",
     "**Option A: PyPI (cluster library)**\n",
     "Install `churnkit` as a cluster library or in `requirements-databricks.txt` for compute-scoped install.\n",
-    "No `sys.path` setup needed \u2014 the package is importable directly.\n",
+    "No `sys.path` setup needed — the package is importable directly.\n",
     "\n",
     "**Option B: Workspace Repo (development / latest code)**\n",
     "Clone the framework repo into Databricks Workspace Repos. Each notebook gets an auto-injected\n",
@@ -1704,16 +1493,16 @@
     "Upload the generated Databricks pipeline to Workspace:\n",
     "```\n",
     "generated_pipelines/databricks/{pipeline_name}/\n",
-    "\u251c\u2500\u2500 config.py                \u2190 Unity Catalog settings\n",
-    "\u251c\u2500\u2500 landing/                 \u2190 raw \u2192 UC Delta tables\n",
-    "\u251c\u2500\u2500 bronze/                  \u2190 per-source Spark notebooks\n",
-    "\u251c\u2500\u2500 silver_featureset_*.py   \u2190 distributed merge\n",
-    "\u251c\u2500\u2500 gold_features_*.py       \u2190 distributed transforms\n",
-    "\u251c\u2500\u2500 training/                \u2190 MLflow experiment tracking\n",
-    "\u2514\u2500\u2500 pipeline_runner.py       \u2190 orchestrator notebook\n",
+    "├── config.py                ← Unity Catalog settings\n",
+    "├── landing/                 ← raw → UC Delta tables\n",
+    "├── bronze/                  ← per-source Spark notebooks\n",
+    "├── silver_featureset_*.py   ← distributed merge\n",
+    "├── gold_features_*.py       ← distributed transforms\n",
+    "├── training/                ← MLflow experiment tracking\n",
+    "└── pipeline_runner.py       ← orchestrator notebook\n",
     "```\n",
     "\n",
-    "Run `pipeline_runner.py` \u2014 it executes all stages sequentially via `dbutils.notebook.run()`."
+    "Run `pipeline_runner.py` — it executes all stages sequentially via `dbutils.notebook.run()`."
    ]
   },
   {

{churnkit-1.2.8a0 → churnkit-1.2.8a2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "churnkit"
-version = "1.02.8a0"
+version = "1.02.8a2"
 description = "Structured ML framework for customer churn prediction -- from exploration notebooks to production pipelines, locally or on Databricks."
 readme = "README.md"
 license = {text = "Apache-2.0"}

{churnkit-1.2.8a0 → churnkit-1.2.8a2}/src/customer_retention/__init__.py

@@ -17,7 +17,7 @@ Main module categories:
                 llm_context, iteration)
 """
 
-__version__ = "1.02.8a0"
+__version__ = "1.02.8a2"
 
 # Environment utilities (always available)
 from .core.compat import (