xplainable-preprocessing 0.1.0__tar.gz

xplainable_preprocessing-0.1.0/PKG-INFO

@@ -0,0 +1,13 @@
+Metadata-Version: 2.4
+Name: xplainable-preprocessing
+Version: 0.1.0
+Summary: Shared preprocessing pipeline package for xplainable
+Requires-Python: >=3.9
+Requires-Dist: cloudpickle>=3.0
+Requires-Dist: numpy>=1.24
+Requires-Dist: pandas>=2.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: scikit-learn>=1.3
+Provides-Extra: dev
+Requires-Dist: pytest-cov; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'

xplainable_preprocessing-0.1.0/docs/dag-pipeline-proposal.md

@@ -0,0 +1,305 @@
+# DAG-Based Feature Pipeline
+
+## Problem
+
+The current `PipelineSpec` is a flat ordered list of steps. This creates three issues:
+
+1. **Ordering burden on the LLM** — the LLM must emit steps in correct dependency order. If step C needs the output of steps A and B, the LLM must place A and B before C. With complex feature graphs this is error-prone.
+
+2. **No parallelism** — independent branches (e.g. extract datetime features vs compute rolling aggregates) run sequentially even though they don't share state.
+
+3. **Post-hoc dependency analysis** — the autotrain agent already has `analyze_feature_dependencies_tool` which makes an extra LLM call to figure out execution order. A DAG makes this structural.
+
+## Proposal
+
+Add an optional `depends_on` field to `StepSpec`. When present on any step, the compiler builds a DAG and resolves execution order via topological sort. When absent on all steps, the pipeline behaves identically to today (flat sequential).
+
+### Spec Changes
+
+```python
+# schema.py
+class StepSpec(BaseModel):
+    id: str
+    type: str
+    columns: Optional[List[str]] = None
+    params: Dict = {}
+    description: Optional[str] = None
+    depends_on: Optional[List[str]] = None  # NEW — step IDs this step requires
+```
+
+```python
+# schema.py
+class PipelineSpec(BaseModel):
+    version: str = "2.0"  # stays 2.0 — depends_on is additive, not breaking
+    steps: List[StepSpec] = []
+```
+
+### Example Spec
+
+```json
+{
+  "version": "2.0",
+  "steps": [
+    {
+      "id": "extract_dow",
+      "type": "DateTimeExtractTransformer",
+      "columns": ["order_date"],
+      "params": {"components": ["dayofweek"]},
+      "depends_on": []
+    },
+    {
+      "id": "rolling_sales",
+      "type": "RollingAggTransformer",
+      "columns": ["sales"],
+      "params": {"group_by": ["store_id"], "window": 7, "operation": "mean", "order_by": "order_date"},
+      "depends_on": []
+    },
+    {
+      "id": "dow_sales_interaction",
+      "type": "ExpressionTransformer",
+      "params": {"expression": "order_date_dayofweek * sales_mean_7", "output_column": "dow_sales_x"},
+      "depends_on": ["extract_dow", "rolling_sales"]
+    },
+    {
+      "id": "drop_raw_date",
+      "type": "DropColumnsTransformer",
+      "params": {"columns": ["order_date"]},
+      "depends_on": ["extract_dow", "rolling_sales"]
+    }
+  ]
+}
+```
+
+This forms the graph:
+
+```
+extract_dow ──┬──> dow_sales_interaction
+              ├──> drop_raw_date
+rolling_sales ┘
+```
+
+Steps `extract_dow` and `rolling_sales` have no dependencies so they can run in parallel. `dow_sales_interaction` and `drop_raw_date` wait for both to complete, then they can also run in parallel.
+
+## Implementation Plan
+
+### Phase 1: Schema + Validation (no runtime changes)
+
+**Files**: `schema.py`
+
+1. Add `depends_on: Optional[List[str]] = None` to `StepSpec`
+2. Add validation in `PipelineSpec`:
+   - All `depends_on` IDs must reference existing step IDs
+   - No cycles (topological sort must succeed)
+   - No self-references
+3. Add `validate_spec()` check: if any step has `depends_on`, verify the DAG is valid
+
+```python
+# In validate_spec():
+def _validate_dag(spec: PipelineSpec) -> None:
+    """Validate DAG constraints if any step uses depends_on."""
+    has_dag = any(step.depends_on is not None for step in spec.steps)
+    if not has_dag:
+        return  # flat pipeline, no DAG validation needed
+
+    step_ids = {step.id for step in spec.steps}
+    for step in spec.steps:
+        if step.depends_on:
+            for dep in step.depends_on:
+                if dep not in step_ids:
+                    raise ValueError(f"Step '{step.id}' depends on unknown step '{dep}'")
+                if dep == step.id:
+                    raise ValueError(f"Step '{step.id}' cannot depend on itself")
+
+    # Cycle detection via topological sort
+    _topological_sort(spec.steps)  # raises ValueError on cycle
+```
+
+**Tests**: Spec with valid DAG parses. Spec with cycle raises. Spec with missing dep raises. Flat spec (no `depends_on`) still works.
+
+### Phase 2: DAG Compiler
+
+**Files**: `compiler.py`, new `dag.py`
+
+Create a `compile_dag()` function that:
+1. Builds an adjacency graph from `depends_on`
+2. Topologically sorts steps into execution "stages" (groups of independent steps)
+3. Returns a `DAGPipeline` (or falls back to `DataFramePipeline` for flat specs)
+
+```python
+# dag.py
+from collections import defaultdict, deque
+
+def topological_stages(steps: list[StepSpec]) -> list[list[StepSpec]]:
+    """Group steps into parallel execution stages via Kahn's algorithm.
+
+    Returns a list of stages. Steps within a stage have no mutual
+    dependencies and can run in parallel.
+    """
+    step_map = {s.id: s for s in steps}
+    in_degree = {s.id: 0 for s in steps}
+    dependents = defaultdict(list)  # step_id -> list of steps that depend on it
+
+    for step in steps:
+        for dep in (step.depends_on or []):
+            in_degree[step.id] += 1
+            dependents[dep].append(step.id)
+
+    stages = []
+    queue = deque(sid for sid, deg in in_degree.items() if deg == 0)
+
+    while queue:
+        # All items currently in queue have in_degree 0 — they form a parallel stage
+        stage = []
+        next_queue = deque()
+        while queue:
+            sid = queue.popleft()
+            stage.append(step_map[sid])
+            for dependent in dependents[sid]:
+                in_degree[dependent] -= 1
+                if in_degree[dependent] == 0:
+                    next_queue.append(dependent)
+        stages.append(stage)
+        queue = next_queue
+
+    # Check for cycles
+    processed = sum(len(s) for s in stages)
+    if processed != len(steps):
+        raise ValueError("Cycle detected in step dependencies")
+
+    return stages
+```
+
+Update `compile_spec()`:
+
+```python
+# compiler.py
+def compile_spec(spec: PipelineSpec) -> DataFramePipeline:
+    has_dag = any(step.depends_on is not None for step in spec.steps)
+    if has_dag:
+        return _compile_dag(spec)
+    return _compile_flat(spec)  # existing logic
+```
+
+### Phase 3: DAGPipeline Runtime
+
+**Files**: new `dag_pipeline.py`
+
+```python
+class DAGPipeline(BaseEstimator, TransformerMixin):
+    """Pipeline that executes steps in DAG-resolved stage order.
+
+    Steps within the same stage are independent and can be run
+    concurrently (future optimization). Steps across stages run
+    sequentially — each stage sees the DataFrame produced by all
+    prior stages.
+    """
+
+    def __init__(self, stages: list[list[tuple[str, TransformerMixin]]]):
+        # stages[i] = [(step_id, transformer), ...] — all independent
+        self.stages = stages
+
+    def fit(self, X: pd.DataFrame, y=None):
+        Xt = X.copy()
+        for stage in self.stages:
+            # All transformers in a stage fit on the same Xt
+            for name, transformer in stage:
+                transformer.fit(Xt, y)
+            # Then apply all transforms (order within stage doesn't matter
+            # because they read from the same Xt snapshot)
+            Xt_snapshot = Xt.copy()
+            for name, transformer in stage:
+                result = transformer.transform(Xt_snapshot)
+                # Merge result columns back into Xt
+                new_cols = [c for c in result.columns if c not in Xt.columns]
+                for col in new_cols:
+                    Xt[col] = result[col]
+                # Handle column modifications (existing cols that changed)
+                modified_cols = [c for c in result.columns if c in Xt.columns and c in (transformer.columns if hasattr(transformer, 'columns') else [])]
+                for col in modified_cols:
+                    Xt[col] = result[col]
+        return self
+
+    def transform(self, X: pd.DataFrame) -> pd.DataFrame:
+        # Same logic but without fit
+        ...
+```
+
+**Key design decisions**:
+
+- **Snapshot per stage**: Each stage gets a snapshot of the DataFrame. Transforms within a stage read from the snapshot (not from each other's outputs). This prevents order-dependence within a stage.
+- **Merge strategy**: New columns are added. Modified columns are overwritten. Dropped columns are removed. This needs careful handling — see "Tricky Bits" below.
+- **Sequential first, parallel later**: Start with sequential execution within stages. Add `joblib.Parallel` or `concurrent.futures` as an optimization later.
+
+### Phase 4: LLM Integration
+
+**Files**: `feature_engineer.py` system prompts
+
+Update the `generate()` and `generate_next()` prompts to tell the LLM about `depends_on`:
+
+```
+Each step_spec can optionally include "depends_on": ["step_id_1", "step_id_2"]
+to declare that this step requires columns created by other steps.
+If a step only uses original dataset columns, omit depends_on or set it to [].
+```
+
+The LLM doesn't need to worry about ordering — just declare dependencies. The compiler handles the rest.
+
+### Phase 5: Autotrain Agent Integration
+
+**Files**: `feature_engineering.py` handler, `batch_apply_steps_tool`
+
+- `batch_apply_steps_tool` already accepts a list of step specs and creates a single `PipelineSpec`. No changes needed — the DAG resolution happens at compile time.
+- `analyze_feature_dependencies_tool` can be simplified to just set `depends_on` on each step based on column analysis, rather than trying to reorder indices.
+- Interactive mode: as the user approves features one at a time, the `depends_on` edges naturally build up. A new feature can reference previously applied feature IDs.
+
+## Tricky Bits
+
+### 1. Schema Propagation
+
+The hardest problem. Within a stage, transformer A might create column `foo` that transformer B in the **next** stage needs. The compiler needs to know the intermediate schema at each stage to validate column references.
+
+**Options**:
+- **Lazy (recommended for v1)**: Don't validate column existence at compile time. Let it fail at runtime with a clear error: "Step 'X' references column 'foo' which doesn't exist. Check depends_on."
+- **Eager (future)**: Add an `output_schema` field to `StepSpec` that declares what columns a step adds/removes/modifies. The compiler walks the DAG and checks that every step's input columns exist in the union of original columns + upstream output columns.
+
+### 2. Column Conflicts Within a Stage
+
+If two steps in the same stage both modify column `age`, the result is ambiguous.
+
+**Solution**: Validate at compile time that steps within a stage don't write to the same columns. Steps that create new columns (ExpressionTransformer) are fine. Steps that modify existing columns (StandardScaler on `age`) must not share target columns with other steps in the same stage.
+
+### 3. Drop Steps
+
+`DropColumnsTransformer` is tricky in a DAG because you need to ensure no downstream step needs the dropped column.
+
+**Solution**: Drop steps should depend on all steps that read the column being dropped. The compiler can warn if a drop step doesn't depend on all readers.
+
+### 4. Fitted State Serialization
+
+Currently `save_pipeline` / `load_pipeline` use cloudpickle on the entire `DataFramePipeline`. A `DAGPipeline` has a `stages` structure instead of flat `steps`. Cloudpickle should handle this transparently since it serializes the object graph. But test this explicitly.
+
+### 5. Backwards Compatibility
+
+A `PipelineSpec` with no `depends_on` on any step is exactly today's flat pipeline. The compiler detects this and returns a `DataFramePipeline`. Zero breaking changes for existing specs in the database.
+
+## Testing Strategy
+
+1. **Unit: `topological_stages()`** — Valid DAG, diamond dependency, linear chain, single node, disjoint subgraphs, cycle detection
+2. **Unit: `DAGPipeline.fit_transform()`** — Two independent branches merge into one step, verify output columns are correct
+3. **Integration: `compile_spec()` with DAG** — Full round-trip: spec with `depends_on` -> compile -> fit -> transform -> verify output
+4. **Regression: Flat specs** — All existing 88 tests must still pass unchanged
+5. **Edge cases**: Step depends on itself, step depends on non-existent ID, two steps create same output column in same stage
+
+## Recommended Implementation Order
+
+| Step | Effort | Risk | What it unblocks |
+|------|--------|------|-----------------|
+| 1. `depends_on` field + validation | Small | None | Everything else |
+| 2. `topological_stages()` | Small | None | DAGPipeline |
+| 3. `DAGPipeline` (sequential within stages) | Medium | Medium | Full DAG execution |
+| 4. `compile_spec()` DAG branch | Small | Low | End-to-end usage |
+| 5. LLM prompt updates | Small | Low | Agent generates DAGs |
+| 6. Parallel execution within stages | Medium | Medium | Performance gain |
+| 7. Schema propagation validation | Large | High | Compile-time safety |
+
+Steps 1-4 give you a working DAG pipeline. Step 5 lets the LLM use it. Steps 6-7 are optimizations you can add later.

xplainable_preprocessing-0.1.0/docs/feature-pipeline-architectures.md

@@ -0,0 +1,363 @@
+# Feature Pipeline Architectures: Alternatives Analysis
+
+## The Core Problem
+
+How should an LLM express feature transformations so they're executable, composable, and don't require manual ordering?
+
+The current system (after the inline step_spec work) has the LLM produce flat, ordered step specs. This works but pushes ordering logic onto the LLM and limits composability. There are several fundamentally different paradigms worth considering.
+
+---
+
+## 1. DAG Pipeline
+
+**See**: `dag-pipeline-proposal.md` for full implementation plan.
+
+Steps declare dependencies via `depends_on`. The compiler topologically sorts and groups into parallel execution stages.
+
+```json
+{
+  "steps": [
+    {"id": "extract_dow", "type": "DateTimeExtractTransformer", "depends_on": []},
+    {"id": "rolling_sales", "type": "RollingAggTransformer", "depends_on": []},
+    {"id": "interaction", "type": "ExpressionTransformer", "depends_on": ["extract_dow", "rolling_sales"]}
+  ]
+}
+```
+
+**Strengths**:
+- Closest to current model — additive change to StepSpec
+- Explicit dependencies are debuggable and inspectable
+- Enables parallel execution within stages
+- LLM no longer needs to worry about ordering
+
+**Weaknesses**:
+- LLM still thinks in terms of transformer types and wiring
+- Schema propagation between stages is complex to validate at compile time
+- Adding a step mid-graph requires understanding the full dependency structure
+
+**Effort**: Medium. Builds directly on existing schema/compiler/pipeline.
+
+**Best for**: Incremental improvement with low risk.
+
+---
+
+## 2. Declarative Column Specs
+
+Instead of "run these transformers in this order", the LLM declares "I want these output columns defined by these expressions." The compiler resolves what transformers to use and in what order.
+
+### Spec Format
+
+```json
+{
+  "version": "3.0",
+  "target_columns": {
+    "age_income_ratio": {
+      "expr": "age / income",
+      "description": "Ratio of age to income"
+    },
+    "log_salary": {
+      "expr": "log(salary)",
+      "description": "Log-transformed salary to reduce skewness"
+    },
+    "order_dayofweek": {
+      "extract": "dayofweek",
+      "from": "order_date",
+      "description": "Day of week from order timestamp"
+    },
+    "sales_7d_avg": {
+      "rolling": "mean",
+      "column": "sales",
+      "window": 7,
+      "group_by": ["store_id"],
+      "order_by": "order_date",
+      "description": "7-day rolling average sales per store"
+    },
+    "risk_category": {
+      "bin": "risk_score",
+      "strategy": "quantile",
+      "n_bins": 5,
+      "description": "Risk score bucketed into quintiles"
+    }
+  },
+  "drop_columns": ["order_date", "customer_name"],
+  "transformations": {
+    "salary": {"impute": "median"},
+    "country": {"encode": "onehot"}
+  }
+}
+```
+
+### How the Compiler Works
+
+1. Parse `target_columns` — each is a column definition with a known pattern (`expr`, `extract`, `rolling`, `bin`, etc.)
+2. Map each pattern to a transformer type:
+   - `expr` → `ExpressionTransformer`
+   - `extract` → `DateTimeExtractTransformer`
+   - `rolling` → `RollingAggTransformer`
+   - `bin` → `KBinsDiscretizer`
+3. Infer dependencies by parsing column references in expressions
+4. Topologically sort and compile to a `DataFramePipeline` (or `DAGPipeline`)
+
+### What the LLM Prompt Looks Like
+
+```
+You are a data scientist. Given a dataset summary, declare what new columns
+would improve model performance.
+
+Available column definition patterns:
+- expr: pandas expression (e.g. "col_a * col_b", "log(price)")
+- extract: datetime component (dayofweek, month, hour, year)
+- rolling: rolling window aggregate (mean, sum, std, min, max)
+- lag: grouped lag features
+- bin: discretize continuous values
+- custom: Python code for anything else
+
+Respond with a JSON object listing desired columns:
+{"target_columns": {"column_name": {"expr": "...", "description": "..."}}}
+```
+
+**Strengths**:
+- Better abstraction for LLMs — "what do you want?" not "how do you build it?"
+- Dependencies are implicit (parsed from expressions) — no `depends_on` to manage
+- Closer to how data scientists think: "I want a column called X"
+- The DSL is small and learnable — ~6 patterns cover 95% of feature engineering
+- Compiler owns all transformer selection logic — single source of truth
+- Naturally deduplicates: two recommendations that want the same column just merge
+
+**Weaknesses**:
+- Requires designing a column definition DSL (the patterns above)
+- Expression parsing needs to extract column references reliably
+- Complex features (custom sklearn transformers) still need an escape hatch
+- More upfront design than DAG, which is purely additive
+
+**Effort**: Medium-large. New spec format, new compiler path, new prompt design.
+
+**Best for**: Clean long-term architecture. Ideal if rebuilding the spec layer.
+
+---
+
+## 3. Expression Graph (Polars-Style Lazy Evaluation)
+
+Every feature is a lazy column expression. The engine builds a computation graph, optimizes it (predicate pushdown, common subexpression elimination, projection pushdown), then executes.
+
+### What It Looks Like
+
+```python
+import polars as pl
+
+features = (
+    df.lazy()
+    .with_columns([
+        (pl.col("age") / pl.col("income")).alias("age_income_ratio"),
+        pl.col("salary").log().alias("log_salary"),
+        pl.col("order_date").dt.weekday().alias("order_dow"),
+        pl.col("sales").rolling_mean(7).over("store_id").alias("sales_7d_avg"),
+    ])
+    .drop(["order_date", "customer_name"])
+    .collect()
+)
+```
+
+### Architecture
+
+- LLM generates Polars expressions (or a JSON DSL that compiles to Polars)
+- The lazy evaluation engine handles ordering, optimization, and execution
+- No manual pipeline construction needed — the engine IS the pipeline
+
+### Spec Format (JSON → Polars)
+
+```json
+{
+  "expressions": [
+    {"alias": "age_income_ratio", "expr": "col('age') / col('income')"},
+    {"alias": "log_salary", "expr": "col('salary').log()"},
+    {"alias": "order_dow", "expr": "col('order_date').dt.weekday()"}
+  ]
+}
+```
+
+A thin compiler converts these to `pl.Expr` objects.
+
+**Strengths**:
+- 10-100x faster than sequential pandas for large datasets
+- Automatic query optimization — the engine handles parallelism and ordering
+- No pipeline/DAG/ordering logic to maintain — the engine does it all
+- Polars expressions are more powerful and composable than sklearn transformers
+- Memory efficient: streaming execution, no intermediate DataFrame copies
+
+**Weaknesses**:
+- Requires Polars dependency (currently using pandas + sklearn)
+- sklearn transformers (StandardScaler, OneHotEncoder, etc.) don't work with Polars natively — need wrappers or reimplementation
+- Fit/transform semantics don't exist in Polars — need to handle stateful transforms (e.g., imputer learns mean from training data, applies to new data) separately
+- LLMs are more familiar with pandas than Polars syntax
+- Large migration from current architecture
+
+**Effort**: Large. New runtime, new serialization, adapter layer for sklearn.
+
+**Best for**: Performance-critical pipelines with large datasets. Consider if pandas becomes the bottleneck.
+
+---
+
+## 4. Feature Store Pattern
+
+Each feature is a standalone, versioned computation with a defined contract: input columns, output columns, and the transformation logic. Features are registered in a catalog and can be composed across models.
+
+### Architecture
+
+```
+FeatureRegistry
+├── age_income_ratio (v1)
+│   ├── inputs: [age, income]
+│   ├── outputs: [age_income_ratio]
+│   ├── transformer: ExpressionTransformer(...)
+│   └── fitted_state: {...}
+├── sales_7d_avg (v2)
+│   ├── inputs: [sales, store_id, order_date]
+│   ├── outputs: [sales_mean_7]
+│   ├── transformer: RollingAggTransformer(...)
+│   └── fitted_state: {...}
+└── ...
+```
+
+### How It Works
+
+1. LLM recommends features → each becomes a registered feature with input/output contract
+2. User selects features → system resolves dependencies from input/output declarations
+3. Compiler builds a pipeline from selected features (ordering resolved from contracts)
+4. Features are cached — once computed for a dataset, they're reused across models
+5. Features are versioned — changing the definition creates a new version
+
+### Spec Format
+
+```json
+{
+  "features": ["age_income_ratio@v1", "sales_7d_avg@v2", "order_dow@v1"],
+  "drop": ["order_date"],
+  "target": "churn"
+}
+```
+
+The spec is just a list of feature references. The registry has all the logic.
+
+**Strengths**:
+- Features become reusable assets across autotrain runs
+- Natural caching: compute once, use everywhere
+- Versioning: change a feature definition without breaking existing models
+- Input/output contracts make dependency resolution trivial
+- Testing is per-feature, not per-pipeline
+- Aligns with industry direction (Feast, Tecton, Featureform)
+
+**Weaknesses**:
+- Heavyweight for single-run autotrain — overhead only pays off with feature reuse
+- Requires a feature registry (new infrastructure)
+- Fitted state becomes per-feature — serialization/loading is more complex
+- LLM needs to know what features already exist in the registry (growing context)
+
+**Effort**: Large. New registry, versioning system, caching layer.
+
+**Best for**: Platform with many models sharing features. Overkill for single autotrain runs today, but could become the right abstraction as the platform scales.
+
+---
+
+## 5. Auto-Feature Synthesis (Featuretools-Style)
+
+Algorithmic feature generation, not LLM-driven. Given columns and their types, systematically generate all meaningful combinations and score them against the target.
+
+### How It Works
+
+1. Define "primitives": ratio, product, sum, difference, log, sqrt, datetime_extract, grouped_agg
+2. Enumerate: for each pair of numeric columns, generate ratio, product, difference. For each datetime, extract all components. For each numeric + categorical pair, generate grouped aggregates.
+3. Score: compute mutual information (or correlation, or permutation importance) between each candidate feature and the target variable
+4. Rank and select: keep the top N features
+
+### Integration with Current System
+
+This doesn't replace LLM recommendations — it complements them:
+
+```
+Auto-mode pipeline:
+1. Auto-feature synthesis → 50 candidate features, scored
+2. Filter to top 10 by mutual information
+3. LLM reviews candidates → refines, combines, adds domain insight
+4. Final feature set → compile and apply
+```
+
+Or simpler:
+
+```
+Auto-mode pipeline:
+1. LLM generates 5 creative features (current flow)
+2. Auto-synthesis generates 20 mechanical features (ratios, interactions)
+3. Score all 25 against target
+4. Keep top N
+```
+
+**Strengths**:
+- Exhaustive: finds features the LLM wouldn't think of (e.g., `col7 * col13 / col2` that happens to be predictive)
+- No hallucination risk — purely data-driven
+- Deterministic and reproducible
+- Can run without any LLM calls
+- Well-established technique (Featuretools, AutoFeat, tsfresh)
+
+**Weaknesses**:
+- Combinatorial explosion: N columns → O(N^2) pairwise features, O(N^3) for triples
+- Generates many useless features — needs aggressive filtering
+- Requires a target variable (supervised) for scoring
+- Features lack semantic meaning — "col3_times_col7" isn't interpretable
+- Doesn't handle complex domain-specific features (e.g., "customer lifetime value")
+
+**Effort**: Medium. Can be implemented as a standalone tool alongside LLM recommendations.
+
+**Best for**: Auto-mode where speed matters more than interpretability. Excellent complement to LLM-generated features.
+
+---
+
+## Comparison Matrix
+
+| Approach | Ordering Problem | LLM Cognitive Load | Performance | Reusability | Migration Effort |
+|----------|-----------------|-------------------|-------------|-------------|-----------------|
+| **Current (flat)** | LLM must order | High | Sequential | None | N/A |
+| **DAG** | Compiler resolves | Medium | Parallelizable | None | Small |
+| **Declarative columns** | Implicit from expressions | Low | Parallelizable | None | Medium |
+| **Expression graph** | Engine resolves | Low | Optimal | None | Large |
+| **Feature store** | Contracts resolve | Low | Cached | Full reuse | Large |
+| **Auto-synthesis** | N/A (scored) | None | Parallel | Cacheable | Medium |
+
+## Recommended Path
+
+### Short-term (now → 1 month)
+**DAG** — additive to existing spec, solves the immediate ordering problem, low risk.
+
+### Medium-term (1-3 months)
+**Declarative column specs** — design the DSL, build a compiler that maps column definitions to the existing transformer infrastructure. This becomes the LLM-facing abstraction while the DAG handles execution underneath.
+
+### Long-term (3-6 months)
+**Feature store** — as the platform handles more models, feature reuse becomes valuable. The declarative column spec format naturally evolves into feature definitions with input/output contracts.
+
+### Orthogonal (can start anytime)
+**Auto-feature synthesis** — independent of pipeline architecture. Add as an auto-mode enhancement alongside LLM recommendations. Start with pairwise numeric interactions + datetime extractions, score by mutual information, filter aggressively.
+
+### Consider later
+**Expression graph (Polars)** — only pursue if pandas performance becomes a bottleneck on real workloads. The migration cost is high and the current DataFrame sizes in autotrain may not justify it.
+
+---
+
+## Architecture Layering
+
+These approaches aren't mutually exclusive. The ideal architecture layers them:
+
+```
+┌─────────────────────────────────────────┐
+│  LLM / Auto-Synthesis (feature ideas)   │  ← what columns to create
+├─────────────────────────────────────────┤
+│  Declarative Column Specs (DSL)         │  ← user-facing format
+├─────────────────────────────────────────┤
+│  DAG Compiler (dependency resolution)   │  ← execution planning
+├─────────────────────────────────────────┤
+│  Pipeline Runtime (execution)           │  ← fit/transform
+├─────────────────────────────────────────┤
+│  Feature Store (versioning + caching)   │  ← persistence + reuse
+└─────────────────────────────────────────┘
+```
+
+Each layer can be built independently. The inline step_spec work we just completed is the foundation — it gives us structured specs flowing from the LLM. Every architecture above builds on that.