flasheda 0.3.0__tar.gz → 0.4.0__tar.gz

{flasheda-0.3.0 → flasheda-0.4.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: flasheda
-Version: 0.3.0
-Summary: Fixed-time EDA for any dataset size, plus validated cleaning code for pandas, polars, DuckDB and PySpark.
+Version: 0.4.0
+Summary: Fixed-time EDA for any dataset size, plus validated cleaning, feature and training code for pandas, polars, DuckDB and PySpark.
 Home-page: https://github.com/sjapanjots/Flash-EDA
 Author: Japanjot Singh
 Author-email: sjapanjots@gmail.com
@@ -9,7 +9,7 @@ License: MIT
 Project-URL: Source, https://github.com/sjapanjots/Flash-EDA
 Project-URL: Changelog, https://github.com/sjapanjots/Flash-EDA/blob/main/CHANGELOG.md
 Project-URL: Issues, https://github.com/sjapanjots/Flash-EDA/issues
-Keywords: eda,data-cleaning,pandas,polars,duckdb,pyspark,code-generation
+Keywords: eda,data-cleaning,pandas,polars,duckdb,pyspark,code-generation,feature-engineering,automl
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Science/Research
 Classifier: Intended Audience :: Developers
@@ -32,11 +32,20 @@ Requires-Dist: polars>=0.20; extra == "fast"
 Requires-Dist: pyarrow>=12; extra == "fast"
 Provides-Extra: spark
 Requires-Dist: pyspark>=3.4; extra == "spark"
+Provides-Extra: ml
+Requires-Dist: scikit-learn>=1.2; extra == "ml"
+Requires-Dist: pyarrow>=12; extra == "ml"
+Provides-Extra: all
+Requires-Dist: duckdb>=0.10; extra == "all"
+Requires-Dist: polars>=0.20; extra == "all"
+Requires-Dist: pyarrow>=12; extra == "all"
+Requires-Dist: scikit-learn>=1.2; extra == "all"
 Provides-Extra: dev
 Requires-Dist: pytest>=7; extra == "dev"
 Requires-Dist: duckdb>=0.10; extra == "dev"
 Requires-Dist: polars>=0.20; extra == "dev"
 Requires-Dist: pyarrow>=12; extra == "dev"
+Requires-Dist: scikit-learn>=1.2; extra == "dev"
 Requires-Dist: build; extra == "dev"
 Requires-Dist: twine; extra == "dev"
 Dynamic: author
@@ -63,9 +72,9 @@ Dynamic: summary
 [![Tests](https://github.com/sjapanjots/Flash-EDA/actions/workflows/tests.yml/badge.svg)](https://github.com/sjapanjots/Flash-EDA/actions/workflows/tests.yml)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](MIT.License)
 
-**Current version: 0.3.0** · [Changelog](CHANGELOG.md)
+**Current version: 0.4.0** · [Changelog](CHANGELOG.md)
 
-Analyse any dataset in a few seconds, then get **validated cleaning code** that runs on the full data, in the engine that fits its size: pandas, polars, DuckDB or PySpark.
+Analyse any dataset in a few seconds, then get **validated cleaning, feature and training code** that runs on the full data, in the engine that fits its size: pandas, polars, DuckDB or PySpark.
 
 FlashEDA analyses a fixed-size random sample, so analysis time barely changes between 100 thousand and 100 million rows.
 
@@ -74,7 +83,9 @@ FlashEDA analyses a fixed-size random sample, so analysis time barely changes be
 ```bash
 pip install flasheda            # core
 pip install "flasheda[fast]"    # + polars, DuckDB, pyarrow (recommended for big files)
+pip install "flasheda[ml]"      # + scikit-learn, for flasheda.train
 pip install "flasheda[spark]"   # + PySpark
+pip install "flasheda[all]"     # everything except PySpark
 ```
 
 ## Quick start
@@ -90,6 +101,11 @@ report.save_html("report.html")    # or report.save_pdf("report.pdf")
 # 2. Clean: get a script for the full data, already checked on the sample
 code = fe.clean("orders.csv", target="churn", engine="polars")
 code.save("clean_orders.py")       # then: python clean_orders.py
+
+# 3. Model: compare models on the sample, get prep + training scripts
+result = fe.train("orders.csv", target="churn", engine="polars")
+result.show()                      # leaderboard
+result.save("prep.py", "train.py") # python prep.py && python train.py → model.joblib
 ```
 
 ## What it does
@@ -102,6 +118,8 @@ code.save("clean_orders.py")       # then: python clean_orders.py
 | **Correlations** | Pearson (numeric) and Cramér's V (categorical) |
 | **Reports** | console, HTML, PDF, JSON |
 | **Code generation** | one cleaning plan → pandas, polars (lazy, streaming), DuckDB SQL (larger than memory) or PySpark |
+| **Features** | date parts, one-hot / frequency encoding, ratios that beat their inputs |
+| **Models** | baseline, linear, random forest, gradient boosting compared with the right metric and split; training script for the winner |
 | **Validation** | generated code is run on the sample and compared with a reference result before you get it |
 
 ### Cleaning steps
@@ -115,6 +133,14 @@ Each step in the generated code has a comment saying why it was added:
 - add `<col>_was_missing` flags when missingness predicts the target
 - fill nulls (median / mode), log-transform skewed columns, clip outliers
 
+### Features and models
+`fe.train()` (or `report.train()`) adds features on top of the cleaning steps, compares models on the sample, and writes two scripts:
+
+- **`prep.py`**: cleaning + features for the full data, in the chosen engine
+- **`train.py`**: scikit-learn training with cross-validation, using the same metric and split, that saves `model.joblib`
+
+The metric and split follow the target: PR-AUC for imbalanced binary targets, MAE for skewed regression targets, time-based splits when there's a date column, and so on.
+
 ### Engine choice
 
 | Rows | Engine |
@@ -135,11 +161,14 @@ Measured on synthetic messy data:
 | Sample a 1M-row CSV (0.1.x → 0.2.0) | 37.3 s → 0.65 s |
 | Analyse + plan + generate + validate, 1M-row CSV with target | 2.5 s |
 | Run the generated polars script on 1M rows | 0.9 s |
+| `fe.train()` on a 1M-row CSV (plan, features, 4 models, checks) | 6.7 s |
+| Generated `prep.py` + `train.py` on 1M rows | 1.3 s + 20 s |
 
 ## Examples
 
 - [`examples/phase1_demo.py`](examples/phase1_demo.py): engine choice and code generation at 0.1M / 1M / 5M rows
 - [`examples/phase2_demo.py`](examples/phase2_demo.py): target analysis and cleaning on a 1M-row churn CSV
+- [`examples/phase3_demo.py`](examples/phase3_demo.py): features, model comparison and training on 1M rows
 
 ## Development
 

{flasheda-0.3.0 → flasheda-0.4.0}/README.md

@@ -7,9 +7,9 @@
 [![Tests](https://github.com/sjapanjots/Flash-EDA/actions/workflows/tests.yml/badge.svg)](https://github.com/sjapanjots/Flash-EDA/actions/workflows/tests.yml)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](MIT.License)
 
-**Current version: 0.3.0** · [Changelog](CHANGELOG.md)
+**Current version: 0.4.0** · [Changelog](CHANGELOG.md)
 
-Analyse any dataset in a few seconds, then get **validated cleaning code** that runs on the full data, in the engine that fits its size: pandas, polars, DuckDB or PySpark.
+Analyse any dataset in a few seconds, then get **validated cleaning, feature and training code** that runs on the full data, in the engine that fits its size: pandas, polars, DuckDB or PySpark.
 
 FlashEDA analyses a fixed-size random sample, so analysis time barely changes between 100 thousand and 100 million rows.
 
@@ -18,7 +18,9 @@ FlashEDA analyses a fixed-size random sample, so analysis time barely changes be
 ```bash
 pip install flasheda            # core
 pip install "flasheda[fast]"    # + polars, DuckDB, pyarrow (recommended for big files)
+pip install "flasheda[ml]"      # + scikit-learn, for flasheda.train
 pip install "flasheda[spark]"   # + PySpark
+pip install "flasheda[all]"     # everything except PySpark
 ```
 
 ## Quick start
@@ -34,6 +36,11 @@ report.save_html("report.html")    # or report.save_pdf("report.pdf")
 # 2. Clean: get a script for the full data, already checked on the sample
 code = fe.clean("orders.csv", target="churn", engine="polars")
 code.save("clean_orders.py")       # then: python clean_orders.py
+
+# 3. Model: compare models on the sample, get prep + training scripts
+result = fe.train("orders.csv", target="churn", engine="polars")
+result.show()                      # leaderboard
+result.save("prep.py", "train.py") # python prep.py && python train.py → model.joblib
 ```
 
 ## What it does
@@ -46,6 +53,8 @@ code.save("clean_orders.py")       # then: python clean_orders.py
 | **Correlations** | Pearson (numeric) and Cramér's V (categorical) |
 | **Reports** | console, HTML, PDF, JSON |
 | **Code generation** | one cleaning plan → pandas, polars (lazy, streaming), DuckDB SQL (larger than memory) or PySpark |
+| **Features** | date parts, one-hot / frequency encoding, ratios that beat their inputs |
+| **Models** | baseline, linear, random forest, gradient boosting compared with the right metric and split; training script for the winner |
 | **Validation** | generated code is run on the sample and compared with a reference result before you get it |
 
 ### Cleaning steps
@@ -59,6 +68,14 @@ Each step in the generated code has a comment saying why it was added:
 - add `<col>_was_missing` flags when missingness predicts the target
 - fill nulls (median / mode), log-transform skewed columns, clip outliers
 
+### Features and models
+`fe.train()` (or `report.train()`) adds features on top of the cleaning steps, compares models on the sample, and writes two scripts:
+
+- **`prep.py`**: cleaning + features for the full data, in the chosen engine
+- **`train.py`**: scikit-learn training with cross-validation, using the same metric and split, that saves `model.joblib`
+
+The metric and split follow the target: PR-AUC for imbalanced binary targets, MAE for skewed regression targets, time-based splits when there's a date column, and so on.
+
 ### Engine choice
 
 | Rows | Engine |
@@ -79,11 +96,14 @@ Measured on synthetic messy data:
 | Sample a 1M-row CSV (0.1.x → 0.2.0) | 37.3 s → 0.65 s |
 | Analyse + plan + generate + validate, 1M-row CSV with target | 2.5 s |
 | Run the generated polars script on 1M rows | 0.9 s |
+| `fe.train()` on a 1M-row CSV (plan, features, 4 models, checks) | 6.7 s |
+| Generated `prep.py` + `train.py` on 1M rows | 1.3 s + 20 s |
 
 ## Examples
 
 - [`examples/phase1_demo.py`](examples/phase1_demo.py): engine choice and code generation at 0.1M / 1M / 5M rows
 - [`examples/phase2_demo.py`](examples/phase2_demo.py): target analysis and cleaning on a 1M-row churn CSV
+- [`examples/phase3_demo.py`](examples/phase3_demo.py): features, model comparison and training on 1M rows
 
 ## Development
 

{flasheda-0.3.0 → flasheda-0.4.0}/flasheda/__init__.py

@@ -9,12 +9,13 @@ import pandas as pd
 
 from . import codegen, engine
 from .clean import clean
+from .modeling import train
 from .engine import available_engines, choose_engine, inspect_source, sample, get_sample_size, set_sample_size
 from .analyzers import overview, missing, numeric, categorical, correlation
 from .analyzers import target as target_analyzer
 from .report import EDAReport
 
-__version__ = "0.3.0"
+__version__ = "0.4.0"
 __all__ = [
     "analyze",
     "available_engines",
@@ -24,6 +25,7 @@ __all__ = [
     "get_sample_size",
     "inspect_source",
     "set_sample_size",
+    "train",
     "EDAReport",
 ]
 

{flasheda-0.3.0 → flasheda-0.4.0}/flasheda/clean/__init__.py

@@ -26,6 +26,7 @@ def clean(
     n: Optional[int] = None,
     random_state: int = 42,
     validate: bool = True,
+    features: bool = False,
     **plan_options,
 ):
     """
@@ -35,11 +36,13 @@ def clean(
     target       : column to predict; enables leakage drops, null-target
                    row drops and missing-value flags
     engine       : "pandas" | "polars" | "duckdb" | "pyspark" (default: by size)
+    features     : also add model-ready features (date parts, encodings, ratios)
     plan_options : switch planner phases off, e.g. group_rare=False
     """
     from .. import analyze  # local import: flasheda/__init__ imports this package
 
     report = analyze(source, n=n, target=target, random_state=random_state, save_pdf=False)
-    code = report.codegen(engine, output=output, validate=validate, **plan_options)
+    code = report.codegen(engine, output=output, validate=validate, features=features,
+                          **plan_options)
     code.report = report
     return code

{flasheda-0.3.0 → flasheda-0.4.0}/flasheda/clean/detectors.py

@@ -17,7 +17,6 @@ NUMERIC_TEXT_MIN_RATIO = 0.95    # share of values that must parse as numbers
 LEADING_ZERO_MAX_RATIO = 0.10    # more than this looks like codes (zip, phone)
 RARE_MIN_SHARE = 0.01            # categories below 1% of rows are "rare"
 RARE_MIN_UNIQUE = 10             # only group columns with more distinct values
-RARE_MAX_KEEP = 25
 MISSING_MIN_ROWS = 30
 MISSING_MIN_GAP = 0.10           # class-share gap between missing / present rows
 MISSING_MIN_EFFECT = 0.20        # regression: mean gap in target std units
@@ -78,7 +77,7 @@ def rare_categories(s: pd.Series) -> Optional[Tuple[tuple, str]]:
     shares = values.value_counts(normalize=True)
     if len(shares) <= RARE_MIN_UNIQUE:
         return None
-    keep = shares[shares >= RARE_MIN_SHARE].index[:RARE_MAX_KEEP]
+    keep = shares[shares >= RARE_MIN_SHARE].index   # at most 100 values can each be ≥ 1%
     if len(keep) == 0 or len(keep) == len(shares):
         return None
     other = "other"

{flasheda-0.3.0 → flasheda-0.4.0}/flasheda/codegen/__init__.py

@@ -16,8 +16,9 @@ import pandas as pd
 
 from .planner import plan_from_report
 from .steps import (
-    ClipOutliers, DropColumns, DropDuplicates, DropNullRows, FillNull, GroupRare, Log1p,
-    MissingIndicator, NormalizeText, Plan, ToNumeric,
+    ClipOutliers, DateParts, DropColumns, DropDuplicates, DropNullRows, FillNull,
+    FrequencyEncode, GroupRare, Log1p, MissingIndicator, NormalizeText, OneHot, Plan, Ratio,
+    ToNumeric,
 )
 from .validate import ValidationResult, compare_frames, run_generated, validate_code
 from .writers import WRITERS, get_writer
@@ -66,16 +67,20 @@ def generate(
 
 __all__ = [
     "ClipOutliers",
+    "DateParts",
     "DropColumns",
     "DropDuplicates",
     "DropNullRows",
     "FillNull",
+    "FrequencyEncode",
     "GeneratedCode",
     "GroupRare",
     "Log1p",
     "MissingIndicator",
     "NormalizeText",
+    "OneHot",
     "Plan",
+    "Ratio",
     "ToNumeric",
     "ValidationResult",
     "WRITERS",

{flasheda-0.3.0 → flasheda-0.4.0}/flasheda/codegen/steps.py

@@ -105,15 +105,87 @@ class Log1p:
     reason: str = ""
 
 
+# ── Feature steps ───────────────────────────────────────────────────────────
+
+DATE_PARTS = ("year", "month", "day", "dayofweek", "hour")
+
+
+@dataclass(frozen=True)
+class DateParts:
+    """
+    Add `<column>_<part>` columns from a date/timestamp column.
+    dayofweek is ISO: Monday = 1 ... Sunday = 7. Unparseable values give null.
+    `is_text` says the column holds ISO-8601 text that must be parsed first.
+    """
+
+    column: str
+    parts: tuple
+    names: tuple
+    is_text: bool = True
+    reason: str = ""
+
+
+@dataclass(frozen=True)
+class Ratio:
+    """Add `name` = numerator / denominator (null where the denominator is 0)."""
+
+    numerator: str
+    denominator: str
+    name: str
+    reason: str = ""
+
+
+@dataclass(frozen=True)
+class OneHot:
+    """Add one boolean column per value: names[i] = (column == values[i])."""
+
+    column: str
+    values: tuple
+    names: tuple
+    reason: str = ""
+
+
+@dataclass(frozen=True)
+class FrequencyEncode:
+    """Replace each value with its share of the sample; unseen or null → `default`."""
+
+    column: str
+    mapping: tuple           # ((value, share), ...)
+    default: float = 0.0
+    reason: str = ""
+
+
 ROW_STEPS = (DropColumns, DropDuplicates, DropNullRows)
-COLUMN_STEPS = (ToNumeric, NormalizeText, GroupRare, MissingIndicator, FillNull, ClipOutliers, Log1p)
+COLUMN_STEPS = (
+    ToNumeric, NormalizeText, GroupRare, MissingIndicator, FillNull, ClipOutliers, Log1p,
+    DateParts, Ratio, OneHot, FrequencyEncode,
+)
 
 Step = Union[
     DropColumns, DropDuplicates, DropNullRows,
     ToNumeric, NormalizeText, GroupRare, MissingIndicator, FillNull, ClipOutliers, Log1p,
+    DateParts, Ratio, OneHot, FrequencyEncode,
 ]
 
 
+def inputs(step) -> tuple:
+    """Columns a column step reads."""
+    if isinstance(step, Ratio):
+        return (step.numerator, step.denominator)
+    return (step.column,)
+
+
+def outputs(step) -> tuple:
+    """New columns a column step adds (empty when it only replaces values)."""
+    if isinstance(step, MissingIndicator):
+        return (step.name,)
+    if isinstance(step, Ratio):
+        return (step.name,)
+    if isinstance(step, (DateParts, OneHot)):
+        return step.names
+    return ()
+
+
 @dataclass
 class Plan:
     steps: List[Step] = field(default_factory=list)
@@ -134,13 +206,20 @@ class Plan:
         them together; row steps keep their position in the order.
         """
         pending: List[Step] = []
+        added: set = set()
         for step in self.steps:
             if isinstance(step, COLUMN_STEPS):
+                # A step reading a column added in this segment needs that
+                # column to exist first, so it starts a new segment.
+                if added.intersection(inputs(step)):
+                    yield "columns", pending
+                    pending, added = [], set()
                 pending.append(step)
+                added.update(outputs(step))
                 continue
             if pending:
                 yield "columns", pending
-                pending = []
+                pending, added = [], set()
             yield "row", step
         if pending:
             yield "columns", pending
@@ -175,6 +254,23 @@ class Plan:
             elif isinstance(step, Log1p):
                 x = out[step.column]
                 out[step.column] = np.sign(x) * np.log1p(np.abs(x))
+            elif isinstance(step, DateParts):
+                ts = out[step.column]
+                if step.is_text:
+                    ts = pd.to_datetime(ts, errors="coerce", format="ISO8601")
+                for part, name in zip(step.parts, step.names):
+                    value = ts.dt.dayofweek + 1 if part == "dayofweek" else getattr(ts.dt, part)
+                    out[name] = value.astype("float64")
+            elif isinstance(step, Ratio):
+                den = out[step.denominator].astype("float64")
+                out[step.name] = out[step.numerator].astype("float64") / den.where(den != 0)
+            elif isinstance(step, OneHot):
+                s = out[step.column]
+                for value, name in zip(step.values, step.names):
+                    out[name] = s.eq(value).fillna(False).astype(bool)
+            elif isinstance(step, FrequencyEncode):
+                mapping = dict(step.mapping)
+                out[step.column] = out[step.column].map(mapping).astype("float64").fillna(step.default)
         return out
 
 
@@ -197,4 +293,12 @@ def _label(step: Step) -> str:
         return f"fill nulls in {step.column!r} with {step.value!r}"
     if isinstance(step, ClipOutliers):
         return f"clip {step.column!r} to [{step.lower}, {step.upper}]"
-    return f"log transform {step.column!r}"
+    if isinstance(step, Log1p):
+        return f"log transform {step.column!r}"
+    if isinstance(step, DateParts):
+        return f"add {', '.join(step.parts)} from {step.column!r}"
+    if isinstance(step, Ratio):
+        return f"add {step.name!r} = {step.numerator!r} / {step.denominator!r}"
+    if isinstance(step, OneHot):
+        return f"one-hot encode {step.column!r} ({len(step.values)} columns)"
+    return f"frequency-encode {step.column!r}"

{flasheda-0.3.0 → flasheda-0.4.0}/flasheda/codegen/writers/duckdb_writer.py

@@ -3,8 +3,9 @@ from __future__ import annotations
 from typing import List, Optional
 
 from ..steps import (
-    ClipOutliers, DropColumns, DropDuplicates, DropNullRows, FillNull, GroupRare, Log1p,
-    MissingIndicator, NormalizeText, Plan, Step, ToNumeric,
+    ClipOutliers, DateParts, DropColumns, DropDuplicates, DropNullRows, FillNull,
+    FrequencyEncode, GroupRare, Log1p, MissingIndicator, NormalizeText, OneHot, Plan, Ratio, Step,
+    ToNumeric,
 )
 from ...engine.source import DUCKDB_CSV_TYPES
 from .base import Writer, source_format
@@ -64,9 +65,31 @@ class DuckDBWriter(Writer):
     def _columns(steps: List[Step]) -> str:
         replaced = {}
         added = {}
+        def current(name: str) -> str:
+            return replaced.get(name, ident(name))
+
         for step in steps:
-            col = replaced.get(step.column, ident(step.column))
-            if isinstance(step, ToNumeric):
+            if isinstance(step, Ratio):
+                num = f"CAST({current(step.numerator)} AS DOUBLE)"
+                den = f"CAST({current(step.denominator)} AS DOUBLE)"
+                added[step.name] = f"{num} / NULLIF({den}, 0)"
+                continue
+            col = current(step.column)
+            if isinstance(step, DateParts):
+                ts = f"TRY_CAST({col} AS TIMESTAMP)" if step.is_text else col
+                for part, name in zip(step.parts, step.names):
+                    fn = {"year": "year", "month": "month", "day": "day",
+                          "dayofweek": "isodow", "hour": "hour"}[part]
+                    added[name] = f"CAST({fn}({ts}) AS DOUBLE)"
+                continue
+            if isinstance(step, OneHot):
+                for value, name in zip(step.values, step.names):
+                    added[name] = f"coalesce({col} = {literal(value)}, false)"
+                continue
+            if isinstance(step, FrequencyEncode):
+                cases = " ".join(f"WHEN {literal(v)} THEN {literal(float(f))}" for v, f in step.mapping)
+                col = f"CASE {col} {cases} ELSE {literal(float(step.default))} END"
+            elif isinstance(step, ToNumeric):
                 col = f"TRY_CAST(trim(CAST({col} AS VARCHAR), ' ') AS DOUBLE)"
             elif isinstance(step, NormalizeText):
                 col = f"lower(trim({col}, ' '))"

{flasheda-0.3.0 → flasheda-0.4.0}/flasheda/codegen/writers/pandas_writer.py

@@ -3,8 +3,9 @@ from __future__ import annotations
 from typing import Optional
 
 from ..steps import (
-    ClipOutliers, DropColumns, DropDuplicates, DropNullRows, FillNull, GroupRare, Log1p,
-    MissingIndicator, NormalizeText, Plan, ToNumeric,
+    ClipOutliers, DateParts, DropColumns, DropDuplicates, DropNullRows, FillNull,
+    FrequencyEncode, GroupRare, Log1p, MissingIndicator, NormalizeText, OneHot, Plan, Ratio,
+    ToNumeric,
 )
 from .base import Writer, source_format
 
@@ -20,7 +21,7 @@ class PandasWriter(Writer):
         body = []
         for step in plan:
             body.append(self.comment(step))
-            body.append(self._line(step))
+            body.extend(self._line(step).split("\n"))
         body.append("return df")
         return (
             "def transform(df: pd.DataFrame) -> pd.DataFrame:\n"
@@ -34,6 +35,10 @@ class PandasWriter(Writer):
             return f"df = df.drop(columns={list(step.columns)!r})"
         if isinstance(step, DropDuplicates):
             return "df = df.drop_duplicates().reset_index(drop=True)"
+        if isinstance(step, Ratio):
+            den = f'df[{step.denominator!r}].astype("float64")'
+            return (f'df[{step.name!r}] = df[{step.numerator!r}].astype("float64") / '
+                    f"{den}.where({den} != 0)")
         c = f"df[{step.column!r}]"
         if isinstance(step, DropNullRows):
             return f"df = df[{c}.notna()].reset_index(drop=True)"
@@ -52,6 +57,19 @@ class PandasWriter(Writer):
             return f"{c} = {c}.clip({step.lower!r}, {step.upper!r})"
         if isinstance(step, Log1p):
             return f"{c} = np.sign({c}) * np.log1p(np.abs({c}))"
+        if isinstance(step, DateParts):
+            src = f'pd.to_datetime({c}, errors="coerce", format="ISO8601")' if step.is_text else c
+            lines = [f"ts = {src}"]
+            for part, name in zip(step.parts, step.names):
+                value = "ts.dt.dayofweek + 1" if part == "dayofweek" else f"ts.dt.{part}"
+                lines.append(f'df[{name!r}] = ({value}).astype("float64")')
+            return "\n".join(lines)
+        if isinstance(step, OneHot):
+            return "\n".join(
+                f"df[{name!r}] = {c}.eq({value!r}).fillna(False).astype(bool)"
+                for value, name in zip(step.values, step.names))
+        if isinstance(step, FrequencyEncode):
+            return f'{c} = {c}.map({dict(step.mapping)!r}).astype("float64").fillna({step.default!r})'
         raise TypeError(f"Unsupported step {step!r}")
 
     def main(self, source: Optional[str]) -> str:

{flasheda-0.3.0 → flasheda-0.4.0}/flasheda/codegen/writers/polars_writer.py

@@ -3,8 +3,9 @@ from __future__ import annotations
 from typing import Dict, List, Optional
 
 from ..steps import (
-    ClipOutliers, DropColumns, DropDuplicates, DropNullRows, FillNull, GroupRare, Log1p,
-    MissingIndicator, NormalizeText, Plan, ToNumeric,
+    ClipOutliers, DateParts, DropColumns, DropDuplicates, DropNullRows, FillNull,
+    FrequencyEncode, GroupRare, Log1p, MissingIndicator, NormalizeText, OneHot, Plan, Ratio,
+    ToNumeric,
 )
 from .base import Writer, source_format
 
@@ -47,10 +48,36 @@ class PolarsWriter(Writer):
         # same column are composed into a single expression, in order.
         exprs: Dict[str, str] = {}
         reasons: Dict[str, List[str]] = {}
+        def current(name: str) -> str:
+            return exprs.get(name, f"pl.col({name!r})")
+
+        def add(name: str, expr: str, reason: str) -> None:
+            exprs[name] = f"({expr}).alias({name!r})"
+            reasons.setdefault(name, []).append(reason)
+
         for step in steps:
-            e = exprs.get(step.column, f"pl.col({step.column!r})")
+            if isinstance(step, Ratio):
+                num = f"{current(step.numerator)}.cast(pl.Float64)"
+                den = f"{current(step.denominator)}.cast(pl.Float64)"
+                add(step.name, f"pl.when({den} != 0).then({num} / {den})", step.reason)
+                continue
+            e = current(step.column)
             out = step.column
-            if isinstance(step, ToNumeric):
+            if isinstance(step, DateParts):
+                ts = f"{e}.str.to_datetime(strict=False)" if step.is_text else e
+                for part, name in zip(step.parts, step.names):
+                    fn = {"year": "year", "month": "month", "day": "day",
+                          "dayofweek": "weekday", "hour": "hour"}[part]
+                    add(name, f"{ts}.dt.{fn}().cast(pl.Float64)", step.reason)
+                continue
+            if isinstance(step, OneHot):
+                for value, name in zip(step.values, step.names):
+                    add(name, f"({e} == {value!r}).fill_null(False)", step.reason)
+                continue
+            if isinstance(step, FrequencyEncode):
+                e = (f"{e}.replace_strict({dict(step.mapping)!r}, default={step.default!r}, "
+                     f"return_dtype=pl.Float64).fill_null({step.default!r})")
+            elif isinstance(step, ToNumeric):
                 e += '.cast(pl.String).str.strip_chars(" ").cast(pl.Float64, strict=False)'
             elif isinstance(step, NormalizeText):
                 e += '.str.strip_chars(" ").str.to_lowercase()'

{flasheda-0.3.0 → flasheda-0.4.0}/flasheda/codegen/writers/pyspark_writer.py

@@ -3,8 +3,9 @@ from __future__ import annotations
 from typing import Optional
 
 from ..steps import (
-    ClipOutliers, DropColumns, DropDuplicates, DropNullRows, FillNull, GroupRare, Log1p,
-    MissingIndicator, NormalizeText, Plan, ToNumeric,
+    ClipOutliers, DateParts, DropColumns, DropDuplicates, DropNullRows, FillNull,
+    FrequencyEncode, GroupRare, Log1p, MissingIndicator, NormalizeText, OneHot, Plan, Ratio,
+    ToNumeric,
 )
 from .base import Writer, source_format
 
@@ -26,7 +27,7 @@ class PySparkWriter(Writer):
         body = []
         for step in plan:
             body.append(self.comment(step))
-            body.append(self._line(step))
+            body.extend(self._line(step).split("\n"))
         body.append("return df")
         return "def transform(df: DataFrame) -> DataFrame:\n" + self.indent(
             [line for line in body if line]
@@ -38,11 +39,30 @@ class PySparkWriter(Writer):
             return f"df = df.drop(*{list(step.columns)!r})"
         if isinstance(step, DropDuplicates):
             return "df = df.dropDuplicates()"
+        if isinstance(step, Ratio):
+            den = f'{col(step.denominator)}.cast("double")'
+            return (f"df = df.withColumn({step.name!r}, "
+                    f'{col(step.numerator)}.cast("double") / F.when({den} != 0, {den}))')
         c = col(step.column)
         if isinstance(step, DropNullRows):
             return f"df = df.filter({c}.isNotNull())"
         if isinstance(step, MissingIndicator):
             return f"df = df.withColumn({step.name!r}, {c}.isNull())"
+        if isinstance(step, DateParts):
+            ts = f"F.to_timestamp({c})" if step.is_text else c
+            spark = {"year": "F.year", "month": "F.month", "day": "F.dayofmonth", "hour": "F.hour"}
+            lines = []
+            for part, name in zip(step.parts, step.names):
+                if part == "dayofweek":   # Spark: Sunday = 1; ISO: Monday = 1
+                    value = f"((F.dayofweek({ts}) + 5) % 7) + 1"
+                else:
+                    value = f"{spark[part]}({ts})"
+                lines.append(f'df = df.withColumn({name!r}, ({value}).cast("double"))')
+            return "\n".join(lines)
+        if isinstance(step, OneHot):
+            return "\n".join(
+                f"df = df.withColumn({name!r}, F.coalesce({c} == {value!r}, F.lit(False)))"
+                for value, name in zip(step.values, step.names))
         if isinstance(step, ToNumeric):
             e = f'F.trim({c}.cast("string")).cast("double")'
         elif isinstance(step, NormalizeText):
@@ -57,6 +77,9 @@ class PySparkWriter(Writer):
                  f".when({c} > {step.upper!r}, F.lit({step.upper!r})).otherwise({c})")
         elif isinstance(step, Log1p):
             e = f"F.signum({c}) * F.log1p(F.abs({c}))"
+        elif isinstance(step, FrequencyEncode):
+            pairs = ", ".join(f"F.lit({v!r}), F.lit({float(f)!r})" for v, f in step.mapping)
+            e = f"F.coalesce(F.create_map({pairs})[{c}], F.lit({step.default!r}))"
         else:
             raise TypeError(f"Unsupported step {step!r}")
         return f"df = df.withColumn({step.column!r}, {e})"