flasheda 0.3.0__tar.gz → 0.5.0__tar.gz

{flasheda-0.3.0 → flasheda-0.5.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: flasheda
-Version: 0.3.0
-Summary: Fixed-time EDA for any dataset size, plus validated cleaning code for pandas, polars, DuckDB and PySpark.
+Version: 0.5.0
+Summary: Fixed-time EDA for any dataset size, plus validated cleaning, feature and training code for pandas, polars, DuckDB and PySpark.
 Home-page: https://github.com/sjapanjots/Flash-EDA
 Author: Japanjot Singh
 Author-email: sjapanjots@gmail.com
@@ -9,7 +9,7 @@ License: MIT
 Project-URL: Source, https://github.com/sjapanjots/Flash-EDA
 Project-URL: Changelog, https://github.com/sjapanjots/Flash-EDA/blob/main/CHANGELOG.md
 Project-URL: Issues, https://github.com/sjapanjots/Flash-EDA/issues
-Keywords: eda,data-cleaning,pandas,polars,duckdb,pyspark,code-generation
+Keywords: eda,data-cleaning,pandas,polars,duckdb,pyspark,code-generation,feature-engineering,automl,data-quality,data-drift
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Science/Research
 Classifier: Intended Audience :: Developers
@@ -32,11 +32,26 @@ Requires-Dist: polars>=0.20; extra == "fast"
 Requires-Dist: pyarrow>=12; extra == "fast"
 Provides-Extra: spark
 Requires-Dist: pyspark>=3.4; extra == "spark"
+Provides-Extra: ml
+Requires-Dist: scikit-learn>=1.2; extra == "ml"
+Requires-Dist: pyarrow>=12; extra == "ml"
+Provides-Extra: sql
+Requires-Dist: sqlalchemy>=1.4; extra == "sql"
+Requires-Dist: pyarrow>=12; extra == "sql"
+Provides-Extra: all
+Requires-Dist: duckdb>=0.10; extra == "all"
+Requires-Dist: polars>=0.20; extra == "all"
+Requires-Dist: pyarrow>=12; extra == "all"
+Requires-Dist: scikit-learn>=1.2; extra == "all"
+Requires-Dist: sqlalchemy>=1.4; extra == "all"
 Provides-Extra: dev
 Requires-Dist: pytest>=7; extra == "dev"
 Requires-Dist: duckdb>=0.10; extra == "dev"
 Requires-Dist: polars>=0.20; extra == "dev"
 Requires-Dist: pyarrow>=12; extra == "dev"
+Requires-Dist: scikit-learn>=1.2; extra == "dev"
+Requires-Dist: pandera; extra == "dev"
+Requires-Dist: sqlalchemy>=1.4; extra == "dev"
 Requires-Dist: build; extra == "dev"
 Requires-Dist: twine; extra == "dev"
 Dynamic: author
@@ -63,9 +78,9 @@ Dynamic: summary
 [![Tests](https://github.com/sjapanjots/Flash-EDA/actions/workflows/tests.yml/badge.svg)](https://github.com/sjapanjots/Flash-EDA/actions/workflows/tests.yml)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](MIT.License)
 
-**Current version: 0.3.0** · [Changelog](CHANGELOG.md)
+**Current version: 0.5.0** · [Changelog](CHANGELOG.md)
 
-Analyse any dataset in a few seconds, then get **validated cleaning code** that runs on the full data, in the engine that fits its size: pandas, polars, DuckDB or PySpark.
+Analyse any dataset in a few seconds, then get **validated cleaning, feature and training code** that runs on the full data, in the engine that fits its size: pandas, polars, DuckDB or PySpark.
 
 FlashEDA analyses a fixed-size random sample, so analysis time barely changes between 100 thousand and 100 million rows.
 
@@ -74,7 +89,10 @@ FlashEDA analyses a fixed-size random sample, so analysis time barely changes be
 ```bash
 pip install flasheda            # core
 pip install "flasheda[fast]"    # + polars, DuckDB, pyarrow (recommended for big files)
+pip install "flasheda[ml]"      # + scikit-learn, for flasheda.train
+pip install "flasheda[sql]"     # + SQLAlchemy, for database tables
 pip install "flasheda[spark]"   # + PySpark
+pip install "flasheda[all]"     # everything except PySpark
 ```
 
 ## Quick start
@@ -90,6 +108,23 @@ report.save_html("report.html")    # or report.save_pdf("report.pdf")
 # 2. Clean: get a script for the full data, already checked on the sample
 code = fe.clean("orders.csv", target="churn", engine="polars")
 code.save("clean_orders.py")       # then: python clean_orders.py
+
+# 3. Model: compare models on the sample, get prep + training scripts
+result = fe.train("orders.csv", target="churn", engine="polars")
+result.show()                      # leaderboard
+result.save("prep.py", "train.py") # python prep.py && python train.py → model.joblib
+
+# 4. Guard: learn rules from good data, check every new file
+contract = fe.analyze("orders_jan.csv").contract()
+fe.check("orders_feb.csv", contract).show()
+open("check.py", "w").write(contract.script())   # python check.py new.csv → exit 1 on failure
+
+# 5. Compare: what changed between two datasets
+fe.compare("orders_jan.csv", "orders_feb.csv").show()
+
+# Databases work like files
+src = fe.sql("postgresql://user:pass@host/shop", table="orders")
+fe.analyze(src, target="churn")
 ```
 
 ## What it does
@@ -102,6 +137,11 @@ code.save("clean_orders.py")       # then: python clean_orders.py
 | **Correlations** | Pearson (numeric) and Cramér's V (categorical) |
 | **Reports** | console, HTML, PDF, JSON |
 | **Code generation** | one cleaning plan → pandas, polars (lazy, streaming), DuckDB SQL (larger than memory) or PySpark |
+| **Features** | date parts, one-hot / frequency encoding, ratios that beat their inputs |
+| **Models** | baseline, linear, random forest, gradient boosting compared with the right metric and split; training script for the winner |
+| **Data contracts** | rules learned from good data; full-data checks; standalone check script for CI; pandera export |
+| **Drift** | schema changes and per-column drift (PSI, KS, new categories, null-rate change) between two datasets |
+| **Databases** | any SQLAlchemy URL; sampling runs in the database; passwords never stored |
 | **Validation** | generated code is run on the sample and compared with a reference result before you get it |
 
 ### Cleaning steps
@@ -115,6 +155,21 @@ Each step in the generated code has a comment saying why it was added:
 - add `<col>_was_missing` flags when missingness predicts the target
 - fill nulls (median / mode), log-transform skewed columns, clip outliers
 
+### Features and models
+`fe.train()` (or `report.train()`) adds features on top of the cleaning steps, compares models on the sample, and writes two scripts:
+
+- **`prep.py`**: cleaning + features for the full data, in the chosen engine
+- **`train.py`**: scikit-learn training with cross-validation, using the same metric and split, that saves `model.joblib`
+
+The metric and split follow the target: PR-AUC for imbalanced binary targets, MAE for skewed regression targets, time-based splits when there's a date column, and so on.
+
+### Data contracts and drift
+A **contract** describes what good data looks like, learned from a sample with tolerances so normal variation passes:
+type, missing %, 1st/99th percentile range (catches unit changes like ×100), no negatives, allowed values, uniqueness of IDs.
+`fe.check()` checks every row of a file in one DuckDB pass; `contract.script()` writes the same check as a standalone script that exits with code 1 on failure, ready for a pipeline or CI job.
+
+`fe.compare(old, new)` reports added/removed columns, type changes and how much each column shifted (PSI: < 0.1 stable, 0.1–0.25 moderate, ≥ 0.25 major). Dates and IDs are shown as range shifts, since they always move.
+
 ### Engine choice
 
 | Rows | Engine |
@@ -135,11 +190,18 @@ Measured on synthetic messy data:
 | Sample a 1M-row CSV (0.1.x → 0.2.0) | 37.3 s → 0.65 s |
 | Analyse + plan + generate + validate, 1M-row CSV with target | 2.5 s |
 | Run the generated polars script on 1M rows | 0.9 s |
+| `fe.train()` on a 1M-row CSV (plan, features, 4 models, checks) | 6.7 s |
+| Generated `prep.py` + `train.py` on 1M rows | 1.3 s + 20 s |
+| Learn a contract from 1M rows / run the check script on 1M rows | 0.8 s / 0.6–0.9 s |
+| Compare two 1M-row files | 0.6 s |
+| Analyse a 1M-row SQLite table (5,000 rows transferred) | 0.3 s |
 
 ## Examples
 
 - [`examples/phase1_demo.py`](examples/phase1_demo.py): engine choice and code generation at 0.1M / 1M / 5M rows
 - [`examples/phase2_demo.py`](examples/phase2_demo.py): target analysis and cleaning on a 1M-row churn CSV
+- [`examples/phase3_demo.py`](examples/phase3_demo.py): features, model comparison and training on 1M rows
+- [`examples/phase4_demo.py`](examples/phase4_demo.py): contracts, check script, drift and a database on 1M rows
 
 ## Development
 

{flasheda-0.3.0 → flasheda-0.5.0}/README.md

@@ -7,9 +7,9 @@
 [![Tests](https://github.com/sjapanjots/Flash-EDA/actions/workflows/tests.yml/badge.svg)](https://github.com/sjapanjots/Flash-EDA/actions/workflows/tests.yml)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](MIT.License)
 
-**Current version: 0.3.0** · [Changelog](CHANGELOG.md)
+**Current version: 0.5.0** · [Changelog](CHANGELOG.md)
 
-Analyse any dataset in a few seconds, then get **validated cleaning code** that runs on the full data, in the engine that fits its size: pandas, polars, DuckDB or PySpark.
+Analyse any dataset in a few seconds, then get **validated cleaning, feature and training code** that runs on the full data, in the engine that fits its size: pandas, polars, DuckDB or PySpark.
 
 FlashEDA analyses a fixed-size random sample, so analysis time barely changes between 100 thousand and 100 million rows.
 
@@ -18,7 +18,10 @@ FlashEDA analyses a fixed-size random sample, so analysis time barely changes be
 ```bash
 pip install flasheda            # core
 pip install "flasheda[fast]"    # + polars, DuckDB, pyarrow (recommended for big files)
+pip install "flasheda[ml]"      # + scikit-learn, for flasheda.train
+pip install "flasheda[sql]"     # + SQLAlchemy, for database tables
 pip install "flasheda[spark]"   # + PySpark
+pip install "flasheda[all]"     # everything except PySpark
 ```
 
 ## Quick start
@@ -34,6 +37,23 @@ report.save_html("report.html")    # or report.save_pdf("report.pdf")
 # 2. Clean: get a script for the full data, already checked on the sample
 code = fe.clean("orders.csv", target="churn", engine="polars")
 code.save("clean_orders.py")       # then: python clean_orders.py
+
+# 3. Model: compare models on the sample, get prep + training scripts
+result = fe.train("orders.csv", target="churn", engine="polars")
+result.show()                      # leaderboard
+result.save("prep.py", "train.py") # python prep.py && python train.py → model.joblib
+
+# 4. Guard: learn rules from good data, check every new file
+contract = fe.analyze("orders_jan.csv").contract()
+fe.check("orders_feb.csv", contract).show()
+open("check.py", "w").write(contract.script())   # python check.py new.csv → exit 1 on failure
+
+# 5. Compare: what changed between two datasets
+fe.compare("orders_jan.csv", "orders_feb.csv").show()
+
+# Databases work like files
+src = fe.sql("postgresql://user:pass@host/shop", table="orders")
+fe.analyze(src, target="churn")
 ```
 
 ## What it does
@@ -46,6 +66,11 @@ code.save("clean_orders.py")       # then: python clean_orders.py
 | **Correlations** | Pearson (numeric) and Cramér's V (categorical) |
 | **Reports** | console, HTML, PDF, JSON |
 | **Code generation** | one cleaning plan → pandas, polars (lazy, streaming), DuckDB SQL (larger than memory) or PySpark |
+| **Features** | date parts, one-hot / frequency encoding, ratios that beat their inputs |
+| **Models** | baseline, linear, random forest, gradient boosting compared with the right metric and split; training script for the winner |
+| **Data contracts** | rules learned from good data; full-data checks; standalone check script for CI; pandera export |
+| **Drift** | schema changes and per-column drift (PSI, KS, new categories, null-rate change) between two datasets |
+| **Databases** | any SQLAlchemy URL; sampling runs in the database; passwords never stored |
 | **Validation** | generated code is run on the sample and compared with a reference result before you get it |
 
 ### Cleaning steps
@@ -59,6 +84,21 @@ Each step in the generated code has a comment saying why it was added:
 - add `<col>_was_missing` flags when missingness predicts the target
 - fill nulls (median / mode), log-transform skewed columns, clip outliers
 
+### Features and models
+`fe.train()` (or `report.train()`) adds features on top of the cleaning steps, compares models on the sample, and writes two scripts:
+
+- **`prep.py`**: cleaning + features for the full data, in the chosen engine
+- **`train.py`**: scikit-learn training with cross-validation, using the same metric and split, that saves `model.joblib`
+
+The metric and split follow the target: PR-AUC for imbalanced binary targets, MAE for skewed regression targets, time-based splits when there's a date column, and so on.
+
+### Data contracts and drift
+A **contract** describes what good data looks like, learned from a sample with tolerances so normal variation passes:
+type, missing %, 1st/99th percentile range (catches unit changes like ×100), no negatives, allowed values, uniqueness of IDs.
+`fe.check()` checks every row of a file in one DuckDB pass; `contract.script()` writes the same check as a standalone script that exits with code 1 on failure, ready for a pipeline or CI job.
+
+`fe.compare(old, new)` reports added/removed columns, type changes and how much each column shifted (PSI: < 0.1 stable, 0.1–0.25 moderate, ≥ 0.25 major). Dates and IDs are shown as range shifts, since they always move.
+
 ### Engine choice
 
 | Rows | Engine |
@@ -79,11 +119,18 @@ Measured on synthetic messy data:
 | Sample a 1M-row CSV (0.1.x → 0.2.0) | 37.3 s → 0.65 s |
 | Analyse + plan + generate + validate, 1M-row CSV with target | 2.5 s |
 | Run the generated polars script on 1M rows | 0.9 s |
+| `fe.train()` on a 1M-row CSV (plan, features, 4 models, checks) | 6.7 s |
+| Generated `prep.py` + `train.py` on 1M rows | 1.3 s + 20 s |
+| Learn a contract from 1M rows / run the check script on 1M rows | 0.8 s / 0.6–0.9 s |
+| Compare two 1M-row files | 0.6 s |
+| Analyse a 1M-row SQLite table (5,000 rows transferred) | 0.3 s |
 
 ## Examples
 
 - [`examples/phase1_demo.py`](examples/phase1_demo.py): engine choice and code generation at 0.1M / 1M / 5M rows
 - [`examples/phase2_demo.py`](examples/phase2_demo.py): target analysis and cleaning on a 1M-row churn CSV
+- [`examples/phase3_demo.py`](examples/phase3_demo.py): features, model comparison and training on 1M rows
+- [`examples/phase4_demo.py`](examples/phase4_demo.py): contracts, check script, drift and a database on 1M rows
 
 ## Development
 

{flasheda-0.3.0 → flasheda-0.5.0}/flasheda/__init__.py

@@ -9,21 +9,28 @@ import pandas as pd
 
 from . import codegen, engine
 from .clean import clean
-from .engine import available_engines, choose_engine, inspect_source, sample, get_sample_size, set_sample_size
+from .modeling import train
+from .contract import check
+from .compare import compare
+from .engine import available_engines, choose_engine, inspect_source, sample, get_sample_size, set_sample_size, sql
 from .analyzers import overview, missing, numeric, categorical, correlation
 from .analyzers import target as target_analyzer
 from .report import EDAReport
 
-__version__ = "0.3.0"
+__version__ = "0.5.0"
 __all__ = [
     "analyze",
     "available_engines",
+    "check",
     "clean",
+    "compare",
     "codegen",
     "engine",
     "get_sample_size",
     "inspect_source",
     "set_sample_size",
+    "sql",
+    "train",
     "EDAReport",
 ]
 
@@ -40,7 +47,8 @@ def analyze(
     target: Optional[str] = None,
 ) -> EDAReport:
     """
-    Analyse a fixed-size sample of `source` (DataFrame, CSV/TSV or Parquet path).
+    Analyse a fixed-size sample of `source`: a DataFrame, a CSV/TSV or Parquet
+    path, or a database table/query from flasheda.sql().
 
     `target` names the column you want to predict; it adds task detection,
     class balance, per-feature predictive power and leakage checks.
@@ -119,7 +127,10 @@ def analyze(
         warnings=warnings,
         target=tgt,
         source=str(info.path) if info.path else None,
-        recommended_engine=choose_engine(info.n_rows, prefer=engine),
+        sql_source=info.sql,
+        # database sources are read with pandas (or exported with .to_parquet())
+        recommended_engine=choose_engine(
+            info.n_rows, prefer=engine or ("pandas" if info.sql is not None else None)),
         sample=sampled,
     )
 

{flasheda-0.3.0 → flasheda-0.5.0}/flasheda/clean/__init__.py

@@ -26,6 +26,7 @@ def clean(
     n: Optional[int] = None,
     random_state: int = 42,
     validate: bool = True,
+    features: bool = False,
     **plan_options,
 ):
     """
@@ -35,11 +36,13 @@ def clean(
     target       : column to predict; enables leakage drops, null-target
                    row drops and missing-value flags
     engine       : "pandas" | "polars" | "duckdb" | "pyspark" (default: by size)
+    features     : also add model-ready features (date parts, encodings, ratios)
     plan_options : switch planner phases off, e.g. group_rare=False
     """
     from .. import analyze  # local import: flasheda/__init__ imports this package
 
     report = analyze(source, n=n, target=target, random_state=random_state, save_pdf=False)
-    code = report.codegen(engine, output=output, validate=validate, **plan_options)
+    code = report.codegen(engine, output=output, validate=validate, features=features,
+                          **plan_options)
     code.report = report
     return code

{flasheda-0.3.0 → flasheda-0.5.0}/flasheda/clean/detectors.py

@@ -17,7 +17,6 @@ NUMERIC_TEXT_MIN_RATIO = 0.95    # share of values that must parse as numbers
 LEADING_ZERO_MAX_RATIO = 0.10    # more than this looks like codes (zip, phone)
 RARE_MIN_SHARE = 0.01            # categories below 1% of rows are "rare"
 RARE_MIN_UNIQUE = 10             # only group columns with more distinct values
-RARE_MAX_KEEP = 25
 MISSING_MIN_ROWS = 30
 MISSING_MIN_GAP = 0.10           # class-share gap between missing / present rows
 MISSING_MIN_EFFECT = 0.20        # regression: mean gap in target std units
@@ -78,7 +77,7 @@ def rare_categories(s: pd.Series) -> Optional[Tuple[tuple, str]]:
     shares = values.value_counts(normalize=True)
     if len(shares) <= RARE_MIN_UNIQUE:
         return None
-    keep = shares[shares >= RARE_MIN_SHARE].index[:RARE_MAX_KEEP]
+    keep = shares[shares >= RARE_MIN_SHARE].index   # at most 100 values can each be ≥ 1%
     if len(keep) == 0 or len(keep) == len(shares):
         return None
     other = "other"

{flasheda-0.3.0 → flasheda-0.5.0}/flasheda/codegen/__init__.py

@@ -16,8 +16,9 @@ import pandas as pd
 
 from .planner import plan_from_report
 from .steps import (
-    ClipOutliers, DropColumns, DropDuplicates, DropNullRows, FillNull, GroupRare, Log1p,
-    MissingIndicator, NormalizeText, Plan, ToNumeric,
+    ClipOutliers, DateParts, DropColumns, DropDuplicates, DropNullRows, FillNull,
+    FrequencyEncode, GroupRare, Log1p, MissingIndicator, NormalizeText, OneHot, Plan, Ratio,
+    ToNumeric,
 )
 from .validate import ValidationResult, compare_frames, run_generated, validate_code
 from .writers import WRITERS, get_writer
@@ -57,25 +58,31 @@ def generate(
     output: str = "cleaned.parquet",
     sample: Optional[pd.DataFrame] = None,
     columns: Optional[List[str]] = None,
+    sql: Any = None,
 ) -> GeneratedCode:
+    """`sql` is a flasheda.sql() source to read from instead of a file."""
     if columns is None and sample is not None:
         columns = [str(c) for c in sample.columns]
-    code = get_writer(engine).render(plan, source, output, columns)
+    code = get_writer(engine).render(plan, source, output, columns, sql=sql)
     return GeneratedCode(engine, code, plan, sample)
 
 
 __all__ = [
     "ClipOutliers",
+    "DateParts",
     "DropColumns",
     "DropDuplicates",
     "DropNullRows",
     "FillNull",
+    "FrequencyEncode",
     "GeneratedCode",
     "GroupRare",
     "Log1p",
     "MissingIndicator",
     "NormalizeText",
+    "OneHot",
     "Plan",
+    "Ratio",
     "ToNumeric",
     "ValidationResult",
     "WRITERS",

{flasheda-0.3.0 → flasheda-0.5.0}/flasheda/codegen/steps.py

@@ -105,15 +105,87 @@ class Log1p:
     reason: str = ""
 
 
+# ── Feature steps ───────────────────────────────────────────────────────────
+
+DATE_PARTS = ("year", "month", "day", "dayofweek", "hour")
+
+
+@dataclass(frozen=True)
+class DateParts:
+    """
+    Add `<column>_<part>` columns from a date/timestamp column.
+    dayofweek is ISO: Monday = 1 ... Sunday = 7. Unparseable values give null.
+    `is_text` says the column holds ISO-8601 text that must be parsed first.
+    """
+
+    column: str
+    parts: tuple
+    names: tuple
+    is_text: bool = True
+    reason: str = ""
+
+
+@dataclass(frozen=True)
+class Ratio:
+    """Add `name` = numerator / denominator (null where the denominator is 0)."""
+
+    numerator: str
+    denominator: str
+    name: str
+    reason: str = ""
+
+
+@dataclass(frozen=True)
+class OneHot:
+    """Add one boolean column per value: names[i] = (column == values[i])."""
+
+    column: str
+    values: tuple
+    names: tuple
+    reason: str = ""
+
+
+@dataclass(frozen=True)
+class FrequencyEncode:
+    """Replace each value with its share of the sample; unseen or null → `default`."""
+
+    column: str
+    mapping: tuple           # ((value, share), ...)
+    default: float = 0.0
+    reason: str = ""
+
+
 ROW_STEPS = (DropColumns, DropDuplicates, DropNullRows)
-COLUMN_STEPS = (ToNumeric, NormalizeText, GroupRare, MissingIndicator, FillNull, ClipOutliers, Log1p)
+COLUMN_STEPS = (
+    ToNumeric, NormalizeText, GroupRare, MissingIndicator, FillNull, ClipOutliers, Log1p,
+    DateParts, Ratio, OneHot, FrequencyEncode,
+)
 
 Step = Union[
     DropColumns, DropDuplicates, DropNullRows,
     ToNumeric, NormalizeText, GroupRare, MissingIndicator, FillNull, ClipOutliers, Log1p,
+    DateParts, Ratio, OneHot, FrequencyEncode,
 ]
 
 
+def inputs(step) -> tuple:
+    """Columns a column step reads."""
+    if isinstance(step, Ratio):
+        return (step.numerator, step.denominator)
+    return (step.column,)
+
+
+def outputs(step) -> tuple:
+    """New columns a column step adds (empty when it only replaces values)."""
+    if isinstance(step, MissingIndicator):
+        return (step.name,)
+    if isinstance(step, Ratio):
+        return (step.name,)
+    if isinstance(step, (DateParts, OneHot)):
+        return step.names
+    return ()
+
+
 @dataclass
 class Plan:
     steps: List[Step] = field(default_factory=list)
@@ -134,13 +206,20 @@ class Plan:
         them together; row steps keep their position in the order.
         """
         pending: List[Step] = []
+        added: set = set()
         for step in self.steps:
             if isinstance(step, COLUMN_STEPS):
+                # A step reading a column added in this segment needs that
+                # column to exist first, so it starts a new segment.
+                if added.intersection(inputs(step)):
+                    yield "columns", pending
+                    pending, added = [], set()
                 pending.append(step)
+                added.update(outputs(step))
                 continue
             if pending:
                 yield "columns", pending
-                pending = []
+                pending, added = [], set()
             yield "row", step
         if pending:
             yield "columns", pending
@@ -175,6 +254,23 @@ class Plan:
             elif isinstance(step, Log1p):
                 x = out[step.column]
                 out[step.column] = np.sign(x) * np.log1p(np.abs(x))
+            elif isinstance(step, DateParts):
+                ts = out[step.column]
+                if step.is_text:
+                    ts = pd.to_datetime(ts, errors="coerce", format="ISO8601")
+                for part, name in zip(step.parts, step.names):
+                    value = ts.dt.dayofweek + 1 if part == "dayofweek" else getattr(ts.dt, part)
+                    out[name] = value.astype("float64")
+            elif isinstance(step, Ratio):
+                den = out[step.denominator].astype("float64")
+                out[step.name] = out[step.numerator].astype("float64") / den.where(den != 0)
+            elif isinstance(step, OneHot):
+                s = out[step.column]
+                for value, name in zip(step.values, step.names):
+                    out[name] = s.eq(value).fillna(False).astype(bool)
+            elif isinstance(step, FrequencyEncode):
+                mapping = dict(step.mapping)
+                out[step.column] = out[step.column].map(mapping).astype("float64").fillna(step.default)
         return out
 
 
@@ -197,4 +293,12 @@ def _label(step: Step) -> str:
         return f"fill nulls in {step.column!r} with {step.value!r}"
     if isinstance(step, ClipOutliers):
         return f"clip {step.column!r} to [{step.lower}, {step.upper}]"
-    return f"log transform {step.column!r}"
+    if isinstance(step, Log1p):
+        return f"log transform {step.column!r}"
+    if isinstance(step, DateParts):
+        return f"add {', '.join(step.parts)} from {step.column!r}"
+    if isinstance(step, Ratio):
+        return f"add {step.name!r} = {step.numerator!r} / {step.denominator!r}"
+    if isinstance(step, OneHot):
+        return f"one-hot encode {step.column!r} ({len(step.values)} columns)"
+    return f"frequency-encode {step.column!r}"

{flasheda-0.3.0 → flasheda-0.5.0}/flasheda/codegen/writers/base.py

@@ -14,7 +14,7 @@ from __future__ import annotations
 import textwrap
 from abc import ABC, abstractmethod
 from pathlib import Path
-from typing import List, Optional
+from typing import Any, List, Optional
 
 from ..steps import Plan, Step
 
@@ -23,6 +23,8 @@ class Writer(ABC):
     engine: str = ""
     install_hint: str = ""
     columns: Optional[List[str]] = None
+    sql: Any = None
+    reads_sql = False           # writers that can load from a database URL
 
     def render(
         self,
@@ -30,9 +32,17 @@ class Writer(ABC):
         source: Optional[str],
         output: str,
         columns: Optional[List[str]] = None,
+        sql: Any = None,
     ) -> str:
-        """`columns` are the source's column names, when known."""
+        """`columns` are the source's column names, when known; `sql` a database source."""
         self.columns = list(columns) if columns is not None else None
+        self.sql = sql
+        if sql is not None and not self.reads_sql:
+            raise ValueError(
+                f"The {self.engine} writer can't read from a database directly. Use engine='pandas', "
+                "or export first: flasheda.sql(...).to_parquet('data.parquet'), then analyze that file.")
+        if sql is not None:
+            source = sql.display
         parts = [
             self._docstring(plan, source),
             self.imports(),

{flasheda-0.3.0 → flasheda-0.5.0}/flasheda/codegen/writers/duckdb_writer.py

@@ -3,8 +3,9 @@ from __future__ import annotations
 from typing import List, Optional
 
 from ..steps import (
-    ClipOutliers, DropColumns, DropDuplicates, DropNullRows, FillNull, GroupRare, Log1p,
-    MissingIndicator, NormalizeText, Plan, Step, ToNumeric,
+    ClipOutliers, DateParts, DropColumns, DropDuplicates, DropNullRows, FillNull,
+    FrequencyEncode, GroupRare, Log1p, MissingIndicator, NormalizeText, OneHot, Plan, Ratio, Step,
+    ToNumeric,
 )
 from ...engine.source import DUCKDB_CSV_TYPES
 from .base import Writer, source_format
@@ -64,9 +65,31 @@ class DuckDBWriter(Writer):
     def _columns(steps: List[Step]) -> str:
         replaced = {}
         added = {}
+        def current(name: str) -> str:
+            return replaced.get(name, ident(name))
+
         for step in steps:
-            col = replaced.get(step.column, ident(step.column))
-            if isinstance(step, ToNumeric):
+            if isinstance(step, Ratio):
+                num = f"CAST({current(step.numerator)} AS DOUBLE)"
+                den = f"CAST({current(step.denominator)} AS DOUBLE)"
+                added[step.name] = f"{num} / NULLIF({den}, 0)"
+                continue
+            col = current(step.column)
+            if isinstance(step, DateParts):
+                ts = f"TRY_CAST({col} AS TIMESTAMP)" if step.is_text else col
+                for part, name in zip(step.parts, step.names):
+                    fn = {"year": "year", "month": "month", "day": "day",
+                          "dayofweek": "isodow", "hour": "hour"}[part]
+                    added[name] = f"CAST({fn}({ts}) AS DOUBLE)"
+                continue
+            if isinstance(step, OneHot):
+                for value, name in zip(step.values, step.names):
+                    added[name] = f"coalesce({col} = {literal(value)}, false)"
+                continue
+            if isinstance(step, FrequencyEncode):
+                cases = " ".join(f"WHEN {literal(v)} THEN {literal(float(f))}" for v, f in step.mapping)
+                col = f"CASE {col} {cases} ELSE {literal(float(step.default))} END"
+            elif isinstance(step, ToNumeric):
                 col = f"TRY_CAST(trim(CAST({col} AS VARCHAR), ' ') AS DOUBLE)"
             elif isinstance(step, NormalizeText):
                 col = f"lower(trim({col}, ' '))"