dataframe-toolkit 0.8.0__tar.gz → 0.9.1__tar.gz

{dataframe_toolkit-0.8.0 → dataframe_toolkit-0.9.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: dataframe-toolkit
-Version: 0.8.0
+Version: 0.9.1
 Summary: A command-line toolkit for DataFrame analysis and manipulation
 Keywords: dataframe,statistics,data analysis,CLI,TSV,pandas
 Author: Gunnar Pálsson
@@ -145,11 +145,12 @@ interactively otherwise. They support `--groupcol` for colour grouping,
 `--subgraphcol` for subplot grids, and figure/font presets for
 publication-quality output.
 
-| Command | Description                                                                                     |
-|---------|-------------------------------------------------------------------------------------------------|
-| `scat`  | Scatter plot; optional OLS/robust fit overlay, bubble size (`--sizecol`), colour (`--colorcol`) |
-| `line`  | Line plot; optional error bars (`--yerr`) or CI bands (`--yci lo,hi`) and fit overlay           |
-| `hist`  | Histogram or KDE (`--kde`); normalisation, cumulative mode, mean±σ annotation (`--stats`)       |
+| Command | Description                                                                                                                                                                         |
+|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `scat`  | Scatter plot; OLS/robust or LOWESS fit (`--fit lowess`, one per group with `-g`), hexagonal binning for large tables (`--hexbin`), bubble size (`--sizecol`), colour (`--colorcol`) |
+| `line`  | Line plot; optional error bars (`--yerr`) or CI bands (`--yci lo,hi`) and fit overlay                                                                                               |
+| `hist`  | Histogram or KDE (`--kde`); normalisation, cumulative mode, mean±σ annotation (`--stats`)                                                                                           |
+| `chi`   | Corner-resolved chi-plot: whether tail dependence sits in the lower or upper corner                                                                                                 |
 
 ### Utilities
 
@@ -186,6 +187,17 @@ dftk dataset fertility -o \
 
 For your own export, `dftk cleannames data.tsv --backend duckdb -o data.parquet` also keeps integer columns with empty cells as integers.
 
+### Large tables: hexbin instead of points
+
+With tens of thousands of points a scatter plot is a solid blob. `--hexbin` counts points per hexagon instead (`--hexlog` for a log colour scale), and `--fit lowess` adds a smooth curve:
+
+```bash
+dftk dataset diamonds -o \
+  | dftk scat ... -x carat -y price --hexbin --hexlog --logx --logy -f diamonds.png
+```
+
+`--colorcol COL` colours each hexagon by the mean of `COL` instead of the count; with `--subgraphcol` all panels share one colour scale.
+
 ### Group summary then plot
 
 ```bash
@@ -239,6 +251,21 @@ All commands accept:
 > parquet and prints *its path* to stdout — `-` would try to parse that path
 > as TSV and fail.
 
+### Comment lines and preambles
+
+dftk has no comment or skip-rows option: `#` is ordinary data, so headers
+like `Count #` and values like `A #1` are read as they are. Drop comment
+lines or a preamble in the shell instead:
+
+```bash
+grep -v '^#' data.tsv | dftk print           # drop every line starting with #
+tail -n +3 data.tsv | dftk print             # skip a 2-line preamble
+dftk print <(grep -v '^#' data.tsv)          # the same, as a filename
+```
+
+With `--backend duckdb`, a whole-line `#` comment below the header is an
+error, and `#` lines above the header are skipped.
+
 Standard output options (available on all tabular commands):
 
 - `-o` / `--output` — controls where output goes; takes an *optional* value:

{dataframe_toolkit-0.8.0 → dataframe_toolkit-0.9.1}/README.md

@@ -116,11 +116,12 @@ interactively otherwise. They support `--groupcol` for colour grouping,
 `--subgraphcol` for subplot grids, and figure/font presets for
 publication-quality output.
 
-| Command | Description                                                                                     |
-|---------|-------------------------------------------------------------------------------------------------|
-| `scat`  | Scatter plot; optional OLS/robust fit overlay, bubble size (`--sizecol`), colour (`--colorcol`) |
-| `line`  | Line plot; optional error bars (`--yerr`) or CI bands (`--yci lo,hi`) and fit overlay           |
-| `hist`  | Histogram or KDE (`--kde`); normalisation, cumulative mode, mean±σ annotation (`--stats`)       |
+| Command | Description                                                                                                                                                                         |
+|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `scat`  | Scatter plot; OLS/robust or LOWESS fit (`--fit lowess`, one per group with `-g`), hexagonal binning for large tables (`--hexbin`), bubble size (`--sizecol`), colour (`--colorcol`) |
+| `line`  | Line plot; optional error bars (`--yerr`) or CI bands (`--yci lo,hi`) and fit overlay                                                                                               |
+| `hist`  | Histogram or KDE (`--kde`); normalisation, cumulative mode, mean±σ annotation (`--stats`)                                                                                           |
+| `chi`   | Corner-resolved chi-plot: whether tail dependence sits in the lower or upper corner                                                                                                 |
 
 ### Utilities
 
@@ -157,6 +158,17 @@ dftk dataset fertility -o \
 
 For your own export, `dftk cleannames data.tsv --backend duckdb -o data.parquet` also keeps integer columns with empty cells as integers.
 
+### Large tables: hexbin instead of points
+
+With tens of thousands of points a scatter plot is a solid blob. `--hexbin` counts points per hexagon instead (`--hexlog` for a log colour scale), and `--fit lowess` adds a smooth curve:
+
+```bash
+dftk dataset diamonds -o \
+  | dftk scat ... -x carat -y price --hexbin --hexlog --logx --logy -f diamonds.png
+```
+
+`--colorcol COL` colours each hexagon by the mean of `COL` instead of the count; with `--subgraphcol` all panels share one colour scale.
+
 ### Group summary then plot
 
 ```bash
@@ -210,6 +222,21 @@ All commands accept:
 > parquet and prints *its path* to stdout — `-` would try to parse that path
 > as TSV and fail.
 
+### Comment lines and preambles
+
+dftk has no comment or skip-rows option: `#` is ordinary data, so headers
+like `Count #` and values like `A #1` are read as they are. Drop comment
+lines or a preamble in the shell instead:
+
+```bash
+grep -v '^#' data.tsv | dftk print           # drop every line starting with #
+tail -n +3 data.tsv | dftk print             # skip a 2-line preamble
+dftk print <(grep -v '^#' data.tsv)          # the same, as a filename
+```
+
+With `--backend duckdb`, a whole-line `#` comment below the header is an
+error, and `#` lines above the header are skipped.
+
 Standard output options (available on all tabular commands):
 
 - `-o` / `--output` — controls where output goes; takes an *optional* value:

{dataframe_toolkit-0.8.0 → dataframe_toolkit-0.9.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.8.0"
+version = "0.9.1"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 keywords = [

{dataframe_toolkit-0.8.0 → dataframe_toolkit-0.9.1}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.8.0"
+version = "0.9.1"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 authors = [

{dataframe_toolkit-0.8.0 → dataframe_toolkit-0.9.1}/src/dftk/commands/merge_cmd.py

@@ -17,6 +17,8 @@ Key features
 * --select [:right:]  expands to all right-file columns, suppressing ``_l``.
 * Both input files can be TSV or Parquet (.parquet extension auto-detected).
   The left file defaults to stdin, making it parquet-pipe friendly.
+* Integer columns that a left, right or outer join leaves gaps in become
+  nullable (Int64) instead of float64.
 """
 
 import argparse
@@ -107,6 +109,35 @@ def _add_cross_col(
     args.drop = drop
 
 
+def _gap_sides(args: argparse.Namespace) -> tuple[bool, bool]:
+    """Return (left, right): which side's columns can get missing values.
+
+    --only runs an outer join before filtering, so both sides count.
+    """
+    return args.type in ("right", "outer"), args.type in ("left", "outer")
+
+
+def _nullable_ints(df: pd.DataFrame, keep: list[str]) -> pd.DataFrame:
+    """Return *df* with numpy integer columns (except *keep*) as Int64 etc.
+
+    A plain int64 column cannot hold a missing value, so pandas turns it into
+    float64 when a join leaves rows unmatched; the nullable type keeps the
+    integers.
+    """
+    cols = [
+        c
+        for c in df.columns
+        if c not in keep
+        and pd.api.types.is_integer_dtype(df[c])
+        and not isinstance(df[c].dtype, pd.api.extensions.ExtensionDtype)
+    ]
+    if not cols:
+        return df
+    # int32 -> Int32, uint8 -> UInt8, ...
+    dtypes = {c: df[c].dtype.name.capitalize().replace("Uint", "UInt") for c in cols}
+    return df.astype(dtypes)
+
+
 def _filter_only(
     df: pd.DataFrame,
     df_left: pd.DataFrame,
@@ -152,6 +183,15 @@ def _do_merge(
 
     suffixes = _expand_select(list(df_left.columns), list(df_right.columns), args)
 
+    # Keep integer columns integers when the join leaves gaps in them.  The
+    # -k keys are filled from both sides, so they never get gaps.
+    keys = args.keys or []
+    left_gaps, right_gaps = _gap_sides(args)
+    if left_gaps:
+        df_left = _nullable_ints(df_left, keys)
+    if right_gaps:
+        df_right = _nullable_ints(df_right, keys)
+
     if args.keys is not None:
         df_merged = pd.merge(
             df_left,
@@ -278,6 +318,14 @@ would be read as if it were the data.
 Named-file self-merge (no stdin required):
 
   dftk merge penguins.tsv -r penguins_adelie.tsv -k species --only left
+
+INTEGER COLUMNS
+---------------
+A left, right or outer join leaves the other side's columns empty in
+unmatched rows.  Integer columns there are made nullable (Int64), so they
+stay integers (100, not 100.0); the -k key columns are not changed.  Read
+TSV input with --backend duckdb to keep integer columns that already have
+empty cells as integers too.
 """
 
 

dataframe_toolkit-0.9.1/src/dftk/commands/scat_cmd.py

@@ -0,0 +1,454 @@
+"""
+dftk.commands.scat_cmd — dftk scat subcommand.
+
+Scatter plot with optional OLS / robust regression line overlay.
+
+Port of dfscat.py with improvements:
+  - --groupcol  colour-codes points by a column
+  - --subgraphcol  splits into a subplot grid
+  - --size / --fontsize  publication-quality figure presets
+  - --fit  overlays an OLS (or robust) regression line via fit_cmd.regress_it,
+    or a LOWESS curve (--fit lowess); with --groupcol, one per group
+  - --hexbin  hexagonal binning for large tables (port of dfhexb.py)
+  - --sizecol  point size from a data column (was --area)
+  - --colorcol  continuous colour mapping from a data column (was --color)
+"""
+
+import argparse
+import sys
+
+import matplotlib.lines as mlines
+import numpy as np
+
+from dftk.commands.base import BaseCommand
+from dftk.common.io import check_cols, io
+from dftk.common.plot import (
+    add_figure_arguments,
+    add_group_arguments,
+    add_legend_arguments,
+    add_xy_arguments,
+    apply_labels,
+    apply_limits,
+    apply_style,
+    apply_ticks,
+    make_figure,
+    make_grouplabel,
+    save_or_show,
+    subgraph_groups,
+    subgraph_layout,
+)
+
+# ---------------------------------------------------------------------------
+# Fit overlay
+# ---------------------------------------------------------------------------
+
+
+def _ols_curve(x, y, args, label: str):
+    """Fit OLS (or RLM) of y on x; return (x_plot, y_plot, legend text)."""
+    import pandas as pd
+
+    from dftk.commands.fit_cmd import regress_it
+
+    fit_args = argparse.Namespace(
+        formula="_y ~ _x" if not args.nointercept else "_y ~ _x - 1",
+        weights=None,
+        robust=getattr(args, "robust", False),
+    )
+    res = regress_it(pd.DataFrame({"_x": x, "_y": y}), fit_args)
+    if label:
+        print(f"== {label}", file=sys.stderr)
+    print(res.summary(), file=sys.stderr)
+
+    slope = res.params["_x"]
+    intercept = 0.0 if args.nointercept else res.params["Intercept"]
+    if args.nointercept:
+        text = f"{slope:+.3g} × {args.xcol}"
+    else:
+        text = f"{intercept:.3g}{slope:+.3g} × {args.xcol}"
+    if getattr(args, "pvalue", False):
+        pval = res.pvalues.get("_x", res.pvalues.iloc[-1])
+        text += f"  -log10(p)={-np.log10(pval):.2g}"
+    return slope, intercept, text
+
+
+def _lowess_curve(x, y, args):
+    """Return the LOWESS curve of y on x as (x_sorted, y_smoothed)."""
+    from statsmodels.nonparametric.smoothers_lowess import lowess
+
+    # delta: fit only every 1% of the x range and interpolate in between.
+    # Without it LOWESS is quadratic in n (10,000 points take seconds,
+    # 200,000 do not finish); with it 1,000,000 points take a few seconds.
+    delta = 0.01 * (x.max() - x.min())
+    curve = lowess(y, x, frac=args.frac, delta=delta)
+    return curve[:, 0], curve[:, 1]
+
+
+def _fit_overlay(ax, x_series, y_series, args, color=None, label: str = ""):
+    """Overlay the --fit line or curve on ax.
+
+    Without *color* (no --groupcol) the fit gets its own red legend entry,
+    as before.  With *color* it is drawn in the group's colour, over that
+    group's x range, and the group legend already identifies it.
+    """
+    import pandas as pd
+
+    data = pd.DataFrame({"x": x_series.values, "y": y_series.values}).dropna()
+    data = data.astype(float)
+    if len(data) < 3:
+        print(
+            f"--fit: skipped {label or 'data'} (fewer than 3 points)", file=sys.stderr
+        )
+        return
+    x, y = data["x"].to_numpy(), data["y"].to_numpy()
+
+    if args.fit == "lowess":
+        x_plot, y_plot = _lowess_curve(x, y, args)
+        text = f"LOWESS (frac={args.frac:.2g})"
+    else:
+        slope, intercept, text = _ols_curve(x, y, args, label)
+        xmin, xmax = (x.min(), x.max()) if color is not None else ax.get_xlim()
+        x_plot = np.linspace(xmin, xmax, 200)
+        y_plot = intercept + slope * x_plot
+
+    if color is None:
+        ax.plot(x_plot, y_plot, "r--", linewidth=1)
+        ax.legend(
+            handles=[mlines.Line2D([], [], color="red", linestyle="--", label=text)],
+            loc=0,
+        )
+    else:
+        ax.plot(x_plot, y_plot, "--", color=color, linewidth=1.2)
+
+
+def _hexbin_ax(ax, df, args):
+    """Draw a hexagonal-bin density plot (or mean of --colorcol per hexagon)."""
+    import matplotlib.pyplot as plt
+
+    cols = [args.xcol, args.ycol] + ([args.colorcol] if args.colorcol else [])
+    data = df[cols].dropna()
+    for flag, col in (("logx", args.xcol), ("logy", args.ycol)):
+        if getattr(args, flag, False):
+            positive = data[col] > 0
+            if not positive.all():
+                print(
+                    f"--hexbin --{flag}: dropped {(~positive).sum()} rows with "
+                    f"{col} <= 0",
+                    file=sys.stderr,
+                )
+                data = data[positive]
+
+    kw = dict(
+        gridsize=args.gridsize,
+        mincnt=1,
+        cmap="viridis",
+        # bin on the log scale, so hexagons are regular on log axes
+        xscale="log" if getattr(args, "logx", False) else "linear",
+        yscale="log" if getattr(args, "logy", False) else "linear",
+    )
+    if args.hexlog:
+        kw["bins"] = "log"
+    if args.colorcol:
+        kw["C"] = data[args.colorcol]
+        kw["reduce_C_function"] = np.mean
+        label = f"mean {args.colorcol}"
+    else:
+        label = "count"
+    hb = ax.hexbin(data[args.xcol], data[args.ycol], **kw)
+    plt.colorbar(hb, ax=ax, label=label)
+
+
+def _share_hexbin_colours(axes) -> None:
+    """Give every --subgraphcol panel the same colour scale, so they compare."""
+    from matplotlib.collections import PolyCollection
+
+    hexbins = [
+        c
+        for row in axes
+        for ax in row
+        for c in ax.collections
+        if isinstance(c, PolyCollection) and c.get_array() is not None
+    ]
+    values = [hb.get_array() for hb in hexbins if hb.get_array().size]
+    if not values:
+        return
+    vmin = min(float(v.min()) for v in values)
+    vmax = max(float(v.max()) for v in values)
+    for hb in hexbins:
+        hb.set_clim(vmin, vmax)
+
+
+# ---------------------------------------------------------------------------
+# Single-axes scatter (reused per subplot)
+# ---------------------------------------------------------------------------
+
+
+def _scatter_ax(ax, df, args, title_suffix: str = ""):
+    """Draw scatter on a single Axes, with optional --groupcol colouring."""
+    import matplotlib.pyplot as plt
+
+    xcol = args.xcol
+    ycol = args.ycol
+    legendloc = tuple(args.legendloc) if args.legendloc else "best"
+
+    scatter_kw = dict(marker=getattr(args, "marker", "o"), alpha=0.7)
+
+    if getattr(args, "hexbin", False):
+        _hexbin_ax(ax, df, args)
+        if getattr(args, "fit", None):
+            _fit_overlay(ax, df[xcol], df[ycol], args)
+    elif args.groupcol is not None:
+        for groupname, gdf in subgraph_groups(
+            df, args.groupcol, getattr(args, "groupcolorder", None)
+        ):
+            label = make_grouplabel(
+                groupname, args.groupcol, getattr(args, "groupcolformat", None)
+            )
+            kw = dict(scatter_kw)
+            if args.sizecol:
+                kw["s"] = gdf[args.sizecol]
+            points = ax.scatter(gdf[xcol], gdf[ycol], label=label, **kw)
+            if getattr(args, "fit", None):
+                color = points.get_facecolor()[0]
+                _fit_overlay(ax, gdf[xcol], gdf[ycol], args, color=color, label=label)
+        ax.legend(loc=legendloc, title=args.legendtitle)
+    else:
+        kw = dict(scatter_kw)
+        if args.sizecol:
+            kw["s"] = df[args.sizecol]
+        if args.colorcol:
+            sc = ax.scatter(
+                df[xcol], df[ycol], c=df[args.colorcol], cmap="viridis", **kw
+            )
+            plt.colorbar(sc, ax=ax, label=args.colorcol)
+        else:
+            label = args.legend or ""
+            ax.scatter(df[xcol], df[ycol], label=label if label else None, **kw)
+            if label:
+                ax.legend(loc=legendloc, title=args.legendtitle)
+
+        if getattr(args, "fit", None):
+            _fit_overlay(ax, df[xcol], df[ycol], args)
+
+    apply_limits(ax, args)
+    apply_ticks(ax, args)
+    apply_labels(ax, args, default_xlabel=xcol or "", default_ylabel=ycol or "")
+    if title_suffix:
+        existing = ax.get_title()
+        ax.set_title(f"{existing}  [{title_suffix}]" if existing else title_suffix)
+
+    ax.grid(True, linewidth=0.4, alpha=0.5)
+
+
+# ---------------------------------------------------------------------------
+# Command
+# ---------------------------------------------------------------------------
+
+_EPILOG = """\
+EXAMPLES
+--------
+Basic scatter:
+
+  dftk scat data.tsv -x weight -y height
+  dftk dataset tips -o | dftk scat ... -x total_bill -y tip
+
+Colour by group:
+
+  dftk dataset tips -o | dftk scat ... -x total_bill -y tip -g day
+
+Subplot grid by time-of-day, colour by smoker:
+
+  dftk dataset tips -o | dftk scat ... -x total_bill -y tip \\
+      --subgraphcol time -g smoker
+
+Fit OLS line, or one per group in the group's colour:
+
+  dftk dataset tips -o | dftk scat ... -x total_bill -y tip --fit
+  dftk dataset tips -o | dftk scat ... -x total_bill -y tip -g smoker --fit
+
+LOWESS curve (local regression; smaller --frac follows the data more
+closely). At the ends of the x range, where few points are left, the curve
+can overshoot the data; a larger --frac or --xlim helps:
+
+  dftk dataset tips -o | dftk scat ... -x total_bill -y tip --fit lowess
+
+Hexagonal binning for large tables (54,000 diamonds), log colour scale and
+log axes; hexagons are binned on the log scale:
+
+  dftk dataset diamonds -o | dftk scat ... -x carat -y price \
+      --hexbin --hexlog --logx --logy
+
+Colour each hexagon by the mean of another column instead of the count:
+
+  dftk dataset diamonds -o | dftk scat ... -x carat -y price \
+      --hexbin --colorcol depth --gridsize 50
+
+One hexbin panel per group, all on the same colour scale:
+
+  dftk dataset diamonds -o | dftk scat ... -x carat -y price \
+      --hexbin --hexlog --subgraphcol cut --gridsize 40
+
+Publication figure (Nature single column, PDF):
+
+  dftk dataset tips -o | dftk scat ... -x total_bill -y tip \\
+      --size single --fontsize publication -f fig.pdf
+
+Bubble chart (point size from column):
+
+  dftk dataset tips -o | dftk scat ... -x total_bill -y tip --sizecol size
+
+Continuous colour mapping:
+
+  dftk dataset tips -o | dftk scat ... -x total_bill -y tip --colorcol tip
+"""
+
+
+class ScatCommand(BaseCommand):
+    name = "scat"
+    help = "Scatter plot with optional regression line overlay."
+
+    def add_arguments(self, parser: argparse.ArgumentParser) -> None:
+        parser.formatter_class = argparse.RawDescriptionHelpFormatter
+        parser.epilog = _EPILOG
+
+        self.add_read_arguments(parser)
+        add_xy_arguments(parser)
+        add_figure_arguments(parser)
+        add_group_arguments(parser)
+        add_legend_arguments(parser)
+
+        s = parser.add_argument_group("scatter options")
+        s.add_argument(
+            "-m",
+            "--marker",
+            default="o",
+            metavar="MARKER",
+            help="Matplotlib marker style (default: o).",
+        )
+        s.add_argument(
+            "--sizecol",
+            default=None,
+            metavar="COL",
+            help="Column to use for point sizes (bubble chart).",
+        )
+        s.add_argument(
+            "--colorcol",
+            default=None,
+            metavar="COL",
+            help="Column for continuous colour mapping.",
+        )
+
+        h = parser.add_argument_group("hexbin options")
+        h.add_argument(
+            "--hexbin",
+            action="store_true",
+            help="Hexagonal binning instead of points, for large tables: "
+            "colour shows the number of points per hexagon, or with "
+            "--colorcol the mean of that column. Not with -g or --sizecol.",
+        )
+        h.add_argument(
+            "--gridsize",
+            type=int,
+            default=100,
+            metavar="N",
+            help="Number of hexagons across the x axis (default: 100).",
+        )
+        h.add_argument(
+            "--hexlog",
+            action="store_true",
+            help="Log colour scale, so sparse areas stay visible.",
+        )
+
+        f = parser.add_argument_group("fit options")
+        f.add_argument(
+            "--fit",
+            nargs="?",
+            const="ols",
+            default=None,
+            choices=["ols", "lowess"],
+            help="Overlay a fit: ols (the default for a bare --fit; a "
+            "regression line, summary to stderr) or lowess (a smooth curve). "
+            "With -g, one per group in the group's colour.",
+        )
+        f.add_argument(
+            "--frac",
+            type=float,
+            default=2 / 3,
+            metavar="F",
+            help="Share of the data used for each local fit in --fit lowess "
+            "(default: 2/3). Smaller values follow the data more closely.",
+        )
+        f.add_argument(
+            "-r",
+            "--robust",
+            action="store_true",
+            help="Use robust regression (RLM) for --fit ols.",
+        )
+        f.add_argument(
+            "--nointercept", action="store_true", help="Suppress intercept in --fit."
+        )
+        f.add_argument(
+            "--pvalue",
+            action="store_true",
+            help="Add −log₁₀(p) to the fit legend label.",
+        )
+
+    def execute(self, args: argparse.Namespace) -> None:
+        if not args.xcol or not args.ycol:
+            raise ValueError("--xcol (-x) and --ycol (-y) are required.")
+
+        df = io.read(args)
+        check_cols(df, [args.xcol, args.ycol], "-x/-y")
+        check_cols(df, args.groupcol, "-g/--groupcol")
+        check_cols(df, args.subgraphcol, "--subgraphcol")
+        if args.sizecol:
+            check_cols(df, [args.sizecol], "--sizecol")
+        if args.colorcol:
+            check_cols(df, [args.colorcol], "--colorcol")
+        if args.hexbin:
+            if args.groupcol:
+                raise ValueError(
+                    "--hexbin cannot be combined with -g/--groupcol: overlaid "
+                    "densities cannot be told apart. Use --subgraphcol instead."
+                )
+            if args.sizecol:
+                raise ValueError("--hexbin cannot be combined with --sizecol.")
+        if args.fit == "lowess" and not 0 < args.frac <= 1:
+            raise ValueError(f"--frac must be in (0, 1], got {args.frac}")
+
+        # Use non-interactive backend when saving to file
+        if args.file:
+            import matplotlib
+
+            matplotlib.use("Agg")
+
+        import matplotlib.pyplot as plt
+
+        if args.usetex:
+            plt.rc("text", usetex=True)
+
+        apply_style(args)
+
+        if args.subgraphcol is not None:
+            groups = subgraph_groups(
+                df, args.subgraphcol, getattr(args, "subgraphorder", None)
+            )
+            nrows, ncols = subgraph_layout(len(groups), args.ncols)
+            fig, axes = make_figure(args, nrows=nrows, ncols=ncols)
+            for idx, (groupname, gdf) in enumerate(groups):
+                ax = axes[idx // ncols][idx % ncols]
+                suffix = make_grouplabel(
+                    groupname, args.subgraphcol, getattr(args, "subgraphformat", None)
+                )
+                _scatter_ax(ax, gdf, args, title_suffix=suffix)
+            # Hide unused axes
+            for idx in range(len(groups), nrows * ncols):
+                axes[idx // ncols][idx % ncols].set_visible(False)
+            if args.hexbin:
+                _share_hexbin_colours(axes)
+            fig.tight_layout()
+        else:
+            fig, axes = make_figure(args)
+            _scatter_ax(axes[0][0], df, args)
+            fig.tight_layout()
+
+        save_or_show(fig, args)

{dataframe_toolkit-0.8.0 → dataframe_toolkit-0.9.1}/src/dftk/common/io.py

@@ -27,6 +27,7 @@ import fnmatch
 import io as _io
 import logging
 import os
+import shutil
 import sys
 import tempfile
 
@@ -365,14 +366,30 @@ class io:
             return _name_columns(df)
 
         def _csv_with_duckdb(path: str) -> pd.DataFrame:
-            # DuckDB keeps integer columns with blanks as Int64 (pandas: float64)
+            # DuckDB keeps integer columns with blanks as Int64 (pandas: float64).
+            # comment="" stops it guessing "#" as a comment character: after one
+            # whole-line "#" comment it cuts every line at "#", headers included
             try:
-                df = duckdb.read_csv(path, sep=sep, header=not noheader).df()
+                df = duckdb.read_csv(
+                    path, sep=sep, header=not noheader, comment=""
+                ).df()
             except Exception as exc:
                 logging.error(f"DuckDB failed to read {path!r}: {exc}")
                 raise
             return _name_columns(df)
 
+        def _spooled_csv_with_duckdb(stream) -> pd.DataFrame:
+            # DuckDB reads its input twice (type sniffing, then data), so a
+            # stream must be copied to a temp file first
+            os.makedirs(DFTK_TMPDIR, exist_ok=True)
+            fd, path = tempfile.mkstemp(suffix=".tsv", dir=DFTK_TMPDIR)
+            try:
+                with os.fdopen(fd, "wb") as fh:
+                    shutil.copyfileobj(stream, fh)
+                return _csv_with_duckdb(path)
+            finally:
+                os.unlink(path)
+
         def _csv_from_file(path: str) -> pd.DataFrame:
             df = _pandas_csv(path)
             df = _with_blank_lines(
@@ -403,21 +420,18 @@ class io:
                 df = pd.read_parquet(filename)
                 df.attrs["_parquet_meta"] = _read_parquet_meta(filename)
             elif backend == "duckdb" and duckdb is not None:
-                df = _csv_with_duckdb(filename)
+                if os.path.exists(filename) and not os.path.isfile(filename):
+                    # A pipe, e.g. <(grep -v '^#' file): one read would use it up
+                    with open(filename, "rb") as fh:
+                        df = _spooled_csv_with_duckdb(fh)
+                else:
+                    df = _csv_with_duckdb(filename)
             else:
                 df = _csv_from_file(filename)
 
         elif backend == "duckdb" and duckdb is not None:
-            # stdin with DuckDB: it reads files, not buffers (that needs fsspec),
-            # so spool stdin to a temp file and read it like a named file
-            os.makedirs(DFTK_TMPDIR, exist_ok=True)
-            fd, path = tempfile.mkstemp(suffix=".tsv", dir=DFTK_TMPDIR)
-            try:
-                with os.fdopen(fd, "wb") as fh:
-                    fh.write(sys.stdin.buffer.read())
-                df = _csv_with_duckdb(path)
-            finally:
-                os.unlink(path)
+            # stdin with DuckDB: it reads files, not buffers (that needs fsspec)
+            df = _spooled_csv_with_duckdb(sys.stdin.buffer)
 
         else:
             # stdin — read TSV directly

dataframe_toolkit-0.8.0/src/dftk/commands/scat_cmd.py

@@ -1,281 +0,0 @@
-"""
-dftk.commands.scat_cmd — dftk scat subcommand.
-
-Scatter plot with optional OLS / robust regression line overlay.
-
-Port of dfscat.py with improvements:
-  - --groupcol  colour-codes points by a column
-  - --subgraphcol  splits into a subplot grid
-  - --size / --fontsize  publication-quality figure presets
-  - --fit  overlays an OLS (or robust) regression line via fit_cmd.regress_it
-  - --sizecol  point size from a data column (was --area)
-  - --colorcol  continuous colour mapping from a data column (was --color)
-"""
-
-import argparse
-import sys
-
-import matplotlib.lines as mlines
-import numpy as np
-
-from dftk.commands.base import BaseCommand
-from dftk.common.io import check_cols, io
-from dftk.common.plot import (
-    add_figure_arguments,
-    add_group_arguments,
-    add_legend_arguments,
-    add_xy_arguments,
-    apply_labels,
-    apply_limits,
-    apply_style,
-    apply_ticks,
-    make_figure,
-    make_grouplabel,
-    save_or_show,
-    subgraph_groups,
-    subgraph_layout,
-)
-
-# ---------------------------------------------------------------------------
-# Fit overlay
-# ---------------------------------------------------------------------------
-
-
-def _fit_overlay(ax, x_series, y_series, args):
-    """Fit a regression line and overlay it on ax. Prints summary to stdout."""
-    import pandas as pd
-
-    from dftk.commands.fit_cmd import regress_it
-
-    df_fit = pd.DataFrame({"_x": x_series.values, "_y": y_series.values}).dropna()
-    fit_args = argparse.Namespace(
-        formula="_y ~ _x" if not args.nointercept else "_y ~ _x - 1",
-        weights=None,
-        robust=getattr(args, "robust", False),
-    )
-    res = regress_it(df_fit, fit_args)
-    print(res.summary(), file=sys.stderr)
-
-    xmin, xmax = ax.get_xlim()
-    X_plot = np.linspace(xmin, xmax, 200)
-
-    if args.nointercept:
-        slope = res.params["_x"]
-        Y_plot = slope * X_plot
-        label_text = f"{slope:+.3g} × {args.xcol}"
-    else:
-        intercept = res.params["Intercept"]
-        slope = res.params["_x"]
-        Y_plot = intercept + slope * X_plot
-        label_text = f"{intercept:.3g}{slope:+.3g} × {args.xcol}"
-
-    if getattr(args, "pvalue", False):
-        pval = res.pvalues.get("_x", res.pvalues.iloc[-1])
-        label_text += f"  -log10(p)={-np.log10(pval):.2g}"
-
-    ax.plot(X_plot, Y_plot, "r--", linewidth=1)
-    ax.legend(
-        handles=[mlines.Line2D([], [], color="red", linestyle="--", label=label_text)],
-        loc=0,
-    )
-
-
-# ---------------------------------------------------------------------------
-# Single-axes scatter (reused per subplot)
-# ---------------------------------------------------------------------------
-
-
-def _scatter_ax(ax, df, args, title_suffix: str = ""):
-    """Draw scatter on a single Axes, with optional --groupcol colouring."""
-    import matplotlib.pyplot as plt
-
-    xcol = args.xcol
-    ycol = args.ycol
-    legendloc = tuple(args.legendloc) if args.legendloc else "best"
-
-    scatter_kw = dict(marker=getattr(args, "marker", "o"), alpha=0.7)
-
-    if args.groupcol is not None:
-        for groupname, gdf in subgraph_groups(
-            df, args.groupcol, getattr(args, "groupcolorder", None)
-        ):
-            label = make_grouplabel(
-                groupname, args.groupcol, getattr(args, "groupcolformat", None)
-            )
-            kw = dict(scatter_kw)
-            if args.sizecol:
-                kw["s"] = gdf[args.sizecol]
-            ax.scatter(gdf[xcol], gdf[ycol], label=label, **kw)
-        ax.legend(loc=legendloc, title=args.legendtitle)
-    else:
-        kw = dict(scatter_kw)
-        if args.sizecol:
-            kw["s"] = df[args.sizecol]
-        if args.colorcol:
-            sc = ax.scatter(
-                df[xcol], df[ycol], c=df[args.colorcol], cmap="viridis", **kw
-            )
-            plt.colorbar(sc, ax=ax, label=args.colorcol)
-        else:
-            label = args.legend or ""
-            ax.scatter(df[xcol], df[ycol], label=label if label else None, **kw)
-            if label:
-                ax.legend(loc=legendloc, title=args.legendtitle)
-
-    if getattr(args, "fit", False) and args.groupcol is None:
-        _fit_overlay(ax, df[xcol], df[ycol], args)
-
-    apply_limits(ax, args)
-    apply_ticks(ax, args)
-    apply_labels(ax, args, default_xlabel=xcol or "", default_ylabel=ycol or "")
-    if title_suffix:
-        existing = ax.get_title()
-        ax.set_title(f"{existing}  [{title_suffix}]" if existing else title_suffix)
-
-    ax.grid(True, linewidth=0.4, alpha=0.5)
-
-
-# ---------------------------------------------------------------------------
-# Command
-# ---------------------------------------------------------------------------
-
-_EPILOG = """\
-EXAMPLES
---------
-Basic scatter:
-
-  dftk scat data.tsv -x weight -y height
-  dftk dataset tips -o | dftk scat ... -x total_bill -y tip
-
-Colour by group:
-
-  dftk dataset tips -o | dftk scat ... -x total_bill -y tip -g day
-
-Subplot grid by time-of-day, colour by smoker:
-
-  dftk dataset tips -o | dftk scat ... -x total_bill -y tip \\
-      --subgraphcol time -g smoker
-
-Fit OLS line:
-
-  dftk dataset tips -o | dftk scat ... -x total_bill -y tip --fit
-
-Publication figure (Nature single column, PDF):
-
-  dftk dataset tips -o | dftk scat ... -x total_bill -y tip \\
-      --size single --fontsize publication -f fig.pdf
-
-Bubble chart (point size from column):
-
-  dftk dataset tips -o | dftk scat ... -x total_bill -y tip --sizecol size
-
-Continuous colour mapping:
-
-  dftk dataset tips -o | dftk scat ... -x total_bill -y tip --colorcol tip
-"""
-
-
-class ScatCommand(BaseCommand):
-    name = "scat"
-    help = "Scatter plot with optional regression line overlay."
-
-    def add_arguments(self, parser: argparse.ArgumentParser) -> None:
-        parser.formatter_class = argparse.RawDescriptionHelpFormatter
-        parser.epilog = _EPILOG
-
-        self.add_read_arguments(parser)
-        add_xy_arguments(parser)
-        add_figure_arguments(parser)
-        add_group_arguments(parser)
-        add_legend_arguments(parser)
-
-        s = parser.add_argument_group("scatter options")
-        s.add_argument(
-            "-m",
-            "--marker",
-            default="o",
-            metavar="MARKER",
-            help="Matplotlib marker style (default: o).",
-        )
-        s.add_argument(
-            "--sizecol",
-            default=None,
-            metavar="COL",
-            help="Column to use for point sizes (bubble chart).",
-        )
-        s.add_argument(
-            "--colorcol",
-            default=None,
-            metavar="COL",
-            help="Column for continuous colour mapping.",
-        )
-
-        f = parser.add_argument_group("fit options")
-        f.add_argument(
-            "--fit",
-            action="store_true",
-            help="Overlay an OLS regression line. Prints summary to stderr.",
-        )
-        f.add_argument(
-            "-r",
-            "--robust",
-            action="store_true",
-            help="Use robust regression (RLM) for --fit.",
-        )
-        f.add_argument(
-            "--nointercept", action="store_true", help="Suppress intercept in --fit."
-        )
-        f.add_argument(
-            "--pvalue",
-            action="store_true",
-            help="Add −log₁₀(p) to the fit legend label.",
-        )
-
-    def execute(self, args: argparse.Namespace) -> None:
-        if not args.xcol or not args.ycol:
-            raise ValueError("--xcol (-x) and --ycol (-y) are required.")
-
-        df = io.read(args)
-        check_cols(df, [args.xcol, args.ycol], "-x/-y")
-        check_cols(df, args.groupcol, "-g/--groupcol")
-        check_cols(df, args.subgraphcol, "--subgraphcol")
-        if args.sizecol:
-            check_cols(df, [args.sizecol], "--sizecol")
-        if args.colorcol:
-            check_cols(df, [args.colorcol], "--colorcol")
-
-        # Use non-interactive backend when saving to file
-        if args.file:
-            import matplotlib
-
-            matplotlib.use("Agg")
-
-        import matplotlib.pyplot as plt
-
-        if args.usetex:
-            plt.rc("text", usetex=True)
-
-        apply_style(args)
-
-        if args.subgraphcol is not None:
-            groups = subgraph_groups(
-                df, args.subgraphcol, getattr(args, "subgraphorder", None)
-            )
-            nrows, ncols = subgraph_layout(len(groups), args.ncols)
-            fig, axes = make_figure(args, nrows=nrows, ncols=ncols)
-            for idx, (groupname, gdf) in enumerate(groups):
-                ax = axes[idx // ncols][idx % ncols]
-                suffix = make_grouplabel(
-                    groupname, args.subgraphcol, getattr(args, "subgraphformat", None)
-                )
-                _scatter_ax(ax, gdf, args, title_suffix=suffix)
-            # Hide unused axes
-            for idx in range(len(groups), nrows * ncols):
-                axes[idx // ncols][idx % ncols].set_visible(False)
-            fig.tight_layout()
-        else:
-            fig, axes = make_figure(args)
-            _scatter_ax(axes[0][0], df, args)
-            fig.tight_layout()
-
-        save_or_show(fig, args)