dataframe-toolkit 0.11.0__tar.gz → 0.11.2__tar.gz

{dataframe_toolkit-0.11.0 → dataframe_toolkit-0.11.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: dataframe-toolkit
-Version: 0.11.0
+Version: 0.11.2
 Summary: A command-line toolkit for DataFrame analysis and manipulation
 Keywords: dataframe,statistics,data analysis,CLI,TSV,pandas
 Author: Gunnar Pálsson
@@ -226,35 +226,77 @@ printf "sample\tfluor\ns1\t60\ns2\t110\n" \
 
 ### Filling missing values
 
-`eval -f` takes pandas column methods (`COL.method(...)`), and `fillna`,
-`ffill`, `bfill` and `interpolate` fill empty cells. Defaults from another
-table, after a merge (a column in both files gets the suffix `_r`):
+`eval -f` takes pandas column methods (`COL.method(...)`); `fillna`,
+`ffill`, `bfill` and `interpolate` fill empty cells and keep the others.
+
+| Formula | Fills with |
+|---|---|
+| `x = x.fillna(y)` | the value in column `y` |
+| `x = x.fillna(0)`, `sex = sex.fillna('unknown')` | a fixed value |
+| `x = x.fillna(y).fillna(0)` | `y` first, then 0 |
+| `x = x.fillna(x.groupby(g).transform('mean'))` | the mean of the row's group (or `'median'`, ...) |
+| `x = x.ffill()` / `x = x.bfill()` | the value above / below (`ffill(limit=1)`: at most one row) |
+| `x = x.groupby(g).ffill()` | the value above, within each group of `g` |
+| `x = x.interpolate()` | a straight line between the values above and below, by row position |
+
+Write to a new column (`x_filled = x.fillna(y)`) to keep the original, and
+mark the gaps first (`missing = x.isna()`) to tell filled values apart.
+Only empty cells are filled: a missing row (a year with no row at all) is
+not added.
+
+**Defaults from another table.** A column that is in both files gets the
+suffix `_r` from `merge`; fill from it, then drop it. Here the defaults are
+each species' median body mass:
 
 ```bash
-dftk merge data.tsv -r defaults.tsv -k type -t left -o \
-  | dftk eval ... -f "depth = depth.fillna(depth_r)" --drop depth_r
+dftk dataset penguins -o | dftk pivot ... -i species -v body_mass_g \
+  -f median --move median_body_mass_g:body_mass_g -o defaults.parquet
+dftk dataset penguins -o \
+  | dftk merge ... -r defaults.parquet -k species -t left -o \
+  | dftk eval ... -f "body_mass_g = body_mass_g.fillna(body_mass_g_r)" \
+      --drop body_mass_g_r
 ```
 
-The group mean, a fixed value, or the last value above (`ffill`) within
-each group:
+**From neighbouring rows** (`co2` is a weekly series with gaps):
 
 ```bash
-dftk dataset penguins -o | dftk eval ... \
-  -f "mean = body_mass_g.groupby(species).transform('mean')" \
-  -f "body_mass_g = body_mass_g.fillna(mean)" --drop mean \
-  -f "sex = sex.fillna('unknown')"
+dftk dataset co2 -o | dftk eval ... -f "ffill = co2.ffill()" \
+  -f "bfill = co2.bfill()" -f "linear = co2.interpolate()"
+```
+
+`ffill` leaves the first rows empty and `bfill` the last ones;
+`x.ffill().fillna(x.bfill())` fills both. `interpolate()` counts one step
+per row, so the rows must be evenly spaced; for an x-based lookup from a
+reference curve, see `dftk interp`.
+
+**Row order and groups.** These methods use the rows in the order they are
+in. Sort first, in a step of its own: `--sortasc` applies to a command's
+output, so in the same command it would come after the fill. When a table
+holds several series (sites, countries, samples), fill within each group,
+or values are taken from the neighbouring group. Rows of a group don't need
+to be next to each other, but within each group they must be in order:
 
+```bash
 dftk dataset healthexp -o \
-  | dftk eval ... -f "le = Life_Expectancy.where(Year % 10 != 5)" -o \
+  | dftk eval ... -f "le = Life_Expectancy.where(Year % 10 != 5)" \
+      --sortasc Country Year -o \
   | dftk eval ... -f "le_filled = le.groupby(Country).ffill()"
 ```
 
-`ffill`, `bfill` and `interpolate` use the rows in the order they are in:
-sort in a step of its own first (`dftk eval data.tsv --sortasc site day -o
-| dftk eval ... -f ...`), and use `groupby(...)` when the table holds
-several series, or values are taken from the neighbouring group.
-`interpolate()` has no grouped form and works by row position. `dftk eval
--h` covers these cases, with examples.
+`interpolate()` has no grouped form. Once each group's rows are together,
+gaps inside a group are right, but a gap at a group's start or end is
+filled from the neighbouring group. To leave those empty, keep only values
+with something before and after them in the same group:
+
+```bash
+dftk dataset healthexp -o \
+  | dftk eval ... -f "le = Life_Expectancy.where(Year % 10 != 5)" \
+      --sortasc Country Year -o \
+  | dftk eval ... -f "before = le.groupby(Country).ffill().notna()" \
+      -f "after = le.groupby(Country).bfill().notna()" \
+      -f "le_linear = le.interpolate().where(before & after)" \
+      --drop before after
+```
 
 ### Bootstrap confidence intervals
 

{dataframe_toolkit-0.11.0 → dataframe_toolkit-0.11.2}/README.md

@@ -197,35 +197,77 @@ printf "sample\tfluor\ns1\t60\ns2\t110\n" \
 
 ### Filling missing values
 
-`eval -f` takes pandas column methods (`COL.method(...)`), and `fillna`,
-`ffill`, `bfill` and `interpolate` fill empty cells. Defaults from another
-table, after a merge (a column in both files gets the suffix `_r`):
+`eval -f` takes pandas column methods (`COL.method(...)`); `fillna`,
+`ffill`, `bfill` and `interpolate` fill empty cells and keep the others.
+
+| Formula | Fills with |
+|---|---|
+| `x = x.fillna(y)` | the value in column `y` |
+| `x = x.fillna(0)`, `sex = sex.fillna('unknown')` | a fixed value |
+| `x = x.fillna(y).fillna(0)` | `y` first, then 0 |
+| `x = x.fillna(x.groupby(g).transform('mean'))` | the mean of the row's group (or `'median'`, ...) |
+| `x = x.ffill()` / `x = x.bfill()` | the value above / below (`ffill(limit=1)`: at most one row) |
+| `x = x.groupby(g).ffill()` | the value above, within each group of `g` |
+| `x = x.interpolate()` | a straight line between the values above and below, by row position |
+
+Write to a new column (`x_filled = x.fillna(y)`) to keep the original, and
+mark the gaps first (`missing = x.isna()`) to tell filled values apart.
+Only empty cells are filled: a missing row (a year with no row at all) is
+not added.
+
+**Defaults from another table.** A column that is in both files gets the
+suffix `_r` from `merge`; fill from it, then drop it. Here the defaults are
+each species' median body mass:
 
 ```bash
-dftk merge data.tsv -r defaults.tsv -k type -t left -o \
-  | dftk eval ... -f "depth = depth.fillna(depth_r)" --drop depth_r
+dftk dataset penguins -o | dftk pivot ... -i species -v body_mass_g \
+  -f median --move median_body_mass_g:body_mass_g -o defaults.parquet
+dftk dataset penguins -o \
+  | dftk merge ... -r defaults.parquet -k species -t left -o \
+  | dftk eval ... -f "body_mass_g = body_mass_g.fillna(body_mass_g_r)" \
+      --drop body_mass_g_r
 ```
 
-The group mean, a fixed value, or the last value above (`ffill`) within
-each group:
+**From neighbouring rows** (`co2` is a weekly series with gaps):
 
 ```bash
-dftk dataset penguins -o | dftk eval ... \
-  -f "mean = body_mass_g.groupby(species).transform('mean')" \
-  -f "body_mass_g = body_mass_g.fillna(mean)" --drop mean \
-  -f "sex = sex.fillna('unknown')"
+dftk dataset co2 -o | dftk eval ... -f "ffill = co2.ffill()" \
+  -f "bfill = co2.bfill()" -f "linear = co2.interpolate()"
+```
+
+`ffill` leaves the first rows empty and `bfill` the last ones;
+`x.ffill().fillna(x.bfill())` fills both. `interpolate()` counts one step
+per row, so the rows must be evenly spaced; for an x-based lookup from a
+reference curve, see `dftk interp`.
 
+**Row order and groups.** These methods use the rows in the order they are
+in. Sort first, in a step of its own: `--sortasc` applies to a command's
+output, so in the same command it would come after the fill. When a table
+holds several series (sites, countries, samples), fill within each group,
+or values are taken from the neighbouring group. Rows of a group don't need
+to be next to each other, but within each group they must be in order:
+
+```bash
 dftk dataset healthexp -o \
-  | dftk eval ... -f "le = Life_Expectancy.where(Year % 10 != 5)" -o \
+  | dftk eval ... -f "le = Life_Expectancy.where(Year % 10 != 5)" \
+      --sortasc Country Year -o \
   | dftk eval ... -f "le_filled = le.groupby(Country).ffill()"
 ```
 
-`ffill`, `bfill` and `interpolate` use the rows in the order they are in:
-sort in a step of its own first (`dftk eval data.tsv --sortasc site day -o
-| dftk eval ... -f ...`), and use `groupby(...)` when the table holds
-several series, or values are taken from the neighbouring group.
-`interpolate()` has no grouped form and works by row position. `dftk eval
--h` covers these cases, with examples.
+`interpolate()` has no grouped form. Once each group's rows are together,
+gaps inside a group are right, but a gap at a group's start or end is
+filled from the neighbouring group. To leave those empty, keep only values
+with something before and after them in the same group:
+
+```bash
+dftk dataset healthexp -o \
+  | dftk eval ... -f "le = Life_Expectancy.where(Year % 10 != 5)" \
+      --sortasc Country Year -o \
+  | dftk eval ... -f "before = le.groupby(Country).ffill().notna()" \
+      -f "after = le.groupby(Country).bfill().notna()" \
+      -f "le_linear = le.interpolate().where(before & after)" \
+      --drop before after
+```
 
 ### Bootstrap confidence intervals
 

{dataframe_toolkit-0.11.0 → dataframe_toolkit-0.11.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.11.0"
+version = "0.11.2"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 keywords = [

{dataframe_toolkit-0.11.0 → dataframe_toolkit-0.11.2}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.11.0"
+version = "0.11.2"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 authors = [

dataframe_toolkit-0.11.2/src/dftk/commands/eval_cmd.py

@@ -0,0 +1,391 @@
+"""
+dftk.commands.eval_cmd — dftk eval subcommand.
+
+Evaluates formulas, constants, and string/row functions against a DataFrame.
+Three formula types are supported, each with its own flag:
+
+  -f  pandas eval expressions  (df.eval, falls back to special_function)
+  -c  constant column assignments
+  -s  string / row functions
+"""
+
+import argparse
+import logging
+import re
+
+import numpy as np
+import pandas as pd
+
+from dftk.commands.base import BaseCommand
+from dftk.common.evalfuncs import (
+    SPECIAL_FUNCTIONS,
+    FormulaMistake,
+    apply_special_function,
+    is_special_call,
+    string_function,
+)
+from dftk.common.io import io
+
+logger = logging.getLogger(__name__)
+
+
+_EPILOG = """\
+FORMULA EXPRESSIONS (-f)
+-------------------------
+Each -f argument is first passed to pandas df.eval().  This handles
+arithmetic and boolean expressions directly on column names:
+
+  dftk eval data.tsv -f "z = x + y * 2"
+  dftk dataset tips -o | dftk eval ... -f "bill_and_tip = total_bill + tip * 2"
+  dftk dataset tips -o | dftk eval ... -f "good_tip = (tip > 3) & (total_bill < 30)"
+
+Column methods are pandas Series methods, written COL.method(...); text
+methods are under .str (name.str.lower(), not name.lower() or tolower()):
+
+  x = x.abs()   x = x.round(1)   x = x.clip(lower=0)   y = name.str.upper()
+
+
+FILLING MISSING VALUES (-f)
+---------------------------
+  -f "x = x.fillna(y)"              from another column (or a value: fillna(0))
+  -f "x = x.fillna(y).fillna(0)"    y first, then 0
+  -f "x = x.ffill()"                the value above (bfill(): the value below)
+  -f "x = x.interpolate()"          a straight line, by row position
+  -f "x = x.groupby(g).ffill()"     within each group of g (also bfill)
+  -f "x = x.fillna(x.groupby(g).transform('mean'))"   the group mean
+
+ffill, bfill and interpolate use the rows in their order: sort first, in a
+step of its own (--sortasc applies to the output), and fill within groups
+when the table holds several series, or values come from the next group:
+
+  dftk dataset healthexp -o \\
+      | dftk eval ... -f "le = Life_Expectancy.where(Year % 10 != 5)" \\
+          --sortasc Country Year -o \\
+      | dftk eval ... -f "le_filled = le.groupby(Country).ffill()"
+
+interpolate() has no grouped form. The README ("Filling missing values")
+covers defaults from a merge, interpolating within groups, and more.
+
+
+DFTK FUNCTIONS (-f)
+-------------------
+A formula whose right-hand side is one call to a function below runs that
+function; any other formula goes to pandas df.eval():
+
+  dest = func(col1, col2, ...)
+
+The call must be the whole right-hand side: for z = sum(a, b) * 2, write
+-f "tmp = sum(a, b)" -f "z = tmp * 2".
+
+Available special functions:
+
+  Row aggregation (applied across named columns):
+    sum  mean  std  min  max  median
+
+  Conditional assignment:
+    where(cond_col, true_val, false_val)
+      — np.where-style conditional; cond_col must be a boolean column.
+        true_val and false_val can each be: a column name, a quoted
+        string literal ("foo"), or a numeric literal.
+      Example:
+        dftk dataset tips -o | dftk eval ... \
+          -f "big = total_bill > 30" \
+          -f "label = where(big, \"large\", \"small\")"
+
+  Bitwise:
+    applymask(col, mask_col)     — col & mask_col
+    overlap(lo1, hi1, lo2, hi2)  — length of interval overlap
+
+  Column aggregation (glob pattern):
+    colsum(col*)     — sum of all columns matching the glob pattern
+
+  NumPy math (element-wise):
+    sign(col)  cbrt(col)  log1p(col)  expm1(col)
+
+  scipy.special — log-gamma / digamma:
+    gamma(col)   lgamma(col)   digamma(col)
+    beta(a, b)   betaln(a, b)
+
+  scipy.special — error functions:
+    erf(col)   erfc(col)   erfinv(col)
+
+  scipy.special — sigmoid / logit:
+    expit(col)   logit(col)
+
+  scipy.special — Bessel functions (fixed order):
+    j0(col)  j1(col)   — Bessel J, order 0 / 1
+    y0(col)  y1(col)   — Bessel Y, order 0 / 1
+    i0(col)  i1(col)   — modified Bessel I, order 0 / 1
+    k0(col)  k1(col)   — modified Bessel K, order 0 / 1
+
+  scipy.special — Bessel functions (parametric order):
+    jn(n, col)   — Bessel J, integer order n
+    yn(n, col)   — Bessel Y, integer order n
+    iv(v, col)   — modified Bessel I, real order v
+    kv(v, col)   — modified Bessel K, real order v
+    (n/v is a numeric literal, e.g.  result = jn(3, x))
+
+  Path functions (applied element-wise):
+    basename  dirname  exists  getsize  realpath
+
+  Statistical functions (see common/stats.py):
+    binom_test(k, n, p)
+    fisher_test(a, b, c, d)      — p-value
+    fisher_OR(a, b, c, d)        — odds ratio
+    boschloo_test(a, b, c, d)    — p-value
+    boschloo_OR(a, b, c, d)      — odds ratio
+    pval2se(effect, pval)
+    t2pval(t, df)
+    chi2_to_neglogp(chi2, df)
+    neglogp_to_chi2(neglogp, df)
+    pval_to_chi2(pval, df)
+    generalized_poisson_nll(x, lam, theta)
+
+
+CONSTANT COLUMNS (-c)
+----------------------
+Add a column with a fixed value:
+
+  dftk dataset tips -o | dftk eval ... -c "batch = 3"
+  dftk dataset tips -o | dftk eval ... -c "region = northeast"
+
+The value is coerced to numeric when possible.
+
+
+STRING / ROW FUNCTIONS (-s)
+----------------------------
+Apply a string or row-level function using the same dest = func(cols) syntax:
+
+  String operations:
+    translate(col, from, to)
+    len(col)
+    join(col1, col2, ...)        — joined with --joinsep (default: "")
+    lower(col)  upper(col)
+    zfill(col, width)
+    find(col, substr)
+    match(col, pattern_col)
+    substr(col, start, length)
+    substring(col, start_col, length)
+    replace(col, old, new)
+    leftsplit(col, sep, idx)     — split on first sep; idx=0 left part, 1 right
+    rightsplit(col, sep, idx)    — split on last sep
+    sort(col)                    — sort characters in string
+    ord(col)                     — minimum ord value of characters
+    template(...)                — Python str.format_map across all columns
+
+  Path operations:
+    basename(col)  stem(col)  realpath(col)  isfile(col)  exists(col)
+
+  Crypto / hash:
+    md5(col)
+
+  DNA / sequence (see common/seq.py):
+    gccontent(col)
+    basecount(col)
+    motifcount(pattern, seq_col)
+    readoffset(read_pos, ref_pos, cigar, seq, count)
+
+  Edit distance (requires python-Levenshtein):
+    levenshtein(col1, col2)
+"""
+
+# ---------------------------------------------------------------------------
+# Core eval logic
+# ---------------------------------------------------------------------------
+
+
+# A message pandas gives for a method that no column has, whatever its dtype
+_NO_METHOD = re.compile(r"'Series' object has no attribute '(\w+)'")
+
+# pandas' message for a function it doesn't know
+_NO_FUNCTION = re.compile(r'"(\w+)" is not a supported function')
+
+
+def _formula_error(formula: str, exc: Exception) -> str:
+    """Return a short message for a mistake pandas found in an -f formula."""
+    msg = str(exc)
+    m = _NO_METHOD.search(msg)
+    if m:
+        name = m.group(1)
+        col = re.search(rf"([A-Za-z_]\w*)\.{name}\b", formula)
+        col = col.group(1) if col else "COL"
+        if hasattr(pd.Series.str, name):
+            hint = f"Text methods are under .str: {col}.str.{name}()."
+        else:
+            hint = f"Text methods are under .str, e.g. {col}.str.lower()."
+        return (
+            f"-f {formula!r}: column has no method {name!r}. {hint}  See dftk eval -h."
+        )
+    m = _NO_FUNCTION.search(msg)
+    if m:
+        name = m.group(1)
+        if name in SPECIAL_FUNCTIONS:
+            return (
+                f"-f {formula!r}: {name}() must be the whole right-hand side, "
+                f"e.g. -f 'tmp = {name}(...)' -f 'z = tmp * 2'."
+            )
+        return f"-f {formula!r}: unknown function {name!r}.  See dftk eval -h."
+    return f"-f {formula!r}: {msg}"
+
+
+def _eval_formula(df: pd.DataFrame, formula: str) -> pd.DataFrame:
+    """Evaluate one -f formula: a special function, or else pandas df.eval()."""
+    if is_special_call(formula):
+        apply_special_function(df, formula)
+        return df
+    try:
+        # Not inplace: that writes into the existing column, which must keep
+        # its dtype, so x = x.fillna(y) or x = x.isna() failed whenever the
+        # result had another dtype
+        result = df.eval(formula)
+    except (NameError, SyntaxError) as exc:
+        # An unknown column (pandas' UndefinedVariableError) or bad syntax
+        raise FormulaMistake(_formula_error(formula, exc)) from exc
+    except (AttributeError, TypeError, KeyError, ValueError) as exc:
+        # Raised while pandas evaluated the user's expression, so most likely
+        # a mistake in it (wrong method, arguments, or .str on numbers)
+        msg = str(exc)
+        mistake = _NO_METHOD.search(msg) or _NO_FUNCTION.search(msg)
+        cls = FormulaMistake if mistake else ValueError
+        raise cls(_formula_error(formula, exc)) from exc
+    if not isinstance(result, pd.DataFrame):
+        # No "dest =": df.eval returned the values instead of a table
+        raise FormulaMistake(
+            f"-f {formula!r} has no assignment; write it as 'name = {formula}'"
+        )
+    return result
+
+
+def _eval(df: pd.DataFrame, args: argparse.Namespace) -> pd.DataFrame:
+    joinsep: str = getattr(args, "joinsep", "")
+    intbool: bool = getattr(args, "intbool", False)
+    existing = set(df.columns)
+
+    # -c: constant columns
+    for expr in getattr(args, "constant", []):
+        colname, constval_str = (x.strip() for x in expr.split("=", 1))
+        try:
+            constval: int | float | str = int(constval_str)
+        except ValueError:
+            try:
+                constval = float(constval_str)
+            except ValueError:
+                constval = constval_str
+        df[colname] = constval
+
+    # -f: special functions, or else pandas df.eval
+    for formula in getattr(args, "formula", []):
+        try:
+            df = _eval_formula(df, formula)
+        except Exception as exc:
+            # An empty table has no column types (pandas reads them as
+            # float64), so on one only mistakes that don't depend on types
+            # are reported; for the rest the column is just added
+            if len(df) != 0 or isinstance(exc, FormulaMistake):
+                raise
+            df[formula.split("=", 1)[0].strip()] = True
+
+    # -s: string / row functions
+    for strfunc in getattr(args, "strfunc", []):
+        try:
+            dest, func, cols = string_function(strfunc, joinsep)
+            if len(df) == 0:
+                df[dest] = True
+            elif len(cols) == 0:
+                df[dest] = df.apply(func, axis=1)
+            elif len(cols) == 1:
+                df[dest] = df[cols[0]].apply(func)
+            else:
+                df[dest] = df[cols].apply(func, axis=1)
+        except Exception:
+            if len(df) == 0:
+                dest = strfunc.split("=", 1)[0].strip()
+                df[dest] = True
+            else:
+                logger.error(f"Could not evaluate string function: {strfunc!r}")
+                raise
+
+    # --intbool: convert newly created boolean columns to int
+    if intbool:
+        new_cols = set(df.columns) - existing
+        for col in new_cols:
+            if df[col].dtype == np.bool_:
+                df[col] = df[col].astype(np.int64)
+
+    return df
+
+
+# ---------------------------------------------------------------------------
+# Command
+# ---------------------------------------------------------------------------
+
+
+class EvalCommand(BaseCommand):
+    @property
+    def name(self) -> str:
+        return "eval"
+
+    @property
+    def help(self) -> str:
+        return "Evaluate formulas, constants, and string functions on a DataFrame"
+
+    def add_arguments(self, parser: argparse.ArgumentParser) -> None:
+        import argparse as ap
+
+        parser.formatter_class = ap.RawDescriptionHelpFormatter
+        parser.epilog = _EPILOG
+
+        io.parser_read(parser)
+
+        g = parser.add_argument_group("formula evaluation")
+        g.add_argument(
+            "-f",
+            "--formula",
+            help=(
+                "pandas eval expression, or special function call "
+                "(dest = func(col1, col2, ...)).  Repeatable."
+            ),
+            default=[],
+            action="append",
+            metavar="EXPR",
+        )
+        g.add_argument(
+            "-c",
+            "--constant",
+            help=(
+                "Add a constant column (dest = value).  "
+                "Value is coerced to numeric when possible.  Repeatable."
+            ),
+            default=[],
+            action="append",
+            metavar="EXPR",
+        )
+        g.add_argument(
+            "-s",
+            "--strfunc",
+            help=(
+                "String or row function (dest = func(col1, col2, ...)).  "
+                "Repeatable.  See --help for full function list."
+            ),
+            default=[],
+            action="append",
+            metavar="EXPR",
+        )
+        g.add_argument(
+            "--joinsep",
+            help="Separator used by the join() string function (default: empty).",
+            default="",
+            metavar="SEP",
+        )
+        g.add_argument(
+            "--intbool",
+            help="Convert newly created boolean columns to int (False=0, True=1).",
+            action="store_true",
+        )
+
+        io.parser_output(parser)
+
+    def execute(self, args: argparse.Namespace) -> None:
+        df = io.read(args)
+        df = _eval(df, args)
+        io.printdf(df, args)

{dataframe_toolkit-0.11.0 → dataframe_toolkit-0.11.2}/src/dftk/commands/implode_cmd.py

@@ -9,6 +9,7 @@ original rows.
 
 import argparse
 
+import numpy as np
 import pandas as pd
 
 from dftk.commands.base import BaseCommand
@@ -68,6 +69,7 @@ def _as_text(s: pd.Series) -> pd.Series:
     if (
         pd.api.types.is_float_dtype(s)
         and len(values)
+        and np.isfinite(values).all()
         and (values == values.round()).all()
     ):
         # Integers with gaps are read as floats: write 10, not 10.0
@@ -97,23 +99,26 @@ def _implode(df: pd.DataFrame, args: argparse.Namespace) -> pd.DataFrame:
 
     order = [c for c in df.columns if c in keys or c in cols]
     work = df[keys + cols + ([poscol] if poscol else [])].reset_index(drop=True)
+    gid = "_gid"  # helper column for the group number; never one of the user's
+    while gid in work.columns:
+        gid = "_" + gid
     if keys:
-        work["_gid"] = work.groupby(keys, sort=False, dropna=False).ngroup()
+        work[gid] = work.groupby(keys, sort=False, dropna=False).ngroup()
     else:
-        work["_gid"] = 0
+        work[gid] = 0
 
     if args.sortvalues:
-        work = work.sort_values(["_gid", *cols], kind="stable", na_position="last")
+        work = work.sort_values([gid, *cols], kind="stable", na_position="last")
     elif poscol:
-        work = work.sort_values(["_gid", poscol], kind="stable")
+        work = work.sort_values([gid, poscol], kind="stable")
     else:
-        work = work.sort_values("_gid", kind="stable")
+        work = work.sort_values(gid, kind="stable")
     if args.unique:
-        work = work.drop_duplicates(subset=["_gid", *cols])
+        work = work.drop_duplicates(subset=[gid, *cols])
 
     for col in cols:
         work[col] = _as_text(work[col])
-    grouped = work.groupby("_gid", sort=True)
+    grouped = work.groupby(gid, sort=True)
     joined = grouped[cols].agg(lambda s: args.sep.join(s.dropna()))
     joined = joined.where(joined != "")
     result = grouped[keys].first().join(joined) if keys else joined