dataframe-toolkit 0.5.0__tar.gz → 0.7.0__tar.gz

{dataframe_toolkit-0.5.0 → dataframe_toolkit-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: dataframe-toolkit
-Version: 0.5.0
+Version: 0.7.0
 Summary: A command-line toolkit for DataFrame analysis and manipulation
 Keywords: dataframe,statistics,data analysis,CLI,TSV,pandas
 Author: Gunnar Pálsson
@@ -38,6 +38,14 @@ for analysts who live in the terminal and want a consistent, pipeable toolkit
 for the full data analysis pipeline — from initial exploration through
 statistical modelling and publication-quality figures.
 
+**No data file handy?** Every example below uses `dftk dataset` to pull in a
+small, real dataset, so you can copy-paste any of them and run them right
+now. Run `dftk dataset --list` to browse everything available (dozens of
+datasets from seaborn, statsmodels, and pydataset). `dftk dataset NAME`
+searches all three sources in that order and returns the first match, so
+`--source` is optional — pass it only to disambiguate a name that exists in
+more than one source, or to restrict the search.
+
 ## Installation
 
 ```bash
@@ -68,28 +76,38 @@ uv tool install .
 ## Quick example
 
 ```bash
-dftk stat data.tsv -c height weight -g sex
-dftk stat data.tsv -c value -g group -o \
-  | dftk pivot -i group -v value_mean -o \
-  | dftk print
+dftk dataset tips | dftk stat - -c tip total_bill -g day
+```
+
+Chain commands with `-o` (write to a temp parquet, print its path) and `...`
+(read a parquet path from stdin) instead of piping TSV text directly — faster
+for large data, and the recommended pattern once a pipeline has more than one
+step:
+
+```bash
+dftk dataset tips -o \
+  | dftk pivot ... -i day -v tip -f mean -o \
+  | dftk print ...
 ```
 
 ## Subcommands
 
 ### Data transformation
 
-| Command   | Description                                                                          |
-|-----------|--------------------------------------------------------------------------------------|
-| `eval`    | Add or modify columns: eval expressions, string/path functions, statistical ops      |
-| `query`   | Filter rows with pandas query expressions or SQL (`--sql` via DuckDB)                |
-| `merge`   | Join two tables on key columns (inner/left/right/outer)                              |
-| `concat`  | Concatenate two or more tables row-wise                                              |
-| `melt`    | Reshape wide-to-long (`pd.melt`)                                                     |
-| `pivot`   | Reshape long-to-wide with per-cell aggregation                                       |
-| `func`    | Column transforms: cumsum, group mean/sum/min/max/count/median/std, rank, qcut:N     |
-| `scale`   | Normalise: z-score, min-shift, sum/max/mean scaling, Blom rank, regression residuals |
-| `interp`  | Interpolate values from a reference curve into a table (1-D lookup)                  |
-| `binx`    | Assign bin indices to a column based on explicit or generated edges                  |
+| Command     | Description                                                                          |
+|-------------|---------------------------------------------------------------------------------------|
+| `eval`      | Add or modify columns: eval expressions, string/path functions, statistical ops      |
+| `query`     | Filter rows with pandas query expressions or SQL (`--sql` via DuckDB)                |
+| `merge`     | Join two tables on key columns (inner/left/right/outer)                              |
+| `concat`    | Concatenate two or more tables row-wise                                              |
+| `melt`      | Reshape wide-to-long (`pd.melt`)                                                     |
+| `pivot`     | Reshape long-to-wide with per-cell aggregation                                       |
+| `func`      | Column transforms: cumsum, group mean/sum/min/max/count/median/std, rank, qcut:N     |
+| `scale`     | Normalise: z-score, min-shift, sum/max/mean scaling, Blom rank, regression residuals |
+| `interp`    | Interpolate values from a reference curve into a table (1-D lookup)                  |
+| `binx`      | Assign bin indices to a column based on explicit or generated edges                  |
+| `transpose` | Flip rows and columns; original column names land in a new `--keycol` column         |
+| `segid`     | Assign a segment ID that increments on each value change in a column                 |
 
 ### Statistics
 
@@ -101,6 +119,7 @@ dftk stat data.tsv -c value -g group -o \
 | `test`     | P-values between column pairs: t-test, Mann-Whitney, Wilcoxon, KS, correlations, bootstrap; groups  |
 | `corr`     | Pairwise column correlations (Pearson/Spearman/Kendall) with optional BCa bootstrap CI              |
 | `describe` | Quick column-level summary (dtype, n, n_unique, n_null, sample values)                              |
+| `info`     | Per-column dtype, null counts, and memory usage; `--summary` for dataset-level totals               |
 | `randvar`  | Sample from a distribution and append as a new column (norm, alpha, beta, …)                        |
 
 ### Plots
@@ -133,38 +152,48 @@ publication-quality output.
 ### Chaining commands
 
 ```bash
-# z-score within group, then fit a model
-dftk scale data.tsv -c expr -g condition -o \
-  | dftk fit - -f "expr_z ~ time + batch" -g condition
+# z-score sepal_length within species, then fit against petal_length
+dftk dataset iris -o \
+  | dftk scale ... -c sepal_length -g species -o \
+  | dftk fit ... -f "sepal_length_scaled ~ petal_length" -g species
 ```
 
 ### Group summary then plot
 
 ```bash
-dftk stat results.tsv -c value -g group -o \
-  | dftk line - -x group -y value_mean --yerr value_sem -f fig.png
+dftk dataset tips -o \
+  | dftk stat ... -c tip -g day -o \
+  | dftk line ... -x day -y mean --yerr sem -f fig.png
 ```
 
 ### Wide-to-long then plot overlaid histograms
 
 ```bash
-dftk melt data.tsv -i sample -d gene -v expression \
-  | dftk hist - -x expression -g gene -k -f dist.png
+dftk dataset iris \
+  | dftk melt - -i species -d measurement -v value \
+  | dftk hist - -x value -g measurement -k -f dist.png
 ```
 
 ### Interpolation (standard curve lookup)
 
 ```bash
-dftk interp samples.tsv --ref stdcurve.tsv \
-    -x fluorescence -v concentration -d conc_ng_ul
+printf "conc\tfluor\n0\t5\n10\t52\n20\t98\n30\t151\n" > stdcurve.tsv
+printf "sample\tfluor\ns1\t60\ns2\t110\n" \
+  | dftk interp - --ref stdcurve.tsv -x fluor --refx fluor -v conc -d conc_ng_ul
 ```
 
 ### Bootstrap confidence intervals
 
+Bootstrap mode repeats the full stat computation N times on resampled data
+and emits one row per resample (tagged with `samplenum`) — pipe into `pivot`
+with percentile aggfuncs (`cilo`/`cihi`) to collapse that into an empirical
+confidence interval:
+
 ```bash
-dftk stat data.tsv -c value -g group --bootstrap 1000 --randomseed 42 -o \
-  | dftk pivot -i group -v value_mean -f mean -o \
-  | dftk stat - -c group_A group_B
+dftk dataset tips -o \
+  | dftk stat ... -c tip -g day --bootstrap 1000 --randomseed 42 -o \
+  | dftk pivot ... -i day -v mean -f mean cilo cihi -o \
+  | dftk print ...
 ```
 
 ## Input/output
@@ -176,9 +205,19 @@ All commands accept:
 - `...` to receive a parquet path from stdin (written by a previous `-o` command)
 - A `.parquet` filename to read a named parquet file directly
 
+> **`-` vs `...`** — these are not interchangeable. Use `-` when the previous
+> command in the pipe wrote plain TSV to stdout (the default). Use `...` when
+> the previous command used bare `-o` (pipe mode), which writes a temp
+> parquet and prints *its path* to stdout — `-` would try to parse that path
+> as TSV and fail.
+
 Standard output options (available on all tabular commands):
 
-- `-o` / `--output` — write a temp parquet for the next piped command instead of TSV to stdout
+- `-o` / `--output` — controls where output goes; takes an *optional* value:
+  - `-o` alone — write a temp parquet to pipe into the next command, printing its path to stdout (auto-deleted once read)
+  - `-o FILE.parquet` — write a named, reusable parquet, printing its path to stdout
+  - `-o FILE` (no `.parquet` extension) — write TSV to `FILE`, with no stdout output
+  - omit `-o` entirely — write TSV to stdout (the default)
 - `--select col1 col2 …` — keep only these columns
 - `--drop col1 col2 …` — remove these columns
 - `--round N` — round numeric output
@@ -193,20 +232,19 @@ any new `--meta` values.
 
 ```bash
 # Tag a file at creation
-dftk eval raw.tsv -f "z = x + y" -o results.parquet \
-    --meta genome=hg38 --meta source=gwas_2024
+dftk dataset iris -o iris.parquet --meta source=seaborn --meta species_col=species
 
 # Inspect annotations
-dftk annotate results.parquet
-# genome   hg38
-# source   gwas_2024
+dftk annotate iris.parquet
+# source        seaborn
+# species_col   species
 
 # Add or update an annotation in-place
-dftk annotate results.parquet --set step=qc_filtered
+dftk annotate iris.parquet --set step=raw
 
 # Annotations survive piping
-dftk eval results.parquet -f "z_scaled = z / 2" -o | dftk scale ... -c z -o scaled.parquet
-dftk annotate scaled.parquet   # genome and source still present
+dftk eval iris.parquet -f "petal_area = petal_length * petal_width" -o iris2.parquet
+dftk annotate iris2.parquet   # source, species_col, and step all still present
 ```
 
 ## Figure options (plot commands)

{dataframe_toolkit-0.5.0 → dataframe_toolkit-0.7.0}/README.md

@@ -10,6 +10,14 @@ for analysts who live in the terminal and want a consistent, pipeable toolkit
 for the full data analysis pipeline — from initial exploration through
 statistical modelling and publication-quality figures.
 
+**No data file handy?** Every example below uses `dftk dataset` to pull in a
+small, real dataset, so you can copy-paste any of them and run them right
+now. Run `dftk dataset --list` to browse everything available (dozens of
+datasets from seaborn, statsmodels, and pydataset). `dftk dataset NAME`
+searches all three sources in that order and returns the first match, so
+`--source` is optional — pass it only to disambiguate a name that exists in
+more than one source, or to restrict the search.
+
 ## Installation
 
 ```bash
@@ -40,28 +48,38 @@ uv tool install .
 ## Quick example
 
 ```bash
-dftk stat data.tsv -c height weight -g sex
-dftk stat data.tsv -c value -g group -o \
-  | dftk pivot -i group -v value_mean -o \
-  | dftk print
+dftk dataset tips | dftk stat - -c tip total_bill -g day
+```
+
+Chain commands with `-o` (write to a temp parquet, print its path) and `...`
+(read a parquet path from stdin) instead of piping TSV text directly — faster
+for large data, and the recommended pattern once a pipeline has more than one
+step:
+
+```bash
+dftk dataset tips -o \
+  | dftk pivot ... -i day -v tip -f mean -o \
+  | dftk print ...
 ```
 
 ## Subcommands
 
 ### Data transformation
 
-| Command   | Description                                                                          |
-|-----------|--------------------------------------------------------------------------------------|
-| `eval`    | Add or modify columns: eval expressions, string/path functions, statistical ops      |
-| `query`   | Filter rows with pandas query expressions or SQL (`--sql` via DuckDB)                |
-| `merge`   | Join two tables on key columns (inner/left/right/outer)                              |
-| `concat`  | Concatenate two or more tables row-wise                                              |
-| `melt`    | Reshape wide-to-long (`pd.melt`)                                                     |
-| `pivot`   | Reshape long-to-wide with per-cell aggregation                                       |
-| `func`    | Column transforms: cumsum, group mean/sum/min/max/count/median/std, rank, qcut:N     |
-| `scale`   | Normalise: z-score, min-shift, sum/max/mean scaling, Blom rank, regression residuals |
-| `interp`  | Interpolate values from a reference curve into a table (1-D lookup)                  |
-| `binx`    | Assign bin indices to a column based on explicit or generated edges                  |
+| Command     | Description                                                                          |
+|-------------|---------------------------------------------------------------------------------------|
+| `eval`      | Add or modify columns: eval expressions, string/path functions, statistical ops      |
+| `query`     | Filter rows with pandas query expressions or SQL (`--sql` via DuckDB)                |
+| `merge`     | Join two tables on key columns (inner/left/right/outer)                              |
+| `concat`    | Concatenate two or more tables row-wise                                              |
+| `melt`      | Reshape wide-to-long (`pd.melt`)                                                     |
+| `pivot`     | Reshape long-to-wide with per-cell aggregation                                       |
+| `func`      | Column transforms: cumsum, group mean/sum/min/max/count/median/std, rank, qcut:N     |
+| `scale`     | Normalise: z-score, min-shift, sum/max/mean scaling, Blom rank, regression residuals |
+| `interp`    | Interpolate values from a reference curve into a table (1-D lookup)                  |
+| `binx`      | Assign bin indices to a column based on explicit or generated edges                  |
+| `transpose` | Flip rows and columns; original column names land in a new `--keycol` column         |
+| `segid`     | Assign a segment ID that increments on each value change in a column                 |
 
 ### Statistics
 
@@ -73,6 +91,7 @@ dftk stat data.tsv -c value -g group -o \
 | `test`     | P-values between column pairs: t-test, Mann-Whitney, Wilcoxon, KS, correlations, bootstrap; groups  |
 | `corr`     | Pairwise column correlations (Pearson/Spearman/Kendall) with optional BCa bootstrap CI              |
 | `describe` | Quick column-level summary (dtype, n, n_unique, n_null, sample values)                              |
+| `info`     | Per-column dtype, null counts, and memory usage; `--summary` for dataset-level totals               |
 | `randvar`  | Sample from a distribution and append as a new column (norm, alpha, beta, …)                        |
 
 ### Plots
@@ -105,38 +124,48 @@ publication-quality output.
 ### Chaining commands
 
 ```bash
-# z-score within group, then fit a model
-dftk scale data.tsv -c expr -g condition -o \
-  | dftk fit - -f "expr_z ~ time + batch" -g condition
+# z-score sepal_length within species, then fit against petal_length
+dftk dataset iris -o \
+  | dftk scale ... -c sepal_length -g species -o \
+  | dftk fit ... -f "sepal_length_scaled ~ petal_length" -g species
 ```
 
 ### Group summary then plot
 
 ```bash
-dftk stat results.tsv -c value -g group -o \
-  | dftk line - -x group -y value_mean --yerr value_sem -f fig.png
+dftk dataset tips -o \
+  | dftk stat ... -c tip -g day -o \
+  | dftk line ... -x day -y mean --yerr sem -f fig.png
 ```
 
 ### Wide-to-long then plot overlaid histograms
 
 ```bash
-dftk melt data.tsv -i sample -d gene -v expression \
-  | dftk hist - -x expression -g gene -k -f dist.png
+dftk dataset iris \
+  | dftk melt - -i species -d measurement -v value \
+  | dftk hist - -x value -g measurement -k -f dist.png
 ```
 
 ### Interpolation (standard curve lookup)
 
 ```bash
-dftk interp samples.tsv --ref stdcurve.tsv \
-    -x fluorescence -v concentration -d conc_ng_ul
+printf "conc\tfluor\n0\t5\n10\t52\n20\t98\n30\t151\n" > stdcurve.tsv
+printf "sample\tfluor\ns1\t60\ns2\t110\n" \
+  | dftk interp - --ref stdcurve.tsv -x fluor --refx fluor -v conc -d conc_ng_ul
 ```
 
 ### Bootstrap confidence intervals
 
+Bootstrap mode repeats the full stat computation N times on resampled data
+and emits one row per resample (tagged with `samplenum`) — pipe into `pivot`
+with percentile aggfuncs (`cilo`/`cihi`) to collapse that into an empirical
+confidence interval:
+
 ```bash
-dftk stat data.tsv -c value -g group --bootstrap 1000 --randomseed 42 -o \
-  | dftk pivot -i group -v value_mean -f mean -o \
-  | dftk stat - -c group_A group_B
+dftk dataset tips -o \
+  | dftk stat ... -c tip -g day --bootstrap 1000 --randomseed 42 -o \
+  | dftk pivot ... -i day -v mean -f mean cilo cihi -o \
+  | dftk print ...
 ```
 
 ## Input/output
@@ -148,9 +177,19 @@ All commands accept:
 - `...` to receive a parquet path from stdin (written by a previous `-o` command)
 - A `.parquet` filename to read a named parquet file directly
 
+> **`-` vs `...`** — these are not interchangeable. Use `-` when the previous
+> command in the pipe wrote plain TSV to stdout (the default). Use `...` when
+> the previous command used bare `-o` (pipe mode), which writes a temp
+> parquet and prints *its path* to stdout — `-` would try to parse that path
+> as TSV and fail.
+
 Standard output options (available on all tabular commands):
 
-- `-o` / `--output` — write a temp parquet for the next piped command instead of TSV to stdout
+- `-o` / `--output` — controls where output goes; takes an *optional* value:
+  - `-o` alone — write a temp parquet to pipe into the next command, printing its path to stdout (auto-deleted once read)
+  - `-o FILE.parquet` — write a named, reusable parquet, printing its path to stdout
+  - `-o FILE` (no `.parquet` extension) — write TSV to `FILE`, with no stdout output
+  - omit `-o` entirely — write TSV to stdout (the default)
 - `--select col1 col2 …` — keep only these columns
 - `--drop col1 col2 …` — remove these columns
 - `--round N` — round numeric output
@@ -165,20 +204,19 @@ any new `--meta` values.
 
 ```bash
 # Tag a file at creation
-dftk eval raw.tsv -f "z = x + y" -o results.parquet \
-    --meta genome=hg38 --meta source=gwas_2024
+dftk dataset iris -o iris.parquet --meta source=seaborn --meta species_col=species
 
 # Inspect annotations
-dftk annotate results.parquet
-# genome   hg38
-# source   gwas_2024
+dftk annotate iris.parquet
+# source        seaborn
+# species_col   species
 
 # Add or update an annotation in-place
-dftk annotate results.parquet --set step=qc_filtered
+dftk annotate iris.parquet --set step=raw
 
 # Annotations survive piping
-dftk eval results.parquet -f "z_scaled = z / 2" -o | dftk scale ... -c z -o scaled.parquet
-dftk annotate scaled.parquet   # genome and source still present
+dftk eval iris.parquet -f "petal_area = petal_length * petal_width" -o iris2.parquet
+dftk annotate iris2.parquet   # source, species_col, and step all still present
 ```
 
 ## Figure options (plot commands)

dataframe_toolkit-0.7.0/pyproject.toml

@@ -0,0 +1,82 @@
+[project]
+name = "dataframe-toolkit"
+version = "0.7.0"
+description = "A command-line toolkit for DataFrame analysis and manipulation"
+readme = "README.md"
+keywords = [
+    "dataframe",
+    "statistics",
+    "data analysis",
+    "CLI",
+    "TSV",
+    "pandas",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "Intended Audience :: Developers",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+    "Topic :: Utilities",
+    "Programming Language :: Python :: 3",
+]
+requires-python = ">=3.12"
+dependencies = [
+    "pandas>=2.1",
+    "numpy>=1.24",
+    "pyarrow>=12.0",
+    "scipy>=1.10",
+    "statsmodels>=0.14",
+    "duckdb>=0.8",
+    "matplotlib>=3.7",
+    "seaborn>=0.12",
+    "tqdm>=4.65",
+]
+
+[[project.authors]]
+name = "Gunnar Pálsson"
+email = "hemingur@gmail.com"
+
+[project.license]
+text = "MIT"
+
+[project.urls]
+Homepage = "https://github.com/hemingur/dataframe-toolkit"
+Repository = "https://github.com/hemingur/dataframe-toolkit"
+
+[project.scripts]
+dftk = "dftk.cli:main"
+
+[build-system]
+requires = ["uv_build>=0.10.4,<0.11.0"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+module-name = "dftk"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 88
+target-version = "py312"
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "W",
+    "F",
+    "I",
+    "UP",
+    "B",
+]
+
+[tool.ruff.lint.isort]
+known-first-party = ["dftk"]
+
+[dependency-groups]
+dev = [
+    "pre-commit>=4.6.0",
+    "pytest>=8.0",
+    "pytest-cov>=7.1.0",
+    "ruff>=0.15.12",
+]

dataframe_toolkit-0.5.0/pyproject.toml → dataframe_toolkit-0.7.0/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.5.0"
+version = "0.7.0"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 authors = [

{dataframe_toolkit-0.5.0 → dataframe_toolkit-0.7.0}/src/dftk/cli.py

@@ -8,6 +8,8 @@ logic and never needs to change when a new subcommand is added.
 
 import argparse
 import logging
+import os
+import signal
 import sys
 
 from dftk import __version__
@@ -47,6 +49,13 @@ def main() -> None:
 
     try:
         args.command_instance.execute(args)
+        sys.stdout.flush()
+    except BrokenPipeError:
+        # The reader closed the pipe early (e.g. `dftk ... | head`).  Exit
+        # quietly with the SIGPIPE status like cat/grep, and point stdout at
+        # devnull so the interpreter's final flush doesn't raise again.
+        os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno())
+        sys.exit(128 + signal.SIGPIPE)
     except FileNotFoundError as exc:
         print(f"File error: {exc}", file=sys.stderr)
         sys.exit(1)

{dataframe_toolkit-0.5.0 → dataframe_toolkit-0.7.0}/src/dftk/commands/__init__.py

@@ -8,6 +8,7 @@
 
 from dftk.commands.annotate_cmd import AnnotateCommand
 from dftk.commands.binx_cmd import BinxCommand
+from dftk.commands.chi_cmd import ChiCommand
 from dftk.commands.clean_cmd import CleanCommand
 from dftk.commands.concat_cmd import ConcatCommand
 from dftk.commands.corr_cmd import CorrCommand
@@ -80,6 +81,7 @@ COMMAND_GROUPS: list[tuple[str, list]] = [
             ScatCommand(),
             LineCommand(),
             HistCommand(),
+            ChiCommand(),
         ],
     ),
     (

{dataframe_toolkit-0.5.0 → dataframe_toolkit-0.7.0}/src/dftk/commands/annotate_cmd.py

@@ -25,6 +25,10 @@ Usage examples:
   dftk eval data.tsv -f "z = x + y" -o result.parquet \\
       --meta genome=hg38 --meta source=my_pipeline
 
+  # Same, built from a bundled dataset:
+  dftk dataset iris -o | dftk eval ... -f "z = sepal_length + petal_length" \\
+      -o result.parquet --meta genome=hg38 --meta source=my_pipeline
+
 Annotations propagate through the parquet pipe: a file annotated at
 creation is re-annotated on every subsequent -o write unless explicitly
 overridden.

{dataframe_toolkit-0.5.0 → dataframe_toolkit-0.7.0}/src/dftk/commands/binx_cmd.py

@@ -74,8 +74,10 @@ USEVALUE (--usevalue)
 EXAMPLES
 --------
   dftk binx data.tsv -c age -b 0:100:10
-  dftk binx data.tsv -c score -b 0,25,50,75,100 -d quartile --usevalue m
-  dftk binx data.tsv -c x -b -3:4:1 -d x_bin -o | dftk stat -g x_bin -s y
+  dftk dataset tips -o | dftk binx ... -c total_bill -b 0:60:10
+  dftk dataset tips -o | dftk binx ... -c tip -b 0,2,4,6,10 -d tip_bracket --usevalue m
+  dftk dataset tips -o | dftk binx ... -c total_bill -b 0:60:10 -d bill_bin -o \\
+      | dftk stat ... -c tip -g bill_bin
 """