dataframe-toolkit 0.12.0__tar.gz → 0.14.0__tar.gz

{dataframe_toolkit-0.12.0 → dataframe_toolkit-0.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: dataframe-toolkit
-Version: 0.12.0
+Version: 0.14.0
 Summary: A command-line toolkit for DataFrame analysis and manipulation
 Keywords: dataframe,statistics,data analysis,CLI,TSV,pandas
 Author: Gunnar Pálsson
@@ -193,6 +193,23 @@ dftk dataset fertility -o \
 
 For your own export, `dftk cleannames data.tsv --backend duckdb -o data.parquet` also keeps integer columns with empty cells as integers.
 
+### JSON files
+
+DuckDB reads JSON itself, so `query --sql` takes a JSON file in `FROM` (a file of records, `[{...}, {...}]`, or JSON Lines, one object per line; `'logs/*.jsonl'` reads several). When the SQL doesn't use the `data` table, dftk reads no input. Nested objects and lists would end up as one text cell each, so flatten them in the same query: `pos.*` makes a column of each field, `len()` counts a list and `unnest()` gives one row per item:
+
+```bash
+printf '%s\n' '{"id": 1, "pos": {"chr": "1", "start": 100}, "tags": ["x", "y"]}' \
+              '{"id": 2, "pos": {"chr": "2", "start": 200}, "tags": []}' \
+  > records.jsonl
+dftk query --sql \
+  "SELECT * EXCLUDE (pos, tags), pos.*, len(tags) AS ntags FROM 'records.jsonl'"
+# id  chr  start  ntags
+# 1   1    100    2
+# 2   2    200    0
+dftk query --sql "SELECT id, unnest(tags) AS tag FROM 'records.jsonl'" -o \
+  | dftk pivot -i tag -v id -f count
+```
+
 ### Large tables: hexbin instead of points
 
 With tens of thousands of points a scatter plot is a solid blob. `--hexbin` counts points per hexagon instead (`--hexlog` for a log colour scale), and `--fit lowess` adds a smooth curve:
@@ -390,6 +407,8 @@ All commands accept:
   a previous command's bare `-o` (that temp file is then read and deleted)
 - A pipe as the filename, e.g. `<(grep -v '^#' data.tsv)` or `<(dftk … -o)`,
   read the same way as stdin
+- JSON through `query --sql "SELECT … FROM 'f.jsonl'"`, which needs no
+  other input (see [JSON files](#json-files))
 
 ```bash
 dftk dataset tips -o | dftk stat -c tip -g day        # parquet path on stdin
@@ -416,6 +435,32 @@ dftk print <(grep -v '^#' data.tsv)          # the same, as a filename
 With `--backend duckdb`, a whole-line `#` comment below the header is an
 error, and `#` lines above the header are skipped.
 
+### Files without a header
+
+`--noheader` reads the first line as data and names the columns `V1`,
+`V2`, …. It only describes the input: the output gets a header (`V1 V2 …`)
+unless you add `--removeheader`. Name the columns with `--move`:
+
+```bash
+dftk print data.txt --noheader --move V1:chr V2:begin V3:end -o named.tsv
+dftk eval data.txt --noheader -f "len = V3 - V2" --removeheader   # no header out
+```
+
+In `merge`, `--noheader` applies to both files, so both sides are named
+`V1 V2 …`. The key appears once and the right side's other columns get the
+`_r` suffix. `--move` runs after the merge, so it renames the merged names:
+
+```bash
+dftk merge -l left.txt -r right.txt -k V1 --noheader
+# V1  V2  V3  V2_r  V3_r
+dftk merge -l left.txt -r right.txt -k V1 --noheader \
+  --move V1:id V2:n V3:s V2_r:n2 V3_r:s2
+# id  n  s  n2  s2
+```
+
+Parquet always has column names, so `--removeheader` with `-o` and
+`--noheader` on parquet input do nothing (dftk warns).
+
 Standard output options (available on all tabular commands):
 
 - `-o` / `--output` — controls where output goes; takes an *optional* value:
@@ -427,6 +472,8 @@ Standard output options (available on all tabular commands):
 - `--drop col1 col2 …` — remove these columns
 - `--round N` — round numeric output
 - `--postquery EXPR` — filter output rows after processing
+- `--move OLD:NEW …` — rename columns
+- `--removeheader` — leave out the header row in TSV output
 - `--meta KEY=VALUE` — embed provenance metadata in parquet output (repeatable)
 
 ### Provenance annotations

{dataframe_toolkit-0.12.0 → dataframe_toolkit-0.14.0}/README.md

@@ -163,6 +163,23 @@ dftk dataset fertility -o \
 
 For your own export, `dftk cleannames data.tsv --backend duckdb -o data.parquet` also keeps integer columns with empty cells as integers.
 
+### JSON files
+
+DuckDB reads JSON itself, so `query --sql` takes a JSON file in `FROM` (a file of records, `[{...}, {...}]`, or JSON Lines, one object per line; `'logs/*.jsonl'` reads several). When the SQL doesn't use the `data` table, dftk reads no input. Nested objects and lists would end up as one text cell each, so flatten them in the same query: `pos.*` makes a column of each field, `len()` counts a list and `unnest()` gives one row per item:
+
+```bash
+printf '%s\n' '{"id": 1, "pos": {"chr": "1", "start": 100}, "tags": ["x", "y"]}' \
+              '{"id": 2, "pos": {"chr": "2", "start": 200}, "tags": []}' \
+  > records.jsonl
+dftk query --sql \
+  "SELECT * EXCLUDE (pos, tags), pos.*, len(tags) AS ntags FROM 'records.jsonl'"
+# id  chr  start  ntags
+# 1   1    100    2
+# 2   2    200    0
+dftk query --sql "SELECT id, unnest(tags) AS tag FROM 'records.jsonl'" -o \
+  | dftk pivot -i tag -v id -f count
+```
+
 ### Large tables: hexbin instead of points
 
 With tens of thousands of points a scatter plot is a solid blob. `--hexbin` counts points per hexagon instead (`--hexlog` for a log colour scale), and `--fit lowess` adds a smooth curve:
@@ -360,6 +377,8 @@ All commands accept:
   a previous command's bare `-o` (that temp file is then read and deleted)
 - A pipe as the filename, e.g. `<(grep -v '^#' data.tsv)` or `<(dftk … -o)`,
   read the same way as stdin
+- JSON through `query --sql "SELECT … FROM 'f.jsonl'"`, which needs no
+  other input (see [JSON files](#json-files))
 
 ```bash
 dftk dataset tips -o | dftk stat -c tip -g day        # parquet path on stdin
@@ -386,6 +405,32 @@ dftk print <(grep -v '^#' data.tsv)          # the same, as a filename
 With `--backend duckdb`, a whole-line `#` comment below the header is an
 error, and `#` lines above the header are skipped.
 
+### Files without a header
+
+`--noheader` reads the first line as data and names the columns `V1`,
+`V2`, …. It only describes the input: the output gets a header (`V1 V2 …`)
+unless you add `--removeheader`. Name the columns with `--move`:
+
+```bash
+dftk print data.txt --noheader --move V1:chr V2:begin V3:end -o named.tsv
+dftk eval data.txt --noheader -f "len = V3 - V2" --removeheader   # no header out
+```
+
+In `merge`, `--noheader` applies to both files, so both sides are named
+`V1 V2 …`. The key appears once and the right side's other columns get the
+`_r` suffix. `--move` runs after the merge, so it renames the merged names:
+
+```bash
+dftk merge -l left.txt -r right.txt -k V1 --noheader
+# V1  V2  V3  V2_r  V3_r
+dftk merge -l left.txt -r right.txt -k V1 --noheader \
+  --move V1:id V2:n V3:s V2_r:n2 V3_r:s2
+# id  n  s  n2  s2
+```
+
+Parquet always has column names, so `--removeheader` with `-o` and
+`--noheader` on parquet input do nothing (dftk warns).
+
 Standard output options (available on all tabular commands):
 
 - `-o` / `--output` — controls where output goes; takes an *optional* value:
@@ -397,6 +442,8 @@ Standard output options (available on all tabular commands):
 - `--drop col1 col2 …` — remove these columns
 - `--round N` — round numeric output
 - `--postquery EXPR` — filter output rows after processing
+- `--move OLD:NEW …` — rename columns
+- `--removeheader` — leave out the header row in TSV output
 - `--meta KEY=VALUE` — embed provenance metadata in parquet output (repeatable)
 
 ### Provenance annotations

{dataframe_toolkit-0.12.0 → dataframe_toolkit-0.14.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.12.0"
+version = "0.14.0"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 keywords = [

{dataframe_toolkit-0.12.0 → dataframe_toolkit-0.14.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.12.0"
+version = "0.14.0"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 authors = [

{dataframe_toolkit-0.12.0 → dataframe_toolkit-0.14.0}/src/dftk/commands/eval_cmd.py

@@ -44,6 +44,12 @@ methods are under .str (name.str.lower(), not name.lower() or tolower()):
 
   x = x.abs()   x = x.round(1)   x = x.clip(lower=0)   y = name.str.upper()
 
+A regular expression test gives True/False; fullmatch checks the whole value
+(match only its start). Here: exactly 7 ASCII letters, empty cells False:
+
+  dftk dataset diamonds -o \\
+      | dftk eval -f "is7 = cut.str.fullmatch('[A-Za-z]{7}')"
+
 
 FILLING MISSING VALUES (-f)
 ---------------------------

{dataframe_toolkit-0.12.0 → dataframe_toolkit-0.14.0}/src/dftk/commands/print_cmd.py

@@ -8,10 +8,13 @@ dftk pipeline:
     dftk stat -g grp -s val -o \\
       | dftk pivot -i grp -v val_mean -o \\
       | dftk print
+
+With --noheader and --move it names the columns of a headerless file:
+
+    dftk print data.txt --noheader --move V1:chr V2:begin V3:end
 """
 
 import argparse
-import sys
 
 from dftk.commands.base import BaseCommand
 from dftk.common.io import io
@@ -24,10 +27,9 @@ class PrintCommand(BaseCommand):
     help = "Convert any input (TSV or parquet pipe) to TSV on stdout"
 
     def add_arguments(self, parser: argparse.ArgumentParser) -> None:
-        # Only the read-side arguments — no -o/--output, since the whole
-        # point of this command is to force TSV to stdout.
-        io.parser_read(parser)
+        # The standard output options (--move, --select, --removeheader, …)
+        # work too; with -o FILE print writes there instead of stdout
+        self.add_io_arguments(parser)
 
     def execute(self, args: argparse.Namespace) -> None:
-        df = io.read(args)
-        df.to_csv(sys.stdout, sep="\t", index=False)
+        io.printdf(io.read(args), args)

{dataframe_toolkit-0.12.0 → dataframe_toolkit-0.14.0}/src/dftk/commands/query_cmd.py

@@ -26,6 +26,7 @@ sql mode  (--sql)
 """
 
 import argparse
+import json
 import logging
 import os
 
@@ -85,6 +86,9 @@ def _sql_query(args: argparse.Namespace, sql: str, table: str) -> pd.DataFrame:
     if direct_parquet:
         rel = conn.read_parquet(filename)
         conn.register(table, rel)
+    elif filename is None and not _uses_table(sql, table):
+        # SQL reads its own files (FROM 'f.jsonl'): don't wait for stdin.
+        pass
     else:
         # stdin, TSV files, or parquet-pipe: read into pandas first.
         df = io.read(args)
@@ -99,6 +103,37 @@ def _sql_query(args: argparse.Namespace, sql: str, table: str) -> pd.DataFrame:
     return _hugeint_to_int64(df, description)
 
 
+def _uses_table(sql: str, table: str) -> bool:
+    """Return True unless DuckDB's parser says *sql* never names *table*.
+
+    Uses json_serialize_sql, which only parses (duckdb.get_table_names also
+    binds, and fails on valid SQL such as SELECT pos.*).  On a parse error
+    the input is read as before and the query itself reports the error.
+    """
+    try:
+        tree = json.loads(
+            duckdb.execute("SELECT json_serialize_sql(?)", [sql]).fetchone()[0]
+        )
+    except Exception:  # an old duckdb without json_serialize_sql
+        return True
+    if tree.get("error"):
+        return True
+    names: set[str] = set()
+
+    def walk(node) -> None:
+        if isinstance(node, dict):
+            if node.get("type") == "BASE_TABLE":
+                names.add(node.get("table_name", "").lower())
+            for value in node.values():
+                walk(value)
+        elif isinstance(node, list):
+            for value in node:
+                walk(value)
+
+    walk(tree)
+    return table.lower() in names
+
+
 def _hugeint_to_int64(df: pd.DataFrame, description) -> pd.DataFrame:
     """Return 128-bit integer result columns as Int64 instead of float64.
 
@@ -197,6 +232,30 @@ CTEs:
      )
      SELECT * FROM ranked WHERE rank <= 3"
 
+JSON FILES
+----------
+DuckDB reads JSON itself: name the file in FROM, in single quotes. A file
+of records ([{...}, {...}]) and JSON Lines (one object per line) both
+work, and 'logs/*.jsonl' reads several files as one table. When the SQL
+doesn't use the data table, dftk reads no input, so nothing is piped in:
+
+  printf '%s\\n' '{"id": 1, "pos": {"chr": "1", "start": 100}, "tags": ["x", "y"]}' \\
+                '{"id": 2, "pos": {"chr": "2", "start": 200}, "tags": []}' \\
+    > records.jsonl
+  dftk query --sql "SELECT * FROM 'records.jsonl'"
+
+Nested objects and lists come out as one text cell each, which later
+commands can't compute on. Flatten them in the same query: pos.* makes a
+column of each field, len() counts a list, unnest() gives a row per item:
+
+  dftk query --sql \\
+    "SELECT * EXCLUDE (pos, tags), pos.*, len(tags) AS ntags
+     FROM 'records.jsonl'"
+  dftk query --sql "SELECT id, unnest(tags) AS tag FROM 'records.jsonl'"
+  dftk query --sql \\
+    "SELECT * EXCLUDE (pos, tags), pos.* FROM 'records.jsonl'" -o \\
+    | dftk stat -c start
+
 OUT-OF-CORE LARGE FILES
 ------------------------
 When the input is a named .parquet file, DuckDB queries it natively without
@@ -254,6 +313,8 @@ class QueryCommand(BaseCommand):
                 "Full DuckDB SQL query.  The input is registered as a virtual "
                 "table named 'data' (see --table).  Named .parquet inputs are "
                 "queried natively without full load into memory.  "
+                "SQL that only reads its own files (FROM 'f.jsonl') "
+                "needs no input.  "
                 "Cannot be combined with -q.  Requires duckdb."
             ),
         )

{dataframe_toolkit-0.12.0 → dataframe_toolkit-0.14.0}/src/dftk/common/io.py

@@ -91,6 +91,13 @@ def _is_temp_pipe_path(path: str) -> bool:
 _MAX_PATH_LINE = 4096
 
 
+def _warn_noheader_parquet(args: argparse.Namespace) -> None:
+    if getattr(args, "noheader", False):
+        logging.warning(
+            "--noheader has no effect on parquet input; parquet always has column names"
+        )
+
+
 def _read_pipe_parquet(path: str) -> pd.DataFrame:
     """Read the parquet file *path* named on stdin; delete it if it's dftk's temp."""
     df = pd.read_parquet(path)
@@ -305,7 +312,11 @@ class io:
         )
         g.add_argument(
             "--noheader",
-            help="File has no header row; columns are named V1, V2, …",
+            help=(
+                "Input has no header row; columns are named V1, V2, … "
+                "(rename with --move V1:NAME).  Output still gets a header "
+                "unless --removeheader."
+            ),
             action="store_true",
         )
         g.add_argument(
@@ -418,6 +429,7 @@ class io:
             if _is_pipe_path(path):
                 rest = stream.read()
                 if not rest.strip():
+                    _warn_noheader_parquet(args)
                     return _read_pipe_parquet(path)
                 stream = _io.BytesIO(rest)
             if backend == "duckdb" and duckdb is not None:
@@ -441,6 +453,7 @@ class io:
                 raise ValueError(
                     f"'...' expects a .parquet path on stdin, got {pipe_path!r}"
                 )
+            _warn_noheader_parquet(args)
             df = _read_pipe_parquet(pipe_path)
 
         elif filename is not None and filename != "-":
@@ -453,6 +466,7 @@ class io:
                     "stdin."
                 )
             if filename.endswith(".parquet"):
+                _warn_noheader_parquet(args)
                 df = pd.read_parquet(filename)
                 df.attrs["_parquet_meta"] = _read_parquet_meta(filename)
             elif os.path.exists(filename) and not os.path.isfile(filename):
@@ -573,7 +587,8 @@ class io:
         )
         g.add_argument(
             "--removeheader",
-            help="Suppress the header row in TSV output.",
+            help="Leave out the header row in TSV output (parquet always "
+            "keeps column names).",
             action="store_true",
         )
         g.add_argument(
@@ -819,10 +834,13 @@ class io:
             except ValueError:
                 logging.warning(f"--meta: expected KEY=VALUE, got {item!r}")
 
-        # Header: suppressed if the input had no header, or --removeheader set
-        header = not getattr(args, "noheader", False)
-        if header and getattr(args, "removeheader", False):
-            header = False
+        # Header: written unless --removeheader (--noheader is about the input)
+        header = not getattr(args, "removeheader", False)
+        if not header and (output == "" or str(output).endswith(".parquet")):
+            logging.warning(
+                "--removeheader has no effect on parquet output (-o); parquet "
+                "always keeps column names"
+            )
 
         if output == "":
             # Pipe mode — write temp parquet, print path

{dataframe_toolkit-0.12.0 → dataframe_toolkit-0.14.0}/src/dftk/skills/dftk/SKILL.md

@@ -75,8 +75,9 @@ left by an interrupted pipe are removed with `dftk clean`.
 ## Options shared by every data command
 
 Input: `--prequery EXPR` (filter rows right after reading), `--noheader`
-(columns become `V1, V2, …`), `--delimiter`, `--readasobject COL…` (keep
-values like IDs as strings).
+(columns become `V1, V2, …`; the output still gets a header unless
+`--removeheader`, and `--move V1:chr …` names them), `--delimiter`,
+`--readasobject COL…` (keep values like IDs as strings).
 
 Output, applied after the command's own work: `--select COL… [+]` (choose
 and order columns; globs allowed; a trailing `+` keeps the remaining
@@ -103,6 +104,7 @@ Transform:
 | New column from a formula: `z = x / y`, `where(...)`, string, DNA and statistical functions | `eval -f`, `eval -c` (constant), `eval -s` (string) |
 | Fill empty cells: from another column, a group mean, the row above (`ffill`, per group with `groupby`) or by interpolation; sort first, fills follow row order | `eval -f "x = x.fillna(y)"`, `"x = x.groupby(g).ffill()"` (see `eval -h`) |
 | Filter rows (pandas expression) or run any DuckDB SQL (`FROM data`) | `query -q` / `query --sql` |
+| Read JSON or JSON Lines (no input needed); flatten nested fields with `pos.*`, lists with `unnest()` | `query --sql "SELECT * EXCLUDE (pos), pos.* FROM 'f.jsonl'"` |
 | Join two tables, including anti-joins | `merge DATAFILE -r OTHER -k KEY [-t left] [--only left]` |
 | Stack tables row-wise | `concat A B C [--sourcecol src]` |
 | Wide to long / long to wide plus aggregation | `melt` / `pivot -i ROWS -g COLS -v VALS -f FUNCS` |