dataframe-toolkit 0.13.0__tar.gz → 0.14.0__tar.gz

{dataframe_toolkit-0.13.0 → dataframe_toolkit-0.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: dataframe-toolkit
-Version: 0.13.0
+Version: 0.14.0
 Summary: A command-line toolkit for DataFrame analysis and manipulation
 Keywords: dataframe,statistics,data analysis,CLI,TSV,pandas
 Author: Gunnar Pálsson
@@ -193,6 +193,23 @@ dftk dataset fertility -o \
 
 For your own export, `dftk cleannames data.tsv --backend duckdb -o data.parquet` also keeps integer columns with empty cells as integers.
 
+### JSON files
+
+DuckDB reads JSON itself, so `query --sql` takes a JSON file in `FROM` (a file of records, `[{...}, {...}]`, or JSON Lines, one object per line; `'logs/*.jsonl'` reads several). When the SQL doesn't use the `data` table, dftk reads no input. Nested objects and lists would end up as one text cell each, so flatten them in the same query: `pos.*` makes a column of each field, `len()` counts a list and `unnest()` gives one row per item:
+
+```bash
+printf '%s\n' '{"id": 1, "pos": {"chr": "1", "start": 100}, "tags": ["x", "y"]}' \
+              '{"id": 2, "pos": {"chr": "2", "start": 200}, "tags": []}' \
+  > records.jsonl
+dftk query --sql \
+  "SELECT * EXCLUDE (pos, tags), pos.*, len(tags) AS ntags FROM 'records.jsonl'"
+# id  chr  start  ntags
+# 1   1    100    2
+# 2   2    200    0
+dftk query --sql "SELECT id, unnest(tags) AS tag FROM 'records.jsonl'" -o \
+  | dftk pivot -i tag -v id -f count
+```
+
 ### Large tables: hexbin instead of points
 
 With tens of thousands of points a scatter plot is a solid blob. `--hexbin` counts points per hexagon instead (`--hexlog` for a log colour scale), and `--fit lowess` adds a smooth curve:
@@ -390,6 +407,8 @@ All commands accept:
   a previous command's bare `-o` (that temp file is then read and deleted)
 - A pipe as the filename, e.g. `<(grep -v '^#' data.tsv)` or `<(dftk … -o)`,
   read the same way as stdin
+- JSON through `query --sql "SELECT … FROM 'f.jsonl'"`, which needs no
+  other input (see [JSON files](#json-files))
 
 ```bash
 dftk dataset tips -o | dftk stat -c tip -g day        # parquet path on stdin
@@ -427,6 +446,18 @@ dftk print data.txt --noheader --move V1:chr V2:begin V3:end -o named.tsv
 dftk eval data.txt --noheader -f "len = V3 - V2" --removeheader   # no header out
 ```
 
+In `merge`, `--noheader` applies to both files, so both sides are named
+`V1 V2 …`. The key appears once and the right side's other columns get the
+`_r` suffix. `--move` runs after the merge, so it renames the merged names:
+
+```bash
+dftk merge -l left.txt -r right.txt -k V1 --noheader
+# V1  V2  V3  V2_r  V3_r
+dftk merge -l left.txt -r right.txt -k V1 --noheader \
+  --move V1:id V2:n V3:s V2_r:n2 V3_r:s2
+# id  n  s  n2  s2
+```
+
 Parquet always has column names, so `--removeheader` with `-o` and
 `--noheader` on parquet input do nothing (dftk warns).
 

{dataframe_toolkit-0.13.0 → dataframe_toolkit-0.14.0}/README.md

@@ -163,6 +163,23 @@ dftk dataset fertility -o \
 
 For your own export, `dftk cleannames data.tsv --backend duckdb -o data.parquet` also keeps integer columns with empty cells as integers.
 
+### JSON files
+
+DuckDB reads JSON itself, so `query --sql` takes a JSON file in `FROM` (a file of records, `[{...}, {...}]`, or JSON Lines, one object per line; `'logs/*.jsonl'` reads several). When the SQL doesn't use the `data` table, dftk reads no input. Nested objects and lists would end up as one text cell each, so flatten them in the same query: `pos.*` makes a column of each field, `len()` counts a list and `unnest()` gives one row per item:
+
+```bash
+printf '%s\n' '{"id": 1, "pos": {"chr": "1", "start": 100}, "tags": ["x", "y"]}' \
+              '{"id": 2, "pos": {"chr": "2", "start": 200}, "tags": []}' \
+  > records.jsonl
+dftk query --sql \
+  "SELECT * EXCLUDE (pos, tags), pos.*, len(tags) AS ntags FROM 'records.jsonl'"
+# id  chr  start  ntags
+# 1   1    100    2
+# 2   2    200    0
+dftk query --sql "SELECT id, unnest(tags) AS tag FROM 'records.jsonl'" -o \
+  | dftk pivot -i tag -v id -f count
+```
+
 ### Large tables: hexbin instead of points
 
 With tens of thousands of points a scatter plot is a solid blob. `--hexbin` counts points per hexagon instead (`--hexlog` for a log colour scale), and `--fit lowess` adds a smooth curve:
@@ -360,6 +377,8 @@ All commands accept:
   a previous command's bare `-o` (that temp file is then read and deleted)
 - A pipe as the filename, e.g. `<(grep -v '^#' data.tsv)` or `<(dftk … -o)`,
   read the same way as stdin
+- JSON through `query --sql "SELECT … FROM 'f.jsonl'"`, which needs no
+  other input (see [JSON files](#json-files))
 
 ```bash
 dftk dataset tips -o | dftk stat -c tip -g day        # parquet path on stdin
@@ -397,6 +416,18 @@ dftk print data.txt --noheader --move V1:chr V2:begin V3:end -o named.tsv
 dftk eval data.txt --noheader -f "len = V3 - V2" --removeheader   # no header out
 ```
 
+In `merge`, `--noheader` applies to both files, so both sides are named
+`V1 V2 …`. The key appears once and the right side's other columns get the
+`_r` suffix. `--move` runs after the merge, so it renames the merged names:
+
+```bash
+dftk merge -l left.txt -r right.txt -k V1 --noheader
+# V1  V2  V3  V2_r  V3_r
+dftk merge -l left.txt -r right.txt -k V1 --noheader \
+  --move V1:id V2:n V3:s V2_r:n2 V3_r:s2
+# id  n  s  n2  s2
+```
+
 Parquet always has column names, so `--removeheader` with `-o` and
 `--noheader` on parquet input do nothing (dftk warns).
 

{dataframe_toolkit-0.13.0 → dataframe_toolkit-0.14.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.13.0"
+version = "0.14.0"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 keywords = [

{dataframe_toolkit-0.13.0 → dataframe_toolkit-0.14.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.13.0"
+version = "0.14.0"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 authors = [

{dataframe_toolkit-0.13.0 → dataframe_toolkit-0.14.0}/src/dftk/commands/query_cmd.py

@@ -26,6 +26,7 @@ sql mode  (--sql)
 """
 
 import argparse
+import json
 import logging
 import os
 
@@ -85,6 +86,9 @@ def _sql_query(args: argparse.Namespace, sql: str, table: str) -> pd.DataFrame:
     if direct_parquet:
         rel = conn.read_parquet(filename)
         conn.register(table, rel)
+    elif filename is None and not _uses_table(sql, table):
+        # SQL reads its own files (FROM 'f.jsonl'): don't wait for stdin.
+        pass
     else:
         # stdin, TSV files, or parquet-pipe: read into pandas first.
         df = io.read(args)
@@ -99,6 +103,37 @@ def _sql_query(args: argparse.Namespace, sql: str, table: str) -> pd.DataFrame:
     return _hugeint_to_int64(df, description)
 
 
+def _uses_table(sql: str, table: str) -> bool:
+    """Return True unless DuckDB's parser says *sql* never names *table*.
+
+    Uses json_serialize_sql, which only parses (duckdb.get_table_names also
+    binds, and fails on valid SQL such as SELECT pos.*).  On a parse error
+    the input is read as before and the query itself reports the error.
+    """
+    try:
+        tree = json.loads(
+            duckdb.execute("SELECT json_serialize_sql(?)", [sql]).fetchone()[0]
+        )
+    except Exception:  # an old duckdb without json_serialize_sql
+        return True
+    if tree.get("error"):
+        return True
+    names: set[str] = set()
+
+    def walk(node) -> None:
+        if isinstance(node, dict):
+            if node.get("type") == "BASE_TABLE":
+                names.add(node.get("table_name", "").lower())
+            for value in node.values():
+                walk(value)
+        elif isinstance(node, list):
+            for value in node:
+                walk(value)
+
+    walk(tree)
+    return table.lower() in names
+
+
 def _hugeint_to_int64(df: pd.DataFrame, description) -> pd.DataFrame:
     """Return 128-bit integer result columns as Int64 instead of float64.
 
@@ -197,6 +232,30 @@ CTEs:
      )
      SELECT * FROM ranked WHERE rank <= 3"
 
+JSON FILES
+----------
+DuckDB reads JSON itself: name the file in FROM, in single quotes. A file
+of records ([{...}, {...}]) and JSON Lines (one object per line) both
+work, and 'logs/*.jsonl' reads several files as one table. When the SQL
+doesn't use the data table, dftk reads no input, so nothing is piped in:
+
+  printf '%s\\n' '{"id": 1, "pos": {"chr": "1", "start": 100}, "tags": ["x", "y"]}' \\
+                '{"id": 2, "pos": {"chr": "2", "start": 200}, "tags": []}' \\
+    > records.jsonl
+  dftk query --sql "SELECT * FROM 'records.jsonl'"
+
+Nested objects and lists come out as one text cell each, which later
+commands can't compute on. Flatten them in the same query: pos.* makes a
+column of each field, len() counts a list, unnest() gives a row per item:
+
+  dftk query --sql \\
+    "SELECT * EXCLUDE (pos, tags), pos.*, len(tags) AS ntags
+     FROM 'records.jsonl'"
+  dftk query --sql "SELECT id, unnest(tags) AS tag FROM 'records.jsonl'"
+  dftk query --sql \\
+    "SELECT * EXCLUDE (pos, tags), pos.* FROM 'records.jsonl'" -o \\
+    | dftk stat -c start
+
 OUT-OF-CORE LARGE FILES
 ------------------------
 When the input is a named .parquet file, DuckDB queries it natively without
@@ -254,6 +313,8 @@ class QueryCommand(BaseCommand):
                 "Full DuckDB SQL query.  The input is registered as a virtual "
                 "table named 'data' (see --table).  Named .parquet inputs are "
                 "queried natively without full load into memory.  "
+                "SQL that only reads its own files (FROM 'f.jsonl') "
+                "needs no input.  "
                 "Cannot be combined with -q.  Requires duckdb."
             ),
         )

{dataframe_toolkit-0.13.0 → dataframe_toolkit-0.14.0}/src/dftk/skills/dftk/SKILL.md

@@ -104,6 +104,7 @@ Transform:
 | New column from a formula: `z = x / y`, `where(...)`, string, DNA and statistical functions | `eval -f`, `eval -c` (constant), `eval -s` (string) |
 | Fill empty cells: from another column, a group mean, the row above (`ffill`, per group with `groupby`) or by interpolation; sort first, fills follow row order | `eval -f "x = x.fillna(y)"`, `"x = x.groupby(g).ffill()"` (see `eval -h`) |
 | Filter rows (pandas expression) or run any DuckDB SQL (`FROM data`) | `query -q` / `query --sql` |
+| Read JSON or JSON Lines (no input needed); flatten nested fields with `pos.*`, lists with `unnest()` | `query --sql "SELECT * EXCLUDE (pos), pos.* FROM 'f.jsonl'"` |
 | Join two tables, including anti-joins | `merge DATAFILE -r OTHER -k KEY [-t left] [--only left]` |
 | Stack tables row-wise | `concat A B C [--sourcecol src]` |
 | Wide to long / long to wide plus aggregation | `melt` / `pivot -i ROWS -g COLS -v VALS -f FUNCS` |