dataframe-toolkit 0.13.0__tar.gz → 0.15.0__tar.gz

{dataframe_toolkit-0.13.0 → dataframe_toolkit-0.15.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: dataframe-toolkit
-Version: 0.13.0
+Version: 0.15.0
 Summary: A command-line toolkit for DataFrame analysis and manipulation
 Keywords: dataframe,statistics,data analysis,CLI,TSV,pandas
 Author: Gunnar Pálsson
@@ -166,7 +166,7 @@ publication-quality output.
 | `sample`   | Random row sampling (with or without replacement, by count or fraction)   |
 | `split`    | Split a dataframe into one file per group                                 |
 | `annotate` | Read and write provenance metadata (genome, source, …) in parquet files  |
-| `print`    | Read any dftk input (TSV, parquet, stdin) and write TSV                |
+| `print`    | Read any dftk input (TSV, parquet, stdin) and write TSV, JSON, Markdown or LaTeX |
 | `clean`    | Remove leftover temp parquet pipe files from interrupted pipelines       |
 | `help`     | List all subcommands or show full help for one: `dftk help stat`       |
 
@@ -193,6 +193,49 @@ dftk dataset fertility -o \
 
 For your own export, `dftk cleannames data.tsv --backend duckdb -o data.parquet` also keeps integer columns with empty cells as integers.
 
+### JSON files
+
+DuckDB reads JSON itself, so `query --sql` takes a JSON file in `FROM` (a file of records, `[{...}, {...}]`, or JSON Lines, one object per line; `'logs/*.jsonl'` reads several). When the SQL doesn't use the `data` table, dftk reads no input. In TSV a nested object or list is shown as one line of JSON (`["x","y"]`), which later commands can't compute on, so flatten it in the same query: `pos.*` makes a column of each field, `len()` counts a list and `unnest()` gives one row per item:
+
+```bash
+printf '%s\n' '{"id": 1, "pos": {"chr": "1", "start": 100}, "tags": ["x", "y"]}' \
+              '{"id": 2, "pos": {"chr": "2", "start": 200}, "tags": []}' \
+  > records.jsonl
+dftk query --sql \
+  "SELECT * EXCLUDE (pos, tags), pos.*, len(tags) AS ntags FROM 'records.jsonl'"
+# id  chr  start  ntags
+# 1   1    100    2
+# 2   2    200    0
+dftk query --sql "SELECT id, unnest(tags) AS tag FROM 'records.jsonl'" -o \
+  | dftk pivot -i tag -v id -f count
+```
+
+A field that is a string in some records and a list in others (a variant catalog's `ReferenceRegion`, say) gets DuckDB type JSON; TSV shows its strings without quotes and its lists as JSON. `json_type()` tells them apart, so a list can become one row per item:
+
+```bash
+printf '%s\n' '{"locus": "AFF2", "region": "chrX:1-60"}' \
+              '{"locus": "ATXN7", "region": ["chr3:1-30", "chr3:30-42"]}' \
+  > catalog.jsonl
+dftk query --sql "
+  SELECT locus,
+         unnest(CASE WHEN json_type(region) = 'ARRAY' THEN region->>'\$[*]'
+                     ELSE [region->>'\$'] END) AS region
+  FROM 'catalog.jsonl'"
+# locus  region
+# AFF2   chrX:1-60
+# ATXN7  chr3:1-30
+# ATXN7  chr3:30-42
+```
+
+To get JSON back, write `-o FILE.json` or `.jsonl`, or `--format json` for stdout (see [Output formats](#output-formats)). Nested values and mixed fields are written as JSON again; a key missing from a record is written as `null`:
+
+```bash
+dftk query --sql "SELECT * FROM 'catalog.jsonl' WHERE locus LIKE 'ATXN%'" --format json
+# [
+#   {"locus": "ATXN7", "region": ["chr3:1-30", "chr3:30-42"]}
+# ]
+```
+
 ### Large tables: hexbin instead of points
 
 With tens of thousands of points a scatter plot is a solid blob. `--hexbin` counts points per hexagon instead (`--hexlog` for a log colour scale), and `--fit lowess` adds a smooth curve:
@@ -390,6 +433,8 @@ All commands accept:
   a previous command's bare `-o` (that temp file is then read and deleted)
 - A pipe as the filename, e.g. `<(grep -v '^#' data.tsv)` or `<(dftk … -o)`,
   read the same way as stdin
+- JSON through `query --sql "SELECT … FROM 'f.jsonl'"`, which needs no
+  other input (see [JSON files](#json-files))
 
 ```bash
 dftk dataset tips -o | dftk stat -c tip -g day        # parquet path on stdin
@@ -401,6 +446,28 @@ Older scripts give `...` as the file to read a parquet path from stdin; that
 still works. In zsh with Oh My Zsh an unquoted `...` becomes `../..`, so
 leave it out (or use `-`) in scripts you share.
 
+### Output formats
+
+Without `-o`, output is TSV on stdout. `-o` alone, or `-o FILE.parquet`, writes parquet and prints the path for the next command; `-o FILE` writes TSV, except for these extensions:
+
+| `-o` | Writes |
+|------|--------|
+| `FILE.json` | JSON array of records |
+| `FILE.jsonl` | JSON Lines, one record per line |
+| `FILE.md` | Markdown table (numeric columns right-aligned) |
+| `FILE.tex` | bare LaTeX `tabular` with booktabs rules |
+
+`--format tsv|json|jsonl|md|tex` writes the same formats to stdout, or overrides the extension. `--digits N` applies to all of them. The `.tex` file is only the tabular, with special characters (`_ & % $ # { } ~ ^ \ | < >`) escaped; it needs `\usepackage{booktabs}`, and goes inside your own `table` environment with `\input{tips_by_day.tex}`. Add `\usepackage[T1]{fontenc}` (pdfLaTeX's default font has no straight `"`, which JSON lists contain):
+
+```bash
+dftk dataset tips -o | dftk pivot -i day -v tip -f mean count --digits 3 --format md
+# | day | mean_tip | count_tip |
+# | --- | ---: | ---: |
+# | Thur | 2.77 | 62 |
+# ...
+dftk dataset tips -o | dftk pivot -i day -v tip -f mean count --digits 3 -o tips_by_day.tex
+```
+
 ### Comment lines and preambles
 
 dftk has no comment or skip-rows option: `#` is ordinary data, so headers
@@ -427,6 +494,18 @@ dftk print data.txt --noheader --move V1:chr V2:begin V3:end -o named.tsv
 dftk eval data.txt --noheader -f "len = V3 - V2" --removeheader   # no header out
 ```
 
+In `merge`, `--noheader` applies to both files, so both sides are named
+`V1 V2 …`. The key appears once and the right side's other columns get the
+`_r` suffix. `--move` runs after the merge, so it renames the merged names:
+
+```bash
+dftk merge -l left.txt -r right.txt -k V1 --noheader
+# V1  V2  V3  V2_r  V3_r
+dftk merge -l left.txt -r right.txt -k V1 --noheader \
+  --move V1:id V2:n V3:s V2_r:n2 V3_r:s2
+# id  n  s  n2  s2
+```
+
 Parquet always has column names, so `--removeheader` with `-o` and
 `--noheader` on parquet input do nothing (dftk warns).
 

{dataframe_toolkit-0.13.0 → dataframe_toolkit-0.15.0}/README.md

@@ -136,7 +136,7 @@ publication-quality output.
 | `sample`   | Random row sampling (with or without replacement, by count or fraction)   |
 | `split`    | Split a dataframe into one file per group                                 |
 | `annotate` | Read and write provenance metadata (genome, source, …) in parquet files  |
-| `print`    | Read any dftk input (TSV, parquet, stdin) and write TSV                |
+| `print`    | Read any dftk input (TSV, parquet, stdin) and write TSV, JSON, Markdown or LaTeX |
 | `clean`    | Remove leftover temp parquet pipe files from interrupted pipelines       |
 | `help`     | List all subcommands or show full help for one: `dftk help stat`       |
 
@@ -163,6 +163,49 @@ dftk dataset fertility -o \
 
 For your own export, `dftk cleannames data.tsv --backend duckdb -o data.parquet` also keeps integer columns with empty cells as integers.
 
+### JSON files
+
+DuckDB reads JSON itself, so `query --sql` takes a JSON file in `FROM` (a file of records, `[{...}, {...}]`, or JSON Lines, one object per line; `'logs/*.jsonl'` reads several). When the SQL doesn't use the `data` table, dftk reads no input. In TSV a nested object or list is shown as one line of JSON (`["x","y"]`), which later commands can't compute on, so flatten it in the same query: `pos.*` makes a column of each field, `len()` counts a list and `unnest()` gives one row per item:
+
+```bash
+printf '%s\n' '{"id": 1, "pos": {"chr": "1", "start": 100}, "tags": ["x", "y"]}' \
+              '{"id": 2, "pos": {"chr": "2", "start": 200}, "tags": []}' \
+  > records.jsonl
+dftk query --sql \
+  "SELECT * EXCLUDE (pos, tags), pos.*, len(tags) AS ntags FROM 'records.jsonl'"
+# id  chr  start  ntags
+# 1   1    100    2
+# 2   2    200    0
+dftk query --sql "SELECT id, unnest(tags) AS tag FROM 'records.jsonl'" -o \
+  | dftk pivot -i tag -v id -f count
+```
+
+A field that is a string in some records and a list in others (a variant catalog's `ReferenceRegion`, say) gets DuckDB type JSON; TSV shows its strings without quotes and its lists as JSON. `json_type()` tells them apart, so a list can become one row per item:
+
+```bash
+printf '%s\n' '{"locus": "AFF2", "region": "chrX:1-60"}' \
+              '{"locus": "ATXN7", "region": ["chr3:1-30", "chr3:30-42"]}' \
+  > catalog.jsonl
+dftk query --sql "
+  SELECT locus,
+         unnest(CASE WHEN json_type(region) = 'ARRAY' THEN region->>'\$[*]'
+                     ELSE [region->>'\$'] END) AS region
+  FROM 'catalog.jsonl'"
+# locus  region
+# AFF2   chrX:1-60
+# ATXN7  chr3:1-30
+# ATXN7  chr3:30-42
+```
+
+To get JSON back, write `-o FILE.json` or `.jsonl`, or `--format json` for stdout (see [Output formats](#output-formats)). Nested values and mixed fields are written as JSON again; a key missing from a record is written as `null`:
+
+```bash
+dftk query --sql "SELECT * FROM 'catalog.jsonl' WHERE locus LIKE 'ATXN%'" --format json
+# [
+#   {"locus": "ATXN7", "region": ["chr3:1-30", "chr3:30-42"]}
+# ]
+```
+
 ### Large tables: hexbin instead of points
 
 With tens of thousands of points a scatter plot is a solid blob. `--hexbin` counts points per hexagon instead (`--hexlog` for a log colour scale), and `--fit lowess` adds a smooth curve:
@@ -360,6 +403,8 @@ All commands accept:
   a previous command's bare `-o` (that temp file is then read and deleted)
 - A pipe as the filename, e.g. `<(grep -v '^#' data.tsv)` or `<(dftk … -o)`,
   read the same way as stdin
+- JSON through `query --sql "SELECT … FROM 'f.jsonl'"`, which needs no
+  other input (see [JSON files](#json-files))
 
 ```bash
 dftk dataset tips -o | dftk stat -c tip -g day        # parquet path on stdin
@@ -371,6 +416,28 @@ Older scripts give `...` as the file to read a parquet path from stdin; that
 still works. In zsh with Oh My Zsh an unquoted `...` becomes `../..`, so
 leave it out (or use `-`) in scripts you share.
 
+### Output formats
+
+Without `-o`, output is TSV on stdout. `-o` alone, or `-o FILE.parquet`, writes parquet and prints the path for the next command; `-o FILE` writes TSV, except for these extensions:
+
+| `-o` | Writes |
+|------|--------|
+| `FILE.json` | JSON array of records |
+| `FILE.jsonl` | JSON Lines, one record per line |
+| `FILE.md` | Markdown table (numeric columns right-aligned) |
+| `FILE.tex` | bare LaTeX `tabular` with booktabs rules |
+
+`--format tsv|json|jsonl|md|tex` writes the same formats to stdout, or overrides the extension. `--digits N` applies to all of them. The `.tex` file is only the tabular, with special characters (`_ & % $ # { } ~ ^ \ | < >`) escaped; it needs `\usepackage{booktabs}`, and goes inside your own `table` environment with `\input{tips_by_day.tex}`. Add `\usepackage[T1]{fontenc}` (pdfLaTeX's default font has no straight `"`, which JSON lists contain):
+
+```bash
+dftk dataset tips -o | dftk pivot -i day -v tip -f mean count --digits 3 --format md
+# | day | mean_tip | count_tip |
+# | --- | ---: | ---: |
+# | Thur | 2.77 | 62 |
+# ...
+dftk dataset tips -o | dftk pivot -i day -v tip -f mean count --digits 3 -o tips_by_day.tex
+```
+
 ### Comment lines and preambles
 
 dftk has no comment or skip-rows option: `#` is ordinary data, so headers
@@ -397,6 +464,18 @@ dftk print data.txt --noheader --move V1:chr V2:begin V3:end -o named.tsv
 dftk eval data.txt --noheader -f "len = V3 - V2" --removeheader   # no header out
 ```
 
+In `merge`, `--noheader` applies to both files, so both sides are named
+`V1 V2 …`. The key appears once and the right side's other columns get the
+`_r` suffix. `--move` runs after the merge, so it renames the merged names:
+
+```bash
+dftk merge -l left.txt -r right.txt -k V1 --noheader
+# V1  V2  V3  V2_r  V3_r
+dftk merge -l left.txt -r right.txt -k V1 --noheader \
+  --move V1:id V2:n V3:s V2_r:n2 V3_r:s2
+# id  n  s  n2  s2
+```
+
 Parquet always has column names, so `--removeheader` with `-o` and
 `--noheader` on parquet input do nothing (dftk warns).
 

{dataframe_toolkit-0.13.0 → dataframe_toolkit-0.15.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.13.0"
+version = "0.15.0"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 keywords = [

{dataframe_toolkit-0.13.0 → dataframe_toolkit-0.15.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.13.0"
+version = "0.15.0"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 authors = [

{dataframe_toolkit-0.13.0 → dataframe_toolkit-0.15.0}/src/dftk/commands/print_cmd.py

@@ -2,7 +2,8 @@
 dftk.commands.print_cmd — the ``dftk print`` subcommand.
 
 Reads any dftk-compatible input (TSV file, TSV on stdin, or a parquet pipe
-path on stdin) and writes TSV to stdout.  Acts as the terminal stage of a
+path on stdin) and writes TSV to stdout, or JSON, a Markdown table or a
+LaTeX tabular with --format json|jsonl|md|tex.  Acts as the terminal stage of a
 dftk pipeline:
 
     dftk stat -g grp -s val -o \\
@@ -12,6 +13,11 @@ dftk pipeline:
 With --noheader and --move it names the columns of a headerless file:
 
     dftk print data.txt --noheader --move V1:chr V2:begin V3:end
+
+A table for a document:
+
+    dftk dataset tips -o | dftk pivot -i day -v tip -f mean -o \\
+      | dftk print --digits 3 --format md
 """
 
 import argparse
@@ -24,7 +30,7 @@ class PrintCommand(BaseCommand):
     """Read any dftk input and write TSV to stdout."""
 
     name = "print"
-    help = "Convert any input (TSV or parquet pipe) to TSV on stdout"
+    help = "Write any input (TSV or parquet pipe) as TSV, JSON, Markdown or LaTeX"
 
     def add_arguments(self, parser: argparse.ArgumentParser) -> None:
         # The standard output options (--move, --select, --removeheader, …)

{dataframe_toolkit-0.13.0 → dataframe_toolkit-0.15.0}/src/dftk/commands/query_cmd.py

@@ -26,12 +26,14 @@ sql mode  (--sql)
 """
 
 import argparse
+import json
 import logging
 import os
 
 import pandas as pd
 
 from dftk.commands.base import BaseCommand
+from dftk.common import docformats
 from dftk.common.io import io
 
 try:
@@ -85,6 +87,9 @@ def _sql_query(args: argparse.Namespace, sql: str, table: str) -> pd.DataFrame:
     if direct_parquet:
         rel = conn.read_parquet(filename)
         conn.register(table, rel)
+    elif filename is None and not _uses_table(sql, table):
+        # SQL reads its own files (FROM 'f.jsonl'): don't wait for stdin.
+        pass
     else:
         # stdin, TSV files, or parquet-pipe: read into pandas first.
         df = io.read(args)
@@ -96,7 +101,42 @@ def _sql_query(args: argparse.Namespace, sql: str, table: str) -> pd.DataFrame:
         df = result.df()
     except Exception as exc:
         raise ValueError(f"DuckDB SQL failed: {exc}") from exc
-    return _hugeint_to_int64(df, description)
+    df = _hugeint_to_int64(df, description)
+    docformats.mark_json_columns(
+        df, [d[0] for d in description or [] if str(d[1]) == "JSON"]
+    )
+    return df
+
+
+def _uses_table(sql: str, table: str) -> bool:
+    """Return True unless DuckDB's parser says *sql* never names *table*.
+
+    Uses json_serialize_sql, which only parses (duckdb.get_table_names also
+    binds, and fails on valid SQL such as SELECT pos.*).  On a parse error
+    the input is read as before and the query itself reports the error.
+    """
+    try:
+        tree = json.loads(
+            duckdb.execute("SELECT json_serialize_sql(?)", [sql]).fetchone()[0]
+        )
+    except Exception:  # an old duckdb without json_serialize_sql
+        return True
+    if tree.get("error"):
+        return True
+    names: set[str] = set()
+
+    def walk(node) -> None:
+        if isinstance(node, dict):
+            if node.get("type") == "BASE_TABLE":
+                names.add(node.get("table_name", "").lower())
+            for value in node.values():
+                walk(value)
+        elif isinstance(node, list):
+            for value in node:
+                walk(value)
+
+    walk(tree)
+    return table.lower() in names
 
 
 def _hugeint_to_int64(df: pd.DataFrame, description) -> pd.DataFrame:
@@ -197,6 +237,41 @@ CTEs:
      )
      SELECT * FROM ranked WHERE rank <= 3"
 
+JSON FILES
+----------
+DuckDB reads JSON itself: name the file in FROM, in single quotes. A file
+of records ([{...}, {...}]) and JSON Lines (one object per line) both
+work, and 'logs/*.jsonl' reads several files as one table. When the SQL
+doesn't use the data table, dftk reads no input, so nothing is piped in:
+
+  printf '%s\\n' '{"id": 1, "pos": {"chr": "1", "start": 100}, "tags": ["x", "y"]}' \\
+                '{"id": 2, "pos": {"chr": "2", "start": 200}, "tags": []}' \\
+    > records.jsonl
+  dftk query --sql "SELECT * FROM 'records.jsonl'"
+
+Nested objects and lists come out as one text cell each, which later
+commands can't compute on. Flatten them in the same query: pos.* makes a
+column of each field, len() counts a list, unnest() gives a row per item:
+
+  dftk query --sql \\
+    "SELECT * EXCLUDE (pos, tags), pos.*, len(tags) AS ntags
+     FROM 'records.jsonl'"
+  dftk query --sql "SELECT id, unnest(tags) AS tag FROM 'records.jsonl'"
+  dftk query --sql \\
+    "SELECT * EXCLUDE (pos, tags), pos.* FROM 'records.jsonl'" -o \\
+    | dftk stat -c start
+
+In TSV output a list or object that is left nested is shown as one line
+of JSON (["x","y"]). A field that is a string in some records and a list
+in others gets DuckDB type JSON; its plain strings are shown without
+quotes. To get JSON back, give -o FILE.json (or .jsonl) or --format json;
+nested values, and such mixed fields, are written as JSON again, and a key
+missing from a record is written as null:
+
+  dftk query --sql "SELECT * FROM 'records.jsonl' WHERE pos.chr = '1'" \\
+    --format json
+  dftk query --sql "SELECT id, tags FROM 'records.jsonl'" -o ids.jsonl
+
 OUT-OF-CORE LARGE FILES
 ------------------------
 When the input is a named .parquet file, DuckDB queries it natively without
@@ -254,6 +329,8 @@ class QueryCommand(BaseCommand):
                 "Full DuckDB SQL query.  The input is registered as a virtual "
                 "table named 'data' (see --table).  Named .parquet inputs are "
                 "queried natively without full load into memory.  "
+                "SQL that only reads its own files (FROM 'f.jsonl') "
+                "needs no input.  "
                 "Cannot be combined with -q.  Requires duckdb."
             ),
         )

dataframe_toolkit-0.15.0/src/dftk/common/docformats.py

@@ -0,0 +1,246 @@
+"""
+dftk.common.docformats — JSON, Markdown and LaTeX output for io.printdf.
+
+Nested values (lists, structs, maps) come from DuckDB as numpy arrays and
+dicts.  TSV, Markdown and LaTeX show them as one line of JSON text; JSON
+output writes them as JSON.
+
+A DuckDB column of type JSON (from a JSON file whose field is a string in one
+record and a list in another) reaches pandas as JSON text, e.g.
+'"chrX:1-2"' and '["chr3:1-2","chr3:2-3"]'.  query_cmd lists such columns in
+the parquet metadata key JSON_COLUMNS_KEY, so they survive -o pipes; the text
+view drops the quotes around a plain string and JSON output parses them.
+"""
+
+import contextlib
+import decimal
+import json
+import math
+
+import numpy as np
+import pandas as pd
+
+JSON_COLUMNS_KEY = "dftk_json_columns"
+
+# Output format by file extension; --format names the same formats
+EXTENSIONS = {".json": "json", ".jsonl": "jsonl", ".md": "md", ".tex": "tex"}
+FORMATS = ("tsv", "json", "jsonl", "md", "tex")
+
+
+def json_columns(df: pd.DataFrame) -> list[str]:
+    """Columns of *df* that hold DuckDB JSON text (see module docstring)."""
+    names = df.attrs.get("_parquet_meta", {}).get(JSON_COLUMNS_KEY, "")
+    return [c for c in names.split(",") if c in df.columns]
+
+
+def mark_json_columns(df: pd.DataFrame, columns: list[str]) -> None:
+    """Record *columns* as JSON text in df.attrs (carried into parquet)."""
+    meta = dict(df.attrs.get("_parquet_meta", {}))
+    if columns:
+        meta[JSON_COLUMNS_KEY] = ",".join(columns)
+    else:
+        meta.pop(JSON_COLUMNS_KEY, None)
+    df.attrs["_parquet_meta"] = meta
+
+
+def rename_json_columns(df: pd.DataFrame, mapping: dict[str, str]) -> None:
+    """Follow a --move rename in the JSON column list."""
+    names = df.attrs.get("_parquet_meta", {}).get(JSON_COLUMNS_KEY)
+    if names:
+        mark_json_columns(df, [mapping.get(c, c) for c in names.split(",")])
+
+
+def _plain(value):
+    """json.dumps fallback for numpy, Decimal and other non-JSON values."""
+    if isinstance(value, np.ndarray):  # MaskedArray.tolist() gives None
+        return value.tolist()
+    if isinstance(value, np.generic):
+        return value.item()
+    if isinstance(value, decimal.Decimal):
+        return float(value)
+    return str(value)
+
+
+def _is_nested(value) -> bool:
+    return isinstance(value, (np.ndarray, list, tuple, dict))
+
+
+def _dumps(value) -> str:
+    return json.dumps(value, default=_plain, ensure_ascii=False, separators=(",", ":"))
+
+
+def nested_as_text(df: pd.DataFrame) -> pd.DataFrame:
+    """Return *df* with nested values and JSON-column text as one-line text."""
+    jcols = set(json_columns(df))
+    out = None
+    for col in df.columns:
+        dtype = df[col].dtype
+        if not (
+            pd.api.types.is_object_dtype(dtype) or pd.api.types.is_string_dtype(dtype)
+        ):
+            continue
+        if col in jcols:
+            new = df[col].map(_json_text_cell)
+        elif df[col].map(_is_nested).any():
+            new = df[col].map(lambda v: _dumps(v) if _is_nested(v) else v)
+        else:
+            continue
+        if out is None:
+            out = df.copy()
+        out[col] = new
+    return df if out is None else out
+
+
+def _json_text_cell(value):
+    """'"chrX:1-2"' → 'chrX:1-2'; a list or object stays JSON text."""
+    if not isinstance(value, str):
+        return value
+    try:
+        parsed = json.loads(value)
+    except ValueError:
+        return value
+    return parsed if isinstance(parsed, str) else value
+
+
+def _json_value(value, digits: int | None, is_json_text: bool):
+    """A cell as a JSON-ready Python value."""
+    if is_json_text and isinstance(value, str):
+        try:
+            return json.loads(value)
+        except ValueError:
+            return value
+    if _is_nested(value):
+        return json.loads(_dumps(value))
+    if value is None or value is pd.NA or value is pd.NaT:
+        return None
+    if isinstance(value, np.generic):
+        value = value.item()
+    if isinstance(value, float):
+        if math.isnan(value) or math.isinf(value):
+            return None
+        if digits is not None:
+            return float(f"{value:.{digits}g}")
+        return value
+    if isinstance(value, (int, bool, str)):
+        return value
+    return _plain(value)
+
+
+def write_json(df: pd.DataFrame, dest, lines: bool, digits: int | None) -> None:
+    """Write *df* as an array of records, or one record per line (*lines*).
+
+    A missing value is written as null; the key is kept.
+    """
+    jcols = set(json_columns(df))
+    cols = list(df.columns)
+    records = (
+        {
+            col: _json_value(value, digits, col in jcols)
+            for col, value in zip(cols, row, strict=True)
+        }
+        for row in df.itertuples(index=False, name=None)
+    )
+    with _open(dest) as fh:
+        if lines:
+            for record in records:
+                fh.write(json.dumps(record, ensure_ascii=False) + "\n")
+        else:
+            items = [json.dumps(r, ensure_ascii=False) for r in records]
+            fh.write("[\n" + ",\n".join("  " + s for s in items) + "\n]\n")
+
+
+def _cell_text(value, float_format: str | None) -> str:
+    if value is None or value is pd.NA or value is pd.NaT:
+        return ""
+    if isinstance(value, (float, np.floating)):
+        if math.isnan(value):
+            return ""
+        return float_format % value if float_format else str(float(value))
+    return str(value)
+
+
+def _is_numeric(series: pd.Series) -> bool:
+    return pd.api.types.is_numeric_dtype(series) and not pd.api.types.is_bool_dtype(
+        series
+    )
+
+
+def _table(df: pd.DataFrame, float_format: str | None, escape):
+    """Escaped header and rows of *df* (nested values as JSON text)."""
+    df = nested_as_text(df)
+    header = [escape(str(c)) for c in df.columns]
+    rows = [
+        [escape(_cell_text(v, float_format)) for v in row]
+        for row in df.itertuples(index=False, name=None)
+    ]
+    numeric = [_is_numeric(df[c]) for c in df.columns]
+    return header, rows, numeric
+
+
+def _escape_markdown(text: str) -> str:
+    return text.replace("\\", "\\\\").replace("|", "\\|").replace("\n", " ")
+
+
+def write_markdown(
+    df: pd.DataFrame, dest, float_format: str | None, header: bool
+) -> None:
+    """Write *df* as a pipe table; numeric columns are right-aligned.
+
+    A Markdown table always has a header row; with --removeheader it is empty.
+    """
+    names, rows, numeric = _table(df, float_format, _escape_markdown)
+    if not header:
+        names = [""] * len(names)
+    rule = ["---:" if num else "---" for num in numeric]
+    with _open(dest) as fh:
+        for cells in [names, rule, *rows]:
+            fh.write("| " + " | ".join(cells) + " |\n")
+
+
+_LATEX_SPECIAL = {
+    "\\": r"\textbackslash{}",
+    "&": r"\&",
+    "%": r"\%",
+    "$": r"\$",
+    "#": r"\#",
+    "_": r"\_",
+    "{": r"\{",
+    "}": r"\}",
+    "~": r"\textasciitilde{}",
+    "^": r"\textasciicircum{}",
+    # The default OT1 font prints these as an em dash and as inverted ! and ?
+    "|": r"\textbar{}",
+    "<": r"\textless{}",
+    ">": r"\textgreater{}",
+    "\n": " ",
+}
+
+
+def _escape_latex(text: str) -> str:
+    return "".join(_LATEX_SPECIAL.get(ch, ch) for ch in text)
+
+
+def write_latex(df: pd.DataFrame, dest, float_format: str | None, header: bool) -> None:
+    """Write *df* as a bare booktabs tabular (needs \\usepackage{booktabs}).
+
+    Numeric columns are right-aligned (r), others left (l).  Wrap it in your
+    own table environment with \\input{file.tex}.  A " (in JSON lists, say)
+    and ~ ^ print as intended with \\usepackage[T1]{fontenc}; the default
+    OT1 font has no straight double quote.
+    """
+    names, rows, numeric = _table(df, float_format, _escape_latex)
+    spec = "".join("r" if num else "l" for num in numeric)
+    with _open(dest) as fh:
+        fh.write(f"\\begin{{tabular}}{{{spec}}}\n\\toprule\n")
+        if header:
+            fh.write(" & ".join(names) + " \\\\\n\\midrule\n")
+        for cells in rows:
+            fh.write(" & ".join(cells) + " \\\\\n")
+        fh.write("\\bottomrule\n\\end{tabular}\n")
+
+
+def _open(dest):
+    """Open *dest* for writing if it is a path; use it as is if a stream."""
+    if isinstance(dest, str):
+        return open(dest, "w", encoding="utf-8", newline="")
+    return contextlib.nullcontext(dest)

{dataframe_toolkit-0.13.0 → dataframe_toolkit-0.15.0}/src/dftk/common/io.py

@@ -10,6 +10,10 @@ Ported from df/src/df/dfio.py with the following changes:
       -o FILE.parquet  → named parquet written to FILE, path printed to stdout
                          (NOT auto-deleted — reusable later in the same pipeline)
       -o FILE          → TSV written to FILE, nothing on stdout
+      -o FILE.json / .jsonl / .md / .tex
+                       → JSON, JSON Lines, Markdown or LaTeX table written to
+                         FILE (common/docformats.py); --format sends the same
+                         formats to stdout
 
   • Parquet pipes: when stdin (no DATAFILE, or -) or a pipe given as
     DATAFILE (<(...)) holds just the path of an existing .parquet file, that
@@ -18,7 +22,7 @@ Ported from df/src/df/dfio.py with the following changes:
     they can be referenced again downstream. The older ``...`` DATAFILE
     marker still works and requires a parquet path.
 
-  • Removed --sqlout and --latex (niche output formats).
+  • Removed --sqlout.  --latex became -o FILE.tex / --format tex.
   • Removed --experimental (unused).
   • parser_printdf kept as an alias for parser_output for compatibility.
 """
@@ -36,6 +40,8 @@ import tempfile
 import numpy as np
 import pandas as pd
 
+from dftk.common import docformats
+
 try:
     import duckdb
 except ImportError:
@@ -529,12 +535,27 @@ class io:
                 "path to stdout (auto-deleted by the next reader).  "
                 "Use -o FILE.parquet to write a named parquet and print its path to "
                 "stdout (not auto-deleted — reusable later in the same pipeline).  "
-                "Use -o FILE (no .parquet) to write TSV to FILE with no stdout output."
+                "Use -o FILE (no .parquet) to write TSV to FILE with no stdout output; "
+                "FILE.json, .jsonl, .md or .tex writes JSON, JSON Lines, a "
+                "Markdown table or a LaTeX tabular instead."
+            ),
+        )
+        g.add_argument(
+            "--format",
+            choices=docformats.FORMATS,
+            default=None,
+            help=(
+                "Output format for stdout or -o FILE, instead of the format "
+                "the file extension gives: tsv, json (array of records), "
+                "jsonl (one record per line), md (Markdown table), tex "
+                "(bare booktabs tabular: \\usepackage{booktabs} and [T1]{fontenc}, "
+                "then \\input the file).  Not with parquet output."
             ),
         )
         g.add_argument(
             "--digits",
-            help="Significant floating-point digits in TSV output",
+            help="Significant floating-point digits in TSV, JSON, Markdown and "
+            "LaTeX output",
             type=int,
             default=None,
         )
@@ -688,6 +709,7 @@ class io:
             try:
                 old, new = item.split(":")
                 df = df.rename(columns={old: new})
+                docformats.rename_json_columns(df, {old: new})
             except Exception:
                 logging.warning(f"--move: failed to rename from {item!r}")
 
@@ -836,12 +858,22 @@ class io:
 
         # Header: written unless --removeheader (--noheader is about the input)
         header = not getattr(args, "removeheader", False)
-        if not header and (output == "" or str(output).endswith(".parquet")):
+        to_parquet = output == "" or str(output).endswith(".parquet")
+        if not header and to_parquet:
             logging.warning(
                 "--removeheader has no effect on parquet output (-o); parquet "
                 "always keeps column names"
             )
 
+        fmt: str | None = getattr(args, "format", None)
+        if fmt is not None and to_parquet:
+            raise ValueError(
+                f"--format {fmt} can't be combined with parquet output (-o, "
+                "-o FILE.parquet): give -o FILE or leave -o out"
+            )
+        if fmt is None and output:
+            fmt = docformats.EXTENSIONS.get(os.path.splitext(output)[1].lower())
+
         if output == "":
             # Pipe mode — write temp parquet, print path
             os.makedirs(DFTK_TMPDIR, exist_ok=True)
@@ -857,13 +889,22 @@ class io:
             _write_parquet(df, output, meta=meta or None)
             print(output)
 
+        elif fmt in ("json", "jsonl"):
+            dest = output if output is not None else sys.stdout
+            docformats.write_json(df, dest, lines=fmt == "jsonl", digits=digits)
+
+        elif fmt in ("md", "tex"):
+            dest = output if output is not None else sys.stdout
+            write = docformats.write_markdown if fmt == "md" else docformats.write_latex
+            write(df, dest, float_format=float_format, header=header)
+
         else:
             # TSV — stdout or named file
             if digits is not None:
                 _warn_digits_exponent(df, digits)
             dest = output if output is not None else sys.stdout
             _write_tsv(
-                df,
+                docformats.nested_as_text(df),
                 dest,
                 sep="\t",
                 index=False,

{dataframe_toolkit-0.13.0 → dataframe_toolkit-0.15.0}/src/dftk/skills/dftk/SKILL.md

@@ -87,6 +87,12 @@ columns), `--drop COL…`, `--move OLD:NEW…` (rename), `--postquery EXPR`,
 `--dropna`, `--na_rep VAL`, `--removeheader`, and
 `--meta KEY=VALUE` (provenance, see below).
 
+Output format: TSV unless `-o FILE` ends in `.json` (array of records),
+`.jsonl`, `.md` (Markdown table) or `.tex` (bare booktabs tabular, for
+`\input` in a LaTeX table); `--format tsv|json|jsonl|md|tex` does the same
+on stdout. Lists and nested values are one line of JSON in TSV and stay
+nested in JSON output.
+
 ## Choosing a command
 
 Look at a table first:
@@ -104,6 +110,7 @@ Transform:
 | New column from a formula: `z = x / y`, `where(...)`, string, DNA and statistical functions | `eval -f`, `eval -c` (constant), `eval -s` (string) |
 | Fill empty cells: from another column, a group mean, the row above (`ffill`, per group with `groupby`) or by interpolation; sort first, fills follow row order | `eval -f "x = x.fillna(y)"`, `"x = x.groupby(g).ffill()"` (see `eval -h`) |
 | Filter rows (pandas expression) or run any DuckDB SQL (`FROM data`) | `query -q` / `query --sql` |
+| Read JSON or JSON Lines (no input needed); flatten nested fields with `pos.*`, lists with `unnest()` | `query --sql "SELECT * EXCLUDE (pos), pos.* FROM 'f.jsonl'"` |
 | Join two tables, including anti-joins | `merge DATAFILE -r OTHER -k KEY [-t left] [--only left]` |
 | Stack tables row-wise | `concat A B C [--sourcecol src]` |
 | Wide to long / long to wide plus aggregation | `melt` / `pivot -i ROWS -g COLS -v VALS -f FUNCS` |