dataframe-toolkit 0.14.0__tar.gz → 0.15.1__tar.gz

{dataframe_toolkit-0.14.0 → dataframe_toolkit-0.15.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: dataframe-toolkit
-Version: 0.14.0
+Version: 0.15.1
 Summary: A command-line toolkit for DataFrame analysis and manipulation
 Keywords: dataframe,statistics,data analysis,CLI,TSV,pandas
 Author: Gunnar Pálsson
@@ -166,7 +166,7 @@ publication-quality output.
 | `sample`   | Random row sampling (with or without replacement, by count or fraction)   |
 | `split`    | Split a dataframe into one file per group                                 |
 | `annotate` | Read and write provenance metadata (genome, source, …) in parquet files  |
-| `print`    | Read any dftk input (TSV, parquet, stdin) and write TSV                |
+| `print`    | Read any dftk input (TSV, parquet, stdin) and write TSV, JSON, Markdown or LaTeX |
 | `clean`    | Remove leftover temp parquet pipe files from interrupted pipelines       |
 | `help`     | List all subcommands or show full help for one: `dftk help stat`       |
 
@@ -195,7 +195,7 @@ For your own export, `dftk cleannames data.tsv --backend duckdb -o data.parquet`
 
 ### JSON files
 
-DuckDB reads JSON itself, so `query --sql` takes a JSON file in `FROM` (a file of records, `[{...}, {...}]`, or JSON Lines, one object per line; `'logs/*.jsonl'` reads several). When the SQL doesn't use the `data` table, dftk reads no input. Nested objects and lists would end up as one text cell each, so flatten them in the same query: `pos.*` makes a column of each field, `len()` counts a list and `unnest()` gives one row per item:
+DuckDB reads JSON itself, so `query --sql` takes a JSON file in `FROM` (a file of records, `[{...}, {...}]`, or JSON Lines, one object per line; `'logs/*.jsonl'` reads several). When the SQL doesn't use the `data` table, dftk reads no input. In TSV a nested object or list is shown as one line of JSON (`["x","y"]`), which later commands can't compute on, so flatten it in the same query: `pos.*` makes a column of each field, `len()` counts a list and `unnest()` gives one row per item:
 
 ```bash
 printf '%s\n' '{"id": 1, "pos": {"chr": "1", "start": 100}, "tags": ["x", "y"]}' \
@@ -210,6 +210,32 @@ dftk query --sql "SELECT id, unnest(tags) AS tag FROM 'records.jsonl'" -o \
   | dftk pivot -i tag -v id -f count
 ```
 
+A field that is a string in some records and a list in others (a variant catalog's `ReferenceRegion`, say) gets DuckDB type JSON; TSV shows its strings without quotes and its lists as JSON. `json_type()` tells them apart, so a list can become one row per item:
+
+```bash
+printf '%s\n' '{"locus": "AFF2", "region": "chrX:1-60"}' \
+              '{"locus": "ATXN7", "region": ["chr3:1-30", "chr3:30-42"]}' \
+  > catalog.jsonl
+dftk query --sql "
+  SELECT locus,
+         unnest(CASE WHEN json_type(region) = 'ARRAY' THEN region->>'\$[*]'
+                     ELSE [region->>'\$'] END) AS region
+  FROM 'catalog.jsonl'"
+# locus  region
+# AFF2   chrX:1-60
+# ATXN7  chr3:1-30
+# ATXN7  chr3:30-42
+```
+
+To get JSON back, write `-o FILE.json` or `.jsonl`, or `--format json` for stdout (see [Output formats](#output-formats)). Nested values and mixed fields are written as JSON again; a key missing from a record is written as `null`:
+
+```bash
+dftk query --sql "SELECT * FROM 'catalog.jsonl' WHERE locus LIKE 'ATXN%'" --format json
+# [
+#   {"locus": "ATXN7", "region": ["chr3:1-30", "chr3:30-42"]}
+# ]
+```
+
 ### Large tables: hexbin instead of points
 
 With tens of thousands of points a scatter plot is a solid blob. `--hexbin` counts points per hexagon instead (`--hexlog` for a log colour scale), and `--fit lowess` adds a smooth curve:
@@ -420,6 +446,28 @@ Older scripts give `...` as the file to read a parquet path from stdin; that
 still works. In zsh with Oh My Zsh an unquoted `...` becomes `../..`, so
 leave it out (or use `-`) in scripts you share.
 
+### Output formats
+
+Without `-o`, output is TSV on stdout. `-o` alone, or `-o FILE.parquet`, writes parquet and prints the path for the next command; `-o FILE` writes TSV, except for these extensions:
+
+| `-o` | Writes |
+|------|--------|
+| `FILE.json` | JSON array of records |
+| `FILE.jsonl` | JSON Lines, one record per line |
+| `FILE.md` | Markdown table (numeric columns right-aligned) |
+| `FILE.tex` | bare LaTeX `tabular` with booktabs rules |
+
+`--format tsv|json|jsonl|md|tex` writes the same formats to stdout, or overrides the extension. `--digits N` applies to all of them. The `.tex` file is only the tabular, with special characters (`_ & % $ # { } ~ ^ \ | < >`) escaped; it needs `\usepackage{booktabs}`, and goes inside your own `table` environment with `\input{tips_by_day.tex}`. Add `\usepackage[T1]{fontenc}` (pdfLaTeX's default font has no straight `"`, which JSON lists contain):
+
+```bash
+dftk dataset tips -o | dftk pivot -i day -v tip -f mean count --digits 3 --format md
+# | day | mean_tip | count_tip |
+# | --- | ---: | ---: |
+# | Thur | 2.77 | 62 |
+# ...
+dftk dataset tips -o | dftk pivot -i day -v tip -f mean count --digits 3 -o tips_by_day.tex
+```
+
 ### Comment lines and preambles
 
 dftk has no comment or skip-rows option: `#` is ordinary data, so headers

{dataframe_toolkit-0.14.0 → dataframe_toolkit-0.15.1}/README.md

@@ -136,7 +136,7 @@ publication-quality output.
 | `sample`   | Random row sampling (with or without replacement, by count or fraction)   |
 | `split`    | Split a dataframe into one file per group                                 |
 | `annotate` | Read and write provenance metadata (genome, source, …) in parquet files  |
-| `print`    | Read any dftk input (TSV, parquet, stdin) and write TSV                |
+| `print`    | Read any dftk input (TSV, parquet, stdin) and write TSV, JSON, Markdown or LaTeX |
 | `clean`    | Remove leftover temp parquet pipe files from interrupted pipelines       |
 | `help`     | List all subcommands or show full help for one: `dftk help stat`       |
 
@@ -165,7 +165,7 @@ For your own export, `dftk cleannames data.tsv --backend duckdb -o data.parquet`
 
 ### JSON files
 
-DuckDB reads JSON itself, so `query --sql` takes a JSON file in `FROM` (a file of records, `[{...}, {...}]`, or JSON Lines, one object per line; `'logs/*.jsonl'` reads several). When the SQL doesn't use the `data` table, dftk reads no input. Nested objects and lists would end up as one text cell each, so flatten them in the same query: `pos.*` makes a column of each field, `len()` counts a list and `unnest()` gives one row per item:
+DuckDB reads JSON itself, so `query --sql` takes a JSON file in `FROM` (a file of records, `[{...}, {...}]`, or JSON Lines, one object per line; `'logs/*.jsonl'` reads several). When the SQL doesn't use the `data` table, dftk reads no input. In TSV a nested object or list is shown as one line of JSON (`["x","y"]`), which later commands can't compute on, so flatten it in the same query: `pos.*` makes a column of each field, `len()` counts a list and `unnest()` gives one row per item:
 
 ```bash
 printf '%s\n' '{"id": 1, "pos": {"chr": "1", "start": 100}, "tags": ["x", "y"]}' \
@@ -180,6 +180,32 @@ dftk query --sql "SELECT id, unnest(tags) AS tag FROM 'records.jsonl'" -o \
   | dftk pivot -i tag -v id -f count
 ```
 
+A field that is a string in some records and a list in others (a variant catalog's `ReferenceRegion`, say) gets DuckDB type JSON; TSV shows its strings without quotes and its lists as JSON. `json_type()` tells them apart, so a list can become one row per item:
+
+```bash
+printf '%s\n' '{"locus": "AFF2", "region": "chrX:1-60"}' \
+              '{"locus": "ATXN7", "region": ["chr3:1-30", "chr3:30-42"]}' \
+  > catalog.jsonl
+dftk query --sql "
+  SELECT locus,
+         unnest(CASE WHEN json_type(region) = 'ARRAY' THEN region->>'\$[*]'
+                     ELSE [region->>'\$'] END) AS region
+  FROM 'catalog.jsonl'"
+# locus  region
+# AFF2   chrX:1-60
+# ATXN7  chr3:1-30
+# ATXN7  chr3:30-42
+```
+
+To get JSON back, write `-o FILE.json` or `.jsonl`, or `--format json` for stdout (see [Output formats](#output-formats)). Nested values and mixed fields are written as JSON again; a key missing from a record is written as `null`:
+
+```bash
+dftk query --sql "SELECT * FROM 'catalog.jsonl' WHERE locus LIKE 'ATXN%'" --format json
+# [
+#   {"locus": "ATXN7", "region": ["chr3:1-30", "chr3:30-42"]}
+# ]
+```
+
 ### Large tables: hexbin instead of points
 
 With tens of thousands of points a scatter plot is a solid blob. `--hexbin` counts points per hexagon instead (`--hexlog` for a log colour scale), and `--fit lowess` adds a smooth curve:
@@ -390,6 +416,28 @@ Older scripts give `...` as the file to read a parquet path from stdin; that
 still works. In zsh with Oh My Zsh an unquoted `...` becomes `../..`, so
 leave it out (or use `-`) in scripts you share.
 
+### Output formats
+
+Without `-o`, output is TSV on stdout. `-o` alone, or `-o FILE.parquet`, writes parquet and prints the path for the next command; `-o FILE` writes TSV, except for these extensions:
+
+| `-o` | Writes |
+|------|--------|
+| `FILE.json` | JSON array of records |
+| `FILE.jsonl` | JSON Lines, one record per line |
+| `FILE.md` | Markdown table (numeric columns right-aligned) |
+| `FILE.tex` | bare LaTeX `tabular` with booktabs rules |
+
+`--format tsv|json|jsonl|md|tex` writes the same formats to stdout, or overrides the extension. `--digits N` applies to all of them. The `.tex` file is only the tabular, with special characters (`_ & % $ # { } ~ ^ \ | < >`) escaped; it needs `\usepackage{booktabs}`, and goes inside your own `table` environment with `\input{tips_by_day.tex}`. Add `\usepackage[T1]{fontenc}` (pdfLaTeX's default font has no straight `"`, which JSON lists contain):
+
+```bash
+dftk dataset tips -o | dftk pivot -i day -v tip -f mean count --digits 3 --format md
+# | day | mean_tip | count_tip |
+# | --- | ---: | ---: |
+# | Thur | 2.77 | 62 |
+# ...
+dftk dataset tips -o | dftk pivot -i day -v tip -f mean count --digits 3 -o tips_by_day.tex
+```
+
 ### Comment lines and preambles
 
 dftk has no comment or skip-rows option: `#` is ordinary data, so headers

{dataframe_toolkit-0.14.0 → dataframe_toolkit-0.15.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.14.0"
+version = "0.15.1"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 keywords = [

{dataframe_toolkit-0.14.0 → dataframe_toolkit-0.15.1}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.14.0"
+version = "0.15.1"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 authors = [

{dataframe_toolkit-0.14.0 → dataframe_toolkit-0.15.1}/src/dftk/commands/merge_cmd.py

@@ -29,6 +29,7 @@ import uuid
 import pandas as pd
 
 from dftk.commands.base import BaseCommand
+from dftk.common import docformats
 from dftk.common.io import check_cols, io
 
 logger = logging.getLogger(__name__)
@@ -113,6 +114,24 @@ def _add_cross_col(
     args.drop = drop
 
 
+def _carry_json_columns(
+    merged: pd.DataFrame,
+    left: pd.DataFrame,
+    right: pd.DataFrame,
+    suffixes: tuple[str, str],
+) -> None:
+    """Mark the JSON-text columns of both inputs in *merged* (see docformats)."""
+    names = []
+    for side, other, suffix in ((left, right, suffixes[0]), (right, left, suffixes[1])):
+        for col in docformats.json_columns(side):
+            name = col
+            if col in other.columns and f"{col}{suffix}" in merged.columns:
+                name = f"{col}{suffix}"
+            if name in merged.columns and name not in names:
+                names.append(name)
+    docformats.mark_json_columns(merged, names)
+
+
 def _gap_sides(args: argparse.Namespace) -> tuple[bool, bool]:
     """Return (left, right): which side's columns can get missing values.
 
@@ -227,6 +246,7 @@ def _do_merge(
     if "_merge" in df_merged.columns:
         df_merged = df_merged.drop(columns=["_merge"])
 
+    _carry_json_columns(df_merged, df_left, df_right, suffixes)
     return df_merged
 
 

{dataframe_toolkit-0.14.0 → dataframe_toolkit-0.15.1}/src/dftk/commands/print_cmd.py

@@ -2,7 +2,8 @@
 dftk.commands.print_cmd — the ``dftk print`` subcommand.
 
 Reads any dftk-compatible input (TSV file, TSV on stdin, or a parquet pipe
-path on stdin) and writes TSV to stdout.  Acts as the terminal stage of a
+path on stdin) and writes TSV to stdout, or JSON, a Markdown table or a
+LaTeX tabular with --format json|jsonl|md|tex.  Acts as the terminal stage of a
 dftk pipeline:
 
     dftk stat -g grp -s val -o \\
@@ -12,6 +13,11 @@ dftk pipeline:
 With --noheader and --move it names the columns of a headerless file:
 
     dftk print data.txt --noheader --move V1:chr V2:begin V3:end
+
+A table for a document:
+
+    dftk dataset tips -o | dftk pivot -i day -v tip -f mean -o \\
+      | dftk print --digits 3 --format md
 """
 
 import argparse
@@ -24,7 +30,7 @@ class PrintCommand(BaseCommand):
     """Read any dftk input and write TSV to stdout."""
 
     name = "print"
-    help = "Convert any input (TSV or parquet pipe) to TSV on stdout"
+    help = "Write any input (TSV or parquet pipe) as TSV, JSON, Markdown or LaTeX"
 
     def add_arguments(self, parser: argparse.ArgumentParser) -> None:
         # The standard output options (--move, --select, --removeheader, …)

{dataframe_toolkit-0.14.0 → dataframe_toolkit-0.15.1}/src/dftk/commands/query_cmd.py

@@ -33,6 +33,7 @@ import os
 import pandas as pd
 
 from dftk.commands.base import BaseCommand
+from dftk.common import docformats
 from dftk.common.io import io
 
 try:
@@ -100,7 +101,11 @@ def _sql_query(args: argparse.Namespace, sql: str, table: str) -> pd.DataFrame:
         df = result.df()
     except Exception as exc:
         raise ValueError(f"DuckDB SQL failed: {exc}") from exc
-    return _hugeint_to_int64(df, description)
+    df = _hugeint_to_int64(df, description)
+    docformats.mark_json_columns(
+        df, [d[0] for d in description or [] if str(d[1]) == "JSON"]
+    )
+    return df
 
 
 def _uses_table(sql: str, table: str) -> bool:
@@ -256,6 +261,17 @@ column of each field, len() counts a list, unnest() gives a row per item:
     "SELECT * EXCLUDE (pos, tags), pos.* FROM 'records.jsonl'" -o \\
     | dftk stat -c start
 
+In TSV output a list or object that is left nested is shown as one line
+of JSON (["x","y"]). A field that is a string in some records and a list
+in others gets DuckDB type JSON; its plain strings are shown without
+quotes. To get JSON back, give -o FILE.json (or .jsonl) or --format json;
+nested values, and such mixed fields, are written as JSON again, and a key
+missing from a record is written as null:
+
+  dftk query --sql "SELECT * FROM 'records.jsonl' WHERE pos.chr = '1'" \\
+    --format json
+  dftk query --sql "SELECT id, tags FROM 'records.jsonl'" -o ids.jsonl
+
 OUT-OF-CORE LARGE FILES
 ------------------------
 When the input is a named .parquet file, DuckDB queries it natively without

dataframe_toolkit-0.15.1/src/dftk/common/docformats.py

@@ -0,0 +1,262 @@
+"""
+dftk.common.docformats — JSON, Markdown and LaTeX output for io.printdf.
+
+Nested values (lists, structs, maps) come from DuckDB as numpy arrays and
+dicts.  TSV, Markdown and LaTeX show them as one line of JSON text; JSON
+output writes them as JSON.
+
+A DuckDB column of type JSON (from a JSON file whose field is a string in one
+record and a list in another) reaches pandas as JSON text, e.g.
+'"chrX:1-2"' and '["chr3:1-2","chr3:2-3"]'.  query_cmd lists such columns in
+the parquet metadata key JSON_COLUMNS_KEY, so they survive -o pipes; the text
+view drops the quotes around a plain string and JSON output parses them.
+"""
+
+import contextlib
+import decimal
+import json
+import math
+
+import numpy as np
+import pandas as pd
+
+JSON_COLUMNS_KEY = "dftk_json_columns"
+
+# Output format by file extension; --format names the same formats
+EXTENSIONS = {".json": "json", ".jsonl": "jsonl", ".md": "md", ".tex": "tex"}
+FORMATS = ("tsv", "json", "jsonl", "md", "tex")
+
+
+def json_columns(df: pd.DataFrame) -> list[str]:
+    """Columns of *df* that hold DuckDB JSON text (see module docstring)."""
+    names = df.attrs.get("_parquet_meta", {}).get(JSON_COLUMNS_KEY, "")
+    return [c for c in names.split(",") if c in df.columns]
+
+
+def mark_json_columns(df: pd.DataFrame, columns: list[str]) -> None:
+    """Record *columns* as JSON text in df.attrs (carried into parquet)."""
+    meta = dict(df.attrs.get("_parquet_meta", {}))
+    if columns:
+        meta[JSON_COLUMNS_KEY] = ",".join(columns)
+    else:
+        meta.pop(JSON_COLUMNS_KEY, None)
+    df.attrs["_parquet_meta"] = meta
+
+
+def rename_json_columns(df: pd.DataFrame, mapping: dict[str, str]) -> None:
+    """Follow a --move rename in the JSON column list."""
+    names = df.attrs.get("_parquet_meta", {}).get(JSON_COLUMNS_KEY)
+    if names:
+        mark_json_columns(df, [mapping.get(c, c) for c in names.split(",")])
+
+
+def _plain(value):
+    """json.dumps fallback for numpy, Decimal and other non-JSON values."""
+    if isinstance(value, np.ndarray):  # MaskedArray.tolist() gives None
+        return value.tolist()
+    if isinstance(value, np.generic):
+        return value.item()
+    if isinstance(value, decimal.Decimal):
+        return float(value)
+    return str(value)
+
+
+def _is_nested(value) -> bool:
+    return isinstance(value, (np.ndarray, list, tuple, dict))
+
+
+def _dumps(value) -> str:
+    text = json.dumps(value, default=_plain, ensure_ascii=False, separators=(",", ":"))
+    if "NaN" in text or "Infinity" in text:
+        # NaN and Infinity aren't JSON: write them as null
+        parsed = json.loads(text, parse_constant=lambda _: None)
+        text = json.dumps(parsed, ensure_ascii=False, separators=(",", ":"))
+    return text
+
+
+def nested_as_text(df: pd.DataFrame) -> pd.DataFrame:
+    """Return *df* with nested values and JSON-column text as one-line text."""
+    jcols = set(json_columns(df))
+    out = None
+    # By position: column names may repeat
+    for i, col in enumerate(df.columns):
+        series = df.iloc[:, i]
+        dtype = series.dtype
+        if not (
+            pd.api.types.is_object_dtype(dtype) or pd.api.types.is_string_dtype(dtype)
+        ):
+            continue
+        if col in jcols:
+            new = series.map(_json_text_cell)
+        elif series.map(_is_nested).any():
+            new = series.map(lambda v: _dumps(v) if _is_nested(v) else v)
+        else:
+            continue
+        if out is None:
+            out = df.copy()
+        out.isetitem(i, new)
+    return df if out is None else out
+
+
+def _json_text_cell(value):
+    """'"chrX:1-2"' → 'chrX:1-2'; a list or object stays JSON text."""
+    if not isinstance(value, str):
+        return value
+    try:
+        parsed = json.loads(value)
+    except ValueError:
+        return value
+    return parsed if isinstance(parsed, str) else value
+
+
+def _json_value(value, digits: int | None, is_json_text: bool):
+    """A cell as a JSON-ready Python value."""
+    if is_json_text and isinstance(value, str):
+        try:
+            return json.loads(value)
+        except ValueError:
+            return value
+    if _is_nested(value):
+        return json.loads(_dumps(value))
+    if value is None or value is pd.NA or value is pd.NaT:
+        return None
+    if isinstance(value, np.generic):
+        value = value.item()
+    if isinstance(value, float):
+        if math.isnan(value) or math.isinf(value):
+            return None
+        if digits is not None:
+            return float(f"{value:.{digits}g}")
+        return value
+    if isinstance(value, (int, bool, str)):
+        return value
+    return _plain(value)
+
+
+def write_json(df: pd.DataFrame, dest, lines: bool, digits: int | None) -> None:
+    """Write *df* as an array of records, or one record per line (*lines*).
+
+    A missing value is written as null; the key is kept.
+    """
+    repeated = df.columns[df.columns.duplicated()].unique().tolist()
+    if repeated:
+        raise ValueError(
+            f"JSON output needs unique column names; repeated: {repeated} "
+            "(rename one with --move OLD:NEW)"
+        )
+    jcols = set(json_columns(df))
+    cols = list(df.columns)
+    records = (
+        {
+            col: _json_value(value, digits, col in jcols)
+            for col, value in zip(cols, row, strict=True)
+        }
+        for row in df.itertuples(index=False, name=None)
+    )
+    with _open(dest) as fh:
+        if lines:
+            for record in records:
+                fh.write(json.dumps(record, ensure_ascii=False) + "\n")
+        else:
+            items = [json.dumps(r, ensure_ascii=False) for r in records]
+            fh.write("[\n" + ",\n".join("  " + s for s in items) + "\n]\n")
+
+
+def _cell_text(value, float_format: str | None) -> str:
+    if value is None or value is pd.NA or value is pd.NaT:
+        return ""
+    if isinstance(value, (float, np.floating)):
+        if math.isnan(value):
+            return ""
+        return float_format % value if float_format else str(float(value))
+    return str(value)
+
+
+def _is_numeric(series: pd.Series) -> bool:
+    return pd.api.types.is_numeric_dtype(series) and not pd.api.types.is_bool_dtype(
+        series
+    )
+
+
+def _table(df: pd.DataFrame, float_format: str | None, escape):
+    """Escaped header and rows of *df* (nested values as JSON text)."""
+    df = nested_as_text(df)
+    header = [escape(str(c)) for c in df.columns]
+    rows = [
+        [escape(_cell_text(v, float_format)) for v in row]
+        for row in df.itertuples(index=False, name=None)
+    ]
+    numeric = [_is_numeric(df.iloc[:, i]) for i in range(df.shape[1])]
+    return header, rows, numeric
+
+
+def _escape_markdown(text: str) -> str:
+    return text.replace("\\", "\\\\").replace("|", "\\|").replace("\n", " ")
+
+
+def write_markdown(
+    df: pd.DataFrame, dest, float_format: str | None, header: bool
+) -> None:
+    """Write *df* as a pipe table; numeric columns are right-aligned.
+
+    A Markdown table always has a header row; with --removeheader it is empty.
+    """
+    names, rows, numeric = _table(df, float_format, _escape_markdown)
+    if not header:
+        names = [""] * len(names)
+    rule = ["---:" if num else "---" for num in numeric]
+    with _open(dest) as fh:
+        for cells in [names, rule, *rows]:
+            fh.write("| " + " | ".join(cells) + " |\n")
+
+
+_LATEX_SPECIAL = {
+    "\\": r"\textbackslash{}",
+    "&": r"\&",
+    "%": r"\%",
+    "$": r"\$",
+    "#": r"\#",
+    "_": r"\_",
+    "{": r"\{",
+    "}": r"\}",
+    # A row starting with [ after \\ would be read as \\[length]
+    "[": "{[}",
+    "]": "{]}",
+    "~": r"\textasciitilde{}",
+    "^": r"\textasciicircum{}",
+    # The default OT1 font prints these as an em dash and as inverted ! and ?
+    "|": r"\textbar{}",
+    "<": r"\textless{}",
+    ">": r"\textgreater{}",
+    "\n": " ",
+}
+
+
+def _escape_latex(text: str) -> str:
+    return "".join(_LATEX_SPECIAL.get(ch, ch) for ch in text)
+
+
+def write_latex(df: pd.DataFrame, dest, float_format: str | None, header: bool) -> None:
+    """Write *df* as a bare booktabs tabular (needs \\usepackage{booktabs}).
+
+    Numeric columns are right-aligned (r), others left (l).  Wrap it in your
+    own table environment with \\input{file.tex}.  A " (in JSON lists, say)
+    and ~ ^ print as intended with \\usepackage[T1]{fontenc}; the default
+    OT1 font has no straight double quote.
+    """
+    names, rows, numeric = _table(df, float_format, _escape_latex)
+    spec = "".join("r" if num else "l" for num in numeric)
+    with _open(dest) as fh:
+        fh.write(f"\\begin{{tabular}}{{{spec}}}\n\\toprule\n")
+        if header:
+            fh.write(" & ".join(names) + " \\\\\n\\midrule\n")
+        for cells in rows:
+            fh.write(" & ".join(cells) + " \\\\\n")
+        fh.write("\\bottomrule\n\\end{tabular}\n")
+
+
+def _open(dest):
+    """Open *dest* for writing if it is a path; use it as is if a stream."""
+    if isinstance(dest, str):
+        return open(dest, "w", encoding="utf-8", newline="")
+    return contextlib.nullcontext(dest)

{dataframe_toolkit-0.14.0 → dataframe_toolkit-0.15.1}/src/dftk/common/io.py

@@ -10,6 +10,10 @@ Ported from df/src/df/dfio.py with the following changes:
       -o FILE.parquet  → named parquet written to FILE, path printed to stdout
                          (NOT auto-deleted — reusable later in the same pipeline)
       -o FILE          → TSV written to FILE, nothing on stdout
+      -o FILE.json / .jsonl / .md / .tex
+                       → JSON, JSON Lines, Markdown or LaTeX table written to
+                         FILE (common/docformats.py); --format sends the same
+                         formats to stdout
 
   • Parquet pipes: when stdin (no DATAFILE, or -) or a pipe given as
     DATAFILE (<(...)) holds just the path of an existing .parquet file, that
@@ -18,7 +22,7 @@ Ported from df/src/df/dfio.py with the following changes:
     they can be referenced again downstream. The older ``...`` DATAFILE
     marker still works and requires a parquet path.
 
-  • Removed --sqlout and --latex (niche output formats).
+  • Removed --sqlout.  --latex became -o FILE.tex / --format tex.
   • Removed --experimental (unused).
   • parser_printdf kept as an alias for parser_output for compatibility.
 """
@@ -36,6 +40,8 @@ import tempfile
 import numpy as np
 import pandas as pd
 
+from dftk.common import docformats
+
 try:
     import duckdb
 except ImportError:
@@ -529,12 +535,27 @@ class io:
                 "path to stdout (auto-deleted by the next reader).  "
                 "Use -o FILE.parquet to write a named parquet and print its path to "
                 "stdout (not auto-deleted — reusable later in the same pipeline).  "
-                "Use -o FILE (no .parquet) to write TSV to FILE with no stdout output."
+                "Use -o FILE (no .parquet) to write TSV to FILE with no stdout output; "
+                "FILE.json, .jsonl, .md or .tex writes JSON, JSON Lines, a "
+                "Markdown table or a LaTeX tabular instead."
+            ),
+        )
+        g.add_argument(
+            "--format",
+            choices=docformats.FORMATS,
+            default=None,
+            help=(
+                "Output format for stdout or -o FILE, instead of the format "
+                "the file extension gives: tsv, json (array of records), "
+                "jsonl (one record per line), md (Markdown table), tex "
+                "(bare booktabs tabular: \\usepackage{booktabs} and [T1]{fontenc}, "
+                "then \\input the file).  Not with parquet output."
             ),
         )
         g.add_argument(
             "--digits",
-            help="Significant floating-point digits in TSV output",
+            help="Significant floating-point digits in TSV, JSON, Markdown and "
+            "LaTeX output",
             type=int,
             default=None,
         )
@@ -688,6 +709,7 @@ class io:
             try:
                 old, new = item.split(":")
                 df = df.rename(columns={old: new})
+                docformats.rename_json_columns(df, {old: new})
             except Exception:
                 logging.warning(f"--move: failed to rename from {item!r}")
 
@@ -836,12 +858,22 @@ class io:
 
         # Header: written unless --removeheader (--noheader is about the input)
         header = not getattr(args, "removeheader", False)
-        if not header and (output == "" or str(output).endswith(".parquet")):
+        to_parquet = output == "" or str(output).endswith(".parquet")
+        if not header and to_parquet:
             logging.warning(
                 "--removeheader has no effect on parquet output (-o); parquet "
                 "always keeps column names"
             )
 
+        fmt: str | None = getattr(args, "format", None)
+        if fmt is not None and to_parquet:
+            raise ValueError(
+                f"--format {fmt} can't be combined with parquet output (-o, "
+                "-o FILE.parquet): give -o FILE or leave -o out"
+            )
+        if fmt is None and output:
+            fmt = docformats.EXTENSIONS.get(os.path.splitext(output)[1].lower())
+
         if output == "":
             # Pipe mode — write temp parquet, print path
             os.makedirs(DFTK_TMPDIR, exist_ok=True)
@@ -857,13 +889,22 @@ class io:
             _write_parquet(df, output, meta=meta or None)
             print(output)
 
+        elif fmt in ("json", "jsonl"):
+            dest = output if output is not None else sys.stdout
+            docformats.write_json(df, dest, lines=fmt == "jsonl", digits=digits)
+
+        elif fmt in ("md", "tex"):
+            dest = output if output is not None else sys.stdout
+            write = docformats.write_markdown if fmt == "md" else docformats.write_latex
+            write(df, dest, float_format=float_format, header=header)
+
         else:
             # TSV — stdout or named file
             if digits is not None:
                 _warn_digits_exponent(df, digits)
             dest = output if output is not None else sys.stdout
             _write_tsv(
-                df,
+                docformats.nested_as_text(df),
                 dest,
                 sep="\t",
                 index=False,

{dataframe_toolkit-0.14.0 → dataframe_toolkit-0.15.1}/src/dftk/skills/dftk/SKILL.md

@@ -87,6 +87,12 @@ columns), `--drop COL…`, `--move OLD:NEW…` (rename), `--postquery EXPR`,
 `--dropna`, `--na_rep VAL`, `--removeheader`, and
 `--meta KEY=VALUE` (provenance, see below).
 
+Output format: TSV unless `-o FILE` ends in `.json` (array of records),
+`.jsonl`, `.md` (Markdown table) or `.tex` (bare booktabs tabular, for
+`\input` in a LaTeX table); `--format tsv|json|jsonl|md|tex` does the same
+on stdout. Lists and nested values are one line of JSON in TSV and stay
+nested in JSON output.
+
 ## Choosing a command
 
 Look at a table first: