dataframe-toolkit 0.11.2__tar.gz → 0.11.3__tar.gz

{dataframe_toolkit-0.11.2 → dataframe_toolkit-0.11.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: dataframe-toolkit
-Version: 0.11.2
+Version: 0.11.3
 Summary: A command-line toolkit for DataFrame analysis and manipulation
 Keywords: dataframe,statistics,data analysis,CLI,TSV,pandas
 Author: Gunnar Pálsson
@@ -20,6 +20,7 @@ Requires-Dist: statsmodels>=0.14
 Requires-Dist: patsy>=0.5.6
 Requires-Dist: duckdb>=0.8
 Requires-Dist: matplotlib>=3.7
+Requires-Dist: pillow>=13.0
 Requires-Dist: seaborn>=0.12
 Requires-Dist: tqdm>=4.65
 Requires-Python: >=3.12
@@ -298,6 +299,70 @@ dftk dataset healthexp -o \
       --drop before after
 ```
 
+### Differences between rows
+
+For the change in a column from one row to the next, `eval -f` takes
+pandas `diff()` and `pct_change()`, also within groups. They use the rows
+in their order, so sort first, in a step of its own:
+
+```bash
+dftk dataset healthexp -o | dftk eval ... --sortasc Country Year -o \
+  | dftk eval ... -f "change = Spending_USD.groupby(Country).diff().round(1)" \
+      -f "pct = (Spending_USD.groupby(Country).pct_change() * 100).round(1)" \
+      -f "years = Year.groupby(Country).diff()"
+```
+
+```text
+Year  Country  Spending_USD  Life_Expectancy  change  pct   years
+1971  Canada   313.391       72.8
+1976  Canada   543.337       73.8             229.9   73.4  5.0
+1979  Canada   692.269       75.1             148.9   27.4  3.0
+1980  Canada   791.812       75.2             99.5    14.4  1.0
+```
+
+The first row of each country gets an empty cell. Without
+`groupby(Country)`, it would be compared with the last row of the
+previous country.
+
+To compare a value with one in another column of the previous or next row, for example the
+gap between segments given by `Chr`, `Begin`, `End`, use SQL's `LAG()` /
+`LEAD()` in `query --sql`:
+
+```bash
+dftk query segments.tsv --sql \
+  'SELECT *, "Begin" - LAG("End") OVER (PARTITION BY Chr ORDER BY "Begin")
+     AS gap
+   FROM data ORDER BY Chr, "Begin"'
+```
+
+```text
+Chr   Begin  End  Id  gap
+chr1  100    200  a
+chr1  250    400  b   50
+chr1  390    500  c   -10
+chr2  50     80   d
+chr2  100    300  e   20
+```
+
+A negative gap is an overlap, and the first segment on each chromosome
+gets an empty cell. `lead("Begin") OVER (...) - "End"` gives the gap to
+the next segment instead.
+
+- `Begin` and `End` are SQL keywords: quote such column names with double
+  quotes and the whole statement with single quotes.
+- No sorting is needed first; `ORDER BY` inside `OVER` sets the order.
+- Integers stay integers.
+
+`eval -f` can do the same with pandas `shift`, but it goes by row order, so
+sort by `Chr` and `Begin` first (results are floats):
+
+```bash
+dftk eval segments.tsv -f 'gap = Begin - End.groupby(Chr).shift()'
+```
+
+Without `groupby(Chr)` the first segment of a chromosome is compared with
+the last one of the previous chromosome.
+
 ### Bootstrap confidence intervals
 
 Bootstrap mode repeats the full stat computation N times on resampled data

{dataframe_toolkit-0.11.2 → dataframe_toolkit-0.11.3}/README.md

@@ -269,6 +269,70 @@ dftk dataset healthexp -o \
       --drop before after
 ```
 
+### Differences between rows
+
+For the change in a column from one row to the next, `eval -f` takes
+pandas `diff()` and `pct_change()`, also within groups. They use the rows
+in their order, so sort first, in a step of its own:
+
+```bash
+dftk dataset healthexp -o | dftk eval ... --sortasc Country Year -o \
+  | dftk eval ... -f "change = Spending_USD.groupby(Country).diff().round(1)" \
+      -f "pct = (Spending_USD.groupby(Country).pct_change() * 100).round(1)" \
+      -f "years = Year.groupby(Country).diff()"
+```
+
+```text
+Year  Country  Spending_USD  Life_Expectancy  change  pct   years
+1971  Canada   313.391       72.8
+1976  Canada   543.337       73.8             229.9   73.4  5.0
+1979  Canada   692.269       75.1             148.9   27.4  3.0
+1980  Canada   791.812       75.2             99.5    14.4  1.0
+```
+
+The first row of each country gets an empty cell. Without
+`groupby(Country)`, it would be compared with the last row of the
+previous country.
+
+To compare a value with one in another column of the previous or next row, for example the
+gap between segments given by `Chr`, `Begin`, `End`, use SQL's `LAG()` /
+`LEAD()` in `query --sql`:
+
+```bash
+dftk query segments.tsv --sql \
+  'SELECT *, "Begin" - LAG("End") OVER (PARTITION BY Chr ORDER BY "Begin")
+     AS gap
+   FROM data ORDER BY Chr, "Begin"'
+```
+
+```text
+Chr   Begin  End  Id  gap
+chr1  100    200  a
+chr1  250    400  b   50
+chr1  390    500  c   -10
+chr2  50     80   d
+chr2  100    300  e   20
+```
+
+A negative gap is an overlap, and the first segment on each chromosome
+gets an empty cell. `lead("Begin") OVER (...) - "End"` gives the gap to
+the next segment instead.
+
+- `Begin` and `End` are SQL keywords: quote such column names with double
+  quotes and the whole statement with single quotes.
+- No sorting is needed first; `ORDER BY` inside `OVER` sets the order.
+- Integers stay integers.
+
+`eval -f` can do the same with pandas `shift`, but it goes by row order, so
+sort by `Chr` and `Begin` first (results are floats):
+
+```bash
+dftk eval segments.tsv -f 'gap = Begin - End.groupby(Chr).shift()'
+```
+
+Without `groupby(Chr)` the first segment of a chromosome is compared with
+the last one of the previous chromosome.
+
 ### Bootstrap confidence intervals
 
 Bootstrap mode repeats the full stat computation N times on resampled data

{dataframe_toolkit-0.11.2 → dataframe_toolkit-0.11.3}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.11.2"
+version = "0.11.3"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 keywords = [
@@ -29,6 +29,7 @@ dependencies = [
     "patsy>=0.5.6",
     "duckdb>=0.8",
     "matplotlib>=3.7",
+    "pillow>=13.0",
     "seaborn>=0.12",
     "tqdm>=4.65",
 ]

{dataframe_toolkit-0.11.2 → dataframe_toolkit-0.11.3}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.11.2"
+version = "0.11.3"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 authors = [
@@ -26,6 +26,7 @@ dependencies = [
     "patsy>=0.5.6",
     "duckdb>=0.8",
     "matplotlib>=3.7",
+    "pillow>=13.0",  # not used directly (matplotlib needs it); floor for security fixes
     "seaborn>=0.12",
     "tqdm>=4.65",
 ]

{dataframe_toolkit-0.11.2 → dataframe_toolkit-0.11.3}/src/dftk/commands/eval_cmd.py

@@ -67,6 +67,24 @@ interpolate() has no grouped form. The README ("Filling missing values")
 covers defaults from a merge, interpolating within groups, and more.
 
 
+DIFFERENCES BETWEEN ROWS (-f)
+-----------------------------
+  -f "dx = x.diff()"                    x minus x in the row above
+  -f "pct = x.pct_change() * 100"       the same as a percentage
+  -f "dx = x.groupby(g).diff()"         within each group of g
+  -f "gap = Begin - End.groupby(Chr).shift()"   another column, row above
+
+Like ffill, these use the rows in their order, so sort first, in a step
+of its own; the first row of each group gets an empty cell:
+
+  dftk dataset healthexp -o | dftk eval ... --sortasc Country Year -o \\
+      | dftk eval ... -f "change = Spending_USD.groupby(Country).diff().round(1)" \\
+          -f "years = Year.groupby(Country).diff()"
+
+query --sql with LAG()/LEAD() does the same without sorting first and
+keeps integers integers (see query -h).
+
+
 DFTK FUNCTIONS (-f)
 -------------------
 A formula whose right-hand side is one call to a function below runs that

{dataframe_toolkit-0.11.2 → dataframe_toolkit-0.11.3}/src/dftk/commands/query_cmd.py

@@ -166,6 +166,28 @@ Window functions:
     "SELECT *, ROW_NUMBER() OVER (PARTITION BY cut ORDER BY price DESC) AS rank
      FROM data"
 
+Differences between rows: LAG() and LEAD() read a column in the previous
+or next row of the window, so one row's start can be compared with the
+previous row's end. Gaps between segments on each chromosome (negative =
+overlap; the first segment of each chromosome gets an empty cell):
+
+  dftk query segments.tsv --sql \\
+    'SELECT *, "Begin" - LAG("End") OVER (PARTITION BY Chr ORDER BY "Begin")
+       AS gap
+     FROM data ORDER BY Chr, "Begin"'
+
+  Begin and End are SQL keywords: put such column names in double quotes,
+  and the whole statement in single quotes. No sorting is needed first;
+  ORDER BY inside OVER sets the row order. The same on a bundled dataset,
+  change since the previous record and years between records per country:
+
+  dftk dataset healthexp -o | dftk query ... --sql \\
+    "SELECT Country, Year, Spending_USD,
+            ROUND(Spending_USD - LAG(Spending_USD) OVER w, 1) AS change,
+            Year - LAG(Year) OVER w AS years
+     FROM data WINDOW w AS (PARTITION BY Country ORDER BY Year)
+     ORDER BY Country, Year"
+
 CTEs:
 
   dftk dataset diamonds -o | dftk query ... --sql \\