dataframe-toolkit 0.11.1__tar.gz → 0.11.3__tar.gz

{dataframe_toolkit-0.11.1 → dataframe_toolkit-0.11.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: dataframe-toolkit
-Version: 0.11.1
+Version: 0.11.3
 Summary: A command-line toolkit for DataFrame analysis and manipulation
 Keywords: dataframe,statistics,data analysis,CLI,TSV,pandas
 Author: Gunnar Pálsson
@@ -20,6 +20,7 @@ Requires-Dist: statsmodels>=0.14
 Requires-Dist: patsy>=0.5.6
 Requires-Dist: duckdb>=0.8
 Requires-Dist: matplotlib>=3.7
+Requires-Dist: pillow>=13.0
 Requires-Dist: seaborn>=0.12
 Requires-Dist: tqdm>=4.65
 Requires-Python: >=3.12
@@ -226,35 +227,141 @@ printf "sample\tfluor\ns1\t60\ns2\t110\n" \
 
 ### Filling missing values
 
-`eval -f` takes pandas column methods (`COL.method(...)`), and `fillna`,
-`ffill`, `bfill` and `interpolate` fill empty cells. Defaults from another
-table, after a merge (a column in both files gets the suffix `_r`):
+`eval -f` takes pandas column methods (`COL.method(...)`); `fillna`,
+`ffill`, `bfill` and `interpolate` fill empty cells and keep the others.
+
+| Formula | Fills with |
+|---|---|
+| `x = x.fillna(y)` | the value in column `y` |
+| `x = x.fillna(0)`, `sex = sex.fillna('unknown')` | a fixed value |
+| `x = x.fillna(y).fillna(0)` | `y` first, then 0 |
+| `x = x.fillna(x.groupby(g).transform('mean'))` | the mean of the row's group (or `'median'`, ...) |
+| `x = x.ffill()` / `x = x.bfill()` | the value above / below (`ffill(limit=1)`: at most one row) |
+| `x = x.groupby(g).ffill()` | the value above, within each group of `g` |
+| `x = x.interpolate()` | a straight line between the values above and below, by row position |
+
+Write to a new column (`x_filled = x.fillna(y)`) to keep the original, and
+mark the gaps first (`missing = x.isna()`) to tell filled values apart.
+Only empty cells are filled: a missing row (a year with no row at all) is
+not added.
+
+**Defaults from another table.** A column that is in both files gets the
+suffix `_r` from `merge`; fill from it, then drop it. Here the defaults are
+each species' median body mass:
 
 ```bash
-dftk merge data.tsv -r defaults.tsv -k type -t left -o \
-  | dftk eval ... -f "depth = depth.fillna(depth_r)" --drop depth_r
+dftk dataset penguins -o | dftk pivot ... -i species -v body_mass_g \
+  -f median --move median_body_mass_g:body_mass_g -o defaults.parquet
+dftk dataset penguins -o \
+  | dftk merge ... -r defaults.parquet -k species -t left -o \
+  | dftk eval ... -f "body_mass_g = body_mass_g.fillna(body_mass_g_r)" \
+      --drop body_mass_g_r
 ```
 
-The group mean, a fixed value, or the last value above (`ffill`) within
-each group:
+**From neighbouring rows** (`co2` is a weekly series with gaps):
 
 ```bash
-dftk dataset penguins -o | dftk eval ... \
-  -f "mean = body_mass_g.groupby(species).transform('mean')" \
-  -f "body_mass_g = body_mass_g.fillna(mean)" --drop mean \
-  -f "sex = sex.fillna('unknown')"
+dftk dataset co2 -o | dftk eval ... -f "ffill = co2.ffill()" \
+  -f "bfill = co2.bfill()" -f "linear = co2.interpolate()"
+```
+
+`ffill` leaves the first rows empty and `bfill` the last ones;
+`x.ffill().fillna(x.bfill())` fills both. `interpolate()` counts one step
+per row, so the rows must be evenly spaced; for an x-based lookup from a
+reference curve, see `dftk interp`.
 
+**Row order and groups.** These methods use the rows in the order they are
+in. Sort first, in a step of its own: `--sortasc` applies to a command's
+output, so in the same command it would come after the fill. When a table
+holds several series (sites, countries, samples), fill within each group,
+or values are taken from the neighbouring group. Rows of a group don't need
+to be next to each other, but within each group they must be in order:
+
+```bash
 dftk dataset healthexp -o \
-  | dftk eval ... -f "le = Life_Expectancy.where(Year % 10 != 5)" -o \
+  | dftk eval ... -f "le = Life_Expectancy.where(Year % 10 != 5)" \
+      --sortasc Country Year -o \
   | dftk eval ... -f "le_filled = le.groupby(Country).ffill()"
 ```
 
-`ffill`, `bfill` and `interpolate` use the rows in the order they are in:
-sort in a step of its own first (`dftk eval data.tsv --sortasc site day -o
-| dftk eval ... -f ...`), and use `groupby(...)` when the table holds
-several series, or values are taken from the neighbouring group.
-`interpolate()` has no grouped form and works by row position. `dftk eval
--h` covers these cases, with examples.
+`interpolate()` has no grouped form. Once each group's rows are together,
+gaps inside a group are right, but a gap at a group's start or end is
+filled from the neighbouring group. To leave those empty, keep only values
+with something before and after them in the same group:
+
+```bash
+dftk dataset healthexp -o \
+  | dftk eval ... -f "le = Life_Expectancy.where(Year % 10 != 5)" \
+      --sortasc Country Year -o \
+  | dftk eval ... -f "before = le.groupby(Country).ffill().notna()" \
+      -f "after = le.groupby(Country).bfill().notna()" \
+      -f "le_linear = le.interpolate().where(before & after)" \
+      --drop before after
+```
+
+### Differences between rows
+
+For the change in a column from one row to the next, `eval -f` takes
+pandas `diff()` and `pct_change()`, also within groups. They use the rows
+in their order, so sort first, in a step of its own:
+
+```bash
+dftk dataset healthexp -o | dftk eval ... --sortasc Country Year -o \
+  | dftk eval ... -f "change = Spending_USD.groupby(Country).diff().round(1)" \
+      -f "pct = (Spending_USD.groupby(Country).pct_change() * 100).round(1)" \
+      -f "years = Year.groupby(Country).diff()"
+```
+
+```text
+Year  Country  Spending_USD  Life_Expectancy  change  pct   years
+1971  Canada   313.391       72.8
+1976  Canada   543.337       73.8             229.9   73.4  5.0
+1979  Canada   692.269       75.1             148.9   27.4  3.0
+1980  Canada   791.812       75.2             99.5    14.4  1.0
+```
+
+The first row of each country gets an empty cell. Without
+`groupby(Country)`, it would be compared with the last row of the
+previous country.
+
+To compare a value with one in another column of the previous or next row, for example the
+gap between segments given by `Chr`, `Begin`, `End`, use SQL's `LAG()` /
+`LEAD()` in `query --sql`:
+
+```bash
+dftk query segments.tsv --sql \
+  'SELECT *, "Begin" - LAG("End") OVER (PARTITION BY Chr ORDER BY "Begin")
+     AS gap
+   FROM data ORDER BY Chr, "Begin"'
+```
+
+```text
+Chr   Begin  End  Id  gap
+chr1  100    200  a
+chr1  250    400  b   50
+chr1  390    500  c   -10
+chr2  50     80   d
+chr2  100    300  e   20
+```
+
+A negative gap is an overlap, and the first segment on each chromosome
+gets an empty cell. `lead("Begin") OVER (...) - "End"` gives the gap to
+the next segment instead.
+
+- `Begin` and `End` are SQL keywords: quote such column names with double
+  quotes and the whole statement with single quotes.
+- No sorting is needed first; `ORDER BY` inside `OVER` sets the order.
+- Integers stay integers.
+
+`eval -f` can do the same with pandas `shift`, but it goes by row order, so
+sort by `Chr` and `Begin` first (results are floats):
+
+```bash
+dftk eval segments.tsv -f 'gap = Begin - End.groupby(Chr).shift()'
+```
+
+Without `groupby(Chr)` the first segment of a chromosome is compared with
+the last one of the previous chromosome.
 
 ### Bootstrap confidence intervals
 

{dataframe_toolkit-0.11.1 → dataframe_toolkit-0.11.3}/README.md

@@ -197,35 +197,141 @@ printf "sample\tfluor\ns1\t60\ns2\t110\n" \
 
 ### Filling missing values
 
-`eval -f` takes pandas column methods (`COL.method(...)`), and `fillna`,
-`ffill`, `bfill` and `interpolate` fill empty cells. Defaults from another
-table, after a merge (a column in both files gets the suffix `_r`):
+`eval -f` takes pandas column methods (`COL.method(...)`); `fillna`,
+`ffill`, `bfill` and `interpolate` fill empty cells and keep the others.
+
+| Formula | Fills with |
+|---|---|
+| `x = x.fillna(y)` | the value in column `y` |
+| `x = x.fillna(0)`, `sex = sex.fillna('unknown')` | a fixed value |
+| `x = x.fillna(y).fillna(0)` | `y` first, then 0 |
+| `x = x.fillna(x.groupby(g).transform('mean'))` | the mean of the row's group (or `'median'`, ...) |
+| `x = x.ffill()` / `x = x.bfill()` | the value above / below (`ffill(limit=1)`: at most one row) |
+| `x = x.groupby(g).ffill()` | the value above, within each group of `g` |
+| `x = x.interpolate()` | a straight line between the values above and below, by row position |
+
+Write to a new column (`x_filled = x.fillna(y)`) to keep the original, and
+mark the gaps first (`missing = x.isna()`) to tell filled values apart.
+Only empty cells are filled: a missing row (a year with no row at all) is
+not added.
+
+**Defaults from another table.** A column that is in both files gets the
+suffix `_r` from `merge`; fill from it, then drop it. Here the defaults are
+each species' median body mass:
 
 ```bash
-dftk merge data.tsv -r defaults.tsv -k type -t left -o \
-  | dftk eval ... -f "depth = depth.fillna(depth_r)" --drop depth_r
+dftk dataset penguins -o | dftk pivot ... -i species -v body_mass_g \
+  -f median --move median_body_mass_g:body_mass_g -o defaults.parquet
+dftk dataset penguins -o \
+  | dftk merge ... -r defaults.parquet -k species -t left -o \
+  | dftk eval ... -f "body_mass_g = body_mass_g.fillna(body_mass_g_r)" \
+      --drop body_mass_g_r
 ```
 
-The group mean, a fixed value, or the last value above (`ffill`) within
-each group:
+**From neighbouring rows** (`co2` is a weekly series with gaps):
 
 ```bash
-dftk dataset penguins -o | dftk eval ... \
-  -f "mean = body_mass_g.groupby(species).transform('mean')" \
-  -f "body_mass_g = body_mass_g.fillna(mean)" --drop mean \
-  -f "sex = sex.fillna('unknown')"
+dftk dataset co2 -o | dftk eval ... -f "ffill = co2.ffill()" \
+  -f "bfill = co2.bfill()" -f "linear = co2.interpolate()"
+```
+
+`ffill` leaves the first rows empty and `bfill` the last ones;
+`x.ffill().fillna(x.bfill())` fills both. `interpolate()` counts one step
+per row, so the rows must be evenly spaced; for an x-based lookup from a
+reference curve, see `dftk interp`.
 
+**Row order and groups.** These methods use the rows in the order they are
+in. Sort first, in a step of its own: `--sortasc` applies to a command's
+output, so in the same command it would come after the fill. When a table
+holds several series (sites, countries, samples), fill within each group,
+or values are taken from the neighbouring group. Rows of a group don't need
+to be next to each other, but within each group they must be in order:
+
+```bash
 dftk dataset healthexp -o \
-  | dftk eval ... -f "le = Life_Expectancy.where(Year % 10 != 5)" -o \
+  | dftk eval ... -f "le = Life_Expectancy.where(Year % 10 != 5)" \
+      --sortasc Country Year -o \
   | dftk eval ... -f "le_filled = le.groupby(Country).ffill()"
 ```
 
-`ffill`, `bfill` and `interpolate` use the rows in the order they are in:
-sort in a step of its own first (`dftk eval data.tsv --sortasc site day -o
-| dftk eval ... -f ...`), and use `groupby(...)` when the table holds
-several series, or values are taken from the neighbouring group.
-`interpolate()` has no grouped form and works by row position. `dftk eval
--h` covers these cases, with examples.
+`interpolate()` has no grouped form. Once each group's rows are together,
+gaps inside a group are right, but a gap at a group's start or end is
+filled from the neighbouring group. To leave those empty, keep only values
+with something before and after them in the same group:
+
+```bash
+dftk dataset healthexp -o \
+  | dftk eval ... -f "le = Life_Expectancy.where(Year % 10 != 5)" \
+      --sortasc Country Year -o \
+  | dftk eval ... -f "before = le.groupby(Country).ffill().notna()" \
+      -f "after = le.groupby(Country).bfill().notna()" \
+      -f "le_linear = le.interpolate().where(before & after)" \
+      --drop before after
+```
+
+### Differences between rows
+
+For the change in a column from one row to the next, `eval -f` takes
+pandas `diff()` and `pct_change()`, also within groups. They use the rows
+in their order, so sort first, in a step of its own:
+
+```bash
+dftk dataset healthexp -o | dftk eval ... --sortasc Country Year -o \
+  | dftk eval ... -f "change = Spending_USD.groupby(Country).diff().round(1)" \
+      -f "pct = (Spending_USD.groupby(Country).pct_change() * 100).round(1)" \
+      -f "years = Year.groupby(Country).diff()"
+```
+
+```text
+Year  Country  Spending_USD  Life_Expectancy  change  pct   years
+1971  Canada   313.391       72.8
+1976  Canada   543.337       73.8             229.9   73.4  5.0
+1979  Canada   692.269       75.1             148.9   27.4  3.0
+1980  Canada   791.812       75.2             99.5    14.4  1.0
+```
+
+The first row of each country gets an empty cell. Without
+`groupby(Country)`, it would be compared with the last row of the
+previous country.
+
+To compare a value with one in another column of the previous or next row, for example the
+gap between segments given by `Chr`, `Begin`, `End`, use SQL's `LAG()` /
+`LEAD()` in `query --sql`:
+
+```bash
+dftk query segments.tsv --sql \
+  'SELECT *, "Begin" - LAG("End") OVER (PARTITION BY Chr ORDER BY "Begin")
+     AS gap
+   FROM data ORDER BY Chr, "Begin"'
+```
+
+```text
+Chr   Begin  End  Id  gap
+chr1  100    200  a
+chr1  250    400  b   50
+chr1  390    500  c   -10
+chr2  50     80   d
+chr2  100    300  e   20
+```
+
+A negative gap is an overlap, and the first segment on each chromosome
+gets an empty cell. `lead("Begin") OVER (...) - "End"` gives the gap to
+the next segment instead.
+
+- `Begin` and `End` are SQL keywords: quote such column names with double
+  quotes and the whole statement with single quotes.
+- No sorting is needed first; `ORDER BY` inside `OVER` sets the order.
+- Integers stay integers.
+
+`eval -f` can do the same with pandas `shift`, but it goes by row order, so
+sort by `Chr` and `Begin` first (results are floats):
+
+```bash
+dftk eval segments.tsv -f 'gap = Begin - End.groupby(Chr).shift()'
+```
+
+Without `groupby(Chr)` the first segment of a chromosome is compared with
+the last one of the previous chromosome.
 
 ### Bootstrap confidence intervals
 

{dataframe_toolkit-0.11.1 → dataframe_toolkit-0.11.3}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.11.1"
+version = "0.11.3"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 keywords = [
@@ -29,6 +29,7 @@ dependencies = [
     "patsy>=0.5.6",
     "duckdb>=0.8",
     "matplotlib>=3.7",
+    "pillow>=13.0",
     "seaborn>=0.12",
     "tqdm>=4.65",
 ]

{dataframe_toolkit-0.11.1 → dataframe_toolkit-0.11.3}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "dataframe-toolkit"
-version = "0.11.1"
+version = "0.11.3"
 description = "A command-line toolkit for DataFrame analysis and manipulation"
 readme = "README.md"
 authors = [
@@ -26,6 +26,7 @@ dependencies = [
     "patsy>=0.5.6",
     "duckdb>=0.8",
     "matplotlib>=3.7",
+    "pillow>=13.0",  # not used directly (matplotlib needs it); floor for security fixes
     "seaborn>=0.12",
     "tqdm>=4.65",
 ]