py-tbparse 0.3.0__tar.gz → 0.4.0__tar.gz

{py_tbparse-0.3.0 → py_tbparse-0.4.0}/AGENTS.md

@@ -14,14 +14,15 @@ type="join">` and 2020.2+ `<relationships>`), inferred relationships,
 dashboards/dashboard-sheets, `validate_relationships`, custom/initial SQL,
 and published-source detection. **Not yet ported** (v2): formatting,
 tooltips, colors, axes, sorts, dashboard layout/actions, analytics
-helpers (calc complexity, field usage, replication brief), and the
+helpers (calc complexity, replication brief), and the
 Shiny-inspector equivalent.
 
 Also added, with no R equivalent (Python-native extras -- see "Non-R
 modules" below): Graphviz DOT export of the relationship graph
 (replaces the R package's igraph/ggraph-based plotting with a
 dependency-free alternative), workbook-to-workbook diff, folder/batch
-analysis across many workbooks, and Jupyter rich display.
+analysis across many workbooks, field renaming, field usage, workbook
+templates, and Jupyter rich display.
 
 ## Setup
 
@@ -120,11 +121,13 @@ beyond the R package's scope:
 | Python module | What it is |
 |---|---|
 | `_tables.py` | Name → `TwbParser`-accessor registry shared by `cli.py`, `webgui.py`, `diff.py`, and `batch.py`. Adding a new extractor to `TwbParser`? Add it here too so it's automatically available everywhere else. |
-| `cli.py` | `py-tbparse` command-line entry point, plus the `diff`/`batch`/`rename` subcommands (dispatched on `sys.argv[1]` before the normal single-workbook argparse parser runs) |
+| `cli.py` | `py-tbparse` command-line entry point, plus the `diff`/`batch`/`rename`/`template` subcommands (dispatched on `sys.argv[1]` before the normal single-workbook argparse parser runs) |
 | `webgui.py` | `py-tbparse-gui`: stdlib-only (`http.server` + vanilla JS) local browser GUI, no GUI toolkit dependency. Loosely fills the role of the R package's `run_twbparser_app`/Shiny inspector. |
 | `graph.py` | `to_dot()`: Graphviz DOT export of joins/relationships (+ optional inferred, as dashed edges). Replaces the R package's igraph/ggraph-based `plot_dependency_graph`/`plot_relationship_graph` with a dependency-free text format any Graphviz-compatible tool can render. |
 | `diff.py` | `diff_tables()`/`diff_workbooks()`: row-level added/removed diff between two workbooks' same-named table, via `_tables.TABLE_SPECS`. No "changed" classification without a natural key — a changed row shows as one removed + one added row. |
-| `rename.py` | `suggest_field_renames()` (clean-name suggestions, optionally matched against a "before" reference), `load_rename_mapping()` (read an edited CSV back), `compare_field_schemas()` (fields with no counterpart across a datasource switch) and `apply_field_renames()`/`build_renamed_workbook()` (write a copy with captions set; never overwrites). Also the `field-renames` table in `_tables.py`. |
+| `rename.py` | `suggest_field_renames()` (clean-name suggestions, optionally matched against a "before" reference), `suggest_renames()` (the same for every kind of object: field, parameter, worksheet, dashboard, datasource, folder, hierarchy; adds a `kind` column), `load_rename_mapping()` (read an edited CSV back), `compare_field_schemas()` (fields with no counterpart across a datasource switch) and `apply_field_renames()`/`build_renamed_workbook()` (write a copy with captions set; never overwrites). Also the `field-renames` and `report-renames` tables in `_tables.py`. A worksheet/dashboard rename must rewrite every reference (`_SHEET_REFERENCES`); if you learn of another place Tableau writes a sheet name, add it there. |
+| `usage.py` | `field_usage()`: for every field, the worksheets, dashboards and calculations that use it, followed through calculations, groups/sets and bins (the `field-usage` table). Python-native rather than a port of the R package's field-usage helper. |
+| `templates.py` | Workbook templates. `make_template()` writes a `.twbx` with a `template.json` manifest (required fields from `usage.py`, parameters, connections; data, extracts and credentials stripped); `read_data()` describes new data (CSV, or a `.twb`/`.twbx`/`.tds`); `suggest_mapping()` pairs fields with columns (reuses `rename._match_key`, type checks against the field's *physical* type, `datatype-customized` fields keep their type); `apply_template()` replaces the datasource's connection, keeping every field's local name, sets parameters and stores `template-answers.json`. A CSV connection is written in the 2020.2+ object-model shape (both `_.fcp.ObjectModelEncapsulateLegacy` relations, object graph, table column, `object-id` per record) when the template uses it -- keep those in step if you touch one. Untested in Tableau itself. |
 | `batch.py` | `scan_folder()`: runs one table across every `.twb`/`.twbx` in a directory, concatenated with a `workbook` column. Skips (with a warning) any file that fails to load/extract rather than aborting the batch. |
 
 ## Porting conventions (read before adding/modifying a function)
@@ -174,6 +177,12 @@ beyond the R package's scope:
   synthetic XML snippet (see `tests/test_joins.py`,
   `tests/test_dashboards.py` for the pattern — many are lifted from the R
   functions' own `@examples` roxygen blocks).
+- `tests/corpus/` is 200 real workbooks (permissive licences, pinned by blob sha in
+  `manifest.csv`, licence texts in `licenses/`). The files are gitignored; fetch with
+  `python scripts/fetch_corpus.py`. `tests/test_corpus.py` skips without them. Run it when you
+  change anything that reads workbook XML: it found a Unicode matching bug the hand-made
+  fixtures could not. Add to the corpus only from repositories whose licence permits
+  redistribution, and record the licence text.
 - `tests/conftest.py` provides `wenjie_xml`, `wenjie_path`,
   `zip_twbx_path` fixtures and an `xml_from_string()` helper. Tests import
   it with `from conftest import xml_from_string` (no `tests/__init__.py`,

{py_tbparse-0.3.0/py_tbparse.egg-info → py_tbparse-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: py-tbparse
-Version: 0.3.0
+Version: 0.4.0
 Summary: Native Python port of the twbparser R package: parse Tableau .twb/.twbx workbooks into pandas DataFrames.
 Author: DDSNA
 License: MIT
@@ -101,6 +101,24 @@ With a `reference` (a workbook, a fields table or a plain list of names), fields
 
 `p.get_field_renames()` does the same from a parser.
 
+**Rename everything in the report, not just fields.** Pass `kinds` (or `--all` / `--kinds` on the command line) and worksheets, dashboards, datasources, parameters, folders and hierarchies are covered too:
+
+```python
+from py_tbparse import TwbParser, suggest_renames, apply_field_renames
+
+p = TwbParser("report.twb")
+suggest_renames(p, only_changed=True)                    # kind, datasource, name, current, suggested, ...
+suggest_renames(p, kinds=["worksheet", "dashboard"])     # just the sheets
+apply_field_renames(p, kinds="all")                      # writes report_renamed.twb
+```
+
+```bash
+py-tbparse rename report.twb --all --only-changed
+py-tbparse rename report.twb --kinds worksheet,dashboard --write-workbook
+```
+
+Each kind is renamed the way Tableau does it: fields, parameters and datasources get a caption (their internal names stay, so formulas and sheets keep working); a worksheet or dashboard is renamed in every place its name is written (the sheet, its window and thumbnail, the zones of dashboards that show it, actions, story points); a folder or hierarchy gets its new name. Worksheets and dashboards share one namespace, as they do in Tableau, so two of them never end up with the same name. A `reference` workbook lends its spelling to objects of the same kind. Datasources that Tableau named itself (`federated.0grg...`) and nobody captioned are left out. The `kind` column also appears in the CSV, so **Edit the suggestions yourself** works for sheets too. The `report-renames` table lists all of it, and the GUI's Field renames view has an "Everything in the report" switch. As with fields, none of this has been opened in Tableau itself.
+
 **Edit the suggestions yourself.** Export them, change the `suggested` column in a spreadsheet (blank means leave the field alone), then apply your version to a copy of the workbook:
 
 ```bash
@@ -116,6 +134,37 @@ To save the result, `p.write_renamed_workbook()` (or `apply_field_renames(p, ...
 
 **When to run it.** Add the new datasource to a *copy* of the workbook first, then run this with the old workbook as `reference` and `datasource=` set to the new source, so only its fields are renamed. Open the fixed copy and use Replace Data Source; fields with matching names should re-link on their own. It also works after references have already broken, but it only fixes names: sheets that point at missing fields stay broken until you replace the source again. (Check this on a copy first; I have not tested the re-linking in Tableau itself.)
 
+### Templates
+
+Turn a finished workbook into a template, then make new workbooks from it with other data. Every sheet, dashboard, calculation and format comes along; only the data changes. The idea comes from Tableau's Accelerators and Power BI's `.pbit` files.
+
+```bash
+py-tbparse template make sales.twbx                         # writes sales.template.twbx
+py-tbparse template show sales.template.twbx                # the fields it needs, and its parameters
+py-tbparse template apply sales.template.twbx --data q3.csv # suggested mapping + what would break; writes nothing
+py-tbparse template apply sales.template.twbx --data q3.csv --mapping-out map.csv   # save the mapping to edit
+py-tbparse template apply sales.template.twbx --data q3.csv --mapping map.csv -p "Top N=10" --write
+```
+
+```python
+from py_tbparse import make_template, load_template, read_data, suggest_mapping, apply_template
+
+t = load_template(make_template("sales.twbx"))
+data = read_data("q3.csv")                      # or a .twb / .twbx / .tds already connected to the new data
+suggest_mapping(t, data)                        # field, required, used_by, mapped_to, status, ...
+apply_template(t, data, params={"Top N": "10"}) # writes sales_q3.twbx
+```
+
+**What a template is.** An ordinary `.twbx` (Tableau still opens it) with a `template.json` manifest inside. The manifest lists the fields the workbook takes from its data, marking a field `required` when a sheet uses it, directly or through calculations, groups and sets. It also lists the parameters and where the data came from. Extracts, packaged data and cached query results are left out (`--keep-data` keeps them as sample data), and user names and passwords are blanked.
+
+**Mapping.** Each required field is matched to a column of the new data by name, ignoring case and separators (`ORDER_DATE` → `Order Date`), with close spellings accepted above `--cutoff`. Types are checked like Tableau's Accelerator mapper: a text column is never offered for a number or a date, while integer vs decimal and date vs date-time map with a warning. A field whose type the author changed in Tableau keeps that type, and Tableau converts the column. Before anything is written you see which sheets would break for each field left without a column; writing then needs `--allow-missing`. Edit the mapping as a CSV, as with renames.
+
+**What gets written.** The template's connection is replaced by one to the new data (a CSV file, or the connection of the workbook / `.tds` you pass). Every field keeps the local name its sheets and formulas use; only the physical column behind it changes. Parameter values are set with `-p NAME=VALUE`, checked against the parameter's type and its list of allowed values. The output (`<template>_<data>.twbx` beside the template, never overwritten) also stores `template-answers.json`: which template, data, mapping and parameters made it, so it can be re-made or checked later.
+
+**Limits.** A CSV feeds one table. A template whose datasource joins several tables needs a workbook or `.tds` as its data, so the joins come along. Excel files are not read directly yet; save as CSV or pass a workbook connected to the sheet. As with renames, I have not opened the generated workbooks in Tableau itself, so check one before relying on it.
+
+`p.get_field_usage()` (or `field_usage(p)`, the `field-usage` table) is the analysis behind `required`: for every field, the sheets, dashboards and calculations that use it.
+
 ### `.twbx` files
 
 A `.twbx` is read directly from the zip and nothing gets written to disk. That means `p.twbx_dir` is `None`, and `p.path` is a made-up `<file>.twbx/<name>.twb` that you can't open. If you want the files out, extract them yourself:
@@ -141,11 +190,14 @@ py-tbparse diff old.twb new.twb datasources
 py-tbparse batch ./workbooks datasources
 py-tbparse rename new.twb --reference old.twb --only-changed   # suggested clean field names
 py-tbparse rename new.twb -r old.twb --datasource federated.abc123 --write-workbook   # and save new_renamed.twb
+py-tbparse rename report.twb --all --write-workbook       # sheets, dashboards, datasources, ... too
+py-tbparse template make sales.twbx                        # see Templates above
+py-tbparse template apply sales.template.twbx --data q3.csv --write
 ```
 
-Tables: `overview`, `datasources`, `parameters`, `fields`, `raw-fields`, `calculated-fields`, `joins`, `relations`, `relationships`, `inferred-relationships`, `dashboards`, `dashboard-sheets`, `custom-sql`, `initial-sql`, `published-refs`.
+Tables: `overview`, `datasources`, `parameters`, `fields`, `raw-fields`, `calculated-fields`, `joins`, `relations`, `relationships`, `inferred-relationships`, `dashboards`, `dashboard-sheets`, `custom-sql`, `initial-sql`, `published-refs`, `field-usage`, `field-renames`, `report-renames`.
 
-`--format` takes `table` (default), `csv` or `json`. `graph` always prints Graphviz text. `rename` takes `--reference`, `--datasource`, `--write-workbook [PATH]`, `--style`, `--cutoff`, `--only-changed`, `--apply MAPPING.csv`, `--missing`, `--format` and `--output`. `diff` and `batch` accept the same table names except `graph`, `validate` and `tables`.
+`--format` takes `table` (default), `csv` or `json`. `graph` always prints Graphviz text. `rename` takes `--reference`, `--datasource`, `--write-workbook [PATH]`, `--style`, `--cutoff`, `--only-changed`, `--all`, `--kinds`, `--apply MAPPING.csv`, `--missing`, `--format` and `--output`. `diff` and `batch` accept the same table names except `graph`, `validate` and `tables`.
 
 ## GUI
 
@@ -172,6 +224,8 @@ pytest
 
 The sample workbooks in `tests/fixtures/` come from the R package. `tests/fixtures/public/` holds real workbooks from Tableau's own [document-api-python](https://github.com/tableau/document-api-python) (MIT), used by the smoke tests.
 
+`tests/corpus/` lists 200 more real workbooks from public repositories with MIT, Apache-2.0, ISC or CC0 licences, for integration tests and as examples (manifest, licence texts and where each file came from are in its README). The files themselves are not in git (about 26 MB): run `python scripts/fetch_corpus.py` to download them, checked against the manifest. `tests/test_corpus.py` then runs every feature over all of them; it skips when they are not fetched.
+
 The GUI tests run the page in headless Chromium through Playwright and fail on any JavaScript error. They skip if the browser isn't installed. To run them:
 
 ```bash

{py_tbparse-0.3.0 → py_tbparse-0.4.0}/README.md

@@ -65,6 +65,24 @@ With a `reference` (a workbook, a fields table or a plain list of names), fields
 
 `p.get_field_renames()` does the same from a parser.
 
+**Rename everything in the report, not just fields.** Pass `kinds` (or `--all` / `--kinds` on the command line) and worksheets, dashboards, datasources, parameters, folders and hierarchies are covered too:
+
+```python
+from py_tbparse import TwbParser, suggest_renames, apply_field_renames
+
+p = TwbParser("report.twb")
+suggest_renames(p, only_changed=True)                    # kind, datasource, name, current, suggested, ...
+suggest_renames(p, kinds=["worksheet", "dashboard"])     # just the sheets
+apply_field_renames(p, kinds="all")                      # writes report_renamed.twb
+```
+
+```bash
+py-tbparse rename report.twb --all --only-changed
+py-tbparse rename report.twb --kinds worksheet,dashboard --write-workbook
+```
+
+Each kind is renamed the way Tableau does it: fields, parameters and datasources get a caption (their internal names stay, so formulas and sheets keep working); a worksheet or dashboard is renamed in every place its name is written (the sheet, its window and thumbnail, the zones of dashboards that show it, actions, story points); a folder or hierarchy gets its new name. Worksheets and dashboards share one namespace, as they do in Tableau, so two of them never end up with the same name. A `reference` workbook lends its spelling to objects of the same kind. Datasources that Tableau named itself (`federated.0grg...`) and nobody captioned are left out. The `kind` column also appears in the CSV, so **Edit the suggestions yourself** works for sheets too. The `report-renames` table lists all of it, and the GUI's Field renames view has an "Everything in the report" switch. As with fields, none of this has been opened in Tableau itself.
+
 **Edit the suggestions yourself.** Export them, change the `suggested` column in a spreadsheet (blank means leave the field alone), then apply your version to a copy of the workbook:
 
 ```bash
@@ -80,6 +98,37 @@ To save the result, `p.write_renamed_workbook()` (or `apply_field_renames(p, ...
 
 **When to run it.** Add the new datasource to a *copy* of the workbook first, then run this with the old workbook as `reference` and `datasource=` set to the new source, so only its fields are renamed. Open the fixed copy and use Replace Data Source; fields with matching names should re-link on their own. It also works after references have already broken, but it only fixes names: sheets that point at missing fields stay broken until you replace the source again. (Check this on a copy first; I have not tested the re-linking in Tableau itself.)
 
+### Templates
+
+Turn a finished workbook into a template, then make new workbooks from it with other data. Every sheet, dashboard, calculation and format comes along; only the data changes. The idea comes from Tableau's Accelerators and Power BI's `.pbit` files.
+
+```bash
+py-tbparse template make sales.twbx                         # writes sales.template.twbx
+py-tbparse template show sales.template.twbx                # the fields it needs, and its parameters
+py-tbparse template apply sales.template.twbx --data q3.csv # suggested mapping + what would break; writes nothing
+py-tbparse template apply sales.template.twbx --data q3.csv --mapping-out map.csv   # save the mapping to edit
+py-tbparse template apply sales.template.twbx --data q3.csv --mapping map.csv -p "Top N=10" --write
+```
+
+```python
+from py_tbparse import make_template, load_template, read_data, suggest_mapping, apply_template
+
+t = load_template(make_template("sales.twbx"))
+data = read_data("q3.csv")                      # or a .twb / .twbx / .tds already connected to the new data
+suggest_mapping(t, data)                        # field, required, used_by, mapped_to, status, ...
+apply_template(t, data, params={"Top N": "10"}) # writes sales_q3.twbx
+```
+
+**What a template is.** An ordinary `.twbx` (Tableau still opens it) with a `template.json` manifest inside. The manifest lists the fields the workbook takes from its data, marking a field `required` when a sheet uses it, directly or through calculations, groups and sets. It also lists the parameters and where the data came from. Extracts, packaged data and cached query results are left out (`--keep-data` keeps them as sample data), and user names and passwords are blanked.
+
+**Mapping.** Each required field is matched to a column of the new data by name, ignoring case and separators (`ORDER_DATE` → `Order Date`), with close spellings accepted above `--cutoff`. Types are checked like Tableau's Accelerator mapper: a text column is never offered for a number or a date, while integer vs decimal and date vs date-time map with a warning. A field whose type the author changed in Tableau keeps that type, and Tableau converts the column. Before anything is written you see which sheets would break for each field left without a column; writing then needs `--allow-missing`. Edit the mapping as a CSV, as with renames.
+
+**What gets written.** The template's connection is replaced by one to the new data (a CSV file, or the connection of the workbook / `.tds` you pass). Every field keeps the local name its sheets and formulas use; only the physical column behind it changes. Parameter values are set with `-p NAME=VALUE`, checked against the parameter's type and its list of allowed values. The output (`<template>_<data>.twbx` beside the template, never overwritten) also stores `template-answers.json`: which template, data, mapping and parameters made it, so it can be re-made or checked later.
+
+**Limits.** A CSV feeds one table. A template whose datasource joins several tables needs a workbook or `.tds` as its data, so the joins come along. Excel files are not read directly yet; save as CSV or pass a workbook connected to the sheet. As with renames, I have not opened the generated workbooks in Tableau itself, so check one before relying on it.
+
+`p.get_field_usage()` (or `field_usage(p)`, the `field-usage` table) is the analysis behind `required`: for every field, the sheets, dashboards and calculations that use it.
+
 ### `.twbx` files
 
 A `.twbx` is read directly from the zip and nothing gets written to disk. That means `p.twbx_dir` is `None`, and `p.path` is a made-up `<file>.twbx/<name>.twb` that you can't open. If you want the files out, extract them yourself:
@@ -105,11 +154,14 @@ py-tbparse diff old.twb new.twb datasources
 py-tbparse batch ./workbooks datasources
 py-tbparse rename new.twb --reference old.twb --only-changed   # suggested clean field names
 py-tbparse rename new.twb -r old.twb --datasource federated.abc123 --write-workbook   # and save new_renamed.twb
+py-tbparse rename report.twb --all --write-workbook       # sheets, dashboards, datasources, ... too
+py-tbparse template make sales.twbx                        # see Templates above
+py-tbparse template apply sales.template.twbx --data q3.csv --write
 ```
 
-Tables: `overview`, `datasources`, `parameters`, `fields`, `raw-fields`, `calculated-fields`, `joins`, `relations`, `relationships`, `inferred-relationships`, `dashboards`, `dashboard-sheets`, `custom-sql`, `initial-sql`, `published-refs`.
+Tables: `overview`, `datasources`, `parameters`, `fields`, `raw-fields`, `calculated-fields`, `joins`, `relations`, `relationships`, `inferred-relationships`, `dashboards`, `dashboard-sheets`, `custom-sql`, `initial-sql`, `published-refs`, `field-usage`, `field-renames`, `report-renames`.
 
-`--format` takes `table` (default), `csv` or `json`. `graph` always prints Graphviz text. `rename` takes `--reference`, `--datasource`, `--write-workbook [PATH]`, `--style`, `--cutoff`, `--only-changed`, `--apply MAPPING.csv`, `--missing`, `--format` and `--output`. `diff` and `batch` accept the same table names except `graph`, `validate` and `tables`.
+`--format` takes `table` (default), `csv` or `json`. `graph` always prints Graphviz text. `rename` takes `--reference`, `--datasource`, `--write-workbook [PATH]`, `--style`, `--cutoff`, `--only-changed`, `--all`, `--kinds`, `--apply MAPPING.csv`, `--missing`, `--format` and `--output`. `diff` and `batch` accept the same table names except `graph`, `validate` and `tables`.
 
 ## GUI
 
@@ -136,6 +188,8 @@ pytest
 
 The sample workbooks in `tests/fixtures/` come from the R package. `tests/fixtures/public/` holds real workbooks from Tableau's own [document-api-python](https://github.com/tableau/document-api-python) (MIT), used by the smoke tests.
 
+`tests/corpus/` lists 200 more real workbooks from public repositories with MIT, Apache-2.0, ISC or CC0 licences, for integration tests and as examples (manifest, licence texts and where each file came from are in its README). The files themselves are not in git (about 26 MB): run `python scripts/fetch_corpus.py` to download them, checked against the manifest. `tests/test_corpus.py` then runs every feature over all of them; it skips when they are not fetched.
+
 The GUI tests run the page in headless Chromium through Playwright and fail on any JavaScript error. They skip if the browser isn't installed. To run them:
 
 ```bash

{py_tbparse-0.3.0 → py_tbparse-0.4.0}/py_tbparse/__init__.py

@@ -21,8 +21,20 @@ from .rename import (
     load_rename_mapping,
     normalize_name,
     suggest_field_renames,
+    suggest_renames,
 )
 from .sql import extract_custom_sql, extract_initial_sql
+from .templates import (
+    Template,
+    TemplateError,
+    apply_template,
+    load_mapping,
+    load_template,
+    make_template,
+    read_data,
+    suggest_mapping,
+)
+from .usage import field_usage
 from .validators import validate_relationships
 from ._xml import extract_twb_from_twbx, twbx_extract_files, twbx_list
 
@@ -56,6 +68,16 @@ __all__ = [
     "load_rename_mapping",
     "normalize_name",
     "suggest_field_renames",
+    "suggest_renames",
+    "field_usage",
+    "Template",
+    "TemplateError",
+    "make_template",
+    "load_template",
+    "read_data",
+    "suggest_mapping",
+    "load_mapping",
+    "apply_template",
 ]
 
 try:

{py_tbparse-0.3.0 → py_tbparse-0.4.0}/py_tbparse/_tables.py

@@ -39,12 +39,28 @@ def _calculated_fields(p: TwbParser, include_parameters: bool = False, **_kw) ->
 
 
 def _field_renames(p: TwbParser, style: str = "title", reference=None, only_changed: bool = False,
-                  datasource=None, **_kw) -> pd.DataFrame:
+                  datasource=None, kinds=None, **_kw) -> pd.DataFrame:
+    if kinds:  # the GUI's "everything in the report" switch
+        return p.get_renames(
+            reference=reference, style=style, only_changed=only_changed, datasource=datasource, kinds=kinds
+        )
     return p.get_field_renames(
         reference=reference, style=style, only_changed=only_changed, datasource=datasource
     )
 
 
+def _report_renames(p: TwbParser, style: str = "title", reference=None, only_changed: bool = False,
+                    datasource=None, **_kw) -> pd.DataFrame:
+    return p.get_renames(reference=reference, style=style, only_changed=only_changed, datasource=datasource)
+
+
+def _field_usage(p: TwbParser, **_kw) -> pd.DataFrame:
+    df = p.get_field_usage()
+    for col in ("sheets", "dashboards", "calculations"):
+        df[col] = df[col].map("; ".join)
+    return df
+
+
 def _joins(p: TwbParser, **_kw) -> pd.DataFrame:
     return p.get_joins()
 
@@ -91,6 +107,8 @@ TABLE_SPECS: dict[str, Callable[..., pd.DataFrame]] = {
     "raw-fields": _raw_fields,
     "calculated-fields": _calculated_fields,
     "field-renames": _field_renames,
+    "report-renames": _report_renames,
+    "field-usage": _field_usage,
     "joins": _joins,
     "relations": _relations,
     "relationships": _relationships,

{py_tbparse-0.3.0 → py_tbparse-0.4.0}/py_tbparse/cli.py

@@ -10,20 +10,33 @@ from __future__ import annotations
 import argparse
 import json
 import sys
+import zipfile
 from pathlib import Path
 
 import pandas as pd
+from lxml import etree
 
 from ._tables import TABLE_NAMES, TABLE_SPECS
 from .batch import scan_folder
 from .diff import diff_workbooks
 from .parser import TwbParser
+from .templates import (
+    apply_template,
+    broken_sheets,
+    load_mapping,
+    load_template,
+    make_template,
+    read_data,
+    suggest_mapping,
+)
 from .rename import (
     STYLES,
     apply_field_renames,
     compare_field_schemas,
+    KINDS,
     load_rename_mapping,
     suggest_field_renames,
+    suggest_renames,
 )
 
 
@@ -165,6 +178,11 @@ def build_rename_arg_parser() -> argparse.ArgumentParser:
         "--cutoff", type=float, default=0.85,
         help="0..1 similarity needed to match a reference name approximately (default: 0.85)",
     )
+    ap.add_argument(
+        "--kinds", metavar="KIND[,KIND...]",
+        help="rename more than fields: any of " + ", ".join(KINDS) + ", or 'all' (default: field)",
+    )
+    ap.add_argument("--all", action="store_true", help="shorthand for --kinds all")
     ap.add_argument("--only-changed", action="store_true", help="hide fields that need no rename")
     ap.add_argument("--datasource", help="only this datasource (its internal name), e.g. the newly added one")
     ap.add_argument(
@@ -227,8 +245,13 @@ def _run_rename(argv: list[str]) -> int:
         return 0
 
     kwargs = dict(reference=ref, style=args.style, fuzzy_cutoff=args.cutoff, datasource=args.datasource)
+    kinds = None
+    if args.all or args.kinds:
+        kinds = KINDS if args.all or args.kinds.strip() == "all" else tuple(
+            k.strip() for k in args.kinds.split(",") if k.strip()
+        )
     try:
-        everything = suggest_field_renames(wb, **kwargs)
+        everything = suggest_renames(wb, kinds=kinds, **kwargs) if kinds else suggest_field_renames(wb, **kwargs)
     except ValueError as e:  # e.g. --cutoff outside 0..1
         print(f"error: {e}", file=sys.stderr)
         return 1
@@ -246,6 +269,104 @@ def _run_rename(argv: list[str]) -> int:
     return 0
 
 
+def build_template_arg_parser() -> argparse.ArgumentParser:
+    ap = argparse.ArgumentParser(
+        prog="py-tbparse template",
+        description="Make a reusable template from a workbook, or apply one to new data.",
+    )
+    sub = ap.add_subparsers(dest="action", required=True)
+
+    mk = sub.add_parser("make", help="save WORKBOOK as a template (.twbx with a template.json manifest)")
+    mk.add_argument("workbook")
+    mk.add_argument("--output", "-o", help="template file (default: <name>.template.twbx beside the workbook)")
+    mk.add_argument("--name", help="template name (default: the workbook's)")
+    mk.add_argument("--description", help="what the template is for")
+    mk.add_argument("--keep-data", action="store_true", help="keep extracts and packaged data as sample data")
+
+    sh = sub.add_parser("show", help="list the fields and parameters a template needs")
+    sh.add_argument("template")
+    sh.add_argument("--format", "-f", choices=["table", "csv", "json"], default="table")
+
+    ap_ = sub.add_parser(
+        "apply", help="map a template's fields to new data; with --write, make the workbook",
+        description="Without --write this only prints the suggested mapping and what would break.",
+    )
+    ap_.add_argument("template")
+    ap_.add_argument("--data", "-d", required=True, help="the new data: a .csv, or a .twb/.twbx/.tds connected to it")
+    ap_.add_argument("--datasource", help="which template datasource to fill (when it has several)")
+    ap_.add_argument("--data-datasource", help="which datasource of a --data workbook to use")
+    ap_.add_argument("--mapping", "-m", help="use this edited mapping CSV instead of the suggestion")
+    ap_.add_argument("--mapping-out", help="save the mapping as CSV, to edit and pass back with --mapping")
+    ap_.add_argument("--param", "-p", action="append", default=[], metavar="NAME=VALUE",
+                     help="set a parameter (repeatable), e.g. -p 'Top N=10'")
+    ap_.add_argument("--cutoff", type=float, default=0.85, help="0..1 similarity for approximate matches")
+    ap_.add_argument("--allow-missing", action="store_true",
+                     help="write even if required fields have no column (their sheets will break)")
+    ap_.add_argument(
+        "--write", "-w", nargs="?", const="", metavar="PATH",
+        help="make the workbook (default PATH: <template>_<data>.twbx beside the template; never overwrites)",
+    )
+    ap_.add_argument("--format", "-f", choices=["table", "csv", "json"], default="table")
+    return ap
+
+
+def _run_template(argv: list[str]) -> int:
+    ap = build_template_arg_parser()
+    args = ap.parse_args(argv)
+    try:
+        if args.action == "make":
+            out = make_template(args.workbook, output_path=args.output, name=args.name,
+                                description=args.description, keep_data=args.keep_data)
+            t = load_template(out)
+            req = int(t.fields()["required"].sum())
+            print(f"wrote {out} ({req} required field(s), {len(t.parameters())} parameter(s))", file=sys.stderr)
+            return 0
+
+        t = load_template(args.template)
+        if args.action == "show":
+            if t.manifest.get("description"):
+                print(t.manifest["description"], file=sys.stderr)
+            _write(_df_text(t.fields(), args.format), None)
+            if args.format == "table" and len(t.parameters()):
+                print("\nParameters:")
+                _write(_df_text(t.parameters(), args.format), None)
+            return 0
+
+        params = {}
+        for item in args.param:
+            if "=" not in item:
+                ap.error(f"--param expects NAME=VALUE, got {item!r}")
+            k, v = item.split("=", 1)
+            params[k.strip()] = v
+        data = read_data(args.data, datasource=args.data_datasource)
+        if args.mapping:
+            mapping = load_mapping(args.mapping)
+        else:
+            mapping = suggest_mapping(t, data, datasource=args.datasource, fuzzy_cutoff=args.cutoff)
+        _write(_df_text(mapping, args.format), None)
+        if args.mapping_out:
+            _write(mapping.to_csv(index=False), args.mapping_out)
+        broken = broken_sheets(t, mapping, datasource=args.datasource)
+        for r in broken.to_dict("records"):
+            name = r["caption"] or r["field"].strip("[]")
+            print(f"missing: {name} (breaks: {r['sheets'] or 'no sheet'})", file=sys.stderr)
+        if args.write is None:
+            if not broken.empty:
+                print("required fields are unmapped; edit the mapping (--mapping-out / --mapping) "
+                      "or pass --allow-missing", file=sys.stderr)
+            return 0
+        report: dict = {}
+        out = apply_template(t, data, mapping=mapping, params=params, output_path=args.write or None,
+                             datasource=args.datasource, allow_missing=args.allow_missing, report=report)
+        print(f"wrote {out} ({report['mapped']} field(s) mapped, {report['missing']} missing, "
+              f"{report['parameters']} parameter(s) set)", file=sys.stderr)
+        return 0
+    except (FileNotFoundError, FileExistsError, ValueError, OSError, zipfile.BadZipFile,
+            json.JSONDecodeError, etree.XMLSyntaxError, pd.errors.ParserError) as e:
+        print(f"error: {e}", file=sys.stderr)
+        return 1
+
+
 def _is_reserved_subcommand(argv: list[str], name: str) -> bool:
     """True if `argv` invokes the `name` subcommand -- but don't let that
     shadow an actual workbook that happens to be named exactly "diff" or
@@ -261,6 +382,8 @@ def main(argv: list[str] | None = None) -> int:
         return _run_batch(raw_argv[1:])
     if _is_reserved_subcommand(raw_argv, "rename"):
         return _run_rename(raw_argv[1:])
+    if _is_reserved_subcommand(raw_argv, "template"):
+        return _run_template(raw_argv[1:])
 
     args = build_arg_parser().parse_args(argv)
 

{py_tbparse-0.3.0 → py_tbparse-0.4.0}/py_tbparse/parser.py

@@ -208,6 +208,21 @@ class TwbParser:
 
         return suggest_field_renames(self, reference=reference, **kwargs)
 
+    def get_field_usage(self) -> pd.DataFrame:
+        """Which sheets, dashboards and calculations use each field; see
+        `usage.field_usage`."""
+        from .usage import field_usage
+
+        return field_usage(self)
+
+    def get_renames(self, reference=None, **kwargs) -> pd.DataFrame:
+        """Suggested clean names for everything in the report (fields,
+        parameters, worksheets, dashboards, datasources, folders,
+        hierarchies); see `rename.suggest_renames`."""
+        from .rename import suggest_renames
+
+        return suggest_renames(self, reference=reference, **kwargs)
+
     def write_renamed_workbook(self, output_path=None, renames=None, overwrite=False, **kwargs) -> str:
         """Save a copy of the workbook with clean field names; returns its
         path. See `rename.apply_field_renames`."""