py-tbparse 0.4.0__tar.gz → 0.4.4__tar.gz

{py_tbparse-0.4.0 → py_tbparse-0.4.4}/AGENTS.md

@@ -122,10 +122,11 @@ beyond the R package's scope:
 |---|---|
 | `_tables.py` | Name → `TwbParser`-accessor registry shared by `cli.py`, `webgui.py`, `diff.py`, and `batch.py`. Adding a new extractor to `TwbParser`? Add it here too so it's automatically available everywhere else. |
 | `cli.py` | `py-tbparse` command-line entry point, plus the `diff`/`batch`/`rename`/`template` subcommands (dispatched on `sys.argv[1]` before the normal single-workbook argparse parser runs) |
-| `webgui.py` | `py-tbparse-gui`: stdlib-only (`http.server` + vanilla JS) local browser GUI, no GUI toolkit dependency. Loosely fills the role of the R package's `run_twbparser_app`/Shiny inspector. |
-| `graph.py` | `to_dot()`: Graphviz DOT export of joins/relationships (+ optional inferred, as dashed edges). Replaces the R package's igraph/ggraph-based `plot_dependency_graph`/`plot_relationship_graph` with a dependency-free text format any Graphviz-compatible tool can render. |
+| `webgui.py` + `webui/` | `py-tbparse-gui`: stdlib-only (`http.server` + vanilla JS) local browser GUI, no GUI toolkit dependency. The server is `webgui.py`; the page is `webui/` (`index.html`, `tokens.css`, `themes.css`, `app.css`, `table.js`, `graph.js`, `app.js`). Loosely fills the role of the R package's `run_twbparser_app`/Shiny inspector. |
+| `graph.py` | `to_dot()`: Graphviz DOT export of joins/relationships (+ optional inferred, as dashed edges); `graph_data()` is the same edges as plain data, which the GUI lays out and draws itself (`webui/graph.js`: layered layout, SVG, pan/zoom, keyboard). Replaces the R package's igraph/ggraph-based `plot_dependency_graph`/`plot_relationship_graph` with a dependency-free text format any Graphviz-compatible tool can render. |
 | `diff.py` | `diff_tables()`/`diff_workbooks()`: row-level added/removed diff between two workbooks' same-named table, via `_tables.TABLE_SPECS`. No "changed" classification without a natural key — a changed row shows as one removed + one added row. |
 | `rename.py` | `suggest_field_renames()` (clean-name suggestions, optionally matched against a "before" reference), `suggest_renames()` (the same for every kind of object: field, parameter, worksheet, dashboard, datasource, folder, hierarchy; adds a `kind` column), `load_rename_mapping()` (read an edited CSV back), `compare_field_schemas()` (fields with no counterpart across a datasource switch) and `apply_field_renames()`/`build_renamed_workbook()` (write a copy with captions set; never overwrites). Also the `field-renames` and `report-renames` tables in `_tables.py`. A worksheet/dashboard rename must rewrite every reference (`_SHEET_REFERENCES`); if you learn of another place Tableau writes a sheet name, add it there. |
+| `report.py` | `workbook_report()`: the overview's report card (summary sentence, health checks, dashboards with their sheets) built from `validate_relationships`, `field_usage` and `usage.missing_references`. Every health item names the table and column filters that list exactly the rows it counted; a test keeps count and rows equal on all 200 corpus workbooks. |
 | `usage.py` | `field_usage()`: for every field, the worksheets, dashboards and calculations that use it, followed through calculations, groups/sets and bins (the `field-usage` table). Python-native rather than a port of the R package's field-usage helper. |
 | `templates.py` | Workbook templates. `make_template()` writes a `.twbx` with a `template.json` manifest (required fields from `usage.py`, parameters, connections; data, extracts and credentials stripped); `read_data()` describes new data (CSV, or a `.twb`/`.twbx`/`.tds`); `suggest_mapping()` pairs fields with columns (reuses `rename._match_key`, type checks against the field's *physical* type, `datatype-customized` fields keep their type); `apply_template()` replaces the datasource's connection, keeping every field's local name, sets parameters and stores `template-answers.json`. A CSV connection is written in the 2020.2+ object-model shape (both `_.fcp.ObjectModelEncapsulateLegacy` relations, object graph, table column, `object-id` per record) when the template uses it -- keep those in step if you touch one. Untested in Tableau itself. |
 | `batch.py` | `scan_folder()`: runs one table across every `.twb`/`.twbx` in a directory, concatenated with a `workbook` column. Skips (with a warning) any file that fails to load/extract rather than aborting the batch. |
@@ -203,21 +204,38 @@ string became a literal newline, splitting a JS string literal across
 two lines) killed the entire script, so no handlers bound and the UI was
 inert — with the whole suite green.
 
-So: **any change to `webgui.py`'s `_PAGE` needs a browser test**, in
+So: **any change to the page (`py_tbparse/webui/`) needs a browser test**, in
+The browser suite is split by concern, all sharing the fixtures in `tests/test_gui_browser.py`:
+`test_gui_table.py` (windowing invariants, pipeline vs. a Python oracle, a seeded random walk;
+`PYTBPARSE_WALK_SEEDS`/`PYTBPARSE_WALK_STEPS` widen it), `test_gui_a11y.py` (roles, keyboard, focus, rendered
+contrast in both themes) and `test_gui_layout.py` (no sideways overflow from 320 px up, layout stability).
+Design decisions the tests pin: only sorting uses a view transition (Chromium sends clicks to the page root while
+one runs); column `MIN_WIDTH` is 80; per-table view settings are keyed by column name; disabled menu items use
+`aria-disabled` so they stay focusable.
+
 `tests/test_gui_browser.py`, which runs the page in real headless
 Chromium via Playwright and fails on any uncaught JS error. The cheap
 structural guards in `test_webgui.py` (unterminated string literals,
 bracket balance) are a backstop, not a substitute.
 
-`_PAGE` is declared as `r"""..."""` (a **raw** string) specifically so
-this can't recur: without `r`, any backslash escape meant for the
-*browser* (`\n`, `\t`, a future `\'`) would need doubling in the Python
-source, and forgetting to double it silently corrupts the embedded JS
-instead of erroring. Keep it raw — write JS escapes the normal JS way
-(`'\n'`, not `'\\n'`).
-
-Any server-side value spliced into `_PAGE` (currently `TABLE_NAMES` and
-the preloaded workbook path) must go through `_json_for_script()`, not
+The page is real files, not a Python string: `py_tbparse/webui/index.html`, `tokens.css` (the design
+tokens), `app.css` and `app.js`. They are served by `webgui.py` from a fixed whitelist under
+`/static/` (`_ASSETS`), so a request can never reach any other file; add a new asset to
+`_ASSETS` and to the `webui/*` package-data (pyproject and MANIFEST.in) or it will not ship.
+`index.html` is the only thing that gets server values: `_render_index()` replaces
+`<!--APP_CONFIG-->` (now in `<head>`, so the saved theme is on `<html>` before the first paint) with the single
+inline `<script>`. Keep it that way (one inline script plus the three external ones, `table.js`, `graph.js`, then
+`app.js`); a test counts them. `populateTables();` must appear exactly once
+in `app.js`, and the structural tests (unterminated string literals, bracket balance) read both served scripts.
+Keep apostrophes out of JS strings and comments (the test counts quotes per line).
+
+`POST /upload` (drag and drop, Open file) takes raw bytes with `Content-Type: application/octet-stream` and the name
+in `X-Filename`: neither is CORS-safelisted, so another site cannot send it without a preflight. It streams to a
+`mkdtemp` directory, refuses over `MAX_UPLOAD_BYTES` (413) and content that does not match the extension, and keeps
+only one upload at a time. An uploaded workbook cannot "create beside the original" (409), only download.
+
+Any server-side value spliced into the page (currently `TABLE_NAMES`, the preloaded workbook path
+and the version, all in `_render_index()`'s config script) must go through `_json_for_script()`, not
 bare `json.dumps()`. `json.dumps` doesn't escape `/`, so a value
 containing the literal text `</script>` closes the script tag early in
 the browser's HTML parser — this is real, not theoretical, since the
@@ -241,6 +259,35 @@ leaves inputs empty and tests time out) and may crash rendering. Don't add
 startup, which makes collection fail for anyone who doesn't have it
 installed.
 
+The table is windowed (`table.js`, class `VTable`): rows have a fixed height (`--row-h`) and only the rows
+near the viewport exist in the page, so code and tests that read rows from the DOM see a window, not the table.
+Use `aria-rowcount` for the size and the column menu's "Copy column values" to read a whole column (the tests do:
+`_copy_column`). Never render every row; the old table needed over 4 s to sort 20,000 rows and could not draw
+50,000. Data work (search index, typed sort) is in `app.js`; each render records the User Timing measure
+`py-tbparse:table`, which the speed tests read. The budgets (first paint of 1,000 rows 150 ms, filtering 50,000 rows
+100 ms) are enforced at twice the figure. Heavy tests skip below 1.5 GB of free memory (`_enough_memory`); this
+sandbox has only 2 CPUs and 7.9 GB, and a 50,000-row DOM render has crashed it before.
+
+The page's colours, spacing, type and motion are tokens in `webui/tokens.css`; use them rather than literals.
+Colour themes are `webui/themes.css`: each theme is a complete colour set in light and dark, selected by
+`<html data-theme data-mode>` (Auto is resolved to light/dark by the head script and kept in step by `app.js`;
+dark colours live under `[data-mode="dark"]`, there is no `prefers-color-scheme` query). To add a theme, add its
+two blocks there, its id to `webgui.THEMES` and its label to `THEME_LABELS` in `app.js`; the token test checks it.
+`tests/test_webui_tokens.py` reads those files and fails if any text/background pair drops below WCAG AA
+4.5:1, if the stylesheet uses an undefined variable, or if text is faded with `opacity` (use `--faint`).
+Motion durations come from the `--dur-*` tokens, which `prefers-reduced-motion` sets to instant; keep new
+animations on those tokens.
+
+`scripts/readme_screenshots.py` retakes the README's four images from the made-up `docs/demo/coffee-shop.twb`
+(built by `scripts/make_demo_workbook.py`, which is the one place that workbook is defined); run both after a
+visible change.
+
+`scripts/gui_screenshots.py WORKBOOK OUT_DIR [--compare BASELINE_DIR]` captures nine GUI states (start,
+overview and fields in light and dark, renames, graph, phone width) deterministically. Use it for GUI
+refactors: a pure refactor must compare all-identical to the baseline taken before it; a redesign is expected to
+differ, so review the new look and re-capture the baseline. The redesign plan and its decisions are in
+`docs/ui-redesign-plan.md`.
+
 ## Commit / PR conventions
 
 Nothing project-specific beyond the harness defaults — see repo commit

{py_tbparse-0.4.0 → py_tbparse-0.4.4}/MANIFEST.in

@@ -5,3 +5,4 @@ recursive-include tests *.py
 include tests/fixtures/*.twb
 include tests/fixtures/*.twbx
 recursive-include docs *.png
+recursive-include py_tbparse/webui *

py_tbparse-0.4.4/PKG-INFO

@@ -0,0 +1,137 @@
+Metadata-Version: 2.4
+Name: py-tbparse
+Version: 0.4.4
+Summary: Native Python port of the twbparser R package: parse Tableau .twb/.twbx workbooks into pandas DataFrames.
+Author: DDSNA
+License: MIT
+Project-URL: Homepage, https://github.com/DDSNA/py-tbparse
+Project-URL: Source, https://github.com/DDSNA/py-tbparse
+Project-URL: Issues, https://github.com/DDSNA/py-tbparse/issues
+Keywords: tableau,twb,twbx,workbook,parser,pandas
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Office/Business
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: lxml>=4.9
+Requires-Dist: pandas>=1.5
+Provides-Extra: test
+Requires-Dist: pytest>=7; extra == "test"
+Provides-Extra: browser
+Requires-Dist: playwright>=1.40; extra == "browser"
+Provides-Extra: dev
+Requires-Dist: build>=1.0; extra == "dev"
+Requires-Dist: twine>=5.0; extra == "dev"
+Dynamic: license-file
+
+# py-tbparse
+
+[![PyPI](https://img.shields.io/pypi/v/py-tbparse)](https://pypi.org/project/py-tbparse/)
+[![Python](https://img.shields.io/pypi/pyversions/py-tbparse)](https://pypi.org/project/py-tbparse/)
+[![CI](https://github.com/DDSNA/py-tbparse/actions/workflows/ci.yml/badge.svg)](https://github.com/DDSNA/py-tbparse/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/DDSNA/py-tbparse/blob/main/LICENSE)
+
+Read Tableau workbooks (`.twb` and `.twbx`) from Python. You get datasources, fields, calculated fields, joins, relationships and dashboards as pandas DataFrames. It is plain Python on top of `lxml` and `pandas`, so Tableau and R are not needed.
+
+It began as a port of PrigasG's R package [twbparser](https://github.com/PrigasG/twbparser). The command-line tool, the browser GUI, renaming, templates, workbook diffing and folder scanning are new here.
+
+![The overview report card for a small demo workbook](https://raw.githubusercontent.com/DDSNA/py-tbparse/main/docs/gui-overview.png)
+
+## What it does
+
+- Lists what is in a workbook: datasources, parameters, fields, calculations, joins, relationships, dashboards, custom SQL, published sources.
+- Checks relationships and finds calculations that refer to fields the workbook does not have.
+- Shows which worksheets, dashboards and calculations use each field.
+- Suggests clean names for fields, sheets, dashboards and other objects, and writes a renamed copy.
+- Turns a finished workbook into a template you can fill with other data.
+- Compares two workbooks, scans a folder of them, and draws the data model as a graph.
+- Works from Python, from the command line, or in a local browser page.
+
+## Install
+
+```bash
+pip install py-tbparse
+```
+
+Python 3.9 or newer. The package, the import and both commands all use the same name: `pip install py-tbparse`, `import py_tbparse`, `py-tbparse`, `py-tbparse-gui`. Releases up to 0.2.0 used `twbparser_py` and `twbparser`; those names are gone.
+
+## Quick start
+
+From Python:
+
+```python
+from py_tbparse import TwbParser
+
+p = TwbParser("workbook.twb")    # or a .twbx
+p.get_overview()                 # counts of everything
+p.get_calculated_fields()        # each calculation and its formula
+p.get_relationships()            # how the tables connect
+```
+
+Every getter returns a DataFrame. The others are `get_datasources`, `get_parameters`, `get_fields`, `get_raw_fields`, `get_joins`, `get_relations`, `get_inferred_relationships`, `get_dashboards`, `get_dashboard_sheets`, `get_custom_sql`, `get_initial_sql`, `get_published_refs` and `get_field_usage`. `get_relationship_graph_dot()` returns the data model as Graphviz text and `validate()` looks for problems in the relationships. In Jupyter, a bare `p` shows the overview.
+
+From the command line:
+
+```bash
+py-tbparse workbook.twb                    # overview
+py-tbparse workbook.twb calculated-fields
+py-tbparse workbook.twb fields --format csv -o fields.csv
+py-tbparse workbook.twb validate           # exit code 2 if it finds a problem
+```
+
+In the browser:
+
+```bash
+py-tbparse-gui workbook.twb
+```
+
+Across workbooks:
+
+```python
+from py_tbparse import TwbParser, diff_workbooks, scan_folder
+
+diff_workbooks(TwbParser("v1.twb"), TwbParser("v2.twb"), table="datasources")
+scan_folder("./workbooks", table="datasources")   # one table, every workbook in the folder
+```
+
+## Guides
+
+- [Command line](https://github.com/DDSNA/py-tbparse/blob/main/docs/cli.md): every table, the `diff`, `batch`, `rename` and `template` commands, and their options.
+- [Browser GUI](https://github.com/DDSNA/py-tbparse/blob/main/docs/gui.md): opening files, the table, the overview, the graph, themes.
+- [Renaming](https://github.com/DDSNA/py-tbparse/blob/main/docs/renaming.md): clean names after a datasource switch, editing the suggestions, what will stay broken.
+- [Templates](https://github.com/DDSNA/py-tbparse/blob/main/docs/templates.md): make a template from a workbook and apply it to new data.
+- [`.twbx` files](https://github.com/DDSNA/py-tbparse/blob/main/docs/twbx.md): they are read straight from the zip; how to extract the contents.
+- [Development](https://github.com/DDSNA/py-tbparse/blob/main/docs/development.md): running the tests, the workbook corpus, the browser tests.
+
+## Limits
+
+- **Nothing the tool writes has been opened in Tableau yet.** That covers renamed workbooks and workbooks made from templates. The XML follows what Tableau writes, but open one on a copy and check it before relying on it. The original file is never modified or overwritten.
+- **Only part of the R package is ported.** Missing: formatting, tooltips, colors, axes and sorts, dashboard layout and actions, calculation complexity, the replication brief and the Shiny inspector. The GUI covers some of what the inspector did.
+- Where the R version has a bug, this one does not copy it: joins and relationships on more than one key, nested joins, the include-parameters option, and calculations with brackets inside brackets.
+- The GUI is meant to run on your own machine for one person. It refuses requests that come from other websites, but there is no login, so do not put it on a shared network.
+
+## Why a Tableau parser
+
+A `.twb` is XML and a `.twbx` is a zip containing one, so reading them from Python is not hard. Power BI's `.pbix` is a binary format built on a proprietary storage engine, and getting into it from code took reverse-engineering projects like [PBIXRay](https://github.com/Hugoberry/pbixray) and [pbi-tools](https://github.com/pbi-tools/pbi-tools). On the server side it goes the other way: Tableau's own Python tooling ([tableauserverclient](https://pypi.org/project/tableauserverclient/) and `tabcmd`) is more mature and more open than what Microsoft has for the Power BI REST API.
+
+## Contributing
+
+Issues and pull requests are welcome at [github.com/DDSNA/py-tbparse](https://github.com/DDSNA/py-tbparse). The test setup is in the [development guide](https://github.com/DDSNA/py-tbparse/blob/main/docs/development.md). `AGENTS.md` describes the code layout and the rules for porting a function from the R package.
+
+## Credit
+
+Based on [twbparser](https://github.com/PrigasG/twbparser) by George Arthur, MIT licensed.
+
+## License
+
+MIT, see [LICENSE](https://github.com/DDSNA/py-tbparse/blob/main/LICENSE).

py_tbparse-0.4.4/README.md

@@ -0,0 +1,101 @@
+# py-tbparse
+
+[![PyPI](https://img.shields.io/pypi/v/py-tbparse)](https://pypi.org/project/py-tbparse/)
+[![Python](https://img.shields.io/pypi/pyversions/py-tbparse)](https://pypi.org/project/py-tbparse/)
+[![CI](https://github.com/DDSNA/py-tbparse/actions/workflows/ci.yml/badge.svg)](https://github.com/DDSNA/py-tbparse/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/DDSNA/py-tbparse/blob/main/LICENSE)
+
+Read Tableau workbooks (`.twb` and `.twbx`) from Python. You get datasources, fields, calculated fields, joins, relationships and dashboards as pandas DataFrames. It is plain Python on top of `lxml` and `pandas`, so Tableau and R are not needed.
+
+It began as a port of PrigasG's R package [twbparser](https://github.com/PrigasG/twbparser). The command-line tool, the browser GUI, renaming, templates, workbook diffing and folder scanning are new here.
+
+![The overview report card for a small demo workbook](https://raw.githubusercontent.com/DDSNA/py-tbparse/main/docs/gui-overview.png)
+
+## What it does
+
+- Lists what is in a workbook: datasources, parameters, fields, calculations, joins, relationships, dashboards, custom SQL, published sources.
+- Checks relationships and finds calculations that refer to fields the workbook does not have.
+- Shows which worksheets, dashboards and calculations use each field.
+- Suggests clean names for fields, sheets, dashboards and other objects, and writes a renamed copy.
+- Turns a finished workbook into a template you can fill with other data.
+- Compares two workbooks, scans a folder of them, and draws the data model as a graph.
+- Works from Python, from the command line, or in a local browser page.
+
+## Install
+
+```bash
+pip install py-tbparse
+```
+
+Python 3.9 or newer. The package, the import and both commands all use the same name: `pip install py-tbparse`, `import py_tbparse`, `py-tbparse`, `py-tbparse-gui`. Releases up to 0.2.0 used `twbparser_py` and `twbparser`; those names are gone.
+
+## Quick start
+
+From Python:
+
+```python
+from py_tbparse import TwbParser
+
+p = TwbParser("workbook.twb")    # or a .twbx
+p.get_overview()                 # counts of everything
+p.get_calculated_fields()        # each calculation and its formula
+p.get_relationships()            # how the tables connect
+```
+
+Every getter returns a DataFrame. The others are `get_datasources`, `get_parameters`, `get_fields`, `get_raw_fields`, `get_joins`, `get_relations`, `get_inferred_relationships`, `get_dashboards`, `get_dashboard_sheets`, `get_custom_sql`, `get_initial_sql`, `get_published_refs` and `get_field_usage`. `get_relationship_graph_dot()` returns the data model as Graphviz text and `validate()` looks for problems in the relationships. In Jupyter, a bare `p` shows the overview.
+
+From the command line:
+
+```bash
+py-tbparse workbook.twb                    # overview
+py-tbparse workbook.twb calculated-fields
+py-tbparse workbook.twb fields --format csv -o fields.csv
+py-tbparse workbook.twb validate           # exit code 2 if it finds a problem
+```
+
+In the browser:
+
+```bash
+py-tbparse-gui workbook.twb
+```
+
+Across workbooks:
+
+```python
+from py_tbparse import TwbParser, diff_workbooks, scan_folder
+
+diff_workbooks(TwbParser("v1.twb"), TwbParser("v2.twb"), table="datasources")
+scan_folder("./workbooks", table="datasources")   # one table, every workbook in the folder
+```
+
+## Guides
+
+- [Command line](https://github.com/DDSNA/py-tbparse/blob/main/docs/cli.md): every table, the `diff`, `batch`, `rename` and `template` commands, and their options.
+- [Browser GUI](https://github.com/DDSNA/py-tbparse/blob/main/docs/gui.md): opening files, the table, the overview, the graph, themes.
+- [Renaming](https://github.com/DDSNA/py-tbparse/blob/main/docs/renaming.md): clean names after a datasource switch, editing the suggestions, what will stay broken.
+- [Templates](https://github.com/DDSNA/py-tbparse/blob/main/docs/templates.md): make a template from a workbook and apply it to new data.
+- [`.twbx` files](https://github.com/DDSNA/py-tbparse/blob/main/docs/twbx.md): they are read straight from the zip; how to extract the contents.
+- [Development](https://github.com/DDSNA/py-tbparse/blob/main/docs/development.md): running the tests, the workbook corpus, the browser tests.
+
+## Limits
+
+- **Nothing the tool writes has been opened in Tableau yet.** That covers renamed workbooks and workbooks made from templates. The XML follows what Tableau writes, but open one on a copy and check it before relying on it. The original file is never modified or overwritten.
+- **Only part of the R package is ported.** Missing: formatting, tooltips, colors, axes and sorts, dashboard layout and actions, calculation complexity, the replication brief and the Shiny inspector. The GUI covers some of what the inspector did.
+- Where the R version has a bug, this one does not copy it: joins and relationships on more than one key, nested joins, the include-parameters option, and calculations with brackets inside brackets.
+- The GUI is meant to run on your own machine for one person. It refuses requests that come from other websites, but there is no login, so do not put it on a shared network.
+
+## Why a Tableau parser
+
+A `.twb` is XML and a `.twbx` is a zip containing one, so reading them from Python is not hard. Power BI's `.pbix` is a binary format built on a proprietary storage engine, and getting into it from code took reverse-engineering projects like [PBIXRay](https://github.com/Hugoberry/pbixray) and [pbi-tools](https://github.com/pbi-tools/pbi-tools). On the server side it goes the other way: Tableau's own Python tooling ([tableauserverclient](https://pypi.org/project/tableauserverclient/) and `tabcmd`) is more mature and more open than what Microsoft has for the Power BI REST API.
+
+## Contributing
+
+Issues and pull requests are welcome at [github.com/DDSNA/py-tbparse](https://github.com/DDSNA/py-tbparse). The test setup is in the [development guide](https://github.com/DDSNA/py-tbparse/blob/main/docs/development.md). `AGENTS.md` describes the code layout and the rules for porting a function from the R package.
+
+## Credit
+
+Based on [twbparser](https://github.com/PrigasG/twbparser) by George Arthur, MIT licensed.
+
+## License
+
+MIT, see [LICENSE](https://github.com/DDSNA/py-tbparse/blob/main/LICENSE).

{py_tbparse-0.4.0 → py_tbparse-0.4.4}/py_tbparse/__init__.py

@@ -10,7 +10,7 @@ from .dashboards import dashboard_sheets, list_dashboards
 from .datasources import extract_datasource_details, extract_named_connections, extract_parameters
 from .diff import diff_tables, diff_workbooks
 from .fields import extract_columns_with_table_source, infer_implicit_relationships
-from .graph import to_dot
+from .graph import graph_data, to_dot
 from .joins import extract_joins
 from .parser import TwbParser
 from .published import extract_published_refs
@@ -51,6 +51,7 @@ __all__ = [
     "infer_implicit_relationships",
     "extract_joins",
     "to_dot",
+    "graph_data",
     "extract_relations",
     "extract_relationships",
     "extract_custom_sql",

{py_tbparse-0.4.0 → py_tbparse-0.4.4}/py_tbparse/_tables.py

@@ -93,6 +93,10 @@ def _initial_sql(p: TwbParser, **_kw) -> pd.DataFrame:
     return p.get_initial_sql()
 
 
+def _missing_references(p: TwbParser, **_kw) -> pd.DataFrame:
+    return p.get_missing_references()
+
+
 def _published_refs(p: TwbParser, **_kw) -> pd.DataFrame:
     return p.get_published_refs()
 
@@ -109,6 +113,7 @@ TABLE_SPECS: dict[str, Callable[..., pd.DataFrame]] = {
     "field-renames": _field_renames,
     "report-renames": _report_renames,
     "field-usage": _field_usage,
+    "missing-references": _missing_references,
     "joins": _joins,
     "relations": _relations,
     "relationships": _relationships,

{py_tbparse-0.4.0 → py_tbparse-0.4.4}/py_tbparse/graph.py

@@ -44,6 +44,57 @@ def _inferred_label(row) -> str:
     return f"{row.left_field} ~ {row.right_field} ({row.reason})"
 
 
+def _collect(joins_df, relationships_df, inferred_df) -> list[tuple[str, str, str, str]]:
+    """Every edge as (left table, right table, label, kind) with kind `join`, `relationship` or `inferred`,
+    in the order they are drawn: joins, relationships, then inferred guesses."""
+    edges = []
+    for left, right, label in _edges_from(joins_df, _join_label):
+        edges.append((left, right, label, "join"))
+    for left, right, label in _edges_from(relationships_df, _relationship_label):
+        edges.append((left, right, label, "relationship"))
+    if inferred_df is not None:
+        for left, right, label in _edges_from(inferred_df, _inferred_label):
+            edges.append((left, right, label, "inferred"))
+    return edges
+
+
+def graph_data(
+    joins_df: pd.DataFrame,
+    relationships_df: pd.DataFrame,
+    inferred_df: pd.DataFrame | None = None,
+) -> dict:
+    """The same graph as `to_dot`, as plain data for the GUI to lay out and draw:
+    `{"nodes": [{"id": ...}], "edges": [{"source", "target", "label", "kind"}]}`.
+    Nodes are sorted by name; edges keep `to_dot`'s order."""
+    edges = _collect(joins_df, relationships_df, inferred_df)
+    nodes = sorted({n for left, right, _, _ in edges for n in (left, right)})
+    return {
+        "nodes": [{"id": n} for n in nodes],
+        "edges": [{"source": l, "target": r, "label": lab, "kind": k} for l, r, lab, k in edges],
+    }
+
+
+def to_dot(
+    joins_df: pd.DataFrame,
+    relationships_df: pd.DataFrame,
+    inferred_df: pd.DataFrame | None = None,
+    graph_name: str = "twb",
+) -> str:
+    """Render join/relationship (and optionally inferred) edges as a
+    Graphviz DOT digraph string.
+
+    Solid edges are real joins/relationships; dashed edges (when
+    `inferred_df` is passed) are `infer_implicit_relationships()` guesses.
+    """
+    edges = [(l, r, lab, kind == "inferred") for l, r, lab, kind in _collect(joins_df, relationships_df, inferred_df)]
+
+    nodes = sorted({n for left, right, _, _ in edges for n in (left, right)})
+    return {
+        "nodes": [{"id": n} for n in nodes],
+        "edges": [{"source": l, "target": r, "label": lab, "kind": k} for l, r, lab, k in edges],
+    }
+
+
 def to_dot(
     joins_df: pd.DataFrame,
     relationships_df: pd.DataFrame,

{py_tbparse-0.4.0 → py_tbparse-0.4.4}/py_tbparse/parser.py

@@ -20,7 +20,7 @@ from .datasources import (
     extract_datasource_details,
 )
 from .fields import _FIELDS_COLUMNS, _INFERRED_COLUMNS, extract_columns_with_table_source, infer_implicit_relationships
-from .graph import to_dot
+from .graph import graph_data, to_dot
 from .joins import _JOIN_COLUMNS, extract_joins
 from .published import _PUBLISHED_COLUMNS, extract_published_refs
 from .relationships import _RELATION_COLUMNS, _RELATIONSHIP_COLUMNS, extract_relations, extract_relationships
@@ -202,6 +202,14 @@ class TwbParser:
             self.get_inferred_relationships() if include_inferred else None,
         )
 
+    def get_relationship_graph_data(self, include_inferred: bool = False) -> dict:
+        """The relationship graph as `{"nodes": [...], "edges": [...]}` (see `graph.graph_data`)."""
+        return graph_data(
+            self.get_joins(),
+            self.get_relationships(),
+            self.get_inferred_relationships() if include_inferred else None,
+        )
+
     def get_field_renames(self, reference=None, **kwargs) -> pd.DataFrame:
         """Suggested clean field names; see `rename.suggest_field_renames`."""
         from .rename import suggest_field_renames
@@ -215,6 +223,19 @@ class TwbParser:
 
         return field_usage(self)
 
+    def get_missing_references(self) -> pd.DataFrame:
+        """Calculations that name a field the workbook does not have; see
+        `usage.missing_references`."""
+        from .usage import missing_references
+
+        return missing_references(self)
+
+    def get_report(self) -> dict:
+        """The workbook report card (summary, health checks); see `report.workbook_report`."""
+        from .report import workbook_report
+
+        return workbook_report(self)
+
     def get_renames(self, reference=None, **kwargs) -> pd.DataFrame:
         """Suggested clean names for everything in the report (fields,
         parameters, worksheets, dashboards, datasources, folders,

py_tbparse-0.4.4/py_tbparse/report.py

@@ -0,0 +1,124 @@
+"""A plain-language report card for a workbook: what is in it and what deserves a look.
+
+Python-native (the R package has no equivalent). It only combines checks that already exist:
+`validate_relationships`, `field_usage` and `missing_references`, plus counts of the things that are
+information rather than problems. The GUI's overview shows it; each item carries the table (and column
+filters) that lists exactly the rows it counted, so a count on the card is never a different number from
+the rows it opens.
+"""
+
+from __future__ import annotations
+
+import os
+
+from .parser import TwbParser
+from .usage import missing_references
+
+PROBLEM, WARNING, INFO = "problem", "warning", "info"
+
+
+def _plural(n: int, one: str, many: str | None = None) -> str:
+    return f"{n} {one if n == 1 else (many or one + 's')}"
+
+
+def _item(severity: str, key: str, title: str, detail: str, count: int, table: str, filters=None) -> dict:
+    return {"id": key, "severity": severity, "title": title, "detail": detail, "count": count,
+            "table": table, "filters": filters or []}
+
+
+def _sheets_by_dashboard(parser: TwbParser) -> list[dict]:
+    doc = parser.xml_doc
+    out = []
+    for db in doc.xpath("/workbook/dashboards/dashboard[@name]"):
+        seen: list[str] = []
+        for z in db.xpath(".//zone[@worksheet]"):
+            if z.get("worksheet") not in seen:
+                seen.append(z.get("worksheet"))
+        out.append({"name": db.get("name"), "sheets": seen})
+    return sorted(out, key=lambda d: d["name"].lower())
+
+
+def workbook_report(parser: TwbParser) -> dict:
+    """`{summary, counts, worksheets, dashboards, health}` for a loaded workbook.
+
+    `health` is a list of items (`severity`, `title`, `detail`, `count`, `table`, `filters`), problems
+    first. An empty list means nothing was found."""
+    doc = parser.xml_doc
+    overview = parser.get_overview().iloc[0].to_dict()
+    overview.pop("file", None)
+    # a .twbx is named by the package the person opened, not by the .twb inside it
+    name = os.path.basename(getattr(parser, "twbx_path", None) or parser.path or "") or "This workbook"
+    sheets = sorted({w.get("name") for w in doc.xpath("/workbook/worksheets/worksheet[@name]")}, key=str.lower)
+    dashboards = _sheets_by_dashboard(parser)
+    on_dashboard = {s for d in dashboards for s in d["sheets"]}
+
+    parts = [_plural(len(dashboards), "dashboard"), _plural(len(sheets), "worksheet"),
+             _plural(int(overview["datasources"]), "datasource")]
+    summary = f"{name} has {parts[0]}, {parts[1]} and {parts[2]}."
+
+    health: list[dict] = []
+
+    from .validators import validate_relationships
+    issues = validate_relationships(parser)["issues"]
+    if "unknown_tables" in issues:
+        n = len(issues["unknown_tables"])
+        health.append(_item(PROBLEM, "unknown-tables", _plural(n, "relationship") + " point at a table the workbook does not have",
+                            "Check the left and right table names against the datasources.", n, "relationships"))
+    if "unknown_fields" in issues:
+        n = len(issues["unknown_fields"])
+        health.append(_item(PROBLEM, "unknown-fields", _plural(n, "relationship") + " use a field that is not in the workbook",
+                            "The join keys may have been renamed or removed.", n, "relationships"))
+
+    missing = missing_references(parser)
+    if len(missing):
+        n = len(missing)
+        health.append(_item(WARNING, "missing-references", _plural(n, "reference") + " in calculations to fields that do not exist",
+                            "A calculation names a field the workbook does not have, so it will show an error in Tableau.",
+                            n, "missing-references"))
+
+    usage = parser.get_field_usage()
+    unused = usage[~usage["used"].astype(bool)] if len(usage) else usage
+    calcs = unused[unused["kind"] == "calculated"] if len(unused) else unused
+    raw = unused[unused["kind"] == "physical"] if len(unused) else unused
+    if len(calcs):
+        n = len(calcs)
+        health.append(_item(WARNING, "unused-calculations", _plural(n, "calculation") + " no worksheet uses",
+                            "Candidates to delete, unless another workbook or a published source needs them.",
+                            n, "field-usage", [{"col": "kind", "text": "calculated"}, {"col": "used", "text": "false"}]))
+    if len(raw):
+        n = len(raw)
+        health.append(_item(INFO, "unused-fields", _plural(n, "field") + " no worksheet uses",
+                            "Ordinary for a wide data source; useful when you are slimming one down.",
+                            n, "field-usage", [{"col": "kind", "text": "physical"}, {"col": "used", "text": "false"}]))
+
+    loose = [s for s in sheets if s not in on_dashboard]
+    if loose and dashboards:
+        health.append(_item(INFO, "sheets-off-dashboards", _plural(len(loose), "worksheet") + " not on any dashboard",
+                            ", ".join(loose[:6]) + (" and more" if len(loose) > 6 else ""), len(loose), ""))   # no table lists these: sheets with no zone
+
+    inferred = int(overview["inferred_relationships"])
+    if inferred:
+        health.append(_item(INFO, "inferred", _plural(inferred, "relationship") + " could be inferred from matching field names",
+                            "They are not modelled in the workbook; this is only a hint.", inferred, "inferred-relationships"))
+    for key, getter, table, title in (
+        ("custom-sql", parser.get_custom_sql, "custom-sql", "custom SQL"),
+        ("initial-sql", parser.get_initial_sql, "initial-sql", "initial SQL"),
+    ):
+        n = len(getter())
+        if n:
+            health.append(_item(INFO, key, f"{n} {title} {'entry' if n == 1 else 'entries'}", "", n, table))
+
+    # A datasource with no connection of its own is a published one, or the Parameters source that every
+    # workbook with parameters has. Only mention it when something other than Parameters is in the list; the
+    # count is still every row the link opens, Parameters included, and the detail says so.
+    pub = parser.get_published_refs()
+    likely = pub[pub["likely_published"].astype(bool)] if len(pub) else pub
+    if len(likely) and (likely["name"] != "Parameters").any():
+        n = len(likely)
+        health.append(_item(INFO, "published", _plural(n, "datasource") + " without a connection of its own",
+                            "Published data sources look like this. So does the Parameters source, which is counted too.",
+                            n, "published-refs", [{"col": "likely_published", "text": "true"}]))
+
+    order = {PROBLEM: 0, WARNING: 1, INFO: 2}
+    health.sort(key=lambda h: order[h["severity"]])
+    return {"summary": summary, "counts": overview, "worksheets": sheets, "dashboards": dashboards, "health": health}