pbi-docs 1.0.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,830 @@
1
+ Metadata-Version: 2.4
2
+ Name: pbi-docs
3
+ Version: 1.0.0
4
+ Summary: AI context engine for Power BI models: indexed, queryable context for LLM agents (MCP server, CLI, docs), zero dependencies
5
+ Author-email: Oscar Rosero <orosero2405@gmail.com>
6
+ License: MIT
7
+ Project-URL: Homepage, https://github.com/Osc2405/pbi-docs
8
+ Project-URL: Repository, https://github.com/Osc2405/pbi-docs
9
+ Project-URL: Issues, https://github.com/Osc2405/pbi-docs/issues
10
+ Project-URL: Changelog, https://github.com/Osc2405/pbi-docs/blob/main/CHANGELOG.md
11
+ Keywords: powerbi,metadata,dax,documentation,bi,pbit,mcp,ai-agent,llm-context
12
+ Classifier: Development Status :: 4 - Beta
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.10
15
+ Classifier: Programming Language :: Python :: 3.11
16
+ Classifier: Programming Language :: Python :: 3.12
17
+ Classifier: Programming Language :: Python :: 3.13
18
+ Classifier: License :: OSI Approved :: MIT License
19
+ Classifier: Operating System :: OS Independent
20
+ Classifier: Intended Audience :: Developers
21
+ Classifier: Intended Audience :: Information Technology
22
+ Classifier: Topic :: Documentation
23
+ Classifier: Topic :: Utilities
24
+ Requires-Python: >=3.10
25
+ Description-Content-Type: text/markdown
26
+ License-File: LICENSE
27
+ Provides-Extra: dev
28
+ Requires-Dist: pytest>=7.0.0; extra == "dev"
29
+ Dynamic: license-file
30
+
31
+ ## pbi-docs — AI Context Engine for Power BI Models
32
+
33
+ [![Tests](https://github.com/Osc2405/pbi-docs/actions/workflows/tests.yml/badge.svg)](https://github.com/Osc2405/pbi-docs/actions/workflows/tests.yml)
34
+ [![Python Version](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
35
+ [![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
36
+ [![Platform](https://img.shields.io/badge/platform-Windows%20%7C%20macOS%20%7C%20Linux-lightgrey)](https://github.com/Osc2405/pbi-docs/actions/workflows/tests.yml)
37
+
38
+ **Turns a Power BI model (`.pbit` or the new `.pbip`/TMDL format) into documentation and
39
+ context an AI agent can actually use** — human-readable Markdown, indexed JSON for LLMs/RAG, a
40
+ query CLI, and a read-only MCP server. Zero external dependencies.
41
+
42
+ **Who it's for:** data engineers documenting dashboards, consultants auditing models they didn't
43
+ build, and anyone connecting an AI agent (Claude, GPT, Copilot) to a Power BI model's structure.
44
+
45
+ ### Demo
46
+
47
+ ![Power BI Model](docs/images/powerbi-sample.png)
48
+ ![CLI Usage](docs/images/CLI_Usage.png)
49
+ ![AI Agent Using the Documentation](docs/images/ChatGPT-demo.gif)
50
+
51
+ ## Quick Start
52
+
53
+ ```powershell
54
+ git clone https://github.com/Osc2405/pbi-docs.git
55
+ cd pbi-docs
56
+ pip install -e .
57
+
58
+ # From a .pbit file...
59
+ pbi-docs --input "data/pbit/my-model.pbit"
60
+ # ...or a PBIP project (folder, .pbip marker, or .SemanticModel/ — auto-detected)
61
+ pbi-docs --input "data/pbip/my-model/"
62
+
63
+ Get-Content "output/my-model.pbit/model_documentation.md"
64
+ ```
65
+
66
+ **Result:** 7 files in `output/<model-name>/` in seconds — human-readable Markdown, JSON/JSONL
67
+ context for AI agents, and an indexed, queryable version for large models.
68
+
69
+ <details>
70
+ <summary><strong>Full walkthrough</strong> — folder structure, expected output, and using the context in Python</summary>
71
+
72
+ The command generates a folder in `output/` with all documentation files:
73
+
74
+ ```
75
+ output/my-model.pbit/ (or output/my-model/ for PBIP)
76
+ ├── metadata.json # Structured model metadata
77
+ ├── model_documentation.md # Human-readable documentation
78
+ ├── agent_context.json # LLM-optimized context (top-20 measures)
79
+ ├── model_context.jsonl # JSONL format for embeddings/RAG
80
+ ├── index.json # Lightweight index + pointers
81
+ ├── relationships.json # All relationships
82
+ └── tables/
83
+ ├── Sales.json # Full detail per table
84
+ └── ...
85
+ ```
86
+
87
+ ```powershell
88
+ Get-Content "output/my-model.pbit/model_documentation.md" | Select-Object -First 15
89
+ ```
90
+ ```markdown
91
+ # my-model - Power BI Data Model
92
+
93
+ **Generated:** 2025-12-22 14:23:29
94
+
95
+ ## Model Summary
96
+
97
+ - **Business Tables:** 9
98
+ - **Total Columns:** 23
99
+ - **Total Measures:** 44
100
+ - **Relationships:** 9
101
+ ```
102
+
103
+ Use the JSON context directly in Python (or point an AI agent at it via `--query` or
104
+ `--mcp-serve` — see [Use Cases](#use-cases) below):
105
+
106
+ ```python
107
+ import json
108
+
109
+ with open("output/my-model.pbit/agent_context.json", "r", encoding="utf-8") as f:
110
+ context = json.load(f)
111
+
112
+ print(f"Model: {context['model_name']}")
113
+ print(f"Key measures: {len(context['key_measures'])}")
114
+ print(f"First measure: {context['key_measures'][0]['name']}")
115
+ ```
116
+ ```
117
+ Model: my-model
118
+ Key measures: 20
119
+ First measure: Revenue Budget
120
+ ```
121
+
122
+ </details>
123
+
124
+ ## Why pbi-docs?
125
+
126
+ | Your Need | pbi-docs Solution |
127
+ |-----------|---------------------|
128
+ | **Document 10+ dashboards fast** | Batch processing with `--batch` |
129
+ | **Support the new PBIP format** | Full TMDL parser, auto-detected from `.pbip` or folder |
130
+ | **Train AI agents on your models** | Indexed JSON/JSONL context, a query CLI, and an MCP server |
131
+ | **Let an AI agent query the model live** | Read-only MCP server (`--mcp-serve`) — validated against a test harness, not yet a live MCP client, see [MCP server](docs/use-cases.md#7-mcp-server---mcp-serve) |
132
+ | **Use it from your AI coding assistant** | Chat-invocable Skill for Claude Code + prompt file for GitHub Copilot |
133
+ | **Actually readable DAX** | Hierarchical indentation (4x better than raw) |
134
+ | **Compare model versions** | Content-aware `--diff`, with impact analysis (`--diff-impact`) |
135
+ | **Zero-cost, zero-install** | Python-only, no .NET dependencies |
136
+
137
+ **Perfect for:** Data engineers onboarding teams, consultants auditing models, organizations building AI copilots for BI.
138
+
139
+ ## Project Status
140
+
141
+ The read/context layer — PBIP/TMDL support, indexed output, query resolver, MCP server — is
142
+ implemented and tested (222 tests). Every claim above is backed by a dated, reproducible report,
143
+ not just asserted: see [Validation](#validation) below. Writing/editing TMDL models and PBIR/report-
144
+ layer parsing are deliberately out of scope for now (see `CHANGELOG.md` and the
145
+ [Roadmap](#roadmap) for why).
146
+
147
+ ## Requirements
148
+ - Python 3.10+ (3.12 recommended)
149
+ - Windows PowerShell (instructions include Windows commands)
150
+
151
+ Optional: virtual environment (`venv`). No external libraries required.
152
+
153
+ ## Installation (Windows/PowerShell)
154
+
155
+ ```powershell
156
+ # 1) Clone or download the repository
157
+ # 2) (Optional) Create and activate virtual environment
158
+ python -m venv venv
159
+ ./venv/Scripts/Activate.ps1
160
+
161
+ # 3) Editable installation (development)
162
+ pip install -e .
163
+
164
+ # Verify Python version
165
+ python --version
166
+ ```
167
+
168
+ If PowerShell blocks activation, run as Administrator:
169
+
170
+ ```powershell
171
+ Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
172
+ ```
173
+
174
+ ---
175
+
176
+ ## Quick Usage
177
+
178
+ ### Basic Commands
179
+
180
+ **Process a `.pbit` file:**
181
+ ```powershell
182
+ pbi-docs --input "data/pbit/my-model.pbit"
183
+ ```
184
+
185
+ **Process a PBIP project (new in v1.0):**
186
+ ```powershell
187
+ # From the .pbip marker file
188
+ pbi-docs --input "data/pbip/my-model.pbip"
189
+
190
+ # From the .SemanticModel folder directly
191
+ pbi-docs --input "data/pbip/my-model.SemanticModel"
192
+
193
+ # From the project root folder (auto-detected)
194
+ pbi-docs --input "data/pbip/my-model/"
195
+ ```
196
+
197
+ **Specify custom output directory:**
198
+ ```powershell
199
+ pbi-docs -i "data/pbit/my-model.pbit" -o "my-results"
200
+ ```
201
+
202
+ **Process multiple files (batch mode — mixed formats supported):**
203
+ ```powershell
204
+ pbi-docs --batch "data/pbit/*.pbit"
205
+ ```
206
+
207
+ ![Batch Processing](docs/images/batch-processing.png)
208
+
209
+ **Compare two versions of a model (mixed `.pbit`/`.pbip` supported):**
210
+ ```powershell
211
+ pbi-docs --diff "data/pbit/model_v1.pbit" "data/pbip/model_v2/"
212
+ ```
213
+
214
+ **Verbose mode (more debugging information):**
215
+ ```powershell
216
+ pbi-docs --input "data/pbit/my-model.pbit" --verbose
217
+ ```
218
+
219
+ **Human-readable indexed output (indented JSON, for debugging — compact by default):**
220
+ ```powershell
221
+ pbi-docs --input "data/pbit/my-model.pbit" --pretty
222
+ ```
223
+
224
+ **Generate documentation in Spanish:**
225
+ ```powershell
226
+ pbi-docs --input "data/pbit/my-model.pbit" --lang es
227
+ ```
228
+
229
+ **Generate documentation in English (default):**
230
+ ```powershell
231
+ pbi-docs --input "data/pbit/my-model.pbit" --lang en
232
+ # Or simply omit --lang (English is the default)
233
+ pbi-docs --input "data/pbit/my-model.pbit"
234
+ ```
235
+
236
+ ### Expected Output
237
+
238
+ When running the command, you'll see messages like:
239
+
240
+ ```
241
+ Processing file: data/pbit/my-model.pbit
242
+ Schema extracted successfully: 11 tables
243
+ Metadata processed: 11 tables, 44 measures
244
+ Metadata saved to: output/my-model.pbit/metadata.json
245
+ Documentation saved to: output/my-model.pbit/model_documentation.md
246
+ Agent context saved to: output/my-model.pbit/agent_context.json
247
+ JSONL context saved to: output/my-model.pbit/model_context.jsonl
248
+ Processing completed successfully for: my-model.pbit
249
+ ```
250
+
251
+ ### Important Notes
252
+
253
+ - **Supported formats:** `.pbit` files (ZIP + JSON TMSL) and `.pbip` projects (TMDL folder structure). `.pbix` files must be exported to `.pbit` from Power BI Desktop (File > Export > Power BI Template).
254
+ - **PBIP entry points:** The `--input` flag accepts a `.pbip` marker file, a `.SemanticModel/` folder, or a project root folder. Format is auto-detected.
255
+ - **Language selection:** Use `--lang en` for English (default) or `--lang es` for Spanish. The language affects the generated `model_documentation.md` and `agent_context.json` files.
256
+ - **Paths with spaces:** Use quotes around paths that contain spaces.
257
+ - **Recommended paths:** Place your files in `data/` or `data/pbit/` to keep the project organized.
258
+
259
+ ---
260
+
261
+ ## Project Structure
262
+
263
+ ```
264
+ pbi-docs/
265
+ ├── pbi_extractor/ # Main package
266
+ │ ├── __init__.py
267
+ │ ├── cli.py # CLI with argparse (auto-detection, --index-format)
268
+ │ ├── extractor.py # .pbit (ZIP+JSON TMSL) extractor
269
+ │ ├── pbip_extractor.py # .pbip / TMDL extractor (new in v1.0)
270
+ │ ├── processor.py # Metadata processing (format-agnostic)
271
+ │ ├── indexed_output.py # index.json + tables/*.json writer (new in v1.0)
272
+ │ ├── formatters.py # Advanced hierarchical DAX formatting
273
+ │ ├── categorizer.py # Table/measure categorization
274
+ │ ├── documentation.py # Markdown generation
275
+ │ ├── diff.py # Model comparison
276
+ │ ├── jsonl_generator.py # JSONL generator for LLMs
277
+ │ └── i18n.py # Translations (en/es)
278
+ ├── tests/
279
+ │ ├── fixtures/
280
+ │ │ └── minimal_pbip/ # TMDL test fixtures (new in v1.0)
281
+ │ ├── test_pbip_extractor.py
282
+ │ ├── test_cli_detection.py
283
+ │ ├── test_indexed_output.py
284
+ │ ├── test_categorizer.py
285
+ │ ├── test_i18n.py
286
+ │ └── test_processor_and_context.py
287
+ ├── githooks/ # Reference pre-commit hook (docs/pre_commit_hook.md)
288
+ ├── data/ # Input model files
289
+ ├── output/ # Generated results
290
+ ├── pyproject.toml # Package configuration
291
+ ├── README.md
292
+ ├── CHANGELOG.md
293
+ └── LICENSE
294
+ ```
295
+
296
+ ---
297
+
298
+ ## Generated Outputs
299
+
300
+ After running the command, a folder is created in `output/` with the model name. Inside you'll find:
301
+
302
+ - **`metadata.json`**
303
+ - `summary`: totals of tables, visible columns, visible measures and relationships.
304
+ - `tables`: each table with `columns` (type, visibility, category) and `measures` (clean expression, format, display folder, category).
305
+ - `relationships`: from/to, cardinality, direction and active status.
306
+
307
+ - **`model_documentation.md`**
308
+ - Model summary (language depends on `--lang` flag, default: English).
309
+ - List of tables (hidden or business), visible columns and measures grouped by category: revenue, cost, margin, percentage, ratio, temporal, etc.
310
+ - Sections with DAX expressions formatted with **hierarchical indentation**.
311
+ - Relationships table with visual representation of table connections.
312
+ - AI Agent Usage Guide with sample questions (translated based on selected language).
313
+
314
+ ![Model Relationships](docs/images/Relationships.png)
315
+
316
+ - **`agent_context.json`**
317
+ - Model name, totals, available tables, key measures (up to 20), temporal columns and sample questions (language depends on `--lang` flag, default: English).
318
+
319
+ - **`model_context.jsonl`**
320
+ - Line-delimited JSON format optimized for embeddings and RAG.
321
+ - Each line is an independent object (table, measure or relationship).
322
+ - Includes formatted DAX and sample prompts.
323
+
324
+ - **`index.json`** *(new in v1.0)*
325
+ - Lightweight model summary with per-table metadata (column/measure counts, categories) and relative paths to all other output files.
326
+ - Allows LLM agents to navigate large models without loading the full `metadata.json`.
327
+
328
+ - **`relationships.json`** *(new in v1.0)*
329
+ - All model relationships in a single focused file.
330
+
331
+ - **`tables/<TableName>.json`** *(new in v1.0)*
332
+ - Full detail for one table (columns + measures including DAX).
333
+ - One file per table, addressable via `index.json`.
334
+
335
+ ---
336
+
337
+ ## `index.json` — open format specification
338
+
339
+ `index.json` is a small, stable contract meant to be consumed directly by any tool — not just
340
+ pbi-docs' own CLI/resolver/MCP server: a lightweight per-table summary (name, column/measure
341
+ counts, categories, format, and a relative path to that table's detail file) plus pointers to
342
+ every other output file, so an agent can navigate a large model without loading `metadata.json`.
343
+ By default it's written **compact** (no indentation); pass `--pretty` for indented JSON.
344
+
345
+ Full field-by-field contract — top-level shape, per-table entry, the `tables/<Name>.json` JSON vs.
346
+ TOON shapes, and the versioning policy — is documented in
347
+ **[docs/index-json-spec.md](docs/index-json-spec.md)**.
348
+
349
+ ---
350
+
351
+ ## Use Cases
352
+
353
+ ### 1. Automatic Dashboard Documentation
354
+
355
+ **Problem:** Your company has multiple undocumented Power BI dashboards. Analysts waste time searching for which measures to use and how tables are related.
356
+
357
+ **Solution:**
358
+ ```powershell
359
+ # Process all dashboards in a folder (English documentation)
360
+ pbi-docs --batch "data/dashboards/*.pbit"
361
+
362
+ # Or generate Spanish documentation for all dashboards
363
+ pbi-docs --batch "data/dashboards/*.pbit" --lang es
364
+ ```
365
+
366
+ ![Batch Processing Example](docs/images/batch-processing.png)
367
+
368
+ **Result:**
369
+ - Each dashboard generates its own documentation in `output/[dashboard-name].pbit/`
370
+ - Documentation ready to share with the team
371
+ - Automatic identification of measures by category (revenue, cost, margin, etc.)
372
+
373
+ **Output example:**
374
+ ```
375
+ output/
376
+ ├── Sales Dashboard.pbit/
377
+ │ ├── model_documentation.md # 23 documented measures
378
+ │ └── metadata.json
379
+ ├── Finance Dashboard.pbit/
380
+ │ ├── model_documentation.md # 31 documented measures
381
+ │ └── metadata.json
382
+ └── Operations Dashboard.pbit/
383
+ ├── model_documentation.md # 18 documented measures
384
+ └── metadata.json
385
+ ```
386
+
387
+ ---
388
+
389
+ ### 2. New Analyst Onboarding
390
+
391
+ **Problem:** New employees need weeks to understand Power BI model structure and which measures to use for each analysis.
392
+
393
+ **Solution:**
394
+ 1. Generate the model documentation (in your preferred language):
395
+ ```powershell
396
+ # English documentation (default)
397
+ pbi-docs --input "data/pbit/my-model.pbit"
398
+
399
+ # Spanish documentation
400
+ pbi-docs --input "data/pbit/my-model.pbit" --lang es
401
+ ```
402
+
403
+ 2. Upload the `model_documentation.md` file to your favorite AI agent (Claude, GPT-4, etc.)
404
+
405
+ 3. The agent can answer questions like:
406
+ - "What revenue measures are available?"
407
+ - "How is Gross Margin calculated?"
408
+ - "What tables are related to Customer?"
409
+
410
+ **Interaction example:**
411
+ ```
412
+ User: What revenue measures does this model have?
413
+
414
+ Agent: The "my-model" model has 11 revenue measures:
415
+ - Total Revenue (simple): SUM([Revenue])
416
+ - YTD Revenue (simple): TOTALYTD(SUM([Revenue]),'Date'[Date])
417
+ - Revenue SPLY (medium): CALCULATE([Total Revenue],SAMEPERIODLASTYEAR('Date'[Date]))
418
+ - Revenue Budget (medium): CALCULATE([Total Revenue], FILTER(Scenario, Scenario[Scenario]="Budget"))
419
+ ...
420
+ ```
421
+
422
+ **Benefit:** Significant reduction in onboarding time by having immediate answers about the model structure.
423
+
424
+ ---
425
+
426
+ ### 3. Model Auditing
427
+
428
+ **Problem:** You need to compare two versions of the same dashboard to identify which measures or relationships changed between releases.
429
+
430
+ **Solution:**
431
+ ```powershell
432
+ # Compare two versions of the model
433
+ pbi-docs --diff "data/pbit/dashboard_v1.pbit" "data/pbit/dashboard_v2.pbit"
434
+ ```
435
+
436
+ **Result:** A `diff_dashboard_v1_vs_dashboard_v2.json` file is generated, content-aware — not just
437
+ which measures/columns/relationships were added or removed, but which existing ones changed
438
+ content (DAX expression, format string, display folder, hidden flag, category, data type,
439
+ cardinality, cross-filtering, active flag).
440
+
441
+ Identity for matching an object across both models:
442
+ - Measures and columns: `(table, name)`.
443
+ - Relationships: `(from_table, from_column, to_table, to_column)` — a relationship that keeps the
444
+ same connected columns but changes cardinality/cross-filtering/active flag shows up in
445
+ `relationships_modified`, not as a remove+add.
446
+
447
+ DAX changes are flagged `"semantic"` or `"cosmetic"` via a whitespace-insensitive comparison of
448
+ `formatted_expression` — this is a text heuristic (DAX has no whitespace-sensitive syntax, so a
449
+ pure reindent compares equal), not a DAX parser; a change to a comment or to non-functional
450
+ casing would still register as semantic.
451
+
452
+ **Output example (`diff_*.json`):**
453
+ ```json
454
+ {
455
+ "a_model": "dashboard_v1",
456
+ "b_model": "dashboard_v2",
457
+ "measures_added": [["Fact", "New Revenue Measure"]],
458
+ "measures_removed": [["Fact", "Deprecated Measure"]],
459
+ "measures_modified": [
460
+ {
461
+ "table": "Fact",
462
+ "name": "Total Sales",
463
+ "changes": {
464
+ "formatted_expression": {"old": "SUM(Fact[Amount])", "new": "SUM(Fact[NetAmount])", "dax_change": "semantic"},
465
+ "display_folder": {"old": "", "new": "Sales"}
466
+ }
467
+ }
468
+ ],
469
+ "columns_added": [],
470
+ "columns_removed": [],
471
+ "columns_modified": [
472
+ {"table": "Fact", "name": "Amount", "changes": {"data_type": {"old": "int64", "new": "decimal"}}}
473
+ ],
474
+ "relationships_added": [],
475
+ "relationships_removed": [],
476
+ "relationships_modified": [
477
+ {
478
+ "from_table": "Fact", "from_column": "DateKey", "to_table": "Date", "to_column": "Date",
479
+ "changes": {"cardinality": {"old": "many:one", "new": "one:one"}}
480
+ }
481
+ ]
482
+ }
483
+ ```
484
+
485
+ **Impact analysis (`--diff-impact`):** add `--diff-impact` (optionally with `--transitive`) to
486
+ also report which measures reference each removed/modified measure — "what changed, and what
487
+ might break" in one call, connecting this diff to the resolver's `find_measure_usages()`:
488
+
489
+ ```powershell
490
+ pbi-docs --diff "data/pbit/dashboard_v1.pbit" "data/pbit/dashboard_v2.pbit" --diff-impact --transitive
491
+ ```
492
+
493
+ ```json
494
+ {
495
+ "measures_removed_impact": [
496
+ {"table": "Fact", "name": "Deprecated Measure", "used_by": [{"table": "Fact", "name": "Margin %"}]}
497
+ ],
498
+ "measures_modified_impact": [
499
+ {"table": "Fact", "name": "Total Sales", "used_by": [{"table": "Fact", "name": "YTD Sales"}]}
500
+ ]
501
+ }
502
+ ```
503
+ Removed measures are checked for usages in the *old* model (those references just broke);
504
+ modified measures are checked in the *new* model (those callers may now behave differently). The
505
+ same capability is exposed to AI agents as the `diff_impact` MCP tool (see
506
+ [MCP server](docs/use-cases.md#7-mcp-server---mcp-serve) in docs/use-cases.md).
507
+
508
+ **Want this enforced automatically before a commit lands?** See
509
+ **[docs/pre_commit_hook.md](docs/pre_commit_hook.md)** — a reference `git` pre-commit hook
510
+ (under [`githooks/`](githooks/)) for repos that version `.pbip`/`.pbit` models, built on exactly
511
+ the command above.
512
+
513
+ ---
514
+
515
+ More use cases — integrating with AI agents/RAG, chat-invocable Skills for Claude Code and
516
+ GitHub Copilot, the `--query` CLI, and the `--mcp-serve` MCP server — are in
517
+ **[docs/use-cases.md](docs/use-cases.md)**.
518
+
519
+ ---
520
+
521
+ ## Validation
522
+
523
+ Every efficiency/correctness claim in this README is backed by a dated, reproducible report
524
+ against the real `Supply Chain Sample.pbip` fixture (not a synthetic toy model) — read these
525
+ before taking "AI-ready" or "token-optimized" at face value:
526
+
527
+ - **[docs/pbip_validation_report.md](docs/pbip_validation_report.md)** — end-to-end validation of
528
+ PBIP/TMDL extraction and both output formats against a real (non-synthetic) export; documents
529
+ 5 bugs found and fixed in the process.
530
+ - **[docs/token_optimization_report.md](docs/token_optimization_report.md)** — measured token
531
+ cost of raw TMDL vs pbi-docs JSON vs TOON across 3 usage scenarios, plus a table-by-table
532
+ breakdown showing TOON is *not* a uniform win (loses on small tables).
533
+ - **[docs/precision_validation_report.md](docs/precision_validation_report.md)** — automated
534
+ proxy for the "does scoped context sacrifice accuracy?" question: 3 isolated agents answer 18
535
+ objectively-gradable questions using only raw TMDL / only JSON / only `--query`. JSON and
536
+ `--query` both scored 18/18; raw TMDL scored 16/18 (the 2 misses were honest `NOT_FOUND` on a
537
+ field TMDL doesn't contain at all, not agent error). Includes an honest caveat about total
538
+ conversation token overhead vs. raw context-source bytes.
539
+ - **[docs/scale_validation_report.md](docs/scale_validation_report.md)** — behavior at 60
540
+ tables/288 measures (synthetic, since no public enterprise-scale PBIP model exists): confirms
541
+ `--index-format auto` and the resolver still hold up, and is transparent about where a fixed
542
+ per-model cost (`index.json`) stops paying for itself at scale.
543
+ - **[docs/human_validation_protocol.md](docs/human_validation_protocol.md)** *(protocol — not yet
544
+ executed)* — the planned human-subject experiment for validating that scoped context doesn't
545
+ cost real users time or accuracy versus raw file dumps.
546
+
547
+ ---
548
+
549
+ ## DAX Formatting Example
550
+
551
+ The formatter now generates **hierarchical indentation** that reflects the logical structure of expressions:
552
+
553
+ ![DAX Formatting](docs/images/DAX.png)
554
+
555
+ ### Before (unformatted):
556
+ ```dax
557
+ CALCULATE([YTD Gross Margin],SAMEPERIODLASTYEAR(DATESYTD('Date'[Date])))
558
+ ```
559
+
560
+ ### After (hierarchical formatting):
561
+ ```dax
562
+ CALCULATE(
563
+ [YTD Gross Margin],
564
+ SAMEPERIODLASTYEAR(
565
+ DATESYTD(
566
+ 'Date'[Date]
567
+ )
568
+ )
569
+ )
570
+ ```
571
+
572
+ **Formatting features:**
573
+ - Each nesting level increases indentation (+4 spaces)
574
+ - Closing parentheses aligned with the start of their function
575
+ - Arguments on separate lines for clarity
576
+ - Preservation of all original arguments
577
+ - Complexity indicators (Simple / Medium / Complex)
578
+
579
+ ---
580
+
581
+ ## Troubleshooting
582
+
583
+ ### Common Errors
584
+
585
+ **Error decoding `DataModelSchema`**:
586
+ - The script tries `utf-8`, `utf-16` and `latin-1`, plus removes comments/trailing commas.
587
+ - If it fails, `schema_snippet.txt` is saved in the output folder for diagnosis.
588
+
589
+ **Paths with spaces/special characters**:
590
+ - Use quotes in the CLI: `pbi-docs -i "data/My File.pbit"`
591
+ - Prefer paths within `data/`.
592
+
593
+ **No documentation generated**:
594
+ - Verify that the file contains `DataModelSchema`.
595
+ - If you're using `.pbix`, export to `.pbit` from Power BI Desktop (File > Export > Power BI Template).
596
+
597
+ **PowerShell blocks venv activation**:
598
+ - Adjust the `ExecutionPolicy` as indicated in installation.
599
+
600
+ ### Verify Installation
601
+
602
+ Run a quick test with your .pbit file:
603
+
604
+ ```powershell
605
+ pbi-docs --input "data/pbit/my-model.pbit"
606
+ ```
607
+
608
+ **Expected output:**
609
+ ```
610
+ Processing file: data/pbit/my-model.pbit
611
+ Schema extracted successfully: 11 tables
612
+ Metadata processed: 11 tables, 44 measures
613
+ Metadata saved to: output/my-model.pbit/metadata.json
614
+ Documentation saved to: output/my-model.pbit/model_documentation.md
615
+ Agent context saved to: output/my-model.pbit/agent_context.json
616
+ JSONL context saved to: output/my-model.pbit/model_context.jsonl
617
+ Processing completed successfully for: my-model.pbit
618
+ ```
619
+
620
+ Then verify that the files were generated correctly:
621
+
622
+ ```powershell
623
+ # List generated files
624
+ Get-ChildItem "output/my-model.pbit/"
625
+
626
+ # View a summary of the documentation
627
+ Get-Content "output/my-model.pbit/model_documentation.md" | Select-Object -First 10
628
+ ```
629
+
630
+ ### Common Installation Issues
631
+
632
+ **Error: "Python not recognized"**
633
+ ```powershell
634
+ # Add Python to PATH or use full path
635
+ C:\Python312\python.exe -m pip install -e .
636
+ ```
637
+
638
+ **Error: "No module named..."**
639
+ This project requires no dependencies. If you see this error, verify your Python version:
640
+ ```powershell
641
+ python --version # Must be 3.10+
642
+ ```
643
+
644
+ **Error: "pbi-docs not recognized"**
645
+ ```powershell
646
+ # Reinstall the package
647
+ pip install -e .
648
+ # Or use python -m
649
+ python -m pbi_extractor.cli --input file.pbit
650
+ ```
651
+
652
+ ---
653
+
654
+ ## Comparison with Alternatives
655
+
656
+ | Feature | pbi-docs | Power BI Helper | Dataedo | Manual (DAX Studio) |
657
+ |---------|------------|-----------------|---------|---------------------|
658
+ | **Free & Open Source** | Yes | No (Paid) | No (Paid) | Yes |
659
+ | **Batch Processing** | Yes - Multiple files | No - One at a time | Yes | No |
660
+ | **Multi-language Support** | Yes - English & Spanish | No | No | No |
661
+ | **AI-Ready Outputs** | Yes - JSON + JSONL | No | Limited | No |
662
+ | **Hierarchical DAX** | Yes | Basic | Basic | No - Raw only |
663
+ | **No Installation** | Yes - pip install | Limited - Desktop app | Limited - Platform | Yes |
664
+ | **Categorization** | Yes - Auto (revenue, cost...) | No - Manual | Yes | No |
665
+ | **Version Diff** | Yes - Built-in | No | Yes | No - Manual |
666
+ | **MCP Server / On-demand Query** | Yes - Built-in, zero deps | No | No | No |
667
+
668
+ pbi-docs is a read/context-compilation layer, a different category from Microsoft's own Power BI
669
+ MCP servers (Modeling MCP for writes, Remote MCP for DAX execution) — complementary rather than
670
+ competing.
671
+
672
+ ---
673
+
674
+ ## Generated Documentation Example
675
+
676
+ <details>
677
+ <summary>View complete example of model_documentation.md (my-model)</summary>
678
+
679
+ ```markdown
680
+ # my-model - Power BI Data Model
681
+
682
+ **Generated:** 2025-12-22 14:23:29
683
+
684
+ ## Tables and Measures
685
+
686
+ ### Fact *(Hidden Table - Measures Only)*
687
+
688
+ **Measures:**
689
+
690
+ ##### Revenue Measures
691
+
692
+ **Total Revenue** *(simple)*
693
+
694
+ ```dax
695
+ SUM([Revenue])
696
+ ```
697
+
698
+ *Format:* `$#,0;($#,0);$#,0`
699
+
700
+ ---
701
+
702
+ **YTD Revenue** *(simple)*
703
+
704
+ ```dax
705
+ TOTALYTD(
706
+ SUM([Revenue]),
707
+ 'Date'[Date])
708
+ ```
709
+
710
+ *Format:* `$#,0;($#,0);$#,0`
711
+
712
+ ---
713
+
714
+ **Revenue SPLY** *(medium)*
715
+
716
+ ```dax
717
+ CALCULATE(
718
+ [Total Revenue],
719
+ SAMEPERIODLASTYEAR(
720
+ 'Date'[Date]))
721
+ ```
722
+
723
+ *Format:* `$#,0;($#,0);$#,0`
724
+
725
+ ##### Margin Measures
726
+
727
+ **Gross Margin** *(simple)*
728
+
729
+ ```dax
730
+ [Total Revenue]-[Total COGS]
731
+ ```
732
+
733
+ *Format:* `$#,0;($#,0);$#,0`
734
+
735
+ ##### Percentage Measures
736
+
737
+ **GM%** *(simple)*
738
+
739
+ ```dax
740
+ DIVIDE(
741
+ [Gross Margin],
742
+ [Total Revenue])
743
+ ```
744
+
745
+ *Format:* `0.0 %;-0.0 %;0.0 %`
746
+
747
+ ### Date
748
+
749
+ **Columns:**
750
+
751
+ | Column | Type | Category |
752
+ |--------|------|----------|
753
+ | `Date` | dateTime | temporal |
754
+ | `Year` | int64 | numeric |
755
+ | `Month` | string | categorical |
756
+
757
+ ### Customer
758
+
759
+ **Columns:**
760
+
761
+ | Column | Type | Category |
762
+ |--------|------|----------|
763
+ | `Name` | string | descriptive |
764
+ | `City` | string | categorical |
765
+ | `State` | string | categorical |
766
+ | `Country/Region` | string | categorical |
767
+
768
+ ## Relationships
769
+
770
+ | From | To | Type | Direction |
771
+ |------|----|----- |-----------|
772
+ | Fact.BU Key | BU.BU Key | many:one | OneDirection |
773
+ | Fact.YearPeriod | Date.YearPeriod | many:one | OneDirection |
774
+ | Fact.Customer Key | Customer.Customer | many:one | OneDirection |
775
+ ```
776
+
777
+ </details>
778
+
779
+ ---
780
+
781
+ ## Contributing
782
+
783
+ Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for the workflow, or use
784
+ [GitHub Issues](https://github.com/Osc2405/pbi-docs/issues) for bug reports and feature requests.
785
+
786
+ ---
787
+
788
+ ## Roadmap
789
+
790
+ The read/context layer (PBIP/TMDL support, indexed output, query resolver, MCP server) is done.
791
+ Writing TMDL models and PBIR/report parsing are deliberately deferred — bigger undertakings, and
792
+ (for writing) Microsoft's own Modeling MCP already covers that space.
793
+
794
+ ---
795
+
796
+ ## Sample Files
797
+
798
+ The example files referenced in this documentation are official sample files provided by Microsoft. You can find these and other Power BI sample files in the [Microsoft Power BI Desktop Samples repository](https://github.com/microsoft/powerbi-desktop-samples).
799
+
800
+ These sample files are excellent for:
801
+ - Testing pbi-docs functionality
802
+ - Learning Power BI data modeling
803
+ - Exploring different DAX patterns and measure types
804
+ - Understanding relationship structures
805
+
806
+ To use these samples:
807
+ 1. Clone or download the repository: `git clone https://github.com/microsoft/powerbi-desktop-samples.git`
808
+ 2. Open the `.pbix` files in Power BI Desktop
809
+ 3. Export them as `.pbit` files (File > Export > Power BI Template)
810
+ 4. Use them with pbi-docs to generate documentation
811
+
812
+ ---
813
+
814
+ ## Author
815
+
816
+ **Oscar Rosero** - Data Engineer | BI Developer
817
+
818
+ - [LinkedIn](https://www.linkedin.com/in/oscrosero24/)
819
+ - [GitHub](https://github.com/Osc2405)
820
+ - orosero2405@gmail.com
821
+
822
+ Built with ❤️ for the Power BI community.
823
+
824
+ ---
825
+
826
+ ## Star History
827
+
828
+ If this tool saves you time, please consider giving it a ⭐ on GitHub!
829
+
830
+