pbi-docs 1.0.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pbi_docs-1.0.0.dist-info/METADATA +830 -0
- pbi_docs-1.0.0.dist-info/RECORD +21 -0
- pbi_docs-1.0.0.dist-info/WHEEL +5 -0
- pbi_docs-1.0.0.dist-info/entry_points.txt +2 -0
- pbi_docs-1.0.0.dist-info/licenses/LICENSE +23 -0
- pbi_docs-1.0.0.dist-info/top_level.txt +1 -0
- pbi_extractor/__init__.py +33 -0
- pbi_extractor/categorizer.py +63 -0
- pbi_extractor/cli.py +458 -0
- pbi_extractor/diff.py +195 -0
- pbi_extractor/documentation.py +187 -0
- pbi_extractor/extractor.py +196 -0
- pbi_extractor/formatters.py +354 -0
- pbi_extractor/i18n.py +181 -0
- pbi_extractor/indexed_output.py +238 -0
- pbi_extractor/jsonl_generator.py +179 -0
- pbi_extractor/mcp_server.py +272 -0
- pbi_extractor/pbip_extractor.py +469 -0
- pbi_extractor/processor.py +261 -0
- pbi_extractor/resolver.py +409 -0
- pbi_extractor/toon_encoder.py +59 -0
|
@@ -0,0 +1,830 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pbi-docs
|
|
3
|
+
Version: 1.0.0
|
|
4
|
+
Summary: AI context engine for Power BI models: indexed, queryable context for LLM agents (MCP server, CLI, docs), zero dependencies
|
|
5
|
+
Author-email: Oscar Rosero <orosero2405@gmail.com>
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/Osc2405/pbi-docs
|
|
8
|
+
Project-URL: Repository, https://github.com/Osc2405/pbi-docs
|
|
9
|
+
Project-URL: Issues, https://github.com/Osc2405/pbi-docs/issues
|
|
10
|
+
Project-URL: Changelog, https://github.com/Osc2405/pbi-docs/blob/main/CHANGELOG.md
|
|
11
|
+
Keywords: powerbi,metadata,dax,documentation,bi,pbit,mcp,ai-agent,llm-context
|
|
12
|
+
Classifier: Development Status :: 4 - Beta
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
18
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
19
|
+
Classifier: Operating System :: OS Independent
|
|
20
|
+
Classifier: Intended Audience :: Developers
|
|
21
|
+
Classifier: Intended Audience :: Information Technology
|
|
22
|
+
Classifier: Topic :: Documentation
|
|
23
|
+
Classifier: Topic :: Utilities
|
|
24
|
+
Requires-Python: >=3.10
|
|
25
|
+
Description-Content-Type: text/markdown
|
|
26
|
+
License-File: LICENSE
|
|
27
|
+
Provides-Extra: dev
|
|
28
|
+
Requires-Dist: pytest>=7.0.0; extra == "dev"
|
|
29
|
+
Dynamic: license-file
|
|
30
|
+
|
|
31
|
+
## pbi-docs — AI Context Engine for Power BI Models
|
|
32
|
+
|
|
33
|
+
[](https://github.com/Osc2405/pbi-docs/actions/workflows/tests.yml)
|
|
34
|
+
[](https://www.python.org/)
|
|
35
|
+
[](LICENSE)
|
|
36
|
+
[](https://github.com/Osc2405/pbi-docs/actions/workflows/tests.yml)
|
|
37
|
+
|
|
38
|
+
**Turns a Power BI model (`.pbit` or the new `.pbip`/TMDL format) into documentation and
|
|
39
|
+
context an AI agent can actually use** — human-readable Markdown, indexed JSON for LLMs/RAG, a
|
|
40
|
+
query CLI, and a read-only MCP server. Zero external dependencies.
|
|
41
|
+
|
|
42
|
+
**Who it's for:** data engineers documenting dashboards, consultants auditing models they didn't
|
|
43
|
+
build, and anyone connecting an AI agent (Claude, GPT, Copilot) to a Power BI model's structure.
|
|
44
|
+
|
|
45
|
+
### Demo
|
|
46
|
+
|
|
47
|
+

|
|
48
|
+

|
|
49
|
+

|
|
50
|
+
|
|
51
|
+
## Quick Start
|
|
52
|
+
|
|
53
|
+
```powershell
|
|
54
|
+
git clone https://github.com/Osc2405/pbi-docs.git
|
|
55
|
+
cd pbi-docs
|
|
56
|
+
pip install -e .
|
|
57
|
+
|
|
58
|
+
# From a .pbit file...
|
|
59
|
+
pbi-docs --input "data/pbit/my-model.pbit"
|
|
60
|
+
# ...or a PBIP project (folder, .pbip marker, or .SemanticModel/ — auto-detected)
|
|
61
|
+
pbi-docs --input "data/pbip/my-model/"
|
|
62
|
+
|
|
63
|
+
Get-Content "output/my-model.pbit/model_documentation.md"
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
**Result:** 7 files in `output/<model-name>/` in seconds — human-readable Markdown, JSON/JSONL
|
|
67
|
+
context for AI agents, and an indexed, queryable version for large models.
|
|
68
|
+
|
|
69
|
+
<details>
|
|
70
|
+
<summary><strong>Full walkthrough</strong> — folder structure, expected output, and using the context in Python</summary>
|
|
71
|
+
|
|
72
|
+
The command generates a folder in `output/` with all documentation files:
|
|
73
|
+
|
|
74
|
+
```
|
|
75
|
+
output/my-model.pbit/ (or output/my-model/ for PBIP)
|
|
76
|
+
├── metadata.json # Structured model metadata
|
|
77
|
+
├── model_documentation.md # Human-readable documentation
|
|
78
|
+
├── agent_context.json # LLM-optimized context (top-20 measures)
|
|
79
|
+
├── model_context.jsonl # JSONL format for embeddings/RAG
|
|
80
|
+
├── index.json # Lightweight index + pointers
|
|
81
|
+
├── relationships.json # All relationships
|
|
82
|
+
└── tables/
|
|
83
|
+
├── Sales.json # Full detail per table
|
|
84
|
+
└── ...
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
```powershell
|
|
88
|
+
Get-Content "output/my-model.pbit/model_documentation.md" | Select-Object -First 15
|
|
89
|
+
```
|
|
90
|
+
```markdown
|
|
91
|
+
# my-model - Power BI Data Model
|
|
92
|
+
|
|
93
|
+
**Generated:** 2025-12-22 14:23:29
|
|
94
|
+
|
|
95
|
+
## Model Summary
|
|
96
|
+
|
|
97
|
+
- **Business Tables:** 9
|
|
98
|
+
- **Total Columns:** 23
|
|
99
|
+
- **Total Measures:** 44
|
|
100
|
+
- **Relationships:** 9
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Use the JSON context directly in Python (or point an AI agent at it via `--query` or
|
|
104
|
+
`--mcp-serve` — see [Use Cases](#use-cases) below):
|
|
105
|
+
|
|
106
|
+
```python
|
|
107
|
+
import json
|
|
108
|
+
|
|
109
|
+
with open("output/my-model.pbit/agent_context.json", "r", encoding="utf-8") as f:
|
|
110
|
+
context = json.load(f)
|
|
111
|
+
|
|
112
|
+
print(f"Model: {context['model_name']}")
|
|
113
|
+
print(f"Key measures: {len(context['key_measures'])}")
|
|
114
|
+
print(f"First measure: {context['key_measures'][0]['name']}")
|
|
115
|
+
```
|
|
116
|
+
```
|
|
117
|
+
Model: my-model
|
|
118
|
+
Key measures: 20
|
|
119
|
+
First measure: Revenue Budget
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
</details>
|
|
123
|
+
|
|
124
|
+
## Why pbi-docs?
|
|
125
|
+
|
|
126
|
+
| Your Need | pbi-docs Solution |
|
|
127
|
+
|-----------|---------------------|
|
|
128
|
+
| **Document 10+ dashboards fast** | Batch processing with `--batch` |
|
|
129
|
+
| **Support the new PBIP format** | Full TMDL parser, auto-detected from `.pbip` or folder |
|
|
130
|
+
| **Train AI agents on your models** | Indexed JSON/JSONL context, a query CLI, and an MCP server |
|
|
131
|
+
| **Let an AI agent query the model live** | Read-only MCP server (`--mcp-serve`) — validated against a test harness, not yet a live MCP client, see [MCP server](docs/use-cases.md#7-mcp-server---mcp-serve) |
|
|
132
|
+
| **Use it from your AI coding assistant** | Chat-invocable Skill for Claude Code + prompt file for GitHub Copilot |
|
|
133
|
+
| **Actually readable DAX** | Hierarchical indentation (4x better than raw) |
|
|
134
|
+
| **Compare model versions** | Content-aware `--diff`, with impact analysis (`--diff-impact`) |
|
|
135
|
+
| **Zero-cost, zero-install** | Python-only, no .NET dependencies |
|
|
136
|
+
|
|
137
|
+
**Perfect for:** Data engineers onboarding teams, consultants auditing models, organizations building AI copilots for BI.
|
|
138
|
+
|
|
139
|
+
## Project Status
|
|
140
|
+
|
|
141
|
+
The read/context layer — PBIP/TMDL support, indexed output, query resolver, MCP server — is
|
|
142
|
+
implemented and tested (222 tests). Every claim above is backed by a dated, reproducible report,
|
|
143
|
+
not just asserted: see [Validation](#validation) below. Writing/editing TMDL models and PBIR/report-
|
|
144
|
+
layer parsing are deliberately out of scope for now (see `CHANGELOG.md` and the
|
|
145
|
+
[Roadmap](#roadmap) for why).
|
|
146
|
+
|
|
147
|
+
## Requirements
|
|
148
|
+
- Python 3.10+ (3.12 recommended)
|
|
149
|
+
- Windows PowerShell (instructions include Windows commands)
|
|
150
|
+
|
|
151
|
+
Optional: virtual environment (`venv`). No external libraries required.
|
|
152
|
+
|
|
153
|
+
## Installation (Windows/PowerShell)
|
|
154
|
+
|
|
155
|
+
```powershell
|
|
156
|
+
# 1) Clone or download the repository
|
|
157
|
+
# 2) (Optional) Create and activate virtual environment
|
|
158
|
+
python -m venv venv
|
|
159
|
+
./venv/Scripts/Activate.ps1
|
|
160
|
+
|
|
161
|
+
# 3) Editable installation (development)
|
|
162
|
+
pip install -e .
|
|
163
|
+
|
|
164
|
+
# Verify Python version
|
|
165
|
+
python --version
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
If PowerShell blocks activation, run as Administrator:
|
|
169
|
+
|
|
170
|
+
```powershell
|
|
171
|
+
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
---
|
|
175
|
+
|
|
176
|
+
## Quick Usage
|
|
177
|
+
|
|
178
|
+
### Basic Commands
|
|
179
|
+
|
|
180
|
+
**Process a `.pbit` file:**
|
|
181
|
+
```powershell
|
|
182
|
+
pbi-docs --input "data/pbit/my-model.pbit"
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
**Process a PBIP project (new in v1.0):**
|
|
186
|
+
```powershell
|
|
187
|
+
# From the .pbip marker file
|
|
188
|
+
pbi-docs --input "data/pbip/my-model.pbip"
|
|
189
|
+
|
|
190
|
+
# From the .SemanticModel folder directly
|
|
191
|
+
pbi-docs --input "data/pbip/my-model.SemanticModel"
|
|
192
|
+
|
|
193
|
+
# From the project root folder (auto-detected)
|
|
194
|
+
pbi-docs --input "data/pbip/my-model/"
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
**Specify custom output directory:**
|
|
198
|
+
```powershell
|
|
199
|
+
pbi-docs -i "data/pbit/my-model.pbit" -o "my-results"
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
**Process multiple files (batch mode — mixed formats supported):**
|
|
203
|
+
```powershell
|
|
204
|
+
pbi-docs --batch "data/pbit/*.pbit"
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+

|
|
208
|
+
|
|
209
|
+
**Compare two versions of a model (mixed `.pbit`/`.pbip` supported):**
|
|
210
|
+
```powershell
|
|
211
|
+
pbi-docs --diff "data/pbit/model_v1.pbit" "data/pbip/model_v2/"
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
**Verbose mode (more debugging information):**
|
|
215
|
+
```powershell
|
|
216
|
+
pbi-docs --input "data/pbit/my-model.pbit" --verbose
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
**Human-readable indexed output (indented JSON, for debugging — compact by default):**
|
|
220
|
+
```powershell
|
|
221
|
+
pbi-docs --input "data/pbit/my-model.pbit" --pretty
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
**Generate documentation in Spanish:**
|
|
225
|
+
```powershell
|
|
226
|
+
pbi-docs --input "data/pbit/my-model.pbit" --lang es
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
**Generate documentation in English (default):**
|
|
230
|
+
```powershell
|
|
231
|
+
pbi-docs --input "data/pbit/my-model.pbit" --lang en
|
|
232
|
+
# Or simply omit --lang (English is the default)
|
|
233
|
+
pbi-docs --input "data/pbit/my-model.pbit"
|
|
234
|
+
```
|
|
235
|
+
|
|
236
|
+
### Expected Output
|
|
237
|
+
|
|
238
|
+
When running the command, you'll see messages like:
|
|
239
|
+
|
|
240
|
+
```
|
|
241
|
+
Processing file: data/pbit/my-model.pbit
|
|
242
|
+
Schema extracted successfully: 11 tables
|
|
243
|
+
Metadata processed: 11 tables, 44 measures
|
|
244
|
+
Metadata saved to: output/my-model.pbit/metadata.json
|
|
245
|
+
Documentation saved to: output/my-model.pbit/model_documentation.md
|
|
246
|
+
Agent context saved to: output/my-model.pbit/agent_context.json
|
|
247
|
+
JSONL context saved to: output/my-model.pbit/model_context.jsonl
|
|
248
|
+
Processing completed successfully for: my-model.pbit
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
### Important Notes
|
|
252
|
+
|
|
253
|
+
- **Supported formats:** `.pbit` files (ZIP + JSON TMSL) and `.pbip` projects (TMDL folder structure). `.pbix` files must be exported to `.pbit` from Power BI Desktop (File > Export > Power BI Template).
|
|
254
|
+
- **PBIP entry points:** The `--input` flag accepts a `.pbip` marker file, a `.SemanticModel/` folder, or a project root folder. Format is auto-detected.
|
|
255
|
+
- **Language selection:** Use `--lang en` for English (default) or `--lang es` for Spanish. The language affects the generated `model_documentation.md` and `agent_context.json` files.
|
|
256
|
+
- **Paths with spaces:** Use quotes around paths that contain spaces.
|
|
257
|
+
- **Recommended paths:** Place your files in `data/` or `data/pbit/` to keep the project organized.
|
|
258
|
+
|
|
259
|
+
---
|
|
260
|
+
|
|
261
|
+
## Project Structure
|
|
262
|
+
|
|
263
|
+
```
|
|
264
|
+
pbi-docs/
|
|
265
|
+
├── pbi_extractor/ # Main package
|
|
266
|
+
│ ├── __init__.py
|
|
267
|
+
│ ├── cli.py # CLI with argparse (auto-detection, --index-format)
|
|
268
|
+
│ ├── extractor.py # .pbit (ZIP+JSON TMSL) extractor
|
|
269
|
+
│ ├── pbip_extractor.py # .pbip / TMDL extractor (new in v1.0)
|
|
270
|
+
│ ├── processor.py # Metadata processing (format-agnostic)
|
|
271
|
+
│ ├── indexed_output.py # index.json + tables/*.json writer (new in v1.0)
|
|
272
|
+
│ ├── formatters.py # Advanced hierarchical DAX formatting
|
|
273
|
+
│ ├── categorizer.py # Table/measure categorization
|
|
274
|
+
│ ├── documentation.py # Markdown generation
|
|
275
|
+
│ ├── diff.py # Model comparison
|
|
276
|
+
│ ├── jsonl_generator.py # JSONL generator for LLMs
|
|
277
|
+
│ └── i18n.py # Translations (en/es)
|
|
278
|
+
├── tests/
|
|
279
|
+
│ ├── fixtures/
|
|
280
|
+
│ │ └── minimal_pbip/ # TMDL test fixtures (new in v1.0)
|
|
281
|
+
│ ├── test_pbip_extractor.py
|
|
282
|
+
│ ├── test_cli_detection.py
|
|
283
|
+
│ ├── test_indexed_output.py
|
|
284
|
+
│ ├── test_categorizer.py
|
|
285
|
+
│ ├── test_i18n.py
|
|
286
|
+
│ └── test_processor_and_context.py
|
|
287
|
+
├── githooks/ # Reference pre-commit hook (docs/pre_commit_hook.md)
|
|
288
|
+
├── data/ # Input model files
|
|
289
|
+
├── output/ # Generated results
|
|
290
|
+
├── pyproject.toml # Package configuration
|
|
291
|
+
├── README.md
|
|
292
|
+
├── CHANGELOG.md
|
|
293
|
+
└── LICENSE
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
---
|
|
297
|
+
|
|
298
|
+
## Generated Outputs
|
|
299
|
+
|
|
300
|
+
After running the command, a folder is created in `output/` with the model name. Inside you'll find:
|
|
301
|
+
|
|
302
|
+
- **`metadata.json`**
|
|
303
|
+
- `summary`: totals of tables, visible columns, visible measures and relationships.
|
|
304
|
+
- `tables`: each table with `columns` (type, visibility, category) and `measures` (clean expression, format, display folder, category).
|
|
305
|
+
- `relationships`: from/to, cardinality, direction and active status.
|
|
306
|
+
|
|
307
|
+
- **`model_documentation.md`**
|
|
308
|
+
- Model summary (language depends on `--lang` flag, default: English).
|
|
309
|
+
- List of tables (hidden or business), visible columns and measures grouped by category: revenue, cost, margin, percentage, ratio, temporal, etc.
|
|
310
|
+
- Sections with DAX expressions formatted with **hierarchical indentation**.
|
|
311
|
+
- Relationships table with visual representation of table connections.
|
|
312
|
+
- AI Agent Usage Guide with sample questions (translated based on selected language).
|
|
313
|
+
|
|
314
|
+

|
|
315
|
+
|
|
316
|
+
- **`agent_context.json`**
|
|
317
|
+
- Model name, totals, available tables, key measures (up to 20), temporal columns and sample questions (language depends on `--lang` flag, default: English).
|
|
318
|
+
|
|
319
|
+
- **`model_context.jsonl`**
|
|
320
|
+
- Line-delimited JSON format optimized for embeddings and RAG.
|
|
321
|
+
- Each line is an independent object (table, measure or relationship).
|
|
322
|
+
- Includes formatted DAX and sample prompts.
|
|
323
|
+
|
|
324
|
+
- **`index.json`** *(new in v1.0)*
|
|
325
|
+
- Lightweight model summary with per-table metadata (column/measure counts, categories) and relative paths to all other output files.
|
|
326
|
+
- Allows LLM agents to navigate large models without loading the full `metadata.json`.
|
|
327
|
+
|
|
328
|
+
- **`relationships.json`** *(new in v1.0)*
|
|
329
|
+
- All model relationships in a single focused file.
|
|
330
|
+
|
|
331
|
+
- **`tables/<TableName>.json`** *(new in v1.0)*
|
|
332
|
+
- Full detail for one table (columns + measures including DAX).
|
|
333
|
+
- One file per table, addressable via `index.json`.
|
|
334
|
+
|
|
335
|
+
---
|
|
336
|
+
|
|
337
|
+
## `index.json` — open format specification
|
|
338
|
+
|
|
339
|
+
`index.json` is a small, stable contract meant to be consumed directly by any tool — not just
|
|
340
|
+
pbi-docs' own CLI/resolver/MCP server: a lightweight per-table summary (name, column/measure
|
|
341
|
+
counts, categories, format, and a relative path to that table's detail file) plus pointers to
|
|
342
|
+
every other output file, so an agent can navigate a large model without loading `metadata.json`.
|
|
343
|
+
By default it's written **compact** (no indentation); pass `--pretty` for indented JSON.
|
|
344
|
+
|
|
345
|
+
Full field-by-field contract — top-level shape, per-table entry, the `tables/<Name>.json` JSON vs.
|
|
346
|
+
TOON shapes, and the versioning policy — is documented in
|
|
347
|
+
**[docs/index-json-spec.md](docs/index-json-spec.md)**.
|
|
348
|
+
|
|
349
|
+
---
|
|
350
|
+
|
|
351
|
+
## Use Cases
|
|
352
|
+
|
|
353
|
+
### 1. Automatic Dashboard Documentation
|
|
354
|
+
|
|
355
|
+
**Problem:** Your company has multiple undocumented Power BI dashboards. Analysts waste time searching for which measures to use and how tables are related.
|
|
356
|
+
|
|
357
|
+
**Solution:**
|
|
358
|
+
```powershell
|
|
359
|
+
# Process all dashboards in a folder (English documentation)
|
|
360
|
+
pbi-docs --batch "data/dashboards/*.pbit"
|
|
361
|
+
|
|
362
|
+
# Or generate Spanish documentation for all dashboards
|
|
363
|
+
pbi-docs --batch "data/dashboards/*.pbit" --lang es
|
|
364
|
+
```
|
|
365
|
+
|
|
366
|
+

|
|
367
|
+
|
|
368
|
+
**Result:**
|
|
369
|
+
- Each dashboard generates its own documentation in `output/[dashboard-name].pbit/`
|
|
370
|
+
- Documentation ready to share with the team
|
|
371
|
+
- Automatic identification of measures by category (revenue, cost, margin, etc.)
|
|
372
|
+
|
|
373
|
+
**Output example:**
|
|
374
|
+
```
|
|
375
|
+
output/
|
|
376
|
+
├── Sales Dashboard.pbit/
|
|
377
|
+
│ ├── model_documentation.md # 23 documented measures
|
|
378
|
+
│ └── metadata.json
|
|
379
|
+
├── Finance Dashboard.pbit/
|
|
380
|
+
│ ├── model_documentation.md # 31 documented measures
|
|
381
|
+
│ └── metadata.json
|
|
382
|
+
└── Operations Dashboard.pbit/
|
|
383
|
+
├── model_documentation.md # 18 documented measures
|
|
384
|
+
└── metadata.json
|
|
385
|
+
```
|
|
386
|
+
|
|
387
|
+
---
|
|
388
|
+
|
|
389
|
+
### 2. New Analyst Onboarding
|
|
390
|
+
|
|
391
|
+
**Problem:** New employees need weeks to understand Power BI model structure and which measures to use for each analysis.
|
|
392
|
+
|
|
393
|
+
**Solution:**
|
|
394
|
+
1. Generate the model documentation (in your preferred language):
|
|
395
|
+
```powershell
|
|
396
|
+
# English documentation (default)
|
|
397
|
+
pbi-docs --input "data/pbit/my-model.pbit"
|
|
398
|
+
|
|
399
|
+
# Spanish documentation
|
|
400
|
+
pbi-docs --input "data/pbit/my-model.pbit" --lang es
|
|
401
|
+
```
|
|
402
|
+
|
|
403
|
+
2. Upload the `model_documentation.md` file to your favorite AI agent (Claude, GPT-4, etc.)
|
|
404
|
+
|
|
405
|
+
3. The agent can answer questions like:
|
|
406
|
+
- "What revenue measures are available?"
|
|
407
|
+
- "How is Gross Margin calculated?"
|
|
408
|
+
- "What tables are related to Customer?"
|
|
409
|
+
|
|
410
|
+
**Interaction example:**
|
|
411
|
+
```
|
|
412
|
+
User: What revenue measures does this model have?
|
|
413
|
+
|
|
414
|
+
Agent: The "my-model" model has 11 revenue measures:
|
|
415
|
+
- Total Revenue (simple): SUM([Revenue])
|
|
416
|
+
- YTD Revenue (simple): TOTALYTD(SUM([Revenue]),'Date'[Date])
|
|
417
|
+
- Revenue SPLY (medium): CALCULATE([Total Revenue],SAMEPERIODLASTYEAR('Date'[Date]))
|
|
418
|
+
- Revenue Budget (medium): CALCULATE([Total Revenue], FILTER(Scenario, Scenario[Scenario]="Budget"))
|
|
419
|
+
...
|
|
420
|
+
```
|
|
421
|
+
|
|
422
|
+
**Benefit:** Significant reduction in onboarding time by having immediate answers about the model structure.
|
|
423
|
+
|
|
424
|
+
---
|
|
425
|
+
|
|
426
|
+
### 3. Model Auditing
|
|
427
|
+
|
|
428
|
+
**Problem:** You need to compare two versions of the same dashboard to identify which measures or relationships changed between releases.
|
|
429
|
+
|
|
430
|
+
**Solution:**
|
|
431
|
+
```powershell
|
|
432
|
+
# Compare two versions of the model
|
|
433
|
+
pbi-docs --diff "data/pbit/dashboard_v1.pbit" "data/pbit/dashboard_v2.pbit"
|
|
434
|
+
```
|
|
435
|
+
|
|
436
|
+
**Result:** A `diff_dashboard_v1_vs_dashboard_v2.json` file is generated, content-aware — not just
|
|
437
|
+
which measures/columns/relationships were added or removed, but which existing ones changed
|
|
438
|
+
content (DAX expression, format string, display folder, hidden flag, category, data type,
|
|
439
|
+
cardinality, cross-filtering, active flag).
|
|
440
|
+
|
|
441
|
+
Identity for matching an object across both models:
|
|
442
|
+
- Measures and columns: `(table, name)`.
|
|
443
|
+
- Relationships: `(from_table, from_column, to_table, to_column)` — a relationship that keeps the
|
|
444
|
+
same connected columns but changes cardinality/cross-filtering/active flag shows up in
|
|
445
|
+
`relationships_modified`, not as a remove+add.
|
|
446
|
+
|
|
447
|
+
DAX changes are flagged `"semantic"` or `"cosmetic"` via a whitespace-insensitive comparison of
|
|
448
|
+
`formatted_expression` — this is a text heuristic (DAX has no whitespace-sensitive syntax, so a
|
|
449
|
+
pure reindent compares equal), not a DAX parser; a change to a comment or to non-functional
|
|
450
|
+
casing would still register as semantic.
|
|
451
|
+
|
|
452
|
+
**Output example (`diff_*.json`):**
|
|
453
|
+
```json
|
|
454
|
+
{
|
|
455
|
+
"a_model": "dashboard_v1",
|
|
456
|
+
"b_model": "dashboard_v2",
|
|
457
|
+
"measures_added": [["Fact", "New Revenue Measure"]],
|
|
458
|
+
"measures_removed": [["Fact", "Deprecated Measure"]],
|
|
459
|
+
"measures_modified": [
|
|
460
|
+
{
|
|
461
|
+
"table": "Fact",
|
|
462
|
+
"name": "Total Sales",
|
|
463
|
+
"changes": {
|
|
464
|
+
"formatted_expression": {"old": "SUM(Fact[Amount])", "new": "SUM(Fact[NetAmount])", "dax_change": "semantic"},
|
|
465
|
+
"display_folder": {"old": "", "new": "Sales"}
|
|
466
|
+
}
|
|
467
|
+
}
|
|
468
|
+
],
|
|
469
|
+
"columns_added": [],
|
|
470
|
+
"columns_removed": [],
|
|
471
|
+
"columns_modified": [
|
|
472
|
+
{"table": "Fact", "name": "Amount", "changes": {"data_type": {"old": "int64", "new": "decimal"}}}
|
|
473
|
+
],
|
|
474
|
+
"relationships_added": [],
|
|
475
|
+
"relationships_removed": [],
|
|
476
|
+
"relationships_modified": [
|
|
477
|
+
{
|
|
478
|
+
"from_table": "Fact", "from_column": "DateKey", "to_table": "Date", "to_column": "Date",
|
|
479
|
+
"changes": {"cardinality": {"old": "many:one", "new": "one:one"}}
|
|
480
|
+
}
|
|
481
|
+
]
|
|
482
|
+
}
|
|
483
|
+
```
|
|
484
|
+
|
|
485
|
+
**Impact analysis (`--diff-impact`):** add `--diff-impact` (optionally with `--transitive`) to
|
|
486
|
+
also report which measures reference each removed/modified measure — "what changed, and what
|
|
487
|
+
might break" in one call, connecting this diff to the resolver's `find_measure_usages()`:
|
|
488
|
+
|
|
489
|
+
```powershell
|
|
490
|
+
pbi-docs --diff "data/pbit/dashboard_v1.pbit" "data/pbit/dashboard_v2.pbit" --diff-impact --transitive
|
|
491
|
+
```
|
|
492
|
+
|
|
493
|
+
```json
|
|
494
|
+
{
|
|
495
|
+
"measures_removed_impact": [
|
|
496
|
+
{"table": "Fact", "name": "Deprecated Measure", "used_by": [{"table": "Fact", "name": "Margin %"}]}
|
|
497
|
+
],
|
|
498
|
+
"measures_modified_impact": [
|
|
499
|
+
{"table": "Fact", "name": "Total Sales", "used_by": [{"table": "Fact", "name": "YTD Sales"}]}
|
|
500
|
+
]
|
|
501
|
+
}
|
|
502
|
+
```
|
|
503
|
+
Removed measures are checked for usages in the *old* model (those references just broke);
|
|
504
|
+
modified measures are checked in the *new* model (those callers may now behave differently). The
|
|
505
|
+
same capability is exposed to AI agents as the `diff_impact` MCP tool (see
|
|
506
|
+
[MCP server](docs/use-cases.md#7-mcp-server---mcp-serve) in docs/use-cases.md).
|
|
507
|
+
|
|
508
|
+
**Want this enforced automatically before a commit lands?** See
|
|
509
|
+
**[docs/pre_commit_hook.md](docs/pre_commit_hook.md)** — a reference `git` pre-commit hook
|
|
510
|
+
(under [`githooks/`](githooks/)) for repos that version `.pbip`/`.pbit` models, built on exactly
|
|
511
|
+
the command above.
|
|
512
|
+
|
|
513
|
+
---
|
|
514
|
+
|
|
515
|
+
More use cases — integrating with AI agents/RAG, chat-invocable Skills for Claude Code and
|
|
516
|
+
GitHub Copilot, the `--query` CLI, and the `--mcp-serve` MCP server — are in
|
|
517
|
+
**[docs/use-cases.md](docs/use-cases.md)**.
|
|
518
|
+
|
|
519
|
+
---
|
|
520
|
+
|
|
521
|
+
## Validation
|
|
522
|
+
|
|
523
|
+
Every efficiency/correctness claim in this README is backed by a dated, reproducible report
|
|
524
|
+
against the real `Supply Chain Sample.pbip` fixture (not a synthetic toy model) — read these
|
|
525
|
+
before taking "AI-ready" or "token-optimized" at face value:
|
|
526
|
+
|
|
527
|
+
- **[docs/pbip_validation_report.md](docs/pbip_validation_report.md)** — end-to-end validation of
|
|
528
|
+
PBIP/TMDL extraction and both output formats against a real (non-synthetic) export; documents
|
|
529
|
+
5 bugs found and fixed in the process.
|
|
530
|
+
- **[docs/token_optimization_report.md](docs/token_optimization_report.md)** — measured token
|
|
531
|
+
cost of raw TMDL vs pbi-docs JSON vs TOON across 3 usage scenarios, plus a table-by-table
|
|
532
|
+
breakdown showing TOON is *not* a uniform win (loses on small tables).
|
|
533
|
+
- **[docs/precision_validation_report.md](docs/precision_validation_report.md)** — automated
|
|
534
|
+
proxy for the "does scoped context sacrifice accuracy?" question: 3 isolated agents answer 18
|
|
535
|
+
objectively-gradable questions using only raw TMDL / only JSON / only `--query`. JSON and
|
|
536
|
+
`--query` both scored 18/18; raw TMDL scored 16/18 (the 2 misses were honest `NOT_FOUND` on a
|
|
537
|
+
field TMDL doesn't contain at all, not agent error). Includes an honest caveat about total
|
|
538
|
+
conversation token overhead vs. raw context-source bytes.
|
|
539
|
+
- **[docs/scale_validation_report.md](docs/scale_validation_report.md)** — behavior at 60
|
|
540
|
+
tables/288 measures (synthetic, since no public enterprise-scale PBIP model exists): confirms
|
|
541
|
+
`--index-format auto` and the resolver still hold up, and is transparent about where a fixed
|
|
542
|
+
per-model cost (`index.json`) stops paying for itself at scale.
|
|
543
|
+
- **[docs/human_validation_protocol.md](docs/human_validation_protocol.md)** *(protocol — not yet
|
|
544
|
+
executed)* — the planned human-subject experiment for validating that scoped context doesn't
|
|
545
|
+
cost real users time or accuracy versus raw file dumps.
|
|
546
|
+
|
|
547
|
+
---
|
|
548
|
+
|
|
549
|
+
## DAX Formatting Example
|
|
550
|
+
|
|
551
|
+
The formatter now generates **hierarchical indentation** that reflects the logical structure of expressions:
|
|
552
|
+
|
|
553
|
+

|
|
554
|
+
|
|
555
|
+
### Before (unformatted):
|
|
556
|
+
```dax
|
|
557
|
+
CALCULATE([YTD Gross Margin],SAMEPERIODLASTYEAR(DATESYTD('Date'[Date])))
|
|
558
|
+
```
|
|
559
|
+
|
|
560
|
+
### After (hierarchical formatting):
|
|
561
|
+
```dax
|
|
562
|
+
CALCULATE(
|
|
563
|
+
[YTD Gross Margin],
|
|
564
|
+
SAMEPERIODLASTYEAR(
|
|
565
|
+
DATESYTD(
|
|
566
|
+
'Date'[Date]
|
|
567
|
+
)
|
|
568
|
+
)
|
|
569
|
+
)
|
|
570
|
+
```
|
|
571
|
+
|
|
572
|
+
**Formatting features:**
|
|
573
|
+
- Each nesting level increases indentation (+4 spaces)
|
|
574
|
+
- Closing parentheses aligned with the start of their function
|
|
575
|
+
- Arguments on separate lines for clarity
|
|
576
|
+
- Preservation of all original arguments
|
|
577
|
+
- Complexity indicators (Simple / Medium / Complex)
|
|
578
|
+
|
|
579
|
+
---
|
|
580
|
+
|
|
581
|
+
## Troubleshooting
|
|
582
|
+
|
|
583
|
+
### Common Errors
|
|
584
|
+
|
|
585
|
+
**Error decoding `DataModelSchema`**:
|
|
586
|
+
- The script tries `utf-8`, `utf-16` and `latin-1`, plus removes comments/trailing commas.
|
|
587
|
+
- If it fails, `schema_snippet.txt` is saved in the output folder for diagnosis.
|
|
588
|
+
|
|
589
|
+
**Paths with spaces/special characters**:
|
|
590
|
+
- Use quotes in the CLI: `pbi-docs -i "data/My File.pbit"`
|
|
591
|
+
- Prefer paths within `data/`.
|
|
592
|
+
|
|
593
|
+
**No documentation generated**:
|
|
594
|
+
- Verify that the file contains `DataModelSchema`.
|
|
595
|
+
- If you're using `.pbix`, export to `.pbit` from Power BI Desktop (File > Export > Power BI Template).
|
|
596
|
+
|
|
597
|
+
**PowerShell blocks venv activation**:
|
|
598
|
+
- Adjust the `ExecutionPolicy` as indicated in installation.
|
|
599
|
+
|
|
600
|
+
### Verify Installation
|
|
601
|
+
|
|
602
|
+
Run a quick test with your .pbit file:
|
|
603
|
+
|
|
604
|
+
```powershell
|
|
605
|
+
pbi-docs --input "data/pbit/my-model.pbit"
|
|
606
|
+
```
|
|
607
|
+
|
|
608
|
+
**Expected output:**
|
|
609
|
+
```
|
|
610
|
+
Processing file: data/pbit/my-model.pbit
|
|
611
|
+
Schema extracted successfully: 11 tables
|
|
612
|
+
Metadata processed: 11 tables, 44 measures
|
|
613
|
+
Metadata saved to: output/my-model.pbit/metadata.json
|
|
614
|
+
Documentation saved to: output/my-model.pbit/model_documentation.md
|
|
615
|
+
Agent context saved to: output/my-model.pbit/agent_context.json
|
|
616
|
+
JSONL context saved to: output/my-model.pbit/model_context.jsonl
|
|
617
|
+
Processing completed successfully for: my-model.pbit
|
|
618
|
+
```
|
|
619
|
+
|
|
620
|
+
Then verify that the files were generated correctly:
|
|
621
|
+
|
|
622
|
+
```powershell
|
|
623
|
+
# List generated files
|
|
624
|
+
Get-ChildItem "output/my-model.pbit/"
|
|
625
|
+
|
|
626
|
+
# View a summary of the documentation
|
|
627
|
+
Get-Content "output/my-model.pbit/model_documentation.md" | Select-Object -First 10
|
|
628
|
+
```
|
|
629
|
+
|
|
630
|
+
### Common Installation Issues
|
|
631
|
+
|
|
632
|
+
**Error: "Python not recognized"**
|
|
633
|
+
```powershell
|
|
634
|
+
# Add Python to PATH or use full path
|
|
635
|
+
C:\Python312\python.exe -m pip install -e .
|
|
636
|
+
```
|
|
637
|
+
|
|
638
|
+
**Error: "No module named..."**
|
|
639
|
+
This project requires no dependencies. If you see this error, verify your Python version:
|
|
640
|
+
```powershell
|
|
641
|
+
python --version # Must be 3.10+
|
|
642
|
+
```
|
|
643
|
+
|
|
644
|
+
**Error: "pbi-docs not recognized"**
|
|
645
|
+
```powershell
|
|
646
|
+
# Reinstall the package
|
|
647
|
+
pip install -e .
|
|
648
|
+
# Or use python -m
|
|
649
|
+
python -m pbi_extractor.cli --input file.pbit
|
|
650
|
+
```
|
|
651
|
+
|
|
652
|
+
---
|
|
653
|
+
|
|
654
|
+
## Comparison with Alternatives
|
|
655
|
+
|
|
656
|
+
| Feature | pbi-docs | Power BI Helper | Dataedo | Manual (DAX Studio) |
|
|
657
|
+
|---------|------------|-----------------|---------|---------------------|
|
|
658
|
+
| **Free & Open Source** | Yes | No (Paid) | No (Paid) | Yes |
|
|
659
|
+
| **Batch Processing** | Yes - Multiple files | No - One at a time | Yes | No |
|
|
660
|
+
| **Multi-language Support** | Yes - English & Spanish | No | No | No |
|
|
661
|
+
| **AI-Ready Outputs** | Yes - JSON + JSONL | No | Limited | No |
|
|
662
|
+
| **Hierarchical DAX** | Yes | Basic | Basic | No - Raw only |
|
|
663
|
+
| **No Installation** | Yes - pip install | Limited - Desktop app | Limited - Platform | Yes |
|
|
664
|
+
| **Categorization** | Yes - Auto (revenue, cost...) | No - Manual | Yes | No |
|
|
665
|
+
| **Version Diff** | Yes - Built-in | No | Yes | No - Manual |
|
|
666
|
+
| **MCP Server / On-demand Query** | Yes - Built-in, zero deps | No | No | No |
|
|
667
|
+
|
|
668
|
+
pbi-docs is a read/context-compilation layer, a different category from Microsoft's own Power BI
|
|
669
|
+
MCP servers (Modeling MCP for writes, Remote MCP for DAX execution) — complementary rather than
|
|
670
|
+
competing.
|
|
671
|
+
|
|
672
|
+
---
|
|
673
|
+
|
|
674
|
+
## Generated Documentation Example
|
|
675
|
+
|
|
676
|
+
<details>
|
|
677
|
+
<summary>View complete example of model_documentation.md (my-model)</summary>
|
|
678
|
+
|
|
679
|
+
```markdown
|
|
680
|
+
# my-model - Power BI Data Model
|
|
681
|
+
|
|
682
|
+
**Generated:** 2025-12-22 14:23:29
|
|
683
|
+
|
|
684
|
+
## Tables and Measures
|
|
685
|
+
|
|
686
|
+
### Fact *(Hidden Table - Measures Only)*
|
|
687
|
+
|
|
688
|
+
**Measures:**
|
|
689
|
+
|
|
690
|
+
##### Revenue Measures
|
|
691
|
+
|
|
692
|
+
**Total Revenue** *(simple)*
|
|
693
|
+
|
|
694
|
+
```dax
|
|
695
|
+
SUM([Revenue])
|
|
696
|
+
```
|
|
697
|
+
|
|
698
|
+
*Format:* `$#,0;($#,0);$#,0`
|
|
699
|
+
|
|
700
|
+
---
|
|
701
|
+
|
|
702
|
+
**YTD Revenue** *(simple)*
|
|
703
|
+
|
|
704
|
+
```dax
|
|
705
|
+
TOTALYTD(
|
|
706
|
+
SUM([Revenue]),
|
|
707
|
+
'Date'[Date])
|
|
708
|
+
```
|
|
709
|
+
|
|
710
|
+
*Format:* `$#,0;($#,0);$#,0`
|
|
711
|
+
|
|
712
|
+
---
|
|
713
|
+
|
|
714
|
+
**Revenue SPLY** *(medium)*
|
|
715
|
+
|
|
716
|
+
```dax
|
|
717
|
+
CALCULATE(
|
|
718
|
+
[Total Revenue],
|
|
719
|
+
SAMEPERIODLASTYEAR(
|
|
720
|
+
'Date'[Date]))
|
|
721
|
+
```
|
|
722
|
+
|
|
723
|
+
*Format:* `$#,0;($#,0);$#,0`
|
|
724
|
+
|
|
725
|
+
##### Margin Measures
|
|
726
|
+
|
|
727
|
+
**Gross Margin** *(simple)*
|
|
728
|
+
|
|
729
|
+
```dax
|
|
730
|
+
[Total Revenue]-[Total COGS]
|
|
731
|
+
```
|
|
732
|
+
|
|
733
|
+
*Format:* `$#,0;($#,0);$#,0`
|
|
734
|
+
|
|
735
|
+
##### Percentage Measures
|
|
736
|
+
|
|
737
|
+
**GM%** *(simple)*
|
|
738
|
+
|
|
739
|
+
```dax
|
|
740
|
+
DIVIDE(
|
|
741
|
+
[Gross Margin],
|
|
742
|
+
[Total Revenue])
|
|
743
|
+
```
|
|
744
|
+
|
|
745
|
+
*Format:* `0.0 %;-0.0 %;0.0 %`
|
|
746
|
+
|
|
747
|
+
### Date
|
|
748
|
+
|
|
749
|
+
**Columns:**
|
|
750
|
+
|
|
751
|
+
| Column | Type | Category |
|
|
752
|
+
|--------|------|----------|
|
|
753
|
+
| `Date` | dateTime | temporal |
|
|
754
|
+
| `Year` | int64 | numeric |
|
|
755
|
+
| `Month` | string | categorical |
|
|
756
|
+
|
|
757
|
+
### Customer
|
|
758
|
+
|
|
759
|
+
**Columns:**
|
|
760
|
+
|
|
761
|
+
| Column | Type | Category |
|
|
762
|
+
|--------|------|----------|
|
|
763
|
+
| `Name` | string | descriptive |
|
|
764
|
+
| `City` | string | categorical |
|
|
765
|
+
| `State` | string | categorical |
|
|
766
|
+
| `Country/Region` | string | categorical |
|
|
767
|
+
|
|
768
|
+
## Relationships
|
|
769
|
+
|
|
770
|
+
| From | To | Type | Direction |
|
|
771
|
+
|------|----|----- |-----------|
|
|
772
|
+
| Fact.BU Key | BU.BU Key | many:one | OneDirection |
|
|
773
|
+
| Fact.YearPeriod | Date.YearPeriod | many:one | OneDirection |
|
|
774
|
+
| Fact.Customer Key | Customer.Customer | many:one | OneDirection |
|
|
775
|
+
```
|
|
776
|
+
|
|
777
|
+
</details>
|
|
778
|
+
|
|
779
|
+
---
|
|
780
|
+
|
|
781
|
+
## Contributing
|
|
782
|
+
|
|
783
|
+
Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for the workflow, or use
|
|
784
|
+
[GitHub Issues](https://github.com/Osc2405/pbi-docs/issues) for bug reports and feature requests.
|
|
785
|
+
|
|
786
|
+
---
|
|
787
|
+
|
|
788
|
+
## Roadmap
|
|
789
|
+
|
|
790
|
+
The read/context layer (PBIP/TMDL support, indexed output, query resolver, MCP server) is done.
|
|
791
|
+
Writing TMDL models and PBIR/report parsing are deliberately deferred — bigger undertakings, and
|
|
792
|
+
(for writing) Microsoft's own Modeling MCP already covers that space.
|
|
793
|
+
|
|
794
|
+
---
|
|
795
|
+
|
|
796
|
+
## Sample Files
|
|
797
|
+
|
|
798
|
+
The example files referenced in this documentation are official sample files provided by Microsoft. You can find these and other Power BI sample files in the [Microsoft Power BI Desktop Samples repository](https://github.com/microsoft/powerbi-desktop-samples).
|
|
799
|
+
|
|
800
|
+
These sample files are excellent for:
|
|
801
|
+
- Testing pbi-docs functionality
|
|
802
|
+
- Learning Power BI data modeling
|
|
803
|
+
- Exploring different DAX patterns and measure types
|
|
804
|
+
- Understanding relationship structures
|
|
805
|
+
|
|
806
|
+
To use these samples:
|
|
807
|
+
1. Clone or download the repository: `git clone https://github.com/microsoft/powerbi-desktop-samples.git`
|
|
808
|
+
2. Open the `.pbix` files in Power BI Desktop
|
|
809
|
+
3. Export them as `.pbit` files (File > Export > Power BI Template)
|
|
810
|
+
4. Use them with pbi-docs to generate documentation
|
|
811
|
+
|
|
812
|
+
---
|
|
813
|
+
|
|
814
|
+
## Author
|
|
815
|
+
|
|
816
|
+
**Oscar Rosero** - Data Engineer | BI Developer
|
|
817
|
+
|
|
818
|
+
- [LinkedIn](https://www.linkedin.com/in/oscrosero24/)
|
|
819
|
+
- [GitHub](https://github.com/Osc2405)
|
|
820
|
+
- orosero2405@gmail.com
|
|
821
|
+
|
|
822
|
+
Built with ❤️ for the Power BI community.
|
|
823
|
+
|
|
824
|
+
---
|
|
825
|
+
|
|
826
|
+
## Star History
|
|
827
|
+
|
|
828
|
+
If this tool saves you time, please consider giving it a ⭐ on GitHub!
|
|
829
|
+
|
|
830
|
+
|