dbdocs 1.2.0__tar.gz → 1.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {dbdocs-1.2.0 → dbdocs-1.3.0}/.gitignore +2 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/PKG-INFO +12 -14
- dbdocs-1.3.0/README.md +67 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/dimensions.py +22 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/__init__.py +14 -1
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/base.py +17 -2
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/dimensions/documentation.py +30 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/dimensions/modeling.py +96 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/dimensions/structure.py +18 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/dimensions/testing.py +19 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/registry.py +14 -1
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/css/style.css +19 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/js/service.js +169 -1
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/js/ui.js +137 -17
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/index.html +3 -2
- dbdocs-1.2.0/README.md +0 -69
- {dbdocs-1.2.0 → dbdocs-1.3.0}/LICENSE +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/__init__.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/__main__.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/cli/__init__.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/cli/main.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/core/__init__.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/core/artifacts.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/core/config.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/core/exceptions.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/core/log.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/__init__.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/_sqlglot_lineage.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/column_lineage.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/erd.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/erd_json.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/graph.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/__init__.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/extractor.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/dimensions/__init__.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/dimensions/governance.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/dimensions/performance.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/nodes.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/main.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/__init__.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/builder.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/css/icons.css +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/favicon.svg +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/graph/index.css +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/graph/index.js +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/js/app.js +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/js/data.js +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/vendor/marked.min.js +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/vendor/minisearch.min.js +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/deploy.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/inject.py +0 -0
- {dbdocs-1.2.0 → dbdocs-1.3.0}/pyproject.toml +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: dbdocs
|
|
3
|
-
Version: 1.
|
|
3
|
+
Version: 1.3.0
|
|
4
4
|
Summary: Alternative dbt docs site: Catalog + ERD + column-level lineage
|
|
5
5
|
Project-URL: Homepage, https://github.com/datnguye/dbt-docs
|
|
6
6
|
Project-URL: Repository, https://github.com/datnguye/dbt-docs
|
|
@@ -45,19 +45,17 @@ Turn your dbt artifacts into a self-contained docs site: a browsable catalog, an
|
|
|
45
45
|
|
|
46
46
|
## Why dbdocs?
|
|
47
47
|
|
|
48
|
-
dbt's built-in docs
|
|
49
|
-
|
|
50
|
-
**
|
|
51
|
-
|
|
52
|
-
- **
|
|
53
|
-
- **
|
|
54
|
-
- **
|
|
55
|
-
- **
|
|
56
|
-
- **
|
|
57
|
-
- **
|
|
58
|
-
- **
|
|
59
|
-
- **Versioned deploys, no plugins.** `dbdocs deploy --version v1.2 --alias latest` generates into a plain directory tree, writes a `versions.json` index, and the SPA renders a version dropdown. No mike, no external tooling, no surprise dependencies.
|
|
60
|
-
- **Catalog navigation + client-side search.** Models, seeds, and snapshots grouped by database and schema; full-text search without a backend.
|
|
48
|
+
dbt's built-in docs stop short of telling you *which upstream column fed this downstream column*, *which tables relate to each other*, or *what changed between builds*. dbdocs fills those gaps — no documentation framework or separate ERD tool to install.
|
|
49
|
+
|
|
50
|
+
- **ERD + column-level lineage** — table relationships ([dbterd](https://github.com/datnguye/dbterd)) and column lineage from compiled SQL ([sqlglot](https://github.com/tobymao/sqlglot)).
|
|
51
|
+
- **Column impact analysis** — downstream dependents for any column.
|
|
52
|
+
- **Deep-link URLs** for every node, column, and DAG view.
|
|
53
|
+
- **Any sqlglot dialect**, auto-detected from your manifest.
|
|
54
|
+
- **Scales to 1 000s of models** without freezing the browser.
|
|
55
|
+
- **Fail-soft** — an unparseable model is skipped, not fatal.
|
|
56
|
+
- **Project Health Check** across the six [dbt-project-evaluator](https://dbt-labs.github.io/dbt-project-evaluator/) dimensions.
|
|
57
|
+
- **Versioned deploys** with a built-in version switcher, no plugins.
|
|
58
|
+
- **Full-text search** across names, columns, descriptions, tags, and SQL at the client-side, no backend.
|
|
61
59
|
- **Dark / light theme.**
|
|
62
60
|
|
|
63
61
|
## Install
|
dbdocs-1.3.0/README.md
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
<p align="center">
|
|
2
|
+
<img src="docs/assets/logo.svg" alt="dbdocs logo" width="220" height="88">
|
|
3
|
+
</p>
|
|
4
|
+
|
|
5
|
+
<p align="center"><b>An alternative dbt docs site — catalog + ERD + column-level lineage + versioned deploys, all in one CLI.</b></p>
|
|
6
|
+
|
|
7
|
+
<p align="center">
|
|
8
|
+
<a href="https://dbdocs.datnguye.me/latest/demo/latest/"><img src="https://img.shields.io/badge/live-demo-FF694A?style=flat&logo=rocket&logoColor=white" alt="live demo"></a>
|
|
9
|
+
<a href="https://dbdocs.datnguye.me/"><img src="https://img.shields.io/badge/docs-visit%20site-blue?style=flat&logo=gitbook&logoColor=white" alt="docs"></a>
|
|
10
|
+
<a href="https://pypi.org/project/dbdocs/"><img src="https://badge.fury.io/py/dbdocs.svg" alt="PyPI version"></a>
|
|
11
|
+
<img src="https://img.shields.io/badge/CLI-Python-FFCE3E?labelColor=14354C&logo=python&logoColor=white" alt="python-cli">
|
|
12
|
+
<a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
|
|
13
|
+
<a href="https://www.python.org"><img src="https://img.shields.io/badge/Python-3.10|3.11|3.12|3.13-3776AB.svg?style=flat&logo=python&logoColor=white" alt="python"></a>
|
|
14
|
+
</p>
|
|
15
|
+
|
|
16
|
+
Turn your dbt artifacts into a self-contained docs site: a browsable catalog, an entity-relationship diagram, an interactive lineage DAG, and **column-level lineage** traced from your compiled SQL — all in one `dbdocs generate`. Serve it with `dbdocs serve`, or deploy versioned builds anywhere a static host will take them.
|
|
17
|
+
|
|
18
|
+
| Catalog | Model page | Lineage DAG | Health Check |
|
|
19
|
+
|---|---|---|---|
|
|
20
|
+
|  |  |  |  |
|
|
21
|
+
|
|
22
|
+
## Why dbdocs?
|
|
23
|
+
|
|
24
|
+
dbt's built-in docs stop short of telling you *which upstream column fed this downstream column*, *which tables relate to each other*, or *what changed between builds*. dbdocs fills those gaps — no documentation framework or separate ERD tool to install.
|
|
25
|
+
|
|
26
|
+
- **ERD + column-level lineage** — table relationships ([dbterd](https://github.com/datnguye/dbterd)) and column lineage from compiled SQL ([sqlglot](https://github.com/tobymao/sqlglot)).
|
|
27
|
+
- **Column impact analysis** — downstream dependents for any column.
|
|
28
|
+
- **Deep-link URLs** for every node, column, and DAG view.
|
|
29
|
+
- **Any sqlglot dialect**, auto-detected from your manifest.
|
|
30
|
+
- **Scales to 1 000s of models** without freezing the browser.
|
|
31
|
+
- **Fail-soft** — an unparseable model is skipped, not fatal.
|
|
32
|
+
- **Project Health Check** across the six [dbt-project-evaluator](https://dbt-labs.github.io/dbt-project-evaluator/) dimensions.
|
|
33
|
+
- **Versioned deploys** with a built-in version switcher, no plugins.
|
|
34
|
+
- **Full-text search** across names, columns, descriptions, tags, and SQL at the client-side, no backend.
|
|
35
|
+
- **Dark / light theme.**
|
|
36
|
+
|
|
37
|
+
## Install
|
|
38
|
+
|
|
39
|
+
```bash
|
|
40
|
+
pip install dbdocs --upgrade
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Requires Python 3.10+.
|
|
44
|
+
|
|
45
|
+
## Quickstart
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
dbt docs generate # writes target/manifest.json + target/catalog.json
|
|
49
|
+
dbdocs generate # builds ./site/ with index.html + dbdocs-data.json.gz
|
|
50
|
+
dbdocs serve # static http server on http://127.0.0.1:8000
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
The site must be served over HTTP (not opened as a local file) because it fetches the data payload at load time. `dbdocs serve` handles that locally; any static host works for deployment.
|
|
54
|
+
|
|
55
|
+
Full walkthrough, configuration, and architecture live in the **[documentation](https://dbdocs.datnguye.me/)**.
|
|
56
|
+
|
|
57
|
+
## Contributing
|
|
58
|
+
|
|
59
|
+
Contributions are welcome — bugs, features, docs, typos. See the **[Contributing Guide](https://dbdocs.datnguye.me/latest/nav/development/contributing-guide.html)**.
|
|
60
|
+
|
|
61
|
+
If dbdocs saves you some clicks, consider [buying me a coffee](https://www.buymeacoffee.com/datnguye).
|
|
62
|
+
|
|
63
|
+
<a href="https://www.buymeacoffee.com/datnguye"><img src="https://img.shields.io/badge/buy%20me%20a%20coffee-donate-yellow.svg?logo=buy-me-a-coffee&logoColor=white&labelColor=ff813f&style=for-the-badge" alt="buy me a coffee"></a>
|
|
64
|
+
|
|
65
|
+
## License
|
|
66
|
+
|
|
67
|
+
[MIT](./LICENSE) © Dat Nguyen
|
|
@@ -50,6 +50,13 @@ class ManifestGraph:
|
|
|
50
50
|
self.models = [n for uid, n in self._nodes.items() if uid.startswith("model.")]
|
|
51
51
|
self.sources = list(self._sources.values())
|
|
52
52
|
self.exposures = list(self._exposures.values())
|
|
53
|
+
# Singular tests are custom-SQL test nodes (no test_metadata); generic
|
|
54
|
+
# tests (unique/not_null/…) carry test_metadata and are excluded.
|
|
55
|
+
self.singular_tests = [
|
|
56
|
+
n
|
|
57
|
+
for uid, n in self._nodes.items()
|
|
58
|
+
if uid.startswith("test.") and getattr(n, "test_metadata", None) is None
|
|
59
|
+
]
|
|
53
60
|
|
|
54
61
|
# Rule thresholds: per-run overrides layered over the DPE defaults.
|
|
55
62
|
self._thresholds = {**DEFAULT_THRESHOLDS, **(thresholds or {})}
|
|
@@ -147,6 +154,21 @@ class ManifestGraph:
|
|
|
147
154
|
access = access or getattr(model, "access", None)
|
|
148
155
|
return str(access or "protected").lower()
|
|
149
156
|
|
|
157
|
+
@staticmethod
|
|
158
|
+
def has_source_freshness(source: Any) -> bool:
|
|
159
|
+
"""Whether a source has a freshness check: a ``loaded_at_field`` plus a
|
|
160
|
+
``warn_after``/``error_after`` threshold count."""
|
|
161
|
+
if not str(getattr(source, "loaded_at_field", "") or "").strip():
|
|
162
|
+
return False
|
|
163
|
+
freshness = getattr(source, "freshness", None)
|
|
164
|
+
if freshness is None:
|
|
165
|
+
return False
|
|
166
|
+
for bound in ("warn_after", "error_after"):
|
|
167
|
+
period = getattr(freshness, bound, None)
|
|
168
|
+
if period is not None and getattr(period, "count", None) is not None:
|
|
169
|
+
return True
|
|
170
|
+
return False
|
|
171
|
+
|
|
150
172
|
@staticmethod
|
|
151
173
|
def contract_enforced(model: Any) -> bool:
|
|
152
174
|
"""Whether the model has an enforced contract (``contract.enforced``)."""
|
|
@@ -24,6 +24,7 @@ of :mod:`base` and :mod:`registry`; no implementation lives here.
|
|
|
24
24
|
|
|
25
25
|
from dbdocs.extract.health.rules.base import DEFAULT_THRESHOLDS, NON_PHYSICAL, docs_url, finding
|
|
26
26
|
from dbdocs.extract.health.rules.dimensions.documentation import (
|
|
27
|
+
documentation_coverage,
|
|
27
28
|
undocumented_models,
|
|
28
29
|
undocumented_source_tables,
|
|
29
30
|
undocumented_sources,
|
|
@@ -35,7 +36,9 @@ from dbdocs.extract.health.rules.dimensions.governance import (
|
|
|
35
36
|
)
|
|
36
37
|
from dbdocs.extract.health.rules.dimensions.modeling import (
|
|
37
38
|
direct_join_to_source,
|
|
39
|
+
downstream_models_dependent_on_source,
|
|
38
40
|
duplicate_sources,
|
|
41
|
+
hard_coded_references,
|
|
39
42
|
model_fanout,
|
|
40
43
|
multiple_sources_joined,
|
|
41
44
|
rejoining_of_upstream_concepts,
|
|
@@ -54,8 +57,13 @@ from dbdocs.extract.health.rules.dimensions.structure import (
|
|
|
54
57
|
model_directories,
|
|
55
58
|
model_naming_conventions,
|
|
56
59
|
source_directories,
|
|
60
|
+
test_directories,
|
|
61
|
+
)
|
|
62
|
+
from dbdocs.extract.health.rules.dimensions.testing import (
|
|
63
|
+
missing_primary_key_tests,
|
|
64
|
+
missing_source_freshness,
|
|
65
|
+
test_coverage,
|
|
57
66
|
)
|
|
58
|
-
from dbdocs.extract.health.rules.dimensions.testing import missing_primary_key_tests, test_coverage
|
|
59
67
|
from dbdocs.extract.health.rules.registry import (
|
|
60
68
|
DIMENSION_RULES,
|
|
61
69
|
ENTRY_POINT_GROUP,
|
|
@@ -81,8 +89,11 @@ __all__ = [
|
|
|
81
89
|
# rule functions (re-exported for direct access / testing)
|
|
82
90
|
"test_coverage",
|
|
83
91
|
"missing_primary_key_tests",
|
|
92
|
+
"missing_source_freshness",
|
|
84
93
|
"direct_join_to_source",
|
|
94
|
+
"downstream_models_dependent_on_source",
|
|
85
95
|
"duplicate_sources",
|
|
96
|
+
"hard_coded_references",
|
|
86
97
|
"model_fanout",
|
|
87
98
|
"multiple_sources_joined",
|
|
88
99
|
"rejoining_of_upstream_concepts",
|
|
@@ -92,12 +103,14 @@ __all__ = [
|
|
|
92
103
|
"staging_dependent_on_marts_or_intermediate",
|
|
93
104
|
"unused_sources",
|
|
94
105
|
"too_many_joins",
|
|
106
|
+
"documentation_coverage",
|
|
95
107
|
"undocumented_models",
|
|
96
108
|
"undocumented_sources",
|
|
97
109
|
"undocumented_source_tables",
|
|
98
110
|
"model_naming_conventions",
|
|
99
111
|
"model_directories",
|
|
100
112
|
"source_directories",
|
|
113
|
+
"test_directories",
|
|
101
114
|
"chained_view_dependencies",
|
|
102
115
|
"exposure_parents_materializations",
|
|
103
116
|
"public_models_without_contracts",
|
|
@@ -14,6 +14,16 @@ DEFAULT_THRESHOLDS = {
|
|
|
14
14
|
"model_fanout": 3,
|
|
15
15
|
"too_many_joins": 7,
|
|
16
16
|
"chained_view_dependencies": 4,
|
|
17
|
+
"documentation_coverage": 100,
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
# DPE anchors that don't equal the rule name kebab-cased. The auto-derived anchor
|
|
21
|
+
# (``rule.replace("_", "-")``) is the DPE heading for most rules; these few have a
|
|
22
|
+
# different published heading, so map the rule name to its real anchor explicitly.
|
|
23
|
+
_RULE_ANCHORS = {
|
|
24
|
+
"too_many_joins": "models-with-too-many-joins",
|
|
25
|
+
"staging_dependent_on_marts_or_intermediate": "staging-models-dependent-on-downstream-models",
|
|
26
|
+
"staging_dependent_on_staging": "staging-models-dependent-on-other-staging-models",
|
|
17
27
|
}
|
|
18
28
|
|
|
19
29
|
# Materializations that are *not* a physical table (for chained-view detection).
|
|
@@ -28,8 +38,13 @@ LAYER_PREFIXES = {
|
|
|
28
38
|
|
|
29
39
|
|
|
30
40
|
def docs_url(category: str, rule: str) -> str:
|
|
31
|
-
"""The DPE docs URL for a rule under *category
|
|
32
|
-
|
|
41
|
+
"""The DPE docs URL for a rule under *category*.
|
|
42
|
+
|
|
43
|
+
The anchor is the rule name kebab-cased, except for the few rules whose DPE
|
|
44
|
+
heading differs (see ``_RULE_ANCHORS``).
|
|
45
|
+
"""
|
|
46
|
+
anchor = _RULE_ANCHORS.get(rule, rule.replace("_", "-"))
|
|
47
|
+
return f"{_RULES_BASE}/{category}/#{anchor}"
|
|
33
48
|
|
|
34
49
|
|
|
35
50
|
def finding(rule: str, category: str, node: str, node_type: str, message: str) -> dict:
|
|
@@ -54,6 +54,36 @@ def undocumented_sources(graph: "ManifestGraph") -> "list[dict]":
|
|
|
54
54
|
return out
|
|
55
55
|
|
|
56
56
|
|
|
57
|
+
def documentation_coverage(graph: "ManifestGraph") -> "list[dict]":
|
|
58
|
+
"""One project-level finding when the share of documented models is below target.
|
|
59
|
+
|
|
60
|
+
Mirrors DPE's ``fct_documentation_coverage`` — an aggregate percentage rather
|
|
61
|
+
than a per-model flag. The ``documentation_coverage`` threshold (default 100)
|
|
62
|
+
is the minimum acceptable percentage; coverage at or above it is clean.
|
|
63
|
+
|
|
64
|
+
Being an aggregate, its finding's ``node`` is the pseudo-id ``"project"`` (not a
|
|
65
|
+
real ``unique_id``); the SPA renders an unresolvable node as plain text rather
|
|
66
|
+
than a node-page link, so this degrades gracefully.
|
|
67
|
+
"""
|
|
68
|
+
models = graph.models
|
|
69
|
+
if not models:
|
|
70
|
+
return []
|
|
71
|
+
documented = sum(1 for m in models if str(getattr(m, "description", "") or "").strip())
|
|
72
|
+
coverage = round(documented * 100 / len(models), 1)
|
|
73
|
+
if coverage >= graph.threshold("documentation_coverage"):
|
|
74
|
+
return []
|
|
75
|
+
return [
|
|
76
|
+
finding(
|
|
77
|
+
"documentation_coverage",
|
|
78
|
+
"documentation",
|
|
79
|
+
"project",
|
|
80
|
+
"project",
|
|
81
|
+
f"Only {coverage}% of models are documented "
|
|
82
|
+
f"({documented}/{len(models)}) — below the configured target.",
|
|
83
|
+
)
|
|
84
|
+
]
|
|
85
|
+
|
|
86
|
+
|
|
57
87
|
def undocumented_source_tables(graph: "ManifestGraph") -> "list[dict]":
|
|
58
88
|
"""Source tables (the individual relations) with no description."""
|
|
59
89
|
out = []
|
|
@@ -4,14 +4,29 @@ Each rule is ``(graph) -> list[dict]``; see ``dbdocs.extract.health.rules``
|
|
|
4
4
|
for the shared finding shape and the registry that assembles these.
|
|
5
5
|
"""
|
|
6
6
|
|
|
7
|
+
import re
|
|
7
8
|
from typing import Any
|
|
8
9
|
|
|
10
|
+
import sqlglot
|
|
11
|
+
from sqlglot import exp
|
|
12
|
+
from sqlglot.errors import SqlglotError
|
|
13
|
+
from sqlglot.optimizer.scope import build_scope
|
|
14
|
+
|
|
9
15
|
from dbdocs.extract.health.rules.base import finding
|
|
10
16
|
|
|
11
17
|
# ``ManifestGraph`` annotations are strings (forward ref) to avoid a circular
|
|
12
18
|
# import (rules ← graph ← analyzer); resolved lazily by the type checker only.
|
|
13
19
|
ManifestGraph = Any
|
|
14
20
|
|
|
21
|
+
# A ``{{ ref(...) }}`` / ``{{ source(...) }}`` call is rewritten to a uniquely
|
|
22
|
+
# numbered sentinel table before parsing (``__dbt_ref_0__``, ``__dbt_ref_1__``, …)
|
|
23
|
+
# — unique so sqlglot's scope analysis doesn't choke on a repeated alias — so any
|
|
24
|
+
# *other* real table left in the SQL is a hard-coded relation. Any remaining
|
|
25
|
+
# ``{{ ... }}`` macro is blanked so it doesn't masquerade as a table.
|
|
26
|
+
_REF_SENTINEL_PREFIX = "__dbt_ref_"
|
|
27
|
+
_REF_OR_SOURCE = re.compile(r"{{[-\s]*(?:ref|source)\s*\(.*?\)[-\s]*}}", re.IGNORECASE | re.DOTALL)
|
|
28
|
+
_OTHER_JINJA = re.compile(r"{[{%].*?[%}]}", re.DOTALL)
|
|
29
|
+
|
|
15
30
|
|
|
16
31
|
def direct_join_to_source(graph: "ManifestGraph") -> "list[dict]":
|
|
17
32
|
"""Models that ref a model AND a source in the same query (missing staging)."""
|
|
@@ -121,6 +136,87 @@ def rejoining_of_upstream_concepts(graph: "ManifestGraph") -> "list[dict]":
|
|
|
121
136
|
return out
|
|
122
137
|
|
|
123
138
|
|
|
139
|
+
def downstream_models_dependent_on_source(graph: "ManifestGraph") -> "list[dict]":
|
|
140
|
+
"""Non-staging models that ref a source directly (only staging should touch sources)."""
|
|
141
|
+
out = []
|
|
142
|
+
for model in graph.models:
|
|
143
|
+
if graph.layer(model) == "staging":
|
|
144
|
+
continue
|
|
145
|
+
if any(p.startswith("source.") for p in graph.parents(model.unique_id)):
|
|
146
|
+
out.append(
|
|
147
|
+
finding(
|
|
148
|
+
"downstream_models_dependent_on_source",
|
|
149
|
+
"modeling",
|
|
150
|
+
model.unique_id,
|
|
151
|
+
"model",
|
|
152
|
+
"A non-staging model selects from a source — route it through a staging model.",
|
|
153
|
+
)
|
|
154
|
+
)
|
|
155
|
+
return out
|
|
156
|
+
|
|
157
|
+
|
|
158
|
+
def _number_refs(raw_code: str) -> str:
|
|
159
|
+
"""Replace each ref()/source() jinja call with a uniquely numbered sentinel."""
|
|
160
|
+
out = []
|
|
161
|
+
last = 0
|
|
162
|
+
for index, match in enumerate(_REF_OR_SOURCE.finditer(raw_code)):
|
|
163
|
+
out.append(raw_code[last : match.start()])
|
|
164
|
+
out.append(f"{_REF_SENTINEL_PREFIX}{index}__")
|
|
165
|
+
last = match.end()
|
|
166
|
+
out.append(raw_code[last:])
|
|
167
|
+
return "".join(out)
|
|
168
|
+
|
|
169
|
+
|
|
170
|
+
def _hard_coded_relations(raw_code: str) -> "list[str]":
|
|
171
|
+
"""Real table names a model's raw SQL selects from outside ref()/source().
|
|
172
|
+
|
|
173
|
+
dbt references are jinja (``{{ ref(...) }}`` / ``{{ source(...) }}``), so they
|
|
174
|
+
are rewritten to uniquely numbered sentinel tables before parsing (so sqlglot's
|
|
175
|
+
scope analysis doesn't choke on a repeated alias); any other real table its
|
|
176
|
+
scope analysis finds is a hard-coded relation. CTEs are excluded by walking
|
|
177
|
+
``scope.selected_sources`` rather than every ``exp.Table``. Unparseable SQL
|
|
178
|
+
yields nothing (fail-soft — one model never sinks the pass).
|
|
179
|
+
"""
|
|
180
|
+
sql = _number_refs(raw_code)
|
|
181
|
+
sql = _OTHER_JINJA.sub(" ", sql)
|
|
182
|
+
if not sql.strip():
|
|
183
|
+
return []
|
|
184
|
+
try:
|
|
185
|
+
# parse_one returns a node or raises for non-empty input (empty is guarded
|
|
186
|
+
# above); build_scope returns None for a non-query statement (DDL/SET).
|
|
187
|
+
root = build_scope(sqlglot.parse_one(sql))
|
|
188
|
+
if root is None:
|
|
189
|
+
return []
|
|
190
|
+
relations = [
|
|
191
|
+
source.sql()
|
|
192
|
+
for scope in root.traverse()
|
|
193
|
+
for _alias, (_node, source) in scope.selected_sources.items()
|
|
194
|
+
if isinstance(source, exp.Table) and not source.name.startswith(_REF_SENTINEL_PREFIX)
|
|
195
|
+
]
|
|
196
|
+
except SqlglotError: # OptimizeError (e.g. duplicate alias) is a SqlglotError subclass.
|
|
197
|
+
return []
|
|
198
|
+
return relations
|
|
199
|
+
|
|
200
|
+
|
|
201
|
+
def hard_coded_references(graph: "ManifestGraph") -> "list[dict]":
|
|
202
|
+
"""Models whose SQL selects from a literal relation instead of ref()/source()."""
|
|
203
|
+
out = []
|
|
204
|
+
for model in graph.models:
|
|
205
|
+
relations = _hard_coded_relations(str(getattr(model, "raw_code", "") or ""))
|
|
206
|
+
if relations:
|
|
207
|
+
shown = ", ".join(sorted(set(relations)))
|
|
208
|
+
out.append(
|
|
209
|
+
finding(
|
|
210
|
+
"hard_coded_references",
|
|
211
|
+
"modeling",
|
|
212
|
+
model.unique_id,
|
|
213
|
+
"model",
|
|
214
|
+
f"Hard-coded relation(s) {shown} — replace with ref()/source().",
|
|
215
|
+
)
|
|
216
|
+
)
|
|
217
|
+
return out
|
|
218
|
+
|
|
219
|
+
|
|
124
220
|
def root_models(graph: "ManifestGraph") -> "list[dict]":
|
|
125
221
|
"""Models with zero parents (no ref/source) — untraceable lineage."""
|
|
126
222
|
out = []
|
|
@@ -57,6 +57,24 @@ def model_directories(graph: "ManifestGraph") -> "list[dict]":
|
|
|
57
57
|
return out
|
|
58
58
|
|
|
59
59
|
|
|
60
|
+
def test_directories(graph: "ManifestGraph") -> "list[dict]":
|
|
61
|
+
"""Singular (custom SQL) tests not stored under a ``tests/`` directory."""
|
|
62
|
+
out = []
|
|
63
|
+
for test in graph.singular_tests:
|
|
64
|
+
path = str(getattr(test, "original_file_path", "") or getattr(test, "path", "") or "")
|
|
65
|
+
if "tests" not in path.split("/"):
|
|
66
|
+
out.append(
|
|
67
|
+
finding(
|
|
68
|
+
"test_directories",
|
|
69
|
+
"structure",
|
|
70
|
+
test.unique_id,
|
|
71
|
+
"test",
|
|
72
|
+
"Singular test is not under a 'tests/' directory.",
|
|
73
|
+
)
|
|
74
|
+
)
|
|
75
|
+
return out
|
|
76
|
+
|
|
77
|
+
|
|
60
78
|
def source_directories(graph: "ManifestGraph") -> "list[dict]":
|
|
61
79
|
"""Source YAML not under a staging/ directory."""
|
|
62
80
|
out = []
|
|
@@ -49,3 +49,22 @@ def missing_primary_key_tests(graph: "ManifestGraph") -> "list[dict]":
|
|
|
49
49
|
)
|
|
50
50
|
)
|
|
51
51
|
return out
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
def missing_source_freshness(graph: "ManifestGraph") -> "list[dict]":
|
|
55
|
+
"""Sources with no freshness check (no ``loaded_at_field`` + warn/error threshold)."""
|
|
56
|
+
out = []
|
|
57
|
+
for src in graph.sources:
|
|
58
|
+
if not graph.has_source_freshness(src):
|
|
59
|
+
name = str(getattr(src, "source_name", "") or "") or src.unique_id
|
|
60
|
+
table = str(getattr(src, "name", "") or "")
|
|
61
|
+
out.append(
|
|
62
|
+
finding(
|
|
63
|
+
"missing_source_freshness",
|
|
64
|
+
"testing",
|
|
65
|
+
src.unique_id,
|
|
66
|
+
"source",
|
|
67
|
+
f"Source '{name}.{table}' has no freshness check configured.",
|
|
68
|
+
)
|
|
69
|
+
)
|
|
70
|
+
return out
|
|
@@ -12,6 +12,7 @@ from typing import Any
|
|
|
12
12
|
|
|
13
13
|
from dbdocs.core.log import logger
|
|
14
14
|
from dbdocs.extract.health.rules.dimensions.documentation import (
|
|
15
|
+
documentation_coverage,
|
|
15
16
|
undocumented_models,
|
|
16
17
|
undocumented_source_tables,
|
|
17
18
|
undocumented_sources,
|
|
@@ -23,7 +24,9 @@ from dbdocs.extract.health.rules.dimensions.governance import (
|
|
|
23
24
|
)
|
|
24
25
|
from dbdocs.extract.health.rules.dimensions.modeling import (
|
|
25
26
|
direct_join_to_source,
|
|
27
|
+
downstream_models_dependent_on_source,
|
|
26
28
|
duplicate_sources,
|
|
29
|
+
hard_coded_references,
|
|
27
30
|
model_fanout,
|
|
28
31
|
multiple_sources_joined,
|
|
29
32
|
rejoining_of_upstream_concepts,
|
|
@@ -42,8 +45,13 @@ from dbdocs.extract.health.rules.dimensions.structure import (
|
|
|
42
45
|
model_directories,
|
|
43
46
|
model_naming_conventions,
|
|
44
47
|
source_directories,
|
|
48
|
+
test_directories,
|
|
49
|
+
)
|
|
50
|
+
from dbdocs.extract.health.rules.dimensions.testing import (
|
|
51
|
+
missing_primary_key_tests,
|
|
52
|
+
missing_source_freshness,
|
|
53
|
+
test_coverage,
|
|
45
54
|
)
|
|
46
|
-
from dbdocs.extract.health.rules.dimensions.testing import missing_primary_key_tests, test_coverage
|
|
47
55
|
|
|
48
56
|
# The built-in rules grouped by dimension, in DPE's published order. These are the
|
|
49
57
|
# *built-ins*. ``DIMENSION_RULES`` (below) starts as a deep copy and is what
|
|
@@ -53,10 +61,13 @@ _BUILTIN_RULES: "dict[str, tuple]" = {
|
|
|
53
61
|
"testing": (
|
|
54
62
|
test_coverage,
|
|
55
63
|
missing_primary_key_tests,
|
|
64
|
+
missing_source_freshness,
|
|
56
65
|
),
|
|
57
66
|
"modeling": (
|
|
58
67
|
direct_join_to_source,
|
|
68
|
+
downstream_models_dependent_on_source,
|
|
59
69
|
duplicate_sources,
|
|
70
|
+
hard_coded_references,
|
|
60
71
|
model_fanout,
|
|
61
72
|
multiple_sources_joined,
|
|
62
73
|
rejoining_of_upstream_concepts,
|
|
@@ -68,6 +79,7 @@ _BUILTIN_RULES: "dict[str, tuple]" = {
|
|
|
68
79
|
too_many_joins,
|
|
69
80
|
),
|
|
70
81
|
"documentation": (
|
|
82
|
+
documentation_coverage,
|
|
71
83
|
undocumented_models,
|
|
72
84
|
undocumented_sources,
|
|
73
85
|
undocumented_source_tables,
|
|
@@ -76,6 +88,7 @@ _BUILTIN_RULES: "dict[str, tuple]" = {
|
|
|
76
88
|
model_naming_conventions,
|
|
77
89
|
model_directories,
|
|
78
90
|
source_directories,
|
|
91
|
+
test_directories,
|
|
79
92
|
),
|
|
80
93
|
"performance": (
|
|
81
94
|
chained_view_dependencies,
|
|
@@ -84,7 +84,26 @@ code, pre { font-family: "SF Mono", "Roboto Mono", Menlo, Consolas, monospace; }
|
|
|
84
84
|
display: block; padding: 8px 12px; color: var(--text); border-bottom: 1px solid var(--border);
|
|
85
85
|
}
|
|
86
86
|
.search-results a:hover, .search-results a.active { background: var(--accent-soft); text-decoration: none; }
|
|
87
|
+
.search-results .sr-empty { padding: 8px 12px; color: var(--text-soft); font-size: 13px; }
|
|
88
|
+
.search-results .sr-title { display: block; }
|
|
87
89
|
.search-results .sr-meta { color: var(--text-soft); font-size: 12px; }
|
|
90
|
+
.search-results .sr-snippet {
|
|
91
|
+
display: flex; align-items: baseline; gap: 6px; margin-top: 3px;
|
|
92
|
+
font-size: 12px; color: var(--text-soft); min-width: 0;
|
|
93
|
+
}
|
|
94
|
+
.search-results .sr-snippet-field {
|
|
95
|
+
flex-shrink: 0; font-size: 10px; text-transform: uppercase; letter-spacing: 0.03em;
|
|
96
|
+
color: var(--text-soft); border: 1px solid var(--border); border-radius: 4px; padding: 0 4px;
|
|
97
|
+
}
|
|
98
|
+
.search-results .sr-snippet-text {
|
|
99
|
+
overflow: hidden; text-overflow: ellipsis; white-space: nowrap; font-family: monospace;
|
|
100
|
+
}
|
|
101
|
+
.search-results .sr-snippet-text mark { background: var(--accent-soft); color: var(--text); border-radius: 2px; padding: 0 1px; }
|
|
102
|
+
/* Off-screen but readable by assistive tech (the #search-status live region). */
|
|
103
|
+
.visually-hidden {
|
|
104
|
+
position: absolute; width: 1px; height: 1px; margin: -1px; padding: 0;
|
|
105
|
+
overflow: hidden; clip: rect(0 0 0 0); clip-path: inset(50%); border: 0; white-space: nowrap;
|
|
106
|
+
}
|
|
88
107
|
.topbar-actions { display: flex; align-items: center; gap: 10px; margin-left: auto; }
|
|
89
108
|
.icon-btn {
|
|
90
109
|
width: 34px; height: 34px; display: inline-flex; align-items: center; justify-content: center;
|
|
@@ -6,8 +6,11 @@
|
|
|
6
6
|
|
|
7
7
|
var DATA = { metadata: {}, nodes: {}, lineage: {}, columnLineage: {}, erd: {}, tree: { byDatabase: {} }, readme: "", health: { enabled: false } };
|
|
8
8
|
var DOWNSTREAM = null;
|
|
9
|
+
// Cache of the per-node indexed search text, keyed by id, populated by
|
|
10
|
+
// searchDocs() and read by searchSnippet() (both pure, DOM-free). Reset on init().
|
|
11
|
+
var SEARCH_TEXT = null;
|
|
9
12
|
|
|
10
|
-
export function init(data) { DATA = data; DOWNSTREAM = null; }
|
|
13
|
+
export function init(data) { DATA = data; DOWNSTREAM = null; SEARCH_TEXT = null; }
|
|
11
14
|
|
|
12
15
|
export function meta() { return DATA.metadata; }
|
|
13
16
|
export function nodes() { return DATA.nodes; }
|
|
@@ -18,6 +21,171 @@ export function counts() { return DATA.metadata.counts || {}; }
|
|
|
18
21
|
|
|
19
22
|
export function shortName(id) { return String(id).split(".").pop(); }
|
|
20
23
|
|
|
24
|
+
/* Flatten possibly-HTML-bearing text to plain words for the search index: drop
|
|
25
|
+
tags, decode the entities mdInline emits, collapse whitespace. A node's own
|
|
26
|
+
`description` is raw markdown (rendered through mdInline at display time), while
|
|
27
|
+
a column's `description` is pre-escaped HTML carrying <br>; this handles both. */
|
|
28
|
+
function stripHtml(text) {
|
|
29
|
+
return String(text == null ? "" : text)
|
|
30
|
+
.replace(/<[^>]*>/g, " ")
|
|
31
|
+
.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">").replace(/"/g, '"')
|
|
32
|
+
.replace(/\s+/g, " ")
|
|
33
|
+
.trim();
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
/* Above this node count we drop the SQL body (raw + compiled) from the search
|
|
37
|
+
index. Indexing every model's SQL doubles that corpus into MiniSearch's
|
|
38
|
+
in-memory inverted index; on a multi-thousand-model project that's tens of MB
|
|
39
|
+
resident for a marginal-value field. Name/column/tag/macro coverage stays —
|
|
40
|
+
only the full-SQL haystack is traded away at scale. */
|
|
41
|
+
var SQL_INDEX_NODE_CAP = 2000;
|
|
42
|
+
|
|
43
|
+
/* The indexed fields, ordered most→least human-relevant. The snippet builder
|
|
44
|
+
walks this order to pick which matched field to excerpt, and ui mirrors it for
|
|
45
|
+
the MiniSearch `fields` list. `label` is the result title (never excerpted as a
|
|
46
|
+
snippet); the rest are the searchable surface: the human-facing text (name,
|
|
47
|
+
description, tags) plus the structural surface a reader searches by — column
|
|
48
|
+
names + descriptions, the warehouse relation, the package, the macros a model
|
|
49
|
+
calls, and the model SQL (raw + compiled). Each excerptable entry carries a
|
|
50
|
+
display label (what ui shows above a snippet, mkdocs-material style) so the
|
|
51
|
+
dropdown can say *why* a result matched; label/name are the title itself and
|
|
52
|
+
are never excerpted, so their display label is null. */
|
|
53
|
+
export var SEARCH_FIELDS = [
|
|
54
|
+
{ key: "label", label: null },
|
|
55
|
+
{ key: "name", label: null },
|
|
56
|
+
{ key: "columns", label: "Column" },
|
|
57
|
+
{ key: "columnDescriptions", label: "Column docs" },
|
|
58
|
+
{ key: "tags", label: "Tag" },
|
|
59
|
+
{ key: "description", label: "Description" },
|
|
60
|
+
{ key: "relation", label: "Relation" },
|
|
61
|
+
{ key: "package", label: "Package" },
|
|
62
|
+
{ key: "macros", label: "Macro" },
|
|
63
|
+
{ key: "code", label: "SQL" },
|
|
64
|
+
];
|
|
65
|
+
|
|
66
|
+
/* The documents fed to the full-text index (one per node). DOM-free — ui builds
|
|
67
|
+
the MiniSearch instance from these. Indexes the fields in SEARCH_FIELDS; the
|
|
68
|
+
SQL body (`code`) is dropped on large projects (see SQL_INDEX_NODE_CAP) to keep
|
|
69
|
+
the index small. storeFields (label/resource_type/schema) are what the dropdown
|
|
70
|
+
renders. The raw per-field text is cached in SEARCH_TEXT so searchSnippet() can
|
|
71
|
+
excerpt the matched field without a second pass. */
|
|
72
|
+
export function searchDocs() {
|
|
73
|
+
var N = DATA.nodes;
|
|
74
|
+
var ids = Object.keys(N);
|
|
75
|
+
var indexCode = ids.length <= SQL_INDEX_NODE_CAP;
|
|
76
|
+
SEARCH_TEXT = {};
|
|
77
|
+
return ids.map(function (id) {
|
|
78
|
+
var n = N[id];
|
|
79
|
+
var cols = n.columns || [];
|
|
80
|
+
var doc = {
|
|
81
|
+
id: id,
|
|
82
|
+
label: n.label,
|
|
83
|
+
name: n.name,
|
|
84
|
+
resource_type: n.resource_type,
|
|
85
|
+
schema: n.schema,
|
|
86
|
+
description: stripHtml(n.description),
|
|
87
|
+
columns: cols.map(function (c) { return c.name; }).join(" "),
|
|
88
|
+
columnDescriptions: cols.map(function (c) { return stripHtml(c.description); }).join(" "),
|
|
89
|
+
tags: (n.tags || []).join(" "),
|
|
90
|
+
relation: n.relation_name || "",
|
|
91
|
+
package: n.package || "",
|
|
92
|
+
macros: (n.macros || []).map(function (m) { return m.name; }).join(" "),
|
|
93
|
+
code: indexCode ? stripHtml((n.raw_code || "") + " " + (n.compiled_code || "")) : "",
|
|
94
|
+
};
|
|
95
|
+
SEARCH_TEXT[id] = doc;
|
|
96
|
+
return doc;
|
|
97
|
+
});
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
/* The MiniSearch `match` object maps each matched document-term to the list of
|
|
101
|
+
fields it hit. Collapse that to the single most-relevant matched field by
|
|
102
|
+
walking SEARCH_FIELDS order (skipping `label`/`name` — the title already shows
|
|
103
|
+
that). Returns the SEARCH_FIELDS entry ({ key, label }), or null when only
|
|
104
|
+
`label`/`name` matched and the title already explains the hit. */
|
|
105
|
+
function topMatchedField(match) {
|
|
106
|
+
var hitFields = {};
|
|
107
|
+
Object.keys(match || {}).forEach(function (term) {
|
|
108
|
+
(match[term] || []).forEach(function (f) { hitFields[f] = true; });
|
|
109
|
+
});
|
|
110
|
+
for (var i = 0; i < SEARCH_FIELDS.length; i++) {
|
|
111
|
+
var f = SEARCH_FIELDS[i];
|
|
112
|
+
if (f.key === "label" || f.key === "name") continue;
|
|
113
|
+
if (hitFields[f.key]) return f;
|
|
114
|
+
}
|
|
115
|
+
return null;
|
|
116
|
+
}
|
|
117
|
+
|
|
118
|
+
/* A windowed excerpt of `text` centered on the first occurrence of any matched
|
|
119
|
+
term, with ~`pad` chars of context either side and ellipses where clipped.
|
|
120
|
+
Case-insensitive find; returns the head of the text when no term is located
|
|
121
|
+
(e.g. a fuzzy/prefix hit whose surface term differs). Pure string work. */
|
|
122
|
+
function excerpt(text, terms, pad) {
|
|
123
|
+
var hay = String(text || "");
|
|
124
|
+
if (!hay) return "";
|
|
125
|
+
var lower = hay.toLowerCase();
|
|
126
|
+
var at = -1;
|
|
127
|
+
for (var i = 0; i < terms.length; i++) {
|
|
128
|
+
var p = lower.indexOf(String(terms[i]).toLowerCase());
|
|
129
|
+
if (p !== -1 && (at === -1 || p < at)) at = p;
|
|
130
|
+
}
|
|
131
|
+
if (at === -1) return hay.length > pad * 2 ? hay.slice(0, pad * 2) + "…" : hay;
|
|
132
|
+
var start = Math.max(0, at - pad);
|
|
133
|
+
var end = Math.min(hay.length, at + pad);
|
|
134
|
+
return (start > 0 ? "…" : "") + hay.slice(start, end).trim() + (end < hay.length ? "…" : "");
|
|
135
|
+
}
|
|
136
|
+
|
|
137
|
+
/* mkdocs-material-style match context for a search hit: which field matched and
|
|
138
|
+
a short excerpt of it around the query term(s), so the dropdown explains *why*
|
|
139
|
+
a result is there (e.g. searching "unique" surfaces the SQL line it lives on).
|
|
140
|
+
`terms` are the matched document terms (hit.terms from MiniSearch). Returns
|
|
141
|
+
{ field, text } or null when only the title matched. DOM-free — ui highlights
|
|
142
|
+
the terms and renders. */
|
|
143
|
+
export function searchSnippet(id, match, terms) {
|
|
144
|
+
var field = topMatchedField(match);
|
|
145
|
+
if (!field) return null;
|
|
146
|
+
var doc = (SEARCH_TEXT && SEARCH_TEXT[id]) || {};
|
|
147
|
+
var text = excerpt(doc[field.key], terms || [], 40);
|
|
148
|
+
if (!text) return null;
|
|
149
|
+
return { field: field.label, text: text };
|
|
150
|
+
}
|
|
151
|
+
|
|
152
|
+
/* Inline search operators (mkdocs-material style) the user types into the box:
|
|
153
|
+
type:<resource_type> restrict hits to a resource_type (model/source/…)
|
|
154
|
+
label:<text> / name:<text> match only against the name/label fields,
|
|
155
|
+
skipping the SQL/description/column noise
|
|
156
|
+
Operators combine with the free-text remainder: `type:model orders` finds
|
|
157
|
+
models whose text matches "orders"; `label:stg` matches names containing
|
|
158
|
+
"stg". A bare query (no operators) searches everything. */
|
|
159
|
+
var SEARCH_OPERATOR = /(\w+):(\S+)/g;
|
|
160
|
+
var NAME_FIELDS = ["label", "name"];
|
|
161
|
+
|
|
162
|
+
/* Parse a raw query into MiniSearch inputs. Returns:
|
|
163
|
+
{ text } the free-text remainder to search (operators stripped)
|
|
164
|
+
{ fields } restrict matching to these fields, or null for all
|
|
165
|
+
{ filterType } restrict hits to this resource_type, or null
|
|
166
|
+
{ operators } the recognized operators (for ui to echo as chips)
|
|
167
|
+
DOM-free — ui owns running mini.search() and rendering. */
|
|
168
|
+
export function parseSearchQuery(raw) {
|
|
169
|
+
var fields = null;
|
|
170
|
+
var filterType = null;
|
|
171
|
+
var operators = [];
|
|
172
|
+
var text = String(raw || "").replace(SEARCH_OPERATOR, function (m, key, val) {
|
|
173
|
+
var k = key.toLowerCase();
|
|
174
|
+
if (k === "type" || k === "resource_type") {
|
|
175
|
+
filterType = val.toLowerCase();
|
|
176
|
+
operators.push({ key: "type", value: filterType });
|
|
177
|
+
return " ";
|
|
178
|
+
}
|
|
179
|
+
if (k === "label" || k === "name") {
|
|
180
|
+
fields = NAME_FIELDS;
|
|
181
|
+
operators.push({ key: "label", value: val });
|
|
182
|
+
return " " + val + " "; // the operator's value is still the text to match
|
|
183
|
+
}
|
|
184
|
+
return m; // unrecognized prefix (e.g. a "schema:foo" the index doesn't carry) — leave it as a literal term
|
|
185
|
+
}).replace(/\s+/g, " ").trim();
|
|
186
|
+
return { text: text, fields: fields, filterType: filterType, operators: operators };
|
|
187
|
+
}
|
|
188
|
+
|
|
21
189
|
/* The searchable text for a node in the sidebar tree filter: its table name and
|
|
22
190
|
its fully-qualified database.schema.table, lowercased. Lets the tree filter
|
|
23
191
|
match either "orders" or "shaman.jf.orders". DOM-free (ui does the show/hide). */
|
|
@@ -651,37 +651,157 @@ function toggleFullscreen(host) {
|
|
|
651
651
|
}
|
|
652
652
|
}
|
|
653
653
|
|
|
654
|
+
var SEARCH_RESULT_CAP = 12;
|
|
655
|
+
|
|
654
656
|
function buildSearch() {
|
|
655
657
|
var input = document.getElementById("search");
|
|
656
658
|
var results = document.getElementById("search-results");
|
|
659
|
+
var status = document.getElementById("search-status");
|
|
657
660
|
if (typeof MiniSearch === "undefined") return;
|
|
658
|
-
var
|
|
659
|
-
var
|
|
660
|
-
|
|
661
|
-
|
|
662
|
-
|
|
661
|
+
var docs = svc.searchDocs();
|
|
662
|
+
var mini = new MiniSearch({
|
|
663
|
+
fields: svc.SEARCH_FIELDS.map(function (f) { return f.key; }),
|
|
664
|
+
storeFields: ["label", "resource_type", "schema"],
|
|
665
|
+
// Names/columns/tags expand fuzzily + by prefix so a half-typed model name
|
|
666
|
+
// still hits. The bulk text fields (description, column docs, SQL) match only
|
|
667
|
+
// on whole terms and at a low weight — otherwise a generic SQL keyword like
|
|
668
|
+
// "unique" fuzzy-floods nearly every model. boost ranks name/column hits well
|
|
669
|
+
// above an incidental SQL-body hit so the title results stay on top.
|
|
670
|
+
searchOptions: {
|
|
671
|
+
prefix: true,
|
|
672
|
+
fuzzy: 0.2,
|
|
673
|
+
boost: { label: 6, name: 6, columns: 3, tags: 2, relation: 2, description: 1, columnDescriptions: 1, code: 0.4 },
|
|
674
|
+
},
|
|
663
675
|
});
|
|
664
|
-
var mini = new MiniSearch({ fields: ["label", "description", "columns"], storeFields: ["label", "resource_type", "schema"], searchOptions: { prefix: true, fuzzy: 0.2, boost: { label: 3 } } });
|
|
665
676
|
mini.addAll(docs);
|
|
666
677
|
|
|
678
|
+
// Build the matched-field snippet (mkdocs-material style): a label for the
|
|
679
|
+
// field that matched + the excerpt, with the matched terms wrapped in <mark>.
|
|
680
|
+
// Highlighting is done by splitting on the matched terms and building text +
|
|
681
|
+
// <mark> nodes (no innerHTML) so user-derived text can never inject markup.
|
|
682
|
+
function highlightNodes(text, terms) {
|
|
683
|
+
var safe = (terms || []).map(function (t) { return String(t).replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); }).filter(Boolean);
|
|
684
|
+
if (!safe.length) return [text];
|
|
685
|
+
var re = new RegExp("(" + safe.join("|") + ")", "ig");
|
|
686
|
+
return String(text).split(re).map(function (piece, i) {
|
|
687
|
+
// split() with a capturing group puts the matched delimiters at odd indices.
|
|
688
|
+
return i % 2 === 1 ? el("mark", null, [piece]) : piece;
|
|
689
|
+
});
|
|
690
|
+
}
|
|
691
|
+
|
|
692
|
+
function snippetNode(h) {
|
|
693
|
+
var snip = svc.searchSnippet(h.id, h.match, h.terms);
|
|
694
|
+
if (!snip) return null;
|
|
695
|
+
return el("span", { class: "sr-snippet" }, [
|
|
696
|
+
el("span", { class: "sr-snippet-field" }, [snip.field]),
|
|
697
|
+
el("span", { class: "sr-snippet-text" }, highlightNodes(snip.text, h.terms)),
|
|
698
|
+
]);
|
|
699
|
+
}
|
|
700
|
+
|
|
701
|
+
// Roving active-descendant state for keyboard nav: index into the rendered
|
|
702
|
+
// option rows, or -1 for "input itself focused, nothing selected".
|
|
703
|
+
var activeIndex = -1;
|
|
704
|
+
|
|
705
|
+
function setExpanded(open) {
|
|
706
|
+
results.hidden = !open;
|
|
707
|
+
input.setAttribute("aria-expanded", open ? "true" : "false");
|
|
708
|
+
if (!open) {
|
|
709
|
+
activeIndex = -1;
|
|
710
|
+
input.removeAttribute("aria-activedescendant");
|
|
711
|
+
status.textContent = "";
|
|
712
|
+
}
|
|
713
|
+
}
|
|
714
|
+
|
|
715
|
+
function optionRows() { return results.querySelectorAll('[role="option"]'); }
|
|
716
|
+
|
|
717
|
+
// Move the roving selection by `delta` (wrapping), reflect it as the .active
|
|
718
|
+
// class + aria-selected, and point the input's aria-activedescendant at it so
|
|
719
|
+
// a screen reader announces the row without moving DOM focus off the input.
|
|
720
|
+
function moveActive(delta) {
|
|
721
|
+
var rows = optionRows();
|
|
722
|
+
if (!rows.length) return;
|
|
723
|
+
if (activeIndex >= 0 && rows[activeIndex]) rows[activeIndex].setAttribute("aria-selected", "false");
|
|
724
|
+
activeIndex = (activeIndex + delta + rows.length) % rows.length;
|
|
725
|
+
rows.forEach(function (r, i) { r.classList.toggle("active", i === activeIndex); });
|
|
726
|
+
var active = rows[activeIndex];
|
|
727
|
+
active.setAttribute("aria-selected", "true");
|
|
728
|
+
input.setAttribute("aria-activedescendant", active.id);
|
|
729
|
+
active.scrollIntoView({ block: "nearest" });
|
|
730
|
+
}
|
|
731
|
+
|
|
667
732
|
function render(hits) {
|
|
668
733
|
clear(results);
|
|
669
|
-
|
|
670
|
-
|
|
671
|
-
|
|
672
|
-
|
|
673
|
-
|
|
674
|
-
|
|
734
|
+
activeIndex = -1;
|
|
735
|
+
input.removeAttribute("aria-activedescendant");
|
|
736
|
+
if (!hits.length) {
|
|
737
|
+
// A non-empty query with no hits gets an explicit cue, mirrored into the
|
|
738
|
+
// #search-status live region for screen readers.
|
|
739
|
+
var noMatches = "No matches.";
|
|
740
|
+
results.appendChild(el("div", { class: "sr-empty" }, [noMatches]));
|
|
741
|
+
status.textContent = noMatches;
|
|
742
|
+
setExpanded(true);
|
|
743
|
+
return;
|
|
744
|
+
}
|
|
745
|
+
var shown = Math.min(hits.length, SEARCH_RESULT_CAP);
|
|
746
|
+
status.textContent = shown === 1 ? "1 result." : shown + " results.";
|
|
747
|
+
hits.slice(0, SEARCH_RESULT_CAP).forEach(function (h, i) {
|
|
748
|
+
var children = [
|
|
749
|
+
el("span", { class: "sr-title" }, [
|
|
750
|
+
el("span", { class: "dot " + h.resource_type }), " " + h.label,
|
|
751
|
+
el("span", { class: "sr-meta" }, [" " + h.resource_type + " · " + h.schema]),
|
|
752
|
+
]),
|
|
753
|
+
];
|
|
754
|
+
var snip = snippetNode(h);
|
|
755
|
+
if (snip) children.push(snip);
|
|
756
|
+
results.appendChild(el("a", {
|
|
757
|
+
id: "search-opt-" + i,
|
|
758
|
+
role: "option",
|
|
759
|
+
"aria-selected": "false",
|
|
760
|
+
href: "#/node/" + encodeURIComponent(h.id),
|
|
761
|
+
onclick: function () { closeSearch(); },
|
|
762
|
+
}, children));
|
|
675
763
|
});
|
|
676
|
-
|
|
764
|
+
setExpanded(true);
|
|
765
|
+
}
|
|
766
|
+
|
|
767
|
+
function closeSearch() { setExpanded(false); input.value = ""; setSearchOpen(false); }
|
|
768
|
+
// Map service's parsed operators to MiniSearch's field restriction + a
|
|
769
|
+
// resource_type filter. A bare `type:` (no free text) scans for the first
|
|
770
|
+
// SEARCH_RESULT_CAP nodes of that type — O(cap), not O(N) on a huge project.
|
|
771
|
+
function runQuery(raw) {
|
|
772
|
+
var p = svc.parseSearchQuery(raw);
|
|
773
|
+
if (!p.text) {
|
|
774
|
+
if (!p.filterType) return [];
|
|
775
|
+
var picked = [];
|
|
776
|
+
for (var i = 0; i < docs.length && picked.length < SEARCH_RESULT_CAP; i++) {
|
|
777
|
+
if (docs[i].resource_type === p.filterType) picked.push(docs[i]);
|
|
778
|
+
}
|
|
779
|
+
return picked;
|
|
780
|
+
}
|
|
781
|
+
var opts = {};
|
|
782
|
+
if (p.fields) opts.fields = p.fields;
|
|
783
|
+
if (p.filterType) opts.filter = function (h) { return h.resource_type === p.filterType; };
|
|
784
|
+
return mini.search(p.text, opts);
|
|
677
785
|
}
|
|
678
786
|
input.addEventListener("input", function () {
|
|
679
787
|
var q = input.value.trim();
|
|
680
|
-
if (!q) {
|
|
681
|
-
render(
|
|
788
|
+
if (!q) { setExpanded(false); return; }
|
|
789
|
+
render(runQuery(q));
|
|
790
|
+
});
|
|
791
|
+
input.addEventListener("focus", function () { if (input.value.trim()) render(runQuery(input.value.trim())); });
|
|
792
|
+
// Keyboard nav: ↑/↓ rove the options, Enter follows the active one (or the
|
|
793
|
+
// first when none is roved), Escape closes. Other keys fall through to typing.
|
|
794
|
+
input.addEventListener("keydown", function (e) {
|
|
795
|
+
if (results.hidden) return;
|
|
796
|
+
if (e.key === "ArrowDown") { e.preventDefault(); moveActive(1); }
|
|
797
|
+
else if (e.key === "ArrowUp") { e.preventDefault(); moveActive(-1); }
|
|
798
|
+
else if (e.key === "Enter") {
|
|
799
|
+
var rows = optionRows();
|
|
800
|
+
var target = rows[activeIndex >= 0 ? activeIndex : 0];
|
|
801
|
+
if (target) { e.preventDefault(); location.hash = target.getAttribute("href"); closeSearch(); }
|
|
802
|
+
} else if (e.key === "Escape") { setExpanded(false); }
|
|
682
803
|
});
|
|
683
|
-
|
|
684
|
-
document.addEventListener("click", function (e) { if (!results.contains(e.target) && e.target !== input) results.hidden = true; });
|
|
804
|
+
document.addEventListener("click", function (e) { if (!results.contains(e.target) && e.target !== input) setExpanded(false); });
|
|
685
805
|
}
|
|
686
806
|
|
|
687
807
|
function initTheme() {
|
|
@@ -21,8 +21,9 @@
|
|
|
21
21
|
<span class="brand-generated" id="brand-generated" hidden></span>
|
|
22
22
|
</div>
|
|
23
23
|
<div class="search-wrap">
|
|
24
|
-
<input id="search" type="search"
|
|
25
|
-
<div id="search-results" class="search-results" hidden></div>
|
|
24
|
+
<input id="search" type="search" role="combobox" aria-label="Search" aria-expanded="false" aria-controls="search-results" aria-autocomplete="list" autocomplete="off" placeholder="Search models, columns, tags…" />
|
|
25
|
+
<div id="search-results" class="search-results" role="listbox" aria-label="Search results" hidden></div>
|
|
26
|
+
<div id="search-status" class="visually-hidden" role="status" aria-live="polite"></div>
|
|
26
27
|
</div>
|
|
27
28
|
<div class="topbar-actions">
|
|
28
29
|
<button id="search-toggle" class="icon-btn search-toggle" aria-label="Toggle search" aria-expanded="false">
|
dbdocs-1.2.0/README.md
DELETED
|
@@ -1,69 +0,0 @@
|
|
|
1
|
-
<p align="center">
|
|
2
|
-
<img src="docs/assets/logo.svg" alt="dbdocs logo" width="220" height="88">
|
|
3
|
-
</p>
|
|
4
|
-
|
|
5
|
-
<p align="center"><b>An alternative dbt docs site — catalog + ERD + column-level lineage + versioned deploys, all in one CLI.</b></p>
|
|
6
|
-
|
|
7
|
-
<p align="center">
|
|
8
|
-
<a href="https://dbdocs.datnguye.me/latest/demo/latest/"><img src="https://img.shields.io/badge/live-demo-FF694A?style=flat&logo=rocket&logoColor=white" alt="live demo"></a>
|
|
9
|
-
<a href="https://dbdocs.datnguye.me/"><img src="https://img.shields.io/badge/docs-visit%20site-blue?style=flat&logo=gitbook&logoColor=white" alt="docs"></a>
|
|
10
|
-
<a href="https://pypi.org/project/dbdocs/"><img src="https://badge.fury.io/py/dbdocs.svg" alt="PyPI version"></a>
|
|
11
|
-
<img src="https://img.shields.io/badge/CLI-Python-FFCE3E?labelColor=14354C&logo=python&logoColor=white" alt="python-cli">
|
|
12
|
-
<a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
|
|
13
|
-
<a href="https://www.python.org"><img src="https://img.shields.io/badge/Python-3.10|3.11|3.12|3.13-3776AB.svg?style=flat&logo=python&logoColor=white" alt="python"></a>
|
|
14
|
-
</p>
|
|
15
|
-
|
|
16
|
-
Turn your dbt artifacts into a self-contained docs site: a browsable catalog, an entity-relationship diagram, an interactive lineage DAG, and **column-level lineage** traced from your compiled SQL — all in one `dbdocs generate`. Serve it with `dbdocs serve`, or deploy versioned builds anywhere a static host will take them.
|
|
17
|
-
|
|
18
|
-
| Catalog | Model page | Lineage DAG | Health Check |
|
|
19
|
-
|---|---|---|---|
|
|
20
|
-
|  |  |  |  |
|
|
21
|
-
|
|
22
|
-
## Why dbdocs?
|
|
23
|
-
|
|
24
|
-
dbt's built-in docs are great — right up until you want to know *which upstream column fed this downstream column*, or *which tables relate to each other*, or *what changed between last week's docs and today's*. dbdocs fills all three gaps without asking you to install a documentation framework or maintain a separate ERD tool.
|
|
25
|
-
|
|
26
|
-
**What you get that nothing else bundles together:**
|
|
27
|
-
|
|
28
|
-
- **ERD + column-level lineage, side by side.** The entity-relationship diagram (powered by [dbterd](https://github.com/datnguye/dbterd)) shows table relationships; column lineage (traced by [sqlglot](https://github.com/tobymao/sqlglot) from compiled SQL) shows exactly which column fed which. Most alternatives give you one or the other — dbdocs gives you both.
|
|
29
|
-
- **Column impact analysis.** Select any column and see its downstream dependents across the project, so you know what a schema change will break before you run it.
|
|
30
|
-
- **Deep-link URLs.** Every focused node, column, and filtered DAG view has a shareable URL. Paste it in Slack and your teammate lands on exactly the right model, column, or graph state.
|
|
31
|
-
- **Any sqlglot-supported dialect.** The dialect for column-lineage parsing is auto-detected from your manifest's `adapter_type` (Snowflake, BigQuery, Redshift, DuckDB, PostgreSQL, Databricks/Spark, Trino, and more — anything sqlglot understands). Override it per-project with `dialect:` in `dbdocs.yml` when auto-detection isn't enough.
|
|
32
|
-
- **Scales without freezing.** Column-lineage parsing fans out across CPU cores automatically above 500 models, so large projects finish in roughly the same wall-clock time as small ones. The DAG is windowed by React Flow, so a 1 000-model graph doesn't turn your browser into a space heater. The payload ships as an external gzip (`dbdocs-data.json.gz`, decompressed client-side) so `index.html` stays tiny regardless of project size.
|
|
33
|
-
- **Fail-soft.** One model with SQL sqlglot can't parse gets skipped and logged — it never sinks the whole generate run.
|
|
34
|
-
- **Project Health Check.** A scorecard across the six [dbt-project-evaluator](https://dbt-labs.github.io/dbt-project-evaluator/) dimensions (testing, modeling, documentation, structure, performance, governance), computed straight from your `manifest.json` — no extra dbt package, no warehouse. When a `run_results.json` is also present (any `dbt build`/`dbt test`; default `<target_dir>/run_results.json`, override with `--run-results`), each test additionally shows up as a pass/fail finding grouped by what it checks (integrity, referential, validity, business logic, freshness). dbdocs only reads the artifacts — it never runs dbt or touches your warehouse. Fail-soft: a missing `run_results.json` just drops the per-test detail; the dimensions still render.
|
|
35
|
-
- **Versioned deploys, no plugins.** `dbdocs deploy --version v1.2 --alias latest` generates into a plain directory tree, writes a `versions.json` index, and the SPA renders a version dropdown. No mike, no external tooling, no surprise dependencies.
|
|
36
|
-
- **Catalog navigation + client-side search.** Models, seeds, and snapshots grouped by database and schema; full-text search without a backend.
|
|
37
|
-
- **Dark / light theme.**
|
|
38
|
-
|
|
39
|
-
## Install
|
|
40
|
-
|
|
41
|
-
```bash
|
|
42
|
-
pip install dbdocs --upgrade
|
|
43
|
-
```
|
|
44
|
-
|
|
45
|
-
Requires Python 3.10+.
|
|
46
|
-
|
|
47
|
-
## Quickstart
|
|
48
|
-
|
|
49
|
-
```bash
|
|
50
|
-
dbt docs generate # writes target/manifest.json + target/catalog.json
|
|
51
|
-
dbdocs generate # builds ./site/ with index.html + dbdocs-data.json.gz
|
|
52
|
-
dbdocs serve # static http server on http://127.0.0.1:8000
|
|
53
|
-
```
|
|
54
|
-
|
|
55
|
-
The site must be served over HTTP (not opened as a local file) because it fetches the data payload at load time. `dbdocs serve` handles that locally; any static host works for deployment.
|
|
56
|
-
|
|
57
|
-
Full walkthrough, configuration, and architecture live in the **[documentation](https://dbdocs.datnguye.me/)**.
|
|
58
|
-
|
|
59
|
-
## Contributing
|
|
60
|
-
|
|
61
|
-
Contributions are welcome — bugs, features, docs, typos. See the **[Contributing Guide](https://dbdocs.datnguye.me/latest/nav/development/contributing-guide.html)**.
|
|
62
|
-
|
|
63
|
-
If dbdocs saves you some clicks, consider [buying me a coffee](https://www.buymeacoffee.com/datnguye).
|
|
64
|
-
|
|
65
|
-
<a href="https://www.buymeacoffee.com/datnguye"><img src="https://img.shields.io/badge/buy%20me%20a%20coffee-donate-yellow.svg?logo=buy-me-a-coffee&logoColor=white&labelColor=ff813f&style=for-the-badge" alt="buy me a coffee"></a>
|
|
66
|
-
|
|
67
|
-
## License
|
|
68
|
-
|
|
69
|
-
[MIT](./LICENSE) © Dat Nguyen
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|