dbdocs 1.2.0__tar.gz → 1.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (52) hide show
  1. {dbdocs-1.2.0 → dbdocs-1.3.0}/.gitignore +2 -0
  2. {dbdocs-1.2.0 → dbdocs-1.3.0}/PKG-INFO +12 -14
  3. dbdocs-1.3.0/README.md +67 -0
  4. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/dimensions.py +22 -0
  5. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/__init__.py +14 -1
  6. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/base.py +17 -2
  7. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/dimensions/documentation.py +30 -0
  8. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/dimensions/modeling.py +96 -0
  9. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/dimensions/structure.py +18 -0
  10. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/dimensions/testing.py +19 -0
  11. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/registry.py +14 -1
  12. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/css/style.css +19 -0
  13. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/js/service.js +169 -1
  14. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/js/ui.js +137 -17
  15. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/index.html +3 -2
  16. dbdocs-1.2.0/README.md +0 -69
  17. {dbdocs-1.2.0 → dbdocs-1.3.0}/LICENSE +0 -0
  18. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/__init__.py +0 -0
  19. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/__main__.py +0 -0
  20. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/cli/__init__.py +0 -0
  21. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/cli/main.py +0 -0
  22. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/core/__init__.py +0 -0
  23. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/core/artifacts.py +0 -0
  24. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/core/config.py +0 -0
  25. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/core/exceptions.py +0 -0
  26. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/core/log.py +0 -0
  27. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/__init__.py +0 -0
  28. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/_sqlglot_lineage.py +0 -0
  29. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/column_lineage.py +0 -0
  30. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/erd.py +0 -0
  31. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/erd_json.py +0 -0
  32. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/graph.py +0 -0
  33. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/__init__.py +0 -0
  34. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/extractor.py +0 -0
  35. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/dimensions/__init__.py +0 -0
  36. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/dimensions/governance.py +0 -0
  37. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/health/rules/dimensions/performance.py +0 -0
  38. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/extract/nodes.py +0 -0
  39. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/main.py +0 -0
  40. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/__init__.py +0 -0
  41. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/builder.py +0 -0
  42. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/css/icons.css +0 -0
  43. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/favicon.svg +0 -0
  44. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/graph/index.css +0 -0
  45. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/graph/index.js +0 -0
  46. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/js/app.js +0 -0
  47. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/js/data.js +0 -0
  48. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/vendor/marked.min.js +0 -0
  49. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/bundle/assets/vendor/minisearch.min.js +0 -0
  50. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/deploy.py +0 -0
  51. {dbdocs-1.2.0 → dbdocs-1.3.0}/dbdocs/site/inject.py +0 -0
  52. {dbdocs-1.2.0 → dbdocs-1.3.0}/pyproject.toml +0 -0
@@ -154,6 +154,8 @@ frontend/coverage/
154
154
  frontend/*.tsbuildinfo
155
155
  frontend/test-results/
156
156
  frontend/playwright-report/
157
+ test-results/
158
+ playwright-report/
157
159
  .eslintcache
158
160
 
159
161
  # npm / yarn / pnpm logs & debug output
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: dbdocs
3
- Version: 1.2.0
3
+ Version: 1.3.0
4
4
  Summary: Alternative dbt docs site: Catalog + ERD + column-level lineage
5
5
  Project-URL: Homepage, https://github.com/datnguye/dbt-docs
6
6
  Project-URL: Repository, https://github.com/datnguye/dbt-docs
@@ -45,19 +45,17 @@ Turn your dbt artifacts into a self-contained docs site: a browsable catalog, an
45
45
 
46
46
  ## Why dbdocs?
47
47
 
48
- dbt's built-in docs are great — right up until you want to know *which upstream column fed this downstream column*, or *which tables relate to each other*, or *what changed between last week's docs and today's*. dbdocs fills all three gaps without asking you to install a documentation framework or maintain a separate ERD tool.
49
-
50
- **What you get that nothing else bundles together:**
51
-
52
- - **ERD + column-level lineage, side by side.** The entity-relationship diagram (powered by [dbterd](https://github.com/datnguye/dbterd)) shows table relationships; column lineage (traced by [sqlglot](https://github.com/tobymao/sqlglot) from compiled SQL) shows exactly which column fed which. Most alternatives give you one or the other — dbdocs gives you both.
53
- - **Column impact analysis.** Select any column and see its downstream dependents across the project, so you know what a schema change will break before you run it.
54
- - **Deep-link URLs.** Every focused node, column, and filtered DAG view has a shareable URL. Paste it in Slack and your teammate lands on exactly the right model, column, or graph state.
55
- - **Any sqlglot-supported dialect.** The dialect for column-lineage parsing is auto-detected from your manifest's `adapter_type` (Snowflake, BigQuery, Redshift, DuckDB, PostgreSQL, Databricks/Spark, Trino, and more — anything sqlglot understands). Override it per-project with `dialect:` in `dbdocs.yml` when auto-detection isn't enough.
56
- - **Scales without freezing.** Column-lineage parsing fans out across CPU cores automatically above 500 models, so large projects finish in roughly the same wall-clock time as small ones. The DAG is windowed by React Flow, so a 1 000-model graph doesn't turn your browser into a space heater. The payload ships as an external gzip (`dbdocs-data.json.gz`, decompressed client-side) so `index.html` stays tiny regardless of project size.
57
- - **Fail-soft.** One model with SQL sqlglot can't parse gets skipped and logged — it never sinks the whole generate run.
58
- - **Project Health Check.** A scorecard across the six [dbt-project-evaluator](https://dbt-labs.github.io/dbt-project-evaluator/) dimensions (testing, modeling, documentation, structure, performance, governance), computed straight from your `manifest.json` — no extra dbt package, no warehouse. When a `run_results.json` is also present (any `dbt build`/`dbt test`; default `<target_dir>/run_results.json`, override with `--run-results`), each test additionally shows up as a pass/fail finding grouped by what it checks (integrity, referential, validity, business logic, freshness). dbdocs only reads the artifacts — it never runs dbt or touches your warehouse. Fail-soft: a missing `run_results.json` just drops the per-test detail; the dimensions still render.
59
- - **Versioned deploys, no plugins.** `dbdocs deploy --version v1.2 --alias latest` generates into a plain directory tree, writes a `versions.json` index, and the SPA renders a version dropdown. No mike, no external tooling, no surprise dependencies.
60
- - **Catalog navigation + client-side search.** Models, seeds, and snapshots grouped by database and schema; full-text search without a backend.
48
+ dbt's built-in docs stop short of telling you *which upstream column fed this downstream column*, *which tables relate to each other*, or *what changed between builds*. dbdocs fills those gaps — no documentation framework or separate ERD tool to install.
49
+
50
+ - **ERD + column-level lineage** — table relationships ([dbterd](https://github.com/datnguye/dbterd)) and column lineage from compiled SQL ([sqlglot](https://github.com/tobymao/sqlglot)).
51
+ - **Column impact analysis** — downstream dependents for any column.
52
+ - **Deep-link URLs** for every node, column, and DAG view.
53
+ - **Any sqlglot dialect**, auto-detected from your manifest.
54
+ - **Scales to 1 000s of models** without freezing the browser.
55
+ - **Fail-soft** — an unparseable model is skipped, not fatal.
56
+ - **Project Health Check** across the six [dbt-project-evaluator](https://dbt-labs.github.io/dbt-project-evaluator/) dimensions.
57
+ - **Versioned deploys** with a built-in version switcher, no plugins.
58
+ - **Full-text search** across names, columns, descriptions, tags, and SQL at the client-side, no backend.
61
59
  - **Dark / light theme.**
62
60
 
63
61
  ## Install
dbdocs-1.3.0/README.md ADDED
@@ -0,0 +1,67 @@
1
+ <p align="center">
2
+ <img src="docs/assets/logo.svg" alt="dbdocs logo" width="220" height="88">
3
+ </p>
4
+
5
+ <p align="center"><b>An alternative dbt docs site — catalog + ERD + column-level lineage + versioned deploys, all in one CLI.</b></p>
6
+
7
+ <p align="center">
8
+ <a href="https://dbdocs.datnguye.me/latest/demo/latest/"><img src="https://img.shields.io/badge/live-demo-FF694A?style=flat&logo=rocket&logoColor=white" alt="live demo"></a>
9
+ <a href="https://dbdocs.datnguye.me/"><img src="https://img.shields.io/badge/docs-visit%20site-blue?style=flat&logo=gitbook&logoColor=white" alt="docs"></a>
10
+ <a href="https://pypi.org/project/dbdocs/"><img src="https://badge.fury.io/py/dbdocs.svg" alt="PyPI version"></a>
11
+ <img src="https://img.shields.io/badge/CLI-Python-FFCE3E?labelColor=14354C&logo=python&logoColor=white" alt="python-cli">
12
+ <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
13
+ <a href="https://www.python.org"><img src="https://img.shields.io/badge/Python-3.10|3.11|3.12|3.13-3776AB.svg?style=flat&logo=python&logoColor=white" alt="python"></a>
14
+ </p>
15
+
16
+ Turn your dbt artifacts into a self-contained docs site: a browsable catalog, an entity-relationship diagram, an interactive lineage DAG, and **column-level lineage** traced from your compiled SQL — all in one `dbdocs generate`. Serve it with `dbdocs serve`, or deploy versioned builds anywhere a static host will take them.
17
+
18
+ | Catalog | Model page | Lineage DAG | Health Check |
19
+ |---|---|---|---|
20
+ | ![catalog](docs/assets/img/demo-catalog.png) | ![model page](docs/assets/img/demo-model-page.png) | ![dag](docs/assets/img/demo-model-dag.png) | ![health check](docs/assets/img/demo-health.png) |
21
+
22
+ ## Why dbdocs?
23
+
24
+ dbt's built-in docs stop short of telling you *which upstream column fed this downstream column*, *which tables relate to each other*, or *what changed between builds*. dbdocs fills those gaps — no documentation framework or separate ERD tool to install.
25
+
26
+ - **ERD + column-level lineage** — table relationships ([dbterd](https://github.com/datnguye/dbterd)) and column lineage from compiled SQL ([sqlglot](https://github.com/tobymao/sqlglot)).
27
+ - **Column impact analysis** — downstream dependents for any column.
28
+ - **Deep-link URLs** for every node, column, and DAG view.
29
+ - **Any sqlglot dialect**, auto-detected from your manifest.
30
+ - **Scales to 1 000s of models** without freezing the browser.
31
+ - **Fail-soft** — an unparseable model is skipped, not fatal.
32
+ - **Project Health Check** across the six [dbt-project-evaluator](https://dbt-labs.github.io/dbt-project-evaluator/) dimensions.
33
+ - **Versioned deploys** with a built-in version switcher, no plugins.
34
+ - **Full-text search** across names, columns, descriptions, tags, and SQL at the client-side, no backend.
35
+ - **Dark / light theme.**
36
+
37
+ ## Install
38
+
39
+ ```bash
40
+ pip install dbdocs --upgrade
41
+ ```
42
+
43
+ Requires Python 3.10+.
44
+
45
+ ## Quickstart
46
+
47
+ ```bash
48
+ dbt docs generate # writes target/manifest.json + target/catalog.json
49
+ dbdocs generate # builds ./site/ with index.html + dbdocs-data.json.gz
50
+ dbdocs serve # static http server on http://127.0.0.1:8000
51
+ ```
52
+
53
+ The site must be served over HTTP (not opened as a local file) because it fetches the data payload at load time. `dbdocs serve` handles that locally; any static host works for deployment.
54
+
55
+ Full walkthrough, configuration, and architecture live in the **[documentation](https://dbdocs.datnguye.me/)**.
56
+
57
+ ## Contributing
58
+
59
+ Contributions are welcome — bugs, features, docs, typos. See the **[Contributing Guide](https://dbdocs.datnguye.me/latest/nav/development/contributing-guide.html)**.
60
+
61
+ If dbdocs saves you some clicks, consider [buying me a coffee](https://www.buymeacoffee.com/datnguye).
62
+
63
+ <a href="https://www.buymeacoffee.com/datnguye"><img src="https://img.shields.io/badge/buy%20me%20a%20coffee-donate-yellow.svg?logo=buy-me-a-coffee&logoColor=white&labelColor=ff813f&style=for-the-badge" alt="buy me a coffee"></a>
64
+
65
+ ## License
66
+
67
+ [MIT](./LICENSE) © Dat Nguyen
@@ -50,6 +50,13 @@ class ManifestGraph:
50
50
  self.models = [n for uid, n in self._nodes.items() if uid.startswith("model.")]
51
51
  self.sources = list(self._sources.values())
52
52
  self.exposures = list(self._exposures.values())
53
+ # Singular tests are custom-SQL test nodes (no test_metadata); generic
54
+ # tests (unique/not_null/…) carry test_metadata and are excluded.
55
+ self.singular_tests = [
56
+ n
57
+ for uid, n in self._nodes.items()
58
+ if uid.startswith("test.") and getattr(n, "test_metadata", None) is None
59
+ ]
53
60
 
54
61
  # Rule thresholds: per-run overrides layered over the DPE defaults.
55
62
  self._thresholds = {**DEFAULT_THRESHOLDS, **(thresholds or {})}
@@ -147,6 +154,21 @@ class ManifestGraph:
147
154
  access = access or getattr(model, "access", None)
148
155
  return str(access or "protected").lower()
149
156
 
157
+ @staticmethod
158
+ def has_source_freshness(source: Any) -> bool:
159
+ """Whether a source has a freshness check: a ``loaded_at_field`` plus a
160
+ ``warn_after``/``error_after`` threshold count."""
161
+ if not str(getattr(source, "loaded_at_field", "") or "").strip():
162
+ return False
163
+ freshness = getattr(source, "freshness", None)
164
+ if freshness is None:
165
+ return False
166
+ for bound in ("warn_after", "error_after"):
167
+ period = getattr(freshness, bound, None)
168
+ if period is not None and getattr(period, "count", None) is not None:
169
+ return True
170
+ return False
171
+
150
172
  @staticmethod
151
173
  def contract_enforced(model: Any) -> bool:
152
174
  """Whether the model has an enforced contract (``contract.enforced``)."""
@@ -24,6 +24,7 @@ of :mod:`base` and :mod:`registry`; no implementation lives here.
24
24
 
25
25
  from dbdocs.extract.health.rules.base import DEFAULT_THRESHOLDS, NON_PHYSICAL, docs_url, finding
26
26
  from dbdocs.extract.health.rules.dimensions.documentation import (
27
+ documentation_coverage,
27
28
  undocumented_models,
28
29
  undocumented_source_tables,
29
30
  undocumented_sources,
@@ -35,7 +36,9 @@ from dbdocs.extract.health.rules.dimensions.governance import (
35
36
  )
36
37
  from dbdocs.extract.health.rules.dimensions.modeling import (
37
38
  direct_join_to_source,
39
+ downstream_models_dependent_on_source,
38
40
  duplicate_sources,
41
+ hard_coded_references,
39
42
  model_fanout,
40
43
  multiple_sources_joined,
41
44
  rejoining_of_upstream_concepts,
@@ -54,8 +57,13 @@ from dbdocs.extract.health.rules.dimensions.structure import (
54
57
  model_directories,
55
58
  model_naming_conventions,
56
59
  source_directories,
60
+ test_directories,
61
+ )
62
+ from dbdocs.extract.health.rules.dimensions.testing import (
63
+ missing_primary_key_tests,
64
+ missing_source_freshness,
65
+ test_coverage,
57
66
  )
58
- from dbdocs.extract.health.rules.dimensions.testing import missing_primary_key_tests, test_coverage
59
67
  from dbdocs.extract.health.rules.registry import (
60
68
  DIMENSION_RULES,
61
69
  ENTRY_POINT_GROUP,
@@ -81,8 +89,11 @@ __all__ = [
81
89
  # rule functions (re-exported for direct access / testing)
82
90
  "test_coverage",
83
91
  "missing_primary_key_tests",
92
+ "missing_source_freshness",
84
93
  "direct_join_to_source",
94
+ "downstream_models_dependent_on_source",
85
95
  "duplicate_sources",
96
+ "hard_coded_references",
86
97
  "model_fanout",
87
98
  "multiple_sources_joined",
88
99
  "rejoining_of_upstream_concepts",
@@ -92,12 +103,14 @@ __all__ = [
92
103
  "staging_dependent_on_marts_or_intermediate",
93
104
  "unused_sources",
94
105
  "too_many_joins",
106
+ "documentation_coverage",
95
107
  "undocumented_models",
96
108
  "undocumented_sources",
97
109
  "undocumented_source_tables",
98
110
  "model_naming_conventions",
99
111
  "model_directories",
100
112
  "source_directories",
113
+ "test_directories",
101
114
  "chained_view_dependencies",
102
115
  "exposure_parents_materializations",
103
116
  "public_models_without_contracts",
@@ -14,6 +14,16 @@ DEFAULT_THRESHOLDS = {
14
14
  "model_fanout": 3,
15
15
  "too_many_joins": 7,
16
16
  "chained_view_dependencies": 4,
17
+ "documentation_coverage": 100,
18
+ }
19
+
20
+ # DPE anchors that don't equal the rule name kebab-cased. The auto-derived anchor
21
+ # (``rule.replace("_", "-")``) is the DPE heading for most rules; these few have a
22
+ # different published heading, so map the rule name to its real anchor explicitly.
23
+ _RULE_ANCHORS = {
24
+ "too_many_joins": "models-with-too-many-joins",
25
+ "staging_dependent_on_marts_or_intermediate": "staging-models-dependent-on-downstream-models",
26
+ "staging_dependent_on_staging": "staging-models-dependent-on-other-staging-models",
17
27
  }
18
28
 
19
29
  # Materializations that are *not* a physical table (for chained-view detection).
@@ -28,8 +38,13 @@ LAYER_PREFIXES = {
28
38
 
29
39
 
30
40
  def docs_url(category: str, rule: str) -> str:
31
- """The DPE docs URL for a rule under *category* (anchor = the rule name)."""
32
- return f"{_RULES_BASE}/{category}/#{rule.replace('_', '-')}"
41
+ """The DPE docs URL for a rule under *category*.
42
+
43
+ The anchor is the rule name kebab-cased, except for the few rules whose DPE
44
+ heading differs (see ``_RULE_ANCHORS``).
45
+ """
46
+ anchor = _RULE_ANCHORS.get(rule, rule.replace("_", "-"))
47
+ return f"{_RULES_BASE}/{category}/#{anchor}"
33
48
 
34
49
 
35
50
  def finding(rule: str, category: str, node: str, node_type: str, message: str) -> dict:
@@ -54,6 +54,36 @@ def undocumented_sources(graph: "ManifestGraph") -> "list[dict]":
54
54
  return out
55
55
 
56
56
 
57
+ def documentation_coverage(graph: "ManifestGraph") -> "list[dict]":
58
+ """One project-level finding when the share of documented models is below target.
59
+
60
+ Mirrors DPE's ``fct_documentation_coverage`` — an aggregate percentage rather
61
+ than a per-model flag. The ``documentation_coverage`` threshold (default 100)
62
+ is the minimum acceptable percentage; coverage at or above it is clean.
63
+
64
+ Being an aggregate, its finding's ``node`` is the pseudo-id ``"project"`` (not a
65
+ real ``unique_id``); the SPA renders an unresolvable node as plain text rather
66
+ than a node-page link, so this degrades gracefully.
67
+ """
68
+ models = graph.models
69
+ if not models:
70
+ return []
71
+ documented = sum(1 for m in models if str(getattr(m, "description", "") or "").strip())
72
+ coverage = round(documented * 100 / len(models), 1)
73
+ if coverage >= graph.threshold("documentation_coverage"):
74
+ return []
75
+ return [
76
+ finding(
77
+ "documentation_coverage",
78
+ "documentation",
79
+ "project",
80
+ "project",
81
+ f"Only {coverage}% of models are documented "
82
+ f"({documented}/{len(models)}) — below the configured target.",
83
+ )
84
+ ]
85
+
86
+
57
87
  def undocumented_source_tables(graph: "ManifestGraph") -> "list[dict]":
58
88
  """Source tables (the individual relations) with no description."""
59
89
  out = []
@@ -4,14 +4,29 @@ Each rule is ``(graph) -> list[dict]``; see ``dbdocs.extract.health.rules``
4
4
  for the shared finding shape and the registry that assembles these.
5
5
  """
6
6
 
7
+ import re
7
8
  from typing import Any
8
9
 
10
+ import sqlglot
11
+ from sqlglot import exp
12
+ from sqlglot.errors import SqlglotError
13
+ from sqlglot.optimizer.scope import build_scope
14
+
9
15
  from dbdocs.extract.health.rules.base import finding
10
16
 
11
17
  # ``ManifestGraph`` annotations are strings (forward ref) to avoid a circular
12
18
  # import (rules ← graph ← analyzer); resolved lazily by the type checker only.
13
19
  ManifestGraph = Any
14
20
 
21
+ # A ``{{ ref(...) }}`` / ``{{ source(...) }}`` call is rewritten to a uniquely
22
+ # numbered sentinel table before parsing (``__dbt_ref_0__``, ``__dbt_ref_1__``, …)
23
+ # — unique so sqlglot's scope analysis doesn't choke on a repeated alias — so any
24
+ # *other* real table left in the SQL is a hard-coded relation. Any remaining
25
+ # ``{{ ... }}`` macro is blanked so it doesn't masquerade as a table.
26
+ _REF_SENTINEL_PREFIX = "__dbt_ref_"
27
+ _REF_OR_SOURCE = re.compile(r"{{[-\s]*(?:ref|source)\s*\(.*?\)[-\s]*}}", re.IGNORECASE | re.DOTALL)
28
+ _OTHER_JINJA = re.compile(r"{[{%].*?[%}]}", re.DOTALL)
29
+
15
30
 
16
31
  def direct_join_to_source(graph: "ManifestGraph") -> "list[dict]":
17
32
  """Models that ref a model AND a source in the same query (missing staging)."""
@@ -121,6 +136,87 @@ def rejoining_of_upstream_concepts(graph: "ManifestGraph") -> "list[dict]":
121
136
  return out
122
137
 
123
138
 
139
+ def downstream_models_dependent_on_source(graph: "ManifestGraph") -> "list[dict]":
140
+ """Non-staging models that ref a source directly (only staging should touch sources)."""
141
+ out = []
142
+ for model in graph.models:
143
+ if graph.layer(model) == "staging":
144
+ continue
145
+ if any(p.startswith("source.") for p in graph.parents(model.unique_id)):
146
+ out.append(
147
+ finding(
148
+ "downstream_models_dependent_on_source",
149
+ "modeling",
150
+ model.unique_id,
151
+ "model",
152
+ "A non-staging model selects from a source — route it through a staging model.",
153
+ )
154
+ )
155
+ return out
156
+
157
+
158
+ def _number_refs(raw_code: str) -> str:
159
+ """Replace each ref()/source() jinja call with a uniquely numbered sentinel."""
160
+ out = []
161
+ last = 0
162
+ for index, match in enumerate(_REF_OR_SOURCE.finditer(raw_code)):
163
+ out.append(raw_code[last : match.start()])
164
+ out.append(f"{_REF_SENTINEL_PREFIX}{index}__")
165
+ last = match.end()
166
+ out.append(raw_code[last:])
167
+ return "".join(out)
168
+
169
+
170
+ def _hard_coded_relations(raw_code: str) -> "list[str]":
171
+ """Real table names a model's raw SQL selects from outside ref()/source().
172
+
173
+ dbt references are jinja (``{{ ref(...) }}`` / ``{{ source(...) }}``), so they
174
+ are rewritten to uniquely numbered sentinel tables before parsing (so sqlglot's
175
+ scope analysis doesn't choke on a repeated alias); any other real table its
176
+ scope analysis finds is a hard-coded relation. CTEs are excluded by walking
177
+ ``scope.selected_sources`` rather than every ``exp.Table``. Unparseable SQL
178
+ yields nothing (fail-soft — one model never sinks the pass).
179
+ """
180
+ sql = _number_refs(raw_code)
181
+ sql = _OTHER_JINJA.sub(" ", sql)
182
+ if not sql.strip():
183
+ return []
184
+ try:
185
+ # parse_one returns a node or raises for non-empty input (empty is guarded
186
+ # above); build_scope returns None for a non-query statement (DDL/SET).
187
+ root = build_scope(sqlglot.parse_one(sql))
188
+ if root is None:
189
+ return []
190
+ relations = [
191
+ source.sql()
192
+ for scope in root.traverse()
193
+ for _alias, (_node, source) in scope.selected_sources.items()
194
+ if isinstance(source, exp.Table) and not source.name.startswith(_REF_SENTINEL_PREFIX)
195
+ ]
196
+ except SqlglotError: # OptimizeError (e.g. duplicate alias) is a SqlglotError subclass.
197
+ return []
198
+ return relations
199
+
200
+
201
+ def hard_coded_references(graph: "ManifestGraph") -> "list[dict]":
202
+ """Models whose SQL selects from a literal relation instead of ref()/source()."""
203
+ out = []
204
+ for model in graph.models:
205
+ relations = _hard_coded_relations(str(getattr(model, "raw_code", "") or ""))
206
+ if relations:
207
+ shown = ", ".join(sorted(set(relations)))
208
+ out.append(
209
+ finding(
210
+ "hard_coded_references",
211
+ "modeling",
212
+ model.unique_id,
213
+ "model",
214
+ f"Hard-coded relation(s) {shown} — replace with ref()/source().",
215
+ )
216
+ )
217
+ return out
218
+
219
+
124
220
  def root_models(graph: "ManifestGraph") -> "list[dict]":
125
221
  """Models with zero parents (no ref/source) — untraceable lineage."""
126
222
  out = []
@@ -57,6 +57,24 @@ def model_directories(graph: "ManifestGraph") -> "list[dict]":
57
57
  return out
58
58
 
59
59
 
60
+ def test_directories(graph: "ManifestGraph") -> "list[dict]":
61
+ """Singular (custom SQL) tests not stored under a ``tests/`` directory."""
62
+ out = []
63
+ for test in graph.singular_tests:
64
+ path = str(getattr(test, "original_file_path", "") or getattr(test, "path", "") or "")
65
+ if "tests" not in path.split("/"):
66
+ out.append(
67
+ finding(
68
+ "test_directories",
69
+ "structure",
70
+ test.unique_id,
71
+ "test",
72
+ "Singular test is not under a 'tests/' directory.",
73
+ )
74
+ )
75
+ return out
76
+
77
+
60
78
  def source_directories(graph: "ManifestGraph") -> "list[dict]":
61
79
  """Source YAML not under a staging/ directory."""
62
80
  out = []
@@ -49,3 +49,22 @@ def missing_primary_key_tests(graph: "ManifestGraph") -> "list[dict]":
49
49
  )
50
50
  )
51
51
  return out
52
+
53
+
54
+ def missing_source_freshness(graph: "ManifestGraph") -> "list[dict]":
55
+ """Sources with no freshness check (no ``loaded_at_field`` + warn/error threshold)."""
56
+ out = []
57
+ for src in graph.sources:
58
+ if not graph.has_source_freshness(src):
59
+ name = str(getattr(src, "source_name", "") or "") or src.unique_id
60
+ table = str(getattr(src, "name", "") or "")
61
+ out.append(
62
+ finding(
63
+ "missing_source_freshness",
64
+ "testing",
65
+ src.unique_id,
66
+ "source",
67
+ f"Source '{name}.{table}' has no freshness check configured.",
68
+ )
69
+ )
70
+ return out
@@ -12,6 +12,7 @@ from typing import Any
12
12
 
13
13
  from dbdocs.core.log import logger
14
14
  from dbdocs.extract.health.rules.dimensions.documentation import (
15
+ documentation_coverage,
15
16
  undocumented_models,
16
17
  undocumented_source_tables,
17
18
  undocumented_sources,
@@ -23,7 +24,9 @@ from dbdocs.extract.health.rules.dimensions.governance import (
23
24
  )
24
25
  from dbdocs.extract.health.rules.dimensions.modeling import (
25
26
  direct_join_to_source,
27
+ downstream_models_dependent_on_source,
26
28
  duplicate_sources,
29
+ hard_coded_references,
27
30
  model_fanout,
28
31
  multiple_sources_joined,
29
32
  rejoining_of_upstream_concepts,
@@ -42,8 +45,13 @@ from dbdocs.extract.health.rules.dimensions.structure import (
42
45
  model_directories,
43
46
  model_naming_conventions,
44
47
  source_directories,
48
+ test_directories,
49
+ )
50
+ from dbdocs.extract.health.rules.dimensions.testing import (
51
+ missing_primary_key_tests,
52
+ missing_source_freshness,
53
+ test_coverage,
45
54
  )
46
- from dbdocs.extract.health.rules.dimensions.testing import missing_primary_key_tests, test_coverage
47
55
 
48
56
  # The built-in rules grouped by dimension, in DPE's published order. These are the
49
57
  # *built-ins*. ``DIMENSION_RULES`` (below) starts as a deep copy and is what
@@ -53,10 +61,13 @@ _BUILTIN_RULES: "dict[str, tuple]" = {
53
61
  "testing": (
54
62
  test_coverage,
55
63
  missing_primary_key_tests,
64
+ missing_source_freshness,
56
65
  ),
57
66
  "modeling": (
58
67
  direct_join_to_source,
68
+ downstream_models_dependent_on_source,
59
69
  duplicate_sources,
70
+ hard_coded_references,
60
71
  model_fanout,
61
72
  multiple_sources_joined,
62
73
  rejoining_of_upstream_concepts,
@@ -68,6 +79,7 @@ _BUILTIN_RULES: "dict[str, tuple]" = {
68
79
  too_many_joins,
69
80
  ),
70
81
  "documentation": (
82
+ documentation_coverage,
71
83
  undocumented_models,
72
84
  undocumented_sources,
73
85
  undocumented_source_tables,
@@ -76,6 +88,7 @@ _BUILTIN_RULES: "dict[str, tuple]" = {
76
88
  model_naming_conventions,
77
89
  model_directories,
78
90
  source_directories,
91
+ test_directories,
79
92
  ),
80
93
  "performance": (
81
94
  chained_view_dependencies,
@@ -84,7 +84,26 @@ code, pre { font-family: "SF Mono", "Roboto Mono", Menlo, Consolas, monospace; }
84
84
  display: block; padding: 8px 12px; color: var(--text); border-bottom: 1px solid var(--border);
85
85
  }
86
86
  .search-results a:hover, .search-results a.active { background: var(--accent-soft); text-decoration: none; }
87
+ .search-results .sr-empty { padding: 8px 12px; color: var(--text-soft); font-size: 13px; }
88
+ .search-results .sr-title { display: block; }
87
89
  .search-results .sr-meta { color: var(--text-soft); font-size: 12px; }
90
+ .search-results .sr-snippet {
91
+ display: flex; align-items: baseline; gap: 6px; margin-top: 3px;
92
+ font-size: 12px; color: var(--text-soft); min-width: 0;
93
+ }
94
+ .search-results .sr-snippet-field {
95
+ flex-shrink: 0; font-size: 10px; text-transform: uppercase; letter-spacing: 0.03em;
96
+ color: var(--text-soft); border: 1px solid var(--border); border-radius: 4px; padding: 0 4px;
97
+ }
98
+ .search-results .sr-snippet-text {
99
+ overflow: hidden; text-overflow: ellipsis; white-space: nowrap; font-family: monospace;
100
+ }
101
+ .search-results .sr-snippet-text mark { background: var(--accent-soft); color: var(--text); border-radius: 2px; padding: 0 1px; }
102
+ /* Off-screen but readable by assistive tech (the #search-status live region). */
103
+ .visually-hidden {
104
+ position: absolute; width: 1px; height: 1px; margin: -1px; padding: 0;
105
+ overflow: hidden; clip: rect(0 0 0 0); clip-path: inset(50%); border: 0; white-space: nowrap;
106
+ }
88
107
  .topbar-actions { display: flex; align-items: center; gap: 10px; margin-left: auto; }
89
108
  .icon-btn {
90
109
  width: 34px; height: 34px; display: inline-flex; align-items: center; justify-content: center;
@@ -6,8 +6,11 @@
6
6
 
7
7
  var DATA = { metadata: {}, nodes: {}, lineage: {}, columnLineage: {}, erd: {}, tree: { byDatabase: {} }, readme: "", health: { enabled: false } };
8
8
  var DOWNSTREAM = null;
9
+ // Cache of the per-node indexed search text, keyed by id, populated by
10
+ // searchDocs() and read by searchSnippet() (both pure, DOM-free). Reset on init().
11
+ var SEARCH_TEXT = null;
9
12
 
10
- export function init(data) { DATA = data; DOWNSTREAM = null; }
13
+ export function init(data) { DATA = data; DOWNSTREAM = null; SEARCH_TEXT = null; }
11
14
 
12
15
  export function meta() { return DATA.metadata; }
13
16
  export function nodes() { return DATA.nodes; }
@@ -18,6 +21,171 @@ export function counts() { return DATA.metadata.counts || {}; }
18
21
 
19
22
  export function shortName(id) { return String(id).split(".").pop(); }
20
23
 
24
+ /* Flatten possibly-HTML-bearing text to plain words for the search index: drop
25
+ tags, decode the entities mdInline emits, collapse whitespace. A node's own
26
+ `description` is raw markdown (rendered through mdInline at display time), while
27
+ a column's `description` is pre-escaped HTML carrying <br>; this handles both. */
28
+ function stripHtml(text) {
29
+ return String(text == null ? "" : text)
30
+ .replace(/<[^>]*>/g, " ")
31
+ .replace(/&amp;/g, "&").replace(/&lt;/g, "<").replace(/&gt;/g, ">").replace(/&quot;/g, '"')
32
+ .replace(/\s+/g, " ")
33
+ .trim();
34
+ }
35
+
36
+ /* Above this node count we drop the SQL body (raw + compiled) from the search
37
+ index. Indexing every model's SQL doubles that corpus into MiniSearch's
38
+ in-memory inverted index; on a multi-thousand-model project that's tens of MB
39
+ resident for a marginal-value field. Name/column/tag/macro coverage stays —
40
+ only the full-SQL haystack is traded away at scale. */
41
+ var SQL_INDEX_NODE_CAP = 2000;
42
+
43
+ /* The indexed fields, ordered most→least human-relevant. The snippet builder
44
+ walks this order to pick which matched field to excerpt, and ui mirrors it for
45
+ the MiniSearch `fields` list. `label` is the result title (never excerpted as a
46
+ snippet); the rest are the searchable surface: the human-facing text (name,
47
+ description, tags) plus the structural surface a reader searches by — column
48
+ names + descriptions, the warehouse relation, the package, the macros a model
49
+ calls, and the model SQL (raw + compiled). Each excerptable entry carries a
50
+ display label (what ui shows above a snippet, mkdocs-material style) so the
51
+ dropdown can say *why* a result matched; label/name are the title itself and
52
+ are never excerpted, so their display label is null. */
53
+ export var SEARCH_FIELDS = [
54
+ { key: "label", label: null },
55
+ { key: "name", label: null },
56
+ { key: "columns", label: "Column" },
57
+ { key: "columnDescriptions", label: "Column docs" },
58
+ { key: "tags", label: "Tag" },
59
+ { key: "description", label: "Description" },
60
+ { key: "relation", label: "Relation" },
61
+ { key: "package", label: "Package" },
62
+ { key: "macros", label: "Macro" },
63
+ { key: "code", label: "SQL" },
64
+ ];
65
+
66
+ /* The documents fed to the full-text index (one per node). DOM-free — ui builds
67
+ the MiniSearch instance from these. Indexes the fields in SEARCH_FIELDS; the
68
+ SQL body (`code`) is dropped on large projects (see SQL_INDEX_NODE_CAP) to keep
69
+ the index small. storeFields (label/resource_type/schema) are what the dropdown
70
+ renders. The raw per-field text is cached in SEARCH_TEXT so searchSnippet() can
71
+ excerpt the matched field without a second pass. */
72
+ export function searchDocs() {
73
+ var N = DATA.nodes;
74
+ var ids = Object.keys(N);
75
+ var indexCode = ids.length <= SQL_INDEX_NODE_CAP;
76
+ SEARCH_TEXT = {};
77
+ return ids.map(function (id) {
78
+ var n = N[id];
79
+ var cols = n.columns || [];
80
+ var doc = {
81
+ id: id,
82
+ label: n.label,
83
+ name: n.name,
84
+ resource_type: n.resource_type,
85
+ schema: n.schema,
86
+ description: stripHtml(n.description),
87
+ columns: cols.map(function (c) { return c.name; }).join(" "),
88
+ columnDescriptions: cols.map(function (c) { return stripHtml(c.description); }).join(" "),
89
+ tags: (n.tags || []).join(" "),
90
+ relation: n.relation_name || "",
91
+ package: n.package || "",
92
+ macros: (n.macros || []).map(function (m) { return m.name; }).join(" "),
93
+ code: indexCode ? stripHtml((n.raw_code || "") + " " + (n.compiled_code || "")) : "",
94
+ };
95
+ SEARCH_TEXT[id] = doc;
96
+ return doc;
97
+ });
98
+ }
99
+
100
+ /* The MiniSearch `match` object maps each matched document-term to the list of
101
+ fields it hit. Collapse that to the single most-relevant matched field by
102
+ walking SEARCH_FIELDS order (skipping `label`/`name` — the title already shows
103
+ that). Returns the SEARCH_FIELDS entry ({ key, label }), or null when only
104
+ `label`/`name` matched and the title already explains the hit. */
105
+ function topMatchedField(match) {
106
+ var hitFields = {};
107
+ Object.keys(match || {}).forEach(function (term) {
108
+ (match[term] || []).forEach(function (f) { hitFields[f] = true; });
109
+ });
110
+ for (var i = 0; i < SEARCH_FIELDS.length; i++) {
111
+ var f = SEARCH_FIELDS[i];
112
+ if (f.key === "label" || f.key === "name") continue;
113
+ if (hitFields[f.key]) return f;
114
+ }
115
+ return null;
116
+ }
117
+
118
+ /* A windowed excerpt of `text` centered on the first occurrence of any matched
119
+ term, with ~`pad` chars of context either side and ellipses where clipped.
120
+ Case-insensitive find; returns the head of the text when no term is located
121
+ (e.g. a fuzzy/prefix hit whose surface term differs). Pure string work. */
122
+ function excerpt(text, terms, pad) {
123
+ var hay = String(text || "");
124
+ if (!hay) return "";
125
+ var lower = hay.toLowerCase();
126
+ var at = -1;
127
+ for (var i = 0; i < terms.length; i++) {
128
+ var p = lower.indexOf(String(terms[i]).toLowerCase());
129
+ if (p !== -1 && (at === -1 || p < at)) at = p;
130
+ }
131
+ if (at === -1) return hay.length > pad * 2 ? hay.slice(0, pad * 2) + "…" : hay;
132
+ var start = Math.max(0, at - pad);
133
+ var end = Math.min(hay.length, at + pad);
134
+ return (start > 0 ? "…" : "") + hay.slice(start, end).trim() + (end < hay.length ? "…" : "");
135
+ }
136
+
137
+ /* mkdocs-material-style match context for a search hit: which field matched and
138
+ a short excerpt of it around the query term(s), so the dropdown explains *why*
139
+ a result is there (e.g. searching "unique" surfaces the SQL line it lives on).
140
+ `terms` are the matched document terms (hit.terms from MiniSearch). Returns
141
+ { field, text } or null when only the title matched. DOM-free — ui highlights
142
+ the terms and renders. */
143
+ export function searchSnippet(id, match, terms) {
144
+ var field = topMatchedField(match);
145
+ if (!field) return null;
146
+ var doc = (SEARCH_TEXT && SEARCH_TEXT[id]) || {};
147
+ var text = excerpt(doc[field.key], terms || [], 40);
148
+ if (!text) return null;
149
+ return { field: field.label, text: text };
150
+ }
151
+
152
+ /* Inline search operators (mkdocs-material style) the user types into the box:
153
+ type:<resource_type> restrict hits to a resource_type (model/source/…)
154
+ label:<text> / name:<text> match only against the name/label fields,
155
+ skipping the SQL/description/column noise
156
+ Operators combine with the free-text remainder: `type:model orders` finds
157
+ models whose text matches "orders"; `label:stg` matches names containing
158
+ "stg". A bare query (no operators) searches everything. */
159
+ var SEARCH_OPERATOR = /(\w+):(\S+)/g;
160
+ var NAME_FIELDS = ["label", "name"];
161
+
162
+ /* Parse a raw query into MiniSearch inputs. Returns:
163
+ { text } the free-text remainder to search (operators stripped)
164
+ { fields } restrict matching to these fields, or null for all
165
+ { filterType } restrict hits to this resource_type, or null
166
+ { operators } the recognized operators (for ui to echo as chips)
167
+ DOM-free — ui owns running mini.search() and rendering. */
168
+ export function parseSearchQuery(raw) {
169
+ var fields = null;
170
+ var filterType = null;
171
+ var operators = [];
172
+ var text = String(raw || "").replace(SEARCH_OPERATOR, function (m, key, val) {
173
+ var k = key.toLowerCase();
174
+ if (k === "type" || k === "resource_type") {
175
+ filterType = val.toLowerCase();
176
+ operators.push({ key: "type", value: filterType });
177
+ return " ";
178
+ }
179
+ if (k === "label" || k === "name") {
180
+ fields = NAME_FIELDS;
181
+ operators.push({ key: "label", value: val });
182
+ return " " + val + " "; // the operator's value is still the text to match
183
+ }
184
+ return m; // unrecognized prefix (e.g. a "schema:foo" the index doesn't carry) — leave it as a literal term
185
+ }).replace(/\s+/g, " ").trim();
186
+ return { text: text, fields: fields, filterType: filterType, operators: operators };
187
+ }
188
+
21
189
  /* The searchable text for a node in the sidebar tree filter: its table name and
22
190
  its fully-qualified database.schema.table, lowercased. Lets the tree filter
23
191
  match either "orders" or "shaman.jf.orders". DOM-free (ui does the show/hide). */
@@ -651,37 +651,157 @@ function toggleFullscreen(host) {
651
651
  }
652
652
  }
653
653
 
654
+ var SEARCH_RESULT_CAP = 12;
655
+
654
656
  function buildSearch() {
655
657
  var input = document.getElementById("search");
656
658
  var results = document.getElementById("search-results");
659
+ var status = document.getElementById("search-status");
657
660
  if (typeof MiniSearch === "undefined") return;
658
- var NODES = svc.nodes();
659
- var docs = Object.keys(NODES).map(function (id) {
660
- var n = NODES[id];
661
- return { id: id, label: n.label, resource_type: n.resource_type, schema: n.schema,
662
- description: n.description, columns: (n.columns || []).map(function (c) { return c.name; }).join(" ") };
661
+ var docs = svc.searchDocs();
662
+ var mini = new MiniSearch({
663
+ fields: svc.SEARCH_FIELDS.map(function (f) { return f.key; }),
664
+ storeFields: ["label", "resource_type", "schema"],
665
+ // Names/columns/tags expand fuzzily + by prefix so a half-typed model name
666
+ // still hits. The bulk text fields (description, column docs, SQL) match only
667
+ // on whole terms and at a low weight — otherwise a generic SQL keyword like
668
+ // "unique" fuzzy-floods nearly every model. boost ranks name/column hits well
669
+ // above an incidental SQL-body hit so the title results stay on top.
670
+ searchOptions: {
671
+ prefix: true,
672
+ fuzzy: 0.2,
673
+ boost: { label: 6, name: 6, columns: 3, tags: 2, relation: 2, description: 1, columnDescriptions: 1, code: 0.4 },
674
+ },
663
675
  });
664
- var mini = new MiniSearch({ fields: ["label", "description", "columns"], storeFields: ["label", "resource_type", "schema"], searchOptions: { prefix: true, fuzzy: 0.2, boost: { label: 3 } } });
665
676
  mini.addAll(docs);
666
677
 
678
+ // Build the matched-field snippet (mkdocs-material style): a label for the
679
+ // field that matched + the excerpt, with the matched terms wrapped in <mark>.
680
+ // Highlighting is done by splitting on the matched terms and building text +
681
+ // <mark> nodes (no innerHTML) so user-derived text can never inject markup.
682
+ function highlightNodes(text, terms) {
683
+ var safe = (terms || []).map(function (t) { return String(t).replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); }).filter(Boolean);
684
+ if (!safe.length) return [text];
685
+ var re = new RegExp("(" + safe.join("|") + ")", "ig");
686
+ return String(text).split(re).map(function (piece, i) {
687
+ // split() with a capturing group puts the matched delimiters at odd indices.
688
+ return i % 2 === 1 ? el("mark", null, [piece]) : piece;
689
+ });
690
+ }
691
+
692
+ function snippetNode(h) {
693
+ var snip = svc.searchSnippet(h.id, h.match, h.terms);
694
+ if (!snip) return null;
695
+ return el("span", { class: "sr-snippet" }, [
696
+ el("span", { class: "sr-snippet-field" }, [snip.field]),
697
+ el("span", { class: "sr-snippet-text" }, highlightNodes(snip.text, h.terms)),
698
+ ]);
699
+ }
700
+
701
+ // Roving active-descendant state for keyboard nav: index into the rendered
702
+ // option rows, or -1 for "input itself focused, nothing selected".
703
+ var activeIndex = -1;
704
+
705
+ function setExpanded(open) {
706
+ results.hidden = !open;
707
+ input.setAttribute("aria-expanded", open ? "true" : "false");
708
+ if (!open) {
709
+ activeIndex = -1;
710
+ input.removeAttribute("aria-activedescendant");
711
+ status.textContent = "";
712
+ }
713
+ }
714
+
715
+ function optionRows() { return results.querySelectorAll('[role="option"]'); }
716
+
717
+ // Move the roving selection by `delta` (wrapping), reflect it as the .active
718
+ // class + aria-selected, and point the input's aria-activedescendant at it so
719
+ // a screen reader announces the row without moving DOM focus off the input.
720
+ function moveActive(delta) {
721
+ var rows = optionRows();
722
+ if (!rows.length) return;
723
+ if (activeIndex >= 0 && rows[activeIndex]) rows[activeIndex].setAttribute("aria-selected", "false");
724
+ activeIndex = (activeIndex + delta + rows.length) % rows.length;
725
+ rows.forEach(function (r, i) { r.classList.toggle("active", i === activeIndex); });
726
+ var active = rows[activeIndex];
727
+ active.setAttribute("aria-selected", "true");
728
+ input.setAttribute("aria-activedescendant", active.id);
729
+ active.scrollIntoView({ block: "nearest" });
730
+ }
731
+
667
732
  function render(hits) {
668
733
  clear(results);
669
- if (!hits.length) { results.hidden = true; return; }
670
- hits.slice(0, 12).forEach(function (h) {
671
- results.appendChild(el("a", { href: "#/node/" + encodeURIComponent(h.id), onclick: function () { results.hidden = true; input.value = ""; setSearchOpen(false); } }, [
672
- el("span", { class: "dot " + h.resource_type }), " " + h.label,
673
- el("span", { class: "sr-meta" }, [" " + h.resource_type + " · " + h.schema]),
674
- ]));
734
+ activeIndex = -1;
735
+ input.removeAttribute("aria-activedescendant");
736
+ if (!hits.length) {
737
+ // A non-empty query with no hits gets an explicit cue, mirrored into the
738
+ // #search-status live region for screen readers.
739
+ var noMatches = "No matches.";
740
+ results.appendChild(el("div", { class: "sr-empty" }, [noMatches]));
741
+ status.textContent = noMatches;
742
+ setExpanded(true);
743
+ return;
744
+ }
745
+ var shown = Math.min(hits.length, SEARCH_RESULT_CAP);
746
+ status.textContent = shown === 1 ? "1 result." : shown + " results.";
747
+ hits.slice(0, SEARCH_RESULT_CAP).forEach(function (h, i) {
748
+ var children = [
749
+ el("span", { class: "sr-title" }, [
750
+ el("span", { class: "dot " + h.resource_type }), " " + h.label,
751
+ el("span", { class: "sr-meta" }, [" " + h.resource_type + " · " + h.schema]),
752
+ ]),
753
+ ];
754
+ var snip = snippetNode(h);
755
+ if (snip) children.push(snip);
756
+ results.appendChild(el("a", {
757
+ id: "search-opt-" + i,
758
+ role: "option",
759
+ "aria-selected": "false",
760
+ href: "#/node/" + encodeURIComponent(h.id),
761
+ onclick: function () { closeSearch(); },
762
+ }, children));
675
763
  });
676
- results.hidden = false;
764
+ setExpanded(true);
765
+ }
766
+
767
+ function closeSearch() { setExpanded(false); input.value = ""; setSearchOpen(false); }
768
+ // Map service's parsed operators to MiniSearch's field restriction + a
769
+ // resource_type filter. A bare `type:` (no free text) scans for the first
770
+ // SEARCH_RESULT_CAP nodes of that type — O(cap), not O(N) on a huge project.
771
+ function runQuery(raw) {
772
+ var p = svc.parseSearchQuery(raw);
773
+ if (!p.text) {
774
+ if (!p.filterType) return [];
775
+ var picked = [];
776
+ for (var i = 0; i < docs.length && picked.length < SEARCH_RESULT_CAP; i++) {
777
+ if (docs[i].resource_type === p.filterType) picked.push(docs[i]);
778
+ }
779
+ return picked;
780
+ }
781
+ var opts = {};
782
+ if (p.fields) opts.fields = p.fields;
783
+ if (p.filterType) opts.filter = function (h) { return h.resource_type === p.filterType; };
784
+ return mini.search(p.text, opts);
677
785
  }
678
786
  input.addEventListener("input", function () {
679
787
  var q = input.value.trim();
680
- if (!q) { results.hidden = true; return; }
681
- render(mini.search(q));
788
+ if (!q) { setExpanded(false); return; }
789
+ render(runQuery(q));
790
+ });
791
+ input.addEventListener("focus", function () { if (input.value.trim()) render(runQuery(input.value.trim())); });
792
+ // Keyboard nav: ↑/↓ rove the options, Enter follows the active one (or the
793
+ // first when none is roved), Escape closes. Other keys fall through to typing.
794
+ input.addEventListener("keydown", function (e) {
795
+ if (results.hidden) return;
796
+ if (e.key === "ArrowDown") { e.preventDefault(); moveActive(1); }
797
+ else if (e.key === "ArrowUp") { e.preventDefault(); moveActive(-1); }
798
+ else if (e.key === "Enter") {
799
+ var rows = optionRows();
800
+ var target = rows[activeIndex >= 0 ? activeIndex : 0];
801
+ if (target) { e.preventDefault(); location.hash = target.getAttribute("href"); closeSearch(); }
802
+ } else if (e.key === "Escape") { setExpanded(false); }
682
803
  });
683
- input.addEventListener("focus", function () { if (input.value.trim()) render(mini.search(input.value.trim())); });
684
- document.addEventListener("click", function (e) { if (!results.contains(e.target) && e.target !== input) results.hidden = true; });
804
+ document.addEventListener("click", function (e) { if (!results.contains(e.target) && e.target !== input) setExpanded(false); });
685
805
  }
686
806
 
687
807
  function initTheme() {
@@ -21,8 +21,9 @@
21
21
  <span class="brand-generated" id="brand-generated" hidden></span>
22
22
  </div>
23
23
  <div class="search-wrap">
24
- <input id="search" type="search" placeholder="Search models, sources, columns…" autocomplete="off" />
25
- <div id="search-results" class="search-results" hidden></div>
24
+ <input id="search" type="search" role="combobox" aria-label="Search" aria-expanded="false" aria-controls="search-results" aria-autocomplete="list" autocomplete="off" placeholder="Search models, columns, tags…" />
25
+ <div id="search-results" class="search-results" role="listbox" aria-label="Search results" hidden></div>
26
+ <div id="search-status" class="visually-hidden" role="status" aria-live="polite"></div>
26
27
  </div>
27
28
  <div class="topbar-actions">
28
29
  <button id="search-toggle" class="icon-btn search-toggle" aria-label="Toggle search" aria-expanded="false">
dbdocs-1.2.0/README.md DELETED
@@ -1,69 +0,0 @@
1
- <p align="center">
2
- <img src="docs/assets/logo.svg" alt="dbdocs logo" width="220" height="88">
3
- </p>
4
-
5
- <p align="center"><b>An alternative dbt docs site — catalog + ERD + column-level lineage + versioned deploys, all in one CLI.</b></p>
6
-
7
- <p align="center">
8
- <a href="https://dbdocs.datnguye.me/latest/demo/latest/"><img src="https://img.shields.io/badge/live-demo-FF694A?style=flat&logo=rocket&logoColor=white" alt="live demo"></a>
9
- <a href="https://dbdocs.datnguye.me/"><img src="https://img.shields.io/badge/docs-visit%20site-blue?style=flat&logo=gitbook&logoColor=white" alt="docs"></a>
10
- <a href="https://pypi.org/project/dbdocs/"><img src="https://badge.fury.io/py/dbdocs.svg" alt="PyPI version"></a>
11
- <img src="https://img.shields.io/badge/CLI-Python-FFCE3E?labelColor=14354C&logo=python&logoColor=white" alt="python-cli">
12
- <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
13
- <a href="https://www.python.org"><img src="https://img.shields.io/badge/Python-3.10|3.11|3.12|3.13-3776AB.svg?style=flat&logo=python&logoColor=white" alt="python"></a>
14
- </p>
15
-
16
- Turn your dbt artifacts into a self-contained docs site: a browsable catalog, an entity-relationship diagram, an interactive lineage DAG, and **column-level lineage** traced from your compiled SQL — all in one `dbdocs generate`. Serve it with `dbdocs serve`, or deploy versioned builds anywhere a static host will take them.
17
-
18
- | Catalog | Model page | Lineage DAG | Health Check |
19
- |---|---|---|---|
20
- | ![catalog](docs/assets/img/demo-catalog.png) | ![model page](docs/assets/img/demo-model-page.png) | ![dag](docs/assets/img/demo-model-dag.png) | ![health check](docs/assets/img/demo-health.png) |
21
-
22
- ## Why dbdocs?
23
-
24
- dbt's built-in docs are great — right up until you want to know *which upstream column fed this downstream column*, or *which tables relate to each other*, or *what changed between last week's docs and today's*. dbdocs fills all three gaps without asking you to install a documentation framework or maintain a separate ERD tool.
25
-
26
- **What you get that nothing else bundles together:**
27
-
28
- - **ERD + column-level lineage, side by side.** The entity-relationship diagram (powered by [dbterd](https://github.com/datnguye/dbterd)) shows table relationships; column lineage (traced by [sqlglot](https://github.com/tobymao/sqlglot) from compiled SQL) shows exactly which column fed which. Most alternatives give you one or the other — dbdocs gives you both.
29
- - **Column impact analysis.** Select any column and see its downstream dependents across the project, so you know what a schema change will break before you run it.
30
- - **Deep-link URLs.** Every focused node, column, and filtered DAG view has a shareable URL. Paste it in Slack and your teammate lands on exactly the right model, column, or graph state.
31
- - **Any sqlglot-supported dialect.** The dialect for column-lineage parsing is auto-detected from your manifest's `adapter_type` (Snowflake, BigQuery, Redshift, DuckDB, PostgreSQL, Databricks/Spark, Trino, and more — anything sqlglot understands). Override it per-project with `dialect:` in `dbdocs.yml` when auto-detection isn't enough.
32
- - **Scales without freezing.** Column-lineage parsing fans out across CPU cores automatically above 500 models, so large projects finish in roughly the same wall-clock time as small ones. The DAG is windowed by React Flow, so a 1 000-model graph doesn't turn your browser into a space heater. The payload ships as an external gzip (`dbdocs-data.json.gz`, decompressed client-side) so `index.html` stays tiny regardless of project size.
33
- - **Fail-soft.** One model with SQL sqlglot can't parse gets skipped and logged — it never sinks the whole generate run.
34
- - **Project Health Check.** A scorecard across the six [dbt-project-evaluator](https://dbt-labs.github.io/dbt-project-evaluator/) dimensions (testing, modeling, documentation, structure, performance, governance), computed straight from your `manifest.json` — no extra dbt package, no warehouse. When a `run_results.json` is also present (any `dbt build`/`dbt test`; default `<target_dir>/run_results.json`, override with `--run-results`), each test additionally shows up as a pass/fail finding grouped by what it checks (integrity, referential, validity, business logic, freshness). dbdocs only reads the artifacts — it never runs dbt or touches your warehouse. Fail-soft: a missing `run_results.json` just drops the per-test detail; the dimensions still render.
35
- - **Versioned deploys, no plugins.** `dbdocs deploy --version v1.2 --alias latest` generates into a plain directory tree, writes a `versions.json` index, and the SPA renders a version dropdown. No mike, no external tooling, no surprise dependencies.
36
- - **Catalog navigation + client-side search.** Models, seeds, and snapshots grouped by database and schema; full-text search without a backend.
37
- - **Dark / light theme.**
38
-
39
- ## Install
40
-
41
- ```bash
42
- pip install dbdocs --upgrade
43
- ```
44
-
45
- Requires Python 3.10+.
46
-
47
- ## Quickstart
48
-
49
- ```bash
50
- dbt docs generate # writes target/manifest.json + target/catalog.json
51
- dbdocs generate # builds ./site/ with index.html + dbdocs-data.json.gz
52
- dbdocs serve # static http server on http://127.0.0.1:8000
53
- ```
54
-
55
- The site must be served over HTTP (not opened as a local file) because it fetches the data payload at load time. `dbdocs serve` handles that locally; any static host works for deployment.
56
-
57
- Full walkthrough, configuration, and architecture live in the **[documentation](https://dbdocs.datnguye.me/)**.
58
-
59
- ## Contributing
60
-
61
- Contributions are welcome — bugs, features, docs, typos. See the **[Contributing Guide](https://dbdocs.datnguye.me/latest/nav/development/contributing-guide.html)**.
62
-
63
- If dbdocs saves you some clicks, consider [buying me a coffee](https://www.buymeacoffee.com/datnguye).
64
-
65
- <a href="https://www.buymeacoffee.com/datnguye"><img src="https://img.shields.io/badge/buy%20me%20a%20coffee-donate-yellow.svg?logo=buy-me-a-coffee&logoColor=white&labelColor=ff813f&style=for-the-badge" alt="buy me a coffee"></a>
66
-
67
- ## License
68
-
69
- [MIT](./LICENSE) © Dat Nguyen
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes