@mriekki/lean 0.4.2 → 0.4.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/assets/components.json +2 -2
- package/assets/core/distribution.json +3 -0
- package/assets/generated/ASSEMBLY.json +22 -7
- package/assets/generated/core/core/learning/ENGINEERING_LEARNING_WORKBOOK.md +4 -1
- package/assets/generated/core/core/learning/LEARNING_MAP.md +22 -0
- package/assets/generated/core/core/learning/SQL_DRILLS.md +80 -17
- package/assets/generated/core/core/process/COACHING_GUIDE.md +3 -1
- package/assets/generated/core/core/process/DATABASE_LEARNING_WORKBENCH.md +346 -0
- package/assets/generated/core/core/skills/coach/SKILL.md +12 -1
- package/assets/generated/core/core/skills/db-lab/SKILL.md +202 -0
- package/assets/generated/core/tools/db_lab/sqlite_workbench.py +463 -0
- package/package.json +1 -1
|
@@ -0,0 +1,202 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: db-lab
|
|
3
|
+
description: "Optional project-aware database learning lab using actual schema, ORM, migration, query, and query-plan evidence for SQL, relational design, debugging, and interview practice."
|
|
4
|
+
argument-hint: "[schema|trace|plan|debug|migration|interview|topic] [light|focused|deep]"
|
|
5
|
+
user-invocable: true
|
|
6
|
+
disable-model-invocation: true
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Database Lab
|
|
10
|
+
|
|
11
|
+
`/db-lab` is an optional hands-on database learning surface. It complements `/coach`; it does not replace the engineering workflow or authorize implementation changes.
|
|
12
|
+
|
|
13
|
+
For detailed usage and examples, read `docs/process/DATABASE_LEARNING_WORKBENCH.md` in a greenfield consumer (or the canonical `core/process/DATABASE_LEARNING_WORKBENCH.md`).
|
|
14
|
+
|
|
15
|
+
## 1. Boundaries
|
|
16
|
+
|
|
17
|
+
- Start from the actual repository and its documented development/test database configuration.
|
|
18
|
+
- Prefer read-only inspection. Do not mutate application data, apply migrations, add/drop indexes, or change production code while in Database Lab mode.
|
|
19
|
+
- Never inspect Production credentials or Production data.
|
|
20
|
+
- Do not invent schema, SQL, query-plan behavior, ORM translation, or index usage. Distinguish inferred behavior from captured evidence.
|
|
21
|
+
- If safe local database/runtime evidence is unavailable, fall back to repository artifacts and say what could not be verified.
|
|
22
|
+
- Generated learning artifacts are allowed only as disposable local output from read-only evidence; do not commit them unless the human explicitly asks for a durable artifact.
|
|
23
|
+
- Discovered implementation defects return to the normal engineering workflow; do not silently fix them here.
|
|
24
|
+
- Learning is optional and never blocks delivery.
|
|
25
|
+
|
|
26
|
+
## 2. Database competencies
|
|
27
|
+
|
|
28
|
+
Use the common L1-L6 learning levels across independent database areas:
|
|
29
|
+
|
|
30
|
+
```text
|
|
31
|
+
relational modeling
|
|
32
|
+
keys + integrity constraints
|
|
33
|
+
normalization + data anomalies
|
|
34
|
+
SQL query construction
|
|
35
|
+
indexes + access patterns
|
|
36
|
+
query plans + performance reasoning
|
|
37
|
+
transactions + concurrency
|
|
38
|
+
ORM translation + abstraction leaks
|
|
39
|
+
schema evolution + migrations
|
|
40
|
+
database debugging
|
|
41
|
+
design defense + interview communication
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Do not collapse these into one generic "SQL level". A developer may be strong in query syntax and weak in relational design, indexing, transactions, or ORM translation.
|
|
45
|
+
|
|
46
|
+
## 3. Schema mode
|
|
47
|
+
|
|
48
|
+
Build an evidence-backed view from the best available sources:
|
|
49
|
+
|
|
50
|
+
```text
|
|
51
|
+
physical database metadata
|
|
52
|
+
ORM model/configuration
|
|
53
|
+
migration history
|
|
54
|
+
schema SQL / DDL
|
|
55
|
+
project architecture and requirements
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Show tables/entities, columns, types, nullability, primary keys, foreign keys, unique/check constraints, indexes, relationship cardinality, cascade behavior, and relevant ORM mappings when evidence exists.
|
|
59
|
+
|
|
60
|
+
When useful, distinguish:
|
|
61
|
+
|
|
62
|
+
```text
|
|
63
|
+
DOMAIN MODEL — what the application/domain expresses
|
|
64
|
+
ORM MODEL — what the ORM maps and tracks
|
|
65
|
+
PHYSICAL MODEL — what the relational database actually stores/enforces
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
## 4. Query trace mode
|
|
69
|
+
|
|
70
|
+
Trace a real application query without guessing:
|
|
71
|
+
|
|
72
|
+
```text
|
|
73
|
+
application/ORM expression
|
|
74
|
+
→ ORM translation preview when available
|
|
75
|
+
→ actual executed SQL when captured
|
|
76
|
+
→ parameters
|
|
77
|
+
→ tables/relationships touched
|
|
78
|
+
→ query plan when available
|
|
79
|
+
→ indexes actually chosen by the planner
|
|
80
|
+
→ materialized/application result
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
Keep translation preview and actual executed command distinct when the framework exposes both.
|
|
84
|
+
|
|
85
|
+
## 5. Query-to-ERD highlighting
|
|
86
|
+
|
|
87
|
+
When the distributed SQLite workbench or another compatible viewer is available, a selected query should drive a linked ERD view:
|
|
88
|
+
|
|
89
|
+
- highlight tables referenced by the SQL;
|
|
90
|
+
- highlight FK edges when both related tables participate in the selected query;
|
|
91
|
+
- highlight qualified query columns when they can be resolved to a table;
|
|
92
|
+
- distinguish candidate indexes from indexes actually selected by the query planner;
|
|
93
|
+
- show query-plan evidence alongside the diagram;
|
|
94
|
+
- allow table/column/index selection for deeper explanation.
|
|
95
|
+
|
|
96
|
+
Never claim the planner used an index merely because it exists or looks relevant.
|
|
97
|
+
|
|
98
|
+
If no interactive viewer is available, provide a temporary Mermaid/text diagram using the same evidence and clearly label it as a fallback rather than a live ERD.
|
|
99
|
+
|
|
100
|
+
## 6. SQLite workbench
|
|
101
|
+
|
|
102
|
+
When `tools/lean-db-lab/sqlite-workbench.py` exists and a safe local development/test SQLite database is known, prefer it for physical-schema and query-plan evidence.
|
|
103
|
+
|
|
104
|
+
The tool:
|
|
105
|
+
|
|
106
|
+
- opens the SQLite database in `mode=ro`;
|
|
107
|
+
- introspects tables/views, columns, primary keys, foreign keys, indexes, and DDL;
|
|
108
|
+
- accepts optional captured-query JSON containing `name`, `sql`, and optional `parameters`;
|
|
109
|
+
- uses `EXPLAIN QUERY PLAN` only for `SELECT`/`WITH` queries;
|
|
110
|
+
- generates self-contained `index.html` plus `workbench.json` learning artifacts;
|
|
111
|
+
- supports interactive table/index inspection and query-to-ERD highlighting.
|
|
112
|
+
|
|
113
|
+
Generate output in an explicitly selected disposable/temp location. Do not write it into tracked project paths by default.
|
|
114
|
+
|
|
115
|
+
The workbench analyzes captured SQL; it does not itself capture Entity Framework commands. Obtain translation/execution evidence through supported framework mechanisms first, then pass the captured SQL into the workbench when useful.
|
|
116
|
+
|
|
117
|
+
## 7. Query-plan/performance mode
|
|
118
|
+
|
|
119
|
+
Use provider-native plan evidence where safely available. Ask the developer to predict likely scans/searches/joins/index behavior before revealing the plan when that improves learning.
|
|
120
|
+
|
|
121
|
+
```text
|
|
122
|
+
query/access pattern
|
|
123
|
+
→ developer prediction
|
|
124
|
+
→ plan evidence
|
|
125
|
+
→ explain scan/search/join behavior
|
|
126
|
+
→ developer proposes index/query change
|
|
127
|
+
→ predict changed plan
|
|
128
|
+
→ compare evidence in a safe sandbox when available
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
Do not teach "add an index" without connecting it to a concrete access pattern, selectivity/order needs, write cost, and plan evidence.
|
|
132
|
+
|
|
133
|
+
## 8. Debug mode
|
|
134
|
+
|
|
135
|
+
```text
|
|
136
|
+
symptom
|
|
137
|
+
→ database hypotheses
|
|
138
|
+
→ distinguishing evidence
|
|
139
|
+
→ SQL / parameters
|
|
140
|
+
→ plan / constraints / transaction state
|
|
141
|
+
→ root cause
|
|
142
|
+
→ fix concept
|
|
143
|
+
→ prevention
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
The human should diagnose before the lab reveals the answer unless they explicitly request explanation-first coaching.
|
|
147
|
+
|
|
148
|
+
## 9. Migration mode
|
|
149
|
+
|
|
150
|
+
Connect:
|
|
151
|
+
|
|
152
|
+
```text
|
|
153
|
+
ORM migration/change
|
|
154
|
+
↔ generated DDL/SQL when available
|
|
155
|
+
↔ schema before
|
|
156
|
+
↔ schema after
|
|
157
|
+
↔ data-safety implications
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
Ask the developer to predict effects before showing generated SQL when useful. Discuss destructive changes, nullability transitions, backfills, constraint creation, and rollback limitations without applying migrations in learning mode.
|
|
161
|
+
|
|
162
|
+
## 10. Interview/design mode
|
|
163
|
+
|
|
164
|
+
Use the active project domain as context, but perturb requirements so the developer must reason rather than reproduce the existing schema.
|
|
165
|
+
|
|
166
|
+
Drive one material question at a time through:
|
|
167
|
+
|
|
168
|
+
```text
|
|
169
|
+
requirements clarification
|
|
170
|
+
→ entities
|
|
171
|
+
→ relationships/cardinality
|
|
172
|
+
→ keys
|
|
173
|
+
→ constraints
|
|
174
|
+
→ normalization
|
|
175
|
+
→ access patterns
|
|
176
|
+
→ indexes
|
|
177
|
+
→ transactions/concurrency
|
|
178
|
+
→ evolution/scaling
|
|
179
|
+
→ design defense
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
Challenge unsupported choices. Ask why a junction table exists, why a key was selected, what deletion means historically, which queries justify an index, and how changed requirements alter the design.
|
|
183
|
+
|
|
184
|
+
## 11. ORM-specific evidence
|
|
185
|
+
|
|
186
|
+
When Entity Framework Core is present, distinguish:
|
|
187
|
+
|
|
188
|
+
- LINQ/application expression;
|
|
189
|
+
- translation preview such as `ToQueryString()` when appropriate;
|
|
190
|
+
- actual executed `DbCommand` from supported logging/interception evidence;
|
|
191
|
+
- generated migration SQL;
|
|
192
|
+
- entity/change-tracking state versus physical SQL operations.
|
|
193
|
+
|
|
194
|
+
Do not present a translation preview as proof of the exact command that executed.
|
|
195
|
+
|
|
196
|
+
For other ORMs, apply the same principle using framework-supported evidence instead of guessing.
|
|
197
|
+
|
|
198
|
+
## 12. Tooling
|
|
199
|
+
|
|
200
|
+
Optional IDE/database extensions may improve exploration but are not workflow authority and must not become a required dependency unless the project explicitly owns that requirement.
|
|
201
|
+
|
|
202
|
+
Prefer provider/framework-native inspection plus repository evidence. The SQLite workbench is a reusable read-only learning aid; future provider/ORM adapters should preserve the same evidence and safety contract.
|
|
@@ -0,0 +1,463 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
from __future__ import annotations
|
|
3
|
+
|
|
4
|
+
import argparse
|
|
5
|
+
import html
|
|
6
|
+
import json
|
|
7
|
+
import re
|
|
8
|
+
import sqlite3
|
|
9
|
+
from pathlib import Path
|
|
10
|
+
from typing import Any
|
|
11
|
+
from urllib.parse import quote
|
|
12
|
+
|
|
13
|
+
|
|
14
|
+
PLAN_INDEX_RE = re.compile(r"\bUSING(?:\s+COVERING)?\s+INDEX\s+([^\s()]+)", re.IGNORECASE)
|
|
15
|
+
PLAN_TABLE_RE = re.compile(r"\b(?:SCAN|SEARCH)\s+([^\s()]+)", re.IGNORECASE)
|
|
16
|
+
TABLE_REF_RE = re.compile(
|
|
17
|
+
r"\b(?:FROM|JOIN|UPDATE|INTO)\s+([\[\]`\"A-Za-z0-9_.]+)(?:\s+(?:AS\s+)?([A-Za-z_][A-Za-z0-9_]*))?",
|
|
18
|
+
re.IGNORECASE,
|
|
19
|
+
)
|
|
20
|
+
QUALIFIED_COLUMN_RE = re.compile(r"\b([A-Za-z_][A-Za-z0-9_]*)\.([A-Za-z_][A-Za-z0-9_]*)\b")
|
|
21
|
+
SQL_KEYWORDS = {
|
|
22
|
+
"where",
|
|
23
|
+
"join",
|
|
24
|
+
"left",
|
|
25
|
+
"right",
|
|
26
|
+
"inner",
|
|
27
|
+
"outer",
|
|
28
|
+
"cross",
|
|
29
|
+
"on",
|
|
30
|
+
"order",
|
|
31
|
+
"group",
|
|
32
|
+
"having",
|
|
33
|
+
"limit",
|
|
34
|
+
"offset",
|
|
35
|
+
"union",
|
|
36
|
+
"intersect",
|
|
37
|
+
"except",
|
|
38
|
+
"returning",
|
|
39
|
+
"set",
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
def _literal(value: str) -> str:
|
|
44
|
+
return "'" + value.replace("'", "''") + "'"
|
|
45
|
+
|
|
46
|
+
|
|
47
|
+
def _clean_identifier(value: str) -> str:
|
|
48
|
+
value = value.strip().strip("[]`\"")
|
|
49
|
+
return value.split(".")[-1]
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
def _open_read_only(db_path: Path) -> sqlite3.Connection:
|
|
53
|
+
resolved = db_path.expanduser().resolve()
|
|
54
|
+
if not resolved.is_file():
|
|
55
|
+
raise FileNotFoundError(f"SQLite database does not exist: {resolved}")
|
|
56
|
+
uri_path = quote(resolved.as_posix(), safe="/:")
|
|
57
|
+
conn = sqlite3.connect(f"file:{uri_path}?mode=ro", uri=True)
|
|
58
|
+
conn.row_factory = sqlite3.Row
|
|
59
|
+
conn.execute("PRAGMA foreign_keys = ON")
|
|
60
|
+
return conn
|
|
61
|
+
|
|
62
|
+
|
|
63
|
+
def _columns(conn: sqlite3.Connection, name: str) -> list[dict[str, Any]]:
|
|
64
|
+
rows = conn.execute(f"PRAGMA table_xinfo({_literal(name)})").fetchall()
|
|
65
|
+
return [
|
|
66
|
+
{
|
|
67
|
+
"cid": row["cid"],
|
|
68
|
+
"name": row["name"],
|
|
69
|
+
"type": row["type"] or "",
|
|
70
|
+
"notNull": bool(row["notnull"]),
|
|
71
|
+
"default": row["dflt_value"],
|
|
72
|
+
"primaryKeyOrder": row["pk"],
|
|
73
|
+
"hidden": row["hidden"],
|
|
74
|
+
}
|
|
75
|
+
for row in rows
|
|
76
|
+
]
|
|
77
|
+
|
|
78
|
+
|
|
79
|
+
def _foreign_keys(conn: sqlite3.Connection, name: str) -> list[dict[str, Any]]:
|
|
80
|
+
rows = conn.execute(f"PRAGMA foreign_key_list({_literal(name)})").fetchall()
|
|
81
|
+
return [
|
|
82
|
+
{
|
|
83
|
+
"id": row["id"],
|
|
84
|
+
"sequence": row["seq"],
|
|
85
|
+
"toTable": row["table"],
|
|
86
|
+
"fromColumn": row["from"],
|
|
87
|
+
"toColumn": row["to"],
|
|
88
|
+
"onUpdate": row["on_update"],
|
|
89
|
+
"onDelete": row["on_delete"],
|
|
90
|
+
"match": row["match"],
|
|
91
|
+
}
|
|
92
|
+
for row in rows
|
|
93
|
+
]
|
|
94
|
+
|
|
95
|
+
|
|
96
|
+
def _indexes(conn: sqlite3.Connection, table: str) -> list[dict[str, Any]]:
|
|
97
|
+
result: list[dict[str, Any]] = []
|
|
98
|
+
for row in conn.execute(f"PRAGMA index_list({_literal(table)})").fetchall():
|
|
99
|
+
name = row["name"]
|
|
100
|
+
columns: list[dict[str, Any]] = []
|
|
101
|
+
for item in conn.execute(f"PRAGMA index_xinfo({_literal(name)})").fetchall():
|
|
102
|
+
if item["key"] == 0:
|
|
103
|
+
continue
|
|
104
|
+
columns.append(
|
|
105
|
+
{
|
|
106
|
+
"sequence": item["seqno"],
|
|
107
|
+
"cid": item["cid"],
|
|
108
|
+
"name": item["name"],
|
|
109
|
+
"descending": bool(item["desc"]),
|
|
110
|
+
"collation": item["coll"],
|
|
111
|
+
}
|
|
112
|
+
)
|
|
113
|
+
sql_row = conn.execute(
|
|
114
|
+
"SELECT sql FROM sqlite_schema WHERE type='index' AND name=?", (name,)
|
|
115
|
+
).fetchone()
|
|
116
|
+
result.append(
|
|
117
|
+
{
|
|
118
|
+
"name": name,
|
|
119
|
+
"unique": bool(row["unique"]),
|
|
120
|
+
"origin": row["origin"],
|
|
121
|
+
"partial": bool(row["partial"]),
|
|
122
|
+
"columns": columns,
|
|
123
|
+
"sql": sql_row["sql"] if sql_row else None,
|
|
124
|
+
}
|
|
125
|
+
)
|
|
126
|
+
return result
|
|
127
|
+
|
|
128
|
+
|
|
129
|
+
def inspect_schema(conn: sqlite3.Connection) -> dict[str, Any]:
|
|
130
|
+
objects = conn.execute(
|
|
131
|
+
"""
|
|
132
|
+
SELECT type, name, tbl_name, sql
|
|
133
|
+
FROM sqlite_schema
|
|
134
|
+
WHERE type IN ('table', 'view')
|
|
135
|
+
AND name NOT LIKE 'sqlite_%'
|
|
136
|
+
ORDER BY type, name
|
|
137
|
+
"""
|
|
138
|
+
).fetchall()
|
|
139
|
+
|
|
140
|
+
entities: list[dict[str, Any]] = []
|
|
141
|
+
relationships: list[dict[str, Any]] = []
|
|
142
|
+
for obj in objects:
|
|
143
|
+
name = obj["name"]
|
|
144
|
+
is_table = obj["type"] == "table"
|
|
145
|
+
columns = _columns(conn, name)
|
|
146
|
+
fks = _foreign_keys(conn, name) if is_table else []
|
|
147
|
+
indexes = _indexes(conn, name) if is_table else []
|
|
148
|
+
fk_columns = {fk["fromColumn"] for fk in fks}
|
|
149
|
+
payload_columns = [
|
|
150
|
+
col for col in columns if col["name"] not in fk_columns and not col["primaryKeyOrder"]
|
|
151
|
+
]
|
|
152
|
+
junction_candidate = len(fks) >= 2 and len(payload_columns) <= 2
|
|
153
|
+
entity = {
|
|
154
|
+
"name": name,
|
|
155
|
+
"kind": obj["type"],
|
|
156
|
+
"sql": obj["sql"],
|
|
157
|
+
"columns": columns,
|
|
158
|
+
"foreignKeys": fks,
|
|
159
|
+
"indexes": indexes,
|
|
160
|
+
"junctionCandidate": junction_candidate,
|
|
161
|
+
}
|
|
162
|
+
entities.append(entity)
|
|
163
|
+
for fk in fks:
|
|
164
|
+
relationships.append(
|
|
165
|
+
{
|
|
166
|
+
"fromTable": name,
|
|
167
|
+
"fromColumn": fk["fromColumn"],
|
|
168
|
+
"toTable": fk["toTable"],
|
|
169
|
+
"toColumn": fk["toColumn"],
|
|
170
|
+
"onUpdate": fk["onUpdate"],
|
|
171
|
+
"onDelete": fk["onDelete"],
|
|
172
|
+
}
|
|
173
|
+
)
|
|
174
|
+
|
|
175
|
+
return {"entities": entities, "relationships": relationships}
|
|
176
|
+
|
|
177
|
+
|
|
178
|
+
def _normalize_parameters(parameters: Any) -> Any:
|
|
179
|
+
if isinstance(parameters, dict):
|
|
180
|
+
return {str(key).lstrip(":@$?"): value for key, value in parameters.items()}
|
|
181
|
+
if isinstance(parameters, list):
|
|
182
|
+
return parameters
|
|
183
|
+
if parameters is None:
|
|
184
|
+
return {}
|
|
185
|
+
raise ValueError("query parameters must be an object, array, or null")
|
|
186
|
+
|
|
187
|
+
|
|
188
|
+
def _aliases_and_tables(sql: str, known_tables: set[str]) -> tuple[dict[str, str], list[str]]:
|
|
189
|
+
aliases: dict[str, str] = {}
|
|
190
|
+
tables: list[str] = []
|
|
191
|
+
for match in TABLE_REF_RE.finditer(sql):
|
|
192
|
+
table = _clean_identifier(match.group(1))
|
|
193
|
+
alias = match.group(2)
|
|
194
|
+
if table not in known_tables:
|
|
195
|
+
continue
|
|
196
|
+
if table not in tables:
|
|
197
|
+
tables.append(table)
|
|
198
|
+
aliases[table] = table
|
|
199
|
+
if alias and alias.lower() not in SQL_KEYWORDS:
|
|
200
|
+
aliases[alias] = table
|
|
201
|
+
return aliases, tables
|
|
202
|
+
|
|
203
|
+
|
|
204
|
+
def _referenced_columns(sql: str, aliases: dict[str, str], known_tables: set[str]) -> list[dict[str, str]]:
|
|
205
|
+
seen: set[tuple[str, str]] = set()
|
|
206
|
+
result: list[dict[str, str]] = []
|
|
207
|
+
for qualifier, column in QUALIFIED_COLUMN_RE.findall(sql):
|
|
208
|
+
table = aliases.get(qualifier)
|
|
209
|
+
if not table and qualifier in known_tables:
|
|
210
|
+
table = qualifier
|
|
211
|
+
if not table:
|
|
212
|
+
continue
|
|
213
|
+
key = (table, column)
|
|
214
|
+
if key in seen:
|
|
215
|
+
continue
|
|
216
|
+
seen.add(key)
|
|
217
|
+
result.append({"table": table, "column": column})
|
|
218
|
+
return result
|
|
219
|
+
|
|
220
|
+
|
|
221
|
+
def _plan_for_query(conn: sqlite3.Connection, sql: str, parameters: Any) -> tuple[list[dict[str, Any]], str | None]:
|
|
222
|
+
first_keyword = re.match(r"\s*(?:--[^\n]*\n\s*)*([A-Za-z]+)", sql)
|
|
223
|
+
if not first_keyword or first_keyword.group(1).upper() not in {"SELECT", "WITH"}:
|
|
224
|
+
return [], "Query plan is only collected for SELECT/WITH statements."
|
|
225
|
+
try:
|
|
226
|
+
rows = conn.execute("EXPLAIN QUERY PLAN " + sql, _normalize_parameters(parameters)).fetchall()
|
|
227
|
+
except sqlite3.Error as exc:
|
|
228
|
+
return [], str(exc)
|
|
229
|
+
return [
|
|
230
|
+
{"id": row[0], "parent": row[1], "notUsed": row[2], "detail": row[3]}
|
|
231
|
+
for row in rows
|
|
232
|
+
], None
|
|
233
|
+
|
|
234
|
+
|
|
235
|
+
def analyze_queries(conn: sqlite3.Connection, query_specs: list[dict[str, Any]], schema: dict[str, Any]) -> list[dict[str, Any]]:
|
|
236
|
+
known_tables = {entity["name"] for entity in schema["entities"]}
|
|
237
|
+
analyzed: list[dict[str, Any]] = []
|
|
238
|
+
for position, spec in enumerate(query_specs, start=1):
|
|
239
|
+
sql = str(spec.get("sql") or "").strip()
|
|
240
|
+
if not sql:
|
|
241
|
+
raise ValueError(f"query #{position} is missing sql")
|
|
242
|
+
name = str(spec.get("name") or f"Query {position}")
|
|
243
|
+
parameters = spec.get("parameters")
|
|
244
|
+
aliases, tables = _aliases_and_tables(sql, known_tables)
|
|
245
|
+
columns = _referenced_columns(sql, aliases, known_tables)
|
|
246
|
+
plan, plan_error = _plan_for_query(conn, sql, parameters)
|
|
247
|
+
used_indexes: list[str] = []
|
|
248
|
+
plan_tables: list[str] = []
|
|
249
|
+
for step in plan:
|
|
250
|
+
detail = step["detail"] or ""
|
|
251
|
+
index_match = PLAN_INDEX_RE.search(detail)
|
|
252
|
+
if index_match:
|
|
253
|
+
index_name = _clean_identifier(index_match.group(1))
|
|
254
|
+
if index_name not in used_indexes:
|
|
255
|
+
used_indexes.append(index_name)
|
|
256
|
+
table_match = PLAN_TABLE_RE.search(detail)
|
|
257
|
+
if table_match:
|
|
258
|
+
table_name = _clean_identifier(table_match.group(1))
|
|
259
|
+
if table_name in known_tables and table_name not in plan_tables:
|
|
260
|
+
plan_tables.append(table_name)
|
|
261
|
+
for table in plan_tables:
|
|
262
|
+
if table not in tables:
|
|
263
|
+
tables.append(table)
|
|
264
|
+
analyzed.append(
|
|
265
|
+
{
|
|
266
|
+
"name": name,
|
|
267
|
+
"sql": sql,
|
|
268
|
+
"parameters": parameters,
|
|
269
|
+
"tables": tables,
|
|
270
|
+
"columns": columns,
|
|
271
|
+
"plan": plan,
|
|
272
|
+
"planError": plan_error,
|
|
273
|
+
"usedIndexes": used_indexes,
|
|
274
|
+
}
|
|
275
|
+
)
|
|
276
|
+
return analyzed
|
|
277
|
+
|
|
278
|
+
|
|
279
|
+
def _load_query_specs(path: Path | None) -> list[dict[str, Any]]:
|
|
280
|
+
if path is None:
|
|
281
|
+
return []
|
|
282
|
+
data = json.loads(path.read_text(encoding="utf-8"))
|
|
283
|
+
if not isinstance(data, list):
|
|
284
|
+
raise ValueError("queries JSON must be an array")
|
|
285
|
+
for item in data:
|
|
286
|
+
if not isinstance(item, dict):
|
|
287
|
+
raise ValueError("each query entry must be an object")
|
|
288
|
+
return data
|
|
289
|
+
|
|
290
|
+
|
|
291
|
+
def build_model(db_path: Path, queries_path: Path | None = None) -> dict[str, Any]:
|
|
292
|
+
with _open_read_only(db_path) as conn:
|
|
293
|
+
schema = inspect_schema(conn)
|
|
294
|
+
queries = analyze_queries(conn, _load_query_specs(queries_path), schema)
|
|
295
|
+
return {
|
|
296
|
+
"database": {
|
|
297
|
+
"path": str(db_path.expanduser().resolve()),
|
|
298
|
+
"engine": "SQLite",
|
|
299
|
+
"sqliteVersion": sqlite3.sqlite_version,
|
|
300
|
+
},
|
|
301
|
+
**schema,
|
|
302
|
+
"queries": queries,
|
|
303
|
+
}
|
|
304
|
+
|
|
305
|
+
|
|
306
|
+
def _json_for_script(value: Any) -> str:
|
|
307
|
+
return json.dumps(value, ensure_ascii=False).replace("</", "<\\/")
|
|
308
|
+
|
|
309
|
+
|
|
310
|
+
def render_html(model: dict[str, Any]) -> str:
|
|
311
|
+
data = _json_for_script(model)
|
|
312
|
+
title = html.escape(Path(model["database"]["path"]).name)
|
|
313
|
+
return f"""<!doctype html>
|
|
314
|
+
<html lang=\"en\">
|
|
315
|
+
<head>
|
|
316
|
+
<meta charset=\"utf-8\">
|
|
317
|
+
<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\">
|
|
318
|
+
<title>Lean DB Lab — {title}</title>
|
|
319
|
+
<style>
|
|
320
|
+
:root {{ color-scheme: light dark; font-family: Inter, ui-sans-serif, system-ui, sans-serif; }}
|
|
321
|
+
* {{ box-sizing: border-box; }}
|
|
322
|
+
body {{ margin: 0; min-height: 100vh; background: Canvas; color: CanvasText; }}
|
|
323
|
+
header {{ padding: 12px 16px; border-bottom: 1px solid color-mix(in srgb, CanvasText 20%, transparent); display:flex; gap:16px; align-items:center; flex-wrap:wrap; }}
|
|
324
|
+
header h1 {{ margin:0; font-size:18px; }}
|
|
325
|
+
.controls {{ display:flex; gap:8px; align-items:center; flex-wrap:wrap; }}
|
|
326
|
+
select, button {{ font: inherit; padding:6px 8px; }}
|
|
327
|
+
.layout {{ display:grid; grid-template-columns:minmax(0,1fr) 360px; min-height:calc(100vh - 58px); }}
|
|
328
|
+
#viewport {{ overflow:auto; position:relative; min-height:700px; background: color-mix(in srgb, Canvas 94%, CanvasText 6%); }}
|
|
329
|
+
#canvas {{ position:relative; min-width:900px; min-height:700px; }}
|
|
330
|
+
#edges {{ position:absolute; inset:0; pointer-events:none; overflow:visible; }}
|
|
331
|
+
.edge {{ stroke: color-mix(in srgb, CanvasText 35%, transparent); stroke-width:2; }}
|
|
332
|
+
.edge.active {{ stroke: CanvasText; stroke-width:4; }}
|
|
333
|
+
.card {{ position:absolute; width:270px; border:1px solid color-mix(in srgb, CanvasText 28%, transparent); border-radius:10px; background:Canvas; box-shadow:0 2px 10px color-mix(in srgb, CanvasText 10%, transparent); overflow:hidden; transition:opacity .15s, transform .15s, border-color .15s; }}
|
|
334
|
+
.card.query-hit {{ border-width:3px; transform:translateY(-2px); }}
|
|
335
|
+
.card.dim {{ opacity:.28; }}
|
|
336
|
+
.card.hidden {{ display:none; }}
|
|
337
|
+
.card h3 {{ margin:0; padding:9px 10px; font-size:14px; border-bottom:1px solid color-mix(in srgb, CanvasText 18%, transparent); cursor:pointer; }}
|
|
338
|
+
.badge {{ font-size:10px; opacity:.7; margin-left:6px; }}
|
|
339
|
+
.columns, .indexes {{ list-style:none; margin:0; padding:0; }}
|
|
340
|
+
.columns li, .indexes li {{ padding:5px 9px; border-bottom:1px solid color-mix(in srgb, CanvasText 9%, transparent); font-size:12px; cursor:pointer; }}
|
|
341
|
+
.columns li.hit {{ font-weight:700; text-decoration:underline; }}
|
|
342
|
+
.indexes li.used {{ font-weight:800; }}
|
|
343
|
+
.indexes li.candidate:not(.used) {{ font-style:italic; }}
|
|
344
|
+
.section-title {{ padding:6px 9px; font-size:10px; font-weight:700; text-transform:uppercase; opacity:.65; }}
|
|
345
|
+
aside {{ border-left:1px solid color-mix(in srgb, CanvasText 20%, transparent); padding:14px; overflow:auto; }}
|
|
346
|
+
aside h2 {{ font-size:16px; margin:0 0 10px; }}
|
|
347
|
+
aside h3 {{ font-size:13px; margin:16px 0 6px; }}
|
|
348
|
+
pre {{ white-space:pre-wrap; overflow-wrap:anywhere; font-size:11px; padding:9px; border-radius:8px; background:color-mix(in srgb, Canvas 88%, CanvasText 12%); }}
|
|
349
|
+
.kv {{ display:grid; grid-template-columns:110px 1fr; gap:4px 8px; font-size:12px; }}
|
|
350
|
+
.legend {{ font-size:11px; opacity:.8; }}
|
|
351
|
+
.legend span {{ margin-right:10px; }}
|
|
352
|
+
@media (max-width: 1000px) {{ .layout {{ grid-template-columns:1fr; }} aside {{ border-left:0; border-top:1px solid color-mix(in srgb, CanvasText 20%, transparent); }} }}
|
|
353
|
+
</style>
|
|
354
|
+
</head>
|
|
355
|
+
<body>
|
|
356
|
+
<header>
|
|
357
|
+
<h1>Lean DB Lab — {title}</h1>
|
|
358
|
+
<div class=\"controls\">
|
|
359
|
+
<label>Query <select id=\"querySelect\"></select></label>
|
|
360
|
+
<label>View <select id=\"groupSelect\"><option value=\"all\">All tables</option><option value=\"query\">Query tables</option><option value=\"connected\">Related tables</option><option value=\"junction\">Junction candidates</option></select></label>
|
|
361
|
+
</div>
|
|
362
|
+
<div class=\"legend\"><span><b>bold index</b> = planner selected</span><span><i>italic index</i> = candidate/relevant</span></div>
|
|
363
|
+
</header>
|
|
364
|
+
<div class=\"layout\">
|
|
365
|
+
<div id=\"viewport\"><div id=\"canvas\"><svg id=\"edges\"></svg><div id=\"cards\"></div></div></div>
|
|
366
|
+
<aside id=\"details\"></aside>
|
|
367
|
+
</div>
|
|
368
|
+
<script>
|
|
369
|
+
const DATA = {data};
|
|
370
|
+
const entities = new Map(DATA.entities.map(e => [e.name, e]));
|
|
371
|
+
let activeQuery = null;
|
|
372
|
+
let selected = null;
|
|
373
|
+
|
|
374
|
+
const qSelect = document.getElementById('querySelect');
|
|
375
|
+
qSelect.innerHTML = '<option value="">No query highlighting</option>' + DATA.queries.map((q,i)=>`<option value="${{i}}">${{escapeHtml(q.name)}}</option>`).join('');
|
|
376
|
+
qSelect.disabled = DATA.queries.length === 0;
|
|
377
|
+
qSelect.addEventListener('change', () => {{ activeQuery = qSelect.value === '' ? null : DATA.queries[Number(qSelect.value)]; render(); }});
|
|
378
|
+
document.getElementById('groupSelect').addEventListener('change', render);
|
|
379
|
+
window.addEventListener('resize', () => requestAnimationFrame(drawEdges));
|
|
380
|
+
|
|
381
|
+
function escapeHtml(value) {{ return String(value ?? '').replace(/[&<>\"']/g, c => ({{'&':'&','<':'<','>':'>','\"':'"',"'":'''}})[c]); }}
|
|
382
|
+
function columnKey(table, column) {{ return `${{table}}.${{column}}`; }}
|
|
383
|
+
function queryColumnKeys() {{ return new Set((activeQuery?.columns || []).map(c => columnKey(c.table,c.column))); }}
|
|
384
|
+
function queryTables() {{ return new Set(activeQuery?.tables || []); }}
|
|
385
|
+
function usedIndexes() {{ return new Set(activeQuery?.usedIndexes || []); }}
|
|
386
|
+
function relatedNames() {{ const names=new Set(); DATA.relationships.forEach(r=>{{names.add(r.fromTable);names.add(r.toTable)}}); return names; }}
|
|
387
|
+
function visibleEntities() {{
|
|
388
|
+
const mode=document.getElementById('groupSelect').value;
|
|
389
|
+
const qTables=queryTables(); const related=relatedNames();
|
|
390
|
+
return DATA.entities.filter(e => mode==='all' || (mode==='query' && qTables.has(e.name)) || (mode==='connected' && related.has(e.name)) || (mode==='junction' && e.junctionCandidate));
|
|
391
|
+
}}
|
|
392
|
+
function candidateIndex(index, table, columnKeys) {{ return (index.columns || []).some((c,i) => i===0 && columnKeys.has(columnKey(table,c.name))); }}
|
|
393
|
+
function layoutPositions(items) {{
|
|
394
|
+
const viewportWidth=Math.max(900, document.getElementById('viewport').clientWidth - 20);
|
|
395
|
+
const cols=Math.max(1, Math.floor(viewportWidth/310)); const gapX=300, gapY=330;
|
|
396
|
+
const positions=new Map(); items.forEach((e,i)=>positions.set(e.name, {{x:20+(i%cols)*gapX,y:20+Math.floor(i/cols)*gapY}}));
|
|
397
|
+
const rows=Math.ceil(items.length/cols); const canvas=document.getElementById('canvas'); canvas.style.width=`${{Math.max(900,cols*gapX+30)}}px`; canvas.style.height=`${{Math.max(700,rows*gapY+40)}}px`; return positions;
|
|
398
|
+
}}
|
|
399
|
+
function render() {{
|
|
400
|
+
const items=visibleEntities(); const positions=layoutPositions(items); const cards=document.getElementById('cards'); cards.innerHTML='';
|
|
401
|
+
const qTables=queryTables(), qCols=queryColumnKeys(), used=usedIndexes();
|
|
402
|
+
items.forEach(e => {{
|
|
403
|
+
const p=positions.get(e.name); const card=document.createElement('section'); card.className='card'; card.dataset.table=e.name; card.style.left=p.x+'px'; card.style.top=p.y+'px';
|
|
404
|
+
if (activeQuery && qTables.has(e.name)) card.classList.add('query-hit'); else if(activeQuery) card.classList.add('dim');
|
|
405
|
+
const kind=e.kind==='view'?'view':(e.junctionCandidate?'table · junction candidate':'table');
|
|
406
|
+
card.innerHTML=`<h3>${{escapeHtml(e.name)}} <span class="badge">${{escapeHtml(kind)}}</span></h3><div class="section-title">Columns</div><ul class="columns">${{e.columns.map(c=>`<li data-column="${{escapeHtml(c.name)}}" class="${{qCols.has(columnKey(e.name,c.name))?'hit':''}}">${{escapeHtml(c.name)}} <span class="badge">${{escapeHtml(c.type || 'typeless')}}${{c.primaryKeyOrder?' · PK':''}}${{c.notNull?' · NOT NULL':''}}</span></li>`).join('')}}</ul><div class="section-title">Indexes</div><ul class="indexes">${{e.indexes.map(ix=>`<li data-index="${{escapeHtml(ix.name)}}" class="${{used.has(ix.name)?'used ':''}}${{candidateIndex(ix,e.name,qCols)?'candidate':''}}">${{escapeHtml(ix.name)}} <span class="badge">${{ix.unique?'unique':'non-unique'}}${{ix.partial?' · partial':''}}</span></li>`).join('') || '<li class="badge">none</li>'}}</ul>`;
|
|
407
|
+
card.querySelector('h3').addEventListener('click',()=>{{selected={{kind:'table',table:e.name}}; renderDetails();}});
|
|
408
|
+
card.querySelectorAll('[data-column]').forEach(node=>node.addEventListener('click',ev=>{{ev.stopPropagation();selected={{kind:'column',table:e.name,column:node.dataset.column}};renderDetails();}}));
|
|
409
|
+
card.querySelectorAll('[data-index]').forEach(node=>node.addEventListener('click',ev=>{{ev.stopPropagation();selected={{kind:'index',table:e.name,index:node.dataset.index}};renderDetails();}}));
|
|
410
|
+
cards.appendChild(card);
|
|
411
|
+
}});
|
|
412
|
+
requestAnimationFrame(drawEdges); renderDetails();
|
|
413
|
+
}}
|
|
414
|
+
function drawEdges() {{
|
|
415
|
+
const svg=document.getElementById('edges'); svg.innerHTML=''; const canvas=document.getElementById('canvas'); const base=canvas.getBoundingClientRect(); const qTables=queryTables();
|
|
416
|
+
DATA.relationships.forEach(r=>{{ const a=document.querySelector(`[data-table="${{CSS.escape(r.fromTable)}}"]`); const b=document.querySelector(`[data-table="${{CSS.escape(r.toTable)}}"]`); if(!a||!b) return; const ar=a.getBoundingClientRect(), br=b.getBoundingClientRect(); const line=document.createElementNS('http://www.w3.org/2000/svg','line'); line.setAttribute('x1',ar.left-base.left+ar.width/2); line.setAttribute('y1',ar.top-base.top+ar.height/2); line.setAttribute('x2',br.left-base.left+br.width/2); line.setAttribute('y2',br.top-base.top+br.height/2); line.setAttribute('class','edge'+(activeQuery&&qTables.has(r.fromTable)&&qTables.has(r.toTable)?' active':'')); svg.appendChild(line); }});
|
|
417
|
+
}}
|
|
418
|
+
function indexDescription(ix) {{ const origin=ix.origin==='pk'?'created for a PRIMARY KEY':ix.origin==='u'?'created for a UNIQUE constraint':'explicitly created'; const cols=(ix.columns||[]).map(c=>c.name+(c.descending?' DESC':'')).join(', '); return `${{ix.unique?'Unique':'Non-unique'}} SQLite index; ${{origin}}${{ix.partial?'; partial index':''}}. Key columns: ${{cols || 'expression/rowid'}}.`; }}
|
|
419
|
+
function renderDetails() {{
|
|
420
|
+
const panel=document.getElementById('details'); let out=`<h2>Database evidence</h2><div class="kv"><b>Engine</b><span>${{escapeHtml(DATA.database.engine)}} ${{escapeHtml(DATA.database.sqliteVersion)}}</span><b>File</b><span>${{escapeHtml(DATA.database.path)}}</span></div>`;
|
|
421
|
+
if(activeQuery) {{ out+=`<h3>Active query — ${{escapeHtml(activeQuery.name)}}</h3><pre>${{escapeHtml(activeQuery.sql)}}</pre><h3>Query plan</h3>${{activeQuery.planError?`<p>${{escapeHtml(activeQuery.planError)}}</p>`:''}}<pre>${{escapeHtml((activeQuery.plan||[]).map(p=>p.detail).join('\n') || 'No plan evidence captured.')}}</pre>`; }}
|
|
422
|
+
if(selected) {{ const e=entities.get(selected.table); if(e) {{ if(selected.kind==='table') out+=`<h3>Table — ${{escapeHtml(e.name)}}</h3><p>${{e.junctionCandidate?'This is a junction/associative-table candidate based on multiple foreign keys and limited payload columns. Verify against domain intent.':''}}</p><pre>${{escapeHtml(e.sql || '')}}</pre>`; if(selected.kind==='column') {{ const c=e.columns.find(x=>x.name===selected.column); out+=`<h3>Column — ${{escapeHtml(e.name)}}.${{escapeHtml(c?.name)}}</h3><div class="kv"><b>Type</b><span>${{escapeHtml(c?.type || 'typeless')}}</span><b>Nullable</b><span>${{c?.notNull?'No':'Yes'}}</span><b>Primary key</b><span>${{c?.primaryKeyOrder?'Yes':'No'}}</span></div>`; }} if(selected.kind==='index') {{ const ix=e.indexes.find(x=>x.name===selected.index); if(ix) out+=`<h3>Index — ${{escapeHtml(ix.name)}}</h3><p>${{escapeHtml(indexDescription(ix))}}</p><p><b>Planner evidence:</b> ${{usedIndexes().has(ix.name)?'selected by the active query plan':'not shown as selected by the active query plan'}}</p><pre>${{escapeHtml(ix.sql || 'SQLite autoindex; no CREATE INDEX statement stored.')}}</pre>`; }} }} }}
|
|
423
|
+
panel.innerHTML=out;
|
|
424
|
+
}}
|
|
425
|
+
render();
|
|
426
|
+
</script>
|
|
427
|
+
</body>
|
|
428
|
+
</html>
|
|
429
|
+
"""
|
|
430
|
+
|
|
431
|
+
|
|
432
|
+
def export_workbench(db_path: Path, out_dir: Path, queries_path: Path | None = None) -> tuple[Path, Path]:
|
|
433
|
+
model = build_model(db_path, queries_path)
|
|
434
|
+
out_dir.mkdir(parents=True, exist_ok=True)
|
|
435
|
+
json_path = out_dir / "workbench.json"
|
|
436
|
+
html_path = out_dir / "index.html"
|
|
437
|
+
json_path.write_text(json.dumps(model, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
|
|
438
|
+
html_path.write_text(render_html(model), encoding="utf-8")
|
|
439
|
+
return html_path, json_path
|
|
440
|
+
|
|
441
|
+
|
|
442
|
+
def _parser() -> argparse.ArgumentParser:
|
|
443
|
+
parser = argparse.ArgumentParser(description="Generate a read-only interactive SQLite learning workbench.")
|
|
444
|
+
sub = parser.add_subparsers(dest="command", required=True)
|
|
445
|
+
export = sub.add_parser("export", help="Inspect a SQLite database and generate self-contained HTML/JSON learning artifacts.")
|
|
446
|
+
export.add_argument("--db", required=True, type=Path, help="Path to the local development/test SQLite database.")
|
|
447
|
+
export.add_argument("--out", required=True, type=Path, help="Output directory for index.html and workbench.json.")
|
|
448
|
+
export.add_argument("--queries", type=Path, help="Optional JSON array of query objects: name, sql, and optional parameters.")
|
|
449
|
+
return parser
|
|
450
|
+
|
|
451
|
+
|
|
452
|
+
def main() -> int:
|
|
453
|
+
args = _parser().parse_args()
|
|
454
|
+
if args.command == "export":
|
|
455
|
+
html_path, json_path = export_workbench(args.db, args.out, args.queries)
|
|
456
|
+
print(f"Generated: {html_path}")
|
|
457
|
+
print(f"Evidence: {json_path}")
|
|
458
|
+
return 0
|
|
459
|
+
raise AssertionError(f"unsupported command: {args.command}")
|
|
460
|
+
|
|
461
|
+
|
|
462
|
+
if __name__ == "__main__":
|
|
463
|
+
raise SystemExit(main())
|