sqlccc 1.2.0__tar.gz → 1.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {sqlccc-1.2.0/sqlccc.egg-info → sqlccc-1.4.0}/PKG-INFO +47 -23
- {sqlccc-1.2.0 → sqlccc-1.4.0}/README.md +46 -22
- {sqlccc-1.2.0 → sqlccc-1.4.0}/pyproject.toml +1 -1
- {sqlccc-1.2.0 → sqlccc-1.4.0}/sqlcc.py +90 -22
- {sqlccc-1.2.0 → sqlccc-1.4.0/sqlccc.egg-info}/PKG-INFO +47 -23
- {sqlccc-1.2.0 → sqlccc-1.4.0}/LICENSE +0 -0
- {sqlccc-1.2.0 → sqlccc-1.4.0}/setup.cfg +0 -0
- {sqlccc-1.2.0 → sqlccc-1.4.0}/sqlccc.egg-info/SOURCES.txt +0 -0
- {sqlccc-1.2.0 → sqlccc-1.4.0}/sqlccc.egg-info/dependency_links.txt +0 -0
- {sqlccc-1.2.0 → sqlccc-1.4.0}/sqlccc.egg-info/entry_points.txt +0 -0
- {sqlccc-1.2.0 → sqlccc-1.4.0}/sqlccc.egg-info/requires.txt +0 -0
- {sqlccc-1.2.0 → sqlccc-1.4.0}/sqlccc.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: sqlccc
|
|
3
|
-
Version: 1.
|
|
3
|
+
Version: 1.4.0
|
|
4
4
|
Summary: A cyclomatic complexity calculator for SQL
|
|
5
5
|
Author-email: Søren Brønsted <soren@bronsted.dk>
|
|
6
6
|
License-Expression: Apache-2.0
|
|
@@ -44,6 +44,10 @@ The SQL can contain several statements, in which case the complexity of all of t
|
|
|
44
44
|
* `ValueError` if the SQL contains no statements, or the dialect is unknown
|
|
45
45
|
* `sqlglot.errors.ParseError` if the SQL can not be parsed
|
|
46
46
|
|
|
47
|
+
`loc(sql, dialect='snowflake')` returns the lines of code, which are the lines with something on them besides
|
|
48
|
+
whitespace and comments. A string which spans several lines counts every line it is on. It raises `TypeError` if the SQL
|
|
49
|
+
is not a string, `ValueError` if the dialect is unknown and `sqlglot.errors.TokenError` if the SQL can not be tokenized.
|
|
50
|
+
|
|
47
51
|
`crap(complexity, coverage)` returns the [CRAP1 score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
|
|
48
52
|
`complexity^2 * (1 - coverage/100)^3 + complexity`, where the coverage is in percent. It is never lower than the
|
|
49
53
|
complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from a number of tests, see
|
|
@@ -54,10 +58,11 @@ complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from
|
|
|
54
58
|
```
|
|
55
59
|
sqlcc target/compiled/my_project/models
|
|
56
60
|
```
|
|
57
|
-
prints the complexity of every `.sql` file found in the directories (or of the files)
|
|
61
|
+
prints the complexity and the lines of code (see `loc`) of every `.sql` file found in the directories (or of the files)
|
|
62
|
+
given, the highest complexity first:
|
|
58
63
|
```
|
|
59
|
-
8 target/compiled/my_project/models/marts/fct_orders.sql
|
|
60
|
-
2 target/compiled/my_project/models/staging/stg_orders.sql
|
|
64
|
+
8 120 target/compiled/my_project/models/marts/fct_orders.sql
|
|
65
|
+
2 35 target/compiled/my_project/models/staging/stg_orders.sql
|
|
61
66
|
```
|
|
62
67
|
followed, on stderr, by a summary. `python -m sqlcc` does the same.
|
|
63
68
|
|
|
@@ -99,13 +104,13 @@ sqlcc --crap --manifest /tmp/dbt_sqlcc/manifest.json --max-crap 100
|
|
|
99
104
|
```
|
|
100
105
|
It prints the models with the highest score first:
|
|
101
106
|
```
|
|
102
|
-
CRAP COMP COV% TESTS MISSING MODEL
|
|
103
|
-
24.9 7 28.6 2 5 model.shop.fct_orders
|
|
104
|
-
2.0 2 100.0 3 0 model.shop.stg_customers
|
|
105
|
-
2.0 2 100.0 5 0 model.shop.stg_orders
|
|
107
|
+
CRAP COMP LOC COV% TESTS MISSING MODEL
|
|
108
|
+
24.9 7 42 28.6 2 5 model.shop.fct_orders
|
|
109
|
+
2.0 2 18 100.0 3 0 model.shop.stg_customers
|
|
110
|
+
2.0 2 15 100.0 5 0 model.shop.stg_orders
|
|
106
111
|
```
|
|
107
|
-
`COMP` is the complexity of the compiled SQL, `COV%` the estimated coverage, `TESTS` the number
|
|
108
|
-
the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
|
|
112
|
+
`COMP` is the complexity of the compiled SQL, `LOC` its lines of code, `COV%` the estimated coverage, `TESTS` the number
|
|
113
|
+
of tests and `MISSING` the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
|
|
109
114
|
|
|
110
115
|
**The coverage is an estimate.** CRAP is defined on the basis path coverage of the automated tests, but a dbt test
|
|
111
116
|
asserts something about the data and does not run the paths of the SQL. The complexity is the number of tests it takes
|
|
@@ -152,16 +157,20 @@ The complexity starts at 1 and each of the following adds to it:
|
|
|
152
157
|
|
|
153
158
|
| Construct | Adds |
|
|
154
159
|
|--------------------------------------------------------------------------------------------------|----------------------------|
|
|
155
|
-
| `WHERE`
|
|
160
|
+
| `WHERE`, `HAVING`, `QUALIFY` | 1 per clause |
|
|
156
161
|
| `AND`, `OR` | 1 each |
|
|
157
|
-
| `IN`, `BETWEEN`, `LIKE`, `ILIKE`
|
|
162
|
+
| `IN`, `BETWEEN`, `LIKE`, `ILIKE`, `SIMILAR TO`, `ANY`, `ALL` | 1 each |
|
|
158
163
|
| `GROUP BY` | 1 |
|
|
159
164
|
| `WHEN` branch of a `CASE`, and `IF`, `IFF` (the `CASE` itself adds nothing) | 1 per branch or call |
|
|
160
165
|
| `COALESCE`, `NVL`, `IFNULL`, `NULLIF`, `NVL2`, `DECODE`, `GREATEST`, `LEAST`, `EXISTS` | 1 per call |
|
|
161
|
-
| `
|
|
162
|
-
| `
|
|
166
|
+
| `LAG`, `LEAD` | 1 per call |
|
|
167
|
+
| `TRY_CAST`, `SAFE_CAST`, `SAFE_DIVIDE`, `TRY_TO_NUMBER`, `TRY_CONVERT` and other `TRY_`/`SAFE_` | 1 per call |
|
|
168
|
+
| `REGEXP_LIKE`, `RLIKE`, `CONTAINS`, `STARTSWITH`, `ENDSWITH`, `ARRAY_CONTAINS`, `ARRAYS_OVERLAP` | 1 per call |
|
|
169
|
+
| `COUNT_IF` | 1 per call |
|
|
163
170
|
| `JOIN` of any kind, including a comma join | 1 per join |
|
|
164
171
|
| `UNION`, `INTERSECT`, `EXCEPT` | 1 per operator |
|
|
172
|
+
| `WITH RECURSIVE`, `CONNECT BY` | 1 each |
|
|
173
|
+
| `WHEN` of a `MERGE` or multi table `INSERT`, `ON CONFLICT`, `ON DUPLICATE KEY` | 1 each |
|
|
165
174
|
|
|
166
175
|
Everything is counted anywhere in the SQL, including subqueries and CTEs. A construct is only counted once.
|
|
167
176
|
|
|
@@ -172,15 +181,30 @@ adds 1 for the join and 1 for the `AND`.
|
|
|
172
181
|
**A function counts when it branches**, which means that the path through the SQL depends on the data, and a test is
|
|
173
182
|
needed for each path. `COALESCE` has a path for a NULL and one for a value. `CAST`, `LOWER`, `TRIM`, `CONCAT` and
|
|
174
183
|
`SUBSTRING` do the same whatever the data is, and so they add nothing. Neither do the aggregates (`COUNT`, `SUM`,
|
|
175
|
-
`MAX`, ...)
|
|
176
|
-
`TRY_CAST` counts, because a value which can not be converted gives NULL, which is a path of its own.
|
|
177
|
-
which
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
+
`MAX`, ...), as they change the grain and do not branch, and `GROUP BY` counts on its own.
|
|
185
|
+
`TRY_CAST` counts, because a value which can not be converted gives NULL, which is a path of its own. So do the other
|
|
186
|
+
functions which give NULL instead of an error, like `TRY_TO_NUMBER`, `TRY_CONVERT`, `TRY_PARSE_JSON` and `SAFE_DIVIDE`,
|
|
187
|
+
also when sqlglot does not know them but their name starts with `TRY_` or `SAFE_`. Any other function which sqlglot does
|
|
188
|
+
not know adds nothing, as nothing can be said about it.
|
|
189
|
+
|
|
190
|
+
**A clause which filters rows counts**, and `WHERE`, `HAVING` and `QUALIFY` do. The expression of the filter can have
|
|
191
|
+
any length, so it is also counted by what it contains, like `AND` and `BETWEEN`. That is why
|
|
192
|
+
`HAVING COUNT(*) BETWEEN 1 AND 3` and `HAVING COUNT(*) >= 1 AND COUNT(*) <= 3` get the same complexity.
|
|
193
|
+
|
|
194
|
+
**A window function does not filter and does not branch**, except `LAG` and `LEAD`: the first or last row of a
|
|
195
|
+
partition has no neighbour, which gives NULL or the default, so they count 1 per call. `ROW_NUMBER`, `RANK`, `SUM() OVER`
|
|
196
|
+
and the like add nothing, and a filter on them is counted by the filter. `QUALIFY ROW_NUMBER() OVER (...) = 1` and
|
|
197
|
+
`SELECT ... FROM (SELECT ROW_NUMBER() OVER (...) AS rn ...) WHERE rn = 1` therefore get the same complexity.
|
|
198
|
+
|
|
199
|
+
**A loop counts**: a recursive CTE (`WITH RECURSIVE`) and `CONNECT BY` repeat until nothing is left, and count 1 each, in
|
|
200
|
+
addition to the `UNION` and the `WHERE` they have.
|
|
201
|
+
|
|
202
|
+
**A statement which branches counts**: each `WHEN` of a `MERGE` (`MATCHED`, `NOT MATCHED`) and of a multi table
|
|
203
|
+
`INSERT ALL WHEN` counts 1, but the `ELSE` of the `INSERT` does not, like the `ELSE` of a `CASE`. `ON CONFLICT` and
|
|
204
|
+
`ON DUPLICATE KEY` count 1. The `IN` of a `PIVOT` lists the columns and is not a predicate, so it does not count, and
|
|
205
|
+
`LIKE ANY (...)` counts as the `LIKE` and not as an `ANY` too.
|
|
206
|
+
|
|
207
|
+
Not counted (yet): comparisons, also `IS NULL` and `IS DISTINCT FROM`, subqueries and CTEs.
|
|
184
208
|
The weights are defined in `WEIGHTS` in `sqlcc.py`. `COALESCE` counts 1 however many arguments it has.
|
|
185
209
|
|
|
186
210
|
Which functions sqlglot knows depends on its version, and an unknown function adds nothing. For example, `IFF` in the
|
|
@@ -28,6 +28,10 @@ The SQL can contain several statements, in which case the complexity of all of t
|
|
|
28
28
|
* `ValueError` if the SQL contains no statements, or the dialect is unknown
|
|
29
29
|
* `sqlglot.errors.ParseError` if the SQL can not be parsed
|
|
30
30
|
|
|
31
|
+
`loc(sql, dialect='snowflake')` returns the lines of code, which are the lines with something on them besides
|
|
32
|
+
whitespace and comments. A string which spans several lines counts every line it is on. It raises `TypeError` if the SQL
|
|
33
|
+
is not a string, `ValueError` if the dialect is unknown and `sqlglot.errors.TokenError` if the SQL can not be tokenized.
|
|
34
|
+
|
|
31
35
|
`crap(complexity, coverage)` returns the [CRAP1 score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
|
|
32
36
|
`complexity^2 * (1 - coverage/100)^3 + complexity`, where the coverage is in percent. It is never lower than the
|
|
33
37
|
complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from a number of tests, see
|
|
@@ -38,10 +42,11 @@ complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from
|
|
|
38
42
|
```
|
|
39
43
|
sqlcc target/compiled/my_project/models
|
|
40
44
|
```
|
|
41
|
-
prints the complexity of every `.sql` file found in the directories (or of the files)
|
|
45
|
+
prints the complexity and the lines of code (see `loc`) of every `.sql` file found in the directories (or of the files)
|
|
46
|
+
given, the highest complexity first:
|
|
42
47
|
```
|
|
43
|
-
8 target/compiled/my_project/models/marts/fct_orders.sql
|
|
44
|
-
2 target/compiled/my_project/models/staging/stg_orders.sql
|
|
48
|
+
8 120 target/compiled/my_project/models/marts/fct_orders.sql
|
|
49
|
+
2 35 target/compiled/my_project/models/staging/stg_orders.sql
|
|
45
50
|
```
|
|
46
51
|
followed, on stderr, by a summary. `python -m sqlcc` does the same.
|
|
47
52
|
|
|
@@ -83,13 +88,13 @@ sqlcc --crap --manifest /tmp/dbt_sqlcc/manifest.json --max-crap 100
|
|
|
83
88
|
```
|
|
84
89
|
It prints the models with the highest score first:
|
|
85
90
|
```
|
|
86
|
-
CRAP COMP COV% TESTS MISSING MODEL
|
|
87
|
-
24.9 7 28.6 2 5 model.shop.fct_orders
|
|
88
|
-
2.0 2 100.0 3 0 model.shop.stg_customers
|
|
89
|
-
2.0 2 100.0 5 0 model.shop.stg_orders
|
|
91
|
+
CRAP COMP LOC COV% TESTS MISSING MODEL
|
|
92
|
+
24.9 7 42 28.6 2 5 model.shop.fct_orders
|
|
93
|
+
2.0 2 18 100.0 3 0 model.shop.stg_customers
|
|
94
|
+
2.0 2 15 100.0 5 0 model.shop.stg_orders
|
|
90
95
|
```
|
|
91
|
-
`COMP` is the complexity of the compiled SQL, `COV%` the estimated coverage, `TESTS` the number
|
|
92
|
-
the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
|
|
96
|
+
`COMP` is the complexity of the compiled SQL, `LOC` its lines of code, `COV%` the estimated coverage, `TESTS` the number
|
|
97
|
+
of tests and `MISSING` the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
|
|
93
98
|
|
|
94
99
|
**The coverage is an estimate.** CRAP is defined on the basis path coverage of the automated tests, but a dbt test
|
|
95
100
|
asserts something about the data and does not run the paths of the SQL. The complexity is the number of tests it takes
|
|
@@ -136,16 +141,20 @@ The complexity starts at 1 and each of the following adds to it:
|
|
|
136
141
|
|
|
137
142
|
| Construct | Adds |
|
|
138
143
|
|--------------------------------------------------------------------------------------------------|----------------------------|
|
|
139
|
-
| `WHERE`
|
|
144
|
+
| `WHERE`, `HAVING`, `QUALIFY` | 1 per clause |
|
|
140
145
|
| `AND`, `OR` | 1 each |
|
|
141
|
-
| `IN`, `BETWEEN`, `LIKE`, `ILIKE`
|
|
146
|
+
| `IN`, `BETWEEN`, `LIKE`, `ILIKE`, `SIMILAR TO`, `ANY`, `ALL` | 1 each |
|
|
142
147
|
| `GROUP BY` | 1 |
|
|
143
148
|
| `WHEN` branch of a `CASE`, and `IF`, `IFF` (the `CASE` itself adds nothing) | 1 per branch or call |
|
|
144
149
|
| `COALESCE`, `NVL`, `IFNULL`, `NULLIF`, `NVL2`, `DECODE`, `GREATEST`, `LEAST`, `EXISTS` | 1 per call |
|
|
145
|
-
| `
|
|
146
|
-
| `
|
|
150
|
+
| `LAG`, `LEAD` | 1 per call |
|
|
151
|
+
| `TRY_CAST`, `SAFE_CAST`, `SAFE_DIVIDE`, `TRY_TO_NUMBER`, `TRY_CONVERT` and other `TRY_`/`SAFE_` | 1 per call |
|
|
152
|
+
| `REGEXP_LIKE`, `RLIKE`, `CONTAINS`, `STARTSWITH`, `ENDSWITH`, `ARRAY_CONTAINS`, `ARRAYS_OVERLAP` | 1 per call |
|
|
153
|
+
| `COUNT_IF` | 1 per call |
|
|
147
154
|
| `JOIN` of any kind, including a comma join | 1 per join |
|
|
148
155
|
| `UNION`, `INTERSECT`, `EXCEPT` | 1 per operator |
|
|
156
|
+
| `WITH RECURSIVE`, `CONNECT BY` | 1 each |
|
|
157
|
+
| `WHEN` of a `MERGE` or multi table `INSERT`, `ON CONFLICT`, `ON DUPLICATE KEY` | 1 each |
|
|
149
158
|
|
|
150
159
|
Everything is counted anywhere in the SQL, including subqueries and CTEs. A construct is only counted once.
|
|
151
160
|
|
|
@@ -156,15 +165,30 @@ adds 1 for the join and 1 for the `AND`.
|
|
|
156
165
|
**A function counts when it branches**, which means that the path through the SQL depends on the data, and a test is
|
|
157
166
|
needed for each path. `COALESCE` has a path for a NULL and one for a value. `CAST`, `LOWER`, `TRIM`, `CONCAT` and
|
|
158
167
|
`SUBSTRING` do the same whatever the data is, and so they add nothing. Neither do the aggregates (`COUNT`, `SUM`,
|
|
159
|
-
`MAX`, ...)
|
|
160
|
-
`TRY_CAST` counts, because a value which can not be converted gives NULL, which is a path of its own.
|
|
161
|
-
which
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
+
`MAX`, ...), as they change the grain and do not branch, and `GROUP BY` counts on its own.
|
|
169
|
+
`TRY_CAST` counts, because a value which can not be converted gives NULL, which is a path of its own. So do the other
|
|
170
|
+
functions which give NULL instead of an error, like `TRY_TO_NUMBER`, `TRY_CONVERT`, `TRY_PARSE_JSON` and `SAFE_DIVIDE`,
|
|
171
|
+
also when sqlglot does not know them but their name starts with `TRY_` or `SAFE_`. Any other function which sqlglot does
|
|
172
|
+
not know adds nothing, as nothing can be said about it.
|
|
173
|
+
|
|
174
|
+
**A clause which filters rows counts**, and `WHERE`, `HAVING` and `QUALIFY` do. The expression of the filter can have
|
|
175
|
+
any length, so it is also counted by what it contains, like `AND` and `BETWEEN`. That is why
|
|
176
|
+
`HAVING COUNT(*) BETWEEN 1 AND 3` and `HAVING COUNT(*) >= 1 AND COUNT(*) <= 3` get the same complexity.
|
|
177
|
+
|
|
178
|
+
**A window function does not filter and does not branch**, except `LAG` and `LEAD`: the first or last row of a
|
|
179
|
+
partition has no neighbour, which gives NULL or the default, so they count 1 per call. `ROW_NUMBER`, `RANK`, `SUM() OVER`
|
|
180
|
+
and the like add nothing, and a filter on them is counted by the filter. `QUALIFY ROW_NUMBER() OVER (...) = 1` and
|
|
181
|
+
`SELECT ... FROM (SELECT ROW_NUMBER() OVER (...) AS rn ...) WHERE rn = 1` therefore get the same complexity.
|
|
182
|
+
|
|
183
|
+
**A loop counts**: a recursive CTE (`WITH RECURSIVE`) and `CONNECT BY` repeat until nothing is left, and count 1 each, in
|
|
184
|
+
addition to the `UNION` and the `WHERE` they have.
|
|
185
|
+
|
|
186
|
+
**A statement which branches counts**: each `WHEN` of a `MERGE` (`MATCHED`, `NOT MATCHED`) and of a multi table
|
|
187
|
+
`INSERT ALL WHEN` counts 1, but the `ELSE` of the `INSERT` does not, like the `ELSE` of a `CASE`. `ON CONFLICT` and
|
|
188
|
+
`ON DUPLICATE KEY` count 1. The `IN` of a `PIVOT` lists the columns and is not a predicate, so it does not count, and
|
|
189
|
+
`LIKE ANY (...)` counts as the `LIKE` and not as an `ANY` too.
|
|
190
|
+
|
|
191
|
+
Not counted (yet): comparisons, also `IS NULL` and `IS DISTINCT FROM`, subqueries and CTEs.
|
|
168
192
|
The weights are defined in `WEIGHTS` in `sqlcc.py`. `COALESCE` counts 1 however many arguments it has.
|
|
169
193
|
|
|
170
194
|
Which functions sqlglot knows depends on its version, and an unknown function adds nothing. For example, `IFF` in the
|
|
@@ -1,8 +1,10 @@
|
|
|
1
1
|
import argparse
|
|
2
|
+
import bisect
|
|
2
3
|
import json
|
|
3
4
|
import os
|
|
4
5
|
import sys
|
|
5
6
|
from collections import Counter
|
|
7
|
+
from collections.abc import Callable
|
|
6
8
|
from dataclasses import dataclass
|
|
7
9
|
from pathlib import Path
|
|
8
10
|
|
|
@@ -15,32 +17,64 @@ MAX_SKIPPED_LISTED = 5
|
|
|
15
17
|
# dbt adapter types that are named differently from the sqlglot dialect
|
|
16
18
|
ADAPTER_DIALECTS = {'sqlserver': 'tsql'}
|
|
17
19
|
|
|
20
|
+
|
|
21
|
+
def _if_arg(name: str) -> Callable[[sg.exp.Expression], int]:
|
|
22
|
+
return lambda node: 1 if node.args.get(name) else 0
|
|
23
|
+
|
|
24
|
+
|
|
25
|
+
def _unless_parent(*classes: type) -> Callable[[sg.exp.Expression], int]:
|
|
26
|
+
return lambda node: 0 if isinstance(node.parent, classes) else 1
|
|
27
|
+
|
|
28
|
+
|
|
29
|
+
def _named(*prefixes: str) -> Callable[[sg.exp.Expression], int]:
|
|
30
|
+
return lambda node: 1 if node.name.upper().startswith(prefixes) else 0
|
|
31
|
+
|
|
32
|
+
|
|
18
33
|
# The weight each construct adds to the complexity. A node gets the weight of the most specific class
|
|
19
34
|
# in its inheritance chain, so a Union counts as the SetOperation it is, and a node which has no class
|
|
20
|
-
# in WEIGHTS adds nothing.
|
|
35
|
+
# in WEIGHTS adds nothing. The weight is a number, or a function of the node when it depends on more
|
|
36
|
+
# than the class.
|
|
21
37
|
WEIGHTS = {
|
|
22
|
-
# Logic:
|
|
23
|
-
|
|
24
|
-
sg.exp.
|
|
38
|
+
# Logic: a filter (WHERE, HAVING, QUALIFY) counts 1, and its expression is counted by what it contains.
|
|
39
|
+
# The IN of a PIVOT lists the columns, and LIKE ANY counts as the LIKE
|
|
40
|
+
sg.exp.Where: 1, sg.exp.Having: 1, sg.exp.Qualify: 1, sg.exp.And: 1, sg.exp.Or: 1,
|
|
41
|
+
sg.exp.In: _unless_parent(sg.exp.Pivot), sg.exp.Between: 1, sg.exp.Like: 1, sg.exp.ILike: 1,
|
|
42
|
+
sg.exp.SimilarTo: 1, sg.exp.Group: 1,
|
|
43
|
+
sg.exp.Any: _unless_parent(sg.exp.Like, sg.exp.ILike, sg.exp.SimilarTo),
|
|
44
|
+
sg.exp.All: _unless_parent(sg.exp.Like, sg.exp.ILike, sg.exp.SimilarTo),
|
|
25
45
|
# Functions: only the ones which branch count, and a CASE through its WHEN branches (sg.exp.If). The others,
|
|
26
46
|
# like CAST and LOWER, take the same path whatever the data is. So do the ones sqlglot does not know
|
|
27
|
-
# (sg.exp.Anonymous), as nothing can be said about them
|
|
47
|
+
# (sg.exp.Anonymous), as nothing can be said about them, except the ones which are named TRY_ or SAFE_
|
|
28
48
|
sg.exp.If: 1, sg.exp.Coalesce: 1, sg.exp.Nullif: 1, sg.exp.Nvl2: 1, sg.exp.DecodeCase: 1,
|
|
29
49
|
sg.exp.Greatest: 1, sg.exp.Least: 1, sg.exp.Exists: 1,
|
|
30
|
-
#
|
|
31
|
-
|
|
50
|
+
# The first or last row of a partition has no neighbour, which gives NULL or the default. The other window
|
|
51
|
+
# functions, like ROW_NUMBER, do not branch, and a filter on them is counted by the filter
|
|
52
|
+
sg.exp.Lag: 1, sg.exp.Lead: 1,
|
|
53
|
+
# A conversion which fails gives NULL, and that is a path of its own. sqlglot sets safe on the ones which do,
|
|
54
|
+
# like TRY_TO_NUMBER, TRY_CONVERT and TRY_PARSE_JSON, but also on CONCAT, which is only about NULL arguments
|
|
55
|
+
sg.exp.TryCast: 1, sg.exp.SafeDivide: 1, sg.exp.Func: _if_arg('safe'), sg.exp.Concat: 0,
|
|
56
|
+
sg.exp.Anonymous: _named('TRY_', 'SAFE_'),
|
|
32
57
|
# Functions which are predicates, like LIKE and IN
|
|
33
58
|
sg.exp.RegexpLike: 1, sg.exp.RegexpILike: 1, sg.exp.Contains: 1, sg.exp.StartsWith: 1,
|
|
34
|
-
sg.exp.EndsWith: 1, sg.exp.ArrayContains: 1, sg.exp.CountIf: 1,
|
|
59
|
+
sg.exp.EndsWith: 1, sg.exp.ArrayContains: 1, sg.exp.ArrayOverlaps: 1, sg.exp.CountIf: 1,
|
|
35
60
|
# Sets: SetOperation is UNION, INTERSECT and EXCEPT
|
|
36
61
|
sg.exp.Join: 1, sg.exp.SetOperation: 1,
|
|
62
|
+
# Loops: a recursive CTE, and CONNECT BY
|
|
63
|
+
sg.exp.With: _if_arg('recursive'), sg.exp.Connect: 1,
|
|
64
|
+
# Branches of a statement: WHEN of MERGE, WHEN of a multi table INSERT (not its ELSE), and ON CONFLICT
|
|
65
|
+
# or ON DUPLICATE KEY
|
|
66
|
+
sg.exp.When: 1, sg.exp.ConditionalInsert: _if_arg('expression'), sg.exp.OnConflict: 1,
|
|
37
67
|
}
|
|
68
|
+
# SAFE.f() of BigQuery returns NULL for an error. It is only known to sqlglot from a later version
|
|
69
|
+
if hasattr(sg.exp, 'SafeFunc'):
|
|
70
|
+
WEIGHTS[sg.exp.SafeFunc] = 1
|
|
38
71
|
|
|
39
72
|
|
|
40
73
|
def _weight(node: sg.exp.Expression) -> int:
|
|
41
74
|
for cls in type(node).__mro__:
|
|
42
75
|
if cls in WEIGHTS:
|
|
43
|
-
|
|
76
|
+
weight = WEIGHTS[cls]
|
|
77
|
+
return weight(node) if callable(weight) else weight
|
|
44
78
|
return 0
|
|
45
79
|
|
|
46
80
|
|
|
@@ -71,6 +105,34 @@ def calc(sql: str, dialect: str = DEFAULT_DIALECT) -> int:
|
|
|
71
105
|
return 1 + sum(_weight(node) for statement in statements for node in statement.walk())
|
|
72
106
|
|
|
73
107
|
|
|
108
|
+
def loc(sql: str, dialect: str = DEFAULT_DIALECT) -> int:
|
|
109
|
+
"""
|
|
110
|
+
Count the lines of code in SQL, which are the lines with something on them besides whitespace and comments.
|
|
111
|
+
|
|
112
|
+
A token which spans several lines, like a string, counts every line it is on.
|
|
113
|
+
|
|
114
|
+
Args:
|
|
115
|
+
sql (str): The SQL string, with any number of statements.
|
|
116
|
+
dialect (str): The SQL dialect to use for tokenizing. See sqlglot documentation for supported dialects.
|
|
117
|
+
|
|
118
|
+
Returns:
|
|
119
|
+
int: The number of lines of code, 0 if there is only whitespace and comments.
|
|
120
|
+
|
|
121
|
+
Raises:
|
|
122
|
+
TypeError: If sql is not a string.
|
|
123
|
+
ValueError: If the dialect is unknown.
|
|
124
|
+
sqlglot.errors.TokenError: If sql can not be tokenized with the dialect.
|
|
125
|
+
"""
|
|
126
|
+
if not isinstance(sql, str):
|
|
127
|
+
raise TypeError(f"sql must be a str, not {type(sql).__name__}")
|
|
128
|
+
line_starts = [0] + [index + 1 for index, char in enumerate(sql) if char == '\n']
|
|
129
|
+
lines = set()
|
|
130
|
+
for token in sg.tokenize(sql, read=dialect):
|
|
131
|
+
first, last = (bisect.bisect_right(line_starts, offset) for offset in (token.start, token.end))
|
|
132
|
+
lines.update(range(first, last + 1))
|
|
133
|
+
return len(lines)
|
|
134
|
+
|
|
135
|
+
|
|
74
136
|
def crap(complexity: int, coverage: float) -> float:
|
|
75
137
|
"""
|
|
76
138
|
Calculate CRAP1, the change risk anti-pattern score: complexity^2 * (1 - coverage/100)^3 + complexity.
|
|
@@ -117,6 +179,7 @@ class ModelScore:
|
|
|
117
179
|
name: str
|
|
118
180
|
complexity: int
|
|
119
181
|
tests: int
|
|
182
|
+
loc: int
|
|
120
183
|
|
|
121
184
|
@property
|
|
122
185
|
def coverage(self) -> float:
|
|
@@ -196,16 +259,19 @@ def _first_line(e: Exception) -> str:
|
|
|
196
259
|
return (str(e).splitlines() or [type(e).__name__])[0]
|
|
197
260
|
|
|
198
261
|
|
|
199
|
-
def _score_files(
|
|
200
|
-
|
|
262
|
+
def _score_files(
|
|
263
|
+
files: list[Path], dialect: str
|
|
264
|
+
) -> tuple[dict[Path, int], dict[Path, int], dict[Path, str], list[Path]]:
|
|
265
|
+
scores, locs, failed, skipped = {}, {}, {}, []
|
|
201
266
|
for file in files:
|
|
202
267
|
try:
|
|
203
|
-
|
|
268
|
+
sql = file.read_text(encoding='utf-8')
|
|
269
|
+
scores[file], locs[file] = calc(sql, dialect=dialect), loc(sql, dialect=dialect)
|
|
204
270
|
except (OSError, UnicodeDecodeError, sg.errors.SqlglotError) as e:
|
|
205
271
|
failed[file] = _first_line(e)
|
|
206
272
|
except ValueError:
|
|
207
273
|
skipped.append(file)
|
|
208
|
-
return scores, failed, skipped
|
|
274
|
+
return scores, locs, failed, skipped
|
|
209
275
|
|
|
210
276
|
|
|
211
277
|
def _read_manifest(path: Path) -> dict:
|
|
@@ -254,7 +320,9 @@ def _score_models(manifest: dict, dialect: str) -> tuple[list[ModelScore], dict[
|
|
|
254
320
|
scores, failed, skipped = [], {}, {}
|
|
255
321
|
for unique_id, node in _models(manifest).items():
|
|
256
322
|
try:
|
|
257
|
-
|
|
323
|
+
sql = _compiled_sql(node)
|
|
324
|
+
scores.append(ModelScore(unique_id, calc(sql, dialect=dialect), tests[unique_id],
|
|
325
|
+
loc(sql, dialect=dialect)))
|
|
258
326
|
except _Skipped as e:
|
|
259
327
|
skipped[unique_id] = str(e)
|
|
260
328
|
except sg.errors.SqlglotError as e:
|
|
@@ -267,8 +335,8 @@ def _score_models(manifest: dict, dialect: str) -> tuple[list[ModelScore], dict[
|
|
|
267
335
|
def _parser() -> argparse.ArgumentParser:
|
|
268
336
|
parser = argparse.ArgumentParser(
|
|
269
337
|
prog='sqlcc',
|
|
270
|
-
description='Calculate the cyclomatic complexity of SQL files, e.g. the models dbt
|
|
271
|
-
'or with --crap the CRAP score of the models in a dbt manifest.',
|
|
338
|
+
description='Calculate the cyclomatic complexity and the lines of code of SQL files, e.g. the models dbt '
|
|
339
|
+
'compiles to target/compiled, or with --crap the CRAP score of the models in a dbt manifest.',
|
|
272
340
|
epilog='Exit status: 0 when everything is scored and nothing exceeds --max or --max-crap, '
|
|
273
341
|
'1 when something exceeds them, '
|
|
274
342
|
'2 when something can not be scored or the arguments are invalid.',
|
|
@@ -298,9 +366,9 @@ def _exceeding(values: dict, limit: float | None) -> dict:
|
|
|
298
366
|
return {key: value for key, value in values.items() if value > limit}
|
|
299
367
|
|
|
300
368
|
|
|
301
|
-
def _print_scores(scores: dict[Path, int]) -> None:
|
|
369
|
+
def _print_scores(scores: dict[Path, int], locs: dict[Path, int]) -> None:
|
|
302
370
|
for file, score in sorted(scores.items(), key=lambda item: (-item[1], str(item[0]))):
|
|
303
|
-
print(f"{score:>6} {file}")
|
|
371
|
+
print(f"{score:>6} {locs[file]:>6} {file}")
|
|
304
372
|
sys.stdout.flush()
|
|
305
373
|
|
|
306
374
|
|
|
@@ -318,9 +386,9 @@ def _print_diagnostics(scores: dict[Path, int], failed: dict[Path, str], skipped
|
|
|
318
386
|
|
|
319
387
|
|
|
320
388
|
def _print_models(scores: list[ModelScore]) -> None:
|
|
321
|
-
print(f"{'CRAP':>9} {'COMP':>5} {'COV%':>6} {'TESTS':>5} {'MISSING':>7} MODEL")
|
|
389
|
+
print(f"{'CRAP':>9} {'COMP':>5} {'LOC':>6} {'COV%':>6} {'TESTS':>5} {'MISSING':>7} MODEL")
|
|
322
390
|
for score in sorted(scores, key=lambda score: (-score.crap, score.name)):
|
|
323
|
-
print(f"{score.crap:>9.1f} {score.complexity:>5} {score.coverage:>6.1f} {score.tests:>5} "
|
|
391
|
+
print(f"{score.crap:>9.1f} {score.complexity:>5} {score.loc:>6} {score.coverage:>6.1f} {score.tests:>5} "
|
|
324
392
|
f"{score.missing:>7} {score.name}")
|
|
325
393
|
sys.stdout.flush()
|
|
326
394
|
|
|
@@ -362,9 +430,9 @@ def _run_files(args: argparse.Namespace) -> int:
|
|
|
362
430
|
print(f"sqlcc: error: {e}", file=sys.stderr)
|
|
363
431
|
return 2
|
|
364
432
|
|
|
365
|
-
scores, failed, skipped = _score_files(files, dialect)
|
|
433
|
+
scores, locs, failed, skipped = _score_files(files, dialect)
|
|
366
434
|
exceeding = _exceeding(scores, args.max_score)
|
|
367
|
-
_print_scores(scores)
|
|
435
|
+
_print_scores(scores, locs)
|
|
368
436
|
_print_diagnostics(scores, failed, skipped, exceeding, args.max_score)
|
|
369
437
|
if failed:
|
|
370
438
|
return 2
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: sqlccc
|
|
3
|
-
Version: 1.
|
|
3
|
+
Version: 1.4.0
|
|
4
4
|
Summary: A cyclomatic complexity calculator for SQL
|
|
5
5
|
Author-email: Søren Brønsted <soren@bronsted.dk>
|
|
6
6
|
License-Expression: Apache-2.0
|
|
@@ -44,6 +44,10 @@ The SQL can contain several statements, in which case the complexity of all of t
|
|
|
44
44
|
* `ValueError` if the SQL contains no statements, or the dialect is unknown
|
|
45
45
|
* `sqlglot.errors.ParseError` if the SQL can not be parsed
|
|
46
46
|
|
|
47
|
+
`loc(sql, dialect='snowflake')` returns the lines of code, which are the lines with something on them besides
|
|
48
|
+
whitespace and comments. A string which spans several lines counts every line it is on. It raises `TypeError` if the SQL
|
|
49
|
+
is not a string, `ValueError` if the dialect is unknown and `sqlglot.errors.TokenError` if the SQL can not be tokenized.
|
|
50
|
+
|
|
47
51
|
`crap(complexity, coverage)` returns the [CRAP1 score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
|
|
48
52
|
`complexity^2 * (1 - coverage/100)^3 + complexity`, where the coverage is in percent. It is never lower than the
|
|
49
53
|
complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from a number of tests, see
|
|
@@ -54,10 +58,11 @@ complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from
|
|
|
54
58
|
```
|
|
55
59
|
sqlcc target/compiled/my_project/models
|
|
56
60
|
```
|
|
57
|
-
prints the complexity of every `.sql` file found in the directories (or of the files)
|
|
61
|
+
prints the complexity and the lines of code (see `loc`) of every `.sql` file found in the directories (or of the files)
|
|
62
|
+
given, the highest complexity first:
|
|
58
63
|
```
|
|
59
|
-
8 target/compiled/my_project/models/marts/fct_orders.sql
|
|
60
|
-
2 target/compiled/my_project/models/staging/stg_orders.sql
|
|
64
|
+
8 120 target/compiled/my_project/models/marts/fct_orders.sql
|
|
65
|
+
2 35 target/compiled/my_project/models/staging/stg_orders.sql
|
|
61
66
|
```
|
|
62
67
|
followed, on stderr, by a summary. `python -m sqlcc` does the same.
|
|
63
68
|
|
|
@@ -99,13 +104,13 @@ sqlcc --crap --manifest /tmp/dbt_sqlcc/manifest.json --max-crap 100
|
|
|
99
104
|
```
|
|
100
105
|
It prints the models with the highest score first:
|
|
101
106
|
```
|
|
102
|
-
CRAP COMP COV% TESTS MISSING MODEL
|
|
103
|
-
24.9 7 28.6 2 5 model.shop.fct_orders
|
|
104
|
-
2.0 2 100.0 3 0 model.shop.stg_customers
|
|
105
|
-
2.0 2 100.0 5 0 model.shop.stg_orders
|
|
107
|
+
CRAP COMP LOC COV% TESTS MISSING MODEL
|
|
108
|
+
24.9 7 42 28.6 2 5 model.shop.fct_orders
|
|
109
|
+
2.0 2 18 100.0 3 0 model.shop.stg_customers
|
|
110
|
+
2.0 2 15 100.0 5 0 model.shop.stg_orders
|
|
106
111
|
```
|
|
107
|
-
`COMP` is the complexity of the compiled SQL, `COV%` the estimated coverage, `TESTS` the number
|
|
108
|
-
the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
|
|
112
|
+
`COMP` is the complexity of the compiled SQL, `LOC` its lines of code, `COV%` the estimated coverage, `TESTS` the number
|
|
113
|
+
of tests and `MISSING` the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
|
|
109
114
|
|
|
110
115
|
**The coverage is an estimate.** CRAP is defined on the basis path coverage of the automated tests, but a dbt test
|
|
111
116
|
asserts something about the data and does not run the paths of the SQL. The complexity is the number of tests it takes
|
|
@@ -152,16 +157,20 @@ The complexity starts at 1 and each of the following adds to it:
|
|
|
152
157
|
|
|
153
158
|
| Construct | Adds |
|
|
154
159
|
|--------------------------------------------------------------------------------------------------|----------------------------|
|
|
155
|
-
| `WHERE`
|
|
160
|
+
| `WHERE`, `HAVING`, `QUALIFY` | 1 per clause |
|
|
156
161
|
| `AND`, `OR` | 1 each |
|
|
157
|
-
| `IN`, `BETWEEN`, `LIKE`, `ILIKE`
|
|
162
|
+
| `IN`, `BETWEEN`, `LIKE`, `ILIKE`, `SIMILAR TO`, `ANY`, `ALL` | 1 each |
|
|
158
163
|
| `GROUP BY` | 1 |
|
|
159
164
|
| `WHEN` branch of a `CASE`, and `IF`, `IFF` (the `CASE` itself adds nothing) | 1 per branch or call |
|
|
160
165
|
| `COALESCE`, `NVL`, `IFNULL`, `NULLIF`, `NVL2`, `DECODE`, `GREATEST`, `LEAST`, `EXISTS` | 1 per call |
|
|
161
|
-
| `
|
|
162
|
-
| `
|
|
166
|
+
| `LAG`, `LEAD` | 1 per call |
|
|
167
|
+
| `TRY_CAST`, `SAFE_CAST`, `SAFE_DIVIDE`, `TRY_TO_NUMBER`, `TRY_CONVERT` and other `TRY_`/`SAFE_` | 1 per call |
|
|
168
|
+
| `REGEXP_LIKE`, `RLIKE`, `CONTAINS`, `STARTSWITH`, `ENDSWITH`, `ARRAY_CONTAINS`, `ARRAYS_OVERLAP` | 1 per call |
|
|
169
|
+
| `COUNT_IF` | 1 per call |
|
|
163
170
|
| `JOIN` of any kind, including a comma join | 1 per join |
|
|
164
171
|
| `UNION`, `INTERSECT`, `EXCEPT` | 1 per operator |
|
|
172
|
+
| `WITH RECURSIVE`, `CONNECT BY` | 1 each |
|
|
173
|
+
| `WHEN` of a `MERGE` or multi table `INSERT`, `ON CONFLICT`, `ON DUPLICATE KEY` | 1 each |
|
|
165
174
|
|
|
166
175
|
Everything is counted anywhere in the SQL, including subqueries and CTEs. A construct is only counted once.
|
|
167
176
|
|
|
@@ -172,15 +181,30 @@ adds 1 for the join and 1 for the `AND`.
|
|
|
172
181
|
**A function counts when it branches**, which means that the path through the SQL depends on the data, and a test is
|
|
173
182
|
needed for each path. `COALESCE` has a path for a NULL and one for a value. `CAST`, `LOWER`, `TRIM`, `CONCAT` and
|
|
174
183
|
`SUBSTRING` do the same whatever the data is, and so they add nothing. Neither do the aggregates (`COUNT`, `SUM`,
|
|
175
|
-
`MAX`, ...)
|
|
176
|
-
`TRY_CAST` counts, because a value which can not be converted gives NULL, which is a path of its own.
|
|
177
|
-
which
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
+
`MAX`, ...), as they change the grain and do not branch, and `GROUP BY` counts on its own.
|
|
185
|
+
`TRY_CAST` counts, because a value which can not be converted gives NULL, which is a path of its own. So do the other
|
|
186
|
+
functions which give NULL instead of an error, like `TRY_TO_NUMBER`, `TRY_CONVERT`, `TRY_PARSE_JSON` and `SAFE_DIVIDE`,
|
|
187
|
+
also when sqlglot does not know them but their name starts with `TRY_` or `SAFE_`. Any other function which sqlglot does
|
|
188
|
+
not know adds nothing, as nothing can be said about it.
|
|
189
|
+
|
|
190
|
+
**A clause which filters rows counts**, and `WHERE`, `HAVING` and `QUALIFY` do. The expression of the filter can have
|
|
191
|
+
any length, so it is also counted by what it contains, like `AND` and `BETWEEN`. That is why
|
|
192
|
+
`HAVING COUNT(*) BETWEEN 1 AND 3` and `HAVING COUNT(*) >= 1 AND COUNT(*) <= 3` get the same complexity.
|
|
193
|
+
|
|
194
|
+
**A window function does not filter and does not branch**, except `LAG` and `LEAD`: the first or last row of a
|
|
195
|
+
partition has no neighbour, which gives NULL or the default, so they count 1 per call. `ROW_NUMBER`, `RANK`, `SUM() OVER`
|
|
196
|
+
and the like add nothing, and a filter on them is counted by the filter. `QUALIFY ROW_NUMBER() OVER (...) = 1` and
|
|
197
|
+
`SELECT ... FROM (SELECT ROW_NUMBER() OVER (...) AS rn ...) WHERE rn = 1` therefore get the same complexity.
|
|
198
|
+
|
|
199
|
+
**A loop counts**: a recursive CTE (`WITH RECURSIVE`) and `CONNECT BY` repeat until nothing is left, and count 1 each, in
|
|
200
|
+
addition to the `UNION` and the `WHERE` they have.
|
|
201
|
+
|
|
202
|
+
**A statement which branches counts**: each `WHEN` of a `MERGE` (`MATCHED`, `NOT MATCHED`) and of a multi table
|
|
203
|
+
`INSERT ALL WHEN` counts 1, but the `ELSE` of the `INSERT` does not, like the `ELSE` of a `CASE`. `ON CONFLICT` and
|
|
204
|
+
`ON DUPLICATE KEY` count 1. The `IN` of a `PIVOT` lists the columns and is not a predicate, so it does not count, and
|
|
205
|
+
`LIKE ANY (...)` counts as the `LIKE` and not as an `ANY` too.
|
|
206
|
+
|
|
207
|
+
Not counted (yet): comparisons, also `IS NULL` and `IS DISTINCT FROM`, subqueries and CTEs.
|
|
184
208
|
The weights are defined in `WEIGHTS` in `sqlcc.py`. `COALESCE` counts 1 however many arguments it has.
|
|
185
209
|
|
|
186
210
|
Which functions sqlglot knows depends on its version, and an unknown function adds nothing. For example, `IFF` in the
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|