sqlccc 1.3.0__tar.gz → 1.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {sqlccc-1.3.0/sqlccc.egg-info → sqlccc-1.4.0}/PKG-INFO +15 -10
- {sqlccc-1.3.0 → sqlccc-1.4.0}/README.md +14 -9
- {sqlccc-1.3.0 → sqlccc-1.4.0}/pyproject.toml +1 -1
- {sqlccc-1.3.0 → sqlccc-1.4.0}/sqlcc.py +48 -13
- {sqlccc-1.3.0 → sqlccc-1.4.0/sqlccc.egg-info}/PKG-INFO +15 -10
- {sqlccc-1.3.0 → sqlccc-1.4.0}/LICENSE +0 -0
- {sqlccc-1.3.0 → sqlccc-1.4.0}/setup.cfg +0 -0
- {sqlccc-1.3.0 → sqlccc-1.4.0}/sqlccc.egg-info/SOURCES.txt +0 -0
- {sqlccc-1.3.0 → sqlccc-1.4.0}/sqlccc.egg-info/dependency_links.txt +0 -0
- {sqlccc-1.3.0 → sqlccc-1.4.0}/sqlccc.egg-info/entry_points.txt +0 -0
- {sqlccc-1.3.0 → sqlccc-1.4.0}/sqlccc.egg-info/requires.txt +0 -0
- {sqlccc-1.3.0 → sqlccc-1.4.0}/sqlccc.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: sqlccc
|
|
3
|
-
Version: 1.
|
|
3
|
+
Version: 1.4.0
|
|
4
4
|
Summary: A cyclomatic complexity calculator for SQL
|
|
5
5
|
Author-email: Søren Brønsted <soren@bronsted.dk>
|
|
6
6
|
License-Expression: Apache-2.0
|
|
@@ -44,6 +44,10 @@ The SQL can contain several statements, in which case the complexity of all of t
|
|
|
44
44
|
* `ValueError` if the SQL contains no statements, or the dialect is unknown
|
|
45
45
|
* `sqlglot.errors.ParseError` if the SQL can not be parsed
|
|
46
46
|
|
|
47
|
+
`loc(sql, dialect='snowflake')` returns the lines of code, which are the lines with something on them besides
|
|
48
|
+
whitespace and comments. A string which spans several lines counts every line it is on. It raises `TypeError` if the SQL
|
|
49
|
+
is not a string, `ValueError` if the dialect is unknown and `sqlglot.errors.TokenError` if the SQL can not be tokenized.
|
|
50
|
+
|
|
47
51
|
`crap(complexity, coverage)` returns the [CRAP1 score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
|
|
48
52
|
`complexity^2 * (1 - coverage/100)^3 + complexity`, where the coverage is in percent. It is never lower than the
|
|
49
53
|
complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from a number of tests, see
|
|
@@ -54,10 +58,11 @@ complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from
|
|
|
54
58
|
```
|
|
55
59
|
sqlcc target/compiled/my_project/models
|
|
56
60
|
```
|
|
57
|
-
prints the complexity of every `.sql` file found in the directories (or of the files)
|
|
61
|
+
prints the complexity and the lines of code (see `loc`) of every `.sql` file found in the directories (or of the files)
|
|
62
|
+
given, the highest complexity first:
|
|
58
63
|
```
|
|
59
|
-
8 target/compiled/my_project/models/marts/fct_orders.sql
|
|
60
|
-
2 target/compiled/my_project/models/staging/stg_orders.sql
|
|
64
|
+
8 120 target/compiled/my_project/models/marts/fct_orders.sql
|
|
65
|
+
2 35 target/compiled/my_project/models/staging/stg_orders.sql
|
|
61
66
|
```
|
|
62
67
|
followed, on stderr, by a summary. `python -m sqlcc` does the same.
|
|
63
68
|
|
|
@@ -99,13 +104,13 @@ sqlcc --crap --manifest /tmp/dbt_sqlcc/manifest.json --max-crap 100
|
|
|
99
104
|
```
|
|
100
105
|
It prints the models with the highest score first:
|
|
101
106
|
```
|
|
102
|
-
CRAP COMP COV% TESTS MISSING MODEL
|
|
103
|
-
24.9 7 28.6 2 5 model.shop.fct_orders
|
|
104
|
-
2.0 2 100.0 3 0 model.shop.stg_customers
|
|
105
|
-
2.0 2 100.0 5 0 model.shop.stg_orders
|
|
107
|
+
CRAP COMP LOC COV% TESTS MISSING MODEL
|
|
108
|
+
24.9 7 42 28.6 2 5 model.shop.fct_orders
|
|
109
|
+
2.0 2 18 100.0 3 0 model.shop.stg_customers
|
|
110
|
+
2.0 2 15 100.0 5 0 model.shop.stg_orders
|
|
106
111
|
```
|
|
107
|
-
`COMP` is the complexity of the compiled SQL, `COV%` the estimated coverage, `TESTS` the number
|
|
108
|
-
the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
|
|
112
|
+
`COMP` is the complexity of the compiled SQL, `LOC` its lines of code, `COV%` the estimated coverage, `TESTS` the number
|
|
113
|
+
of tests and `MISSING` the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
|
|
109
114
|
|
|
110
115
|
**The coverage is an estimate.** CRAP is defined on the basis path coverage of the automated tests, but a dbt test
|
|
111
116
|
asserts something about the data and does not run the paths of the SQL. The complexity is the number of tests it takes
|
|
@@ -28,6 +28,10 @@ The SQL can contain several statements, in which case the complexity of all of t
|
|
|
28
28
|
* `ValueError` if the SQL contains no statements, or the dialect is unknown
|
|
29
29
|
* `sqlglot.errors.ParseError` if the SQL can not be parsed
|
|
30
30
|
|
|
31
|
+
`loc(sql, dialect='snowflake')` returns the lines of code, which are the lines with something on them besides
|
|
32
|
+
whitespace and comments. A string which spans several lines counts every line it is on. It raises `TypeError` if the SQL
|
|
33
|
+
is not a string, `ValueError` if the dialect is unknown and `sqlglot.errors.TokenError` if the SQL can not be tokenized.
|
|
34
|
+
|
|
31
35
|
`crap(complexity, coverage)` returns the [CRAP1 score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
|
|
32
36
|
`complexity^2 * (1 - coverage/100)^3 + complexity`, where the coverage is in percent. It is never lower than the
|
|
33
37
|
complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from a number of tests, see
|
|
@@ -38,10 +42,11 @@ complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from
|
|
|
38
42
|
```
|
|
39
43
|
sqlcc target/compiled/my_project/models
|
|
40
44
|
```
|
|
41
|
-
prints the complexity of every `.sql` file found in the directories (or of the files)
|
|
45
|
+
prints the complexity and the lines of code (see `loc`) of every `.sql` file found in the directories (or of the files)
|
|
46
|
+
given, the highest complexity first:
|
|
42
47
|
```
|
|
43
|
-
8 target/compiled/my_project/models/marts/fct_orders.sql
|
|
44
|
-
2 target/compiled/my_project/models/staging/stg_orders.sql
|
|
48
|
+
8 120 target/compiled/my_project/models/marts/fct_orders.sql
|
|
49
|
+
2 35 target/compiled/my_project/models/staging/stg_orders.sql
|
|
45
50
|
```
|
|
46
51
|
followed, on stderr, by a summary. `python -m sqlcc` does the same.
|
|
47
52
|
|
|
@@ -83,13 +88,13 @@ sqlcc --crap --manifest /tmp/dbt_sqlcc/manifest.json --max-crap 100
|
|
|
83
88
|
```
|
|
84
89
|
It prints the models with the highest score first:
|
|
85
90
|
```
|
|
86
|
-
CRAP COMP COV% TESTS MISSING MODEL
|
|
87
|
-
24.9 7 28.6 2 5 model.shop.fct_orders
|
|
88
|
-
2.0 2 100.0 3 0 model.shop.stg_customers
|
|
89
|
-
2.0 2 100.0 5 0 model.shop.stg_orders
|
|
91
|
+
CRAP COMP LOC COV% TESTS MISSING MODEL
|
|
92
|
+
24.9 7 42 28.6 2 5 model.shop.fct_orders
|
|
93
|
+
2.0 2 18 100.0 3 0 model.shop.stg_customers
|
|
94
|
+
2.0 2 15 100.0 5 0 model.shop.stg_orders
|
|
90
95
|
```
|
|
91
|
-
`COMP` is the complexity of the compiled SQL, `COV%` the estimated coverage, `TESTS` the number
|
|
92
|
-
the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
|
|
96
|
+
`COMP` is the complexity of the compiled SQL, `LOC` its lines of code, `COV%` the estimated coverage, `TESTS` the number
|
|
97
|
+
of tests and `MISSING` the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
|
|
93
98
|
|
|
94
99
|
**The coverage is an estimate.** CRAP is defined on the basis path coverage of the automated tests, but a dbt test
|
|
95
100
|
asserts something about the data and does not run the paths of the SQL. The complexity is the number of tests it takes
|
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import argparse
|
|
2
|
+
import bisect
|
|
2
3
|
import json
|
|
3
4
|
import os
|
|
4
5
|
import sys
|
|
@@ -104,6 +105,34 @@ def calc(sql: str, dialect: str = DEFAULT_DIALECT) -> int:
|
|
|
104
105
|
return 1 + sum(_weight(node) for statement in statements for node in statement.walk())
|
|
105
106
|
|
|
106
107
|
|
|
108
|
+
def loc(sql: str, dialect: str = DEFAULT_DIALECT) -> int:
|
|
109
|
+
"""
|
|
110
|
+
Count the lines of code in SQL, which are the lines with something on them besides whitespace and comments.
|
|
111
|
+
|
|
112
|
+
A token which spans several lines, like a string, counts every line it is on.
|
|
113
|
+
|
|
114
|
+
Args:
|
|
115
|
+
sql (str): The SQL string, with any number of statements.
|
|
116
|
+
dialect (str): The SQL dialect to use for tokenizing. See sqlglot documentation for supported dialects.
|
|
117
|
+
|
|
118
|
+
Returns:
|
|
119
|
+
int: The number of lines of code, 0 if there is only whitespace and comments.
|
|
120
|
+
|
|
121
|
+
Raises:
|
|
122
|
+
TypeError: If sql is not a string.
|
|
123
|
+
ValueError: If the dialect is unknown.
|
|
124
|
+
sqlglot.errors.TokenError: If sql can not be tokenized with the dialect.
|
|
125
|
+
"""
|
|
126
|
+
if not isinstance(sql, str):
|
|
127
|
+
raise TypeError(f"sql must be a str, not {type(sql).__name__}")
|
|
128
|
+
line_starts = [0] + [index + 1 for index, char in enumerate(sql) if char == '\n']
|
|
129
|
+
lines = set()
|
|
130
|
+
for token in sg.tokenize(sql, read=dialect):
|
|
131
|
+
first, last = (bisect.bisect_right(line_starts, offset) for offset in (token.start, token.end))
|
|
132
|
+
lines.update(range(first, last + 1))
|
|
133
|
+
return len(lines)
|
|
134
|
+
|
|
135
|
+
|
|
107
136
|
def crap(complexity: int, coverage: float) -> float:
|
|
108
137
|
"""
|
|
109
138
|
Calculate CRAP1, the change risk anti-pattern score: complexity^2 * (1 - coverage/100)^3 + complexity.
|
|
@@ -150,6 +179,7 @@ class ModelScore:
|
|
|
150
179
|
name: str
|
|
151
180
|
complexity: int
|
|
152
181
|
tests: int
|
|
182
|
+
loc: int
|
|
153
183
|
|
|
154
184
|
@property
|
|
155
185
|
def coverage(self) -> float:
|
|
@@ -229,16 +259,19 @@ def _first_line(e: Exception) -> str:
|
|
|
229
259
|
return (str(e).splitlines() or [type(e).__name__])[0]
|
|
230
260
|
|
|
231
261
|
|
|
232
|
-
def _score_files(
|
|
233
|
-
|
|
262
|
+
def _score_files(
|
|
263
|
+
files: list[Path], dialect: str
|
|
264
|
+
) -> tuple[dict[Path, int], dict[Path, int], dict[Path, str], list[Path]]:
|
|
265
|
+
scores, locs, failed, skipped = {}, {}, {}, []
|
|
234
266
|
for file in files:
|
|
235
267
|
try:
|
|
236
|
-
|
|
268
|
+
sql = file.read_text(encoding='utf-8')
|
|
269
|
+
scores[file], locs[file] = calc(sql, dialect=dialect), loc(sql, dialect=dialect)
|
|
237
270
|
except (OSError, UnicodeDecodeError, sg.errors.SqlglotError) as e:
|
|
238
271
|
failed[file] = _first_line(e)
|
|
239
272
|
except ValueError:
|
|
240
273
|
skipped.append(file)
|
|
241
|
-
return scores, failed, skipped
|
|
274
|
+
return scores, locs, failed, skipped
|
|
242
275
|
|
|
243
276
|
|
|
244
277
|
def _read_manifest(path: Path) -> dict:
|
|
@@ -287,7 +320,9 @@ def _score_models(manifest: dict, dialect: str) -> tuple[list[ModelScore], dict[
|
|
|
287
320
|
scores, failed, skipped = [], {}, {}
|
|
288
321
|
for unique_id, node in _models(manifest).items():
|
|
289
322
|
try:
|
|
290
|
-
|
|
323
|
+
sql = _compiled_sql(node)
|
|
324
|
+
scores.append(ModelScore(unique_id, calc(sql, dialect=dialect), tests[unique_id],
|
|
325
|
+
loc(sql, dialect=dialect)))
|
|
291
326
|
except _Skipped as e:
|
|
292
327
|
skipped[unique_id] = str(e)
|
|
293
328
|
except sg.errors.SqlglotError as e:
|
|
@@ -300,8 +335,8 @@ def _score_models(manifest: dict, dialect: str) -> tuple[list[ModelScore], dict[
|
|
|
300
335
|
def _parser() -> argparse.ArgumentParser:
|
|
301
336
|
parser = argparse.ArgumentParser(
|
|
302
337
|
prog='sqlcc',
|
|
303
|
-
description='Calculate the cyclomatic complexity of SQL files, e.g. the models dbt
|
|
304
|
-
'or with --crap the CRAP score of the models in a dbt manifest.',
|
|
338
|
+
description='Calculate the cyclomatic complexity and the lines of code of SQL files, e.g. the models dbt '
|
|
339
|
+
'compiles to target/compiled, or with --crap the CRAP score of the models in a dbt manifest.',
|
|
305
340
|
epilog='Exit status: 0 when everything is scored and nothing exceeds --max or --max-crap, '
|
|
306
341
|
'1 when something exceeds them, '
|
|
307
342
|
'2 when something can not be scored or the arguments are invalid.',
|
|
@@ -331,9 +366,9 @@ def _exceeding(values: dict, limit: float | None) -> dict:
|
|
|
331
366
|
return {key: value for key, value in values.items() if value > limit}
|
|
332
367
|
|
|
333
368
|
|
|
334
|
-
def _print_scores(scores: dict[Path, int]) -> None:
|
|
369
|
+
def _print_scores(scores: dict[Path, int], locs: dict[Path, int]) -> None:
|
|
335
370
|
for file, score in sorted(scores.items(), key=lambda item: (-item[1], str(item[0]))):
|
|
336
|
-
print(f"{score:>6} {file}")
|
|
371
|
+
print(f"{score:>6} {locs[file]:>6} {file}")
|
|
337
372
|
sys.stdout.flush()
|
|
338
373
|
|
|
339
374
|
|
|
@@ -351,9 +386,9 @@ def _print_diagnostics(scores: dict[Path, int], failed: dict[Path, str], skipped
|
|
|
351
386
|
|
|
352
387
|
|
|
353
388
|
def _print_models(scores: list[ModelScore]) -> None:
|
|
354
|
-
print(f"{'CRAP':>9} {'COMP':>5} {'COV%':>6} {'TESTS':>5} {'MISSING':>7} MODEL")
|
|
389
|
+
print(f"{'CRAP':>9} {'COMP':>5} {'LOC':>6} {'COV%':>6} {'TESTS':>5} {'MISSING':>7} MODEL")
|
|
355
390
|
for score in sorted(scores, key=lambda score: (-score.crap, score.name)):
|
|
356
|
-
print(f"{score.crap:>9.1f} {score.complexity:>5} {score.coverage:>6.1f} {score.tests:>5} "
|
|
391
|
+
print(f"{score.crap:>9.1f} {score.complexity:>5} {score.loc:>6} {score.coverage:>6.1f} {score.tests:>5} "
|
|
357
392
|
f"{score.missing:>7} {score.name}")
|
|
358
393
|
sys.stdout.flush()
|
|
359
394
|
|
|
@@ -395,9 +430,9 @@ def _run_files(args: argparse.Namespace) -> int:
|
|
|
395
430
|
print(f"sqlcc: error: {e}", file=sys.stderr)
|
|
396
431
|
return 2
|
|
397
432
|
|
|
398
|
-
scores, failed, skipped = _score_files(files, dialect)
|
|
433
|
+
scores, locs, failed, skipped = _score_files(files, dialect)
|
|
399
434
|
exceeding = _exceeding(scores, args.max_score)
|
|
400
|
-
_print_scores(scores)
|
|
435
|
+
_print_scores(scores, locs)
|
|
401
436
|
_print_diagnostics(scores, failed, skipped, exceeding, args.max_score)
|
|
402
437
|
if failed:
|
|
403
438
|
return 2
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: sqlccc
|
|
3
|
-
Version: 1.
|
|
3
|
+
Version: 1.4.0
|
|
4
4
|
Summary: A cyclomatic complexity calculator for SQL
|
|
5
5
|
Author-email: Søren Brønsted <soren@bronsted.dk>
|
|
6
6
|
License-Expression: Apache-2.0
|
|
@@ -44,6 +44,10 @@ The SQL can contain several statements, in which case the complexity of all of t
|
|
|
44
44
|
* `ValueError` if the SQL contains no statements, or the dialect is unknown
|
|
45
45
|
* `sqlglot.errors.ParseError` if the SQL can not be parsed
|
|
46
46
|
|
|
47
|
+
`loc(sql, dialect='snowflake')` returns the lines of code, which are the lines with something on them besides
|
|
48
|
+
whitespace and comments. A string which spans several lines counts every line it is on. It raises `TypeError` if the SQL
|
|
49
|
+
is not a string, `ValueError` if the dialect is unknown and `sqlglot.errors.TokenError` if the SQL can not be tokenized.
|
|
50
|
+
|
|
47
51
|
`crap(complexity, coverage)` returns the [CRAP1 score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
|
|
48
52
|
`complexity^2 * (1 - coverage/100)^3 + complexity`, where the coverage is in percent. It is never lower than the
|
|
49
53
|
complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from a number of tests, see
|
|
@@ -54,10 +58,11 @@ complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from
|
|
|
54
58
|
```
|
|
55
59
|
sqlcc target/compiled/my_project/models
|
|
56
60
|
```
|
|
57
|
-
prints the complexity of every `.sql` file found in the directories (or of the files)
|
|
61
|
+
prints the complexity and the lines of code (see `loc`) of every `.sql` file found in the directories (or of the files)
|
|
62
|
+
given, the highest complexity first:
|
|
58
63
|
```
|
|
59
|
-
8 target/compiled/my_project/models/marts/fct_orders.sql
|
|
60
|
-
2 target/compiled/my_project/models/staging/stg_orders.sql
|
|
64
|
+
8 120 target/compiled/my_project/models/marts/fct_orders.sql
|
|
65
|
+
2 35 target/compiled/my_project/models/staging/stg_orders.sql
|
|
61
66
|
```
|
|
62
67
|
followed, on stderr, by a summary. `python -m sqlcc` does the same.
|
|
63
68
|
|
|
@@ -99,13 +104,13 @@ sqlcc --crap --manifest /tmp/dbt_sqlcc/manifest.json --max-crap 100
|
|
|
99
104
|
```
|
|
100
105
|
It prints the models with the highest score first:
|
|
101
106
|
```
|
|
102
|
-
CRAP COMP COV% TESTS MISSING MODEL
|
|
103
|
-
24.9 7 28.6 2 5 model.shop.fct_orders
|
|
104
|
-
2.0 2 100.0 3 0 model.shop.stg_customers
|
|
105
|
-
2.0 2 100.0 5 0 model.shop.stg_orders
|
|
107
|
+
CRAP COMP LOC COV% TESTS MISSING MODEL
|
|
108
|
+
24.9 7 42 28.6 2 5 model.shop.fct_orders
|
|
109
|
+
2.0 2 18 100.0 3 0 model.shop.stg_customers
|
|
110
|
+
2.0 2 15 100.0 5 0 model.shop.stg_orders
|
|
106
111
|
```
|
|
107
|
-
`COMP` is the complexity of the compiled SQL, `COV%` the estimated coverage, `TESTS` the number
|
|
108
|
-
the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
|
|
112
|
+
`COMP` is the complexity of the compiled SQL, `LOC` its lines of code, `COV%` the estimated coverage, `TESTS` the number
|
|
113
|
+
of tests and `MISSING` the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
|
|
109
114
|
|
|
110
115
|
**The coverage is an estimate.** CRAP is defined on the basis path coverage of the automated tests, but a dbt test
|
|
111
116
|
asserts something about the data and does not run the paths of the SQL. The complexity is the number of tests it takes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|