sqlccc 1.3.0__tar.gz → 1.4.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: sqlccc
3
- Version: 1.3.0
3
+ Version: 1.4.0
4
4
  Summary: A cyclomatic complexity calculator for SQL
5
5
  Author-email: Søren Brønsted <soren@bronsted.dk>
6
6
  License-Expression: Apache-2.0
@@ -44,6 +44,10 @@ The SQL can contain several statements, in which case the complexity of all of t
44
44
  * `ValueError` if the SQL contains no statements, or the dialect is unknown
45
45
  * `sqlglot.errors.ParseError` if the SQL can not be parsed
46
46
 
47
+ `loc(sql, dialect='snowflake')` returns the lines of code, which are the lines with something on them besides
48
+ whitespace and comments. A string which spans several lines counts every line it is on. It raises `TypeError` if the SQL
49
+ is not a string, `ValueError` if the dialect is unknown and `sqlglot.errors.TokenError` if the SQL can not be tokenized.
50
+
47
51
  `crap(complexity, coverage)` returns the [CRAP1 score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
48
52
  `complexity^2 * (1 - coverage/100)^3 + complexity`, where the coverage is in percent. It is never lower than the
49
53
  complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from a number of tests, see
@@ -54,10 +58,11 @@ complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from
54
58
  ```
55
59
  sqlcc target/compiled/my_project/models
56
60
  ```
57
- prints the complexity of every `.sql` file found in the directories (or of the files) given, the highest first:
61
+ prints the complexity and the lines of code (see `loc`) of every `.sql` file found in the directories (or of the files)
62
+ given, the highest complexity first:
58
63
  ```
59
- 8 target/compiled/my_project/models/marts/fct_orders.sql
60
- 2 target/compiled/my_project/models/staging/stg_orders.sql
64
+ 8 120 target/compiled/my_project/models/marts/fct_orders.sql
65
+ 2 35 target/compiled/my_project/models/staging/stg_orders.sql
61
66
  ```
62
67
  followed, on stderr, by a summary. `python -m sqlcc` does the same.
63
68
 
@@ -99,13 +104,13 @@ sqlcc --crap --manifest /tmp/dbt_sqlcc/manifest.json --max-crap 100
99
104
  ```
100
105
  It prints the models with the highest score first:
101
106
  ```
102
- CRAP COMP COV% TESTS MISSING MODEL
103
- 24.9 7 28.6 2 5 model.shop.fct_orders
104
- 2.0 2 100.0 3 0 model.shop.stg_customers
105
- 2.0 2 100.0 5 0 model.shop.stg_orders
107
+ CRAP COMP LOC COV% TESTS MISSING MODEL
108
+ 24.9 7 42 28.6 2 5 model.shop.fct_orders
109
+ 2.0 2 18 100.0 3 0 model.shop.stg_customers
110
+ 2.0 2 15 100.0 5 0 model.shop.stg_orders
106
111
  ```
107
- `COMP` is the complexity of the compiled SQL, `COV%` the estimated coverage, `TESTS` the number of tests and `MISSING`
108
- the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
112
+ `COMP` is the complexity of the compiled SQL, `LOC` its lines of code, `COV%` the estimated coverage, `TESTS` the number
113
+ of tests and `MISSING` the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
109
114
 
110
115
  **The coverage is an estimate.** CRAP is defined on the basis path coverage of the automated tests, but a dbt test
111
116
  asserts something about the data and does not run the paths of the SQL. The complexity is the number of tests it takes
@@ -28,6 +28,10 @@ The SQL can contain several statements, in which case the complexity of all of t
28
28
  * `ValueError` if the SQL contains no statements, or the dialect is unknown
29
29
  * `sqlglot.errors.ParseError` if the SQL can not be parsed
30
30
 
31
+ `loc(sql, dialect='snowflake')` returns the lines of code, which are the lines with something on them besides
32
+ whitespace and comments. A string which spans several lines counts every line it is on. It raises `TypeError` if the SQL
33
+ is not a string, `ValueError` if the dialect is unknown and `sqlglot.errors.TokenError` if the SQL can not be tokenized.
34
+
31
35
  `crap(complexity, coverage)` returns the [CRAP1 score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
32
36
  `complexity^2 * (1 - coverage/100)^3 + complexity`, where the coverage is in percent. It is never lower than the
33
37
  complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from a number of tests, see
@@ -38,10 +42,11 @@ complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from
38
42
  ```
39
43
  sqlcc target/compiled/my_project/models
40
44
  ```
41
- prints the complexity of every `.sql` file found in the directories (or of the files) given, the highest first:
45
+ prints the complexity and the lines of code (see `loc`) of every `.sql` file found in the directories (or of the files)
46
+ given, the highest complexity first:
42
47
  ```
43
- 8 target/compiled/my_project/models/marts/fct_orders.sql
44
- 2 target/compiled/my_project/models/staging/stg_orders.sql
48
+ 8 120 target/compiled/my_project/models/marts/fct_orders.sql
49
+ 2 35 target/compiled/my_project/models/staging/stg_orders.sql
45
50
  ```
46
51
  followed, on stderr, by a summary. `python -m sqlcc` does the same.
47
52
 
@@ -83,13 +88,13 @@ sqlcc --crap --manifest /tmp/dbt_sqlcc/manifest.json --max-crap 100
83
88
  ```
84
89
  It prints the models with the highest score first:
85
90
  ```
86
- CRAP COMP COV% TESTS MISSING MODEL
87
- 24.9 7 28.6 2 5 model.shop.fct_orders
88
- 2.0 2 100.0 3 0 model.shop.stg_customers
89
- 2.0 2 100.0 5 0 model.shop.stg_orders
91
+ CRAP COMP LOC COV% TESTS MISSING MODEL
92
+ 24.9 7 42 28.6 2 5 model.shop.fct_orders
93
+ 2.0 2 18 100.0 3 0 model.shop.stg_customers
94
+ 2.0 2 15 100.0 5 0 model.shop.stg_orders
90
95
  ```
91
- `COMP` is the complexity of the compiled SQL, `COV%` the estimated coverage, `TESTS` the number of tests and `MISSING`
92
- the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
96
+ `COMP` is the complexity of the compiled SQL, `LOC` its lines of code, `COV%` the estimated coverage, `TESTS` the number
97
+ of tests and `MISSING` the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
93
98
 
94
99
  **The coverage is an estimate.** CRAP is defined on the basis path coverage of the automated tests, but a dbt test
95
100
  asserts something about the data and does not run the paths of the SQL. The complexity is the number of tests it takes
@@ -1,6 +1,6 @@
1
1
  [project]
2
2
  name = "sqlccc"
3
- version = "1.3.0"
3
+ version = "1.4.0"
4
4
  authors = [
5
5
  { name="Søren Brønsted", email="soren@bronsted.dk" },
6
6
  ]
@@ -1,4 +1,5 @@
1
1
  import argparse
2
+ import bisect
2
3
  import json
3
4
  import os
4
5
  import sys
@@ -104,6 +105,34 @@ def calc(sql: str, dialect: str = DEFAULT_DIALECT) -> int:
104
105
  return 1 + sum(_weight(node) for statement in statements for node in statement.walk())
105
106
 
106
107
 
108
+ def loc(sql: str, dialect: str = DEFAULT_DIALECT) -> int:
109
+ """
110
+ Count the lines of code in SQL, which are the lines with something on them besides whitespace and comments.
111
+
112
+ A token which spans several lines, like a string, counts every line it is on.
113
+
114
+ Args:
115
+ sql (str): The SQL string, with any number of statements.
116
+ dialect (str): The SQL dialect to use for tokenizing. See sqlglot documentation for supported dialects.
117
+
118
+ Returns:
119
+ int: The number of lines of code, 0 if there is only whitespace and comments.
120
+
121
+ Raises:
122
+ TypeError: If sql is not a string.
123
+ ValueError: If the dialect is unknown.
124
+ sqlglot.errors.TokenError: If sql can not be tokenized with the dialect.
125
+ """
126
+ if not isinstance(sql, str):
127
+ raise TypeError(f"sql must be a str, not {type(sql).__name__}")
128
+ line_starts = [0] + [index + 1 for index, char in enumerate(sql) if char == '\n']
129
+ lines = set()
130
+ for token in sg.tokenize(sql, read=dialect):
131
+ first, last = (bisect.bisect_right(line_starts, offset) for offset in (token.start, token.end))
132
+ lines.update(range(first, last + 1))
133
+ return len(lines)
134
+
135
+
107
136
  def crap(complexity: int, coverage: float) -> float:
108
137
  """
109
138
  Calculate CRAP1, the change risk anti-pattern score: complexity^2 * (1 - coverage/100)^3 + complexity.
@@ -150,6 +179,7 @@ class ModelScore:
150
179
  name: str
151
180
  complexity: int
152
181
  tests: int
182
+ loc: int
153
183
 
154
184
  @property
155
185
  def coverage(self) -> float:
@@ -229,16 +259,19 @@ def _first_line(e: Exception) -> str:
229
259
  return (str(e).splitlines() or [type(e).__name__])[0]
230
260
 
231
261
 
232
- def _score_files(files: list[Path], dialect: str) -> tuple[dict[Path, int], dict[Path, str], list[Path]]:
233
- scores, failed, skipped = {}, {}, []
262
+ def _score_files(
263
+ files: list[Path], dialect: str
264
+ ) -> tuple[dict[Path, int], dict[Path, int], dict[Path, str], list[Path]]:
265
+ scores, locs, failed, skipped = {}, {}, {}, []
234
266
  for file in files:
235
267
  try:
236
- scores[file] = calc(file.read_text(encoding='utf-8'), dialect=dialect)
268
+ sql = file.read_text(encoding='utf-8')
269
+ scores[file], locs[file] = calc(sql, dialect=dialect), loc(sql, dialect=dialect)
237
270
  except (OSError, UnicodeDecodeError, sg.errors.SqlglotError) as e:
238
271
  failed[file] = _first_line(e)
239
272
  except ValueError:
240
273
  skipped.append(file)
241
- return scores, failed, skipped
274
+ return scores, locs, failed, skipped
242
275
 
243
276
 
244
277
  def _read_manifest(path: Path) -> dict:
@@ -287,7 +320,9 @@ def _score_models(manifest: dict, dialect: str) -> tuple[list[ModelScore], dict[
287
320
  scores, failed, skipped = [], {}, {}
288
321
  for unique_id, node in _models(manifest).items():
289
322
  try:
290
- scores.append(ModelScore(unique_id, calc(_compiled_sql(node), dialect=dialect), tests[unique_id]))
323
+ sql = _compiled_sql(node)
324
+ scores.append(ModelScore(unique_id, calc(sql, dialect=dialect), tests[unique_id],
325
+ loc(sql, dialect=dialect)))
291
326
  except _Skipped as e:
292
327
  skipped[unique_id] = str(e)
293
328
  except sg.errors.SqlglotError as e:
@@ -300,8 +335,8 @@ def _score_models(manifest: dict, dialect: str) -> tuple[list[ModelScore], dict[
300
335
  def _parser() -> argparse.ArgumentParser:
301
336
  parser = argparse.ArgumentParser(
302
337
  prog='sqlcc',
303
- description='Calculate the cyclomatic complexity of SQL files, e.g. the models dbt compiles to target/compiled, '
304
- 'or with --crap the CRAP score of the models in a dbt manifest.',
338
+ description='Calculate the cyclomatic complexity and the lines of code of SQL files, e.g. the models dbt '
339
+ 'compiles to target/compiled, or with --crap the CRAP score of the models in a dbt manifest.',
305
340
  epilog='Exit status: 0 when everything is scored and nothing exceeds --max or --max-crap, '
306
341
  '1 when something exceeds them, '
307
342
  '2 when something can not be scored or the arguments are invalid.',
@@ -331,9 +366,9 @@ def _exceeding(values: dict, limit: float | None) -> dict:
331
366
  return {key: value for key, value in values.items() if value > limit}
332
367
 
333
368
 
334
- def _print_scores(scores: dict[Path, int]) -> None:
369
+ def _print_scores(scores: dict[Path, int], locs: dict[Path, int]) -> None:
335
370
  for file, score in sorted(scores.items(), key=lambda item: (-item[1], str(item[0]))):
336
- print(f"{score:>6} {file}")
371
+ print(f"{score:>6} {locs[file]:>6} {file}")
337
372
  sys.stdout.flush()
338
373
 
339
374
 
@@ -351,9 +386,9 @@ def _print_diagnostics(scores: dict[Path, int], failed: dict[Path, str], skipped
351
386
 
352
387
 
353
388
  def _print_models(scores: list[ModelScore]) -> None:
354
- print(f"{'CRAP':>9} {'COMP':>5} {'COV%':>6} {'TESTS':>5} {'MISSING':>7} MODEL")
389
+ print(f"{'CRAP':>9} {'COMP':>5} {'LOC':>6} {'COV%':>6} {'TESTS':>5} {'MISSING':>7} MODEL")
355
390
  for score in sorted(scores, key=lambda score: (-score.crap, score.name)):
356
- print(f"{score.crap:>9.1f} {score.complexity:>5} {score.coverage:>6.1f} {score.tests:>5} "
391
+ print(f"{score.crap:>9.1f} {score.complexity:>5} {score.loc:>6} {score.coverage:>6.1f} {score.tests:>5} "
357
392
  f"{score.missing:>7} {score.name}")
358
393
  sys.stdout.flush()
359
394
 
@@ -395,9 +430,9 @@ def _run_files(args: argparse.Namespace) -> int:
395
430
  print(f"sqlcc: error: {e}", file=sys.stderr)
396
431
  return 2
397
432
 
398
- scores, failed, skipped = _score_files(files, dialect)
433
+ scores, locs, failed, skipped = _score_files(files, dialect)
399
434
  exceeding = _exceeding(scores, args.max_score)
400
- _print_scores(scores)
435
+ _print_scores(scores, locs)
401
436
  _print_diagnostics(scores, failed, skipped, exceeding, args.max_score)
402
437
  if failed:
403
438
  return 2
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: sqlccc
3
- Version: 1.3.0
3
+ Version: 1.4.0
4
4
  Summary: A cyclomatic complexity calculator for SQL
5
5
  Author-email: Søren Brønsted <soren@bronsted.dk>
6
6
  License-Expression: Apache-2.0
@@ -44,6 +44,10 @@ The SQL can contain several statements, in which case the complexity of all of t
44
44
  * `ValueError` if the SQL contains no statements, or the dialect is unknown
45
45
  * `sqlglot.errors.ParseError` if the SQL can not be parsed
46
46
 
47
+ `loc(sql, dialect='snowflake')` returns the lines of code, which are the lines with something on them besides
48
+ whitespace and comments. A string which spans several lines counts every line it is on. It raises `TypeError` if the SQL
49
+ is not a string, `ValueError` if the dialect is unknown and `sqlglot.errors.TokenError` if the SQL can not be tokenized.
50
+
47
51
  `crap(complexity, coverage)` returns the [CRAP1 score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
48
52
  `complexity^2 * (1 - coverage/100)^3 + complexity`, where the coverage is in percent. It is never lower than the
49
53
  complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from a number of tests, see
@@ -54,10 +58,11 @@ complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from
54
58
  ```
55
59
  sqlcc target/compiled/my_project/models
56
60
  ```
57
- prints the complexity of every `.sql` file found in the directories (or of the files) given, the highest first:
61
+ prints the complexity and the lines of code (see `loc`) of every `.sql` file found in the directories (or of the files)
62
+ given, the highest complexity first:
58
63
  ```
59
- 8 target/compiled/my_project/models/marts/fct_orders.sql
60
- 2 target/compiled/my_project/models/staging/stg_orders.sql
64
+ 8 120 target/compiled/my_project/models/marts/fct_orders.sql
65
+ 2 35 target/compiled/my_project/models/staging/stg_orders.sql
61
66
  ```
62
67
  followed, on stderr, by a summary. `python -m sqlcc` does the same.
63
68
 
@@ -99,13 +104,13 @@ sqlcc --crap --manifest /tmp/dbt_sqlcc/manifest.json --max-crap 100
99
104
  ```
100
105
  It prints the models with the highest score first:
101
106
  ```
102
- CRAP COMP COV% TESTS MISSING MODEL
103
- 24.9 7 28.6 2 5 model.shop.fct_orders
104
- 2.0 2 100.0 3 0 model.shop.stg_customers
105
- 2.0 2 100.0 5 0 model.shop.stg_orders
107
+ CRAP COMP LOC COV% TESTS MISSING MODEL
108
+ 24.9 7 42 28.6 2 5 model.shop.fct_orders
109
+ 2.0 2 18 100.0 3 0 model.shop.stg_customers
110
+ 2.0 2 15 100.0 5 0 model.shop.stg_orders
106
111
  ```
107
- `COMP` is the complexity of the compiled SQL, `COV%` the estimated coverage, `TESTS` the number of tests and `MISSING`
108
- the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
112
+ `COMP` is the complexity of the compiled SQL, `LOC` its lines of code, `COV%` the estimated coverage, `TESTS` the number
113
+ of tests and `MISSING` the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
109
114
 
110
115
  **The coverage is an estimate.** CRAP is defined on the basis path coverage of the automated tests, but a dbt test
111
116
  asserts something about the data and does not run the paths of the SQL. The complexity is the number of tests it takes
File without changes
File without changes