sqlccc 1.2.0__tar.gz → 1.4.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: sqlccc
3
- Version: 1.2.0
3
+ Version: 1.4.0
4
4
  Summary: A cyclomatic complexity calculator for SQL
5
5
  Author-email: Søren Brønsted <soren@bronsted.dk>
6
6
  License-Expression: Apache-2.0
@@ -44,6 +44,10 @@ The SQL can contain several statements, in which case the complexity of all of t
44
44
  * `ValueError` if the SQL contains no statements, or the dialect is unknown
45
45
  * `sqlglot.errors.ParseError` if the SQL can not be parsed
46
46
 
47
+ `loc(sql, dialect='snowflake')` returns the lines of code, which are the lines with something on them besides
48
+ whitespace and comments. A string which spans several lines counts every line it is on. It raises `TypeError` if the SQL
49
+ is not a string, `ValueError` if the dialect is unknown and `sqlglot.errors.TokenError` if the SQL can not be tokenized.
50
+
47
51
  `crap(complexity, coverage)` returns the [CRAP1 score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
48
52
  `complexity^2 * (1 - coverage/100)^3 + complexity`, where the coverage is in percent. It is never lower than the
49
53
  complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from a number of tests, see
@@ -54,10 +58,11 @@ complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from
54
58
  ```
55
59
  sqlcc target/compiled/my_project/models
56
60
  ```
57
- prints the complexity of every `.sql` file found in the directories (or of the files) given, the highest first:
61
+ prints the complexity and the lines of code (see `loc`) of every `.sql` file found in the directories (or of the files)
62
+ given, the highest complexity first:
58
63
  ```
59
- 8 target/compiled/my_project/models/marts/fct_orders.sql
60
- 2 target/compiled/my_project/models/staging/stg_orders.sql
64
+ 8 120 target/compiled/my_project/models/marts/fct_orders.sql
65
+ 2 35 target/compiled/my_project/models/staging/stg_orders.sql
61
66
  ```
62
67
  followed, on stderr, by a summary. `python -m sqlcc` does the same.
63
68
 
@@ -99,13 +104,13 @@ sqlcc --crap --manifest /tmp/dbt_sqlcc/manifest.json --max-crap 100
99
104
  ```
100
105
  It prints the models with the highest score first:
101
106
  ```
102
- CRAP COMP COV% TESTS MISSING MODEL
103
- 24.9 7 28.6 2 5 model.shop.fct_orders
104
- 2.0 2 100.0 3 0 model.shop.stg_customers
105
- 2.0 2 100.0 5 0 model.shop.stg_orders
107
+ CRAP COMP LOC COV% TESTS MISSING MODEL
108
+ 24.9 7 42 28.6 2 5 model.shop.fct_orders
109
+ 2.0 2 18 100.0 3 0 model.shop.stg_customers
110
+ 2.0 2 15 100.0 5 0 model.shop.stg_orders
106
111
  ```
107
- `COMP` is the complexity of the compiled SQL, `COV%` the estimated coverage, `TESTS` the number of tests and `MISSING`
108
- the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
112
+ `COMP` is the complexity of the compiled SQL, `LOC` its lines of code, `COV%` the estimated coverage, `TESTS` the number
113
+ of tests and `MISSING` the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
109
114
 
110
115
  **The coverage is an estimate.** CRAP is defined on the basis path coverage of the automated tests, but a dbt test
111
116
  asserts something about the data and does not run the paths of the SQL. The complexity is the number of tests it takes
@@ -152,16 +157,20 @@ The complexity starts at 1 and each of the following adds to it:
152
157
 
153
158
  | Construct | Adds |
154
159
  |--------------------------------------------------------------------------------------------------|----------------------------|
155
- | `WHERE` | 1 per clause |
160
+ | `WHERE`, `HAVING`, `QUALIFY` | 1 per clause |
156
161
  | `AND`, `OR` | 1 each |
157
- | `IN`, `BETWEEN`, `LIKE`, `ILIKE` | 1 each |
162
+ | `IN`, `BETWEEN`, `LIKE`, `ILIKE`, `SIMILAR TO`, `ANY`, `ALL` | 1 each |
158
163
  | `GROUP BY` | 1 |
159
164
  | `WHEN` branch of a `CASE`, and `IF`, `IFF` (the `CASE` itself adds nothing) | 1 per branch or call |
160
165
  | `COALESCE`, `NVL`, `IFNULL`, `NULLIF`, `NVL2`, `DECODE`, `GREATEST`, `LEAST`, `EXISTS` | 1 per call |
161
- | `TRY_CAST`, `SAFE_CAST` | 1 per call |
162
- | `REGEXP_LIKE`, `RLIKE`, `CONTAINS`, `STARTSWITH`, `ENDSWITH`, `ARRAY_CONTAINS`, `COUNT_IF` | 1 per call |
166
+ | `LAG`, `LEAD` | 1 per call |
167
+ | `TRY_CAST`, `SAFE_CAST`, `SAFE_DIVIDE`, `TRY_TO_NUMBER`, `TRY_CONVERT` and other `TRY_`/`SAFE_` | 1 per call |
168
+ | `REGEXP_LIKE`, `RLIKE`, `CONTAINS`, `STARTSWITH`, `ENDSWITH`, `ARRAY_CONTAINS`, `ARRAYS_OVERLAP` | 1 per call |
169
+ | `COUNT_IF` | 1 per call |
163
170
  | `JOIN` of any kind, including a comma join | 1 per join |
164
171
  | `UNION`, `INTERSECT`, `EXCEPT` | 1 per operator |
172
+ | `WITH RECURSIVE`, `CONNECT BY` | 1 each |
173
+ | `WHEN` of a `MERGE` or multi table `INSERT`, `ON CONFLICT`, `ON DUPLICATE KEY` | 1 each |
165
174
 
166
175
  Everything is counted anywhere in the SQL, including subqueries and CTEs. A construct is only counted once.
167
176
 
@@ -172,15 +181,30 @@ adds 1 for the join and 1 for the `AND`.
172
181
  **A function counts when it branches**, which means that the path through the SQL depends on the data, and a test is
173
182
  needed for each path. `COALESCE` has a path for a NULL and one for a value. `CAST`, `LOWER`, `TRIM`, `CONCAT` and
174
183
  `SUBSTRING` do the same whatever the data is, and so they add nothing. Neither do the aggregates (`COUNT`, `SUM`,
175
- `MAX`, ...) and the window functions, as they change the grain and do not branch, and `GROUP BY` counts on its own.
176
- `TRY_CAST` counts, because a value which can not be converted gives NULL, which is a path of its own. A function
177
- which sqlglot does not know adds nothing either, as nothing can be said about it.
178
-
179
- `HAVING` adds nothing itself. Its expression can have any length, so it is counted by what it contains, like `AND`
180
- and `BETWEEN`. That is also why `HAVING COUNT(*) BETWEEN 1 AND 3` and `HAVING COUNT(*) >= 1 AND COUNT(*) <= 3` get the
181
- same complexity.
182
-
183
- Not counted (yet): comparisons, subqueries, CTEs, window functions in themselves and `QUALIFY`.
184
+ `MAX`, ...), as they change the grain and do not branch, and `GROUP BY` counts on its own.
185
+ `TRY_CAST` counts, because a value which can not be converted gives NULL, which is a path of its own. So do the other
186
+ functions which give NULL instead of an error, like `TRY_TO_NUMBER`, `TRY_CONVERT`, `TRY_PARSE_JSON` and `SAFE_DIVIDE`,
187
+ also when sqlglot does not know them but their name starts with `TRY_` or `SAFE_`. Any other function which sqlglot does
188
+ not know adds nothing, as nothing can be said about it.
189
+
190
+ **A clause which filters rows counts**, and `WHERE`, `HAVING` and `QUALIFY` do. The expression of the filter can have
191
+ any length, so it is also counted by what it contains, like `AND` and `BETWEEN`. That is why
192
+ `HAVING COUNT(*) BETWEEN 1 AND 3` and `HAVING COUNT(*) >= 1 AND COUNT(*) <= 3` get the same complexity.
193
+
194
+ **A window function does not filter and does not branch**, except `LAG` and `LEAD`: the first or last row of a
195
+ partition has no neighbour, which gives NULL or the default, so they count 1 per call. `ROW_NUMBER`, `RANK`, `SUM() OVER`
196
+ and the like add nothing, and a filter on them is counted by the filter. `QUALIFY ROW_NUMBER() OVER (...) = 1` and
197
+ `SELECT ... FROM (SELECT ROW_NUMBER() OVER (...) AS rn ...) WHERE rn = 1` therefore get the same complexity.
198
+
199
+ **A loop counts**: a recursive CTE (`WITH RECURSIVE`) and `CONNECT BY` repeat until nothing is left, and count 1 each, in
200
+ addition to the `UNION` and the `WHERE` they have.
201
+
202
+ **A statement which branches counts**: each `WHEN` of a `MERGE` (`MATCHED`, `NOT MATCHED`) and of a multi table
203
+ `INSERT ALL WHEN` counts 1, but the `ELSE` of the `INSERT` does not, like the `ELSE` of a `CASE`. `ON CONFLICT` and
204
+ `ON DUPLICATE KEY` count 1. The `IN` of a `PIVOT` lists the columns and is not a predicate, so it does not count, and
205
+ `LIKE ANY (...)` counts as the `LIKE` and not as an `ANY` too.
206
+
207
+ Not counted (yet): comparisons, also `IS NULL` and `IS DISTINCT FROM`, subqueries and CTEs.
184
208
  The weights are defined in `WEIGHTS` in `sqlcc.py`. `COALESCE` counts 1 however many arguments it has.
185
209
 
186
210
  Which functions sqlglot knows depends on its version, and an unknown function adds nothing. For example, `IFF` in the
@@ -28,6 +28,10 @@ The SQL can contain several statements, in which case the complexity of all of t
28
28
  * `ValueError` if the SQL contains no statements, or the dialect is unknown
29
29
  * `sqlglot.errors.ParseError` if the SQL can not be parsed
30
30
 
31
+ `loc(sql, dialect='snowflake')` returns the lines of code, which are the lines with something on them besides
32
+ whitespace and comments. A string which spans several lines counts every line it is on. It raises `TypeError` if the SQL
33
+ is not a string, `ValueError` if the dialect is unknown and `sqlglot.errors.TokenError` if the SQL can not be tokenized.
34
+
31
35
  `crap(complexity, coverage)` returns the [CRAP1 score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
32
36
  `complexity^2 * (1 - coverage/100)^3 + complexity`, where the coverage is in percent. It is never lower than the
33
37
  complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from a number of tests, see
@@ -38,10 +42,11 @@ complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from
38
42
  ```
39
43
  sqlcc target/compiled/my_project/models
40
44
  ```
41
- prints the complexity of every `.sql` file found in the directories (or of the files) given, the highest first:
45
+ prints the complexity and the lines of code (see `loc`) of every `.sql` file found in the directories (or of the files)
46
+ given, the highest complexity first:
42
47
  ```
43
- 8 target/compiled/my_project/models/marts/fct_orders.sql
44
- 2 target/compiled/my_project/models/staging/stg_orders.sql
48
+ 8 120 target/compiled/my_project/models/marts/fct_orders.sql
49
+ 2 35 target/compiled/my_project/models/staging/stg_orders.sql
45
50
  ```
46
51
  followed, on stderr, by a summary. `python -m sqlcc` does the same.
47
52
 
@@ -83,13 +88,13 @@ sqlcc --crap --manifest /tmp/dbt_sqlcc/manifest.json --max-crap 100
83
88
  ```
84
89
  It prints the models with the highest score first:
85
90
  ```
86
- CRAP COMP COV% TESTS MISSING MODEL
87
- 24.9 7 28.6 2 5 model.shop.fct_orders
88
- 2.0 2 100.0 3 0 model.shop.stg_customers
89
- 2.0 2 100.0 5 0 model.shop.stg_orders
91
+ CRAP COMP LOC COV% TESTS MISSING MODEL
92
+ 24.9 7 42 28.6 2 5 model.shop.fct_orders
93
+ 2.0 2 18 100.0 3 0 model.shop.stg_customers
94
+ 2.0 2 15 100.0 5 0 model.shop.stg_orders
90
95
  ```
91
- `COMP` is the complexity of the compiled SQL, `COV%` the estimated coverage, `TESTS` the number of tests and `MISSING`
92
- the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
96
+ `COMP` is the complexity of the compiled SQL, `LOC` its lines of code, `COV%` the estimated coverage, `TESTS` the number
97
+ of tests and `MISSING` the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
93
98
 
94
99
  **The coverage is an estimate.** CRAP is defined on the basis path coverage of the automated tests, but a dbt test
95
100
  asserts something about the data and does not run the paths of the SQL. The complexity is the number of tests it takes
@@ -136,16 +141,20 @@ The complexity starts at 1 and each of the following adds to it:
136
141
 
137
142
  | Construct | Adds |
138
143
  |--------------------------------------------------------------------------------------------------|----------------------------|
139
- | `WHERE` | 1 per clause |
144
+ | `WHERE`, `HAVING`, `QUALIFY` | 1 per clause |
140
145
  | `AND`, `OR` | 1 each |
141
- | `IN`, `BETWEEN`, `LIKE`, `ILIKE` | 1 each |
146
+ | `IN`, `BETWEEN`, `LIKE`, `ILIKE`, `SIMILAR TO`, `ANY`, `ALL` | 1 each |
142
147
  | `GROUP BY` | 1 |
143
148
  | `WHEN` branch of a `CASE`, and `IF`, `IFF` (the `CASE` itself adds nothing) | 1 per branch or call |
144
149
  | `COALESCE`, `NVL`, `IFNULL`, `NULLIF`, `NVL2`, `DECODE`, `GREATEST`, `LEAST`, `EXISTS` | 1 per call |
145
- | `TRY_CAST`, `SAFE_CAST` | 1 per call |
146
- | `REGEXP_LIKE`, `RLIKE`, `CONTAINS`, `STARTSWITH`, `ENDSWITH`, `ARRAY_CONTAINS`, `COUNT_IF` | 1 per call |
150
+ | `LAG`, `LEAD` | 1 per call |
151
+ | `TRY_CAST`, `SAFE_CAST`, `SAFE_DIVIDE`, `TRY_TO_NUMBER`, `TRY_CONVERT` and other `TRY_`/`SAFE_` | 1 per call |
152
+ | `REGEXP_LIKE`, `RLIKE`, `CONTAINS`, `STARTSWITH`, `ENDSWITH`, `ARRAY_CONTAINS`, `ARRAYS_OVERLAP` | 1 per call |
153
+ | `COUNT_IF` | 1 per call |
147
154
  | `JOIN` of any kind, including a comma join | 1 per join |
148
155
  | `UNION`, `INTERSECT`, `EXCEPT` | 1 per operator |
156
+ | `WITH RECURSIVE`, `CONNECT BY` | 1 each |
157
+ | `WHEN` of a `MERGE` or multi table `INSERT`, `ON CONFLICT`, `ON DUPLICATE KEY` | 1 each |
149
158
 
150
159
  Everything is counted anywhere in the SQL, including subqueries and CTEs. A construct is only counted once.
151
160
 
@@ -156,15 +165,30 @@ adds 1 for the join and 1 for the `AND`.
156
165
  **A function counts when it branches**, which means that the path through the SQL depends on the data, and a test is
157
166
  needed for each path. `COALESCE` has a path for a NULL and one for a value. `CAST`, `LOWER`, `TRIM`, `CONCAT` and
158
167
  `SUBSTRING` do the same whatever the data is, and so they add nothing. Neither do the aggregates (`COUNT`, `SUM`,
159
- `MAX`, ...) and the window functions, as they change the grain and do not branch, and `GROUP BY` counts on its own.
160
- `TRY_CAST` counts, because a value which can not be converted gives NULL, which is a path of its own. A function
161
- which sqlglot does not know adds nothing either, as nothing can be said about it.
162
-
163
- `HAVING` adds nothing itself. Its expression can have any length, so it is counted by what it contains, like `AND`
164
- and `BETWEEN`. That is also why `HAVING COUNT(*) BETWEEN 1 AND 3` and `HAVING COUNT(*) >= 1 AND COUNT(*) <= 3` get the
165
- same complexity.
166
-
167
- Not counted (yet): comparisons, subqueries, CTEs, window functions in themselves and `QUALIFY`.
168
+ `MAX`, ...), as they change the grain and do not branch, and `GROUP BY` counts on its own.
169
+ `TRY_CAST` counts, because a value which can not be converted gives NULL, which is a path of its own. So do the other
170
+ functions which give NULL instead of an error, like `TRY_TO_NUMBER`, `TRY_CONVERT`, `TRY_PARSE_JSON` and `SAFE_DIVIDE`,
171
+ also when sqlglot does not know them but their name starts with `TRY_` or `SAFE_`. Any other function which sqlglot does
172
+ not know adds nothing, as nothing can be said about it.
173
+
174
+ **A clause which filters rows counts**, and `WHERE`, `HAVING` and `QUALIFY` do. The expression of the filter can have
175
+ any length, so it is also counted by what it contains, like `AND` and `BETWEEN`. That is why
176
+ `HAVING COUNT(*) BETWEEN 1 AND 3` and `HAVING COUNT(*) >= 1 AND COUNT(*) <= 3` get the same complexity.
177
+
178
+ **A window function does not filter and does not branch**, except `LAG` and `LEAD`: the first or last row of a
179
+ partition has no neighbour, which gives NULL or the default, so they count 1 per call. `ROW_NUMBER`, `RANK`, `SUM() OVER`
180
+ and the like add nothing, and a filter on them is counted by the filter. `QUALIFY ROW_NUMBER() OVER (...) = 1` and
181
+ `SELECT ... FROM (SELECT ROW_NUMBER() OVER (...) AS rn ...) WHERE rn = 1` therefore get the same complexity.
182
+
183
+ **A loop counts**: a recursive CTE (`WITH RECURSIVE`) and `CONNECT BY` repeat until nothing is left, and count 1 each, in
184
+ addition to the `UNION` and the `WHERE` they have.
185
+
186
+ **A statement which branches counts**: each `WHEN` of a `MERGE` (`MATCHED`, `NOT MATCHED`) and of a multi table
187
+ `INSERT ALL WHEN` counts 1, but the `ELSE` of the `INSERT` does not, like the `ELSE` of a `CASE`. `ON CONFLICT` and
188
+ `ON DUPLICATE KEY` count 1. The `IN` of a `PIVOT` lists the columns and is not a predicate, so it does not count, and
189
+ `LIKE ANY (...)` counts as the `LIKE` and not as an `ANY` too.
190
+
191
+ Not counted (yet): comparisons, also `IS NULL` and `IS DISTINCT FROM`, subqueries and CTEs.
168
192
  The weights are defined in `WEIGHTS` in `sqlcc.py`. `COALESCE` counts 1 however many arguments it has.
169
193
 
170
194
  Which functions sqlglot knows depends on its version, and an unknown function adds nothing. For example, `IFF` in the
@@ -1,6 +1,6 @@
1
1
  [project]
2
2
  name = "sqlccc"
3
- version = "1.2.0"
3
+ version = "1.4.0"
4
4
  authors = [
5
5
  { name="Søren Brønsted", email="soren@bronsted.dk" },
6
6
  ]
@@ -1,8 +1,10 @@
1
1
  import argparse
2
+ import bisect
2
3
  import json
3
4
  import os
4
5
  import sys
5
6
  from collections import Counter
7
+ from collections.abc import Callable
6
8
  from dataclasses import dataclass
7
9
  from pathlib import Path
8
10
 
@@ -15,32 +17,64 @@ MAX_SKIPPED_LISTED = 5
15
17
  # dbt adapter types that are named differently from the sqlglot dialect
16
18
  ADAPTER_DIALECTS = {'sqlserver': 'tsql'}
17
19
 
20
+
21
+ def _if_arg(name: str) -> Callable[[sg.exp.Expression], int]:
22
+ return lambda node: 1 if node.args.get(name) else 0
23
+
24
+
25
+ def _unless_parent(*classes: type) -> Callable[[sg.exp.Expression], int]:
26
+ return lambda node: 0 if isinstance(node.parent, classes) else 1
27
+
28
+
29
+ def _named(*prefixes: str) -> Callable[[sg.exp.Expression], int]:
30
+ return lambda node: 1 if node.name.upper().startswith(prefixes) else 0
31
+
32
+
18
33
  # The weight each construct adds to the complexity. A node gets the weight of the most specific class
19
34
  # in its inheritance chain, so a Union counts as the SetOperation it is, and a node which has no class
20
- # in WEIGHTS adds nothing.
35
+ # in WEIGHTS adds nothing. The weight is a number, or a function of the node when it depends on more
36
+ # than the class.
21
37
  WEIGHTS = {
22
- # Logic: HAVING is not weighted, its expression varies in length and is counted by what it contains
23
- sg.exp.Where: 1, sg.exp.And: 1, sg.exp.Or: 1, sg.exp.In: 1, sg.exp.Between: 1,
24
- sg.exp.Like: 1, sg.exp.ILike: 1, sg.exp.Group: 1,
38
+ # Logic: a filter (WHERE, HAVING, QUALIFY) counts 1, and its expression is counted by what it contains.
39
+ # The IN of a PIVOT lists the columns, and LIKE ANY counts as the LIKE
40
+ sg.exp.Where: 1, sg.exp.Having: 1, sg.exp.Qualify: 1, sg.exp.And: 1, sg.exp.Or: 1,
41
+ sg.exp.In: _unless_parent(sg.exp.Pivot), sg.exp.Between: 1, sg.exp.Like: 1, sg.exp.ILike: 1,
42
+ sg.exp.SimilarTo: 1, sg.exp.Group: 1,
43
+ sg.exp.Any: _unless_parent(sg.exp.Like, sg.exp.ILike, sg.exp.SimilarTo),
44
+ sg.exp.All: _unless_parent(sg.exp.Like, sg.exp.ILike, sg.exp.SimilarTo),
25
45
  # Functions: only the ones which branch count, and a CASE through its WHEN branches (sg.exp.If). The others,
26
46
  # like CAST and LOWER, take the same path whatever the data is. So do the ones sqlglot does not know
27
- # (sg.exp.Anonymous), as nothing can be said about them.
47
+ # (sg.exp.Anonymous), as nothing can be said about them, except the ones which are named TRY_ or SAFE_
28
48
  sg.exp.If: 1, sg.exp.Coalesce: 1, sg.exp.Nullif: 1, sg.exp.Nvl2: 1, sg.exp.DecodeCase: 1,
29
49
  sg.exp.Greatest: 1, sg.exp.Least: 1, sg.exp.Exists: 1,
30
- # A conversion which fails gives NULL, and that is a path of its own
31
- sg.exp.TryCast: 1,
50
+ # The first or last row of a partition has no neighbour, which gives NULL or the default. The other window
51
+ # functions, like ROW_NUMBER, do not branch, and a filter on them is counted by the filter
52
+ sg.exp.Lag: 1, sg.exp.Lead: 1,
53
+ # A conversion which fails gives NULL, and that is a path of its own. sqlglot sets safe on the ones which do,
54
+ # like TRY_TO_NUMBER, TRY_CONVERT and TRY_PARSE_JSON, but also on CONCAT, which is only about NULL arguments
55
+ sg.exp.TryCast: 1, sg.exp.SafeDivide: 1, sg.exp.Func: _if_arg('safe'), sg.exp.Concat: 0,
56
+ sg.exp.Anonymous: _named('TRY_', 'SAFE_'),
32
57
  # Functions which are predicates, like LIKE and IN
33
58
  sg.exp.RegexpLike: 1, sg.exp.RegexpILike: 1, sg.exp.Contains: 1, sg.exp.StartsWith: 1,
34
- sg.exp.EndsWith: 1, sg.exp.ArrayContains: 1, sg.exp.CountIf: 1,
59
+ sg.exp.EndsWith: 1, sg.exp.ArrayContains: 1, sg.exp.ArrayOverlaps: 1, sg.exp.CountIf: 1,
35
60
  # Sets: SetOperation is UNION, INTERSECT and EXCEPT
36
61
  sg.exp.Join: 1, sg.exp.SetOperation: 1,
62
+ # Loops: a recursive CTE, and CONNECT BY
63
+ sg.exp.With: _if_arg('recursive'), sg.exp.Connect: 1,
64
+ # Branches of a statement: WHEN of MERGE, WHEN of a multi table INSERT (not its ELSE), and ON CONFLICT
65
+ # or ON DUPLICATE KEY
66
+ sg.exp.When: 1, sg.exp.ConditionalInsert: _if_arg('expression'), sg.exp.OnConflict: 1,
37
67
  }
68
+ # SAFE.f() of BigQuery returns NULL for an error. It is only known to sqlglot from a later version
69
+ if hasattr(sg.exp, 'SafeFunc'):
70
+ WEIGHTS[sg.exp.SafeFunc] = 1
38
71
 
39
72
 
40
73
  def _weight(node: sg.exp.Expression) -> int:
41
74
  for cls in type(node).__mro__:
42
75
  if cls in WEIGHTS:
43
- return WEIGHTS[cls]
76
+ weight = WEIGHTS[cls]
77
+ return weight(node) if callable(weight) else weight
44
78
  return 0
45
79
 
46
80
 
@@ -71,6 +105,34 @@ def calc(sql: str, dialect: str = DEFAULT_DIALECT) -> int:
71
105
  return 1 + sum(_weight(node) for statement in statements for node in statement.walk())
72
106
 
73
107
 
108
+ def loc(sql: str, dialect: str = DEFAULT_DIALECT) -> int:
109
+ """
110
+ Count the lines of code in SQL, which are the lines with something on them besides whitespace and comments.
111
+
112
+ A token which spans several lines, like a string, counts every line it is on.
113
+
114
+ Args:
115
+ sql (str): The SQL string, with any number of statements.
116
+ dialect (str): The SQL dialect to use for tokenizing. See sqlglot documentation for supported dialects.
117
+
118
+ Returns:
119
+ int: The number of lines of code, 0 if there is only whitespace and comments.
120
+
121
+ Raises:
122
+ TypeError: If sql is not a string.
123
+ ValueError: If the dialect is unknown.
124
+ sqlglot.errors.TokenError: If sql can not be tokenized with the dialect.
125
+ """
126
+ if not isinstance(sql, str):
127
+ raise TypeError(f"sql must be a str, not {type(sql).__name__}")
128
+ line_starts = [0] + [index + 1 for index, char in enumerate(sql) if char == '\n']
129
+ lines = set()
130
+ for token in sg.tokenize(sql, read=dialect):
131
+ first, last = (bisect.bisect_right(line_starts, offset) for offset in (token.start, token.end))
132
+ lines.update(range(first, last + 1))
133
+ return len(lines)
134
+
135
+
74
136
  def crap(complexity: int, coverage: float) -> float:
75
137
  """
76
138
  Calculate CRAP1, the change risk anti-pattern score: complexity^2 * (1 - coverage/100)^3 + complexity.
@@ -117,6 +179,7 @@ class ModelScore:
117
179
  name: str
118
180
  complexity: int
119
181
  tests: int
182
+ loc: int
120
183
 
121
184
  @property
122
185
  def coverage(self) -> float:
@@ -196,16 +259,19 @@ def _first_line(e: Exception) -> str:
196
259
  return (str(e).splitlines() or [type(e).__name__])[0]
197
260
 
198
261
 
199
- def _score_files(files: list[Path], dialect: str) -> tuple[dict[Path, int], dict[Path, str], list[Path]]:
200
- scores, failed, skipped = {}, {}, []
262
+ def _score_files(
263
+ files: list[Path], dialect: str
264
+ ) -> tuple[dict[Path, int], dict[Path, int], dict[Path, str], list[Path]]:
265
+ scores, locs, failed, skipped = {}, {}, {}, []
201
266
  for file in files:
202
267
  try:
203
- scores[file] = calc(file.read_text(encoding='utf-8'), dialect=dialect)
268
+ sql = file.read_text(encoding='utf-8')
269
+ scores[file], locs[file] = calc(sql, dialect=dialect), loc(sql, dialect=dialect)
204
270
  except (OSError, UnicodeDecodeError, sg.errors.SqlglotError) as e:
205
271
  failed[file] = _first_line(e)
206
272
  except ValueError:
207
273
  skipped.append(file)
208
- return scores, failed, skipped
274
+ return scores, locs, failed, skipped
209
275
 
210
276
 
211
277
  def _read_manifest(path: Path) -> dict:
@@ -254,7 +320,9 @@ def _score_models(manifest: dict, dialect: str) -> tuple[list[ModelScore], dict[
254
320
  scores, failed, skipped = [], {}, {}
255
321
  for unique_id, node in _models(manifest).items():
256
322
  try:
257
- scores.append(ModelScore(unique_id, calc(_compiled_sql(node), dialect=dialect), tests[unique_id]))
323
+ sql = _compiled_sql(node)
324
+ scores.append(ModelScore(unique_id, calc(sql, dialect=dialect), tests[unique_id],
325
+ loc(sql, dialect=dialect)))
258
326
  except _Skipped as e:
259
327
  skipped[unique_id] = str(e)
260
328
  except sg.errors.SqlglotError as e:
@@ -267,8 +335,8 @@ def _score_models(manifest: dict, dialect: str) -> tuple[list[ModelScore], dict[
267
335
  def _parser() -> argparse.ArgumentParser:
268
336
  parser = argparse.ArgumentParser(
269
337
  prog='sqlcc',
270
- description='Calculate the cyclomatic complexity of SQL files, e.g. the models dbt compiles to target/compiled, '
271
- 'or with --crap the CRAP score of the models in a dbt manifest.',
338
+ description='Calculate the cyclomatic complexity and the lines of code of SQL files, e.g. the models dbt '
339
+ 'compiles to target/compiled, or with --crap the CRAP score of the models in a dbt manifest.',
272
340
  epilog='Exit status: 0 when everything is scored and nothing exceeds --max or --max-crap, '
273
341
  '1 when something exceeds them, '
274
342
  '2 when something can not be scored or the arguments are invalid.',
@@ -298,9 +366,9 @@ def _exceeding(values: dict, limit: float | None) -> dict:
298
366
  return {key: value for key, value in values.items() if value > limit}
299
367
 
300
368
 
301
- def _print_scores(scores: dict[Path, int]) -> None:
369
+ def _print_scores(scores: dict[Path, int], locs: dict[Path, int]) -> None:
302
370
  for file, score in sorted(scores.items(), key=lambda item: (-item[1], str(item[0]))):
303
- print(f"{score:>6} {file}")
371
+ print(f"{score:>6} {locs[file]:>6} {file}")
304
372
  sys.stdout.flush()
305
373
 
306
374
 
@@ -318,9 +386,9 @@ def _print_diagnostics(scores: dict[Path, int], failed: dict[Path, str], skipped
318
386
 
319
387
 
320
388
  def _print_models(scores: list[ModelScore]) -> None:
321
- print(f"{'CRAP':>9} {'COMP':>5} {'COV%':>6} {'TESTS':>5} {'MISSING':>7} MODEL")
389
+ print(f"{'CRAP':>9} {'COMP':>5} {'LOC':>6} {'COV%':>6} {'TESTS':>5} {'MISSING':>7} MODEL")
322
390
  for score in sorted(scores, key=lambda score: (-score.crap, score.name)):
323
- print(f"{score.crap:>9.1f} {score.complexity:>5} {score.coverage:>6.1f} {score.tests:>5} "
391
+ print(f"{score.crap:>9.1f} {score.complexity:>5} {score.loc:>6} {score.coverage:>6.1f} {score.tests:>5} "
324
392
  f"{score.missing:>7} {score.name}")
325
393
  sys.stdout.flush()
326
394
 
@@ -362,9 +430,9 @@ def _run_files(args: argparse.Namespace) -> int:
362
430
  print(f"sqlcc: error: {e}", file=sys.stderr)
363
431
  return 2
364
432
 
365
- scores, failed, skipped = _score_files(files, dialect)
433
+ scores, locs, failed, skipped = _score_files(files, dialect)
366
434
  exceeding = _exceeding(scores, args.max_score)
367
- _print_scores(scores)
435
+ _print_scores(scores, locs)
368
436
  _print_diagnostics(scores, failed, skipped, exceeding, args.max_score)
369
437
  if failed:
370
438
  return 2
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: sqlccc
3
- Version: 1.2.0
3
+ Version: 1.4.0
4
4
  Summary: A cyclomatic complexity calculator for SQL
5
5
  Author-email: Søren Brønsted <soren@bronsted.dk>
6
6
  License-Expression: Apache-2.0
@@ -44,6 +44,10 @@ The SQL can contain several statements, in which case the complexity of all of t
44
44
  * `ValueError` if the SQL contains no statements, or the dialect is unknown
45
45
  * `sqlglot.errors.ParseError` if the SQL can not be parsed
46
46
 
47
+ `loc(sql, dialect='snowflake')` returns the lines of code, which are the lines with something on them besides
48
+ whitespace and comments. A string which spans several lines counts every line it is on. It raises `TypeError` if the SQL
49
+ is not a string, `ValueError` if the dialect is unknown and `sqlglot.errors.TokenError` if the SQL can not be tokenized.
50
+
47
51
  `crap(complexity, coverage)` returns the [CRAP1 score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
48
52
  `complexity^2 * (1 - coverage/100)^3 + complexity`, where the coverage is in percent. It is never lower than the
49
53
  complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from a number of tests, see
@@ -54,10 +58,11 @@ complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from
54
58
  ```
55
59
  sqlcc target/compiled/my_project/models
56
60
  ```
57
- prints the complexity of every `.sql` file found in the directories (or of the files) given, the highest first:
61
+ prints the complexity and the lines of code (see `loc`) of every `.sql` file found in the directories (or of the files)
62
+ given, the highest complexity first:
58
63
  ```
59
- 8 target/compiled/my_project/models/marts/fct_orders.sql
60
- 2 target/compiled/my_project/models/staging/stg_orders.sql
64
+ 8 120 target/compiled/my_project/models/marts/fct_orders.sql
65
+ 2 35 target/compiled/my_project/models/staging/stg_orders.sql
61
66
  ```
62
67
  followed, on stderr, by a summary. `python -m sqlcc` does the same.
63
68
 
@@ -99,13 +104,13 @@ sqlcc --crap --manifest /tmp/dbt_sqlcc/manifest.json --max-crap 100
99
104
  ```
100
105
  It prints the models with the highest score first:
101
106
  ```
102
- CRAP COMP COV% TESTS MISSING MODEL
103
- 24.9 7 28.6 2 5 model.shop.fct_orders
104
- 2.0 2 100.0 3 0 model.shop.stg_customers
105
- 2.0 2 100.0 5 0 model.shop.stg_orders
107
+ CRAP COMP LOC COV% TESTS MISSING MODEL
108
+ 24.9 7 42 28.6 2 5 model.shop.fct_orders
109
+ 2.0 2 18 100.0 3 0 model.shop.stg_customers
110
+ 2.0 2 15 100.0 5 0 model.shop.stg_orders
106
111
  ```
107
- `COMP` is the complexity of the compiled SQL, `COV%` the estimated coverage, `TESTS` the number of tests and `MISSING`
108
- the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
112
+ `COMP` is the complexity of the compiled SQL, `LOC` its lines of code, `COV%` the estimated coverage, `TESTS` the number
113
+ of tests and `MISSING` the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
109
114
 
110
115
  **The coverage is an estimate.** CRAP is defined on the basis path coverage of the automated tests, but a dbt test
111
116
  asserts something about the data and does not run the paths of the SQL. The complexity is the number of tests it takes
@@ -152,16 +157,20 @@ The complexity starts at 1 and each of the following adds to it:
152
157
 
153
158
  | Construct | Adds |
154
159
  |--------------------------------------------------------------------------------------------------|----------------------------|
155
- | `WHERE` | 1 per clause |
160
+ | `WHERE`, `HAVING`, `QUALIFY` | 1 per clause |
156
161
  | `AND`, `OR` | 1 each |
157
- | `IN`, `BETWEEN`, `LIKE`, `ILIKE` | 1 each |
162
+ | `IN`, `BETWEEN`, `LIKE`, `ILIKE`, `SIMILAR TO`, `ANY`, `ALL` | 1 each |
158
163
  | `GROUP BY` | 1 |
159
164
  | `WHEN` branch of a `CASE`, and `IF`, `IFF` (the `CASE` itself adds nothing) | 1 per branch or call |
160
165
  | `COALESCE`, `NVL`, `IFNULL`, `NULLIF`, `NVL2`, `DECODE`, `GREATEST`, `LEAST`, `EXISTS` | 1 per call |
161
- | `TRY_CAST`, `SAFE_CAST` | 1 per call |
162
- | `REGEXP_LIKE`, `RLIKE`, `CONTAINS`, `STARTSWITH`, `ENDSWITH`, `ARRAY_CONTAINS`, `COUNT_IF` | 1 per call |
166
+ | `LAG`, `LEAD` | 1 per call |
167
+ | `TRY_CAST`, `SAFE_CAST`, `SAFE_DIVIDE`, `TRY_TO_NUMBER`, `TRY_CONVERT` and other `TRY_`/`SAFE_` | 1 per call |
168
+ | `REGEXP_LIKE`, `RLIKE`, `CONTAINS`, `STARTSWITH`, `ENDSWITH`, `ARRAY_CONTAINS`, `ARRAYS_OVERLAP` | 1 per call |
169
+ | `COUNT_IF` | 1 per call |
163
170
  | `JOIN` of any kind, including a comma join | 1 per join |
164
171
  | `UNION`, `INTERSECT`, `EXCEPT` | 1 per operator |
172
+ | `WITH RECURSIVE`, `CONNECT BY` | 1 each |
173
+ | `WHEN` of a `MERGE` or multi table `INSERT`, `ON CONFLICT`, `ON DUPLICATE KEY` | 1 each |
165
174
 
166
175
  Everything is counted anywhere in the SQL, including subqueries and CTEs. A construct is only counted once.
167
176
 
@@ -172,15 +181,30 @@ adds 1 for the join and 1 for the `AND`.
172
181
  **A function counts when it branches**, which means that the path through the SQL depends on the data, and a test is
173
182
  needed for each path. `COALESCE` has a path for a NULL and one for a value. `CAST`, `LOWER`, `TRIM`, `CONCAT` and
174
183
  `SUBSTRING` do the same whatever the data is, and so they add nothing. Neither do the aggregates (`COUNT`, `SUM`,
175
- `MAX`, ...) and the window functions, as they change the grain and do not branch, and `GROUP BY` counts on its own.
176
- `TRY_CAST` counts, because a value which can not be converted gives NULL, which is a path of its own. A function
177
- which sqlglot does not know adds nothing either, as nothing can be said about it.
178
-
179
- `HAVING` adds nothing itself. Its expression can have any length, so it is counted by what it contains, like `AND`
180
- and `BETWEEN`. That is also why `HAVING COUNT(*) BETWEEN 1 AND 3` and `HAVING COUNT(*) >= 1 AND COUNT(*) <= 3` get the
181
- same complexity.
182
-
183
- Not counted (yet): comparisons, subqueries, CTEs, window functions in themselves and `QUALIFY`.
184
+ `MAX`, ...), as they change the grain and do not branch, and `GROUP BY` counts on its own.
185
+ `TRY_CAST` counts, because a value which can not be converted gives NULL, which is a path of its own. So do the other
186
+ functions which give NULL instead of an error, like `TRY_TO_NUMBER`, `TRY_CONVERT`, `TRY_PARSE_JSON` and `SAFE_DIVIDE`,
187
+ also when sqlglot does not know them but their name starts with `TRY_` or `SAFE_`. Any other function which sqlglot does
188
+ not know adds nothing, as nothing can be said about it.
189
+
190
+ **A clause which filters rows counts**, and `WHERE`, `HAVING` and `QUALIFY` do. The expression of the filter can have
191
+ any length, so it is also counted by what it contains, like `AND` and `BETWEEN`. That is why
192
+ `HAVING COUNT(*) BETWEEN 1 AND 3` and `HAVING COUNT(*) >= 1 AND COUNT(*) <= 3` get the same complexity.
193
+
194
+ **A window function does not filter and does not branch**, except `LAG` and `LEAD`: the first or last row of a
195
+ partition has no neighbour, which gives NULL or the default, so they count 1 per call. `ROW_NUMBER`, `RANK`, `SUM() OVER`
196
+ and the like add nothing, and a filter on them is counted by the filter. `QUALIFY ROW_NUMBER() OVER (...) = 1` and
197
+ `SELECT ... FROM (SELECT ROW_NUMBER() OVER (...) AS rn ...) WHERE rn = 1` therefore get the same complexity.
198
+
199
+ **A loop counts**: a recursive CTE (`WITH RECURSIVE`) and `CONNECT BY` repeat until nothing is left, and count 1 each, in
200
+ addition to the `UNION` and the `WHERE` they have.
201
+
202
+ **A statement which branches counts**: each `WHEN` of a `MERGE` (`MATCHED`, `NOT MATCHED`) and of a multi table
203
+ `INSERT ALL WHEN` counts 1, but the `ELSE` of the `INSERT` does not, like the `ELSE` of a `CASE`. `ON CONFLICT` and
204
+ `ON DUPLICATE KEY` count 1. The `IN` of a `PIVOT` lists the columns and is not a predicate, so it does not count, and
205
+ `LIKE ANY (...)` counts as the `LIKE` and not as an `ANY` too.
206
+
207
+ Not counted (yet): comparisons, also `IS NULL` and `IS DISTINCT FROM`, subqueries and CTEs.
184
208
  The weights are defined in `WEIGHTS` in `sqlcc.py`. `COALESCE` counts 1 however many arguments it has.
185
209
 
186
210
  Which functions sqlglot knows depends on its version, and an unknown function adds nothing. For example, `IFF` in the
File without changes
File without changes