sqlccc 1.1.1__tar.gz → 1.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
sqlccc-1.2.0/PKG-INFO ADDED
@@ -0,0 +1,200 @@
1
+ Metadata-Version: 2.4
2
+ Name: sqlccc
3
+ Version: 1.2.0
4
+ Summary: A cyclomatic complexity calculator for SQL
5
+ Author-email: Søren Brønsted <soren@bronsted.dk>
6
+ License-Expression: Apache-2.0
7
+ Project-URL: Homepage, https://github.com/sorenbronsted/sqlcc
8
+ Project-URL: Issues, https://github.com/sorenbronsted/sqlcc/issues
9
+ Classifier: Programming Language :: Python :: 3
10
+ Classifier: Operating System :: OS Independent
11
+ Requires-Python: >=3.11
12
+ Description-Content-Type: text/markdown
13
+ License-File: LICENSE
14
+ Requires-Dist: sqlglot<31,>=27.29.0
15
+ Dynamic: license-file
16
+
17
+ # sqlcc
18
+ A [cyclomatic complexity](https://en.wikipedia.org/wiki/Cyclomatic_complexity)
19
+ calculator for SQL
20
+
21
+ ## Installation
22
+
23
+ ```
24
+ pip install sqlccc
25
+ ```
26
+ The package is named `sqlccc` (three c's), but the module you import is `sqlcc`.
27
+
28
+ ## Usage
29
+
30
+ ```python
31
+ import sqlcc
32
+
33
+ cc = sqlcc.calc('select a, b from foo')
34
+ cc = sqlcc.calc('select `a` from `foo`', dialect='mysql')
35
+ ```
36
+ which returns a number representing the cyclomatic complexity of the SQL.
37
+ The SQL is parsed with the [sqlglot](https://github.com/tobymao/sqlglot) dialect given by `dialect`,
38
+ which defaults to `snowflake`.
39
+
40
+ The SQL can contain several statements, in which case the complexity of all of them is summed.
41
+
42
+ `calc` raises
43
+ * `TypeError` if the SQL is not a string
44
+ * `ValueError` if the SQL contains no statements, or the dialect is unknown
45
+ * `sqlglot.errors.ParseError` if the SQL can not be parsed
46
+
47
+ `crap(complexity, coverage)` returns the [CRAP1 score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
48
+ `complexity^2 * (1 - coverage/100)^3 + complexity`, where the coverage is in percent. It is never lower than the
49
+ complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from a number of tests, see
50
+ [CRAP for dbt models](#crap-for-dbt-models).
51
+
52
+ ## Command line
53
+
54
+ ```
55
+ sqlcc target/compiled/my_project/models
56
+ ```
57
+ prints the complexity of every `.sql` file found in the directories (or of the files) given, the highest first:
58
+ ```
59
+ 8 target/compiled/my_project/models/marts/fct_orders.sql
60
+ 2 target/compiled/my_project/models/staging/stg_orders.sql
61
+ ```
62
+ followed, on stderr, by a summary. `python -m sqlcc` does the same.
63
+
64
+ | Option | Effect |
65
+ |--------------------|-------------------------------------------------------------------------------------------------------|
66
+ | `-d`, `--dialect` | The sqlglot dialect, see below for the default |
67
+ | `--manifest FILE` | A dbt `manifest.json` to find the dialect in. With `--crap` also where the models and tests are found |
68
+ | `--max N` | Exit with status 1 if a file, or with `--crap` a model, has a complexity above `N`, and name it |
69
+ | `--crap` | Score the models in the manifest by CRAP instead of the files, see below |
70
+ | `--max-crap N` | With `--crap`, exit with status 1 if a model has a CRAP score above `N`, and name it |
71
+
72
+ The exit status is 0 when everything is scored and nothing exceeds `--max` or `--max-crap`, 1 when something exceeds them,
73
+ and 2 when something can not be parsed, or the arguments are invalid. A file without SQL statements is skipped with a note.
74
+
75
+ ### With dbt
76
+
77
+ sqlcc can not parse the Jinja in dbt models, so run it on the SQL that dbt compiles:
78
+
79
+ ```
80
+ dbt compile
81
+ sqlcc target/compiled/my_project/models --max 50
82
+ ```
83
+ * The dialect is the `adapter_type` of the `manifest.json` in, or above, the given path, which for the path above is
84
+ `target/manifest.json`. It is mapped to the sqlglot dialect of the same name, except `sqlserver`, which is `tsql`.
85
+ Without a manifest it is `snowflake`. `--dialect` always wins.
86
+ * dbt compiles the tests of a `schema.yml` into a directory named `schema.yml`. These are skipped when a directory is
87
+ searched, but not when you name such a file. The singular tests are compiled to `tests` next to `models` and are not
88
+ skipped, so name the `models` directory and not the project.
89
+
90
+ ## CRAP for dbt models
91
+
92
+ `--crap` ranks the models of a dbt project by their [CRAP score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
93
+ which is the complexity of the model weighted by how little it is tested. Compile the project into a folder of its own,
94
+ and point sqlcc at the manifest there:
95
+
96
+ ```
97
+ dbt compile --target-path /tmp/dbt_sqlcc
98
+ sqlcc --crap --manifest /tmp/dbt_sqlcc/manifest.json --max-crap 100
99
+ ```
100
+ It prints the models with the highest score first:
101
+ ```
102
+ CRAP COMP COV% TESTS MISSING MODEL
103
+ 24.9 7 28.6 2 5 model.shop.fct_orders
104
+ 2.0 2 100.0 3 0 model.shop.stg_customers
105
+ 2.0 2 100.0 5 0 model.shop.stg_orders
106
+ ```
107
+ `COMP` is the complexity of the compiled SQL, `COV%` the estimated coverage, `TESTS` the number of tests and `MISSING`
108
+ the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
109
+
110
+ **The coverage is an estimate.** CRAP is defined on the basis path coverage of the automated tests, but a dbt test
111
+ asserts something about the data and does not run the paths of the SQL. The complexity is the number of tests it takes
112
+ to cover every basis path, so each test is assumed to cover one path: `COV% = min(100, TESTS / COMP * 100)`. Ten
113
+ `not_null` tests count as ten paths, so the number says something about how much a model is tested, not how well.
114
+
115
+ The tests are counted as follows:
116
+ * A generic test, e.g. `unique`, `relationships` or one of your own, counts for the model it is attached to. A
117
+ `relationships` test also depends on the model it refers to, but that model gets nothing.
118
+ * A singular test, a `.sql` file in `tests`, counts in full for every model it depends on through `ref()`. A table
119
+ named directly in the SQL is not seen.
120
+ * A unit test counts for the model it tests.
121
+ * A test with `severity: warn` counts. A disabled test does not, because it is not in the manifest.
122
+
123
+ Only the definitions in the manifest are used, not whether the tests passed, and a test of a model downstream does not
124
+ count for the models it reads.
125
+
126
+ Only the models of the project which made the manifest are scored, not the ones in packages. The SQL is the
127
+ `compiled_code` in the manifest. A Python model, a model which is not compiled, and a model without SQL statements are
128
+ skipped. A few are named on stderr, and more are counted, e.g. `1363 models skipped, it is not compiled, run dbt compile`.
129
+ Nothing scored at all is an error, so a manifest from `dbt parse` does not pass a check by being empty.
130
+
131
+ **Every dbt command writes `manifest.json`, and only the models which that command compiled have a `compiled_code`.**
132
+ After `dbt run -s some_model` the manifest has the SQL of `some_model` and its tests and nothing else, and while it
133
+ runs, or after `dbt parse` or `dbt ls`, it has none. Without a `--manifest`, sqlcc reads `target/manifest.json`, which
134
+ the next dbt command overwrites. That is why the example above compiles into its own `--target-path`, and why a
135
+ `dbt compile --select` gives a score for the selected models only.
136
+
137
+ The CRAP score is never lower than the complexity, and the lack of tests weighs more the more complex the model is:
138
+
139
+ | Coverage | CRAP at complexity 12 | CRAP at complexity 40 |
140
+ |----------|-----------------------|-----------------------|
141
+ | 0% | 156 | 1640 |
142
+ | 50% | 30 | 240 |
143
+ | 80% | 13.2 | 52.8 |
144
+ | 100% | 12 | 40 |
145
+
146
+ The limit of 30 which is known from CRAP was found for the complexity of Java methods. The complexity here only counts
147
+ the functions which branch, so choose a `--max-crap` for your own project.
148
+
149
+ ## How the complexity is calculated
150
+
151
+ The complexity starts at 1 and each of the following adds to it:
152
+
153
+ | Construct | Adds |
154
+ |--------------------------------------------------------------------------------------------------|----------------------------|
155
+ | `WHERE` | 1 per clause |
156
+ | `AND`, `OR` | 1 each |
157
+ | `IN`, `BETWEEN`, `LIKE`, `ILIKE` | 1 each |
158
+ | `GROUP BY` | 1 |
159
+ | `WHEN` branch of a `CASE`, and `IF`, `IFF` (the `CASE` itself adds nothing) | 1 per branch or call |
160
+ | `COALESCE`, `NVL`, `IFNULL`, `NULLIF`, `NVL2`, `DECODE`, `GREATEST`, `LEAST`, `EXISTS` | 1 per call |
161
+ | `TRY_CAST`, `SAFE_CAST` | 1 per call |
162
+ | `REGEXP_LIKE`, `RLIKE`, `CONTAINS`, `STARTSWITH`, `ENDSWITH`, `ARRAY_CONTAINS`, `COUNT_IF` | 1 per call |
163
+ | `JOIN` of any kind, including a comma join | 1 per join |
164
+ | `UNION`, `INTERSECT`, `EXCEPT` | 1 per operator |
165
+
166
+ Everything is counted anywhere in the SQL, including subqueries and CTEs. A construct is only counted once.
167
+
168
+ A join counts 1, like `WHERE` and `GROUP BY`: the rows which match go on, and the others are left out, or become NULL
169
+ for an outer join. The condition of the join is counted by what it contains, so `JOIN b ON b.c1 = 'x' AND b.c2 = 'y'`
170
+ adds 1 for the join and 1 for the `AND`.
171
+
172
+ **A function counts when it branches**, which means that the path through the SQL depends on the data, and a test is
173
+ needed for each path. `COALESCE` has a path for a NULL and one for a value. `CAST`, `LOWER`, `TRIM`, `CONCAT` and
174
+ `SUBSTRING` do the same whatever the data is, and so they add nothing. Neither do the aggregates (`COUNT`, `SUM`,
175
+ `MAX`, ...) and the window functions, as they change the grain and do not branch, and `GROUP BY` counts on its own.
176
+ `TRY_CAST` counts, because a value which can not be converted gives NULL, which is a path of its own. A function
177
+ which sqlglot does not know adds nothing either, as nothing can be said about it.
178
+
179
+ `HAVING` adds nothing itself. Its expression can have any length, so it is counted by what it contains, like `AND`
180
+ and `BETWEEN`. That is also why `HAVING COUNT(*) BETWEEN 1 AND 3` and `HAVING COUNT(*) >= 1 AND COUNT(*) <= 3` get the
181
+ same complexity.
182
+
183
+ Not counted (yet): comparisons, subqueries, CTEs, window functions in themselves and `QUALIFY`.
184
+ The weights are defined in `WEIGHTS` in `sqlcc.py`. `COALESCE` counts 1 however many arguments it has.
185
+
186
+ Which functions sqlglot knows depends on its version, and an unknown function adds nothing. For example, `IFF` in the
187
+ Databricks dialect is unknown to sqlglot 30.0 but known to 30.5, and the same SQL can therefore get a different
188
+ complexity.
189
+
190
+ ## Development
191
+
192
+ ```
193
+ python3 -m venv .venv-linux
194
+ .venv-linux/bin/pip install -r requirements-dev.txt
195
+ make test # runs the tests
196
+ make test-coverage # runs the tests and prints which lines of the code they cover, using coverage.py
197
+ make coverage # prints the cyclomatic complexity of the Python code, using radon
198
+ ```
199
+ `make` uses `.venv-linux/bin/python` when it exists, and else `python3`. Use `make test PYTHON=path/to/python` to run
200
+ with another interpreter.
sqlccc-1.2.0/README.md ADDED
@@ -0,0 +1,184 @@
1
+ # sqlcc
2
+ A [cyclomatic complexity](https://en.wikipedia.org/wiki/Cyclomatic_complexity)
3
+ calculator for SQL
4
+
5
+ ## Installation
6
+
7
+ ```
8
+ pip install sqlccc
9
+ ```
10
+ The package is named `sqlccc` (three c's), but the module you import is `sqlcc`.
11
+
12
+ ## Usage
13
+
14
+ ```python
15
+ import sqlcc
16
+
17
+ cc = sqlcc.calc('select a, b from foo')
18
+ cc = sqlcc.calc('select `a` from `foo`', dialect='mysql')
19
+ ```
20
+ which returns a number representing the cyclomatic complexity of the SQL.
21
+ The SQL is parsed with the [sqlglot](https://github.com/tobymao/sqlglot) dialect given by `dialect`,
22
+ which defaults to `snowflake`.
23
+
24
+ The SQL can contain several statements, in which case the complexity of all of them is summed.
25
+
26
+ `calc` raises
27
+ * `TypeError` if the SQL is not a string
28
+ * `ValueError` if the SQL contains no statements, or the dialect is unknown
29
+ * `sqlglot.errors.ParseError` if the SQL can not be parsed
30
+
31
+ `crap(complexity, coverage)` returns the [CRAP1 score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
32
+ `complexity^2 * (1 - coverage/100)^3 + complexity`, where the coverage is in percent. It is never lower than the
33
+ complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from a number of tests, see
34
+ [CRAP for dbt models](#crap-for-dbt-models).
35
+
36
+ ## Command line
37
+
38
+ ```
39
+ sqlcc target/compiled/my_project/models
40
+ ```
41
+ prints the complexity of every `.sql` file found in the directories (or of the files) given, the highest first:
42
+ ```
43
+ 8 target/compiled/my_project/models/marts/fct_orders.sql
44
+ 2 target/compiled/my_project/models/staging/stg_orders.sql
45
+ ```
46
+ followed, on stderr, by a summary. `python -m sqlcc` does the same.
47
+
48
+ | Option | Effect |
49
+ |--------------------|-------------------------------------------------------------------------------------------------------|
50
+ | `-d`, `--dialect` | The sqlglot dialect, see below for the default |
51
+ | `--manifest FILE` | A dbt `manifest.json` to find the dialect in. With `--crap` also where the models and tests are found |
52
+ | `--max N` | Exit with status 1 if a file, or with `--crap` a model, has a complexity above `N`, and name it |
53
+ | `--crap` | Score the models in the manifest by CRAP instead of the files, see below |
54
+ | `--max-crap N` | With `--crap`, exit with status 1 if a model has a CRAP score above `N`, and name it |
55
+
56
+ The exit status is 0 when everything is scored and nothing exceeds `--max` or `--max-crap`, 1 when something exceeds them,
57
+ and 2 when something can not be parsed, or the arguments are invalid. A file without SQL statements is skipped with a note.
58
+
59
+ ### With dbt
60
+
61
+ sqlcc can not parse the Jinja in dbt models, so run it on the SQL that dbt compiles:
62
+
63
+ ```
64
+ dbt compile
65
+ sqlcc target/compiled/my_project/models --max 50
66
+ ```
67
+ * The dialect is the `adapter_type` of the `manifest.json` in, or above, the given path, which for the path above is
68
+ `target/manifest.json`. It is mapped to the sqlglot dialect of the same name, except `sqlserver`, which is `tsql`.
69
+ Without a manifest it is `snowflake`. `--dialect` always wins.
70
+ * dbt compiles the tests of a `schema.yml` into a directory named `schema.yml`. These are skipped when a directory is
71
+ searched, but not when you name such a file. The singular tests are compiled to `tests` next to `models` and are not
72
+ skipped, so name the `models` directory and not the project.
73
+
74
+ ## CRAP for dbt models
75
+
76
+ `--crap` ranks the models of a dbt project by their [CRAP score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
77
+ which is the complexity of the model weighted by how little it is tested. Compile the project into a folder of its own,
78
+ and point sqlcc at the manifest there:
79
+
80
+ ```
81
+ dbt compile --target-path /tmp/dbt_sqlcc
82
+ sqlcc --crap --manifest /tmp/dbt_sqlcc/manifest.json --max-crap 100
83
+ ```
84
+ It prints the models with the highest score first:
85
+ ```
86
+ CRAP COMP COV% TESTS MISSING MODEL
87
+ 24.9 7 28.6 2 5 model.shop.fct_orders
88
+ 2.0 2 100.0 3 0 model.shop.stg_customers
89
+ 2.0 2 100.0 5 0 model.shop.stg_orders
90
+ ```
91
+ `COMP` is the complexity of the compiled SQL, `COV%` the estimated coverage, `TESTS` the number of tests and `MISSING`
92
+ the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
93
+
94
+ **The coverage is an estimate.** CRAP is defined on the basis path coverage of the automated tests, but a dbt test
95
+ asserts something about the data and does not run the paths of the SQL. The complexity is the number of tests it takes
96
+ to cover every basis path, so each test is assumed to cover one path: `COV% = min(100, TESTS / COMP * 100)`. Ten
97
+ `not_null` tests count as ten paths, so the number says something about how much a model is tested, not how well.
98
+
99
+ The tests are counted as follows:
100
+ * A generic test, e.g. `unique`, `relationships` or one of your own, counts for the model it is attached to. A
101
+ `relationships` test also depends on the model it refers to, but that model gets nothing.
102
+ * A singular test, a `.sql` file in `tests`, counts in full for every model it depends on through `ref()`. A table
103
+ named directly in the SQL is not seen.
104
+ * A unit test counts for the model it tests.
105
+ * A test with `severity: warn` counts. A disabled test does not, because it is not in the manifest.
106
+
107
+ Only the definitions in the manifest are used, not whether the tests passed, and a test of a model downstream does not
108
+ count for the models it reads.
109
+
110
+ Only the models of the project which made the manifest are scored, not the ones in packages. The SQL is the
111
+ `compiled_code` in the manifest. A Python model, a model which is not compiled, and a model without SQL statements are
112
+ skipped. A few are named on stderr, and more are counted, e.g. `1363 models skipped, it is not compiled, run dbt compile`.
113
+ Nothing scored at all is an error, so a manifest from `dbt parse` does not pass a check by being empty.
114
+
115
+ **Every dbt command writes `manifest.json`, and only the models which that command compiled have a `compiled_code`.**
116
+ After `dbt run -s some_model` the manifest has the SQL of `some_model` and its tests and nothing else, and while it
117
+ runs, or after `dbt parse` or `dbt ls`, it has none. Without a `--manifest`, sqlcc reads `target/manifest.json`, which
118
+ the next dbt command overwrites. That is why the example above compiles into its own `--target-path`, and why a
119
+ `dbt compile --select` gives a score for the selected models only.
120
+
121
+ The CRAP score is never lower than the complexity, and the lack of tests weighs more the more complex the model is:
122
+
123
+ | Coverage | CRAP at complexity 12 | CRAP at complexity 40 |
124
+ |----------|-----------------------|-----------------------|
125
+ | 0% | 156 | 1640 |
126
+ | 50% | 30 | 240 |
127
+ | 80% | 13.2 | 52.8 |
128
+ | 100% | 12 | 40 |
129
+
130
+ The limit of 30 which is known from CRAP was found for the complexity of Java methods. The complexity here only counts
131
+ the functions which branch, so choose a `--max-crap` for your own project.
132
+
133
+ ## How the complexity is calculated
134
+
135
+ The complexity starts at 1 and each of the following adds to it:
136
+
137
+ | Construct | Adds |
138
+ |--------------------------------------------------------------------------------------------------|----------------------------|
139
+ | `WHERE` | 1 per clause |
140
+ | `AND`, `OR` | 1 each |
141
+ | `IN`, `BETWEEN`, `LIKE`, `ILIKE` | 1 each |
142
+ | `GROUP BY` | 1 |
143
+ | `WHEN` branch of a `CASE`, and `IF`, `IFF` (the `CASE` itself adds nothing) | 1 per branch or call |
144
+ | `COALESCE`, `NVL`, `IFNULL`, `NULLIF`, `NVL2`, `DECODE`, `GREATEST`, `LEAST`, `EXISTS` | 1 per call |
145
+ | `TRY_CAST`, `SAFE_CAST` | 1 per call |
146
+ | `REGEXP_LIKE`, `RLIKE`, `CONTAINS`, `STARTSWITH`, `ENDSWITH`, `ARRAY_CONTAINS`, `COUNT_IF` | 1 per call |
147
+ | `JOIN` of any kind, including a comma join | 1 per join |
148
+ | `UNION`, `INTERSECT`, `EXCEPT` | 1 per operator |
149
+
150
+ Everything is counted anywhere in the SQL, including subqueries and CTEs. A construct is only counted once.
151
+
152
+ A join counts 1, like `WHERE` and `GROUP BY`: the rows which match go on, and the others are left out, or become NULL
153
+ for an outer join. The condition of the join is counted by what it contains, so `JOIN b ON b.c1 = 'x' AND b.c2 = 'y'`
154
+ adds 1 for the join and 1 for the `AND`.
155
+
156
+ **A function counts when it branches**, which means that the path through the SQL depends on the data, and a test is
157
+ needed for each path. `COALESCE` has a path for a NULL and one for a value. `CAST`, `LOWER`, `TRIM`, `CONCAT` and
158
+ `SUBSTRING` do the same whatever the data is, and so they add nothing. Neither do the aggregates (`COUNT`, `SUM`,
159
+ `MAX`, ...) and the window functions, as they change the grain and do not branch, and `GROUP BY` counts on its own.
160
+ `TRY_CAST` counts, because a value which can not be converted gives NULL, which is a path of its own. A function
161
+ which sqlglot does not know adds nothing either, as nothing can be said about it.
162
+
163
+ `HAVING` adds nothing itself. Its expression can have any length, so it is counted by what it contains, like `AND`
164
+ and `BETWEEN`. That is also why `HAVING COUNT(*) BETWEEN 1 AND 3` and `HAVING COUNT(*) >= 1 AND COUNT(*) <= 3` get the
165
+ same complexity.
166
+
167
+ Not counted (yet): comparisons, subqueries, CTEs, window functions in themselves and `QUALIFY`.
168
+ The weights are defined in `WEIGHTS` in `sqlcc.py`. `COALESCE` counts 1 however many arguments it has.
169
+
170
+ Which functions sqlglot knows depends on its version, and an unknown function adds nothing. For example, `IFF` in the
171
+ Databricks dialect is unknown to sqlglot 30.0 but known to 30.5, and the same SQL can therefore get a different
172
+ complexity.
173
+
174
+ ## Development
175
+
176
+ ```
177
+ python3 -m venv .venv-linux
178
+ .venv-linux/bin/pip install -r requirements-dev.txt
179
+ make test # runs the tests
180
+ make test-coverage # runs the tests and prints which lines of the code they cover, using coverage.py
181
+ make coverage # prints the cyclomatic complexity of the Python code, using radon
182
+ ```
183
+ `make` uses `.venv-linux/bin/python` when it exists, and else `python3`. Use `make test PYTHON=path/to/python` to run
184
+ with another interpreter.
@@ -1,12 +1,15 @@
1
1
  [project]
2
2
  name = "sqlccc"
3
- version = "1.1.1"
3
+ version = "1.2.0"
4
4
  authors = [
5
5
  { name="Søren Brønsted", email="soren@bronsted.dk" },
6
6
  ]
7
- description = "A cyclomatix complexity calculator for SQL"
7
+ description = "A cyclomatic complexity calculator for SQL"
8
8
  readme = "README.md"
9
9
  requires-python = ">=3.11"
10
+ dependencies = [
11
+ "sqlglot>=27.29.0,<31",
12
+ ]
10
13
  classifiers = [
11
14
  "Programming Language :: Python :: 3",
12
15
  "Operating System :: OS Independent",
@@ -14,6 +17,9 @@ classifiers = [
14
17
  license = "Apache-2.0"
15
18
  license-files = ["LICENSE"]
16
19
 
20
+ [project.scripts]
21
+ sqlcc = "sqlcc:main"
22
+
17
23
  [project.urls]
18
24
  Homepage = "https://github.com/sorenbronsted/sqlcc"
19
25
  Issues = "https://github.com/sorenbronsted/sqlcc/issues"
sqlccc-1.2.0/sqlcc.py ADDED
@@ -0,0 +1,418 @@
1
+ import argparse
2
+ import json
3
+ import os
4
+ import sys
5
+ from collections import Counter
6
+ from dataclasses import dataclass
7
+ from pathlib import Path
8
+
9
+ import sqlglot as sg
10
+
11
+ DEFAULT_DIALECT = 'snowflake'
12
+ DEFAULT_MANIFEST = Path('target') / 'manifest.json'
13
+ MAX_SKIPPED_LISTED = 5
14
+
15
+ # dbt adapter types that are named differently from the sqlglot dialect
16
+ ADAPTER_DIALECTS = {'sqlserver': 'tsql'}
17
+
18
+ # The weight each construct adds to the complexity. A node gets the weight of the most specific class
19
+ # in its inheritance chain, so a Union counts as the SetOperation it is, and a node which has no class
20
+ # in WEIGHTS adds nothing.
21
+ WEIGHTS = {
22
+ # Logic: HAVING is not weighted, its expression varies in length and is counted by what it contains
23
+ sg.exp.Where: 1, sg.exp.And: 1, sg.exp.Or: 1, sg.exp.In: 1, sg.exp.Between: 1,
24
+ sg.exp.Like: 1, sg.exp.ILike: 1, sg.exp.Group: 1,
25
+ # Functions: only the ones which branch count, and a CASE through its WHEN branches (sg.exp.If). The others,
26
+ # like CAST and LOWER, take the same path whatever the data is. So do the ones sqlglot does not know
27
+ # (sg.exp.Anonymous), as nothing can be said about them.
28
+ sg.exp.If: 1, sg.exp.Coalesce: 1, sg.exp.Nullif: 1, sg.exp.Nvl2: 1, sg.exp.DecodeCase: 1,
29
+ sg.exp.Greatest: 1, sg.exp.Least: 1, sg.exp.Exists: 1,
30
+ # A conversion which fails gives NULL, and that is a path of its own
31
+ sg.exp.TryCast: 1,
32
+ # Functions which are predicates, like LIKE and IN
33
+ sg.exp.RegexpLike: 1, sg.exp.RegexpILike: 1, sg.exp.Contains: 1, sg.exp.StartsWith: 1,
34
+ sg.exp.EndsWith: 1, sg.exp.ArrayContains: 1, sg.exp.CountIf: 1,
35
+ # Sets: SetOperation is UNION, INTERSECT and EXCEPT
36
+ sg.exp.Join: 1, sg.exp.SetOperation: 1,
37
+ }
38
+
39
+
40
+ def _weight(node: sg.exp.Expression) -> int:
41
+ for cls in type(node).__mro__:
42
+ if cls in WEIGHTS:
43
+ return WEIGHTS[cls]
44
+ return 0
45
+
46
+
47
+ def calc(sql: str, dialect: str = DEFAULT_DIALECT) -> int:
48
+ """
49
+ Calculate the cyclomatic complexity of SQL.
50
+
51
+ The complexity is 1 plus the weight of every construct in WEIGHTS found in the SQL. If the SQL
52
+ contains more than one statement, the weights of all statements are summed.
53
+
54
+ Args:
55
+ sql (str): The SQL string, with one or more statements.
56
+ dialect (str): The SQL dialect to use for parsing. See sqlglot documentation for supported dialects.
57
+
58
+ Returns:
59
+ int: The calculated cyclomatic complexity.
60
+
61
+ Raises:
62
+ TypeError: If sql is not a string.
63
+ ValueError: If sql contains no statements, or the dialect is unknown.
64
+ sqlglot.errors.ParseError: If sql can not be parsed with the dialect.
65
+ """
66
+ if not isinstance(sql, str):
67
+ raise TypeError(f"sql must be a str, not {type(sql).__name__}")
68
+ statements = [statement for statement in sg.parse(sql, dialect=dialect) if statement is not None]
69
+ if not statements:
70
+ raise ValueError("sql contains no statements")
71
+ return 1 + sum(_weight(node) for statement in statements for node in statement.walk())
72
+
73
+
74
+ def crap(complexity: int, coverage: float) -> float:
75
+ """
76
+ Calculate CRAP1, the change risk anti-pattern score: complexity^2 * (1 - coverage/100)^3 + complexity.
77
+
78
+ Args:
79
+ complexity (int): The cyclomatic complexity, see calc.
80
+ coverage (float): The coverage in percent, from 0 to 100.
81
+
82
+ Returns:
83
+ float: The CRAP score. It is never lower than the complexity.
84
+
85
+ Raises:
86
+ ValueError: If coverage is not from 0 to 100.
87
+ """
88
+ if not 0 <= coverage <= 100:
89
+ raise ValueError(f"coverage must be from 0 to 100, not {coverage}")
90
+ return complexity ** 2 * (1 - coverage / 100) ** 3 + complexity
91
+
92
+
93
+ def coverage_from_tests(tests: int, complexity: int) -> float:
94
+ """
95
+ Estimate the basis path coverage in percent from the number of tests.
96
+
97
+ The complexity is the number of tests it takes to cover every basis path. When each test is
98
+ assumed to cover one path, the coverage is the number of tests relative to the complexity.
99
+
100
+ Args:
101
+ tests (int): The number of tests.
102
+ complexity (int): The cyclomatic complexity, see calc.
103
+
104
+ Returns:
105
+ float: The coverage in percent, at most 100.
106
+
107
+ Raises:
108
+ ValueError: If complexity is lower than 1.
109
+ """
110
+ if complexity < 1:
111
+ raise ValueError(f"complexity must be at least 1, not {complexity}")
112
+ return min(100.0, 100.0 * tests / complexity)
113
+
114
+
115
+ @dataclass(frozen=True)
116
+ class ModelScore:
117
+ name: str
118
+ complexity: int
119
+ tests: int
120
+
121
+ @property
122
+ def coverage(self) -> float:
123
+ return coverage_from_tests(self.tests, self.complexity)
124
+
125
+ @property
126
+ def crap(self) -> float:
127
+ return crap(self.complexity, self.coverage)
128
+
129
+ @property
130
+ def missing(self) -> int:
131
+ return max(0, self.complexity - self.tests)
132
+
133
+
134
+ class _Skipped(Exception):
135
+ pass
136
+
137
+
138
+ def _sql_files(path: Path) -> list[Path]:
139
+ if not path.exists():
140
+ raise ValueError(f"{path} does not exist")
141
+ if path.is_file():
142
+ return [path]
143
+ # dbt compiles the generic tests of a schema.yml into a directory named schema.yml
144
+ return sorted(
145
+ file for file in path.rglob('*.sql')
146
+ if not any(part.endswith(('.yml', '.yaml')) for part in file.relative_to(path).parts[:-1])
147
+ )
148
+
149
+
150
+ def _find_manifest(paths: list[Path]) -> Path | None:
151
+ for path in paths:
152
+ resolved = path.resolve()
153
+ for directory in [resolved, *resolved.parents]:
154
+ if (directory / 'manifest.json').is_file():
155
+ return directory / 'manifest.json'
156
+ return None
157
+
158
+
159
+ def _read_json_object(path: Path) -> dict:
160
+ try:
161
+ content = json.loads(path.read_text(encoding='utf-8'))
162
+ except (OSError, ValueError) as e:
163
+ raise ValueError(f"can not read {path}: {e}")
164
+ if not isinstance(content, dict):
165
+ raise ValueError(f"{path} is not a JSON object")
166
+ return content
167
+
168
+
169
+ def _manifest_dialect(manifest: dict) -> str | None:
170
+ adapter = manifest.get('metadata', {}).get('adapter_type')
171
+ return ADAPTER_DIALECTS.get(adapter, adapter)
172
+
173
+
174
+ def _dialect_from_manifest(path: Path) -> str | None:
175
+ try:
176
+ return _manifest_dialect(_read_json_object(path))
177
+ except ValueError as e:
178
+ raise ValueError(f"can not read the dialect from {path}, use --dialect instead: {e}")
179
+
180
+
181
+ def _valid_dialect(dialect: str | None) -> str:
182
+ dialect = dialect or DEFAULT_DIALECT
183
+ sg.Dialect.get_or_raise(dialect)
184
+ return dialect
185
+
186
+
187
+ def _resolve_dialect(dialect: str | None, manifest: Path | None, paths: list[Path]) -> str:
188
+ manifest = manifest or _find_manifest(paths)
189
+ if not dialect and manifest:
190
+ dialect = _dialect_from_manifest(manifest)
191
+ return _valid_dialect(dialect)
192
+
193
+
194
+ def _first_line(e: Exception) -> str:
195
+ # a sqlglot error message can span several lines, with the offending SQL below the first
196
+ return (str(e).splitlines() or [type(e).__name__])[0]
197
+
198
+
199
+ def _score_files(files: list[Path], dialect: str) -> tuple[dict[Path, int], dict[Path, str], list[Path]]:
200
+ scores, failed, skipped = {}, {}, []
201
+ for file in files:
202
+ try:
203
+ scores[file] = calc(file.read_text(encoding='utf-8'), dialect=dialect)
204
+ except (OSError, UnicodeDecodeError, sg.errors.SqlglotError) as e:
205
+ failed[file] = _first_line(e)
206
+ except ValueError:
207
+ skipped.append(file)
208
+ return scores, failed, skipped
209
+
210
+
211
+ def _read_manifest(path: Path) -> dict:
212
+ manifest = _read_json_object(path)
213
+ if not isinstance(manifest.get('nodes'), dict):
214
+ raise ValueError(f"{path} is not a dbt manifest")
215
+ return manifest
216
+
217
+
218
+ def _test_counts(manifest: dict) -> Counter:
219
+ """
220
+ Count the tests of each node in the manifest.
221
+
222
+ A generic test belongs to the node it is attached to only, even though a relationships test also
223
+ depends on the node it refers to. A singular test belongs to every node it depends on, and so does
224
+ a unit test. Disabled tests are not in the manifest nodes.
225
+ """
226
+ counts = Counter()
227
+ for node in manifest['nodes'].values():
228
+ if node.get('resource_type') == 'test':
229
+ attached = node.get('attached_node')
230
+ counts.update([attached] if attached else node.get('depends_on', {}).get('nodes', []))
231
+ for unit_test in manifest.get('unit_tests', {}).values():
232
+ counts.update(unit_test.get('depends_on', {}).get('nodes', []))
233
+ return counts
234
+
235
+
236
+ def _models(manifest: dict) -> dict[str, dict]:
237
+ project = manifest.get('metadata', {}).get('project_name')
238
+ return {
239
+ unique_id: node for unique_id, node in manifest['nodes'].items()
240
+ if node.get('resource_type') == 'model' and (project is None or node.get('package_name') == project)
241
+ }
242
+
243
+
244
+ def _compiled_sql(node: dict) -> str:
245
+ if node.get('language', 'sql') != 'sql':
246
+ raise _Skipped("it is not a SQL model")
247
+ if not node.get('compiled_code'):
248
+ raise _Skipped("it is not compiled, run dbt compile")
249
+ return node['compiled_code']
250
+
251
+
252
+ def _score_models(manifest: dict, dialect: str) -> tuple[list[ModelScore], dict[str, str], dict[str, str]]:
253
+ tests = _test_counts(manifest)
254
+ scores, failed, skipped = [], {}, {}
255
+ for unique_id, node in _models(manifest).items():
256
+ try:
257
+ scores.append(ModelScore(unique_id, calc(_compiled_sql(node), dialect=dialect), tests[unique_id]))
258
+ except _Skipped as e:
259
+ skipped[unique_id] = str(e)
260
+ except sg.errors.SqlglotError as e:
261
+ failed[unique_id] = _first_line(e)
262
+ except ValueError:
263
+ skipped[unique_id] = "it contains no SQL statements"
264
+ return scores, failed, skipped
265
+
266
+
267
+ def _parser() -> argparse.ArgumentParser:
268
+ parser = argparse.ArgumentParser(
269
+ prog='sqlcc',
270
+ description='Calculate the cyclomatic complexity of SQL files, e.g. the models dbt compiles to target/compiled, '
271
+ 'or with --crap the CRAP score of the models in a dbt manifest.',
272
+ epilog='Exit status: 0 when everything is scored and nothing exceeds --max or --max-crap, '
273
+ '1 when something exceeds them, '
274
+ '2 when something can not be scored or the arguments are invalid.',
275
+ )
276
+ parser.add_argument('paths', nargs='*', type=Path, metavar='PATH',
277
+ help='SQL files, or directories which are searched for *.sql. '
278
+ 'Compiled dbt tests, in directories named *.yml, are skipped. Not used with --crap.')
279
+ parser.add_argument('-d', '--dialect',
280
+ help=f'The sqlglot dialect. Default is the adapter_type of the dbt manifest.json '
281
+ f'in or above PATH, or else {DEFAULT_DIALECT}.')
282
+ parser.add_argument('--manifest', type=Path,
283
+ help=f'Use this dbt manifest.json to find the dialect. With --crap it is also where the models '
284
+ f'and tests are found, and the default is {DEFAULT_MANIFEST}.')
285
+ parser.add_argument('--max', type=int, dest='max_score',
286
+ help='Exit with status 1 if the complexity of a file, or with --crap a model, is above this.')
287
+ parser.add_argument('--crap', action='store_true',
288
+ help='Calculate the CRAP score of the models in the dbt manifest, from their compiled SQL and '
289
+ 'tests. Run dbt compile first.')
290
+ parser.add_argument('--max-crap', type=float,
291
+ help='With --crap, exit with status 1 if the CRAP score of a model is above this.')
292
+ return parser
293
+
294
+
295
+ def _exceeding(values: dict, limit: float | None) -> dict:
296
+ if limit is None:
297
+ return {}
298
+ return {key: value for key, value in values.items() if value > limit}
299
+
300
+
301
+ def _print_scores(scores: dict[Path, int]) -> None:
302
+ for file, score in sorted(scores.items(), key=lambda item: (-item[1], str(item[0]))):
303
+ print(f"{score:>6} {file}")
304
+ sys.stdout.flush()
305
+
306
+
307
+ def _print_diagnostics(scores: dict[Path, int], failed: dict[Path, str], skipped: list[Path],
308
+ exceeding: dict[Path, int], max_score: int | None) -> None:
309
+ for file, message in failed.items():
310
+ print(f"{file}: {message}", file=sys.stderr)
311
+ for file in skipped:
312
+ print(f"{file}: skipped, it contains no SQL statements", file=sys.stderr)
313
+ for file, score in exceeding.items():
314
+ print(f"{file}: complexity {score} exceeds --max {max_score}", file=sys.stderr)
315
+ if scores:
316
+ print(f"{len(scores)} files scored, max {max(scores.values())}, "
317
+ f"average {sum(scores.values()) / len(scores):.1f}", file=sys.stderr)
318
+
319
+
320
+ def _print_models(scores: list[ModelScore]) -> None:
321
+ print(f"{'CRAP':>9} {'COMP':>5} {'COV%':>6} {'TESTS':>5} {'MISSING':>7} MODEL")
322
+ for score in sorted(scores, key=lambda score: (-score.crap, score.name)):
323
+ print(f"{score.crap:>9.1f} {score.complexity:>5} {score.coverage:>6.1f} {score.tests:>5} "
324
+ f"{score.missing:>7} {score.name}")
325
+ sys.stdout.flush()
326
+
327
+
328
+ def _print_skipped(skipped: dict[str, str]) -> None:
329
+ names_by_reason: dict[str, list[str]] = {}
330
+ for name, reason in skipped.items():
331
+ names_by_reason.setdefault(reason, []).append(name)
332
+ for reason, names in names_by_reason.items():
333
+ if len(names) > MAX_SKIPPED_LISTED:
334
+ print(f"{len(names)} models skipped, {reason}", file=sys.stderr)
335
+ else:
336
+ for name in names:
337
+ print(f"{name}: skipped, {reason}", file=sys.stderr)
338
+
339
+
340
+ def _print_model_diagnostics(scores: list[ModelScore], failed: dict[str, str], skipped: dict[str, str],
341
+ too_complex: dict[str, int], too_crappy: dict[str, float],
342
+ args: argparse.Namespace) -> None:
343
+ for name, message in failed.items():
344
+ print(f"{name}: {message}", file=sys.stderr)
345
+ _print_skipped(skipped)
346
+ for name, complexity in too_complex.items():
347
+ print(f"{name}: complexity {complexity} exceeds --max {args.max_score}", file=sys.stderr)
348
+ for name, score in too_crappy.items():
349
+ print(f"{name}: CRAP {score:.1f} exceeds --max-crap {args.max_crap:g}", file=sys.stderr)
350
+ if scores:
351
+ print(f"{len(scores)} models scored, max CRAP {max(score.crap for score in scores):.1f}, "
352
+ f"{sum(1 for score in scores if score.tests == 0)} without tests", file=sys.stderr)
353
+
354
+
355
+ def _run_files(args: argparse.Namespace) -> int:
356
+ try:
357
+ dialect = _resolve_dialect(args.dialect, args.manifest, args.paths)
358
+ files = [file for path in args.paths for file in _sql_files(path)]
359
+ if not files:
360
+ raise ValueError("no .sql files found")
361
+ except ValueError as e:
362
+ print(f"sqlcc: error: {e}", file=sys.stderr)
363
+ return 2
364
+
365
+ scores, failed, skipped = _score_files(files, dialect)
366
+ exceeding = _exceeding(scores, args.max_score)
367
+ _print_scores(scores)
368
+ _print_diagnostics(scores, failed, skipped, exceeding, args.max_score)
369
+ if failed:
370
+ return 2
371
+ return 1 if exceeding else 0
372
+
373
+
374
+ def _run_crap(args: argparse.Namespace) -> int:
375
+ try:
376
+ manifest = _read_manifest(args.manifest or DEFAULT_MANIFEST)
377
+ dialect = _valid_dialect(args.dialect or _manifest_dialect(manifest))
378
+ except ValueError as e:
379
+ print(f"sqlcc: error: {e}", file=sys.stderr)
380
+ return 2
381
+
382
+ scores, failed, skipped = _score_models(manifest, dialect)
383
+ too_complex = _exceeding({score.name: score.complexity for score in scores}, args.max_score)
384
+ too_crappy = _exceeding({score.name: score.crap for score in scores}, args.max_crap)
385
+ _print_models(scores)
386
+ _print_model_diagnostics(scores, failed, skipped, too_complex, too_crappy, args)
387
+ if failed:
388
+ return 2
389
+ if not scores:
390
+ print("sqlcc: error: no models could be scored", file=sys.stderr)
391
+ return 2
392
+ return 1 if too_complex or too_crappy else 0
393
+
394
+
395
+ def _main(argv: list[str] | None) -> int:
396
+ parser = _parser()
397
+ args = parser.parse_args(argv)
398
+ if args.crap and args.paths:
399
+ parser.error("--crap reads the models from the manifest, so PATH can not be used")
400
+ if not args.crap and not args.paths:
401
+ parser.error("the following arguments are required: PATH, or use --crap")
402
+ if args.max_crap is not None and not args.crap:
403
+ parser.error("--max-crap can only be used with --crap")
404
+ return _run_crap(args) if args.crap else _run_files(args)
405
+
406
+
407
+ def main(argv: list[str] | None = None) -> int:
408
+ try:
409
+ return _main(argv)
410
+ except BrokenPipeError:
411
+ # The reader of the output, e.g. head, is done. Point stdout to nowhere, so Python does not fail again
412
+ # when it flushes it on exit. 141 is the status a shell reports for a process which got SIGPIPE.
413
+ os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno())
414
+ return 141
415
+
416
+
417
+ if __name__ == '__main__':
418
+ sys.exit(main())
@@ -0,0 +1,200 @@
1
+ Metadata-Version: 2.4
2
+ Name: sqlccc
3
+ Version: 1.2.0
4
+ Summary: A cyclomatic complexity calculator for SQL
5
+ Author-email: Søren Brønsted <soren@bronsted.dk>
6
+ License-Expression: Apache-2.0
7
+ Project-URL: Homepage, https://github.com/sorenbronsted/sqlcc
8
+ Project-URL: Issues, https://github.com/sorenbronsted/sqlcc/issues
9
+ Classifier: Programming Language :: Python :: 3
10
+ Classifier: Operating System :: OS Independent
11
+ Requires-Python: >=3.11
12
+ Description-Content-Type: text/markdown
13
+ License-File: LICENSE
14
+ Requires-Dist: sqlglot<31,>=27.29.0
15
+ Dynamic: license-file
16
+
17
+ # sqlcc
18
+ A [cyclomatic complexity](https://en.wikipedia.org/wiki/Cyclomatic_complexity)
19
+ calculator for SQL
20
+
21
+ ## Installation
22
+
23
+ ```
24
+ pip install sqlccc
25
+ ```
26
+ The package is named `sqlccc` (three c's), but the module you import is `sqlcc`.
27
+
28
+ ## Usage
29
+
30
+ ```python
31
+ import sqlcc
32
+
33
+ cc = sqlcc.calc('select a, b from foo')
34
+ cc = sqlcc.calc('select `a` from `foo`', dialect='mysql')
35
+ ```
36
+ which returns a number representing the cyclomatic complexity of the SQL.
37
+ The SQL is parsed with the [sqlglot](https://github.com/tobymao/sqlglot) dialect given by `dialect`,
38
+ which defaults to `snowflake`.
39
+
40
+ The SQL can contain several statements, in which case the complexity of all of them is summed.
41
+
42
+ `calc` raises
43
+ * `TypeError` if the SQL is not a string
44
+ * `ValueError` if the SQL contains no statements, or the dialect is unknown
45
+ * `sqlglot.errors.ParseError` if the SQL can not be parsed
46
+
47
+ `crap(complexity, coverage)` returns the [CRAP1 score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
48
+ `complexity^2 * (1 - coverage/100)^3 + complexity`, where the coverage is in percent. It is never lower than the
49
+ complexity. `coverage_from_tests(tests, complexity)` estimates the coverage from a number of tests, see
50
+ [CRAP for dbt models](#crap-for-dbt-models).
51
+
52
+ ## Command line
53
+
54
+ ```
55
+ sqlcc target/compiled/my_project/models
56
+ ```
57
+ prints the complexity of every `.sql` file found in the directories (or of the files) given, the highest first:
58
+ ```
59
+ 8 target/compiled/my_project/models/marts/fct_orders.sql
60
+ 2 target/compiled/my_project/models/staging/stg_orders.sql
61
+ ```
62
+ followed, on stderr, by a summary. `python -m sqlcc` does the same.
63
+
64
+ | Option | Effect |
65
+ |--------------------|-------------------------------------------------------------------------------------------------------|
66
+ | `-d`, `--dialect` | The sqlglot dialect, see below for the default |
67
+ | `--manifest FILE` | A dbt `manifest.json` to find the dialect in. With `--crap` also where the models and tests are found |
68
+ | `--max N` | Exit with status 1 if a file, or with `--crap` a model, has a complexity above `N`, and name it |
69
+ | `--crap` | Score the models in the manifest by CRAP instead of the files, see below |
70
+ | `--max-crap N` | With `--crap`, exit with status 1 if a model has a CRAP score above `N`, and name it |
71
+
72
+ The exit status is 0 when everything is scored and nothing exceeds `--max` or `--max-crap`, 1 when something exceeds them,
73
+ and 2 when something can not be parsed, or the arguments are invalid. A file without SQL statements is skipped with a note.
74
+
75
+ ### With dbt
76
+
77
+ sqlcc can not parse the Jinja in dbt models, so run it on the SQL that dbt compiles:
78
+
79
+ ```
80
+ dbt compile
81
+ sqlcc target/compiled/my_project/models --max 50
82
+ ```
83
+ * The dialect is the `adapter_type` of the `manifest.json` in, or above, the given path, which for the path above is
84
+ `target/manifest.json`. It is mapped to the sqlglot dialect of the same name, except `sqlserver`, which is `tsql`.
85
+ Without a manifest it is `snowflake`. `--dialect` always wins.
86
+ * dbt compiles the tests of a `schema.yml` into a directory named `schema.yml`. These are skipped when a directory is
87
+ searched, but not when you name such a file. The singular tests are compiled to `tests` next to `models` and are not
88
+ skipped, so name the `models` directory and not the project.
89
+
90
+ ## CRAP for dbt models
91
+
92
+ `--crap` ranks the models of a dbt project by their [CRAP score](https://testing.googleblog.com/2011/02/this-code-is-crap.html),
93
+ which is the complexity of the model weighted by how little it is tested. Compile the project into a folder of its own,
94
+ and point sqlcc at the manifest there:
95
+
96
+ ```
97
+ dbt compile --target-path /tmp/dbt_sqlcc
98
+ sqlcc --crap --manifest /tmp/dbt_sqlcc/manifest.json --max-crap 100
99
+ ```
100
+ It prints the models with the highest score first:
101
+ ```
102
+ CRAP COMP COV% TESTS MISSING MODEL
103
+ 24.9 7 28.6 2 5 model.shop.fct_orders
104
+ 2.0 2 100.0 3 0 model.shop.stg_customers
105
+ 2.0 2 100.0 5 0 model.shop.stg_orders
106
+ ```
107
+ `COMP` is the complexity of the compiled SQL, `COV%` the estimated coverage, `TESTS` the number of tests and `MISSING`
108
+ the number of tests it takes to get to 100%. `CRAP` is `COMP^2 * (1 - COV/100)^3 + COMP`.
109
+
110
+ **The coverage is an estimate.** CRAP is defined on the basis path coverage of the automated tests, but a dbt test
111
+ asserts something about the data and does not run the paths of the SQL. The complexity is the number of tests it takes
112
+ to cover every basis path, so each test is assumed to cover one path: `COV% = min(100, TESTS / COMP * 100)`. Ten
113
+ `not_null` tests count as ten paths, so the number says something about how much a model is tested, not how well.
114
+
115
+ The tests are counted as follows:
116
+ * A generic test, e.g. `unique`, `relationships` or one of your own, counts for the model it is attached to. A
117
+ `relationships` test also depends on the model it refers to, but that model gets nothing.
118
+ * A singular test, a `.sql` file in `tests`, counts in full for every model it depends on through `ref()`. A table
119
+ named directly in the SQL is not seen.
120
+ * A unit test counts for the model it tests.
121
+ * A test with `severity: warn` counts. A disabled test does not, because it is not in the manifest.
122
+
123
+ Only the definitions in the manifest are used, not whether the tests passed, and a test of a model downstream does not
124
+ count for the models it reads.
125
+
126
+ Only the models of the project which made the manifest are scored, not the ones in packages. The SQL is the
127
+ `compiled_code` in the manifest. A Python model, a model which is not compiled, and a model without SQL statements are
128
+ skipped. A few are named on stderr, and more are counted, e.g. `1363 models skipped, it is not compiled, run dbt compile`.
129
+ Nothing scored at all is an error, so a manifest from `dbt parse` does not pass a check by being empty.
130
+
131
+ **Every dbt command writes `manifest.json`, and only the models which that command compiled have a `compiled_code`.**
132
+ After `dbt run -s some_model` the manifest has the SQL of `some_model` and its tests and nothing else, and while it
133
+ runs, or after `dbt parse` or `dbt ls`, it has none. Without a `--manifest`, sqlcc reads `target/manifest.json`, which
134
+ the next dbt command overwrites. That is why the example above compiles into its own `--target-path`, and why a
135
+ `dbt compile --select` gives a score for the selected models only.
136
+
137
+ The CRAP score is never lower than the complexity, and the lack of tests weighs more the more complex the model is:
138
+
139
+ | Coverage | CRAP at complexity 12 | CRAP at complexity 40 |
140
+ |----------|-----------------------|-----------------------|
141
+ | 0% | 156 | 1640 |
142
+ | 50% | 30 | 240 |
143
+ | 80% | 13.2 | 52.8 |
144
+ | 100% | 12 | 40 |
145
+
146
+ The limit of 30 which is known from CRAP was found for the complexity of Java methods. The complexity here only counts
147
+ the functions which branch, so choose a `--max-crap` for your own project.
148
+
149
+ ## How the complexity is calculated
150
+
151
+ The complexity starts at 1 and each of the following adds to it:
152
+
153
+ | Construct | Adds |
154
+ |--------------------------------------------------------------------------------------------------|----------------------------|
155
+ | `WHERE` | 1 per clause |
156
+ | `AND`, `OR` | 1 each |
157
+ | `IN`, `BETWEEN`, `LIKE`, `ILIKE` | 1 each |
158
+ | `GROUP BY` | 1 |
159
+ | `WHEN` branch of a `CASE`, and `IF`, `IFF` (the `CASE` itself adds nothing) | 1 per branch or call |
160
+ | `COALESCE`, `NVL`, `IFNULL`, `NULLIF`, `NVL2`, `DECODE`, `GREATEST`, `LEAST`, `EXISTS` | 1 per call |
161
+ | `TRY_CAST`, `SAFE_CAST` | 1 per call |
162
+ | `REGEXP_LIKE`, `RLIKE`, `CONTAINS`, `STARTSWITH`, `ENDSWITH`, `ARRAY_CONTAINS`, `COUNT_IF` | 1 per call |
163
+ | `JOIN` of any kind, including a comma join | 1 per join |
164
+ | `UNION`, `INTERSECT`, `EXCEPT` | 1 per operator |
165
+
166
+ Everything is counted anywhere in the SQL, including subqueries and CTEs. A construct is only counted once.
167
+
168
+ A join counts 1, like `WHERE` and `GROUP BY`: the rows which match go on, and the others are left out, or become NULL
169
+ for an outer join. The condition of the join is counted by what it contains, so `JOIN b ON b.c1 = 'x' AND b.c2 = 'y'`
170
+ adds 1 for the join and 1 for the `AND`.
171
+
172
+ **A function counts when it branches**, which means that the path through the SQL depends on the data, and a test is
173
+ needed for each path. `COALESCE` has a path for a NULL and one for a value. `CAST`, `LOWER`, `TRIM`, `CONCAT` and
174
+ `SUBSTRING` do the same whatever the data is, and so they add nothing. Neither do the aggregates (`COUNT`, `SUM`,
175
+ `MAX`, ...) and the window functions, as they change the grain and do not branch, and `GROUP BY` counts on its own.
176
+ `TRY_CAST` counts, because a value which can not be converted gives NULL, which is a path of its own. A function
177
+ which sqlglot does not know adds nothing either, as nothing can be said about it.
178
+
179
+ `HAVING` adds nothing itself. Its expression can have any length, so it is counted by what it contains, like `AND`
180
+ and `BETWEEN`. That is also why `HAVING COUNT(*) BETWEEN 1 AND 3` and `HAVING COUNT(*) >= 1 AND COUNT(*) <= 3` get the
181
+ same complexity.
182
+
183
+ Not counted (yet): comparisons, subqueries, CTEs, window functions in themselves and `QUALIFY`.
184
+ The weights are defined in `WEIGHTS` in `sqlcc.py`. `COALESCE` counts 1 however many arguments it has.
185
+
186
+ Which functions sqlglot knows depends on its version, and an unknown function adds nothing. For example, `IFF` in the
187
+ Databricks dialect is unknown to sqlglot 30.0 but known to 30.5, and the same SQL can therefore get a different
188
+ complexity.
189
+
190
+ ## Development
191
+
192
+ ```
193
+ python3 -m venv .venv-linux
194
+ .venv-linux/bin/pip install -r requirements-dev.txt
195
+ make test # runs the tests
196
+ make test-coverage # runs the tests and prints which lines of the code they cover, using coverage.py
197
+ make coverage # prints the cyclomatic complexity of the Python code, using radon
198
+ ```
199
+ `make` uses `.venv-linux/bin/python` when it exists, and else `python3`. Use `make test PYTHON=path/to/python` to run
200
+ with another interpreter.
@@ -5,4 +5,6 @@ sqlcc.py
5
5
  sqlccc.egg-info/PKG-INFO
6
6
  sqlccc.egg-info/SOURCES.txt
7
7
  sqlccc.egg-info/dependency_links.txt
8
+ sqlccc.egg-info/entry_points.txt
9
+ sqlccc.egg-info/requires.txt
8
10
  sqlccc.egg-info/top_level.txt
@@ -0,0 +1,2 @@
1
+ [console_scripts]
2
+ sqlcc = sqlcc:main
@@ -0,0 +1 @@
1
+ sqlglot<31,>=27.29.0
sqlccc-1.1.1/PKG-INFO DELETED
@@ -1,25 +0,0 @@
1
- Metadata-Version: 2.4
2
- Name: sqlccc
3
- Version: 1.1.1
4
- Summary: A cyclomatix complexity calculator for SQL
5
- Author-email: Søren Brønsted <soren@bronsted.dk>
6
- License-Expression: Apache-2.0
7
- Project-URL: Homepage, https://github.com/sorenbronsted/sqlcc
8
- Project-URL: Issues, https://github.com/sorenbronsted/sqlcc/issues
9
- Classifier: Programming Language :: Python :: 3
10
- Classifier: Operating System :: OS Independent
11
- Requires-Python: >=3.11
12
- Description-Content-Type: text/markdown
13
- License-File: LICENSE
14
- Dynamic: license-file
15
-
16
- # sqlcc
17
- A [cyclomatix complexity](https://en.wikipedia.org/wiki/Cyclomatic_complexity)
18
- calculator for SQL
19
-
20
- ## Usage
21
-
22
- ```bash
23
- sqlcc <file.sql>
24
- ```
25
- which output a number representing the cyclomatic complexity of the SQL file.
sqlccc-1.1.1/README.md DELETED
@@ -1,10 +0,0 @@
1
- # sqlcc
2
- A [cyclomatix complexity](https://en.wikipedia.org/wiki/Cyclomatic_complexity)
3
- calculator for SQL
4
-
5
- ## Usage
6
-
7
- ```bash
8
- sqlcc <file.sql>
9
- ```
10
- which output a number representing the cyclomatic complexity of the SQL file.
sqlccc-1.1.1/sqlcc.py DELETED
@@ -1,30 +0,0 @@
1
- import sqlglot as sg
2
-
3
- """
4
- Calculate the cyclomatic complexity of a SQL query.
5
-
6
- Args:
7
- sql (str): The SQL query string.
8
- dialect (str): The SQL dialect to use for parsing. See sqlglot documentation for supported dialects.
9
-
10
- Returns:
11
- int: The calculated cyclomatic complexity.
12
- """
13
- def calc(sql: str, dialect='snowflake') -> int:
14
- result = 1
15
- parsed = sg.parse_one(sql,dialect=dialect)
16
- cyclomatics = [
17
- # Logic
18
- sg.exp.And, sg.exp.Or, sg.exp.In, sg.exp.Between, sg.exp.Like, sg.exp.Group,
19
- # Columns
20
- sg.exp.Case, sg.exp.Func,
21
- # Sets
22
- sg.exp.Join
23
- ]
24
- for cyclom in cyclomatics:
25
- for item in parsed.find_all(cyclom):
26
- if item.key == sg.exp.Join.key:
27
- result = result + 2
28
- else:
29
- result += 1
30
- return result
@@ -1,25 +0,0 @@
1
- Metadata-Version: 2.4
2
- Name: sqlccc
3
- Version: 1.1.1
4
- Summary: A cyclomatix complexity calculator for SQL
5
- Author-email: Søren Brønsted <soren@bronsted.dk>
6
- License-Expression: Apache-2.0
7
- Project-URL: Homepage, https://github.com/sorenbronsted/sqlcc
8
- Project-URL: Issues, https://github.com/sorenbronsted/sqlcc/issues
9
- Classifier: Programming Language :: Python :: 3
10
- Classifier: Operating System :: OS Independent
11
- Requires-Python: >=3.11
12
- Description-Content-Type: text/markdown
13
- License-File: LICENSE
14
- Dynamic: license-file
15
-
16
- # sqlcc
17
- A [cyclomatix complexity](https://en.wikipedia.org/wiki/Cyclomatic_complexity)
18
- calculator for SQL
19
-
20
- ## Usage
21
-
22
- ```bash
23
- sqlcc <file.sql>
24
- ```
25
- which output a number representing the cyclomatic complexity of the SQL file.
File without changes
File without changes