codeanalyzer-python 0.3.1__tar.gz → 1.0.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/.gitignore +8 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/PKG-INFO +248 -61
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/README.md +244 -59
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/__main__.py +86 -4
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/core.py +175 -82
- codeanalyzer_python-1.0.1/codeanalyzer/dataflow/__init__.py +35 -0
- codeanalyzer_python-1.0.1/codeanalyzer/dataflow/access_paths.py +563 -0
- codeanalyzer_python-1.0.1/codeanalyzer/dataflow/alias.py +93 -0
- codeanalyzer_python-1.0.1/codeanalyzer/dataflow/builder.py +688 -0
- codeanalyzer_python-1.0.1/codeanalyzer/dataflow/cfg.py +605 -0
- codeanalyzer_python-1.0.1/codeanalyzer/dataflow/defuse.py +113 -0
- codeanalyzer_python-1.0.1/codeanalyzer/dataflow/dominance.py +140 -0
- codeanalyzer_python-1.0.1/codeanalyzer/dataflow/identity.py +91 -0
- codeanalyzer_python-1.0.1/codeanalyzer/dataflow/pdg.py +100 -0
- codeanalyzer_python-1.0.1/codeanalyzer/dataflow/scalpel_oracle.py +269 -0
- codeanalyzer_python-1.0.1/codeanalyzer/dataflow/scc.py +91 -0
- codeanalyzer_python-1.0.1/codeanalyzer/dataflow/sdg.py +424 -0
- codeanalyzer_python-1.0.1/codeanalyzer/dataflow/slicing.py +93 -0
- codeanalyzer_python-1.0.1/codeanalyzer/dataflow/summaries.py +217 -0
- codeanalyzer_python-1.0.1/codeanalyzer/dataflow/syntactic.py +26 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/neo4j/bolt.py +19 -4
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/neo4j/cypher.py +9 -3
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/neo4j/emit.py +8 -3
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/neo4j/project.py +241 -60
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/neo4j/rows.py +18 -15
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/neo4j/schema.py +43 -7
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/options/options.py +4 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/schema/__init__.py +19 -0
- codeanalyzer_python-1.0.1/codeanalyzer/schema/assign_ids.py +37 -0
- codeanalyzer_python-1.0.1/codeanalyzer/schema/call_graph_ids.py +12 -0
- codeanalyzer_python-1.0.1/codeanalyzer/schema/ids.py +23 -0
- codeanalyzer_python-1.0.1/codeanalyzer/schema/l1_body.py +29 -0
- codeanalyzer_python-1.0.1/codeanalyzer/schema/l2_callees.py +36 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/schema/py_schema.py +141 -30
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/semantic_analysis/call_graph.py +24 -27
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/semantic_analysis/pycg/pycg_analysis.py +77 -16
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/semantic_analysis/pycg/shard_planner.py +7 -7
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/syntactic_analysis/symbol_table_builder.py +65 -27
- codeanalyzer_python-1.0.1/docs/handoff/README.md +160 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1/docs/handoff}/schema.neo4j.json +109 -10
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/pyproject.toml +8 -2
- codeanalyzer_python-1.0.1/schema.neo4j.json +388 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/LICENSE +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/NOTICE +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/__init__.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/config/__init__.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/config/config.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/jedi/__init__.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/jedi/jedi.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/neo4j/__init__.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/options/__init__.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/provenance.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/py.typed +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/semantic_analysis/__init__.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/semantic_analysis/pycg/__init__.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/semantic_analysis/pycg/pycg_exceptions.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/syntactic_analysis/__init__.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/syntactic_analysis/exceptions.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/syntactic_analysis/import_resolver.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/utils/__init__.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/utils/logging.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/codeanalyzer/utils/progress_bar.py +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/schema-uml.drawio +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/test/fixtures/whole_applications/flask/README.md +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/test/fixtures/whole_applications/flask/examples/celery/README.md +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/test/fixtures/whole_applications/flask/requirements-skip/README.md +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/test/fixtures/whole_applications/flask/src/flask/sansio/README.md +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/test/fixtures/whole_applications/requests/LICENSE +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/test/fixtures/whole_applications/requests/NOTICE +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/test/fixtures/whole_applications/requests/README.md +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/test/fixtures/whole_applications/requests/docs/_themes/LICENSE +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/test/fixtures/whole_applications/requests/ext/LICENSE +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/test/fixtures/whole_applications/xarray/LICENSE +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/test/fixtures/whole_applications/xarray/README.md +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/test/fixtures/whole_applications/xarray/properties/README.md +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/test/fixtures/whole_applications/xarray/xarray/datatree_/LICENSE +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/test/fixtures/whole_applications/xarray/xarray/datatree_/README.md +0 -0
- {codeanalyzer_python-0.3.1 → codeanalyzer_python-1.0.1}/test/fixtures/whole_applications/xarray/xarray/datatree_/docs/README.md +0 -0
|
@@ -187,3 +187,11 @@ node_modules/
|
|
|
187
187
|
|
|
188
188
|
# Track this repo's CLAUDE.md even though a global gitignore excludes CLAUDE.md
|
|
189
189
|
!CLAUDE.md
|
|
190
|
+
|
|
191
|
+
# Track AGENTS.md (a symlink to CLAUDE.md) past the same global ignore
|
|
192
|
+
!AGENTS.md
|
|
193
|
+
|
|
194
|
+
# Track the schema decision log (SDK-model input) past a global .claude ignore
|
|
195
|
+
!.claude/
|
|
196
|
+
.claude/*
|
|
197
|
+
!.claude/SCHEMA_DECISIONS.md
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: codeanalyzer-python
|
|
3
|
-
Version: 0.
|
|
4
|
-
Summary: Static
|
|
3
|
+
Version: 1.0.1
|
|
4
|
+
Summary: Static analysis for Python — canonical schema v2 (symbol table, call graph, and native CFG/PDG/SDG dataflow) as analysis.json or a Neo4j property graph.
|
|
5
5
|
Author-email: Rahul Krishna <i.m.ralk@gmail.com>
|
|
6
6
|
License-File: LICENSE
|
|
7
7
|
License-File: NOTICE
|
|
@@ -33,6 +33,8 @@ Requires-Dist: typing-extensions<6.0.0,>=4.5.0; python_version >= '3.11'
|
|
|
33
33
|
Requires-Dist: uv>=0.5.0
|
|
34
34
|
Provides-Extra: neo4j
|
|
35
35
|
Requires-Dist: neo4j<6.0.0,>=5.0.0; extra == 'neo4j'
|
|
36
|
+
Provides-Extra: scalpel
|
|
37
|
+
Requires-Dist: python-scalpel>=1.0b0; extra == 'scalpel'
|
|
36
38
|
Description-Content-Type: text/markdown
|
|
37
39
|
|
|
38
40
|
<div align="center">
|
|
@@ -41,7 +43,7 @@ Description-Content-Type: text/markdown
|
|
|
41
43
|
|
|
42
44
|
# codeanalyzer-python (`canpy`)
|
|
43
45
|
|
|
44
|
-
**A Python static-analysis toolkit — the CLDK backend that emits
|
|
46
|
+
**A Python static-analysis toolkit — the CLDK backend that emits the canonical schema v2 Code Property Graph, as `analysis.json` or a Neo4j property graph.**
|
|
45
47
|
|
|
46
48
|
[](https://pypi.org/project/codeanalyzer-python/)
|
|
47
49
|
[](https://github.com/codellm-devkit/codeanalyzer-python/releases/latest)
|
|
@@ -52,18 +54,19 @@ Description-Content-Type: text/markdown
|
|
|
52
54
|
|
|
53
55
|
---
|
|
54
56
|
|
|
55
|
-
`canpy` is a static analyzer for Python built on [Jedi](https://jedi.readthedocs.io/),
|
|
56
|
-
[
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
[CLDK](https://github.com/codellm-devkit/python-sdk), mirroring its
|
|
57
|
+
`canpy` is a static analyzer for Python built on [Jedi](https://jedi.readthedocs.io/),
|
|
58
|
+
[PyCG](https://github.com/vitsalis/PyCG), and [Tree-sitter](https://tree-sitter.github.io/). It
|
|
59
|
+
emits the **canonical CodeLLM-DevKit (CLDK) schema v2** — a single, additive Code Property Graph
|
|
60
|
+
tree — either as `analysis.json` or projected into a **Neo4j property graph**. It is the Python
|
|
61
|
+
backend behind [CLDK](https://github.com/codellm-devkit/python-sdk), mirroring its
|
|
61
62
|
[TypeScript](https://github.com/codellm-devkit/codeanalyzer-typescript) (`cants`) and
|
|
62
63
|
[Java](https://github.com/codellm-devkit/codeanalyzer-java) siblings.
|
|
63
64
|
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
65
|
+
The payload is **one tree grown one layer at a time** across four analysis levels (`-a 1|2|3|4`): a
|
|
66
|
+
symbol table, a call graph, intraprocedural control- and data-dependence graphs, and a whole-program
|
|
67
|
+
interprocedural system dependence graph. Each level is a strict superset of the one below it
|
|
68
|
+
(`analysis.json(-a 1) ⊆ … ⊆ analysis.json(-a 4)`), so a consumer can request exactly the depth it
|
|
69
|
+
needs.
|
|
67
70
|
|
|
68
71
|
## Table of Contents
|
|
69
72
|
|
|
@@ -77,6 +80,9 @@ and merges them with the Jedi-derived edges, also backfilling callees Jedi could
|
|
|
77
80
|
- [Usage](#usage)
|
|
78
81
|
- [Options](#options)
|
|
79
82
|
- [Examples](#examples)
|
|
83
|
+
- [Analysis levels](#analysis-levels)
|
|
84
|
+
- [Architecture & Tooling](#architecture--tooling)
|
|
85
|
+
- [Output shape (canonical schema v2)](#output-shape-canonical-schema-v2)
|
|
80
86
|
- [Output targets](#output-targets)
|
|
81
87
|
- [`analysis.json` (default)](#analysisjson-default)
|
|
82
88
|
- [Neo4j graph](#neo4j-graph)
|
|
@@ -86,15 +92,21 @@ and merges them with the Jedi-derived edges, also backfilling callees Jedi could
|
|
|
86
92
|
|
|
87
93
|
## Features
|
|
88
94
|
|
|
95
|
+
- **Canonical schema v2** — one additive Code Property Graph tree (`schema_version` `2.0.0`),
|
|
96
|
+
stamped with `language`, `max_level`, `analyzer{name,version}`, and (at L3+) `k_limit`, rooted
|
|
97
|
+
at a single `application` node with durable `can://` ids on every callable and above.
|
|
89
98
|
- **Symbol table** — modules, classes, functions, methods, variables, decorators, imports, and
|
|
90
|
-
docstrings, with precise source spans.
|
|
91
|
-
- **Call graph** — Jedi's lexical resolver
|
|
92
|
-
|
|
93
|
-
|
|
99
|
+
docstrings, with precise byte-offset source spans; each module carries its `source` once.
|
|
100
|
+
- **Call graph** — Jedi's lexical resolver at level 1, enriched with **PyCG**-resolved edges at
|
|
101
|
+
level 2 (provenance-tagged, coupling-aware sharding for large apps).
|
|
102
|
+
- **Dataflow graphs** — native, per-callable exceptional **CFG** plus **control-** and
|
|
103
|
+
**data-dependence** edges (`cfg`/`cdg`/`ddg`) at level 3, stitched into a whole-program
|
|
104
|
+
**interprocedural SDG** (synthetic parameter vertices, `param_in`/`param_out`/`summary`,
|
|
105
|
+
alias-aware DDG) at level 4 — all built in-process from the stdlib `ast`.
|
|
94
106
|
- **Neo4j output** — project the analysis into a labeled property graph: a self-contained
|
|
95
107
|
`graph.cypher` snapshot, or an **incremental** push to a live database over Bolt.
|
|
96
108
|
- **Versioned schema** — a machine-readable, version-stamped Neo4j schema contract (`--emit schema`),
|
|
97
|
-
checked in as `schema.neo4j.json` and shipped with every release.
|
|
109
|
+
checked in as `schema.neo4j.json` (`2.0.0`) and shipped with every release.
|
|
98
110
|
- **Incremental cache** — per-file results are cached under `.codeanalyzer`; `--lazy` (default)
|
|
99
111
|
reuses them, `--eager` forces a clean rebuild. `--ray` distributes the work across cores.
|
|
100
112
|
- **Compact output** — canonical `analysis.json`, or binary `analysis.msgpack` for smaller artifacts.
|
|
@@ -131,6 +143,13 @@ For the optional **live Neo4j push** (`--emit neo4j --neo4j-uri …`), install t
|
|
|
131
143
|
pip install 'codeanalyzer-python[neo4j]'
|
|
132
144
|
```
|
|
133
145
|
|
|
146
|
+
For the **Scalpel-backed points-to oracle** at level 4, install the `scalpel` extra. It is optional:
|
|
147
|
+
when it is absent, level 4 automatically falls back to the built-in type-based oracle.
|
|
148
|
+
|
|
149
|
+
```sh
|
|
150
|
+
pip install 'codeanalyzer-python[scalpel]'
|
|
151
|
+
```
|
|
152
|
+
|
|
134
153
|
### Install via shell script
|
|
135
154
|
|
|
136
155
|
Install the CLI as an isolated tool with the one-line installer (provisions via uv / pipx / pip):
|
|
@@ -184,19 +203,19 @@ $ canpy --help
|
|
|
184
203
|
│ --version Show the canpy │
|
|
185
204
|
│ version and │
|
|
186
205
|
│ exit. │
|
|
187
|
-
│ --input -i
|
|
206
|
+
│ --input -i <path> Path to the │
|
|
188
207
|
│ project root │
|
|
189
208
|
│ directory (not │
|
|
190
209
|
│ required for │
|
|
191
210
|
│ --emit schema). │
|
|
192
|
-
│ --output -o
|
|
211
|
+
│ --output -o <path> Output directory │
|
|
193
212
|
│ for artifacts. │
|
|
194
|
-
│ --format -f
|
|
213
|
+
│ --format -f <json|msgpack> Output format │
|
|
195
214
|
│ for --emit json: │
|
|
196
215
|
│ json or msgpack. │
|
|
197
216
|
│ [default: json] │
|
|
198
|
-
│ --emit
|
|
199
|
-
│ ma
|
|
217
|
+
│ --emit <json|neo4j|sche Output target: │
|
|
218
|
+
│ ma> json │
|
|
200
219
|
│ (analysis.json, │
|
|
201
220
|
│ default) | neo4j │
|
|
202
221
|
│ (graph.cypher or │
|
|
@@ -206,13 +225,13 @@ $ canpy --help
|
|
|
206
225
|
│ schema.json │
|
|
207
226
|
│ contract). │
|
|
208
227
|
│ [default: json] │
|
|
209
|
-
│ --app-name
|
|
228
|
+
│ --app-name <str> Logical │
|
|
210
229
|
│ application name │
|
|
211
230
|
│ for the graph │
|
|
212
231
|
│ :PyApplication │
|
|
213
232
|
│ anchor (default: │
|
|
214
233
|
│ input dir name). │
|
|
215
|
-
│ --neo4j-uri
|
|
234
|
+
│ --neo4j-uri <str> Push the graph │
|
|
216
235
|
│ to a live Neo4j │
|
|
217
236
|
│ over Bolt │
|
|
218
237
|
│ (incremental); │
|
|
@@ -220,11 +239,11 @@ $ canpy --help
|
|
|
220
239
|
│ graph.cypher. │
|
|
221
240
|
│ [env var: │
|
|
222
241
|
│ NEO4J_URI] │
|
|
223
|
-
│ --neo4j-user
|
|
242
|
+
│ --neo4j-user <str> Neo4j username. │
|
|
224
243
|
│ [env var: │
|
|
225
244
|
│ NEO4J_USERNAME] │
|
|
226
245
|
│ [default: neo4j] │
|
|
227
|
-
│ --neo4j-password
|
|
246
|
+
│ --neo4j-password <str> Neo4j password. │
|
|
228
247
|
│ Prefer the env │
|
|
229
248
|
│ var over the │
|
|
230
249
|
│ flag (the flag │
|
|
@@ -234,17 +253,53 @@ $ canpy --help
|
|
|
234
253
|
│ [env var: │
|
|
235
254
|
│ NEO4J_PASSWORD] │
|
|
236
255
|
│ [default: neo4j] │
|
|
237
|
-
│ --neo4j-database
|
|
256
|
+
│ --neo4j-database <str> Neo4j database │
|
|
238
257
|
│ name (default: │
|
|
239
258
|
│ server default). │
|
|
240
259
|
│ [env var: │
|
|
241
260
|
│ NEO4J_DATABASE] │
|
|
242
|
-
│ --analysis-level -a
|
|
243
|
-
│ [1<=x<=
|
|
261
|
+
│ --analysis-level -a <int range> Analysis depth: │
|
|
262
|
+
│ [1<=x<=4] 1=symbol │
|
|
244
263
|
│ table+Jedi call │
|
|
245
264
|
│ graph, 2=+PyCG │
|
|
246
|
-
│ call graph
|
|
265
|
+
│ call graph, │
|
|
266
|
+
│ 3=+native │
|
|
267
|
+
│ intraprocedural │
|
|
268
|
+
│ dataflow │
|
|
269
|
+
│ (CFG/PDG), │
|
|
270
|
+
│ 4=+interprocedu… │
|
|
271
|
+
│ SDG │
|
|
272
|
+
│ (param/summary │
|
|
273
|
+
│ edges, │
|
|
274
|
+
│ alias-aware │
|
|
275
|
+
│ DDG). │
|
|
247
276
|
│ [default: 1] │
|
|
277
|
+
│ --graphs <str> Level 3+ only: │
|
|
278
|
+
│ comma-separated │
|
|
279
|
+
│ program-graph │
|
|
280
|
+
│ sections to emit │
|
|
281
|
+
│ (cfg, dfg, pdg, │
|
|
282
|
+
│ sdg). Default: │
|
|
283
|
+
│ cfg,dfg,pdg. │
|
|
284
|
+
│ `dfg` emits the │
|
|
285
|
+
│ PDG's data edges │
|
|
286
|
+
│ only; `sdg` │
|
|
287
|
+
│ requires -a 4. │
|
|
288
|
+
│ [default: │
|
|
289
|
+
│ cfg,dfg,pdg] │
|
|
290
|
+
│ --graph-field-de… <int range> Level 3 only: │
|
|
291
|
+
│ [x>=1] k-limit on │
|
|
292
|
+
│ access-path │
|
|
293
|
+
│ depth (x.f.g.h │
|
|
294
|
+
│ with k=3 becomes │
|
|
295
|
+
│ x.f.g.*). │
|
|
296
|
+
│ Mandatory bound │
|
|
297
|
+
│ — it is what │
|
|
298
|
+
│ guarantees the │
|
|
299
|
+
│ interprocedural │
|
|
300
|
+
│ fixpoint │
|
|
301
|
+
│ terminates. │
|
|
302
|
+
│ [default: 3] │
|
|
248
303
|
│ --ray --no-ray Enable Ray for │
|
|
249
304
|
│ distributed │
|
|
250
305
|
│ analysis. │
|
|
@@ -269,12 +324,12 @@ $ canpy --help
|
|
|
269
324
|
│ environment │
|
|
270
325
|
│ instead. │
|
|
271
326
|
│ [default: venv] │
|
|
272
|
-
│ --file-name
|
|
327
|
+
│ --file-name <path> Analyze only the │
|
|
273
328
|
│ specified file │
|
|
274
329
|
│ (relative to │
|
|
275
330
|
│ input │
|
|
276
331
|
│ directory). │
|
|
277
|
-
│ --cache-dir -c
|
|
332
|
+
│ --cache-dir -c <path> Directory to │
|
|
278
333
|
│ store analysis │
|
|
279
334
|
│ cache. Defaults │
|
|
280
335
|
│ to │
|
|
@@ -288,7 +343,7 @@ $ canpy --help
|
|
|
288
343
|
│ retained. │
|
|
289
344
|
│ [default: │
|
|
290
345
|
│ keep-cache] │
|
|
291
|
-
│ -v
|
|
346
|
+
│ -v <int> Increase │
|
|
292
347
|
│ verbosity: -v, │
|
|
293
348
|
│ -vv, -vvv │
|
|
294
349
|
│ [default: 0] │
|
|
@@ -315,7 +370,7 @@ $ canpy --help
|
|
|
315
370
|
│ Jedi-only edges. │
|
|
316
371
|
│ [default: │
|
|
317
372
|
│ no-pycg-shard] │
|
|
318
|
-
│ --pycg-shard-cei…
|
|
373
|
+
│ --pycg-shard-cei… <int range> Maximum files │
|
|
319
374
|
│ [x>=1] per shard when │
|
|
320
375
|
│ --pycg-shard is │
|
|
321
376
|
│ active (default │
|
|
@@ -337,7 +392,7 @@ $ canpy --help
|
|
|
337
392
|
│ heavy import │
|
|
338
393
|
│ graphs. │
|
|
339
394
|
│ [default: 100] │
|
|
340
|
-
│ --pycg-shard-tim…
|
|
395
|
+
│ --pycg-shard-tim… <int range> Per-shard │
|
|
341
396
|
│ [x>=0] wall-clock │
|
|
342
397
|
│ timeout in │
|
|
343
398
|
│ seconds when │
|
|
@@ -365,7 +420,7 @@ $ canpy --help
|
|
|
365
420
|
│ ignored on │
|
|
366
421
|
│ Windows. │
|
|
367
422
|
│ [default: 120] │
|
|
368
|
-
│ --pycg-shard-str…
|
|
423
|
+
│ --pycg-shard-str… <jedi|package> How --pycg-shard │
|
|
369
424
|
│ groups files │
|
|
370
425
|
│ (level 2 only). │
|
|
371
426
|
│ 'jedi' (default) │
|
|
@@ -387,7 +442,7 @@ $ canpy --help
|
|
|
387
442
|
│ one-shard-per-p… │
|
|
388
443
|
│ grouping. │
|
|
389
444
|
│ [default: jedi] │
|
|
390
|
-
│ --pycg-max-iter
|
|
445
|
+
│ --pycg-max-iter <int range> Cap on PyCG's │
|
|
391
446
|
│ [x>=-1] fixpoint passes │
|
|
392
447
|
│ per │
|
|
393
448
|
│ shard/project │
|
|
@@ -441,14 +496,13 @@ $ canpy --help
|
|
|
441
496
|
canpy --input ./my-python-project --output ./out --format msgpack # → ./out/analysis.msgpack
|
|
442
497
|
```
|
|
443
498
|
|
|
444
|
-
3. **
|
|
499
|
+
3. **Enrich the call graph with PyCG (level 2):**
|
|
445
500
|
```sh
|
|
446
|
-
canpy --input ./my-python-project
|
|
501
|
+
canpy --input ./my-python-project -a 2
|
|
447
502
|
```
|
|
448
|
-
|
|
449
|
-
(
|
|
450
|
-
Jedi
|
|
451
|
-
integration is experimental; the CLI is downloaded into `<cache_dir>/codeql/` on first use.
|
|
503
|
+
Level 1 edges come from Jedi's lexical resolution. `-a 2` runs **PyCG** and merges its
|
|
504
|
+
flow-sensitive edges in (RPC / third-party / dynamically-dispatched targets), backfilling
|
|
505
|
+
callees Jedi could not resolve. Every edge is provenance-tagged (e.g. `jedi`, `pycg`).
|
|
452
506
|
|
|
453
507
|
4. **Emit a Neo4j snapshot, or push to a live database:**
|
|
454
508
|
```sh
|
|
@@ -468,31 +522,163 @@ $ canpy --help
|
|
|
468
522
|
canpy --input ./my-python-project --eager --cache-dir /path/to/custom-cache
|
|
469
523
|
```
|
|
470
524
|
|
|
471
|
-
|
|
472
|
-
|
|
473
|
-
|
|
525
|
+
7. **Dataflow graphs — intraprocedural (level 3) and interprocedural (level 4):**
|
|
526
|
+
```sh
|
|
527
|
+
canpy --input ./my-python-project -a 3 --output ./out # per-callable cfg/cdg/ddg
|
|
528
|
+
canpy --input ./my-python-project -a 4 --output ./out # + interprocedural SDG
|
|
529
|
+
canpy --input ./my-python-project -a 3 --graphs cfg,pdg # scope the emitted sections
|
|
530
|
+
canpy --input ./my-python-project -a 4 --graphs sdg # sdg requires -a 4
|
|
531
|
+
canpy --input ./my-python-project -a 3 --graph-field-depth 2 # tighter access-path k-limit
|
|
532
|
+
```
|
|
533
|
+
Levels 3 and 4 also enrich the Neo4j projection (`--emit neo4j`) with the CPG overlay
|
|
534
|
+
(`:PyCFGNode` nodes wired by `PY_CFG_NEXT`/`PY_CDG`/`PY_DDG`, plus the level-4
|
|
535
|
+
`PY_PARAM_IN`/`PY_PARAM_OUT`/`PY_SUMMARY` edges — the cross-language dataflow vocabulary,
|
|
536
|
+
PY_-namespaced like every other row family so multi-language databases never mingle
|
|
537
|
+
analyzers' edges).
|
|
538
|
+
|
|
539
|
+
## Analysis levels
|
|
540
|
+
|
|
541
|
+
Each level is the same tree grown one layer deeper, plus the edge family over that new layer. The
|
|
542
|
+
levels are cumulative and additive — `analysis.json(-a 1) ⊆ … ⊆ analysis.json(-a 4)`.
|
|
543
|
+
|
|
544
|
+
| Level | Flag | What it adds | Where it lands |
|
|
545
|
+
| --- | --- | --- | --- |
|
|
546
|
+
| **1** | `-a 1` (default) | Symbol table, Jedi call graph, and `call` nodes in each callable's `body` | `body` calls (`callee: null`) |
|
|
547
|
+
| **2** | `-a 2` | PyCG call-graph enrichment; each call's `callee` backfilled to a `can://` id | `call_graph`, `body` callees |
|
|
548
|
+
| **3** | `-a 3` | Native **intraprocedural** CFG/CDG/DDG (syntactic, name-equality, `prov: ["ssa"]`) | `cfg`, `cdg`, `ddg`, `@entry`/`@exit` on each callable |
|
|
549
|
+
| **4** | `-a 4` | **Interprocedural** SDG: synthetic param vertices, alias-aware DDG (`prov: ["points-to"]`) | `param_in`, `param_out`, `summary`, semantic `ddg` |
|
|
550
|
+
|
|
551
|
+
`-a 1`/`-a 2` timings and output are unaffected by the heavier levels — nothing at level 3+ runs
|
|
552
|
+
unless requested. Flag gating: `--graphs sdg` requires `-a 4`; `--graphs cfg,dfg,pdg` and
|
|
553
|
+
`--graph-field-depth` require `-a 3`.
|
|
554
|
+
|
|
555
|
+
## Architecture & Tooling
|
|
556
|
+
|
|
557
|
+
The dataflow substrate is hand-built from the standard library so every graph node joins back to a
|
|
558
|
+
symbol-table signature by construction
|
|
559
|
+
([#67](https://github.com/codellm-devkit/codeanalyzer-python/issues/67)):
|
|
560
|
+
|
|
561
|
+
- **CFG source:** a hand-built **exceptional** control-flow graph from the stdlib `ast` module — the
|
|
562
|
+
same parse the symbol-table builder uses. One synthetic `@entry`/`@exit` per callable,
|
|
563
|
+
statement-level nodes keyed `line:col` in source order, with exception / `yield` / `await` edges
|
|
564
|
+
first-class.
|
|
565
|
+
- **Def-use source:** hand-built **reaching definitions** (a classic forward worklist) over
|
|
566
|
+
k-limited access paths (`--graph-field-depth`, default 3) — there is no usable SSA library for
|
|
567
|
+
Python. This yields the level-3 syntactic DDG (name-equality, `prov: ["ssa"]`).
|
|
568
|
+
- **Points-to oracle (level 4):** the **Scalpel** may-alias oracle — `ScalpelAliasOracle`
|
|
569
|
+
(`codeanalyzer/dataflow/scalpel_oracle.py`) — consumes Scalpel's SSA copy/const facts to answer
|
|
570
|
+
`may_alias(path_a, path_b)`, adding the alias-aware DDG edges (`prov: ["points-to"]`) and the
|
|
571
|
+
interprocedural summaries. `python-scalpel` is an **optional dependency**
|
|
572
|
+
(`pip install 'codeanalyzer-python[scalpel]'`); when it is absent or cannot resolve a construct,
|
|
573
|
+
the analyzer automatically falls back to the built-in `TypeBasedAliasOracle` (Jedi-inferred types;
|
|
574
|
+
unknown types conservatively alias), keeping the `may_alias` interface total. Call dispatch comes
|
|
575
|
+
from the merged Jedi(+PyCG) call graph, treated as a frozen oracle.
|
|
576
|
+
- **Summaries:** relational formal-in → formal-out flows composed bottom-up over the Tarjan SCC
|
|
577
|
+
condensation of the call graph, a monotone fixpoint within SCCs; globals ride as extra formals,
|
|
578
|
+
closure captures bind at definition sites.
|
|
579
|
+
- **Slicing and taint are the SDK's responsibility.** A backward slicer ships in-process
|
|
580
|
+
(`codeanalyzer.dataflow.slicing`), but only as an **internal validation utility** for the L3/L4
|
|
581
|
+
gates — it is not a product surface. Once the SDG is emitted, slicing and taint become
|
|
582
|
+
language-independent labeled reachability and belong to the CLDK SDK across the provider/client
|
|
583
|
+
boundary; the analyzer emits the `summary` substrate and **no `taint_flows` section**.
|
|
584
|
+
- **Precision posture:** sound-leaning and over-approximate — prefer false positives to missed
|
|
585
|
+
flows. **Known unsoundness (documented, not silently absorbed):** `eval`/`exec`, reflection
|
|
586
|
+
(`getattr`/`setattr` with dynamic names), monkey-patching, C extensions, `import` side effects,
|
|
587
|
+
and module top-level statements (globals are modeled as formals instead).
|
|
588
|
+
|
|
589
|
+
## Output shape (canonical schema v2)
|
|
590
|
+
|
|
591
|
+
Every run produces the same envelope — an `Analysis` document — regardless of level; deeper levels
|
|
592
|
+
just populate more of the same tree:
|
|
474
593
|
|
|
475
|
-
|
|
594
|
+
```jsonc
|
|
595
|
+
{
|
|
596
|
+
"schema_version": "2.0.0",
|
|
597
|
+
"language": "python",
|
|
598
|
+
"max_level": 4, // the level this run was produced at
|
|
599
|
+
"k_limit": 3, // access-path depth bound (--graph-field-depth); L3+ only
|
|
600
|
+
"analyzer": { "name": "codeanalyzer-python", "version": "1.0.0" },
|
|
601
|
+
"application": {
|
|
602
|
+
"id": "can://python/<app>",
|
|
603
|
+
"kind": "application",
|
|
604
|
+
"symbol_table": { // relative POSIX path → module
|
|
605
|
+
"pkg/mod.py": {
|
|
606
|
+
"id": "can://python/<app>/pkg/mod.py",
|
|
607
|
+
"kind": "module",
|
|
608
|
+
"source": "…full file text, stored once per module…",
|
|
609
|
+
"types": { "<Class>": { "id": "…", "kind": "class", "callables": { /* methods */ } } },
|
|
610
|
+
"functions": { "<sig>": { /* callable, see below */ } }
|
|
611
|
+
}
|
|
612
|
+
},
|
|
613
|
+
"call_graph": [ { "src": "can://…/main(a)", "dst": "can://…/helper(x)",
|
|
614
|
+
"weight": 1, "prov": ["jedi", "pycg"] } ],
|
|
615
|
+
"external_symbols": { // imported/builtin call targets, keyed by id
|
|
616
|
+
"can://python/<app>/@external/os/getcwd":
|
|
617
|
+
{ "id": "can://python/<app>/@external/os/getcwd", "kind": "external",
|
|
618
|
+
"name": "getcwd", "module": "os" }
|
|
619
|
+
},
|
|
620
|
+
"param_in": [ { "src": "can://…/main(a)@6:4/actual_in:0", "dst": "can://…/helper(x)@formal_in:0" } ],
|
|
621
|
+
"param_out": [ { "src": "can://…/helper(x)@formal_out", "dst": "can://…/main(a)@6:4/actual_out" } ]
|
|
622
|
+
}
|
|
623
|
+
}
|
|
624
|
+
```
|
|
476
625
|
|
|
477
|
-
A
|
|
626
|
+
A **callable** (function or method) carries its own CPG, keyed by node id:
|
|
478
627
|
|
|
479
628
|
```jsonc
|
|
480
629
|
{
|
|
481
|
-
"
|
|
482
|
-
"
|
|
630
|
+
"id": "can://…/main(a)", "kind": "function",
|
|
631
|
+
"span": { "start": [5, 0], "end": [7, 12], "bytes": [43, 86] }, // byte offsets into module.source
|
|
632
|
+
"body": { // node id → node
|
|
633
|
+
"@entry": { "kind": "entry" },
|
|
634
|
+
"6:4": { "kind": "statement", "span": { … } },
|
|
635
|
+
"6:8": { "kind": "call", "span": { … }, "callee": "can://…/helper(x)" }, // callee null until L2
|
|
636
|
+
"@formal_in:0": { "kind": "formal_in", "of": "a" }, // L4 param vertices
|
|
637
|
+
"6:4/actual_in:0": { "kind": "actual_in", "of": "a", "parent": "6:4" },
|
|
638
|
+
"@exit": { "kind": "exit" }
|
|
639
|
+
},
|
|
640
|
+
"cfg": [ { "src": "@entry", "dst": "6:4", "kind": "fallthrough" } ], // L3
|
|
641
|
+
"cdg": [ { "src": "@entry", "dst": "6:4" } ], // L3
|
|
642
|
+
"ddg": [ { "src": "6:4", "dst": "7:4", "var": "h", "prov": ["ssa"] } ], // L3 ssa / L4 points-to
|
|
643
|
+
"summary": [ { "src": "6:4/actual_in:0", "dst": "6:4/actual_out" } ] // L4
|
|
483
644
|
}
|
|
484
645
|
```
|
|
485
646
|
|
|
486
|
-
|
|
487
|
-
|
|
647
|
+
Notable properties:
|
|
648
|
+
|
|
649
|
+
- **Durable `can://` ids** identify every node at callable granularity and above
|
|
650
|
+
(`can://python/<app>/<file>/<callable-sig>`); nodes below a callable use ordinal ids
|
|
651
|
+
(`@entry`, `@exit`, `line:col`, `@formal_in:N`, `line:col/actual_in:N`).
|
|
652
|
+
- **`source` lives once per module**; every node's text is the `module.source[span.bytes]` slice.
|
|
653
|
+
- **Cross-function edges** — `call_graph`, `param_in`, `param_out` — live at **application** scope;
|
|
654
|
+
the intraprocedural `cfg`/`cdg`/`ddg` and the `summary` edges live **on the callable**.
|
|
655
|
+
- **No dangling endpoints** — every `call_graph` `src`/`dst` joins the id space: declared
|
|
656
|
+
callables by their tree id, imported/builtin targets by a `…/@external/<module>/<name>` id
|
|
657
|
+
homed in `application.external_symbols`.
|
|
658
|
+
- **Breaking change from v1:** there is no more flat top-level `symbol_table`/`call_graph`, and no
|
|
659
|
+
separate program-graphs section. Everything now hangs off `application`, and the dataflow graphs
|
|
660
|
+
are inlined on each callable. Read `analysis.application.symbol_table` (was
|
|
661
|
+
`analysis.symbol_table`) and `analysis.application.call_graph` (was `analysis.call_graph`).
|
|
662
|
+
|
|
663
|
+
## Output targets
|
|
664
|
+
|
|
665
|
+
`canpy` builds one analysis in memory and can emit it three ways (`--emit`):
|
|
666
|
+
|
|
667
|
+
### `analysis.json` (default)
|
|
668
|
+
|
|
669
|
+
The `Analysis` envelope described above. By default it is printed to stdout as JSON; with `--output`
|
|
670
|
+
it is written to `analysis.json` (or `analysis.msgpack` with `--format msgpack`, a more compact
|
|
671
|
+
binary format).
|
|
488
672
|
|
|
489
673
|
### Neo4j graph
|
|
490
674
|
|
|
491
|
-
`--emit neo4j` projects the same analysis into a labeled property graph. Every node
|
|
492
|
-
`Py`-prefixed and every relationship type is `PY_`-prefixed (e.g. `:PyClass`, `PY_CALLS`)
|
|
493
|
-
language analyzers can share one database without label or relationship-type collisions.
|
|
494
|
-
are keyed by their
|
|
495
|
-
decorators, and call sites are relationships
|
|
675
|
+
`--emit neo4j` projects the same schema v2.0.0 analysis into a labeled property graph. Every node
|
|
676
|
+
label is `Py`-prefixed and every relationship type is `PY_`-prefixed (e.g. `:PyClass`, `PY_CALLS`)
|
|
677
|
+
so multiple language analyzers can share one database without label or relationship-type collisions.
|
|
678
|
+
Declarations are keyed by their **`can://` id** under a shared `:PySymbol` label; calls, imports,
|
|
679
|
+
inheritance, decorators, and call sites are relationships. At `-a 3`/`-a 4` the projection gains the
|
|
680
|
+
**CPG overlay** — `:PyCFGNode` nodes (statements, and at level 4 the parameter vertices) wired by
|
|
681
|
+
`PY_CFG_NEXT`/`PY_CDG`/`PY_DDG`, plus the level-4 `PY_PARAM_IN`/`PY_PARAM_OUT`/`PY_SUMMARY` edges:
|
|
496
682
|
|
|
497
683
|
- **Without `--neo4j-uri`** — writes a self-contained `graph.cypher` (constraints + indexes, a scoped
|
|
498
684
|
wipe, then batched `MERGE`s). Load it with `cypher-shell < graph.cypher`. Needs no extra
|
|
@@ -519,10 +705,11 @@ canpy -i ./my-project --emit neo4j # credentials picked up from the environm
|
|
|
519
705
|
### Schema contract
|
|
520
706
|
|
|
521
707
|
`--emit schema` writes the machine-readable, version-stamped Neo4j schema (`schema.json`: node labels,
|
|
522
|
-
relationships, properties, constraints, and indexes). It needs no
|
|
523
|
-
as `schema.neo4j.json` and bundled in every release as a GitHub
|
|
524
|
-
validate producer/consumer compatibility without invoking the tool.
|
|
525
|
-
the
|
|
708
|
+
relationships, properties, constraints, and indexes; currently `schema_version` `2.0.0`). It needs no
|
|
709
|
+
project and is checked into the repo as `schema.neo4j.json` and bundled in every release as a GitHub
|
|
710
|
+
Release asset, so a consumer can validate producer/consumer compatibility without invoking the tool.
|
|
711
|
+
The shape of the contract matches the
|
|
712
|
+
[`codeanalyzer-typescript`](https://github.com/codellm-devkit/codeanalyzer-typescript) backend.
|
|
526
713
|
|
|
527
714
|
A UML of the `analysis.json` schema (the `PyApplication` containment tree) is checked in as
|
|
528
715
|
[`schema-uml.drawio`](./schema-uml.drawio), and the property-graph schema as
|