trailmark 0.1.3__tar.gz → 0.2.2__tar.gz

{trailmark-0.1.3 → trailmark-0.2.2}/AGENTS.md

@@ -5,9 +5,10 @@ Development context for AI agents working on this codebase.
 ## Project Overview
 
 Trailmark parses source code into directed graphs of functions, classes,
-calls, and semantic metadata for security analysis. It supports 16
-languages via tree-sitter (plus a bundled Circom grammar) and uses
-rustworkx for graph traversal.
+calls, and semantic metadata for security analysis. It supports more than
+twenty language frontends via tree-sitter-language-pack plus bundled
+custom grammars, and uses rustworkx for graph traversal. Treat
+`README.md` as the authoritative public feature list.
 
 ## Architecture
 
@@ -19,17 +20,22 @@ CodeGraph (data) -> GraphStore (indexed storage) -> QueryEngine (facade)
 - **GraphStore** wraps CodeGraph in a rustworkx PyDiGraph. Validates
   node existence. Returns model objects.
 - **QueryEngine** resolves names to node IDs, delegates to GraphStore,
-  returns plain dicts for JSON serialization.
+  and returns JSON-serializable data structures for user-facing APIs.
 
 ## Key Conventions
 
-- All public methods return `False` or `[]` for missing nodes. Never raise.
-- QueryEngine returns dicts; GraphStore returns model objects.
+- Query/read methods should return `False`, `[]`, or empty sets for
+  missing nodes where practical. Factory methods and invalid-input paths
+  may raise `ValueError` or exit via the CLI.
+- QueryEngine returns JSON-serializable data structures; GraphStore
+  returns model objects and node IDs.
 - Node IDs follow `module:function`, `module:Class`, `module:Class.method`.
 - Edge confidence: `certain`, `inferred`, `uncertain`.
 - Annotation sources: `"llm"`, `"docstring"`, `"manual"`.
 - Frozen dataclasses for immutable data. `CodeGraph` is mutable.
 - No relative (`..`) imports. Use `from trailmark.*`.
+- Update `README.md` and `tests/test_readme.py` when documented behavior
+  changes.
 
 ## File Layout
 
@@ -46,6 +52,7 @@ src/trailmark/
     python/        # One subpackage per language
     javascript/
     ...
+  analysis/        # Higher-level passes: entrypoints, diff, augment, preanalysis
   storage/
     graph_store.py # GraphStore: rustworkx-backed indexed storage
   query/
@@ -57,10 +64,11 @@ tests/             # pytest test suite
 ## Running Checks
 
 ```bash
-uv run ruff check --fix src/ tests/
-uv run ruff format src/ tests/
-uv run ty check
-pytest -q
+uv sync --all-groups
+uv run ruff check src/ tests/
+uv run ruff format --check src/ tests/
+uv tool install ty && ty check
+uv run pytest -q tests/
 ```
 
 ## Mutation Testing
@@ -85,6 +93,10 @@ This is not needed on Linux/CI (Ubuntu).
 - Follow the three-layer pattern: add to CodeGraph first, then GraphStore
   (with validation), then QueryEngine (with name resolution and dict
   conversion).
-- Add tests at each layer: `test_models.py`, `test_storage.py`,
-  `test_query.py`.
-- Update `README.md` if adding user-facing API.
+- Analysis passes live under `trailmark.analysis` and should usually
+  persist results as annotations or named subgraphs rather than
+  bypassing the graph model.
+- Add tests at each relevant layer: `test_models.py`, `test_storage.py`,
+  `test_query.py`, plus focused parser/analysis/doc regression tests.
+- Update `README.md` and the matching regression tests if adding or
+  changing user-facing API.

{trailmark-0.1.3 → trailmark-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: trailmark
-Version: 0.1.3
+Version: 0.2.2
 Summary: Parse source code into a queryable graph of functions, classes, calls, and semantic annotations
 Project-URL: Homepage, https://github.com/trailofbits/trailmark
 Project-URL: Repository, https://github.com/trailofbits/trailmark
@@ -60,7 +60,7 @@ A language-specific parser walks the directory, parses each file into a tree-sit
 | Language | Extensions | Key constructs |
 | --- | --- | --- |
 | Python | `.py` | functions, classes, methods |
-| JavaScript | `.js`, `.jsx` | functions, classes, arrow functions |
+| JavaScript | `.js`, `.jsx`, `.mjs`, `.cjs` | functions, classes, arrow functions |
 | TypeScript | `.ts`, `.tsx` | functions, classes, interfaces, enums |
 | PHP | `.php` | functions, classes, interfaces, traits |
 | Ruby | `.rb` | methods, classes, modules |
@@ -76,6 +76,10 @@ A language-specific parser walks the directory, parses each file into a tree-sit
 | Haskell | `.hs` | functions, data types, type classes, instances |
 | Erlang | `.erl` | functions, records, behaviours, modules |
 | Miden Assembly | `.masm` | procedures, entrypoints, constants, invocations |
+| Swift | `.swift` | functions, classes, structs, enums, protocols, extensions |
+| Objective-C | `.m`, `.mm`, `.h` | C functions, classes, methods (selector-based naming) |
+| Kotlin | `.kt`, `.kts` | functions, classes, interfaces, data classes, objects, methods |
+| Dart | `.dart` | functions, classes, abstract classes, methods, constructors |
 
 ```mermaid
 flowchart TD
@@ -114,15 +118,27 @@ The `GraphStore` loads the `CodeGraph` into a rustworkx `PyDiGraph` and builds b
 The `QueryEngine` provides a high-level API over the indexed graph:
 
 | Method | Description |
-|---|---|
-| `callers_of(name)` | All functions that call the named target |
-| `callees_of(name)` | All functions called by the named source |
+| --- | --- |
+| `callers_of(name)` | Direct callers of the named target |
+| `callees_of(name)` | Direct callees of the named source |
+| `ancestors_of(name)` | Every function that can transitively reach the target (upward slice) |
+| `reachable_from(name)` | Every function transitively reachable from the source |
 | `paths_between(src, dst)` | All simple call paths between two nodes |
+| `entrypoint_paths_to(name)` | Paths from any detected entrypoint to the target |
 | `attack_surface()` | Entrypoints tagged with trust level and asset value |
 | `complexity_hotspots(n)` | Functions with cyclomatic complexity ≥ n |
-| `annotate(name, kind, desc, source)` | Add a semantic annotation to a node |
+| `functions_that_raise(exc)` | Functions whose parser-detected exception list includes `exc` |
+| `annotate(name, kind, description, source)` | Add a semantic annotation to a node |
 | `annotations_of(name, kind=None)` | Get annotations for a node, optionally filtered by kind |
+| `nodes_with_annotation(kind)` | Every node tagged with the given annotation kind |
 | `clear_annotations(name, kind=None)` | Remove annotations from a node |
+| `diff_against(other)` | Structural diff of this engine's graph vs. another |
+| `preanalysis()` | Run the built-in pre-analysis passes and store annotations/subgraphs |
+| `augment_sarif(path)` | Merge SARIF findings into the graph |
+| `augment_weaudit(path)` | Merge weAudit findings into the graph |
+| `findings(kind=None)` | Return nodes carrying finding-style annotations |
+| `subgraph(name)` | Return the nodes in a named subgraph |
+| `subgraph_names()` | List every named subgraph currently on the graph |
 | `summary()` | Node counts, edge counts, dependencies |
 | `to_json()` | Full graph export |
 
@@ -227,15 +243,27 @@ graph TD
 
 ## Installation
 
+The examples below track the current development branch. For the latest
+published package, install from PyPI. For the exact feature set described
+here, install from a checkout and run commands via `uv run`.
+
 ```bash
+# Latest published release
 uv pip install trailmark
+
+# Current checkout / development branch
+uv sync --all-groups
 ```
 
-Requires Python ≥ 3.13.
+Requires Python ≥ 3.12.
 
 ## Usage
 
 ```bash
+# Report the installed version
+trailmark --version     # or: trailmark -V
+trailmark version       # subcommand form
+
 # Full JSON graph (Python, the default)
 trailmark analyze path/to/project
 
@@ -243,33 +271,148 @@ trailmark analyze path/to/project
 trailmark analyze --language rust path/to/project
 trailmark analyze --language javascript path/to/project
 
+# Polyglot: auto-detect and merge every supported language found in the
+# tree, or pass an explicit comma-separated list.
+trailmark analyze --language auto path/to/project
+trailmark analyze --language python,rust,solidity path/to/project
+
 # Summary statistics
 trailmark analyze --summary path/to/project
 
 # Complexity hotspots (threshold >= 10)
 trailmark analyze --complexity 10 path/to/project
+
+# Augment the graph with external findings (SARIF from static analyzers,
+# weAudit findings from the VS Code extension). Each --sarif / --weaudit
+# flag is repeatable. Add --json to print the augmented graph.
+trailmark augment --sarif results.sarif path/to/project
+trailmark augment --weaudit findings.json path/to/project
+trailmark augment --sarif a.sarif --sarif b.sarif --json path/to/project
+
+# List detected entrypoints (attack surface). Uses heuristic detection
+# (main() functions, pyproject.toml [project.scripts]) plus an optional
+# override file at .trailmark/entrypoints.toml (see below).
+trailmark entrypoints path/to/project
+trailmark entrypoints --json path/to/project
+
+# Structural diff between two code graphs. Accepts directory paths or
+# git refs (branches, tags, commits). Surfaces added/removed nodes,
+# call-edge changes, and — most usefully — attack-surface changes.
+trailmark diff before/ after/
+trailmark diff --repo . main HEAD          # compare git refs
+trailmark diff --json before/ after/        # machine-readable output
+```
+
+### Entrypoint detection
+
+Trailmark automatically populates `graph.entrypoints` so `attack_surface()`, taint propagation, and privilege-boundary crossing have data to work with. Detection runs in four layers, each overriding the last:
+
+1. **Generic `main` heuristic.** Any function named `main` in any language. Tagged `user_input` / `trusted_internal` / `low`.
+2. **Framework-aware scan.** Decorator, attribute, and visibility patterns per language — see the table below.
+3. **`pyproject.toml [project.scripts]`.** Explicit CLI targets get an upgraded trust/asset classification.
+4. **Repo-local override file.** Hand-curated entrypoints in `.trailmark/entrypoints.toml` always win.
+
+Framework coverage:
+
+| Language | Frameworks detected |
+| --- | --- |
+| Python | Flask, FastAPI, aiohttp, Click, Typer, Celery |
+| JavaScript / TypeScript | NestJS, Next.js (App Router + Pages API), AWS Lambda |
+| Java | Spring MVC / WebFlux, JAX-RS, Kafka listeners, servlets |
+| C# | ASP.NET Core, Azure Functions |
+| PHP | Symfony `#[Route]` attributes + legacy annotations |
+| Rust | actix-web, rocket, FFI exports (`#[no_mangle]`, `pub extern "C"`), async-main attributes |
+| Solidity | `external` / `public` visibility |
+| Cairo / StarkNet | `#[external]`, `#[view]`, `#[l1_handler]`, `#[constructor]` |
+| Circom | `component main` declarations |
+| Miden Assembly | `export.<name>` directives |
+| Haskell | top-level `main ::` / `main =` |
+| Erlang | functions listed in `-export([...])` |
+| Swift | `@main` app attribute |
+| Objective-C | `UIApplicationDelegate` lifecycle selectors (e.g. `application:openURL:options:`) |
+| Kotlin | Spring MVC / WebFlux annotations (shared with Java), Android component lifecycle methods (`onCreate`, `onReceive`, `onBind`, ...) |
+| Dart | `@pragma('vm:entry-point')` native-callable markers |
+| Go | `http.HandleFunc` / `http.Handle` stdlib registrations, gin/chi/echo-style `<router>.GET/POST/...` handler registrations |
+| Ruby | Rails controller actions (classes inheriting `ApplicationController` / `ActionController::*`), Sidekiq worker `perform` methods |
+| C / C++ | `extern "C"` linkage, `__attribute__((visibility("default")))`, `__declspec(dllexport)` |
+
+For anything the heuristics miss, declare entrypoints explicitly in `.trailmark/entrypoints.toml` at the project root. The file supports both single-node and rule-based entries:
+
+```toml
+# Single-node entry
+[[entrypoint]]
+node = "my_module:handle_request"  # node id, or "module.path:function"
+kind = "api"                       # user_input | api | database | file_system | third_party
+trust = "untrusted_external"       # untrusted_external | semi_trusted_external | trusted_internal
+asset_value = "high"               # high | medium | low
+description = "HTTP POST /auth"
+
+# Rule: every PHP script under public_html/ is a web-exposed entrypoint.
+[[entrypoint]]
+file_glob = "public_html/**/*.php"
+kind = "user_input"
+trust = "untrusted_external"
+asset_value = "high"
+description = "Web-exposed PHP script"
+
+# Rule: any function that takes a PSR-7 request object.
+[[entrypoint]]
+param_type = "ServerRequestInterface"
+kind = "api"
+trust = "untrusted_external"
+asset_value = "high"
+description = "PSR-7 HTTP handler"
+
+# Rule: functions named `handle_*`.
+[[entrypoint]]
+name_regex = "^handle_"
+kind = "api"
+trust = "untrusted_external"
+
+# Rule: conditions compose with AND — web.py files AND name starts with handle_.
+[[entrypoint]]
+file_glob = "public/*.py"
+name_regex = "^handle_"
+kind = "api"
+trust = "untrusted_external"
 ```
 
+Later entries override earlier ones when two rules tag the same node, so place broad rules first and specific corrections after.
+
+See [docs/entrypoint-patterns.md](docs/entrypoint-patterns.md) for the full reference, including frameworks not yet implemented (Express / Koa / Fastify, Laravel, Cobra, axum, warp, clap, and others) with grep-ready patterns contributors can use to add new detectors.
+
 ### Programmatic API
 
 ```python
 from trailmark.query.api import QueryEngine
 
+# Single-language (default) or auto-detect + merge across all languages
 engine = QueryEngine.from_directory("path/to/project")
+engine = QueryEngine.from_directory("path/to/project", language="auto")
+engine = QueryEngine.from_directory("path/to/project", language="python,rust")
 
-# Who calls this function?
+# Direct neighbors
 engine.callers_of("handle_request")
-
-# What does this function call?
 engine.callees_of("handle_request")
 
-# Call paths from entrypoint to sensitive function
+# Transitive slicing — who could reach this sink, or what could it reach?
+engine.ancestors_of("Auth._check_sig")
+engine.reachable_from("handle_request")
+
+# Attack-surface paths from any detected entrypoint
+engine.entrypoint_paths_to("Auth._check_sig")
+
+# All call paths between two nodes
 engine.paths_between("handle_request", "Auth._check_sig")
 
 # Functions with cyclomatic complexity >= 10
 engine.complexity_hotspots(10)
 
-# Add a semantic annotation
+# What functions can raise a given exception? (uses parser-detected
+# exception_types; no runtime tracing required)
+engine.functions_that_raise("PermissionError")
+
+# Add and query semantic annotations
 from trailmark.models.annotations import AnnotationKind
 
 engine.annotate(
@@ -278,10 +421,23 @@ engine.annotate(
     "Caller has already authenticated the session token",
     source="llm",
 )
-
-# Retrieve annotations
 engine.annotations_of("handle_request")
-engine.annotations_of("handle_request", kind=AnnotationKind.ASSUMPTION)
+engine.nodes_with_annotation(AnnotationKind.FINDING)
+
+# Diff against an earlier snapshot of the same codebase
+before = QueryEngine.from_directory("before/")
+diff = engine.diff_against(before)
+# diff contains: summary_delta, nodes {added/removed/modified},
+# edges {added/removed}, entrypoints {added/removed/modified}
+
+# Run the built-in audit-oriented preanalysis passes
+engine.preanalysis()
+engine.findings()
+engine.subgraph_names()
+
+# Programmatic augmentation hooks for external tooling
+engine.augment_sarif("results.sarif")
+engine.augment_weaudit("findings.json")
 ```
 
 ## Development
@@ -298,7 +454,7 @@ uv run ruff format
 uv tool install ty && ty check
 
 # Tests
-uv run pytest -q
+uv run pytest -q tests/
 
 # Mutation testing (on macOS, set this env var to avoid rustworkx fork segfaults)
 OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES uv run mutmut run