@dataparade/cli 0.0.3 → 0.0.5

package/patterns/python.md

@@ -0,0 +1,211 @@
+# Python Patterns (CLI Scanner)
+
+This document describes the Python-specific patterns implemented by the CLI codebase scanner for **DP-P0-CLI-703 – Python pattern detector**.
+
+The Python analyzer emits `RawFinding[]` using the **shared pattern IDs** from `core/types/detection.ts`:
+
+- `express_route`
+- `database_connection`
+- `external_api_call`
+- `auth_middleware`
+- `config_file`
+- `env_variable`
+
+These pattern IDs align with the TypeScript/JavaScript analyzer so that the classifier and data-flow detector can treat Python and TS/JS findings consistently.
+
+## Route / Handler Detection (`express_route`)
+
+The Python analyzer reuses the `express_route` pattern ID for web routes and handlers in common Python web frameworks.
+
+### FastAPI
+
+- **Detected when:**
+  - The module imports `FastAPI` or any symbol from `fastapi`.
+  - A function has a decorator starting with `app.get`, `app.post`, `app.put`, `app.delete`, or `app.patch`.
+- **Emitted finding:**
+  - `pattern`: `express_route`
+  - `name`: `"<METHOD> <function_name>"` (e.g. `GET read_item`)
+  - `properties`:
+    - `framework`: `"fastapi"`
+    - `httpMethods`: `[ "<METHOD>" ]`
+
+**Example:**
+
+```python
+from fastapi import FastAPI
+
+app = FastAPI()
+
+@app.get("/items/{item_id}")
+async def read_item(item_id: int):
+    ...
+```
+
+### Flask
+
+- **Detected when:**
+  - The module imports from `flask` or imports `Flask`.
+  - A function has a decorator starting with `app.route`, `bp.route`, or `blueprint.route`.
+- **Emitted finding:**
+  - `pattern`: `express_route`
+  - `name`: `"FLASK_ROUTE <function_name>"`
+  - `properties`:
+    - `framework`: `"flask"`
+
+**Example:**
+
+```python
+from flask import Flask
+
+app = Flask(__name__)
+
+@app.route("/users", methods=["GET"])
+def list_users():
+    ...
+```
+
+### Django / Django REST Framework
+
+- **Detected when:**
+  - The file path ends with `urls.py` or imports start with `django.`.
+  - A line contains `path("...")`, `re_path("...")`, or `url("...")`.
+- **Emitted finding:**
+  - `pattern`: `express_route`
+  - `name`: `"DJANGO_ROUTE <path>"`
+  - `properties`:
+    - `framework`: `"django"`
+    - `path`: `"<route_path>"`
+
+**Example (`urls.py`):**
+
+```python
+from django.urls import path
+from . import views
+
+urlpatterns = [
+    path("users/", views.user_list, name="user-list"),
+]
+```
+
+### Starlette
+
+- **Detected when:**
+  - The module imports Starlette (module name contains `"starlette"`).
+  - A function has a decorator starting with `app.route`.
+- **Emitted finding:**
+  - `pattern`: `express_route`
+  - `name`: `"STARLETTE_ROUTE <function_name>"`
+  - `properties`:
+    - `framework`: `"starlette"`
+
+### Bottle
+
+- **Detected when:**
+  - The module imports from `bottle` or imports `Bottle`.
+  - A function has decorators `@route`, `@get`, `@post`, `@put`, or `@delete`.
+- **Emitted finding:**
+  - `pattern`: `express_route`
+  - `name`: `"BOTTLE_ROUTE <function_name>"`
+  - `properties`:
+    - `framework`: `"bottle"`
+
+## Database / ORM Detection (`database_connection`)
+
+The Python analyzer emits `database_connection` for database clients and ORM usage.
+
+### psycopg2
+
+- **Detected when:**
+  - The module imports from `psycopg2` / `psycopg`.
+  - A module-level call contains `connect` (e.g. `psycopg2.connect(...)`).
+- **Properties:**
+  - `client`: `"psycopg2"`
+  - `databaseType`: `"postgres"`
+
+### SQLAlchemy
+
+- **Detected when:**
+  - The module imports from `sqlalchemy` or imports `create_engine`.
+  - A module-level call contains `create_engine`.
+- **Properties:**
+  - `client`: `"sqlalchemy"`
+  - `databaseType`: `"sql"`
+
+### Django ORM
+
+- **Detected when:**
+  - The module imports from `django.db` or the content includes `.objects.`.
+- **Properties:**
+  - `client`: `"django_orm"`
+  - `databaseType`: `"sql"`
+
+## External HTTP / API Calls (`external_api_call`)
+
+The Python analyzer emits `external_api_call` for HTTP clients and SDKs.
+
+### HTTP client libraries
+
+- **requests**
+  - Detected when the module imports `requests` and a module-level call starts with `requests.`.
+  - `name`: `"requests_call"`
+  - `properties.url`: first `http(s)` URL literal found in the file, if any.
+
+- **httpx**
+  - Detected when the module imports `httpx` and a module-level call starts with `httpx.`.
+  - `name`: `"httpx_call"`
+
+- **aiohttp**
+  - Detected when the module imports `aiohttp` or `ClientSession` and a module-level call references `ClientSession` or `.get`.
+  - `name`: `"aiohttp_call"`
+
+### SDKs
+
+The analyzer also emits `external_api_call` findings when the module imports:
+
+- `boto3` → `name`: `"boto3_client"`, `serviceName`: `"aws"`
+- `stripe` → `name`: `"stripe_sdk"`, `serviceName`: `"stripe"`
+- `sendgrid` → `name`: `"sendgrid_sdk"`, `serviceName`: `"sendgrid"`
+- `twilio` → `name`: `"twilio_sdk"`, `serviceName`: `"twilio"`
+- `openai` → `name`: `"openai_sdk"`, `serviceName`: `"openai"`
+
+## Auth Patterns (`auth_middleware`)
+
+The Python analyzer emits `auth_middleware` findings for common auth patterns.
+
+- **JWT-based auth**
+  - Detected when:
+    - The module imports a module containing `"jwt"`, imports `jwt`, or
+    - The content includes `jwt.encode(` or `jwt.decode(`.
+  - `name`: `"jwt_auth"`
+  - `properties.strategy`: `"jwt"`
+
+- **Auth decorators / permissions**
+  - Detected when module-level calls reference `login_required` or `IsAuthenticated`.
+  - `name`: `"auth_decorator"`
+
+- **Generic auth libraries**
+  - Detected when imports contain `"auth"` or `"authentication"`.
+  - `name`: `"auth_library"`
+
+## Config and Environment (`config_file`, `env_variable`)
+
+### Environment variables (`env_variable`)
+
+- **Detected when:**
+  - The module imports `os`, and lines reference:
+    - `os.environ["KEY"]` or `os.environ['KEY']`
+    - `os.getenv("KEY")` or `os.getenv('KEY')`
+- **Emitted finding:**
+  - `pattern`: `env_variable`
+  - `name`: `"os.environ[KEY]"`
+  - `properties.key`: `"KEY"`
+
+### Config files (`config_file`)
+
+- **Django settings**
+  - Detected when the file path ends with `settings.py`.
+  - `name`: `"django_settings"`
+
+- **dotenv configuration**
+  - Detected when the module imports `os` and the content includes `load_dotenv(`.
+  - `name`: `"dotenv_config"`

package/patterns/python.patterns.yaml

@@ -0,0 +1,160 @@
+# Python pattern config. All entries share: patternId, optional confidence (default 0.8).
+
+db_clients:
+  - id: psycopg2
+    patternId: database_connection
+    databaseType: postgres
+    importModules:
+      - psycopg2
+      - psycopg
+    callNames:
+      - connect
+    confidence: 0.9
+
+  - id: sqlalchemy
+    patternId: database_connection
+    databaseType: sql
+    importModules:
+      - sqlalchemy
+    callNames:
+      - create_engine
+    confidence: 0.9
+
+  - id: django_orm
+    patternId: database_connection
+    databaseType: sql
+    importModules:
+      - django.db
+    heuristics:
+      usesObjectsAttribute: true
+    confidence: 0.85
+
+auth:
+  jwt:
+    patternId: auth_middleware
+    importModules:
+      - jwt
+    contentRegexes:
+      - "jwt\\.encode\\("
+      - "jwt\\.decode\\("
+    strategy: jwt
+    confidence: 0.85
+
+  decorators:
+    - id: login_required
+      patternId: auth_middleware
+      callNames:
+        - login_required
+      confidence: 0.8
+    - id: is_authenticated
+      patternId: auth_middleware
+      callNames:
+        - IsAuthenticated
+      confidence: 0.8
+
+routes:
+  frameworks:
+    - id: fastapi
+      patternId: express_route
+      importModules:
+        - fastapi
+      decoratorPrefixes:
+        - app.get
+        - app.post
+        - app.put
+        - app.delete
+        - app.patch
+      confidence: 0.9
+
+    - id: flask
+      patternId: express_route
+      importModules:
+        - flask
+      decoratorPrefixes:
+        - app.route
+        - bp.route
+        - blueprint.route
+      confidence: 0.9
+
+    - id: starlette
+      patternId: express_route
+      importModules:
+        - starlette
+      decoratorPrefixes:
+        - app.route
+      confidence: 0.85
+
+    - id: bottle
+      patternId: express_route
+      importModules:
+        - bottle
+      decoratorNames:
+        - route
+        - get
+        - post
+        - put
+        - delete
+      confidence: 0.85
+
+    - id: django_urls
+      patternId: express_route
+      importModules:
+        - django.
+      urlsFileSuffix: "urls.py"
+      pathRegex: "(path|re_path|url)\\(\\s*['\\\"]([^'\\\"]+)['\\\"]"
+      confidence: 0.9
+
+env_config:
+  envVariable:
+    patternId: env_variable
+    regex: "os\\.(environ\\[['\\\"][^'\\\"]+['\\\"]\\]|getenv\\(\\s*['\\\"][^'\\\"]+['\\\"])"
+    confidence: 0.85
+
+  djangoSettings:
+    patternId: config_file
+    fileSuffix: "settings.py"
+    name: django_settings
+    confidence: 0.9
+
+  dotenvConfig:
+    patternId: config_file
+    requiresOsImport: true
+    contentSubstring: "load_dotenv("
+    name: dotenv_config
+    confidence: 0.85
+
+external_apis:
+  httpClients:
+    - id: requests
+      patternId: external_api_call
+      clientName: requests
+      importModules:
+        - requests
+      urlRegex: "['\\\"](https?:\\/\\/[^\\s'\\\"]+)['\\\"]"
+      confidence: 0.85
+
+    - id: httpx
+      patternId: external_api_call
+      clientName: httpx
+      importModules:
+        - httpx
+      urlRegex: "['\\\"](https?:\\/\\/[^\\s'\\\"]+)['\\\"]"
+      confidence: 0.85
+
+    - id: aiohttp
+      patternId: external_api_call
+      clientName: aiohttp
+      importModules:
+        - aiohttp
+      urlRegex: "['\\\"](https?:\\/\\/[^\\s'\\\"]+)['\\\"]"
+      confidence: 0.85
+
+    - id: supabase_python
+      patternId: external_api_call
+      clientName: supabase
+      importModules:
+        - supabase
+      callNames:
+        - create_client
+      confidence: 0.85
+

package/patterns/terraform.md

@@ -0,0 +1,37 @@
+# Terraform patterns (CLI)
+
+The Terraform analyzer performs **static, best-effort** parsing of `.tf` and `.tfvars` files. It does **not** run `terraform` or evaluate full HCL2 (heredocs, complex expressions, and `dynamic` blocks may be incomplete).
+
+**All matching rules** (block openers, cross-reference regexes, resource-type → `componentSubType` / `cloud_provider` hints, provider display names, and **satellite** resources folded into a parent) live in [`terraform.patterns.yaml`](terraform.patterns.yaml) and are loaded via `terraform-detection-config.ts`, consistent with `typescript.patterns.yaml` / `python.patterns.yaml`.
+
+**AWS resource type coverage:** hand-written regex hints in YAML are merged with **`aws-terraform-service-hints.generated.json`**, produced from **`@cdktf/provider-aws`** (tracks HashiCorp **[`terraform-provider-aws`](https://github.com/hashicorp/terraform-provider-aws)** ~> 6.x) and listing **~1.7k** `aws_*` strings in **`aws-terraform-catalog.snapshot.json`**. Generated `^aws_<service>_` rows are inserted immediately before the generic `^aws_` rule so uncommon services get a sensible `componentSubType` instead of falling through to `application`.
+
+**Azure (`azurerm`) resource type coverage:** **`azure-terraform-service-hints.generated.json`** + **`azure-terraform-catalog.snapshot.json`** come from **`@cdktf/provider-azurerm`** (~1.2k `azurerm_*` types), which tracks the upstream **[`hashicorp/terraform-provider-azurerm`](https://github.com/hashicorp/terraform-provider-azurerm)** Azure Resource Manager provider; hints are merged before **`azurerm_default_family`**. The **`azapi_*`** provider has no CDKTF package in this flow; it stays covered by the hand-written **`^azapi_`** rule in [`terraform.patterns.yaml`](terraform.patterns.yaml).
+
+**Kubernetes resource type coverage:** **`kubernetes-terraform-service-hints.generated.json`** + **`kubernetes-terraform-catalog.snapshot.json`** come from **`@cdktf/provider-kubernetes`** (HashiCorp **`kubernetes_*`** resources for in-cluster workloads, services, ingress, secrets, PVCs, etc.); hints merge before **`kubernetes_default_family`**. Provider topology in **`provider-topology.rules.yaml`** maps **`provider "kubernetes"`** to managed nodes (workload, service, ingress, config, storage, namespace).
+
+Regenerate all provider snapshots from `cli/` with **`pnpm run generate:terraform-provider-hints`** (or AWS / Azure / Kubernetes only — see **`cli/README.md`**).
+
+## Pattern IDs
+
+| Pattern ID | Meaning |
+|---|---|
+| `terraform_resource` | `resource` or `data` block (address `type.name` or `data.type.name`). |
+| `terraform_module` | `module` block. |
+| `terraform_provider` | `provider` block (emitted as a `third_party` component). |
+
+## Finding properties (representative)
+
+- `terraform_address`: stable resource key used for cross-reference flow edges.
+- `terraform_references`: other addresses referenced from the block body (string array), including references from **merged satellite** blocks (see below).
+- `terraform_satellites` (optional): when a child resource is configured as a **satellite** of a primary resource in YAML (e.g. S3 ACL / ownership controls scoped by `bucket = …`), it is **not** emitted as its own node; metadata is attached here on the parent finding.
+- `resource_type`, `block_name`, `cloud_provider`, `componentSubType` (classifier hint).
+- `section_id` / `section_label`: derived from the file path for layout grouping.
+- After **`applyDeterministicInferenceFallbacks`** (same pipeline step as TypeScript scans), matching AWS/Azure/Kubernetes/… resources may also have **`managed_by_provider`**, **`managed_service_key`**, and **`generated_by: "provider_topology_fallback"`** when they align with `provider-topology.rules.yaml` managed service nodes (e.g. labels **Aws S3**, **Aws Lambda**, **Kubernetes workload**). This mirrors TS SDK topology and enables the same provider→managed edge styling in graph export. Resources not covered by a managed node still get a **`provider → resource`** `api_call` via **`appendTerraformBareProviderAttachmentFlows`** inside that fallback pass.
+
+## Provider topology (parity with TypeScript)
+
+Static Terraform scans do **not** run a second topology engine. The **same** function, **`applyDeterministicInferenceFallbacks`** (`cli/src/ai-enrichment/fallbacks.ts`), loads **`cli/patterns/provider-topology.rules.yaml`**. For **TypeScript/JavaScript**, usage signals come from **`providerUsageCorpus(provider, flows)`**. For **Terraform on AWS**, each `managedServiceNode` lists **`terraformResourceTypePrefixes`** / **`terraformResourceTypes`** (HashiCorp `aws_*` resource type names, e.g. `aws_s3_bucket`, prefix `aws_lambda_`); matching uses **`resource_type` only**, not block labels like `lambda_bucket`. Providers without those fields (e.g. Supabase) still use legacy **`usageSignals`** matching on Terraform metadata. Shared helpers: **`cli/src/ai-enrichment/provider-topology-shared.ts`**. **`componentMatchesProvider`** treats Terraform’s **`provider_name`** (e.g. `aws`) as a match key so `provider "aws"` nodes participate in the AWS rule.
+
+- References inside `module.*` outputs are not resolved.
+- Optional merge from plan/state: **`terraform show -json`** via CLI `--terraform-json` / `--terraform-plan` (see `cli/README.md` and Epic 29 task DP-P0-CLI-906).