@gpzhang2001/sharpkit-skills 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +201 -0
- package/README.md +12 -0
- package/THIRD_PARTY_NOTICES.md +48 -0
- package/lib/index.d.ts +2027 -0
- package/lib/index.d.ts.map +1 -0
- package/lib/index.js +70 -0
- package/lib/index.js.map +1 -0
- package/package.json +46 -0
- package/skills/analysis/counterevidence.md +185 -0
- package/skills/analysis/fix_verification.md +129 -0
- package/skills/analysis/severity_calibration.md +130 -0
- package/skills/analysis/source_aware_discovery.md +211 -0
- package/skills/cloud/aws.md +231 -0
- package/skills/cloud/azure.md +262 -0
- package/skills/cloud/gcp.md +194 -0
- package/skills/cloud/kubernetes.md +223 -0
- package/skills/coordination/root_agent.md +105 -0
- package/skills/coordination/source_aware_whitebox.md +47 -0
- package/skills/custom/api_spec_testing.md +61 -0
- package/skills/custom/dependency_cve_scanning.md +341 -0
- package/skills/custom/npx_confusion.md +233 -0
- package/skills/custom/source_aware_sast.md +192 -0
- package/skills/frameworks/django.md +214 -0
- package/skills/frameworks/fastapi.md +191 -0
- package/skills/frameworks/nestjs.md +225 -0
- package/skills/frameworks/nextjs.md +228 -0
- package/skills/protocols/graphql.md +276 -0
- package/skills/protocols/oauth.md +185 -0
- package/skills/reconnaissance/asset_discovery.md +150 -0
- package/skills/reconnaissance/infrastructure_lifecycle.md +226 -0
- package/skills/scan_modes/deep.md +164 -0
- package/skills/scan_modes/diff.md +86 -0
- package/skills/scan_modes/quick.md +68 -0
- package/skills/scan_modes/standard.md +99 -0
- package/skills/technologies/active_directory.md +233 -0
- package/skills/technologies/auth0.md +188 -0
- package/skills/technologies/electron_desktop_apps.md +181 -0
- package/skills/technologies/firebase.md +263 -0
- package/skills/technologies/grafana_prometheus.md +189 -0
- package/skills/technologies/llm_applications.md +257 -0
- package/skills/technologies/supabase.md +268 -0
- package/skills/tooling/agent_browser.md +551 -0
- package/skills/tooling/ffuf.md +72 -0
- package/skills/tooling/httpx.md +82 -0
- package/skills/tooling/hurl.md +99 -0
- package/skills/tooling/hypothesis.md +100 -0
- package/skills/tooling/katana.md +102 -0
- package/skills/tooling/naabu.md +68 -0
- package/skills/tooling/nmap.md +66 -0
- package/skills/tooling/nuclei.md +67 -0
- package/skills/tooling/python.md +109 -0
- package/skills/tooling/semgrep.md +72 -0
- package/skills/tooling/sqlmap.md +67 -0
- package/skills/tooling/subfinder.md +66 -0
- package/skills/vulnerabilities/agentic_system_security.md +207 -0
- package/skills/vulnerabilities/argument_injection.md +157 -0
- package/skills/vulnerabilities/authentication_jwt.md +166 -0
- package/skills/vulnerabilities/broken_function_level_authorization.md +154 -0
- package/skills/vulnerabilities/browser_security.md +192 -0
- package/skills/vulnerabilities/business_logic.md +178 -0
- package/skills/vulnerabilities/csrf.md +198 -0
- package/skills/vulnerabilities/header_injection.md +216 -0
- package/skills/vulnerabilities/http_request_smuggling.md +255 -0
- package/skills/vulnerabilities/idor.md +217 -0
- package/skills/vulnerabilities/information_disclosure.md +187 -0
- package/skills/vulnerabilities/insecure_deserialization.md +210 -0
- package/skills/vulnerabilities/insecure_file_uploads.md +194 -0
- package/skills/vulnerabilities/llm_prompt_injection.md +187 -0
- package/skills/vulnerabilities/mass_assignment.md +153 -0
- package/skills/vulnerabilities/nosql_injection.md +288 -0
- package/skills/vulnerabilities/open_redirect.md +165 -0
- package/skills/vulnerabilities/path_traversal_lfi_rfi.md +218 -0
- package/skills/vulnerabilities/prototype_pollution.md +142 -0
- package/skills/vulnerabilities/race_conditions.md +181 -0
- package/skills/vulnerabilities/rce.md +250 -0
- package/skills/vulnerabilities/semantic_confusion.md +189 -0
- package/skills/vulnerabilities/sql_injection.md +190 -0
- package/skills/vulnerabilities/ssrf.md +186 -0
- package/skills/vulnerabilities/ssti.md +270 -0
- package/skills/vulnerabilities/subdomain_takeover.md +167 -0
- package/skills/vulnerabilities/weak_password_detection.md +200 -0
- package/skills/vulnerabilities/xss.md +206 -0
- package/skills/vulnerabilities/xxe.md +223 -0
- package/src/index.ts +89 -0
|
@@ -0,0 +1,257 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: llm-applications
|
|
3
|
+
description: "End-to-end security testing for LLM, RAG, embedding, agent, and model-serving applications. Covers the OWASP Top 10 for LLM Applications 2026 (LLM01-LLM10): prompt injection, sensitive disclosure, excessive agency, supply chain, data/model poisoning, unbounded consumption, misinformation, hidden context exposure, vector weaknesses, and improper output handling. Use for architecture mapping, source review, black-box testing, and complete LLM application assessments."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# LLM Application Security
|
|
7
|
+
|
|
8
|
+
Use this as the umbrella workflow for the [OWASP Top 10 for LLM Applications 2026](https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/). Load `llm_prompt_injection` for deeper LLM01 testing and the relevant conventional vulnerability skill when an LLM-controlled value reaches a browser, query, command, URL, file, or authorization sink.
|
|
9
|
+
|
|
10
|
+
Treat the identifiers as a coverage taxonomy, not as report titles. Classify a finding by its technical root cause and affected trust boundary. One exploit chain may contain several OWASP categories, while one root cause should not become ten duplicate reports.
|
|
11
|
+
|
|
12
|
+
The LLM list covers the model as a component of an application. When a model acts through tools, persistent memory, peer agents, or autonomous workflows, apply this list and pair the assessment with the OWASP Top 10 for Agentic Applications 2026; do not force every agentic failure into an LLM category.
|
|
13
|
+
|
|
14
|
+
## Architecture and Evidence Map
|
|
15
|
+
|
|
16
|
+
Map the complete system before testing prompts:
|
|
17
|
+
|
|
18
|
+
```text
|
|
19
|
+
users / tenants / external content
|
|
20
|
+
-> API, UI, file and multimodal ingestion
|
|
21
|
+
-> prompt builder, policy and orchestration
|
|
22
|
+
-> model/provider and context window
|
|
23
|
+
-> memory, cache, RAG retrieval and vector index
|
|
24
|
+
-> tools, MCP servers, plugins and peer agents
|
|
25
|
+
-> output parsers, renderers and downstream systems
|
|
26
|
+
-> logs, traces, feedback, evaluation and training pipelines
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
For every edge, record:
|
|
30
|
+
|
|
31
|
+
- **Data authority:** who creates, reads, updates, deletes, approves, and owns the data; tenant and sensitivity; retention and training use.
|
|
32
|
+
- **Action authority:** caller identity, downstream identity, permissions, authorization checks, confirmation, transaction boundaries, and audit evidence.
|
|
33
|
+
- **Transformation:** serialization, chunking, embedding, retrieval, reranking, prompt placement, output parsing, and cache keys.
|
|
34
|
+
- **Runtime identity:** application build, provider, model and revision, prompt revision, tool set, feature flags, corpus/index snapshot, temperature/seed where available, and quota policy.
|
|
35
|
+
|
|
36
|
+
Do not treat the model as an authorization principal or a trusted parser. Put deterministic authentication, authorization, validation, and policy enforcement outside the model.
|
|
37
|
+
|
|
38
|
+
## 2026 Coverage Matrix
|
|
39
|
+
|
|
40
|
+
| OWASP 2026 risk | Security invariant to test | Primary route |
|
|
41
|
+
|---|---|---|
|
|
42
|
+
| LLM01:2026 Prompt Injection | Untrusted instructions cannot cross a meaningful policy or authority boundary | `llm_prompt_injection` |
|
|
43
|
+
| LLM02:2026 Sensitive Information Disclosure | A response, context, cache, trace, training path, or retrieval result reveals only data authorized for the caller | This skill + `information_disclosure` |
|
|
44
|
+
| LLM03:2026 Excessive Agency | Tools expose only required functionality, permissions, and autonomy, with complete mediation at the action | This skill + `broken_function_level_authorization` / `business_logic` |
|
|
45
|
+
| LLM04:2026 Supply Chain | Every model, adapter, dataset, tokenizer, prompt, plugin, package, image, and hosted API has verified provenance and an immutable deployment identity | This skill + `dependency_cve_scanning` / `source_aware_sast` |
|
|
46
|
+
| LLM05:2026 Data and Model Poisoning | Attacker-influenced training, tuning, feedback, memory, or embedding data cannot persistently alter protected behavior unnoticed | This skill |
|
|
47
|
+
| LLM06:2026 Unbounded Consumption | Every request, recursive action, queue, and billable operation has enforceable cumulative resource and cost bounds | This skill + `business_logic` / `race_conditions` |
|
|
48
|
+
| LLM07:2026 Misinformation | Unsupported output cannot silently drive a security-sensitive or high-impact decision | This skill + `business_logic` |
|
|
49
|
+
| LLM08:2026 Hidden Context Exposure | Hidden instructions and operational context contain no secrets and reveal no security-relevant logic or capability that materially increases attacker power | This skill + `llm_prompt_injection` / `information_disclosure` |
|
|
50
|
+
| LLM09:2026 Vector and Embedding Weaknesses | Ingestion and retrieval preserve tenant, source, document authorization, and embedding confidentiality across the index lifecycle | This skill + `idor` / `information_disclosure` |
|
|
51
|
+
| LLM10:2026 Improper Output Handling | Model output remains untrusted until the actual downstream grammar and sink validate it | This skill + the sink-specific vulnerability skill |
|
|
52
|
+
|
|
53
|
+
## Assessment Workflow
|
|
54
|
+
|
|
55
|
+
1. Inventory every LLM-backed feature, model endpoint, ingestion route, retrieval source, tool, output consumer, and feedback/training path.
|
|
56
|
+
2. Build the data-and-authority map above for each user role and tenant.
|
|
57
|
+
3. Create a test matrix across application build, model/revision, prompt revision, tool configuration, identity, corpus snapshot, and quota tier.
|
|
58
|
+
4. Use controlled records with distinct per-user and per-tenant markers to distinguish context, retrieval, cache, memory, and training leakage.
|
|
59
|
+
5. Establish a normal baseline and matched negative control before adversarial variants. Run repeated trials and report success counts because model behavior is stochastic.
|
|
60
|
+
6. Validate the application-side effect, retrieved record, rendered sink, downstream authorization result, resource meter, or persistent model change. Model narration alone is not evidence of that effect.
|
|
61
|
+
7. Label each claim **architecture-confirmed**, **dynamically verified**, **candidate**, or **disproven**. Do not turn an unsafe architecture property into a claimed exploit, or ignore a confirmed control defect merely because downstream impact has not yet been exercised.
|
|
62
|
+
8. Report the smallest technical root cause that explains the demonstrated impact, then document related OWASP categories as chain context.
|
|
63
|
+
|
|
64
|
+
## Source Review
|
|
65
|
+
|
|
66
|
+
Trace source to sink around:
|
|
67
|
+
|
|
68
|
+
- provider SDK calls, local inference servers, model gateways, and fallback providers
|
|
69
|
+
- system/developer prompts, templates, message-role conversion, context truncation, reasoning channels, and prompt caches
|
|
70
|
+
- file, URL, email, image/audio/video, connector, tool-result, peer-agent, and memory ingestion
|
|
71
|
+
- embedding generation, collection/namespace selection, metadata filters, reranking, hybrid search, and retrieval caches
|
|
72
|
+
- function/tool definitions, MCP clients/servers, generic HTTP/shell/SQL tools, peer-agent delegation, and approval handlers
|
|
73
|
+
- model output parsers, HTML/Markdown renderers, terminals/IDEs/logs, code execution, query builders, URLs, file paths, templates, and policy decisions
|
|
74
|
+
- training/fine-tuning jobs, adapters, datasets, feedback stores, evaluation corpora, model registries, and runtime downloads
|
|
75
|
+
- token accounting, request limits, concurrency, retries, agent-loop depth, fan-out, async queues, streaming cancellation, and provider billing
|
|
76
|
+
|
|
77
|
+
Record both forward and reverse reachability: attacker-controlled input to privileged consumer, and privileged consumer back to every input or model output that can influence it.
|
|
78
|
+
|
|
79
|
+
## Optional Tool Routing
|
|
80
|
+
|
|
81
|
+
Use tools only when they match the deployed surface. Treat generated cases and scanner labels as leads until the application-side boundary is validated.
|
|
82
|
+
|
|
83
|
+
- **[Promptfoo](https://github.com/promptfoo/promptfoo)** — use for repeatable model/application trials, custom adversarial cases, graders, provider comparisons, and success-rate regression. Install the reviewed version locally with `npm install --save-dev --save-exact promptfoo@0.122.0`, then invoke `./node_modules/.bin/promptfoo redteam run`. Define explicit plugins, assertions, `numTests`, `maxConcurrency`, and `delay`; provider calls may transmit test data and incur cost. Its `owasp:llm` preset still uses the 2025 category mapping in version 0.122.0, so build or select tests from the 2026 matrix above and do not present the preset report as complete 2026 coverage.
|
|
84
|
+
- **[MCP Inspector](https://github.com/modelcontextprotocol/inspector)** — use for LLM01/LLM03 surface mapping when MCP servers are present. Install the reviewed version with `npm install --save-dev --save-exact @modelcontextprotocol/inspector@2.2.0`, then use `./node_modules/.bin/mcp-inspector --cli --config <reviewed-config> --server <name> --method tools/list` and the equivalent `resources/list` / `prompts/list` operations. Starting a stdio server executes that configured process, initialization/list handlers may have side effects, and `tools/call` can perform the real action; inspect the target and credentials before invoking it.
|
|
85
|
+
- **[ModelScan](https://github.com/protectai/modelscan)** — use for LLM04 static triage of supported H5, Pickle, and SavedModel artifacts before loading them, for example `uvx modelscan==0.8.8 -p <artifact>`. Run it as an untrusted-file parser in an isolated analysis environment. A clean result covers only the scanner's supported formats and signatures; it does not establish artifact provenance, integrity, or absence of behavioral backdoors.
|
|
86
|
+
|
|
87
|
+
## LLM01:2026 Prompt Injection
|
|
88
|
+
|
|
89
|
+
Load `llm_prompt_injection` and test direct, indirect, stored, cross-modal, tool-result, memory, intermediate-reasoning, and multi-turn instruction paths. Include content from web pages, documents, messages, metadata, OCR, images/audio/video, retrieved chunks, tools, MCP servers, and peer agents.
|
|
90
|
+
|
|
91
|
+
For each delivery path, record provenance as untrusted, semi-trusted, or trusted-by-the-operator but attacker-writable through another workflow. Test plain, split, multilingual, encoded, invisible-Unicode, and multimodal representations where the deployed preprocessing makes them relevant.
|
|
92
|
+
|
|
93
|
+
Define the violated invariant before testing: unauthorized data access, an unauthorized action, corruption of a protected decision, persistent behavior change, or unsafe downstream output. A jailbreak or changed tone without a security-relevant boundary is not automatically an application vulnerability.
|
|
94
|
+
|
|
95
|
+
Distinguish:
|
|
96
|
+
|
|
97
|
+
- **Prompt injection:** input changes model behavior contrary to application policy.
|
|
98
|
+
- **Jailbreak:** model safety behavior is bypassed; application impact depends on the product's requirements and connected capabilities.
|
|
99
|
+
- **Poisoning:** attacker influence persists in training, feedback, memory, or an indexed corpus and affects later users or decisions.
|
|
100
|
+
|
|
101
|
+
## LLM02:2026 Sensitive Information Disclosure
|
|
102
|
+
|
|
103
|
+
Inventory sensitive data in prompts, reasoning or scratchpad traces, retrieved chunks, tool results, memory, caches, logs, training/feedback stores, model outputs, and provider retention paths.
|
|
104
|
+
|
|
105
|
+
Test separately for:
|
|
106
|
+
|
|
107
|
+
- cross-user and cross-tenant context, memory, cache, and retrieval leakage
|
|
108
|
+
- secrets or private records inserted into prompts, tool schemas/results, errors, traces, or telemetry
|
|
109
|
+
- retained user content later used for training, evaluation, or another user's response
|
|
110
|
+
- training-data membership or memorization when the tested model and data provenance make that claim meaningful
|
|
111
|
+
- model/provider options that expose logits, log probabilities, hidden metadata, raw context, or internal reasoning
|
|
112
|
+
|
|
113
|
+
Use distinct markers for each principal and storage stage. A fabricated secret or hallucinated record is not disclosure; correlate the output to a real record and its unauthorized source.
|
|
114
|
+
|
|
115
|
+
## LLM03:2026 Excessive Agency
|
|
116
|
+
|
|
117
|
+
Create a capability ledger for every tool and peer agent:
|
|
118
|
+
|
|
119
|
+
```text
|
|
120
|
+
tool -> exposed operations -> downstream identity -> permissions
|
|
121
|
+
-> caller/user binding -> argument validation -> authorization
|
|
122
|
+
-> side effects -> retry/idempotency -> audit evidence
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Test the three independent causes:
|
|
126
|
+
|
|
127
|
+
- **Excessive functionality:** unused, generic, administrative, shell, arbitrary-URL, or broad CRUD tools remain callable.
|
|
128
|
+
- **Excessive permissions:** tools use a shared/service identity or scopes broader than the initiating user and requested operation.
|
|
129
|
+
- **Excessive autonomy:** consequential actions execute without human or deterministic authorization appropriate to the exact action, object, arguments, identity, and current state.
|
|
130
|
+
|
|
131
|
+
Tool descriptions, model instructions, hidden channel names, and confirmation prose are not authorization controls. Enforce authorization again at the tool/downstream system. Test delegation, recursive plans, retries, race/state changes between approval and execution, and whether untrusted tool results become new instructions.
|
|
132
|
+
|
|
133
|
+
Prove the accepted tool call and downstream result. A model saying it invoked a tool is not evidence that the action occurred.
|
|
134
|
+
|
|
135
|
+
## LLM04:2026 Supply Chain
|
|
136
|
+
|
|
137
|
+
Build an inventory beyond ordinary packages:
|
|
138
|
+
|
|
139
|
+
- base models, weights, tokenizers, configuration, adapters/LoRA, quantizations, and model-conversion outputs
|
|
140
|
+
- training, tuning, evaluation, and embedding datasets
|
|
141
|
+
- prompt/template repositories, skills, plugins, MCP servers, hosted model APIs, and model gateways
|
|
142
|
+
- Python/JavaScript/native dependencies, containers, drivers, accelerators, and serving infrastructure
|
|
143
|
+
|
|
144
|
+
For each component, record origin, owner, license/terms, exact revision or digest, hash/signature/attestation, review status, update channel, runtime downloads, and effective permissions. Resolve every model alias, branch, mutable tag, adapter, and custom-code dependency to the artifact actually loaded. Identify who can mutate the source, promotion record, cache, or registry and whether the promoted artifact matches its claimed identity.
|
|
145
|
+
|
|
146
|
+
Inspect model loading as code loading. Pickle-compatible weights, custom model/tokenizer code, conversion hooks, package installation, and remote-code trust options can execute during acquisition or load. Trace the selected loader, artifact format, revision, initialization hooks, and resulting process or file activity.
|
|
147
|
+
|
|
148
|
+
Trace model-generated dependency names through every package runner, installer, build file, and registry lookup. A fabricated package recommendation is LLM07 misinformation; accepting or auto-installing an unverified name, namespace, or registry artifact is the LLM04 supply-chain boundary. Verify ownership and provenance rather than treating a registry response alone as proof of safety.
|
|
149
|
+
|
|
150
|
+
Use `dependency_cve_scanning` for verified known-CVE software versions. A malicious or tampered model, dataset, adapter, prompt, or plugin is a different supply-chain finding and requires provenance plus behavioral or loader evidence.
|
|
151
|
+
|
|
152
|
+
## LLM05:2026 Data and Model Poisoning
|
|
153
|
+
|
|
154
|
+
Map who can contribute to every pre-training, fine-tuning, preference, feedback, evaluation, memory, and embedding dataset. Record moderation, approval, deduplication, weighting, precedence, versioning, rollback, and the delay before data affects production.
|
|
155
|
+
|
|
156
|
+
Test:
|
|
157
|
+
|
|
158
|
+
- targeted trigger/backdoor behavior versus broad quality degradation
|
|
159
|
+
- poisoned examples that survive normalization, deduplication, chunking, or retraining
|
|
160
|
+
- feedback loops where model output or user ratings become future training data
|
|
161
|
+
- shared memory or indexed content that persists across users, sessions, or releases
|
|
162
|
+
- compromised adapters, merged models, or fine-tuning jobs that alter only a narrow topic, identity, or trigger
|
|
163
|
+
|
|
164
|
+
Compare clean and candidate snapshots with a fixed evaluation corpus and repeated trials. Trace a candidate record into the exact training/index snapshot and demonstrate persistence plus a protected behavior change. One retrieved malicious instruction may be LLM01 rather than proof that the model or dataset was poisoned.
|
|
165
|
+
|
|
166
|
+
Classify provenance/distribution compromise under LLM04 and durable corruption of data, weights, adapters, templates, or model behavior under LLM05. Record both when one chain crosses both boundaries, but do not duplicate the same root cause.
|
|
167
|
+
|
|
168
|
+
## LLM06:2026 Unbounded Consumption
|
|
169
|
+
|
|
170
|
+
Inventory every resource multiplier:
|
|
171
|
+
|
|
172
|
+
- input and output tokens, context windows, image/audio/video/document processing, embeddings, reranking, and model tier
|
|
173
|
+
- requests per user/key/IP/tenant, concurrency, batch size, and organization-wide budget
|
|
174
|
+
- agent iterations, tool calls, peer-agent fan-out, retries, provider failover, and recursive workflows
|
|
175
|
+
- upload count/size, chunk count, index growth, queued/background jobs, and retained outputs
|
|
176
|
+
- streaming connections, disconnect cancellation, timeouts, cache behavior, and partial failures
|
|
177
|
+
- logprobs or repeated-query surfaces that increase extraction or model-replication risk
|
|
178
|
+
|
|
179
|
+
Model cumulative work, not isolated limits: depth × fan-out × retries × failovers × model/tool cost. Test limits at request, identity, tenant, and global layers. Confirm that alternate keys, endpoints, models, encodings, streaming, retries, and concurrent requests cannot bypass accounting. Verify cancellation stops upstream inference and tool work, and that failed/retried operations do not bill or enqueue without bounds.
|
|
180
|
+
|
|
181
|
+
Record measured requests, tokens, tool calls, queue growth, latency, and provider-side cost/usage. Increase load in controlled steps; do not infer denial of service, model extraction, or financial impact from the mere absence of a UI counter.
|
|
182
|
+
|
|
183
|
+
## LLM07:2026 Misinformation
|
|
184
|
+
|
|
185
|
+
Define a trusted answer set and the downstream decision before testing. Separate ordinary model fallibility from a security or business-logic flaw.
|
|
186
|
+
|
|
187
|
+
Exercise:
|
|
188
|
+
|
|
189
|
+
- absent, ambiguous, stale, and mutually contradictory sources
|
|
190
|
+
- fabricated, mismatched, or forged citations, quotations, evidence, and task-completion claims
|
|
191
|
+
- adversarial sources that rank above authoritative material
|
|
192
|
+
- confidence language and UI cues that overstate certainty
|
|
193
|
+
- generated code, policy, medical/legal/financial guidance, identity matching, fraud/risk decisions, and other outputs consumed without verification
|
|
194
|
+
- automated actions triggered by unsupported claims
|
|
195
|
+
|
|
196
|
+
Measure claim support, citation coverage and entailment, source authority, abstention, and decision error across a repeatable corpus rather than reporting one hallucinated answer. Report when unsupported output crosses a defined trust boundary or drives a protected decision without required verification; otherwise record it as a quality/reliability issue.
|
|
197
|
+
|
|
198
|
+
## LLM08:2026 Hidden Context Exposure
|
|
199
|
+
|
|
200
|
+
Inventory non-user-facing content available to the model: system and developer instructions, retrieved policy text, user-profile context, tool/function schemas, workflow criteria, internal roles, reasoning scaffolds, and operational configuration.
|
|
201
|
+
|
|
202
|
+
Test extraction, inference, and reconstruction separately. Compare purported hidden context with the deployed revision, a unique marker, or observed capability because models can fabricate plausible prompts and tool lists.
|
|
203
|
+
|
|
204
|
+
Classify the result by what it exposes:
|
|
205
|
+
|
|
206
|
+
- embedded credentials, tokens, private records, or connection material -> LLM02 disclosure, with LLM08 as the exposure path
|
|
207
|
+
- hidden rules, trust boundaries, tool schemas, or workflow logic that materially improve an attack -> LLM08
|
|
208
|
+
- authorization, filtering, or privilege controls that depend on hidden-context secrecy or model obedience -> the underlying deterministic-control failure
|
|
209
|
+
- generic instructions with no sensitive content, security reliance, or material attacker advantage -> no standalone vulnerability
|
|
210
|
+
|
|
211
|
+
Assume hidden context is discoverable. Keep secrets and security-critical decisions outside it, and test the underlying control even when exact prompt wording cannot be recovered.
|
|
212
|
+
|
|
213
|
+
## LLM09:2026 Vector and Embedding Weaknesses
|
|
214
|
+
|
|
215
|
+
Map ingestion authorization separately from retrieval authorization. Preserve source identity, tenant, document ACL, classification, retention, and deletion state through chunking, embedding, indexing, replication, reranking, and caching.
|
|
216
|
+
|
|
217
|
+
Test:
|
|
218
|
+
|
|
219
|
+
- authorization inside vector search, filtering after top-k but before context construction, and filtering only after the model sees candidates
|
|
220
|
+
- shared collections/namespaces and missing, inconsistent, or fail-open tenant filters
|
|
221
|
+
- metadata-filter injection, type confusion, duplicate keys, or precedence differences
|
|
222
|
+
- oversampling/reranking/hybrid-search stages that drop earlier authorization constraints
|
|
223
|
+
- stale embeddings after source ACL changes, deletion, tenant moves, or index rebuilds
|
|
224
|
+
- retrieval and answer caches keyed without user, tenant, role, corpus version, or filter state
|
|
225
|
+
- cross-tenant existence inference through IDs, scores, timing, citations, or chunk metadata even when final text is refused
|
|
226
|
+
- adversarial or duplicate content that dominates nearest-neighbor retrieval
|
|
227
|
+
- embedding export, inversion, reconstruction, or linkage when vectors are returned or broadly readable
|
|
228
|
+
|
|
229
|
+
Use at least two principals and distinct documents. Inspect raw candidate IDs, context-bound chunks, and the final answer. Post-search filtering may cause ranking interference or expose candidates to an intermediate service without proving that the model or user received another tenant's content; state the exact boundary crossed.
|
|
230
|
+
|
|
231
|
+
Do not apply LLM09 merely because an application retrieves documents. Require an embedding or vector-similarity property; route authorization flaws in vectorless retrieval to the conventional access-control or information-disclosure skill.
|
|
232
|
+
|
|
233
|
+
## LLM10:2026 Improper Output Handling
|
|
234
|
+
|
|
235
|
+
Treat every model-generated string, object, URL, code block, tool argument, control sequence, and structured-output field as attacker-influenceable.
|
|
236
|
+
|
|
237
|
+
Trace output into its actual consumer:
|
|
238
|
+
|
|
239
|
+
- HTML, Markdown, email, office-document, terminal, IDE, log, and rich-text renderers
|
|
240
|
+
- shell/process APIs, SQL/NoSQL queries, templates, expressions, interpreters, and generated code accepted into builds
|
|
241
|
+
- URLs, webhooks, redirects, image fetches, browser navigation, and server-side requests
|
|
242
|
+
- file paths, archive entries, object keys, configuration, logs, and serialized objects
|
|
243
|
+
- authorization, moderation, routing, pricing, eligibility, or workflow decisions
|
|
244
|
+
|
|
245
|
+
Validate with the sink-specific skill (`xss`, `sql_injection`, `nosql_injection`, `rce`, `ssrf`, `path_traversal_lfi_rfi`, `ssti`, or `insecure_deserialization`). JSON/schema conformance does not establish authorization or semantic safety; validate types, ranges, identities, destinations, and business rules after parsing.
|
|
246
|
+
|
|
247
|
+
## Reproducibility and Reporting
|
|
248
|
+
|
|
249
|
+
- Preserve application/model/prompt/tool/corpus versions and all generation parameters available to the application.
|
|
250
|
+
- Compare baseline and adversarial trials, record attempt and success counts, and distinguish deterministic application behavior from stochastic model behavior.
|
|
251
|
+
- Validate authorization, data origin, downstream effects, persistence, or measured consumption outside the model transcript.
|
|
252
|
+
- Split reports when weaknesses have independent reproductions, trust boundaries, owners, or remediations. Otherwise report one technical root cause and mention additional OWASP mappings as chain context.
|
|
253
|
+
- Use `create_dependency_report` only for verified advisory-matched dependency CVEs. Use `create_vulnerability_report` for dynamically verified application, model, RAG, agent, or supply-chain findings.
|
|
254
|
+
|
|
255
|
+
## Summary
|
|
256
|
+
|
|
257
|
+
Test the LLM application as a data-and-authority system, not as a chatbot prompt. Complete 2026 coverage requires model behavior, application code, retrieval, tools, supply chain, downstream sinks, and resource controls to be evaluated together while keeping their root causes distinct.
|
|
@@ -0,0 +1,268 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: supabase
|
|
3
|
+
description: Supabase security testing covering Row Level Security, PostgREST, Edge Functions, and service key exposure
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Supabase
|
|
7
|
+
|
|
8
|
+
Security testing for Supabase applications. Focus on mis-scoped Row Level Security (RLS), unsafe RPCs, leaked `service_role` keys, lax Storage policies, and Edge Functions trusting headers without binding to issuer/audience/tenant.
|
|
9
|
+
|
|
10
|
+
## Attack Surface
|
|
11
|
+
|
|
12
|
+
**Data Access**
|
|
13
|
+
- PostgREST: table CRUD, filters, embeddings, RPC (remote functions)
|
|
14
|
+
- GraphQL: pg_graphql over Postgres schema with RLS interaction
|
|
15
|
+
- Realtime: replication subscriptions, broadcast/presence channels
|
|
16
|
+
|
|
17
|
+
**Storage**
|
|
18
|
+
- Buckets, objects, signed URLs, public/private policies
|
|
19
|
+
|
|
20
|
+
**Authentication**
|
|
21
|
+
- Auth (GoTrue): JWTs, cookie/session, magic links, OAuth flows
|
|
22
|
+
|
|
23
|
+
**Server-Side**
|
|
24
|
+
- Edge Functions (Deno): server-side code calling Supabase with secrets
|
|
25
|
+
|
|
26
|
+
## Architecture
|
|
27
|
+
|
|
28
|
+
**Endpoints**
|
|
29
|
+
- REST: `https://<ref>.supabase.co/rest/v1/<table>`
|
|
30
|
+
- RPC: `https://<ref>.supabase.co/rest/v1/rpc/<fn>`
|
|
31
|
+
- Storage: `https://<ref>.supabase.co/storage/v1`
|
|
32
|
+
- GraphQL: `https://<ref>.supabase.co/graphql/v1`
|
|
33
|
+
- Realtime: `wss://<ref>.supabase.co/realtime/v1`
|
|
34
|
+
- Auth: `https://<ref>.supabase.co/auth/v1`
|
|
35
|
+
- Functions: `https://<ref>.functions.supabase.co/`
|
|
36
|
+
|
|
37
|
+
**Headers**
|
|
38
|
+
- `apikey: <anon-or-service>` — identifies project
|
|
39
|
+
- `Authorization: Bearer <JWT>` — binds user context
|
|
40
|
+
|
|
41
|
+
**Roles**
|
|
42
|
+
- `anon`, `authenticated` — standard roles
|
|
43
|
+
- `service_role` — bypasses RLS, must never be client-exposed
|
|
44
|
+
|
|
45
|
+
**Key Principle**
|
|
46
|
+
`auth.uid()` returns current user UUID from JWT. Policies must never trust client-supplied IDs over server context.
|
|
47
|
+
|
|
48
|
+
## High-Value Targets
|
|
49
|
+
|
|
50
|
+
- Tables with sensitive data (users, orders, payments, PII)
|
|
51
|
+
- RPC functions (especially `SECURITY DEFINER`)
|
|
52
|
+
- Storage buckets with private files
|
|
53
|
+
- Edge Functions with `service_role` access
|
|
54
|
+
- Export/report endpoints generating signed outputs
|
|
55
|
+
- Admin/staff routes and privilege-granting endpoints
|
|
56
|
+
|
|
57
|
+
## Reconnaissance
|
|
58
|
+
|
|
59
|
+
**Enumerate Surfaces**
|
|
60
|
+
```
|
|
61
|
+
/rest/v1/<table>
|
|
62
|
+
/rest/v1/rpc/<fn>
|
|
63
|
+
/storage/v1/object/public/<bucket>/
|
|
64
|
+
/storage/v1/object/list/<bucket>?prefix=
|
|
65
|
+
/graphql/v1
|
|
66
|
+
/auth/v1
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
**Obtain Principals**
|
|
70
|
+
- Unauthenticated (anon key only)
|
|
71
|
+
- Basic user A, user B
|
|
72
|
+
- Admin/staff (if available)
|
|
73
|
+
- Check if `service_role` key leaked in client bundle or Edge Function responses
|
|
74
|
+
|
|
75
|
+
## Key Vulnerabilities
|
|
76
|
+
|
|
77
|
+
### Row Level Security (RLS)
|
|
78
|
+
|
|
79
|
+
Enable RLS on every non-public table; absence or "permit-all" policies → bulk exposure.
|
|
80
|
+
|
|
81
|
+
**Common Gaps**
|
|
82
|
+
- Policies check `auth.uid()` for SELECT but forget UPDATE/DELETE/INSERT
|
|
83
|
+
- Missing tenant constraints (`org_id`/`tenant_id`) allow cross-tenant access
|
|
84
|
+
- Policies rely on client-provided columns (`user_id` in payload) instead of JWT
|
|
85
|
+
- Complex joins where policy is applied after filters, enabling inference via counts
|
|
86
|
+
|
|
87
|
+
**Tests**
|
|
88
|
+
```bash
|
|
89
|
+
# Compare row counts for two users
|
|
90
|
+
GET /rest/v1/<table>?select=*&Prefer=count=exact
|
|
91
|
+
|
|
92
|
+
# Cross-tenant probe
|
|
93
|
+
GET /rest/v1/<table>?org_id=eq.<other_org>
|
|
94
|
+
GET /rest/v1/<table>?or=(org_id.eq.other,org_id.is.null)
|
|
95
|
+
|
|
96
|
+
# Write-path
|
|
97
|
+
PATCH /rest/v1/<table>?id=eq.<foreign_id>
|
|
98
|
+
DELETE /rest/v1/<table>?id=eq.<foreign_id>
|
|
99
|
+
POST /rest/v1/<table> with foreign owner_id
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
### PostgREST & REST
|
|
103
|
+
|
|
104
|
+
**Filters**
|
|
105
|
+
- `eq`, `neq`, `lt`, `gt`, `ilike`, `or`, `is`, `in`
|
|
106
|
+
- Embed relations: `select=*,profile(*)`—exploits overfetch if resolvers skip per-row checks
|
|
107
|
+
- Search leaks: generous `LIKE`/`ILIKE` filters combined with missing RLS → mass disclosure via wildcard queries
|
|
108
|
+
|
|
109
|
+
**Headers**
|
|
110
|
+
- `Prefer: return=representation` — echo writes
|
|
111
|
+
- `Prefer: count=exact` — exposure via counts
|
|
112
|
+
- `Accept-Profile`/`Content-Profile` — select schema
|
|
113
|
+
|
|
114
|
+
**IDOR Patterns**
|
|
115
|
+
```
|
|
116
|
+
/rest/v1/<table>?select=*&id=eq.<other_id>
|
|
117
|
+
/rest/v1/<table>?select=*&slug=eq.<other_slug>
|
|
118
|
+
/rest/v1/<table>?select=*&email=eq.<other_email>
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
**Mass Assignment**
|
|
122
|
+
- If RPC not used, PATCH can update unintended columns
|
|
123
|
+
- Verify restricted columns via database permissions/policies
|
|
124
|
+
|
|
125
|
+
### RPC Functions
|
|
126
|
+
|
|
127
|
+
RPC endpoints map to SQL functions. `SECURITY DEFINER` bypasses RLS unless carefully coded; `SECURITY INVOKER` respects caller.
|
|
128
|
+
|
|
129
|
+
**Anti-Patterns**
|
|
130
|
+
- `SECURITY DEFINER` + missing owner checks → vertical/horizontal bypass
|
|
131
|
+
- `set search_path` left to public; function resolves unsafe objects
|
|
132
|
+
- Trusting client-supplied `user_id`/`tenant_id` rather than `auth.uid()`
|
|
133
|
+
|
|
134
|
+
**Tests**
|
|
135
|
+
```bash
|
|
136
|
+
# Call as different users with foreign IDs
|
|
137
|
+
POST /rest/v1/rpc/<fn> {"user_id": "<foreign_id>"}
|
|
138
|
+
|
|
139
|
+
# Remove JWT entirely
|
|
140
|
+
Authorization: Bearer <anon_token>
|
|
141
|
+
```
|
|
142
|
+
Verify functions perform explicit ownership/tenant checks inside SQL.
|
|
143
|
+
|
|
144
|
+
### Storage
|
|
145
|
+
|
|
146
|
+
**Buckets**
|
|
147
|
+
- Public vs private; objects in `storage.objects` with RLS-like policies
|
|
148
|
+
|
|
149
|
+
**Misconfigurations**
|
|
150
|
+
```bash
|
|
151
|
+
# Public bucket with sensitive data
|
|
152
|
+
GET /storage/v1/object/public/<bucket>/<path>
|
|
153
|
+
|
|
154
|
+
# List prefixes without auth
|
|
155
|
+
GET /storage/v1/object/list/<bucket>?prefix=
|
|
156
|
+
|
|
157
|
+
# Signed URL reuse across tenants/paths
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
**Content-Type Abuse**
|
|
161
|
+
- Upload HTML/SVG served as `text/html` or `image/svg+xml`
|
|
162
|
+
- Verify `X-Content-Type-Options: nosniff` and `Content-Disposition: attachment`
|
|
163
|
+
|
|
164
|
+
**Path Confusion**
|
|
165
|
+
- Mixed case, URL-encoding, `..` segments may be rejected at UI but accepted by API
|
|
166
|
+
- Test path normalization differences between client validation and server handling
|
|
167
|
+
|
|
168
|
+
### Realtime
|
|
169
|
+
|
|
170
|
+
**Endpoint**: `wss://<ref>.supabase.co/realtime/v1`
|
|
171
|
+
|
|
172
|
+
**Risks**
|
|
173
|
+
- Channel names derived from table/schema/filters leaking other users' updates when RLS or channel guards are weak
|
|
174
|
+
- Broadcast/presence channels allowing cross-room join/publish without auth
|
|
175
|
+
|
|
176
|
+
**Tests**
|
|
177
|
+
- Subscribe to `public:realtime` changes on protected tables; confirm visibility aligns with RLS
|
|
178
|
+
- Attempt joining other users' channels: `room:<user_id>`, `org:<org_id>`
|
|
179
|
+
|
|
180
|
+
### GraphQL
|
|
181
|
+
|
|
182
|
+
**Endpoint**: `/graphql/v1` using pg_graphql with RLS
|
|
183
|
+
|
|
184
|
+
**Risks**
|
|
185
|
+
- Introspection reveals schema relations
|
|
186
|
+
- Overfetch via nested relations where resolvers skip per-row ownership checks
|
|
187
|
+
- Global node IDs leaked and reusable via different viewers
|
|
188
|
+
|
|
189
|
+
**Tests**
|
|
190
|
+
- Compare REST vs GraphQL responses for same principal and query shape
|
|
191
|
+
- Query deep nested fields; verify RLS holds at each edge
|
|
192
|
+
|
|
193
|
+
### Auth & Tokens
|
|
194
|
+
|
|
195
|
+
GoTrue issues JWTs with claims (`sub=uid`, `role`, `aud=authenticated`).
|
|
196
|
+
|
|
197
|
+
**Verification Requirements**
|
|
198
|
+
- Issuer, audience, expiration, signature, tenant context
|
|
199
|
+
|
|
200
|
+
**Pitfalls**
|
|
201
|
+
- Storing tokens in localStorage → XSS exfiltration
|
|
202
|
+
- Treating `apikey` as identity (it's project-scoped, not user identity)
|
|
203
|
+
- Exposing `service_role` key in client bundle or Edge Function responses
|
|
204
|
+
- Refresh token mismanagement leading to long-lived sessions beyond intended TTL
|
|
205
|
+
|
|
206
|
+
**Tests**
|
|
207
|
+
- Replay tokens across services; check audience/issuer pinning
|
|
208
|
+
- Try downgraded tokens (expired/other audience) against custom endpoints
|
|
209
|
+
|
|
210
|
+
### Edge Functions
|
|
211
|
+
|
|
212
|
+
Deno-based functions often initialize Supabase client with `service_role`.
|
|
213
|
+
|
|
214
|
+
**Risks**
|
|
215
|
+
- Trusting Authorization/apikey headers without verifying JWT against issuer/audience
|
|
216
|
+
- CORS: wildcard origins with credentials; reflected Authorization in responses
|
|
217
|
+
- SSRF via fetch; secrets exposed via error traces or logs
|
|
218
|
+
|
|
219
|
+
**Tests**
|
|
220
|
+
- Call functions with and without Authorization; compare behavior
|
|
221
|
+
- Try foreign resource IDs in payloads; verify server re-derives user/tenant from JWT
|
|
222
|
+
- Attempt to reach internal endpoints (metadata services) via function fetch
|
|
223
|
+
|
|
224
|
+
### Tenant Isolation
|
|
225
|
+
|
|
226
|
+
Ensure every query joins or filters by `tenant_id`/`org_id` derived from JWT context, not client input.
|
|
227
|
+
|
|
228
|
+
**Tests**
|
|
229
|
+
- Change subdomain/header/path tenant selectors while keeping JWT tenant constant
|
|
230
|
+
- Export/report endpoints: confirm queries execute under caller scope
|
|
231
|
+
|
|
232
|
+
## Bypass Techniques
|
|
233
|
+
|
|
234
|
+
- Content-type switching: `application/json` ↔ `application/x-www-form-urlencoded` ↔ `multipart/form-data`
|
|
235
|
+
- Parameter pollution: duplicate keys in JSON/query (PostgREST chooses last/first depending on parser)
|
|
236
|
+
- GraphQL+REST parity probing: protections often drift; fetch via the weaker path
|
|
237
|
+
- Race windows: parallel writes to bypass post-insert ownership updates
|
|
238
|
+
|
|
239
|
+
## Blind Enumeration
|
|
240
|
+
|
|
241
|
+
- Use `Prefer: count=exact` and ETag/length diffs to infer unauthorized rows
|
|
242
|
+
- Conditional requests (`If-None-Match`) to detect object existence
|
|
243
|
+
- Storage signed URLs: timing/length deltas to map valid vs invalid tokens
|
|
244
|
+
|
|
245
|
+
## Testing Methodology
|
|
246
|
+
|
|
247
|
+
1. **Inventory surfaces** - Map REST, Storage, GraphQL, Realtime, Auth, Functions endpoints
|
|
248
|
+
2. **Obtain principals** - Collect tokens for anon, user A/B, admin; check for `service_role` leaks
|
|
249
|
+
3. **Build matrix** - Resource × Action × Principal
|
|
250
|
+
4. **REST vs GraphQL** - Test both to find parity gaps
|
|
251
|
+
5. **Seed IDs** - Start with list/search endpoints to gather IDs
|
|
252
|
+
6. **Cross-principal** - Swap IDs, tenants, and transports across principals
|
|
253
|
+
|
|
254
|
+
## Tooling
|
|
255
|
+
|
|
256
|
+
- PostgREST: httpie/curl + jq; enumerate tables; fuzz filters (`or=`, `ilike`, `neq`, `is.null`)
|
|
257
|
+
- GraphQL: graphql-inspector, voyager; deep queries for field-level enforcement
|
|
258
|
+
- Realtime: custom ws client; subscribe to suspicious channels; diff payloads per principal
|
|
259
|
+
- Storage: enumerate bucket listing APIs; script signed URL patterns
|
|
260
|
+
- Auth/JWT: jwt-cli/jose to validate audience/issuer; replay against Edge Functions
|
|
261
|
+
- Policy diffing: maintain request sets per role; compare results across releases
|
|
262
|
+
|
|
263
|
+
## Validation Requirements
|
|
264
|
+
|
|
265
|
+
- Owner vs non-owner requests for REST/GraphQL showing unauthorized access (content or metadata)
|
|
266
|
+
- Mis-scoped RPC or Storage signed URL usable by another user/tenant
|
|
267
|
+
- Realtime or GraphQL exposure matching missing policy checks
|
|
268
|
+
- Minimal reproducible requests with role contexts documented
|