@query-ai/digital-workers 1.0.0

package/skills/detection-engineer/SKILL.md

@@ -0,0 +1,170 @@
+---
+name: detection-engineer
+description: Use when hunt findings need to be converted into production-ready detection artifacts (FSQL queries, Sigma rules, Query recipes) and coverage gaps need remediation plans
+---
+
+# Detection Engineer
+
+## Iron Law
+
+**EVERY DETECTION MUST HAVE A DOCUMENTED FALSE POSITIVE CONDITION.** A DETECTION WITHOUT TUNING GUIDANCE IS A NOISE GENERATOR.
+
+## When to Invoke
+
+Called by `threat-hunt` orchestrator at Phase 4 after hunt-pattern-analyzer classifies findings.
+
+## Process
+
+### Step 1: Receive Patterns
+
+Receive patterns from hunt-pattern-analyzer — both findings (things we CAN see) and coverage gaps (things we CAN'T see).
+
+### Step 2: Convert Findings into Detections
+
+For each finding:
+
+a. **Generate FSQL detection queries** — iterate with `FSQL_Query_Generation` -> `Search_FSQL_SCHEMA` -> `Validate_FSQL_Query` until correct. Do NOT fire-and-forget. Treat MCP tools as collaborators.
+
+b. **Generate Query platform recipes** via `RECIPE_FROM_FSQL_Query_Generation`.
+
+c. **Generate Sigma rules** for portable detections.
+
+d. **Test each detection** against the hunt time window — document hit rate and false positives. Use SUMMARIZE to measure hit rate and false positive distribution efficiently:
+
+   ```
+   -- Hit rate: how many events match the detection?
+   SUMMARIZE COUNT detection_finding.message
+   WITH <your_detection_filter> AFTER 7d
+
+   -- False positive rate: what percentage are already resolved/benign?
+   SUMMARIZE COUNT detection_finding.status_id GROUP BY detection_finding.status_id
+   WITH <your_detection_filter> AFTER 7d
+
+   -- Host distribution: is this firing on one host or many?
+   SUMMARIZE COUNT detection_finding.message GROUP BY detection_finding.device.hostname
+   WITH <your_detection_filter> AFTER 7d
+   ```
+
+   A detection that fires 500 times on 1 host is different from one that fires 5 times across 100 hosts. SUMMARIZE tells you this in one query instead of manually counting QUERY results.
+
+   > **Constraints:** SUMMARIZE has known execution limits — `status_id` filtering fails on detection_finding (use GROUP BY instead), `FROM` not supported, high-cardinality GROUP BY can overflow. If SUMMARIZE returns empty, fall back to QUERY. See fsql-expert Layer 1c for workarounds and check `summarize_support` in the environment profile.
+
+e. **Document false positive conditions** for EVERY detection.
+
+#### MCP Tool Interaction — Iterative, Not Fire-and-Forget
+
+For each detection query:
+
+1. Call `FSQL_Query_Generation` with the behavioral pattern description
+2. Review the generated query — is it accurate? Does it use the right fields?
+3. Call `Search_FSQL_SCHEMA` to verify field paths exist
+4. Call `Validate_FSQL_Query` — if validation fails, adjust and re-validate
+5. Call `Execute_FSQL_Query` against the hunt time window to test hit rate
+6. Call `RECIPE_FROM_FSQL_Query_Generation` with the validated query
+7. Document the detection with FP conditions
+
+### Step 3: Build Gap Remediation Plan
+
+For each coverage gap:
+
+a. **Document the gap**: what data is missing, which TTPs it blocks, what connector/event type/field would fill it.
+
+b. **Assess impact**: what threats are invisible because of this gap.
+
+c. **Propose remediation**: specific connector deployment, field mapping request, or platform config change.
+
+d. **Prioritize by risk**: gaps that blind us to high-impact TTPs rank higher.
+
+## Output
+
+Two deliverables are returned to the calling orchestrator.
+
+### Deliverable 1 — Detection Package
+
+```
+DETECTION PACKAGE
+━━━━━━━━━━━━━━━━
+Hunt: [hunt ID]
+Date: [date]
+Findings converted: [N]
+
+Detection 1: [name]
+  Trigger: [what behavior this detects]
+  MITRE ATT&CK: [technique ID] — [technique name]
+
+  FSQL Detection Query:
+    [the validated FSQL query]
+
+  Sigma Rule:
+    title: [detection name]
+    status: experimental
+    description: [what it detects]
+    logsource:
+      category: [category]
+      product: [product]
+    detection:
+      selection:
+        [field]: [value]
+      condition: selection
+    falsepositives:
+      - [documented FP condition 1]
+      - [documented FP condition 2]
+    level: [medium/high/critical]
+    tags:
+      - attack.[tactic]
+      - attack.[technique_id]
+
+  Query Recipe: [recipe output from RECIPE_FROM_FSQL_Query_Generation]
+
+  Validation Results:
+    FSQL validation: PASS
+    Test against hunt window: [N] hits, [N] false positives
+    Estimated FP rate: [percentage or qualitative assessment]
+
+  False Positive Conditions:
+    1. [specific condition that would trigger this detection benignly]
+    2. [another FP condition]
+    Tuning guidance: [how to reduce FPs without losing true positives]
+
+Detection 2: [...]
+```
+
+### Deliverable 2 — Gap Remediation Plan
+
+```
+GAP REMEDIATION PLAN
+━━━━━━━━━━━━━━━━━━━
+Hunt: [hunt ID]
+Date: [date]
+
+Gap 1: [description]
+  Missing: [event type / field / connector capability]
+  Blocks: [MITRE techniques that cannot be hunted]
+  Impact: [HIGH/MEDIUM/LOW] — [what threats are invisible]
+  Remediation: [specific action — deploy X, configure Y, request mapping Z]
+  Priority: [1-5, 1 = critical blind spot]
+
+Gap 2: [...]
+
+SUMMARY:
+  Detections created: [N]
+  Gaps identified: [N]
+  TTPs fully covered: [list]
+  TTPs partially covered: [list]
+  TTPs blind: [list — these are the risks]
+```
+
+**Return detection package and gap remediation plan to the threat-hunt orchestrator. Do not present to the user or wait for input.**
+
+## Red Flags
+
+| Red Flag | Correct Action |
+|----------|---------------|
+| "Detection query works, ship it" without testing | STOP. Every detection must be tested against the hunt time window. Document hit rate AND false positives. |
+| Detection without false positive documentation | STOP. A detection without FP conditions is a noise generator. Document at least one FP scenario. |
+| Accepting first FSQL_Query_Generation output without review | STOP. Iterate. Verify fields via Search_FSQL_SCHEMA. Validate. Cross-reference. |
+| Skipping Sigma rule generation | STOP. Sigma rules provide portability. Not every environment uses Query. Generate both FSQL and Sigma. |
+| Coverage gap without remediation recommendation | STOP. Identifying a gap without proposing a fix is half the job. Be specific: which connector, which field, which configuration. |
+| Gap remediation without priority | STOP. Not all gaps are equal. A gap that blinds you to lateral movement is more critical than one that blocks geolocation. Prioritize by risk. |
+| Hardcoding connector names in detections | STOP. Detections should use OCSF event types and field paths, not connector-specific references. They must work across any connector that maps to the same OCSF schema. |
+| Not generating a Query recipe | STOP. The recipe is the deployment artifact for the Query platform. Always call RECIPE_FROM_FSQL_Query_Generation. |

package/skills/evidence-quality-checker/SKILL.md

@@ -0,0 +1,109 @@
+---
+name: evidence-quality-checker
+description: Use at Gate 2 and Gate 3 exits to verify data quality and analytical reasoning before proceeding — catches status filtering errors, vendor-label anchoring, and missing-data gaps before they cascade into false escalations
+---
+
+# Evidence Quality Checker
+
+## Iron Law
+
+**CHECK THE EVIDENCE QUALITY BEFORE BUILDING THE NARRATIVE.**
+
+Reasoning errors cascade. A missing `status_id` filter in Gate 2 becomes a false APT attribution in Gate 4 and a wrong escalation in Gate 6. These checkpoints exist to catch errors before they compound.
+
+## When to Invoke
+
+- **Gate 2 exit**: Data quality pass — invoked by `alert-investigation` after enrichment, before severity scoring
+- **Gate 3 exit**: Analytical reasoning pass — invoked by `alert-investigation` after severity scoring, before specialist invocation
+
+## Reasoning Principles
+
+Five principles anchor all checks. Understand the *why* before running the *what*.
+
+### 1. Status before narrative
+
+An alert's `status_id` determines whether it's actionable. RESOLVED/Benign alerts are closed investigations, not active threats. Always separate findings by status before drawing conclusions.
+
+This applies to **findings events only** (`detection_finding`, `security_finding`, `vulnerability_finding`, `incident_finding`). Note: `authentication.status_id` means SUCCESS/FAILURE (auth result, different concept). Telemetry events (`network_activity`, `process_activity`, `file_activity`) don't have an alert lifecycle `status_id`.
+
+### 2. Vendor labels are hypotheses, not facts
+
+"Dukozy malware detected" means a signature matched. "RESOLVED/Benign" means the same vendor investigated and dismissed it. The label and the resolution come from the same source — you can't trust one and ignore the other.
+
+### 3. Volume is not evidence
+
+"~600 alerts" is a count, not a finding. 600 resolved alerts is less concerning than 3 unresolved ones. Signal/noise separation must happen before any alert count appears in a report.
+
+### 4. Absence is a finding
+
+If file hashes are empty, authentication logs are unavailable, or a connector returns nothing, that's not a blank to skip over. It constrains what conclusions you can reach. State it explicitly: "Cannot confirm X because Y data is unavailable."
+
+### 5. Confidence tracks evidence, not severity
+
+High-severity source labels don't mean high-confidence findings. Confidence comes from corroboration across independent sources, hash verification, behavioral correlation. If your only evidence is vendor labels, confidence is MEDIUM at best. Five detections from the same platform are one source, not five.
+
+---
+
+## Gate 2 Checkpoint — Data Quality
+
+Run at Gate 2 exit, before severity scoring. Five binary checks, designed for <60 seconds.
+
+Review the enrichment queries and results from Gate 2, then evaluate each check:
+
+| # | Check | If No |
+|---|-------|-------|
+| 1 | Do all **findings** lookback queries (time range beyond the initial intake window, e.g., 7d/30d) include `status_id` in field selectors? | Re-run with `status_id` before proceeding |
+| 2 | Have findings results been separated into NEW vs. RESOLVED vs. null buckets? | Separate now — count only NEW as active |
+| 3 | Are any IOC types expected but absent (e.g., no file hashes in malware detections)? | Document as data gap with impact on conclusions |
+| 4 | Did any enrichment queries return zero results? | Note which data sources returned empty and why |
+| 5 | Is alert count reported without status breakdown? | Add status breakdown before any count reaches the report |
+
+---
+
+## Gate 3 Checkpoint — Analytical Reasoning
+
+Run at Gate 3 exit, after severity scoring, before specialist invocation. Eight checks, ~90 seconds.
+
+Review the severity scoring, Five W's, and accumulated evidence, then evaluate each check:
+
+| # | Check | If No |
+|---|-------|-------|
+| 1 | For each vendor detection label cited as evidence: has its `status_id` and `status_detail` been checked? | Check now — a RESOLVED/Benign detection is not evidence of active compromise |
+| 2 | Is the threat actor attribution supported by NEW/unresolved detections, not just vendor labels? | Downgrade attribution confidence or remove claim |
+| 3 | Does the severity score reflect actual evidence quality, or is it anchored to source severity labels? | Re-score Confidence dimension based on evidence, not labels |
+| 4 | Are all "data not available" findings documented with impact on conclusions? | Add to Five W's and report |
+| 5 | Have the appropriate specialist skills been identified for Gate 4 invocation based on alert types? | Identify now and ensure Gate 4 invokes them |
+| 6 | Can every conclusion in the Five W's be traced to a specific query result — and if sourced from findings data, verified as `status_id = NEW`? | Remove or downgrade unsupported conclusions |
+| 7 | If `status_detail = "UnsupportedAlertType"`, has this been flagged as potential integration gap rather than confirmed threat? | Add caveat to findings |
+| 8 | If the environment profile was used to skip queries (known FPs, unpopulated fields, broken observables), is each skip justified by a profile entry with `last_verified` < 7 days ago? | Re-verify stale entries before relying on them — run the query instead of skipping |
+
+---
+
+## Output Format
+
+After running each checkpoint, output:
+
+```
+EVIDENCE QUALITY CHECK — [Gate 2 | Gate 3]
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+[1] [Check description]: PASS | FAIL
+    [If FAIL: what was found and what action to take]
+
+[2] [Check description]: PASS | FAIL
+    ...
+
+Result: ALL PASS | [N] FAILURES — fix before proceeding
+```
+
+If any check FAILs, fix the issue before proceeding to the next gate. Do not defer fixes to later gates or to the report.
+
+**After the check completes — whether all checks pass or after failures are fixed — immediately continue to the next gate without pausing.** This is an inline quality gate, not a stopping point. Do not wait for user input. Print the results and keep going.
+
+## Red Flags
+
+| Red Flag | Correct Action |
+|----------|---------------|
+| "All checks passed" in under 10 seconds | STOP. You didn't actually check. Review each item against the evidence. |
+| "This check doesn't apply to this investigation" | It might not. But state WHY it doesn't apply — don't just skip. |
+| "I'll fix this in the report" | STOP. Fix it now. The point is to prevent errors from cascading, not to annotate them later. |

package/skills/fsql-expert/SKILL.md

@@ -0,0 +1,308 @@
+---
+name: fsql-expert
+description: Use when any FSQL query needs to be authored, validated, or executed against the Query Data Mesh — handles all data access for investigations
+---
+
+# FSQL Expert
+
+## ABSOLUTE RULES — Read These First
+
+**1. NEVER use Bash, cat, python, jq, or any shell command to process query results.** If results overflow to a file, your query was too broad. Re-run with specific field selectors or tighter filters. DO NOT cat the file. DO NOT pipe to python. NEVER.
+
+**2. NEVER use `**` on broad queries.** Use specific field selectors: `detection_finding.message, detection_finding.severity_id, detection_finding.time, detection_finding.observables`. Only use `**` when scoped to a single host or single event type.
+
+**3. ALWAYS validate before execute.** Every query goes through `Validate_FSQL_Query` first. The query you validate must be identical to the query you execute.
+
+## Iron Law
+
+**NO QUERY HITS THE MESH WITHOUT VALIDATION FIRST.**
+
+Every FSQL query must be validated via `Validate_FSQL_Query` before execution via `Execute_FSQL_Query`.
+
+## Overview
+
+You are the data access layer for all Digital Workers investigations. Every piece of evidence comes through you. Your job is to author precise FSQL queries, validate them against the live schema, execute them against the mesh, and return structured results.
+
+## Available MCP Tools
+
+The Query Data Mesh provides these MCP tools. **Call them directly as MCP tools — do NOT use ToolSearch to find them.** They are provided by the QueryDemoMCP MCP server configured in `.mcp.json`.
+
+| Tool | Purpose |
+|------|---------|
+| `Execute_FSQL_Query` | Run a validated FSQL query against the mesh. Returns OCSF events. |
+| `Validate_FSQL_Query` | Check if an FSQL query (QUERY or SUMMARIZE) is syntactically valid before execution. |
+| `Search_FSQL_SCHEMA` | Search the schema vector database for attributes and event types. |
+| `FSQL_Connectors` | List all available connectors in the mesh (data sources). |
+| `FSQL_Query_Generation` | Generate FSQL from natural language (useful for complex queries). |
+| `KQL_TO_FSQL_Query_Generation` | Convert KQL queries to FSQL. |
+| `SIGMA_TO_FSQL_Query_Generation` | Convert Sigma detection rules to FSQL. |
+
+## Reference — MANDATORY Session Start
+
+**Step 1 (REQUIRED): Read the MCP syntax reference.** At the start of every investigation session, read `fsql://docs/syntax-reference` via `ReadMcpResourceTool` (server: `QueryDemoMCP`). This unlocks advanced operators you will need — set operations (UNION/INTERSECTION/EXCEPT), type filters (@ip, @user), ~40 observable types, and depth modifiers. Sessions that skip this step produce simpler, less comprehensive queries. Do not proceed to query authoring until you have read this resource.
+
+**Step 2: Read the investigation cheat sheet.** See `fsql-reference.md` in this skill directory for investigation-specific patterns — the Layer 1a/1b discovery approach, event-type-specific field selectors, telemetry pivot patterns, and status filtering rules. This is your playbook for how to structure investigation queries.
+
+**How these two resources work together:**
+- The **cheat sheet** gives you the investigation methodology — Layer 1a discovery, pivot patterns, field selectors per event type, status_id rules
+- The **MCP syntax reference** gives you the full operator toolkit — set operations, type filters, advanced observables, category selectors
+- Use both. Agents that only use the cheat sheet write correct but basic queries. Agents that also internalize the syntax reference write sophisticated queries using set operations and type filters that find more evidence.
+
+**Additional MCP resources** (read as needed during investigation):
+- `fsql://docs/categories-and-events` — all OCSF categories and event classes (read when pivoting to unfamiliar event types)
+- `fsql://docs/kql-conversion-guide` — converting KQL queries to FSQL
+- `fsql://docs/sigma-logsource-mappings` — converting Sigma detection rules to FSQL
+
+**Step 3: Read the environment profile.** If `digital-workers/learned/environment-profile.json` exists, read it. This tells you what previous investigations learned about this mesh — which fields are populated per connector, which observables work, which event types have data, query performance limits, and known false positives. Use this knowledge to avoid re-discovering known gaps.
+
+Key profile sections and how they affect query authoring:
+- **`field_population[connector_id]`** — Skip known-unpopulated field paths on specific connectors. Use the `workaround` field for alternatives (e.g., `%ip` instead of `device.hostname`). Check if other connectors producing the same event type have the field before giving up entirely.
+- **`event_type_availability[connector_id]`** — Skip event types marked `no_data` on all connectors. If some connectors are `untested`, probe one with a `FROM <connector_id>` query before declaring a gap.
+- **`observable_support`** — Skip observables marked `not_working`. Check `connectors_tested` — if not all relevant connectors have been tested, consider probing an untested one.
+- **`query_performance`** — Respect batch size limits, known overflow patterns, and query reliability notes. For `%ip` discovery scans, always use single-IP queries (IN operator is unreliable).
+- **`known_false_positives`** — If a detection message is listed with `last_verified` < 7 days ago, skip re-verification and cite the profile (include the last_verified date and sample size in your findings).
+- **`systemic_patterns`** — Recurring alert patterns that are un-actionable but not vendor-resolved FPs (e.g., `UnsupportedAlertType` alerts with no enrichment data). Fast-track through investigation — minimal enrichment, profile-cited disposition.
+- **`connector_behaviors`** — Connector-level quirks that affect all event types (e.g., "status_id always null" on CrowdStrike). Use these to interpret findings correctly — a null status_id from a connector that never populates it is different from a genuinely unknown status.
+- **`summarize_support[event_type]`** — Check before writing SUMMARIZE queries. If the event type is `not_working`, skip SUMMARIZE and use QUERY. If `partial`, check `known_issues` for the specific filter or GROUP BY you plan to use. If `untested`, try SUMMARIZE and update the profile with the result.
+
+If the profile doesn't exist, proceed normally. Your discoveries during this investigation will create it.
+
+## The Three-Layer Process
+
+### Layer 1: Author the Query
+
+Using the FSQL syntax reference, construct a query for the investigation need:
+- **Use specific field selectors, not `**`.** Select only the fields you need (e.g., `detection_finding.message, detection_finding.severity_id, detection_finding.time, detection_finding.observables`). The `**` wildcard returns the entire OCSF event with all nested objects, which produces massive payloads. Only use `**` when you genuinely need every field.
+- Choose the right event type(s) or use category selectors (`#network`, `#findings`)
+- Use observable shortcuts (`%ip`, `%hash`, `%email`) for broad searches
+- Apply appropriate filters and time ranges
+- Scope with `FROM` clause only when targeting specific connectors
+- For complex queries, use `FSQL_Query_Generation` to generate from natural language, then review and adjust
+
+### Layer 2: Discover Schema (When Needed)
+
+Before querying unfamiliar event types or filtering on fields you haven't used, call `Search_FSQL_SCHEMA`:
+
+```
+Search_FSQL_SCHEMA(query="detection_finding severity", limit=10)
+```
+
+This tells you:
+- What fields actually exist for this event type (`fsql_path`)
+- Whether a field is a string, enum, array, IP, etc. (`data_type`)
+- Enum values if applicable (`enum_values`)
+- Attribute descriptions for understanding field semantics
+
+Also use `FSQL_Connectors` to see what data sources are available in the mesh.
+
+**ALWAYS run schema search when:**
+- First time querying an event type in this investigation (before writing the query, not after it fails)
+- Layer 1a discovery reveals an event type you haven't queried before — look up its fields before writing the Layer 1b query
+- Unsure if a field path is correct
+- Need to know if a field is an enum (use `IN`) vs. string (use `CONTAINS`)
+
+**Skip schema search when:**
+- Using well-known patterns from the reference (e.g., `detection_finding.severity_id IN HIGH, CRITICAL`)
+- Repeating a query pattern that already succeeded in this investigation
+
+### Layer 3: Validate Then Execute
+
+1. **Validate:** Call `Validate_FSQL_Query` with your query — returns `is_valid: true/false` with error detail
+2. **Check:** If validation fails, read the error detail, fix the query, validate again
+3. **Execute:** Only after `is_valid: true`, call `Execute_FSQL_Query` to run against the mesh
+4. **Self-correct (MANDATORY):** If execution returns empty results when data is expected, OR returns an error about field paths, you MUST call `Search_FSQL_SCHEMA` to discover the correct fields and retry with fixed paths. Do not document a data gap without first attempting schema search and retry. The schema search tells you the exact field paths — use them.
+5. **IOC extraction fallback chain.** When querying for IOCs (users, IPs, hostnames) from detection findings:
+   - **Try structured fields first:** `detection_finding.actor.user.email_addr`, `detection_finding.finding_info`, `detection_finding.observables`
+   - **If structured fields are empty:** Query `detection_finding.raw_data` for the same alerts. Extract IOCs from JSON keys — common patterns: `impacted_assets` (user: hostname), `impacted_ips`, `entities`, `evidence`
+   - **Check the profile:** If `field_population` shows these fields as `unpopulated` on the relevant connector, skip straight to raw_data (saves a wasted query)
+   - **Do NOT probe multiple field paths.** One structured query + one raw_data query max. If both are empty, document the gap.
+6. **Update the environment profile.** After any of these outcomes, update `digital-workers/learned/environment-profile.json` immediately (do not wait until end of investigation):
+   - **Query returned 0 results on a field path** → Add `field_population` entry for the connector(s) that produce this event type. If the query used a `FROM` clause, attribute to that connector with `confidence: "high"`. Otherwise, attribute to all connectors producing the event type with `confidence: "medium"`.
+   - **Observable query errored or returned empty** → Add `observable_support` entry with `status: "not_working"` and note which `connectors_tested`.
+   - **Query overflowed (results saved to file)** → Add `query_performance` entry with the pattern description and threshold.
+   - **Event type returned 0 results** → Add `event_type_availability` entry for the relevant connector(s).
+   - **Query returned data for a field/event type previously marked unpopulated** → Update the existing entry to `status: "populated"` or `status: "has_data"` and refresh `last_verified`.
+   - **SUMMARIZE succeeded on an event type** → Add or update `summarize_support` entry with `status: "working"`, note `filters_tested` and `group_by_tested`.
+   - **SUMMARIZE failed or returned empty** → Add or update `summarize_support` entry with `status: "not_working"` or `"partial"`, document the failure in `known_issues`.
+   - **SUMMARIZE succeeded on a previously `not_working` event type** → Update to `status: "working"` (platform may have fixed the issue).
+
+   When updating, always set `last_verified` to today's date, `source_investigation` to the current investigation ID, and preserve existing entries for other connectors.
+
+## Red Flags
+
+| Red Flag | Correct Action |
+|----------|---------------|
+| "I'll just run this query and see" | STOP. Call `Validate_FSQL_Query` first. |
+| "This field name looks right" | STOP. Use `Search_FSQL_SCHEMA` to verify. |
+| "No results — the data isn't there" | STOP. Try broader query, different time range, observable search. Check `FSQL_Connectors` for available data sources. |
+| "I'll skip validation, this is a simple query" | STOP. ALL queries get validated. Simple queries have simple validations. |
+| Running the same failed query a second time | STOP. Use `Search_FSQL_SCHEMA` to understand why it failed. Fix before retrying. |
+| Using Bash, cat, or Python to parse MCP tool results | STOP. **Never use Bash commands to process query results.** Analyze inline results directly. If results overflow to a file, DO NOT read the file — re-run the query with specific field selectors to get a smaller result that fits inline. The file is a signal that your query was too broad. |
+| Query result was saved to a file (too large for context) | STOP. **Do not cat, Read, or process the file.** Your query was too broad. Re-run with specific field selectors, tighter filters (single host, single event type), or a narrower time range. |
+| Using ToolSearch to find MCP tools | STOP. Call `Execute_FSQL_Query`, `Validate_FSQL_Query`, etc. directly. They are MCP tools, not deferred tools. |
+
+## Query Strategy for Investigations
+
+### Layered Query Approach
+
+#### Layer 1a — Discovery Scan (always start here)
+
+Find ALL activity for IOCs across the mesh without overflowing. Use `*.message, *.time` — this returns one lightweight row per matching event with the `__event` field showing which event type it came from.
+
+**BATCH same-type IOCs using `IN` where reliable — but use single queries for `%ip`:**
+
+**`%ip` discovery: one query per IP.** The `%ip IN` operator is unreliable on discovery scans — 4-IP batches always error, 2-IP batches are intermittent. Use single-IP queries for guaranteed results:
+
+```
+-- IP discovery: one query per IP (reliable)
+QUERY *.message, *.time WITH %ip = '10.0.0.1' AFTER 24h
+QUERY *.message, *.time WITH %ip = '10.0.0.2' AFTER 24h
+
+-- Hashes: IN batching works reliably
+QUERY *.message, *.time WITH %hash IN 'f6c3023f', 'a1b2c3d4' AFTER 7d
+```
+
+Read the `__event` field in the results — it tells you where your IOCs appeared (e.g., `detection_finding`, `email_activity`, `osint_inventory_info`, `process_activity`). This is how you discover event types you wouldn't have thought to query.
+
+**Never use bare `QUERY %hash = 'x'` or `QUERY ** WITH %hash = 'x'` for discovery.** These return full OCSF events and will overflow on any IOC with significant activity.
+
+#### Layer 1b — Targeted Detail
+
+Once Layer 1a tells you which event types have hits, query those specific event types with the fields you need:
+
+```
+-- Layer 1a showed hits in email_activity and detection_finding
+QUERY email_activity.message, email_activity.time, email_activity.actor.user.email_addr
+WITH %hash = 'f6c3023f' AFTER 7d
+
+QUERY detection_finding.message, detection_finding.severity_id, detection_finding.status_id,
+      detection_finding.device.hostname
+WITH %hash = 'f6c3023f' AND detection_finding.status_id = NEW AFTER 7d
+```
+
+If you don't know the field paths for an event type, run `Search_FSQL_SCHEMA` first (see Layer 2).
+
+#### Layer 1c — Aggregation & Counting
+
+When Layer 1a/1b results return many records and you need distributions or unique counts rather than individual events, use SUMMARIZE:
+
+```
+-- Status distribution (see which alerts are NEW vs RESOLVED)
+SUMMARIZE COUNT detection_finding.status_id
+GROUP BY detection_finding.status_id
+WITH detection_finding.severity_id IN HIGH, CRITICAL AFTER 7d
+
+-- Alert type breakdown with status (GROUP BY status_id instead of filtering on it)
+SUMMARIZE COUNT detection_finding.message
+GROUP BY detection_finding.message, detection_finding.status_id
+WITH detection_finding.severity_id IN HIGH, CRITICAL AFTER 24h
+
+-- Per-host alert distribution
+SUMMARIZE COUNT detection_finding.message
+GROUP BY detection_finding.device.hostname, detection_finding.severity_id
+AFTER 7d
+
+-- Authentication: unique IPs per user (works reliably with all filters)
+SUMMARIZE COUNT DISTINCT authentication.device.ip
+GROUP BY authentication.actor.user.email_addr
+WITH authentication.status_id = FAILURE AFTER 24h
+```
+
+**When to use SUMMARIZE vs QUERY:**
+- Need individual events (IOCs, timelines, raw_data)? → QUERY with field selectors
+- Need counts, distributions, or unique entity counts? → SUMMARIZE
+- Unsure? Start with QUERY. Switch to SUMMARIZE when you catch yourself manually counting results.
+
+**Rules:**
+- All fields must reference the same OCSF event class (cross-event-class fails validation)
+- Validate before execute applies — pass SUMMARIZE queries through `Validate_FSQL_Query`
+- SUMMARIZE queries do NOT need the `VALIDATE` prefix — the tool adds it automatically
+- Include status_id in GROUP BY on findings lookbacks to separate NEW from RESOLVED (see constraints below)
+
+**Known execution constraints:**
+- **detection_finding + `status_id = NEW`:** Executor errors or returns empty. Workaround — omit `status_id` from the WITH filter, add it to GROUP BY instead:
+  ```
+  -- FAILS: status_id as a filter
+  SUMMARIZE COUNT detection_finding.message WITH detection_finding.status_id = NEW AFTER 24h
+
+  -- WORKS: status_id as a GROUP BY dimension
+  SUMMARIZE COUNT detection_finding.message GROUP BY detection_finding.message, detection_finding.status_id AFTER 24h
+  -- Read the status_id column to identify which rows are NEW vs RESOLVED
+  ```
+- **`FROM` not supported:** SUMMARIZE always queries all connectors. For connector-specific analysis, use QUERY with FROM.
+- **High-cardinality GROUP BY overflows:** GROUP BY severity_id, status_id, hostname = safe (low cardinality, <100 values). GROUP BY IP, hash, username = always scope with a WITH filter first. Unfiltered GROUP BY on network_activity.src_endpoint.ip overflowed at 3.1M chars in testing.
+- **email_activity, file_activity:** SUMMARIZE execution fails on these event types. Use QUERY.
+- **Fallback rule:** If SUMMARIZE returns empty `{}` or "No data were processed" error, fall back to QUERY with field selectors. Do not document a data gap based on empty SUMMARIZE results.
+- **Check the environment profile** (`summarize_support` section) before writing SUMMARIZE queries. If the event type is `not_working`, skip SUMMARIZE. If `partial`, check `known_issues`. If `untested`, try SUMMARIZE and update the profile with the result.
+
+#### Layer 2 — Category with Key Fields
+
+Query by OCSF category when investigating a class of activity rather than a specific IOC:
+
+```
+QUERY #network.src_endpoint.ip, #network.dst_endpoint.ip, #network.message, #network.time
+WITH #network.src_endpoint.ip = '10.0.0.1' AFTER 48h
+```
+
+#### Layer 3 — Full Event (scoped, rare)
+
+Use `**` only when scoped to a single host AND single event type AND narrow time window:
+
+```
+QUERY process_activity.** WITH process_activity.device.hostname = 'BD-2578' AFTER 24h
+```
+
+**Never** use `**` on broad observable searches or multi-host queries.
+
+### Follow-Up Queries
+
+When initial results reveal IOCs, entities, or patterns:
+- Extract IOCs (IPs, hashes, domains, usernames) from results
+- Immediately author follow-up queries to search for those IOCs across the mesh
+- Continue until the investigation picture is complete or no new leads emerge
+
+### Time Range Strategy
+
+- Start with the alert's time window (usually `AFTER 24h`)
+- Expand to `AFTER 7d` if looking for patterns or persistence
+- Use `AFTER 30d` for threat hunting or campaign correlation
+- Narrow with `BEFORE` and `AFTER` for precise timeline reconstruction
+
+### Status Awareness on Lookback Queries
+
+**When expanding time ranges beyond the initial 24h window on findings queries (`detection_finding`, `security_finding`, `vulnerability_finding`, `incident_finding`), always include `status_id` in your field selectors.** Historical findings may have been resolved as benign. If you build an investigation narrative on RESOLVED/Benign alerts, you will produce a false escalation.
+
+Note: `authentication.status_id` means auth result (SUCCESS/FAILURE) — a different concept. Telemetry events (`network_activity`, `process_activity`, `file_activity`) don't have an alert lifecycle `status_id`. This rule applies to findings events only.
+
+```
+-- WRONG: no status_id — treats resolved alerts as active threats
+QUERY detection_finding.message, detection_finding.severity_id
+WITH detection_finding.device.hostname = 'BD-2578' AFTER 30d
+
+-- RIGHT: includes status_id so you can separate active from resolved
+QUERY detection_finding.message, detection_finding.severity_id, detection_finding.status_id
+WITH detection_finding.device.hostname = 'BD-2578' AFTER 30d
+```
+
+When analyzing results, separate findings by status:
+- `status_id = NEW` → actionable, include in investigation
+- `status_id = RESOLVED` + `status_detail = "Benign"` → already closed, do NOT cite as evidence of active compromise
+- `status_id = null` → unknown, investigate cautiously
+
+### Status Detail Interpretation
+
+When analyzing `status_id` results, also check `status_detail`. Common patterns:
+- `status_detail = "Benign"` with `status_id = RESOLVED` — vendor investigated and closed as false positive
+- `status_detail = "UnsupportedAlertType"` with `status_id = NEW` — the integration cannot auto-resolve this alert type. `NEW` may reflect an integration gap, not a genuine untriaged alert. Flag this ambiguity in findings.
+
+## Output Format
+
+**Return results to the calling skill and continue. Do not present to the user or wait for input — the calling skill determines next steps.**
+
+When returning results to the calling skill, always include:
+1. **The FSQL query executed** (exact text)
+2. **Validation status** (passed EXPLAIN GRAPHQL)
+3. **Result summary** (count, key fields, notable findings)
+4. **Raw results** (structured data for further analysis)
+5. **Suggested follow-up queries** (if results suggest additional investigation paths)
+6. **Data completeness flags** — If expected fields are empty (e.g., all file hash fields are null on malware detection findings), flag this in the summary: "WARNING: [field] is empty across all results — [impact on investigation]"