log10x-mcp 1.30.21 → 1.30.22

package/build/product-kb/docs/faq/stacks/splunk/compatibility.md

@@ -3,26 +3,27 @@ icon: material/check-circle-outline
 title: "Compatibility"
 ---
 
-Splunk dashboards, Universal Forwarders, testing on your own environment, HEC integration, Splunk Enterprise on-premises, and the full Splunk Cloud KV-Store pilot checklist.
+Splunk dashboards, Universal Forwarders, testing on your own environment, HEC integration, Splunk Enterprise on-premises, and Splunk Cloud Platform.
 
 ??? tenx-compat "Does 10x work with my existing Splunk dashboards and queries"
 
-    **Yes.** 10x preserves all Splunk field mappings and metadata.
+    **Yes.** Compact events keep `host`, `source`, `sourcetype` and `index`; field extractions apply to the expanded text.
 
     **Fields preserved:**
 
-    - `_time`, `host`, `source`, `sourcetype`
-    - `_raw`, `index`
-    - All field extractions (props.conf, transforms.conf)
-    - Custom metadata and tags
+    - `host`, `source`, `sourcetype`, `index`
+    - `_raw`, expanded at search time
+    - Field extractions, applied to the expanded text
 
-    **Functionality preserved:**
+    `_time` on a compact event is the time Splunk indexed it; the original timestamp is in the expanded line.
 
-    - Saved searches and dashboards require zero changes
-    - Scheduled alerts are compiled once at save time (via the **10x Compile Alert** view) so they run natively server-side
-    - SPL queries return the same results
+    **How searches reach compact events:**
 
-    **How optimization works:** The 10x Engine uses [template-based](https://doc.log10x.com/run/transform/){target="_blank"} encoding, not field removal. Each repeated log event is replaced by a compact reference to its template while maintaining full searchability in Splunk.
+    - Classic dashboards in an app that carries `dashboard.js` keep their SPL unchanged; Dashboard Studio panels use `tenxsearch`
+    - The search bar, saved searches and the REST API wrap the search in the `tenxsearch` command
+    - Alerts created in the **Compile Alert** view are compiled once at save time, so they run natively server-side; an alert saved from the search page wraps its search in `tenxsearch`
+
+    **How optimization works:** The 10x Engine uses [template-based](https://doc.log10x.com/run/transform/){target="_blank"} encoding, keeping every value. Each repeated log event is replaced by a compact reference to its template while maintaining full searchability in Splunk.
 
 ??? tenx-compat "Does this work with Universal Forwarders"
 
@@ -36,7 +37,7 @@ Splunk dashboards, Universal Forwarders, testing on your own environment, HEC in
 
     1. **[Cost POC](https://doc.log10x.com/apps/mcp/)**, install the 10x MCP and ask it to run a POC on your Splunk logs. The engine runs on your own machine, results in minutes. No account, no credit card.
     2. **[Reporter](https://doc.log10x.com/apps/reporter/)**, Deploy as a DaemonSet alongside your forwarder for cost visibility before logs reach Splunk. Alternatively, the [MCP server](https://github.com/log-10x/log10x-mcp){target="_blank"}'s SIEM-sample tool connects to Splunk via REST API for agentless Splunk-side analysis (no DaemonSet).
-    3. **[Receiver](https://doc.log10x.com/apps/receiver/deploy/)**, Deploy via Helm chart alongside your forwarder. Filter mode (sampling) or [Compact mode](https://doc.log10x.com/apps/receiver/compact/) (lossless shrink via the [10x for Splunk](https://doc.log10x.com/apps/receiver/compact/splunk/) app). ~30 min setup.
+    3. **[Receiver](https://doc.log10x.com/apps/receiver/deploy/)**, Deploy via Helm chart alongside your forwarder. Filter mode (sampling) or [Compact mode](https://doc.log10x.com/apps/receiver/compact/) (lossless shrink via the [Log10x App](https://doc.log10x.com/apps/receiver/compact/splunk/)). ~30 min setup.
     4. **[Retriever](https://doc.log10x.com/apps/retriever/)**, Route events to S3, stream selected data to Splunk on-demand.
 
     Each step is independent, start with the cost POC to see your reduction ratio, then move to production when ready.
@@ -68,11 +69,11 @@ Splunk dashboards, Universal Forwarders, testing on your own environment, HEC in
 
     **Agentless Splunk analysis:** The [MCP server](https://github.com/log-10x/log10x-mcp){target="_blank"}'s SIEM-sample tool queries Splunk REST API for cost analysis without deploying a DaemonSet.
 
-    **Compatibility:** Works with Splunk Cloud Platform (all regions), Splunk Enterprise 8.x, 9.x, and 10.x. No changes to HEC configuration required.
+    **Compatibility:** The Receiver and Reporter send to any Splunk HEC endpoint, on Splunk Cloud Platform or Splunk Enterprise. No changes to HEC configuration required. The Log10x App, which expands compact events, supports Splunk Enterprise 9.4 through 10.4.
 
 ??? tenx-compat "Works with Splunk Enterprise on-premises"
 
-    **Yes.** Works with both Splunk Cloud and Splunk Enterprise on-premises (8.x, 9.x, 10.x), across single-instance, distributed, and clustered deployments.
+    **Yes.** The Receiver, Reporter and Retriever work with Splunk Enterprise on-premises across single-instance, distributed, and clustered deployments. The Log10x App supports Splunk Enterprise 9.4 through 10.4.
 
     On-premises, the Receiver sidecar and Reporter DaemonSet forward to HEC or to Heavy Forwarders, using either HEC token or Heavy Forwarder S2S authentication.
 
@@ -85,234 +86,10 @@ Splunk dashboards, Universal Forwarders, testing on your own environment, HEC in
 
     **[Retriever](https://doc.log10x.com/apps/retriever/){target="_blank"} for on-premises:** Works with AWS S3, Azure Blobs, and any S3-compatible [object storage](https://doc.log10x.com/run/input/objectStorage/){target="_blank"}. Returns the exact offloaded events to Splunk Enterprise HEC endpoints on-demand.
 
-??? tenx-compat "Splunk Cloud: KV Store Setup & Pilot Checklist"
-
-    The [10x for Splunk](https://github.com/log-10x/splunk-app){target="_blank"} app expands compact events at search time using a KV Store collection. Here's the complete setup and pilot validation checklist for Splunk Cloud.
-
-    **Before You Start**
-
-    - Admin or Power User access to your Splunk Cloud instance
-    - Splunk Cloud supporting KV Store (all modern instances do)
-    - Ability to create HTTP Event Collector (HEC) tokens
-    - Two HEC tokens configured: one for templates (`tenx_dml_raw_json` sourcetype), one for encoded events
-
-    **Day 1: App Installation & KV Store Setup**
-
-    1. **Install 10x for Splunk app**
-       - [ ] Download from [GitHub](https://github.com/log-10x/splunk-app/releases){target="_blank"}
-       - [ ] Upload via Settings > Apps > Install app from file
-       - [ ] Restart (if prompted by Splunk Cloud)
-       - [ ] Verify: Settings > Apps > Confirm "10x for Splunk" appears in app list
-
-    2. **Create KV Store collection**
-       - [ ] Go to Settings > Advanced Search > Collections
-       - [ ] Create new collection named `tenx_dml`
-       - [ ] Schema fields (automatically generated, verify all present):
-         ```
-         _key (primary key)
-         pattern_hash (string)
-         pattern (string)
-         pattern_parts (array)
-         part_0 (string)
-         pattern_terminator (string)
-         timestamp_format (string)
-         ```
-       - [ ] Verify: `| inputlookup tenx-dml-lookup | stats count` returns 0 (empty)
-
-    3. **Create indexes for template data**
-       - [ ] **Create `tenx_dml` index** (required for storing templates):
-         ```
-         Settings → Indexes → New Index
-         Name: tenx_dml
-         Data type: Events
-         Max size: 10GB (adjust based on your expected template volume)
-         Retention: 30+ days (templates are reference data, not logs)
-         ```
-         Verify: `| rest /services/data/indexes | search title="tenx_dml"`
-       - [ ] **Optional: Create separate index for encoded events:**
-         ```
-         Settings → Indexes → New Index
-         Name: encoded_events (or your preferred name)
-         Data type: Events
-         Max size: Depends on log volume
-         ```
-         Or use `main` index if preferred (encoded events are searchable until expansion)
-
-    4. **Verify props.conf and transforms.conf**
-       - [ ] Check Settings > Field Extractions > Verify `tenx_encoded` sourcetype has `REPORT-tenx` extraction
-       - [ ] Verify transforms.conf has `tenx-hash-vars-extraction` and `tenx-dml-lookup` defined
-       - [ ] If missing, manually add via Settings > Add data > Source type settings
-
-    **Phase 4: Enable HTTP Event Collector (HEC) globally**
-
-    4a. **Enable HEC globally** (required before creating tokens):
-       - [ ] Settings → Data Inputs → HTTP Event Collector → Global Settings
-       - [ ] Toggle "All Tokens" to **ENABLED**
-       - [ ] Set Default Input Port: `8088` (or your custom port)
-       - [ ] Enable SSL: **YES** (recommended for production)
-       - [ ] Click Save
-       - [ ] Verify: `| rest /services/data/inputs/http | search disabled=0`
-
-    4b. **Create HEC Token 1 (for templates)**:
-       - [ ] Settings → Data Inputs → HTTP Event Collector → New Token
-       - [ ] Name: `tenx-templates`
-       - [ ] Source Type: `tenx_dml_raw_json`
-       - [ ] Index: `tenx_dml` (created in step 3)
-       - [ ] Indexes allowed: `tenx_dml` (restrict to this index only)
-       - [ ] Disabled: NO
-       - [ ] Click **Save Token**
-       - [ ] Copy the token value (save it for later)
-
-    4c. **Create HEC Token 2 (for encoded events)**:
-       - [ ] Settings → Data Inputs → HTTP Event Collector → New Token
-       - [ ] Name: `tenx-encoded`
-       - [ ] Source Type: `tenx_encoded`
-       - [ ] Index: Your target index (where searchable events go)
-       - [ ] Indexes allowed: Your target index
-       - [ ] Disabled: NO
-       - [ ] Click **Save Token**
-       - [ ] Copy the token value (save it for later)
-
-    **Week 1: Data Ingestion & KV Store Population**
-
-    5. **Send template data via HEC** (or via your log forwarder)
-       - [ ] **Via curl** (for testing):
-         ```bash
-         SPLUNK_HOST="your-splunk-cloud.splunkcloud.com"
-         SPLUNK_PORT="8088"
-         HEC_TOKEN="<your-tenx-templates-token>"
-
-         curl -k https://$SPLUNK_HOST:$SPLUNK_PORT/services/collector/event \
-           -H "Authorization: Splunk $HEC_TOKEN" \
-           -H "Content-Type: application/json" \
-           -d '{
-             "event": {
-               "templateHash": "abc123def456",
-               "template": "User %s logged in from %s",
-               "templateParts": ["User", "logged in from"]
-             },
-             "sourcetype": "tenx_dml_raw_json",
-             "index": "tenx_dml"
-           }'
-         ```
-       - [ ] **Via Fluentd/Fluent Bit:** Configure your forwarder output to send to this HEC endpoint with same token and sourcetype
-       - [ ] Verify templates arriving: `index=tenx_dml sourcetype=tenx_dml_raw_json | head 10`
-
-    6. **Send encoded events via HEC** (or via your log forwarder)
-       - [ ] **Via curl** (for testing):
-         ```bash
-         SPLUNK_HOST="your-splunk-cloud.splunkcloud.com"
-         SPLUNK_PORT="8088"
-         HEC_TOKEN="<your-tenx-encoded-token>"
-
-         curl -k https://$SPLUNK_HOST:$SPLUNK_PORT/services/collector/event \
-           -H "Authorization: Splunk $HEC_TOKEN" \
-           -H "Content-Type: application/json" \
-           -d '{
-             "event": "~abc123def456,admin,192.168.1.1",
-             "sourcetype": "tenx_encoded",
-             "index": "main"
-           }'
-         ```
-       - [ ] **Via Fluentd/Fluent Bit:** Configure your forwarder output to send to this HEC endpoint with same token and sourcetype
-       - [ ] Verify encoded events arriving: `index=main sourcetype=tenx_encoded | head 10`
-
-    7. **Wait for KV Store population**
-       - [ ] The "Consume KV" saved search runs every 2 minutes (automatic)
-       - [ ] Check: `index=_internal savedsearch_name="Consume KV" | table _time, status, result_count` (verify no errors)
-       - [ ] Check KV store: `| inputlookup tenx-dml-lookup | stats count` (should be > 0 after 2-3 min)
-
-    8. **Monitor template consumption**
-       - [ ] Run: `sourcetype=tenx_dml_pure | stats count` (confirms templates are searchable; this sourcetype lands in the index set by `dest_dml_index`, `main` by default)
-       - [ ] Run: `| inputlookup tenx-dml-lookup | head 5 | table _key, pattern, timestamp_format` (verify structure)
-
-    **Week 2: Expansion Validation & Performance Testing**
-
-    9. **Test basic expansion**
-       - [ ] Run: `index=your_target_index sourcetype=tenx_encoded | head 10 | \`tenx-inflate\``
-       - [ ] Verify: All fields (`_raw`, `_time`, `host`, `source`, etc.) are restored to original values
-       - [ ] Check: No `tenx_hash`, `tenx_var_*` fields remain in final output (cleanup working)
-
-    10. **Test debug mode**
-       - [ ] Run: `index=your_target_index sourcetype=tenx_encoded | head 1 | \`tenx-inflate-debug\` | table *`
-       - [ ] Verify: `_raw` field matches original (unencoded) log format
-       - [ ] Check: `tenx_ts_sec` correctly detects timestamp precision (milliseconds vs nanoseconds)
-
-    11. **Test field extractions and searches post-expansion**
-       - [ ] Run existing saved search/dashboard on expanded data
-       - [ ] Verify: All field extractions work (extractions applied post-expansion)
-       - [ ] Check: Alerts trigger correctly on expanded events
-       - [ ] Compare: Results match pre-optimization historical logs (sample query on same time range)
-
-    12. **Measure search performance**
-       - [ ] Run: `index=your_target_index sourcetype=tenx_encoded earliest=-1h | \`tenx-inflate\` | stats count`
-       - [ ] Note: The `tenx-inflate` macro adds a per-event decode step at search time; expect a small overhead relative to searching un-encoded events
-       - [ ] Compare: Same query without expansion vs with expansion, and validate the overhead on your own data volume
-
-    **Post-Pilot: Production Deployment**
-
-    13. **Enable analytics dashboard**
-       - [ ] Open: App launcher > 10x for Splunk > Analytics Dashboard
-       - [ ] Verify: Shows total compact events, reduction ratio, storage savings
-       - [ ] Check: Panels populate when opened (the dashboard runs its searches on view)
-
-    14. **Set up monitoring and alerts**
-       - [ ] Monitor KV store size: `| inputlookup tenx-dml-lookup | stats count` (alert on growth or error against the threshold you set)
-       - [ ] Monitor expansion failures: Check tenx app logs for errors
-       - [ ] Optional: Set up dashboard for Splunk license impact (GB before/after)
-
-    15. **Risk Mitigation & Rollback**
-       - [ ] **Rollback procedure:** Simply disable 10x for Splunk app:
-          1. Settings > Apps > 10x for Splunk > Disable
-          2. Re-run searches without `\`tenx-inflate\`` macro (searches work on encoded raw data until disabled)
-          3. KV Store collection remains; can re-enable app without data loss
-       - [ ] **Zero data loss:** Encoded events remain in index; templates preserved in KV Store
-       - [ ] **Retention:** Configure KV Store collection retention if needed (Settings > Collections)
-
-    **Splunk Cloud Limitations & Workarounds**
-
-    - **No custom Python alert actions**, Covered. App uses standard KV Store and Search hooks (no custom Python required)
-    - **Limited app customization**, App config available in local/default folders; can override via local/ without modifying default/
-    - **Network egress**, All data stays within Splunk Cloud. No external calls needed after app installation
-    - **KV Store max size**, Bounded by your Splunk KV Store limits, one entry per template. Monitor via `| inputlookup tenx-dml-lookup | stats count`. If approaching your limit, consider archiving old templates
-
-    **Forwarder Configuration Examples**
-
-    **Fluentd:**
-    ```yaml
-    <match encoded_events>
-      @type http_buffered
-      endpoint_url https://<splunk-host>:8088/services/collector/event
-      serializer json
-      auth_type basic
-      auth_key "Splunk <your-hec-token>"
-      <buffer>
-        flush_interval 10s
-      </buffer>
-    </match>
-    ```
-
-    **Fluent Bit:**
-    ```ini
-    [OUTPUT]
-    Name http
-    Match *
-    Host <splunk-host>
-    Port 8088
-    URI /services/collector/event
-    header Authorization Splunk <your-hec-token>
-    header Content-Type application/json
-    json_date_key timestamp
-    Format json
-    ```
+??? tenx-compat "Does the Log10x App run on Splunk Cloud Platform"
 
-    **Universal Forwarder:**
-    The Universal Forwarder integration uses a file relay pattern, not a custom forwarder output. Fluent Bit with the 10x sidecar reads logs from the original folder, filters and (optionally) compacts them, and writes the processed output to a second folder. The Universal Forwarder monitors that second folder with a standard `inputs.conf` and ships to the indexers over its normal S2S protocol. The UF keeps its existing configuration; no custom outputs are required.
+    The [Log10x App](https://doc.log10x.com/apps/receiver/compact/splunk/) is tested on Splunk Enterprise 9.4, 10.0, 10.2 and 10.4, and the package passes AppInspect's Splunk Cloud checks. On Splunk Cloud Platform, an uploaded app goes through Splunk's own vetting before it installs.
 
-    **Support & Troubleshooting**
+    Setup is the same on both: install the app, create the `tenx_dml` index and two HEC tokens, point the Receiver at them, set the `tenx-events` macro, and run the backfill search once if templates were indexed before the app. The app creates its KV Store collection itself. See the [Quickstart](https://doc.log10x.com/apps/receiver/compact/splunk/#quickstart).
 
-    - **Templates not in KV Store:** Check saved search logs: `index=_internal savedsearch_name="Consume KV"`
-    - **Expansion returns empty:** Verify template format in `tenx_dml_raw_json` index, ensure KV Store has matching hash
-    - **Performance degradation:** Limit time ranges in searches; filter by `tenx_hash` before expansion for large datasets
-    - **HEC token rejected:** Verify token is enabled and not deleted: `| rest /services/data/inputs/http`
-    - **For detailed troubleshooting:** See [10x for Splunk Troubleshooting Guide](https://github.com/log-10x/splunk-app#troubleshooting){target="_blank"} on GitHub
+    The app runs Python on the search head: the `tenxsearch` command, the `/tenx-search` and `/tenx-alert` endpoints, and the alert action that stores templates. It makes no outbound calls.

package/build/product-kb/docs/faq/stacks/splunk/index.md

@@ -19,7 +19,7 @@ Cut [Splunk](https://www.log10x.com/splunk.html){target="_blank"} costs by a mod
 
 - :material-currency-usd: **[Optimization](optimization.md)**
 
-    How the 10x for Splunk app expands compact events, search-time overhead, license-tier reduction, and what happens to filtered events.
+    How the Log10x App expands compact events, search-time overhead, license-tier reduction, and what happens to filtered events.
 
 - :material-scale-balance: **[Comparisons](comparisons.md)**
 

package/build/product-kb/docs/faq/stacks/splunk/kv-store.md

@@ -3,196 +3,78 @@ icon: material/database-check
 title: "KV Store Validation & Diagnostics"
 ---
 
-Validating the KV Store, diagnosing "Consume KV" silent failures, monitoring capacity, recovering from template/event ordering issues, and distributed-cluster setup.
+Validating the KV Store, diagnosing "Consume KV", loading templates indexed earlier, sizing, and search head clusters.
 
 ??? tenx-kvstore "How do I validate that KV Store is working correctly"
 
-    **Quick Health Check (run all three):**
+    Run each search in the Log10x App.
 
-    1. **Verify KV collection exists:**
+    1. **Templates are arriving:**
+       ```spl
+       index=tenx_dml sourcetype=tenx_dml_raw_json | stats count
        ```
-       | rest /servicesNS/nobody/tenx-for-splunk/storage/collections/config
-       | search title="tenx_dml"
-       ```
-       Expected: Returns 1 result. If 0 results, collection wasn't created.
+       Expected: more than 0. If 0, check the templates HEC token points at `tenx_dml`.
 
-    2. **Check KV store population:**
-       ```
+    2. **Templates are stored:**
+       ```spl
        | inputlookup tenx-dml-lookup | stats count
        ```
-       Expected: Shows N (number of templates). If 0, no templates loaded yet.
+       Expected: close to the number of distinct templates sent.
 
-    3. **Verify "Consume KV" scheduled search is running:**
+    3. **"Consume KV" is running:**
+       ```spl
+       index=_internal sourcetype=scheduler savedsearch_name="Consume KV"
+       | stats latest(status) as status, latest(_time) as last_run
+       | eval last_run=strftime(last_run, "%F %T")
        ```
-       | index=_internal savedsearch_name="Consume KV"
-       | stats latest(status) as status, latest(_time) as last_run by savedsearch_name
-       ```
-       Expected: `status=success`, `last_run` within last 2 minutes.
-
-    **If any check fails, see troubleshooting below.**
+       Expected: `status=success`, `last_run` within the last 5 minutes.
 
-??? tenx-kvstore "\"Consume KV\" scheduled search is failing silently"
+    The **Diagnostics** dashboard runs these checks and more on one page.
 
-    The "Consume KV" search populates templates from `tenx_dml` index into the KV Store. If it fails, templates won't be available for expansion.
+??? tenx-kvstore "\"Consume KV\" is not storing templates"
 
-    **Diagnostic procedure:**
-
-    ```
-    Step 1: Check scheduler logs
-    | index=_internal sourcetype=scheduler savedsearch_name="Consume KV"
-    | table _time, status, result_count, alert_action
-    | stats latest(*) as * by status
-    ```
+    "Consume KV" runs every 5 minutes over the last 7 minutes of `tenx_dml_raw_json` and hands the results to the `tenx_dml_to_kv` alert action, which writes the KV Store.
 
-    **Common failure modes:**
+    | Symptom | Cause | Fix |
+    |---------|-------|-----|
+    | No scheduler entries | The search is disabled | Enable it in **Settings** → **Searches, reports, and alerts** |
+    | `success`, KV count stays 0 | No templates in the last 7 minutes | Check the templates token, then run the backfill search below |
+    | `success`, KV count stays 0, templates present | The alert action failed | Read `$SPLUNK_HOME/var/log/splunk/tenx_dml_to_kv.log` |
 
-    | Status | Cause | Fix |
-    |--------|-------|-----|
-    | `error` | Search syntax error in saved search | Edit saved search "Consume KV" and verify query syntax |
-    | `success` / count=0 | No templates in `tenx_dml` index | Run: `\| index=tenx_dml \| stats count`, if 0, send templates via HEC |
-    | `failure` | Alert action (tenx_dml_to_kv.py) failed | Check: `\| index=_internal sourcetype=action_handler savedsearch_name="Consume KV"` |
-    | No results | Search never ran | Verify: Scheduler is enabled (Settings > Scheduled Searches) |
-
-    **Recovery steps:**
-
-    ```
-    1. Verify templates exist:
-       | index=tenx_dml sourcetype=tenx_dml_raw_json | stats count
-
-    2. Force immediate execution:
-       Click saved search "Consume KV" > Run
-       (Or use: | savedsearch "Consume KV")
-
-    3. Wait 2 minutes and verify population:
-       | inputlookup tenx-dml-lookup | stats count
-       (Should show > 0)
+    The alert action's log is also searchable:
 
-    4. If still 0, check KV collection exists:
-       | rest /servicesNS/nobody/tenx-for-splunk/storage/collections/config
+    ```spl
+    index=_internal source=*tenx_dml_to_kv.log* (ERROR OR WARNING)
     ```
 
-??? tenx-kvstore "How do I monitor KV Store size and capacity"
+??? tenx-kvstore "Templates were indexed before the app was installed"
 
-    KV Store size affects search performance. Monitor it proactively:
+    "Consume KV" reads only the last 7 minutes, so templates indexed earlier, or while it was disabled or failing, are never stored and their events stay compact.
 
-    **Monthly capacity check:**
+    Run this search once, as an admin or power user:
 
-    ```
-    | inputlookup tenx-dml-lookup
-    | stats count as num_templates, max(timestamp_format) as latest_update
-    ```
-
-    **Growth and partitioning guidance:**
-
-    The KV Store holds one row per template, so size grows with the number of
-    templates rather than event volume. Track `num_templates` over time
-    and watch expansion latency (query below). If lookup latency climbs well above
-    your normal baseline as the template count grows, consider archiving old
-    templates or partitioning across collections.
-
-    **If the template count keeps growing:**
-
-    Archive old templates by exporting them out of the collection and removing
-    them from `tenx_dml`:
-    ```
-    | inputlookup tenx-dml-lookup
-    | search timestamp_format < "2024-01-01"
-    | ... (export to archive)
-    ```
-
-    The `tenx-inflate` macro reads a single collection (`tenx_dml`), so keep all
-    active templates in that one collection rather than splitting them.
-
-    **Monitor expansion latency:**
-
-    ```
-    index=<your-index> sourcetype=tenx_encoded
-    | `tenx-inflate`
-    | stats avg(eval(round(relative_time(now(), "now") - _time, 3))) as inflate_latency_sec
+    ```spl
+    index=tenx_dml sourcetype=tenx_dml_raw_json earliest=-30d | sendalert tenx_dml_to_kv
     ```
 
-    Establish a baseline for `inflate_latency_sec` when the collection is small,
-    then watch for it rising well above that baseline as the template count grows.
+    Widen `earliest` to reach older templates. Templates already stored are skipped, so running it again is safe. The app's **Backfill KV** saved search holds the same search, but the **Run** button does not fire its alert action. Encoded events need no re-indexing: they expand as soon as their template is in the KV Store.
 
-??? tenx-kvstore "What if I accidentally send encoded events before templates are loaded"
+??? tenx-kvstore "How large does the KV Store get"
 
-    If encoded events arrive before templates, expansion will fail silently until templates load.
+    One row per template, not per event, so size follows the number of distinct templates. The **Analytics** dashboard shows the count as **Active Templates**; the same number from SPL:
 
-    **Prevention:**
-
-    Always verify template population BEFORE sending encoded events:
-
-    ```
-    # Wait for this to return > 0:
+    ```spl
     | inputlookup tenx-dml-lookup | stats count
     ```
 
-    **Recovery (if already happened):**
-
-    1. Load the missing templates
-       - Re-send template data via HEC (same format as before)
-       - Wait ~2 minutes for "Consume KV" to process (it runs every 2 minutes)
-
-    2. Re-index the encoded events (optional)
-       ```
-       # If using Kubernetes:
-       kubectl delete pod <forwarder-pod-name>  # Triggers reprocessing
-
-       # If using file-based forwarder:
-       # Delete offset tracking file, restart forwarder
-       ```
-
-    3. Verify recovery:
-       ```
-       | index=<your-index> sourcetype=tenx_encoded
-       | head 10 | `tenx-inflate`
-       # Should now return expanded events
-       ```
-
-??? tenx-kvstore "Distributed KV Store setup for multi-node Splunk clusters"
-
-    For production Splunk clusters, KV Store can be:
-    - **Replicated** (HA across nodes)
-    - **Partitioned** (scaled across multiple collections)
+    The `tenx-inflate` macro reads the single `tenx_dml` collection; keep every active template there.
 
-    **For a multi-node Splunk cluster:**
+??? tenx-kvstore "Search head clusters"
 
-    KV Store replication across search heads follows your cluster's KV Store
-    configuration, so confirm the `tenx_dml` collection is present and consistent
-    on each node:
+    The `tenx_dml` collection is defined in the app's `default/collections.conf` and replicates with the rest of the cluster's KV Store. Deploy the app to every search head with the deployer. Each member reports the same count:
 
-    ```
-    # On each node:
-    | rest /servicesNS/nobody/tenx-for-splunk/storage/collections/config
-    | search title="tenx_dml"
-    | table label, acl{}.perms
-    ```
-
-    Each node should return the same collection.
-
-    **Collection schema:**
-
-    The `tenx_dml` collection is defined in the app's `default/collections.conf`.
-    The `tenx-inflate` macro reads these fields at search time:
-
-    ```
-    [tenx_dml]
-    field.pattern_hash = string
-    field.pattern = string
-    field.pattern_parts = array
-    field.part_0 = string
-    field.pattern_terminator = string
-    field.timestamp_format = string
-    ```
-
-    **Monitoring cluster KV Store health:**
-
-    ```
-    | rest /servicesNS/nobody/tenx-for-splunk/storage/collections/data/tenx_dml
-    | stats count as templates_primary
-    | append
-      [| rest /servicesNS/nobody/tenx-for-splunk/storage/collections/data/tenx_dml
-       | stats count as templates_replica]
+    ```spl
+    | inputlookup tenx-dml-lookup | stats count
     ```
 
-    Both should be equal (healthy replication).
+    Only admins and power users can write the collection. The scheduled "Consume KV" search runs on one member, and its writes replicate to the others.

package/build/product-kb/docs/faq/stacks/splunk/optimization.md

@@ -3,11 +3,11 @@ icon: material/currency-usd
 title: "Optimization"
 ---
 
-How the 10x for Splunk app expands compact events at search time, search-time overhead, potential license-tier reduction, and what happens to events the Receiver filters out.
+How the Log10x App expands compact events at search time, search-time overhead, potential license-tier reduction, and what happens to events the Receiver filters out.
 
-??? tenx-costopt "How does the 10x for Splunk app expand optimized events"
+??? tenx-costopt "How does the Log10x App expand optimized events"
 
-    On Splunk and self-hosted Elasticsearch or OpenSearch, the Receiver compacts events losslessly before they hit Splunk, so the saving lands on your license bill. The open-source [10x for Splunk](https://doc.log10x.com/apps/receiver/compact/splunk/){target="_blank"} app keeps that saving invisible to searchers: it automatically expands compact events back to their full original form before displaying results.
+    On Splunk and self-hosted Elasticsearch or OpenSearch, the Receiver compacts events losslessly before they hit Splunk, so the saving lands on your license bill. The open-source [Log10x App](https://doc.log10x.com/apps/receiver/compact/splunk/){target="_blank"} keeps that saving invisible to searchers: it automatically expands compact events back to their full original form before displaying results.
 
     **How it works:**
 
@@ -30,7 +30,7 @@ How the 10x for Splunk app expands compact events at search time, search-time ov
     - Top templates by usage
     - Expansion success rate
 
-    **User experience:** Classic dashboards keep their panel SPL unchanged. From the search bar a query is wrapped in the app's `tenxsearch` command, since Splunk's search page loads no app JavaScript. Scheduled alerts are compiled once at save time into a native saved search. All three read the original full-fidelity data.
+    **User experience:** Classic dashboards keep their panel SPL unchanged. From the search bar a query is wrapped in the app's `tenxsearch` command, since Splunk's search page loads no app JavaScript. Alerts created in the app's Compile Alert view are compiled once into a native saved search. All three read the original full-fidelity data.
 
     **Open source:** Available on [GitHub](https://github.com/log-10x/splunk-app){target="_blank"}.
 

package/build/product-kb/docs/run/input/forwarder/fluentbit/index.md

@@ -95,7 +95,7 @@ tenx @run/input/forwarder/fluentbit @apps/receiver
     # ... destination config
 ```
 
-For Splunk integration see the [10x for Splunk](https://doc.log10x.com/apps/receiver/compact/splunk/) documentation. For Kubernetes deployment, add the `log10x/edge-10x` sidecar on top of the official Fluent Bit chart via an `extraContainers` values overlay, see the [Helm chart overlay](https://doc.log10x.com/apps/receiver/deploy/#fluent-bit).
+For Splunk integration see the [Log10x App](https://doc.log10x.com/apps/receiver/compact/splunk/) documentation. For Kubernetes deployment, add the `log10x/edge-10x` sidecar on top of the official Fluent Bit chart via an `extraContainers` values overlay, see the [Helm chart overlay](https://doc.log10x.com/apps/receiver/deploy/#fluent-bit).
 
 ## :material-wrench-outline: Config Files
 

package/build/product-kb/docs/run/input/forwarder/fluentd/index.md

@@ -101,7 +101,7 @@ tenx @run/input/forwarder/fluentd @apps/receiver
 </label>
 ```
 
-For Splunk integration see the [10x for Splunk](https://doc.log10x.com/apps/receiver/compact/splunk/) documentation. For Kubernetes deployment, add the `log10x/edge-10x` sidecar on top of the official Fluentd chart via a kustomize post-renderer overlay, see the [Helm chart overlay](https://doc.log10x.com/apps/receiver/deploy/#fluentd).
+For Splunk integration see the [Log10x App](https://doc.log10x.com/apps/receiver/compact/splunk/) documentation. For Kubernetes deployment, add the `log10x/edge-10x` sidecar on top of the official Fluentd chart via a kustomize post-renderer overlay, see the [Helm chart overlay](https://doc.log10x.com/apps/receiver/deploy/#fluentd).
 
 ## :material-wrench-outline: Config Files
 

package/build/product-kb/docs/run/input/forwarder/otel-collector/index.md

@@ -139,7 +139,7 @@ service:
       exporters: [debug]
 ```
 
-For Splunk integration see the [10x for Splunk](https://doc.log10x.com/apps/receiver/compact/splunk/) documentation. For Kubernetes deployment via the official OpenTelemetry Collector Helm chart see the [Helm sidecar overlay](https://doc.log10x.com/apps/receiver/deploy/#otel-collector).
+For Splunk integration see the [Log10x App](https://doc.log10x.com/apps/receiver/compact/splunk/) documentation. For Kubernetes deployment via the official OpenTelemetry Collector Helm chart see the [Helm sidecar overlay](https://doc.log10x.com/apps/receiver/deploy/#otel-collector).
 
 ## :material-wrench-outline: Config Files
 

package/build/product-kb/docs/run/input/forwarder/vector/index.md

@@ -116,7 +116,7 @@ sinks:
     encoding: { codec: json }
 ```
 
-For Splunk integration see the [10x for Splunk](https://doc.log10x.com/apps/receiver/compact/splunk/) documentation. For Kubernetes deployment via the official Vector Helm chart see the [Helm sidecar overlay](https://doc.log10x.com/apps/receiver/deploy/#vector).
+For Splunk integration see the [Log10x App](https://doc.log10x.com/apps/receiver/compact/splunk/) documentation. For Kubernetes deployment via the official Vector Helm chart see the [Helm sidecar overlay](https://doc.log10x.com/apps/receiver/deploy/#vector).
 
 ## :material-wrench-outline: Config Files