log10x-mcp 1.30.39 → 1.30.40

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. package/README.md +1 -1
  2. package/build/product-kb/docs/api/js.md +4 -4
  3. package/build/product-kb/docs/apps/mcp/run.md +5 -4
  4. package/build/product-kb/docs/apps/mcp/tools/compile/compile-link.md +2 -2
  5. package/build/product-kb/docs/apps/mcp/tools/compile/compile-status.md +2 -2
  6. package/build/product-kb/docs/apps/mcp/tools/compile/compile.md +6 -6
  7. package/build/product-kb/docs/apps/mcp/tools/compile/index.md +1 -1
  8. package/build/product-kb/docs/apps/receiver/deploy.md +1 -1
  9. package/build/product-kb/docs/apps/receiver/faq.md +16 -17
  10. package/build/product-kb/docs/apps/receiver/index.md +2 -2
  11. package/build/product-kb/docs/apps/receiver/run.md +5 -4
  12. package/build/product-kb/docs/apps/reporter/run.md +5 -4
  13. package/build/product-kb/docs/apps/retriever/run.md +5 -4
  14. package/build/product-kb/docs/compile/faq.md +13 -13
  15. package/build/product-kb/docs/compile/index.md +1 -1
  16. package/build/product-kb/docs/compile/link/index.md +2 -1
  17. package/build/product-kb/docs/compile/pull/index.md +3 -1
  18. package/build/product-kb/docs/compile/scan/index.md +5 -5
  19. package/build/product-kb/docs/compile/scanner/logMethods/index.md +1 -1
  20. package/build/product-kb/docs/compile/scanner/text/index.md +1 -1
  21. package/build/product-kb/docs/engine/launcher/extension.md +24 -25
  22. package/build/product-kb/docs/faq/apps/index.md +1 -1
  23. package/build/product-kb/docs/faq/apps/receiver.md +16 -17
  24. package/build/product-kb/docs/faq/general.md +5 -5
  25. package/build/product-kb/docs/faq/pricing/node-counting.md +1 -1
  26. package/build/product-kb/docs/faq/stacks/aws-cloudwatch.md +4 -4
  27. package/build/product-kb/docs/faq/stacks/azure-monitor.md +2 -2
  28. package/build/product-kb/docs/faq/stacks/datadog/comparisons.md +3 -3
  29. package/build/product-kb/docs/faq/stacks/datadog/integration.md +1 -1
  30. package/build/product-kb/docs/faq/stacks/elasticsearch/optimization.md +1 -1
  31. package/build/product-kb/docs/faq/stacks/splunk/index.md +1 -1
  32. package/build/product-kb/docs/run/initialize/message/index.md +66 -31
  33. package/build/product-kb/docs/run/receive/rate/index.md +2 -2
  34. package/build/product-kb/docs/run/transform/structure/index.md +2 -2
  35. package/build/product-kb/docs/run/transform/symbol/index.md +3 -3
  36. package/build/tools/configure-engine.d.ts +3 -2
  37. package/build/tools/configure-engine.js +6 -5
  38. package/build/tools/configure-engine.js.map +1 -1
  39. package/package.json +1 -1
package/README.md CHANGED
@@ -20,7 +20,7 @@ This server is open source under MIT. A Log10x product license (API key) unlocks
20
20
  | "This message type is noise, cut it." | Sample, drop, compact, tier down, or offload a chosen message type, with examples and trend so the action is clear. `log10x_pattern_mitigate`, `log10x_pattern_examples`, `log10x_pattern_trend` |
21
21
  | "We are 100% Lambda, no cluster. Install there." | The install plan comes back for the serverless estate: the engine as a Lambda extension paired with the function's OTel collector, a CDK construct, and an AWS-native recurring tick over an S3 config plane. `log10x_advise_install`, `log10x_setup_recurring` |
22
22
  | "Get me the actual events I offloaded." | Fetch the exact events back on demand from your own S3. `log10x_retriever_query`, `log10x_offload_add`, `log10x_retriever_register` |
23
- | "Recognize more of my message types." | Build a custom symbol library from your own logs so the engine finds more savings. `log10x_compile`, `log10x_compile_link`, `log10x_compile_status` |
23
+ | "Recognize more of my message types." | Compile your own source code, binaries and container images into a symbol library so the engine names more of your message types. `log10x_compile`, `log10x_compile_link`, `log10x_compile_status` |
24
24
 
25
25
  ## Install
26
26
 
@@ -1270,7 +1270,7 @@ this.status = this.symbol("ENUM", "Status"); //capture the 'ENABLED value above
1270
1270
  ### .symbolSequence(symbolContexts,fieldName,maxLen) ⇒ <code>string</code>
1271
1271
  Returns a sequence of [symbol](https://doc.log10x.com/run/transform/structure/#symbols) tokens of a specified type.
1272
1272
 
1273
- This function selects a the longest set of symbol and delimiter tokens from the object's
1273
+ This function selects the symbol and delimiter tokens of the origin ranked highest by the [message](https://doc.log10x.com/run/initialize/message/) comparator from the object's
1274
1274
  [TenXTemplate](https://doc.log10x.com/run/template/) based on their context within the source code/binary file from which they originated.
1275
1275
 
1276
1276
  To capture a symbol sequence for the following event:
@@ -1286,7 +1286,7 @@ this.message = symbolSequence("log,class,exec"); // = 'Partition__not_found_comp
1286
1286
  ```
1287
1287
 
1288
1288
  The `symbolContext` argument controls the [symbol contexts](https://doc.log10x.com/run/transform/symbol/#contexts) to search for;
1289
- if the first type does not yield a result, the next one is tried etc.
1289
+ the listed contexts are evaluated in a single pass, so list order acts as a filter rather than a precedence chain.
1290
1290
 
1291
1291
  Supported values: package, class, method, log, enum, const, text, exec, any (case insensitive).
1292
1292
 
@@ -1342,7 +1342,7 @@ and [time-series](https://doc.log10x.com/run/output/metric/) output.
1342
1342
  ### .symbolOrigin(symbolContext) ⇒ <code>string</code>
1343
1343
  Returns a the source code/binary origin of 'symbol' tokens of a specified type.
1344
1344
 
1345
- This function selects a the origin (i.e. the source code or binary executable which emitted) the longest set of [symbol](https://doc.log10x.com/run/transform/structure/#symbols) and delimiter tokens from the object's
1345
+ This function selects the origin (i.e. the source code or binary executable which emitted) the highest-ranked set of [symbol](https://doc.log10x.com/run/transform/structure/#symbols) and delimiter tokens from the object's
1346
1346
  TenXTemplate.
1347
1347
 
1348
1348
  To capture the origin value for the following event:
@@ -1356,7 +1356,7 @@ The following call can be used:
1356
1356
  this.origin = symbolOrigin("log,class,exec"); // CacheManager.scala'
1357
1357
  ```
1358
1358
 
1359
- The 'symbolContext' argument controls the [symbol contexts](https://doc.log10x.com/run/transform/symbol/#contexts) to search for; if the first type does not yield a result, the next one is tried etc.
1359
+ The 'symbolContext' argument controls the [symbol contexts](https://doc.log10x.com/run/transform/symbol/#contexts) to search for; the listed contexts are evaluated in a single pass, so list order acts as a filter rather than a precedence chain.
1360
1360
 
1361
1361
  Supported values: PACKAGE, CLASS, METHOD, LOG, ENUM, CONST, TEXT, EXEC (case insensitive).
1362
1362
 
@@ -587,8 +587,8 @@ To configure the MCP test-drive app, [:material-cog: Edit](https://doc.log10x.co
587
587
  contexts: log,exec
588
588
 
589
589
  # 'messageField' specifies the field name to assign the message pattern to target TenXObjects: the symbol
590
- # tokens of the origin ranked highest by coverage, meaning the count of distinct library-known tokens it
591
- # explains. Span length is the last tiebreak. The origin file name itself goes to 'originField'.
590
+ # tokens of the origin ranked highest on the five-key comparator, message-text width first. Span length
591
+ # is the last tiebreak. The origin file name itself goes to 'originField'.
592
592
  messageField: message_pattern
593
593
 
594
594
  # 'messageHashField' specifies the field name to assign a stable, URL-safe pattern hash (xxHash64 of the messageField value, base64url, 11 chars). Safe as a SIEM query term or forwarder filter key.
@@ -610,8 +610,9 @@ To configure the MCP test-drive app, [:material-cog: Edit](https://doc.log10x.co
610
610
  # indented detail, banners) so the message-template skips its own message-pattern compute for them.
611
611
  # The check fires only for templates that don't already pass the (groupSize > 1) || groupHead arms;
612
612
  # i.e., for single-event templates with no head signal. A non-negator line in that case (e.g., bare
613
- # "Hello World", "Heartbeat received from node-7") still gets a pattern; a negator-matching orphan
614
- # (e.g., a lone "\tat com.foo.Bar(...)") does not.
613
+ # "Hello World", "Heartbeat received from node-7") gets a pattern from its library words; a negator-matching
614
+ # orphan (e.g., a lone "\tat com.foo.Bar(...)") skips origin selection and is named by a hash of its shape,
615
+ # template_<hash>.
615
616
  message:
616
617
  negators:
617
618
  - " " # any indented line (stack frames, indented app detail, multi-line JSON, banners)
@@ -36,7 +36,7 @@ A very large units tree returns a running `job_id` to poll with [`log10x_compile
36
36
 
37
37
  ## :material-check-decagram-outline: Prerequisites
38
38
 
39
- The Compiler flavor, docker-first: `auto` and `docker` pull `log10x/compiler-10x:1.1.125` (pinned by digest), and `local` runs a Compiler-flavor `tenx`. The Runtime flavor is refused; the `mode` argument of [`log10x_compile`](compile.md) lists the three flavors.
39
+ The Compiler flavor, docker-first: `auto` and `docker` pull `log10x/compiler-10x:1.1.136` (pinned by digest), and `local` runs a Compiler-flavor `tenx`. The Runtime flavor is refused; the `mode` argument of [`log10x_compile`](compile.md) lists the three flavors.
40
40
 
41
41
  A link into a folder another compile or link job is still writing joins that job instead of starting a second one.
42
42
 
@@ -138,7 +138,7 @@ This is the CLI tier, so it needs no Kubernetes, deployed app, or Log10x account
138
138
  "job_id": "7b2e9c41-3f5a-4d8e-9b21-6c0f1a2d7e44",
139
139
  "job_status": "completed",
140
140
  "mode": "docker",
141
- "image": "log10x/compiler-10x:1.1.125@sha256:ee875d49b9a25ef98b6b9edfe69a4b60888fd68fd370e8a0020f1b4777ecf12b",
141
+ "image": "log10x/compiler-10x:1.1.136@sha256:ffcd9890935804b06fcb1791994dfbb32cdb270f3e35fc423ede78ced2097b1f",
142
142
  "compiler": null,
143
143
  "exit_code": 0,
144
144
  "elapsed_ms": 6240,
@@ -149,8 +149,8 @@ The per-language scan failures and the symbol-type counts need a `compiler-10x`
149
149
  "job_id": "compile-7c2f",
150
150
  "job_status": "completed",
151
151
  "mode": "docker",
152
- "image": "log10x/compiler-10x:1.1.125@sha256:ee875d49b9a25ef98b6b9edfe69a4b60888fd68fd370e8a0020f1b4777ecf12b",
153
- "compiler": "docker:log10x/compiler-10x:1.1.125@sha256:ee875d49b9a25ef98b6b9edfe69a4b60888fd68fd370e8a0020f1b4777ecf12b",
152
+ "image": "log10x/compiler-10x:1.1.136@sha256:ffcd9890935804b06fcb1791994dfbb32cdb270f3e35fc423ede78ced2097b1f",
153
+ "compiler": "docker:log10x/compiler-10x:1.1.136@sha256:ffcd9890935804b06fcb1791994dfbb32cdb270f3e35fc423ede78ced2097b1f",
154
154
  "exit_code": 0,
155
155
  "elapsed_ms": 184220,
156
156
  "timed_out": false,
@@ -22,7 +22,7 @@ Sources combine freely in one call:
22
22
 
23
23
  Docker pulls are daemonless. GitHub and Artifactory pulls each need a token, even for public GitHub repos.
24
24
 
25
- The [Compiler app](../../../compiler/) runs the Compiler flavor, docker-first via the `log10x/compiler-10x:1.1.125` image (pinned by digest) or a local Compiler-flavor `tenx`. It runs without Kubernetes, a deployed app, or a Log10x account.
25
+ The [Compiler app](../../../compiler/) runs the Compiler flavor, docker-first via the `log10x/compiler-10x:1.1.136` image (pinned by digest) or a local Compiler-flavor `tenx`. It runs without Kubernetes, a deployed app, or a Log10x account.
26
26
 
27
27
  What gets scanned, per the shipped scanner config:
28
28
 
@@ -38,7 +38,7 @@ The output folder is keyed on the sources, the library name, and the compiler (t
38
38
 
39
39
  What a re-run skips: the engine skips every local file whose checksum already has a unit in the folder. Pulled sources cost a remote round-trip each run (GitHub branch head, image manifest digest, Artifactory listing) and are downloaded again only when the remote changed; Helm charts are rendered again every run. Only the scan of unchanged files is skipped.
40
40
 
41
- The engine's own whole-scan cap (`scanOperationTimeout`, bundled default 10 minutes) is raised to 90% of `timeout_ms` on the command line, so the engine cap follows `timeout_ms`. That needs compiler 1.1.89 or newer; the pinned default is 1.1.125, and an older local engine keeps its 10 minute cap. A run the engine stopped early is reported as `incomplete` by [Status](compile-status.md).
41
+ The engine's own whole-scan cap (`scanOperationTimeout`, bundled default 10 minutes) is raised to 90% of `timeout_ms` on the command line, so the engine cap follows `timeout_ms`. That needs compiler 1.1.89 or newer; the pinned default is 1.1.136, and an older local engine keeps its 10 minute cap. A run the engine stopped early is reported as `incomplete` by [Status](compile-status.md).
42
42
 
43
43
  ## :material-code-braces: Example
44
44
 
@@ -208,14 +208,14 @@ Archives are scanned as-is: `.jar`, `.war`, `.ear`, `.zip` and tarballs are expa
208
208
  "local"
209
209
  ],
210
210
  "default": "auto",
211
- "description": "Execution backend. `auto` (default) prefers Docker (the compiler image, guaranteed compiler flavor) and falls back to a local compiler-flavor tenx; the backend is chosen once, before the output folder is keyed. `docker` forces the image (LOG10X_COMPILER_IMAGE or LOG10X_TENX_IMAGE, default log10x/compiler-10x:1.1.125 pinned by digest, so every run goes through the same compiler). `local` forces the binary (LOG10X_TENX_PATH or `tenx` on PATH) and refuses it unless its version banner reports the compiler flavor (`compiler`, or `cloud` on an engine built before the flavor rename). Of the three shipped flavors only `compiler` can compile: `runtime` (native binary) and `runtime-jvm` (JVM-packaged runtime, the only runtime available on Windows) are both refused, since neither carries the `generate` pipeline unit. With a local install, local-folder compilation and GitHub pull (REST API + token) work out of the box; docker_images pull additionally needs a container engine (podman or docker) on the host. The docker `compiler-10x` image bundles all of those (podman included, daemonless), which is why Docker is the default."
211
+ "description": "Execution backend. `auto` (default) prefers Docker (the compiler image, guaranteed compiler flavor) and falls back to a local compiler-flavor tenx; the backend is chosen once, before the output folder is keyed. `docker` forces the image (LOG10X_COMPILER_IMAGE or LOG10X_TENX_IMAGE, default log10x/compiler-10x:1.1.136 pinned by digest, so every run goes through the same compiler). `local` forces the binary (LOG10X_TENX_PATH or `tenx` on PATH) and refuses it unless its version banner reports the compiler flavor (`compiler`, or `cloud` on an engine built before the flavor rename). Of the three shipped flavors only `compiler` can compile: `runtime` (native binary) and `runtime-jvm` (JVM-packaged runtime, the only runtime available on Windows) are both refused, since neither carries the `generate` pipeline unit. With a local install, local-folder compilation and GitHub pull (REST API + token) work out of the box; docker_images pull additionally needs a container engine (podman or docker) on the host. The docker `compiler-10x` image bundles all of those (podman included, daemonless), which is why Docker is the default."
212
212
  },
213
213
  "timeout_ms": {
214
214
  "type": "integer",
215
215
  "minimum": 10000,
216
216
  "maximum": 3600000,
217
217
  "default": 1800000,
218
- "description": "Hard cap on compile wall time in milliseconds. Default 1,800,000 (30 min). The engine's own whole-scan cap (scanOperationTimeout, bundled default 10 min) is raised to 90% of this value on the command line, so the engine cap follows this value (needs compiler 1.1.89+; the pinned default is 1.1.125; an older local engine keeps its 10 min cap). A re-run over unchanged local sources skips every file whose checksum already has a unit and finishes in seconds. Pulled sources (GitHub, images, Artifactory) are re-checked remotely each run and downloaded again only when the remote changed; Helm charts are rendered again every run."
218
+ "description": "Hard cap on compile wall time in milliseconds. Default 1,800,000 (30 min). The engine's own whole-scan cap (scanOperationTimeout, bundled default 10 min) is raised to 90% of this value on the command line, so the engine cap follows this value (needs compiler 1.1.89+; the pinned default is 1.1.136; an older local engine keeps its 10 min cap). A re-run over unchanged local sources skips every file whose checksum already has a unit and finishes in seconds. Pulled sources (GitHub, images, Artifactory) are re-checked remotely each run and downloaded again only when the remote changed; Helm charts are rendered again every run."
219
219
  },
220
220
  "max_wait_ms": {
221
221
  "type": "integer",
@@ -264,8 +264,8 @@ Archives are scanned as-is: `.jar`, `.war`, `.ear`, `.zip` and tarballs are expa
264
264
  "job_id": "9f2c1a3e-7b4d-4e1a-9f0c-2a1b3c4d5e6f",
265
265
  "job_status": "completed",
266
266
  "mode": "docker",
267
- "image": "log10x/compiler-10x:1.1.125@sha256:ee875d49b9a25ef98b6b9edfe69a4b60888fd68fd370e8a0020f1b4777ecf12b",
268
- "compiler": "docker:log10x/compiler-10x:1.1.125@sha256:ee875d49b9a25ef98b6b9edfe69a4b60888fd68fd370e8a0020f1b4777ecf12b",
267
+ "image": "log10x/compiler-10x:1.1.136@sha256:ffcd9890935804b06fcb1791994dfbb32cdb270f3e35fc423ede78ced2097b1f",
268
+ "compiler": "docker:log10x/compiler-10x:1.1.136@sha256:ffcd9890935804b06fcb1791994dfbb32cdb270f3e35fc423ede78ced2097b1f",
269
269
  "exit_code": 0,
270
270
  "elapsed_ms": 38120,
271
271
  "timed_out": false,
@@ -52,7 +52,7 @@ To re-link an existing units tree into a library with no source scan, use [`log1
52
52
 
53
53
  ## :material-check-decagram-outline: Prerequisites
54
54
 
55
- The compile family runs the [Compiler app](../../../compiler/), which executes the [compile pipeline](../../../../compile/), docker-first. By default it pulls `log10x/compiler-10x:1.1.125`, pinned by digest, so every run goes through the same compiler.
55
+ The compile family runs the [Compiler app](../../../compiler/), which executes the [compile pipeline](../../../../compile/), docker-first. By default it pulls `log10x/compiler-10x:1.1.136`, pinned by digest, so every run goes through the same compiler.
56
56
 
57
57
  That image bundles the following, so no host docker socket is needed:
58
58
 
@@ -2179,7 +2179,7 @@ The OTel Collector overlay moves the sidecar's OTLP listener with `TENX_OTEL_COL
2179
2179
 
2180
2180
  **Phase C: processed events flowing**
2181
2181
 
2182
- The receiver stamps a `tenx_hash` field on every record it hands back, so that field arriving at the destination is the proof that events completed the round trip through 10x. For a real destination, query for the field there: Elasticsearch `exists: tenx_hash`, Splunk `tenx_hash=*`, Datadog `@tenx_hash:*`. For the stdout / `debug` placeholder destination from Step 3, count it on the container that writes to the destination. Pick your forwarder:
2182
+ The receiver hands each record back byte-for-byte unless `symbolMessageHashField` names a field (the forwarder args above). Name one, for example `symbolMessageHashField tenx_hash`, and the receiver adds that field to every record it hands back, so the field arriving at the destination is the proof that events completed the round trip through 10x. For a real destination, query for the field there: Elasticsearch `exists: tenx_hash`, Splunk `tenx_hash=*`, Datadog `@tenx_hash:*`. For the stdout / `debug` placeholder destination from Step 3, count it on the container that writes to the destination. Pick your forwarder:
2183
2183
 
2184
2184
  === ":simple-fluentd: Fluentd"
2185
2185
 
@@ -3,7 +3,7 @@ icon: material/help-circle-outline
3
3
  title: "FAQ"
4
4
  ---
5
5
 
6
- The [Receiver](https://doc.log10x.com/apps/receiver/) is the execution arm of the 10x pipeline. The rate regulator finds which patterns are over the byte cap set for their service, and an action decides what happens to that excess. The keep-everything actions (compact, offload, tier_down) cut cost without losing data; sample and drop are lossy actions you opt into; pass leaves the pattern untouched. An AI agent picks the action per service through the log10x MCP (the `configure_engine` tool); the engine enforces it, and the decision travels as a config change through the GitOps repo. The Receiver runs as a sidecar alongside your forwarder.
6
+ The [Receiver](https://doc.log10x.com/apps/receiver/) is the execution arm of the 10x pipeline. The rate regulator finds which patterns are over the byte cap set for their service, and an action decides what happens to that excess. The keep-everything actions (compact, offload, tier_down) cut cost without losing data; sample and drop, the Receiver's Filter modes, are lossy: they apply only to the share of a pattern over a cap the operator sets, and that share is dropped unless the operator chooses another action; pass leaves the pattern untouched. An AI agent picks the action per service through the log10x MCP (the `configure_engine` tool); the engine enforces it, and the decision travels as a config change through the GitOps repo. The Receiver runs as a sidecar alongside your forwarder.
7
7
 
8
8
  <div class="grid cards" markdown>
9
9
 
@@ -27,31 +27,30 @@ The [Receiver](https://doc.log10x.com/apps/receiver/) is the execution arm of th
27
27
  - **offload**: route the pattern to customer-owned object storage (S3 or any S3-compatible bucket) instead of the destination.
28
28
  - **drop**: stop forwarding the pattern.
29
29
 
30
- An AI agent picks the action per service through the log10x MCP (the `configure_engine` tool), which turns a target percent or budget into a per-pattern action set carried as a cap/action CSV. The CSV lands in the config repo through a GitOps PR and hot-reloads on the next pull. The sample action runs in [per-node budget mode](https://doc.log10x.com/run/receive/rate/){target="_blank"} or [mute file mode](https://doc.log10x.com/run/receive/rate/#mute-file-mode-declarative-field-set-caps){target="_blank"}, a declarative field-set keyed cap file committed to git and pulled via GitOps.
30
+ An AI agent picks the action per service through the log10x MCP (the `configure_engine` tool), which turns a target percent or budget into per-service caps and actions carried as cap and action CSV files. The files land in the config repo through a GitOps PR and hot-reload on the next pull. Per-pattern rates come from the [mute file](https://doc.log10x.com/run/receive/rate/#protection-list){target="_blank"}, a CSV keyed by pattern, committed to git and pulled the same way.
31
31
 
32
32
  Log10x is normally driven by an AI agent (Claude, or a model the customer brings) through the log10x MCP server, which installs, configures, and queries via MCP tools. The manual steps in these docs are the same operations without an agent.
33
33
 
34
34
  ??? tenx-overview "How do budget policies work"
35
35
 
36
- Configuration is a YAML file, written by hand or generated by the MCP `configure_engine` tool and delivered through GitOps. The [rate receiver](https://doc.log10x.com/run/receive/rate/){target="_blank"} tracks per-event-type spend using automatic [symbol identity](https://doc.log10x.com/run/initialize/message/){target="_blank"} enrichment. You configure:
36
+ Configuration is a YAML file, written by hand or generated by the MCP `configure_engine` tool and delivered through GitOps. The [rate receiver](https://doc.log10x.com/run/receive/rate/){target="_blank"} caps the bytes each pattern ([symbol identity](https://doc.log10x.com/run/initialize/message/){target="_blank"}) may spend in its container over a rolling window. You configure:
37
37
 
38
- - **Budget per hour**, target ingestion cost rate (e.g., $1.50/hour)
39
- - **Max share per event type**, prevents any single event type from dominating the budget (e.g., 20%)
40
- - **Severity floors**, ERROR events keep a higher minimum share than DEBUG noise
38
+ - **A byte cap per pattern per container**, fleet-wide or per container from a cap file; the MCP derives it from a monthly dollar budget
39
+ - **A share guard**, a pattern below a minimum share of its container's volume (default 5%) is left alone
40
+ - **Severity floors**, an over-cap pattern keeps at least a set share of each level (by default ERROR 50%, WARN 30%, INFO 10%)
41
+ - **An action per service** for the excess: drop (the default), offload, tier_down, compact, sample or pass
41
42
 
42
- Example policy: cap total ingestion at $500/hour, cap any single event type at 20% of the budget, prioritize ERROR over INFO.
43
-
44
- When an event type exceeds its max share, the Receiver samples it down proportionally. For [multi-app environments](https://doc.log10x.com/run/receive/rate/#multi-app-regulation){target="_blank"} (Kubernetes), cap per-app budgets using the [container name](https://doc.log10x.com/run/initialize/k8s/){target="_blank"} field, scaling replicas doesn't bypass limits.
43
+ Cost is reported per pattern; the cap itself is in bytes. Caps are scoped by the [container name](https://doc.log10x.com/run/receive/rate/#containers){target="_blank"}, so scaling replicas doesn't bypass them.
45
44
 
46
45
  ??? tenx-overview "What happens to logs when budget limits are reached (sample action)"
47
46
 
48
- The [rate receiver](https://doc.log10x.com/run/receive/rate/){target="_blank"} applies cost-based sampling. Each event is either **retained** or **dropped** based on how much its event type has spent relative to the budget:
47
+ The [rate receiver](https://doc.log10x.com/run/receive/rate/){target="_blank"} decides per event, on bytes:
49
48
 
50
- - **Under budget**, all events flow through normally
51
- - **Event type over its max share**, that type gets sampled down proportionally (e.g., an event type consuming 60% of the budget with a 20% cap gets sampled to ~33%)
52
- - **Severity floors**, ERROR events keep a higher minimum share of the over-cap slice than DEBUG noise
49
+ - **At or below the pattern's cap**, every event flows through
50
+ - **Over the cap**, each level keeps its floor share and the rest takes the container's action (drop when none is set); nothing targets the excess back down to the cap line
51
+ - **Severity floors**, ERROR events keep a higher share of the over-cap slice than DEBUG noise
53
52
 
54
- The result: noisy event types are automatically throttled while critical events are preserved. The Receiver exports [metrics](https://doc.log10x.com/run/output/metric/){target="_blank"} tracking what each pattern's action did, so policies can be tuned over time. Routed-away events carry the label `routeState` (for example `routeState="drop"`).
53
+ The result: a noisy pattern is throttled while its errors keep flowing. The Receiver exports [metrics](https://doc.log10x.com/run/output/metric/){target="_blank"} tracking what each pattern's action did, so policies can be tuned over time. Routed-away events carry the label `routeState` (for example `routeState="drop"`).
55
54
 
56
55
  ### :material-cog-outline: Configuration
57
56
 
@@ -69,10 +68,10 @@ The [Receiver](https://doc.log10x.com/apps/receiver/) is the execution arm of th
69
68
 
70
69
  The [rate receiver](https://doc.log10x.com/run/receive/rate/){target="_blank"} uses two mechanisms that naturally protect critical events:
71
70
 
72
- - **Severity floors**, ERROR and WARN events keep a higher minimum share during sampling. Even when their pattern is over its cap, critical-severity events keep flowing at a higher rate than DEBUG noise
73
- - **Max share targeting**, sampling only kicks in when a specific [event type](https://doc.log10x.com/run/initialize/message/){target="_blank"} exceeds its configured share of the budget (e.g., 20%). Low-volume event types, which security and authentication logs typically are, stay well under their share and pass through unaffected
71
+ - **Severity floors**, ERROR and WARN events keep a higher minimum share of the over-cap slice. Even when their pattern is over its cap, critical-severity events keep flowing at a higher rate than DEBUG noise
72
+ - **Per-pattern caps**, the regulator engages only on a [pattern](https://doc.log10x.com/run/initialize/message/){target="_blank"} over its byte cap and above the share guard (default 5% of its container's volume). Low-volume patterns, which security and authentication logs typically are, stay under their cap and pass through unaffected
74
73
 
75
- The Receiver targets noisy, high-volume event types that dominate your budget, not broad categories. Security events that don't spike beyond their budget share flow through without sampling.
74
+ The Receiver targets noisy, high-volume patterns, not broad categories. Security events that stay under their cap flow through untouched.
76
75
 
77
76
  ??? tenx-capabilities "How do I monitor budget consumption"
78
77
 
@@ -7,7 +7,7 @@ icon: "material/pipe-valve"
7
7
  ---
8
8
  The Receiver runs as a forwarder sidecar and acts on each pattern before events reach your analyzer.
9
9
 
10
- It picks one action per pattern: **pass, sample, compact, tier_down, offload, or drop**. An AI agent sets the action per service through the [log10x MCP](https://doc.log10x.com/apps/mcp/) (`configure_engine`); the engine enforces it, carried as a config change through the [GitOps](https://doc.log10x.com/engine/gitops/) repo.
10
+ Its actions are **pass, sample, compact, tier_down, offload, and drop**. The action is set per service and applies to every pattern over its cap in that service. An AI agent sets the action per service through the [log10x MCP](https://doc.log10x.com/apps/mcp/) (`configure_engine`); the engine enforces it, carried as a config change through the [GitOps](https://doc.log10x.com/engine/gitops/) repo.
11
11
 
12
12
  ## :material-cog-transfer-outline: Workflow
13
13
 
@@ -71,7 +71,7 @@ The Receiver executes as a [forwarder sidecar](https://doc.log10x.com/engine/lau
71
71
 
72
72
  === ":material-file-document-edit-outline: Mute File (GitOps)"
73
73
 
74
- A declarative file keyed by the joined `rateReceiverFieldNames` values (e.g. `symbolMessage`, `container`) caps specific patterns with an explicit sample rate and expiry. Operators (or an AI assistant via the [Log10x MCP](https://github.com/log-10x/log10x-mcp)) append entries based on [Reporter](https://doc.log10x.com/apps/reporter/) cost attribution, commit to git, and every receiver pulls the file on its next reload. Each mute is diff-reviewed, self-expires, and maps 1:1 to the field-sets the Reporter attributes cost to. See [mute file mode](https://doc.log10x.com/run/receive/rate/#mute-file-mode-declarative-field-set-caps).
74
+ A declarative file keyed by the joined `rateReceiverFieldNames` values (e.g. `symbolMessage`, `container`) caps specific patterns with an explicit sample rate and expiry. Operators (or an AI assistant via the [Log10x MCP](https://github.com/log-10x/log10x-mcp)) append entries based on [Reporter](https://doc.log10x.com/apps/reporter/) cost attribution, commit to git, and every receiver pulls the file on its next reload. Each mute is diff-reviewed, self-expires, and maps 1:1 to the field-sets the Reporter attributes cost to. See [mute file mode](https://doc.log10x.com/run/receive/rate/#protection-list).
75
75
 
76
76
  ## :material-shield-check-outline: Safety & Reliability
77
77
 
@@ -1303,8 +1303,8 @@ To configure the Receiver app, [:material-cog: Edit](https://doc.log10x.com/conf
1303
1303
  contexts: log,exec
1304
1304
 
1305
1305
  # 'messageField' specifies the field name to assign the message pattern to target TenXObjects: the symbol
1306
- # tokens of the origin ranked highest by coverage, meaning the count of distinct library-known tokens it
1307
- # explains. Span length is the last tiebreak. The origin file name itself goes to 'originField'.
1306
+ # tokens of the origin ranked highest on the five-key comparator, message-text width first. Span length
1307
+ # is the last tiebreak. The origin file name itself goes to 'originField'.
1308
1308
  messageField: message_pattern
1309
1309
 
1310
1310
  # 'messageHashField' specifies the field name to assign a stable, URL-safe pattern hash (xxHash64 of the messageField value, base64url, 11 chars). Safe as a SIEM query term or forwarder filter key.
@@ -1326,8 +1326,9 @@ To configure the Receiver app, [:material-cog: Edit](https://doc.log10x.com/conf
1326
1326
  # indented detail, banners) so the message-template skips its own message-pattern compute for them.
1327
1327
  # The check fires only for templates that don't already pass the (groupSize > 1) || groupHead arms;
1328
1328
  # i.e., for single-event templates with no head signal. A non-negator line in that case (e.g., bare
1329
- # "Hello World", "Heartbeat received from node-7") still gets a pattern; a negator-matching orphan
1330
- # (e.g., a lone "\tat com.foo.Bar(...)") does not.
1329
+ # "Hello World", "Heartbeat received from node-7") gets a pattern from its library words; a negator-matching
1330
+ # orphan (e.g., a lone "\tat com.foo.Bar(...)") skips origin selection and is named by a hash of its shape,
1331
+ # template_<hash>.
1331
1332
  message:
1332
1333
  negators:
1333
1334
  - " " # any indented line (stack frames, indented app detail, multi-line JSON, banners)
@@ -826,8 +826,8 @@ To configure the Reporter app, [:material-cog: Edit](https://doc.log10x.com/conf
826
826
  contexts: log,exec
827
827
 
828
828
  # 'messageField' specifies the field name to assign the message pattern to target TenXObjects: the symbol
829
- # tokens of the origin ranked highest by coverage, meaning the count of distinct library-known tokens it
830
- # explains. Span length is the last tiebreak. The origin file name itself goes to 'originField'.
829
+ # tokens of the origin ranked highest on the five-key comparator, message-text width first. Span length
830
+ # is the last tiebreak. The origin file name itself goes to 'originField'.
831
831
  messageField: message_pattern
832
832
 
833
833
  # 'messageHashField' specifies the field name to assign a stable, URL-safe pattern hash (xxHash64 of the messageField value, base64url, 11 chars). Safe as a SIEM query term or forwarder filter key.
@@ -849,8 +849,9 @@ To configure the Reporter app, [:material-cog: Edit](https://doc.log10x.com/conf
849
849
  # indented detail, banners) so the message-template skips its own message-pattern compute for them.
850
850
  # The check fires only for templates that don't already pass the (groupSize > 1) || groupHead arms;
851
851
  # i.e., for single-event templates with no head signal. A non-negator line in that case (e.g., bare
852
- # "Hello World", "Heartbeat received from node-7") still gets a pattern; a negator-matching orphan
853
- # (e.g., a lone "\tat com.foo.Bar(...)") does not.
852
+ # "Hello World", "Heartbeat received from node-7") gets a pattern from its library words; a negator-matching
853
+ # orphan (e.g., a lone "\tat com.foo.Bar(...)") skips origin selection and is named by a hash of its shape,
854
+ # template_<hash>.
854
855
  message:
855
856
  negators:
856
857
  - " " # any indented line (stack frames, indented app detail, multi-line JSON, banners)
@@ -1373,8 +1373,8 @@ To configure the Retriever app, [:material-cog: Edit](https://doc.log10x.com/con
1373
1373
  contexts: log,exec
1374
1374
 
1375
1375
  # 'messageField' specifies the field name to assign the message pattern to target TenXObjects: the symbol
1376
- # tokens of the origin ranked highest by coverage, meaning the count of distinct library-known tokens it
1377
- # explains. Span length is the last tiebreak. The origin file name itself goes to 'originField'.
1376
+ # tokens of the origin ranked highest on the five-key comparator, message-text width first. Span length
1377
+ # is the last tiebreak. The origin file name itself goes to 'originField'.
1378
1378
  messageField: message_pattern
1379
1379
 
1380
1380
  # 'messageHashField' specifies the field name to assign a stable, URL-safe pattern hash (xxHash64 of the messageField value, base64url, 11 chars). Safe as a SIEM query term or forwarder filter key.
@@ -1396,8 +1396,9 @@ To configure the Retriever app, [:material-cog: Edit](https://doc.log10x.com/con
1396
1396
  # indented detail, banners) so the message-template skips its own message-pattern compute for them.
1397
1397
  # The check fires only for templates that don't already pass the (groupSize > 1) || groupHead arms;
1398
1398
  # i.e., for single-event templates with no head signal. A non-negator line in that case (e.g., bare
1399
- # "Hello World", "Heartbeat received from node-7") still gets a pattern; a negator-matching orphan
1400
- # (e.g., a lone "\tat com.foo.Bar(...)") does not.
1399
+ # "Hello World", "Heartbeat received from node-7") gets a pattern from its library words; a negator-matching
1400
+ # orphan (e.g., a lone "\tat com.foo.Bar(...)") skips origin selection and is named by a hash of its shape,
1401
+ # template_<hash>.
1401
1402
  message:
1402
1403
  negators:
1403
1404
  - " " # any indented line (stack frames, indented app detail, multi-line JSON, banners)
@@ -26,13 +26,13 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
26
26
 
27
27
  ??? tenx-security "What information does the compiler generate"
28
28
 
29
- The compiler extracts **symbol values**, tokens that appear in log statements, such as string literals, enum values, and format patterns. Extracted string constants, from log statement formats (e.g., `"error in %s"`), [binary executables](https://doc.log10x.com/compile/scanner/executable/) (via `strings`), or [text/JSON/YAML files](https://doc.log10x.com/compile/scanner/text/), are stored as **64-bit hashes** in the symbol library. The original plain text is never written to the output. The runtime engine matches incoming events against these hashes, not against your source strings.
29
+ The compiler extracts **symbol values**, tokens that appear in log statements, such as string literals, enum values, and format patterns. Extracted string constants, from log statement formats (e.g., `"error in %s"`), [binary executables](https://doc.log10x.com/compile/scanner/executable/) (via `strings`), or [text/JSON/YAML files](https://doc.log10x.com/compile/scanner/text/), are stored as **64-bit hashes** in the symbol library. The original plain text of a string is never written to the output; file names and class, method and enum names are. The runtime engine matches incoming events against these hashes, not against your source strings.
30
30
 
31
31
  Class and method names are stored alongside hashes to identify the source of each log statement, comparable to what appears in a stack trace.
32
32
 
33
33
  The compiler does **not** extract application logic, control flow, full source file contents, or environment variables. It does not read `.env` files or runtime state.
34
34
 
35
- Output is a [symbol library](https://doc.log10x.com/compile/link/#symbol-library) file (`.10x.tar`) containing hash mappings and their source locations. You can inspect the contents before deploying (`tar tf library.10x.tar`).
35
+ Output is a [symbol library](https://doc.log10x.com/compile/link/#symbol-library) file (`.10x.tar`) containing hashes and the file names they came from. You can inspect the contents before deploying (`tar tf library.10x.tar`).
36
36
 
37
37
  ??? tenx-security "What permissions does the compiler need, and how do I scope access"
38
38
 
@@ -45,7 +45,7 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
45
45
  | [Helm charts](https://doc.log10x.com/compile/pull/helm/) | Same as GitHub + Docker | Resolves chart image references, then pulls each image |
46
46
  | [Artifactory](https://doc.log10x.com/compile/pull/artifactory/) | `ARTIFACTORY_TOKEN` | Read access to target repositories |
47
47
 
48
- You control exactly which repos and images the compiler accesses via your [configuration](https://doc.log10x.com/compile/test/#config-files). The compiler only pulls sources you explicitly list, it does not discover or scan anything beyond what you configure.
48
+ You control exactly which repos and images the compiler accesses via your [configuration](https://doc.log10x.com/compile/test/#config-files). The compiler only pulls sources you explicitly list, and the images and repositories those sources reference (a Helm chart's images, an image's source repository, a `go.mod` file's dependencies).
49
49
 
50
50
  All processing runs inside your infrastructure (CI/CD pipeline or k8s CronJob). No source code or symbol data is sent to Log10x or any external service.
51
51
 
@@ -73,7 +73,7 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
73
73
 
74
74
  ??? tenx-info "Do I need to run the compiler, or do the built-in defaults cover my stack"
75
75
 
76
- The 10x runtime ships with a [default symbol library](https://doc.log10x.com/compile/pull/#default-symbols) covering 131 open-source repositories and 14 public container images across Java, Python, Node.js, Go, C++, Rust, Ruby, .NET, and Scala, including Kubernetes, OpenTelemetry, Spring Boot, Django, Express, nginx, PostgreSQL, Redis, Kafka, and many more.
76
+ The 10x runtime ships with a [default symbol library](https://doc.log10x.com/compile/pull/#default-symbols) covering the open-source repositories and public container images listed in the manifest shipped inside the library archive, across Java, Python, Node.js, Go, C++, Rust, Ruby, .NET, and Scala, including Kubernetes, OpenTelemetry, Spring Boot, Django, Express, nginx, PostgreSQL, Redis, Kafka, and many more.
77
77
 
78
78
  **Start with the defaults.** Run a local cost POC on your own log files to see how much the built-in library covers. The runtime reports which events matched known symbols and which did not.
79
79
 
@@ -95,7 +95,7 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
95
95
 
96
96
  ??? tenx-coverage "My stack uses common open-source frameworks, do I need to run the compiler at all"
97
97
 
98
- The 10x runtime ships with a [default symbol library](https://doc.log10x.com/compile/pull/#default-symbols) covering 131 open-source repositories and 14 public container images. If your stack is covered by the defaults, you don't need to run the compiler.
98
+ The 10x runtime ships with a [default symbol library](https://doc.log10x.com/compile/pull/#default-symbols) covering the repositories and container images listed in the manifest shipped inside the library archive. If your stack is covered by the defaults, you don't need to run the compiler.
99
99
 
100
100
  Run the compiler to **add custom symbols** for your proprietary application code on top of the defaults. This improves template accuracy for your specific log formats. The compiler output supplements the default library, not replaces it.
101
101
 
@@ -107,23 +107,23 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
107
107
 
108
108
  1. **[Docker image scanning](https://doc.log10x.com/compile/pull/docker/)**, Pull any Docker image from any registry (public or private), including closed-source vendor agents. The compiler exports and scans the image filesystem for symbol content. Images are only re-pulled when their SHA256 changes, making incremental runs efficient. Works for `nginx:latest`, Datadog agent images, Splunk UF, or any vendor-provided image.
109
109
 
110
- 2. **[Executable scanning](https://doc.log10x.com/compile/scanner/executable/)**, Runs the OS `strings` utility on compiled binaries (`.so`, `.dll`, `.dynlib`) to extract symbol values. Runs inside your infrastructure; output is only 64-bit hashes (no binary content leaves your network).
110
+ 2. **[Executable scanning](https://doc.log10x.com/compile/scanner/executable/)**, Runs the OS `strings` utility on compiled binaries (`.so`, `.dll`, `.dynlib`) to extract symbol values. Runs inside your infrastructure; output is the binary's file name plus 64-bit hashes of its strings (no binary content leaves your network).
111
111
 
112
112
  3. **JIT fallback**, For truly opaque sources (AWS-managed service logs, SaaS tools with no Docker image), the JIT runtime handles them automatically at runtime. The engine creates a template on the first event seen, giving ~80% efficiency without any compiler setup needed.
113
113
 
114
114
  ??? tenx-coverage "Do you support custom and proprietary log formats, or just the default library"
115
115
 
116
- **Both.** The default library covers common frameworks, but the compiler lets you add custom symbols for proprietary logging patterns. Custom formats work out of the box with 80% coverage from the built-in library.
116
+ **Both.** The default library covers common frameworks, but the compiler lets you add custom symbols for proprietary logging patterns. Custom formats work out of the box with the built-in library.
117
117
 
118
- **How it works:** The runtime [tokenizes](https://doc.log10x.com/run/transform/structure/) incoming events and matches tokens against the symbol library. With 80% built-in coverage, most generic tokens (class names, log levels, timestamps, method names) already match. For logs from your proprietary services, the runtime still processes them, but template inference is less precise.
118
+ **How it works:** The runtime [tokenizes](https://doc.log10x.com/run/transform/structure/) incoming events and matches tokens against the symbol library. With the built-in library, most generic tokens (class names, log levels, method names) already match; timestamps are [variables](https://doc.log10x.com/run/transform/structure/#variables), not library words. For logs from your proprietary services, the runtime still processes them, but template inference is less precise.
119
119
 
120
- **When to optimize:** If you want accurate [message extraction](https://doc.log10x.com/run/initialize/message/) for cost attribution and metrics, capturing the intended "essence" of each custom event type, run the compiler on your custom code. This generates additional symbols, so the runtime recognizes your proprietary patterns and builds more accurate templates. The compiler is iterative: initial runs take 10-30 minutes to scan your code. Subsequent runs only process changed files, completing in seconds, perfect for CI/CD pipelines. You point it at your repositories once, then it automatically supplements the default symbol library on each deploy.
120
+ **When to optimize:** If you want accurate [message extraction](https://doc.log10x.com/run/initialize/message/) for cost attribution and metrics, capturing the intended "essence" of each custom event type, run the compiler on your custom code. This generates additional symbols, so the runtime recognizes your proprietary patterns and builds more accurate templates. The compiler is iterative: an initial run scans all of your code; later runs scan only what changed. Subsequent runs only process changed files, completing in seconds, perfect for CI/CD pipelines. You point it at your repositories once, then it automatically supplements the default symbol library on each deploy.
121
121
 
122
122
  See [real-world before/after examples](https://doc.log10x.com/apps/receiver/){target="_blank"} showing how symbol coverage improves lossless volume reduction on production logs.
123
123
 
124
124
  ??? tenx-coverage "My custom format isn't in the library, does it still work"
125
125
 
126
- Yes. The engine creates a template for every event type it encounters, there are no "unrecognized" events. The [symbol library](https://doc.log10x.com/run/symbol/) provides vocabulary that determines how much of each event is classified as structure (low-cardinality, stored in the template) vs variable data (high-cardinality, shipped per event). The [default library](https://doc.log10x.com/compile/pull/#default-symbols) covers 145 pinned sources and delivers strong reduction on most workloads.
126
+ Yes. The engine creates a template for every event type it encounters, there are no "unrecognized" events. The [symbol library](https://doc.log10x.com/run/symbol/) provides vocabulary that determines how much of each event is classified as structure (low-cardinality, stored in the template) vs variable data (high-cardinality, shipped per event). The [default library](https://doc.log10x.com/compile/pull/#default-symbols) covers the sources listed in its manifest, each pinned to a commit or image digest, and delivers strong reduction on most workloads.
127
127
 
128
128
  Running the compiler on custom application code enriches the vocabulary further, producing richer templates, more accurate [message extraction](https://doc.log10x.com/run/initialize/message/), and higher reduction ratios. Without it, the format still works using the default vocabulary, with no configuration or manual regex rules.
129
129
 
@@ -131,7 +131,7 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
131
131
 
132
132
  ??? tenx-operations "How often do I need to run the compiler, and what is the maintenance overhead"
133
133
 
134
- Minimal. Initial runs take a few minutes; incremental runs on unchanged code complete in seconds. Deploy as a [k8s CronJob](https://doc.log10x.com/compile/deploy/) (every 30 minutes) or trigger from CI/CD on commits.
134
+ Minimal. Initial runs take a few minutes; incremental runs on unchanged code complete in seconds. Deploy as a [k8s CronJob](https://doc.log10x.com/compile/deploy/) on a schedule you choose, or trigger from CI/CD on commits. A rebuilt library can rename the affected patterns once, so a schedule is a choice with that consequence. To pin the library, load it from a branch you update deliberately, or set `syncInterval` to null in the [GitHub](https://doc.log10x.com/config/github/) pull so the runtime loads it once at start-up.
135
135
 
136
136
  How it stays lightweight:
137
137
 
@@ -149,7 +149,7 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
149
149
 
150
150
  To troubleshoot:
151
151
 
152
- - Extract and inspect [symbol library contents](https://doc.log10x.com/compile/link/#file-structure) (`tar tf library.10x.tar`), verify your target symbols were captured
152
+ - Extract and inspect [symbol library contents](https://doc.log10x.com/compile/link/#file-structure) (`tar tf library.10x.tar`); units list file, class and method names, and each string only as a hash
153
153
  - Run the engine locally on test logs and enable [debug output](https://doc.log10x.com/run/transform/symbol/#debug) to see symbol matching and template assignment in detail
154
154
  - Check the engine [log file](https://doc.log10x.com/manage/logging/#log-file-location) for runtime template assignment messages
155
155
 
@@ -162,7 +162,7 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
162
162
  - **[GitHub repos](https://doc.log10x.com/compile/pull/github/)**, extracts symbols from source code and compiled binaries
163
163
  - **[Artifactory](https://doc.log10x.com/compile/pull/artifactory/)**, pulls from artifact repositories
164
164
 
165
- The runtime ships with a [default symbol library](https://doc.log10x.com/compile/pull/#default-symbols) covering 131 open-source repositories and 14 public container images (Spring Boot, Django, Express, Kafka, Kubernetes, and more). Most environments work out of the box, custom compilation is only needed for proprietary logging frameworks or application-specific log formats.
165
+ The runtime ships with a [default symbol library](https://doc.log10x.com/compile/pull/#default-symbols) covering the repositories and container images listed in the manifest shipped inside the library archive (Spring Boot, Django, Express, Kafka, Kubernetes, and more). Most environments work out of the box, custom compilation is only needed for proprietary logging frameworks or application-specific log formats.
166
166
 
167
167
  The compiler commits custom symbol libraries to GitHub as part of its pipeline. Edge and cloud apps pull them automatically via the [`@github` launch macro](https://doc.log10x.com/config/github/) at startup and poll for changes at a configurable interval. See [compiler workflow](https://doc.log10x.com/compile/#workflow) for the full CI/CD → Pull → Scan → Link → Push → Distribute pipeline.
168
168
 
@@ -8,7 +8,7 @@ icon: "material/cogs"
8
8
  ---
9
9
  The `AOT compile` pipeline produces [symbol library](https://doc.log10x.com/compile/link/#symbol-library) files that enable the [run](https://doc.log10x.com/run/) stream processor pipeline to transform input events into typed [TenXObjects](https://doc.log10x.com/api/js/#TenXObject "Provide structured, reflective access to log/trace events read from input(s)."). To launch this pipeline use the [compiler app](https://doc.log10x.com/apps/compiler/).
10
10
 
11
- The 10x Engine's [default library](https://doc.log10x.com/compile/pull/#default-symbols) covers 150+ frameworks. Running the compiler on your own environment's repos is **optional** and enables the runtime to increase its level of efficiency in aggregating and reducing event volume.
11
+ The 10x Engine ships a [default library](https://doc.log10x.com/compile/pull/#default-symbols); the manifest inside it lists every repository and container image it was built from. See [naming](https://doc.log10x.com/run/initialize/message/#message-extraction) for how a line gets its pattern name. Running the compiler on your own environment's repos is **optional** and enables the runtime to increase its level of efficiency in aggregating and reducing event volume.
12
12
 
13
13
  ## :material-cog-transfer-outline: Workflow
14
14
 
@@ -24,6 +24,7 @@ Operating on typed TenXObjects enables direct access to [symbol](https://doc.log
24
24
  Each symbol library `.10x.tar` archive file contains:
25
25
 
26
26
  - A `.10x.json` file combining the symbol unit files generated during the [scan](https://doc.log10x.com/compile/scan/ "Capture symbol values from source code/binary files and link to an output symbol library") phase.
27
+ - A `manifest.10x.json` file listing each source with the commit or digest it was built from.
27
28
  - A `.10x.pb` Protocol Buffer file which provides a reverse in-mem index of the symbols values contained within the JSON for fast loading and random access. To learn more see the [.proto IDL](https://github.com/log-10x/pipeline-extensions/blob/main/api-extensions/src/main/proto/FsSymbolUnitsIndex.proto){target="\_blank"}.
28
29
 
29
30
  ## :material-wrench-outline: Config Files
@@ -290,7 +291,7 @@ Token types read from input source/binary files written to output.
290
291
 
291
292
  |Type|Default|
292
293
  |---|---|
293
- |List|\[package, class, enum, log, text, exec\]|
294
+ |List|\[class, enum, log\]|
294
295
 
295
296
  Filter symbol values to write to [outputSymbolLibraryFile](https://doc.log10x.com/compile/link/#outputsymbollibraryfile "location of output symbol library file") based on their [context](https://doc.log10x.com/run/transform/symbol/#contexts).
296
297
  This option reduces the size of the output symbol library file by omitting symbol types not required at run time.
@@ -13,6 +13,8 @@ Pulling [symbol files](https://doc.log10x.com/compile/scanner/symbol/ "Discover
13
13
 
14
14
  The 10x runtime comes with a built-in symbol library that includes symbols for industry-standard frameworks and infrastructure components. This pre-compiled symbol library enables immediate log optimization capabilities without requiring users to scan and compile symbols for common technologies and popular open-source projects.
15
15
 
16
+ The library's [manifest](https://doc.log10x.com/compile/link/#file-structure) lists each source with the commit or digest it was built from. From container images the library takes the strings in their binaries. Test files, vendored copies and log files inside the repositories are left out. Each project keeps its own license: the library holds hashes of its strings and the names of its files, classes and methods, not its source text.
17
+
16
18
  ??? tenx-symbols "Container Images (Helm Charts)"
17
19
 
18
20
  Symbols extracted from Docker container images and their source code dependencies via Helm chart scanning.
@@ -21,7 +23,7 @@ The 10x runtime comes with a built-in symbol library that includes symbols for i
21
23
  |------------|------------|
22
24
  | [open-telemetry/opentelemetry-demo](https://github.com/open-telemetry/opentelemetry-demo) | Polyglot microservices demo (14 services in Go, Java, Node.js, Python, .NET, Rust, PHP, Ruby, C++) |
23
25
  | [ingress-nginx/ingress-nginx](https://github.com/kubernetes/ingress-nginx) | NGINX binary, HTTP error codes, upstream timeouts, load balancer logs |
24
- | [prometheus-community/kube-prometheus-stack](https://github.com/prometheus-community/helm-charts) | Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics |
26
+ | [prometheus-community/kube-prometheus-stack](https://github.com/prometheus-community/helm-charts) | Prometheus, Prometheus Operator, node-exporter, kube-state-metrics |
25
27
 
26
28
  ??? tenx-symbols "Java Frameworks"
27
29
 
@@ -691,7 +691,7 @@ Maximum number of input files to scan per sub-process.
691
691
 
692
692
  |Type|Default|Category|
693
693
  |---|---|---|
694
- |Number|40|Subprocess|
694
+ |Number|200|Subprocess|
695
695
 
696
696
  Sets the maximum number of source/binary input files to scan by a single 'compile' sub-process.
697
697
  Set to 0 to unlimited.
@@ -720,7 +720,7 @@ Timeout interval for scanning a source code /binary input file.
720
720
 
721
721
  |Type|Default|Category|
722
722
  |---|---|---|
723
- |String|20s|Parallel|
723
+ |String|30s|Parallel|
724
724
 
725
725
  Sets the timeout interval when scanning a source code /binary input file before dropping it.
726
726
  Set to null to ignore.
@@ -768,9 +768,9 @@ It is a standard programming practice to use string formats to log messages in t
768
768
  log.error("could not connect to {} with status {}", host, status);
769
769
  ```
770
770
 
771
- When an TenXTemplate tokenizer processing an event identifies a format prefix (e.g., `%`)
772
- it skips it and appends subsequent symbol tokens (e.g., `with status`)
773
- to the symbol sequence preceding them (e.g., `could not connect to`).
771
+ The scanner splits the literal at each format prefix (e.g., `%`) into sections
772
+ (`could not connect to`, `with status`) and stores each section; at run time a
773
+ multi-section format matches only when every section is present on the line.
774
774
 
775
775
 
776
776
  #### :material-menu-right-outline:**`maxSymbolUnitsPerToken`**
@@ -14,7 +14,7 @@ logger.error("could not connect to {}", host);
14
14
  cerr << "could not connect to" << host << << std::endl;
15
15
  ```
16
16
 
17
- The string constants receive a 'log' context, unlike non-logging calls:
17
+ The string constants receive a 'log' context. So does every literal in a method that logs, throws or raises (in Java source, Python and Scala, only those preceding the logging or throwing statement), and any literal carrying a format placeholder. A literal in a method that does neither keeps a 'const' context:
18
18
 
19
19
  ```js
20
20
  foo("could not connect to " + host);
@@ -9,7 +9,7 @@ Extracts [symbol](https://doc.log10x.com/run/transform/structure/#symbols) value
9
9
 
10
10
  Parses input by splitting lines with delimiters or using a [JsonFactory](https://fasterxml.github.io/jackson-core/javadoc/2.8/com/fasterxml/jackson/core/JsonFactory.html){target="\_blank"} for structured token reading.
11
11
 
12
- When source code for a log format is unavailable (e.g., third-party services), scan a sample log to extract symbols for future parsing.
12
+ When source code for a log format is unavailable (e.g., third-party services), a sample log can be scanned at compile time as an optional input to extract its symbols. Nothing is learned at run time.
13
13
 
14
14
  !!! note "Size limit"
15
15