log10x-mcp 1.30.39 → 1.30.40
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/build/product-kb/docs/api/js.md +4 -4
- package/build/product-kb/docs/apps/mcp/run.md +5 -4
- package/build/product-kb/docs/apps/mcp/tools/compile/compile-link.md +2 -2
- package/build/product-kb/docs/apps/mcp/tools/compile/compile-status.md +2 -2
- package/build/product-kb/docs/apps/mcp/tools/compile/compile.md +6 -6
- package/build/product-kb/docs/apps/mcp/tools/compile/index.md +1 -1
- package/build/product-kb/docs/apps/receiver/deploy.md +1 -1
- package/build/product-kb/docs/apps/receiver/faq.md +16 -17
- package/build/product-kb/docs/apps/receiver/index.md +2 -2
- package/build/product-kb/docs/apps/receiver/run.md +5 -4
- package/build/product-kb/docs/apps/reporter/run.md +5 -4
- package/build/product-kb/docs/apps/retriever/run.md +5 -4
- package/build/product-kb/docs/compile/faq.md +13 -13
- package/build/product-kb/docs/compile/index.md +1 -1
- package/build/product-kb/docs/compile/link/index.md +2 -1
- package/build/product-kb/docs/compile/pull/index.md +3 -1
- package/build/product-kb/docs/compile/scan/index.md +5 -5
- package/build/product-kb/docs/compile/scanner/logMethods/index.md +1 -1
- package/build/product-kb/docs/compile/scanner/text/index.md +1 -1
- package/build/product-kb/docs/engine/launcher/extension.md +24 -25
- package/build/product-kb/docs/faq/apps/index.md +1 -1
- package/build/product-kb/docs/faq/apps/receiver.md +16 -17
- package/build/product-kb/docs/faq/general.md +5 -5
- package/build/product-kb/docs/faq/pricing/node-counting.md +1 -1
- package/build/product-kb/docs/faq/stacks/aws-cloudwatch.md +4 -4
- package/build/product-kb/docs/faq/stacks/azure-monitor.md +2 -2
- package/build/product-kb/docs/faq/stacks/datadog/comparisons.md +3 -3
- package/build/product-kb/docs/faq/stacks/datadog/integration.md +1 -1
- package/build/product-kb/docs/faq/stacks/elasticsearch/optimization.md +1 -1
- package/build/product-kb/docs/faq/stacks/splunk/index.md +1 -1
- package/build/product-kb/docs/run/initialize/message/index.md +66 -31
- package/build/product-kb/docs/run/receive/rate/index.md +2 -2
- package/build/product-kb/docs/run/transform/structure/index.md +2 -2
- package/build/product-kb/docs/run/transform/symbol/index.md +3 -3
- package/build/tools/configure-engine.d.ts +3 -2
- package/build/tools/configure-engine.js +6 -5
- package/build/tools/configure-engine.js.map +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -20,7 +20,7 @@ This server is open source under MIT. A Log10x product license (API key) unlocks
|
|
|
20
20
|
| "This message type is noise, cut it." | Sample, drop, compact, tier down, or offload a chosen message type, with examples and trend so the action is clear. `log10x_pattern_mitigate`, `log10x_pattern_examples`, `log10x_pattern_trend` |
|
|
21
21
|
| "We are 100% Lambda, no cluster. Install there." | The install plan comes back for the serverless estate: the engine as a Lambda extension paired with the function's OTel collector, a CDK construct, and an AWS-native recurring tick over an S3 config plane. `log10x_advise_install`, `log10x_setup_recurring` |
|
|
22
22
|
| "Get me the actual events I offloaded." | Fetch the exact events back on demand from your own S3. `log10x_retriever_query`, `log10x_offload_add`, `log10x_retriever_register` |
|
|
23
|
-
| "Recognize more of my message types." |
|
|
23
|
+
| "Recognize more of my message types." | Compile your own source code, binaries and container images into a symbol library so the engine names more of your message types. `log10x_compile`, `log10x_compile_link`, `log10x_compile_status` |
|
|
24
24
|
|
|
25
25
|
## Install
|
|
26
26
|
|
|
@@ -1270,7 +1270,7 @@ this.status = this.symbol("ENUM", "Status"); //capture the 'ENABLED value above
|
|
|
1270
1270
|
### .symbolSequence(symbolContexts,fieldName,maxLen) ⇒ <code>string</code>
|
|
1271
1271
|
Returns a sequence of [symbol](https://doc.log10x.com/run/transform/structure/#symbols) tokens of a specified type.
|
|
1272
1272
|
|
|
1273
|
-
This function selects
|
|
1273
|
+
This function selects the symbol and delimiter tokens of the origin ranked highest by the [message](https://doc.log10x.com/run/initialize/message/) comparator from the object's
|
|
1274
1274
|
[TenXTemplate](https://doc.log10x.com/run/template/) based on their context within the source code/binary file from which they originated.
|
|
1275
1275
|
|
|
1276
1276
|
To capture a symbol sequence for the following event:
|
|
@@ -1286,7 +1286,7 @@ this.message = symbolSequence("log,class,exec"); // = 'Partition__not_found_comp
|
|
|
1286
1286
|
```
|
|
1287
1287
|
|
|
1288
1288
|
The `symbolContext` argument controls the [symbol contexts](https://doc.log10x.com/run/transform/symbol/#contexts) to search for;
|
|
1289
|
-
|
|
1289
|
+
the listed contexts are evaluated in a single pass, so list order acts as a filter rather than a precedence chain.
|
|
1290
1290
|
|
|
1291
1291
|
Supported values: package, class, method, log, enum, const, text, exec, any (case insensitive).
|
|
1292
1292
|
|
|
@@ -1342,7 +1342,7 @@ and [time-series](https://doc.log10x.com/run/output/metric/) output.
|
|
|
1342
1342
|
### .symbolOrigin(symbolContext) ⇒ <code>string</code>
|
|
1343
1343
|
Returns a the source code/binary origin of 'symbol' tokens of a specified type.
|
|
1344
1344
|
|
|
1345
|
-
This function selects
|
|
1345
|
+
This function selects the origin (i.e. the source code or binary executable which emitted) the highest-ranked set of [symbol](https://doc.log10x.com/run/transform/structure/#symbols) and delimiter tokens from the object's
|
|
1346
1346
|
TenXTemplate.
|
|
1347
1347
|
|
|
1348
1348
|
To capture the origin value for the following event:
|
|
@@ -1356,7 +1356,7 @@ The following call can be used:
|
|
|
1356
1356
|
this.origin = symbolOrigin("log,class,exec"); // CacheManager.scala'
|
|
1357
1357
|
```
|
|
1358
1358
|
|
|
1359
|
-
The 'symbolContext' argument controls the [symbol contexts](https://doc.log10x.com/run/transform/symbol/#contexts) to search for;
|
|
1359
|
+
The 'symbolContext' argument controls the [symbol contexts](https://doc.log10x.com/run/transform/symbol/#contexts) to search for; the listed contexts are evaluated in a single pass, so list order acts as a filter rather than a precedence chain.
|
|
1360
1360
|
|
|
1361
1361
|
Supported values: PACKAGE, CLASS, METHOD, LOG, ENUM, CONST, TEXT, EXEC (case insensitive).
|
|
1362
1362
|
|
|
@@ -587,8 +587,8 @@ To configure the MCP test-drive app, [:material-cog: Edit](https://doc.log10x.co
|
|
|
587
587
|
contexts: log,exec
|
|
588
588
|
|
|
589
589
|
# 'messageField' specifies the field name to assign the message pattern to target TenXObjects: the symbol
|
|
590
|
-
# tokens of the origin ranked highest
|
|
591
|
-
#
|
|
590
|
+
# tokens of the origin ranked highest on the five-key comparator, message-text width first. Span length
|
|
591
|
+
# is the last tiebreak. The origin file name itself goes to 'originField'.
|
|
592
592
|
messageField: message_pattern
|
|
593
593
|
|
|
594
594
|
# 'messageHashField' specifies the field name to assign a stable, URL-safe pattern hash (xxHash64 of the messageField value, base64url, 11 chars). Safe as a SIEM query term or forwarder filter key.
|
|
@@ -610,8 +610,9 @@ To configure the MCP test-drive app, [:material-cog: Edit](https://doc.log10x.co
|
|
|
610
610
|
# indented detail, banners) so the message-template skips its own message-pattern compute for them.
|
|
611
611
|
# The check fires only for templates that don't already pass the (groupSize > 1) || groupHead arms;
|
|
612
612
|
# i.e., for single-event templates with no head signal. A non-negator line in that case (e.g., bare
|
|
613
|
-
# "Hello World", "Heartbeat received from node-7")
|
|
614
|
-
# (e.g., a lone "\tat com.foo.Bar(...)")
|
|
613
|
+
# "Hello World", "Heartbeat received from node-7") gets a pattern from its library words; a negator-matching
|
|
614
|
+
# orphan (e.g., a lone "\tat com.foo.Bar(...)") skips origin selection and is named by a hash of its shape,
|
|
615
|
+
# template_<hash>.
|
|
615
616
|
message:
|
|
616
617
|
negators:
|
|
617
618
|
- " " # any indented line (stack frames, indented app detail, multi-line JSON, banners)
|
|
@@ -36,7 +36,7 @@ A very large units tree returns a running `job_id` to poll with [`log10x_compile
|
|
|
36
36
|
|
|
37
37
|
## :material-check-decagram-outline: Prerequisites
|
|
38
38
|
|
|
39
|
-
The Compiler flavor, docker-first: `auto` and `docker` pull `log10x/compiler-10x:1.1.
|
|
39
|
+
The Compiler flavor, docker-first: `auto` and `docker` pull `log10x/compiler-10x:1.1.136` (pinned by digest), and `local` runs a Compiler-flavor `tenx`. The Runtime flavor is refused; the `mode` argument of [`log10x_compile`](compile.md) lists the three flavors.
|
|
40
40
|
|
|
41
41
|
A link into a folder another compile or link job is still writing joins that job instead of starting a second one.
|
|
42
42
|
|
|
@@ -138,7 +138,7 @@ This is the CLI tier, so it needs no Kubernetes, deployed app, or Log10x account
|
|
|
138
138
|
"job_id": "7b2e9c41-3f5a-4d8e-9b21-6c0f1a2d7e44",
|
|
139
139
|
"job_status": "completed",
|
|
140
140
|
"mode": "docker",
|
|
141
|
-
"image": "log10x/compiler-10x:1.1.
|
|
141
|
+
"image": "log10x/compiler-10x:1.1.136@sha256:ffcd9890935804b06fcb1791994dfbb32cdb270f3e35fc423ede78ced2097b1f",
|
|
142
142
|
"compiler": null,
|
|
143
143
|
"exit_code": 0,
|
|
144
144
|
"elapsed_ms": 6240,
|
|
@@ -149,8 +149,8 @@ The per-language scan failures and the symbol-type counts need a `compiler-10x`
|
|
|
149
149
|
"job_id": "compile-7c2f",
|
|
150
150
|
"job_status": "completed",
|
|
151
151
|
"mode": "docker",
|
|
152
|
-
"image": "log10x/compiler-10x:1.1.
|
|
153
|
-
"compiler": "docker:log10x/compiler-10x:1.1.
|
|
152
|
+
"image": "log10x/compiler-10x:1.1.136@sha256:ffcd9890935804b06fcb1791994dfbb32cdb270f3e35fc423ede78ced2097b1f",
|
|
153
|
+
"compiler": "docker:log10x/compiler-10x:1.1.136@sha256:ffcd9890935804b06fcb1791994dfbb32cdb270f3e35fc423ede78ced2097b1f",
|
|
154
154
|
"exit_code": 0,
|
|
155
155
|
"elapsed_ms": 184220,
|
|
156
156
|
"timed_out": false,
|
|
@@ -22,7 +22,7 @@ Sources combine freely in one call:
|
|
|
22
22
|
|
|
23
23
|
Docker pulls are daemonless. GitHub and Artifactory pulls each need a token, even for public GitHub repos.
|
|
24
24
|
|
|
25
|
-
The [Compiler app](../../../compiler/) runs the Compiler flavor, docker-first via the `log10x/compiler-10x:1.1.
|
|
25
|
+
The [Compiler app](../../../compiler/) runs the Compiler flavor, docker-first via the `log10x/compiler-10x:1.1.136` image (pinned by digest) or a local Compiler-flavor `tenx`. It runs without Kubernetes, a deployed app, or a Log10x account.
|
|
26
26
|
|
|
27
27
|
What gets scanned, per the shipped scanner config:
|
|
28
28
|
|
|
@@ -38,7 +38,7 @@ The output folder is keyed on the sources, the library name, and the compiler (t
|
|
|
38
38
|
|
|
39
39
|
What a re-run skips: the engine skips every local file whose checksum already has a unit in the folder. Pulled sources cost a remote round-trip each run (GitHub branch head, image manifest digest, Artifactory listing) and are downloaded again only when the remote changed; Helm charts are rendered again every run. Only the scan of unchanged files is skipped.
|
|
40
40
|
|
|
41
|
-
The engine's own whole-scan cap (`scanOperationTimeout`, bundled default 10 minutes) is raised to 90% of `timeout_ms` on the command line, so the engine cap follows `timeout_ms`. That needs compiler 1.1.89 or newer; the pinned default is 1.1.
|
|
41
|
+
The engine's own whole-scan cap (`scanOperationTimeout`, bundled default 10 minutes) is raised to 90% of `timeout_ms` on the command line, so the engine cap follows `timeout_ms`. That needs compiler 1.1.89 or newer; the pinned default is 1.1.136, and an older local engine keeps its 10 minute cap. A run the engine stopped early is reported as `incomplete` by [Status](compile-status.md).
|
|
42
42
|
|
|
43
43
|
## :material-code-braces: Example
|
|
44
44
|
|
|
@@ -208,14 +208,14 @@ Archives are scanned as-is: `.jar`, `.war`, `.ear`, `.zip` and tarballs are expa
|
|
|
208
208
|
"local"
|
|
209
209
|
],
|
|
210
210
|
"default": "auto",
|
|
211
|
-
"description": "Execution backend. `auto` (default) prefers Docker (the compiler image, guaranteed compiler flavor) and falls back to a local compiler-flavor tenx; the backend is chosen once, before the output folder is keyed. `docker` forces the image (LOG10X_COMPILER_IMAGE or LOG10X_TENX_IMAGE, default log10x/compiler-10x:1.1.
|
|
211
|
+
"description": "Execution backend. `auto` (default) prefers Docker (the compiler image, guaranteed compiler flavor) and falls back to a local compiler-flavor tenx; the backend is chosen once, before the output folder is keyed. `docker` forces the image (LOG10X_COMPILER_IMAGE or LOG10X_TENX_IMAGE, default log10x/compiler-10x:1.1.136 pinned by digest, so every run goes through the same compiler). `local` forces the binary (LOG10X_TENX_PATH or `tenx` on PATH) and refuses it unless its version banner reports the compiler flavor (`compiler`, or `cloud` on an engine built before the flavor rename). Of the three shipped flavors only `compiler` can compile: `runtime` (native binary) and `runtime-jvm` (JVM-packaged runtime, the only runtime available on Windows) are both refused, since neither carries the `generate` pipeline unit. With a local install, local-folder compilation and GitHub pull (REST API + token) work out of the box; docker_images pull additionally needs a container engine (podman or docker) on the host. The docker `compiler-10x` image bundles all of those (podman included, daemonless), which is why Docker is the default."
|
|
212
212
|
},
|
|
213
213
|
"timeout_ms": {
|
|
214
214
|
"type": "integer",
|
|
215
215
|
"minimum": 10000,
|
|
216
216
|
"maximum": 3600000,
|
|
217
217
|
"default": 1800000,
|
|
218
|
-
"description": "Hard cap on compile wall time in milliseconds. Default 1,800,000 (30 min). The engine's own whole-scan cap (scanOperationTimeout, bundled default 10 min) is raised to 90% of this value on the command line, so the engine cap follows this value (needs compiler 1.1.89+; the pinned default is 1.1.
|
|
218
|
+
"description": "Hard cap on compile wall time in milliseconds. Default 1,800,000 (30 min). The engine's own whole-scan cap (scanOperationTimeout, bundled default 10 min) is raised to 90% of this value on the command line, so the engine cap follows this value (needs compiler 1.1.89+; the pinned default is 1.1.136; an older local engine keeps its 10 min cap). A re-run over unchanged local sources skips every file whose checksum already has a unit and finishes in seconds. Pulled sources (GitHub, images, Artifactory) are re-checked remotely each run and downloaded again only when the remote changed; Helm charts are rendered again every run."
|
|
219
219
|
},
|
|
220
220
|
"max_wait_ms": {
|
|
221
221
|
"type": "integer",
|
|
@@ -264,8 +264,8 @@ Archives are scanned as-is: `.jar`, `.war`, `.ear`, `.zip` and tarballs are expa
|
|
|
264
264
|
"job_id": "9f2c1a3e-7b4d-4e1a-9f0c-2a1b3c4d5e6f",
|
|
265
265
|
"job_status": "completed",
|
|
266
266
|
"mode": "docker",
|
|
267
|
-
"image": "log10x/compiler-10x:1.1.
|
|
268
|
-
"compiler": "docker:log10x/compiler-10x:1.1.
|
|
267
|
+
"image": "log10x/compiler-10x:1.1.136@sha256:ffcd9890935804b06fcb1791994dfbb32cdb270f3e35fc423ede78ced2097b1f",
|
|
268
|
+
"compiler": "docker:log10x/compiler-10x:1.1.136@sha256:ffcd9890935804b06fcb1791994dfbb32cdb270f3e35fc423ede78ced2097b1f",
|
|
269
269
|
"exit_code": 0,
|
|
270
270
|
"elapsed_ms": 38120,
|
|
271
271
|
"timed_out": false,
|
|
@@ -52,7 +52,7 @@ To re-link an existing units tree into a library with no source scan, use [`log1
|
|
|
52
52
|
|
|
53
53
|
## :material-check-decagram-outline: Prerequisites
|
|
54
54
|
|
|
55
|
-
The compile family runs the [Compiler app](../../../compiler/), which executes the [compile pipeline](../../../../compile/), docker-first. By default it pulls `log10x/compiler-10x:1.1.
|
|
55
|
+
The compile family runs the [Compiler app](../../../compiler/), which executes the [compile pipeline](../../../../compile/), docker-first. By default it pulls `log10x/compiler-10x:1.1.136`, pinned by digest, so every run goes through the same compiler.
|
|
56
56
|
|
|
57
57
|
That image bundles the following, so no host docker socket is needed:
|
|
58
58
|
|
|
@@ -2179,7 +2179,7 @@ The OTel Collector overlay moves the sidecar's OTLP listener with `TENX_OTEL_COL
|
|
|
2179
2179
|
|
|
2180
2180
|
**Phase C: processed events flowing**
|
|
2181
2181
|
|
|
2182
|
-
The receiver
|
|
2182
|
+
The receiver hands each record back byte-for-byte unless `symbolMessageHashField` names a field (the forwarder args above). Name one, for example `symbolMessageHashField tenx_hash`, and the receiver adds that field to every record it hands back, so the field arriving at the destination is the proof that events completed the round trip through 10x. For a real destination, query for the field there: Elasticsearch `exists: tenx_hash`, Splunk `tenx_hash=*`, Datadog `@tenx_hash:*`. For the stdout / `debug` placeholder destination from Step 3, count it on the container that writes to the destination. Pick your forwarder:
|
|
2183
2183
|
|
|
2184
2184
|
=== ":simple-fluentd: Fluentd"
|
|
2185
2185
|
|
|
@@ -3,7 +3,7 @@ icon: material/help-circle-outline
|
|
|
3
3
|
title: "FAQ"
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
The [Receiver](https://doc.log10x.com/apps/receiver/) is the execution arm of the 10x pipeline. The rate regulator finds which patterns are over the byte cap set for their service, and an action decides what happens to that excess. The keep-everything actions (compact, offload, tier_down) cut cost without losing data; sample and drop are lossy
|
|
6
|
+
The [Receiver](https://doc.log10x.com/apps/receiver/) is the execution arm of the 10x pipeline. The rate regulator finds which patterns are over the byte cap set for their service, and an action decides what happens to that excess. The keep-everything actions (compact, offload, tier_down) cut cost without losing data; sample and drop, the Receiver's Filter modes, are lossy: they apply only to the share of a pattern over a cap the operator sets, and that share is dropped unless the operator chooses another action; pass leaves the pattern untouched. An AI agent picks the action per service through the log10x MCP (the `configure_engine` tool); the engine enforces it, and the decision travels as a config change through the GitOps repo. The Receiver runs as a sidecar alongside your forwarder.
|
|
7
7
|
|
|
8
8
|
<div class="grid cards" markdown>
|
|
9
9
|
|
|
@@ -27,31 +27,30 @@ The [Receiver](https://doc.log10x.com/apps/receiver/) is the execution arm of th
|
|
|
27
27
|
- **offload**: route the pattern to customer-owned object storage (S3 or any S3-compatible bucket) instead of the destination.
|
|
28
28
|
- **drop**: stop forwarding the pattern.
|
|
29
29
|
|
|
30
|
-
An AI agent picks the action per service through the log10x MCP (the `configure_engine` tool), which turns a target percent or budget into
|
|
30
|
+
An AI agent picks the action per service through the log10x MCP (the `configure_engine` tool), which turns a target percent or budget into per-service caps and actions carried as cap and action CSV files. The files land in the config repo through a GitOps PR and hot-reload on the next pull. Per-pattern rates come from the [mute file](https://doc.log10x.com/run/receive/rate/#protection-list){target="_blank"}, a CSV keyed by pattern, committed to git and pulled the same way.
|
|
31
31
|
|
|
32
32
|
Log10x is normally driven by an AI agent (Claude, or a model the customer brings) through the log10x MCP server, which installs, configures, and queries via MCP tools. The manual steps in these docs are the same operations without an agent.
|
|
33
33
|
|
|
34
34
|
??? tenx-overview "How do budget policies work"
|
|
35
35
|
|
|
36
|
-
Configuration is a YAML file, written by hand or generated by the MCP `configure_engine` tool and delivered through GitOps. The [rate receiver](https://doc.log10x.com/run/receive/rate/){target="_blank"}
|
|
36
|
+
Configuration is a YAML file, written by hand or generated by the MCP `configure_engine` tool and delivered through GitOps. The [rate receiver](https://doc.log10x.com/run/receive/rate/){target="_blank"} caps the bytes each pattern ([symbol identity](https://doc.log10x.com/run/initialize/message/){target="_blank"}) may spend in its container over a rolling window. You configure:
|
|
37
37
|
|
|
38
|
-
- **
|
|
39
|
-
- **
|
|
40
|
-
- **Severity floors**,
|
|
38
|
+
- **A byte cap per pattern per container**, fleet-wide or per container from a cap file; the MCP derives it from a monthly dollar budget
|
|
39
|
+
- **A share guard**, a pattern below a minimum share of its container's volume (default 5%) is left alone
|
|
40
|
+
- **Severity floors**, an over-cap pattern keeps at least a set share of each level (by default ERROR 50%, WARN 30%, INFO 10%)
|
|
41
|
+
- **An action per service** for the excess: drop (the default), offload, tier_down, compact, sample or pass
|
|
41
42
|
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
When an event type exceeds its max share, the Receiver samples it down proportionally. For [multi-app environments](https://doc.log10x.com/run/receive/rate/#multi-app-regulation){target="_blank"} (Kubernetes), cap per-app budgets using the [container name](https://doc.log10x.com/run/initialize/k8s/){target="_blank"} field, scaling replicas doesn't bypass limits.
|
|
43
|
+
Cost is reported per pattern; the cap itself is in bytes. Caps are scoped by the [container name](https://doc.log10x.com/run/receive/rate/#containers){target="_blank"}, so scaling replicas doesn't bypass them.
|
|
45
44
|
|
|
46
45
|
??? tenx-overview "What happens to logs when budget limits are reached (sample action)"
|
|
47
46
|
|
|
48
|
-
The [rate receiver](https://doc.log10x.com/run/receive/rate/){target="_blank"}
|
|
47
|
+
The [rate receiver](https://doc.log10x.com/run/receive/rate/){target="_blank"} decides per event, on bytes:
|
|
49
48
|
|
|
50
|
-
- **
|
|
51
|
-
- **
|
|
52
|
-
- **Severity floors**, ERROR events keep a higher
|
|
49
|
+
- **At or below the pattern's cap**, every event flows through
|
|
50
|
+
- **Over the cap**, each level keeps its floor share and the rest takes the container's action (drop when none is set); nothing targets the excess back down to the cap line
|
|
51
|
+
- **Severity floors**, ERROR events keep a higher share of the over-cap slice than DEBUG noise
|
|
53
52
|
|
|
54
|
-
The result: noisy
|
|
53
|
+
The result: a noisy pattern is throttled while its errors keep flowing. The Receiver exports [metrics](https://doc.log10x.com/run/output/metric/){target="_blank"} tracking what each pattern's action did, so policies can be tuned over time. Routed-away events carry the label `routeState` (for example `routeState="drop"`).
|
|
55
54
|
|
|
56
55
|
### :material-cog-outline: Configuration
|
|
57
56
|
|
|
@@ -69,10 +68,10 @@ The [Receiver](https://doc.log10x.com/apps/receiver/) is the execution arm of th
|
|
|
69
68
|
|
|
70
69
|
The [rate receiver](https://doc.log10x.com/run/receive/rate/){target="_blank"} uses two mechanisms that naturally protect critical events:
|
|
71
70
|
|
|
72
|
-
- **Severity floors**, ERROR and WARN events keep a higher minimum share
|
|
73
|
-
- **
|
|
71
|
+
- **Severity floors**, ERROR and WARN events keep a higher minimum share of the over-cap slice. Even when their pattern is over its cap, critical-severity events keep flowing at a higher rate than DEBUG noise
|
|
72
|
+
- **Per-pattern caps**, the regulator engages only on a [pattern](https://doc.log10x.com/run/initialize/message/){target="_blank"} over its byte cap and above the share guard (default 5% of its container's volume). Low-volume patterns, which security and authentication logs typically are, stay under their cap and pass through unaffected
|
|
74
73
|
|
|
75
|
-
The Receiver targets noisy, high-volume
|
|
74
|
+
The Receiver targets noisy, high-volume patterns, not broad categories. Security events that stay under their cap flow through untouched.
|
|
76
75
|
|
|
77
76
|
??? tenx-capabilities "How do I monitor budget consumption"
|
|
78
77
|
|
|
@@ -7,7 +7,7 @@ icon: "material/pipe-valve"
|
|
|
7
7
|
---
|
|
8
8
|
The Receiver runs as a forwarder sidecar and acts on each pattern before events reach your analyzer.
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
Its actions are **pass, sample, compact, tier_down, offload, and drop**. The action is set per service and applies to every pattern over its cap in that service. An AI agent sets the action per service through the [log10x MCP](https://doc.log10x.com/apps/mcp/) (`configure_engine`); the engine enforces it, carried as a config change through the [GitOps](https://doc.log10x.com/engine/gitops/) repo.
|
|
11
11
|
|
|
12
12
|
## :material-cog-transfer-outline: Workflow
|
|
13
13
|
|
|
@@ -71,7 +71,7 @@ The Receiver executes as a [forwarder sidecar](https://doc.log10x.com/engine/lau
|
|
|
71
71
|
|
|
72
72
|
=== ":material-file-document-edit-outline: Mute File (GitOps)"
|
|
73
73
|
|
|
74
|
-
A declarative file keyed by the joined `rateReceiverFieldNames` values (e.g. `symbolMessage`, `container`) caps specific patterns with an explicit sample rate and expiry. Operators (or an AI assistant via the [Log10x MCP](https://github.com/log-10x/log10x-mcp)) append entries based on [Reporter](https://doc.log10x.com/apps/reporter/) cost attribution, commit to git, and every receiver pulls the file on its next reload. Each mute is diff-reviewed, self-expires, and maps 1:1 to the field-sets the Reporter attributes cost to. See [mute file mode](https://doc.log10x.com/run/receive/rate/#
|
|
74
|
+
A declarative file keyed by the joined `rateReceiverFieldNames` values (e.g. `symbolMessage`, `container`) caps specific patterns with an explicit sample rate and expiry. Operators (or an AI assistant via the [Log10x MCP](https://github.com/log-10x/log10x-mcp)) append entries based on [Reporter](https://doc.log10x.com/apps/reporter/) cost attribution, commit to git, and every receiver pulls the file on its next reload. Each mute is diff-reviewed, self-expires, and maps 1:1 to the field-sets the Reporter attributes cost to. See [mute file mode](https://doc.log10x.com/run/receive/rate/#protection-list).
|
|
75
75
|
|
|
76
76
|
## :material-shield-check-outline: Safety & Reliability
|
|
77
77
|
|
|
@@ -1303,8 +1303,8 @@ To configure the Receiver app, [:material-cog: Edit](https://doc.log10x.com/conf
|
|
|
1303
1303
|
contexts: log,exec
|
|
1304
1304
|
|
|
1305
1305
|
# 'messageField' specifies the field name to assign the message pattern to target TenXObjects: the symbol
|
|
1306
|
-
# tokens of the origin ranked highest
|
|
1307
|
-
#
|
|
1306
|
+
# tokens of the origin ranked highest on the five-key comparator, message-text width first. Span length
|
|
1307
|
+
# is the last tiebreak. The origin file name itself goes to 'originField'.
|
|
1308
1308
|
messageField: message_pattern
|
|
1309
1309
|
|
|
1310
1310
|
# 'messageHashField' specifies the field name to assign a stable, URL-safe pattern hash (xxHash64 of the messageField value, base64url, 11 chars). Safe as a SIEM query term or forwarder filter key.
|
|
@@ -1326,8 +1326,9 @@ To configure the Receiver app, [:material-cog: Edit](https://doc.log10x.com/conf
|
|
|
1326
1326
|
# indented detail, banners) so the message-template skips its own message-pattern compute for them.
|
|
1327
1327
|
# The check fires only for templates that don't already pass the (groupSize > 1) || groupHead arms;
|
|
1328
1328
|
# i.e., for single-event templates with no head signal. A non-negator line in that case (e.g., bare
|
|
1329
|
-
# "Hello World", "Heartbeat received from node-7")
|
|
1330
|
-
# (e.g., a lone "\tat com.foo.Bar(...)")
|
|
1329
|
+
# "Hello World", "Heartbeat received from node-7") gets a pattern from its library words; a negator-matching
|
|
1330
|
+
# orphan (e.g., a lone "\tat com.foo.Bar(...)") skips origin selection and is named by a hash of its shape,
|
|
1331
|
+
# template_<hash>.
|
|
1331
1332
|
message:
|
|
1332
1333
|
negators:
|
|
1333
1334
|
- " " # any indented line (stack frames, indented app detail, multi-line JSON, banners)
|
|
@@ -826,8 +826,8 @@ To configure the Reporter app, [:material-cog: Edit](https://doc.log10x.com/conf
|
|
|
826
826
|
contexts: log,exec
|
|
827
827
|
|
|
828
828
|
# 'messageField' specifies the field name to assign the message pattern to target TenXObjects: the symbol
|
|
829
|
-
# tokens of the origin ranked highest
|
|
830
|
-
#
|
|
829
|
+
# tokens of the origin ranked highest on the five-key comparator, message-text width first. Span length
|
|
830
|
+
# is the last tiebreak. The origin file name itself goes to 'originField'.
|
|
831
831
|
messageField: message_pattern
|
|
832
832
|
|
|
833
833
|
# 'messageHashField' specifies the field name to assign a stable, URL-safe pattern hash (xxHash64 of the messageField value, base64url, 11 chars). Safe as a SIEM query term or forwarder filter key.
|
|
@@ -849,8 +849,9 @@ To configure the Reporter app, [:material-cog: Edit](https://doc.log10x.com/conf
|
|
|
849
849
|
# indented detail, banners) so the message-template skips its own message-pattern compute for them.
|
|
850
850
|
# The check fires only for templates that don't already pass the (groupSize > 1) || groupHead arms;
|
|
851
851
|
# i.e., for single-event templates with no head signal. A non-negator line in that case (e.g., bare
|
|
852
|
-
# "Hello World", "Heartbeat received from node-7")
|
|
853
|
-
# (e.g., a lone "\tat com.foo.Bar(...)")
|
|
852
|
+
# "Hello World", "Heartbeat received from node-7") gets a pattern from its library words; a negator-matching
|
|
853
|
+
# orphan (e.g., a lone "\tat com.foo.Bar(...)") skips origin selection and is named by a hash of its shape,
|
|
854
|
+
# template_<hash>.
|
|
854
855
|
message:
|
|
855
856
|
negators:
|
|
856
857
|
- " " # any indented line (stack frames, indented app detail, multi-line JSON, banners)
|
|
@@ -1373,8 +1373,8 @@ To configure the Retriever app, [:material-cog: Edit](https://doc.log10x.com/con
|
|
|
1373
1373
|
contexts: log,exec
|
|
1374
1374
|
|
|
1375
1375
|
# 'messageField' specifies the field name to assign the message pattern to target TenXObjects: the symbol
|
|
1376
|
-
# tokens of the origin ranked highest
|
|
1377
|
-
#
|
|
1376
|
+
# tokens of the origin ranked highest on the five-key comparator, message-text width first. Span length
|
|
1377
|
+
# is the last tiebreak. The origin file name itself goes to 'originField'.
|
|
1378
1378
|
messageField: message_pattern
|
|
1379
1379
|
|
|
1380
1380
|
# 'messageHashField' specifies the field name to assign a stable, URL-safe pattern hash (xxHash64 of the messageField value, base64url, 11 chars). Safe as a SIEM query term or forwarder filter key.
|
|
@@ -1396,8 +1396,9 @@ To configure the Retriever app, [:material-cog: Edit](https://doc.log10x.com/con
|
|
|
1396
1396
|
# indented detail, banners) so the message-template skips its own message-pattern compute for them.
|
|
1397
1397
|
# The check fires only for templates that don't already pass the (groupSize > 1) || groupHead arms;
|
|
1398
1398
|
# i.e., for single-event templates with no head signal. A non-negator line in that case (e.g., bare
|
|
1399
|
-
# "Hello World", "Heartbeat received from node-7")
|
|
1400
|
-
# (e.g., a lone "\tat com.foo.Bar(...)")
|
|
1399
|
+
# "Hello World", "Heartbeat received from node-7") gets a pattern from its library words; a negator-matching
|
|
1400
|
+
# orphan (e.g., a lone "\tat com.foo.Bar(...)") skips origin selection and is named by a hash of its shape,
|
|
1401
|
+
# template_<hash>.
|
|
1401
1402
|
message:
|
|
1402
1403
|
negators:
|
|
1403
1404
|
- " " # any indented line (stack frames, indented app detail, multi-line JSON, banners)
|
|
@@ -26,13 +26,13 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
|
|
|
26
26
|
|
|
27
27
|
??? tenx-security "What information does the compiler generate"
|
|
28
28
|
|
|
29
|
-
The compiler extracts **symbol values**, tokens that appear in log statements, such as string literals, enum values, and format patterns. Extracted string constants, from log statement formats (e.g., `"error in %s"`), [binary executables](https://doc.log10x.com/compile/scanner/executable/) (via `strings`), or [text/JSON/YAML files](https://doc.log10x.com/compile/scanner/text/), are stored as **64-bit hashes** in the symbol library. The original plain text is never written to the output. The runtime engine matches incoming events against these hashes, not against your source strings.
|
|
29
|
+
The compiler extracts **symbol values**, tokens that appear in log statements, such as string literals, enum values, and format patterns. Extracted string constants, from log statement formats (e.g., `"error in %s"`), [binary executables](https://doc.log10x.com/compile/scanner/executable/) (via `strings`), or [text/JSON/YAML files](https://doc.log10x.com/compile/scanner/text/), are stored as **64-bit hashes** in the symbol library. The original plain text of a string is never written to the output; file names and class, method and enum names are. The runtime engine matches incoming events against these hashes, not against your source strings.
|
|
30
30
|
|
|
31
31
|
Class and method names are stored alongside hashes to identify the source of each log statement, comparable to what appears in a stack trace.
|
|
32
32
|
|
|
33
33
|
The compiler does **not** extract application logic, control flow, full source file contents, or environment variables. It does not read `.env` files or runtime state.
|
|
34
34
|
|
|
35
|
-
Output is a [symbol library](https://doc.log10x.com/compile/link/#symbol-library) file (`.10x.tar`) containing
|
|
35
|
+
Output is a [symbol library](https://doc.log10x.com/compile/link/#symbol-library) file (`.10x.tar`) containing hashes and the file names they came from. You can inspect the contents before deploying (`tar tf library.10x.tar`).
|
|
36
36
|
|
|
37
37
|
??? tenx-security "What permissions does the compiler need, and how do I scope access"
|
|
38
38
|
|
|
@@ -45,7 +45,7 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
|
|
|
45
45
|
| [Helm charts](https://doc.log10x.com/compile/pull/helm/) | Same as GitHub + Docker | Resolves chart image references, then pulls each image |
|
|
46
46
|
| [Artifactory](https://doc.log10x.com/compile/pull/artifactory/) | `ARTIFACTORY_TOKEN` | Read access to target repositories |
|
|
47
47
|
|
|
48
|
-
You control exactly which repos and images the compiler accesses via your [configuration](https://doc.log10x.com/compile/test/#config-files). The compiler only pulls sources you explicitly list,
|
|
48
|
+
You control exactly which repos and images the compiler accesses via your [configuration](https://doc.log10x.com/compile/test/#config-files). The compiler only pulls sources you explicitly list, and the images and repositories those sources reference (a Helm chart's images, an image's source repository, a `go.mod` file's dependencies).
|
|
49
49
|
|
|
50
50
|
All processing runs inside your infrastructure (CI/CD pipeline or k8s CronJob). No source code or symbol data is sent to Log10x or any external service.
|
|
51
51
|
|
|
@@ -73,7 +73,7 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
|
|
|
73
73
|
|
|
74
74
|
??? tenx-info "Do I need to run the compiler, or do the built-in defaults cover my stack"
|
|
75
75
|
|
|
76
|
-
The 10x runtime ships with a [default symbol library](https://doc.log10x.com/compile/pull/#default-symbols) covering
|
|
76
|
+
The 10x runtime ships with a [default symbol library](https://doc.log10x.com/compile/pull/#default-symbols) covering the open-source repositories and public container images listed in the manifest shipped inside the library archive, across Java, Python, Node.js, Go, C++, Rust, Ruby, .NET, and Scala, including Kubernetes, OpenTelemetry, Spring Boot, Django, Express, nginx, PostgreSQL, Redis, Kafka, and many more.
|
|
77
77
|
|
|
78
78
|
**Start with the defaults.** Run a local cost POC on your own log files to see how much the built-in library covers. The runtime reports which events matched known symbols and which did not.
|
|
79
79
|
|
|
@@ -95,7 +95,7 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
|
|
|
95
95
|
|
|
96
96
|
??? tenx-coverage "My stack uses common open-source frameworks, do I need to run the compiler at all"
|
|
97
97
|
|
|
98
|
-
The 10x runtime ships with a [default symbol library](https://doc.log10x.com/compile/pull/#default-symbols) covering
|
|
98
|
+
The 10x runtime ships with a [default symbol library](https://doc.log10x.com/compile/pull/#default-symbols) covering the repositories and container images listed in the manifest shipped inside the library archive. If your stack is covered by the defaults, you don't need to run the compiler.
|
|
99
99
|
|
|
100
100
|
Run the compiler to **add custom symbols** for your proprietary application code on top of the defaults. This improves template accuracy for your specific log formats. The compiler output supplements the default library, not replaces it.
|
|
101
101
|
|
|
@@ -107,23 +107,23 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
|
|
|
107
107
|
|
|
108
108
|
1. **[Docker image scanning](https://doc.log10x.com/compile/pull/docker/)**, Pull any Docker image from any registry (public or private), including closed-source vendor agents. The compiler exports and scans the image filesystem for symbol content. Images are only re-pulled when their SHA256 changes, making incremental runs efficient. Works for `nginx:latest`, Datadog agent images, Splunk UF, or any vendor-provided image.
|
|
109
109
|
|
|
110
|
-
2. **[Executable scanning](https://doc.log10x.com/compile/scanner/executable/)**, Runs the OS `strings` utility on compiled binaries (`.so`, `.dll`, `.dynlib`) to extract symbol values. Runs inside your infrastructure; output is
|
|
110
|
+
2. **[Executable scanning](https://doc.log10x.com/compile/scanner/executable/)**, Runs the OS `strings` utility on compiled binaries (`.so`, `.dll`, `.dynlib`) to extract symbol values. Runs inside your infrastructure; output is the binary's file name plus 64-bit hashes of its strings (no binary content leaves your network).
|
|
111
111
|
|
|
112
112
|
3. **JIT fallback**, For truly opaque sources (AWS-managed service logs, SaaS tools with no Docker image), the JIT runtime handles them automatically at runtime. The engine creates a template on the first event seen, giving ~80% efficiency without any compiler setup needed.
|
|
113
113
|
|
|
114
114
|
??? tenx-coverage "Do you support custom and proprietary log formats, or just the default library"
|
|
115
115
|
|
|
116
|
-
**Both.** The default library covers common frameworks, but the compiler lets you add custom symbols for proprietary logging patterns. Custom formats work out of the box with
|
|
116
|
+
**Both.** The default library covers common frameworks, but the compiler lets you add custom symbols for proprietary logging patterns. Custom formats work out of the box with the built-in library.
|
|
117
117
|
|
|
118
|
-
**How it works:** The runtime [tokenizes](https://doc.log10x.com/run/transform/structure/) incoming events and matches tokens against the symbol library. With
|
|
118
|
+
**How it works:** The runtime [tokenizes](https://doc.log10x.com/run/transform/structure/) incoming events and matches tokens against the symbol library. With the built-in library, most generic tokens (class names, log levels, method names) already match; timestamps are [variables](https://doc.log10x.com/run/transform/structure/#variables), not library words. For logs from your proprietary services, the runtime still processes them, but template inference is less precise.
|
|
119
119
|
|
|
120
|
-
**When to optimize:** If you want accurate [message extraction](https://doc.log10x.com/run/initialize/message/) for cost attribution and metrics, capturing the intended "essence" of each custom event type, run the compiler on your custom code. This generates additional symbols, so the runtime recognizes your proprietary patterns and builds more accurate templates. The compiler is iterative: initial
|
|
120
|
+
**When to optimize:** If you want accurate [message extraction](https://doc.log10x.com/run/initialize/message/) for cost attribution and metrics, capturing the intended "essence" of each custom event type, run the compiler on your custom code. This generates additional symbols, so the runtime recognizes your proprietary patterns and builds more accurate templates. The compiler is iterative: an initial run scans all of your code; later runs scan only what changed. Subsequent runs only process changed files, completing in seconds, perfect for CI/CD pipelines. You point it at your repositories once, then it automatically supplements the default symbol library on each deploy.
|
|
121
121
|
|
|
122
122
|
See [real-world before/after examples](https://doc.log10x.com/apps/receiver/){target="_blank"} showing how symbol coverage improves lossless volume reduction on production logs.
|
|
123
123
|
|
|
124
124
|
??? tenx-coverage "My custom format isn't in the library, does it still work"
|
|
125
125
|
|
|
126
|
-
Yes. The engine creates a template for every event type it encounters, there are no "unrecognized" events. The [symbol library](https://doc.log10x.com/run/symbol/) provides vocabulary that determines how much of each event is classified as structure (low-cardinality, stored in the template) vs variable data (high-cardinality, shipped per event). The [default library](https://doc.log10x.com/compile/pull/#default-symbols) covers
|
|
126
|
+
Yes. The engine creates a template for every event type it encounters, there are no "unrecognized" events. The [symbol library](https://doc.log10x.com/run/symbol/) provides vocabulary that determines how much of each event is classified as structure (low-cardinality, stored in the template) vs variable data (high-cardinality, shipped per event). The [default library](https://doc.log10x.com/compile/pull/#default-symbols) covers the sources listed in its manifest, each pinned to a commit or image digest, and delivers strong reduction on most workloads.
|
|
127
127
|
|
|
128
128
|
Running the compiler on custom application code enriches the vocabulary further, producing richer templates, more accurate [message extraction](https://doc.log10x.com/run/initialize/message/), and higher reduction ratios. Without it, the format still works using the default vocabulary, with no configuration or manual regex rules.
|
|
129
129
|
|
|
@@ -131,7 +131,7 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
|
|
|
131
131
|
|
|
132
132
|
??? tenx-operations "How often do I need to run the compiler, and what is the maintenance overhead"
|
|
133
133
|
|
|
134
|
-
Minimal. Initial runs take a few minutes; incremental runs on unchanged code complete in seconds. Deploy as a [k8s CronJob](https://doc.log10x.com/compile/deploy/)
|
|
134
|
+
Minimal. Initial runs take a few minutes; incremental runs on unchanged code complete in seconds. Deploy as a [k8s CronJob](https://doc.log10x.com/compile/deploy/) on a schedule you choose, or trigger from CI/CD on commits. A rebuilt library can rename the affected patterns once, so a schedule is a choice with that consequence. To pin the library, load it from a branch you update deliberately, or set `syncInterval` to null in the [GitHub](https://doc.log10x.com/config/github/) pull so the runtime loads it once at start-up.
|
|
135
135
|
|
|
136
136
|
How it stays lightweight:
|
|
137
137
|
|
|
@@ -149,7 +149,7 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
|
|
|
149
149
|
|
|
150
150
|
To troubleshoot:
|
|
151
151
|
|
|
152
|
-
- Extract and inspect [symbol library contents](https://doc.log10x.com/compile/link/#file-structure) (`tar tf library.10x.tar`),
|
|
152
|
+
- Extract and inspect [symbol library contents](https://doc.log10x.com/compile/link/#file-structure) (`tar tf library.10x.tar`); units list file, class and method names, and each string only as a hash
|
|
153
153
|
- Run the engine locally on test logs and enable [debug output](https://doc.log10x.com/run/transform/symbol/#debug) to see symbol matching and template assignment in detail
|
|
154
154
|
- Check the engine [log file](https://doc.log10x.com/manage/logging/#log-file-location) for runtime template assignment messages
|
|
155
155
|
|
|
@@ -162,7 +162,7 @@ The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from
|
|
|
162
162
|
- **[GitHub repos](https://doc.log10x.com/compile/pull/github/)**, extracts symbols from source code and compiled binaries
|
|
163
163
|
- **[Artifactory](https://doc.log10x.com/compile/pull/artifactory/)**, pulls from artifact repositories
|
|
164
164
|
|
|
165
|
-
The runtime ships with a [default symbol library](https://doc.log10x.com/compile/pull/#default-symbols) covering
|
|
165
|
+
The runtime ships with a [default symbol library](https://doc.log10x.com/compile/pull/#default-symbols) covering the repositories and container images listed in the manifest shipped inside the library archive (Spring Boot, Django, Express, Kafka, Kubernetes, and more). Most environments work out of the box, custom compilation is only needed for proprietary logging frameworks or application-specific log formats.
|
|
166
166
|
|
|
167
167
|
The compiler commits custom symbol libraries to GitHub as part of its pipeline. Edge and cloud apps pull them automatically via the [`@github` launch macro](https://doc.log10x.com/config/github/) at startup and poll for changes at a configurable interval. See [compiler workflow](https://doc.log10x.com/compile/#workflow) for the full CI/CD → Pull → Scan → Link → Push → Distribute pipeline.
|
|
168
168
|
|
|
@@ -8,7 +8,7 @@ icon: "material/cogs"
|
|
|
8
8
|
---
|
|
9
9
|
The `AOT compile` pipeline produces [symbol library](https://doc.log10x.com/compile/link/#symbol-library) files that enable the [run](https://doc.log10x.com/run/) stream processor pipeline to transform input events into typed [TenXObjects](https://doc.log10x.com/api/js/#TenXObject "Provide structured, reflective access to log/trace events read from input(s)."). To launch this pipeline use the [compiler app](https://doc.log10x.com/apps/compiler/).
|
|
10
10
|
|
|
11
|
-
The 10x Engine
|
|
11
|
+
The 10x Engine ships a [default library](https://doc.log10x.com/compile/pull/#default-symbols); the manifest inside it lists every repository and container image it was built from. See [naming](https://doc.log10x.com/run/initialize/message/#message-extraction) for how a line gets its pattern name. Running the compiler on your own environment's repos is **optional** and enables the runtime to increase its level of efficiency in aggregating and reducing event volume.
|
|
12
12
|
|
|
13
13
|
## :material-cog-transfer-outline: Workflow
|
|
14
14
|
|
|
@@ -24,6 +24,7 @@ Operating on typed TenXObjects enables direct access to [symbol](https://doc.log
|
|
|
24
24
|
Each symbol library `.10x.tar` archive file contains:
|
|
25
25
|
|
|
26
26
|
- A `.10x.json` file combining the symbol unit files generated during the [scan](https://doc.log10x.com/compile/scan/ "Capture symbol values from source code/binary files and link to an output symbol library") phase.
|
|
27
|
+
- A `manifest.10x.json` file listing each source with the commit or digest it was built from.
|
|
27
28
|
- A `.10x.pb` Protocol Buffer file which provides a reverse in-mem index of the symbols values contained within the JSON for fast loading and random access. To learn more see the [.proto IDL](https://github.com/log-10x/pipeline-extensions/blob/main/api-extensions/src/main/proto/FsSymbolUnitsIndex.proto){target="\_blank"}.
|
|
28
29
|
|
|
29
30
|
## :material-wrench-outline: Config Files
|
|
@@ -290,7 +291,7 @@ Token types read from input source/binary files written to output.
|
|
|
290
291
|
|
|
291
292
|
|Type|Default|
|
|
292
293
|
|---|---|
|
|
293
|
-
|List|\[
|
|
294
|
+
|List|\[class, enum, log\]|
|
|
294
295
|
|
|
295
296
|
Filter symbol values to write to [outputSymbolLibraryFile](https://doc.log10x.com/compile/link/#outputsymbollibraryfile "location of output symbol library file") based on their [context](https://doc.log10x.com/run/transform/symbol/#contexts).
|
|
296
297
|
This option reduces the size of the output symbol library file by omitting symbol types not required at run time.
|
|
@@ -13,6 +13,8 @@ Pulling [symbol files](https://doc.log10x.com/compile/scanner/symbol/ "Discover
|
|
|
13
13
|
|
|
14
14
|
The 10x runtime comes with a built-in symbol library that includes symbols for industry-standard frameworks and infrastructure components. This pre-compiled symbol library enables immediate log optimization capabilities without requiring users to scan and compile symbols for common technologies and popular open-source projects.
|
|
15
15
|
|
|
16
|
+
The library's [manifest](https://doc.log10x.com/compile/link/#file-structure) lists each source with the commit or digest it was built from. From container images the library takes the strings in their binaries. Test files, vendored copies and log files inside the repositories are left out. Each project keeps its own license: the library holds hashes of its strings and the names of its files, classes and methods, not its source text.
|
|
17
|
+
|
|
16
18
|
??? tenx-symbols "Container Images (Helm Charts)"
|
|
17
19
|
|
|
18
20
|
Symbols extracted from Docker container images and their source code dependencies via Helm chart scanning.
|
|
@@ -21,7 +23,7 @@ The 10x runtime comes with a built-in symbol library that includes symbols for i
|
|
|
21
23
|
|------------|------------|
|
|
22
24
|
| [open-telemetry/opentelemetry-demo](https://github.com/open-telemetry/opentelemetry-demo) | Polyglot microservices demo (14 services in Go, Java, Node.js, Python, .NET, Rust, PHP, Ruby, C++) |
|
|
23
25
|
| [ingress-nginx/ingress-nginx](https://github.com/kubernetes/ingress-nginx) | NGINX binary, HTTP error codes, upstream timeouts, load balancer logs |
|
|
24
|
-
| [prometheus-community/kube-prometheus-stack](https://github.com/prometheus-community/helm-charts) | Prometheus,
|
|
26
|
+
| [prometheus-community/kube-prometheus-stack](https://github.com/prometheus-community/helm-charts) | Prometheus, Prometheus Operator, node-exporter, kube-state-metrics |
|
|
25
27
|
|
|
26
28
|
??? tenx-symbols "Java Frameworks"
|
|
27
29
|
|
|
@@ -691,7 +691,7 @@ Maximum number of input files to scan per sub-process.
|
|
|
691
691
|
|
|
692
692
|
|Type|Default|Category|
|
|
693
693
|
|---|---|---|
|
|
694
|
-
|Number|
|
|
694
|
+
|Number|200|Subprocess|
|
|
695
695
|
|
|
696
696
|
Sets the maximum number of source/binary input files to scan by a single 'compile' sub-process.
|
|
697
697
|
Set to 0 to unlimited.
|
|
@@ -720,7 +720,7 @@ Timeout interval for scanning a source code /binary input file.
|
|
|
720
720
|
|
|
721
721
|
|Type|Default|Category|
|
|
722
722
|
|---|---|---|
|
|
723
|
-
|String|
|
|
723
|
+
|String|30s|Parallel|
|
|
724
724
|
|
|
725
725
|
Sets the timeout interval when scanning a source code /binary input file before dropping it.
|
|
726
726
|
Set to null to ignore.
|
|
@@ -768,9 +768,9 @@ It is a standard programming practice to use string formats to log messages in t
|
|
|
768
768
|
log.error("could not connect to {} with status {}", host, status);
|
|
769
769
|
```
|
|
770
770
|
|
|
771
|
-
|
|
772
|
-
|
|
773
|
-
|
|
771
|
+
The scanner splits the literal at each format prefix (e.g., `%`) into sections
|
|
772
|
+
(`could not connect to`, `with status`) and stores each section; at run time a
|
|
773
|
+
multi-section format matches only when every section is present on the line.
|
|
774
774
|
|
|
775
775
|
|
|
776
776
|
#### :material-menu-right-outline:**`maxSymbolUnitsPerToken`**
|
|
@@ -14,7 +14,7 @@ logger.error("could not connect to {}", host);
|
|
|
14
14
|
cerr << "could not connect to" << host << << std::endl;
|
|
15
15
|
```
|
|
16
16
|
|
|
17
|
-
The string constants receive a 'log' context,
|
|
17
|
+
The string constants receive a 'log' context. So does every literal in a method that logs, throws or raises (in Java source, Python and Scala, only those preceding the logging or throwing statement), and any literal carrying a format placeholder. A literal in a method that does neither keeps a 'const' context:
|
|
18
18
|
|
|
19
19
|
```js
|
|
20
20
|
foo("could not connect to " + host);
|
|
@@ -9,7 +9,7 @@ Extracts [symbol](https://doc.log10x.com/run/transform/structure/#symbols) value
|
|
|
9
9
|
|
|
10
10
|
Parses input by splitting lines with delimiters or using a [JsonFactory](https://fasterxml.github.io/jackson-core/javadoc/2.8/com/fasterxml/jackson/core/JsonFactory.html){target="\_blank"} for structured token reading.
|
|
11
11
|
|
|
12
|
-
When source code for a log format is unavailable (e.g., third-party services),
|
|
12
|
+
When source code for a log format is unavailable (e.g., third-party services), a sample log can be scanned at compile time as an optional input to extract its symbols. Nothing is learned at run time.
|
|
13
13
|
|
|
14
14
|
!!! note "Size limit"
|
|
15
15
|
|