@grepr/cli 1.11.5 → 1.11.6-674b35e

package/build/dist/docs-index/912c4feb-57ad-4fd3-8c93-f0a4e85a27bd.txt

@@ -0,0 +1,136 @@
+Control what a Grepr AI agent can reach with a service account, tools, skills, and MCP integrations, and whether its pipeline changes need approval.
+
+# Control what an AI agent can do
+
+An agent's reach is bounded in layers, and a capability has to be allowed by every layer to work:
+
+1. The **service account** the agent acts as. Its roles are the outer boundary and cap everything the agent can reach in Grepr.
+2. The **tools and skills** you enable. These narrow the agent to specific operations within what the service account permits.
+3. The **MCP integrations** you enable, and the tools you allow on each. These decide what the agent can reach in other systems.
+
+A fourth setting, the change approval mode, decides whether the agent can make automatic changes to your pipeline.
+
+These settings are on the agent's **Capabilities** step, and on the matching step of the create wizard.
+
+## Choose the service account an agent acts as \[#service-account]
+
+Every call an agent makes to Grepr authenticates as the service account you select in the **Service account** field. The service account's roles decide which pipelines, datasets, and integrations the agent can read and write to.
+
+The service account is the boundary that holds even if a tool grant is over-permissive. An agent with the pipeline editing tool but a service account that can only view jobs cannot change a pipeline. Grant the narrowest role that lets the agent finish its job. To create one, see [_](/admin/service-accounts).
+
+If the **Service account** menu is empty, create a service account before you continue.
+
+## What every agent can already do \[#sandbox-baseline]
+
+Each investigation runs in its own sandbox, and inside that sandbox an agent can:
+
+* Run shell commands.
+* Run Python for local analysis and scripting.
+* Query files it downloaded with DuckDB.
+* Read, create, and edit files in its own workspace.
+
+These are part of the sandbox and cannot be turned off. They act only on the sandbox, which is discarded when the investigation ends, so they do not reach your data or your pipelines. Everything that does reach Grepr is granted separately, as described next.
+
+## Grant access to Grepr with tools and skills \[#grepr-access]
+
+The **Grepr access** section is where you decide what the agent can do. You grant access in two ways, and most agents use both:
+
+* Enable a skill, which gives the agent guidance for a task and, when the skill needs them, the tools that task requires.
+* Pin an individual tool, which grants that tool with no accompanying guidance.
+
+Prefer skills. A skill tells the agent how to use what it grants, so an agent with a skill usually needs fewer turns than one holding the bare tool. Pin a tool when you want a capability that no skill attached covers.
+
+### Abilities and reference skills \[#skill-kinds]
+
+Skills are grouped by whether enabling one is a permission decision:
+
+* **Abilities** carry tools. Enabling one grants the agent the operations that skill needs, so treat it as granting access.
+* **Reference** skills carry only knowledge. Enabling one changes what the agent knows, never what it can do, so it is safe to enable freely.
+
+To read what a skill covers and what it grants before you enable it, open its detail from the selector. To learn what skills exist and how to write your own, see [_](/ai-agents/skills).
+
+### How much a skill grants \[#grant-scope]
+
+How wide a skill's grant is depends on its scope, and the difference matters when you are reasoning about least privilege:
+
+* A **platform skill**, which Grepr maintains, grants exactly the operations its definition lists. A skill that needs only to preview a pipeline change gets the preview operation and not the one that applies it.
+* An **organization skill**, which you create, grants the whole *tool family* behind each tool you approve on it, meaning every operation that tool covers rather than only the ones the skill uses. Approving the pipeline editing tool on your own skill grants everything that tool covers, not only the part your instructions use.
+
+So an organization skill is as broad as the tools you approve on it. To keep a grant narrow, approve fewer tools on the skill rather than relying on the skill's instructions to limit the agent.
+
+Enabling a tool directly always grants its whole family, the same way an organization skill does.
+
+### Tool reference \[#tool-table]
+
+| Tool | What it lets the agent do |
+| --- | --- |
+| `query_logs` | Query the logs in your Grepr datasets. |
+| `get_pipeline` | Read the configuration of your pipelines and templates. |
+| `edit_pipeline` | Propose a change to a pipeline's configuration, and preview a change before proposing it. |
+| `grok` | Build and test Grok parsers. |
+| `datasets` | List, read, and manage your Grepr datasets. |
+| `integrations` | List and read your Grepr integrations. |
+| `docs` | Search the Grepr documentation and validate SQL. |
+| `backfill` | Replay logs from the data lake to a vendor destination. |
+| `investigations` | Read agent configuration, past investigations, and their transcripts. |
+| `github_code` | Clone, edit, and publish source repositories. Requires an enabled GitHub MCP integration. |
+
+An agent's pipeline changes are scoped to a pipeline's Grok parsers and its SQL transforms, so granting the pipeline editing tool keeps the rest of the pipeline configuration out of the agent's reach. Both are add-only. An agent can add a Grok rule or add a SQL transform, and it cannot change or remove one that is already there, so its proposals accumulate alongside your configuration rather than rewriting it.
+
+A SQL transform an agent adds is a filter and nothing further. It reads the pipeline's logs and selects them unchanged against a single condition, so it cannot join, aggregate, read a dataset, register a function, or keep state. It routes the events it matches to the agent signal sink, which you configure yourself when you set the pipeline up. An agent can add routes to that sink, and it cannot create, change, or remove the sink. That limit is on what an agent adds by itself. A SQL output you route to the agent signal sink yourself carries no such restriction, so you can build the event an agent receives however you need. See [_](/ai-agents/subscriptions#agent-signal-sink).
+
+The tools an agent uses to record and search its memory are not in this list. They are granted by the agent's memory settings instead, so that turning memory on is the single decision that controls them. See [_](/ai-agents/memory).
+
+## Let an agent call an MCP server's tools \[#mcp-integrations]
+
+The **MCP integrations** section lists the MCP integrations in your organization and which of their tools the agent can use. Two agents can share one integration with different tool access enabled.
+
+Edit the integration's allowed tools and choose either every tool the server publishes or only the ones you select. Grepr reads the tool list from the server and marks each tool as read only or writing; for example, you can allow an agent to read incidents without allowing it to change them.
+
+Allowing every tool means the agent gains any tool the server adds later, which is convenient for a read-only server and worth avoiding for one that can make changes. Selecting individual tools keeps the enabled tools fixed until it is manually changed.
+
+Two rules apply to GitHub:
+
+* An agent can enable at most one GitHub MCP integration.
+* The `github_code` tool needs an enabled GitHub MCP integration, because that integration is how it reaches source code.
+
+To create an integration, see [_](/integrations/mcp).
+
+## Decide whether pipeline changes need approval \[#change-approval]
+
+An agent that decides a pipeline needs a configuration update does not edit it directly. It submits the change, and the **Pipeline changes** setting decides what happens next:
+
+| Mode | What happens when the agent proposes a change |
+| --- | --- |
+| **Require human approval** | Grepr records the change as a suggestion on the pipeline. Nothing changes until you review and apply it. |
+| **Auto-apply** | Grepr verifies the change and applies it. A running pipeline restarts to pick it up. No person is involved. |
+
+**Require human approval** is the default, and it is the right setting while you are learning how an agent behaves. Auto-apply an agent's changes only after its suggestions have consistently been ones you would have made yourself.
+
+Grepr enforces the mode at the operation level, so while an agent can always preview pipeline changes, it cannot bypass **Require human approval** to apply them.
+
+A change that lands on a running pipeline restarts it, which interrupts processing briefly.
+
+To review suggestions, see [_](/ai-agents/investigations#suggestions).
+
+## Frequently Asked Questions
+
+### How do I limit what a Grepr AI agent can reach?
+
+Reach is bounded in layers. The service account the agent acts as is the outer boundary and caps everything it can reach in Grepr. Within that, the tools and skills you enable narrow it to specific operations, and the MCP integrations you enable with their allowed tools decide what it can reach in other systems.
+
+### What can a Grepr AI agent do without any tools enabled?
+
+Inside its own sandbox it can always run shell commands and Python, query downloaded files with DuckDB, and read and write files in its workspace. These act only on the sandbox, which is discarded when the investigation ends, and they do not reach your data or pipelines.
+
+### What is the difference between an ability and a reference skill?
+
+An ability carries tools, so enabling one grants the agent the operations that skill needs and is a permission decision. A reference skill carries only knowledge, so enabling one changes what the agent knows but never what it can do.
+
+### Do platform skills and organization skills grant the same access?
+
+No. A platform skill grants exactly the operations its definition lists, which can be narrower than a whole tool. An organization skill grants the whole tool family behind each tool you approve on it, so keep its grant narrow by approving fewer tools.
+
+### Can a Grepr AI agent change my pipeline without approval?
+
+Only when you set its change approval mode to Auto-apply. The default, Require human approval, records every proposed change as a suggestion for you to review. Grepr enforces the mode server-side, so an agent has no route around it whatever its instructions say.

package/build/dist/docs-index/948ca32a-5acd-49c8-996a-1504b46f2193.txt

@@ -0,0 +1,78 @@
+Two ways to send logs to several vendor accounts, by creating a pipeline for each group or by grouping the reducer and filtering each sink.
+
+# Route logs to several vendor accounts
+
+Your logs do not all belong in the same place. Separate teams, environments, or business units each own an account in your observability platform, and a log has to reach the account that owns it. Each account you send to is its own Grepr integration.
+
+A Grepr pipeline is a free-form processing graph rather than a fixed shape, so you can arrange this in more than one way. Here are two examples, and you can combine them.
+
+## Create a pipeline for each group \[#pipeline-per-group]
+
+Create one pipeline for each group of logs that belongs together, filter its source to that group, and give it the sink for that group's account.
+
+Each pipeline summarizes only its own logs, so you can tune each one on its own. Choose this when the groups need different configurations, or when you want them isolated from each other.
+
+To create a pipeline, see [_](/tutorials/first-pipeline). To limit what one ingests, see [_](/transforms/filter-events).
+
+## Group the reducer and filter each sink \[#grouped-sinks]
+
+Run a single pipeline, add a sink for each account, and filter each sink to that account's logs. Choose this when the groups have the same configuration, and you would rather manage one pipeline than several.
+
+Routing rests on a *routing key*, which is a field whose value names the account a log belongs to, such as `account` or `team`. Two settings act on it: the reducer groups on the key, and each sink filters on it.
+
+Grouping is what makes the filtering correct. A summary carries the values of the fields it was grouped on, so grouping on the routing key gives every summary exactly one account, and each sink's filter then matches its own account and no other.
+
+> Configure the grouping before you add the sinks, because leaving it out fails quietly. Without the routing key in the group-by, logs bound for different accounts aggregate together. A tag then carries every value the merged messages had, so every account receives the same summary, reporting counts that include the other accounts' logs. An attribute is merged instead, and the default merge strategy replaces values that differ with a wildcard, so no filter matches and the summary reaches no account.
+
+### Step 1: Put the routing key on every log \[#step1-routing-key]
+
+The routing key has to be on each log before the reducer sees it, holding one value per account. If your collectors already tag logs with the owning team or environment, use that tag. If the value is inside the message, parse it into an attribute or a tag first with the Grok parser or the JSON parser. See [_](/transforms/grok) and [_](/transforms/json-parsing).
+
+A tag and an attribute both work as the routing key. Choose one whose values are few and stable, because each value aggregates separately. Decide what happens to logs carrying no value for the key, which match no filter and so reach no account: give them a default value when you parse, or add a sink that catches them.
+
+### Step 2: Group the reducer on the routing key \[#step2-group]
+
+1. On the pipeline's **Reducer** step, click **Edit** on the **Fields** card.
+2. If the routing key is listed, set its **Behaviour** to **Match or split**.
+3. If it is not listed, enter it in the **Add field** row, set the **Behaviour** to **Match or split**, and click **Add**.
+4. Click **Save**.
+
+The **Match or split** behavior keeps a field out of aggregation, so messages whose values differ are never merged into one pattern. Enter tag keys plainly, and prefix attribute paths with `@`. An attribute is merged as well as grouped, and the default merge strategy keeps the value that every message in the group agrees on. If you set a different merge strategy for the routing attribute, check that it preserves that value rather than combining or discarding it. See [_](/transforms/reducer/configuration#attribute-configuration).
+
+### Step 3: Create a Grepr integration for each account \[#step3-integrations]
+
+Create one Grepr integration for each account you send to. An integration authenticates to a single account, so its credential is what decides where its logs land. Name each integration after its account, because that name is what you select when you add the sink. See [_](/integrations).
+
+### Step 4: Add a filtered sink for each account \[#step4-sinks]
+
+Repeat for each account:
+
+1. On the pipeline's **Sinks** step, click **Add sink**.
+2. In the **Sink** menu, select the integration for the account. Grepr fills in a **Name**, which you can change.
+3. In the **Query language** menu, select the syntax you want to write the filter in.
+4. In the **Filter Query** field, enter a query that matches only that account's logs, such as `team:platform`.
+5. Click **Add**, then save the pipeline.
+
+A sink with an empty filter forwards everything, so give each sink a filter query that matches it to one account. A sink's card reports its filter and shows **All messages** when it has none, which is how you spot one you missed. The filter runs on everything the reducer emits, so summaries, samples, passthrough messages, and exceptions all follow the same routing. To learn about the filter syntax, see [_](/queries/datadog) and [_](/queries/new-relic).
+
+## Frequently Asked Questions
+
+### Can one Grepr pipeline send logs to several accounts of the same vendor?
+
+Yes. Add one sink for each account, each pointing at that account's integration, and give each sink a filter that matches only that account's logs. Group the reducer on the same routing key the filters use, so that each summary belongs to exactly one account.
+
+### Why does the reducer need to group on the routing key?
+
+Without it, logs for different accounts aggregate together and the summary no longer identifies one account. If the key is a tag, the summary carries every value the merged messages had, so each account's filter matches it and every account receives counts that include the other accounts' logs. If the key is an attribute, the default merge strategy replaces the differing values with a wildcard, so no filter matches and the summary reaches no account.
+
+### Should I create a pipeline for each account or filter the sinks of one pipeline?
+
+Create a pipeline for each group when the groups need different reduction settings or different exceptions, or when you want them isolated. Filter the sinks of one pipeline when the groups share one reduction policy and you would rather keep a single pipeline to change.
+
+### What happens to logs that have no value for the routing key?
+
+They match no sink's filter and reach no vendor account. Give them a default value when you parse the routing key, or add a sink whose filter matches them so they land somewhere you can see.
+
+### Does a sink filter apply to summarized logs as well as raw ones?
+
+Yes. The filter runs on everything the reducer emits, so summaries, samples, passthrough messages, and exceptions all follow the same routing.

package/build/dist/docs-index/98eff00a-e5e6-4554-bfa8-3cda94790750.txt

@@ -0,0 +1,301 @@
+Configure the Grepr log reducer to reduce the volume of logs forwarded to your observability vendor while preserving access to your raw logs.
+
+# Configure the log reducer in a Grepr pipeline
+
+This page describes the transformations the log reducer applies to incoming log messages and how to configure them to meet your requirements. When you create log reduction pipelines, Grepr adds a log reducer with default settings that control how log messages are aggregated, grouped, tokenized, and summarized. These settings are suitable for many log-processing workloads, but you can modify these settings to meet your requirements through the Grepr UI, the REST API, or the Grepr CLI. This page describes each setting and how to configure the setting in the UI. For the API documentation, see the [LogReducer API specification](/apis/api-spec#schema/LogReducer). For the CLI, see [The Grepr CLI](/cli).
+
+The log reducer transforms logs into summarized events through a multi-step process. The reducer divides the incoming log stream into consecutive *aggregation windows* and produces summaries for each window separately. An aggregation window is a fixed-duration, tumbling window over which the reducer counts and aggregates the messages that match each pattern. Tumbling windows are contiguous and non-overlapping, so each log belongs to exactly one window. The aggregation window is two minutes by default.
+
+> To change the aggregation window duration, you must use the Grepr REST API. You cannot use the Grepr UI to change the window duration. Before changing this value, Grepr recommends contacting <support@grepr.ai> for guidance.
+
+The following describes the core steps in this process and includes links with details on changing the default behavior for each step:
+
+* Masking replaces dynamic values such as timestamps, IDs, and IP addresses with specific tokens. By normalizing values that represent the same entity but might differ slightly in format or content, masking prevents them from being split incorrectly in the next step. See [Enable grouping of dynamic fields by assigning common tokens](#mask-configuration).
+* Tokenizing breaks log messages into tokens based on configurable delimiters, creating a structured representation optimized for grouping. See [Configure delimiters used to split log messages](#delimiter-configuration).
+* Grouping uses similarity metrics to group messages into patterns. A configurable similarity threshold determines how closely messages must match to be considered part of the same pattern. You can use grouping to retain the values of important fields by excluding them from aggregation. See [Configure how messages are grouped](#grouping-configuration).
+* Sampling sends a configurable number of raw sample messages per pattern. This number is referred to as the *deduplication threshold* throughout this documentation. After this threshold is reached, you can configure the reducer to stop forwarding that pattern until the end of the aggregation window, or use logarithmic sampling. See [Configure a sampling strategy](#sampling-configuration).
+* Summarizing generates, at the end of each aggregation window, concise summaries of aggregated messages that matched an identified pattern. For details on the fields included in the summarized output events, see [Summary message metadata](/transforms/reducer#summary-metadata).
+
+To modify the configuration settings in the UI, on the overview page for your pipeline, click **Reducer** in the left-hand navigation menu to open the **Edit Reducer** dialog. The dialog contains a form to configure **Grouping Configuration**, **Aggregation Configuration**, **Sampling Configuration**, **Mask Configuration**, **Delimiter Configuration**, and **Attribute Configuration** values. The following sections describe these settings.
+
+## Configure how messages are grouped \[#grouping-configuration]
+
+Grouping controls how incoming messages are partitioned into independent groups before pattern detection, and the level of similarity messages must meet to be considered part of the same pattern.
+
+### Group messages by a field \[#group-by-configuration]
+
+Messages are aggregated separately for each unique combination of values across the fields you group on, so patterns from different services, hosts, or other dimensions are never merged. Grouping on `service`, `host`, and the `http.url.path` attribute aggregates messages independently for each unique combination of those three values.
+
+To group on a field, on the pipeline's **Reducer** step, click **Edit** on the **Fields** card, find the field, and set its behavior to **Match or split**. A field with this behavior is kept out of aggregation, so messages whose values for it differ are never merged into one pattern.
+
+Attribute paths are written with an `@` prefix and dot notation for nested attributes, such as `@http.url.path`. Tag keys are written as the key name alone.
+
+Use grouping to retain the values of important fields by excluding them from aggregation, such as high-cardinality identifiers like user IDs. To keep a field that is currently inside the message, parse it into an attribute or a tag with the **JSON Parser** or **Grok Parser** step earlier in the pipeline, then group on that field. See [Transform JSON strings into JSON objects](/transforms/json-parsing) and [Parse log messages to enrich log events with the Grok parser](/transforms/grok).
+
+Grouping also decides how a summary can be routed, because a summary carries the tag values of the fields it was grouped on. To use grouping to send each group's logs to a different destination, see [_](/solution-guides/route-logs-to-several-accounts).
+
+### Similarity threshold (%)
+
+The similarity threshold controls how close a set of tokens must be to an existing pattern for the reducer to consider the message a match for that pattern. To set this threshold, enter a whole number from 20 to 100. This value represents the percentage of tokens that must match between the incoming message and the pattern.
+
+* A higher value requires a closer match to group messages into the same pattern. Higher values produce more distinct patterns, each containing fewer of the tokens defined by [regular expression masks](#mask-configuration), and reduce volume less aggressively.
+* A lower value allows grouping messages with more variation into the same pattern. Lower values lead to greater reduction, but the resulting summary messages contain more mask-defined tokens.
+
+## Configure how messages are aggregated \[#aggregation-configuration]
+
+The **Aggregation Configuration** section controls how many raw messages are forwarded before aggregation begins for a pattern and how summary messages are formatted.
+
+### Minimum number of samples
+
+To set a *deduplication threshold* for messages, in the **Minimum number of samples** field, enter a whole number of two or greater. This value is the minimum count of messages matching a pattern that the reducer forwards as raw messages inside a single aggregation window before it begins aggregating additional matches into a summary message. Until this threshold is reached, unaggregated messages are forwarded to your sinks.
+
+Use a lower value to begin aggregation sooner and maximize volume reduction. Use a higher value when a pattern has a higher threshold for the number of duplicate messages to forward before beginning aggregation.
+
+### Add repeat count prefix to summary messages
+
+To have the reducer prepend a prefix to each summary message that indicates the number of raw messages aggregated and the duration of the aggregation period, select **Add repeat count prefix to summary messages**.
+
+Clear **Add repeat count prefix to summary messages** if your downstream system needs to parse the summary message body without changes. For example, if summary messages use JSON formatting, the leading prefix would cause a syntax error.
+
+### Choose which fields form a pattern \[#structured-configuration]
+
+To reduce logs whose useful information is in attributes, on the reducer step of your pipeline, click **Edit** on the **Patterns** card. Use **Build patterns from** to select the fields that determine patterns:
+
+| Build patterns from | Behavior |
+| --- | --- |
+| **Message only** | Uses only the message. Empty messages pass through without reduction. This is the default. |
+| **Message, or attributes when empty** | Uses the message when it is nonempty, otherwise the selected attributes. Summaries of empty-message events have a null message. |
+| **Attributes only** | Ignores the message for pattern matching. Generated summaries have a null message; original samples and exceptions retain their messages. |
+| **Message and attributes** | Uses the selected attribute structure for exact grouping and combines message tokens with any attribute-value tokens for similarity. Summaries contain only the message text, and events with an empty message produce summaries with a null message. |
+
+When attributes participate, their selected key structure must match exactly. Key order does not matter. Adding a selected key, changing nesting, or changing a null field to a populated scalar separates patterns. When the message is read from an attribute that holds a string, that attribute does not participate either: it is used as the message or, with **Attributes only**, ignored along with the message. An attribute holding anything else, such as an object body, provides no message and participates like any other attribute.
+
+With the default, **Message only**, attributes are not used. The attribute settings below appear when you set **Build patterns from** to an option that uses attributes.
+
+Use **Attribute values** to choose how selected attributes affect similarity scoring:
+
+* **Compare values**: Use values as part of the similarity calculations. A mask applies to a value only when it matches the entire value.
+* **Ignore values**: Attribute values are not used for similarity scoring.
+
+### Include or exclude attribute subtrees \[#attribute-prefixes]
+
+Use **Include attributes** and **Exclude attributes** to select the attribute structure and values that participate in pattern matching. Enter each attribute path the same way as in the **Fields** dialog: `@` followed by the path in dot notation, or a JSON array for a key that contains a dot. For example, `@request.headers` selects the nested `request.headers` subtree, while `@["request.headers"]` selects a top-level key containing a literal dot.
+
+Leave the include field empty to include all attributes. Exclusions take precedence over includes. Prefixes select complete subtrees and address keys inside array elements without an array index. Excluded fields, ancestors containing only excluded fields, and array elements containing only excluded fields contribute neither structure nor value tokens. Selected containers that are empty in the event remain part of the shape.
+
+These settings affect pattern matching. They do not remove attributes from events or change their configured merge strategies. An excluded attribute can still be an explicit grouping value. If no message tokens or selected attribute structure remain, the event passes through.
+
+For example, this reducer configuration uses request keys for matching while requiring the excluded tenant value to match exactly:
+
+```json
+{
+  "type": "log-reducer",
+  "tokenizationInput": "ATTRIBUTES_ONLY",
+  "attributeTokenization": "KEYS_ONLY",
+  "includedAttributePrefixes": [["request"]],
+  "excludedAttributePrefixes": [["request", "tenant_id"], ["request", "trace_id"]],
+  "partitionByAttributePaths": [["request", "tenant_id"]]
+}
+```
+
+The `pattern-matcher` operation supports the same API settings and preserves the messages of forwarded events. Both operations default `tokenizationInput` to `MESSAGE_ONLY`, which is **Message only**, so attributes are not used. When attributes are used, `attributeTokenization` defaults to `KEYS_AND_VALUES`, which is **Compare values**, and empty attribute lists select every attribute.
+
+## Configure a sampling strategy \[#sampling-configuration]
+
+The **Sampling Configuration** section controls whether and how additional raw messages are forwarded for a pattern after the minimum number of samples is reached in the current aggregation window. To configure the sampling strategy for a reducer, in the **Sampling strategy** menu, select one of the following options:
+
+### No Additional Sampling
+
+During each aggregation window, Grepr passes through a configurable number of sample messages for each pattern before beginning aggregation. After the deduplication threshold is reached for a specific pattern, Grepr stops sending messages for that pattern until the end of the aggregation window, at which point a summary is emitted. This ensures that a minimum number of raw messages always pass through unmodified. Low-frequency messages that usually contain important troubleshooting information pass through unmodified.
+
+While this default behavior maximizes reduction, the actual count of messages seen at the vendor is independent of the number of original raw messages. Instead, the count is based on the messages summarized in each aggregation window. Features in external log aggregators that depend on message counts by pattern, such as Datadog's "top patterns" capability, might not work as expected. To address this, select the following logarithmic sampling option.
+
+### Window-Based Logarithmic
+
+With logarithmic sampling, high-volume patterns that occur in the aggregation window are sampled more heavily than low-volume patterns.
+
+When you select this option, to configure the rate of additional sampling, enter a whole number of two or greater in the **Logarithm Base** field. The reducer process uses this base value to increase the sample size logarithmically. For example, if the base is set to two and the deduplication threshold is set to four, Grepr sends one additional sample message when the number of messages in the aggregation window first exceeds two raised to the power of the deduplication threshold + 1 (`2^5 or 32`). Additional samples are then sent at the 64th, 128th, 256th, and subsequent messages. This ensures that an unexpectedly large increase in messages matching a pattern still appears as an increase in message count at the vendor, but with a logarithmically smaller magnitude.
+
+## Enable grouping of dynamic fields by assigning common tokens \[#mask-configuration]
+
+The **Mask Configuration** section defines the regular expression masks that the reducer uses to replace variable parts of a log message, such as timestamps, IDs, or IP addresses, with specific tokens before the message is tokenized and grouped. Masking normalizes variable content so that messages that differ only in these values are grouped into the same pattern.
+
+The reducer ships with the following default masks. Each appears as a row in the **Mask Configuration** section and can be enabled, disabled, edited, or reset to its original pattern:
+
+| Mask | Default state | Description |
+| --- | --- | --- |
+| `timestamp` | Enabled | Matches many timestamp formats, such as `2024-04-26T15:30:45.123Z`, `26/04/2024 15:30:45`, or `15:30:45`. |
+| `ipport` | Enabled | Matches IPv4 addresses and optional ports, such as `192.168.0.1` or `192.168.0.1:8080`. |
+| `number` | Enabled | Matches integers and decimal numbers up to 100 digits. |
+| `uuid` | Enabled | Matches UUIDs, such as `123e4567-e89b-12d3-a456-426614174000`. |
+| `awsarn` | Disabled | Matches AWS ARNs. |
+| `awstoken` | Disabled | Matches AWS session tokens. |
+
+For each mask row, you can:
+
+* Select or clear the checkbox on the left to enable or disable the mask. Only enabled masks are applied to incoming messages.
+* Click **Edit** to display the regular expression for the mask and modify the pattern. The pattern field cannot be empty.
+* For default masks, click the restore icon to reset the pattern to its original value.
+* For custom masks, click the delete icon to remove the mask.
+
+### Enable grouping of non-default formats by assigning custom tokens
+
+To ensure the correct grouping of fields with formats that aren't handled by the default masks, you can create custom masks. For example, you might have a field with values that include delimiter characters, and you need to prevent the reducer from splitting the single field on those delimiters.
+
+When you add a custom mask, you must enter a name for the mask that:
+
+* Contains only lowercase alphabetic characters. Spaces, digits, and symbols are not allowed.
+* Is no more than 50 characters long.
+* Is unique across all masks in this reducer.
+
+Enter the mask in the **Regular Expression Pattern** text box. A custom mask pattern supports the following common regular expression constructs:
+
+* Literal characters and character classes, such as `[a-z]`, along with predefined classes such as `\d`, `\w`, and `\s`.
+* Quantifiers, such as `*`, `+`, `?`, and `{n,m}`.
+* Alternation with `|` and grouping with `(` and `)`.
+* Anchors and word boundaries, such as `^`, `$`, and `\b`.
+
+> Custom mask patterns do not support lookahead or lookbehind assertions or backreferences. A pattern that uses one of these constructs passes the validation in the form but prevents the reducer from processing messages. To confirm that a custom mask matches the values you expect, test the pattern against sample messages.
+
+## Configure delimiters used to split log messages \[#delimiter-configuration]
+
+The **Delimiter Configuration** section defines the characters that the reducer uses to split a log message into tokens during clustering. After masking, the reducer splits each message into tokens using these characters and then calculates similarity against known patterns. Characters that are not in the delimiter list are treated as part of a token.
+
+The default delimiter set contains common punctuation and whitespace characters: `:`, `#`, `[`, `]`, `(`, `)`, `{`, `}`, `|`, `,`, `;`, `"`, `'`, space, `\t`, `\n`, and `\r`.
+
+> The default set intentionally omits `@` and `.` so that email addresses and dotted identifiers, such as fully qualified class names, are not split across tokens.
+
+To modify the delimiter list:
+
+* To remove a delimiter, hover over its chip in the delimiter list and click the delete icon that appears.
+* To add a delimiter, enter the character in the **Add delimiter** field and click **Add** or press **Enter**. For control characters, enter the escape sequence: `\t` for tab, `\n` for newline, `\r` for carriage return, `\b` for backspace, `\f` for form feed, `\v` for vertical tab, or `\0` for null. Duplicate delimiters are rejected.
+* To restore the default delimiter set, click **Reset to Defaults**.
+
+## Configure how attributes are merged \[#attribute-configuration]
+
+The log reducer groups events using the configured [pattern inputs](#structured-configuration). Attribute values can differ within a group, so the reducer provides multiple strategies to merge them. These merge strategies also apply to attributes excluded from pattern matching.
+
+To control how the reducer combines attributes, you use the **Attribute Configuration** section to specify merge strategies for individual attribute paths and configure a default strategy that applies to any attribute path without an explicit configuration.
+
+The reducer supports the following merge strategy types:
+
+| Strategy | Behavior |
+| --- | --- |
+| **Exact Match** | If all aggregated messages have the same value for the attribute, the summary preserves that value. If they differ, the summary stores a wildcard, `*`. |
+| **Sample** | Collects up to a configured **Sample Limit** of values from the aggregated messages. When the limit is reached, additional values are dropped. |
+| **Preserve All** | Collects every distinct or repeated value from the aggregated messages, up to a configured **Preserve Limit**. When the incoming data exceeds the limit, the reducer emits the current summary and starts a new one, rather than dropping values. |
+| **Sum** | Sums numeric values across aggregated messages and stores the result as a scalar number. Non-numeric values are ignored. Collections of numbers are summed element by element. |
+| **Min** | Keeps the minimum numeric value across aggregated messages. Non-numeric values are ignored. |
+| **Max** | Keeps the maximum numeric value across aggregated messages. Non-numeric values are ignored. |
+| **Average** | Computes the arithmetic mean of numeric values across aggregated messages. Non-numeric values are ignored. |
+
+To see examples and learn more about the merge strategies, see [Merge strategy examples](/transforms/reducer/configuration/merge-strategy-examples).
+
+### Configure merge strategies
+
+To add a merge strategy, in the **Attribute merging strategies** section, click the plus (+) icon. Select a strategy from the **Strategy** menu and enter the target attribute path. If any part of the target path uses dot notation, enter the target path using dot notation, such as `http.response.bytes`, or a JSON array, such as `["http", "response", "bytes"]`. Each entry must have a non-empty path, and each path must be unique within the list.
+
+When you select **Sample** or **Preserve All**, you can configure the following settings:
+
+* **Sample Limit** or **Preserve Limit**: A whole number that's one or greater. These values cap the size of the collected values. **Sample** defaults to 10 and **Preserve All** defaults to 1000.
+* **Keep unique values only**: When selected, the reducer stores the collected values as a set and discards duplicates. When cleared, the reducer stores the values as a list and preserves duplicates.
+
+When you configure more than one strategy for the same attribute path, the reducer writes a separate attribute in the summary for each strategy, using a suffix such as `_sum`, `_avg`, or `_min`. The first strategy in the list also writes to the base attribute name. For example, configuring **Sum** and then **Average** for `bytes_sent` produces `bytes_sent_sum`, which is also written to `bytes_sent`, and `bytes_sent_avg`. To change the order in which strategies are applied, drag them in the list.
+
+> Adding multiple strategies, whether to a specific path or the default, adds additional attributes to the summary and requires more processing per message. Use multiple strategies only when your analytics or dashboards need more than one aggregated form of the same attribute.
+
+Each strategy type can be used only once per attribute path.
+
+### Configure the default strategy
+
+The **Default Strategy** section defines the strategies that apply to every attribute path that is not configured under **Attribute merging strategies**. You must configure at least one default strategy. **Exact Match** is the default.
+
+As with path-specific strategies, you can add multiple default strategies. When you add multiple default strategies, each strategy produces a suffixed attribute in the summary, and the first strategy also writes to the base attribute name.
+
+## Send new-pattern signals to AI agents \[#agent-signals]
+
+The reducer identifies the patterns in your logs, which also makes it the first place a pattern that has never occurred before becomes visible. You can have the pipeline report each new pattern to your Grepr AI agents, so an agent investigates behavior that is new in your systems without you writing an alert for it first.
+
+To turn this on, on the reducer step of your pipeline, open the **Advanced** settings and select **Emit new-pattern signals to AI agents**.
+
+Turning it on emits nothing by itself. A signal reaches an agent only when that agent is subscribed to the pipeline, and Grepr refuses a subscription to a pipeline that has this setting off. So turn it on first, then subscribe the agent.
+
+Each pattern signals an agent once rather than on every occurrence, for as long as the reducer keeps that pattern in its state, so a pattern that keeps matching does not keep starting investigations. To learn how signals reach an agent and how to bound the volume, see [_](/ai-agents/subscriptions).
+
+## Frequently Asked Questions
+
+### What is the aggregation window?
+
+The aggregation window is a fixed-duration, tumbling window over which the reducer counts and aggregates the messages that match each pattern. The reducer divides the incoming log stream into consecutive windows and produces a summary separately for each window. The window is two minutes by default. You cannot change the window duration in the Grepr UI; you must use the REST API, and Grepr recommends contacting support for guidance before changing it.
+
+### How do I prevent messages from different services being aggregated together?
+
+Add the tag key that identifies each service, such as service or env, to the Group-by values field. The reducer aggregates messages independently for each unique combination of group-by values, so patterns from different services are never merged.
+
+### What does the similarity threshold control?
+
+The similarity threshold is the percentage of tokens that must match between an incoming message and an existing pattern for the message to be considered part of that pattern. Higher values produce more distinct patterns and less aggressive reduction. Lower values produce broader patterns and more aggressive reduction, at the cost of more mask-defined tokens in each summary.
+
+### Why should I raise the minimum number of samples?
+
+The minimum number of samples is how many raw matching messages are forwarded for a pattern in an aggregation window before aggregation begins. Raise it when you want more raw samples of each pattern to be sent to your observability backend. Lower it to begin aggregation sooner and reduce more volume.
+
+### When should I disable the repeat count prefix?
+
+Disable the repeat count prefix when downstream systems parse the summary message body directly, such as when summary messages contain JSON. The prefix would otherwise prepend non-JSON text and break parsing.
+
+### What's the difference between the two sampling strategies?
+
+No Additional Sampling stops forwarding raw messages for a pattern as soon as the minimum number of samples has been reached in an aggregation window. Window-Based Logarithmic continues to forward additional raw samples at a logarithmically decreasing rate, preserving more visibility into high-volume patterns.
+
+### How does the logarithm base affect sampling?
+
+With Window-Based Logarithmic sampling, a larger logarithm base forwards fewer additional samples as message volume grows, and a smaller base forwards more. The base must be at least two.
+
+### What are masks and why does the reducer apply them?
+
+Masks are regular expressions that identify variable parts of a log message, such as timestamps, IDs, or IP addresses, and replace them with specific tokens before clustering. This normalizes messages that differ only in those values so they can be grouped into the same pattern.
+
+### Why are my JSON-only or fully structured logs not being reduced?
+
+On the Patterns card, set Build patterns from to 'Message, or attributes when empty', 'Attributes only', or 'Message and attributes'. The default, 'Message only', passes empty-message events through without reduction. Selected attribute keys define exact grouping boundaries. With 'Compare values', selected values also participate in similarity matching; with 'Ignore values', they do not. Summaries of patterns without message text have a null message.
+
+### Can I add my own masks?
+
+Yes. Click Add Custom Mask in the Mask Configuration section. The mask name must contain only lowercase letters, be no more than 50 characters long, and be unique across all masks in the reducer. Then enter the regular expression pattern to apply.
+
+### Which regular expression features can I use in a custom mask?
+
+A custom mask pattern supports common regular expression constructs, including literal characters and character classes such as \[a-z], \d, \w, and \s; quantifiers such as \*, +, ?, and {n,m}; alternation and grouping; and anchors and word boundaries such as ^, $, and \b. Lookahead and lookbehind assertions and backreferences are not supported. A pattern that uses an unsupported construct passes the form's validation but prevents the reducer from processing messages, so test a custom pattern against sample messages.
+
+### What are delimiters and when should I change them?
+
+Delimiters are the characters the reducer uses to split messages into tokens during clustering. The defaults cover common punctuation and whitespace but omit @ and . so email addresses and dotted identifiers are not split. Change the list when your log format uses other separators, such as a custom field delimiter.
+
+### How do I add a tab, newline, or carriage return delimiter?
+
+Enter the escape sequence in the Add delimiter field: \t for tab, \n for newline, or \r for carriage return. Additional control-character escape sequences, including \b, \f, \v, and \0, are also supported.
+
+### What does each attribute merge strategy do?
+
+Exact Match keeps a single value if all messages agree and stores a wildcard otherwise. Sample collects up to a limit of values and drops overflow. Preserve All collects every value up to a limit and emits the current summary when the limit would be exceeded. Sum, Min, Max, and Average compute the corresponding aggregation over numeric values, ignoring non-numeric values.
+
+### What is the difference between Sample and Preserve All?
+
+Sample stops collecting once its limit is reached and discards any further values in the same summary. Preserve All emits the current summary and starts a new one when adding more values would exceed its limit, so no value is discarded. Sample has a lower default limit; Preserve All has a higher default limit.
+
+### When does the 'Keep unique values only' checkbox matter?
+
+For Sample and Preserve All strategies, it controls how collected values are stored. When selected, values are stored as a set and duplicates are discarded. When cleared, values are stored as a list, and duplicates are preserved.
+
+### How do I target a specific attribute with its own strategy?
+
+Add an entry under Attribute merging strategies, enter the attribute path using dot notation or a JSON array, and configure one or more strategies. Paths must be unique within the list, and each entry must have a non-empty path.
+
+### What happens when I select multiple strategies for the same attribute?
+
+The reducer produces a separate attribute in the summary for each strategy, suffixed with the strategy name, such as _sum or _avg. The first strategy in the list also writes to the base attribute name. Each strategy type can be used only once per attribute path.
+
+### What does the default strategy apply to?
+
+The default strategy applies to every attribute path that does not have an explicit entry in Attribute merging strategies. Exact Match is the default, but you can configure one or more other strategies. As with path-specific strategies, additional default strategies produce suffixed attributes and require more processing.