@grepr/cli 1.11.5-ca28c41 → 1.11.5-dbd05e7

package/build/dist/docs-index/a2b96132-120f-4e1a-8007-fb801f60e28f.txt

@@ -1,244 +0,0 @@
-Configure the Grepr log reducer to reduce the volume of logs forwarded to your observability vendor while preserving access to your raw logs.
-
-# Configure the log reducer in a Grepr pipeline
-
-This page describes the transformations the log reducer applies to incoming log messages and how to configure them to meet your requirements. When you create log reduction pipelines, Grepr adds a log reducer with default settings that control how log messages are aggregated, grouped, tokenized, and summarized. These settings are suitable for many log-processing workloads, but you can modify these settings to meet your requirements through the Grepr UI, the REST API, or the Grepr CLI. This page describes each setting and how to configure the setting in the UI. For the API documentation, see the [LogReducer API specification](/apis/api-spec#schema/LogReducer). For the CLI, see [The Grepr CLI](/cli).
-
-The log reducer transforms logs into summarized events through a multi-step process. The reducer divides the incoming log stream into consecutive *aggregation windows* and produces summaries for each window separately. An aggregation window is a fixed-duration, tumbling window over which the reducer counts and aggregates the messages that match each pattern. Tumbling windows are contiguous and non-overlapping, so each log belongs to exactly one window. The aggregation window is two minutes by default.
-
-> To change the aggregation window duration, you must use the Grepr REST API. You cannot use the Grepr UI to change the window duration. Before changing this value, Grepr recommends contacting <support@grepr.ai> for guidance.
-
-The following describes the core steps in this process and includes links with details on changing the default behavior for each step:
-
-* Masking replaces dynamic values such as timestamps, IDs, and IP addresses with specific tokens. By normalizing values that represent the same entity but might differ slightly in format or content, masking prevents them from being split incorrectly in the next step. See [Enable grouping of dynamic fields by assigning common tokens](#mask-configuration).
-* Tokenizing breaks log messages into tokens based on configurable delimiters, creating a structured representation optimized for grouping. See [Configure delimiters used to split log messages](#delimiter-configuration).
-* Grouping uses similarity metrics to group messages into patterns. A configurable similarity threshold determines how closely messages must match to be considered part of the same pattern. You can use grouping to retain the values of important fields by excluding them from aggregation. See [Configure how messages are grouped](#grouping-configuration).
-* Sampling sends a configurable number of raw sample messages per pattern. This number is referred to as the *deduplication threshold* throughout this documentation. After this threshold is reached, you can configure the reducer to stop forwarding that pattern until the end of the aggregation window, or use logarithmic sampling. See [Configure a sampling strategy](#sampling-configuration).
-* Summarizing generates, at the end of each aggregation window, concise summaries of aggregated messages that matched an identified pattern. For details on the fields included in the summarized output events, see [Summary message metadata](/transforms/reducer#summary-metadata).
-
-To modify the configuration settings in the UI, on the overview page for your pipeline, click **Reducer** in the left-hand navigation menu to open the **Edit Reducer** dialog. The dialog contains a form to configure **Grouping Configuration**, **Aggregation Configuration**, **Sampling Configuration**, **Mask Configuration**, **Delimiter Configuration**, and **Attribute Configuration** values. The following sections describe these settings.
-
-## Configure how messages are grouped \[#grouping-configuration]
-
-Grouping controls how incoming messages are partitioned into independent groups before pattern detection, and the level of similarity messages must meet to be considered part of the same pattern.
-
-### Group messages by a field \[#group-by-configuration]
-
-Messages are aggregated separately for each unique combination of values across the fields you group on, so patterns from different services, hosts, or other dimensions are never merged. Grouping on `service`, `host`, and the `http.url.path` attribute aggregates messages independently for each unique combination of those three values.
-
-To group on a field, on the pipeline's **Reducer** step, click **Edit** on the **Fields** card, find the field, and set its behavior to **Match or split**. A field with this behavior is kept out of aggregation, so messages whose values for it differ are never merged into one pattern.
-
-Attribute paths are written with an `@` prefix and dot notation for nested attributes, such as `@http.url.path`. Tag keys are written as the key name alone.
-
-Use grouping to retain the values of important fields by excluding them from aggregation, such as high-cardinality identifiers like user IDs. To keep a field that is currently inside the message, parse it into an attribute or a tag with the **JSON Parser** or **Grok Parser** step earlier in the pipeline, then group on that field. See [Transform JSON strings into JSON objects](/transforms/json-parsing) and [Parse log messages to enrich log events with the Grok parser](/transforms/grok).
-
-Grouping also decides how a summary can be routed, because a summary carries the tag values of the fields it was grouped on. To use grouping to send each group's logs to a different destination, see [_](/solution-guides/route-logs-to-several-accounts).
-
-### Similarity threshold (%)
-
-The similarity threshold controls how close a set of tokens must be to an existing pattern for the reducer to consider the message a match for that pattern. To set this threshold, enter a whole number from 20 to 100. This value represents the percentage of tokens that must match between the incoming message and the pattern.
-
-* A higher value requires a closer match to group messages into the same pattern. Higher values produce more distinct patterns, each containing fewer of the tokens defined by [regular expression masks](#mask-configuration), and reduce volume less aggressively.
-* A lower value allows grouping messages with more variation into the same pattern. Lower values lead to greater reduction, but the resulting summary messages contain more mask-defined tokens.
-
-## Configure how messages are aggregated \[#aggregation-configuration]
-
-The **Aggregation Configuration** section controls how many raw messages are forwarded before aggregation begins for a pattern and how summary messages are formatted.
-
-### Minimum number of samples
-
-To set a *deduplication threshold* for messages, in the **Minimum number of samples** field, enter a whole number of two or greater. This value is the minimum count of messages matching a pattern that the reducer forwards as raw messages inside a single aggregation window before it begins aggregating additional matches into a summary message. Until this threshold is reached, unaggregated messages are forwarded to your sinks.
-
-Use a lower value to begin aggregation sooner and maximize volume reduction. Use a higher value when a pattern has a higher threshold for the number of duplicate messages to forward before beginning aggregation.
-
-### Add repeat count prefix to summary messages
-
-To have the reducer prepend a prefix to each summary message that indicates the number of raw messages aggregated and the duration of the aggregation period, select **Add repeat count prefix to summary messages**.
-
-Clear **Add repeat count prefix to summary messages** if your downstream system needs to parse the summary message body without changes. For example, if summary messages use JSON formatting, the leading prefix would cause a syntax error.
-
-## Configure a sampling strategy \[#sampling-configuration]
-
-The **Sampling Configuration** section controls whether and how additional raw messages are forwarded for a pattern after the minimum number of samples is reached in the current aggregation window. To configure the sampling strategy for a reducer, in the **Sampling strategy** menu, select one of the following options:
-
-### No Additional Sampling
-
-During each aggregation window, Grepr passes through a configurable number of sample messages for each pattern before beginning aggregation. After the deduplication threshold is reached for a specific pattern, Grepr stops sending messages for that pattern until the end of the aggregation window, at which point a summary is emitted. This ensures that a minimum number of raw messages always pass through unmodified. Low-frequency messages that usually contain important troubleshooting information pass through unmodified.
-
-While this default behavior maximizes reduction, the actual count of messages seen at the vendor is independent of the number of original raw messages. Instead, the count is based on the messages summarized in each aggregation window. Features in external log aggregators that depend on message counts by pattern, such as Datadog's "top patterns" capability, might not work as expected. To address this, select the following logarithmic sampling option.
-
-### Window-Based Logarithmic
-
-With logarithmic sampling, high-volume patterns that occur in the aggregation window are sampled more heavily than low-volume patterns.
-
-When you select this option, to configure the rate of additional sampling, enter a whole number of two or greater in the **Logarithm Base** field. The reducer process uses this base value to increase the sample size logarithmically. For example, if the base is set to two and the deduplication threshold is set to four, Grepr sends one additional sample message when the number of messages in the aggregation window first exceeds two raised to the power of the deduplication threshold + 1 (`2^5 or 32`). Additional samples are then sent at the 64th, 128th, 256th, and subsequent messages. This ensures that an unexpectedly large increase in messages matching a pattern still appears as an increase in message count at the vendor, but with a logarithmically smaller magnitude.
-
-## Enable grouping of dynamic fields by assigning common tokens \[#mask-configuration]
-
-The **Mask Configuration** section defines the regular expression masks that the reducer uses to replace variable parts of a log message, such as timestamps, IDs, or IP addresses, with specific tokens before the message is tokenized and grouped. Masking normalizes variable content so that messages that differ only in these values are grouped into the same pattern.
-
-The reducer ships with the following default masks. Each appears as a row in the **Mask Configuration** section and can be enabled, disabled, edited, or reset to its original pattern:
-
-| Mask | Default state | Description |
-| --- | --- | --- |
-| `timestamp` | Enabled | Matches many timestamp formats, such as `2024-04-26T15:30:45.123Z`, `26/04/2024 15:30:45`, or `15:30:45`. |
-| `ipport` | Enabled | Matches IPv4 addresses and optional ports, such as `192.168.0.1` or `192.168.0.1:8080`. |
-| `number` | Enabled | Matches integers and decimal numbers up to 100 digits. |
-| `uuid` | Enabled | Matches UUIDs, such as `123e4567-e89b-12d3-a456-426614174000`. |
-| `awsarn` | Disabled | Matches AWS ARNs. |
-| `awstoken` | Disabled | Matches AWS session tokens. |
-
-For each mask row, you can:
-
-* Select or clear the checkbox on the left to enable or disable the mask. Only enabled masks are applied to incoming messages.
-* Click **Edit** to display the regular expression for the mask and modify the pattern. The pattern field cannot be empty.
-* For default masks, click the restore icon to reset the pattern to its original value.
-* For custom masks, click the delete icon to remove the mask.
-
-### Enable grouping of non-default formats by assigning custom tokens
-
-To ensure the correct grouping of fields with formats that aren't handled by the default masks, you can create custom masks. For example, you might have a field with values that include delimiter characters, and you need to prevent the reducer from splitting the single field on those delimiters.
-
-When you add a custom mask, you must enter a name for the mask that:
-
-* Contains only lowercase alphabetic characters. Spaces, digits, and symbols are not allowed.
-* Is no more than 50 characters long.
-* Is unique across all masks in this reducer.
-
-Enter the mask in the **Regular Expression Pattern** text box. A custom mask pattern supports the following common regular expression constructs:
-
-* Literal characters and character classes, such as `[a-z]`, along with predefined classes such as `\d`, `\w`, and `\s`.
-* Quantifiers, such as `*`, `+`, `?`, and `{n,m}`.
-* Alternation with `|` and grouping with `(` and `)`.
-* Anchors and word boundaries, such as `^`, `$`, and `\b`.
-
-> Custom mask patterns do not support lookahead or lookbehind assertions or backreferences. A pattern that uses one of these constructs passes the validation in the form but prevents the reducer from processing messages. To confirm that a custom mask matches the values you expect, test the pattern against sample messages.
-
-## Configure delimiters used to split log messages \[#delimiter-configuration]
-
-The **Delimiter Configuration** section defines the characters that the reducer uses to split a log message into tokens during clustering. After masking, the reducer splits each message into tokens using these characters and then calculates similarity against known patterns. Characters that are not in the delimiter list are treated as part of a token.
-
-The default delimiter set contains common punctuation and whitespace characters: `:`, `#`, `[`, `]`, `(`, `)`, `{`, `}`, `|`, `,`, `;`, `"`, `'`, space, `\t`, `\n`, and `\r`.
-
-> The default set intentionally omits `@` and `.` so that email addresses and dotted identifiers, such as fully qualified class names, are not split across tokens.
-
-To modify the delimiter list:
-
-* To remove a delimiter, hover over its chip in the delimiter list and click the delete icon that appears.
-* To add a delimiter, enter the character in the **Add delimiter** field and click **Add** or press **Enter**. For control characters, enter the escape sequence: `\t` for tab, `\n` for newline, `\r` for carriage return, `\b` for backspace, `\f` for form feed, `\v` for vertical tab, or `\0` for null. Duplicate delimiters are rejected.
-* To restore the default delimiter set, click **Reset to Defaults**.
-
-## Configure how attributes are merged \[#attribute-configuration]
-
-To group log messages, the log reducer uses only the text in the `message` field. However, because of the possible variability of the attribute values associated with the grouped messages, the reducer provides multiple strategies to merge them.
-
-To control how the reducer combines attributes, you use the **Attribute Configuration** section to specify merge strategies for individual attribute paths and configure a default strategy that applies to any attribute path without an explicit configuration.
-
-The reducer supports the following merge strategy types:
-
-| Strategy | Behavior |
-| --- | --- |
-| **Exact Match** | If all aggregated messages have the same value for the attribute, the summary preserves that value. If they differ, the summary stores a wildcard, `*`. |
-| **Sample** | Collects up to a configured **Sample Limit** of values from the aggregated messages. When the limit is reached, additional values are dropped. |
-| **Preserve All** | Collects every distinct or repeated value from the aggregated messages, up to a configured **Preserve Limit**. When the incoming data exceeds the limit, the reducer emits the current summary and starts a new one, rather than dropping values. |
-| **Sum** | Sums numeric values across aggregated messages and stores the result as a scalar number. Non-numeric values are ignored. Collections of numbers are summed element by element. |
-| **Min** | Keeps the minimum numeric value across aggregated messages. Non-numeric values are ignored. |
-| **Max** | Keeps the maximum numeric value across aggregated messages. Non-numeric values are ignored. |
-| **Average** | Computes the arithmetic mean of numeric values across aggregated messages. Non-numeric values are ignored. |
-
-To see examples and learn more about the merge strategies, see [Merge strategy examples](/transforms/reducer/configuration/merge-strategy-examples).
-
-### Configure merge strategies
-
-To add a merge strategy, in the **Attribute merging strategies** section, click the plus (+) icon. Select a strategy from the **Strategy** menu and enter the target attribute path. If any part of the target path uses dot notation, enter the target path using dot notation, such as `http.response.bytes`, or a JSON array, such as `["http", "response", "bytes"]`. Each entry must have a non-empty path, and each path must be unique within the list.
-
-When you select **Sample** or **Preserve All**, you can configure the following settings:
-
-* **Sample Limit** or **Preserve Limit**: A whole number that's one or greater. These values cap the size of the collected values. **Sample** defaults to 10 and **Preserve All** defaults to 1000.
-* **Keep unique values only**: When selected, the reducer stores the collected values as a set and discards duplicates. When cleared, the reducer stores the values as a list and preserves duplicates.
-
-When you configure more than one strategy for the same attribute path, the reducer writes a separate attribute in the summary for each strategy, using a suffix such as `_sum`, `_avg`, or `_min`. The first strategy in the list also writes to the base attribute name. For example, configuring **Sum** and then **Average** for `bytes_sent` produces `bytes_sent_sum`, which is also written to `bytes_sent`, and `bytes_sent_avg`. To change the order in which strategies are applied, drag them in the list.
-
-> Adding multiple strategies, whether to a specific path or the default, adds additional attributes to the summary and requires more processing per message. Use multiple strategies only when your analytics or dashboards need more than one aggregated form of the same attribute.
-
-Each strategy type can be used only once per attribute path.
-
-### Configure the default strategy
-
-The **Default Strategy** section defines the strategies that apply to every attribute path that is not configured under **Attribute merging strategies**. You must configure at least one default strategy. **Exact Match** is the default.
-
-As with path-specific strategies, you can add multiple default strategies. When you add multiple default strategies, each strategy produces a suffixed attribute in the summary, and the first strategy also writes to the base attribute name.
-
-## Frequently Asked Questions
-
-### What is the aggregation window?
-
-The aggregation window is a fixed-duration, tumbling window over which the reducer counts and aggregates the messages that match each pattern. The reducer divides the incoming log stream into consecutive windows and produces a summary separately for each window. The window is two minutes by default. You cannot change the window duration in the Grepr UI; you must use the REST API, and Grepr recommends contacting support for guidance before changing it.
-
-### How do I prevent messages from different services being aggregated together?
-
-Add the tag key that identifies each service, such as service or env, to the Group-by values field. The reducer aggregates messages independently for each unique combination of group-by values, so patterns from different services are never merged.
-
-### What does the similarity threshold control?
-
-The similarity threshold is the percentage of tokens that must match between an incoming message and an existing pattern for the message to be considered part of that pattern. Higher values produce more distinct patterns and less aggressive reduction. Lower values produce broader patterns and more aggressive reduction, at the cost of more mask-defined tokens in each summary.
-
-### Why should I raise the minimum number of samples?
-
-The minimum number of samples is how many raw matching messages are forwarded for a pattern in an aggregation window before aggregation begins. Raise it when you want more raw samples of each pattern to be sent to your observability backend. Lower it to begin aggregation sooner and reduce more volume.
-
-### When should I disable the repeat count prefix?
-
-Disable the repeat count prefix when downstream systems parse the summary message body directly, such as when summary messages contain JSON. The prefix would otherwise prepend non-JSON text and break parsing.
-
-### What's the difference between the two sampling strategies?
-
-No Additional Sampling stops forwarding raw messages for a pattern as soon as the minimum number of samples has been reached in an aggregation window. Window-Based Logarithmic continues to forward additional raw samples at a logarithmically decreasing rate, preserving more visibility into high-volume patterns.
-
-### How does the logarithm base affect sampling?
-
-With Window-Based Logarithmic sampling, a larger logarithm base forwards fewer additional samples as message volume grows, and a smaller base forwards more. The base must be at least two.
-
-### What are masks and why does the reducer apply them?
-
-Masks are regular expressions that identify variable parts of a log message, such as timestamps, IDs, or IP addresses, and replace them with specific tokens before clustering. This normalizes messages that differ only in those values so they can be grouped into the same pattern.
-
-### Can I add my own masks?
-
-Yes. Click Add Custom Mask in the Mask Configuration section. The mask name must contain only lowercase letters, be no more than 50 characters long, and be unique across all masks in the reducer. Then enter the regular expression pattern to apply.
-
-### Which regular expression features can I use in a custom mask?
-
-A custom mask pattern supports common regular expression constructs, including literal characters and character classes such as \[a-z], \d, \w, and \s; quantifiers such as \*, +, ?, and {n,m}; alternation and grouping; and anchors and word boundaries such as ^, $, and \b. Lookahead and lookbehind assertions and backreferences are not supported. A pattern that uses an unsupported construct passes the form's validation but prevents the reducer from processing messages, so test a custom pattern against sample messages.
-
-### What are delimiters and when should I change them?
-
-Delimiters are the characters the reducer uses to split messages into tokens during clustering. The defaults cover common punctuation and whitespace but omit @ and . so email addresses and dotted identifiers are not split. Change the list when your log format uses other separators, such as a custom field delimiter.
-
-### How do I add a tab, newline, or carriage return delimiter?
-
-Enter the escape sequence in the Add delimiter field: \t for tab, \n for newline, or \r for carriage return. Additional control-character escape sequences, including \b, \f, \v, and \0, are also supported.
-
-### What does each attribute merge strategy do?
-
-Exact Match keeps a single value if all messages agree and stores a wildcard otherwise. Sample collects up to a limit of values and drops overflow. Preserve All collects every value up to a limit and emits the current summary when the limit would be exceeded. Sum, Min, Max, and Average compute the corresponding aggregation over numeric values, ignoring non-numeric values.
-
-### What is the difference between Sample and Preserve All?
-
-Sample stops collecting once its limit is reached and discards any further values in the same summary. Preserve All emits the current summary and starts a new one when adding more values would exceed its limit, so no value is discarded. Sample has a lower default limit; Preserve All has a higher default limit.
-
-### When does the 'Keep unique values only' checkbox matter?
-
-For Sample and Preserve All strategies, it controls how collected values are stored. When selected, values are stored as a set and duplicates are discarded. When cleared, values are stored as a list, and duplicates are preserved.
-
-### How do I target a specific attribute with its own strategy?
-
-Add an entry under Attribute merging strategies, enter the attribute path using dot notation or a JSON array, and configure one or more strategies. Paths must be unique within the list, and each entry must have a non-empty path.
-
-### What happens when I select multiple strategies for the same attribute?
-
-The reducer produces a separate attribute in the summary for each strategy, suffixed with the strategy name, such as _sum or _avg. The first strategy in the list also writes to the base attribute name. Each strategy type can be used only once per attribute path.
-
-### What does the default strategy apply to?
-
-The default strategy applies to every attribute path that does not have an explicit entry in Attribute merging strategies. Exact Match is the default, but you can configure one or more other strategies. As with path-specific strategies, additional default strategies produce suffixed attributes and require more processing.

package/build/dist/docs-index/a60fd596-a4f3-4191-9892-dea555ae2597.txt

@@ -1,50 +0,0 @@
-# ReadOrgConstraints
-
-**Type**: `object`
-
-## Properties
-
-- name: maxAnthropicIntegrations (optional)
-  type: integer
-
-- name: maxDataWarehouseIntegrations (optional)
-  type: integer
-
-- name: maxDatadogApiKeysPerIntegration (optional)
-  type: integer
-
-- name: maxDatadogIntegrations (optional)
-  type: integer
-
-- name: maxGeminiIntegrations (optional)
-  type: integer
-
-- name: maxLiteLlmIntegrations (optional)
-  type: integer
-
-- name: maxMcpIntegrations (optional)
-  type: integer
-
-- name: maxNewRelicIntegrations (optional)
-  type: integer
-
-- name: maxOpenAiIntegrations (optional)
-  type: integer
-
-- name: maxOtlpIntegrations (optional)
-  type: integer
-
-- name: maxPipelines (optional)
-  type: integer
-
-- name: maxS3DataWarehouseIntegrations (optional)
-  type: integer
-
-- name: maxS3VectorIndexIntegrations (optional)
-  type: integer
-
-- name: maxSplunkIntegrations (optional)
-  type: integer
-
-- name: maxSumoIntegrations (optional)
-  type: integer

package/build/dist/docs-index/a79f0674-f22d-4d6b-805c-786e4417aeb0.txt

@@ -1,40 +0,0 @@
-# Preview a vendor estimation run
-
-**POST** `/v1/integrations/{integrationId}/exceptions/vendor-estimation-runs/preview`
-
-**Operation ID**: `previewVendorEstimationRun`
-**Summary**: Preview a vendor estimation run
-Classifies the exceptions a run would cover — every one the integration has imported — and returns how many distinct queries it would send to the vendor and which exceptions cannot be counted because their query never parsed. Writes nothing.
-
-## Tags
-- Integration Exceptions
-## Parameters
-
-- **integrationId** (required) - path
-  - Integration id
-  - Type: string
-
-## Request Body
-
-**Content-Type**: `application/json`
-
-**Schema**: `CreateVendorEstimationRunRequest`
-
-## Responses
-
-### 200
-
-Preview computed
-
-- **Content-Type**: `application/json` / `VendorEstimationRunPreview`
-### 400
-
-Bad Request - Window is inverted or scope is invalid.
-
-### 401
-
-Unauthorized
-
-### 404
-
-Not Found - Integration not found.

package/build/dist/docs-index/a8576386-37fd-4ed5-b8d7-779ff364f00d.txt

@@ -1,44 +0,0 @@
-# Start a vendor estimation run
-
-**POST** `/v1/integrations/{integrationId}/exceptions/vendor-estimation-runs`
-
-**Operation ID**: `createVendorEstimationRun`
-**Summary**: Start a vendor estimation run
-Creates a run over the given scope and window covering every exception the integration has imported, with one result row per distinct predicate, and queues its first slice. One non-terminal run per organization: starting a second returns 409 while the first is still in flight.
-
-## Tags
-- Integration Exceptions
-## Parameters
-
-- **integrationId** (required) - path
-  - Integration id
-  - Type: string
-
-## Request Body
-
-**Content-Type**: `application/json`
-
-**Schema**: `CreateVendorEstimationRunRequest`
-
-## Responses
-
-### 201
-
-Run created
-
-- **Content-Type**: `application/json` / `VendorEstimationRun`
-### 400
-
-Bad Request - Window is inverted or scope is invalid.
-
-### 401
-
-Unauthorized
-
-### 404
-
-Not Found - Integration not found.
-
-### 409
-
-Conflict - The organization already has a run that has not finished.

package/build/dist/docs-index/b4c452f7-2dfd-4254-9821-36203148d230.txt

@@ -1,18 +0,0 @@
-# Get the current maxTokens window of one LLM integration
-
-**GET** `/v1/llm-usage/{integrationId}`
-
-**Operation ID**: `currentWindow`
-**Summary**: Get the current maxTokens window of one LLM integration
-## Parameters
-
-- **integrationId** (required) - path
-  - Type: string
-
-## Responses
-
-### 200
-
-Current window retrieved
-
-- **Content-Type**: `application/json` / `LlmUsageWindow`

package/build/dist/docs-index/b8754b51-a6aa-453e-95f5-da9747c85f72.txt

@@ -1,217 +0,0 @@
-Learn how to configure how Grepr integrates with observability platforms and storage systems to support data ingestion, processing, and output in your pipelines.
-
-# Configure how Grepr integrates with observability platforms and cloud storage
-
-To support data ingestion and data output for your Grepr pipelines, you connect Grepr to external observability platforms and storage systems. To support these connections, the Grepr platform uses *integrations*. A Grepr integration provides the configuration details to connect to an external system, enabling data exchange between Grepr and that system.
-
-After configuring an integration, you can use it in your Grepr pipelines to create *sources* to ingest data from agents and collectors, such as FluentD or the OpenTelemetry collector, and *sinks* to send data to an observability platform or storage.
-
-You can use the Grepr UI or the REST API to create and manage integrations. To use the UI, see [Manage integrations in the Grepr UI](#manage-integrations-in-ui). To use the API, see the documentation for each [supported integration](#supported-integrations).
-
-## Integration types
-
-The Grepr platform includes two integration types:
-
-* Observability platform integrations: Connections to monitoring, logging, and APM platforms, such as Datadog, Splunk, New Relic, and Sumo Logic.
-* Cloud storage integrations: Connections to cloud storage. The Grepr data lake uses cloud storage to store your raw data and metadata.
-
-## Supported platform and storage integrations \[#supported-integrations]
-
-The Grepr platform includes the following integrations. For details on configuring and using an integration, refer to the linked documentation page. For a complete list of supported integrations, supported functionality, and details on vendor-specific capabilities with some integrations, see [Grepr supported vendor and storage integrations](/integrations/support-matrix).
-
-### Datadog
-
-The Datadog integration connects to the Datadog observability platform and supports sourcing data from several Datadog-compatible collectors and sending processed data to Datadog.
-
-See [Configure Datadog connections with a Grepr integration](/integrations/datadog).
-
-### New Relic
-
-The New Relic integration connects to New Relic's observability platform. Use this integration to create sources that retrieve log data from New Relic collectors and sinks that send processed data to New Relic for monitoring and analysis.
-
-See [Configure New Relic connections with a Grepr integration](/integrations/new-relic).
-
-### Splunk
-
-The Splunk integration provides connectivity to Splunk through Splunk's HTTP Event Collector (HEC) or S2S over HTTP. This integration supports reading data from Splunk collectors and writing data to Splunk indexes.
-
-See [Configure Splunk connections with a Grepr integration](/integrations/splunk).
-
-### Sumo Logic
-
-The Sumo Logic integration connects to Sumo Logic's cloud-based log management and monitoring platform using a Sumo Logic HTTP Source. You can use the integration to create sources that ingest data from Sumo Logic collectors and sinks that send processed data to Sumo Logic.
-
-See [Configure Sumo Logic connections with a Grepr integration](/integrations/sumo-logic).
-
-### OpenTelemetry
-
-The Grepr OpenTelemetry integration lets you ingest logs and Application Performance Monitoring (APM) data into Grepr from a collector that supports the OpenTelemetry Protocol over HTTP, or OTLP/HTTP, process them with a Grepr pipeline, and then send the processed data to any observability vendor that accepts OTLP/HTTP requests.
-
-See [Configure connections to OpenTelemetry compatible platforms](/integrations/open-telemetry).
-
-### Grafana Cloud
-
-You can configure Grepr's OpenTelemetry integration to connect with Grafana Cloud using the OpenTelemetry Protocol (OTLP). Use an OpenTelemetry integration to create sources that ingest data from OTLP-compatible collectors and sinks that send processed log data to Grafana Cloud.
-
-See [Configure Grafana Cloud connections with a Grepr integration](/integrations/grafana).
-
-### Amazon CloudWatch
-
-You can configure Grepr's OpenTelemetry integration to connect with Amazon CloudWatch using the OpenTelemetry Protocol (OTLP). Use an OpenTelemetry integration to create sources that ingest data from OTLP-compatible collectors and sinks that send processed log data to a CloudWatch log group.
-
-See [Configure Amazon CloudWatch connections with a Grepr integration](/integrations/cloudwatch).
-
-### Google Cloud Platform (GCP)
-
-The GCP integration as a source enables you to stream logs from Google Cloud services to Grepr for processing and analysis, and then forward the processed logs to any sink created using a Grepr-supported integration, such as Datadog, Splunk, or New Relic.
-
-See [Google Cloud Platform observability with Grepr](/integrations/gcp).
-
-### Cloud storage
-
-The Grepr data lake uses Amazon S3 to store, manage, and support querying the raw data from your Grepr pipelines. Depending on your requirements, you can choose from two S3 integrations: a Grepr-hosted integration, which uses a Grepr-managed S3 bucket, and an integration that uses an S3 bucket in your account. Both integrations provide secure, managed, and efficient storage for your data, along with an optimized query interface.
-
-To configure the Grepr-hosted integration, you only need to assign a name to the integration. The Grepr platform manages the S3 bucket, including its deployment and configuration.
-
-Using the integration with your own S3 bucket requires that you deploy and configure an S3 bucket in your AWS account, but it provides more control over your data. See [Host a Grepr data lake with the Amazon S3 integration](/integrations/aws-s3).
-
-## Manage integrations in the Grepr UI \[#manage-integrations-in-ui]
-
-The **Integrations** page displays your integrations as cards, organized by observability vendor integrations and storage integrations. In each section, cards are ordered by recent activity, with the most active integrations appearing first.
-
-In addition to the integration name and its type, for example, a Datadog integration, if the integration is assigned to one or more teams, the team names appear in the header of the card.
-
-You can get a visual overview of the health and usage of your integrations using the charts included on each integration card. If an integration has recent activity, such as requests received from agents by a vendor integration or the amount of data written to a storage integration, a time-series chart of that activity is shown on the card.
-
-* Observability vendor integrations display requests received from agents over the last 30 minutes, color-coded by HTTP status code.
-
-  * To view more detailed metrics or more details about a pipeline that uses the integration, select it from the **Pipelines** menu.
-
-* Storage integrations display data written to the integration over the last 24 hours, broken down by dataset.
-
-  * To view and query a dataset stored on a storage integration, select it from the **Datasets** menu.
-
-To view more details about the metrics in either chart type, hover over any point in the chart to see a tooltip with the value and timestamp for that point.
-
-### Add an integration
-
-To create an integration, click **Add new** next to the section heading for the type you want to add. To learn how to create and configure specific integrations, see the documentation for each [supported integration](#supported-integrations).
-
-### Edit an integration
-
-To edit an integration, click the three-dot menu on the integration card and select **Edit**. Update the fields in the dialog and click **Save**.
-
-### Delete an integration
-
-To delete an integration, click the three-dot menu on the integration card and select **Delete**. Confirm the deletion in the dialog that appears.
-
-### Identify log events that should bypass aggregation \[#passthrough-estimation]
-
-When you create an integration with an observability vendor that supports *exception parsing*, Grepr automatically parses queries used by monitors, dashboards, or alerts in your observability vendor platform. You can use this list of queries to identify log messages that should be treated as *exceptions* to log reduction in Grepr pipelines. In other words, messages that should bypass aggregation to ensure that critical alerts and dashboards continue to work as expected. To help with identifying exceptions, the Grepr UI displays a *passthrough percentage* estimate for each query. This estimate is the percentage of events, over a configurable time window, that would bypass aggregation.
-
-You can also choose to automatically add and remove exceptions by enabling **Auto-sync exceptions** when you add the integration to your pipelines. To learn more about enabling and using auto-sync when you create or edit a pipeline, see [Prevent aggregation of messages used in your observability platform](/transforms/reducer/exceptions#vendor-exceptions).
-
-If an integration does not support exception parsing, the **Parsed Alerts** button on the integration card is disabled. See [Supported integrations: observability platforms](/integrations/support-matrix#supported-observability-integrations).
-
-To view the log queries parsed from an integration and configure passthrough estimation:
-
-1. On the **Integrations** page, on the card for a vendor integration that supports query parsing, click **Parsed Alerts**.
-
-   The **Parsed alert queries** dialog displays a table with the following columns:
-
-   * **Id**: The unique identifier for this alert.
-   * **Name**: The alert or dashboard name from your vendor.
-   * **Exception Type**: The type of vendor resource the query came from, such as a monitor or dashboard widget.
-   * **Query**: The log query extracted from the alert or dashboard.
-   * **Passthrough %**: The percentage of raw logs in your configured dataset that match this query. This value is displayed only if you've configured passthrough estimation.
-   * **Last Updated**: When the passthrough percentage was last calculated. Hover over the value to see the exact date and time.
-
-   You can filter the table by name or query text and filter by exception type using the multi-select filter in the **Exception Type** column. To manage which columns are visible, click the **Manage columns** icon in the upper-right corner of the table.
-
-2. To configure passthrough estimation and view the percentage of logs that would bypass aggregation for each query, click **Configure Passthrough Estimation**.
-
-   The **Passthrough Estimation Configuration** dialog lists the datasets that pipelines using the integration write to. Each row shows whether passthrough estimation is configured for that dataset and which pipelines use it.
-
-   > When you configure passthrough estimation for a dataset, it applies to all integrations or pipelines that use that dataset as a sink.
-
-3. Expand the row for the dataset you want to configure.
-
-4. Under **Filter Query (Optional)**, select your preferred query syntax and enter a query to filter for logs to include in the estimation analysis. Use this field when your dataset contains logs from multiple sources, and you want Grepr to use only the logs relevant to this integration. Leave this field empty to include all logs in the dataset.
-
-5. In the **Lookback Duration** menu, select the time window for calculating percentages. Options range from 15 minutes to 24 hours. A shorter window scans less data, reducing costs, but may not be representative if your log patterns vary throughout the day. A longer window provides more representative results but increases the cost of the analysis queries.
-
-6. In the **Refresh Period** menu, select how often Grepr recomputes each estimation for this dataset. Options range from 1 hour to 30 days, with a default of 7 days. A shorter period uses more recent logs to calculate estimates, resulting in more up-to-date values, but increases query load on the data lake.
-
-7. In the **Log Limit** field, enter the maximum number of logs to scan per estimation run. The default is 10,000,000. Larger limits provide more representative results, but increase the cost of each estimation query.
-
-8. To save a new configuration, click **Enable**. To save edits to an existing configuration, click **Save changes**.
-
-Grepr's impact estimation cycle runs once an hour for any datasets due for refresh, based on each dataset's **Refresh Period**. New or recently changed queries are included in the next cycle. A first estimate typically appears within an hour, though it might take longer if a large number of queries are queued for processing.
-
-To stop estimating for a dataset, expand its row and click **Disable estimation**. The dataset remains in the list as **Not configured** so you can re-enable estimation later.
-
-### Interpret passthrough percentages
-
-> Although two queries might use the same log events, their estimates will be equal. Additionally, the passthrough rate remains constant and is not the sum of the estimates. For example, if you enable exceptions for two queries that use the same log events, the passthrough rate doesn't double, but remains equal to the rate when only one exception is enabled.
-
-The **Passthrough %** value for a query shows what fraction of your raw logs that alert would require to pass through without aggregation. Use these values to evaluate the impact of adding each alert as a pipeline exception:
-
-A percentage less than 1% is considered low, and an exception should have minimal impact on aggregation efficiency and storage.
-
-You should review carefully any percentage above 1% before creating an exception for the query.
-
-A dash (—) in the **Passthrough %** column indicates that no estimate is available yet, either because passthrough estimation is not configured, or because Grepr hasn't finished calculating the estimate. If passthrough estimation is configured, the first estimate typically appears within an hour, and later refreshes occur based on the dataset's **Refresh Period**.
-
-The **Passthrough Estimation Configuration** dialog also includes the **Recent Activity** table at the bottom. This section shows the most recent log events related to exception parsing, giving you visibility into recent integration activity without leaving the dialog. To view the complete log history for the integration, click **View all logs** to go to the **System Logs** page, pre-filtered to show all activity for the selected integration.
-
-### Configure passthrough estimation from a pipeline
-
-You can also configure passthrough estimation directly from a pipeline. On the pipeline details page, in the left-hand navigation menu, click **Exceptions**. In the **Actions** column for an **Avoid impacting dashboards/alerts** exception, click the **⋯** menu and select **Configure estimations** to open a dialog that includes all datasets configured as sinks for the pipeline. Any configuration changes you make to the dataset apply to every other integration or pipeline that populates that dataset.
-
-To have Grepr recompute impact percentages during the next impact estimation cycle, click the **⋯** menu and select **Reset estimations** to clear the cached impact percentages for the datasets configured as sinks for the pipeline. Use this when you've changed alert queries in your observability vendor and want updated estimates without waiting for the next scheduled refresh.
-
-## Limitations \[#limitations]
-
-Grepr might split log events over 900KB, causing them to lose their original structured format. Each split event includes a common `grepr.splitOriginId` attribute if you need to reconstruct the original message.
-
-## Frequently Asked Questions
-
-### What is a Grepr integration?
-
-A Grepr integration provides the configuration details needed to connect Grepr to an external system, such as an observability platform or cloud storage. Integrations enable data exchange between Grepr and external systems, allowing you to create sources for data ingestion and sinks for data output in your pipelines.
-
-### What types of integrations does Grepr support?
-
-Grepr supports two integration types: observability platform integrations (connections to monitoring, logging, and APM platforms like Datadog, Splunk, New Relic, and Sumo Logic) and cloud storage integrations (connections to cloud storage for the Grepr data lake).
-
-### What is a source in a Grepr pipeline?
-
-A source in a Grepr pipeline is an operator that ingests data from an external system or agent. Sources use integrations to connect to observability platforms or collectors like FluentD or OpenTelemetry to pull data into Grepr for processing.
-
-### What is a sink in a Grepr pipeline?
-
-A sink in a Grepr pipeline is an operator that outputs processed data to an external system. Sinks use integrations to send Grepr-processed logs to observability platforms, data lakes, or other storage systems based on your pipeline configuration.
-
-### Can I use multiple integrations in a single pipeline?
-
-Yes, you can configure pipelines with multiple sources to ingest data from different platforms and multiple sinks to send processed data to different destinations. This flexibility allows you to build complex multi-source, multi-destination data processing workflows.
-
-### Which observability platforms does Grepr support?
-
-Grepr supports integrations with Datadog, Splunk, New Relic, Sumo Logic, Grafana Cloud, and Google Cloud Platform (GCP). Each integration enables different source and sink capabilities depending on the platform's features and capabilities.
-
-### How do I configure a Grepr integration?
-
-To configure an integration, you need to provide connection details specific to your external system, such as API keys, tokens, endpoints, or credentials. The specific steps vary by integration type. Detailed instructions for each integration are available in the platform-specific documentation pages.
-
-### What is the Grepr data lake integration?
-
-The Grepr data lake uses Amazon S3 to store raw log data. You can configure either a Grepr-hosted integration (using a Grepr-managed S3 bucket) where you only provide a name, or an integration with your own AWS account (requiring you to configure and manage the S3 bucket yourself for more control).
-
-### What happens to logs older than 48 hours from external sources?
-
-Grepr might drop incoming data from external integration sources with timestamps older than 48 hours. This limitation helps optimize storage and processing while allowing you to retain raw logs in the data lake for longer-term analysis.
-
-### What happens when a log event is larger than 900KB?
-
-If a log event exceeds 900KB, Grepr might split it into smaller events, which can cause them to lose their original structured format. Each split event includes a common 'grepr.splitOriginId' attribute that you can use to reconstruct the original message from its component parts.