@grepr/cli 1.11.5 → 1.11.6-48496dd

package/build/dist/docs-index/7dae2da2-2ad4-4a46-83f6-5f9827e912fb.txt

@@ -1,66 +0,0 @@
-This page provides a comprehensive overview of the Grepr-supported integrations with observability platforms and cloud storage, and the features supported by those integrations.
-
-# Grepr supported vendor and storage integrations
-
-This page provides an overview of the observability platform and cloud storage integrations supported by the Grepr platform. This page also specifies the features supported by each integration, including whether it can be used as a source or a sink.
-
-> If you require an integration that Grepr does not support, contact <support@grepr.ai>.
-
-## Supported integrations: cloud storage
-
-Grepr supports Amazon S3 for storing, querying, and backfilling raw data. See [Host a Grepr data lake with the Amazon S3 integration](/integrations/aws-s3).
-
-## Supported integrations: observability platforms \[#supported-observability-integrations]
-
-A Grepr integration provides the configuration details needed to connect to an external system, such as an observability platform. Integrations support moving data into Grepr and between Grepr and external systems by enabling the creation of sources and sinks in your Grepr pipelines.
-
-Some integrations support the automatic creation of exceptions based on information returned by vendor tools. These exceptions ensure that Grepr passes through important messages without aggregation. For some integrations, this feature is available when the integration is enabled as a *cloud source*, which allows Grepr to read and process data from the external service. To learn more, see [Identify log events that should bypass aggregation](/integrations#passthrough-estimation)
-
-The following are the observability platform and tool integrations supported by Grepr as data sources and sinks:
-
-| Integration | As source | As sink | Exception parsing |
-| --- | --- | --- | --- |
-| Datadog | Yes | Yes | Yes. See [Enable Datadog selective reduction](/integrations/datadog#enable-selective-reduction). |
-| New Relic | Yes | Yes | Yes. See [Enable New Relic selective reduction](/integrations/new-relic#enable-selective-reduction). |
-| Splunk | HEC, HTTP | HEC only | No |
-| Google Cloud Platform | Yes | No \[^gcpsink] | Not applicable \[^gcpep] |
-| Grafana Cloud | Yes | Yes | No |
-| OpenTelemetry | Yes | Yes | Not applicable |
-| Sumo Logic | Yes | Yes | No |
-| Amazon S3 \[^awsstorage] | Yes | No | Not applicable |
-
-\[^gcpsink]: Although Grepr supports streaming data from GCP, you cannot use GCP as a sink. Instead, you configure a supported sink for the output from your Grepr pipeline. To learn more, see [Google Cloud Platform observability with Grepr](/integrations/gcp). \[^gcpep]: Because the exception parsing feature is configured as a separate step in your pipeline, support for this feature is based on the sink configured on the pipeline. \[^awsstorage]: The support mentioned in this table is for reading from arbitrary files in Amazon S3 and writing to files in arbitrary formats, such as JSON or plain text. To learn about using S3 as data lake storage, see [Host a Grepr data lake with the Amazon S3 integration](/integrations/aws-s3).
-
-## Frequently Asked Questions
-
-### Which observability platforms can I use as sources in Grepr?
-
-Grepr supports the following platforms as sources: Datadog, Splunk (HEC and HTTP), Google Cloud Platform, Grafana Cloud, New Relic, OpenTelemetry, Sumo Logic, and Amazon S3. Each platform provides different capabilities for ingesting data into Grepr pipelines.
-
-### Which observability platforms can I use as sinks in Grepr?
-
-Grepr supports the following platforms as sinks: Datadog, Splunk (HEC only, not HTTP), Grafana Cloud, New Relic, and OpenTelemetry. Note that GCP and Amazon S3 have specific limitations: GCP cannot be used as a sink, and S3 supports arbitrary format writing but not Iceberg data lake functionality.
-
-### What is exception parsing in Grepr integrations?
-
-Exception parsing is a feature that automatically creates exceptions from information returned by vendor tools, ensuring that important messages pass through without aggregation. This feature is supported by Datadog and Splunk integrations, and is available when the integration is enabled as a cloud source.
-
-### Does Grepr support OpenTelemetry?
-
-Yes, Grepr supports OpenTelemetry as both a source and a sink. You can use the OpenTelemetry Protocol (OTLP) to ingest data from compatible collectors or output processed logs to platforms like Grafana Cloud that accept OTLP data.
-
-### What is the difference between Splunk HEC and HTTP support?
-
-Splunk HEC (HTTP Event Collector) is supported for both source (reading) and sink (writing) operations. Splunk HTTP support is available as a source only. When configuring a Splunk sink, use the HEC integration for data output.
-
-### Can I use Google Cloud Platform (GCP) as a data sink?
-
-No, GCP cannot be used as a sink in Grepr pipelines. While Grepr supports streaming logs from GCP sources, for data output you must configure a different supported sink such as Datadog, Splunk, New Relic, Grafana Cloud, or OpenTelemetry.
-
-### How does exception parsing work with GCP integrations?
-
-Exception parsing for GCP is not applicable as a direct feature. Instead, the exception parsing feature is configured as a separate step in your pipeline, and support is based on the sink you configure for your pipeline output.
-
-### What cloud storage options does Grepr support?
-
-Grepr supports Amazon S3 for storing and querying raw data in the Grepr data lake. You can use either a Grepr-hosted S3 integration or configure your own S3 bucket in your AWS account. S3 can also be used as a source for reading arbitrary files in any format.

package/build/dist/docs-index/7f5e5121-38c8-4604-a32a-95f2528aab04.txt

@@ -1,17 +0,0 @@
-# CreateVendorEstimationRunRequest
-
-**Type**: `object`
-
-## Properties
-
-- name: scopeQuery (optional)
-  type: string
-  description: Vendor query limiting the estimate to part of the customer's traffic. Applied to both the matched count and the total. Leave empty for all logs.
-
-- name: windowEnd (required)
-  type: string
-  description: Absolute end of the window to count over; must be after the start
-
-- name: windowStart (required)
-  type: string
-  description: Absolute start of the window to count over

package/build/dist/docs-index/8b5faafd-ec31-4008-bbd9-76103ad81ab4.txt

@@ -1,37 +0,0 @@
-# QueryJobInfo
-
-Information about a query job.
-
-**Type**: `object`
-
-## Properties
-
-- name: flinkJobId (required)
-  type: `JobID`
-
-- name: integrationId (optional)
-  type: string
-  description: The integration id used for the query job. This determines where the results will be stored.
-
-- name: jobExpiry (required)
-  type: string
-  description: The time at which the job will expire. After this time, the job is bound to be terminated and any new results will not be made available.
-
-- name: metadata (optional)
-  type: `QueryJobMetadata`
-
-- name: nextHeartbeatTime (required)
-  type: string
-  description: The time after which the client should send a heartbeat. If no new heartbeats are received a certain amount of time after this, the job is bound to be terminated. To send heartbeats, do a query read.
-
-- name: resultPath (required)
-  type: string
-  description: The path where results from the query job will be available.
-
-- name: results (optional)
-  type: object
-  description: The results of the query job. The keys are the names of the result files and the values are the paths to them.
-
-- name: status (required)
-  type: enum: `PENDING`, `RUNNING`, `FINISHED`, `UNKNOWN`, `FAILED`, `TIMEOUT`, `CANCELLED`
-  description: The status of the query job.

package/build/dist/docs-index/8cea8038-f933-4f46-9213-38ea818cb53f.txt

@@ -1,16 +0,0 @@
-# QueryJobMetadata
-
-Metadata about a query job.
-
-**Type**: `object`
-
-## Properties
-
-- name: numTasksFinished (optional)
-  type: integer
-
-- name: numTasksPending (optional)
-  type: integer
-
-- name: numTasksRunning (optional)
-  type: integer

package/build/dist/docs-index/964c4a0e-d236-4b42-b7d4-05cde9e949c7.txt

@@ -1,21 +0,0 @@
-# VendorEstimationRunPreview
-
-**Type**: `object`
-
-## Properties
-
-- name: distinctQueries (required)
-  type: integer
-  description: Distinct countable predicates among those exceptions, and so the number of vendor calls the run would make. Smaller than the exception count wherever several exceptions carry the same query.
-
-- name: exceptions (required)
-  type: integer
-  description: Exceptions the run would cover: every one the integration has imported
-
-- name: queuedAhead (optional)
-  type: integer
-  description: Runs that would start before this one if it were created now. Absent when a slot is free.
-
-- name: unparseable (required)
-  type: array of `UnparseableException`
-  description: Exceptions whose query could not be parsed at import and so cannot be counted, each with the parse error

package/build/dist/docs-index/a6dd6059-7108-44df-8ea8-4edbcdd786cc.txt

@@ -1,26 +0,0 @@
-# Read
-
-**Type**: `object`
-
-## Properties
-
-- name: accessConfig (required)
-  type: `AccessConfig`
-
-- name: constraints (required)
-  type: `ReadOrgConstraints`
-
-- name: featureFlags (optional)
-  type: array of enum: `DATADOG_BROWSER_LOGS_INTEGRATION`, `METRICS_SOURCE`, `S3_LOGS_FILE_SOURCE`, `PIPELINE_TEMPLATES`, `AI_WORKFLOWS`, `SEARCH_ICEBERG_SOURCE`, `QUERY_JOBS`, `AUTH0_SSO_CONNECTION`, `TRINO_QUERY_ENGINE`, `UNKNOWN`
-
-- name: name (required)
-  type: string
-
-- name: plan (required)
-  type: `Plan`
-
-- name: ssoConnectionEnabled (optional)
-  type: boolean
-
-- name: ssoRequestedAt (optional)
-  type: string

package/build/dist/docs-index/ae8bc94a-cb1a-41e7-9e36-3b7250469d04.txt

@@ -1,17 +0,0 @@
-# TableConfigRead
-
-**Type**: `object`
-
-## Properties
-
-- name: effectiveSpec (required)
-  type: array of `EffectivePartitionField`
-  description: Full effective partition spec, including system-controlled rows.
-
-- name: promotedTagKeys (optional)
-  type: array of string
-  description: Bare tag keys promoted to top-level columns.
-
-- name: sortFields (optional)
-  type: array of `SortFieldConfig`
-  description: Effective sort fields, in precedence order.

package/build/dist/docs-index/b46490c4-85e1-41bd-bdc0-9c661d963c6b.txt

@@ -1,217 +0,0 @@
-Learn how to configure how Grepr integrates with observability platforms and storage systems to support data ingestion, processing, and output in your pipelines.
-
-# Configure how Grepr integrates with observability platforms and cloud storage
-
-To support data ingestion and data output for your Grepr pipelines, you connect Grepr to external observability platforms and storage systems. To support these connections, the Grepr platform uses *integrations*. A Grepr integration provides the configuration details to connect to an external system, enabling data exchange between Grepr and that system.
-
-After configuring an integration, you can use it in your Grepr pipelines to create *sources* to ingest data from agents and collectors, such as FluentD or the OpenTelemetry collector, and *sinks* to send data to an observability platform or storage.
-
-You can use the Grepr UI or the REST API to create and manage integrations. To use the UI, see [Manage integrations in the Grepr UI](#manage-integrations-in-ui). To use the API, see the documentation for each [supported integration](#supported-integrations).
-
-## Integration types
-
-The Grepr platform includes two integration types:
-
-* Observability platform integrations: Connections to monitoring, logging, and APM platforms, such as Datadog, Splunk, New Relic, and Sumo Logic.
-* Cloud storage integrations: Connections to cloud storage. The Grepr data lake uses cloud storage to store your raw data and metadata.
-
-## Supported platform and storage integrations \[#supported-integrations]
-
-The Grepr platform includes the following integrations. For details on configuring and using an integration, refer to the linked documentation page. For a complete list of supported integrations, supported functionality, and details on vendor-specific capabilities with some integrations, see [Grepr supported vendor and storage integrations](/integrations/support-matrix).
-
-### Datadog
-
-The Datadog integration connects to the Datadog observability platform and supports sourcing data from several Datadog-compatible collectors and sending processed data to Datadog.
-
-See [Configure Datadog connections with a Grepr integration](/integrations/datadog).
-
-### New Relic
-
-The New Relic integration connects to New Relic's observability platform. Use this integration to create sources that retrieve log data from New Relic collectors and sinks that send processed data to New Relic for monitoring and analysis.
-
-See [Configure New Relic connections with a Grepr integration](/integrations/new-relic).
-
-### Splunk
-
-The Splunk integration provides connectivity to Splunk through Splunk's HTTP Event Collector (HEC) or S2S over HTTP. This integration supports reading data from Splunk collectors and writing data to Splunk indexes.
-
-See [Configure Splunk connections with a Grepr integration](/integrations/splunk).
-
-### Sumo Logic
-
-The Sumo Logic integration connects to Sumo Logic's cloud-based log management and monitoring platform using a Sumo Logic HTTP Source. You can use the integration to create sources that ingest data from Sumo Logic collectors and sinks that send processed data to Sumo Logic.
-
-See [Configure Sumo Logic connections with a Grepr integration](/integrations/sumo-logic).
-
-### OpenTelemetry
-
-The Grepr OpenTelemetry integration lets you ingest logs and Application Performance Monitoring (APM) data into Grepr from a collector that supports the OpenTelemetry Protocol over HTTP, or OTLP/HTTP, process them with a Grepr pipeline, and then send the processed data to any observability vendor that accepts OTLP/HTTP requests.
-
-See [Configure connections to OpenTelemetry compatible platforms](/integrations/open-telemetry).
-
-### Grafana Cloud
-
-You can configure Grepr's OpenTelemetry integration to connect with Grafana Cloud using the OpenTelemetry Protocol (OTLP). Use an OpenTelemetry integration to create sources that ingest data from OTLP-compatible collectors and sinks that send processed log data to Grafana Cloud.
-
-See [Configure Grafana Cloud connections with a Grepr integration](/integrations/grafana).
-
-### Amazon CloudWatch
-
-You can configure Grepr's OpenTelemetry integration to connect with Amazon CloudWatch using the OpenTelemetry Protocol (OTLP). Use an OpenTelemetry integration to create sources that ingest data from OTLP-compatible collectors and sinks that send processed log data to a CloudWatch log group.
-
-See [Configure Amazon CloudWatch connections with a Grepr integration](/integrations/cloudwatch).
-
-### Google Cloud Platform (GCP)
-
-The GCP integration as a source enables you to stream logs from Google Cloud services to Grepr for processing and analysis, and then forward the processed logs to any sink created using a Grepr-supported integration, such as Datadog, Splunk, or New Relic.
-
-See [Google Cloud Platform observability with Grepr](/integrations/gcp).
-
-### Cloud storage
-
-The Grepr data lake uses Amazon S3 to store, manage, and support querying the raw data from your Grepr pipelines. Depending on your requirements, you can choose from two S3 integrations: a Grepr-hosted integration, which uses a Grepr-managed S3 bucket, and an integration that uses an S3 bucket in your account. Both integrations provide secure, managed, and efficient storage for your data, along with an optimized query interface.
-
-To configure the Grepr-hosted integration, you only need to assign a name to the integration. The Grepr platform manages the S3 bucket, including its deployment and configuration.
-
-Using the integration with your own S3 bucket requires that you deploy and configure an S3 bucket in your AWS account, but it provides more control over your data. See [Host a Grepr data lake with the Amazon S3 integration](/integrations/aws-s3).
-
-## Manage integrations in the Grepr UI \[#manage-integrations-in-ui]
-
-The **Integrations** page displays your integrations as cards, organized by observability vendor integrations and storage integrations. In each section, cards are ordered by recent activity, with the most active integrations appearing first.
-
-In addition to the integration name and its type, for example, a Datadog integration, if the integration is assigned to one or more teams, the team names appear in the header of the card.
-
-You can get a visual overview of the health and usage of your integrations using the charts included on each integration card. If an integration has recent activity, such as requests received from agents by a vendor integration or the amount of data written to a storage integration, a time-series chart of that activity is shown on the card.
-
-* Observability vendor integrations display requests received from agents over the last 30 minutes, color-coded by HTTP status code.
-
-  * To view more detailed metrics or more details about a pipeline that uses the integration, select it from the **Pipelines** menu.
-
-* Storage integrations display data written to the integration over the last 24 hours, broken down by dataset.
-
-  * To view and query a dataset stored on a storage integration, select it from the **Datasets** menu.
-
-To view more details about the metrics in either chart type, hover over any point in the chart to see a tooltip with the value and timestamp for that point.
-
-### Add an integration
-
-To create an integration, click **Add new** next to the section heading for the type you want to add. To learn how to create and configure specific integrations, see the documentation for each [supported integration](#supported-integrations).
-
-### Edit an integration
-
-To edit an integration, click the three-dot menu on the integration card and select **Edit**. Update the fields in the dialog and click **Save**.
-
-### Delete an integration
-
-To delete an integration, click the three-dot menu on the integration card and select **Delete**. Confirm the deletion in the dialog that appears.
-
-### Identify log events that should bypass aggregation \[#passthrough-estimation]
-
-When you create an integration with an observability vendor that supports *exception parsing*, Grepr automatically parses queries used by monitors, dashboards, or alerts in your observability vendor platform. You can use this list of queries to identify log messages that should be treated as *exceptions* to log reduction in Grepr pipelines. In other words, messages that should bypass aggregation to ensure that critical alerts and dashboards continue to work as expected. To help with identifying exceptions, the Grepr UI displays a *passthrough percentage* estimate for each query. This estimate is the percentage of events, over a configurable time window, that would bypass aggregation.
-
-You can also choose to automatically add and remove exceptions by enabling **Auto-sync exceptions** when you add the integration to your pipelines. To learn more about enabling and using auto-sync when you create or edit a pipeline, see [Prevent aggregation of messages used in your observability platform](/transforms/reducer/exceptions#vendor-exceptions).
-
-If an integration does not support exception parsing, the **Parsed Alerts** button on the integration card is disabled. See [Supported integrations: observability platforms](/integrations/support-matrix#supported-observability-integrations).
-
-To view the log queries parsed from an integration and configure passthrough estimation:
-
-1. On the **Integrations** page, on the card for a vendor integration that supports query parsing, click **Parsed Alerts**.
-
-   The **Parsed alert queries** dialog displays a table with the following columns:
-
-   * **Id**: The unique identifier for this alert.
-   * **Name**: The alert or dashboard name from your vendor.
-   * **Exception Type**: The type of vendor resource the query came from, such as a monitor or dashboard widget.
-   * **Query**: The log query extracted from the alert or dashboard.
-   * **Passthrough %**: The percentage of raw logs in your configured dataset that match this query. This value is displayed only if you've configured passthrough estimation.
-   * **Last Updated**: When the passthrough percentage was last calculated. Hover over the value to see the exact date and time.
-
-   You can filter the table by name or query text and filter by exception type using the multi-select filter in the **Exception Type** column. To manage which columns are visible, click the **Manage columns** icon in the upper-right corner of the table.
-
-2. To configure passthrough estimation and view the percentage of logs that would bypass aggregation for each query, click **Configure Passthrough Estimation**.
-
-   The **Passthrough Estimation Configuration** dialog lists the datasets that pipelines using the integration write to. Each row shows whether passthrough estimation is configured for that dataset and which pipelines use it.
-
-   > When you configure passthrough estimation for a dataset, it applies to all integrations or pipelines that use that dataset as a sink.
-
-3. Expand the row for the dataset you want to configure.
-
-4. Under **Filter Query (Optional)**, select your preferred query syntax and enter a query to filter for logs to include in the estimation analysis. Use this field when your dataset contains logs from multiple sources, and you want Grepr to use only the logs relevant to this integration. Leave this field empty to include all logs in the dataset.
-
-5. In the **Lookback Duration** menu, select the time window for calculating percentages. Options range from 15 minutes to 24 hours. A shorter window scans less data, reducing costs, but may not be representative if your log patterns vary throughout the day. A longer window provides more representative results but increases the cost of the analysis queries.
-
-6. In the **Refresh Period** menu, select how often Grepr recomputes each estimation for this dataset. Options range from 1 hour to 30 days, with a default of 7 days. A shorter period uses more recent logs to calculate estimates, resulting in more up-to-date values, but increases query load on the data lake.
-
-7. In the **Log Limit** field, enter the maximum number of logs to scan per estimation run. The default is 10,000,000. Larger limits provide more representative results, but increase the cost of each estimation query.
-
-8. To save a new configuration, click **Enable**. To save edits to an existing configuration, click **Save changes**.
-
-Grepr's impact estimation cycle runs once an hour for any datasets due for refresh, based on each dataset's **Refresh Period**. New or recently changed queries are included in the next cycle. A first estimate typically appears within an hour, though it might take longer if a large number of queries are queued for processing.
-
-To stop estimating for a dataset, expand its row and click **Disable estimation**. The dataset remains in the list as **Not configured** so you can re-enable estimation later.
-
-### Interpret passthrough percentages
-
-> Although two queries might use the same log events, their estimates will be equal. Additionally, the passthrough rate remains constant and is not the sum of the estimates. For example, if you enable exceptions for two queries that use the same log events, the passthrough rate doesn't double, but remains equal to the rate when only one exception is enabled.
-
-The **Passthrough %** value for a query shows what fraction of your raw logs that alert would require to pass through without aggregation. Use these values to evaluate the impact of adding each alert as a pipeline exception:
-
-A percentage less than 1% is considered low, and an exception should have minimal impact on aggregation efficiency and storage.
-
-You should review carefully any percentage above 1% before creating an exception for the query.
-
-A dash (—) in the **Passthrough %** column indicates that no estimate is available yet, either because passthrough estimation is not configured, or because Grepr hasn't finished calculating the estimate. If passthrough estimation is configured, the first estimate typically appears within an hour, and later refreshes occur based on the dataset's **Refresh Period**.
-
-The **Passthrough Estimation Configuration** dialog also includes the **Recent Activity** table at the bottom. This section shows the most recent log events related to exception parsing, giving you visibility into recent integration activity without leaving the dialog. To view the complete log history for the integration, click **View all logs** to go to the **System Logs** page, pre-filtered to show all activity for the selected integration.
-
-### Configure passthrough estimation from a pipeline
-
-You can also configure passthrough estimation directly from a pipeline. On the pipeline details page, in the left-hand navigation menu, click **Exceptions**. In the **Actions** column for an **Avoid impacting dashboards/alerts** exception, click the **⋯** menu and select **Configure estimations** to open a dialog that includes all datasets configured as sinks for the pipeline. Any configuration changes you make to the dataset apply to every other integration or pipeline that populates that dataset.
-
-To have Grepr recompute impact percentages during the next impact estimation cycle, click the **⋯** menu and select **Reset estimations** to clear the cached impact percentages for the datasets configured as sinks for the pipeline. Use this when you've changed alert queries in your observability vendor and want updated estimates without waiting for the next scheduled refresh.
-
-## Limitations \[#limitations]
-
-Grepr might split log events over 900KB, causing them to lose their original structured format. Each split event includes a common `grepr.splitOriginId` attribute if you need to reconstruct the original message.
-
-## Frequently Asked Questions
-
-### What is a Grepr integration?
-
-A Grepr integration provides the configuration details needed to connect Grepr to an external system, such as an observability platform or cloud storage. Integrations enable data exchange between Grepr and external systems, allowing you to create sources for data ingestion and sinks for data output in your pipelines.
-
-### What types of integrations does Grepr support?
-
-Grepr supports two integration types: observability platform integrations (connections to monitoring, logging, and APM platforms like Datadog, Splunk, New Relic, and Sumo Logic) and cloud storage integrations (connections to cloud storage for the Grepr data lake).
-
-### What is a source in a Grepr pipeline?
-
-A source in a Grepr pipeline is an operator that ingests data from an external system or agent. Sources use integrations to connect to observability platforms or collectors like FluentD or OpenTelemetry to pull data into Grepr for processing.
-
-### What is a sink in a Grepr pipeline?
-
-A sink in a Grepr pipeline is an operator that outputs processed data to an external system. Sinks use integrations to send Grepr-processed logs to observability platforms, data lakes, or other storage systems based on your pipeline configuration.
-
-### Can I use multiple integrations in a single pipeline?
-
-Yes, you can configure pipelines with multiple sources to ingest data from different platforms and multiple sinks to send processed data to different destinations. This flexibility allows you to build complex multi-source, multi-destination data processing workflows.
-
-### Which observability platforms does Grepr support?
-
-Grepr supports integrations with Datadog, Splunk, New Relic, Sumo Logic, Grafana Cloud, and Google Cloud Platform (GCP). Each integration enables different source and sink capabilities depending on the platform's features and capabilities.
-
-### How do I configure a Grepr integration?
-
-To configure an integration, you need to provide connection details specific to your external system, such as API keys, tokens, endpoints, or credentials. The specific steps vary by integration type. Detailed instructions for each integration are available in the platform-specific documentation pages.
-
-### What is the Grepr data lake integration?
-
-The Grepr data lake uses Amazon S3 to store raw log data. You can configure either a Grepr-hosted integration (using a Grepr-managed S3 bucket) where you only provide a name, or an integration with your own AWS account (requiring you to configure and manage the S3 bucket yourself for more control).
-
-### What happens to logs older than 48 hours from external sources?
-
-Grepr might drop incoming data from external integration sources with timestamps older than 48 hours. This limitation helps optimize storage and processing while allowing you to retain raw logs in the data lake for longer-term analysis.
-
-### What happens when a log event is larger than 900KB?
-
-If a log event exceeds 900KB, Grepr might split it into smaller events, which can cause them to lose their original structured format. Each split event includes a common 'grepr.splitOriginId' attribute that you can use to reconstruct the original message from its component parts.

package/build/dist/docs-index/b70f8239-19a8-4f4b-a4df-417bebca69b9.txt

@@ -1,75 +0,0 @@
-Learn how Grepr integrates with Datadog, Splunk, New Relic, and other platforms to reduce observability costs and provide real-time log processing and analysis.
-
-# What is Grepr?
-
-Grepr is an enterprise-grade data storage, processing, and analytics platform for your observability data. Grepr collects, transforms, and analyzes your log data and then sends the processed logs to your observability platform.
-
-Before sending log data to the destination platform, Grepr uses machine learning to analyze and intelligently aggregate the data. During normal operations, Grepr reduces your observability costs by reducing the volume of data sent to your observability tools, while retaining all of your raw data in low-cost cloud storage. If an incident or anomaly occurs, Grepr increases the granularity of the forwarded data to ensure that sufficient data is available for troubleshooting. Grepr also uses the raw data to backfill any historical data relevant to the incident.
-
-Grepr supports popular observability platforms, including Datadog, Splunk, New Relic, Sumo Logic, Grafana Cloud, and OpenTelemetry-compatible platforms.
-
-If you're just getting started with Grepr, follow the **Get started** guides below. If you're already using Grepr and need information on creating pipelines to process logs or to query raw logs stored in Grepr, see the guides under **Process and explore your logs**. If you need to automate your Grepr tasks or implement clients that use Grepr features, or use JavaScript to code reusable components for your pipelines, see the **Developers** documentation.
-
-## Get started
-
-[Create your first pipeline](/tutorials/first-pipeline)
-
-[Learn about the platform](/grepr-platform)
-
-## Process and explore your logs
-
-[Connect to external tools](/integrations)
-
-[Process and transform data](/transforms)
-
-[Query logs in the data lake](/queries)
-
-## Developers
-
-[Use the Grepr Rest API](/apis)
-
-[Use the Grepr CLI](/cli)
-
-[Custom pipeline components](/templates)
-
-## Frequently Asked Questions
-
-### What is Grepr?
-
-Grepr is an enterprise-grade data storage, processing, and analytics platform for observability data. It collects, transforms, and analyzes log data, then sends processed logs to your observability platform while retaining raw data in low-cost cloud storage.
-
-### How does Grepr reduce observability costs?
-
-Grepr uses machine learning to intelligently aggregate and reduce the volume of log data sent to your observability tools during normal operations, while automatically increasing data granularity during incidents. This significantly reduces costs while maintaining full raw data access.
-
-### What observability platforms does Grepr integrate with?
-
-Grepr supports popular platforms including Datadog, Splunk, New Relic, Sumo Logic, Grafana Cloud, and OpenTelemetry-compatible platforms.
-
-### What is the Grepr data lake?
-
-The Grepr data lake is a secure cloud storage layer built on Apache Iceberg and S3 that stores your raw log data. It provides efficient querying capabilities and serves as the foundation for historical analysis and incident backfills.
-
-### Can I query raw logs in Grepr?
-
-Yes, you can query raw logs stored in the Grepr data lake using the query interface or REST API. The data lake supports powerful queries with tags, time ranges, and custom filters to analyze historical log data.
-
-### How do I create a Grepr pipeline?
-
-You can create pipelines using the Grepr UI, REST API, or CLI. Pipelines consist of data sources, transformations (like log reduction and parsing), and destinations. Follow the 'Create your first pipeline' guide to get started.
-
-### What happens to my data during an incident?
-
-During incidents or anomalies, Grepr automatically increases the granularity of forwarded data to ensure sufficient detail for troubleshooting. It also backtransfers historical raw logs relevant to the incident from the data lake.
-
-### Is my data secure in Grepr?
-
-Yes, Grepr maintains a SOC 2 Type II report and a HIPAA attestation, and implements enterprise security controls including encryption in transit (TLS 1.2+) and at rest, role-based access controls, and secure identity management with Auth0.
-
-### Can I use Grepr with my existing observability tools?
-
-Yes, Grepr is designed to integrate seamlessly with existing observability platforms. You continue using your current tools while Grepr handles data reduction and optimization in the background.
-
-### How can I automate Grepr tasks?
-
-You can automate tasks using the Grepr REST APIs, CLI, or custom JavaScript components in pipelines. The CLI supports job creation, queries, integration management, and dataset operations.

package/build/dist/docs-index/b90b2b69-69d2-4064-9fe1-8f5dec7ea8c4.txt

@@ -1,18 +0,0 @@
-# Get the current maxTokens window of one LLM integration
-
-**GET** `/v1/llm-usage/{integrationId}`
-
-**Operation ID**: `currentWindow`
-**Summary**: Get the current maxTokens window of one LLM integration
-## Parameters
-
-- **integrationId** (required) - path
-  - Type: string
-
-## Responses
-
-### 200
-
-Current window retrieved
-
-- **Content-Type**: `application/json` / `LlmUsageWindow`

package/build/dist/docs-index/bd26bd6b-07d0-4548-b38f-3e5788af3140.txt

@@ -1,111 +0,0 @@
-Process data in Grepr pipelines using built-in tools for parsing, filtering, enrichment, and transformation of log events.
-
-# Transform data in a Grepr pipeline
-
-A Grepr transform is a processing step that modifies, enriches, or restructures data as it flows through a pipeline. As records move through a pipeline, transforms allow you to parse structured and semi-structured data, filter records, and use a familiar SQL interface to perform more complex transformation of records as they move through a pipeline. Transforms are configured as pipeline steps that process data sequentially. Each transform receives input data, applies its logic, and passes the result to the next step in the pipeline.
-
-Most transforms are available in the Grepr pipelines UI. Some transforms are available only in the Grepr REST API or provide more capabilities when used with the API, such as [filters](#filter) and [branching the processing of records](#message-routing) based on query matches.
-
-The following sections describe the transforms available in Grepr pipelines.
-
-## Transform semi-structured records with the Grok parser \[#grok-parser]
-
-The Grok transform parses unstructured text using pattern matching with regular expressions and named capture groups. Use this transform to extract fields from log lines, error messages, and other text-based data where you need to identify specific patterns and convert them to structured fields.
-
-See [Grok parser](/transforms/grok).
-
-## Convert JSON messages into structured objects \[#json-parser]
-
-The JSON parser transforms JSON strings into structured JSON objects. This transform handles nested objects, arrays, and various data types, making it useful for processing API responses, configuration data, and structured log entries.
-
-See [Transform JSON strings into JSON objects](/transforms/json-parsing).
-
-## Restructure JSON objects with a remapper \[#remapper]
-
-The remapper transform moves or copies attributes to top-level fields or to tags in a JSON object. You can also use rename rules to move attribute values from one path to another within the same event. In the pipelines UI, a remapper with customizable default attributes is automatically added to a pipeline.
-
-See [Remap JSON attribute fields to top-level log event fields](/transforms/remapper).
-
-## Transform records with SQL \[#sql-transform]
-
-The SQL transform allows you to use ANSI standard SQL queries to filter, transform, and enrich data records. The SQL transform provides a familiar SQL interface to perform a number of data processing and transformation tasks.
-
-You can use the SQL transform in the Grepr UI or REST API. In the UI, the SQL transform is available when you configure filters.
-
-To learn about using the SQL transform in the Grepr API, see the [SQL Operation specification](/apis/api-spec#schema/SQLOperation).
-
-For a complete reference and examples, see [Transform events with SQL](/transforms/sql-transform).
-
-## Filter records based on conditions \[#filter]
-
-The Filter transform includes or excludes data records based on specified conditions. You can create filters using familiar query interfaces, such as SQL, as well as Datadog and New Relic log query-like syntaxes.
-
-In the Grepr pipelines UI, filters are located in predefined locations in the pipeline. To learn how to configure filters, see [Filter events in a Grepr pipeline](/transforms/filter-events).
-
-To have more flexibility for where filters are included in a pipeline, use the REST API. See [LogsFilter](/apis/api-spec#schema/LogsFilter).
-
-## Reduce log volumes with dynamic aggregation \[#reducer]
-
-The Grepr log reducer uses clustering algorithms to identify patterns in your log data and aggregate similar log messages. The output of this aggregation includes summaries and samples of similar messages. The log reducer is a core step in a log reduction pipeline, where it runs before processed log events are sent to your configured sinks. By sending summaries of similar messages rather than multiple events with redundant information, Grepr significantly reduces the volume of shipped logs and your observability costs. The log reduction pipeline also persists the complete set of raw logs to a low-cost data lake, enabling you to analyze and troubleshoot issues when necessary.
-
-To learn more about how Grepr reduces log volumes with dynamic aggregation, see [Reduce noise and improve the usability of your logs with the log reducer](/transforms/reducer).
-
-To learn how to create a log reduction pipeline, see [Tutorial: Build your first Grepr pipeline](/tutorials/first-pipeline).
-
-To learn how to use the resulting summarized log events in your observability platform along with raw logs stored in Grepr, see [Troubleshoot incidents with Grepr-processed data and your observability tool](/tutorials/troubleshooting-example).
-
-## Transform and enrich log messages with the REST API \[#log-transform-action]
-
-You can use the LogTransformAction in the REST API to specify actions that transform log messages, such as adding or removing attributes and tags. See [LogTransformAction](/apis/api-spec#schema/LogTransformAction).
-
-## Route the processing of messages based on query matches \[#message-routing]
-
-You can route log messages to be processed by different pipelines based on query matches. Because the pipelines UI doesn't support branching, Grepr recommends creating multiple pipelines that ingest data using the same source, and then use filters to route data to the correct pipeline.
-
-You can also use the REST API to create more complex pipeline configurations. See [LogsBranch](/apis/api-spec#schema/LogsBranch).
-
-## Frequently Asked Questions
-
-### What transforms are available in Grepr pipelines?
-
-Grepr provides several transform types: Grok parser for parsing unstructured text with pattern matching, JSON parser for converting JSON strings into structured objects, remapper for restructuring JSON fields and attributes, SQL transform for complex filtering and enrichment using SQL queries, and Filter transform for conditional record processing. Most transforms are available in the UI, while some advanced options are available via REST API.
-
-### When should I use the Grok parser transform?
-
-Use the Grok parser when you need to extract structured data from unstructured text logs. It's ideal for parsing error messages, application logs, system logs, and other text-based data where you need to identify specific patterns and convert them into structured fields using regular expressions and named capture groups.
-
-### When should I use the JSON parser transform?
-
-Use the JSON parser when your log messages contain JSON strings that need to be converted into structured JSON objects. This is useful for processing API responses, configuration data, and structured log entries. The JSON parser handles nested objects, arrays, and various data types automatically.
-
-### What is the remapper transform and when is it used?
-
-The remapper transform moves or copies attributes to top-level fields or to tags in a JSON object. It's useful for standardizing log event structure by moving well-known fields like message, severity, and host from nested attributes to top-level fields. You can configure whether the source attribute is preserved or removed after remapping. The remapper also supports rename rules for moving attribute values from one path to another. The remapper is automatically added to pipelines in the UI with customizable default attributes.
-
-### When should I use the SQL transform?
-
-Use the SQL transform when you need complex filtering, transformation, or enrichment of log records. SQL provides a familiar interface for advanced operations like conditional logic, string manipulation, aggregations, and joining data. It's available in the UI for filters and via REST API for more flexibility.
-
-### Can I combine multiple transforms in a pipeline?
-
-Yes, transforms are configured as sequential pipeline steps. Each transform receives input from the previous step, applies its logic, and passes the result to the next step. This allows you to create complex data processing workflows by combining parsing (Grok), restructuring (remapper), and filtering/enrichment (SQL) transforms.
-
-### What is the Filter transform used for?
-
-The Filter transform includes or excludes data records based on specified conditions. You can use filters with SQL, Datadog, and New Relic-like query syntaxes. In the UI, filters are located in predefined pipeline positions, while the REST API provides more flexibility for filter placement.
-
-### Can I use the REST API for more advanced transform options?
-
-Yes, the REST API provides access to advanced transform options not available in the UI. For example, you can use LogTransformAction to add or remove attributes and tags, LogsBranch for complex message routing based on query matches, and more flexible filter placement. See the API specification for complete details on transform capabilities.
-
-### How do I choose between Grok and SQL transforms?
-
-Use Grok when you need pattern-based parsing of unstructured text to extract specific fields. Use SQL when you need complex conditional logic, calculations, string transformations, or filtering. Grok is specialized for text parsing, while SQL is more general-purpose for data manipulation and enrichment.
-
-### How does Grepr reduce log volumes and observability costs?
-
-Grepr uses clustering algorithms to identify patterns in log data and aggregates similar log messages. Instead of sending multiple redundant log events, Grepr sends summaries and samples of similar messages to your observability platform. This significantly reduces the volume of shipped logs while retaining the ability to analyze and troubleshoot issues. Raw logs are retained in a low-cost data lake for future reference.
-
-### What happens if a transform fails on a record?
-
-Transform behavior depends on the specific transform type and configuration. Parsing errors (like Grok or JSON parsing) typically skip records or add error attributes depending on your configuration. SQL transforms filter out records that don't match conditions. Review transform-specific documentation for error handling details.