@grepr/cli 1.11.5-a603268 → 1.11.5-bcb5435

package/build/dist/docs-index/7c4dd67b-a984-4dec-944d-4597177f44f6.txt

@@ -0,0 +1,168 @@
+Turn on Grepr AI agent memory so an agent records what it concluded, counts recurrences, and searches past investigations during a new one.
+
+# Remember past investigations
+
+Without memory, every investigation starts from nothing. An agent that diagnosed a problem last week reaches the same conclusion the same slow way this week, and nobody learns that the problem has now happened five times.
+
+*Memory* changes that. An agent with memory enabled records what it concluded when it reaches a diagnosis, recognizes when a later investigation reaches the same conclusion and counts the recurrence, and searches those records at the start of a new investigation to find out whether it has seen this before.
+
+Both halves show up in the investigation's tool calls. A memory search runs near the start, before the agent has looked at anything else, and the record is written at the end, when it has a diagnosis.
+
+The agent decides what to record and whether something is a recurrence. Grepr does not extract conclusions on its behalf or merge records by similarity, because the investigating agent is the only thing that saw the evidence.
+
+Memory is off until you turn it on and choose where the memories are stored.
+
+## What an agent records \[#what-recorded]
+
+An agent records at most one memory per investigation, written when it reaches a diagnosis rather than after the fact. Each memory holds:
+
+| Field | What it holds |
+| --- | --- |
+| Title | A short name for the problem. |
+| Symptoms | What was observed. |
+| Root cause | What the agent concluded was responsible. |
+| Root cause confidence | Whether the cause is **confirmed**, **suspected**, or **unknown**. |
+| Ruled out | Causes the agent checked and eliminated, so a later investigation does not repeat that work. |
+| Resolution | What resolved it, or what was attempted. |
+| Significance | How much the conclusion matters. See [Significance](#classification). |
+| Entity tags | The services, pipelines, and environments the memory is about. See [Entity tags](#tagging). |
+| Recurrence count | How many investigations reached this same conclusion. |
+
+An agent can correct its own memory for an investigation while that investigation is running, so an early diagnosis it later disproves does not stay on the record. Correcting a memory does not count as a new recurrence.
+
+### Significance \[#classification]
+
+Significance is how an agent classifies what it found, and it decides how the memory can be found later:
+
+| Significance | Meaning |
+| --- | --- |
+| Incident | A real problem was identified. |
+| Observation | Not a problem, but a conclusion worth keeping. |
+| Inconclusive | The agent could not determine whether there was a problem. |
+| Benign | Triaged as no problem. |
+
+Benign memories are recorded but not indexed for search by meaning or by text. They are reachable only by a timeline search over their entity tags. This keeps the large number of routine all-clear findings out of the way of searches for real problems, at the cost of making them harder to find. If you want an agent to notice that a given service has been triaged as benign many times, search the timeline for that service rather than searching by text.
+
+## Turn on memory \[#enable]
+
+Memory needs somewhere to store its search index and a model to build that index with, so there are three things to set up.
+
+### Step 1: Create a vector index integration \[#vector-index]
+
+1. On the **Integrations** page, next to the **S3 Vector Index** section heading, click **Add new**.
+2. In the **Add Vector Index** dialog, enter a **Name** and the number of **Dimensions** the embedding model you plan to use produces, then click **Create**.
+
+The dimensions must match the model's output, so choose the embedding model first and use its dimension count. An index's dimensions are fixed after creation.
+
+### Step 2: Create an embedding integration \[#embedding-integration]
+
+Memory embeds each record with a model from an LLM integration that provides embeddings. If you do not have one, create it before you continue. See [_](/integrations/llm).
+
+An agent's reasoning model and its embedding model are separate choices. An agent can reason with one integration and embed with another.
+
+### Step 3: Turn on memory for the agent \[#agent-memory]
+
+1. On the agent's page, go to the **Memory** step.
+2. Turn on **Remember past investigations**.
+3. Under **Storage**, select the **Vector index integration**, the **Embedding credentials**, and the **Embedding model**.
+4. Save the agent.
+
+Grepr creates the search index when you create the vector index integration in Step 1, and discards the integration if the index cannot be created. Saving the agent then checks that the embedding model actually works against that index, by embedding a test value with it. So a wrong model or an unusable credential stops the save rather than surfacing later as memories that were never indexed.
+
+### Choose an embedding model and keep it \[#embedding-model]
+
+A memory is embedded with the model the agent had when it was recorded, and a query can only be compared against embeddings from the same model. Changing an agent's embedding model therefore cuts its new memories off from the ones it already recorded, as far as searching by meaning is concerned. Treat the choice as long lived.
+
+Using the same model across several agents does not pool their memories into one semantic search. A search by meaning always targets one agent, as described next.
+
+## How an agent searches its memory \[#search-modes]
+
+A search runs in exactly one of three modes, and which mode fits depends on what the agent is trying to find:
+
+| Mode | What it finds | Use it to answer |
+| --- | --- | --- |
+| Semantic | Investigations whose conclusions mean something similar to the query. | Has a problem like this happened before? |
+| Text | Investigations mentioning specific words, using full-text search. | Has this exact error string or identifier come up before? |
+| Timeline | Investigations about given entities, ordered by time. | What has been happening to this service recently? |
+
+Semantic search finds a past incident described in different words, which text search misses. Text search finds a literal, such as an error code or a host name, which semantic search can rank poorly. Timeline search needs entity tags rather than a query, and it is the only mode that returns benign memories.
+
+The modes also differ in how much they cover, which matters when you read an empty result:
+
+* A semantic search covers one agent at a time, because a query is only comparable against embeddings made with that agent's model. An empty semantic result means nothing similar was found within what the search could reach. It is not evidence that the agent recorded nothing similar, and it is not evidence that your organization has not seen the problem. A memory embedded with a model the agent no longer uses is unreachable until it is embedded again, and a search that cannot reach the embedding provider returns empty as well. When an empty result matters, confirm it with a text or timeline search, which read the records directly.
+* A text or timeline search covers every agent in your organization by default, so it is the one to use when you want the organization's whole history.
+
+So to find out whether any agent has seen something, search by text or timeline, or repeat the semantic search against each agent you care about.
+
+A result carries the memory's content along with its recurrence count, how long ago it was recorded, when it was last seen, and whether it is active, recent, or historical. There is no relevance score, because the agent judges from the content and the provenance rather than a ranking number.
+
+Search returns at most 50 results and has no pagination, so a broad query can be incomplete. Narrowing by entity tags or by significance is more effective than raising the limit.
+
+## Entity tags \[#tagging]
+
+Entity tags are what make memories findable by the thing they are about, such as a service, a pipeline, or an environment. Each memory carries two kinds:
+
+* Tags Grepr derives from the triggering signal, which identify the source and the signal type.
+* Tags the agent adds, which name the entities it concluded the memory is about.
+
+Grepr derives its tags using a mapping per signal type, which extracts values from the signal payload by path. The shipped mappings cover the common cases:
+
+| Signal type | Tags derived from the signal |
+| --- | --- |
+| Log pattern | The pipeline, the pattern, and the service, environment, and Kubernetes namespace from the sample log event. |
+| Webhook | The integration, and the service and host from the payload. |
+| Manual | The signal type only. |
+
+The webhook mapping reads the service and host from field names Grepr expects. If your payloads name those fields differently, add a mapping so the tags resolve. On the agent's **Memory** step, under **Entity tags**, add a mapping for the signal type, then add each tag key with the paths to read it from. Grepr tries the paths in order and uses the first that resolves.
+
+Consistent tag values matter more than complete ones. A service tagged `checkout` in some memories and `checkout-svc` in others splits its history in two, and a timeline search finds only half. Grepr gives an agent the tag values already in use in your organization when it searches, so an agent that searches before recording reuses existing values rather than inventing new ones.
+
+## Who can read and write memories \[#scope]
+
+* Reads are organization-wide. Any agent can search any other agent's memories in your organization. Operational history is only useful shared, so an agent investigating a service finds what another agent concluded about the same service. Permission is uniform across the modes, but coverage is not: a semantic search still examines one agent at a time. See [How an agent searches its memory](#search-modes).
+* Writes are owner-only. A memory can be written or corrected only by the agent that ran the investigation, so one agent cannot overwrite another's conclusions.
+
+When an agent finds that a past memory describes what it is now seeing, it links its investigation to that memory, which raises the recurrence count rather than creating a near-duplicate record. Each contributing investigation counts once.
+
+Memories outlive the agent that wrote them. Deleting an agent leaves its memories searchable by your other agents.
+
+## Search memories from the CLI \[#cli]
+
+The Grepr CLI searches the same memories an agent does, which is how you read what your agents have concluded.
+
+```bash
+# Find investigations mentioning a literal string
+grepr investigation:memory:search --mode text --query "connection refused" --limit 5
+
+# Find investigations whose conclusions are similar in meaning
+grepr investigation:memory:search --mode semantic --agent-id <agent-id> --query "intermittent ingestion failures"
+
+# See what has been happening to one service lately
+grepr investigation:memory:search --mode timeline --entity-tag entity:service=checkout
+```
+
+A semantic search needs an agent id, because a query can only be compared inside the index that agent embeds into. Text and timeline searches cover every agent in your organization by default. Timeline searches need at least one entity tag.
+
+The CLI is how you read what your agents have concluded outside of an investigation. To learn more about the CLI, see [_](/cli).
+
+## Frequently Asked Questions
+
+### What does Grepr AI agent memory store?
+
+One memory per investigation, holding a title, the symptoms, the root cause and the agent's confidence in it, the causes it ruled out, the resolution, a significance classification, entity tags, and a count of how many investigations reached the same conclusion.
+
+### What do I need to turn on agent memory?
+
+Three things: a vector index integration whose dimensions match your embedding model, an LLM integration that supplies the embedding model, and the embedding model name. Grepr creates the search index when you create the vector index integration. Saving the agent then checks that the embedding model works against that index, so a wrong model stops the save.
+
+### How does an agent search its memory?
+
+In one of three modes. Semantic search finds conclusions that mean something similar to the query, and covers one agent at a time. Text search finds specific words, such as an error code. Timeline search finds memories about given entities ordered by time, and is the only mode that returns benign memories. Text and timeline searches cover every agent in the organization by default.
+
+### Can one Grepr AI agent read another agent's memories?
+
+Yes, permission is organization-wide, so any agent can search any other agent's memories in your organization. Coverage differs by mode: text and timeline searches span every agent by default, while a search by meaning targets one agent at a time, because a query is only comparable against embeddings made with that agent's model. Writes are owner-only.
+
+### How do I read what my agents have remembered?
+
+Search them with the grepr investigation:memory:search command, choosing text, semantic, or timeline mode, or read them in an investigation that searched them. The Memory step of an agent holds the settings rather than the records.

package/build/dist/docs-index/82747906-5dce-4dad-99a2-849f4de6c4d8.txt

@@ -0,0 +1,136 @@
+Control what a Grepr AI agent can reach with a service account, tools, skills, and MCP integrations, and whether its pipeline changes need approval.
+
+# Control what an AI agent can do
+
+An agent's reach is bounded in layers, and a capability has to be allowed by every layer to work:
+
+1. The **service account** the agent acts as. Its roles are the outer boundary and cap everything the agent can reach in Grepr.
+2. The **tools and skills** you enable. These narrow the agent to specific operations within what the service account permits.
+3. The **MCP integrations** you enable, and the tools you allow on each. These decide what the agent can reach in other systems.
+
+A fourth setting, the change approval mode, decides whether the agent can make automatic changes to your pipeline.
+
+These settings are on the agent's **Capabilities** step, and on the matching step of the create wizard.
+
+## Choose the service account an agent acts as \[#service-account]
+
+Every call an agent makes to Grepr authenticates as the service account you select in the **Service account** field. The service account's roles decide which pipelines, datasets, and integrations the agent can read and write to.
+
+The service account is the boundary that holds even if a tool grant is over-permissive. An agent with the pipeline editing tool but a service account that can only view jobs cannot change a pipeline. Grant the narrowest role that lets the agent finish its job. To create one, see [_](/admin/service-accounts).
+
+If the **Service account** menu is empty, create a service account before you continue.
+
+## What every agent can already do \[#sandbox-baseline]
+
+Each investigation runs in its own sandbox, and inside that sandbox an agent can:
+
+* Run shell commands.
+* Run Python for local analysis and scripting.
+* Query files it downloaded with DuckDB.
+* Read, create, and edit files in its own workspace.
+
+These are part of the sandbox and cannot be turned off. They act only on the sandbox, which is discarded when the investigation ends, so they do not reach your data or your pipelines. Everything that does reach Grepr is granted separately, as described next.
+
+## Grant access to Grepr with tools and skills \[#grepr-access]
+
+The **Grepr access** section is where you decide what the agent can do. You grant access in two ways, and most agents use both:
+
+* Enable a skill, which gives the agent guidance for a task and, when the skill needs them, the tools that task requires.
+* Pin an individual tool, which grants that tool with no accompanying guidance.
+
+Prefer skills. A skill tells the agent how to use what it grants, so an agent with a skill usually needs fewer turns than one holding the bare tool. Pin a tool when you want a capability that no skill attached covers.
+
+### Abilities and reference skills \[#skill-kinds]
+
+Skills are grouped by whether enabling one is a permission decision:
+
+* **Abilities** carry tools. Enabling one grants the agent the operations that skill needs, so treat it as granting access.
+* **Reference** skills carry only knowledge. Enabling one changes what the agent knows, never what it can do, so it is safe to enable freely.
+
+To read what a skill covers and what it grants before you enable it, open its detail from the selector. To learn what skills exist and how to write your own, see [_](/ai-agents/skills).
+
+### How much a skill grants \[#grant-scope]
+
+How wide a skill's grant is depends on its scope, and the difference matters when you are reasoning about least privilege:
+
+* A **platform skill**, which Grepr maintains, grants exactly the operations its definition lists. A skill that needs only to preview a pipeline change gets the preview operation and not the one that applies it.
+* An **organization skill**, which you create, grants the whole *tool family* behind each tool you approve on it, meaning every operation that tool covers rather than only the ones the skill uses. Approving the pipeline editing tool on your own skill grants everything that tool covers, not only the part your instructions use.
+
+So an organization skill is as broad as the tools you approve on it. To keep a grant narrow, approve fewer tools on the skill rather than relying on the skill's instructions to limit the agent.
+
+Enabling a tool directly always grants its whole family, the same way an organization skill does.
+
+### Tool reference \[#tool-table]
+
+| Tool | What it lets the agent do |
+| --- | --- |
+| `query_logs` | Query the logs in your Grepr datasets. |
+| `get_pipeline` | Read the configuration of your pipelines and templates. |
+| `edit_pipeline` | Propose a change to a pipeline's configuration, and preview a change before proposing it. |
+| `grok` | Build and test Grok parsers. |
+| `datasets` | List, read, and manage your Grepr datasets. |
+| `integrations` | List and read your Grepr integrations. |
+| `docs` | Search the Grepr documentation and validate SQL. |
+| `backfill` | Replay logs from the data lake to a vendor destination. |
+| `investigations` | Read agent configuration, past investigations, and their transcripts. |
+| `github_code` | Clone, edit, and publish source repositories. Requires an enabled GitHub MCP integration. |
+
+An agent's pipeline changes are scoped to a pipeline's Grok parsers and its SQL transforms, so granting the pipeline editing tool keeps the rest of the pipeline configuration out of the agent's reach. Both are add-only. An agent can add a Grok rule or add a SQL transform, and it cannot change or remove one that is already there, so its proposals accumulate alongside your configuration rather than rewriting it.
+
+A SQL transform an agent adds is a filter and nothing further. It reads the pipeline's logs and selects them unchanged against a single condition, so it cannot join, aggregate, read a dataset, register a function, or keep state. It routes the events it matches to the agent signal sink, which you configure yourself when you set the pipeline up. An agent can add routes to that sink, and it cannot create, change, or remove the sink. That limit is on what an agent adds by itself. A SQL output you route to the agent signal sink yourself carries no such restriction, so you can build the event an agent receives however you need. See [_](/ai-agents/subscriptions#agent-signal-sink).
+
+The tools an agent uses to record and search its memory are not in this list. They are granted by the agent's memory settings instead, so that turning memory on is the single decision that controls them. See [_](/ai-agents/memory).
+
+## Let an agent call an MCP server's tools \[#mcp-integrations]
+
+The **MCP integrations** section lists the MCP integrations in your organization and which of their tools the agent can use. Two agents can share one integration with different tool access enabled.
+
+Edit the integration's allowed tools and choose either every tool the server publishes or only the ones you select. Grepr reads the tool list from the server and marks each tool as read only or writing; for example, you can allow an agent to read incidents without allowing it to change them.
+
+Allowing every tool means the agent gains any tool the server adds later, which is convenient for a read-only server and worth avoiding for one that can make changes. Selecting individual tools keeps the enabled tools fixed until it is manually changed.
+
+Two rules apply to GitHub:
+
+* An agent can enable at most one GitHub MCP integration.
+* The `github_code` tool needs an enabled GitHub MCP integration, because that integration is how it reaches source code.
+
+To create an integration, see [_](/integrations/mcp).
+
+## Decide whether pipeline changes need approval \[#change-approval]
+
+An agent that decides a pipeline needs a configuration update does not edit it directly. It submits the change, and the **Pipeline changes** setting decides what happens next:
+
+| Mode | What happens when the agent proposes a change |
+| --- | --- |
+| **Require human approval** | Grepr records the change as a suggestion on the pipeline. Nothing changes until you review and apply it. |
+| **Auto-apply** | Grepr verifies the change and applies it. A running pipeline restarts to pick it up. No person is involved. |
+
+**Require human approval** is the default, and it is the right setting while you are learning how an agent behaves. Auto-apply an agent's changes only after its suggestions have consistently been ones you would have made yourself.
+
+Grepr enforces the mode at the operation level, so while an agent can always preview pipeline changes, it cannot bypass **Require human approval** to apply them.
+
+A change that lands on a running pipeline restarts it, which interrupts processing briefly.
+
+To review suggestions, see [_](/ai-agents/investigations#suggestions).
+
+## Frequently Asked Questions
+
+### How do I limit what a Grepr AI agent can reach?
+
+Reach is bounded in layers. The service account the agent acts as is the outer boundary and caps everything it can reach in Grepr. Within that, the tools and skills you enable narrow it to specific operations, and the MCP integrations you enable with their allowed tools decide what it can reach in other systems.
+
+### What can a Grepr AI agent do without any tools enabled?
+
+Inside its own sandbox it can always run shell commands and Python, query downloaded files with DuckDB, and read and write files in its workspace. These act only on the sandbox, which is discarded when the investigation ends, and they do not reach your data or pipelines.
+
+### What is the difference between an ability and a reference skill?
+
+An ability carries tools, so enabling one grants the agent the operations that skill needs and is a permission decision. A reference skill carries only knowledge, so enabling one changes what the agent knows but never what it can do.
+
+### Do platform skills and organization skills grant the same access?
+
+No. A platform skill grants exactly the operations its definition lists, which can be narrower than a whole tool. An organization skill grants the whole tool family behind each tool you approve on it, so keep its grant narrow by approving fewer tools.
+
+### Can a Grepr AI agent change my pipeline without approval?
+
+Only when you set its change approval mode to Auto-apply. The default, Require human approval, records every proposed change as a suggestion for you to review. Grepr enforces the mode server-side, so an agent has no route around it whatever its instructions say.

package/build/dist/docs-index/831c0e53-84aa-4af5-bd38-813581480d60.txt

@@ -0,0 +1,26 @@
+# Read
+
+**Type**: `object`
+
+## Properties
+
+- name: accessConfig (required)
+  type: `AccessConfig`
+
+- name: constraints (required)
+  type: `ReadOrgConstraints`
+
+- name: featureFlags (optional)
+  type: array of enum: `DATADOG_BROWSER_LOGS_INTEGRATION`, `METRICS_SOURCE`, `S3_LOGS_FILE_SOURCE`, `PIPELINE_TEMPLATES`, `AI_WORKFLOWS`, `SEARCH_ICEBERG_SOURCE`, `MULTI_GROK_PARSER`, `AUTH0_SSO_CONNECTION`, `TRINO_QUERY_ENGINE`, `UNKNOWN`
+
+- name: name (required)
+  type: string
+
+- name: plan (required)
+  type: `Plan`
+
+- name: ssoConnectionEnabled (optional)
+  type: boolean
+
+- name: ssoRequestedAt (optional)
+  type: string

package/build/dist/docs-index/8620f20a-e2b2-4686-b1e9-3e887b080143.txt

@@ -0,0 +1,78 @@
+Two ways to send logs to several vendor accounts, by creating a pipeline for each group or by grouping the reducer and filtering each sink.
+
+# Route logs to several vendor accounts
+
+Your logs do not all belong in the same place. Separate teams, environments, or business units each own an account in your observability platform, and a log has to reach the account that owns it. Each account you send to is its own Grepr integration.
+
+A Grepr pipeline is a free-form processing graph rather than a fixed shape, so you can arrange this in more than one way. Here are two examples, and you can combine them.
+
+## Create a pipeline for each group \[#pipeline-per-group]
+
+Create one pipeline for each group of logs that belongs together, filter its source to that group, and give it the sink for that group's account.
+
+Each pipeline summarizes only its own logs, so you can tune each one on its own. Choose this when the groups need different configurations, or when you want them isolated from each other.
+
+To create a pipeline, see [_](/tutorials/first-pipeline). To limit what one ingests, see [_](/transforms/filter-events).
+
+## Group the reducer and filter each sink \[#grouped-sinks]
+
+Run a single pipeline, add a sink for each account, and filter each sink to that account's logs. Choose this when the groups have the same configuration, and you would rather manage one pipeline than several.
+
+Routing rests on a *routing key*, which is a field whose value names the account a log belongs to, such as `account` or `team`. Two settings act on it: the reducer groups on the key, and each sink filters on it.
+
+Grouping is what makes the filtering correct. A summary carries the values of the fields it was grouped on, so grouping on the routing key gives every summary exactly one account, and each sink's filter then matches its own account and no other.
+
+> Configure the grouping before you add the sinks, because leaving it out fails quietly. Without the routing key in the group-by, logs bound for different accounts aggregate together. A tag then carries every value the merged messages had, so every account receives the same summary, reporting counts that include the other accounts' logs. An attribute is merged instead, and the default merge strategy replaces values that differ with a wildcard, so no filter matches and the summary reaches no account.
+
+### Step 1: Put the routing key on every log \[#step1-routing-key]
+
+The routing key has to be on each log before the reducer sees it, holding one value per account. If your collectors already tag logs with the owning team or environment, use that tag. If the value is inside the message, parse it into an attribute or a tag first with the Grok parser or the JSON parser. See [_](/transforms/grok) and [_](/transforms/json-parsing).
+
+A tag and an attribute both work as the routing key. Choose one whose values are few and stable, because each value aggregates separately. Decide what happens to logs carrying no value for the key, which match no filter and so reach no account: give them a default value when you parse, or add a sink that catches them.
+
+### Step 2: Group the reducer on the routing key \[#step2-group]
+
+1. On the pipeline's **Reducer** step, click **Edit** on the **Fields** card.
+2. If the routing key is listed, set its **Behaviour** to **Match or split**.
+3. If it is not listed, enter it in the **Add field** row, set the **Behaviour** to **Match or split**, and click **Add**.
+4. Click **Save**.
+
+The **Match or split** behavior keeps a field out of aggregation, so messages whose values differ are never merged into one pattern. Enter tag keys plainly, and prefix attribute paths with `@`. An attribute is merged as well as grouped, and the default merge strategy keeps the value that every message in the group agrees on. If you set a different merge strategy for the routing attribute, check that it preserves that value rather than combining or discarding it. See [_](/transforms/reducer/configuration#attribute-configuration).
+
+### Step 3: Create a Grepr integration for each account \[#step3-integrations]
+
+Create one Grepr integration for each account you send to. An integration authenticates to a single account, so its credential is what decides where its logs land. Name each integration after its account, because that name is what you select when you add the sink. See [_](/integrations).
+
+### Step 4: Add a filtered sink for each account \[#step4-sinks]
+
+Repeat for each account:
+
+1. On the pipeline's **Sinks** step, click **Add sink**.
+2. In the **Sink** menu, select the integration for the account. Grepr fills in a **Name**, which you can change.
+3. In the **Query language** menu, select the syntax you want to write the filter in.
+4. In the **Filter Query** field, enter a query that matches only that account's logs, such as `team:platform`.
+5. Click **Add**, then save the pipeline.
+
+A sink with an empty filter forwards everything, so give each sink a filter query that matches it to one account. A sink's card reports its filter and shows **All messages** when it has none, which is how you spot one you missed. The filter runs on everything the reducer emits, so summaries, samples, passthrough messages, and exceptions all follow the same routing. To learn about the filter syntax, see [_](/queries/datadog) and [_](/queries/new-relic).
+
+## Frequently Asked Questions
+
+### Can one Grepr pipeline send logs to several accounts of the same vendor?
+
+Yes. Add one sink for each account, each pointing at that account's integration, and give each sink a filter that matches only that account's logs. Group the reducer on the same routing key the filters use, so that each summary belongs to exactly one account.
+
+### Why does the reducer need to group on the routing key?
+
+Without it, logs for different accounts aggregate together and the summary no longer identifies one account. If the key is a tag, the summary carries every value the merged messages had, so each account's filter matches it and every account receives counts that include the other accounts' logs. If the key is an attribute, the default merge strategy replaces the differing values with a wildcard, so no filter matches and the summary reaches no account.
+
+### Should I create a pipeline for each account or filter the sinks of one pipeline?
+
+Create a pipeline for each group when the groups need different reduction settings or different exceptions, or when you want them isolated. Filter the sinks of one pipeline when the groups share one reduction policy and you would rather keep a single pipeline to change.
+
+### What happens to logs that have no value for the routing key?
+
+They match no sink's filter and reach no vendor account. Give them a default value when you parse the routing key, or add a sink whose filter matches them so they land somewhere you can see.
+
+### Does a sink filter apply to summarized logs as well as raw ones?
+
+Yes. The filter runs on everything the reducer emits, so summaries, samples, passthrough messages, and exceptions all follow the same routing.

package/build/dist/docs-index/8c07f999-822b-4ce1-9e1c-7864b42be053.txt

@@ -0,0 +1,113 @@
+Process data in Grepr pipelines using built-in tools for parsing, filtering, enrichment, and transformation of log events.
+
+# Transform data in a Grepr pipeline
+
+A Grepr transform is a processing step that modifies, enriches, or restructures data as it flows through a pipeline. As records move through a pipeline, transforms allow you to parse structured and semi-structured data, filter records, and use a familiar SQL interface to perform more complex transformation of records as they move through a pipeline. Transforms are configured as pipeline steps that process data sequentially. Each transform receives input data, applies its logic, and passes the result to the next step in the pipeline.
+
+Most transforms are available in the Grepr pipelines UI. Some transforms are available only in the Grepr REST API or provide more capabilities when used with the API, such as [filters](#filter) and [branching the processing of records](#message-routing) based on query matches.
+
+The following sections describe the transforms available in Grepr pipelines.
+
+## Transform semi-structured records with the Grok parser \[#grok-parser]
+
+The Grok transform parses unstructured text using pattern matching with regular expressions and named capture groups. Use this transform to extract fields from log lines, error messages, and other text-based data where you need to identify specific patterns and convert them to structured fields.
+
+See [Grok parser](/transforms/grok).
+
+## Convert JSON messages into structured objects \[#json-parser]
+
+The JSON parser transforms JSON strings into structured JSON objects. This transform handles nested objects, arrays, and various data types, making it useful for processing API responses, configuration data, and structured log entries.
+
+See [Transform JSON strings into JSON objects](/transforms/json-parsing).
+
+## Restructure JSON objects with a remapper \[#remapper]
+
+The remapper transform moves or copies attributes to top-level fields or to tags in a JSON object. You can also use rename rules to move attribute values from one path to another within the same event. In the pipelines UI, a remapper with customizable default attributes is automatically added to a pipeline.
+
+See [Remap JSON attribute fields to top-level log event fields](/transforms/remapper).
+
+## Transform records with SQL \[#sql-transform]
+
+The SQL transform allows you to use ANSI standard SQL queries to filter, transform, and enrich data records. The SQL transform provides a familiar SQL interface to perform a number of data processing and transformation tasks.
+
+You can use the SQL transform in the Grepr UI or REST API. In the UI, the SQL transform is available when you configure filters.
+
+To learn about using the SQL transform in the Grepr API, see the [SQL Operation specification](/apis/api-spec#schema/SQLOperation).
+
+For a complete reference and examples, see [Transform events with SQL](/transforms/sql-transform).
+
+## Filter records based on conditions \[#filter]
+
+The Filter transform includes or excludes data records based on specified conditions. You can create filters using familiar query interfaces, such as SQL, as well as Datadog and New Relic log query-like syntaxes.
+
+In the Grepr pipelines UI, filters are located in predefined locations in the pipeline. To learn how to configure filters, see [Filter events in a Grepr pipeline](/transforms/filter-events).
+
+To have more flexibility for where filters are included in a pipeline, use the REST API. See [LogsFilter](/apis/api-spec#schema/LogsFilter).
+
+## Reduce log volumes with dynamic aggregation \[#reducer]
+
+The Grepr log reducer uses clustering algorithms to identify patterns in your log data and aggregate similar log messages. The output of this aggregation includes summaries and samples of similar messages. The log reducer is a core step in a log reduction pipeline, where it runs before processed log events are sent to your configured sinks. By sending summaries of similar messages rather than multiple events with redundant information, Grepr significantly reduces the volume of shipped logs and your observability costs. The log reduction pipeline also persists the complete set of raw logs to a low-cost data lake, enabling you to analyze and troubleshoot issues when necessary.
+
+To learn more about how Grepr reduces log volumes with dynamic aggregation, see [Reduce noise and improve the usability of your logs with the log reducer](/transforms/reducer).
+
+To learn how to create a log reduction pipeline, see [Tutorial: Build your first Grepr pipeline](/tutorials/first-pipeline).
+
+To learn how to use the resulting summarized log events in your observability platform along with raw logs stored in Grepr, see [Troubleshoot incidents with Grepr-processed data and your observability tool](/tutorials/troubleshooting-example).
+
+## Transform and enrich log messages with the REST API \[#log-transform-action]
+
+You can use the LogTransformAction in the REST API to specify actions that transform log messages, such as adding or removing attributes and tags. See [LogTransformAction](/apis/api-spec#schema/LogTransformAction).
+
+## Route the processing of messages based on query matches \[#message-routing]
+
+You can route log messages to be processed by different pipelines based on query matches. Because the pipelines UI doesn't support branching, Grepr recommends creating multiple pipelines that ingest data using the same source, and then use filters to route data to the correct pipeline.
+
+Sending logs to different destinations, rather than processing them differently, is a separate problem with its own approaches. See [_](/solution-guides/route-logs-to-several-accounts).
+
+You can also use the REST API to create more complex pipeline configurations. See [LogsBranch](/apis/api-spec#schema/LogsBranch).
+
+## Frequently Asked Questions
+
+### What transforms are available in Grepr pipelines?
+
+Grepr provides several transform types: Grok parser for parsing unstructured text with pattern matching, JSON parser for converting JSON strings into structured objects, remapper for restructuring JSON fields and attributes, SQL transform for complex filtering and enrichment using SQL queries, and Filter transform for conditional record processing. Most transforms are available in the UI, while some advanced options are available via REST API.
+
+### When should I use the Grok parser transform?
+
+Use the Grok parser when you need to extract structured data from unstructured text logs. It's ideal for parsing error messages, application logs, system logs, and other text-based data where you need to identify specific patterns and convert them into structured fields using regular expressions and named capture groups.
+
+### When should I use the JSON parser transform?
+
+Use the JSON parser when your log messages contain JSON strings that need to be converted into structured JSON objects. This is useful for processing API responses, configuration data, and structured log entries. The JSON parser handles nested objects, arrays, and various data types automatically.
+
+### What is the remapper transform and when is it used?
+
+The remapper transform moves or copies attributes to top-level fields or to tags in a JSON object. It's useful for standardizing log event structure by moving well-known fields like message, severity, and host from nested attributes to top-level fields. You can configure whether the source attribute is preserved or removed after remapping. The remapper also supports rename rules for moving attribute values from one path to another. The remapper is automatically added to pipelines in the UI with customizable default attributes.
+
+### When should I use the SQL transform?
+
+Use the SQL transform when you need complex filtering, transformation, or enrichment of log records. SQL provides a familiar interface for advanced operations like conditional logic, string manipulation, aggregations, and joining data. It's available in the UI for filters and via REST API for more flexibility.
+
+### Can I combine multiple transforms in a pipeline?
+
+Yes, transforms are configured as sequential pipeline steps. Each transform receives input from the previous step, applies its logic, and passes the result to the next step. This allows you to create complex data processing workflows by combining parsing (Grok), restructuring (remapper), and filtering/enrichment (SQL) transforms.
+
+### What is the Filter transform used for?
+
+The Filter transform includes or excludes data records based on specified conditions. You can use filters with SQL, Datadog, and New Relic-like query syntaxes. In the UI, filters are located in predefined pipeline positions, while the REST API provides more flexibility for filter placement.
+
+### Can I use the REST API for more advanced transform options?
+
+Yes, the REST API provides access to advanced transform options not available in the UI. For example, you can use LogTransformAction to add or remove attributes and tags, LogsBranch for complex message routing based on query matches, and more flexible filter placement. See the API specification for complete details on transform capabilities.
+
+### How do I choose between Grok and SQL transforms?
+
+Use Grok when you need pattern-based parsing of unstructured text to extract specific fields. Use SQL when you need complex conditional logic, calculations, string transformations, or filtering. Grok is specialized for text parsing, while SQL is more general-purpose for data manipulation and enrichment.
+
+### How does Grepr reduce log volumes and observability costs?
+
+Grepr uses clustering algorithms to identify patterns in log data and aggregates similar log messages. Instead of sending multiple redundant log events, Grepr sends summaries and samples of similar messages to your observability platform. This significantly reduces the volume of shipped logs while retaining the ability to analyze and troubleshoot issues. Raw logs are retained in a low-cost data lake for future reference.
+
+### What happens if a transform fails on a record?
+
+Transform behavior depends on the specific transform type and configuration. Parsing errors (like Grok or JSON parsing) typically skip records or add error attributes depending on your configuration. SQL transforms filter out records that don't match conditions. Review transform-specific documentation for error handling details.

package/build/dist/docs-index/8ea4d998-11b7-473e-8275-cfb406c0e950.txt

@@ -0,0 +1,79 @@
+Learn how Grepr integrates with Datadog, Splunk, New Relic, and other platforms to reduce observability costs and provide real-time log processing and analysis.
+
+# What is Grepr?
+
+Grepr is an enterprise-grade data storage, processing, and analytics platform for your observability data. Grepr collects, transforms, and analyzes your log data and then sends the processed logs to your observability platform.
+
+Before sending log data to the destination platform, Grepr uses machine learning to analyze and intelligently aggregate the data. During normal operations, Grepr reduces your observability costs by reducing the volume of data sent to your observability tools, while retaining all of your raw data in low-cost cloud storage. If an incident or anomaly occurs, Grepr increases the granularity of the forwarded data to ensure that sufficient data is available for troubleshooting. Grepr also uses the raw data to backfill any historical data relevant to the incident.
+
+Grepr supports popular observability platforms, including Datadog, Splunk, New Relic, Sumo Logic, Grafana Cloud, and OpenTelemetry-compatible platforms.
+
+Grepr AI agents build on the same data. An agent examines your logs and pipeline configuration with the capabilities you grant it and reports what it found. Start an investigation yourself, or connect an agent to a pipeline or another system so it investigates on its own. To learn more, see [_](/ai-agents).
+
+If you're just getting started with Grepr, follow the **Get started** guides below. If you're already using Grepr and need information on creating pipelines to process logs or to query raw logs stored in Grepr, see the guides under **Process and explore your logs**. If you need to automate your Grepr tasks or implement clients that use Grepr features, or use JavaScript to code reusable components for your pipelines, see the **Developers** documentation.
+
+## Get started
+
+[Create your first pipeline](/tutorials/first-pipeline)
+
+[Learn about the platform](/grepr-platform)
+
+## Process and explore your logs
+
+[Connect to external tools](/integrations)
+
+[Process and transform data](/transforms)
+
+[Query logs in the data lake](/queries)
+
+[Grepr AI agents](/ai-agents)
+
+## Developers
+
+[Use the Grepr Rest API](/apis)
+
+[Use the Grepr CLI](/cli)
+
+[Custom pipeline components](/templates)
+
+## Frequently Asked Questions
+
+### What is Grepr?
+
+Grepr is an enterprise-grade data storage, processing, and analytics platform for observability data. It collects, transforms, and analyzes log data, then sends processed logs to your observability platform while retaining raw data in low-cost cloud storage.
+
+### How does Grepr reduce observability costs?
+
+Grepr uses machine learning to intelligently aggregate and reduce the volume of log data sent to your observability tools during normal operations, while automatically increasing data granularity during incidents. This significantly reduces costs while maintaining full raw data access.
+
+### What observability platforms does Grepr integrate with?
+
+Grepr supports popular platforms including Datadog, Splunk, New Relic, Sumo Logic, Grafana Cloud, and OpenTelemetry-compatible platforms.
+
+### What is the Grepr data lake?
+
+The Grepr data lake is a secure cloud storage layer built on Apache Iceberg and S3 that stores your raw log data. It provides efficient querying capabilities and serves as the foundation for historical analysis and incident backfills.
+
+### Can I query raw logs in Grepr?
+
+Yes, you can query raw logs stored in the Grepr data lake using the query interface or REST API. The data lake supports powerful queries with tags, time ranges, and custom filters to analyze historical log data.
+
+### How do I create a Grepr pipeline?
+
+You can create pipelines using the Grepr UI, REST API, or CLI. Pipelines consist of data sources, transformations (like log reduction and parsing), and destinations. Follow the 'Create your first pipeline' guide to get started.
+
+### What happens to my data during an incident?
+
+During incidents or anomalies, Grepr automatically increases the granularity of forwarded data to ensure sufficient detail for troubleshooting. It also backtransfers historical raw logs relevant to the incident from the data lake.
+
+### Is my data secure in Grepr?
+
+Yes, Grepr maintains a SOC 2 Type II report and a HIPAA attestation, and implements enterprise security controls including encryption in transit (TLS 1.2+) and at rest, role-based access controls, and secure identity management with Auth0.
+
+### Can I use Grepr with my existing observability tools?
+
+Yes, Grepr is designed to integrate seamlessly with existing observability platforms. You continue using your current tools while Grepr handles data reduction and optimization in the background.
+
+### How can I automate Grepr tasks?
+
+You can automate tasks using the Grepr REST APIs, CLI, or custom JavaScript components in pipelines. The CLI supports job creation, queries, integration management, and dataset operations.