@f4bioo/berry-shield 2026.3.3-1

package/docs/wiki/engine/performance.md

@@ -0,0 +1,114 @@
+---
+summary: "Engine reference for performance profile, optimization strategy, and runtime tradeoffs"
+read_when:
+  - You are evaluating runtime cost of scanning and redaction
+  - You are tuning patterns for better latency and memory behavior
+  - You are validating behavior on constrained environments
+title: "performance"
+---
+
+# `Performance and optimization`
+
+Berry Shield is designed for practical security execution under real runtime constraints.
+The performance strategy prioritizes predictable behavior, bounded memory growth on common paths, and acceptable latency during content scanning.
+
+## Overview
+
+Engine performance is shaped by:
+- payload size and nesting depth
+- pattern set breadth and regex complexity
+- number of scan/interception hook invocations
+- match frequency (more matches generally means more replacement work)
+
+The current implementation favors change-only transformation and fast no-match returns where possible.
+
+## Optimization strategy
+
+### Lazy cloning
+
+Deep-cloning every payload on every pass is expensive.
+Berry runtime traversal follows a lazy approach:
+- keep original references when no changes are required
+- clone object branches only after actual transformation is needed
+
+This reduces memory churn and lowers garbage-collection pressure on no-change paths.
+
+### Circular-reference guard
+
+Recursive traversal uses seen-set protection for object graphs.
+This avoids repeated traversal loops and helps keep runtime stable for cyclic payloads.
+
+### Fast-path behavior
+
+Primitive/no-scan paths return quickly.
+On string paths, original content can be preserved when no redaction match is found.
+
+### Pattern caching and reuse
+
+Effective pattern sets are prepared and reused by category.
+This avoids rebuilding full pattern arrays on every operation path.
+
+## Runtime flow (cost perspective)
+
+```mermaid
+flowchart TD
+    A[Payload enters engine path] --> B{Fast path possible?}
+    B -->|yes| F[Return unchanged quickly]
+    B -->|no| T[Traverse and evaluate patterns]
+    T --> C{Any transformation needed?}
+    C -->|no| U[Reuse original references]
+    C -->|yes| L[Clone changed branches only]
+    L --> O[Return transformed payload]
+    U --> O
+```
+
+## LLM context efficiency impact
+
+Redaction and cleanup can reduce repeated sensitive-noise in model-visible artifacts and persisted transcripts.
+Operationally, this can improve context quality by:
+- reducing repetitive secret/token fragments
+- minimizing irrelevant high-entropy payload noise
+- preserving clearer task-relevant content after sanitation
+
+## Constrained environment profile
+
+Berry aims to stay usable on constrained devices and low-overhead deployments by combining:
+- lazy clone semantics
+- circular guard traversal
+- practical pattern tuning expectations
+
+Outcome depends on workload shape.
+Large deeply nested payloads with broad aggressive patterns still increase cost and should be tuned intentionally.
+
+## Practical tuning guidance
+
+1. Prefer targeted custom patterns over broad catch-all expressions.
+2. Validate new patterns with CLI test flows before production use.
+3. Use audit cycles to measure match volume and false-positive pressure.
+4. Keep tool outputs scoped where possible to reduce unnecessary scan volume.
+
+## Limits and caveats
+
+- Performance behavior remains workload-dependent and cannot be constant across all payload classes.
+- Regex quality directly affects both latency and detection reliability.
+- CLI safe-match timeout behavior applies to CLI matcher paths, not every runtime layer path.
+
+## Validation checklist
+
+1. Confirm no-match payloads preserve original references.
+2. Confirm changed payloads clone only modified branches.
+3. Confirm cyclic payloads are processed without recursion failure.
+4. Confirm catastrophic regex patterns are bounded in CLI safe matcher paths.
+
+## Related pages
+- [engine index](README.md)
+- [redaction](redaction.md)
+- [match engine](match-engine.md)
+- [decision posture](../decision/posture.md)
+
+---
+
+## Navigation
+- [Back to Engine Index](README.md)
+- [Back to Wiki Index](../README.md)
+

package/docs/wiki/engine/redaction.md

@@ -0,0 +1,120 @@
+---
+summary: "Engine reference for redaction behavior, pattern provenance, and transformation flow"
+read_when:
+  - You need to understand how sensitive content is detected and transformed
+  - You are validating output sanitation behavior in runtime layers
+  - You are reviewing pattern provenance and redaction behavior
+title: "redaction"
+---
+
+# `Redaction engine`
+
+The redaction engine is the central transformation layer for sensitive-content handling in Berry Shield.
+It is used by runtime layers that need to detect, classify, and sanitize secrets and PII in content payloads.
+
+## Overview
+
+In runtime terms, the engine provides:
+- text-level redaction for string payloads
+- recursive traversal for arrays and nested objects
+- key-based masking for sensitive field names
+- match metadata for audit/reporting paths
+
+This behavior feeds both observability (audit paths) and mitigation (enforce paths), depending on layer mode decisions.
+
+## Detection intelligence and pattern provenance
+
+The detection model combines multiple sources:
+
+- Static regex families for common credential and PII classes.
+- Community-derived secret patterns from Gitleaks source configuration.
+- Key-based masking for sensitive JSON/object field names (for example password/token-style keys).
+
+### Gitleaks provenance in Berry
+
+Berry includes a generated pattern source derived from Gitleaks configuration and tracked in source with explicit provenance metadata.
+This keeps pattern evolution practical while preserving local runtime control over placeholder behavior and layer-specific decisions.
+
+Important nuance:
+- Gitleaks-derived rules contribute to detection coverage.
+- Final behavior (observe, redact, deny, allow) still depends on Berry layer and mode logic.
+
+## Unescape handling
+
+Escaped payloads can hide sensitive tokens in operational logs and tool responses.
+Before regex evaluation on string paths, engine logic applies a practical unescape pass for common escaped forms.
+
+Why this matters:
+- improves detection on JSON-escaped and partially encoded outputs
+- reduces false negatives caused by escaped delimiters or unicode escape sequences
+
+## Runtime flow
+
+```mermaid
+flowchart TD
+    A[Raw payload] --> B{Payload type}
+    B -->|string| S[Unescape + string redaction path]
+    B -->|object/array| W[Recursive walk path]
+    S --> R[Pattern replacement loop]
+    W --> J{JSON-like string value?}
+    J -->|yes| P[Parse and recurse]
+    J -->|no| K[Key-based and value recursion]
+    K --> L[Lazy clone changed branches]
+    R --> O[Redaction result]
+    P --> O
+    L --> O
+```
+
+## Core behavior details
+
+### String redaction path
+
+- normalizes escaped content candidate
+- evaluates active patterns in sequence
+- replaces matched segments with pattern placeholders
+- preserves original text when no match exists
+
+### Recursive walk path
+
+- traverses arrays and objects deeply
+- guards against circular references with seen-set tracking
+- can parse JSON-like strings and recurse into parsed values
+- returns redaction stats (count and matched type names)
+
+### Key-based masking path
+
+For object keys recognized as sensitive:
+- primitive values may be replaced with key-derived mask marker
+- this complements regex detection where value format alone may be ambiguous
+
+## Where the redaction engine is used
+
+- [Pulp layer](../layers/pulp.md): output-side sanitation and redaction outcomes
+- [Leaf layer](../layers/leaf.md): inbound sensitivity signaling for audit visibility
+- [Thorn layer](../layers/thorn.md): match-only helper path for interception checks
+
+## Limits and caveats
+
+- Regex coverage is heuristic and pattern quality-dependent.
+- JSON-string parsing applies only when payload is parseable as JSON shape.
+- Unescape logic uses practical common forms, not universal decoding for all encodings.
+- Large payload size and pattern count directly affect processing cost.
+
+## Validation checklist
+
+1. Verify known secret strings map to expected placeholders.
+2. Verify nested-object traversal redacts only affected branches.
+3. Verify circular structures are handled without traversal failures.
+4. Verify escaped-token samples are detected after normalization.
+
+## Related pages
+- [engine index](README.md)
+- [match engine](match-engine.md)
+- [performance](performance.md)
+- [decision patterns](../decision/patterns.md)
+
+---
+
+## Navigation
+- [Back to Engine Index](README.md)
+- [Back to Wiki Index](../README.md)

package/docs/wiki/layers/README.md

@@ -0,0 +1,68 @@
+---
+summary: "Layer index for Berry Shield runtime architecture and cross-layer interaction map"
+read_when:
+  - You need a quick map of all six layers
+  - You are tracing a user-input to output-security path
+  - You are onboarding to Berry Shield internals
+title: "Layers Reference"
+---
+
+# `Layers reference`
+
+This page is the entry point for Berry Shield layer architecture.
+It explains what each layer owns and how layers interact during runtime.
+
+## Layer map
+
+- [root](root.md): policy injection strategy at turn start
+- [leaf](leaf.md): incoming-message audit and sensitive-signal logging
+- [stem](stem.md): gate tool for pre-operation allow/deny decisions
+- [thorn](thorn.md): hook-level pre-tool-call interception and blocking
+- [vine](vine.md): external-content trust guard for prompt-injection hardening
+- [pulp](pulp.md): output scanning, sanitation, and policy-block hygiene
+
+## End-to-end interaction (single view)
+
+```mermaid
+flowchart TD
+    U[User message] --> LEAF[Leaf: inbound audit signals]
+    U --> ROOT[Root: turn-start policy decision]
+    ROOT --> A[Agent reasoning]
+    A --> VINE[Vine: external-content trust guard]
+    A --> STEM[Stem: gate tool check]
+    A --> THORN[Thorn: before_tool_call interception]
+    VINE --> STEM
+    VINE --> THORN
+    STEM --> TOOLS[Tool execution path]
+    THORN --> TOOLS
+    TOOLS --> PULP[Pulp: output scan and sanitation]
+    A --> PULP
+    PULP --> OUT[Assistant response]
+```
+
+## Responsibility boundaries
+
+- Root controls policy presence in context, not hard operation execution.
+- Leaf provides observability on inbound content, not inbound blocking.
+- Stem evaluates operation intent through a gate tool path.
+- Thorn intercepts tool calls on hook path when host runtime supports invocation.
+- Vine tracks external-content risk signals and adds trust-aware guardrails.
+- Pulp sanitizes output paths and strips leaked policy snippets.
+
+## How to read this section
+
+Use this index in two passes:
+1. Read each layer page for exact runtime behavior and limits.
+2. Return to this map to confirm where a failure or mismatch belongs.
+
+## Related pages
+- [wiki index](../README.md)
+- [decision modes](../decision/modes.md)
+- [decision patterns](../decision/patterns.md)
+- [decision posture](../decision/posture.md)
+
+---
+
+## Navigation
+- [Back to Wiki Index](../README.md)
+- [Back to Repository README](../../../README.md)

package/docs/wiki/layers/leaf.md

@@ -0,0 +1,126 @@
+---
+summary: "Layer reference for Berry.Leaf (incoming message audit and sensitive-signal logging)"
+read_when:
+  - You need to understand what Leaf actually does in runtime
+  - You are validating input-observability behavior
+  - You are debugging why incoming messages are logged but not blocked
+title: "leaf"
+---
+
+# `Berry.Leaf`
+
+Berry.Leaf is the **incoming-message audit layer**.
+
+Despite the name Leaf, this layer is not a file blocker.  
+Its runtime role is: observe inbound user text, detect sensitive signals, and emit audit logs.
+
+## What Leaf does
+
+- Hooks into message_received.
+- Scans inbound message text with redaction patterns (secrets + PII).
+- Produces audit signals (containsSecrets, containsPII, sensitiveTypes).
+- Writes structured debug logs and warning logs for sensitive hits.
+
+In code terms, this is an observation pipeline, not an enforcement pipeline.
+
+## What Leaf does not do
+
+- It does not block incoming user messages.
+- It does not rewrite incoming user content before model processing.
+- It does not decide tool allow/deny.
+- It does not redact outbound messages.
+
+Reason: `message_received` is used as an observe-only path in this plugin design.
+
+## Runtime flow
+
+```mermaid
+flowchart TD
+    U[User Message] --> MR[Hook: message_received]
+    MR --> L[Berry.Leaf]
+    L --> S[Pattern Scan: secrets + PII]
+    S --> D{Sensitive match?}
+    D -->|No| LOG0[Debug audit log]
+    D -->|Yes| LOG1[Warn + structured audit log]
+    LOG0 --> NEXT[Continue runtime flow]
+    LOG1 --> NEXT
+```
+
+## Data extracted by Leaf
+
+For each non-empty inbound message, Leaf derives:
+
+- timestamp
+- session identifier (when available in context)
+- source/sender fields
+- message length
+- secret/PII presence flags
+- sensitive type list matched by patterns
+
+This gives operators evidence for risk monitoring without storing transformed user text as policy output.
+
+## How Leaf interacts with other layers
+
+Leaf is one signal producer in a layered system.
+
+### With Root
+- Root controls policy injection into agent context.
+- Leaf does not influence Root decisions directly.
+- Both can operate in the same turn, but on different concerns (guidance vs observability).
+
+### With Stem
+- Stem enforces security gate outcomes (`berry_check`) on operations.
+- Leaf can reveal that risky intent appeared in user input before operation attempts.
+- Stem is enforcement; Leaf is telemetry.
+
+### With Thorn
+- Thorn intercepts runtime tool calls (`before_tool_call`) when available.
+- Leaf can show early input indicators that later map to Thorn block events.
+
+### With Pulp
+- Pulp handles output-side redaction/hygiene.
+- Leaf handles input-side detection/auditing.
+- Together, they provide inbound and outbound visibility.
+
+## Operational value
+
+Leaf is useful for:
+
+- measuring sensitive-input frequency
+- building audit trails for compliance/security review
+- correlating "user asked for risky action" with later deny/redact events
+- tuning detection rules with real conversational signals
+
+## Limits and caveats
+
+- Leaf is only as strong as pattern coverage and matching quality.
+- If context fields are missing from runtime, session-level correlation can degrade.
+- Leaf should not be interpreted as a user-input firewall.
+- Use Leaf together with Stem/Thorn/Pulp for active mitigation posture.
+
+## Validation checklist
+
+Use this quick check during runtime validation:
+
+1. Send benign message and confirm debug audit entry appears.
+2. Send message containing test secret/PII marker and confirm warning audit entry appears.
+3. Confirm message still proceeds (Leaf observation behavior).
+4. Trigger a risky operation and correlate Leaf signal with Stem/Thorn outcomes.
+
+## See layers
+
+- [root](root.md)
+- [stem](stem.md)
+- [thorn](thorn.md)
+- [pulp](pulp.md)
+
+## Related pages
+- [layers index](README.md)
+- [decision modes](../decision/modes.md)
+- [decision patterns](../decision/patterns.md)
+
+---
+
+## Navigation
+- [Back to Layers Index](README.md)
+- [Back to Wiki Index](../README.md)

package/docs/wiki/layers/pulp.md

@@ -0,0 +1,139 @@
+---
+summary: "Layer reference for Berry.Pulp (output scanning, redaction paths, and policy-block hygiene)"
+read_when:
+  - You need to understand output-side sanitation behavior
+  - You are validating audit vs enforce output outcomes
+  - You are debugging redaction events or leaked policy snippets
+title: "pulp"
+---
+
+# `Berry.Pulp`
+
+Berry.Pulp is the **output scanning and sanitation layer**.
+
+It inspects tool results and outgoing messages, applies sensitive-pattern scanning, and handles mode-specific redaction behavior.
+
+## What Pulp does
+
+- Hooks into tool_result_persist for tool output persistence path.
+- Hooks into message_sending for outgoing assistant message path.
+- Runs pattern-based scan over content payloads.
+- Emits structured redaction events in both audit and enforce paths.
+- In enforce mode, returns sanitized content when redaction is required.
+- Applies policy-block hygiene by stripping leaked policy snippets from outgoing messages.
+
+## What Pulp does not do
+
+- It does not block tool execution requests.
+- It does not classify command/file operation risk for allow/deny.
+- It does not guarantee what the model already consumed before persistence-time hook execution.
+- It does not replace operation gate logic from Stem or interception logic from Thorn.
+
+## Runtime flow
+
+```mermaid
+flowchart TD
+    A[Content reaches output path] --> H{Hook path}
+    H -->|tool_result_persist| TP[Scan tool result content]
+    H -->|message_sending| MS[Strip leaked policy block, then scan]
+    TP --> M{Sensitive match?}
+    MS --> M
+    M -->|No| PASS[Return original/unchanged path]
+    M -->|Yes + audit| WR[Emit would_redact event]
+    M -->|Yes + enforce| RD[Emit redacted event and return sanitized content]
+    WR --> PASS
+```
+
+## Decision inputs
+
+Pulp consumes:
+- content payload from hook event
+- effective redaction patterns (built-in + custom)
+- runtime mode (audit or enforce)
+- hook path context (tool_result_persist or message_sending)
+
+## Decision behavior (high level)
+
+### Tool result persist path
+- scans tool result message content
+- if sensitive match:
+  - audit: emits observation event and keeps original content
+  - enforce: emits redaction event and returns sanitized message
+
+### Message sending path
+- first strips leaked policy block markers from outgoing content
+- then scans resulting content for sensitive match
+- if sensitive match:
+  - audit: emits observation event and generally does not rewrite for redaction
+  - enforce: emits redaction event and returns sanitized content
+- if only policy block leak is found, returns content with policy block removed
+
+## Policy-block hygiene behavior
+
+Pulp includes a dedicated outgoing hygiene step:
+- detects leaked policy wrapper snippets
+- removes them from outgoing message content
+- logs that policy leakage was stripped
+
+This hygiene step is independent from secret/PII redaction outcomes.
+
+## How Pulp interacts with other layers
+
+### With Stem
+- Stem decides whether risky operations should proceed.
+- Pulp sanitizes output content after content exists.
+
+### With Thorn
+- Thorn can stop risky calls before execution.
+- Pulp handles content-side sanitation for outputs that still reach output hooks.
+
+### With Root
+- Root injects security guidance text into context.
+- Pulp can remove leaked policy snippets before user delivery in outgoing messages.
+
+### With Leaf
+- Leaf provides inbound audit visibility.
+- Pulp provides outbound sanitation visibility.
+
+## Operational value
+
+Pulp is useful for:
+- reducing secret/PII exposure in stored and outbound content paths
+- providing structured output-side audit evidence
+- containing leaked policy-block echoes in user-visible responses
+
+## Limits and caveats
+
+- tool_result_persist is persistence-time; model output in the same turn may have already seen raw content.
+- behavior depends on host runtime hook availability/invocation.
+- pattern quality and coverage control redaction quality.
+- should be operated with Stem/Thorn for stronger defense-in-depth.
+
+## Validation checklist
+
+1. Send benign output content and confirm unchanged behavior.
+2. Produce output containing sensitive marker and confirm mode-specific outcome:
+   - audit: observation event
+   - enforce: sanitized output path
+3. Force leaked policy wrapper in outgoing content and confirm stripping behavior.
+4. Confirm report includes expected would_redact or redacted event counts.
+
+## See layers
+
+- [root](root.md)
+- [stem](stem.md)
+- [thorn](thorn.md)
+- [leaf](leaf.md)
+
+## Related pages
+- [layers index](README.md)
+- [decision modes](../decision/modes.md)
+- [decision patterns](../decision/patterns.md)
+- [decision posture](../decision/posture.md)
+- [engine redaction](../engine/redaction.md)
+
+---
+
+## Navigation
+- [Back to Layers Index](README.md)
+- [Back to Wiki Index](../README.md)

package/docs/wiki/layers/root.md

@@ -0,0 +1,130 @@
+---
+summary: "Layer reference for Berry.Root (policy injection strategy at agent turn start)"
+read_when:
+  - You need to understand how policy text is injected per turn
+  - You are validating profile/adaptive behavior in real sessions
+  - You are debugging repeated policy text or missing reminders
+title: "root"
+---
+
+# `Berry.Root`
+
+Berry.Root is the **policy-injection layer** that runs at agent turn start.
+
+Its role is to decide whether the turn receives:
+- full policy block
+- short reminder block
+- no injection
+
+This decision is computed from session state + policy config (profile/adaptive/retention).
+
+## What Root does
+
+- Hooks into before_agent_start.
+- Computes per-turn decision (full, short, none) using shared policy state.
+- Injects policy text through prependContext when decision is full or short.
+- Uses session identity when available; falls back to global_session and forces full for safety.
+- Clears per-session state on session_end.
+
+## What Root does not do
+
+- It does not hard-block tool execution by itself.
+- It does not redact content.
+- It does not classify file/command risk directly.
+- It does not replace Stem/Thorn/Pulp enforcement paths.
+
+Root is guidance and control-plane orchestration for prompt policy presence.
+
+## Runtime flow
+
+```mermaid
+flowchart TD
+    T[New Agent Turn] --> H[Hook: before_agent_start]
+    H --> R[Berry.Root]
+    R --> D{Decision}
+    D -->|full| F[Inject full policy block]
+    D -->|short| S[Inject short reminder block]
+    D -->|none| N[Skip injection]
+    F --> C[Continue turn]
+    S --> C
+    N --> C
+```
+
+## Decision inputs
+
+Root consumes:
+- session key (sessionId/sessionKey when available)
+- provider/message provider identity
+- profile (`strict`, `balanced`, `minimal`)
+- adaptive settings (stale window, escalation turns, heartbeat, global escalation)
+- retention limits for state tracking
+
+## Decision behavior (high level)
+
+- strict: full policy every turn.
+- balanced: first turn full; later turns typically none; short reminder on stale/heartbeat turns.
+- minimal: first turn none; mostly none; short reminder only on heartbeat.
+- forced-full windows can be triggered by adaptive escalation signals.
+- provider changes can trigger full reinjection.
+- missing session identity forces full policy for safety.
+
+## How Root interacts with other layers
+
+### With Stem
+- Root instructs model behavior (call berry_check first).
+- Stem is the actual gate tool that returns allow/deny decisions.
+- Root improves compliance probability; Stem provides enforceable gate responses.
+
+### With Thorn
+- Root guidance reduces risky tool attempts.
+- Thorn can still block tool calls at hook level where available.
+
+### With Pulp
+- Root influences what the agent attempts to output.
+- Pulp sanitizes output paths when sensitive material appears.
+
+### With Leaf
+- Root controls policy visibility cadence.
+- Leaf provides inbound observability; it does not drive Root decisions directly.
+
+## Operational value
+
+Root is useful for:
+- reducing repeated policy spam while preserving safety reminders
+- adapting policy visibility by session behavior
+- keeping first-turn security context explicit when required
+- supporting low-noise profiles without disabling enforcement layers
+
+## Limits and caveats
+
+- Root is instruction-level control, not hard execution enforcement.
+- If runtime context identity is missing, global safety fallback can increase policy verbosity.
+- Excessive full injections increase token cost.
+- Too little reinjection can reduce model adherence in long sessions.
+
+## Validation checklist
+
+1. Start a new session and confirm expected first-turn behavior for chosen profile.
+2. Send follow-up turns and confirm full/short/none cadence matches profile + adaptive config.
+3. Trigger denied events and verify forced-full window behavior on subsequent turns.
+4. End session and confirm state is cleared for the previous session id.
+
+## See layers
+
+- [leaf](leaf.md)
+- [stem](stem.md)
+- [thorn](thorn.md)
+- [pulp](pulp.md)
+
+## Related pages
+- [layers index](README.md)
+- [decision modes](../decision/modes.md)
+- [decision posture](../decision/posture.md)
+- [CLI profile](../operation/cli/profile.md)
+- [CLI policy](../operation/cli/policy.md)
+
+---
+
+## Navigation
+- [Back to Layers Index](README.md)
+- [Back to Wiki Index](../README.md)