rag-security-protocol 0.1.1rc1__tar.gz

rag_security_protocol-0.1.1rc1/.coderabbit.yaml

@@ -0,0 +1,107 @@
+# yaml-language-server: $schema=https://storage.googleapis.com/coderabbit_public_assets/schema.v2.json
+# The standards live in AGENTS.md, which CodeRabbit reads on its own. Path
+# instructions add only what is specific to a path; add one when a review misses it.
+
+language: 'en-US'
+
+# 250 char limit.
+tone_instructions: >-
+  Be terse and concrete. One sentence per finding, naming the failure. No praise,
+  no summarising what the diff does, no restating a rule without applying it. If
+  the diff is clean, say so and stop.
+
+reviews:
+  profile: 'chill'
+  poem: false
+  high_level_summary: true
+  request_changes_workflow: false
+
+  auto_review:
+    enabled: true
+    drafts: false
+
+  path_instructions:
+    - path: 'rsp/**/*.py'
+      instructions: >-
+        The two that matter most here: every error path must end in BLOCK unless
+        a per-plugin on_error says otherwise, and a plugin-supplied span must be
+        validated (in range, start <= end, on a UTF-8 boundary) before anything
+        slices with it.
+
+    - path: 'rsp/guards.py'
+      instructions: >-
+        Under 80 lines of code and no detection logic.
+
+    - path: '{plugins,examples}/**'
+      instructions: >-
+        Adapters wrap unmodified upstream binaries and contain no detection
+        logic — flag any the wrapped tool already performs. Under 80 lines of
+        code, except where a brace-per-line language pays for syntax rather
+        than behaviour. One JSON object on stdout per call; diagnostics on
+        stderr.
+
+    - path: 'examples/**/*.ts'
+      instructions: >-
+        Position arithmetic must produce byte offsets into the UTF-8 encoding of
+        the content, never string indices — flag any use of .length, .slice, or
+        .indexOf on a string where an offset is being computed. The conversion
+        functions must stay callable without the wrapped binary installed.
+
+    - path: 'SPEC.md'
+      instructions: >-
+        Flag any new MUST without a conformance fixture in the same PR, and any
+        clause without a Rationale.
+
+  tools:
+    # One entry per language present in the tree. The LLM review is
+    # language-agnostic; these are not.
+    ruff:
+      enabled: true          # skipped automatically once ruff runs in CI
+    oxc:
+      enabled: true          # TypeScript, no config file of its own to keep
+    biome:
+      enabled: true
+    eslint:
+      enabled: false         # would mean an eslint config and a dev dependency
+                             # for a hundred lines of example code
+    shellcheck:
+      enabled: true
+    actionlint:
+      enabled: true
+    zizmor:
+      enabled: true
+    yamllint:
+      enabled: true
+    gitleaks:
+      enabled: true          # the demo corpus will carry fabricated secrets
+    languagetool:
+      enabled: false         # prose nitpicks on a doc-heavy repo
+    markdownlint:
+      enabled: false
+
+  pre_merge_checks:
+    # A coverage threshold asks for more docstrings; AGENTS.md's Style asks for fewer.
+    docstrings:
+      mode: 'off'
+
+    custom_checks:
+      - name: 'Fail closed'
+        mode: 'warning'
+        instructions: >-
+          If this PR adds or changes an error path in rsp/ — an except handler, a
+          timeout branch, a parse failure, a size cap — verify it results in
+          BLOCK unless an explicit per-plugin on_error setting says otherwise.
+          Report a failure naming the file and line if such a path results in
+          ALLOW, skips the plugin silently, or lets the exception escape.
+          Scope: an error path is one that makes a response untrustworthy, per
+          SPEC.md E3. Failures that cost only diagnostics — an unreadable or
+          truncated stderr — are not error paths, and code that lets them pass
+          is correct, not a finding.
+
+      - name: 'MUST has a fixture'
+        mode: 'warning'
+        instructions: >-
+          If this PR adds or modifies a MUST, MUST NOT, or SHALL clause in
+          SPEC.md, verify the same PR adds or updates a conformance case under
+          conformance/cases/ that exercises it. Report a failure naming each
+          clause left without one.

rag_security_protocol-0.1.1rc1/.gitattributes

@@ -0,0 +1,5 @@
+# Linguist counts examples/ as documentation and leaves it out of the language
+# bar. Here the six adapters under it are implementations: the conformance kit
+# runs them as plugins, and the claim this repository makes is that a plugin is
+# a command in any language. A bar reading 100% Python says the opposite.
+examples/** linguist-documentation=false

rag_security_protocol-0.1.1rc1/.github/workflows/ci.yml

@@ -0,0 +1,71 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+# nltk, which the sentence splitter pulls in, refuses the hardlinks uv makes by default.
+env:
+  UV_LINK_MODE: copy
+
+# A third party's action is pinned to a hash: whoever can move its tag can run
+# code with this job's token. GitHub's own are pinned to a major tag (AGENTS.md).
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+        with:
+          # Nothing pushes, so the token is not left behind in .git/config.
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6.0.1
+        with:
+          enable-cache: true
+      # --frozen fails when uv.lock is out of date with pyproject.toml.
+      - run: uv sync --frozen
+      - run: uv run ruff check rsp/ tests/ plugins/ conformance/ examples/gitleaks-py/ examples/host/
+      - run: uv run ruff format --check rsp/ tests/ plugins/ conformance/ examples/gitleaks-py/ examples/host/
+
+      # actionlint for expression and matrix mistakes, zizmor for workflow security.
+      - run: uvx --from actionlint-py==1.7.12.24 actionlint .github/workflows/*.yml
+      - run: uvx zizmor==1.9.0 --persona=regular .github/workflows/
+
+  test:
+    strategy:
+      fail-fast: false
+      matrix:
+        # From the floor in pyproject.toml to whatever is current.
+        python: ["3.11", "3.12", "3.13", "3.14"]
+        os: [ubuntu-latest]
+        # One macOS job: process groups and pipe buffers (16 KiB, not 64) differ from Linux.
+        include:
+          - python: "3.12"
+            os: macos-latest
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v5
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6.0.1
+        with:
+          enable-cache: true
+          python-version: ${{ matrix.python }}
+      - run: uv sync --frozen
+      # Unit tests, conformance cases and drift checks are all pytest.
+      - run: uv run pytest tests/ examples/gitleaks-py/ -q --cov --cov-report=xml
+
+      # One upload: coverage does not differ across the matrix.
+      - if: matrix.os == 'ubuntu-latest' && matrix.python == '3.12'
+        uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1
+        with:
+          files: coverage.xml
+          token: ${{ secrets.CODECOV_TOKEN }}
+          fail_ci_if_error: false

rag_security_protocol-0.1.1rc1/.github/workflows/plugins.yml

@@ -0,0 +1,191 @@
+name: Plugins
+
+# Separate from CI, and scheduled, because a failure here can mean upstream changed:
+# gitleaks retiring a rule changes a verdict without any pull request.
+
+on:
+  push:
+    branches: [main]
+    paths: &paths
+      - "examples/**"
+      - "conformance/**"
+      - "tests/**"
+      # The suite runs the host too.
+      - "rsp/**"
+      - "pyproject.toml"
+      - "SPEC.md"
+      - ".github/workflows/plugins.yml"
+  pull_request:
+    paths: *paths
+  schedule:
+    - cron: "23 5 * * 1"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: plugins-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  # One pin for the binary and for the test corpus in its source archive.
+  GITLEAKS_VERSION: 8.30.1
+  UV_LINK_MODE: copy  # see ci.yml
+
+jobs:
+  gitleaks:
+    # One job for every adapter over this binary: the cases are shared, and
+    # disagreement is the finding. Two OSes, because the binary and the pipe differ.
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - runner: ubuntu-latest
+            platform: linux_x64
+          - runner: macos-latest
+            platform: darwin_arm64
+    runs-on: ${{ matrix.runner }}
+    steps:
+      - uses: actions/checkout@v5
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6.0.1
+        with:
+          enable-cache: true
+      # The image supplied Node 22, where the test runner rejected a directory.
+      - uses: actions/setup-node@v6
+        with:
+          node-version: '24'
+          cache: npm
+          cache-dependency-path: examples/gitleaks-ts/package-lock.json
+      - uses: actions/setup-go@v7
+        with:
+          go-version: '1.24'
+          cache: false  # stdlib only, so there is no go.sum to key on
+
+      - name: Install gitleaks
+        run: |
+          set -euo pipefail
+          base="https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}"
+          asset="gitleaks_${GITLEAKS_VERSION}_${{ matrix.platform }}.tar.gz"
+          sums="gitleaks_${GITLEAKS_VERSION}_checksums.txt"
+          curl -sSfLO "${base}/${asset}"
+          curl -sSfLO "${base}/${sums}"
+          grep " ${asset}\$" "${sums}" | shasum -a 256 --check --strict -
+          tar -xzf "${asset}" gitleaks
+          sudo install -m 755 gitleaks /usr/local/bin/gitleaks
+          gitleaks version
+
+      - name: Unit tests and types
+        working-directory: examples/gitleaks-ts
+        run: |
+          npm ci
+          npm run typecheck
+          npm test
+
+      # Built once, not `go run` per call: the corpus suite calls it four hundred times.
+      - name: Build and test the Go plugin
+        working-directory: examples/gitleaks-go
+        run: |
+          test -z "$(gofmt -l .)"
+          go vet ./...
+          # -count=1: the offset table lives outside this module, so the cache misses changes.
+          go test -count=1 ./...
+          go build -o "${RUNNER_TEMP}/rsp-gitleaks-go" .
+
+      - uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: '21'
+
+      # Keyed on the Makefile, which holds both the version and the checksum:
+      # a jar restored from this cache meets the same check a fetched one does,
+      # and a rate-limited Maven Central has failed this job once already.
+      - uses: actions/cache@v6
+        with:
+          path: examples/gitleaks-java/lib
+          key: gson-${{ hashFiles('examples/gitleaks-java/Makefile') }}
+
+      # gson is pinned and checksummed by the Makefile, the way this job pins
+      # gitleaks: Java has no JSON in its standard library either.
+      - name: Build and test the Java plugin
+        working-directory: examples/gitleaks-java
+        run: make test
+
+      # macOS only, and the manifest says so rather than this file: the
+      # runner ships Swift, while Linux would mean a 748MB tarball or
+      # swift-actions/setup-swift, whose key import is currently broken
+      # upstream. What Swift demonstrates — that the byte discipline can be
+      # the compiler's — it demonstrates on one platform, and the tests
+      # skipping elsewhere say which reason it was.
+      - name: Build and test the Swift plugin
+        if: runner.os == 'macOS'
+        working-directory: examples/gitleaks-swift
+        run: make test
+
+      - name: Install cJSON
+        run: |
+          if [ "$RUNNER_OS" = "Linux" ]; then
+            sudo apt-get update && sudo apt-get install -y libcjson-dev
+          else
+            brew install cjson
+          fi
+
+      # Twice: once as it ships, once under the sanitizers. Refusing a Match
+      # shorter than its span is memory safety, not behaviour, so no assertion
+      # here can see it go — ASan can.
+      - name: Build and test the C plugin
+        working-directory: examples/gitleaks-c
+        run: |
+          make test
+          make clean
+          make CFLAGS="-O1 -g -std=c11 -Wall -Wextra -Werror -fsanitize=address,undefined $EXTRA_I" \
+               LDFLAGS="-fsanitize=address,undefined $EXTRA_L" test
+          make clean
+          make rsp-gitleaks-c
+        env:
+          EXTRA_I: ${{ runner.os == 'macOS' && '-I/opt/homebrew/include' || '' }}
+          EXTRA_L: ${{ runner.os == 'macOS' && '-L/opt/homebrew/lib' || '' }}
+
+      # Release build: a debug build spends longer starting than scanning.
+      - name: Build and test the Rust plugin
+        working-directory: examples/gitleaks-rs
+        run: |
+          cargo fmt --check
+          cargo clippy --all-targets -- -D warnings
+          cargo test
+          cargo build --release
+          cp target/release/rsp-gitleaks-rs "${RUNNER_TEMP}/rsp-gitleaks-rs"
+
+      # Other people's test secrets, kept out of the checkout.
+      - name: Fetch the corpus
+        run: |
+          set -euo pipefail
+          mkdir -p "${RUNNER_TEMP}/corpus"
+          curl -sSfL \
+            "https://github.com/gitleaks/gitleaks/archive/refs/tags/v${GITLEAKS_VERSION}.tar.gz" \
+            | tar -xz -C "${RUNNER_TEMP}/corpus" --strip-components=2 \
+              "gitleaks-${GITLEAKS_VERSION}/testdata"
+          find "${RUNNER_TEMP}/corpus" -type f | wc -l
+
+      # The Python adapter needs no build step, but `pytest tests/` does not
+      # reach examples/, so it is named rather than assumed.
+      # Nothing skips here, so this is the only run of every clause against every plugin.
+      - name: Every case, every tool present
+        run: uv run --frozen pytest tests/ examples/gitleaks-py/ -q --html=report.html --self-contained-html
+        env:
+          RSP_REQUIRE_ALL_PLUGINS: "1"
+          RSP_GITLEAKS_CORPUS: ${{ runner.temp }}/corpus
+          RSP_PLUGIN_GITLEAKS_GO: ${{ runner.temp }}/rsp-gitleaks-go
+          RSP_PLUGIN_GITLEAKS_RS: ${{ runner.temp }}/rsp-gitleaks-rs
+          RSP_PLUGIN_GITLEAKS_C: ${{ github.workspace }}/examples/gitleaks-c/rsp-gitleaks-c
+          RSP_PLUGIN_GITLEAKS_JAVA: ${{ github.workspace }}/examples/gitleaks-java/rsp-gitleaks-java
+          RSP_PLUGIN_GITLEAKS_SWIFT: ${{ github.workspace }}/examples/gitleaks-swift/.build/release/rsp-gitleaks-swift
+
+      # always(): a failed run's report is the one worth reading. One OS is enough.
+      - if: always() && matrix.runner == 'ubuntu-latest'
+        uses: actions/upload-artifact@v7
+        with:
+          name: conformance-report
+          path: report.html

rag_security_protocol-0.1.1rc1/.github/workflows/publish.yml

@@ -0,0 +1,49 @@
+# Publishing is a tag push, and the tag is the only trigger: a release that can
+# be made by hand is a release whose contents nobody can derive from the tree.
+name: Publish
+
+on:
+  push:
+    tags: ["v*"]
+
+permissions:
+  contents: read
+
+env:
+  UV_LINK_MODE: copy  # see ci.yml
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      # Restated, because a job's block replaces the workflow's rather than
+      # adding to it, and checkout would be left reading a public repository
+      # only by its being public.
+      contents: read
+      # Trusted publishing: PyPI verifies this run, so no token is stored here.
+      id-token: write
+    steps:
+      - uses: actions/checkout@v5
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6.0.1
+        with:
+          # A cache another run could have written has no business deciding
+          # what goes to PyPI.
+          enable-cache: false
+
+      # The suite before the upload, because a published version cannot be taken
+      # back — PyPI refuses a reused filename even after a delete.
+      - run: uv sync --frozen --extra llamaindex
+      - run: uv run --frozen pytest tests/ examples/gitleaks-py/ -q
+
+      - name: The tag and the version must be the same release
+        run: |
+          set -euo pipefail
+          declared=$(uv run --frozen python -c 'import tomllib; print(tomllib.load(open("pyproject.toml","rb"))["project"]["version"])')
+          tagged=${GITHUB_REF_NAME#v}
+          test "$declared" = "$tagged" || { echo "pyproject says $declared, the tag says $tagged"; exit 1; }
+
+      - run: uv build
+      - uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2

rag_security_protocol-0.1.1rc1/.gitignore

@@ -0,0 +1,19 @@
+.venv/
+__pycache__/
+*.pyc
+.coverage
+coverage.xml
+.ruff_cache/
+.pytest_cache/
+report.html
+
+# Build output. Committed once by a `git add -A` after a branch switch took
+# the .gitignore that had been covering it, so the rule lives here now, where
+# no checkout can remove it while the artefacts remain.
+*.o
+examples/gitleaks-c/rsp-gitleaks-c
+examples/gitleaks-c/test-gitleaks
+examples/gitleaks-java/classes/
+examples/gitleaks-java/lib/
+examples/gitleaks-java/rsp-gitleaks-java
+examples/gitleaks-swift/.build/

rag_security_protocol-0.1.1rc1/.python-version

@@ -0,0 +1 @@
+3.12

rag_security_protocol-0.1.1rc1/AGENTS.md

@@ -0,0 +1,104 @@
+# AGENTS.md
+
+RSP is a protocol. `SPEC.md` is the deliverable; the code is evidence that it
+works. Decisions (D1–D9) and open questions (Q1–Q8) are in the
+[wiki](https://github.com/PCBZ/rsp/wiki) — cite them by number, don't re-argue them.
+
+## Layout
+
+Each layer knows strictly less than the one above it, and the imports go one way.
+
+- `rsp/process.py` — start a plugin, feed it, drain it, reap it. Knows no JSON.
+- `rsp/codec.py` — a request in, a parsed response out. Knows no plugin identity.
+- `rsp/handshake.py` — who a plugin says it is.
+- `rsp/spans.py` — byte offsets and redaction. Knows no verdicts.
+- `rsp/runtime.py` — the deciding layer: dispatch, compose, fail closed.
+- `rsp/guards.py` — LlamaIndex adapter. Knows no transport.
+- `rsp/config.py` — plugins from a file. Not in the spec; a host may ignore it.
+- `rsp/ingest.py` — a directory through the LlamaIndex pipeline, for the demo.
+- `rsp/cli.py` — `rsp validate` and `rsp ingest`.
+- `plugins/*` — untrusted third-party code. Detection lives only here.
+
+A change that makes a lower layer import a higher one is a design change, not a
+convenience.
+
+## Commands
+
+```bash
+uv sync
+uv run ruff check rsp/ tests/ plugins/ conformance/ examples/gitleaks-py/ examples/host/
+uv run ruff format --check rsp/ tests/ plugins/ conformance/ examples/gitleaks-py/ examples/host/
+uv run pytest
+```
+
+## Rules
+
+- Every error path ends in `BLOCK` unless that plugin's `on_error` says otherwise (D3).
+- Validate a plugin's span before slicing with it: in range, `start <= end`, on a UTF-8 boundary (D6).
+- Findings never reach an embedding or the LLM — both `excluded_*_metadata_keys` (Q8).
+- Never log, embed, or echo matched content. Demo secrets are fabricated.
+- No detection logic in `rsp/`. No dependencies in the core; `llama-index-core` is an extra.
+- Ignore unknown protocol fields (D8).
+- Cache key is `hash(content + plugin_version + plugin_config)`; never cache a plugin that didn't declare determinism (D4).
+- Timeout every read and wait, then kill and reap. Plugin commands are argv lists, never `shell=True`.
+- Strictest verdict wins, `BLOCK` short-circuits (D9).
+- `guards.py` and each plugin adapter contain no detection logic and stay
+  small: under 80 lines of code, counting neither docstrings nor comments nor
+  blank lines. The budget is on `guards.py` and on the file wrapping the tool
+  — `gitleaks.c`, `Gitleaks.java`, `gitleaks.rs` — not on everything beside
+  it. A strict gate reads the bytes T4 binds before anything is deserialised,
+  and a span helper converts someone else's positions; those are other jobs,
+  in other files, and unbudgeted is not unbounded. A language that puts a
+  closing brace or a struct tag on its own line pays a fixed tax that is not
+  growth, so those lines do not count either —
+  `examples/gitleaks-go/gitleaks.go` is 136 lines, of which 80 do work. A
+  test does the counting, so a disagreement is about the work rather than
+  about the arithmetic. Judge the work, and say which it is when the number
+  goes over.
+
+## Conventions
+
+- Work top-down: write the call site first, then the code it calls. A file may import a module that does not exist yet — don't stub it, make it run, or test it. This is about the order you write in, not the order things sit in a file.
+- Every MUST in `SPEC.md` gets a conformance fixture the same day. Fixtures assert protocol behaviour, not implementation.
+- Comments cite decisions by number: `# span application is the runtime's job (D6)`.
+- Never an issue number. `D6` and `S1` resolve inside a clone; `#13` resolves
+  only against a live tracker, and only until that issue is renumbered. Say the
+  reason or cite the clause — a test enforces this.
+- `ruff` owns formatting and annotation style.
+- A workflow pins an action from `actions/*` to its major tag and everything
+  else to a hash, because a third party's tag can be moved and one of these
+  workflows uploads a release. The lint job enforces half of that: `zizmor`
+  refuses an unpinned third party and accepts any ref under `actions/*`, so
+  `actions/checkout@main` passes it. The major tag is the convention, not the
+  check.
+- One issue per PR, titled `#N Short summary`. Out-of-scope findings become
+  issues, not scope creep. A PR carries the labels of the issue it closes, or
+  of the areas it touched where it closes none, so that what a change touched
+  is answerable without reading it.
+
+## Style
+
+Written down so that a style finding cites a rule rather than a preference.
+`ruff` enforces what it can (`pyproject.toml`); the rest is for reviewers.
+
+- A comment says why, never what. If the code already says it, delete the comment.
+- One line where one will do. Cite the clause instead of restating it: `(D6)`,
+  `(S3, V3)`, `§4`.
+- A reason lives in one place. Protocol rationale belongs to `SPEC.md`; code
+  cites the clause. The same reason is not written in two files.
+- No history. Not what the code used to do, not the bug that prompted a check —
+  `git log` keeps that. Keep the reason that still holds.
+- Docstrings: a one-line summary. A body only for what the caller cannot learn
+  from the signature, after a blank line, with the closing `"""` on its own line.
+  A test's name says what holds; its docstring, if any, says why.
+- In `rsp/`, a name used only inside its module starts with `_`: no underscore
+  means public API. A caught exception is `exc`.
+- Constants come straight after the imports. A helper sits above its first
+  caller; `main` comes last.
+- Every parameter and return in `rsp/` is annotated.
+
+## Reviewers
+
+Skip formatting, non-running files, missing tests for call sites, spawn-per-call
+performance (Q6), and matters of taste. One sentence per finding, naming the
+failure. Nothing to say is a valid review.