runflect 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- runflect-0.1.0/PKG-INFO +720 -0
- runflect-0.1.0/README.md +688 -0
- runflect-0.1.0/pyproject.toml +46 -0
- runflect-0.1.0/setup.cfg +4 -0
- runflect-0.1.0/src/runflect/__init__.py +26 -0
- runflect-0.1.0/src/runflect/__main__.py +5 -0
- runflect-0.1.0/src/runflect/bench/__init__.py +4 -0
- runflect-0.1.0/src/runflect/bench/fixtures.py +33 -0
- runflect-0.1.0/src/runflect/bench/metrics.py +91 -0
- runflect-0.1.0/src/runflect/bench/regression.py +320 -0
- runflect-0.1.0/src/runflect/bench/report.py +76 -0
- runflect-0.1.0/src/runflect/bench/runner.py +549 -0
- runflect-0.1.0/src/runflect/bench/schema.py +244 -0
- runflect-0.1.0/src/runflect/bench/suites/v0_1.json +106 -0
- runflect-0.1.0/src/runflect/cli.py +862 -0
- runflect-0.1.0/src/runflect/integrations/__init__.py +1 -0
- runflect-0.1.0/src/runflect/integrations/langgraph.py +453 -0
- runflect-0.1.0/src/runflect/integrations/opentelemetry.py +408 -0
- runflect-0.1.0/src/runflect/policy.py +553 -0
- runflect-0.1.0/src/runflect/policy_files.py +402 -0
- runflect-0.1.0/src/runflect/pricing.py +139 -0
- runflect-0.1.0/src/runflect/recorder.py +433 -0
- runflect-0.1.0/src/runflect/security_simulation.py +714 -0
- runflect-0.1.0/src/runflect/storage.py +220 -0
- runflect-0.1.0/src/runflect/supply_chain.py +875 -0
- runflect-0.1.0/src/runflect/terminal.py +85 -0
- runflect-0.1.0/src/runflect/web_api.py +1536 -0
- runflect-0.1.0/src/runflect/web_cli.py +9 -0
- runflect-0.1.0/src/runflect.egg-info/PKG-INFO +720 -0
- runflect-0.1.0/src/runflect.egg-info/SOURCES.txt +44 -0
- runflect-0.1.0/src/runflect.egg-info/dependency_links.txt +1 -0
- runflect-0.1.0/src/runflect.egg-info/entry_points.txt +3 -0
- runflect-0.1.0/src/runflect.egg-info/requires.txt +16 -0
- runflect-0.1.0/src/runflect.egg-info/top_level.txt +1 -0
- runflect-0.1.0/tests/test_bench.py +203 -0
- runflect-0.1.0/tests/test_bench_regression.py +134 -0
- runflect-0.1.0/tests/test_cli_ux.py +198 -0
- runflect-0.1.0/tests/test_langgraph.py +608 -0
- runflect-0.1.0/tests/test_opentelemetry.py +271 -0
- runflect-0.1.0/tests/test_policy.py +349 -0
- runflect-0.1.0/tests/test_policy_files.py +191 -0
- runflect-0.1.0/tests/test_pricing.py +81 -0
- runflect-0.1.0/tests/test_recorder.py +284 -0
- runflect-0.1.0/tests/test_security_simulation.py +315 -0
- runflect-0.1.0/tests/test_supply_chain.py +167 -0
- runflect-0.1.0/tests/test_web_api.py +628 -0
runflect-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,720 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: runflect
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Local-first evidence capture for AI agent executions
|
|
5
|
+
Project-URL: Homepage, https://github.com/bevinkatti/Runflect
|
|
6
|
+
Project-URL: Repository, https://github.com/bevinkatti/Runflect
|
|
7
|
+
Project-URL: Documentation, https://github.com/bevinkatti/Runflect#readme
|
|
8
|
+
Project-URL: Issues, https://github.com/bevinkatti/Runflect/issues
|
|
9
|
+
Classifier: Development Status :: 3 - Alpha
|
|
10
|
+
Classifier: Environment :: Console
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Operating System :: OS Independent
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
18
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
19
|
+
Classifier: Topic :: System :: Monitoring
|
|
20
|
+
Requires-Python: >=3.10
|
|
21
|
+
Description-Content-Type: text/markdown
|
|
22
|
+
Provides-Extra: langgraph
|
|
23
|
+
Requires-Dist: langgraph<2.0,>=1.2.14; extra == "langgraph"
|
|
24
|
+
Provides-Extra: opentelemetry
|
|
25
|
+
Requires-Dist: opentelemetry-api<2.0,>=1.27; extra == "opentelemetry"
|
|
26
|
+
Requires-Dist: opentelemetry-sdk<2.0,>=1.27; extra == "opentelemetry"
|
|
27
|
+
Provides-Extra: web
|
|
28
|
+
Requires-Dist: fastapi<1.0,>=0.115; extra == "web"
|
|
29
|
+
Requires-Dist: uvicorn<1.0,>=0.30; extra == "web"
|
|
30
|
+
Provides-Extra: supply-chain
|
|
31
|
+
Requires-Dist: tomli<3,>=2; python_version < "3.11" and extra == "supply-chain"
|
|
32
|
+
|
|
33
|
+
# Runflect
|
|
34
|
+
|
|
35
|
+
### The reliability and security toolkit for AI agents.
|
|
36
|
+
|
|
37
|
+
Trace executions. Enforce tool-use policies. Benchmark agent behavior. Investigate failures with inspectable evidence.
|
|
38
|
+
|
|
39
|
+
An agent calls a tool it shouldn't. A workflow fails halfway through. A policy change introduces a regression you didn't anticipate.
|
|
40
|
+
|
|
41
|
+
When something goes wrong, scattered logs and inconsistent traces make it difficult to understand what happened, identify the failure, and verify whether a fix actually works.
|
|
42
|
+
|
|
43
|
+
**Runflect turns agent execution into actionable evidence.**
|
|
44
|
+
|
|
45
|
+
Record execution traces, control tool access through explicit policies, evaluate security boundaries with repeatable scenarios, and catch regressions through benchmark comparisons—all in one local-first toolkit for developers building AI agents.
|
|
46
|
+
**Observe execution. Enforce boundaries. Test before you trust.**
|
|
47
|
+
|
|
48
|
+
[Quickstart](#quickstart) · [Features](#what-runflect-provides) · [CLI](#cli-20) · [Architecture](docs/architecture.md) · [Local dashboard](docs/web.md)
|
|
49
|
+
|
|
50
|
+
> **Project status:** Runflect is an early-stage developer preview. The distribution name is `runflect`, but public PyPI installation is not yet available. Install from a source checkout using the instructions below.
|
|
51
|
+
|
|
52
|
+
> **Important:** Runflect provides observability and policy enforcement at instrumented boundaries. It does not reveal hidden model reasoning, guarantee agent security, or automatically explain every failure.
|
|
53
|
+
|
|
54
|
+
---
|
|
55
|
+
|
|
56
|
+
## What Runflect provides
|
|
57
|
+
|
|
58
|
+
Runflect brings agent execution observability, tool-use controls, evaluation, and local security evidence into one developer workflow.
|
|
59
|
+
|
|
60
|
+
| Capability | What it does |
|
|
61
|
+
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
62
|
+
| **Execution tracing** | Records runs, nested spans, timing, status, and bounded typed attributes in a versioned SQLite database. |
|
|
63
|
+
| **LangGraph integration** | Optional synchronous and asynchronous adapters record observed graph, node, tool, and supported model callback events. |
|
|
64
|
+
| **Tool-use policies** | `ToolGuard` supports allow, deny, approval-required, and configurable default-deny decisions around wrapped tool calls. |
|
|
65
|
+
| **Policy lifecycle** | Validate policies, import immutable revisions, review history, evaluate scenarios offline, and explicitly activate revisions. |
|
|
66
|
+
| **Runflect Bench** | Run deterministic evaluation scenarios and inspect outcomes, metrics, and host-specific timing observations. |
|
|
67
|
+
| **Regression testing** | Save versioned baselines, compare compatible reports, apply supported thresholds, and use CI-compatible exit codes. |
|
|
68
|
+
| **Security simulation** | Execute deterministic offline mock scenarios against the actual guarded execution path and retain evidence-backed reports. |
|
|
69
|
+
| **Negative controls** | Exercise a deliberately unguarded prohibited mock action to verify that the test harness detects the expected failure. |
|
|
70
|
+
| **Supply-chain inventory** | Scan supported local manifests and configuration files, build a dependency/configuration evidence graph, and report bounded static findings. |
|
|
71
|
+
| **Cost observations** | Calculate bounded model-cost observations from explicit token usage and an operator-supplied pricing configuration. |
|
|
72
|
+
| **OpenTelemetry export** | Explicitly export completed recorded traces through an application-configured OpenTelemetry `SpanExporter`. |
|
|
73
|
+
| **Local web interface** | Explore traces and configured policy, Bench, Security, and Supply Chain reports through a read-only local API and dashboard. |
|
|
74
|
+
| **CLI 2.0** | Use a redesigned terminal experience with a workspace homepage, contextual help, structured output, clearer diagnostics, and improved run inspection. |
|
|
75
|
+
|
|
76
|
+
These capabilities share local execution records, policy evidence, reports, and explicit references. Relationships are shown only when the available evidence supports them.
|
|
77
|
+
|
|
78
|
+
## Who is it for?
|
|
79
|
+
|
|
80
|
+
- **AI agent developers** who need to inspect tool calls, execution paths, and failures.
|
|
81
|
+
- **Framework engineers** integrating consistent tracing and policy boundaries into agent applications.
|
|
82
|
+
- **AI platform teams** building applications where tool access needs explicit authorization rules.
|
|
83
|
+
- **Evaluation engineers** who need repeatable scenarios, baseline comparisons, and regression checks.
|
|
84
|
+
- **Security engineers** who want to test specific guarded execution boundaries and review the evidence.
|
|
85
|
+
- **Developer-tool builders** who want local traces, structured reports, and optional OpenTelemetry export.
|
|
86
|
+
|
|
87
|
+
Runflect is a developer toolkit, not a hosted multi-user service or a replacement for application-specific security controls.
|
|
88
|
+
|
|
89
|
+
---
|
|
90
|
+
|
|
91
|
+
## CLI 2.0
|
|
92
|
+
|
|
93
|
+
The terminal is a primary interface to Runflect, not an afterthought.
|
|
94
|
+
|
|
95
|
+
CLI 2.0 focuses on making execution evidence and evaluation results easier to discover, inspect, and use in local development and CI workflows.
|
|
96
|
+
|
|
97
|
+
### A more useful terminal experience
|
|
98
|
+
|
|
99
|
+
- **Workspace homepage:** Running `runflect` without arguments presents a compact overview of the local workspace, database status, available capabilities, and useful next commands.
|
|
100
|
+
- **Contextual help:** Consistent command descriptions, options, examples, and recovery guidance across command groups.
|
|
101
|
+
- **Execution inspection:** Readable run listings and nested span details, with bounded output and safer metadata handling.
|
|
102
|
+
- **Clear diagnostics:** More actionable errors for invalid arguments, missing data, and operational failures.
|
|
103
|
+
- **Structured output:** JSON output for supported commands, suitable for scripts and automation.
|
|
104
|
+
- **Terminal-aware rendering:** Respect for color settings and non-interactive output, with a plain-text experience for logs and CI.
|
|
105
|
+
- **Module invocation:** Support for `python -m runflect` alongside the `runflect` executable.
|
|
106
|
+
- **Version reporting:** Inspect the installed version with `runflect --version`.
|
|
107
|
+
|
|
108
|
+
### Available command groups
|
|
109
|
+
|
|
110
|
+
| Command | Purpose |
|
|
111
|
+
| ------------------------------ | --------------------------------------------------------------- |
|
|
112
|
+
| `runflect` | Open the terminal homepage. |
|
|
113
|
+
| `runflect --help` | Explore commands and global options. |
|
|
114
|
+
| `runflect list` | List recorded agent executions. |
|
|
115
|
+
| `runflect show RUN_ID` | Inspect a run and its nested spans. |
|
|
116
|
+
| `runflect policy --help` | Explore policy validation, history, evaluation, and activation. |
|
|
117
|
+
| `runflect bench --help` | Run evaluations, manage baselines, and compare reports. |
|
|
118
|
+
| `runflect security --help` | Validate and execute deterministic security scenarios. |
|
|
119
|
+
| `runflect supply-chain --help` | Inspect the local repository scanning workflow. |
|
|
120
|
+
|
|
121
|
+
The terminal homepage distinguishes a missing database from available recorded data. It does not manufacture activity or scores to make the workspace look populated.
|
|
122
|
+
|
|
123
|
+
See the [CLI 2.0 guide](docs/cli-v2.md) for command behavior, output formats, exit codes, and the PowerShell acceptance walkthrough.
|
|
124
|
+
|
|
125
|
+
---
|
|
126
|
+
|
|
127
|
+
## What makes Runflect different?
|
|
128
|
+
|
|
129
|
+
Agent observability, tool permissions, and evaluation are often treated as separate concerns. Runflect brings them together around explicit execution evidence.
|
|
130
|
+
|
|
131
|
+
### 1. Understand execution
|
|
132
|
+
|
|
133
|
+
Record application-defined runs and nested spans. Inspect timing, statuses, relationships, and safe metadata instead of reconstructing every failure from scattered logs.
|
|
134
|
+
|
|
135
|
+
### 2. Control tool access
|
|
136
|
+
|
|
137
|
+
Wrap selected tool calls with `ToolGuard`. Evaluate policy decisions, audit guarded execution, and prevent a guarded tool from running when its pre-execution audit cannot be recorded successfully.
|
|
138
|
+
|
|
139
|
+
### 3. Evaluate before activation
|
|
140
|
+
|
|
141
|
+
Validate and import policy revisions, evaluate them against deterministic scenarios, inspect the results, and activate a revision explicitly.
|
|
142
|
+
|
|
143
|
+
### 4. Test security boundaries
|
|
144
|
+
|
|
145
|
+
Run versioned offline scenarios against the actual guarded execution path. Review the evidence behind each verdict, including deliberate negative controls.
|
|
146
|
+
|
|
147
|
+
### 5. Detect evaluation regressions
|
|
148
|
+
|
|
149
|
+
Save compatible Bench baselines, compare new reports, and apply supported regression thresholds in automated workflows.
|
|
150
|
+
|
|
151
|
+
### 6. Inspect declared dependencies
|
|
152
|
+
|
|
153
|
+
Scan supported repository files and connect findings to their source locations. Preserve uncertainty when a file format is unsupported or runtime attribution is missing.
|
|
154
|
+
|
|
155
|
+
### 7. Keep evidence accessible
|
|
156
|
+
|
|
157
|
+
Use the CLI, optional local dashboard, JSON reports, and explicit OpenTelemetry export to work with recorded evidence.
|
|
158
|
+
|
|
159
|
+
**The objective is not to claim that every failure or attack can be prevented. It is to make agent behavior easier to inspect, test, and review.**
|
|
160
|
+
|
|
161
|
+
---
|
|
162
|
+
|
|
163
|
+
## How it works
|
|
164
|
+
|
|
165
|
+
The recorder observes application execution that developers choose to record. `ToolGuard` evaluates and audits policy decisions around wrapped tool calls. Policy evaluation can run offline without executing the underlying tools. Bench and security simulations use controlled local fixtures. The optional web API reads existing Runflect data without executing tools or modifying policies.
|
|
166
|
+
|
|
167
|
+
```mermaid
|
|
168
|
+
flowchart LR
|
|
169
|
+
App[Agent application] --> Recorder[Runflect recorder]
|
|
170
|
+
App --> Guard[Optional ToolGuard]
|
|
171
|
+
Guard --> Tool[Wrapped tool]
|
|
172
|
+
Guard --> Audit[Policy audit event]
|
|
173
|
+
Recorder --> DB[(Local SQLite)]
|
|
174
|
+
Audit --> DB
|
|
175
|
+
|
|
176
|
+
Policy[Policy and scenarios] --> Eval[Offline policy evaluation]
|
|
177
|
+
Eval --> DB
|
|
178
|
+
|
|
179
|
+
Bench[Bench scenarios] --> BenchReport[Versioned JSON report]
|
|
180
|
+
Security[Security simulations] --> SecurityReport[Evidence-backed report]
|
|
181
|
+
Repo[Explicit repository path] --> Scanner[Local static scanner]
|
|
182
|
+
Scanner --> ScanReport[Inventory and findings]
|
|
183
|
+
|
|
184
|
+
DB --> CLI[Runflect CLI]
|
|
185
|
+
DB --> API[Optional read-only local API]
|
|
186
|
+
BenchReport --> API
|
|
187
|
+
SecurityReport --> API
|
|
188
|
+
ScanReport --> API
|
|
189
|
+
API --> Web[Local web dashboard]
|
|
190
|
+
DB --> OTel[Explicit OpenTelemetry export]
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
Runflect is local-first. The core does not require a hosted control plane, external model provider, or network connection for its offline recording, policy, Bench, and security-fixture workflows.
|
|
194
|
+
|
|
195
|
+
---
|
|
196
|
+
|
|
197
|
+
## Quickstart
|
|
198
|
+
|
|
199
|
+
The core supports Python 3.10 or newer and has no mandatory runtime dependencies.
|
|
200
|
+
|
|
201
|
+
The package is not yet available from the public PyPI index. Install from the source checkout.
|
|
202
|
+
|
|
203
|
+
### 1. Install Runflect
|
|
204
|
+
|
|
205
|
+
From the repository root:
|
|
206
|
+
|
|
207
|
+
```powershell
|
|
208
|
+
python -m pip install -e .
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
Verify the CLI:
|
|
212
|
+
|
|
213
|
+
```powershell
|
|
214
|
+
runflect --version
|
|
215
|
+
runflect --help
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
You can also invoke it as a Python module:
|
|
219
|
+
|
|
220
|
+
```powershell
|
|
221
|
+
python -m runflect --help
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
### 2. Record your first execution
|
|
225
|
+
|
|
226
|
+
Run the bundled deterministic example:
|
|
227
|
+
|
|
228
|
+
```powershell
|
|
229
|
+
python examples/quickstart.py
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
Inspect recorded runs:
|
|
233
|
+
|
|
234
|
+
```powershell
|
|
235
|
+
runflect list --db .runflect/traces.sqlite3
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
The example prints the run ID. Inspect it with:
|
|
239
|
+
|
|
240
|
+
```powershell
|
|
241
|
+
runflect show RUN_ID --db .runflect/traces.sqlite3
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
Replace `RUN_ID` with the identifier printed by the example.
|
|
245
|
+
|
|
246
|
+
The example records two nested workflow steps and does not require a model provider or external service. It writes to `.runflect/traces.sqlite3` unless `RUNFLECT_DB` is set.
|
|
247
|
+
|
|
248
|
+
Its `compose-answer` span explicitly attributes itself to `file:examples/quickstart.py`, the scanner's stable source-file identifier when scanning this checkout. This is caller-supplied attribution, not proof that the file caused or supplied a runtime dependency.
|
|
249
|
+
|
|
250
|
+
### 3. Use the Python API directly
|
|
251
|
+
|
|
252
|
+
```python
|
|
253
|
+
from runflect import run
|
|
254
|
+
|
|
255
|
+
with run("support-answer", db_path=".runflect/traces.sqlite3") as execution:
|
|
256
|
+
with execution.span(
|
|
257
|
+
"retrieve-context",
|
|
258
|
+
input={"query": "refund policy"},
|
|
259
|
+
) as span:
|
|
260
|
+
result = "Refunds are available within 30 days."
|
|
261
|
+
span.set_output({"result": result})
|
|
262
|
+
|
|
263
|
+
print(execution.run_id)
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
The recorder stores values only when application code passes them to `input` or `set_output`.
|
|
267
|
+
|
|
268
|
+
Common sensitive mapping keys are redacted by default. Each payload is capped at 64 KiB, and oversized payloads are omitted with a safe span attribute. Error messages are omitted by default.
|
|
269
|
+
|
|
270
|
+
Review [Security and privacy](#security-and-privacy) before recording application data.
|
|
271
|
+
|
|
272
|
+
---
|
|
273
|
+
|
|
274
|
+
## LangGraph integration
|
|
275
|
+
|
|
276
|
+
Runflect provides optional synchronous and asynchronous LangGraph adapters.
|
|
277
|
+
|
|
278
|
+
Install the integration:
|
|
279
|
+
|
|
280
|
+
```powershell
|
|
281
|
+
python -m pip install -e ".[langgraph]"
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
Run the example:
|
|
285
|
+
|
|
286
|
+
```powershell
|
|
287
|
+
python examples/langgraph_quickstart.py
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
The example builds and invokes a local LangGraph with a deterministic tool fixture and prints run IDs for success and failure cases.
|
|
291
|
+
|
|
292
|
+
The adapter supports:
|
|
293
|
+
|
|
294
|
+
- `graph.invoke()`
|
|
295
|
+
- `await graph.ainvoke()`
|
|
296
|
+
- Observed graph, node, and tool callback events
|
|
297
|
+
- Parent-child relationships between supported spans
|
|
298
|
+
- Explicit model/provider labels and provider-supplied integer token usage when supported callbacks provide them
|
|
299
|
+
|
|
300
|
+
Prompts and model responses are omitted by default.
|
|
301
|
+
|
|
302
|
+
The asynchronous example is:
|
|
303
|
+
|
|
304
|
+
```powershell
|
|
305
|
+
python examples/langgraph_async_quickstart.py
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
The supported optional dependency range is `langgraph>=1.2.14,<2.0`. Consult the [architecture notes](docs/architecture.md#scope) for tested integration details and boundaries.
|
|
309
|
+
|
|
310
|
+
### Optional model-cost attribution
|
|
311
|
+
|
|
312
|
+
Reliability can calculate bounded cost observations when explicit token usage is available and you provide a versioned pricing file through `RUNFLECT_PRICING_CONFIG`.
|
|
313
|
+
|
|
314
|
+
Runflect does not invent default prices, infer missing token usage, or treat an unknown cost as zero. Partial pricing coverage remains partial. These observations are estimates, not invoice reconciliation.
|
|
315
|
+
|
|
316
|
+
See [reliability documentation](docs/reliability.md#optional-model-cost-attribution).
|
|
317
|
+
|
|
318
|
+
---
|
|
319
|
+
|
|
320
|
+
## Policies and scenario evaluation
|
|
321
|
+
|
|
322
|
+
Run the policy example:
|
|
323
|
+
|
|
324
|
+
```powershell
|
|
325
|
+
python examples/policy_quickstart.py
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
Validate a policy:
|
|
329
|
+
|
|
330
|
+
```powershell
|
|
331
|
+
runflect policy validate examples/policies/support-tools.json --json
|
|
332
|
+
```
|
|
333
|
+
|
|
334
|
+
Import an immutable policy revision:
|
|
335
|
+
|
|
336
|
+
```powershell
|
|
337
|
+
runflect policy import examples/policies/support-tools.json --db .runflect/policy.sqlite3 --json
|
|
338
|
+
```
|
|
339
|
+
|
|
340
|
+
Evaluate the revision against offline scenarios:
|
|
341
|
+
|
|
342
|
+
```powershell
|
|
343
|
+
runflect policy evaluate support-tools 1.0.0 --scenarios examples/scenarios/support-tools.json --db .runflect/policy.sqlite3 --json
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
Activate it explicitly after review:
|
|
347
|
+
|
|
348
|
+
```powershell
|
|
349
|
+
runflect policy activate support-tools 1.0.0 --db .runflect/policy.sqlite3
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
Inspect its history:
|
|
353
|
+
|
|
354
|
+
```powershell
|
|
355
|
+
runflect policy history support-tools --db .runflect/policy.sqlite3
|
|
356
|
+
```
|
|
357
|
+
|
|
358
|
+
`ToolGuard` supports allow, deny, approval-required, and configurable default-deny decisions.
|
|
359
|
+
|
|
360
|
+
**Importing or evaluating a policy does not activate it.** Activation is a separate operation. Scenario evaluation checks policy decisions without executing the underlying example tools.
|
|
361
|
+
|
|
362
|
+
The guard applies only to calls made through its wrapper. Calls made outside that boundary can bypass it.
|
|
363
|
+
|
|
364
|
+
---
|
|
365
|
+
|
|
366
|
+
## Security simulation
|
|
367
|
+
|
|
368
|
+
Run deterministic, versioned offline scenarios against the actual `ToolGuard` and recorder.
|
|
369
|
+
|
|
370
|
+
Validate a suite:
|
|
371
|
+
|
|
372
|
+
```powershell
|
|
373
|
+
runflect security validate examples/security/core-v1.json --json
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
Execute it with an explicit evidence database and report path:
|
|
377
|
+
|
|
378
|
+
```powershell
|
|
379
|
+
runflect security run --suite examples/security/core-v1.json --db .runflect/security-simulation.sqlite3 --output .runflect/security-report.json
|
|
380
|
+
```
|
|
381
|
+
|
|
382
|
+
The bundled core suite uses fixed in-process mock tools to exercise scenarios involving:
|
|
383
|
+
|
|
384
|
+
- Unauthorized tool calls
|
|
385
|
+
- Synthetic secret handling
|
|
386
|
+
- Prompt-like untrusted instructions
|
|
387
|
+
- Dangerous tool chains
|
|
388
|
+
- Harness call budgets
|
|
389
|
+
- Approval-required calls
|
|
390
|
+
|
|
391
|
+
An optional `langgraph_fake_model` scenario exercises the supported LangGraph callback, model, tool, and `ToolGuard` path without a model provider, network access, credentials, or external side effects.
|
|
392
|
+
|
|
393
|
+
A deliberate unguarded negative control can execute a prohibited mock operation. The expected result is a failing verdict with evidence that the operation executed. This tests whether the harness detects that specific unsafe outcome.
|
|
394
|
+
|
|
395
|
+
Reports distinguish `PASS`, `FAIL`, `INCONCLUSIVE`, `ERROR`, and `SKIPPED`.
|
|
396
|
+
|
|
397
|
+
**A passing fixture means only that the tested scenario produced its expected result.** It does not establish general prompt-injection resistance or comprehensive agent security.
|
|
398
|
+
|
|
399
|
+
See [Security simulation](docs/security-simulation.md) for verdict rules, report evidence, exit codes, and limitations.
|
|
400
|
+
|
|
401
|
+
---
|
|
402
|
+
|
|
403
|
+
## Runflect Bench
|
|
404
|
+
|
|
405
|
+
Run the bundled deterministic evaluation suite and write a versioned report:
|
|
406
|
+
|
|
407
|
+
```powershell
|
|
408
|
+
New-Item -ItemType Directory -Force .runflect | Out-Null
|
|
409
|
+
|
|
410
|
+
runflect bench run --output .runflect/bench-report.json
|
|
411
|
+
```
|
|
412
|
+
|
|
413
|
+
The report includes scenario outcomes, metric numerators and denominators, runtime information, and host-specific timing samples.
|
|
414
|
+
|
|
415
|
+
### Save a baseline
|
|
416
|
+
|
|
417
|
+
```powershell
|
|
418
|
+
runflect bench baseline save --report .runflect/bench-report.json --output .runflect/bench-baseline.json --label main
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
### Compare against a baseline
|
|
422
|
+
|
|
423
|
+
```powershell
|
|
424
|
+
runflect bench compare --baseline .runflect/bench-baseline.json --current .runflect/bench-report.json --output .runflect/bench-comparison.json
|
|
425
|
+
```
|
|
426
|
+
|
|
427
|
+
The comparison workflow supports compatible versioned baselines, configured thresholds, and CI-compatible exit codes.
|
|
428
|
+
|
|
429
|
+
A failed configured threshold returns a nonzero exit code. A requested timing gate without enough samples is reported as `INCONCLUSIVE`, with exit code 4.
|
|
430
|
+
|
|
431
|
+
The local Bench view reads a persisted comparison report; it does not execute benchmarks itself.
|
|
432
|
+
|
|
433
|
+
Baseline policy bypasses are an explicit comparison control, not an enforced configuration. Timing results depend on the host and workload and are not performance guarantees.
|
|
434
|
+
|
|
435
|
+
Read the [Bench report schema](docs/bench-report-v1.md) and [Bench regression guide](docs/bench-regression.md) before consuming reports programmatically.
|
|
436
|
+
|
|
437
|
+
---
|
|
438
|
+
|
|
439
|
+
## Local repository and supply-chain scanning
|
|
440
|
+
|
|
441
|
+
Run a scan against an explicit repository path:
|
|
442
|
+
|
|
443
|
+
```powershell
|
|
444
|
+
New-Item -ItemType Directory -Force .runflect | Out-Null
|
|
445
|
+
|
|
446
|
+
runflect supply-chain scan . --output .runflect/supply-chain-report.json
|
|
447
|
+
```
|
|
448
|
+
|
|
449
|
+
Fail when findings meet the configured severity threshold:
|
|
450
|
+
|
|
451
|
+
```powershell
|
|
452
|
+
runflect supply-chain scan . --fail-on high
|
|
453
|
+
```
|
|
454
|
+
|
|
455
|
+
The scanner reads supported Python and npm dependency files, selected Docker and GitHub Actions configuration, MCP server declarations, and literal Python model references.
|
|
456
|
+
|
|
457
|
+
It builds a versioned inventory/evidence graph and reports bounded static findings such as:
|
|
458
|
+
|
|
459
|
+
- Non-exact dependency declarations
|
|
460
|
+
- GitHub Actions references that are not pinned to commit SHAs
|
|
461
|
+
- Other supported configuration patterns
|
|
462
|
+
|
|
463
|
+
Reports include counts, findings, source locations when available, parse errors, unsupported formats, and coverage limitations.
|
|
464
|
+
|
|
465
|
+
The scanner does not execute repository code, download vulnerability advisories, establish package exploitability, verify artifact provenance, or prove runtime reachability.
|
|
466
|
+
|
|
467
|
+
A successful scan does not mean that the repository has no vulnerabilities. It means the scan completed according to the implemented rules and configured failure threshold.
|
|
468
|
+
|
|
469
|
+
See the [Supply Chain guide](docs/supply-chain.md) for exact coverage and limitations.
|
|
470
|
+
|
|
471
|
+
---
|
|
472
|
+
|
|
473
|
+
## OpenTelemetry export
|
|
474
|
+
|
|
475
|
+
Runflect supports explicit export of completed recorded traces through an OpenTelemetry `SpanExporter` supplied by your application.
|
|
476
|
+
|
|
477
|
+
Install the optional dependencies:
|
|
478
|
+
|
|
479
|
+
```powershell
|
|
480
|
+
python -m pip install -e ".[opentelemetry]"
|
|
481
|
+
```
|
|
482
|
+
|
|
483
|
+
Then call the export function:
|
|
484
|
+
|
|
485
|
+
```python
|
|
486
|
+
from runflect.integrations.opentelemetry import export_trace
|
|
487
|
+
|
|
488
|
+
# Supply an existing run ID and an exporter configured by your application.
|
|
489
|
+
export_trace(run_id, exporter, db_path=".runflect/traces.sqlite3")
|
|
490
|
+
```
|
|
491
|
+
|
|
492
|
+
Replace `run_id` and `exporter` with your actual run identifier and configured exporter.
|
|
493
|
+
|
|
494
|
+
Export is explicit and uses existing recorded traces. Exported telemetry leaves the local database boundary, so review your destination and data-handling requirements first.
|
|
495
|
+
|
|
496
|
+
See [OpenTelemetry export](docs/opentelemetry.md) for the mapping, privacy behavior, and failure limits.
|
|
497
|
+
|
|
498
|
+
---
|
|
499
|
+
|
|
500
|
+
## Run the local web interface
|
|
501
|
+
|
|
502
|
+
Runflect includes an optional read-only dashboard for inspecting local traces and configured reports.
|
|
503
|
+
|
|
504
|
+
The web app uses Next.js 16 and requires Node.js 20.9 or newer. The Python API is an optional `web` extra.
|
|
505
|
+
|
|
506
|
+
### 1. Install the API dependencies
|
|
507
|
+
|
|
508
|
+
From the repository root:
|
|
509
|
+
|
|
510
|
+
```powershell
|
|
511
|
+
python -m pip install -e ".[web,supply-chain]"
|
|
512
|
+
|
|
513
|
+
New-Item -ItemType Directory -Force .runflect | Out-Null
|
|
514
|
+
|
|
515
|
+
$env:RUNFLECT_DB = (Join-Path (Get-Location) ".runflect\traces.sqlite3")
|
|
516
|
+
$env:RUNFLECT_BENCH_REPORT = (Join-Path (Get-Location) ".runflect\bench-report.json")
|
|
517
|
+
$env:RUNFLECT_SUPPLY_CHAIN_REPORT = (Join-Path (Get-Location) ".runflect\supply-chain-report.json")
|
|
518
|
+
$env:RUNFLECT_SECURITY_REPORT = (Join-Path (Get-Location) ".runflect\security-report.json")
|
|
519
|
+
|
|
520
|
+
python examples/quickstart.py
|
|
521
|
+
python examples/policy_quickstart.py
|
|
522
|
+
|
|
523
|
+
runflect bench run --output .runflect/bench-report.json
|
|
524
|
+
runflect supply-chain scan . --output .runflect/supply-chain-report.json
|
|
525
|
+
|
|
526
|
+
runflect security run --suite examples/security/core-v1.json --db $env:RUNFLECT_DB --output .runflect/security-report.json
|
|
527
|
+
|
|
528
|
+
runflect-web-api
|
|
529
|
+
```
|
|
530
|
+
|
|
531
|
+
### 2. Start the frontend
|
|
532
|
+
|
|
533
|
+
In a second terminal:
|
|
534
|
+
|
|
535
|
+
```powershell
|
|
536
|
+
Set-Location web
|
|
537
|
+
|
|
538
|
+
npm ci
|
|
539
|
+
|
|
540
|
+
$env:NEXT_PUBLIC_RUNFLECT_API_URL = "http://127.0.0.1:8765"
|
|
541
|
+
|
|
542
|
+
npm run dev -- --hostname 127.0.0.1
|
|
543
|
+
```
|
|
544
|
+
|
|
545
|
+
Open:
|
|
546
|
+
|
|
547
|
+
- Landing page: `http://127.0.0.1:3000`
|
|
548
|
+
- Dashboard: `http://127.0.0.1:3000/dashboard`
|
|
549
|
+
|
|
550
|
+
The setup commands generate local data and reports before starting the API.
|
|
551
|
+
|
|
552
|
+
The dashboard includes:
|
|
553
|
+
|
|
554
|
+
- Run overview and trace inspection
|
|
555
|
+
- Policy history and decisions
|
|
556
|
+
- Bench results and persisted comparisons
|
|
557
|
+
- Security simulation reports
|
|
558
|
+
- Supply-chain inventory and findings
|
|
559
|
+
- Evidence-linked navigation where explicit references exist
|
|
560
|
+
|
|
561
|
+
The API binds to loopback, allows only configured localhost browser origins, serves GET requests only, and does not execute tools or modify policies.
|
|
562
|
+
|
|
563
|
+
Supply Chain links to spans only when the available evidence contains the exact matching unique component ID. That relationship represents explicit caller attribution, not proven dependency use or causality.
|
|
564
|
+
|
|
565
|
+
The local API is a read-only convenience, not a hosted service or complete security boundary.
|
|
566
|
+
|
|
567
|
+
See the [web guide](docs/web.md), [Supply Chain guide](docs/supply-chain.md), and [Security simulation guide](docs/security-simulation.md) for further details.
|
|
568
|
+
|
|
569
|
+
### Optional LangGraph traces in the dashboard
|
|
570
|
+
|
|
571
|
+
Install the LangGraph extra as well:
|
|
572
|
+
|
|
573
|
+
```powershell
|
|
574
|
+
python -m pip install -e ".[langgraph,web,supply-chain]"
|
|
575
|
+
```
|
|
576
|
+
|
|
577
|
+
With `RUNFLECT_DB` pointing to the same local database, run:
|
|
578
|
+
|
|
579
|
+
```powershell
|
|
580
|
+
python examples/langgraph_quickstart.py
|
|
581
|
+
```
|
|
582
|
+
|
|
583
|
+
The example records one successful and one deliberately failed LangGraph execution using a fixed tool and deterministic fake chat model. It does not contact a model provider or external network.
|
|
584
|
+
|
|
585
|
+
---
|
|
586
|
+
|
|
587
|
+
## Example developer workflow
|
|
588
|
+
|
|
589
|
+
A typical local workflow looks like this:
|
|
590
|
+
|
|
591
|
+
1. **Record:** Run `python examples/quickstart.py` and inspect executions using `runflect list` and `runflect show`.
|
|
592
|
+
2. **Protect:** Wrap sensitive tool calls with `ToolGuard`, using `examples/policy_quickstart.py` as a starting point.
|
|
593
|
+
3. **Evaluate:** Import and evaluate a policy revision against offline scenarios, then review it before explicit activation.
|
|
594
|
+
4. **Benchmark:** Run `runflect bench run` and preserve the versioned report.
|
|
595
|
+
5. **Compare:** Save a baseline and compare subsequent reports to detect supported regressions.
|
|
596
|
+
6. **Simulate:** Run `runflect security run` and inspect scenario verdicts and recorded mock traces.
|
|
597
|
+
7. **Inspect dependencies:** Run `runflect supply-chain scan .` to review supported repository evidence.
|
|
598
|
+
8. **Investigate:** Follow exact evidence references across the local CLI and dashboard when those relationships are available.
|
|
599
|
+
9. **Export:** Optionally export a completed trace through your configured OpenTelemetry exporter.
|
|
600
|
+
|
|
601
|
+
The purpose is to create a repeatable loop:
|
|
602
|
+
|
|
603
|
+
**Record → Protect → Evaluate → Inspect → Improve**
|
|
604
|
+
|
|
605
|
+
This is an evidence-driven developer workflow, not an automatic root-cause engine or a guarantee that every agent failure can be reproduced.
|
|
606
|
+
|
|
607
|
+
---
|
|
608
|
+
|
|
609
|
+
## Security and privacy
|
|
610
|
+
|
|
611
|
+
Runflect is designed for local inspection, but local-first does not mean risk-free.
|
|
612
|
+
|
|
613
|
+
- **Explicit capture:** The core recorder does not automatically inspect agent state. It stores values explicitly passed to span inputs and outputs.
|
|
614
|
+
- **Default redaction:** Common sensitive mapping keys are recursively replaced with `[REDACTED]` by default. Setting `redact=False` disables this protection.
|
|
615
|
+
- **Payload limits:** Recorded payloads are bounded. Oversized payloads are omitted with a safe span attribute.
|
|
616
|
+
- **Error messages:** Error messages are omitted by default. `capture_error_messages=True` opts into storing messages up to 4,096 characters after key-based redaction.
|
|
617
|
+
- **LangGraph privacy:** Input/output capture is off by default and can be opted into using `capture_inputs=True` and/or `capture_outputs=True`. Exception messages are omitted by the adapter.
|
|
618
|
+
- **Guard boundaries:** `ToolGuard` protects calls made through its wrapper. Direct calls outside that boundary can bypass it.
|
|
619
|
+
- **Audit behavior:** A pre-execution policy audit failure blocks a guarded tool. A later audit or trace write failure cannot undo an external side effect.
|
|
620
|
+
- **Local storage:** SQLite is not encryption or a tamper-proof ledger. Protect the database with host filesystem controls.
|
|
621
|
+
- **Approvals:** The approval authority is in-process, not a distributed approval service.
|
|
622
|
+
- **Web interface:** The API is a local read-only convenience with loopback and Host/CORS checks, not a complete security boundary.
|
|
623
|
+
- **Scanner scope:** Repository scanning is bounded static analysis. It does not execute code or download vulnerability advisories.
|
|
624
|
+
- **OpenTelemetry:** Export is explicit. Exported telemetry leaves the local database boundary and should be handled accordingly.
|
|
625
|
+
|
|
626
|
+
Runflect is not a sandbox, does not prevent arbitrary code from bypassing wrappers, and does not guarantee secure agent behavior.
|
|
627
|
+
|
|
628
|
+
Read the [threat model](docs/threat-model.md), [architecture and guarantee boundaries](docs/architecture.md#boundaries-and-failure-behavior), and [release checklist](docs/release-checklist.md#guarantee-boundaries).
|
|
629
|
+
|
|
630
|
+
---
|
|
631
|
+
|
|
632
|
+
## Current status and limitations
|
|
633
|
+
|
|
634
|
+
Runflect is an early-stage developer preview. Its implemented capabilities include local Python execution recording and CLI inspection, synchronous and asynchronous LangGraph callbacks, guarded tool policies and audit records, immutable policy history, offline scenario evaluation, deterministic Bench and regression comparisons, security simulation fixtures, explicit OpenTelemetry export, an optional local read-only web interface, and bounded static repository scanning.
|
|
635
|
+
|
|
636
|
+
The CLI 2.0 redesign adds a terminal homepage, contextual help, module invocation, structured output options, clearer diagnostics, and improved run inspection.
|
|
637
|
+
|
|
638
|
+
The following limitations remain important:
|
|
639
|
+
|
|
640
|
+
- **Model instrumentation:** Provider/model callback spans and token counts are available only when supported callbacks supply the necessary metadata. Streaming-token capture and cross-process checkpoint resume are unsupported.
|
|
641
|
+
- **Cost attribution:** Cost observations require a user-supplied versioned rate file and complete explicit input/output token usage. They are partial estimates, not invoice reconciliation.
|
|
642
|
+
- **Policy controls:** Policy argument constraints, distributed approvals, and durable external approval workflows are unavailable.
|
|
643
|
+
- **Security coverage:** The offline fixtures exercise specific deterministic boundaries. The optional LangGraph fake-model scenario is not a real-provider attack evaluation and does not establish general prompt-injection resistance.
|
|
644
|
+
- **Runtime enforcement:** Calls outside a guarded wrapper are not protected by that wrapper.
|
|
645
|
+
- **Investigation:** Evidence-linked navigation exists where exact references are available. A general causal graph, automatic root-cause analysis, and general-purpose deterministic replay are not provided.
|
|
646
|
+
- **Supply chain:** The scanner does not provide complete SBOM coverage, known-vulnerability advisory matching, model provenance verification, or proven runtime dependency reachability.
|
|
647
|
+
- **Hosting:** Remote ingestion, multi-user hosting, encryption, retention management, and tamper-proof audit storage are unavailable.
|
|
648
|
+
- **Benchmarking:** Functional scenarios are deterministic within their fixtures. Timing measurements depend on the host and workload and are not performance guarantees.
|
|
649
|
+
- **Distribution:** The package has not yet been published to public PyPI. Source-checkout installation is the supported installation path for now.
|
|
650
|
+
|
|
651
|
+
These are known product boundaries, not capabilities to rely on today.
|
|
652
|
+
|
|
653
|
+
---
|
|
654
|
+
|
|
655
|
+
## Development and testing
|
|
656
|
+
|
|
657
|
+
Install the optional integrations used by the full test suite:
|
|
658
|
+
|
|
659
|
+
```powershell
|
|
660
|
+
python -m pip install -e ".[langgraph,opentelemetry]"
|
|
661
|
+
```
|
|
662
|
+
|
|
663
|
+
Run the Python tests:
|
|
664
|
+
|
|
665
|
+
```powershell
|
|
666
|
+
python -m unittest discover -s tests -v
|
|
667
|
+
```
|
|
668
|
+
|
|
669
|
+
Compile the Python source and examples:
|
|
670
|
+
|
|
671
|
+
```powershell
|
|
672
|
+
python -m compileall -q src tests examples
|
|
673
|
+
```
|
|
674
|
+
|
|
675
|
+
Run the release smoke test:
|
|
676
|
+
|
|
677
|
+
```powershell
|
|
678
|
+
python examples/release_smoke.py
|
|
679
|
+
```
|
|
680
|
+
|
|
681
|
+
The release smoke test exercises the deterministic Bench suite, security scenarios, policy/audit behavior, and trace export.
|
|
682
|
+
|
|
683
|
+
### Frontend checks
|
|
684
|
+
|
|
685
|
+
From the `web/` directory:
|
|
686
|
+
|
|
687
|
+
```powershell
|
|
688
|
+
npm ci
|
|
689
|
+
npm run typecheck
|
|
690
|
+
npm run build
|
|
691
|
+
```
|
|
692
|
+
|
|
693
|
+
Run browser tests if the repository's Playwright setup is installed and configured:
|
|
694
|
+
|
|
695
|
+
```powershell
|
|
696
|
+
npm run test:e2e
|
|
697
|
+
```
|
|
698
|
+
|
|
699
|
+
### Useful project documentation
|
|
700
|
+
|
|
701
|
+
- [CLI design, commands, exit codes, and PowerShell acceptance](docs/cli-v2.md)
|
|
702
|
+
- [Architecture and guarantee boundaries](docs/architecture.md)
|
|
703
|
+
- [Local threat model and tested boundaries](docs/threat-model.md)
|
|
704
|
+
- [Windows PowerShell acceptance walkthrough](docs/manual-acceptance-powershell.md)
|
|
705
|
+
- [Bench report schema](docs/bench-report-v1.md)
|
|
706
|
+
- [Bench baselines and regression comparisons](docs/bench-regression.md)
|
|
707
|
+
- [Custom offline evaluation suite](examples/scenarios/custom-evaluation-v1.json)
|
|
708
|
+
- [Local reliability observations and cost attribution](docs/reliability.md)
|
|
709
|
+
- [OpenTelemetry export](docs/opentelemetry.md)
|
|
710
|
+
- [Local repository scanning](docs/supply-chain.md)
|
|
711
|
+
- [Local web interface](docs/web.md)
|
|
712
|
+
- [Release readiness checklist](docs/release-checklist.md)
|
|
713
|
+
|
|
714
|
+
---
|
|
715
|
+
|
|
716
|
+
## Contributing and license
|
|
717
|
+
|
|
718
|
+
Runflect is under active development. Please review the repository's current contribution guidance and license before contributing, reusing, or distributing the code.
|
|
719
|
+
|
|
720
|
+
The repository checkout used to prepare this README did not contain a `LICENSE` file or a contribution guide. Add the intended license and contribution instructions before making public release claims.
|