runflect 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (46) hide show
  1. runflect-0.1.0/PKG-INFO +720 -0
  2. runflect-0.1.0/README.md +688 -0
  3. runflect-0.1.0/pyproject.toml +46 -0
  4. runflect-0.1.0/setup.cfg +4 -0
  5. runflect-0.1.0/src/runflect/__init__.py +26 -0
  6. runflect-0.1.0/src/runflect/__main__.py +5 -0
  7. runflect-0.1.0/src/runflect/bench/__init__.py +4 -0
  8. runflect-0.1.0/src/runflect/bench/fixtures.py +33 -0
  9. runflect-0.1.0/src/runflect/bench/metrics.py +91 -0
  10. runflect-0.1.0/src/runflect/bench/regression.py +320 -0
  11. runflect-0.1.0/src/runflect/bench/report.py +76 -0
  12. runflect-0.1.0/src/runflect/bench/runner.py +549 -0
  13. runflect-0.1.0/src/runflect/bench/schema.py +244 -0
  14. runflect-0.1.0/src/runflect/bench/suites/v0_1.json +106 -0
  15. runflect-0.1.0/src/runflect/cli.py +862 -0
  16. runflect-0.1.0/src/runflect/integrations/__init__.py +1 -0
  17. runflect-0.1.0/src/runflect/integrations/langgraph.py +453 -0
  18. runflect-0.1.0/src/runflect/integrations/opentelemetry.py +408 -0
  19. runflect-0.1.0/src/runflect/policy.py +553 -0
  20. runflect-0.1.0/src/runflect/policy_files.py +402 -0
  21. runflect-0.1.0/src/runflect/pricing.py +139 -0
  22. runflect-0.1.0/src/runflect/recorder.py +433 -0
  23. runflect-0.1.0/src/runflect/security_simulation.py +714 -0
  24. runflect-0.1.0/src/runflect/storage.py +220 -0
  25. runflect-0.1.0/src/runflect/supply_chain.py +875 -0
  26. runflect-0.1.0/src/runflect/terminal.py +85 -0
  27. runflect-0.1.0/src/runflect/web_api.py +1536 -0
  28. runflect-0.1.0/src/runflect/web_cli.py +9 -0
  29. runflect-0.1.0/src/runflect.egg-info/PKG-INFO +720 -0
  30. runflect-0.1.0/src/runflect.egg-info/SOURCES.txt +44 -0
  31. runflect-0.1.0/src/runflect.egg-info/dependency_links.txt +1 -0
  32. runflect-0.1.0/src/runflect.egg-info/entry_points.txt +3 -0
  33. runflect-0.1.0/src/runflect.egg-info/requires.txt +16 -0
  34. runflect-0.1.0/src/runflect.egg-info/top_level.txt +1 -0
  35. runflect-0.1.0/tests/test_bench.py +203 -0
  36. runflect-0.1.0/tests/test_bench_regression.py +134 -0
  37. runflect-0.1.0/tests/test_cli_ux.py +198 -0
  38. runflect-0.1.0/tests/test_langgraph.py +608 -0
  39. runflect-0.1.0/tests/test_opentelemetry.py +271 -0
  40. runflect-0.1.0/tests/test_policy.py +349 -0
  41. runflect-0.1.0/tests/test_policy_files.py +191 -0
  42. runflect-0.1.0/tests/test_pricing.py +81 -0
  43. runflect-0.1.0/tests/test_recorder.py +284 -0
  44. runflect-0.1.0/tests/test_security_simulation.py +315 -0
  45. runflect-0.1.0/tests/test_supply_chain.py +167 -0
  46. runflect-0.1.0/tests/test_web_api.py +628 -0
@@ -0,0 +1,720 @@
1
+ Metadata-Version: 2.4
2
+ Name: runflect
3
+ Version: 0.1.0
4
+ Summary: Local-first evidence capture for AI agent executions
5
+ Project-URL: Homepage, https://github.com/bevinkatti/Runflect
6
+ Project-URL: Repository, https://github.com/bevinkatti/Runflect
7
+ Project-URL: Documentation, https://github.com/bevinkatti/Runflect#readme
8
+ Project-URL: Issues, https://github.com/bevinkatti/Runflect/issues
9
+ Classifier: Development Status :: 3 - Alpha
10
+ Classifier: Environment :: Console
11
+ Classifier: Intended Audience :: Developers
12
+ Classifier: Operating System :: OS Independent
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.10
15
+ Classifier: Programming Language :: Python :: 3.11
16
+ Classifier: Programming Language :: Python :: 3.12
17
+ Classifier: Programming Language :: Python :: 3.13
18
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
19
+ Classifier: Topic :: System :: Monitoring
20
+ Requires-Python: >=3.10
21
+ Description-Content-Type: text/markdown
22
+ Provides-Extra: langgraph
23
+ Requires-Dist: langgraph<2.0,>=1.2.14; extra == "langgraph"
24
+ Provides-Extra: opentelemetry
25
+ Requires-Dist: opentelemetry-api<2.0,>=1.27; extra == "opentelemetry"
26
+ Requires-Dist: opentelemetry-sdk<2.0,>=1.27; extra == "opentelemetry"
27
+ Provides-Extra: web
28
+ Requires-Dist: fastapi<1.0,>=0.115; extra == "web"
29
+ Requires-Dist: uvicorn<1.0,>=0.30; extra == "web"
30
+ Provides-Extra: supply-chain
31
+ Requires-Dist: tomli<3,>=2; python_version < "3.11" and extra == "supply-chain"
32
+
33
+ # Runflect
34
+
35
+ ### The reliability and security toolkit for AI agents.
36
+
37
+ Trace executions. Enforce tool-use policies. Benchmark agent behavior. Investigate failures with inspectable evidence.
38
+
39
+ An agent calls a tool it shouldn't. A workflow fails halfway through. A policy change introduces a regression you didn't anticipate.
40
+
41
+ When something goes wrong, scattered logs and inconsistent traces make it difficult to understand what happened, identify the failure, and verify whether a fix actually works.
42
+
43
+ **Runflect turns agent execution into actionable evidence.**
44
+
45
+ Record execution traces, control tool access through explicit policies, evaluate security boundaries with repeatable scenarios, and catch regressions through benchmark comparisons—all in one local-first toolkit for developers building AI agents.
46
+ **Observe execution. Enforce boundaries. Test before you trust.**
47
+
48
+ [Quickstart](#quickstart) · [Features](#what-runflect-provides) · [CLI](#cli-20) · [Architecture](docs/architecture.md) · [Local dashboard](docs/web.md)
49
+
50
+ > **Project status:** Runflect is an early-stage developer preview. The distribution name is `runflect`, but public PyPI installation is not yet available. Install from a source checkout using the instructions below.
51
+
52
+ > **Important:** Runflect provides observability and policy enforcement at instrumented boundaries. It does not reveal hidden model reasoning, guarantee agent security, or automatically explain every failure.
53
+
54
+ ---
55
+
56
+ ## What Runflect provides
57
+
58
+ Runflect brings agent execution observability, tool-use controls, evaluation, and local security evidence into one developer workflow.
59
+
60
+ | Capability | What it does |
61
+ | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
62
+ | **Execution tracing** | Records runs, nested spans, timing, status, and bounded typed attributes in a versioned SQLite database. |
63
+ | **LangGraph integration** | Optional synchronous and asynchronous adapters record observed graph, node, tool, and supported model callback events. |
64
+ | **Tool-use policies** | `ToolGuard` supports allow, deny, approval-required, and configurable default-deny decisions around wrapped tool calls. |
65
+ | **Policy lifecycle** | Validate policies, import immutable revisions, review history, evaluate scenarios offline, and explicitly activate revisions. |
66
+ | **Runflect Bench** | Run deterministic evaluation scenarios and inspect outcomes, metrics, and host-specific timing observations. |
67
+ | **Regression testing** | Save versioned baselines, compare compatible reports, apply supported thresholds, and use CI-compatible exit codes. |
68
+ | **Security simulation** | Execute deterministic offline mock scenarios against the actual guarded execution path and retain evidence-backed reports. |
69
+ | **Negative controls** | Exercise a deliberately unguarded prohibited mock action to verify that the test harness detects the expected failure. |
70
+ | **Supply-chain inventory** | Scan supported local manifests and configuration files, build a dependency/configuration evidence graph, and report bounded static findings. |
71
+ | **Cost observations** | Calculate bounded model-cost observations from explicit token usage and an operator-supplied pricing configuration. |
72
+ | **OpenTelemetry export** | Explicitly export completed recorded traces through an application-configured OpenTelemetry `SpanExporter`. |
73
+ | **Local web interface** | Explore traces and configured policy, Bench, Security, and Supply Chain reports through a read-only local API and dashboard. |
74
+ | **CLI 2.0** | Use a redesigned terminal experience with a workspace homepage, contextual help, structured output, clearer diagnostics, and improved run inspection. |
75
+
76
+ These capabilities share local execution records, policy evidence, reports, and explicit references. Relationships are shown only when the available evidence supports them.
77
+
78
+ ## Who is it for?
79
+
80
+ - **AI agent developers** who need to inspect tool calls, execution paths, and failures.
81
+ - **Framework engineers** integrating consistent tracing and policy boundaries into agent applications.
82
+ - **AI platform teams** building applications where tool access needs explicit authorization rules.
83
+ - **Evaluation engineers** who need repeatable scenarios, baseline comparisons, and regression checks.
84
+ - **Security engineers** who want to test specific guarded execution boundaries and review the evidence.
85
+ - **Developer-tool builders** who want local traces, structured reports, and optional OpenTelemetry export.
86
+
87
+ Runflect is a developer toolkit, not a hosted multi-user service or a replacement for application-specific security controls.
88
+
89
+ ---
90
+
91
+ ## CLI 2.0
92
+
93
+ The terminal is a primary interface to Runflect, not an afterthought.
94
+
95
+ CLI 2.0 focuses on making execution evidence and evaluation results easier to discover, inspect, and use in local development and CI workflows.
96
+
97
+ ### A more useful terminal experience
98
+
99
+ - **Workspace homepage:** Running `runflect` without arguments presents a compact overview of the local workspace, database status, available capabilities, and useful next commands.
100
+ - **Contextual help:** Consistent command descriptions, options, examples, and recovery guidance across command groups.
101
+ - **Execution inspection:** Readable run listings and nested span details, with bounded output and safer metadata handling.
102
+ - **Clear diagnostics:** More actionable errors for invalid arguments, missing data, and operational failures.
103
+ - **Structured output:** JSON output for supported commands, suitable for scripts and automation.
104
+ - **Terminal-aware rendering:** Respect for color settings and non-interactive output, with a plain-text experience for logs and CI.
105
+ - **Module invocation:** Support for `python -m runflect` alongside the `runflect` executable.
106
+ - **Version reporting:** Inspect the installed version with `runflect --version`.
107
+
108
+ ### Available command groups
109
+
110
+ | Command | Purpose |
111
+ | ------------------------------ | --------------------------------------------------------------- |
112
+ | `runflect` | Open the terminal homepage. |
113
+ | `runflect --help` | Explore commands and global options. |
114
+ | `runflect list` | List recorded agent executions. |
115
+ | `runflect show RUN_ID` | Inspect a run and its nested spans. |
116
+ | `runflect policy --help` | Explore policy validation, history, evaluation, and activation. |
117
+ | `runflect bench --help` | Run evaluations, manage baselines, and compare reports. |
118
+ | `runflect security --help` | Validate and execute deterministic security scenarios. |
119
+ | `runflect supply-chain --help` | Inspect the local repository scanning workflow. |
120
+
121
+ The terminal homepage distinguishes a missing database from available recorded data. It does not manufacture activity or scores to make the workspace look populated.
122
+
123
+ See the [CLI 2.0 guide](docs/cli-v2.md) for command behavior, output formats, exit codes, and the PowerShell acceptance walkthrough.
124
+
125
+ ---
126
+
127
+ ## What makes Runflect different?
128
+
129
+ Agent observability, tool permissions, and evaluation are often treated as separate concerns. Runflect brings them together around explicit execution evidence.
130
+
131
+ ### 1. Understand execution
132
+
133
+ Record application-defined runs and nested spans. Inspect timing, statuses, relationships, and safe metadata instead of reconstructing every failure from scattered logs.
134
+
135
+ ### 2. Control tool access
136
+
137
+ Wrap selected tool calls with `ToolGuard`. Evaluate policy decisions, audit guarded execution, and prevent a guarded tool from running when its pre-execution audit cannot be recorded successfully.
138
+
139
+ ### 3. Evaluate before activation
140
+
141
+ Validate and import policy revisions, evaluate them against deterministic scenarios, inspect the results, and activate a revision explicitly.
142
+
143
+ ### 4. Test security boundaries
144
+
145
+ Run versioned offline scenarios against the actual guarded execution path. Review the evidence behind each verdict, including deliberate negative controls.
146
+
147
+ ### 5. Detect evaluation regressions
148
+
149
+ Save compatible Bench baselines, compare new reports, and apply supported regression thresholds in automated workflows.
150
+
151
+ ### 6. Inspect declared dependencies
152
+
153
+ Scan supported repository files and connect findings to their source locations. Preserve uncertainty when a file format is unsupported or runtime attribution is missing.
154
+
155
+ ### 7. Keep evidence accessible
156
+
157
+ Use the CLI, optional local dashboard, JSON reports, and explicit OpenTelemetry export to work with recorded evidence.
158
+
159
+ **The objective is not to claim that every failure or attack can be prevented. It is to make agent behavior easier to inspect, test, and review.**
160
+
161
+ ---
162
+
163
+ ## How it works
164
+
165
+ The recorder observes application execution that developers choose to record. `ToolGuard` evaluates and audits policy decisions around wrapped tool calls. Policy evaluation can run offline without executing the underlying tools. Bench and security simulations use controlled local fixtures. The optional web API reads existing Runflect data without executing tools or modifying policies.
166
+
167
+ ```mermaid
168
+ flowchart LR
169
+ App[Agent application] --> Recorder[Runflect recorder]
170
+ App --> Guard[Optional ToolGuard]
171
+ Guard --> Tool[Wrapped tool]
172
+ Guard --> Audit[Policy audit event]
173
+ Recorder --> DB[(Local SQLite)]
174
+ Audit --> DB
175
+
176
+ Policy[Policy and scenarios] --> Eval[Offline policy evaluation]
177
+ Eval --> DB
178
+
179
+ Bench[Bench scenarios] --> BenchReport[Versioned JSON report]
180
+ Security[Security simulations] --> SecurityReport[Evidence-backed report]
181
+ Repo[Explicit repository path] --> Scanner[Local static scanner]
182
+ Scanner --> ScanReport[Inventory and findings]
183
+
184
+ DB --> CLI[Runflect CLI]
185
+ DB --> API[Optional read-only local API]
186
+ BenchReport --> API
187
+ SecurityReport --> API
188
+ ScanReport --> API
189
+ API --> Web[Local web dashboard]
190
+ DB --> OTel[Explicit OpenTelemetry export]
191
+ ```
192
+
193
+ Runflect is local-first. The core does not require a hosted control plane, external model provider, or network connection for its offline recording, policy, Bench, and security-fixture workflows.
194
+
195
+ ---
196
+
197
+ ## Quickstart
198
+
199
+ The core supports Python 3.10 or newer and has no mandatory runtime dependencies.
200
+
201
+ The package is not yet available from the public PyPI index. Install from the source checkout.
202
+
203
+ ### 1. Install Runflect
204
+
205
+ From the repository root:
206
+
207
+ ```powershell
208
+ python -m pip install -e .
209
+ ```
210
+
211
+ Verify the CLI:
212
+
213
+ ```powershell
214
+ runflect --version
215
+ runflect --help
216
+ ```
217
+
218
+ You can also invoke it as a Python module:
219
+
220
+ ```powershell
221
+ python -m runflect --help
222
+ ```
223
+
224
+ ### 2. Record your first execution
225
+
226
+ Run the bundled deterministic example:
227
+
228
+ ```powershell
229
+ python examples/quickstart.py
230
+ ```
231
+
232
+ Inspect recorded runs:
233
+
234
+ ```powershell
235
+ runflect list --db .runflect/traces.sqlite3
236
+ ```
237
+
238
+ The example prints the run ID. Inspect it with:
239
+
240
+ ```powershell
241
+ runflect show RUN_ID --db .runflect/traces.sqlite3
242
+ ```
243
+
244
+ Replace `RUN_ID` with the identifier printed by the example.
245
+
246
+ The example records two nested workflow steps and does not require a model provider or external service. It writes to `.runflect/traces.sqlite3` unless `RUNFLECT_DB` is set.
247
+
248
+ Its `compose-answer` span explicitly attributes itself to `file:examples/quickstart.py`, the scanner's stable source-file identifier when scanning this checkout. This is caller-supplied attribution, not proof that the file caused or supplied a runtime dependency.
249
+
250
+ ### 3. Use the Python API directly
251
+
252
+ ```python
253
+ from runflect import run
254
+
255
+ with run("support-answer", db_path=".runflect/traces.sqlite3") as execution:
256
+ with execution.span(
257
+ "retrieve-context",
258
+ input={"query": "refund policy"},
259
+ ) as span:
260
+ result = "Refunds are available within 30 days."
261
+ span.set_output({"result": result})
262
+
263
+ print(execution.run_id)
264
+ ```
265
+
266
+ The recorder stores values only when application code passes them to `input` or `set_output`.
267
+
268
+ Common sensitive mapping keys are redacted by default. Each payload is capped at 64 KiB, and oversized payloads are omitted with a safe span attribute. Error messages are omitted by default.
269
+
270
+ Review [Security and privacy](#security-and-privacy) before recording application data.
271
+
272
+ ---
273
+
274
+ ## LangGraph integration
275
+
276
+ Runflect provides optional synchronous and asynchronous LangGraph adapters.
277
+
278
+ Install the integration:
279
+
280
+ ```powershell
281
+ python -m pip install -e ".[langgraph]"
282
+ ```
283
+
284
+ Run the example:
285
+
286
+ ```powershell
287
+ python examples/langgraph_quickstart.py
288
+ ```
289
+
290
+ The example builds and invokes a local LangGraph with a deterministic tool fixture and prints run IDs for success and failure cases.
291
+
292
+ The adapter supports:
293
+
294
+ - `graph.invoke()`
295
+ - `await graph.ainvoke()`
296
+ - Observed graph, node, and tool callback events
297
+ - Parent-child relationships between supported spans
298
+ - Explicit model/provider labels and provider-supplied integer token usage when supported callbacks provide them
299
+
300
+ Prompts and model responses are omitted by default.
301
+
302
+ The asynchronous example is:
303
+
304
+ ```powershell
305
+ python examples/langgraph_async_quickstart.py
306
+ ```
307
+
308
+ The supported optional dependency range is `langgraph>=1.2.14,<2.0`. Consult the [architecture notes](docs/architecture.md#scope) for tested integration details and boundaries.
309
+
310
+ ### Optional model-cost attribution
311
+
312
+ Reliability can calculate bounded cost observations when explicit token usage is available and you provide a versioned pricing file through `RUNFLECT_PRICING_CONFIG`.
313
+
314
+ Runflect does not invent default prices, infer missing token usage, or treat an unknown cost as zero. Partial pricing coverage remains partial. These observations are estimates, not invoice reconciliation.
315
+
316
+ See [reliability documentation](docs/reliability.md#optional-model-cost-attribution).
317
+
318
+ ---
319
+
320
+ ## Policies and scenario evaluation
321
+
322
+ Run the policy example:
323
+
324
+ ```powershell
325
+ python examples/policy_quickstart.py
326
+ ```
327
+
328
+ Validate a policy:
329
+
330
+ ```powershell
331
+ runflect policy validate examples/policies/support-tools.json --json
332
+ ```
333
+
334
+ Import an immutable policy revision:
335
+
336
+ ```powershell
337
+ runflect policy import examples/policies/support-tools.json --db .runflect/policy.sqlite3 --json
338
+ ```
339
+
340
+ Evaluate the revision against offline scenarios:
341
+
342
+ ```powershell
343
+ runflect policy evaluate support-tools 1.0.0 --scenarios examples/scenarios/support-tools.json --db .runflect/policy.sqlite3 --json
344
+ ```
345
+
346
+ Activate it explicitly after review:
347
+
348
+ ```powershell
349
+ runflect policy activate support-tools 1.0.0 --db .runflect/policy.sqlite3
350
+ ```
351
+
352
+ Inspect its history:
353
+
354
+ ```powershell
355
+ runflect policy history support-tools --db .runflect/policy.sqlite3
356
+ ```
357
+
358
+ `ToolGuard` supports allow, deny, approval-required, and configurable default-deny decisions.
359
+
360
+ **Importing or evaluating a policy does not activate it.** Activation is a separate operation. Scenario evaluation checks policy decisions without executing the underlying example tools.
361
+
362
+ The guard applies only to calls made through its wrapper. Calls made outside that boundary can bypass it.
363
+
364
+ ---
365
+
366
+ ## Security simulation
367
+
368
+ Run deterministic, versioned offline scenarios against the actual `ToolGuard` and recorder.
369
+
370
+ Validate a suite:
371
+
372
+ ```powershell
373
+ runflect security validate examples/security/core-v1.json --json
374
+ ```
375
+
376
+ Execute it with an explicit evidence database and report path:
377
+
378
+ ```powershell
379
+ runflect security run --suite examples/security/core-v1.json --db .runflect/security-simulation.sqlite3 --output .runflect/security-report.json
380
+ ```
381
+
382
+ The bundled core suite uses fixed in-process mock tools to exercise scenarios involving:
383
+
384
+ - Unauthorized tool calls
385
+ - Synthetic secret handling
386
+ - Prompt-like untrusted instructions
387
+ - Dangerous tool chains
388
+ - Harness call budgets
389
+ - Approval-required calls
390
+
391
+ An optional `langgraph_fake_model` scenario exercises the supported LangGraph callback, model, tool, and `ToolGuard` path without a model provider, network access, credentials, or external side effects.
392
+
393
+ A deliberate unguarded negative control can execute a prohibited mock operation. The expected result is a failing verdict with evidence that the operation executed. This tests whether the harness detects that specific unsafe outcome.
394
+
395
+ Reports distinguish `PASS`, `FAIL`, `INCONCLUSIVE`, `ERROR`, and `SKIPPED`.
396
+
397
+ **A passing fixture means only that the tested scenario produced its expected result.** It does not establish general prompt-injection resistance or comprehensive agent security.
398
+
399
+ See [Security simulation](docs/security-simulation.md) for verdict rules, report evidence, exit codes, and limitations.
400
+
401
+ ---
402
+
403
+ ## Runflect Bench
404
+
405
+ Run the bundled deterministic evaluation suite and write a versioned report:
406
+
407
+ ```powershell
408
+ New-Item -ItemType Directory -Force .runflect | Out-Null
409
+
410
+ runflect bench run --output .runflect/bench-report.json
411
+ ```
412
+
413
+ The report includes scenario outcomes, metric numerators and denominators, runtime information, and host-specific timing samples.
414
+
415
+ ### Save a baseline
416
+
417
+ ```powershell
418
+ runflect bench baseline save --report .runflect/bench-report.json --output .runflect/bench-baseline.json --label main
419
+ ```
420
+
421
+ ### Compare against a baseline
422
+
423
+ ```powershell
424
+ runflect bench compare --baseline .runflect/bench-baseline.json --current .runflect/bench-report.json --output .runflect/bench-comparison.json
425
+ ```
426
+
427
+ The comparison workflow supports compatible versioned baselines, configured thresholds, and CI-compatible exit codes.
428
+
429
+ A failed configured threshold returns a nonzero exit code. A requested timing gate without enough samples is reported as `INCONCLUSIVE`, with exit code 4.
430
+
431
+ The local Bench view reads a persisted comparison report; it does not execute benchmarks itself.
432
+
433
+ Baseline policy bypasses are an explicit comparison control, not an enforced configuration. Timing results depend on the host and workload and are not performance guarantees.
434
+
435
+ Read the [Bench report schema](docs/bench-report-v1.md) and [Bench regression guide](docs/bench-regression.md) before consuming reports programmatically.
436
+
437
+ ---
438
+
439
+ ## Local repository and supply-chain scanning
440
+
441
+ Run a scan against an explicit repository path:
442
+
443
+ ```powershell
444
+ New-Item -ItemType Directory -Force .runflect | Out-Null
445
+
446
+ runflect supply-chain scan . --output .runflect/supply-chain-report.json
447
+ ```
448
+
449
+ Fail when findings meet the configured severity threshold:
450
+
451
+ ```powershell
452
+ runflect supply-chain scan . --fail-on high
453
+ ```
454
+
455
+ The scanner reads supported Python and npm dependency files, selected Docker and GitHub Actions configuration, MCP server declarations, and literal Python model references.
456
+
457
+ It builds a versioned inventory/evidence graph and reports bounded static findings such as:
458
+
459
+ - Non-exact dependency declarations
460
+ - GitHub Actions references that are not pinned to commit SHAs
461
+ - Other supported configuration patterns
462
+
463
+ Reports include counts, findings, source locations when available, parse errors, unsupported formats, and coverage limitations.
464
+
465
+ The scanner does not execute repository code, download vulnerability advisories, establish package exploitability, verify artifact provenance, or prove runtime reachability.
466
+
467
+ A successful scan does not mean that the repository has no vulnerabilities. It means the scan completed according to the implemented rules and configured failure threshold.
468
+
469
+ See the [Supply Chain guide](docs/supply-chain.md) for exact coverage and limitations.
470
+
471
+ ---
472
+
473
+ ## OpenTelemetry export
474
+
475
+ Runflect supports explicit export of completed recorded traces through an OpenTelemetry `SpanExporter` supplied by your application.
476
+
477
+ Install the optional dependencies:
478
+
479
+ ```powershell
480
+ python -m pip install -e ".[opentelemetry]"
481
+ ```
482
+
483
+ Then call the export function:
484
+
485
+ ```python
486
+ from runflect.integrations.opentelemetry import export_trace
487
+
488
+ # Supply an existing run ID and an exporter configured by your application.
489
+ export_trace(run_id, exporter, db_path=".runflect/traces.sqlite3")
490
+ ```
491
+
492
+ Replace `run_id` and `exporter` with your actual run identifier and configured exporter.
493
+
494
+ Export is explicit and uses existing recorded traces. Exported telemetry leaves the local database boundary, so review your destination and data-handling requirements first.
495
+
496
+ See [OpenTelemetry export](docs/opentelemetry.md) for the mapping, privacy behavior, and failure limits.
497
+
498
+ ---
499
+
500
+ ## Run the local web interface
501
+
502
+ Runflect includes an optional read-only dashboard for inspecting local traces and configured reports.
503
+
504
+ The web app uses Next.js 16 and requires Node.js 20.9 or newer. The Python API is an optional `web` extra.
505
+
506
+ ### 1. Install the API dependencies
507
+
508
+ From the repository root:
509
+
510
+ ```powershell
511
+ python -m pip install -e ".[web,supply-chain]"
512
+
513
+ New-Item -ItemType Directory -Force .runflect | Out-Null
514
+
515
+ $env:RUNFLECT_DB = (Join-Path (Get-Location) ".runflect\traces.sqlite3")
516
+ $env:RUNFLECT_BENCH_REPORT = (Join-Path (Get-Location) ".runflect\bench-report.json")
517
+ $env:RUNFLECT_SUPPLY_CHAIN_REPORT = (Join-Path (Get-Location) ".runflect\supply-chain-report.json")
518
+ $env:RUNFLECT_SECURITY_REPORT = (Join-Path (Get-Location) ".runflect\security-report.json")
519
+
520
+ python examples/quickstart.py
521
+ python examples/policy_quickstart.py
522
+
523
+ runflect bench run --output .runflect/bench-report.json
524
+ runflect supply-chain scan . --output .runflect/supply-chain-report.json
525
+
526
+ runflect security run --suite examples/security/core-v1.json --db $env:RUNFLECT_DB --output .runflect/security-report.json
527
+
528
+ runflect-web-api
529
+ ```
530
+
531
+ ### 2. Start the frontend
532
+
533
+ In a second terminal:
534
+
535
+ ```powershell
536
+ Set-Location web
537
+
538
+ npm ci
539
+
540
+ $env:NEXT_PUBLIC_RUNFLECT_API_URL = "http://127.0.0.1:8765"
541
+
542
+ npm run dev -- --hostname 127.0.0.1
543
+ ```
544
+
545
+ Open:
546
+
547
+ - Landing page: `http://127.0.0.1:3000`
548
+ - Dashboard: `http://127.0.0.1:3000/dashboard`
549
+
550
+ The setup commands generate local data and reports before starting the API.
551
+
552
+ The dashboard includes:
553
+
554
+ - Run overview and trace inspection
555
+ - Policy history and decisions
556
+ - Bench results and persisted comparisons
557
+ - Security simulation reports
558
+ - Supply-chain inventory and findings
559
+ - Evidence-linked navigation where explicit references exist
560
+
561
+ The API binds to loopback, allows only configured localhost browser origins, serves GET requests only, and does not execute tools or modify policies.
562
+
563
+ Supply Chain links to spans only when the available evidence contains the exact matching unique component ID. That relationship represents explicit caller attribution, not proven dependency use or causality.
564
+
565
+ The local API is a read-only convenience, not a hosted service or complete security boundary.
566
+
567
+ See the [web guide](docs/web.md), [Supply Chain guide](docs/supply-chain.md), and [Security simulation guide](docs/security-simulation.md) for further details.
568
+
569
+ ### Optional LangGraph traces in the dashboard
570
+
571
+ Install the LangGraph extra as well:
572
+
573
+ ```powershell
574
+ python -m pip install -e ".[langgraph,web,supply-chain]"
575
+ ```
576
+
577
+ With `RUNFLECT_DB` pointing to the same local database, run:
578
+
579
+ ```powershell
580
+ python examples/langgraph_quickstart.py
581
+ ```
582
+
583
+ The example records one successful and one deliberately failed LangGraph execution using a fixed tool and deterministic fake chat model. It does not contact a model provider or external network.
584
+
585
+ ---
586
+
587
+ ## Example developer workflow
588
+
589
+ A typical local workflow looks like this:
590
+
591
+ 1. **Record:** Run `python examples/quickstart.py` and inspect executions using `runflect list` and `runflect show`.
592
+ 2. **Protect:** Wrap sensitive tool calls with `ToolGuard`, using `examples/policy_quickstart.py` as a starting point.
593
+ 3. **Evaluate:** Import and evaluate a policy revision against offline scenarios, then review it before explicit activation.
594
+ 4. **Benchmark:** Run `runflect bench run` and preserve the versioned report.
595
+ 5. **Compare:** Save a baseline and compare subsequent reports to detect supported regressions.
596
+ 6. **Simulate:** Run `runflect security run` and inspect scenario verdicts and recorded mock traces.
597
+ 7. **Inspect dependencies:** Run `runflect supply-chain scan .` to review supported repository evidence.
598
+ 8. **Investigate:** Follow exact evidence references across the local CLI and dashboard when those relationships are available.
599
+ 9. **Export:** Optionally export a completed trace through your configured OpenTelemetry exporter.
600
+
601
+ The purpose is to create a repeatable loop:
602
+
603
+ **Record → Protect → Evaluate → Inspect → Improve**
604
+
605
+ This is an evidence-driven developer workflow, not an automatic root-cause engine or a guarantee that every agent failure can be reproduced.
606
+
607
+ ---
608
+
609
+ ## Security and privacy
610
+
611
+ Runflect is designed for local inspection, but local-first does not mean risk-free.
612
+
613
+ - **Explicit capture:** The core recorder does not automatically inspect agent state. It stores values explicitly passed to span inputs and outputs.
614
+ - **Default redaction:** Common sensitive mapping keys are recursively replaced with `[REDACTED]` by default. Setting `redact=False` disables this protection.
615
+ - **Payload limits:** Recorded payloads are bounded. Oversized payloads are omitted with a safe span attribute.
616
+ - **Error messages:** Error messages are omitted by default. `capture_error_messages=True` opts into storing messages up to 4,096 characters after key-based redaction.
617
+ - **LangGraph privacy:** Input/output capture is off by default and can be opted into using `capture_inputs=True` and/or `capture_outputs=True`. Exception messages are omitted by the adapter.
618
+ - **Guard boundaries:** `ToolGuard` protects calls made through its wrapper. Direct calls outside that boundary can bypass it.
619
+ - **Audit behavior:** A pre-execution policy audit failure blocks a guarded tool. A later audit or trace write failure cannot undo an external side effect.
620
+ - **Local storage:** SQLite is not encryption or a tamper-proof ledger. Protect the database with host filesystem controls.
621
+ - **Approvals:** The approval authority is in-process, not a distributed approval service.
622
+ - **Web interface:** The API is a local read-only convenience with loopback and Host/CORS checks, not a complete security boundary.
623
+ - **Scanner scope:** Repository scanning is bounded static analysis. It does not execute code or download vulnerability advisories.
624
+ - **OpenTelemetry:** Export is explicit. Exported telemetry leaves the local database boundary and should be handled accordingly.
625
+
626
+ Runflect is not a sandbox, does not prevent arbitrary code from bypassing wrappers, and does not guarantee secure agent behavior.
627
+
628
+ Read the [threat model](docs/threat-model.md), [architecture and guarantee boundaries](docs/architecture.md#boundaries-and-failure-behavior), and [release checklist](docs/release-checklist.md#guarantee-boundaries).
629
+
630
+ ---
631
+
632
+ ## Current status and limitations
633
+
634
+ Runflect is an early-stage developer preview. Its implemented capabilities include local Python execution recording and CLI inspection, synchronous and asynchronous LangGraph callbacks, guarded tool policies and audit records, immutable policy history, offline scenario evaluation, deterministic Bench and regression comparisons, security simulation fixtures, explicit OpenTelemetry export, an optional local read-only web interface, and bounded static repository scanning.
635
+
636
+ The CLI 2.0 redesign adds a terminal homepage, contextual help, module invocation, structured output options, clearer diagnostics, and improved run inspection.
637
+
638
+ The following limitations remain important:
639
+
640
+ - **Model instrumentation:** Provider/model callback spans and token counts are available only when supported callbacks supply the necessary metadata. Streaming-token capture and cross-process checkpoint resume are unsupported.
641
+ - **Cost attribution:** Cost observations require a user-supplied versioned rate file and complete explicit input/output token usage. They are partial estimates, not invoice reconciliation.
642
+ - **Policy controls:** Policy argument constraints, distributed approvals, and durable external approval workflows are unavailable.
643
+ - **Security coverage:** The offline fixtures exercise specific deterministic boundaries. The optional LangGraph fake-model scenario is not a real-provider attack evaluation and does not establish general prompt-injection resistance.
644
+ - **Runtime enforcement:** Calls outside a guarded wrapper are not protected by that wrapper.
645
+ - **Investigation:** Evidence-linked navigation exists where exact references are available. A general causal graph, automatic root-cause analysis, and general-purpose deterministic replay are not provided.
646
+ - **Supply chain:** The scanner does not provide complete SBOM coverage, known-vulnerability advisory matching, model provenance verification, or proven runtime dependency reachability.
647
+ - **Hosting:** Remote ingestion, multi-user hosting, encryption, retention management, and tamper-proof audit storage are unavailable.
648
+ - **Benchmarking:** Functional scenarios are deterministic within their fixtures. Timing measurements depend on the host and workload and are not performance guarantees.
649
+ - **Distribution:** The package has not yet been published to public PyPI. Source-checkout installation is the supported installation path for now.
650
+
651
+ These are known product boundaries, not capabilities to rely on today.
652
+
653
+ ---
654
+
655
+ ## Development and testing
656
+
657
+ Install the optional integrations used by the full test suite:
658
+
659
+ ```powershell
660
+ python -m pip install -e ".[langgraph,opentelemetry]"
661
+ ```
662
+
663
+ Run the Python tests:
664
+
665
+ ```powershell
666
+ python -m unittest discover -s tests -v
667
+ ```
668
+
669
+ Compile the Python source and examples:
670
+
671
+ ```powershell
672
+ python -m compileall -q src tests examples
673
+ ```
674
+
675
+ Run the release smoke test:
676
+
677
+ ```powershell
678
+ python examples/release_smoke.py
679
+ ```
680
+
681
+ The release smoke test exercises the deterministic Bench suite, security scenarios, policy/audit behavior, and trace export.
682
+
683
+ ### Frontend checks
684
+
685
+ From the `web/` directory:
686
+
687
+ ```powershell
688
+ npm ci
689
+ npm run typecheck
690
+ npm run build
691
+ ```
692
+
693
+ Run browser tests if the repository's Playwright setup is installed and configured:
694
+
695
+ ```powershell
696
+ npm run test:e2e
697
+ ```
698
+
699
+ ### Useful project documentation
700
+
701
+ - [CLI design, commands, exit codes, and PowerShell acceptance](docs/cli-v2.md)
702
+ - [Architecture and guarantee boundaries](docs/architecture.md)
703
+ - [Local threat model and tested boundaries](docs/threat-model.md)
704
+ - [Windows PowerShell acceptance walkthrough](docs/manual-acceptance-powershell.md)
705
+ - [Bench report schema](docs/bench-report-v1.md)
706
+ - [Bench baselines and regression comparisons](docs/bench-regression.md)
707
+ - [Custom offline evaluation suite](examples/scenarios/custom-evaluation-v1.json)
708
+ - [Local reliability observations and cost attribution](docs/reliability.md)
709
+ - [OpenTelemetry export](docs/opentelemetry.md)
710
+ - [Local repository scanning](docs/supply-chain.md)
711
+ - [Local web interface](docs/web.md)
712
+ - [Release readiness checklist](docs/release-checklist.md)
713
+
714
+ ---
715
+
716
+ ## Contributing and license
717
+
718
+ Runflect is under active development. Please review the repository's current contribution guidance and license before contributing, reusing, or distributing the code.
719
+
720
+ The repository checkout used to prepare this README did not contain a `LICENSE` file or a contribution guide. Add the intended license and contribution instructions before making public release claims.