driftguard-tokepi 0.5.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- driftguard_tokepi-0.5.0/CHANGELOG.md +451 -0
- driftguard_tokepi-0.5.0/LICENSE +21 -0
- driftguard_tokepi-0.5.0/MANIFEST.in +5 -0
- driftguard_tokepi-0.5.0/PKG-INFO +195 -0
- driftguard_tokepi-0.5.0/README.md +148 -0
- driftguard_tokepi-0.5.0/driftguard/__init__.py +20 -0
- driftguard_tokepi-0.5.0/driftguard/adapters/__init__.py +21 -0
- driftguard_tokepi-0.5.0/driftguard/adapters/claude_agent.py +106 -0
- driftguard_tokepi-0.5.0/driftguard/adapters/langchain.py +148 -0
- driftguard_tokepi-0.5.0/driftguard/adapters/mcp.py +161 -0
- driftguard_tokepi-0.5.0/driftguard/adapters/openai_agents.py +116 -0
- driftguard_tokepi-0.5.0/driftguard/adapters/python_agent.py +276 -0
- driftguard_tokepi-0.5.0/driftguard/adapters/usage.py +50 -0
- driftguard_tokepi-0.5.0/driftguard/agent_analysis.py +274 -0
- driftguard_tokepi-0.5.0/driftguard/baseline.py +118 -0
- driftguard_tokepi-0.5.0/driftguard/client.py +247 -0
- driftguard_tokepi-0.5.0/driftguard/gateway/__init__.py +19 -0
- driftguard_tokepi-0.5.0/driftguard/gateway/cli.py +126 -0
- driftguard_tokepi-0.5.0/driftguard/gateway/config.py +127 -0
- driftguard_tokepi-0.5.0/driftguard/gateway/proxy.py +222 -0
- driftguard_tokepi-0.5.0/driftguard/policy.py +135 -0
- driftguard_tokepi-0.5.0/driftguard/server/__init__.py +31 -0
- driftguard_tokepi-0.5.0/driftguard/server/app.py +648 -0
- driftguard_tokepi-0.5.0/driftguard/server/cli.py +56 -0
- driftguard_tokepi-0.5.0/driftguard/server/db.py +298 -0
- driftguard_tokepi-0.5.0/driftguard/server/notifications.py +66 -0
- driftguard_tokepi-0.5.0/driftguard/server/ratelimit.py +91 -0
- driftguard_tokepi-0.5.0/driftguard/server/realtime.py +93 -0
- driftguard_tokepi-0.5.0/driftguard/server/retention.py +71 -0
- driftguard_tokepi-0.5.0/driftguard/server/schemas.py +243 -0
- driftguard_tokepi-0.5.0/driftguard/server/security.py +80 -0
- driftguard_tokepi-0.5.0/driftguard/server/service.py +1009 -0
- driftguard_tokepi-0.5.0/driftguard/server/settings.py +140 -0
- driftguard_tokepi-0.5.0/driftguard/server/static/assets/charts-CJQ8HGlc.js +103 -0
- driftguard_tokepi-0.5.0/driftguard/server/static/assets/index-C2YdrKjD.js +342 -0
- driftguard_tokepi-0.5.0/driftguard/server/static/assets/index-CgZZ5_dJ.css +1 -0
- driftguard_tokepi-0.5.0/driftguard/server/static/driftguard.svg +4 -0
- driftguard_tokepi-0.5.0/driftguard/server/static/index.html +21 -0
- driftguard_tokepi-0.5.0/driftguard_tokepi.egg-info/PKG-INFO +195 -0
- driftguard_tokepi-0.5.0/driftguard_tokepi.egg-info/SOURCES.txt +44 -0
- driftguard_tokepi-0.5.0/driftguard_tokepi.egg-info/dependency_links.txt +1 -0
- driftguard_tokepi-0.5.0/driftguard_tokepi.egg-info/entry_points.txt +3 -0
- driftguard_tokepi-0.5.0/driftguard_tokepi.egg-info/requires.txt +28 -0
- driftguard_tokepi-0.5.0/driftguard_tokepi.egg-info/top_level.txt +1 -0
- driftguard_tokepi-0.5.0/pyproject.toml +90 -0
- driftguard_tokepi-0.5.0/setup.cfg +4 -0
|
@@ -0,0 +1,451 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to DriftGuard are recorded here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses
|
|
5
|
+
[semantic versioning](https://semver.org/) (pre-1.0: minor versions may break).
|
|
6
|
+
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
## [0.5.0] - 2026-11-11
|
|
10
|
+
|
|
11
|
+
First release published to PyPI. It contains the 0.4.0 foundation and
|
|
12
|
+
everything built on top of it: the activity log, the signal box dashboard
|
|
13
|
+
redesign, the analytics (baseline comparison, spread, latency), the MCP gateway,
|
|
14
|
+
and the framework adapters.
|
|
15
|
+
|
|
16
|
+
### Changed
|
|
17
|
+
- **`driftguard-server` fails gracefully without the server extra.** Running it
|
|
18
|
+
from a base `pip install driftguard-tokepi` now prints `pip install "driftguard-tokepi[server]"`
|
|
19
|
+
instead of a `ModuleNotFoundError` traceback. Importing `driftguard.server` is
|
|
20
|
+
lazy, so the entry point and the `driftguard-mcp-proxy` command never pull in
|
|
21
|
+
FastAPI.
|
|
22
|
+
|
|
23
|
+
### Added
|
|
24
|
+
- **Framework adapters for LangChain, the OpenAI Agents SDK and the Claude Agent
|
|
25
|
+
SDK.** `DriftGuardCallbackHandler` (a LangChain/LangGraph callback handler),
|
|
26
|
+
`DriftGuardRunHooks` (OpenAI Agents SDK run hooks) and
|
|
27
|
+
`DriftGuardClaudeAgentHooks` (Claude Agent SDK tool hooks) each record one
|
|
28
|
+
agent event per tool call — with its duration and outcome — and attribute each
|
|
29
|
+
model turn's tokens to the next tool, reusing `AgentContext`. Each plugs into
|
|
30
|
+
the framework's own hook mechanism in one or two lines. None of the frameworks
|
|
31
|
+
is a dependency: each adapter subclasses its framework's base when installed
|
|
32
|
+
and still works for testing without it. 10 new tests.
|
|
33
|
+
- **Latency as a first-class metric.** Telemetry accepts `latency_ms`, a new
|
|
34
|
+
`latency_limit_ms` policy rule (default 10,000 ms) flags requests slower than
|
|
35
|
+
the limit, and the reading flows through the summary (`averages.latency_ms`,
|
|
36
|
+
`violation_rates.latency_limit_ms`), the Overview's All readings, the Drift
|
|
37
|
+
Risk Breakdown, the Policy page and its impact preview, and the Test Telemetry
|
|
38
|
+
form. A slow request adds 0.15 to its risk score — enough to tip a borderline
|
|
39
|
+
event into `warning`, never enough to flag a fast one, and an event that
|
|
40
|
+
reports no latency is never flagged. The SDK already forwards it
|
|
41
|
+
(`capture_metrics(latency_ms=...)`). Existing databases gain the two columns on
|
|
42
|
+
startup. 9 new backend tests and a frontend preview test.
|
|
43
|
+
- **MCP gateway (`driftguard-mcp-proxy`).** A stdio proxy that sits between any
|
|
44
|
+
MCP client and a downstream MCP server, forwards every JSON-RPC message
|
|
45
|
+
unchanged, and records each `tools/call` as a DriftGuard agent event with its
|
|
46
|
+
duration and outcome — so closed agents that speak MCP can be observed with no
|
|
47
|
+
code change. One session's calls group under one task, so a retried failing
|
|
48
|
+
tool is flagged as blocked. Config comes from flags, `DRIFTGUARD_*` env vars or
|
|
49
|
+
a JSON file (the same keys as the Claude Code integration). It logs only to
|
|
50
|
+
stderr, passes a message it can't parse through untouched, and resolves the
|
|
51
|
+
server command on `PATH` (including `.cmd` wrappers on Windows). New
|
|
52
|
+
`driftguard.gateway` package, 13 tests.
|
|
53
|
+
- **View filters in the address.** Alerts keeps its tab and severity in the
|
|
54
|
+
address (`#/alerts?tab=resolved&severity=critical`) and Logs keeps its search,
|
|
55
|
+
kind, outcome and task (`#/logs?q=pytest&outcome=failed`), so a filtered list
|
|
56
|
+
can be linked. Defaults are left out and the values go when you leave the view.
|
|
57
|
+
Back and forward move between them, and a link to another view carries only
|
|
58
|
+
the global filters.
|
|
59
|
+
- **Plate panels.** The Overview's timeline, Drift Risk Breakdown, Top Root
|
|
60
|
+
Causes and Open Alerts are named with engraved plate labels over a sleeper
|
|
61
|
+
rule, the same idiom as the instruments, instead of a plain title over a
|
|
62
|
+
hairline. `Panel` gains `variant="plate"`.
|
|
63
|
+
- **Bulk resolve and reopen in one request.** `PATCH /projects/{id}/alerts` takes
|
|
64
|
+
`{"ids": [...], "resolved": bool}` for up to 500 alerts and returns the alerts
|
|
65
|
+
now in that state plus any ids that weren't the project's. Ids from other
|
|
66
|
+
projects are never touched. Each changed alert is still pushed on the
|
|
67
|
+
WebSocket. The Alerts view's selection actions use it, so resolving a hundred
|
|
68
|
+
alerts is one round trip. 3 new tests.
|
|
69
|
+
- **Frontend unit tests.** The dashboard gains Vitest with jsdom: 33 tests over
|
|
70
|
+
hash routing and shareable links, the keyboard shortcuts (including the `g`
|
|
71
|
+
timeout, typing, modifiers and dialogs), input validation, which scores a
|
|
72
|
+
source reports, the baseline wording, and the Policy impact preview, whose
|
|
73
|
+
maths moved from the view into `lib/policyPreview.ts`. `npm test` runs in CI
|
|
74
|
+
before the build.
|
|
75
|
+
- **Demo slides.** `docs/demo.md` links a nine-card deck that follows the running
|
|
76
|
+
order, built from the current screenshots.
|
|
77
|
+
- **`startup.bat`** for Windows development: creates the environment on first
|
|
78
|
+
run, starts the API on port 8000 against the local SQLite file and the Vite dev
|
|
79
|
+
server on port 5173 with hot reload. `demo.bat` stays the one-click demo.
|
|
80
|
+
- **Claude Code templates** (`integrations/claude_code/templates/`): a
|
|
81
|
+
`driftguard.json` and a `settings.local.json` with all six hooks, ready to copy
|
|
82
|
+
into another repository's `.claude/` folder. Three placeholders to fill in; no
|
|
83
|
+
key is shipped.
|
|
84
|
+
- **Signal box redesign of the dashboard.** The Overview now leads with one
|
|
85
|
+
status plate, then a track diagram of the most urgent task, then four
|
|
86
|
+
instruments.
|
|
87
|
+
- The status plate shows one lamp and one word (`Critical`, `Warning`,
|
|
88
|
+
`Stable` or `No data`), the diagnosis sentence, the recommendation and a
|
|
89
|
+
single action.
|
|
90
|
+
- The track draws each tool a task used as a block section, with the
|
|
91
|
+
blocking tool held at a red signal. It also appears on Agent Diagnosis and
|
|
92
|
+
in the task drawer.
|
|
93
|
+
- Open alerts, mean risk score, response quality and tool failure rate sit in
|
|
94
|
+
one strip. Gauges draw each reading against its policy floor or limit.
|
|
95
|
+
Everything else is under a collapsed **All readings**.
|
|
96
|
+
- The timeline draws the prompt-token limit and both score floors as dashed
|
|
97
|
+
lines, hatches the span below the floors, and offers its data as a table.
|
|
98
|
+
- **Baseline comparison and sample-size guard.** `GET /projects/{id}/summary`
|
|
99
|
+
gains `baseline`: the window's mean retrieval score, response quality and risk
|
|
100
|
+
score against the same length of time just before it (at least 24 hours).
|
|
101
|
+
- A change counts only when it is at least 3 standard errors wide (a Welch
|
|
102
|
+
two-sample z-test) and at least 0.05 in size.
|
|
103
|
+
- With under 10 events on either side it says "too few events to compare"
|
|
104
|
+
instead of giving a verdict. "All time" has no comparison.
|
|
105
|
+
- The Overview shows it under Mean risk score, Response quality and Mean
|
|
106
|
+
retrieval score, and adds a "Worse than usual" line to the status plate. It is
|
|
107
|
+
advice only: it raises no alert and doesn't change the status.
|
|
108
|
+
- `driftguard/baseline.py` holds the maths, with 14 new tests.
|
|
109
|
+
- **Response quality for Claude Code, from outcome checks.** When a turn runs tests,
|
|
110
|
+
a linter, a type check or a build, the hook scores the turn's last model call 1.0
|
|
111
|
+
if the last check passed and 0.0 if it failed. Turns that run no check get no
|
|
112
|
+
score, so the average is over checked turns only. Pass or fail also reads the
|
|
113
|
+
output for what failing runs print, because a pipe into `tail` hides the exit code,
|
|
114
|
+
and a command that never started is ignored. The basis is in the event's metadata.
|
|
115
|
+
`"quality_from_checks": false` turns it off. There is deliberately **no retrieval
|
|
116
|
+
score** for Claude Code: it has no retrieval step, and scoring empty searches would
|
|
117
|
+
raise false degradation warnings. 54 new tests.
|
|
118
|
+
- **Scores a source never sends are left off the dashboard.** The Overview drops the
|
|
119
|
+
Response quality instrument (three instruments instead of four), the Mean retrieval
|
|
120
|
+
score reading, the matching Drift Risk Breakdown row, and the timeline's line, axis
|
|
121
|
+
and legend entry, and All readings says what was left out. A score counts as sent if
|
|
122
|
+
it appears in the window or the period before. Scores sent for only some events are
|
|
123
|
+
drawn as dots, not joined by a line. The Policy preview counts only events that
|
|
124
|
+
carry the reading ("8 of 25") and says when none do.
|
|
125
|
+
- **Spread behind each average.** `GET /projects/{id}/summary` gains `spread`:
|
|
126
|
+
the median and the 10th and 90th percentiles of each metric, from the latest
|
|
127
|
+
5,000 readings in the window. The Overview shows them under Mean risk score,
|
|
128
|
+
Response quality and Mean retrieval score, with the share of events under the
|
|
129
|
+
floor, so an average can't hide a split. It appears from 10 events. 6 new tests.
|
|
130
|
+
- **Restore defaults on Policy.** A button fills every field with the thresholds a
|
|
131
|
+
new project starts with, for you to review and save. It is backed by a new
|
|
132
|
+
`GET /projects/{id}/policy/defaults` (session or key), so the defaults live only
|
|
133
|
+
on the server. 2 new tests.
|
|
134
|
+
- **Attempt timeline in the task drawer.** One row per tool call on a shared time
|
|
135
|
+
axis, with each bar ending when the attempt was recorded and running back by its
|
|
136
|
+
duration. A retry loop shows as a bunch of red bars. It shows the latest 40
|
|
137
|
+
attempts and has a text summary.
|
|
138
|
+
- **Delete account from the dashboard.** Project Settings has a "Your account"
|
|
139
|
+
panel. Deleting lists the projects that will go, and asks for your email and
|
|
140
|
+
password. A wrong password deletes nothing. It uses the existing
|
|
141
|
+
`DELETE /auth/me`.
|
|
142
|
+
- **Keyboard shortcuts.** `g` then `o` `d` `l` `e` `a` `n` `p` `s` `t` jumps
|
|
143
|
+
between views, `/` focuses the search box, `r` refreshes and `?` lists them.
|
|
144
|
+
They are off while typing or with a dialog open.
|
|
145
|
+
- **Shareable links.** The address carries the view, the project and both
|
|
146
|
+
filters (`#/alerts?env=prod&range=7d&project=showcase`). Opening a link
|
|
147
|
+
applies them. **Reset filters** appears when a filter is not at its default.
|
|
148
|
+
- **Bulk resolve and reopen** on Alerts, with select-all. The Alerts tabs now
|
|
149
|
+
respond to arrow keys, Home and End, and the new-alert form opens from a
|
|
150
|
+
**New alert** button.
|
|
151
|
+
- **Policy impact preview.** Each threshold says how many of the loaded events
|
|
152
|
+
or tasks it flags, before and after your edit. Fields validate as you leave
|
|
153
|
+
them. The save bar lists every change and warns when coverage drops.
|
|
154
|
+
- **Logs view** in the dashboard: the full history of agent activity, newest
|
|
155
|
+
first.
|
|
156
|
+
- Server-side search, plus kind and outcome filters.
|
|
157
|
+
- Live updates, with a pause button.
|
|
158
|
+
- A detail drawer, including "show only this task".
|
|
159
|
+
- CSV and JSON export.
|
|
160
|
+
- **Agent event kinds.** `kind` accepts `tool_call` (the default), `llm_call`,
|
|
161
|
+
`prompt`, `response`, `session_start` and `session_end`. Non-tool kinds don't
|
|
162
|
+
need `tool_name` or `status`, and are ignored by the diagnosis and agent alerts.
|
|
163
|
+
- **Opt-in content storage.**
|
|
164
|
+
- Agent events accept `input` and `output` previews.
|
|
165
|
+
- The server stores them only when the project's new `capture_content`
|
|
166
|
+
setting is on. It's off by default.
|
|
167
|
+
- Stored content is scrubbed, cut to 2,000 characters, and pruned by retention.
|
|
168
|
+
- `PATCH /projects/{id}` changes a project's name or `capture_content`.
|
|
169
|
+
- **Overview agent activity.** `GET /projects/{id}/summary` gains
|
|
170
|
+
`agent_activity`, computed from the same rows as Logs: tool calls, failures,
|
|
171
|
+
failure rate, tasks, finished turns, tokens used and average tool duration. The
|
|
172
|
+
Overview shows these as a row of cards. Retrieval and quality cards say "Not
|
|
173
|
+
reported by this source" when the telemetry has no such score.
|
|
174
|
+
- `GET /projects/{id}/agent-events` gains `kind`, `tool_name`, `agent_name`,
|
|
175
|
+
`outcome`, `q` (search) and `before_id` (paging).
|
|
176
|
+
- `GET /projects/{id}/agent-events/export?format=csv|json` downloads up to 10,000
|
|
177
|
+
filtered events.
|
|
178
|
+
- The WebSocket sends `{"type": "activity", "count": n}` after agent events are
|
|
179
|
+
ingested. Agent Diagnosis and Logs use it to refresh live.
|
|
180
|
+
- **SDK.**
|
|
181
|
+
- `DriftGuardClient(capture_content=True)` sends content previews. Without it,
|
|
182
|
+
`input` and `output` are removed before events are queued.
|
|
183
|
+
- With it on, `@driftguard_tool` and the MCP adapter send tool arguments and
|
|
184
|
+
results.
|
|
185
|
+
- `AgentContext.record_activity(kind, ...)` logs prompts, model calls and
|
|
186
|
+
other non-tool activity.
|
|
187
|
+
|
|
188
|
+
- `examples/seed_showcase.py` fills a project with a week of varied sample data
|
|
189
|
+
for every dashboard view, using only its API key.
|
|
190
|
+
- **Claude Code integration** (`integrations/claude_code/`): a hook script
|
|
191
|
+
that reports a Claude Code session to DriftGuard with no code changes.
|
|
192
|
+
- Each prompt becomes a task, and each tool call a `tool_call` event.
|
|
193
|
+
- Failures come from `PostToolUseFailure`.
|
|
194
|
+
- Prompts, turn endings and session start/end appear in Logs.
|
|
195
|
+
- Tool calls get their duration, model and tokens from the session
|
|
196
|
+
transcript. Cached context isn't counted, and each model call is counted
|
|
197
|
+
once. A later tool call from the same model call shows 0.
|
|
198
|
+
- Each turn is reported once, when Claude stops, as one `response`. It holds
|
|
199
|
+
the prompt, the answer, the turn's duration and the tokens of all its model
|
|
200
|
+
calls.
|
|
201
|
+
- Session start and end are opt-in (`"session_events": true`).
|
|
202
|
+
- Each finished turn sends LLM telemetry, one event per model call, with new
|
|
203
|
+
input tokens and the full context size. Turn it off with
|
|
204
|
+
`"send_telemetry": false`.
|
|
205
|
+
- A turn whose start isn't in the hook state falls back to the last prompt you
|
|
206
|
+
typed. It skips messages Claude Code inserts, and ignores starts older than
|
|
207
|
+
6 hours.
|
|
208
|
+
- It uses only the standard library, gives up after 2 seconds, never fails
|
|
209
|
+
Claude Code, and logs problems to `.claude/driftguard-hook.log`.
|
|
210
|
+
|
|
211
|
+
### Changed
|
|
212
|
+
- **New look.** The dashboard moves from a pure-black graphite console to a dark
|
|
213
|
+
black enamel panel with ivory type. Colour now appears only where
|
|
214
|
+
something deviates (red, amber, green lamps, each with an icon and a word).
|
|
215
|
+
Fonts are Barlow, Barlow Condensed and JetBrains Mono, replacing Space
|
|
216
|
+
Grotesk and Plus Jakarta Sans. The logo is the text "DriftGuard" alone, with no
|
|
217
|
+
symbol, and the browser-tab icon is a plain "D". The decorative background grid
|
|
218
|
+
is removed.
|
|
219
|
+
- **True-black canvas and brighter colours.** The page background is pure black
|
|
220
|
+
with panels lifted a shade, ivory text and the route colour are brighter, and
|
|
221
|
+
the red, amber, green and steel lamp colours are more luminous.
|
|
222
|
+
- The Logs kind filter now offers only Tool call, Prompt and Response. Model
|
|
223
|
+
call, Session start and Session end are no longer listed there, though rows of
|
|
224
|
+
those kinds still appear under All activity.
|
|
225
|
+
- **Themed dropdowns.** Every select uses a themed control. In Chrome and Edge
|
|
226
|
+
the open list is a black panel with a hairline border, a highlighted row and a
|
|
227
|
+
check mark on the chosen item. Other browsers keep their own list in the same
|
|
228
|
+
colours.
|
|
229
|
+
- **Layout fixes from the design review.** The track diagram's sections share the
|
|
230
|
+
full width. Agent Diagnosis gains a gauge of the longest run of failures in a
|
|
231
|
+
row against the block threshold. Top Root Causes moves under Drift Risk
|
|
232
|
+
Breakdown and Open Alerts runs full width, so there are no empty gaps. On a
|
|
233
|
+
phone the chart legend is one scrolling line, Test Telemetry is an icon, and the
|
|
234
|
+
filter row wraps.
|
|
235
|
+
- The baseline line says **No clear change** instead of "Steady", and every verdict
|
|
236
|
+
shows the events it rests on, for example "(89 events now, 93 before)". Under 30
|
|
237
|
+
events on a side it adds that a small drop could be missed. The research study
|
|
238
|
+
found a real 0.09 drop is detected only about a third of the time at 10 events,
|
|
239
|
+
so "steady" over-claimed.
|
|
240
|
+
- Disabled navigation, placeholder and footer text now meets 4.5:1 contrast.
|
|
241
|
+
The header selects have visible labels (**Env**, **Range**), and the
|
|
242
|
+
**Updated** time shows from tablet width up. The sidebar groups its nine
|
|
243
|
+
views into Monitor and Configure.
|
|
244
|
+
- The Overview moves Total events, Tokens at stake, retrieval score and the
|
|
245
|
+
agent counts into a collapsed **All readings** list. The "Active Alerts"
|
|
246
|
+
panel is now **Open Alerts**, and the Alerts panel title follows the tab.
|
|
247
|
+
- Agent Diagnosis shows three headline instruments, with the other counts
|
|
248
|
+
under **All counts**.
|
|
249
|
+
- The dashboard's Refresh button also reloads the project list, and the list
|
|
250
|
+
reloads when you return to the tab. A project created through the API or a
|
|
251
|
+
script no longer needs a page reload to appear.
|
|
252
|
+
- Existing databases gain `projects.capture_content` and
|
|
253
|
+
`agent_events.kind`/`input_text`/`output_text` automatically on startup.
|
|
254
|
+
Existing agent events become `tool_call` events.
|
|
255
|
+
|
|
256
|
+
### Documentation
|
|
257
|
+
- **Research: the baseline study** (`research/baseline_study.py`,
|
|
258
|
+
`research/BASELINE_STUDY.md`). A seeded simulation of the product's own rules,
|
|
259
|
+
fixed floors against the baseline comparison, with known truth. It reports
|
|
260
|
+
detection rates, false alarms and delays for one decision per window, for
|
|
261
|
+
noisier data, for checking after every event, and for slow decline, with 11
|
|
262
|
+
tests. Findings: the two rules fail in opposite places so both stay; the
|
|
263
|
+
10-event guard works but "Steady" over-claims at low counts; and a 3-sigma
|
|
264
|
+
check repeated at every event raises false alarms in 5 percent of streams, which
|
|
265
|
+
needs a rule of several checks in a row before anything alerts. The research
|
|
266
|
+
README now explains what each part can and can't show.
|
|
267
|
+
- **New guide: Measuring retrieval score and response quality**
|
|
268
|
+
(`docs/scoring-guide.md`). How to calculate each score, what to do when a source
|
|
269
|
+
has no evaluator (and why a proxy from telemetry is only a rough signal),
|
|
270
|
+
choosing the floors from your own data, how the scores become alerts, and what
|
|
271
|
+
the numbers can't tell you. The Logs Details column and its troubleshooting row
|
|
272
|
+
now explain that a dash means content storage is off.
|
|
273
|
+
- **Documentation refresh.** The architecture diagram is restyled to the new
|
|
274
|
+
look (SVG and PNG) and notes the dashboard's shortcuts and shared links. The
|
|
275
|
+
dashboard guide, demo outline, setup guides and architecture page use the new
|
|
276
|
+
names (**Env**, **Range**, instruments, track), and every screenshot is
|
|
277
|
+
retaken with a full week of data, plus new ones for Policy and the shortcut
|
|
278
|
+
sheet. `docs/development.md` lists the design files.
|
|
279
|
+
- **`PRODUCT.md`** records the product's users, purpose, positioning and
|
|
280
|
+
constraints for design work.
|
|
281
|
+
- **Design system** (`docs/design-system.md`) rewritten for the signal box
|
|
282
|
+
look. The **dashboard guide** covers the new Overview, shortcuts, shared
|
|
283
|
+
links, bulk actions and the policy preview, and its screenshots are
|
|
284
|
+
retaken.
|
|
285
|
+
- **`SETUP.md`**: a step-by-step guide for teammates, from a fresh laptop to
|
|
286
|
+
a running DriftGuard with sample data, on Windows, macOS and Linux.
|
|
287
|
+
- **Getting started** (`docs/quickstart.md`) rewritten: three ways to run it
|
|
288
|
+
(`demo.bat`, from source, Docker), four ways to send data (sample data, live
|
|
289
|
+
demo, Claude Code, SDK), and a troubleshooting table.
|
|
290
|
+
- New **How DriftGuard works** (`docs/concepts.md`): the problem, the two kinds
|
|
291
|
+
of events, a glossary, scoring and diagnosis with worked examples, and a
|
|
292
|
+
60-second summary.
|
|
293
|
+
- **Architecture diagram** as an image (`docs/images/architecture.svg`, plus a
|
|
294
|
+
PNG for slides), and new diagrams of the dashboard and of the telemetry, agent
|
|
295
|
+
event and Claude Code hook flows. Also a server module table and a
|
|
296
|
+
view-to-endpoint table.
|
|
297
|
+
- **Dashboard guide** rewritten with screenshots and a table for every card and
|
|
298
|
+
column, plus a map of which source fills which view.
|
|
299
|
+
- The demo guide gains a 15-minute presentation outline. The README gains a
|
|
300
|
+
"Start here" table, the diagram and a screenshot.
|
|
301
|
+
|
|
302
|
+
## [0.4.0] - unreleased
|
|
303
|
+
|
|
304
|
+
A rebuild of the foundation. v0.3 described several features as done that
|
|
305
|
+
didn't work: authentication, PostgreSQL, live alerts, and retention. This
|
|
306
|
+
release implements them, with tests.
|
|
307
|
+
|
|
308
|
+
### Security
|
|
309
|
+
- **Removed the unauthenticated `default` account.** v0.3 let any request with
|
|
310
|
+
no token act as a shared account, so anyone could list, read, and change
|
|
311
|
+
those projects.
|
|
312
|
+
- **Real authentication.** Email and password accounts, with passwords hashed by
|
|
313
|
+
scrypt. `POST /auth/register`, `/auth/login`, `/auth/logout`, `GET /auth/me`,
|
|
314
|
+
and `DELETE /auth/me`. v0.3's `POST /auth/session?account_id=` handed out a
|
|
315
|
+
token to anyone who knew an account ID.
|
|
316
|
+
- Sessions are revocable. Each JWT's `jti` is stored, so logout and account
|
|
317
|
+
deletion invalidate it.
|
|
318
|
+
- **Project API keys are hashed**, shown once, and can be rotated
|
|
319
|
+
(`POST /projects/{id}/api-key`). Existing plaintext keys are hashed on first
|
|
320
|
+
start. Keys now have a `dg_live_` prefix.
|
|
321
|
+
- **Fixed project takeover.** v0.3 used `INSERT OR REPLACE`, so creating a
|
|
322
|
+
project with another account's ID overwrote it. Creating an existing ID now
|
|
323
|
+
returns `409`.
|
|
324
|
+
- A project API key can no longer change policy. Policy changes, deletion, key
|
|
325
|
+
rotation, and alert changes need the owner's session.
|
|
326
|
+
- The WebSocket requires a session token or API key, sent as a subprotocol so
|
|
327
|
+
it stays out of URL access logs.
|
|
328
|
+
- Production refuses to start without a strong `DRIFTGUARD_SECRET_KEY`.
|
|
329
|
+
- Rate limits: separate limits for login (per IP), the API (per session or
|
|
330
|
+
key, 300/min), and ingestion (per key). The
|
|
331
|
+
Redis limiter is atomic and uses wall-clock time; v0.3 used
|
|
332
|
+
`time.monotonic()`, which isn't comparable across processes.
|
|
333
|
+
- Scrubbing now covers API keys and tokens (OpenAI, Anthropic, GitHub, AWS,
|
|
334
|
+
Google, Slack, bearer, private keys). Card numbers need a valid Luhn checksum,
|
|
335
|
+
so ordinary long numbers are no longer redacted.
|
|
336
|
+
- Security headers on every response, and HSTS in production.
|
|
337
|
+
- Upgraded to FastAPI 0.14x and Starlette 1.x. Starlette 0.38 had published
|
|
338
|
+
advisories.
|
|
339
|
+
|
|
340
|
+
### Added
|
|
341
|
+
- `driftguard.server`: an app factory (`create_app`) and a working
|
|
342
|
+
`driftguard-server` command. The old entry point pointed at an ASGI object and
|
|
343
|
+
couldn't run.
|
|
344
|
+
- **Automatic alerts.**
|
|
345
|
+
- Critical telemetry raises a `drift` alert, at most one per root cause per
|
|
346
|
+
cooldown.
|
|
347
|
+
- A blocked agent task raises one `agent` alert. Its wasted-token figure keeps
|
|
348
|
+
updating, and the alert resolves itself when the task recovers.
|
|
349
|
+
- Alerts have `id`, `source`, `task_id`, `root_cause`, and
|
|
350
|
+
`resolved`/`resolved_at`.
|
|
351
|
+
- `PATCH /projects/{id}/alerts/{alert_id}` resolves or reopens an alert.
|
|
352
|
+
- **Server-side scoring.** Each ingested event stores `risk_score`, `severity`,
|
|
353
|
+
`root_cause`, and `recommendation`.
|
|
354
|
+
- **Batch ingestion:** `POST /events/{id}/batch` and
|
|
355
|
+
`POST /agent-events/{id}/batch`, up to 500 events each.
|
|
356
|
+
- Events accept `occurred_at` and `metadata`. Agent events accept `input_hash`.
|
|
357
|
+
- `GET /projects/{id}/summary` rewritten: SQL aggregates, `violation_rates`,
|
|
358
|
+
`events_by_severity`, `top_root_causes`, and open-alert totals.
|
|
359
|
+
- `GET /projects/{id}/agent-events` with a `task_id` filter. Event lists page
|
|
360
|
+
with `before_id`.
|
|
361
|
+
- Realtime alerts at `WS /ws/projects/{id}` with an `action` field (`created`,
|
|
362
|
+
`resolved`, `reopened`), fanned out across workers through Redis.
|
|
363
|
+
- Slack and generic webhook notifications, delivered on a background thread.
|
|
364
|
+
- A scheduled retention job (`DRIFTGUARD_RETENTION_DAYS`). It prunes telemetry,
|
|
365
|
+
agent events, resolved alerts, and expired sessions. v0.3 had the code but
|
|
366
|
+
never ran it.
|
|
367
|
+
- **Agent diagnosis v2.**
|
|
368
|
+
- Per-task results in `tasks`.
|
|
369
|
+
- Recovery detection: a later success of the same tool.
|
|
370
|
+
- Policy settings `blocked_after_failures` and `retry_window_minutes`.
|
|
371
|
+
- Redundant-call detection.
|
|
372
|
+
- Recommendations based on the error type.
|
|
373
|
+
- **Adapters.**
|
|
374
|
+
- `@driftguard_tool` supports async functions. `AgentContext` uses
|
|
375
|
+
contextvars and supports `async with` and `auto_sync`.
|
|
376
|
+
- `record_llm_usage()` attributes model tokens to tool calls.
|
|
377
|
+
- `usage_from_response()` reads OpenAI, Anthropic, and Gemini responses.
|
|
378
|
+
- `MCPMiddleware.instrument_session()` wraps an MCP `ClientSession`.
|
|
379
|
+
- Inputs are fingerprinted automatically.
|
|
380
|
+
- **SDK.**
|
|
381
|
+
- Batched sync; failed batches stay queued in order (v0.3 could drop or
|
|
382
|
+
duplicate events).
|
|
383
|
+
- A bounded queue and client-side `occurred_at` timestamps.
|
|
384
|
+
- An optional `project_id` default, plus `flush()` and
|
|
385
|
+
`sync_agent_events_async()`.
|
|
386
|
+
- **Dashboard** moved into `frontend/` from the separate `Design-` repo,
|
|
387
|
+
rebranded to DriftGuard, and rebuilt against the real API.
|
|
388
|
+
- Schema setup is safe when several workers start at once (a PostgreSQL
|
|
389
|
+
advisory lock; retried on SQLite).
|
|
390
|
+
- The server closes its database connection pool on shutdown.
|
|
391
|
+
- Docker: a multi-stage, non-root image that bundles the dashboard, with a
|
|
392
|
+
healthcheck. Compose uses healthchecks, and Postgres and Redis are no longer
|
|
393
|
+
exposed on the host.
|
|
394
|
+
- CI: ruff, pytest on Python 3.10–3.13, the full server suite on PostgreSQL 16
|
|
395
|
+
and Redis 7, the dashboard build, and security scans that fail on findings.
|
|
396
|
+
Releases check that the wheel bundles the dashboard.
|
|
397
|
+
- `examples/live_demo.py` and [docs/demo.md](docs/demo.md): a step-by-step live
|
|
398
|
+
demo covering drift, a blocked agent, and automatic recovery.
|
|
399
|
+
`demo.bat` starts the whole demo on Windows in one step.
|
|
400
|
+
- Docs: new API reference, architecture, configuration, deployment,
|
|
401
|
+
development, and roadmap pages. `DESIGN.md` moved to `docs/design-system.md`. All existing docs were rewritten to match the
|
|
402
|
+
code. A `LICENSE` file was added (MIT, as `pyproject.toml` already declared).
|
|
403
|
+
|
|
404
|
+
### Changed
|
|
405
|
+
- **Package extras.** `pip install driftguard` now installs only `httpx`. The
|
|
406
|
+
server needs `driftguard[server]`; add `[postgres]` and `[redis]` as needed,
|
|
407
|
+
or `[all]`. NumPy and pandas are no longer dependencies.
|
|
408
|
+
- **Persistence rewritten** on SQLAlchemy Core. PostgreSQL works; v0.3 used
|
|
409
|
+
SQLite-only SQL and silently fell back to a local SQLite file. There are no
|
|
410
|
+
in-memory caches, so it's safe with several workers. Filtering happens in SQL.
|
|
411
|
+
- `recommend_mitigation` and `check_drift` use the project policy's thresholds;
|
|
412
|
+
v0.3 compared against hardcoded values.
|
|
413
|
+
- Root cause text: "trajectory quality degradation" is now "response quality
|
|
414
|
+
degradation".
|
|
415
|
+
- The policy returns `blocked_after_failures` as an integer (`3`, not `3.0`),
|
|
416
|
+
matching what `PUT /projects/{id}/policy` accepts.
|
|
417
|
+
- `.env.example` lists every server variable, including the `driftguard-server`
|
|
418
|
+
defaults `DRIFTGUARD_HOST`, `PORT` and `DRIFTGUARD_WORKERS`.
|
|
419
|
+
- Configuration uses `DRIFTGUARD_*` names. `JWT_SECRET`, `API_SECRET`, and
|
|
420
|
+
`RATE_LIMIT_PER_MINUTE` still work but are deprecated.
|
|
421
|
+
- The research prototypes (`data`, `detector`, `benchmark`, `experiment`,
|
|
422
|
+
`reporting`, and the demo API) moved to `research/`.
|
|
423
|
+
|
|
424
|
+
### Removed / breaking
|
|
425
|
+
- These routes are gone:
|
|
426
|
+
- `POST /accounts` → use `/auth/register`
|
|
427
|
+
- `POST /auth/session` → use `/auth/login`
|
|
428
|
+
- `GET /accounts/me` → use `/auth/me`
|
|
429
|
+
- `POST /alerts` → use `POST /projects/{id}/alerts`
|
|
430
|
+
- `GET /alerts/{id}` → use `GET /projects/{id}/alerts`
|
|
431
|
+
- `GET /events/{id}` → use `GET /projects/{id}/events`
|
|
432
|
+
- `/projects/{id}/metrics` and `/projects/{id}/dashboard` → use
|
|
433
|
+
`/projects/{id}/summary`
|
|
434
|
+
- `WS /ws/alerts/{id}` → use `WS /ws/projects/{id}`
|
|
435
|
+
- `driftguard.routes`, `driftguard.service`, `driftguard.middleware`,
|
|
436
|
+
`driftguard.config`, `driftguard.workers`, and `driftguard.notifications` moved
|
|
437
|
+
under `driftguard.server`. `driftguard.analytics` and
|
|
438
|
+
`driftguard.dashboard_data` were removed; `/summary` replaces them.
|
|
439
|
+
- `driftguard.compute_drift_scores` was removed from the public API. It was the
|
|
440
|
+
synthetic research detector and now lives in `research.detector`. Use
|
|
441
|
+
`driftguard.evaluate_drift` for LLM telemetry.
|
|
442
|
+
- `capture_metrics` no longer adds a `project` key to the payload.
|
|
443
|
+
- `requirements.txt` was removed; `pyproject.toml` is the single source of
|
|
444
|
+
dependencies.
|
|
445
|
+
- The legacy static dashboard (`static/dashboard.js`, `styles.css`) and the
|
|
446
|
+
1,200-line embedded HTML dashboard in `api.py` were removed.
|
|
447
|
+
|
|
448
|
+
## [0.3.0] - 2026-08
|
|
449
|
+
|
|
450
|
+
- The SDK client, FastAPI server, project API keys, agent event ingestion, and
|
|
451
|
+
the first blocking-tool diagnosis.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Sean Pereira
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|