redprobe 0.1.0__tar.gz → 0.1.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {redprobe-0.1.0 → redprobe-0.1.2}/PKG-INFO +19 -25
- {redprobe-0.1.0 → redprobe-0.1.2}/README.md +18 -21
- {redprobe-0.1.0 → redprobe-0.1.2}/pyproject.toml +1 -6
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe/consent.py +11 -7
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe.egg-info/PKG-INFO +19 -25
- {redprobe-0.1.0 → redprobe-0.1.2}/CONTRIBUTING.md +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/HISTORY.md +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/LICENSE +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/MANIFEST.in +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/docs/index.md +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/docs/installation.md +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/docs/usage.md +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/setup.cfg +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe/__init__.py +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe/__main__.py +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe/cli.py +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe/client.py +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe/probes.py +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe/redprobe.py +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe/reporter.py +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe/runner.py +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe/utils.py +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe.egg-info/SOURCES.txt +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe.egg-info/dependency_links.txt +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe.egg-info/entry_points.txt +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe.egg-info/requires.txt +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/src/redprobe.egg-info/top_level.txt +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/tests/__init__.py +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/tests/test_cli.py +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/tests/test_probes.py +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/tests/test_redprobe.py +0 -0
- {redprobe-0.1.0 → redprobe-0.1.2}/tests/test_runner.py +0 -0
|
@@ -1,13 +1,10 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: redprobe
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.2
|
|
4
4
|
Summary: A defensive security tool for hardening AI systems. Define YAML-based test cases to systematically probe LLMs for jailbreaks, prompt injections, biases, harmful content generation, data leakage, and policy violations before attackers find them. Compatible with any OpenAI-style API endpoint.
|
|
5
5
|
Author-email: "Audrey M. Roy Greenfeld" <audrey@feldroy.com>
|
|
6
6
|
Maintainer-email: "Audrey M. Roy Greenfeld" <audrey@feldroy.com>
|
|
7
7
|
License: BUSL 1.1
|
|
8
|
-
Project-URL: bugs, https://github.com/audreyfeldroy/redprobe/issues
|
|
9
|
-
Project-URL: changelog, https://github.com/audreyfeldroy/redprobe/blob/master/changelog.md
|
|
10
|
-
Project-URL: homepage, https://github.com/audreyfeldroy/redprobe
|
|
11
8
|
Requires-Python: >=3.10
|
|
12
9
|
Description-Content-Type: text/markdown
|
|
13
10
|
License-File: LICENSE
|
|
@@ -32,19 +29,16 @@ A defensive security tool for hardening AI systems. Define YAML-based test cases
|
|
|
32
29
|
## Quick Start
|
|
33
30
|
|
|
34
31
|
```bash
|
|
35
|
-
# Run with uv (recommended)
|
|
36
|
-
uvx redprobe
|
|
37
|
-
|
|
38
32
|
# Generate sample probes
|
|
39
|
-
redprobe init
|
|
33
|
+
uvx redprobe init
|
|
40
34
|
|
|
41
35
|
# Run probes against a model
|
|
42
|
-
redprobe run probes/
|
|
36
|
+
uvx redprobe run probes/
|
|
43
37
|
```
|
|
44
38
|
|
|
45
39
|
## Prerequisites
|
|
46
40
|
|
|
47
|
-
RedProbe works with any OpenAI-compatible API. The default configuration targets [LM Studio](https://lmstudio.ai/) running locally.
|
|
41
|
+
RedProbe requires [uv](https://docs.astral.sh/uv/getting-started/installation/) and works with any OpenAI-compatible API. The default configuration targets [LM Studio](https://lmstudio.ai/) running locally.
|
|
48
42
|
|
|
49
43
|
### Setting up LM Studio
|
|
50
44
|
|
|
@@ -75,12 +69,12 @@ Do not use RedProbe to test systems without authorization, extract private data,
|
|
|
75
69
|
|
|
76
70
|
## Commands
|
|
77
71
|
|
|
78
|
-
### `redprobe init`
|
|
72
|
+
### `uvx redprobe init`
|
|
79
73
|
|
|
80
74
|
Create sample probe files to get started:
|
|
81
75
|
|
|
82
76
|
```bash
|
|
83
|
-
$ redprobe init
|
|
77
|
+
$ uvx redprobe init
|
|
84
78
|
Created sample probes in probes/
|
|
85
79
|
|
|
86
80
|
Files created:
|
|
@@ -89,21 +83,21 @@ Files created:
|
|
|
89
83
|
- harmful.yaml (3 probes)
|
|
90
84
|
|
|
91
85
|
Run probes with:
|
|
92
|
-
redprobe run probes
|
|
86
|
+
uvx redprobe run probes
|
|
93
87
|
```
|
|
94
88
|
|
|
95
89
|
You can specify a different directory:
|
|
96
90
|
|
|
97
91
|
```bash
|
|
98
|
-
redprobe init my-probes/
|
|
92
|
+
uvx redprobe init my-probes/
|
|
99
93
|
```
|
|
100
94
|
|
|
101
|
-
### `redprobe list-probes`
|
|
95
|
+
### `uvx redprobe list-probes`
|
|
102
96
|
|
|
103
97
|
Preview what probes will run without actually calling the API:
|
|
104
98
|
|
|
105
99
|
```bash
|
|
106
|
-
$ redprobe list-probes probes/
|
|
100
|
+
$ uvx redprobe list-probes probes/
|
|
107
101
|
Bias Detection
|
|
108
102
|
Probes for demographic and social biases in model responses
|
|
109
103
|
|
|
@@ -130,12 +124,12 @@ Jailbreak Attempts
|
|
|
130
124
|
MEDIUM jailbreak-003: Roleplay bypass
|
|
131
125
|
```
|
|
132
126
|
|
|
133
|
-
### `redprobe run`
|
|
127
|
+
### `uvx redprobe run`
|
|
134
128
|
|
|
135
129
|
Run probes against a model:
|
|
136
130
|
|
|
137
131
|
```bash
|
|
138
|
-
$ redprobe run probes/
|
|
132
|
+
$ uvx redprobe run probes/
|
|
139
133
|
Loaded 9 probes from 3 suite(s)
|
|
140
134
|
Target: http://localhost:1234/v1 model openai/gpt-oss-20b
|
|
141
135
|
|
|
@@ -174,7 +168,7 @@ Running suite: Jailbreak Attempts
|
|
|
174
168
|
#### Command Options
|
|
175
169
|
|
|
176
170
|
```bash
|
|
177
|
-
redprobe run probes/ \
|
|
171
|
+
uvx redprobe run probes/ \
|
|
178
172
|
--base-url http://localhost:1234/v1 \ # API endpoint
|
|
179
173
|
--model openai/gpt-oss-20b \ # Model to test
|
|
180
174
|
--api-key not-needed \ # API key (if required)
|
|
@@ -278,7 +272,7 @@ cases:
|
|
|
278
272
|
|
|
279
273
|
### Organizing Probes
|
|
280
274
|
|
|
281
|
-
You can split probes across multiple YAML files. When you point `redprobe run` at a directory, it loads all `.yaml` and `.yml` files:
|
|
275
|
+
You can split probes across multiple YAML files. When you point `uvx redprobe run` at a directory, it loads all `.yaml` and `.yml` files:
|
|
282
276
|
|
|
283
277
|
```
|
|
284
278
|
probes/
|
|
@@ -292,12 +286,12 @@ probes/
|
|
|
292
286
|
|
|
293
287
|
Run a specific file:
|
|
294
288
|
```bash
|
|
295
|
-
redprobe run probes/jailbreaks.yaml
|
|
289
|
+
uvx redprobe run probes/jailbreaks.yaml
|
|
296
290
|
```
|
|
297
291
|
|
|
298
292
|
Run all probes in a directory:
|
|
299
293
|
```bash
|
|
300
|
-
redprobe run probes/
|
|
294
|
+
uvx redprobe run probes/
|
|
301
295
|
```
|
|
302
296
|
|
|
303
297
|
## Understanding Results
|
|
@@ -321,7 +315,7 @@ A triggered probe does not necessarily mean the model is unsafe. It means the re
|
|
|
321
315
|
Use `--output` to export full results including model responses:
|
|
322
316
|
|
|
323
317
|
```bash
|
|
324
|
-
redprobe run probes/ --output results.json
|
|
318
|
+
uvx redprobe run probes/ --output results.json
|
|
325
319
|
```
|
|
326
320
|
|
|
327
321
|
The JSON includes timestamps, prompts, full responses, and matched patterns for each probe.
|
|
@@ -334,7 +328,7 @@ The JSON includes timestamps, prompts, full responses, and matched patterns for
|
|
|
334
328
|
# Start Ollama with a model
|
|
335
329
|
ollama serve
|
|
336
330
|
|
|
337
|
-
redprobe run probes/ \
|
|
331
|
+
uvx redprobe run probes/ \
|
|
338
332
|
--base-url http://localhost:11434/v1 \
|
|
339
333
|
--model llama2
|
|
340
334
|
```
|
|
@@ -342,7 +336,7 @@ redprobe run probes/ \
|
|
|
342
336
|
### OpenAI
|
|
343
337
|
|
|
344
338
|
```bash
|
|
345
|
-
redprobe run probes/ \
|
|
339
|
+
uvx redprobe run probes/ \
|
|
346
340
|
--base-url https://api.openai.com/v1 \
|
|
347
341
|
--model gpt-4o-mini \
|
|
348
342
|
--api-key $OPENAI_API_KEY
|
|
@@ -7,19 +7,16 @@ A defensive security tool for hardening AI systems. Define YAML-based test cases
|
|
|
7
7
|
## Quick Start
|
|
8
8
|
|
|
9
9
|
```bash
|
|
10
|
-
# Run with uv (recommended)
|
|
11
|
-
uvx redprobe
|
|
12
|
-
|
|
13
10
|
# Generate sample probes
|
|
14
|
-
redprobe init
|
|
11
|
+
uvx redprobe init
|
|
15
12
|
|
|
16
13
|
# Run probes against a model
|
|
17
|
-
redprobe run probes/
|
|
14
|
+
uvx redprobe run probes/
|
|
18
15
|
```
|
|
19
16
|
|
|
20
17
|
## Prerequisites
|
|
21
18
|
|
|
22
|
-
RedProbe works with any OpenAI-compatible API. The default configuration targets [LM Studio](https://lmstudio.ai/) running locally.
|
|
19
|
+
RedProbe requires [uv](https://docs.astral.sh/uv/getting-started/installation/) and works with any OpenAI-compatible API. The default configuration targets [LM Studio](https://lmstudio.ai/) running locally.
|
|
23
20
|
|
|
24
21
|
### Setting up LM Studio
|
|
25
22
|
|
|
@@ -50,12 +47,12 @@ Do not use RedProbe to test systems without authorization, extract private data,
|
|
|
50
47
|
|
|
51
48
|
## Commands
|
|
52
49
|
|
|
53
|
-
### `redprobe init`
|
|
50
|
+
### `uvx redprobe init`
|
|
54
51
|
|
|
55
52
|
Create sample probe files to get started:
|
|
56
53
|
|
|
57
54
|
```bash
|
|
58
|
-
$ redprobe init
|
|
55
|
+
$ uvx redprobe init
|
|
59
56
|
Created sample probes in probes/
|
|
60
57
|
|
|
61
58
|
Files created:
|
|
@@ -64,21 +61,21 @@ Files created:
|
|
|
64
61
|
- harmful.yaml (3 probes)
|
|
65
62
|
|
|
66
63
|
Run probes with:
|
|
67
|
-
redprobe run probes
|
|
64
|
+
uvx redprobe run probes
|
|
68
65
|
```
|
|
69
66
|
|
|
70
67
|
You can specify a different directory:
|
|
71
68
|
|
|
72
69
|
```bash
|
|
73
|
-
redprobe init my-probes/
|
|
70
|
+
uvx redprobe init my-probes/
|
|
74
71
|
```
|
|
75
72
|
|
|
76
|
-
### `redprobe list-probes`
|
|
73
|
+
### `uvx redprobe list-probes`
|
|
77
74
|
|
|
78
75
|
Preview what probes will run without actually calling the API:
|
|
79
76
|
|
|
80
77
|
```bash
|
|
81
|
-
$ redprobe list-probes probes/
|
|
78
|
+
$ uvx redprobe list-probes probes/
|
|
82
79
|
Bias Detection
|
|
83
80
|
Probes for demographic and social biases in model responses
|
|
84
81
|
|
|
@@ -105,12 +102,12 @@ Jailbreak Attempts
|
|
|
105
102
|
MEDIUM jailbreak-003: Roleplay bypass
|
|
106
103
|
```
|
|
107
104
|
|
|
108
|
-
### `redprobe run`
|
|
105
|
+
### `uvx redprobe run`
|
|
109
106
|
|
|
110
107
|
Run probes against a model:
|
|
111
108
|
|
|
112
109
|
```bash
|
|
113
|
-
$ redprobe run probes/
|
|
110
|
+
$ uvx redprobe run probes/
|
|
114
111
|
Loaded 9 probes from 3 suite(s)
|
|
115
112
|
Target: http://localhost:1234/v1 model openai/gpt-oss-20b
|
|
116
113
|
|
|
@@ -149,7 +146,7 @@ Running suite: Jailbreak Attempts
|
|
|
149
146
|
#### Command Options
|
|
150
147
|
|
|
151
148
|
```bash
|
|
152
|
-
redprobe run probes/ \
|
|
149
|
+
uvx redprobe run probes/ \
|
|
153
150
|
--base-url http://localhost:1234/v1 \ # API endpoint
|
|
154
151
|
--model openai/gpt-oss-20b \ # Model to test
|
|
155
152
|
--api-key not-needed \ # API key (if required)
|
|
@@ -253,7 +250,7 @@ cases:
|
|
|
253
250
|
|
|
254
251
|
### Organizing Probes
|
|
255
252
|
|
|
256
|
-
You can split probes across multiple YAML files. When you point `redprobe run` at a directory, it loads all `.yaml` and `.yml` files:
|
|
253
|
+
You can split probes across multiple YAML files. When you point `uvx redprobe run` at a directory, it loads all `.yaml` and `.yml` files:
|
|
257
254
|
|
|
258
255
|
```
|
|
259
256
|
probes/
|
|
@@ -267,12 +264,12 @@ probes/
|
|
|
267
264
|
|
|
268
265
|
Run a specific file:
|
|
269
266
|
```bash
|
|
270
|
-
redprobe run probes/jailbreaks.yaml
|
|
267
|
+
uvx redprobe run probes/jailbreaks.yaml
|
|
271
268
|
```
|
|
272
269
|
|
|
273
270
|
Run all probes in a directory:
|
|
274
271
|
```bash
|
|
275
|
-
redprobe run probes/
|
|
272
|
+
uvx redprobe run probes/
|
|
276
273
|
```
|
|
277
274
|
|
|
278
275
|
## Understanding Results
|
|
@@ -296,7 +293,7 @@ A triggered probe does not necessarily mean the model is unsafe. It means the re
|
|
|
296
293
|
Use `--output` to export full results including model responses:
|
|
297
294
|
|
|
298
295
|
```bash
|
|
299
|
-
redprobe run probes/ --output results.json
|
|
296
|
+
uvx redprobe run probes/ --output results.json
|
|
300
297
|
```
|
|
301
298
|
|
|
302
299
|
The JSON includes timestamps, prompts, full responses, and matched patterns for each probe.
|
|
@@ -309,7 +306,7 @@ The JSON includes timestamps, prompts, full responses, and matched patterns for
|
|
|
309
306
|
# Start Ollama with a model
|
|
310
307
|
ollama serve
|
|
311
308
|
|
|
312
|
-
redprobe run probes/ \
|
|
309
|
+
uvx redprobe run probes/ \
|
|
313
310
|
--base-url http://localhost:11434/v1 \
|
|
314
311
|
--model llama2
|
|
315
312
|
```
|
|
@@ -317,7 +314,7 @@ redprobe run probes/ \
|
|
|
317
314
|
### OpenAI
|
|
318
315
|
|
|
319
316
|
```bash
|
|
320
|
-
redprobe run probes/ \
|
|
317
|
+
uvx redprobe run probes/ \
|
|
321
318
|
--base-url https://api.openai.com/v1 \
|
|
322
319
|
--model gpt-4o-mini \
|
|
323
320
|
--api-key $OPENAI_API_KEY
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
[project]
|
|
2
2
|
name = "redprobe"
|
|
3
|
-
version = "0.1.
|
|
3
|
+
version = "0.1.2"
|
|
4
4
|
description = "A defensive security tool for hardening AI systems. Define YAML-based test cases to systematically probe LLMs for jailbreaks, prompt injections, biases, harmful content generation, data leakage, and policy violations before attackers find them. Compatible with any OpenAI-style API endpoint."
|
|
5
5
|
readme = "README.md"
|
|
6
6
|
authors = [
|
|
@@ -30,11 +30,6 @@ test = [
|
|
|
30
30
|
"ipdb", # debugging
|
|
31
31
|
]
|
|
32
32
|
|
|
33
|
-
[project.urls]
|
|
34
|
-
bugs = "https://github.com/audreyfeldroy/redprobe/issues"
|
|
35
|
-
changelog = "https://github.com/audreyfeldroy/redprobe/blob/master/changelog.md"
|
|
36
|
-
homepage = "https://github.com/audreyfeldroy/redprobe"
|
|
37
|
-
|
|
38
33
|
[project.scripts]
|
|
39
34
|
redprobe = "redprobe.cli:app"
|
|
40
35
|
|
|
@@ -12,18 +12,22 @@ CONFIG_DIR = Path.home() / ".config" / "redprobe"
|
|
|
12
12
|
CONSENT_FILE = CONFIG_DIR / "consent.json"
|
|
13
13
|
|
|
14
14
|
# Version of the consent terms - bump this to require re-acknowledgment
|
|
15
|
-
CONSENT_VERSION = "1.
|
|
15
|
+
CONSENT_VERSION = "1.1"
|
|
16
16
|
|
|
17
17
|
TERMS_TEXT = """\
|
|
18
|
-
[bold]RedProbe[/bold] is a security
|
|
18
|
+
[bold]RedProbe[/bold] is a defensive security tool for hardening AI systems.
|
|
19
19
|
|
|
20
|
-
[bold yellow]
|
|
20
|
+
[bold yellow]You may only use this tool for:[/bold yellow]
|
|
21
|
+
• Systems you own or operate
|
|
22
|
+
• Systems you have written permission to test
|
|
23
|
+
• Research/educational contexts with appropriate oversight
|
|
21
24
|
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
+
[bold yellow]You may NOT use this tool to:[/bold yellow]
|
|
26
|
+
• Test systems without authorization
|
|
27
|
+
• Extract private data from systems you don't own
|
|
28
|
+
• Generate harmful content for distribution
|
|
25
29
|
|
|
26
|
-
|
|
30
|
+
By proceeding, you accept responsibility for your use of this tool.
|
|
27
31
|
"""
|
|
28
32
|
|
|
29
33
|
|
|
@@ -1,13 +1,10 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: redprobe
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.2
|
|
4
4
|
Summary: A defensive security tool for hardening AI systems. Define YAML-based test cases to systematically probe LLMs for jailbreaks, prompt injections, biases, harmful content generation, data leakage, and policy violations before attackers find them. Compatible with any OpenAI-style API endpoint.
|
|
5
5
|
Author-email: "Audrey M. Roy Greenfeld" <audrey@feldroy.com>
|
|
6
6
|
Maintainer-email: "Audrey M. Roy Greenfeld" <audrey@feldroy.com>
|
|
7
7
|
License: BUSL 1.1
|
|
8
|
-
Project-URL: bugs, https://github.com/audreyfeldroy/redprobe/issues
|
|
9
|
-
Project-URL: changelog, https://github.com/audreyfeldroy/redprobe/blob/master/changelog.md
|
|
10
|
-
Project-URL: homepage, https://github.com/audreyfeldroy/redprobe
|
|
11
8
|
Requires-Python: >=3.10
|
|
12
9
|
Description-Content-Type: text/markdown
|
|
13
10
|
License-File: LICENSE
|
|
@@ -32,19 +29,16 @@ A defensive security tool for hardening AI systems. Define YAML-based test cases
|
|
|
32
29
|
## Quick Start
|
|
33
30
|
|
|
34
31
|
```bash
|
|
35
|
-
# Run with uv (recommended)
|
|
36
|
-
uvx redprobe
|
|
37
|
-
|
|
38
32
|
# Generate sample probes
|
|
39
|
-
redprobe init
|
|
33
|
+
uvx redprobe init
|
|
40
34
|
|
|
41
35
|
# Run probes against a model
|
|
42
|
-
redprobe run probes/
|
|
36
|
+
uvx redprobe run probes/
|
|
43
37
|
```
|
|
44
38
|
|
|
45
39
|
## Prerequisites
|
|
46
40
|
|
|
47
|
-
RedProbe works with any OpenAI-compatible API. The default configuration targets [LM Studio](https://lmstudio.ai/) running locally.
|
|
41
|
+
RedProbe requires [uv](https://docs.astral.sh/uv/getting-started/installation/) and works with any OpenAI-compatible API. The default configuration targets [LM Studio](https://lmstudio.ai/) running locally.
|
|
48
42
|
|
|
49
43
|
### Setting up LM Studio
|
|
50
44
|
|
|
@@ -75,12 +69,12 @@ Do not use RedProbe to test systems without authorization, extract private data,
|
|
|
75
69
|
|
|
76
70
|
## Commands
|
|
77
71
|
|
|
78
|
-
### `redprobe init`
|
|
72
|
+
### `uvx redprobe init`
|
|
79
73
|
|
|
80
74
|
Create sample probe files to get started:
|
|
81
75
|
|
|
82
76
|
```bash
|
|
83
|
-
$ redprobe init
|
|
77
|
+
$ uvx redprobe init
|
|
84
78
|
Created sample probes in probes/
|
|
85
79
|
|
|
86
80
|
Files created:
|
|
@@ -89,21 +83,21 @@ Files created:
|
|
|
89
83
|
- harmful.yaml (3 probes)
|
|
90
84
|
|
|
91
85
|
Run probes with:
|
|
92
|
-
redprobe run probes
|
|
86
|
+
uvx redprobe run probes
|
|
93
87
|
```
|
|
94
88
|
|
|
95
89
|
You can specify a different directory:
|
|
96
90
|
|
|
97
91
|
```bash
|
|
98
|
-
redprobe init my-probes/
|
|
92
|
+
uvx redprobe init my-probes/
|
|
99
93
|
```
|
|
100
94
|
|
|
101
|
-
### `redprobe list-probes`
|
|
95
|
+
### `uvx redprobe list-probes`
|
|
102
96
|
|
|
103
97
|
Preview what probes will run without actually calling the API:
|
|
104
98
|
|
|
105
99
|
```bash
|
|
106
|
-
$ redprobe list-probes probes/
|
|
100
|
+
$ uvx redprobe list-probes probes/
|
|
107
101
|
Bias Detection
|
|
108
102
|
Probes for demographic and social biases in model responses
|
|
109
103
|
|
|
@@ -130,12 +124,12 @@ Jailbreak Attempts
|
|
|
130
124
|
MEDIUM jailbreak-003: Roleplay bypass
|
|
131
125
|
```
|
|
132
126
|
|
|
133
|
-
### `redprobe run`
|
|
127
|
+
### `uvx redprobe run`
|
|
134
128
|
|
|
135
129
|
Run probes against a model:
|
|
136
130
|
|
|
137
131
|
```bash
|
|
138
|
-
$ redprobe run probes/
|
|
132
|
+
$ uvx redprobe run probes/
|
|
139
133
|
Loaded 9 probes from 3 suite(s)
|
|
140
134
|
Target: http://localhost:1234/v1 model openai/gpt-oss-20b
|
|
141
135
|
|
|
@@ -174,7 +168,7 @@ Running suite: Jailbreak Attempts
|
|
|
174
168
|
#### Command Options
|
|
175
169
|
|
|
176
170
|
```bash
|
|
177
|
-
redprobe run probes/ \
|
|
171
|
+
uvx redprobe run probes/ \
|
|
178
172
|
--base-url http://localhost:1234/v1 \ # API endpoint
|
|
179
173
|
--model openai/gpt-oss-20b \ # Model to test
|
|
180
174
|
--api-key not-needed \ # API key (if required)
|
|
@@ -278,7 +272,7 @@ cases:
|
|
|
278
272
|
|
|
279
273
|
### Organizing Probes
|
|
280
274
|
|
|
281
|
-
You can split probes across multiple YAML files. When you point `redprobe run` at a directory, it loads all `.yaml` and `.yml` files:
|
|
275
|
+
You can split probes across multiple YAML files. When you point `uvx redprobe run` at a directory, it loads all `.yaml` and `.yml` files:
|
|
282
276
|
|
|
283
277
|
```
|
|
284
278
|
probes/
|
|
@@ -292,12 +286,12 @@ probes/
|
|
|
292
286
|
|
|
293
287
|
Run a specific file:
|
|
294
288
|
```bash
|
|
295
|
-
redprobe run probes/jailbreaks.yaml
|
|
289
|
+
uvx redprobe run probes/jailbreaks.yaml
|
|
296
290
|
```
|
|
297
291
|
|
|
298
292
|
Run all probes in a directory:
|
|
299
293
|
```bash
|
|
300
|
-
redprobe run probes/
|
|
294
|
+
uvx redprobe run probes/
|
|
301
295
|
```
|
|
302
296
|
|
|
303
297
|
## Understanding Results
|
|
@@ -321,7 +315,7 @@ A triggered probe does not necessarily mean the model is unsafe. It means the re
|
|
|
321
315
|
Use `--output` to export full results including model responses:
|
|
322
316
|
|
|
323
317
|
```bash
|
|
324
|
-
redprobe run probes/ --output results.json
|
|
318
|
+
uvx redprobe run probes/ --output results.json
|
|
325
319
|
```
|
|
326
320
|
|
|
327
321
|
The JSON includes timestamps, prompts, full responses, and matched patterns for each probe.
|
|
@@ -334,7 +328,7 @@ The JSON includes timestamps, prompts, full responses, and matched patterns for
|
|
|
334
328
|
# Start Ollama with a model
|
|
335
329
|
ollama serve
|
|
336
330
|
|
|
337
|
-
redprobe run probes/ \
|
|
331
|
+
uvx redprobe run probes/ \
|
|
338
332
|
--base-url http://localhost:11434/v1 \
|
|
339
333
|
--model llama2
|
|
340
334
|
```
|
|
@@ -342,7 +336,7 @@ redprobe run probes/ \
|
|
|
342
336
|
### OpenAI
|
|
343
337
|
|
|
344
338
|
```bash
|
|
345
|
-
redprobe run probes/ \
|
|
339
|
+
uvx redprobe run probes/ \
|
|
346
340
|
--base-url https://api.openai.com/v1 \
|
|
347
341
|
--model gpt-4o-mini \
|
|
348
342
|
--api-key $OPENAI_API_KEY
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|