redprobe 0.1.1__tar.gz → 0.1.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {redprobe-0.1.1 → redprobe-0.1.3}/PKG-INFO +22 -22
- {redprobe-0.1.1 → redprobe-0.1.3}/README.md +21 -21
- {redprobe-0.1.1 → redprobe-0.1.3}/pyproject.toml +1 -1
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe/consent.py +11 -7
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe.egg-info/PKG-INFO +22 -22
- {redprobe-0.1.1 → redprobe-0.1.3}/CONTRIBUTING.md +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/HISTORY.md +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/LICENSE +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/MANIFEST.in +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/docs/index.md +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/docs/installation.md +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/docs/usage.md +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/setup.cfg +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe/__init__.py +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe/__main__.py +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe/cli.py +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe/client.py +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe/probes.py +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe/redprobe.py +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe/reporter.py +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe/runner.py +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe/utils.py +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe.egg-info/SOURCES.txt +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe.egg-info/dependency_links.txt +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe.egg-info/entry_points.txt +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe.egg-info/requires.txt +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/src/redprobe.egg-info/top_level.txt +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/tests/__init__.py +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/tests/test_cli.py +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/tests/test_probes.py +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/tests/test_redprobe.py +0 -0
- {redprobe-0.1.1 → redprobe-0.1.3}/tests/test_runner.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: redprobe
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.3
|
|
4
4
|
Summary: A defensive security tool for hardening AI systems. Define YAML-based test cases to systematically probe LLMs for jailbreaks, prompt injections, biases, harmful content generation, data leakage, and policy violations before attackers find them. Compatible with any OpenAI-style API endpoint.
|
|
5
5
|
Author-email: "Audrey M. Roy Greenfeld" <audrey@feldroy.com>
|
|
6
6
|
Maintainer-email: "Audrey M. Roy Greenfeld" <audrey@feldroy.com>
|
|
@@ -29,19 +29,19 @@ A defensive security tool for hardening AI systems. Define YAML-based test cases
|
|
|
29
29
|
## Quick Start
|
|
30
30
|
|
|
31
31
|
```bash
|
|
32
|
-
# Run with uv (recommended)
|
|
33
|
-
uvx redprobe
|
|
34
|
-
|
|
35
32
|
# Generate sample probes
|
|
36
|
-
redprobe init
|
|
33
|
+
uvx redprobe init
|
|
37
34
|
|
|
38
35
|
# Run probes against a model
|
|
39
|
-
redprobe run probes/
|
|
36
|
+
uvx redprobe run probes/
|
|
37
|
+
|
|
38
|
+
# Upgrade to latest version
|
|
39
|
+
uvx redprobe@latest
|
|
40
40
|
```
|
|
41
41
|
|
|
42
42
|
## Prerequisites
|
|
43
43
|
|
|
44
|
-
RedProbe works with any OpenAI-compatible API. The default configuration targets [LM Studio](https://lmstudio.ai/) running locally.
|
|
44
|
+
RedProbe requires [uv](https://docs.astral.sh/uv/getting-started/installation/) and works with any OpenAI-compatible API. The default configuration targets [LM Studio](https://lmstudio.ai/) running locally.
|
|
45
45
|
|
|
46
46
|
### Setting up LM Studio
|
|
47
47
|
|
|
@@ -72,12 +72,12 @@ Do not use RedProbe to test systems without authorization, extract private data,
|
|
|
72
72
|
|
|
73
73
|
## Commands
|
|
74
74
|
|
|
75
|
-
### `redprobe init`
|
|
75
|
+
### `uvx redprobe init`
|
|
76
76
|
|
|
77
77
|
Create sample probe files to get started:
|
|
78
78
|
|
|
79
79
|
```bash
|
|
80
|
-
$ redprobe init
|
|
80
|
+
$ uvx redprobe init
|
|
81
81
|
Created sample probes in probes/
|
|
82
82
|
|
|
83
83
|
Files created:
|
|
@@ -86,21 +86,21 @@ Files created:
|
|
|
86
86
|
- harmful.yaml (3 probes)
|
|
87
87
|
|
|
88
88
|
Run probes with:
|
|
89
|
-
redprobe run probes
|
|
89
|
+
uvx redprobe run probes
|
|
90
90
|
```
|
|
91
91
|
|
|
92
92
|
You can specify a different directory:
|
|
93
93
|
|
|
94
94
|
```bash
|
|
95
|
-
redprobe init my-probes/
|
|
95
|
+
uvx redprobe init my-probes/
|
|
96
96
|
```
|
|
97
97
|
|
|
98
|
-
### `redprobe list-probes`
|
|
98
|
+
### `uvx redprobe list-probes`
|
|
99
99
|
|
|
100
100
|
Preview what probes will run without actually calling the API:
|
|
101
101
|
|
|
102
102
|
```bash
|
|
103
|
-
$ redprobe list-probes probes/
|
|
103
|
+
$ uvx redprobe list-probes probes/
|
|
104
104
|
Bias Detection
|
|
105
105
|
Probes for demographic and social biases in model responses
|
|
106
106
|
|
|
@@ -127,12 +127,12 @@ Jailbreak Attempts
|
|
|
127
127
|
MEDIUM jailbreak-003: Roleplay bypass
|
|
128
128
|
```
|
|
129
129
|
|
|
130
|
-
### `redprobe run`
|
|
130
|
+
### `uvx redprobe run`
|
|
131
131
|
|
|
132
132
|
Run probes against a model:
|
|
133
133
|
|
|
134
134
|
```bash
|
|
135
|
-
$ redprobe run probes/
|
|
135
|
+
$ uvx redprobe run probes/
|
|
136
136
|
Loaded 9 probes from 3 suite(s)
|
|
137
137
|
Target: http://localhost:1234/v1 model openai/gpt-oss-20b
|
|
138
138
|
|
|
@@ -171,7 +171,7 @@ Running suite: Jailbreak Attempts
|
|
|
171
171
|
#### Command Options
|
|
172
172
|
|
|
173
173
|
```bash
|
|
174
|
-
redprobe run probes/ \
|
|
174
|
+
uvx redprobe run probes/ \
|
|
175
175
|
--base-url http://localhost:1234/v1 \ # API endpoint
|
|
176
176
|
--model openai/gpt-oss-20b \ # Model to test
|
|
177
177
|
--api-key not-needed \ # API key (if required)
|
|
@@ -275,7 +275,7 @@ cases:
|
|
|
275
275
|
|
|
276
276
|
### Organizing Probes
|
|
277
277
|
|
|
278
|
-
You can split probes across multiple YAML files. When you point `redprobe run` at a directory, it loads all `.yaml` and `.yml` files:
|
|
278
|
+
You can split probes across multiple YAML files. When you point `uvx redprobe run` at a directory, it loads all `.yaml` and `.yml` files:
|
|
279
279
|
|
|
280
280
|
```
|
|
281
281
|
probes/
|
|
@@ -289,12 +289,12 @@ probes/
|
|
|
289
289
|
|
|
290
290
|
Run a specific file:
|
|
291
291
|
```bash
|
|
292
|
-
redprobe run probes/jailbreaks.yaml
|
|
292
|
+
uvx redprobe run probes/jailbreaks.yaml
|
|
293
293
|
```
|
|
294
294
|
|
|
295
295
|
Run all probes in a directory:
|
|
296
296
|
```bash
|
|
297
|
-
redprobe run probes/
|
|
297
|
+
uvx redprobe run probes/
|
|
298
298
|
```
|
|
299
299
|
|
|
300
300
|
## Understanding Results
|
|
@@ -318,7 +318,7 @@ A triggered probe does not necessarily mean the model is unsafe. It means the re
|
|
|
318
318
|
Use `--output` to export full results including model responses:
|
|
319
319
|
|
|
320
320
|
```bash
|
|
321
|
-
redprobe run probes/ --output results.json
|
|
321
|
+
uvx redprobe run probes/ --output results.json
|
|
322
322
|
```
|
|
323
323
|
|
|
324
324
|
The JSON includes timestamps, prompts, full responses, and matched patterns for each probe.
|
|
@@ -331,7 +331,7 @@ The JSON includes timestamps, prompts, full responses, and matched patterns for
|
|
|
331
331
|
# Start Ollama with a model
|
|
332
332
|
ollama serve
|
|
333
333
|
|
|
334
|
-
redprobe run probes/ \
|
|
334
|
+
uvx redprobe run probes/ \
|
|
335
335
|
--base-url http://localhost:11434/v1 \
|
|
336
336
|
--model llama2
|
|
337
337
|
```
|
|
@@ -339,7 +339,7 @@ redprobe run probes/ \
|
|
|
339
339
|
### OpenAI
|
|
340
340
|
|
|
341
341
|
```bash
|
|
342
|
-
redprobe run probes/ \
|
|
342
|
+
uvx redprobe run probes/ \
|
|
343
343
|
--base-url https://api.openai.com/v1 \
|
|
344
344
|
--model gpt-4o-mini \
|
|
345
345
|
--api-key $OPENAI_API_KEY
|
|
@@ -7,19 +7,19 @@ A defensive security tool for hardening AI systems. Define YAML-based test cases
|
|
|
7
7
|
## Quick Start
|
|
8
8
|
|
|
9
9
|
```bash
|
|
10
|
-
# Run with uv (recommended)
|
|
11
|
-
uvx redprobe
|
|
12
|
-
|
|
13
10
|
# Generate sample probes
|
|
14
|
-
redprobe init
|
|
11
|
+
uvx redprobe init
|
|
15
12
|
|
|
16
13
|
# Run probes against a model
|
|
17
|
-
redprobe run probes/
|
|
14
|
+
uvx redprobe run probes/
|
|
15
|
+
|
|
16
|
+
# Upgrade to latest version
|
|
17
|
+
uvx redprobe@latest
|
|
18
18
|
```
|
|
19
19
|
|
|
20
20
|
## Prerequisites
|
|
21
21
|
|
|
22
|
-
RedProbe works with any OpenAI-compatible API. The default configuration targets [LM Studio](https://lmstudio.ai/) running locally.
|
|
22
|
+
RedProbe requires [uv](https://docs.astral.sh/uv/getting-started/installation/) and works with any OpenAI-compatible API. The default configuration targets [LM Studio](https://lmstudio.ai/) running locally.
|
|
23
23
|
|
|
24
24
|
### Setting up LM Studio
|
|
25
25
|
|
|
@@ -50,12 +50,12 @@ Do not use RedProbe to test systems without authorization, extract private data,
|
|
|
50
50
|
|
|
51
51
|
## Commands
|
|
52
52
|
|
|
53
|
-
### `redprobe init`
|
|
53
|
+
### `uvx redprobe init`
|
|
54
54
|
|
|
55
55
|
Create sample probe files to get started:
|
|
56
56
|
|
|
57
57
|
```bash
|
|
58
|
-
$ redprobe init
|
|
58
|
+
$ uvx redprobe init
|
|
59
59
|
Created sample probes in probes/
|
|
60
60
|
|
|
61
61
|
Files created:
|
|
@@ -64,21 +64,21 @@ Files created:
|
|
|
64
64
|
- harmful.yaml (3 probes)
|
|
65
65
|
|
|
66
66
|
Run probes with:
|
|
67
|
-
redprobe run probes
|
|
67
|
+
uvx redprobe run probes
|
|
68
68
|
```
|
|
69
69
|
|
|
70
70
|
You can specify a different directory:
|
|
71
71
|
|
|
72
72
|
```bash
|
|
73
|
-
redprobe init my-probes/
|
|
73
|
+
uvx redprobe init my-probes/
|
|
74
74
|
```
|
|
75
75
|
|
|
76
|
-
### `redprobe list-probes`
|
|
76
|
+
### `uvx redprobe list-probes`
|
|
77
77
|
|
|
78
78
|
Preview what probes will run without actually calling the API:
|
|
79
79
|
|
|
80
80
|
```bash
|
|
81
|
-
$ redprobe list-probes probes/
|
|
81
|
+
$ uvx redprobe list-probes probes/
|
|
82
82
|
Bias Detection
|
|
83
83
|
Probes for demographic and social biases in model responses
|
|
84
84
|
|
|
@@ -105,12 +105,12 @@ Jailbreak Attempts
|
|
|
105
105
|
MEDIUM jailbreak-003: Roleplay bypass
|
|
106
106
|
```
|
|
107
107
|
|
|
108
|
-
### `redprobe run`
|
|
108
|
+
### `uvx redprobe run`
|
|
109
109
|
|
|
110
110
|
Run probes against a model:
|
|
111
111
|
|
|
112
112
|
```bash
|
|
113
|
-
$ redprobe run probes/
|
|
113
|
+
$ uvx redprobe run probes/
|
|
114
114
|
Loaded 9 probes from 3 suite(s)
|
|
115
115
|
Target: http://localhost:1234/v1 model openai/gpt-oss-20b
|
|
116
116
|
|
|
@@ -149,7 +149,7 @@ Running suite: Jailbreak Attempts
|
|
|
149
149
|
#### Command Options
|
|
150
150
|
|
|
151
151
|
```bash
|
|
152
|
-
redprobe run probes/ \
|
|
152
|
+
uvx redprobe run probes/ \
|
|
153
153
|
--base-url http://localhost:1234/v1 \ # API endpoint
|
|
154
154
|
--model openai/gpt-oss-20b \ # Model to test
|
|
155
155
|
--api-key not-needed \ # API key (if required)
|
|
@@ -253,7 +253,7 @@ cases:
|
|
|
253
253
|
|
|
254
254
|
### Organizing Probes
|
|
255
255
|
|
|
256
|
-
You can split probes across multiple YAML files. When you point `redprobe run` at a directory, it loads all `.yaml` and `.yml` files:
|
|
256
|
+
You can split probes across multiple YAML files. When you point `uvx redprobe run` at a directory, it loads all `.yaml` and `.yml` files:
|
|
257
257
|
|
|
258
258
|
```
|
|
259
259
|
probes/
|
|
@@ -267,12 +267,12 @@ probes/
|
|
|
267
267
|
|
|
268
268
|
Run a specific file:
|
|
269
269
|
```bash
|
|
270
|
-
redprobe run probes/jailbreaks.yaml
|
|
270
|
+
uvx redprobe run probes/jailbreaks.yaml
|
|
271
271
|
```
|
|
272
272
|
|
|
273
273
|
Run all probes in a directory:
|
|
274
274
|
```bash
|
|
275
|
-
redprobe run probes/
|
|
275
|
+
uvx redprobe run probes/
|
|
276
276
|
```
|
|
277
277
|
|
|
278
278
|
## Understanding Results
|
|
@@ -296,7 +296,7 @@ A triggered probe does not necessarily mean the model is unsafe. It means the re
|
|
|
296
296
|
Use `--output` to export full results including model responses:
|
|
297
297
|
|
|
298
298
|
```bash
|
|
299
|
-
redprobe run probes/ --output results.json
|
|
299
|
+
uvx redprobe run probes/ --output results.json
|
|
300
300
|
```
|
|
301
301
|
|
|
302
302
|
The JSON includes timestamps, prompts, full responses, and matched patterns for each probe.
|
|
@@ -309,7 +309,7 @@ The JSON includes timestamps, prompts, full responses, and matched patterns for
|
|
|
309
309
|
# Start Ollama with a model
|
|
310
310
|
ollama serve
|
|
311
311
|
|
|
312
|
-
redprobe run probes/ \
|
|
312
|
+
uvx redprobe run probes/ \
|
|
313
313
|
--base-url http://localhost:11434/v1 \
|
|
314
314
|
--model llama2
|
|
315
315
|
```
|
|
@@ -317,7 +317,7 @@ redprobe run probes/ \
|
|
|
317
317
|
### OpenAI
|
|
318
318
|
|
|
319
319
|
```bash
|
|
320
|
-
redprobe run probes/ \
|
|
320
|
+
uvx redprobe run probes/ \
|
|
321
321
|
--base-url https://api.openai.com/v1 \
|
|
322
322
|
--model gpt-4o-mini \
|
|
323
323
|
--api-key $OPENAI_API_KEY
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
[project]
|
|
2
2
|
name = "redprobe"
|
|
3
|
-
version = "0.1.
|
|
3
|
+
version = "0.1.3"
|
|
4
4
|
description = "A defensive security tool for hardening AI systems. Define YAML-based test cases to systematically probe LLMs for jailbreaks, prompt injections, biases, harmful content generation, data leakage, and policy violations before attackers find them. Compatible with any OpenAI-style API endpoint."
|
|
5
5
|
readme = "README.md"
|
|
6
6
|
authors = [
|
|
@@ -12,18 +12,22 @@ CONFIG_DIR = Path.home() / ".config" / "redprobe"
|
|
|
12
12
|
CONSENT_FILE = CONFIG_DIR / "consent.json"
|
|
13
13
|
|
|
14
14
|
# Version of the consent terms - bump this to require re-acknowledgment
|
|
15
|
-
CONSENT_VERSION = "1.
|
|
15
|
+
CONSENT_VERSION = "1.1"
|
|
16
16
|
|
|
17
17
|
TERMS_TEXT = """\
|
|
18
|
-
[bold]RedProbe[/bold] is a security
|
|
18
|
+
[bold]RedProbe[/bold] is a defensive security tool for hardening AI systems.
|
|
19
19
|
|
|
20
|
-
[bold yellow]
|
|
20
|
+
[bold yellow]You may only use this tool for:[/bold yellow]
|
|
21
|
+
• Systems you own or operate
|
|
22
|
+
• Systems you have written permission to test
|
|
23
|
+
• Research/educational contexts with appropriate oversight
|
|
21
24
|
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
+
[bold yellow]You may NOT use this tool to:[/bold yellow]
|
|
26
|
+
• Test systems without authorization
|
|
27
|
+
• Extract private data from systems you don't own
|
|
28
|
+
• Generate harmful content for distribution
|
|
25
29
|
|
|
26
|
-
|
|
30
|
+
By proceeding, you accept responsibility for your use of this tool.
|
|
27
31
|
"""
|
|
28
32
|
|
|
29
33
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: redprobe
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.3
|
|
4
4
|
Summary: A defensive security tool for hardening AI systems. Define YAML-based test cases to systematically probe LLMs for jailbreaks, prompt injections, biases, harmful content generation, data leakage, and policy violations before attackers find them. Compatible with any OpenAI-style API endpoint.
|
|
5
5
|
Author-email: "Audrey M. Roy Greenfeld" <audrey@feldroy.com>
|
|
6
6
|
Maintainer-email: "Audrey M. Roy Greenfeld" <audrey@feldroy.com>
|
|
@@ -29,19 +29,19 @@ A defensive security tool for hardening AI systems. Define YAML-based test cases
|
|
|
29
29
|
## Quick Start
|
|
30
30
|
|
|
31
31
|
```bash
|
|
32
|
-
# Run with uv (recommended)
|
|
33
|
-
uvx redprobe
|
|
34
|
-
|
|
35
32
|
# Generate sample probes
|
|
36
|
-
redprobe init
|
|
33
|
+
uvx redprobe init
|
|
37
34
|
|
|
38
35
|
# Run probes against a model
|
|
39
|
-
redprobe run probes/
|
|
36
|
+
uvx redprobe run probes/
|
|
37
|
+
|
|
38
|
+
# Upgrade to latest version
|
|
39
|
+
uvx redprobe@latest
|
|
40
40
|
```
|
|
41
41
|
|
|
42
42
|
## Prerequisites
|
|
43
43
|
|
|
44
|
-
RedProbe works with any OpenAI-compatible API. The default configuration targets [LM Studio](https://lmstudio.ai/) running locally.
|
|
44
|
+
RedProbe requires [uv](https://docs.astral.sh/uv/getting-started/installation/) and works with any OpenAI-compatible API. The default configuration targets [LM Studio](https://lmstudio.ai/) running locally.
|
|
45
45
|
|
|
46
46
|
### Setting up LM Studio
|
|
47
47
|
|
|
@@ -72,12 +72,12 @@ Do not use RedProbe to test systems without authorization, extract private data,
|
|
|
72
72
|
|
|
73
73
|
## Commands
|
|
74
74
|
|
|
75
|
-
### `redprobe init`
|
|
75
|
+
### `uvx redprobe init`
|
|
76
76
|
|
|
77
77
|
Create sample probe files to get started:
|
|
78
78
|
|
|
79
79
|
```bash
|
|
80
|
-
$ redprobe init
|
|
80
|
+
$ uvx redprobe init
|
|
81
81
|
Created sample probes in probes/
|
|
82
82
|
|
|
83
83
|
Files created:
|
|
@@ -86,21 +86,21 @@ Files created:
|
|
|
86
86
|
- harmful.yaml (3 probes)
|
|
87
87
|
|
|
88
88
|
Run probes with:
|
|
89
|
-
redprobe run probes
|
|
89
|
+
uvx redprobe run probes
|
|
90
90
|
```
|
|
91
91
|
|
|
92
92
|
You can specify a different directory:
|
|
93
93
|
|
|
94
94
|
```bash
|
|
95
|
-
redprobe init my-probes/
|
|
95
|
+
uvx redprobe init my-probes/
|
|
96
96
|
```
|
|
97
97
|
|
|
98
|
-
### `redprobe list-probes`
|
|
98
|
+
### `uvx redprobe list-probes`
|
|
99
99
|
|
|
100
100
|
Preview what probes will run without actually calling the API:
|
|
101
101
|
|
|
102
102
|
```bash
|
|
103
|
-
$ redprobe list-probes probes/
|
|
103
|
+
$ uvx redprobe list-probes probes/
|
|
104
104
|
Bias Detection
|
|
105
105
|
Probes for demographic and social biases in model responses
|
|
106
106
|
|
|
@@ -127,12 +127,12 @@ Jailbreak Attempts
|
|
|
127
127
|
MEDIUM jailbreak-003: Roleplay bypass
|
|
128
128
|
```
|
|
129
129
|
|
|
130
|
-
### `redprobe run`
|
|
130
|
+
### `uvx redprobe run`
|
|
131
131
|
|
|
132
132
|
Run probes against a model:
|
|
133
133
|
|
|
134
134
|
```bash
|
|
135
|
-
$ redprobe run probes/
|
|
135
|
+
$ uvx redprobe run probes/
|
|
136
136
|
Loaded 9 probes from 3 suite(s)
|
|
137
137
|
Target: http://localhost:1234/v1 model openai/gpt-oss-20b
|
|
138
138
|
|
|
@@ -171,7 +171,7 @@ Running suite: Jailbreak Attempts
|
|
|
171
171
|
#### Command Options
|
|
172
172
|
|
|
173
173
|
```bash
|
|
174
|
-
redprobe run probes/ \
|
|
174
|
+
uvx redprobe run probes/ \
|
|
175
175
|
--base-url http://localhost:1234/v1 \ # API endpoint
|
|
176
176
|
--model openai/gpt-oss-20b \ # Model to test
|
|
177
177
|
--api-key not-needed \ # API key (if required)
|
|
@@ -275,7 +275,7 @@ cases:
|
|
|
275
275
|
|
|
276
276
|
### Organizing Probes
|
|
277
277
|
|
|
278
|
-
You can split probes across multiple YAML files. When you point `redprobe run` at a directory, it loads all `.yaml` and `.yml` files:
|
|
278
|
+
You can split probes across multiple YAML files. When you point `uvx redprobe run` at a directory, it loads all `.yaml` and `.yml` files:
|
|
279
279
|
|
|
280
280
|
```
|
|
281
281
|
probes/
|
|
@@ -289,12 +289,12 @@ probes/
|
|
|
289
289
|
|
|
290
290
|
Run a specific file:
|
|
291
291
|
```bash
|
|
292
|
-
redprobe run probes/jailbreaks.yaml
|
|
292
|
+
uvx redprobe run probes/jailbreaks.yaml
|
|
293
293
|
```
|
|
294
294
|
|
|
295
295
|
Run all probes in a directory:
|
|
296
296
|
```bash
|
|
297
|
-
redprobe run probes/
|
|
297
|
+
uvx redprobe run probes/
|
|
298
298
|
```
|
|
299
299
|
|
|
300
300
|
## Understanding Results
|
|
@@ -318,7 +318,7 @@ A triggered probe does not necessarily mean the model is unsafe. It means the re
|
|
|
318
318
|
Use `--output` to export full results including model responses:
|
|
319
319
|
|
|
320
320
|
```bash
|
|
321
|
-
redprobe run probes/ --output results.json
|
|
321
|
+
uvx redprobe run probes/ --output results.json
|
|
322
322
|
```
|
|
323
323
|
|
|
324
324
|
The JSON includes timestamps, prompts, full responses, and matched patterns for each probe.
|
|
@@ -331,7 +331,7 @@ The JSON includes timestamps, prompts, full responses, and matched patterns for
|
|
|
331
331
|
# Start Ollama with a model
|
|
332
332
|
ollama serve
|
|
333
333
|
|
|
334
|
-
redprobe run probes/ \
|
|
334
|
+
uvx redprobe run probes/ \
|
|
335
335
|
--base-url http://localhost:11434/v1 \
|
|
336
336
|
--model llama2
|
|
337
337
|
```
|
|
@@ -339,7 +339,7 @@ redprobe run probes/ \
|
|
|
339
339
|
### OpenAI
|
|
340
340
|
|
|
341
341
|
```bash
|
|
342
|
-
redprobe run probes/ \
|
|
342
|
+
uvx redprobe run probes/ \
|
|
343
343
|
--base-url https://api.openai.com/v1 \
|
|
344
344
|
--model gpt-4o-mini \
|
|
345
345
|
--api-key $OPENAI_API_KEY
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|