orizu 0.1.4 → 0.1.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +220 -0
- package/package.json +1 -1
package/README.md
ADDED
|
@@ -0,0 +1,220 @@
|
|
|
1
|
+
# Orizu CLI
|
|
2
|
+
|
|
3
|
+
## What is Orizu
|
|
4
|
+
|
|
5
|
+
Orizu is a platform that helps you build continually learning agents and other LLM applications. It does so in a scientific, measurable manner by first helping you build evals and then helping you hill climb on them.
|
|
6
|
+
|
|
7
|
+
## How Orizu works
|
|
8
|
+
|
|
9
|
+
There are 3 key pillars to Orizu:
|
|
10
|
+
1. **Human evaluation:** We consider human evaluation to be the gold standard, particularly when the humans involved are people with deep context and understanding of the output being evaluated. This is often the cross-functional team behind a product (product managers, engineers, designers, content writers, researchers, etc) or external experts (lawyers, engineers, writers, etc). Any good evaluation system relies on the presence of such high quality judgment to compare to. We provide tools that simplify the process of collecting the feedback from this audience.
|
|
11
|
+
2. **Auto evaluation:** Ultimately, human experts can only review so much. As your data scales, you need to encode their knowledge, their opinions, their taste and judgment into auto-evaluations. We automate this for you using the data we help you collect above.
|
|
12
|
+
3. **Optimization:** We turn your evals into an asset for you by building an automated hill-climbing workflow. We've started with prompt optimization because multiple studies and our own experience with our customers has shown strong measurable lifts with even relatively small datasets. Over time, we'll introduce additional capabilities, including optimizing your skills and other contextual information, and even model weights if your model provider allows finetuning.
|
|
13
|
+
|
|
14
|
+
## CLI and Agent Skills
|
|
15
|
+
|
|
16
|
+
We've created a CLI for you to use so that you can perform any task that you can on our platform – from admin stuff like creating teams and projects to uploading datasets and assigning reviews – from your terminal. The obvious reason we did this is so that you can have your coding agents do this for you.
|
|
17
|
+
|
|
18
|
+
Note: the one task we do _not_ let you perform from your CLI is any reviews assigned to you. We want to ensure that the data being collected is from the intended human expert.
|
|
19
|
+
|
|
20
|
+
## Requirements
|
|
21
|
+
|
|
22
|
+
- [Node.js 20+](https://nodejs.org/en/download)
|
|
23
|
+
- Access to an Orizu web app/API
|
|
24
|
+
- A valid Orizu account
|
|
25
|
+
|
|
26
|
+
By default, the CLI talks to `https://orizu.ai`.
|
|
27
|
+
|
|
28
|
+
## Install
|
|
29
|
+
|
|
30
|
+
Install the CLI globally with your package manager:
|
|
31
|
+
|
|
32
|
+
| Package manager | Command |
|
|
33
|
+
| --- | --- |
|
|
34
|
+
| npm | `npm i -g orizu` |
|
|
35
|
+
| pnpm | `pnpm add -g orizu` |
|
|
36
|
+
| Bun | `bun add -g orizu` |
|
|
37
|
+
|
|
38
|
+
Install the companion coding agent skill using [Skills.sh](https://skills.sh).
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
npx skills add raveeshbhalla/orizu-cli --skill orizu-cli
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
## Authentication
|
|
45
|
+
|
|
46
|
+
**Sign in**
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
orizu login
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
This will open a browser tab for you to login with. You must either have an account on the [Orizu platform](https://orizu.ai) or it will help you create one.
|
|
53
|
+
|
|
54
|
+
**Check the signed-in user**
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
orizu whoami
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
**Sign out**
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
orizu logout
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
## Common Commands
|
|
67
|
+
|
|
68
|
+
<table>
|
|
69
|
+
<tr>
|
|
70
|
+
<th>Task</th>
|
|
71
|
+
<th>Command</th>
|
|
72
|
+
</tr>
|
|
73
|
+
<tr>
|
|
74
|
+
<td colspan="2"><strong>Teams</strong></td>
|
|
75
|
+
</tr>
|
|
76
|
+
<tr>
|
|
77
|
+
<td>List teams</td>
|
|
78
|
+
<td><code>orizu teams list</code></td>
|
|
79
|
+
</tr>
|
|
80
|
+
<tr>
|
|
81
|
+
<td>Create a team</td>
|
|
82
|
+
<td><code>orizu teams create --name "Ops Eval"</code></td>
|
|
83
|
+
</tr>
|
|
84
|
+
<tr>
|
|
85
|
+
<td>List team members</td>
|
|
86
|
+
<td><code>orizu teams members list --team ops-eval</code></td>
|
|
87
|
+
</tr>
|
|
88
|
+
<tr>
|
|
89
|
+
<td>Add a team member</td>
|
|
90
|
+
<td><code>orizu teams members add --email person@example.com --team ops-eval</code></td>
|
|
91
|
+
</tr>
|
|
92
|
+
<tr>
|
|
93
|
+
<td colspan="2"><strong>Projects</strong></td>
|
|
94
|
+
</tr>
|
|
95
|
+
<tr>
|
|
96
|
+
<td>List projects</td>
|
|
97
|
+
<td><code>orizu projects list --team ops-eval</code></td>
|
|
98
|
+
</tr>
|
|
99
|
+
<tr>
|
|
100
|
+
<td>Create a project</td>
|
|
101
|
+
<td><code>orizu projects create --name "Support QA" --team ops-eval</code></td>
|
|
102
|
+
</tr>
|
|
103
|
+
<tr>
|
|
104
|
+
<td colspan="2"><strong>Datasets</strong></td>
|
|
105
|
+
</tr>
|
|
106
|
+
<tr>
|
|
107
|
+
<td>Upload a dataset</td>
|
|
108
|
+
<td>
|
|
109
|
+
<pre><code>orizu datasets upload \
|
|
110
|
+
--project ops-eval/support-qa \
|
|
111
|
+
--file ./datasets/support.jsonl \
|
|
112
|
+
--name "Support Batch 1"</code></pre>
|
|
113
|
+
</td>
|
|
114
|
+
</tr>
|
|
115
|
+
<tr>
|
|
116
|
+
<td>Download a dataset</td>
|
|
117
|
+
<td><code>orizu datasets download --dataset <datasetId> --format jsonl --out ./dataset.jsonl</code></td>
|
|
118
|
+
</tr>
|
|
119
|
+
<tr>
|
|
120
|
+
<td>Append dataset rows</td>
|
|
121
|
+
<td><code>orizu datasets append --dataset <datasetId> --file ./datasets/support-extra.jsonl</code></td>
|
|
122
|
+
</tr>
|
|
123
|
+
<tr>
|
|
124
|
+
<td>Lock a dataset</td>
|
|
125
|
+
<td><code>orizu datasets lock --dataset <datasetId> --reason "Finalize for labeling"</code></td>
|
|
126
|
+
</tr>
|
|
127
|
+
<tr>
|
|
128
|
+
<td colspan="2"><strong>Apps</strong></td>
|
|
129
|
+
</tr>
|
|
130
|
+
<tr>
|
|
131
|
+
<td>List apps</td>
|
|
132
|
+
<td><code>orizu apps list --project ops-eval/support-qa</code></td>
|
|
133
|
+
</tr>
|
|
134
|
+
<tr>
|
|
135
|
+
<td>Create an app</td>
|
|
136
|
+
<td>
|
|
137
|
+
<pre><code>orizu apps create \
|
|
138
|
+
--project ops-eval/support-qa \
|
|
139
|
+
--name "Support Labeler" \
|
|
140
|
+
--dataset <datasetId> \
|
|
141
|
+
--file ./apps/SupportLabeler.tsx \
|
|
142
|
+
--input-schema ./schemas/support-input.json \
|
|
143
|
+
--output-schema ./schemas/support-output.json</code></pre>
|
|
144
|
+
</td>
|
|
145
|
+
</tr>
|
|
146
|
+
<tr>
|
|
147
|
+
<td>Update an app</td>
|
|
148
|
+
<td>
|
|
149
|
+
<pre><code>orizu apps update \
|
|
150
|
+
--app <appId> \
|
|
151
|
+
--file ./apps/SupportLabeler.tsx \
|
|
152
|
+
--input-schema ./schemas/support-input.json \
|
|
153
|
+
--output-schema ./schemas/support-output.json</code></pre>
|
|
154
|
+
</td>
|
|
155
|
+
</tr>
|
|
156
|
+
<tr>
|
|
157
|
+
<td>Link a dataset to an app</td>
|
|
158
|
+
<td><code>orizu apps link-dataset --app <appId> --dataset <datasetId></code></td>
|
|
159
|
+
</tr>
|
|
160
|
+
<tr>
|
|
161
|
+
<td colspan="2"><strong>Tasks</strong></td>
|
|
162
|
+
</tr>
|
|
163
|
+
<tr>
|
|
164
|
+
<td>Create a task</td>
|
|
165
|
+
<td>
|
|
166
|
+
<pre><code>orizu tasks create \
|
|
167
|
+
--project ops-eval/support-qa \
|
|
168
|
+
--dataset <datasetId> \
|
|
169
|
+
--app <appId> \
|
|
170
|
+
--title "Support QA Round 1" \
|
|
171
|
+
--assignees <userIdOrEmail1,userIdOrEmail2> \
|
|
172
|
+
--labels-per-item 2</code></pre>
|
|
173
|
+
</td>
|
|
174
|
+
</tr>
|
|
175
|
+
<tr>
|
|
176
|
+
<td>Check task status</td>
|
|
177
|
+
<td><code>orizu tasks status --task <taskId></code></td>
|
|
178
|
+
</tr>
|
|
179
|
+
<tr>
|
|
180
|
+
<td>Pause a task</td>
|
|
181
|
+
<td><code>orizu tasks pause --task <taskId></code></td>
|
|
182
|
+
</tr>
|
|
183
|
+
<tr>
|
|
184
|
+
<td>Resume a task</td>
|
|
185
|
+
<td><code>orizu tasks unpause --task <taskId></code></td>
|
|
186
|
+
</tr>
|
|
187
|
+
<tr>
|
|
188
|
+
<td>Export task results</td>
|
|
189
|
+
<td><code>orizu tasks export --task <taskId> --format csv --out ./support-round1.csv</code></td>
|
|
190
|
+
</tr>
|
|
191
|
+
</table>
|
|
192
|
+
|
|
193
|
+
## Supported Data Formats
|
|
194
|
+
|
|
195
|
+
Dataset upload, append, and edit-rows commands support:
|
|
196
|
+
|
|
197
|
+
- `.csv`
|
|
198
|
+
- `.json` files containing an array of objects
|
|
199
|
+
- `.jsonl` files containing one object per line
|
|
200
|
+
|
|
201
|
+
Task and dataset export support:
|
|
202
|
+
|
|
203
|
+
- `csv`
|
|
204
|
+
- `json`
|
|
205
|
+
- `jsonl`
|
|
206
|
+
|
|
207
|
+
## Interactive And Automated Usage
|
|
208
|
+
|
|
209
|
+
Many commands can prompt for missing team, project, app, dataset, or task selections in an interactive terminal. In scripts and CI, pass explicit flags instead:
|
|
210
|
+
|
|
211
|
+
```bash
|
|
212
|
+
orizu tasks export --task <taskId> --format jsonl --out ./labels.jsonl
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
Use `--json` on supported task and app commands when automation needs structured output.
|
|
216
|
+
|
|
217
|
+
## More Documentation
|
|
218
|
+
|
|
219
|
+
- [docs/cli.md](docs/cli.md): complete CLI guide and command reference
|
|
220
|
+
- [skills/orizu-cli/references/cli-reference.md](skills/orizu-cli/references/cli-reference.md): compact command matrix and end-to-end flows
|