@pickled-dev/cli 0.14.0 → 0.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +57 -7
- package/dist/index.js +179 -176
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -70,13 +70,37 @@ Default `terminal` format is plain text suited to CI logs. Use `--format markdow
|
|
|
70
70
|
|
|
71
71
|
Run agent scenarios against registered sources.
|
|
72
72
|
|
|
73
|
-
| Option | Description
|
|
74
|
-
| --------------------- |
|
|
75
|
-
| `--json` | Output as JSON
|
|
76
|
-
| `-o, --output <file>` | Save JSON report to file
|
|
77
|
-
| `-v, --verbose` | Show progress while scenarios run
|
|
78
|
-
| `-t, --threshold <n>` | Minimum score percent needed to pass
|
|
79
|
-
| `--target <name>` | Run only the named target (overrides matrix)
|
|
73
|
+
| Option | Description |
|
|
74
|
+
| --------------------- | ------------------------------------------------------------------- |
|
|
75
|
+
| `--json` | Output as JSON |
|
|
76
|
+
| `-o, --output <file>` | Save JSON report to file |
|
|
77
|
+
| `-v, --verbose` | Show progress while scenarios run |
|
|
78
|
+
| `-t, --threshold <n>` | Minimum score percent needed to pass |
|
|
79
|
+
| `--target <name>` | Run only the named target (overrides matrix) |
|
|
80
|
+
| `--scenario <name>` | Run only the named scenario (CI-matrix-friendly) |
|
|
81
|
+
| `--interface <name>` | Matrix cell filter: run only cells with this interface |
|
|
82
|
+
| `--source <name>` | Matrix cell filter: run only cells with this source id |
|
|
83
|
+
| `--toolset <name>` | Matrix cell filter: run only cells with this toolset name |
|
|
84
|
+
|
|
85
|
+
The cell filters work with `scenario.matrix` declarations. Designed for GitHub Actions matrix usage where each CI job runs one cell:
|
|
86
|
+
|
|
87
|
+
```yaml
|
|
88
|
+
# .github/workflows/pickled-matrix.yml
|
|
89
|
+
strategy:
|
|
90
|
+
matrix:
|
|
91
|
+
interface: [codex, claude_code]
|
|
92
|
+
source: [docs_site, readme]
|
|
93
|
+
toolset: [none]
|
|
94
|
+
steps:
|
|
95
|
+
- run: |
|
|
96
|
+
pickled check \
|
|
97
|
+
--interface "${{ matrix.interface }}" \
|
|
98
|
+
--source "${{ matrix.source }}" \
|
|
99
|
+
--toolset "${{ matrix.toolset }}" \
|
|
100
|
+
--output "pickled-report-${{ matrix.interface }}-${{ matrix.source }}-${{ matrix.toolset }}.json"
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Each job uploads one receipt; a later job can merge or compare them. Full-matrix runs without filters work too; they just produce one report covering every declared cell.
|
|
80
104
|
|
|
81
105
|
## Example Output
|
|
82
106
|
|
|
@@ -110,6 +134,32 @@ Review fired traps before trusting this surface.
|
|
|
110
134
|
| `Ungrounded` | No valid citations, or every citation is unknown. |
|
|
111
135
|
| `Error` | The target failed before Pickled could score the response. |
|
|
112
136
|
|
|
137
|
+
## Sources
|
|
138
|
+
|
|
139
|
+
Sources are what scenarios cite. Three loader types:
|
|
140
|
+
|
|
141
|
+
- **`file` (default)** - a path to one local file. The string form (`readme: ./README.md`) implicitly uses this.
|
|
142
|
+
- **`url`** - an `http(s)://` path. Fetched on every `pickled check` run.
|
|
143
|
+
- **`codebase`** - a glob expanded into one logical source whose content is every matched file concatenated with file-separator headers. Useful when you want the agent to answer from a directory of JSDoc, per-package agent docs, or examples.
|
|
144
|
+
|
|
145
|
+
Codebase sources are always explicit:
|
|
146
|
+
|
|
147
|
+
```yaml
|
|
148
|
+
docs:
|
|
149
|
+
sources:
|
|
150
|
+
readme: ./README.md # file (string form)
|
|
151
|
+
docs_site: https://example.com/docs.md # url (string form, http prefix)
|
|
152
|
+
jsdoc:
|
|
153
|
+
type: codebase
|
|
154
|
+
path: "packages/**/src/**/*.ts"
|
|
155
|
+
exclude: ["**/*.test.ts"] # codebase-only
|
|
156
|
+
maxBytes: 524288 # optional; default 256 KB soft cap
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
Codebase loader safety defaults: skips directories (`onlyFiles`), does not follow symlinks, rejects glob patterns containing `..` segments. Files are read in lexicographic order so the same config produces the same content for reproducible LLM calls. The audit's trap cross-reference scans each matched file individually so findings carry per-file `source_id:path:line`.
|
|
160
|
+
|
|
161
|
+
URL sources are NOT scanned by the audit's trap cross-reference in v1; they are fetched only during `pickled check`.
|
|
162
|
+
|
|
113
163
|
## Targets
|
|
114
164
|
|
|
115
165
|
Pickled ships three target shapes today. Each target is a distinct surface that exercises the agent differently; results are comparable but not identical.
|