simula-gen 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- simula_gen-0.1.0/LICENSE +21 -0
- simula_gen-0.1.0/PKG-INFO +424 -0
- simula_gen-0.1.0/README.md +400 -0
- simula_gen-0.1.0/pyproject.toml +37 -0
- simula_gen-0.1.0/setup.cfg +4 -0
- simula_gen-0.1.0/simula/__init__.py +5 -0
- simula_gen-0.1.0/simula/cli.py +120 -0
- simula_gen-0.1.0/simula/config.py +80 -0
- simula_gen-0.1.0/simula/console.py +213 -0
- simula_gen-0.1.0/simula/data_models.py +295 -0
- simula_gen-0.1.0/simula/diversity.py +115 -0
- simula_gen-0.1.0/simula/evaluate.py +345 -0
- simula_gen-0.1.0/simula/generate.py +440 -0
- simula_gen-0.1.0/simula/models.py +225 -0
- simula_gen-0.1.0/simula/prompts.py +418 -0
- simula_gen-0.1.0/simula/taxonomy.py +379 -0
- simula_gen-0.1.0/simula/utils.py +181 -0
- simula_gen-0.1.0/simula_gen.egg-info/PKG-INFO +424 -0
- simula_gen-0.1.0/simula_gen.egg-info/SOURCES.txt +28 -0
- simula_gen-0.1.0/simula_gen.egg-info/dependency_links.txt +1 -0
- simula_gen-0.1.0/simula_gen.egg-info/entry_points.txt +2 -0
- simula_gen-0.1.0/simula_gen.egg-info/requires.txt +16 -0
- simula_gen-0.1.0/simula_gen.egg-info/top_level.txt +1 -0
- simula_gen-0.1.0/tests/test_config.py +341 -0
- simula_gen-0.1.0/tests/test_diversity.py +42 -0
- simula_gen-0.1.0/tests/test_evaluate.py +78 -0
- simula_gen-0.1.0/tests/test_models.py +60 -0
- simula_gen-0.1.0/tests/test_pipeline.py +420 -0
- simula_gen-0.1.0/tests/test_taxonomy_sampling.py +111 -0
- simula_gen-0.1.0/tests/test_utils.py +31 -0
simula_gen-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Mercity AI
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,424 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: simula-gen
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Schema-driven synthetic data generator inspired by reasoning-first taxonomy coverage.
|
|
5
|
+
License: MIT
|
|
6
|
+
Requires-Python: >=3.11
|
|
7
|
+
Description-Content-Type: text/markdown
|
|
8
|
+
License-File: LICENSE
|
|
9
|
+
Requires-Dist: jsonschema>=4.22.0
|
|
10
|
+
Requires-Dist: openai>=1.50.0
|
|
11
|
+
Requires-Dist: pydantic>=2.7.0
|
|
12
|
+
Requires-Dist: python-dotenv>=1.0.0
|
|
13
|
+
Requires-Dist: PyYAML>=6.0.1
|
|
14
|
+
Requires-Dist: rich>=13.0.0
|
|
15
|
+
Provides-Extra: diversity
|
|
16
|
+
Requires-Dist: numpy>=1.26.0; extra == "diversity"
|
|
17
|
+
Requires-Dist: scikit-learn>=1.4.0; extra == "diversity"
|
|
18
|
+
Requires-Dist: sentence-transformers>=2.7.0; extra == "diversity"
|
|
19
|
+
Provides-Extra: dev
|
|
20
|
+
Requires-Dist: simula-gen[diversity]; extra == "dev"
|
|
21
|
+
Requires-Dist: pytest>=8.0.0; extra == "dev"
|
|
22
|
+
Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
|
|
23
|
+
Dynamic: license-file
|
|
24
|
+
|
|
25
|
+
<p align="center">
|
|
26
|
+
<img src="assets/hero.png" alt="Simula" width="640">
|
|
27
|
+
</p>
|
|
28
|
+
|
|
29
|
+
<p align="center">
|
|
30
|
+
<img alt="Python" src="https://img.shields.io/badge/python-%E2%89%A53.11-blue">
|
|
31
|
+
<a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-green"></a>
|
|
32
|
+
<img alt="Version" src="https://img.shields.io/badge/version-0.1.0-blue">
|
|
33
|
+
<img alt="Status" src="https://img.shields.io/badge/status-pilot-orange">
|
|
34
|
+
<a href="https://arxiv.org/abs/2603.29791"><img alt="Paper" src="https://img.shields.io/badge/paper-arXiv%3A2603.29791-b31b1b"></a>
|
|
35
|
+
<a href="https://github.com/Mercity-AI"><img alt="Built by Mercity" src="https://img.shields.io/badge/built%20by-Mercity-1f6feb"></a>
|
|
36
|
+
</p>
|
|
37
|
+
|
|
38
|
+
# simula
|
|
39
|
+
|
|
40
|
+
> 📖 **[Concepts & Configuration](DOCS.md)** — the pipeline, the model roles, and every config knob, with diagrams.
|
|
41
|
+
|
|
42
|
+
`simula` is a small CLI-first framework for generating schema-shaped or free-text synthetic datasets with taxonomy-guided coverage. It is inspired by the Simula-style workflow: map the conceptual space first, sample from that space, generate records, critique them, preserve lineage, and then trim/evaluate the result.
|
|
43
|
+
|
|
44
|
+
The current implementation is intentionally lean. It uses JSON/YAML files, OpenAI-compatible chat endpoints, JSON Schema validation, and human-inspectable artifacts.
|
|
45
|
+
|
|
46
|
+
## What It Does
|
|
47
|
+
|
|
48
|
+
- Builds taxonomies from a dataset description and optional user-provided factors.
|
|
49
|
+
- Generates sampling strategies over taxonomy branches.
|
|
50
|
+
- Samples taxonomy mixes and turns them into meta-prompts.
|
|
51
|
+
- Generates JSON records matching a user-defined schema, or direct free-form text when `schema: null`.
|
|
52
|
+
- Validates records with a JSON Schema subset.
|
|
53
|
+
- Runs a simple critic/refine loop.
|
|
54
|
+
- Writes every model response to `llm_calls.jsonl` as soon as it returns.
|
|
55
|
+
- Writes raw, accepted, and final datasets as JSONL.
|
|
56
|
+
- Supports n-gram dedupe, coverage reports, optional reassignment coverage, diversity metrics, and optional complexity scoring.
|
|
57
|
+
|
|
58
|
+
## Install
|
|
59
|
+
|
|
60
|
+
From the repository root:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
python -m pip install -e ".[dev]"
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
The base install is light. The optional embedding-based **diversity** metric needs heavier packages
|
|
67
|
+
(numpy, scikit-learn, sentence-transformers / PyTorch); install them only if you enable it:
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
python -m pip install -e ".[diversity]"
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
You can also run the CLI without installing:
|
|
74
|
+
|
|
75
|
+
```bash
|
|
76
|
+
python -m simula.cli validate examples/basic_qa.yaml
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
## Quick Start
|
|
80
|
+
|
|
81
|
+
Run the fake-model smoke test. This does not call any external API:
|
|
82
|
+
|
|
83
|
+
```bash
|
|
84
|
+
python -m simula.cli run examples/basic_qa.yaml
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
Artifacts are written to:
|
|
88
|
+
|
|
89
|
+
```text
|
|
90
|
+
runs/basic_qa/
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
For a real OpenRouter/OpenAI-compatible run, put the key named by `provider.api_key_env` in a
|
|
94
|
+
gitignored `.env` file at the project root:
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
echo 'OPENROUTER_API_KEY=sk-or-...' > .env # gitignored; the only source of API keys
|
|
98
|
+
python -m simula.cli taxonomy examples/query_extraction_gemini.yaml
|
|
99
|
+
python -m simula.cli generate examples/query_extraction_gemini.yaml
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
Key resolution: the project-root `.env` file is read directly (via `dotenv_values`) and is the
|
|
103
|
+
**only** source of API keys — a shell-exported variable is deliberately ignored, so the key always
|
|
104
|
+
comes from the file you can see. A missing key warns at `validate` and fails real model calls. Never
|
|
105
|
+
commit real keys.
|
|
106
|
+
|
|
107
|
+
The query extraction example currently uses:
|
|
108
|
+
|
|
109
|
+
```text
|
|
110
|
+
google/gemini-3-flash-preview
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
through the OpenRouter-compatible API.
|
|
114
|
+
|
|
115
|
+
## CLI Commands
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
simula validate CONFIG.yaml
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
Validates the YAML config, JSON Schema subset, model role config, and output path. It makes no model calls.
|
|
122
|
+
|
|
123
|
+
```bash
|
|
124
|
+
simula taxonomy CONFIG.yaml
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
Builds `taxonomy.json` and, depending on `taxonomy.review_mode`, either accepts it automatically or lets the user review it.
|
|
128
|
+
|
|
129
|
+
```bash
|
|
130
|
+
simula generate CONFIG.yaml
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
Loads or builds the taxonomy, builds strategies and meta prompts if missing, generates data, validates records, runs critic/refine, and writes dataset artifacts. It does not run evaluation.
|
|
134
|
+
|
|
135
|
+
Pass `--stop-after {taxonomy,strategies,meta_prompts}` (or set `generation.stop_after` in the
|
|
136
|
+
config) to halt once that stage's artifact is written. Edit the artifact, then rerun: existing
|
|
137
|
+
artifacts are reused, so the run picks up where it stopped. `--stop-after none` overrides a
|
|
138
|
+
config-set stop for one invocation. To regenerate a stage instead, delete its file
|
|
139
|
+
(`taxonomy.json`, `strategies.json`, `meta_prompts.jsonl`) and rerun.
|
|
140
|
+
|
|
141
|
+
```bash
|
|
142
|
+
simula evaluate CONFIG.yaml
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
Runs dedupe, coverage, and optional complexity scoring. It reads `dataset.final.jsonl` and writes
|
|
146
|
+
the deduped/decontaminated result to a separate `dataset.evaluated.jsonl` (it never rewrites
|
|
147
|
+
`dataset.final.jsonl`), plus `eval_report.json`.
|
|
148
|
+
|
|
149
|
+
```bash
|
|
150
|
+
simula run CONFIG.yaml
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
Runs taxonomy, generation, and evaluation end to end.
|
|
154
|
+
|
|
155
|
+
## Config Overview
|
|
156
|
+
|
|
157
|
+
Each run is controlled by a single YAML file. **`examples/template.yaml`** is a copy-me skeleton with
|
|
158
|
+
every key at its default, and **`CONFIG.md`** is the full per-field reference. Only two things are
|
|
159
|
+
required — a non-empty `description` and a `model` id for each of the three roles; everything else has
|
|
160
|
+
a default, so the minimum viable config is small:
|
|
161
|
+
|
|
162
|
+
```yaml
|
|
163
|
+
description: "Describe the dataset to generate."
|
|
164
|
+
|
|
165
|
+
schema: # omit or set null for free-text mode
|
|
166
|
+
type: object
|
|
167
|
+
required: ["input", "output"]
|
|
168
|
+
properties:
|
|
169
|
+
input: { type: string }
|
|
170
|
+
output: { type: string }
|
|
171
|
+
|
|
172
|
+
provider:
|
|
173
|
+
base_url: "https://openrouter.ai/api/v1"
|
|
174
|
+
api_key_env: "OPENROUTER_API_KEY" # variable name read from the project-root .env
|
|
175
|
+
|
|
176
|
+
models:
|
|
177
|
+
strategic: { model: "google/gemini-3-flash-preview" }
|
|
178
|
+
bulk: { model: "google/gemini-3-flash-preview" }
|
|
179
|
+
critic: { model: "google/gemini-3-flash-preview" }
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
Copy `examples/template.yaml` for the full set of knobs (`project`, `taxonomy`, `strategy`,
|
|
183
|
+
`sampling`, `generation`, `evaluation`); the sections below explain the ones worth steering by hand.
|
|
184
|
+
|
|
185
|
+
### Model Roles
|
|
186
|
+
|
|
187
|
+
- `strategic`: factor discovery, taxonomy expansion, strategy generation.
|
|
188
|
+
- `bulk`: meta-prompt generation, complexification, record generation, repair, refinement.
|
|
189
|
+
- `critic`: semantic critique and optional complexity scoring.
|
|
190
|
+
|
|
191
|
+
All model roles use the same OpenAI-compatible chat completions interface. A role can use `"model": "fake"` for deterministic local tests.
|
|
192
|
+
|
|
193
|
+
### Prompt Overrides
|
|
194
|
+
|
|
195
|
+
Built-in prompt defaults live in `simula/prompts.py`. To override them for one run, point config at a Python module:
|
|
196
|
+
|
|
197
|
+
```yaml
|
|
198
|
+
prompts:
|
|
199
|
+
module: "config/prompts.py"
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
Paths are resolved relative to the YAML file. The module may override any subset of the built-in prompt functions, and missing functions fall back to the defaults. It may also override `SYSTEM_JSON` and `SYSTEM_TEXT`.
|
|
203
|
+
|
|
204
|
+
```python
|
|
205
|
+
SYSTEM_JSON = "Return valid JSON only."
|
|
206
|
+
|
|
207
|
+
|
|
208
|
+
def strategy_prompt(description, taxonomy, guidance=None):
|
|
209
|
+
return f"""
|
|
210
|
+
Dataset description:
|
|
211
|
+
{description}
|
|
212
|
+
|
|
213
|
+
Taxonomy:
|
|
214
|
+
{taxonomy}
|
|
215
|
+
|
|
216
|
+
Create only combinations that make semantic sense for this dataset.
|
|
217
|
+
Return JSON with a strategies array.
|
|
218
|
+
""".strip()
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
Override functions must keep the same parameter names as their built-in counterparts in `simula/prompts.py`. `simula validate` imports the module and rejects missing files, import failures, non-string system prompts, and incompatible function signatures before any model call runs.
|
|
222
|
+
|
|
223
|
+
### Strategy Guidance
|
|
224
|
+
|
|
225
|
+
Strategies decide which taxonomy branches combine and how often each combination is sampled. To steer that without writing a prompt module, set free-text `strategy.guidance`:
|
|
226
|
+
|
|
227
|
+
```yaml
|
|
228
|
+
strategy:
|
|
229
|
+
guidance: |
|
|
230
|
+
- Make billing + calm + simple the most common combination.
|
|
231
|
+
- Never combine enterprise_sso with mobile_app; they don't coexist.
|
|
232
|
+
- Keep a dedicated strategy for furious + needs_escalation, even though it is rare.
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
The guidance is woven into the strategy prompt, so the generated `strategies.json` reflects your intent (root combinations and weights) before any bulk generation runs. Guidance is a nudge interpreted by the strategic model, not a hard constraint; for guarantees, edit `strategies.json` directly or override `strategy_prompt`. When unset (`null`), the built-in prompt is used unchanged.
|
|
236
|
+
|
|
237
|
+
### Sampling
|
|
238
|
+
|
|
239
|
+
Decoding params resolve in three layers, last one wins: built-in defaults (`temperature: 0.7`, `max_tokens: 32768`) → per-role config under `models.<role>` → per-task overrides under `sampling.tasks`. Each task is handled by exactly one role, so naming a task is enough — no role needs to be specified.
|
|
240
|
+
|
|
241
|
+
```yaml
|
|
242
|
+
sampling:
|
|
243
|
+
tasks:
|
|
244
|
+
generate: {temperature: 1.1, top_p: 0.95, min_p: 0.05}
|
|
245
|
+
repair: {temperature: 0.0}
|
|
246
|
+
meta_prompt: {temperature: 1.1}
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
Valid task names are the `TaskType` values: `factor_discovery`, `node_expansion`, `taxonomy_critic`, `level_plan`, `strategy`, `meta_prompt`, `complexify`, `generate`, `repair`, `semantic_critic`, `refine`, `complexity_score`, `node_assign`.
|
|
250
|
+
|
|
251
|
+
OpenAI-compatible params (`temperature`, `top_p`, `max_tokens`, `frequency_penalty`, `presence_penalty`, `stop`, `seed`) are sent as top-level call kwargs. Anything else (`min_p`, `top_k`, `repetition_penalty`, …) is passed through `extra_body` so provider-specific knobs work without lock-in. The resolved params are recorded per call in `llm_calls.jsonl`.
|
|
252
|
+
|
|
253
|
+
Connection lives on `provider` (`base_url`, `api_key_env`, `timeout_seconds`), shared by all roles;
|
|
254
|
+
the non-decoding role keys (`model`, `extra_body`) are never treated as decoding params.
|
|
255
|
+
|
|
256
|
+
**Per-request timeout.** Each real call uses a default timeout of **180 seconds** so a hung or
|
|
257
|
+
rate-limited provider connection cannot stall a worker for the SDK default (~600s). Override with
|
|
258
|
+
`provider.timeout_seconds`.
|
|
259
|
+
|
|
260
|
+
**Reasoning models.** There is no automatic model-id detection. If a model emits hidden reasoning
|
|
261
|
+
tokens (e.g. DeepSeek R1/V4, OpenAI o-series) and you want them excluded from output and your
|
|
262
|
+
`max_tokens` budget, set it explicitly per role:
|
|
263
|
+
|
|
264
|
+
```yaml
|
|
265
|
+
models:
|
|
266
|
+
bulk:
|
|
267
|
+
model: "deepseek/deepseek-v4-flash"
|
|
268
|
+
max_tokens: 16384
|
|
269
|
+
extra_body: {reasoning: {effort: low, exclude: true}}
|
|
270
|
+
```
|
|
271
|
+
|
|
272
|
+
`simula validate` rejects unknown task names and non-numeric values before any model call runs. With no `sampling` block, behavior is unchanged except the larger default `max_tokens`. That 32K default is batteries-included: roles without an explicit `max_tokens` can now emit much larger (and pricier) completions than before — set `models.<role>.max_tokens` or a per-task `max_tokens` to cap it.
|
|
273
|
+
|
|
274
|
+
### Schema Support
|
|
275
|
+
|
|
276
|
+
The MVP supports a practical JSON Schema subset:
|
|
277
|
+
|
|
278
|
+
- `object`
|
|
279
|
+
- `string`
|
|
280
|
+
- `number`
|
|
281
|
+
- `integer`
|
|
282
|
+
- `boolean`
|
|
283
|
+
- `array`
|
|
284
|
+
- `enum`
|
|
285
|
+
- `required`
|
|
286
|
+
- nested objects and arrays
|
|
287
|
+
|
|
288
|
+
Every generated record must validate against the schema before it can be accepted.
|
|
289
|
+
|
|
290
|
+
Set `schema: null` or omit `schema` to use free-text mode. In that mode `record` is a string, JSON repair is skipped, and the critic still returns a JSON verdict.
|
|
291
|
+
|
|
292
|
+
## Artifacts
|
|
293
|
+
|
|
294
|
+
Each run writes human-inspectable files under `project.output_dir`:
|
|
295
|
+
|
|
296
|
+
```text
|
|
297
|
+
taxonomy.json
|
|
298
|
+
strategies.json
|
|
299
|
+
meta_prompts.jsonl
|
|
300
|
+
dataset.raw.jsonl
|
|
301
|
+
dataset.accepted.jsonl
|
|
302
|
+
dataset.final.jsonl
|
|
303
|
+
dataset.evaluated.jsonl
|
|
304
|
+
run_state.json
|
|
305
|
+
llm_calls.jsonl
|
|
306
|
+
eval_report.json
|
|
307
|
+
cost_summary.json
|
|
308
|
+
embeddings.cache.npz
|
|
309
|
+
```
|
|
310
|
+
|
|
311
|
+
`dataset.final.jsonl` is the generator's output. `dataset.evaluated.jsonl` is written by
|
|
312
|
+
`evaluate`/`run` and holds the deduped/decontaminated rows, leaving `dataset.final.jsonl`
|
|
313
|
+
untouched.
|
|
314
|
+
|
|
315
|
+
`meta_prompts.jsonl` holds one row per attempt (`attempt_index`, sampled `strategy_id` +
|
|
316
|
+
`taxonomy_mix`, `meta_prompt`, `complexified`). Like `taxonomy.json` and `strategies.json` it is
|
|
317
|
+
reused when present — stop with `--stop-after meta_prompts`, hand-edit the prompts, and rerun to
|
|
318
|
+
generate from the edited versions.
|
|
319
|
+
|
|
320
|
+
`llm_calls.jsonl` is especially useful while a run is still active. Every successful model response is appended immediately with:
|
|
321
|
+
|
|
322
|
+
- timestamp
|
|
323
|
+
- model role
|
|
324
|
+
- model name
|
|
325
|
+
- duration
|
|
326
|
+
- system prompt
|
|
327
|
+
- user prompt
|
|
328
|
+
- raw response
|
|
329
|
+
|
|
330
|
+
`dataset.final.jsonl` contains rows with full lineage:
|
|
331
|
+
|
|
332
|
+
- `id`
|
|
333
|
+
- `attempt_index`
|
|
334
|
+
- `record`
|
|
335
|
+
- `output_format`
|
|
336
|
+
- `taxonomy_mix`
|
|
337
|
+
- `strategy_id`
|
|
338
|
+
- `meta_prompt`
|
|
339
|
+
- `complexified`
|
|
340
|
+
- `generator_model`
|
|
341
|
+
- `critic_verdicts`
|
|
342
|
+
- `schema_valid`
|
|
343
|
+
- `accepted`
|
|
344
|
+
- `rejection_reason`
|
|
345
|
+
- `created_at`
|
|
346
|
+
|
|
347
|
+
## Query Extraction Example
|
|
348
|
+
|
|
349
|
+
The query extraction example generates records like:
|
|
350
|
+
|
|
351
|
+
```json
|
|
352
|
+
{
|
|
353
|
+
"query": "Compare Amtrak and Greyhound bus tickets from New York Penn Station to Washington DC for this Friday...",
|
|
354
|
+
"extraction": {
|
|
355
|
+
"intent": "transportation comparison",
|
|
356
|
+
"search_terms": ["Amtrak", "Greyhound", "New York", "Washington DC"],
|
|
357
|
+
"attributes": {
|
|
358
|
+
"domain": "travel",
|
|
359
|
+
"category": "ground transportation",
|
|
360
|
+
"entities": ["Amtrak", "Greyhound"],
|
|
361
|
+
"descriptors": ["cheapest", "less than 4 hours"],
|
|
362
|
+
"quantities": ["under 4 hours"]
|
|
363
|
+
},
|
|
364
|
+
"filters": {
|
|
365
|
+
"location": "New York to Washington DC",
|
|
366
|
+
"time": "this Friday",
|
|
367
|
+
"sort": "cheapest first"
|
|
368
|
+
},
|
|
369
|
+
"exclusions": [],
|
|
370
|
+
"ambiguities": ["Exact date for this Friday"]
|
|
371
|
+
}
|
|
372
|
+
}
|
|
373
|
+
```
|
|
374
|
+
|
|
375
|
+
Run it in two phases if you want to inspect the taxonomy before spending generation calls:
|
|
376
|
+
|
|
377
|
+
```bash
|
|
378
|
+
export OPENROUTER_API_KEY="..."
|
|
379
|
+
python -m simula.cli taxonomy examples/query_extraction_gemini.yaml
|
|
380
|
+
python -m simula.cli generate examples/query_extraction_gemini.yaml
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
Evaluation is separate:
|
|
384
|
+
|
|
385
|
+
```bash
|
|
386
|
+
python -m simula.cli evaluate examples/query_extraction_gemini.yaml
|
|
387
|
+
```
|
|
388
|
+
|
|
389
|
+
For diversity in JSON mode, set `evaluation.diversity.text_field` to a dotted field path such as `query` or `extraction.intent` to embed a specific field instead of the full JSON blob (a leading `$.` is accepted too). This is plain nested-key access, not full JSONPath. Embeddings are cached under the run directory and reused on later `evaluate` runs.
|
|
390
|
+
|
|
391
|
+
## Tests
|
|
392
|
+
|
|
393
|
+
```bash
|
|
394
|
+
pytest -q
|
|
395
|
+
```
|
|
396
|
+
|
|
397
|
+
Current test coverage includes:
|
|
398
|
+
|
|
399
|
+
- config defaults and validation
|
|
400
|
+
- JSON Schema subset validation
|
|
401
|
+
- JSONL IO
|
|
402
|
+
- deterministic taxonomy sampling
|
|
403
|
+
- dedupe
|
|
404
|
+
- coverage reports
|
|
405
|
+
- JSON repair path
|
|
406
|
+
- fake-model end-to-end generation
|
|
407
|
+
- CLI validation
|
|
408
|
+
|
|
409
|
+
## Current Limitations
|
|
410
|
+
|
|
411
|
+
- Generation is thread-based and pilot-scale, not distributed.
|
|
412
|
+
- The model client only supports OpenAI-compatible chat completions.
|
|
413
|
+
- The critic is simple and single-pass by default.
|
|
414
|
+
- Complexity scoring is optional and relatively basic.
|
|
415
|
+
- Taxonomy coverage for generated data uses lineage, not independent LLM assignment.
|
|
416
|
+
- There is no database, web UI, fine-tuning harness, multimodal support, or production queue.
|
|
417
|
+
|
|
418
|
+
## Development Notes
|
|
419
|
+
|
|
420
|
+
- Keep generated outputs under `runs/`; it is gitignored.
|
|
421
|
+
- Avoid committing API keys or local `.env` files.
|
|
422
|
+
- Prefer editing example YAMLs rather than hardcoding task-specific behavior.
|
|
423
|
+
- Keep built-in prompt defaults centralized in `simula/prompts.py`; use prompt modules for per-run overrides.
|
|
424
|
+
- Use `llm_calls.jsonl` when debugging model behavior.
|