simula-gen 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Mercity AI
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,424 @@
1
+ Metadata-Version: 2.4
2
+ Name: simula-gen
3
+ Version: 0.1.0
4
+ Summary: Schema-driven synthetic data generator inspired by reasoning-first taxonomy coverage.
5
+ License: MIT
6
+ Requires-Python: >=3.11
7
+ Description-Content-Type: text/markdown
8
+ License-File: LICENSE
9
+ Requires-Dist: jsonschema>=4.22.0
10
+ Requires-Dist: openai>=1.50.0
11
+ Requires-Dist: pydantic>=2.7.0
12
+ Requires-Dist: python-dotenv>=1.0.0
13
+ Requires-Dist: PyYAML>=6.0.1
14
+ Requires-Dist: rich>=13.0.0
15
+ Provides-Extra: diversity
16
+ Requires-Dist: numpy>=1.26.0; extra == "diversity"
17
+ Requires-Dist: scikit-learn>=1.4.0; extra == "diversity"
18
+ Requires-Dist: sentence-transformers>=2.7.0; extra == "diversity"
19
+ Provides-Extra: dev
20
+ Requires-Dist: simula-gen[diversity]; extra == "dev"
21
+ Requires-Dist: pytest>=8.0.0; extra == "dev"
22
+ Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
23
+ Dynamic: license-file
24
+
25
+ <p align="center">
26
+ <img src="assets/hero.png" alt="Simula" width="640">
27
+ </p>
28
+
29
+ <p align="center">
30
+ <img alt="Python" src="https://img.shields.io/badge/python-%E2%89%A53.11-blue">
31
+ <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-green"></a>
32
+ <img alt="Version" src="https://img.shields.io/badge/version-0.1.0-blue">
33
+ <img alt="Status" src="https://img.shields.io/badge/status-pilot-orange">
34
+ <a href="https://arxiv.org/abs/2603.29791"><img alt="Paper" src="https://img.shields.io/badge/paper-arXiv%3A2603.29791-b31b1b"></a>
35
+ <a href="https://github.com/Mercity-AI"><img alt="Built by Mercity" src="https://img.shields.io/badge/built%20by-Mercity-1f6feb"></a>
36
+ </p>
37
+
38
+ # simula
39
+
40
+ > 📖 **[Concepts & Configuration](DOCS.md)** — the pipeline, the model roles, and every config knob, with diagrams.
41
+
42
+ `simula` is a small CLI-first framework for generating schema-shaped or free-text synthetic datasets with taxonomy-guided coverage. It is inspired by the Simula-style workflow: map the conceptual space first, sample from that space, generate records, critique them, preserve lineage, and then trim/evaluate the result.
43
+
44
+ The current implementation is intentionally lean. It uses JSON/YAML files, OpenAI-compatible chat endpoints, JSON Schema validation, and human-inspectable artifacts.
45
+
46
+ ## What It Does
47
+
48
+ - Builds taxonomies from a dataset description and optional user-provided factors.
49
+ - Generates sampling strategies over taxonomy branches.
50
+ - Samples taxonomy mixes and turns them into meta-prompts.
51
+ - Generates JSON records matching a user-defined schema, or direct free-form text when `schema: null`.
52
+ - Validates records with a JSON Schema subset.
53
+ - Runs a simple critic/refine loop.
54
+ - Writes every model response to `llm_calls.jsonl` as soon as it returns.
55
+ - Writes raw, accepted, and final datasets as JSONL.
56
+ - Supports n-gram dedupe, coverage reports, optional reassignment coverage, diversity metrics, and optional complexity scoring.
57
+
58
+ ## Install
59
+
60
+ From the repository root:
61
+
62
+ ```bash
63
+ python -m pip install -e ".[dev]"
64
+ ```
65
+
66
+ The base install is light. The optional embedding-based **diversity** metric needs heavier packages
67
+ (numpy, scikit-learn, sentence-transformers / PyTorch); install them only if you enable it:
68
+
69
+ ```bash
70
+ python -m pip install -e ".[diversity]"
71
+ ```
72
+
73
+ You can also run the CLI without installing:
74
+
75
+ ```bash
76
+ python -m simula.cli validate examples/basic_qa.yaml
77
+ ```
78
+
79
+ ## Quick Start
80
+
81
+ Run the fake-model smoke test. This does not call any external API:
82
+
83
+ ```bash
84
+ python -m simula.cli run examples/basic_qa.yaml
85
+ ```
86
+
87
+ Artifacts are written to:
88
+
89
+ ```text
90
+ runs/basic_qa/
91
+ ```
92
+
93
+ For a real OpenRouter/OpenAI-compatible run, put the key named by `provider.api_key_env` in a
94
+ gitignored `.env` file at the project root:
95
+
96
+ ```bash
97
+ echo 'OPENROUTER_API_KEY=sk-or-...' > .env # gitignored; the only source of API keys
98
+ python -m simula.cli taxonomy examples/query_extraction_gemini.yaml
99
+ python -m simula.cli generate examples/query_extraction_gemini.yaml
100
+ ```
101
+
102
+ Key resolution: the project-root `.env` file is read directly (via `dotenv_values`) and is the
103
+ **only** source of API keys — a shell-exported variable is deliberately ignored, so the key always
104
+ comes from the file you can see. A missing key warns at `validate` and fails real model calls. Never
105
+ commit real keys.
106
+
107
+ The query extraction example currently uses:
108
+
109
+ ```text
110
+ google/gemini-3-flash-preview
111
+ ```
112
+
113
+ through the OpenRouter-compatible API.
114
+
115
+ ## CLI Commands
116
+
117
+ ```bash
118
+ simula validate CONFIG.yaml
119
+ ```
120
+
121
+ Validates the YAML config, JSON Schema subset, model role config, and output path. It makes no model calls.
122
+
123
+ ```bash
124
+ simula taxonomy CONFIG.yaml
125
+ ```
126
+
127
+ Builds `taxonomy.json` and, depending on `taxonomy.review_mode`, either accepts it automatically or lets the user review it.
128
+
129
+ ```bash
130
+ simula generate CONFIG.yaml
131
+ ```
132
+
133
+ Loads or builds the taxonomy, builds strategies and meta prompts if missing, generates data, validates records, runs critic/refine, and writes dataset artifacts. It does not run evaluation.
134
+
135
+ Pass `--stop-after {taxonomy,strategies,meta_prompts}` (or set `generation.stop_after` in the
136
+ config) to halt once that stage's artifact is written. Edit the artifact, then rerun: existing
137
+ artifacts are reused, so the run picks up where it stopped. `--stop-after none` overrides a
138
+ config-set stop for one invocation. To regenerate a stage instead, delete its file
139
+ (`taxonomy.json`, `strategies.json`, `meta_prompts.jsonl`) and rerun.
140
+
141
+ ```bash
142
+ simula evaluate CONFIG.yaml
143
+ ```
144
+
145
+ Runs dedupe, coverage, and optional complexity scoring. It reads `dataset.final.jsonl` and writes
146
+ the deduped/decontaminated result to a separate `dataset.evaluated.jsonl` (it never rewrites
147
+ `dataset.final.jsonl`), plus `eval_report.json`.
148
+
149
+ ```bash
150
+ simula run CONFIG.yaml
151
+ ```
152
+
153
+ Runs taxonomy, generation, and evaluation end to end.
154
+
155
+ ## Config Overview
156
+
157
+ Each run is controlled by a single YAML file. **`examples/template.yaml`** is a copy-me skeleton with
158
+ every key at its default, and **`CONFIG.md`** is the full per-field reference. Only two things are
159
+ required — a non-empty `description` and a `model` id for each of the three roles; everything else has
160
+ a default, so the minimum viable config is small:
161
+
162
+ ```yaml
163
+ description: "Describe the dataset to generate."
164
+
165
+ schema: # omit or set null for free-text mode
166
+ type: object
167
+ required: ["input", "output"]
168
+ properties:
169
+ input: { type: string }
170
+ output: { type: string }
171
+
172
+ provider:
173
+ base_url: "https://openrouter.ai/api/v1"
174
+ api_key_env: "OPENROUTER_API_KEY" # variable name read from the project-root .env
175
+
176
+ models:
177
+ strategic: { model: "google/gemini-3-flash-preview" }
178
+ bulk: { model: "google/gemini-3-flash-preview" }
179
+ critic: { model: "google/gemini-3-flash-preview" }
180
+ ```
181
+
182
+ Copy `examples/template.yaml` for the full set of knobs (`project`, `taxonomy`, `strategy`,
183
+ `sampling`, `generation`, `evaluation`); the sections below explain the ones worth steering by hand.
184
+
185
+ ### Model Roles
186
+
187
+ - `strategic`: factor discovery, taxonomy expansion, strategy generation.
188
+ - `bulk`: meta-prompt generation, complexification, record generation, repair, refinement.
189
+ - `critic`: semantic critique and optional complexity scoring.
190
+
191
+ All model roles use the same OpenAI-compatible chat completions interface. A role can use `"model": "fake"` for deterministic local tests.
192
+
193
+ ### Prompt Overrides
194
+
195
+ Built-in prompt defaults live in `simula/prompts.py`. To override them for one run, point config at a Python module:
196
+
197
+ ```yaml
198
+ prompts:
199
+ module: "config/prompts.py"
200
+ ```
201
+
202
+ Paths are resolved relative to the YAML file. The module may override any subset of the built-in prompt functions, and missing functions fall back to the defaults. It may also override `SYSTEM_JSON` and `SYSTEM_TEXT`.
203
+
204
+ ```python
205
+ SYSTEM_JSON = "Return valid JSON only."
206
+
207
+
208
+ def strategy_prompt(description, taxonomy, guidance=None):
209
+ return f"""
210
+ Dataset description:
211
+ {description}
212
+
213
+ Taxonomy:
214
+ {taxonomy}
215
+
216
+ Create only combinations that make semantic sense for this dataset.
217
+ Return JSON with a strategies array.
218
+ """.strip()
219
+ ```
220
+
221
+ Override functions must keep the same parameter names as their built-in counterparts in `simula/prompts.py`. `simula validate` imports the module and rejects missing files, import failures, non-string system prompts, and incompatible function signatures before any model call runs.
222
+
223
+ ### Strategy Guidance
224
+
225
+ Strategies decide which taxonomy branches combine and how often each combination is sampled. To steer that without writing a prompt module, set free-text `strategy.guidance`:
226
+
227
+ ```yaml
228
+ strategy:
229
+ guidance: |
230
+ - Make billing + calm + simple the most common combination.
231
+ - Never combine enterprise_sso with mobile_app; they don't coexist.
232
+ - Keep a dedicated strategy for furious + needs_escalation, even though it is rare.
233
+ ```
234
+
235
+ The guidance is woven into the strategy prompt, so the generated `strategies.json` reflects your intent (root combinations and weights) before any bulk generation runs. Guidance is a nudge interpreted by the strategic model, not a hard constraint; for guarantees, edit `strategies.json` directly or override `strategy_prompt`. When unset (`null`), the built-in prompt is used unchanged.
236
+
237
+ ### Sampling
238
+
239
+ Decoding params resolve in three layers, last one wins: built-in defaults (`temperature: 0.7`, `max_tokens: 32768`) → per-role config under `models.<role>` → per-task overrides under `sampling.tasks`. Each task is handled by exactly one role, so naming a task is enough — no role needs to be specified.
240
+
241
+ ```yaml
242
+ sampling:
243
+ tasks:
244
+ generate: {temperature: 1.1, top_p: 0.95, min_p: 0.05}
245
+ repair: {temperature: 0.0}
246
+ meta_prompt: {temperature: 1.1}
247
+ ```
248
+
249
+ Valid task names are the `TaskType` values: `factor_discovery`, `node_expansion`, `taxonomy_critic`, `level_plan`, `strategy`, `meta_prompt`, `complexify`, `generate`, `repair`, `semantic_critic`, `refine`, `complexity_score`, `node_assign`.
250
+
251
+ OpenAI-compatible params (`temperature`, `top_p`, `max_tokens`, `frequency_penalty`, `presence_penalty`, `stop`, `seed`) are sent as top-level call kwargs. Anything else (`min_p`, `top_k`, `repetition_penalty`, …) is passed through `extra_body` so provider-specific knobs work without lock-in. The resolved params are recorded per call in `llm_calls.jsonl`.
252
+
253
+ Connection lives on `provider` (`base_url`, `api_key_env`, `timeout_seconds`), shared by all roles;
254
+ the non-decoding role keys (`model`, `extra_body`) are never treated as decoding params.
255
+
256
+ **Per-request timeout.** Each real call uses a default timeout of **180 seconds** so a hung or
257
+ rate-limited provider connection cannot stall a worker for the SDK default (~600s). Override with
258
+ `provider.timeout_seconds`.
259
+
260
+ **Reasoning models.** There is no automatic model-id detection. If a model emits hidden reasoning
261
+ tokens (e.g. DeepSeek R1/V4, OpenAI o-series) and you want them excluded from output and your
262
+ `max_tokens` budget, set it explicitly per role:
263
+
264
+ ```yaml
265
+ models:
266
+ bulk:
267
+ model: "deepseek/deepseek-v4-flash"
268
+ max_tokens: 16384
269
+ extra_body: {reasoning: {effort: low, exclude: true}}
270
+ ```
271
+
272
+ `simula validate` rejects unknown task names and non-numeric values before any model call runs. With no `sampling` block, behavior is unchanged except the larger default `max_tokens`. That 32K default is batteries-included: roles without an explicit `max_tokens` can now emit much larger (and pricier) completions than before — set `models.<role>.max_tokens` or a per-task `max_tokens` to cap it.
273
+
274
+ ### Schema Support
275
+
276
+ The MVP supports a practical JSON Schema subset:
277
+
278
+ - `object`
279
+ - `string`
280
+ - `number`
281
+ - `integer`
282
+ - `boolean`
283
+ - `array`
284
+ - `enum`
285
+ - `required`
286
+ - nested objects and arrays
287
+
288
+ Every generated record must validate against the schema before it can be accepted.
289
+
290
+ Set `schema: null` or omit `schema` to use free-text mode. In that mode `record` is a string, JSON repair is skipped, and the critic still returns a JSON verdict.
291
+
292
+ ## Artifacts
293
+
294
+ Each run writes human-inspectable files under `project.output_dir`:
295
+
296
+ ```text
297
+ taxonomy.json
298
+ strategies.json
299
+ meta_prompts.jsonl
300
+ dataset.raw.jsonl
301
+ dataset.accepted.jsonl
302
+ dataset.final.jsonl
303
+ dataset.evaluated.jsonl
304
+ run_state.json
305
+ llm_calls.jsonl
306
+ eval_report.json
307
+ cost_summary.json
308
+ embeddings.cache.npz
309
+ ```
310
+
311
+ `dataset.final.jsonl` is the generator's output. `dataset.evaluated.jsonl` is written by
312
+ `evaluate`/`run` and holds the deduped/decontaminated rows, leaving `dataset.final.jsonl`
313
+ untouched.
314
+
315
+ `meta_prompts.jsonl` holds one row per attempt (`attempt_index`, sampled `strategy_id` +
316
+ `taxonomy_mix`, `meta_prompt`, `complexified`). Like `taxonomy.json` and `strategies.json` it is
317
+ reused when present — stop with `--stop-after meta_prompts`, hand-edit the prompts, and rerun to
318
+ generate from the edited versions.
319
+
320
+ `llm_calls.jsonl` is especially useful while a run is still active. Every successful model response is appended immediately with:
321
+
322
+ - timestamp
323
+ - model role
324
+ - model name
325
+ - duration
326
+ - system prompt
327
+ - user prompt
328
+ - raw response
329
+
330
+ `dataset.final.jsonl` contains rows with full lineage:
331
+
332
+ - `id`
333
+ - `attempt_index`
334
+ - `record`
335
+ - `output_format`
336
+ - `taxonomy_mix`
337
+ - `strategy_id`
338
+ - `meta_prompt`
339
+ - `complexified`
340
+ - `generator_model`
341
+ - `critic_verdicts`
342
+ - `schema_valid`
343
+ - `accepted`
344
+ - `rejection_reason`
345
+ - `created_at`
346
+
347
+ ## Query Extraction Example
348
+
349
+ The query extraction example generates records like:
350
+
351
+ ```json
352
+ {
353
+ "query": "Compare Amtrak and Greyhound bus tickets from New York Penn Station to Washington DC for this Friday...",
354
+ "extraction": {
355
+ "intent": "transportation comparison",
356
+ "search_terms": ["Amtrak", "Greyhound", "New York", "Washington DC"],
357
+ "attributes": {
358
+ "domain": "travel",
359
+ "category": "ground transportation",
360
+ "entities": ["Amtrak", "Greyhound"],
361
+ "descriptors": ["cheapest", "less than 4 hours"],
362
+ "quantities": ["under 4 hours"]
363
+ },
364
+ "filters": {
365
+ "location": "New York to Washington DC",
366
+ "time": "this Friday",
367
+ "sort": "cheapest first"
368
+ },
369
+ "exclusions": [],
370
+ "ambiguities": ["Exact date for this Friday"]
371
+ }
372
+ }
373
+ ```
374
+
375
+ Run it in two phases if you want to inspect the taxonomy before spending generation calls:
376
+
377
+ ```bash
378
+ export OPENROUTER_API_KEY="..."
379
+ python -m simula.cli taxonomy examples/query_extraction_gemini.yaml
380
+ python -m simula.cli generate examples/query_extraction_gemini.yaml
381
+ ```
382
+
383
+ Evaluation is separate:
384
+
385
+ ```bash
386
+ python -m simula.cli evaluate examples/query_extraction_gemini.yaml
387
+ ```
388
+
389
+ For diversity in JSON mode, set `evaluation.diversity.text_field` to a dotted field path such as `query` or `extraction.intent` to embed a specific field instead of the full JSON blob (a leading `$.` is accepted too). This is plain nested-key access, not full JSONPath. Embeddings are cached under the run directory and reused on later `evaluate` runs.
390
+
391
+ ## Tests
392
+
393
+ ```bash
394
+ pytest -q
395
+ ```
396
+
397
+ Current test coverage includes:
398
+
399
+ - config defaults and validation
400
+ - JSON Schema subset validation
401
+ - JSONL IO
402
+ - deterministic taxonomy sampling
403
+ - dedupe
404
+ - coverage reports
405
+ - JSON repair path
406
+ - fake-model end-to-end generation
407
+ - CLI validation
408
+
409
+ ## Current Limitations
410
+
411
+ - Generation is thread-based and pilot-scale, not distributed.
412
+ - The model client only supports OpenAI-compatible chat completions.
413
+ - The critic is simple and single-pass by default.
414
+ - Complexity scoring is optional and relatively basic.
415
+ - Taxonomy coverage for generated data uses lineage, not independent LLM assignment.
416
+ - There is no database, web UI, fine-tuning harness, multimodal support, or production queue.
417
+
418
+ ## Development Notes
419
+
420
+ - Keep generated outputs under `runs/`; it is gitignored.
421
+ - Avoid committing API keys or local `.env` files.
422
+ - Prefer editing example YAMLs rather than hardcoding task-specific behavior.
423
+ - Keep built-in prompt defaults centralized in `simula/prompts.py`; use prompt modules for per-run overrides.
424
+ - Use `llm_calls.jsonl` when debugging model behavior.