isynth-ai-data-config 0.8.0__tar.gz

isynth_ai_data_config-0.8.0/PKG-INFO

@@ -0,0 +1,441 @@
+Metadata-Version: 2.4
+Name: isynth-ai-data-config
+Version: 0.8.0
+Summary: Generate iSynth data_config test-data definitions from a plain-language request, using an LLM that reads the environment's own metadata.
+Author-email: Markus Herrmann <markus.herrmann@itopia.ch>
+License-Expression: LicenseRef-Proprietary
+Project-URL: Homepage, https://isynth.io
+Keywords: isynth,mcp,llm,data-config,test-data,synthetic-data
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+Requires-Dist: litellm>=1.0
+Requires-Dist: fastmcp>=4.0
+Requires-Dist: openpyxl>=3.0
+Requires-Dist: python-box>=7.0
+Requires-Dist: python-dotenv>=1.0
+Requires-Dist: httpx>=0.28
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: starlette>=1.0
+
+# isynth-ai-data-config
+
+Generate iSynth `data_config` files from a request written in plain language, instead
+of hand-writing YAML against a data model you have to memorise first.
+
+You describe the test data you want — *"a private client with one account, no
+portfolio, no phone number"* — and you get back a ready `data_config` that iSynth can
+execute:
+
+```yaml
+simple_client:
+  based_on: banking.individual_client
+  number_of_copies: 1
+  parameters:
+    number_of_accounts: 1
+    number_of_portfolios: 0
+    number_of_phone_numbers: 0
+    number_of_email_addresses: 0
+  composers: []
+```
+
+An LLM writes that file, but it does not guess the vocabulary. `banking.individual_client`,
+`number_of_accounts` and every other name in the output come from **your** environment's
+own metadata, which the model reads through a set of tools while it works. The draft is
+then validated against that same metadata, and anything that does not check out goes
+back to the model for correction before you ever see it.
+
+## Some iSynth vocabulary first
+
+If you have not worked with iSynth: it is a platform for generating synthetic test
+data. A few terms show up throughout this document.
+
+| Term | What it means |
+|---|---|
+| **env** (environment) | One iSynth project — its own data model, its own directory on disk. This package is installed per env. |
+| **object type** | An entity in that model: `NaturalPerson`, `Account`, `PostalAddress`. |
+| **constellation template** | A reusable bundle of objects that belong together — a client with their accounts and addresses. It is the `based_on` value of a case. |
+| **composer** | A building block that adds objects to a constellation, in a defined order. |
+| **data_config** | The YAML file this package produces: one or more named cases, each saying what to build. iSynth executes it to create the actual data. |
+
+Everything above already exists in your env before this package is installed. The
+package only reads it.
+
+## How it works
+
+The model is not given a dump of your data model. It gets 14 tools and fetches only
+what it needs:
+
+| Tool group | Examples |
+|---|---|
+| Object types | `list_object_types`, `get_object_type` |
+| Composers | `list_composers`, `get_composer`, `get_composer_order` |
+| Constellation templates | `list_constellation_templates`, `get_constellation_template` |
+| Enums and functions | `list_enums`, `get_enum`, `list_functions`, `get_function` |
+| Curated examples | `list_ai_training_examples`, `get_ai_training_example` |
+| Validation | `validate_data_config` |
+
+Those tools are plain Python functions in one registry. Generation calls them
+**in-process** — no HTTP, no running server. The same registry is also served over MCP
+(streamable-http) for external clients such as IDE agents; that is optional and covered
+below.
+
+LLM access goes through [LiteLLM](https://www.litellm.ai/), so a local model in LM
+Studio and a hosted one are configured the same way. The model must support **native
+tool calling**.
+
+## Requirements
+
+- **Python 3.10** or newer.
+- **An iSynth env**, with these already in place:
+
+  | Path | Where it comes from |
+  |---|---|
+  | `__generated__/*.json` | the iSynth action **Initialize environment** (object types, composers, enums, functions) |
+  | `constellations/__generated__/**/*.json` | generated from the env's constellation modules |
+  | `data_configs/ai_training/` | **must exist**, may be empty — otherwise `list_ai_training_examples` raises `FileNotFoundError` |
+  | `__generated__/composer_sequence.txt` | written once by the restart plugin, see *Composer order* below |
+
+  Generation works without training examples, but noticeably worse: they are what the
+  model orients itself on.
+
+- **A tool-calling LLM** — local (LM Studio) or hosted. It must handle *native*
+  function calling over several turns: choose a tool, read the result, choose the next,
+  produce valid JSON arguments, and finally answer with a single JSON object. Prompt-
+  engineered tool use is not enough. See *LLM settings* below.
+
+## Install
+
+```
+pip install isynth-ai-data-config
+```
+
+Or add `isynth-ai-data-config` to your env's `requirements.txt` and install
+that as usual.
+
+Installing gives you the library. It does **not** yet put anything in the iSynth UI —
+that needs the plugin wrappers from *Full integration* below. If you only want the MCP
+tool server, or want to call `generate_data_order()` from your own code, you can stop
+after *Configuration*.
+
+### Alternative: copy the folder
+
+The package can also be copied wholesale into an env instead of installed — the way it
+was distributed before it was published, still supported for an env that cannot reach
+the private index. The steps (copying the folder, including its `requirements.txt` from
+the env's own list, the Docker build context and the `.dockerignore` that keeps it
+small) are in `install-by-copying.md`, which ships inside the package.
+
+## Configuration
+
+### Model
+
+Defaults ship in the package's own `settings.yml`. To deviate, repeat the same key in
+your **env's** `settings.yml` — the env wins:
+
+```yaml
+LLM_MODEL: "lm_studio/qwen3.8-27b-gguf"
+LLM_API_BASE: "http://192.168.1.100:1234/v1"
+```
+
+`${VARIABLE_NAME}` substitution works as in any iSynth `settings.yml`. You never edit a
+file inside the package, which is what keeps it upgradable.
+
+Only keys the package declares are read from the env; an `LLM_*`/`AI_*` key with a typo
+is reported as a warning naming the key. The ones you are most likely to touch:
+
+| Key | Default | What it does |
+|---|---|---|
+| `LLM_MODEL` | `openai/gpt-5.6-sol` | any LiteLLM model identifier |
+| `LLM_API_BASE` | — | for local or self-hosted endpoints |
+| `LLM_REQUEST_TIMEOUT` | `1800` | seconds; local models can be slow |
+| `LLM_MAX_OUTPUT_TOKENS` | `10000` | |
+| `AI_MAX_TOOL_TURNS` | `20` | budget for the tool-calling loop |
+| `AI_MAX_VALIDATION_ROUNDS` | `2` | correction rounds; `0` disables validation |
+| `AI_MAX_TRAINING_EXAMPLES` | `3` | how many curated examples may be fetched |
+
+The full list, each with a comment explaining it, is in the package's `settings.yml`.
+
+### LLM settings
+
+The package sends **no sampling parameters** — no temperature, no top_p. Those come
+from the defaults of whatever serves the model (LM Studio, vLLM, the hosted API). The
+one exception is `LLM_CHAT_TEMPLATE_KWARGS`, which travels with each request. So these
+are settings you make on the server, not here. **A model vendor's own recommendation
+always takes precedence over the table below.**
+
+| Setting | Suggested | Why |
+|---|---|---|
+| Reasoning / thinking mode | **off** | Decode dominates the runtime, and thinking tokens are decode. Measured on a local 27B: 85% of all generated tokens went into reasoning. Risk of thinking blocks or empty answers instead of JSON. |
+| Tool / function calling | **on, native** | Mandatory — the tools are passed over the API as `tools=[...]`. |
+| Temperature | 0.1–0.2 to start | Qwen warns against greedy decoding (`0.0` causes repetition loops, it recommends `0.7`); gpt-oss is specified for `1.0`. Start low, raise if you see loops. |
+| Top P | 0.8–0.9 | Keep conservative for structured output. |
+| Top K | 20–40 | Qwen recommends 20. |
+| Repeat penalty | 1.0–1.05 | Higher values corrupt recurring field names. |
+| Context length | 64K | A run including two correction rounds needs ~15–25K tokens. Use 128K only if you need it. |
+| Max response tokens | 2048–4096 | Per assistant turn, tool-call arguments included. The final JSON and the largest tool argument stay under 1K. |
+| Prompt truncation | **disabled** | Set the overflow policy to stop at the limit. The transcript grows over the correction rounds; "truncate middle" would silently cut away rules or examples. |
+| Structured output / JSON schema | use with care | Forcing a schema on every answer can suppress tool calls. `validate_data_config` checks the final JSON anyway. |
+| Quantization | 6–8 bit where memory allows | 4-bit produces measurably less precise tool arguments. Drop to 4-bit only as a memory fallback. |
+| Seed | fixed | Makes a validation run reproducible. |
+
+Running locally, pick a model by class rather than by name — a specific ranking dates
+quickly. With 16 GB expect to be limited to a small MoE or an aggressively quantized
+model, and to lose precision in tool arguments; 24–32 GB comfortably runs a mid-size
+MoE with few active parameters at 4-bit; from 64 GB a dense model in the 27–35B range
+at 6–8 bit becomes practical. Mixture-of-experts models with roughly 3–4B active
+parameters give the best interactive latency, because decode speed, not model size, is
+what a run waits on.
+
+### API keys
+
+Keys do **not** belong in `settings.yml`. They go in `<env>/.keys`, in dotenv format.
+Anything ending in `_API_KEY` or `_API_BASE` is picked up:
+
+```
+OPENAI_API_KEY=sk-...
+ANTHROPIC_API_KEY=sk-ant-...
+```
+
+A missing file is not an error — an env that sets its keys as real environment
+variables does not need one.
+
+### GitHub Copilot
+
+`LLM_MODEL: "github_copilot/<model>"` runs on a Copilot subscription instead of a
+provider API key. There is no key to put in `.keys`: litellm authenticates with a
+GitHub OAuth token that it obtains once through a device flow and then keeps in a
+token directory, as two files.
+
+| File | What it is |
+|---|---|
+| `access-token` | the long-lived GitHub token, written by the device flow |
+| `api-key.json` | the short-lived Copilot key, refreshed automatically |
+
+The default directory is `~/.config/litellm/github_copilot`, which inside a container
+is lost on the next rebuild. So set the directory **first**, then log in.
+
+**1. Point the token directory at the env.** In `<env>/.keys`, next to the other
+credentials:
+
+```
+GITHUB_COPILOT_TOKEN_DIR=/isynth_envs/<env>/.copilot
+```
+
+Every `GITHUB_COPILOT_*` variable in `.keys` is exported, the same way as an
+`_API_KEY`. Add `.copilot/` to the env's `.gitignore` — it holds a credential.
+
+**2. Log in once, interactively.** The device flow prints a code and waits for
+someone at a browser, so do it by hand rather than letting it happen inside a plugin
+call. Importing `ai_data_config.llm_helper` is what applies step 1 — with plain
+`import litellm` the token would land in the container's home directory instead:
+
+```bash
+docker exec -it <appserver> sh -c 'cd /isynth_envs/<env> && python3 -c "
+import ai_data_config.llm_helper, litellm
+litellm.completion(model=\"github_copilot/<model>\",
+                   messages=[{\"role\": \"user\", \"content\": \"hi\"}])
+"'
+```
+
+It prints `Please visit https://github.com/login/device and enter code XXXX-XXXX`.
+Open that page, enter the code, approve. The call then answers normally.
+
+**3. Check that both files are there.**
+
+```bash
+docker exec <appserver> ls /isynth_envs/<env>/.copilot
+# access-token  api-key.json
+```
+
+**4. Set the model.** In the env's `settings.yml`:
+
+```yaml
+LLM_MODEL: "github_copilot/<model>"
+```
+
+Leave `LLM_API_BASE` empty — litellm resolves `https://api.githubcopilot.com` itself.
+Which models you can name is Copilot's list, not litellm's.
+
+**5. Verify**, with the check under *Full integration → Check that it works*. From
+here on nothing is Copilot-specific; generation, the MCP tools and validation runs
+behave as with any other model.
+
+Steps 1 and 2 are needed once per env. The `api-key.json` expires regularly and is
+refreshed without asking; the `access-token` survives until it is revoked on GitHub.
+
+Two notes on the integration. It is litellm's **provider** integration
+(`docs.litellm.ai/docs/providers/github_copilot`) — not the *tutorial* page of a
+similar name, which points the Copilot IDE extension at a LiteLLM proxy and is the
+opposite direction. And tool calling, which this package depends on, works even
+though that page does not mention it.
+
+For a non-default setup the remaining variables also go in `.keys`:
+`_ACCESS_TOKEN_FILE` and `_API_KEY_FILE` rename the two files; `_API_BASE`,
+`_DEVICE_CODE_URL`, `_ACCESS_TOKEN_URL` and `_API_KEY_URL` point at a GitHub
+Enterprise endpoint.
+
+## Using it from Python
+
+```python
+from ai_data_config.llm_helper import generate_data_order
+
+data_config = generate_data_order("a private client with one account, no portfolio")
+```
+
+`generate_data_order(prompt, content=None)` builds the messages, runs the tool-calling
+loop, then validates and orders the result. `content` is an existing `data_config` to
+modify rather than start from scratch. It returns a dict, ready to be dumped as YAML.
+
+This needs no running MCP server and no iSynth runtime — only the env's generated
+metadata on disk, found via `ENV_HOME`.
+
+---
+
+# Full integration
+
+The steps that put the package into the iSynth UI. Skip any you do not need.
+
+## 1. Plugin wrappers
+
+The four files in `plugins/` are only the iSynth UI; the functionality lives in the
+package. Copy them into your env's `plugins/` folder.
+
+Each wrapper needs `base.plugin` (the iSynth plugin API), most also `base.logging`.
+Beyond that:
+
+| Wrapper | Calls | Additionally needs |
+|---|---|---|
+| `plugins/ai_data_config.py` | `llm_helper.generate_data_order()` | — |
+| `plugins/validate_ai_prompts.py` | `validation.config_validator.prepare_and_execute_validation_run()` | — |
+| `plugins/restart_mcp_server.py` | `mcp_server.admin.request_restart()` | `base.composers.Composer`, `settings.BASE_DIR` |
+| `plugins/bill_of_materials.py` | `bill_of_materials.calculate_bill_of_materials()` | `base.plugin_pro.ParameterType` |
+
+Afterwards run **"Reload plugins"** from the context menu of the `plugins/` folder in
+the UI — the REST API serves new definitions immediately, the UI only after that.
+
+## 2. Composer order (once)
+
+`get_composer_order()` reads `__generated__/composer_sequence.txt`. Without that file it
+returns an **empty list**, and the system prompt then fails to tell the model which
+order composer entries have to be in.
+
+The restart plugin writes it: run **Restart MCP Server** with the checkbox ticked. It
+fetches the order via `Composer.execution_sequence()` and writes the file *before*
+restarting the server, so it appears even when no MCP server is running yet (the
+restart call then fails, which is fine).
+
+Repeat after any model change that adds, removes or reorders composers. The file is
+read at import time, so a running process only picks up a change after a restart.
+
+## 3. MCP server service (optional)
+
+Only needed for external MCP clients (IDE agents) and for the **Restart MCP Server**
+plugin. Generation and validation runs do **not** need it — they call the same tools
+in-process and keep working with the service stopped.
+
+```yaml
+  i-mcpserver:
+    build:
+      context: ../
+      dockerfile: deployment/Dockerfile
+    restart: unless-stopped          # safety net, should os.execv fail on restart
+    environment:
+      - ENV_HOME=/isynth_envs/<env>  # must match the mount target below
+      - MCP_TRANSPORT=streamable-http
+      - MCP_HOST=0.0.0.0
+      - MCP_PORT=8010
+      - MCP_PATH=/mcp
+      - MCP_WAIT_FOR_GENERATED_TIMEOUT=60
+    volumes:
+      - "../:/isynth_envs/<env>"
+    working_dir: /isynth_envs/<env>  # so "-m ai_data_config.mcp_server" resolves
+    networks: [<the appserver's network>]
+    ports:
+      - "8010:8010"
+    command: "python3 -m ai_data_config.mcp_server"
+```
+
+**The service name is not free.** `mcp_server/admin.py` has
+`DEFAULT_BASE_URL = "http://i-mcpserver:8010"`, which is where the restart plugin looks.
+Either the service is called `i-mcpserver`, or the appserver service sets
+`MCP_SERVER_URL` to the right address.
+
+Installed from the index, the package also puts an `ai-data-config-mcp` command on
+`PATH`, equivalent to `python3 -m ai_data_config.mcp_server`.
+
+## 4. Check that it works
+
+```bash
+# package loads, settings arrive, tools answer
+docker exec <appserver> sh -c 'cd /isynth_envs/<env> && python3 -c "
+from ai_data_config import config, llm_helper
+from ai_data_config.mcp_server.tools import TOOLS, call_tool
+print(\"ENV_HOME :\", config.ENV_HOME)
+print(\"LLM_MODEL:\", llm_helper.DEFAULT_MODEL)
+print(\"Tools    :\", len(TOOLS))
+print(\"Composer :\", call_tool(\"get_composer_order\", {})[:3], \"...\")
+print(\"Examples :\", len(call_tool(\"list_ai_training_examples\", {})))
+"'
+
+# MCP server, if you set one up
+curl -s http://localhost:8010/health
+```
+
+Expected: 14 tools, a **non-empty** composer order, the number of training examples, and
+`{"status":"ok",...}`. An empty composer list means step 2 is missing.
+
+# What needs iSynth and what does not
+
+`llm_helper` and `mcp_server` import nothing from `base` — generation and the MCP tools
+run in any directory where the dependencies are installed, iSynth or not.
+
+Only `bill_of_materials.py` needs the iSynth runtime (`base.constellation_utils`,
+`base.data_config_utils`), and through it the validation run, which compares the bill of
+materials of a generated `data_config` against the case's reference. `base` is part of
+the iSynth image and cannot be installed from an index, so importing that module
+elsewhere raises a `ModuleNotFoundError` saying so.
+
+The MCP server container is needed by none of this except `restart_mcp_server`.
+
+# Troubleshooting
+
+| Symptom | Cause |
+|---|---|
+| `FileNotFoundError: ai_training_dir source not found` | `data_configs/ai_training/` missing — create it, empty is fine |
+| `FileNotFoundError: ... __generated__/...` | **Initialize environment** has not run in the env yet |
+| `ModuleNotFoundError: No module named 'box'` or similar | dependencies not installed — see *Install* |
+| `ModuleNotFoundError: ... needs the iSynth runtime ('base')` | `bill_of_materials` or the validation run is running outside an iSynth env |
+| `MCP server at ... did not answer` | service is not called `i-mcpserver` and `MCP_SERVER_URL` is not set |
+| Plugin does not appear in the UI | run "Reload plugins" from the context menu of the `plugins/` folder |
+| Env override is ignored | key not declared in the package's `settings.yml` — the log warning names it |
+| `get_composer_order()` is empty | `__generated__/composer_sequence.txt` missing — see *Composer order* |
+| Logs are empty | not on STDOUT: the package logs with `propagate=False` into `<env>/logs/*.log` |
+
+# Further reading
+
+`install-by-copying.md` ships alongside this file: the alternative to installing,
+copying the folder into an env and wiring up its dependencies and the Docker build
+yourself.
+
+# Building the package
+
+```bash
+python -m build ai_data_config/        # -> ai_data_config/dist/*.whl and *.tar.gz
+python -m twine upload ai_data_config/dist/*   # -> pypi.org
+```
+
+The build runs from inside the package folder, where `pyproject.toml` sits, because the
+folder is meant to travel as one unit. Dependencies are declared only in
+`requirements.txt`; `pyproject.toml` reads them from there via
+`dynamic = ["dependencies"]`, so there is no second list to drift out of sync. The
+package is published on pypi.org; its licence stays proprietary.
+
+In CI (`.gitlab-ci.yml` in the repository root) this runs automatically: every change
+under `ai_data_config/` is built and checked, and pushing a tag
+`ai_data_config-v<version>` (matching `version` in `pyproject.toml`) publishes the
+package to the project's GitLab package registry.