usersim 0.2.0__tar.gz → 0.3.0__tar.gz

{usersim-0.2.0/src/usersim.egg-info → usersim-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: usersim
-Version: 0.2.0
+Version: 0.3.0
 Summary: Improve models and agents with population-grounded simulated users, even before production traffic exists.
 License-Expression: Apache-2.0
 Project-URL: Homepage, https://github.com/NVIDIA-NeMo/UserSim
@@ -50,7 +50,7 @@ Dynamic: license-file
 > trajectories: evaluation signal, failure analysis, and curated training data,
 > before you have production traffic to learn from.
 
-**14 probes · 11 shipped locales across 9 countries (+ 22 preview India-language variants) · 8M+ census-grounded personas · 3,407 tools across 143 toolsets for agentic runs · 12 evaluator scorers · 4,000+ tests · offline `usersim smoke` in seconds**
+**14 probes · 11 shipped locales across 9 countries (+ 22 preview India-language variants) · 8M+ census-grounded personas · 3,406 tools across 143 toolsets for agentic runs · 12 evaluator scorers · 4,000+ tests · offline `usersim smoke` in seconds**
 
 <img src="https://raw.githubusercontent.com/NVIDIA-NeMo/UserSim/main/docs/images/hero.jpg" width="800" alt="NeMo User Sim: Population-Grounded User Simulation. Simulated users in the USA, Brazil, France, India and Japan, each speaking their own language under a different probe, on a world map beside headline counts for people represented, languages, locales and probes.">
 
@@ -116,8 +116,8 @@ walks through each layer and where it extends.
   grounded in census distributions.
 - **Sovereign-AI builders** with no user base yet. You get an in-language
   evaluation surface from day one, drawn from your country's actual census
-  distribution: USA, Japan, India, Singapore, Brazil, France and Korea ship
-  today.
+  distribution: USA, UK, Germany, France, Brazil, India, Singapore, Japan and
+  Korea ship today.
 - **Agent builders** putting a tool-using system under multi-turn load. Probes
   execute tools against a mock environment that holds state across turns, so a
   run surfaces scope creep, unauthorized calls, prompt-injection susceptibility
@@ -175,11 +175,11 @@ in the NeMo Gym repository.
 <img src="https://raw.githubusercontent.com/NVIDIA-NeMo/UserSim/main/docs/images/getting_started.jpg" width="800" alt="From install to capability report in five commands: pip install usersim, download personas with data-designer, then usersim simulate, eval and report. Optional steps fix the panel, replay stored rows or select training data, with a locale picker and the probe groups below.">
 
 Python 3.10+, in a virtual environment. Python 3.10 is supported through the
-0.2 releases.
+0.3 releases.
 
 ```bash
 python -m venv .venv && source .venv/bin/activate
-pip install "usersim>=0.2.0"      # in a uv project: uv add "usersim>=0.2.0"
+pip install "usersim>=0.3.0"      # in a uv project: uv add "usersim>=0.3.0"
 
 # Verify the install offline: no LLM calls, no network, CI-ready.
 usersim smoke
@@ -232,9 +232,9 @@ These steps make live model calls, using the keys and personas above.
 usersim simulate --locale en_US --num-rows 30 --out output/trajectories
 
 # 2. Score the assistant under test: the evaluator model's judgments, plus
-#    the scorers that match the default probe mix.
+#    each probe's own scorers and the three that apply to every probe.
 usersim eval     --trajectories output/trajectories --out output/evaluations \
-                 --scorers tool_use,language_compliance,response_shape,refusal_basics
+                 --scorers auto
 
 # 3. Render the capability dashboard.
 usersim report   --trajectories output/trajectories \
@@ -252,9 +252,13 @@ usersim simulate --locale ja_JP --num-rows 30 \
   --out output/trajectories
 ```
 
-Score a run with the scorers for its probes. For this one, step 2 takes
-`--scorers sov_ai_facts,sov_ai_dynamic,sov_ai_multilingual_parity,language_compliance`;
-[`docs/outputs.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/outputs.md) lists every scorer.
+With `--scorers auto`, step 2 scores this run with the three `sov_ai` scorers
+as well; [`docs/outputs.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/outputs.md)
+lists every scorer. A probe that takes settings gets them with
+`--probe-option LABEL.SETTING=VALUE`, such as
+`--probe-option financial_services.tier_mix=0.3`;
+[`docs/probes.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/probes.md#a-probes-settings)
+lists each probe's settings.
 
 ### Curate training data
 
@@ -334,6 +338,8 @@ full plan and exits before the first network call.
 
 **Extending it**
 
+- [`docs/extensions.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/extensions.md): a probe and a scorer in a package of your own, installed next to `usersim`
+- [`docs/extension_api.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/extension_api.md): every name an extension imports from `usersim`
 - [`docs/engine/AUTHORING_A_PROBE.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/engine/AUTHORING_A_PROBE.md): end-to-end tutorial for a new probe
 - [`docs/engine/README.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/engine/README.md): the substrate's full API surface
 - [`templates/probe/`](https://github.com/NVIDIA-NeMo/UserSim/tree/main/templates/probe/): copy-and-edit starters, one per probe shape

{usersim-0.2.0 → usersim-0.3.0}/README.md

@@ -10,7 +10,7 @@
 > trajectories: evaluation signal, failure analysis, and curated training data,
 > before you have production traffic to learn from.
 
-**14 probes · 11 shipped locales across 9 countries (+ 22 preview India-language variants) · 8M+ census-grounded personas · 3,407 tools across 143 toolsets for agentic runs · 12 evaluator scorers · 4,000+ tests · offline `usersim smoke` in seconds**
+**14 probes · 11 shipped locales across 9 countries (+ 22 preview India-language variants) · 8M+ census-grounded personas · 3,406 tools across 143 toolsets for agentic runs · 12 evaluator scorers · 4,000+ tests · offline `usersim smoke` in seconds**
 
 <img src="https://raw.githubusercontent.com/NVIDIA-NeMo/UserSim/main/docs/images/hero.jpg" width="800" alt="NeMo User Sim: Population-Grounded User Simulation. Simulated users in the USA, Brazil, France, India and Japan, each speaking their own language under a different probe, on a world map beside headline counts for people represented, languages, locales and probes.">
 
@@ -76,8 +76,8 @@ walks through each layer and where it extends.
   grounded in census distributions.
 - **Sovereign-AI builders** with no user base yet. You get an in-language
   evaluation surface from day one, drawn from your country's actual census
-  distribution: USA, Japan, India, Singapore, Brazil, France and Korea ship
-  today.
+  distribution: USA, UK, Germany, France, Brazil, India, Singapore, Japan and
+  Korea ship today.
 - **Agent builders** putting a tool-using system under multi-turn load. Probes
   execute tools against a mock environment that holds state across turns, so a
   run surfaces scope creep, unauthorized calls, prompt-injection susceptibility
@@ -135,11 +135,11 @@ in the NeMo Gym repository.
 <img src="https://raw.githubusercontent.com/NVIDIA-NeMo/UserSim/main/docs/images/getting_started.jpg" width="800" alt="From install to capability report in five commands: pip install usersim, download personas with data-designer, then usersim simulate, eval and report. Optional steps fix the panel, replay stored rows or select training data, with a locale picker and the probe groups below.">
 
 Python 3.10+, in a virtual environment. Python 3.10 is supported through the
-0.2 releases.
+0.3 releases.
 
 ```bash
 python -m venv .venv && source .venv/bin/activate
-pip install "usersim>=0.2.0"      # in a uv project: uv add "usersim>=0.2.0"
+pip install "usersim>=0.3.0"      # in a uv project: uv add "usersim>=0.3.0"
 
 # Verify the install offline: no LLM calls, no network, CI-ready.
 usersim smoke
@@ -192,9 +192,9 @@ These steps make live model calls, using the keys and personas above.
 usersim simulate --locale en_US --num-rows 30 --out output/trajectories
 
 # 2. Score the assistant under test: the evaluator model's judgments, plus
-#    the scorers that match the default probe mix.
+#    each probe's own scorers and the three that apply to every probe.
 usersim eval     --trajectories output/trajectories --out output/evaluations \
-                 --scorers tool_use,language_compliance,response_shape,refusal_basics
+                 --scorers auto
 
 # 3. Render the capability dashboard.
 usersim report   --trajectories output/trajectories \
@@ -212,9 +212,13 @@ usersim simulate --locale ja_JP --num-rows 30 \
   --out output/trajectories
 ```
 
-Score a run with the scorers for its probes. For this one, step 2 takes
-`--scorers sov_ai_facts,sov_ai_dynamic,sov_ai_multilingual_parity,language_compliance`;
-[`docs/outputs.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/outputs.md) lists every scorer.
+With `--scorers auto`, step 2 scores this run with the three `sov_ai` scorers
+as well; [`docs/outputs.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/outputs.md)
+lists every scorer. A probe that takes settings gets them with
+`--probe-option LABEL.SETTING=VALUE`, such as
+`--probe-option financial_services.tier_mix=0.3`;
+[`docs/probes.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/probes.md#a-probes-settings)
+lists each probe's settings.
 
 ### Curate training data
 
@@ -294,6 +298,8 @@ full plan and exits before the first network call.
 
 **Extending it**
 
+- [`docs/extensions.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/extensions.md): a probe and a scorer in a package of your own, installed next to `usersim`
+- [`docs/extension_api.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/extension_api.md): every name an extension imports from `usersim`
 - [`docs/engine/AUTHORING_A_PROBE.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/engine/AUTHORING_A_PROBE.md): end-to-end tutorial for a new probe
 - [`docs/engine/README.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/engine/README.md): the substrate's full API surface
 - [`templates/probe/`](https://github.com/NVIDIA-NeMo/UserSim/tree/main/templates/probe/): copy-and-edit starters, one per probe shape

{usersim-0.2.0 → usersim-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "usersim"
-version = "0.2.0"
+version = "0.3.0"
 description = "Improve models and agents with population-grounded simulated users, even before production traffic exists."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -87,7 +87,8 @@ Issues = "https://github.com/NVIDIA-NeMo/UserSim/issues"
 # A dependency group rather than an optional-dependency extra, so `uv sync`
 # installs it by default and `pip install usersim[dev]` cannot pull test
 # and lint tooling into a user's environment. `make test` and the hermeticity
-# guard in conftest.py both depend on this group being present.
+# checks in usersim.testing.pytest_plugin both depend on this group being
+# present.
 dev = [
     "jupyter",
     "pytest>=8.0",
@@ -101,18 +102,22 @@ dev = [
     # without this.
     "pytest-xdist>=3.6,<4",
     # Applies the [tool.pytest.ini_options] env block below, which sets
-    # USERSIM_DISABLE_EXTENSIONS before any conftest loads. A root conftest.py
-    # would run after one that imports a registry, which is too late to keep
-    # discovery out of the suite. conftest.py fails the run if this is absent.
+    # USERSIM_DISABLE_EXTENSIONS before any conftest loads. Setting it from the
+    # root conftest.py, or the plugin it loads, would come after any conftest
+    # that imports a registry, which is too late to keep discovery out of the
+    # suite. usersim.testing.pytest_plugin fails the run if this is absent.
     "pytest-env>=1.1,<2",
     # Collects `async def` tests. `asyncio_mode = "auto"` below means they need
     # no decorator, so a contributor writing one does not have to know this is
-    # here. conftest.py fails the run if it is absent, because without it every
-    # async test is skipped with a warning rather than failing.
+    # here. usersim.testing.pytest_plugin fails the run if it is absent,
+    # because without it no async test runs.
     "pytest-asyncio>=1.0,<2",
     # tests/test_network_guard.py unwraps exception groups, which Python 3.10
     # has only through this backport.
     "exceptiongroup>=1.0; python_version < '3.11'",
+    # Validates each tool spec in the tool_calling catalog against the JSON
+    # Schema 2020-12 meta-schema, in tests/probes/tool_calling/test_catalog.py.
+    "jsonschema>=4.0,<5",
 ]
 # Empty on purpose: the gate runs a pinned pip-licenses through `uv tool run`,
 # so this group exists only to scope an isolated environment holding the
@@ -163,14 +168,16 @@ include = ["usersim*"]
     "models_*.toml",
 ]
 "usersim.engine" = [
-    # The asset banks. setuptools ships package data only when it is named,
-    # and a wheel without the banks fails every probe at asset resolution.
-    # `make check-dist` fails if the wheel has none.
-    "assets/**/*.yaml",
-    "assets/**/*.yml",
-    "assets/**/*.json",
-    "assets/**/*.parquet",
-    "assets/**/*.md",
+    # The asset banks, each in its probe's own folder, under probes/<folder>/assets/.
+    # setuptools ships package data only when it is named, and a wheel without the
+    # banks fails every probe at asset resolution. `make check-dist` fails if the
+    # wheel misses any bank file.
+    "probes/*/assets/**/*.yml",
+    "probes/*/assets/**/*.json",
+    "probes/*/assets/**/*.parquet",
+    "probes/*/assets/**/*.md",
+    # Every other YAML, the region specs an asset domain beside its probe reads
+    # among them: without those an installed package reports no locales to generate.
     "**/*.yaml",
 ]
 
@@ -180,9 +187,9 @@ include = ["usersim*"]
 # vectors `gen-assets` writes beside a bank stay local, as do the judge
 # scorecards `evaluate-assets` writes there.
 "usersim.engine" = [
-    "assets/financial_services/**/embeddings.parquet",
-    "assets/financial_services/**/_eval_scorecard.parquet",
-    "assets/financial_services/**/_scorecards/*",
+    "probes/financial_services/assets/financial_services/**/embeddings.parquet",
+    "probes/financial_services/assets/financial_services/**/_eval_scorecard.parquet",
+    "probes/financial_services/assets/financial_services/**/_scorecards/*",
 ]
 
 [tool.pytest.ini_options]

usersim-0.3.0/src/usersim/asset_gen/__init__.py

@@ -0,0 +1,11 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""asset_gen: the generic dispatch for building asset banks offline.
+
+``registry.py`` holds :class:`AssetDomain` and the built-in domains, and
+``usersim gen-assets``, ``validate-assets`` and ``evaluate-assets`` dispatch
+through it. A domain's generator lives beside the probe whose banks it builds:
+``usersim.engine.probes.financial_services.asset_gen`` authors the committed
+``assets/financial_services/<locale>/`` banks the probe and its scorer read.
+"""

usersim-0.3.0/src/usersim/asset_gen/registry.py

@@ -0,0 +1,101 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Asset-domain registry: the one generic layer over per-domain asset generation.
+
+The `usersim gen-assets / validate-assets / evaluate-assets --domain <d>` CLIs
+dispatch through here. Each domain registers an :class:`AssetDomain` carrying its
+entry functions (load_spec / describe_plan / generate / validate / evaluate) + its
+region_spec directory + an ``add_gen_args`` hook that injects domain-specific CLI
+flags. The heavy domain modules are imported lazily inside the callables so
+``usersim --help`` stays fast; ``add_gen_args`` is pure argparse (no import).
+
+A built-in domain lives beside the probe whose banks it builds, in its package's
+``asset_domain.py``; an extension's domain registers under ``usersim.asset_domains``.
+Neither adds a CLI subcommand. There is deliberately NO generic
+``asset_gen/core/`` engine -- generation internals are domain-specific; only this
+thin dispatch is shared.
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Callable
+
+from usersim.engine.core.builtin_packages import builtin_modules
+
+
+@dataclass(frozen=True)
+class AssetDomain:
+    name: str
+    #: directory holding ``<locale>.yaml`` region specs for this domain.
+    region_spec_dir: Path
+    #: add domain-specific generation flags to a ``gen-assets`` parser (argparse only).
+    add_gen_args: Callable[[argparse.ArgumentParser], None]
+    #: load + validate a region spec from a path -> a domain spec object.
+    load_spec: Callable[[Any], Any]
+    #: print an offline plan summary for ``gen-assets`` (also used by --dry-run).
+    describe_plan: Callable[[Any, argparse.Namespace], None]
+    #: run live generation: ``(spec, models, out_dir, args) -> out_dir``.
+    generate: Callable[[Any, Any, Any, argparse.Namespace], Any]
+    #: run the offline validation gate: ``(bank_dir, spec | None) -> ValidationReport``.
+    validate: Callable[[Any, Any | None], Any]
+    #: run the DD-judge evaluation scorecard (network-gated); set in the asset_eval bucket.
+    evaluate: Callable[..., Any] | None = None
+
+
+# ── built-in domains ─────────────────────────────────────────────────────
+
+
+def _builtin_domains() -> dict[str, AssetDomain]:
+    """The domain each built-in package's ``asset_domain.py`` returns from its ``asset_domain()`` factory.
+
+    The module lives beside the probe whose banks it builds and imports none of its generator.
+    """
+    domains = {}
+    for module in builtin_modules("asset_domain"):
+        domain = importlib.import_module(module).asset_domain()
+        domains[domain.name] = domain
+    return domains
+
+
+# ── registry access ──────────────────────────────────────────────────────
+
+_DOMAINS: dict[str, AssetDomain] | None = None
+
+
+def _domains() -> dict[str, AssetDomain]:
+    """Built-in domains merged with any advertised under ``usersim.asset_domains``.
+
+    An entry point may resolve to an ``AssetDomain`` or to a zero-argument
+    factory returning one, as each built-in is a factory. A domain imports its
+    pipeline inside its callables; building one eagerly at import time would drag
+    those dependencies into every ``--help``.
+    """
+    global _DOMAINS
+    if _DOMAINS is None:
+        from usersim.engine.core._extensions import ASSET_DOMAINS, merge_extensions
+
+        merged = merge_extensions(ASSET_DOMAINS, _builtin_domains(), what="asset domain")
+        _DOMAINS = {name: obj if isinstance(obj, AssetDomain) else obj() for name, obj in merged.items()}
+    return _DOMAINS
+
+
+def clear_domain_cache() -> None:
+    """Drop the memoized registry — for tests that toggle extension discovery."""
+    global _DOMAINS
+    _DOMAINS = None
+
+
+def domain_names() -> tuple[str, ...]:
+    return tuple(sorted(_domains()))
+
+
+def get_domain(name: str) -> AssetDomain:
+    domains = _domains()
+    if name not in domains:
+        raise KeyError(f"unknown asset domain {name!r}; known: {sorted(domains)}")
+    return domains[name]

{usersim-0.2.0 → usersim-0.3.0}/src/usersim/cli/__init__.py

@@ -15,7 +15,7 @@ run from a script or CI without opening Jupyter:
 Each subcommand is implemented in its own module under :mod:`cli`.
 The shared pipeline-construction helpers live in :mod:`cli._pipeline`
 (used by both the CLI and the notebooks under ``notebooks/``). Model
-configuration is loaded from a TOML file via :mod:`cli._models`; a
+configuration is loaded from a TOML file via :mod:`usersim.engine.models_file`; a
 sensible default ships at ``cli/models_default.toml``.
 
 Identity: ``persona_uuid`` is a content hash of the persona, and
@@ -34,7 +34,7 @@ import logging
 import sys
 
 from usersim import __version__
-from usersim.cli._errors import ConfigError
+from usersim.engine.errors import ConfigError
 
 
 def _register_extension_commands(subparsers: argparse._SubParsersAction) -> None:
@@ -50,23 +50,32 @@ def _register_extension_commands(subparsers: argparse._SubParsersAction) -> None
     """
     import logging
 
-    from usersim.engine.core._extensions import COMMANDS, load_extensions
+    from usersim.engine.core._extensions import (
+        COMMANDS,
+        extension_source,
+        load_extensions,
+        note_extension_problem,
+    )
 
     for name, register in load_extensions(COMMANDS):
         if name in subparsers.choices:
             logging.getLogger("usersim.engine").warning(
-                "  |-- extensions: command %r is already provided by this package; ignoring the contributed one.",
+                "  |-- extensions: command %r from %s is already built in; ignoring it.",
                 name,
+                extension_source(COMMANDS, name),
             )
+            note_extension_problem(COMMANDS, name, f"command {name!r} is already built in")
             continue
         try:
             register(subparsers)
         except Exception as exc:
             logging.getLogger("usersim.engine").warning(
-                "  |-- extensions: command %r failed to register and was skipped: %s",
+                "  |-- extensions: command %r from %s failed to register and was skipped: %s",
                 name,
+                extension_source(COMMANDS, name),
                 exc,
             )
+            note_extension_problem(COMMANDS, name, f"failed to register: {exc}")
 
 
 def _build_parser() -> argparse.ArgumentParser:

usersim-0.3.0/src/usersim/cli/_bank_dir.py

@@ -0,0 +1,27 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""The folder an asset command reads or writes when it is given none."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from usersim.engine.core._assets import shipped_assets_dir
+from usersim.engine.errors import ConfigError
+
+
+def default_bank_dir(domain: str, locale: str, *, flag: str) -> Path:
+    """The ``locale`` bank in the asset folder of the probe that shares ``domain``'s name.
+
+    An asset domain builds the bank of the probe with its name, so that probe's
+    own asset folder is where the bank lives. A domain with no such probe has
+    no default folder: ``ConfigError`` asks for one with ``flag``.
+    """
+    try:
+        return shipped_assets_dir(domain) / locale
+    except LookupError:
+        raise ConfigError(
+            f"asset domain {domain!r} has no probe of that name whose asset folder could hold its bank; "
+            f"pass {flag} to name the folder"
+        ) from None