xp-libs-analytics 0.4.0__tar.gz → 0.5.0__tar.gz

{xp_libs_analytics-0.4.0 → xp_libs_analytics-0.5.0}/.github/pull_request_template.md

@@ -1,13 +1,13 @@
 ## Description and motivation
 
 ## Affected contracts / V1 operators / public API
-- [ ] Updated `QUERY_LANGUAGE_V1.md` / semantic version decision if applicable
+- [ ] Updated `docs/query-language.md` and reviewed versioning if the query contract changed
 - [ ] New/changed error codes and catalog parity (single `catalog.json`)
 - [ ] Updated agent skill and machine-valid examples if behavior changed
 
 ## Safety and threat model
 - [ ] No unsafe SQL/Python/UDF/network/file access introduced
-- [ ] Authorization and handle ownership unchanged or explicitly reviewed
+- [ ] Reviewed authorization and context-local dataset access
 - [ ] Resource budgets and input/output leakage considered
 - [ ] No real records/secrets in fixtures, logs or artifacts
 

{xp_libs_analytics-0.4.0 → xp_libs_analytics-0.5.0}/.github/workflows/ci.yml

@@ -44,6 +44,7 @@ jobs:
       - run: python scripts/check_critical_coverage.py coverage.json
       - run: python examples/run_examples.py --lint-only
       - run: python examples/run_examples.py --all
+      - run: python -m examples.registered_queries
       - run: python scripts/resource_probe.py
       - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
         if: always()

{xp_libs_analytics-0.4.0 → xp_libs_analytics-0.5.0}/CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## 0.5.0 — Parameterized registered queries
+
+- Add key-only MCP execution for host-registered YAML or typed queries. Add a typed SDK entry point that bypasses YAML parsing while retaining snapshot planning and authorization.
+- Bind strict runtime parameters into fixed YAML value positions and support host-owned Polars transformations over authorized projections.
+- Expose parameter schema discovery and optional generated MCP tools. Support native execution in the OS-limited worker using a host-provided transformation registry.
+
 
 ## 0.4.0 — Bounded string and fuzzy query matching
 
@@ -24,7 +30,7 @@
 ## 0.2.0 — local build
 
 - Rename the distribution/CLI to `xp-libs-analytics` and imports to `xp_libs_analytics`.
-- Preserve the independently usable YAML/analytics core and optional FastMCP adapter; add a bring-your-own-MCP example and import-isolation regression test.
+- Preserve the independently usable YAML analytics core and optional FastMCP adapter. Add a custom MCP adapter example and an import isolation regression test.
 - Add optional Inspect AI agent evaluations, deterministic scenario/oracle contracts, real MCP recovery seeding, semantic/result scorers, memory-only Inspect logs, sanitized reporting and skill/baseline comparison.
 - Add 105 synthetic scenarios and offline/live test separation. Live results are recorded separately from deterministic test evidence.
 

{xp_libs_analytics-0.4.0 → xp_libs_analytics-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: xp-libs-analytics
-Version: 0.4.0
+Version: 0.5.0
 Summary: Schema-on-read, in-memory analytics for agent-generated YAML queries
 Project-URL: Source, https://github.com/rrodrigoa/xp-libs-analytics
 Project-URL: Issues, https://github.com/rrodrigoa/xp-libs-analytics/issues
@@ -45,7 +45,7 @@ Description-Content-Type: text/markdown
 
 # XP Libs Analytics
 
-A Python library for bounded, in-memory snapshot analytics over JSON records with unknown, heterogeneous schemas. Queries are restricted YAML, validated with strict Pydantic models and compiled to fixed Polars expressions. The optional standalone FastMCP adapter supplies tools, an agent skill, and 28 read-only query examples.
+A Python library for bounded, in-memory snapshot analytics over JSON records with unknown, heterogeneous schemas. The engine accepts restricted YAML and host-authored typed query definitions, then plans and executes them using fixed Polars expressions. The optional standalone FastMCP adapter supports YAML authoring or execution by registered query key.
 
 Python 3.12+ · MIT · Import: `xp_libs_analytics` · Distribution: `xp-libs-analytics`
 
@@ -109,7 +109,7 @@ Input accepts JSON objects, arrays of objects, NDJSON, or Python iterables of JS
 | Grouping | Up to four scalar dimensions; missing and explicit null groups remain separate |
 | Time | Explicit RFC3339/epoch casts, absolute or relative UTC ranges, epoch-aligned tumbling/sliding windows |
 | Results | HAVING, stable sorting, limit/truncation, strict JSON and quality/provenance evidence |
-| Errors | 73 catalog codes, one canonical packaged message catalog, safe diagnostic locations |
+| Errors | 79 catalog codes, one canonical packaged message catalog, safe diagnostic locations |
 
 See the [query language](docs/query-language.md), [examples](examples/README.md), and [agent skill](skills/generic-analytics/SKILL.md). `NO_DATA` reports absent matching observations. `INSUFFICIENT_DATA` applies the host's minimum sample policy. Unknown sampling cadence yields `coverage: UNKNOWN`; counts do not imply duration.
 
@@ -166,7 +166,7 @@ def validation_tool(query_yaml: str) -> dict:
     return analytics.validate_query(query_yaml)
 ```
 
-Register those callables with your chosen server. Select the authorized execution context and identity in the host, enforce transport authentication, and use `isolated=True` for untrusted execution. The bundled FastMCP adapter is an optional convenience layer.
+A custom MCP adapter can register these callables. The host selects the authorized execution context and identity, enforces transport authentication, and uses `isolated=True` for untrusted execution. The library also includes an optional FastMCP adapter.
 
 ```sh
 xp-libs-analytics --help
@@ -178,6 +178,47 @@ Tools: `analytics_capabilities`, `analytics_list_datasets`, `analytics_load_json
 
 The local stdio server inherits its launcher's authority; one process is one local context. The default backend uses isolated workers. HTTP access fails closed unless a host supplies an authenticated `principal_resolver` and configures FastMCP transport authentication. A host-supplied `engine` deliberately selects that engine's context for local use. Hosted sessions can supply `context_resolver`, a trusted callable returning their authorized engine/context. No tool accepts a context ID or principal. The identity resolver creates distinct namespaces per authenticated principal and reapplies current field/scopes policy per request. The adapter caps cached principal contexts at 128; restarting clears local state. It is not a turnkey public multi-tenant service.
 
+### Registered query keys
+
+Hosts can expose a fixed set of queries through `create_server(registered_queries=...)`.
+The host registers the datasets and supplies the query definitions at startup.
+
+```python
+from xp_libs_analytics import AnalyticsEngine, AnalyticsQuery
+from xp_libs_analytics.mcp.server import create_server
+
+if __name__ == "__main__":
+    engine = AnalyticsEngine(isolated=True)
+    engine.register_records("observations", [{"value": 2}, {"value": 4}])
+
+    count_yaml = """version: analytics.query/v1
+dataset: {name: observations}
+aggregations: [{op: count, as: rows}]
+"""
+    sum_query = AnalyticsQuery.model_validate(
+        {
+            "version": "analytics.query/v1",
+            "dataset": {"name": "observations"},
+            "aggregations": [{"op": "sum", "field": "/value", "as": "total"}],
+        }
+    )
+
+    server = create_server(engine, registered_queries={"count": count_yaml, "sum": sum_query})
+    server.run(transport="stdio")
+```
+
+The client calls `analytics_query_by_key` with `{"key": "sum"}` and receives the standard analytics result or error. Keys are case-sensitive, start with an ASCII letter, and contain up to 64 ASCII letters, digits, underscores, or hyphens. The host distributes the allowed keys to clients. For HTTP, the host configures transport authentication and supplies a trusted context resolver that selects an engine with the registered datasets.
+
+YAML entries use restricted parsing on each call. Typed entries and direct SDK calls to `engine.query_typed(sum_query)` bypass the analytics YAML parser. FastMCP independently imports PyYAML for its skill provider. Both query forms resolve the current dataset snapshot and run authorization, cost planning, and execution on each request.
+
+Registered queries can also accept a host-defined Pydantic parameter model. Use
+`ParameterizedYamlQuery` for fixed YAML or `PolarsQuery` for a trusted Polars
+transformation. Clients pass values as `{"key": "sum", "parameters": {"minimum": 3}}`
+and can retrieve the input schema with `analytics_describe_query`. Set
+`generated_tools=True` to expose each query as an `analytics_registered_<key>` tool
+with named arguments. See [registered query configuration](docs/registered-queries.md)
+and run `python -m examples.registered_queries` for an end-to-end example.
+
 ## Development and evidence
 
 Optional real-model evaluations use Inspect AI and the actual MCP adapter. From this checkout, install `python -m pip install -e '.[dev,mcp,agent-eval]'`, then run `python -m evals.agent_skill.cli validate`. Live evaluation accepts the model connection only at execution time; see [evaluation usage](evals/agent_skill/README.md). Normal CI runs offline evaluator tests and excludes live model tests.

{xp_libs_analytics-0.4.0 → xp_libs_analytics-0.5.0}/README.md

@@ -1,6 +1,6 @@
 # XP Libs Analytics
 
-A Python library for bounded, in-memory snapshot analytics over JSON records with unknown, heterogeneous schemas. Queries are restricted YAML, validated with strict Pydantic models and compiled to fixed Polars expressions. The optional standalone FastMCP adapter supplies tools, an agent skill, and 28 read-only query examples.
+A Python library for bounded, in-memory snapshot analytics over JSON records with unknown, heterogeneous schemas. The engine accepts restricted YAML and host-authored typed query definitions, then plans and executes them using fixed Polars expressions. The optional standalone FastMCP adapter supports YAML authoring or execution by registered query key.
 
 Python 3.12+ · MIT · Import: `xp_libs_analytics` · Distribution: `xp-libs-analytics`
 
@@ -64,7 +64,7 @@ Input accepts JSON objects, arrays of objects, NDJSON, or Python iterables of JS
 | Grouping | Up to four scalar dimensions; missing and explicit null groups remain separate |
 | Time | Explicit RFC3339/epoch casts, absolute or relative UTC ranges, epoch-aligned tumbling/sliding windows |
 | Results | HAVING, stable sorting, limit/truncation, strict JSON and quality/provenance evidence |
-| Errors | 73 catalog codes, one canonical packaged message catalog, safe diagnostic locations |
+| Errors | 79 catalog codes, one canonical packaged message catalog, safe diagnostic locations |
 
 See the [query language](docs/query-language.md), [examples](examples/README.md), and [agent skill](skills/generic-analytics/SKILL.md). `NO_DATA` reports absent matching observations. `INSUFFICIENT_DATA` applies the host's minimum sample policy. Unknown sampling cadence yields `coverage: UNKNOWN`; counts do not imply duration.
 
@@ -121,7 +121,7 @@ def validation_tool(query_yaml: str) -> dict:
     return analytics.validate_query(query_yaml)
 ```
 
-Register those callables with your chosen server. Select the authorized execution context and identity in the host, enforce transport authentication, and use `isolated=True` for untrusted execution. The bundled FastMCP adapter is an optional convenience layer.
+A custom MCP adapter can register these callables. The host selects the authorized execution context and identity, enforces transport authentication, and uses `isolated=True` for untrusted execution. The library also includes an optional FastMCP adapter.
 
 ```sh
 xp-libs-analytics --help
@@ -133,6 +133,47 @@ Tools: `analytics_capabilities`, `analytics_list_datasets`, `analytics_load_json
 
 The local stdio server inherits its launcher's authority; one process is one local context. The default backend uses isolated workers. HTTP access fails closed unless a host supplies an authenticated `principal_resolver` and configures FastMCP transport authentication. A host-supplied `engine` deliberately selects that engine's context for local use. Hosted sessions can supply `context_resolver`, a trusted callable returning their authorized engine/context. No tool accepts a context ID or principal. The identity resolver creates distinct namespaces per authenticated principal and reapplies current field/scopes policy per request. The adapter caps cached principal contexts at 128; restarting clears local state. It is not a turnkey public multi-tenant service.
 
+### Registered query keys
+
+Hosts can expose a fixed set of queries through `create_server(registered_queries=...)`.
+The host registers the datasets and supplies the query definitions at startup.
+
+```python
+from xp_libs_analytics import AnalyticsEngine, AnalyticsQuery
+from xp_libs_analytics.mcp.server import create_server
+
+if __name__ == "__main__":
+    engine = AnalyticsEngine(isolated=True)
+    engine.register_records("observations", [{"value": 2}, {"value": 4}])
+
+    count_yaml = """version: analytics.query/v1
+dataset: {name: observations}
+aggregations: [{op: count, as: rows}]
+"""
+    sum_query = AnalyticsQuery.model_validate(
+        {
+            "version": "analytics.query/v1",
+            "dataset": {"name": "observations"},
+            "aggregations": [{"op": "sum", "field": "/value", "as": "total"}],
+        }
+    )
+
+    server = create_server(engine, registered_queries={"count": count_yaml, "sum": sum_query})
+    server.run(transport="stdio")
+```
+
+The client calls `analytics_query_by_key` with `{"key": "sum"}` and receives the standard analytics result or error. Keys are case-sensitive, start with an ASCII letter, and contain up to 64 ASCII letters, digits, underscores, or hyphens. The host distributes the allowed keys to clients. For HTTP, the host configures transport authentication and supplies a trusted context resolver that selects an engine with the registered datasets.
+
+YAML entries use restricted parsing on each call. Typed entries and direct SDK calls to `engine.query_typed(sum_query)` bypass the analytics YAML parser. FastMCP independently imports PyYAML for its skill provider. Both query forms resolve the current dataset snapshot and run authorization, cost planning, and execution on each request.
+
+Registered queries can also accept a host-defined Pydantic parameter model. Use
+`ParameterizedYamlQuery` for fixed YAML or `PolarsQuery` for a trusted Polars
+transformation. Clients pass values as `{"key": "sum", "parameters": {"minimum": 3}}`
+and can retrieve the input schema with `analytics_describe_query`. Set
+`generated_tools=True` to expose each query as an `analytics_registered_<key>` tool
+with named arguments. See [registered query configuration](docs/registered-queries.md)
+and run `python -m examples.registered_queries` for an end-to-end example.
+
 ## Development and evidence
 
 Optional real-model evaluations use Inspect AI and the actual MCP adapter. From this checkout, install `python -m pip install -e '.[dev,mcp,agent-eval]'`, then run `python -m evals.agent_skill.cli validate`. Live evaluation accepts the model connection only at execution time; see [evaluation usage](evals/agent_skill/README.md). Normal CI runs offline evaluator tests and excludes live model tests.

xp_libs_analytics-0.5.0/SECURITY.md

@@ -0,0 +1,15 @@
+# Security Policy
+
+XP Libs Analytics executes analytical queries over potentially sensitive JSON records. Vulnerability reports should use synthetic records and omit secrets, access tokens, personal information, private schema, and cross-context dataset metadata.
+
+## Reporting vulnerabilities
+
+Use the repository's private security advisory form (Security → Advisories → Report a vulnerability) when private reporting is available.
+
+## Supported versions
+
+Security fixes are distributed as new package versions. The [changelog](CHANGELOG.md) records version-specific changes.
+
+## Threat model priorities
+
+The library's threat model covers cross-context name resolution, snapshot identity, untrusted tool input, resource exhaustion, type and cast handling, metadata disclosure, dependency and package integrity, and unauthorized output. See the [end-to-end security review](docs/security-review.md) for controls and test evidence.

{xp_libs_analytics-0.4.0 → xp_libs_analytics-0.5.0}/docs/architecture.md

@@ -16,6 +16,8 @@ The worker memory ceiling is absolute. POSIX address-space limits include native
 
 MCP resources are read-only and allowlisted. Under modern MCP, local requests inherit launcher identity rather than a per-request connection ID. Remote hosts must supply transport authentication and a trusted principal or execution-context resolver; default HTTP data operations are denied.
 
+The optional key-only MCP surface resolves caller-supplied keys through a host-configured mapping. A YAML entry enters restricted parsing. A typed entry enters through strict model validation and skips the analytics YAML parser. Both paths bind the current snapshot under the trusted context, plan against its schema and budgets, and use the shared execution and result path.
+
 MCP never accepts agent-supplied identity or context selectors. Authenticated principal contexts are isolated and policy views preserve names while applying current scopes/field restrictions. A context resolver can instead select a host-managed session/request context. Names in other contexts are indistinguishable from nonexistent names. Local stdio deliberately has one launcher context per process. Reserved register_stream rejects without consuming sources; no continuous semantics are implied.
 
 The shared matching package owns strict rule models, Unicode normalization, bounded LIKE/Rust regex validation, per-query distinct RapidFuzz scoring and privacy-cleared diagnostics. Matching plans compile to native Polars expressions/membership without row UDFs. Typed datetime equality bypasses string matching after explicit casts. Safe independent nonfuzzy conjunctions can bound candidate work; OR/NOT branches are never pushed independently. There is no persistent derived-value cache. Regex/fuzzy and requested diagnostics default to the OS-limited worker; explain/validate counts candidates without running scorers. Optional diagnostics have an explicitly narrower post-all-filters/pre-aggregation leaf-hit scope. Entity inference is deferred; see docs/adr/entity-inference-deferred.md.

xp_libs_analytics-0.5.0/docs/registered-queries.md

@@ -0,0 +1,64 @@
+# Parameterized registered queries
+
+The host supplies `registered_queries` to `create_server`. Each entry accepts fixed
+YAML text, an `AnalyticsQuery`, a `ParameterizedYamlQuery`, or a `PolarsQuery`. The
+adapter resolves the key within the server and selects the engine through the
+trusted context. Dataset bindings remain local to that engine.
+
+Use `analytics_describe_query(key)` to obtain the parameter JSON Schema. Call
+`analytics_query_by_key(key, parameters)` with a JSON object. Omitting the object
+applies the model defaults and still enforces required fields. Fixed queries accept
+an omitted or empty object. With `generated_tools=True`, the tool
+`analytics_registered_<key>` accepts the same values as named arguments. Both
+surfaces validate requests against the same model.
+
+Parameter models use strict JSON validation. Integer and boolean values require
+their JSON types. Enums use their declared JSON values and datetimes use RFC 3339
+strings. The engine forbids undeclared fields and applies byte, depth, node,
+collection, and string limits before model validation. Effective values,
+including defaults, are also subject to the resource limits. Host models can
+enable Pydantic's `validate_default` setting to validate defaults against field
+constraints. Model validators and serializers execute as trusted host code.
+
+## Fixed YAML
+
+Create `ParameterizedYamlQuery(yaml, parameters=Model)` and invoke it directly with
+`engine.query_parameterized(definition, values)` or register it with the MCP
+adapter. Place `{$param: minimum}` where the query expects a whole literal value.
+Supported positions include predicate values, membership lists, range bounds,
+matching pattern values and thresholds, and time range bounds. Placeholder names
+refer to Python model fields. A bound string is a literal even if it contains YAML
+syntax. The dataset name, field paths, operators, aggregate aliases, and other
+structural parts come from the fixed template. The resulting query goes through
+the normal typed planner.
+
+## Native Polars
+
+Create `PolarsQuery(dataset, fields, transform, parameters, output_schema)`
+and invoke it with `engine.query_polars(definition, values)` or register it with the
+MCP adapter. The host supplies the dataset name, JSON Pointer paths, and a callable
+that takes a Polars DataFrame and the validated Pydantic model, then returns a
+Polars DataFrame. The input contains every row from the leased snapshot, a
+`row_index`, one column per authorized scalar path, and one Boolean
+`present:<path>` column per path. Missing values have a false presence flag and a
+null cell. Explicit nulls have a true presence flag and a null cell. Container and
+mixed-type projections are rejected.
+
+The output must match the declared column names, order, and types. Supported
+output types are `integer` (Int64), `float` (Float64), `boolean`, `string`,
+`datetime` (timezone-aware), and `null`. Values can be nullable. Oversized outputs
+return a structured error. `max_output_rows` limits returned rows after the
+transformation, and the engine enforces its result byte limit.
+
+For isolated execution, construct the engine with `isolated=True` and an explicit
+`native_transforms={"count": count_values}` registry. The transformation and the
+parameter model must be top-level definitions in importable host modules. The
+worker resolves the host-selected references after setting its resource limits.
+MCP arguments supply parameter values only. Native transformations have the
+privileges of the host or worker process and must come from trusted application
+code. Use operating-system or container controls to restrict process access as
+needed. The OS worker limits apply to execution, and the host bounds projection
+and parameter processing.
+
+Run `python -m examples.registered_queries` from the source checkout with the MCP
+extra installed. The example uses one parameter model with both query types.

{xp_libs_analytics-0.4.0 → xp_libs_analytics-0.5.0}/docs/security-review.md

@@ -1,15 +1,29 @@
-# V1 security review
+# End-to-end security review for V1
 
-The local review covered YAML/JSON parsing, scalar typing, field/presence normalization, ownership, metadata, quotas, compilation, native worker failure, tools/resources, and release permissions. Executed tests and OS probes are recorded in `docs/evidence`.
+XP Libs Analytics executes bounded snapshot queries over JSON records. The host controls which records enter the engine, which context handles each request, and what that context can read. YAML expresses the query. The Python SDK enforces the same query contract for direct callers and callers using the optional MCP adapter.
+
+```text
+Trusted host ── records ──► bounded admission ──► immutable snapshot
+     │                                                  │
+     └──────── context, permissions and limits ───┐     │
+                                                   ▼     ▼
+Caller ── YAML via SDK or MCP ──► parse and type ──► authorize and plan
+                                                          │
+                                                    approved plan
+                                                          ▼
+Caller ◄── bounded JSON or coded error ◄──────────── execute
+```
+
+The review covers these boundaries from admission through response, as well as tool and release permissions. Executed tests and OS probes are recorded in [verification evidence](evidence/verification.md).
 
 - YAML is byte/event/depth/node/collection/scalar bounded before construction. Tags, directives, anchors, aliases, merges and duplicate keys are rejected. Scalar resolvers are isolated from global PyYAML behavior. Query keys cannot introduce SQL, code, plugins, host paths or network sources.
 - Strict disjoint AST variants forbid unused and unknown fields. Public errors use only the packaged catalog. Invalid unknown-key locations are redacted; parser excerpts, records and backend exception strings are never returned.
 - Snapshots copy admitted records, preserve missing versus null and examine all rows for typing/casts. Integer precision/overflow and nonfinite results are checked. Opaque internal column names prevent output alias collisions. Casting precedes WHERE and cannot bypass field permissions.
-- Ownership is enforced for describe/validate/explain/query/release. Hidden and nonexistent paths are indistinguishable for field-restricted principals; foreign and unknown handles receive identical denial. Host principal changes select immutable policy configurations instead of mutating shared request authority.
+- Ownership is enforced for dataset discovery, description, query validation, planning, execution, and unregistering. Hidden and nonexistent paths are indistinguishable for field-restricted principals. Dataset names outside the current context are indistinguishable from unknown names. The engine applies the host's current policy without mutating shared request authority.
 - Input bytes, rows, paths, record size, recursion, store bytes, handle count, result bytes, groups, distinct cardinality, windows, overlap and expanded rows have explicit budgets. Quotas/concurrency are shared by owner across SDK views of the same store. A low result limit cannot bypass planning.
 - Workers have OS CPU/memory ceilings and parent-enforced deadlines; native abort/pipe loss returns a canonical error. Windows Job Object and POSIX rlimit adapters have fail-closed API tests. OS probes demonstrate memory denial and CPU termination; host/container restrictions remain necessary for host-side admission and egress.
 - Default MCP local identity follows launcher authority under modern per-request connections. Default HTTP data operations fail closed. Authenticated remote hosting requires a trusted resolver and transport authentication; this package does not claim to configure an identity provider. Upload is disabled by default. Resource names are an allowlist, with no caller filesystem routing.
 - Observability hooks contain only IDs/hashes/status/timing, never records, query text or field values. Hook failures cannot change query outcomes. The package performs no default payload logging.
 - CI does not use `pull_request_target`. Actions are commit-pinned. Only protected release jobs have OIDC write authority; TestPyPI promotion compares wheel bytes and checks hashes before PyPI. Account-level protection still requires maintainers.
 
-Scope limits: this is a local code/test review, not an external penetration certification. No live internet-facing authentication deployment or hosted GitHub security scan was performed. The deterministic agent suite verifies tool workflows and evidence handling; it does not measure a live language model's query-generation accuracy. Exact decimal, continuous queries and multi-input semantics are outside V1.
+The evidence for this review comes from source inspection and local tests. The deterministic agent suite verifies tool workflows and result handling. Live model accuracy, internet-facing authentication deployments, and independent penetration testing fall outside this review. Exact decimal arithmetic, continuous queries, and multi-input execution fall outside V1.

{xp_libs_analytics-0.4.0 → xp_libs_analytics-0.5.0}/docs/yaml-security.md

@@ -1,6 +1,6 @@
 # Restricted YAML Query Parser — Normative Security Contract (V1)
 
-**Scope:** YAML is the only public query authoring format in V1. JSON/NDJSON records remain the analytics input, and query results/error envelopes remain JSON. This file governs YAML decoding before typed Pydantic validation. It does not permit arbitrary general-purpose YAML.
+This document defines restricted YAML decoding before typed Pydantic validation. It applies to caller-authored YAML and host-registered YAML queries. Host-authored typed queries enter through model validation. Both paths operate on snapshots of JSON or NDJSON records and return JSON results or errors.
 
 ## Decision and rationale
 
@@ -34,7 +34,7 @@
 Parser fidelity:
 
 1. Enforce UTF-8 and raw byte size **before** parser work. Suggested default `max_query_bytes=65_536`, configurable lower by trusted policy. Also bound string length (`max_scalar_chars=1_024` except perhaps approved fixed query metadata), nesting (`max_yaml_depth=16`), total nodes/events (`max_yaml_nodes=4_096`), mapping entries and sequences (`max_yaml_collection_items=1_024`); versioned server defaults are normative once selected.
-2. Use PyYAML's `SafeLoader` as a *base class for a custom loader*. Never use `yaml.Loader`, `FullLoader`, `unsafe_load`, `!!python/*` or arbitrary constructors. `yaml.safe_load` **by itself is NOT enough**: it does not enforce this entire subset, does not provide our duplicate-key rules, and its default implicit scalar resolver differs from the desired contract.
+2. The restricted loader extends PyYAML's `SafeLoader` with the library's duplicate-key rules, scalar resolution, and allowed syntax. Query parsing excludes `yaml.Loader`, `FullLoader`, `unsafe_load`, `!!python/*`, and arbitrary constructors.
 3. Preflight parser events **with budgets**. Reject `AliasEvent`, any `anchor` on scalar/sequence/mapping events, explicit tags, directives, multi-documents and unsupported events. Stop after the first permitted document and reject further documents. Parse deterministically, with a hard deadline/worker budget and bounded untrusted text. Avoid regex-only prechecks: anchors, aliases and tags must be detected by parsing, not substring search (quoted `"*alias"` is just a string).
 4. Clone only the safe scalar resolver table before modifying it; don't mutate global `yaml.SafeLoader` implicit resolvers. Install a narrowly documented set of JSON-compatible scalar recognizers: lowercase true/false; exact null; JSON-style decimal integers and finite decimal/exponent floats; no timestamp or binary auto-resolution; no YAML 1.1 aliases for booleans. Treat leading-zero or plus-prefixed numerals as strings unless explicitly supported by a later version. Validate `int64` bounds and numeric finiteness at the typed semantic layer; never accept Python bool as numeric.
 5. Override mapping constructor to reject duplicate keys **at every depth**, including flow-style maps, before converting to dict; require string keys, no merge key, and preserve source marks in a safe `{query_path → (line,column)}` index. Convert positions to 1-based line/column. Use local source-mark metadata, never return raw YAML excerpts or tokens. A duplicate-key failure may include the duplicate key's **safe path** and line/column, but must not echo field values.

{xp_libs_analytics-0.4.0 → xp_libs_analytics-0.5.0}/evals/agent_skill/README.md

@@ -18,7 +18,7 @@ For an Ollama deployment, install Ollama separately, start it with `ollama serve
 
 ## Supply connection settings only at runtime
 
-Set MODEL_ID in your process/shell to an installed native-tool-capable model ID. There is no model default.
+Set `MODEL_ID` in the evaluation process environment to identify an installed model with native tool support. The evaluation runner requires an explicit model ID.
 
 The CLI prompts for an API base URL if `--endpoint` is omitted. It never saves that answer. For example:
 
@@ -32,7 +32,7 @@ python -m evals.agent_skill.cli run --model "$MODEL_ID" --suite adversarial
 python -m evals.agent_skill.cli compare --model "$MODEL_ID" --scenario aggregate-count --repetitions 3
 ```
 
-To pass a runtime variable explicitly, use `--endpoint "$API_BASE"` in your shell. The variable's value is supplied by you, never stored in a config or `.env` file. An optional credential can be supplied through the process-only `XP_ANALYTICS_EVAL_API_KEY` environment variable; clear it after use. Shell history/process listings are controlled by your shell and OS; interactive prompting avoids putting the address in command history.
+Pass the endpoint at runtime with `--endpoint "$API_BASE"` or enter it at the interactive prompt. The prompt keeps the address out of command history. Supply any required credential through the process environment variable `XP_ANALYTICS_EVAL_API_KEY` and clear it after use.
 
 Other controls: `--seed`, `--temperature` (default zero), `--max-turns`, `--max-tool-calls`, `--timeout`, `--report`, `--compare-skill`, `--verbose`, and explicit `--traces` for redacted message traces. Runs are serial; concurrency defaults to one. Preflight checks model listing where available, native tool calling with a harmless echo tool, actual analytics MCP discovery, and skill references. It fails without silently emulating tools.
 

{xp_libs_analytics-0.4.0 → xp_libs_analytics-0.5.0}/examples/README.md

@@ -11,7 +11,7 @@ python examples/run_examples.py --all
 
 `--lint-only` applies the restricted parser and strict Pydantic AST. `--all` additionally registers synthetic records, registers them under example_data, semantically validates and executes every query, and unregisters each binding. Runtime errors fail the runner.
 
-Each query is a complete document. `dataset.name: example_data` selects the logical name registered by the runner. You can run the same YAML unchanged after registering your own records as example_data. No internal ID is injected into YAML. Inputs are `data/events.json`, equivalent `events.ndjson`, `mixed_types.json`, and `matching_events.json`. Fields are illustrative discovered paths, never built-in business rules.
+Each query is a complete document. `dataset.name: example_data` selects the logical name registered by the runner. The same YAML can execute against any snapshot registered as `example_data` with compatible fields and types. The YAML uses the logical name throughout. Example inputs are `data/events.json`, its equivalent `events.ndjson`, `mixed_types.json`, and `matching_events.json`. The engine discovers their field paths from the admitted records.
 
 | Example | Behavior |
 |---|---|

xp_libs_analytics-0.5.0/examples/registered_queries.py

@@ -0,0 +1,60 @@
+"""Run both parameterized query forms through a local MCP client."""
+
+import asyncio
+from typing import Annotated
+
+import polars as pl
+from fastmcp import Client
+from pydantic import BaseModel, Field
+
+from xp_libs_analytics import AnalyticsEngine, ParameterizedYamlQuery, PolarsQuery
+from xp_libs_analytics.mcp.server import create_server
+
+
+class Minimum(BaseModel):
+    minimum: Annotated[int, Field(ge=0, le=100)] = 2
+
+
+def count_values(frame: pl.DataFrame, parameters: BaseModel) -> pl.DataFrame:
+    assert isinstance(parameters, Minimum)  # noqa: S101 -- host-owned example contract
+    return frame.filter(pl.col("/value") >= parameters.minimum).select(
+        pl.len().cast(pl.Int64).alias("count")
+    )
+
+
+YAML_COUNT = ParameterizedYamlQuery(
+    """version: analytics.query/v1
+dataset: {name: observations}
+filters:
+  - {field: /value, op: gte, value: {$param: minimum}}
+aggregations:
+  - {op: count, as: count}
+""",
+    Minimum,
+)
+POLARS_COUNT = PolarsQuery("observations", ("/value",), count_values, Minimum, {"count": "integer"})
+
+
+async def main() -> None:
+    engine = AnalyticsEngine()
+    engine.register_records("observations", [{"value": 1}, {"value": 3}, {"value": 4}])
+    server = create_server(
+        engine,
+        registered_queries={"yaml_count": YAML_COUNT, "polars_count": POLARS_COUNT},
+        generated_tools=True,
+    )
+    async with Client(server) as client:
+        yaml_result = await client.call_tool(
+            "analytics_query_by_key", {"key": "yaml_count", "parameters": {"minimum": 3}}
+        )
+        polars_result = await client.call_tool("analytics_registered_polars_count", {"minimum": 3})
+        for result in (yaml_result, polars_result):
+            if result.structured_content is None or result.structured_content["rows"] != [
+                {"count": 2}
+            ]:
+                raise RuntimeError("Unexpected registered query result")
+            print(result.structured_content["rows"])
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

{xp_libs_analytics-0.4.0 → xp_libs_analytics-0.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "xp-libs-analytics"
-version = "0.4.0"
+version = "0.5.0"
 description = "Schema-on-read, in-memory analytics for agent-generated YAML queries"
 readme = "README.md"
 requires-python = ">=3.12"

{xp_libs_analytics-0.4.0 → xp_libs_analytics-0.5.0}/scripts/smoke.py

@@ -6,7 +6,9 @@ from dataclasses import replace
 from importlib.metadata import distribution
 from importlib.resources import files
 
-from xp_libs_analytics import AnalyticsEngine
+from pydantic import BaseModel
+
+from xp_libs_analytics import AnalyticsEngine, ParameterizedYamlQuery
 
 assert "fastmcp" not in sys.modules
 assert any(
@@ -14,7 +16,7 @@ assert any(
     for p in distribution("xp-libs-analytics").files
 )
 catalog = json.loads(files("xp_libs_analytics.errors").joinpath("catalog.json").read_text())
-assert len(catalog) == 73
+assert len(catalog) == 79
 bundle = files("xp_libs_analytics").joinpath("skills/generic-analytics")
 assert bundle.joinpath("SKILL.md").is_file()
 assert bundle.joinpath("references/query-language-full.md").is_file()
@@ -22,6 +24,22 @@ assert bundle.joinpath("references/string-matching-full.md").is_file()
 assert bundle.joinpath("examples/queries/01_count_all.yaml").is_file()
 assert len(list(bundle.joinpath("examples/queries").iterdir())) == 28
 engine = AnalyticsEngine()
+
+
+class Parameters(BaseModel):
+    minimum: int = 1
+
+
+engine.register_records("registered_smoke", [{"value": 0}, {"value": 2}])
+registered = ParameterizedYamlQuery(
+    """version: analytics.query/v1
+dataset: {name: registered_smoke}
+filters: [{field: /value, op: gte, value: {$param: minimum}}]
+aggregations: [{op: count, as: rows}]
+""",
+    Parameters,
+)
+assert engine.query_parameterized(registered)["rows"] == [{"rows": 1}]
 handle = engine.register_records("dataset_1", [{"x": 2}, {"x": 4}, {}])
 text = f"""version: analytics.query/v1
 dataset: {{name: {handle.name}}}

{xp_libs_analytics-0.4.0 → xp_libs_analytics-0.5.0}/src/xp_libs_analytics/__init__.py

@@ -1,8 +1,10 @@
 """Bounded snapshot analytics over arbitrary JSON records and restricted YAML."""
 
+from xp_libs_analytics.engine.native import PolarsQuery
 from xp_libs_analytics.errors import AnalyticsError
 from xp_libs_analytics.models import AnalyticsQuery
 from xp_libs_analytics.policy import Limits, Principal
+from xp_libs_analytics.registered import ParameterizedYamlQuery
 from xp_libs_analytics.registry import DatasetInfo, DatasetRegistry
 from xp_libs_analytics.sdk import AnalyticsEngine
 
@@ -13,6 +15,8 @@ __all__ = [
     "DatasetInfo",
     "DatasetRegistry",
     "Limits",
+    "ParameterizedYamlQuery",
+    "PolarsQuery",
     "Principal",
 ]
-__version__ = "0.4.0"
+__version__ = "0.5.0"