apcore-toolkit 0.4.2__tar.gz → 0.5.0__tar.gz

{apcore_toolkit-0.4.2 → apcore_toolkit-0.5.0}/CHANGELOG.md

@@ -2,6 +2,46 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.5.0] - 2026-04-21
+
+### Added
+
+- **`BindingLoader`** (`apcore_toolkit.binding_loader`) — parses `.binding.yaml` files back into `ScannedModule` objects, the inverse of `YAMLWriter`. Unlike `apcore.BindingLoader`, this is pure data: no target import, no Registry mutation. Enables verification, merging, diffing, and round-trip workflows.
+  - `load(path, *, strict=False)` — single file or directory of `*.binding.yaml`.
+  - `load_data(data, *, strict=False)` — pre-parsed YAML dict.
+  - Loose mode (default): only `module_id + target` required; missing fields use defaults.
+  - Strict mode: additionally requires `input_schema + output_schema`.
+  - `spec_version` validated; missing/unsupported versions WARN but do not fail.
+  - `annotations` parsed via `ModuleAnnotations.from_dict`; malformed values degrade to `None` with WARN.
+  - `examples` entries validated individually; malformed ones skipped with WARN.
+  - `BindingLoadError` exception carries `file_path`, `module_id`, `missing_fields`, `reason`.
+- **`ScannedModule.display`** — new top-level optional field (`dict | None`) holding the sparse display overlay for binding YAML persistence. Distinct from `metadata["display"]` (resolved form produced by `DisplayResolver`).
+- **New feature doc**: `docs/features/binding-loader.md`; `display-overlay.md` and `output-writers.md` updated.
+
+### Changed
+
+- **`YAMLWriter._build_binding`** — emits top-level `display:` key only when `ScannedModule.display is not None` (skip when None keeps output clean).
+- **`serializers.module_to_dict`** — includes `display` key in output.
+- **`AIEnhancer._build_prompt`** — confidence template is now built dynamically from `gaps`. When `annotations` is in gaps, the prompt requests per-field confidence for every `_ANNOTATION_FIELD_VALIDATORS` field (`annotations.readonly`, `annotations.streaming`, `annotations.cache_ttl`, ...). Previously the template hard-coded `{"description": 0.0, "documentation": 0.0}` only, causing all annotation-field confidence lookups to fall back to `0.0` and fail the threshold check — annotation enhancement silently never took effect. Fixes symmetry with `_enhance_module`'s `ann_conf.get(f"annotations.{field_name}", ...)` read path.
+
+### Dependencies
+
+- **`apcore >= 0.19.0`** — picks up the expanded `ModuleAnnotations` (12 fields incl. `streaming`, `cacheable`, `cache_ttl`, `cache_key_fields`, `paginated`, `pagination_style`, `extra`). No toolkit code changes were needed for the type itself — `_build_annotation_field_validators` reflects the updated dataclass automatically.
+
+### Tests
+
+- +34 new tests: 24 for `BindingLoader` (parsing, strict/loose modes, spec_version, file & directory loading, round-trip with `YAMLWriter`), 5 for the prompt confidence block, and 5 hardening tests (display deep-copy, malformed-shape warn, recursive glob, UTF-8 encoding, null-field error wording).
+- Updated `test_field_count` (13 → 14) and `test_all_expected_keys` for the new `display` field.
+- Total suite: 440 tests.
+
+### Hardening (post-review)
+
+- **`BindingLoader`**: warns (rather than silently drops) malformed `display` values that are not a mapping; `load()` gained a `recursive: bool = False` kwarg for nested binding layouts; `read_text` now forces UTF-8 decoding so non-ASCII aliases round-trip on non-UTF-8 locales; required-field validation now rejects wrong-type scalars (e.g. `module_id: 42`, `target: true`) and empty strings in addition to absent/null, matching the Rust loader's contract — error wording is "missing or invalid required fields"; nested `input_schema`/`output_schema`/`metadata` are now deep-copied via `copy.deepcopy` so caller mutation does not leak back into the parsed YAML source graph.
+- **`YAMLWriter`**: `display` is now deep-copied into the emitted binding (defensive parity with the TypeScript/Rust writers) so post-write mutation of `ScannedModule.display` cannot leak into the file. File writes are now atomic: the payload is written to `<name>.<pid>.tmp`, `fsync`ed, then `os.replace`d onto the final path (matches the TypeScript `tmp + rename` and Rust `tmp + sync_all + rename` writers). A process crash mid-write no longer leaves a partial YAML file that `BindingLoader` would fail to parse. A pre-write check refuses to overwrite a symlink at the target path (defence-in-depth against TOCTOU).
+- **`BaseScanner.deduplicate_ids`**: pre-scans all input `module_id`s so generated `_N` suffixes never collide with an ID already present in the input. Input `[a, a, a_2]` now yields `[a, a_3, a_2]` instead of the previous buggy `[a, a_2, a_2]`. Matches the TypeScript and Rust implementations.
+- **`resolve_target` / `RegistryWriter.write`**: new `allowed_prefixes: list[str] | None` kwarg (forwarded from `RegistryWriter.write` through `_to_function_module` to `resolve_target`). When set, `resolve_target` rejects any module path outside the listed prefixes **before** calling `importlib.import_module`, raising `PermissionError`. Mitigates arbitrary-code-execution via forged binding files (e.g. a malicious `target: "os:system"` injected into untrusted YAML). Parity with the TypeScript SDK's `allowedPrefixes` option, adapted to Python's module-name import model. Boundary-aware: `"myapp"` permits `myapp.views` but NOT `myappx.foo`. Rust does not need this because `resolve_target` is parse-only and the `HandlerFactory` is the security boundary.
+- **`ScannedModule.display`**: moved to the END of the dataclass so existing positional `ScannedModule(...)` callers are not broken by the new field.
+
 ## [0.4.1] - 2026-03-25
 
 ### Added

apcore_toolkit-0.5.0/LICENSE

@@ -0,0 +1,17 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   Copyright 2024 AI Partner Up
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

{apcore_toolkit-0.4.2 → apcore_toolkit-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: apcore-toolkit
-Version: 0.4.2
+Version: 0.5.0
 Summary: Shared scanner, schema extraction, and output toolkit for apcore framework adapters
 Project-URL: Homepage, https://aiperceivable.com
 Project-URL: Repository, https://github.com/aiperceivable/apcore-toolkit-python
@@ -8,9 +8,10 @@ Project-URL: Documentation, https://github.com/aiperceivable/apcore-toolkit-pyth
 Project-URL: Issues, https://github.com/aiperceivable/apcore-toolkit-python/issues
 Author-email: aiperceivable <tercel.yi@gmail.com>
 License-Expression: Apache-2.0
+License-File: LICENSE
 Keywords: apcore,mcp,openapi,pydantic,scanner,schema,toolkit,yaml
 Requires-Python: >=3.11
-Requires-Dist: apcore>=0.18.0
+Requires-Dist: apcore>=0.19.0
 Requires-Dist: pydantic>=2.0
 Requires-Dist: pyyaml>=6.0
 Provides-Extra: dev
@@ -18,10 +19,12 @@ Requires-Dist: apdev[dev]>=0.2.1; extra == 'dev'
 Requires-Dist: httpx>=0.24; extra == 'dev'
 Requires-Dist: mypy>=1.0; extra == 'dev'
 Requires-Dist: pytest-cov>=4.0; extra == 'dev'
-Requires-Dist: pytest>=7.0; extra == 'dev'
-Requires-Dist: ruff>=0.1; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.9; extra == 'dev'
 Provides-Extra: http-proxy
 Requires-Dist: httpx>=0.24; extra == 'http-proxy'
+Provides-Extra: json-schema
+Requires-Dist: jsonschema>=4.0; extra == 'json-schema'
 Description-Content-Type: text/markdown
 
 <div align="center">
@@ -48,6 +51,8 @@ pip install apcore-toolkit
 | `ScannedModule` | Canonical dataclass representing a scanned endpoint |
 | `BaseScanner` | Abstract base class for framework scanners with filtering and deduplication |
 | `YAMLWriter` | Generates `.binding.yaml` files for `apcore.BindingLoader` |
+| `BindingLoader` | Parses `.binding.yaml` files back into `ScannedModule` objects (pure-data inverse of `YAMLWriter`, with loose/strict modes) |
+| `BindingLoadError` | Exception raised when binding parsing fails; carries `file_path`, `module_id`, `missing_fields`, `reason` |
 | `PythonWriter` | Generates `@module`-decorated Python wrapper files |
 | `RegistryWriter` | Registers modules directly into an `apcore.Registry` |
 | `HTTPProxyRegistryWriter` | Registers HTTP proxy modules that forward requests to a running API |
@@ -249,7 +254,7 @@ Input and output schemas are inferred from PEP 484 type annotations. Use `includ
 ## Requirements
 
 - Python >= 3.11
-- apcore >= 0.14.0
+- apcore >= 0.19.0
 - pydantic >= 2.0
 - PyYAML >= 6.0
 

{apcore_toolkit-0.4.2 → apcore_toolkit-0.5.0}/README.md

@@ -22,6 +22,8 @@ pip install apcore-toolkit
 | `ScannedModule` | Canonical dataclass representing a scanned endpoint |
 | `BaseScanner` | Abstract base class for framework scanners with filtering and deduplication |
 | `YAMLWriter` | Generates `.binding.yaml` files for `apcore.BindingLoader` |
+| `BindingLoader` | Parses `.binding.yaml` files back into `ScannedModule` objects (pure-data inverse of `YAMLWriter`, with loose/strict modes) |
+| `BindingLoadError` | Exception raised when binding parsing fails; carries `file_path`, `module_id`, `missing_fields`, `reason` |
 | `PythonWriter` | Generates `@module`-decorated Python wrapper files |
 | `RegistryWriter` | Registers modules directly into an `apcore.Registry` |
 | `HTTPProxyRegistryWriter` | Registers HTTP proxy modules that forward requests to a running API |
@@ -223,7 +225,7 @@ Input and output schemas are inferred from PEP 484 type annotations. Use `includ
 ## Requirements
 
 - Python >= 3.11
-- apcore >= 0.14.0
+- apcore >= 0.19.0
 - pydantic >= 2.0
 - PyYAML >= 6.0
 

{apcore_toolkit-0.4.2 → apcore_toolkit-0.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "apcore-toolkit"
-version = "0.4.2"
+version = "0.5.0"
 description = "Shared scanner, schema extraction, and output toolkit for apcore framework adapters"
 requires-python = ">=3.11"
 readme = "README.md"
@@ -23,17 +23,18 @@ keywords = [
     "toolkit",
 ]
 dependencies = [
-    "apcore>=0.18.0",
+    "apcore>=0.19.0",
     "pydantic>=2.0",
     "PyYAML>=6.0",
 ]
 
 [project.optional-dependencies]
 http-proxy = ["httpx>=0.24"]
+json-schema = ["jsonschema>=4.0"]
 dev = [
-    "pytest>=7.0",
+    "pytest>=8.0",
     "pytest-cov>=4.0",
-    "ruff>=0.1",
+    "ruff>=0.9",
     "mypy>=1.0",
     "apdev[dev]>=0.2.1",
     "httpx>=0.24",

{apcore_toolkit-0.4.2 → apcore_toolkit-0.5.0}/src/apcore_toolkit/__init__.py

@@ -6,6 +6,7 @@ Public API re-exports for convenient access to core types and utilities.
 from importlib.metadata import PackageNotFoundError
 from importlib.metadata import version as _get_version
 from apcore_toolkit.ai_enhancer import AIEnhancer, Enhancer
+from apcore_toolkit.binding_loader import BindingLoader, BindingLoadError
 from apcore_toolkit.display import DisplayResolver
 from apcore_toolkit.formatting import to_markdown
 from apcore_toolkit.openapi import (
@@ -32,10 +33,23 @@ from apcore_toolkit.output.verifiers import (
 from apcore_toolkit.output.yaml_writer import YAMLWriter
 from apcore_toolkit.pydantic_utils import flatten_pydantic_params, resolve_target
 from apcore_toolkit.convention_scanner import ConventionScanner
-from apcore_toolkit.scanner import BaseScanner
+from apcore_toolkit.http_verb_map import (
+    SCANNER_VERB_MAP,
+    extract_path_param_names,
+    generate_suggested_alias,
+    has_path_params,
+    resolve_http_verb,
+    substitute_path_params,
+)
+from apcore_toolkit.scanner import (
+    BaseScanner,
+    deduplicate_ids,
+    filter_modules,
+    infer_annotations_from_method,
+)
 from apcore_toolkit.schema_utils import enrich_schema_descriptions
 from apcore_toolkit.serializers import annotations_to_dict, module_to_dict, modules_to_dicts
-from apcore_toolkit.types import ScannedModule
+from apcore_toolkit.types import ScannedModule, clone_module, create_scanned_module
 
 try:
     __version__ = _get_version("apcore-toolkit")
@@ -44,6 +58,8 @@ except PackageNotFoundError:
 
 __all__ = [
     "AIEnhancer",
+    "BindingLoadError",
+    "BindingLoader",
     "DisplayResolver",
     "BaseScanner",
     "ConventionScanner",
@@ -54,6 +70,7 @@ __all__ = [
     "PythonWriter",
     "RegistryVerifier",
     "RegistryWriter",
+    "SCANNER_VERB_MAP",
     "ScannedModule",
     "SyntaxVerifier",
     "Verifier",
@@ -63,17 +80,27 @@ __all__ = [
     "YAMLVerifier",
     "YAMLWriter",
     "annotations_to_dict",
+    "clone_module",
+    "create_scanned_module",
+    "deduplicate_ids",
     "deep_resolve_refs",
     "enrich_schema_descriptions",
     "extract_input_schema",
     "extract_output_schema",
+    "extract_path_param_names",
+    "filter_modules",
     "flatten_pydantic_params",
+    "generate_suggested_alias",
     "get_writer",
+    "has_path_params",
+    "infer_annotations_from_method",
     "module_to_dict",
     "modules_to_dicts",
+    "resolve_http_verb",
     "resolve_ref",
     "resolve_schema",
     "resolve_target",
     "run_verifier_chain",
+    "substitute_path_params",
     "to_markdown",
 ]

apcore_toolkit-0.5.0/src/apcore_toolkit/_type_mapping.py

@@ -0,0 +1,19 @@
+"""Shared Python ↔ JSON Schema type vocabulary.
+
+Single source of truth for the 6-type mapping used by ConventionScanner
+(Python→JSON Schema) and PythonWriter (JSON Schema→Python). Adding a new
+type here propagates to both automatically.
+"""
+
+from __future__ import annotations
+
+PYTHON_TO_JSON_SCHEMA: dict[str, str] = {
+    "str": "string",
+    "int": "integer",
+    "float": "number",
+    "bool": "boolean",
+    "list": "array",
+    "dict": "object",
+}
+
+JSON_SCHEMA_TO_PYTHON: dict[str, str] = {v: k for k, v in PYTHON_TO_JSON_SCHEMA.items()}

{apcore_toolkit-0.4.2 → apcore_toolkit-0.5.0}/src/apcore_toolkit/ai_enhancer.py

@@ -111,6 +111,11 @@ class AIEnhancer:
         timeout: int | None = None,
     ) -> None:
         self.endpoint = endpoint or os.environ.get("APCORE_AI_ENDPOINT", _DEFAULT_ENDPOINT)
+        from urllib.parse import urlparse as _urlparse
+
+        _parsed = _urlparse(self.endpoint)
+        if _parsed.scheme not in ("http", "https"):
+            raise ValueError(f"APCORE_AI_ENDPOINT must use http or https scheme, got: {self.endpoint!r}")
         self.model = model or os.environ.get("APCORE_AI_MODEL", _DEFAULT_MODEL)
         self.threshold = (
             threshold if threshold is not None else self._parse_float_env("APCORE_AI_THRESHOLD", _DEFAULT_THRESHOLD)
@@ -158,10 +163,13 @@ class AIEnhancer:
         For each module, identifies missing fields and calls the SLM to
         generate them. Only fields above the confidence threshold are applied.
 
-        Modules with gaps are collected into batches of ``batch_size``
-        (configured via ``APCORE_AI_BATCH_SIZE``, default 5). Each batch
-        shares a single prompt/API call where possible, reducing round-trips.
-        When batch_size is 1, behaviour is identical to per-module processing.
+        ``batch_size`` (configured via ``APCORE_AI_BATCH_SIZE``, default 5)
+        currently controls only the outer iteration granularity — each
+        module still produces its own prompt and API call. The setting
+        is retained so a future implementation can coalesce prompts
+        without changing the caller-facing configuration. When
+        ``batch_size`` is 1, behaviour is identical to per-module
+        processing. **It does not currently reduce API round-trips.**
 
         Args:
             modules: List of ScannedModule instances (post-scan).
@@ -182,15 +190,13 @@ class AIEnhancer:
                 results.append(module)
                 pending.append((idx, module, gaps))
 
-        # Process pending modules in batches
-        for batch_start in range(0, len(pending), self.batch_size):
-            batch = pending[batch_start : batch_start + self.batch_size]
-            for idx, module, gaps in batch:
-                try:
-                    enhanced = self._enhance_module(module, gaps)
-                    results[idx] = enhanced
-                except Exception:
-                    logger.warning("AI enhancement failed for %s, keeping original", module.module_id, exc_info=True)
+        # TODO: coalesce batch_size modules into a single API call to reduce round-trips
+        for idx, module, gaps in pending:
+            try:
+                enhanced = self._enhance_module(module, gaps)
+                results[idx] = enhanced
+            except Exception:
+                logger.error("AI enhancement failed for %s, keeping original", module.module_id, exc_info=True)
 
         return results
 
@@ -217,35 +223,63 @@ class AIEnhancer:
         confidence: dict[str, float] = {}
         warnings: list[str] = list(module.warnings)
 
-        # Apply description if above threshold
-        if "description" in gaps and "description" in parsed:
-            desc_conf = parsed.get("confidence", {}).get("description", 0.0)
-            confidence["description"] = desc_conf
-            if desc_conf >= self.threshold:
-                updates["description"] = parsed["description"]
+        # Guard: SLM may return confidence as a non-dict (e.g. "high" or 1).
+        # Treat any non-dict value as absent — all fields default to 0.0.
+        confidence_raw = parsed.get("confidence")
+        if confidence_raw is not None and not isinstance(confidence_raw, dict):
+            logger.warning(
+                "Module '%s': SLM returned non-dict 'confidence' (%s) — treating as absent.",
+                module.module_id,
+                type(confidence_raw).__name__,
+            )
+        confidence_parsed: dict[str, Any] = confidence_raw if isinstance(confidence_raw, dict) else {}
+
+        def _apply_simple(field: str) -> None:
+            """Apply a simple scalar field from parsed SLM output if confidence is sufficient."""
+            if field not in gaps or field not in parsed:
+                return
+            raw_conf = confidence_parsed.get(field, 0.0)
+            if not isinstance(raw_conf, (int, float)) or isinstance(raw_conf, bool):
+                logger.warning(
+                    "Module '%s': non-numeric confidence for %r (%r) — treating as 0.0",
+                    module.module_id,
+                    field,
+                    raw_conf,
+                )
+                field_conf: float = 0.0
             else:
-                warnings.append(f"Low confidence ({desc_conf:.2f}) for description — skipped. Review manually.")
-
-        # Apply documentation if above threshold
-        if "documentation" in gaps and "documentation" in parsed:
-            doc_conf = parsed.get("confidence", {}).get("documentation", 0.0)
-            confidence["documentation"] = doc_conf
-            if doc_conf >= self.threshold:
-                updates["documentation"] = parsed["documentation"]
+                field_conf = float(raw_conf)
+            confidence[field] = field_conf
+            if field_conf >= self.threshold:
+                updates[field] = parsed[field]
             else:
-                warnings.append(f"Low confidence ({doc_conf:.2f}) for documentation — skipped. Review manually.")
+                warnings.append(f"Low confidence ({field_conf:.2f}) for {field} — skipped. Review manually.")
+
+        _apply_simple("description")
+        _apply_simple("documentation")
 
         # Apply annotations if above threshold. Field set is derived from
         # ModuleAnnotations at import time, so adding new fields upstream
         # automatically widens what the SLM may populate (extra excluded).
         if "annotations" in gaps and "annotations" in parsed and isinstance(parsed["annotations"], dict):
             ann_data = parsed["annotations"]
-            ann_conf = parsed.get("confidence", {})
             accepted: dict[str, Any] = {}
             for field_name, validate in _ANNOTATION_FIELD_VALIDATORS.items():
                 if field_name not in ann_data or not validate(ann_data[field_name]):
                     continue
-                field_conf = ann_conf.get(f"annotations.{field_name}", ann_conf.get(field_name, 0.0))
+                raw_ann_conf = confidence_parsed.get(
+                    f"annotations.{field_name}", confidence_parsed.get(field_name, 0.0)
+                )
+                if not isinstance(raw_ann_conf, (int, float)) or isinstance(raw_ann_conf, bool):
+                    logger.warning(
+                        "Module '%s': non-numeric confidence for 'annotations.%s' (%r) — treating as 0.0",
+                        module.module_id,
+                        field_name,
+                        raw_ann_conf,
+                    )
+                    field_conf = 0.0
+                else:
+                    field_conf = float(raw_ann_conf)
                 confidence[f"annotations.{field_name}"] = field_conf
                 if field_conf >= self.threshold:
                     accepted[field_name] = ann_data[field_name]
@@ -257,14 +291,7 @@ class AIEnhancer:
                 base = module.annotations or DEFAULT_ANNOTATIONS
                 updates["annotations"] = replace(base, **accepted)
 
-        # Apply input_schema if above threshold
-        if "input_schema" in gaps and "input_schema" in parsed:
-            schema_conf = parsed.get("confidence", {}).get("input_schema", 0.0)
-            confidence["input_schema"] = schema_conf
-            if schema_conf >= self.threshold:
-                updates["input_schema"] = parsed["input_schema"]
-            else:
-                warnings.append(f"Low confidence ({schema_conf:.2f}) for input_schema — skipped. Review manually.")
+        _apply_simple("input_schema")
 
         if not updates:
             return replace(module, warnings=warnings) if warnings != module.warnings else module
@@ -314,8 +341,23 @@ class AIEnhancer:
         if "input_schema" in gaps:
             parts.append('  "input_schema": <JSON Schema object for function parameters>,')
 
+        # Build confidence keys dynamically from gaps so the SLM is told to
+        # supply confidence for every field it is being asked to fill. The
+        # read side (_enhance_module) looks up annotations.<name> per-field
+        # keys via _ANNOTATION_FIELD_VALIDATORS; keep the prompt and the
+        # read logic symmetric by enumerating the same validator set.
+        confidence_keys: list[str] = []
+        if "description" in gaps:
+            confidence_keys.append("description")
+        if "documentation" in gaps:
+            confidence_keys.append("documentation")
+        if "input_schema" in gaps:
+            confidence_keys.append("input_schema")
+        if "annotations" in gaps:
+            confidence_keys.extend(f"annotations.{name}" for name in _ANNOTATION_FIELD_VALIDATORS)
+
         parts.append('  "confidence": {')
-        parts.append('    "description": 0.0, "documentation": 0.0')
+        parts.append("    " + ", ".join(f'"{k}": 0.0' for k in confidence_keys))
         parts.append("  }")
         parts.append("}")
         parts.append("")
@@ -369,6 +411,9 @@ class AIEnhancer:
             text = "\n".join(lines)
 
         try:
-            return json.loads(text)
+            result = json.loads(text)
         except json.JSONDecodeError as exc:
             raise ValueError(f"SLM returned invalid JSON: {exc}") from exc
+        if not isinstance(result, dict):
+            raise ValueError(f"SLM returned non-dict JSON ({type(result).__name__}); expected a JSON object")
+        return result