yaml-reference 2.6.0__tar.gz → 2.6.2__tar.gz

yaml_reference-2.6.2/.github/copilot-instructions.md

@@ -0,0 +1,141 @@
+# Copilot Instructions for yaml-reference
+
+## Project Overview
+
+**yaml-reference** is a Python library that extends `ruamel.yaml` with cross-file YAML composition using custom tags (`!reference`, `!reference-all`, `!flatten`, `!merge`). It's built to be a reference implementation of the [yaml-reference-specs](https://github.com/dsillman2000/yaml-reference-specs) specification.
+
+## Build, Test, and Lint
+
+**Tool chain:** `uv` (Python package manager) + `pytest` (testing) + `ruff` (linting/formatting)
+
+### Install dependencies
+```bash
+uv sync
+```
+
+### Run all tests
+```bash
+uv run pytest tests/ -v
+```
+
+### Run a single test file
+```bash
+uv run pytest tests/unit/test_reference.py -v
+```
+
+### Run tests matching a pattern
+```bash
+uv run pytest tests/unit/test_reference.py::test_reference_load -v
+```
+
+### Run spec compliance tests (tests against yaml-reference-specs)
+```bash
+make spec-test
+```
+
+### Code formatting
+```bash
+uv run ruff format
+# or
+make format
+```
+
+### Linting and auto-fix
+```bash
+uv run ruff check --fix
+# or
+make lint
+```
+
+### Run full quality check (format + lint + test)
+```bash
+make check
+```
+
+### Build package
+```bash
+uv build
+```
+
+## Architecture
+
+The library is structured in two key parts:
+
+### Core Module (`yaml_reference/__init__.py`)
+- **Reference & ReferenceAll classes**: Represent the `!reference` and `!reference-all` YAML tags as Python objects
+- **parse_yaml_with_references()**: Parses YAML, returning Reference/ReferenceAll objects without resolving them (one layer only)
+- **load_yaml_with_references()**: Fully recursively resolves all references, returning a complete Python dict
+- **Flatten & Merge classes**: Represent `!flatten` and `!merge` tag logic
+- **YAML loader setup**: Registers custom constructors with `ruamel.yaml.YAML` for each tag
+
+### CLI Module (`yaml_reference/cli.py`)
+- Simple entry point that calls the core loading functions
+- Outputs JSON to stdout (compatible with spec tests)
+- Takes optional `--allow` flag for path restrictions
+
+### Test Structure (`tests/unit/`)
+- `test_reference.py`: Tests for `!reference` and `!reference-all` tag resolution
+- `test_flatten.py`: Tests for `!flatten` tag behavior
+- `test_merge.py`: Tests for `!merge` tag behavior
+- `conftest.py`: Pytest fixtures and test utilities
+
+## Key Conventions & Design Patterns
+
+### Security-First Path Handling
+1. **Relative paths only**: All references must use relative paths (e.g., `path: "config/db.yaml"`). Absolute paths raise `ValueError`.
+2. **Path restriction by default**: References can only access files in the same directory or subdirectories (no `..` to escape). Use `allow_paths` parameter to explicitly allow other directory trees.
+3. **Security invariant**: Disallowed files are **never opened or read into memory**. Path filtering happens before file I/O.
+4. **Silent omission (for `!reference-all`)**: When a glob pattern matches files outside allowed paths, those files are silently dropped from results and the function returns `rc=0` (not an error).
+
+### YAML Tag Implementation Pattern
+Each custom tag follows this pattern:
+1. Define a class with `yaml_tag` attribute
+2. Implement `@classmethod from_yaml(cls, constructor, node)` to parse from YAML
+3. Register constructor with the YAML loader in `__init__.py`
+4. The class instance persists through `parse_yaml_with_references()`, allowing layer-by-layer resolution
+
+### Reference Resolution Order
+1. **Circular reference detection** occurs during recursive resolution by tracking a "resolution stack"
+2. **Anchors** (optional parameter): If specified, extract only the anchored section from the referenced file
+3. **Recursive expansion**: `load_yaml_with_references()` recursively expands all tags, applying `!flatten` and `!merge` logic as it encounters them
+
+### Error Handling
+- **ValueError** for spec violations: absolute paths, circular references, invalid anchors
+- **FileNotFoundError** for missing referenced files
+- **Glob errors**: Return empty list `[]` if glob matches no files (silent omission)
+
+### Spec Compliance Testing
+The project tests against `yaml-reference-specs`, a Go-based reference implementation. The spec tests verify:
+- Correct expansion of all four tags
+- Proper error detection (bad paths, missing files, circular refs)
+- Path restriction enforcement
+- Edge cases like empty globs and nested composition
+
+Run with: `make spec-test` or `scripts/spec-test.sh`
+
+## Pre-commit Hooks
+
+The repository enforces conventional commits and code quality via pre-commit:
+- **ruff-check** and **ruff-format**: Ensures consistent style
+- **conventional-pre-commit**: Enforces Conventional Commits format (e.g., `feat:`, `fix:`, `docs:`)
+- Standard hooks: trailing whitespace, EOF fixers, YAML validation
+
+Install hooks with: `pre-commit install`
+
+## Common Workflows
+
+### Adding a new tag type
+1. Create a class in `yaml_reference/__init__.py` with `yaml_tag` attribute and `from_yaml()` classmethod
+2. Register the constructor after the class definition
+3. Add resolution logic (handle in recursive expansion)
+4. Write tests in `tests/unit/test_*.py` following existing patterns
+5. Update README.md with usage example
+
+### Debugging a reference resolution issue
+1. Use `parse_yaml_with_references()` to see raw Reference objects before resolution
+2. Add print statements or use a debugger to trace the `_resolve_references()` recursive calls
+3. Check the resolution stack to verify circular reference detection is working
+4. Run a specific test with `-v` flag to see detailed assertion output
+
+### Updating error messages
+Ensure error messages follow this pattern: include the problematic value, the path of the file where the error occurred, and the specific constraint violated. This helps spec tests verify proper error handling.

{yaml_reference-2.6.0 → yaml_reference-2.6.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: yaml-reference
-Version: 2.6.0
+Version: 2.6.2
 Summary: Extension package built on top of  `ruamel.yaml` to support cross-file references in YAML files using tags `!reference` and `!reference-all`.
 Project-URL: Repository, https://github.com/dsillman2000/yaml-reference.git
 Author-email: David Sillman <dsillman2000@gmail.com>
@@ -40,7 +40,7 @@ uv add yaml-reference
 ```
 
 ## Spec
-![Spec Status](https://img.shields.io/badge/spec%20v0.2.6--3-passing-brightgreen?link=https%3A%2F%2Fgithub.com%2Fdsillman2000%2Fyaml-reference-specs%2Ftree%2Fv0.2.6-3)
+![Spec Status](https://img.shields.io/badge/spec%20v0.2.6--4-passing-brightgreen?link=https%3A%2F%2Fgithub.com%2Fdsillman2000%2Fyaml-reference-specs%2Ftree%2Fv0.2.6-4)
 
 This Python library implements the YAML specification for cross-file references and YAML composition in YAML files using tags `!reference`, `!reference-all`, `!flatten`, and `!merge` as defined in the [yaml-reference-specs project](https://github.com/dsillman2000/yaml-reference-specs).
 
@@ -286,6 +286,21 @@ yaml-reference compile input.yml --allow /allowed/path1 --allow /allowed/path2
 
 Whether or not `allow_paths` is specified, the default behavior is to allow references to files in the same directory as the source YAML file (or subdirectories). "Back-navigating" out of a the root directory is not allowed (".." local references in a root YAML file). This provides a secure baseline to prevent unsafe access which is not explicitly allowed.
 
+### Glob matching behavior for `!reference-all`
+
+`!reference-all` applies **silent-omission semantics** when individual glob matches fall outside the allowed path set. Disallowed paths are filtered out *before* any file is opened (security invariant: disallowed file contents are never loaded into memory). The result is the subset of glob matches that are both reachable and allowed:
+
+| Scenario | Behaviour | Exit |
+|---|---|---|
+| Glob matches zero files | Returns `[]` | `rc=0` |
+| Some matched files are outside `allow_paths` | Disallowed files are silently dropped; remaining files are returned | `rc=0` |
+| All matched files are outside `allow_paths` | Returns `[]` | `rc=0` |
+| Glob pattern is absolute (starts with `/`) | Hard error – `ValueError` raised | `rc=1` |
+| A matched file is the calling file (self-reference) | Hard error – circularity `ValueError` raised | `rc=1` |
+| A matched file transitively references the caller | Hard error – circularity `ValueError` raised | `rc=1` |
+
+This design keeps `!reference-all` resilient against partially-populated directory trees while still enforcing absolute-path and circularity invariants as hard failures.
+
 ### Absolute path restrictions
 
 References using absolute paths (e.g., `/tmp/file.yml`) are explicitly rejected with a `ValueError`. All reference paths must be relative to the source file's directory. If you absolutely must reference an absolute path, relative paths to symlinks can be used. Note that their target directories must be explicitly allowed to avoid permission errors (see the above section about "Path restriction and `allow_paths`").

{yaml_reference-2.6.0 → yaml_reference-2.6.2}/README.md

@@ -14,7 +14,7 @@ uv add yaml-reference
 ```
 
 ## Spec
-![Spec Status](https://img.shields.io/badge/spec%20v0.2.6--3-passing-brightgreen?link=https%3A%2F%2Fgithub.com%2Fdsillman2000%2Fyaml-reference-specs%2Ftree%2Fv0.2.6-3)
+![Spec Status](https://img.shields.io/badge/spec%20v0.2.6--4-passing-brightgreen?link=https%3A%2F%2Fgithub.com%2Fdsillman2000%2Fyaml-reference-specs%2Ftree%2Fv0.2.6-4)
 
 This Python library implements the YAML specification for cross-file references and YAML composition in YAML files using tags `!reference`, `!reference-all`, `!flatten`, and `!merge` as defined in the [yaml-reference-specs project](https://github.com/dsillman2000/yaml-reference-specs).
 
@@ -260,6 +260,21 @@ yaml-reference compile input.yml --allow /allowed/path1 --allow /allowed/path2
 
 Whether or not `allow_paths` is specified, the default behavior is to allow references to files in the same directory as the source YAML file (or subdirectories). "Back-navigating" out of a the root directory is not allowed (".." local references in a root YAML file). This provides a secure baseline to prevent unsafe access which is not explicitly allowed.
 
+### Glob matching behavior for `!reference-all`
+
+`!reference-all` applies **silent-omission semantics** when individual glob matches fall outside the allowed path set. Disallowed paths are filtered out *before* any file is opened (security invariant: disallowed file contents are never loaded into memory). The result is the subset of glob matches that are both reachable and allowed:
+
+| Scenario | Behaviour | Exit |
+|---|---|---|
+| Glob matches zero files | Returns `[]` | `rc=0` |
+| Some matched files are outside `allow_paths` | Disallowed files are silently dropped; remaining files are returned | `rc=0` |
+| All matched files are outside `allow_paths` | Returns `[]` | `rc=0` |
+| Glob pattern is absolute (starts with `/`) | Hard error – `ValueError` raised | `rc=1` |
+| A matched file is the calling file (self-reference) | Hard error – circularity `ValueError` raised | `rc=1` |
+| A matched file transitively references the caller | Hard error – circularity `ValueError` raised | `rc=1` |
+
+This design keeps `!reference-all` resilient against partially-populated directory trees while still enforcing absolute-path and circularity invariants as hard failures.
+
 ### Absolute path restrictions
 
 References using absolute paths (e.g., `/tmp/file.yml`) are explicitly rejected with a `ValueError`. All reference paths must be relative to the source file's directory. If you absolutely must reference an absolute path, relative paths to symlinks can be used. Note that their target directories must be explicitly allowed to avoid permission errors (see the above section about "Path restriction and `allow_paths`").

{yaml_reference-2.6.0 → yaml_reference-2.6.2}/tests/unit/test_reference.py

@@ -155,11 +155,73 @@ def test_allow_paths_load_yaml_with_references(stage_files):
     )
     assert data["all"][0]["outside"] == "outside_value"
 
-    # Test with allow_paths that doesn't include the referenced file (should fail, !reference-all)
-    with pytest.raises(PermissionError):
-        load_yaml_with_references(
-            stg / "inner/with_all.yml", allow_paths=[stg / "some"]
-        )
+    # Test with allow_paths that doesn't include the referenced file (!reference-all):
+    # disallowed files are silently omitted, so the result is an empty list.
+    data = load_yaml_with_references(
+        stg / "inner/with_all.yml", allow_paths=[stg / "some"]
+    )
+    assert data["all"] == []
+
+
+def test_reference_all_empty_glob(stage_files):
+    """Test that !reference-all returns [] without error when glob matches no files."""
+    files = {
+        "test.yml": "data: !reference-all { glob: ./nonexistent/*.yml }",
+    }
+    stg = stage_files(files)
+    data = load_yaml_with_references(stg / "test.yml")
+    assert data["data"] == []
+
+
+def test_reference_all_silently_omits_disallowed_paths(stage_files):
+    """Test that !reference-all silently omits files outside the allowed paths.
+
+    Relative-path violations (glob matches beyond the allowed directory tree) must be
+    dropped *before* reading the file (security invariant) and must NOT trigger an
+    error; the result is simply the subset of paths that are allowed.
+    """
+    files = {
+        # test.yml lives in sub/; globs into ../chapters/ which is outside sub/
+        "sub/test.yml": "data: !reference-all { glob: ../chapters/*.yml }",
+        "chapters/file1.yml": "val: 1",
+        "chapters/file2.yml": "val: 2",
+    }
+    stg = stage_files(files)
+
+    # allow_paths=[stg/"other"] => effective = [stg/"other", stg/"sub"].
+    # Neither covers stg/"chapters", so all matches are silently omitted.
+    data = load_yaml_with_references(stg / "sub/test.yml", allow_paths=[stg / "other"])
+    assert data["data"] == []
+
+    # When we explicitly allow chapters/, both files are included.
+    data = load_yaml_with_references(
+        stg / "sub/test.yml", allow_paths=[stg / "chapters"]
+    )
+    assert len(data["data"]) == 2
+    assert {"val": 1} in data["data"]
+    assert {"val": 2} in data["data"]
+
+
+def test_reference_all_partial_disallow(stage_files):
+    """Test that !reference-all includes only the allowed subset of glob matches.
+
+    When a glob expands to paths spanning multiple directories and only some of those
+    directories are in allow_paths, the disallowed paths are silently omitted while the
+    allowed paths are still included in the result.
+    """
+    files = {
+        "sub/test.yml": "data: !reference-all { glob: ../*/file.yml }",
+        "allowed/file.yml": "kind: allowed",
+        "blocked/file.yml": "kind: blocked",
+    }
+    stg = stage_files(files)
+
+    # Only stg/"allowed" is in allow_paths (plus the default stg/"sub").
+    # stg/"blocked"/file.yml is not covered, so it is silently omitted.
+    data = load_yaml_with_references(
+        stg / "sub/test.yml", allow_paths=[stg / "allowed"]
+    )
+    assert data["data"] == [{"kind": "allowed"}]
 
 
 @pytest.mark.parametrize(

{yaml_reference-2.6.0 → yaml_reference-2.6.2}/yaml_reference/__init__.py

@@ -400,6 +400,32 @@ def _recursively_attribute_location_to_references(data: Any, base_path: Path):
     return data
 
 
+def _is_path_allowed(path: Path, allow_paths: Sequence[Path]) -> bool:
+    """Check whether a resolved path is accessible given the allow_paths configuration.
+
+    Unlike `_check_file_path`, this never raises; it returns `False` for paths that
+    do not exist, are not regular files, or fall outside every entry in *allow_paths*.
+    An empty *allow_paths* sequence means "no directory restrictions" (all existing files
+    are considered allowed).
+
+    Args:
+        path: Resolved, absolute path to check.
+        allow_paths: List of allowed directory paths.
+
+    Returns:
+        True if the path is an accessible file within an allowed directory (or no
+        restrictions are in place). False otherwise.
+    """
+    if not path.exists() or not path.is_file():
+        return False
+    if not allow_paths:
+        return True
+    for allow_path in allow_paths:
+        if path.is_relative_to(allow_path):
+            return True
+    return False
+
+
 def _check_and_track_path(path: Path, visited_paths: set[Path]) -> None:
     """
     Check for circular reference and add path to visited set.
@@ -481,12 +507,26 @@ def _recursively_resolve_references(
     elif isinstance(data, ReferenceAll):
         glob_results = Path(data.location).parent.glob(data.glob)
         abs_paths = [path.resolve() for path in glob_results]
+
+        # Empty glob match -> silent omission, return empty list.
         if not abs_paths:
-            raise FileNotFoundError(
-                f'No files found matching glob pattern "{data.glob}" in directory "{Path(data.location).parent}"'
-            )
-        abs_paths = sorted(abs_paths, key=lambda x: str(x))
+            return []
+
+        # Precompute allowed paths sequence once to avoid repeated list()
+        # construction in the comprehension below.
+        allowed_paths_seq = list(allow_paths)
 
+        # Security invariant: filter out disallowed / nonexistent paths *before*
+        # opening any file.  Relative-path violations are silently omitted here;
+        # absolute-path violations are caught earlier in ReferenceAll.__init__.
+        abs_paths = [p for p in abs_paths if _is_path_allowed(p, allowed_paths_seq)]
+
+        # All matched paths were disallowed → silent omission, return empty list.
+        if not abs_paths:
+            return []
+
+        # Sort only the allowed paths to avoid sorting entries that will be dropped.
+        abs_paths = sorted(abs_paths, key=lambda x: str(x))
         resolved_items = []
         for path in abs_paths:
             # Check for circular reference and track path