Gen3SchemaDev 2.6.0__tar.gz → 2.6.2__tar.gz

{gen3schemadev-2.6.0 → gen3schemadev-2.6.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: Gen3SchemaDev
-Version: 2.6.0
+Version: 2.6.2
 Summary: Tool for data modelling in Gen3
 License: Apache 2.0
 License-File: LICENSE
@@ -35,18 +35,9 @@ This repository aims to provide the documentation, learning materials, and softw
 - [Guide to creating your first dictionary](docs/gen3schemadev/first_dictionary.md)
 - [Troubleshooting](docs/gen3schemadev/troubleshooting.md)
 
-### Fixing `$ref` properties in an existing dictionary
+### Null description placeholders
 
-JSON Schema draft-04 (which Gen3 uses) ignores any keyword sitting as a direct sibling of `$ref`, so properties written as `{description: ..., $ref: _definitions.yaml#/X}` lose their description in the data-dictionary viewer. The `fix-refs` command rewrites such properties in place, moving the `$ref` into an `allOf` list so annotations survive resolution:
-
-```bash
-gen3schemadev fix-refs -y path/to/dictionary --dry-run   # preview changes
-gen3schemadev fix-refs -y path/to/dictionary             # rewrite in place
-```
-
-The command is idempotent and never touches bare refs, refs already inside `allOf`/`anyOf`/`oneOf`, the `properties: {$ref: ...}` merge construct, or `_definitions.yaml`/`_terms.yaml`/`_settings.yaml`.
-
-Both `fix-refs` and `validate` also warn about `description: null` placeholders anywhere in the dictionary (commonly in `_definitions.yaml`): the Gen3 metaschema requires descriptions to be strings, and such placeholders fail metaschema validation once exposed through bare or `allOf`-wrapped refs. Remove the null keys to resolve the warning.
+`gen3schemadev validate` warns about `description: null` placeholders anywhere in the dictionary (commonly in `_definitions.yaml`'s enum definitions). The Gen3 metaschema requires descriptions to be strings; null placeholders cause "No Description" in the data-dictionary viewer and metaschema validation failures that surface on resolved node schemas, far from the offending definition. Remove the null `description` keys to resolve the warning.
 
 
 ## Deep dive into Gen3 Data Modelling

{gen3schemadev-2.6.0 → gen3schemadev-2.6.2}/README.md

@@ -12,18 +12,9 @@ This repository aims to provide the documentation, learning materials, and softw
 - [Guide to creating your first dictionary](docs/gen3schemadev/first_dictionary.md)
 - [Troubleshooting](docs/gen3schemadev/troubleshooting.md)
 
-### Fixing `$ref` properties in an existing dictionary
+### Null description placeholders
 
-JSON Schema draft-04 (which Gen3 uses) ignores any keyword sitting as a direct sibling of `$ref`, so properties written as `{description: ..., $ref: _definitions.yaml#/X}` lose their description in the data-dictionary viewer. The `fix-refs` command rewrites such properties in place, moving the `$ref` into an `allOf` list so annotations survive resolution:
-
-```bash
-gen3schemadev fix-refs -y path/to/dictionary --dry-run   # preview changes
-gen3schemadev fix-refs -y path/to/dictionary             # rewrite in place
-```
-
-The command is idempotent and never touches bare refs, refs already inside `allOf`/`anyOf`/`oneOf`, the `properties: {$ref: ...}` merge construct, or `_definitions.yaml`/`_terms.yaml`/`_settings.yaml`.
-
-Both `fix-refs` and `validate` also warn about `description: null` placeholders anywhere in the dictionary (commonly in `_definitions.yaml`): the Gen3 metaschema requires descriptions to be strings, and such placeholders fail metaschema validation once exposed through bare or `allOf`-wrapped refs. Remove the null keys to resolve the warning.
+`gen3schemadev validate` warns about `description: null` placeholders anywhere in the dictionary (commonly in `_definitions.yaml`'s enum definitions). The Gen3 metaschema requires descriptions to be strings; null placeholders cause "No Description" in the data-dictionary viewer and metaschema validation failures that surface on resolved node schemas, far from the offending definition. Remove the null `description` keys to resolve the warning.
 
 
 ## Deep dive into Gen3 Data Modelling

{gen3schemadev-2.6.0 → gen3schemadev-2.6.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "Gen3SchemaDev"
-version = "2.6.0"
+version = "2.6.2"
 description = "Tool for data modelling in Gen3"
 authors = [
     {name = "JoshuaHarris391",email = "harjo391@gmail.com"}

{gen3schemadev-2.6.0 → gen3schemadev-2.6.2}/src/gen3schemadev/cli.py

@@ -20,13 +20,7 @@ from gen3schemadev.validators.metaschema_validator import validate_schema_with_m
 from importlib.metadata import version
 from gen3schemadev.ddvis import visualise_with_docker
 from gen3schemadev.validators.rule_validator import RuleValidator
-from gen3schemadev.refs import (
-    WRAPPED,
-    SKIP_REASONS,
-    fix_yaml_dir,
-    find_null_descriptions,
-    scan_dir_null_descriptions,
-)
+from gen3schemadev.refs import find_null_descriptions
 
 
 def print_null_description_warning(hits):
@@ -168,27 +162,6 @@ def main():
         help="Set logging level to DEBUG"
     )
 
-    # Create 'fix-refs' subcommand
-    fixrefs_parser = subparsers.add_parser(
-        "fix-refs",
-        help="Rewrite properties with '$ref' siblings into draft-04-safe allOf form (in place)"
-    )
-    fixrefs_parser.add_argument(
-        "-y", "--yamldir",
-        required=True,
-        help="Directory of Gen3 YAML files to rewrite in place (recursive)"
-    )
-    fixrefs_parser.add_argument(
-        "--dry-run",
-        action="store_true",
-        help="Report what would change without writing files"
-    )
-    fixrefs_parser.add_argument(
-        "--debug",
-        action="store_true",
-        help="Set logging level to DEBUG"
-    )
-
     args = parser.parse_args()
     
     # Handle case where no command is provided
@@ -361,45 +334,5 @@ def main():
         output_path = args.output or "input_example.yaml"
         write_yaml(init_yaml, output_path)
 
-    elif args.command == "fix-refs":
-        if not os.path.isdir(args.yamldir):
-            print(f"ERROR: Directory not found: {args.yamldir}", file=sys.stderr)
-            sys.exit(1)
-
-        if args.dry_run:
-            print(f"Dry run: scanning {args.yamldir} (no files will be written)")
-        else:
-            print(f"Fixing $ref sibling annotations in: {args.yamldir}")
-
-        reports = fix_yaml_dir(args.yamldir, dry_run=args.dry_run)
-
-        total_wrapped = 0
-        files_changed = 0
-        for report in reports:
-            print(report["path"])
-            if report["skipped_file"]:
-                print(f"  skipped file ({report['skipped_file']})")
-                continue
-            if not report["changes"]:
-                print("  no $ref properties found")
-                continue
-            for prop_name, action, preserved in report["changes"]:
-                if action == WRAPPED:
-                    total_wrapped += 1
-                    print(f"  wrapped: {prop_name} (preserved: {', '.join(preserved)})")
-                else:
-                    print(f"  skipped ({SKIP_REASONS[action]}): {prop_name}")
-            if report["rewritten"] or (args.dry_run and any(c[1] == WRAPPED for c in report["changes"])):
-                files_changed += 1
-
-        files_unchanged = len(reports) - files_changed
-        verb = "would be wrapped" if args.dry_run else "wrapped"
-        print(f"\nSummary: {total_wrapped} properties {verb} in {files_changed} files; "
-              f"{files_unchanged} files unchanged.")
-
-        # Wrapping is what exposes null 'description' placeholders in shared
-        # definitions to metaschema validation, so surface them here too.
-        print_null_description_warning(scan_dir_null_descriptions(args.yamldir))
-
 if __name__ == "__main__":
     main()

{gen3schemadev-2.6.0 → gen3schemadev-2.6.2}/src/gen3schemadev/converter.py

@@ -17,8 +17,6 @@ from dataclasses import dataclass, asdict
 from typing import Protocol, runtime_checkable, Dict, Any, List
 import logging
 
-from gen3schemadev.refs import wrap_ref_siblings
-
 
 logger = logging.getLogger(__name__)
 
@@ -490,11 +488,11 @@ def format_datetime(prop_dict: dict) -> dict:
     Formats a property dictionary with a 'type' of 'datetime' to use a $ref
     to the Gen3 _definitions.yaml#/datetime definition.
 
-    Annotations such as 'description' are preserved by wrapping the $ref in
-    an allOf list: under JSON Schema draft-04, keywords sitting as direct
-    siblings of $ref are ignored during resolution, so they must live beside
-    allOf instead. A datetime property with no annotations becomes a bare
-    $ref. If the property is not of type 'datetime', it is returned unchanged.
+    Annotations such as 'description' are preserved as siblings of the $ref:
+    Gen3's resolver merges sibling keys over the referenced definition, so
+    the property's own description survives resolution. A datetime property
+    with no annotations becomes a bare $ref. If the property is not of type
+    'datetime', it is returned unchanged.
 
     Example input:
         {
@@ -508,9 +506,7 @@ def format_datetime(prop_dict: dict) -> dict:
         {
             "collection_date": {
                 "description": "Date and time of collection (datetime)",
-                "allOf": [
-                    {"$ref": "_definitions.yaml#/datetime"}
-                ]
+                "$ref": "_definitions.yaml#/datetime"
             }
         }
 
@@ -532,9 +528,8 @@ def format_datetime(prop_dict: dict) -> dict:
         value = prop_dict[first_key]
 
         if isinstance(value, dict) and value.get('type') == 'datetime':
-            ref_prop = {k: v for k, v in value.items() if k != 'type'}
-            ref_prop['$ref'] = "_definitions.yaml#/datetime"
-            formatted_props, _ = wrap_ref_siblings(ref_prop, first_key)
+            formatted_props = {k: v for k, v in value.items() if k != 'type'}
+            formatted_props['$ref'] = "_definitions.yaml#/datetime"
         else:
             formatted_props = value
 

gen3schemadev-2.6.2/src/gen3schemadev/refs.py

@@ -0,0 +1,74 @@
+"""Diagnostics for ``$ref`` usage in Gen3 data dictionaries.
+
+Gen3's resolver merges a property's sibling keys over the referenced
+definition, so writing annotations next to a ``$ref`` is the normal,
+working form::
+
+    atrial_fibrillation:
+      description: "Self-reported atrial fibrillation."
+      $ref: "_definitions.yaml#/enum_yes_no"
+
+The real hazard is a ``description: null`` placeholder inside a shared
+definition (commonly the enum definitions in ``_definitions.yaml``): the
+Gen3 metaschema requires ``description`` to be a string, and the null shows
+up as "No Description" in the data-dictionary viewer and fails metaschema
+validation on resolved node schemas — far from the definition that carries
+it. ``find_null_descriptions`` reports every such placeholder by path so
+the root cause is named up front.
+"""
+
+from __future__ import annotations
+
+import logging
+
+logger = logging.getLogger(__name__)
+
+_COMBINATORS = ("allOf", "anyOf", "oneOf")
+
+
+def has_ref(prop) -> bool:
+    """
+    Return True if a property definition references another schema, either
+    via a top-level ``$ref`` or via a ``$ref`` inside an allOf/anyOf/oneOf
+    list item.
+    """
+    if not isinstance(prop, dict):
+        return False
+    if "$ref" in prop:
+        return True
+    for combinator in _COMBINATORS:
+        items = prop.get(combinator)
+        if isinstance(items, list):
+            if any(isinstance(item, dict) and "$ref" in item for item in items):
+                return True
+    return False
+
+
+def find_null_descriptions(node, path: str = "") -> list:
+    """
+    Recursively find every ``description`` key whose value is null.
+
+    The Gen3 metaschema requires ``description`` to be a string, so a
+    ``description: null`` placeholder (common in generated shared
+    definitions) is invalid. Because the property's own description merges
+    over the definition's during resolution, the null often stays hidden
+    until it surfaces on some resolved node schema — where the resulting
+    metaschema failure points far away from the definition that caused it.
+
+    Returns a list of dotted paths to each offender, with list items shown
+    as ``[i]`` (e.g. ``"enum_yes_no.description"``,
+    ``"properties.status.anyOf[0].description"``). Non-dict/list input
+    yields an empty list.
+    """
+    hits = []
+    if isinstance(node, dict):
+        for key, value in node.items():
+            key_path = f"{path}.{key}" if path else str(key)
+            if key == "description" and value is None:
+                hits.append(key_path)
+            else:
+                hits.extend(find_null_descriptions(value, key_path))
+    elif isinstance(node, list):
+        for i, item in enumerate(node):
+            hits.extend(find_null_descriptions(item, f"{path}[{i}]"))
+    return hits

{gen3schemadev-2.6.0 → gen3schemadev-2.6.2}/src/gen3schemadev/schema/schema_templates/program.yaml

@@ -24,9 +24,8 @@ properties:
   type:
     type: string
   id:
+    $ref: _definitions.yaml#/UUID
     systemAlias: node_id
-    allOf:
-    - $ref: _definitions.yaml#/UUID
   name:
     type: string
     description: Full name/title of the program.

{gen3schemadev-2.6.0 → gen3schemadev-2.6.2}/src/gen3schemadev/schema/schema_templates/project.yaml

@@ -35,10 +35,9 @@ properties:
   type:
     type: string
   id:
+    $ref: _definitions.yaml#/UUID
     systemAlias: node_id
     description: UUID for the project.
-    allOf:
-    - $ref: _definitions.yaml#/UUID
   name:
     type: string
     description: Display name/brief description for the project.
@@ -63,11 +62,10 @@ properties:
     description: The ID of the source providing support/grant resources.
     type: string
   programs:
+    $ref: _definitions.yaml#/to_one
     description: 'Indicates that the Study is logically part of the indicated Study.
 
       '
-    allOf:
-    - $ref: _definitions.yaml#/to_one
   state:
     description: 'The possible states a project can be in.  All but `open` are
 

gen3schemadev-2.6.0/src/gen3schemadev/refs.py

@@ -1,262 +0,0 @@
-"""Utilities for making ``$ref`` properties safe under JSON Schema draft-04.
-
-Gen3 compiles data dictionaries with draft-04 semantics, where any keyword
-sitting as a direct sibling of ``$ref`` is ignored during resolution. A
-property written as::
-
-    atrial_fibrillation:
-      description: "Self-reported atrial fibrillation."
-      $ref: "_definitions.yaml#/enum_yes_no"
-
-loses its ``description`` (and ``termDef``, ``term``, etc.) when resolved by
-strict tooling such as dictionaryutils, so the data-dictionary viewer shows
-"No Description". The fix is structural: keep the annotations at the top
-level and move the ``$ref`` into an ``allOf`` list::
-
-    atrial_fibrillation:
-      description: "Self-reported atrial fibrillation."
-      allOf:
-      - $ref: "_definitions.yaml#/enum_yes_no"
-
-This module is the single source of truth for that transformation. It is
-used by the schema generator (``converter.format_datetime``), the rule
-validator, and the ``gen3schemadev fix-refs`` CLI command.
-"""
-
-from __future__ import annotations
-
-import fnmatch
-import logging
-import os
-
-logger = logging.getLogger(__name__)
-
-# Actions recorded for each property examined by the transform.
-WRAPPED = "wrapped"
-BARE_REF = "bare-ref"
-ALREADY_WRAPPED = "already-wrapped"
-NO_REF = "no-ref"
-PROPERTIES_MERGE = "properties-merge"
-
-# Human-readable reasons for skipped properties, keyed by action.
-SKIP_REASONS = {
-    BARE_REF: "bare $ref, no siblings to preserve",
-    ALREADY_WRAPPED: "already wrapped in allOf/anyOf/oneOf",
-    PROPERTIES_MERGE: "$ref key in 'properties' is a Gen3 properties-merge",
-}
-
-_COMBINATORS = ("allOf", "anyOf", "oneOf")
-
-
-def has_ref(prop) -> bool:
-    """
-    Return True if a property definition references another schema, either
-    via a top-level ``$ref`` or via a ``$ref`` inside an allOf/anyOf/oneOf
-    list item.
-    """
-    if not isinstance(prop, dict):
-        return False
-    if "$ref" in prop:
-        return True
-    for combinator in _COMBINATORS:
-        items = prop.get(combinator)
-        if isinstance(items, list):
-            if any(isinstance(item, dict) and "$ref" in item for item in items):
-                return True
-    return False
-
-
-def wrap_ref_siblings(prop, prop_name: str = "") -> "tuple":
-    """
-    Rewrite a single property definition so sibling annotations survive
-    draft-04 ``$ref`` resolution.
-
-    Returns a ``(new_prop, action)`` tuple. The transformation only fires
-    when the property dict has a top-level ``$ref`` alongside at least one
-    other key; in that case the siblings keep their original order and the
-    ``$ref`` moves into an ``allOf`` list appended last. All other shapes
-    are returned unchanged:
-
-    - not a dict, or no ``$ref`` anywhere -> ``NO_REF``
-    - ``$ref`` only inside allOf/anyOf/oneOf -> ``ALREADY_WRAPPED`` (idempotent)
-    - ``{"$ref": ...}`` with no siblings -> ``BARE_REF`` (nothing is lost)
-    """
-    if not isinstance(prop, dict) or not has_ref(prop):
-        return prop, NO_REF
-
-    if "$ref" not in prop:
-        return prop, ALREADY_WRAPPED
-
-    if len(prop) == 1:
-        return prop, BARE_REF
-
-    new_prop = {k: v for k, v in prop.items() if k != "$ref"}
-    if isinstance(new_prop.get("allOf"), list):
-        # Rare edge: the property already has an allOf next to the $ref.
-        # Append rather than overwrite so existing constraints are kept.
-        new_prop["allOf"] = new_prop["allOf"] + [{"$ref": prop["$ref"]}]
-    else:
-        new_prop["allOf"] = [{"$ref": prop["$ref"]}]
-    logger.debug(f"Wrapped $ref of property '{prop_name}' in allOf.")
-    return new_prop, WRAPPED
-
-
-def fix_schema(schema: dict) -> "tuple":
-    """
-    Apply ``wrap_ref_siblings`` to every property of a node schema.
-
-    Only the direct values of ``schema["properties"]`` are examined:
-    sibling-``$ref`` annotations occur at the property level in Gen3 node
-    schemas, and a shallow walk cannot accidentally rewrite the Gen3
-    properties-merge construct (a ``$ref`` *key* directly inside the
-    ``properties`` block), which is recorded as ``PROPERTIES_MERGE`` and
-    left untouched.
-
-    Returns ``(new_schema, changes)`` where changes is a list of
-    ``(prop_name, action, preserved_keys)`` tuples. ``preserved_keys`` names
-    the sibling annotation keys kept for ``WRAPPED`` entries and is an empty
-    tuple otherwise.
-    """
-    changes = []
-    props = schema.get("properties") if isinstance(schema, dict) else None
-    if not isinstance(props, dict):
-        return schema, changes
-
-    new_props = {}
-    for prop_name, prop in props.items():
-        if prop_name == "$ref":
-            new_props[prop_name] = prop
-            changes.append((prop_name, PROPERTIES_MERGE, ()))
-            continue
-        new_prop, action = wrap_ref_siblings(prop, prop_name)
-        new_props[prop_name] = new_prop
-        preserved = ()
-        if action == WRAPPED:
-            preserved = tuple(k for k in prop.keys() if k != "$ref")
-        if action != NO_REF:
-            changes.append((prop_name, action, preserved))
-
-    new_schema = dict(schema)
-    new_schema["properties"] = new_props
-    return new_schema, changes
-
-
-def find_null_descriptions(node, path: str = "") -> list:
-    """
-    Recursively find every ``description`` key whose value is null.
-
-    The Gen3 metaschema requires ``description`` to be a string, so a
-    ``description: null`` placeholder (common in generated shared
-    definitions) is invalid. With a direct ``$ref`` the resolver happens to
-    mask it — the referencing property's own description merges over the
-    definition's null — but a bare or allOf-wrapped ref exposes the null in
-    the resolved node schema, where metaschema validation fails far away
-    from the definition that caused it.
-
-    Returns a list of dotted paths to each offender, with list items shown
-    as ``[i]`` (e.g. ``"enum_yes_no.description"``,
-    ``"properties.status.anyOf[0].description"``). Non-dict/list input
-    yields an empty list.
-    """
-    hits = []
-    if isinstance(node, dict):
-        for key, value in node.items():
-            key_path = f"{path}.{key}" if path else str(key)
-            if key == "description" and value is None:
-                hits.append(key_path)
-            else:
-                hits.extend(find_null_descriptions(value, key_path))
-    elif isinstance(node, list):
-        for i, item in enumerate(node):
-            hits.extend(find_null_descriptions(item, f"{path}[{i}]"))
-    return hits
-
-
-def scan_dir_null_descriptions(yaml_dir: str) -> list:
-    """
-    Scan every YAML file under ``yaml_dir`` (recursively) for null-valued
-    ``description`` keys.
-
-    Unlike ``fix_yaml_dir``, underscore files (``_definitions.yaml``,
-    ``_terms.yaml``, ...) are INCLUDED — shared definitions are where the
-    null placeholders usually live, and where they do the most damage
-    because every referencing property inherits them on resolution.
-
-    Returns ``"relpath: dotted.path"`` strings, one per offender.
-    """
-    from gen3schemadev.utils import load_yaml
-
-    hits = []
-    for root, _dirs, files in os.walk(yaml_dir):
-        for fname in sorted(files):
-            if not (fnmatch.fnmatch(fname, "*.yaml") or fnmatch.fnmatch(fname, "*.yml")):
-                continue
-            path = os.path.join(root, fname)
-            rel_path = os.path.relpath(path, yaml_dir)
-            schema = load_yaml(path)
-            for hit in find_null_descriptions(schema):
-                hits.append(f"{rel_path}: {hit}")
-    return hits
-
-
-def fix_yaml_dir(yaml_dir: str, dry_run: bool = False) -> list:
-    """
-    Rewrite every YAML node schema under ``yaml_dir`` (recursively) so that
-    ``$ref`` sibling annotations survive draft-04 resolution.
-
-    Files whose basename starts with an underscore (``_definitions.yaml``,
-    ``_terms.yaml``, ``_settings.yaml``) are skipped entirely: adding
-    structure or descriptions to shared definitions would force them onto
-    every referencing property. Files with no changes are never rewritten,
-    which keeps the operation idempotent. With ``dry_run=True`` nothing is
-    written at all.
-
-    Returns a list of per-file report dicts::
-
-        {"path": <relative path>, "skipped_file": <reason or None>,
-         "changes": [(prop_name, action, preserved_keys), ...],
-         "rewritten": <bool>}
-    """
-    from gen3schemadev.utils import load_yaml, write_yaml
-
-    reports = []
-    for root, _dirs, files in os.walk(yaml_dir):
-        for fname in sorted(files):
-            if not (fnmatch.fnmatch(fname, "*.yaml") or fnmatch.fnmatch(fname, "*.yml")):
-                continue
-            path = os.path.join(root, fname)
-            rel_path = os.path.relpath(path, yaml_dir)
-
-            if fname.startswith("_"):
-                reports.append({
-                    "path": rel_path,
-                    "skipped_file": "definitions/terms/settings file",
-                    "changes": [],
-                    "rewritten": False,
-                })
-                continue
-
-            schema = load_yaml(path)
-            if not isinstance(schema, dict):
-                reports.append({
-                    "path": rel_path,
-                    "skipped_file": "not a mapping, no properties to fix",
-                    "changes": [],
-                    "rewritten": False,
-                })
-                continue
-
-            new_schema, changes = fix_schema(schema)
-            wrapped = [c for c in changes if c[1] == WRAPPED]
-            rewritten = False
-            if wrapped and not dry_run:
-                write_yaml(new_schema, path)
-                rewritten = True
-
-            reports.append({
-                "path": rel_path,
-                "skipped_file": None,
-                "changes": changes,
-                "rewritten": rewritten,
-            })
-    return reports