zs-yaml 0.3.0__tar.gz

zs_yaml-0.3.0/LICENSE

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2024, NDS e.V. Association
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

zs_yaml-0.3.0/MANIFEST.in

@@ -0,0 +1,26 @@
+# Explicitly include specific files from the root directory
+include README.md
+include requirements.txt
+include LICENSE
+
+# Include all Python files in the package directory and subdirectories
+recursive-include zs_yaml *.py
+
+# Include all documentation resources
+recursive-include docs *
+
+# Exclude cache directories
+prune **/__pycache__
+prune **/.cache
+
+# Exclude version control directories
+prune .git
+
+# Exclude test directories
+prune **/tests
+
+# Exclude development and IDE-specific files
+global-exclude .DS_Store
+global-exclude *.swp
+global-exclude .vscode
+global-exclude .idea

zs_yaml-0.3.0/PKG-INFO

@@ -0,0 +1,174 @@
+Metadata-Version: 2.1
+Name: zs-yaml
+Version: 0.3.0
+Summary: A package that allows using yaml to define zserio objects.
+Home-page: https://github.com/ndsev/zs_yaml
+Author: Navigation Data Standard e.V.
+Author-email: support@nds-association.org
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: PyYAML~=6.0
+Requires-Dist: zserio~=2.12
+
+# `zs-yaml`
+
+*Your Easy Path to zserio Test Data*
+
+<img src="doc/zs-yaml.png" alt="" height="100">
+
+**zs-yaml** adds YAML as a text format to be used with zserio in order to improve the user experience when manually creating or analyzing test data for zserio.
+
+- **YAML Format**: Uses YAML's human-readable format instead of JSON as the primary format for editing, making data handling and transformation more intuitive and less cluttered.
+- **Metadata Inclusion**: Automatically includes metadata in the YAML, eliminating the need for users to manually identify the correct type when importing JSON data, ensuring seamless (de-)serialization.
+- **Custom Transformations**: Allows for hooking in custom transformations so that users can work with familiar formats (e.g., dates or coordinate representations) instead of thinking in unfamiliar formats.
+
+## Example
+
+### Zserio Schema Module Creation
+
+1. **Create the Zserio schema**:
+   - Create a file named `person.zs` with the following content:
+
+     ```zserio
+     package person;
+
+     struct Person
+     {
+         string name;
+         string birthdate;
+         string birth_location;
+         string current_job;
+         int32 income;
+         RoleExperience experience[];
+     };
+
+     struct RoleExperience
+     {
+         string role;
+         int32 years;
+     };
+     ```
+
+2. **Compile the Zserio schema and generate Python APIs**:
+   - Run the following command to compile the schema and generate Python sources:
+
+     ```sh
+     # Generated sources needs type infos so that 'json <-> bin'
+     # conversion works, which is utilized by zs-yaml
+     zserio person.zs -withTypeInfoCode -python zs_gen_api
+     ```
+
+3. **Ensure that the schema modules are available in the Python environment**:
+   - Export the `PYTHONPATH` to include the directory containing the generated Python sources:
+
+     ```sh
+     export PYTHONPATH="zs_gen_api"
+     ```
+
+### Use zs-yaml to create data
+
+Using **zs-yaml**, you can define the same data in a more human-readable YAML format and include necessary metadata along with a custom transformation for the birthdate:
+
+```yaml
+# 1) Metadata is used to specify the type needed for
+#    (de-)serialization and custom transform functions
+# 2) User are free to use their preferred date format
+#    for the birth data as the a normalization function
+#    get invoked.
+# 3) Yaml allows avoiding clutter and adding comments
+#    like this one :)
+
+_meta:
+  schema_module: person.api
+  schema_type: Person
+  transformation_module: "./transformations.py"
+
+name: John Doe
+
+birthdate:
+  _f: normalize_date
+  _a: "01/01/1990"
+birth_location: Springfield, USA
+
+current_job: Software Engineer
+income: 75000
+
+experience:
+  - role: Intern
+    years: 1
+  - role: Junior Developer
+    years: 2
+  - role: Senior Developer
+    years: 3
+```
+
+#### Example Transformation Module (transformations.py)
+
+```python
+def normalize_date(date_str):
+    from datetime import datetime
+    for fmt in ("%Y-%m-%d", "%d/%m/%Y", "%m/%d/%Y"):
+        try:
+            return datetime.strptime(date_str, fmt).strftime("%Y-%m-%d")
+        except ValueError:
+            pass
+    raise ValueError(f"Date {date_str} is not in a recognized format")
+```
+
+#### Creating the Binary Representation
+
+After you have installed `zs-yaml`, call `zs-yaml` to convert your YAML file to a binary representation:
+
+```sh
+# Install zs-yaml if not already installed.
+
+# Create the binary representation from the YAML file
+zs-yaml person.yaml person.bin
+```
+
+## Installation
+
+Currently, the Python package is only available at test.pypi. Once the official release is available,
+you can install the package without specifying an index URL.
+
+```bash
+python -m pip install --upgrade zs-yaml
+```
+
+## Usage
+
+The main entry point for the application is `main.py`. It accepts arguments for specifying the input and output file paths. You can run the application as follows:
+
+```bash
+zs-yaml input.yaml output.bin
+zs-yaml input.bin output.yaml
+```
+
+### Notes
+
+- You have to use the exact same order of field in the yaml as defined by the zserio schema, because zserio expects this
+- When converting from binary to YAML, the target YAML file must already exist and contain the necessary metadata.
+- The minimal metadata content in the target YAML file should be:
+
+```yaml
+_meta:
+  schema_module: <module_name>
+  schema_type: <type_name>
+```
+
+### Future Considerations
+
+In the future, we may consider using YAML tags for a more integrated approach to handling transformations directly within the YAML files, simplifying syntax and enhancing readability.
+
+## Project Structure
+
+- `setup.py`: Script for setting up the project and installing dependencies.
+- `zs_yaml/`: Directory containing the actual `zs-yaml` implementation.
+
+## Reference
+
+This project references the zserio serialization framework. For more information about zserio, visit the [zserio GitHub project](https://github.com/ndsev/zserio).

zs_yaml-0.3.0/README.md

@@ -0,0 +1,158 @@
+# `zs-yaml`
+
+*Your Easy Path to zserio Test Data*
+
+<img src="doc/zs-yaml.png" alt="" height="100">
+
+**zs-yaml** adds YAML as a text format to be used with zserio in order to improve the user experience when manually creating or analyzing test data for zserio.
+
+- **YAML Format**: Uses YAML's human-readable format instead of JSON as the primary format for editing, making data handling and transformation more intuitive and less cluttered.
+- **Metadata Inclusion**: Automatically includes metadata in the YAML, eliminating the need for users to manually identify the correct type when importing JSON data, ensuring seamless (de-)serialization.
+- **Custom Transformations**: Allows for hooking in custom transformations so that users can work with familiar formats (e.g., dates or coordinate representations) instead of thinking in unfamiliar formats.
+
+## Example
+
+### Zserio Schema Module Creation
+
+1. **Create the Zserio schema**:
+   - Create a file named `person.zs` with the following content:
+
+     ```zserio
+     package person;
+
+     struct Person
+     {
+         string name;
+         string birthdate;
+         string birth_location;
+         string current_job;
+         int32 income;
+         RoleExperience experience[];
+     };
+
+     struct RoleExperience
+     {
+         string role;
+         int32 years;
+     };
+     ```
+
+2. **Compile the Zserio schema and generate Python APIs**:
+   - Run the following command to compile the schema and generate Python sources:
+
+     ```sh
+     # Generated sources needs type infos so that 'json <-> bin'
+     # conversion works, which is utilized by zs-yaml
+     zserio person.zs -withTypeInfoCode -python zs_gen_api
+     ```
+
+3. **Ensure that the schema modules are available in the Python environment**:
+   - Export the `PYTHONPATH` to include the directory containing the generated Python sources:
+
+     ```sh
+     export PYTHONPATH="zs_gen_api"
+     ```
+
+### Use zs-yaml to create data
+
+Using **zs-yaml**, you can define the same data in a more human-readable YAML format and include necessary metadata along with a custom transformation for the birthdate:
+
+```yaml
+# 1) Metadata is used to specify the type needed for
+#    (de-)serialization and custom transform functions
+# 2) User are free to use their preferred date format
+#    for the birth data as the a normalization function
+#    get invoked.
+# 3) Yaml allows avoiding clutter and adding comments
+#    like this one :)
+
+_meta:
+  schema_module: person.api
+  schema_type: Person
+  transformation_module: "./transformations.py"
+
+name: John Doe
+
+birthdate:
+  _f: normalize_date
+  _a: "01/01/1990"
+birth_location: Springfield, USA
+
+current_job: Software Engineer
+income: 75000
+
+experience:
+  - role: Intern
+    years: 1
+  - role: Junior Developer
+    years: 2
+  - role: Senior Developer
+    years: 3
+```
+
+#### Example Transformation Module (transformations.py)
+
+```python
+def normalize_date(date_str):
+    from datetime import datetime
+    for fmt in ("%Y-%m-%d", "%d/%m/%Y", "%m/%d/%Y"):
+        try:
+            return datetime.strptime(date_str, fmt).strftime("%Y-%m-%d")
+        except ValueError:
+            pass
+    raise ValueError(f"Date {date_str} is not in a recognized format")
+```
+
+#### Creating the Binary Representation
+
+After you have installed `zs-yaml`, call `zs-yaml` to convert your YAML file to a binary representation:
+
+```sh
+# Install zs-yaml if not already installed.
+
+# Create the binary representation from the YAML file
+zs-yaml person.yaml person.bin
+```
+
+## Installation
+
+Currently, the Python package is only available at test.pypi. Once the official release is available,
+you can install the package without specifying an index URL.
+
+```bash
+python -m pip install --upgrade zs-yaml
+```
+
+## Usage
+
+The main entry point for the application is `main.py`. It accepts arguments for specifying the input and output file paths. You can run the application as follows:
+
+```bash
+zs-yaml input.yaml output.bin
+zs-yaml input.bin output.yaml
+```
+
+### Notes
+
+- You have to use the exact same order of field in the yaml as defined by the zserio schema, because zserio expects this
+- When converting from binary to YAML, the target YAML file must already exist and contain the necessary metadata.
+- The minimal metadata content in the target YAML file should be:
+
+```yaml
+_meta:
+  schema_module: <module_name>
+  schema_type: <type_name>
+```
+
+### Future Considerations
+
+In the future, we may consider using YAML tags for a more integrated approach to handling transformations directly within the YAML files, simplifying syntax and enhancing readability.
+
+## Project Structure
+
+- `setup.py`: Script for setting up the project and installing dependencies.
+- `zs_yaml/`: Directory containing the actual `zs-yaml` implementation.
+
+## Reference
+
+This project references the zserio serialization framework. For more information about zserio, visit the [zserio GitHub project](https://github.com/ndsev/zserio).

zs_yaml-0.3.0/VERSION

@@ -0,0 +1 @@
+0.1.3

zs_yaml-0.3.0/doc/zs-yaml.svg

@@ -0,0 +1,90 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+   viewBox="0 0 76.940711 22.655753"
+   id="zserio-logo"
+   version="1.1"
+   sodipodi:docname="zs-yaml.svg"
+   width="76.940712"
+   height="22.655754"
+   inkscape:version="1.3.2 (091e20ef0f, 2023-11-25)"
+   xml:space="preserve"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:dc="http://purl.org/dc/elements/1.1/"><metadata
+     id="metadata21"><rdf:RDF><cc:Work
+         rdf:about=""><dc:format>image/svg+xml</dc:format><dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" /></cc:Work></rdf:RDF></metadata><defs
+     id="defs19" /><sodipodi:namedview
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1"
+     objecttolerance="10"
+     gridtolerance="10"
+     guidetolerance="10"
+     inkscape:pageopacity="0"
+     inkscape:pageshadow="2"
+     inkscape:window-width="1920"
+     inkscape:window-height="935"
+     id="namedview17"
+     showgrid="false"
+     inkscape:zoom="5.6462916"
+     inkscape:cx="29.134167"
+     inkscape:cy="0.79698329"
+     inkscape:window-x="0"
+     inkscape:window-y="32"
+     inkscape:window-maximized="1"
+     inkscape:showpageshadow="2"
+     inkscape:pagecheckerboard="0"
+     inkscape:deskcolor="#d1d1d1"
+     inkscape:current-layer="zserio-logo" /><rect
+     style="fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:0.444658"
+     id="rect3"
+     width="76.940712"
+     height="22.655754"
+     x="0"
+     y="0" /><g
+     id="zserio-logo-serio"
+     fill="currentColor"
+     transform="matrix(0.96869663,0,0,0.96869663,5.4172856,2.76989)"><path
+       d="m 21.733415,11.886839 q 0,1.508249 -1.251904,2.474278 -1.242093,0.966048 -3.400868,0.966048 -1.222375,0 -2.24756,-0.285874 -1.015376,-0.295725 -1.705339,-0.640746 v -2.079999 h 0.09857 q 0.877324,0.660455 1.951776,1.054813 1.074434,0.38445 2.06028,0.38445 1.222375,0 1.912339,-0.394309 0.689963,-0.394309 0.690033,-1.242094 0,-0.650605 -0.37459,-0.985767 -0.37459,-0.335162 -1.439184,-0.571739 -0.394309,-0.08872 -1.035094,-0.207009 -0.630886,-0.118294 -1.153309,-0.256297 -1.449092,-0.384449 -2.06028,-1.1237795 -0.601318,-0.74918 -0.601318,-1.8335618 0,-0.6801737 0.276016,-1.2815312 0.285874,-0.6013179 0.857614,-1.0744331 0.552031,-0.4633054 1.399746,-0.7294612 0.857614,-0.2760152 1.912338,-0.2760152 0.985757,0 1.991214,0.2464369 1.015375,0.2365873 1.68562,0.5815989 V 6.593252 h -0.09857 Q 20.491193,6.0707998 19.475886,5.7159287 18.460511,5.3511981 17.484672,5.3511981 q -1.015375,0 -1.715248,0.394309 -0.699893,0.3844494 -0.699893,1.1533088 0,0.6801737 0.423878,1.0251853 0.414018,0.3450117 1.340589,0.5618801 0.512592,0.1182937 1.143499,0.2365873 0.640745,0.1182937 1.064623,0.2168685 1.291341,0.2957242 1.991214,1.0153751 0.699893,0.7294608 0.699893,1.9320578 z"
+       inkscape:connector-curvature="0"
+       id="path5" /></g><path
+     id="zserio-logo-z"
+     d="M 16.939948,17.513033 H 6.5404496 V 15.235944 L 12.376159,9.0354854 H 6.7770766 V 6.442894 H 16.831742 v 2.2376514 l -5.747023,6.1905486 h 5.855427 z"
+     inkscape:connector-curvature="0"
+     style="fill:#ff0000" /><g
+     id="g2"
+     transform="matrix(0.08129594,0,0,0.08129594,82.698086,11.100326)"><g
+       id="g1"
+       transform="matrix(1.3333333,0,0,1.3333333,-558.97901,-69.293599)"><polygon
+         points="1,109 73,1 258,1 258,19 318,109 "
+         style="fill:#ffffff;stroke:#000000;stroke-width:1"
+         id="polygon1" /><polygon
+         points="1,109 37,55 1,1 25,1 49,37 73,1 97,1 25,109 "
+         style="fill:#000000;stroke:#000000;stroke-width:1"
+         id="polygon2" /><polygon
+         points="127,1 103,1 31,109 55,109 103,37 103,109 127,109 "
+         style="fill:#000000;stroke:#000000;stroke-width:1"
+         id="polygon3" /><polygon
+         points="228,1 198,1 180,37 156,1 132,1 132,109 156,109 156,37 180,73 204,37 204,109 228,109 "
+         style="fill:#000000;stroke:#000000;stroke-width:1"
+         id="polygon4" /><polygon
+         points="318,109 302,85 258,85 258,1 234,1 234,109 "
+         style="fill:#000000;stroke:#000000;stroke-width:1"
+         id="polygon5" /></g></g><rect
+     style="fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.784084"
+     id="rect2"
+     width="6.5856104"
+     height="1.8180964"
+     x="29.616592"
+     y="10.931295" /><rect
+     style="fill:none;fill-opacity:1;stroke:#311620;stroke-width:0.721635;stroke-opacity:0.642554"
+     id="rect4"
+     width="73.180183"
+     height="19.131836"
+     x="2.0607479"
+     y="1.8520763" /></svg>

zs_yaml-0.3.0/requirements.txt

@@ -0,0 +1,2 @@
+PyYAML~=6.0
+zserio~=2.12

zs_yaml-0.3.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

zs_yaml-0.3.0/setup.py

@@ -0,0 +1,78 @@
+import os
+from setuptools import setup, find_packages
+import subprocess
+
+
+# Utility function to read file content into a string
+def read(fname):
+    return open(os.path.join(os.path.dirname(__file__), fname),
+                'r', encoding='utf-8').read()
+
+# DOCUMENTATION
+def read_for_pypi(fname):
+    """Read a file and replace relative image URLs with absolute URLs."""
+    content = read(fname)
+    # Replace relative image URLs with absolute URLs
+    # Adjust the base URL to where your images are actually hosted
+    base_url = "https://raw.githubusercontent.com/ndsev/zs_yaml/main/"
+    content = content.replace("](images/", f"]({base_url}images/")
+    return content
+
+
+# REQUIREMENTS
+def read_requirements(fname):
+    return [line.strip() for line in read(fname).splitlines() if line.strip() and not line.startswith('#')]
+
+# VERSION INFO
+def get_version_info():
+    version_file = os.path.join(os.path.dirname(__file__), 'zs_yaml', '_version.py')
+    version_info = {}
+    with open(version_file, 'r', encoding='utf-8') as f:
+        exec(f.read(), version_info)
+
+    # Get the Git commit ID if possible
+    try:
+        commit_id = subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).decode('ascii').strip()
+        version_info['__commit_id__'] = commit_id
+    except:
+        pass  # If Git is not available or not a Git repository, use the default "unknown"
+
+    return version_info
+
+version_info = get_version_info()
+
+# Write the updated version info back to the file
+with open(os.path.join('zs_yaml', '_version.py'), 'w', encoding='utf-8') as f:
+    f.write(f"__version__ = '{version_info['__version__']}'\n")
+    f.write(f"__commit_id__ = '{version_info['__commit_id__']}'\n")
+
+
+setup(
+    name='zs-yaml',
+    version=version_info['__version__'],
+    author="Navigation Data Standard e.V.",
+    author_email="support@nds-association.org",
+    description='A package that allows using yaml to define zserio objects.',
+    long_description=read_for_pypi('README.md'),
+    long_description_content_type="text/markdown",
+    url="https://github.com/ndsev/zs_yaml",
+    packages=find_packages(),
+    install_requires=read_requirements('requirements.txt'),
+    classifiers=[
+        "Programming Language :: Python :: 3",
+        "License :: OSI Approved :: MIT License",
+        "Operating System :: OS Independent",
+    ],
+    python_requires='>=3.8',
+    entry_points={
+        'console_scripts': [
+            'zs-yaml=zs_yaml.main:main'
+        ],
+    },
+    include_package_data=True,
+    package_data={
+        'zs_yaml': ['images/*'],
+    },
+    ext_modules=[],
+    cmdclass={}
+)

zs_yaml-0.3.0/zs_yaml/__init__.py

@@ -0,0 +1,6 @@
+from .convert import yaml_to_json, yaml_to_bin, bin_to_yaml, json_to_yaml
+from .yaml_transformer import YamlTransformer
+from ._version import __version__, __commit_id__
+
+def get_version_info():
+    return f"{__version__} (commit: {__commit_id__})"

zs_yaml-0.3.0/zs_yaml/_version.py

@@ -0,0 +1,2 @@
+__version__ = '0.3.0'
+__commit_id__ = 'a89ec0a'

zs_yaml-0.3.0/zs_yaml/built_in_transformations.py

@@ -0,0 +1,75 @@
+import json
+import importlib
+import zserio
+import yaml
+import copy
+
+def insert_yaml_as_extern(transformer, file, template_args=None):
+    """
+    Include external YAML by transforming it to JSON and using zserio.
+
+    Args:
+        transformer (YamlTransformer): The transformer instance.
+        file (str): Path to the external YAML file.
+        template_args (dict, optional): A dictionary of template arguments for placeholder replacement.
+
+    Returns:
+        dict: A dictionary containing the binary data and its bit size.
+    """
+    abs_path = transformer.resolve_path(file)
+    external_transformer = transformer.__class__(abs_path, template_args, initial_transformations=transformer.transformations)
+    processed_data = external_transformer.transform()
+    meta = external_transformer.get_meta()
+
+    schema_module = meta.get('schema_module')
+    schema_type = meta.get('schema_type')
+
+    if not schema_module or not schema_type:
+        raise ValueError(f"Error: schema_module and schema_type must be specified in the _meta section of the YAML file: {file}")
+
+    json_string = json.dumps(processed_data)
+
+    # Convert JSON to binary using zserio
+    module = importlib.import_module(schema_module)
+    ImportedType = getattr(module, schema_type)
+    zserio_object = zserio.from_json_string(ImportedType, json_string)
+    writer = zserio.BitStreamWriter()
+    zserio_object.write(writer)
+    bits = zserio.BitBuffer(writer.byte_array, writer.bitposition)
+
+    # Encode the binary data
+    encoded_bytes = list(bits.buffer)
+    data = {"buffer": encoded_bytes, "bitSize": bits.bitsize}
+
+    return data
+
+def insert_yaml(transformer, file, template_args=None):
+    """
+    Insert YAML content directly without converting to binary.
+
+    Args:
+        transformer (YamlTransformer): The transformer instance.
+        file (str): Path to the external YAML file.
+        template_args (dict, optional): A dictionary of template arguments for placeholder replacement.
+
+    Returns:
+        dict: The processed YAML data.
+    """
+    abs_path = transformer.resolve_path(file)
+    external_transformer = transformer.__class__(abs_path, template_args, initial_transformations=transformer.transformations)
+    processed_data = external_transformer.transform()
+    return processed_data
+
+def repeat_node(transformer, node, count):
+    """
+    Repeat a specific node n times and return as a list.
+
+    Args:
+        transformer (YamlTransformer): The transformer instance.
+        node: The node to be repeated.
+        count (int): The number of times to repeat the node.
+
+    Returns:
+        list: A list containing the repeated node.
+    """
+    return [copy.deepcopy(node) for _ in range(count)]

zs_yaml-0.3.0/zs_yaml/convert.py

@@ -0,0 +1,164 @@
+"""
+This module contains the logic to cover the following conversions:
+- yaml to bin: Converts a YAML file to a binary file using zserio serialization.
+               Custom transformations can be applied to the YAML data before conversion.
+- bin to yaml: Converts a binary file to a YAML file using zserio deserialization.
+               The conversion requires the schema module and type information to be
+               specified in the _meta section of the YAML file.
+- json to yaml: Converts a JSON file to a YAML file. This is a plain conversion without
+                applying any transformations.
+- yaml to json: Converts a YAML file to a JSON file. Custom transformations can be applied
+                to the YAML data before conversion.
+"""
+
+import yaml
+import json
+import importlib
+import zserio
+import tempfile
+import os
+
+from .yaml_transformer import YamlTransformer
+
+def _json_to_zs_bin(json_input_path, bin_output_path, module_name, type_name):
+    """
+    Converts a JSON file to a binary file using Zserio serialization.
+
+    Args:
+        json_input_path (str): Path to the input JSON file.
+        bin_output_path (str): Path to the output binary file.
+        module_name (str): Name of the Zserio module containing the schema.
+        type_name (str): Name of the Zserio type to be used for serialization.
+
+    Raises:
+        ValueError: If the specified Zserio type is not found in the module.
+    """
+    module = importlib.import_module(module_name)
+    ImportedType = getattr(module, type_name)
+    if ImportedType is None:
+        raise ValueError(f"Type {type_name} not found in module {module_name}")
+    zserio_object = zserio.from_json_file(ImportedType, json_input_path)
+    zserio.serialize_to_file(zserio_object, bin_output_path)
+
+def yaml_to_yaml(yaml_input_path, yaml_output_path=None):
+    """
+    Applies all transformations and template substitution to
+    the input YAML to come up with the output YAML.
+
+    Args:
+        yaml_input_path (str): Path to the input YAML file.
+        yaml_output_path (str): Path to the output YAML file.
+    """
+    transformer = YamlTransformer(yaml_input_path)
+    transformed_data = transformer.transform()
+    meta = transformer.get_meta()
+
+    if yaml_output_path:
+        output_data = {'_meta': meta}
+        output_data.update(transformed_data)
+        with open(yaml_output_path, 'w') as yaml_file:
+            yaml.dump(output_data, yaml_file, default_flow_style=False, sort_keys=False)
+
+    return transformed_data, meta
+
+
+def yaml_to_json(yaml_input_path, json_output_path):
+    """
+    Converts a YAML file to a JSON file.
+
+    Args:
+        yaml_input_path (str): Path to the input YAML file.
+        json_output_path (str): Path to the output JSON file.
+
+    Returns:
+        dict: The _meta section from the YAML file, if present.
+    """
+    transformed_data, meta = yaml_to_yaml(yaml_input_path)
+
+    with open(json_output_path, 'w') as json_file:
+        json.dump(transformed_data, json_file, indent=2)
+
+    return meta
+
+
+def json_to_yaml(json_input_path, yaml_output_path):
+    """
+    Converts a JSON file to a YAML file. This is a plain conversion without applying
+    any transformations.
+
+    Args:
+        json_input_path (str): Path to the input JSON file.
+        yaml_output_path (str): Path to the output YAML file.
+    """
+    with open(json_input_path, 'r') as json_file:
+        data = json.load(json_file)
+    with open(yaml_output_path, 'w') as yaml_file:
+        yaml.safe_dump(data, yaml_file, default_flow_style=False, sort_keys=False)
+
+
+def yaml_to_bin(yaml_input_path, bin_output_path):
+    """
+    Converts a YAML file to a binary file using Zserio serialization.
+
+    Args:
+        yaml_input_path (str): Path to the input YAML file.
+        bin_output_path (str): Path to the output binary file.
+
+    Raises:
+        ValueError: If schema_module and schema_type are not specified in the _meta
+            section of the YAML file.
+    """
+    with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as temp_json_file:
+            temp_json_path = temp_json_file.name
+
+    try:
+        meta = yaml_to_json(yaml_input_path, temp_json_path)
+
+        schema_module = meta.get('schema_module')
+        schema_type = meta.get('schema_type')
+
+        if not schema_module or not schema_type:
+            raise ValueError("Error: schema_module and schema_type must be specified in the _meta section for binary output")
+
+        _json_to_zs_bin(temp_json_path, bin_output_path, schema_module, schema_type)
+    finally:
+        os.remove(temp_json_path)
+
+
+def bin_to_yaml(bin_input_path, yaml_output_path):
+    """
+    Converts a binary file to a YAML file using Zserio deserialization.
+
+    Args:
+        bin_input_path (str): Path to the input binary file.
+        yaml_output_path (str): Path to the output YAML file.
+
+    Raises:
+        ValueError: If schema_module and schema_type are not specified in the _meta
+            section of the YAML file, or if the specified Zserio type is not found
+            in the module.
+    """
+    with open(yaml_output_path, 'r') as yaml_file:
+        meta = yaml.safe_load(yaml_file)
+
+    schema_module = meta.get('_meta', {}).get('schema_module')
+    schema_type = meta.get('_meta', {}).get('schema_type')
+    if not schema_module or not schema_type:
+        raise ValueError("Error: schema_module and schema_type must be specified in the _meta section of the YAML file")
+
+    module = importlib.import_module(schema_module)
+    ImportedType = getattr(module, schema_type)
+    if ImportedType is None:
+        raise ValueError(f"Type {schema_type} not found in module {schema_module}")
+
+    zserio_object = zserio.deserialize_from_file(ImportedType, bin_input_path)
+    json_data = zserio.to_json_string(zserio_object)
+
+    data = json.loads(json_data)
+
+    # Create a new dictionary to ensure _meta comes first
+    final_data = {'_meta': meta['_meta']}
+    final_data.update(data)
+
+    with open(yaml_output_path, 'w') as yaml_file:
+        yaml.safe_dump(final_data, yaml_file, default_flow_style=False, sort_keys=False)

zs_yaml-0.3.0/zs_yaml/main.py

@@ -0,0 +1,119 @@
+import argparse
+import traceback
+import sys
+import os
+from zs_yaml.convert import (
+    yaml_to_bin,
+    bin_to_yaml,
+    yaml_to_json,
+    json_to_yaml,
+    yaml_to_yaml
+    )
+from zs_yaml import get_version_info
+import time
+
+def parse_arguments():
+    parser = argparse.ArgumentParser(
+        formatter_class=argparse.RawTextHelpFormatter,
+
+        description=
+        f'%(prog)s {get_version_info()}\n\n'
+        'Converts between YAML, JSON, and binary formats. '
+        'To convert from binary to YAML, the target YAML file must already exist with metadata. '
+        'The metadata is required to identify the correct Python type for deserialization.\n\n'
+        'The minimal metadata content in the target YAML file should be:\n'
+        ' _meta:\n'
+        ' schema_module: <module_name>\n'
+        ' schema_type: <type_name>',
+
+        usage='%(prog)s <input_path> [output_path]\n\n'
+        'Example usage:\n'
+        ' %(prog)s input.yaml output.bin\n'
+        ' %(prog)s input.bin output.yaml\n'
+        ' %(prog)s input.yaml output.json\n'
+        ' %(prog)s input.json output.yaml\n'
+        ' %(prog)s input.yaml (output will be inferred as binary if not specified)'
+    )
+    parser.add_argument('input_path', type=str, help='Path to the input file (YAML, JSON, or binary)')
+    parser.add_argument('output_path', type=str, nargs='?', help='Path to the output file (YAML, JSON, or binary)')
+    parser.add_argument('--version', action='version', version=f'%(prog)s {get_version_info()}')
+
+    if len(sys.argv) < 2:
+        parser.print_help(sys.stderr)
+        sys.exit(1)
+
+    return parser.parse_args()
+
+def process_yaml_input(input_path, output_path):
+    if not output_path:
+        output_extension = '.bin'
+        output_path = os.path.splitext(input_path)[0] + output_extension
+    else:
+        output_extension = os.path.splitext(output_path)[1].lower()
+
+    if output_extension == '.bin' or output_extension == '':
+        yaml_to_bin(input_path, output_path)
+    elif output_extension == '.json':
+        yaml_to_json(input_path, output_path)
+    elif output_extension == '.yaml':
+        yaml_to_yaml(input_path, output_path)
+    else:
+        raise ValueError("Unsupported output file extension for YAML input")
+
+    return output_path
+
+def process_binary_input(input_path, output_path):
+    if not output_path:
+        raise ValueError("Output path must be specified for binary input")
+    bin_to_yaml(input_path, output_path)
+    return output_path
+
+def process_json_input(input_path, output_path):
+    if not output_path:
+        output_extension = '.yaml'
+        output_path = os.path.splitext(input_path)[0] + output_extension
+    else:
+        output_extension = os.path.splitext(output_path)[1].lower()
+
+    if output_extension == '.yaml':
+        json_to_yaml(input_path, output_path)
+    else:
+        raise ValueError("Unsupported output file extension for JSON input")
+
+    return output_path
+
+def get_file_size(file_path):
+    return os.path.getsize(file_path)
+
+def print_summary(start_time, output_path):
+    end_time = time.time()
+    execution_time = end_time - start_time
+    output_size = get_file_size(output_path)
+    print(f"Generated {output_path} (size: {output_size/1024.0:.0f}KB) in {execution_time:.2f} seconds.")
+
+def main():
+    args = parse_arguments()
+
+    input_extension = os.path.splitext(args.input_path)[1].lower()
+
+    start_time = time.time()
+
+    try:
+        if input_extension == '.yaml':
+            output_path = process_yaml_input(args.input_path, args.output_path)
+        elif input_extension == '.bin' or input_extension == '':
+            output_path = process_binary_input(args.input_path, args.output_path)
+        elif input_extension == '.json':
+            output_path = process_json_input(args.input_path, args.output_path)
+        else:
+            raise ValueError("Unsupported input file extension")
+
+        print_summary(start_time, output_path)
+
+    except Exception as e:
+        print(f"Error processing file: {e}")
+        traceback.print_exc()
+        sys.exit(1)
+
+if __name__ == "__main__":
+    main()

zs_yaml-0.3.0/zs_yaml/yaml_transformer.py

@@ -0,0 +1,112 @@
+import importlib
+import importlib.util
+import json
+import os
+import yaml
+import zs_yaml.built_in_transformations
+from string import Template
+
+class YamlTransformer:
+    """
+    Encapsulates the transformation of yaml data using
+    invocations of registered transformation functions.
+    """
+
+    # Class-level cache for loaded modules
+    _loaded_modules = {}
+
+    def __init__(self, yaml_file_path, template_args=None, initial_transformations=None):
+        self.yaml_file_path = yaml_file_path
+        self.yaml_dir = os.path.dirname(os.path.abspath(yaml_file_path))
+        self.transformations = initial_transformations or {}
+        self._load_functions(zs_yaml.built_in_transformations)
+        self.meta = {}
+        self.template_args = template_args
+        self.yaml_data = None
+
+    def resolve_path(self, path):
+        if os.path.isabs(path):
+            return path
+        return os.path.normpath(os.path.join(self.yaml_dir, path))
+
+    def transform(self):
+        if self.yaml_data is None:
+            self._load_yaml()
+
+        transformation_module = self.meta.get('transformation_module')
+
+        if transformation_module:
+            self._load_functions(transformation_module)
+
+        return self._process(self.yaml_data)
+
+    def to_json(self):
+        processed_data = self.transform()
+        return json.dumps(processed_data, indent=2)
+
+    def get_meta(self):
+        if self.yaml_data is None:
+            self._load_yaml()
+        return self.meta
+
+    @classmethod
+    def _load_module_from_file(cls, module_name, file_path):
+        abs_path = os.path.abspath(file_path)
+        if abs_path in cls._loaded_modules:
+            return cls._loaded_modules[abs_path]
+
+        spec = importlib.util.spec_from_file_location(module_name, abs_path)
+        module = importlib.util.module_from_spec(spec)
+        spec.loader.exec_module(module)
+        cls._loaded_modules[abs_path] = module
+        return module
+
+    def _load_functions(self, transformation_module):
+        if isinstance(transformation_module, str):
+            if transformation_module.endswith('.py'):
+                abs_path = self.resolve_path(transformation_module)
+                transformation_module = self._load_module_from_file('custom_transformations', abs_path)
+            else:
+                transformation_module = importlib.import_module(transformation_module)
+
+        for name, func in vars(transformation_module).items():
+            if callable(func):
+                self._register_function(name, func)
+
+    def _register_function(self, name, func):
+        if name in self.transformations:
+            if self.transformations[name] != func:
+                raise ValueError(f"Attempting to register a different function with an existing name: {name}")
+        else:
+            self.transformations[name] = func
+
+    def _get_function(self, name):
+        return self.transformations.get(name)
+
+    def _process(self, data):
+        if isinstance(data, list):
+            return [self._process(item) for item in data]
+        elif isinstance(data, dict):
+            if '_f' in data and '_a' in data:
+                func = self._get_function(data['_f'])
+                args = data['_a']
+                if func and callable(func):
+                    if isinstance(args, dict):
+                        return func(self, **args)
+                    else:
+                        return func(self, args)
+                else:
+                    raise ValueError(f"Function {data['_f']} not found or is not callable")
+            else:
+                return {key: self._process(value) for key, value in data.items()}
+        return data
+
+    def _load_yaml(self):
+        with open(self.yaml_file_path, 'r') as yaml_file:
+            content = yaml_file.read()
+
+        if self.template_args:
+            content = Template(content).safe_substitute(self.template_args)
+
+        self.yaml_data = yaml.safe_load(content)
+        self.meta = self.yaml_data.pop('_meta', {})

zs_yaml-0.3.0/zs_yaml.egg-info/PKG-INFO

@@ -0,0 +1,174 @@
+Metadata-Version: 2.1
+Name: zs-yaml
+Version: 0.3.0
+Summary: A package that allows using yaml to define zserio objects.
+Home-page: https://github.com/ndsev/zs_yaml
+Author: Navigation Data Standard e.V.
+Author-email: support@nds-association.org
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: PyYAML~=6.0
+Requires-Dist: zserio~=2.12
+
+# `zs-yaml`
+
+*Your Easy Path to zserio Test Data*
+
+<img src="doc/zs-yaml.png" alt="" height="100">
+
+**zs-yaml** adds YAML as a text format to be used with zserio in order to improve the user experience when manually creating or analyzing test data for zserio.
+
+- **YAML Format**: Uses YAML's human-readable format instead of JSON as the primary format for editing, making data handling and transformation more intuitive and less cluttered.
+- **Metadata Inclusion**: Automatically includes metadata in the YAML, eliminating the need for users to manually identify the correct type when importing JSON data, ensuring seamless (de-)serialization.
+- **Custom Transformations**: Allows for hooking in custom transformations so that users can work with familiar formats (e.g., dates or coordinate representations) instead of thinking in unfamiliar formats.
+
+## Example
+
+### Zserio Schema Module Creation
+
+1. **Create the Zserio schema**:
+   - Create a file named `person.zs` with the following content:
+
+     ```zserio
+     package person;
+
+     struct Person
+     {
+         string name;
+         string birthdate;
+         string birth_location;
+         string current_job;
+         int32 income;
+         RoleExperience experience[];
+     };
+
+     struct RoleExperience
+     {
+         string role;
+         int32 years;
+     };
+     ```
+
+2. **Compile the Zserio schema and generate Python APIs**:
+   - Run the following command to compile the schema and generate Python sources:
+
+     ```sh
+     # Generated sources needs type infos so that 'json <-> bin'
+     # conversion works, which is utilized by zs-yaml
+     zserio person.zs -withTypeInfoCode -python zs_gen_api
+     ```
+
+3. **Ensure that the schema modules are available in the Python environment**:
+   - Export the `PYTHONPATH` to include the directory containing the generated Python sources:
+
+     ```sh
+     export PYTHONPATH="zs_gen_api"
+     ```
+
+### Use zs-yaml to create data
+
+Using **zs-yaml**, you can define the same data in a more human-readable YAML format and include necessary metadata along with a custom transformation for the birthdate:
+
+```yaml
+# 1) Metadata is used to specify the type needed for
+#    (de-)serialization and custom transform functions
+# 2) User are free to use their preferred date format
+#    for the birth data as the a normalization function
+#    get invoked.
+# 3) Yaml allows avoiding clutter and adding comments
+#    like this one :)
+
+_meta:
+  schema_module: person.api
+  schema_type: Person
+  transformation_module: "./transformations.py"
+
+name: John Doe
+
+birthdate:
+  _f: normalize_date
+  _a: "01/01/1990"
+birth_location: Springfield, USA
+
+current_job: Software Engineer
+income: 75000
+
+experience:
+  - role: Intern
+    years: 1
+  - role: Junior Developer
+    years: 2
+  - role: Senior Developer
+    years: 3
+```
+
+#### Example Transformation Module (transformations.py)
+
+```python
+def normalize_date(date_str):
+    from datetime import datetime
+    for fmt in ("%Y-%m-%d", "%d/%m/%Y", "%m/%d/%Y"):
+        try:
+            return datetime.strptime(date_str, fmt).strftime("%Y-%m-%d")
+        except ValueError:
+            pass
+    raise ValueError(f"Date {date_str} is not in a recognized format")
+```
+
+#### Creating the Binary Representation
+
+After you have installed `zs-yaml`, call `zs-yaml` to convert your YAML file to a binary representation:
+
+```sh
+# Install zs-yaml if not already installed.
+
+# Create the binary representation from the YAML file
+zs-yaml person.yaml person.bin
+```
+
+## Installation
+
+Currently, the Python package is only available at test.pypi. Once the official release is available,
+you can install the package without specifying an index URL.
+
+```bash
+python -m pip install --upgrade zs-yaml
+```
+
+## Usage
+
+The main entry point for the application is `main.py`. It accepts arguments for specifying the input and output file paths. You can run the application as follows:
+
+```bash
+zs-yaml input.yaml output.bin
+zs-yaml input.bin output.yaml
+```
+
+### Notes
+
+- You have to use the exact same order of field in the yaml as defined by the zserio schema, because zserio expects this
+- When converting from binary to YAML, the target YAML file must already exist and contain the necessary metadata.
+- The minimal metadata content in the target YAML file should be:
+
+```yaml
+_meta:
+  schema_module: <module_name>
+  schema_type: <type_name>
+```
+
+### Future Considerations
+
+In the future, we may consider using YAML tags for a more integrated approach to handling transformations directly within the YAML files, simplifying syntax and enhancing readability.
+
+## Project Structure
+
+- `setup.py`: Script for setting up the project and installing dependencies.
+- `zs_yaml/`: Directory containing the actual `zs-yaml` implementation.
+
+## Reference
+
+This project references the zserio serialization framework. For more information about zserio, visit the [zserio GitHub project](https://github.com/ndsev/zserio).

zs_yaml-0.3.0/zs_yaml.egg-info/SOURCES.txt

@@ -0,0 +1,20 @@
+LICENSE
+MANIFEST.in
+README.md
+VERSION
+requirements.txt
+setup.py
+doc/zs-yaml.png
+doc/zs-yaml.svg
+zs_yaml/__init__.py
+zs_yaml/_version.py
+zs_yaml/built_in_transformations.py
+zs_yaml/convert.py
+zs_yaml/main.py
+zs_yaml/yaml_transformer.py
+zs_yaml.egg-info/PKG-INFO
+zs_yaml.egg-info/SOURCES.txt
+zs_yaml.egg-info/dependency_links.txt
+zs_yaml.egg-info/entry_points.txt
+zs_yaml.egg-info/requires.txt
+zs_yaml.egg-info/top_level.txt

zs_yaml-0.3.0/zs_yaml.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

zs_yaml-0.3.0/zs_yaml.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+zs-yaml = zs_yaml.main:main

zs_yaml-0.3.0/zs_yaml.egg-info/requires.txt

@@ -0,0 +1,2 @@
+PyYAML~=6.0
+zserio~=2.12

zs_yaml-0.3.0/zs_yaml.egg-info/top_level.txt

@@ -0,0 +1 @@
+zs_yaml