vtlengine 1.1rc2__tar.gz → 1.2.0__tar.gz

vtlengine-1.2.0/PKG-INFO

@@ -0,0 +1,92 @@
+Metadata-Version: 2.3
+Name: vtlengine
+Version: 1.2.0
+Summary: Run and Validate VTL Scripts
+License: AGPL-3.0
+Keywords: vtl,sdmx,vtlengine,Validation and Transformation Language
+Author: MeaningfulData
+Author-email: info@meaningfuldata.eu
+Maintainer: Francisco Javier Hernandez del Caño
+Maintainer-email: javier.hernandez@meaningfuldata.eu
+Requires-Python: >=3.9
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: Intended Audience :: Science/Research
+Classifier: Typing :: Typed
+Provides-Extra: all
+Provides-Extra: s3
+Requires-Dist: antlr4-python3-runtime (>=4.13.2,<4.14)
+Requires-Dist: duckdb (>=1.1,<1.2)
+Requires-Dist: fsspec (>=2022.11.0,<2023.0) ; extra == "all"
+Requires-Dist: fsspec (>=2022.11.0,<2023.0) ; extra == "s3"
+Requires-Dist: jsonschema (>=3.2.0,<5.0)
+Requires-Dist: networkx (>=2.8,<3.0)
+Requires-Dist: numpy (>=1.23.2,<2) ; python_version < "3.13"
+Requires-Dist: numpy (>=2.1.0) ; python_version >= "3.13"
+Requires-Dist: pandas (>=2.1.4,<3.0)
+Requires-Dist: pysdmx[xml] (>=1.4.0rc1,<2.0)
+Requires-Dist: s3fs (>=2022.11.0,<2023.0) ; extra == "all"
+Requires-Dist: s3fs (>=2022.11.0,<2023.0) ; extra == "s3"
+Requires-Dist: sqlglot (>=22.2.0,<23.0)
+Project-URL: Authors, https://github.com/Meaningful-Data/vtlengine/graphs/contributors
+Project-URL: Documentation, https://docs.vtlengine.meaningfuldata.eu
+Project-URL: IssueTracker, https://github.com/Meaningful-Data/vtlengine/issues
+Project-URL: MeaningfulData, https://www.meaningfuldata.eu/
+Project-URL: Repository, https://github.com/Meaningful-Data/vtlengine
+Description-Content-Type: text/markdown
+
+# VTL Engine
+
+|              |                                                                                                                                                                                |
+|--------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Testing      | [![Testing](https://github.com/Meaningful-Data/vtlengine/actions/workflows/testing.yml/badge.svg)](https://github.com/Meaningful-Data/vtlengine/actions/workflows/testing.yml) |
+| Package      | [![PyPI Latest Release](https://img.shields.io/pypi/v/vtlengine.svg)](https://pypi.org/project/vtlengine/)                                                                     |
+| License      | [![License - AGPL 3.0](https://img.shields.io/pypi/l/vtlengine.svg)](https://github.com/Meaningful-Data/vtlengine/blob/main/LICENSE.md)                                        |
+| Mentioned in | [![Mentioned in Awesome Official Statistics ](https://awesome.re/mentioned-badge.svg)](https://github.com/SNStatComp/awesome-official-statistics-software)                                               |
+
+## Introduction
+
+The VTL Engine is a Python library that allows you to validate, format and execute VTL scripts.
+
+It is a Python-based library around
+the [VTL Language 2.1](https://sdmx-twg.github.io/vtl/2.1/html/index.html).
+
+## Useful Links
+
+- [MeaningfulData: who we are](https://www.meaningfuldata.eu)
+- [Documentation](https://docs.vtlengine.meaningfuldata.eu)
+- [Source Code](https://github.com/Meaningful-Data/vtlengine)
+- [Bug Tracker](https://github.com/Meaningful-Data/vtlengine/issues?q=is%3Aopen+is%3Aissue+label%3Abug)
+- [New features Tracker](https://github.com/Meaningful-Data/vtlengine/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement)
+
+## Installation
+
+### Requirements
+
+The VTL Engine requires Python 3.9 or higher.
+
+### Install with pip
+
+To install the VTL Engine on any Operating System, you can use pip:
+
+```bash
+
+pip install vtlengine
+
+```
+
+*Note: it is recommended to install the VTL Engine in a virtual environment.*
+
+### S3 extra
+
+If you want to use the S3 functionality, you can install the VTL Engine with the `s3` extra:
+
+```bash
+pip install vtlengine[s3]
+```
+
+## Documentation
+
+The documentation for the VTL Engine is available
+at [docs.vtlengine.meaningfuldata.eu](https://docs.vtlengine.meaningfuldata.eu).

vtlengine-1.2.0/README.md

@@ -0,0 +1,54 @@
+# VTL Engine
+
+|              |                                                                                                                                                                                |
+|--------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Testing      | [![Testing](https://github.com/Meaningful-Data/vtlengine/actions/workflows/testing.yml/badge.svg)](https://github.com/Meaningful-Data/vtlengine/actions/workflows/testing.yml) |
+| Package      | [![PyPI Latest Release](https://img.shields.io/pypi/v/vtlengine.svg)](https://pypi.org/project/vtlengine/)                                                                     |
+| License      | [![License - AGPL 3.0](https://img.shields.io/pypi/l/vtlengine.svg)](https://github.com/Meaningful-Data/vtlengine/blob/main/LICENSE.md)                                        |
+| Mentioned in | [![Mentioned in Awesome Official Statistics ](https://awesome.re/mentioned-badge.svg)](https://github.com/SNStatComp/awesome-official-statistics-software)                                               |
+
+## Introduction
+
+The VTL Engine is a Python library that allows you to validate, format and execute VTL scripts.
+
+It is a Python-based library around
+the [VTL Language 2.1](https://sdmx-twg.github.io/vtl/2.1/html/index.html).
+
+## Useful Links
+
+- [MeaningfulData: who we are](https://www.meaningfuldata.eu)
+- [Documentation](https://docs.vtlengine.meaningfuldata.eu)
+- [Source Code](https://github.com/Meaningful-Data/vtlengine)
+- [Bug Tracker](https://github.com/Meaningful-Data/vtlengine/issues?q=is%3Aopen+is%3Aissue+label%3Abug)
+- [New features Tracker](https://github.com/Meaningful-Data/vtlengine/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement)
+
+## Installation
+
+### Requirements
+
+The VTL Engine requires Python 3.9 or higher.
+
+### Install with pip
+
+To install the VTL Engine on any Operating System, you can use pip:
+
+```bash
+
+pip install vtlengine
+
+```
+
+*Note: it is recommended to install the VTL Engine in a virtual environment.*
+
+### S3 extra
+
+If you want to use the S3 functionality, you can install the VTL Engine with the `s3` extra:
+
+```bash
+pip install vtlengine[s3]
+```
+
+## Documentation
+
+The documentation for the VTL Engine is available
+at [docs.vtlengine.meaningfuldata.eu](https://docs.vtlengine.meaningfuldata.eu).

{vtlengine-1.1rc2 → vtlengine-1.2.0}/pyproject.toml

@@ -1,10 +1,10 @@
 [project]
 name = "vtlengine"
-version = "1.1rc2"
+version = "1.2.0"
 description = "Run and Validate VTL Scripts"
 license = "AGPL-3.0"
 readme = "README.md"
-requires-python = ">=3.9,<4"
+requires-python = ">=3.9"
 authors = [
     {name = "MeaningfulData", email = "info@meaningfuldata.eu"},
 ]
@@ -25,13 +25,15 @@ keywords = ['vtl', 'sdmx', 'vtlengine', 'Validation and Transformation Language'
 dependencies = [
     # PyPi dependencies
     "duckdb>=1.1,<1.2",
-
+    "pysdmx[xml]>=1.4.0rc1,<2.0",
     # APT-supported dependencies
     "jsonschema>=3.2.0,<5.0",
     "sqlglot>=22.2.0,<23.0",
-    "antlr4-python3-runtime>=4.9.2,<4.10",
-    "pandas>=2.1.4,<2.2",
+    "antlr4-python3-runtime>=4.13.2,<4.14",
+    "pandas>=2.1.4,<3.0",
     "networkx>=2.8,<3.0",
+    "numpy>=1.23.2,<2 ; python_version < '3.13'",
+    "numpy>=2.1.0; python_version >= '3.13'",
 ]
 
 [project.optional-dependencies]
@@ -45,20 +47,26 @@ MeaningfulData = 'https://www.meaningfuldata.eu/'
 IssueTracker = 'https://github.com/Meaningful-Data/vtlengine/issues'
 Authors = 'https://github.com/Meaningful-Data/vtlengine/graphs/contributors'
 
+[tool.poetry.dependencies]
+python = ">=3.9,<4.0"
+
 [tool.poetry.group.dev.dependencies]
-pytest = "^8.3"
-pytest-cov = "^6.0.0"
+pytest = "^8.4"
+pytest-cov = "^6.2.1"
+pytest-xdist = "^3.8.0"
 line-profiler-pycharm = "^1.2.0"
-mypy = "1.14.1"
-pandas-stubs = "^2.1.4.231227"
-ruff = "^0.9.4"
-types-jsonschema = "4.23.0.20241208"
+mypy = "1.16.1"
+pandas-stubs = "2.2.2.240807"
+ruff = "^0.12.3"
+types-jsonschema = "4.24.0.20250708"
 
 [tool.poetry.group.docs.dependencies]
 sphinx = "^7.4.7"
 sphinx-rtd-theme = "^3.0.2"
 toml = "^0.10.2"
 
+
+
 [tool.ruff]
 line-length = 100
 lint.mccabe.max-complexity = 20
@@ -68,7 +76,7 @@ lint.select = [
 # TODO: check S608 (duckdb querys)
 lint.ignore = ["B023", "B028", "B904", "C403", "D100", "D101", "D102", "D103", "D104", "D105",
     "D107", "D200", "D201", "D202", "D203", "D205", "D209", "D212", "D213", "D301",
-    "D400", "D401", "D404", "D411", "D413", "D415", "D419", "E203", "S320", "S608"]
+    "D400", "D401", "D404", "D411", "D413", "D415", "D419", "E203", "S608"]
 lint.exclude = ["*/Grammar/*"]
 
 [tool.ruff.lint.per-file-ignores]

{vtlengine-1.1rc2 → vtlengine-1.2.0}/src/vtlengine/API/_InternalApi.py

@@ -1,18 +1,37 @@
 import json
 import os
 from pathlib import Path
-from typing import Any, Dict, List, Optional, Union
+from typing import Any, Dict, List, Literal, Optional, Tuple, Union
 
 import jsonschema
 import pandas as pd
+from pysdmx.model.dataflow import Component as SDMXComponent
+from pysdmx.model.dataflow import DataStructureDefinition, Schema
+from pysdmx.model.dataflow import Role as SDMX_Role
+from pysdmx.model.vtl import (
+    Ruleset,
+    RulesetScheme,
+    Transformation,
+    TransformationScheme,
+    UserDefinedOperator,
+    UserDefinedOperatorScheme,
+)
 
+from vtlengine import AST as AST
 from vtlengine.__extras_check import __check_s3_extra
-from vtlengine.AST import PersistentAssignment, Start
+from vtlengine.AST import Assignment, DPRuleset, HRuleset, Operator, PersistentAssignment, Start
+from vtlengine.AST.ASTString import ASTString
 from vtlengine.DataTypes import SCALAR_TYPES
-from vtlengine.Exceptions import InputValidationException, check_key
+from vtlengine.Exceptions import (
+    InputValidationException,
+    SemanticError,
+    check_key,
+)
 from vtlengine.files.parser import _fill_dataset_empty_data, _validate_pandas
 from vtlengine.Model import (
-    Component,
+    Component as VTL_Component,
+)
+from vtlengine.Model import (
     Dataset,
     ExternalRoutine,
     Role,
@@ -20,18 +39,23 @@ from vtlengine.Model import (
     Scalar,
     ValueDomain,
 )
+from vtlengine.Utils import VTL_DTYPES_MAPPING, VTL_ROLE_MAPPING
 
 base_path = Path(__file__).parent
 schema_path = base_path / "data" / "schema"
+sdmx_csv_path = base_path / "data" / "sdmx_csv"
 with open(schema_path / "json_schema_2.1.json", "r") as file:
     schema = json.load(file)
 
 
-def _load_dataset_from_structure(structures: Dict[str, Any]) -> Dict[str, Any]:
+def _load_dataset_from_structure(
+    structures: Dict[str, Any],
+) -> Tuple[Dict[str, Any], Dict[str, Any]]:
     """
     Loads a dataset with the structure given.
     """
     datasets = {}
+    scalars = {}
 
     if "datasets" in structures:
         for dataset_json in structures["datasets"]:
@@ -66,7 +90,7 @@ def _load_dataset_from_structure(structures: Dict[str, Any]) -> Dict[str, Any]:
                         else:
                             component["nullable"] = False
 
-                    components[component["name"]] = Component(
+                    components[component["name"]] = VTL_Component(
                         name=component["name"],
                         data_type=SCALAR_TYPES[component["data_type"]],
                         role=Role(component["role"]),
@@ -77,7 +101,7 @@ def _load_dataset_from_structure(structures: Dict[str, Any]) -> Dict[str, Any]:
                 for component in dataset_json["DataStructure"]:
                     check_key("data_type", SCALAR_TYPES.keys(), component["type"])
                     check_key("role", Role_keys, component["role"])
-                    components[component["name"]] = Component(
+                    components[component["name"]] = VTL_Component(
                         name=component["name"],
                         data_type=SCALAR_TYPES[component["type"]],
                         role=Role(component["role"]),
@@ -93,8 +117,8 @@ def _load_dataset_from_structure(structures: Dict[str, Any]) -> Dict[str, Any]:
                 data_type=SCALAR_TYPES[scalar_json["type"]],
                 value=None,
             )
-            datasets[scalar_name] = scalar  # type: ignore[assignment]
-    return datasets
+            scalars[scalar_name] = scalar
+    return datasets, scalars
 
 
 def _load_single_datapoint(datapoint: Union[str, Path]) -> Dict[str, Any]:
@@ -142,7 +166,9 @@ def _load_datapoints_path(
     return _load_single_datapoint(datapoints)
 
 
-def _load_datastructure_single(data_structure: Union[Dict[str, Any], Path]) -> Dict[str, Dataset]:
+def _load_datastructure_single(
+    data_structure: Union[Dict[str, Any], Path],
+) -> Tuple[Dict[str, Dataset], Dict[str, Scalar]]:
     """
     Loads a single data structure.
     """
@@ -153,13 +179,15 @@ def _load_datastructure_single(data_structure: Union[Dict[str, Any], Path]) -> D
     if not data_structure.exists():
         raise Exception("Invalid datastructure. Input does not exist")
     if data_structure.is_dir():
-        datasets: Dict[str, Any] = {}
+        datasets: Dict[str, Dataset] = {}
+        scalars: Dict[str, Scalar] = {}
         for f in data_structure.iterdir():
             if f.suffix != ".json":
                 continue
-            dataset = _load_datastructure_single(f)
-            datasets = {**datasets, **dataset}
-        return datasets
+            ds, sc = _load_datastructure_single(f)
+            datasets = {**datasets, **ds}
+            scalars = {**scalars, **sc}
+        return datasets, scalars
     else:
         if data_structure.suffix != ".json":
             raise Exception("Invalid datastructure. Must have .json extension")
@@ -169,8 +197,8 @@ def _load_datastructure_single(data_structure: Union[Dict[str, Any], Path]) -> D
 
 
 def load_datasets(
-    data_structure: Union[Dict[str, Any], Path, List[Union[Dict[str, Any], Path]]],
-) -> Dict[str, Dataset]:
+    data_structure: Union[Dict[str, Any], Path, List[Dict[str, Any]], List[Path]],
+) -> Tuple[Dict[str, Dataset], Dict[str, Scalar]]:
     """
     Loads multiple datasets.
 
@@ -188,21 +216,42 @@ def load_datasets(
     if isinstance(data_structure, dict):
         return _load_datastructure_single(data_structure)
     if isinstance(data_structure, list):
-        ds_structures: Dict[str, Any] = {}
+        ds_structures: Dict[str, Dataset] = {}
+        scalar_structures: Dict[str, Scalar] = {}
         for x in data_structure:
-            result = _load_datastructure_single(x)
-            ds_structures = {**ds_structures, **result}  # Overwrite ds_structures dict.
-        return ds_structures
+            ds, sc = _load_datastructure_single(x)
+            ds_structures = {**ds_structures, **ds}  # Overwrite ds_structures dict.
+            scalar_structures = {**scalar_structures, **sc}  # Overwrite scalar_structures dict.
+        return ds_structures, scalar_structures
     return _load_datastructure_single(data_structure)
 
 
-def load_datasets_with_data(data_structures: Any, datapoints: Optional[Any] = None) -> Any:
+def _handle_scalars_values(
+    scalars: Dict[str, Scalar],
+    scalar_values: Optional[Dict[str, Optional[Union[int, str, bool, float]]]] = None,
+) -> None:
+    if scalar_values is None:
+        return
+    # Handling scalar values with the scalar dict
+    for name, value in scalar_values.items():
+        if name not in scalars:
+            raise Exception(f"Not found scalar {name} in datastructures")
+        # Casting value to scalar data type
+        scalars[name].value = scalars[name].data_type.cast(value)
+
+
+def load_datasets_with_data(
+    data_structures: Any,
+    datapoints: Optional[Any] = None,
+    scalar_values: Optional[Dict[str, Optional[Union[int, str, bool, float]]]] = None,
+) -> Any:
     """
     Loads the dataset structures and fills them with the data contained in the datapoints.
 
     Args:
         data_structures: Dict, Path or a List of dicts or Paths.
         datapoints: Dict, Path or a List of Paths.
+        scalar_values: Dict with the scalar values.
 
     Returns:
         A dict with the structure and a pandas dataframe with the data.
@@ -210,17 +259,18 @@ def load_datasets_with_data(data_structures: Any, datapoints: Optional[Any] = No
     Raises:
         Exception: If the Path is wrong or the file is invalid.
     """
-    datasets = load_datasets(data_structures)
+    datasets, scalars = load_datasets(data_structures)
     if datapoints is None:
         for dataset in datasets.values():
             if isinstance(dataset, Dataset):
                 _fill_dataset_empty_data(dataset)
-        return datasets, None
+        _handle_scalars_values(scalars, scalar_values)
+        return datasets, scalars, None
     if isinstance(datapoints, dict):
         # Handling dictionary of Pandas Dataframes
         for dataset_name, data in datapoints.items():
             if dataset_name not in datasets:
-                raise Exception(f"Not found dataset {dataset_name}")
+                raise Exception(f"Not found dataset {dataset_name} in datastructures.")
             datasets[dataset_name].data = _validate_pandas(
                 datasets[dataset_name].components, data, dataset_name
             )
@@ -229,14 +279,17 @@ def load_datasets_with_data(data_structures: Any, datapoints: Optional[Any] = No
                 datasets[dataset_name].data = pd.DataFrame(
                     columns=list(datasets[dataset_name].components.keys())
                 )
-        return datasets, None
+        _handle_scalars_values(scalars, scalar_values)
+        return datasets, scalars, None
     # Handling dictionary of paths
     dict_datapoints = _load_datapoints_path(datapoints)
     for dataset_name, _ in dict_datapoints.items():
         if dataset_name not in datasets:
-            raise Exception(f"Not found dataset {dataset_name}")
+            raise Exception(f"Not found dataset {dataset_name} in datastructures.")
 
-    return datasets, dict_datapoints
+    _handle_scalars_values(scalars, scalar_values)
+
+    return datasets, scalars, dict_datapoints
 
 
 def load_vtl(input: Union[str, Path]) -> str:
@@ -345,8 +398,8 @@ def load_external_routines(input: Union[Dict[str, Any], Path, str]) -> Any:
 
 
 def _return_only_persistent_datasets(
-    datasets: Dict[str, Dataset], ast: Start
-) -> Dict[str, Dataset]:
+    datasets: Dict[str, Union[Dataset, Scalar]], ast: Start
+) -> Dict[str, Union[Dataset, Scalar]]:
     """
     Returns only the datasets with a persistent assignment.
     """
@@ -393,3 +446,209 @@ def _check_output_folder(output_folder: Union[str, Path]) -> None:
         if output_folder.suffix != "":
             raise ValueError("Output folder must be a Path or S3 URI to a directory")
         os.mkdir(output_folder)
+
+
+def to_vtl_json(dsd: Union[DataStructureDefinition, Schema], dataset_name: str) -> Dict[str, Any]:
+    """
+    Converts a pysdmx `DataStructureDefinition` or `Schema` into a VTL-compatible JSON
+    representation.
+
+    This function extracts and transforms the components (dimensions, measures, and attributes)
+    from the given SDMX data structure and maps them into a dictionary format that conforms
+    to the expected VTL data structure json schema.
+
+    Args:
+        dsd: An instance of `DataStructureDefinition` or `Schema` from the `pysdmx` model.
+        dataset_name: The name of the resulting VTL dataset.
+
+    Returns:
+            A dictionary representing the dataset in VTL format, with keys for dataset name and its
+            components, including their name, role, data type, and nullability.
+    """
+    components = []
+    NAME = "name"
+    ROLE = "role"
+    TYPE = "type"
+    NULLABLE = "nullable"
+
+    _components: List[SDMXComponent] = []
+    _components.extend(dsd.components.dimensions)
+    _components.extend(dsd.components.measures)
+    _components.extend(dsd.components.attributes)
+
+    for c in _components:
+        _type = VTL_DTYPES_MAPPING[c.dtype]
+        _nullability = c.role != SDMX_Role.DIMENSION
+        _role = VTL_ROLE_MAPPING[c.role]
+
+        component = {
+            NAME: c.id,
+            ROLE: _role,
+            TYPE: _type,
+            NULLABLE: _nullability,
+        }
+
+        components.append(component)
+
+    result = {"datasets": [{"name": dataset_name, "DataStructure": components}]}
+
+    return result
+
+
+def __generate_transformation(
+    child: Union[Assignment, PersistentAssignment], is_persistent: bool, count: int
+) -> Transformation:
+    expression = ASTString().render(ast=child.right)
+    result = child.left.value  # type: ignore[attr-defined]
+    return Transformation(
+        id=f"T{count}",
+        expression=expression,
+        is_persistent=is_persistent,
+        result=result,
+        name=f"Transformation {result}",
+    )
+
+
+def __generate_udo(child: Operator, count: int) -> UserDefinedOperator:
+    operator_definition = ASTString().render(ast=child)
+    return UserDefinedOperator(
+        id=f"UDO{count}",
+        operator_definition=operator_definition,
+        name=f"UserDefinedOperator {child.op}",
+    )
+
+
+def __generate_ruleset(child: Union[DPRuleset, HRuleset], count: int) -> Ruleset:
+    ruleset_definition = ASTString().render(ast=child)
+    ruleset_type: Literal["datapoint", "hierarchical"] = (
+        "datapoint" if isinstance(child, DPRuleset) else "hierarchical"
+    )
+    ruleset_scope: Literal["variable", "valuedomain"] = (
+        "variable" if child.signature_type == "variable" else "valuedomain"
+    )
+    return Ruleset(
+        id=f"R{count}",
+        ruleset_definition=ruleset_definition,
+        ruleset_type=ruleset_type,
+        ruleset_scope=ruleset_scope,
+        name=f"{ruleset_type.capitalize()} ruleset {child.name}",
+    )
+
+
+def ast_to_sdmx(ast: AST.Start, agency_id: str, id: str, version: str) -> TransformationScheme:
+    """
+    Converts a vtl AST into an SDMX compatible `TransformationScheme` object, following
+    the pysdmx model.
+
+    This function iterates over the child nodes of the given AST and categorizes each into one of
+    the following types:
+    - `PersistentAssignment`: Represents a persistent transformation. These are added to the
+    transformation list with a persistence flag.
+    - `Assignment`: Represents a temporary (non-persistent) transformation. These are added to the
+    transformation list without the persistence flag
+    - `DPRuleset` or `HRuleset`: Represent validation rule sets.
+    These are collected and wrapped into a `RulesetScheme` object.
+    - `Operator`: Defines user-defined operators. These are collected
+    into a `UserDefinedOperatorScheme` object.
+
+    After parsing all AST elements:
+    - If any rulesets were found, a `RulesetScheme` is created and added to the references.
+    - If any user-defined operators were found, a `UserDefinedOperatorScheme` is created and added
+    to the references.
+    - A `TransformationScheme` object is constructed with all collected transformations and any
+    additional references.
+
+    Args:
+        ast: The root node of the vtl ast representing the set of
+        vtl expressions.
+        agency_id: The identifier of the agency defining the SDMX structure as a string.
+        id: The identifier of the transformation scheme as a string.
+        version: The version of the transformation scheme given as a string.
+
+    Returns:
+        TransformationScheme: A fully constructed transformation scheme that includes
+        transformations, and optionally rule sets and user-defined operator schemes,
+        suitable for SDMX.
+
+    """
+    list_transformation = []
+    list_udos = []
+    list_rulesets = []
+    count_transformation = 0
+    count_udo = 0
+    count_ruleset = 0
+
+    for child in ast.children:
+        if isinstance(child, PersistentAssignment):
+            count_transformation += 1
+            list_transformation.append(
+                __generate_transformation(
+                    child=child, is_persistent=True, count=count_transformation
+                )
+            )
+        elif isinstance(child, Assignment):
+            count_transformation += 1
+            list_transformation.append(
+                __generate_transformation(
+                    child=child, is_persistent=False, count=count_transformation
+                )
+            )
+        elif isinstance(child, (DPRuleset, HRuleset)):
+            count_ruleset += 1
+            list_rulesets.append(__generate_ruleset(child=child, count=count_ruleset))
+        elif isinstance(child, Operator):
+            count_udo += 1
+            list_udos.append(__generate_udo(child=child, count=count_udo))
+
+    references: Any = {}
+    if list_rulesets:
+        references["ruleset_schemes"] = [
+            RulesetScheme(
+                items=list_rulesets,
+                agency=agency_id,
+                id="RS1",
+                vtl_version="2.1",
+                version=version,
+                name=f"RulesetScheme {id}-RS",
+            )
+        ]
+    if list_udos:
+        references["user_defined_operator_schemes"] = [
+            UserDefinedOperatorScheme(
+                items=list_udos,
+                agency=agency_id,
+                id="UDS1",
+                vtl_version="2.1",
+                version=version,
+                name=f"UserDefinedOperatorScheme {id}-UDS",
+            )
+        ]
+
+    transformation_scheme = TransformationScheme(
+        items=list_transformation,
+        agency=agency_id,
+        id="TS1",
+        vtl_version="2.1",
+        version=version,
+        name=f"TransformationScheme {id}",
+        **references,
+    )
+
+    return transformation_scheme
+
+
+def _check_script(script: Union[str, TransformationScheme, Path]) -> str:
+    """
+    Check if the TransformationScheme object is valid to generate a vtl script.
+    """
+    if not isinstance(script, (str, TransformationScheme, Path)):
+        raise SemanticError("0-1-1-1", format_=type(script).__name__)
+    if isinstance(script, TransformationScheme):
+        from pysdmx.toolkit.vtl import (
+            generate_vtl_script,
+        )
+
+        vtl_script = generate_vtl_script(script, model_validation=True)
+        return vtl_script
+    else:
+        return str(script)