relatics-toolkit 1.0.0__tar.gz

relatics_toolkit-1.0.0/PKG-INFO

@@ -0,0 +1,100 @@
+Metadata-Version: 2.3
+Name: relatics-toolkit
+Version: 1.0.0
+Summary: A package that enables fast and easy ETL processing of a Relatics workspace.
+Requires-Dist: mkdocs>=1.6.1
+Requires-Dist: mkdocs-material>=9.7.7
+Requires-Dist: mkdocstrings[python]>=1.0.6
+Requires-Dist: pandas>=3.0.3
+Requires-Dist: pyarrow>=25.0.1
+Requires-Dist: pytest>=9.1.1
+Requires-Dist: pytest-mock>=3.15.1
+Requires-Python: >=3.14
+Description-Content-Type: text/markdown
+
+# Relatics Extractor
+[![Tests](https://github.com/nvvliet-croonwolterendros/relatics-extractor/actions/workflows/test.yml/badge.svg)](https://github.com/nvvliet-croonwolterendros/relatics-extractor/actions/workflows/test.yml)
+
+A Python package for extracting, parsing, and transforming data from Relatics (a requirements management tool) into database-ready pandas DataFrames.
+
+## Overview
+
+The Relatics Extractor is designed to streamline the process of extracting data from Relatics API endpoints and converting it into structured, normalized tables suitable for database storage. It handles complex relationships between entities, nested XML structures, and provides robust error handling throughout the extraction pipeline.
+
+## Key Features
+
+- **Authentication**: Implements OAuth 2.0 token-based authentication with Relatics API
+- **Data Extraction**: Handles API calls to retrieve elements and their relationships
+- **XML Parsing**: Parses deeply nested XML structures into pandas DataFrames
+- **Schema Validation**: Validates data against predefined schemas and normalizes tables
+- **Relationship Management**: Properly handles different relationship cardinalities:
+  - :1 (one-to-one) relations are embedded in element tables  
+  - :n (many-to-one) relations become link tables
+- **Multithreading Support**: Uses ThreadPoolExecutor for faster processing of multiple elements
+- **Error Handling**: Robust error handling with logging and failure tracking
+
+## Documentation
+
+Documentation can be found [here](https://nvvliet-croonwolterendros.github.io/relatics-extractor/)
+
+## Installation
+
+```bash
+# Install from source
+pip install .
+
+# Or install in development mode
+pip install -e .
+```
+
+## Usage
+
+### Basic Setup
+
+```python
+from relatics_extractor import Extractor
+
+# Initialize the extractor with your credentials
+extractor = Extractor(
+    client_id="your_client_id",
+    client_secret="your_client_secret", 
+    environment="your_environment"
+)
+```
+
+### Running ETL Pipeline
+
+```python
+# Run the extraction process for multiple workspaces
+tables = extractor.run_etl_fast(
+    workspaces=["workspace1", "workspace2"],
+    element_operation="GetElements",
+    datamodel_operation="GetDatamodel"
+)
+
+# Access extracted tables
+for table_name, df in tables.items():
+    print(f"Table: {table_name}")
+    print(df.head())
+```
+
+## Architecture
+
+The extractor follows a clear pipeline architecture:
+
+1. **Extraction**: Uses `RelaticsClient` to fetch data from API endpoints
+2. **Parsing**: Parses XML responses using `parse_xml` function  
+3. **Validation**: Validates schemas using `validator.py`
+4. **Transformation**: Transforms data using `transformer.py` to normalize relationships
+
+## Key Components
+
+- **Extractor**: Main class that orchestrates the full ETL pipeline
+- **RelaticsClient**: Handles OAuth 2.0 authentication and API requests  
+- **parse_xml**: Parses deeply nested XML structures
+- **transformer**: Transforms raw data into structured tables with proper relationship handling
+- **validator**: Validates schema compliance and normalizes table structures
+
+## License
+
+MIT

relatics_toolkit-1.0.0/README.md

@@ -0,0 +1,86 @@
+# Relatics Extractor
+[![Tests](https://github.com/nvvliet-croonwolterendros/relatics-extractor/actions/workflows/test.yml/badge.svg)](https://github.com/nvvliet-croonwolterendros/relatics-extractor/actions/workflows/test.yml)
+
+A Python package for extracting, parsing, and transforming data from Relatics (a requirements management tool) into database-ready pandas DataFrames.
+
+## Overview
+
+The Relatics Extractor is designed to streamline the process of extracting data from Relatics API endpoints and converting it into structured, normalized tables suitable for database storage. It handles complex relationships between entities, nested XML structures, and provides robust error handling throughout the extraction pipeline.
+
+## Key Features
+
+- **Authentication**: Implements OAuth 2.0 token-based authentication with Relatics API
+- **Data Extraction**: Handles API calls to retrieve elements and their relationships
+- **XML Parsing**: Parses deeply nested XML structures into pandas DataFrames
+- **Schema Validation**: Validates data against predefined schemas and normalizes tables
+- **Relationship Management**: Properly handles different relationship cardinalities:
+  - :1 (one-to-one) relations are embedded in element tables  
+  - :n (many-to-one) relations become link tables
+- **Multithreading Support**: Uses ThreadPoolExecutor for faster processing of multiple elements
+- **Error Handling**: Robust error handling with logging and failure tracking
+
+## Documentation
+
+Documentation can be found [here](https://nvvliet-croonwolterendros.github.io/relatics-extractor/)
+
+## Installation
+
+```bash
+# Install from source
+pip install .
+
+# Or install in development mode
+pip install -e .
+```
+
+## Usage
+
+### Basic Setup
+
+```python
+from relatics_extractor import Extractor
+
+# Initialize the extractor with your credentials
+extractor = Extractor(
+    client_id="your_client_id",
+    client_secret="your_client_secret", 
+    environment="your_environment"
+)
+```
+
+### Running ETL Pipeline
+
+```python
+# Run the extraction process for multiple workspaces
+tables = extractor.run_etl_fast(
+    workspaces=["workspace1", "workspace2"],
+    element_operation="GetElements",
+    datamodel_operation="GetDatamodel"
+)
+
+# Access extracted tables
+for table_name, df in tables.items():
+    print(f"Table: {table_name}")
+    print(df.head())
+```
+
+## Architecture
+
+The extractor follows a clear pipeline architecture:
+
+1. **Extraction**: Uses `RelaticsClient` to fetch data from API endpoints
+2. **Parsing**: Parses XML responses using `parse_xml` function  
+3. **Validation**: Validates schemas using `validator.py`
+4. **Transformation**: Transforms data using `transformer.py` to normalize relationships
+
+## Key Components
+
+- **Extractor**: Main class that orchestrates the full ETL pipeline
+- **RelaticsClient**: Handles OAuth 2.0 authentication and API requests  
+- **parse_xml**: Parses deeply nested XML structures
+- **transformer**: Transforms raw data into structured tables with proper relationship handling
+- **validator**: Validates schema compliance and normalizes table structures
+
+## License
+
+MIT

relatics_toolkit-1.0.0/pyproject.toml

@@ -0,0 +1,46 @@
+[project]
+name = "relatics-toolkit"
+version = "1.0.0"
+description = "A package that enables fast and easy ETL processing of a Relatics workspace."
+readme = "README.md"
+requires-python = ">=3.14"
+dependencies = [
+    "mkdocs>=1.6.1",
+    "mkdocs-material>=9.7.7",
+    "mkdocstrings[python]>=1.0.6",
+    "pandas>=3.0.3",
+    "pyarrow>=25.0.1",
+    "pytest>=9.1.1",
+    "pytest-mock>=3.15.1",
+]
+
+[dependency-groups]
+dev = ["ruff>=0.16.6"]
+
+[tool.pytest.ini_options]
+pythonpath = ["."]
+
+[tool.ruff]
+line-length = 88
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "F",
+    "I",
+    "UP",
+    "B",
+]
+
+[tool.ruff.format]
+quote-style = "double"
+
+[[tool.uv.index]]
+name = "testpypi"
+url = "https://test.pypi.org/simple/"
+publish-url = "https://test.pypi.org/legacy/"
+explicit = true
+
+[build-system]
+requires = ["uv_build>=0.12.5,<0.13"]
+build-backend = "uv_build"

relatics_toolkit-1.0.0/pyproject.toml.orig

@@ -0,0 +1,42 @@
+[project]
+name = "relatics-toolkit"
+version = "1.0.0"
+description = "A package that enables fast and easy ETL processing of a Relatics workspace."
+readme = "README.md"
+requires-python = ">=3.14"
+dependencies = [
+    "mkdocs>=1.6.1",
+    "mkdocs-material>=9.7.7",
+    "mkdocstrings[python]>=1.0.6",
+    "pandas>=3.0.3",
+    "pyarrow>=25.0.1",
+    "pytest>=9.1.1",
+    "pytest-mock>=3.15.1",
+]
+
+[dependency-groups]
+dev = [
+    "ruff>=0.16.6",
+]
+
+[tool.pytest.ini_options]
+pythonpath = ["."]
+
+[tool.ruff]
+line-length = 88
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B"]
+
+[tool.ruff.format]
+quote-style = "double"
+
+[[tool.uv.index]]
+name = "testpypi"
+url = "https://test.pypi.org/simple/"
+publish-url = "https://test.pypi.org/legacy/"
+explicit = true
+
+[build-system]
+requires = ["uv_build>=0.12.5,<0.13"]
+build-backend = "uv_build"

relatics_toolkit-1.0.0/src/relatics_toolkit/__init__.py

@@ -0,0 +1,5 @@
+from relatics_toolkit.extraction_service import extract_element_tables
+from relatics_toolkit.ingestion.relatics_client import RelaticsClient
+from relatics_toolkit.ingestion.xml_parser import parse_xml
+
+__all__ = ["RelaticsClient", "extract_element_tables", "parse_xml"]

relatics_toolkit-1.0.0/src/relatics_toolkit/extraction_service.py

@@ -0,0 +1,166 @@
+import logging
+from concurrent.futures import ThreadPoolExecutor, as_completed
+
+import pandas as pd
+
+from relatics_toolkit.ingestion.relatics_client import RelaticsClient
+from relatics_toolkit.ingestion.xml_parser import parse_xml
+from relatics_toolkit.processing.schema import SCHEMA
+from relatics_toolkit.processing.transformer import create_element_tables
+from relatics_toolkit.processing.validator import normalize_tables, validate_schema
+
+logger = logging.getLogger(__name__)
+
+
+def extract_element_tables(
+    client: RelaticsClient,
+    workspace_elements: dict[str, list[str]],
+    operation: str,
+    parallel: bool = False,
+    max_workers: int | None = None,
+    inline_relations: list[str] | None = None,
+) -> dict[str, pd.DataFrame]:
+    """
+    Extracts and transforms Relatics elements into normalized tables.
+
+    For each configured workspace and element combination, retrieves the
+    corresponding Relatics XML payload, validates the extracted schema,
+    normalizes the resulting tables, and applies business transformations.
+
+    Processing can be executed sequentially or in parallel.
+
+    Args:
+        client: Configured Relatics API client.
+        workspace_elements: Mapping of workspace IDs to lists of element IDs
+            that should be extracted.
+        operation: Relatics operation name used to retrieve the
+            element data.
+        parallel: Whether element extraction should be executed in
+            parallel.
+        max_workers: Maximum number of worker threads used when
+            run_parallel is True. If None, the ThreadPoolExecutor
+            default is used.
+        inline_relations: Relation names of Relations to R2 Elements
+            whose values should be materialized directly in the
+            resulting element tables (must be to-one relations).
+
+    Returns:
+        Dictionary mapping table names to transformed pandas DataFrames.
+
+    Raises:
+        Exception: Any exception raised during retrieval, validation,
+            normalization, or transformation of element data.
+    """
+    jobs = [
+        (workspace_id, element_id)
+        for workspace_id, element_ids in workspace_elements.items()
+        for element_id in element_ids
+    ]
+
+    logger.info(
+        "Starting extraction for %s elements (parallel=%s)",
+        len(jobs),
+        parallel,
+    )
+
+    tables: dict[str, pd.DataFrame] = {}
+
+    if parallel:
+        with ThreadPoolExecutor(max_workers=max_workers) as executor:
+            future_map = {
+                executor.submit(
+                    _process_element,
+                    element_id=element_id,
+                    client=client,
+                    workspace_id=workspace_id,
+                    operation=operation,
+                    inline_relations=inline_relations,
+                ): (workspace_id, element_id)
+                for workspace_id, element_id in jobs
+            }
+
+            for future in as_completed(future_map):
+                workspace_id, element_id = future_map[future]
+
+                try:
+                    _merge_tables(tables, future.result())
+                except Exception:
+                    logger.exception(
+                        "Failed processing workspace_id=%s element_id=%s",
+                        workspace_id,
+                        element_id,
+                    )
+                    raise
+    else:
+        for workspace_id, element_id in jobs:
+            try:
+                _merge_tables(
+                    tables,
+                    _process_element(
+                        element_id=element_id,
+                        client=client,
+                        workspace_id=workspace_id,
+                        operation=operation,
+                        inline_relations=inline_relations,
+                    ),
+                )
+            except Exception:
+                logger.exception(
+                    "Failed processing workspace_id=%s element_id=%s",
+                    workspace_id,
+                    element_id,
+                )
+                raise
+
+    logger.info(
+        "Extraction completed successfully. Generated %s tables.",
+        len(tables),
+    )
+
+    return tables
+
+
+def _process_element(
+    element_id: str,
+    client: RelaticsClient,
+    workspace_id: str,
+    operation: str,
+    schema: dict[str, dict] = SCHEMA,
+    inline_relations: list[str] | None = None,
+) -> dict[str, pd.DataFrame]:
+    """
+    Processes a Relatics element with its first order relations and properties.
+    """
+    parameters = {"ConfigurationOfRef": element_id}
+
+    element_data = client.get_request(
+        workspace_id=workspace_id,
+        operation=operation,
+        parameters=parameters,
+    )
+
+    tables = {table_name: parse_xml(element_data, table_name) for table_name in schema}
+    normalized_tables = normalize_tables(tables=tables, schema=schema)
+    validate_schema(tables=normalized_tables, schema=schema)
+    transformed_tables = create_element_tables(
+        tables=normalized_tables, inline_relations=inline_relations
+    )
+
+    for table in transformed_tables.values():
+        table["workspace_id"] = workspace_id
+
+    return transformed_tables
+
+
+def _merge_tables(
+    tables: dict[str, pd.DataFrame],
+    element_tables: dict[str, pd.DataFrame],
+) -> None:
+    for table_name, df in element_tables.items():
+        if table_name in tables:
+            tables[table_name] = pd.concat(
+                [tables[table_name], df],
+                ignore_index=True,
+            )
+        else:
+            tables[table_name] = df

relatics_toolkit-1.0.0/src/relatics_toolkit/ingestion/relatics_client.py

@@ -0,0 +1,104 @@
+import requests
+import xml.etree.ElementTree as ET
+from typing import Dict, Tuple
+import logging
+
+logger = logging.getLogger(__name__)
+
+class TokenRequestError(Exception):
+    """Raised when token retrieval fails."""
+
+class APIRequestError(Exception):
+    """Raised when the API call fails."""
+
+class XMLParseError(Exception):
+    """Raised when XML parsing fails."""
+
+class RelaticsClient:
+    """Client for making OAuth2 get requests to relatics.
+    
+    Client for making requests to relatics webservices. Will automatically request a token and subsequently do a get request to a webservice.
+
+    Args:
+        client_id (str): the OAUTH client id obtained from the Relatics environment studio.
+        client_secret (str): the OAUTH client secret obtained from the Relatics environment studio.
+        environment (str): The subdomain of relaticonline. in https://example.relaticsonline.com, example is the environment string.
+
+    Returns:
+        RelaticsClient (RelaticsClient): Class capable of doing get requests to webservices in the specified environment.
+    """
+    
+    def __init__(self, client_id: str, client_secret: str, environment: str) -> None:
+        self.client_id = client_id
+        self.client_secret = client_secret
+        self.environment = environment
+    
+    def get_request(self, workspace_id: str, operation: str, parameters: Dict[str, str] = {}) -> ET.Element:
+        """
+        Executes a GET request to the Relatics DataExchange API.
+
+        Args:
+            workspace_id: Workspace identifier
+            operation: API operation name
+            parameters: Query parameters
+
+        Returns:
+            result: Parsed XML root element
+        """
+        api_endpoint = f"https://{self.environment}.relaticsonline.com/DataExchange/{workspace_id}/{operation}"
+        
+        access_token, token_type = self._get_token()
+        
+        headers = {
+            "Authorization": f"{token_type} {access_token}",
+            "Content-Type": "application/json",
+        }
+        
+        body = {
+            "Parameters": parameters
+        }
+        
+        try:
+            api_response = requests.get(api_endpoint, headers=headers, json=body)
+        except requests.RequestException as e:
+            raise APIRequestError(f"API Request failed: {str(e)}") from e
+        
+        if api_response.status_code != 200:
+            raise APIRequestError(f"API request failed with status code {api_response.status_code}: {api_response.text}")
+                
+        try:
+            return ET.fromstring(api_response.content)
+        except ET.ParseError as e:
+            raise XMLParseError("Invalid XML in API response") from e
+    
+    def _get_token(self) -> Tuple[str,str]:
+        """Retrieves OAuth2 token."""
+        
+        token_endpoint = f"https://{self.environment}.relaticsonline.com/oauth2/token"
+        
+        headers = {
+            "Content-Type": "application/x-www-form-urlencoded"
+        }
+
+        data = {
+            "grant_type": "client_credentials",
+            "client_id": self.client_id,
+            "client_secret": self.client_secret
+        }
+        
+        try:
+            token_response = requests.post(token_endpoint, data=data, headers=headers)
+        except requests.RequestException as e:
+            raise TokenRequestError("Error while requesting token") from e
+            
+        if token_response.status_code != 200:
+            raise TokenRequestError(f"Token request failed with status code {token_response.status_code}: {token_response.text}")
+
+        token_json = token_response.json()
+        access_token = token_json.get('access_token')
+        token_type = token_json.get('token_type')
+
+        if not access_token or not token_type:
+            raise TokenRequestError("Missing access_token or token_type in response")
+        
+        return access_token, token_type 

relatics_toolkit-1.0.0/src/relatics_toolkit/ingestion/xml_parser.py

@@ -0,0 +1,99 @@
+from collections import defaultdict
+import pandas as pd
+import itertools
+import xml.etree.ElementTree as ET
+
+def parse_xml(root:ET.Element, report_part:str) -> pd.DataFrame:
+    """Function to convert a relatics report part into a pandas DataFrame.
+
+    Relatics report parts can be deeply nested, this function recusively unpacks the XML and returns a single DataFrame.
+
+    Args:
+        root (xml.etree.ElementTree): The complete xml.etree.ElementTree xml as obtained from the relatics webservice. Can be easily obtained from the RelaticsClient.
+        report_part: Specific report part to unpack into a pandas dataframe.
+
+    Returns:
+        df: An unpacked pandas DataFrame of a specific report part.
+
+    """
+    start_element = root.find(report_part)
+    
+    nested_rows = []
+    
+    if start_element is not None:
+        nested_rows.append(_parse_xml_to_dict(start_element=start_element))
+        
+    unpacked_rows = []
+    
+    for row in nested_rows:
+        unpacked_rows.extend(_recursive_unpack(row))
+    
+    df = pd.DataFrame(unpacked_rows)
+    
+    df.columns = [col.split('.')[-1] for col in df.columns]
+
+    return df
+
+def _parse_xml_to_dict(start_element):
+    dictionary = defaultdict(list)
+
+    # Collect all direct children of the current element
+    for child_element in start_element:
+        dictionary[child_element.tag].append(child_element)
+
+    # Process collected elements
+    for child_tag in list(dictionary.keys()):
+        unpacked_element_list = []
+        for child_element in dictionary[child_tag]:
+            unpacked_element = child_element.attrib
+
+            # Recurse if the element has children
+            if list(child_element):
+                children_dict = _parse_xml_to_dict(child_element)
+                if children_dict:
+                    unpacked_element.update(children_dict)
+
+            # If we got something meaningful, keep it
+            if unpacked_element:
+                unpacked_element_list.append(unpacked_element)
+
+        dictionary[child_tag] = unpacked_element_list
+
+    return dict(dictionary)
+
+def _recursive_unpack(nested_row):
+    """Recursively unpack a nested dict with list-of-dict values into flat records."""
+    base_record = {}
+    unpacked_lists = []
+
+    for key, value in nested_row.items():
+        if isinstance(value, list) and value and all(isinstance(item, dict) for item in value):
+            # Recursively unpack each item in the list
+            new_list = []
+            for item in value:
+                flattened_items = _recursive_unpack(item)
+                for flat in flattened_items:
+                    # Add prefix to each key
+                    prefixed = {f"{key}.{k}": v for k, v in flat.items()}
+                    new_list.append(prefixed)
+            unpacked_lists.append(new_list)
+        elif isinstance(value, dict):
+            # Recursively flatten the nested dict
+            nested = _recursive_unpack(value)
+            for flat in nested:
+                base_record.update({f"{key}.{k}": v for k, v in flat.items()})
+        elif value:
+            base_record[key] = value
+
+    # If there are unpacked lists, compute cartesian product
+    if unpacked_lists:
+        result = []
+        for combo in itertools.product(*unpacked_lists):
+            combined = dict(base_record)  # copy base record
+            for item in combo:
+                combined.update(item)
+            result.append(combined)
+        return result
+    else:
+        return [base_record]
+    

relatics_toolkit-1.0.0/src/relatics_toolkit/processing/schema.py

@@ -0,0 +1,38 @@
+SCHEMA = {
+    "Element": {
+        "R1ElementID": {"not_null": True, "unique": True},
+        "R1Element": {"not_null": True, "unique": True},
+    },
+    "ElementInstances": {
+        "R1InstanceID": {"not_null": True, "unique": True},
+        "R1Instance": {"not_null": True, "unique": False},
+        "R1InstanceDescription": {"not_null": False, "unique": False, "default": None},
+        "R1InstanceRichText": {"not_null": False, "unique": False, "default": None},
+    },
+    "Properties": {
+        "Property": {"not_null": True, "unique": True},
+    },
+    "PropertyInstances": {
+        "R1InstanceID": {"not_null": True, "unique": False},
+        "Property": {"not_null": True, "unique": False},
+        "PropertyInstance": {"not_null": False, "unique": False},
+    },
+    "Relations": {
+        "RelationID": {"not_null": True, "unique": False},
+        "Relation": {"not_null": True, "unique": False},
+        "Cardinality": {"not_null": False, "unique": False},
+        "R1Element": {"not_null": True, "unique": False},
+        "R2ElementID": {"not_null": True, "unique": False},
+        "R2Element": {"not_null": True, "unique": False},
+        "ChildR2ElementID": {"not_null": False, "unique": False, "default": None},
+        "ChildR2Element": {"not_null": False, "unique": False, "default": None},
+    },
+    "RelationInstances": {
+        "RelationID": {"not_null": True, "unique": False},
+        "Cardinality": {"not_null": False, "unique": False},
+        "R1InstanceID": {"not_null": True, "unique": False},
+        "R2Element": {"not_null": True, "unique": False},
+        "R2InstanceID": {"not_null": True, "unique": False},
+        "R2Instance": {"not_null": True, "unique": False},
+    },
+}

relatics_toolkit-1.0.0/src/relatics_toolkit/processing/transformer.py

@@ -0,0 +1,420 @@
+import logging
+import re
+import unicodedata
+from typing import Literal
+
+import pandas as pd
+
+logger = logging.getLogger(__name__)
+
+R1ELEMENT_COL = "R1Element"
+
+R1INSTANCE_COL = "R1Instance"
+R1INSTANCEID_COL = "R1InstanceID"
+
+PROPERTY_COL = "Property"
+PROPERTYINSTANCE_COL = "PropertyInstance"
+
+RELATION_COL = "Relation"
+RELATIONID_COL = "RelationID"
+CARDINALITY_COL = "Cardinality"
+
+R2ELEMENT_COL = "R2Element"
+R2ELEMENTID_COL = "R2ElementID"
+
+CHILDR2ELEMENT_COL = "ChildR2Element"
+CHILDR2ELEMENTID_COL = "ChildR2ElementID"
+
+R2INSTANCE_COL = "R2Instance"
+R2INSTANCEID_COL = "R2InstanceID"
+
+DEFAULT_COLUMN_MAP = {
+    R1INSTANCEID_COL: "guid",
+    R1INSTANCE_COL: "naam",
+    "R1InstanceDescription": "description",
+    "R1InstanceRichText": "rich_text",
+}
+
+
+def create_element_tables(
+    tables: dict[str, pd.DataFrame],
+    column_map: dict[str, str] = DEFAULT_COLUMN_MAP,
+    inline_relations: list[str] | None = None,
+) -> dict[str, pd.DataFrame]:
+    """
+    Create the element table and associated link tables for an element.
+
+    The element table is built by joining:
+    - property values
+    - to-one relation references
+    - directly resolved relation values
+
+    Regular to-one relations are represented as <element>_guid
+    columns containing the related R2 instance identifier. Relations
+    listed in inline_relations are represented directly by their
+    related R2 instance value.
+
+    To-many relations are returned as separate link tables.
+
+    Args:
+        tables: Dict with the 6 raw Relatics report tables ("Element",
+            "ElementInstances", "Properties", "PropertyInstances",
+            "Relations", "RelationInstances").
+        column_map: Mapping used to rename the final element table's
+            columns.
+        inline_relations: Relation names of Relations to R2 Elements
+            whose values will be materialized directly in the
+            resulting element tables.
+
+    Returns:
+        Mapping of table names to DataFrames containing the element table
+        and any associated link tables.
+    """
+    element_df = tables["Element"].copy()
+    element_instances_df = tables["ElementInstances"].copy()
+    properties_df = tables["Properties"].copy()
+    property_instances_df = tables["PropertyInstances"].copy()
+    relations_df = tables["Relations"].copy()
+    relation_instances_df = tables["RelationInstances"].copy()
+
+    r1_element = _normalize_value(element_df[R1ELEMENT_COL][0])
+
+    relations_df, relation_instances_df = _transform_relations_table(
+        relations_df=relations_df, relation_instances_df=relation_instances_df
+    )
+    property_table = _create_property_table(
+        properties_df=properties_df, property_instances_df=property_instances_df
+    )
+    to_one_relations_table = _create_to_one_relations_table(
+        relations_df=relations_df,
+        relation_instances_df=relation_instances_df,
+        inline_relations=inline_relations,
+    )
+
+    sub_tables: list = [
+        df
+        for df in (
+            property_table,
+            to_one_relations_table,
+        )
+        if not df.columns.empty
+    ]
+
+    element_table = (
+        element_instances_df.set_index(R1INSTANCEID_COL)
+        .join(sub_tables, how="left")
+        .reset_index()
+    )
+    element_table = element_table.rename(columns=column_map)
+
+    link_tables = _create_link_tables(
+        r1_element=r1_element,
+        relations_df=relations_df,
+        relation_instances_df=relation_instances_df,
+    )
+
+    return {
+        f"raw_relatics__{r1_element}": element_table,
+        **link_tables,
+    }
+
+
+def _transform_relations_table(
+    relations_df: pd.DataFrame, relation_instances_df: pd.DataFrame
+) -> tuple[pd.DataFrame, pd.DataFrame]:
+    """
+    Normalize and disambiguate relation target element names.
+
+    - Coalesces child R2 element data into the primary R2 columns.
+    - Normalizes element and relation names.
+    - Validates that each Relation/R2Element combination is unique.
+    - Renames duplicate and self-referencing R2Elements to ensure unique
+        SQL-safe column names.
+    - Applies the same renaming to relation instances.
+    """
+    if (
+        CHILDR2ELEMENT_COL in relations_df.columns
+        and CHILDR2ELEMENTID_COL in relations_df.columns
+    ):
+        relations_df[R2ELEMENT_COL] = (
+            relations_df[CHILDR2ELEMENT_COL]
+            .replace("", None)
+            .combine_first(relations_df[R2ELEMENT_COL])
+        )
+        relations_df[R2ELEMENTID_COL] = (
+            relations_df[CHILDR2ELEMENTID_COL]
+            .replace("", None)
+            .combine_first(relations_df[R2ELEMENTID_COL])
+        )
+        relations_df = relations_df.drop(
+            [CHILDR2ELEMENT_COL, CHILDR2ELEMENTID_COL], axis=1
+        )
+
+    relations_df[[R1ELEMENT_COL, RELATION_COL, R2ELEMENT_COL]] = relations_df[
+        [R1ELEMENT_COL, RELATION_COL, R2ELEMENT_COL]
+    ].map(_normalize_value)
+    relation_instances_df[R2ELEMENT_COL] = relation_instances_df[R2ELEMENT_COL].apply(
+        _normalize_value
+    )
+
+    duplicates_mask = relations_df.duplicated(
+        subset=[RELATION_COL, R2ELEMENT_COL], keep=False
+    )
+    if duplicates_mask.any():
+        logger.error("Duplicate Relation/R2Element combinations found.")
+        logger.debug(relations_df[duplicates_mask].to_dict())
+        raise RuntimeError(
+            "relations_df cannot contain duplicated Relation + R2Element pairs."
+        )
+
+    rename_rows = relations_df.loc[
+        relations_df[R2ELEMENT_COL].duplicated(keep=False)
+        | (relations_df[R1ELEMENT_COL] == relations_df[R2ELEMENT_COL])
+    ].copy()
+
+    rename_rows[R2ELEMENT_COL] = (
+        rename_rows[RELATION_COL] + "_" + rename_rows[R2ELEMENT_COL]
+    )
+
+    rename_map = rename_rows.set_index(RELATIONID_COL)[R2ELEMENT_COL].to_dict()
+
+    relations_df[R2ELEMENT_COL] = (
+        relations_df[RELATIONID_COL].map(rename_map).fillna(relations_df[R2ELEMENT_COL])
+    )
+    relation_instances_df[R2ELEMENT_COL] = (
+        relation_instances_df[RELATIONID_COL]
+        .map(rename_map)
+        .fillna(relation_instances_df[R2ELEMENT_COL])
+    )
+
+    return relations_df, relation_instances_df
+
+
+def _create_property_table(
+    properties_df: pd.DataFrame, property_instances_df: pd.DataFrame
+) -> pd.DataFrame:
+    """
+    Create a property table indexed by R1 instance.
+
+    Property names are normalized and become columns.
+    A column is included for every property defined in properties_df,
+    even if no property instances exist.
+    """
+    property_instances_df = property_instances_df.copy()
+
+    property_columns = (
+        properties_df[PROPERTY_COL].apply(_normalize_value).unique().tolist()
+    )
+
+    property_instances_df[PROPERTY_COL] = property_instances_df[PROPERTY_COL].apply(
+        _normalize_value
+    )
+
+    try:
+        return (
+            property_instances_df.pivot(
+                index=R1INSTANCEID_COL,
+                columns=PROPERTY_COL,
+                values=PROPERTYINSTANCE_COL,
+            )
+            .reindex(columns=property_columns)
+            .rename_axis(columns=None)
+        )
+
+    except ValueError:
+        logger.exception(
+            "Failed to pivot property instances. "
+            "Normalization may have created duplicate property names."
+        )
+        raise
+
+
+def _create_to_one_relations_table(
+    relations_df: pd.DataFrame,
+    relation_instances_df: pd.DataFrame,
+    inline_relations: list[str] | None = None,
+) -> pd.DataFrame:
+    """
+    Create a table containing to-one relation data.
+
+    For regular to-one relations, each referenced element becomes a
+    <element>_guid column containing the related R2 instance
+    id.
+
+    Relations listed in inline_relations are represented by their
+    resolved value instead of an R2 instance id.
+    """
+    inline_relations = inline_relations or []
+    if inline_relations:
+        inline_relations = [_normalize_value(value) for value in inline_relations]
+
+    all_to_one_relations_df = _filter_cardinality(
+        df=relations_df,
+        cardinality="one",
+    )
+
+    to_one_relations_df = all_to_one_relations_df[
+        ~all_to_one_relations_df[RELATION_COL].isin(inline_relations)
+    ].copy()
+
+    to_one_relation_columns = [
+        f"{element}_guid" for element in to_one_relations_df[R2ELEMENT_COL]
+    ]
+
+    to_one_relation_instances_df = relation_instances_df[
+        relation_instances_df[RELATIONID_COL].isin(to_one_relations_df[RELATIONID_COL])
+    ].copy()
+
+    to_one_relation_instances_df[R2ELEMENT_COL] = (
+        to_one_relation_instances_df[R2ELEMENT_COL].astype("string") + "_guid"
+    )
+
+    to_one_relations_table = (
+        to_one_relation_instances_df.pivot(
+            index=R1INSTANCEID_COL,
+            columns=R2ELEMENT_COL,
+            values=R2INSTANCEID_COL,
+        )
+        .reindex(columns=to_one_relation_columns)
+        .rename_axis(columns=None)
+    )
+
+    if not inline_relations:
+        return to_one_relations_table
+
+    inline_relations_df = all_to_one_relations_df[
+        all_to_one_relations_df[RELATION_COL].isin(inline_relations)
+    ].copy()
+
+    inline_relations_table = _create_inline_relations_table(
+        inline_relations_df=inline_relations_df,
+        relation_instances_df=relation_instances_df,
+    )
+
+    return pd.concat(
+        [inline_relations_table, to_one_relations_table],
+        axis=1,
+    )
+
+
+def _create_inline_relations_table(
+    inline_relations_df: pd.DataFrame,
+    relation_instances_df: pd.DataFrame,
+) -> pd.DataFrame:
+    """
+    Create a table containing directly resolved property values.
+
+    Each referenced property element becomes a column containing the
+    related R2 instance value.
+    """
+    inline_relation_instances_df = relation_instances_df[
+        relation_instances_df[RELATIONID_COL].isin(inline_relations_df[RELATIONID_COL])
+    ].copy()
+
+    inline_relation_columns = inline_relations_df[R2ELEMENT_COL].tolist()
+
+    return (
+        inline_relation_instances_df.pivot(
+            index=R1INSTANCEID_COL,
+            columns=R2ELEMENT_COL,
+            values=R2INSTANCE_COL,
+        )
+        .reindex(columns=inline_relation_columns)
+        .rename_axis(columns=None)
+    )
+
+
+def _create_link_tables(
+    r1_element: str, relations_df: pd.DataFrame, relation_instances_df: pd.DataFrame
+) -> dict[str, pd.DataFrame]:
+    """
+    Create link tables for all to-many relations.
+
+    Each R2Element receives its own link table containing the R1 and R2
+    instance identifiers. Table and column names are normalized to make
+    them SQL safe.
+    """
+    to_many_relations_df = _filter_cardinality(relations_df, "many")
+    to_many_relation_instances_df = _filter_cardinality(relation_instances_df, "many")
+
+    link_tables: dict[str, pd.DataFrame] = {}
+
+    for r2_element in to_many_relations_df[R2ELEMENT_COL]:
+        table_name = f"raw_relatics__{r1_element}_{r2_element}"
+
+        mask = to_many_relation_instances_df[R2ELEMENT_COL] == str(r2_element)
+
+        link_table = to_many_relation_instances_df.loc[
+            mask, [R1INSTANCEID_COL, R2INSTANCEID_COL]
+        ].reset_index(drop=True)
+
+        link_tables[table_name] = link_table.rename(
+            columns={
+                R1INSTANCEID_COL: f"{r1_element}_guid",
+                R2INSTANCEID_COL: f"{r2_element}_guid",
+            }
+        )
+
+    return link_tables
+
+
+def _filter_cardinality(
+    df: pd.DataFrame,
+    cardinality: Literal["many", "one"],
+) -> pd.DataFrame:
+    """
+    Filter rows by relation cardinality.
+
+    Cardinality values are normalized to either ':1' or ':n'
+    before filtering.
+    """
+    cardinality_mask = df[CARDINALITY_COL].map(
+        lambda x: ":n" if "n" in str(x).split(":")[-1] else ":1"
+    )
+
+    if cardinality == "many":
+        return df[cardinality_mask == ":n"]
+
+    if cardinality == "one":
+        return df[cardinality_mask == ":1"]
+
+    raise ValueError(f"Invalid cardinality '{cardinality}'. Expected 'one' or 'many'.")
+
+
+def _normalize_value(val: str, max_length: int = 63) -> str:
+    """
+    Normalize text for use as SQL table and column names.
+    """
+    if pd.isna(val) or val is None:
+        return ""
+
+    val = str(val)
+
+    # Replace special characters
+    val = val.replace("&", "_en_").replace("€", "_euro_").replace("+", "_plus_")
+
+    # Unicode -> ASCII
+    val = unicodedata.normalize("NFKD", val)
+    val = val.encode("ascii", "ignore").decode("ascii")
+
+    # Lowercase
+    val = val.lower()
+
+    # Replace invalid characters
+    val = re.sub(r"[^a-z0-9_]", "_", val)
+
+    # Collapse underscores
+    val = re.sub(r"_+", "_", val)
+
+    # Strip leading/trailing underscores
+    val = val.strip("_")
+
+    if not val:
+        return ""
+
+    # Prevent leading digit
+    if val[0].isdigit():
+        val = f"no_num_{val}"
+
+    # Trim to PostgreSQL identifier length
+    return val[:max_length]

relatics_toolkit-1.0.0/src/relatics_toolkit/processing/validator.py

@@ -0,0 +1,120 @@
+from collections import defaultdict
+
+import pandas as pd
+
+from relatics_toolkit.processing.schema import SCHEMA
+
+
+def normalize_tables(
+    tables: dict[str, pd.DataFrame], schema: dict[str, dict[str, dict]] = SCHEMA
+) -> dict[str, pd.DataFrame]:
+    """
+    Add missing optional columns to tables according to the schema.
+
+    Columns marked 'not_null=False' are added when missing and
+    initialized with their configured default value. The resulting
+    tables are validated against the schema before being returned.
+    """
+
+    normalized_tables: dict[str, pd.DataFrame] = {}
+
+    for table_name, table_schema in schema.items():
+        table = tables.get(table_name)
+
+        if table is not None:
+            df = table.copy()
+
+            for column_name, column_rules in table_schema.items():
+                if column_name not in df.columns:
+                    df[column_name] = column_rules.get("default")
+
+            df = df.dropna(how="all").reset_index(drop=True)
+
+            normalized_tables[table_name] = df
+
+    return normalized_tables
+
+
+def validate_schema(
+    tables: dict[str, pd.DataFrame],
+    schema: dict[str, dict[str, dict]],
+) -> None:
+    """
+    Validate tables against the configured schema.
+
+    Checks:
+    - Required tables exist.
+    - Required columns exist.
+    - Columns marked 'not_null' contain no null values.
+    - Columns marked 'unique' contain no duplicate non-null values.
+    """
+    errors = defaultdict(list)
+
+    for table_name, table_schema in schema.items():
+        if table_name not in tables:
+            errors["missing_tables"].append({"table": table_name})
+            continue
+
+        df = tables[table_name]
+
+        missing_columns = [
+            column for column in table_schema if column not in df.columns
+        ]
+
+        if missing_columns:
+            errors["missing_columns"].append(
+                {
+                    "table": table_name,
+                    "columns": missing_columns,
+                }
+            )
+            continue
+
+        null_columns = [
+            column
+            for column, column_rules in table_schema.items()
+            if column_rules.get("not_null") and df[column].isna().any()
+        ]
+
+        if null_columns:
+            errors["null_values"].append(
+                {
+                    "table": table_name,
+                    "columns": null_columns,
+                }
+            )
+
+        duplicate_columns = [
+            column
+            for column, column_rules in table_schema.items()
+            if column_rules.get("unique") and df[column].dropna().duplicated().any()
+        ]
+
+        if duplicate_columns:
+            errors["duplicate_values"].append(
+                {
+                    "table": table_name,
+                    "columns": duplicate_columns,
+                }
+            )
+
+    if errors:
+        raise RuntimeError(format_validation_errors(errors))
+
+
+def format_validation_errors(errors: dict[str, list[dict]]) -> str:
+    sections = []
+
+    for category, items in errors.items():
+        lines = []
+
+        for item in items:
+            if category == "missing_tables":
+                lines.append(item["table"])
+
+            else:
+                lines.append(f"{item['table']}: {', '.join(item['columns'])}")
+
+        sections.append(f"{category}:\n- " + "\n- ".join(lines))
+
+    return "Schema validation failed.\n\n" + "\n\n".join(sections)