relationalai 0.13.4__tar.gz → 1.0.0a1__tar.gz

relationalai-1.0.0a1/PKG-INFO

@@ -0,0 +1,44 @@
+Metadata-Version: 2.4
+Name: relationalai
+Version: 1.0.0a1
+Summary: RelationalAI Library and CLI
+Author-email: RelationalAI <support@relational.ai>
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: lqp==0.1.19
+Requires-Dist: snowflake-connector-python[secure-local-storage]
+Requires-Dist: snowflake-snowpark-python>=1.38.0
+Requires-Dist: typing-extensions
+Requires-Dist: rich
+Requires-Dist: nicegui>3.0.0
+Requires-Dist: numpy<2
+Requires-Dist: pandas
+Requires-Dist: colorama
+Requires-Dist: inquirerpy
+Requires-Dist: click==8.2.1
+Requires-Dist: gravis
+Requires-Dist: toml
+Requires-Dist: tomlkit
+Requires-Dist: requests
+Requires-Dist: pyarrow
+Requires-Dist: websockets
+Requires-Dist: aiohttp
+Requires-Dist: packaging
+Requires-Dist: opentelemetry-sdk
+Requires-Dist: opentelemetry-api
+Requires-Dist: opentelemetry-exporter-otlp-proto-http
+Requires-Dist: wcwidth
+Requires-Dist: cryptography
+Requires-Dist: pyjwt
+Requires-Dist: more-itertools
+Requires-Dist: sqlglot
+Requires-Dist: pyyaml
+Requires-Dist: pydantic<2.12.0,>=2.0.0
+Requires-Dist: pydantic-settings<2.11.0,>=2.0.0
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: pytest-mock; extra == "dev"
+Requires-Dist: pytest-snapshot; extra == "dev"
+Requires-Dist: pytest-timeout; extra == "dev"
+Requires-Dist: ruff; extra == "dev"

relationalai-1.0.0a1/README.md

@@ -0,0 +1,160 @@
+# PyRel
+
+This repository contains the source code for PyRel, the Python library for RelationalAI.
+
+## Documentation
+
+The documentation for PyRel can be found at https://private.relational.ai/early-access/pyrel.
+
+## Development
+
+### Local
+
+```bash
+make install
+make sync
+```
+
+### Docstrings
+
+API reference documentation is generated from docstrings in the code.
+Please ensure that all public modules, attributes, functions, and classes, and class methods have docstrings formatted according to the [NumPy docstring standard](https://numpydoc.readthedocs.io/en/latest/format.html).
+
+#### Docstring Templates
+
+To help maintain consistency in docstring formatting, you can use the following templates for common constructs:
+
+**Modules:**
+
+Add a short description of the module's purpose to the top of each module file above the imports, and include the `__include_in_docs__` attribute set to `True`.
+
+```python
+"""
+The `<module_name>` module provides `<brief_description>`.
+"""
+import ...
+
+__include_in_docs__ = True
+```
+
+**Module Attributes:**
+
+Add docstrings to module-level attributes directly below their definitions:
+
+```python
+THE_MEANING: int = 42
+"""The meaning of life, the universe, and everything."""
+```
+
+**Functions:**
+
+Import the `include_in_docs` decorator from `util.docutils` and apply it to the public function.
+Then add a docstring with a brief description, parameters, return value, and one or two usage examples.
+
+> [!IMPORTANT]
+> Functions must be fully typed with type hints for all public parameters and return values.
+
+```python
+from util.docutils import include_in_docs
+
+@include_in_docs
+def add(a: int, b: int) -> int:
+    """Add two integers.
+
+    Parameters
+    ----------
+    a : int
+        The first integer to add.
+    b : int
+        The second integer to add.
+
+    Returns
+    -------
+    int
+        The sum of the two integers.
+
+    Examples
+    --------
+
+    >>> add(2, 3)
+    5
+    >>> add(-1, 1)
+    0
+    """
+    return a + b
+```
+
+**Classes:**
+
+Import the `include_in_docs` decorator from `util.docutils` and apply it to the public class and its public methods.
+Add a docstring to the class with a brief description, parameters for the constructor, and one or two usage examples.
+Add docstrings to each public method with a brief description, parameters, return value, and one or two usage examples.
+
+> [!NOTE]
+> You do not need to add a docstring to the `__init__` method; include constructor parameters in the class docstring instead.
+
+> [!NOTE]
+> Methods are documented just like functions. You do not need to document the `self` parameter in methods.
+
+```python
+from util.docutils import include_in_docs
+
+@include_in_docs
+class Calculator:
+    """A simple calculator class for basic arithmetic operations.
+
+    Parameters
+    ----------
+    name : str
+        The name of the calculator.
+
+    Examples
+    --------
+
+    >>> calc = Calculator("MyCalc")
+    >>> calc.add(2, 3)
+    5
+    """
+    def __init__(self, name: str) -> None:
+        self.name = name
+
+    @include_in_docs
+    def add(self, a: int, b: int) -> int:
+        """Add two integers.
+
+        Parameters
+        ----------
+        a : int
+            The first integer to add.
+        b : int
+            The second integer to add.
+
+        Returns
+        -------
+        int
+            The sum of the two integers.
+
+        Examples
+        --------
+
+        >>> calc.add(2, 3)
+        5
+        >>> calc.add(-1, 1)
+        0
+        """
+        return a + b
+```
+
+#### Previewing Documentation Changes
+
+When you open a pull request that modifies docstrings, a preview of the updated documentation will be automatically built and deployed to a temporary URL.
+To view the preview, look for a comment from the "Docs Preview" GitHub Action in your pull request.
+The comment will contain a link to the preview site where you can review the changes.
+Open the link and navigate to the `/api/python/` section and use the left-hand sidebar to find the updated documentation for the modified modules, classes, functions, or attributes.
+
+The API docs are organized by module, so you can easily locate the relevant sections.
+For example, if you modified the `relationalai.semantics.std.string` module, the docstring will can be found at:
+
+```
+https://<preview-url>/api/python/semantics/std/strings
+```

relationalai-1.0.0a1/pyproject.toml

@@ -0,0 +1,70 @@
+[project]
+name = 'relationalai'
+version = '1.0.0a1'
+description = 'RelationalAI Library and CLI'
+readme="docs/pypi/README.md"
+authors = [
+    { name="RelationalAI", email="support@relational.ai" },
+]
+requires-python = ">= 3.10"
+dependencies = [
+    "lqp==0.1.19",
+    "snowflake-connector-python[secure-local-storage]",
+    "snowflake-snowpark-python>=1.38.0",
+    "typing-extensions",
+    "rich",
+    "nicegui>3.0.0",
+    "numpy<2",
+    "pandas",
+    "colorama",
+    "inquirerpy",
+    "click==8.2.1",
+    "gravis",
+    "toml",
+    "tomlkit",
+    "requests",
+    "pyarrow",
+    "websockets",
+    "aiohttp",
+    "packaging",
+    "opentelemetry-sdk",
+    "opentelemetry-api",
+    "opentelemetry-exporter-otlp-proto-http",
+    "wcwidth",
+    "cryptography",
+    "pyjwt",
+    "more-itertools",
+    "sqlglot",
+    "pyyaml",
+    # TODO: Add back when confocal is published to PyPI
+    # "confocal @ git+https://github.com/joshuafcole/confocal.git",
+    "pydantic>=2.0.0,<2.12.0",
+    "pydantic-settings>=2.0.0,<2.11.0",
+]
+[project.optional-dependencies]
+dev = [
+    "pytest",
+    "pytest-cov",
+    "pytest-mock",
+    "pytest-snapshot",
+    "pytest-timeout",
+    "ruff",
+]
+
+[project.scripts]
+rai = "v0.relationalai.tools.cli:cli"
+rai-dev = "relationalai.tools.dev:cli"
+
+[tool.ruff]
+extend-exclude = ["*.ipynb"]
+exclude = []
+
+[tool.pyright]
+include = ["src", "tests", "example"]
+exclude = ["**/node_modules", "**/__pycache__", "**/.*"]
+useLibraryCodeForTypes = true
+reportMissingImports = true
+reportMissingTypeStubs = false
+pythonVersion = "3.10"
+typeCheckingMode = "standard"
+reportUnusedExpression = false

relationalai-1.0.0a1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

relationalai-1.0.0a1/src/relationalai/__init__.py

@@ -0,0 +1,7 @@
+"""
+The RelationalAI Python SDK, also known as PyRel.
+"""
+
+import importlib.metadata
+
+__version__ = importlib.metadata.version(__package__ or __name__)

relationalai-1.0.0a1/src/relationalai/config/__init__.py

@@ -0,0 +1,56 @@
+"""
+RelationalAI Multi-Connection Configuration System
+
+This package contains the new Pydantic-based configuration system with:
+- Multi-connection support (Snowflake, DuckDB)
+- Multi-profile support (dev, prod, staging, etc.)
+- Discriminated unions for type-safe connection handling
+- Auto-loading from multiple sources (raiconfig.yaml, Snowflake connections, DBT profiles)
+- Environment variable templating support (YAML)
+- Profile override system (profile values override config defaults)
+- Type-safe validation via Pydantic
+"""
+
+from .config import Config, _Config, RAIConfig, ConfigFromDBT, ConfigFromSnowflake
+from .config_fields import EngineConfig, DataConfig, CompilerConfig, ModelConfig, ReasonerConfig, DebugConfig
+from .connections import (
+    ConnectionConfig,
+    SnowflakeConnection,
+    DuckDBConnection,
+    UsernamePasswordAuth,
+    UsernamePasswordMFAAuth,
+    ExternalBrowserAuth,
+    JWTAuth,
+    OAuthAuth,
+    ProgrammaticAccessTokenAuth,
+)
+
+__all__ = [
+    # Main classes
+    "Config",
+    "_Config",
+    "RAIConfig",
+    "ConfigFromDBT",
+    "ConfigFromSnowflake",
+
+    # Nested config models
+    "EngineConfig",
+    "DataConfig",
+    "CompilerConfig",
+    "ModelConfig",
+    "ReasonerConfig",
+    "DebugConfig",
+
+    # Connection types
+    "ConnectionConfig",
+    "SnowflakeConnection",
+    "DuckDBConnection",
+
+    # Snowflake authenticators
+    "UsernamePasswordAuth",
+    "UsernamePasswordMFAAuth",
+    "ExternalBrowserAuth",
+    "JWTAuth",
+    "OAuthAuth",
+    "ProgrammaticAccessTokenAuth",
+]

relationalai-1.0.0a1/src/relationalai/config/config.py

@@ -0,0 +1,289 @@
+"""
+Main configuration class for PyRel (YAML-based).
+
+"""
+
+from __future__ import annotations
+
+from abc import ABC
+from typing import Any, TypeVar, overload, Literal, TYPE_CHECKING
+from pydantic import Field, model_validator
+from pydantic_settings import SettingsConfigDict
+
+try:
+    from confocal import BaseConfig
+except ImportError:
+    # Confocal not yet published to PyPI - use pydantic_settings as fallback
+    # Config system not actively used yet, this is just to prevent import errors
+    from pydantic_settings import BaseSettings as BaseConfig  # type: ignore[misc, assignment]
+
+if TYPE_CHECKING:
+    import snowflake.snowpark
+    import duckdb
+
+from .connections import ConnectionConfig, BaseConnection, SnowflakeConnection, DuckDBConnection
+from .config_fields import EngineConfig, DataConfig, CompilerConfig, ModelConfig, ReasonerConfig, DebugConfig
+from .external.dbt_converter import convert_dbt_to_rai
+from .external.snowflake_converter import convert_snowflake_to_rai
+from .external.utils import find_dbt_profiles_file, find_snowflake_config_file
+
+# TypeVar for generic connection retrieval
+T = TypeVar('T', bound=BaseConnection)
+
+
+class _Config(BaseConfig, ABC): # type: ignore for now until we publish confocal
+    """Base configuration class with common fields and methods."""
+
+    active_profile: str | None = Field(
+        default=None,
+        description="Currently active profile name (auto-defaults if only one profile)"
+    )
+
+    profiles: dict[str, dict[str, Any]] = Field(
+        default_factory=dict,
+        alias="profile",
+        description="Profiles available (dev, prod, staging, etc.) - profiles overlay config fields"
+    )
+
+    connections: dict[str, ConnectionConfig] = Field(
+        default_factory=dict,
+        description="Connections available (snowflake, duckdb, etc.)"
+    )
+
+    default_connection: str | None = Field(
+        default=None,
+        description="Default connection name (auto-defaults if only one connection)"
+    )
+
+    use_graph_index: bool = Field(
+        default=True,
+        description="Enabling graph index"
+    )
+    use_direct_access: bool = Field(
+        default=False,
+        description="Use direct access mode"
+    )
+    describe_optimization: str | None = Field(default=None, description="Describe optimization")
+    enable_otel_handler: bool = Field(
+        default=False,
+        description="Enable OpenTelemetry handler"
+    )
+
+    # Nested config objects
+    engine: EngineConfig = Field(
+        default_factory=EngineConfig,
+        description="Engine configuration"
+    )
+    data: DataConfig = Field(
+        default_factory=DataConfig,
+        description="Data loading and streaming configuration"
+    )
+    compiler: CompilerConfig = Field(
+        default_factory=CompilerConfig,
+        description="Compiler configuration"
+    )
+    model: ModelConfig = Field(
+        default_factory=ModelConfig,
+        description="Model configuration"
+    )
+    reasoner: ReasonerConfig = Field(
+        default_factory=ReasonerConfig,
+        description="Reasoner configuration"
+    )
+    debug: DebugConfig = Field(
+        default_factory=DebugConfig,
+        description="Debug configuration"
+    )
+
+    @model_validator(mode='before')
+    @classmethod
+    def set_default_snowflake_authenticator(cls, data: Any):
+        """Set default authenticator for Snowflake connections in raw data."""
+        if not isinstance(data, dict):
+            return data
+
+        def process_connections(connections):
+            if not isinstance(connections, dict):
+                return
+            for conn_data in connections.values():
+                if isinstance(conn_data, dict) and conn_data.get('type') == 'snowflake':
+                    conn_data.setdefault('authenticator', 'username_password')
+
+        # Process top-level connections
+        if 'connections' in data:
+            process_connections(data['connections'])
+
+        # Process connections in profiles (before overlay)
+        if 'profiles' in data and isinstance(data['profiles'], dict):
+            for profile_data in data['profiles'].values():
+                if isinstance(profile_data, dict) and 'connections' in profile_data:
+                    process_connections(profile_data['connections'])
+
+        return data
+
+    @model_validator(mode='after')
+    def validate_connections(self):
+        if not self.connections:
+            raise ValueError("Config must have at least one connection")
+
+        # Auto-set default_connection if only one connection exists and not already set
+        if len(self.connections) == 1 and self.default_connection is None:
+            self.default_connection = next(iter(self.connections.keys()))
+
+        return self
+
+    def get_default_connection(self) -> ConnectionConfig:
+        default_conn_name = self.default_connection
+
+        if default_conn_name is None:
+            if len(self.connections) == 1:
+                default_conn_name = list(self.connections.keys())[0]
+            else:
+                raise ValueError(
+                    f"Multiple connections available but no default_connection specified. "
+                    f"Available: {list(self.connections.keys())}"
+                )
+
+        if default_conn_name not in self.connections:
+            raise ValueError(
+                f"default_connection '{default_conn_name}' not found. "
+                f"Available: {list(self.connections.keys())}"
+            )
+
+        return self.connections[default_conn_name]
+
+    def get_connection(self, connection_type: type[T], name: str | None = None) -> T:
+        if name is None:
+            connection = self.get_default_connection()
+        else:
+            if name not in self.connections:
+                raise ValueError(
+                    f"Connection '{name}' not found. "
+                    f"Available: {list(self.connections.keys())}"
+                )
+            connection = self.connections[name]
+
+        if not isinstance(connection, connection_type):
+            # Provide helpful error message with the actual connection class hierarchy
+            actual_type = type(connection).__name__
+            # Check if it's a Snowflake connection
+            if isinstance(connection, SnowflakeConnection):
+                connection_class = "SnowflakeConnection"
+            elif isinstance(connection, DuckDBConnection):
+                connection_class = "DuckDBConnection"
+            else:
+                connection_class = actual_type
+
+            raise ValueError(
+                f"Connection '{name or self.default_connection}' is not of type {connection_type.__name__}. "
+                f"Got: {connection_class} (actual: {actual_type})"
+            )
+
+        return connection
+
+    @overload
+    def get_session(self, connection_type: type[SnowflakeConnection]) -> snowflake.snowpark.Session: ...
+
+    @overload
+    def get_session(self, connection_type: type[DuckDBConnection]) -> duckdb.DuckDBPyConnection: ...
+
+    @overload
+    def get_session(self, connection_type: None = None) -> snowflake.snowpark.Session | duckdb.DuckDBPyConnection: ...
+
+    def get_session(self, connection_type: type[SnowflakeConnection] | type[DuckDBConnection] | None = None) -> snowflake.snowpark.Session | duckdb.DuckDBPyConnection:
+        if connection_type is None:
+            connection = self.get_default_connection()
+            return connection.get_session()
+        else:
+            connection = self.get_connection(connection_type)
+            return connection.get_session()
+
+
+# =============================================================================
+# Config Source Classes
+# =============================================================================
+
+class RAIConfig(_Config):
+    """Config loaded from raiconfig.yaml."""
+
+    source: Literal["rai"] = Field(default="rai", exclude=True)
+
+    model_config = SettingsConfigDict(
+        yaml_file="raiconfig.yaml",
+        extra="ignore",
+        nested_model_default_partial_update=True,
+    )
+
+
+class ConfigFromSnowflake(_Config):
+    source: Literal["snowflake"] = Field(default="snowflake", exclude=True)
+
+    model_config = SettingsConfigDict(
+        toml_file=find_snowflake_config_file(),
+        extra="ignore",
+        nested_model_default_partial_update=True,
+    )
+
+    @model_validator(mode='before')
+    @classmethod
+    def convert_snowflake_structure(cls, data: Any):
+        if not isinstance(data, dict):
+            return data
+        return convert_snowflake_to_rai(data)
+
+class ConfigFromDBT(_Config):
+    source: Literal["dbt"] = Field(default="dbt", exclude=True)
+
+    model_config = SettingsConfigDict(
+        yaml_file=find_dbt_profiles_file(),
+        extra="ignore",
+        nested_model_default_partial_update=True,
+    )
+
+    @model_validator(mode='before')
+    @classmethod
+    def convert_dbt_structure(cls, data: Any):
+        """Convert DBT profiles.yml format to RAI Config format."""
+        if not isinstance(data, dict):
+            return data
+
+        return convert_dbt_to_rai(data)
+
+
+# =============================================================================
+# Config Factory - Tries each source in order
+# =============================================================================
+
+def Config(**data) -> _Config:
+    """
+    Create Config by trying multiple sources in priority order:
+    1. RAIConfig (raiconfig.yaml) - or direct data if provided
+    2. ConfigFromSnowflake (config.toml) - only if no data provided
+    3. ConfigFromDBT (profiles.yml) - only if no data provided
+    """
+    sources = [
+        ("RAIConfig (raiconfig.yaml)", RAIConfig, True),      # passes **data
+        ("ConfigFromSnowflake (config.toml)", ConfigFromSnowflake, False),
+        ("ConfigFromDBT (profiles.yml)", ConfigFromDBT, False),
+    ]
+
+    errors = []
+
+    for name, source_cls, accepts_data in sources:
+        try:
+            if accepts_data:
+                return source_cls(**data)
+            else:
+                return source_cls()
+        except Exception as e:
+            errors.append((name, str(e)))
+
+    # All failed, create helpful error message
+    error_lines = [f"  ❌ {name}: {error}" for name, error in errors]
+    error_summary = "\n".join(error_lines)
+
+    raise FileNotFoundError(
+        f"Could not load config from any source:\n\n{error_summary}\n\n"
+        f"To fix: Correct one of the config files above, or pass config programmatically:\n"
+        f"   Config(connections={{...}})"
+    )