dbt-cube-sync 0.1.0a1__tar.gz

dbt_cube_sync-0.1.0a1/PKG-INFO

@@ -0,0 +1,230 @@
+Metadata-Version: 2.4
+Name: dbt-cube-sync
+Version: 0.1.0a1
+Summary: Synchronization tool for dbt models to Cube.js schemas and BI tools
+Author: Ponder
+Requires-Python: >=3.9,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Dist: click (>=8.1.7,<9.0.0)
+Requires-Dist: jinja2 (>=3.1.2,<4.0.0)
+Requires-Dist: pydantic (>=2.5.0,<3.0.0)
+Requires-Dist: pyyaml (>=6.0,<7.0)
+Requires-Dist: requests (>=2.31.0,<3.0.0)
+Description-Content-Type: text/markdown
+
+# dbt-cube-sync
+
+A powerful synchronization tool that creates a seamless pipeline from dbt models to Cube.js schemas and BI tools (Superset, Tableau, PowerBI).
+
+## Features
+
+- 🔄 **dbt → Cube.js**: Auto-generate Cube.js schemas from dbt models with metrics
+- 📊 **Cube.js → BI Tools**: Sync schemas to multiple BI platforms
+- 🏗️ **Extensible Architecture**: Plugin-based connector system for easy BI tool integration
+- 🐳 **Docker Support**: Containerized execution with orchestration support
+- 🎯 **CLI Interface**: Simple command-line tools for automation
+
+## Supported BI Tools
+
+- ✅ **Apache Superset** - Full implementation
+- 🚧 **Tableau** - Placeholder (coming soon)
+- 🚧 **PowerBI** - Placeholder (coming soon)
+
+## Installation
+
+### Using Poetry (Development)
+
+```bash
+cd dbt-cube-sync
+poetry install
+poetry run dbt-cube-sync --help
+```
+
+### Using Docker
+
+```bash
+docker build -t dbt-cube-sync .
+docker run --rm dbt-cube-sync --help
+```
+
+## Quick Start
+
+### 1. Create Configuration File
+
+```bash
+# Create sample config
+dbt-cube-sync create-config sync-config.yaml
+
+# Edit the config file with your BI tool credentials
+```
+
+### 2. Generate Cube.js Schemas
+
+```bash
+# Generate from dbt manifest
+dbt-cube-sync generate-cubes \\
+  --dbt-manifest ./DbtEducationalDataProject/target/manifest.json \\
+  --output-dir ./cube/conf/cube_output
+```
+
+### 3. Sync to BI Tool
+
+```bash
+# Sync to Superset
+dbt-cube-sync sync-bi superset \\
+  --cube-dir ./cube/conf/cube_output \\
+  --config-file ./sync-config.yaml
+```
+
+### 4. Full Pipeline
+
+```bash
+# Complete dbt → Cube.js → Superset pipeline
+dbt-cube-sync full-sync \\
+  --dbt-manifest ./DbtEducationalDataProject/target/manifest.json \\
+  --cube-dir ./cube/conf/cube_output \\
+  --bi-connector superset \\
+  --config-file ./sync-config.yaml
+```
+
+## Configuration
+
+### Sample Configuration (`sync-config.yaml`)
+
+```yaml
+connectors:
+  superset:
+    type: superset
+    url: http://localhost:8088
+    username: admin
+    password: admin
+    database_name: Cube
+    
+  tableau:
+    type: tableau
+    url: https://your-tableau-server.com
+    username: your-username
+    password: your-password
+    
+  powerbi:
+    type: powerbi
+    # PowerBI specific configuration
+```
+
+## CLI Commands
+
+### `generate-cubes`
+Generate Cube.js schema files from dbt models.
+
+**Options:**
+- `--dbt-manifest` / `-m`: Path to dbt manifest.json file
+- `--output-dir` / `-o`: Output directory for Cube.js files
+- `--template-dir` / `-t`: Directory containing Cube.js templates
+
+### `sync-bi`
+Sync Cube.js schemas to BI tool datasets.
+
+**Arguments:**
+- `connector`: BI tool type (`superset`, `tableau`, `powerbi`)
+
+**Options:**
+- `--cube-dir` / `-c`: Directory containing Cube.js files
+- `--config-file` / `-f`: Configuration file for BI tool connection
+
+### `full-sync`
+Complete pipeline: dbt models → Cube.js schemas → BI tool datasets.
+
+**Options:**
+- `--dbt-manifest` / `-m`: Path to dbt manifest.json file
+- `--cube-dir` / `-c`: Directory for Cube.js files
+- `--template-dir` / `-t`: Directory containing Cube.js templates
+- `--bi-connector` / `-b`: BI tool to sync to
+- `--config-file` / `-f`: Configuration file for BI tool connection
+
+## Architecture
+
+```
+dbt models (with metrics) 
+    ↓
+dbt-cube-sync generate-cubes
+    ↓
+Cube.js schemas
+    ↓
+dbt-cube-sync sync-bi [connector]
+    ↓
+BI Tool Datasets (Superset/Tableau/PowerBI)
+```
+
+### Project Structure
+
+```
+dbt-cube-sync/
+├── dbt_cube_sync/
+│   ├── cli.py                 # CLI interface
+│   ├── config.py             # Configuration management
+│   ├── core/
+│   │   ├── dbt_parser.py     # dbt manifest parser
+│   │   ├── cube_generator.py # Cube.js generator
+│   │   └── models.py         # Pydantic data models
+│   └── connectors/
+│       ├── base.py           # Abstract base connector
+│       ├── superset.py       # Superset implementation
+│       ├── tableau.py        # Tableau placeholder
+│       └── powerbi.py        # PowerBI placeholder
+├── Dockerfile                # Container definition
+├── pyproject.toml            # Poetry configuration
+└── README.md
+```
+
+## Adding New BI Connectors
+
+1. Create a new connector class inheriting from `BaseConnector`
+2. Implement the required abstract methods
+3. Register the connector using `ConnectorRegistry.register()`
+
+Example:
+```python
+from .base import BaseConnector, ConnectorRegistry
+
+class MyBIConnector(BaseConnector):
+    def _validate_config(self):
+        # Validation logic
+        pass
+    
+    def connect(self):
+        # Connection logic
+        pass
+    
+    def sync_cube_schemas(self, cube_dir):
+        # Sync implementation
+        pass
+
+# Register the connector
+ConnectorRegistry.register('mybi', MyBIConnector)
+```
+
+## Docker Integration
+
+The tool is designed to work in containerized environments with proper dependency orchestration:
+
+1. **dbt docs**: Runs `dbt build` then serves documentation
+2. **dbt-cube-sync**: Runs sync pipeline after dbt and Cube.js are ready  
+3. **BI Tools**: Receive synced datasets after sync completes
+
+## Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Implement your changes
+4. Add tests if applicable
+5. Submit a pull request
+
+## License
+
+MIT License - see LICENSE file for details.

dbt_cube_sync-0.1.0a1/README.md

@@ -0,0 +1,210 @@
+# dbt-cube-sync
+
+A powerful synchronization tool that creates a seamless pipeline from dbt models to Cube.js schemas and BI tools (Superset, Tableau, PowerBI).
+
+## Features
+
+- 🔄 **dbt → Cube.js**: Auto-generate Cube.js schemas from dbt models with metrics
+- 📊 **Cube.js → BI Tools**: Sync schemas to multiple BI platforms
+- 🏗️ **Extensible Architecture**: Plugin-based connector system for easy BI tool integration
+- 🐳 **Docker Support**: Containerized execution with orchestration support
+- 🎯 **CLI Interface**: Simple command-line tools for automation
+
+## Supported BI Tools
+
+- ✅ **Apache Superset** - Full implementation
+- 🚧 **Tableau** - Placeholder (coming soon)
+- 🚧 **PowerBI** - Placeholder (coming soon)
+
+## Installation
+
+### Using Poetry (Development)
+
+```bash
+cd dbt-cube-sync
+poetry install
+poetry run dbt-cube-sync --help
+```
+
+### Using Docker
+
+```bash
+docker build -t dbt-cube-sync .
+docker run --rm dbt-cube-sync --help
+```
+
+## Quick Start
+
+### 1. Create Configuration File
+
+```bash
+# Create sample config
+dbt-cube-sync create-config sync-config.yaml
+
+# Edit the config file with your BI tool credentials
+```
+
+### 2. Generate Cube.js Schemas
+
+```bash
+# Generate from dbt manifest
+dbt-cube-sync generate-cubes \\
+  --dbt-manifest ./DbtEducationalDataProject/target/manifest.json \\
+  --output-dir ./cube/conf/cube_output
+```
+
+### 3. Sync to BI Tool
+
+```bash
+# Sync to Superset
+dbt-cube-sync sync-bi superset \\
+  --cube-dir ./cube/conf/cube_output \\
+  --config-file ./sync-config.yaml
+```
+
+### 4. Full Pipeline
+
+```bash
+# Complete dbt → Cube.js → Superset pipeline
+dbt-cube-sync full-sync \\
+  --dbt-manifest ./DbtEducationalDataProject/target/manifest.json \\
+  --cube-dir ./cube/conf/cube_output \\
+  --bi-connector superset \\
+  --config-file ./sync-config.yaml
+```
+
+## Configuration
+
+### Sample Configuration (`sync-config.yaml`)
+
+```yaml
+connectors:
+  superset:
+    type: superset
+    url: http://localhost:8088
+    username: admin
+    password: admin
+    database_name: Cube
+    
+  tableau:
+    type: tableau
+    url: https://your-tableau-server.com
+    username: your-username
+    password: your-password
+    
+  powerbi:
+    type: powerbi
+    # PowerBI specific configuration
+```
+
+## CLI Commands
+
+### `generate-cubes`
+Generate Cube.js schema files from dbt models.
+
+**Options:**
+- `--dbt-manifest` / `-m`: Path to dbt manifest.json file
+- `--output-dir` / `-o`: Output directory for Cube.js files
+- `--template-dir` / `-t`: Directory containing Cube.js templates
+
+### `sync-bi`
+Sync Cube.js schemas to BI tool datasets.
+
+**Arguments:**
+- `connector`: BI tool type (`superset`, `tableau`, `powerbi`)
+
+**Options:**
+- `--cube-dir` / `-c`: Directory containing Cube.js files
+- `--config-file` / `-f`: Configuration file for BI tool connection
+
+### `full-sync`
+Complete pipeline: dbt models → Cube.js schemas → BI tool datasets.
+
+**Options:**
+- `--dbt-manifest` / `-m`: Path to dbt manifest.json file
+- `--cube-dir` / `-c`: Directory for Cube.js files
+- `--template-dir` / `-t`: Directory containing Cube.js templates
+- `--bi-connector` / `-b`: BI tool to sync to
+- `--config-file` / `-f`: Configuration file for BI tool connection
+
+## Architecture
+
+```
+dbt models (with metrics) 
+    ↓
+dbt-cube-sync generate-cubes
+    ↓
+Cube.js schemas
+    ↓
+dbt-cube-sync sync-bi [connector]
+    ↓
+BI Tool Datasets (Superset/Tableau/PowerBI)
+```
+
+### Project Structure
+
+```
+dbt-cube-sync/
+├── dbt_cube_sync/
+│   ├── cli.py                 # CLI interface
+│   ├── config.py             # Configuration management
+│   ├── core/
+│   │   ├── dbt_parser.py     # dbt manifest parser
+│   │   ├── cube_generator.py # Cube.js generator
+│   │   └── models.py         # Pydantic data models
+│   └── connectors/
+│       ├── base.py           # Abstract base connector
+│       ├── superset.py       # Superset implementation
+│       ├── tableau.py        # Tableau placeholder
+│       └── powerbi.py        # PowerBI placeholder
+├── Dockerfile                # Container definition
+├── pyproject.toml            # Poetry configuration
+└── README.md
+```
+
+## Adding New BI Connectors
+
+1. Create a new connector class inheriting from `BaseConnector`
+2. Implement the required abstract methods
+3. Register the connector using `ConnectorRegistry.register()`
+
+Example:
+```python
+from .base import BaseConnector, ConnectorRegistry
+
+class MyBIConnector(BaseConnector):
+    def _validate_config(self):
+        # Validation logic
+        pass
+    
+    def connect(self):
+        # Connection logic
+        pass
+    
+    def sync_cube_schemas(self, cube_dir):
+        # Sync implementation
+        pass
+
+# Register the connector
+ConnectorRegistry.register('mybi', MyBIConnector)
+```
+
+## Docker Integration
+
+The tool is designed to work in containerized environments with proper dependency orchestration:
+
+1. **dbt docs**: Runs `dbt build` then serves documentation
+2. **dbt-cube-sync**: Runs sync pipeline after dbt and Cube.js are ready  
+3. **BI Tools**: Receive synced datasets after sync completes
+
+## Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Implement your changes
+4. Add tests if applicable
+5. Submit a pull request
+
+## License
+
+MIT License - see LICENSE file for details.

dbt_cube_sync-0.1.0a1/dbt_cube_sync/__init__.py

@@ -0,0 +1,5 @@
+"""
+dbt-cube-sync: Synchronization tool for dbt models to Cube.js schemas and BI tools
+"""
+
+__version__ = "0.1.0"

dbt_cube_sync-0.1.0a1/dbt_cube_sync/cli.py

@@ -0,0 +1,135 @@
+"""
+CLI interface for dbt-cube-sync tool
+"""
+import click
+import sys
+from pathlib import Path
+from typing import Optional
+
+from .core.dbt_parser import DbtParser
+from .core.cube_generator import CubeGenerator
+from .connectors.base import ConnectorRegistry
+from .config import Config
+
+# Import connectors to register them
+from .connectors import superset, tableau, powerbi
+
+
+class CustomGroup(click.Group):
+    def get_command(self, ctx, cmd_name):
+        # Handle common mistake of typing dbt-cube-sync twice
+        if cmd_name == 'dbt-cube-sync':
+            click.echo("❌ Error: You typed 'dbt-cube-sync' twice!")
+            click.echo("💡 Just run: dbt-cube-sync <command>")
+            click.echo("\nAvailable commands:")
+            click.echo("  dbt-cube-sync --help                                    # Show help")
+            click.echo("  dbt-cube-sync --version                                 # Show version")
+            click.echo("  dbt-cube-sync dbt-to-cube -m manifest -c catalog -o output # Generate Cube.js schemas")
+            click.echo("  dbt-cube-sync cube-to-bi superset -c cubes -u url -n user -p pass -d Cube # Sync to BI tool")
+            ctx.exit(1)
+        
+        return super().get_command(ctx, cmd_name)
+
+
+@click.group(cls=CustomGroup)
+@click.version_option()
+def main():
+    """dbt-cube-sync: Synchronization tool for dbt models to Cube.js schemas and BI tools"""
+    pass
+
+
+@main.command()
+@click.option('--manifest', '-m', 
+              required=True,
+              help='Path to dbt manifest.json file')
+@click.option('--catalog', '-c',
+              required=True,
+              help='Path to dbt catalog.json file')
+@click.option('--output', '-o',
+              required=True,
+              help='Output directory for Cube.js files')
+@click.option('--template-dir', '-t',
+              default='./cube/templates',
+              help='Directory containing Cube.js templates')
+def dbt_to_cube(manifest: str, catalog: str, output: str, template_dir: str):
+    """Generate Cube.js schemas from dbt models"""
+    try:
+        click.echo("🔄 Parsing dbt manifest...")
+        parser = DbtParser(manifest, catalog)
+        models = parser.parse_models()
+        
+        click.echo(f"📊 Found {len(models)} dbt models")
+        
+        click.echo("🏗️  Generating Cube.js schemas...")
+        generator = CubeGenerator(template_dir, output)
+        generated_files = generator.generate_cube_files(models)
+        
+        click.echo(f"✅ Generated {len(generated_files)} Cube.js files:")
+        for file_path in generated_files:
+            click.echo(f"   • {file_path}")
+            
+    except Exception as e:
+        click.echo(f"❌ Error: {str(e)}", err=True)
+        sys.exit(1)
+
+
+@main.command()
+@click.argument('bi_tool', type=click.Choice(['superset', 'tableau', 'powerbi']))
+@click.option('--cube-files', '-c',
+              required=True,
+              help='Directory containing Cube.js metric files')
+@click.option('--url', '-u',
+              required=True,
+              help='BI tool URL (e.g., http://localhost:8088)')
+@click.option('--username', '-n',
+              required=True,
+              help='BI tool username')
+@click.option('--password', '-p',
+              required=True,
+              help='BI tool password')
+@click.option('--cube-connection-name', '-d',
+              default='Cube',
+              help='Name of the Cube database connection in the BI tool (default: Cube)')
+def cube_to_bi(bi_tool: str, cube_files: str, url: str, username: str, password: str, cube_connection_name: str):
+    """Sync Cube.js schemas to BI tool datasets"""
+    try:
+        click.echo(f"🔄 Connecting to {bi_tool.title()} at {url}...")
+        
+        # Create connector config from command line params
+        connector_config = {
+            'url': url,
+            'username': username,
+            'password': password,
+            'database_name': cube_connection_name
+        }
+        
+        connector_instance = ConnectorRegistry.get_connector(bi_tool, **connector_config)
+        
+        click.echo(f"📊 Syncing Cube.js schemas to {bi_tool.title()}...")
+        results = connector_instance.sync_cube_schemas(cube_files)
+        
+        successful = sum(1 for r in results if r.status == 'success')
+        failed = sum(1 for r in results if r.status == 'failed')
+        
+        click.echo(f"✅ Sync complete: {successful} successful, {failed} failed")
+        
+        # Show detailed results
+        for result in results:
+            status_emoji = "✅" if result.status == 'success' else "❌"
+            click.echo(f"   {status_emoji} {result.file_or_dataset}: {result.message}")
+            
+    except Exception as e:
+        click.echo(f"❌ Error: {str(e)}", err=True)
+        sys.exit(1)
+
+
+
+@main.command()
+def version():
+    """Show version information"""
+    from . import __version__
+    click.echo(f"dbt-cube-sync version {__version__}")
+
+
+if __name__ == '__main__':
+    main()

dbt_cube_sync-0.1.0a1/dbt_cube_sync/config.py

@@ -0,0 +1,121 @@
+"""
+Configuration management for dbt-cube-sync
+"""
+import yaml
+from typing import Dict, Any, Optional
+from pathlib import Path
+from pydantic import BaseModel, validator
+
+
+class ConnectorConfig(BaseModel):
+    """Configuration for a BI tool connector"""
+    type: str
+    url: Optional[str] = None
+    username: Optional[str] = None
+    password: Optional[str] = None
+    database_name: Optional[str] = "Cube"
+    
+    @validator('type')
+    def validate_type(cls, v):
+        supported_types = ['superset', 'tableau', 'powerbi']
+        if v not in supported_types:
+            raise ValueError(f"Unsupported connector type: {v}. Supported: {supported_types}")
+        return v
+
+
+class Config(BaseModel):
+    """Main configuration class"""
+    connectors: Dict[str, ConnectorConfig] = {}
+    
+    @classmethod
+    def load_from_file(cls, config_path: str) -> 'Config':
+        """
+        Load configuration from YAML file
+        
+        Args:
+            config_path: Path to the YAML configuration file
+            
+        Returns:
+            Config instance
+        """
+        config_file = Path(config_path)
+        if not config_file.exists():
+            raise FileNotFoundError(f"Configuration file not found: {config_path}")
+        
+        with open(config_file, 'r') as f:
+            data = yaml.safe_load(f)
+        
+        # Parse connector configurations
+        connectors = {}
+        for name, connector_data in data.get('connectors', {}).items():
+            connectors[name] = ConnectorConfig(**connector_data)
+        
+        return cls(connectors=connectors)
+    
+    def get_connector_config(self, connector_name: str) -> Dict[str, Any]:
+        """
+        Get configuration for a specific connector
+        
+        Args:
+            connector_name: Name of the connector (e.g., 'superset')
+            
+        Returns:
+            Dictionary with connector configuration
+        """
+        if connector_name not in self.connectors:
+            available = list(self.connectors.keys())
+            raise ValueError(f"Connector '{connector_name}' not found in config. Available: {available}")
+        
+        config = self.connectors[connector_name]
+        return config.dict()
+    
+    def save_to_file(self, config_path: str) -> None:
+        """
+        Save configuration to YAML file
+        
+        Args:
+            config_path: Path to save the configuration file
+        """
+        data = {
+            'connectors': {
+                name: config.dict() for name, config in self.connectors.items()
+            }
+        }
+        
+        config_file = Path(config_path)
+        config_file.parent.mkdir(parents=True, exist_ok=True)
+        
+        with open(config_file, 'w') as f:
+            yaml.dump(data, f, default_flow_style=False, indent=2)
+    
+    @classmethod
+    def create_sample_config(cls, config_path: str) -> None:
+        """
+        Create a sample configuration file
+        
+        Args:
+            config_path: Path to create the sample configuration file
+        """
+        sample_config = cls(
+            connectors={
+                'superset': ConnectorConfig(
+                    type='superset',
+                    url='http://localhost:8088',
+                    username='admin',
+                    password='admin',
+                    database_name='Cube'
+                ),
+                'tableau': ConnectorConfig(
+                    type='tableau',
+                    url='https://your-tableau-server.com',
+                    username='your-username',
+                    password='your-password'
+                ),
+                'powerbi': ConnectorConfig(
+                    type='powerbi',
+                    # Add PowerBI specific configuration fields here
+                )
+            }
+        )
+        
+        sample_config.save_to_file(config_path)

dbt_cube_sync-0.1.0a1/dbt_cube_sync/connectors/__init__.py

@@ -0,0 +1 @@
+"""BI tool connectors for syncing Cube.js schemas"""