starrocks-br 0.1.0__tar.gz → 0.2.0__tar.gz

{starrocks_br-0.1.0 → starrocks_br-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: starrocks-br
-Version: 0.1.0
+Version: 0.2.0
 Summary: StarRocks Backup and Restore automation tool
 Requires-Python: >=3.9
 Requires-Dist: click<9,>=8.1.7

{starrocks_br-0.1.0 → starrocks_br-0.2.0}/README.md

@@ -4,13 +4,83 @@
 
 The StarRocks Backup & Restore tool provides production-grade automation for backup and restore operations.
 
+**Important:** This tool requires StarRocks 3.5 or later. Earlier versions are not supported due to differences in the `SHOW FRONTENDS` and `SHOW BACKENDS` command output formats, which are used for cluster health checks.
+
+## Summary
+
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [Configuration](#configuration)
+  - [Password Management](#password-management)
+  - [Connecting with TLS/SSL](#connecting-with-tlsssl)
+- [Commands](#commands)
+- [Example Usage Scenarios](#example-usage-scenarios)
+- [Error Handling](#error-handling)
+- [Monitoring](#monitoring)
+- [Testing](#testing)
+- [Project Status](#project-status)
+
 ## Installation
 
-### Option 1: Using Devbox (Recommended for Development)
+### Option 1: Install from PyPI (Recommended for Production)
+
+We recommend using a virtual environment to ensure proper script availability and dependency isolation:
+
+```bash
+# Create and activate a virtual environment
+python3 -m venv .venv
+source .venv/bin/activate  # On Linux/Mac
+# .venv\Scripts\activate    # On Windows
+
+# Install the package from PyPI
+pip install starrocks-br==0.1.0
+
+# Verify the installation
+starrocks-br --help
+```
+
+**Note:** Always activate the virtual environment before using the tool. The `starrocks-br` command will only be available when the virtual environment is activated.
+
+### Option 2: Standalone Executable (Alternative if PyPI Installation Fails)
+
+If you encounter problems when installing via PyPI (often due to `mysql-connector-python` native extension issues), you can build a standalone executable that bundles all dependencies.
+
+**Note:** This requires cloning the repository first.
+
+```bash
+# Clone the repository
+git clone https://github.com/deep-bi/starrocks-br
+cd starrocks-br
+
+# Create and activate virtual environment
+python3 -m venv .venv
+source .venv/bin/activate  # On Linux/Mac
+# .venv\Scripts\activate    # On Windows
+
+# Install dependencies
+pip install -e .
+
+# Build the standalone executable
+./build_executable.sh
+
+# The executable will be created in: dist/starrocks-br
+# You can now distribute or use this executable directly
+./dist/starrocks-br --help
+```
+
+**Note:** The executable is platform-specific. Build it on the same OS/architecture where you'll use it, or build it on the target machine.
+
+### Option 3: Using Devbox (Recommended for Development)
+
+**Note:** This requires cloning the repository first.
 
 Devbox provides a reproducible development environment with all required tools.
 
 ```bash
+# Clone the repository
+git clone https://github.com/deep-bi/starrocks-br
+cd starrocks-br
+
 # Install devbox (if not already installed)
 curl -fsSL https://get.jetpack.io/devbox | bash
 
@@ -26,18 +96,67 @@ starrocks-br --help
 pytest
 ```
 
-### Option 2: Manual Setup
+### Option 4: Manual Development Setup
 
 ```bash
-# Activate virtual environment
+# Clone the repository
+git clone https://github.com/deep-bi/starrocks-br
+cd starrocks-br
+
+# Create and activate virtual environment
+python3 -m venv .venv
 source .venv/bin/activate
 
-# The CLI is already installed as: starrocks-br
+# Install in editable mode with development dependencies
+pip install -e ".[dev]"
+
+# The CLI is now available as: starrocks-br
 ```
 
+## Quick Start
+
+After installing from PyPI, follow these steps:
+
+1. **Activate your virtual environment** (if not already active):
+   ```bash
+   source .venv/bin/activate  # On Linux/Mac
+   # .venv\Scripts\activate    # On Windows
+   ```
+
+2. **Verify installation:**
+   ```bash
+   starrocks-br --help
+   ```
+
+3. **Create your `config.yaml` file** (see Configuration section below)
+
+4. **Set your password as an environment variable:**
+   ```bash
+   export STARROCKS_PASSWORD="your_password"
+   ```
+   
+   On Windows (PowerShell):
+   ```powershell
+   $env:STARROCKS_PASSWORD="your_password"
+   ```
+   
+   On Windows (Command Prompt):
+   ```cmd
+   set STARROCKS_PASSWORD=your_password
+   ```
+
+5. **Initialize the ops schema:**
+   ```bash
+   starrocks-br init --config config.yaml
+   ```
+
+6. **Start using the tool** - see Commands section below for details
+
 ## Configuration
 
-Create a `config.yaml` file with your StarRocks connection details:
+**Important:** After installing the package, you need to create your own `config.yaml` file. This file is **not included in the package** - each user creates it with their own StarRocks connection details. You can place it anywhere and reference it using the `--config` parameter.
+
+Create a `config.yaml` file in your working directory (or any location you prefer) with your StarRocks connection details:
 
 ```yaml
 host: "127.0.0.1"
@@ -55,6 +174,53 @@ The database password must be provided via the `STARROCKS_PASSWORD` environment
 export STARROCKS_PASSWORD="your_password"
 ```
 
+### Connecting with TLS/SSL
+
+The tool can make secure connections to StarRocks using TLS. Add an optional `tls` section to your `config.yaml` when you need encryption.
+
+#### Scenario 1: Server Authentication (Most Common)
+
+Use this setup when the client only needs to verify the StarRocks server certificate.
+
+```yaml
+host: "127.0.0.1"
+port: 9030
+user: "root"
+database: "your_database"
+repository: "your_repo_name"
+
+tls:
+  enabled: true
+  ca_cert: "/path/to/ca.pem"
+```
+
+- `enabled`: Turns TLS on or off.
+- `ca_cert`: Certificate Authority file used to validate the server certificate.
+- `verify_server_cert` (optional, default `true`): Disable only if you need to skip certificate validation.
+
+#### Scenario 2: Mutual TLS (mTLS)
+
+Use this when both the client and server must present certificates.
+
+```yaml
+host: "127.0.0.1"
+port: 9030
+user: "root"
+database: "your_database"
+repository: "your_repo_name"
+
+tls:
+  enabled: true
+  ca_cert: "/path/to/ca.pem"
+  client_cert: "/path/to/client-cert.pem"
+  client_key: "/path/to/client-key.pem"
+```
+
+- `client_cert`: Client certificate presented to the server.
+- `client_key`: Private key paired with the client certificate.
+
+Regardless of the scenario, the connection defaults to modern TLS versions (`TLSv1.2`, `TLSv1.3`). Provide a `tls_versions` list if you need different protocol settings.
+
 **Note:** The repository must be created in StarRocks using the `CREATE REPOSITORY` command before running backups. For example:
 
 ```sql
@@ -113,7 +279,7 @@ starrocks-br backup full --config config.yaml --group <group_name>
 **Parameters:**
 - `--group`: The inventory group to back up.
 
-**Flow:**
+**Internal flow:**
 1. Load config → verify cluster health → ensure repository exists
 2. Reserve job slot (prevent concurrent backups)
 3. Query `ops.table_inventory` for all tables in the specified group.
@@ -133,7 +299,7 @@ starrocks-br backup incremental --config config.yaml --group <group_name>
 - `--group`: The inventory group to back up.
 - `--baseline-backup` (Optional): Specify a backup label to use as the baseline instead of the latest full backup.
 
-**Flow:**
+**Internal flow:**
 1. Load config → verify cluster health → ensure repository exists
 2. Reserve job slot
 3. Find the latest successful full backup for the group to use as a baseline.
@@ -159,7 +325,8 @@ starrocks-br restore \
 **Parameters:**
 - `--config`: Path to config YAML file (required)
 - `--target-label`: Backup label to restore to (required)
-- `--group`: Optional inventory group to filter tables to restore
+- `--group`: Optional inventory group to filter tables to restore (cannot be used with `--table`)
+- `--table`: Optional table name to restore (table name only, database comes from config). Cannot be used with `--group`
 - `--rename-suffix`: Suffix for temporary tables during restore (default: `_restored`)
 
 **How it works:**
@@ -167,14 +334,18 @@ starrocks-br restore \
 - **For incremental backups**: Automatically restores the base full backup first, then applies the incremental
 - **Safety mechanism**: Uses temporary tables with the specified suffix, then performs atomic rename to make restored data live
 
-**Two Restore Modes:**
-- **Disaster Recovery**: Restore all tables from a backup (omit `--group` parameter)
-- **Surgical Restore**: Restore only specific table groups (use `--group` parameter)
+**Three Restore Modes:**
+- **Disaster Recovery**: Restore all tables from a backup (omit both `--group` and `--table` parameters)
+- **Surgical Restore by Group**: Restore only specific table groups (use `--group` parameter)
+- **Single Table Restore**: Restore a specific table (use `--table` parameter). The table name should not include the database prefix - the database comes from the config file.
+
+**Table Name Format:**
+When using `--table`, provide only the table name (e.g., `fact_sales`), not `database.table_name`. The database is taken from the `database` field in your config file. For multiple tables, set up an inventory group and use `--group` instead.
 
 **Purpose of `--rename-suffix`:**
 The restore process creates temporary tables with the specified suffix (e.g., `table_restored`) to avoid conflicts with existing tables. Once the restore is complete and verified, the tool performs atomic renames to swap the original tables with the restored data. This ensures data safety and allows for rollback if needed.
 
-**Flow:**
+**Internal flow:**
 1. Load config → verify cluster health → ensure repository exists
 2. Find the correct restore sequence (full backup + optional incremental)
 3. Get tables from backup manifest (optionally filtered by group)
@@ -224,6 +395,12 @@ starrocks-br restore \
 starrocks-br restore \
   --config config.yaml \
   --target-label sales_db_20251014_full
+
+# Restore a single table from a backup
+starrocks-br restore \
+  --config config.yaml \
+  --target-label sales_db_20251015_inc \
+  --table fact_sales
 ```
 
 ## Error Handling
@@ -301,4 +478,4 @@ pytest tests/test_cli.py -v
 - Exponential backoff retry for job conflicts
 - Disk space precheck (requires external monitoring)
 - Formal runbooks and DR drill procedures
-- Monitoring dashboards and alerting integration
+- Monitoring dashboards and alerting integration

{starrocks_br-0.1.0 → starrocks_br-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "starrocks-br"
-version = "0.1.0"
+version = "0.2.0"
 description = "StarRocks Backup and Restore automation tool"
 requires-python = ">=3.9"
 dependencies = [

{starrocks_br-0.1.0 → starrocks_br-0.2.0}/src/starrocks_br/cli.py

@@ -15,6 +15,40 @@ from . import schema
 from . import logger
 
 
+def _handle_snapshot_exists_error(error_details: dict, label: str, config: str, repository: str, backup_type: str, group: str, baseline_backup: str = None) -> None:
+    """Handle snapshot_exists error by providing helpful guidance to the user.
+    
+    Args:
+        error_details: Error details dict containing error_type and snapshot_name
+        label: The backup label that was generated
+        config: Path to config file
+        repository: Repository name
+        backup_type: Type of backup ('incremental' or 'full')
+        group: Inventory group name
+        baseline_backup: Optional baseline backup label (for incremental backups)
+    """
+    snapshot_name = error_details.get('snapshot_name', label)
+    logger.error(f"Snapshot '{snapshot_name}' already exists in the repository.")
+    logger.info("")
+    logger.info("This typically happens when:")
+    logger.info("  • The CLI lost connectivity during a previous backup operation")
+    logger.info("  • The backup completed on the server, but backup_history wasn't updated")
+    logger.info("")
+    logger.info("To resolve this, retry the backup with a custom label using --name:")
+    
+    if backup_type == 'incremental':
+        retry_cmd = f"  starrocks-br backup incremental --config {config} --group {group} --name {snapshot_name}_retry"
+        if baseline_backup:
+            retry_cmd += f" --baseline-backup {baseline_backup}"
+        logger.info(retry_cmd)
+    else:
+        logger.info(f"  starrocks-br backup full --config {config} --group {group} --name {snapshot_name}_retry")
+    
+    logger.info("")
+    logger.tip("You can verify the existing backup by checking the repository or running:")
+    logger.tip(f"  SHOW SNAPSHOT ON {repository} WHERE Snapshot = '{snapshot_name}'")
+
+
 @click.group()
 def cli():
     """StarRocks Backup & Restore automation tool."""
@@ -43,7 +77,8 @@ def init(config):
             port=cfg['port'],
             user=cfg['user'],
             password=os.getenv('STARROCKS_PASSWORD'),
-            database=cfg['database']
+            database=cfg['database'],
+            tls_config=cfg.get('tls'),
         )
         
         with database:
@@ -102,7 +137,8 @@ def backup_incremental(config, baseline_backup, group, name):
             port=cfg['port'],
             user=cfg['user'],
             password=os.getenv('STARROCKS_PASSWORD'),
-            database=cfg['database']
+            database=cfg['database'],
+            tls_config=cfg.get('tls'),
         )
         
         with database:
@@ -174,6 +210,13 @@ def backup_incremental(config, baseline_backup, group, name):
                 logger.success(f"Backup completed successfully: {result['final_status']['state']}")
                 sys.exit(0)
             else:
+                error_details = result.get('error_details')
+                if error_details and error_details.get('error_type') == 'snapshot_exists':
+                    _handle_snapshot_exists_error(
+                        error_details, label, config, cfg['repository'], 'incremental', group, baseline_backup
+                    )
+                    sys.exit(1)
+                
                 state = result.get('final_status', {}).get('state', 'UNKNOWN')
                 if state == "LOST":
                     logger.critical("Backup tracking lost!")
@@ -215,7 +258,8 @@ def backup_full(config, group, name):
             port=cfg['port'],
             user=cfg['user'],
             password=os.getenv('STARROCKS_PASSWORD'),
-            database=cfg['database']
+            database=cfg['database'],
+            tls_config=cfg.get('tls'),
         )
         
         with database:
@@ -274,6 +318,13 @@ def backup_full(config, group, name):
                 logger.success(f"Backup completed successfully: {result['final_status']['state']}")
                 sys.exit(0)
             else:
+                error_details = result.get('error_details')
+                if error_details and error_details.get('error_type') == 'snapshot_exists':
+                    _handle_snapshot_exists_error(
+                        error_details, label, config, cfg['repository'], 'full', group
+                    )
+                    sys.exit(1)
+                
                 state = result.get('final_status', {}).get('state', 'UNKNOWN')
                 if state == "LOST":
                     logger.critical("Backup tracking lost!")
@@ -300,8 +351,10 @@ def backup_full(config, group, name):
 @click.option('--config', required=True, help='Path to config YAML file')
 @click.option('--target-label', required=True, help='Backup label to restore to')
 @click.option('--group', help='Optional inventory group to filter tables to restore')
+@click.option('--table', help='Optional table name to restore (table name only, database comes from config). Cannot be used with --group.')
 @click.option('--rename-suffix', default='_restored', help='Suffix for temporary tables during restore (default: _restored)')
-def restore_command(config, target_label, group, rename_suffix):
+@click.option('--yes', is_flag=True, help='Skip confirmation prompt and proceed automatically')
+def restore_command(config, target_label, group, table, rename_suffix, yes):
     """Restore data to a specific point in time using intelligent backup chain resolution.
     
     This command automatically determines the correct sequence of backups needed for restore:
@@ -311,9 +364,23 @@ def restore_command(config, target_label, group, rename_suffix):
     The restore process uses temporary tables with the specified suffix for safety, then performs
     an atomic rename to make the restored data live.
     
-    Flow: load config → find restore pair → get tables from backup → execute restore flow
+    Flow: load config → check health → ensure repository → find restore pair → get tables from backup → execute restore flow
     """
     try:
+        if group and table:
+            logger.error("Cannot specify both --group and --table. Use --table for single table restore or --group for inventory group restore.")
+            sys.exit(1)
+        
+        if table:
+            table = table.strip()
+            if not table:
+                logger.error("Table name cannot be empty")
+                sys.exit(1)
+            
+            if '.' in table:
+                logger.error("Table name must not include database prefix. Use 'table_name' not 'database.table_name'. Database comes from config file.")
+                sys.exit(1)
+        
         cfg = config_module.load_config(config)
         config_module.validate_config(cfg)
         
@@ -322,7 +389,8 @@ def restore_command(config, target_label, group, rename_suffix):
             port=cfg['port'],
             user=cfg['user'],
             password=os.getenv('STARROCKS_PASSWORD'),
-            database=cfg['database']
+            database=cfg['database'],
+            tls_config=cfg.get('tls'),
         )
         
         with database:
@@ -332,6 +400,17 @@ def restore_command(config, target_label, group, rename_suffix):
                 logger.warning("Remember to populate ops.table_inventory with your backup groups!")
                 sys.exit(1) # Exit if schema was just created, requires user action
             
+            healthy, message = health.check_cluster_health(database)
+            if not healthy:
+                logger.error(f"Cluster health check failed: {message}")
+                sys.exit(1)
+            
+            logger.success(f"Cluster health: {message}")
+            
+            repository.ensure_repository(database, cfg['repository'])
+            
+            logger.success(f"Repository '{cfg['repository']}' verified")
+            
             logger.info(f"Finding restore sequence for target backup: {target_label}")
             
             try:
@@ -342,11 +421,24 @@ def restore_command(config, target_label, group, rename_suffix):
                 sys.exit(1)
             
             logger.info("Determining tables to restore from backup manifest...")
-            tables_to_restore = restore.get_tables_from_backup(database, target_label, group)
+            
+            try:
+                tables_to_restore = restore.get_tables_from_backup(
+                    database, 
+                    target_label, 
+                    group=group, 
+                    table=table, 
+                    database=cfg['database'] if table else None
+                )
+            except ValueError as e:
+                logger.error(str(e))
+                sys.exit(1)
             
             if not tables_to_restore:
                 if group:
                     logger.warning(f"No tables found in backup '{target_label}' for group '{group}'")
+                elif table:
+                    logger.warning(f"No tables found in backup '{target_label}' for table '{table}'")
                 else:
                     logger.warning(f"No tables found in backup '{target_label}'")
                 sys.exit(1)
@@ -359,7 +451,8 @@ def restore_command(config, target_label, group, rename_suffix):
                 cfg['repository'],
                 restore_pair,
                 tables_to_restore,
-                rename_suffix
+                rename_suffix,
+                skip_confirmation=yes
             )
             
             if result['success']:
@@ -375,6 +468,9 @@ def restore_command(config, target_label, group, rename_suffix):
     except ValueError as e:
         logger.error(f"Configuration error: {e}")
         sys.exit(1)
+    except RuntimeError as e:
+        logger.error(f"{e}")
+        sys.exit(1)
     except Exception as e:
         logger.error(f"Unexpected error: {e}")
         sys.exit(1)

starrocks_br-0.2.0/src/starrocks_br/config.py

@@ -0,0 +1,64 @@
+import yaml
+from typing import Any, Dict, Optional
+
+
+def load_config(config_path: str) -> Dict[str, Any]:
+    """Load and parse YAML configuration file.
+    
+    Args:
+        config_path: Path to the YAML config file
+        
+    Returns:
+        Dictionary containing configuration
+        
+    Raises:
+        FileNotFoundError: If config file doesn't exist
+        yaml.YAMLError: If config file is not valid YAML
+    """
+    with open(config_path, 'r') as f:
+        config = yaml.safe_load(f)
+    
+    if not isinstance(config, dict):
+        raise ValueError("Config must be a dictionary")
+    
+    return config
+
+
+def validate_config(config: Dict[str, Any]) -> None:
+    """Validate that config contains required fields.
+    
+    Args:
+        config: Configuration dictionary
+        
+    Raises:
+        ValueError: If required fields are missing
+    """
+    required_fields = ['host', 'port', 'user', 'database', 'repository']
+    
+    for field in required_fields:
+        if field not in config:
+            raise ValueError(f"Missing required config field: {field}")
+
+    _validate_tls_section(config.get('tls'))
+
+
+def _validate_tls_section(tls_config) -> None:
+    if tls_config is None:
+        return
+
+    if not isinstance(tls_config, dict):
+        raise ValueError("TLS configuration must be a dictionary")
+
+    enabled = bool(tls_config.get('enabled', False))
+
+    if enabled and not tls_config.get('ca_cert'):
+        raise ValueError("TLS configuration requires 'ca_cert' when 'enabled' is true")
+
+    if 'verify_server_cert' in tls_config and not isinstance(tls_config['verify_server_cert'], bool):
+        raise ValueError("TLS configuration field 'verify_server_cert' must be a boolean if provided")
+
+    if 'tls_versions' in tls_config:
+        tls_versions = tls_config['tls_versions']
+        if not isinstance(tls_versions, list) or not all(isinstance(version, str) for version in tls_versions):
+            raise ValueError("TLS configuration field 'tls_versions' must be a list of strings if provided")
+