dtaas-services 0.3.0__tar.gz

dtaas_services-0.3.0/PKG-INFO

@@ -0,0 +1,347 @@
+Metadata-Version: 2.3
+Name: dtaas-services
+Version: 0.3.0
+Summary: DTaaS Platform Services Management CLI
+License: INTO-CPS-Association
+Author: Mohamed Abdulkarim
+Requires-Python: >=3.10,<4.0
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: click (>=8.1.7,<9.0.0)
+Requires-Dist: httpx (>=0.28.1,<0.29.0)
+Requires-Dist: python-dotenv (>=1.0.1,<2.0.0)
+Requires-Dist: python-gitlab (>=8.1.0,<9.0.0)
+Requires-Dist: python-on-whales (>=0.75.0,<0.76.0)
+Requires-Dist: rich (>=14.2.0,<15.0.0)
+Description-Content-Type: text/markdown
+
+# Platform Services
+
+It is recommended to install certain third-party software for use
+by digital twins running inside the DTaaS software.
+_These services can only be installed in secure (TLS) mode._
+
+The following services can be installed:
+
+* **Influx** time-series database and dashboard service
+* **Grafana** visualization and dashboard service
+* **RabbitMQ** AMQP broker and its' management interface
+  The **MQTT plugin** of this broker has been enabled.
+  So, it can also be used as **MQTT** broker.
+* **MongoDB** database server
+* **ThingsBoard** IoT device management and data visualization platform
+(with PostgreSQL backend)
+* **GitLab** OAuth2 identity provider and Git repository service
+
+## Directory Structure
+
+* **config** is used for storing the service configuration
+* **data** is used by the services for storing data
+* **certs** is used for storing the TLS certificates needed by the services.
+* **cli** contains a CLI package for automated service management
+
+## Installation Methods
+
+You can install and manage the services using:
+
+**CLI Package :** Automated CLI tool for easy service management.
+
+## DTaaS Services CLI
+
+A command-line tool for managing DTaaS platform services including MongoDB,
+InfluxDB, RabbitMQ, Grafana, ThingsBoard, and GitLab.
+
+## Features
+
+* **Project Initialization:** Generate project structure with config and data directories
+* **Automated Setup:** One command setup of TLS certificates and permissions
+* **Service Management:** Start, stop, and check status of all services
+* **User Management:** Easy creation of user accounts in InfluxDB,
+RabbitMQ, ThingsBoard, and GitLab
+* **Cross platform:** Works on Linux, macOS, and Windows
+* **Configuration-driven:** Reads settings from `config/services.env`
+
+## Installation
+
+### Prerequisites
+
+* Python 3.10 or higher
+* Docker and Docker Compose
+* TLS certificates
+
+### Install from Wheel Package
+
+Install the standalone wheel package using pip:
+
+```bash
+pip install dtaas_services-0.3.0-py3-none-any.whl
+```
+
+This installs the `dtaas-services` command.
+
+To verify the installation:
+
+```bash
+dtaas-services --help
+```
+
+## Quick Start
+
+1. Navigate to where you want to set up the services and generate the project
+structure and run:
+
+   ```bash
+   dtaas-services generate-project
+   ```
+
+   This creates:
+   * `config/` directory with configuration templates
+   * `data/` directory for service data
+   * `compose.services.yml` for main services
+   * `compose.thingsboard.yml` for ThingsBoard and PostgreSQL
+
+2. Update `config/services.env` with your environment values:
+   * `HOSTNAME`: Public hostname of the DTaaS server (used for certificate paths)
+   * `GITLAB_PORT`: Port for the local GitLab instance (default: `8090`)
+   * `GITLAB_ROOT_NEW_PASSWORD`: Strong password to set for the GitLab root admin
+   * `TB_SYSADMIN_NEW_PASSWORD`: New password for the ThingsBoard system admin
+   * Port numbers for each service
+   * `TB_TENANT_NEW_PASSWORD`: New password for the tenant admin
+   * `SERVICES_UID`: User ID for service file ownership
+   * `SERVICES_GID`: Group ID for service file ownership
+
+3. Update `config/credentials.csv` with user accounts (format: `username,password,email`)
+
+**Options:**
+
+* `--path` Directory to generate project structure (default: current directory)
+
+**Example:**
+
+```bash
+dtaas-services generate-project --path /path/to/project
+```
+
+## Usage
+
+### Service Setup
+
+After generating the project and configuring your settings:
+
+```bash
+dtaas-services setup
+```
+
+This command will:
+
+* Copy TLS certificates to the correct locations
+* Set up MongoDB certificates and permissions
+* Set up InfluxDB certificates and permissions
+* Set up RabbitMQ certificates and permissions
+* Set up PostgreSQL and ThingsBoard certificates and permissions
+
+Make sure you run the clean command right after
+
+```bash
+dtaas-services clean
+```
+
+### Managing Services
+
+Now you can manage services by simple commands (Start, Stop, Remove and Restart)
+
+Example:
+
+```bash
+dtaas-services start
+```
+
+you can specify a services to manage
+
+```bash
+dtaas-services stop -s influxdb
+```
+
+Remove services and their volumes:
+
+```bash
+dtaas-services remove -v
+# Specify
+dtaas-services remove -v -s <service_name>
+
+```
+
+### User Account Management
+
+1. Edit `config/credentials.csv` with user accounts (format: `username,password,email`)
+
+2. Add users to all supported services:
+
+   ```bash
+   dtaas-services user add
+   ```
+
+   This creates user accounts in InfluxDB, RabbitMQ, ThingsBoard, and GitLab
+   (each service is skipped gracefully if it is not running).
+   For GitLab, a Personal Access Token is created for each new user and
+   saved to `config/gitlab_user_tokens.json`.
+
+3. Add users to a specific service:
+
+   ```bash
+   dtaas-services user add -s rabbitmq
+   ```
+
+## ThingsBoard
+
+It is recommended to install the third-party software ThingsBoard
+for use by digital twins
+running inside the DTaaS software.
+This service can only be installed in secure (TLS) mode.
+
+The steps given above install two services:
+
+* **ThingsBoard** is an IoT device management and data visualization platform
+* **PostgreSQL** is a database server for ThingsBoard
+
+### ThingsBoard Directory Structure
+
+* **config** is used for storing the service configuration
+* **data** is used by the services for storing data
+* **log** is used by the services for logging
+* **certs** is used for storing the TLS certificates needed by the services
+* **script** contains scripts for creating user accounts and service management
+
+### ThingsBoard Installation
+
+> **Warning:** Running `dtaas-services install -s thingsboard` more than once
+> will re-run the ThingsBoard schema migration against an already-populated
+> PostgreSQL database, which can corrupt it. If you need to reinstall from
+> scratch, run `dtaas-services clean -s "postgres,thingsboard"` first to wipe
+> all data before re-running the install command.
+> **Note:** It's recommended to specify the service explicitly with `-s <service>`
+> when installing.
+> Make sure that you have runned the clean command before the installation
+
+```bash
+#  (It starts PostgreSQL if it's not running, and it checks its health)
+dtaas-services install -s thingsboard
+```
+
+```bash
+dtaas-services start -s thingsboard
+```
+
+```bash
+
+#  After installation, wait some time before adding users
+#  This creates the tenant, tenant admin and users.
+dtaas-services user add -s thingsboard
+```
+
+Reset the ThingsBoard sysadmin and tenant admin passwords using values
+configured in `config/services.env`:
+
+```bash
+dtaas-services user reset-password -s thingsboard
+```
+
+This command:
+
+* Changes the sysadmin password from the default (`"sysadmin"`) to `TB_SYSADMIN_NEW_PASSWORD`
+* Changes the tenant admin password from the default (`"tenant"`) to
+  `TB_TENANT_ADMIN_PASSWORD`
+
+### GitLab Installation
+
+**Prerequisites:**
+
+* The GitLab container joins the `dtaas-services` Docker network
+  (`platform-services`), which is created automatically when the other
+  platform services are running. Start them first with
+  `dtaas-services start` before installing GitLab.
+* Set `REACT_APP_AUTH_AUTHORITY` in the client config file
+  (`deploy/config/client/env.js` for server deployments, or
+  `deploy/config/client/env.local.js` for localhost) to
+  `https://<hostname>:<GITLAB_PORT>/gitlab`.
+
+> **Note:** The DTaaS client uses `react-oidc-context`, which forces
+> redirects to use HTTPS. GitLab must therefore be served over HTTPS
+> at `https://<hostname>:<GITLAB_PORT>/gitlab`
+
+To install and configure the local GitLab instance:
+
+```bash
+dtaas-services install -s gitlab
+```
+
+GitLab takes 5–10 minutes to become healthy after the first start.
+The install command checks GitLab’s readiness and returns immediately:
+
+* **If GitLab is healthy**: the command runs post-install setup
+  (password reset, PAT creation, OAuth app registration) and
+  saves the access token to `config/gitlab_tokens.json`.
+* **If GitLab is still starting**: the command prints a status hint
+  and exits. Check progress with `dtaas-services status -s gitlab`
+  and re-run `dtaas-services install -s gitlab` once the status
+  shows "healthy".
+
+> **Note:** After a successful setup, `config/gitlab_tokens.json` is backed
+> up to `config/backup_gitlab_tokens.json` and the `root_password` entry is
+> removed from the live tokens file.
+> If the GitLab installation becomes corrupted before the root password is
+> changed, the initial password can be found in the backup file:
+> `config/backup_gitlab_tokens.json`.
+> If that file is also missing, re-install GitLab to generate a new
+> initial password.
+> **Warning:** `config/password.env.current` is managed automatically by
+> the CLI and tracks the current service passwords. Do **not** edit or delete
+> this file manually — doing so may cause password reset commands to fail.
+
+To complete the OAuth2 integration with DTaaS and set up GitLab Runner,
+follow the [integration guide](GITLAB_INTEGRATION.md) and
+[runner setup guide](../runner/GITLAB-RUNNER.md).
+
+Reset the GitLab root admin password using the value configured in
+`config/services.env` (`GITLAB_ROOT_NEW_PASSWORD`):
+
+```bash
+dtaas-services user reset-password -s gitlab
+```
+
+The command reads the new password from `GITLAB_ROOT_NEW_PASSWORD`
+and applies it via the GitLab API.
+
+## GitLab Post-Install Flow
+
+The `dtaas-services install -s gitlab` command performs the following steps
+automatically:
+
+1. Starts the GitLab Docker container
+2. Checks if GitLab is healthy (non-blocking — exits immediately if not ready)
+3. Reads the auto-generated root password from the container
+   (`/etc/gitlab/initial_root_password`)
+4. Creates an initial Personal Access Token and saves it to
+   `config/gitlab_tokens.json`
+5. Creates Server and Client OAuth application tokens.
+
+## Troubleshooting
+
+### Permission Issues (Linux/macOS)
+
+If you encounter permission errors when setting up services,
+ensure you run the setup command with appropriate privileges:
+
+```bash
+sudo -E env PATH="$PATH" dtaas-services setup
+```
+
+### Thingsboard connection error
+
+After starting thingsboard and before adding users or changing passwords,
+it needs some time to initialize then you can add users.
+

dtaas_services-0.3.0/README.md

@@ -0,0 +1,325 @@
+# Platform Services
+
+It is recommended to install certain third-party software for use
+by digital twins running inside the DTaaS software.
+_These services can only be installed in secure (TLS) mode._
+
+The following services can be installed:
+
+* **Influx** time-series database and dashboard service
+* **Grafana** visualization and dashboard service
+* **RabbitMQ** AMQP broker and its' management interface
+  The **MQTT plugin** of this broker has been enabled.
+  So, it can also be used as **MQTT** broker.
+* **MongoDB** database server
+* **ThingsBoard** IoT device management and data visualization platform
+(with PostgreSQL backend)
+* **GitLab** OAuth2 identity provider and Git repository service
+
+## Directory Structure
+
+* **config** is used for storing the service configuration
+* **data** is used by the services for storing data
+* **certs** is used for storing the TLS certificates needed by the services.
+* **cli** contains a CLI package for automated service management
+
+## Installation Methods
+
+You can install and manage the services using:
+
+**CLI Package :** Automated CLI tool for easy service management.
+
+## DTaaS Services CLI
+
+A command-line tool for managing DTaaS platform services including MongoDB,
+InfluxDB, RabbitMQ, Grafana, ThingsBoard, and GitLab.
+
+## Features
+
+* **Project Initialization:** Generate project structure with config and data directories
+* **Automated Setup:** One command setup of TLS certificates and permissions
+* **Service Management:** Start, stop, and check status of all services
+* **User Management:** Easy creation of user accounts in InfluxDB,
+RabbitMQ, ThingsBoard, and GitLab
+* **Cross platform:** Works on Linux, macOS, and Windows
+* **Configuration-driven:** Reads settings from `config/services.env`
+
+## Installation
+
+### Prerequisites
+
+* Python 3.10 or higher
+* Docker and Docker Compose
+* TLS certificates
+
+### Install from Wheel Package
+
+Install the standalone wheel package using pip:
+
+```bash
+pip install dtaas_services-0.3.0-py3-none-any.whl
+```
+
+This installs the `dtaas-services` command.
+
+To verify the installation:
+
+```bash
+dtaas-services --help
+```
+
+## Quick Start
+
+1. Navigate to where you want to set up the services and generate the project
+structure and run:
+
+   ```bash
+   dtaas-services generate-project
+   ```
+
+   This creates:
+   * `config/` directory with configuration templates
+   * `data/` directory for service data
+   * `compose.services.yml` for main services
+   * `compose.thingsboard.yml` for ThingsBoard and PostgreSQL
+
+2. Update `config/services.env` with your environment values:
+   * `HOSTNAME`: Public hostname of the DTaaS server (used for certificate paths)
+   * `GITLAB_PORT`: Port for the local GitLab instance (default: `8090`)
+   * `GITLAB_ROOT_NEW_PASSWORD`: Strong password to set for the GitLab root admin
+   * `TB_SYSADMIN_NEW_PASSWORD`: New password for the ThingsBoard system admin
+   * Port numbers for each service
+   * `TB_TENANT_NEW_PASSWORD`: New password for the tenant admin
+   * `SERVICES_UID`: User ID for service file ownership
+   * `SERVICES_GID`: Group ID for service file ownership
+
+3. Update `config/credentials.csv` with user accounts (format: `username,password,email`)
+
+**Options:**
+
+* `--path` Directory to generate project structure (default: current directory)
+
+**Example:**
+
+```bash
+dtaas-services generate-project --path /path/to/project
+```
+
+## Usage
+
+### Service Setup
+
+After generating the project and configuring your settings:
+
+```bash
+dtaas-services setup
+```
+
+This command will:
+
+* Copy TLS certificates to the correct locations
+* Set up MongoDB certificates and permissions
+* Set up InfluxDB certificates and permissions
+* Set up RabbitMQ certificates and permissions
+* Set up PostgreSQL and ThingsBoard certificates and permissions
+
+Make sure you run the clean command right after
+
+```bash
+dtaas-services clean
+```
+
+### Managing Services
+
+Now you can manage services by simple commands (Start, Stop, Remove and Restart)
+
+Example:
+
+```bash
+dtaas-services start
+```
+
+you can specify a services to manage
+
+```bash
+dtaas-services stop -s influxdb
+```
+
+Remove services and their volumes:
+
+```bash
+dtaas-services remove -v
+# Specify
+dtaas-services remove -v -s <service_name>
+
+```
+
+### User Account Management
+
+1. Edit `config/credentials.csv` with user accounts (format: `username,password,email`)
+
+2. Add users to all supported services:
+
+   ```bash
+   dtaas-services user add
+   ```
+
+   This creates user accounts in InfluxDB, RabbitMQ, ThingsBoard, and GitLab
+   (each service is skipped gracefully if it is not running).
+   For GitLab, a Personal Access Token is created for each new user and
+   saved to `config/gitlab_user_tokens.json`.
+
+3. Add users to a specific service:
+
+   ```bash
+   dtaas-services user add -s rabbitmq
+   ```
+
+## ThingsBoard
+
+It is recommended to install the third-party software ThingsBoard
+for use by digital twins
+running inside the DTaaS software.
+This service can only be installed in secure (TLS) mode.
+
+The steps given above install two services:
+
+* **ThingsBoard** is an IoT device management and data visualization platform
+* **PostgreSQL** is a database server for ThingsBoard
+
+### ThingsBoard Directory Structure
+
+* **config** is used for storing the service configuration
+* **data** is used by the services for storing data
+* **log** is used by the services for logging
+* **certs** is used for storing the TLS certificates needed by the services
+* **script** contains scripts for creating user accounts and service management
+
+### ThingsBoard Installation
+
+> **Warning:** Running `dtaas-services install -s thingsboard` more than once
+> will re-run the ThingsBoard schema migration against an already-populated
+> PostgreSQL database, which can corrupt it. If you need to reinstall from
+> scratch, run `dtaas-services clean -s "postgres,thingsboard"` first to wipe
+> all data before re-running the install command.
+> **Note:** It's recommended to specify the service explicitly with `-s <service>`
+> when installing.
+> Make sure that you have runned the clean command before the installation
+
+```bash
+#  (It starts PostgreSQL if it's not running, and it checks its health)
+dtaas-services install -s thingsboard
+```
+
+```bash
+dtaas-services start -s thingsboard
+```
+
+```bash
+
+#  After installation, wait some time before adding users
+#  This creates the tenant, tenant admin and users.
+dtaas-services user add -s thingsboard
+```
+
+Reset the ThingsBoard sysadmin and tenant admin passwords using values
+configured in `config/services.env`:
+
+```bash
+dtaas-services user reset-password -s thingsboard
+```
+
+This command:
+
+* Changes the sysadmin password from the default (`"sysadmin"`) to `TB_SYSADMIN_NEW_PASSWORD`
+* Changes the tenant admin password from the default (`"tenant"`) to
+  `TB_TENANT_ADMIN_PASSWORD`
+
+### GitLab Installation
+
+**Prerequisites:**
+
+* The GitLab container joins the `dtaas-services` Docker network
+  (`platform-services`), which is created automatically when the other
+  platform services are running. Start them first with
+  `dtaas-services start` before installing GitLab.
+* Set `REACT_APP_AUTH_AUTHORITY` in the client config file
+  (`deploy/config/client/env.js` for server deployments, or
+  `deploy/config/client/env.local.js` for localhost) to
+  `https://<hostname>:<GITLAB_PORT>/gitlab`.
+
+> **Note:** The DTaaS client uses `react-oidc-context`, which forces
+> redirects to use HTTPS. GitLab must therefore be served over HTTPS
+> at `https://<hostname>:<GITLAB_PORT>/gitlab`
+
+To install and configure the local GitLab instance:
+
+```bash
+dtaas-services install -s gitlab
+```
+
+GitLab takes 5–10 minutes to become healthy after the first start.
+The install command checks GitLab’s readiness and returns immediately:
+
+* **If GitLab is healthy**: the command runs post-install setup
+  (password reset, PAT creation, OAuth app registration) and
+  saves the access token to `config/gitlab_tokens.json`.
+* **If GitLab is still starting**: the command prints a status hint
+  and exits. Check progress with `dtaas-services status -s gitlab`
+  and re-run `dtaas-services install -s gitlab` once the status
+  shows "healthy".
+
+> **Note:** After a successful setup, `config/gitlab_tokens.json` is backed
+> up to `config/backup_gitlab_tokens.json` and the `root_password` entry is
+> removed from the live tokens file.
+> If the GitLab installation becomes corrupted before the root password is
+> changed, the initial password can be found in the backup file:
+> `config/backup_gitlab_tokens.json`.
+> If that file is also missing, re-install GitLab to generate a new
+> initial password.
+> **Warning:** `config/password.env.current` is managed automatically by
+> the CLI and tracks the current service passwords. Do **not** edit or delete
+> this file manually — doing so may cause password reset commands to fail.
+
+To complete the OAuth2 integration with DTaaS and set up GitLab Runner,
+follow the [integration guide](GITLAB_INTEGRATION.md) and
+[runner setup guide](../runner/GITLAB-RUNNER.md).
+
+Reset the GitLab root admin password using the value configured in
+`config/services.env` (`GITLAB_ROOT_NEW_PASSWORD`):
+
+```bash
+dtaas-services user reset-password -s gitlab
+```
+
+The command reads the new password from `GITLAB_ROOT_NEW_PASSWORD`
+and applies it via the GitLab API.
+
+## GitLab Post-Install Flow
+
+The `dtaas-services install -s gitlab` command performs the following steps
+automatically:
+
+1. Starts the GitLab Docker container
+2. Checks if GitLab is healthy (non-blocking — exits immediately if not ready)
+3. Reads the auto-generated root password from the container
+   (`/etc/gitlab/initial_root_password`)
+4. Creates an initial Personal Access Token and saves it to
+   `config/gitlab_tokens.json`
+5. Creates Server and Client OAuth application tokens.
+
+## Troubleshooting
+
+### Permission Issues (Linux/macOS)
+
+If you encounter permission errors when setting up services,
+ensure you run the setup command with appropriate privileges:
+
+```bash
+sudo -E env PATH="$PATH" dtaas-services setup
+```
+
+### Thingsboard connection error
+
+After starting thingsboard and before adding users or changing passwords,
+it needs some time to initialize then you can add users.

dtaas_services-0.3.0/dtaas_services/cmd.py

@@ -0,0 +1,37 @@
+"""DTaaS Services CLI main entry point."""
+
+import click
+from .commands import service_ops, setup_ops, user_ops
+
+
+@click.group()
+def services():
+    """Manage DTaaS platform services."""
+
+
+# Register setup and installation commands
+services.add_command(setup_ops.generate_project, name="generate-project")
+services.add_command(setup_ops.setup)
+services.add_command(setup_ops.install)
+
+# Register service operational commands
+services.add_command(service_ops.start)
+services.add_command(service_ops.stop)
+services.add_command(service_ops.restart)
+services.add_command(service_ops.status)
+services.add_command(service_ops.remove)
+services.add_command(service_ops.clean)
+
+
+# Register user management command group
+@services.group()
+def user():
+    """User account management for services."""
+
+
+user.add_command(user_ops.add)
+user.add_command(user_ops.reset_password)
+
+
+if __name__ == "__main__":
+    services()

dtaas_services-0.3.0/dtaas_services/commands/__init__.py

@@ -0,0 +1 @@
+"""DTaaS Services CLI command modules."""