fabricks 2024.7.1.5__tar.gz

fabricks-2024.7.1.5/PKG-INFO

@@ -0,0 +1,212 @@
+Metadata-Version: 2.1
+Name: fabricks
+Version: 2024.7.1.5
+Summary: 
+Author: BMS DWH Team
+Author-email: bi_support@bmsuisse.ch
+Requires-Python: >=3.9,<4
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: azure-data-tables (>=12.5.0,<13.0.0)
+Requires-Dist: azure-identity (>=1.10.0)
+Requires-Dist: azure-storage-blob (>=12.14.1)
+Requires-Dist: azure-storage-queue (>=12.10.0,<13.0.0)
+Requires-Dist: databricks-sdk (>=0.29.0)
+Requires-Dist: jinja2 (>=2.11.3)
+Requires-Dist: python-dotenv (>=1.0.1)
+Requires-Dist: sqlglot (>=22.1.1)
+Description-Content-Type: text/markdown
+
+# Fabricks
+
+Fabricks is a Python framework developed to help create a lake house in Databricks. It simplifies the process of building and maintaining data pipelines by providing a standardized approach to defining and managing data processing workflows.
+
+Though Fabricks is currently really meant to be run on Databricks, the code using Fabricks is really portable - you'll almost exclusively
+write SQL-Select Code - no need to manually write DDL/DML/Merge queries. Later on we might add support for other platforms as well, eg. DuckDB or Open Source Spark.
+
+
+
+## Features
+
+- YAML configuration files: Fabricks uses YAML files for configuration, making it easy to define and modify workflows without requiring significant changes to the code.
+- SQL files for business logic: Business logic is defined in SQL files, providing a familiar and powerful tool for data processing.
+- Version control: Fabricks supports version control, ensuring that changes are tracked and can be rolled back if necessary.
+- Seamless integration of new data sources: Fabricks can easily integrate new data sources into existing workflows.
+- Change Data Capture: Fabricks supports Change Data Capture, allowing it to track and handle changes in the data over time.
+- Drop and create: Fabricks can drop and create tables as needed, providing flexibility in managing the data schema.
+
+## Getting Started
+
+To get started with Fabricks, you'll need to install it and set up your first project. Here's a basic guide on how to do that:
+
+### Installation
+
+To install Fabricks, you need to install the library on your Databricks cluster. Follow the steps below:
+
+1. Navigate to your Databricks workspace.
+2. Select the cluster where you want to install the library.
+3. Click on the `Libraries` tab.
+4. Click on `Install New`.
+5. Choose `PyPI` from the library source dropdown.
+6. Enter `fabricks` in the package text box.
+7. Click `Install`.
+
+After the library is installed, you can import it in your notebooks or scripts using `import fabricks`.
+
+### Setting Up Your First Project
+
+# Fabricks Runtime Configuration
+
+The Fabricks runtime configuration is defined in a YAML file. This file specifies the settings for the Fabricks runtime, including options for the runtime environment, path options, Spark options, and the configuration for different stages of the data pipeline (bronze, silver, gold, etc.).
+
+A sample can be found in the [tests](tests/runtime/fabricks/conf.5589296195699698.yml)
+
+## Configuration Options
+
+- `name`: The name of the configuration.
+- `options`: General options for the runtime. This includes:
+  - `secret_scope`: The name of the secret scope in Databricks.
+  - `timeout`: The timeout for the runtime in seconds.
+  - `workers`: The number of workers for the runtime.
+- `path_options`: Options for the storage path. This includes:
+  - `storage`: The storage path for the data.
+- `spark_options`: Options for Spark. This includes:
+  - `sql`: SQL options for Spark.
+
+## Data Pipeline Stages
+
+The configuration file defines the settings for different stages of the data pipeline:
+
+- `bronze`: The initial stage of the data pipeline. This includes:
+  - `name`: The name of the stage.
+  - `path_options`: Options for the storage path.
+  - `options`: Options for the stage.
+  
+  For some samples, see in the [tests/runtime/bronze Folder](tests/runtime/bronze)
+- `silver`: The intermediate stage of the data pipeline. This includes:
+  - `name`: The name of the stage.
+  - `path_options`: Options for the storage path.
+  - `options`: Options for the stage.
+
+  For some samples, see in the [tests/runtime/silver Folder](tests/runtime/silver)
+- `gold`: The final stage of the data pipeline. This includes:
+  - `name`: The name of the stage.
+  - `path_options`: Options for the storage path.
+  - `options`: Options for the stage.
+
+  
+  For some samples, see in the [tests/runtime/gold Folder](tests/runtime/gold)
+
+The folder names and the stage names can be configures in the main Fabricks config, you don't have t o stick with the defaults
+
+## Other Configurations
+
+- `powerbi`: Configuration for PowerBI integration.
+- `databases`: Configuration for the databases.
+- `credentials`: Credentials for accessing different resources.
+- `variables`: Variables used in the configuration.
+
+Please note that this is a basic documentation based on the provided YAML file. The actual configuration options may vary depending on the specific requirements of your project.
+
+# Bronze Step Configuration
+
+The "bronze" step in Fabricks is the initial stage of the data pipeline. It is responsible for ingesting raw data and storing it in a "bronze" table for further processing. The configuration for the "bronze" step is defined in a YAML file. Each job in the "bronze" step has the following configuration options:
+
+- `step`: The step in the data pipeline. For these jobs, it is always "bronze".
+- `topic`: The topic of the job. This is usually the name of the data source.
+- `item`: The item of the job. This is usually the name of the specific data item being processed.
+- `tags`: Tags for the job. These can be used to categorize or filter jobs.
+- `options`: Options for the job. This includes:
+  - `mode`: The mode of the job. This can be "append", "memory", or "register".
+  - `uri`: The URI of the data source. This is usually an Azure Blob File System (ABFS) URI.
+  - `parser`: The parser to use for the data. This can be "monarch", "parquet", etc.
+  - `keys`: The keys for the data. These are the columns that uniquely identify each row in the data.
+  - `source`: The source of the data. This is usually the same as the topic.
+  - `extender`: The extender for the data. This is used to extend the data with additional columns or transformations.
+  - `encrypted_columns`: The columns in the data that are encrypted. These columns will be decrypted during the "bronze" step.
+  - `calculated_columns`: The columns in the data that are calculated. These columns will be calculated during the "bronze" step.
+
+Here's an example of a "bronze" step job:
+
+```yaml
+- job:
+    step: bronze
+    topic: king
+    item: scd1
+    tags: [test]
+    options:
+      mode: append
+      uri: abfss://fabricks@$datahub/raw/king
+      parser: monarch
+      keys: [id]
+      source: king
+```
+
+# Silver Step Configuration
+
+The "silver" step in Fabricks is the intermediate stage of the data pipeline. It is responsible for processing the raw data ingested in the "bronze" step and storing it in a "silver" table for further processing. The configuration for the "silver" step is defined in a YAML file. Each job in the "silver" step has the following configuration options:
+
+- `step`: The step in the data pipeline. For these jobs, it is always "silver".
+- `topic`: The topic of the job. This is usually the name of the data source.
+- `item`: The item of the job. This is usually the name of the specific data item being processed.
+- `tags`: Tags for the job. These can be used to categorize or filter jobs.
+- `options`: Options for the job. This includes:
+  - `mode`: The mode of the job. This can be "update", "memory", "latest", "append", "combine", etc.
+  - `change_data_capture`: The type of Change Data Capture (CDC) to use. This can be "scd1", "scd2", "nocdc", etc.
+  - `parents`: The parent jobs that this job depends on. These are usually "bronze" step jobs.
+  - `extender`: The extender for the data. This is used to extend the data with additional columns or transformations.
+  - `order_duplicate_by`: The order to use when removing duplicates. This can be "asc" or "desc".
+  - `check_options`: Options for checking the data. This includes:
+    - `max_rows`: The maximum number of rows to check.
+  - `stream`: Whether to stream the data. This can be "true" or "false".
+
+Here's an example of a "silver" step job:
+
+```yaml
+- job:
+    step: silver
+    topic: king_and_queen
+    item: scd1
+    tags: [test]
+    options:
+      mode: update
+      change_data_capture: scd1
+      parents: [bronze.queen_scd1, bronze.king_scd1]
+```
+# Gold Step Configuration
+
+The "gold" step in Fabricks is the final stage of the data pipeline. It is responsible for processing the data from the "silver" step and storing it in a "gold" table for consumption. The configuration for the "gold" step is defined in a YAML file. Each job in the "gold" step has the following configuration options:
+
+- `step`: The step in the data pipeline. For these jobs, it is always "gold".
+- `topic`: The topic of the job. This is usually the name of the data source.
+- `item`: The item of the job. This is usually the name of the specific data item being processed.
+- `tags`: Tags for the job. These can be used to categorize or filter jobs.
+- `options`: Options for the job. This includes:
+  - `mode`: The mode of the job. This can be "complete", "memory", etc.
+  - `change_data_capture`: The type of Change Data Capture (CDC) to use. This can be "scd1", "scd2", "nocdc", etc.
+
+Here's an example of a "gold" step job:
+
+```yaml
+- job:
+    step: gold
+    topic: scd2
+    item: complete
+    tags: [test]
+    options:
+      change_data_capture: scd2
+      mode: complete
+````
+
+## Usage
+
+// Instructions on how to use the framework go here
+
+
+## License
+
+This project is licensed under the terms of the MIT license.
+

fabricks-2024.7.1.5/README.md

@@ -0,0 +1,189 @@
+# Fabricks
+
+Fabricks is a Python framework developed to help create a lake house in Databricks. It simplifies the process of building and maintaining data pipelines by providing a standardized approach to defining and managing data processing workflows.
+
+Though Fabricks is currently really meant to be run on Databricks, the code using Fabricks is really portable - you'll almost exclusively
+write SQL-Select Code - no need to manually write DDL/DML/Merge queries. Later on we might add support for other platforms as well, eg. DuckDB or Open Source Spark.
+
+
+
+## Features
+
+- YAML configuration files: Fabricks uses YAML files for configuration, making it easy to define and modify workflows without requiring significant changes to the code.
+- SQL files for business logic: Business logic is defined in SQL files, providing a familiar and powerful tool for data processing.
+- Version control: Fabricks supports version control, ensuring that changes are tracked and can be rolled back if necessary.
+- Seamless integration of new data sources: Fabricks can easily integrate new data sources into existing workflows.
+- Change Data Capture: Fabricks supports Change Data Capture, allowing it to track and handle changes in the data over time.
+- Drop and create: Fabricks can drop and create tables as needed, providing flexibility in managing the data schema.
+
+## Getting Started
+
+To get started with Fabricks, you'll need to install it and set up your first project. Here's a basic guide on how to do that:
+
+### Installation
+
+To install Fabricks, you need to install the library on your Databricks cluster. Follow the steps below:
+
+1. Navigate to your Databricks workspace.
+2. Select the cluster where you want to install the library.
+3. Click on the `Libraries` tab.
+4. Click on `Install New`.
+5. Choose `PyPI` from the library source dropdown.
+6. Enter `fabricks` in the package text box.
+7. Click `Install`.
+
+After the library is installed, you can import it in your notebooks or scripts using `import fabricks`.
+
+### Setting Up Your First Project
+
+# Fabricks Runtime Configuration
+
+The Fabricks runtime configuration is defined in a YAML file. This file specifies the settings for the Fabricks runtime, including options for the runtime environment, path options, Spark options, and the configuration for different stages of the data pipeline (bronze, silver, gold, etc.).
+
+A sample can be found in the [tests](tests/runtime/fabricks/conf.5589296195699698.yml)
+
+## Configuration Options
+
+- `name`: The name of the configuration.
+- `options`: General options for the runtime. This includes:
+  - `secret_scope`: The name of the secret scope in Databricks.
+  - `timeout`: The timeout for the runtime in seconds.
+  - `workers`: The number of workers for the runtime.
+- `path_options`: Options for the storage path. This includes:
+  - `storage`: The storage path for the data.
+- `spark_options`: Options for Spark. This includes:
+  - `sql`: SQL options for Spark.
+
+## Data Pipeline Stages
+
+The configuration file defines the settings for different stages of the data pipeline:
+
+- `bronze`: The initial stage of the data pipeline. This includes:
+  - `name`: The name of the stage.
+  - `path_options`: Options for the storage path.
+  - `options`: Options for the stage.
+  
+  For some samples, see in the [tests/runtime/bronze Folder](tests/runtime/bronze)
+- `silver`: The intermediate stage of the data pipeline. This includes:
+  - `name`: The name of the stage.
+  - `path_options`: Options for the storage path.
+  - `options`: Options for the stage.
+
+  For some samples, see in the [tests/runtime/silver Folder](tests/runtime/silver)
+- `gold`: The final stage of the data pipeline. This includes:
+  - `name`: The name of the stage.
+  - `path_options`: Options for the storage path.
+  - `options`: Options for the stage.
+
+  
+  For some samples, see in the [tests/runtime/gold Folder](tests/runtime/gold)
+
+The folder names and the stage names can be configures in the main Fabricks config, you don't have t o stick with the defaults
+
+## Other Configurations
+
+- `powerbi`: Configuration for PowerBI integration.
+- `databases`: Configuration for the databases.
+- `credentials`: Credentials for accessing different resources.
+- `variables`: Variables used in the configuration.
+
+Please note that this is a basic documentation based on the provided YAML file. The actual configuration options may vary depending on the specific requirements of your project.
+
+# Bronze Step Configuration
+
+The "bronze" step in Fabricks is the initial stage of the data pipeline. It is responsible for ingesting raw data and storing it in a "bronze" table for further processing. The configuration for the "bronze" step is defined in a YAML file. Each job in the "bronze" step has the following configuration options:
+
+- `step`: The step in the data pipeline. For these jobs, it is always "bronze".
+- `topic`: The topic of the job. This is usually the name of the data source.
+- `item`: The item of the job. This is usually the name of the specific data item being processed.
+- `tags`: Tags for the job. These can be used to categorize or filter jobs.
+- `options`: Options for the job. This includes:
+  - `mode`: The mode of the job. This can be "append", "memory", or "register".
+  - `uri`: The URI of the data source. This is usually an Azure Blob File System (ABFS) URI.
+  - `parser`: The parser to use for the data. This can be "monarch", "parquet", etc.
+  - `keys`: The keys for the data. These are the columns that uniquely identify each row in the data.
+  - `source`: The source of the data. This is usually the same as the topic.
+  - `extender`: The extender for the data. This is used to extend the data with additional columns or transformations.
+  - `encrypted_columns`: The columns in the data that are encrypted. These columns will be decrypted during the "bronze" step.
+  - `calculated_columns`: The columns in the data that are calculated. These columns will be calculated during the "bronze" step.
+
+Here's an example of a "bronze" step job:
+
+```yaml
+- job:
+    step: bronze
+    topic: king
+    item: scd1
+    tags: [test]
+    options:
+      mode: append
+      uri: abfss://fabricks@$datahub/raw/king
+      parser: monarch
+      keys: [id]
+      source: king
+```
+
+# Silver Step Configuration
+
+The "silver" step in Fabricks is the intermediate stage of the data pipeline. It is responsible for processing the raw data ingested in the "bronze" step and storing it in a "silver" table for further processing. The configuration for the "silver" step is defined in a YAML file. Each job in the "silver" step has the following configuration options:
+
+- `step`: The step in the data pipeline. For these jobs, it is always "silver".
+- `topic`: The topic of the job. This is usually the name of the data source.
+- `item`: The item of the job. This is usually the name of the specific data item being processed.
+- `tags`: Tags for the job. These can be used to categorize or filter jobs.
+- `options`: Options for the job. This includes:
+  - `mode`: The mode of the job. This can be "update", "memory", "latest", "append", "combine", etc.
+  - `change_data_capture`: The type of Change Data Capture (CDC) to use. This can be "scd1", "scd2", "nocdc", etc.
+  - `parents`: The parent jobs that this job depends on. These are usually "bronze" step jobs.
+  - `extender`: The extender for the data. This is used to extend the data with additional columns or transformations.
+  - `order_duplicate_by`: The order to use when removing duplicates. This can be "asc" or "desc".
+  - `check_options`: Options for checking the data. This includes:
+    - `max_rows`: The maximum number of rows to check.
+  - `stream`: Whether to stream the data. This can be "true" or "false".
+
+Here's an example of a "silver" step job:
+
+```yaml
+- job:
+    step: silver
+    topic: king_and_queen
+    item: scd1
+    tags: [test]
+    options:
+      mode: update
+      change_data_capture: scd1
+      parents: [bronze.queen_scd1, bronze.king_scd1]
+```
+# Gold Step Configuration
+
+The "gold" step in Fabricks is the final stage of the data pipeline. It is responsible for processing the data from the "silver" step and storing it in a "gold" table for consumption. The configuration for the "gold" step is defined in a YAML file. Each job in the "gold" step has the following configuration options:
+
+- `step`: The step in the data pipeline. For these jobs, it is always "gold".
+- `topic`: The topic of the job. This is usually the name of the data source.
+- `item`: The item of the job. This is usually the name of the specific data item being processed.
+- `tags`: Tags for the job. These can be used to categorize or filter jobs.
+- `options`: Options for the job. This includes:
+  - `mode`: The mode of the job. This can be "complete", "memory", etc.
+  - `change_data_capture`: The type of Change Data Capture (CDC) to use. This can be "scd1", "scd2", "nocdc", etc.
+
+Here's an example of a "gold" step job:
+
+```yaml
+- job:
+    step: gold
+    topic: scd2
+    item: complete
+    tags: [test]
+    options:
+      change_data_capture: scd2
+      mode: complete
+````
+
+## Usage
+
+// Instructions on how to use the framework go here
+
+
+## License
+
+This project is licensed under the terms of the MIT license.

fabricks-2024.7.1.5/fabricks/api/__init__.py

@@ -0,0 +1,7 @@
+from fabricks.api.core import get_job, get_jobs, get_step
+
+__all__ = [
+    "get_job",
+    "get_jobs",
+    "get_step",
+]

fabricks-2024.7.1.5/fabricks/api/cdc/__init__.py

@@ -0,0 +1,6 @@
+from fabricks.api.cdc.nocdc import NoCDC
+from fabricks.api.cdc.scd1 import SCD1
+from fabricks.api.cdc.scd2 import SCD2
+from fabricks.cdc.cdc import CDC
+
+__all__ = ["CDC", "SCD1", "SCD2", "NoCDC"]

fabricks-2024.7.1.5/fabricks/api/cdc/nocdc.py

@@ -0,0 +1,3 @@
+from fabricks.cdc.nocdc import NoCDC
+
+__all__ = ["NoCDC"]

fabricks-2024.7.1.5/fabricks/api/cdc/scd1.py

@@ -0,0 +1,3 @@
+from fabricks.cdc.scd1 import SCD1
+
+__all__ = ["SCD1"]

fabricks-2024.7.1.5/fabricks/api/cdc/scd2.py

@@ -0,0 +1,3 @@
+from fabricks.cdc.scd2 import SCD2
+
+__all__ = ["SCD2"]

fabricks-2024.7.1.5/fabricks/api/context.py

@@ -0,0 +1,31 @@
+from databricks.sdk.runtime import dbutils, spark
+
+from fabricks.context import BRONZE, GOLD, SECRET_SCOPE, SILVER
+from fabricks.core.jobs.base.types import Bronzes, Golds, Silvers
+
+# spark
+SPARK = spark
+DBUTILS = dbutils
+
+# step
+BRONZES = Bronzes
+SILVERS = Silvers
+GOLDS = Golds
+STEPS = BRONZES + SILVERS + GOLDS
+
+
+__all__ = [
+    "BRONZE",
+    "Bronzes",
+    "BRONZES",
+    "DBUTILS",
+    "GOLD",
+    "Golds",
+    "GOLDS",
+    "SECRET_SCOPE",
+    "SILVER",
+    "Silvers",
+    "SILVERS",
+    "SPARK",
+    "STEPS",
+]

fabricks-2024.7.1.5/fabricks/api/core.py

@@ -0,0 +1,4 @@
+from fabricks.core.jobs import get_job, get_jobs
+from fabricks.core.steps import get_step
+
+__all__ = ["get_job", "get_jobs", "get_step"]

fabricks-2024.7.1.5/fabricks/api/extenders.py

@@ -0,0 +1,3 @@
+from fabricks.core.extenders import extender
+
+__all__ = ["extender"]

fabricks-2024.7.1.5/fabricks/api/log.py

@@ -0,0 +1,3 @@
+from fabricks.context.log import Logger, TableLogger, flush
+
+__all__ = ["Logger", "TableLogger", "flush"]

fabricks-2024.7.1.5/fabricks/api/metastore/__init__.py

@@ -0,0 +1,10 @@
+from fabricks.api.metastore.database import Database
+from fabricks.api.metastore.table import Table
+from fabricks.api.metastore.view import View, create_or_replace_view
+
+__all__ = [
+    "create_or_replace_view",
+    "Database",
+    "Table",
+    "View",
+]

fabricks-2024.7.1.5/fabricks/api/metastore/database.py

@@ -0,0 +1,3 @@
+from fabricks.metastore import Database
+
+__all__ = ["Database"]

fabricks-2024.7.1.5/fabricks/api/metastore/table.py

@@ -0,0 +1,3 @@
+from fabricks.metastore import Table
+
+__all__ = ["Table"]

fabricks-2024.7.1.5/fabricks/api/metastore/view.py

@@ -0,0 +1,6 @@
+from fabricks.metastore import View
+
+create_or_replace_view = View.create_or_replace
+
+
+__all__ = ["View", "create_or_replace_view"]

fabricks-2024.7.1.5/fabricks/api/notebooks/cluster.py

@@ -0,0 +1,6 @@
+# Databricks notebook source
+from databricks.sdk.runtime import dbutils
+
+# COMMAND ----------
+
+dbutils.notebook.exit("exit (0)")

fabricks-2024.7.1.5/fabricks/api/notebooks/deploy/fabricks.py

@@ -0,0 +1,147 @@
+# Databricks notebook source
+# MAGIC %pip install python-dotenv
+
+# COMMAND ----------
+
+import os
+import subprocess
+from pathlib import Path
+
+from databricks.sdk.runtime import dbutils
+from dotenv import load_dotenv
+
+# COMMAND ----------
+
+load_dotenv()
+
+# COMMAND ----------
+
+try:
+    dbutils.fs.ls("mnt/fabricks")
+except Exception:
+    print("fabricks container not mounted")
+
+# COMMAND ----------
+
+try:
+    dbutils.fs.ls("mnt/fabricks/versions")
+except Exception:
+    print("fabricks not found")
+
+# COMMAND ----------
+
+runtime = os.environ.get("path_runtime")
+assert runtime
+if not Path(runtime).exists():
+    print("runtime not found")
+
+# COMMAND ----------
+
+notebooks = os.environ.get("path_notebooks")
+assert notebooks
+if not Path(notebooks).exists():
+    print("notebooks not found")
+
+# COMMAND ----------
+
+scripts = os.environ.get("path_scripts")
+assert scripts
+if not Path(scripts).exists():
+    print("scripts not found")
+
+# COMMAND ----------
+
+abfss_wheels = "abfss://fabricks-wheels@bmsstaprdeuwsoftware.dfs.core.windows.net"
+
+# COMMAND ----------
+
+version = os.environ.get("fabricks_version")
+
+# COMMAND ----------
+
+mnt_version = f"dbfs:/mnt/fabricks/versions/{version}"
+fuse_mnt_version = f"/dbfs/mnt/fabricks/versions/{version}"
+
+# COMMAND ----------
+
+try:
+    for f in dbutils.fs.ls(mnt_version):
+        dbutils.fs.rm(f.path, True)
+except Exception:
+    pass
+
+dbutils.fs.rm(mnt_version, True)
+dbutils.fs.mkdirs(mnt_version)
+
+# COMMAND ----------
+
+dbutils.fs.rm(mnt_version, True)
+dbutils.fs.mkdirs(mnt_version)
+
+# COMMAND ----------
+
+print("copying version to", f"{mnt_version}/version")
+
+for f in dbutils.fs.ls(f"{abfss_wheels}/{version}"):
+    to = f"{mnt_version}/{f.name}"
+
+    try:
+        dbutils.fs.ls(to)
+    except Exception:
+        print("uploading", f.name)
+        dbutils.fs.cp(f.path, to, recurse=True)
+
+# COMMAND ----------
+
+print("pip install requirements.txt")
+
+out = subprocess.run(
+    [
+        "pip",
+        "install",
+        "--no-index",
+        f"--find-links={fuse_mnt_version}/wheels",
+        "-r",
+        f"{fuse_mnt_version}/requirements.txt",
+    ],
+    capture_output=True,
+)
+
+if out.returncode == 1:
+    raise ValueError(out.stderr)
+
+# COMMAND ----------
+
+latest = os.environ.get("latest")
+assert latest
+latest = latest.lower() == "true"
+
+# COMMAND ----------
+
+versions = [version, "2023.12.*"] if latest else [version]
+
+# COMMAND ----------
+
+print("deploy init script")
+
+for v in versions:
+    path = f"{scripts}/{v}.sh"
+
+    with open(path, "w") as sh:
+        sh.write(
+            f"""
+            sudo echo FABRICKS_RUNTIME={runtime} >> /etc/environment
+            sudo echo FABRICKS_NOTEBOOKS={notebooks}/{version} >> /etc/environment
+            sudo echo FABRICKS_VERSION={version} >> /etc/environment
+
+            /databricks/python/bin/pip install --no-index --find-links='{fuse_mnt_version}/wheels' -r '{fuse_mnt_version}/requirements.txt'
+            /databricks/python/bin/pip install sqlglot
+            /databricks/python/bin/pip install jinja2
+            """.replace("            ", "").strip()
+        )
+
+# COMMAND ----------
+
+dbutils.notebook.exit("exit (0)")
+
+# COMMAND ----------