salesforce-data-customcode 0.1.4__tar.gz → 0.1.6__tar.gz

salesforce_data_customcode-0.1.6/PKG-INFO

@@ -0,0 +1,261 @@
+Metadata-Version: 2.3
+Name: salesforce-data-customcode
+Version: 0.1.6
+Summary: Data Cloud Custom Code SDK
+License: Apache-2.0
+Requires-Python: >=3.10,<3.12
+Classifier: Development Status :: 4 - Beta
+Classifier: Operating System :: Unix
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Requires-Dist: click (>=8.1.8,<9.0.0)
+Requires-Dist: loguru (>=0.7.3,<0.8.0)
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: pydantic (>=1.8.2,<3.0.0)
+Requires-Dist: pyspark
+Requires-Dist: pyyaml (>=6.0,<7.0)
+Requires-Dist: salesforce-cdp-connector
+Requires-Dist: setuptools_scm (>=7.1.0,<8.0.0)
+Description-Content-Type: text/markdown
+
+# Data Cloud Custom Code SDK
+
+This package provides a development kit for creating custom data transformations in [Data Cloud](https://www.salesforce.com/data/). It allows you to write your own data processing logic in Python while leveraging Data Cloud's infrastructure for data access and running data transformations, mapping execution into Data Cloud data structures like [Data Model Objects](https://help.salesforce.com/s/articleView?id=data.c360_a_data_model_objects.htm&type=5) and [Data Lake Objects](https://help.salesforce.com/s/articleView?id=sf.c360_a_data_lake_objects.htm&language=en_US&type=5).
+
+More specifically, this codebase gives you ability to test code locally before pushing to Data Cloud's remote execution engine, greatly reducing how long it takes to develop.
+
+Use of this project with Salesforce is subject to the [TERMS OF USE](./TERMS_OF_USE.md)
+
+## Prerequisites
+
+- Python 3.11 (If your system version is different, we recommend using [pyenv](https://github.com/pyenv/pyenv) to configure 3.11)
+- [Azul Zulu OpenJDK 17.x](https://www.azul.com/downloads/?version=java-17-lts&package=jdk#zulu)
+- Docker support like [Docker Desktop](https://docs.docker.com/desktop/)
+- A salesforce org, with some DLOs or DMOs with data
+- A [connected app](#creating-a-connected-app)
+
+## Installation
+The SDK can be downloaded directly from PyPI with `pip`:
+```
+pip install salesforce-data-customcode
+```
+
+You can verify it was properly installed via CLI:
+```
+datacustomcode version
+```
+
+## Quick start
+Ensure you have all the [prerequisites](#prerequisites) prepared on your machine.
+
+To get started, create a directory and initialize a new project with the CLI:
+```zsh
+mkdir datacloud && cd datacloud
+python3.11 -m venv .venv
+source .venv/bin/activate
+pip install salesforce-data-customcode
+datacustomcode init my_package
+```
+
+This will yield all necessary files to get started:
+```
+.
+├── Dockerfile
+├── README.md
+├── requirements.txt
+├── requirements-dev.txt
+├── payload
+│   ├── config.json
+│   ├── entrypoint.py
+├── jupyterlab.sh
+└── requirements.txt
+```
+* `Dockerfile` <span style="color:grey;font-style:italic;">(Do not update)</span> – Development container emulating the remote execution environment.
+* `requirements-dev.txt` <span style="color:grey;font-style:italic;">(Do not update)</span> – These are the dependencies for the development environment.
+* `jupyterlab.sh` <span style="color:grey;font-style:italic;">(Do not update)</span> – Helper script for setting up Jupyter.
+* `requirements.txt` – Here you define the requirements that you will need for your script.
+* `payload` – This folder will be compressed and deployed to the remote execution environment.
+  * `config.json` – This config defines permissions on the back and can be generated programmatically with `scan` CLI method.
+  * `entrypoint.py` – The script that defines the data transformation logic.
+
+A functional entrypoint.py is provided so you can run once you've configured your connected app:
+```zsh
+cd my_package
+datacustomcode configure
+datacustomcode run ./payload/entrypoint.py
+```
+
+> [!IMPORTANT]
+> The example entrypoint.py requires a `Account_Home__dll` DLO to be present.  And in order to deploy the script (next step), the output DLO (which is `Account_Home_copy__dll` in the example entrypoint.py) also needs to exist and be in the same dataspace as `Account_Home__dll`.
+
+After modifying the `entrypoint.py` as needed, using any dependencies you add in the `.venv` virtual environment, you can run this script in Data Cloud:
+```zsh
+datacustomcode scan ./payload/entrypoint.py
+datacustomcode deploy --path ./payload --name my_custom_script
+```
+
+> [!TIP]
+> The `deploy` process can take several minutes.  If you'd like more feedback on the underlying process, you can add `--debug` to the command like `datacustomcode --debug deploy --path ./payload --name my_custom_script`
+
+You can now use the Salesforce Data Cloud UI to find the created Data Transform and use the `Run Now` button to run it.
+Once the Data Transform run is successful, check the DLO your script is writing to and verify the correct records were added.
+
+## API
+
+Your entry point script will define logic using the `Client` object which wraps data access layers.
+
+You should only need the following methods:
+* `read_dlo(name)` – Read from a Data Lake Object by name
+* `read_dmo(name)` – Read from a Data Model Object by name
+* `write_to_dlo(name, spark_dataframe, write_mode)` – Write to a Data Model Object by name with a Spark dataframe
+* `write_to_dmo(name, spark_dataframe, write_mode)` – Write to a Data Lake Object by name with a Spark dataframe
+
+For example:
+```
+from datacustomcode import Client
+
+client = Client()
+
+sdf = client.read_dlo('my_DLO')
+# some transformations
+# ...
+client.write_to_dlo('output_DLO')
+```
+
+
+> [!WARNING]
+> Currently we only support reading from DMOs and writing to DMOs or reading from DLOs and writing to DLOs, but they cannot mix.
+
+
+## CLI
+
+The Data Cloud Custom Code SDK provides a command-line interface (CLI) with the following commands:
+
+### Global Options
+- `--debug`: Enable debug-level logging
+
+### Commands
+
+#### `datacustomcode version`
+Display the current version of the package.
+
+#### `datacustomcode configure`
+Configure credentials for connecting to Data Cloud.
+
+Options:
+- `--profile TEXT`: Credential profile name (default: "default")
+- `--username TEXT`: Salesforce username
+- `--password TEXT`: Salesforce password
+- `--client-id TEXT`: Connected App Client ID
+- `--client-secret TEXT`: Connected App Client Secret
+- `--login-url TEXT`: Salesforce login URL
+
+#### `datacustomcode deploy`
+Deploy a transformation job to Data Cloud.
+
+Options:
+- `--profile TEXT`: Credential profile name (default: "default")
+- `--path TEXT`: Path to the code directory (default: ".")
+- `--name TEXT`: Name of the transformation job [required]
+- `--version TEXT`: Version of the transformation job (default: "0.0.1")
+- `--description TEXT`: Description of the transformation job (default: "")
+
+#### `datacustomcode init`
+Initialize a new development environment with a template.
+
+Argument:
+- `DIRECTORY`: Directory to create project in (default: ".")
+
+#### `datacustomcode scan`
+Scan a Python file to generate a Data Cloud configuration.
+
+Argument:
+- `FILENAME`: Python file to scan
+
+Options:
+- `--config TEXT`: Path to save the configuration file (default: same directory as FILENAME)
+- `--dry-run`: Preview the configuration without saving to a file
+
+#### `datacustomcode run`
+Run an entrypoint file locally for testing.
+
+Argument:
+- `ENTRYPOINT`: Path to entrypoint Python file
+
+Options:
+- `--config-file TEXT`: Path to configuration file
+- `--dependencies TEXT`: Additional dependencies (can be specified multiple times)
+
+#### `datacustomcode zip`
+Zip a transformation job in preparation to upload to Data Cloud.
+
+Options:
+- `--path TEXT`: Path to the code directory (default: ".")
+
+## Docker usage
+
+After initializing a project with `datacustomcode init my_package`, you might notice a Dockerfile.  This file isn't used for the
+[Quick Start](#quick-start) approach above, which uses virtual environments, until the `zip` or `deploy` commands are used.  When using dependencies
+that include [native features](https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html#using-pyspark-native-features)
+like C++ or C interop, the platform and architecture may be different between your machine and Data Cloud compute.  This is all taken care of
+in the `zip` and `deploy` commands, which utilize the Dockerfile which starts `FROM` an image compatible with Data Cloud.  However, you may
+want to build, run, and test your script on your machine using the same platform and architecture as Data Cloud.  You can use the sections below
+to test your script in this manner.
+
+### VS Code Dev Containers
+
+Within your `init`ed package, you will find a `.devcontainer` folder which allows you to run a docker container while developing inside of it.
+
+Read more about Dev Containers here: https://code.visualstudio.com/docs/devcontainers/containers.
+
+1. Install the VS Code extension "Dev Containers" by microsoft.com.
+1. Open your package folder in VS Code, ensuring that the `.devcontainer` folder is at the root of the File Explorer
+1. Bring up the Command Palette (on mac: Cmd + Shift + P), and select "Dev Containers: Rebuild and Reopen in Container"
+1. Allow the docker image to be built, then you're ready to develop
+1. Now if you open a terminal (within the Dev Container window) and `datacustomcode run ./payload/entrypoint.py`, it will run inside a docker container that more closely resembles Data Cloud compute than your machine
+
+> [!IMPORTANT]
+> Dev Containers get their own tmp file storage, so you'll need to re-run `datacustomcode configure` every time you "Rebuild and Reopen in Container".
+
+### JupyterLab
+
+Within your `init`ed package, you will find a `jupyterlab.sh` file that can open a jupyter notebook for you.  Jupyter notebooks, in
+combination with Data Cloud's [Query Editor](https://help.salesforce.com/s/articleView?id=data.c360_a_add_queries_to_a_query_workspace.htm&type=5)
+and [Data Explorer](https://help.salesforce.com/s/articleView?id=data.c360_a_data_explorer.htm&type=5), can be extremely helpful for data
+exploration.  Instead of running an entire script, one can run one code cell at a time as they discover and experiment with the DLO or DMO data.
+
+You can read more about Jupyter Notebooks here: https://jupyter.org/
+
+1. Within the root project of your package folder, run `./jupyterlab.sh start`
+1. Double-click on "account.ipynb" file, which provides a starting point for a notebook
+1. Use shift+enter to execute each cell within the notebook.  Add/edit/delete cells of code as needed for your data exploration.
+1. Don't forget to run `./jupyterlab.sh stop` to stop the docker container
+
+> [!IMPORTANT]
+> JupyterLab uses its own tmp file storage, so you'll need to re-run `datacustomcode configure` each time you `./jupyterlab.sh start`.
+
+## Prerequisite details
+
+### Creating a connected app
+
+1. Log in to salesforce as an admin. In the top right corner, click on the gear icon and go to `Setup`
+2. In the left hand side, search for "App Manager" and select the `App Manager` underneath `Apps`
+3. Click on `New Connected App` in the upper right
+4. Fill in the required fields within the `Basic Information` section
+5. Under the `API (Enable OAuth Settings)` section:
+    1. Click on the checkbox to Enable OAuth Settings.
+    2. Provide a callback URL like http://localhost:55555/callback
+    3. In the Selected OAuth Scopes, make sure that `refresh_token`, `api`, `cdp_query_api`, `cdp_profile_api` is selected.
+    4. Click on Save to save the connected app
+6. From the detail page that opens up afterwards, click the "Manage Consumer Details" button to find your client id and client secret
+7. Go back to `Setup`, then `OAuth and OpenID Connect Settings`, and enable the "Allow OAuth Username-Password Flows" option
+
+You now have all fields necessary for the `datacustomcode configure` command.
+
+## Other docs
+
+[Troubleshooting](./docs/troubleshooting.md)
+

salesforce_data_customcode-0.1.6/README.md

@@ -0,0 +1,237 @@
+# Data Cloud Custom Code SDK
+
+This package provides a development kit for creating custom data transformations in [Data Cloud](https://www.salesforce.com/data/). It allows you to write your own data processing logic in Python while leveraging Data Cloud's infrastructure for data access and running data transformations, mapping execution into Data Cloud data structures like [Data Model Objects](https://help.salesforce.com/s/articleView?id=data.c360_a_data_model_objects.htm&type=5) and [Data Lake Objects](https://help.salesforce.com/s/articleView?id=sf.c360_a_data_lake_objects.htm&language=en_US&type=5).
+
+More specifically, this codebase gives you ability to test code locally before pushing to Data Cloud's remote execution engine, greatly reducing how long it takes to develop.
+
+Use of this project with Salesforce is subject to the [TERMS OF USE](./TERMS_OF_USE.md)
+
+## Prerequisites
+
+- Python 3.11 (If your system version is different, we recommend using [pyenv](https://github.com/pyenv/pyenv) to configure 3.11)
+- [Azul Zulu OpenJDK 17.x](https://www.azul.com/downloads/?version=java-17-lts&package=jdk#zulu)
+- Docker support like [Docker Desktop](https://docs.docker.com/desktop/)
+- A salesforce org, with some DLOs or DMOs with data
+- A [connected app](#creating-a-connected-app)
+
+## Installation
+The SDK can be downloaded directly from PyPI with `pip`:
+```
+pip install salesforce-data-customcode
+```
+
+You can verify it was properly installed via CLI:
+```
+datacustomcode version
+```
+
+## Quick start
+Ensure you have all the [prerequisites](#prerequisites) prepared on your machine.
+
+To get started, create a directory and initialize a new project with the CLI:
+```zsh
+mkdir datacloud && cd datacloud
+python3.11 -m venv .venv
+source .venv/bin/activate
+pip install salesforce-data-customcode
+datacustomcode init my_package
+```
+
+This will yield all necessary files to get started:
+```
+.
+├── Dockerfile
+├── README.md
+├── requirements.txt
+├── requirements-dev.txt
+├── payload
+│   ├── config.json
+│   ├── entrypoint.py
+├── jupyterlab.sh
+└── requirements.txt
+```
+* `Dockerfile` <span style="color:grey;font-style:italic;">(Do not update)</span> – Development container emulating the remote execution environment.
+* `requirements-dev.txt` <span style="color:grey;font-style:italic;">(Do not update)</span> – These are the dependencies for the development environment.
+* `jupyterlab.sh` <span style="color:grey;font-style:italic;">(Do not update)</span> – Helper script for setting up Jupyter.
+* `requirements.txt` – Here you define the requirements that you will need for your script.
+* `payload` – This folder will be compressed and deployed to the remote execution environment.
+  * `config.json` – This config defines permissions on the back and can be generated programmatically with `scan` CLI method.
+  * `entrypoint.py` – The script that defines the data transformation logic.
+
+A functional entrypoint.py is provided so you can run once you've configured your connected app:
+```zsh
+cd my_package
+datacustomcode configure
+datacustomcode run ./payload/entrypoint.py
+```
+
+> [!IMPORTANT]
+> The example entrypoint.py requires a `Account_Home__dll` DLO to be present.  And in order to deploy the script (next step), the output DLO (which is `Account_Home_copy__dll` in the example entrypoint.py) also needs to exist and be in the same dataspace as `Account_Home__dll`.
+
+After modifying the `entrypoint.py` as needed, using any dependencies you add in the `.venv` virtual environment, you can run this script in Data Cloud:
+```zsh
+datacustomcode scan ./payload/entrypoint.py
+datacustomcode deploy --path ./payload --name my_custom_script
+```
+
+> [!TIP]
+> The `deploy` process can take several minutes.  If you'd like more feedback on the underlying process, you can add `--debug` to the command like `datacustomcode --debug deploy --path ./payload --name my_custom_script`
+
+You can now use the Salesforce Data Cloud UI to find the created Data Transform and use the `Run Now` button to run it.
+Once the Data Transform run is successful, check the DLO your script is writing to and verify the correct records were added.
+
+## API
+
+Your entry point script will define logic using the `Client` object which wraps data access layers.
+
+You should only need the following methods:
+* `read_dlo(name)` – Read from a Data Lake Object by name
+* `read_dmo(name)` – Read from a Data Model Object by name
+* `write_to_dlo(name, spark_dataframe, write_mode)` – Write to a Data Model Object by name with a Spark dataframe
+* `write_to_dmo(name, spark_dataframe, write_mode)` – Write to a Data Lake Object by name with a Spark dataframe
+
+For example:
+```
+from datacustomcode import Client
+
+client = Client()
+
+sdf = client.read_dlo('my_DLO')
+# some transformations
+# ...
+client.write_to_dlo('output_DLO')
+```
+
+
+> [!WARNING]
+> Currently we only support reading from DMOs and writing to DMOs or reading from DLOs and writing to DLOs, but they cannot mix.
+
+
+## CLI
+
+The Data Cloud Custom Code SDK provides a command-line interface (CLI) with the following commands:
+
+### Global Options
+- `--debug`: Enable debug-level logging
+
+### Commands
+
+#### `datacustomcode version`
+Display the current version of the package.
+
+#### `datacustomcode configure`
+Configure credentials for connecting to Data Cloud.
+
+Options:
+- `--profile TEXT`: Credential profile name (default: "default")
+- `--username TEXT`: Salesforce username
+- `--password TEXT`: Salesforce password
+- `--client-id TEXT`: Connected App Client ID
+- `--client-secret TEXT`: Connected App Client Secret
+- `--login-url TEXT`: Salesforce login URL
+
+#### `datacustomcode deploy`
+Deploy a transformation job to Data Cloud.
+
+Options:
+- `--profile TEXT`: Credential profile name (default: "default")
+- `--path TEXT`: Path to the code directory (default: ".")
+- `--name TEXT`: Name of the transformation job [required]
+- `--version TEXT`: Version of the transformation job (default: "0.0.1")
+- `--description TEXT`: Description of the transformation job (default: "")
+
+#### `datacustomcode init`
+Initialize a new development environment with a template.
+
+Argument:
+- `DIRECTORY`: Directory to create project in (default: ".")
+
+#### `datacustomcode scan`
+Scan a Python file to generate a Data Cloud configuration.
+
+Argument:
+- `FILENAME`: Python file to scan
+
+Options:
+- `--config TEXT`: Path to save the configuration file (default: same directory as FILENAME)
+- `--dry-run`: Preview the configuration without saving to a file
+
+#### `datacustomcode run`
+Run an entrypoint file locally for testing.
+
+Argument:
+- `ENTRYPOINT`: Path to entrypoint Python file
+
+Options:
+- `--config-file TEXT`: Path to configuration file
+- `--dependencies TEXT`: Additional dependencies (can be specified multiple times)
+
+#### `datacustomcode zip`
+Zip a transformation job in preparation to upload to Data Cloud.
+
+Options:
+- `--path TEXT`: Path to the code directory (default: ".")
+
+## Docker usage
+
+After initializing a project with `datacustomcode init my_package`, you might notice a Dockerfile.  This file isn't used for the
+[Quick Start](#quick-start) approach above, which uses virtual environments, until the `zip` or `deploy` commands are used.  When using dependencies
+that include [native features](https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html#using-pyspark-native-features)
+like C++ or C interop, the platform and architecture may be different between your machine and Data Cloud compute.  This is all taken care of
+in the `zip` and `deploy` commands, which utilize the Dockerfile which starts `FROM` an image compatible with Data Cloud.  However, you may
+want to build, run, and test your script on your machine using the same platform and architecture as Data Cloud.  You can use the sections below
+to test your script in this manner.
+
+### VS Code Dev Containers
+
+Within your `init`ed package, you will find a `.devcontainer` folder which allows you to run a docker container while developing inside of it.
+
+Read more about Dev Containers here: https://code.visualstudio.com/docs/devcontainers/containers.
+
+1. Install the VS Code extension "Dev Containers" by microsoft.com.
+1. Open your package folder in VS Code, ensuring that the `.devcontainer` folder is at the root of the File Explorer
+1. Bring up the Command Palette (on mac: Cmd + Shift + P), and select "Dev Containers: Rebuild and Reopen in Container"
+1. Allow the docker image to be built, then you're ready to develop
+1. Now if you open a terminal (within the Dev Container window) and `datacustomcode run ./payload/entrypoint.py`, it will run inside a docker container that more closely resembles Data Cloud compute than your machine
+
+> [!IMPORTANT]
+> Dev Containers get their own tmp file storage, so you'll need to re-run `datacustomcode configure` every time you "Rebuild and Reopen in Container".
+
+### JupyterLab
+
+Within your `init`ed package, you will find a `jupyterlab.sh` file that can open a jupyter notebook for you.  Jupyter notebooks, in
+combination with Data Cloud's [Query Editor](https://help.salesforce.com/s/articleView?id=data.c360_a_add_queries_to_a_query_workspace.htm&type=5)
+and [Data Explorer](https://help.salesforce.com/s/articleView?id=data.c360_a_data_explorer.htm&type=5), can be extremely helpful for data
+exploration.  Instead of running an entire script, one can run one code cell at a time as they discover and experiment with the DLO or DMO data.
+
+You can read more about Jupyter Notebooks here: https://jupyter.org/
+
+1. Within the root project of your package folder, run `./jupyterlab.sh start`
+1. Double-click on "account.ipynb" file, which provides a starting point for a notebook
+1. Use shift+enter to execute each cell within the notebook.  Add/edit/delete cells of code as needed for your data exploration.
+1. Don't forget to run `./jupyterlab.sh stop` to stop the docker container
+
+> [!IMPORTANT]
+> JupyterLab uses its own tmp file storage, so you'll need to re-run `datacustomcode configure` each time you `./jupyterlab.sh start`.
+
+## Prerequisite details
+
+### Creating a connected app
+
+1. Log in to salesforce as an admin. In the top right corner, click on the gear icon and go to `Setup`
+2. In the left hand side, search for "App Manager" and select the `App Manager` underneath `Apps`
+3. Click on `New Connected App` in the upper right
+4. Fill in the required fields within the `Basic Information` section
+5. Under the `API (Enable OAuth Settings)` section:
+    1. Click on the checkbox to Enable OAuth Settings.
+    2. Provide a callback URL like http://localhost:55555/callback
+    3. In the Selected OAuth Scopes, make sure that `refresh_token`, `api`, `cdp_query_api`, `cdp_profile_api` is selected.
+    4. Click on Save to save the connected app
+6. From the detail page that opens up afterwards, click the "Manage Consumer Details" button to find your client id and client secret
+7. Go back to `Setup`, then `OAuth and OpenID Connect Settings`, and enable the "Allow OAuth Username-Password Flows" option
+
+You now have all fields necessary for the `datacustomcode configure` command.
+
+## Other docs
+
+[Troubleshooting](./docs/troubleshooting.md)

{salesforce_data_customcode-0.1.4 → salesforce_data_customcode-0.1.6}/pyproject.toml

@@ -18,7 +18,7 @@ license = "Apache-2.0"
 name = "salesforce-data-customcode"
 readme = "README.md"
 requires-python = ">=3.10,<3.12"
-version = "0.1.4"
+version = "0.1.6"
 
 [tool.black]
 exclude = '''

{salesforce_data_customcode-0.1.4 → salesforce_data_customcode-0.1.6}/src/datacustomcode/cli.py

@@ -69,6 +69,15 @@ def configure(
     ).update_ini(profile=profile)
 
 
+@cli.command()
+@click.argument("path", default="payload")
+def zip(path: str):
+    from datacustomcode.deploy import zip
+
+    logger.debug("Zipping project")
+    zip(path)
+
+
 @cli.command()
 @click.option("--profile", default="default")
 @click.option("--path", default="payload")
@@ -127,8 +136,11 @@ def init(directory: str):
 @click.argument("filename")
 @click.option("--config")
 @click.option("--dry-run", is_flag=True)
-def scan(filename: str, config: str, dry_run: bool):
-    from datacustomcode.scan import dc_config_json_from_file
+@click.option(
+    "--no-requirements", is_flag=True, help="Skip generating requirements.txt file"
+)
+def scan(filename: str, config: str, dry_run: bool, no_requirements: bool):
+    from datacustomcode.scan import dc_config_json_from_file, write_requirements_file
 
     config_location = config or os.path.join(os.path.dirname(filename), "config.json")
     click.echo(
@@ -143,6 +155,13 @@ def scan(filename: str, config: str, dry_run: bool):
         with open(config_location, "w") as f:
             json.dump(config_json, f, indent=2)
 
+        if not no_requirements:
+            requirements_path = write_requirements_file(filename)
+            click.echo(
+                "Generated requirements file: "
+                + click.style(requirements_path, fg="blue", bold=True)
+            )
+
 
 @cli.command()
 @click.argument("entrypoint")

{salesforce_data_customcode-0.1.4 → salesforce_data_customcode-0.1.6}/src/datacustomcode/deploy.py

@@ -169,25 +169,14 @@ def prepare_dependency_archive(directory: str) -> None:
         archive_file = os.path.join(archives_dir, DEPENDENCIES_ARCHIVE_NAME)
         with tarfile.open(archive_file, "w:gz") as tar:
             for file in os.listdir(temp_dir):
+                # Exclude requirements.txt from the archive
+                if file == "requirements.txt":
+                    continue
                 tar.add(os.path.join(temp_dir, file), arcname=file)
 
         logger.debug(f"Dependencies downloaded and archived to {archive_file}")
 
 
-def zip_and_upload_directory(directory: str, file_upload_url: str) -> None:
-    file_upload_url = unescape(file_upload_url)
-
-    logger.debug(f"Zipping directory... {directory}")
-    shutil.make_archive(ZIP_FILE_NAME.rstrip(".zip"), "zip", directory)
-
-    logger.debug(f"Uploading deployment to {file_upload_url}")
-    with open(ZIP_FILE_NAME, "rb") as zip_file:
-        response = requests.put(
-            file_upload_url, data=zip_file, headers={"Content-Type": "application/zip"}
-        )
-        response.raise_for_status()
-
-
 class DeploymentsResponse(BaseModel):
     deploymentStatus: str
 
@@ -325,6 +314,69 @@ def create_data_transform(
     return response
 
 
+def has_nonempty_requirements_file(directory: str) -> bool:
+    """
+    Check if requirements.txt exists in the given directory and has at least
+    one non-comment line.
+    Args:
+        directory (str): The directory to check for requirements.txt.
+    Returns:
+        bool: True if requirements.txt exists and has a non-comment line,
+        False otherwise.
+    """
+    # Look for requirements.txt in the parent directory of the given directory
+    requirements_path = os.path.join(os.path.dirname(directory), "requirements.txt")
+
+    try:
+        if os.path.isfile(requirements_path):
+            with open(requirements_path, "r", encoding="utf-8") as f:
+                for line in f:
+                    # Consider non-empty if any line is not a comment (ignoring
+                    # leading whitespace)
+                    if line.strip() and not line.lstrip().startswith("#"):
+                        return True
+    except Exception as e:
+        logger.error(f"Error reading requirements.txt: {e}")
+    return False
+
+
+def upload_zip(file_upload_url: str) -> None:
+    file_upload_url = unescape(file_upload_url)
+    with open(ZIP_FILE_NAME, "rb") as zip_file:
+        response = requests.put(
+            file_upload_url, data=zip_file, headers={"Content-Type": "application/zip"}
+        )
+        response.raise_for_status()
+
+
+def zip(
+    directory: str,
+):
+    # Create a zip file excluding .DS_Store files
+    import zipfile
+
+    # prepare payload only if requirements.txt is non-empty
+    if has_nonempty_requirements_file(directory):
+        prepare_dependency_archive(directory)
+    else:
+        logger.info(
+            f"Skipping dependency archive: requirements.txt is missing or empty "
+            f"in {directory}"
+        )
+
+    logger.debug(f"Zipping directory... {directory}")
+
+    with zipfile.ZipFile(ZIP_FILE_NAME, "w", zipfile.ZIP_DEFLATED) as zipf:
+        for root, dirs, files in os.walk(directory):
+            # Skip .DS_Store files when adding to zip
+            for file in files:
+                if file != ".DS_Store":
+                    file_path = os.path.join(root, file)
+                    zipf.write(file_path)
+
+    logger.debug(f"Created zip file: {ZIP_FILE_NAME}")
+
+
 def deploy_full(
     directory: str,
     metadata: TransformationJobMetadata,
@@ -340,7 +392,8 @@ def deploy_full(
 
     # create deployment and upload payload
     deployment = create_deployment(access_token, metadata)
-    zip_and_upload_directory(directory, deployment.fileUploadUrl)
+    zip(directory)
+    upload_zip(deployment.fileUploadUrl)
     wait_for_deployment(access_token, metadata, callback)
 
     # create data transform

{salesforce_data_customcode-0.1.4 → salesforce_data_customcode-0.1.6}/src/datacustomcode/scan.py

@@ -15,9 +15,12 @@
 from __future__ import annotations
 
 import ast
+import os
 from typing import (
     Any,
+    ClassVar,
     Dict,
+    Set,
     Union,
 )
 
@@ -131,6 +134,137 @@ class ClientMethodVisitor(ast.NodeVisitor):
         )
 
 
+class ImportVisitor(ast.NodeVisitor):
+    """AST Visitor that extracts external package imports from Python code."""
+
+    # Standard library modules that should be excluded from requirements
+    STANDARD_LIBS: ClassVar[set[str]] = {
+        "abc",
+        "argparse",
+        "ast",
+        "asyncio",
+        "base64",
+        "collections",
+        "configparser",
+        "contextlib",
+        "copy",
+        "csv",
+        "datetime",
+        "enum",
+        "functools",
+        "glob",
+        "hashlib",
+        "http",
+        "importlib",
+        "inspect",
+        "io",
+        "itertools",
+        "json",
+        "logging",
+        "math",
+        "os",
+        "pathlib",
+        "pickle",
+        "random",
+        "re",
+        "shutil",
+        "site",
+        "socket",
+        "sqlite3",
+        "string",
+        "subprocess",
+        "sys",
+        "tempfile",
+        "threading",
+        "time",
+        "traceback",
+        "typing",
+        "uuid",
+        "warnings",
+        "xml",
+        "zipfile",
+    }
+
+    # Additional packages to exclude from requirements.txt
+    EXCLUDED_PACKAGES: ClassVar[set[str]] = {
+        "datacustomcode",  # Internal package
+        "pyspark",  # Provided by the runtime environment
+    }
+
+    def __init__(self) -> None:
+        self.imports: Set[str] = set()
+
+    def visit_Import(self, node: ast.Import) -> None:
+        """Visit an import statement (e.g., import os, sys)."""
+        for name in node.names:
+            # Get the top-level package name
+            package = name.name.split(".")[0]
+            if (
+                package not in self.STANDARD_LIBS
+                and package not in self.EXCLUDED_PACKAGES
+                and not package.startswith("_")
+            ):
+                self.imports.add(package)
+        self.generic_visit(node)
+
+    def visit_ImportFrom(self, node: ast.ImportFrom) -> None:
+        """Visit a from-import statement (e.g., from os import path)."""
+        if node.module is not None:
+            # Get the top-level package
+            package = node.module.split(".")[0]
+            if (
+                package not in self.STANDARD_LIBS
+                and package not in self.EXCLUDED_PACKAGES
+                and not package.startswith("_")
+            ):
+                self.imports.add(package)
+        self.generic_visit(node)
+
+
+def scan_file_for_imports(file_path: str) -> Set[str]:
+    """Scan a Python file for external package imports."""
+    with open(file_path, "r") as f:
+        code = f.read()
+        tree = ast.parse(code)
+        visitor = ImportVisitor()
+        visitor.visit(tree)
+        return visitor.imports
+
+
+def write_requirements_file(file_path: str) -> str:
+    """
+    Scan a Python file for imports and write them to requirements.txt.
+
+    Args:
+        file_path: Path to the Python file to scan
+
+    Returns:
+        Path to the generated requirements.txt file
+    """
+    imports = scan_file_for_imports(file_path)
+
+    # Write requirements.txt in the parent directory of the Python file
+    file_dir = os.path.dirname(file_path)
+    parent_dir = os.path.dirname(file_dir) if file_dir else "."
+    requirements_path = os.path.join(parent_dir, "requirements.txt")
+
+    # If the file exists, read existing requirements and merge with new ones
+    existing_requirements = set()
+    if os.path.exists(requirements_path):
+        with open(requirements_path, "r") as f:
+            existing_requirements = {line.strip() for line in f if line.strip()}
+
+    # Merge existing requirements with newly discovered ones
+    all_requirements = existing_requirements.union(imports)
+
+    # Write the combined requirements
+    with open(requirements_path, "w") as f:
+        for package in sorted(all_requirements):
+            f.write(f"{package}\n")
+
+    return requirements_path
+
+
 def scan_file(file_path: str) -> DataAccessLayerCalls:
     """Scan a single Python file for Client read/write method calls."""
     with open(file_path, "r") as f:

salesforce_data_customcode-0.1.6/src/datacustomcode/templates/account.ipynb

@@ -0,0 +1,86 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from datacustomcode.client import Client\n",
+    "from datacustomcode.io.writer.base import WriteMode\n",
+    "from pyspark.sql.functions import col, upper"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "client = Client()\n",
+    "\n",
+    "df = client.read_dlo(\"Account_Home__dll\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Perform transformations on the DataFrame\n",
+    "df_upper1 = df.withColumn(\"Description__c\", upper(col(\"Description__c\")))\n",
+    "\n",
+    "# Drop specific columns related to relationships\n",
+    "df_upper1 = df_upper1.drop(\"KQ_ParentId__c\")\n",
+    "df_upper1 = df_upper1.drop(\"KQ_Id__c\")\n",
+    "\n",
+    "df_upper1.show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Save the transformed DataFrame\n",
+    "dlo_name = \"Account_Home_copy__dll\"\n",
+    "client.write_to_dlo(dlo_name, df_upper1, write_mode=WriteMode.APPEND)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

{salesforce_data_customcode-0.1.4 → salesforce_data_customcode-0.1.6}/src/datacustomcode/templates/jupyterlab.sh

@@ -48,13 +48,13 @@ check_docker() {
 # Function to start Jupyter server
 start_jupyter() {
     echo "Building the docker image"
-    docker build -t datacloud-byoc .
+    docker build -t datacloud-customcode .
 
     echo "Running the docker container"
     docker run -d --rm -p 8888:8888 \
         -v $(pwd):/workspace \
         --name jupyter-server \
-        datacloud-byoc jupyter lab \
+        datacloud-customcode jupyter lab \
         --ip=0.0.0.0 \
         --port=8888 \
         --no-browser \

salesforce_data_customcode-0.1.4/PKG-INFO

@@ -1,159 +0,0 @@
-Metadata-Version: 2.3
-Name: salesforce-data-customcode
-Version: 0.1.4
-Summary: Data Cloud Custom Code SDK
-License: Apache-2.0
-Requires-Python: >=3.10,<3.12
-Classifier: Development Status :: 4 - Beta
-Classifier: Operating System :: Unix
-Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Requires-Dist: click (>=8.1.8,<9.0.0)
-Requires-Dist: loguru (>=0.7.3,<0.8.0)
-Requires-Dist: numpy
-Requires-Dist: pandas
-Requires-Dist: pydantic (>=1.8.2,<3.0.0)
-Requires-Dist: pyspark
-Requires-Dist: pyyaml (>=6.0,<7.0)
-Requires-Dist: salesforce-cdp-connector
-Requires-Dist: setuptools_scm (>=7.1.0,<8.0.0)
-Description-Content-Type: text/markdown
-
-# Data Cloud Custom Code SDK
-
-<img src="https://img.shields.io/badge/version-0.1.0-blue" alt="license">
-
-This package provides a development kit for creating custom data transformations in [Data Cloud](https://www.salesforce.com/data/). It allows you to write your own data processing logic in Python while leveraging Data Cloud's infrastructure for data access and running data transformations, mapping execution into Data Cloud data structures like [Data Model Objects](https://help.salesforce.com/s/articleView?id=data.c360_a_data_model_objects.htm&type=5) and [Data Lake Objects](https://help.salesforce.com/s/articleView?id=sf.c360_a_data_lake_objects.htm&language=en_US&type=5).
-
-More specifically, this codebase gives you ability to test code locally before pushing to Data Cloud's remote execution engine, greatly reducing how long it takes to develop.
-
-Use of this project with Salesforce is subject to the [TERMS OF USE](./TERMS_OF_USE.md)
-
-## Installation
-The SDK can be downloaded directly from PyPI with `pip`:
-```
-pip install salesforce-data-customcode
-```
-
-You can verify it was properly installed via CLI:
-```
-datacustomcode version
-```
-
-## Development Setup
-We offer two built-in development interfaces: `devcontainers` and Jupyter, but you can set up any tool you would like manually.
-
-To get started, use the CLI to initialize a new development environment:
-```
-datacustomcode init [DIRECTORY TO DUMP NEW REPO]
-```
-
-This will yield all necessary files to get started:
-```
-.
-├── Dockerfile
-├── README.md
-├── requirements.txt
-├── requirements-dev.txt
-├── payload
-│   ├── config.json
-│   ├── entrypoint.py
-├── jupyterlab.sh
-└── requirements.txt
-```
-* `Dockerfile` <span style="color:grey;font-style:italic;">(Do not update)</span> – Development container emulating the remote execution environment.
-* `requirements-dev.txt` <span style="color:grey;font-style:italic;">(Do not update)</span> – These are the dependencies for the development environment.
-* `jupyterlab.sh` <span style="color:grey;font-style:italic;">(Do not update)</span> – Helper script for setting up Jupyter.
-* `requirements.txt` – Here you define the requirements that you will need remotely
-* `payload` – This folder will be compressed and deployed to the remote execution environment.
-  * `config.json` – This config defines permissions on the back and can be generated programmatically with `scan` CLI method.
-  * `entrypoint.py` – The script that defines the data transformation logic.
-
-## API
-
-You entry point script will define logic using the `Client` object which wraps data access layers.
-
-You should only need the following methods:
-* `read_dlo(name)` – Read from a Data Lake Object by name
-* `read_dmo(name)` – Read from a Data Model Object by name
-* `write_to_dlo(name, spark_dataframe, write_mode)` – Write to a Data Model Object by name with a Spark dataframe
-* `write_to_dmo(name, spark_dataframe, write_mode)` – Write to a Data Lake Object by name with a Spark dataframe
-
-For example:
-```
-from datacustomcode import Client
-
-client = Client()
-
-sdf = client.read_dlo('my_DLO')
-# some transformations
-# ...
-client.write_to_dlo('output_DLO')
-```
-
-
-> [!WARNING]
-> Currently we only support reading from DMOs and writing to DMOs or reading from DLOs and writing to DLOs, but they cannot mix.
-
-
-## CLI
-
-The Data Cloud Custom Code SDK provides a command-line interface (CLI) with the following commands:
-
-### Global Options
-- `--debug`: Enable debug-level logging
-
-### Commands
-
-#### `datacustomcode version`
-Display the current version of the package.
-
-#### `datacustomcode configure`
-Configure credentials for connecting to Data Cloud.
-
-Options:
-- `--profile TEXT`: Credential profile name (default: "default")
-- `--username TEXT`: Salesforce username
-- `--password TEXT`: Salesforce password
-- `--client-id TEXT`: Connected App Client ID
-- `--client-secret TEXT`: Connected App Client Secret
-- `--login-url TEXT`: Salesforce login URL
-
-#### `datacustomcode deploy`
-Deploy a transformation job to Data Cloud.
-
-Options:
-- `--profile TEXT`: Credential profile name (default: "default")
-- `--path TEXT`: Path to the code directory (default: ".")
-- `--name TEXT`: Name of the transformation job [required]
-- `--version TEXT`: Version of the transformation job (default: "0.0.1")
-- `--description TEXT`: Description of the transformation job (default: "")
-
-#### `datacustomcode init`
-Initialize a new development environment with a template.
-
-Argument:
-- `DIRECTORY`: Directory to create project in (default: ".")
-
-#### `datacustomcode scan`
-Scan a Python file to generate a Data Cloud configuration.
-
-Argument:
-- `FILENAME`: Python file to scan
-
-Options:
-- `--config TEXT`: Path to save the configuration file (default: same directory as FILENAME)
-- `--dry-run`: Preview the configuration without saving to a file
-
-#### `datacustomcode run`
-Run an entrypoint file locally for testing.
-
-Argument:
-- `ENTRYPOINT`: Path to entrypoint Python file
-
-Options:
-- `--config-file TEXT`: Path to configuration file
-- `--dependencies TEXT`: Additional dependencies (can be specified multiple times)
-

salesforce_data_customcode-0.1.4/README.md

@@ -1,135 +0,0 @@
-# Data Cloud Custom Code SDK
-
-<img src="https://img.shields.io/badge/version-0.1.0-blue" alt="license">
-
-This package provides a development kit for creating custom data transformations in [Data Cloud](https://www.salesforce.com/data/). It allows you to write your own data processing logic in Python while leveraging Data Cloud's infrastructure for data access and running data transformations, mapping execution into Data Cloud data structures like [Data Model Objects](https://help.salesforce.com/s/articleView?id=data.c360_a_data_model_objects.htm&type=5) and [Data Lake Objects](https://help.salesforce.com/s/articleView?id=sf.c360_a_data_lake_objects.htm&language=en_US&type=5).
-
-More specifically, this codebase gives you ability to test code locally before pushing to Data Cloud's remote execution engine, greatly reducing how long it takes to develop.
-
-Use of this project with Salesforce is subject to the [TERMS OF USE](./TERMS_OF_USE.md)
-
-## Installation
-The SDK can be downloaded directly from PyPI with `pip`:
-```
-pip install salesforce-data-customcode
-```
-
-You can verify it was properly installed via CLI:
-```
-datacustomcode version
-```
-
-## Development Setup
-We offer two built-in development interfaces: `devcontainers` and Jupyter, but you can set up any tool you would like manually.
-
-To get started, use the CLI to initialize a new development environment:
-```
-datacustomcode init [DIRECTORY TO DUMP NEW REPO]
-```
-
-This will yield all necessary files to get started:
-```
-.
-├── Dockerfile
-├── README.md
-├── requirements.txt
-├── requirements-dev.txt
-├── payload
-│   ├── config.json
-│   ├── entrypoint.py
-├── jupyterlab.sh
-└── requirements.txt
-```
-* `Dockerfile` <span style="color:grey;font-style:italic;">(Do not update)</span> – Development container emulating the remote execution environment.
-* `requirements-dev.txt` <span style="color:grey;font-style:italic;">(Do not update)</span> – These are the dependencies for the development environment.
-* `jupyterlab.sh` <span style="color:grey;font-style:italic;">(Do not update)</span> – Helper script for setting up Jupyter.
-* `requirements.txt` – Here you define the requirements that you will need remotely
-* `payload` – This folder will be compressed and deployed to the remote execution environment.
-  * `config.json` – This config defines permissions on the back and can be generated programmatically with `scan` CLI method.
-  * `entrypoint.py` – The script that defines the data transformation logic.
-
-## API
-
-You entry point script will define logic using the `Client` object which wraps data access layers.
-
-You should only need the following methods:
-* `read_dlo(name)` – Read from a Data Lake Object by name
-* `read_dmo(name)` – Read from a Data Model Object by name
-* `write_to_dlo(name, spark_dataframe, write_mode)` – Write to a Data Model Object by name with a Spark dataframe
-* `write_to_dmo(name, spark_dataframe, write_mode)` – Write to a Data Lake Object by name with a Spark dataframe
-
-For example:
-```
-from datacustomcode import Client
-
-client = Client()
-
-sdf = client.read_dlo('my_DLO')
-# some transformations
-# ...
-client.write_to_dlo('output_DLO')
-```
-
-
-> [!WARNING]
-> Currently we only support reading from DMOs and writing to DMOs or reading from DLOs and writing to DLOs, but they cannot mix.
-
-
-## CLI
-
-The Data Cloud Custom Code SDK provides a command-line interface (CLI) with the following commands:
-
-### Global Options
-- `--debug`: Enable debug-level logging
-
-### Commands
-
-#### `datacustomcode version`
-Display the current version of the package.
-
-#### `datacustomcode configure`
-Configure credentials for connecting to Data Cloud.
-
-Options:
-- `--profile TEXT`: Credential profile name (default: "default")
-- `--username TEXT`: Salesforce username
-- `--password TEXT`: Salesforce password
-- `--client-id TEXT`: Connected App Client ID
-- `--client-secret TEXT`: Connected App Client Secret
-- `--login-url TEXT`: Salesforce login URL
-
-#### `datacustomcode deploy`
-Deploy a transformation job to Data Cloud.
-
-Options:
-- `--profile TEXT`: Credential profile name (default: "default")
-- `--path TEXT`: Path to the code directory (default: ".")
-- `--name TEXT`: Name of the transformation job [required]
-- `--version TEXT`: Version of the transformation job (default: "0.0.1")
-- `--description TEXT`: Description of the transformation job (default: "")
-
-#### `datacustomcode init`
-Initialize a new development environment with a template.
-
-Argument:
-- `DIRECTORY`: Directory to create project in (default: ".")
-
-#### `datacustomcode scan`
-Scan a Python file to generate a Data Cloud configuration.
-
-Argument:
-- `FILENAME`: Python file to scan
-
-Options:
-- `--config TEXT`: Path to save the configuration file (default: same directory as FILENAME)
-- `--dry-run`: Preview the configuration without saving to a file
-
-#### `datacustomcode run`
-Run an entrypoint file locally for testing.
-
-Argument:
-- `ENTRYPOINT`: Path to entrypoint Python file
-
-Options:
-- `--config-file TEXT`: Path to configuration file
-- `--dependencies TEXT`: Additional dependencies (can be specified multiple times)