salesforce-data-customcode 6.1.0rc1__tar.gz → 7.0.0rc2__tar.gz

{salesforce_data_customcode-6.1.0rc1 → salesforce_data_customcode-7.0.0rc2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: salesforce-data-customcode
-Version: 6.1.0rc1
+Version: 7.0.0rc2
 Summary: Data Cloud Custom Code SDK
 License-Expression: Apache-2.0
 License-File: LICENSE.txt
@@ -170,15 +170,22 @@ Your Python dependencies can be packaged as .py files, .zip archives (containing
 
 ## API
 
-Your entry point script will define logic using the `Client` object which wraps data access layers.
+Your entry point script will define logic using the `Client` object (for batch transforms) or the `StreamingClient` object (for streaming delta transforms), which wrap the data access layers. Both are singletons; a single transform should use one or the other, not both.
 
-You should only need the following methods:
+For a batch transform, use `Client`. You should only need the following methods:
 * `find_file_path(file_name)` – Resolve a bundled file (placed under `payload/files/`) to a `pathlib.Path` that exists. Works the same locally and inside Data Cloud — see [Bundled file resolution](#bundled-file-resolution) below for the full lookup order. Raises `FileNotFoundError` if the file isn't found.
 * `read_dlo(name)` – Read from a Data Lake Object by name
 * `read_dmo(name)` – Read from a Data Model Object by name
 * `write_to_dlo(name, spark_dataframe, write_mode)` – Write to a Data Model Object by name with a Spark dataframe
 * `write_to_dmo(name, spark_dataframe, write_mode)` – Write to a Data Lake Object by name with a Spark dataframe
 
+For a streaming (delta) transform, use `StreamingClient`, which exposes the streaming counterparts:
+* `read_dlo_deltas()` – Read the streaming change feed (deltas) of a Data Lake Object as a streaming DataFrame.
+* `read_dmo_deltas()` – Read the streaming change feed (deltas) of a Data Model Object as a streaming DataFrame.
+* `write_dlo_deltas(name, spark_dataframe)` – Write a streaming DataFrame of deltas to a Data Lake Object; returns the started `StreamingQuery`
+
+`find_file_path`, `llm_gateway_generate_text`, and `einstein_predict` are available on both clients.
+
 For example:
 ```python
 from datacustomcode import Client
@@ -194,6 +201,36 @@ client.write_to_dlo('output_DLO')
 > [!WARNING]
 > Currently we only support reading from DMOs and writing to DMOs or reading from DLOs and writing to DLOs, but they cannot mix.
 
+### Streaming (delta) transforms
+
+Streaming BYOC transforms process a Data Lake Object's Change Data Feed continuously instead of reading a bounded snapshot. Use a `StreamingClient` and its `*_deltas` methods in place of the batch `Client` read/write methods:
+
+```python
+from pyspark.sql.functions import col, upper
+
+from datacustomcode import StreamingClient
+
+client = StreamingClient()
+
+# read_dlo_deltas returns a *streaming* DataFrame over the change feed.
+# The runtime resolves the single streaming source, so no name is passed.
+deltas = client.read_dlo_deltas()
+
+# Ordinary PySpark transform.
+transformed = deltas.withColumn("description__c", upper(col("description__c")))
+
+# write_dlo_deltas starts a streaming query and returns the StreamingQuery.
+# The runtime owns the trigger and checkpoint location; you
+# choose only the target table.
+query = client.write_dlo_deltas("Output__dll", transformed)
+query.awaitTermination()
+```
+
+Notes:
+
+- These methods only run inside the Data Cloud streaming (`DELTA_SYNC`) runtime. Locally (`datacustomcode run`) they raise `NotImplementedError`, since there is no change feed to stream.
+- A complete runnable entry point is provided in [`examples/streaming_deltas/entrypoint.py`](src/datacustomcode/templates/script/examples/streaming_deltas/entrypoint.py).
+
 ### Bundled file resolution
 
 Place bundled files (CSVs, prompt files, etc.) under `payload/files/`. The same `client.find_file_path("data.csv")` call resolves consistently across all three runtimes:
@@ -497,7 +534,7 @@ exploration.  Instead of running an entire script, one can run one code cell at
 
 You can read more about Jupyter Notebooks here: https://jupyter.org/
 
-1. Within the root project of your package folder, run `./jupyterlab.sh start`
+1. Within the root project of your package folder, run `./jupyterlab.sh start`. This prints an access token and opens an already-authenticated JupyterLab session in your browser. If the browser doesn't open automatically, copy the printed `http://localhost:8888/?token=...` URL into your browser.
 1. Double-click on "account.ipynb" file, which provides a starting point for a notebook
 1. Use shift+enter to execute each cell within the notebook.  Add/edit/delete cells of code as needed for your data exploration.
 1. Don't forget to run `./jupyterlab.sh stop` to stop the docker container
@@ -588,5 +625,5 @@ If you're using OAuth Tokens authentication, the initial configure will retrieve
 ## Other docs
 
 - [Troubleshooting](./docs/troubleshooting.md)
-- [For Contributors](./FOR_CONTRIBUTORS.md)
+- [Contributing](./CONTRIBUTING.md)
 

{salesforce_data_customcode-6.1.0rc1 → salesforce_data_customcode-7.0.0rc2}/README.md

@@ -145,15 +145,22 @@ Your Python dependencies can be packaged as .py files, .zip archives (containing
 
 ## API
 
-Your entry point script will define logic using the `Client` object which wraps data access layers.
+Your entry point script will define logic using the `Client` object (for batch transforms) or the `StreamingClient` object (for streaming delta transforms), which wrap the data access layers. Both are singletons; a single transform should use one or the other, not both.
 
-You should only need the following methods:
+For a batch transform, use `Client`. You should only need the following methods:
 * `find_file_path(file_name)` – Resolve a bundled file (placed under `payload/files/`) to a `pathlib.Path` that exists. Works the same locally and inside Data Cloud — see [Bundled file resolution](#bundled-file-resolution) below for the full lookup order. Raises `FileNotFoundError` if the file isn't found.
 * `read_dlo(name)` – Read from a Data Lake Object by name
 * `read_dmo(name)` – Read from a Data Model Object by name
 * `write_to_dlo(name, spark_dataframe, write_mode)` – Write to a Data Model Object by name with a Spark dataframe
 * `write_to_dmo(name, spark_dataframe, write_mode)` – Write to a Data Lake Object by name with a Spark dataframe
 
+For a streaming (delta) transform, use `StreamingClient`, which exposes the streaming counterparts:
+* `read_dlo_deltas()` – Read the streaming change feed (deltas) of a Data Lake Object as a streaming DataFrame.
+* `read_dmo_deltas()` – Read the streaming change feed (deltas) of a Data Model Object as a streaming DataFrame.
+* `write_dlo_deltas(name, spark_dataframe)` – Write a streaming DataFrame of deltas to a Data Lake Object; returns the started `StreamingQuery`
+
+`find_file_path`, `llm_gateway_generate_text`, and `einstein_predict` are available on both clients.
+
 For example:
 ```python
 from datacustomcode import Client
@@ -169,6 +176,36 @@ client.write_to_dlo('output_DLO')
 > [!WARNING]
 > Currently we only support reading from DMOs and writing to DMOs or reading from DLOs and writing to DLOs, but they cannot mix.
 
+### Streaming (delta) transforms
+
+Streaming BYOC transforms process a Data Lake Object's Change Data Feed continuously instead of reading a bounded snapshot. Use a `StreamingClient` and its `*_deltas` methods in place of the batch `Client` read/write methods:
+
+```python
+from pyspark.sql.functions import col, upper
+
+from datacustomcode import StreamingClient
+
+client = StreamingClient()
+
+# read_dlo_deltas returns a *streaming* DataFrame over the change feed.
+# The runtime resolves the single streaming source, so no name is passed.
+deltas = client.read_dlo_deltas()
+
+# Ordinary PySpark transform.
+transformed = deltas.withColumn("description__c", upper(col("description__c")))
+
+# write_dlo_deltas starts a streaming query and returns the StreamingQuery.
+# The runtime owns the trigger and checkpoint location; you
+# choose only the target table.
+query = client.write_dlo_deltas("Output__dll", transformed)
+query.awaitTermination()
+```
+
+Notes:
+
+- These methods only run inside the Data Cloud streaming (`DELTA_SYNC`) runtime. Locally (`datacustomcode run`) they raise `NotImplementedError`, since there is no change feed to stream.
+- A complete runnable entry point is provided in [`examples/streaming_deltas/entrypoint.py`](src/datacustomcode/templates/script/examples/streaming_deltas/entrypoint.py).
+
 ### Bundled file resolution
 
 Place bundled files (CSVs, prompt files, etc.) under `payload/files/`. The same `client.find_file_path("data.csv")` call resolves consistently across all three runtimes:
@@ -472,7 +509,7 @@ exploration.  Instead of running an entire script, one can run one code cell at
 
 You can read more about Jupyter Notebooks here: https://jupyter.org/
 
-1. Within the root project of your package folder, run `./jupyterlab.sh start`
+1. Within the root project of your package folder, run `./jupyterlab.sh start`. This prints an access token and opens an already-authenticated JupyterLab session in your browser. If the browser doesn't open automatically, copy the printed `http://localhost:8888/?token=...` URL into your browser.
 1. Double-click on "account.ipynb" file, which provides a starting point for a notebook
 1. Use shift+enter to execute each cell within the notebook.  Add/edit/delete cells of code as needed for your data exploration.
 1. Don't forget to run `./jupyterlab.sh stop` to stop the docker container
@@ -563,4 +600,4 @@ If you're using OAuth Tokens authentication, the initial configure will retrieve
 ## Other docs
 
 - [Troubleshooting](./docs/troubleshooting.md)
-- [For Contributors](./FOR_CONTRIBUTORS.md)
+- [Contributing](./CONTRIBUTING.md)

{salesforce_data_customcode-6.1.0rc1 → salesforce_data_customcode-7.0.0rc2}/pyproject.toml

@@ -18,7 +18,7 @@ license = "Apache-2.0"
 name = "salesforce-data-customcode"
 readme = "README.md"
 requires-python = ">=3.10,<3.12"
-version = "6.1.0rc1"
+version = "7.0.0rc2"
 
 [tool.black]
 exclude = '''

{salesforce_data_customcode-6.1.0rc1 → salesforce_data_customcode-7.0.0rc2}/src/datacustomcode/__init__.py

@@ -13,6 +13,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from datacustomcode.spark.column_hints import install_column_casing_hints
+
+# Friendlier errors for wrong-case column references. Idempotent.
+install_column_casing_hints()
+
 __all__ = [
     "AuthType",
     "Client",
@@ -23,6 +28,7 @@ __all__ = [
     "QueryAPIDataCloudReader",
     "SparkEinsteinPredictions",
     "SparkLLMGateway",
+    "StreamingClient",
     "einstein_predict_col",
     "llm_gateway_generate_text_col",
 ]
@@ -34,6 +40,10 @@ def __getattr__(name: str):
         from datacustomcode.client import Client
 
         return Client
+    elif name == "StreamingClient":
+        from datacustomcode.client import StreamingClient
+
+        return StreamingClient
     elif name == "AuthType":
         from datacustomcode.credentials import AuthType
 

{salesforce_data_customcode-6.1.0rc1 → salesforce_data_customcode-7.0.0rc2}/src/datacustomcode/cli.py

@@ -283,10 +283,19 @@ def deploy(
 )
 @click.option(
     "--use-in-feature",
-    default="SearchIndexChunking",
-    help="Feature where this function will be used (only applicable for function).",
+    "-u",
+    default=None,
+    help=(
+        "Invoke option for this package. For scripts: 'BatchTransform' "
+        "(default) or 'StreamingTransform'. For functions: 'SearchIndexChunking'."
+    ),
 )
 def init(directory: str, code_type: str, use_in_feature: Optional[str]):
+    from datacustomcode.constants import (
+        SCRIPT_USE_IN_FEATURE_BATCH,
+        SCRIPT_USE_IN_FEATURE_OPTIONS,
+        SCRIPT_USE_IN_FEATURE_STREAMING,
+    )
     from datacustomcode.scan import (
         dc_config_json_from_file,
         update_config,
@@ -294,9 +303,23 @@ def init(directory: str, code_type: str, use_in_feature: Optional[str]):
     )
     from datacustomcode.template import copy_function_template, copy_script_template
 
+    streaming = False
+    if code_type == "script":
+        use_in_feature = use_in_feature or SCRIPT_USE_IN_FEATURE_BATCH
+        if use_in_feature not in SCRIPT_USE_IN_FEATURE_OPTIONS:
+            click.secho(
+                f"Error: Invalid --use-in-feature '{use_in_feature}' for a "
+                f"script. Valid options: {', '.join(SCRIPT_USE_IN_FEATURE_OPTIONS)}.",
+                fg="red",
+            )
+            raise click.Abort()
+        streaming = use_in_feature == SCRIPT_USE_IN_FEATURE_STREAMING
+    else:
+        use_in_feature = use_in_feature or "SearchIndexChunking"
+
     click.echo("Copying template to " + click.style(directory, fg="blue", bold=True))
     if code_type == "script":
-        copy_script_template(directory)
+        copy_script_template(directory, streaming=streaming)
     elif code_type == "function":
         copy_function_template(directory, use_in_feature)
     entrypoint_path = os.path.join(directory, PAYLOAD_DIR, ENTRYPOINT_FILE)
@@ -306,7 +329,9 @@ def init(directory: str, code_type: str, use_in_feature: Optional[str]):
     sdk_config = {"type": code_type}
     write_sdk_config(directory, sdk_config)
 
-    config_json = dc_config_json_from_file(entrypoint_path, code_type)
+    config_json = dc_config_json_from_file(
+        entrypoint_path, code_type, streaming=streaming
+    )
     with open(config_location, "w") as f:
         json.dump(config_json, f, indent=2)