salesforce-data-customcode 7.0.0rc3__tar.gz → 7.0.0rc5__tar.gz

{salesforce_data_customcode-7.0.0rc3 → salesforce_data_customcode-7.0.0rc5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: salesforce-data-customcode
-Version: 7.0.0rc3
+Version: 7.0.0rc5
 Summary: Data Cloud Custom Code SDK
 License-Expression: Apache-2.0
 License-File: LICENSE.txt
@@ -176,13 +176,14 @@ For a batch transform, use `Client`. You should only need the following methods:
 * `find_file_path(file_name)` – Resolve a bundled file (placed under `payload/files/`) to a `pathlib.Path` that exists. Works the same locally and inside Data Cloud — see [Bundled file resolution](#bundled-file-resolution) below for the full lookup order. Raises `FileNotFoundError` if the file isn't found.
 * `read_dlo(name)` – Read from a Data Lake Object by name
 * `read_dmo(name)` – Read from a Data Model Object by name
-* `write_to_dlo(name, spark_dataframe, write_mode)` – Write to a Data Model Object by name with a Spark dataframe
-* `write_to_dmo(name, spark_dataframe, write_mode)` – Write to a Data Lake Object by name with a Spark dataframe
+* `write_to_dlo(name, spark_dataframe, write_mode)` – Write to a Data Lake Object by name with a Spark dataframe
+* `write_to_dmo(name, spark_dataframe, write_mode)` – Write to a Data Model Object by name with a Spark dataframe
 
 For a streaming (delta) transform, use `StreamingClient`, which exposes the streaming counterparts:
 * `read_dlo_deltas()` – Read the streaming change feed (deltas) of a Data Lake Object as a streaming DataFrame.
 * `read_dmo_deltas()` – Read the streaming change feed (deltas) of a Data Model Object as a streaming DataFrame.
 * `write_dlo_deltas(name, spark_dataframe)` – Write a streaming DataFrame of deltas to a Data Lake Object; returns the started `StreamingQuery`
+* `write_dmo_deltas(name, spark_dataframe)` – Write a streaming DataFrame of deltas to a Data Model Object; returns the started `StreamingQuery`
 
 `find_file_path`, `llm_gateway_generate_text`, and `einstein_predict` are available on both clients.
 
@@ -201,6 +202,27 @@ client.write_to_dlo('output_DLO')
 > [!WARNING]
 > Currently we only support reading from DMOs and writing to DMOs or reading from DLOs and writing to DLOs, but they cannot mix.
 
+### Writing Currency fields
+
+A DLO or DMO with a Currency field has a system field, `cdp_sys_record_currency__c`, that holds each record's ISO currency code. Data Explorer UI uses it to display currency values, and shows a blank value if it's null (Query Editor still shows it).
+
+Code Extensions don't set `cdp_sys_record_currency__c` for you, since only your script knows what currency its amounts are in. When writing to an object with Currency fields, set it yourself:
+
+```python
+from pyspark.sql.functions import lit
+
+# Either source has Currency fields: keep its cdp_sys_record_currency__c in your output
+df = source.select("id__c", "amount__c", "cdp_sys_record_currency__c")
+
+# OR amounts are in a known currency
+df = df.withColumn("cdp_sys_record_currency__c", lit("EUR"))
+```
+
+- Use an ISO code that exists in your org's configured currency rates.
+- When aggregating, group by the currency code so amounts in different currencies aren't summed together.
+
+See [`docs/examples/script/currency_totals`](docs/examples/script/currency_totals) for a complete example.
+
 ### Streaming (delta) transforms
 
 Streaming BYOC transforms process a Data Lake Object's Change Data Feed continuously instead of reading a bounded snapshot. Use a `StreamingClient` and its `*_deltas` methods in place of the batch `Client` read/write methods:
@@ -226,6 +248,15 @@ query = client.write_dlo_deltas("Output__dll", transformed)
 query.awaitTermination()
 ```
 
+A streaming transform can also read and write Data Model Objects with `read_dmo_deltas()` and `write_dmo_deltas()`:
+
+```python
+deltas = client.read_dmo_deltas()
+transformed = deltas.withColumn("description__c", upper(col("description__c")))
+query = client.write_dmo_deltas("Output__dlm", transformed)
+query.awaitTermination()
+```
+
 Notes:
 
 - These methods only run inside the Data Cloud streaming (`DELTA_SYNC`) runtime. Locally (`datacustomcode run`) they raise `NotImplementedError`, since there is no change feed to stream.
@@ -249,6 +280,29 @@ If none of these exist, `find_file_path` raises `FileNotFoundError` with the lis
 
 `$LIBRARY_PATH` is set automatically to the root of the package at runtime inside Data Cloud.
 
+### Generating lineage
+
+`generate_lineage` runs a script entry point locally and derives the field-level `dataLineage` section of `config.json` from the Spark plans of the DataFrames it writes:
+
+```python
+from datacustomcode.lineage import generate_lineage
+
+result = generate_lineage("payload/entrypoint.py", sf_cli_org="my-org")
+print(result.to_dict()["dataLineage"])
+# [{"targetField": "Output__dll.name__c",
+#   "sourceFields": ["Input__dll.first__c", "Input__dll.last__c"]}, ...]
+```
+
+The entry point runs exactly as it would locally (same config, reader, and credentials), except that writes are traced and validated against the target DLO instead of being performed. It runs in a separate Python process, which inherits your working directory, environment, and `sys.path`; errors raised by the script are re-raised from `generate_lineage`. Reads use the reader's `default_row_limit` (1000 by default; set it in your `config.yaml`).
+
+> **Review the output before you use it.** The generated lineage is a starting point for review, not a guarantee. It reflects only the code paths and data your local run exercised, and it can't see inside UDFs. Accurate and complete `dataLineage` in your package is your responsibility as the package author.
+
+- A target field's sources are the fields its *value* is computed from. Fields that only decide which rows, groups or branch are used aren't sources, however the logic is written: filter conditions, join keys, `groupBy` keys, sort keys, window `partitionBy`/`orderBy` keys, `when()`/`if()` conditions, and aggregate `FILTER (WHERE …)` clauses. For example, `groupBy("region").agg(sum("amount"))` and `sum("amount").over(Window.partitionBy("region"))` both trace to `amount` only.
+- Fields computed only from literals are listed in `noSources`. So are fields whose value doesn't come from any input field's value, such as `row_number()`, `rank()`, `count("*")`, or `sum(when(cond, 1))`.
+- Fields that can't be traced are listed in `unresolved` with a reason and get no lineage. This covers data that didn't come from `read_dlo`/`read_dmo` (for example a `toPandas()` → `createDataFrame` round trip), `mapInPandas`, and subqueries.
+- `pairCount` is the total number of (target, source) field pairs.
+- Only batch script packages are supported.
+
 
 ## CLI
 
@@ -364,6 +418,8 @@ Options:
 #### `datacustomcode zip`
 Zip a transformation job in preparation to upload to Data Cloud. Make sure to change directory into your code package folder (e.g., `my_package`) before running this command.
 
+Symlinks inside the payload folder are not followed or packaged. Each symlinked file that is skipped is reported with a warning, and symlinked directories are not included. `datacustomcode deploy` zips the same way.
+
 Arguments:
 - `PATH`: Path to the code directory i.e. the payload folder (default: "payload")
 

{salesforce_data_customcode-7.0.0rc3 → salesforce_data_customcode-7.0.0rc5}/README.md

@@ -151,13 +151,14 @@ For a batch transform, use `Client`. You should only need the following methods:
 * `find_file_path(file_name)` – Resolve a bundled file (placed under `payload/files/`) to a `pathlib.Path` that exists. Works the same locally and inside Data Cloud — see [Bundled file resolution](#bundled-file-resolution) below for the full lookup order. Raises `FileNotFoundError` if the file isn't found.
 * `read_dlo(name)` – Read from a Data Lake Object by name
 * `read_dmo(name)` – Read from a Data Model Object by name
-* `write_to_dlo(name, spark_dataframe, write_mode)` – Write to a Data Model Object by name with a Spark dataframe
-* `write_to_dmo(name, spark_dataframe, write_mode)` – Write to a Data Lake Object by name with a Spark dataframe
+* `write_to_dlo(name, spark_dataframe, write_mode)` – Write to a Data Lake Object by name with a Spark dataframe
+* `write_to_dmo(name, spark_dataframe, write_mode)` – Write to a Data Model Object by name with a Spark dataframe
 
 For a streaming (delta) transform, use `StreamingClient`, which exposes the streaming counterparts:
 * `read_dlo_deltas()` – Read the streaming change feed (deltas) of a Data Lake Object as a streaming DataFrame.
 * `read_dmo_deltas()` – Read the streaming change feed (deltas) of a Data Model Object as a streaming DataFrame.
 * `write_dlo_deltas(name, spark_dataframe)` – Write a streaming DataFrame of deltas to a Data Lake Object; returns the started `StreamingQuery`
+* `write_dmo_deltas(name, spark_dataframe)` – Write a streaming DataFrame of deltas to a Data Model Object; returns the started `StreamingQuery`
 
 `find_file_path`, `llm_gateway_generate_text`, and `einstein_predict` are available on both clients.
 
@@ -176,6 +177,27 @@ client.write_to_dlo('output_DLO')
 > [!WARNING]
 > Currently we only support reading from DMOs and writing to DMOs or reading from DLOs and writing to DLOs, but they cannot mix.
 
+### Writing Currency fields
+
+A DLO or DMO with a Currency field has a system field, `cdp_sys_record_currency__c`, that holds each record's ISO currency code. Data Explorer UI uses it to display currency values, and shows a blank value if it's null (Query Editor still shows it).
+
+Code Extensions don't set `cdp_sys_record_currency__c` for you, since only your script knows what currency its amounts are in. When writing to an object with Currency fields, set it yourself:
+
+```python
+from pyspark.sql.functions import lit
+
+# Either source has Currency fields: keep its cdp_sys_record_currency__c in your output
+df = source.select("id__c", "amount__c", "cdp_sys_record_currency__c")
+
+# OR amounts are in a known currency
+df = df.withColumn("cdp_sys_record_currency__c", lit("EUR"))
+```
+
+- Use an ISO code that exists in your org's configured currency rates.
+- When aggregating, group by the currency code so amounts in different currencies aren't summed together.
+
+See [`docs/examples/script/currency_totals`](docs/examples/script/currency_totals) for a complete example.
+
 ### Streaming (delta) transforms
 
 Streaming BYOC transforms process a Data Lake Object's Change Data Feed continuously instead of reading a bounded snapshot. Use a `StreamingClient` and its `*_deltas` methods in place of the batch `Client` read/write methods:
@@ -201,6 +223,15 @@ query = client.write_dlo_deltas("Output__dll", transformed)
 query.awaitTermination()
 ```
 
+A streaming transform can also read and write Data Model Objects with `read_dmo_deltas()` and `write_dmo_deltas()`:
+
+```python
+deltas = client.read_dmo_deltas()
+transformed = deltas.withColumn("description__c", upper(col("description__c")))
+query = client.write_dmo_deltas("Output__dlm", transformed)
+query.awaitTermination()
+```
+
 Notes:
 
 - These methods only run inside the Data Cloud streaming (`DELTA_SYNC`) runtime. Locally (`datacustomcode run`) they raise `NotImplementedError`, since there is no change feed to stream.
@@ -224,6 +255,29 @@ If none of these exist, `find_file_path` raises `FileNotFoundError` with the lis
 
 `$LIBRARY_PATH` is set automatically to the root of the package at runtime inside Data Cloud.
 
+### Generating lineage
+
+`generate_lineage` runs a script entry point locally and derives the field-level `dataLineage` section of `config.json` from the Spark plans of the DataFrames it writes:
+
+```python
+from datacustomcode.lineage import generate_lineage
+
+result = generate_lineage("payload/entrypoint.py", sf_cli_org="my-org")
+print(result.to_dict()["dataLineage"])
+# [{"targetField": "Output__dll.name__c",
+#   "sourceFields": ["Input__dll.first__c", "Input__dll.last__c"]}, ...]
+```
+
+The entry point runs exactly as it would locally (same config, reader, and credentials), except that writes are traced and validated against the target DLO instead of being performed. It runs in a separate Python process, which inherits your working directory, environment, and `sys.path`; errors raised by the script are re-raised from `generate_lineage`. Reads use the reader's `default_row_limit` (1000 by default; set it in your `config.yaml`).
+
+> **Review the output before you use it.** The generated lineage is a starting point for review, not a guarantee. It reflects only the code paths and data your local run exercised, and it can't see inside UDFs. Accurate and complete `dataLineage` in your package is your responsibility as the package author.
+
+- A target field's sources are the fields its *value* is computed from. Fields that only decide which rows, groups or branch are used aren't sources, however the logic is written: filter conditions, join keys, `groupBy` keys, sort keys, window `partitionBy`/`orderBy` keys, `when()`/`if()` conditions, and aggregate `FILTER (WHERE …)` clauses. For example, `groupBy("region").agg(sum("amount"))` and `sum("amount").over(Window.partitionBy("region"))` both trace to `amount` only.
+- Fields computed only from literals are listed in `noSources`. So are fields whose value doesn't come from any input field's value, such as `row_number()`, `rank()`, `count("*")`, or `sum(when(cond, 1))`.
+- Fields that can't be traced are listed in `unresolved` with a reason and get no lineage. This covers data that didn't come from `read_dlo`/`read_dmo` (for example a `toPandas()` → `createDataFrame` round trip), `mapInPandas`, and subqueries.
+- `pairCount` is the total number of (target, source) field pairs.
+- Only batch script packages are supported.
+
 
 ## CLI
 
@@ -339,6 +393,8 @@ Options:
 #### `datacustomcode zip`
 Zip a transformation job in preparation to upload to Data Cloud. Make sure to change directory into your code package folder (e.g., `my_package`) before running this command.
 
+Symlinks inside the payload folder are not followed or packaged. Each symlinked file that is skipped is reported with a warning, and symlinked directories are not included. `datacustomcode deploy` zips the same way.
+
 Arguments:
 - `PATH`: Path to the code directory i.e. the payload folder (default: "payload")
 

{salesforce_data_customcode-7.0.0rc3 → salesforce_data_customcode-7.0.0rc5}/pyproject.toml

@@ -18,7 +18,7 @@ license = "Apache-2.0"
 name = "salesforce-data-customcode"
 readme = "README.md"
 requires-python = ">=3.10,<3.12"
-version = "7.0.0rc3"
+version = "7.0.0rc5"
 
 [tool.black]
 exclude = '''
@@ -88,10 +88,10 @@ warn_unused_configs = true
 
 [tool.poetry]
 include = [
-  {path = "src/datacustomcode/templates/**/*"},
-  {path = "src/datacustomcode/config.yaml"}
+  { path = "src/datacustomcode/templates/**/*" },
+  { path = "src/datacustomcode/config.yaml" }
 ]
-packages = [{include = "datacustomcode", from = "src"}]
+packages = [{ include = "datacustomcode", from = "src" }]
 
 [tool.poetry.build]
 generate-setup-file = false
@@ -115,7 +115,7 @@ coverage = ">=7.0.0,<8.0.0"
 ipykernel = "^6.29.5"
 mypy = "*"
 pipreqs = "*"
-poetry-dynamic-versioning = {extras = ["plugin"], version = "^1.8.2"}
+poetry-dynamic-versioning = { extras = ["plugin"], version = "^1.8.2" }
 pre-commit = "*"
 pytest = "*"
 pytest-cov = "*"
@@ -127,7 +127,7 @@ types-requests = "*"
 [tool.poetry.plugins."poetry.plugin"]
 
 [tool.poetry.requires-plugins]
-poetry-dynamic-versioning = {version = ">=1.0.0,<2.0.0", extras = ["plugin"]}
+poetry-dynamic-versioning = { version = ">=1.0.0,<2.0.0", extras = ["plugin"] }
 
 [tool.poetry.scripts]
 datacustomcode = "datacustomcode.cli:cli"
@@ -152,7 +152,6 @@ testpaths = ["tests"]
 [tool.ruff]
 fix = true
 line-length = 88
-target-version = 'py310'
 lint.ignore = [
   # do not assign a lambda expression, use a def
   'E731',
@@ -211,6 +210,7 @@ lint.select = [
   'RUF',
   'S102'
 ]
+target-version = 'py310'
 
 [tool.setuptools.package_data]
 datacustomcode = ["templates/**/*", "config.yaml"]

{salesforce_data_customcode-7.0.0rc3 → salesforce_data_customcode-7.0.0rc5}/src/datacustomcode/client.py

@@ -231,11 +231,12 @@ def einstein_predict_col(
 def named_credential_request_col(
     request: "HTTPRequest",
     body: Optional["Column"] = None,
+    path: Optional["Column"] = None,
 ) -> "Column":
     """Build a Spark Column that makes one Named Credential callout per row.
 
-    The endpoint, method, and headers are fixed for the call (taken from
-    ``request``); only ``body`` varies per row. Use this instead of
+    The method and headers are fixed for the call (taken from ``request``);
+    ``body`` and, optionally, ``path`` vary per row. Use this instead of
     :meth:`Client.named_credential_request` when the callout runs across a
     DataFrame so each row is dispatched independently rather than one-shot on
     the driver.
@@ -253,13 +254,20 @@ def named_credential_request_col(
             headers are applied to every row.
         body: Optional per-row ``Column`` holding the request body as a
             string (or null for no body).
+        path: Optional per-row ``Column`` appended to ``request.url``, e.g.
+            ``concat(lit("?address="), col("address"))`` or ``lit("/v1/models")``.
+            The callout always stays on ``request.url``'s Named Credential: a
+            path not starting with ``/`` or ``?`` is joined with ``/``. A null
+            or empty row value uses ``request.url`` as is.
 
     Returns:
         A Spark ``Column`` of ``StructType`` with fields ``status``,
         ``response``, ``error_code``, and ``error_message``.
     """
     named_credential = Client()._get_spark_named_credential()
-    return named_credential.request_col(request, body=body)
+    if path is None:
+        return named_credential.request_col(request, body=body)
+    return named_credential.request_col(request, body=body, path=path)
 
 
 class DataCloudObjectType(Enum):
@@ -368,7 +376,13 @@ class _BaseClient:
             instance._spark_named_credential = spark_named_credential
             # Initialize Readers and Writers from config
             # and/or provided reader and writer
-            if reader is None or writer is None:
+            reader_forced = (
+                config.reader_config is not None and config.reader_config.force
+            )
+            writer_forced = (
+                config.writer_config is not None and config.writer_config.force
+            )
+            if reader is None or writer is None or reader_forced or writer_forced:
                 # We need a spark because we will initialize readers and writers.
                 # Reuse the process-wide session if one client already built it,
                 # so a Client and a StreamingClient share a single connection.
@@ -396,9 +410,7 @@ class _BaseClient:
                 raise ValueError(
                     "Reader config is required when reader is not provided"
                 )
-            elif reader is None or (
-                config.reader_config is not None and config.reader_config.force
-            ):
+            elif reader is None or reader_forced:
                 reader_init = config.reader_config.to_object(spark)  # type: ignore
             else:
                 reader_init = reader
@@ -406,9 +418,7 @@ class _BaseClient:
                 raise ValueError(
                     "Writer config is required when writer is not provided"
                 )
-            elif writer is None or (
-                config.writer_config is not None and config.writer_config.force
-            ):
+            elif writer is None or writer_forced:
                 writer_init = config.writer_config.to_object(spark)  # type: ignore
             else:
                 writer_init = writer
@@ -716,6 +726,25 @@ class StreamingClient(_BaseClient):
         self._validate_data_layer_history_does_not_contain(DataCloudObjectType.DMO)
         return self._writer.write_dlo_deltas(name, dataframe, **kwargs)  # type: ignore[no-any-return]
 
+    def write_dmo_deltas(
+        self, name: str, dataframe: PySparkDataFrame, **kwargs
+    ) -> StreamingQuery:
+        """Write a streaming DataFrame of deltas to a DMO in Data Cloud.
+
+        Starts a streaming query that writes each micro-batch to the
+        target DMO and returns the ``StreamingQuery`` handle; the caller
+        typically calls ``query.awaitTermination()``.
+
+        Args:
+            name: The name of the DMO to write to.
+            dataframe: The streaming PySpark DataFrame to write.
+
+        Returns:
+            The started ``StreamingQuery``.
+        """
+        self._validate_data_layer_history_does_not_contain(DataCloudObjectType.DLO)
+        return self._writer.write_dmo_deltas(name, dataframe, **kwargs)  # type: ignore[no-any-return]
+
     def auto_write_to_dlo(self, name: str, dataframe: PySparkDataFrame) -> None:
         """Write a PySpark DataFrame to a DLO in Data Cloud automatically picking
         the WriteMode.

{salesforce_data_customcode-7.0.0rc3 → salesforce_data_customcode-7.0.0rc5}/src/datacustomcode/deploy.py

@@ -415,13 +415,7 @@ class DataTransformConfig(BaseConfig):
     def _validate_layers(self) -> "DataTransformConfig":
         read_is_dlo = isinstance(self.permissions.read, DloPermission)
         write_is_dlo = isinstance(self.permissions.write, DloPermission)
-        if self.is_streaming:
-            if not write_is_dlo:
-                raise ValueError(
-                    "A streaming transform must write to a DLO "
-                    "(permissions.write must be a 'dlo' entry)."
-                )
-        elif read_is_dlo != write_is_dlo:
+        if read_is_dlo != write_is_dlo:
             raise ValueError(
                 "permissions.read and permissions.write must both reference "
                 "DLOs or both reference DMOs (got "
@@ -630,12 +624,15 @@ def zip(
     logger.debug(f"Zipping directory... {directory}")
 
     with zipfile.ZipFile(ZIP_FILE_NAME, "w", zipfile.ZIP_DEFLATED) as zipf:
-        for root, dirs, files in os.walk(directory):
-            # Skip .DS_Store and local credentials.
+        for root, dirs, files in os.walk(directory, followlinks=False):
+            # Skip symlinks, .DS_Store and local credentials.
             for file in files:
+                abs_path = os.path.join(root, file)
+                arcname = os.path.relpath(abs_path, directory)
+                if os.path.islink(abs_path):
+                    logger.warning(f"Skipping symlink file {arcname}")
+                    continue
                 if file not in EXCLUDED_FILES:
-                    abs_path = os.path.join(root, file)
-                    arcname = os.path.relpath(abs_path, directory)
                     zipf.write(abs_path, arcname)
 
     logger.debug(f"Created zip file: {ZIP_FILE_NAME}")

{salesforce_data_customcode-7.0.0rc3 → salesforce_data_customcode-7.0.0rc5}/src/datacustomcode/io/writer/base.py

@@ -87,7 +87,7 @@ class BaseDataCloudWriter(BaseDataAccessLayer):
         Args:
             name: Target Data Lake Object name.
             dataframe: Streaming PySpark DataFrame produced from a
-                ``read_dlo_deltas`` / ``read_dmo_deltas`` source.
+                ``read_dlo_deltas`` source.
 
         Returns:
             The started ``StreamingQuery``; the caller drives its lifecycle
@@ -102,3 +102,35 @@ class BaseDataCloudWriter(BaseDataAccessLayer):
             "streaming runtime; the local writer does not support streaming "
             "deltas."
         )
+
+    def write_dmo_deltas(
+        self, name: str, dataframe: PySparkDataFrame
+    ) -> StreamingQuery:
+        """Write a streaming DataFrame of deltas to a Data Model Object.
+
+        Streaming counterpart to :meth:`write_to_dmo`. Starts a streaming query
+        that writes each micro-batch to the target DMO via the Data Cloud
+        streaming sink and returns the resulting ``StreamingQuery`` handle. The
+        runtime owns the trigger and checkpoint location; callers pass only the
+        table name. Concrete streaming behavior is provided by the deployed
+        Data Cloud runtime; the base implementation raises
+        :class:`NotImplementedError`.
+
+        Args:
+            name: Target Data Model Object name.
+            dataframe: Streaming PySpark DataFrame produced from a
+                ``read_dmo_deltas`` source.
+
+        Returns:
+            The started ``StreamingQuery``; the caller drives its lifecycle
+            (typically ``query.awaitTermination()``).
+
+        Raises:
+            NotImplementedError: If the active writer does not support streaming
+                deltas (e.g. the local development writers).
+        """
+        raise NotImplementedError(
+            "write_dmo_deltas is only supported when running in the Data Cloud "
+            "streaming runtime; the local writer does not support streaming "
+            "deltas."
+        )

salesforce_data_customcode-7.0.0rc5/src/datacustomcode/lineage/__init__.py

@@ -0,0 +1,137 @@
+# Copyright (c) 2025, Salesforce, Inc.
+# SPDX-License-Identifier: Apache-2
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Generate config.json ``dataLineage`` by tracing a script's Spark plans."""
+
+from __future__ import annotations
+
+import builtins
+import json
+import os
+import subprocess
+import sys
+import tempfile
+from typing import (
+    Any,
+    Dict,
+    NoReturn,
+    Optional,
+    Sequence,
+)
+
+from datacustomcode.lineage.capture import (
+    LineageCaptureReader,
+    LineageCaptureWriter,
+    LineageCollector,
+    LineageResult,
+    install_capture,
+)
+from datacustomcode.lineage.tracer import LineageTraceError
+
+__all__ = [
+    "LineageCaptureReader",
+    "LineageCaptureWriter",
+    "LineageCollector",
+    "LineageResult",
+    "LineageTraceError",
+    "generate_lineage",
+    "install_capture",
+]
+
+
+def generate_lineage(
+    entrypoint: str,
+    config_file: Optional[str] = None,
+    dependencies: Sequence[str] = (),
+    profile: str = "default",
+    sf_cli_org: Optional[str] = None,
+) -> LineageResult:
+    """Run a script entrypoint and return the field-level lineage it produces.
+
+    The entrypoint runs exactly as with ``run`` (same config resolution, reader
+    and credentials), except that writes are traced instead of performed. It runs
+    in a fresh Python process, so no client or config from the caller's process
+    leaks into the capture. The process inherits this one's working directory,
+    environment, ``sys.path`` and stdout/stderr.
+
+    Args:
+        entrypoint: The script entrypoint to run.
+        config_file: Optional SDK config file, as for ``run``.
+        dependencies: Modules to import before running, as for ``run``.
+        profile: The credentials profile to use.
+        sf_cli_org: Optional SF CLI org alias or username.
+
+    Returns:
+        The captured :class:`LineageResult`.
+
+    Raises:
+        ValueError: If the package is not a batch script.
+        Exception: Whatever the run raised, re-raised as the same built-in
+            exception type or :class:`LineageTraceError`, else ``RuntimeError``.
+    """
+    with tempfile.TemporaryDirectory(prefix="datacustomcode-lineage-") as tmp:
+        request_file = os.path.join(tmp, "request.json")
+        response_file = os.path.join(tmp, "response.json")
+        with open(request_file, "w", encoding="utf-8") as f:
+            json.dump(
+                {
+                    "args": {
+                        "entrypoint": entrypoint,
+                        "config_file": config_file,
+                        "dependencies": list(dependencies),
+                        "profile": profile,
+                        "sf_cli_org": sf_cli_org,
+                    },
+                    "response_file": response_file,
+                },
+                f,
+            )
+        # The worker sees the same modules this process does
+        env = dict(os.environ)
+        env["PYTHONPATH"] = os.pathsep.join(path for path in sys.path if path)
+        completed = subprocess.run(
+            [sys.executable, "-m", "datacustomcode.lineage._worker", request_file],
+            env=env,
+            check=False,
+        )
+        if not os.path.exists(response_file):
+            raise RuntimeError(
+                f"Lineage generation exited with code {completed.returncode} "
+                "without a result"
+            )
+        with open(response_file, encoding="utf-8") as f:
+            response = json.load(f)
+    if "error" in response:
+        _raise(response["error"])
+    return LineageResult.from_dict(response["result"])
+
+
+def _raise(error: Dict[str, Any]) -> NoReturn:
+    """Re-raise an error reported by the worker as the closest local type."""
+    message = error["message"]
+    if error["type"] == LineageTraceError.__name__:
+        raise LineageTraceError(message)
+    exc_type = getattr(builtins, error["type"], None)
+    if (
+        error["module"] == "builtins"
+        and isinstance(exc_type, type)
+        and issubclass(exc_type, Exception)
+    ):
+        try:
+            exc = exc_type(message)
+        except TypeError:  # e.g. UnicodeDecodeError takes more arguments
+            pass
+        else:
+            raise exc
+    raise RuntimeError(f"{error['module']}.{error['type']}: {message}")