expops 0.1.19.dev0__tar.gz → 0.1.22.dev0__tar.gz

{expops-0.1.19.dev0 → expops-0.1.22.dev0}/.github/workflows/release.yml

@@ -40,7 +40,7 @@ jobs:
       id-token: write
     environment:
       name: pypi
-      url: https://pypi.org/project/mlops-platform/
+      url: https://pypi.org/project/expops/
     steps:
       - name: Download dist artifacts
         uses: actions/download-artifact@v4

{expops-0.1.19.dev0 → expops-0.1.22.dev0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: expops
-Version: 0.1.19.dev0
+Version: 0.1.22.dev0
 Summary: MLOps Platform with step-based pipeline execution
 License:                     GNU GENERAL PUBLIC LICENSE
                                Version 3, 29 June 2007
@@ -722,6 +722,7 @@ The installed CLI command is **`expops`**.
 
 ### Prerequisites
 
+- Python 3.10
 - Git installed and configured
 - Access to the project repository
 - Required dependencies installed
@@ -731,7 +732,7 @@ The installed CLI command is **`expops`**.
 ```bash
 # Clone the repository
 git clone <repository-url>
-cd mlops-platform
+cd expops-platform
 
 # Create and activate a virtual environment
 python -m venv .venv

{expops-0.1.19.dev0 → expops-0.1.22.dev0}/README.md

@@ -16,6 +16,7 @@ The installed CLI command is **`expops`**.
 
 ### Prerequisites
 
+- Python 3.10
 - Git installed and configured
 - Access to the project repository
 - Required dependencies installed
@@ -25,7 +26,7 @@ The installed CLI command is **`expops`**.
 ```bash
 # Clone the repository
 git clone <repository-url>
-cd mlops-platform
+cd expops-platform
 
 # Create and activate a virtual environment
 python -m venv .venv

{expops-0.1.19.dev0 → expops-0.1.22.dev0}/docs/features/caching.md

@@ -7,12 +7,12 @@ ExpOps provides intelligent multi-level caching and reproducibility guarantees.
 When caching is enabled, the platform separates **lightweight manifests** from **heavy artifacts**:
 
 - **Step/process manifests (`.pkl` files)**:
-    - Stored under a cache prefix (for example `cache/steps/steps/` in GCS).
+    - In GCS, stored under a project-scoped cache path: `gs://<bucket>/<project_id>/cache/steps/<run_id>/...`.
     - Contain small Python dictionaries describing the result of a step or process.
     - Heavy artifacts (models, large arrays) are represented as *references* rather than inlined objects.
-- **Models (`models/` prefix)**:
+- **Models (`model_spill/` prefix)**:
     - Any recognized model object (for example, most `sklearn` and `xgboost` estimators) is always serialized separately.
-    - Stored as `.joblib` blobs under a `models/<run_id>/<process_name>/...` path.
+    - Stored as `.joblib` blobs under a `model_spill/<process_name>/...` path.
 - **Payloads (`payloads/` prefix)**:
     - Large array-like payloads (NumPy arrays, lists/tuples coercible to arrays) above a small size threshold are stored as compressed `.npz` blobs.
 
@@ -66,7 +66,26 @@ Remote backend for shared caching:
 - Persistent storage.
 - Requires GCP credentials.
 
-When an object store is configured, manifests are written as `.pkl` blobs and heavy artifacts are written under the `models/` and `payloads/` prefixes in the same bucket. When no object store is configured, all of these artifacts are stored on the local filesystem under the project cache directory.
+When an object store is configured, manifests are written as `.pkl` blobs and heavy artifacts are written under the `model_spill/` and `payloads/` prefixes in the same bucket. When no object store is configured, all of these artifacts are stored on the local filesystem under the project cache directory.
+
+With GCS enabled, the layout within the bucket is project-scoped:
+
+- Step/process manifests are stored under `gs://<bucket>/<project_id>/cache/steps/<run_id>/...`.
+- Spilled models are stored under `gs://<bucket>/<project_id>/model_spill/<process_name>/...`.
+- Spilled large payloads are stored under `gs://<bucket>/<project_id>/payloads/<run_id>/<process_name>/...`.
+
+When no object store is configured, manifests and heavy artifacts are stored on the local filesystem under the project's hidden runtime cache directory, typically under:
+
+- `.<project_id>/cache/<version_hash>/<encoded_probe_path>/steps/` for step/process manifests.
+- `.<project_id>/cache/<version_hash>/<encoded_probe_path>/model_spill/` for spilled models.
+- `.<project_id>/cache/<version_hash>/<encoded_probe_path>/payloads/` for spilled large payloads.
+
+**Artifacts** (charts, reproducibility outputs) use the same version/probe layout for consistency:
+
+- **Local**: `.<project_id>/artifacts/<version_hash>/<encoded_probe_path>/<filename>.png`
+- **GCS**: `gs://<bucket>/<project_id>/artifacts/<version_hash>/<encoded_probe_path>/<filename>.png`
+
+Both cache and artifacts share the same `version_hash` (from project config + scripts) and `encoded_probe_path` (from process/step identity).
 
 ## Configuration
 

{expops-0.1.19.dev0 → expops-0.1.22.dev0}/docs/features/reporting.md

@@ -16,8 +16,8 @@ Functions decorated with `@chart()` generate visualizations. **Chart functions h
 
 **Every static chart function MUST:**
 1. Accept `metrics` as the first parameter (Dict[str, Any])
-2. Accept `ctx` as the second parameter (ChartContext)
-3. Use `ctx.savefig()` to save figures
+2. Accept `ctx` as the second parameter (ChartContext), for metrics and optional context
+3. Use `plt.savefig()` to save figures (the process runs with cwd set to the output directory; all files written there are synced to artifacts)
 
 ```python
 from expops.reporting import chart, ChartContext
@@ -29,7 +29,7 @@ def plot_metrics(metrics: Dict[str, Any], ctx: ChartContext) -> None:
     """
     Chart function signature requirements:
     - metrics: Dict containing metrics from probe_paths (REQUIRED)
-    - ctx: ChartContext for saving figures (REQUIRED)
+    - ctx: ChartContext for metrics and run context (REQUIRED)
     - Returns: None (void function)
     """
     # Access metrics directly from the metrics dict
@@ -45,8 +45,8 @@ def plot_metrics(metrics: Dict[str, Any], ctx: ChartContext) -> None:
     fig, ax = plt.subplots(figsize=(10, 6))
     # ... plotting code ...
     
-    # MUST use ctx.savefig() to save the figure
-    ctx.savefig('plot_metrics.png', fig=fig, dpi=150)
+    # Use plt.savefig() with a relative path; cwd is the process output dir
+    plt.savefig('plot_metrics.png', dpi=150)
     plt.close(fig)
 ```
 
@@ -77,6 +77,7 @@ The chart function receives:
 ```python
 @chart()
 def my_chart(metrics: Dict[str, Any], ctx: ChartContext) -> None:
+    import matplotlib.pyplot as plt
     # metrics['train'] contains metrics from train_model process
     train_data = metrics.get('train', {})
     
@@ -113,10 +114,11 @@ Common patterns:
 
 ### Output
 
-Static charts produce image files (PNG) saved to:
+Static charts produce image files (PNG) saved under the unified artifacts layout:
 ```
-artifacts/charts/<run-id>/
+.<project_id>/artifacts/<version_hash>/<encoded_probe_path>/<chart_name>.png
 ```
+In GCS: `gs://<bucket>/<project_id>/artifacts/<version_hash>/<encoded_probe_path>/<chart_name>.png`
 
 ## Dynamic Charts
 

{expops-0.1.19.dev0 → expops-0.1.22.dev0}/docs/getting-started/creating-a-project.md

@@ -10,6 +10,13 @@ Templates provide a pre-configured project structure with example code:
 expops create my-project --template sklearn-basic
 ```
 
+You can also create *in the current directory* (project name inferred from the folder name):
+
+```bash
+mkdir my-project && cd my-project
+expops create --template sklearn-basic
+```
+
 Available templates:
 - `sklearn-basic`: Runnable project skeleton with a tiny sklearn model
 - `premier-league`: Comprehensive ML project with cluster config and dynamic charts
@@ -26,6 +33,13 @@ expops create my-project
 
 This creates a minimal project structure that you can customize.
 
+You can also create *in the current directory* (project name inferred from the folder name):
+
+```bash
+mkdir my-project && cd my-project
+expops create
+```
+
 ## Project Structure
 
 After creation, your project will have the following structure:
@@ -33,15 +47,28 @@ After creation, your project will have the following structure:
 ```
 my-project/
 ├── .gitignore
+├── .my-project/
+│   ├── envs/
+│   ├── logs/
+│   ├── cache/<version_hash>/<encoded_probe_path>/
+│   ├── artifacts/<version_hash>/<encoded_probe_path>/
+│   └── metrics.sqlite
 ├── configs/
 │   └── project_config.yaml
-├── models/
-├── charts/
+├── src/
+│   ├── models/
+│   └── charts/
 ├── data/
 ├── requirements.txt
 └── ...
 ```
 
+The `.my-project/envs/` directory holds the Python virtual environments for
+this project (for example `.my-project/envs/my-project-env`). Older versions
+of the platform used a workspace-level `.venvs/` folder; that location is now
+deprecated for project runs, and new environments are created only under each
+project’s hidden `.my-project/envs/` directory.
+
 ## Configuration Notes
 
 ### Caching and Web UI

{expops-0.1.19.dev0 → expops-0.1.22.dev0}/docs/getting-started/quick-start.md

@@ -67,7 +67,7 @@ experiment:
       # object_store:
       #   type: gcs
       #   bucket: <your-gcs-bucket-name>
-      #   prefix: sklearn-basic/cache/steps
+      #   # Manifests will be stored under gs://<bucket>/<project_id>/cache/steps/ automatically.
 ```
 
 **Setup steps for GCP (Firestore)**:

{expops-0.1.19.dev0 → expops-0.1.22.dev0}/docs/project-structure/configuration.md

@@ -20,6 +20,7 @@ environment:       # Named environments (requirements/env files + type)
 reproducibility:   # Random seed configuration
 data:              # Data sources and data path for hashing
 experiment:        # Model framework, paths, parameters, pipeline, cache
+execution:         # Optional: e.g. execution.workspace.base_dir (temp dir; default tmp)
 reporting:         # Chart entrypoints, probe paths, reporting environment
 ```
 
@@ -36,6 +37,7 @@ reporting:         # Chart entrypoints, probe paths, reporting environment
   - See [Caching & Reproducibility](../features/caching.md) and [Backends](../advanced/backends.md) for details
 - **`reporting`**: Chart entrypoints and chart definitions
   - See [Reporting Features](../features/reporting.md) for details
+- **`execution.workspace`**: Optional. Per-process temporary workspace base path. Set `base_dir` to control where process run directories are created (e.g. `base_dir: "/tmp"`). Default is `tmp` (i.e. `/tmp` on Unix).
 
 ### Defaults that reduce config
 

expops-0.1.22.dev0/docs/project-structure/overview.md

@@ -0,0 +1,59 @@
+# Project Structure Overview
+
+Each ExpOps project follows a standardized directory structure that organizes configuration, code, data, and artifacts.
+
+## Directory Layout
+
+```
+my-project/
+├── configs/
+│   ├── project_config.yaml      # Main project configuration
+│   └── compute_config.yaml      # Optional cluster configuration
+├── src/
+│   ├── <model_name>.py          # Model implementation (main script)
+│   ├── plot_metrics.py           # Static chart generation
+│   ├── plot_metrics.js          # Optional dynamic chart generation
+│   └── ...                      # Other project modules
+├── data/                         # Input datasets
+├── requirements.txt              # Main project dependencies
+├── requirements-charts.txt       # Chart/reporting dependencies
+└── .<project_id>/                # Runtime directory (hidden)
+    ├── envs/                     # Project virtual environments
+    ├── logs/                     # Execution logs
+    ├── cache/                    # Step/process cache
+    │   └── <version_hash>/<encoded_probe_path>/
+    └── artifacts/                # Charts and other artifacts
+        └── <version_hash>/<encoded_probe_path>/
+```
+
+For GCP-backed backends (e.g. Firestore), set the path to your credentials file in your config (e.g. `credentials_json: src/firestore.json` or `credentials_json: firestore.json`).
+
+## Key Components
+
+### Configuration Files
+
+The `configs/` directory contains all project configuration:
+- **project_config.yaml**: Main configuration (required)
+- **compute_config.yaml**: Cluster execution settings (optional)
+
+See [Configuration Files](configuration.md) for details.
+
+### Model Code
+
+The `src/` directory contains your ML pipeline and reporting scripts. The main model script (e.g. `<model_name>.py`) is referenced in `project_config.yaml` under `scripts.main`:
+- Process definitions with `@process()` decorator
+- Step functions with `@step()` decorator
+- Pipeline logic and data transformations
+
+See [Model Code](model-code.md) for details.
+
+### Chart Generation
+
+Chart scripts live under `src/` and are referenced in config (`scripts.reporting`, `scripts.reporting_js`):
+- **plot_metrics.py**: Static PNG chart generation
+- **plot_metrics.js**: Optional dynamic interactive charts
+
+### Dependencies
+
+- **requirements.txt**: Main dependencies for training/inference
+- **requirements-charts.txt**: Reporting/chart dependencies

{expops-0.1.19.dev0 → expops-0.1.22.dev0}/docs/templates/premier-league.md

@@ -20,19 +20,33 @@ expops create my-project --template premier-league
 
 ```
 my-project/
+├── .my-project/
+│   ├── envs/
+│   ├── logs/
+│   ├── cache/<version_hash>/<encoded_probe_path>/
+│   ├── artifacts/<version_hash>/<encoded_probe_path>/
+│   └── metrics.sqlite
 ├── configs/
 │   ├── project_config.yaml
 │   └── compute_config.yaml
-├── models/
-│   └── premier_league_model.py
-├── charts/
-│   ├── plot_metrics.py
-│   ├── plot_metrics.js
-│   └── requirements.txt
+├── src/
+│   ├── models/
+│   │   └── premier_league_model.py
+│   └── charts/
+│       ├── plot_metrics.py
+│       └── plot_metrics.js
 ├── requirements.txt
+├── requirements-charts.txt
 └── data/
+    └── England CSV.csv
 ```
 
+When you run the template, its virtual environments are created under the
+project-local hidden `.my-project/envs/` directory (for example
+`.my-project/envs/premier-league-env`). The older workspace-level `.venvs/`
+directory is kept only for legacy, non-project usage and is no longer used
+for project runs.
+
 ## Features Demonstrated
 
 ### Distributed Execution

{expops-0.1.19.dev0 → expops-0.1.22.dev0}/docs/templates/sklearn-basic.md

@@ -20,17 +20,31 @@ expops create my-project --template sklearn-basic
 
 ```
 my-project/
+├── .my-project/
+│   ├── envs/
+│   ├── logs/
+│   ├── cache/<version_hash>/<encoded_probe_path>/
+│   ├── artifacts/<version_hash>/<encoded_probe_path>/
+│   └── metrics.sqlite
 ├── configs/
 │   └── project_config.yaml
-├── models/
-│   └── sklearn_model.py
-├── charts/
-│   ├── plot_metrics.py
-│   └── requirements.txt
+├── src/
+│   ├── models/
+│   │   └── model.py
+│   └── charts/
+│       └── plot_metrics.py
 ├── requirements.txt
+├── requirements-charts.txt
 └── data/
+    └── train.csv
 ```
 
+Project-specific Python virtual environments are created under the hidden
+runtime directory `.my-project/envs/` (for example
+`.my-project/envs/sklearn-basic-env`). Earlier versions of the platform used
+a workspace-level `.venvs/` directory; that location is now deprecated for
+project runs.
+
 ## Running
 
 Run locally:
@@ -49,6 +63,6 @@ The template uses:
 ## Customization
 
 You can customize:
-- Experiment parameters in `project_config.yaml`
-- Pipeline steps in `models/sklearn_model.py`
-- Chart visualizations in `charts/plot_metrics.py`
+- Experiment parameters in `configs/project_config.yaml`
+- Pipeline steps in `src/models/model.py`
+- Chart visualizations in `src/charts/plot_metrics.py`

{expops-0.1.19.dev0 → expops-0.1.22.dev0}/pyproject.toml

@@ -55,8 +55,8 @@ include-package-data = true
 where = ["src"]
 
 [tool.setuptools.package-data]
-"mlops.web" = ["ui/*"]
-"mlops" = [
+"expops.web" = ["ui/*"]
+"expops" = [
   "templates/*/*",
   "templates/*/*/*",
   "templates/*/*/*/*",
@@ -64,9 +64,12 @@ where = ["src"]
 
 [tool.setuptools_scm]
 local_scheme = "no-local-version"
-write_to = "src/mlops/_version.py"
+write_to = "src/expops/_version.py"
+# Use this version when building without git (e.g. on a cluster with only copied files)
+fallback_version = "0.0.0"
 
 [tool.pytest.ini_options]
 pythonpath = ["src"]
 
-
+[tool.ruff.lint]
+ignore = ["F401", "F841", "E402"]

{expops-0.1.19.dev0/src/mlops → expops-0.1.22.dev0/src/expops}/__main__.py

@@ -2,7 +2,7 @@
 """
 MLOps Platform __main__ module
 
-Allows the package to be run with `python -m mlops.main`
+Allows the package to be run with `python -m expops.main`
 """
 
 from .main import main

{expops-0.1.19.dev0/src/mlops → expops-0.1.22.dev0/src/expops}/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '0.1.19.dev0'
-__version_tuple__ = version_tuple = (0, 1, 19, 'dev0')
+__version__ = version = '0.1.22.dev0'
+__version_tuple__ = version_tuple = (0, 1, 22, 'dev0')
 
-__commit_id__ = commit_id = 'g4b7fdd9d1'
+__commit_id__ = commit_id = 'ga532cd5e9'

{expops-0.1.19.dev0/src/mlops → expops-0.1.22.dev0/src/expops}/adapters/__init__.py

@@ -1,7 +1,7 @@
 """Adapter plugins package.
 
 This package contains subpackages that each expose an `Adapter` symbol
-for plugin discovery (e.g., `mlops.adapters.custom.Adapter`).
+for plugin discovery (e.g., `expops.adapters.custom.Adapter`).
 """
 
 __all__ = [

{expops-0.1.19.dev0/src/mlops → expops-0.1.22.dev0/src/expops}/adapters/config_schema.py

@@ -66,7 +66,6 @@ class AdapterConfig(BaseModel):
                         ],
                         "execution": {
                             "parallel": True,
-                            "failure_mode": "stop",
                             "max_workers": 4
                         }
                     }

{expops-0.1.19.dev0/src/mlops → expops-0.1.22.dev0/src/expops}/adapters/custom/custom_adapter.py

@@ -9,11 +9,11 @@ import time
 from pathlib import Path
 from typing import Any, Dict, Optional
 
-from mlops.core import StepStateManager, get_context_factory, get_step_registry, set_current_context
-from mlops.core.compute_config import load_compute_config, map_compute_to_executor
-from mlops.core.custom_model_base import MLOpsCustomModelBase
-from mlops.core.dask_networkx_executor import DaskNetworkXExecutor
-from mlops.core.networkx_parser import parse_networkx_pipeline_from_config
+from expops.core import StepStateManager, get_context_factory, get_step_registry, set_current_context
+from expops.core.compute_config import load_compute_config, map_compute_to_executor
+from expops.core.runtime.custom_model_base import MLOpsCustomModelBase
+from expops.core.execution.dask_networkx_executor import DaskNetworkXExecutor
+from expops.core.graph.networkx_parser import parse_networkx_pipeline_from_config
 
 from ..base import ModelAdapter
 from ..config_schema import AdapterConfig
@@ -75,6 +75,8 @@ class CustomModelAdapter(ModelAdapter):
         self.step_registry = get_step_registry()
         self.step_state_manager: StepStateManager | None = None
         self.networkx_executor: DaskNetworkXExecutor | None = None
+        self._local_cluster: Any = None
+        self._local_client: Any = None
         self.logger = logging.getLogger(__name__)
         self._script_paths: list[Path] = []
         self._chart_script_paths: list[Path] = []
@@ -84,7 +86,7 @@ class CustomModelAdapter(ModelAdapter):
     def _repo_root(self) -> Path:
         """Best-effort workspace root resolution (where projects live)."""
         try:
-            from mlops.core.workspace import get_workspace_root
+            from expops.core.workspace import get_workspace_root
             return get_workspace_root()
         except Exception:
             return Path.cwd()
@@ -128,7 +130,7 @@ class CustomModelAdapter(ModelAdapter):
         except Exception:
             pass
         try:
-            from mlops.core.workspace import infer_source_root
+            from expops.core.workspace import infer_source_root
 
             src_root = infer_source_root()
             if src_root:
@@ -145,7 +147,8 @@ class CustomModelAdapter(ModelAdapter):
         """Initialize the adapter: import model module, configure caching, and set up the executor."""
         self.logger = logging.getLogger(__name__)
 
-        # Load scripts section from config file
+        # Load scripts and execution.workspace from config file
+        full_config: Dict[str, Any] = {}
         try:
             if self.project_path:
                 config_path = self.project_path / "configs" / "project_config.yaml"
@@ -159,6 +162,12 @@ class CustomModelAdapter(ModelAdapter):
                         self._default_script = next(iter(self._scripts_section)) if self._scripts_section else None
         except Exception as e:
             self.logger.warning(f"Failed to load scripts section from config: {e}")
+
+        execution_workspace = getattr(self.run_context, "execution_workspace", None) if self.run_context else None
+        if execution_workspace is None and full_config:
+            from expops.runtime.context import parse_execution_workspace_config
+            exec_cfg = full_config.get("execution") if isinstance(full_config.get("execution"), dict) else {}
+            execution_workspace = parse_execution_workspace_config(exec_cfg.get("workspace") if isinstance(exec_cfg.get("workspace"), dict) else None)
         
         scripts_section = self._scripts_section
         default_script = self._default_script
@@ -172,7 +181,7 @@ class CustomModelAdapter(ModelAdapter):
         missing_script_paths: list[str] = []
 
         # Import resolve_script_path helper
-        from mlops.core.pipeline_utils import resolve_script_path
+        from expops.core.graph.pipeline_utils import resolve_script_path
 
         for proc in process_cfgs:
             if not isinstance(proc, dict):
@@ -293,14 +302,27 @@ class CustomModelAdapter(ModelAdapter):
         cache_config = _as_dict(cache_config)
         cache_ttl_hours = cache_config.get("ttl_hours", 24) if isinstance(cache_config, dict) else 24
 
-        step_cache_dir = (self.project_path / "cache" / "steps") if self.project_path else Path("step_cache")
+        step_cache_dir: Path
+        env_cache_dir = os.getenv("MLOPS_STEP_CACHE_DIR")
+        if env_cache_dir:
+            step_cache_dir = Path(env_cache_dir)
+        elif self.project_path:
+            try:
+                from expops.core.workspace import get_project_runtime_root
+
+                runtime_root = get_project_runtime_root(self.project_path, self.project_path.name)
+                step_cache_dir = runtime_root / "cache" / "steps"
+            except Exception:
+                step_cache_dir = self.project_path / "cache" / "steps"
+        else:
+            step_cache_dir = Path("step_cache")
 
         project_id_for_ns = self.project_path.name if self.project_path else "default"
         backend_cfg = _as_dict(cache_config.get("backend", {}))
 
         # Centralized KV/object-store creation (config -> env override -> safe fallback).
         try:
-            from mlops.storage.factory import create_kv_store, create_object_store
+            from expops.storage.factory import create_kv_store, create_object_store
         except Exception:
             create_kv_store = None  # type: ignore[assignment]
             create_object_store = None  # type: ignore[assignment]
@@ -319,16 +341,19 @@ class CustomModelAdapter(ModelAdapter):
                 project_root=self.project_path,
             )
         else:
-            from mlops.storage.adapters.memory_store import InMemoryStore
+            from expops.storage.adapters.memory_store import InMemoryStore
             kv_store = InMemoryStore(project_id_for_ns)
 
         obj_store = None
+        obj_prefix = None
         if create_object_store:
             try:
                 obj_store = create_object_store(cache_config if isinstance(cache_config, dict) else {}, env=os.environ)
+                if obj_store is not None:
+                    obj_prefix = f"{project_id_for_ns}/cache"
             except Exception:
                 obj_store = None
-        obj_prefix = None
+                obj_prefix = None
 
         self.step_state_manager = StepStateManager(
             cache_dir=step_cache_dir,
@@ -337,6 +362,7 @@ class CustomModelAdapter(ModelAdapter):
             cache_ttl_hours=cache_ttl_hours,
             object_store=obj_store,
             object_prefix=obj_prefix,
+            project_root=self.project_path,
         )
 
         compute_cfg = {}
@@ -399,7 +425,26 @@ class CustomModelAdapter(ModelAdapter):
                 pass
 
         scheduler_address = executor_config.get("scheduler_address") or os.environ.get("DASK_SCHEDULER_ADDRESS")
-        scheduler_mode = "distributed" if scheduler_address else "threads"
+        initial_client: Any = None
+        if not scheduler_address and execution_workspace is not None:
+            try:
+                try:
+                    from distributed import Client, LocalCluster
+                except Exception:
+                    from dask.distributed import Client, LocalCluster
+                self._local_cluster = LocalCluster(
+                    n_workers=max(1, n_workers),
+                    threads_per_worker=1,
+                )
+                self._local_client = Client(self._local_cluster)
+                initial_client = self._local_client
+                scheduler_address = getattr(self._local_cluster, "scheduler_address", None) or getattr(getattr(self._local_client, "scheduler", None), "address", None)
+                self.logger.info(f"Started local process workers (strict_process_workers): n_workers={n_workers}, scheduler={scheduler_address}")
+            except Exception as e:
+                self.logger.warning(f"Failed to start local process workers, falling back to threads: {e}")
+                self._local_cluster = None
+                self._local_client = None
+        scheduler_mode = "distributed" if (scheduler_address or initial_client) else "threads"
 
         extra_files_to_upload: list[str] = []
         try:
@@ -407,6 +452,12 @@ class CustomModelAdapter(ModelAdapter):
             extra_files_to_upload.extend([str(path) for path in self._chart_script_paths if path])
         except Exception:
             extra_files_to_upload = []
+        # Paths to upload but not load (chart/reporting scripts run in subprocess with reporting env).
+        extra_files_no_load: list[str] = []
+        try:
+            extra_files_no_load = [str(path) for path in self._chart_script_paths if path]
+        except Exception:
+            pass
         # Upload reporting entrypoint for worker-side chart imports (best-effort).
         try:
             rep_cfg_text = os.environ.get("MLOPS_REPORTING_CONFIG") or ""
@@ -419,6 +470,7 @@ class CustomModelAdapter(ModelAdapter):
                         p = self._repo_root() / p
                     if p.exists():
                         extra_files_to_upload.append(str(p))
+                        extra_files_no_load.append(str(p))
         except Exception:
             pass
 
@@ -429,11 +481,13 @@ class CustomModelAdapter(ModelAdapter):
             n_workers=n_workers,
             scheduler_mode=scheduler_mode,
             scheduler_address=scheduler_address,
-            client=None,
+            client=initial_client,
             extra_files_to_upload=extra_files_to_upload,
+            extra_files_no_load=extra_files_no_load,
             min_workers=min_workers_override,
             wait_for_workers_sec=wait_for_workers_override,
             dask_config_overrides=dask_overrides,
+            run_context=self.run_context,
         )
         mode_label = "distributed" if scheduler_mode == "distributed" else "threads"
         self.logger.info(f"Initialized Dask NetworkX executor with {n_workers} workers ({mode_label} scheduler)")
@@ -442,7 +496,7 @@ class CustomModelAdapter(ModelAdapter):
 
         # Make state manager available for manual step-level caching in decorators
         try:
-            from mlops.core.step_system import set_state_manager as _set_sm
+            from expops.core.step_system import set_state_manager as _set_sm
             _set_sm(self.step_state_manager)
         except Exception:
             pass