seamless-workflow 0.2.0__tar.gz

seamless_workflow-0.2.0/PKG-INFO

@@ -0,0 +1,157 @@
+Metadata-Version: 2.4
+Name: seamless-workflow
+Version: 0.2.0
+Summary: Reactive workflow Context layer for Seamless
+Author: Sjoerd de Vries
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/sjdv1982/seamless
+Project-URL: Repository, https://github.com/sjdv1982/seamless-workflow
+Project-URL: Issues, https://github.com/sjdv1982/seamless/issues
+Keywords: seamless,workflow,reactive,checksum,reproducible
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: seamless-core>=0.2
+Requires-Dist: seamless-transformer>=0.8.0
+Requires-Dist: traitlets
+Requires-Dist: ipywidgets
+Requires-Dist: aiohttp>=3.9
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+
+# seamless-workflow
+
+Reactive workflow `Context` layer for Seamless.
+
+Context nodes are exposed through the canonical `seamless.Cell` and
+`seamless_transformer.Transformer` handles. The experimental wrapper/view surface
+described by the earlier context handoff documents is superseded by
+[`context-internals-followup-plan.md`](../seamless/context-internals-followup-plan.md).
+
+Construct cells with `Cell("int")` or `Cell(celltype="int")`; the default type
+is `"mixed"` without a typed source. Initial references are keyword-only, for example
+`Cell("int", checksum=checksum)`. Assign values with `.set(value)`.
+
+`celltype` is the output type; `input_celltype` is read-only and follows the
+source or the constant's declared serialization type. Creating `ctx.a = ctx.b`
+copies `b.celltype` once. Rewiring an existing cell retains its type. Retyping
+converts the original input; `.build().run()` agrees with `.value`. A cell's own
+conversion failure reports `failed` with `.exception`.
+
+`cell.source` reports the configured source. On a projection it reports the edge
+at that path, else the nearest enclosing source, else None. `.checksum` reads
+the produced checksum. Public `input_ref` and `target_celltype` are retired.
+
+`ctx.tf.pins.x` returns a fresh Pin, also when unwired. Read `.value`, `.checksum`,
+`.buffer`, `.source`, `.input_celltype`, `.state`, or `.exception` explicitly.
+`pin.celltype` and `ctx.tf.celltypes.x` are linked. Pins convert their original
+inputs before transformation construction; a failed conversion leaves the Pin
+failed and the Transformer blocked on its name. A Pin is not a source; connect
+`pin.source`. There are no sub-pin targets, validators, or mounts.
+
+For Cells and Pins, `.value`, `.buffer`, and `.checksum` assignments declare a
+new input and detach connections; `set`, `set_buffer`, and `set_checksum` check
+ownership. `set_checksum(cs, input_celltype=...)` declares the input encoding.
+`.value = None` stores canonical null; `.buffer = None` and `.checksum = None`
+clear the input. All Cell types allow null. Required pins allow null only for
+plain/mixed/bytes; optional pins of any type drop null before conversion.
+A connected optional pin whose upstream has no checksum blocks.
+
+`del ctx.a` deletes a node. `del ctx.tf.pins.x` deletes a declaration only for
+signatureless code; use `.checksum = None` to clear a fixed-signature pin.
+
+Whole Context cells can be mounted to files:
+
+```python
+from seamless.workflow import Cell, Context, Transformer
+
+with Context() as ctx:
+    ctx.config = Cell("plain")
+    ctx.config.mount("config.json")
+    ctx.output = Cell("text")
+    ctx.output.set("ready")
+    ctx.output.mount("output.txt", mode="w")
+    report = ctx.mounts.sync(timeout=10)
+```
+
+`mount(path, mode="rw", authority="file", persistent=True)` blocks through the
+initial read and first write attempt. Modes are `r`, `w`, and `rw`. Authority
+chooses the initial winner; subsequent file edits are authoritative inputs in
+sensing modes. A missing path supplies no value, while zero-byte and `null\n`
+files supply null. At mount time, missing, zero-byte, and an empty directory do
+not override an existing cell value. A sensing mount on a cell without a value
+installs null; `file-strict` instead fails when the path is missing. Invalid or
+unreadable input fails the cell, preserves its stored last good value, and
+continues monitoring. Delivery failures appear on `ctx.output.mount.error` and
+retry with backoff; they do not invalidate the cell's value.
+
+`ctx.compute()` waits only for graph work. Call `ctx.mounts.sync()` before
+reading mounted outputs externally, or `await ctx.mounts.synchronization()` in
+async code. The returned mapping contains status and errors per node path.
+Synchronization establishes a finite filesystem cut; continuously changing
+external files can prevent completion, so interactive clients can set a timeout.
+
+Use `del ctx.config.mount` to unmount. `persistent=False` conditionally deletes
+unchanged files at unmount or close. Explicit Context close flushes already
+requested persistent deliveries, bounded by `close(timeout=60)`. A foreign
+modification is never deleted based solely on its path. Two incompatible mounts
+of overlapping paths within one process are refused. Write-only mounts pause
+after three foreign-write reassertions in 20 seconds; inspect `.mount.error` and
+call `.mount.clear_error()` to resume.
+
+Null writes truncate an existing file to zero bytes, including `.gz` and `.zst`
+paths, but do not create a missing file. Deleting a sensed file preserves the
+stored cell value; under `file-strict` it raises a recoverable sense error. An
+initially empty directory supplies no value, while a directory emptied after
+mounting reads as `{}`. Directory null delivery leaves the tree in place and the
+mount out of sync. Clearing a mounted cell is refused; unmount first. These rules
+are separate from explicit nonpersistent unmount cleanup.
+
+Text/code, JSON scalar/plain, bytes, binary, and mixed celltypes are supported.
+`.gz` and `.zst` paths compress canonical bytes. `folder` and `deepfolder` use
+directories, with per-leaf atomic replacement. `folder` mounts in `r`, `w` or
+`rw`; `deepfolder` is read-only (sensing) and may only be mounted with
+`mode="r"`. Directory trees are not replaced
+atomically. Mounts follow a target symlink and preserve it; atomic file writes
+break hardlink sharing. Cross-process exclusion and filesystem aliases such as
+hardlinks are not covered by the registry.
+
+Graphs retain mount specifications (format `0.4`). Legacy 0.2/0.3 graphs load
+only when `target_celltype` is absent or equals `celltype`; conflicting graphs
+raise `PathError`. Constant producers retain their input encoding as `value.celltype`. **Loading a graph can write
+files:** use `ctx.set_graph(graph, mounts=False)` for graphs of unknown origin.
+Sensing cells cannot receive incoming edges; mounted celltypes cannot change.
+Unmount first. Standalone cells, sub-path projections, transformer pins and code
+handles cannot be mounted directly: mount a whole cell and connect it instead.
+
+The transport currently uses polling and a bounded daemon I/O pool. Diagnostics
+include `seamless_workflow.diagnostics.record_attachments(ctx)` and the
+controllable `attachments.manual.ManualDriver` for ordering tests.
+
+Set `SEAMLESS_MOUNT_NATIVE=1` before the first mount to enable Linux inotify hints
+alongside polling. Unsupported platforms and failed watches retain polling.
+
+For notebooks, install `seamless-workflow[jupyter]` and use the public widget
+helpers:
+
+```python
+from IPython.display import display
+from seamless_workflow.jupyter import traitlet, output
+
+display(output(ctx.c))
+traitlet(ctx.a).link(slider)
+```
+
+The widget hub has its own runtime attachment slot and can coexist with
+`ctx.a.mount("a.txt")` on the same cell. `ctx.mounts.sync()` reports both
+sessions; `del ctx.a.mount` detaches only the file slot.
+
+Widget attachments are runtime state and are never serialized. The underlying
+`attachments.widget.WidgetDriver` remains an internal transport.

seamless_workflow-0.2.0/README.md

@@ -0,0 +1,129 @@
+# seamless-workflow
+
+Reactive workflow `Context` layer for Seamless.
+
+Context nodes are exposed through the canonical `seamless.Cell` and
+`seamless_transformer.Transformer` handles. The experimental wrapper/view surface
+described by the earlier context handoff documents is superseded by
+[`context-internals-followup-plan.md`](../seamless/context-internals-followup-plan.md).
+
+Construct cells with `Cell("int")` or `Cell(celltype="int")`; the default type
+is `"mixed"` without a typed source. Initial references are keyword-only, for example
+`Cell("int", checksum=checksum)`. Assign values with `.set(value)`.
+
+`celltype` is the output type; `input_celltype` is read-only and follows the
+source or the constant's declared serialization type. Creating `ctx.a = ctx.b`
+copies `b.celltype` once. Rewiring an existing cell retains its type. Retyping
+converts the original input; `.build().run()` agrees with `.value`. A cell's own
+conversion failure reports `failed` with `.exception`.
+
+`cell.source` reports the configured source. On a projection it reports the edge
+at that path, else the nearest enclosing source, else None. `.checksum` reads
+the produced checksum. Public `input_ref` and `target_celltype` are retired.
+
+`ctx.tf.pins.x` returns a fresh Pin, also when unwired. Read `.value`, `.checksum`,
+`.buffer`, `.source`, `.input_celltype`, `.state`, or `.exception` explicitly.
+`pin.celltype` and `ctx.tf.celltypes.x` are linked. Pins convert their original
+inputs before transformation construction; a failed conversion leaves the Pin
+failed and the Transformer blocked on its name. A Pin is not a source; connect
+`pin.source`. There are no sub-pin targets, validators, or mounts.
+
+For Cells and Pins, `.value`, `.buffer`, and `.checksum` assignments declare a
+new input and detach connections; `set`, `set_buffer`, and `set_checksum` check
+ownership. `set_checksum(cs, input_celltype=...)` declares the input encoding.
+`.value = None` stores canonical null; `.buffer = None` and `.checksum = None`
+clear the input. All Cell types allow null. Required pins allow null only for
+plain/mixed/bytes; optional pins of any type drop null before conversion.
+A connected optional pin whose upstream has no checksum blocks.
+
+`del ctx.a` deletes a node. `del ctx.tf.pins.x` deletes a declaration only for
+signatureless code; use `.checksum = None` to clear a fixed-signature pin.
+
+Whole Context cells can be mounted to files:
+
+```python
+from seamless.workflow import Cell, Context, Transformer
+
+with Context() as ctx:
+    ctx.config = Cell("plain")
+    ctx.config.mount("config.json")
+    ctx.output = Cell("text")
+    ctx.output.set("ready")
+    ctx.output.mount("output.txt", mode="w")
+    report = ctx.mounts.sync(timeout=10)
+```
+
+`mount(path, mode="rw", authority="file", persistent=True)` blocks through the
+initial read and first write attempt. Modes are `r`, `w`, and `rw`. Authority
+chooses the initial winner; subsequent file edits are authoritative inputs in
+sensing modes. A missing path supplies no value, while zero-byte and `null\n`
+files supply null. At mount time, missing, zero-byte, and an empty directory do
+not override an existing cell value. A sensing mount on a cell without a value
+installs null; `file-strict` instead fails when the path is missing. Invalid or
+unreadable input fails the cell, preserves its stored last good value, and
+continues monitoring. Delivery failures appear on `ctx.output.mount.error` and
+retry with backoff; they do not invalidate the cell's value.
+
+`ctx.compute()` waits only for graph work. Call `ctx.mounts.sync()` before
+reading mounted outputs externally, or `await ctx.mounts.synchronization()` in
+async code. The returned mapping contains status and errors per node path.
+Synchronization establishes a finite filesystem cut; continuously changing
+external files can prevent completion, so interactive clients can set a timeout.
+
+Use `del ctx.config.mount` to unmount. `persistent=False` conditionally deletes
+unchanged files at unmount or close. Explicit Context close flushes already
+requested persistent deliveries, bounded by `close(timeout=60)`. A foreign
+modification is never deleted based solely on its path. Two incompatible mounts
+of overlapping paths within one process are refused. Write-only mounts pause
+after three foreign-write reassertions in 20 seconds; inspect `.mount.error` and
+call `.mount.clear_error()` to resume.
+
+Null writes truncate an existing file to zero bytes, including `.gz` and `.zst`
+paths, but do not create a missing file. Deleting a sensed file preserves the
+stored cell value; under `file-strict` it raises a recoverable sense error. An
+initially empty directory supplies no value, while a directory emptied after
+mounting reads as `{}`. Directory null delivery leaves the tree in place and the
+mount out of sync. Clearing a mounted cell is refused; unmount first. These rules
+are separate from explicit nonpersistent unmount cleanup.
+
+Text/code, JSON scalar/plain, bytes, binary, and mixed celltypes are supported.
+`.gz` and `.zst` paths compress canonical bytes. `folder` and `deepfolder` use
+directories, with per-leaf atomic replacement. `folder` mounts in `r`, `w` or
+`rw`; `deepfolder` is read-only (sensing) and may only be mounted with
+`mode="r"`. Directory trees are not replaced
+atomically. Mounts follow a target symlink and preserve it; atomic file writes
+break hardlink sharing. Cross-process exclusion and filesystem aliases such as
+hardlinks are not covered by the registry.
+
+Graphs retain mount specifications (format `0.4`). Legacy 0.2/0.3 graphs load
+only when `target_celltype` is absent or equals `celltype`; conflicting graphs
+raise `PathError`. Constant producers retain their input encoding as `value.celltype`. **Loading a graph can write
+files:** use `ctx.set_graph(graph, mounts=False)` for graphs of unknown origin.
+Sensing cells cannot receive incoming edges; mounted celltypes cannot change.
+Unmount first. Standalone cells, sub-path projections, transformer pins and code
+handles cannot be mounted directly: mount a whole cell and connect it instead.
+
+The transport currently uses polling and a bounded daemon I/O pool. Diagnostics
+include `seamless_workflow.diagnostics.record_attachments(ctx)` and the
+controllable `attachments.manual.ManualDriver` for ordering tests.
+
+Set `SEAMLESS_MOUNT_NATIVE=1` before the first mount to enable Linux inotify hints
+alongside polling. Unsupported platforms and failed watches retain polling.
+
+For notebooks, install `seamless-workflow[jupyter]` and use the public widget
+helpers:
+
+```python
+from IPython.display import display
+from seamless_workflow.jupyter import traitlet, output
+
+display(output(ctx.c))
+traitlet(ctx.a).link(slider)
+```
+
+The widget hub has its own runtime attachment slot and can coexist with
+`ctx.a.mount("a.txt")` on the same cell. `ctx.mounts.sync()` reports both
+sessions; `del ctx.a.mount` detaches only the file slot.
+
+Widget attachments are runtime state and are never serialized. The underlying
+`attachments.widget.WidgetDriver` remains an internal transport.

seamless_workflow-0.2.0/pyproject.toml

@@ -0,0 +1,52 @@
+[build-system]
+requires = ["setuptools>=69", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "seamless-workflow"
+version = "0.2.0"
+description = "Reactive workflow Context layer for Seamless"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+authors = [{name = "Sjoerd de Vries"}]
+dependencies = [
+    "seamless-core>=0.2",
+    "seamless-transformer>=0.8.0",
+    "traitlets", 
+    "ipywidgets",
+    "aiohttp>=3.9"
+]
+keywords = ["seamless", "workflow", "reactive", "checksum", "reproducible"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Operating System :: OS Independent",
+    "Topic :: Scientific/Engineering",
+    "Topic :: Software Development :: Libraries",
+]
+
+[project.scripts]
+seamless-serve-graph = "seamless_workflow.serve_graph:main"
+
+[project.optional-dependencies]
+dev = ["pytest"]
+
+[project.urls]
+Homepage = "https://github.com/sjdv1982/seamless"
+Repository = "https://github.com/sjdv1982/seamless-workflow"
+Issues = "https://github.com/sjdv1982/seamless/issues"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["seamless_workflow*"]
+exclude = ["tests*"]
+
+[tool.setuptools.package-data]
+seamless_workflow = ["attachments/share/static/*.js"]
+
+[tool.pytest.ini_options]
+pythonpath = ["."]

seamless_workflow-0.2.0/seamless_workflow/__init__.py

@@ -0,0 +1,40 @@
+"""Reactive workflow layer for Seamless."""
+
+__version__ = "0.1.0"
+from .context import Context
+from seamless import Cell
+from seamless_transformer import Transformer
+from .endpoints import BoundEndpoint
+from . import shareserver
+from .errors import (
+    AuthorityError,
+    DependencyError,
+    PathError,
+    ReadOnlyEndpointError,
+    StaleWorkflowHandleError,
+    ValueUnavailableError,
+    ClosedContextError,
+    ConcurrentUpdateError,
+    ControllerFailedError,
+    ReentrantContextError,
+    WorkflowExecutionError,
+)
+
+__all__ = [
+    "AuthorityError",
+    "BoundEndpoint",
+    "Context",
+    "Cell",
+    "Transformer",
+    "DependencyError",
+    "PathError",
+    "ReadOnlyEndpointError",
+    "StaleWorkflowHandleError",
+    "ValueUnavailableError",
+    "ClosedContextError",
+    "ConcurrentUpdateError",
+    "ControllerFailedError",
+    "ReentrantContextError",
+    "WorkflowExecutionError",
+    "shareserver",
+]

seamless_workflow-0.2.0/seamless_workflow/adapters.py

@@ -0,0 +1,50 @@
+"""Narrow substrate adapters used by the workflow layer."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from seamless import Buffer, Checksum
+
+
+def checksum_for_value(
+    value: Any, celltype: str = "mixed", *, checksum_is_value: bool = False
+) -> Checksum:
+    """Serialize a Python value and return its checksum.
+
+    An explicit Checksum is already serialized state, and is returned as it is.
+    The exception is a written value (``checksum_is_value``) of celltype
+    ``checksum``: there a Checksum is the value itself.
+    """
+
+    # Ordinary 64-character strings are never reinterpreted as checksums.
+    if isinstance(value, Checksum) and not (checksum_is_value and celltype == "checksum"):
+        return value
+    buffer = Buffer(value, celltype)
+    checksum = buffer.get_checksum()
+    buffer.tempref()
+    return checksum
+
+
+def buffer_for_checksum(checksum: Checksum | str | bytes | None):
+    if checksum is None:
+        return None
+    return Checksum(checksum).resolve()
+
+
+def value_for_checksum(checksum: Checksum | str | bytes | None, celltype: str = "mixed"):
+    if checksum is None:
+        return None
+    return Checksum(checksum).resolve(celltype)
+
+
+def normalize_checksum(checksum: Checksum | str | bytes) -> Checksum:
+    return Checksum(checksum)
+
+
+__all__ = [
+    "checksum_for_value",
+    "buffer_for_checksum",
+    "value_for_checksum",
+    "normalize_checksum",
+]

seamless_workflow-0.2.0/seamless_workflow/assignment.py

@@ -0,0 +1,8 @@
+"""Assignment helper namespace.
+
+The implementation currently keeps assignment methods on :mod:`context` because they
+need direct access to graph mutation internals. This module is intentionally present
+as the public scaffold extension point named by the implementation handoff.
+"""
+
+__all__: list[str] = []

seamless_workflow-0.2.0/seamless_workflow/attachments/__init__.py

@@ -0,0 +1,3 @@
+"""Context attachments and file mounts."""
+from .spec import AttachmentSpec
+from .session import MountError, ConflictError, SyncReport

seamless_workflow-0.2.0/seamless_workflow/attachments/api.py

@@ -0,0 +1,217 @@
+"""Caller-side mount preparation and public handles."""
+import asyncio
+import weakref
+from concurrent.futures import Future
+from uuid import uuid4
+from .spec import AttachmentSpec
+from .fs.service import get_service
+from ..errors import ClosedContextError, ControllerFailedError
+
+
+def make_sink(controller):
+    reference = weakref.ref(controller)
+    def sink(operation, payload):
+        def fail(error):
+            if hasattr(payload, 'release'):
+                payload.release()
+            reply = Future()
+            reply.set_exception(error)
+            return reply
+        owner = reference()
+        if owner is None:
+            return fail(ClosedContextError("Context is closed"))
+        try:
+            reply = owner.enqueue(operation, (payload,), klass=3 if operation == '_mount_observed' else 5,
+                                  internal=operation != '_mount_observed')
+        except (ClosedContextError, ControllerFailedError) as exc:
+            return fail(exc)
+        except BaseException as exc:
+            return fail(exc)
+        else:
+            if hasattr(payload, 'release'):
+                # Also covers accepted messages discarded by stop or poison.
+                reply.add_done_callback(lambda future: payload.release())
+            return reply
+    return sink
+
+
+class MountHandle:
+    def __init__(self, backend):
+        self.backend = backend
+
+    def _context(self):
+        b = self.backend
+        b.context._check_public_caller()
+        if getattr(b, "local_path", ()) or getattr(b, "readonly", False):
+            raise AttributeError('Only whole Context cell nodes can be mounted')
+        from ..errors import NodeError, StaleWorkflowHandleError
+
+        try:
+            b._node()
+        except StaleWorkflowHandleError as exc:
+            raise NodeError('Mounts require an existing whole cell node') from exc
+        return b.context, b.node_path
+
+    def __call__(self, path, mode='rw', authority='file', *, persistent=True):
+        ctx, node_path = self._context()
+        spec = AttachmentSpec(path, mode, authority, persistent)
+        celltype = ctx._controller.call('_mount_validate', node_path, spec, klass=4)
+        service = get_service()
+        registration = service.reserve(spec, celltype, uuid4().hex, make_sink(ctx._controller))
+        try:
+            observation = service.initial_read(registration).result()
+            try:
+                initial = ctx._controller.call('_mount_attach', node_path, spec, registration, observation, klass=2)
+            finally: observation.release()
+            initial.result()
+        except Exception:
+            service.unregister(registration).result()
+            raise
+
+    def unmount(self):
+        ctx, path = self._context()
+        future = ctx._controller.call('_mount_detach', path, klass=2)
+        if future is not None: future.result(get_service().delivery_timeout)
+
+    @property
+    def spec(self):
+        ctx, path = self._context()
+        return ctx._node_snapshot(path).mount
+
+    @property
+    def status(self):
+        ctx, path = self._context()
+        return ctx._controller.call('_mount_status', path, klass=4)
+
+    @property
+    def error(self):
+        status = self.status
+        return None if status is None else status['error']
+
+    def clear_error(self):
+        ctx, path = self._context()
+        ctx._controller.call('_mount_clear_error', path, klass=2)
+
+
+class ContextMounts:
+    def __init__(self, context): self.context = context
+
+    def sync(self, timeout=None):
+        ctx = self.context
+        ctx._check_public_caller()
+        future = ctx._controller.call('_mount_sync', klass=1)
+        try: return future.result(timeout)
+        finally:
+            if ctx._controller.accepting: ctx._controller.call('_withdraw_wait', future, klass=1)
+
+    async def synchronization(self, timeout=None):
+        ctx = self.context
+        ctx._check_public_caller()
+        installed = ctx._controller.submit('_mount_sync', klass=1)
+        try: future = await asyncio.shield(asyncio.wrap_future(installed))
+        except asyncio.CancelledError:
+            def withdraw(reply):
+                if not reply.cancelled() and reply.exception() is None:
+                    ctx._controller.notify('_withdraw_wait', (reply.result(),), klass=1)
+            installed.add_done_callback(withdraw)
+            raise
+        try:
+            return await asyncio.wait_for(asyncio.shield(asyncio.wrap_future(future)), timeout)
+        finally:
+            if ctx._controller.accepting:
+                await asyncio.wrap_future(ctx._controller.submit('_withdraw_wait', (future,), klass=1))
+
+    @property
+    def errors(self):
+        report = self.context._controller.call('_mount_report', klass=4)
+        return {p: s['error'] or s['sense_error'] for p, s in report.items() if s['error'] or s['sense_error']}
+
+
+def load_graph(ctx, data, *, mounts=True, shares=True):
+    context_shares = ctx.shares
+    with context_shares._lock:
+        return _load_graph(
+            ctx, data, mounts=mounts, shares=shares,
+            context_shares=context_shares,
+        )
+
+
+def _load_graph(ctx, data, *, mounts=True, shares=True, context_shares):
+    """Load a graph, optionally restoring its file mounts, HTTP shares and downloads."""
+    import copy
+    from .spec import validate_celltype
+    from ..serialization import prepare_graph
+    graph = prepare_graph(copy.deepcopy(data))
+    old = ctx._controller.call('_mount_registrations', klass=4)
+    old_shares = tuple(
+        registration for registration in old
+        if getattr(getattr(registration, 'spec', None), 'driver', None) == 'share'
+    )
+    old_downloads = tuple(context_shares._downloads.values())
+    download_specs = graph.downloads if shares else {}
+    graph.downloads = dict(download_specs)
+    prepared, reservations = [], []
+    service = get_service() if mounts and any(n.mount for n in graph.nodes.values()) else None
+    share_prepared = []
+    download_registrations = []
+    share_driver = (
+        context_shares._driver
+        if shares and (download_specs or any('share' in n.attachments for n in graph.nodes.values()))
+        else None
+    )
+    try:
+        reads = []
+        for path, node in graph.nodes.items():
+            spec, node.mount = node.mount, None
+            share_spec = node.attachments.pop('share', None)
+            if spec is not None and mounts:
+                celltype = node.cell_config.celltype
+                validate_celltype(celltype, spec.mode)
+                reg = service.reserve(spec, celltype, uuid4().hex, make_sink(ctx._controller), replaces=old)
+                reservations.append(reg)
+                reads.append(('mount', path, spec, reg, service.initial_read(reg)))
+            if share_spec is not None and shares:
+                celltype = node.cell_config.celltype
+                reg = share_driver.reserve(
+                    share_spec, celltype, uuid4().hex, make_sink(ctx._controller),
+                    replaces=old_shares + old_downloads, staged=True, node_path=path,
+                )
+                reservations.append(reg)
+                reads.append(('share', path, share_spec, reg, share_driver.initial_read(reg)))
+        for download_spec in download_specs.values():
+            reg = share_driver.reserve_download(
+                download_spec, context_shares._download_provider(download_spec),
+                replaces=old_shares + old_downloads, staged=True,
+            )
+            reservations.append(reg)
+            download_registrations.append(reg)
+        if share_driver is not None:
+            share_driver.start()
+        for kind, path, spec, reg, future in reads:
+            prepared_item = (path, spec, reg, future.result())
+            (prepared if kind == 'mount' else share_prepared).append(prepared_item)
+        initial = ctx._controller.call(
+            'set_graph', graph, mount_prepared=tuple(prepared),
+            share_prepared=tuple(share_prepared),
+            download_registrations=tuple(download_registrations), klass=2,
+        )
+        for future in initial: future.result()
+        for reg in download_registrations:
+            share_driver.activate(reg).result(share_driver.delivery_timeout)
+    except BaseException:
+        for reg in reversed(reservations):
+            try:
+                future = reg.service.unregister(reg)
+                if future is not None:
+                    future.result(getattr(reg.service, 'delivery_timeout', 60))
+            except BaseException:
+                pass
+        raise
+    finally:
+        for _, _, _, observation in (*prepared, *share_prepared): observation.release()
+    # The old graph's downloads go with it, except those whose key a new
+    # share or download took over (already removed by the commit).
+    for reg in old_downloads:
+        if not reg.closed:
+            context_shares._driver.unregister(reg).result(context_shares._driver.delivery_timeout)
+    context_shares._downloads = {reg.spec.path: reg for reg in download_registrations}

seamless_workflow-0.2.0/seamless_workflow/attachments/fs/__init__.py

@@ -0,0 +1 @@
+"""Process-global filesystem transport."""