opengris-scaler 1.12.9__tar.gz → 1.12.12__tar.gz

{opengris_scaler-1.12.9 → opengris_scaler-1.12.12}/CONTRIBUTING.md

@@ -7,6 +7,7 @@ This document describes the contribution process and governance policies of the
 
 ## Helpful Resources
 
+* [Development Guidelines](https://finos.github.io/opengris-scaler/tutorials/development.html)
 
 * [README.md](./README.md)
 
@@ -103,4 +104,4 @@ This document MAY be amended by a vote of the Maintainers according to the Maint
 
 ## Code of Conduct
 
-Please see the FINOS [Community Code of Conduct](https://www.finos.org/code-of-conduct).
+Please see the FINOS [Community Code of Conduct](https://www.finos.org/code-of-conduct).

{opengris_scaler-1.12.9 → opengris_scaler-1.12.12}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: opengris-scaler
-Version: 1.12.9
+Version: 1.12.12
 Summary: OpenGRIS Scaler Distribution Framework
 Author-Email: Citi <opensource@citi.com>
 License: Apache 2.0
@@ -365,6 +365,8 @@ with Client(address="tcp://127.0.0.1:2345") as client:
     print(future.result())  # 21
 ```
 
+**Note**: When creating a `Client` inside a task (nested client), the `address` parameter is optional. If omitted, the client automatically uses the scheduler address from the worker context. If provided, the specified address takes precedence.
+
 ## Task Routing and Capability Management
 
 > **Note**: This feature is experimental and may change in future releases.

{opengris_scaler-1.12.9 → opengris_scaler-1.12.12}/README.md

@@ -322,6 +322,8 @@ with Client(address="tcp://127.0.0.1:2345") as client:
     print(future.result())  # 21
 ```
 
+**Note**: When creating a `Client` inside a task (nested client), the `address` parameter is optional. If omitted, the client automatically uses the scheduler address from the worker context. If provided, the specified address takes precedence.
+
 ## Task Routing and Capability Management
 
 > **Note**: This feature is experimental and may change in future releases.

{opengris_scaler-1.12.9 → opengris_scaler-1.12.12}/docs/source/index.rst

@@ -28,3 +28,4 @@ Content
    tutorials/features
    tutorials/configuration
    tutorials/examples
+   tutorials/development

opengris_scaler-1.12.12/docs/source/tutorials/development.rst

@@ -0,0 +1,193 @@
+=======================
+Development Guidelines
+=======================
+
+.. contents:: Table of Contents
+    :depth: 2
+
+Please be aware that contributing code to this repo requires following several opinionated rules.
+Pull requests that ignore the guidelines are unlikely to be accepted.
+
+These development guidelines will be updated whenever we need to clarify missing rules or adjust existing guidance.
+
+This project consists of both Python and C++ code, so the guidelines cover expectations for each language.
+
+Both humans and AI contributors should follow these guidelines.
+
+Common Guidelines
+-----------------
+
+Naming
+~~~~~~
+
+Names should be crystal clear within the code context. That context can be a function, module (file), or library.
+In general, prefer the most specific name unless a conventional name is already established.
+
+Bad examples:
+ * data1, data2
+ * a, b, c, d, e
+ * df
+ * function1
+ * array
+
+Avoid abbreviations whenever possible. Abbreviations make the code harder to read and debug unless the term is widely
+known or an industry standard.
+
+Numbers
+~~~~~~~
+
+Magic numbers are not allowed. Every constant should be assigned to a well-named variable that explains its purpose.
+
+Object-Oriented Programming (OOP)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+OOP is available in both C++ and Python. Use classes to encapsulate related behavior and data when those pieces belong
+together.
+
+**We allow subtyping, but we do not allow inheritance.**
+ * Subtyping: allowed - override member functions that have no implementations (interfaces in C++).
+ * Inheritance: disallowed - do not override functions that already provide implementations.
+
+Class and static methods are allowed in Python. Remember that class methods provide no benefit over module-level
+functions unless the behavior is tied to class-level state. Place the function in the scope that matches its
+responsibility.
+
+Functions
+~~~~~~~~~
+
+ * Please refactor aggressively, except in test cases where limited duplication can aid readability.
+   Shared test helpers should still live in utility functions.
+ * Express complex conditional filtering logic with guard clauses to improve readability.
+
+Python Guidelines
+-----------------
+
+ * We follow PEP 8 closely, with the exception of a 120-character line width. Refer to PEP 8 for the remaining details.
+ * Python type hints are mandatory for all function arguments and return types, and mypy will check them.
+ * We use the Black formatter; run `black -l 120 -C`.
+ * All Python imports must be absolute.
+ * We do not allow any content in `__init__.py` files unless a special case (such as the project root)
+   requires it for user convenience.
+
+Test Cases
+~~~~~~~~~~
+
+ * Python test cases should use the standard `unittest` module, and each test class should live in its own Python file
+   named with the `test_*.py` prefix.
+ * Mirror the unit-test directory structure to the code structure because the `tests` directory lives at the repo root.
+
+C++ Guidelines
+--------------
+
+Naming
+~~~~~~
+
+Please use PascalCase for namespaces and classes. Capitalize the first character, for example:
+ * SomeClass
+ * LongNameClass
+
+Use all capital letters for global variables or macros. System or third-party library definitions are exempt from this
+rule:
+ * GLOBAL_VARIABLE
+
+Use camelCase for variable and function names, with the first character lower case:
+ * addSomeValue
+ * deleteSomeValue
+
+Keep abbreviations uppercase, for example:
+ * TCPAcceptor instead of TcpAcceptor
+ * IOSocket instead of IoSocket
+
+File names should use snake_case. Use the `.h` extension for headers and `.cpp` for source files:
+ * header files: message_connection.h
+ * source files: message_connection.cpp
+
+Formatting
+~~~~~~~~~~
+
+Please use the `.clang-format` file in the repo root for code formatting.
+
+Namespaces
+~~~~~~~~~~
+
+ * Do not add `using namespace` directives like `using namespace std;`.
+ * Always use fully qualified names such as `std::cout`.
+
+Includes
+~~~~~~~~
+
+ * Remove include files when none of their symbols are used.
+ * Always include the header that defines the symbols you rely on; avoid depending on transitive includes. For example:
+   * There are files `common.h`, `some_module.h`, and `application.cpp`.
+   * At the top of `common.h`, there is `#include <cstring>`.
+   * At the top of `some_module.h`, there is `#include "common.h"`.
+   * At the top of `application.cpp`, there is `#include "some_module.h"`.
+   * Even though `application.cpp` compiles because `<cstring>` is indirectly included through `common.h`, explicitly
+     include `<cstring>` in `application.cpp`.
+
+Struct/Class
+~~~~~~~~~~~~
+
+In C++, structs and classes are the same except for default access. Use `struct` for passive data structures with minimal
+helpers. Use `class` for stateful types, and do not expose fields directly. Provide setters and getters instead.
+
+.. code:: cpp
+
+    struct Address {
+        std::string domain;
+        int port;
+    };
+
+    // If a data structure has internal state that consumers should not mutate directly, provide methods to modify that
+    // state instead of exposing the members.
+    class Client {
+    public:
+        void send(const std::vector<uint8_t>& buffer);
+
+        int getMessageCount() const;
+
+    private:
+        int messageCount;
+    };
+
+Code Layout
+~~~~~~~~~~~
+
+For any header or source file, order elements as follows:
+ * system library includes
+ * standard library includes
+ * third-party library includes
+ * in-library includes
+ * global variables
+ * API classes and functions
+ * internal functions that are not meant to be used outside the file
+
+Class definitions should follow this order:
+
+.. code:: cpp
+
+    // List public members first, then protected, then private sections.
+    // Place using declarations at the top, followed by constructors, copy/move constructors, and the destructor.
+    // Declare member functions before member variables.
+    class Foobar {
+    public:
+        // Using types should go first.
+        using Type = some_namespace::SomeType;
+
+        Foobar();
+        Foobar(const std::string& foobar) : _foobar(foobar) {}
+        ~Foobar();
+
+        const std::string& getCurrentFoobar() const;
+
+    protected:
+        void someProtectedMethod();
+
+        int _someProtectedVariable;
+
+    private:
+        void somePrivateMethod();
+
+        std::string _foobar;
+        int _somePrivateVariable;
+    };

{opengris_scaler-1.12.9 → opengris_scaler-1.12.12}/docs/source/tutorials/features.rst

@@ -118,6 +118,28 @@ Tasks can depend on other tasks' result without using graph.
 .. literalinclude:: ../../../examples/nested_client.py
    :language: python
 
+Nested Client Scheduler Address
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When creating a ``Client`` inside a task (nested client), the scheduler ``address`` parameter is optional. If omitted, the client automatically detects and uses the scheduler address from the worker context. This is particularly useful when workers are behind firewalls or NAT, where the scheduler address accessible from inside the worker may differ from external addresses.
+
+**Automatic Address Detection:**
+
+.. code:: python
+
+    from scaler import Client
+
+    def nested_task():
+        # Automatically uses worker's scheduler address if no address is provided to Client()
+        with Client() as client:
+            result = client.submit(lambda x: x * 2, 5).result()
+        return result
+
+    # Main client
+    client = Client(address="tcp://127.0.0.1:2345")
+    future = client.submit(nested_task)
+    print(future.result())  # 10
+
 Dynamically Building Graph Within a Task
 ----------------------------------------
 

{opengris_scaler-1.12.9 → opengris_scaler-1.12.12}/scaler/client/client.py

@@ -51,7 +51,7 @@ class _CallNode:
 class Client:
     def __init__(
         self,
-        address: str,
+        address: Optional[str] = None,
         profiling: bool = False,
         timeout_seconds: int = DEFAULT_CLIENT_TIMEOUT_SECONDS,
         heartbeat_interval_seconds: int = DEFAULT_HEARTBEAT_INTERVAL_SECONDS,
@@ -61,8 +61,9 @@ class Client:
         """
         The Scaler Client used to send tasks to a scheduler.
 
-        :param address: Address of Scheduler to submit work to
-        :type address: str
+        :param address: Address of Scheduler to submit work to. If None, will attempt to auto-detect
+                       when running inside a worker context.
+        :type address: Optional[str]
         :param profiling: If True, the returned futures will have the `task_duration()` property enabled.
         :type profiling: bool
         :param timeout_seconds: Seconds until heartbeat times out
@@ -72,6 +73,7 @@ class Client:
         :param stream_output: If True, stdout/stderr will be streamed to client during task execution
         :type stream_output: bool
         """
+        address = self._resolve_scheduler_address(address)
         self.__initialize__(address, profiling, timeout_seconds, heartbeat_interval_seconds, serializer, stream_output)
 
     def __initialize__(
@@ -380,6 +382,10 @@ class Client:
         disconnect from connected scheduler, this will not shut down the scheduler
         """
 
+        # Handle case where client wasn't fully initialized
+        if not hasattr(self, "_stop_event"):
+            return
+
         if self._stop_event.is_set():
             self.__destroy()
             return
@@ -633,3 +639,20 @@ class Client:
         assert current_task is not None
 
         return retrieve_task_flags_from_task(current_task).priority
+
+    def _resolve_scheduler_address(self, address: Optional[str]) -> str:
+        """Resolve the scheduler address based on the provided address and worker context."""
+        # Provided address always takes precedence
+        if address is not None:
+            return address
+
+        # No address provided, check if we're running inside a worker context
+        current_processor = Processor.get_current_processor()
+        if current_processor is None:
+            raise ValueError(
+                "No scheduler address provided and not running inside a worker context. "
+                "Please provide a scheduler address when creating the Client outside of a worker."
+            )
+
+        # Return the scheduler address from the current processor
+        return current_processor.scheduler_address().to_address()

{opengris_scaler-1.12.9 → opengris_scaler-1.12.12}/scaler/entry_points/object_storage_server.py

@@ -1,5 +1,6 @@
 import argparse
 import logging
+import sys
 
 from scaler.config.loader import load_config
 from scaler.config.section.object_storage_server import ObjectStorageServerConfig
@@ -31,11 +32,14 @@ def main():
 
     log_format_str, log_level_str, log_paths = get_logger_info(logging.getLogger())
 
-    ObjectStorageServer().run(
-        oss_config.object_storage_address.host,
-        oss_config.object_storage_address.port,
-        oss_config.object_storage_address.identity,
-        log_level_str,
-        log_format_str,
-        log_paths,
-    )
+    try:
+        ObjectStorageServer().run(
+            oss_config.object_storage_address.host,
+            oss_config.object_storage_address.port,
+            oss_config.object_storage_address.identity,
+            log_level_str,
+            log_format_str,
+            log_paths,
+        )
+    except KeyboardInterrupt:
+        sys.exit(0)

{opengris_scaler-1.12.9 → opengris_scaler-1.12.12}/scaler/object_storage/CMakeLists.txt

@@ -31,11 +31,9 @@ set_target_properties(py_object_storage_server PROPERTIES
 )
 
 target_link_libraries(py_object_storage_server PRIVATE
-    $<$<CXX_COMPILER_ID:GNU,Clang>:-Wl,-Bstatic>
     capnp_objs
     ymq_objs
     object_storage_server_objs
-    $<$<CXX_COMPILER_ID:GNU,Clang>:-Wl,-Bdynamic>
     CapnProto::capnp
     CapnProto::kj
     Python3::Module

{opengris_scaler-1.12.9 → opengris_scaler-1.12.12}/scaler/object_storage/object_storage_server.cpp

@@ -138,6 +138,8 @@ void ObjectStorageServer::processRequests(std::function<bool()> running)
             auto maybeMessageFuture = ymq::futureRecvMessage(_ioSocket);
             while (maybeMessageFuture.wait_for(100ms) == std::future_status::timeout) {
                 if (!running()) {
+                    _logger.log(scaler::ymq::Logger::LoggingLevel::info, "ObjectStorageServer: stopped by user");
+                    pendingRequests.clear();
                     return;
                 }
             }

opengris_scaler-1.12.12/scaler/version.txt

@@ -0,0 +1 @@
+1.12.12

{opengris_scaler-1.12.9 → opengris_scaler-1.12.12}/scaler/worker/agent/processor/processor.py

@@ -37,6 +37,7 @@ class Processor(multiprocessing.get_context("spawn").Process):  # type: ignore
         self,
         event_loop: str,
         agent_address: ZMQConfig,
+        scheduler_address: ZMQConfig,
         storage_address: ObjectStorageConfig,
         preload: Optional[str],
         resume_event: Optional[EventType],
@@ -50,6 +51,7 @@ class Processor(multiprocessing.get_context("spawn").Process):  # type: ignore
 
         self._event_loop = event_loop
         self._agent_address = agent_address
+        self._scheduler_address = scheduler_address
         self._storage_address = storage_address
         self._preload = preload
 
@@ -74,6 +76,10 @@ class Processor(multiprocessing.get_context("spawn").Process):  # type: ignore
         """Returns the current Processor instance controlling the current process, if any."""
         return _current_processor.get()
 
+    def scheduler_address(self) -> ZMQConfig:
+        """Returns the scheduler address this processor's worker is connected to."""
+        return self._scheduler_address
+
     def current_task(self) -> Optional[Task]:
         return self._current_task
 

{opengris_scaler-1.12.9 → opengris_scaler-1.12.12}/scaler/worker/agent/processor_holder.py

@@ -19,6 +19,7 @@ class ProcessorHolder:
         self,
         event_loop: str,
         agent_address: ZMQConfig,
+        scheduler_address: ZMQConfig,
         storage_address: ObjectStorageConfig,
         preload: Optional[str],
         garbage_collect_interval_seconds: int,
@@ -43,6 +44,7 @@ class ProcessorHolder:
         self._processor = Processor(
             event_loop=event_loop,
             agent_address=agent_address,
+            scheduler_address=scheduler_address,
             storage_address=storage_address,
             preload=preload,
             resume_event=self._resume_event,

{opengris_scaler-1.12.9 → opengris_scaler-1.12.12}/scaler/worker/agent/processor_manager.py

@@ -23,6 +23,7 @@ class VanillaProcessorManager(ProcessorManager):
         identity: WorkerID,
         event_loop: str,
         address_internal: ZMQConfig,
+        scheduler_address: ZMQConfig,
         preload: Optional[str],
         garbage_collect_interval_seconds: int,
         trim_memory_threshold_bytes: int,
@@ -34,6 +35,7 @@ class VanillaProcessorManager(ProcessorManager):
 
         self._identity = identity
         self._event_loop = event_loop
+        self._scheduler_address = scheduler_address
         self._preload = preload
 
         self._garbage_collect_interval_seconds = garbage_collect_interval_seconds
@@ -299,6 +301,7 @@ class VanillaProcessorManager(ProcessorManager):
         self._current_holder = ProcessorHolder(
             self._event_loop,
             self._address_internal,
+            self._scheduler_address,
             storage_address,
             self._preload,
             self._garbage_collect_interval_seconds,

{opengris_scaler-1.12.9 → opengris_scaler-1.12.12}/scaler/worker/worker.py

@@ -139,6 +139,7 @@ class Worker(multiprocessing.get_context("spawn").Process):  # type: ignore
             identity=self._ident,
             event_loop=self._event_loop,
             address_internal=self._address_internal,
+            scheduler_address=self._address,
             preload=self._preload,
             garbage_collect_interval_seconds=self._garbage_collect_interval_seconds,
             trim_memory_threshold_bytes=self._trim_memory_threshold_bytes,

{opengris_scaler-1.12.9 → opengris_scaler-1.12.12}/tests/test_client.py

@@ -320,6 +320,14 @@ class TestClient(unittest.TestCase):
             # but new tasks should work fine
             self.assertEqual(client.submit(round, 3.14).result(), 3.0)
 
+    def test_client_no_address_outside_worker(self):
+        """Test that creating a Client without an address outside worker context raises ValueError."""
+        with self.assertRaises(ValueError) as context:
+            Client()
+
+        self.assertIn("No scheduler address provided", str(context.exception))
+        self.assertIn("not running inside a worker context", str(context.exception))
+
 
 class TestClientPreload(unittest.TestCase):
     # Separate class for preload functionality with separate cluster to avoid interfering with time-sensitive tests

{opengris_scaler-1.12.9 → opengris_scaler-1.12.12}/tests/test_nested_task.py

@@ -34,6 +34,18 @@ class TestNestedTask(unittest.TestCase):
             result = client.submit(fibonacci, client, 8).result()
             self.assertEqual(result, 21)
 
+    def test_nested_task_auto_address(self) -> None:
+        """Test nested task with automatic scheduler address detection."""
+        with Client(self.address) as client:
+            result = client.submit(nested_task_auto_address, 5).result()
+            self.assertEqual(result, 25)
+
+    def test_nested_task_explicit_address(self) -> None:
+        """Test nested task with explicit scheduler address (honors user-provided address)."""
+        with Client(self.address) as client:
+            result = client.submit(nested_task_explicit_address, self.address, 7).result()
+            self.assertEqual(result, 49)
+
 
 def parent_task_arg_client(client: Client) -> int:
     iterable = [(v,) for v in range(0, N_TASKS)]
@@ -60,3 +72,25 @@ def fibonacci(client: Client, n: int):
         a = client.submit(fibonacci, client, n - 1)
         b = client.submit(fibonacci, client, n - 2)
         return a.result() + b.result()
+
+
+def nested_task_auto_address(value: int) -> int:
+    """Test function that creates a nested client without providing an address."""
+    # Client should automatically detect worker context and use worker's scheduler address
+    client = Client()
+    try:
+        result = client.submit(nested_task, value).result()
+        return result
+    finally:
+        client.disconnect()
+
+
+def nested_task_explicit_address(scheduler_address: str, value: int) -> int:
+    """Test function that creates a nested client with an explicit address."""
+    # Client should honor the explicitly provided address even though running in worker context
+    client = Client(address=scheduler_address)
+    try:
+        result = client.submit(nested_task, value).result()
+        return result
+    finally:
+        client.disconnect()

opengris_scaler-1.12.9/scaler/version.txt

@@ -1 +0,0 @@
-1.12.9