opengris-scaler 1.12.8__tar.gz → 1.12.11__tar.gz

{opengris_scaler-1.12.8 → opengris_scaler-1.12.11}/CONTRIBUTING.md

@@ -7,6 +7,7 @@ This document describes the contribution process and governance policies of the
 
 ## Helpful Resources
 
+* [Development Guidelines](https://finos.github.io/opengris-scaler/tutorials/development.html)
 
 * [README.md](./README.md)
 
@@ -103,4 +104,4 @@ This document MAY be amended by a vote of the Maintainers according to the Maint
 
 ## Code of Conduct
 
-Please see the FINOS [Community Code of Conduct](https://www.finos.org/code-of-conduct).
+Please see the FINOS [Community Code of Conduct](https://www.finos.org/code-of-conduct).

{opengris_scaler-1.12.8 → opengris_scaler-1.12.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: opengris-scaler
-Version: 1.12.8
+Version: 1.12.11
 Summary: OpenGRIS Scaler Distribution Framework
 Author-Email: Citi <opensource@citi.com>
 License: Apache 2.0

{opengris_scaler-1.12.8 → opengris_scaler-1.12.11}/docs/source/index.rst

@@ -28,3 +28,4 @@ Content
    tutorials/features
    tutorials/configuration
    tutorials/examples
+   tutorials/development

opengris_scaler-1.12.11/docs/source/tutorials/development.rst

@@ -0,0 +1,193 @@
+=======================
+Development Guidelines
+=======================
+
+.. contents:: Table of Contents
+    :depth: 2
+
+Please be aware that contributing code to this repo requires following several opinionated rules.
+Pull requests that ignore the guidelines are unlikely to be accepted.
+
+These development guidelines will be updated whenever we need to clarify missing rules or adjust existing guidance.
+
+This project consists of both Python and C++ code, so the guidelines cover expectations for each language.
+
+Both humans and AI contributors should follow these guidelines.
+
+Common Guidelines
+-----------------
+
+Naming
+~~~~~~
+
+Names should be crystal clear within the code context. That context can be a function, module (file), or library.
+In general, prefer the most specific name unless a conventional name is already established.
+
+Bad examples:
+ * data1, data2
+ * a, b, c, d, e
+ * df
+ * function1
+ * array
+
+Avoid abbreviations whenever possible. Abbreviations make the code harder to read and debug unless the term is widely
+known or an industry standard.
+
+Numbers
+~~~~~~~
+
+Magic numbers are not allowed. Every constant should be assigned to a well-named variable that explains its purpose.
+
+Object-Oriented Programming (OOP)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+OOP is available in both C++ and Python. Use classes to encapsulate related behavior and data when those pieces belong
+together.
+
+**We allow subtyping, but we do not allow inheritance.**
+ * Subtyping: allowed - override member functions that have no implementations (interfaces in C++).
+ * Inheritance: disallowed - do not override functions that already provide implementations.
+
+Class and static methods are allowed in Python. Remember that class methods provide no benefit over module-level
+functions unless the behavior is tied to class-level state. Place the function in the scope that matches its
+responsibility.
+
+Functions
+~~~~~~~~~
+
+ * Please refactor aggressively, except in test cases where limited duplication can aid readability.
+   Shared test helpers should still live in utility functions.
+ * Express complex conditional filtering logic with guard clauses to improve readability.
+
+Python Guidelines
+-----------------
+
+ * We follow PEP 8 closely, with the exception of a 120-character line width. Refer to PEP 8 for the remaining details.
+ * Python type hints are mandatory for all function arguments and return types, and mypy will check them.
+ * We use the Black formatter; run `black -l 120 -C`.
+ * All Python imports must be absolute.
+ * We do not allow any content in `__init__.py` files unless a special case (such as the project root)
+   requires it for user convenience.
+
+Test Cases
+~~~~~~~~~~
+
+ * Python test cases should use the standard `unittest` module, and each test class should live in its own Python file
+   named with the `test_*.py` prefix.
+ * Mirror the unit-test directory structure to the code structure because the `tests` directory lives at the repo root.
+
+C++ Guidelines
+--------------
+
+Naming
+~~~~~~
+
+Please use PascalCase for namespaces and classes. Capitalize the first character, for example:
+ * SomeClass
+ * LongNameClass
+
+Use all capital letters for global variables or macros. System or third-party library definitions are exempt from this
+rule:
+ * GLOBAL_VARIABLE
+
+Use camelCase for variable and function names, with the first character lower case:
+ * addSomeValue
+ * deleteSomeValue
+
+Keep abbreviations uppercase, for example:
+ * TCPAcceptor instead of TcpAcceptor
+ * IOSocket instead of IoSocket
+
+File names should use snake_case. Use the `.h` extension for headers and `.cpp` for source files:
+ * header files: message_connection.h
+ * source files: message_connection.cpp
+
+Formatting
+~~~~~~~~~~
+
+Please use the `.clang-format` file in the repo root for code formatting.
+
+Namespaces
+~~~~~~~~~~
+
+ * Do not add `using namespace` directives like `using namespace std;`.
+ * Always use fully qualified names such as `std::cout`.
+
+Includes
+~~~~~~~~
+
+ * Remove include files when none of their symbols are used.
+ * Always include the header that defines the symbols you rely on; avoid depending on transitive includes. For example:
+   * There are files `common.h`, `some_module.h`, and `application.cpp`.
+   * At the top of `common.h`, there is `#include <cstring>`.
+   * At the top of `some_module.h`, there is `#include "common.h"`.
+   * At the top of `application.cpp`, there is `#include "some_module.h"`.
+   * Even though `application.cpp` compiles because `<cstring>` is indirectly included through `common.h`, explicitly
+     include `<cstring>` in `application.cpp`.
+
+Struct/Class
+~~~~~~~~~~~~
+
+In C++, structs and classes are the same except for default access. Use `struct` for passive data structures with minimal
+helpers. Use `class` for stateful types, and do not expose fields directly. Provide setters and getters instead.
+
+.. code:: cpp
+
+    struct Address {
+        std::string domain;
+        int port;
+    };
+
+    // If a data structure has internal state that consumers should not mutate directly, provide methods to modify that
+    // state instead of exposing the members.
+    class Client {
+    public:
+        void send(const std::vector<uint8_t>& buffer);
+
+        int getMessageCount() const;
+
+    private:
+        int messageCount;
+    };
+
+Code Layout
+~~~~~~~~~~~
+
+For any header or source file, order elements as follows:
+ * system library includes
+ * standard library includes
+ * third-party library includes
+ * in-library includes
+ * global variables
+ * API classes and functions
+ * internal functions that are not meant to be used outside the file
+
+Class definitions should follow this order:
+
+.. code:: cpp
+
+    // List public members first, then protected, then private sections.
+    // Place using declarations at the top, followed by constructors, copy/move constructors, and the destructor.
+    // Declare member functions before member variables.
+    class Foobar {
+    public:
+        // Using types should go first.
+        using Type = some_namespace::SomeType;
+
+        Foobar();
+        Foobar(const std::string& foobar) : _foobar(foobar) {}
+        ~Foobar();
+
+        const std::string& getCurrentFoobar() const;
+
+    protected:
+        void someProtectedMethod();
+
+        int _someProtectedVariable;
+
+    private:
+        void somePrivateMethod();
+
+        std::string _foobar;
+        int _somePrivateVariable;
+    };

{opengris_scaler-1.12.8 → opengris_scaler-1.12.11}/scaler/entry_points/object_storage_server.py

@@ -1,5 +1,6 @@
 import argparse
 import logging
+import sys
 
 from scaler.config.loader import load_config
 from scaler.config.section.object_storage_server import ObjectStorageServerConfig
@@ -31,11 +32,14 @@ def main():
 
     log_format_str, log_level_str, log_paths = get_logger_info(logging.getLogger())
 
-    ObjectStorageServer().run(
-        oss_config.object_storage_address.host,
-        oss_config.object_storage_address.port,
-        oss_config.object_storage_address.identity,
-        log_level_str,
-        log_format_str,
-        log_paths,
-    )
+    try:
+        ObjectStorageServer().run(
+            oss_config.object_storage_address.host,
+            oss_config.object_storage_address.port,
+            oss_config.object_storage_address.identity,
+            log_level_str,
+            log_format_str,
+            log_paths,
+        )
+    except KeyboardInterrupt:
+        sys.exit(0)

{opengris_scaler-1.12.8 → opengris_scaler-1.12.11}/scaler/object_storage/CMakeLists.txt

@@ -31,11 +31,9 @@ set_target_properties(py_object_storage_server PROPERTIES
 )
 
 target_link_libraries(py_object_storage_server PRIVATE
-    $<$<CXX_COMPILER_ID:GNU,Clang>:-Wl,-Bstatic>
     capnp_objs
     ymq_objs
     object_storage_server_objs
-    $<$<CXX_COMPILER_ID:GNU,Clang>:-Wl,-Bdynamic>
     CapnProto::capnp
     CapnProto::kj
     Python3::Module

{opengris_scaler-1.12.8 → opengris_scaler-1.12.11}/scaler/object_storage/object_storage_server.cpp

@@ -29,7 +29,8 @@ void ObjectStorageServer::run(
     ObjectStorageServer::Identity identity,
     std::string log_level,
     std::string log_format,
-    std::vector<std::string> log_paths)
+    std::vector<std::string> log_paths,
+    std::function<bool()> running)
 {
     _logger = scaler::ymq::Logger(log_format, std::move(log_paths), scaler::ymq::Logger::stringToLogLevel(log_level));
 
@@ -44,7 +45,7 @@ void ObjectStorageServer::run(
 
         _logger.log(scaler::ymq::Logger::LoggingLevel::info, "ObjectStorageServer: started");
 
-        processRequests();
+        processRequests(running);
 
         _ioContext.removeIOSocket(_ioSocket);
     } catch (const std::exception& e) {
@@ -117,7 +118,7 @@ void ObjectStorageServer::closeServerReadyFds()
     }
 }
 
-void ObjectStorageServer::processRequests()
+void ObjectStorageServer::processRequests(std::function<bool()> running)
 {
     using namespace std::chrono_literals;
     Identity lastMessageIdentity;
@@ -134,7 +135,15 @@ void ObjectStorageServer::processRequests()
                 }
             });
 
-            auto maybeMessage = ymq::syncRecvMessage(_ioSocket);
+            auto maybeMessageFuture = ymq::futureRecvMessage(_ioSocket);
+            while (maybeMessageFuture.wait_for(100ms) == std::future_status::timeout) {
+                if (!running()) {
+                    _logger.log(scaler::ymq::Logger::LoggingLevel::info, "ObjectStorageServer: stopped by user");
+                    pendingRequests.clear();
+                    return;
+                }
+            }
+            auto maybeMessage = maybeMessageFuture.get();
 
             if (!maybeMessage) {
                 auto error = maybeMessage.error();

{opengris_scaler-1.12.8 → opengris_scaler-1.12.11}/scaler/object_storage/object_storage_server.h

@@ -35,7 +35,8 @@ public:
         Identity identity                  = "ObjectStorageServer",
         std::string log_level              = "INFO",
         std::string log_format             = "%(levelname)s: %(message)s",
-        std::vector<std::string> log_paths = {"/dev/stdout"});
+        std::vector<std::string> log_paths = {"/dev/stdout"},
+        std::function<bool()> running      = []() { return true; });
 
     void waitUntilReady();
 
@@ -78,7 +79,7 @@ private:
 
     void closeServerReadyFds();
 
-    void processRequests();
+    void processRequests(std::function<bool()> stopCondition);
 
     void processSetRequest(std::shared_ptr<Client> client, std::pair<ObjectRequestHeader, Bytes> request);
 

{opengris_scaler-1.12.8 → opengris_scaler-1.12.11}/scaler/object_storage/pymod_object_storage_server.cpp

@@ -1,6 +1,8 @@
 #define PY_SSIZE_T_CLEAN
 #include <Python.h>
+#include <pyerrors.h>
 
+#include "scaler/io/ymq/pymod_ymq/gil.h"
 #include "scaler/object_storage/object_storage_server.h"
 
 extern "C" {
@@ -52,10 +54,19 @@ static PyObject* PyObjectStorageServerRun(PyObject* self, PyObject* args)
         logging_paths.push_back(PyUnicode_AsUTF8(path_obj));
     }
 
+    auto running = []() -> bool {
+        AcquireGIL gil;
+        (void)gil;
+        return PyErr_CheckSignals() == 0;
+    };
+
     ((PyObjectStorageServer*)self)
-        ->server.run(addr, std::to_string(port), identity, log_level, log_format, std::move(logging_paths));
+        ->server.run(
+            addr, std::to_string(port), identity, log_level, log_format, std::move(logging_paths), std::move(running));
 
-    Py_RETURN_NONE;
+    // TODO: Ideally, run should return a bool and we return failure with nullptr.
+    return nullptr;
+    // Py_RETURN_NONE;
 }
 
 static PyObject* PyObjectStorageServerWaitUntilReady(PyObject* self, [[maybe_unused]] PyObject* args)

opengris_scaler-1.12.11/scaler/version.txt

@@ -0,0 +1 @@
+1.12.11

opengris_scaler-1.12.11/tests/test_cluster_config.py

@@ -0,0 +1,49 @@
+import os
+import tempfile
+import unittest
+from argparse import Namespace
+
+from scaler.config.loader import load_config
+from scaler.config.section.cluster import ClusterConfig
+
+
+class TestClusterConfig(unittest.TestCase):
+    """Tests for the ClusterConfig class."""
+
+    def setUp(self):
+        self.temp_dir = tempfile.TemporaryDirectory()
+        self.config_path = os.path.join(self.temp_dir.name, "cluster.toml")
+        with open(self.config_path, "w") as f:
+            f.write(
+                """
+scheduler_address = "tcp://127.0.0.1:5555"
+num_of_workers = 2
+worker_names = "w1,w2"
+"""
+            )
+
+    def tearDown(self):
+        self.temp_dir.cleanup()
+
+    def test_load_from_toml_only(self):
+        """Test loading ClusterConfig from a flat TOML file."""
+        config = load_config(ClusterConfig, self.config_path, Namespace())
+        self.assertEqual(str(config.scheduler_address), "tcp://127.0.0.1:5555")
+        self.assertEqual(config.num_of_workers, 2)
+        self.assertEqual(config.worker_names.names, ["w1", "w2"])
+
+    def test_command_line_overrides_toml(self):
+        """Test that argparse args correctly override TOML values for ClusterConfig."""
+        worker_list = [f"worker_{i}" for i in range(20)]
+        args = Namespace(num_of_workers=20, worker_names=",".join(worker_list), heartbeat_interval_seconds=99)
+
+        config = load_config(ClusterConfig, self.config_path, args)
+
+        self.assertEqual(config.num_of_workers, 20)
+        self.assertEqual(config.worker_names.names, worker_list)
+        self.assertEqual(config.heartbeat_interval_seconds, 99)
+        self.assertEqual(str(config.scheduler_address), "tcp://127.0.0.1:5555")
+
+
+if __name__ == "__main__":
+    unittest.main()

opengris_scaler-1.12.11/tests/test_config_types.py

@@ -0,0 +1,50 @@
+import unittest
+
+from scaler.config.types.worker import WorkerCapabilities, WorkerNames
+from scaler.config.types.zmq import ZMQConfig
+
+
+class TestConfigTypes(unittest.TestCase):
+    """Tests for individual ConfigType helper classes."""
+
+    def test_zmq_config_validation(self):
+        """Test ZMQConfig.from_string raises ValueError for malformed strings."""
+        with self.assertRaises(ValueError):
+            ZMQConfig.from_string("this-is-not-a-valid-address")
+        with self.assertRaises(ValueError):
+            ZMQConfig.from_string("tcp://127.0.0.1")
+        with self.assertRaises(ValueError):
+            ZMQConfig.from_string("badprotocol://127.0.0.1:1234")
+
+        cfg = ZMQConfig.from_string("ipc://a-valid-path")
+        self.assertEqual(cfg.host, "a-valid-path")
+
+    def test_worker_names_config_value(self):
+        """Test the WorkerNames ConfigType class."""
+        wn = WorkerNames.from_string(" worker1 , worker2 ")
+        self.assertEqual(wn.names, ["worker1", "worker2"])
+        self.assertEqual(str(wn), "worker1,worker2")
+        self.assertEqual(len(wn), 2)
+        wn_empty = WorkerNames.from_string("")
+        self.assertEqual(wn_empty.names, [])
+
+    def test_worker_capabilities_config_value(self):
+        """Test the WorkerCapabilities ConfigType class."""
+        wc = WorkerCapabilities.from_string(" gpu=2, linux ")
+        self.assertEqual(wc.capabilities, {"gpu": 2, "linux": -1})
+        self.assertIn("gpu=2", str(wc))
+        self.assertIn("linux", str(wc))
+
+    def test_worker_capabilities_invalid_input(self):
+        """Test that WorkerCapabilities raises an error for non-integer values."""
+        with self.assertRaises(ValueError):
+            WorkerCapabilities.from_string("gpu=two")
+
+    def test_worker_capabilities_invalid_value_in_string(self):
+        """Test that WorkerCapabilities.from_string raises a helpful ValueError for non-integer values."""
+        with self.assertRaisesRegex(ValueError, "Expected an integer, but got 'MostPowerful'"):
+            WorkerCapabilities.from_string("linux,cpu=MostPowerful")
+
+
+if __name__ == "__main__":
+    unittest.main()

opengris_scaler-1.12.8/tests/test_config.py → opengris_scaler-1.12.11/tests/test_scheduler_config.py

@@ -3,58 +3,13 @@ import tempfile
 import unittest
 from argparse import Namespace
 
-from scaler.config.section.cluster import ClusterConfig
+from scaler.config import defaults
 from scaler.config.loader import load_config
 from scaler.config.section.scheduler import SchedulerConfig
 from scaler.config.types.object_storage_server import ObjectStorageConfig
-from scaler.config.types.worker import WorkerCapabilities, WorkerNames
-from scaler.config.types.zmq import ZMQConfig
-from scaler.config import defaults
 from scaler.scheduler.allocate_policy.allocate_policy import AllocatePolicy
 
 
-class TestConfigTypes(unittest.TestCase):
-    """Tests for individual ConfigType helper classes."""
-
-    def test_zmq_config_validation(self):
-        """Test ZMQConfig.from_string raises ValueError for malformed strings."""
-        with self.assertRaises(ValueError):
-            ZMQConfig.from_string("this-is-not-a-valid-address")
-        with self.assertRaises(ValueError):
-            ZMQConfig.from_string("tcp://127.0.0.1")
-        with self.assertRaises(ValueError):
-            ZMQConfig.from_string("badprotocol://127.0.0.1:1234")
-
-        cfg = ZMQConfig.from_string("ipc://a-valid-path")
-        self.assertEqual(cfg.host, "a-valid-path")
-
-    def test_worker_names_config_value(self):
-        """Test the WorkerNames ConfigType class."""
-        wn = WorkerNames.from_string(" worker1 , worker2 ")
-        self.assertEqual(wn.names, ["worker1", "worker2"])
-        self.assertEqual(str(wn), "worker1,worker2")
-        self.assertEqual(len(wn), 2)
-        wn_empty = WorkerNames.from_string("")
-        self.assertEqual(wn_empty.names, [])
-
-    def test_worker_capabilities_config_value(self):
-        """Test the WorkerCapabilities ConfigType class."""
-        wc = WorkerCapabilities.from_string(" gpu=2, linux ")
-        self.assertEqual(wc.capabilities, {"gpu": 2, "linux": -1})
-        self.assertIn("gpu=2", str(wc))
-        self.assertIn("linux", str(wc))
-
-    def test_worker_capabilities_invalid_input(self):
-        """Test that WorkerCapabilities raises an error for non-integer values."""
-        with self.assertRaises(ValueError):
-            WorkerCapabilities.from_string("gpu=two")
-
-    def test_worker_capabilities_invalid_value_in_string(self):
-        """Test that WorkerCapabilities.from_string raises a helpful ValueError for non-integer values."""
-        with self.assertRaisesRegex(ValueError, "Expected an integer, but got 'MostPowerful'"):
-            WorkerCapabilities.from_string("linux,cpu=MostPowerful")
-
-
 class TestSchedulerConfig(unittest.TestCase):
     """Tests for the SchedulerConfig class."""
 
@@ -136,44 +91,5 @@ allocate_policy = "even"
         self.assertEqual(config.logging_paths, ("/var/log/scaler.log", "/dev/stdout"))
 
 
-class TestClusterConfig(unittest.TestCase):
-    """Tests for the ClusterConfig class."""
-
-    def setUp(self):
-        self.temp_dir = tempfile.TemporaryDirectory()
-        self.config_path = os.path.join(self.temp_dir.name, "cluster.toml")
-        with open(self.config_path, "w") as f:
-            f.write(
-                """
-scheduler_address = "tcp://127.0.0.1:5555"
-# FIX: Changed num_of_workers to match the number of worker_names
-num_of_workers = 2
-worker_names = "w1,w2"
-"""
-            )
-
-    def tearDown(self):
-        self.temp_dir.cleanup()
-
-    def test_load_from_toml_only(self):
-        """Test loading ClusterConfig from a flat TOML file."""
-        config = load_config(ClusterConfig, self.config_path, Namespace())
-        self.assertEqual(str(config.scheduler_address), "tcp://127.0.0.1:5555")
-        self.assertEqual(config.num_of_workers, 2)
-        self.assertEqual(config.worker_names.names, ["w1", "w2"])
-
-    def test_command_line_overrides_toml(self):
-        """Test that argparse args correctly override TOML values for ClusterConfig."""
-        worker_list = [f"worker_{i}" for i in range(20)]
-        args = Namespace(num_of_workers=20, worker_names=",".join(worker_list), heartbeat_interval_seconds=99)
-
-        config = load_config(ClusterConfig, self.config_path, args)
-
-        self.assertEqual(config.num_of_workers, 20)
-        self.assertEqual(config.worker_names.names, worker_list)
-        self.assertEqual(config.heartbeat_interval_seconds, 99)
-        self.assertEqual(str(config.scheduler_address), "tcp://127.0.0.1:5555")
-
-
 if __name__ == "__main__":
     unittest.main()

opengris_scaler-1.12.8/scaler/version.txt

@@ -1 +0,0 @@
-1.12.8