k3wsjobd 0.1.1__tar.gz

k3wsjobd-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 张炎泼
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

k3wsjobd-0.1.1/PKG-INFO

@@ -0,0 +1,101 @@
+Metadata-Version: 2.4
+Name: k3wsjobd
+Version: 0.1.1
+Summary: Gevent-based WebSocket server for async job processing
+Author-email: Zhang Yanpo <drdr.xp@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/pykit3/k3wsjobd
+Project-URL: Documentation, https://k3wsjobd.readthedocs.io
+Keywords: websocket,gevent,job,async,server
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: gevent-websocket
+Requires-Dist: psutil
+Requires-Dist: k3thread
+Requires-Dist: k3utfjson
+Requires-Dist: k3jobq
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: coverage; extra == "dev"
+Provides-Extra: publish
+Requires-Dist: build; extra == "publish"
+Requires-Dist: twine; extra == "publish"
+Requires-Dist: pk3; extra == "publish"
+Provides-Extra: docs
+Requires-Dist: mkdocs>=1.5; extra == "docs"
+Requires-Dist: mkdocs-material>=9.0; extra == "docs"
+Requires-Dist: mkdocstrings[python]>=0.24; extra == "docs"
+Dynamic: license-file
+
+# k3wsjobd
+
+[![Action-CI](https://github.com/pykit3/k3wsjobd/actions/workflows/python-package.yml/badge.svg)](https://github.com/pykit3/k3wsjobd/actions/workflows/python-package.yml)
+[![Build Status](https://travis-ci.com/pykit3/k3wsjobd.svg?branch=master)](https://travis-ci.com/pykit3/k3wsjobd)
+[![Documentation Status](https://readthedocs.org/projects/k3wsjobd/badge/?version=stable)](https://k3wsjobd.readthedocs.io/en/stable/?badge=stable)
+[![Package](https://img.shields.io/pypi/pyversions/k3wsjobd)](https://pypi.org/project/k3wsjobd)
+
+This module is a gevent based websocket server. When the server receives a job description from a client, it runs that job asynchronously in a thread, and reports the progress back to the client periodically.
+
+k3wsjobd is a component of [pykit3] project: a python3 toolkit set.
+
+
+This module is a gevent based websocket server. When the server receives a job description from a client,
+it runs that job asynchronously in a thread, and reports the progress back to the client periodically.
+
+
+
+
+# Install
+
+```
+pip install k3wsjobd
+```
+
+# Synopsis
+
+```python
+
+from geventwebsocket import Resource, WebSocketServer
+import k3wsjobd
+from k3wsjobd import logging
+
+
+def run():
+    k3wsjobd.run(ip='127.0.0.1', port=33445)
+
+
+if __name__ == "__main__":
+    logger = logging.getLogger()
+    logger.setLevel(logging.INFO)
+
+    file_handler = logging.FileHandler('wsjobd.log')
+    formatter = logging.Formatter('[%(asctime)s, %(levelname)s] %(message)s')
+    file_handler.setFormatter(formatter)
+    logger.addHandler(file_handler)
+    run()
+
+```
+
+#   Author
+
+Zhang Yanpo (张炎泼) <drdr.xp@gmail.com>
+
+#   Copyright and License
+
+The MIT License (MIT)
+
+Copyright (c) 2015 Zhang Yanpo (张炎泼) <drdr.xp@gmail.com>
+
+
+[pykit3]: https://github.com/pykit3

k3wsjobd-0.1.1/README.md

@@ -0,0 +1,61 @@
+# k3wsjobd
+
+[![Action-CI](https://github.com/pykit3/k3wsjobd/actions/workflows/python-package.yml/badge.svg)](https://github.com/pykit3/k3wsjobd/actions/workflows/python-package.yml)
+[![Build Status](https://travis-ci.com/pykit3/k3wsjobd.svg?branch=master)](https://travis-ci.com/pykit3/k3wsjobd)
+[![Documentation Status](https://readthedocs.org/projects/k3wsjobd/badge/?version=stable)](https://k3wsjobd.readthedocs.io/en/stable/?badge=stable)
+[![Package](https://img.shields.io/pypi/pyversions/k3wsjobd)](https://pypi.org/project/k3wsjobd)
+
+This module is a gevent based websocket server. When the server receives a job description from a client, it runs that job asynchronously in a thread, and reports the progress back to the client periodically.
+
+k3wsjobd is a component of [pykit3] project: a python3 toolkit set.
+
+
+This module is a gevent based websocket server. When the server receives a job description from a client,
+it runs that job asynchronously in a thread, and reports the progress back to the client periodically.
+
+
+
+
+# Install
+
+```
+pip install k3wsjobd
+```
+
+# Synopsis
+
+```python
+
+from geventwebsocket import Resource, WebSocketServer
+import k3wsjobd
+from k3wsjobd import logging
+
+
+def run():
+    k3wsjobd.run(ip='127.0.0.1', port=33445)
+
+
+if __name__ == "__main__":
+    logger = logging.getLogger()
+    logger.setLevel(logging.INFO)
+
+    file_handler = logging.FileHandler('wsjobd.log')
+    formatter = logging.Formatter('[%(asctime)s, %(levelname)s] %(message)s')
+    file_handler.setFormatter(formatter)
+    logger.addHandler(file_handler)
+    run()
+
+```
+
+#   Author
+
+Zhang Yanpo (张炎泼) <drdr.xp@gmail.com>
+
+#   Copyright and License
+
+The MIT License (MIT)
+
+Copyright (c) 2015 Zhang Yanpo (张炎泼) <drdr.xp@gmail.com>
+
+
+[pykit3]: https://github.com/pykit3

k3wsjobd-0.1.1/__init__.py

@@ -0,0 +1,33 @@
+"""
+This module is a gevent based websocket server. When the server receives a job description from a client,
+it runs that job asynchronously in a thread, and reports the progress back to the client periodically.
+
+"""
+
+from importlib.metadata import version
+
+__version__ = version("k3wsjobd")
+
+from .wsjobd import (
+    JobdWebSocketApplication,
+    Job,
+    run,
+    SystemOverloadError,
+    JobError,
+    InvalidMessageError,
+    InvalidProgressError,
+    LoadingError,
+    JobNotInSessionError,
+)
+
+__all__ = [
+    "JobdWebSocketApplication",
+    "Job",
+    "run",
+    "SystemOverloadError",
+    "JobError",
+    "InvalidMessageError",
+    "InvalidProgressError",
+    "LoadingError",
+    "JobNotInSessionError",
+]

k3wsjobd-0.1.1/k3wsjobd.egg-info/PKG-INFO

@@ -0,0 +1,101 @@
+Metadata-Version: 2.4
+Name: k3wsjobd
+Version: 0.1.1
+Summary: Gevent-based WebSocket server for async job processing
+Author-email: Zhang Yanpo <drdr.xp@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/pykit3/k3wsjobd
+Project-URL: Documentation, https://k3wsjobd.readthedocs.io
+Keywords: websocket,gevent,job,async,server
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: gevent-websocket
+Requires-Dist: psutil
+Requires-Dist: k3thread
+Requires-Dist: k3utfjson
+Requires-Dist: k3jobq
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: coverage; extra == "dev"
+Provides-Extra: publish
+Requires-Dist: build; extra == "publish"
+Requires-Dist: twine; extra == "publish"
+Requires-Dist: pk3; extra == "publish"
+Provides-Extra: docs
+Requires-Dist: mkdocs>=1.5; extra == "docs"
+Requires-Dist: mkdocs-material>=9.0; extra == "docs"
+Requires-Dist: mkdocstrings[python]>=0.24; extra == "docs"
+Dynamic: license-file
+
+# k3wsjobd
+
+[![Action-CI](https://github.com/pykit3/k3wsjobd/actions/workflows/python-package.yml/badge.svg)](https://github.com/pykit3/k3wsjobd/actions/workflows/python-package.yml)
+[![Build Status](https://travis-ci.com/pykit3/k3wsjobd.svg?branch=master)](https://travis-ci.com/pykit3/k3wsjobd)
+[![Documentation Status](https://readthedocs.org/projects/k3wsjobd/badge/?version=stable)](https://k3wsjobd.readthedocs.io/en/stable/?badge=stable)
+[![Package](https://img.shields.io/pypi/pyversions/k3wsjobd)](https://pypi.org/project/k3wsjobd)
+
+This module is a gevent based websocket server. When the server receives a job description from a client, it runs that job asynchronously in a thread, and reports the progress back to the client periodically.
+
+k3wsjobd is a component of [pykit3] project: a python3 toolkit set.
+
+
+This module is a gevent based websocket server. When the server receives a job description from a client,
+it runs that job asynchronously in a thread, and reports the progress back to the client periodically.
+
+
+
+
+# Install
+
+```
+pip install k3wsjobd
+```
+
+# Synopsis
+
+```python
+
+from geventwebsocket import Resource, WebSocketServer
+import k3wsjobd
+from k3wsjobd import logging
+
+
+def run():
+    k3wsjobd.run(ip='127.0.0.1', port=33445)
+
+
+if __name__ == "__main__":
+    logger = logging.getLogger()
+    logger.setLevel(logging.INFO)
+
+    file_handler = logging.FileHandler('wsjobd.log')
+    formatter = logging.Formatter('[%(asctime)s, %(levelname)s] %(message)s')
+    file_handler.setFormatter(formatter)
+    logger.addHandler(file_handler)
+    run()
+
+```
+
+#   Author
+
+Zhang Yanpo (张炎泼) <drdr.xp@gmail.com>
+
+#   Copyright and License
+
+The MIT License (MIT)
+
+Copyright (c) 2015 Zhang Yanpo (张炎泼) <drdr.xp@gmail.com>
+
+
+[pykit3]: https://github.com/pykit3

k3wsjobd-0.1.1/k3wsjobd.egg-info/SOURCES.txt

@@ -0,0 +1,16 @@
+LICENSE
+README.md
+__init__.py
+pyproject.toml
+synopsis.py
+wsjobd.py
+./__init__.py
+./synopsis.py
+./wsjobd.py
+k3wsjobd.egg-info/PKG-INFO
+k3wsjobd.egg-info/SOURCES.txt
+k3wsjobd.egg-info/dependency_links.txt
+k3wsjobd.egg-info/requires.txt
+k3wsjobd.egg-info/top_level.txt
+test/test_doctest.py
+test/test_k3wsjobd.py

k3wsjobd-0.1.1/k3wsjobd.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

k3wsjobd-0.1.1/k3wsjobd.egg-info/requires.txt

@@ -0,0 +1,20 @@
+gevent-websocket
+psutil
+k3thread
+k3utfjson
+k3jobq
+
+[dev]
+pytest>=7.0
+ruff
+coverage
+
+[docs]
+mkdocs>=1.5
+mkdocs-material>=9.0
+mkdocstrings[python]>=0.24
+
+[publish]
+build
+twine
+pk3

k3wsjobd-0.1.1/k3wsjobd.egg-info/top_level.txt

@@ -0,0 +1 @@
+k3wsjobd

k3wsjobd-0.1.1/pyproject.toml

@@ -0,0 +1,63 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "k3wsjobd"
+version = "0.1.1"
+description = "Gevent-based WebSocket server for async job processing"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.9"
+authors = [
+    { name = "Zhang Yanpo", email = "drdr.xp@gmail.com" }
+]
+keywords = ["websocket", "gevent", "job", "async", "server"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Topic :: Software Development :: Libraries",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+]
+dependencies = [
+    "gevent-websocket",
+    "psutil",
+    "k3thread",
+    "k3utfjson",
+    "k3jobq",
+]
+
+[project.urls]
+Homepage = "https://github.com/pykit3/k3wsjobd"
+Documentation = "https://k3wsjobd.readthedocs.io"
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0",
+    "ruff",
+    "coverage",
+]
+publish = [
+    "build",
+    "twine",
+    "pk3",
+]
+docs = [
+    "mkdocs>=1.5",
+    "mkdocs-material>=9.0",
+    "mkdocstrings[python]>=0.24",
+]
+
+[tool.setuptools]
+packages = ["k3wsjobd"]
+
+[tool.setuptools.package-dir]
+k3wsjobd = "."
+
+[tool.ruff]
+line-length = 120

k3wsjobd-0.1.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

k3wsjobd-0.1.1/synopsis.py

@@ -0,0 +1,17 @@
+import k3wsjobd
+from k3wsjobd import logging
+
+
+def run():
+    k3wsjobd.run(ip="127.0.0.1", port=33445)
+
+
+if __name__ == "__main__":
+    logger = logging.getLogger()
+    logger.setLevel(logging.INFO)
+
+    file_handler = logging.FileHandler("wsjobd.log")
+    formatter = logging.Formatter("[%(asctime)s, %(levelname)s] %(message)s")
+    file_handler.setFormatter(formatter)
+    logger.addHandler(file_handler)
+    run()

k3wsjobd-0.1.1/test/test_doctest.py

@@ -0,0 +1,8 @@
+import doctest
+
+import k3wsjobd
+
+
+def load_tests(loader, tests, ignore):
+    tests.addTests(doctest.DocTestSuite(k3wsjobd))
+    return tests

k3wsjobd-0.1.1/test/test_k3wsjobd.py

@@ -0,0 +1,472 @@
+#!/usr/bin/env python
+# coding: utf-8
+
+import os
+import random
+import time
+import unittest
+
+import websocket
+
+import k3proc
+import k3utfjson
+import k3ut
+from k3wsjobd import Job
+from k3wsjobd.test.wsjobd_server import PORT
+
+dd = k3ut.dd
+
+random.seed(time.time())
+this_base = os.path.dirname(__file__)
+
+
+def subproc(script, env=None):
+    if env is None:
+        env = dict(
+            PYTHONPATH=this_base + "/../..",
+        )
+
+    return k3proc.shell_script(script, env=env)
+
+
+class TestWsjobd(unittest.TestCase):
+    @classmethod
+    def _clean(cls):
+        try:
+            subproc("python {b}/wsjobd_server.py stop".format(b=this_base))
+        except Exception as e:
+            dd("failed to stop wsjobd server: " + repr(e))
+
+        time.sleep(0.1)
+
+    @classmethod
+    def setUpClass(cls):
+        cls._clean()
+        subproc("python {b}/wsjobd_server.py start".format(b=this_base))
+        time.sleep(1)
+
+    @classmethod
+    def tearDownClass(cls):
+        cls._clean()
+
+    def _create_client(self):
+        ws = websocket.WebSocket()
+        ws.connect("ws://127.0.0.1:%d" % PORT)
+        ws.timeout = 6
+        return ws
+
+    def setUp(self):
+        self.ws = self._create_client()
+
+    def tearDown(self):
+        self.ws.close()
+
+    def get_random_ident(self):
+        return "random_ident_%d" % random.randint(10000, 99999)
+
+    def _wait_for_result(self, ws):
+        # wait for test_job_echo.run to fillin resp['result']
+        for _ in range(3):
+            resp = k3utfjson.load(ws.recv())
+            if "result" in resp:
+                break
+            time.sleep(0.1)
+        return resp
+
+    def test_invalid_jobdesc(self):
+        cases = (
+            ("foo", "not json"),
+            (k3utfjson.dump("foo"), "not dict"),
+            (k3utfjson.dump({}), "no func"),
+            (k3utfjson.dump({"func": "foo"}), "no ident"),
+            (k3utfjson.dump({"ident": "bar"}), "no func"),
+            (k3utfjson.dump({"ident": "bar", "func": {}}), "invalid func"),
+            (k3utfjson.dump({"ident": 44, "func": "foo"}), "invalid ident"),
+            (k3utfjson.dump({"ident": "foo", "func": "foo", "jobs_dir": {}}), "invalid jobs_dir"),
+        )
+        for msg, desc in cases:
+            ws = self._create_client()
+            ws.send(msg)
+            resp = k3utfjson.load(ws.recv())
+            self.assertIn("err", resp, desc)
+            ws.close()
+
+    def test_normal_job(self):
+        job_desc = {
+            "func": "test_job_normal.run",
+            "ident": self.get_random_ident(),
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        self.ws.send(k3utfjson.dump(job_desc))
+
+        resp = self._wait_for_result(self.ws)
+        dd(resp)
+        self.assertEqual("foo", resp["result"], "test get result")
+
+    def test_report_interval(self):
+        job_desc = {
+            "func": "test_job_loop_10.run",
+            "ident": self.get_random_ident(),
+            "progress": {
+                "interval": 0.5,
+            },
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        self.ws.send(k3utfjson.dump(job_desc))
+
+        self.ws.recv()
+        first_report_time = time.time()
+
+        self.ws.recv()
+        second_report_time = time.time()
+
+        actual_interval = second_report_time - first_report_time
+
+        diff = actual_interval - job_desc["progress"]["interval"]
+
+        # tolerate 0.1 second of difference
+        self.assertLess(diff, 0.1)
+
+    def test_invalid_progress_key(self):
+        job_desc = {
+            "func": "test_job_progress_key.run",
+            "ident": self.get_random_ident(),
+            "progress": {
+                "key": "inexistent",
+            },
+            "report_system_load": True,
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        self.ws.send(k3utfjson.dump(job_desc))
+
+        resp = self.ws.recv()
+        dd(repr(resp))
+        resp = k3utfjson.load(resp)
+        self.assertIsNone(resp)
+
+    def test_progress_key(self):
+        job_desc = {
+            "func": "test_job_progress_key.run",
+            "ident": self.get_random_ident(),
+            "progress": {
+                "key": "foo",
+            },
+            # progress_sender may try to read data["foo"] before job starts.
+            # need to preset a value.
+            "foo": "0%",
+            "report_system_load": True,
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        self.ws.send(k3utfjson.dump(job_desc))
+
+        resp = self.ws.recv()
+        dd(repr(resp))
+        resp = k3utfjson.load(resp)
+        # progress may be set by job runner or job desc
+        self.assertIn(resp, ("80%", "0%"))
+
+    def test_check_system_load(self):
+        job_desc = {
+            "func": "test_job_normal.run",
+            "ident": self.get_random_ident(),
+            "check_load": {
+                "mem_low_threshold": 100 * 1024**3,
+                "cpu_low_threshold": 0,
+            },
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        self.ws.send(k3utfjson.dump(job_desc))
+
+        resp = self.ws.recv()
+        resp = k3utfjson.load(resp)
+        self.assertEqual("SystemOverloadError", resp["err"])
+
+        job_desc = {
+            "func": "test_job_normal.run",
+            "ident": self.get_random_ident(),
+            "check_load": {
+                "cpu_low_threshold": 100.1,
+                "mem_low_threshold": 0,
+            },
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        ws2 = self._create_client()
+        ws2.send(k3utfjson.dump(job_desc))
+
+        resp = ws2.recv()
+        resp = k3utfjson.load(resp)
+        ws2.close()
+        self.assertEqual("SystemOverloadError", resp["err"])
+
+    def test_max_client_number(self):
+        job_desc = {
+            "func": "test_job_loop_10.run",
+            "ident": self.get_random_ident(),
+            "check_load": {
+                "max_client_number": 1,
+                "cpu_low_threshold": 0,
+                "mem_low_threshold": 0,
+            },
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        self.ws.send(k3utfjson.dump(job_desc))
+
+        resp = self.ws.recv()
+        resp = k3utfjson.load(resp)
+        self.assertNotIn("err", resp)
+
+        job_desc = {
+            "func": "test_job_loop_10.run",
+            "ident": self.get_random_ident(),
+            "check_load": {
+                "max_client_number": 1,
+                "cpu_low_threshold": 0,
+                "mem_low_threshold": 0,
+            },
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        ws2 = self._create_client()
+        ws2.send(k3utfjson.dump(job_desc))
+
+        resp = ws2.recv()
+        resp = k3utfjson.load(resp)
+        self.assertEqual("SystemOverloadError", resp["err"])
+        ws2.close()
+
+    def test_module_not_exists(self):
+        job_desc = {
+            "func": "foo.bar",
+            "ident": self.get_random_ident(),
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        self.ws.send(k3utfjson.dump(job_desc))
+
+        resp = k3utfjson.load(self.ws.recv())
+        self.assertIn("err", resp)
+
+    def test_report_system_load(self):
+        job_desc = {
+            "func": "test_job_normal.run",
+            "ident": self.get_random_ident(),
+            "report_system_load": True,
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        self.ws.send(k3utfjson.dump(job_desc))
+
+        resp = k3utfjson.load(self.ws.recv())
+        self.assertIn("mem_available", resp["system_load"])
+        self.assertIn("cpu_idle_percent", resp["system_load"])
+        self.assertIn("client_number", resp["system_load"])
+
+    def test_same_ident_same_job(self):
+        ident = self.get_random_ident()
+        job_desc = {
+            "func": "test_job_echo.run",
+            "ident": ident,
+            "echo": "foo",
+            "sleep_time": 10,
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        self.ws.send(k3utfjson.dump(job_desc))
+
+        resp = k3utfjson.load(self.ws.recv())
+        self.assertEqual("foo", resp["result"])
+
+        job_desc = {
+            "func": "test_job_echo.run",
+            "ident": ident,
+            "echo": "bar",
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        ws = self._create_client()
+        ws.send(k3utfjson.dump(job_desc))
+
+        resp = k3utfjson.load(ws.recv())
+        ws.close()
+        # not bar, because the ident is same as the first job,
+        # if job exists, it will not create a new one
+        self.assertEqual("foo", resp["result"])
+        self.assertEqual("foo", resp["echo"])
+
+    def test_same_ident_different_job(self):
+        ident = self.get_random_ident()
+        job_desc = {
+            "func": "test_job_echo.run",
+            "ident": ident,
+            "echo": "foo",
+            "sleep_time": 0.1,
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        self.ws.send(k3utfjson.dump(job_desc))
+
+        resp = self._wait_for_result(self.ws)
+        self.assertEqual("foo", resp["result"])
+
+        time.sleep(0.2)
+
+        job_desc = {
+            "func": "test_job_echo.run",
+            "ident": ident,
+            "echo": "bar",
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        ws2 = self._create_client()
+        ws2.send(k3utfjson.dump(job_desc))
+
+        resp = self._wait_for_result(ws2)
+        ws2.close()
+        # old job with the same ident has exit, it will create a new one
+        self.assertEqual("bar", resp["result"])
+        self.assertEqual("bar", resp["echo"])
+
+    def test_client_close(self):
+        ident = self.get_random_ident()
+        job_desc = {
+            "func": "test_job_echo.run",
+            "ident": ident,
+            "echo": "foo",
+            "time_sleep": 10,
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        self.ws.send(k3utfjson.dump(job_desc))
+        self.ws.close()
+        self.ws = self._create_client()
+
+        job_desc = {
+            "func": "test_job_echo.run",
+            "ident": ident,
+            "echo": "bar",
+            "time_sleep": 10,
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        self.ws.send(k3utfjson.dump(job_desc))
+
+        resp = self.ws.recv()
+        resp = k3utfjson.load(resp)
+        self.assertEqual("foo", resp["result"])
+
+    def test_invalid_cpu_sample_interval(self):
+        job_desc = {
+            "func": "test_job_normal.run",
+            "ident": self.get_random_ident(),
+            "cpu_sample_interval": "foo",
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        self.ws.send(k3utfjson.dump(job_desc))
+
+        resp = k3utfjson.load(self.ws.recv())
+        self.assertEqual("InvalidMessageError", resp["err"])
+
+    def test_invalid_check_load_args(self):
+        cases = (
+            {"check_load": {"mem_low_threshold": "foo", "cpu_low_threshold": 0}},
+            {"check_load": {"cpu_low_threshold": None, "mem_low_threshold": 0}},
+            {"check_load": {"max_client_number": {}, "cpu_low_threshold": 0, "mem_low_threshold": 0}},
+        )
+
+        job_desc = {
+            "func": "test_job_normal.run",
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        for case in cases:
+            case.update(job_desc)
+            case["ident"] = self.get_random_ident()
+
+            ws = self._create_client()
+            ws.send(k3utfjson.dump(case))
+
+            resp = k3utfjson.load(ws.recv())
+            ws.close()
+            self.assertEqual("InvalidMessageError", resp["err"])
+
+    def test_func_not_exists(self):
+        ident = self.get_random_ident()
+        job_desc = {
+            "func": "test_job_echo.func_not_exists",
+            "ident": ident,
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+        }
+
+        self.ws.send(k3utfjson.dump(job_desc))
+        resp = self.ws.recv()
+        resp = k3utfjson.load(resp)
+        self.assertEqual("LoadingError", resp["err"])
+
+    def test_worker_exception(self):
+        ident = self.get_random_ident()
+        job_desc = {
+            "func": "test_job_worker_exception.run",
+            "ident": ident,
+            "jobs_dir": "k3wsjobd/test/test_jobs",
+            "progress": {
+                "interval": 0.1,
+            },
+        }
+
+        self.ws.send(k3utfjson.dump(job_desc))
+
+        for i in range(10):
+            resp = self.ws.recv()
+            self.assertNotIn("err", resp)
+
+        with self.assertRaises(Exception):
+            for i in range(3):
+                self.ws.recv()
+
+    def test_create_job(self):
+        self.assertEqual(0, len(Job.sessions))
+
+        def f(self):
+            time.sleep(0.2)
+            return
+
+        Job("channel", {"ident": "a"}, f)
+        joba = Job.sessions["a"]
+
+        self.assertEqual(1, len(Job.sessions))
+
+        Job("channel", {"ident": "a"}, f)
+        joba1 = Job.sessions["a"]
+
+        #  joba already exists
+        self.assertEqual(joba1, joba)
+        self.assertEqual(1, len(Job.sessions))
+
+        time.sleep(0.15)
+        Job("channel", {"ident": "b"}, f)
+
+        self.assertEqual(2, len(Job.sessions))
+
+        time.sleep(0.15)
+        self.assertEqual(1, len(Job.sessions))
+
+        time.sleep(0.1)
+        self.assertEqual(0, len(Job.sessions))
+
+        #  test use same ident after first one exit
+        Job("channel", {"ident": "a"}, f)
+        joba1 = Job.sessions["a"]
+
+        time.sleep(0.25)
+
+        Job("channel", {"ident": "a"}, f)
+        joba2 = Job.sessions["a"]
+        self.assertNotEqual(joba1, joba2)

k3wsjobd-0.1.1/wsjobd.py

@@ -0,0 +1,355 @@
+#!/usr/bin/env python
+# coding: utf-8
+
+import logging
+import threading
+from collections import OrderedDict
+
+import psutil
+from geventwebsocket import Resource
+from geventwebsocket import WebSocketApplication
+from geventwebsocket import WebSocketError
+from geventwebsocket import WebSocketServer
+
+import k3thread
+import k3utfjson
+import k3jobq
+
+logger = logging.getLogger(__name__)
+
+MEM_AVAILABLE = "mem_available"
+CPU_IDLE_PERCENT = "cpu_idle_percent"
+CLIENT_NUMBER = "client_number"
+JOBS_DIR = "jobs"
+
+# - `mem_low_threshold`
+# set the min size of available memory the system should have, if not satified, return error. the default is 500M
+#
+# - `cpu_low_threshold`
+# set the min idle percent of the system cpu, if not satified, return error. the default is 3
+#
+# - `max_client_number`
+# set the max client number, if this number reached, new connection will be refused. the default is 1000
+
+
+CHECK_LOAD_PARAMS = {
+    "mem_low_threshold": {
+        "load_name": MEM_AVAILABLE,
+        "default": 500 * 1024**2,  # 500M
+        "greater": True,
+    },
+    "cpu_low_threshold": {
+        "load_name": CPU_IDLE_PERCENT,
+        "default": 3,  # 3%
+        "greater": True,
+    },
+    "max_client_number": {
+        "load_name": CLIENT_NUMBER,
+        "default": 1000,
+        "greater": False,
+    },
+}
+
+
+class SystemOverloadError(Exception):
+    pass
+
+
+class JobError(Exception):
+    pass
+
+
+class InvalidMessageError(JobError):
+    pass
+
+
+class InvalidProgressError(InvalidMessageError):
+    pass
+
+
+class LoadingError(JobError):
+    pass
+
+
+class JobNotInSessionError(JobError):
+    pass
+
+
+class Job(object):
+    lock = threading.RLock()
+    sessions = {}
+
+    def __init__(self, channel, msg, func):
+        """
+        Job description is a string formatted in json, it is used to tell wsjobd what to do,
+         it can contain the following fields:
+        :param channel:
+        report_system_load: a boolean to indicate whether to report system load, if set to true and the progress info
+        is a dict, then the system load info will be add to progress dict by key `system_load`, the value is also a dict
+        , which contains three fields: `mem_available`, `cpu_idle_percent`, `client_number`.
+        cpu_sample_interval: set the interval used by psutil.cpu_times_percent, the default is 0.02
+        :param msg:
+        check_load: a dict to enable system load check, also to set customed thresholds, the can contain the following fields
+        ident : required. the identifier of a job, it is used to prevent from creating the same job repeatedly.
+        progress: a dict to set progress reporting related settings, it can contain the following fields:
+            - `interval`: the interval of progress reporting, default is 5 seconds
+        `key`: the sub field in which the progress info located
+        :param func: required. the function of job, it contain module name and function name, seperated by a dot,
+        the module shoud in the `jobs` directory.
+        """
+        self.ident = msg["ident"]
+        self.channel = channel
+        self.data = msg
+        self.worker = func
+        self.ctx = {}
+        self.err = None
+        self.progress_available = threading.Event()
+
+        if self.ident in self.sessions:
+            logger.info(
+                "job: %s already exists, created by chennel %s" % (self.ident, repr(self.sessions[self.ident].channel))
+            )
+            return
+        else:
+            self.sessions[self.ident] = self
+            logger.info(
+                ("inserted job: %s to sessions by channel %s, " + "there are %d jobs in sessions now")
+                % (self.ident, repr(self.channel), len(self.sessions))
+            )
+
+        self.thread = k3thread.start(target=self.work, args=(), daemon=True)
+
+    def work(self):
+        logger.info("job %s started, the data is: %s" % (self.ident, self.data))
+
+        try:
+            self.worker(self)
+        except Exception as e:
+            logger.exception("job %s got exception: %s" % (self.ident, repr(e)))
+            self.err = e
+        finally:
+            logger.info("job %s ended" % self.ident)
+            self.close()
+
+    def close(self):
+        with self.lock:
+            del self.sessions[self.ident]
+            logger.info(
+                ("removed job: %s from sessions, there are %d " + "jobs in sessions now")
+                % (self.ident, len(self.sessions))
+            )
+
+
+def get_or_create_job(channel, msg, func):
+    with Job.lock:
+        Job(channel, msg, func)
+
+        job = Job.sessions.get(msg["ident"])
+
+    return job
+
+
+def progress_sender(job, channel, interval=5, stat=None):
+    stat = stat or (lambda data: data)
+    data = job.data
+
+    i = 10
+    try:
+        while True:
+            # if thread died due to some reason, still send 10 stats
+            if not job.thread.is_alive():
+                logger.info("job %s died: %s" % (job.ident, repr(job.err)))
+                if i == 0:
+                    channel.ws.close()
+                    break
+                i -= 1
+
+            logger.info("jod %s on channel %s send progress: %s" % (job.ident, repr(channel), repr(stat(data))))
+
+            to_send = stat(data)
+            if channel.report_system_load and isinstance(to_send, dict):
+                to_send["system_load"] = channel.get_system_load()
+
+            channel.ws.send(k3utfjson.dump(to_send))
+
+            if job.progress_available.wait(interval):
+                job.progress_available.clear()
+
+    except WebSocketError as e:
+        if channel.ws.closed:
+            logger.info("the client has closed the connection")
+        else:
+            logger.exception(
+                ("got websocket error when sending progress on" + " channel %s: %s") % (repr(channel), repr(e))
+            )
+
+    except Exception as e:
+        logger.exception("got exception when sending progress on channel %s: %s" % (repr(channel), repr(e)))
+        channel.ws.close()
+
+
+class JobdWebSocketApplication(WebSocketApplication):
+    jobq_mgr = None
+
+    def on_open(self):
+        logger.info("on open, the channel is: " + repr(self))
+        self.ignore_message = False
+
+    def _parse_request(self, message):
+        try:
+            try:
+                msg = k3utfjson.load(message)
+            except Exception:
+                raise InvalidMessageError("message is not a vaild json string: %s" % message)
+
+            self._check_msg(msg)
+
+            self.report_system_load = msg.get("report_system_load") is True
+            self.cpu_sample_interval = msg.get("cpu_sample_interval", 0.02)
+
+            if not isinstance(self.cpu_sample_interval, (int, float)):
+                raise InvalidMessageError("cpu_sample_interval is not a number")
+
+            check_load = msg.get("check_load")
+            if isinstance(check_load, dict):
+                self._check_system_load(check_load)
+
+            self.jobs_dir = msg.get("jobs_dir", JOBS_DIR)
+
+            self._setup_response(msg)
+            return
+
+        except SystemOverloadError as e:
+            logger.info("system overload on chennel %s, %s" % (repr(self), repr(e)))
+            self._send_err_and_close(e)
+
+        except JobError as e:
+            logger.info("error on channel %s while handling message, %s" % (repr(self), repr(e)))
+            self._send_err_and_close(e)
+
+        except Exception as e:
+            logger.exception(("exception on channel %s while handling " + "message, %s") % (repr(self), repr(e)))
+            self._send_err_and_close(e)
+
+    def on_message(self, message):
+        logger.info("on message, the channel is: %s, the message is: %s" % (repr(self), message))
+        if self.ignore_message:
+            return
+
+        else:
+            self.ignore_message = True
+
+        self.jobq_mgr.put((self, message))
+
+    def _send_err_and_close(self, err):
+        try:
+            err_msg = {
+                "err": err.__class__.__name__,
+                "val": err.args,
+            }
+            self.ws.send(k3utfjson.dump(err_msg))
+        except Exception as e:
+            logger.error(("error on channel %s while sending back error " + "message, %s") % (repr(self), repr(e)))
+
+    def get_system_load(self):
+        return {
+            MEM_AVAILABLE: psutil.virtual_memory().available,
+            CPU_IDLE_PERCENT: psutil.cpu_times_percent(self.cpu_sample_interval).idle,
+            CLIENT_NUMBER: len(self.protocol.server.clients),
+        }
+
+    def _check_system_load(self, check_load):
+        system_load = self.get_system_load()
+
+        for param_name, param_attr in CHECK_LOAD_PARAMS.items():
+            param_value = check_load.get(param_name, param_attr["default"])
+
+            if not isinstance(param_value, (int, float)):
+                raise InvalidMessageError("%s is not a number" % param_name)
+
+            load_name = param_attr["load_name"]
+            diff = system_load[load_name] - param_value
+
+            if not param_attr["greater"]:
+                diff = 0 - diff
+
+            if diff < 0:
+                raise SystemOverloadError(
+                    "%s: %d is %s than: %d"
+                    % (load_name, system_load[load_name], param_attr["greater"] and "less" or "greater", param_value)
+                )
+
+    def _check_msg(self, msg):
+        if not isinstance(msg, dict):
+            raise InvalidMessageError("message is not dictionary")
+
+        if "ident" not in msg:
+            raise InvalidMessageError("'ident' is not in message")
+
+        if "func" not in msg:
+            raise InvalidMessageError("'func' is not in message")
+
+    def _setup_response(self, msg):
+        func = self._get_func_by_name(msg)
+        channel = self
+        job = get_or_create_job(channel, msg, func)
+
+        if job is None:
+            raise JobNotInSessionError("job not in sessions: " + repr(Job.sessions))
+
+        progress = msg.get("progress", {})
+
+        if progress in (None, False):
+            return
+
+        if not isinstance(progress, dict):
+            raise InvalidProgressError("the progress in message is not a dictionary")
+
+        interval = progress.get("interval", 5)
+        progress_key = progress.get("key")
+
+        def get_progress(r):
+            if progress_key is None:
+                return r
+            return r.get(progress_key)
+
+        k3thread.start(target=progress_sender, args=(job, channel, interval, get_progress), daemon=True)
+
+    def _get_func_by_name(self, msg):
+        mod_func = self.jobs_dir.split("/") + msg["func"].split(".")
+        mod_path = ".".join(mod_func[:-1])
+        func_name = mod_func[-1]
+
+        try:
+            mod = __import__(mod_path)
+        except (ImportError, SyntaxError) as e:
+            raise LoadingError("failed to import %s: %s" % (mod_path, repr(e)))
+
+        for mod_name in mod_path.split(".")[1:]:
+            mod = getattr(mod, mod_name)
+
+        logger.info("mod imported from: " + repr(mod.__file__))
+
+        try:
+            func = getattr(mod, func_name)
+        except AttributeError:
+            raise LoadingError("function not found: " + repr(func_name))
+
+        return func
+
+    def on_close(self, reason):
+        logger.info("on close, the channel is: " + repr(self))
+
+
+def _parse_request(args):
+    app, msg = args
+    app._parse_request(msg)
+
+
+def run(ip="127.0.0.1", port=63482, jobq_thread_count=10):
+    JobdWebSocketApplication.jobq_mgr = k3jobq.JobManager([(_parse_request, jobq_thread_count)])
+
+    WebSocketServer(
+        (ip, port),
+        Resource(OrderedDict({"/": JobdWebSocketApplication})),
+    ).serve_forever()