dsmq 1.3.0__tar.gz → 1.4.0__tar.gz

{dsmq-1.3.0 → dsmq-1.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dsmq
-Version: 1.3.0
+Version: 1.4.0
 Summary: A dead simple message queue
 License-File: LICENSE
 Requires-Python: >=3.10
@@ -152,6 +152,16 @@ connected to the server.
 in the topic, or the topic doesn't yet exist,
 returns `""`.
 
+### `get_latest(topic)`
+
+Get the *most recent* eligible message from the queue named `topic`.
+All the messages older than that in the queue become ineligible and never
+get seen by the client.
+- `topic` (str)
+- returns str, the content of the message. If there was no eligble message
+in the topic, or the topic doesn't yet exist,
+returns `""`.
+
 ### `get_wait(topic)`
 
 A variant of `get()` that retries a few times until it gets

{dsmq-1.3.0 → dsmq-1.4.0}/README.md

@@ -142,6 +142,16 @@ connected to the server.
 in the topic, or the topic doesn't yet exist,
 returns `""`.
 
+### `get_latest(topic)`
+
+Get the *most recent* eligible message from the queue named `topic`.
+All the messages older than that in the queue become ineligible and never
+get seen by the client.
+- `topic` (str)
+- returns str, the content of the message. If there was no eligble message
+in the topic, or the topic doesn't yet exist,
+returns `""`.
+
 ### `get_wait(topic)`
 
 A variant of `get()` that retries a few times until it gets

dsmq-1.4.0/post.md

@@ -0,0 +1,244 @@
+# dsmq
+
+I'd like to introduce [dsmq](https://github.com/brohrer/dsmq),
+the Dead Simple Message Queue, to the world.
+
+Part mail room, part bulletin board, dsmq is a central location for sharing messages
+between processes, even when they are running on different computers.
+
+Its defining characteristic is bare-bones simplicity.
+
+![
+A screenshot of the dsmq GitHub repository:
+src/dsmq,
+.python-version,
+LICENSE,
+README.md,
+pyproject.toml,
+README,
+MIT license,
+Dead Simple Message Queue
+What it does
+Part mail room, part bulletin board, dsmq is a central location
+for sharing messages between processes, even when they are running
+on computers scattered around the world.
+Its defining characteristic is bare-bones simplicity.
+](https://brandonrohrer.com/images/dsmq/dsmq_00.png)
+
+## What dsmq does
+
+A message queue lets different processes talk to each other, within or across machines.
+Message queues are a waystation, a place to publish messages and hold them until
+they get picked up by the process that needs them.
+
+In dsmq, a program running the message queue starts up first (the server).
+It handles all the
+receiving, delivering, sorting, and storing of messages.
+
+Other programs (the clients) connect to the server. They add messages to a queue
+or read messages from a queue. Each queue is a separate topic. 
+
+## Why message queues?
+Message queues are invaluable for distributed systems of all sorts,
+but my favorite application is robotics.
+Robots typically have several (or many) processes doing different things at different
+speeds. Communication between processes is a fundamental part of any moderately
+complex automated system.
+When [ROS](https://www.ros.org) (the Robot Operating System) was
+released, one of the big gifts it gave to robot builders was a reliable way to
+pass messages.
+
+## Why another message queue?
+
+There are lots of [message queues](https://en.wikipedia.org/wiki/Message_queue)
+in the world, and some are quite well known--Amazon SQS,
+RabbitMQ, Apache Kafka to name a few. It's fair to ask why this one was created.
+
+The official reason for dsmq's existence is that all the other available options
+are pretty heavy. Take RabbitMQ for instance, a popular open source message queue.
+It has hundreds of associated repositories and it's core
+[rabbitmq-server](https://github.com/rabbitmq/rabbitmq-server) repo has many thousands
+of lines of code. It's a heavy lift to import this to support a small robotics project,
+and code running on small edge devices may struggle to run it at all.
+
+RabbitMQ is also mature and optimized and dockerized and multi-platform
+and fully featured and a lot of other things a robot doesn't need. It would
+be out of balance to use it for a small project.
+
+![https://brandonrohrer.com/images/dsmq/dsmq_01.png]("""
+Screenshot of the RabbitMQ-server GitHub repo showing that it
+is many times larger than dsmq""")
+
+dsmq is only about 200 lines of Python, including client and server code. It's *tiny*.
+Good for reading and understanding front-to-back when you're integrating it with
+your project.
+
+But the real reason is that I wanted to understand how a message queue might work
+and the best way I know to learn this is to build one.
+
+## How to use dsmq
+
+### Install it
+
+```
+pip install dsmq
+```
+or
+```
+uv add dsmq
+```
+
+### Spin up a dsmq server
+
+```python
+from dsmq import dsmq
+dsmq.start_server(host="127.0.0.1", port=12345)
+```
+
+### Connect a client to a dsmq server
+
+```python
+mq = dsmq.connect_to_server(host="127.0.0.1", port=12345)
+```
+
+### Add a message to a topic queue
+
+```python
+topic = "greetings"
+msg = "hello world!"
+mq.put(topic, msg)
+```
+
+### Read a message from a topic queue
+
+```python
+topic = "greetings"
+msg = mq.get(topic)
+```
+
+### Run a demo
+
+0. Open 3 separate terminal windows.
+1. In the first, run `src/dsmq/dsmq.py`.
+2. In the second, run `src/dsmq/example_put_client.py`.
+3. In the third, run `src/dsmq/example_get_client.py`.
+
+Alternatively, if you're on Linux just run `src/dsmq/demo_linux.py`.
+(Linux has some process forking capabilities that Windows doesn't and
+that macOS forbids. It makes for a nice self-contained multiprocess demo.)
+
+## How it works
+
+- Many clients can read messages from the same topic queue. dsmq uses a one-to-many
+publication model.
+
+- A client will get the oldest message available on a requested topic.
+Queues are first-in-first-out.
+
+- Clients will only be able to get messages that were added to a queue after the
+time they connected to the server. Any messages older that that won't be visible.
+
+- dsmq is asyncronous. There are no guarantees about how soon a message will arrive
+at its intended client.
+
+- dsmq is backed by an in-memory SQLite database. If your message volumes
+get larger than your RAM, you will reach an out-of-memory condition.
+```python
+sqlite_conn = sqlite3.connect("file:mem1?mode=memory&cache=shared")
+```
+- `put()` and `get()` operations are fairly quick--less than 100 $`\mu`$s of processing
+time plus any network latency--so it can comfortably handle requests at rates of
+hundreds of times per second. But if you have several clients reading and writing
+at 1 kHz or more, you may overload the queue.
+
+- A client will not be able to read any of the messages that were put into
+a queue before it connected.
+
+- Messages older than 600 seconds will be deleted from the queue.
+- In case of contention for the lock on the database, failed queries will be
+retried several times, with exponential backoff. None of this is visible to
+the clients, but it helps keep the internal operations reliable.
+
+```python
+for i_retry in range(_n_retries):
+    try:
+        cursor.execute("""...
+```
+
+- To keep things bare bones, the connections are all implemented in the
+socket layer, rather than WebSockets or http in the application layer.
+```python
+self.dsmq_conn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+self.dsmq_conn.connect((host, port))
+```
+- ...however, working on this lower level meant dsmq had to reinvent the wheel of
+message headers. Each dsmq message is sent with a header that tells it exactly
+how many bytes to expect in it. The header format is simplistic--a message
+with one million plus the number of byes in the following message. Doing it this
+way means that the header is
+exactly 23 bytes long every time. (As long as the message is less than nine million
+bytes long.)
+
+- Every time a new client connects to the server, a new thread is created.
+That thread listens for any `get()` or `put()` requests the client might make
+and handles them immediately.
+```python
+Thread(target=_handle_client_connection, args=(socket_conn,)).start()
+```
+
+- dsmq retrieves the oldest eligible message from the queue. If a client wants
+to ensure it is getting the most recent message from the queue, it will need
+to iteratively get messages until there are no more left to be gotten.
+```python
+msg_str = "<no response>"
+response = None
+while response != "":
+    msg_str = response
+    response = self.mq.get("<topic>")
+```
+
+- dsmq messages are text fields, but Python dictionaries are a very convenient and
+common format for passing structured messages. Use the `json` library to convert
+from dictionaries to strings and back.
+```python
+topic = "update"
+msg_dict = {"timestep": 374, "value": 3.897}
+msg_str = json.dumps(msg_dict)
+dsmq.put(topic, msg_str)
+
+msg_str = dsmq.get(topic)
+msg_dict = json.loads(msg_str)
+```
+
+- dsmq is opinionated. Parameters for controlling behavior are set to reasonable
+defaults and not exposed to the user. The additional complexity in the API is
+not worth the value of making them user-controlled.
+However they are also clearly labeled and very easy to find. If anyone cares enough to
+play with them, they are strongly encouraged to fork dsmq and make it their own.
+
+```python
+_message_length_offset = 1_000_000
+_header_length = 23
+_n_retries = 5
+_first_retry = 0.01  # seconds
+_time_to_live = 600.0  # seconds
+```
+
+- dsmq is deliberately built to have as few dependencies as it can get away with.
+As of this writing, it doesn't depend on any third party packages and just a handful
+of core packages: `json`, `socket`, `sqlite3`, `sys`, and `threading`.
+
+# Dead Simple
+
+Dead simple is an aesthetic.
+It says that the ideal is achieved not when nothing more can be added,
+but when nothing more can be taken away.
+It is the aims to follow the apocryphal advice of Albert Einstein, to make
+a thing as simple as possible, but no simpler.
+
+Dead simple is like keystroke golfing, but instead of minimizing the number of
+characters, it minimizes the number of concepts a developer or a user
+has to hold in their head.
+
+I've tried to embody it in dsmq.
+dsmq has fewer lines of code than RabbitMQ has *repositories*. And that tickles me. 

{dsmq-1.3.0 → dsmq-1.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dsmq"
-version = "1.3.0"
+version = "1.4.0"
 description = "A dead simple message queue"
 readme = "README.md"
 requires-python = ">=3.10"

{dsmq-1.3.0 → dsmq-1.4.0}/src/dsmq/client.py

@@ -18,6 +18,7 @@ def connect(host=_default_host, port=_default_port, verbose=False):
 class DSMQClientSideConnection:
     def __init__(self, host, port, verbose=False):
         self.uri = f"ws://{host}:{port}"
+        self.port = port
         self.verbose = verbose
         if self.verbose:
             print(f"Connecting to dsmq server at {self.uri}")
@@ -40,13 +41,41 @@ class DSMQClientSideConnection:
 
     def get(self, topic):
         msg = {"action": "get", "topic": topic}
-        self.websocket.send(json.dumps(msg))
+        try:
+            self.websocket.send(json.dumps(msg))
+        except ConnectionClosedError:
+            return ""
+
+        try:
+            msg_text = self.websocket.recv()
+        except ConnectionClosedError:
+            self.close()
+            return ""
+
+        msg = json.loads(msg_text)
+        return msg["message"]
+
+    def get_latest(self, topic):
+        """
+        A variant of `get()` that grabs the latest available message
+        (if there is one) rather than grabbing the oldest unread message.
+        It will not go back to read older ones on subsequent calls;
+        it will leave them unread.
+        """
+        msg = {"action": "get_latest", "topic": topic}
+        try:
+            self.websocket.send(json.dumps(msg))
+        except ConnectionClosedError:
+            return ""
+
         try:
             msg_text = self.websocket.recv()
         except ConnectionClosedError:
             self.close()
+            return ""
 
         msg = json.loads(msg_text)
+
         return msg["message"]
 
     def get_wait(self, topic):
@@ -64,7 +93,10 @@ class DSMQClientSideConnection:
 
     def put(self, topic, msg_body):
         msg_dict = {"action": "put", "topic": topic, "message": msg_body}
-        self.websocket.send(json.dumps(msg_dict))
+        try:
+            self.websocket.send(json.dumps(msg_dict))
+        except ConnectionClosedError:
+            return
 
     def shutdown_server(self):
         msg_dict = {"action": "shutdown", "topic": ""}

{dsmq-1.3.0 → dsmq-1.4.0}/src/dsmq/demo.py

@@ -1,7 +1,12 @@
 import multiprocessing as mp
-from dsmq.server import serve
-import dsmq.example_get_client
-import dsmq.example_put_client
+
+# spawn is the default method on macOS,
+# starting in Python 3.14 it will be the default in Linux too.
+mp.set_start_method("spawn")
+
+from dsmq.server import serve  # noqa: E402
+import dsmq.example_get_client  # noqa: E402
+import dsmq.example_put_client  # noqa: E402
 
 HOST = "127.0.0.1"
 PORT = 25252

dsmq-1.4.0/src/dsmq/server.py

@@ -0,0 +1,305 @@
+import json
+import os
+import sqlite3
+import sys
+from threading import Thread
+import time
+from websockets.sync.server import serve as ws_serve
+from websockets.exceptions import ConnectionClosedError, ConnectionClosedOK
+
+_default_host = "127.0.0.1"
+_default_port = 30008
+_max_queue_length = 10
+_shutdown_pause = 1.0  # seconds
+_time_between_cleanup = 3.0  # seconds
+_time_to_keep = 0.3  # seconds
+
+# Make this global so it's easy to share
+dsmq_server = None
+
+
+def serve(
+    host=_default_host,
+    port=_default_port,
+    name="mqdb",
+    verbose=False,
+):
+    """
+    For best results, start this running in its own process and walk away.
+    """
+    # May occasionally create files with this name.
+    # https://sqlite.org/inmemorydb.html
+    # "...parts of a temporary database might be flushed to disk if the
+    # database becomes large or if SQLite comes under memory pressure."
+    global _db_name
+    _db_name = f"file:{name}?mode=memory&cache=shared"
+
+    cleanup_temp_files()
+    sqlite_conn = sqlite3.connect(_db_name)
+    cursor = sqlite_conn.cursor()
+
+    # Tweak the connection to make it faster
+    # and keep long-term latency more predictable.
+    # These also make it more susceptible to corruption during shutdown,
+    # but since dsmq is meant to be ephemeral, that's not a concern.
+    # See https://www.sqlite.org/pragma.html
+    #
+    cursor.execute("PRAGMA journal_mode = OFF")
+    # cursor.execute("PRAGMA journal_mode = MEMORY")
+    # cursor.execute("PRAGMA synchronous = NORMAL")
+    cursor.execute("PRAGMA synchronous = OFF")
+    cursor.execute("PRAGMA secure_delete = OFF")
+    cursor.execute("PRAGMA temp_store = MEMORY")
+
+    cursor.execute("""
+CREATE TABLE IF NOT EXISTS messages (timestamp DOUBLE, topic TEXT, message TEXT)
+    """)
+
+    # Making this global in scope is a way to make it available
+    # to the shutdown operation. It's an awkward construction,
+    # and a method of last resort.
+    global dsmq_server
+
+    try:
+        with ws_serve(request_handler, host, port) as dsmq_server:
+            dsmq_server.serve_forever()
+
+    except OSError:
+        # Catch the case where the address is already in use
+        if verbose:
+            print()
+            print(f"Found a dsmq server already running on {host} on port {port}.")
+            print("    Closing it down.")
+
+            def shutdown_gracefully(server_to_shutdown):
+                server_to_shutdown.shutdown()
+
+            Thread(target=shutdown_gracefully, args=(dsmq_server,)).start()
+            time.sleep(_shutdown_pause)
+
+            with ws_serve(request_handler, host, port) as dsmq_server:
+                dsmq_server.serve_forever()
+
+    if verbose:
+        print()
+        print(f"Server started at {host} on port {port}.")
+        print("Waiting for clients...")
+
+    sqlite_conn.close()
+    time.sleep(_shutdown_pause)
+    cleanup_temp_files()
+
+
+def cleanup_temp_files():
+    # Under some condition
+    # (which I haven't yet been able to pin down)
+    # a file is generated with the db name.
+    # If it is not removed, it gets
+    # treated as a SQLite db on disk,
+    # which dramatically slows it down,
+    # especially the way it's used here for
+    # rapid-fire one-item reads and writes.
+    global _db_name
+    filenames = os.listdir()
+    for filename in filenames:
+        if filename[: len(_db_name)] == _db_name:
+            try:
+                os.remove(filename)
+            except FileNotFoundError:
+                pass
+
+
+def request_handler(websocket):
+    global _db_name
+    sqlite_conn = sqlite3.connect(_db_name)
+    cursor = sqlite_conn.cursor()
+
+    client_creation_time = time.time()
+    last_read_times = {}
+    time_of_last_purge = time.time()
+
+    try:
+        for msg_text in websocket:
+            msg = json.loads(msg_text)
+            topic = msg["topic"]
+            action = msg["action"]
+            timestamp = time.time()
+
+            if action == "put":
+                msg["timestamp"] = timestamp
+
+                try:
+                    cursor.execute(
+                        """
+                        INSERT INTO messages (timestamp, topic, message)
+                        VALUES (:timestamp, :topic, :message)
+                        """,
+                        (msg),
+                    )
+                    sqlite_conn.commit()
+                except sqlite3.OperationalError:
+                    pass
+
+            elif action == "get":
+                try:
+                    last_read_time = last_read_times[topic]
+                except KeyError:
+                    last_read_times[topic] = client_creation_time
+                    last_read_time = last_read_times[topic]
+                msg["last_read_time"] = last_read_time
+
+                try:
+                    cursor.execute(
+                        """
+                        SELECT message,
+                        timestamp
+                        FROM messages
+                        WHERE topic = :topic
+                        AND timestamp > :last_read_time
+                        ORDER BY timestamp ASC
+                        LIMIT 1
+                        """,
+                        msg,
+                    )
+                except sqlite3.OperationalError:
+                    pass
+
+                try:
+                    result = cursor.fetchall()[0]
+                    message = result[0]
+                    timestamp = result[1]
+                    last_read_times[topic] = timestamp
+                except IndexError:
+                    # Handle the case where no results are returned
+                    message = ""
+
+                websocket.send(json.dumps({"message": message}))
+
+            elif action == "get_latest":
+                try:
+                    last_read_time = last_read_times[topic]
+                except KeyError:
+                    last_read_times[topic] = client_creation_time
+                    last_read_time = last_read_times[topic]
+                msg["last_read_time"] = last_read_time
+
+                try:
+                    cursor.execute(
+                        """
+                        SELECT message,
+                        timestamp
+                        FROM messages
+                        WHERE topic = :topic
+                        AND timestamp > :last_read_time
+                        ORDER BY timestamp DESC
+                        LIMIT 1;
+                        """,
+                        msg,
+                    )
+                except sqlite3.OperationalError:
+                    pass
+
+                try:
+                    result = cursor.fetchall()[0]
+                    message = result[0]
+                    timestamp = result[1]
+                    last_read_times[topic] = timestamp
+                except IndexError:
+                    # Handle the case where no results are returned
+                    message = ""
+
+                websocket.send(json.dumps({"message": message}))
+
+            elif action == "shutdown":
+                # Run this from a separate thread to prevent deadlock
+                global dsmq_server
+
+                def shutdown_gracefully(server_to_shutdown):
+                    server_to_shutdown.shutdown()
+
+                Thread(target=shutdown_gracefully, args=(dsmq_server,)).start()
+                break
+            else:
+                raise RuntimeWarning(
+                    "dsmq client action must either be\n"
+                    + "'put', 'get', 'get_wait', 'get_latest', or 'shutdown'"
+                )
+
+            # Periodically clean out messages to keep individual queues at
+            # a manageable length and the overall mq small.
+            if time.time() - time_of_last_purge > _time_between_cleanup:
+                cutoff_time = time.time() - _time_to_keep
+                try:
+                    cursor.execute(
+                        """
+                        DELETE
+                        FROM messages
+                        WHERE topic = :topic
+                        AND timestamp < :cutoff_time
+                        """,
+                        {
+                            "cutoff_time": cutoff_time,
+                            "topic": topic,
+                        },
+                    )
+                    sqlite_conn.commit()
+                    time_of_last_purge = time.time()
+
+                    cursor.execute(
+                        """
+                        DELETE
+                        FROM messages
+                        WHERE topic = :topic
+                        AND timestamp IN (
+                          SELECT timestamp
+                          FROM (
+                              SELECT timestamp,
+                              RANK() OVER (ORDER BY timestamp DESC) recency_rank
+                              FROM messages
+                              WHERE topic = :topic
+                          )
+                          WHERE recency_rank >= :max_queue_length
+                        )
+                        """,
+                        {
+                            "max_queue_length": _max_queue_length,
+                            "topic": topic,
+                        },
+                    )
+                    sqlite_conn.commit()
+                    time_of_last_purge = time.time()
+
+                except sqlite3.OperationalError:
+                    # Database may be locked. Try again next time.
+                    pass
+
+    except (ConnectionClosedError, ConnectionClosedOK):
+        # Something happened on the other end and this handler
+        # is no longer needed.
+        pass
+
+    sqlite_conn.close()
+
+
+if __name__ == "__main__":
+    if len(sys.argv) == 3:
+        host = sys.argv[1]
+        port = int(sys.argv[2])
+        serve(host=host, port=port)
+    elif len(sys.argv) == 2:
+        host = sys.argv[1]
+        serve(host=host)
+    elif len(sys.argv) == 1:
+        serve()
+    else:
+        print(
+            """
+Try one of these:
+$ python3 server.py
+
+$ python3 server.py 127.0.0.1
+
+$ python3 server.py 127.0.0.1 25853
+
+"""
+        )