twainmq 0.2.3__tar.gz

twainmq-0.2.3/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Mike Crowe
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

twainmq-0.2.3/PKG-INFO

@@ -0,0 +1,65 @@
+Metadata-Version: 2.4
+Name: twainmq
+Version: 0.2.3
+Summary: A file-based message queue. Kafka‑inspired. Twain‑simple.
+Author-email: Mike Crowe <you@example.com>
+License: MIT
+Project-URL: homepage, https://github.com/BlueRaven314/TwainMQ
+Project-URL: repository, https://github.com/BlueRaven314/TwainMQ
+Project-URL: issues, https://github.com/BlueRaven314/TwainMQ/issues
+Keywords: messaging,message-queue,queue,pubsub,publish-subscribe,log-based,event-stream,event-log,file-based,lightweight,local,stream-processing,consumer-groups,partitioning,offsets,kafka-like
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: dataclasses-jsonschema>=2.16.0
+Provides-Extra: docs
+Requires-Dist: mkdocs; extra == "docs"
+Requires-Dist: mkdocs-material; extra == "docs"
+Requires-Dist: mkdocstrings[python]; extra == "docs"
+Requires-Dist: mkdocs-gen-files; extra == "docs"
+Requires-Dist: mkdocs-literate-nav; extra == "docs"
+Requires-Dist: mkdocs-section-index; extra == "docs"
+Dynamic: license-file
+
+<p align="center">
+  <img src="docs/TwainLogoTransparent.png" alt="TwainMQ Logo" width="200">
+</p>
+
+<h1 align="center">TwainMQ</h1>
+
+**TwainMQ** is a lightweight, file‑based message log inspired by Kafka’s design principles but without the operational overhead of running a broker or cluster. It provides durable, append‑only topics, partitioned message streams, consumer groups with load balancing, and offset managemen, all without using a central service or broker.
+
+TwainMQ is designed for:
+
+- local development workflows  
+- embedded systems  
+- single‑machine (or small local network) data pipelines  
+- teaching and experimentation with log‑based messaging  
+- applications that need Kafka‑like semantics without a server  
+
+It requires **no services**, **no central authority**, just a directory where the files will live.  All coordination is done via the log files.
+
+---
+
+## Features
+
+- **No running services or brokers**, just an API and distributed self-organisation
+- **File‑based topics** stored as append‑only logs
+- **Partitions** for parallelism and ordering guarantees
+- **Consumer groups** with automatic load balancing
+- **Offset tracking and commits**
+- **Typed messages** (supports dataclasses)
+- **Simple API** for producers and consumers
+
+## Documentation
+
+Full documentation is available at:
+
+**https://blueraven314.github.io/TwainMQ/**
+
+## Why “TwainMQ”?  
+
+Kafka messaging is powerful, but it’s also famously complex and sometimes hard going, rather like reading Franz Kafka. Mark Twain, on the other hand, is known for clarity and accessibility. TwainMQ takes inspiration from Kafka’s log‑based design, but aims to be simple, easy to use, and friendly to work with.

twainmq-0.2.3/README.md

@@ -0,0 +1,39 @@
+<p align="center">
+  <img src="docs/TwainLogoTransparent.png" alt="TwainMQ Logo" width="200">
+</p>
+
+<h1 align="center">TwainMQ</h1>
+
+**TwainMQ** is a lightweight, file‑based message log inspired by Kafka’s design principles but without the operational overhead of running a broker or cluster. It provides durable, append‑only topics, partitioned message streams, consumer groups with load balancing, and offset managemen, all without using a central service or broker.
+
+TwainMQ is designed for:
+
+- local development workflows  
+- embedded systems  
+- single‑machine (or small local network) data pipelines  
+- teaching and experimentation with log‑based messaging  
+- applications that need Kafka‑like semantics without a server  
+
+It requires **no services**, **no central authority**, just a directory where the files will live.  All coordination is done via the log files.
+
+---
+
+## Features
+
+- **No running services or brokers**, just an API and distributed self-organisation
+- **File‑based topics** stored as append‑only logs
+- **Partitions** for parallelism and ordering guarantees
+- **Consumer groups** with automatic load balancing
+- **Offset tracking and commits**
+- **Typed messages** (supports dataclasses)
+- **Simple API** for producers and consumers
+
+## Documentation
+
+Full documentation is available at:
+
+**https://blueraven314.github.io/TwainMQ/**
+
+## Why “TwainMQ”?  
+
+Kafka messaging is powerful, but it’s also famously complex and sometimes hard going, rather like reading Franz Kafka. Mark Twain, on the other hand, is known for clarity and accessibility. TwainMQ takes inspiration from Kafka’s log‑based design, but aims to be simple, easy to use, and friendly to work with.

twainmq-0.2.3/pyproject.toml

@@ -0,0 +1,63 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "twainmq"
+version = "0.2.3"
+description = "A file-based message queue. Kafka‑inspired. Twain‑simple."
+
+authors = [
+    { name = "Mike Crowe", email = "you@example.com" }
+]
+
+readme = "README.md"
+license = { text = "MIT" }
+requires-python = ">=3.8"
+
+keywords = [
+    "messaging",
+    "message-queue",
+    "queue",
+    "pubsub",
+    "publish-subscribe",
+    "log-based",
+    "event-stream",
+    "event-log",
+    "file-based",
+    "lightweight",
+    "local",
+    "stream-processing",
+    "consumer-groups",
+    "partitioning",
+    "offsets",
+    "kafka-like"
+]
+
+dependencies = [
+    "dataclasses-jsonschema>=2.16.0",
+]
+
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+]
+
+[project.urls]
+homepage = "https://github.com/BlueRaven314/TwainMQ"
+repository = "https://github.com/BlueRaven314/TwainMQ"
+issues = "https://github.com/BlueRaven314/TwainMQ/issues"
+
+[project.optional-dependencies]
+docs = [
+    "mkdocs",
+    "mkdocs-material",
+    "mkdocstrings[python]",
+    "mkdocs-gen-files",
+    "mkdocs-literate-nav",
+    "mkdocs-section-index",
+]
+
+[tool.setuptools.packages.find]
+where = ["src"]

twainmq-0.2.3/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

twainmq-0.2.3/src/twainmq/__init__.py

@@ -0,0 +1,24 @@
+"""
+
+TwainMQ is a broker free messaging system.  Unlike similar tools like Apache Kafka and RabbitMQ, TwainMQ does not have any running services or brokers to coordinate messages, being run entirely from files.  Think of it like SQLite is to traditional database systems.
+
+To set up a TwainMQ message queue you simply need to specify a directory for it to use, either locally or on a network share.
+
+Messages are limited in size to ensure that they can be written quickly.  The size limit is currently set at slightly less than 4kb.
+
+Messages can be either plain UTF-8 or raw bytes.
+
+Messages are organised by key, partition and topic.
+
+## Keys
+
+u8
+u16
+u32
+u64
+str
+
+"""
+
+from .core import Twain, TwainMQConsumer, TwainMQProducer
+from .utils import multi_poll

twainmq-0.2.3/src/twainmq/agent_base.py

@@ -0,0 +1,44 @@
+from abc import ABC
+from datetime import datetime, timezone
+import json
+
+from twainmq.encoding import find_key_char_width
+from twainmq.errors import ConfigNotFoundError
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from .core import Twain
+
+class TwainMQBase(ABC):
+    def __init__(self, twain: "Twain", topic: str):
+        self._topic = topic
+        self._twain = twain
+        config_path = self._twain._config_path(topic)
+        if not config_path.is_file():
+            raise ConfigNotFoundError(f"No config for {topic}: cannot find file {config_path}")
+        with config_path.open("r") as f:
+            config = json.load(f)
+        self._key_width = config["key_width"]
+        self._message_types = config["message_types"]
+        self._message_types_rev = {i: msg_type for msg_type, i in self._message_types.items()}
+        self._n_partitions = int(config["partitions"])
+        self._key_chars = find_key_char_width(self._key_width)
+    
+    @property
+    def topic(self):
+        return self._topic
+    
+    @property
+    def _topic_dir(self):
+        return self._twain.root_dir / self._topic
+    
+    @property
+    def key_width(self):
+        """The width of the key in bytes.  0 indicates a string key.
+        """
+        return self._key_width
+
+    @property
+    def chunk_str_now(self):
+        return f"{datetime.now(timezone.utc):%Y%m%d}"

twainmq-0.2.3/src/twainmq/atomic_append.py

@@ -0,0 +1,100 @@
+import os
+from pathlib import Path
+import errno
+import ctypes
+from ctypes import wintypes
+from pathlib import Path
+import os
+import errno
+from pathlib import Path
+
+# Win32 constants
+FILE_APPEND_DATA        = 0x0004
+FILE_SHARE_READ         = 0x00000001
+FILE_SHARE_WRITE        = 0x00000002
+OPEN_ALWAYS             = 4
+FILE_ATTRIBUTE_NORMAL   = 0x80
+INVALID_HANDLE_VALUE    = wintypes.HANDLE(-1).value
+
+if os.name == "nt":
+    kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
+
+def atomic_append_win(path, data: bytes) -> None:
+    """Atomic appending to a file under Windows.
+    
+    This makes use of the windows FILE_APPEND_DATA mode which doesn't check the file length first or use a cursor
+    it simply dumps the data on the end of the file.
+    """
+    # Accept Path or str
+    if isinstance(path, Path):
+        path = str(path)
+
+    # 1. Open with FILE_APPEND_DATA only (critical)
+    handle = kernel32.CreateFileW(
+        wintypes.LPCWSTR(path),
+        FILE_APPEND_DATA,                         # <-- atomic append mode
+        FILE_SHARE_READ | FILE_SHARE_WRITE,
+        None,
+        OPEN_ALWAYS,
+        FILE_ATTRIBUTE_NORMAL,
+        None,
+    )
+
+    if handle == INVALID_HANDLE_VALUE:
+        raise OSError(ctypes.get_last_error(), "CreateFileW failed")
+
+    try:
+        # 2. WriteFile (one syscall, atomic append)
+        written = wintypes.DWORD(0)
+        ok = kernel32.WriteFile(
+            handle,
+            data,
+            len(data),
+            ctypes.byref(written),
+            None,
+        )
+
+        if not ok or written.value != len(data):
+            raise OSError(ctypes.get_last_error(), "WriteFile failed")
+
+    finally:
+        kernel32.CloseHandle(handle)
+        
+def atomic_append_posix(path, data: bytes, mode: int = 0o644) -> None:
+    """Atomic append function for POSIX systems.
+
+    This should work by using O_APPEND but this is not properly tested yet!
+
+    Short write might in theory happen, but shouldn't, when we can test this better
+    we will know if this is a problem and come up with a solution.
+    """
+    # Accept Path or str
+    if isinstance(path, Path):
+        path = os.fspath(path)
+
+    # Open with O_APPEND so the kernel moves the file offset atomically
+    flags = os.O_WRONLY | os.O_APPEND | os.O_CREAT
+
+    # Add close-on-exec if available (good hygiene)
+    flags |= getattr(os, "O_CLOEXEC", 0)
+
+    fd = os.open(path, flags, mode)
+    try:
+        # Exactly one write() syscall
+        n = os.write(fd, data)
+
+        # POSIX guarantees full writes to regular files unless an error occurs.
+        # If we ever see a short write, treat it as fatal.
+        if n != len(data):
+            raise OSError(errno.EIO, f"short write: wrote {n} of {len(data)} bytes")
+
+    finally:
+        os.close(fd)
+
+def atomic_append(path, data: bytes, mode: int = 0o644) -> None:
+    """Atomic writing used by producers to allow multiple producers to write to the same 
+    log file at the same time without locks"""
+    if os.name == "nt":
+        atomic_append_win(path, data)
+    else:
+        atomic_append_posix(path, data)