priomesh-python 0.1.0__tar.gz → 0.2.0__tar.gz

{priomesh_python-0.1.0/src/priomesh_python.egg-info → priomesh_python-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: priomesh-python
-Version: 0.1.0
+Version: 0.2.0
 Summary: A pluggable priority and fair-queue scheduling library
 License: MIT License
         
@@ -61,14 +61,13 @@ cd priomesh-python
 python -m pip install -e .
 ```
 
-Once the project is published to PyPI, end users will be able to install it
-with:
+Install the latest published release from PyPI:
 
 ```bash
 python -m pip install priomesh-python
 ```
 
-Until then, users can install directly from the GitHub repository:
+To install directly from the GitHub repository instead:
 
 ```bash
 python -m pip install "git+https://github.com/PrioMesh/priomesh-python.git"
@@ -108,11 +107,53 @@ sub-queue within that level. Jobs within each priority/sub-queue pair are
 returned in FIFO order. Calling `get()` on an empty queue waits until a job is
 put; `empty()` checks whether the queue currently has any jobs.
 
+By default, the queue is unbounded. Pass a positive `maxsize` to limit the
+number of jobs waiting in the queue across all priorities and sub-queues.
+When it is full, `put()` waits until a `get()` removes a job and frees a slot.
+`maxsize=0` keeps the queue unbounded; negative values raise `ValueError`.
+
 Call `shutdown()` to stop accepting jobs. It performs a graceful shutdown:
 `put()` raises `QueueShutDown`, while `get()` continues returning jobs that
 were already queued. Once the queue is drained, waiting and future `get()`
-calls raise `QueueShutDown`, allowing consumers to exit. The example above
-shows a consumer draining the queue and stopping cleanly.
+calls raise `QueueShutDown`, allowing consumers to exit. A `put()` waiting
+because the queue is full is also woken by shutdown and raises
+`QueueShutDown`. The example above shows a consumer draining the queue and
+stopping cleanly.
+
+`put()` and `get()` accept an optional `timeout` in seconds. The default
+`None` waits without a deadline; otherwise the timeout must be a finite
+positive number. Zero, negative, `NaN`, and infinite timeout values raise
+`ValueError`. The timeout is implemented with
+`asyncio.wait_for()`, so an expired timeout raises `asyncio.TimeoutError`
+(`TimeoutError` on Python 3.11 and newer). Cancelling the task awaiting either
+operation also cancels the operation. If the operation is cancelled while
+waiting for a job or for queue capacity, it does not remove or add a job.
+
+```python
+import asyncio
+from priomesh import Job, Priority, PriorityFairQueue
+
+async def main() -> None:
+    queue = PriorityFairQueue()
+    try:
+        await queue.get(timeout=1.0)
+    except asyncio.TimeoutError:
+        print("No job arrived before the timeout")
+
+    bounded_queue = PriorityFairQueue(maxsize=1)
+    await bounded_queue.put(Job("queued", Priority.NORMAL))
+    try:
+        await bounded_queue.put(Job("waiting", Priority.NORMAL), timeout=1.0)
+    except asyncio.TimeoutError:
+        print("No capacity became available before the timeout")
+
+
+asyncio.run(main())
+```
+
+For manual cancellation, create a task for `put()` or `get()` and call its
+`cancel()` method. Timeout and cancellation handling use `asyncio`; they do
+not use a separate queue-specific cancellation API.
 
 ## Scheduling policies
 

{priomesh_python-0.1.0 → priomesh_python-0.2.0}/README.md

@@ -20,14 +20,13 @@ cd priomesh-python
 python -m pip install -e .
 ```
 
-Once the project is published to PyPI, end users will be able to install it
-with:
+Install the latest published release from PyPI:
 
 ```bash
 python -m pip install priomesh-python
 ```
 
-Until then, users can install directly from the GitHub repository:
+To install directly from the GitHub repository instead:
 
 ```bash
 python -m pip install "git+https://github.com/PrioMesh/priomesh-python.git"
@@ -67,11 +66,53 @@ sub-queue within that level. Jobs within each priority/sub-queue pair are
 returned in FIFO order. Calling `get()` on an empty queue waits until a job is
 put; `empty()` checks whether the queue currently has any jobs.
 
+By default, the queue is unbounded. Pass a positive `maxsize` to limit the
+number of jobs waiting in the queue across all priorities and sub-queues.
+When it is full, `put()` waits until a `get()` removes a job and frees a slot.
+`maxsize=0` keeps the queue unbounded; negative values raise `ValueError`.
+
 Call `shutdown()` to stop accepting jobs. It performs a graceful shutdown:
 `put()` raises `QueueShutDown`, while `get()` continues returning jobs that
 were already queued. Once the queue is drained, waiting and future `get()`
-calls raise `QueueShutDown`, allowing consumers to exit. The example above
-shows a consumer draining the queue and stopping cleanly.
+calls raise `QueueShutDown`, allowing consumers to exit. A `put()` waiting
+because the queue is full is also woken by shutdown and raises
+`QueueShutDown`. The example above shows a consumer draining the queue and
+stopping cleanly.
+
+`put()` and `get()` accept an optional `timeout` in seconds. The default
+`None` waits without a deadline; otherwise the timeout must be a finite
+positive number. Zero, negative, `NaN`, and infinite timeout values raise
+`ValueError`. The timeout is implemented with
+`asyncio.wait_for()`, so an expired timeout raises `asyncio.TimeoutError`
+(`TimeoutError` on Python 3.11 and newer). Cancelling the task awaiting either
+operation also cancels the operation. If the operation is cancelled while
+waiting for a job or for queue capacity, it does not remove or add a job.
+
+```python
+import asyncio
+from priomesh import Job, Priority, PriorityFairQueue
+
+async def main() -> None:
+    queue = PriorityFairQueue()
+    try:
+        await queue.get(timeout=1.0)
+    except asyncio.TimeoutError:
+        print("No job arrived before the timeout")
+
+    bounded_queue = PriorityFairQueue(maxsize=1)
+    await bounded_queue.put(Job("queued", Priority.NORMAL))
+    try:
+        await bounded_queue.put(Job("waiting", Priority.NORMAL), timeout=1.0)
+    except asyncio.TimeoutError:
+        print("No capacity became available before the timeout")
+
+
+asyncio.run(main())
+```
+
+For manual cancellation, create a task for `put()` or `get()` and call its
+`cancel()` method. Timeout and cancellation handling use `asyncio`; they do
+not use a separate queue-specific cancellation API.
 
 ## Scheduling policies
 

{priomesh_python-0.1.0 → priomesh_python-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "priomesh-python"
-version = "0.1.0"
+version = "0.2.0"
 description = "A pluggable priority and fair-queue scheduling library"
 readme = "README.md"
 requires-python = ">=3.10"

{priomesh_python-0.1.0 → priomesh_python-0.2.0}/src/priomesh/queue.py

@@ -1,4 +1,5 @@
 import asyncio
+import math
 from collections import deque
 from .models import Priority, SubQueue, Job
 
@@ -14,6 +15,11 @@ class QueueShutDown(Exception):
     """Raised when an operation is attempted on a shut-down queue."""
 
 
+def _validate_timeout(timeout: float | None) -> None:
+    if timeout is not None and (not math.isfinite(timeout) or timeout <= 0):
+        raise ValueError("timeout must be a finite positive number or None")
+
+
 class PriorityFairQueue:
     """Schedule jobs by priority and distribute service fairly among subqueues.
 
@@ -26,6 +32,7 @@ class PriorityFairQueue:
         self,
         priority_policy: PriorityPolicy | None = None,
         subqueue_policy: SubQueuePolicy | None = None,
+        maxsize: int = 0,
     ) -> None:
         self._priority_policy = priority_policy or RandomPriorityPolicy()
         self._subqueue_policy = subqueue_policy or AgingWeightedRandomSubQueuePolicy()
@@ -34,11 +41,28 @@ class PriorityFairQueue:
         self._condition = asyncio.Condition()
         self._shutdown = False
 
+        if maxsize < 0:
+            raise ValueError("maxsize must be non-negative")
+        self._maxsize = maxsize
+        self._size = 0
+
     def empty(self) -> bool:
         return not self._rotation
 
-    async def put(self, job: Job) -> None:
+    async def put(self, job: Job, timeout: float | None = None) -> None:
+        _validate_timeout(timeout)
+        if timeout is None:
+            await self._put(job)
+        else:
+            await asyncio.wait_for(self._put(job), timeout=timeout)
+
+    async def _put(self, job: Job) -> None:
         async with self._condition:
+            while self._maxsize > 0 and self._size >= self._maxsize:
+                if self._shutdown:
+                    raise QueueShutDown("cannot put into a shut-down queue")
+                await self._condition.wait()
+
             if self._shutdown:
                 raise QueueShutDown("cannot put into a shut-down queue")
             level_queues = self._queues.setdefault(job.priority, {})
@@ -53,9 +77,16 @@ class PriorityFairQueue:
                 rotation.append(SubQueue(name=job.subqueue, waiting_rounds=0))
             else:
                 subqueue.pending_jobs += 1
+            self._size += 1
             self._condition.notify()
 
-    async def get(self) -> Job:
+    async def get(self, timeout: float | None = None) -> Job:
+        _validate_timeout(timeout)
+        if timeout is None:
+            return await self._get()
+        return await asyncio.wait_for(self._get(), timeout=timeout)
+
+    async def _get(self) -> Job:
         async with self._condition:
             while True:
                 active_priorities = [p for p, r in self._rotation.items() if r]
@@ -67,6 +98,8 @@ class PriorityFairQueue:
                     r.remove(subqueue)
                     jobs = self._queues[priority][subqueue.name]
                     job = jobs.popleft()
+                    self._size -= 1
+                    self._condition.notify_all()
                     subqueue.pending_jobs -= 1
                     if jobs:
                         r.append(subqueue)
@@ -85,4 +118,10 @@ class PriorityFairQueue:
     async def shutdown(self) -> None:
         async with self._condition:
             self._shutdown = True
-            self._condition.notify_all()  # 唤醒所有等待的协程，促使其检查关闭态并退出
+            self._condition.notify_all()  # 唤醒所有等待的协程，促使其检查关闭态并退出
+
+    def qsize(self) -> int:
+        return self._size
+
+    def full(self) -> bool:
+        return self._maxsize > 0 and self._size >= self._maxsize

{priomesh_python-0.1.0 → priomesh_python-0.2.0/src/priomesh_python.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: priomesh-python
-Version: 0.1.0
+Version: 0.2.0
 Summary: A pluggable priority and fair-queue scheduling library
 License: MIT License
         
@@ -61,14 +61,13 @@ cd priomesh-python
 python -m pip install -e .
 ```
 
-Once the project is published to PyPI, end users will be able to install it
-with:
+Install the latest published release from PyPI:
 
 ```bash
 python -m pip install priomesh-python
 ```
 
-Until then, users can install directly from the GitHub repository:
+To install directly from the GitHub repository instead:
 
 ```bash
 python -m pip install "git+https://github.com/PrioMesh/priomesh-python.git"
@@ -108,11 +107,53 @@ sub-queue within that level. Jobs within each priority/sub-queue pair are
 returned in FIFO order. Calling `get()` on an empty queue waits until a job is
 put; `empty()` checks whether the queue currently has any jobs.
 
+By default, the queue is unbounded. Pass a positive `maxsize` to limit the
+number of jobs waiting in the queue across all priorities and sub-queues.
+When it is full, `put()` waits until a `get()` removes a job and frees a slot.
+`maxsize=0` keeps the queue unbounded; negative values raise `ValueError`.
+
 Call `shutdown()` to stop accepting jobs. It performs a graceful shutdown:
 `put()` raises `QueueShutDown`, while `get()` continues returning jobs that
 were already queued. Once the queue is drained, waiting and future `get()`
-calls raise `QueueShutDown`, allowing consumers to exit. The example above
-shows a consumer draining the queue and stopping cleanly.
+calls raise `QueueShutDown`, allowing consumers to exit. A `put()` waiting
+because the queue is full is also woken by shutdown and raises
+`QueueShutDown`. The example above shows a consumer draining the queue and
+stopping cleanly.
+
+`put()` and `get()` accept an optional `timeout` in seconds. The default
+`None` waits without a deadline; otherwise the timeout must be a finite
+positive number. Zero, negative, `NaN`, and infinite timeout values raise
+`ValueError`. The timeout is implemented with
+`asyncio.wait_for()`, so an expired timeout raises `asyncio.TimeoutError`
+(`TimeoutError` on Python 3.11 and newer). Cancelling the task awaiting either
+operation also cancels the operation. If the operation is cancelled while
+waiting for a job or for queue capacity, it does not remove or add a job.
+
+```python
+import asyncio
+from priomesh import Job, Priority, PriorityFairQueue
+
+async def main() -> None:
+    queue = PriorityFairQueue()
+    try:
+        await queue.get(timeout=1.0)
+    except asyncio.TimeoutError:
+        print("No job arrived before the timeout")
+
+    bounded_queue = PriorityFairQueue(maxsize=1)
+    await bounded_queue.put(Job("queued", Priority.NORMAL))
+    try:
+        await bounded_queue.put(Job("waiting", Priority.NORMAL), timeout=1.0)
+    except asyncio.TimeoutError:
+        print("No capacity became available before the timeout")
+
+
+asyncio.run(main())
+```
+
+For manual cancellation, create a task for `put()` or `get()` and call its
+`cancel()` method. Timeout and cancellation handling use `asyncio`; they do
+not use a separate queue-specific cancellation API.
 
 ## Scheduling policies
 

{priomesh_python-0.1.0 → priomesh_python-0.2.0}/tests/test_queue.py

@@ -1,4 +1,5 @@
 import asyncio
+import math
 import unittest
 
 from priomesh import Job, Priority, PriorityFairQueue, QueueShutDown
@@ -207,6 +208,168 @@ class PriorityFairQueueTests(unittest.IsolatedAsyncioTestCase):
             with self.assertRaises(QueueShutDown):
                 await asyncio.wait_for(t, timeout=1)
 
+    async def test_maxsize_neg(self) -> None:
+        with self.assertRaises(ValueError):
+            PriorityFairQueue(maxsize=-1)
+
+    async def test_maxsize_zero(self) -> None:
+        queue = PriorityFairQueue(maxsize=0)
+        for i in range(10):
+            await queue.put(Job(f"job-{i}", Priority.NORMAL))
+        self.assertEqual(queue.qsize(), 10)
+        self.assertFalse(queue.full())
+        while not queue.empty():
+            await queue.get()
+        self.assertEqual(queue.qsize(), 0)
+
+    async def test_qsize_and_full_track_jobs(self) -> None:
+        queue = PriorityFairQueue(maxsize=2)
+
+        self.assertEqual(queue.qsize(), 0)
+        self.assertFalse(queue.full())
+
+        await queue.put(Job("job-1", Priority.NORMAL, "tenant-a"))
+        self.assertEqual(queue.qsize(), 1)
+        self.assertFalse(queue.full())
+
+        await queue.put(Job("job-2", Priority.HIGH, "tenant-b"))
+        self.assertEqual(queue.qsize(), 2)
+        self.assertTrue(queue.full())
+
+        await queue.get()
+        self.assertEqual(queue.qsize(), 1)
+        self.assertFalse(queue.full())
+
+        await queue.get()
+        self.assertEqual(queue.qsize(), 0)
+        self.assertFalse(queue.full())
+
+    async def test_full_is_false_for_unbounded_queue(self) -> None:
+        queue = PriorityFairQueue()
+        await queue.put(Job("job-1", Priority.NORMAL))
+
+        self.assertEqual(queue.qsize(), 1)
+        self.assertFalse(queue.full())
+
+    async def test_maxsize_1(self) -> None:
+        queue = PriorityFairQueue(maxsize=1)
+        await queue.put(Job("job-1", Priority.NORMAL))
+
+        t = asyncio.create_task(
+            queue.put(Job("job-2", Priority.NORMAL))
+        )
+        await asyncio.sleep(0)
+        self.assertFalse(t.done())
+        await queue.get()
+
+        await asyncio.wait_for(t, timeout=1)
+        await queue.get()
+        self.assertTrue(t.done())
+        self.assertTrue(queue.empty())
+
+    async def test_shutdown_wakes_put_waiting_on_full_queue(self) -> None:
+        queue = PriorityFairQueue(maxsize=1)
+        await queue.put(Job("job-1", Priority.NORMAL))
+
+        put_task = asyncio.create_task(
+            queue.put(Job("job-2", Priority.NORMAL))
+        )
+        await asyncio.sleep(0)
+        self.assertFalse(put_task.done())
+
+        await queue.shutdown()
+
+        with self.assertRaises(QueueShutDown):
+            await asyncio.wait_for(put_task, timeout=1)
+
+        self.assertEqual((await queue.get()).name, "job-1")
+        with self.assertRaises(QueueShutDown):
+            await queue.get()
+
+    async def test_get_times_out_when_queue_is_empty(self) -> None:
+        queue = PriorityFairQueue()
+
+        with self.assertRaises(asyncio.TimeoutError):
+            await queue.get(timeout=0.01)
+
+        self.assertTrue(queue.empty())
+        self.assertEqual(queue.qsize(), 0)
+
+        job = Job("job-1", Priority.NORMAL)
+        await queue.put(job)
+        self.assertEqual(await queue.get(), job)
+
+    async def test_put_times_out_when_queue_is_full(self) -> None:
+        queue = PriorityFairQueue(maxsize=1)
+        first_job = Job("job-1", Priority.NORMAL)
+        await queue.put(first_job)
+
+        with self.assertRaises(asyncio.TimeoutError):
+            await queue.put(Job("job-2", Priority.NORMAL), timeout=0.01)
+
+        self.assertEqual(queue.qsize(), 1)
+        self.assertTrue(queue.full())
+        self.assertEqual(await queue.get(), first_job)
+
+    async def test_cancelled_get_does_not_remove_jobs(self) -> None:
+        queue = PriorityFairQueue()
+        get_task = asyncio.create_task(queue.get())
+        await asyncio.sleep(0)
+        self.assertFalse(get_task.done())
+
+        get_task.cancel()
+        with self.assertRaises(asyncio.CancelledError):
+            await get_task
+
+        job = Job("job-1", Priority.NORMAL)
+        await queue.put(job)
+        self.assertEqual(queue.qsize(), 1)
+        self.assertEqual(await queue.get(), job)
+
+    async def test_cancelled_put_does_not_enqueue_job(self) -> None:
+        queue = PriorityFairQueue(maxsize=1)
+        first_job = Job("job-1", Priority.NORMAL)
+        await queue.put(first_job)
+
+        put_task = asyncio.create_task(
+            queue.put(Job("job-2", Priority.NORMAL))
+        )
+        await asyncio.sleep(0)
+        self.assertFalse(put_task.done())
+
+        put_task.cancel()
+        with self.assertRaises(asyncio.CancelledError):
+            await put_task
+
+        self.assertEqual(queue.qsize(), 1)
+        self.assertTrue(queue.full())
+        self.assertEqual(await queue.get(), first_job)
+
+    async def test_put_rejects_invalid_timeouts(self) -> None:
+        queue = PriorityFairQueue()
+        job = Job("job-1", Priority.NORMAL)
+
+        for timeout in (0, -1, math.inf, -math.inf, math.nan):
+            with self.subTest(timeout=timeout):
+                with self.assertRaises(ValueError):
+                    await queue.put(job, timeout=timeout)
+
+        self.assertTrue(queue.empty())
+        self.assertEqual(queue.qsize(), 0)
+
+    async def test_get_rejects_invalid_timeouts_without_consuming_jobs(self) -> None:
+        queue = PriorityFairQueue()
+        job = Job("job-1", Priority.NORMAL)
+        await queue.put(job)
+
+        for timeout in (0, -1, math.inf, -math.inf, math.nan):
+            with self.subTest(timeout=timeout):
+                with self.assertRaises(ValueError):
+                    await queue.get(timeout=timeout)
+
+        self.assertEqual(queue.qsize(), 1)
+        self.assertEqual(await queue.get(), job)
+
 if __name__ == "__main__":
     """
     python -m pip install .