PyPI - amps-python-client - Versions diffs - 5.3.4.1__zip - Mend

amps-python-client 5.3.4.1__zip

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (550) hide show

amps-python-client-5.3.4.1/docs/python/html/chapters/queues.html ADDED Viewed

@@ -0,0 +1,352 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>7. Using Queues &#8212; Python Developer Guide develop
+ documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     'develop',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+    <link rel="next" title="8. Delta Publish and Subscribe" href="deltas.html" />
+    <link rel="prev" title="6. State of the World" href="sow.html" />
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+  </head>
+  <body role="document">
+  <div class="document">
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper">
+            <p class="logo"><a href="../index.html">
+              <img class="logo" src="../_static/flag_logo.png" alt="Logo"/>
+            </a></p>
+  <h3><a href="../index.html">Table Of Contents</a></h3>
+  <ul>
+<li><a class="reference internal" href="#">7. Using Queues</a><ul>
+<li><a class="reference internal" href="#backlog-and-smart-pipelining">Backlog and Smart Pipelining</a><ul>
+<li><a class="reference internal" href="#subscription-backlog">Subscription Backlog</a></li>
+<li><a class="reference internal" href="#acknowledging-messages">Acknowledging Messages</a></li>
+<li><a class="reference internal" href="#automatic-acknowledgment">Automatic Acknowledgment</a></li>
+<li><a class="reference internal" href="#message-convenience-method">Message Convenience Method</a></li>
+<li><a class="reference internal" href="#acknowledgment-batching">Acknowledgment Batching</a></li>
+<li><a class="reference internal" href="#returning-a-message-to-the-queue">Returning a Message to the Queue</a></li>
+<li><a class="reference internal" href="#manual-acknowledgment">Manual Acknowledgment</a></li>
+</ul>
+</li>
+</ul>
+</li>
+</ul>
+<div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+      <li>Previous: <a href="sow.html" title="previous chapter">6. State of the World</a></li>
+      <li>Next: <a href="deltas.html" title="next chapter">8. Delta Publish and Subscribe</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+  <div class="section" id="using-queues">
+<h1>7. Using Queues<a class="headerlink" href="#using-queues" title="Permalink to this headline">¶</a></h1>
+<p>AMPS message queues provide a high-performance way of distributing
+messages across a set of workers. The <em>AMPS User Guide</em> describes AMPS
+queues in detail, including the features of AMPS referred to in this
+chapter. This chapter does not describe AMPS queues in detail, but
+instead explains how to use the AMPS Python client with AMPS queues.</p>
+<p>To publish messages to an AMPS queue, publishers simply publish to any
+topic that is collected by the queue. There is no difference between
+publishing to a queue and publishing to any other topic, and a publisher
+does not need to be aware that the topic will be collected into a queue.</p>
+<p>Subscribers must be aware that they are subscribing to a queue, and
+acknowledge messages from the queue when the message is processed.</p>
+<div class="section" id="backlog-and-smart-pipelining">
+<h2>Backlog and Smart Pipelining<a class="headerlink" href="#backlog-and-smart-pipelining" title="Permalink to this headline">¶</a></h2>
+<p>AMPS queues are designed for high-volume applications that need minimal
+latency and overhead. One of the features that helps performance is the
+<em>subscription backlog</em> feature, which allows applications to receive
+multiple messages at a time. The subscription backlog sets the maximum
+number of unacknowledged messages that AMPS will provide to the
+subscription.</p>
+<p>When the subscription backlog is larger than <code class="docutils literal"><span class="pre">1</span></code>, AMPS delivers
+additional messages to a subscriber before the subscriber has
+acknowledged the first message received. This technique allows
+subscribers to process messages as fast as possible, without ever having
+to wait for messages to be delivered. The technique of providing a
+consistent flow of messages to the application is called <em>smart
+pipelining</em>.</p>
+<div class="section" id="subscription-backlog">
+<h3>Subscription Backlog<a class="headerlink" href="#subscription-backlog" title="Permalink to this headline">¶</a></h3>
+<p>The AMPS server determines the backlog for each subscription. An
+application can set the maximum backlog that it is willing to accept
+with the <code class="docutils literal"><span class="pre">max_backlog</span></code> option. Depending on the configuration of the
+queue (or queues) specified in the subscription, AMPS may assign a
+smaller backlog to the subscription. If no <code class="docutils literal"><span class="pre">max_backlog</span></code> option is
+specified, AMPS uses a <code class="docutils literal"><span class="pre">max_backlog</span></code> of <code class="docutils literal"><span class="pre">1</span></code> for that subscription.</p>
+<p>In general, applications that have a constant flow of messages perform
+better with a <code class="docutils literal"><span class="pre">max_backlog</span></code> setting higher than <code class="docutils literal"><span class="pre">1</span></code>. The reason for
+this is that, with a backlog greater than <code class="docutils literal"><span class="pre">1</span></code>, the application can
+always have a message waiting when the previous message is processed.
+Setting the optimum <code class="docutils literal"><span class="pre">max_backlog</span></code> is a matter of understanding the
+messaging pattern of your application and how quickly your application
+can process messages.</p>
+<p>To request a <code class="docutils literal"><span class="pre">max_backlog</span></code> for a subscription, you explicitly set the
+option on the subscribe command, as shown below:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">command</span> <span class="o">=</span> <span class="n">Command</span><span class="p">(</span><span class="s2">&quot;subscribe&quot;</span><span class="p">)</span> \
+            <span class="o">.</span><span class="n">set_topic</span><span class="p">(</span><span class="s2">&quot;my_queue&quot;</span> <span class="p">)</span> \
+            <span class="o">.</span><span class="n">set_options</span><span class="p">(</span><span class="s2">&quot;max_backlog=10&quot;</span><span class="p">)</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="acknowledging-messages">
+<h3>Acknowledging Messages<a class="headerlink" href="#acknowledging-messages" title="Permalink to this headline">¶</a></h3>
+<p>For each message delivered on a subscription, AMPS counts the message
+against the subscription backlog until the message is explicitly
+acknowledged. In addition, when a queue specifies <code class="docutils literal"><span class="pre">at-least-once</span></code>
+delivery, AMPS retains the message in the queue until the message
+expires or until the message has been explicitly acknowledged and
+removed from the queue. From the point of view of the AMPS server,
+acknowledgment is implemented as a <code class="docutils literal"><span class="pre">sow_delete</span></code> from the queue with
+the bookmarks of the messages to remove. The AMPS Python client provides
+several ways to make it easier for applications to create and send the
+appropriate <code class="docutils literal"><span class="pre">sow_delete</span></code>.</p>
+</div>
+<div class="section" id="automatic-acknowledgment">
+<h3>Automatic Acknowledgment<a class="headerlink" href="#automatic-acknowledgment" title="Permalink to this headline">¶</a></h3>
+<p>The AMPS client allows you to specify that messages should be
+automatically acknowledged. When this mode is on, AMPS acknowledges the
+message automatically in the following cases:</p>
+<ul class="simple">
+<li><strong>Asynchronous Message Processing Interface</strong> - The message handler
+returns without throwing an exception.</li>
+<li><strong>Synchronous Message Processing Interface</strong> - The application requests
+the next message from the <code class="docutils literal"><span class="pre">MessageStream</span></code>.</li>
+</ul>
+<p>AMPS batches acknowledgments created with this method, as described in
+the following section.</p>
+<p>To enable automatic acknowledgment batching, use the <code class="docutils literal"><span class="pre">set_auto_ack()</span></code>
+method.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">client</span><span class="o">.</span><span class="n">set_auto_ack</span><span class="p">(</span><span class="kc">True</span><span class="p">)</span>  <span class="c1"># enable AutoAck</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="message-convenience-method">
+<h3>Message Convenience Method<a class="headerlink" href="#message-convenience-method" title="Permalink to this headline">¶</a></h3>
+<p>The AMPS Python client provides a convenience method, <code class="docutils literal"><span class="pre">ack()</span></code>, on
+delivered messages. When the application is finished with the message,
+the application simply calls <code class="docutils literal"><span class="pre">ack()</span></code> on the message. (This, in turn,
+provides the topic and bookmark to the <code class="docutils literal"><span class="pre">ack()</span></code> method of the client
+that received the message.)</p>
+<p>For messages that originated from a queue with <code class="docutils literal"><span class="pre">at-least-once</span></code>
+semantics, this adds the bookmark from the message to the batch of
+messages to acknowledge. For other messages, this method has no effect.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">message</span><span class="o">.</span><span class="n">ack</span><span class="p">()</span> <span class="c1"># Add this message to the next</span>
+              <span class="c1"># acknowledgment batch.</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="acknowledgment-batching">
+<h3>Acknowledgment Batching<a class="headerlink" href="#acknowledgment-batching" title="Permalink to this headline">¶</a></h3>
+<p>The AMPS Python client automatically batches acknowledgments when
+either of the convenience methods are used. Batching acknowledgments
+reduces the number of round-trips to AMPS, which reduces network traffic
+and improves overall performance. AMPS sends the batch of
+acknowledgments when the number of acknowledgments exceeds a specified
+size, or when the amount of time since the last batch was sent exceeds a
+specified timeout.</p>
+<p>You can set the number of messages to batch and the maximum amount of
+time between batches as shown below:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">client</span><span class="o">.</span><span class="n">set_ack_batch_size</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>  <span class="c1"># Send batch after 10 messages</span>
+<span class="n">client</span><span class="o">.</span><span class="n">set_ack_timeout</span><span class="p">(</span><span class="mi">1000</span><span class="p">)</span>  <span class="c1"># ... or 1 second</span>
+</pre></div>
+</div>
+<p>The AMPS Python client is aware of the subscription backlog for a
+subscription. When AMPS returns the acknowledgment for a subscription
+that contains queues, AMPS includes information on the subscription
+backlog for the subscription. If the requested batch size is larger than
+the subscription backlog, the AMPS Python client adjusts the requested
+batch size to match the subscription backlog.</p>
+<p>60East recommends tuning the batch size to improve application performance.
+A value of 1/3 of the smallest <code class="docutils literal"><span class="pre">max_backlog</span></code> value is a good initial
+starting point for testing. 60East does not recommend setting the batch size
+larger than 1/2 of the <code class="docutils literal"><span class="pre">max_backlog</span></code> value without testing to ensure
+that the application does not run out of messages to process while the
+acknowledgment is being sent to AMPS.</p>
+</div>
+<div class="section" id="returning-a-message-to-the-queue">
+<h3>Returning a Message to the Queue<a class="headerlink" href="#returning-a-message-to-the-queue" title="Permalink to this headline">¶</a></h3>
+<p>A subscriber can also explicitly release a message back to the queue.
+AMPS returns the message to the queue and redelivers the message just
+as though the lease had expired. To do this, the subscriber sends a
+<code class="docutils literal"><span class="pre">sow_delete</span></code> command with the bookmark of the message to release and
+the <code class="docutils literal"><span class="pre">cancel</span></code> option.</p>
+<p>When using automatic acknowledgments and the asynchronous API, AMPS
+will cancel a message if an exception is thrown from the message
+handler.</p>
+<p>To return a message to the queue, you can build a <code class="docutils literal"><span class="pre">sow_delete</span></code>
+acknowledgment using the <code class="docutils literal"><span class="pre">Command</span></code> class, or pass an option to the
+<code class="docutils literal"><span class="pre">ack()</span></code> method on the message.</p>
+<table border="1" class="docutils">
+<colgroup>
+<col width="28%" />
+<col width="72%" />
+</colgroup>
+<thead valign="bottom">
+<tr class="row-odd"><th class="head"><strong>Option</strong></th>
+<th class="head"><strong>Result</strong></th>
+</tr>
+</thead>
+<tbody valign="top">
+<tr class="row-even"><td><code class="docutils literal"><span class="pre">cancel</span></code></td>
+<td>Returns the message to the queue.</td>
+</tr>
+<tr class="row-odd"><td><code class="docutils literal"><span class="pre">expire</span></code></td>
+<td>Immediately expires the message from the queue.</td>
+</tr>
+</tbody>
+</table>
+<p>For example, to return a message to a queue, call <code class="docutils literal"><span class="pre">ack()</span></code> on the message
+and pass the <code class="docutils literal"><span class="pre">cancel</span></code> option.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">message</span><span class="o">.</span><span class="n">ack</span><span class="p">(</span><span class="s2">&quot;cancel&quot;</span><span class="p">)</span>
+</pre></div>
+</div>
+<p><strong>Example 7.1:</strong> <em>Simple queue cancel</em></p>
+</div>
+<div class="section" id="manual-acknowledgment">
+<h3>Manual Acknowledgment<a class="headerlink" href="#manual-acknowledgment" title="Permalink to this headline">¶</a></h3>
+<p>60East generally recommends that applications use an <code class="docutils literal"><span class="pre">ack()</span></code> method
+to acknowledge messages during normal processing. This approach works
+properly from within a message handler, provides batching support as
+described elsewhere in this chapter, and is generally both easier to
+code and more efficient.</p>
+<p>However, in some situations, you may need to manually acknowledge
+messages in the queue. This is most common when an application needs
+to operate on all messages with certain characteristics, rather than
+acknowledging individual messages. For example, an application
+that is doing updates to an order may want to cancel an order by
+both publishing a cancellation and immediately expiring all other
+messages in the queue for that order.  With manual
+acknowledgment, that application can use a filter to remove all
+previous updates for that order, then publish the cancellation.</p>
+<p>To manually acknowledge processed messages and remove the messages from
+the queue, applications use the <code class="docutils literal"><span class="pre">sow_delete</span></code> command. To remove
+specific messages from the queue, provide the bookmarks of those
+messages. To remove messages that match a given filter, provide
+the filter. Notice that AMPS only supports
+using a bookmark with <code class="docutils literal"><span class="pre">sow_delete</span></code> when removing messages from a
+queue, not when removing records from a SOW.</p>
+<p>For example, given a <code class="docutils literal"><span class="pre">Message</span></code> object to acknowledge and a client, the
+code below acknowledges the message.</p>
+<div class="code python highlight-default" id="simple-acknowledge"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">acknowledgeSingle</span><span class="p">(</span><span class="n">client</span><span class="p">,</span><span class="n">message</span><span class="p">):</span>
+    <span class="n">acknowledge</span> <span class="o">=</span> <span class="n">Command</span><span class="p">(</span><span class="s2">&quot;sow_delete&quot;</span><span class="p">)</span>
+    <span class="n">acknowledge</span><span class="o">.</span><span class="n">set_topic</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">get_topic</span><span class="p">())</span>\
+               <span class="o">.</span><span class="n">set_bookmark</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">get_bookmark</span><span class="p">())</span>
+    <span class="n">client</span><span class="o">.</span><span class="n">execute_async</span><span class="p">(</span><span class="n">acknowledge</span><span class="p">,</span><span class="kc">None</span><span class="p">)</span>
+</pre></div>
+</div>
+<p><strong>Example 7.2:</strong> <em>Simple queue acknowledgment</em></p>
+<p>In the example above, the program creates a <code class="docutils literal"><span class="pre">sow_delete</span></code> command,
+specifies the topic and the bookmark, and then sends the command to the server.
+Since the program does not need or expect a response from AMPS, this function
+provides <code class="docutils literal"><span class="pre">None</span></code> as the message handler.</p>
+<p>While this method works, creating and sending an acknowledgment for
+each individual message can be inefficient if your application is
+processing a large volume of messages. Rather than acknowledging each
+message individually, your application can build a comma-delimited list
+of bookmarks from the processed messages and acknowledge all of the
+messages at the same time. In this case, it&#8217;s important to be sure that
+the number of messages you wait for is less than the maximum backlog &#8211;
+the number of messages your client can have unacknowledged at a given
+time. Notice that both automatic acknowledgment and the helper method
+on the <code class="docutils literal"><span class="pre">Message</span></code> object take the maximum backlog into account.</p>
+<p>When constructing a command to acknowledge queue messages,  AMPS allows an
+application to specify a filter rather than a set of bookmarks. AMPS interprets
+this as the client requesting acknowledgment of all messages that match the
+filter. (This  may include messages that the client has not received, subject
+to the <code class="docutils literal"><span class="pre">Leasing</span></code> model for the queue.)</p>
+<p>As a more typical example of manual acknowledgment, the code below expires
+all messages for a given <code class="docutils literal"><span class="pre">id</span></code> that have a status other than <code class="docutils literal"><span class="pre">cancel</span></code>. An
+application might do this to halt processing of an order that it is about
+to cancel.</p>
+<div class="highlight-python" id="filter-acknowledge"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">removePending</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">orderId</span><span class="p">):</span>
+    <span class="n">acknowledge</span> <span class="o">=</span> <span class="n">Command</span><span class="p">(</span><span class="s2">&quot;sow_delete&quot;</span><span class="p">)</span>
+    <span class="n">acknowledge</span><span class="o">.</span><span class="n">set_topic</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">get_topic</span><span class="p">())</span>\
+               <span class="o">.</span><span class="n">set_filter</span><span class="p">(</span><span class="s2">&quot;/id = &#39;</span><span class="si">%s</span><span class="s2">&#39; and /status != &#39;cancel&#39;&quot;</span> <span class="o">%</span> <span class="n">orderId</span><span class="p">)</span>\
+               <span class="o">.</span><span class="n">set_options</span><span class="p">(</span><span class="s2">&quot;expire&quot;</span><span class="p">)</span>
+    <span class="n">client</span><span class="o">.</span><span class="n">execute_async</span><span class="p">(</span><span class="n">acknowledge</span><span class="p">,</span><span class="kc">None</span><span class="p">)</span>
+</pre></div>
+</div>
+<p><strong>Example 7.3:</strong> <em>Queue acknowledgment with a filter</em></p>
+<p>In the example above, the program specifies a topic and a filter to
+use to find the messages that should be removed. In this case, the program
+also provides the <cite>expire</cite> option to indicate that the messages have been
+removed from the queue rather than successfully processed (of course, whether
+this is the correct behavior for a canceled order depends on the expected
+message flow for your application).</p>
+<p>Notice that, as described in <a class="reference internal" href="subscriptions.html#client-threading"><span class="std std-ref">Understanding Threading and Message Handlers</span></a>, this method
+of acknowledging a message should not be used from a message handler unless
+the <code class="docutils literal"><span class="pre">sow_delete</span></code> is sent from a different client than the client that
+called the message handler.  Instead, 60East recommends using the <code class="docutils literal"><span class="pre">ack()</span></code>
+function from within a message handler.</p>
+</div>
+</div>
+</div>
+          </div>
+        </div>
+      </div>
+    <div class="clearer"></div>
+  </div>
+    <div class="footer">
+      &copy;2022, 60East Technologies, Inc..
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.5.3</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+    </div>
+  </body>
+</html>