amps-python-client 5.3.4.1__zip

amps-python-client-5.3.4.1/docs/python/html/_sources/chapters/queues.rst.txt

@@ -0,0 +1,282 @@
+
+7. Using Queues
+===============
+
+AMPS message queues provide a high-performance way of distributing
+messages across a set of workers. The *AMPS User Guide* describes AMPS
+queues in detail, including the features of AMPS referred to in this
+chapter. This chapter does not describe AMPS queues in detail, but
+instead explains how to use the AMPS Python client with AMPS queues.
+
+To publish messages to an AMPS queue, publishers simply publish to any
+topic that is collected by the queue. There is no difference between
+publishing to a queue and publishing to any other topic, and a publisher
+does not need to be aware that the topic will be collected into a queue.
+
+Subscribers must be aware that they are subscribing to a queue, and
+acknowledge messages from the queue when the message is processed.
+
+Backlog and Smart Pipelining
+----------------------------
+
+AMPS queues are designed for high-volume applications that need minimal
+latency and overhead. One of the features that helps performance is the
+*subscription backlog* feature, which allows applications to receive
+multiple messages at a time. The subscription backlog sets the maximum
+number of unacknowledged messages that AMPS will provide to the
+subscription.
+
+When the subscription backlog is larger than ``1``, AMPS delivers
+additional messages to a subscriber before the subscriber has
+acknowledged the first message received. This technique allows
+subscribers to process messages as fast as possible, without ever having
+to wait for messages to be delivered. The technique of providing a
+consistent flow of messages to the application is called *smart
+pipelining*.
+
+Subscription Backlog
+^^^^^^^^^^^^^^^^^^^^
+
+The AMPS server determines the backlog for each subscription. An
+application can set the maximum backlog that it is willing to accept
+with the ``max_backlog`` option. Depending on the configuration of the
+queue (or queues) specified in the subscription, AMPS may assign a
+smaller backlog to the subscription. If no ``max_backlog`` option is
+specified, AMPS uses a ``max_backlog`` of ``1`` for that subscription.
+
+In general, applications that have a constant flow of messages perform
+better with a ``max_backlog`` setting higher than ``1``. The reason for
+this is that, with a backlog greater than ``1``, the application can
+always have a message waiting when the previous message is processed.
+Setting the optimum ``max_backlog`` is a matter of understanding the
+messaging pattern of your application and how quickly your application
+can process messages.
+
+To request a ``max_backlog`` for a subscription, you explicitly set the
+option on the subscribe command, as shown below:
+
+.. code:: python
+
+    command = Command("subscribe") \
+                .set_topic("my_queue" ) \
+                .set_options("max_backlog=10")
+
+Acknowledging Messages
+^^^^^^^^^^^^^^^^^^^^^^^
+
+For each message delivered on a subscription, AMPS counts the message
+against the subscription backlog until the message is explicitly
+acknowledged. In addition, when a queue specifies ``at-least-once``
+delivery, AMPS retains the message in the queue until the message
+expires or until the message has been explicitly acknowledged and
+removed from the queue. From the point of view of the AMPS server,
+acknowledgment is implemented as a ``sow_delete`` from the queue with
+the bookmarks of the messages to remove. The AMPS Python client provides
+several ways to make it easier for applications to create and send the
+appropriate ``sow_delete``.
+
+Automatic Acknowledgment
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The AMPS client allows you to specify that messages should be
+automatically acknowledged. When this mode is on, AMPS acknowledges the
+message automatically in the following cases:
+
+-  **Asynchronous Message Processing Interface** - The message handler
+   returns without throwing an exception.
+
+-  **Synchronous Message Processing Interface** - The application requests
+   the next message from the ``MessageStream``.
+
+AMPS batches acknowledgments created with this method, as described in
+the following section.
+
+To enable automatic acknowledgment batching, use the ``set_auto_ack()``
+method.
+
+.. code:: python
+
+    client.set_auto_ack(True)  # enable AutoAck
+
+Message Convenience Method
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The AMPS Python client provides a convenience method, ``ack()``, on
+delivered messages. When the application is finished with the message,
+the application simply calls ``ack()`` on the message. (This, in turn,
+provides the topic and bookmark to the ``ack()`` method of the client
+that received the message.)
+
+For messages that originated from a queue with ``at-least-once``
+semantics, this adds the bookmark from the message to the batch of
+messages to acknowledge. For other messages, this method has no effect.
+
+.. code:: python
+
+    message.ack() # Add this message to the next
+                  # acknowledgment batch.
+
+Acknowledgment Batching
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The AMPS Python client automatically batches acknowledgments when
+either of the convenience methods are used. Batching acknowledgments
+reduces the number of round-trips to AMPS, which reduces network traffic
+and improves overall performance. AMPS sends the batch of
+acknowledgments when the number of acknowledgments exceeds a specified
+size, or when the amount of time since the last batch was sent exceeds a
+specified timeout.
+
+You can set the number of messages to batch and the maximum amount of
+time between batches as shown below:
+
+.. code:: python
+
+    client.set_ack_batch_size(10)  # Send batch after 10 messages
+    client.set_ack_timeout(1000)  # ... or 1 second
+
+The AMPS Python client is aware of the subscription backlog for a
+subscription. When AMPS returns the acknowledgment for a subscription
+that contains queues, AMPS includes information on the subscription
+backlog for the subscription. If the requested batch size is larger than
+the subscription backlog, the AMPS Python client adjusts the requested
+batch size to match the subscription backlog.
+
+60East recommends tuning the batch size to improve application performance.
+A value of 1/3 of the smallest ``max_backlog`` value is a good initial
+starting point for testing. 60East does not recommend setting the batch size
+larger than 1/2 of the ``max_backlog`` value without testing to ensure
+that the application does not run out of messages to process while the
+acknowledgment is being sent to AMPS.
+
+Returning a Message to the Queue
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A subscriber can also explicitly release a message back to the queue.
+AMPS returns the message to the queue and redelivers the message just
+as though the lease had expired. To do this, the subscriber sends a
+``sow_delete`` command with the bookmark of the message to release and
+the ``cancel`` option.
+
+When using automatic acknowledgments and the asynchronous API, AMPS
+will cancel a message if an exception is thrown from the message
+handler.
+
+To return a message to the queue, you can build a ``sow_delete``
+acknowledgment using the ``Command`` class, or pass an option to the
+``ack()`` method on the message.
+
++-------------------+------------------------------------------------+
+| **Option**        | **Result**                                     |
++===================+================================================+
+| ``cancel``        | Returns the message to the queue.              |
++-------------------+------------------------------------------------+ 
+| ``expire``        | Immediately expires the message from the queue.|
++-------------------+------------------------------------------------+ 
+
+For example, to return a message to a queue, call ``ack()`` on the message
+and pass the ``cancel`` option.
+
+
+.. code:: python
+
+    message.ack("cancel")
+
+**Example 7.1:** *Simple queue cancel*
+
+
+Manual Acknowledgment
+^^^^^^^^^^^^^^^^^^^^^^
+
+60East generally recommends that applications use an ``ack()`` method
+to acknowledge messages during normal processing. This approach works
+properly from within a message handler, provides batching support as
+described elsewhere in this chapter, and is generally both easier to
+code and more efficient.
+
+However, in some situations, you may need to manually acknowledge
+messages in the queue. This is most common when an application needs
+to operate on all messages with certain characteristics, rather than
+acknowledging individual messages. For example, an application
+that is doing updates to an order may want to cancel an order by
+both publishing a cancellation and immediately expiring all other
+messages in the queue for that order.  With manual
+acknowledgment, that application can use a filter to remove all
+previous updates for that order, then publish the cancellation.
+
+To manually acknowledge processed messages and remove the messages from
+the queue, applications use the ``sow_delete`` command. To remove
+specific messages from the queue, provide the bookmarks of those
+messages. To remove messages that match a given filter, provide
+the filter. Notice that AMPS only supports
+using a bookmark with ``sow_delete`` when removing messages from a
+queue, not when removing records from a SOW.
+
+For example, given a ``Message`` object to acknowledge and a client, the
+code below acknowledges the message.
+
+.. _#simple-acknowledge:
+
+.. code:: python
+
+    def acknowledgeSingle(client,message):
+        acknowledge = Command("sow_delete")
+        acknowledge.set_topic(message.get_topic())\
+                   .set_bookmark(message.get_bookmark())
+        client.execute_async(acknowledge,None)
+
+**Example 7.2:** *Simple queue acknowledgment*
+
+
+In the example above, the program creates a ``sow_delete`` command, 
+specifies the topic and the bookmark, and then sends the command to the server. 
+Since the program does not need or expect a response from AMPS, this function 
+provides ``None`` as the message handler.
+
+While this method works, creating and sending an acknowledgment for
+each individual message can be inefficient if your application is
+processing a large volume of messages. Rather than acknowledging each
+message individually, your application can build a comma-delimited list
+of bookmarks from the processed messages and acknowledge all of the
+messages at the same time. In this case, it's important to be sure that
+the number of messages you wait for is less than the maximum backlog --
+the number of messages your client can have unacknowledged at a given
+time. Notice that both automatic acknowledgment and the helper method
+on the ``Message`` object take the maximum backlog into account.
+
+When constructing a command to acknowledge queue messages,  AMPS allows an
+application to specify a filter rather than a set of bookmarks. AMPS interprets
+this as the client requesting acknowledgment of all messages that match the
+filter. (This  may include messages that the client has not received, subject
+to the ``Leasing`` model for the queue.)
+
+As a more typical example of manual acknowledgment, the code below expires
+all messages for a given ``id`` that have a status other than ``cancel``. An
+application might do this to halt processing of an order that it is about
+to cancel.
+
+.. _#filter-acknowledge:
+
+.. code-block:: python
+
+    def removePending(client, orderId):
+        acknowledge = Command("sow_delete")
+        acknowledge.set_topic(message.get_topic())\
+                   .set_filter("/id = '%s' and /status != 'cancel'" % orderId)\
+                   .set_options("expire")
+        client.execute_async(acknowledge,None)
+
+**Example 7.3:** *Queue acknowledgment with a filter*
+
+In the example above, the program specifies a topic and a filter to 
+use to find the messages that should be removed. In this case, the program 
+also provides the `expire` option to indicate that the messages have been 
+removed from the queue rather than successfully processed (of course, whether
+this is the correct behavior for a canceled order depends on the expected
+message flow for your application).
+
+Notice that, as described in :ref:`#client-threading`, this method
+of acknowledging a message should not be used from a message handler unless
+the ``sow_delete`` is sent from a different client than the client that
+called the message handler.  Instead, 60East recommends using the ``ack()``
+function from within a message handler.

amps-python-client-5.3.4.1/docs/python/html/_sources/chapters/sow.rst.txt

@@ -0,0 +1,292 @@
+6. State of the World
+=====================
+
+AMPS State of the World (SOW) allows you to automatically keep and query
+the latest information about a topic on the AMPS server, without
+building a separate database. Using SOW lets you build impressively
+high-performance applications that provide rich experiences to users.
+The AMPS Python client lets you query SOW topics and subscribe to
+changes with ease.
+
+Performing SOW Queries
+----------------------
+
+To begin, we will look at a simple example of issuing a SOW query.
+
+.. _#basic-sow-query:
+
+.. code:: python
+
+    for message in client.sow("orders", "/symbol='ROL'"):
+        if message.get_command() == Message.Command.GroupBegin:
+            print("--- Begin SOW Results ---")
+        if message.get_command() == Message.Command.SOW:
+            print(message.get_data())
+        if message.get_command() == Message.Command.GroupEnd:
+            print("--- End SOW Results ---")
+
+**Example 6.1:** *Basic SOW query*
+
+In the example above, the ``execute_sow_query()`` function invokes 
+``Client.sow()`` to initiate a SOW query on the ``orders`` topic, for 
+all entries that have a symbol of ``'ROL'``.
+
+As the query executes, messages containing the data of matching entries
+have a ``Command`` of value ``sow`` or ``SOW``, so as those arrive, we
+write them to the console.
+
+As with ``subscribe``, the ``sow`` command also provides an asynchronous
+mode, where you provide a message handler.
+
+.. _#asynchronous-sow-query:
+
+.. code:: python
+
+    def on_message_handler(message):
+        if message.get_command() == Message.Command.GroupBegin:
+            print("--- Begin SOW Results ---")
+        if message.get_command() == Message.Command.SOW:
+            print(message.get_data())
+        if message.get_command() == Message.Command.GroupEnd:
+            print("--- End SOW Results ---")
+
+    def execute_sow_query(client):
+        return client.execute_async(               \
+                    Command("sow")                 \
+                      .set_topic("orders")         \
+                      .set_filter("/symbol='ROL'"),\
+                   on_message_handler)
+
+**Example 6.2:** *Asynchronous SOW query*
+
+In the example above, the ``execute_sow_query()`` function invokes 
+``Client.execute_async()`` to initiate a SOW query on the ``orders`` 
+topic, for all entries that have a symbol of 'ROL'.
+
+As the query executes, the ``on_message_handler()`` function is invoked
+for each matching entry in the topic. Messages containing the data of
+matching entries have a ``Command`` of value ``sow``, so as those
+arrive, we write them to the console.
+
+SOW and Subscribe
+-----------------
+
+Imagine an application that displays real time information about the
+position and status of a fleet of delivery vans. When the application
+starts, it should display the current location of each of the vans along
+with their current status. As vans move around the city and post other
+status updates, the application should keep its display up to date. Vans
+upload information to the system by posting messages to a van
+``location`` topic, configured with a key of ``van_id`` on the AMPS
+server.
+
+In this application, it is important to not only stay up-to-date on the
+latest information about each van, but to ensure all of the active vans
+are displayed as soon as the application starts. Combining a SOW with a
+subscription to the topic is exactly what is needed, and that is
+accomplished by the ``Client.sow_and_subscribe()`` method. As with the
+other methods for receiving messages, the AMPS Python client provides a
+basic, synchronous form of ``sow_and subscribe`` that provides you with
+a ``MessageStream`` to iterate over, and an asynchronous form that
+requires a message handler.
+
+First, let's look at an example that uses the basic form of
+``sow_and_subscribe``:
+
+.. _#simple-sow-and-sub-example:
+
+.. code:: python
+
+    def report_van_position(client):
+        # sow_and_subscribe command to begin receiving information about all of the active
+        # delivery vans in the system. All of the vans in the system now are returned as
+        # Message objects whose get_command returns sow. New messages coming in are
+        # returned as Message objects whose get_command returns publish.
+        for message in client.execute(Command("sow_and_subscribe")\
+                .set_topic("van_location")\
+                .set_filter("/status = 'ACTIVE'")\
+                .set_batch_size(100)\
+                .set_options("oof")):
+                # Notice here that we specified the oof option to the command. Setting this option
+                # causes AMPS to send Out-of-Focus (OOF) messages for the topic.
+                # OOF messages are sent when an entry that was sent to us in the past no
+                # longer matches our query. This happens when an entry is removed from the SOW
+                # cache via a sow_delete operation, when the entry expires (as specified by the
+                # expiration time on the message or by the configuration of that topic on the AMPS
+                # server), or when the entry no longer matches the content filter specified. In
+                # our case, if a van’s status changes to something other than ACTIVE, it no longer
+                # matches the content filter, and becomes out of focus. When this occurs, a
+                # Message is sent with Command set to oof. We use OOF messages to remove vans from
+                # the display as they become inactive, expire, or are deleted.
+          if (message.get_command() == Message.Command.SOW or
+              message.get_command() == Message.Command.Publish):          
+               
+              # For each of these messages we call add_or_update_van(), that presumably adds the
+              # van to our application’s display. As vans send updates to the AMPS server, those
+              # are also received by the client because of the subscription placed by
+              # sow_and_subscribe(). Our application does not need to distinguish between
+              # updates and the original set of vans we found via the SOW query, so we use
+              # add_or_update_van() to display the new position of vans as well.
+              add_or_update_van(message)
+          elif message.get_command() == Message.Command.OOF:
+              remove_van(message)
+
+**Example 6.3:** *Using sow_and_subscribe*
+
+Now we will look at an example that uses the asynchronous form of
+``sow_and_subscribe``:
+
+.. code:: python
+
+    def update_van_position(message):
+        if (message.get_command() == Message.Command.SOW or 
+                message == Message.Command.Publish): 
+           add_or_update_van(message) 
+        elif message.get_command() == Message.Command.OOF:
+            remove_van(message)
+
+
+    def subscribe_to_van_location(client):
+        client.execute_async(
+               Command("sow_and_subscribe")          \
+                 .set_topic("van_location")          \
+                 .set_filter("/status = 'ACTIVE'")   \
+                 .set_batch_size(100)                \
+                 .set_options("oof"),                \
+           update_van_position)
+
+**Example 6.4:** *Using sow_and_subscribe*
+
+Notice that the two forms have the same result. However, one form
+performs processing on a background thread, and blocks the client from
+receiving messages while that processing happens, while the other form
+processes messages on the main thread and allows the background thread
+to continue to receive messages while processing occurs. In both cases,
+the calls to ``add_or_update_van`` and ``remove_van`` receive the same
+data
+
+.. include:: ./batch-size.inc
+Client-Side Conflation
+----------------------
+
+In many cases, applications that use SOW topics only need the current
+value of a message at the time the message is processed, rather than
+processing each change that led to the current value. On the server
+side, AMPS provides *conflated topics* to meet this need. Conflated
+topics are described in more detail in the *AMPS User Guide*, and
+require no special handling on the client side.
+
+In some cases, though, it's important to conflate messages on the client
+side. This can be particularly useful for applications that do expensive
+processing on each message, applications that are more efficient when
+processing batches of messages, or for situations where you cannot
+provide an appropriate conflation interval for the server to use.
+
+A ``MessageStream`` has the ability to conflate messages received for a
+subscription to a SOW topic, view, or conflated topic. When conflation
+is enabled, for each message received, the client checks to see whether
+it has already received an unprocessed message with the same ``SowKey``.
+If so, the client replaces the unprocessed message with the new message.
+The application never receives the message that has been replaced.
+
+To enable client-side conflation, you call ``conflate()`` on the
+``MessageStream``, and then use the ``MessageStream`` as usual:
+
+.. code:: python
+
+    # SOW query and subscribe
+    results = ampsClient.sow_and_subscribe("orders", "/symbol == 'ROL'")
+
+    # Turn on conflation
+    results.conflate()
+
+    # Process the results
+    for message in results:
+        # process message here
+
+When a ``MessageStream`` is used for a subscription that does not
+include ``SowKeys`` (such as a subscription to a topic that does not
+have a SOW), the ``MessageStream`` will allow you to turn on conflation,
+but no conflation will occur.
+
+When using client-side conflation with delta subscriptions, bear in mind
+that client-side conflation replaces the whole message, and does not
+attempt to merge deltas. This means that updates can be lost when
+messages are replaced. For some applications (for example, a ticker
+application that simply sends delta updates that replace the current
+price), this causes no problems. For other applications (for example,
+when several processors may be updating different fields of a message
+simultaneously), using conflation with deltas could result in lost data,
+and server-side conflation is a safer alternative.
+
+.. index:: sow_delete, sow_delete_by_keys, sow_delete_by_data, sow
+
+Managing SOW Contents
+---------------------
+
+AMPS allows applications to manage the contents of the SOW by explicitly
+deleting messages that are no longer relevant. For example, if a
+particular delivery van is retired from service, the application can
+remove the record for the van by deleting the record for the van.
+
+The client provides the following methods for deleting records from the
+SOW:
+
+-  ``sow_delete`` - accepts a topic and filter, and deletes all messages
+   that match the filter from the topic specified.
+
+-  ``sow_delete_by_keys`` - accepts a set of SOW keys as a comma-delimited
+   string and deletes messages for those keys, regardless of the
+   contents of the messages. SOW keys are provided in the header of a
+   SOW message, and are the internal identifier AMPS uses for that SOW
+   message.
+
+-  ``sow_delete_by_data`` - accepts a topic and message, and deletes the
+   SOW record that would be updated by that message.
+
+The most efficient way to remove messages from the SOW is to use
+``sow_delete_by_keys`` or ``sow_delete_by_data``, since those options
+allow AMPS to exactly target the message or messages to be removed.
+Many applications use ``sow_delete``, since this is the most
+flexible method for removing items from the SOW when the application
+does not have information on the exact messages to be removed.
+
+In either case, AMPS sends an OOF message to all subscribers who have
+received updates for the messages removed, as described in the previous
+section.
+
+The simple form of the ``sow_delete`` command returns a ``Message``.
+This ``Message`` is an acknowledgment that contains information on the
+delete command. For example, the following snippet simply prints
+informational text with the number of messages deleted:
+
+.. code:: python
+
+    msg = client.sow_delete(
+        "sow_topic",
+        "/id IN (42, 64, 37)"
+    )
+
+    print("Got an %s message containing %s: deleted %s SOW entries" % \
+          (msg.get_command(), msg.get_ack_type(), msg.get_matches()))
+
+The ``sow_delete`` command also provides an asynchronous version that
+requires a message handler. This message handler is designed to receive
+``sow_delete`` response messages from AMPS:
+
+.. code:: python
+
+    def delete_handler(m):
+        print("Got an %s message containing %s: deleted %s SOW entries" % \
+            (m.get_command(), m.get_ack_type(), m.get_matches()))
+
+            client.execute_async(
+                Command("sow_delete")     \
+                .set_topic("sow_topic") \
+                .set_filter("/id IN (42, 64, 37)"), \
+                delete_handler
+            )
+
+Acknowledging messages from a queue uses a form of the ``sow_delete``
+command that is only supported for queues. Acknowledgment is discussed
+in the chapter on queues.