amps-python-client 5.3.4.1__zip → 5.3.4.2__zip

amps-python-client-5.3.4.1/docs/python/html/_sources/chapters/queues.rst.txt

@@ -1,282 +0,0 @@
-
-7. Using Queues
-===============
-
-AMPS message queues provide a high-performance way of distributing
-messages across a set of workers. The *AMPS User Guide* describes AMPS
-queues in detail, including the features of AMPS referred to in this
-chapter. This chapter does not describe AMPS queues in detail, but
-instead explains how to use the AMPS Python client with AMPS queues.
-
-To publish messages to an AMPS queue, publishers simply publish to any
-topic that is collected by the queue. There is no difference between
-publishing to a queue and publishing to any other topic, and a publisher
-does not need to be aware that the topic will be collected into a queue.
-
-Subscribers must be aware that they are subscribing to a queue, and
-acknowledge messages from the queue when the message is processed.
-
-Backlog and Smart Pipelining
-----------------------------
-
-AMPS queues are designed for high-volume applications that need minimal
-latency and overhead. One of the features that helps performance is the
-*subscription backlog* feature, which allows applications to receive
-multiple messages at a time. The subscription backlog sets the maximum
-number of unacknowledged messages that AMPS will provide to the
-subscription.
-
-When the subscription backlog is larger than ``1``, AMPS delivers
-additional messages to a subscriber before the subscriber has
-acknowledged the first message received. This technique allows
-subscribers to process messages as fast as possible, without ever having
-to wait for messages to be delivered. The technique of providing a
-consistent flow of messages to the application is called *smart
-pipelining*.
-
-Subscription Backlog
-^^^^^^^^^^^^^^^^^^^^
-
-The AMPS server determines the backlog for each subscription. An
-application can set the maximum backlog that it is willing to accept
-with the ``max_backlog`` option. Depending on the configuration of the
-queue (or queues) specified in the subscription, AMPS may assign a
-smaller backlog to the subscription. If no ``max_backlog`` option is
-specified, AMPS uses a ``max_backlog`` of ``1`` for that subscription.
-
-In general, applications that have a constant flow of messages perform
-better with a ``max_backlog`` setting higher than ``1``. The reason for
-this is that, with a backlog greater than ``1``, the application can
-always have a message waiting when the previous message is processed.
-Setting the optimum ``max_backlog`` is a matter of understanding the
-messaging pattern of your application and how quickly your application
-can process messages.
-
-To request a ``max_backlog`` for a subscription, you explicitly set the
-option on the subscribe command, as shown below:
-
-.. code:: python
-
-    command = Command("subscribe") \
-                .set_topic("my_queue" ) \
-                .set_options("max_backlog=10")
-
-Acknowledging Messages
-^^^^^^^^^^^^^^^^^^^^^^^
-
-For each message delivered on a subscription, AMPS counts the message
-against the subscription backlog until the message is explicitly
-acknowledged. In addition, when a queue specifies ``at-least-once``
-delivery, AMPS retains the message in the queue until the message
-expires or until the message has been explicitly acknowledged and
-removed from the queue. From the point of view of the AMPS server,
-acknowledgment is implemented as a ``sow_delete`` from the queue with
-the bookmarks of the messages to remove. The AMPS Python client provides
-several ways to make it easier for applications to create and send the
-appropriate ``sow_delete``.
-
-Automatic Acknowledgment
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The AMPS client allows you to specify that messages should be
-automatically acknowledged. When this mode is on, AMPS acknowledges the
-message automatically in the following cases:
-
--  **Asynchronous Message Processing Interface** - The message handler
-   returns without throwing an exception.
-
--  **Synchronous Message Processing Interface** - The application requests
-   the next message from the ``MessageStream``.
-
-AMPS batches acknowledgments created with this method, as described in
-the following section.
-
-To enable automatic acknowledgment batching, use the ``set_auto_ack()``
-method.
-
-.. code:: python
-
-    client.set_auto_ack(True)  # enable AutoAck
-
-Message Convenience Method
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The AMPS Python client provides a convenience method, ``ack()``, on
-delivered messages. When the application is finished with the message,
-the application simply calls ``ack()`` on the message. (This, in turn,
-provides the topic and bookmark to the ``ack()`` method of the client
-that received the message.)
-
-For messages that originated from a queue with ``at-least-once``
-semantics, this adds the bookmark from the message to the batch of
-messages to acknowledge. For other messages, this method has no effect.
-
-.. code:: python
-
-    message.ack() # Add this message to the next
-                  # acknowledgment batch.
-
-Acknowledgment Batching
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The AMPS Python client automatically batches acknowledgments when
-either of the convenience methods are used. Batching acknowledgments
-reduces the number of round-trips to AMPS, which reduces network traffic
-and improves overall performance. AMPS sends the batch of
-acknowledgments when the number of acknowledgments exceeds a specified
-size, or when the amount of time since the last batch was sent exceeds a
-specified timeout.
-
-You can set the number of messages to batch and the maximum amount of
-time between batches as shown below:
-
-.. code:: python
-
-    client.set_ack_batch_size(10)  # Send batch after 10 messages
-    client.set_ack_timeout(1000)  # ... or 1 second
-
-The AMPS Python client is aware of the subscription backlog for a
-subscription. When AMPS returns the acknowledgment for a subscription
-that contains queues, AMPS includes information on the subscription
-backlog for the subscription. If the requested batch size is larger than
-the subscription backlog, the AMPS Python client adjusts the requested
-batch size to match the subscription backlog.
-
-60East recommends tuning the batch size to improve application performance.
-A value of 1/3 of the smallest ``max_backlog`` value is a good initial
-starting point for testing. 60East does not recommend setting the batch size
-larger than 1/2 of the ``max_backlog`` value without testing to ensure
-that the application does not run out of messages to process while the
-acknowledgment is being sent to AMPS.
-
-Returning a Message to the Queue
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-A subscriber can also explicitly release a message back to the queue.
-AMPS returns the message to the queue and redelivers the message just
-as though the lease had expired. To do this, the subscriber sends a
-``sow_delete`` command with the bookmark of the message to release and
-the ``cancel`` option.
-
-When using automatic acknowledgments and the asynchronous API, AMPS
-will cancel a message if an exception is thrown from the message
-handler.
-
-To return a message to the queue, you can build a ``sow_delete``
-acknowledgment using the ``Command`` class, or pass an option to the
-``ack()`` method on the message.
-
-+-------------------+------------------------------------------------+
-| **Option**        | **Result**                                     |
-+===================+================================================+
-| ``cancel``        | Returns the message to the queue.              |
-+-------------------+------------------------------------------------+ 
-| ``expire``        | Immediately expires the message from the queue.|
-+-------------------+------------------------------------------------+ 
-
-For example, to return a message to a queue, call ``ack()`` on the message
-and pass the ``cancel`` option.
-
-
-.. code:: python
-
-    message.ack("cancel")
-
-**Example 7.1:** *Simple queue cancel*
-
-
-Manual Acknowledgment
-^^^^^^^^^^^^^^^^^^^^^^
-
-60East generally recommends that applications use an ``ack()`` method
-to acknowledge messages during normal processing. This approach works
-properly from within a message handler, provides batching support as
-described elsewhere in this chapter, and is generally both easier to
-code and more efficient.
-
-However, in some situations, you may need to manually acknowledge
-messages in the queue. This is most common when an application needs
-to operate on all messages with certain characteristics, rather than
-acknowledging individual messages. For example, an application
-that is doing updates to an order may want to cancel an order by
-both publishing a cancellation and immediately expiring all other
-messages in the queue for that order.  With manual
-acknowledgment, that application can use a filter to remove all
-previous updates for that order, then publish the cancellation.
-
-To manually acknowledge processed messages and remove the messages from
-the queue, applications use the ``sow_delete`` command. To remove
-specific messages from the queue, provide the bookmarks of those
-messages. To remove messages that match a given filter, provide
-the filter. Notice that AMPS only supports
-using a bookmark with ``sow_delete`` when removing messages from a
-queue, not when removing records from a SOW.
-
-For example, given a ``Message`` object to acknowledge and a client, the
-code below acknowledges the message.
-
-.. _#simple-acknowledge:
-
-.. code:: python
-
-    def acknowledgeSingle(client,message):
-        acknowledge = Command("sow_delete")
-        acknowledge.set_topic(message.get_topic())\
-                   .set_bookmark(message.get_bookmark())
-        client.execute_async(acknowledge,None)
-
-**Example 7.2:** *Simple queue acknowledgment*
-
-
-In the example above, the program creates a ``sow_delete`` command, 
-specifies the topic and the bookmark, and then sends the command to the server. 
-Since the program does not need or expect a response from AMPS, this function 
-provides ``None`` as the message handler.
-
-While this method works, creating and sending an acknowledgment for
-each individual message can be inefficient if your application is
-processing a large volume of messages. Rather than acknowledging each
-message individually, your application can build a comma-delimited list
-of bookmarks from the processed messages and acknowledge all of the
-messages at the same time. In this case, it's important to be sure that
-the number of messages you wait for is less than the maximum backlog --
-the number of messages your client can have unacknowledged at a given
-time. Notice that both automatic acknowledgment and the helper method
-on the ``Message`` object take the maximum backlog into account.
-
-When constructing a command to acknowledge queue messages,  AMPS allows an
-application to specify a filter rather than a set of bookmarks. AMPS interprets
-this as the client requesting acknowledgment of all messages that match the
-filter. (This  may include messages that the client has not received, subject
-to the ``Leasing`` model for the queue.)
-
-As a more typical example of manual acknowledgment, the code below expires
-all messages for a given ``id`` that have a status other than ``cancel``. An
-application might do this to halt processing of an order that it is about
-to cancel.
-
-.. _#filter-acknowledge:
-
-.. code-block:: python
-
-    def removePending(client, orderId):
-        acknowledge = Command("sow_delete")
-        acknowledge.set_topic(message.get_topic())\
-                   .set_filter("/id = '%s' and /status != 'cancel'" % orderId)\
-                   .set_options("expire")
-        client.execute_async(acknowledge,None)
-
-**Example 7.3:** *Queue acknowledgment with a filter*
-
-In the example above, the program specifies a topic and a filter to 
-use to find the messages that should be removed. In this case, the program 
-also provides the `expire` option to indicate that the messages have been 
-removed from the queue rather than successfully processed (of course, whether
-this is the correct behavior for a canceled order depends on the expected
-message flow for your application).
-
-Notice that, as described in :ref:`#client-threading`, this method
-of acknowledging a message should not be used from a message handler unless
-the ``sow_delete`` is sent from a different client than the client that
-called the message handler.  Instead, 60East recommends using the ``ack()``
-function from within a message handler.

amps-python-client-5.3.4.1/docs/python/html/_sources/chapters/sow.rst.txt

@@ -1,292 +0,0 @@
-6. State of the World
-=====================
-
-AMPS State of the World (SOW) allows you to automatically keep and query
-the latest information about a topic on the AMPS server, without
-building a separate database. Using SOW lets you build impressively
-high-performance applications that provide rich experiences to users.
-The AMPS Python client lets you query SOW topics and subscribe to
-changes with ease.
-
-Performing SOW Queries
-----------------------
-
-To begin, we will look at a simple example of issuing a SOW query.
-
-.. _#basic-sow-query:
-
-.. code:: python
-
-    for message in client.sow("orders", "/symbol='ROL'"):
-        if message.get_command() == Message.Command.GroupBegin:
-            print("--- Begin SOW Results ---")
-        if message.get_command() == Message.Command.SOW:
-            print(message.get_data())
-        if message.get_command() == Message.Command.GroupEnd:
-            print("--- End SOW Results ---")
-
-**Example 6.1:** *Basic SOW query*
-
-In the example above, the ``execute_sow_query()`` function invokes 
-``Client.sow()`` to initiate a SOW query on the ``orders`` topic, for 
-all entries that have a symbol of ``'ROL'``.
-
-As the query executes, messages containing the data of matching entries
-have a ``Command`` of value ``sow`` or ``SOW``, so as those arrive, we
-write them to the console.
-
-As with ``subscribe``, the ``sow`` command also provides an asynchronous
-mode, where you provide a message handler.
-
-.. _#asynchronous-sow-query:
-
-.. code:: python
-
-    def on_message_handler(message):
-        if message.get_command() == Message.Command.GroupBegin:
-            print("--- Begin SOW Results ---")
-        if message.get_command() == Message.Command.SOW:
-            print(message.get_data())
-        if message.get_command() == Message.Command.GroupEnd:
-            print("--- End SOW Results ---")
-
-    def execute_sow_query(client):
-        return client.execute_async(               \
-                    Command("sow")                 \
-                      .set_topic("orders")         \
-                      .set_filter("/symbol='ROL'"),\
-                   on_message_handler)
-
-**Example 6.2:** *Asynchronous SOW query*
-
-In the example above, the ``execute_sow_query()`` function invokes 
-``Client.execute_async()`` to initiate a SOW query on the ``orders`` 
-topic, for all entries that have a symbol of 'ROL'.
-
-As the query executes, the ``on_message_handler()`` function is invoked
-for each matching entry in the topic. Messages containing the data of
-matching entries have a ``Command`` of value ``sow``, so as those
-arrive, we write them to the console.
-
-SOW and Subscribe
------------------
-
-Imagine an application that displays real time information about the
-position and status of a fleet of delivery vans. When the application
-starts, it should display the current location of each of the vans along
-with their current status. As vans move around the city and post other
-status updates, the application should keep its display up to date. Vans
-upload information to the system by posting messages to a van
-``location`` topic, configured with a key of ``van_id`` on the AMPS
-server.
-
-In this application, it is important to not only stay up-to-date on the
-latest information about each van, but to ensure all of the active vans
-are displayed as soon as the application starts. Combining a SOW with a
-subscription to the topic is exactly what is needed, and that is
-accomplished by the ``Client.sow_and_subscribe()`` method. As with the
-other methods for receiving messages, the AMPS Python client provides a
-basic, synchronous form of ``sow_and subscribe`` that provides you with
-a ``MessageStream`` to iterate over, and an asynchronous form that
-requires a message handler.
-
-First, let's look at an example that uses the basic form of
-``sow_and_subscribe``:
-
-.. _#simple-sow-and-sub-example:
-
-.. code:: python
-
-    def report_van_position(client):
-        # sow_and_subscribe command to begin receiving information about all of the active
-        # delivery vans in the system. All of the vans in the system now are returned as
-        # Message objects whose get_command returns sow. New messages coming in are
-        # returned as Message objects whose get_command returns publish.
-        for message in client.execute(Command("sow_and_subscribe")\
-                .set_topic("van_location")\
-                .set_filter("/status = 'ACTIVE'")\
-                .set_batch_size(100)\
-                .set_options("oof")):
-                # Notice here that we specified the oof option to the command. Setting this option
-                # causes AMPS to send Out-of-Focus (OOF) messages for the topic.
-                # OOF messages are sent when an entry that was sent to us in the past no
-                # longer matches our query. This happens when an entry is removed from the SOW
-                # cache via a sow_delete operation, when the entry expires (as specified by the
-                # expiration time on the message or by the configuration of that topic on the AMPS
-                # server), or when the entry no longer matches the content filter specified. In
-                # our case, if a van’s status changes to something other than ACTIVE, it no longer
-                # matches the content filter, and becomes out of focus. When this occurs, a
-                # Message is sent with Command set to oof. We use OOF messages to remove vans from
-                # the display as they become inactive, expire, or are deleted.
-          if (message.get_command() == Message.Command.SOW or
-              message.get_command() == Message.Command.Publish):          
-               
-              # For each of these messages we call add_or_update_van(), that presumably adds the
-              # van to our application’s display. As vans send updates to the AMPS server, those
-              # are also received by the client because of the subscription placed by
-              # sow_and_subscribe(). Our application does not need to distinguish between
-              # updates and the original set of vans we found via the SOW query, so we use
-              # add_or_update_van() to display the new position of vans as well.
-              add_or_update_van(message)
-          elif message.get_command() == Message.Command.OOF:
-              remove_van(message)
-
-**Example 6.3:** *Using sow_and_subscribe*
-
-Now we will look at an example that uses the asynchronous form of
-``sow_and_subscribe``:
-
-.. code:: python
-
-    def update_van_position(message):
-        if (message.get_command() == Message.Command.SOW or 
-                message == Message.Command.Publish): 
-           add_or_update_van(message) 
-        elif message.get_command() == Message.Command.OOF:
-            remove_van(message)
-
-
-    def subscribe_to_van_location(client):
-        client.execute_async(
-               Command("sow_and_subscribe")          \
-                 .set_topic("van_location")          \
-                 .set_filter("/status = 'ACTIVE'")   \
-                 .set_batch_size(100)                \
-                 .set_options("oof"),                \
-           update_van_position)
-
-**Example 6.4:** *Using sow_and_subscribe*
-
-Notice that the two forms have the same result. However, one form
-performs processing on a background thread, and blocks the client from
-receiving messages while that processing happens, while the other form
-processes messages on the main thread and allows the background thread
-to continue to receive messages while processing occurs. In both cases,
-the calls to ``add_or_update_van`` and ``remove_van`` receive the same
-data
-
-.. include:: ./batch-size.inc
-Client-Side Conflation
-----------------------
-
-In many cases, applications that use SOW topics only need the current
-value of a message at the time the message is processed, rather than
-processing each change that led to the current value. On the server
-side, AMPS provides *conflated topics* to meet this need. Conflated
-topics are described in more detail in the *AMPS User Guide*, and
-require no special handling on the client side.
-
-In some cases, though, it's important to conflate messages on the client
-side. This can be particularly useful for applications that do expensive
-processing on each message, applications that are more efficient when
-processing batches of messages, or for situations where you cannot
-provide an appropriate conflation interval for the server to use.
-
-A ``MessageStream`` has the ability to conflate messages received for a
-subscription to a SOW topic, view, or conflated topic. When conflation
-is enabled, for each message received, the client checks to see whether
-it has already received an unprocessed message with the same ``SowKey``.
-If so, the client replaces the unprocessed message with the new message.
-The application never receives the message that has been replaced.
-
-To enable client-side conflation, you call ``conflate()`` on the
-``MessageStream``, and then use the ``MessageStream`` as usual:
-
-.. code:: python
-
-    # SOW query and subscribe
-    results = ampsClient.sow_and_subscribe("orders", "/symbol == 'ROL'")
-
-    # Turn on conflation
-    results.conflate()
-
-    # Process the results
-    for message in results:
-        # process message here
-
-When a ``MessageStream`` is used for a subscription that does not
-include ``SowKeys`` (such as a subscription to a topic that does not
-have a SOW), the ``MessageStream`` will allow you to turn on conflation,
-but no conflation will occur.
-
-When using client-side conflation with delta subscriptions, bear in mind
-that client-side conflation replaces the whole message, and does not
-attempt to merge deltas. This means that updates can be lost when
-messages are replaced. For some applications (for example, a ticker
-application that simply sends delta updates that replace the current
-price), this causes no problems. For other applications (for example,
-when several processors may be updating different fields of a message
-simultaneously), using conflation with deltas could result in lost data,
-and server-side conflation is a safer alternative.
-
-.. index:: sow_delete, sow_delete_by_keys, sow_delete_by_data, sow
-
-Managing SOW Contents
----------------------
-
-AMPS allows applications to manage the contents of the SOW by explicitly
-deleting messages that are no longer relevant. For example, if a
-particular delivery van is retired from service, the application can
-remove the record for the van by deleting the record for the van.
-
-The client provides the following methods for deleting records from the
-SOW:
-
--  ``sow_delete`` - accepts a topic and filter, and deletes all messages
-   that match the filter from the topic specified.
-
--  ``sow_delete_by_keys`` - accepts a set of SOW keys as a comma-delimited
-   string and deletes messages for those keys, regardless of the
-   contents of the messages. SOW keys are provided in the header of a
-   SOW message, and are the internal identifier AMPS uses for that SOW
-   message.
-
--  ``sow_delete_by_data`` - accepts a topic and message, and deletes the
-   SOW record that would be updated by that message.
-
-The most efficient way to remove messages from the SOW is to use
-``sow_delete_by_keys`` or ``sow_delete_by_data``, since those options
-allow AMPS to exactly target the message or messages to be removed.
-Many applications use ``sow_delete``, since this is the most
-flexible method for removing items from the SOW when the application
-does not have information on the exact messages to be removed.
-
-In either case, AMPS sends an OOF message to all subscribers who have
-received updates for the messages removed, as described in the previous
-section.
-
-The simple form of the ``sow_delete`` command returns a ``Message``.
-This ``Message`` is an acknowledgment that contains information on the
-delete command. For example, the following snippet simply prints
-informational text with the number of messages deleted:
-
-.. code:: python
-
-    msg = client.sow_delete(
-        "sow_topic",
-        "/id IN (42, 64, 37)"
-    )
-
-    print("Got an %s message containing %s: deleted %s SOW entries" % \
-          (msg.get_command(), msg.get_ack_type(), msg.get_matches()))
-
-The ``sow_delete`` command also provides an asynchronous version that
-requires a message handler. This message handler is designed to receive
-``sow_delete`` response messages from AMPS:
-
-.. code:: python
-
-    def delete_handler(m):
-        print("Got an %s message containing %s: deleted %s SOW entries" % \
-            (m.get_command(), m.get_ack_type(), m.get_matches()))
-
-            client.execute_async(
-                Command("sow_delete")     \
-                .set_topic("sow_topic") \
-                .set_filter("/id IN (42, 64, 37)"), \
-                delete_handler
-            )
-
-Acknowledging messages from a queue uses a form of the ``sow_delete``
-command that is only supported for queues. Acknowledgment is discussed
-in the chapter on queues.