amps-python-client 5.3.4.1__zip → 5.3.4.2__zip

amps-python-client-5.3.4.1/docs/user-guide/html/_sources/chapters/queues.rst.txt

@@ -1,1351 +0,0 @@
-.. |tip| image:: ../../common/svg/tip.svg
-
-.. _#ug-queues:
-
-.. _queues:
-
-Message Queues
-==============
-
-.. index:: queuing, message queues,
-
-AMPS includes high performance queuing built on
-the AMPS messaging engine and transaction log. AMPS message queues
-combine elements of classic message queuing with the advanced messaging
-features of AMPS, including content filtering, aggregation and
-projection, historical replay, and so on. This chapter presents an
-overview of queues.
-
-AMPS message queues help you easily solve some common messaging
-problems:
-
--  Ensuring that a message is only processed once.
-
--  Distributing tasks across workers in a fair manner.
-
--  Ensuring that a message is delivered to and processed by a worker.
-
--  Ensuring that when a worker fails to process a message, that message
-   is redelivered.
-
--  Setting limits for retries in processing a message, and taking an
-   action when those limits are exceeded (such as moving the message to
-   a dead-letter queue).
-
-While it's possible to create applications with these properties by
-using the other features of AMPS, message queues provide these functions
-built into the AMPS server. In addition, message queues allow you to:
-
--  Replicate messages between AMPS instances while preserving delivery
-   guarantees.
-
--  Create views and aggregates based on the current contents of a queue.
-
--  Filter messages into and out of a queue.
-
--  Provide a single published message to multiple queues.
-
--  Aggregate multiple topics into a single queue.
-
-
-.. tip::
-
-   Use message queues when it is important to ensure that a message
-   is processed once, by a single consumer. When it is important to
-   distribute messages to a number of consumers, use the AMPS
-   publish/subscribe delivery model rather than a message queue.
-
-The following diagram illustrates a simple usage of a queue to
-distribute work across three consumers:
-
-.. image:: ../../common/chapters/queues/svg/queue_usage_overview.svg
-   :width: 80.0%
-
-This diagram shows a simple use of AMPS queues to distribute work. In
-the diagram, the transaction log is configured to record a topic named
-``Work``. AMPS is also configured with a queue named ``WorkToDo``,
-which is based on the underlying topic ``Work``. The publisher publishes
-three messages to the topic ``Work``, and AMPS includes those messages
-in the ``WorkToDo`` queue. Each message is delivered to one of the three
-subscribers to the ``WorkToDo`` queue. Unlike pub/sub messaging, each
-subscriber only receives one message, and each message is delivered to
-only one subscriber.
-
-Notice that, even though AMPS provides queue semantics over the
-``WorkToDo`` topic, the messages are recorded in the transaction log
-once, in the ``Work`` topic. Other subscribers could subscribe to the
-``Work`` topic to receive the full stream of messages, or do a bookmark
-replay over the ``Work`` topic to recreate the message flow or audit the
-messages published to that topic.
-
-Also notice that, while the ``Work`` topic (the underlying topic) must be
-recorded in the transaction log, there is no need to define the ``Work`` topic
-in the SOW section of the configuration file. Queues use the transaction log to
-determine which messages to enqueue, and when a message is acknowledged, the
-acknowledgment is also recorded in the transaction log.
-
-Getting Started with AMPS Queues
----------------------------------
-
-.. _AMPS Configuration Reference: ../../../configuration/html/index.html
-
-To add a simple queue to AMPS, add the following options to your
-configuration file.
-
-First, create a transaction log that will record the messages for the
-queue and the state of the queue, as described in 
-:ref:`#ug-txlog`. 
-You add the transaction log
-entry if your AMPS configuration does not already have one. Otherwise,
-you can simply add a ``Topic`` statement or modify an existing ``Topic``
-statement to record the messages. The sample below captures any JSON
-messages published to the ``Work`` topic, and also tracks the state of the queue itself:
-
-.. code-block:: xml
-
-    <AMPSConfig>
-        ...
-
-        <TransactionLog>
-             <JournalDirectory>./journals</JournalDirectory>
-             <Topic>
-                <Name>Work</Name>
-                <MessageType>json</MessageType>
-             </Topic>
-             <Topic>
-                <Name>WorkToDo</Name>
-                <MessageType>json</MessageType>
-             </Topic>
-        </TransactionLog>
-
-       ...
-    </AMPSConfig>
-
-Next, declare the queue topic itself. Queues are defined in the SOW
-element of the ``AMPSConfig`` file, as shown below:
-
-.. code-block:: xml
-
-    <AMPSConfig>
-        ...
-
-        <SOW>
-            <Queue>
-                <Name>WorkToDo</Name>
-                <MessageType>json</MessageType>
-                <Semantics>at-most-once</Semantics>
-                <UnderlyingTopic>Work</UnderlyingTopic>
-            </Queue>
-        </SOW>
-
-        ...
-    </AMPSConfig>
-
-These simple configuration changes create an AMPS message queue. Notice
-that the ``Topic`` for the queue in this case is ``WorkToDo``, which
-includes every message published to the underlying topic ``Work``. You
-could also use a regular expression to include messages to more than one
-topic, or leave out the ``UnderlyingTopic`` to include only messages
-published to the topic with the same name as the queue.
-
-This simple queue provides each message that arrives for the queue to at
-most one subscriber. After AMPS delivers the message to one subscriber,
-AMPS removes the message from the queue without waiting for the
-subscriber to acknowledge the message.
-
-While it's easy to create a simple queue, AMPS offers a rich queuing
-model that is designed to meet a wide variety of queuing needs. The
-options are described in the following sections and the `AMPS Configuration Reference`_.
-
-Queue Replication Types
-^^^^^^^^^^^^^^^^^^^^^^^
-
-AMPS supports three different replication types for a queue. The
-configuration tag used to define a queue specifies the replication
-type for the queue.
-
-.. list-table:: *Queue replication types*
-     :header-rows: 1
-
-     * - **Configuration Tag**
-
-       - **Replication Type**
-
-       - **Description**
-
-     * - ``Queue``
-         
-
-       - Distributed queue
-         
-
-       - Fully distributed queue.
-         
-
-     * - ``LocalQueue``
-         
-
-       - Local queue
-         
-
-       - Queue exists on a single
-         instance, delivery guarantees
-         only apply to that instance.
-
-         Cannot be replicated.
-
-     * - ``GroupLocalQueue``
-         
-
-       - Queue replicated
-         to specific instances,
-         most often in the
-         same replication
-         ``Group``
-         
-
-       - Distributed queue on a
-         specific set of instances.
-         
-         In version 5.3.4 and
-         higher, the instances that
-         contain the queue must all
-         be in the same ``Group``
-         *or* define the same
-         ``GroupLocalQueueDomain``
-         tag.
-         
-         In earlier versions of
-         AMPS, every queue with
-         the same name in the
-         replication fabric is
-         treated as an instance of
-         the same queue and must
-         have the same definition
-         and the same
-         ``InitialOwner``.
-
-
-By default, AMPS queues are *distributed queues*. That is, if the queue
-topic or the underlying topics are replicated, AMPS provides the queue
-delivery guarantees (that is, ``at-most-once`` or ``at-least-once`` delivery)
-as though all of the instances were delivering
-messages from a single queue.
-
-
-AMPS provides local queues (where each instance has a separate,
-independent queue) when a queue is defined with the ``LocalQueue`` tag. In
-this case, AMPS does not allow replication of the queue topic, and all
-management of the queue is local to the individual instance.
-
-AMPS also supports distributed queues that are restricted to a set of
-instances within a mesh of replicated instances.  These queues are defined
-with the ``GroupLocalQueue`` tag, to indicate that the queue is
-restricted to a subset of the servers involved in replication. In this
-case, only the servers that define the ``GroupLocalQueue`` participate in
-message delivery for the queue. In addition, the group must choose the
-instance that will initially own messages for the queue, as described in
-the section on distributed queues and :ref:`queue message ownership <#ug-queue-message-ownership>`.
-In version 5.3.4 and later, the identity of a ``GroupLocalQueue`` is defined
-by the ``Name`` of the queue combined with the ``GroupLocalQueueDomain``,
-if one is set, or the ``Group`` of the AMPS instance if the definition
-does not include a ``GroupLocalQueueDomain`` tag.
-In previous versions of AMPS, the identity of the queue is determined only
-by the ``Name`` of the group local queue, regardless of the instance that
-defines the queue.
-
-.. tip::
-
-   When defining a ``GroupLocalQueue``, the instances that participate
-   in the queue do not need to be in the same replication group:
-   any instance in a replicated mesh that defines the queue can
-   host an instance of the queue, regardless of whether the instance
-   is in the same replication group or not. However, if the instances
-   that host the queue are in different groups, the queue definitions
-   should set the same ``GroupLocalQueueDomain`` to be considered
-   to be the same queue.
-
-Each instance of AMPS manages its own subscriptions and backlog,
-regardless of the replication model. Delivery guarantees are
-provided by managing the ownership of each message in the queue,
-as described in the :ref:`#ug-replicated-queues` section. Only messages
-owned by the current instance will be delivered to subscribers,
-and only the subscriptions on the current instance are considered
-for delivery fairness and limits such as ``MaxBacklog`` (the overall
-limit for leases from the queue).
-
-Understanding AMPS Queuing
----------------------------
-
-AMPS message queues take advantage of the full historical and
-transactional power of the AMPS engine. Each queue is implemented as a
-view over an underlying topic or set of topics. Each of the underlying
-topics must be recorded in a transaction log. Publishers publish to the
-underlying topic, and the messages are recorded in the transaction log.
-Consumers simply subscribe to the queue. AMPS tracks which messages have
-been delivered to subscribers and which messages have been processed by
-subscribers. AMPS delivers the next available message to the next
-subscriber.
-
-Unlike traditional queues, which require consumers to poll for messages,
-AMPS queues use a subscription model. In this model, each queue consumer
-requests that AMPS provide messages from the queue. The consumer can
-also request a maximum number of messages to have outstanding from the
-queue at any given time, referred to as the backlog for that consumer.
-When a message is available, and the consumer has fewer messages
-outstanding than the backlog for that consumer, AMPS delivers the
-message to the consumer. This improves latency and conserves bandwidth,
-since there is no need for consumers to repeatedly poll the queue to see
-if work is available. In addition, the server maintains an overall view
-of the consumers, which allows the server to control message
-distribution strategies to optimize for latency, optimize to
-deliver to clients with the most unused capacity, or optimize for
-general fairness.
-
-The following diagram presents a simplified view of an AMPS queue:
-
-.. image:: ../../common/chapters/queues/svg/AMPS-Queues-Intro.svg
-
-As the diagram indicates, a queue tracks a set of messages in the
-transaction log. The messages the queue is currently tracking are
-considered to be in the queue. When the queue delivers a message, it
-marks the message as having been delivered (shown as *leased* in the
-diagram above). Messages that have been processed are no longer tracked
-by the queue (for example, the message for the order 1 in the diagram
-above). When a message has been delivered and processed, that event is
-recorded in the transaction log to ensure that the queue meets the
-delivery guarantees even across restarts of AMPS.
-
-Since queues are implemented as views over underlying topics, AMPS
-allows you to create any number of queues over the same underlying
-topic. Each queue tracks messages to the topic independently, and can
-have different policies for delivery and fairness. When a queue topic
-has a different name than the underlying topic, you can subscribe to the
-underlying topic directly, and that subscription is to the underlying
-(non-queue) topic. When a queue topic has the same name as the
-underlying topic (the default), all subscriptions to that topic are to
-the queue. (Notice that bookmark subscriptions to a queue are pub/sub
-subscriptions that replay from the underlying topic in the transaction log,
-so the behavior in that case is the same as if the subscription was directly
-to the underlying topic.)
-
-Likewise, AMPS queues work seamlessly with the AMPS entitlement system.
-Permissions to queues are managed the same way permissions are managed
-to any other topic, as described in the 
-:ref:`Entitlement <#ug-entitlement>` section of the *AMPS User Guide*.
-
-While a message is in a queue, AMPS does not retain an extra copy of the message.
-Instead, AMPS retains in memory a small data structure indicating
-the state of the message and the position of the message in the transaction log.
-The amount of memory consumed by a queue is approximately 200 bytes per message,
-regardless of the size of the messages.
-
-AMPS queues provide a variety of options to help you tailor the behavior
-of each queue to meet your application's needs.
-
-Delivery Semantics
-^^^^^^^^^^^^^^^^^^
-
-AMPS queues deliver a message to a single subscriber at a time. In the
-most common case, a message is delivered to exactly one subscriber, and that
-subscriber processes the message.
-
-In the case that a subscriber does not successfully process a message,
-AMPS provides two delivery semantics to precisely specify the handling of
-the unprocessed message:
-
--  With ``at-least-once`` delivery, AMPS delivers the message to one
-   subscriber at a time, and expects that subscriber to explicitly
-   remove the message from the queue when the message has been received
-   and processed. With this guarantee, each message from the queue must
-   be processed within a specified timeout, or lease period. AMPS tracks
-   the amount of time since the message was sent to the subscriber. If
-   the subscriber has not responded by removing the message within the
-   lease period, AMPS revokes the lease and the message is available to
-   another subscriber. AMPS allows you to set limits on the number of
-   times a message is made available to another subscriber, using the
-   ``MaxCancels`` and ``MaxDeliveries`` configuration options.
-
-   In this model, receiving a message is the equivalent of a
-   non-destructive get from a traditional queue. To acknowledge and
-   remove the message, a subscriber uses the ``sow_delete`` command with
-   the bookmark of the message.
-
-   Leases are broken and messages are returned to the queue if the lease
-   holder disconnects from AMPS. This ensures that, if a message
-   processor fails or loses its connection to AMPS, the message can
-   immediately be processed by another message processor.
-
--  With ``at-most-once`` delivery, AMPS removes the message from the
-   queue as soon as the message is sent to a subscriber. However, the
-   subscriber still needs to acknowledge that the message was processed,
-   so that AMPS can track the subscription backlog, as described below.
-
-   In this model, receiving a message is the equivalent of a destructive
-   get from a traditional queue. The message is immediately removed by
-   AMPS, and is no longer available in the queue.
-
-Choosing Delivery Semantics to Handle Failures
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Regardless of the delivery semantics you choose, during typical message
-processing from a queue, AMPS delivers each message in the queue to a single
-subscriber, with no duplicates or redelivery.
-
-The difference between ``at-most-once`` and ``at-least-once`` semantics is
-important in cases where a failure happens. With ``at-most-once`` delivery,
-the message will not be redelivered even if the subscriber fails to process
-the message. With ``at-least-once`` delivery, the message will be redelivered
-to subscribers until the message is either explicitly acknowledged, explicitly
-removed, or expired based on the queue policy for expiration or retries.
-
-With either setting, a message is delivered exactly once, to a single subscriber,
-during normal processing.
-
-Consider the following recommendations when deciding how you would like AMPS to
-handle cases where the subscriber that receives the message fails to process the
-message:
-
-1. For ephemeral or lower-value data, where message loss in the case of failure is
-   preferable to duplicating message delivery, consider ``at-most-once`` semantics.
-   For example, processing a stream of events from sensors might fall into this
-   category: each message should be processed once, in order, during normal operation,
-   but missing a single data point from time to time may be less disruptive than
-   reconciling duplicates. 
-2. For higher value data, where duplicate message delivery in the event of failure is preferable
-   to message loss, consider ``at-least-once`` semantics. For example, if inserting the same
-   data into a database twice would simply be an in-place update of the same record with the
-   same information, having a duplicate insert happen occasionally in the event of failure
-   may be preferable to losing a message when a failure occurs.
-3. For higher value data, where manual intervention is required to reconcile messages
-   when there is a question as to whether the message has been correctly processed,
-   consider using ``at-least-once`` message delivery with a ``MaxDeliveries`` setting
-   and an action to move failed messages to a dead-letter queue for manual reconciliation.
-4. For higher value data that is part of a transactional dataflow, where processors
-   maintain state, it can be useful to include information about the message processed
-   in the transaction. This can be used with ``at-least-once`` messaging to guarantee that
-   the message is only processed once: if a processor receives a message that is already
-   marked as processed in the transactional store, the processor knows that the message
-   has already been acted on and should be acknowledged without further processing.
-   This is the most common pattern used to produce the result that each message is
-   processed "exactly once". This pattern involves additional overhead and processing,
-   trading off increased work in the application for stronger guarantees that a message
-   is only processed one time.
-
-Subscription Backlog
-^^^^^^^^^^^^^^^^^^^^
-
-For efficiency, queues in AMPS use a push model of delivery, providing
-messages to consumers when the message becomes available rather than
-requiring the consumer to poll the queue. To manage the workload among
-consumers, AMPS queues keep track of a *subscription backlog*. This
-backlog is the number of messages that have been provided to an
-individual subscription that have not yet been acknowledged. This
-backlog helps AMPS provide strong delivery guarantees while still
-optimizing for high throughput processing. AMPS calculates the
-subscription backlog for each subscription by calculating the
-*minimum* of the following:
-
--  The *minimum* ``MaxPerSubscriptionBacklog`` setting for the queues
-   matched by the subscription
-
-   *or* 
-
--  The ``max_backlog`` specified on the subscribe command
-
-Notice that, if a subscriber does not provide a ``max_backlog`` on a
-subscription, AMPS defaults to a ``max_backlog`` of ``1``. In practical
-terms, this means that an application must explicitly specify a backlog
-to be able to receive more than one message from a queue at a time,
-regardless of the queue configuration.
-
-Subscribers request a ``max_backlog`` by adding the request to the
-options string of the ``subscribe`` command. For example, to request a
-``max_backlog`` of 10, a subscriber includes ``max_backlog=10`` in the
-options for the command.
-
-.. tip::
-  To improve concurrency for subscribers, 60East recommends using a
-  backlog of at least ``2``. This allows efficient pipelined delivery, as
-  the consumer can be processing one message while the previous message is
-  being acknowledged. With a ``max_backlog`` higher than ``1``, the
-  consumer never needs to be stopped waiting for the next message from the
-  queue.
-
-Delivery Fairness
-^^^^^^^^^^^^^^^^^^
-
-When a queue provides ``at-least-once`` delivery, AMPS provides three
-different algorithms for distributing messages among subscribers. Each
-algorithm has different performance and fairness guarantees. For
-``at-most-once`` delivery, AMPS supports only the ``round-robin`` method
-of distributing messages.
-
-.. list-table:: *Message delivery strategies*
-     :header-rows: 1
-
-     * - Algorithm
-
-       - Description
-
-     * - ``fast``
-         
-
-       - This strategy optimizes for the lowest latency.
-         
-         AMPS delivers the message to the first subscription found that does not have a full
-         backlog. With this algorithm, AMPS tries to minimize the time spent determining which
-         subscription receives the message without attempting to distribute messages fairly
-         across subscriptions.
-
-     * - ``round-robin``
-         
-
-       - This strategy optimizes for general fairness across subscriptions.
-         
-         AMPS delivers the message to the next available subscription that does not have a full
-         backlog. With this algorithm, AMPS delivers messages evenly among the subscribers
-         that have space in their backlog.
-
-     * - ``proportional``
-         
-
-       - This strategy optimizes for delivery to subscriptions with the most unused capacity.
-         
-         AMPS delivers the message to the subscription that has the highest proportion of
-         backlog capacity unused. AMPS determines this by taking the ratio of unacknowledged
-         messages to the maximum backlog.
-         
-         For example, if there are three active subscribers for the queue, with backlog settings
-         and outstanding messages as follows:
-         
-         * Subscriber ``Inky``: ``max_backlog=2``, currently leased 1 message
-         
-         * Subscriber ``Blinky``: ``max_backlog=4``, currently leased 3 messages
-         
-         * Subscriber ``Clyde``: ``max_backlog=10``, currently leased 4 messages
-         
-         In this case, with ``proportional`` delivery, a new message for the queue will be
-         delivered to Clyde, since that subscriber has only filled 40% of the backlog, as
-         compared with 50% for Inky and 75% for Blinky.
-         
-         If more than one subscription has the same unused capacity, AMPS delivers the
-         message to the first subscription found with that capacity.
-
-
-
-AMPS defaults to ``proportional`` delivery for ``at-least-once`` queues
-and defaults to ``round-robin`` (the only valid delivery model) for
-``at-most-once`` queues.
-
-.. tip::
-
-   Each instance of AMPS manages delivery strategy from among the
-   messages that it currently owns and the subscriptions that are
-   present on that instance.
-
-   Delivery strategies apply only to a single instance, and are
-   not applied across instances.
-
-Acknowledging Messages
-^^^^^^^^^^^^^^^^^^^^^^^
-
-Subscribers must acknowledge each message to indicate to AMPS that a
-message has been processed. The point at which a subscriber acknowledges
-a message depends on the exact processing that the subscriber performs
-and the processing guarantees for the application. In general,
-applications acknowledge messages at the point at which the processing
-has a result that is durable and which would require an explicit action
-(such as another message) to change. 
-
-Some common points at which to acknowledge a message are:
-
--  When processing is fully completed.
-
--  When work is performed that would require a compensating action (that
-   is, when information is committed to a database or forwarded to a
-   downstream system).
-
--  When work is submitted to a processor that is guaranteed to either
-   succeed or explicitly indicate failure.
-
-To acknowledge a message, the subscriber typically uses the acknowledge
-convenience methods in the AMPS client. These commands issue a
-``sow_delete`` command with the bookmark from the message to
-acknowledge. AMPS allows subscribers to acknowledge multiple messages
-simultaneously by providing a comma-delimited list of bookmarks in the
-``sow_delete`` command: the AMPS clients provide facilities to batch
-acknowledgments for efficiency.
-
-AMPS allows an application to acknowledge messages by providing a filter on a
-``sow_delete`` command. In this case, the ``sow_delete`` acknowledges all
-messages that match the filter, regardless of whether the application that
-sends the command has a current lease on a given message or not. (The
-``Leasing`` parameter on the queue specifies whether AMPS allows a client to
-successfully acknowledge messages that it does not currently have leased.)
-
-For queues that use the ``at-least-once`` delivery model, there are two
-additional options available for acknowledging messages. 
-
-* **To return a message to the queue without processing it**, the
-  subscriber provides the ``cancel`` option on the acknowledgment.
-  In this case, AMPS returns the message to the queue just as though
-  the lease had expired. If the message is eligible for redelivery
-  (that is, it has not exceeded the maximum time or maximum cancels
-  configured for the queue), it is redelivered. Otherwise, the
-  message is expired from the queue. The ``MaxCancels`` option
-  allows you to configure how many times a message can be returned
-  to the queue for redelivery before AMPS expires the message.
-* **To immediately remove a message from the queue**, the subscriber
-  provides the ``expire`` option. In this case, AMPS does not return
-  the message to the queue for redelivery, but instead immediately
-  expires the message from the queue and triggers any configured
-  ``amps-action-on-sow-expire-message`` actions that monitor the
-  queue. The ``expire`` option can be especially helpful if your
-  application can determine that the message cannot be successfully
-  processed (for example, the message is unparseable or contains
-  invalid data).
-
-Options for acknowledging messages delivered from ``at-least-once`` queues:
-
-.. list-table::  *Acknowledgement options*
-     :header-rows: 1
-
-     * - **Option**
-         
-
-       - **Result**
-         
-
-     * - <none>
-         
-
-       - Message is considered to be successfully processed
-         and removed from the queue.
-
-     * - ``cancel``
-         
-
-       - Message is returned to the queue. The count of cancels
-         for this message is incremented, and if the count
-         is greater than the configured ``MaxCancels`` for the
-         queue, the message is expired.
-
-     * - ``expire``
-
-       - Message is immediately expired from the queue.
-
-
-For ``at-most-once`` queues, these options to acknowledgments have no effect,
-since the message is guaranteed not to be redelivered even if processing fails.
-
-Persisting Metadata for Queues     
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-A queue can, optionally, save the state of the queue as
-metadata in the journal directory. This can reduce the
-active memory footprint required for large queues, since
-the state of the queue can be paged out if necessary.
-This can also potentially improve recovery time if there
-are a large number of unacknowledged messages in the queue
-when AMPS starts. To persist metadata for a queue,
-set the ``FileBackedMetadata`` option to ``enabled``
-for the queue.
-
-.. tip::
-
-  When ``FileBackedMetadata`` is enabled, queue
-  state that is not currently needed can be paged
-  out to the file. This can substantially reduce
-  the active memory needed for large queues that
-  are infrequently used.
-
-  Notice that the state will only be paged
-  out as necessary. While this option increases
-  AMPS ability to manage queue state in
-  low-memory situations, AMPS will still
-  retain state in memory if memory is
-  available to improve performance.
-
-Optional Message Delivery Behaviors
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-AMPS queues provide the following optional message delivery behaviors:
-
-* A ``Priority`` can be specified for a queue, in which case
-  AMPS delivers messages in priority order rather than the
-  order in which the instance received the messages.
-
-* A ``BarrierExpression`` can be specified to provide a
-  synchronization point. When a message matches the
-  expression, all previous messages must be
-  acknowledged before AMPS will deliver the message
-  that matches the expression, or subsequent messages.
-
-These options are described in more detail in the
-:ref:`Advanced Queue Configuration<#ug-queues-advanced>` section below.
-
-Message Flow for Queues
-^^^^^^^^^^^^^^^^^^^^^^^
-
-The message flow for AMPS queues is as follows. The message flow differs
-depending on whether the queue is configured for ``at-most-once``
-delivery or ``at-least-once`` delivery.
-
-When the queue is configured for ``at-most-once`` delivery:
-
-1. A publisher publishes a message to an underlying topic.
-
-2. The message becomes available in the queue.
-
-3. For a message that is not a barrier message, the message is published to a subscriber when:
-
-   -  There is a subscription that matches the message, and the
-      subscriber is entitled to see the message
-
-   -  The message is the oldest message in the queue that matches the
-      subscription (if no ``Priority`` is specified for the queue)
-      or it has the highest priority value for messages in the queue
-      and is the oldest message at that priority (if ``Priority``
-      is specified).
-
-   -  The subscription has remaining capacity in its backlog
-
-   -  The subscription is the next subscription to receive the message
-      as determined by the delivery fairness for the queue
-
-   AMPS removes the message from the queue when the message is
-   published.
-
-4. For a message that is a barrier message, AMPS does not deliver the message
-   until all previous messages have been acknowledged. AMPS does not deliver
-   messages after the barrier message until the barrier message has been
-   delivered.
-
-5. If no subscription has requested the message, and the message has
-   been in the queue longer than the ``Expiration`` time, AMPS removes
-   the message from the queue. With AMPS queues, message expiration is
-   considered to be a normal way for the message to leave the queue, and
-   is not considered an error.
-
-6. The subscriber processes the message, and acknowledges the message
-   when processing is finished to indicate to AMPS that the subscriber
-   has capacity for another message.
-
-When the queue is configured for ``at-least-once`` delivery:
-
-1. A publisher publishes a message to an underlying topic.
-
-2. The message becomes available in the queue.
-
-3. For a message that is not a barrier message, the message is published to a subscriber when:
-
-   -  There is a subscription that matches the message
-
-   -  The message is the oldest message in the queue that matches the
-      subscription (if no ``Priority`` is specified for the queue) or
-      it has the highest priority value for messages in the queue
-      and is the oldest message at that priority (if ``Priority``
-      is specified).
-   -  The subscription has remaining capacity in its backlog
-
-   -  The subscription is the next subscription to receive a message as
-      determined by the delivery fairness for the queue
-
-   AMPS calculates the lease time for the message and provides that time
-   to the subscriber along with the message.
-
-4. For a message that is a barrier message (that is, the queue
-   has a ``BarrierExpression`` specified and the message matches
-   that expression), AMPS does not deliver the message
-   until all previous messages have been acknowledged. AMPS will not deliver
-   messages after the barrier message until the barrier message is delivered.
-
-5. If the message has been in the queue longer than the ``Expiration``
-   time, and there is no current lease on the message, AMPS removes the
-   message from the queue.
-
-6. If a subscriber has received the message, but has not removed the
-   message from the queue at the time the lease expires, AMPS returns
-   the message to the queue if the message has been in the queue less
-   than the ``Expiration`` time, and it has been delivered fewer than 
-   the ``MaxDeliveries`` limit. If the message has been in the queue
-   longer than the ``Expiration`` time or has been delivered more than
-   the number of times specified by ``MaxDeliveries``, AMPS removes the
-   message from the queue when the lease expires.
-
-7. The subscriber processes the message, and removes the message from
-   the queue by acknowledging the message (which is translated by the
-   AMPS client into the appropriate ``sow_delete`` command).
-
-Startup and Recovery for Queues
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-When the metadata for a queue is not persisted to a file,
-or if that file is not fully up to date with the transaction
-log, AMPS recovers the current state of a queue from the
-transaction log when the instance starts.
-
-The point at which AMPS chooses to start recovery
-for a given queue is as follows:
-
-* If this is a new queue,
-  or the configured ``RecoveryPoint`` for the queue
-  has changed since the last time AMPS started,
-  begin queue recovery at the configured recovery
-  point. (This defaults to the beginning of the
-  transaction log if no explicit ``RecoveryPoint`` is
-  configured for the queue.)
-
-* Otherwise, start from the most recent point in
-  the transaction log at which all previous messages
-  have been acknowledged. This recovery point is
-  stored in the ``queues.ack`` file in the journal
-  directory for all queues in the instance.  If that
-  recovery point is older than the journals currently
-  available, begin from the beginning of the
-  transaction log.
-
-To recover the state of the queue, AMPS proceeds through
-the journal file from the recovery point and restores
-the state of the queue based on the operations in the
-transaction log for that queue.
-
-AMPS determines the oldest recovery point for all of
-the queues in the instance, and reads journals
-from that point forward to the end of the transaction log.
-AMPS rebuilds the state of each queue as the replay progresses
-past the recovery point for that queue.
-
-
-Advanced Messaging and Queues
------------------------------
-
-Queues are implemented as AMPS topics which lets you use the advanced
-messaging features of AMPS to create your queues and provide insight
-into your queues. For example, consumers can use content filtering to
-select the messages from the queue that they want to consume. You can
-use content filters to select only a subset of messages published to an
-underlying topic to populate the queue. You can even create a view that
-aggregates data from multiple topics, and use that view as the
-underlying topic for the queue. Since messages for queues are recorded
-in the transaction log, you can easily replay the messages that the
-queue provided to subscribers by using a bookmark subscription.
-
-Querying Queues as a View
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-For each queue, AMPS provides a view of the currently available
-messages. Applications can query this queue just as though it were a
-view. For example, if you have a queue named ``PendingOrders``, you can
-see the currently available messages in the queue by querying the queue
-as though it were a view, with a ``sow`` command.
-
-A query of a queue is *read-only*. AMPS does not lease the returned
-messages to the querying application, or remove them from the queue.
-
-
-.. _#ug-sow-underlying-queue:
-
-Topic in the State of the World as an Underlying Topic for a Queue
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-AMPS fully supports a topic that maintains a SOW as an underlying topic
-for a queue. Since a queue records every individual publish to a topic
-(rather than simply preserving the current state of a distinct message
-identified by a SOW key), each publish to the SOW topic creates a new
-message in the queue.
-
-AMPS does not provide Out-of-Focus messages to the queue. Only publish
-messages are added to the queue.
-
-Deleting a message from an underlying topic that maintains a SOW does
-not remove the corresponding messages from the queue. Likewise, when a
-message expires from the SOW, it is not removed from the queue.
-
-When a message is published to a topic in the SOW, and that topic uses
-expiration, the message is stamped with the expiration value
-set for the topic before being written to the transaction log.
-This means that the message will have the same expiration in the queue.
-
-Delta Messaging with Queues
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-AMPS delta subscriptions rely on being able to determine the last state
-of a message delivered on a subscription and providing a set of changes
-to the subscriber. With AMPS queues, AMPS treats each update to a SOW
-record as a new message, so there's no previous state that would
-generate a delta message. When an underlying topic of a queue is a SOW
-topic, AMPS supports delta publish to that underlying topic. The full,
-merged message is added to the queue.
-
-AMPS allows delta subscriptions to a queue, but treats each message as a
-new publish and delivers the full message.
-
-Views and Aggregated Subscriptions over Queues
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-AMPS fully supports creating a view or an aggregated subscription with a
-queue as an underlying topic. In both cases, AMPS operates on the
-messages that are currently available in the queue. When a message is
-leased, that message is no longer available to the queue and does not
-appear in the view or the aggregated subscription. If the message is
-returned to the queue, then the message is again available to the view
-or aggregated subscription. When a message expires, that message is no
-longer available in the view or aggregated subscription.
-
-Views and aggregated subscriptions are considered to be a query of the
-queue, so they are *read-only*. Views and aggregated subscriptions do
-not lease messages from the queue and do not affect message delivery.
-
-Views over queues can be useful to show constantly-updated aggregates of
-the activity in the queue. For example, you could create an aggregate
-that shows the total value of unprocessed orders currently in the queue.
-
-Bookmark Subscriptions to Queues
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-A queue itself supports `at_most_once` or `at_least_once` delivery.
-It does not provide a way to replay messages once a subscriber
-has acknowledged the message.
-
-To support replay of messages from a queue, AMPS translates a
-bookmark subscription to a queue to be a bookmark subscription
-to the underlying topic for the queue. This allows you to replay
-messages from the underlying topic *without* queue delivery semantics.
-That is, a bookmark subscription to a queue becomes a
-publish/subscribe bookmark subscription to the underlying topic,
-*not* a subscription to the queue.  Messages from this subscription do
-not have ``at-most-once`` or ``at-least-once`` delivery, and do not need
-to be acknowledged. The subscription is a publish/subscribe bookmark
-subscription, just as though there was no queue for the topic.
-
-To get queuing semantics, do not include a bookmark on subscriptions to
-a queue.
-
-Queue Subscriptions Compared to Bookmark Replays
-------------------------------------------------
-
-At first glance, a subscription to a queue can look very similar to a bookmark
-replay. In both cases, messages are provided in order from the transaction log.
-In both cases, a message that is not processed can be retrieved and redelivered.
-In both cases, AMPS allows a subscription to exactly specify the
-messages of interest using content filtering.
-
-There are a few main differences:
-
-* **Delivery Model**
-
-  With a bookmark subscription, a message from a given topic can be delivered
-  to any number of subscribers and processed multiple times. A given
-  subscriber can replay the same messages any number of times, if needed.
-
-  With a queue subscription, the server guarantees that once a message
-  is processed, it is not delivered from that topic again.
-
-* **Delivery Limits**
-
-  With a bookmark subscription, by default AMPS will deliver messages to the
-  subscription as fast as the physical hardware, disk, and network
-  allow.
-
-  With a queue subscription, AMPS intentionally limits the number of messages
-  delivered to a subscriber, to help ensure that messages are processed in a 
-  timely manner. Since each message should be delivered only once, delivering  
-  every message to a single subscriber would run the risk of having a slow consumer  
-  holding messages that it is unable to process, while a faster consumer sits idle.
-  
-- **Acknowledgment**
-
-  With a bookmark subscription, the application and client libraries are
-  responsible for tracking which messages have been processed and providing
-  the correct recovery point. There is no state stored in the AMPS server
-  as to which messages have been consumed by a given subscription.
-
-  With a queue subscription, the AMPS server tracks whether a given message
-  has been processed. A queue subscriber must notify the server that it
-  is finished with a given message and is ready for more work. The AMPS
-  client libraries contain support for making this process easy, as well
-  as the ability to batch acknowledgments to provide high throughput
-  while maintaining efficient processing and the guarantees for queues.
-
-Replacing Queue Subscriptions
------------------------------
-
-Queues support the ``replace`` option for subscriptions. As with
-subscriptions to other topics, queue subscriptions can replace the
-content filter, the topic, the options, or all of the above. Replacement
-is atomic. The queue consumer is guaranteed that, after the replace
-occurs, only messages that match the new subscription will be delivered.
-
-Replacing queue subscriptions differs from unsubscribing and
-resubscribing with new parameters in two ways:
-
-1. AMPS does not break message leases or adjust the number of
-   currently-unacknowledged messages for the subscription, even if the
-   messages no longer match the current subscription. AMPS makes no
-   assumptions about the state of the messages, and requires the
-   subscriber to acknowledge them or allow the lease to expire.
-
-2. AMPS may change the maximum backlog for the subscription if either
-   the ``max_backlog`` option *or* the topic for the subscription has
-   changed. AMPS adjusts the backlog using the same logic as when the
-   subscription was entered: the maximum backlog will be the smaller of
-   the option set by the consumer or the limit on the queue. This can
-   result in a situation where the consumer has more messages leased
-   than the current maximum for the subscription, and no new messages
-   will be delivered until that number drops below the current maximum.
-
-   For example, if the consumer has a requested ``max_backlog`` of 10 and
-   updates a subscription from a queue with a configured maximum of 10
-   to a queue with a configured maximum of 5, the new backlog for the
-   subscription will be 5. However, the consumer may still have 10
-   messages outstanding.
-
-In all other ways, AMPS behaves as though the replaced subscription was
-a new subscription to the queue.
-
-.. index:: dead-letter queue, poison message handling
-
-Handling Unprocessed Messages
------------------------------
-
-Queuing applications occasionally encounter data that cannot be successfully
-processed or completed. AMPS allows you to configure actions that occur when
-a message expires from a message queue. You can create a dead-letter queue or
-topic by using the ``amps-action-on-sow-expire-message`` action in conjunction
-with the ``MaxCancels`` and ``MaxDeliveries`` options and the ``cancel``
-and ``expire`` options on the ``sow_delete`` command. 
-
-As an example, suppose your queue ``Jobs`` is used to coordinate computation
-jobs, and in the rare case where a job cannot be completed successfully, you
-would like AMPS to automatically move the job to another queue, ``FailedJobs``.
-AMPS provides ``MaxCancels`` and ``MaxDeliveries`` options to specify how many
-attempts it should make to deliver a message to a subscriber:
-
-.. code-block:: xml
-
-    <AMPSConfig>
-        ...
-
-        <SOW>
-            <Queue>
-                <Name>Jobs</Name>
-                <MessageType>json</MessageType>
-                <MaxCancels>3</MaxCancels>
-                <MaxDeliveries>10</MaxDeliveries>
-            </Queue>
-            <Queue>
-                <Name>FailedJobs</Name>
-                <MessageType>json</MessageType>
-            </Queue>
-        </SOW>
-    </AMPSConfig>
-
-In this configuration, AMPS will automatically expire a message from ``Jobs``
-on the 3rd time it receives a ``sow_delete`` with the ``cancel`` operation
-for that message, or after the 10th unsuccessful delivery of the message.
-By default, expiring a message simply means the message is removed from the
-queue, but in this case, we also want the ``FailedJobs`` queue to receive
-the message so that it can be manually reviewed later. To accomplish this,
-AMPS provides the ``amps-action-on-sow-expire-message`` action which can be
-configured to do so:
-
-.. code-block:: xml
-
-    <AMPSConfig>
-        ...
-        <Actions>
-            <Action>
-                <On>
-                    <Module>amps-action-on-sow-expire-message</Module>
-                    <Options>
-                        <Topic>Jobs</Topic>
-                        <MessageType>json</MessageType>
-                    </Options>
-                </On>
-                <Do>
-                    <Module>amps-action-do-publish-message</Module>
-                    <Options>
-                        <Topic>FailedJobs</Topic>
-                        <MessageType>json</MessageType>
-                        <Data>{ "message": {{AMPS_DATA}}, "reason": "{{AMPS_REASON}}" }</Data>
-                    </Options>
-                </Do>
-            </Action>
-        </Actions>
-        ...
-    </AMPSConfig>
-
-With this action configured, any expired message from ``Jobs`` will result in a
-message automatically published to ``FailedJobs`` containing the original
-message inside the ``message`` element, and an AMPS-supplied reason code in
-the ``reason`` field.
-
-
-.. _#ug-queues-advanced:
-
-Advanced Queue Configuration
-----------------------------
-
-This section describes more advanced configuration settings for
-AMPS message queues.
-
-Using Multiple Underlying Topics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-AMPS queues can contain messages from any number of underlying topics. This provides
-a flexible delivery model, and allows applications to populate multiple queues
-with a single publish to AMPS, which simplifies publisher code, reduces bandwidth,
-and ensures that the message is provided to all queues from the same point in
-the message stream.
-
-To create a queue that includes messages from underlying topics, you provide a
-regular expression that matches the set of topic names that contain messages for
-the queue.  You also provide a ``DefaultPublishTarget`` that specifies the topic
-name for AMPS to use when a message is published directly to the queue topic.
-
-For example, you might configure a set of topics as follows:
-
-.. code-block:: xml
-
-  <SOW>
-    ...
-    <Queue>
-       <Name>ORDERS_ANALYTICS</Name>
-       <MessageType>json</MessageType>
-       <UnderlyingTopic>^ORDERS$|^ORDERS_ANALYTICS_DIRECT$</UnderlyingTopic>
-       <DefaultPublishTarget>ORDERS_ANALYTICS_DIRECT</DefaultPublishTarget>
-    </Queue>
-    <Queue>
-       <Name>ORDERS_RISK</Name>
-       <MessageType>json</MessageType>
-       <UnderlyingTopic>^ORDERS$|^ORDERS_RISK_DIRECT$</UnderlyingTopic>
-       <DefaultPublishTarget>ORDERS_RISK_DIRECT</DefaultPublishTarget>
-    </Queue>
-    ...
-  </SOW>
-
-In this case, when a message is published to the ``ORDERS`` topic, both the
-``ORDERS_ANALYTICS`` and the ``ORDERS_RISK`` queues deliver the message.
-However, a publisher can also publish directly to each queue by publishing
-a message to the ``_DIRECT`` topic for that queue. Furthermore, any publish
-to the name of the queue will be routed to the appropriate ``_DIRECT`` topic.
-
-The following table demonstrates how messages are provided to topics with
-this configuration:
-
-.. list-table:: 
-     :header-rows: 1
-
-     * - Publish To
-
-       - Results
-
-     * - ``ORDERS``
-         
-
-       - Both ``ORDERS_ANALYTICS`` and ``ORDERS_RISK`` enqueue the
-         message, since ``ORDERS`` matches the ``UnderlyingTopic`` of
-         both queues.
-
-     * - ``ORDERS_ANALYTICS``
-         
-
-       - The message is published to the ``DefaultPublishTarget``
-         of ``ORDERS_ANALYTICS``, which is ``ORDERS_ANALYTICS_DIRECT``.
-         The message is then enqueued to ``ORDERS_ANALYTICS``, since
-         ``ORDERS_ANALYTICS_DIRECT`` matches the ``UnderlyingTopic`` of
-         ``ORDERS_ANALYTICS``.
-
-     * - ``ORDERS_RISK``
-         
-
-       - The message is published to the ``DefaultPublishTarget``
-         of ``ORDERS_RISK``, which is ``ORDERS_RISK_DIRECT``.
-         The message is then enqueued to ``ORDERS_RISK``, since
-         ``ORDERS_RISK_DIRECT`` matches the ``UnderlyingTopic``
-         ``ORDERS_RISK``.
-
-     * - ``ORDERS_ANALYTICS_DIRECT``
-         
-
-       - The message is published to ``ORDERS_ANALYTICS_DIRECT``,
-         and is then enqueued to ``ORDERS_ANALYTICS``.
-
-     * - ``ORDERS_RISK_DIRECT``
-         
-
-       - The message is published to ``ORDERS_RISK_DIRECT``,
-         and is then enqueued to ``ORDERS_RISK``.
-
-
-
-Priority Queues
-^^^^^^^^^^^^^^^
-
-AMPS includes the ability to specify that messages from a queue are delivered
-in order of *priority* rather than being delivered strictly in publication order.
-To enable prioritization on a ``Queue``, add the ``Priority`` tag to the queue
-configuration and specify the field or expression to use to set the priority.
-When a queue definition specifies ``Priority``, AMPS orders delivery based on
-a descending sort of the value calculated for the ``Priority`` rather than
-in descending order of age.
-
-``Priority`` expressions use the same grammar as other expressions in AMPS, as
-described in :ref:`#ug-amps-expressions`. The results of the ``Priority`` expression 
-are interpreted as an ``unsigned long``. That is, the result should be a positive 
-integer that can be represented in 64 bits. Non-numeric values or ``NULL`` are treated 
-as ``NaN``, and have the lowest priority.
-
-For example, to create a queue that delivers messages in order of the product of
-the price and quantity specified in the message, you could use the following
-queue definition:
-
-.. code-block:: xml
-
-    <AMPSConfig>
-        ...
-        <!-- Other configuration, including
-             recording  that OrderPriority
-             and Orders are recorded in the
-             transaction log.
-          -->
-
-        <SOW>
-            ...
-            <Queue>
-                <Name>OrderPriority</Name>
-                <MessageType>json</MessageType>
-                <Semantics>at-least-once</Semantics>
-                <UnderlyingTopic>Orders</UnderlyingTopic>
-                <Priority>/price * /qty</Priority>
-            </Queue>
-        </SOW>
-
-        ...
-    </AMPSConfig>
-
-Synchronizing Work with Barrier Messages
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-AMPS queues provide the ability to synchronize work across a set of
-subscribers by sending the same message to all subscribers simultaneously,
-when the queue is fully processed up to the point of the message.
-
-These messages are referred to as *barrier messages*, since they are roughly
-equivalent to the concept of a barrier in multithreaded programming.
-
-Barrier messages simplify coordination between current subscribers to
-a queue.  They allow you to easily coordinate operations (such as end of day
-processing), update reference data at the precise point in the message
-stream that the update should take effect, send a shutdown message
-when all available work is processed, and so on. Since barrier messages
-are integrated with queue delivery, there is no need to write extra code
-to correlate the barrier message with the correct point in the queue,
-to manage multiple subscriptions, or to write code that guarantees 
-delivery to all current subscribers.
-
-To specify which messages should be treated as barrier messages, the
-queue configuration includes a ``BarrierExpression`` element that contains
-a filter. All messages that match that filter are considered barrier
-expressions.
-
-When a message matches the ``BarrierExpression``, AMPS delivers the
-message in the following way:
-
-* AMPS does not deliver the message until all previous messages in the
-  queue are acknowledged.
-* AMPS does not deliver messages that are after the barrier message
-  until the barrier message is sent to subscribers.
-* When all previous messages in the queue are acknowledged, AMPS
-  delivers the barrier message to all current subscribers to the
-  queue.
-* AMPS immediately removes the message from the queue, without
-  requiring acknowledgment from any subscriber.
-
-A barrier message sent to a subscriber does not count toward that subscriber's
-backlog, since the barrier message is immediately acknowledged. When
-sending the barrier message, AMPS does not consider the current backlog
-of the subscriber (so, subscribers that request ``max_backlog=0`` or which
-use a regular expression topic and have their backlog filled by messages
-from other topics will still receive the barrier message).
-
-AMPS does, however, apply content filters and entitlement filters for each
-subscription when delivering the message. If the content filter or entitlement
-filter for a subscription does not match the barrier message, that subscription
-will not receive the message. The fact that this subscription does not
-receive the actual barrier message does not affect the behavior of the
-barrier itself. A subscription that does not match the message will still
-see message delivery pause until the barrier is released, but will not
-receive the actual barrier message. 
-
-Each AMPS instance manages the delivery of barrier messages for the subscribers
-connected to that instance, when all messages prior to the barrier message in
-the local queue have been acknowledged.
-
-Limiting Currently Deliverable Messages
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-In some cases, it may be important to limit the maximum number of
-messages that AMPS considers to be part of the currently active,
-deliverable part of the queue. This can be advantageous in cases
-where memory is limited, or in cases where it is expected that
-a queue is only processed infrequently (for example, a
-reconciliation process that runs at the end of each day).
-
-The ``TargetQueueDepth`` option on a queue allows you to set
-the target depth for the number of messages that AMPS will
-consider to be deliverable at a given time.
-
-If the number of messages for the queue exceeds the ``TargetQueueDepth``
-on an instance, messages beyond the specified depth are considered to
-be inactive (though still part of the queue). Those messages are not
-deliverable to subscribers on this instance. They do not appear in
-a ``sow`` query of the queue, and are not considered when a view
-over the queue is calculated.  A ``sow_delete`` that uses a filter
-will only apply to the messages in the queue that are currently deliverable.
-As messages are acknowledged from the queue, AMPS will add messages
-to the active part of the queue until the active part of the queue
-once again reaches the ``TargetQueueDepth``.  For messages that are
-not currently deliverable, AMPS will not read those messages
-from the transaction log and will not maintain queue state for
-those messages. This also means that, if no messages within the
-``TargetQueueDepth`` match a given subscription, no messages will
-be delivered to that subscription even if later messages in the
-transaction log match the subscription.
-
-Although AMPS does not consider messages that are currently inactive
-to be deliverable, AMPS will preserve queue delivery guarantees in
-the event that a message that is not currently deliverable is
-acknowledged before it enters the queue and becomes deliverable.
-To do this, AMPS keeps a list of acknowledgments that
-have been received for messages that are not in the queue. When
-a message that would be added to the active part of the queue
-has already been acknowledged, AMPS does not add that message to
-the queue. Applications that use ``TargetQueueDepth`` to limit memory
-growth should avoid a pattern where messages are acknowledged before
-they enter the queue to reduce memory consumption for acknowledgments
-that cannot yet be processed.
-
-Likewise, if a transfer request from another instance arrives for
-a message that is in the transaction log, but is beyond the limit
-set by the ``TargetQueueDepth``, AMPS will process messages further
-into the queue until it reaches the message to be transferred. At
-that point, AMPS will transfer the message and stop processing further
-messages until either:
-
-* The current depth drops below the ``TargetQueueDepth``, *or*
-
-* AMPS once again receives a request for a message that is in the
-  transaction log, but beyond the current active messages in the queue.
-
-.. important::
-
-  A ``TargetQueueDepth`` *cannot* be set for a queue that specifies
-  a ``Priority`` or a ``BarrierExpression``.
-
-
-.. include:: ../../common/chapters/queues/queues.inc
-
-