amps-python-client 5.3.4.1__zip → 5.3.4.2__zip

amps-python-client-5.3.4.2/docs/python/html/_sources/chapters/high-availability.rst.txt

@@ -0,0 +1,879 @@
+.. index:: HAClient
+
+
+9. High Availability 
+====================
+
+The AMPS Python Client provides an easy way to create highly-available
+applications using AMPS, via the ``HAClient`` class. ``HAClient``
+derives from ``Client`` and offers the same methods, but also adds
+protection against network, server, and client outages.
+
+Using ``HAClient`` allows applications to automatically:
+
+-  Recover from temporary disconnects between client and server.
+
+-  Failover from one server to another when a server becomes
+   unavailable.
+
+Since the ``HAClient`` automatically manages failover and
+reconnection, 60East recommends using the ``HAClient`` for applications
+that need to:
+
+-  Automatically reconnect and resume work in the case of disconnection.
+
+-  Ensure no messages are lost or duplicated after a reconnect or
+   failover.
+
+-  Persist messages and bookmarks on disk for protection against client
+   failure.
+
+You can choose how your application uses ``HAClient`` features. For
+example, you might need automatic reconnection, but have no need to
+resume subscriptions or republish messages. The high availability
+behavior in ``HAClient`` is provided by implementations of defined
+interfaces. You can combine different implementations provided by 60East
+to meet your needs, and implement those interfaces to provide your own
+policies.
+
+Some of these features require specific configuration settings on your
+AMPS instance(s). This chapter mentions these features and describes how
+to use them from the AMPS Java client. You can find full documentation
+for these settings and server features in the *AMPS User Guide*.
+
+.. include:: ./ha-overview-python.inc
+Reconnection with HAClient
+--------------------------
+
+The most important difference between ``Client`` and ``HAClient`` is
+that ``HAClient`` automatically provides a reconnect handler.
+
+This description provides a high-level framework for understanding the
+components involved in failover with the ``HAClient``. The components
+are described in more detail in the following sections.
+
+The ``HAClient`` reconnect handler performs the following steps when
+reconnecting:
+
+1. Calls the ``ServerChooser`` to determine the next URI to connect to
+   and the authenticator to use for that connection.
+
+   If the connection fails, calls ``get_error`` on the ``ServerChooser``
+   to get a description of the failure, sends an exception to the
+   exception listener and stops the reconnection process.
+
+2. Calls the ``DelayStrategy`` to determine how long to wait before
+   attempting to reconnect and waits for that period of time.
+
+3. Connects to the AMPS server. If the connection fails, calls
+   ``report_failure`` on the ``ServerChooser`` and begins the process
+   again.
+
+4. Logs on to the AMPS server. If the connection fails, calls
+   ``report_failure`` on the ``ServerChooser`` and begins the process
+   again.
+
+5. Calls ``report_success`` on the ``ServerChooser``.
+
+6. Receives the bookmark for the last message that the server has
+   persisted. Discards any older messages from the ``PublishStore``.
+
+7. Republishes any messages in the ``PublishStore`` that have not been
+   persisted by the server.
+
+8. Re-establishes subscriptions using the ``SubscriptionManager`` for
+   the client. For bookmark subscriptions, the reconnect handler uses
+   the ``BookmarkStore`` for the client to determine the most recent
+   bookmark, and re-subscribes with that bookmark. For subscriptions that
+   do not use a bookmark, the ``SubscriptionManager`` simply re-enters
+   the subscription, meaning that it is entered at the point at which
+   the ``HAClient`` reconnects.
+
+The ``ServerChooser``, ``DelayStrategy``, ``PublishStore``,
+and ``BookmarkStore`` are all extension points
+for the ``HAClient``. You can adapt the failover and recovery behavior
+by setting a different object for the behavior you want to customize on
+the ``HAClient`` or by providing your own implementation.
+
+For example, the convenience methods in the previous section customize
+the behavior of the ``PublishStore`` and ``BookmarkStore`` by providing
+either memory-backed or file-backed stores.
+
+The reconnection process runs on the thread that discovers the
+disconnection. This means that, in the event that an application
+thread discovers the disconnection as a result of a call to
+the Python AMPS client, that call may not return until a
+connection is re-established (or until the server chooser
+indicates failure, in which case the application will
+receive an exception).
+
+The Python client includes a ``retry_on_disconnect`` setting that
+controls this retry behavior when the client is disconnected. When
+set to ``True`` (the default), any call to the Client that results
+in a command being sent to AMPS may block until a connection is
+re-established.  When set to ``False``, the ``HAClient`` will
+retry the connection a single time and throw an exception
+if the connection cannot be re-established.
+
+Regardless of the ``retry_on_disconnect`` setting, a call
+to ``publish`` will result in the message being stored in
+the ``PublishStore`` for the client if one is set.
+
+
+Choosing Store Durability
+-------------------------
+
+.. index:: Method; create_file_backed(), createFileBacked(), Method; createMemoryBacked(), 
+    create_memory_backed(),
+
+If your application needs reliable publish to AMPS, install a
+``PublishStore`` in the ``HAClient``. If your application needs to
+resume replays from the transaction log, install a ``BookmarkStore`` in
+the ``HAClient``.
+
+These stores provide the following capabilities:
+
+-  A *bookmark store* tracks received messages and is used to resume
+   subscriptions that replay from the transaction log.
+
+-  A *publish store* tracks published messages and is used to ensure that
+   messages are persisted in AMPS.
+
+The AMPS client provides a memory-backed version of each store and a
+file-backed version of each store. The store interface is public, and an
+application can create and provide a custom store as necessary. An
+``HAClient`` can use either a memory backed store or a file backed store
+for protection. Each method provides resilience to different failures, as 
+described below:
+
+-  *Memory-backed stores* provide recovery disconnection from AMPS by
+   storing messages and bookmarks in your process' address space. This
+   is the highest performance option for working with AMPS in a highly
+   available manner. The trade-off with this method is there is no
+   protection from a crash or failure of your client application. If
+   your application is terminated prematurely or, if the application
+   terminates at the same time as an AMPS instance failure or network
+   outage, then messages may be lost or duplicated. The state of
+   bookmark replays will be lost when the application shuts down.
+   Messages in the publish store when the application shuts down
+   will not be maintained through a restart, so the application will
+   not be able to attempt any necessary redelivery when the application restarts.
+
+   A memory-backed store should only be used by one instance of a client at
+   a time.
+
+-  *File-backed stores* provide recovery after client failure and
+   disconnection from AMPS by storing messages and bookmarks on disk. To
+   use this protection method, the ``create_file_backed`` method
+   requests additional arguments for the two files that will be used for
+   both bookmark storage and message storage. If these files exist and
+   are non-empty (as they would be after a client application is
+   restarted), the ``HAClient`` loads their contents and ensures
+   synchronization with the AMPS server once connected. The performance
+   of this option depends heavily on the speed of the device on which
+   these files are placed. When the files do not exist (as they would
+   the first time a client starts on a given system), the ``HAClient``
+   creates and initializes the files, and in this case the client does
+   not have a point at which to resume the subscription or messages to
+   republish.
+
+   A store file should only be used by one instance of a client at
+   a time.
+
+   When using file-backed stores, 60East recommends periodically removing
+   unneeded entries by calling the ``prune()`` method. The precise strategy
+   that your application uses to call ``prune()`` depends on the nature of the
+   application. Most applications call ``prune()`` when the application exits.
+
+   There are two basic strategies that applications follow while the
+   application runs:
+
+   * Install a resize handler and call ``prune()`` after a specified number
+     of resize operations or when the store reaches a specific size.
+
+   * Call ``prune()`` after a specific number of messages are processed (for
+     example, every 10,000 messages received or every 1,000 updates completed).
+
+   Regardless of the strategy, it is best to call ``prune()`` when the application
+   is idle, since the ``prune()`` call rewrites the log file.
+
+
+The store interface is public and an application can create and provide
+a custom store as necessary. While clients provide convenience methods
+for creating file-backed and memory-backed ``HAClient`` objects with the
+appropriate stores, you can also create and set the stores in your
+application code. For the AMPS Python client, stores are implemented in
+C++. You can implement stores using C++, and use the technique described
+in Chapter 12 of this guide - *Using the C++ Client*, to set the store on 
+the client.
+
+Starting in 5.3.2.0, the underlying AMPS client contains a recovery point
+adapter interface to make it easier to add a custom persistence layer
+to a bookmark store. The distribution includes a recovery point adapter
+that can store bookmark recovery information in an AMPS SOW topic.
+
+The ``HAClient`` provides convenience methods for creating clients and
+setting stores. You can also construct an ``HAClient`` and set the store
+implementations you choose.
+
+In this example, we create several clients. The first client uses memory
+stores for both bookmarks and publishes. The second client uses files
+for both bookmarks and publishes. The third client uses a file for
+bookmarks. The third client does not set a store for publishes, which
+means that AMPS provides the default store (and no outgoing messages are
+stored). The final client does not specify any stores, so has no
+persistence for published messages or bookmark subscriptions, but can
+take advantage of the automatic failover and reconnection in the
+``HAClient``.
+
+.. code:: python
+
+    # Memory publish store, memory bookmark store
+    memoryClient = AMPS.HAClient("lessImportantMessages")
+
+    # File-backed publish store, file-backed bookmark store
+    diskClient = AMPS.HAClient("moreImportantMessages",
+              "/mnt/fastDisk/moreImportantMessages.outgoing",
+              "/mnt/fastDisk/moreImportantMessages.incoming")
+
+    # No-op publish store, file-backed bookmark store
+    subscriberClient = AMPS.HAClient("subscriber", no_store=True)
+    subscriberClient.set_bookmark_store(     \
+                AMPS.MMapBookmarkStore("/mnt/fastdisk/bookmark.store"))
+
+    # No-op publish store, no-op bookmark store
+    # Failover behavior only.
+    streamReader = AMPS.HAClient("streamReader",no_store=True)
+
+**Example 9.1:** *HAClient creation example*
+
+.. include:: ./sow_bookmark_store.inc
+Capacity Planning and Store Sizing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When an application uses a file-backed store, it is important to make
+sure that there is enough space available on the file system to
+be able to manage the store.
+
+For logged bookmark stores, an application needs to keep a bookmark record for
+each message received, each message discarded, and the persisted
+acknowledgments delivered by the server approximately once a second.
+Each bookmark entry consumes roughly 70 bytes of storage *plus* the length
+of the subscription ID for the subscription receiving the message. The logged
+bookmark store retains entries until an application explicitly calls
+``prune()``. The capacity needed for a logged bookmark store will
+depend on the strategy that the application uses for pruning the file.
+
+For a file-backed publish store, the application needs to be able to
+store published messages until the AMPS server that the publisher is
+connected to acknowledges those messages as persisted. The volume of
+messages that needs to be stored depends on the failover policy for
+the server -- that is, the maximum amount of time that the server will
+allow a downstream instance to fail to acknowledge a message before
+the server downgrades that connection to ``async`` acknowledgment.
+By default, AMPS does not downgrade connections: this policy must
+be set explicitly using the AMPS actions. As an example, if the
+server is configured to downgrade connections that are more than
+120 seconds behind, then -- for disaster recovery -- the application
+must have the capacity to store 120 seconds of published messages
+at peak publishing load. However, unlike the logged bookmark store, a
+file-backed publish store removes messages from the store and reuses
+the space once AMPS has acknowledged the message.
+
+Connections and the Server Chooser 
+----------------------------------
+
+.. index:: HAClient Client, Client Interface, ServerChooser Interface,
+
+Unlike ``Client``, the ``HAClient`` attempts to keep itself connected to
+an AMPS instance at all times, by automatically reconnecting or failing
+over when it detects that the client is disconnected. When you are using
+the ``Client`` directly, your disconnect handler usually takes care of
+reconnection. ``HAClient``, on the other hand, provides a disconnect
+handler that automatically reconnects to the current server or to the
+next available server.
+
+To inform the ``HAClient`` of the addresses of the AMPS instances in
+your system, you pass a ``ServerChooser`` instance to the ``HAClient``.
+``ServerChooser`` acts as a smart enumerator over the servers available:
+``HAClient`` calls ``ServerChooser`` methods to inquire about what
+server should be connected, and calls methods to indicate whether a
+given server succeeded or failed.
+
+.. index:: Method; connect_and_logon(), connect_and_logon() method,
+
+The AMPS Python client provides a simple implementation of
+``ServerChooser``, called ``DefaultServerChooser``, that provides very
+simple logic for reconnecting. This server chooser is most suitable for
+basic testing, or in cases where an application should simply rotate
+through a list of servers. For most applications, you implement the
+``ServerChooser`` interface yourself for more advanced logic, such as
+choosing a backup server based on your network topology, or limiting the
+number of times your application should try to reconnect to a given
+address.
+
+To connect to AMPS, you provide a ``ServerChooser`` to ``HAClient`` and
+then call ``connect_and_logon()`` to create the first connection:
+
+.. code:: python
+
+    memoryClient = AMPS.HAClient("myClient")
+
+    # primary.amps.xyz.com is the primary AMPS instance, and
+    # secondary.amps.xyz.com is the secondary
+    chooser = AMPS.DefaultServerChooser()
+    chooser.add("tcp://primary.amps.xyz.com:12345/fix")
+    chooser.add("tcp://secondary.amps.xyz.com:12345/fix")
+    memoryClient.set_server_chooser(chooser)
+    memoryClient.connect_and_logon()
+    ...
+    myClient.disconnect()
+                        
+**Example 9.2:** *Multiple HAClient creation example*
+
+.. index:: Disconnection, Method; disconnect(), Method;
+    connect_and_logon(),
+
+Similar to ``Client``, ``HAClient`` remains connected to the server until
+``disconnect()`` is called. Unlike ``Client``, ``HAClient``
+automatically attempts to reconnect to your server if it detects a
+disconnect and if that server cannot be connected, fails over to the
+next server provided by the ``ServerChooser``. In this example, the call
+to ``connect_and_logon()`` attempts to connect and login to
+``primary.amps.xyz.com``, and returns if that is successful. If it
+cannot connect, it tries ``secondary.amps.xyz.com``, and continues
+trying servers from the ``ServerChooser`` until a connection is
+established. Likewise, if it detects a disconnection while the client is
+in use, then ``HAClient`` attempts to reconnect to the server with which
+it was most recently connected; if that is not possible, then it moves
+on to the next server provided by the ``ServerChooser``.
+
+The default ``ServerChooser`` simply provides the next URL in the
+sequence. This strategy works for many applications. If you need a
+different strategy, you can implement your own logic for failover by
+creating a class derived from ``ServerChooser``.
+
+Setting a Reconnect Delay and Timeout
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. index:: ReconnectDelayStrategy, reconnection delay, reconnection
+    timeout,
+
+You can control the amount of time between reconnection attempts and 
+set a total amount of time for the ``HAClient`` to attempt to reconnect.
+
+The AMPS Python client includes a method for setting a delay strategy on
+a client, ``set_reconnect_delay_strategy``. This method accepts an
+instance of any type that provides the methods
+``get_connect_wait_duration`` and ``reset``, as described in the API
+documentation.
+
+While you can easily implement your own delay strategy, the client also
+provides two delay strategies:
+
+-  ``FixedDelayStrategy`` provides the same delay each time the
+   ``HAClient`` tries to reconnect.
+
+-  ``ExponentialDelayStrategy`` provides an exponential backoff until a
+   connection attempt succeeds.
+
+To use either of these classes, you simply create an instance, set the
+appropriate parameters, and install that instance as the delay strategy
+for the ``HAClient``. For example, the following code sets up a
+reconnect delay that starts at 200ms and increases the delay by 1.5
+times after each failure. The strategy allows a maximum delay between
+connection attempts of 5 seconds, and will not retry longer than 60
+seconds.
+
+.. code:: python
+
+    theClient = AMPS.HAClient("myClient")
+
+    theClient.set_reconnect_delay_strategy(      \
+       AMPS.ExponentialDelayStrategy(            \
+                       initial_delay=200,        \
+                       backoff_exponent=1.5,     \
+                       maximum_delay=5000,       \
+                       maximum_retry_time=60000) \
+                     )
+
+Implementing a Server Chooser
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. index:: ServerChooser Interface,
+
+As described above, you provide the ``HAClient``
+with connection strings to one or more AMPS servers using a
+``ServerChooser``. The purpose of a ``ServerChooser`` is to provide
+information to the ``HAClient``. A ``ServerChooser`` does not manage the
+reconnection process, and should not call methods on the ``HAClient``.
+
+A ``ServerChooser`` has two required responsibilities to the
+``HAClient``:
+
+-  Tells the ``HAClient`` the connection string for the server to
+   connect to. If there are no servers, or the ``ServerChooser`` wants
+   the connection to fail, the ``ServerChooser`` returns an empty
+   string.
+
+   To provide this information, the ``ServerChooser`` implements the
+   ``get_current_uri()`` method.
+
+-  Provides an ``Authenticator`` for the current connection string. This
+   is especially important for installations where different servers
+   require different credentials or authentication tokens must be reset
+   after each connection attempt.
+
+   To provide the authenticator, the ``ServerChooser`` implements the
+   ``get_current_authenticator()`` method.
+
+The ``HAClient`` calls the ``get_current_uri()`` and
+``get_current_authenticator()`` methods each time it needs to make a
+connection.
+
+Each time a connection succeeds, the ``HAClient`` calls the
+``report_success()`` method of the ``ServerChooser``. Each time a
+connection fails, the ``HAClient`` calls the ``report_failure()`` method
+of the ``ServerChooser``. The ``HAClient`` does not require the
+``ServerChooser`` to take any particular action when it calls these
+methods. These methods are provided for the ``HAClient`` to do internal
+maintenance, logging, or record keeping. For example, an ``HAClient``
+might keep a list of available URIs with a current failure count, and
+skip over URIs that have failed more than 5 consecutive times until all
+URIs in the list have failed more than 5 consecutive times.
+
+When the ``ServerChooser`` returns an empty string from
+``get_current_uri()``, indicating that no servers are available for
+connection, the ``HAClient`` calls the ``get_error()`` method on the
+``ServerChooser``, if one is provided, and includes the string returned
+by ``get_error()`` in the generated exception.
+
+Heartbeats and Failure Detection 
+---------------------------------
+
+.. index:: Heartbeat,
+
+Use of the ``HAClient`` allows your application to quickly recover from
+detected connection failures. By default, connection failure detection
+occurs when AMPS receives an operating system error on the connection.
+This system may result in unpredictable delays in detecting a connection
+failure on the client, particularly when failures in network routing
+hardware occur, and the client primarily acts as a subscriber.
+
+The heartbeat feature of the AMPS client allows connection failure to be
+detected quickly. Heartbeats ensure that regular messages are sent
+between the AMPS client and server on a predictable schedule. The AMPS
+client and server both assume disconnection has occurred if these
+regular heartbeats cease, ensuring disconnection is detected in a timely
+manner. To use the heartbeat feature, call the ``set_heartbeat`` method on
+``Client`` or ``HAClient``:
+
+.. code:: python
+
+    memoryClient = AMPS.HAClient("importantStuff")
+    ...
+    memoryClient.set_heartbeat(3)
+    memoryClient.connect_and_logon()
+    ...
+
+**Example 9.3:** *Heartbeat example*
+
+.. index:: Method; set_heartbeat(), set_heartbeat() method,
+
+Method ``set_heartbeat`` takes one parameter: the heartbeat interval. The
+heartbeat interval specifies the periodicity of heartbeat messages sent
+by the server: the value ``3`` indicates messages are sent on a
+three-second interval. If the client receives no messages in a
+six-second window (two heartbeat intervals), the connection is assumed
+to be dead, and the ``HAClient`` attempts reconnection. An additional
+variant of ``set_heartbeat`` allows the idle period to be set to a value
+other than two heartbeat intervals. (The server, however, will always consider
+a connection to be closed after two heartbeat intervals without any traffic.)
+
+Notice that, for ``HAClient``, ``setHeartbeat`` must be called *before*
+the client is connected. For ``Client``, ``setHeartbeat`` must be called
+*after* the client is connected.
+
+.. caution::
+
+   Heartbeats are serviced on the receive thread created by the AMPS
+   client. Your application must not block the receive thread for longer
+   than the heartbeat interval or the application is subject to being
+   disconnected.
+
+Considerations for Publishers
+-----------------------------
+
+.. index:: Method; publish(), publish() method, Store interface, 
+    Method; get_unpersisted_count(), get_unpersisted_count() method,
+
+Publishing with an ``HAClient`` is nearly identical to regular
+publishing; you simply call the ``publish()`` method with your message's
+topic and data. The AMPS client sends the message to AMPS, and then
+returns from the ``publish()`` call. For maximum performance, the client
+does not wait for the AMPS server to acknowledge that the message has
+been received.
+
+When an ``HAClient`` sets a publish store, the publish store retains a
+copy of each outgoing message and requests that AMPS acknowledge that
+the message has been persisted. The AMPS server acknowledges messages
+back to the publisher. Acknowledgments can be delivered for multiple
+messages at periodic intervals (for topics recorded in the transaction
+log) or after each message (for topics that are not recorded in the
+transaction log). When an acknowledgment for a message is received, the
+``HAClient`` removes that message from the bookmark store. When a connection
+to a server is made, the ``HAClient`` automatically determines which
+messages from the publish store (if any) the server has not processed,
+and replays those messages to the server once the connection is
+established.
+
+For reliable publishers, the application must choose how best to handle
+application shutdown. For example, it is possible for the network to
+fail immediately after the publisher sends the message, while the
+message is still in transit. In this case, the publisher has sent the
+message, but the server has not processed it and acknowledged it. During
+normal operation, the ``HAClient`` will automatically connect and retry
+the message. On shutdown, however, the application must decide whether
+to wait for messages to be acknowledged, or whether to exit.
+
+Publish store implementations provide an ``unpersisted_count()`` method
+that reports the number of messages that have not yet been acknowledged
+by the AMPS server. When the ``unpersisted_count()`` reaches ``0``,
+there are no unpersisted messages in the local publish store.
+
+For the highest level of safety, an application can wait until the
+``unpersisted_count()`` reaches ``0``, which indicates that all of the
+messages have been persisted to the instance that the application is
+connected to, and the synchronous replication destinations configured
+for that instance. When a synchronous replication destination goes
+offline, this approach will cause the publisher to wait to exit until
+the destination comes back online or until the destination is downgraded
+to asynchronous replication.
+
+For applications that are shut down periodically for short periods of
+time (for example, applications that are only offline during a weekly
+maintenance window), another approach is to use the ``publish_flush()``
+method to ensure that messages are delivered to AMPS, and then rely on
+the connection logic to replay messages as necessary when the
+application restarts.
+
+For example, the following code flushes messages to AMPS, then warns if
+not all messages have been acknowledged:
+
+.. code:: python
+
+    client = AMPS.HAClient("ha-publisher",
+             "/mnt/fastDisk/moreImportantMessages.outgoing",
+             "/mnt/fastDisk/moreImportantMessages.incoming")
+    ...
+    client.connect_and_logon()
+
+    # Publish messages
+    ...
+
+
+    # We think we are done, but the server may not
+    # have received or acknowledged all messages yet.
+
+    # Wait until the server has received all messages.
+    # The program could also specify a timeout in this
+    # command to avoid blocking forever if the network
+    # is down or all servers are offline.
+
+    client.publish_flush()
+
+    # Print warning to the console if messages have
+    # been published but not yet acknowledged as
+    # persisted
+
+    if (client.get_unpersisted_count() > 0):
+        print( "all messages have been published, " \
+            + " but not all have been persisted" )
+        
+    client.disconnect()
+                        
+**Example 9.4:** *HAPublisher*
+
+.. index:: Method; disconnect(), disconnect() method, Method; publish(), 
+    publish() method, Method; get_unpersisted_count(),
+    get_unpersisted_count() method
+
+In this example, the client sends each
+message immediately when ``publish()`` is called. If AMPS becomes
+unavailable between the final ``publish()`` and the ``disconnect()``, or
+one of the servers that the AMPS instance replicates to is offline, the
+client may not have received a persisted acknowledgment for all of the
+published messages. For example, if a message has not yet been persisted
+by all of the servers in the replication fabric that are connected with
+synchronous replication, AMPS will not have acknowledged the message.
+
+Before shutting down the client, the code does two things:
+
+- First, the code flushes messages to the server to ensure that all 
+  messages have been delivered to AMPS.
+
+- Next, the code checks to see if all of the messages in the publish store
+  have been acknowledged as persisted by AMPS. If the messages have not
+  been acknowledged, they will remain in the publish store file and will
+  be published to AMPS, if necessary, the next time the application
+  connects. An application may choose to loop until
+  ``get_unpersisted_count()`` returns ``0``, or (as we do in this case)
+  simply warn that AMPS has not confirmed that the messages are fully
+  persisted. The behavior you choose in your application should be
+  consistent with the high-availability guarantees your application needs
+  to provide.
+
+.. caution::
+
+   AMPS uses the name of the ``HAClient`` to determine the
+   origin of messages. For the AMPS server to correctly
+   identify duplicate messages, each instance of an
+   application that publishes messages must use a distinct
+   name. That name must be consistent across different runs
+   of the application.
+
+If your application crashes or is terminated, some published messages
+may not have been persisted in the AMPS server. If you use the
+file-based store—in other words, if you provide file names for
+persistent storage when you create the ``HAClient``—the ``HAClient``
+will recover the messages, and once logged on, will correlate the
+message store to what the AMPS server has received, re-publishing any
+missing messages. This occurs automatically when ``HAClient`` connects,
+without any explicit consideration in your code, other than ensuring
+that the same file name is used to create the ``HAClient`` if recovery
+is desired.
+
+.. warning::
+
+    AMPS provides persisted acknowledgment messages for
+    topics that do not have a transaction log enabled.
+    However, the level of durability provided for topics with
+    no transaction log is minimal. Learn more about
+    transaction logs in the *AMPS User Guide*.
+
+
+.. include:: ./detecting-early-failover-python.inc
+
+Considerations for Subscribers
+------------------------------
+
+``HAClient`` provides two important features for applications that
+subscribe to one or more topics: re-subscription, and a bookmark store
+to track the correct point at which to resume a bookmark subscription.
+
+Resubscription with Asynchronous Message Processing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Any asynchronous subscription placed using an ``HAClient`` is
+automatically reinstated after a disconnect or a failover. These
+subscriptions are placed in an in-memory ``SubscriptionManager``, which
+is created automatically when the ``HAClient`` is instantiated.
+
+When a re-subscription occurs, the AMPS Python client re-executes the
+command as originally submitted, including the original topic, options,
+and so on. AMPS sends the subscriber any messages for the specified
+topic (or topic expression) that are published after the subscription is
+placed. For a ``sow_and_subscribe`` command, this means that the client
+re-issues the full command, including the SOW query as well as the
+subscription.
+
+Resubscription with Synchronous Message Processing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``HAClient`` (starting with the AMPS Python client version 4.3.1.1)
+does not track synchronous message processing subscriptions in the
+``SubscriptionManager``. The reason for this is to preserve the iterator
+semantics. That is, once the ``MessageStream`` indicates that there are
+no more elements in the stream, it does not suddenly produce more
+elements.
+
+To re-subscribe when the ``HAClient`` fails over, you can simply re-issue
+the subscription. For example, the snippet below re-issues the subscribe
+command when the message stream ends:
+
+.. code:: python
+
+    while still_need_to_process:
+     # Exiting the for loop is the end of stream.
+     # For a subscribe, this likely means that the
+     # client has disconnected.
+     try:  
+       for message in client.subscribe("messages"):
+         # process messages here
+         # check condition on still_need_to_process
+         if still_need_to_process == False: break
+     except AMPS.DisconnectedException as e:
+         pass
+
+**Example 9.5:** *Resubscription*
+
+Bookmark Stores
+^^^^^^^^^^^^^^^^
+
+In cases where it is critical not to miss a single message, it is
+important to be able to resume a subscription at the exact point that a
+failure occurred. In this case, simply recreating a subscription isn't
+sufficient. Even though the subscription is recreated, the subscriber
+may have been disconnected at precisely the wrong time and will not see
+the message.
+
+To ensure delivery of every message from a topic or set of topics, the
+AMPS ``HAClient`` includes a ``BookmarkStore`` that, combined with the
+bookmark subscription and transaction log functionality in the AMPS
+server, ensures that clients receive any messages that might have been
+missed. The client stores the bookmark associated with each message
+received, and tracks whether the application has processed that message;
+if a disconnect occurs, the client uses the ``BookmarkStore`` to determine
+the correct resubscription point, and sends that bookmark to AMPS when
+it re-subscribes. AMPS then replays messages from its transaction log
+from the point after the specified bookmark, thus ensuring the client is
+completely up-to-date.
+
+``HAClient`` helps you to take advantage of this bookmark mechanism
+through the ``BookmarkStore`` interface and ``bookmarkSubscribe()``
+method on ``Client``. When you create subscriptions with
+``bookmarkSubscribe()``, whenever a disconnection or failover occurs,
+your application automatically re-subscribes to the message after the
+last message it processed. ``HAClients`` created by
+``createFileBacked()`` additionally store these bookmarks on disk, so
+that the application can restart with the appropriate message if the
+client application fails and restarts.
+
+To take advantage of bookmark subscriptions, do the following:
+
+-  Ensure the topic(s) to be subscribed to are included in a transaction
+   log. See the *AMPS User Guide* for information on how to specify the
+   contents of a transaction log.
+
+  .. index:: SubId,
+
+-  Use ``bookmark_subscribe()`` instead of ``subscribe()`` when
+   creating a ``subscription()`` and decide how the application will
+   manage subscription identifiers (SubIds). If you are using a
+   ``Command`` object, you can simply set the bookmark on that object.
+
+  .. index:: Method; discard(), discard() method,
+
+-  Use the ``discard()`` method in message handlers to indicate 
+   when a message has been fully processed by the application,
+   that is, when the application does not need to receive
+   the message again if the application fails over.
+
+The following example creates a bookmark subscription against a
+transaction-logged topic and fully processes each message as soon as it
+is delivered:
+
+.. code:: python
+
+    class MessagePrinter(object):
+        def __init__(self, client):
+            self._client = client
+
+        def __call__(self, message):
+           print (message.get_data())
+           self._client.discard(message)
+
+        
+    ...
+
+    client = AMPS.HAClient(
+        "aClient",
+        "/logs/aClient.publishLog",
+        "/logs/aClient.subscribeLog")
+
+    # Create ServerChooser, populate chooser, connect client
+    ...
+
+    client.execute_async(                        \
+        AMPS.Command("subscribe")                            \
+            .set_topic("myTopic")                            \
+            .set_bookmark(AMPS.Client.Bookmarks.MOST_RECENT) \
+            .set_sub_id("MySubID"),                          \
+            MessagePrinter(client))
+
+**Example 9.6:** *HAClient subscription*
+
+In this example, the client is a file-backed client, meaning that
+arriving bookmarks will be stored in a file (``aClient.subscribeLog``).
+Storing these bookmarks in a file allows the application to restart the
+subscription from the last message processed, in the event of either
+server or client failure.
+
+.. tip::
+
+     For optimum performance, it is critical to discard every 
+     message once its processing is complete. If a message is
+     never discarded, it remains in the bookmark store. During
+     re-subscription, ``HAClient`` always restarts the
+     bookmark subscription with the oldest undiscarded
+     message, and then filters out any more recent messages
+     that have been discarded. If an old message remains in
+     the store, but is no longer important for the
+     application’s functioning, then the client and the AMPS
+     server will incur unnecessary network, disk and CPU
+     activity.
+
+The fourth parameter, ``sub_id``, specifies an identifier to be used for
+this subscription. Passing ``None`` causes ``HAClient`` to generate one
+and return it, like most other ``Client`` functions. However, if you
+wish to resume a subscription from a previous point after the
+application has terminated and restarted, the application must pass the
+same subscription ID as during its previous run. Passing a different
+subscription ID bypasses any recovery mechanisms, creating an entirely
+new subscription. When you use an existing subscription ID, the
+``HAClient`` locates the last-used bookmark for that subscription in the
+local store, and attempts to re-subscribe from that point.
+
+Below are the different bookmark types that can be used to enable different 
+recovery strategies for an application:
+
+.. index:: Method; subscribe(), subscribe() method, Parameters; NOW,
+   NOW parameter
+
+-  ``Client.Bookmarks.NOW`` specifies that the subscription
+   should begin from the moment the server receives the subscription
+   request. This results in the same messages being delivered as if you
+   had invoked ``subscribe()`` instead, except that the messages will be
+   accompanied by bookmarks. This is also the behavior that results if
+   you supply an invalid bookmark.
+
+.. index:: Parameters; EPOCH, EPOCH parameter,
+
+-  ``Client.Bookmarks.EPOCH``
+   specifies that the subscription should begin from the beginning of
+   the AMPS transaction log (that is, the first entry in the oldest
+   journal file for the transaction log).
+
+  .. index:: Parameters; MOST_RECENT, MOST_RECENT parameter,
+
+-  ``Client.Bookmarks.MOST_RECENT`` specifies that the
+   subscription should begin from the last-used message in the
+   associated ``BookmarkStore``. Alternatively, if this subscription has
+   not been seen before, it instructs the subscription to begin with
+   ``EPOCH``. This is the most common value for this parameter and is
+   the value used in the preceding example. By using ``MOST_RECENT``,
+   the application automatically resumes from wherever the subscription
+   left off, taking into account any messages that have already been
+   processed and discarded.
+
+When the ``HAClient`` re-subscribes after a disconnection and
+reconnection, it always uses ``MOST_RECENT``, ensuring that the
+continued subscription always begins from the last message used before
+the disconnect, so that no messages are missed.
+
+Conclusion
+----------
+
+With only a few changes, most AMPS applications can take advantage of
+the ``HAClient`` and associated classes to become more highly-available
+and resilient. Using the ``PublishStore``, publishers can ensure that
+every message published has actually been persisted by AMPS. Using
+``BookmarkStore``, subscribers can make sure that there are no gaps or
+duplicates in the messages received. ``HAClient`` makes both kinds of
+applications more resilient to network and server outages, as well as temporary
+issues. By utilizing the file based ``HAClient``, clients can recover
+their state after an unexpected termination or crash. Though
+``HAClient`` provides useful defaults for the ``Store``,
+``BookmarkStore``, and ``ServerChooser``, you
+can customize any or all of these to the specific needs of your
+application and architecture.