PyPI - amps-python-client - Versions diffs - 5.3.4.1__zip → 5.3.4.3__zip - Mend

amps-python-client 5.3.4.1zip → 5.3.4.3zip

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (918) hide show

amps-python-client-5.3.4.3/docs/python/html/chapters/high-availability.html ADDED Viewed

@@ -0,0 +1,1017 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    <title>9. High Availability &#8212; Python Developer Guide 5.3.4.0
+ documentation</title>
+    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '5.3.4.0',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="index" title="Index" href="../genindex.html" />
+    <link rel="search" title="Search" href="../search.html" />
+    <link rel="next" title="10. AMPS Programming: Working with Commands" href="creating-commands.html" />
+    <link rel="prev" title="8. Delta Publish and Subscribe" href="deltas.html" />
+  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
+  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
+  </head>
+  <body role="document">
+  <div class="document">
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper">
+            <p class="logo"><a href="../index.html">
+              <img class="logo" src="../_static/flag_logo.png" alt="Logo"/>
+            </a></p>
+  <h3><a href="../index.html">Table Of Contents</a></h3>
+  <ul>
+<li><a class="reference internal" href="#">9. High Availability</a><ul>
+<li><a class="reference internal" href="#overview-of-haclient">Overview of HAClient</a></li>
+<li><a class="reference internal" href="#reconnection-with-haclient">Reconnection with HAClient</a></li>
+<li><a class="reference internal" href="#choosing-store-durability">Choosing Store Durability</a><ul>
+<li><a class="reference internal" href="#using-the-sow-recovery-point-adapter">Using the SOW Recovery Point Adapter</a><ul>
+<li><a class="reference internal" href="#amps-topic-configuration">AMPS Topic Configuration</a></li>
+<li><a class="reference internal" href="#constructing-a-client-for-the-adapter">Constructing a Client for the Adapter</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#capacity-planning-and-store-sizing">Capacity Planning and Store Sizing</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#connections-and-the-server-chooser">Connections and the Server Chooser</a><ul>
+<li><a class="reference internal" href="#setting-a-reconnect-delay-and-timeout">Setting a Reconnect Delay and Timeout</a></li>
+<li><a class="reference internal" href="#implementing-a-server-chooser">Implementing a Server Chooser</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#heartbeats-and-failure-detection">Heartbeats and Failure Detection</a></li>
+<li><a class="reference internal" href="#considerations-for-publishers">Considerations for Publishers</a></li>
+<li><a class="reference internal" href="#detecting-failover-ahead-of-replication">Detecting Failover Ahead of Replication</a></li>
+<li><a class="reference internal" href="#considerations-for-subscribers">Considerations for Subscribers</a><ul>
+<li><a class="reference internal" href="#resubscription-with-asynchronous-message-processing">Resubscription with Asynchronous Message Processing</a></li>
+<li><a class="reference internal" href="#resubscription-with-synchronous-message-processing">Resubscription with Synchronous Message Processing</a></li>
+<li><a class="reference internal" href="#bookmark-stores">Bookmark Stores</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#conclusion">Conclusion</a></li>
+</ul>
+</li>
+</ul>
+<div class="relations">
+<h3>Related Topics</h3>
+<ul>
+  <li><a href="../index.html">Documentation overview</a><ul>
+      <li>Previous: <a href="deltas.html" title="previous chapter">8. Delta Publish and Subscribe</a></li>
+      <li>Next: <a href="creating-commands.html" title="next chapter">10. AMPS Programming: Working with Commands</a></li>
+  </ul></li>
+</ul>
+</div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+  <div class="section" id="high-availability">
+<span id="index-0"></span><h1>9. High Availability<a class="headerlink" href="#high-availability" title="Permalink to this headline">¶</a></h1>
+<p>The AMPS Python Client provides an easy way to create highly-available
+applications using AMPS, via the <code class="docutils literal"><span class="pre">HAClient</span></code> class. <code class="docutils literal"><span class="pre">HAClient</span></code>
+derives from <code class="docutils literal"><span class="pre">Client</span></code> and offers the same methods, but also adds
+protection against network, server, and client outages.</p>
+<p>Using <code class="docutils literal"><span class="pre">HAClient</span></code> allows applications to automatically:</p>
+<ul class="simple">
+<li>Recover from temporary disconnects between client and server.</li>
+<li>Failover from one server to another when a server becomes
+unavailable.</li>
+</ul>
+<p>Since the <code class="docutils literal"><span class="pre">HAClient</span></code> automatically manages failover and
+reconnection, 60East recommends using the <code class="docutils literal"><span class="pre">HAClient</span></code> for applications
+that need to:</p>
+<ul class="simple">
+<li>Automatically reconnect and resume work in the case of disconnection.</li>
+<li>Ensure no messages are lost or duplicated after a reconnect or
+failover.</li>
+<li>Persist messages and bookmarks on disk for protection against client
+failure.</li>
+</ul>
+<p>You can choose how your application uses <code class="docutils literal"><span class="pre">HAClient</span></code> features. For
+example, you might need automatic reconnection, but have no need to
+resume subscriptions or republish messages. The high availability
+behavior in <code class="docutils literal"><span class="pre">HAClient</span></code> is provided by implementations of defined
+interfaces. You can combine different implementations provided by 60East
+to meet your needs, and implement those interfaces to provide your own
+policies.</p>
+<p>Some of these features require specific configuration settings on your
+AMPS instance(s). This chapter mentions these features and describes how
+to use them from the AMPS Java client. You can find full documentation
+for these settings and server features in the <em>AMPS User Guide</em>.</p>
+<div class="section" id="overview-of-haclient">
+<h2>Overview of HAClient<a class="headerlink" href="#overview-of-haclient" title="Permalink to this headline">¶</a></h2>
+<p><code class="docutils literal"><span class="pre">HAClient</span></code> derives from <code class="docutils literal"><span class="pre">Client</span></code> and offers the same methods for
+sending commands to AMPS and receiving messages from AMPS.</p>
+<p>The <code class="docutils literal"><span class="pre">HAClient</span></code> differs from the <code class="docutils literal"><span class="pre">Client</span></code> in two ways:</p>
+<ul class="simple">
+<li>The <code class="docutils literal"><span class="pre">HAClient</span></code> automatically installs a disconnect handler that
+reconnects to AMPS and resumes active (asynchronous) subscriptions.
+The disconnect handler optionally replays <code class="docutils literal"><span class="pre">publish</span></code> and <code class="docutils literal"><span class="pre">sow_delete</span></code>
+messages that have not been acknowledged by AMPS, using a
+<code class="docutils literal"><span class="pre">PublishStore</span></code>.  The disconnect handler can optionally resume
+replays from the transaction log at a point that guarantees
+no messages are skipped and no duplicates are delivered to the
+application, using a <code class="docutils literal"><span class="pre">BookmarkStore</span></code>.</li>
+<li>The <code class="docutils literal"><span class="pre">HAClient</span></code> includes the infrastructure needed for
+client failover, including a list of connection strings
+and their associated authentication mechanisms (provided by
+the <code class="docutils literal"><span class="pre">ServerChooser</span></code>), and options for controlling backoff
+behavior for reconnects (provided by the <code class="docutils literal"><span class="pre">DelayStrategy</span></code>).
+As a result, the <code class="docutils literal"><span class="pre">HAClient</span></code> provides a <code class="docutils literal"><span class="pre">connect_and_logon()</span></code>
+function for establishing a connection to AMPS, rather than
+treating these as independent steps that an application must
+manage itself.</li>
+</ul>
+<p>If your application needs to automatically reconnect to AMPS,
+60East recommends using the <code class="docutils literal"><span class="pre">HAClient</span></code> and the automatically
+provided disconnect handler rather than using a <code class="docutils literal"><span class="pre">Client</span></code>
+or replacing the <code class="docutils literal"><span class="pre">HAClient</span></code> default disconnect handler.</p>
+</div>
+<div class="section" id="reconnection-with-haclient">
+<h2>Reconnection with HAClient<a class="headerlink" href="#reconnection-with-haclient" title="Permalink to this headline">¶</a></h2>
+<p>The most important difference between <code class="docutils literal"><span class="pre">Client</span></code> and <code class="docutils literal"><span class="pre">HAClient</span></code> is
+that <code class="docutils literal"><span class="pre">HAClient</span></code> automatically provides a reconnect handler.</p>
+<p>This description provides a high-level framework for understanding the
+components involved in failover with the <code class="docutils literal"><span class="pre">HAClient</span></code>. The components
+are described in more detail in the following sections.</p>
+<p>The <code class="docutils literal"><span class="pre">HAClient</span></code> reconnect handler performs the following steps when
+reconnecting:</p>
+<ol class="arabic">
+<li><p class="first">Calls the <code class="docutils literal"><span class="pre">ServerChooser</span></code> to determine the next URI to connect to
+and the authenticator to use for that connection.</p>
+<p>If the connection fails, calls <code class="docutils literal"><span class="pre">get_error</span></code> on the <code class="docutils literal"><span class="pre">ServerChooser</span></code>
+to get a description of the failure, sends an exception to the
+exception listener and stops the reconnection process.</p>
+</li>
+<li><p class="first">Calls the <code class="docutils literal"><span class="pre">DelayStrategy</span></code> to determine how long to wait before
+attempting to reconnect and waits for that period of time.</p>
+</li>
+<li><p class="first">Connects to the AMPS server. If the connection fails, calls
+<code class="docutils literal"><span class="pre">report_failure</span></code> on the <code class="docutils literal"><span class="pre">ServerChooser</span></code> and begins the process
+again.</p>
+</li>
+<li><p class="first">Logs on to the AMPS server. If the connection fails, calls
+<code class="docutils literal"><span class="pre">report_failure</span></code> on the <code class="docutils literal"><span class="pre">ServerChooser</span></code> and begins the process
+again.</p>
+</li>
+<li><p class="first">Calls <code class="docutils literal"><span class="pre">report_success</span></code> on the <code class="docutils literal"><span class="pre">ServerChooser</span></code>.</p>
+</li>
+<li><p class="first">Receives the bookmark for the last message that the server has
+persisted. Discards any older messages from the <code class="docutils literal"><span class="pre">PublishStore</span></code>.</p>
+</li>
+<li><p class="first">Republishes any messages in the <code class="docutils literal"><span class="pre">PublishStore</span></code> that have not been
+persisted by the server.</p>
+</li>
+<li><p class="first">Re-establishes subscriptions using the <code class="docutils literal"><span class="pre">SubscriptionManager</span></code> for
+the client. For bookmark subscriptions, the reconnect handler uses
+the <code class="docutils literal"><span class="pre">BookmarkStore</span></code> for the client to determine the most recent
+bookmark, and re-subscribes with that bookmark. For subscriptions that
+do not use a bookmark, the <code class="docutils literal"><span class="pre">SubscriptionManager</span></code> simply re-enters
+the subscription, meaning that it is entered at the point at which
+the <code class="docutils literal"><span class="pre">HAClient</span></code> reconnects.</p>
+</li>
+</ol>
+<p>The <code class="docutils literal"><span class="pre">ServerChooser</span></code>, <code class="docutils literal"><span class="pre">DelayStrategy</span></code>, <code class="docutils literal"><span class="pre">PublishStore</span></code>,
+and <code class="docutils literal"><span class="pre">BookmarkStore</span></code> are all extension points
+for the <code class="docutils literal"><span class="pre">HAClient</span></code>. You can adapt the failover and recovery behavior
+by setting a different object for the behavior you want to customize on
+the <code class="docutils literal"><span class="pre">HAClient</span></code> or by providing your own implementation.</p>
+<p>For example, the convenience methods in the previous section customize
+the behavior of the <code class="docutils literal"><span class="pre">PublishStore</span></code> and <code class="docutils literal"><span class="pre">BookmarkStore</span></code> by providing
+either memory-backed or file-backed stores.</p>
+<p>The reconnection process runs on the thread that discovers the
+disconnection. This means that, in the event that an application
+thread discovers the disconnection as a result of a call to
+the Python AMPS client, that call may not return until a
+connection is re-established (or until the server chooser
+indicates failure, in which case the application will
+receive an exception).</p>
+<p>The Python client includes a <code class="docutils literal"><span class="pre">retry_on_disconnect</span></code> setting that
+controls this retry behavior when the client is disconnected. When
+set to <code class="docutils literal"><span class="pre">True</span></code> (the default), any call to the Client that results
+in a command being sent to AMPS may block until a connection is
+re-established.  When set to <code class="docutils literal"><span class="pre">False</span></code>, the <code class="docutils literal"><span class="pre">HAClient</span></code> will
+retry the connection a single time and throw an exception
+if the connection cannot be re-established.</p>
+<p>Regardless of the <code class="docutils literal"><span class="pre">retry_on_disconnect</span></code> setting, a call
+to <code class="docutils literal"><span class="pre">publish</span></code> will result in the message being stored in
+the <code class="docutils literal"><span class="pre">PublishStore</span></code> for the client if one is set.</p>
+</div>
+<div class="section" id="choosing-store-durability">
+<h2>Choosing Store Durability<a class="headerlink" href="#choosing-store-durability" title="Permalink to this headline">¶</a></h2>
+<p id="index-1">If your application needs reliable publish to AMPS, install a
+<code class="docutils literal"><span class="pre">PublishStore</span></code> in the <code class="docutils literal"><span class="pre">HAClient</span></code>. If your application needs to
+resume replays from the transaction log, install a <code class="docutils literal"><span class="pre">BookmarkStore</span></code> in
+the <code class="docutils literal"><span class="pre">HAClient</span></code>.</p>
+<p>These stores provide the following capabilities:</p>
+<ul class="simple">
+<li>A <em>bookmark store</em> tracks received messages and is used to resume
+subscriptions that replay from the transaction log.</li>
+<li>A <em>publish store</em> tracks published messages and is used to ensure that
+messages are persisted in AMPS.</li>
+</ul>
+<p>The AMPS client provides a memory-backed version of each store and a
+file-backed version of each store. The store interface is public, and an
+application can create and provide a custom store as necessary. An
+<code class="docutils literal"><span class="pre">HAClient</span></code> can use either a memory backed store or a file backed store
+for protection. Each method provides resilience to different failures, as
+described below:</p>
+<ul>
+<li><p class="first"><em>Memory-backed stores</em> provide recovery disconnection from AMPS by
+storing messages and bookmarks in your process&#8217; address space. This
+is the highest performance option for working with AMPS in a highly
+available manner. The trade-off with this method is there is no
+protection from a crash or failure of your client application. If
+your application is terminated prematurely or, if the application
+terminates at the same time as an AMPS instance failure or network
+outage, then messages may be lost or duplicated. The state of
+bookmark replays will be lost when the application shuts down.
+Messages in the publish store when the application shuts down
+will not be maintained through a restart, so the application will
+not be able to attempt any necessary redelivery when the application restarts.</p>
+<p>A memory-backed store should only be used by one instance of a client at
+a time.</p>
+</li>
+<li><p class="first"><em>File-backed stores</em> provide recovery after client failure and
+disconnection from AMPS by storing messages and bookmarks on disk. To
+use this protection method, the <code class="docutils literal"><span class="pre">create_file_backed</span></code> method
+requests additional arguments for the two files that will be used for
+both bookmark storage and message storage. If these files exist and
+are non-empty (as they would be after a client application is
+restarted), the <code class="docutils literal"><span class="pre">HAClient</span></code> loads their contents and ensures
+synchronization with the AMPS server once connected. The performance
+of this option depends heavily on the speed of the device on which
+these files are placed. When the files do not exist (as they would
+the first time a client starts on a given system), the <code class="docutils literal"><span class="pre">HAClient</span></code>
+creates and initializes the files, and in this case the client does
+not have a point at which to resume the subscription or messages to
+republish.</p>
+<p>A store file should only be used by one instance of a client at
+a time.</p>
+<p>When using file-backed stores, 60East recommends periodically removing
+unneeded entries by calling the <code class="docutils literal"><span class="pre">prune()</span></code> method. The precise strategy
+that your application uses to call <code class="docutils literal"><span class="pre">prune()</span></code> depends on the nature of the
+application. Most applications call <code class="docutils literal"><span class="pre">prune()</span></code> when the application exits.</p>
+<p>There are two basic strategies that applications follow while the
+application runs:</p>
+<ul class="simple">
+<li>Install a resize handler and call <code class="docutils literal"><span class="pre">prune()</span></code> after a specified number
+of resize operations or when the store reaches a specific size.</li>
+<li>Call <code class="docutils literal"><span class="pre">prune()</span></code> after a specific number of messages are processed (for
+example, every 10,000 messages received or every 1,000 updates completed).</li>
+</ul>
+<p>Regardless of the strategy, it is best to call <code class="docutils literal"><span class="pre">prune()</span></code> when the application
+is idle, since the <code class="docutils literal"><span class="pre">prune()</span></code> call rewrites the log file.</p>
+</li>
+</ul>
+<p>The store interface is public and an application can create and provide
+a custom store as necessary. While clients provide convenience methods
+for creating file-backed and memory-backed <code class="docutils literal"><span class="pre">HAClient</span></code> objects with the
+appropriate stores, you can also create and set the stores in your
+application code. For the AMPS Python client, stores are implemented in
+C++. You can implement stores using C++, and use the technique described
+in Chapter 12 of this guide - <em>Using the C++ Client</em>, to set the store on
+the client.</p>
+<p>Starting in 5.3.2.0, the underlying AMPS client contains a recovery point
+adapter interface to make it easier to add a custom persistence layer
+to a bookmark store. The distribution includes a recovery point adapter
+that can store bookmark recovery information in an AMPS SOW topic.</p>
+<p>The <code class="docutils literal"><span class="pre">HAClient</span></code> provides convenience methods for creating clients and
+setting stores. You can also construct an <code class="docutils literal"><span class="pre">HAClient</span></code> and set the store
+implementations you choose.</p>
+<p>In this example, we create several clients. The first client uses memory
+stores for both bookmarks and publishes. The second client uses files
+for both bookmarks and publishes. The third client uses a file for
+bookmarks. The third client does not set a store for publishes, which
+means that AMPS provides the default store (and no outgoing messages are
+stored). The final client does not specify any stores, so has no
+persistence for published messages or bookmark subscriptions, but can
+take advantage of the automatic failover and reconnection in the
+<code class="docutils literal"><span class="pre">HAClient</span></code>.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Memory publish store, memory bookmark store</span>
+<span class="n">memoryClient</span> <span class="o">=</span> <span class="n">AMPS</span><span class="o">.</span><span class="n">HAClient</span><span class="p">(</span><span class="s2">&quot;lessImportantMessages&quot;</span><span class="p">)</span>
+<span class="c1"># File-backed publish store, file-backed bookmark store</span>
+<span class="n">diskClient</span> <span class="o">=</span> <span class="n">AMPS</span><span class="o">.</span><span class="n">HAClient</span><span class="p">(</span><span class="s2">&quot;moreImportantMessages&quot;</span><span class="p">,</span>
+          <span class="s2">&quot;/mnt/fastDisk/moreImportantMessages.outgoing&quot;</span><span class="p">,</span>
+          <span class="s2">&quot;/mnt/fastDisk/moreImportantMessages.incoming&quot;</span><span class="p">)</span>
+<span class="c1"># No-op publish store, file-backed bookmark store</span>
+<span class="n">subscriberClient</span> <span class="o">=</span> <span class="n">AMPS</span><span class="o">.</span><span class="n">HAClient</span><span class="p">(</span><span class="s2">&quot;subscriber&quot;</span><span class="p">,</span> <span class="n">no_store</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
+<span class="n">subscriberClient</span><span class="o">.</span><span class="n">set_bookmark_store</span><span class="p">(</span>     \
+            <span class="n">AMPS</span><span class="o">.</span><span class="n">MMapBookmarkStore</span><span class="p">(</span><span class="s2">&quot;/mnt/fastdisk/bookmark.store&quot;</span><span class="p">))</span>
+<span class="c1"># No-op publish store, no-op bookmark store</span>
+<span class="c1"># Failover behavior only.</span>
+<span class="n">streamReader</span> <span class="o">=</span> <span class="n">AMPS</span><span class="o">.</span><span class="n">HAClient</span><span class="p">(</span><span class="s2">&quot;streamReader&quot;</span><span class="p">,</span><span class="n">no_store</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
+</pre></div>
+</div>
+<p><strong>Example 9.1:</strong> <em>HAClient creation example</em></p>
+<div class="section" id="using-the-sow-recovery-point-adapter">
+<h3>Using the SOW Recovery Point Adapter<a class="headerlink" href="#using-the-sow-recovery-point-adapter" title="Permalink to this headline">¶</a></h3>
+<p>The AMPS client also includes the ability to use a SOW topic to store bookmark
+state for a bookmark store. This can be a useful option in a situation
+where an application needs a persistent bookmark store, but does not have
+the ability to store a file on the filesystem, or where an application
+has a bookmark file, but wants to have the ability to resume the subscription
+if the file is lost or damaged, or if the application is started on a
+system that does not have access to the file.</p>
+<p>To use the SOW topic recovery point adapter, you create a bookmark store of
+the type you would like to use for the <code class="docutils literal"><span class="pre">Client</span></code>, passing an
+adapter when you construct the store.  You then set this bookmark store as
+the store for the <code class="docutils literal"><span class="pre">Client</span></code> to use. The constructor for the
+SOW recovery adapter allows you to customize the topic name and
+field names used to store the recovery point information in AMPS.
+As with the <code class="docutils literal"><span class="pre">RecoveryPointAdapter</span></code> interface
+in general, it is possible to customize the behavior of the SOW recovery
+point adapter by overriding the provided methods.</p>
+<p>This section describes how to use the adapter with the default settings.
+Should you need to change the behavior of the class, you would adjust
+the guidance in this section accordingly. (For example, if you override
+methods to produce a message with a different set of keys or
+a different message format, you would update the topic definition
+accordingly).</p>
+<div class="section" id="amps-topic-configuration">
+<h4>AMPS Topic Configuration<a class="headerlink" href="#amps-topic-configuration" title="Permalink to this headline">¶</a></h4>
+<p>To store recovery point state in AMPS, the AMPS instance
+that will store the recovery point state must define a <code class="docutils literal"><span class="pre">SOW/Topic</span></code>
+to hold the recovery point data.</p>
+<p>By default, the adapter uses a topic named <code class="docutils literal"><span class="pre">/ADMIN/bookmark_store</span></code> of
+<code class="docutils literal"><span class="pre">json</span></code> message type, with the <code class="docutils literal"><span class="pre">/clientName</span></code> and <code class="docutils literal"><span class="pre">/subId</span></code> fields
+as keys, similar to the following definition:</p>
+<div class="highlight-xml"><div class="highlight"><pre><span></span><span class="nt">&lt;Topic&gt;</span>
+   <span class="nt">&lt;Name&gt;</span>/ADMIN/bookmark_store<span class="nt">&lt;/Name&gt;</span>
+   <span class="nt">&lt;MessageType&gt;</span>json<span class="nt">&lt;/MessageType&gt;</span>
+   <span class="nt">&lt;Key&gt;</span>/clientName<span class="nt">&lt;/Key&gt;</span>
+   <span class="nt">&lt;Key&gt;</span>/subId<span class="nt">&lt;/Key&gt;</span>
+   <span class="c">&lt;!-- Storage/persistence configuration here.</span>
+<span class="c">        In most cases, this topic should be</span>
+<span class="c">        persisted to a file, but that is not</span>
+<span class="c">        a requirement.  --&gt;</span>
+<span class="nt">&lt;/Topic&gt;</span>
+</pre></div>
+</div>
+<p>You must include this definition, or an equivalent definition,
+in the configuration file for the AMPS instance that will host
+the recovery point.</p>
+<p>If you define a topic with a different configuration (for
+example, different key names, a different topic name or a
+different message type), you must ensure that the
+adapter that you create uses the same parameters as those
+configured on the server.</p>
+</div>
+<div class="section" id="constructing-a-client-for-the-adapter">
+<h4>Constructing a Client for the Adapter<a class="headerlink" href="#constructing-a-client-for-the-adapter" title="Permalink to this headline">¶</a></h4>
+<p>The AMPS SOW Recovery Point Adapter requires a <code class="docutils literal"><span class="pre">Client</span></code> or <code class="docutils literal"><span class="pre">HAClient</span></code>
+connected to the instance that contains the SOW topic. The Adapter will
+use this client to recover bookmark state and store bookmarks in AMPS.
+Notice that this client <strong>must not</strong> be a client that the Adapter is
+keeping state for. This must be a completely separate client instance,
+otherwise the client may deadlock while updating the store.</p>
+<p>The client must be connected and logged in to the instance that
+contains the SOW topic, using the message type defined for the topic.</p>
+</div>
+</div>
+<div class="section" id="capacity-planning-and-store-sizing">
+<h3>Capacity Planning and Store Sizing<a class="headerlink" href="#capacity-planning-and-store-sizing" title="Permalink to this headline">¶</a></h3>
+<p>When an application uses a file-backed store, it is important to make
+sure that there is enough space available on the file system to
+be able to manage the store.</p>
+<p>For logged bookmark stores, an application needs to keep a bookmark record for
+each message received, each message discarded, and the persisted
+acknowledgments delivered by the server approximately once a second.
+Each bookmark entry consumes roughly 70 bytes of storage <em>plus</em> the length
+of the subscription ID for the subscription receiving the message. The logged
+bookmark store retains entries until an application explicitly calls
+<code class="docutils literal"><span class="pre">prune()</span></code>. The capacity needed for a logged bookmark store will
+depend on the strategy that the application uses for pruning the file.</p>
+<p>For a file-backed publish store, the application needs to be able to
+store published messages until the AMPS server that the publisher is
+connected to acknowledges those messages as persisted. The volume of
+messages that needs to be stored depends on the failover policy for
+the server &#8211; that is, the maximum amount of time that the server will
+allow a downstream instance to fail to acknowledge a message before
+the server downgrades that connection to <code class="docutils literal"><span class="pre">async</span></code> acknowledgment.
+By default, AMPS does not downgrade connections: this policy must
+be set explicitly using the AMPS actions. As an example, if the
+server is configured to downgrade connections that are more than
+120 seconds behind, then &#8211; for disaster recovery &#8211; the application
+must have the capacity to store 120 seconds of published messages
+at peak publishing load. However, unlike the logged bookmark store, a
+file-backed publish store removes messages from the store and reuses
+the space once AMPS has acknowledged the message.</p>
+</div>
+</div>
+<div class="section" id="connections-and-the-server-chooser">
+<h2>Connections and the Server Chooser<a class="headerlink" href="#connections-and-the-server-chooser" title="Permalink to this headline">¶</a></h2>
+<p id="index-2">Unlike <code class="docutils literal"><span class="pre">Client</span></code>, the <code class="docutils literal"><span class="pre">HAClient</span></code> attempts to keep itself connected to
+an AMPS instance at all times, by automatically reconnecting or failing
+over when it detects that the client is disconnected. When you are using
+the <code class="docutils literal"><span class="pre">Client</span></code> directly, your disconnect handler usually takes care of
+reconnection. <code class="docutils literal"><span class="pre">HAClient</span></code>, on the other hand, provides a disconnect
+handler that automatically reconnects to the current server or to the
+next available server.</p>
+<p>To inform the <code class="docutils literal"><span class="pre">HAClient</span></code> of the addresses of the AMPS instances in
+your system, you pass a <code class="docutils literal"><span class="pre">ServerChooser</span></code> instance to the <code class="docutils literal"><span class="pre">HAClient</span></code>.
+<code class="docutils literal"><span class="pre">ServerChooser</span></code> acts as a smart enumerator over the servers available:
+<code class="docutils literal"><span class="pre">HAClient</span></code> calls <code class="docutils literal"><span class="pre">ServerChooser</span></code> methods to inquire about what
+server should be connected, and calls methods to indicate whether a
+given server succeeded or failed.</p>
+<p id="index-3">The AMPS Python client provides a simple implementation of
+<code class="docutils literal"><span class="pre">ServerChooser</span></code>, called <code class="docutils literal"><span class="pre">DefaultServerChooser</span></code>, that provides very
+simple logic for reconnecting. This server chooser is most suitable for
+basic testing, or in cases where an application should simply rotate
+through a list of servers. For most applications, you implement the
+<code class="docutils literal"><span class="pre">ServerChooser</span></code> interface yourself for more advanced logic, such as
+choosing a backup server based on your network topology, or limiting the
+number of times your application should try to reconnect to a given
+address.</p>
+<p>To connect to AMPS, you provide a <code class="docutils literal"><span class="pre">ServerChooser</span></code> to <code class="docutils literal"><span class="pre">HAClient</span></code> and
+then call <code class="docutils literal"><span class="pre">connect_and_logon()</span></code> to create the first connection:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">memoryClient</span> <span class="o">=</span> <span class="n">AMPS</span><span class="o">.</span><span class="n">HAClient</span><span class="p">(</span><span class="s2">&quot;myClient&quot;</span><span class="p">)</span>
+<span class="c1"># primary.amps.xyz.com is the primary AMPS instance, and</span>
+<span class="c1"># secondary.amps.xyz.com is the secondary</span>
+<span class="n">chooser</span> <span class="o">=</span> <span class="n">AMPS</span><span class="o">.</span><span class="n">DefaultServerChooser</span><span class="p">()</span>
+<span class="n">chooser</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s2">&quot;tcp://primary.amps.xyz.com:12345/fix&quot;</span><span class="p">)</span>
+<span class="n">chooser</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s2">&quot;tcp://secondary.amps.xyz.com:12345/fix&quot;</span><span class="p">)</span>
+<span class="n">memoryClient</span><span class="o">.</span><span class="n">set_server_chooser</span><span class="p">(</span><span class="n">chooser</span><span class="p">)</span>
+<span class="n">memoryClient</span><span class="o">.</span><span class="n">connect_and_logon</span><span class="p">()</span>
+<span class="o">...</span>
+<span class="n">myClient</span><span class="o">.</span><span class="n">disconnect</span><span class="p">()</span>
+</pre></div>
+</div>
+<p><strong>Example 9.2:</strong> <em>Multiple HAClient creation example</em></p>
+<p id="index-4">Similar to <code class="docutils literal"><span class="pre">Client</span></code>, <code class="docutils literal"><span class="pre">HAClient</span></code> remains connected to the server until
+<code class="docutils literal"><span class="pre">disconnect()</span></code> is called. Unlike <code class="docutils literal"><span class="pre">Client</span></code>, <code class="docutils literal"><span class="pre">HAClient</span></code>
+automatically attempts to reconnect to your server if it detects a
+disconnect and if that server cannot be connected, fails over to the
+next server provided by the <code class="docutils literal"><span class="pre">ServerChooser</span></code>. In this example, the call
+to <code class="docutils literal"><span class="pre">connect_and_logon()</span></code> attempts to connect and login to
+<code class="docutils literal"><span class="pre">primary.amps.xyz.com</span></code>, and returns if that is successful. If it
+cannot connect, it tries <code class="docutils literal"><span class="pre">secondary.amps.xyz.com</span></code>, and continues
+trying servers from the <code class="docutils literal"><span class="pre">ServerChooser</span></code> until a connection is
+established. Likewise, if it detects a disconnection while the client is
+in use, then <code class="docutils literal"><span class="pre">HAClient</span></code> attempts to reconnect to the server with which
+it was most recently connected; if that is not possible, then it moves
+on to the next server provided by the <code class="docutils literal"><span class="pre">ServerChooser</span></code>.</p>
+<p>The default <code class="docutils literal"><span class="pre">ServerChooser</span></code> simply provides the next URL in the
+sequence. This strategy works for many applications. If you need a
+different strategy, you can implement your own logic for failover by
+creating a class derived from <code class="docutils literal"><span class="pre">ServerChooser</span></code>.</p>
+<div class="section" id="setting-a-reconnect-delay-and-timeout">
+<h3>Setting a Reconnect Delay and Timeout<a class="headerlink" href="#setting-a-reconnect-delay-and-timeout" title="Permalink to this headline">¶</a></h3>
+<p id="index-5">You can control the amount of time between reconnection attempts and
+set a total amount of time for the <code class="docutils literal"><span class="pre">HAClient</span></code> to attempt to reconnect.</p>
+<p>The AMPS Python client includes a method for setting a delay strategy on
+a client, <code class="docutils literal"><span class="pre">set_reconnect_delay_strategy</span></code>. This method accepts an
+instance of any type that provides the methods
+<code class="docutils literal"><span class="pre">get_connect_wait_duration</span></code> and <code class="docutils literal"><span class="pre">reset</span></code>, as described in the API
+documentation.</p>
+<p>While you can easily implement your own delay strategy, the client also
+provides two delay strategies:</p>
+<ul class="simple">
+<li><code class="docutils literal"><span class="pre">FixedDelayStrategy</span></code> provides the same delay each time the
+<code class="docutils literal"><span class="pre">HAClient</span></code> tries to reconnect.</li>
+<li><code class="docutils literal"><span class="pre">ExponentialDelayStrategy</span></code> provides an exponential backoff until a
+connection attempt succeeds.</li>
+</ul>
+<p>To use either of these classes, you simply create an instance, set the
+appropriate parameters, and install that instance as the delay strategy
+for the <code class="docutils literal"><span class="pre">HAClient</span></code>. For example, the following code sets up a
+reconnect delay that starts at 200ms and increases the delay by 1.5
+times after each failure. The strategy allows a maximum delay between
+connection attempts of 5 seconds, and will not retry longer than 60
+seconds.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">theClient</span> <span class="o">=</span> <span class="n">AMPS</span><span class="o">.</span><span class="n">HAClient</span><span class="p">(</span><span class="s2">&quot;myClient&quot;</span><span class="p">)</span>
+<span class="n">theClient</span><span class="o">.</span><span class="n">set_reconnect_delay_strategy</span><span class="p">(</span>      \
+   <span class="n">AMPS</span><span class="o">.</span><span class="n">ExponentialDelayStrategy</span><span class="p">(</span>            \
+                   <span class="n">initial_delay</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span>        \
+                   <span class="n">backoff_exponent</span><span class="o">=</span><span class="mf">1.5</span><span class="p">,</span>     \
+                   <span class="n">maximum_delay</span><span class="o">=</span><span class="mi">5000</span><span class="p">,</span>       \
+                   <span class="n">maximum_retry_time</span><span class="o">=</span><span class="mi">60000</span><span class="p">)</span> \
+                 <span class="p">)</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="implementing-a-server-chooser">
+<h3>Implementing a Server Chooser<a class="headerlink" href="#implementing-a-server-chooser" title="Permalink to this headline">¶</a></h3>
+<p id="index-6">As described above, you provide the <code class="docutils literal"><span class="pre">HAClient</span></code>
+with connection strings to one or more AMPS servers using a
+<code class="docutils literal"><span class="pre">ServerChooser</span></code>. The purpose of a <code class="docutils literal"><span class="pre">ServerChooser</span></code> is to provide
+information to the <code class="docutils literal"><span class="pre">HAClient</span></code>. A <code class="docutils literal"><span class="pre">ServerChooser</span></code> does not manage the
+reconnection process, and should not call methods on the <code class="docutils literal"><span class="pre">HAClient</span></code>.</p>
+<p>A <code class="docutils literal"><span class="pre">ServerChooser</span></code> has two required responsibilities to the
+<code class="docutils literal"><span class="pre">HAClient</span></code>:</p>
+<ul>
+<li><p class="first">Tells the <code class="docutils literal"><span class="pre">HAClient</span></code> the connection string for the server to
+connect to. If there are no servers, or the <code class="docutils literal"><span class="pre">ServerChooser</span></code> wants
+the connection to fail, the <code class="docutils literal"><span class="pre">ServerChooser</span></code> returns an empty
+string.</p>
+<p>To provide this information, the <code class="docutils literal"><span class="pre">ServerChooser</span></code> implements the
+<code class="docutils literal"><span class="pre">get_current_uri()</span></code> method.</p>
+</li>
+<li><p class="first">Provides an <code class="docutils literal"><span class="pre">Authenticator</span></code> for the current connection string. This
+is especially important for installations where different servers
+require different credentials or authentication tokens must be reset
+after each connection attempt.</p>
+<p>To provide the authenticator, the <code class="docutils literal"><span class="pre">ServerChooser</span></code> implements the
+<code class="docutils literal"><span class="pre">get_current_authenticator()</span></code> method.</p>
+</li>
+</ul>
+<p>The <code class="docutils literal"><span class="pre">HAClient</span></code> calls the <code class="docutils literal"><span class="pre">get_current_uri()</span></code> and
+<code class="docutils literal"><span class="pre">get_current_authenticator()</span></code> methods each time it needs to make a
+connection.</p>
+<p>Each time a connection succeeds, the <code class="docutils literal"><span class="pre">HAClient</span></code> calls the
+<code class="docutils literal"><span class="pre">report_success()</span></code> method of the <code class="docutils literal"><span class="pre">ServerChooser</span></code>. Each time a
+connection fails, the <code class="docutils literal"><span class="pre">HAClient</span></code> calls the <code class="docutils literal"><span class="pre">report_failure()</span></code> method
+of the <code class="docutils literal"><span class="pre">ServerChooser</span></code>. The <code class="docutils literal"><span class="pre">HAClient</span></code> does not require the
+<code class="docutils literal"><span class="pre">ServerChooser</span></code> to take any particular action when it calls these
+methods. These methods are provided for the <code class="docutils literal"><span class="pre">HAClient</span></code> to do internal
+maintenance, logging, or record keeping. For example, an <code class="docutils literal"><span class="pre">HAClient</span></code>
+might keep a list of available URIs with a current failure count, and
+skip over URIs that have failed more than 5 consecutive times until all
+URIs in the list have failed more than 5 consecutive times.</p>
+<p>When the <code class="docutils literal"><span class="pre">ServerChooser</span></code> returns an empty string from
+<code class="docutils literal"><span class="pre">get_current_uri()</span></code>, indicating that no servers are available for
+connection, the <code class="docutils literal"><span class="pre">HAClient</span></code> calls the <code class="docutils literal"><span class="pre">get_error()</span></code> method on the
+<code class="docutils literal"><span class="pre">ServerChooser</span></code>, if one is provided, and includes the string returned
+by <code class="docutils literal"><span class="pre">get_error()</span></code> in the generated exception.</p>
+</div>
+</div>
+<div class="section" id="heartbeats-and-failure-detection">
+<h2>Heartbeats and Failure Detection<a class="headerlink" href="#heartbeats-and-failure-detection" title="Permalink to this headline">¶</a></h2>
+<p id="index-7">Use of the <code class="docutils literal"><span class="pre">HAClient</span></code> allows your application to quickly recover from
+detected connection failures. By default, connection failure detection
+occurs when AMPS receives an operating system error on the connection.
+This system may result in unpredictable delays in detecting a connection
+failure on the client, particularly when failures in network routing
+hardware occur, and the client primarily acts as a subscriber.</p>
+<p>The heartbeat feature of the AMPS client allows connection failure to be
+detected quickly. Heartbeats ensure that regular messages are sent
+between the AMPS client and server on a predictable schedule. The AMPS
+client and server both assume disconnection has occurred if these
+regular heartbeats cease, ensuring disconnection is detected in a timely
+manner. To use the heartbeat feature, call the <code class="docutils literal"><span class="pre">set_heartbeat</span></code> method on
+<code class="docutils literal"><span class="pre">Client</span></code> or <code class="docutils literal"><span class="pre">HAClient</span></code>:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">memoryClient</span> <span class="o">=</span> <span class="n">AMPS</span><span class="o">.</span><span class="n">HAClient</span><span class="p">(</span><span class="s2">&quot;importantStuff&quot;</span><span class="p">)</span>
+<span class="o">...</span>
+<span class="n">memoryClient</span><span class="o">.</span><span class="n">set_heartbeat</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
+<span class="n">memoryClient</span><span class="o">.</span><span class="n">connect_and_logon</span><span class="p">()</span>
+<span class="o">...</span>
+</pre></div>
+</div>
+<p><strong>Example 9.3:</strong> <em>Heartbeat example</em></p>
+<p id="index-8">Method <code class="docutils literal"><span class="pre">set_heartbeat</span></code> takes one parameter: the heartbeat interval. The
+heartbeat interval specifies the periodicity of heartbeat messages sent
+by the server: the value <code class="docutils literal"><span class="pre">3</span></code> indicates messages are sent on a
+three-second interval. If the client receives no messages in a
+six-second window (two heartbeat intervals), the connection is assumed
+to be dead, and the <code class="docutils literal"><span class="pre">HAClient</span></code> attempts reconnection. An additional
+variant of <code class="docutils literal"><span class="pre">set_heartbeat</span></code> allows the idle period to be set to a value
+other than two heartbeat intervals. (The server, however, will always consider
+a connection to be closed after two heartbeat intervals without any traffic.)</p>
+<p>Notice that, for <code class="docutils literal"><span class="pre">HAClient</span></code>, <code class="docutils literal"><span class="pre">setHeartbeat</span></code> must be called <em>before</em>
+the client is connected. For <code class="docutils literal"><span class="pre">Client</span></code>, <code class="docutils literal"><span class="pre">setHeartbeat</span></code> must be called
+<em>after</em> the client is connected.</p>
+<div class="admonition caution">
+<p class="first admonition-title">Caution</p>
+<p class="last">Heartbeats are serviced on the receive thread created by the AMPS
+client. Your application must not block the receive thread for longer
+than the heartbeat interval or the application is subject to being
+disconnected.</p>
+</div>
+</div>
+<div class="section" id="considerations-for-publishers">
+<h2>Considerations for Publishers<a class="headerlink" href="#considerations-for-publishers" title="Permalink to this headline">¶</a></h2>
+<p id="index-9">Publishing with an <code class="docutils literal"><span class="pre">HAClient</span></code> is nearly identical to regular
+publishing; you simply call the <code class="docutils literal"><span class="pre">publish()</span></code> method with your message&#8217;s
+topic and data. The AMPS client sends the message to AMPS, and then
+returns from the <code class="docutils literal"><span class="pre">publish()</span></code> call. For maximum performance, the client
+does not wait for the AMPS server to acknowledge that the message has
+been received.</p>
+<p>When an <code class="docutils literal"><span class="pre">HAClient</span></code> sets a publish store, the publish store retains a
+copy of each outgoing message and requests that AMPS acknowledge that
+the message has been persisted. The AMPS server acknowledges messages
+back to the publisher. Acknowledgments can be delivered for multiple
+messages at periodic intervals (for topics recorded in the transaction
+log) or after each message (for topics that are not recorded in the
+transaction log). When an acknowledgment for a message is received, the
+<code class="docutils literal"><span class="pre">HAClient</span></code> removes that message from the bookmark store. When a connection
+to a server is made, the <code class="docutils literal"><span class="pre">HAClient</span></code> automatically determines which
+messages from the publish store (if any) the server has not processed,
+and replays those messages to the server once the connection is
+established.</p>
+<p>For reliable publishers, the application must choose how best to handle
+application shutdown. For example, it is possible for the network to
+fail immediately after the publisher sends the message, while the
+message is still in transit. In this case, the publisher has sent the
+message, but the server has not processed it and acknowledged it. During
+normal operation, the <code class="docutils literal"><span class="pre">HAClient</span></code> will automatically connect and retry
+the message. On shutdown, however, the application must decide whether
+to wait for messages to be acknowledged, or whether to exit.</p>
+<p>Publish store implementations provide an <code class="docutils literal"><span class="pre">unpersisted_count()</span></code> method
+that reports the number of messages that have not yet been acknowledged
+by the AMPS server. When the <code class="docutils literal"><span class="pre">unpersisted_count()</span></code> reaches <code class="docutils literal"><span class="pre">0</span></code>,
+there are no unpersisted messages in the local publish store.</p>
+<p>For the highest level of safety, an application can wait until the
+<code class="docutils literal"><span class="pre">unpersisted_count()</span></code> reaches <code class="docutils literal"><span class="pre">0</span></code>, which indicates that all of the
+messages have been persisted to the instance that the application is
+connected to, and the synchronous replication destinations configured
+for that instance. When a synchronous replication destination goes
+offline, this approach will cause the publisher to wait to exit until
+the destination comes back online or until the destination is downgraded
+to asynchronous replication.</p>
+<p>For applications that are shut down periodically for short periods of
+time (for example, applications that are only offline during a weekly
+maintenance window), another approach is to use the <code class="docutils literal"><span class="pre">publish_flush()</span></code>
+method to ensure that messages are delivered to AMPS, and then rely on
+the connection logic to replay messages as necessary when the
+application restarts.</p>
+<p>For example, the following code flushes messages to AMPS, then warns if
+not all messages have been acknowledged:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">client</span> <span class="o">=</span> <span class="n">AMPS</span><span class="o">.</span><span class="n">HAClient</span><span class="p">(</span><span class="s2">&quot;ha-publisher&quot;</span><span class="p">,</span>
+         <span class="s2">&quot;/mnt/fastDisk/moreImportantMessages.outgoing&quot;</span><span class="p">,</span>
+         <span class="s2">&quot;/mnt/fastDisk/moreImportantMessages.incoming&quot;</span><span class="p">)</span>
+<span class="o">...</span>
+<span class="n">client</span><span class="o">.</span><span class="n">connect_and_logon</span><span class="p">()</span>
+<span class="c1"># Publish messages</span>
+<span class="o">...</span>
+<span class="c1"># We think we are done, but the server may not</span>
+<span class="c1"># have received or acknowledged all messages yet.</span>
+<span class="c1"># Wait until the server has received all messages.</span>
+<span class="c1"># The program could also specify a timeout in this</span>
+<span class="c1"># command to avoid blocking forever if the network</span>
+<span class="c1"># is down or all servers are offline.</span>
+<span class="n">client</span><span class="o">.</span><span class="n">publish_flush</span><span class="p">()</span>
+<span class="c1"># Print warning to the console if messages have</span>
+<span class="c1"># been published but not yet acknowledged as</span>
+<span class="c1"># persisted</span>
+<span class="k">if</span> <span class="p">(</span><span class="n">client</span><span class="o">.</span><span class="n">get_unpersisted_count</span><span class="p">()</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">):</span>
+    <span class="nb">print</span><span class="p">(</span> <span class="s2">&quot;all messages have been published, &quot;</span> \
+        <span class="o">+</span> <span class="s2">&quot; but not all have been persisted&quot;</span> <span class="p">)</span>
+<span class="n">client</span><span class="o">.</span><span class="n">disconnect</span><span class="p">()</span>
+</pre></div>
+</div>
+<p><strong>Example 9.4:</strong> <em>HAPublisher</em></p>
+<p id="index-10">In this example, the client sends each
+message immediately when <code class="docutils literal"><span class="pre">publish()</span></code> is called. If AMPS becomes
+unavailable between the final <code class="docutils literal"><span class="pre">publish()</span></code> and the <code class="docutils literal"><span class="pre">disconnect()</span></code>, or
+one of the servers that the AMPS instance replicates to is offline, the
+client may not have received a persisted acknowledgment for all of the
+published messages. For example, if a message has not yet been persisted
+by all of the servers in the replication fabric that are connected with
+synchronous replication, AMPS will not have acknowledged the message.</p>
+<p>Before shutting down the client, the code does two things:</p>
+<ul class="simple">
+<li>First, the code flushes messages to the server to ensure that all
+messages have been delivered to AMPS.</li>
+<li>Next, the code checks to see if all of the messages in the publish store
+have been acknowledged as persisted by AMPS. If the messages have not
+been acknowledged, they will remain in the publish store file and will
+be published to AMPS, if necessary, the next time the application
+connects. An application may choose to loop until
+<code class="docutils literal"><span class="pre">get_unpersisted_count()</span></code> returns <code class="docutils literal"><span class="pre">0</span></code>, or (as we do in this case)
+simply warn that AMPS has not confirmed that the messages are fully
+persisted. The behavior you choose in your application should be
+consistent with the high-availability guarantees your application needs
+to provide.</li>
+</ul>
+<div class="admonition caution">
+<p class="first admonition-title">Caution</p>
+<p class="last">AMPS uses the name of the <code class="docutils literal"><span class="pre">HAClient</span></code> to determine the
+origin of messages. For the AMPS server to correctly
+identify duplicate messages, each instance of an
+application that publishes messages must use a distinct
+name. That name must be consistent across different runs
+of the application.</p>
+</div>
+<p>If your application crashes or is terminated, some published messages
+may not have been persisted in the AMPS server. If you use the
+file-based store—in other words, if you provide file names for
+persistent storage when you create the <code class="docutils literal"><span class="pre">HAClient</span></code>—the <code class="docutils literal"><span class="pre">HAClient</span></code>
+will recover the messages, and once logged on, will correlate the
+message store to what the AMPS server has received, re-publishing any
+missing messages. This occurs automatically when <code class="docutils literal"><span class="pre">HAClient</span></code> connects,
+without any explicit consideration in your code, other than ensuring
+that the same file name is used to create the <code class="docutils literal"><span class="pre">HAClient</span></code> if recovery
+is desired.</p>
+<div class="admonition warning">
+<p class="first admonition-title">Warning</p>
+<p class="last">AMPS provides persisted acknowledgment messages for
+topics that do not have a transaction log enabled.
+However, the level of durability provided for topics with
+no transaction log is minimal. Learn more about
+transaction logs in the <em>AMPS User Guide</em>.</p>
+</div>
+</div>
+<div class="section" id="detecting-failover-ahead-of-replication">
+<h2>Detecting Failover Ahead of Replication<a class="headerlink" href="#detecting-failover-ahead-of-replication" title="Permalink to this headline">¶</a></h2>
+<p>AMPS replication provides two different acknowledgment modes
+for outgoing replication links from an instance:</p>
+<ul class="simple">
+<li>For a link in <code class="docutils literal"><span class="pre">sync</span></code> acknowledgment mode, a message must
+be successfully acknowledged by the downstream instance of AMPS
+before this instance of AMPS will acknowledge the message.</li>
+<li>For a link in <code class="docutils literal"><span class="pre">async</span></code> acknowledgment mode, this link is
+not considered for acknowledging the message. In this mode,
+the downstream side of the replication link may not have
+received or processed the message at the time that
+the publisher receives an acknowledgment.</li>
+</ul>
+<p>As described in the <em>AMPS User Guide</em>, a publisher must not
+failover from one instance of AMPS to another instance when
+any link between those instances uses <code class="docutils literal"><span class="pre">async</span></code> acknowledgment
+<em>unless</em> replication is certain to have reached that instance.
+(For example, if replication is taking a maximum of 1.2 seconds
+between the instances and the publisher has been disconnected for
+30 seconds, all messages from that publisher will have been
+replicated).</p>
+<p>To help detect a situation where a publisher may be
+&#8220;jumping ahead&#8221; of messages that it has published, but which
+have not yet been replicated, the AMPS client allows an application
+to consider it to be an error to make a connection to a server
+that has not received messages previously published by the application.</p>
+<p>To enable this behavior, set the <code class="docutils literal"><span class="pre">set_error_on_publish_gap()</span></code> method on
+the <code class="docutils literal"><span class="pre">PublishStore</span></code> that is set for the client.
+When this property is set, the client will consider it to be an
+error to connect to a server that has not received messages
+previously published by the client, and consider the connection
+to have failed.</p>
+<p>Notice that an application that uses this approach may need to
+handle situations where no server has received the message, particularly
+if the replication configuration uses automated replication downgrade.</p>
+</div>
+<div class="section" id="considerations-for-subscribers">
+<h2>Considerations for Subscribers<a class="headerlink" href="#considerations-for-subscribers" title="Permalink to this headline">¶</a></h2>
+<p><code class="docutils literal"><span class="pre">HAClient</span></code> provides two important features for applications that
+subscribe to one or more topics: re-subscription, and a bookmark store
+to track the correct point at which to resume a bookmark subscription.</p>
+<div class="section" id="resubscription-with-asynchronous-message-processing">
+<h3>Resubscription with Asynchronous Message Processing<a class="headerlink" href="#resubscription-with-asynchronous-message-processing" title="Permalink to this headline">¶</a></h3>
+<p>Any asynchronous subscription placed using an <code class="docutils literal"><span class="pre">HAClient</span></code> is
+automatically reinstated after a disconnect or a failover. These
+subscriptions are placed in an in-memory <code class="docutils literal"><span class="pre">SubscriptionManager</span></code>, which
+is created automatically when the <code class="docutils literal"><span class="pre">HAClient</span></code> is instantiated.</p>
+<p>When a re-subscription occurs, the AMPS Python client re-executes the
+command as originally submitted, including the original topic, options,
+and so on. AMPS sends the subscriber any messages for the specified
+topic (or topic expression) that are published after the subscription is
+placed. For a <code class="docutils literal"><span class="pre">sow_and_subscribe</span></code> command, this means that the client
+re-issues the full command, including the SOW query as well as the
+subscription.</p>
+</div>
+<div class="section" id="resubscription-with-synchronous-message-processing">
+<h3>Resubscription with Synchronous Message Processing<a class="headerlink" href="#resubscription-with-synchronous-message-processing" title="Permalink to this headline">¶</a></h3>
+<p>The <code class="docutils literal"><span class="pre">HAClient</span></code> (starting with the AMPS Python client version 4.3.1.1)
+does not track synchronous message processing subscriptions in the
+<code class="docutils literal"><span class="pre">SubscriptionManager</span></code>. The reason for this is to preserve the iterator
+semantics. That is, once the <code class="docutils literal"><span class="pre">MessageStream</span></code> indicates that there are
+no more elements in the stream, it does not suddenly produce more
+elements.</p>
+<p>To re-subscribe when the <code class="docutils literal"><span class="pre">HAClient</span></code> fails over, you can simply re-issue
+the subscription. For example, the snippet below re-issues the subscribe
+command when the message stream ends:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="k">while</span> <span class="n">still_need_to_process</span><span class="p">:</span>
+ <span class="c1"># Exiting the for loop is the end of stream.</span>
+ <span class="c1"># For a subscribe, this likely means that the</span>
+ <span class="c1"># client has disconnected.</span>
+ <span class="k">try</span><span class="p">:</span>
+   <span class="k">for</span> <span class="n">message</span> <span class="ow">in</span> <span class="n">client</span><span class="o">.</span><span class="n">subscribe</span><span class="p">(</span><span class="s2">&quot;messages&quot;</span><span class="p">):</span>
+     <span class="c1"># process messages here</span>
+     <span class="c1"># check condition on still_need_to_process</span>
+     <span class="k">if</span> <span class="n">still_need_to_process</span> <span class="o">==</span> <span class="kc">False</span><span class="p">:</span> <span class="k">break</span>
+ <span class="k">except</span> <span class="n">AMPS</span><span class="o">.</span><span class="n">DisconnectedException</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+     <span class="k">pass</span>
+</pre></div>
+</div>
+<p><strong>Example 9.5:</strong> <em>Resubscription</em></p>
+</div>
+<div class="section" id="bookmark-stores">
+<h3>Bookmark Stores<a class="headerlink" href="#bookmark-stores" title="Permalink to this headline">¶</a></h3>
+<p>In cases where it is critical not to miss a single message, it is
+important to be able to resume a subscription at the exact point that a
+failure occurred. In this case, simply recreating a subscription isn&#8217;t
+sufficient. Even though the subscription is recreated, the subscriber
+may have been disconnected at precisely the wrong time and will not see
+the message.</p>
+<p>To ensure delivery of every message from a topic or set of topics, the
+AMPS <code class="docutils literal"><span class="pre">HAClient</span></code> includes a <code class="docutils literal"><span class="pre">BookmarkStore</span></code> that, combined with the
+bookmark subscription and transaction log functionality in the AMPS
+server, ensures that clients receive any messages that might have been
+missed. The client stores the bookmark associated with each message
+received, and tracks whether the application has processed that message;
+if a disconnect occurs, the client uses the <code class="docutils literal"><span class="pre">BookmarkStore</span></code> to determine
+the correct resubscription point, and sends that bookmark to AMPS when
+it re-subscribes. AMPS then replays messages from its transaction log
+from the point after the specified bookmark, thus ensuring the client is
+completely up-to-date.</p>
+<p><code class="docutils literal"><span class="pre">HAClient</span></code> helps you to take advantage of this bookmark mechanism
+through the <code class="docutils literal"><span class="pre">BookmarkStore</span></code> interface and <code class="docutils literal"><span class="pre">bookmarkSubscribe()</span></code>
+method on <code class="docutils literal"><span class="pre">Client</span></code>. When you create subscriptions with
+<code class="docutils literal"><span class="pre">bookmarkSubscribe()</span></code>, whenever a disconnection or failover occurs,
+your application automatically re-subscribes to the message after the
+last message it processed. <code class="docutils literal"><span class="pre">HAClients</span></code> created by
+<code class="docutils literal"><span class="pre">createFileBacked()</span></code> additionally store these bookmarks on disk, so
+that the application can restart with the appropriate message if the
+client application fails and restarts.</p>
+<p>To take advantage of bookmark subscriptions, do the following:</p>
+<ul class="simple">
+<li>Ensure the topic(s) to be subscribed to are included in a transaction
+log. See the <em>AMPS User Guide</em> for information on how to specify the
+contents of a transaction log.</li>
+</ul>
+<blockquote>
+<div></div></blockquote>
+<ul class="simple" id="index-11">
+<li>Use <code class="docutils literal"><span class="pre">bookmark_subscribe()</span></code> instead of <code class="docutils literal"><span class="pre">subscribe()</span></code> when
+creating a <code class="docutils literal"><span class="pre">subscription()</span></code> and decide how the application will
+manage subscription identifiers (SubIds). If you are using a
+<code class="docutils literal"><span class="pre">Command</span></code> object, you can simply set the bookmark on that object.</li>
+</ul>
+<blockquote>
+<div></div></blockquote>
+<ul class="simple" id="index-12">
+<li>Use the <code class="docutils literal"><span class="pre">discard()</span></code> method in message handlers to indicate
+when a message has been fully processed by the application,
+that is, when the application does not need to receive
+the message again if the application fails over.</li>
+</ul>
+<p>The following example creates a bookmark subscription against a
+transaction-logged topic and fully processes each message as soon as it
+is delivered:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">MessagePrinter</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">client</span><span class="p">):</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">_client</span> <span class="o">=</span> <span class="n">client</span>
+    <span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">):</span>
+       <span class="nb">print</span> <span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">get_data</span><span class="p">())</span>
+       <span class="bp">self</span><span class="o">.</span><span class="n">_client</span><span class="o">.</span><span class="n">discard</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
+<span class="o">...</span>
+<span class="n">client</span> <span class="o">=</span> <span class="n">AMPS</span><span class="o">.</span><span class="n">HAClient</span><span class="p">(</span>
+    <span class="s2">&quot;aClient&quot;</span><span class="p">,</span>
+    <span class="s2">&quot;/logs/aClient.publishLog&quot;</span><span class="p">,</span>
+    <span class="s2">&quot;/logs/aClient.subscribeLog&quot;</span><span class="p">)</span>
+<span class="c1"># Create ServerChooser, populate chooser, connect client</span>
+<span class="o">...</span>
+<span class="n">client</span><span class="o">.</span><span class="n">execute_async</span><span class="p">(</span>                        \
+    <span class="n">AMPS</span><span class="o">.</span><span class="n">Command</span><span class="p">(</span><span class="s2">&quot;subscribe&quot;</span><span class="p">)</span>                            \
+        <span class="o">.</span><span class="n">set_topic</span><span class="p">(</span><span class="s2">&quot;myTopic&quot;</span><span class="p">)</span>                            \
+        <span class="o">.</span><span class="n">set_bookmark</span><span class="p">(</span><span class="n">AMPS</span><span class="o">.</span><span class="n">Client</span><span class="o">.</span><span class="n">Bookmarks</span><span class="o">.</span><span class="n">MOST_RECENT</span><span class="p">)</span> \
+        <span class="o">.</span><span class="n">set_sub_id</span><span class="p">(</span><span class="s2">&quot;MySubID&quot;</span><span class="p">),</span>                          \
+        <span class="n">MessagePrinter</span><span class="p">(</span><span class="n">client</span><span class="p">))</span>
+</pre></div>
+</div>
+<p><strong>Example 9.6:</strong> <em>HAClient subscription</em></p>
+<p>In this example, the client is a file-backed client, meaning that
+arriving bookmarks will be stored in a file (<code class="docutils literal"><span class="pre">aClient.subscribeLog</span></code>).
+Storing these bookmarks in a file allows the application to restart the
+subscription from the last message processed, in the event of either
+server or client failure.</p>
+<div class="admonition tip">
+<p class="first admonition-title">Tip</p>
+<p class="last">For optimum performance, it is critical to discard every
+message once its processing is complete. If a message is
+never discarded, it remains in the bookmark store. During
+re-subscription, <code class="docutils literal"><span class="pre">HAClient</span></code> always restarts the
+bookmark subscription with the oldest undiscarded
+message, and then filters out any more recent messages
+that have been discarded. If an old message remains in
+the store, but is no longer important for the
+application’s functioning, then the client and the AMPS
+server will incur unnecessary network, disk and CPU
+activity.</p>
+</div>
+<p>The fourth parameter, <code class="docutils literal"><span class="pre">sub_id</span></code>, specifies an identifier to be used for
+this subscription. Passing <code class="docutils literal"><span class="pre">None</span></code> causes <code class="docutils literal"><span class="pre">HAClient</span></code> to generate one
+and return it, like most other <code class="docutils literal"><span class="pre">Client</span></code> functions. However, if you
+wish to resume a subscription from a previous point after the
+application has terminated and restarted, the application must pass the
+same subscription ID as during its previous run. Passing a different
+subscription ID bypasses any recovery mechanisms, creating an entirely
+new subscription. When you use an existing subscription ID, the
+<code class="docutils literal"><span class="pre">HAClient</span></code> locates the last-used bookmark for that subscription in the
+local store, and attempts to re-subscribe from that point.</p>
+<p>Below are the different bookmark types that can be used to enable different
+recovery strategies for an application:</p>
+<ul class="simple" id="index-13">
+<li><code class="docutils literal"><span class="pre">Client.Bookmarks.NOW</span></code> specifies that the subscription
+should begin from the moment the server receives the subscription
+request. This results in the same messages being delivered as if you
+had invoked <code class="docutils literal"><span class="pre">subscribe()</span></code> instead, except that the messages will be
+accompanied by bookmarks. This is also the behavior that results if
+you supply an invalid bookmark.</li>
+</ul>
+<ul class="simple" id="index-14">
+<li><code class="docutils literal"><span class="pre">Client.Bookmarks.EPOCH</span></code>
+specifies that the subscription should begin from the beginning of
+the AMPS transaction log (that is, the first entry in the oldest
+journal file for the transaction log).</li>
+</ul>
+<blockquote>
+<div></div></blockquote>
+<ul class="simple" id="index-15">
+<li><code class="docutils literal"><span class="pre">Client.Bookmarks.MOST_RECENT</span></code> specifies that the
+subscription should begin from the last-used message in the
+associated <code class="docutils literal"><span class="pre">BookmarkStore</span></code>. Alternatively, if this subscription has
+not been seen before, it instructs the subscription to begin with
+<code class="docutils literal"><span class="pre">EPOCH</span></code>. This is the most common value for this parameter and is
+the value used in the preceding example. By using <code class="docutils literal"><span class="pre">MOST_RECENT</span></code>,
+the application automatically resumes from wherever the subscription
+left off, taking into account any messages that have already been
+processed and discarded.</li>
+</ul>
+<p>When the <code class="docutils literal"><span class="pre">HAClient</span></code> re-subscribes after a disconnection and
+reconnection, it always uses <code class="docutils literal"><span class="pre">MOST_RECENT</span></code>, ensuring that the
+continued subscription always begins from the last message used before
+the disconnect, so that no messages are missed.</p>
+</div>
+</div>
+<div class="section" id="conclusion">
+<h2>Conclusion<a class="headerlink" href="#conclusion" title="Permalink to this headline">¶</a></h2>
+<p>With only a few changes, most AMPS applications can take advantage of
+the <code class="docutils literal"><span class="pre">HAClient</span></code> and associated classes to become more highly-available
+and resilient. Using the <code class="docutils literal"><span class="pre">PublishStore</span></code>, publishers can ensure that
+every message published has actually been persisted by AMPS. Using
+<code class="docutils literal"><span class="pre">BookmarkStore</span></code>, subscribers can make sure that there are no gaps or
+duplicates in the messages received. <code class="docutils literal"><span class="pre">HAClient</span></code> makes both kinds of
+applications more resilient to network and server outages, as well as temporary
+issues. By utilizing the file based <code class="docutils literal"><span class="pre">HAClient</span></code>, clients can recover
+their state after an unexpected termination or crash. Though
+<code class="docutils literal"><span class="pre">HAClient</span></code> provides useful defaults for the <code class="docutils literal"><span class="pre">Store</span></code>,
+<code class="docutils literal"><span class="pre">BookmarkStore</span></code>, and <code class="docutils literal"><span class="pre">ServerChooser</span></code>, you
+can customize any or all of these to the specific needs of your
+application and architecture.</p>
+</div>
+</div>
+          </div>
+        </div>
+      </div>
+    <div class="clearer"></div>
+  </div>
+    <div class="footer">
+      &copy;2022, 60East Technologies, Inc..
+      |
+      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.5.3</a>
+      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
+    </div>
+  </body>
+</html>

amps-python-client 5.3.4.1__zip → 5.3.4.3__zip

amps-python-client 5.3.4.1zip → 5.3.4.3zip