PyPI - amps-python-client - Versions diffs - 5.3.4.1__zip → 5.3.4.2__zip - Mend

amps-python-client 5.3.4.1zip → 5.3.4.2zip

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (917) hide show

amps-python-client-5.3.4.1/docs/user-guide/html/chapters/txlog.html DELETED Viewed

@@ -1,1013 +0,0 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
-  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<html xmlns="http://www.w3.org/1999/xhtml">
-  <head>
-    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
-    <title>14. Transactional Messaging and Bookmark Subscriptions &#8212; AMPS User Guide develop documentation</title>
-    <link rel="stylesheet" href="../_static/alabaster.css" type="text/css" />
-    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
-    <script type="text/javascript">
-      var DOCUMENTATION_OPTIONS = {
-        URL_ROOT:    '../',
-        VERSION:     'develop',
-        COLLAPSE_INDEX: false,
-        FILE_SUFFIX: '.html',
-        HAS_SOURCE:  true,
-        SOURCELINK_SUFFIX: '.txt'
-      };
-    </script>
-    <script type="text/javascript" src="../_static/jquery.js"></script>
-    <script type="text/javascript" src="../_static/underscore.js"></script>
-    <script type="text/javascript" src="../_static/doctools.js"></script>
-    <link rel="index" title="Index" href="../genindex.html" />
-    <link rel="search" title="Search" href="../search.html" />
-    <link rel="next" title="15. Message Queues" href="queues.html" />
-    <link rel="prev" title="13. Aggregating and Analyzing Data in AMPS" href="views.html" />
-  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
-  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
-  </head>
-  <body role="document">
-  <div class="document">
-      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
-        <div class="sphinxsidebarwrapper">
-            <p class="logo"><a href="../index.html">
-              <img class="logo" src="../_static/flag_logo.png" alt="Logo"/>
-            </a></p>
-  <h3><a href="../index.html">Table Of Contents</a></h3>
-  <ul>
-<li><a class="reference internal" href="#">14. Transactional Messaging and Bookmark Subscriptions</a><ul>
-<li><a class="reference internal" href="#recording-and-replaying-messages-with-transaction-logs">Recording and Replaying Messages with Transaction Logs</a><ul>
-<li><a class="reference internal" href="#understanding-message-persistence">Understanding Message Persistence</a></li>
-<li><a class="reference internal" href="#configuring-a-transaction-log">Configuring a Transaction Log</a></li>
-<li><a class="reference internal" href="#replaying-messages-with-bookmark-subscription">Replaying Messages with Bookmark Subscription</a><ul>
-<li><a class="reference internal" href="#replay-of-full-transaction-log">Replay of Full Transaction Log</a></li>
-<li><a class="reference internal" href="#bookmark-replay-from-now">Bookmark Replay from NOW</a></li>
-<li><a class="reference internal" href="#bookmark-replay-with-a-bookmark">Bookmark Replay with a Bookmark</a></li>
-<li><a class="reference internal" href="#developer-note-the-most-recent-value">Developer Note: the MOST_RECENT value</a></li>
-<li><a class="reference internal" href="#bookmark-replay-from-a-moment-in-time">Bookmark Replay from a Moment in Time</a></li>
-<li><a class="reference internal" href="#bookmark-replay-with-a-starting-and-stopping-point">Bookmark Replay with a Starting and Stopping Point</a></li>
-<li><a class="reference internal" href="#content-and-topic-filtering">Content and Topic Filtering</a></li>
-<li><a class="reference internal" href="#delivery-rate-control-for-bookmark-subscriptions">Delivery Rate Control for Bookmark Subscriptions</a></li>
-<li><a class="reference internal" href="#pausing-and-resuming-bookmark-subscriptions">Pausing and Resuming Bookmark Subscriptions</a></li>
-<li><a class="reference internal" href="#conflation-and-bookmark-subscriptions">Conflation and Bookmark Subscriptions</a></li>
-<li><a class="reference internal" href="#requesting-message-timestamps">Requesting Message Timestamps</a></li>
-</ul>
-</li>
-<li><a class="reference internal" href="#selecting-message-durability-options">Selecting Message Durability Options</a><ul>
-<li><a class="reference internal" href="#using-the-fully-durable-option-for-bookmark-subscriptions">Using the &#8216;fully_durable&#8217; Option for Bookmark Subscriptions</a></li>
-<li><a class="reference internal" href="#using-the-live-option-for-bookmark-subscriptions">Using the &#8216;live&#8217; Option for Bookmark Subscriptions</a></li>
-</ul>
-</li>
-<li><a class="reference internal" href="#managing-journal-files">Managing Journal Files</a><ul>
-<li><a class="reference internal" href="#reference-to-file-types">Reference to File Types</a></li>
-</ul>
-</li>
-</ul>
-</li>
-<li><a class="reference internal" href="#using-amps-grep-to-find-messages-in-the-transaction-log">Using amps-grep to Find Messages in the Transaction Log</a><ul>
-<li><a class="reference internal" href="#finding-messages-from-a-specific-client">Finding Messages from a Specific Client</a></li>
-<li><a class="reference internal" href="#finding-messages-for-a-specific-topic">Finding Messages for a Specific Topic</a></li>
-<li><a class="reference internal" href="#finding-messages-with-specific-data">Finding Messages with Specific Data</a></li>
-</ul>
-</li>
-</ul>
-</li>
-</ul>
-<div class="relations">
-<h3>Related Topics</h3>
-<ul>
-  <li><a href="../index.html">Documentation overview</a><ul>
-      <li>Previous: <a href="views.html" title="previous chapter">13. Aggregating and Analyzing Data in AMPS</a></li>
-      <li>Next: <a href="queues.html" title="next chapter">15. Message Queues</a></li>
-  </ul></li>
-</ul>
-</div>
-<div id="searchbox" style="display: none" role="search">
-  <h3>Quick search</h3>
-    <form class="search" action="../search.html" method="get">
-      <div><input type="text" name="q" /></div>
-      <div><input type="submit" value="Go" /></div>
-      <input type="hidden" name="check_keywords" value="yes" />
-      <input type="hidden" name="area" value="default" />
-    </form>
-</div>
-<script type="text/javascript">$('#searchbox').show(0);</script>
-        </div>
-      </div>
-      <div class="documentwrapper">
-        <div class="bodywrapper">
-          <div class="body" role="main">
-  <div class="section" id="transactional-messaging-and-bookmark-subscriptions">
-<span id="txlog"></span><span id="ug-txlog"></span><span id="index-0"></span><h1>14. Transactional Messaging and Bookmark Subscriptions<a class="headerlink" href="#transactional-messaging-and-bookmark-subscriptions" title="Permalink to this headline">¶</a></h1>
-<p>AMPS includes support for transactional messaging, which includes
-persistence, consistency across restarts, and replay of messages
-published to AMPS. AMPS message queues use the transaction log
-to hold the messages in the queue. Transactional messaging is
-also the basis for replication, a key component of the
-high-availability capability in AMPS (as described in <a class="reference internal" href="ha.html#ug-ha"><span class="std std-ref">Highly Available AMPS Installations</span></a>).</p>
-<p>AMPS message queues use the transaction log as a
-persistent record of the messages that have entered the queue, the order
-of those messages, and which messages have been acknowledged and removed
-from the queue. All of these capabilities rely on the AMPS <em>transaction
-log</em>. The transaction log maintains a record of messages. You can choose
-which messages are included in the transaction log by specifying the
-message types and topics you want to record.</p>
-<p>The AMPS transaction log differs from transaction logging in a
-conventional relational database system. Unlike transaction logs that
-are intended solely to maintain the consistency of data in the system,
-the AMPS transaction log is fully queryable through the AMPS client
-APIs. For applications that need access to historical information, or
-applications that need to be able to recover state in the event of a
-client restart, the transaction log allows you to do this, relying on
-AMPS as the definitive single version of the state of the application.
-There is no need for complex logic to handle reconciliation or state
-restoration in the client. AMPS handles the difficult parts of this
-process, and the transaction log guarantees consistency.</p>
-<p>Topics covered by a transaction log are able to provide reliable
-messaging with strict consistency guarantees.</p>
-<p>When a transaction log is enabled, topics covered by the transaction log
-provide <em>atomic broadcast</em> from that instance. This means that the
-instance enforces a repeatable ordering on the messages, and guarantees
-that all subscribers receive messages reliably, in a consistent order,
-and with no gaps or duplicates.</p>
-<p>Enabling a transaction log in an instance also enables the following
-behaviors:</p>
-<ul class="simple">
-<li>When a transaction log is enabled, AMPS uses the client name provided
-by each client to manage the sequencing and integrity of the message
-stream, and to quickly detect and respond to reconnection. To allow this,
-each connection to AMPS must have a unique client name. If a duplicate
-client name is detected, one of the clients is assumed to be defunct and
-is disconnected.</li>
-<li>When a transaction log is enabled, persisted acknowledgments to a publish
-are conflated, as described in <a class="reference internal" href="acks.html#ug-ack-conflation"><span class="std std-ref">Acknowledgment Conflation and Publish Acknowledgments</span></a>.</li>
-</ul>
-<div class="section" id="recording-and-replaying-messages-with-transaction-logs">
-<span id="ha-transaction-log"></span><h2>Recording and Replaying Messages with Transaction Logs<a class="headerlink" href="#recording-and-replaying-messages-with-transaction-logs" title="Permalink to this headline">¶</a></h2>
-<p id="index-1">AMPS includes the ability to record messages in a <em>transaction log</em>, and
-replay those messages at a later time. This capability is key for high
-availability, since it gives subscribers the ability to resume a
-subscription at a point in time without missing messages. This
-capability is also the foundation of replication, since it gives AMPS
-the ability to preserve message streams to be synchronized to an
-instance that has gone offline.</p>
-<p>The transaction log in AMPS contains a sequential, historical record of
-messages. Each message is identified by a <code class="docutils literal"><span class="pre">bookmark</span></code>, a unique
-identifier that AMPS uses to locate the message within the overall set
-of recorded messages. The transaction log can record messages for a
-topic, a set of topics, or for filtered content on one or more topics.</p>
-<p>An application can request a subscription that replays messages from the
-transaction log. Subscriptions that replay from the transaction log are
-called <em>bookmark subscriptions</em>, since the subscription begins at a
-specific point in the transaction log identified by a specific bookmark.
-Bookmark subscriptions provide topic and content filtering in the same
-way that normal subscriptions do, and provide a set of unique
-capabilities (such as the ability to pause and resume the subscription)
-that are made possible because the subscription is provided from a
-persistent record of the message stream. Bookmark subscriptions are also
-key to high availability with AMPS. When a client is recovering from a
-restart or failure, this ability to replay allows a client to fill gaps
-in received messages and resume subscriptions without missing a message.
-This feature also allows new clients to receive an exact replay of a
-message stream. Replay from the transaction log is also useful for
-auditing, quality assurance, and backtesting.</p>
-<p>The transaction log is used in AMPS replication to ensure that all
-servers in a replication group are continually synchronized should one
-of them experience an interruption in service. For example, say an AMPS
-instance, as a member of a replication group, goes down. When it comes
-back up, it can query another AMPS instance for all of the messages it
-did not receive, thereby catching up to a point of synchronization with
-the other instances. This feature, when coupled with AMPS replication,
-ensures that message subscriptions are always available and up-to-date.</p>
-<p>The AMPS transaction log records messages that are received from a
-publisher and events that affect those messages such as <code class="docutils literal"><span class="pre">sow_delete</span></code>
-commands. AMPS does not record messages that are created through a view,
-out-of-focus messages, or event status messages created by AMPS.</p>
-<p>When a subscriber requests a replay from the transaction log, AMPS will
-deliver <code class="docutils literal"><span class="pre">publish</span></code> messages from the transaction log in exactly the
-order in which the instance recorded them. Messages that are not
-stored in the transaction log (for example, out-of-focus messages or
-event messages, messages that would have been produced by a view,
-acknowledgment messages to clients) are not delivered as part of
-the replay.</p>
-<div class="section" id="understanding-message-persistence">
-<h3>Understanding Message Persistence<a class="headerlink" href="#understanding-message-persistence" title="Permalink to this headline">¶</a></h3>
-<p>To take advantage of transactional messaging, the publisher and the AMPS
-instance work together to ensure that messages are written to persistent
-storage. AMPS lets the publisher know when the message is persisted, so
-that the publisher knows that it no longer needs to track the message.</p>
-<p>When a publisher publishes a message to AMPS, the publisher assigns each
-message a unique sequence number. Once the message has been written to
-persistent storage, AMPS uses the sequence number to acknowledge the
-message and let the publisher know that the message is persisted. Once
-AMPS has acknowledged the message, the publisher considers the message
-published. For safety, AMPS always writes a message to the local
-transaction log before acknowledging that the message is persisted. If
-the topic is configured for synchronous replication, all replication
-destinations have to persist the message before AMPS will acknowledge
-that the message is persisted.</p>
-<p>For efficiency, AMPS may not acknowledge each individual message.
-Instead, AMPS acknowledges the most recent persisted message to indicate
-that all previous messages have also been persisted, as described in
-<a class="reference internal" href="acks.html#ug-ack-conflation"><span class="std std-ref">Acknowledgment Conflation and Publish Acknowledgments</span></a>. Publishers that
-need transactional messaging do not wait for acknowledgment to publish
-more messages. Instead, publishers retain messages that haven&#8217;t been
-acknowledged, and republish messages that haven&#8217;t been acknowledged if
-failover occurs. The AMPS client libraries include this functionality
-for persistent messaging (see descriptions of the <em>publish store</em> in
-client library documentation). See the <a class="reference internal" href="ha.html#ug-guaranteed-publishing"><span class="std std-ref">Guaranteed Publishing</span></a> section
-of this guide for further details.</p>
-</div>
-<div class="section" id="configuring-a-transaction-log">
-<h3>Configuring a Transaction Log<a class="headerlink" href="#configuring-a-transaction-log" title="Permalink to this headline">¶</a></h3>
-<p id="index-2">Before demonstrating the power of the transaction log, we will first
-show how to configure the transaction log in the AMPS configuration
-file.</p>
-<div class="highlight-xml"><div class="highlight"><pre><span></span><span class="c">&lt;!-- All transaction log definitions are contained within the TransactionLog block.</span>
-<span class="c">    The following global settings apply to all Topic blocks defined within the</span>
-<span class="c">    TransactionLog: JournalDirectory, PreallocatedJournalFiles, and JournalSize. --&gt;</span>
-<span class="nt">&lt;TransactionLog&gt;</span>
-    <span class="c">&lt;!-- The JournalDirectory is the filesystem location where journal files and journal</span>
-<span class="c">        index files will be stored. --&gt;</span>
-    <span class="nt">&lt;JournalDirectory&gt;</span>./amps/journal/<span class="nt">&lt;/JournalDirectory&gt;</span>
-    <span class="c">&lt;!-- The JournalArchiveDirectory is the filesystem location to which AMPS will</span>
-<span class="c">        archive journals. Notice that AMPS does not archive files by default. You</span>
-<span class="c">        configure an action to archive journal files --&gt;</span>
-    <span class="nt">&lt;JournalArchiveDirectory&gt;</span>/mnt/somedev0/amps/journal<span class="nt">&lt;/JournalArchiveDirectory&gt;</span>
-    <span class="c">&lt;!-- PreallocatedJournalFiles defines the number of journal files AMPS will create as</span>
-<span class="c">        part of the server startup. Default: 2 Minimum: 1 --&gt;</span>
-    <span class="nt">&lt;PreallocatedJournalFiles&gt;</span>1<span class="nt">&lt;/PreallocatedJournalFiles&gt;</span>
-    <span class="c">&lt;!-- The JournalSize is the approximate size of the journal files</span>
-<span class="c">         that AMPS will create. --&gt;</span>
-    <span class="nt">&lt;JournalSize&gt;</span>10MB<span class="nt">&lt;/JournalSize&gt;</span> <span class="c">&lt;!-- Suitable for development work,</span>
-<span class="c">                                         a production deployment would</span>
-<span class="c">                                         typically use 250MB to 2GB</span>
-<span class="c">                                         depending on message size,</span>
-<span class="c">                                         message volume volume</span>
-<span class="c">                                         and retention policy. --&gt;</span>
-    <span class="c">&lt;!-- When a Topic is specified, then all messages which match exactly the specified</span>
-<span class="c">        topic or regular expression will be included in the transaction log. Otherwise,</span>
-<span class="c">        AMPS initializes the transaction logging, but does not record any messages to</span>
-<span class="c">        the transaction log. --&gt;</span>
-    <span class="nt">&lt;Topic&gt;</span>
-        <span class="nt">&lt;Name&gt;</span>orders<span class="nt">&lt;/Name&gt;</span>
-        <span class="nt">&lt;MessageType&gt;</span>nvfix<span class="nt">&lt;/MessageType&gt;</span>
-    <span class="nt">&lt;/Topic&gt;</span>
-<span class="nt">&lt;/TransactionLog&gt;</span>
-</pre></div>
-</div>
-</div>
-<div class="section" id="replaying-messages-with-bookmark-subscription">
-<span id="ug-bookmark-sub"></span><span id="index-3"></span><h3>Replaying Messages with Bookmark Subscription<a class="headerlink" href="#replaying-messages-with-bookmark-subscription" title="Permalink to this headline">¶</a></h3>
-<p>One of the most useful and powerful features in AMPS is <em>bookmark
-subscription</em>, which is enabled by the transaction log. With bookmark
-subscription, an application requests a subscription that starts at a
-specific point in the transaction log. AMPS begins the subscription at
-the specified point, and provides messages from the transaction log.</p>
-<p>Each message in the transaction log has a <em>bookmark</em>. A <em>bookmark</em> is an
-opaque, unique identifier that is added by AMPS to each message recorded
-in the transaction log. For messages provided from a transaction log,
-the field is included in the <code class="docutils literal"><span class="pre">Bookmark</span></code> header of the message. AMPS
-guarantees that bookmarks for the instance are monotonically increasing,
-which enables AMPS to rapidly find an individual bookmark within the
-transaction log.</p>
-<p>A bookmark subscription simply requests that AMPS begin the subscription
-with the first message following the bookmark provided with the
-subscription. AMPS locates the bookmark in the transaction log, and
-begins the subscription at that point in time.</p>
-<p>One way to think about a bookmark subscription is that AMPS publishes to
-the subscribing client only those messages that:</p>
-<ol class="arabic simple">
-<li>Have bookmarks after the provided bookmark</li>
-<li>Match the subscription&#8217;s <code class="docutils literal"><span class="pre">Topic</span></code> and <code class="docutils literal"><span class="pre">Filter</span></code></li>
-<li>Have been written to the transaction log</li>
-</ol>
-<p>AMPS provides these messages in the order in which they were recorded to
-the transaction log. Since a bookmark subscription requires a
-transaction log, when a client requests a bookmark subscription for a
-topic that is not being recorded in the transaction log, AMPS returns an
-error.</p>
-<p>If the subscription requests a <code class="docutils literal"><span class="pre">completed</span></code> acknowledgment, that
-acknowledgment will be delivered to the subscription once replay has completed.
-Messages delivered to the subscription after the acknowledgment is delivered
-are from new publishes.</p>
-<p>AMPS allows an application to submit a comma-delimited list of bookmark
-values as the bookmark for a subscription request. In this case, AMPS
-begins replay at the oldest bookmark in the list. The client controls
-the bookmark provided on the subscription request. For a bookmark
-subscription, the AMPS server does not keep a persistent record of which
-bookmarks a specific client or subscription has processed. The AMPS
-client libraries provide facilities for easily tracking the messages
-which an application has processed so the application can resume at the
-appropriate point in the transaction log.</p>
-<p>Requesting replay from the transaction log is how AMPS applications
-manage <em>resumable subscriptions</em>. The application keeps track of which
-messages have been processed, and requests replay from the appropriate
-point in the transaction log to resume the message stream. This record-keeping
-is built into the AMPS client libraries, and most often handled
-transparently when a <em>bookmark store</em> is set for the client. Notice that
-this means that the AMPS server itself does not track the progress of
-individual subscriptions, nor does the application need to inform the server
-of how far the subscription has progressed. The application state needed
-to resume the subscription is entirely handled on the application side,
-with no involvement by the AMPS server. (For details on how specific
-client libraries manage the application state, see the <em>Developer Guide</em>
-for that client library.)</p>
-<div class="admonition tip">
-<p class="first admonition-title">Tip</p>
-<p class="last">Bookmark subscriptions are provided from the transaction log
-rather than the live publish stream. This lets AMPS adapt the
-pace of replay to the pace at which the subscriber is consuming
-replayed messages without triggering slow client offlining.</p>
-</div>
-<div class="admonition tip">
-<p class="first admonition-title">Tip</p>
-<p class="last">While there are similarities between a bookmark subscription
-used for replay and a State of the World (SOW) query, the transaction
-log and SOW are independent features that can be used separately.
-The SOW gives  a snapshot of the current view of the latest data, while the
-journal is capable of playback of previous messages. Historical
-SOW queries provide a snapshot of the SOW at a defined point in
-the past, and are provided by the SOW database rather than the
-transaction log.</p>
-</div>
-<p>There are different ways that a client can request a bookmark replay
-from the transaction log. Each of these bookmark types meets a
-different need and enables a different recovery strategy that an
-application can use. The sections below describe the recovery types, the
-cases in which they can be used, and how the 60East clients implement
-them.</p>
-<div class="section" id="replay-of-full-transaction-log">
-<h4>Replay of Full Transaction Log<a class="headerlink" href="#replay-of-full-transaction-log" title="Permalink to this headline">¶</a></h4>
-<p id="index-4">The epoch bookmark, when requested on a subscription, will replay the
-transaction log starting at the very beginning.
-Once the transaction log has been replayed in its entirety, then the
-subscriber will begin receiving messages on the live incoming stream of
-messages. A subscriber does this by requesting a <code class="docutils literal"><span class="pre">0</span></code> in the
-<code class="docutils literal"><span class="pre">bookmark</span></code> header field of their subscription. The AMPS clients
-provide a constant for epoch, typically represented as <code class="docutils literal"><span class="pre">EPOCH</span></code>.</p>
-<p>This type of bookmark can be used in a case where the subscriber has
-begun after the start of an event, and needs to catch up on all of the
-messages that have been published to the topic.</p>
-<p>To ensure that no messages from the subscription are lost during the
-replay, AMPS replays messages from the transaction log until the client
-reaches the last message in the transaction log. Once all of the
-existing messages in the transaction log have been sent to the client,
-AMPS will cut over to the live subscription stream and provide messages
-to the client as soon as they are persisted.</p>
-</div>
-<div class="section" id="bookmark-replay-from-now">
-<h4>Bookmark Replay from NOW<a class="headerlink" href="#bookmark-replay-from-now" title="Permalink to this headline">¶</a></h4>
-<p id="index-5">The NOW bookmark, when requested on a subscription, declines to replay
-any messages from the transaction log, and instead begins streaming
-messages from the live stream - returning any messages that would be
-published to the transaction log that match the subscription&#8217;s <code class="docutils literal"><span class="pre">Topic</span></code>
-and <code class="docutils literal"><span class="pre">Filter</span></code>.</p>
-<p>This type of bookmark is used when a client is concerned with messages
-that will be published to the transaction log, but is unconcerned with
-replaying the historical messages in the transaction log. This strategy
-is often used for applications that want to ensure that they do not miss
-messages, even if the application temporarily loses connectivity, but
-are not concerned with older messages. For this case, the application
-subscribes with NOW when the application starts, and then re-establishes
-the subscription with the most recently-processed bookmark if
-connectivity is lost. This resubscription behavior is typically handled
-by the client reconnection logic (as in the 60East <code class="docutils literal"><span class="pre">HAClient</span></code>
-implementations).</p>
-<p>The NOW bookmark is performed using a subscribe query with &#8220;0|1|&#8221; as
-the <code class="docutils literal"><span class="pre">bookmark</span></code> field. The AMPS clients provide a constant for this
-value, typically represented as <code class="docutils literal"><span class="pre">NOW</span></code>.</p>
-</div>
-<div class="section" id="bookmark-replay-with-a-bookmark">
-<h4>Bookmark Replay with a Bookmark<a class="headerlink" href="#bookmark-replay-with-a-bookmark" title="Permalink to this headline">¶</a></h4>
-<p id="index-6">Clients that store the bookmarks from published messages can use those
-bookmarks to recover from an interruption in service. By placing a
-subscribe query with the last bookmark recorded, a client will get a
-replay of all messages persisted to the transaction log after that
-bookmark. Once the replay has completed, the subscription will then cut
-over to the live stream of messages.</p>
-<p>To perform a bookmark replay, the client places a bookmark subscription
-with the bookmark at which to start the subscription.</p>
-<p>A bookmark subscription must always start at a specific point in the
-transaction log. The AMPS server uses a bookmark to locate that
-point in the transaction log and begin replay at that point.</p>
-<p>If a bookmark is unknown (that is, the transaction log
-does not contain that bookmark), the AMPS server does not have a specific
-point at which to begin the replay and will assume that the subscriber
-has failed over from another instance that has not yet replicated
-publishes to this instance, and begin replay at NOW (the end of the
-transaction log).</p>
-</div>
-<div class="section" id="developer-note-the-most-recent-value">
-<h4>Developer Note: the MOST_RECENT value<a class="headerlink" href="#developer-note-the-most-recent-value" title="Permalink to this headline">¶</a></h4>
-<p id="index-7">The AMPS client libraries provide a special constant value that requests
-that the library look up the appropriate recovery point in the bookmark
-store and then provide that recovery point in the subscription
-request. This special value is typically represented as
-<code class="docutils literal"><span class="pre">MOST_RECENT</span></code>, <code class="docutils literal"><span class="pre">RECENT</span></code>, or <code class="docutils literal"><span class="pre">recent</span></code>. When the application requests a
-bookmark subscription with a bookmark of <code class="docutils literal"><span class="pre">MOST_RECENT</span></code>, the client library
-looks for the most recent bookmarks processed for that subscription, then
-provides the appropriate bookmark or list of bookmarks when resubscribing.
-This process helps to ensure that the subscription begins at the last
-processed message, and the application receives the next unprocessed message
-for the subscription. If there is no record of a subscription, the AMPS clients
-will start with <code class="docutils literal"><span class="pre">EPOCH</span></code>, so that the first time a subscription is
-entered, the application gets the full record of available messages.</p>
-<p>It&#8217;s important to remember that the AMPS server has no knowledge of the
-<code class="docutils literal"><span class="pre">MOST_RECENT</span></code> value. <code class="docutils literal"><span class="pre">MOST_RECENT</span></code> itself is never sent to AMPS and never
-appears in the AMPS log. <code class="docutils literal"><span class="pre">MOST_RECENT</span></code> is simply a request to the AMPS
-client library to look up the exact bookmark value to provide to AMPS. The
-AMPS client libraries always translate a request for <code class="docutils literal"><span class="pre">MOST_RECENT</span></code>
-into either a specific value (typically a list of bookmarks) or <code class="docutils literal"><span class="pre">EPOCH</span></code>.</p>
-</div>
-<div class="section" id="bookmark-replay-from-a-moment-in-time">
-<h4>Bookmark Replay from a Moment in Time<a class="headerlink" href="#bookmark-replay-from-a-moment-in-time" title="Permalink to this headline">¶</a></h4>
-<p id="index-8">The final type of bookmark supported is the ASCII-formatted timestamp.
-When using a timestamp as the bookmark value, the transaction log
-replays all messages that occurred after the timestamp, and then cuts
-over to the live subscription once the replay stream has been consumed.</p>
-<p>This bookmark has the format of <code class="docutils literal"><span class="pre">YYYYmmddTHHMMSS[Z]</span></code> where:</p>
-<ul class="simple">
-<li><code class="docutils literal"><span class="pre">YYYY</span></code> is the four digit year.</li>
-<li><code class="docutils literal"><span class="pre">mm</span></code> is the two digit month.</li>
-<li><code class="docutils literal"><span class="pre">dd</span></code> is the two digit day.</li>
-<li><code class="docutils literal"><span class="pre">T</span></code> is the character separator between the date and time.</li>
-<li><code class="docutils literal"><span class="pre">HH</span></code> is the two digit hour.</li>
-<li><code class="docutils literal"><span class="pre">MM</span></code> is the two digit minute.</li>
-<li><code class="docutils literal"><span class="pre">SS</span></code> is the two digit second.</li>
-<li><code class="docutils literal"><span class="pre">Z</span></code> is an optional timezone specifier. AMPS timestamps are always
-in UTC, regardless of whether the timezone is included. AMPS only
-accepts a literal value of <code class="docutils literal"><span class="pre">Z</span></code> for a timezone specifier.</li>
-</ul>
-<p>For example, a timestamp for January 2nd, 2015, at 12:35:</p>
-<div class="code highlight-default"><div class="highlight"><pre><span></span><span class="mi">20150102</span><span class="n">T123500Z</span>
-</pre></div>
-</div>
-<p>With a timestamp, the AMPS server begins replay at the closest point
-in the current transaction log, even if there is no message recorded
-at that exact moment. This means that a timestamp prior to the
-beginning of the journal will start a replay at EPOCH.</p>
-</div>
-<div class="section" id="bookmark-replay-with-a-starting-and-stopping-point">
-<h4>Bookmark Replay with a Starting and Stopping Point<a class="headerlink" href="#bookmark-replay-with-a-starting-and-stopping-point" title="Permalink to this headline">¶</a></h4>
-<p>As of version 5.3.2, AMPS allows a subscriber to specify the
-point at which a bookmark replay should stop. When specified,
-a subscriber will not receive further messages after the replay
-reaches the stopping point. Also, when a stopping point is specified
-and a <code class="docutils literal"><span class="pre">completed</span></code> acknowledgment is requested for the subscription,
-AMPS will return the <code class="docutils literal"><span class="pre">completed</span></code> acknowledgment (if requested)
-when the stopping point is reached, rather than when replay reaches
-the point in the transaction log at which the subscription was
-entered.</p>
-<p>To set a starting and stopping point, a subscription provides
-a subscription range specifier with the bookmark. The format of the
-subscription range specifier is as follows:</p>
-<div class="code highlight-default"><div class="highlight"><pre><span></span><span class="o">&lt;</span><span class="n">begin_interval_specifier</span><span class="o">&gt;</span> <span class="o">&lt;</span><span class="n">begin_bookmarks</span><span class="o">&gt;</span> <span class="p">:</span> <span class="o">&lt;</span><span class="n">end_bookmarks</span><span class="o">&gt;</span> <span class="o">&lt;</span><span class="n">end_interval_specifier</span><span class="o">&gt;</span>
-</pre></div>
-</div>
-<p>The <em>begin_interval_specifier</em> is one of:</p>
-<table border="1" class="docutils">
-<colgroup>
-<col width="50%" />
-<col width="50%" />
-</colgroup>
-<thead valign="bottom">
-<tr class="row-odd"><th class="head"><strong>Specifier</strong></th>
-<th class="head"><strong>Behavior</strong></th>
-</tr>
-</thead>
-<tbody valign="top">
-<tr class="row-even"><td><code class="docutils literal"><span class="pre">(</span></code></td>
-<td><em>Exclusive replay.</em>
-Begin immediately <em>after</em> the bookmark  or range
-specified. The specified bookmark will <em>not</em>
-be present in the replay.</td>
-</tr>
-<tr class="row-odd"><td><code class="docutils literal"><span class="pre">[</span></code></td>
-<td><em>Inclusive replay.</em>
-Begin immediately <em>before</em> the bookmark or range
-specified. The specified bookmark will
-be present in the replay.</td>
-</tr>
-</tbody>
-</table>
-<p>The <em>end_interval_specifier</em> is one of:</p>
-<table border="1" class="docutils">
-<colgroup>
-<col width="50%" />
-<col width="50%" />
-</colgroup>
-<thead valign="bottom">
-<tr class="row-odd"><th class="head"><strong>Specifier</strong></th>
-<th class="head"><strong>Behavior</strong></th>
-</tr>
-</thead>
-<tbody valign="top">
-<tr class="row-even"><td><code class="docutils literal"><span class="pre">)</span></code></td>
-<td><em>Exclusive replay.</em>
-End immediately <em>before</em> the bookmark or range
-specified. The specified bookmark will <em>not</em>
-be present in the replay.</td>
-</tr>
-<tr class="row-odd"><td><code class="docutils literal"><span class="pre">]</span></code></td>
-<td><em>Inclusive replay.</em>
-End immediately <em>after</em> the bookmark or range
-specified. The specified bookmark will
-be present in the replay.</td>
-</tr>
-</tbody>
-</table>
-<p>Bookmarks provided for a subscription range specifier can be either
-timestamps (as described above) or literal bookmarks provided by AMPS.</p>
-<p>For example, to replay messages received by the instance on June 4, 2020
-we could construct an interval beginning at midnight on June 4 UTC
-(inclusive) and ending at midnight on June 5 UTC (exclusive), as follows:</p>
-<div class="code highlight-default"><div class="highlight"><pre><span></span><span class="p">[</span><span class="mi">20200604</span><span class="n">T000000</span><span class="p">:</span><span class="mi">20200605</span><span class="n">T000000</span><span class="p">)</span>
-</pre></div>
-</div>
-<p>AMPS allows future timestamps in a range specifier. For example, it would
-be valid for an application to enter a subscription that collects messages
-for a full business day and then completes, even if the application is
-started before business hours.  AMPS will begin delivering messages at the
-start time specified, and deliver a <code class="docutils literal"><span class="pre">completed</span></code> ack (if requested) and
-stop delivering messages at the end time specified.</p>
-<p>The starting and stopping points are most often provided as timestamps.
-AMPS also allows an application to specify starting and stopping points
-using message bookmarks. In this case, rather than finding the specific
-time specified, AMPS finds the point in the log at which the bookmark
-was recorded.</p>
-<p>An application can choose to submit a list of bookmarks as the starting or
-stopping point. In this case, AMPS will find the <em>earliest</em> bookmark in the
-starting point list (as recorded in the local transaction log) and the <em>latest</em>
-bookmark in the stopping point list (as recorded in the local transaction log), and
-replay as though those two bookmarks had been provided to the replay command.</p>
-</div>
-<div class="section" id="content-and-topic-filtering">
-<h4>Content and Topic Filtering<a class="headerlink" href="#content-and-topic-filtering" title="Permalink to this headline">¶</a></h4>
-<p id="index-9">As with all other subscriptions, bookmark subscriptions support content
-filtering.</p>
-<p>Bookmark subscriptions provide only messages from topics that are
-recorded in the transaction log. In other words, when a bookmark
-subscription uses a topic regular expression, only messages from topics
-that are recorded in the transaction log are provided to the
-subscription. This ensures that a bookmark subscription provides a
-consistent, repeatable stream of messages. The topics provided to the
-subscription are the same during replay, when only messages recorded in
-the transaction log are available, and after replay completes, when
-every publish to AMPS is available. This also ensures that a bookmark
-subscription that replays messages for a specific timeframe gets the
-same messages as bookmark subscribers that had active subscriptions
-during that timeframe.</p>
-<p>Content filtering is covered in greater detail in
-<a class="reference internal" href="amps_expressions.html#ug-amps-expressions"><span class="std std-ref">AMPS Expressions</span></a>.</p>
-</div>
-<div class="section" id="delivery-rate-control-for-bookmark-subscriptions">
-<h4>Delivery Rate Control for Bookmark Subscriptions<a class="headerlink" href="#delivery-rate-control-for-bookmark-subscriptions" title="Permalink to this headline">¶</a></h4>
-<p id="index-10">AMPS allows subscribers to specify the maximum delivery rate for
-messages delivered from a bookmark subscription. A subscriber specifies
-the maximum rate at which AMPS should deliver messages to the
-subscription. AMPS then limits the rate at which replay from the
-transaction log occurs so that the overall rate does not exceed the
-specified maximum.  Rate control is not available for subscriptions
-that use the <code class="docutils literal"><span class="pre">live</span></code> option.</p>
-<p>To request rate control, a subscriber provides the <code class="docutils literal"><span class="pre">rate</span></code> option on
-the subscription. A rate can be specified in either messages per second,
-number of bytes delivered per second, or a multiple of the original
-delivery rate. For example, the following subscription option limits
-delivery to 1000 messages per second:</p>
-<div class="code highlight-default"><div class="highlight"><pre><span></span><span class="n">rate</span><span class="o">=</span><span class="mi">1000</span>
-</pre></div>
-</div>
-<p>To limit delivery to 500KB per second, a subscriber would provide this
-option:</p>
-<div class="code highlight-default"><div class="highlight"><pre><span></span><span class="n">rate</span><span class="o">=</span><span class="mi">500</span><span class="n">KB</span>
-</pre></div>
-</div>
-<p>To limit delivery to double the speed at which messages were originally
-published, a subscriber would provide this option:</p>
-<div class="code highlight-default"><div class="highlight"><pre><span></span><span class="n">rate</span><span class="o">=</span><span class="mi">2</span><span class="n">X</span>
-</pre></div>
-</div>
-<p>To limit delivery to half the speed at which messages were originally
-published, a subscriber would provide this option:</p>
-<div class="code highlight-default"><div class="highlight"><pre><span></span><span class="n">rate</span><span class="o">=</span><span class="mf">.5</span><span class="n">X</span>
-</pre></div>
-</div>
-<p>When using a <code class="docutils literal"><span class="pre">rate</span></code> that is a factor of the original replay speed,
-you may want AMPS to skip over long gaps. For example, you may want
-to do load testing by replaying several days&#8217; worth of operations
-at a <code class="docutils literal"><span class="pre">5x</span></code> multiplier. In that case, however, your load test does
-not need to be idle when there are gaps during which no messages are
-produced (for example, outside of trading hours or during holidays). For
-this situation, AMPS provides a <code class="docutils literal"><span class="pre">rate_max_gap</span></code> option that sets the maximum
-amount of time for a replay to wait to produce a message.  For example,
-with an option string like:</p>
-<div class="code highlight-default"><div class="highlight"><pre><span></span><span class="n">rate</span><span class="o">=</span><span class="mi">5</span><span class="n">X</span><span class="p">,</span><span class="n">rate_max_gap</span><span class="o">=</span><span class="mi">10</span><span class="n">s</span>
-</pre></div>
-</div>
-<p>AMPS will attempt to produce messages at 5 times the original publish
-rate. In the event that there is a gap between messages of more than 50
-seconds in the original publish stream (that is, 10 seconds in the replay),
-AMPS will wait for 10 seconds and then &#8220;skip ahead&#8221; to the next message in
-the replay.</p>
-</div>
-<div class="section" id="pausing-and-resuming-bookmark-subscriptions">
-<h4>Pausing and Resuming Bookmark Subscriptions<a class="headerlink" href="#pausing-and-resuming-bookmark-subscriptions" title="Permalink to this headline">¶</a></h4>
-<p>As of version 5.0, AMPS offers the ability to pause a bookmark
-subscription. When a subscriber requests that AMPS pause the
-subscription, AMPS stops providing messages from the bookmark
-subscription, but does not remove the subscription. The subscriber can
-then resume the subscription, and AMPS will again begin providing
-messages from the subscription. While the subscription is paused, AMPS
-maintains a record of the current position in the transaction log, and
-begins replay from that point.</p>
-<p>This feature is most useful for starting replay of a number of
-subscriptions at the same point in the transaction log, and
-ensuring that those subscriptions are resumed together and
-progress at the same rate. This can be useful for testing purposes,
-or for reconstructing a sequence of events that involve multiple
-subscriptions.</p>
-<p>Notice that sending a <code class="docutils literal"><span class="pre">pause</span></code> command for a subscription does not
-affect messages that have already been sent to a client, and that
-bookmark replays will automatically pace themselves to the rate that
-a client is consuming messages. This means that, while <code class="docutils literal"><span class="pre">pause</span></code>
-can be used to request that AMPS temporarily halt a subscription, this
-option is not recommended to control message rate for a client
-that is oversubscribed.</p>
-<p id="index-11">An application may create a subscription in the paused state by including
-<code class="docutils literal"><span class="pre">pause</span></code> as an option on the initial <code class="docutils literal"><span class="pre">subscribe</span></code> command. To pause an
-active subscription, a subscriber sends a <code class="docutils literal"><span class="pre">subscribe</span></code> command with the
-existing subscription ID and the <code class="docutils literal"><span class="pre">pause</span></code> option. To resume a subscription,
-a subscriber sends a <code class="docutils literal"><span class="pre">subscribe</span></code> command with the subscription ID (or a
-comma-separated list of subscription IDs) and the <code class="docutils literal"><span class="pre">resume</span></code> option. The
-AMPS clients provide convenience functions or constants for the <code class="docutils literal"><span class="pre">pause</span></code>
-and <code class="docutils literal"><span class="pre">resume</span></code> options.</p>
-<p>AMPS allows a given client to pause or resume multiple subscriptions at
-once.</p>
-<p>When multiple bookmark subscriptions are resumed at the same time, AMPS
-will attempt to combine replay for the subscriptions. When AMPS can
-combine replay, AMPS will guarantee that messages across subscriptions
-are delivered from the same replay, which can help to preserve order
-across subscriptions. AMPS can combine subscriptions when they are
-delivered to the same client connection, were paused at the same
-bookmark, delivered at the same rate and are resumed with the same
-command. This feature can be useful for synchronizing message delivery
-across a number of subscriptions. When using <code class="docutils literal"><span class="pre">pause</span></code> and <code class="docutils literal"><span class="pre">resume</span></code>
-for this purpose, an application typically includes the <code class="docutils literal"><span class="pre">pause</span></code> option
-on a number of subscriptions when the subscriptions are created, and
-then resumes the subscriptions when the application is ready to begin
-the replay.</p>
-<p>Pausing a subscription stops AMPS from sending messages to the client
-once the pause command is processed. However, any messages already on
-the network, or in a network buffer on the client or the server will be
-delivered to the client.</p>
-<p>AMPS allows you to begin a subscription in the paused state by providing
-the <code class="docutils literal"><span class="pre">pause</span></code> option when creating the subscription.</p>
-<p>AMPS removes a paused subscription if the subscriber disconnects: for
-restarting a subscription across subscriber restarts, use the basic
-bookmark subscription features as described above.</p>
-</div>
-<div class="section" id="conflation-and-bookmark-subscriptions">
-<h4>Conflation and Bookmark Subscriptions<a class="headerlink" href="#conflation-and-bookmark-subscriptions" title="Permalink to this headline">¶</a></h4>
-<p>AMPS supports subscription conflation for bookmark subscriptions, as
-described in <a class="reference internal" href="pub_sub.html#ug-pub-sub-conflation"><span class="std std-ref">Conflated Subscriptions</span></a>.</p>
-<p>Conflation for bookmark subscriptions works the same way that conflation
-for regular subscriptions works. Messages from the replay are held by
-AMPS for the conflation interval. If during that interval the replay
-finds a message with the same <code class="docutils literal"><span class="pre">conflation_key</span></code> value, AMPS replaces the
-held message with the message from the replay. At the end of the
-conflation interval, AMPS provides the currently held message to the
-subscriber. During replay from the transaction log, the conflation interval
-refers to the timeline of the messages being replayed. That is,
-a conflation interval of <code class="docutils literal"><span class="pre">1s</span></code> will provide conflated messages with
-the same <code class="docutils literal"><span class="pre">conflation_key</span></code> value published during the same second,
-even if during transaction log replay the messages are replayed at a
-much higher rate.</p>
-<p>When using conflation, the bookmark provided on a message post conflation,
-is the bookmark for the <em>first conflated message during the interval</em>
-rather than the message that AMPS delivers at the end of the conflation interval.</p>
-</div>
-<div class="section" id="requesting-message-timestamps">
-<h4>Requesting Message Timestamps<a class="headerlink" href="#requesting-message-timestamps" title="Permalink to this headline">¶</a></h4>
-<p>Messages that are replayed from the transaction log will not have the
-timestamp field of the header populated by default. In order to request
-timestamps, provide the <code class="docutils literal"><span class="pre">timestamp</span></code> option when creating the subscription.</p>
-</div>
-</div>
-<div class="section" id="selecting-message-durability-options">
-<h3>Selecting Message Durability Options<a class="headerlink" href="#selecting-message-durability-options" title="Permalink to this headline">¶</a></h3>
-<p>AMPS supports two distinct options for specifying message durability. By
-default, messages are provided to a bookmark subscription when they are
-persisted to the local transaction log.</p>
-<p>Once replay from the transaction log is finished, AMPS sends messages to
-subscribers as the messages are processed. By default, AMPS waits until
-a message is persisted to the local transaction log before sending the
-message to subscribers. Since each message delivered is persisted,
-this approach ensures that the sequence of messages is consistent for
-this instance across client and server restarts, and that messages that
-are received by a subscriber will be available after a restart.</p>
-<p>AMPS provides options that a subscriber can use to change the point at
-which AMPS delivers messages once replay from the transaction log has
-finished.</p>
-<div class="section" id="using-the-fully-durable-option-for-bookmark-subscriptions">
-<h4>Using the &#8216;fully_durable&#8217; Option for Bookmark Subscriptions<a class="headerlink" href="#using-the-fully-durable-option-for-bookmark-subscriptions" title="Permalink to this headline">¶</a></h4>
-<p>With the <code class="docutils literal"><span class="pre">fully_durable</span></code> option, once replay from the transaction log
-is finished, AMPS sends a message to the subscriber only when the
-message has been persisted in the local transaction log and all
-synchronous downstream replication destinations have acknowledged the
-message. This option is useful for applications where processing of a
-message should not begin until more than one AMPS instance has persisted
-the message.</p>
-<p>This option will typically introduce more latency for incoming messages
-when those messages must be replicated. When this option is used and one
-or more of the synchronous downstream replication destinations that
-receives messages for this topic is offline, the instance will not
-deliver incoming messages until that destination comes back online or is
-downgraded to asynchronous replication.</p>
-</div>
-<div class="section" id="using-the-live-option-for-bookmark-subscriptions">
-<h4>Using the &#8216;live&#8217; Option for Bookmark Subscriptions<a class="headerlink" href="#using-the-live-option-for-bookmark-subscriptions" title="Permalink to this headline">¶</a></h4>
-<p>In some cases, reducing latency may be more important than consistency.
-To support these cases, AMPS provides a <code class="docutils literal"><span class="pre">live</span></code> option on bookmark
-subscriptions. For bookmark subscriptions that use the <code class="docutils literal"><span class="pre">live</span></code> option,
-once replay has finished, AMPS sends messages to subscribers <em>before</em>
-the message has been persisted. This can provide a small reduction in
-latency at the expense of increasing the risk of inconsistency upon
-failover. For example, if a publisher does not republish a message after
-failover, your application may receive a message that is not stored in
-the transaction log and that other applications have not received.</p>
-<div class="admonition warning">
-<p class="first admonition-title">Warning</p>
-<p class="last">The <code class="docutils literal"><span class="pre">live</span></code> option increases the risk of inconsistent data
-between your application and AMPS in the event of a failover. 60East
-recommends using this option only if the risk is acceptable and
-your application requires the small latency reduction this
-option provides.</p>
-</div>
-<p>Since the <code class="docutils literal"><span class="pre">live</span></code> option does not wait for messages to be persisted,
-subscriptions that use this option are subject to slow client offlining
-after replay from the transaction log is complete.</p>
-<p>The <code class="docutils literal"><span class="pre">rate</span></code>, <code class="docutils literal"><span class="pre">pause</span></code>, and <code class="docutils literal"><span class="pre">resume</span></code> options are not supported with
-the <code class="docutils literal"><span class="pre">live</span></code> option.</p>
-</div>
-</div>
-<div class="section" id="managing-journal-files">
-<h3>Managing Journal Files<a class="headerlink" href="#managing-journal-files" title="Permalink to this headline">¶</a></h3>
-<p id="index-12">The design of the journal files for the transaction log are such that
-AMPS can archive, compress and remove these files while AMPS is running.
-AMPS actions provide integrated administration for journal files, as
-described in <a class="reference internal" href="actions.html#ug-actions"><span class="std std-ref">Automating AMPS with Actions</span></a>.</p>
-<p id="index-13">Archiving a file copies the file to an archival directory, typically
-located on higher-capacity but higher-latency storage. Compressing a
-file compresses the file in place. Archived and compressed journal files
-are still accessible to clients for replay and for AMPS to use in
-rebuilding any SOW files that are damaged or removed.</p>
-<p>When defining a policy for archiving, compressing or removing files,
-keep in mind the amount of time for which clients will need to replay
-data. Once journal files have been deleted, the messages in those files
-are no longer available for clients to replay or for AMPS to use in
-recreating a SOW file. If journal files are removed, and a SOW file is
-retained, this means that the SOW may have data that is not in the
-transaction log.</p>
-<div class="admonition caution">
-<p class="first admonition-title">Caution</p>
-<p>While AMPS is running, the <code class="docutils literal"><span class="pre">amps-action-do-remove-journal</span></code>
-action is the only way to safely remove a journal file. This
-action correctly updates the internal AMPS data structures that
-refer to the journal file.</p>
-<p class="last">Likewise, the <code class="docutils literal"><span class="pre">amps-action-do-archive-journal</span></code> action is the
-only way to safely move a journal file to the archive directory
-while AMPS is running, and the <code class="docutils literal"><span class="pre">amps-action-do-compress-journal</span></code>
-action is the only way to safely compress journals while AMPS
-is running.</p>
-</div>
-<p>To determine how best to manage your journal files, consider your
-application&#8217;s access pattern to the recorded messages. Most applications
-have a period of time (often a day or a week) where historical data is
-in heavy use, and a period of time (often a week, or a month) where data
-is infrequently used. One common strategy is to create the journal files
-on high-throughput storage. The files are archived to slower,
-higher-capacity storage after a short period of time, compressed, and
-then removed after a longer period of time. This strategy preserves
-space on high-throughput storage, while still allowing the journals to
-be used. For example, if your applications frequently replay data for
-the last day, occasionally replay data older than the last week, and
-never request data older than one month, a management strategy that
-meets these needs would be to archive files after one day, compress them
-after a week, and remove them after one month. Archival, compression,
-and removal should be done using AMPS actions.</p>
-<div class="admonition caution">
-<p class="first admonition-title">Caution</p>
-<p class="last">If you remove journal files when AMPS is shut down, keep in mind
-that the removal of journal files must be sequential and <em>cannot</em>
-leave gaps in the remaining files. For example, say there
-are three journal files, <code class="docutils literal"><span class="pre">001</span></code>, <code class="docutils literal"><span class="pre">002</span></code> and <code class="docutils literal"><span class="pre">003</span></code>. If only
-<code class="docutils literal"><span class="pre">002</span></code> is removed, then the next AMPS restart could potentially
-overwrite the journal file <code class="docutils literal"><span class="pre">003</span></code>, causing an unrecoverable
-problem.</p>
-</div>
-<p>When using AMPS actions to manage journal files, AMPS ensures that all
-replays from a journal file are complete, all queue messages in that
-journal file have been delivered (and acknowledged, if required), and
-all messages from a journal file have been successfully replicated
-before removing the file.</p>
-<div class="section" id="reference-to-file-types">
-<h4>Reference to File Types<a class="headerlink" href="#reference-to-file-types" title="Permalink to this headline">¶</a></h4>
-<p>AMPS creates the following types of files as part of creating and
-managing the transaction log. Notice that this includes both files that
-contain messages (<em>journal files</em>) and a set of files created by AMPS
-to improve efficiency when the instance is restarting and recovering the
-state of the transaction log.</p>
-<p>The files for a specific instance are prefixed with the instance <code class="docutils literal"><span class="pre">Name</span></code>.
-An AMPS instance will only create files that are prefixed with the <code class="docutils literal"><span class="pre">Name</span></code>
-of the instance, and on startup will only recover files that are prefixed
-with the <code class="docutils literal"><span class="pre">Name</span></code> of the instance.</p>
-<table border="1" class="docutils" id="id1">
-<caption><span class="caption-text"><em>Files created by AMPS when a transaction log is configured</em></span><a class="headerlink" href="#id1" title="Permalink to this table">¶</a></caption>
-<colgroup>
-<col width="33%" />
-<col width="33%" />
-<col width="33%" />
-</colgroup>
-<thead valign="bottom">
-<tr class="row-odd"><th class="head">Extension</th>
-<th class="head">File Type</th>
-<th class="head">Description</th>
-</tr>
-</thead>
-<tbody valign="top">
-<tr class="row-even"><td><code class="docutils literal"><span class="pre">.journal</span></code></td>
-<td>Journal file</td>
-<td><p class="first">These files contain the messages that
-comprise the transaction log.</p>
-<p class="last">AMPS always writes new messages to an
-uncompressed journal file.</p>
-</td>
-</tr>
-<tr class="row-odd"><td><code class="docutils literal"><span class="pre">.journal.gz</span></code></td>
-<td>Compressed
-journal file</td>
-<td><p class="first">These files contain messages that
-comprise the transaction log.</p>
-<p class="last">These files have been compressed by AMPS as
-a result of the
-<code class="docutils literal"><span class="pre">amps-action-do-compress-journal</span></code>
-action. Other than being compressed, they
-are treated identically to uncompressed
-journal files.</p>
-</td>
-</tr>
-<tr class="row-even"><td><code class="docutils literal"><span class="pre">.index.gz</span></code></td>
-<td>Journal index
-file</td>
-<td><p class="first">These files are used during
-recovery to help AMPS quickly rebuild
-its references to the content of the
-transaction log without having to completely
-reprocess each file. Each index file contains
-index information for the corresponding
-journal file.</p>
-<p class="last">These files do not contain messages.</p>
-</td>
-</tr>
-<tr class="row-odd"><td><code class="docutils literal"><span class="pre">.topic.index</span></code></td>
-<td>Topic index
-file</td>
-<td><p class="first">These files are used during
-replay to help AMPS quickly locate messages
-for a given topic. Creating a topic index
-is optional, and is enabled through instance
-level <code class="docutils literal"><span class="pre">Tuning</span></code> configuration.</p>
-<p class="last">These files do not contain messages.</p>
-</td>
-</tr>
-<tr class="row-even"><td><code class="docutils literal"><span class="pre">.clients.ack</span></code></td>
-<td>Clients
-acknowledgment
-cache</td>
-<td><p class="first">Used during recovery to help AMPS quickly
-identify the last message persisted from
-each publisher without having to reprocess
-each journal file.</p>
-<p class="last">These files do not contain messages.</p>
-</td>
-</tr>
-<tr class="row-odd"><td><code class="docutils literal"><span class="pre">.queues.ack</span></code></td>
-<td>Queue
-acknowledgment
-cache</td>
-<td><p class="first">For each queue, this file stores the point in
-the transaction log for which that queue has
-been completely processed (that is, all
-messages prior to that point in the transaction
-log have been acknowledged or expired). On
-recovery, AMPS can begin restoring the state
-of the queue from that point rather than
-reprocessing the entire transaction log.</p>
-<p class="last">These files do not contain messages.</p>
-</td>
-</tr>
-<tr class="row-even"><td><code class="docutils literal"><span class="pre">.queue.cache</span></code></td>
-<td>Queue
-metadata
-cache</td>
-<td><p class="first">For a queue that specifies that metadata
-be cached in a file, this is the file that
-contains the cache. If this file is removed,
-the queue state will be restored from the
-transaction log (using the recovery point
-stored in the <code class="docutils literal"><span class="pre">queues.ack</span></code> file).</p>
-<p class="last">These files do not contain messages or message
-headers. They contain delivery state for
-messages in the queue and the location of those
-messages in the transaction log.</p>
-</td>
-</tr>
-</tbody>
-</table>
-</div>
-</div>
-</div>
-<div class="section" id="using-amps-grep-to-find-messages-in-the-transaction-log">
-<span id="txlog-finding-entries"></span><h2>Using amps-grep to Find Messages in the Transaction Log<a class="headerlink" href="#using-amps-grep-to-find-messages-in-the-transaction-log" title="Permalink to this headline">¶</a></h2>
-<p>The transaction log maintains a full record of the messages published
-to the topics recorded in the transaction log.</p>
-<p>Finding an exact sequence of messages recorded by an instance can
-be useful in troubleshooting issues with applications or AMPS itself.
-The simplest way to find a sequence of messages in the AMPS
-transaction log is by using the <code class="docutils literal"><span class="pre">amps-grep</span></code> utility.</p>
-<p>When used to find records in a transaction log, <code class="docutils literal"><span class="pre">amps-grep</span></code> also
-includes the header information for each journal file in the output,
-to help in cases where the intent of searching the journal is to
-be able to preserve a sequence of messages or analyze the journals
-further.</p>
-<div class="section" id="finding-messages-from-a-specific-client">
-<h3>Finding Messages from a Specific Client<a class="headerlink" href="#finding-messages-from-a-specific-client" title="Permalink to this headline">¶</a></h3>
-<p>When troubleshooting a publisher, it&#8217;s often useful to be able
-to see the set of messages published by a particular publisher.</p>
-<p>To find messages published from a specific publisher, use the
-following general pattern:</p>
-<pre>$amps-grep --client=<em>client_name</em> <em>journal_directory</em>/*.journal &gt; out.txt</pre><p>The <code class="docutils literal"><span class="pre">amps-grep</span></code> command calculates the set of possible client name hashes for the
-given client name and returns the transaction log entries for those messages.
-The Linux shell then writes those entries to the <code class="docutils literal"><span class="pre">out.txt</span></code> file.</p>
-</div>
-<div class="section" id="finding-messages-for-a-specific-topic">
-<h3>Finding Messages for a Specific Topic<a class="headerlink" href="#finding-messages-for-a-specific-topic" title="Permalink to this headline">¶</a></h3>
-<p>When troubleshooting message flow on a given topic, it&#8217;s often useful to be able
-to see the full set of messages published to that topic.</p>
-<p>To find messages published to a specific topic, provide the topic name to the <code class="docutils literal"><span class="pre">amps-grep</span></code>
-utility as follows:</p>
-<pre>$amps-grep <em>topic_name</em> <em>journal_directory</em>/*.journal &gt; out.txt</pre><p>The <code class="docutils literal"><span class="pre">amps-grep</span></code> command searches the journal files for all occurrences of the <em>topic_name</em> provided
-and returns the transaction log entries for those messages.</p>
-</div>
-<div class="section" id="finding-messages-with-specific-data">
-<h3>Finding Messages with Specific Data<a class="headerlink" href="#finding-messages-with-specific-data" title="Permalink to this headline">¶</a></h3>
-<p>In some cases, it can also be useful to find records with specific data. For example,
-this might be helpful to see the set of publishes to a specific key within a SOW topic,
-regardless of the publisher.</p>
-<p>To find messages with specific data, provide the data to the <code class="docutils literal"><span class="pre">amps-grep</span></code> utility as
-follows:</p>
-<pre>$amps-grep <em>data</em> <em>journal_directory</em>/*.journal &gt; out.txt</pre><p>The <code class="docutils literal"><span class="pre">amps-grep</span></code> command searches the journal files for all occurrences of the <em>data</em> provided
-and returns the transaction log entries for those messages.</p>
-</div>
-</div>
-</div>
-          </div>
-        </div>
-      </div>
-    <div class="clearer"></div>
-  </div>
-    <div class="footer">
-      &copy;2023 60East Technologies, Inc. (version develop).
-      |
-      Powered by <a href="http://sphinx-doc.org/">Sphinx 1.5.3</a>
-      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
-    </div>
-  </body>
-</html>

amps-python-client 5.3.4.1__zip → 5.3.4.2__zip

amps-python-client 5.3.4.1zip → 5.3.4.2zip