amps-python-client 5.3.4.1__zip

amps-python-client-5.3.4.1/docs/user-guide/html/_sources/chapters/delta.rst.txt

@@ -0,0 +1,556 @@
+.. index:: delta, incremental message update, Command; delta publish,
+
+.. _#ug-delta:
+
+.. _delta:
+
+Delta Messaging
+===============
+
+AMPS delta messaging allows applications to work with only the changed
+parts of a message in the SOW. In high-performance messaging, it's
+important that applications not waste time or bandwidth for messages
+that they aren't going to use.
+
+Delta messaging has two distinct aspects:
+
+1. *Delta Subscribe* - allows subscribers to receive just the fields that
+   are updated within a message.
+
+2. *Delta Publish* - allows publishers to update and add fields within a
+   message by publishing only the updates into the SOW.
+
+While these features are often used together, the features are
+independent. For example, a subscriber can request a regular
+subscription even if a publisher is publishing deltas. Likewise, a
+subscriber can request a delta subscription even if a publisher is
+publishing full messages.
+
+To be able to use delta messages, the message type for the subscription
+must support delta messages. All of the included AMPS message types,
+except for ``binary`` and ``struct``, support delta messages, with the
+limitations described in each section below. For custom message types,
+contact the message type implementer to determine whether delta support is
+implemented.
+
+These features can often improve performance in environments where
+bandwidth is at a premium. Since these features require AMPS to parse,
+compare, and create messages, these features can consume somewhat more
+CPU on the AMPS server than a simple publish or subscribe, particularly
+for large messages with complex structure (such as deeply-nested XML).
+
+Delta Subscribe
+----------------
+
+Delta subscribe allows applications to receive only the changed parts of
+a message when an update is made to a record in the SOW. When a delta
+subscription is active, AMPS compares the new state of the message to
+the old state of the message, creates a message for the difference, and
+sends the difference message to subscribers. Using this approach can
+simplify processing on the client side, and can improve performance when
+network bandwidth is the most important constraint.
+
+For example, consider a SOW that contains the following messages, with
+the ``order`` field as the key of the SOW topic:
+
+.. image:: ../png/delta_original_messages.png
+
+Now, consider an update that changes the status of order number 3:
+
+.. code-block:: javascript
+
+    {
+        "order":3,
+        "customer":"Patrick",
+        "status":"pending",
+        "qty":1000,
+        "ticker":"MSFT"
+    }
+
+For a regular subscription, subscribers receive the entire message. With
+a delta subscription, subscribers receive just the key of the SOW topic
+and any changed fields:
+
+.. image:: ../png/delta_update_message.png
+
+This can significantly reduce the amount of network traffic,
+and can also simplify processing for subscribers, since the only
+information sent is the information needed by the subscriber to take
+action on the message.
+
+When the ``oof`` option is specified on a delta subscription, AMPS will deliver
+the full message when a previously out-of-focus message comes into focus,
+even if only some of the fields in the message have changed.
+
+Using Delta Subscribe
+^^^^^^^^^^^^^^^^^^^^^^
+
+Since a client must process delta subscriptions using substantially
+different logic than regular subscriptions, delta subscription is
+implemented as a separate set of AMPS commands rather than simply as an
+option on subscribe commands. AMPS supports two different ways to
+request a delta subscription:
+
+.. list-table:: *Delta subscribe commands*
+     :header-rows: 1
+
+     * - Command
+
+       - Result
+
+     * - ``delta_subscribe``
+         
+
+       - Register a delta subscription,
+         starting with newly received
+         messages.
+
+     * - ``sow_and_delta_subscribe``
+         
+
+       - Replay the state of the SOW and
+         atomically register a delta
+         subscription.
+
+
+Applications most commonly use ``sow_and_delta_subscribe`` to receive
+the current state of messages in the SOW before they begin receiving
+deltas.
+
+Options for Delta Subscribe
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The delta subscribe command accepts several options that are unique to
+delta subscriptions. These options control the precise behavior of delta
+messages:
+
+.. list-table:: *Options for delta subscribe*
+     :header-rows: 1
+
+     * - Option
+
+       - Result
+
+     * - ``no_empties``
+         
+
+       - Do not send messages if no data fields
+         have been updated.
+         
+         By default, AMPS will publish a delta
+         for every publish to the record, even if
+         the data has not changed. By specifying
+         this option, AMPS will only send messages
+         when there is changed data.
+
+     * - ``no_sowkey``
+         
+
+       - Do not include the AMPS generated SowKey
+         with messages.
+         
+         By default, AMPS includes this key to help
+         you identify unique records within the SOW.
+
+     * - ``send_keys``
+         
+
+       - Include the SOW key fields in the message.
+         
+         Since the SOW key fields indicate which
+         message to update, without this option,
+         updates to delta messages will never
+         contain the SOW key fields. For views, the
+         SOW key fields are the fields specified in
+         the ``Grouping`` element.
+         
+         *AMPS accepts this option for backward
+         compatibility. As of AMPS 4.0, this option
+         is included on delta subscriptions by
+         default.*
+
+     * - ``oof``
+         
+
+       - AMPS will deliver out of focus messages on
+         this subscription.
+         
+         When focus tracking is enabled, AMPS will
+         also deliver the full message to a
+         subscription when a previously
+         out-of-focus message comes into focus.
+
+
+
+Delta subscriptions also support the options provided for regular
+subscriptions, including the timestamp option and the conflation options
+described in the section on :ref:`#ug-pub-sub-conflation`.
+
+
+Identifying Changed Records
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When an application that uses delta subscriptions receives a message,
+that message can either be a new record or an update to an existing
+record. AMPS offers two strategies for an application to tell whether
+the record is a new record or an existing record, and identify which
+record has changed if the message is an update to an existing record.
+
+The two basic approaches are as follows:
+
+1. By default, each message delivered through a delta subscription
+   contains a SowKey header field. This field is the identifier that
+   AMPS assigns to track a distinct record in the SOW. If the
+   application has previously received a SowKey with that value, then
+   the new message is an update to the record with that SowKey value. If
+   the application has not previously received a SowKey with that value,
+   then the new message contains a new record.
+
+2. Delta messages can also contain the key fields from the SOW in the
+   body of the message. This is controlled by the ``send_keys`` option
+   on the subscription, which is always enabled as of AMPS 4.0. With
+   this approach, the application parses the body of the message to find
+   the key. If the application has previously received the key, then the
+   message is an update to that existing record. Otherwise, the message
+   contains a new record.
+
+In either case, AMPS delivers the information the application needs to
+determine if the record is new or changed. The application chooses how
+to interpret that information, and what actions to take based on the
+changes to the record.
+
+AMPS also supports out-of-focus notification for delta subscriptions, as
+described in :ref:`#ug-oof`. If your
+application needs to know when a
+record is deleted, expires, or no longer matches a subscription, you can
+use out-of-focus messages to be notified.
+
+Conflated Subscriptions and Delta Subscribe
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+AMPS provides subscription conflation on delta subscriptions. When
+conflation is enabled, each delta message during the conflation interval
+is merged into the conflated message. The message that is delivered is
+the merge of all of the deltas that arrived during the conflation
+interval.
+
+Since AMPS combines successive delta messages into a single update, a
+delta subscription that uses conflation may receive values that are
+identical to the previous values. For example, consider the following
+record in a SOW that uses ``/id`` as the key:
+
+.. code-block:: javascript
+
+    {
+        "id": 99,
+        "status":"open",
+        "notes":"none",
+        "xref":82
+    }
+
+Assume that the following updates to the record are published during the
+conflation interval:
+
+.. code-block:: javascript
+
+    {"id": 99, "status":"questioned", "notes":"none", "xref":82}
+    {"id": 99, "status":"questioned", "notes":"jcarlo hold", "xref":82}
+    {"id": 99, "status":"cleared", "notes":"none", "xref":82}
+    {"id": 99, "status":"open", "notes":"none", "xref":82}
+
+At the end of the conflation interval, the subscription will receive the
+delta message:
+
+.. code-block:: javascript
+
+    {
+        "id": 99,
+        "status":"open",
+        "notes":"none"
+    }
+
+The ``/id`` field is included because that field is the key of the SOW,
+and all of the delta messages produced during the conflation interval
+contained that key. The ``/status`` and ``/notes`` fields are included
+because there were changes to these values during the conflation
+interval. The delta messages produced during the conflation interval
+contained changed values, so the merged update contains those fields and
+the state of the values at the end of the conflation interval. The
+``/xref`` field is not included, because none of the delta messages
+produced during the conflation interval contained that field.
+
+Select Lists and Delta Subscribe
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When a ``delta_subscribe`` or ``sow_and_delta_subscribe`` command provides a
+select list and the ``no_empties`` option, only changes to fields included in the
+select list will prompt a publish to the client. Changes to fields that the subscriber does not receive
+do not produce a publish.
+
+Delta Subscribe Support
+~~~~~~~~~~~~~~~~~~~~~~~
+
+To produce delta messages, the message type and the topic must both
+support delta subscribe. When this is not the case, AMPS accepts the
+subscription, but provides full messages rather than delta messages.
+
+All of the basic message types provided with AMPS support delta
+subscribe with the exception of the ``binary`` message type, ``struct`` message
+types, and protobuf message types that use protobuf version 3. Composite message
+types support delta subscribe if they use the ``composite-local`` definition,
+as described in the section on :ref:`Composite Message Types <#ug-messagetypes-composite>`.
+
+AMPS queues do not support delta subscribe. AMPS accepts a delta
+subscription for a queue, but produces full messages from the queue.
+
+Bookmark subscriptions do not support delta subscribe. AMPS does not accept 
+a delta bookmark subscription.
+
+All other AMPS topic types that are based on a SOW support delta
+subscribe. AMPS topics that do not use a SOW do not support delta
+subscribe, and instead produce full messages.
+
+Multiple Subscriptions and Delta Subscribe
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When a single connection to AMPS has multiple subscriptions, AMPS sends
+the message to that client once, with information on the set of
+subscriptions that match. AMPS sends a message that will include the
+requested data for all of the matching subscriptions. For example, if a
+message matches one subscription that requests full messages and another
+subscription from the same connection that requests deltas, both
+subscriptions will receive a full message. If your application depends
+on receiving deltas, take care that the application does not issue
+non-delta subscriptions for the same set of messages on the same
+connection.
+
+Delta Publish
+-------------
+
+Delta publish allows publishers to update a message in the SOW by
+providing just the key fields for the SOW and the data to update. When
+AMPS receives a delta publish, AMPS parses the incoming message and the
+existing messages, identifies changed fields, and creates an updated
+message that merges changed fields from the delta publish into the
+existing message. If there is no existing message, the result of a delta
+publish is the same as if the update were merged into an empty message:
+the fields in the delta publish, and only those fields, are present
+in the result.
+
+This can be particularly useful in cases where more than one worker acts
+on a record. For example, an order fulfillment application may need to
+check inventory, to ensure that the order is available, and check credit
+to be sure that the customer is approved for the order. These checks may
+be run in parallel, by different worker processes. With delta publish,
+each worker process updates the part of the record that the worker is
+responsible for, without affecting any other part of the record. Delta
+publish saves the worker from having to query the record and construct a
+full update, and eliminates the possibility of incorrect updates when
+two workers try to update the record at the same time.
+
+For example, consider an order published to the SOW:
+
+.. code-block:: javascript
+
+    {
+        "id":735,
+        "customer":"Patrick",
+        "item":90123,
+        "qty":1000,
+        "state":"new"
+    }
+
+Using delta publishing, two independent workers can operate on the
+record in parallel, safely making updates and preparing the record for a
+final fulfillment process.
+
+The inventory worker process is responsible for checking inventory. This
+worker subscribes to messages where
+the ``/state = 'new' AND /inventory IS NULL AND /credit IS NULL``. This
+process receives the new message and verifies that the inventory system
+contains 1000 of the item # 90123. When it verifies this, it uses delta
+publish to publish the following update:
+
+.. code-block:: javascript
+
+    {
+        "id":735,
+        "inventory":"available"
+    }
+
+The credit worker process verifies that the customer is permitted to
+bill for the total amount. Like the inventory worker, this worker
+subscribes to messages where the
+``/state = 'new' and /inventory IS NULL and /credit IS NULL``. This
+process receives the new message and verifies that the customer is
+allowed to bill the total value of the order. When the check is
+complete, the credit worker publishes this message:
+
+.. code-block:: javascript
+
+    {
+        "id":735,
+        "credit":"approved"
+    }
+
+After both of these processes run, the SOW contains the following
+record:
+
+.. code-block:: javascript
+
+    {
+        "id": 735,
+        "credit": "approved",
+        "inventory": "available",
+        "customer": "Patrick",
+        "item": 90123,
+        "qty": 1000,
+        "state": "new"
+    }
+
+The fulfillment worker would subscribe to messages where
+``/state = 'new' AND /inventory IS NOT NULL AND /credit IS NOT NULL``.
+
+Using Delta Publish
+^^^^^^^^^^^^^^^^^^^
+
+Since delta messages must be processed and merged into the existing
+SOW record, AMPS provides a distinct command for delta publish.
+
+.. list-table:: *Delta publish command*
+     :header-rows: 1
+
+     * - Command
+
+       - Result
+
+     * - ``delta_publish``
+         
+
+       - Publish a delta message. If no record exists in the
+         SOW, add the message to the SOW. If a record exists
+         in the SOW, merge the data from this record into the
+         existing record.
+
+
+Understanding Delta Publish
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When AMPS receives a delta publish request, AMPS first performs any preprocessing
+or enrichment specified for the incoming message. AMPS then fully parses both
+the incoming message and the destination message, and then merges the contents
+of the incoming update into the existing message. The mechanism that
+merges the messages is message-type independent, and does not rely on
+the syntax or semantics of any particular message type.
+
+The message is merged before the message is persisted to the SOW,
+recorded in the transaction log, replicated, or delivered to subscribers.
+
+Since delta publish uses a very simple, compact syntax  -- a partial message
+-- for updates, AMPS makes certain assumptions about the intent of an update:
+
+1. AMPS replaces the contents of the existing message based on the
+   parsed identifiers in the update. An update to an anonymous element
+   or the root level of a subdocument is a *full replacement* of that
+   value.
+
+   For example, consider the following JSON document::
+
+       {"id":42, "contents":{"packages":[{"box":"chocolates"},
+                                         {"bowl":"noodles"}]}}
+
+   An update to the ``packages`` field of the document *replaces* the
+   subdocument, so providing the following ``delta_publish``::
+
+        {"id":42, "contents":{"packages":[{"basket":"eggs"}]}}
+
+   Replaces the previous value of ``packages``, and produces the message::
+
+        {"id":42, "contents":{"packages":[{"basket":"eggs"}]}}
+
+2. A missing field in a delta publish means that the content
+   of that field is unchanged.
+
+   Since AMPS treats the absence of a field in a delta update to
+   mean that the previous value is unchanged, a ``delta_publish``
+   cannot be used to remove fields from a document.
+
+   For example, consider the following JSON document::
+
+      {"id":42", "flowers":"roses"}
+
+   A ``delta_publish`` that attempts to remove the ``flowers`` field
+   by simply removing it will be treated as though there is no update
+   to that field. In other words, a ``delta_publish`` of::
+
+      {"id":42}
+
+   will produce no change to the ``flowers`` field, and result in a
+   document of::
+
+      {"id":42, "flowers":"roses"}
+
+   To remove a field from a message, republish the full message.
+
+3. AMPS does not evaluate whether a field has a different value than
+   the previous message or has a republish of the previous value
+   when processing a ``delta_publish`` -- any field present in the
+   delta publish will replace the corresponding field in the existing
+   message.
+
+   For example, consider an application that advertises whether
+   a given system is accepting requests with messages along the
+   lines of::
+
+      {"systemId":"A34-Astro", "status":"available",
+                               "commandAccepted":["print","scan","fax"]}
+
+   Now, the system sends a delta publish update that does not change the status::
+
+     {"systemId":"A34-Astr", "status":"available"}
+
+   Even though no value in the message has changed, the publish is still
+   processed, and the publish will still be delivered to subscribers.
+   The fact that it replaces an existing value with the same value
+   does not matter for AMPS. This is still considered an update. (An individual
+   subscriber can choose not to receive updates that do not change the value of
+   a field by using the ``no_empties`` option with a delta subscription.)
+
+
+Delta Publish Support
+~~~~~~~~~~~~~~~~~~~~~~
+
+To accept delta publishes, the message type and the topic must both
+support delta publish. When this is not the case, AMPS accepts the
+publish, but may not produce the expected results.
+
+All of the basic message types provided with AMPS support delta publish
+with the exception of the ``binary`` message type and protobuf message types
+that use protobuf version 3. Composite message types support delta publish
+if they use the ``composite-local`` definition, as described in the section
+on composite message types. Types that do not support delta publish, produce
+the full, literal message provided with a delta publish command (rather than
+merging the publish into the previous state of the message).
+
+When a topic uses the ``composite-local`` message type, parts of the
+composite that are provided as empty (that is, zero-length) are
+considered to be unchanged, and the merged message contains the existing
+contents of that part. This provides a convenient way to update only one
+part of a composite message, without having to republish data that has
+not changed. For example, a ``composite-local`` type contains a JSON
+part and a binary part can modify the JSON part without having to
+republish the full binary part.
+
+AMPS queues support delta publish to an underlying topic, if that
+underlying topic maintains a SOW. The merged message is provided to the
+AMPS queue.
+
+All other AMPS topic types that are based on a SOW and accept publish
+commands support delta publish. AMPS topics that do not use a SOW do not
+support delta publish, so publishing a delta message to those topics
+produces the full, literal message from the publish command rather than
+a merged message. Without a SOW configured for the topic, AMPS does not
+track the current value of a message, and therefore does not have a way
+to merge the publish into an existing message.
+
+