amps-python-client 5.3.4.1__zip → 5.3.4.3__zip

amps-python-client-5.3.4.1/docs/user-guide/html/_sources/chapters/messageTypes.rst.txt

@@ -1,1398 +0,0 @@
-.. index:: Message types, Message types; FIX, Message types; NVFIX,
-   Message types; XML, Message types; JSON, Message types; BSON, Message
-   types; composite, Message types; protobuf, Message types; msgpack
-
-.. _#ug-messagetypes:
-
-.. _messageTypes:
-
-Message Types
-==================
-
-Message communication between the publisher and subscriber in AMPS is
-managed through the use of message types. Message types define the data
-contained within an AMPS message. Each topic has a specific message
-type. Transports used for publishers and subscribers can also define
-specific message types. For a given transport, AMPS will only process
-messages of the type or types that the transport accepts.
-
-When AMPS needs to use the data within a message, AMPS uses the message
-type to parse the message into an internal representation. AMPS uses the
-same internal representation for all message types. Likewise, if AMPS
-needs to create a new message from a set of values (for example, for a
-view), AMPS uses the message type to serialize that set of values into
-the correct format. AMPS filters, commands, processing flow, and so
-forth are the same for every message type. Message types do not change
-how AMPS processes messages. A message type simply allows AMPS to work
-with data of a particular format.
-
-In some cases, a given message type cannot support all of the
-capabilities in AMPS. For example, the unparsed ``binary`` message type
-allows arbitrary payloads. This can be extremely useful, but because
-there is no set format for that message type, none of the capabilities
-that rely on parsing data are supported by the ``binary`` message type.
-Where a message type cannot provide a specific capability to AMPS, those
-limitations are described below.
-
-Except where limitations are described in this section, all message
-types provided with the AMPS server support all AMPS features. The AMPS
-engine itself is message-type agnostic. There is no difference in
-configuring a SOW that uses a composite type than there is configuring a
-SOW that uses JSON, or BFlat, or Google Protocol buffers.
-
-Message types in AMPS are implemented as plug-in modules. For more
-information on plug-in modules, contact 60East support for access to the
-AMPS Server SDK.
-
-Default Message Types
-----------------------
-
-AMPS automatically loads modules for the following message types:
-
-.. list-table:: *AMPS Default Message Types*
-     :header-rows: 1
-
-     * - Message Type
-
-       - Description
-
-     * - ``bson``
-
-       - Binary JSON (BSON) messages. See http://www.bsonspec.org for information on this format.
-
-     * - ``bflat``
-         
-
-       - BFlat, a schemaless message format based on key-value pairs that includes support for binary
-         representations of numeric data. See http://bflat.io for information on this format.
-
-     * - ``fix``
-         
-
-       - FIX messages using numeric tags. FIX is a standard format widely used in the financial
-         industry. See https://www.fixtrading.org/what-is-fix/ for more information on
-         this format.
-
-     * - ``json``
-         
-
-       - JSON (JavaScript Object Notation) messages. See http://www.json.org for information on this
-         format.
-
-     * - ``msgpack``
-         
-
-       - MessagePack messages. MessagePack is a schemaless serialization format designed to
-         efficiently encode data. See http://msgpack.org/index.html for more information on
-         MessagePack.
-
-     * - ``nvfix``
-         
-
-       - NVFIX (name/value FIX) messages. NVFIX uses the same basic format as FIX, but allows tags to
-         contain any byte that is not ``=`` or the configured field separator character (by default,
-         the ASCII ``SOH`` character.) By contrast, FIX requires that tags are numeric.
-
-     * - ``xml``
-
-       - XML messages (of any schema)
-
-     * - ``binary``
-         
-
-       - Uninterpreted binary payload. Because this module does not attempt to parse the payload, it
-         does not support content filtering, views and aggregates. Likewise, because there is no set
-         format for the payload, this message type cannot support features that construct messages
-         (such as delta messaging, ``/AMPS/.*`` topic subscriptions and ``stats`` acks).
-
-     * - ``protobuf``
-         
-
-       - Google protocol buffer messages. To use this message type, you must configure a
-         ``MessageType`` with the format of the messages (the ``.proto`` files).
-
-
-With these message types, AMPS automatically loads the module that
-provides the message type. AMPS declares message types for all of the
-above message types except for ``protobuf``.
-
-.. index:: message parsing, message validation; not enforced by AMPS,
-
-For efficiency, AMPS only parses the content of a message if required,
-and only to the extent required. For example, if AMPS only needs to find
-the ``id`` tag in an NVFIX message, AMPS will not fully parse the
-message, but will stop parsing the message after finding the ``id`` tag.
-This provides significant performance improvements, and also means that
-AMPS does not verify the format or validity of messages unless it needs
-to parse the messages. When AMPS parses a message, it may only partially
-parse a message, and may not detect corruption or invalid format in a
-message if that corruption occurs after the point at which AMPS has all
-of the required information from the message.
-
-The FIX and NVFIX message types support configuration of the field and
-message delimiters.
-
-AMPS also allows you to create new message types by assembling existing
-message types into a composite message. Composite message types are
-described in :ref:`Composite Messages <#ug-messagetypes-composite>`, and
-require additional configuration:
-
-.. list-table:: *AMPS Composite Message Types*
-     :header-rows: 1
-
-     * - Message Type Name
-
-       - Description
-
-     * - ``composite-global``
-         
-
-       - Composite message type that combines message parts for content filtering. This
-         message type combines one or more existing message types into a message. This
-         type is described in more detail in
-         :ref:`Composite Messages<#ug-messagetypes-composite>`.
-
-     * - ``composite-local``
-         
-
-       - Composite message type, filterable by individual parts. This message type
-         combines one or more existing message types into a message. This type is
-         described in more detail in
-         :ref:`Composite Messages <#ug-messagetypes-composite>`.
-
-
-.. index:: bflat,
-
-BFlat Messages
----------------
-
-The BFlat message format combines the simplicity and efficiency of
-simple, schemaless data formats such as FIX and NVFIX with the ability
-to manage binary data and preserve the full precision of numeric values.
-BFlat is especially useful for applications that deal with binary data
-or precise numeric values while demanding high levels of throughput.
-
-A BFlat message is composed of any number of tag/value pairs, similar to
-FIX and NVFIX messages. Tags and values can contain any value and can
-be of any length. Unlike formats such as FIX, there are no reserved
-characters. In practical terms, the name of a tag must be a valid XPath
-identifier to filter the message in AMPS. However, this is a limitation
-of XPath and not of the BFlat message format.
-
-The BFlat message type supports all AMPS features and there are no
-special considerations when using the BFlat message type.
-
-Open-source libraries for producing and parsing BFlat messages are available
-from the `BFlat project <http://bflat.io>`_ site.
-
-BFlat Data Types
-^^^^^^^^^^^^^^^^^
-
-BFlat messages are strongly typed.  For numeric values, BFlat can preserve
-the precise value of the following numeric types:
-
-.. list-table:: *BFlat Numeric Types*
-     :header-rows: 1
-     :widths: 25 75
-
-
-     * - Type
-
-       - Description
-
-     * - int8
-
-       - 8-bit integer
-
-     * - int16
-
-       - 16-bit integer
-
-     * - int32
-
-       - 32-bit integer
-
-     * - int64
-
-       - 64-bit integer
-
-     * - double
-         
-
-       - 64-bit IEEE 754 floating point
-         number
-
-     * - datetime
-         
-
-       - UTC datetime containing milliseconds
-         since Unix epoch (64-bit
-         representation)
-
-     * - leb128
-         
-
-       - Signed LEB128 integer (variable
-         length)
-
-
-BFlat also includes the following non-numeric types:
-
-.. list-table:: *BFlat non-numeric Types*
-     :header-rows: 1
-     :widths: 25 75
-
-     * - Type
-
-       - Description
-
-     * - null
-
-       - Empty field
-
-     * - string
-         
-
-       - String of bytes
-         
-         BFlat does not specify the encoding
-         of a string; this is up to the
-         application to determine.
-
-     * - binary
-
-       - Untyped sequence of bytes
-
-
-BFlat includes the ability to represent arrays of values.
-
-
-Representing Data in BFlat
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-By convention, BFlat serializers should choose the most compact representation of a
-given value. For example, if an integer value can fit into 8 bits, it should be
-serialized as a value of type ``int8``. An application that parses BFlat should
-assume that a serializer may use different types for values in the same field,
-to optimize for a compact representation, and should not rely on a value being
-of a specific type.
-
-For example, a ``TotalValue`` field that can have any integer value could
-be serialized with a type of ``int8`` for values that fit in 8 bits, but could
-be serialized with a integer type that includes more bits if the value is larger,
-and could be serialized with the ``leb128`` type for integer values that cannot
-be represented in 64 bits.
-
-When the AMPS server must serialize a BFlat message (for example, when projecting
-a view or as a result of enrichment), AMPS tries to optimize for the most compact message size,
-which could change the representation if the serializer did not choose the most
-compact type for a value.
-
-
-
-.. index:: messagepack, msgpack
-
-MessagePack Messages
---------------------
-
-AMPS fully supports MessagePack messages, with the following
-implementation decisions to represent MessagePack messages in
-the AMPS type system. See :ref:`#ug-expressions-data-types` for
-information on the AMPS data types. Notice, in particular, that
-the AMPS expression language supports automatic type conversion,
-so while this table shows the default AMPS representation for
-a given MessagePack type, AMPS will convert a value as needed
-once the message has been parsed.
-
-.. list-table:: *MessagePack Types and AMPS Types*
-     :header-rows: 1
-
-     * - MessagePack Type
-
-       - AMPS Representation
-
-     * - nil
-
-       - NULL
-
-     * - bool
-
-       - Boolean
-
-     * - int (all widths)
-
-       - Integer
-
-     * - float (all widths)
-
-       - Float
-
-     * - str (all widths)
-
-       - String
-
-     * - bin (all widths)
-
-       - String
-
-     * - array (all widths)
-
-       - array of AMPS values
-
-     * - map (all widths)
-
-       - nested AMPS values
-
-     * - ext (all widths)
-
-       - String
-
-
-Notice that AMPS does not attempt to interpret extension types, and instead
-represents them as arrays of bytes (a String in the AMPS type system).
-
-.. _#ug-messagetypes-composite:
-
-Composite Messages
--------------------
-
-Sometimes, applications only need to filter on a small subset of the
-fields in a message. Sometimes applications need to send and receive
-messages that cannot be meaningfully parsed by AMPS, such as images or
-audio files. For these cases, AMPS provides a composite message type
-that lets you create a new message type by combining existing message
-types.
-
-For example, you might create a message type that includes three parts:
-the metadata for an image as a ``json`` document, a small JPG thumbnail
-as a ``binary`` message part, and a full size PNG image as another
-``binary`` message part.
-
-Composite messages can also be useful when the message itself is large
-or resource-intensive to parse. In this case, you can create a message
-type that includes the information needed to filter messages in a JSON
-or NVFIX part, and include the full message in the unparsed payload of
-the composite message, as described below.
-
-AMPS provides two different types of composite messages. Messages
-created using the ``composite-local`` module preserve information about
-the individual parts for filtering, aggregation, and projection.
-Messages created using the ``composite-global`` module treat the
-individual parts as elements of a single document.
-
-
-.. index:: unparsed payload,
-
-
-Configuring Composite Message Types
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To use a composite message type, you must first configure the type by
-declaring it in the ``MessageTypes`` section of the AMPS configuration
-file. The declaration contains the name of the new composite message
-type, specifies that the new type is composite, and lists the parts of
-the composite message type.
-
-For example, the ``MessageType`` element below declares a new composite
-message type named ``images``. The new type contains a ``json`` document
-at the beginning of the message, followed by two uninterpreted binary
-message parts. AMPS will combine the XPath identifiers for all message
-parts into a single set of identifiers. Notice that, because only one
-part of the message type is parsable, using ``composite-global``
-simplifies the identifiers for the message.
-
-.. code-block:: xml
-
-    <MessageTypes>
-        ...
-        
-        <MessageType>
-            <Name>images</Name>
-            <Module>composite-global</Module>
-            <MessageType>json</MessageType>
-            <MessageType>binary</MessageType>
-            <MessageType>binary</MessageType>
-        </MessageType>
-
-        ...
-
-    </MessageTypes>
-
-The ``MessageType`` entries for the composite message can be any AMPS
-message type, including both the built-in types and any previously
-defined message type.
-
-Once the new composite message type is created, you can use the new type
-in the configuration file.
-
-Composite message types have the following restrictions:
-
--  Delta subscribe and delta publish are not supported for message types
-   that use ``composite-global``.
-
--  Views, joins, and aggregation cannot project message types that use
-   ``composite-global``. (However, composite message types that use
-   ``composite-global`` *can* be an ``UnderlyingTopic`` or one of the
-   topics in a ``Join``.)
-
--  Composite message types do not support features that automatically
-   construct messages, such as subscriptions to the ``AMPS/.*`` topics and
-   stats acks, regardless of the module the type uses.
-
-Unparsed Payload Section
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-All composite message types, regardless of how they are defined, provide
-an *unparsed payload* section. The unparsed payload section does not
-need to be declared in the ``MessageType`` declaration. As the name
-suggests, AMPS does not parse or interpret this section, so the unparsed
-payload can contain any content of any type. In an AMPS client,
-the unparsed payload section is represented as bytes that appear
-at the end of the message, after the last defined part.
-
-The unparsed payload is included to simplify the common technique where
-a message type contains a header that is used for filtering followed by
-an unparsed binary. If your composite message type contains a single
-binary part, consider using the unparsed payload section in your
-application rather than declaring a binary message part. Notice,
-however, that since AMPS does not parse or interpret this section,
-it may not appear in serialized representations of the message
-(for example, log messages, Galvanometer display, and so on).
-
-Content Filtering with Composite Message Types
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Composite message types support filtering on the contents of the
-composite message. There are some simple conventions to remember when
-constructing expressions to filter on. For more details about content
-filtering, see :ref:`Filtering Subscriptions by Content<#pub-sub-content>`.
-
-These conventions are consistent anywhere that AMPS needs to find a
-value within the composite message type. That includes content filters
-for client subscriptions, identifying SOW keys, creating views and
-aggregates, creating conflated topics, and so on.
-
-composite-global
-~~~~~~~~~~~~~~~~
-
-When using the ``composite-global`` message type, AMPS combines all
-parts of the message into a unified set of XPath identifiers. AMPS
-creates the set of identifiers for each part of the message. If
-different parts of the message contain the same identifier, AMPS treats
-that identifier as though the identifier contained an array of values:
-AMPS creates an array that contains all of the values in the different
-parts of the message. Message types that do not support content
-filtering do not provide XPath identifiers.
-
-For example, consider the message below for a ``composite-global``
-message type that includes two ``json`` parts and a ``binary`` part:
-
-.. code::
-
-    {"id":1,"data":"sample","message":"part one message"}
-    {"message":"another part","customer":"Awesome Amalgamated, Ltd."}
-    0xDEEA0934DF23A37780934...
-
-AMPS constructs the following set of XPath identifiers and values:
-
-.. list-table:: *Composite-global Message Identifiers*
-     :header-rows: 1
-
-     * - Identifier
-
-       - Value
-
-     * - ``/id``
-
-       - ``1``
-
-     * - ``/data``
-
-       - ``"sample"``
-
-     * - ``/message``
-         
-
-       - ``["part one message", "another part"]``
-
-     * - ``/customer``
-
-       - ``"Awesome Amalgamated, Ltd."``
-
-
-
-In short, when using ``composite-global``, AMPS combines the parsable
-parts of the message into a single global set of XPath values, and
-ignores any part of the message that cannot be parsed.
-
-composite-local
-~~~~~~~~~~~~~~~~
-
-When using the ``composite-local`` message type, AMPS creates a distinct
-set of XPath identifiers for each part of the message. AMPS adds an
-XPath step with the position of the message part at the beginning of the
-identifier. Message types that do not support content filtering do not
-provide XPath identifiers, and AMPS skips over them.
-
-For example, consider the message below for a ``composite-local``
-message type that includes two ``json`` parts and a ``binary`` part:
-
-.. code::
-
-    {"id":1,"data":"sample","message":"part one message"}
-    {"message":"another part","customer":"Awesome Amalgamated, Ltd."}
-    0xDEEA0934DF23A37780934...
-
-AMPS constructs the following set of XPath identifiers and values:
-
-.. list-table:: *Composite-local Message Identifiers*
-     :header-rows: 1
-
-     * - Identifier
-
-       - Value
-
-     * - ``/0/id``
-
-       - ``1``
-
-     * - ``/0/data``
-
-       - ``"sample"``
-
-     * - ``/0/message``
-
-       - ``"part one message"``
-
-     * - ``/1/message``
-
-       - ``"another part"``
-
-     * - ``/1/customer``
-
-       - ``"Awesome Amalgamated, Ltd."``
-
-
-
-In short, when using ``composite-local``, AMPS creates XPath identifiers
-for each part of the message, using the position of the message part
-within the composite as the first part of the identifier. AMPS skips
-over any part of the message that cannot be parsed, and simply produces
-no values for that part of the message.
-
-Choosing a Composite Type
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To choose which composite type best fits your application, consider the
-following factors:
-
--  If you need to use delta messaging with this message type, use
-   ``composite-local``.
-
--  If there may be redundant field names in the parts of the message,
-   and it is important to be able to filter based on which part contains
-   the field, use ``composite-local``.
-
--  If you need to be able to create views of this type, use
-   ``composite-local``.
-
-Otherwise, ``composite-global`` may be easier and more straightforward
-for client filtering, since clients do not need to know the detailed
-structure of the message type to be able to filter on the message.
-
-Protobuf Message Types
-------------------------
-
-Protocol buffers, or protobufs for short, is an efficient, automated
-mechanism for serializing structured data. AMPS supports Google protobuf
-messages (version 2 and version 3) as a message format.
-
-Since Google protocol buffers use a fixed format for messages, to use
-protobuf, you must configure AMPS with the definition of the messages
-AMPS will process. This involves defining a ``MessageType``. You must
-define a ``MessageType`` for AMPS to be able to parse protobuf messages.
-
-60East recommends that the ``.proto`` files used with AMPS explicitly
-declare the protocol buffer syntax version used. If there is no explicit
-declaration, AMPS assumes the file uses protocol buffer 2 syntax.
-
-The AMPS engine is message-type agnostic. Except for the limitations
-described in this section, there is no difference to the AMPS engine
-between message types that use protocol buffers and other message types
-such as JSON or XML or FIX.
-
-Configuring Protobuf Message Types
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To use a protobuf message, you must first edit the configuration file to
-include a new ``MessageType``. Then, specify the path to the protobuf
-file and the name of the protobuf file itself inside the
-``MessageType``. Below is a sample configuration of a protobuf message
-type:
-
-.. code-block:: xml
-
-    ...
-
-    <MessageType>
-        <Name>my-protobuf-messages</Name>
-        <Module>protobuf</Module>
-        <ProtoPath>proto-archive;/mnt/shared/protofiles</ProtoPath>
-        <ProtoFile>proto-archive/person.proto</ProtoFile>
-        <Type>MyNamespace.Message</Type>
-    </MessageType>
-
-    ...
-
-Each message type references a ``ProtoFile``, and specifies a single
-top-level type from the file. The ``ProtoFile`` may include other files
-through the standard protocol buffer include mechanism. Likewise, the
-top-level type may be any valid protocol buffer definition, including
-definitions that contain other types.
-
-Once the protocol buffer ``MessageType`` is created as described
-above, you must either create a ``Transport`` that specifies that
-message type exactly, or you must create a ``Transport`` that can
-accept any known message type and ensure that the client specifies the
-new message type (in the example case, ``my-protobuf-message``) in the
-connect string.
-
-When creating a ``protobuf`` message type, you must provide the
-following parameters:
-
-.. list-table:: *Protobuf Message Type Parameters*
-     :header-rows: 1
-
-     * - Parameter
-
-       - Description
-
-     * - .. code::
-         
-           Name
-
-       - The name of the new, customized message type. The rest of the configuration file will
-         use this name to refer to the message type.
-         
-
-     * - .. code::
-         
-           Module
-
-       - The module that contains the message type. Use ``protobuf`` for protocol buffer
-         messages.
-         
-
-     * - .. code::
-         
-           ProtoPath
-         
-
-       - The path in which to search for ``.proto`` files. The content of this element has the
-         following syntax:
-         
-         .. code::
-         
-           alias ; full-path
-         
-         The alias provides a short identifier to use when searching for .proto files. The
-         full path is the path that is substituted for that identifier.
-         
-         For example, in the sample above, ``proto-archive`` is an alias for
-         ``/mnt/shared/protofiles``.
-         
-         A configuration may omit the alias and simply provide the path. For example:
-         
-         .. code::
-         
-           ;/mnt/repository/protodefs
-         
-         You may specify any number of ``ProtoPath`` declarations.
-
-     * - .. code::
-         
-           ProtoFile
-
-       - The name of the ``.proto`` file to use for this message type. To use an alias, prefix
-         the name of the file with the alias, as shown in the example above.
-         
-
-     * - .. code::
-         
-           Type
-
-       - The name of the type inside the ``.proto`` file to use for this message type. AMPS
-         requires a single type.
-         
-
-
-Filtering with Protobuf Messages
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To filter protobuf messages, there are a couple of conventions you must
-remember. AMPS XPath identifiers begin at the outermost message, so you
-can simply use member names for that message. If you have nested
-messages, you use the name of the nested message and the member name
-when creating an XPath identifier.
-
-For example, suppose you have the following definition in a
-``.proto`` file:
-
-.. code-block:: cpp
-
-    message person {
-        required string name = 1;
-        required int32 personID = 2;
-    }
-
-To access the ``personID`` data member, you simply use the name of the
-data member as the XPath identifier. An example filter that verifies
-that a ``personID`` is greater than 1000 would be:
-
-.. code::
-
-    /personID > 1000
-
-If you have nested messages, you simply provide the path to the nested
-message you want to access.
-
-Let's assume that the ``person`` message from the above example was
-nested inside another message with the name of ``record``. The example
-filter below shows how to access the nested ``person`` message, and then
-filter to the ``personID``:
-
-.. code::
-
-    /person/personID > 1000
-
-In this case, the first part of the identifier (``/person``) specifies
-the sub-message. The second part of the identifier (``/personID``)
-specifies the field within that sub-message. Notice that, as always,
-there is no need to specify the name of the message for the outermost
-message.
-
-Working with Multiple Protocol Buffer Types
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Some applications require messages of different types: for example, an
-inventory management system may work with customer records,
-inventory records, and shipping order records.
-
-When using protocol buffers, each of these messages would use a
-different ``.proto`` file, and therefore would be a different
-message type. Unlike a self-describing format such as JSON or
-XML, the serialized form of a protocol buffer message type
-does not automatically contain any information about the type
-of message or the fields that the message contains. Therefore,
-each protocol buffer message type is best considered as a
-completely distinct type. For example, the parser created for
-an order record and the parser created for a customer record
-are different. Unlike self-describing formats, it is not possible
-to use a single parser for these types, or for a parser to
-correctly handle a previously-unknown message structure.
-
-There are two approaches to working with multiple protocol
-buffer types in an AMPS application:
-
-1. Keep the message types distinct. Each message type requires
-   a separate connection to AMPS. The advantage of this approach
-   is that the ``.proto`` files can be maintained and updated
-   separately. Each connection has a distinct type and only
-   needs to handle messages of that type. The disadvantage of
-   this approach is that the application must make a connection
-   to AMPS for each type of message received.
-
-2. Create a "container" type that can
-   *optionally* contain any of the needed message types. The
-   advantage of this approach is that this requires only a
-   single connection to AMPS. Since there is a single 
-   "container" type, a topic can hold this "container" type
-   and have heterogeneous actual contents. The disadvantage
-   to this approach is that it requires a consumer to
-   understand the "container" type and changes to the
-   contained types may need to be carefully managed across
-   the consumers that use the container. A "container" type
-   is typically a ``oneof`` of the contained types.
-
-For example, you might define a container as follows:
-
-.. code-block:: cpp
-
-    message Container {
-
-        oneof {
-          Order      order_type = 1;
-          Payment    payment_type = 2;
-        }
-    }
-
-    message Order {
-        required string customer_id = 1;
-        ...
-    }
-
-    message Payment {
-        required string customer_id = 1;
-        ...
-    }
-
-In this case, the container type will include **either** an
-``Order`` or a ``Payment``.
-
-
-.. _#ug-protobuf-union-types:
-
-Union Types
-^^^^^^^^^^^
-
-When using a protocol buffer message type that contains a union, you can
-navigate the union using the names defined in the top-level element. For
-example, given the union defined below:
-
-.. code-block:: cpp
-
-    message MyUnion {
-        optional Order      order_type = 1;
-        optional Payment    payment_type = 2;
-    }
-
-    message Order {
-        required string customer_id = 1;
-        ...
-    }
-
-    message Payment {
-        required string customer_id = 1;
-        ...
-    }
-
-Providing a filter of ``/order_type IS NOT NULL`` will return all of the
-``MyUnion`` messages that contain an ``Order``, while providing a filter
-of ``/payment_type/customer_id = '42'`` will return only the ``MyUnion``
-messages that contain a ``Payment`` message with a ``customer_id`` of
-``42``.
-
-Limitations of the Protobuf Message Type
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Because the ``protobuf`` message type requires a specific, fixed
-definition for messages, AMPS does not support operations that construct
-messages that may contain arbitrary values. In particular, protobuf does
-not support:
-
--  Creating a View with a ``protobuf`` type as the ``MessageType``. AMPS
-   allows you to aggregate protobuf messages and project the results as
-   another type, but the destination ``MessageType`` for a View cannot be a
-   ``protobuf`` message type.
-
--  Creating an aggregated subscription for a topic that contains messages
-   of  a ``protobuf`` message type.
-
--  Subscriptions to AMPS internal topics. Protobuf message types do not
-   support creating messages for AMPS internal topics, such as
-   ``/AMPS/ClientStatus``.
-
--  Enriching or preprocessing ``protobuf`` message types. AMPS does
-   not support enrichment or preprocessing of ``protobuf`` messages.
-
-
-Protocol buffer version 3 messages provide fixed default values for
-omitted fields. This means that there is no reliable way for AMPS to
-determine if a missing field has been intentionally left out of the
-message, or simply contains the fixed default value. The result
-is an additional limitation for protocol buffer version 3 message types:
-
-- Protocol buffer version 3 message types do not support delta publish
-  or delta subscribe.
-
-Protocol buffer version 2 message types can require that specific
-fields are provided in a message (that is, fields can be marked
-required). The result is an additional limitation for
-protocol buffer version 2 message types:
-
-- Protocol buffer version 2 message types do not support providing a
-  subset of fields in a message by specifying a select list.
-
-There are no other limitations in working with protocol buffer message
-types.
-
-Working with Optional Default Values
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Google protocol buffers provide the ability for a message to have fields
-that are both *optional*, so they need not be provided in the serialized
-message, and *defaulted*, so that there is a specific value interpreted
-when there is no value provided.
-
-When no value is provided in the serialized message for an optional
-default value, AMPS interprets the message differently depending on the
-context:
-
--  For most uses, AMPS interprets the message as though the value is
-   *present and set to the default value*. This means that you can
-   filter on optional default values, use them as SOW keys, and
-   aggregate optional default values regardless of whether a value is
-   present in the serialized message.
-
--  For *delta messaging* with protocol buffer version 2, AMPS treats an
-   optional default value as though there is *no value present*. AMPS does
-   not provide the default value. This means that a delta update must provide
-   the default value *explicitly* in the serialized message to set the field to the
-   default value. This also means that, if the value present in the
-   message is not the default value, but was not changed on the current
-   update, AMPS will not emit that value in messages to delta
-   subscribers. (Since delta messaging is not supported with protocol
-   buffer version 3, this issue does not arise with that version.)
-
-
-Struct Message Types
----------------------
-
-AMPS includes a message type that allows the server to parse and interpret
-binary data in a fixed format. When configuring the message type, you must
-include a definition of the format.
-
-This format is designed to allow AMPS to process messages serialized from
-raw memory, such as would be specified in a C-language ``struct``.
-
-Configuring a Struct Message Type
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To configure a ``struct`` message type, you must define each field that
-AMPS will use. This does not necessarily have to match the original definition
-of the data.  It is possible to "skip over" parts of the binary data
-that AMPS should ignore by declaring that data to be padding.
-
-A ``struct`` message type definition must include one or more ``Field`` elements,
-specified in the order in which the data appears in the message. Each ``Field`` specifies
-the name of the field, and the type and length of the data to be used for that
-``Field``.  The specifier for a field is composed of: *field name* ``=`` *data format specifier*
-where the field name is the XPath identifier that AMPS will use for this field and the
-data format specifier is a specifier for the type of data and number of bytes for this field.  
-
-The data format specifier is modeled after the specifiers for the Python
-``struct`` module. The format requires a data type specifier. The data
-type specifier may be preceded by an optional byte order specifier. For
-variable-width data types (for example, strings), include
-an optional byte order specifier and an optional count specifier.
-
-The following data type specifiers are recognized by the module:
-
-.. list-table::  *Data type specifiers*
-     :header-rows: 1
-
-     * - **Specifier**
-
-       - **C Type**
-
-       - **Size (bytes)**
-
-       - **AMPS Type**
-
-     * - ``x``
-         
-
-       - *n/a*
-         
-
-       - (number of
-         bytes must
-         be specified)
-         
-
-       - Padding bytes, ignored by AMPS
-         
-
-     * - ``c``
-         
-
-       - ``char``
-         
-
-       - 1
-         (number of bytes may
-         be specified)
-
-       - string
-         
-
-     * - ``b``
-         
-
-       - ``signed char``
-         
-
-       - 1
-         
-
-       - integer
-         
-
-     * - ``B``
-         
-
-       - ``unsigned char``
-         
-
-       - 1
-         
-
-       - integer
-         
-
-     * - ``?``
-         
-
-       - ``bool``
-         
-         (as though
-         ``stdbool.h`` were
-         included)
-
-       - 1
-         
-
-       - boolean
-         
-
-     * - ``h``
-         
-
-       - ``short``
-         
-
-       - 2
-         
-
-       - integer
-         
-
-     * - ``H``
-         
-
-       - ``unsigned short``
-         
-
-       - 2
-         
-
-       - integer
-         
-
-     * - ``i``
-
-       - ``int``
-
-       - 4
-
-       - integer
-
-     * - ``I``
-
-       - ``unsigned int``
-
-       - 4
-
-       - integer
-
-     * - ``l``
-
-       - ``long``
-
-       - 4
-
-       - integer
-
-     * - ``L``
-
-       - ``unsigned long``
-
-       - 4
-
-       - integer
-
-     * - ``q``
-
-       - ``long long``
-
-       - 8
-
-       - integer
-
-     * - ``Q``
-         
-
-       - ``unsigned long long``
-         
-
-       - 8
-         
-
-       - integer
-         
-
-     * - ``n``
-         
-
-       - ``ssize_t``
-         
-
-       - 
-
-       - integer
-         
-
-     * - ``N``
-         
-
-       - ``size_t``
-         
-
-       - 
-
-       - integer
-         
-
-     * - ``f``
-
-       - ``float``
-
-       - 4
-
-       - double
-
-     * - ``d``
-
-       - ``double``
-
-       - 8
-
-       - double
-
-     * - ``s``
-         
-
-       - ``char[]``
-         
-
-       - Number of bytes
-         must be specified,
-         AMPS will interpret
-         the specified number of
-         bytes as the string.
-
-       - string
-         
-
-     * - ``S``
-         
-
-       - ``char[]``
-         
-
-       - Number of bytes
-         must be specified,
-         AMPS will interpret
-         the string up to the
-         first NULL character
-         or the number of
-         bytes specified.
-
-       - string
-         
-
-     * - ``p``
-         
-
-       - ``uint8_t``
-         followed by
-         ``char[]``
-         
-
-       - Number of bytes
-         must be specified.
-         
-         AMPS will consume the
-         number of bytes
-         specified in the
-         ``Field`` configuration.
-         The first byte of data
-         specifies the length of
-         the string. Only that
-         number of bytes, starting
-         with the second byte of
-         the data, will be
-         interpreted as the value.
-
-       - string
-         
-
-
-The byte order specifiers are as follows:
-
-.. list-table::  *Byte order specifiers*
-     :header-rows: 1
-
-     * - **Specifier**
-
-       - **Byte Order**
-
-     * - ``@``
-         
-
-       - Native (little-endian for AMPS server)
-         
-
-     * - ``=``
-         
-
-       - Native (little-endian for AMPS server)
-         
-
-     * - ``<``
-
-       - Little-endian
-
-     * - ``>``
-
-       - Big-endian
-
-
-If no byte order is specified, AMPS assumes a little-endian byte order to
-match the native byte order of the AMPS server.
-
-For example, given the following C struct:
-
-.. code-block:: c
-
-   struct data_type
-   {
-      int32_t  id;
-      int32_t  internal_id;
-      float    price;
-      char     label[32];
-      char     code[16];
-      char     routing_instructions[32];
-   };
-
-A message type declaration that would make the `id`, `price`, and `code`
-members available to AMPS could be constructed as follows:
-
-.. code-block:: xml
-
-  <MessageType>
-     <Name>sample_struct_type</Name>
-     <Module>struct</Module>
-     <Field>/id = i</Field>
-     <Field>/ignored = 4x</Field>
-     <Field>/price = f</Field>
-     <Field>/ignored = 32x</Field>
-     <Field>/code = 16s</Field>
-  </MessageType>
-
-This configuration declares that the message will be interpreted as follows:
-
-* The first four bytes of the message will be interpreted
-  as a (little-endian) integer, and that value will be considered to be the `/id`
-  field of the message.
-* The next four bytes are skipped as padding -- notice that,
-  although a field name is required in the specifier syntax, padding bytes are
-  ignored by AMPS, so there is no field named ``/ignored`` generated. However, if
-  the message is pretty-printed (or displayed in Galvanometer), AMPS will indicate
-  that there are padding bytes present for this field.
-* The next four bytes are interpreted as a (little-endian) float, and that value
-  will be considered the value of the ``/price`` field.
-* The next 32 bytes (the ``label`` in the C struct) are skipped as padding. Again,
-  if the message is pretty-printed (or displayed in Galvanometer), AMPS will indicate
-  that there are padding bytes present for this field.
-* The next 16 bytes of the message are an AMPS string that will be used for the value
-  of the ``/code`` field.
-* Any remaining bytes (the ``routing_instructions`` from the C struct, in this case)
-  are ignored. If the message is pretty-printed (or displayed in Galvanometer), AMPS
-  will indicate that there are extra bytes present.
-
-Limitations of Struct Message Types
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Since the ``struct`` message type requires a specific, fixed
-definition for messages, AMPS does not support operations that
-construct messages that may contain arbitrary values. In
-particular, message types defined using the ``struct`` message type
-do not support:
-
-* Creating a View with a ``struct`` message type as the ``MessageType``.
-  AMPS allows you to aggregate ``struct`` message types and project the
-  results as another message type, but the destination ``MessageType`` for
-  a View cannot be a ``struct`` message type.
-
-* Creating an aggregated subscription for a topic that contains
-  messages of a ``struct`` message type.
-
-* Subscriptions to AMPS internal topics (for example, ``/AMPS/ClientStatus``).
-
-* Enriching or preprocessing messages of a ``struct`` message type.
-
-* Delta publish or delta subscribe.
-* Select lists.
-
-Loading Additional Message Types
---------------------------------
-
-AMPS includes the ability to load custom message types in external
-modules. As with all AMPS modules, custom message types are compiled
-into shared object files. AMPS dynamically loads these message types on
-startup, using the information provided in the configuration file. Once
-you have loaded and declared those types, you can use the type just as
-you use the default message types.
-
-For example, the configuration below creates a message type named
-``custom-type`` that uses a module named ``libmy-type-module.so`` and
-specifies a transport for messages of that type:
-
-.. code-block:: xml
-
-    <Modules>
-        <Module>
-            <!-- Specifies the name to use to refer to this module in the rest of the
-                configuration file -->
-            <Name>custom-type-module</Name>
-        
-            <!-- Path to the library to load for this module. In this example, the path is a
-                relative path below the directory where AMPS is started. -->
-            <Library>./custom-modules/libmy-type-module.so</Library>
-        </Module>
-    </Modules>
-
-    <MessageTypes>
-        <MessageType>
-            <!-- The name to use for this message type in the rest of the configuration file. -->
-            <Name>custom-type</Name>
-        
-            <!-- Reference to the module that implements this message type, using the Name
-                defined in the Module configuration. -->
-            <Module>custom-type-module</Module>
-        </MessageType>
-    </MessageTypes>
-
-    <Transports>
-        <Transport>
-            <Name>custom-type-tcp</Name>
-            <Type>tcp</Type>
-            <InetAddr>9008</InetAddr>
-        
-            <!-- The message type that this transport uses, using the Name defined in the
-                MessageType configuration. -->
-            <MessageType>custom-type</MessageType>
-            <Protocol>amps</Protocol>
-        </Transport>
-    </Transports>
-
-Once a message type has been declared, you can use it in exactly the
-same way you use the default message types.
-
-Notice, however, that custom-developed message types may only provide
-support for a subset of the features of AMPS. For example, the
-``binary`` message type provided with AMPS does not support features
-that require AMPS to parse or construct a message, as described above.
-The developer of the message type must provide information on what
-capabilities the message type provides.