moat-kv 0.70.24__py3-none-any.whl → 0.71.0__py3-none-any.whl

docs/source/common_protocol.rst

@@ -0,0 +1,47 @@
+=================
+MoaT-KV's protocol
+=================
+
+MoaT-KV's native protocol (both client/server and server/server) is based on
+MsgPack. Strings must be valid UTF-8 and are distinct from binary buffers.
+
+++++++++++++++++++
+MsgPack Extensions
+++++++++++++++++++
+
+MoaT-KV is expected to be a transparent protocol. Unknown extensions
+must not cause the reader to crash, and should be round-trip-safe: a client
+which reads an object and modifies attribute A should not modify attribute
+B even if B contains an element with an unknown extension.
+
+This should, but currently does not, apply to extensions 2 and 3.
+
+--------
+DateTime
+--------
+
+A client must support MsgPack's Timestamp extension.
+
+------
+Bignum
+------
+
+Extension 2 is used for unsigned arbitrary-sized integers.
+
+-----
+Paths
+-----
+
+Extension 3 packages a MsgPack path.
+
+MoaT-KV uses Path objects to refer to its nodes. Paths are lists which may
+contain arbitrary msgpack data structures but should be limited to UTF-8
+strings and non-negative integers.
+
+A client must support both Path objects and plain lists when reading. It
+should send Path elements. Using plain lists instead is supported but
+not recommended.
+
+In this documentation's protocol examples, Paths are shown as ``P('some.string')``
+for readability.
+

docs/source/debugging.rst

@@ -0,0 +1,70 @@
+==============================
+Fixing MoaT-KV network problems
+==============================
+
+As the MoaT-KV network is fully asynchronous, there's no way to avoid
+getting into trouble – there's no arbitration of inconsistent data.
+
+This document explains how to get back out, if necessary.
+
+Missing data
+============
+
+See the `Server protocol <server_protocol>` for details on how MoaT-KV
+works. From that document it's obvious that when a node increments its
+``tick`` but the associated data gets lost (e.g. if the node or its Serf
+agent crashes), you have a problem.
+
+Worse: a server will not start if the "missing" list is non-empty. The
+problem is that stale data causes difficult-to-resolve inconsistencies
+when written to. TODO: allow the server to be in maintainer-only mode when
+that happens.
+
+First, run ``moat kv internal state -ndmrk``. Your output will look
+somewhat like this::
+
+    deleted:  # Ticks known to be deleted
+      test1:
+      - 12
+    known:  # Ticks known to be superseded
+      test1:
+      - 1
+      - - 3
+        - 10
+      test2:
+      - 1
+    missing:  # Ticks we need to worry about
+      test1:
+      - 2
+    node: test1  # the server we just asked
+    nodes:  # all known nodes and their ticks
+      test1: 12
+      test2: 1
+    remote_missing: {}  # used in recovery
+    tock: 82  # MoaT-KV's global event counter
+    
+This is not healthy: The ``missing`` element contains data. You can
+manually mark the offending data as stale::
+
+   one $ moat kv internal mark test1 2
+   known:
+      test1:
+      - - 1
+        - 11
+      test2:
+      - 1
+    node: test1
+    tock: 92  # If this is not higher than before, clean your glasses ;-)
+    one $
+
+This shows that the offending ``tick`` has been successfully added to the
+``known`` list. Calling ``moat kv internal state -m`` verifies that
+the list is now empty.
+
+Use the ``--broadcast`` flag to send this message to all MoaT-KV servers,
+not just the one you're a client of.
+
+This action will allow the bad record to re-surface when the node that has
+the record reconnects, assuming that there is one. You can use the ``mark``
+command's ``--deleted`` flag to ensure that it will be discarded instead.
+

docs/source/extend.rst

@@ -0,0 +1,37 @@
+================
+Extending MoaT-KV
+================
+
+MoaT-KV comes with a built-in extension mechanism for its command line,
+based on Python namespaces and import hackery.
+
+Yor extension needs to ship a ``moat.kv.NAME`` module, with a
+``_main.py`` file that exports a ``cli`` command (usually used as an
+`asyncclick.group`). This adds the subcommand ``NAME`` to ``moat kv``.
+
+Command line helper
+===================
+
+MoaT-KV commands follow a standard scheme (TODO some don't yet):
+
+* what kind of object do you want to affect
+* the name of the object to affect (or create)
+* [ maybe start over with a sub-object ]
+* the action you want to take
+* some options affecting the action, and/or
+* the generic set of parameter+value options (``-v``/``-e``/``-p``)
+
+In order to simplify implementing that, there's a couple of helper methods.
+
+`moat.kv.obj.command.std_command` takes a ``click.Group`` command and
+attaches a subgroup with standard add/set/delete commands to it. The
+new group is returned so you can attach more commands to it if you want.
+
+`moat.util.attr_args` attaches MoaT-KV's generic parameter+value options.
+
+`moat.util.process_args` takes a dict (usually) and the generic options'
+variables (``set_``, ``vars_``, ``eval_``, ``path_``, ``proxy_``)
+and applies them.
+
+It's the caller's job to verify that the result is sane. TODO: support
+using a validation library (probably jsonschema).

docs/source/history.rst

@@ -0,0 +1,36 @@
+Release history
+===============
+
+.. currentmodule:: moat.kv
+
+.. towncrier release notes start
+
+Migration from DistKV
+=====================
+
+As of 2023-05, the protocol is unchanged.
+
+To re-use the DistKV data as-is, copy ``/etc/distkv.cfg``  to
+``/etc/moat/moat.cfg``. Then edit ``moat.cfg``:
+
+* Move the ``logging:`` entry to the bottom if it isn't there already
+* Indent everything above that entry two spaces
+* Insert a line ``kv:`` as the first line of the file (not indented!)
+
+Then, add these items as appropriate::
+
+    kv:
+      inv:
+        prefix: !P :.distkv.inventory
+      config:
+        prefix: !P :.distkv.config
+      errors:
+        prefix: !P :.distkv.errors
+      codes:
+        prefix: !P :.distkv.code.proc
+      modules:
+        prefix: !P :.distkv.code.module
+      runner:
+        prefix: !P :.distkv.run
+        state: !P :.distkv.state
+    

docs/source/index.rst

@@ -0,0 +1,75 @@
+.. documentation master file, created by
+   sphinx-quickstart on Sat Jan 21 19:11:14 2017.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+
+===============================================
+MoaT-KV: A distributed no-master key-value store
+===============================================
+
+Rationale
+=========
+
+Any kind of distributed storage is subject to the CAP theorem (also called
+"Brewer's theorem"): you can't get all of (global) Consistency,
+Availability, and Partition tolerance. The problem is that you do want all
+three of these.
+
+One way around this problem is to recognize that on most KV storage
+systems, any given record is rarely (if ever) changed by more than one
+entity at the same time. Thus, a simple gossip protocol is sufficient
+for distributing data.
+
+MoaT-KV is intended to be used in a mostly-RAM architecture. There is no
+disk-based storage backend; snapshots and event logs are used to restore a
+system, if necessary.
+
+See `Protocol Overview <overview.html>` for details about MoaT-KV's
+choices.
+
+API
+===
+
+MoaT-KV offers an efficient interface to access and change data.
+
+
+Status
+======
+
+MoaT-KV is in production use as the backbone of the author's home and office
+automation setup.
+
+Note that as of MoaT-KV 0.30, multi-word paths were replaced with dotted
+strings. Some pieces of documentation might still reflect the old style.
+
+.. toctree::
+   :maxdepth: 2
+
+   overview.rst
+   tutorial.rst
+   startup.rst
+   command_line.rst
+   common_protocol.rst
+   client_protocol.rst
+   server_protocol.rst
+   auth.rst
+   acls.rst
+   code.rst
+   model.rst
+   translator.rst
+   debugging.rst
+   extend.rst
+   related.rst
+
+   TODO.rst
+   history.rst
+
+====================
+ Indices and tables
+====================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+* :ref:`glossary`

docs/source/model.rst

@@ -0,0 +1,54 @@
+==========
+Data Model
+==========
+
+This section documents some of MoaT-KV's server-internal classes.
+
+
+.. automodule:: moat.kv.model
+   :members:
+
+ACLs
+----
+
+ACL checks are performed by :class:`~moat.kv.types.ACLFinder`. This class
+collects all relevant ACL entries for any given (sub)path, sorted by
+depth-first specificty. This basically means that you collect all ACLs
+that could possibly match a path and sort them; the ``+`` and ``#``
+wildcards get sorted last. Then the system picks the first entry that
+actually has a value.
+
+This basically means that if you have a path ``a b c d e f g`` and ACLs ``a
+b # g`` and ``a # d e f g``, the first ACL will match because ``b`` is
+more specific than ``#``, even though the second ACL is longer and thus
+could be regarded as being more specific. However, the current rule is more
+stable when used with complex ACLs and thus more secure.
+
+.. autoclass:: moat.kv.types.ACLFinder
+   :members:
+
+Helper methods and classes
+--------------------------
+
+.. autoclass:: moat.kv.util.MsgWriter
+   :members:
+
+.. automodule:: moat.kv.util
+   :members:
+
+.. py:data:: moat.kv.util.NotGiven
+
+   This object marks the absence of information where simply not using the
+   data element or keyword at all would be inconvenient.
+
+   For instance, in ``def fn(value=NotGiven, **kw)`` you'd need to test
+   ``'value'  in kw``, or use an exception. The problem is that this would
+   not show up in the function's signature.
+
+   With ``NotGiven`` you can simply test ``value is`` (or ``is not``) ``NotGiven``.
+
+.. automodule:: moat.kv.runner
+   :members:
+
+.. automodule:: moat.kv.actor
+   :members:

docs/source/overview.rst

@@ -0,0 +1,83 @@
+=======================
+Principles of operation
+=======================
+
+MoaT-KV relies on the fact that on most KV storage systems, any given record
+is rarely (if ever) changed by more than one entity at the same time. Thus,
+a simple gossip protocol is sufficient for distributing data.
+
+To recover from missed changes, each node in a MoaT-KV network maintains a
+change counter ("tick"). All data records (:class:`moat.kv.model.Entry`) are
+tagged with a chain of events (:class:`moat.kv.model.NodeEvent`), consisting
+of the ``n`` most recent ``(node, tick)`` values which changed this
+entry. Nodes do not appear in a chain more than once. Dropped ticks
+are added to a per-node list of "known"(-to-have-been-superseded) counter
+values.
+
+The maximum chain length is determined by the number of partitions a MoaT-KV
+network might split into. Thus the network guarantees that it is possible
+which side of a split modified a record when the split is healed.
+
+If both sides did, the conflict is resolved deterministically.
+TODO: when this happens, send a notification to clients.
+
+After a network split, a four-step protocol re-synchronizes the
+participants:
+
+* broadcast the current counters
+
+* broadcast known-value and known-deleted lists
+
+* broadcast a list of missing node events
+
+* broadcast the missed data
+
+MoaT-KV does not have a master node, much less a consensus-based election
+system (Raft, Paxos, …). Instead, MoaT-KV uses an `asyncactor.Actor` to
+compile a short list of available servers that's broadcast every few
+seconds.
+
+When a partitioned network is re-joined, the current housekeepers are
+responsible for driving and monitoring the re-sync protocol.
+
+
+Storage
+=======
+
+MoaT-KV is intended to be used in a mostly-RAM architecture. There is no
+disk-based storage backend; snapshots and event logs are used to restore a
+system, if necessary. Feeding old snapshots to a running system is mostly
+benign, but see below.
+
+MoaT-KV runs on top of MQTT.
+It supports all data types that can be transmitted by
+`MsgPack <https://github.com/msgpack/msgpack/blob/master/spec.md>`.
+
+TODO: MsgPack has extension types, so constructing Python objects is possible.
+
+Record Deletion
+===============
+
+Deleting data records is when MoaT-KV's synchronization protocol breaks
+down, because MoaT-KV can't attach chains to records which no longer exist.
+
+MoaT-KV fixes this by keeping a separate record of deleted entries, or
+rather their chain links. This works well for mostly-static storages but
+becomes a problem on more dynamic systems.
+
+Thus, periodic clean-up is required. This is achieved by creating a
+separate "Delete" Actor group which contains every system with persistent
+storage plus one system per network that's not already covered.
+
+When every node of this group is online, they periodically broadcast a
+tuple of ``tock`` values: one which signals that deletions with earlier
+``tock``\s may safely be flushed, and a high-water limit for the next
+round.
+
+A node that receives this tuple compares the received first value with the
+last transmission's second. If it's higher, deletions may have been missed,
+most likely due to a network outage between that node and the closest Delete
+member. Since the records are now gone, the node will connect to one of the
+Delete group members and send a list of each entry's last-change chain links.
+The recipient will re-broadcast any misses as "new" deletions.
+

docs/source/related.rst

@@ -0,0 +1,89 @@
+============================
+Plugins and related software
+============================
+
+Additions to this list will be appreciated!
+
+
+Home Assistant
+==============
+
+`Home Assistant`__ is a front-end for home automation. (Actually it's more
+than that, but ``moat.kv`` uses it as a front-end and prefers to do the
+automation part itself.)
+
+`moat.kv-hass <https://github.com/M-o-a-T/disthass>`__ documents how to
+connect Home Assistant to the ``moat.kv`` system and helps with creating the
+data structures that teach Home Assistant about moat.kv-controlled sensors
+and actors.
+
+
+KNX
+===
+
+KNX is a serial bus for building control.
+
+`knxd <https://github.com/knxd/knxd/>`__ is a server commonly used to talk to KNX interfaces.
+
+`xknx <https://github.com/XKNX/xknx>`__ is a Python package you can use to talk to ``knxd``.
+
+`moat.kv-knx <https://github.com/M-o-a-T/distknx>`__ connects values stored
+in moat.kv to devices on the KNX bus.
+
+ 
+1wire
+=====
+
+`1wire <https://en.wikipedia.org/wiki/1-Wire>`__ is a two- or three-wire
+bus (one signal wire, somewhat-optional 5V power, ground) that is
+frequently used to connect inexpensive sensors and actors to a computer.
+
+`OWFS <https://www.owfs.org/>`__ is the server commonly used on Linux
+systems to talk to 1wire.
+
+`asyncowfs <https://github.com/M-o-a-T/asyncowfs>`__ is a Python package
+that provides a high-level object-oriented async interface to OWFS.
+
+`moat-kv-owfs <https://github.com/M-o-a-T/distknx>`__ uses ``asyncowfs`` to
+connect values stored in ``moat.kv`` to attributes if 1wire devices.
+
+
+Inventory Management
+====================
+
+`moat-kv-inv <https://github.com/M-o-a-T/moat-kv-inv>`__ is a command-line
+extension that stores of hosts, networks and cables. It contains templating
+code so you can auto-create the configuration for your router (if it's text
+instead of some binary format).
+
+
+Akumuli
+=======
+
+`Akumuli <https://akumuli.org/>`__ is a time series database.
+
+`asyncakumuli <https://github.com/M-o-a-T/asyncakumuli>`__ is a Python package
+that provides an async interface to Akumuli.
+
+`moat.kv-akumuli <https://github.com/M-o-a-T/distakumuli>`__ implements a
+background task that monitors values stored in ``moat.kv`` and mirrors them
+into Akumuli, thus saving their history.
+
+
+Binary I/O
+==========
+
+`asyncgpio <https://github.com/M-o-a-T/asyncgpio>`__ is a Python package
+that provides structured access to your computer's I/O ports.
+
+`moat.kv-gpio <https://github.com/M-o-a-T/distgpio>`__ contains code that
+mirrors a binary value stored in ``moat.kv`` to a GPIO pin and vice versa.
+
+
+Wago I/O controllers
+====================
+
+The German company `WAGO Kontakttechnik <https://www.wago.com>`__ makes the
+``750-*`` line of extensible rugged controllers with various modules.
+
+