moat-kv 0.71.0__py3-none-any.whl → 0.71.7__py3-none-any.whl

docs/source/tutorial.rst

@@ -1,711 +0,0 @@
-===================
-The MoaT-KV tutorial
-===================
-
-Installation
-============
-
-This part is easy. ``pip install moat-kv``.
-
-You now have, or should have, a ``moat`` command-line utility. If not,
-use this script::
-
-   #!/usr/bin/env python3
-
-   import sys
-   import moat.__main__
-
-
-You also need a running `MQTT <https://mqtt.org>` message broker.
-
-Start the server
-================
-
-You start an initial server with this command::
-
-   $ moat kv server -i Testing $(hostname)
-   Running.
-
-By default, your MoaT-KV server will talk to the local MQTT process.
-You can configure the destination by adapting the config file::
-
-   $ moat kv -C kv.server.mqtt.uri=mqtt://your-server:1883 server -i Testing $(hostname)
-
-You can now retrieve the root value::
-
-   $ moat kv data :
-   "Testing"
-   $
-
-As the purpose of MoaT-KV is to be a *distributed* key-value storage, 
-you can start another server on a different host::
-
-   two $ moat kv -C server.mqtt.uri=mqtt://your-server:1883 server $(hostname)
-   Running.
-
-
-This will take a few seconds for the servers to sync up with each other.
-You can verify that the second server has successfully synced up::
-
-   two $ moat kv data :
-   "Testing"
-   two $
-
-The root value is not special; by convention, it contains some data about the current
-MoaT-KV network.
-
-You can now kill the first server and restart it::
-
-   $ moat kv server $(hostname)
-   Running.
-
-You must **never** start a server with the ``-i`` option unless you're
-creating a new and separate MoaT-KV network.
-
-You can create separate networks by changing the ``server.root`` config
-variable. Such networks do not collide with each other, other than sharing
-Serf gossip bandwidth.
-
-
-Data commands
-=============
-
-You might want to add an alias for "moat kv data" so that you don't
-have to type so much. In ``bash``::
-
-   $ mkd() { moat kv data "$@"; }
-
-Then, you can store arbitrary data at random MoaT-KV nodes::
-
-   $ mkd one.two.three set -e : 123
-   $ mkd one.two.three.four set -e : 1234
-   $ mkd one.two.three.four.five set -v one XXX -e two 2
-   $ mkd one.two.three
-   123
-   $ mkd one.two.three.four.five
-   one: XXX
-   two: 2
-   $
-
-The ``-e`` flag tells the ``set`` command to evaluate the given data as a
-Python expression. You can store numbers, True/False/None, binary and
-Unicode strings, and lists/tuples/hashes composed of these.
-
-Stored values may be data structures, and you can selectively change them::
-
-   $ mkd one.two.three.four.five set -e one 1
-   $ mkd one.two.three.four.five
-   one: 1
-   two: 2
-
-The colon we used after ``-e`` is the empty path. More about paths below.
-
-All entries' values are independent. MoaT-KV's storage is organized
-hierarchically, (among other reasons) for ease of retrieval::
-
-    $ mkd one get -rd_
-    two:
-      three:
-        _: 123
-        four:
-          _: 1234
-          five:
-            _: 
-              one: 1
-              two: 2
-    $
-
-MoaT-KV also stores some internal data, under a special ``null`` root key.
-You can use ``moat kv internal dump :`` to display them.
-
-Path specification
-------------------
-
-MoaT-KV uses "paths" to access entries (and the partial values in them).
-We chose the dot as a path separator because it's more visually distinctive
-than a slash.
-
-In MoaT-KV, paths elements are not limited to strings; integers can
-also be path elements, as can ``True``, ``False``, ``None``, and tuples
-composed from them. We use colons instead of dots to mark those.
-The colon is also used as an escape characters for path elements that
-contain dots or colons; it is easy to type and doesn't occur often,
-while the traditional Unix escape character (backslash ``\\``) is
-hard to type in some locales and must be duplicated almost everywhere you
-want to actually use it.
-
-A space is encoded as ``:_``. While a literal space is not a problem, it
-needs to be escaped on the command line. Experience shows that people tend
-to forget that. A "real" underscore ``_`` is not escaped.
-
-There's also the empty path (i.e. the top of MoaT-KV's entry hierarchy,
-not the same as a path that consists of an empty-string element!) which is
-coded as a stand-alone ``:`` for much the same reason.
-
-Anything else that follows a colon is evaluated as a Python expression.
-
-Thus:
-
-==== ==========
-Code   Meaning
----- ----------
- :.  literal ``.``
- ::  literal ``:``
- :_    space
-==== ==========
- :t    True
- :f    False
- :n    None
- :e  empty string
- :x  hex integer
- :b  binary integer
- :y  hex bytestring
- :v  literal bytestring
- :XX eval(XX)
-
-==== ==========
-
-The first three are inline escape sequences while the others start a new
-element.
-
-Hex number input is purely a convenience; integers in paths are always
-printed in decimal form. While you also could use ``:0x…`` in place of
-``:x…``, the latter reduces visual clutter:
-
-.. warning::
-
-   Yes, MoaT-KV supports tuples as part of paths. You probably should not use
-   this feature without a very good reason. "My key consists of three
-   random integers and I want to avoid the overhead of storing a lot of
-   intermediate entries" would be an example of a good reason.
-   
-   MoaT-KV also allows you to use both ``False``, an integer zero, and a
-   floating-point zero as path elements. This is dangerous because Python's
-   comparison and hashing operators treat them as being equal. (Same for
-   ``True`` and 1; same for floating point numbers without fractions and
-   the corresponding integers.)
-
-   Floating point numbers are also dangerous for a different reason: floats 
-   that are not a fractional power of two, such as 1/3, are inexact.
-   Thus you might end up with five different entries for what was meant to
-   be ``1/3``.
-
-   Bottom line:
-
-   * If you do need paths elements with sub-integer numbers, consider
-     scaling them up using using ``int(num*1000)``, or fractional numbers
-     (stored as a numerator,denominator tuple), or ``str(Decimal(…))``.
-
-   * Don't use multiple numeric types as child nodes of a single parent.
-
-
-Persistent storage
-==================
-
-MoaT-KV keeps everything in memory.
-
-As this is not optimal if there is a power failure (or, for single-node
-systems, a server crash or OS update or …), MoaT-KV has a built-in
-mechanism to save its state to disk.
-
-Automatic state save+restore
-----------------------------
-
-Do this::
-
-    $ echo MODE=hybrid >>/etc/moat/kv.env
-    $ /usr/lib/moat/kv/rotate
-    $ systemctl restart moat-kv
-
-The MoaT-KV server will now auto-save the current state every 15 minutes,
-log all changes, and load the most-recent state from disk when it's
-restarted.
-
-Use ``MODE=master`` if you use a stand-alone MoaT server.
-
-Manual state save+restore
--------------------------
-
-You can also save state manually::
-
-   $ moat kv log dest /var/local/lib/moat/kv/$(date +%Y%m%d).state
-
-This command writes the current state to the given file. The server keeps the
-file open and appends new records to it. The ``log dest`` has options to
-write an incremental change record or to create a one-shot dump.
-
-Incremental records are guaranteed to not have missing or duplicate records.
-
-When you need to restart your MoaT-KV system from scratch, tell it to use the
-newest saved state file::
-
-    $ moat kv server -l $(ls -t /var/local/lib/moat/kv/*.state | head -1) $(hostname)
-    Running.
-
-If your state dump files are incremental, you should instead do
-something like this::
-
-    $ moat kv server -l <(cat /var/local/lib/moat/kv/*.state) $(hostname)
-    Running.
-
-These commands are mostly-safe to use on a network that's already
-running; your node may run with old state for a few seconds until it
-retrieves the updates that happened while it was down. An option to delay
-startup until that process has completed is somewhere on the TODO list.
-
-In a typical MoaT-KV network, at most two or three nodes will use persistent
-storage; all others simply sync up with one of their peers whenever they
-are restarted.
-
-
-Authorization
-=============
-
-MoaT-KV initially doesn't use an authorization scheme. However,
-advanced uses require the ability to distinguish between users.
-
-Let's set up a "root" user::
-
-    $ moat kv auth -m password user add name=joe password?=Code
-    Code: ******
-    $ moat kv auth -m password user list
-    joe
-    $ moat kv auth -m password init -s
-    Authorization switched to password
-    $
-
-(The input at the "Code:" prompt is not echoed.)
-
-After this point, you can no longer use MoaT-KV without a password::
-
-    $ mkd :
-    ClientAuthRequiredError: You need to log in using: password
-    $
-
-    $ moat kv -a "password name=joe password?=Code" data :
-    Code: ******
-    "Root"
-    $
-
-Internal data are stored in a separate MoaT-KV subtree that starts with a ``None`` value.
-You can display it::
-
-    $ moat kv -a "password name=joe password=test123" data internal dump :
-    null:
-      auth:
-        _:
-          current: password
-        password:
-          user:
-            joe:
-              _:
-                _aux: null
-                password: !!binary |
-                  7NcYcNGWMxapfjrDQIyYNa2M8PPBvHA1J8MCZVNPda4=
-
-As you can see, passwords are encrypted -- hashed, actually. The exact
-scheme depends on the auth method.
-
-NB: nothing prevents you from using the string ``"null"`` as an ordinary
-key name::
-
-   $ moat kv -a "password name=joe password=test123" data null.foo set -v : bar
-   $ moat kv -a "password name=joe password=test123" data : get -rd_
-   …
-   'null':
-     foo:
-       _: bar
-
-For experimentation, there's also a ``_test`` authorization method which
-only exposes a user name::
-
-   $ moat kv auth -m _test user add name=joe
-   $ moat kv auth -m _test user add name=root
-   $ moat kv auth -m _test init
-   $ moat kv data :
-   ClientAuthRequiredError: You need to log in using: _test
-   $ mkv() { moat kv -a "_test name=joe" "$@"; }
-   $ mkv data :
-   123
-   $
-
-We'll use this user, and the shell alias, in the following sections.
-
-ACLs and distributed servers
-----------------------------
-
-MoaT-KV servers actually use the client protocol when they sync up. Thus, when you
-set up authorization, you must teach your servers to authenticate to their
-peer::
-
-   $ moat kv -C connect.auth="_test name=joe" server $(hostname)
-
-You typically store that in a configuration file::
-
-    kv:
-      conn:
-        auth: "_test name=joe"
-        host: 127.0.0.1
-
-``moat`` auto-reads the configuration from a few paths, or you can use
-the ``moat -c test.cfg`` flag.
-
-Access restrictions
-===================
-
-A user can be restricted from accessing or modifying MoaT-KV data.
-
-Let's say that we'd like to create a "write-only" data storage::
-
-   $ moat kv -a "_test name=root" acl set writeonly -a xc 'wom.#'
-   $ moat kv -a "_test name=root" auth user set param joe acl writeonly
-   $ mkv data wom.foo.bar set -e : 42
-   $ mkv data wom.foo.bar set -e : 43
-   ServerError: (<AclEntry:[None, 'acl', 'writeonly', 'wom', '#']@<NodeEvent:<Node: test1 @10> @4 1> ='cx'>, 'w')
-   $ mkv data wom.foo
-   ServerError: (<AclEntry:[None, 'acl', 'writeonly', 'wom', '#']@<NodeEvent:<Node: test1 @10> @4 1> ='cx'>, 'r')
-   $
-
-As you can see, this allows the user to write to arbitrary values to the
-"wom" tree, but Joe cannot change anything – nor can he read the values
-which he wrote.
-
-Note that we also created a "root" user who doesn't have ACL restrictions.
-If we had not, we'd now be locked out of our MoaT-KV storage because "no
-matching ACL" means "no access".
-
-A user who has an ACL set can no longer modify the system, because the
-``None`` element that separates system data from the rest cannot match a
-wildcard. ACLs for system entries are on the TODO list; so are user groups
-or roles or whatever. Code welcome.
-
-
-
-Code execution
-==============
-
-MoaT-KV doesn't just store passive data: you can also use it to distribute
-actual computing. We'll demonstrate that here.
-
-First we feed some interesting code into MoaT-KV::
-
-    $ mkv code set the.answer <<END
-    > print("Forty-Two!")
-    > return 42
-    > END
-
-Then we set up a one-shot run-anywhere instance::
-
-   $ mkv run set -c the.answer -t 0 a.question
-
-This doesn't actually execute any code because the executor is not part of
-the MoaT-KV server. (The server may gain an option to do that too, but
-not yet.) So we run it::
-
-   $ mkv run all
-   Forty-Two!
-
-(Initially this takes some time, because the ``run`` command needs to
-co-ordinate with other runners. There aren't any, others, of course, but
-MoaT-KV can't know that.)
-
-The code will not run again unless we either re-set ``--time``, or set a
-repeat timer with ``--repeat``.
-
-Start times are mostly-accurate. There are two reasons why they might not
-be:
-
-* the co-ordination system has a periodic window where it waits for the
-  next coordinator. This causes a delay of up to two seconds.
-
-* TODO: The current leader might decide that it's too busy and wants to
-  delegate starting a particular job to some other node in the cluster.
-  This incurs some delay, more if the recipient is no longer available.
-
-This method will run the code in question on any node. You can also run
-code on one specific node; simply do::
-
-   $ mkv run -n $(hostname) set -c "same answer" -t 0 a.question
-   $ mkv run -n $(hostname) all
-
-The one-node-only runner and the any-node runner are distinct. There's also
-a way to designate a subgroup of hosts (like "all with a 1wire interface")
-and to run a job on any / all of them. See ``moat kv run --help`` for details.
-
-
-Errors
-======
-
-Nobody is perfect, and neither is code. Sometimes things break.
-MoaT-KV remembers errors. To demonstrate, let's first provoke one::
-
-    $ mkv code set the.error <<END
-    > raise RuntimeError("Owch")
-    > END
-    $ mkv run set -c the.error -t 0 what.me.worry
-    $ mkv run all  # if it's not still running
-    20:24:13.935 WARNING:moat.kv.errors:Error ('.moat', 'kv', 'error', 'test1', 16373) test1: Exception: Owch
-
-The list of errors is now no longer empty::
-
-   $ mkv error list -d_
-   [ some YAML ]
-
-You can limit the error list to specific subtrees. This command has the
-same effect::
-
-   $ mkv error list -d_ :.moat.kv.run.any
-
-except that the path is shortened for improved useability.
-
-Error details are available; add the ``-a`` option. You can also filter
-errors on a specific node, which only includes that node's details.
-
-
-The Python API
-==============
-
-Command lines are all well and good, but MoaT-KV gets really interesting
-when you use it from Python.
-
-Let's start by simply setting some value::
-
-   import anyio
-   from moat.kv.client import open_client
-   from moat.util import P
-
-   async def dkv_example():
-      async with open_client() as client:
-         client.set(P("one.two.three"), value=("Test",42,False), chain=None)
-
-   anyio.run(dkv_example)
-
-That was easy. Now we'd like to update that entry::
-
-   from moat.util import P
-   async def dkv_example():
-      async with open_client() as client:
-         res = client.get(P("one.two.three"), nchain=2)
-         ret = client.set(P("one.two.three"), value=("Test",v[1]+1,False), chain=res.chain)
-         assert res.chain != ret.chain
-
-The ``chain`` parameter is important: it tells MoaT-KV which change caused
-the old value. So if somebody else changes your ``one.two.three`` entry
-while your program was running, you get a collision and the ``set`` fails.
-
-``set`` returns a new chain so you can update your value multiple times.
-
-Deleting an entry clears the chain because the source of a non-existing value
-doesn't matter.
-
-.. warning::
-   MoaT-KV is an asynchronous distributed system. Thus, asuming that you
-   have more than one MoaT-KV server, this does not prevent your ``set``
-   command from being ignored; it just reduces the window when this could
-   happen from the time since the last ``get`` to a couple of milliseconds.
-
-
-Watching for Changes
---------------------
-
-The result of the previous ``get`` was static. If somebody else
-subsequently changes it, you wouldn't know. Let's fix that::
-
-   async def dkv_example():
-      async with open_client() as client:
-         async with client.watch(P("one.two"), fetch=True) as watcher:
-            async for res in watcher:
-               if 'path' not in res:
-                  continue
-               if 'value' in res:
-                  print(f"{path}= {res.value}")
-               else:
-                  print(f"{path}: deleted")
-
-``fetch=True`` will send the current state in addition to any changes.
-The ``'path' not in res`` test filters the notification that tells you that
-the subtree you requested is complete. The result's path doesn't contain
-the prefix you used in ``watch`` because you already know it.
-
-if you need two ``watch`` at the same time, create separate tasks. Feed the
-resuts through a common queue if you want to process them in a comon
-function.
-
-Active objects
---------------
-
-While watching for changes is nice, organizing the resulting objects tends
-to be tedious. MoaT-KV comes with a couple of classes that does this for you::
-
-   from moat.kv.obj import ClientRoot, ClientEntry
-   from moat.util import NotGiven
-
-   class OneEntry(ClientEntry):
-      async def set(self, value):
-         await super().set_value()
-         path = ' '.join(str(x) for x in self.subpath)
-         if value is NotGiven:
-            print(f"{path}= {value}")
-         else:
-            print(f"{path}: deleted")
-
-   class OneRoot(ClientRoot):
-      @classmethod
-      def child_type(cls, name):
-         return OneEntry
-
-   async def dkv_example():
-      async with open_client() as client:
-         async with client.mirror("one", root_type=OneRoot) as root:
-            # At this point you have the sub-tree in memory
-            assert root['two']['three'].value[1] >= 42
-
-            await anyio.sleep_forever()
-         pass
-         # at this point the sub-tree is still there, but won't be updated
-
-except that in a real program you'd do some real work instead of sleeping.
-
-Verification
-============
-
-Complex data should be clean. Storing ``"Hello there!"`` in a value that
-the rest of your code expects to be an integer is likely to have unwanted
-effects.
-
-For this example, we'd like to enforce that all ``quota`` values in our
-site statistics are integer percentages.
-
-First, we define the type::
-
-    $ ./kv client type set -g 0 -g -2 -g 123 -b 1.2 -b '"Hello"' int <<END
-    > if int(value) != value: raise ValueError("not an integer")
-    > END
-    $
-
-As you can see, data types must be accompanied by example values that include
-both "good" and "bad" examples.
-
-You can also declare subtypes::
-
-    $ mkv type set -g 0 -g 99 -g 100 -b -1 -b 101 int.percent <<END
-    > if not (0 <= value <= 100): raise ValueError("not a percentage")
-    > END
-    $
-
-The example values, both good and bad, must pass the supertype's checks.
-
-Now we associate the test with our data::
-
-    $ mkv type match -t int.percent 'stats.#.quota'
-
-Then we store some value::
-
-    $ mkv data stats.foo.bar.quota set -v : 123
-    ServerError: ValueError("not an integer")
-
-Oops: non-string values need to be evaluated. Better::
-
-    $ mkv data stats.foo.bar.quota set -e : 123
-    ServerError: ValueError('not a percentage')
-    $ mkv data stats.foo.bar.quota set -e : 12
-    $
-
-MoaT-KV does not test that existing values match your restrictions.
-
-
-Data mangling
-=============
-
-Structured data are great, but some clients want boring single-value items.
-For instance, some home automation systems want to use ``"ON"`` and
-``"OFF"`` messages, while your active code is much happier with a ``bool``
-value – or even a mapping that also carries the time of last change, so that
-a ``turn off after 15 minutes`` rule will actually work.
-
-Let's write a simple number codec::
-
-    $ mkv codec set -i '"12.5"' 12.5 -o 13.25 '"13.25"' float.str
-    Enter the Python script to encode 'value'.
-    return str(value)
-    Enter the Python script to decode 'value'.
-    return float(value)
-    ^D
-
-As you can see, you need to give the codec some examples. Here they're
-symmetric but that's not a requirement; for instance, a ``bool`` codec for our
-home automation system could accept a wide range of ``true``-ish or
-``false``-ish strings but it would always output ``ON`` and ``OFF``.
-
-Associating this codec with a path is slightly more involved::
-
-    $ mkv codec convert -c float.str floatval 'monitor.#.value'
-
-This associates
-
-* the float-to-string codec we just created
-
-* all paths that start with ``monitor`` and end with ``value``
-
-with the codec list named ``floatval``. As not every user needs stringified
-numbers, we also need to tell MoaT-KV which users to apply this codec to::
-
-    $ mkv auth user modify --aux codec=floatval name=joe
-	
-Thus, Joe will read and write ``value`` entries as strings::
-
-    $ mkv data monitor.a.b.c.value set -v : 99.5
-    $ mkv data monitor.a.b.c.thing set -v : 12.3
-    $ mkv data monitor get -rd_
-    a:
-      b:
-        c:
-          value:
-            _:
-              99.5
-          thing:
-            _:
-              '12.3'
-
-This is especially helpful if Joe is in fact an MQTT gateway which only
-receives and transmits strings. A real-world application would use
-binary strings, not Unicode strings.
-
-
-Limitations
------------
-
-MoaT-KV currently can't translate paths, or merge many values to one entry's attributes.
-
-You can use either active objects (add some code to their ``set_value``
-methods) or code objects (listen to A and write to B) to effect such
-translations. There are some caveats:
-
-* All such data are stored twice.
-
-* Replacing a value with the exact same value still counts as a change.
-  Don't set up an endless loop.
-
-* You need to verify that the two trees match when you start up, and decide
-  which is more correct. (The ``tock`` stamp will help you here.) Don't
-  overwrite changes that arrive while you do that.
-
-
-Dynamic configuration
-=====================
-
-For some use cases, you might want to configure MoaT-KV dynamically instead
-of by a static configuration file.
-
-This is not always feasible; in particular, the "logging" and "server"
-sections are imported once. Also, options used for connecting to another
-MoaT-KV server cannot be set dynamically because you need them before the
-data are available.
-
-Other options may be overridden by storing a new values at ``.moat kv config
-<name>``. It is not possible to be more specific. (TODO)
-
-If a client's ACLs do not allow reading a config entry, it will be silently
-ignored.
-
-A config entry's ``_watch`` property will trigger when the entry is updated.