mavlab-logbook-client 0.1.1__tar.gz

mavlab_logbook_client-0.1.1/PKG-INFO

@@ -0,0 +1,95 @@
+Metadata-Version: 2.4
+Name: mavlab-logbook-client
+Version: 0.1.1
+Summary: Download and upload drone flight logs of a MAVLab flight log database (Paparazzi, ArduPilot, PX4, Betaflight)
+License-Expression: GPL-3.0-only
+Project-URL: Source, https://github.com/tudelft/paparazzi_log_parsing/tree/master/python/mavlab-logbook-client
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+
+# mavlab-logbook-client
+
+Download and upload drone flight logs of a [MAVLab flight log database](https://github.com/tudelft/paparazzi_log_parsing)
+from Python, without a browser download or drag and drop. Works with every log the server stores (Paparazzi
+`.data`/`.log`, ArduPilot `.bin`/`.log`/`.tlog`, PX4 `.ulg`, Betaflight `.BFL`, …). Pure Python 3.10+, no dependencies.
+The API follows the [MATLAB tools](https://github.com/tudelft/paparazzi_log_parsing/tree/master/matlab), which share
+the login and the cache.
+
+```sh
+pip install mavlab-logbook-client
+```
+
+```python
+import mavlab_logbook_client as ml
+
+ml.connect("logs.example.org")                     # or ml.connect("logs.example.org", key="logdb_…")
+rows = ml.logbook()                                # the flights you may see, newest first (as pprz_logbook)
+numbers = [r["flight"] for r in rows if "Hex" in r["aircraft"]]
+mats = ml.fetch([1234, 1240])                      # flight numbers, as in /flight/1234: MATLAB exports, one per log
+mats = ml.fetch([1234, 1240], scope="flight")      # one .mat per number instead (as pprz_fetch)
+logs = ml.fetch_raw([1234, 1240])                  # the original logs, untouched: one path per log
+logs[0]                      # e.g. ~/.cache/mavlab-logbook-client/logs/<sha256>/24_10_30__16_27_37_SD.data (.log next to it)
+receipts = ml.upload(["flight.data", "LOG00030.BFL"], wait=True)
+receipts[0]["flight_ids"]                          # the flights the server found, for ml.fetch
+```
+
+`connect(server, key)` does the authentication. The server is required (`https://` is added to a bare host name;
+`http://` is refused except for `localhost`, so keys never travel unencrypted);
+the key defaults to `LOGDB_API_KEY` (a key from your account page, as for `upload_logs.py`; not used when
+`LOGDB_SERVER` names another server), else the key of the last browser login to that server. Values taken from the environment are announced and named
+in errors. Without a key, `connect` opens the browser: sign in, check that the page shows the same code as the
+terminal and press **Allow**. That key acts as you for a week and is stored for the next `connect`.
+`connect(..., login=False)` raises `AuthError` instead (for unattended scripts); `insecure=True` accepts a
+self-signed certificate. `connect` returns a `Client` with the same methods, for using several servers at once.
+
+`ml.revoke_all()` revokes your browser-login keys on every computer (Python and MATLAB) and forgets the stored one.
+Keys made on the account page are revoked there.
+
+**The logbook** (`ml.logbook()`, as MATLAB's `pprz_logbook`) is a list of dicts, one per flight, newest first:
+`flight` (the number for `fetch`), `date` (takeoff, UTC `datetime`; `None` when unknown), `date_ok` (`False` when
+the log's clock was not set and no date was entered), `aircraft`, `duration_min`, `pic`, `project`, `goal`, `verdict`,
+`simulation` (`True` for a simulated flight) and `log`. It holds the flights the parser judged real; `all=True` adds uncertain and ground
+records. Paper records are left out. `pandas.DataFrame(rows)` makes a table of it.
+
+**Fetching** works as MATLAB's `pprz_fetch`: `fetch` downloads the MATLAB exports (`.mat`, v7.3 = HDF5, e.g. for
+`h5py`; the format is in the [MATLAB README](https://github.com/tudelft/paparazzi_log_parsing/tree/master/matlab))
+and exists for Paparazzi logs. `scope` chooses what each file holds: `"selected"` (default) the given flights of
+each log, 10 s before takeoff to 10 s after landing; `"all"` every flight of those logs; `"log"` the whole log; these
+return one file per log. `"flight"` returns one file per number. Exports are cached (shared with MATLAB) and
+reused while they would be identical; a change of the log, parser or logbook data fetches them again.
+
+`fetch_raw` downloads the original log files that hold the flights (`.data` with its `.log`, `.BFL`, `.ulg`, …),
+whole and untouched, one path per log. One never changes (it is named by its SHA-256, which is checked after each
+download). For both, `force=True` downloads again; without a connection or when the server fails (HTTP 5xx) the
+cached copies are used, with a warning.
+
+**Uploads** pair a `.data` with the `.log` next to it (whichever of the two you pass). A log already on the server
+is skipped. `name="Hex 1/LOG00030.BFL"` sets the stored name of a single file, which can link it to a fleet
+aircraft. With `wait=True` each call returns once the server has parsed the log, else as soon as it accepted it.
+
+## Command line
+
+```sh
+mavlab-logbook-client --server logs.example.org status                       # signs in if needed; user, key and cache
+mavlab-logbook-client --server logs.example.org logbook                      # tab-separated flights; --all
+mavlab-logbook-client --server logs.example.org fetch 1234 1240               # prints the .mat paths; --scope flight
+mavlab-logbook-client --server logs.example.org fetch-raw 1234 1240           # prints the paths of the original logs
+mavlab-logbook-client --server logs.example.org upload flight.data LOG00030.BFL --wait
+mavlab-logbook-client --server logs.example.org revoke-all                    # keys of browser logins, all computers
+mavlab-logbook-client --server logs.example.org --key logdb_… fetch 1234      # as connect(server, key)
+```
+
+## Where files are kept
+
+| | Linux | macOS | Windows |
+|---|---|---|---|
+| Login (`config.json`) | `~/.config/mavlab-logbook-client` | `~/Library/Application Support/mavlab-logbook-client` | `%APPDATA%\mavlab-logbook-client` |
+| Cache | `~/.cache/mavlab-logbook-client` | `~/Library/Caches/mavlab-logbook-client` | `%LOCALAPPDATA%\mavlab-logbook-client\Cache` |
+
+`config.json` holds API keys and is readable by you only; `XDG_CONFIG_HOME`/`XDG_CACHE_HOME` are respected on
+Linux. `MAVLAB_LOGBOOK_CACHE` (or `ml.set_cache_dir(folder)`, also seen by MATLAB) and `MAVLAB_LOGBOOK_CONFIG` choose
+other folders. The cache holds `mat/<server>/` (exports, from `fetch` and MATLAB), `logs/<sha256>/` (original files) and
+`flights/<server>/` (which log each flight is in, for offline use); delete it at any time to free space.

mavlab_logbook_client-0.1.1/README.md

@@ -0,0 +1,83 @@
+# mavlab-logbook-client
+
+Download and upload drone flight logs of a [MAVLab flight log database](https://github.com/tudelft/paparazzi_log_parsing)
+from Python, without a browser download or drag and drop. Works with every log the server stores (Paparazzi
+`.data`/`.log`, ArduPilot `.bin`/`.log`/`.tlog`, PX4 `.ulg`, Betaflight `.BFL`, …). Pure Python 3.10+, no dependencies.
+The API follows the [MATLAB tools](https://github.com/tudelft/paparazzi_log_parsing/tree/master/matlab), which share
+the login and the cache.
+
+```sh
+pip install mavlab-logbook-client
+```
+
+```python
+import mavlab_logbook_client as ml
+
+ml.connect("logs.example.org")                     # or ml.connect("logs.example.org", key="logdb_…")
+rows = ml.logbook()                                # the flights you may see, newest first (as pprz_logbook)
+numbers = [r["flight"] for r in rows if "Hex" in r["aircraft"]]
+mats = ml.fetch([1234, 1240])                      # flight numbers, as in /flight/1234: MATLAB exports, one per log
+mats = ml.fetch([1234, 1240], scope="flight")      # one .mat per number instead (as pprz_fetch)
+logs = ml.fetch_raw([1234, 1240])                  # the original logs, untouched: one path per log
+logs[0]                      # e.g. ~/.cache/mavlab-logbook-client/logs/<sha256>/24_10_30__16_27_37_SD.data (.log next to it)
+receipts = ml.upload(["flight.data", "LOG00030.BFL"], wait=True)
+receipts[0]["flight_ids"]                          # the flights the server found, for ml.fetch
+```
+
+`connect(server, key)` does the authentication. The server is required (`https://` is added to a bare host name;
+`http://` is refused except for `localhost`, so keys never travel unencrypted);
+the key defaults to `LOGDB_API_KEY` (a key from your account page, as for `upload_logs.py`; not used when
+`LOGDB_SERVER` names another server), else the key of the last browser login to that server. Values taken from the environment are announced and named
+in errors. Without a key, `connect` opens the browser: sign in, check that the page shows the same code as the
+terminal and press **Allow**. That key acts as you for a week and is stored for the next `connect`.
+`connect(..., login=False)` raises `AuthError` instead (for unattended scripts); `insecure=True` accepts a
+self-signed certificate. `connect` returns a `Client` with the same methods, for using several servers at once.
+
+`ml.revoke_all()` revokes your browser-login keys on every computer (Python and MATLAB) and forgets the stored one.
+Keys made on the account page are revoked there.
+
+**The logbook** (`ml.logbook()`, as MATLAB's `pprz_logbook`) is a list of dicts, one per flight, newest first:
+`flight` (the number for `fetch`), `date` (takeoff, UTC `datetime`; `None` when unknown), `date_ok` (`False` when
+the log's clock was not set and no date was entered), `aircraft`, `duration_min`, `pic`, `project`, `goal`, `verdict`,
+`simulation` (`True` for a simulated flight) and `log`. It holds the flights the parser judged real; `all=True` adds uncertain and ground
+records. Paper records are left out. `pandas.DataFrame(rows)` makes a table of it.
+
+**Fetching** works as MATLAB's `pprz_fetch`: `fetch` downloads the MATLAB exports (`.mat`, v7.3 = HDF5, e.g. for
+`h5py`; the format is in the [MATLAB README](https://github.com/tudelft/paparazzi_log_parsing/tree/master/matlab))
+and exists for Paparazzi logs. `scope` chooses what each file holds: `"selected"` (default) the given flights of
+each log, 10 s before takeoff to 10 s after landing; `"all"` every flight of those logs; `"log"` the whole log; these
+return one file per log. `"flight"` returns one file per number. Exports are cached (shared with MATLAB) and
+reused while they would be identical; a change of the log, parser or logbook data fetches them again.
+
+`fetch_raw` downloads the original log files that hold the flights (`.data` with its `.log`, `.BFL`, `.ulg`, …),
+whole and untouched, one path per log. One never changes (it is named by its SHA-256, which is checked after each
+download). For both, `force=True` downloads again; without a connection or when the server fails (HTTP 5xx) the
+cached copies are used, with a warning.
+
+**Uploads** pair a `.data` with the `.log` next to it (whichever of the two you pass). A log already on the server
+is skipped. `name="Hex 1/LOG00030.BFL"` sets the stored name of a single file, which can link it to a fleet
+aircraft. With `wait=True` each call returns once the server has parsed the log, else as soon as it accepted it.
+
+## Command line
+
+```sh
+mavlab-logbook-client --server logs.example.org status                       # signs in if needed; user, key and cache
+mavlab-logbook-client --server logs.example.org logbook                      # tab-separated flights; --all
+mavlab-logbook-client --server logs.example.org fetch 1234 1240               # prints the .mat paths; --scope flight
+mavlab-logbook-client --server logs.example.org fetch-raw 1234 1240           # prints the paths of the original logs
+mavlab-logbook-client --server logs.example.org upload flight.data LOG00030.BFL --wait
+mavlab-logbook-client --server logs.example.org revoke-all                    # keys of browser logins, all computers
+mavlab-logbook-client --server logs.example.org --key logdb_… fetch 1234      # as connect(server, key)
+```
+
+## Where files are kept
+
+| | Linux | macOS | Windows |
+|---|---|---|---|
+| Login (`config.json`) | `~/.config/mavlab-logbook-client` | `~/Library/Application Support/mavlab-logbook-client` | `%APPDATA%\mavlab-logbook-client` |
+| Cache | `~/.cache/mavlab-logbook-client` | `~/Library/Caches/mavlab-logbook-client` | `%LOCALAPPDATA%\mavlab-logbook-client\Cache` |
+
+`config.json` holds API keys and is readable by you only; `XDG_CONFIG_HOME`/`XDG_CACHE_HOME` are respected on
+Linux. `MAVLAB_LOGBOOK_CACHE` (or `ml.set_cache_dir(folder)`, also seen by MATLAB) and `MAVLAB_LOGBOOK_CONFIG` choose
+other folders. The cache holds `mat/<server>/` (exports, from `fetch` and MATLAB), `logs/<sha256>/` (original files) and
+`flights/<server>/` (which log each flight is in, for offline use); delete it at any time to free space.

mavlab_logbook_client-0.1.1/mavlab_logbook_client/__init__.py

@@ -0,0 +1,70 @@
+"""Download and upload flight logs of a MAVLab flight log database from Python.
+
+    import mavlab_logbook_client as ml
+    ml.connect("logs.example.org")     # signs in through the browser when needed
+    rows = ml.logbook()                # the flights you may see, newest first
+    mats = ml.fetch([1234, 1240])      # flight numbers, as in /flight/1234; MATLAB exports, as pprz_fetch
+    logs = ml.fetch_raw([1234, 1240])  # the original logs (.data with .log, .BFL, .ulg, ...)
+    ml.upload("LOG00030.BFL", wait=True)
+
+See connect() and Client for the details; the command line tool is ``mavlab-logbook-client``.
+"""
+from .client import AuthError, Client, Error, HTTPError, ProcessingFailed
+from .settings import cache_dir, set_cache_dir
+
+__version__ = "0.1.1"
+__all__ = ["Client", "connect", "logbook", "fetch", "fetch_raw", "upload", "revoke_all", "cache_dir", "set_cache_dir",
+           "Error", "AuthError", "HTTPError", "ProcessingFailed"]
+
+_client = None
+
+
+def connect(server, key=None, *, login=True, insecure=False):
+    """Connect to the logbook server for logbook(), fetch(), fetch_raw(), upload() and revoke_all(), and return the
+    Client.
+
+    server: URL or host name, e.g. "logs.example.org" (https:// is added; http:// only for localhost).
+    key: API key from the account page; default LOGDB_API_KEY, else the key of the last browser login to this
+    server, else a new browser login (sign in, compare the code, press Allow; the key lasts a week).
+    Values taken from the environment are announced. login=False raises AuthError instead of opening the browser.
+    """
+    global _client
+    client = Client(server, key, login=login, insecure=insecure)
+    try:
+        user = client.me()
+    except OSError as exc:
+        raise Error(f"cannot reach {client._origin()}: {exc}") from exc
+    print(f"Connected to {client.server} as {user.get('display_name', '?')}.", flush=True)
+    _client = client
+    return client
+
+
+def _connected():
+    if _client is None:
+        raise Error("not connected: call mavlab_logbook_client.connect() first")
+    return _client
+
+
+def logbook(all=False):
+    """The logbook flights you may see, newest first (dicts with flight, date, aircraft, …); see Client.logbook."""
+    return _connected().logbook(all=all)
+
+
+def fetch(flights, scope="selected", force=False):
+    """Local paths of the MATLAB exports (.mat) of flights by number, as MATLAB's pprz_fetch; see Client.fetch."""
+    return _connected().fetch(flights, scope=scope, force=force)
+
+
+def fetch_raw(flights, force=False):
+    """Local paths of the original logs of flights by number, one per log; see Client.fetch_raw."""
+    return _connected().fetch_raw(flights, force=force)
+
+
+def upload(paths, name=None, wait=False, skip_existing=True):
+    """Upload log files; see Client.upload."""
+    return _connected().upload(paths, name=name, wait=wait, skip_existing=skip_existing)
+
+
+def revoke_all():
+    """Revoke your API keys from browser logins (all computers); see Client.revoke_all."""
+    return _connected().revoke_all()

mavlab_logbook_client-0.1.1/mavlab_logbook_client/__main__.py

@@ -0,0 +1,75 @@
+"""Command line: mavlab-logbook-client [--server URL] [--key KEY] status | logbook | fetch | fetch-raw | upload |
+revoke-all."""
+import argparse
+import sys
+
+from . import __version__, settings
+from .client import Client, Error
+
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(prog="mavlab-logbook-client",
+                                 description="Download and upload logs of a flight log database.")
+    ap.add_argument("--version", action="version", version=__version__)
+    ap.add_argument("--server", required=True, help="server URL or host name")
+    ap.add_argument("--key", help="API key (default: LOGDB_API_KEY, else the stored browser login, else a new one)")
+    ap.add_argument("--insecure", action="store_true", help="do not verify the https certificate (self-signed)")
+    sub = ap.add_subparsers(dest="command", required=True)
+    sub.add_parser("status", help="sign in if needed and show the server, user, key and cache folder")
+    p = sub.add_parser("logbook", help="list the flights you may see, newest first, tab-separated")
+    p.add_argument("--all", action="store_true", help="also flights judged uncertain or ground")
+    p = sub.add_parser("fetch", help="download the MATLAB exports (.mat) of flights by number and print their paths")
+    p.add_argument("flights", nargs="+", type=int, help="flight numbers (the number in /flight/<number>)")
+    p.add_argument("--scope", choices=("selected", "all", "log", "flight"), default="selected",
+                   help="the given flights of each log, all its flights, the whole log, or one file per flight")
+    p.add_argument("--force", action="store_true", help="download again even when cached")
+    p = sub.add_parser("fetch-raw", help="download the original logs of flights by number and print their paths")
+    p.add_argument("flights", nargs="+", type=int, help="flight numbers (the number in /flight/<number>)")
+    p.add_argument("--force", action="store_true", help="download again even when cached")
+    p = sub.add_parser("upload", help="upload log files")
+    p.add_argument("files", nargs="+", help="log files (.data with its .log next to it, .BFL, .ulg, .bin, ...)")
+    p.add_argument("--name", help="stored name of a single file, e.g. 'Hex 1/LOG00030.BFL'")
+    p.add_argument("--wait", action="store_true", help="wait until the server has parsed each log")
+    p.add_argument("--again", action="store_true", help="upload even when the server already has the log")
+    sub.add_parser("revoke-all", help="revoke your keys from browser logins (all computers)")
+    a = ap.parse_args(argv)
+    try:
+        client = Client(a.server, a.key, insecure=a.insecure)
+        if a.command == "status":
+            stored = settings.entry(client.server) or {}
+            user = client.me()
+            print(f"Server: {client.server}")
+            print(f"User: {user.get('display_name')}")
+            print(f"Key: {client.key_source}" + (f", until {stored['expires_at']}"
+                  if client.key_source == "browser login" and stored.get("expires_at") else ""))
+            print(f"Cache: {settings.cache_dir()}")
+        elif a.command == "logbook":
+            columns = ("flight", "date", "date_ok", "aircraft", "duration_min", "pic", "project", "goal", "verdict",
+                       "simulation", "log")
+            print("\t".join(columns))
+            for r in client.logbook(all=a.all):
+                r["date"] = r["date"].strftime("%Y-%m-%d %H:%M:%S") if r["date"] else ""
+                r["duration_min"] = f"{r['duration_min']:.1f}" if r["duration_min"] is not None else ""
+                print("\t".join("" if r[c] is None else str(r[c]).replace("\t", " ") for c in columns))
+        elif a.command == "fetch":
+            for path in client.fetch(a.flights, scope=a.scope, force=a.force):
+                print(path)
+        elif a.command == "fetch-raw":
+            for path in client.fetch_raw(a.flights, force=a.force):
+                print(path)
+        elif a.command == "upload":
+            for r in client.upload(a.files, name=a.name, wait=a.wait, skip_existing=not a.again):
+                extra = f" log {r['log_id']}, flights {r.get('flight_ids')}" if r.get("log_id") else ""
+                print(f"{r.get('name')}: {r['status']}{extra}")
+        elif a.command == "revoke-all":
+            client.revoke_all()
+    except (Error, OSError, ValueError) as exc:
+        print(f"mavlab-logbook-client: {exc}", file=sys.stderr)
+        return 1
+    except KeyboardInterrupt:
+        return 130
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())