frugal-iot-server 0.3.9 → 2.0.0

package/INSTALLATION.md

@@ -503,9 +503,9 @@ npx --no frugal-iot-addorganization myfarm "My Farm" you@example.com +6112345678
 
 The arguments are: organization id, display name, your email, your phone (`+` and digits only),
 and a password. That one command writes `config.d/organizations/myfarm.yaml`, creates a login account
-named after the organization (`myfarm`), grants it admin rights, creates its OTA directory, **and adds
-the organization's account to the broker's password file** — which is why the broker had to be
-installed first.
+named after the organization (`myfarm`), grants it admin, read and write rights, creates its OTA
+directory, **and adds the organization's account to the broker's password file** — which is why the
+broker had to be installed first.
 
 * **Organization id `myfarm`** — must be 1–10 lower-case letters or digits. It becomes the first part
   of every MQTT topic, so it must match what your sensor nodes are configured to publish to.
@@ -833,6 +833,15 @@ in the instant the link returned. A gap is honest; that would not be. Controls a
 other way round, and *are* queued, so turning something on from production while the Pi is offline
 takes effect when it reconnects.
 
+**People, as well as readings.** A bridge relays topics, not accounts, so on its own it does not let
+someone registered on production log in here. Step 11b also prints a *replica token*, and step 11c
+asks for it; with it, this Pi pulls that organization's logins and permissions from production every
+fifteen minutes, so the same people can use this Pi's dashboard — including while production is
+unreachable, which is the point of the Pi. It is optional: leave the token blank and the bridge
+relays readings as before. Nothing derived from a password travels either way, and each server
+issues its own broker credentials, so the same person has different ones here and there. Revoking a
+permission on production removes it here on the next pull. See SECURITY.md.
+
 #### 11a. Preparing a production server to accept bridges
 
 Once per production server, not per Pi — and if bridging to `frugaliot.naturalinnovation.org`
@@ -908,12 +917,16 @@ installed by hand, the written procedure for doing that.
 > Note the asymmetry with the Pi end below: a *certificate* change needs only `reload`, but adding
 > or changing a *bridge* needs a full `restart`, because Mosquitto does not reload bridges.
 
-**Turn on access control.** Without it, any account that can log in to the broker can publish and
-subscribe anywhere on it, so a per-Pi account would be no more confined than the organization's
-own. This is the part to plan carefully, because Mosquitto's ACL file is **deny-by-default**: the
-moment the broker names an `acl_file`, every existing account with no entry in it stops working.
-So the file that restricts new bridge accounts has to grant the existing organizations what they
-already have, in the same change.
+**Turn on access control** — if this broker predates the release that ships it. Since
+`extras/aclfile` became part of the base install, a Pi set up with `install-pi.sh` already has this
+and there is nothing to do here; the steps below are for a broker built before that, or one whose
+configuration was written by hand.
+
+Without it, any account that can log in to the broker can publish and subscribe anywhere on it, so
+a per-Pi account would be no more confined than the organization's own. This is the part to plan
+carefully, because Mosquitto's ACL file is **deny-by-default**: the moment the broker names an
+`acl_file`, every existing account with no entry in it stops working. So the file that restricts new
+bridge accounts has to grant the existing organizations what they already have, in the same change.
 
 ```
 sudo tee /etc/mosquitto/aclfile >/dev/null <<'EOF'
@@ -923,12 +936,21 @@ sudo tee /etc/mosquitto/aclfile >/dev/null <<'EOF'
 # first element of every Frugal IoT topic is the organization id - so this is what the nodes,
 # dashboards and the server's own logger already do. It just stops being optional.
 pattern readwrite %u/#
+
+# Whether a bridge is up, for frugal-iot-diagnostic. "pattern readwrite %u/#" does not match $SYS,
+# so without this the diagnostic reports every bridge as never having connected. Mosquitto logs a
+# warning that this pattern contains no %u - harmless, and a "topic" line would apply to anonymous
+# clients only, which is not what is wanted.
+pattern read $SYS/broker/connection/+/state
 EOF
-sudo chmod 644 /etc/mosquitto/aclfile
+sudo chown mosquitto:mosquitto /etc/mosquitto/aclfile
+sudo chmod 600 /etc/mosquitto/aclfile
 ```
 
-`chmod 644` matters: like the password file and the TLS key, the broker reads this *after* dropping
-to the `mosquitto` user, so a root-only file stops it starting. Then name it in the configuration
+The ownership matters: like the password file and the TLS key, the broker reads this *after*
+dropping to the `mosquitto` user, so a root-only file stops it starting. Mode 600 owned by
+`mosquitto` rather than 644 owned by root — 2.0.21 accepts the latter but warns on every start that
+a file it does not own, or one that is world readable, "will be refused by a future version". Then name it in the configuration
 and restart:
 
 ```
@@ -949,10 +971,12 @@ out, delete `/etc/mosquitto/conf.d/zy-frugal-iot-acl.conf` and restart.
 
 Nothing needs adding here per Pi: `frugal-iot-addbridge-prod` in 11b appends each bridge's own rule.
 
-> One thing this costs: `pattern readwrite %u/#` also denies `$SYS`, so the production broker can no
-> longer tell anyone which bridges are currently connected. The Pi still reports its own bridge
-> state to `frugal-iot-diagnostic`, which is where you would look anyway, so this is a fair trade —
-> but it is why production's dashboard cannot show a site as up or down.
+> `pattern readwrite %u/#` on its own denies `$SYS`, which would stop `frugal-iot-diagnostic`
+> reporting whether a bridge is connected — it reads `$SYS/broker/connection/+/state` as the
+> organization. The second rule above restores exactly that one topic and nothing else. Reading all
+> of `$SYS` would also work and would hand every account the broker's client counts, subscription
+> counts and traffic totals, which on a multi-organization broker tells each organization about the
+> others.
 
 #### 11b. Authorizing this Pi (run on the production server)
 
@@ -1204,10 +1228,19 @@ command for each. That matters most for the files you copy somewhere else — a
 `extras/mosquitto.conf` or `extras/frugaliot.service` does nothing until you install it again:
 
 ```
-sudo cp extras/mosquitto.conf /etc/mosquitto/conf.d/frugal-iot.conf && sudo systemctl restart mosquitto
-sudo cp extras/frugaliot.service /etc/systemd/system/ && sudo systemctl daemon-reload && sudo systemctl restart frugaliot
+sudo cp node_modules/frugal-iot-server/extras/mosquitto.conf /etc/mosquitto/conf.d/frugal-iot.conf
+sudo cp node_modules/frugal-iot-server/extras/aclfile /etc/mosquitto/aclfile
+sudo chown mosquitto:mosquitto /etc/mosquitto/aclfile && sudo chmod 600 /etc/mosquitto/aclfile
+sudo systemctl restart mosquitto
+sudo cp node_modules/frugal-iot-server/extras/frugaliot.service /etc/systemd/system/ && sudo systemctl daemon-reload && sudo systemctl restart frugaliot
 ```
 
+Note those read from `node_modules/frugal-iot-server/extras/`, **not** from `extras/` in this
+directory. `frugal-iot-init` keeps your local copy and only tells you it differs, so the local one is
+the *old* version — copying it would reinstall what you already have. Re-running the install script
+(`bash node_modules/frugal-iot-server/scripts/install-pi.sh`) does the same job and reads from the
+same place.
+
 > It does not compare `config.yaml`, `config.d/mqtt.yaml`, `config.d/logger.yaml` or
 > `config.d/server.yaml`, because those hold your own settings and would differ every time. If a
 > release note mentions a new setting in one of them, compare it yourself:
@@ -1223,6 +1256,32 @@ do. On a Pi, compare the two files above and copy across the settings you want.
 Your organizations, accounts, database and logged data are untouched by an upgrade — they live in
 this directory, not in `node_modules`.
 
+### One thing to do by hand after upgrading: grant WRITE
+
+Commanding a device through the API (`/devices/action`, `/devices/property`) now requires a **WRITE**
+permission, which is separate from READ and is *not* implied by ADMIN. New organizations get it
+automatically, but an upgrade does not touch your database, so on an existing server nobody has it
+yet and switching a relay on answers 401.
+
+Grant it from the dashboard — **Admin → People**, pick the person, choose `WRITE` — or in one
+statement, to everyone who already administers an organization:
+
+```
+sqlite3 frugal-iot.db "INSERT OR IGNORE INTO permissions (id, capability, org)
+  SELECT id, 'WRITE', org FROM permissions WHERE capability = 'ADMIN';"
+sudo systemctl restart frugaliot
+```
+
+Check who has it:
+
+```
+sqlite3 frugal-iot.db "SELECT id, org FROM permissions WHERE capability = 'WRITE';"
+```
+
+Do **not** grant WRITE to id `0`. That row means "every logged-in account", including anyone who has
+just registered themselves, and handing them write access is exactly what the WRITE check exists to
+prevent. (Read is a different matter — see the `(0, 'READ', ...)` rows, which are there on purpose.)
+
 ---
 
 ## Open questions

package/config.d/schema/modules.yaml

@@ -523,11 +523,16 @@ soc:
       name: Battery
       min: 0
       max: 16000
+      # controlfloat is "log: false" because most control inputs are wired from a topic that is
+      # already logged in its own right. Battery and panel are not - the node measures and
+      # publishes them here and nowhere else, so without this they were never archived.
+      log: true
     - leaf: panel
       leaf_from: controlfloat
       name: Panel
       min: 0
       max: 30000
+      log: true
     - leaf: profile
       leaf_from: controlint
       name: Battery type
@@ -554,6 +559,16 @@ soilmodbus:
     - leaf: temperature
       name: "Soil Temperature"
       color: brown
+# The node's clock - what time it thinks it is and how far from UTC it displays. Infrastructure
+# rather than a sensor, so it belongs in the Frugal_IoT drop-down alongside ota, and neither leaf is
+# logged. Without this entry both leaves resolved to no schema at all, so every message from them
+# reached valueFromText with no type and was reported as "Unrecognized message type: undefined".
+time:
+  name: Time
+  insidefrugaliot: true
+  topics:
+    - leaf: epoch
+    - leaf: offset
 ultrasonic:
   name: "Ultrasonic Distance"
   topics:

package/config.d/schema/topics.yaml

@@ -273,6 +273,18 @@ eco2:
     significantvalue: 5
   rw:       r
   graphable:  true
+# Unix time in seconds, as the node's clock currently reads it - published by the "time" module so
+# the client can see how far a node has drifted. A clock reading rather than a measurement: logging
+# it would fill the CSVs with a number that only ever counts up.
+epoch:
+  leaf:     epoch
+  name:     Epoch
+  type:     int
+  log:      false
+  display:  text
+  color:    "#000000"
+  rw:       r
+  units:    s
 gas:
   leaf:     gas
   name:     Gas Resistance
@@ -481,6 +493,17 @@ name:
   display:  text
   rw:       w
   retain:   true
+# How far the node is displaying from UTC, so it can show local time. Like epoch, a setting the node
+# reports back rather than anything measured, so not logged.
+offset:
+  leaf:     offset
+  name:     UTC Offset
+  type:     int
+  log:      false
+  display:  text
+  color:    "#000000"
+  rw:       r
+  units:    min
 on:
   leaf:     on
   name:     On

package/extras/aclfile

@@ -0,0 +1,38 @@
+# Frugal IoT access control. Named by acl_file in extras/mosquitto.conf.
+#
+# DENY BY DEFAULT: the moment the broker names an acl_file, an account with no rule here can still
+# log in but can reach no topic at all. So adding an account means adding its rule in the same
+# change - see scripts/addbridge-prod.zsh, which appends one per Pi.
+#
+# This file is read AFTER mosquitto drops privileges, so it must be readable by the mosquitto user.
+# Install it as mosquitto:mosquitto mode 600 - NOT root-owned 644. Mosquitto 2.0.21 warns
+# "owner is not mosquitto ... Future versions will refuse to load this file" and separately
+# "has world readable permissions", so 644 root works today and is on notice to stop working. The
+# rule is the same one the password file follows: owned by whoever opens it.
+
+# Each organization reaches only its own topic tree. "%u" is the connecting account name, and the
+# first element of every Frugal IoT topic is the organization id - so this is what the nodes,
+# dashboards and the server's own logger already do. It just stops being optional.
+pattern readwrite %u/#
+
+# Whether a bridge is up. With "notifications" left on (the default) a bridge publishes a retained
+# 1 or 0 to $SYS/broker/connection/<remote_clientid>/state, and that is what frugal-iot-diagnostic
+# reads to say whether a site is connected.
+#
+# Needed because "pattern readwrite %u/#" does not match $SYS, so without this line the diagnostic
+# reports every bridge as never having connected - measured, not guessed. This is the narrowest rule
+# that fixes it: it exposes which bridges exist and whether each is up, and nothing else. Reading
+# all of $SYS ("pattern read $SYS/#") would also work and would hand every account the broker's
+# client counts, subscription counts and traffic totals, which on a multi-organization broker tells
+# each organization about the others.
+#
+# Mosquitto logs "Warning: ACL pattern ... does not contain '%c' or '%u'" for the line below, on
+# every start and reload. It is harmless and expected: a "pattern" line is the only way to say
+# "every authenticated account", because a bare "topic" line applies to anonymous clients only.
+# Do not "fix" it by changing it to a topic line - that would silently grant nothing.
+pattern read $SYS/broker/connection/+/state
+
+# Bridge connections are exempt from this file on their local broker - a bridge with no
+# local_username still carries traffic in both directions with the rules above in force (measured on
+# mosquitto 2.0.21). So a Pi's own bridge needs nothing here; it is the far end, on production, that
+# needs a rule, and addbridge-prod.zsh writes it.

package/extras/mosquitto.conf

@@ -9,6 +9,35 @@ protocol websockets
 # directory has to be writable by that same user. See INSTALLATION.md step A5.
 password_file /var/lib/mosquitto/passwords
 
+# Which topics each account may reach. Without this, every account that can log in can publish and
+# subscribe anywhere on the broker, across all organizations - so an organization's own password,
+# which is handed to every node and every logged-in browser, would reach every other organization's
+# data too.
+#
+# DENY BY DEFAULT: naming this file stops any account that has no rule in it. The shipped file
+# grants each organization its own topic tree, which is all the nodes, dashboards and logger use, so
+# nothing should notice. But an account added by hand needs a rule adding by hand.
+#
+# Installed from extras/aclfile - which must be in place BEFORE this line takes effect, or the
+# broker will not start.
+acl_file /etc/mosquitto/aclfile
+
+# Accounts and permissions the server can create and remove while the broker is running, over
+# $CONTROL/dynamic-security/v1. This is what lets each user and each node have its own credential
+# rather than sharing the organization's - see SECURITY-REVIEW.md.
+#
+# The plugin's path is set by the installer, because it varies by distribution: Debian and
+# Raspberry Pi OS put it in /usr/lib/<architecture>/, others in a mosquitto/ subdirectory.
+#
+# Both files must exist before this takes effect or the broker will not start. The state file is
+# created by "mosquitto_ctrl dynsec init", which install-pi.sh does.
+#
+# Plugin checks run BEFORE the built-in password_file and acl_file checks, and an account dynsec
+# does not know is passed through to them - measured on 2.0.21, and what makes it possible to move
+# accounts across one at a time rather than all at once.
+plugin PLUGIN_PATH_SET_BY_INSTALLER
+plugin_opt_config_file /var/lib/mosquitto/dynamic-security.json
+
 # A node on a weak WiFi signal reconnects often, and each connection and disconnection would be a
 # line in /var/log/mosquitto/mosquitto.log. On an SD card those writes wear the card out, and
 # nothing is lost by dropping them, since the readings themselves are recorded by the server.

package/frugal-iot-createdb.sql

@@ -20,12 +20,93 @@ CREATE TABLE IF NOT EXISTS `users` (
   `email` TEXT,
   `phone` TEXT
 );
+-- `project` scopes a permission to one project of an organization. Empty string means the whole
+-- organization, which is how every row created before this column existed behaves.
+--
+-- NOT NULL DEFAULT '' rather than nullable, and that matters: SQLite treats NULLs as DISTINCT in a
+-- UNIQUE constraint, so with a nullable column `(2,'READ','dev',NULL)` could be inserted twice and
+-- the constraint that used to prevent duplicate organization-wide rows would silently stop working.
+-- An empty string compares equal to itself, so uniqueness holds. It also keeps the queries simple:
+-- `project = ?` everywhere, never `IS NULL`.
 CREATE TABLE IF NOT EXISTS `permissions` (
   `id` INTEGER NOT NULL,
   `capability` TEXT NOT NULL,
   `org` TEXT NOT NULL,
-  UNIQUE(`id`, `capability`, `org`)
+  `project` TEXT NOT NULL DEFAULT '',
+  UNIQUE(`id`, `capability`, `org`, `project`)
+);
+-- One row per node that has enrolled, holding the broker credential it was issued.
+--
+-- The password is stored, not derived: a node keeps its copy in LittleFS and cannot recompute
+-- anything, so deriving it from a secret would mean that losing the secret strands the whole fleet.
+-- Storing it here is also what makes "the database is the source of truth, and dynsec can be
+-- rebuilt from it" true for nodes as well as users.
+--
+-- No last_seen column on purpose: the logger already tracks that in memory and serves it through
+-- reportNodes(), and writing it here would be a database write per reading - the SD-card wear the
+-- logger's own buffering exists to avoid.
+--
+-- `lora` records whether the firmware was built with LoRaMesher, declared at enrolment. It decides
+-- whether the node joins <org>-gateways, which grants the broader publish a gateway cannot avoid
+-- needing (it republishes other nodes' readings under its own account). Scoped by what the build
+-- can do rather than by what it is doing at the time, because any node that sees WiFi can promote
+-- itself to gateway at runtime.
+CREATE TABLE IF NOT EXISTS `nodes` (
+  `org` TEXT NOT NULL,
+  `project` TEXT NOT NULL,
+  `nodeid` TEXT NOT NULL,
+  `password` TEXT NOT NULL,
+  `lora` INTEGER NOT NULL DEFAULT 0,
+  `enrolled_at` INTEGER NOT NULL,
+  UNIQUE(`org`, `project`, `nodeid`)
 );
+
+-- A Pi authorised to bridge into this server, and the token it pulls users with (SECURITY.md S11).
+--
+-- A bridge relays topics, not accounts, so a person who logs into this server has no account on the
+-- Pi's broker. GET /replica/:org lets the Pi fetch the logins and permissions for the organizations
+-- it hosts, so the same people can log in there - including when this server is unreachable, which
+-- is the reason the Pi exists.
+--
+-- Nothing derived from a password travels: the Pi authenticates the login itself against the
+-- replicated hash and derives its own broker credential from its own user_secret, so the two
+-- brokers issue different passwords for the same person and neither secret is shared.
+--
+-- One token per Pi per organization, so one site can be revoked without disturbing another. The
+-- token is a bearer credential for reading that organization's logins - not for changing anything.
+CREATE TABLE IF NOT EXISTS `bridges` (
+  `org` TEXT NOT NULL,
+  `site` TEXT NOT NULL,
+  `token` TEXT NOT NULL,
+  `created_at` INTEGER NOT NULL,
+  `last_pull` INTEGER,
+  UNIQUE(`org`, `site`)
+);
+
+-- An admin's decision about one node, for the Enrolled column on the dashboard's Nodes card
+-- (SECURITY.md S12). Two states are worth storing:
+--
+--   approved  the next enrolment request from this node id is accepted whatever secret it presents,
+--             and without having to prove it holds a credential - for a node whose secret was
+--             withdrawn, or that was flashed with none, and which cannot be reached physically.
+--             Consumed on use, so an approval admits one node once.
+--   denied    its broker account is deleted and enrolment is refused - the kill switch for a node
+--             publishing bad readings.
+--
+-- Keyed by (org, nodeid), NOT by project: a node states its own project, so keying on it would let
+-- a denied node return by claiming a different one.
+--
+-- Persisted, unlike the record of failed attempts, because a restart must not un-deny a node.
+CREATE TABLE IF NOT EXISTS `node_grants` (
+  `org` TEXT NOT NULL,
+  `project` TEXT NOT NULL DEFAULT '',
+  `nodeid` TEXT NOT NULL,
+  `state` TEXT NOT NULL,
+  `created_by` TEXT NOT NULL DEFAULT '',
+  `created_at` INTEGER NOT NULL,
+  UNIQUE(`org`, `nodeid`)
+);
+
 CREATE TABLE IF NOT EXISTS `projects` (
   `org` TEXT NOT NULL,
   `id` TEXT NOT NULL,