frugal-iot-server 0.3.8 → 0.3.9

package/INSTALLATION.md

@@ -40,42 +40,6 @@ Any of these work. What differs is only how long the install takes, and how much
 A Zero W runs the server perfectly well once installed. It is *installing* that is slow, because a
 single 1 GHz core has to unpack several hundred packages and compile one of them.
 
-## The short way: one script
-
-Steps 3 to 8 below can be done for you. Once you can ssh into the Pi (steps 1 and 2, which need a
-person with an SD card):
-
-```
-curl -fsSLO https://raw.githubusercontent.com/mitra42/frugal-iot-server/main/scripts/install-pi.sh
-bash install-pi.sh --org myfarm --name "My Farm" --email you@example.com --phone +61123456789
-```
-
-It asks for anything it needs that you did not pass, says what it is doing as it goes, and stops at
-the first thing that fails - leaving a log, and telling you to run `frugal-iot-diagnostic`. Running
-it again after a failure carries on rather than starting over, which matters on a Pi Zero W where one
-step takes about 40 minutes. `--dry-run` checks the arguments and shows the plan without touching the
-machine.
-
-**Passwords** are asked for, not invented, unless you say otherwise: it prompts for each (twice, not
-echoed) and generates one if you just press Enter. Pass them as `--superuser-password`,
-`--login-password` and `--broker-password` to skip the prompts, `--random-passwords` to have all
-three generated, or `--yes` to ask nothing at all — which is what to use over ssh with no terminal.
-A password you chose is not written to the log; a generated one has to be, since otherwise you would
-have no way of knowing it.
-
-**It tells you whether a reboot is needed**, rather than leaving you to wonder — it notices a kernel
-or boot-firmware package in the upgrade, the images in `/boot` changing underneath it, and
-`/var/run/reboot-required`. If none of those happened it says so, and there is nothing more to do.
-
-Fetch it with `curl -O` and then run it, rather than piping curl into bash - piped, it has no
-terminal to ask questions at.
-
-It finishes with a live server and a tested broker. It cannot do step 9, pointing your nodes at it,
-because a node learns its broker by being flashed with it - so it prints the settings to give them.
-Nor step 10, HTTPS.
-
-The steps below are what it does, in the same order, if you would rather do it by hand or need to
-understand what went wrong.
 
 ---
 ### 0. What you need before you start
@@ -163,6 +127,7 @@ From your laptop's terminal:
 ```
 ssh pi@frugaliot.local
 ```
+> **Pi Zero W:** This can take a long time to connect, try `ping frugaliot.local` first to check it is alive
 
 Say `yes` to the fingerprint question, then give the password you set in Imager.
 
@@ -213,6 +178,44 @@ the password — either way `nmtui` is the fix, and it is not worth more time th
 (`sudo nmcli device wifi list` shows each nearby network's SSID and whether it is WPA2 or WPA3;
 WPA3 cannot use a derived key at all.)
 
+## 3-8. The short way: one script
+
+Steps 3 to 8 below can be done for you. Once you can ssh into the Pi (steps 1 and 2, which need a
+person with an SD card):
+
+```
+curl -fSLO https://raw.githubusercontent.com/mitra42/frugal-iot-server/main/scripts/install-pi.sh
+bash install-pi.sh --org myfarm --name "My Farm" --email you@example.com --phone +61123456789
+```
+
+It asks for anything it needs that you did not pass, says what it is doing as it goes, and stops at
+the first thing that fails - leaving a log, and telling you to run `frugal-iot-diagnostic`. Running
+it again after a failure carries on rather than starting over, which matters on a Pi Zero W where one
+step takes about 40 minutes. `--dry-run` checks the arguments and shows the plan without touching the
+machine.
+
+**Passwords** are asked for, not invented, unless you say otherwise: it prompts for each (twice, not
+echoed) and generates one if you just press Enter. Pass them as `--superuser-password`,
+`--login-password` and `--broker-password` to skip the prompts, `--random-passwords` to have all
+three generated, or `--yes` to ask nothing at all — which is what to use over ssh with no terminal.
+A password you chose is not written to the log; a generated one has to be, since otherwise you would
+have no way of knowing it.
+
+**It tells you whether a reboot is needed**, rather than leaving you to wonder — it notices a kernel
+or boot-firmware package in the upgrade, the images in `/boot` changing underneath it, and
+`/var/run/reboot-required`. If none of those happened it says so, and there is nothing more to do.
+
+Fetch it with `curl -O` and then run it, rather than piping curl into bash - piped, it has no
+terminal to ask questions at.
+
+It finishes with a live server and a tested broker. It cannot do step 9, pointing your nodes at it,
+because a node learns its broker by being flashed with it - so it prints the settings to give them.
+Nor step 10, HTTPS.
+
+The steps below are what it does, in the same order, if you would rather do it by hand or need to
+understand what went wrong, otherwise skip to Step 9.
+
+
 ### 3. Update the operating system and install the prerequisites
 
 Logged in, bring the system up to date and reboot:
@@ -692,6 +695,101 @@ mosquitto_sub -h localhost -u myfarm -P '<broker-password>' -t '#' -v
 Every reading from every node should scroll past. Seeing anything here also proves the broker's
 port 1883 is reachable from off the Pi, which is what the nodes need.
 
+### 9a. Optional: outgoing mail, so people can reset a forgotten password
+
+Without this, a forgotten password can only be fixed by you, on the Pi, with
+`npx --no frugal-iot-setpassword <username> <new-password>`. The login page's "Forgot password?"
+link says "Password reset is not available on this server" rather than pretending to send anything.
+
+An offline Pi cannot send mail at all, so skip this unless it has internet access.
+
+Mail sent straight from a home broadband connection is almost always treated as spam, so this
+relays through somebody else's SMTP server rather than sending directly. Gmail is the usual choice
+and is written out in full below; any other provider works the same way.
+
+#### With Gmail
+
+Use a Google account you are willing to have the Pi send as. A separate account for the purpose is
+better than your own mailbox — the password ends up in a file on an SD card in a shed.
+
+**1. Turn on 2-Step Verification** on that account, at
+<https://myaccount.google.com/signinoptions/twosv>.
+
+This is not optional and it is where most people get stuck: **app passwords do not exist until
+2-Step Verification is on**. Until then the page in step 2 simply says the setting is not available,
+without explaining why.
+
+**2. Create an app password** at <https://myaccount.google.com/apppasswords>.
+
+Type a name for it — anything you will recognise later, such as `Frugal IoT Pi` — and create it.
+Google shows you **16 lowercase letters, in four groups of four**, once. Copy them now; you cannot
+come back and read it again, only delete it and make another.
+
+The groups are only there to make it readable. **Type it into the config with the spaces removed**,
+as one 16-character word.
+
+**3. Fill in `config.d/email.yaml`:**
+
+```yaml
+host: smtp.gmail.com
+port: 587
+user: yourname@gmail.com
+pass: "abcdefghijklmnop"          # the 16 characters from step 2, spaces removed
+from: Frugal IoT <yourname@gmail.com>
+```
+
+Things that catch people out with Gmail specifically:
+
+* **`from:` must be the same address as `user:`.** Gmail rewrites the sender to the account you
+  authenticated as, so a different address here does not fail — it quietly arrives as something
+  else, which is harder to diagnose than an error. (An address you have set up under Gmail's
+  "Send mail as" is the one exception.)
+* **`user:` is the full address**, including `@gmail.com`, not just the part before it.
+* **`Username and Password not accepted`** in the server's log means the login password was used
+  instead of an app password. Gmail has not accepted account passwords over SMTP for years.
+* **Port 465 also works** if 587 is blocked where the Pi is; the software works out the encryption
+  from the port number, so change nothing else.
+* **A free Gmail account can send around 500 messages a day.** Password resets will not come close.
+* **Google Workspace administrators can switch app passwords off** for a whole domain. If step 2
+  offers you nothing on a work account, that is why, and you will need a real SMTP relay instead.
+
+#### With any other provider
+
+The same four settings, with that provider's SMTP host:
+
+```yaml
+host: smtp.example.org
+port: 587           # 465 is implicit TLS, 587 and 25 are STARTTLS
+user: frugaliot@example.org
+pass: an-app-password-not-your-login-password
+from: Frugal IoT <frugaliot@example.org>
+```
+
+Fastmail, Zoho, Proton (via its bridge) and the rest all issue app passwords in much the same way,
+and for the same reason: one leaking off a Pi then costs you one mailbox rather than the account.
+
+#### Checking it, either way
+
+Restart the server and look at its output:
+
+* `Sending mail via smtp.gmail.com:587 as ...` — configured.
+* `Not sending mail (no host/from in config.d/email.yaml) ...` — it did not find a `host` and a
+  `from`, so the reset link will say it is unavailable.
+
+Then use "Forgot password?" on the login page with your own account. If the mail never arrives, the
+reason is in the server's log — the page deliberately gives the same answer whether or not the
+address is known to it, so it cannot be used to find out who has an account here.
+
+Two more things worth knowing:
+
+* `pass` sits in a file on the SD card in plain text. Give that mailbox nothing else to lose.
+* If the Pi is behind a proxy that does not set `X-Forwarded-Host`, the link in the mail will point
+  at the wrong address. Set `baseurl: https://your.address` in the same file.
+
+The reset code itself is stored nowhere — it is a hash of the account, its current password and the
+time, valid between five and ten minutes and dead the moment it is used. There is no table to
+maintain and nothing to clean up. Restarting the server invalidates any code already sent.
+
 ### 10. HTTPS and over-the-air firmware updates
 
 **To be written.** Everything above gives you a plain HTTP server on your local network, which is
@@ -726,9 +824,6 @@ It comes in three parts, and only the first is a one-off:
 * **11b — authorizing this Pi on that server.** One command, run there, once per Pi.
 * **11c — pointing this Pi at it.** One command, run here, once per Pi.
 
-If someone has already done 11a for the server you are bridging to — the usual case once the first
-Pi is working — **start at 11b**.
-
 **What you get, and what you do not.** While the link is up, readings appear on production within
 a second or so. While it is down, the Pi records everything as usual and production simply has a
 gap — the readings taken during an outage never reach it. That is a deliberate choice: the
@@ -740,8 +835,8 @@ takes effect when it reconnects.
 
 #### 11a. Preparing a production server to accept bridges
 
-Once per production server, not per Pi — and someone has probably already done it for
-`frugaliot.naturalinnovation.org`. If so, skip to 11b.
+Once per production server, not per Pi — and if bridging to `frugaliot.naturalinnovation.org`
+this has already been done. If so, skip to 11b.
 
 **Add a listener for bridges.** A Mosquitto bridge speaks MQTT or MQTT-over-TLS and cannot use
 WebSockets, so the existing `wss://` path that browsers and servers use cannot carry this. It needs
@@ -894,7 +989,7 @@ Once per Pi. Run the command 11b printed, from this installation's directory:
 
 ```
 cd ~/frugal-iot
-npx --no frugal-iot-addbridge-pi --org <org-id> --host <prod-host> --account bridge-<site-name>
+npx --no frugal-iot-addbridge-pi <org-id> <prod-host> bridge-<site-name>
 ```
 
 It asks for the password rather than taking it on the command line, so it stays out of your shell
@@ -1060,7 +1155,11 @@ npx --no frugal-iot-clearretained 'myfarm/lotus/+/sht/temperture/#'
 ```
 
 Quote the pattern or the shell will expand it. That lists what is retained and changes nothing;
-add `--delete` to the same command to remove it.
+add the word `delete` at the end of the same command to remove it.
+
+> `delete` is a bare word rather than a `--delete` flag on purpose. npm parses the command line of
+> an `npx` invocation itself and swallows any `--flag` it does not recognise, so `--delete` would
+> never reach the script and it would quietly list instead of deleting.
 
 > Look before deleting. A node's `min`, `max`, `color` and `wired` settings are retained messages
 > too, and they are how the dashboard knows how to draw it — delete those and the node has to be

package/config.d/email.yaml

@@ -0,0 +1,23 @@
+# Outgoing mail. Only used by the password reset, which is unavailable until this is filled in -
+# the login page says so rather than pretending to have sent something.
+#
+# Leave it commented out on a server that should not send mail.
+#
+# For Gmail - the common case, written out step by step in INSTALLATION.md section 9a:
+#   1. Turn on 2-Step Verification: https://myaccount.google.com/signinoptions/twosv
+#      App passwords do not exist until you do, which is where most people get stuck.
+#   2. Create an app password:      https://myaccount.google.com/apppasswords
+#      You get 16 lowercase letters in four groups of four, shown once. Remove the spaces.
+#   3. "from" must be the same address as "user" - Gmail silently rewrites anything else.
+#
+# host: smtp.gmail.com
+# port: 587           # 465 is implicit TLS, 587 and 25 are STARTTLS; secure: is derived from this
+# user: yourname@gmail.com
+# pass: "abcdefghijklmnop"        # the app password, NOT your Google account password
+# from: Frugal IoT <yourname@gmail.com>
+#
+# Any other provider is the same four settings with its own SMTP host.
+#
+# The reset email contains a link back to this server. Set baseurl if the server cannot work out
+# its own public address from the request - behind a proxy that does not set X-Forwarded-Host, say.
+# baseurl: https://frugaliot.example.org

package/config.d/schema/devices.yaml

@@ -59,5 +59,3 @@ soil:
 sonoff:
   # The relay is what someone wants to see; the control that drives it comes second.
   front:    [relay/on, controlhysteresis]
-temp:
-  front:    [aht20/temperature, bmp280/pressure, controlhysteresis]

package/config.d/schema/modules.yaml

@@ -20,11 +20,89 @@ aht21:
   topics:
     - leaf: temperature
     - leaf: humidity
+# The DS18B20 probes on an FF-OpenMPPT board - see frugal-iot-irrigation. Separate modules rather
+# than one, because the three measure quite different things and want different ranges; the class
+# is the same Sensor_DS18B20 as the `ds18b20` module below.
+airtemp:
+  name: Air Temperature
+  topics:
+    - leaf: airtemp
+      leaf_from: temperature
+      name: Air Temperature
+      min: -20
+      max: 60
+analog:
+  name: "Analog Out"
+  topics:
+    - leaf: volts
 battery:
   name: "Battery"
   insidefrugaliot: true
   topics:
     - leaf:   battery
+# How much of its rated capacity the battery still has, inferred from an overnight discharge.
+# A trend over weeks, not a reading; see control_health.h. Named batteryhealth because `health` is
+# already taken by the device's own reporting.
+batteryhealth:
+  name: Battery Health
+  topics:
+    - leaf: health
+      leaf_from: percent
+      name: Health
+      duplicates:
+        significantdate: 86400000
+        significantvalue: 1
+    - leaf: state
+      leaf_from: controltext
+      name: State
+      rw: r
+    - leaf: soc
+      leaf_from: controlfloat
+      name: Charge
+      min: 0
+      max: 100
+    - leaf: capacity
+      leaf_from: controlfloat
+      name: Capacity Ah
+      min: 0
+      max: 500
+      wireable: false
+    - leaf: load
+      leaf_from: controlfloat
+      name: Average load A
+      min: 0
+      max: 50
+      wireable: false
+    - leaf: active
+      leaf_from: controlint
+      name: Irrigation active
+      min: 0
+      max: 16
+    - leaf: panelwatts
+      leaf_from: controlfloat
+      name: Panel watts
+      min: 0
+      max: 2000
+      wireable: false
+    - leaf: storageratio
+      leaf_from: controlfloat
+      name: Storage ratio
+      min: 0
+      max: 50
+      rw: r
+      wireable: false
+    - leaf: advice
+      leaf_from: controltext
+      name: Advice
+      rw: r
+batttemp:
+  name: Battery Temperature
+  topics:
+    - leaf: batttemp
+      leaf_from: temperature
+      name: Battery Temperature
+      min: -20
+      max: 60
 bme280:
   name: "BME280"
   topics:
@@ -131,11 +209,11 @@ ens160:
       leaf_from: controlfloat
       name: Temperature
       max:  50
-      color: red
+      color: "#ff0000"
     - leaf: humidity
       leaf_from: controlfloat
       name: Humidity
-      color: blue
+      color: "#0000ff"
     - leaf: aqi
     - leaf: tvoc
     - leaf: eco2
@@ -177,6 +255,16 @@ health:
   topics:
     - leaf:     wifibars
     - leaf:     wifissid
+# The charge controller's own heatsink. Either a DS18B20 or a pair of diodes read on an ADC pin,
+# depending on the board - the reading means the same either way.
+heatsink:
+  name: Heatsink Temperature
+  topics:
+    - leaf: heatsink
+      leaf_from: temperature
+      name: Heatsink Temperature
+      min: 0
+      max: 120
 ht:
   name: HT
   topics:
@@ -190,11 +278,64 @@ ina219:
     - leaf: current
     - leaf: power
     - leaf: load
+irrigation:
+  name: Irrigation
+  topics:
+    - leaf: hour
+      leaf_from: controlint
+      name: Start hour
+      min: 0
+      max: 23
+    - leaf: minute
+      leaf_from: controlint
+      name: Start minute
+      min: 0
+      max: 59
+    - leaf: maxminutes
+      leaf_from: controlfloat
+      name: Max minutes per sector
+      min: 0
+      max: 120
+    - leaf: enabled
+      leaf_from: controlintoggle
+      name: Enabled
+    - leaf: tank
+      leaf_from: controlfloat
+      name: Tank level
+      color: "#0000ff"
+    - leaf: tankstart
+      leaf_from: controlfloat
+      name: Tank level to start
+    - leaf: tankempty
+      leaf_from: controlfloat
+      name: Tank level to stop
+    - leaf: power
+      leaf_from: controlintoggle
+      name: Power ok
+      wireable: true
+    - leaf: solar
+      leaf_from: controlintoggle
+      name: Input power
+      wireable: true
+    - leaf: pump
+      leaf_from: controlouttoggle
+      name: Pump
+    - leaf: active
+      leaf_from: controlint
+      name: Active sector
+      min: 0
+      max: 16
 lcd:
   summary: false   # a message is not a reading
   name: LCD
   topics:
     - leaf: message
+# The switched load output - OSPIT's pin 14 in its "load" role, opened by the low-voltage
+# disconnect. An Actuator_Digital, so the same shape as `relay`.
+load:
+  name: Load
+  topics:
+    - leaf:     on
 ledbuiltin:
   name: LED
   slot: ledbuiltin
@@ -211,6 +352,96 @@ lux:
   name: Light meter
   topics:
     - leaf:     lux
+# Solar charge control - see frugal-iot-irrigation. `step` is the raw DAC value because that is
+# unambiguous; `vmpp` is what the node PREDICTS that step asks the hardware for, which is a guess
+# until it has been checked against a meter.
+mppt:
+  name: Charge Control
+  topics:
+    - leaf: automatic
+      leaf_from: controlintoggle
+      name: Automatic
+    - leaf: state
+      leaf_from: controltext
+      name: State
+      rw: r
+    - leaf: profile
+      leaf_from: controlint
+      name: Battery type
+      min: 0
+      max: 3
+      wireable: false
+    - leaf: chargeend
+      leaf_from: controlfloat
+      name: Charge end
+      min: 10000
+      max: 16000
+      wireable: false
+    - leaf: tempcoeff
+      leaf_from: controlfloat
+      name: Temp coefficient
+      min: 0
+      max: 60
+      wireable: false
+    - leaf: hotcharge
+      leaf_from: controlfloat
+      name: Hot battery
+      min: 10000
+      max: 16000
+      wireable: false
+    - leaf: batttemp
+      leaf_from: controlfloat
+      name: Battery temp
+      min: -20
+      max: 60
+    - leaf: heatsink
+      leaf_from: controlfloat
+      name: Heatsink temp
+      min: 0
+      max: 120
+    - leaf: target
+      leaf_from: controlfloat
+      name: Target
+      min: 10000
+      max: 16000
+      rw: r
+      wireable: false
+    - leaf: step
+      leaf_from: controlint
+      name: DAC step
+      min: 0
+      max: 255
+      wireable: false
+    - leaf: panel
+      leaf_from: controlfloat
+      name: Panel
+      min: 0
+      max: 30000
+    - leaf: battery
+      leaf_from: controlfloat
+      name: Battery
+      min: 0
+      max: 16000
+    - leaf: voc
+      leaf_from: controlfloat
+      name: Open circuit
+      min: 0
+      max: 30000
+      rw: r
+      wireable: false
+    - leaf: vmpp
+      leaf_from: controlfloat
+      name: Panel target
+      min: 0
+      max: 30
+      rw: r
+      wireable: false
+    - leaf: dacvolts
+      leaf_from: controlfloat
+      name: DAC volts
+      min: 0
+      max: 3.3
+      rw: r
 ms5803:
   name: MS5803
   topics:
@@ -221,6 +452,25 @@ ota:
   insidefrugaliot: true
   topics:
     - leaf:     key
+panel:
+  name: Solar Panel
+  topics:
+    - leaf: panel
+      leaf_from: voltage
+      name: Panel Voltage
+      max: 30000
+pcbtemp:
+  name: Board Temperature
+  topics:
+    - leaf: pcbtemp
+      leaf_from: temperature
+      name: Board Temperature
+      min: -20
+      max: 100
+pump:
+  name: Pump
+  topics:
+    - leaf:     on
 rain:
   name: Rain
   topics:
@@ -230,16 +480,93 @@ relay:
   name: Relay
   topics:
     - leaf:     on
+# One sector of an irrigation run - see modules "irrigation". A node with several sectors publishes
+# sector1, sector2... and the client resolves those numbered instances back to this entry, so only
+# the one definition is needed here however many sectors a node has.
+sector:
+  name: Irrigation sector
+  topics:
+    - leaf: moisture
+      leaf_from: controlfloat
+      name: Moisture
+      color: "#a52a2a"
+    - leaf: target
+      leaf_from: controlfloat
+      name: Target
+    - leaf: enable
+      leaf_from: controlintoggle
+      name: Enable
+    - leaf: valve
+      leaf_from: controlouttoggle
+      name: Valve
 sht:
   name: SHT
   topics:
     - leaf: temperature
     - leaf: humidity
+# State of charge, estimated from battery voltage - reporting only, never controlled from. See
+# control_soc.h in frugal-iot-irrigation for what the estimate is and is not worth.
+soc:
+  name: State of Charge
+  topics:
+    - leaf: soc
+      leaf_from: percent
+      name: Charge
+      duplicates:
+        significantdate: 900000
+        significantvalue: 1
+    - leaf: charging
+      leaf_from: controlouttoggle
+      name: Charging
+    - leaf: battery
+      leaf_from: controlfloat
+      name: Battery
+      min: 0
+      max: 16000
+    - leaf: panel
+      leaf_from: controlfloat
+      name: Panel
+      min: 0
+      max: 30000
+    - leaf: profile
+      leaf_from: controlint
+      name: Battery type
+      min: 0
+      max: 3
+      wireable: false
 soil:
   name: Soil
   topics:
     - leaf: soil
+tank:
+  name: Water Tank
+  topics:
+    - leaf: tank
+soilmodbus:
+  name: "Soil Probe"
+  topics:
+    # Reuses the shared humidity and temperature definitions - same ranges, logging and
+    # duplicate rules - overriding only what soil makes different from air.
+    - leaf: humidity
+      name: "Soil Moisture"
+      color: brown
+      units: "%"      # Volumetric water content, not the relative humidity the shared leaf means
+    - leaf: temperature
+      name: "Soil Temperature"
+      color: brown
 ultrasonic:
   name: "Ultrasonic Distance"
   topics:
     - leaf: ultrasonic
+# The irrigation valves - Actuator_Digital instances, so the same shape as `relay`. A node with
+# several publishes valve1, valve2... which the client resolves back to this entry.
+# A switched USB supply - an Actuator_Digital, so the same shape as `relay`. On the FF board this
+# shares a pin with the third irrigation valve; a node has one or the other.
+usb:
+  name: USB Supply
+  topics:
+    - leaf:     on
+valve:
+  name: Valve
+  topics:
+    - leaf:     on