frugal-iot-server 2.0.7 → 2.0.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/GRAFANA.md +114 -0
- package/INSTALL-OPENSEARCH.md +415 -0
- package/config.d/opensearch.yaml +23 -0
- package/extras/grafana/frugaliot-readings.json +166 -0
- package/frugal-iot-createdb.sql +38 -0
- package/frugal-iot-server.js +47 -5
- package/lib/admin-guards.js +26 -0
- package/lib/config-for-user.js +10 -0
- package/lib/destinations.js +121 -0
- package/lib/forwarders.js +312 -0
- package/lib/grafana.js +420 -0
- package/lib/opensearch-access.js +245 -0
- package/lib/schema-columns.js +1 -0
- package/package.json +10 -4
- package/public/service-worker.js +1 -1
- package/scripts/backfill-opensearch.js +66 -0
- package/scripts/grafana-token.js +62 -0
package/GRAFANA.md
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
# Charting your readings in Grafana
|
|
2
|
+
|
|
3
|
+
Grafana can chart an organization's readings straight from this server, from the same CSV files the
|
|
4
|
+
logger writes. Nothing is copied into a second database, so Grafana sees the **complete** history of
|
|
5
|
+
every topic, not just what was exported somewhere.
|
|
6
|
+
|
|
7
|
+
It needs nothing extra on the server, and Grafana can run **on your own computer**: all it asks of the
|
|
8
|
+
server is ordinary HTTPS on the port the dashboard already uses, with a token you make for it. (To
|
|
9
|
+
run Grafana on the server instead, see [INSTALL-OPENSEARCH.md](INSTALL-OPENSEARCH.md).)
|
|
10
|
+
|
|
11
|
+
Grafana reads the server with its free **Infinity** plugin, which can call any URL that returns JSON.
|
|
12
|
+
|
|
13
|
+
> **Status:** the server, the data source, the starter dashboard and its topic list were followed on
|
|
14
|
+
> Grafana 13.3 with Infinity 4.1.5 (a Mac, Homebrew install), and the dashboard was loaded in a
|
|
15
|
+
> browser to confirm it draws real readings. Two things are written but not yet tried against a real
|
|
16
|
+
> Grafana: the downloadable data source file (it holds the same settings that worked when set through
|
|
17
|
+
> Grafana's API), and the Grafana card on the dashboard clicked through by hand. Correct this page
|
|
18
|
+
> where they differ.
|
|
19
|
+
|
|
20
|
+
## Setting it up
|
|
21
|
+
|
|
22
|
+
Anyone who can **read** the organization can make a token for themselves - you do not need to be an
|
|
23
|
+
administrator. A token is yours: Grafana sees what *you* can see, so a token made by someone who can
|
|
24
|
+
read only one project shows that project only.
|
|
25
|
+
|
|
26
|
+
1. **Install the Infinity plugin in Grafana.** In Grafana: Administration → Plugins, search for
|
|
27
|
+
"Infinity", install it, and restart Grafana if it asks. (Installing Grafana itself is in
|
|
28
|
+
[INSTALL-OPENSEARCH.md](INSTALL-OPENSEARCH.md).)
|
|
29
|
+
2. **Make a token.** In the dashboard, open the project's back, then the **Grafana** card. Type a name
|
|
30
|
+
for this Grafana (so it can be revoked on its own later) and press *Make a token*. The token is shown
|
|
31
|
+
**once** - only a hash is kept, so it cannot be looked up again. If you lose it, make one with the
|
|
32
|
+
same name, which replaces it.
|
|
33
|
+
3. **Give Grafana the data source.** Either press *Download the data source file* in the same place and
|
|
34
|
+
put it in Grafana's `provisioning/datasources` folder, then restart Grafana; or add it by hand in
|
|
35
|
+
Grafana under Connections → Data sources → Add → **Infinity**:
|
|
36
|
+
- Authentication: **Bearer Token**, with the token
|
|
37
|
+
- Allowed hosts: the address you reached this dashboard at, e.g. `https://frugaliot.naturalinnovation.org`
|
|
38
|
+
4. **Import the dashboard.** In the Grafana card press *Download the dashboard*. In Grafana choose
|
|
39
|
+
Dashboards → New → Import, upload the file, and pick the data source from step 3 when it asks.
|
|
40
|
+
|
|
41
|
+
The dashboard has a list of every topic at the top to choose from, and the time range at the top right.
|
|
42
|
+
The file already has this server and organization filled in.
|
|
43
|
+
|
|
44
|
+
A token lets one Grafana read **one organization**, read-only, and nothing else - and only the
|
|
45
|
+
projects of it that the person who made the token may read. For another
|
|
46
|
+
organization, make its own token, data source and import - the dashboard file is separate for each, so
|
|
47
|
+
importing a second does not replace the first. *Revoke* on the card cuts off a Grafana at once.
|
|
48
|
+
|
|
49
|
+
## What the server provides
|
|
50
|
+
|
|
51
|
+
For building your own panels. All of these need the header `Authorization: Bearer fiot_...`, and the
|
|
52
|
+
token must be for the `<org>` in the URL.
|
|
53
|
+
|
|
54
|
+
| URL | Returns |
|
|
55
|
+
|---|---|
|
|
56
|
+
| `/grafana/<org>/health` | `{"status":"ok"}` |
|
|
57
|
+
| `/grafana/<org>/topics` | every topic with readings: `[{topic, project, node, module, leaf}]`. Narrow with `?project=` and `?node=` |
|
|
58
|
+
| `/grafana/<org>/data?topic=<topic>&from=<time>&to=<time>&maxPoints=<n>` | `[{"time": "2026-09-24T00:00:01.000Z", "value": 22.5}, ...]` |
|
|
59
|
+
|
|
60
|
+
- `topic` is the full topic, `<org>/<project>/<node>/<module>/<leaf>`, e.g.
|
|
61
|
+
`dev/lotus/esp32-944104/aht20/temperature`.
|
|
62
|
+
- `from` and `to` are epoch **milliseconds** (which is what Grafana's `${__from}` and `${__to}` give), or
|
|
63
|
+
an ISO date. Without them you get the last 24 hours.
|
|
64
|
+
- Past `maxPoints` (default 1000, at most 5000) readings are averaged into equal time buckets, so a year
|
|
65
|
+
of data does not travel as millions of rows.
|
|
66
|
+
- Readings that are not numbers (descriptions, colours) are left out - they cannot be charted.
|
|
67
|
+
|
|
68
|
+
A panel in Grafana, using the Infinity data source: Type **JSON**, Parser **Backend**, Format **Time
|
|
69
|
+
series**, URL
|
|
70
|
+
|
|
71
|
+
```
|
|
72
|
+
https://your-server/grafana/dev/data?topic=dev/lotus/esp32-944104/aht20/temperature&from=${__from}&to=${__to}&maxPoints=1000
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
and columns `time` → *Timestamp*, `value` → *Number*. Use a fixed `maxPoints`: Grafana has no macro for a
|
|
76
|
+
panel's width in pixels, and `${__interval_ms}` is a duration, not a count.
|
|
77
|
+
|
|
78
|
+
## Keeping it safe
|
|
79
|
+
|
|
80
|
+
- **A token is a password.** Anyone holding it can read what its maker can read, and the data
|
|
81
|
+
source file the card offers has it inside, so do not leave that file lying about or send it by mail.
|
|
82
|
+
The server cannot show a token again, only replace it.
|
|
83
|
+
- It is **read-only and for one organization**. It cannot change anything, and the server refuses it for
|
|
84
|
+
any other organization.
|
|
85
|
+
- It **acts as the person who made it**, and is checked against their permissions every time it is
|
|
86
|
+
used. If their access to the organization or a project is removed, the token loses it at once - there
|
|
87
|
+
is nothing to revoke. Someone cannot use a token to see more than they could on the dashboard.
|
|
88
|
+
- You see and revoke your own tokens. An administrator of the whole organization sees everyone's and
|
|
89
|
+
can revoke any, for example when a Grafana is lost.
|
|
90
|
+
- **Revoke** on the card cuts a Grafana off at once. Make one token per Grafana, named for it, so one
|
|
91
|
+
can go without disturbing the rest, and revoke any that is no longer used - the card shows when each
|
|
92
|
+
was last used. Tokens do not expire on their own.
|
|
93
|
+
- Grafana should reach the server over **HTTPS**, since the token travels in a header.
|
|
94
|
+
- On a Grafana that other people use, give **each organization its own Grafana organization**, with its
|
|
95
|
+
own data source and token: everyone in a Grafana organization shares its data sources, so anyone who
|
|
96
|
+
can edit a dashboard there can read whatever the token allows.
|
|
97
|
+
- The server limits what a token can cost it: long histories are averaged as they are read, only a few
|
|
98
|
+
reads run at once - and only two for any one organization - (Grafana is told to retry), and the topic
|
|
99
|
+
list is cached for a minute. Each user can hold up to 10 tokens per organization.
|
|
100
|
+
|
|
101
|
+
## From a shell instead
|
|
102
|
+
|
|
103
|
+
If you can get to the server's directory, `npx --no frugal-iot-grafana-token add <org> <username> <name>` makes a
|
|
104
|
+
token for that user, and `list` and `revoke` do what they say. It is the same as the card, except you name whose token it is.
|
|
105
|
+
|
|
106
|
+
## Notes
|
|
107
|
+
|
|
108
|
+
- Readings are written to disk every few minutes, so the server first asks the logger to write out what
|
|
109
|
+
it holds; a chart that reaches "now" is not short.
|
|
110
|
+
- The CSV files can contain a sensor's error value (an SHT sensor reports -45 or 130 when it fails).
|
|
111
|
+
They are real readings as far as the logger knows, so they appear on charts; set the panel's axis
|
|
112
|
+
limits, or filter in Grafana, to leave them out.
|
|
113
|
+
- The first topic in the list is simply the first alphabetically, and may be a sensor that has stopped,
|
|
114
|
+
which shows "No data" for a recent time range. Pick another.
|
|
@@ -0,0 +1,415 @@
|
|
|
1
|
+
# Installing OpenSearch, Grafana and Dashboards
|
|
2
|
+
|
|
3
|
+
This is for **browsing and charting your readings**. All of it is optional - the server and logger work
|
|
4
|
+
without any of it, and the readings are always safe in the CSV files under `data/`.
|
|
5
|
+
|
|
6
|
+
## What the pieces are
|
|
7
|
+
|
|
8
|
+
OpenSearch is **two programs**, and it matters which is which:
|
|
9
|
+
|
|
10
|
+
- **OpenSearch** is the database. It stores and searches the readings, and listens on port 9200.
|
|
11
|
+
- **OpenSearch Dashboards** is a separate web app for searching and charting what is in OpenSearch. It
|
|
12
|
+
holds no readings: it asks OpenSearch for them, over port 9200, using one login in its own
|
|
13
|
+
configuration file. (It also keeps its own saved dashboards *inside* OpenSearch, in an index.)
|
|
14
|
+
|
|
15
|
+
**Grafana** is a third, unrelated program that makes charts. It can read the server directly
|
|
16
|
+
([GRAFANA.md](GRAFANA.md)), needing no OpenSearch at all.
|
|
17
|
+
|
|
18
|
+
## The ways to use it
|
|
19
|
+
|
|
20
|
+
| | Readings are kept in | You look at them with | Set up |
|
|
21
|
+
|---|---|---|---|
|
|
22
|
+
| **Grafana, reading the server** *(start here)* | the server's CSV files, all of them | Grafana on your computer | [GRAFANA.md](GRAFANA.md); needs no OpenSearch |
|
|
23
|
+
| **(a) OpenSearch on the server** | OpenSearch on the production server | OpenSearch Dashboards on **your own computer** | sections 2, 3, 4, 5 |
|
|
24
|
+
| **(b) OpenSearch on your own machine** | OpenSearch on a machine you run, which the server sends readings to | whatever you like - you run it all | section 6 |
|
|
25
|
+
|
|
26
|
+
Either of (a) and (b) can be used with Grafana too.
|
|
27
|
+
|
|
28
|
+
**None of this is for the Raspberry Pi bridge.**
|
|
29
|
+
|
|
30
|
+
> **Status:** the Grafana and Mac steps were followed on macOS with Homebrew (OpenSearch 3.9.0,
|
|
31
|
+
> Dashboards 3.9.0, Grafana 13.3.0). The Linux steps, the logins for Dashboards, and the Apache setup
|
|
32
|
+
> are written from OpenSearch's documentation and have **not been run on a Linux server** - correct them
|
|
33
|
+
> where they differ. [Open questions](#open-questions) lists what to check first.
|
|
34
|
+
|
|
35
|
+
We use **OpenSearch**, not Elasticsearch: it is Apache-licensed, installs with Homebrew and apt, and
|
|
36
|
+
Grafana can read it. OpenSearch Dashboards is its fork of Kibana, so Kibana guides are close but not
|
|
37
|
+
identical. (The reasoning is also in a comment above the `OpenSearch` class in frugal-iot-logger's
|
|
38
|
+
`index.js`, in case the decision is revisited.)
|
|
39
|
+
|
|
40
|
+
How heavy is each? OpenSearch is the heavy one: a Java program with its own 1 GB heap by default.
|
|
41
|
+
Grafana and Dashboards are small by comparison (about 60 MB and 20 MB resident on a Mac, which
|
|
42
|
+
under-reports, so read that loosely).
|
|
43
|
+
|
|
44
|
+
---
|
|
45
|
+
|
|
46
|
+
## 1. On your own computer: Grafana
|
|
47
|
+
|
|
48
|
+
On a Mac:
|
|
49
|
+
|
|
50
|
+
```sh
|
|
51
|
+
brew install grafana
|
|
52
|
+
brew services start grafana
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Open <http://localhost:3000>. Grafana's default login is `admin` / `admin`, and it asks you to change it.
|
|
56
|
+
Then add the Infinity plugin - in Grafana, Administration → Plugins → "Infinity" - or from the shell:
|
|
57
|
+
|
|
58
|
+
```sh
|
|
59
|
+
/opt/homebrew/opt/grafana/bin/grafana cli --homepath /opt/homebrew/opt/grafana/share/grafana \
|
|
60
|
+
--pluginsDir /opt/homebrew/var/lib/grafana/plugins plugins install yesoreyeram-infinity-datasource
|
|
61
|
+
brew services restart grafana
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
For other systems, Grafana's own download page has installers for [Windows, Linux and others](https://grafana.com/grafana/download).
|
|
65
|
+
|
|
66
|
+
Then continue at [GRAFANA.md](GRAFANA.md) to make a token and chart your readings.
|
|
67
|
+
|
|
68
|
+
To stop it: `brew services stop grafana` (Homebrew restarts it at login until you do).
|
|
69
|
+
|
|
70
|
+
---
|
|
71
|
+
|
|
72
|
+
---
|
|
73
|
+
|
|
74
|
+
## 2. (a) On the server: OpenSearch
|
|
75
|
+
|
|
76
|
+
Debian or Ubuntu.
|
|
77
|
+
|
|
78
|
+
`apt-get` does the installing. The `curl` line just downloads the signing key for OpenSearch's
|
|
79
|
+
repository, which apt needs in order to trust it:
|
|
80
|
+
|
|
81
|
+
```sh
|
|
82
|
+
sudo apt-get update && sudo apt-get -y install lsb-release ca-certificates curl gnupg2
|
|
83
|
+
sudo mkdir -p /etc/apt/keyrings
|
|
84
|
+
curl -fsSL https://artifacts.opensearch.org/publickeys/opensearch-release.pgp \
|
|
85
|
+
| sudo gpg --dearmor -o /etc/apt/keyrings/opensearch.gpg
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Then add OpenSearch's apt repository - the `3.x/apt` line from
|
|
89
|
+
[their Debian guide](https://docs.opensearch.org/latest/install-and-configure/install-opensearch/debian/),
|
|
90
|
+
which is printed there for your architecture - and install, choosing an admin password now:
|
|
91
|
+
|
|
92
|
+
```sh
|
|
93
|
+
sudo apt-get update
|
|
94
|
+
sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD='<choose-a-strong-password>' apt-get install opensearch
|
|
95
|
+
sudo systemctl enable opensearch
|
|
96
|
+
sudo systemctl start opensearch
|
|
97
|
+
curl https://localhost:9200 -u 'admin:<that password>' --insecure
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
(That last `curl` only checks that it is answering. `--insecure` here means "do not check the
|
|
101
|
+
certificate", because the one it ships with is self-signed.)
|
|
102
|
+
|
|
103
|
+
This install **has the security plugin switched on**: HTTPS, a login, and that self-signed certificate.
|
|
104
|
+
The password is only used to set up that demo security configuration. Config is in
|
|
105
|
+
`/etc/opensearch/opensearch.yml` and the memory setting in `/etc/opensearch/jvm.options`.
|
|
106
|
+
|
|
107
|
+
- **Memory:** the file ships with `-Xms1g` and `-Xmx1g` (a 1 GB heap). Give both lines the same value;
|
|
108
|
+
more helps with a lot of history, and the machine needs that much free *beyond* what the server,
|
|
109
|
+
broker and logger use. The data is small: all 598,000 readings of the `dev` organization came to
|
|
110
|
+
about 35 MB.
|
|
111
|
+
- **If it will not start**, look at `journalctl -u opensearch`. A common cause on Linux is the kernel
|
|
112
|
+
setting `vm.max_map_count` being below 262144; `sudo sysctl -w vm.max_map_count=262144` fixes it until
|
|
113
|
+
reboot, and a line in `/etc/sysctl.d/` makes it permanent.
|
|
114
|
+
- **Leave OpenSearch listening on the machine itself** (the default). Only Apache, below, is open to the
|
|
115
|
+
internet - never port 9200 directly. Check under [Keeping it safe](#keeping-it-safe).
|
|
116
|
+
|
|
117
|
+
### Tell the server about it
|
|
118
|
+
|
|
119
|
+
Make `config.d/opensearch.yaml` in the server's directory (a commented example ships there), and restart
|
|
120
|
+
the server:
|
|
121
|
+
|
|
122
|
+
```yaml
|
|
123
|
+
opensearch:
|
|
124
|
+
node: https://localhost:9200 # where THIS server reaches OpenSearch: on this machine
|
|
125
|
+
username: admin # what the forwarder writes readings as
|
|
126
|
+
password: <the password you chose>
|
|
127
|
+
insecure: true # OpenSearch's demo certificate cannot be checked; fine on this machine
|
|
128
|
+
admin: # the account that makes the per-organization logins (section 4)
|
|
129
|
+
username: admin
|
|
130
|
+
password: <the same>
|
|
131
|
+
public: https://search.example.org # the address people outside reach it at - section 3
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
This file is never sent to a browser. Using `admin` for everything is the simplest thing that works, not
|
|
135
|
+
the best: a role that can only write `frugaliot-*` for the forwarder would be better, and is not yet
|
|
136
|
+
written. Leave `admin:` and `public:` out until you have done sections 3 and 4.
|
|
137
|
+
|
|
138
|
+
### Send readings to it
|
|
139
|
+
|
|
140
|
+
In the dashboard, open the project's back, then the **Forwarders** card, and add **OpenSearch on this
|
|
141
|
+
server**. There is no address or password to type: they come from the file above. Optionally set *Only
|
|
142
|
+
these topics*, e.g. `dev/lotus/#`; leave it empty to send everything that is logged.
|
|
143
|
+
|
|
144
|
+
It applies at once, with no restart, and creates what it needs in OpenSearch (an index template that
|
|
145
|
+
makes `value` a number) the first time it sends. An organization's administrator can add it for their own
|
|
146
|
+
organization, so only do this if you are happy for them to have their readings in OpenSearch.
|
|
147
|
+
|
|
148
|
+
### Loading the history you already have
|
|
149
|
+
|
|
150
|
+
A forwarder only sends readings from the moment it is switched on. To bring in everything already on
|
|
151
|
+
disk, from the server's directory (safe to repeat, and safe while the server is running):
|
|
152
|
+
|
|
153
|
+
```sh
|
|
154
|
+
npx --no frugal-iot-backfill-opensearch dev dry-run # count what would be sent
|
|
155
|
+
npx --no frugal-iot-backfill-opensearch dev # send it
|
|
156
|
+
npx --no frugal-iot-backfill-opensearch dev 2026-09-01 # only files from that day on
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
### Looking after it
|
|
160
|
+
|
|
161
|
+
- **Old months.** Readings go into one index per organization per month, named like
|
|
162
|
+
`frugaliot-dev-2026.09`. To drop a month: `curl -X DELETE https://localhost:9200/frugaliot-dev-2026.02 -u admin:... --insecure`.
|
|
163
|
+
The CSV files are untouched.
|
|
164
|
+
- **A gap while OpenSearch was down.** The forwarder does not retry a batch it could not send; the CSV
|
|
165
|
+
files have everything, and the backfill above fills the gap.
|
|
166
|
+
- **Starting afresh.** Delete the indices (`frugaliot-*`) and run the backfill again.
|
|
167
|
+
|
|
168
|
+
---
|
|
169
|
+
|
|
170
|
+
---
|
|
171
|
+
|
|
172
|
+
## 3. (a) Apache: HTTPS in front of OpenSearch
|
|
173
|
+
|
|
174
|
+
Dashboards on someone's computer reaches OpenSearch across the internet, so its login and the readings
|
|
175
|
+
must travel over HTTPS. OpenSearch itself stays on the machine (port 9200, loopback); **Apache is put
|
|
176
|
+
in front of OpenSearch**, not in front of Dashboards. It is the same Apache that already sits in front
|
|
177
|
+
of this server - add one more site to it, on its own name, e.g. `search.example.org`.
|
|
178
|
+
|
|
179
|
+
```sh
|
|
180
|
+
sudo a2enmod ssl proxy proxy_http headers
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
```apache
|
|
184
|
+
<VirtualHost *:443>
|
|
185
|
+
ServerName search.example.org
|
|
186
|
+
# your certificate, as for the server's own site (e.g. from certbot)
|
|
187
|
+
SSLEngine on
|
|
188
|
+
SSLCertificateFile /etc/letsencrypt/live/search.example.org/fullchain.pem
|
|
189
|
+
SSLCertificateKeyFile /etc/letsencrypt/live/search.example.org/privkey.pem
|
|
190
|
+
|
|
191
|
+
# Apache to OpenSearch, on this machine. OpenSearch's own certificate is the self-signed demo one,
|
|
192
|
+
# so do not try to check it - the connection never leaves the machine.
|
|
193
|
+
SSLProxyEngine on
|
|
194
|
+
SSLProxyVerify none
|
|
195
|
+
SSLProxyCheckPeerCN off
|
|
196
|
+
SSLProxyCheckPeerName off
|
|
197
|
+
ProxyPreserveHost off
|
|
198
|
+
ProxyPass / https://127.0.0.1:9200/
|
|
199
|
+
ProxyPassReverse / https://127.0.0.1:9200/
|
|
200
|
+
|
|
201
|
+
# Nobody outside manages logins or roles. The server does that itself, on the machine, not through
|
|
202
|
+
# here - and this keeps the admin account's management calls off the internet.
|
|
203
|
+
<LocationMatch "^/_plugins/_security/api">
|
|
204
|
+
Require all denied
|
|
205
|
+
</LocationMatch>
|
|
206
|
+
</VirtualHost>
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
Then `sudo apachectl configtest && sudo systemctl reload apache2`. Check from your own computer:
|
|
210
|
+
`curl https://search.example.org` should answer 401 (it wants a login) - not a block of JSON, and not a
|
|
211
|
+
timeout. Put the address in `public:` in `config.d/opensearch.yaml`.
|
|
212
|
+
|
|
213
|
+
Port 9200 itself stays closed to the network (firewall) - only Apache's 443 is open.
|
|
214
|
+
|
|
215
|
+
---
|
|
216
|
+
|
|
217
|
+
## 4. (a) A login for each person
|
|
218
|
+
|
|
219
|
+
Dashboards needs a login to OpenSearch, and it must not be able to read another organization's readings.
|
|
220
|
+
Each person who wants Dashboards opens, in the dashboard, the project's back, then the **OpenSearch** card,
|
|
221
|
+
and presses **Make a login**. It is theirs, named `frugaliot_<org>_<username>` (for example
|
|
222
|
+
`frugaliot_dev_mitra`), so the same person has a separate one in each organization. The server then,
|
|
223
|
+
through OpenSearch's security API:
|
|
224
|
+
|
|
225
|
+
- makes (or refreshes) a **role** `frugaliot_<org>`, shared by that organization's people, that may **read**
|
|
226
|
+
the organization's readings (`frugaliot-<org>-*`) and nothing else, plus do anything to the
|
|
227
|
+
organization's own saved-dashboards index (`.kibana_<org>*`) - so the people of an organization share
|
|
228
|
+
their saved dashboards, and two organizations do not see each other's;
|
|
229
|
+
- makes the person's **user** with a new password, shown once, and connects it to the role.
|
|
230
|
+
|
|
231
|
+
*Replace the password* makes a new one; *Remove my login* deletes it, and nobody else's. An administrator
|
|
232
|
+
of the whole organization also sees everyone's logins on the card and can remove one - do that when
|
|
233
|
+
someone leaves, because **OpenSearch does not notice a change to who may read** on the dashboard: a login
|
|
234
|
+
stays until it is removed.
|
|
235
|
+
|
|
236
|
+
It needs **READ for the whole organization**: the role covers all of it, so someone who can read only
|
|
237
|
+
some projects is told to use Grafana, whose tokens follow project permissions. It works only for an
|
|
238
|
+
organization whose id is **lower-case letters and digits**, a username of letters, digits and `. _ -`, and
|
|
239
|
+
only with the `admin:` and `public:` lines of `config.d/opensearch.yaml` filled in. The card says what is
|
|
240
|
+
missing.
|
|
241
|
+
|
|
242
|
+
---
|
|
243
|
+
|
|
244
|
+
## 5. (a) OpenSearch Dashboards, on your own computer
|
|
245
|
+
|
|
246
|
+
Whoever wants it installs OpenSearch Dashboards **on their own computer** - not on the server. On a Mac:
|
|
247
|
+
|
|
248
|
+
```sh
|
|
249
|
+
mkdir -p /opt/homebrew/var/run # Homebrew forgets this folder, and Dashboards will not start without it
|
|
250
|
+
brew install opensearch-dashboards
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
(Other systems: [OpenSearch's download page](https://opensearch.org/downloads.html).) Then put the lines the
|
|
254
|
+
card showed into the Dashboards configuration file (`/opt/homebrew/etc/opensearch-dashboards/opensearch_dashboards.yml`
|
|
255
|
+
on a Mac), start it (`brew services start opensearch-dashboards`) and open <http://localhost:5601>:
|
|
256
|
+
|
|
257
|
+
```yaml
|
|
258
|
+
opensearch.hosts: [ "https://search.example.org" ]
|
|
259
|
+
opensearch.username: "frugaliot_dev"
|
|
260
|
+
opensearch.password: "..."
|
|
261
|
+
opensearchDashboards.index: ".kibana_dev"
|
|
262
|
+
opensearch.requestHeadersAllowlist: [ authorization ]
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
The `opensearchDashboards.index` line is what gives the organization its own place to keep saved
|
|
266
|
+
dashboards; without it Dashboards uses `.kibana`, which this login cannot touch. Then, in Dashboards:
|
|
267
|
+
**Dashboards Management → Index patterns → Create**, enter `frugaliot-*`, and choose `@timestamp` as the
|
|
268
|
+
time field.
|
|
269
|
+
|
|
270
|
+
You can use the same login in **Grafana** instead, with its OpenSearch data source and the same address.
|
|
271
|
+
|
|
272
|
+
---
|
|
273
|
+
|
|
274
|
+
## 6. (b) OpenSearch on your own machine
|
|
275
|
+
|
|
276
|
+
If you already run OpenSearch somewhere (or want to), the server can send readings to it. Nothing needs
|
|
277
|
+
installing on the Frugal IoT server. In the dashboard, the **Forwarders** card, add **OpenSearch on
|
|
278
|
+
another server**, with:
|
|
279
|
+
|
|
280
|
+
- its URL - **https**, reachable from the Frugal IoT server over the internet;
|
|
281
|
+
- a username and password, or an API key, for an account that can write indices (`frugaliot-*`) and
|
|
282
|
+
create the index template;
|
|
283
|
+
- optionally *Only these topics*.
|
|
284
|
+
|
|
285
|
+
What it must be: on a **public address**, with a **certificate that can be checked** (a real one, not a
|
|
286
|
+
self-signed demo certificate). The server refuses an address on its own machine or on a private network
|
|
287
|
+
(`localhost`, `192.168.x.x`, `10.x.x.x`) and cannot be told to skip the certificate check. So a computer
|
|
288
|
+
at home behind a router will not work unless you make it reachable (a forwarded port, a tunnel, or put
|
|
289
|
+
OpenSearch on a small cloud machine). The forwarder only sends **new** readings; to load history, the
|
|
290
|
+
operator runs the backfill from section 2 (it reads the same forwarder).
|
|
291
|
+
|
|
292
|
+
---
|
|
293
|
+
|
|
294
|
+
## Keeping it safe
|
|
295
|
+
|
|
296
|
+
Which login can see what:
|
|
297
|
+
|
|
298
|
+
| | Can read | How it is kept in |
|
|
299
|
+
|---|---|---|
|
|
300
|
+
| Grafana token | what its maker may READ, in one organization | checked by this server on every request |
|
|
301
|
+
| Dashboards login (one per person per organization) | the whole of one organization's readings (`frugaliot-<org>-*`) | OpenSearch's own role; stays until removed |
|
|
302
|
+
| the `admin` login in `config.d/opensearch.yaml` | everything, and can change security | stays on the machine, never sent anywhere |
|
|
303
|
+
|
|
304
|
+
1. **OpenSearch listens on the machine itself only.** Check, on the server:
|
|
305
|
+
```sh
|
|
306
|
+
sudo ss -ltnp | grep -E ':(9200|9300)\b'
|
|
307
|
+
```
|
|
308
|
+
Both should show `127.0.0.1` or `[::1]`, not `0.0.0.0` or `*`. If not, look for a `network.host` line
|
|
309
|
+
in `/etc/opensearch/opensearch.yml` and remove it. Block 9200 and 9300 in your firewall as well.
|
|
310
|
+
2. **Only Apache's HTTPS port faces the internet**, and it refuses `/_plugins/_security/api`. Do not
|
|
311
|
+
expose OpenSearch's own port, and do not run Dashboards on the server.
|
|
312
|
+
3. **The demo security setup is for trying things out.** It ships certificates that are the same on every
|
|
313
|
+
OpenSearch install, and accounts that are not meant for production. The `admin` account is now
|
|
314
|
+
reachable through Apache for anyone who guesses its password, so choose a long one, and consider
|
|
315
|
+
changing OpenSearch's built-in accounts you do not use (Security → Internal users in Dashboards).
|
|
316
|
+
Rate-limiting repeated failures (`fail2ban`) in front of that site is worth doing.
|
|
317
|
+
4. **Passwords in the database.** The Forwarders card's password for *another server* is stored in the
|
|
318
|
+
clear in the server's database (`frugal-iot.db`), like every other credential in this system, and
|
|
319
|
+
never sent back. The `config.d/opensearch.yaml` file holds `admin`'s password: protect it like
|
|
320
|
+
`secrets.yaml`.
|
|
321
|
+
5. **Anything put in front of Grafana for other people needs HTTPS**, and one Grafana organization per
|
|
322
|
+
Frugal IoT organization, each with its own data source and token - everyone in a Grafana organization
|
|
323
|
+
shares its data sources.
|
|
324
|
+
|
|
325
|
+
---
|
|
326
|
+
|
|
327
|
+
## 7. Optional: Grafana on the server too
|
|
328
|
+
|
|
329
|
+
Optional - for charts that need no setup on anyone's computer. Everyone who uses it then needs a login
|
|
330
|
+
to this Grafana, which is one more thing to look after. Because it is on the same machine as the
|
|
331
|
+
server, its data source can reach the server at `http://localhost:8080` and nothing has to be exposed.
|
|
332
|
+
|
|
333
|
+
```sh
|
|
334
|
+
sudo apt-get install -y apt-transport-https wget gnupg
|
|
335
|
+
sudo mkdir -p /etc/apt/keyrings
|
|
336
|
+
sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
|
|
337
|
+
sudo chmod 644 /etc/apt/keyrings/grafana.asc
|
|
338
|
+
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" \
|
|
339
|
+
| sudo tee -a /etc/apt/sources.list.d/grafana.list
|
|
340
|
+
sudo apt-get update
|
|
341
|
+
sudo apt-get install grafana
|
|
342
|
+
sudo systemctl daemon-reload
|
|
343
|
+
sudo systemctl enable grafana-server
|
|
344
|
+
sudo systemctl start grafana-server
|
|
345
|
+
sudo grafana cli plugins install yesoreyeram-infinity-datasource
|
|
346
|
+
sudo systemctl restart grafana-server
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
It listens on port 3000, with Grafana's default login `admin` / `admin`, which it asks you to change.
|
|
350
|
+
Do not expose port 3000 to the internet without HTTPS in front of it. Then continue at
|
|
351
|
+
[GRAFANA.md](GRAFANA.md).
|
|
352
|
+
|
|
353
|
+
---
|
|
354
|
+
|
|
355
|
+
## For developers: trying the OpenSearch forwarder on a Mac
|
|
356
|
+
|
|
357
|
+
Not something users do. To work on the logger's OpenSearch forwarder you want OpenSearch on your own
|
|
358
|
+
machine:
|
|
359
|
+
|
|
360
|
+
```sh
|
|
361
|
+
brew install opensearch opensearch-dashboards
|
|
362
|
+
brew services start opensearch
|
|
363
|
+
curl http://localhost:9200 # a block of JSON with "cluster_name"
|
|
364
|
+
```
|
|
365
|
+
|
|
366
|
+
Homebrew's OpenSearch has **no security plugin**: it listens on `http://localhost:9200` with no login,
|
|
367
|
+
and only your own machine can reach it. Make `config.d/opensearch.yaml` in your server's directory contain
|
|
368
|
+
just:
|
|
369
|
+
|
|
370
|
+
```yaml
|
|
371
|
+
opensearch:
|
|
372
|
+
node: http://localhost:9200
|
|
373
|
+
```
|
|
374
|
+
|
|
375
|
+
and restart the server. The Forwarders card then offers **OpenSearch on this server**. The Dashboards
|
|
376
|
+
access card will say it cannot make logins, since that needs the security plugin; on your own Mac you
|
|
377
|
+
do not need one, because Dashboards reaches `http://localhost:9200` with no login.
|
|
378
|
+
|
|
379
|
+
Dashboards will not start until a folder exists that Homebrew forgot to create:
|
|
380
|
+
|
|
381
|
+
```sh
|
|
382
|
+
mkdir -p /opt/homebrew/var/run
|
|
383
|
+
brew services start opensearch-dashboards
|
|
384
|
+
```
|
|
385
|
+
|
|
386
|
+
(Without it the service shows `error 1` in `brew services list`, and the log
|
|
387
|
+
`/opt/homebrew/var/log/opensearch-dashboards.log` says it cannot open `opensearchDashboards.pid`.)
|
|
388
|
+
It is then at <http://localhost:5601>; create the `frugaliot-*` index pattern as in [section 5](#5-opensearch-dashboards-on-your-own-computer).
|
|
389
|
+
|
|
390
|
+
To stop them: `brew services stop opensearch-dashboards opensearch`.
|
|
391
|
+
|
|
392
|
+
---
|
|
393
|
+
|
|
394
|
+
---
|
|
395
|
+
|
|
396
|
+
## Open questions
|
|
397
|
+
|
|
398
|
+
Please add what you find:
|
|
399
|
+
|
|
400
|
+
- **The Linux steps have not been run**: the exact `apt` repository line, and the Apache site above.
|
|
401
|
+
- **The role in section 4 is untried.** Its actions are the ones OpenSearch's documentation names for
|
|
402
|
+
Dashboards, and may be missing one - symptoms are Dashboards failing to start, or an index pattern that
|
|
403
|
+
cannot be created. If so, OpenSearch's log names the refused action; add it to `roleFor` in
|
|
404
|
+
`lib/opensearch-access.js`.
|
|
405
|
+
- **`opensearchDashboards.index`** is meant to give each organization its own saved-objects index. Check
|
|
406
|
+
that two organizations' Dashboards do not see each other's saved dashboards, and that neither can open
|
|
407
|
+
the other's `.kibana_<org>`.
|
|
408
|
+
- **A write-only role for the forwarder**, in place of `admin`. Not written, because it cannot be tested
|
|
409
|
+
on a Mac. It needs to be able to install the index template too, or `value` loses its decimals (the
|
|
410
|
+
forwarder logs when it cannot).
|
|
411
|
+
- **Limiting a login to one project** (OpenSearch can filter by the `project` field of each reading) is not
|
|
412
|
+
done: a Dashboards login sees the whole organization, so it needs READ for all of it.
|
|
413
|
+
- **Logins outlive a change of permissions.** Removing someone's READ on the dashboard does not remove their
|
|
414
|
+
OpenSearch login; an administrator removes it on the card. Doing it automatically (the server already
|
|
415
|
+
notes when a user's permissions change) is not built.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# This server's own OpenSearch - the one INSTALL-OPENSEARCH.md sets up on the server itself.
|
|
2
|
+
#
|
|
3
|
+
# Leave it all commented out on a server without one. With it filled in, an organization's
|
|
4
|
+
# administrator can choose "OpenSearch on this server" on the dashboard's Forwarders card, and make a
|
|
5
|
+
# login for OpenSearch Dashboards on the Dashboards access card. Neither shows an address or a password:
|
|
6
|
+
# they come from here, and never leave this machine.
|
|
7
|
+
#
|
|
8
|
+
# opensearch:
|
|
9
|
+
# node: http://localhost:9200 # where this server reaches OpenSearch - on this machine
|
|
10
|
+
# username: frugaliot_logger # what the forwarder writes readings as
|
|
11
|
+
# password: change-me
|
|
12
|
+
# # insecure: true # only for OpenSearch's demo certificate, which no one can check
|
|
13
|
+
#
|
|
14
|
+
# # The account that makes and removes the per-organization logins on the Dashboards access card.
|
|
15
|
+
# # It must be allowed to change security settings, which is much more than the logger needs - so it
|
|
16
|
+
# # is separate, and you can leave it out if you would rather make those logins by hand.
|
|
17
|
+
# admin:
|
|
18
|
+
# username: admin
|
|
19
|
+
# password: change-me
|
|
20
|
+
#
|
|
21
|
+
# # The address Dashboards on someone's own computer uses to reach OpenSearch: https, through
|
|
22
|
+
# # Apache (INSTALL-OPENSEARCH.md), never the port 9200 above.
|
|
23
|
+
# public: https://search.example.org
|