@noforeignland/signalk-to-noforeignland 1.0.1-beta.8 → 1.1.0-beta.3

package/CHANGELOG.md

@@ -1,3 +1,7 @@
+1.1.0
+* CHANGE: Refactored Project structure, see PROJECT_STRUCTURE.md
+* CHANGE: Use phrase GNSS instead of GPS - Thanks Piotr
+
 1.0.1
 * CHANGE: Cleanup previous installs and migrate plugin config, removes depricated old plugins.
 
@@ -24,17 +28,17 @@ The "new" one has to be installed once manually, after that everything is back t
 0.1.29-beta.1
 * CHANGE: After talking to Treppo from Sugnal K core team we should use the data path "noforeignland.*", code and docs changed.
 * CHANGE: README.md info added and format optimized
-* CHANGE: GPS source at the end for setPluginStatus - Cerbo has a long source name and truncates it.
+* CHANGE: GNSS source at the end for setPluginStatus - Cerbo has a long source name and truncates it.
 * CHANGE: Use "npm install instead" of "npm ci" to push from Github to npmjs using OIDC authentication
 * CHANGE: Moved navigation.position source from path noforeignland.status to noforeignland.source
 
 0.1.28
-* BUGFIX: No GPS data found - introduced in 0.1.27 - Thanks Piotr
+* BUGFIX: No GNSS data found - introduced in 0.1.27 - Thanks Piotr
 
 0.1.28-beta.2
 * EXPERIMENTAL: Signal K data path to visualize the plugin behaviour or error in Node Red, KIP, etc. Using plugin.signalk-to-noforeignland.* for now. See README.md for details.
 * CHANGE: Using app.getDataDirPath() to store transient data, thanks Jeremy, they are now in "plugin-config-data/signalk-to-noforeignland/nfl-track" and renamed to pending.json1 and sent.json1
-* CHANGE: Optimize GPS detection if multiple navigation.position from different sources exist.
+* CHANGE: Optimize GNSS detection if multiple navigation.position from different sources exist.
 * CHANGE: PluginStatus optimized for limited space available.
 * CHANGE: doc/beta_install.md changed for new file structure.
 
@@ -46,13 +50,13 @@ The "new" one has to be installed once manually, after that everything is back t
 
 0.1.27-beta.1
 * NEW: NPMJS requires new method for publishing. The old tokens will expire Nov 19th, 2025, so moving to OIDC authentication.
-* NEW: Check the GPS status in navigation.position, else retry and throw PluginError on Dashboard
+* NEW: Check the GNSS status in navigation.position, else retry and throw PluginError on Dashboard
 * CHANGE: Keep track data on disk rewritten and migrate old files to new structure, so nfl-track-sent.jsonl becomes a continuous archive of all sent track data over time, when enabled. New Logic:
     * New points accumulate in nfl-track-pending.jsonl
     * Send succeeds → API confirms receipt
     * If keepFiles=true: The content of pending file is appended to nfl-track-sent.jsonl (line 588)
     * Pending file is deleted
-    * Next GPS points → create a new pending file
+    * Next GNSS points → create a new pending file
     * Next successful send → appends again to the same nfl-track-sent.jsonl
 
 0.1.26

package/PROJECT_STRUCTURE.md

@@ -0,0 +1,357 @@
+# Project Structure
+
+This document describes the architecture and organization of the SignalK to Noforeignland plugin.
+
+## Overview
+
+This plugin follows a modular architecture where each responsibility is encapsulated in a dedicated module. The main orchestrator ([index.js](index.js)) coordinates all modules through a class-based approach.
+
+## Directory Layout
+
+```
+nfl-signalk/
+  index.js                      # Main plugin entry point & orchestrator
+  cleanup-old-plugin.js         # Postinstall script for removing old versions
+  package.json                  # NPM package configuration
+  lib/                          # Core modules directory
+    ConfigManager.js            # Configuration handling & migration
+    TrackLogger.js              # Position subscription & logging
+    TrackSender.js              # API communication & track sending
+    HealthMonitor.js            # Position data health checks
+    PluginCleanup.js            # Runtime cleanup of old plugins
+    TrackMigration.js           # Track file migration utilities
+    DataPathEmitter.js          # SignalK delta emission
+    DirectoryUtils.js           # File system utilities
+  README.md                     # User documentation
+  CHANGELOG.md                  # Version history
+  PROJECT_STRUCTURE.md          # This file
+```
+
+## Module Responsibilities
+
+### Core Orchestrator
+
+#### [index.js](index.js)
+**Purpose**: Main plugin class that orchestrates all modules and implements SignalK plugin lifecycle.
+
+**Key Responsibilities**:
+- Exports SignalK plugin object with `start()` and `stop()` methods
+- Initializes all module instances in correct order
+- Manages plugin state (options, upSince, cron, lastSuccessfulTransfer)
+- Coordinates between modules during startup sequence
+- Handles plugin status and error reporting to SignalK UI
+- Implements CRON-based interval for data sending
+
+**Key Methods**:
+- `start(options, restartPlugin)`: 13-step startup sequence
+- `stop()`: Cleanup and shutdown
+- `interval()`: CRON job handler for sending tracks
+- `handleSavePoint(lastPosition)`: Callback when track point is saved
+- `handleHealthy()`: Callback when health check passes
+- `setPluginStatus(status)`: Update UI status
+- `setPluginError(error)`: Report error to UI
+
+### Configuration & Migration
+
+#### [lib/ConfigManager.js](lib/ConfigManager.js)
+**Purpose**: Handles plugin configuration, schema definition, and config migration.
+
+**Key Responsibilities**:
+- Defines plugin schema (mandatory/advanced/expert settings groups)
+- Migrates old flat config structure to new nested structure
+- Flattens nested config back to internal format with defaults
+- Validates boat API key
+- Resolves track directory paths (absolute vs relative)
+- Randomizes CRON schedule to avoid server load spikes
+
+**Config Structure**:
+```javascript
+{
+  mandatory: { boatApiKey: string },
+  advanced: { minMove, minSpeed, sendWhileMoving, ping_api_every_24h },
+  expert: { filterSource, trackDir, keepFiles, trackFrequency, apiCron,
+            internetTestTimeout, apiTimeout }
+}
+```
+
+### Position Tracking
+
+#### [lib/TrackLogger.js](lib/TrackLogger.js)
+**Purpose**: Subscribes to position updates and logs them to disk.
+
+**Key Responsibilities**:
+- Subscribes to `navigation.position` and optionally `navigation.speedOverGround`
+- Implements GNSS source auto-selection (stick to one source unless stale)
+- Supports manual source filtering via `filterSource` config
+- Validates positions (rejects coordinates near 0,0)
+- Applies movement filters (minSpeed, minMove, trackFrequency)
+- Implements 24-hour keepalive ping
+- Saves track points to JSONL file (`pending.jsonl`)
+- Calculates distances using equirectangular approximation
+
+**GNSS Source Logic**:
+- If `filterSource` is set: only accept that specific source
+- If not set: auto-select first source, switch only if stale (>5 min)
+- Prevents position jumps when multiple GNSS devices are present
+
+**Track Point Format**:
+```json
+{"lat": 37.8136, "lon": -122.4784, "t": "2025-01-15T10:30:00.000Z"}
+```
+
+#### [lib/TrackSender.js](lib/TrackSender.js)
+**Purpose**: Sends accumulated track data to the Noforeignland API.
+
+**Key Responsibilities**:
+- Reads track points from `pending.jsonl`
+- Tests internet connectivity via DNS lookups (8.8.8.8, 1.1.1.1)
+- Checks if boat is moving (based on last position timestamp)
+- Sends track data to NFL API with retry logic (3 attempts)
+- Handles successful sends (archive to `sent.jsonl` or delete)
+- Validates position coordinates before sending
+
+**API Request Format**:
+```javascript
+POST https://www.noforeignland.com/home/api/v1/boat/tracking/track
+Headers: { 'X-NFL-API-Key': pluginApiKey }
+Body: {
+  timestamp: ms,
+  track: [[ms, lat, lon], [ms, lat, lon], ...],
+  boatApiKey: userBoatApiKey
+}
+```
+
+**Retry Logic**:
+- Attempt 1: base timeout (default 30s)
+- Attempt 2: 2x timeout + 2s delay
+- Attempt 3: 3x timeout + 4s delay
+
+### Health & Monitoring
+
+#### [lib/HealthMonitor.js](lib/HealthMonitor.js)
+**Purpose**: Monitors GNSS position data freshness and reports issues.
+
+**Key Responsibilities**:
+- Initial check: 2 minutes after startup
+- Periodic checks: every 5 minutes
+- Detects stale position data (>5 minutes old)
+- Detects missing position data
+- Provides context-aware error messages (filtered vs auto-selected source)
+- Calls error/healthy callbacks to update plugin status
+
+**Health States**:
+- Healthy: Position received within last 5 minutes
+- Warning: No position for >5 minutes
+- Error: No position ever received (or 2+ min after startup)
+
+### Cleanup & Migration
+
+#### [lib/PluginCleanup.js](lib/PluginCleanup.js)
+**Purpose**: Runtime cleanup of old plugin versions.
+
+**Key Responsibilities**:
+- Detects Victron Cerbo vs standard SignalK paths
+- Migrates config from old `signalk-to-noforeignland` to new scoped name
+- Removes old plugin directories (`signalk-to-noforeignland`, `signalk-to-nfl`)
+- Multiple retry attempts (immediate, 5s, 15s, 30s delays)
+- Falls back to manual instructions if removal fails
+- Returns promise resolving to 'all_removed' or 'partial_removal'
+
+**Old Plugins Removed**:
+- `signalk-to-noforeignland` (unscoped predecessor)
+- `signalk-to-nfl` (deprecated alias)
+
+#### [cleanup-old-plugin.js](cleanup-old-plugin.js)
+**Purpose**: Postinstall script for immediate cleanup during npm install.
+
+**Key Responsibilities**:
+- Runs via `npm postinstall` hook
+- Uses direct `fs.rmSync()` instead of npm uninstall (safer during install)
+- 2-second delay to allow npm to complete current operation
+- Detects Victron Cerbo vs standard SignalK paths
+- Migrates config files
+- Provides fallback manual instructions
+
+**Why Two Cleanup Mechanisms?**
+1. **Postinstall**: Runs during installation, removes old plugins immediately
+2. **Runtime**: Handles cases where postinstall failed (permissions, timing)
+
+#### [lib/TrackMigration.js](lib/TrackMigration.js)
+**Purpose**: Migrates track files from old naming/location schemes.
+
+**Key Responsibilities**:
+- Migrates old file names to new naming scheme:
+  - `nfl-track.jsonl` -> `pending.jsonl`
+  - `nfl-track-pending.jsonl` -> `pending.jsonl`
+  - `nfl-track-sent.jsonl` -> `sent.jsonl`
+- Migrates track files from old plugin directory to new data directory
+- Removes old track directory if empty
+- Non-destructive: only migrates if target doesn't exist
+
+### SignalK Integration
+
+#### [lib/DataPathEmitter.js](lib/DataPathEmitter.js)
+**Purpose**: Creates and updates SignalK data paths for external integrations.
+
+**Key Responsibilities**:
+- Emits SignalK deltas to create data paths
+- Updates status paths on state changes
+- Manages error state independently
+- Provides both ISO8601 and locale-formatted timestamps
+
+**Data Paths Created**:
+```
+noforeignland.savepoint             # ISO8601 timestamp of last save
+noforeignland.savepoint_local       # Locale string of last save
+noforeignland.sent_to_api           # ISO8601 timestamp of last API transfer
+noforeignland.sent_to_api_local     # Locale string of last API transfer
+noforeignland.status                # Status string (save/transfer/source info)
+noforeignland.status_boolean        # 0 = OK, 1 = error
+noforeignland.source                # Active GNSS source name
+```
+
+**Usage**: These paths can be consumed by Node-RED, KIP dashboards, or other SignalK consumers.
+
+#### [lib/DirectoryUtils.js](lib/DirectoryUtils.js)
+**Purpose**: File system utilities with proper error handling.
+
+**Key Responsibilities**:
+- Creates directories recursively
+- Checks read/write permissions
+- Provides context-specific error messages
+- Handles common error codes (EACCES, EPERM, ETIMEDOUT)
+
+## Data Flow
+
+### Startup Sequence
+```
+1. ConfigManager: Migrate old config -> flatten -> validate
+2. DirectoryUtils: Create track directory
+3. PluginCleanup: Remove old plugins (async, non-blocking)
+4. TrackMigration: Migrate old track files
+5. Initialize: TrackLogger, TrackSender, HealthMonitor
+6. ConfigManager: Randomize CRON schedule
+7. TrackLogger: Start position subscription
+8. CRON: Start interval job
+9. HealthMonitor: Start health checks
+```
+
+### Position Logging Flow
+```
+navigation.position delta
+  -> TrackLogger.doOnValue()
+    -> handleSourceSelection() [filter/auto-select]
+    -> isValidPosition() [validate lat/lon]
+    -> shouldLogPosition() [check distance/frequency]
+    -> savePoint() [append to pending.jsonl]
+    -> callback to index.handleSavePoint()
+      -> DataPathEmitter.emitSavepoint()
+      -> setPluginStatus()
+```
+
+### Track Sending Flow (CRON Interval)
+```
+CRON trigger
+  -> index.interval()
+    -> TrackSender.isBoatMoving() [check last position age]
+    -> TrackSender.hasTrackData() [check pending.jsonl exists]
+    -> TrackSender.testInternet() [DNS lookup test]
+    -> TrackSender.sendTrack()
+      -> createTrackFromFile() [read pending.jsonl]
+      -> sendWithRetry() [POST to API, 3 attempts]
+      -> handleSuccessfulSend() [archive or delete]
+    -> DataPathEmitter.emitApiTransfer()
+    -> setPluginStatus()
+```
+
+### Health Monitoring Flow
+```
+HealthMonitor timer (every 5 min)
+  -> performHealthCheck()
+    -> check lastPositionReceived timestamp
+    -> if stale/missing: call onError callback
+      -> index.setPluginError()
+        -> DataPathEmitter.setError()
+        -> app.setPluginError()
+    -> if healthy: call onHealthy callback
+      -> index.handleHealthy()
+        -> setPluginStatus() [clear error if needed]
+```
+
+## File Locations
+
+### Victron Cerbo GX
+- SignalK directory: `/data/conf/signalk`
+- Plugin installed at: `/data/conf/signalk/node_modules/@noforeignland/signalk-to-noforeignland`
+- Config: `/data/conf/signalk/plugin-config-data/@noforeignland-signalk-to-noforeignland.json`
+- Track files: `/data/conf/signalk/plugin-config-data/@noforeignland-signalk-to-noforeignland/nfl-track/`
+
+### Standard SignalK
+- SignalK directory: `~/.signalk`
+- Plugin installed at: `~/.signalk/node_modules/@noforeignland/signalk-to-noforeignland`
+- Config: `~/.signalk/plugin-config-data/@noforeignland-signalk-to-noforeignland.json`
+- Track files: `~/.signalk/plugin-config-data/@noforeignland-signalk-to-noforeignland/nfl-track/`
+
+## Development Notes
+
+### Adding New Features
+1. Create module in `lib/` if it's a distinct responsibility
+2. Instantiate in [index.js](index.js) constructor
+3. Initialize in `start()` method at appropriate step
+4. Clean up in `stop()` method
+5. Update this document
+
+### Configuration Changes
+1. Update schema in [ConfigManager.js](lib/ConfigManager.js) `getSchema()`
+2. Add migration logic in `migrateOldConfig()` if breaking change
+3. Update `flattenConfig()` to include new fields with defaults
+4. Test both old and new config formats
+
+### Track File Format Changes
+1. Update `TrackLogger.savePoint()` for writing
+2. Update `TrackSender.createTrackFromFile()` for reading
+3. Add migration logic in [TrackMigration.js](lib/TrackMigration.js) if needed
+
+### Testing Checklist
+- [ ] Test on Victron Cerbo GX (path detection, limited storage)
+- [ ] Test on standard SignalK installation
+- [ ] Test config migration from old versions
+- [ ] Test with multiple GNSS sources (auto-selection)
+- [ ] Test with filtered source
+- [ ] Test with no internet connection
+- [ ] Test with slow internet (timeout scenarios)
+- [ ] Test cleanup of old plugins
+- [ ] Test track file migration
+
+## Dependencies
+
+```json
+{
+  "cron": "^2.1.0",           // CRON job scheduling
+  "fs-extra": "^10.1.0",      // Enhanced file system operations
+  "node-fetch": "^2.6.7"      // HTTP requests to API
+}
+```
+
+## Version History Notes
+
+- **1.1.0**: Refactored to modular structure
+- **1.0.x**: Scoped package name, cleanup mechanism
+- **0.1.x**: Original flat structure (deprecated)
+
+## Known Technical Debt
+
+1. **Empty process.exit()**: [cleanup-old-plugin.js](cleanup-old-plugin.js) setTimeout may not complete
+2. **Error timing**: [PluginCleanup.js](lib/PluginCleanup.js) shows error before retries complete
+3. **No test suite**: Manual testing only, no automated tests
+
+## SignalK Plugin Conventions
+
+This plugin follows SignalK plugin conventions:
+- Exports function returning `{id, name, schema, start, stop}`
+- Uses `app.debug()` for logging
+- Uses `app.setPluginStatus()` and `app.setPluginError()` for UI feedback
+- Subscribes via `app.subscriptionmanager.subscribe()`
+- Emits deltas via `app.handleMessage(pluginId, delta)`
+- Config stored in `plugin-config-data/` directory
+- Data stored in plugin's data directory

package/README.md

@@ -2,8 +2,7 @@
 Effortlessly log your boat's movement to **noforeignland.com**
 
 ## Important for 0.1.x users
-Since 1.0.0 this package now released unter npmjs org noforeignland in version 1.0.0 due to deployment changes in npmjs and the need of trustworthy sources. All npmjs keys will be revoked Nov 19th, 2025. 
-Users of the old (depricated) 0.1.x versions need to install this new package once manually using the Signal K Appstore. The Plugin Configuration will be kept from the 0.1.x install. 
+Upgrade to >1.1.0 by installing this plugin from the Appstore again. Your old config will be migrated and old plugins will be automatically removed.
 
 ## Features
 * Automatically log your position to noforeignland.com
@@ -20,7 +19,7 @@ Users of the old (depricated) 0.1.x versions need to install this new package on
 
 ## Requirements
 * An internet connection is required in order to update noforeignland.com
-* A navigation.position data path inside Signal K for self, which is your current GPS position
+* A navigation.position data path inside Signal K for self, which is your current GNSS position
 * A **noforeignland.com** account
 * Your Boat API Key from the **noforeignland.com** website: 
   * Account > Settings > Boat tracking > API Key
@@ -43,4 +42,15 @@ noforeignland.status                       - string            - Status & Error
 noforeignland.status_boolean               - number            - 0 = normal operation, 1 = error
 noforeignland.source - string              - string            - data source of navigation.position
 notifications.noforeignland.status_boolean - json object       - auto created
-```
+```
+
+https://github.com/noforeignland/nfl-signalk/issues
+
+# Virctron Cerbo GX Users
+
+## Limited storage
+Signal K can quickly exhaust the small onboard storage of the device, especially when a lot of logging is enabled and not properly configured. Even concider to not enable "Keep track files on disk" if you are moving a lot.
+
+Vicron Energy addressed this here:
+https://www.victronenergy.com/live/venus-os:large#disk_space_issues_data_partition_full
+

package/doc/beta_install_cerbo.md

@@ -55,11 +55,14 @@ cd /data/conf/signalk
 
 npm i @noforeignland/signalk-to-noforeignland@beta
 
+# reset owner properly, else package belongs to root
+chown -R signalk:signalk /data/conf/signalk/*
+
 ```
 
 2. Restart Server & Check logs
 ```
-Restart Server vom Webgui and check logs in Webgui
+ svc -t /service/signalk-server
 ```
 
 
@@ -70,10 +73,14 @@ cd /data/conf/signalk
 
 # Here use the beta version you want to install
 npm i @noforeignland/signalk-to-noforeignland@1.0.1-beta.5
+
+# reset owner properly, else package belongs to root
+chown -R signalk:signalk /data/conf/signalk/*
+
 ```
 2. Restart Server & Check logs
 ```
-Restart Server vom Webgui and check logs in Webgui
+ svc -t /service/signalk-server
 ```
 
 
@@ -98,9 +105,12 @@ npm pack
 cd /data/conf/signalk
 npm install ~/dev/nfl-signalk-main/<your_npm_pack.tgz>
 
+# reset owner properly, else package belongs to root
+chown -R signalk:signalk /data/conf/signalk/*
+
 ```
 
 3. Restart Server & Check logs
 ```
-Restart Server vom Webgui and check logs in Webgui
+svc -t /service/signalk-server
 ```

package/doc/dev

@@ -1,17 +0,0 @@
-
-INTERNAL NOTES ONLY, IGNORE.
-
-
-cd ~/dirk/nfl-signalk
-
-# Aktuelle Version zu Beta machen
-npm version 1.0.1-beta.1 --no-git-tag-version
-
-# Package bauen
-npm pack
-
-# Mit beta tag publishen (nicht als 'latest')
-npm publish --tag beta
-
-# Falls du noch nicht eingeloggt bist:
-npm login