webgate 0.4.2__tar.gz → 0.5.0__tar.gz

{webgate-0.4.2 → webgate-0.5.0}/CHANGELOG.md

@@ -1,5 +1,41 @@
 # Changelog
 
+## v0.5.0 (2026-04-15)
+
+### Features
+
+- **Multi-instance HA deployment** -- run N webgate workers behind a load balancer, sharing a PostgreSQL database. Only one worker at a time performs server-connectivity probes (leader election via a singleton lease row); all workers serve REST + WS traffic normally.
+- **`compose.ha.yml`** reference deployment: 2 webgate replicas + Postgres + nginx LB with `ip_hash` sticky sessions. Verified end-to-end including automatic failover.
+- **`/api/health`** now reports `instance_id` (per-worker UUID) and `monitor_role` (`leader` / `follower`) so LB health checks and observability can tell replicas apart.
+
+### Configuration
+
+| Variable | Default | Description |
+|---|---|---|
+| `WEBGATE_INSTANCE_ID` | auto (UUID) | Stable identifier for this worker |
+| `WEBGATE_DISABLE_MONITOR` | `false` | Skip leader election entirely (pure follower worker, useful if a separate process owns the monitor) |
+
+### Details
+
+- New `monitor_lease` singleton table (auto-created at startup) holds the current leader's instance id and expiry. 90 s TTL with 30 s heartbeat.
+- On leader loss / expiry, any other worker picks up the probe loop within ~1 check cycle (≤ 90 s).
+- Dialect-agnostic (works on SQLite for single-instance dev, PostgreSQL for real HA).
+
+### Known limitation
+
+- Shared terminal sessions (`/api/ws/terminal/join/{token}`) still require owner and joiner to land on the same worker. Sticky-session routing handles same-browser joins; cross-worker cross-engineer sharing needs Redis pub/sub (planned in v0.5.x).
+
+### Verified
+
+`compose.ha.yml` stack (2 replicas + Postgres + nginx):
+
+- Both replicas show `{"instance_id":"…","monitor_role":"follower"|"leader"}`
+- Exactly one replica holds the lease row in Postgres
+- Servers created on replica A immediately visible from replica B (shared DB)
+- Killed the leader → follower promoted automatically, LB kept serving
+
+---
+
 ## v0.4.2 (2026-04-15)
 
 ### Features

{webgate-0.4.2 → webgate-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: webgate
-Version: 0.4.2
+Version: 0.5.0
 Summary: Self-hosted web application for remote server management via SSH terminal and SFTP file browser
 Project-URL: Homepage, https://github.com/kalexnolasco/webgate
 Project-URL: Documentation, https://kalexnolasco.github.io/webgate/
@@ -552,6 +552,48 @@ labels:
   - "traefik.http.services.webgate.loadbalancer.server.port=8443"
 ```
 
+### Multi-instance HA (N replicas + PostgreSQL)
+
+Run multiple webgate workers behind a load balancer, all sharing one Postgres database. Only one worker probes server connectivity at a time (leader election via a singleton lease row); the rest handle REST/WS traffic normally.
+
+```mermaid
+flowchart LR
+    U["Users"]
+    LB["Load balancer<br/>(sticky: ip_hash)"]
+    W1["webgate #1<br/>(leader)"]
+    W2["webgate #2<br/>(follower)"]
+    W3["webgate #3<br/>(follower)"]
+    PG[(PostgreSQL<br/>shared state)]
+    LEASE[["monitor_lease<br/>(singleton row)"]]
+    U --> LB
+    LB --> W1
+    LB --> W2
+    LB --> W3
+    W1 --> PG
+    W2 --> PG
+    W3 --> PG
+    W1 -- holds --> LEASE
+    W2 -. watches .-> LEASE
+    W3 -. watches .-> LEASE
+    style LB fill:#ffcc02,stroke:#e6a800,color:#333
+    style W1 fill:#5cb85c,stroke:#449d44,color:#fff
+    style W2 fill:#e8f0fe,stroke:#4a90d9
+    style W3 fill:#e8f0fe,stroke:#4a90d9
+    style PG fill:#fff3e0,stroke:#ff9800
+```
+
+Reference stack:
+
+```bash
+export WEBGATE_SECRET_KEY=$(openssl rand -hex 32)
+docker compose -f compose.ha.yml up -d
+curl -s http://localhost:8443/api/health   # shows instance_id + monitor_role
+```
+
+[`compose.ha.yml`](compose.ha.yml) spins up 2 webgate replicas + Postgres + nginx with `ip_hash` sticky sessions. On leader loss, the lease expires within 90 seconds and another replica picks it up automatically.
+
+> **Known limitation**: live shared-terminal sessions still need owner and joiner on the same worker. Sticky sessions mitigate it for same-browser joins; true cross-worker fan-out needs Redis pub/sub — planned for a later v0.5.x.
+
 ### Public read-only demo (Fly.io)
 
 The repo includes [`Dockerfile.demo`](Dockerfile.demo) (webgate + sandboxed sshd via supervisord) and [`fly.toml`](fly.toml). Deploy:

{webgate-0.4.2 → webgate-0.5.0}/README.md

@@ -503,6 +503,48 @@ labels:
   - "traefik.http.services.webgate.loadbalancer.server.port=8443"
 ```
 
+### Multi-instance HA (N replicas + PostgreSQL)
+
+Run multiple webgate workers behind a load balancer, all sharing one Postgres database. Only one worker probes server connectivity at a time (leader election via a singleton lease row); the rest handle REST/WS traffic normally.
+
+```mermaid
+flowchart LR
+    U["Users"]
+    LB["Load balancer<br/>(sticky: ip_hash)"]
+    W1["webgate #1<br/>(leader)"]
+    W2["webgate #2<br/>(follower)"]
+    W3["webgate #3<br/>(follower)"]
+    PG[(PostgreSQL<br/>shared state)]
+    LEASE[["monitor_lease<br/>(singleton row)"]]
+    U --> LB
+    LB --> W1
+    LB --> W2
+    LB --> W3
+    W1 --> PG
+    W2 --> PG
+    W3 --> PG
+    W1 -- holds --> LEASE
+    W2 -. watches .-> LEASE
+    W3 -. watches .-> LEASE
+    style LB fill:#ffcc02,stroke:#e6a800,color:#333
+    style W1 fill:#5cb85c,stroke:#449d44,color:#fff
+    style W2 fill:#e8f0fe,stroke:#4a90d9
+    style W3 fill:#e8f0fe,stroke:#4a90d9
+    style PG fill:#fff3e0,stroke:#ff9800
+```
+
+Reference stack:
+
+```bash
+export WEBGATE_SECRET_KEY=$(openssl rand -hex 32)
+docker compose -f compose.ha.yml up -d
+curl -s http://localhost:8443/api/health   # shows instance_id + monitor_role
+```
+
+[`compose.ha.yml`](compose.ha.yml) spins up 2 webgate replicas + Postgres + nginx with `ip_hash` sticky sessions. On leader loss, the lease expires within 90 seconds and another replica picks it up automatically.
+
+> **Known limitation**: live shared-terminal sessions still need owner and joiner on the same worker. Sticky sessions mitigate it for same-browser joins; true cross-worker fan-out needs Redis pub/sub — planned for a later v0.5.x.
+
 ### Public read-only demo (Fly.io)
 
 The repo includes [`Dockerfile.demo`](Dockerfile.demo) (webgate + sandboxed sshd via supervisord) and [`fly.toml`](fly.toml). Deploy:

webgate-0.5.0/compose.ha.yml

@@ -0,0 +1,62 @@
+# High-availability deployment: 2 webgate workers share a Postgres DB
+# behind a nginx load balancer with sticky sessions.
+#
+# Run:  WEBGATE_SECRET_KEY=$(openssl rand -hex 32) docker compose -f compose.ha.yml up -d
+# Open: http://localhost:8443/  (login: admin / admin)
+#
+# /api/health reports the instance_id + monitor_role so you can see which
+# replica answered. Only one replica at a time becomes the monitor leader
+# via a lease row in the shared Postgres DB.
+#
+# Known limitation: shared terminal sessions (live "Share" URLs) still
+# require the owner and joiner to land on the same worker. Sticky sessions
+# by IP hash handle same-browser joins; cross-engineer sharing between
+# workers needs Redis pub/sub (planned for v0.5.x).
+
+services:
+  postgres:
+    image: postgres:16-alpine
+    container_name: webgate-pg
+    restart: unless-stopped
+    environment:
+      POSTGRES_USER: webgate
+      POSTGRES_PASSWORD: webgate
+      POSTGRES_DB: webgate
+    volumes:
+      - webgate-pg:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U webgate"]
+      interval: 5s
+      timeout: 3s
+      retries: 10
+
+  webgate-1: &webgate
+    image: kalexnolasco/webgate:latest
+    restart: unless-stopped
+    depends_on:
+      postgres:
+        condition: service_healthy
+    environment:
+      WEBGATE_SECRET_KEY: "${WEBGATE_SECRET_KEY:?run: export WEBGATE_SECRET_KEY=$(openssl rand -hex 32)}"
+      WEBGATE_DB_URL: "postgresql+asyncpg://webgate:webgate@postgres:5432/webgate"
+      WEBGATE_FIRST_RUN: "true"
+
+  webgate-2:
+    <<: *webgate
+    # Same image + env, different instance. The monitor leader-election
+    # picks exactly one of these to do the probing.
+
+  nginx:
+    image: nginx:alpine
+    container_name: webgate-lb
+    restart: unless-stopped
+    depends_on:
+      - webgate-1
+      - webgate-2
+    ports:
+      - "8443:80"
+    volumes:
+      - ./deploy/nginx-ha.conf:/etc/nginx/conf.d/default.conf:ro
+
+volumes:
+  webgate-pg:

webgate-0.5.0/deploy/nginx-ha.conf

@@ -0,0 +1,36 @@
+# nginx HA config for compose.ha.yml
+#
+# ip_hash is the minimum needed for WebSocket affinity without a shared
+# session store — all requests from the same source IP always hit the
+# same upstream. Good enough for this reference setup; in a real cluster
+# swap for sticky cookies via nginx Plus or Traefik.
+
+upstream webgate_cluster {
+    ip_hash;
+    server webgate-1:8443 max_fails=3 fail_timeout=10s;
+    server webgate-2:8443 max_fails=3 fail_timeout=10s;
+}
+
+server {
+    listen 80;
+
+    # Terminal WebSocket — needs long timeouts and Upgrade passthrough.
+    location /api/ws/ {
+        proxy_pass http://webgate_cluster;
+        proxy_http_version 1.1;
+        proxy_set_header Upgrade    $http_upgrade;
+        proxy_set_header Connection "upgrade";
+        proxy_set_header Host       $host;
+        proxy_read_timeout 86400s;
+        proxy_send_timeout 86400s;
+    }
+
+    location / {
+        proxy_pass http://webgate_cluster;
+        proxy_set_header Host              $host;
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        client_max_body_size 100m;
+    }
+}

{webgate-0.4.2 → webgate-0.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "webgate"
-version = "0.4.2"
+version = "0.5.0"
 description = "Self-hosted web application for remote server management via SSH terminal and SFTP file browser"
 readme = "README.md"
 license = "MIT"

{webgate-0.4.2 → webgate-0.5.0}/src/webgate/app.py

@@ -59,8 +59,12 @@ def create_app() -> FastAPI:
     )
 
     @app.get("/api/health")
-    async def health() -> dict[str, str]:  # pyright: ignore[reportUnusedFunction]
-        return {"status": "ok"}
+    async def health() -> dict[str, object]:  # pyright: ignore[reportUnusedFunction]
+        return {
+            "status": "ok",
+            "instance_id": server_monitor.instance_id,
+            "monitor_role": "leader" if server_monitor.is_leader else "follower",
+        }
 
     @app.get("/api/config")
     async def public_config() -> dict[str, object]:  # pyright: ignore[reportUnusedFunction]

{webgate-0.4.2 → webgate-0.5.0}/src/webgate/config.py

@@ -27,6 +27,10 @@ class Settings(BaseSettings):
     ldap_group_filter: str = "(member={dn})"  # AD: (member:1.2.840.113556.1.4.1941:={dn})
     ldap_group_map: str = "{}"  # JSON: {"ldap-group-cn": "webgate-group-name"}
     ldap_admin_groups: str = "[]"  # JSON list of LDAP group CNs that grant admin
+
+    # Multi-instance HA
+    instance_id: str = ""  # Unique per worker; auto-generated UUID if empty
+    disable_monitor: bool = False  # Skip leader election; never run server monitor
     log_level: str = "info"
     session_timeout: int = 3600
     max_upload_size: int = 104857600  # 100MB

{webgate-0.4.2 → webgate-0.5.0}/src/webgate/db/engine.py

@@ -33,6 +33,8 @@ async def get_session() -> AsyncGenerator[AsyncSession]:
 # (table, column, sqlite_def, postgres_def)
 _MIGRATIONS: list[tuple[str, str, str, str]] = [
     ("servers", "sftp_read_only", "BOOLEAN DEFAULT 0", "BOOLEAN DEFAULT FALSE"),
+    # Singleton lease table for monitor leader election (id always = 1).
+    # Created lazily by ServerMonitor.start() on first call; rows added there too.
     ("users", "totp_secret", "VARCHAR(255) DEFAULT ''", "VARCHAR(255) DEFAULT ''"),
     ("users", "totp_enabled", "BOOLEAN DEFAULT 0", "BOOLEAN DEFAULT FALSE"),
     (

webgate-0.5.0/src/webgate/servers/monitor.py

@@ -0,0 +1,212 @@
+"""Background server connectivity monitor with leader election.
+
+In a multi-instance deployment, only one worker should actively probe servers.
+We use a tiny singleton row in `monitor_lease` as a lease: workers try to claim
+it; the holder heartbeats every LEASE_RENEW seconds; when the lease expires
+without a renewal, any other worker can take over.
+
+Followers keep the loop alive but skip the actual probing — they still serve
+`/api/servers/status` from the leader's writes via the DB row each status is
+persisted to (we store statuses on `Server` rows for cross-worker reads).
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import logging
+import time
+import uuid
+from dataclasses import dataclass
+from datetime import UTC, datetime, timedelta
+
+import asyncssh
+from sqlalchemy import select, text
+
+from webgate.config import settings
+from webgate.db.engine import async_session_factory, engine
+from webgate.servers.crypto import decrypt_value
+from webgate.servers.models import Server
+
+logger = logging.getLogger(__name__)
+
+CHECK_INTERVAL = 60  # seconds between full check cycles (leader only)
+CONNECT_TIMEOUT = 5  # seconds per SSH connect attempt
+MAX_CONCURRENT = 10  # max parallel checks
+LEASE_TTL = 90  # seconds; longer than CHECK_INTERVAL so a slow cycle doesn't drop the lease
+LEASE_RENEW = 30  # seconds; heartbeat interval
+
+
+@dataclass
+class ServerStatus:
+    online: bool
+    last_checked: datetime
+    latency_ms: float | None = None
+    error: str | None = None
+
+
+class ServerMonitor:
+    """Periodically check SSH connectivity for all servers, with leader election."""
+
+    def __init__(self) -> None:
+        self._statuses: dict[int, ServerStatus] = {}
+        self._task: asyncio.Task[None] | None = None
+        self._instance_id: str = settings.instance_id or str(uuid.uuid4())
+        self._is_leader: bool = False
+
+    @property
+    def instance_id(self) -> str:
+        return self._instance_id
+
+    @property
+    def is_leader(self) -> bool:
+        return self._is_leader
+
+    async def start(self) -> None:
+        if settings.disable_monitor:
+            logger.info("Monitor disabled by WEBGATE_DISABLE_MONITOR (instance %s)", self._instance_id)
+            return
+        await self._ensure_lease_table()
+        self._task = asyncio.create_task(self._loop())
+        logger.info("Server monitor started (instance %s)", self._instance_id)
+
+    async def stop(self) -> None:
+        if self._task is not None:
+            self._task.cancel()
+            with contextlib.suppress(asyncio.CancelledError):
+                await self._task
+            self._task = None
+        if self._is_leader:
+            await self._release_lease()
+        logger.info("Server monitor stopped (instance %s)", self._instance_id)
+
+    def get_status(self, server_id: int) -> ServerStatus | None:
+        return self._statuses.get(server_id)
+
+    def get_all_statuses(self) -> dict[int, ServerStatus]:
+        return dict(self._statuses)
+
+    # --- leader election ---------------------------------------------------
+
+    async def _ensure_lease_table(self) -> None:
+        ddl = (
+            "CREATE TABLE IF NOT EXISTS monitor_lease ("
+            "id INTEGER PRIMARY KEY, "
+            "instance_id VARCHAR(64) NOT NULL, "
+            "expires_at TIMESTAMP NOT NULL"
+            ")"
+        )
+        async with engine.begin() as conn:
+            with contextlib.suppress(Exception):
+                await conn.execute(text(ddl))
+
+    async def _try_claim(self) -> bool:
+        """Atomically claim the singleton lease. Returns True if we are the leader."""
+        # Store as naive UTC so we work with both SQLite's TEXT storage and
+        # Postgres' TIMESTAMP WITHOUT TIME ZONE.
+        now = datetime.now(UTC).replace(tzinfo=None)
+        new_expiry = now + timedelta(seconds=LEASE_TTL)
+        async with engine.begin() as conn:
+            try:
+                row = (await conn.execute(text("SELECT instance_id, expires_at FROM monitor_lease WHERE id = 1"))).fetchone()
+                if row is None:
+                    await conn.execute(
+                        text("INSERT INTO monitor_lease (id, instance_id, expires_at) VALUES (1, :iid, :exp)"),
+                        {"iid": self._instance_id, "exp": new_expiry},
+                    )
+                    return True
+                # Postgres returns datetime, SQLite may return a string.
+                expires_raw = row[1]
+                expires = expires_raw if isinstance(expires_raw, datetime) else datetime.fromisoformat(str(expires_raw))
+                if expires.tzinfo is not None:
+                    expires = expires.astimezone(UTC).replace(tzinfo=None)
+                if row[0] == self._instance_id or expires < now:
+                    res = await conn.execute(
+                        text(
+                            "UPDATE monitor_lease SET instance_id = :iid, expires_at = :exp "
+                            "WHERE id = 1 AND (instance_id = :iid OR expires_at < :now)"
+                        ),
+                        {"iid": self._instance_id, "exp": new_expiry, "now": now},
+                    )
+                    return (res.rowcount or 0) > 0
+                return False
+            except Exception as exc:
+                logger.warning("Lease claim failed: %s", exc)
+                return False
+
+    async def _release_lease(self) -> None:
+        async with engine.begin() as conn:
+            with contextlib.suppress(Exception):
+                await conn.execute(
+                    text("DELETE FROM monitor_lease WHERE id = 1 AND instance_id = :iid"),
+                    {"iid": self._instance_id},
+                )
+
+    # --- main loop ---------------------------------------------------------
+
+    async def _loop(self) -> None:
+        last_renew = 0.0
+        while True:
+            try:
+                if not self._is_leader:
+                    self._is_leader = await self._try_claim()
+                    if self._is_leader:
+                        logger.info("Became monitor leader (instance %s)", self._instance_id)
+                if self._is_leader:
+                    now = time.monotonic()
+                    if now - last_renew > LEASE_RENEW:
+                        renewed = await self._try_claim()
+                        if not renewed:
+                            logger.warning("Lost monitor lease (instance %s) -> stepping down", self._instance_id)
+                            self._is_leader = False
+                            await asyncio.sleep(LEASE_RENEW)
+                            continue
+                        last_renew = now
+                    await self._check_all()
+                else:
+                    # Followers wake up roughly once per check interval to retry leadership.
+                    await asyncio.sleep(min(LEASE_RENEW, CHECK_INTERVAL))
+                    continue
+            except Exception:
+                logger.exception("Error in monitor loop (instance %s)", self._instance_id)
+            await asyncio.sleep(CHECK_INTERVAL)
+
+    async def _check_all(self) -> None:
+        async with async_session_factory() as session:
+            result = await session.execute(select(Server))
+            servers = result.scalars().all()
+
+        semaphore = asyncio.Semaphore(MAX_CONCURRENT)
+
+        async def _check_one(server: Server) -> None:
+            async with semaphore:
+                status = await self._check_server(server)
+                self._statuses[server.id] = status
+
+        await asyncio.gather(*[_check_one(s) for s in servers])
+
+    async def _check_server(self, server: Server) -> ServerStatus:
+        password = decrypt_value(server.encrypted_password) if server.encrypted_password else None
+        private_key_str = (
+            decrypt_value(server.encrypted_private_key) if server.encrypted_private_key else None
+        )
+        kwargs: dict[str, object] = {
+            "host": server.hostname, "port": server.port, "username": server.username,
+            "known_hosts": None,
+        }
+        if private_key_str:
+            kwargs["client_keys"] = [asyncssh.import_private_key(private_key_str)]
+        elif password:
+            kwargs["password"] = password
+        now = datetime.now(UTC)
+        start = time.monotonic()
+        try:
+            conn = await asyncio.wait_for(asyncssh.connect(**kwargs), timeout=CONNECT_TIMEOUT)  # type: ignore[arg-type]
+            elapsed = (time.monotonic() - start) * 1000
+            conn.close()
+            return ServerStatus(online=True, last_checked=now, latency_ms=round(elapsed, 1))
+        except Exception as exc:
+            return ServerStatus(online=False, last_checked=now, error=str(exc))
+
+
+server_monitor = ServerMonitor()

webgate-0.4.2/src/webgate/servers/monitor.py

@@ -1,117 +0,0 @@
-from __future__ import annotations
-
-import asyncio
-import contextlib
-import logging
-import time
-from dataclasses import dataclass
-from datetime import UTC, datetime
-
-import asyncssh
-from sqlalchemy import select
-
-from webgate.db.engine import async_session_factory
-from webgate.servers.crypto import decrypt_value
-from webgate.servers.models import Server
-
-logger = logging.getLogger(__name__)
-
-CHECK_INTERVAL = 60  # seconds between full check cycles
-CONNECT_TIMEOUT = 5  # seconds per SSH connect attempt
-MAX_CONCURRENT = 10  # max parallel checks
-
-
-@dataclass
-class ServerStatus:
-    online: bool
-    last_checked: datetime
-    latency_ms: float | None = None
-    error: str | None = None
-
-
-class ServerMonitor:
-    """Background monitor that periodically checks SSH connectivity for all servers."""
-
-    def __init__(self) -> None:
-        self._statuses: dict[int, ServerStatus] = {}
-        self._task: asyncio.Task[None] | None = None
-
-    async def start(self) -> None:
-        """Launch the background check loop."""
-        self._task = asyncio.create_task(self._check_loop())
-        logger.info("Server monitor started")
-
-    async def stop(self) -> None:
-        """Cancel the background task."""
-        if self._task is not None:
-            self._task.cancel()
-            with contextlib.suppress(asyncio.CancelledError):
-                await self._task
-            self._task = None
-        logger.info("Server monitor stopped")
-
-    def get_status(self, server_id: int) -> ServerStatus | None:
-        return self._statuses.get(server_id)
-
-    def get_all_statuses(self) -> dict[int, ServerStatus]:
-        return dict(self._statuses)
-
-    async def _check_loop(self) -> None:
-        """Run connectivity checks every CHECK_INTERVAL seconds."""
-        while True:
-            try:
-                await self._check_all()
-            except Exception:
-                logger.exception("Error during server status check cycle")
-            await asyncio.sleep(CHECK_INTERVAL)
-
-    async def _check_all(self) -> None:
-        """Load all servers from DB and check each one concurrently."""
-        async with async_session_factory() as session:
-            result = await session.execute(select(Server))
-            servers = result.scalars().all()
-
-        semaphore = asyncio.Semaphore(MAX_CONCURRENT)
-
-        async def _check_one(server: Server) -> None:
-            async with semaphore:
-                status = await self._check_server(server)
-                self._statuses[server.id] = status
-
-        await asyncio.gather(*[_check_one(s) for s in servers])
-
-    async def _check_server(self, server: Server) -> ServerStatus:
-        """Attempt an SSH connection to determine if the server is reachable."""
-        password = decrypt_value(server.encrypted_password) if server.encrypted_password else None
-        private_key_str = (
-            decrypt_value(server.encrypted_private_key)
-            if server.encrypted_private_key
-            else None
-        )
-
-        kwargs: dict[str, object] = {
-            "host": server.hostname,
-            "port": server.port,
-            "username": server.username,
-            "known_hosts": None,
-        }
-        if private_key_str:
-            kwargs["client_keys"] = [asyncssh.import_private_key(private_key_str)]
-        elif password:
-            kwargs["password"] = password
-
-        now = datetime.now(UTC)
-        start = time.monotonic()
-        try:
-            conn = await asyncio.wait_for(
-                asyncssh.connect(**kwargs),  # type: ignore[arg-type]
-                timeout=CONNECT_TIMEOUT,
-            )
-            elapsed = (time.monotonic() - start) * 1000
-            conn.close()
-            return ServerStatus(online=True, last_checked=now, latency_ms=round(elapsed, 1))
-        except Exception as exc:
-            return ServerStatus(online=False, last_checked=now, error=str(exc))
-
-
-server_monitor = ServerMonitor()