keepup-admin 0.1.0__tar.gz → 0.2.0__tar.gz

{keepup_admin-0.1.0/keepup_admin.egg-info → keepup_admin-0.2.0}/PKG-INFO

@@ -1,11 +1,13 @@
 Metadata-Version: 2.4
 Name: keepup-admin
-Version: 0.1.0
+Version: 0.2.0
 Summary: An admin framework over FastAPI: plugins, panel, users, replicas
 Author-email: Fedor Sergeev <fksergeev@gmail.com>
 License-Expression: Apache-2.0
-Project-URL: Homepage, https://github.com/FedorSergeev/keepup-admin
+Project-URL: Homepage, https://fedorsergeev.github.io/keepup-admin/
 Project-URL: Repository, https://github.com/FedorSergeev/keepup-admin
+Project-URL: Changelog, https://github.com/FedorSergeev/keepup-admin/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/FedorSergeev/keepup-admin/issues
 Keywords: fastapi,admin,plugins,framework
 Classifier: Framework :: FastAPI
 Classifier: Programming Language :: Python :: 3.11
@@ -17,25 +19,47 @@ License-File: LICENSE
 License-File: NOTICE
 License-File: THIRD-PARTY.md
 Requires-Dist: fastapi<1,>=0.128
-Requires-Dist: starlette<1,>=0.52
+Requires-Dist: starlette<2,>=1.3.1
 Requires-Dist: pydantic<3,>=2.11
 Requires-Dist: sqlalchemy<3,>=2.0
 Requires-Dist: psycopg2-binary>=2.9
 Requires-Dist: apscheduler<4,>=3.10
 Requires-Dist: bcrypt<5,>=4.0
-Requires-Dist: python-jose<4,>=3.5
+Requires-Dist: pyjwt[crypto]<3,>=2.10
 Requires-Dist: python-multipart>=0.0.20
 Requires-Dist: PyYAML<7,>=6.0
 Requires-Dist: psutil>=7.0
 Requires-Dist: prometheus-client>=0.24
 Requires-Dist: requests>=2.31
 Requires-Dist: httpx>=0.28
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == "test"
+Requires-Dist: pytest-asyncio>=0.24; extra == "test"
+Requires-Dist: pytest-cov>=5; extra == "test"
 Provides-Extra: postgres
 Requires-Dist: asyncpg>=0.31; extra == "postgres"
 Dynamic: license-file
 
 # keepup
 
+[![tests](https://github.com/FedorSergeev/keepup-admin/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/FedorSergeev/keepup-admin/actions/workflows/tests.yml)
+[![security](https://github.com/FedorSergeev/keepup-admin/actions/workflows/security.yml/badge.svg?branch=main)](https://github.com/FedorSergeev/keepup-admin/actions/workflows/security.yml)
+[![PyPI](https://img.shields.io/pypi/v/keepup-admin)](https://pypi.org/project/keepup-admin/)
+[![licence](https://img.shields.io/pypi/l/keepup-admin)](LICENSE)
+
+[Project page](https://fedorsergeev.github.io/keepup-admin/) ·
+[Package](https://pypi.org/project/keepup-admin/) ·
+[Changes](CHANGELOG.md)
+
+Four badges, not six checks. These answer what somebody deciding whether to
+install asks: does it run on my Python, is there a known vulnerability in what
+it pulls in, what will I get, and may I use it. `quality`, `build` and `codeql`
+are housekeeping and live on the Actions page -- six badges in a row stop being
+read at all.
+
+A badge shows; it does not require. What keeps a red check from being merged is
+a branch rule, not a picture -- see `doc/keepup.md`.
+
 An admin framework over FastAPI. You give it settings; it gives you an
 application with a panel, a plugin runtime, users, a database layer, an audit,
 metrics and coordination between replicas already wired.
@@ -63,7 +87,7 @@ so the import name is free.
 | **Panel** | the shell, visual themes, and the catalogue of sections with role grants |
 | **Users** | local accounts, roles, panel sessions, sign-in through an external provider |
 | **Database** | a SQLAlchemy layer with a pool, table declarations for two dialects |
-| **Replicas** | distributed locks, leader election, a message bus, a cluster registry |
+| **Replicas** | distributed locks, a message bus, a cluster registry, performance settings in one place |
 | **Observability** | audit of incoming requests, an event log, metrics, log shipping |
 
 ## What it does not decide for you
@@ -98,13 +122,26 @@ between replicas needs.
 ## Tests
 
 ```bash
-ci/tests/keepup.sh          # the framework's own suite, needs no application
+pip install -e ".[test]"
+pytest tests/
 ```
 
-The suite runs against SQLite with no network and holds a coverage floor. The
-plugin runtime is the least covered part of the framework, and deliberately so
-for now: the test that exercises it hardest stands up a real application's
-plugins and therefore lives with that application. See `doc/keepup.md`.
+**Install first, then test.** Pointing pytest at the checkout does not work on
+its own: `import keepup` has to resolve, and the directory a clone lands in is
+called `keepup-admin`, so nothing is named `keepup` until the package is
+installed. The `test` extra carries what the suite needs beyond the package —
+notably `cryptography`, which the framework itself never imports and the OIDC
+tests sign tokens with.
+
+The suite runs against SQLite with no network, needs no application beside the
+package, and holds a coverage floor declared in `pyproject.toml`. Checks that
+genuinely need a consumer of the framework find one by trait and skip with a
+reason when there is none, so a skip here always says what is missing.
+
+Every one of these runs on push and on a pull request; see `.github/workflows/`.
+`security.yml` also runs weekly, because that check goes red without anybody
+touching the repository — an advisory gets published against a version that was
+fine yesterday.
 
 ## Licence
 

{keepup_admin-0.1.0 → keepup_admin-0.2.0}/README.md

@@ -1,5 +1,23 @@
 # keepup
 
+[![tests](https://github.com/FedorSergeev/keepup-admin/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/FedorSergeev/keepup-admin/actions/workflows/tests.yml)
+[![security](https://github.com/FedorSergeev/keepup-admin/actions/workflows/security.yml/badge.svg?branch=main)](https://github.com/FedorSergeev/keepup-admin/actions/workflows/security.yml)
+[![PyPI](https://img.shields.io/pypi/v/keepup-admin)](https://pypi.org/project/keepup-admin/)
+[![licence](https://img.shields.io/pypi/l/keepup-admin)](LICENSE)
+
+[Project page](https://fedorsergeev.github.io/keepup-admin/) ·
+[Package](https://pypi.org/project/keepup-admin/) ·
+[Changes](CHANGELOG.md)
+
+Four badges, not six checks. These answer what somebody deciding whether to
+install asks: does it run on my Python, is there a known vulnerability in what
+it pulls in, what will I get, and may I use it. `quality`, `build` and `codeql`
+are housekeeping and live on the Actions page -- six badges in a row stop being
+read at all.
+
+A badge shows; it does not require. What keeps a red check from being merged is
+a branch rule, not a picture -- see `doc/keepup.md`.
+
 An admin framework over FastAPI. You give it settings; it gives you an
 application with a panel, a plugin runtime, users, a database layer, an audit,
 metrics and coordination between replicas already wired.
@@ -27,7 +45,7 @@ so the import name is free.
 | **Panel** | the shell, visual themes, and the catalogue of sections with role grants |
 | **Users** | local accounts, roles, panel sessions, sign-in through an external provider |
 | **Database** | a SQLAlchemy layer with a pool, table declarations for two dialects |
-| **Replicas** | distributed locks, leader election, a message bus, a cluster registry |
+| **Replicas** | distributed locks, a message bus, a cluster registry, performance settings in one place |
 | **Observability** | audit of incoming requests, an event log, metrics, log shipping |
 
 ## What it does not decide for you
@@ -62,13 +80,26 @@ between replicas needs.
 ## Tests
 
 ```bash
-ci/tests/keepup.sh          # the framework's own suite, needs no application
+pip install -e ".[test]"
+pytest tests/
 ```
 
-The suite runs against SQLite with no network and holds a coverage floor. The
-plugin runtime is the least covered part of the framework, and deliberately so
-for now: the test that exercises it hardest stands up a real application's
-plugins and therefore lives with that application. See `doc/keepup.md`.
+**Install first, then test.** Pointing pytest at the checkout does not work on
+its own: `import keepup` has to resolve, and the directory a clone lands in is
+called `keepup-admin`, so nothing is named `keepup` until the package is
+installed. The `test` extra carries what the suite needs beyond the package —
+notably `cryptography`, which the framework itself never imports and the OIDC
+tests sign tokens with.
+
+The suite runs against SQLite with no network, needs no application beside the
+package, and holds a coverage floor declared in `pyproject.toml`. Checks that
+genuinely need a consumer of the framework find one by trait and skip with a
+reason when there is none, so a skip here always says what is missing.
+
+Every one of these runs on push and on a pull request; see `.github/workflows/`.
+`security.yml` also runs weekly, because that check goes red without anybody
+touching the repository — an advisory gets published against a version that was
+fine yesterday.
 
 ## Licence
 

{keepup_admin-0.1.0 → keepup_admin-0.2.0}/audit.py

@@ -17,7 +17,6 @@ recorded says so by name -- ``configure(redaction=keep_as_is)``.
 import asyncio
 import json
 import logging
-import os
 import time
 from contextlib import asynccontextmanager
 from datetime import datetime
@@ -26,8 +25,8 @@ from typing import Any, Dict, Optional
 from sqlalchemy import CheckConstraint, Column, DateTime, Index, Integer, Text, UniqueConstraint
 from sqlalchemy.dialects import postgresql
 
-from keepup import tables
-from keepup.db import DatabaseManager, DatabaseManagerV2, db_config
+from keepup import retention, tables
+from keepup.db import DatabaseManagerV2, db_config
 from keepup.instance import get_instance_id
 
 #: What an application may import from this module. Everything else is
@@ -132,6 +131,15 @@ incoming_requests_lock = asyncio.Lock()
 
 BUFFER_FLUSH_INTERVAL = 125
 BUFFER_MAX_SIZE = 100
+#: A request running this long is written without its outcome (keepup-41): the
+#: process that would have finished it has most likely gone.
+IN_FLIGHT_STALE_SECONDS = 3600
+
+
+def _as_datetime(value) -> datetime:
+    if isinstance(value, str):
+        return datetime.fromisoformat(value.replace('Z', ''))
+    return value
 
 
 class IncomingRequestLogger:
@@ -211,104 +219,86 @@ class IncomingRequestLogger:
             return True
 
     @staticmethod
-    async def flush_buffer() -> int:
-        """Write the buffered requests to the database."""
-        from keepup.db import DatabaseManager
-
+    async def flush_buffer(include_in_flight: bool = False, now: Optional[datetime] = None) -> int:
+        """Write the finished requests of the buffer to the database.
+
+        A request still running stays in the buffer (keepup-41): written by a
+        timed flush and dropped from the buffer, it could no longer receive its
+        outcome -- end_request found nothing, logged "not found", and the table
+        kept a request with no status. Two exceptions: a request running longer
+        than IN_FLIGHT_STALE_SECONDS is written without an outcome, since the
+        process that would finish it has most likely gone and the buffer must
+        not grow forever; and at shutdown (``include_in_flight``) everything is
+        written -- nothing will finish afterwards.
+        """
+        now = now or datetime.utcnow()
+        # The buffer lock is taken by the start and the end of every request, so
+        # it is held only to pick the rows and take them out -- never across the
+        # write, which runs in a worker thread and may take as long as the
+        # database does (keepup-44).
         async with incoming_requests_lock:
-            if not incoming_requests_buffer:
-                return 0
-
-            requests_to_insert = list(incoming_requests_buffer.values())
-            inserted_count = 0
-
-            try:
-                values = []
-                for req in requests_to_insert:
-                    values.append((
-                        req['instance_id'],
-                        req['method'],
-                        req['endpoint'],
-                        req['host'],
-                        json.dumps(req['request_data']) if req['request_data'] else None,
-                        req['request_start_at'],
-                        req.get('request_end_at'),
-                        req.get('duration_ms'),
-                        req.get('http_status'),
-                        json.dumps(req.get('response_data')) if req.get('response_data') else None,
-                        req.get('error_message'),
-                        req['created_at']
-                    ))
-
-                if values:
-                    if db_config.is_postgres():
-                        query = '''
-                        INSERT INTO incoming_requests 
-                        (instance_id, method, endpoint, host, request_data, request_start_at, 
-                         request_end_at, duration_ms, http_status, response_data, error_message, created_at)
-                        VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
-                        ON CONFLICT (instance_id, method, endpoint, request_start_at) 
-                        DO NOTHING
-                        '''
-                    else:
-                        query = '''
-                        INSERT OR IGNORE INTO incoming_requests 
-                        (instance_id, method, endpoint, host, request_data, request_start_at, 
-                         request_end_at, duration_ms, http_status, response_data, error_message, created_at)
-                        VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
-                        '''
-
-                    if db_config.is_postgres():
-                        conn = DatabaseManager.get_connection()
-                        cursor = conn.cursor()
-                        try:
-                            cursor.executemany(query, values)
-                            inserted_count = cursor.rowcount
-                            conn.commit()
-                        finally:
-                            cursor.close()
-                            conn.close()
-                    else:
-                        conn = DatabaseManager.get_connection()
-                        cursor = conn.cursor()
-                        try:
-                            cursor.executemany(query, values)
-                            inserted_count = cursor.rowcount
-                            conn.commit()
-                        finally:
-                            cursor.close()
-                            conn.close()
-
-                    if inserted_count > 0:
-                        inserted_ids = []
-                        for req_id, req_data in incoming_requests_buffer.items():
-                            for v in values:
-                                if (req_data['instance_id'] == v[0] and
-                                        req_data['method'] == v[1] and
-                                        req_data['endpoint'] == v[2] and
-                                        req_data['request_start_at'] == v[5]):
-                                    inserted_ids.append(req_id)
-                                    break
-
-                        for req_id in inserted_ids:
-                            incoming_requests_buffer.pop(req_id, None)
-
-                        logger.info(f"Flushed {inserted_count} incoming requests to database")
-
-            except Exception as e:
-                logger.error(f"Error flushing incoming requests buffer: {str(e)}")
-
-            return inserted_count
+            chosen = {
+                request_id: req for request_id, req in incoming_requests_buffer.items()
+                if include_in_flight or 'request_end_at' in req
+                or (now - _as_datetime(req['request_start_at'])).total_seconds()
+                >= IN_FLIGHT_STALE_SECONDS
+            }
+            for request_id in chosen:
+                incoming_requests_buffer.pop(request_id, None)
+        if not chosen:
+            return 0
+
+        rows = [{
+            'instance_id': req['instance_id'],
+            'method': req['method'],
+            'endpoint': req['endpoint'],
+            'host': req['host'],
+            'request_data': json.dumps(req['request_data']) if req['request_data'] else None,
+            'request_start_at': req['request_start_at'],
+            'request_end_at': req.get('request_end_at'),
+            'duration_ms': req.get('duration_ms'),
+            'http_status': req.get('http_status'),
+            'response_data': (json.dumps(req.get('response_data'))
+                              if req.get('response_data') else None),
+            'error_message': req.get('error_message'),
+            'created_at': req['created_at'],
+        } for req in chosen.values()]
+        columns = ("instance_id, method, endpoint, host, request_data, request_start_at, "
+                   "request_end_at, duration_ms, http_status, response_data, error_message, "
+                   "created_at")
+        placeholders = ", ".join(":" + c.strip() for c in columns.split(","))
+        if db_config.is_postgres():
+            query = (f"INSERT INTO incoming_requests ({columns}) VALUES ({placeholders}) "
+                     f"ON CONFLICT (instance_id, method, endpoint, request_start_at) DO NOTHING")
+        else:
+            query = f"INSERT OR IGNORE INTO incoming_requests ({columns}) VALUES ({placeholders})"
+
+        try:
+            inserted_count = await DatabaseManagerV2.execute_many_async(query, rows)
+        except Exception as e:
+            # Back into the buffer: the next flush tries again. A request that
+            # somehow reappeared meanwhile keeps its newer entry.
+            logger.error(f"Error flushing incoming requests buffer: {str(e)}")
+            async with incoming_requests_lock:
+                for request_id, req in chosen.items():
+                    incoming_requests_buffer.setdefault(request_id, req)
+            return 0
+
+        # Every chosen request is gone from the buffer, written or ignored as a
+        # duplicate alike: an ignored one is already in the table.
+        if inserted_count:
+            logger.info(f"Flushed {inserted_count} incoming requests to database")
+        return inserted_count
 
 
 #: What PostgreSQL enforces with named constraints, SQLite held as a unique
 #: index of the same name and no checks at all; each dialect keeps what it had.
 # --- how long a row lives ----------------------------------------------------
 
-#: How many days an audit row is kept. The table used to have no sweep at all,
-#: while metric snapshots and application events both had one: it grew for as
-#: long as the deployment ran, and every row in it was a second place a value
-#: from a query string lived.
+#: How many days an audit row is kept. The table used to have no sweep at all:
+#: it grew for as long as the deployment ran, and every row in it was a second
+#: place a value from a query string lived. The event log is swept the same way
+#: (keepup/retention.py).
 DEFAULT_AUDIT_RETENTION_DAYS = 30
 #: How often the sweep runs.
 AUDIT_SWEEP_INTERVAL_SECONDS = 3600
@@ -325,35 +315,12 @@ AUDIT_RETENTION_LOCK = "incoming_requests_retention"
 
 
 def retention_days():
-    """How many days rows are kept, from the environment or the default.
-
-    Returns:
-        A positive number of days; rubbish in the variable means the default
-        and a line in the log, because a sweep that switched itself off
-        silently would be worse than one that swept too much.
-    """
-    raw = os.getenv("AUDIT_RETENTION_DAYS")
-    if raw in (None, ""):
-        return DEFAULT_AUDIT_RETENTION_DAYS
-    try:
-        value = int(raw)
-    except (TypeError, ValueError):
-        logger.warning("AUDIT_RETENTION_DAYS=%r is not a number, using %s",
-                       raw, DEFAULT_AUDIT_RETENTION_DAYS)
-        return DEFAULT_AUDIT_RETENTION_DAYS
-    if value <= 0:
-        logger.warning("AUDIT_RETENTION_DAYS=%s must be positive, using %s",
-                       value, DEFAULT_AUDIT_RETENTION_DAYS)
-        return DEFAULT_AUDIT_RETENTION_DAYS
-    return value
+    """How many days audit rows are kept (``AUDIT_RETENTION_DAYS``, 30 by default)."""
+    return retention.retention_days("AUDIT_RETENTION_DAYS", DEFAULT_AUDIT_RETENTION_DAYS)
 
 
 def purge_old_requests(days=None, now=None):
-    """Delete audit rows older than the retention period.
-
-    The boundary is computed here rather than in the statement: the interval
-    syntax differs between the engines, and a query written for one does not
-    run on the other.
+    """Delete audit rows older than the retention period, in chunks.
 
     Args:
         days: how many days to keep; read from the environment when omitted.
@@ -362,27 +329,10 @@ def purge_old_requests(days=None, now=None):
     Returns:
         How many rows were deleted.
     """
-    from datetime import timedelta
-
-    keep_days = days if days is not None else retention_days()
-    cutoff = (now or datetime.utcnow()) - timedelta(days=keep_days)
-
-    removed = 0
-    for _ in range(AUDIT_MAX_CHUNKS_PER_PASS):
-        rows = DatabaseManagerV2.execute(
-            "SELECT id FROM incoming_requests WHERE created_at < :cutoff "
-            "ORDER BY created_at ASC LIMIT :limit",
-            {"cutoff": cutoff, "limit": AUDIT_DELETE_CHUNK_ROWS})
-        if not rows:
-            break
-        ids = {f"i{index}": row["id"] for index, row in enumerate(rows)}
-        placeholders = ", ".join(f":{key}" for key in ids)
-        DatabaseManagerV2.execute_commit(
-            f"DELETE FROM incoming_requests WHERE id IN ({placeholders})", ids)
-        removed += len(rows)
-        if len(rows) < AUDIT_DELETE_CHUNK_ROWS:
-            break
-    return removed
+    return retention.purge_older_than(
+        "incoming_requests", "created_at",
+        days if days is not None else retention_days(),
+        AUDIT_DELETE_CHUNK_ROWS, AUDIT_MAX_CHUNKS_PER_PASS, now=now)
 
 
 async def audit_retention_background():

{keepup_admin-0.1.0 → keepup_admin-0.2.0}/auth/config.py

@@ -16,7 +16,7 @@ and silently getting local sign-in is an open door where nobody looked for one.
 import os
 from enum import Enum
 from pathlib import Path
-from typing import Any, Dict, Optional
+from typing import Any, Dict
 
 import yaml
 from pydantic import BaseModel, ConfigDict, Field