udata-hydra 2.0.0.dev2235__tar.gz → 2.0.0.dev2384__tar.gz

{udata_hydra-2.0.0.dev2235 → udata_hydra-2.0.0.dev2384}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: udata-hydra
-Version: 2.0.0.dev2235
+Version: 2.0.0.dev2384
 Summary: Async crawler and parsing service for data.gouv.fr
 License: MIT
 Author: Opendata Team
@@ -105,6 +105,11 @@ Then you can run the tests with `poetry run pytest`.
 
 If you would like to see print statements as they are executed, you can pass the -s flag to pytest (`poetry run pytest -s`). However, note that this can sometimes be difficult to parse.
 
+### Tests coverage
+
+Pytest automatically uses the `coverage` package to generate a coverage report, which is displayed at the end of the test run in the terminal.
+The coverage is configured in the `pypoject.toml` file, in the `[tool.pytest.ini_options]` section.
+You can also override the coverage report configuration when running the tests by passing some flags like `--cov-report` to pytest. See [the pytest-cov documentation](https://pytest-cov.readthedocs.io/en/latest/config.html) for more information.
 
 ## API
 
@@ -115,7 +120,32 @@ poetry install
 poetry run adev runserver udata_hydra/app.py
 ```
 
-### Get latest check
+### Routes/endpoints
+
+The API serves the following endpoints:
+
+*Related to checks:*
+- `GET` on `/api/checks/latest/` to get the latest check for a given URL or resource
+- `GET` on `/api/checks/all/` to get all checks for a given URL or resource
+
+*Related to resources:*
+- `POST` on `/api/resources/` to receive a resource creation event from a source. It will create a new resource in the DB "catalog" table and mark it as priority for next crawling
+- `PUT` on `/api/resources/` to update a resource in the DB "catalog" table
+- `DELETE` on `/api/resources/` to delete a resource in the DB "catalog" table
+
+> :warning: **Warning: the following routes are deprecated and need be removed in the future:**
+> - `POST` on `/api/resource/created` -> use `POST` on `/api/resources/` instead
+> - `POST` on `/api/resource/updated` -> use `PUT` on `/api/resources/` instead
+> - `POST` on `/api/resource/deleted` -> use `DELET`E on `/api/resources/` instead
+
+*Related to some status and health check:*
+- `GET` on `/api/status/crawler/` to get the crawling status
+- `GET` on `/api/status/worker/` to get the worker status
+- `GET` on `/api/stats/` to get the crawling stats
+
+More details about some enpoints are provided below with examples, but not for all of them:
+
+#### Get latest check
 
 Works with `?url={url}` and `?resource_id={resource_id}`.
 
@@ -152,7 +182,7 @@ $ curl -s "http://localhost:8000/api/checks/latest/?url=http://opendata-sig.sain
 }
 ```
 
-### Get all checks for an URL or resource
+#### Get all checks for an URL or resource
 
 Works with `?url={url}` and `?resource_id={resource_id}`.
 
@@ -192,7 +222,7 @@ $ curl -s "http://localhost:8000/api/checks/all/?url=http://www.drees.sante.gouv
 ]
 ```
 
-### Get crawling status
+#### Get crawling status
 
 ```bash
 $ curl -s "http://localhost:8000/api/status/crawler/" | json_pp
@@ -205,7 +235,7 @@ $ curl -s "http://localhost:8000/api/status/crawler/" | json_pp
 }
 ```
 
-### Get worker status
+#### Get worker status
 
 ```bash
 $ curl -s "http://localhost:8000/api/status/worker/" | json_pp
@@ -218,7 +248,7 @@ $ curl -s "http://localhost:8000/api/status/worker/" | json_pp
 }
 ```
 
-### Get crawling stats
+#### Get crawling stats
 
 ```bash
 $ curl -s "http://localhost:8000/api/stats/" | json_pp

{udata_hydra-2.0.0.dev2235 → udata_hydra-2.0.0.dev2384}/README.md

@@ -69,6 +69,11 @@ Then you can run the tests with `poetry run pytest`.
 
 If you would like to see print statements as they are executed, you can pass the -s flag to pytest (`poetry run pytest -s`). However, note that this can sometimes be difficult to parse.
 
+### Tests coverage
+
+Pytest automatically uses the `coverage` package to generate a coverage report, which is displayed at the end of the test run in the terminal.
+The coverage is configured in the `pypoject.toml` file, in the `[tool.pytest.ini_options]` section.
+You can also override the coverage report configuration when running the tests by passing some flags like `--cov-report` to pytest. See [the pytest-cov documentation](https://pytest-cov.readthedocs.io/en/latest/config.html) for more information.
 
 ## API
 
@@ -79,7 +84,32 @@ poetry install
 poetry run adev runserver udata_hydra/app.py
 ```
 
-### Get latest check
+### Routes/endpoints
+
+The API serves the following endpoints:
+
+*Related to checks:*
+- `GET` on `/api/checks/latest/` to get the latest check for a given URL or resource
+- `GET` on `/api/checks/all/` to get all checks for a given URL or resource
+
+*Related to resources:*
+- `POST` on `/api/resources/` to receive a resource creation event from a source. It will create a new resource in the DB "catalog" table and mark it as priority for next crawling
+- `PUT` on `/api/resources/` to update a resource in the DB "catalog" table
+- `DELETE` on `/api/resources/` to delete a resource in the DB "catalog" table
+
+> :warning: **Warning: the following routes are deprecated and need be removed in the future:**
+> - `POST` on `/api/resource/created` -> use `POST` on `/api/resources/` instead
+> - `POST` on `/api/resource/updated` -> use `PUT` on `/api/resources/` instead
+> - `POST` on `/api/resource/deleted` -> use `DELET`E on `/api/resources/` instead
+
+*Related to some status and health check:*
+- `GET` on `/api/status/crawler/` to get the crawling status
+- `GET` on `/api/status/worker/` to get the worker status
+- `GET` on `/api/stats/` to get the crawling stats
+
+More details about some enpoints are provided below with examples, but not for all of them:
+
+#### Get latest check
 
 Works with `?url={url}` and `?resource_id={resource_id}`.
 
@@ -116,7 +146,7 @@ $ curl -s "http://localhost:8000/api/checks/latest/?url=http://opendata-sig.sain
 }
 ```
 
-### Get all checks for an URL or resource
+#### Get all checks for an URL or resource
 
 Works with `?url={url}` and `?resource_id={resource_id}`.
 
@@ -156,7 +186,7 @@ $ curl -s "http://localhost:8000/api/checks/all/?url=http://www.drees.sante.gouv
 ]
 ```
 
-### Get crawling status
+#### Get crawling status
 
 ```bash
 $ curl -s "http://localhost:8000/api/status/crawler/" | json_pp
@@ -169,7 +199,7 @@ $ curl -s "http://localhost:8000/api/status/crawler/" | json_pp
 }
 ```
 
-### Get worker status
+#### Get worker status
 
 ```bash
 $ curl -s "http://localhost:8000/api/status/worker/" | json_pp
@@ -182,7 +212,7 @@ $ curl -s "http://localhost:8000/api/status/worker/" | json_pp
 }
 ```
 
-### Get crawling stats
+#### Get crawling stats
 
 ```bash
 $ curl -s "http://localhost:8000/api/stats/" | json_pp

{udata_hydra-2.0.0.dev2235 → udata_hydra-2.0.0.dev2384}/pyproject.toml

@@ -9,7 +9,7 @@ readme = "README.md"
 
 [tool.poetry]
 name = "udata-hydra"
-version = "2.0.0.dev2235"
+version = "2.0.0.dev2384"
 description = "Async crawler and parsing service for data.gouv.fr"
 authors = ["Opendata Team <opendatateam@data.gouv.fr>"]
 license = "MIT"
@@ -47,6 +47,7 @@ mypy = "^1.11.0"
 nest_asyncio = "^1.5.5"
 pytest = "^7.1.1"
 pytest-asyncio = "^0.18.3"
+pytest-cov = "^5.0.0"
 pytest-mock = "^3.7.0"
 ruff = "^0.5.2"
 
@@ -66,21 +67,22 @@ module = [
 ]
 ignore_missing_imports = true
 
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+markers = [
+    "slow: marks tests as slow (deselect with '-m \"not slow\"')",
+    "catalog_harvested: use catalog_harvested.csv as source",
+]
+addopts = "--cov=udata_hydra/ --cov-report term-missing"
+
 [tool.ruff]
-lint = { select = ["I"] } # also sort imports with an isort rule
+lint = { extend-select = ["I"] } # also sort imports with an isort rule
 line-length = 100
 
 [build-system]
 requires = ["poetry-core>=1.0.0"]
 build-backend = "poetry.core.masonry.api"
 
-[tool.pytest.ini_options]
-asyncio_mode = "strict"
-markers = [
-    "slow: marks tests as slow (deselect with '-m \"not slow\"')",
-    "catalog_harvested: use catalog_harvested.csv as source",
-]
-
 [tool.poetry.scripts]
 udata-hydra = "udata_hydra.cli:run"
 udata-hydra-crawl = "udata_hydra.crawl:run"

udata_hydra-2.0.0.dev2384/udata_hydra/app.py

@@ -0,0 +1,29 @@
+import os
+
+from aiohttp import web
+
+from udata_hydra import context
+from udata_hydra.routes import routes
+
+
+async def app_factory() -> web.Application:
+    async def app_startup(app):
+        app["pool"] = await context.pool()
+
+    async def app_cleanup(app):
+        if "pool" in app:
+            await app["pool"].close()
+
+    app = web.Application()
+    app.add_routes(routes)
+    app.on_startup.append(app_startup)
+    app.on_cleanup.append(app_cleanup)
+    return app
+
+
+def run():
+    web.run_app(app_factory(), path=os.environ.get("HYDRA_APP_SOCKET_PATH"))
+
+
+if __name__ == "__main__":
+    run()

udata_hydra-2.0.0.dev2384/udata_hydra/routes/__init__.py

@@ -0,0 +1,23 @@
+from aiohttp import web
+
+from udata_hydra.routes.checks import get_all_checks, get_latest_check
+from udata_hydra.routes.resources import create_resource, delete_resource, update_resource
+from udata_hydra.routes.status import get_crawler_status, get_stats, get_worker_status
+
+# routes = web.RouteTableDef()
+routes: list = [
+    # Routes for checks
+    web.get("/api/checks/latest/", get_latest_check),
+    web.get("/api/checks/all/", get_all_checks),
+    # Routes for resources
+    web.post("/api/resources/", create_resource),
+    web.put("/api/resources/", update_resource),
+    web.delete("/api/resources/", delete_resource),
+    web.post("/api/resource/created/", create_resource),  # TODO: legacy, to remove
+    web.post("/api/resource/updated/", update_resource),  # TODO: legacy, to remove
+    web.post("/api/resource/deleted/", delete_resource),  # TODO: legacy, to remove
+    # Routes for statuses
+    web.get("/api/status/crawler/", get_crawler_status),
+    web.get("/api/status/worker/", get_worker_status),
+    web.get("/api/stats/", get_stats),
+]

udata_hydra-2.0.0.dev2384/udata_hydra/routes/checks.py

@@ -0,0 +1,31 @@
+from aiohttp import web
+
+from udata_hydra.db.check import Check
+from udata_hydra.schemas import CheckSchema
+
+
+def _get_args(request, params=("url", "resource_id")) -> list:
+    """Get GET parameters from request"""
+    data = [request.query.get(param) for param in params]
+    if not any(data):
+        raise web.HTTPBadRequest()
+    return data
+
+
+async def get_latest_check(request: web.Request) -> web.Response:
+    """Get the latest check for a given URL or resource_id"""
+    url, resource_id = _get_args(request)
+    data = await Check.get_latest(url, resource_id)
+    if not data:
+        raise web.HTTPNotFound()
+    if data["deleted"]:
+        raise web.HTTPGone()
+    return web.json_response(CheckSchema().dump(dict(data)))
+
+
+async def get_all_checks(request: web.Request) -> web.Response:
+    url, resource_id = _get_args(request)
+    data = await Check.get_all(url, resource_id)
+    if not data:
+        raise web.HTTPNotFound()
+    return web.json_response([CheckSchema().dump(dict(r)) for r in data])

udata_hydra-2.0.0.dev2384/udata_hydra/routes/resources.py

@@ -0,0 +1,83 @@
+import json
+
+from aiohttp import web
+from marshmallow import ValidationError
+
+from udata_hydra import config
+from udata_hydra.db.resource import Resource
+from udata_hydra.schemas import ResourceQuerySchema
+from udata_hydra.utils.minio import delete_resource_from_minio
+
+
+async def create_resource(request: web.Request) -> web.Response:
+    """Endpoint to receive a resource creation event from a source
+    Will create a new resource in the DB "catalog" table and mark it as priority for next crawling
+    Respond with a 200 status code and a JSON body with a message key set to "created"
+    If error, respond with a 400 status code
+    """
+    try:
+        payload = await request.json()
+        valid_payload: dict = ResourceQuerySchema().load(payload)
+    except ValidationError as err:
+        raise web.HTTPBadRequest(text=json.dumps(err.messages))
+
+    resource = valid_payload["document"]
+    if not resource:
+        raise web.HTTPBadRequest(text="Missing document body")
+
+    dataset_id = valid_payload["dataset_id"]
+    resource_id = valid_payload["resource_id"]
+
+    await Resource.insert(
+        dataset_id=dataset_id,
+        resource_id=resource_id,
+        url=resource["url"],
+        priority=True,
+    )
+
+    return web.json_response({"message": "created"})
+
+
+async def update_resource(request: web.Request) -> web.Response:
+    """Endpoint to receive a resource update event from a source
+    Will update an existing resource in the DB "catalog" table and mark it as priority for next crawling
+    Respond with a 200 status code and a JSON body with a message key set to "updated"
+    If error, respond with a 400 status code
+    """
+    try:
+        payload = await request.json()
+        valid_payload: dict = ResourceQuerySchema().load(payload)
+    except ValidationError as err:
+        raise web.HTTPBadRequest(text=json.dumps(err.messages))
+
+    resource = valid_payload["document"]
+    if not resource:
+        raise web.HTTPBadRequest(text="Missing document body")
+
+    dataset_id = valid_payload["dataset_id"]
+    resource_id = valid_payload["resource_id"]
+
+    await Resource.update_or_insert(dataset_id, resource_id, resource["url"])
+
+    return web.json_response({"message": "updated"})
+
+
+async def delete_resource(request: web.Request) -> web.Response:
+    try:
+        payload = await request.json()
+        valid_payload: dict = ResourceQuerySchema().load(payload)
+    except ValidationError as err:
+        raise web.HTTPBadRequest(text=json.dumps(err.messages))
+
+    dataset_id = valid_payload["dataset_id"]
+    resource_id = valid_payload["resource_id"]
+
+    pool = request.app["pool"]
+    async with pool.acquire() as connection:
+        if config.SAVE_TO_MINIO:
+            delete_resource_from_minio(dataset_id, resource_id)
+        # Mark resource as deleted in catalog table
+        q = f"""UPDATE catalog SET deleted = TRUE WHERE resource_id = '{resource_id}';"""
+        await connection.execute(q)
+
+    return web.json_response({"message": "deleted"})

udata_hydra-2.0.0.dev2384/udata_hydra/routes/status.py

@@ -0,0 +1,125 @@
+from datetime import datetime, timedelta, timezone
+
+from aiohttp import web
+from humanfriendly import parse_timespan
+
+from udata_hydra import config, context
+from udata_hydra.crawl import get_excluded_clause
+from udata_hydra.worker import QUEUES
+
+
+async def get_crawler_status(request: web.Request) -> web.Response:
+    q = f"""
+        SELECT
+            SUM(CASE WHEN last_check IS NULL THEN 1 ELSE 0 END) AS count_left,
+            SUM(CASE WHEN last_check IS NOT NULL THEN 1 ELSE 0 END) AS count_checked
+        FROM catalog
+        WHERE {get_excluded_clause()}
+        AND catalog.deleted = False
+    """
+    stats_catalog = await request.app["pool"].fetchrow(q)
+
+    since = parse_timespan(config.SINCE)
+    since = datetime.now(timezone.utc) - timedelta(seconds=since)
+    q = f"""
+        SELECT
+            SUM(CASE WHEN checks.created_at <= $1 THEN 1 ELSE 0 END) AS count_outdated
+            --, SUM(CASE WHEN checks.created_at > $1 THEN 1 ELSE 0 END) AS count_fresh
+        FROM catalog, checks
+        WHERE {get_excluded_clause()}
+        AND catalog.last_check = checks.id
+        AND catalog.deleted = False
+    """
+    stats_checks = await request.app["pool"].fetchrow(q, since)
+
+    count_left = stats_catalog["count_left"] + (stats_checks["count_outdated"] or 0)
+    # all w/ a check, minus those with an outdated checked
+    count_checked = stats_catalog["count_checked"] - (stats_checks["count_outdated"] or 0)
+    total = stats_catalog["count_left"] + stats_catalog["count_checked"]
+    rate_checked = round(stats_catalog["count_checked"] / total * 100, 1)
+    rate_checked_fresh = round(count_checked / total * 100, 1)
+
+    return web.json_response(
+        {
+            "total": total,
+            "pending_checks": count_left,
+            "fresh_checks": count_checked,
+            "checks_percentage": rate_checked,
+            "fresh_checks_percentage": rate_checked_fresh,
+        }
+    )
+
+
+async def get_worker_status(request: web.Request) -> web.Response:
+    res = {"queued": {q: len(context.queue(q)) for q in QUEUES}}
+    return web.json_response(res)
+
+
+async def get_stats(request: web.Request) -> web.Response:
+    q = f"""
+        SELECT count(*) AS count_checked
+        FROM catalog
+        WHERE {get_excluded_clause()}
+        AND last_check IS NOT NULL
+        AND catalog.deleted = False
+    """
+    stats_catalog = await request.app["pool"].fetchrow(q)
+
+    q = f"""
+        SELECT
+            SUM(CASE WHEN error IS NULL AND timeout = False THEN 1 ELSE 0 END) AS count_ok,
+            SUM(CASE WHEN error IS NOT NULL THEN 1 ELSE 0 END) AS count_error,
+            SUM(CASE WHEN timeout = True THEN 1 ELSE 0 END) AS count_timeout
+        FROM catalog, checks
+        WHERE {get_excluded_clause()}
+        AND catalog.last_check = checks.id
+        AND catalog.deleted = False
+    """
+    stats_status = await request.app["pool"].fetchrow(q)
+
+    def cmp_rate(key):
+        if stats_catalog["count_checked"] == 0:
+            return 0
+        return round(stats_status[key] / stats_catalog["count_checked"] * 100, 1)
+
+    q = f"""
+        SELECT checks.status, count(*) as count FROM checks, catalog
+        WHERE catalog.last_check = checks.id
+        AND checks.status IS NOT NULL
+        AND {get_excluded_clause()}
+        AND last_check IS NOT NULL
+        AND catalog.deleted = False
+        GROUP BY checks.status
+        ORDER BY count DESC;
+    """
+    res = await request.app["pool"].fetch(q)
+    return web.json_response(
+        {
+            "status": sorted(
+                [
+                    {
+                        "label": s,
+                        "count": stats_status[f"count_{s}"] or 0,
+                        "percentage": cmp_rate(f"count_{s}"),
+                    }
+                    for s in ["error", "timeout", "ok"]
+                ],
+                key=lambda x: x["count"],
+                reverse=True,
+            ),
+            "status_codes": [
+                {
+                    "code": r["status"],
+                    "count": r["count"],
+                    "percentage": round(r["count"] / sum(r["count"] for r in res) * 100, 1),
+                }
+                for r in res
+            ],
+        }
+    )
+
+
+async def health(request: web.Request) -> web.Response:
+    test_connection = await request.app["pool"].fetchrow("SELECT 1")
+    assert next(test_connection.values()) == 1
+    return web.HTTPOk()

udata_hydra-2.0.0.dev2235/udata_hydra/app.py

@@ -1,269 +0,0 @@
-import json
-import os
-from datetime import datetime, timedelta, timezone
-
-from aiohttp import web
-from humanfriendly import parse_timespan
-from marshmallow import ValidationError
-
-from udata_hydra import config, context
-from udata_hydra.crawl import get_excluded_clause
-from udata_hydra.db.check import Check
-from udata_hydra.db.resource import Resource
-from udata_hydra.logger import setup_logging
-from udata_hydra.schemas import CheckSchema, ResourceQuerySchema
-from udata_hydra.utils.minio import delete_resource_from_minio
-from udata_hydra.worker import QUEUES
-
-log = setup_logging()
-routes = web.RouteTableDef()
-
-
-def _get_args(request, params=("url", "resource_id")) -> list:
-    """Get GET parameters from request"""
-    data = [request.query.get(param) for param in params]
-    if not any(data):
-        raise web.HTTPBadRequest()
-    return data
-
-
-@routes.post("/api/resource/created/")
-async def resource_created(request: web.Request) -> web.Response:
-    """Endpoint to receive a resource creation event from a source
-    Will create a new resource in the DB "catalog" table and mark it as priority for next crawling
-    Respond with a 200 status code and a JSON body with a message key set to "created"
-    If error, respond with a 400 status code
-    """
-    try:
-        payload = await request.json()
-        valid_payload: dict = ResourceQuerySchema().load(payload)
-    except ValidationError as err:
-        raise web.HTTPBadRequest(text=json.dumps(err.messages))
-
-    resource = valid_payload["document"]
-    if not resource:
-        raise web.HTTPBadRequest(text="Missing document body")
-
-    dataset_id = valid_payload["dataset_id"]
-    resource_id = valid_payload["resource_id"]
-
-    await Resource.insert(
-        dataset_id=dataset_id,
-        resource_id=resource_id,
-        url=resource["url"],
-        priority=True,
-    )
-
-    return web.json_response({"message": "created"})
-
-
-@routes.post("/api/resource/updated/")
-async def resource_updated(request: web.Request) -> web.Response:
-    """Endpoint to receive a resource update event from a source
-    Will update an existing resource in the DB "catalog" table and mark it as priority for next crawling
-    Respond with a 200 status code and a JSON body with a message key set to "updated"
-    If error, respond with a 400 status code
-    """
-    try:
-        payload = await request.json()
-        valid_payload: dict = ResourceQuerySchema().load(payload)
-    except ValidationError as err:
-        raise web.HTTPBadRequest(text=json.dumps(err.messages))
-
-    resource = valid_payload["document"]
-    if not resource:
-        raise web.HTTPBadRequest(text="Missing document body")
-
-    dataset_id = valid_payload["dataset_id"]
-    resource_id = valid_payload["resource_id"]
-
-    await Resource.update_or_insert(dataset_id, resource_id, resource["url"])
-
-    return web.json_response({"message": "updated"})
-
-
-@routes.post("/api/resource/deleted/")
-async def resource_deleted(request: web.Request) -> web.Response:
-    try:
-        payload = await request.json()
-        valid_payload: dict = ResourceQuerySchema().load(payload)
-    except ValidationError as err:
-        raise web.HTTPBadRequest(text=json.dumps(err.messages))
-
-    dataset_id = valid_payload["dataset_id"]
-    resource_id = valid_payload["resource_id"]
-
-    pool = request.app["pool"]
-    async with pool.acquire() as connection:
-        if config.SAVE_TO_MINIO:
-            delete_resource_from_minio(dataset_id, resource_id)
-        # Mark resource as deleted in catalog table
-        q = f"""UPDATE catalog SET deleted = TRUE WHERE resource_id = '{resource_id}';"""
-        await connection.execute(q)
-
-    return web.json_response({"message": "deleted"})
-
-
-@routes.get("/api/checks/latest/")
-async def get_check(request: web.Request) -> web.Response:
-    """Get the latest check for a given URL or resource_id"""
-    url, resource_id = _get_args(request)
-    data = await Check.get_latest(url, resource_id)
-    if not data:
-        raise web.HTTPNotFound()
-    if data["deleted"]:
-        raise web.HTTPGone()
-    return web.json_response(CheckSchema().dump(dict(data)))
-
-
-@routes.get("/api/checks/all/")
-async def get_checks(request: web.Request) -> web.Response:
-    url, resource_id = _get_args(request)
-    data = await Check.get_all(url, resource_id)
-    if not data:
-        raise web.HTTPNotFound()
-    return web.json_response([CheckSchema().dump(dict(r)) for r in data])
-
-
-@routes.get("/api/status/crawler/")
-async def status_crawler(request: web.Request) -> web.Response:
-    q = f"""
-        SELECT
-            SUM(CASE WHEN last_check IS NULL THEN 1 ELSE 0 END) AS count_left,
-            SUM(CASE WHEN last_check IS NOT NULL THEN 1 ELSE 0 END) AS count_checked
-        FROM catalog
-        WHERE {get_excluded_clause()}
-        AND catalog.deleted = False
-    """
-    stats_catalog = await request.app["pool"].fetchrow(q)
-
-    since = parse_timespan(config.SINCE)
-    since = datetime.now(timezone.utc) - timedelta(seconds=since)
-    q = f"""
-        SELECT
-            SUM(CASE WHEN checks.created_at <= $1 THEN 1 ELSE 0 END) AS count_outdated
-            --, SUM(CASE WHEN checks.created_at > $1 THEN 1 ELSE 0 END) AS count_fresh
-        FROM catalog, checks
-        WHERE {get_excluded_clause()}
-        AND catalog.last_check = checks.id
-        AND catalog.deleted = False
-    """
-    stats_checks = await request.app["pool"].fetchrow(q, since)
-
-    count_left = stats_catalog["count_left"] + (stats_checks["count_outdated"] or 0)
-    # all w/ a check, minus those with an outdated checked
-    count_checked = stats_catalog["count_checked"] - (stats_checks["count_outdated"] or 0)
-    total = stats_catalog["count_left"] + stats_catalog["count_checked"]
-    rate_checked = round(stats_catalog["count_checked"] / total * 100, 1)
-    rate_checked_fresh = round(count_checked / total * 100, 1)
-
-    return web.json_response(
-        {
-            "total": total,
-            "pending_checks": count_left,
-            "fresh_checks": count_checked,
-            "checks_percentage": rate_checked,
-            "fresh_checks_percentage": rate_checked_fresh,
-        }
-    )
-
-
-@routes.get("/api/status/worker/")
-async def status_worker(request: web.Request) -> web.Response:
-    res = {"queued": {q: len(context.queue(q)) for q in QUEUES}}
-    return web.json_response(res)
-
-
-@routes.get("/api/stats/")
-async def stats(request: web.Request) -> web.Response:
-    q = f"""
-        SELECT count(*) AS count_checked
-        FROM catalog
-        WHERE {get_excluded_clause()}
-        AND last_check IS NOT NULL
-        AND catalog.deleted = False
-    """
-    stats_catalog = await request.app["pool"].fetchrow(q)
-
-    q = f"""
-        SELECT
-            SUM(CASE WHEN error IS NULL AND timeout = False THEN 1 ELSE 0 END) AS count_ok,
-            SUM(CASE WHEN error IS NOT NULL THEN 1 ELSE 0 END) AS count_error,
-            SUM(CASE WHEN timeout = True THEN 1 ELSE 0 END) AS count_timeout
-        FROM catalog, checks
-        WHERE {get_excluded_clause()}
-        AND catalog.last_check = checks.id
-        AND catalog.deleted = False
-    """
-    stats_status = await request.app["pool"].fetchrow(q)
-
-    def cmp_rate(key):
-        if stats_catalog["count_checked"] == 0:
-            return 0
-        return round(stats_status[key] / stats_catalog["count_checked"] * 100, 1)
-
-    q = f"""
-        SELECT checks.status, count(*) as count FROM checks, catalog
-        WHERE catalog.last_check = checks.id
-        AND checks.status IS NOT NULL
-        AND {get_excluded_clause()}
-        AND last_check IS NOT NULL
-        AND catalog.deleted = False
-        GROUP BY checks.status
-        ORDER BY count DESC;
-    """
-    res = await request.app["pool"].fetch(q)
-    return web.json_response(
-        {
-            "status": sorted(
-                [
-                    {
-                        "label": s,
-                        "count": stats_status[f"count_{s}"] or 0,
-                        "percentage": cmp_rate(f"count_{s}"),
-                    }
-                    for s in ["error", "timeout", "ok"]
-                ],
-                key=lambda x: x["count"],
-                reverse=True,
-            ),
-            "status_codes": [
-                {
-                    "code": r["status"],
-                    "count": r["count"],
-                    "percentage": round(r["count"] / sum(r["count"] for r in res) * 100, 1),
-                }
-                for r in res
-            ],
-        }
-    )
-
-
-@routes.get("/api/health/")
-async def health(request: web.Request) -> web.Response:
-    test_connection = await request.app["pool"].fetchrow("SELECT 1")
-    assert next(test_connection.values()) == 1
-    return web.HTTPOk()
-
-
-async def app_factory() -> web.Application:
-    async def app_startup(app):
-        app["pool"] = await context.pool()
-
-    async def app_cleanup(app):
-        if "pool" in app:
-            await app["pool"].close()
-
-    app = web.Application()
-    app.add_routes(routes)
-    app.on_startup.append(app_startup)
-    app.on_cleanup.append(app_cleanup)
-    return app
-
-
-def run():
-    web.run_app(app_factory(), path=os.environ.get("HYDRA_APP_SOCKET_PATH"))
-
-
-if __name__ == "__main__":
-    run()

udata_hydra-2.0.0.dev2235/udata_hydra/utils/json.py

@@ -1,11 +0,0 @@
-import json
-
-
-def is_json_file(filename: str) -> bool:
-    """Checks if a string is a valid json"""
-    try:
-        with open(filename) as f:
-            json.load(f)
-            return True
-    except ValueError:
-        return False