udata-hydra-csvapi 0.2.0.dev0__tar.gz → 0.2.2__tar.gz

{udata_hydra_csvapi-0.2.0.dev0 → udata_hydra_csvapi-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: udata-hydra-csvapi
-Version: 0.2.0.dev0
+Version: 0.2.2
 Summary: API for CSV converted by udata-hydra
 License: MIT
 Author: data.gouv.fr
@@ -130,7 +130,7 @@ curl http://localhost:8005/api/resources/aaaaaaaa-1111-bbbb-2222-cccccccccccc/da
 }
 ```
 
-This endpoint can be queried with the following operators as query string (replacing `column_name` with the name of an actual column):
+This endpoint can be queried with the following operators as query string (replacing `column_name` with the name of an actual column), if the column type allows it (see the swagger for each column's allowed parameter):
 
 ```
 # sort by column
@@ -160,8 +160,30 @@ column_name__strictly_less=value
 
 # strictly greater
 column_name__strictly_greater=value
+
+# group by values
+column_name__groupby
+
+# count values
+column_name__count
+
+# mean / average
+column_name__avg
+
+# minimum
+column_name__min
+
+# maximum
+column_name__max
+
+# sum
+column_name__sum
 ```
 
+> /!\ WARNING: aggregation requests are only available for resources that are listed in the `ALLOW_AGGREGATION` list of the config file.
+
+> NB : passing an aggregation operator (`count`, `avg`, `min`, `max`, `sum`) returns a column that is named `<column_name>__<operator>` (for instance: `?birth__groupby&score__sum` will return a list of dicts with the keys `birth` and `score__sum`).
+
 For instance:
 ```shell
 curl http://localhost:8005/api/resources/aaaaaaaa-1111-bbbb-2222-cccccccccccc/data/?score__greater=0.9&decompte__exact=13
@@ -203,6 +225,31 @@ returns
 }
 ```
 
+With filters and aggregators (filtering is always done **before** aggregation, no matter the order in the parameters):
+```shell
+curl http://localhost:8005/api/resources/aaaaaaaa-1111-bbbb-2222-cccccccccccc/data/?decompte__groupby&birth__less=1996&score__avg
+```
+i.e. `decompte` and average of `score` for all rows where `birth<="1996"`, grouped by `decompte`, returns
+```json
+{
+    "data": [
+        {
+            "decompte": 55,
+            "score__avg": 0.7123333333333334
+        },
+        {
+            "decompte": 27,
+            "score__avg": 0.6068888888888889
+        },
+        {
+            "decompte": 23,
+            "score__avg": 0.4603333333333334
+        },
+        ...
+    ]
+}
+```
+
 Pagination is made through queries with `page` and `page_size`:
 ```shell
 curl http://localhost:8005/api/resources/aaaaaaaa-1111-bbbb-2222-cccccccccccc/data/?page=2&page_size=30
@@ -227,3 +274,7 @@ You can run the following command to lint and format the code:
 poetry run ruff check --fix && poetry run ruff format
 ```
 
+### Releases
+
+The release process uses [bump'X](https://github.com/datagouv/bumpx).
+

{udata_hydra_csvapi-0.2.0.dev0 → udata_hydra_csvapi-0.2.2}/README.md

@@ -112,7 +112,7 @@ curl http://localhost:8005/api/resources/aaaaaaaa-1111-bbbb-2222-cccccccccccc/da
 }
 ```
 
-This endpoint can be queried with the following operators as query string (replacing `column_name` with the name of an actual column):
+This endpoint can be queried with the following operators as query string (replacing `column_name` with the name of an actual column), if the column type allows it (see the swagger for each column's allowed parameter):
 
 ```
 # sort by column
@@ -142,8 +142,30 @@ column_name__strictly_less=value
 
 # strictly greater
 column_name__strictly_greater=value
+
+# group by values
+column_name__groupby
+
+# count values
+column_name__count
+
+# mean / average
+column_name__avg
+
+# minimum
+column_name__min
+
+# maximum
+column_name__max
+
+# sum
+column_name__sum
 ```
 
+> /!\ WARNING: aggregation requests are only available for resources that are listed in the `ALLOW_AGGREGATION` list of the config file.
+
+> NB : passing an aggregation operator (`count`, `avg`, `min`, `max`, `sum`) returns a column that is named `<column_name>__<operator>` (for instance: `?birth__groupby&score__sum` will return a list of dicts with the keys `birth` and `score__sum`).
+
 For instance:
 ```shell
 curl http://localhost:8005/api/resources/aaaaaaaa-1111-bbbb-2222-cccccccccccc/data/?score__greater=0.9&decompte__exact=13
@@ -185,6 +207,31 @@ returns
 }
 ```
 
+With filters and aggregators (filtering is always done **before** aggregation, no matter the order in the parameters):
+```shell
+curl http://localhost:8005/api/resources/aaaaaaaa-1111-bbbb-2222-cccccccccccc/data/?decompte__groupby&birth__less=1996&score__avg
+```
+i.e. `decompte` and average of `score` for all rows where `birth<="1996"`, grouped by `decompte`, returns
+```json
+{
+    "data": [
+        {
+            "decompte": 55,
+            "score__avg": 0.7123333333333334
+        },
+        {
+            "decompte": 27,
+            "score__avg": 0.6068888888888889
+        },
+        {
+            "decompte": 23,
+            "score__avg": 0.4603333333333334
+        },
+        ...
+    ]
+}
+```
+
 Pagination is made through queries with `page` and `page_size`:
 ```shell
 curl http://localhost:8005/api/resources/aaaaaaaa-1111-bbbb-2222-cccccccccccc/data/?page=2&page_size=30
@@ -208,3 +255,7 @@ You can run the following command to lint and format the code:
 ```shell
 poetry run ruff check --fix && poetry run ruff format
 ```
+
+### Releases
+
+The release process uses [bump'X](https://github.com/datagouv/bumpx).

{udata_hydra_csvapi-0.2.0.dev0 → udata_hydra_csvapi-0.2.2}/api_tabular/app.py

@@ -1,4 +1,5 @@
 import os
+from datetime import datetime, timezone
 
 import aiohttp_cors
 import sentry_sdk
@@ -17,6 +18,7 @@ from api_tabular.utils import (
     build_link_with_page,
     build_sql_query_string,
     build_swagger_file,
+    get_app_version,
     url_for,
 )
 
@@ -93,10 +95,11 @@ async def resource_data(request):
         offset = 0
 
     try:
-        sql_query = build_sql_query_string(query_string, page_size, offset)
-    except ValueError:
-        raise QueryException(400, None, "Invalid query string", "Malformed query")
-
+        sql_query = build_sql_query_string(query_string, resource_id, page_size, offset)
+    except ValueError as e:
+        raise QueryException(400, None, "Invalid query string", f"Malformed query: {e}")
+    except PermissionError as e:
+        raise QueryException(403, None, "Unauthorized parameters", str(e))
     resource = await get_resource(request.app["csession"], resource_id, ["parsing_table"])
     response, total = await get_resource_data(request.app["csession"], resource, sql_query)
 
@@ -121,9 +124,11 @@ async def resource_data_csv(request):
     query_string = request.query_string.split("&") if request.query_string else []
 
     try:
-        sql_query = build_sql_query_string(query_string)
+        sql_query = build_sql_query_string(query_string, resource_id)
     except ValueError:
         raise QueryException(400, None, "Invalid query string", "Malformed query")
+    except PermissionError as e:
+        raise QueryException(403, None, "Unauthorized parameters", str(e))
 
     resource = await get_resource(request.app["csession"], resource_id, ["parsing_table"])
 
@@ -143,12 +148,20 @@ async def resource_data_csv(request):
 
 @routes.get(r"/health/")
 async def get_health(request):
-    return web.HTTPOk()
+    """Return health check status"""
+    start_time = request.app["start_time"]
+    current_time = datetime.now(timezone.utc)
+    uptime_seconds = (current_time - start_time).total_seconds()
+    return web.json_response(
+        {"status": "ok", "version": request.app["app_version"], "uptime_seconds": uptime_seconds}
+    )
 
 
 async def app_factory():
     async def on_startup(app):
         app["csession"] = ClientSession()
+        app["start_time"] = datetime.now(timezone.utc)
+        app["app_version"] = await get_app_version()
 
     async def on_cleanup(app):
         await app["csession"].close()

{udata_hydra_csvapi-0.2.0.dev0 → udata_hydra_csvapi-0.2.2}/api_tabular/config_default.toml

@@ -6,3 +6,4 @@ PAGE_SIZE_DEFAULT = 20
 PAGE_SIZE_MAX = 50
 BATCH_SIZE = 50000
 DOC_PATH = "/api/doc"
+ALLOW_AGGREGATION = []  # list of resource_ids

{udata_hydra_csvapi-0.2.0.dev0 → udata_hydra_csvapi-0.2.2}/api_tabular/metrics.py

@@ -78,9 +78,9 @@ async def metrics_data(request):
     else:
         offset = 0
     try:
-        sql_query = build_sql_query_string(query_string, page_size, offset)
-    except ValueError:
-        raise QueryException(400, None, "Invalid query string", "Malformed query")
+        sql_query = build_sql_query_string(query_string, page_size=page_size, offset=offset)
+    except ValueError as e:
+        raise QueryException(400, None, "Invalid query string", f"Malformed query: {e}")
 
     response, total = await get_object_data(request.app["csession"], model, sql_query)
 
@@ -104,8 +104,8 @@ async def metrics_data_csv(request):
 
     try:
         sql_query = build_sql_query_string(query_string)
-    except ValueError:
-        raise QueryException(400, None, "Invalid query string", "Malformed query")
+    except ValueError as e:
+        raise QueryException(400, None, "Invalid query string", f"Malformed query: {e}")
 
     response_headers = {
         "Content-Disposition": f'attachment; filename="{model}.csv"',

{udata_hydra_csvapi-0.2.0.dev0 → udata_hydra_csvapi-0.2.2}/api_tabular/utils.py

@@ -1,3 +1,7 @@
+from collections import defaultdict
+from typing import Optional
+
+import tomllib
 import yaml
 from aiohttp.web_request import Request
 from aiohttp.web_response import Response
@@ -5,13 +9,37 @@ from aiohttp.web_response import Response
 from api_tabular import config
 
 TYPE_POSSIBILITIES = {
-    "string": ["compare", "contains", "differs", "exact", "in", "sort"],
-    "float": ["compare", "differs", "exact", "in", "sort"],
-    "int": ["compare", "differs", "exact", "in", "sort"],
-    "bool": ["differs", "exact", "in", "sort"],
-    "date": ["compare", "contains", "differs", "exact", "in", "sort"],
-    "datetime": ["compare", "contains", "differs", "exact", "in", "sort"],
-    "json": ["contains", "exact", "in"],
+    "string": ["compare", "contains", "differs", "exact", "in", "sort", "groupby", "count"],
+    "float": [
+        "compare",
+        "differs",
+        "exact",
+        "in",
+        "sort",
+        "groupby",
+        "count",
+        "avg",
+        "max",
+        "min",
+        "sum",
+    ],
+    "int": [
+        "compare",
+        "differs",
+        "exact",
+        "in",
+        "sort",
+        "groupby",
+        "count",
+        "avg",
+        "max",
+        "min",
+        "sum",
+    ],
+    "bool": ["differs", "exact", "in", "sort", "groupby", "count"],
+    "date": ["compare", "contains", "differs", "exact", "in", "sort", "groupby", "count"],
+    "datetime": ["compare", "contains", "differs", "exact", "in", "sort", "groupby", "count"],
+    "json": ["contains", "differs", "exact", "in", "groupby", "count"],
 }
 
 MAP_TYPES = {
@@ -21,50 +49,168 @@ MAP_TYPES = {
     "float": "number",
 }
 
+OPERATORS_DESCRIPTIONS = {
+    "exact": {
+        "name": "{}__exact=value",
+        "description": "Exact match in column: {}",
+    },
+    "differs": {
+        "name": "{}__differs=value",
+        "description": "Differs from in column: {}",
+    },
+    "contains": {
+        "name": "{}__contains=value",
+        "description": "String contains in column: {}",
+    },
+    "in": {
+        "name": "{}__in=value1,value2,...",
+        "description": "Value in list in column: {}",
+    },
+    "groupby": {
+        "name": "{}__groupby",
+        "description": "Performs `group by values` operation in column: {}",
+        "is_aggregator": True,
+    },
+    "count": {
+        "name": "{}__count",
+        "description": "Performs `count values` operation in column: {}",
+        "is_aggregator": True,
+    },
+    "avg": {
+        "name": "{}__avg",
+        "description": "Performs `mean` operation in column: {}",
+        "is_aggregator": True,
+    },
+    "min": {
+        "name": "{}__min",
+        "description": "Performs `minimum` operation in column: {}",
+        "is_aggregator": True,
+    },
+    "max": {
+        "name": "{}__max",
+        "description": "Performs `maximum` operation in column: {}",
+        "is_aggregator": True,
+    },
+    "sum": {
+        "name": "{}__sum",
+        "description": "Performs `sum` operation in column: {}",
+        "is_aggregator": True,
+    },
+}
+
+
+def is_aggregation_allowed(resource_id: str):
+    return resource_id in config.ALLOW_AGGREGATION
+
+
+async def get_app_version() -> str:
+    """Parse pyproject.toml and return the version or an error."""
+    try:
+        with open("pyproject.toml", "rb") as f:
+            pyproject = tomllib.load(f)
+        return pyproject.get("tool", {}).get("poetry", {}).get("version", "unknown")
+    except FileNotFoundError:
+        return "unknown (pyproject.toml not found)"
+    except Exception as e:
+        return f"unknown ({str(e)})"
+
 
-def build_sql_query_string(request_arg: list, page_size: int = None, offset: int = 0) -> str:
+def build_sql_query_string(
+    request_arg: list,
+    resource_id: Optional[str] = None,
+    page_size: int = None,
+    offset: int = 0,
+) -> str:
     sql_query = []
+    aggregators = defaultdict(list)
     sorted = False
     for arg in request_arg:
-        argument, value = arg.split("=")
-        if "__" in argument:
-            *column_split, comparator = argument.split("__")
-            normalized_comparator = comparator.lower()
-            # handling headers with "__" and special characters
-            # we're escaping the " because they are the encapsulators of the label
-            column = '"{}"'.format("__".join(column_split).replace('"', '\\"'))
-
-            if normalized_comparator == "sort":
-                if value == "asc":
-                    sql_query.append(f"order={column}.asc,__id.asc")
-                elif value == "desc":
-                    sql_query.append(f"order={column}.desc,__id.asc")
-                sorted = True
-            elif normalized_comparator == "exact":
-                sql_query.append(f"{column}=eq.{value}")
-            elif normalized_comparator == "differs":
-                sql_query.append(f"{column}=neq.{value}")
-            elif normalized_comparator == "contains":
-                sql_query.append(f"{column}=ilike.*{value}*")
-            elif normalized_comparator == "in":
-                sql_query.append(f"{column}=in.({value})")
-            elif normalized_comparator == "less":
-                sql_query.append(f"{column}=lte.{value}")
-            elif normalized_comparator == "greater":
-                sql_query.append(f"{column}=gte.{value}")
-            elif normalized_comparator == "strictly_less":
-                sql_query.append(f"{column}=lt.{value}")
-            elif normalized_comparator == "strictly_greater":
-                sql_query.append(f"{column}=gt.{value}")
+        _split = arg.split("=")
+        # filters are expected to have the syntax `<column_name>__<operator>=<value>`
+        if len(_split) == 2:
+            _filter, _sorted = add_filter(*_split)
+            if _filter:
+                sorted = sorted or _sorted
+                sql_query.append(_filter)
+        # aggregators are expected to have the syntax `<column_name>__<operator>`
+        elif len(_split) == 1:
+            column, operator = add_aggregator(_split[0])
+            if column:
+                aggregators[operator].append(column)
+        else:
+            raise ValueError(f"argument '{arg}' could not be parsed")
+    if aggregators:
+        if resource_id and not is_aggregation_allowed(resource_id):
+            raise PermissionError(
+                f"Aggregation parameters `{'`, `'.join(aggregators.keys())}` "
+                f"are not allowed for resource '{resource_id}'"
+            )
+        agg_query = "select="
+        for operator in aggregators:
+            if operator == "groupby":
+                agg_query += f"{','.join(aggregators[operator])},"
+            else:
+                for column in aggregators[operator]:
+                    # aggregated columns are named `<column_name>__<operator>`
+                    # we pop the heading and trailing " that were added upstream
+                    # and put them around the new column name
+                    agg_query += f'"{column[1:-1]}__{operator}":{column}.{operator}(),'
+        # we pop the trailing comma (it's always there, by construction)
+        sql_query.append(agg_query[:-1])
     if page_size:
         sql_query.append(f"limit={page_size}")
     if offset >= 1:
         sql_query.append(f"offset={offset}")
-    if not sorted:
+    if not sorted and not aggregators:
         sql_query.append("order=__id.asc")
     return "&".join(sql_query)
 
 
+def get_column_and_operator(argument: str) -> tuple[str, str]:
+    *column_split, comparator = argument.split("__")
+    normalized_comparator = comparator.lower()
+    # handling headers with "__" and special characters
+    # we're escaping the " because they are the encapsulators of the label
+    column = '"{}"'.format("__".join(column_split).replace('"', '\\"'))
+    return column, normalized_comparator
+
+
+def add_filter(argument: str, value: str) -> tuple[Optional[str], bool]:
+    if argument in ["page", "page_size"]:  # processed differently
+        return None, False
+    if "__" in argument:
+        column, normalized_comparator = get_column_and_operator(argument)
+        if normalized_comparator == "sort":
+            q = f"order={column}.{value}"
+            return q, True
+        elif normalized_comparator == "exact":
+            return f"{column}=eq.{value}", False
+        elif normalized_comparator == "differs":
+            return f"{column}=neq.{value}", False
+        elif normalized_comparator == "contains":
+            return f"{column}=ilike.*{value}*", False
+        elif normalized_comparator == "in":
+            return f"{column}=in.({value})", False
+        elif normalized_comparator == "less":
+            return f"{column}=lte.{value}", False
+        elif normalized_comparator == "greater":
+            return f"{column}=gte.{value}", False
+        elif normalized_comparator == "strictly_less":
+            return f"{column}=lt.{value}", False
+        elif normalized_comparator == "strictly_greater":
+            return f"{column}=gt.{value}", False
+    raise ValueError(f"argument '{argument}={value}' could not be parsed")
+
+
+def add_aggregator(argument: str) -> tuple[str, str]:
+    operator = None
+    if "__" in argument:
+        column, operator = get_column_and_operator(argument)
+    if operator in ["avg", "count", "max", "min", "sum", "groupby"]:
+        return column, operator
+    raise ValueError(f"argument '{argument}' could not be parsed")
+
+
 def process_total(res: Response) -> int:
     # the Content-Range looks like this: '0-49/21777'
     # see https://docs.postgrest.org/en/stable/references/api/pagination_count.html
@@ -73,25 +219,25 @@ def process_total(res: Response) -> int:
     return int(str_total)
 
 
-def external_url(url):
+def external_url(url) -> str:
     return f"{config.SCHEME}://{config.SERVER_NAME}{url}"
 
 
-def build_link_with_page(request: Request, query_string: str, page: int, page_size: int):
+def build_link_with_page(request: Request, query_string: str, page: int, page_size: int) -> str:
     q = [string for string in query_string if not string.startswith("page")]
     q.extend([f"page={page}", f"page_size={page_size}"])
     rebuilt_q = "&".join(q)
     return external_url(f"{request.path}?{rebuilt_q}")
 
 
-def url_for(request: Request, route: str, *args, **kwargs):
+def url_for(request: Request, route: str, *args, **kwargs) -> str:
     router = request.app.router
     if kwargs.pop("_external", None):
         return external_url(router[route].url_for(**kwargs))
     return router[route].url_for(**kwargs)
 
 
-def swagger_parameters(resource_columns):
+def swagger_parameters(resource_columns: dict, resource_id: str) -> list:
     parameters_list = [
         {
             "name": "page",
@@ -112,42 +258,23 @@ def swagger_parameters(resource_columns):
     # see metier_to_python here: https://github.com/datagouv/csv-detective/blob/master/csv_detective/explore_csv.py
     # see cast for db here: https://github.com/datagouv/hydra/blob/main/udata_hydra/analysis/csv.py
     for key, value in resource_columns.items():
-        if "exact" in TYPE_POSSIBILITIES[value["python_type"]]:
-            parameters_list.extend(
-                [
-                    {
-                        "name": f"{key}__exact=value",
-                        "in": "query",
-                        "description": f"Exact match in column: {key}",
-                        "required": False,
-                        "schema": {"type": "string"},
-                    },
-                ]
-            )
-        if "differs" in TYPE_POSSIBILITIES[value["python_type"]]:
-            parameters_list.extend(
-                [
-                    {
-                        "name": f"{key}__differs=value",
-                        "in": "query",
-                        "description": f"Differs from in column: {key}",
-                        "required": False,
-                        "schema": {"type": "string"},
-                    },
-                ]
-            )
-        if "in" in TYPE_POSSIBILITIES[value["python_type"]]:
-            parameters_list.extend(
-                [
-                    {
-                        "name": f"{key}__in=value1,value2,...",
-                        "in": "query",
-                        "description": f"Value in list in column: {key}",
-                        "required": False,
-                        "schema": {"type": "string"},
-                    },
-                ]
-            )
+        for op in OPERATORS_DESCRIPTIONS:
+            if not is_aggregation_allowed(resource_id) and OPERATORS_DESCRIPTIONS[op].get(
+                "is_aggregator"
+            ):
+                continue
+            if op in TYPE_POSSIBILITIES[value["python_type"]]:
+                parameters_list.extend(
+                    [
+                        {
+                            "name": OPERATORS_DESCRIPTIONS[op]["name"].format(key),
+                            "in": "query",
+                            "description": OPERATORS_DESCRIPTIONS[op]["description"].format(key),
+                            "required": False,
+                            "schema": {"type": "string"},
+                        },
+                    ]
+                )
         if "sort" in TYPE_POSSIBILITIES[value["python_type"]]:
             parameters_list.extend(
                 [
@@ -167,18 +294,6 @@ def swagger_parameters(resource_columns):
                     },
                 ]
             )
-        if "contains" in TYPE_POSSIBILITIES[value["python_type"]]:
-            parameters_list.extend(
-                [
-                    {
-                        "name": f"{key}__contains=value",
-                        "in": "query",
-                        "description": f"String contains in column: {key}",
-                        "required": False,
-                        "schema": {"type": "string"},
-                    },
-                ]
-            )
         if "compare" in TYPE_POSSIBILITIES[value["python_type"]]:
             parameters_list.extend(
                 [
@@ -215,7 +330,7 @@ def swagger_parameters(resource_columns):
     return parameters_list
 
 
-def swagger_component(resource_columns):
+def swagger_component(resource_columns: dict) -> dict:
     resource_prop_dict = {}
     for key, value in resource_columns.items():
         type = MAP_TYPES.get(value["python_type"], "string")
@@ -262,8 +377,8 @@ def swagger_component(resource_columns):
     return component_dict
 
 
-def build_swagger_file(resource_columns, rid):
-    parameters_list = swagger_parameters(resource_columns)
+def build_swagger_file(resource_columns: dict, rid: str) -> str:
+    parameters_list = swagger_parameters(resource_columns, rid)
     component_dict = swagger_component(resource_columns)
     swagger_dict = {
         "openapi": "3.0.3",

{udata_hydra_csvapi-0.2.0.dev0 → udata_hydra_csvapi-0.2.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "udata-hydra-csvapi"
-version = "0.2.0.dev"
+version = "0.2.2"
 description = "API for CSV converted by udata-hydra"
 authors = ["data.gouv.fr <opendatateam@data.gouv.fr>"]
 license = "MIT"
@@ -17,9 +17,10 @@ sentry-sdk = "^2.13.0"
 [tool.poetry.group.dev.dependencies]
 aiohttp-devtools = "^1.0.post0"
 aioresponses = "^0.7.4"
-bumpr = "^0.3.8"
+bumpx = "^0.3.10"
 pytest = "^7.2.1"
 pytest-asyncio = "^0.20.3"
+pytest-mock = "^3.14.0"
 ruff = "^0.6.5"
 
 [tool.ruff]