udata-hydra 2.0.0.dev1648__tar.gz → 2.0.0.dev1729__tar.gz

{udata_hydra-2.0.0.dev1648 → udata_hydra-2.0.0.dev1729}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: udata-hydra
-Version: 2.0.0.dev1648
+Version: 2.0.0.dev1729
 Summary: Async crawler and parsing service for data.gouv.fr
 License: MIT
 Author: Opendata Team
@@ -45,6 +45,17 @@ Since it's called _hydra_, it also has mythical powers embedded:
 - if the remote resource is a CSV, convert it to a PostgreSQL table, ready for APIfication
 - send crawl and analysis info to a udata instance
 
+## Architecture schema
+
+The architecture for the full workflow is the following:
+
+![Full workflow architecture](docs/archi-idd-IDD.drawio.png)
+
+
+The hydra crawler is one of the components of the architecture. It will check if resource is available, analyze the type of file if the resource has been modified, and analyze the CSV content. It will also convert CSV resources to database tables and send the data to a udata instance. 
+
+![Crawler architecture](docs/hydra.drawio.png)
+
 ## CLI
 
 ### Create database structure
@@ -62,7 +73,7 @@ Install udata-hydra dependencies and cli.
 
 `poetry run udata-hydra-crawl`
 
-It will crawl (forever) the catalog according to config set in `config.py`.
+It will crawl (forever) the catalog according to config set in `udata_hydra/config.toml`, with a default config in `udata_hydra/config_default.toml`.
 
 `BATCH_SIZE` URLs are queued at each loop run.
 
@@ -90,7 +101,7 @@ Converted CSV tables will be stored in the database specified via `config.DATABA
 
 ### Run
 
-```
+```bash
 poetry install
 poetry run adev runserver udata_hydra/app.py
 ```
@@ -99,7 +110,7 @@ poetry run adev runserver udata_hydra/app.py
 
 Works with `?url={url}` and `?resource_id={resource_id}`.
 
-```
+```bash
 $ curl -s "http://localhost:8000/api/checks/latest/?url=http://opendata-sig.saintdenis.re/datasets/661e19974bcc48849bbff7c9637c5c28_1.csv" | json_pp
 {
    "status" : 200,
@@ -136,7 +147,7 @@ $ curl -s "http://localhost:8000/api/checks/latest/?url=http://opendata-sig.sain
 
 Works with `?url={url}` and `?resource_id={resource_id}`.
 
-```
+```bash
 $ curl -s "http://localhost:8000/api/checks/all/?url=http://www.drees.sante.gouv.fr/IMG/xls/er864.xls" | json_pp
 [
    {
@@ -174,7 +185,7 @@ $ curl -s "http://localhost:8000/api/checks/all/?url=http://www.drees.sante.gouv
 
 ### Get crawling status
 
-```
+```bash
 $ curl -s "http://localhost:8000/api/status/crawler/" | json_pp
 {
    "fresh_checks_percentage" : 0.4,
@@ -187,7 +198,7 @@ $ curl -s "http://localhost:8000/api/status/crawler/" | json_pp
 
 ### Get worker status
 
-```
+```bash
 $ curl -s "http://localhost:8000/api/status/worker/" | json_pp
 {
    "queued" : {
@@ -200,7 +211,7 @@ $ curl -s "http://localhost:8000/api/status/worker/" | json_pp
 
 ### Get crawling stats
 
-```
+```bash
 $ curl -s "http://localhost:8000/api/stats/" | json_pp
 {
    "status" : [
@@ -313,3 +324,17 @@ Refer to each section to learn how to launch them. The only differences from dev
 - use `HYDRA_SETTINGS` env var to point to your custom `config.toml`
 - use `HYDRA_APP_SOCKET_PATH` to configure where aiohttp should listen to a [reverse proxy connection (eg nginx)](https://docs.aiohttp.org/en/stable/deployment.html#nginx-configuration) and use `udata-hydra-app` to launch the app server
 
+## Contributing
+
+Before contributing to the repository and making any PR, it is necessary to initialize the pre-commit hooks:
+```bash
+pre-commit install
+```
+Once this is done, code formatting and linting, as well as import sorting, will be automatically checked before each commit.
+
+If you cannot use pre-commit, it is necessary to format, lint, and sort imports with [Ruff](https://docs.astral.sh/ruff/) before committing:
+```bash
+ruff check --fix --select I .
+ruff format .
+```
+

{udata_hydra-2.0.0.dev1648 → udata_hydra-2.0.0.dev1729}/README.md

@@ -9,6 +9,17 @@ Since it's called _hydra_, it also has mythical powers embedded:
 - if the remote resource is a CSV, convert it to a PostgreSQL table, ready for APIfication
 - send crawl and analysis info to a udata instance
 
+## Architecture schema
+
+The architecture for the full workflow is the following:
+
+![Full workflow architecture](docs/archi-idd-IDD.drawio.png)
+
+
+The hydra crawler is one of the components of the architecture. It will check if resource is available, analyze the type of file if the resource has been modified, and analyze the CSV content. It will also convert CSV resources to database tables and send the data to a udata instance. 
+
+![Crawler architecture](docs/hydra.drawio.png)
+
 ## CLI
 
 ### Create database structure
@@ -26,7 +37,7 @@ Install udata-hydra dependencies and cli.
 
 `poetry run udata-hydra-crawl`
 
-It will crawl (forever) the catalog according to config set in `config.py`.
+It will crawl (forever) the catalog according to config set in `udata_hydra/config.toml`, with a default config in `udata_hydra/config_default.toml`.
 
 `BATCH_SIZE` URLs are queued at each loop run.
 
@@ -54,7 +65,7 @@ Converted CSV tables will be stored in the database specified via `config.DATABA
 
 ### Run
 
-```
+```bash
 poetry install
 poetry run adev runserver udata_hydra/app.py
 ```
@@ -63,7 +74,7 @@ poetry run adev runserver udata_hydra/app.py
 
 Works with `?url={url}` and `?resource_id={resource_id}`.
 
-```
+```bash
 $ curl -s "http://localhost:8000/api/checks/latest/?url=http://opendata-sig.saintdenis.re/datasets/661e19974bcc48849bbff7c9637c5c28_1.csv" | json_pp
 {
    "status" : 200,
@@ -100,7 +111,7 @@ $ curl -s "http://localhost:8000/api/checks/latest/?url=http://opendata-sig.sain
 
 Works with `?url={url}` and `?resource_id={resource_id}`.
 
-```
+```bash
 $ curl -s "http://localhost:8000/api/checks/all/?url=http://www.drees.sante.gouv.fr/IMG/xls/er864.xls" | json_pp
 [
    {
@@ -138,7 +149,7 @@ $ curl -s "http://localhost:8000/api/checks/all/?url=http://www.drees.sante.gouv
 
 ### Get crawling status
 
-```
+```bash
 $ curl -s "http://localhost:8000/api/status/crawler/" | json_pp
 {
    "fresh_checks_percentage" : 0.4,
@@ -151,7 +162,7 @@ $ curl -s "http://localhost:8000/api/status/crawler/" | json_pp
 
 ### Get worker status
 
-```
+```bash
 $ curl -s "http://localhost:8000/api/status/worker/" | json_pp
 {
    "queued" : {
@@ -164,7 +175,7 @@ $ curl -s "http://localhost:8000/api/status/worker/" | json_pp
 
 ### Get crawling stats
 
-```
+```bash
 $ curl -s "http://localhost:8000/api/stats/" | json_pp
 {
    "status" : [
@@ -276,3 +287,17 @@ For example, to set the log level to `DEBUG` when initializing the database, use
 Refer to each section to learn how to launch them. The only differences from dev to prod are:
 - use `HYDRA_SETTINGS` env var to point to your custom `config.toml`
 - use `HYDRA_APP_SOCKET_PATH` to configure where aiohttp should listen to a [reverse proxy connection (eg nginx)](https://docs.aiohttp.org/en/stable/deployment.html#nginx-configuration) and use `udata-hydra-app` to launch the app server
+
+## Contributing
+
+Before contributing to the repository and making any PR, it is necessary to initialize the pre-commit hooks:
+```bash
+pre-commit install
+```
+Once this is done, code formatting and linting, as well as import sorting, will be automatically checked before each commit.
+
+If you cannot use pre-commit, it is necessary to format, lint, and sort imports with [Ruff](https://docs.astral.sh/ruff/) before committing:
+```bash
+ruff check --fix --select I .
+ruff format .
+```

{udata_hydra-2.0.0.dev1648 → udata_hydra-2.0.0.dev1729}/pyproject.toml

@@ -1,6 +1,15 @@
+[project]
+name = "udata-hydra"
+version = "2.0.0.dev"
+description = "Async crawler and parsing service for data.gouv.fr"
+authors = [{ name = "Opendata Team", email = "opendatateam@data.gouv.fr" }]
+requires-python = ">=3.9,<3.13"
+license = { text = "MIT" }
+readme = "README.md"
+
 [tool.poetry]
 name = "udata-hydra"
-version = "2.0.0.dev1648"
+version = "2.0.0.dev1729"
 description = "Async crawler and parsing service for data.gouv.fr"
 authors = ["Opendata Team <opendatateam@data.gouv.fr>"]
 license = "MIT"
@@ -31,8 +40,6 @@ python-dateutil = "^2.8.2"
 csv-detective = "0.7.1"
 
 [tool.poetry.group.dev.dependencies]
-flake8-quotes = "^3.3.1"
-flake8 = "^4.0.1"
 gunicorn = "^20.1.0"
 aiohttp-devtools = "^1.0.post0"
 aioresponses = "^0.7.3"
@@ -41,7 +48,11 @@ pytest = "^7.1.1"
 pytest-mock = "^3.7.0"
 pytest-asyncio = "^0.18.3"
 bumpr = "^0.3.8"
+ruff = "^0.5.2"
 
+[tool.ruff]
+lint = { select = ["I"] } # also sort imports with an isort rule
+line-length = 100
 
 [build-system]
 requires = ["poetry-core>=1.0.0"]

udata_hydra-2.0.0.dev1729/setup.py

@@ -0,0 +1,59 @@
+# -*- coding: utf-8 -*-
+from setuptools import setup
+
+packages = \
+['udata_hydra',
+ 'udata_hydra.analysis',
+ 'udata_hydra.migrations',
+ 'udata_hydra.utils']
+
+package_data = \
+{'': ['*'], 'udata_hydra.migrations': ['csv/*', 'main/*']}
+
+install_requires = \
+['aiocontextvars>=0.2.2,<0.3.0',
+ 'aiohttp>=3.8.1,<4.0.0',
+ 'asyncpg>=0.27.0,<0.28.0',
+ 'boto3>=1.21.21,<2.0.0',
+ 'coloredlogs>=15.0.1,<16.0.0',
+ 'csv-detective==0.7.1',
+ 'dateparser>=1.1.7,<2.0.0',
+ 'humanfriendly>=10.0,<11.0',
+ 'marshmallow>=3.14.1,<4.0.0',
+ 'minicli>=0.5.0,<0.6.0',
+ 'pandas>=1.3.3,<2.0.0',
+ 'progressist>=0.1.0,<0.2.0',
+ 'python-dateutil>=2.8.2,<3.0.0',
+ 'python-magic>=0.4.25,<0.5.0',
+ 'redis>=4.1.4,<5.0.0',
+ 'rq>=1.11.1,<2.0.0',
+ 'sentry-sdk>=1.11.1,<2.0.0',
+ 'sqlalchemy>=1.4.46,<2.0.0',
+ 'str2bool>=1.1,<2.0',
+ 'str2float>=0.0.9,<0.0.10',
+ 'toml>=0.10.2,<0.11.0']
+
+entry_points = \
+{'console_scripts': ['udata-hydra = udata_hydra.cli:run',
+                     'udata-hydra-app = udata_hydra.app:run',
+                     'udata-hydra-crawl = udata_hydra.crawl:run']}
+
+setup_kwargs = {
+    'name': 'udata-hydra',
+    'version': '2.0.0.dev1729',
+    'description': 'Async crawler and parsing service for data.gouv.fr',
+    'long_description': '# udata-hydra 🦀\n\n`udata-hydra` is an async metadata crawler for [data.gouv.fr](https://www.data.gouv.fr).\n\nURLs are crawled via _aiohttp_, catalog and crawled metadata are stored in a _PostgreSQL_ database.\n\nSince it\'s called _hydra_, it also has mythical powers embedded:\n- analyse remote resource metadata over time to detect changes in the smartest way possible\n- if the remote resource is a CSV, convert it to a PostgreSQL table, ready for APIfication\n- send crawl and analysis info to a udata instance\n\n## Architecture schema\n\nThe architecture for the full workflow is the following:\n\n![Full workflow architecture](docs/archi-idd-IDD.drawio.png)\n\n\nThe hydra crawler is one of the components of the architecture. It will check if resource is available, analyze the type of file if the resource has been modified, and analyze the CSV content. It will also convert CSV resources to database tables and send the data to a udata instance. \n\n![Crawler architecture](docs/hydra.drawio.png)\n\n## CLI\n\n### Create database structure\n\nInstall udata-hydra dependencies and cli.\n`poetry install`\n\n`poetry run udata-hydra migrate`\n\n### Load (UPSERT) latest catalog version from data.gouv.fr\n\n`poetry run udata-hydra load-catalog`\n\n## Crawler\n\n`poetry run udata-hydra-crawl`\n\nIt will crawl (forever) the catalog according to config set in `udata_hydra/config.toml`, with a default config in `udata_hydra/config_default.toml`.\n\n`BATCH_SIZE` URLs are queued at each loop run.\n\nThe crawler will start with URLs never checked and then proceed with URLs crawled before `SINCE` interval. It will then wait until something changes (catalog or time).\n\nThere\'s a by-domain backoff mecanism. The crawler will wait when, for a given domain in a given batch, `BACKOFF_NB_REQ` is exceeded in a period of `BACKOFF_PERIOD` seconds. It will retry until the backoff is lifted.\n\nIf an URL matches one of the `EXCLUDED_PATTERNS`, it will never be checked.\n\n## Worker\n\nA job queuing system is used to process long-running tasks. Launch the worker with the following command:\n\n`poetry run rq worker -c udata_hydra.worker`\n\nMonitor worker status:\n\n`poetry run rq info -c udata_hydra.worker --interval 1`\n\n## CSV conversion to database\n\nConverted CSV tables will be stored in the database specified via `config.DATABASE_URL_CSV`. For tests it\'s same database as for the catalog. Locally, `docker compose` will launch two distinct database containers.\n\n## API\n\n### Run\n\n```bash\npoetry install\npoetry run adev runserver udata_hydra/app.py\n```\n\n### Get latest check\n\nWorks with `?url={url}` and `?resource_id={resource_id}`.\n\n```bash\n$ curl -s "http://localhost:8000/api/checks/latest/?url=http://opendata-sig.saintdenis.re/datasets/661e19974bcc48849bbff7c9637c5c28_1.csv" | json_pp\n{\n   "status" : 200,\n   "catalog_id" : 64148,\n   "deleted" : false,\n   "error" : null,\n   "created_at" : "2021-02-06T12:19:08.203055",\n   "response_time" : 0.830198049545288,\n   "url" : "http://opendata-sig.saintdenis.re/datasets/661e19974bcc48849bbff7c9637c5c28_1.csv",\n   "domain" : "opendata-sig.saintdenis.re",\n   "timeout" : false,\n   "id" : 114750,\n   "dataset_id" : "5c34944606e3e73d4a551889",\n   "resource_id" : "b3678c59-5b35-43ad-9379-fce29e5b56fe",\n   "headers" : {\n      "content-disposition" : "attachment; filename=\\"xn--Dlimitation_des_cantons-bcc.csv\\"",\n      "server" : "openresty",\n      "x-amz-meta-cachetime" : "191",\n      "last-modified" : "Wed, 29 Apr 2020 02:19:04 GMT",\n      "content-encoding" : "gzip",\n      "content-type" : "text/csv",\n      "cache-control" : "must-revalidate",\n      "etag" : "\\"20415964703d9ccc4815d7126aa3a6d8\\"",\n      "content-length" : "207",\n      "date" : "Sat, 06 Feb 2021 12:19:08 GMT",\n      "x-amz-meta-contentlastmodified" : "2018-11-19T09:38:28.490Z",\n      "connection" : "keep-alive",\n      "vary" : "Accept-Encoding"\n   }\n}\n```\n\n### Get all checks for an URL or resource\n\nWorks with `?url={url}` and `?resource_id={resource_id}`.\n\n```bash\n$ curl -s "http://localhost:8000/api/checks/all/?url=http://www.drees.sante.gouv.fr/IMG/xls/er864.xls" | json_pp\n[\n   {\n      "domain" : "www.drees.sante.gouv.fr",\n      "dataset_id" : "53d6eadba3a72954d9dd62f5",\n      "timeout" : false,\n      "deleted" : false,\n      "response_time" : null,\n      "error" : "Cannot connect to host www.drees.sante.gouv.fr:443 ssl:True [SSLCertVerificationError: (1, \\"[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for \'www.drees.sante.gouv.fr\'. (_ssl.c:1122)\\")]",\n      "catalog_id" : 232112,\n      "url" : "http://www.drees.sante.gouv.fr/IMG/xls/er864.xls",\n      "headers" : {},\n      "id" : 165107,\n      "created_at" : "2021-02-06T14:32:47.675854",\n      "resource_id" : "93dfd449-9d26-4bb0-a6a9-ee49b1b8a4d7",\n      "status" : null\n   },\n   {\n      "timeout" : false,\n      "deleted" : false,\n      "response_time" : null,\n      "error" : "Cannot connect to host www.drees.sante.gouv.fr:443 ssl:True [SSLCertVerificationError: (1, \\"[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for \'www.drees.sante.gouv.fr\'. (_ssl.c:1122)\\")]",\n      "domain" : "www.drees.sante.gouv.fr",\n      "dataset_id" : "53d6eadba3a72954d9dd62f5",\n      "created_at" : "2020-12-24T17:06:58.158125",\n      "resource_id" : "93dfd449-9d26-4bb0-a6a9-ee49b1b8a4d7",\n      "status" : null,\n      "catalog_id" : 232112,\n      "url" : "http://www.drees.sante.gouv.fr/IMG/xls/er864.xls",\n      "headers" : {},\n      "id" : 65092\n   }\n]\n```\n\n### Get crawling status\n\n```bash\n$ curl -s "http://localhost:8000/api/status/crawler/" | json_pp\n{\n   "fresh_checks_percentage" : 0.4,\n   "pending_checks" : 142153,\n   "total" : 142687,\n   "fresh_checks" : 534,\n   "checks_percentage" : 0.4\n}\n```\n\n### Get worker status\n\n```bash\n$ curl -s "http://localhost:8000/api/status/worker/" | json_pp\n{\n   "queued" : {\n      "default" : 0,\n      "high" : 825,\n      "low" : 655\n   }\n}\n```\n\n### Get crawling stats\n\n```bash\n$ curl -s "http://localhost:8000/api/stats/" | json_pp\n{\n   "status" : [\n      {\n         "count" : 525,\n         "percentage" : 98.3,\n         "label" : "ok"\n      },\n      {\n         "label" : "error",\n         "percentage" : 1.3,\n         "count" : 7\n      },\n      {\n         "label" : "timeout",\n         "percentage" : 0.4,\n         "count" : 2\n      }\n   ],\n   "status_codes" : [\n      {\n         "code" : 200,\n         "count" : 413,\n         "percentage" : 78.7\n      },\n      {\n         "code" : 501,\n         "percentage" : 12.4,\n         "count" : 65\n      },\n      {\n         "percentage" : 6.1,\n         "count" : 32,\n         "code" : 404\n      },\n      {\n         "code" : 500,\n         "percentage" : 2.7,\n         "count" : 14\n      },\n      {\n         "code" : 502,\n         "count" : 1,\n         "percentage" : 0.2\n      }\n   ]\n}\n```\n\n## Using Webhook integration\n\n** Set the config values**\n\nCreate a `config.toml` where your service and commands are launched, or specify a path to a TOML file via the `HYDRA_SETTINGS` environment variable. `config.toml` or equivalent will override values from `udata_hydra/config_default.toml`, lookup there for values that can/need to be defined.\n\n```toml\nUDATA_URI = "https://dev.local:7000/api/2"\nUDATA_URI_API_KEY = "example.api.key"\nSENTRY_DSN = "https://{my-sentry-dsn}"\n```\n\nThe webhook integration sends HTTP messages to `udata` when resources are analyzed or checked to fill resources extras.\n\nRegarding analysis, there is a phase called "change detection". It will try to guess if a resource has been modified based on different criterions:\n- harvest modified date in catalog\n- content-length and last-modified headers\n- checksum comparison over time\n\nThe payload should look something like:\n\n```json\n{\n   "analysis:content-length": 91661,\n   "analysis:mime-type": "application/zip",\n   "analysis:checksum": "bef1de04601dedaf2d127418759b16915ba083be",\n   "analysis:last-modified-at": "2022-11-27T23:00:54.762000",\n   "analysis:last-modified-detection": "harvest-resource-metadata",\n}\n```\n\n## Development\n\n### docker-compose\n\nMultiple docker-compose files are provided:\n- a minimal `docker-compose.yml` with two PostgreSQL containers (one for catalog and metadata, the other for converted CSV to database)\n- `docker-compose.broker.yml` adds a Redis broker\n- `docker-compose.test.yml` launches a test DB, needed to run tests\n\nNB: you can launch compose from multiple files like this: `docker-compose -f docker-compose.yml -f docker-compose.test.yml up`\n\n### Logging & Debugging\n\nThe log level can be adjusted using the environment variable LOG_LEVEL.\nFor example, to set the log level to `DEBUG` when initializing the database, use `LOG_LEVEL="DEBUG" udata-hydra init_db `.\n\n### Writing a migration\n\n1. Add a file named `migrations/{YYYYMMDD}_{description}.sql` and write the SQL you need to perform migration.\n2. `udata-hydra migrate` will migrate the database as needeed.\n\n## Deployment\n\n3 services need to be deployed for the full stack to run:\n- worker\n- api / app\n- crawler\n\nRefer to each section to learn how to launch them. The only differences from dev to prod are:\n- use `HYDRA_SETTINGS` env var to point to your custom `config.toml`\n- use `HYDRA_APP_SOCKET_PATH` to configure where aiohttp should listen to a [reverse proxy connection (eg nginx)](https://docs.aiohttp.org/en/stable/deployment.html#nginx-configuration) and use `udata-hydra-app` to launch the app server\n\n## Contributing\n\nBefore contributing to the repository and making any PR, it is necessary to initialize the pre-commit hooks:\n```bash\npre-commit install\n```\nOnce this is done, code formatting and linting, as well as import sorting, will be automatically checked before each commit.\n\nIf you cannot use pre-commit, it is necessary to format, lint, and sort imports with [Ruff](https://docs.astral.sh/ruff/) before committing:\n```bash\nruff check --fix --select I .\nruff format .\n```\n',
+    'author': 'Opendata Team',
+    'author_email': 'opendatateam@data.gouv.fr',
+    'maintainer': 'None',
+    'maintainer_email': 'None',
+    'url': 'None',
+    'packages': packages,
+    'package_data': package_data,
+    'install_requires': install_requires,
+    'entry_points': entry_points,
+    'python_requires': '>=3.9,<4.0',
+}
+
+
+setup(**setup_kwargs)

{udata_hydra-2.0.0.dev1648 → udata_hydra-2.0.0.dev1729}/udata_hydra/analysis/csv.py

@@ -3,36 +3,42 @@ import hashlib
 import json
 import logging
 import os
-import pytz
 import sys
-
 from datetime import datetime
 from typing import Any
 
+import pytz
 import sentry_sdk
-
-from csv_detective.explore_csv import routine as csv_detective_routine
 from csv_detective.detection import engine_to_file
+from csv_detective.explore_csv import routine as csv_detective_routine
 from progressist import ProgressBar
 from sqlalchemy import (
-    MetaData, Table, Column,
-    BigInteger, String, Float, Boolean, Integer, JSON, Date, DateTime,
+    JSON,
+    BigInteger,
+    Boolean,
+    Column,
+    Date,
+    DateTime,
+    Float,
+    Integer,
+    MetaData,
+    String,
+    Table,
 )
 from sqlalchemy.dialects.postgresql import asyncpg
 from sqlalchemy.schema import CreateTable
 from str2bool import str2bool
 from str2float import str2float
 
-from udata_hydra import context, config
+from udata_hydra import config, context
 from udata_hydra.analysis import helpers
 from udata_hydra.analysis.errors import ParseException
-from udata_hydra.utils.db import get_check, compute_insert_query, update_check
+from udata_hydra.utils import queue
+from udata_hydra.utils.db import compute_insert_query, get_check, update_check
 from udata_hydra.utils.file import download_resource
-from udata_hydra.utils.timer import Timer
 from udata_hydra.utils.http import send
 from udata_hydra.utils.reader import Reader
-from udata_hydra.utils import queue
-
+from udata_hydra.utils.timer import Timer
 
 log = logging.getLogger("udata-hydra")
 
@@ -82,15 +88,22 @@ async def notify_udata(check_id):
             "document": {
                 "analysis:parsing:error": check["parsing_error"],
                 "analysis:parsing:started_at": check["parsing_started_at"].isoformat()
-                if check["parsing_started_at"] else None,
+                if check["parsing_started_at"]
+                else None,
                 "analysis:parsing:finished_at": check["parsing_finished_at"].isoformat()
-                if check["parsing_finished_at"] else None,
-            }
+                if check["parsing_finished_at"]
+                else None,
+            },
         }
         queue.enqueue(send, _priority="high", **payload)
 
 
-async def analyse_csv(check_id: int = None, url: str = None, file_path: str = None, debug_insert: bool = False) -> None:
+async def analyse_csv(
+    check_id: int = None,
+    url: str = None,
+    file_path: str = None,
+    debug_insert: bool = False,
+) -> None:
     """Launch csv analysis from a check or an URL (debug), using previsously downloaded file at file_path if any"""
     if not config.CSV_ANALYSIS_ENABLED:
         log.debug("CSV_ANALYSIS_ENABLED turned off, skipping.")
@@ -105,10 +118,14 @@ async def analyse_csv(check_id: int = None, url: str = None, file_path: str = No
     exception_file = str(check.get("resource_id", "")) in exceptions
 
     headers = json.loads(check.get("headers") or "{}")
-    tmp_file = open(file_path, "rb") if file_path else await download_resource(
-        url=url,
-        headers=headers,
-        max_size_allowed=None if exception_file else float(config.MAX_FILESIZE_ALLOWED["csv"]),
+    tmp_file = (
+        open(file_path, "rb")
+        if file_path
+        else await download_resource(
+            url=url,
+            headers=headers,
+            max_size_allowed=None if exception_file else float(config.MAX_FILESIZE_ALLOWED["csv"]),
+        )
     )
     table_name = hashlib.md5(url.encode("utf-8")).hexdigest()
     timer.mark("download-file")
@@ -121,10 +138,13 @@ async def analyse_csv(check_id: int = None, url: str = None, file_path: str = No
         await csv_to_db(tmp_file.name, csv_inspection, table_name, debug_insert=debug_insert)
         timer.mark("csv-to-db")
         if check_id:
-            await update_check(check_id, {
-                "parsing_table": table_name,
-                "parsing_finished_at": datetime.now(pytz.UTC),
-            })
+            await update_check(
+                check_id,
+                {
+                    "parsing_table": table_name,
+                    "parsing_finished_at": datetime.now(pytz.UTC),
+                },
+            )
         await csv_to_db_index(table_name, csv_inspection, check)
     except ParseException as e:
         await handle_parse_exception(e, check_id, table_name)
@@ -196,11 +216,9 @@ async def csv_to_db(file_path: str, inspection: dict, table_name: str, debug_ins
     await db.execute(q)
     with Reader(file_path, inspection) as reader:
         records = (
-            [
-                smart_cast(t, v, failsafe=True)
-                for t, v in zip(columns.values(), line)
-            ]
-            for line in reader if line
+            [smart_cast(t, v, failsafe=True) for t, v in zip(columns.values(), line)]
+            for line in reader
+            if line
         )
         # this use postgresql COPY from an iterator, it's fast but might be difficult to debug
         if not debug_insert:
@@ -223,13 +241,21 @@ async def csv_to_db_index(table_name: str, inspection: dict, check: dict):
     """Store meta info about a converted CSV table in `DATABASE_URL_CSV.tables_index`"""
     db = await context.pool("csv")
     q = "INSERT INTO tables_index(parsing_table, csv_detective, resource_id, url) VALUES($1, $2, $3, $4)"
-    await db.execute(q, table_name, json.dumps(inspection), check.get("resource_id"), check.get("url"))
+    await db.execute(
+        q,
+        table_name,
+        json.dumps(inspection),
+        check.get("resource_id"),
+        check.get("url"),
+    )
 
 
 async def perform_csv_inspection(file_path):
     """Launch csv-detective against given file"""
     try:
-        return csv_detective_routine(file_path, output_profile=True, num_rows=-1, save_results=False)
+        return csv_detective_routine(
+            file_path, output_profile=True, num_rows=-1, save_results=False
+        )
     except Exception as e:
         raise ParseException("csv_detective") from e
 
@@ -242,7 +268,7 @@ async def delete_table(table_name: str):
 
 async def handle_parse_exception(e: Exception, check_id: int, table_name: str) -> None:
     """Specific ParsingError handling. Enriches sentry w/ context if available,
-       and store error if in a check context. Also cleanup :table_name: if needed."""
+    and store error if in a check context. Also cleanup :table_name: if needed."""
     db = await context.pool("csv")
     await db.execute(f'DROP TABLE IF EXISTS "{table_name}"')
     if check_id:
@@ -257,7 +283,10 @@ async def handle_parse_exception(e: Exception, check_id: int, table_name: str) -
         # e.__cause__ let us access the "inherited" error of ParseException (raise e from cause)
         # it's called explicit exception chaining and it's very cool, look it up (PEP 3134)!
         err = f"{e.step}:sentry:{event_id}" if config.SENTRY_DSN else f"{e.step}:{str(e.__cause__)}"
-        await update_check(check_id, {"parsing_error": err, "parsing_finished_at": datetime.now(pytz.UTC)})
+        await update_check(
+            check_id,
+            {"parsing_error": err, "parsing_finished_at": datetime.now(pytz.UTC)},
+        )
         log.error("Parsing error", exc_info=e)
     else:
         raise e

{udata_hydra-2.0.0.dev1648 → udata_hydra-2.0.0.dev1729}/udata_hydra/analysis/errors.py

@@ -1,5 +1,4 @@
 class ParseException(Exception):
-
     def __init__(self, step, *args: object) -> None:
         self.step = step
         super().__init__(*args)

{udata_hydra-2.0.0.dev1648 → udata_hydra-2.0.0.dev1729}/udata_hydra/analysis/helpers.py

@@ -1,7 +1,8 @@
 from datetime import date, datetime
 
 from dateparser import parse as date_parser
-from dateutil.parser import parse as dateutil_parser, ParserError
+from dateutil.parser import ParserError
+from dateutil.parser import parse as dateutil_parser
 
 
 def to_json(value: str) -> str:

{udata_hydra-2.0.0.dev1648 → udata_hydra-2.0.0.dev1729}/udata_hydra/analysis/resource.py

@@ -1,21 +1,19 @@
-from enum import Enum
 import json
 import logging
 import os
-import pytz
-
 from datetime import datetime
+from enum import Enum
 from typing import Tuple, Union
 
 import magic
-
+import pytz
 from dateparser import parse as date_parser
 
-from udata_hydra import context, config
-from udata_hydra.utils import queue
+from udata_hydra import config, context
 from udata_hydra.analysis.csv import analyse_csv
+from udata_hydra.utils import queue
 from udata_hydra.utils.csv import detect_tabular_from_headers
-from udata_hydra.utils.db import update_check, get_check
+from udata_hydra.utils.db import get_check, update_check
 from udata_hydra.utils.file import compute_checksum_from_file, download_resource
 from udata_hydra.utils.http import send
 
@@ -60,8 +58,7 @@ async def process_resource(check_id: int, is_first_check: bool) -> None:
     # could it be a CSV? If we get hints, we will analyse the file further depending on change status
     is_tabular, file_format = await detect_tabular_from_headers(check)
     max_size_allowed = (
-        None if str(resource_id) in exceptions
-        else float(config.MAX_FILESIZE_ALLOWED[file_format])
+        None if str(resource_id) in exceptions else float(config.MAX_FILESIZE_ALLOWED[file_format])
     )
 
     # if the change status is NO_GUESS or HAS_CHANGED, let's download the file to get more infos
@@ -79,18 +76,25 @@ async def process_resource(check_id: int, is_first_check: bool) -> None:
             dl_analysis["analysis:checksum"] = compute_checksum_from_file(tmp_file.name)
             # Check if checksum has been modified if we don't have other hints
             if change_status == Change.NO_GUESS:
-                change_status,  change_payload = await detect_resource_change_from_checksum(
-                    resource_id, dl_analysis["analysis:checksum"])
+                (
+                    change_status,
+                    change_payload,
+                ) = await detect_resource_change_from_checksum(
+                    resource_id, dl_analysis["analysis:checksum"]
+                )
             dl_analysis["analysis:mime-type"] = magic.from_file(tmp_file.name, mime=True)
         finally:
             if tmp_file and not is_tabular:
                 os.remove(tmp_file.name)
-            await update_check(check_id, {
-                "checksum": dl_analysis.get("analysis:checksum"),
-                "analysis_error": dl_analysis.get("analysis:error"),
-                "filesize": dl_analysis.get("analysis:content-length"),
-                "mime_type": dl_analysis.get("analysis:mime-type"),
-            })
+            await update_check(
+                check_id,
+                {
+                    "checksum": dl_analysis.get("analysis:checksum"),
+                    "analysis_error": dl_analysis.get("analysis:error"),
+                    "filesize": dl_analysis.get("analysis:content-length"),
+                    "mime_type": dl_analysis.get("analysis:mime-type"),
+                },
+            )
 
     if change_status == Change.HAS_CHANGED:
         await store_last_modified_date(change_payload or {}, resource_id, check_id)
@@ -119,11 +123,14 @@ async def store_last_modified_date(change_analysis, resource_id, check_id) -> No
         async with pool.acquire() as conn:
             await conn.execute(
                 "UPDATE checks SET detected_last_modified_at = $1 WHERE id = $2",
-                last_modified, check_id
+                last_modified,
+                check_id,
             )
 
 
-async def detect_resource_change_from_checksum(resource_id, new_checksum) -> Tuple[Change, Union[dict, None]]:
+async def detect_resource_change_from_checksum(
+    resource_id, new_checksum
+) -> Tuple[Change, Union[dict, None]]:
     """
     Checks if resource checksum has changed over time
     Returns a tuple with a Change status and an optional payload:
@@ -150,7 +157,9 @@ async def detect_resource_change_from_checksum(resource_id, new_checksum) -> Tup
     return Change.NO_GUESS, None
 
 
-async def detect_resource_change_from_last_modified_header(data: dict) -> Tuple[Change, Union[dict, None]]:
+async def detect_resource_change_from_last_modified_header(
+    data: dict,
+) -> Tuple[Change, Union[dict, None]]:
     # last modified header check
 
     if len(data) == 1 and data[0]["last_modified"]:
@@ -172,7 +181,9 @@ async def detect_resource_change_from_last_modified_header(data: dict) -> Tuple[
     return Change.HAS_NOT_CHANGED, None
 
 
-async def detect_resource_change_from_content_length_header(data: dict) -> Tuple[Change, Union[dict, None]]:
+async def detect_resource_change_from_content_length_header(
+    data: dict,
+) -> Tuple[Change, Union[dict, None]]:
     # content-length variation between current and last check
     if len(data) <= 1 or not data[0]["content_length"]:
         return Change.NO_GUESS, None
@@ -185,7 +196,9 @@ async def detect_resource_change_from_content_length_header(data: dict) -> Tuple
     return Change.HAS_NOT_CHANGED, None
 
 
-async def detect_resource_change_on_early_hints(resource_id: str) -> Tuple[Change, Union[dict, None]]:
+async def detect_resource_change_on_early_hints(
+    resource_id: str,
+) -> Tuple[Change, Union[dict, None]]:
     """
     Try to guess if a resource has been modified from harvest and headers in check data:
     - last-modified header value if it can be found and parsed
@@ -223,7 +236,10 @@ async def detect_resource_change_on_early_hints(resource_id: str) -> Tuple[Chang
         return change_status, change_payload
 
     # if not, let's see if we can infer a modifification date from last-modified headers
-    change_status, change_payload = await detect_resource_change_from_last_modified_header(data)
+    (
+        change_status,
+        change_payload,
+    ) = await detect_resource_change_from_last_modified_header(data)
     if change_status != Change.NO_GUESS:
         return change_status, change_payload
 
@@ -231,7 +247,9 @@ async def detect_resource_change_on_early_hints(resource_id: str) -> Tuple[Chang
     return await detect_resource_change_from_content_length_header(data)
 
 
-async def detect_resource_change_from_harvest(checks_data: dict, resource_id: str) -> Tuple[Change, Union[dict, None]]:
+async def detect_resource_change_from_harvest(
+    checks_data: dict, resource_id: str
+) -> Tuple[Change, Union[dict, None]]:
     """
     Checks if resource has a harvest.modified_at
     Returns a tuple with a Change status and an optional payload: