ugrc-sweeper 2.0.1__tar.gz

ugrc-sweeper-2.0.1/LICENSE

@@ -0,0 +1,9 @@
+MIT License
+
+Copyright (c) UGRC contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

ugrc-sweeper-2.0.1/MANIFEST.in

@@ -0,0 +1,3 @@
+include src/sweeper/*.json
+include src/sweeper/sweepers/*.html
+exclude config.json

ugrc-sweeper-2.0.1/PKG-INFO

@@ -0,0 +1,160 @@
+Metadata-Version: 2.1
+Name: ugrc-sweeper
+Version: 2.0.1
+Summary: CLI tool for making good data
+Home-page: https://github.com/agrc/sweeper
+Author: UGRC
+Author-email: ugrc-developers@utah.gov
+License: MIT
+Platform: UNKNOWN
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: Unix
+Classifier: Operating System :: POSIX
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Utilities
+Requires-Python: >=3
+Description-Content-Type: text/markdown
+Provides-Extra: tests
+License-File: LICENSE
+
+# ugrc-sweeper [![PyPI version](https://badge.fury.io/py/ugrc-sweeper.svg)](https://badge.fury.io/py/ugrc-sweeper)[![Push Events](https://github.com/agrc/sweeper/actions/workflows/push.yml/badge.svg)](https://github.com/agrc/sweeper/actions/workflows/push.yml)
+
+The data cleaning service.
+
+![sweeper_sm](https://user-images.githubusercontent.com/325813/90411835-91c4c080-e069-11ea-9d03-f3e60421b835.png)
+
+## Available Sweepers
+
+### Addresses
+
+Checks that addresses have minimum required parts and optionally normalizes them.
+
+### Duplicates
+
+Checks for duplicate features.
+
+### Empties
+
+Checks for empty geometries.
+
+### Metadata
+
+Checks to make sure that the metadata meets [the Basic SGID Metadata Requirements](https://gis.utah.gov/about/policy/metadata/#basic-sgid-metadata).
+
+#### Tags
+
+Checks to make sure that existing tags are cased appropriately. This mean that the are title-cased other than known abbreviations (e.g. UGRC, BLM) and articles (e.g. a, the, of).
+
+This check also verifies that the data set contains a tag that matches the database name (e.g. `SGID`) and the schema (e.g. `Cadastre`).
+
+`--try-fix` adds missing required tags and title-cases any existing tags.
+
+#### Summary
+
+Checks to make sure that the summary is less than 2048 characters (a limitation of AGOL) and that it is shorter than the description.
+
+#### Description
+
+Checks to make sure that the description contains a link to a data page on gis.utah.gov.
+
+#### Use Limitations
+
+Checks to make sure that the text in this section matches the [official text for UGRC](src/sweeper/sweepers/UseLimitations.html).
+
+`--try-fix` updates the text to match the official text.
+
+## Parsing Addresses
+
+This project contains a module that can be used as a standalone address parser, `sweeper.address_parser`. This allows developer to take advantage of sweepers advanced address parsing and normalization without having to run the entire sweeper process.
+
+### Usage Example
+
+```python
+from sweeper.address_parser import Address
+
+address = Address('123 South Main Street')
+print(address)
+
+'''
+--> Parsed Address:
+{'address_number': '123',
+ 'normalized': '123 S MAIN ST',
+ 'prefix_direction': 'S',
+ 'street_name': 'MAIN',
+ 'street_type': 'ST'}
+'''
+```
+
+### Available Address class properties
+
+All properties default to None if there is no parsed value.
+
+`address_number`
+
+`address_number_suffix`
+
+`prefix_direction`
+
+`street_name`
+
+`street_direction`
+
+`street_type`
+
+`unit_type`
+
+`unit_id`
+If no `unit_type` is found, this property is prefixed with `#` (e.g. `# 3`). If `unit_type` is found, `#` is stripped from this property.
+
+`city`
+
+`zip_code`
+
+`po_box`
+The PO Box if a po-box-type address was entered (e.g. `po_box` would be `1` for `p.o. box 1`).
+
+`normalized`
+A normalized string representing the entire address that was passed into the constructor. PO Boxes are normalized in this format `PO BOX <number>`.
+
+## Installation (requires Pro 2.7+)
+
+<!-- Current conda install arcpy -c esri seems to be wonky; just clone to be safe -->
+
+1. clone arcgis conda environment
+   - `conda create --name sweeper --clone arcgispro-py3`
+1. activate environment
+   - `activate sweeper`
+1. install sweeper
+   - `pip install ugrc-sweeper`
+1. Optionally duplicate `config.sample.json` as `config.json` in the folder where you will run sweeper.
+
+> [!CAUTION]
+> This is required for the following functions:
+>
+> - `--scheduled` argument (required for sending emails)
+> - `--change-detect` argument
+> - using user-specific connection files via the `CONNECTIONS_FOLDER` config value
+
+## Exclusions
+
+Tables can be skipped by adding values to the `EXCLUSIONS.<sweeper_key>` config array. These values are matched against table names using [fnmatch](https://docs.python.org/3/library/fnmatch.html#fnmatch.fnmatch). Note that these do not apply when using the `--table-name` argument.
+
+## Development
+
+1. clone arcgis conda environment
+   - `conda create --name sweeper --clone arcgispro-py3`
+1. activate environment
+   - `activate sweeper`
+1. install required dependencies to work on sweeper
+   - `pip install -e ".[tests]"`
+1. `test_metadata.py` uses a SQL database that needs to be restored via `src/sweeper/tests/data/Sweeper.bak` to your local SQL Server.
+1. run sweeper: `sweeper`
+1. test: `pytest`
+1. lint: `ruff check .`
+1. format: `ruff format .`
+
+

ugrc-sweeper-2.0.1/pyproject.toml

@@ -0,0 +1,10 @@
+[tool.ruff]
+line-length = 120
+[tool.ruff.lint]
+ignore = ["E501"]
+[tool.pytest.ini_options]
+minversion = "6.0"
+testpaths = ["tests", "src"]
+norecursedirs = [".env", "data", "maps", ".github", ".vscode"]
+console_output_style = "count"
+addopts = "--cov-branch --cov=sweeper --cov-report term --cov-report xml:cov.xml --instafail -p no:faulthandler"

ugrc-sweeper-2.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

ugrc-sweeper-2.0.1/setup.py

@@ -0,0 +1,65 @@
+#!/usr/bin/env python
+# -*- encoding: utf-8 -*-
+"""
+setup.py
+A module that installs sweeper as a module
+"""
+
+import glob
+from os.path import basename, splitext
+from pathlib import Path
+
+from setuptools import find_packages, setup
+
+setup(
+    name="ugrc-sweeper",
+    version="2.0.1",
+    license="MIT",
+    description="CLI tool for making good data",
+    long_description=(Path(__file__).parent / "readme.md").read_text(),
+    long_description_content_type="text/markdown",
+    author="UGRC",
+    author_email="ugrc-developers@utah.gov",
+    url="https://github.com/agrc/sweeper",
+    packages=find_packages("src"),
+    package_dir={"": "src"},
+    py_modules=[splitext(basename(i))[0] for i in glob.glob("src/*.py")],
+    python_requires=">=3",
+    include_package_data=True,
+    zip_safe=False,
+    classifiers=[
+        "Development Status :: 5 - Production/Stable",
+        "Intended Audience :: Developers",
+        "License :: OSI Approved :: MIT License",
+        "Operating System :: Unix",
+        "Operating System :: POSIX",
+        "Operating System :: Microsoft :: Windows",
+        "Programming Language :: Python",
+        "Programming Language :: Python :: 3 :: Only",
+        "Topic :: Utilities",
+    ],
+    keywords=[],
+    install_requires=[
+        "agrc-supervisor==3.*",
+        "agrc-usaddress==0.*",
+        "beautifulsoup4==4.*",
+        "docopt==0.*",
+        "html5lib==1.*",
+        "xxhash==3.*",
+    ],
+    dependency_links=[],
+    extras_require={
+        "tests": [
+            "pytest-cov==5.*",
+            "pytest-instafail==0.5.*",
+            "pytest-mock==3.*",
+            "pytest-watch==4.*",
+            "pytest==8.*",
+            "ruff==0.*",
+        ],
+    },
+    setup_requires=[
+        "pytest-runner",
+    ],
+    entry_points={"console_scripts": ["sweeper = sweeper.__main__:main"]},
+)

ugrc-sweeper-2.0.1/src/sweeper/__main__.py

@@ -0,0 +1,207 @@
+#!/usr/bin/env python
+# * coding: utf8 *
+"""
+sweeper
+
+Usage:
+  sweeper sweep duplicates  --workspace=<workspace> [--table-name=<table_name> --verbose --try-fix --change-detect --scheduled --save-report=<report_path> --backup-to=<backup_path>]
+  sweeper sweep empties     --workspace=<workspace> [--table-name=<table_name> --verbose --try-fix --change-detect --scheduled --save-report=<report_path> --backup-to=<backup_path>]
+  sweeper sweep invalids    --workspace=<workspace> [--table-name=<table_name> --verbose --try-fix --change-detect --scheduled --save-report=<report_path> --backup-to=<backup_path>]
+  sweeper sweep addresses   --workspace=<workspace> --table-name=<table-name> --field-name=<field_name> [--verbose --try-fix --save-report=<report_path> --backup-to=<backup_path>]
+  sweeper sweep metadata    --workspace=<workspace> [--table-name=<table_name> --verbose --try-fix --change-detect --scheduled --save-report=<report_path> --backup-to=<backup_path>]
+  sweeper sweep             --workspace=<workspace> [--table-name=<table_name> --verbose --try-fix --change-detect --scheduled --save-report=<report_path> --backup-to=<backup_path>]
+
+Arguments:
+  workspace     - path to workspace eg: `c:\\my.gdb`
+  table_name    - name of feature class or table eg: `Roads` (needs to be fully qualified (eg: `SGID.Transportation.Roads`) for metadata sweeper)
+  report_path   - folder to save report to eg: `c:\\temp`
+  backup_path   - place to create a temp gdb and import original table
+  field_name    - name of the field to check
+
+Examples:
+  sweeper sweep           --workspace=c:\\data\\thing --try-fix --save-report=c:\\temp --backup-to=c:\\temp\\backup.gdb
+  sweeper sweep addresses --workspace=c:\\data\\thing --try-fix --save-report=c:\\temp --backup-to=c:\\temp\\backup.gdb --field-name=ADDRESS
+"""
+
+import datetime
+import logging
+import logging.handlers
+import sys
+from pathlib import Path
+
+import pkg_resources
+from docopt import docopt
+from supervisor.message_handlers import SendGridHandler
+from supervisor.models import MessageDetails, Supervisor
+
+from . import backup, config, report, utilities, workspace_info
+from .sweepers.addresses import AddressTest
+from .sweepers.duplicates import DuplicateTest
+from .sweepers.empties import EmptyTest
+from .sweepers.metadata import MetadataTest
+
+
+def main():
+    """Main entry point for program. Parse arguments and pass to sweeper modules."""
+    args = docopt(__doc__, version=pkg_resources.require("ugrc-sweeper")[0].version)
+
+    log = setup_logging(args["--save-report"], args["--scheduled"])
+
+    if args["--scheduled"]:
+        #: set up supervisor, add email handler
+        sweeper_supervisor = Supervisor()
+        sweeper_supervisor.add_message_handler(
+            SendGridHandler(
+                {
+                    "from_address": "noreply@utah.gov",
+                    "to_addresses": config.get_config("TO_ADDRESSES"),
+                    "api_key": config.get_config("SENDGRID_API_KEY"),
+                },
+                client_name="ugrc-sweeper",
+                client_version=pkg_resources.require("ugrc-sweeper")[0].version,
+            )
+        )
+
+    #: backup input file before quality checks
+    if args["--backup-to"]:
+        backup.backup_data(args["--workspace"], args["--table-name"], args["--backup-to"])
+
+    #: create a list to hold the instantiated objects.
+    closet = []
+
+    #: check what quality check to run.
+    if args["duplicates"]:
+        closet.append(DuplicateTest(args["--workspace"], args["--table-name"]))
+    elif args["invalids"]:
+        raise NotImplementedError('"Invalids" sweep/check not implemented yet.')
+    elif args["empties"]:
+        closet.append(EmptyTest(args["--workspace"], args["--table-name"]))
+    elif args["addresses"]:
+        closet.append(AddressTest(args["--workspace"], args["--table-name"], args["--field-name"]))
+    elif args["metadata"]:
+        closet.append(MetadataTest(args["--workspace"], args["--table-name"]))
+    else:
+        closet.append(DuplicateTest(args["--workspace"], args["--table-name"]))
+        closet.append(EmptyTest(args["--workspace"], args["--table-name"]))
+        closet.append(MetadataTest(args["--workspace"], args["--table-name"]))
+
+    reports = execute_sweepers(closet, args["--try-fix"], args["--change-detect"], log)
+
+    report.print_report(reports)
+
+    if args["--save-report"]:
+        report.save_report(reports, args["--save-report"])
+
+    if args["--scheduled"]:
+        report.add_to_log(reports)
+
+        final_message = report.format_message(reports)
+        log.info(final_message.getvalue())
+
+        #: Build and send summary message
+        summary_message = MessageDetails()
+        summary_message.message = final_message.getvalue()
+        summary_message.attachments = [config.LOG_FILE_PATH]
+        summary_message.subject = f"Sweeper Report {datetime.datetime.today()}"
+
+        sweeper_supervisor.notify(summary_message)
+
+
+def execute_sweepers(closet, try_fix, using_change_detection, log):
+    """
+    orchestrate the sweeper calls.
+
+    closet: array of sweepers.
+    try_fix: bool whether to fix or not.
+    """
+
+    feature_class_names = []
+    reports = []
+
+    def run_tool(tool):
+        reports.append(tool.sweep())
+
+        if try_fix:
+            reports.append(tool.try_fix())
+
+            #: run sweeper again to ensure all errors were fixed.
+            reports.append(tool.sweep())
+
+    log.info(f"running {len(closet)} sweepers. try fix: {try_fix}")
+    for tool in closet:
+        log.info(f"running sweeper: {tool.key}")
+        if tool.table_name:
+            run_tool(tool)
+
+            continue
+
+        #: get feature class names once
+        if len(feature_class_names) == 0:
+            if using_change_detection:
+                log.info("Getting table names from change detection table")
+                feature_class_names = workspace_info.get_change_detection()
+            else:
+                log.info("Missing table name, executing over workspace")
+                feature_class_names = workspace_info.get_featureclasses(tool.workspace)
+                if any("SGID." in fc for fc in feature_class_names):
+                    feature_class_names = [fc.split("SGID.", 1)[1] for fc in feature_class_names if "SGID." in fc]
+
+            #: apply exclusions
+            if config.has_config():
+                try:
+                    exclusions_config = config.get_config("EXCLUSIONS")
+                except KeyError:
+                    exclusions_config = {}
+
+                exclusions = exclusions_config.get(tool.key, [])
+            feature_class_names = utilities.apply_exclusions(feature_class_names, exclusions)
+
+        log.info(f"feature_class_names is: {feature_class_names}")
+
+        if using_change_detection and feature_class_names is None:
+            #: reset variable to empty list
+            log.info("Change detection found no updated tables")
+            feature_class_names = []
+
+            continue
+
+        #: explode sweeper class for each feature class
+        for table_name in feature_class_names:
+            new_tool = tool.clone(table_name, tool.workspace)
+
+            run_tool(new_tool)
+
+    if using_change_detection:
+        workspace_info.update_last_check_date()
+
+    return reports
+
+
+def setup_logging(save_report, scheduled):
+    logger = logging.getLogger("sweeper")
+    logger.setLevel(logging.INFO)
+
+    formatter = logging.Formatter(
+        fmt="%(levelname)-7s %(asctime)s %(module)10s:%(lineno)5s %(message)s", datefmt="%m-%d %H:%M:%S"
+    )
+
+    #: always set up console_handler
+    console_handler = logging.StreamHandler(stream=sys.stdout)
+    console_handler.setFormatter(formatter)
+
+    #: use log file when report location not provided and when running from scheduled task
+    if scheduled and not save_report:
+        log_file = Path(config.LOG_FILE_PATH)
+        file_handler = logging.handlers.RotatingFileHandler(log_file, backupCount=10)
+        file_handler.doRollover()
+        file_handler.setFormatter(formatter)
+
+        logger.addHandler(file_handler)
+
+    logger.addHandler(console_handler)
+
+    return logger
+
+
+if __name__ == "__main__":
+    sys.exit(main())