geocrime-stl 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- geocrime_stl-0.1.0/LICENSE +21 -0
- geocrime_stl-0.1.0/PKG-INFO +85 -0
- geocrime_stl-0.1.0/README.md +72 -0
- geocrime_stl-0.1.0/pyproject.toml +23 -0
- geocrime_stl-0.1.0/setup.cfg +4 -0
- geocrime_stl-0.1.0/src/geocrime_stl/__init__.py +24 -0
- geocrime_stl-0.1.0/src/geocrime_stl/config.py +75 -0
- geocrime_stl-0.1.0/src/geocrime_stl/etl/extract.py +93 -0
- geocrime_stl-0.1.0/src/geocrime_stl/etl/transform.py +160 -0
- geocrime_stl-0.1.0/src/geocrime_stl.egg-info/PKG-INFO +85 -0
- geocrime_stl-0.1.0/src/geocrime_stl.egg-info/SOURCES.txt +12 -0
- geocrime_stl-0.1.0/src/geocrime_stl.egg-info/dependency_links.txt +1 -0
- geocrime_stl-0.1.0/src/geocrime_stl.egg-info/requires.txt +4 -0
- geocrime_stl-0.1.0/src/geocrime_stl.egg-info/top_level.txt +1 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 lweber89
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: geocrime-stl
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: An ETL for fetching/cleaning St. Louis crime data sets.
|
|
5
|
+
Requires-Python: >=3.8
|
|
6
|
+
Description-Content-Type: text/markdown
|
|
7
|
+
License-File: LICENSE
|
|
8
|
+
Requires-Dist: geopandas>=1.1.3
|
|
9
|
+
Requires-Dist: matplotlib>=3.10.9
|
|
10
|
+
Requires-Dist: pandas>=3.0.3
|
|
11
|
+
Requires-Dist: requests>=2.34.2
|
|
12
|
+
Dynamic: license-file
|
|
13
|
+
|
|
14
|
+
# geocrime-stl
|
|
15
|
+
|
|
16
|
+
[](https://opensource.org/licenses/MIT)
|
|
17
|
+
[](https://www.python.org/downloads/)
|
|
18
|
+
|
|
19
|
+
A Python utility for cleaning and standardizing St. Louis Metropolitan Police Department (SLMPD) crime data for spatial analysis.
|
|
20
|
+
|
|
21
|
+
## ✨ Features
|
|
22
|
+
|
|
23
|
+
* **End-to-End ETL:** Fetch, clean, and standardize monthly SLMPD crime data files automatically.
|
|
24
|
+
* **Spatial Formats:** Convert cleaned DataFrames seamlessly into GeoJSON, GeoPackage (GPKG), or CSV.
|
|
25
|
+
* **Analysis Ready:** Access built-in methods for minimal data summarization and visualization.
|
|
26
|
+
|
|
27
|
+
## 🌐 Data Source & Disclaimer
|
|
28
|
+
|
|
29
|
+
This utility programmatically fetches publicly available crime data directly from the **[St. Louis Metropolitan Police Department (SLMPD) Stats Page](https://slmpd.org/stats/)**.
|
|
30
|
+
|
|
31
|
+
⚠️ **Data Disclaimer & Limitations:**
|
|
32
|
+
* **Availability:** This tool relies entirely on the upstream availability and hosting structure of the SLMPD website. If their servers are down or their URL layout changes, the extraction pipeline may fail.
|
|
33
|
+
* **Data Integrity & Spatial Boundaries:** The data processed is strictly "as-is" from the published monthly files. To ensure high data quality for spatial analysis, this utility applies two strict filtering constraints:
|
|
34
|
+
1. **Temporal Filtering:** Because the SLMPD frequently updates historical records retroactively, this utility isolates and preserves only the records belonging to the target month.
|
|
35
|
+
2. **Spatial Clipping:** Records containing invalid, malformed, or erroneous latitude/longitude coordinates that plot completely outside the official St. Louis city boundaries are automatically dropped.
|
|
36
|
+
* **Project Status:** This is an independent, open-source utility. It is not affiliated with, endorsed by, or officially maintained by the SLMPD or the City of St. Louis.
|
|
37
|
+
|
|
38
|
+
## 🚀 Quick Start
|
|
39
|
+
|
|
40
|
+
### Install
|
|
41
|
+
``` bash
|
|
42
|
+
pip install geocrime-stl
|
|
43
|
+
```
|
|
44
|
+
### Fetch/Clean/Visualize Data
|
|
45
|
+
```python
|
|
46
|
+
import geocrime_stl as gc
|
|
47
|
+
|
|
48
|
+
data_pkg = gc.run_pipeline(4, 2026)
|
|
49
|
+
|
|
50
|
+
gc.generate_monthly_metrics(data_pkg)
|
|
51
|
+
|
|
52
|
+
gc.plot_monthly_maps(data_pkg)
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
## 🛠️ Local Setup (Alternative)
|
|
56
|
+
|
|
57
|
+
If you want to run the demo notebook locally or explore the source code, you can clone the repository directly:
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
git clone https://github.com/lweber89/geocrime-stl.git
|
|
61
|
+
cd geocrime-stl
|
|
62
|
+
pip install -e .
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
## 📓 Demo & Tutorial
|
|
66
|
+
To see a step-by-step walkthrough of the entire ETL process, check out the interactive Jupyter notebook included in this package:
|
|
67
|
+
|
|
68
|
+
👉 **[View the Demo Notebook](./notebooks/demo.ipynb)**
|
|
69
|
+
|
|
70
|
+
Note: If you have cloned the repository locally, you can open this file using Jupyter Lab or Jupyter Notebook to run the cells yourself.
|
|
71
|
+
|
|
72
|
+
## 🤝 Contributing & Pull Requests
|
|
73
|
+
Thank you for your interest in the project!
|
|
74
|
+
|
|
75
|
+
⚠️ Please Note: This is a personal project I developed and maintain in an effort to better understand the implementation of Python in geospatial data engineering / solutions architecture. I am not accepting pull requests or code contributions at this time.
|
|
76
|
+
|
|
77
|
+
If you find a bug or have an idea, feel free to open an Issue to let me know, or feel free to fork the repository and adapt it for your own personal needs!
|
|
78
|
+
|
|
79
|
+
## 📜 License
|
|
80
|
+
This project is licensed under the MIT License - see the LICENSE file for details.
|
|
81
|
+
|
|
82
|
+
## 🔗 Project Links
|
|
83
|
+
GitHub Repository: https://github.com/lweber89/geocrime-stl
|
|
84
|
+
|
|
85
|
+
PyPI Home: https://pypi.org/project/geocrime-stl/
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
# geocrime-stl
|
|
2
|
+
|
|
3
|
+
[](https://opensource.org/licenses/MIT)
|
|
4
|
+
[](https://www.python.org/downloads/)
|
|
5
|
+
|
|
6
|
+
A Python utility for cleaning and standardizing St. Louis Metropolitan Police Department (SLMPD) crime data for spatial analysis.
|
|
7
|
+
|
|
8
|
+
## ✨ Features
|
|
9
|
+
|
|
10
|
+
* **End-to-End ETL:** Fetch, clean, and standardize monthly SLMPD crime data files automatically.
|
|
11
|
+
* **Spatial Formats:** Convert cleaned DataFrames seamlessly into GeoJSON, GeoPackage (GPKG), or CSV.
|
|
12
|
+
* **Analysis Ready:** Access built-in methods for minimal data summarization and visualization.
|
|
13
|
+
|
|
14
|
+
## 🌐 Data Source & Disclaimer
|
|
15
|
+
|
|
16
|
+
This utility programmatically fetches publicly available crime data directly from the **[St. Louis Metropolitan Police Department (SLMPD) Stats Page](https://slmpd.org/stats/)**.
|
|
17
|
+
|
|
18
|
+
⚠️ **Data Disclaimer & Limitations:**
|
|
19
|
+
* **Availability:** This tool relies entirely on the upstream availability and hosting structure of the SLMPD website. If their servers are down or their URL layout changes, the extraction pipeline may fail.
|
|
20
|
+
* **Data Integrity & Spatial Boundaries:** The data processed is strictly "as-is" from the published monthly files. To ensure high data quality for spatial analysis, this utility applies two strict filtering constraints:
|
|
21
|
+
1. **Temporal Filtering:** Because the SLMPD frequently updates historical records retroactively, this utility isolates and preserves only the records belonging to the target month.
|
|
22
|
+
2. **Spatial Clipping:** Records containing invalid, malformed, or erroneous latitude/longitude coordinates that plot completely outside the official St. Louis city boundaries are automatically dropped.
|
|
23
|
+
* **Project Status:** This is an independent, open-source utility. It is not affiliated with, endorsed by, or officially maintained by the SLMPD or the City of St. Louis.
|
|
24
|
+
|
|
25
|
+
## 🚀 Quick Start
|
|
26
|
+
|
|
27
|
+
### Install
|
|
28
|
+
``` bash
|
|
29
|
+
pip install geocrime-stl
|
|
30
|
+
```
|
|
31
|
+
### Fetch/Clean/Visualize Data
|
|
32
|
+
```python
|
|
33
|
+
import geocrime_stl as gc
|
|
34
|
+
|
|
35
|
+
data_pkg = gc.run_pipeline(4, 2026)
|
|
36
|
+
|
|
37
|
+
gc.generate_monthly_metrics(data_pkg)
|
|
38
|
+
|
|
39
|
+
gc.plot_monthly_maps(data_pkg)
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
## 🛠️ Local Setup (Alternative)
|
|
43
|
+
|
|
44
|
+
If you want to run the demo notebook locally or explore the source code, you can clone the repository directly:
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
git clone https://github.com/lweber89/geocrime-stl.git
|
|
48
|
+
cd geocrime-stl
|
|
49
|
+
pip install -e .
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
## 📓 Demo & Tutorial
|
|
53
|
+
To see a step-by-step walkthrough of the entire ETL process, check out the interactive Jupyter notebook included in this package:
|
|
54
|
+
|
|
55
|
+
👉 **[View the Demo Notebook](./notebooks/demo.ipynb)**
|
|
56
|
+
|
|
57
|
+
Note: If you have cloned the repository locally, you can open this file using Jupyter Lab or Jupyter Notebook to run the cells yourself.
|
|
58
|
+
|
|
59
|
+
## 🤝 Contributing & Pull Requests
|
|
60
|
+
Thank you for your interest in the project!
|
|
61
|
+
|
|
62
|
+
⚠️ Please Note: This is a personal project I developed and maintain in an effort to better understand the implementation of Python in geospatial data engineering / solutions architecture. I am not accepting pull requests or code contributions at this time.
|
|
63
|
+
|
|
64
|
+
If you find a bug or have an idea, feel free to open an Issue to let me know, or feel free to fork the repository and adapt it for your own personal needs!
|
|
65
|
+
|
|
66
|
+
## 📜 License
|
|
67
|
+
This project is licensed under the MIT License - see the LICENSE file for details.
|
|
68
|
+
|
|
69
|
+
## 🔗 Project Links
|
|
70
|
+
GitHub Repository: https://github.com/lweber89/geocrime-stl
|
|
71
|
+
|
|
72
|
+
PyPI Home: https://pypi.org/project/geocrime-stl/
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=61.0"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "geocrime-stl"
|
|
7
|
+
version = "0.1.0"
|
|
8
|
+
description = "An ETL for fetching/cleaning St. Louis crime data sets."
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
requires-python = ">=3.8"
|
|
11
|
+
dependencies = [
|
|
12
|
+
"geopandas>=1.1.3",
|
|
13
|
+
"matplotlib>=3.10.9",
|
|
14
|
+
"pandas>=3.0.3",
|
|
15
|
+
"requests>=2.34.2"
|
|
16
|
+
]
|
|
17
|
+
|
|
18
|
+
[tool.setuptools.packages.find]
|
|
19
|
+
where = ["src"]
|
|
20
|
+
include = ["geocrime_stl"]
|
|
21
|
+
|
|
22
|
+
[tool.setuptools.package-data]
|
|
23
|
+
geocrime_stl = ["data/*.geojson"]
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
from __future__ import annotations
|
|
2
|
+
|
|
3
|
+
from .config import CITY_BNDY, NBHD_BNDY, STL_MAP_CONFIG
|
|
4
|
+
|
|
5
|
+
# Expose the master pipeline orchestrator directly to the root namespace
|
|
6
|
+
from .etl.transform import run_pipeline
|
|
7
|
+
|
|
8
|
+
# Expose the monthly summary and plottin methods to the root namespace
|
|
9
|
+
from .insights.monthly_summary import generate_monthly_metrics, plot_monthly_maps
|
|
10
|
+
|
|
11
|
+
# Expose the utilities and exporters cleanly to the root namespace
|
|
12
|
+
from .utils.export import export_to_csv, export_to_geojson, export_to_gpkg
|
|
13
|
+
|
|
14
|
+
__all__ = [
|
|
15
|
+
"CITY_BNDY",
|
|
16
|
+
"NBHD_BNDY",
|
|
17
|
+
"STL_MAP_CONFIG",
|
|
18
|
+
"run_pipeline",
|
|
19
|
+
"export_to_csv",
|
|
20
|
+
"export_to_geojson",
|
|
21
|
+
"export_to_gpkg",
|
|
22
|
+
"plot_monthly_maps",
|
|
23
|
+
"generate_monthly_metrics"
|
|
24
|
+
]
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
from pathlib import Path
|
|
2
|
+
from typing import NamedTuple
|
|
3
|
+
|
|
4
|
+
# --- Paths ---
|
|
5
|
+
# .parent takes us to 'geocrime_stl', the second .parent takes us out to 'src'
|
|
6
|
+
# The third .parent takes us to the root project directory where 'data' lives!
|
|
7
|
+
BASE_DIR = Path(__file__).resolve().parent.parent.parent
|
|
8
|
+
DATA_DIR = BASE_DIR / "data"
|
|
9
|
+
|
|
10
|
+
CITY_BNDY = DATA_DIR / "stl_boundary.geojson"
|
|
11
|
+
NBHD_BNDY = DATA_DIR / "stl_neighborhoods.geojson"
|
|
12
|
+
CITY_BNDY_WGS84 = DATA_DIR / "stl_boundary_wgs84.geojson"
|
|
13
|
+
|
|
14
|
+
|
|
15
|
+
# --- Map Configuration ---
|
|
16
|
+
class MapConfig(NamedTuple):
|
|
17
|
+
coords: tuple[float, float]
|
|
18
|
+
zoom: float
|
|
19
|
+
height: str
|
|
20
|
+
bbox: tuple[float, float, float, float]
|
|
21
|
+
|
|
22
|
+
STL_MAP_CONFIG = MapConfig(
|
|
23
|
+
coords=(38.65428167189044, -90.25320053100587),
|
|
24
|
+
zoom=11.5,
|
|
25
|
+
height="925px",
|
|
26
|
+
bbox=(38.5318, -90.3205, 38.7744, -90.1663),
|
|
27
|
+
)
|
|
28
|
+
|
|
29
|
+
|
|
30
|
+
# --- SLMPD ETL Settings ---
|
|
31
|
+
BASE_URL = "https://slmpd.org/wp-content/uploads"
|
|
32
|
+
|
|
33
|
+
COLUMN_RENAMES = {
|
|
34
|
+
"Latitude": "lat",
|
|
35
|
+
"Longitude": "lon",
|
|
36
|
+
"IncidentNum": "inc_#",
|
|
37
|
+
"NIBRS": "nibrs_code",
|
|
38
|
+
"NIBRSCategory": "nibrs_cat",
|
|
39
|
+
"CrimeAgainst": "off_type",
|
|
40
|
+
"IncidentLocation": "address",
|
|
41
|
+
"District": "district",
|
|
42
|
+
"Neighborhood": "nbhd",
|
|
43
|
+
"NbhdNum": "nbhd_num",
|
|
44
|
+
"FirearmUsed": "firearm",
|
|
45
|
+
"IncidentNature": "inc_desc",
|
|
46
|
+
"Offense": "offense",
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
COLUMNS_TO_DROP = (
|
|
50
|
+
"IncidentDate",
|
|
51
|
+
"OccurredFromTime",
|
|
52
|
+
"SRS_UCR",
|
|
53
|
+
"IncidentTopSRS_UCR",
|
|
54
|
+
"IntersectionOtherLoc",
|
|
55
|
+
"IncidentSupplemented",
|
|
56
|
+
"LastSuppDate",
|
|
57
|
+
"VictimNum",
|
|
58
|
+
"FelMisdCit",
|
|
59
|
+
)
|
|
60
|
+
FINAL_SCHEMA_COLUMNS = [
|
|
61
|
+
"inc_#",
|
|
62
|
+
"date_time",
|
|
63
|
+
"nibrs_cat",
|
|
64
|
+
"nibrs_code",
|
|
65
|
+
"offense",
|
|
66
|
+
"inc_desc",
|
|
67
|
+
"off_type",
|
|
68
|
+
"firearm",
|
|
69
|
+
"address",
|
|
70
|
+
"nbhd",
|
|
71
|
+
"nbhd_num",
|
|
72
|
+
"district",
|
|
73
|
+
"lat",
|
|
74
|
+
"lon",
|
|
75
|
+
]
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
from __future__ import annotations
|
|
2
|
+
|
|
3
|
+
import logging
|
|
4
|
+
from datetime import datetime
|
|
5
|
+
|
|
6
|
+
import requests
|
|
7
|
+
|
|
8
|
+
from geocrime_stl.config import BASE_URL
|
|
9
|
+
|
|
10
|
+
logger = logging.getLogger(__name__)
|
|
11
|
+
|
|
12
|
+
class ExtractionError(Exception):
|
|
13
|
+
"""Raised if the target URL cannot be formed or network operations fail."""
|
|
14
|
+
pass
|
|
15
|
+
|
|
16
|
+
|
|
17
|
+
def construct_url(
|
|
18
|
+
month_input: str | int | None = None,
|
|
19
|
+
year_input: str | int | None = None
|
|
20
|
+
) -> tuple[str, int, int]:
|
|
21
|
+
"""Parses flexible date inputs and builds the target SLMPD data asset URL."""
|
|
22
|
+
try:
|
|
23
|
+
now = datetime.now()
|
|
24
|
+
if now.month == 1:
|
|
25
|
+
default_month = 12
|
|
26
|
+
default_year = now.year - 1
|
|
27
|
+
else:
|
|
28
|
+
default_month = now.month - 1
|
|
29
|
+
default_year = now.year
|
|
30
|
+
|
|
31
|
+
# Parse year input into consistent types
|
|
32
|
+
if year_input is None:
|
|
33
|
+
year_int = default_year
|
|
34
|
+
year_str = str(year_int)
|
|
35
|
+
else:
|
|
36
|
+
year_str = str(year_input).strip()
|
|
37
|
+
year_int = int(year_str)
|
|
38
|
+
|
|
39
|
+
# Parse month input into consistent types
|
|
40
|
+
if month_input is None:
|
|
41
|
+
month_int = default_month
|
|
42
|
+
month_str = datetime.strptime(str(month_int), "%m").strftime("%B")
|
|
43
|
+
else:
|
|
44
|
+
clean_month = str(month_input).strip()
|
|
45
|
+
try:
|
|
46
|
+
month_int = int(clean_month)
|
|
47
|
+
month_str = datetime.strptime(str(month_int), "%m").strftime("%B")
|
|
48
|
+
except ValueError:
|
|
49
|
+
try:
|
|
50
|
+
dt = datetime.strptime(clean_month.capitalize(), "%B")
|
|
51
|
+
except ValueError:
|
|
52
|
+
dt = datetime.strptime(clean_month.capitalize(), "%b")
|
|
53
|
+
month_int = dt.month
|
|
54
|
+
month_str = dt.strftime("%B")
|
|
55
|
+
|
|
56
|
+
# Handle the city's specific upload folder schedule offset
|
|
57
|
+
folder_month_int = (month_int % 12) + 1
|
|
58
|
+
folder_month_str = f"{folder_month_int:02d}"
|
|
59
|
+
folder_year_str = str(year_int + 1) if month_int == 12 else year_str
|
|
60
|
+
|
|
61
|
+
url = f"{BASE_URL}/{folder_year_str}/{folder_month_str}/{month_str}{year_str}.csv"
|
|
62
|
+
return url, month_int, year_int
|
|
63
|
+
|
|
64
|
+
except Exception as e:
|
|
65
|
+
raise ExtractionError(f"Invalid month ('{month_input}') or year ('{year_input}') mapping: {e}") from e
|
|
66
|
+
|
|
67
|
+
|
|
68
|
+
def fetch_crime_data_csv(url: str, keep_raw_csv: bool = False) -> bytes:
|
|
69
|
+
"""Downloads raw network bytes. Standardized on 'keep_raw_csv' to match project rules."""
|
|
70
|
+
headers = {"User-Agent": "Mozilla/5.0"}
|
|
71
|
+
filename = url.split("/")[-1] or "downloaded_data.csv"
|
|
72
|
+
|
|
73
|
+
logger.info(f"Attempting network download from: {url}")
|
|
74
|
+
try:
|
|
75
|
+
response = requests.get(url, headers=headers, timeout=10)
|
|
76
|
+
response.raise_for_status()
|
|
77
|
+
raw_bytes = response.content
|
|
78
|
+
except requests.exceptions.HTTPError as http_err:
|
|
79
|
+
raise ExtractionError(f"HTTP error downloading data asset: {http_err}") from http_err
|
|
80
|
+
except requests.exceptions.RequestException as net_err:
|
|
81
|
+
raise ExtractionError(f"Network transport error reaching server: {net_err}") from net_err
|
|
82
|
+
except Exception as e:
|
|
83
|
+
raise ExtractionError(f"Unexpected extraction execution failure: {e}") from e
|
|
84
|
+
|
|
85
|
+
if keep_raw_csv:
|
|
86
|
+
try:
|
|
87
|
+
with open(filename, "wb") as f:
|
|
88
|
+
f.write(raw_bytes)
|
|
89
|
+
logger.info(f"💾 Raw CSV backup cached locally: {filename}")
|
|
90
|
+
except IOError as io_err:
|
|
91
|
+
logger.warning(f"⚠️ Could not write raw data local cache: {io_err}")
|
|
92
|
+
|
|
93
|
+
return raw_bytes
|
|
@@ -0,0 +1,160 @@
|
|
|
1
|
+
from __future__ import annotations
|
|
2
|
+
|
|
3
|
+
import logging
|
|
4
|
+
from dataclasses import dataclass
|
|
5
|
+
|
|
6
|
+
import geopandas as gpd
|
|
7
|
+
import pandas as pd
|
|
8
|
+
|
|
9
|
+
from geocrime_stl.config import (
|
|
10
|
+
CITY_BNDY_WGS84,
|
|
11
|
+
COLUMN_RENAMES,
|
|
12
|
+
COLUMNS_TO_DROP,
|
|
13
|
+
FINAL_SCHEMA_COLUMNS,
|
|
14
|
+
)
|
|
15
|
+
|
|
16
|
+
logger = logging.getLogger(__name__)
|
|
17
|
+
|
|
18
|
+
|
|
19
|
+
@dataclass(frozen=True)
|
|
20
|
+
class CrimeDataPackage:
|
|
21
|
+
"""Wrapper that bundles structured monthly crime data with its target temporal metadata."""
|
|
22
|
+
|
|
23
|
+
df: pd.DataFrame
|
|
24
|
+
month: int
|
|
25
|
+
year: int
|
|
26
|
+
|
|
27
|
+
|
|
28
|
+
def raw_csv_to_dataframe(csv_bytes: bytes, **kwargs) -> pd.DataFrame:
|
|
29
|
+
"""Converts raw byte stream cleanly into an in-memory Pandas DataFrame.
|
|
30
|
+
|
|
31
|
+
Accepts optional kwargs to pass directly to pd.read_csv if needed down the road.
|
|
32
|
+
"""
|
|
33
|
+
import io
|
|
34
|
+
|
|
35
|
+
try:
|
|
36
|
+
return pd.read_csv(io.BytesIO(csv_bytes), **kwargs)
|
|
37
|
+
except Exception as e:
|
|
38
|
+
raise ValueError(f"Failed to decode raw byte stream to DataFrame: {e}") from e
|
|
39
|
+
|
|
40
|
+
|
|
41
|
+
def clean_and_spatial_clip(
|
|
42
|
+
df: pd.DataFrame, month: int, year: int
|
|
43
|
+
) -> CrimeDataPackage:
|
|
44
|
+
"""Normalizes features, handles type enforcement, and spatially clips observations
|
|
45
|
+
|
|
46
|
+
to St. Louis City limits. Returns a populated CrimeDataPackage wrapper.
|
|
47
|
+
"""
|
|
48
|
+
# Handle the original empty CSV dataset scenario gracefully
|
|
49
|
+
if df.empty:
|
|
50
|
+
logger.warning(
|
|
51
|
+
f"⚠️ Ingested DataFrame was empty. Skipping cleaning pipelines for {month}/{year}."
|
|
52
|
+
)
|
|
53
|
+
return CrimeDataPackage(df=df, month=month, year=year)
|
|
54
|
+
|
|
55
|
+
# Date Alignments & Conversions
|
|
56
|
+
df["IncidentDate"] = pd.to_datetime(
|
|
57
|
+
df["IncidentDate"], format="%m/%d/%Y", exact=False, errors="coerce"
|
|
58
|
+
)
|
|
59
|
+
|
|
60
|
+
# Filter by date alignment (keeps only requested month/year observations)
|
|
61
|
+
df = df[(df["IncidentDate"].dt.month == month) & (df["IncidentDate"].dt.year == year)].copy()
|
|
62
|
+
|
|
63
|
+
# String Timestamp Synthesis
|
|
64
|
+
df["OccurredFromTime"] = df["OccurredFromTime"].fillna("00:00")
|
|
65
|
+
combined_time_str = (
|
|
66
|
+
df["IncidentDate"].dt.strftime("%Y-%m-%d") + " " + df["OccurredFromTime"].astype(str)
|
|
67
|
+
)
|
|
68
|
+
df["date_time"] = pd.to_datetime(combined_time_str, errors="coerce")
|
|
69
|
+
|
|
70
|
+
# Structural Re-mapping & Column Dropping
|
|
71
|
+
df = df.rename(columns=COLUMN_RENAMES)
|
|
72
|
+
df = df.drop(columns=COLUMNS_TO_DROP, errors="ignore")
|
|
73
|
+
|
|
74
|
+
# Data Type Enforcement
|
|
75
|
+
df["district"] = pd.to_numeric(df["district"], errors="coerce")
|
|
76
|
+
df["district"] = df["district"].astype("Int64").astype("string")
|
|
77
|
+
df["inc_#"] = df["inc_#"].astype("string")
|
|
78
|
+
|
|
79
|
+
# Regex Character Stripping (e.g., "G-88" -> 88)
|
|
80
|
+
df["nbhd_num"] = df["nbhd_num"].astype(str).str.extract(r"(\d+)", expand=False)
|
|
81
|
+
df["nbhd_num"] = pd.to_numeric(df["nbhd_num"], errors="coerce").astype("Int64")
|
|
82
|
+
df["off_type"] = df["off_type"].replace("Unspecified", "Other")
|
|
83
|
+
|
|
84
|
+
# Coordinate Sanitization
|
|
85
|
+
df["lon"] = pd.to_numeric(df["lon"], errors="coerce")
|
|
86
|
+
df["lat"] = pd.to_numeric(df["lat"], errors="coerce")
|
|
87
|
+
df = df.dropna(subset=["lon", "lat"]).copy()
|
|
88
|
+
|
|
89
|
+
# Apply Final Schema Alignment from global configuration
|
|
90
|
+
existing_cols = [col for col in FINAL_SCHEMA_COLUMNS if col in df.columns]
|
|
91
|
+
df = df[existing_cols]
|
|
92
|
+
|
|
93
|
+
# Spatial Boundary Clipping Execution
|
|
94
|
+
try:
|
|
95
|
+
clipped_df = clip_df(df)
|
|
96
|
+
return CrimeDataPackage(df=clipped_df, month=month, year=year)
|
|
97
|
+
except Exception as e:
|
|
98
|
+
logger.warning(f"⚠️ Spatial fencing bypassed due to pipeline engine block: {e}")
|
|
99
|
+
logger.info("▶️ Flowing original full metropolitan boundary coordinates forward.")
|
|
100
|
+
return CrimeDataPackage(df=df, month=month, year=year)
|
|
101
|
+
|
|
102
|
+
|
|
103
|
+
def clip_df(df: pd.DataFrame) -> pd.DataFrame:
|
|
104
|
+
"""Restricts point coordinates to the formal polygon limits of St. Louis City
|
|
105
|
+
|
|
106
|
+
using an inner spatial join.
|
|
107
|
+
"""
|
|
108
|
+
if not CITY_BNDY_WGS84.exists():
|
|
109
|
+
raise FileNotFoundError(
|
|
110
|
+
f"Could not locate boundary file asset at designated config path: {CITY_BNDY_WGS84}"
|
|
111
|
+
)
|
|
112
|
+
|
|
113
|
+
# Temporarily switch structure to an explicit GeoDataFrame ('gdf')
|
|
114
|
+
gdf = gpd.GeoDataFrame(
|
|
115
|
+
df, geometry=gpd.points_from_xy(df.lon, df.lat), crs="EPSG:4326"
|
|
116
|
+
)
|
|
117
|
+
|
|
118
|
+
# Ingest the boundary file directly using your pathlib setup
|
|
119
|
+
stl_boundary = gpd.read_file(CITY_BNDY_WGS84).to_crs("EPSG:4326")
|
|
120
|
+
|
|
121
|
+
# Inner join drops any points falling outside the boundary polygon limits
|
|
122
|
+
gdf = gpd.sjoin(gdf, stl_boundary[["geometry"]], how="inner", predicate="within").copy()
|
|
123
|
+
|
|
124
|
+
if "index_right" in gdf.columns:
|
|
125
|
+
gdf = gdf.drop(columns=["index_right"])
|
|
126
|
+
|
|
127
|
+
# Revert back to standard DataFrame structure and throw away spatial metadata
|
|
128
|
+
return pd.DataFrame(gdf.drop(columns=["geometry"])).copy()
|
|
129
|
+
|
|
130
|
+
def run_pipeline(month: int, year: int) -> CrimeDataPackage:
|
|
131
|
+
"""Runs the entire ingestion, cleaning, and spatial-clipping pipeline
|
|
132
|
+
|
|
133
|
+
for a given month and year. Safely catches missing assets gracefully.
|
|
134
|
+
"""
|
|
135
|
+
# Change the import to target the module directly
|
|
136
|
+
from geocrime_stl.etl import extract
|
|
137
|
+
|
|
138
|
+
# Build the target URL
|
|
139
|
+
url, m_int, y_int = extract.construct_url(month_input=month, year_input=year)
|
|
140
|
+
|
|
141
|
+
# Fetch the data stream from SLMPD with built-in safety net
|
|
142
|
+
try:
|
|
143
|
+
raw_bytes = extract.fetch_crime_data_csv(url, keep_raw_csv=False)
|
|
144
|
+
except extract.ExtractionError as e:
|
|
145
|
+
# Catch 404s or network drops gracefully
|
|
146
|
+
logger.warning(
|
|
147
|
+
f"🛑 Unable to retrieve data for {month}/{year}. "
|
|
148
|
+
f"The asset might not be published yet. Details: {e}"
|
|
149
|
+
)
|
|
150
|
+
# Return an empty dataframe package so the user's notebook doesn't crash
|
|
151
|
+
import pandas as pd
|
|
152
|
+
return CrimeDataPackage(df=pd.DataFrame(), month=m_int, year=y_int)
|
|
153
|
+
|
|
154
|
+
# Decode bytes to DataFrame (Only runs if extraction succeeded!)
|
|
155
|
+
raw_df = raw_csv_to_dataframe(raw_bytes)
|
|
156
|
+
|
|
157
|
+
# Clean, type-enforce, and clip to city boundaries
|
|
158
|
+
data_package = clean_and_spatial_clip(raw_df, m_int, y_int)
|
|
159
|
+
|
|
160
|
+
return data_package
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: geocrime-stl
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: An ETL for fetching/cleaning St. Louis crime data sets.
|
|
5
|
+
Requires-Python: >=3.8
|
|
6
|
+
Description-Content-Type: text/markdown
|
|
7
|
+
License-File: LICENSE
|
|
8
|
+
Requires-Dist: geopandas>=1.1.3
|
|
9
|
+
Requires-Dist: matplotlib>=3.10.9
|
|
10
|
+
Requires-Dist: pandas>=3.0.3
|
|
11
|
+
Requires-Dist: requests>=2.34.2
|
|
12
|
+
Dynamic: license-file
|
|
13
|
+
|
|
14
|
+
# geocrime-stl
|
|
15
|
+
|
|
16
|
+
[](https://opensource.org/licenses/MIT)
|
|
17
|
+
[](https://www.python.org/downloads/)
|
|
18
|
+
|
|
19
|
+
A Python utility for cleaning and standardizing St. Louis Metropolitan Police Department (SLMPD) crime data for spatial analysis.
|
|
20
|
+
|
|
21
|
+
## ✨ Features
|
|
22
|
+
|
|
23
|
+
* **End-to-End ETL:** Fetch, clean, and standardize monthly SLMPD crime data files automatically.
|
|
24
|
+
* **Spatial Formats:** Convert cleaned DataFrames seamlessly into GeoJSON, GeoPackage (GPKG), or CSV.
|
|
25
|
+
* **Analysis Ready:** Access built-in methods for minimal data summarization and visualization.
|
|
26
|
+
|
|
27
|
+
## 🌐 Data Source & Disclaimer
|
|
28
|
+
|
|
29
|
+
This utility programmatically fetches publicly available crime data directly from the **[St. Louis Metropolitan Police Department (SLMPD) Stats Page](https://slmpd.org/stats/)**.
|
|
30
|
+
|
|
31
|
+
⚠️ **Data Disclaimer & Limitations:**
|
|
32
|
+
* **Availability:** This tool relies entirely on the upstream availability and hosting structure of the SLMPD website. If their servers are down or their URL layout changes, the extraction pipeline may fail.
|
|
33
|
+
* **Data Integrity & Spatial Boundaries:** The data processed is strictly "as-is" from the published monthly files. To ensure high data quality for spatial analysis, this utility applies two strict filtering constraints:
|
|
34
|
+
1. **Temporal Filtering:** Because the SLMPD frequently updates historical records retroactively, this utility isolates and preserves only the records belonging to the target month.
|
|
35
|
+
2. **Spatial Clipping:** Records containing invalid, malformed, or erroneous latitude/longitude coordinates that plot completely outside the official St. Louis city boundaries are automatically dropped.
|
|
36
|
+
* **Project Status:** This is an independent, open-source utility. It is not affiliated with, endorsed by, or officially maintained by the SLMPD or the City of St. Louis.
|
|
37
|
+
|
|
38
|
+
## 🚀 Quick Start
|
|
39
|
+
|
|
40
|
+
### Install
|
|
41
|
+
``` bash
|
|
42
|
+
pip install geocrime-stl
|
|
43
|
+
```
|
|
44
|
+
### Fetch/Clean/Visualize Data
|
|
45
|
+
```python
|
|
46
|
+
import geocrime_stl as gc
|
|
47
|
+
|
|
48
|
+
data_pkg = gc.run_pipeline(4, 2026)
|
|
49
|
+
|
|
50
|
+
gc.generate_monthly_metrics(data_pkg)
|
|
51
|
+
|
|
52
|
+
gc.plot_monthly_maps(data_pkg)
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
## 🛠️ Local Setup (Alternative)
|
|
56
|
+
|
|
57
|
+
If you want to run the demo notebook locally or explore the source code, you can clone the repository directly:
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
git clone https://github.com/lweber89/geocrime-stl.git
|
|
61
|
+
cd geocrime-stl
|
|
62
|
+
pip install -e .
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
## 📓 Demo & Tutorial
|
|
66
|
+
To see a step-by-step walkthrough of the entire ETL process, check out the interactive Jupyter notebook included in this package:
|
|
67
|
+
|
|
68
|
+
👉 **[View the Demo Notebook](./notebooks/demo.ipynb)**
|
|
69
|
+
|
|
70
|
+
Note: If you have cloned the repository locally, you can open this file using Jupyter Lab or Jupyter Notebook to run the cells yourself.
|
|
71
|
+
|
|
72
|
+
## 🤝 Contributing & Pull Requests
|
|
73
|
+
Thank you for your interest in the project!
|
|
74
|
+
|
|
75
|
+
⚠️ Please Note: This is a personal project I developed and maintain in an effort to better understand the implementation of Python in geospatial data engineering / solutions architecture. I am not accepting pull requests or code contributions at this time.
|
|
76
|
+
|
|
77
|
+
If you find a bug or have an idea, feel free to open an Issue to let me know, or feel free to fork the repository and adapt it for your own personal needs!
|
|
78
|
+
|
|
79
|
+
## 📜 License
|
|
80
|
+
This project is licensed under the MIT License - see the LICENSE file for details.
|
|
81
|
+
|
|
82
|
+
## 🔗 Project Links
|
|
83
|
+
GitHub Repository: https://github.com/lweber89/geocrime-stl
|
|
84
|
+
|
|
85
|
+
PyPI Home: https://pypi.org/project/geocrime-stl/
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
LICENSE
|
|
2
|
+
README.md
|
|
3
|
+
pyproject.toml
|
|
4
|
+
src/geocrime_stl/__init__.py
|
|
5
|
+
src/geocrime_stl/config.py
|
|
6
|
+
src/geocrime_stl.egg-info/PKG-INFO
|
|
7
|
+
src/geocrime_stl.egg-info/SOURCES.txt
|
|
8
|
+
src/geocrime_stl.egg-info/dependency_links.txt
|
|
9
|
+
src/geocrime_stl.egg-info/requires.txt
|
|
10
|
+
src/geocrime_stl.egg-info/top_level.txt
|
|
11
|
+
src/geocrime_stl/etl/extract.py
|
|
12
|
+
src/geocrime_stl/etl/transform.py
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
geocrime_stl
|