ranch-network-wrangler 0.0.1__tar.gz

ranch_network_wrangler-0.0.1/LICENSE

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2022, WSP | Systems Analysis Group, Metropolitan Transportation Commission (MTC), Metropolitan Council, Miami-Dade TPO, Contra Costa Transportation Authority (CCTA), Alameda County Transportation Commission(ACTC), and Bay Area Rapid Transit (BART)
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

ranch_network_wrangler-0.0.1/PKG-INFO

@@ -0,0 +1,160 @@
+Metadata-Version: 2.4
+Name: ranch-network-wrangler
+Version: 0.0.1
+Summary: A Python package for making travel model networks
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/network-wrangler/Ranch
+Project-URL: Repository, https://github.com/network-wrangler/Ranch
+Project-URL: Documentation, https://wsp-sag.github.io/Ranch/
+Classifier: Development Status :: 1 - Planning
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.7
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: docker
+Requires-Dist: numpy
+Requires-Dist: pandas>1.0
+Requires-Dist: geopandas>=0.8.0
+Requires-Dist: networkx
+Requires-Dist: osmnx
+Requires-Dist: partridge>=1.1
+Requires-Dist: peartree
+Requires-Dist: pyproj
+Requires-Dist: scipy
+Requires-Dist: shapely
+Dynamic: license-file
+
+# Ranch
+This is the Python package for building standard roadway and transit networks from scratch. Before becoming `Ranch`, it was also referred to as the `Standard Network Building Pipeline`, which consisted of a series of Python notebooks.
+
+It aims to have the following functionality:
+
+ 1. build OSM-based roadway network in standard [`Network Wrangler`](https://github.com/wsp-sag/network_wrangler) format
+ 2. build GTFS-based transit network in standard [`Network Wrangler`](https://github.com/wsp-sag/network_wrangler) format
+ 3. build travel model centroid connectors
+
+For the latest code developments, check out the `develop` branch.
+
+## Installation
+
+Install the published package with `uv`:
+
+```bash
+uv add ranch-network-wrangler
+```
+
+Or install it with pip:
+
+```bash
+python -m pip install ranch-network-wrangler
+```
+
+The Python import package remains `ranch`.
+
+### Requirements
+- Ranch requires user to install [SharedStreets' Node.js implementation](https://github.com/sharedstreets/sharedstreets-js#sharedstreets-nodejs--javascript), using the Docker installation. User will first install Docker on the machine, and then build SharedStreets image using this [Dockerfile](https://github.com/wsp-sag/Ranch/blob/develop/ranch/Dockerfile). See more instruction below.
+
+- Ranch uses Python 3.7 and above. Runtime requirements are installed with the package. Development dependencies are managed by uv and can be installed with `uv sync`. Install the documentation dependencies as well with `uv sync --group doc`.
+
+  The intallation instructions use the [`conda`](https://conda.io/en/latest/) virtual environment manager and some use the ['git'](https://git-scm.com/downloads) version control software.
+
+### Ranch Installation from GitHub
+If you are managing multiple python versions, we suggest using [`virtualenv`](https://virtualenv.pypa.io/en/latest/) or [`conda`](https://conda.io/en/latest/) virtual environments. All commands should executed in a conda command prompt, not powershell or the system command prompt. Do not add conda to the system path during installation. This may cause problems with other programs that require python 2.7 to be placed in the system path.
+
+Example using a conda environment and using the package manager [pip](https://pip.pypa.io/en/stable/) to install Ranch from the source on GitHub.
+
+#### Basic Installation
+
+```bash
+conda config --add channels conda-forge
+conda create python=3.7 rtree geopandas folium osmnx -n <my_ranch_environment>
+conda activate <my_ranch_environment>
+pip install git+https://github.com/wsp-sag/Ranch@master
+```
+Ranch can be installed in several ways depending on the user's needs. The above installation is the simplest method and is appropriate when the user does not anticipate needing to update Ranch. An update will require rebuilding the ranch environment. Installing from clone is slightly more involved and requires the user to have a git manager on their machine, but permits the user to install Ranch with the -e, edit, option so that their Ranch installation can be updated through pulling new commits from the Ranch repo without a full reinstallation.
+
+Note: if you wanted to install from a specific tag/version number or branch, replace `@master` with `@<branchname>`  or `@tag`
+
+#### From Clone (Recommended for project team)
+If you are going to be working on Ranch locally, you might want to clone it to your local machine and install it from the clone.  The -e will install it in [editable mode](https://pip.pypa.io/en/stable/reference/pip_install/?highlight=editable#editable-installs).
+
+```bash
+conda config --add channels conda-forge
+conda create python=3.7 rtree geopandas folium osmnx -n <my_ranch_environment>
+conda activate <my_ranch_environment>
+git clone -b develop https://github.com/wsp-sag/Ranch
+cd Ranch
+pip install -e .
+```
+
+Note: if you are not part of the project team and want to contribute code bxack to the project, please fork before you clone and then add the original repository to your upstream origin list per [these directions on github](https://help.github.com/en/articles/fork-a-repo).
+
+### Docker & SharedStreets
+Before running Ranch, user need to install SharedStreets via Docker. The steps are as following:
+
+ 1. Install Docker.
+ 2. Open command line prompt, change base directory to the Ranch directory
+ ```bash
+ cd path/to/your/ranch/directory/
+ ```
+ 3. Build SharedStreets image using this [Dockerfile](https://github.com/wsp-sag/Ranch/blob/develop/ranch/Dockerfile), name the local image as `shst`
+ ```bash
+ cd ranch
+ docker build -t shst .
+ ```
+ 4. To test if SharedStreets image is built successfully, user can run the following to start a container node
+ ```bash
+ docker run -it --rm -v /path/to/your/local/directory:/usr/node/ shst:latest /bin/bash
+ ```
+ if succeeded, user can see a node being launched
+
+## Documentation
+
+Documentation is located at [https://wsp-sag.github.io/Ranch/](https://wsp-sag.github.io/Ranch/)
+
+Edit the source of the documentation  in the `/docs` folder.
+
+To build the documentation locally, install its dependencies from the Ranch folder:
+
+```bash
+uv sync --group doc
+```
+
+To build the documentation, navigate to the `/docs` folder and execute the command: `make html`
+
+```bash
+cd docs
+make html
+```
+
+## Usage
+
+To learn basic Ranch functionality, please refer to the following jupyter notebooks in the `/notebooks` directory:  
+
+ - `Ranch-Demo.ipynb`
+
+Jupyter notebooks can be started by activating the Ranch conda environment and typing `jupyter notebook`:
+
+```bash
+conda activate <my_ranch_environment>
+jupyter notebook
+```
+
+## Troubleshooting
+
+**Issue: Conda is unable to install a library or to update to a specific library version**
+Try installing libraries from conda-forge
+
+```bash
+conda install -c conda-forge *library*
+```
+
+**Issue: User does not have permission to install in directories**
+Try running Anaconda Prompt as an administrator.
+
+## Client Contact and Relationship
+Before becoming `Ranch`, the `Standard Network Building Pipeline` consisted of a series of python notebooks, which have been implemented to build networks for regions including Minneapolis (Met Council), Bay Area (MTC), and Southeast Florida (SERPM). Those agencies all had different variations of the Standard Network Building Pipeline notebooks due to many reasons, e.g. steps in the pipeline have been getting revised and methods have been evolving, also agencies required different network features. However, the fundamental steps in the process are the same across regions. To standardize the network building process so that the same basic steps can be applied to any region, `Ranch` was created as a Python package that hosts universal network building steps. Our agency partners so far include Met Council, MTC, Miami-Dade TPO, and BART.

ranch_network_wrangler-0.0.1/README.md

@@ -0,0 +1,130 @@
+# Ranch
+This is the Python package for building standard roadway and transit networks from scratch. Before becoming `Ranch`, it was also referred to as the `Standard Network Building Pipeline`, which consisted of a series of Python notebooks.
+
+It aims to have the following functionality:
+
+ 1. build OSM-based roadway network in standard [`Network Wrangler`](https://github.com/wsp-sag/network_wrangler) format
+ 2. build GTFS-based transit network in standard [`Network Wrangler`](https://github.com/wsp-sag/network_wrangler) format
+ 3. build travel model centroid connectors
+
+For the latest code developments, check out the `develop` branch.
+
+## Installation
+
+Install the published package with `uv`:
+
+```bash
+uv add ranch-network-wrangler
+```
+
+Or install it with pip:
+
+```bash
+python -m pip install ranch-network-wrangler
+```
+
+The Python import package remains `ranch`.
+
+### Requirements
+- Ranch requires user to install [SharedStreets' Node.js implementation](https://github.com/sharedstreets/sharedstreets-js#sharedstreets-nodejs--javascript), using the Docker installation. User will first install Docker on the machine, and then build SharedStreets image using this [Dockerfile](https://github.com/wsp-sag/Ranch/blob/develop/ranch/Dockerfile). See more instruction below.
+
+- Ranch uses Python 3.7 and above. Runtime requirements are installed with the package. Development dependencies are managed by uv and can be installed with `uv sync`. Install the documentation dependencies as well with `uv sync --group doc`.
+
+  The intallation instructions use the [`conda`](https://conda.io/en/latest/) virtual environment manager and some use the ['git'](https://git-scm.com/downloads) version control software.
+
+### Ranch Installation from GitHub
+If you are managing multiple python versions, we suggest using [`virtualenv`](https://virtualenv.pypa.io/en/latest/) or [`conda`](https://conda.io/en/latest/) virtual environments. All commands should executed in a conda command prompt, not powershell or the system command prompt. Do not add conda to the system path during installation. This may cause problems with other programs that require python 2.7 to be placed in the system path.
+
+Example using a conda environment and using the package manager [pip](https://pip.pypa.io/en/stable/) to install Ranch from the source on GitHub.
+
+#### Basic Installation
+
+```bash
+conda config --add channels conda-forge
+conda create python=3.7 rtree geopandas folium osmnx -n <my_ranch_environment>
+conda activate <my_ranch_environment>
+pip install git+https://github.com/wsp-sag/Ranch@master
+```
+Ranch can be installed in several ways depending on the user's needs. The above installation is the simplest method and is appropriate when the user does not anticipate needing to update Ranch. An update will require rebuilding the ranch environment. Installing from clone is slightly more involved and requires the user to have a git manager on their machine, but permits the user to install Ranch with the -e, edit, option so that their Ranch installation can be updated through pulling new commits from the Ranch repo without a full reinstallation.
+
+Note: if you wanted to install from a specific tag/version number or branch, replace `@master` with `@<branchname>`  or `@tag`
+
+#### From Clone (Recommended for project team)
+If you are going to be working on Ranch locally, you might want to clone it to your local machine and install it from the clone.  The -e will install it in [editable mode](https://pip.pypa.io/en/stable/reference/pip_install/?highlight=editable#editable-installs).
+
+```bash
+conda config --add channels conda-forge
+conda create python=3.7 rtree geopandas folium osmnx -n <my_ranch_environment>
+conda activate <my_ranch_environment>
+git clone -b develop https://github.com/wsp-sag/Ranch
+cd Ranch
+pip install -e .
+```
+
+Note: if you are not part of the project team and want to contribute code bxack to the project, please fork before you clone and then add the original repository to your upstream origin list per [these directions on github](https://help.github.com/en/articles/fork-a-repo).
+
+### Docker & SharedStreets
+Before running Ranch, user need to install SharedStreets via Docker. The steps are as following:
+
+ 1. Install Docker.
+ 2. Open command line prompt, change base directory to the Ranch directory
+ ```bash
+ cd path/to/your/ranch/directory/
+ ```
+ 3. Build SharedStreets image using this [Dockerfile](https://github.com/wsp-sag/Ranch/blob/develop/ranch/Dockerfile), name the local image as `shst`
+ ```bash
+ cd ranch
+ docker build -t shst .
+ ```
+ 4. To test if SharedStreets image is built successfully, user can run the following to start a container node
+ ```bash
+ docker run -it --rm -v /path/to/your/local/directory:/usr/node/ shst:latest /bin/bash
+ ```
+ if succeeded, user can see a node being launched
+
+## Documentation
+
+Documentation is located at [https://wsp-sag.github.io/Ranch/](https://wsp-sag.github.io/Ranch/)
+
+Edit the source of the documentation  in the `/docs` folder.
+
+To build the documentation locally, install its dependencies from the Ranch folder:
+
+```bash
+uv sync --group doc
+```
+
+To build the documentation, navigate to the `/docs` folder and execute the command: `make html`
+
+```bash
+cd docs
+make html
+```
+
+## Usage
+
+To learn basic Ranch functionality, please refer to the following jupyter notebooks in the `/notebooks` directory:  
+
+ - `Ranch-Demo.ipynb`
+
+Jupyter notebooks can be started by activating the Ranch conda environment and typing `jupyter notebook`:
+
+```bash
+conda activate <my_ranch_environment>
+jupyter notebook
+```
+
+## Troubleshooting
+
+**Issue: Conda is unable to install a library or to update to a specific library version**
+Try installing libraries from conda-forge
+
+```bash
+conda install -c conda-forge *library*
+```
+
+**Issue: User does not have permission to install in directories**
+Try running Anaconda Prompt as an administrator.
+
+## Client Contact and Relationship
+Before becoming `Ranch`, the `Standard Network Building Pipeline` consisted of a series of python notebooks, which have been implemented to build networks for regions including Minneapolis (Met Council), Bay Area (MTC), and Southeast Florida (SERPM). Those agencies all had different variations of the Standard Network Building Pipeline notebooks due to many reasons, e.g. steps in the pipeline have been getting revised and methods have been evolving, also agencies required different network features. However, the fundamental steps in the process are the same across regions. To standardize the network building process so that the same basic steps can be applied to any region, `Ranch` was created as a Python package that hosts universal network building steps. Our agency partners so far include Met Council, MTC, Miami-Dade TPO, and BART.

ranch_network_wrangler-0.0.1/pyproject.toml

@@ -0,0 +1,70 @@
+[build-system]
+requires = ["setuptools>=77", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "ranch-network-wrangler"
+version = "0.0.1"
+description = "A Python package for making travel model networks"
+readme = "README.md"
+requires-python = ">=3.7"
+license = "Apache-2.0"
+license-files = ["LICENSE"]
+classifiers = [
+    "Development Status :: 1 - Planning",
+    "Natural Language :: English",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.7",
+]
+dependencies = [
+    "docker",
+    "numpy",
+    "pandas>1.0",
+    "geopandas>=0.8.0",
+    "networkx",
+    "osmnx",
+    "partridge>=1.1",
+    "peartree",
+    "pyproj",
+    "scipy",
+    "shapely",
+]
+
+[dependency-groups]
+dev = [
+    "geojson",
+    "jsonschema",
+    "folium",
+    "black",
+    "isort",
+    "pytest",
+    "jupyter",
+    "notebook",
+    "pytest-notebook",
+]
+doc = [
+    "recommonmark",
+    "sphinx",
+    "sphinx-autodoc-typehints",
+    "sphinx-rtd-theme",
+]
+
+[project.urls]
+Homepage = "https://github.com/network-wrangler/Ranch"
+Repository = "https://github.com/network-wrangler/Ranch"
+Documentation = "https://wsp-sag.github.io/Ranch/"
+
+[tool.setuptools]
+packages = ["ranch"]
+include-package-data = true
+
+[tool.uv]
+default-groups = ["dev"]
+
+[[tool.uv.index]]
+name = "testpypi"
+url = "https://test.pypi.org/simple/"
+publish-url = "https://test.pypi.org/legacy/"
+explicit = true

ranch_network_wrangler-0.0.1/ranch/__init__.py

@@ -0,0 +1,34 @@
+__version__ = "0.0.1"
+
+import os
+from datetime import datetime
+
+from .logger import RanchLogger, setupLogging
+from .osm import run_osmnx_extraction
+from .parameters import Parameters
+from .roadway import Roadway
+from .sharedstreets import read_shst_extraction, run_shst_extraction, run_shst_match
+from .transit import Transit
+from .utils import link_df_to_geojson, point_df_to_geojson
+
+__all__ = [
+    "RanchLogger",
+    "setupLogging",
+    "Roadway",
+    "run_shst_extraction",
+    "read_shst_extraction",
+    "run_shst_match",
+    "run_osmnx_extraction",
+    "link_df_to_geojson",
+    "point_df_to_geojson",
+    "Parameters",
+    "Transit",
+]
+
+setupLogging(
+    log_filename=os.path.join(
+        os.getcwd(),
+        "ranch_{}.log".format(datetime.now().strftime("%Y_%m_%d__%H_%M_%S")),
+    ),
+    log_to_console=True,
+)

ranch_network_wrangler-0.0.1/ranch/logger.py

@@ -0,0 +1,47 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Utility functions for logging
+
+import logging
+import sys
+
+__all__ = ["RanchLogger", "setupLogging"]
+
+
+# for all the Wrangler logging needs!
+RanchLogger = logging.getLogger("RanchLogger")
+
+
+def setupLogging(
+    level: int = None, log_filename: str = None, log_to_console: bool = False
+):
+    """
+    Sets up the RanchLogger w.r.t. the debug file location and if logging to console.
+
+    args:
+        level: the level of logging that will be recorded
+        log_filename: the location of the log file that will get created to add the DEBUG log
+        log_to_console: if True, logging will go to the console at INFO level
+    """
+
+    if level is None:
+        RanchLogger.setLevel(logging.DEBUG)
+    else:
+        RanchLogger.setLevel(level)
+
+    FORMAT = logging.Formatter(
+        "%(asctime)-15s %(levelname)s: %(message)s", datefmt="%Y-%m-%d %H:%M:%S,"
+    )
+
+    if log_filename:
+        file_handler = logging.FileHandler(log_filename)
+        file_handler.setLevel(logging.DEBUG)
+        file_handler.setFormatter(FORMAT)
+        RanchLogger.addHandler(file_handler)
+
+    if log_to_console:
+        console_handler = logging.StreamHandler(sys.stdout)
+        console_handler.setLevel(logging.INFO)
+        console_handler.setFormatter(FORMAT)
+        RanchLogger.addHandler(console_handler)

ranch_network_wrangler-0.0.1/ranch/osm.py

@@ -0,0 +1,200 @@
+import json
+import os
+
+import geopandas as gpd
+import networkx as nx
+import osmnx as ox
+import pandas as pd
+from pyproj import CRS
+
+from .logger import RanchLogger
+from .parameters import standard_crs, alt_standard_crs
+from .utils import link_df_to_geojson, point_df_to_geojson
+
+__all__ = ["run_osmnx_extraction"]
+
+
+def run_osmnx_extraction(input_polygon_file: str, output_dir: str):
+    """
+    run osmnx extraction with input polygon
+
+    Args:
+        input_area_polygon_file: input polygon file that defines the extract region
+        output_extraction_file: output file that stores the extraction
+    """
+
+    if not input_polygon_file:
+        msg = "Missing polygon file for sharedstreet extraction."
+        RanchLogger.error(msg)
+        raise ValueError(msg)
+
+    if not output_dir:
+        msg = "Please specify output filename for extraction result."
+        RanchLogger.error(msg)
+        raise ValueError(msg)
+
+    if input_polygon_file:
+        filename, file_extension = os.path.splitext(input_polygon_file)
+        if file_extension in [".shp", ".geojson"]:
+            polygon_gdf = gpd.read_file(input_polygon_file)
+
+        else:
+            msg = "Invalid boundary file, should be .shp or .geojson"
+            RanchLogger.error(msg)
+            raise ValueError(msg)
+
+    # avoid conversion between WGS lat-long and NAD lat-long
+    if polygon_gdf.crs == alt_standard_crs:
+        polygon_gdf.crs = standard_crs
+
+    # convert to lat-long
+    polygon_gdf = polygon_gdf.to_crs(standard_crs)
+
+    boundary = polygon_gdf.geometry.unary_union
+
+    # OSM extraction
+    G_drive = ox.graph_from_polygon(boundary, network_type="all", simplify=False)
+
+    link_gdf = ox.graph_to_gdfs(G_drive, nodes=False, edges=True)
+    node_gdf = ox.graph_to_gdfs(G_drive, nodes=True, edges=False)
+
+    # write out osm extraction
+    link_prop = link_gdf.drop("geometry", axis=1).columns.tolist()
+    link_geojson = link_df_to_geojson(link_gdf, link_prop)
+
+    with open(os.path.join(output_dir, "link.geojson"), "w") as f:
+        json.dump(link_geojson, f)
+
+    node_prop = node_gdf.drop("geometry", axis=1).columns.tolist()
+    node_geojson = point_df_to_geojson(node_gdf, node_prop)
+
+    with open(os.path.join(output_dir, "node.geojson"), "w") as f:
+        json.dump(node_geojson, f)
+
+
+def add_two_way_osm(link_gdf, osmnx_link):
+    """
+    for osm with oneway = False, add the reverse direction to complete
+
+    Parameters
+    ------------
+    osm link from shst extraction, plus shst info
+
+    return
+    ------------
+    complete osm link
+    """
+    osm_link_gdf = link_gdf.copy()
+    osm_link_gdf["wayId"] = osm_link_gdf["wayId"].astype(int)
+    osm_link_gdf.drop("name", axis=1, inplace=True)
+
+    osmnx_link_gdf = osmnx_link.copy()
+
+    osmnx_link_gdf.drop_duplicates(subset=["osmid"], inplace=True)
+    osmnx_link_gdf.drop(["length", "u", "v", "geometry"], axis=1, inplace=True)
+
+    RanchLogger.info(
+        "shst extraction has {} geometries".format(osm_link_gdf.id.nunique())
+    )
+    RanchLogger.info("shst extraction has {} osm links".format(osm_link_gdf.shape[0]))
+
+    osm_link_gdf["u"] = osm_link_gdf.nodeIds.apply(lambda x: int(x[0]))
+    osm_link_gdf["v"] = osm_link_gdf.nodeIds.apply(lambda x: int(x[-1]))
+
+    RanchLogger.info("---joining osm shst with osmnx data---")
+    osm_link_gdf = pd.merge(
+        osm_link_gdf, osmnx_link_gdf, left_on=["wayId"], right_on=["osmid"], how="left"
+    )
+
+    reverse_osm_link_gdf = osm_link_gdf[
+        (osm_link_gdf.oneWay == False)
+        & (osm_link_gdf.forwardReferenceId != osm_link_gdf.backReferenceId)
+        & (osm_link_gdf.u != osm_link_gdf.v)
+    ].copy()
+
+    RanchLogger.info(
+        "shst extraction has {} two-way osm links".format(reverse_osm_link_gdf.shape[0])
+    )
+    RanchLogger.info(
+        "and they are {} geometrys".format(reverse_osm_link_gdf.id.nunique())
+    )
+
+    reverse_osm_link_gdf.rename(
+        columns={
+            "u": "v",
+            "v": "u",
+            "forwardReferenceId": "backReferenceId",
+            "backReferenceId": "forwardReferenceId",
+            "fromIntersectionId": "toIntersectionId",
+            "toIntersectionId": "fromIntersectionId",
+        },
+        inplace=True,
+    )
+
+    reverse_osm_link_gdf["reverse_out"] = 1
+
+    osm_link_gdf = pd.concat(
+        [osm_link_gdf, reverse_osm_link_gdf], sort=False, ignore_index=True
+    )
+
+    osm_link_gdf.rename(
+        columns={
+            "forwardReferenceId": "shstReferenceId",
+            "geometryId": "shstGeometryId",
+        },
+        inplace=True,
+    )
+
+    osm_link_gdf.drop("backReferenceId", axis=1, inplace=True)
+
+    RanchLogger.info(
+        "after join, ther are {} osm links from shst extraction, \
+    out of which there are {} links that do not have osm info, \
+    due to shst extraction (default tile 181224) contains {} osm ids that are not included in latest OSM extraction, \
+    e.g. private streets, closed streets.".format(
+            len(osm_link_gdf),
+            len(osm_link_gdf[osm_link_gdf.osmid.isnull()]),
+            osm_link_gdf[osm_link_gdf.osmid.isnull()].wayId.nunique(),
+        )
+    )
+
+    RanchLogger.info(
+        "after join, there are {} shst referencies".format(
+            osm_link_gdf.groupby(["shstReferenceId", "shstGeometryId"]).count().shape[0]
+        )
+    )
+
+    return osm_link_gdf
+
+
+def highway_attribute_list_to_value(x, highway_to_roadway_dict, roadway_hierarchy_dict):
+    """
+    clean up osm highway, and map to standard roadway
+    """
+    if type(x.highway) == list:
+        value_list = list(set([highway_to_roadway_dict[c] for c in x.highway]))
+        if len(value_list) == 1:
+            if value_list[0] != "":
+                return value_list[0]
+            else:
+                if type(x.roadClass) == list:
+                    return highway_to_roadway_dict[x.roadClass[0].lower()]
+                else:
+                    return highway_to_roadway_dict[x.roadClass.lower()]
+
+        else:
+            ret_val = value_list[0]
+            ret_val_level = roadway_hierarchy_dict[ret_val]
+            for c in value_list:
+                val_level = roadway_hierarchy_dict[c]
+                if val_level < ret_val_level:
+                    ret_val = c
+                    ret_val_level = val_level
+                else:
+                    continue
+            return ret_val
+    else:
+        if x.highway == "":
+            return highway_to_roadway_dict[x.roadClass.lower()]
+        else:
+            return highway_to_roadway_dict[x.highway]