uk_bin_collection 0.81.0__tar.gz → 0.82.1__tar.gz

{uk_bin_collection-0.81.0 → uk_bin_collection-0.82.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: uk_bin_collection
-Version: 0.81.0
+Version: 0.82.1
 Summary: Python Lib to collect UK Bin Data
 Author: Robert Bradley
 Author-email: robbrad182@gmail.com
@@ -126,6 +126,124 @@ Install can be done via
 ## UPRN Finder
 Some councils make use of the UPRN (Unique property reference number) to identify your property. You can find yours [here](https://www.findmyaddress.co.uk/search) or [here](https://uprn.uk/).
 
+---
+## Selenium
+Some councils need Selenium to run the scrape on behalf of Home Assistant. The easiest way to do this is run Selenium as in a Docker container. However you do this the Home Assistant server must be able to reach the Selenium server
+
+### Instructions for Windows, Linux, and Mac
+
+#### Step 1: Install Docker
+
+##### Windows
+
+1.  **Download Docker Desktop for Windows:**
+    
+    *   Go to the Docker website: Docker Desktop for Windows
+    *   Download and install Docker Desktop.
+2.  **Run Docker Desktop:**
+    
+    *   After installation, run Docker Desktop.
+    *   Follow the on-screen instructions to complete the setup.
+    *   Ensure Docker is running by checking the Docker icon in the system tray.
+
+##### Linux
+
+1.  **Install Docker:**
+    
+    *   Open a terminal and run the following commands:
+                       
+        ```bash
+        sudo apt-get update
+        sudo apt-get install \
+            apt-transport-https \
+            ca-certificates \
+            curl \
+            gnupg \
+            lsb-release
+        curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
+        echo \
+          "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
+          $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+        sudo apt-get update
+        sudo apt-get install docker-ce docker-ce-cli containerd.io 
+        ```
+        
+2.  **Start Docker:**
+    
+    *   Run the following command to start Docker:
+                        
+        ```bash
+        sudo systemctl start docker
+        ```
+        
+3.  **Enable Docker to start on boot:**
+    
+    bash
+    
+    Copy code
+    
+    ```bash
+    sudo systemctl enable docker
+    ```
+    
+
+##### Mac
+
+1.  **Download Docker Desktop for Mac:**
+    
+    *   Go to the Docker website: Docker Desktop for Mac
+    *   Download and install Docker Desktop.
+2.  **Run Docker Desktop:**
+    
+    *   After installation, run Docker Desktop.
+    *   Follow the on-screen instructions to complete the setup.
+    *   Ensure Docker is running by checking the Docker icon in the menu bar.
+
+#### Step 2: Pull and Run Selenium Standalone Chrome Docker Image
+
+1.  **Open a terminal or command prompt:**
+    
+2.  **Pull the Selenium Standalone Chrome image:**
+       
+    ```bash
+    docker pull selenium/standalone-chrome
+    ```
+    
+4.  **Run the Selenium Standalone Chrome container:**
+        
+    ```bash
+    docker run -d -p 4444:4444 --name selenium-chrome selenium/standalone-chrome
+    ```
+    
+
+#### Step 3: Test the Selenium Server
+
+1.  **Navigate to the Selenium server URL in your web browser:**
+    *   Open a web browser and go to `http://localhost:4444`
+    *   You should see the Selenium Grid console.
+
+#### Step 4: Supply the Selenium Server URL to UKBinCollectionData
+
+1.  **Find the `UKBinCollectionData` project:**
+    
+    *   Go to the GitHub repository: [UKBinCollectionData](https://github.com/robbrad/UKBinCollectionData)
+2.  **Supply the Selenium Server URL:**
+    
+    *   Typically, the URL will be `http://localhost:4444/wd/hub`
+    *   You might need to update a configuration file or environment variable in the project to use this URL. Check the project's documentation for specific instructions.
+
+### Summary of Commands
+
+**Windows/Linux/Mac:**
+
+```bash
+docker pull selenium/standalone-chrome docker run -d -p 4444:4444 --name selenium-chrome selenium/standalone-chrome
+```
+
+**Selenium Server URL:**
+
+*   `http://localhost:4444/wd/hub`
+
 ---
 
 ## Requesting your council

{uk_bin_collection-0.81.0 → uk_bin_collection-0.82.1}/README.md

@@ -103,6 +103,124 @@ Install can be done via
 ## UPRN Finder
 Some councils make use of the UPRN (Unique property reference number) to identify your property. You can find yours [here](https://www.findmyaddress.co.uk/search) or [here](https://uprn.uk/).
 
+---
+## Selenium
+Some councils need Selenium to run the scrape on behalf of Home Assistant. The easiest way to do this is run Selenium as in a Docker container. However you do this the Home Assistant server must be able to reach the Selenium server
+
+### Instructions for Windows, Linux, and Mac
+
+#### Step 1: Install Docker
+
+##### Windows
+
+1.  **Download Docker Desktop for Windows:**
+    
+    *   Go to the Docker website: Docker Desktop for Windows
+    *   Download and install Docker Desktop.
+2.  **Run Docker Desktop:**
+    
+    *   After installation, run Docker Desktop.
+    *   Follow the on-screen instructions to complete the setup.
+    *   Ensure Docker is running by checking the Docker icon in the system tray.
+
+##### Linux
+
+1.  **Install Docker:**
+    
+    *   Open a terminal and run the following commands:
+                       
+        ```bash
+        sudo apt-get update
+        sudo apt-get install \
+            apt-transport-https \
+            ca-certificates \
+            curl \
+            gnupg \
+            lsb-release
+        curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
+        echo \
+          "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
+          $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+        sudo apt-get update
+        sudo apt-get install docker-ce docker-ce-cli containerd.io 
+        ```
+        
+2.  **Start Docker:**
+    
+    *   Run the following command to start Docker:
+                        
+        ```bash
+        sudo systemctl start docker
+        ```
+        
+3.  **Enable Docker to start on boot:**
+    
+    bash
+    
+    Copy code
+    
+    ```bash
+    sudo systemctl enable docker
+    ```
+    
+
+##### Mac
+
+1.  **Download Docker Desktop for Mac:**
+    
+    *   Go to the Docker website: Docker Desktop for Mac
+    *   Download and install Docker Desktop.
+2.  **Run Docker Desktop:**
+    
+    *   After installation, run Docker Desktop.
+    *   Follow the on-screen instructions to complete the setup.
+    *   Ensure Docker is running by checking the Docker icon in the menu bar.
+
+#### Step 2: Pull and Run Selenium Standalone Chrome Docker Image
+
+1.  **Open a terminal or command prompt:**
+    
+2.  **Pull the Selenium Standalone Chrome image:**
+       
+    ```bash
+    docker pull selenium/standalone-chrome
+    ```
+    
+4.  **Run the Selenium Standalone Chrome container:**
+        
+    ```bash
+    docker run -d -p 4444:4444 --name selenium-chrome selenium/standalone-chrome
+    ```
+    
+
+#### Step 3: Test the Selenium Server
+
+1.  **Navigate to the Selenium server URL in your web browser:**
+    *   Open a web browser and go to `http://localhost:4444`
+    *   You should see the Selenium Grid console.
+
+#### Step 4: Supply the Selenium Server URL to UKBinCollectionData
+
+1.  **Find the `UKBinCollectionData` project:**
+    
+    *   Go to the GitHub repository: [UKBinCollectionData](https://github.com/robbrad/UKBinCollectionData)
+2.  **Supply the Selenium Server URL:**
+    
+    *   Typically, the URL will be `http://localhost:4444/wd/hub`
+    *   You might need to update a configuration file or environment variable in the project to use this URL. Check the project's documentation for specific instructions.
+
+### Summary of Commands
+
+**Windows/Linux/Mac:**
+
+```bash
+docker pull selenium/standalone-chrome docker run -d -p 4444:4444 --name selenium-chrome selenium/standalone-chrome
+```
+
+**Selenium Server URL:**
+
+*   `http://localhost:4444/wd/hub`
+
 ---
 
 ## Requesting your council

{uk_bin_collection-0.81.0 → uk_bin_collection-0.82.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "uk_bin_collection"
-version = "0.81.0"
+version = "0.82.1"
 description = "Python Lib to collect UK Bin Data"
 readme = "README.md"
 authors = ["Robert Bradley <robbrad182@gmail.com>"]

{uk_bin_collection-0.81.0 → uk_bin_collection-0.82.1}/uk_bin_collection/tests/input.json

@@ -283,10 +283,10 @@
         "wiki_name": "Doncaster Council"
     },
     "DorsetCouncil": {
-        "url": "https://gi.dorsetcouncil.gov.uk/mapping/mylocal/viewresults/100040711049",
-        "wiki_command_url_override": "https://gi.dorsetcouncil.gov.uk/mapping/mylocal/viewresults/XXXXXXXX",
-        "wiki_name": "Dorset Council",
-        "wiki_note": "Replace XXXXXXXX with UPRN."
+        "skip_get_url": true,
+        "url": "https://www.dorsetcouncil.gov.uk/",
+        "uprn": "100040711049",
+        "wiki_name": "Dorset Council"
     },
     "DoverDistrictCouncil": {
         "url": "https://collections.dover.gov.uk/property/100060908340",
@@ -573,12 +573,13 @@
         "wiki_note": "Pass the house name/number plus the name of the street with the postcode parameter, wrapped in double quotes.  Check the address in the web site first. This version will only pick the first SHOW button returned by the search or if it is fully unique.  The search is not very predictable (e.g. house number 4 returns 14,24,4,44 etc.)."
     },
     "MidSussexDistrictCouncil": {
-        "house_number": "OAKLANDS",
+        "house_number": "OAKLANDS, OAKLANDS ROAD RH16 1SS",
         "postcode": "RH16 1SS",
         "skip_get_url": true,
         "url": "https://www.midsussex.gov.uk/waste-recycling/bin-collection/",
         "wiki_name": "Mid Sussex District Council",
-        "wiki_note": "Pass the name of the street with the house number parameter, wrapped in double quotes"
+        "wiki_note": "Pass the name of the street with the house number parameter, wrapped in double quotes",
+        "web_driver": "http://selenium:4444"
     },
     "MiltonKeynesCityCouncil": {
         "uprn": "Fullers Slade",

{uk_bin_collection-0.81.0 → uk_bin_collection-0.82.1}/uk_bin_collection/tests/step_defs/test_validate_council.py

@@ -1,13 +1,13 @@
+import json
 import logging
 import traceback
-from typing import Any, Generator, Callable
-import json
-
-import pytest
-from pytest_bdd import scenario, given, when, then, parsers
 from functools import wraps
+from typing import Any, Callable, Generator
 
+import pytest
+from pytest_bdd import given, parsers, scenario, then, when
 from step_helpers import file_handler
+
 from uk_bin_collection.uk_bin_collection import collect_data
 
 logging.basicConfig(level=logging.INFO, format="%(levelname)s - %(message)s")

{uk_bin_collection-0.81.0 → uk_bin_collection-0.82.1}/uk_bin_collection/uk_bin_collection/collect_data.py

@@ -11,8 +11,8 @@ from uk_bin_collection.uk_bin_collection.get_bin_data import (
 _LOGGER = logging.getLogger(__name__)
 
 
-# Dynamically importing the council processor
 def import_council_module(module_name, src_path="councils"):
+    """Dynamically import the council processor module."""
     module_path = os.path.realpath(os.path.join(os.path.dirname(__file__), src_path))
     if module_path not in sys.path:
         sys.path.append(module_path)
@@ -25,6 +25,7 @@ class UKBinCollectionApp:
         self.parsed_args = None
 
     def setup_arg_parser(self):
+        """Setup the argument parser for the script."""
         self.parser = argparse.ArgumentParser(
             description="UK Bin Collection Data Parser"
         )
@@ -38,8 +39,7 @@ class UKBinCollectionApp:
             "-p",
             "--postcode",
             type=str,
-            help="Postcode to parse - should include a space and be wrapped in "
-            "double-quotes",
+            help="Postcode to parse - should include a space and be wrapped in double quotes",
             required=False,
         )
         self.parser.add_argument(
@@ -58,6 +58,7 @@ class UKBinCollectionApp:
         self.parser.add_argument(
             "-w",
             "--web_driver",
+            type=str,
             help="URL for remote Selenium web driver - should be wrapped in double quotes",
             required=False,
         )
@@ -74,7 +75,13 @@ class UKBinCollectionApp:
             help="Should Selenium be headless. Defaults to true. Can be set to false to debug council",
         )
         self.parser.set_defaults(headless=True)
-
+        self.parser.add_argument(
+            "--local_browser",
+            dest="local_browser",
+            action="store_true",
+            help="Should Selenium be run on a remote server or locally. Defaults to false.",
+            required=False,
+        )
         self.parser.add_argument(
             "-d",
             "--dev_mode",
@@ -85,43 +92,37 @@ class UKBinCollectionApp:
         self.parsed_args = None
 
     def set_args(self, args):
+        """Parse the arguments from the command line."""
         self.parsed_args = self.parser.parse_args(args)
 
     def run(self):
-        council_module_str = self.parsed_args.module
-        council_module = import_council_module(council_module_str)
-        address_url = self.parsed_args.URL
-        postcode = self.parsed_args.postcode
-        paon = self.parsed_args.number
-        uprn = self.parsed_args.uprn
-        skip_get_url = self.parsed_args.skip_get_url
-        web_driver = self.parsed_args.web_driver
-        headless = self.parsed_args.headless
-        dev_mode = self.parsed_args.dev_mode
-
+        """Run the application with the provided arguments."""
+        council_module = import_council_module(self.parsed_args.module)
         return self.client_code(
             council_module.CouncilClass(),
-            address_url,
-            postcode=postcode,
-            paon=paon,
-            uprn=uprn,
-            skip_get_url=skip_get_url,
-            web_driver=web_driver,
-            headless=headless,
-            dev_mode=dev_mode,
-            council_module_str=council_module_str,
+            self.parsed_args.URL,
+            postcode=self.parsed_args.postcode,
+            paon=self.parsed_args.number,
+            uprn=self.parsed_args.uprn,
+            skip_get_url=self.parsed_args.skip_get_url,
+            web_driver=self.parsed_args.web_driver,
+            headless=self.parsed_args.headless,
+            local_browser=self.parsed_args.local_browser,
+            dev_mode=self.parsed_args.dev_mode,
+            council_module_str=self.parsed_args.module,
         )
 
-    def client_code(self, get_bin_data_class, address_url, **kwargs) -> None:
+    def client_code(self, get_bin_data_class, address_url, **kwargs):
         """
-        The client code calls the template method to execute the algorithm. Client
-        code does not have to know the concrete class of an object it works with,
-        as long as it works with objects through the interface of their base class.
+        Call the template method to execute the algorithm. Client code does not need
+        to know the concrete class of an object it works with, as long as it works with
+        objects through the interface of their base class.
         """
         return get_bin_data_class.template_method(address_url, **kwargs)
 
 
 def run():
+    """Set up logging and run the application."""
     global _LOGGER
     _LOGGER = setup_logging(LOGGING_CONFIG, None)
     app = UKBinCollectionApp()

{uk_bin_collection-0.81.0 → uk_bin_collection-0.82.1}/uk_bin_collection/uk_bin_collection/common.py

@@ -1,18 +1,19 @@
 import calendar
-import holidays
 import json
 import os
-import pandas as pd
 import re
-import requests
 from datetime import datetime, timedelta
-from dateutil.parser import parse
 from enum import Enum
+
+import holidays
+import pandas as pd
+import requests
+from dateutil.parser import parse
 from selenium import webdriver
-from selenium.webdriver.chrome.service import Service as ChromeService
-from webdriver_manager.chrome import ChromeDriverManager
 from selenium.common.exceptions import WebDriverException
+from selenium.webdriver.chrome.service import Service as ChromeService
 from urllib3.exceptions import MaxRetryError
+from webdriver_manager.chrome import ChromeDriverManager
 
 date_format = "%d/%m/%Y"
 days_of_week = {

uk_bin_collection-0.82.1/uk_bin_collection/uk_bin_collection/councils/DorsetCouncil.py

@@ -0,0 +1,42 @@
+from bs4 import BeautifulSoup, element
+from uk_bin_collection.uk_bin_collection.common import *
+from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
+
+
+# import the wonderful Beautiful Soup and the URL grabber
+class CouncilClass(AbstractGetBinDataClass):
+    """
+    Concrete classes have to implement all abstract operations of the
+    base class. They can also override some operations with a default
+    implementation.
+    """
+
+    def parse_data(self, page: str, **kwargs) -> dict:
+        data = {"bins": []}
+        collections = []
+        url_base = "https://geoapi.dorsetcouncil.gov.uk/v1/services/"
+        url_types = ["recyclingday", "refuseday", "foodwasteday", "gardenwasteday"]
+
+        uprn = kwargs.get("uprn")
+        # Check the UPRN is valid
+        check_uprn(uprn)
+
+        for url_type in url_types:
+            response = requests.get(f"{url_base}{url_type}/{uprn}")
+            if response.status_code != 200:
+                raise ConnectionError(f"Could not fetch from {url_type} endpoint")
+            json_data = response.json()["values"][0]
+            collections.append((f"{json_data.get('type')} bin", datetime.strptime(json_data.get('dateNextVisit'), "%Y-%m-%d")))
+
+        # Sort the text and date elements by date
+        ordered_data = sorted(collections, key=lambda x: x[1])
+
+        # Put the elements into the dictionary
+        for item in ordered_data:
+            dict_data = {
+                "type": item[0],
+                "collectionDate": item[1].strftime(date_format),
+            }
+            data["bins"].append(dict_data)
+
+        return data

uk_bin_collection-0.82.1/uk_bin_collection/uk_bin_collection/councils/MidSussexDistrictCouncil.py

@@ -0,0 +1,168 @@
+import logging
+import time
+
+from bs4 import BeautifulSoup
+from selenium.webdriver.common.by import By
+from selenium.webdriver.support import expected_conditions as EC
+from selenium.webdriver.support.ui import Select
+from selenium.webdriver.support.wait import WebDriverWait
+from selenium.webdriver.common.keys import Keys
+
+from uk_bin_collection.uk_bin_collection.common import *
+from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
+
+# Set up logging
+logging.basicConfig(
+    level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s"
+)
+
+
+class CouncilClass(AbstractGetBinDataClass):
+    """
+    Concrete class for Mid-Sussex District Council implementing AbstractGetBinDataClass.
+    """
+
+    def parse_data(self, page: str, **kwargs) -> dict:
+        driver = None
+        try:
+            data = {"bins": []}
+            collections = []
+            user_paon = kwargs.get("paon")
+            user_postcode = kwargs.get("postcode")
+            web_driver = kwargs.get("web_driver")
+            headless = kwargs.get("headless")
+            check_postcode(user_postcode)
+
+            # Create Selenium webdriver
+            driver = create_webdriver(web_driver, headless, None, __name__)
+
+            driver.get("https://www.midsussex.gov.uk/waste-recycling/bin-collection/")
+            wait = WebDriverWait(driver, 60)
+
+            try:
+                logging.info("Cookies")
+                cookie_window = wait.until(
+                    EC.presence_of_element_located(
+                        (By.XPATH, '//div[@id="ccc-content"]')
+                    )
+                )
+                time.sleep(2)
+                accept_cookies = WebDriverWait(driver, timeout=10).until(
+                    EC.presence_of_element_located(
+                        (By.XPATH, '//button[@id="ccc-recommended-settings"]')
+                    )
+                )
+                accept_cookies.send_keys(Keys.ENTER)
+                accept_cookies.click()
+                accept_cookies_close = WebDriverWait(driver, timeout=10).until(
+                    EC.presence_of_element_located(
+                        (By.XPATH, '//button[@id="ccc-close"]')
+                    )
+                )
+                accept_cookies_close.send_keys(Keys.ENTER)
+                accept_cookies_close.click()
+            except:
+                print(
+                    "Accept cookies banner not found or clickable within the specified time."
+                )
+                pass
+
+            def click_element(by, value):
+                element = wait.until(EC.element_to_be_clickable((by, value)))
+                driver.execute_script("arguments[0].scrollIntoView();", element)
+                element.click()
+
+            logging.info("Entering postcode")
+            input_element_postcode = wait.until(
+                EC.presence_of_element_located(
+                    (By.XPATH, '//input[@id="PostCodeStep_strAddressSearch"]')
+                )
+            )
+
+            input_element_postcode.send_keys(user_postcode)
+
+            logging.info("Entering postcode")
+
+            click_element(By.XPATH, "//button[contains(text(), 'Search')]")
+
+            logging.info("Selecting address")
+            dropdown = wait.until(
+                EC.element_to_be_clickable((By.ID, "StrAddressSelect"))
+            )
+
+            dropdown_options = wait.until(
+                EC.presence_of_element_located(
+                    (By.XPATH, "//select[@id='StrAddressSelect']/option")
+                )
+            )
+            dropdownSelect = Select(dropdown)
+            dropdownSelect.select_by_visible_text(str(user_paon))
+
+            click_element(By.XPATH, "//button[contains(text(), 'Select')]")
+
+            logging.info("Waiting for bin schedule")
+            bin_results = wait.until(
+                EC.presence_of_element_located(
+                    (By.XPATH, f"//strong[contains(text(), '{user_paon}')]")
+                )
+            )
+
+            # Make a BS4 object
+            soup = BeautifulSoup(driver.page_source, features="html.parser")
+
+            # Find the table with bin collection data
+            table = soup.find("table", class_="collDates")
+            if table:
+                rows = table.find_all("tr")[1:]  # Skip the header row
+            else:
+                rows = []
+
+            # Extract the data from the table and format it according to the JSON schema
+            bins = []
+            date_pattern = re.compile(r"(\d{2}) (\w+) (\d{4})")
+
+            for row in rows:
+                cols = row.find_all("td")
+                if len(cols) < 3:
+                    print("Skipping row, not enough columns:", row)
+                    continue  # Skip rows that do not have enough columns
+
+                collection_type = cols[1].text.strip()
+                collection_date = cols[2].text.strip()
+
+                # Convert the collection date to the required format
+                date_match = date_pattern.search(collection_date)
+                if date_match:
+                    day, month, year = date_match.groups()
+                    month_number = {
+                        "January": "01",
+                        "February": "02",
+                        "March": "03",
+                        "April": "04",
+                        "May": "05",
+                        "June": "06",
+                        "July": "07",
+                        "August": "08",
+                        "September": "09",
+                        "October": "10",
+                        "November": "11",
+                        "December": "12",
+                    }.get(month, "00")
+
+                    formatted_date = f"{day}/{month_number}/{year}"
+                    bins.append(
+                        {"type": collection_type, "collectionDate": formatted_date}
+                    )
+                else:
+                    print("Date pattern not found in:", collection_date)
+
+            # Create the final JSON structure
+            bin_data = {"bins": bins}
+            return bin_data
+        except Exception as e:
+            logging.error(f"An error occurred: {e}")
+            raise
+
+        finally:
+            if driver:
+                driver.quit()

{uk_bin_collection-0.81.0 → uk_bin_collection-0.82.1}/uk_bin_collection/uk_bin_collection/councils/ThreeRiversDistrictCouncil.py

@@ -30,7 +30,7 @@ class CouncilClass(AbstractGetBinDataClass):
 
             # Create Selenium webdriver
             driver = create_webdriver(web_driver, headless, None, __name__)
-            if headless:
+            if not headless:
                 driver.set_window_size(1920, 1080)
 
             driver.get(