requestium 0.4.0__tar.gz → 0.5.1__tar.gz

{requestium-0.4.0/requestium.egg-info → requestium-0.5.1}/PKG-INFO

@@ -1,25 +1,85 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: requestium
-Version: 0.4.0
-Summary: Adds a Selenium webdriver and parsel's parser to a request's Session object, and makes switching between them seamless. Handles cookie, proxy and header transfer.
-Home-page: https://github.com/tryolabs/requestium
-Author: Joaquin Alori
-Author-email: joaquin@tryolabs.com
-License: MIT
-Classifier: Development Status :: 3 - Alpha
+Version: 0.5.1
+Author-email: Joaquin Alori <joaquin@tryolabs.com>
+Maintainer-email: Judson Neer <jkudson.neer@gmail.com>
+License: BSD 3-Clause License
+        
+        Copyright (c) 2017, Tryolabs
+        All rights reserved.
+        
+        Redistribution and use in source and binary forms, with or without
+        modification, are permitted provided that the following conditions are met:
+        
+        * Redistributions of source code must retain the above copyright notice, this
+          list of conditions and the following disclaimer.
+        
+        * Redistributions in binary form must reproduce the above copyright notice,
+          this list of conditions and the following disclaimer in the documentation
+          and/or other materials provided with the distribution.
+        
+        * Neither the name of the copyright holder nor the names of its
+          contributors may be used to endorse or promote products derived from
+          this software without specific prior written permission.
+        
+        THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+        AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+        IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+        DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+        FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+        DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+        SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+        CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+        OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+        OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+        
+Project-URL: source, https://github.com/tryolabs/requestium
+Project-URL: download, https://pypi.org/project/requestium/#files
+Project-URL: issues, https://github.com/tryolabs/requestium/issues
+Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
 Classifier: Natural Language :: English
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Programming Language :: Python :: 2
-Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Operating System :: POSIX
+Classifier: Operating System :: OS Independent
+Classifier: Environment :: Web Environment
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: parsel>=1.7
+Requires-Dist: requests>=2.27
+Requires-Dist: selenium>=4.32
+Requires-Dist: tldextract>=5.3
+Provides-Extra: dev
+Requires-Dist: bandit[sarif]==1.9.2; extra == "dev"
+Requires-Dist: mypy==1.19.1; extra == "dev"
+Requires-Dist: pre-commit~=4.5.0; extra == "dev"
+Requires-Dist: pytest-cov~=7.0.0; extra == "dev"
+Requires-Dist: pytest-xdist~=3.8.0; extra == "dev"
+Requires-Dist: pytest~=9.0.0; extra == "dev"
+Requires-Dist: ruff==0.14.11; extra == "dev"
+Requires-Dist: ssort==0.16.0; extra == "dev"
+Requires-Dist: types-requests==2.32.4.20260107; extra == "dev"
+Dynamic: license-file
 
 ![Requestium](https://user-images.githubusercontent.com/14966348/32966130-8bb15b00-cbb7-11e7-9faf-85963ec5bd82.png)
 ========
 
-[![Build Status](https://travis-ci.org/tryolabs/requestium.svg?branch=master)](https://travis-ci.org/tryolabs/requestium)
 [![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)
+[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/requestium)](https://pypi.org/project/requestium/)
+[![PyPI](https://img.shields.io/pypi/v/requestium?color=blue)](https://pypi.org/project/requestium/)
+[![Coverage](https://raw.githubusercontent.com/tryolabs/requestium/python-coverage-comment-action-data/badge.svg)](https://htmlpreview.github.io/?https://github.com/tryolabs/requestium/blob/python-coverage-comment-action-data/htmlcov/index.html)
+[![Python checks](https://github.com/tryolabs/requestium/actions/workflows/python.yml/badge.svg)](https://github.com/tryolabs/requestium/actions/workflows/python.yml)
 
 Requestium is a Python library that merges the power of [Requests](https://github.com/requests/requests), [Selenium](https://github.com/SeleniumHQ/selenium), and [Parsel](https://github.com/scrapy/parsel) into a single integrated tool for automatizing web actions.
 
@@ -53,7 +113,7 @@ options = {'arguments': ['headless']}
 s = Session(webdriver_path='./chromedriver', default_timeout=15, webdriver_options=options)
 ```
 
-Since headless mode is common, there's a shortcut for it by pecifying `headless=True`.
+Since headless mode is common, there's a shortcut for it by specifying `headless=True`.
 
 ```python
 from requestium import Session, Keys

{requestium-0.4.0 → requestium-0.5.1}/README.md

@@ -1,8 +1,11 @@
 ![Requestium](https://user-images.githubusercontent.com/14966348/32966130-8bb15b00-cbb7-11e7-9faf-85963ec5bd82.png)
 ========
 
-[![Build Status](https://travis-ci.org/tryolabs/requestium.svg?branch=master)](https://travis-ci.org/tryolabs/requestium)
 [![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)
+[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/requestium)](https://pypi.org/project/requestium/)
+[![PyPI](https://img.shields.io/pypi/v/requestium?color=blue)](https://pypi.org/project/requestium/)
+[![Coverage](https://raw.githubusercontent.com/tryolabs/requestium/python-coverage-comment-action-data/badge.svg)](https://htmlpreview.github.io/?https://github.com/tryolabs/requestium/blob/python-coverage-comment-action-data/htmlcov/index.html)
+[![Python checks](https://github.com/tryolabs/requestium/actions/workflows/python.yml/badge.svg)](https://github.com/tryolabs/requestium/actions/workflows/python.yml)
 
 Requestium is a Python library that merges the power of [Requests](https://github.com/requests/requests), [Selenium](https://github.com/SeleniumHQ/selenium), and [Parsel](https://github.com/scrapy/parsel) into a single integrated tool for automatizing web actions.
 
@@ -36,7 +39,7 @@ options = {'arguments': ['headless']}
 s = Session(webdriver_path='./chromedriver', default_timeout=15, webdriver_options=options)
 ```
 
-Since headless mode is common, there's a shortcut for it by pecifying `headless=True`.
+Since headless mode is common, there's a shortcut for it by specifying `headless=True`.
 
 ```python
 from requestium import Session, Keys

requestium-0.5.1/pyproject.toml

@@ -0,0 +1,145 @@
+[project]
+name = "requestium"
+version = "0.5.1"
+readme = { file = "README.md", content-type = "text/markdown" }
+requires-python = ">=3.10"
+license = { file = "LICENSE" }
+authors = [
+    { name = "Joaquin Alori", email = "joaquin@tryolabs.com" }
+]
+maintainers = [
+    { name = "Judson Neer", email = "jkudson.neer@gmail.com" },
+]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "License :: OSI Approved :: BSD License",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Programming Language :: Python :: 3.14",
+  "Natural Language :: English",
+  "Operating System :: MacOS",
+  "Operating System :: Microsoft :: Windows",
+  "Operating System :: POSIX",
+  "Operating System :: OS Independent",
+  "Environment :: Web Environment",
+  "Programming Language :: Python :: Implementation :: CPython",
+  "Topic :: Software Development :: Libraries :: Python Modules",
+  "Topic :: Software Development :: Testing",
+]
+dependencies = [
+    "parsel>=1.7",
+    "requests>=2.27",
+    "selenium>=4.32",
+    "tldextract>=5.3",
+]
+
+[project.urls]
+source = "https://github.com/tryolabs/requestium"
+download = "https://pypi.org/project/requestium/#files"
+issues = "https://github.com/tryolabs/requestium/issues"
+
+[project.optional-dependencies]
+dev = [
+    "bandit[sarif]==1.9.2",
+    "mypy==1.19.1",
+    "pre-commit~=4.5.0",
+    "pytest-cov~=7.0.0",
+    "pytest-xdist~=3.8.0",
+    "pytest~=9.0.0",
+    "ruff==0.14.11",
+    "ssort==0.16.0",
+    "types-requests==2.32.4.20260107",
+]
+
+[tool]
+
+[tool.ruff]
+line-length = 160
+include = [
+    "requestium/**/*.py",
+    "tests/**/*.py",
+]
+
+[tool.ruff.lint]
+extend-select = [
+    "A",  # flake8-builtins (A)
+    "ANN",  # flake8-annotations (ANN)
+    "ARG",  # flake8-unused-arguments (ARG)
+    "B",  # flake8-bugbear (B)
+    "BLE",  # flake8-blind-except (BLE)
+    "C4",  # flake8-comprehensions (C4)
+    "C90",  # mccabe (C90)
+    "D",  # pydocstyle (D)
+    "DTZ",  # flake8-datetimez (DTZ)
+    "E",  # pycodestyle-error (E)
+    "EM",  # flake8-errmsg (EM)
+    "EXE",  # flake8-executable (EXE)
+    "F",  # Pyflakes (F)
+    "FA",  # flake8-future-annotations (FA)
+    "FBT",  # flake8-boolean-trap (FBT)
+    "FIX",  # flake8-fixme (FIX)
+    "G",  # flake8-logging-format (G)
+    "I",  # isort (I)
+    "ICN",  # flake8-import-conventions (ICN)
+    "INP",  # flake8-no-pep420 (INP)
+    "INT",  # flake8-gettext (INT)
+    "ISC",  # flake8-implicit-str-concat (ISC)
+    "LOG",  # flake8-logging (LOG)
+    "N",  # pep8-naming (N)
+    "PIE",  # flake8-pie (PIE)
+    "PL",  # Pylint (PL)
+    "PT",  # flake8-pytest-style (PT)
+    "PTH",  # flake8-use-pathlib (PTH)
+    "Q",  # flake8-quotes (Q)
+    "RET",  # flake8-return (RET)
+    "RSE",  # flake8-raise (RSE)
+    "RUF",  # Ruff-specific rules (RUF)
+    "SIM",  # flake8-simplify (SIM)
+    "SLF",  # flake8-self (SLF)
+    "T10",  # flake8-debugger (T10)
+    "T20",  # flake8-print (T20)
+    "TC",  # flake8-type-checking (TC)
+    "TD",  # flake8-todos (TD)
+    "TID",  # flake8-tidy-imports (TID)
+    "UP",  # pyupgrade (UP)
+    "W",  # pycodestyle-warning (W)
+    "YTT",  # flake8-2020 (YTT)
+]
+ignore = [
+    "ANN002",  # missing-type-args (ANN002)
+    "ANN003",  # missing-type-kwargs (ANN003)
+    "D100",  # undocumented-public-module (D100)
+    "D102",  # undocumented-public-method (D102)
+    "D103",  # undocumented-public-function (D103)
+    "D104",  # undocumented-public-package (D104)
+    "D107",  # undocumented-public-init (D107)
+    "D203",  # incorrect-blank-line-before-class (D203)
+    "D212",  # multi-line-summary-first-line (D212)
+    "FIX002",  # line-contains-todo (FIX002)
+]
+
+[tool.ruff.lint.per-file-ignores]
+"**/{tests,docs}/*" = [
+    "SLF001",  # private-member-access (SLF001)
+    "PLR2004",  # magic-value-comparison (PLR2004)
+]
+
+[tool.pytest]
+addopts = ["-n", "auto", "--cov=requestium", "--no-cov-on-fail"]
+testpaths = [
+    "tests",
+]
+
+[tool.coverage]
+
+[tool.coverage.run]
+branch = true
+relative_files = true
+
+[tool.coverage.report]
+exclude_also = [
+    "logger\\.",
+]

{requestium-0.4.0 → requestium-0.5.1}/requestium/__init__.py

@@ -1,4 +1,5 @@
 from selenium.common import exceptions  # noqa: F401
+from selenium.webdriver.common.by import By  # noqa: F401
 from selenium.webdriver.common.keys import Keys  # noqa: F401
 from selenium.webdriver.support.ui import Select  # noqa: F401
 

requestium-0.5.1/requestium/requestium.py

@@ -0,0 +1,6 @@
+from .requestium_mixin import (  # noqa: F401
+    DriverMixin,
+    _ensure_click,
+)
+from .requestium_response import RequestiumResponse  # noqa: F401
+from .requestium_session import Session  # noqa: F401

requestium-0.5.1/requestium/requestium_mixin.py

@@ -0,0 +1,255 @@
+from __future__ import annotations
+
+import functools
+import time
+import warnings
+from typing import TYPE_CHECKING, Any
+
+import tldextract
+from parsel.selector import Selector, SelectorList
+from selenium.common.exceptions import NoSuchWindowException, WebDriverException
+from selenium.webdriver.common.by import By, ByType
+from selenium.webdriver.remote.webdriver import WebDriver as RemoteWebDriver
+from selenium.webdriver.support import expected_conditions
+from selenium.webdriver.support.ui import WebDriverWait
+
+if TYPE_CHECKING:
+    from selenium.webdriver.remote.webelement import WebElement
+
+
+DEFAULT_TIMEOUT: float = 0.5
+
+
+def _ensure_click(self: WebElement) -> None:
+    """
+    Ensure a click gets made, because Selenium can be a bit buggy about clicks.
+
+    This method gets added to the selenium element returned in '__ensure_element_by_xpath'.
+    We should probably add it to more selenium methods, such as all the 'find**' methods though.
+
+    I wrote this method out of frustration with chromedriver and its problems with clicking
+    items that need to be scrolled to in order to be clickable. In '__ensure_element_by_xpath' we
+    scroll to the item before returning it, but chrome has some problems if it doesn't get some
+    time to scroll to the item. This method ensures chromes gets enough time to scroll to the item
+    before clicking it. I tried SEVERAL more 'correct' methods to get around this, but none of them
+    worked 100% of the time (waiting for the element to be 'clickable' does not work).
+    """
+    # We ensure the element is scrolled into the middle of the viewport to ensure that
+    # it is clickable. There are two main ways an element may not be clickable:
+    #   - It is outside of the viewport
+    #   - It is under a banner or toolbar
+    # This script solves both cases
+    script = (
+        "var viewPortHeight = Math.max("
+        "document.documentElement.clientHeight, window.innerHeight || 0);"
+        "var elementTop = arguments[0].getBoundingClientRect().top;"
+        "window.scrollBy(0, elementTop-(viewPortHeight/2));"
+    )
+    self.parent.execute_script(script, self)  # parent = the webdriver
+
+    exception_message = ""
+    for _ in range(10):
+        try:
+            self.click()
+            return
+        except WebDriverException as e:
+            exception_message = str(e)
+            time.sleep(0.2)
+    msg = f"Couldn't click item after trying 10 times, got error message: \n{exception_message}"
+    raise WebDriverException(msg)
+
+
+class DriverMixin(RemoteWebDriver):
+    """Provides helper methods to our driver classes."""
+
+    def __init__(self, *args, **kwargs) -> None:
+        self.default_timeout = kwargs.pop("default_timeout", DEFAULT_TIMEOUT)
+        super().__init__(*args, **kwargs)
+
+    def try_add_cookie(self, cookie: dict[str, Any]) -> bool:
+        """
+        Attempt to add the cookie.
+
+        Suppress any errors, and simply detect success or failure.
+        """
+        try:
+            self.add_cookie(cookie)
+        except WebDriverException as e:
+            if e.msg and not e.msg.__contains__("Couldn't add the following cookie to the webdriver"):
+                raise WebDriverException from e
+        return self.is_cookie_in_driver(cookie)
+
+    def ensure_add_cookie(self, cookie: dict[str, Any], override_domain: str | None = None) -> None:
+        """
+        Add a cookie to the driver and check to ensure it has been added.
+
+        Selenium needs the driver to be currently at the domain of the cookie
+        before allowing you to add it, so we need to get through this limitation.
+
+        The cookie parameter is a dict which must contain the keys (name, value, domain) and
+        may contain the keys (path, expiry).
+
+        We first check that we aren't currently in the cookie's domain, if we aren't, we GET
+        the cookie's domain and then add the cookies to the driver.
+
+        We can override the cookie's domain using 'override_domain'. The use for this
+        parameter is that sometimes GETting the cookie's domain redirects you to a different
+        sub domain, and therefore adding the cookie fails. So sometimes the user may
+        need to override the cookie's domain to a less strict one, Eg.: 'site.com' instead of
+        'home.site.com', in this way even if the site redirects us to a subdomain, the cookie will
+        stick. If you set the domain to '', the cookie gets added with whatever domain the browser
+        is currently at (at least in chrome it does), so this ensures the cookie gets added.
+
+        It also retries adding the cookie with a more permissive domain if it fails in the first
+        try, and raises an exception if that fails. The standard selenium behaviour in this case
+        was to not do anything, which was very hard to debug.
+        """
+        if override_domain:
+            cookie["domain"] = override_domain
+
+        cookie_domain = cookie["domain"] if cookie["domain"][0] != "." else cookie["domain"][1:]
+        try:
+            browser_domain = tldextract.extract(self.current_url).fqdn
+        except (AttributeError, NoSuchWindowException):
+            browser_domain = ""
+        if cookie_domain not in browser_domain:
+            # TODO @joaqo: Check if hardcoding 'http' causes trouble.
+            # https://github.com/tryolabs/requestium/issues/97
+            # Consider using a new proxy for this next request to not cause an anomalous
+            # request. This way their server sees our ip address as continuously having the
+            # same cookies and not have a request mid-session with no cookies
+            self.get("http://" + cookie_domain)
+
+        cookie_added = self.try_add_cookie(cookie)
+
+        # If we fail adding the cookie, retry with a more permissive domain
+        if not cookie_added:
+            cookie["domain"] = tldextract.extract(cookie["domain"]).top_domain_under_public_suffix
+            cookie_added = self.try_add_cookie(cookie)
+            if not cookie_added:
+                msg = f"Couldn't add the following cookie to the webdriver: {cookie}"
+                raise WebDriverException(msg)
+
+    def is_cookie_in_driver(self, cookie: dict[str, Any]) -> bool:
+        """
+        We check that the cookie is correctly added to the driver.
+
+        We only compare name, value and domain, as the rest can produce false negatives.
+        We are a bit lenient when comparing domains.
+        """
+        for driver_cookie in self.get_cookies():
+            name_matches = cookie["name"] == driver_cookie["name"]
+            value_matches = cookie["value"] == driver_cookie["value"]
+            domain_matches = driver_cookie["domain"] in (cookie["domain"], "." + cookie["domain"])
+            if name_matches and value_matches and domain_matches:
+                return True
+        return False
+
+    def ensure_element_by_id(self, selector: str, state: str | None = "present", timeout: float | None = None) -> WebElement | None:
+        return self.ensure_element(By.ID, selector, state, timeout)
+
+    def ensure_element_by_name(self, selector: str, state: str | None = "present", timeout: float | None = None) -> WebElement | None:
+        return self.ensure_element(By.NAME, selector, state, timeout)
+
+    def ensure_element_by_xpath(self, selector: str, state: str | None = "present", timeout: float | None = None) -> WebElement | None:
+        return self.ensure_element(By.XPATH, selector, state, timeout)
+
+    def ensure_element_by_link_text(self, selector: str, state: str | None = "present", timeout: float | None = None) -> WebElement | None:
+        return self.ensure_element(By.LINK_TEXT, selector, state, timeout)
+
+    def ensure_element_by_partial_link_text(self, selector: str, state: str | None = "present", timeout: float | None = None) -> WebElement | None:
+        return self.ensure_element(By.PARTIAL_LINK_TEXT, selector, state, timeout)
+
+    def ensure_element_by_tag_name(self, selector: str, state: str | None = "present", timeout: float | None = None) -> WebElement | None:
+        return self.ensure_element(By.TAG_NAME, selector, state, timeout)
+
+    def ensure_element_by_class_name(self, selector: str, state: str | None = "present", timeout: float | None = None) -> WebElement | None:
+        return self.ensure_element(By.CLASS_NAME, selector, state, timeout)
+
+    def ensure_element_by_css_selector(self, selector: str, state: str | None = "present", timeout: float | None = None) -> WebElement | None:
+        return self.ensure_element(By.CSS_SELECTOR, selector, state, timeout)
+
+    def ensure_element(self, locator: ByType | str, selector: str, state: str | None = "present", timeout: float | None = None) -> WebElement | None:
+        """
+        Wait until an element appears or disappears in the browser.
+
+        The webdriver runs in parallel with our scripts, so we must wait for it everytime it
+        runs javascript. Selenium automatically waits till a page loads when GETing it,
+        but it doesn't do this when it runs javascript and makes AJAX requests.
+        So we must explicitly wait in that case.
+
+        The 'locator' argument defines what strategy we use to search for the element.
+        It expects standard names from the By class in selenium.webdriver.common.by.
+        https://www.selenium.dev/selenium/docs/api/py/webdriver/selenium.webdriver.common.by.html
+
+        The 'state' argument allows us to chose between waiting for the element to be visible,
+        clickable, present, or invisible. Presence is more inclusive, but sometimes we want to
+        know if the element is visible. Careful, its not always intuitive what Selenium considers
+        to be a visible element. We can also wait for it to be clickable, although this method
+        is a bit buggy in selenium, an element can be 'clickable' according to selenium and
+        still fail when we try to click it.
+
+        More info at: http://selenium-python.readthedocs.io/waits.html
+        """
+        locators_compatibility = {
+            "link_text": By.LINK_TEXT,
+            "partial_link_text": By.PARTIAL_LINK_TEXT,
+            "tag_name": By.TAG_NAME,
+            "class_name": By.CLASS_NAME,
+            "css_selector": By.CSS_SELECTOR,
+        }
+        if locator in locators_compatibility:
+            warnings.warn(
+                """
+                Support for locator strategy names with underscores is deprecated.
+                Use strategies from Selenium's By class (importable from selenium.webdriver.common.by).
+                """,
+                DeprecationWarning,
+                stacklevel=2,
+            )
+            locator = locators_compatibility[locator]
+
+        if not timeout:
+            timeout = self.default_timeout or DEFAULT_TIMEOUT
+
+        if state == "visible":
+            element = WebDriverWait(self, timeout).until(expected_conditions.visibility_of_element_located((locator, selector)))
+        elif state == "clickable":
+            element = WebDriverWait(self, timeout).until(expected_conditions.element_to_be_clickable((locator, selector)))
+        elif state == "present":
+            element = WebDriverWait(self, timeout).until(expected_conditions.presence_of_element_located((locator, selector)))
+        elif state == "invisible":
+            WebDriverWait(self, timeout).until(expected_conditions.invisibility_of_element_located((locator, selector)))
+            element = None
+        else:
+            msg = f"The 'state' argument must be 'visible', 'clickable', 'present' or 'invisible', not '{state}'"
+            raise ValueError(msg)
+
+        # We add this method to our element to provide a more robust click. Chromedriver
+        # sometimes needs some time before it can click an item, specially if it needs to
+        # scroll into it first. This method ensures clicks don't fail because of this.
+        if element:
+            element.ensure_click = functools.partial(_ensure_click, element)  # type: ignore[attr-defined]
+        return element
+
+    @property
+    def selector(self) -> Selector:
+        """
+        Returns the current state of the browser in a Selector.
+
+        We re-parse the site on each xpath, css, re call because we are running a web browser
+        and the site may change between calls
+        """
+        return Selector(text=self.page_source)
+
+    def xpath(self, *args, **kwargs) -> SelectorList[Selector]:
+        return self.selector.xpath(*args, **kwargs)
+
+    def css(self, *args, **kwargs) -> SelectorList[Selector]:
+        return self.selector.css(*args, **kwargs)
+
+    def re(self, *args, **kwargs) -> list[str]:
+        return self.selector.re(*args, **kwargs)
+
+    def re_first(self, *args, **kwargs) -> str | None:
+        return self.selector.re_first(*args, **kwargs)

requestium-0.5.1/requestium/requestium_response.py

@@ -0,0 +1,34 @@
+from __future__ import annotations
+
+import requests
+from parsel.selector import Selector, SelectorList
+from requests import Response
+
+
+class RequestiumResponse(requests.Response):
+    """Adds xpath, css, and regex methods to a normal requests response object."""
+
+    def __init__(self, response: Response) -> None:
+        super().__init__()
+        self.__class__ = type(response.__class__.__name__, (self.__class__, response.__class__), response.__dict__)
+
+    @property
+    def selector(self) -> Selector:
+        """
+        Returns the response text in a Selector.
+
+        We re-parse the text on each xpath, css, re call in case the encoding has changed.
+        """
+        return Selector(text=self.text)
+
+    def xpath(self, *args, **kwargs) -> SelectorList[Selector]:
+        return self.selector.xpath(*args, **kwargs)
+
+    def css(self, *args, **kwargs) -> SelectorList[Selector]:
+        return self.selector.css(*args, **kwargs)
+
+    def re(self, *args, **kwargs) -> list[str]:
+        return self.selector.re(*args, **kwargs)
+
+    def re_first(self, *args, **kwargs) -> str | None:
+        return self.selector.re_first(*args, **kwargs)