fraudcrawler 0.3.3__tar.gz

fraudcrawler-0.3.3/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 veanu
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

fraudcrawler-0.3.3/PKG-INFO

@@ -0,0 +1,163 @@
+Metadata-Version: 2.3
+Name: fraudcrawler
+Version: 0.3.3
+Summary: Intelligent Market Monitoring
+Home-page: https://github/open-veanu/fraudcrawler
+License: MIT
+Author: Domingo Bertus
+Author-email: hello@veanu.ch
+Requires-Python: >=3.11,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: aiohttp (>=3.11.14,<4.0.0)
+Requires-Dist: openai (>=1.68.2,<2.0.0)
+Requires-Dist: pandas (>=2.2.3,<3.0.0)
+Requires-Dist: pydantic-settings (>=2.8.1,<3.0.0)
+Requires-Dist: requests (>=2.32.3,<3.0.0)
+Project-URL: Repository, https://github/open-veanu/fraudcrawler
+Description-Content-Type: text/markdown
+
+# open-veanu/fraudcrawler
+Intelligent Market Monitoring
+
+The pipeline for monitoring the market has the folling main steps:
+1. search for a given term using SerpAPI
+2. get product information using ZyteAPI
+3. assess relevance of the found products using an OpenAI API
+
+## Installation
+```bash
+python3.11 -m venv .venv
+source .venv/bin/activate
+pip install fraudcrawler
+```
+
+## Usage
+### `.env` file
+Make sure to create an `.env` file with the necessary API keys and credentials (c.f. `.env.example` file).
+
+### Run demo pipeline
+```bash
+python -m fraudcrawler.launch_demo_pipeline
+```
+
+### Customize the pipeline
+Start by initializing the client
+```python
+from fraudcrawler import FraudCrawlerClient
+
+# Initialize the client
+client = FraudCrawlerClient()
+```
+
+For setting up the search we need 5 main objects.
+
+#### `search_term: str`
+The search term for the query (similar to search terms used within major search providers).
+
+#### `language: Language`
+The language used in SerpAPI ('hl' parameter), as well as for the optional search term enrichement (e.g. finding similar and related search terms). `language=Language('German')` creates an object having a language name and a language code as: `Language(name='German', code='de')`.
+
+#### `location: Location`
+The location used in SerpAPI ('gl' parameter). `location=Location('Switzerland')` creates an object having a location name and a location code as `Location(name='Switzerland', code='ch')`.
+
+#### `deepness: Deepness`
+Defines the search depth with the number of results to retrieve and optional enrichment parameters.
+
+#### `prompts: List[Prompt]`
+The list of prompts to classify a given product with (multiple) LLM calls. Each prompt object has a `name`, a `context` (used for defining the user prompt), a `system_prompt` (for defining the classification task), `allowed_classes` (a list of possible classes) and optionally `default_if_missing` (a default class if anything goes wrong).
+
+```python
+from fraudcrawler import Language, Location, Deepness, Prompt
+# Setup the search
+search_term = "sildenafil"
+language = Language(name="German")
+location = Location(name="Switzerland")
+deepness = Deepness(num_results=50)
+prompts = [
+    Prompt(
+        name="relevance",
+        context="This organization is interested in medical products and drugs.",
+        system_prompt=(
+            "You are a helpful and intelligent assistant. Your task is to classify any given product "
+            "as either relevant (1) or not relevant (0), strictly based on the context and product details provided by the user. "
+            "You must consider all aspects of the given context and make a binary decision accordingly. "
+            "If the product aligns with the user's needs, classify it as 1 (relevant); otherwise, classify it as 0 (not relevant). "
+            "Respond only with the number 1 or 0."
+        ),
+        allowed_classes=[0, 1],
+    )
+]
+```
+
+(Optional) Add search term enrichement. This will find related search terms (in a given language) and search for these as well.
+```python
+from fraudcrawler import Enrichment
+deepness.enrichment = Enrichment(
+    additional_terms=5,
+    additional_urls_per_term=10
+)
+```
+
+(Optional) Add marketplaces where we explicitely want to look for (this will focus your search as the :site parameter for a google search)
+```python
+from fraudcrawler import Host
+marketplaces = [
+    Host(name="International", domains="zavamed.com,apomeds.com"),
+    Host(name="National", domains="netdoktor.ch, nobelpharma.ch"),
+]
+```
+
+(Optional) Exclude urls (where you don't want to find products)
+```python
+excluded_urls = [
+    Host(name="Compendium", domains="compendium.ch"),
+]
+```
+
+(Optional) Exclude previously collected urls (intends to save credits)
+```python
+previously_collected_urls = [
+    https://pharmaciedelabateliere.ch/shop/sante/douleurs-inflammations/dafalgan-cpr-eff-500-mg-16-pce/,
+    https://eiche.ch/product/schmerzmittel-52cd81d5d206a/dafalgan-brausetabletten-1336653,
+]
+```
+
+And finally run the pipeline
+```python
+# Execute the pipeline
+client.execute(
+    search_term=search_term,
+    language=language,
+    location=location,
+    deepness=deepness,
+    prompts=prompts,
+    # marketplaces=marketplaces,    # Uncomment this for using marketplaces
+    # excluded_urls=excluded_urls   # Uncomment this for using excluded_urls
+    # previously_collected_urls=previously_collected_urls    # Uncomment this for using previously_selected_urls
+)
+```
+This creates a file with name pattern `<search_term>_<language.code>_<location.code>_<datetime[%Y%m%d%H%M%S]>.csv` inside the folder `data/results/`.
+
+Once the pipeline terminated the results can be loaded and examined as follows:
+```python
+df = client.load_results()
+print(df.head(n=10))
+```
+
+If the client has been used to run multiple pipelines, an overview of the available results (for a given instance of 
+`FraudCrawlerClient`) can be obtained with
+```python
+client.print_available_results()
+```
+
+## Contributing
+see `CONTRIBUTING.md`
+
+### Async Setup
+The following image provides a schematic representation of the package's async setup.
+![Async Setup](https://github.com/open-veanu/fraudcrawler/raw/master/docs/assets/images/Fraudcrawler_Async_Setup.svg)
+

fraudcrawler-0.3.3/README.md

@@ -0,0 +1,140 @@
+# open-veanu/fraudcrawler
+Intelligent Market Monitoring
+
+The pipeline for monitoring the market has the folling main steps:
+1. search for a given term using SerpAPI
+2. get product information using ZyteAPI
+3. assess relevance of the found products using an OpenAI API
+
+## Installation
+```bash
+python3.11 -m venv .venv
+source .venv/bin/activate
+pip install fraudcrawler
+```
+
+## Usage
+### `.env` file
+Make sure to create an `.env` file with the necessary API keys and credentials (c.f. `.env.example` file).
+
+### Run demo pipeline
+```bash
+python -m fraudcrawler.launch_demo_pipeline
+```
+
+### Customize the pipeline
+Start by initializing the client
+```python
+from fraudcrawler import FraudCrawlerClient
+
+# Initialize the client
+client = FraudCrawlerClient()
+```
+
+For setting up the search we need 5 main objects.
+
+#### `search_term: str`
+The search term for the query (similar to search terms used within major search providers).
+
+#### `language: Language`
+The language used in SerpAPI ('hl' parameter), as well as for the optional search term enrichement (e.g. finding similar and related search terms). `language=Language('German')` creates an object having a language name and a language code as: `Language(name='German', code='de')`.
+
+#### `location: Location`
+The location used in SerpAPI ('gl' parameter). `location=Location('Switzerland')` creates an object having a location name and a location code as `Location(name='Switzerland', code='ch')`.
+
+#### `deepness: Deepness`
+Defines the search depth with the number of results to retrieve and optional enrichment parameters.
+
+#### `prompts: List[Prompt]`
+The list of prompts to classify a given product with (multiple) LLM calls. Each prompt object has a `name`, a `context` (used for defining the user prompt), a `system_prompt` (for defining the classification task), `allowed_classes` (a list of possible classes) and optionally `default_if_missing` (a default class if anything goes wrong).
+
+```python
+from fraudcrawler import Language, Location, Deepness, Prompt
+# Setup the search
+search_term = "sildenafil"
+language = Language(name="German")
+location = Location(name="Switzerland")
+deepness = Deepness(num_results=50)
+prompts = [
+    Prompt(
+        name="relevance",
+        context="This organization is interested in medical products and drugs.",
+        system_prompt=(
+            "You are a helpful and intelligent assistant. Your task is to classify any given product "
+            "as either relevant (1) or not relevant (0), strictly based on the context and product details provided by the user. "
+            "You must consider all aspects of the given context and make a binary decision accordingly. "
+            "If the product aligns with the user's needs, classify it as 1 (relevant); otherwise, classify it as 0 (not relevant). "
+            "Respond only with the number 1 or 0."
+        ),
+        allowed_classes=[0, 1],
+    )
+]
+```
+
+(Optional) Add search term enrichement. This will find related search terms (in a given language) and search for these as well.
+```python
+from fraudcrawler import Enrichment
+deepness.enrichment = Enrichment(
+    additional_terms=5,
+    additional_urls_per_term=10
+)
+```
+
+(Optional) Add marketplaces where we explicitely want to look for (this will focus your search as the :site parameter for a google search)
+```python
+from fraudcrawler import Host
+marketplaces = [
+    Host(name="International", domains="zavamed.com,apomeds.com"),
+    Host(name="National", domains="netdoktor.ch, nobelpharma.ch"),
+]
+```
+
+(Optional) Exclude urls (where you don't want to find products)
+```python
+excluded_urls = [
+    Host(name="Compendium", domains="compendium.ch"),
+]
+```
+
+(Optional) Exclude previously collected urls (intends to save credits)
+```python
+previously_collected_urls = [
+    https://pharmaciedelabateliere.ch/shop/sante/douleurs-inflammations/dafalgan-cpr-eff-500-mg-16-pce/,
+    https://eiche.ch/product/schmerzmittel-52cd81d5d206a/dafalgan-brausetabletten-1336653,
+]
+```
+
+And finally run the pipeline
+```python
+# Execute the pipeline
+client.execute(
+    search_term=search_term,
+    language=language,
+    location=location,
+    deepness=deepness,
+    prompts=prompts,
+    # marketplaces=marketplaces,    # Uncomment this for using marketplaces
+    # excluded_urls=excluded_urls   # Uncomment this for using excluded_urls
+    # previously_collected_urls=previously_collected_urls    # Uncomment this for using previously_selected_urls
+)
+```
+This creates a file with name pattern `<search_term>_<language.code>_<location.code>_<datetime[%Y%m%d%H%M%S]>.csv` inside the folder `data/results/`.
+
+Once the pipeline terminated the results can be loaded and examined as follows:
+```python
+df = client.load_results()
+print(df.head(n=10))
+```
+
+If the client has been used to run multiple pipelines, an overview of the available results (for a given instance of 
+`FraudCrawlerClient`) can be obtained with
+```python
+client.print_available_results()
+```
+
+## Contributing
+see `CONTRIBUTING.md`
+
+### Async Setup
+The following image provides a schematic representation of the package's async setup.
+![Async Setup](https://github.com/open-veanu/fraudcrawler/raw/master/docs/assets/images/Fraudcrawler_Async_Setup.svg)

fraudcrawler-0.3.3/fraudcrawler/__init__.py

@@ -0,0 +1,30 @@
+from fraudcrawler.scraping.serp import SerpApi
+from fraudcrawler.scraping.enrich import Enricher
+from fraudcrawler.scraping.zyte import ZyteApi
+from fraudcrawler.processing.processor import Processor
+from fraudcrawler.base.orchestrator import Orchestrator, ProductItem
+from fraudcrawler.base.client import FraudCrawlerClient
+from fraudcrawler.base.base import (
+    Deepness,
+    Enrichment,
+    Host,
+    Language,
+    Location,
+    Prompt,
+)
+
+__all__ = [
+    "SerpApi",
+    "Enricher",
+    "ZyteApi",
+    "Processor",
+    "Orchestrator",
+    "ProductItem",
+    "FraudCrawlerClient",
+    "Language",
+    "Location",
+    "Host",
+    "Deepness",
+    "Enrichment",
+    "Prompt",
+]

fraudcrawler-0.3.3/fraudcrawler/base/base.py

@@ -0,0 +1,145 @@
+import json
+import logging
+from pydantic import BaseModel, field_validator, model_validator
+from pydantic_settings import BaseSettings
+from typing import List
+
+import aiohttp
+
+from fraudcrawler.settings import (
+    GOOGLE_LANGUAGES_FILENAME,
+    GOOGLE_LOCATIONS_FILENAME,
+    PROCESSOR_DEFAULT_IF_MISSING,
+)
+
+logger = logging.getLogger(__name__)
+
+# Load google locations and languages
+with open(GOOGLE_LOCATIONS_FILENAME, "r") as gfile:
+    _locs = json.load(gfile)
+_LOCATION_CODES = {loc["name"]: loc["country_code"].lower() for loc in _locs}
+with open(GOOGLE_LANGUAGES_FILENAME, "r") as gfile:
+    _langs = json.load(gfile)
+_LANGUAGE_CODES = {lang["language_name"]: lang["language_code"] for lang in _langs}
+
+
+# Base classes
+class Setup(BaseSettings):
+    """Class for loading environment variables."""
+
+    # Crawler ENV variables
+    serpapi_key: str
+    dataforseo_user: str
+    dataforseo_pwd: str
+    zyteapi_key: str
+    openaiapi_key: str
+
+    class Config:
+        env_file = ".env"
+        env_file_encoding = "utf-8"
+
+
+class Host(BaseModel):
+    """Model for host details (e.g. `Host(name="Galaxus", domains="galaxus.ch, digitec.ch")`)."""
+
+    name: str
+    domains: str | List[str]
+
+    @field_validator("domains", mode="before")
+    def split_domains_if_str(cls, val):
+        if isinstance(val, str):
+            return [dom.strip() for dom in val.split(",")]
+        return val
+
+
+class Location(BaseModel):
+    """Model for location details (e.g. `Location(name="Switzerland", code="ch")`)."""
+
+    name: str
+    code: str = ""
+
+    @model_validator(mode="before")
+    def set_code(cls, values):
+        """Set the location code if not provided and make it lower case."""
+        name = values.get("name")
+        code = values.get("code")
+        if code is None or not len(code):
+            code = _LOCATION_CODES.get(name)
+            if code is None:
+                raise ValueError(f'Location code not found for location name="{name}"')
+        code = code.lower()
+        return {"name": name, "code": code}
+
+
+class Language(BaseModel):
+    """Model for language details (e.g. `Language(name="German", code="de")`)."""
+
+    name: str
+    code: str = ""
+
+    @model_validator(mode="before")
+    def set_code(cls, values):
+        """Set the language code if not provided and make it lower case."""
+        name = values.get("name")
+        code = values.get("code")
+        if code is None or not len(code):
+            code = _LANGUAGE_CODES.get(name)
+            if code is None:
+                raise ValueError(f'Language code not found for language name="{name}"')
+        code = code.lower()
+        return {"name": name, "code": code}
+
+
+class Enrichment(BaseModel):
+    """Model for enriching initial search_term with alternative ones."""
+
+    additional_terms: int
+    additional_urls_per_term: int
+
+
+class Deepness(BaseModel):
+    """Model for search depth."""
+
+    num_results: int
+    enrichment: Enrichment | None = None
+
+
+class Prompt(BaseModel):
+    """Model for prompts."""
+
+    name: str
+    context: str
+    system_prompt: str
+    allowed_classes: List[int]
+    default_if_missing: int = PROCESSOR_DEFAULT_IF_MISSING
+
+
+class AsyncClient:
+    """Base class for sub-classes using async HTTP requests."""
+
+    @staticmethod
+    async def get(
+        url: str,
+        headers: dict | None = None,
+        params: dict | None = None,
+    ) -> dict:
+        """Async GET request of a given URL returning the data."""
+        async with aiohttp.ClientSession(headers=headers) as session:
+            async with session.get(url=url, params=params) as response:
+                response.raise_for_status()
+                json_ = await response.json()
+        return json_
+
+    @staticmethod
+    async def post(
+        url: str,
+        headers: dict | None = None,
+        data: List[dict] | dict | None = None,
+        auth: aiohttp.BasicAuth | None = None,
+    ) -> dict:
+        """Async POST request of a given URL returning the data."""
+        async with aiohttp.ClientSession(headers=headers) as session:
+            async with session.post(url=url, json=data, auth=auth) as response:
+                response.raise_for_status()
+                json_ = await response.json()
+        return json_

fraudcrawler-0.3.3/fraudcrawler/base/client.py

@@ -0,0 +1,134 @@
+import asyncio
+import csv
+from datetime import datetime
+import logging
+from pathlib import Path
+from pydantic import BaseModel
+from typing import List
+
+import pandas as pd
+
+from fraudcrawler.settings import ROOT_DIR
+from fraudcrawler.base.base import Setup, Language, Location, Deepness, Host, Prompt
+from fraudcrawler.base.orchestrator import Orchestrator, ProductItem
+
+logger = logging.getLogger(__name__)
+
+_RESULTS_DIR = ROOT_DIR / "data" / "results"
+
+
+class Results(BaseModel):
+    """The results of the product search."""
+
+    search_term: str
+    filename: Path | None = None
+
+
+class FraudCrawlerClient(Orchestrator):
+    """The main client for FraudCrawler."""
+
+    _filename_template = "{search_term}_{language}_{location}_{timestamp}.csv"
+
+    def __init__(self):
+        setup = Setup()
+        super().__init__(
+            serpapi_key=setup.serpapi_key,
+            dataforseo_user=setup.dataforseo_user,
+            dataforseo_pwd=setup.dataforseo_pwd,
+            zyteapi_key=setup.zyteapi_key,
+            openaiapi_key=setup.openaiapi_key,
+        )
+
+        self._results_dir = _RESULTS_DIR
+        if not self._results_dir.exists():
+            self._results_dir.mkdir(parents=True)
+        self._results: List[Results] = []
+
+    async def _collect_results(
+        self, queue_in: asyncio.Queue[ProductItem | None]
+    ) -> None:
+        """Collects the results from the given queue_in and saves it as csv.
+
+        Args:
+            queue_in: The input queue containing the results.
+        """
+        products = []
+        while True:
+            product = await queue_in.get()
+            if product is None:
+                queue_in.task_done()
+                break
+
+            products.append(product.model_dump())
+            queue_in.task_done()
+
+        # Convert the list of products to a DataFrame
+        df = pd.json_normalize(products)
+        cols = [c.split(".")[-1] for c in df.columns]
+        if len(cols) != len(set(cols)):
+            logger.error("Duplicate columns after json_normalize.")
+        else:
+            df.columns = cols
+
+        # Save the DataFrame to a CSV file
+        filename = self._results[-1].filename
+        df.to_csv(filename, index=False, quoting=csv.QUOTE_ALL)
+        logger.info(f"Results saved to {filename}")
+
+    def execute(
+        self,
+        search_term: str,
+        language: Language,
+        location: Location,
+        deepness: Deepness,
+        prompts: List[Prompt],
+        marketplaces: List[Host] | None = None,
+        excluded_urls: List[Host] | None = None,
+    ) -> None:
+        """Runs the pipeline steps: serp, enrich, zyte, process, and collect the results.
+
+        Args:
+            search_term: The search term for the query.
+            language: The language to use for the query.
+            location: The location to use for the query.
+            deepness: The search depth and enrichment details.
+            prompts: The list of prompts to use for classification.
+            marketplaces: The marketplaces to include in the search.
+            excluded_urls: The URLs to exclude from the search.
+        """
+        timestamp = datetime.today().strftime("%Y%m%d%H%M%S")
+        filename = self._results_dir / self._filename_template.format(
+            search_term=search_term,
+            language=language.code,
+            location=location.code,
+            timestamp=timestamp,
+        )
+        self._results.append(Results(search_term=search_term, filename=filename))
+
+        asyncio.run(
+            super().run(
+                search_term=search_term,
+                language=language,
+                location=location,
+                deepness=deepness,
+                prompts=prompts,
+                marketplaces=marketplaces,
+                excluded_urls=excluded_urls,
+            )
+        )
+
+    def load_results(self, index: int = -1) -> pd.DataFrame:
+        """Loads the results from the saved .csv files.
+
+        Args:
+            index: The index of the results to load (`incex=-1` are the results for the most recent run).
+        """
+
+        results = self._results[index]
+        return pd.read_csv(results.filename)
+
+    def print_available_results(self) -> None:
+        """Prints the available results."""
+        n_res = len(self._results)
+        for i, res in enumerate(self._results):
+            print(f"index={-n_res + i}: {res.search_term} - {res.filename}")