ai-url-aggregator 0.1__tar.gz

ai_url_aggregator-0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Carlos A. Planchón
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ai_url_aggregator-0.1/PKG-INFO

@@ -0,0 +1,137 @@
+Metadata-Version: 2.2
+Name: ai_url_aggregator
+Version: 0.1
+Summary: Extracts all text results from an XPath query on a parsel Selector object.
+Author-email: "Carlos A. Planchón" <carlosandresplanchonprestes@gmail.com>
+License: MIT License
+Project-URL: repository, https://github.com/carlosplanchon/ai_url_aggregator.git
+Keywords: openai,url,research,grabber
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Build Tools
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.13
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai
+Requires-Dist: requests
+
+```markdown
+# ai_url_aggregator
+
+> **Note**: *This is a small experimental library, provided as-is.
+
+**ai_url_aggregator** is a Python tool that leverages **Perplexity** and **OpenAI** to search the internet for relevant URLs, filter and deduplicate them, check their availability, and then select the most important ones based on GPT analysis.
+
+---
+
+## Features
+
+1. **Search Across Models**  
+   Uses Perplexity’s `sonar-reasoning` model to query the internet for URLs related to your prompt.
+2. **Clean & Filter**  
+   - Prefers `https://` links when both `http://` and `https://` are found for the same domain.  
+   - Removes duplicates by collecting results into a `set`.
+3. **Online Check**  
+   - Verifies each URL’s availability (status codes `200` or `403`) using `requests`.
+4. **Relevance Ranking**  
+   - Uses an OpenAI model to select the most important websites from the deduplicated list of online URLs.
+
+---
+
+## Installation
+
+### 1. Install via PyPI
+
+```bash
+pip install ai_url_aggregator
+```
+
+### 2. Set Environment Variables
+
+You must provide your **Perplexity** and **OpenAI** API keys:
+
+```bash
+export PERPLEXITY_API_KEY="PERPLEXITY_API_KEY"
+export OPENAI_API_KEY="OPENAI_API_KEY"
+```
+
+Replace `"PERPLEXITY_API_KEY"` and `"OPENAI_API_KEY"` with your actual API keys.
+
+### 3. (Optional) Install from Source
+
+1. **Clone or Download** this repository.  
+2. **Install Dependencies**:
+   ```bash
+   pip install -r requirements.txt
+   ```
+   This ensures all required libraries (like `openai`, `requests`, etc.) are installed.
+
+---
+
+## How It Works
+
+1. **`query_models(query: str) -> list[str]`**  
+   - Sends a query to Perplexity’s `sonar-reasoning` model.
+   - Parses the Perplexity output with an OpenAI model into a structured list of URLs.
+
+2. **`keep_https(urls: list[str]) -> list[str]`**  
+   - Selects `https://` versions of URLs when duplicates exist, else keeps `http://`.
+
+3. **`execute_query_multiple_times(query: str, num_runs: int) -> list[str]`**  
+   - Runs the query multiple times to gather more URLs.
+   - Deduplicates results using a `set`.
+
+4. **`check_urls_online(urls: list[str]) -> list[str]`**  
+   - Pings each URL to see if it’s reachable (status `200` or `403`).
+
+5. **`search_for_web_urls(query: str, num_runs: int) -> list[str]`**  
+   - Brings all the above together:
+     1. Executes a query multiple times.
+     2. Prefers HTTPS versions of each domain.
+     3. Verifies URL reachability.
+     4. Returns a final list of online, deduplicated URLs.
+
+6. **`get_top_relevant_websites(website_urls: list[str]) -> list[Website]`**  
+   - Uses an OpenAI model to select the most relevant (important) websites from the final list of URLs.
+
+---
+
+## Usage Example
+
+Once installed and your environment variables are set, you can do:
+
+```python
+import prettyprinter
+from ai_url_aggregator import (
+    search_for_web_urls,
+    get_top_relevant_websites
+)
+
+# Optional: install prettyprinter extras for nicer output
+prettyprinter.install_extras()
+
+# Example query:
+query = "Give me a list of all the real state agencies in Uruguay."
+
+# Step 1: Get a cleaned, deduplicated, and verified list of URLs
+online_urls = search_for_web_urls(query=query)
+
+print("--- Online URLs ---")
+prettyprinter.cpprint(online_urls)
+
+# Step 2: Get the most important websites from the final list
+most_important_websites = get_top_relevant_websites(website_urls=online_urls)
+
+print("--- Most Important Websites ---")
+prettyprinter.cpprint(most_important_websites)
+```
+
+---
+
+## License
+
+This project is distributed under the **MIT License**. See `LICENSE` for more information.
+
+---
+
+All suggestions and improvements are welcome!

ai_url_aggregator-0.1/README.md

@@ -0,0 +1,120 @@
+```markdown
+# ai_url_aggregator
+
+> **Note**: *This is a small experimental library, provided as-is.
+
+**ai_url_aggregator** is a Python tool that leverages **Perplexity** and **OpenAI** to search the internet for relevant URLs, filter and deduplicate them, check their availability, and then select the most important ones based on GPT analysis.
+
+---
+
+## Features
+
+1. **Search Across Models**  
+   Uses Perplexity’s `sonar-reasoning` model to query the internet for URLs related to your prompt.
+2. **Clean & Filter**  
+   - Prefers `https://` links when both `http://` and `https://` are found for the same domain.  
+   - Removes duplicates by collecting results into a `set`.
+3. **Online Check**  
+   - Verifies each URL’s availability (status codes `200` or `403`) using `requests`.
+4. **Relevance Ranking**  
+   - Uses an OpenAI model to select the most important websites from the deduplicated list of online URLs.
+
+---
+
+## Installation
+
+### 1. Install via PyPI
+
+```bash
+pip install ai_url_aggregator
+```
+
+### 2. Set Environment Variables
+
+You must provide your **Perplexity** and **OpenAI** API keys:
+
+```bash
+export PERPLEXITY_API_KEY="PERPLEXITY_API_KEY"
+export OPENAI_API_KEY="OPENAI_API_KEY"
+```
+
+Replace `"PERPLEXITY_API_KEY"` and `"OPENAI_API_KEY"` with your actual API keys.
+
+### 3. (Optional) Install from Source
+
+1. **Clone or Download** this repository.  
+2. **Install Dependencies**:
+   ```bash
+   pip install -r requirements.txt
+   ```
+   This ensures all required libraries (like `openai`, `requests`, etc.) are installed.
+
+---
+
+## How It Works
+
+1. **`query_models(query: str) -> list[str]`**  
+   - Sends a query to Perplexity’s `sonar-reasoning` model.
+   - Parses the Perplexity output with an OpenAI model into a structured list of URLs.
+
+2. **`keep_https(urls: list[str]) -> list[str]`**  
+   - Selects `https://` versions of URLs when duplicates exist, else keeps `http://`.
+
+3. **`execute_query_multiple_times(query: str, num_runs: int) -> list[str]`**  
+   - Runs the query multiple times to gather more URLs.
+   - Deduplicates results using a `set`.
+
+4. **`check_urls_online(urls: list[str]) -> list[str]`**  
+   - Pings each URL to see if it’s reachable (status `200` or `403`).
+
+5. **`search_for_web_urls(query: str, num_runs: int) -> list[str]`**  
+   - Brings all the above together:
+     1. Executes a query multiple times.
+     2. Prefers HTTPS versions of each domain.
+     3. Verifies URL reachability.
+     4. Returns a final list of online, deduplicated URLs.
+
+6. **`get_top_relevant_websites(website_urls: list[str]) -> list[Website]`**  
+   - Uses an OpenAI model to select the most relevant (important) websites from the final list of URLs.
+
+---
+
+## Usage Example
+
+Once installed and your environment variables are set, you can do:
+
+```python
+import prettyprinter
+from ai_url_aggregator import (
+    search_for_web_urls,
+    get_top_relevant_websites
+)
+
+# Optional: install prettyprinter extras for nicer output
+prettyprinter.install_extras()
+
+# Example query:
+query = "Give me a list of all the real state agencies in Uruguay."
+
+# Step 1: Get a cleaned, deduplicated, and verified list of URLs
+online_urls = search_for_web_urls(query=query)
+
+print("--- Online URLs ---")
+prettyprinter.cpprint(online_urls)
+
+# Step 2: Get the most important websites from the final list
+most_important_websites = get_top_relevant_websites(website_urls=online_urls)
+
+print("--- Most Important Websites ---")
+prettyprinter.cpprint(most_important_websites)
+```
+
+---
+
+## License
+
+This project is distributed under the **MIT License**. See `LICENSE` for more information.
+
+---
+
+All suggestions and improvements are welcome!

ai_url_aggregator-0.1/ai_url_aggregator/__init__.py

@@ -0,0 +1,210 @@
+#!/usr/bin/env python3
+
+from openai import OpenAI
+
+import os
+import json
+
+from pydantic import BaseModel
+from pydantic import Field
+from typing import List
+
+
+import requests
+
+
+PERPLEXITY_API_KEY = os.environ["PERPLEXITY_API_KEY"]
+OPENAI_API_KEY = os.environ["OPENAI_API_KEY"]
+
+
+perplexity_client = OpenAI(
+    api_key=PERPLEXITY_API_KEY,
+    base_url="https://api.perplexity.ai"
+)
+
+openai_client = OpenAI(
+    api_key=OPENAI_API_KEY
+)
+
+
+class UrlList(BaseModel):
+    urls: List[str]
+
+
+def query_models(query: str):
+    perplexity_messages = [
+        {
+            "role": "user",
+            "content": (
+                query
+            ),
+        },
+    ]
+
+    # MODEL:
+    # https://sonar.perplexity.ai/
+    # demo chat completion without streaming.
+    response = perplexity_client.chat.completions.create(
+        # model="sonar-pro",
+        # sonar-reasoning is the Deepseek without censorship hosted in the US.
+        model="sonar-reasoning",
+        messages=perplexity_messages,
+    )
+    # print(response)
+
+    response_content: str = response.choices[0].message.content
+    print(response_content)
+
+    openai_messages = [
+        {
+            "role": "user",
+            "content": (
+                response_content
+            ),
+        },
+    ]
+
+    url_list_response = openai_client.beta.chat.completions.parse(
+        model="gpt-4o",
+        messages=openai_messages,
+        response_format=UrlList,
+        temperature=0
+    )
+
+    url_list: list[str] = url_list_response.choices[0].message.parsed.urls
+
+    return url_list
+
+
+def keep_https(urls: list[str]) -> list[str]:
+    # Create a dictionary to store the HTTPS version of each domain
+    https_urls = {}
+
+    for url in urls:
+        try:
+            # Extract the domain (without protocol)
+            domain = url.split('//')[1]
+
+            # If the URL is HTTPS, store it in the dictionary
+            if url.startswith('https://'):
+                https_urls[domain] = url
+
+            # If the URL is HTTP and no HTTPS version has been stored yet,
+            # store the HTTP version.
+            elif url.startswith('http://') and domain not in https_urls:
+                https_urls[domain] = url
+
+        except Exception:
+            ...
+
+    # Return the values (URLs) from the dictionary
+    return list(https_urls.values())
+
+
+def execute_query_multiple_times(
+    query: str,
+    num_runs: int = 5
+        ) -> list[str]:
+    print("--- GET UNIQUE URLS ---")
+    # Initialize an empty set to store unique URLs
+    unique_urls = set()
+
+    # Execute the function `num_runs` times
+    for i in range(num_runs):
+        print(f"i: {i}")
+        # Get the URLs from the function
+        urls: list[str] = query_models(query=query)
+
+        urls = [url.rstrip("/") for url in urls]
+
+        # Add the URLs to the set (duplicates will be automatically ignored)
+        unique_urls.update(urls)
+
+    # Convert the set back to a list if needed
+    unique_urls_list = list(unique_urls)
+
+    # Return the list of unique URLs
+    return unique_urls_list
+
+
+def check_urls_online(urls: list[str]) -> list[str]:
+    online_urls = []
+
+    for url in urls:
+        try:
+            # Set a timeout to avoid waiting too long.
+            response = requests.head(url, timeout=5)
+            if response.status_code in [200, 403]:
+                online_urls.append(url)
+            else:
+                ...
+        except requests.exceptions.RequestException:
+            # Fallback to GET if HEAD fails
+            try:
+                response = requests.get(url, timeout=5)
+                if response.status_code in [200, 403]:
+                    online_urls.append(url)
+            except Exception:
+                ...
+    return online_urls
+
+
+def search_for_web_urls(
+    query: str,
+    num_runs=5
+        ) -> list[str]:
+    query_response: list[str] = execute_query_multiple_times(
+        query=query,
+        num_runs=num_runs
+    )
+    prefer_https_urls = keep_https(
+        urls=query_response
+    )
+    online_urls: list[str] = check_urls_online(
+        urls=prefer_https_urls
+    )
+
+    return online_urls
+
+
+class Website(BaseModel):
+    name: str = Field(description="The name of the website.")
+    url: str = Field(description="The url of the website.")
+
+
+class WebsiteList(BaseModel):
+    websites_list: list[Website] = Field(description="A short list.")
+
+
+def get_top_relevant_websites(website_urls: list[str]) -> list[Website]:
+    websites_list_dump = json.dumps(website_urls)
+
+    # Get important websites with GPT.
+    openai_messages = [
+        {
+            "role": "user",
+            "content": (
+                websites_list_dump
+            ),
+        },
+        {
+            "role": "system",
+            "content": (
+                "Select the most important websites from this JSON. Return a short list."
+            ),
+        }
+    ]
+
+    important_websites_fitered_by_gpt =\
+        openai_client.beta.chat.completions.parse(
+            model="gpt-4o",
+            messages=openai_messages,
+            response_format=WebsiteList,
+            temperature=0
+        )
+
+    parsed_websites_list: list[Website] =\
+        important_websites_fitered_by_gpt.choices[
+            0].message.parsed.websites_list
+
+    return parsed_websites_list

ai_url_aggregator-0.1/ai_url_aggregator.egg-info/PKG-INFO

@@ -0,0 +1,137 @@
+Metadata-Version: 2.2
+Name: ai_url_aggregator
+Version: 0.1
+Summary: Extracts all text results from an XPath query on a parsel Selector object.
+Author-email: "Carlos A. Planchón" <carlosandresplanchonprestes@gmail.com>
+License: MIT License
+Project-URL: repository, https://github.com/carlosplanchon/ai_url_aggregator.git
+Keywords: openai,url,research,grabber
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Build Tools
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.13
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai
+Requires-Dist: requests
+
+```markdown
+# ai_url_aggregator
+
+> **Note**: *This is a small experimental library, provided as-is.
+
+**ai_url_aggregator** is a Python tool that leverages **Perplexity** and **OpenAI** to search the internet for relevant URLs, filter and deduplicate them, check their availability, and then select the most important ones based on GPT analysis.
+
+---
+
+## Features
+
+1. **Search Across Models**  
+   Uses Perplexity’s `sonar-reasoning` model to query the internet for URLs related to your prompt.
+2. **Clean & Filter**  
+   - Prefers `https://` links when both `http://` and `https://` are found for the same domain.  
+   - Removes duplicates by collecting results into a `set`.
+3. **Online Check**  
+   - Verifies each URL’s availability (status codes `200` or `403`) using `requests`.
+4. **Relevance Ranking**  
+   - Uses an OpenAI model to select the most important websites from the deduplicated list of online URLs.
+
+---
+
+## Installation
+
+### 1. Install via PyPI
+
+```bash
+pip install ai_url_aggregator
+```
+
+### 2. Set Environment Variables
+
+You must provide your **Perplexity** and **OpenAI** API keys:
+
+```bash
+export PERPLEXITY_API_KEY="PERPLEXITY_API_KEY"
+export OPENAI_API_KEY="OPENAI_API_KEY"
+```
+
+Replace `"PERPLEXITY_API_KEY"` and `"OPENAI_API_KEY"` with your actual API keys.
+
+### 3. (Optional) Install from Source
+
+1. **Clone or Download** this repository.  
+2. **Install Dependencies**:
+   ```bash
+   pip install -r requirements.txt
+   ```
+   This ensures all required libraries (like `openai`, `requests`, etc.) are installed.
+
+---
+
+## How It Works
+
+1. **`query_models(query: str) -> list[str]`**  
+   - Sends a query to Perplexity’s `sonar-reasoning` model.
+   - Parses the Perplexity output with an OpenAI model into a structured list of URLs.
+
+2. **`keep_https(urls: list[str]) -> list[str]`**  
+   - Selects `https://` versions of URLs when duplicates exist, else keeps `http://`.
+
+3. **`execute_query_multiple_times(query: str, num_runs: int) -> list[str]`**  
+   - Runs the query multiple times to gather more URLs.
+   - Deduplicates results using a `set`.
+
+4. **`check_urls_online(urls: list[str]) -> list[str]`**  
+   - Pings each URL to see if it’s reachable (status `200` or `403`).
+
+5. **`search_for_web_urls(query: str, num_runs: int) -> list[str]`**  
+   - Brings all the above together:
+     1. Executes a query multiple times.
+     2. Prefers HTTPS versions of each domain.
+     3. Verifies URL reachability.
+     4. Returns a final list of online, deduplicated URLs.
+
+6. **`get_top_relevant_websites(website_urls: list[str]) -> list[Website]`**  
+   - Uses an OpenAI model to select the most relevant (important) websites from the final list of URLs.
+
+---
+
+## Usage Example
+
+Once installed and your environment variables are set, you can do:
+
+```python
+import prettyprinter
+from ai_url_aggregator import (
+    search_for_web_urls,
+    get_top_relevant_websites
+)
+
+# Optional: install prettyprinter extras for nicer output
+prettyprinter.install_extras()
+
+# Example query:
+query = "Give me a list of all the real state agencies in Uruguay."
+
+# Step 1: Get a cleaned, deduplicated, and verified list of URLs
+online_urls = search_for_web_urls(query=query)
+
+print("--- Online URLs ---")
+prettyprinter.cpprint(online_urls)
+
+# Step 2: Get the most important websites from the final list
+most_important_websites = get_top_relevant_websites(website_urls=online_urls)
+
+print("--- Most Important Websites ---")
+prettyprinter.cpprint(most_important_websites)
+```
+
+---
+
+## License
+
+This project is distributed under the **MIT License**. See `LICENSE` for more information.
+
+---
+
+All suggestions and improvements are welcome!

ai_url_aggregator-0.1/ai_url_aggregator.egg-info/SOURCES.txt

@@ -0,0 +1,9 @@
+LICENSE
+README.md
+pyproject.toml
+ai_url_aggregator/__init__.py
+ai_url_aggregator.egg-info/PKG-INFO
+ai_url_aggregator.egg-info/SOURCES.txt
+ai_url_aggregator.egg-info/dependency_links.txt
+ai_url_aggregator.egg-info/requires.txt
+ai_url_aggregator.egg-info/top_level.txt

ai_url_aggregator-0.1/ai_url_aggregator.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

ai_url_aggregator-0.1/ai_url_aggregator.egg-info/requires.txt

@@ -0,0 +1,2 @@
+openai
+requests

ai_url_aggregator-0.1/ai_url_aggregator.egg-info/top_level.txt

@@ -0,0 +1 @@
+ai_url_aggregator

ai_url_aggregator-0.1/pyproject.toml

@@ -0,0 +1,31 @@
+[build-system]
+requires = ["setuptools"]
+build-backend = "setuptools.build_meta"
+
+[options]
+package_dir = "ai_url_aggregator"
+packages = ["ai_url_aggregator"]
+
+[project]
+name = "ai_url_aggregator"
+version = "0.1"
+authors = [
+    {name = "Carlos A. Planchón", email = "carlosandresplanchonprestes@gmail.com"}
+]
+description = "Extracts all text results from an XPath query on a parsel Selector object."
+classifiers = [
+    "Intended Audience :: Developers",
+    "Topic :: Software Development :: Build Tools",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3.13",
+]
+license = {text = "MIT License"}
+keywords = ["openai", "url", "research", "grabber"]
+readme = "README.md"
+dependencies=[
+    "openai",
+    "requests"
+]
+
+[project.urls]
+repository = "https://github.com/carlosplanchon/ai_url_aggregator.git"

ai_url_aggregator-0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+