pocong 1.0.0__tar.gz → 1.1.0__tar.gz

pocong-1.1.0/PKG-INFO

@@ -0,0 +1,229 @@
+Metadata-Version: 2.4
+Name: pocong
+Version: 1.1.0
+Summary: Python Oriented Crawling Ongoing (POCONG): a simple crawling framework
+Home-page: https://gitlab.com/mohsin3107/pocong
+Author: Singgih
+Author-email: singgih@alkode.id
+License: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Internet :: WWW/HTTP
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: Scrapy>=2.5.0
+Requires-Dist: pandas>=1.3.0
+Requires-Dist: requests>=2.25.0
+Requires-Dist: Click>=7.0
+Requires-Dist: mechanize>=0.4.0
+Requires-Dist: html2text>=2020.1.16
+Requires-Dist: fake-useragent>=1.1.0
+Requires-Dist: beautifulsoup4>=4.9.0
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-mock; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: moto; extra == "dev"
+Requires-Dist: tox; extra == "dev"
+Requires-Dist: flake8; extra == "dev"
+Requires-Dist: flake8-import-order; extra == "dev"
+Requires-Dist: flake8-print; extra == "dev"
+Requires-Dist: flake8-builtins; extra == "dev"
+Requires-Dist: pep8-naming; extra == "dev"
+Requires-Dist: pre-commit; extra == "dev"
+Requires-Dist: rope; extra == "dev"
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+<p align="center">
+  <img src="https://i.ibb.co.com/35P4Nq9x/Screenshot-2025-08-22-at-18-40-11.png?width=128" alt="POCONG Logo" width="128"/>
+</p>
+
+# POCONG 🪦
+**Python Oriented Crawling ON Going**
+
+POCONG is a lightweight web crawling framework built in Python.
+
+## Features
+
+- 🔒 **Get Free Proxy**: Automatic proxy fetching, validation, and rotation from free proxy sources
+- 🌐 **Dynamic Media Web Scraping**: Extract content, metadata, and media information from web pages with proxy support
+- 📱 **Social Media Scraping**: Extract data from social media platforms *(coming soon)*
+- 🛒 **E-commerce Scraping**: Extract product information from e-commerce websites *(coming soon)*
+
+## Installation
+```bash
+pip install pocong
+```
+
+## Usage: Get Proxy from proxy_spiders
+
+You can use the `get_proxy` and `get_proxy_random` methods from `proxy_spiders` to fetch working proxies.
+
+```python
+from pocong.proxy_spiders import GetProxy
+
+gp = GetProxy()
+
+# Get the first working proxy
+proxy = gp.get_proxy()
+print("First working proxy:", proxy)
+```
+```python
+from pocong.proxy_spiders import GetProxy
+
+gp = GetProxy()
+
+# Get a random working proxy
+random_proxy = gp.get_proxy_random()
+print("Random working proxy:", random_proxy)
+```
+
+Sample output:
+```
+First working proxy: {'ip': '123.45.67.89', 'port': '8080', 'https': 'yes', ...}
+Random working proxy: {'ip': '98.76.54.32', 'port': '3128', 'https': 'yes', ...}
+```
+
+You can use the returned proxy dictionary with the `requests` library, for example:
+
+```python
+import requests
+
+proxy = gp.get_proxy()
+if proxy:
+    proxies = {
+        'http': f"http://{proxy['ip']}:{proxy['port']}",
+        'https': f"http://{proxy['ip']}:{proxy['port']}"
+    }
+    response = requests.get('https://httpbin.org/ip', proxies=proxies)
+    print(response.json())
+else:
+    print("No working proxy found.")
+```
+
+- `get_proxy()` will return the first working proxy found.
+- `get_proxy_random()` will return a random working proxy (with up to 20 retries).
+
+Both methods return a dictionary with proxy details (e.g., `{ 'ip': '...', 'port': '...', ... }`) or `None` if no working proxy is found.
+
+## Usage: Dynamic Media Web Scraping
+
+The `DynamicScrapingNews` class provides comprehensive web scraping capabilities with built-in proxy support for extracting content, metadata, and media information from web pages.
+
+### Basic Usage
+
+```python
+from pocong.media_spiders import DynamicScrapingNews
+
+# Simple scraping without proxy
+scraper = DynamicScrapingNews("https://example.com", use_proxy=False)
+result = scraper.scrape()
+
+# Extract specific information
+print(f"Title: {result['title']}")
+print(f"URL: {result['url']}")
+print(f"Media: {result['media']}")
+print(f"Published: {result['published_date']}")
+print(f"Text content: {result['text'][:200]}...")  # First 200 chars
+```
+
+### Proxy Configuration Options
+
+#### 1. Automatic Proxy (Default)
+```python
+# Uses automatic proxy fetching
+scraper = DynamicScrapingNews("https://example.com")
+result = scraper.scrape()
+```
+
+#### 2. Manual Proxy Configuration
+```python
+# Method 1: IP:Port format
+scraper = DynamicScrapingNews("https://example.com", 
+                              manual_proxy="192.168.1.1:8080")
+
+# Method 2: Full URL format
+scraper = DynamicScrapingNews("https://example.com", 
+                              manual_proxy="http://192.168.1.1:8080")
+
+# Method 3: Dictionary format
+scraper = DynamicScrapingNews("https://example.com", 
+                              manual_proxy={"ip": "192.168.1.1", "port": "8080"})
+
+result = scraper.scrape()
+```
+
+#### 3. No Proxy
+```python
+# Disable proxy completely
+scraper = DynamicScrapingNews("https://example.com", use_proxy=False)
+result = scraper.scrape()
+```
+
+#### 4. Manual Proxy Override
+```python
+# Manual proxy overrides use_proxy setting
+scraper = DynamicScrapingNews("https://example.com", 
+                              use_proxy=False, 
+                              manual_proxy="192.168.1.1:8080")
+result = scraper.scrape()
+```
+
+### Complete Example with Proxy Integration
+
+```python
+from pocong.proxy_spiders import GetProxy
+from pocong.media_spiders import DynamicScrapingNews
+
+# Get a working proxy
+proxy = GetProxy().get_proxy()
+print(f"Using proxy: {proxy}")
+
+# Use automatic proxy (default behavior)
+scraper = DynamicScrapingNews("https://example.com")
+result = scraper.scrape()
+
+# Use manual proxy with ip:port format
+scraper = DynamicScrapingNews("https://example.com", 
+                              manual_proxy=f"{proxy['ip']}:{proxy['port']}")
+result = scraper.scrape()
+
+# Use manual proxy with dictionary format
+scraper = DynamicScrapingNews("https://example.com", 
+                              manual_proxy={"ip": proxy['ip'], "port": proxy['port']})
+result = scraper.scrape()
+```
+
+### Extracted Data Structure
+
+The `scrape()` method returns a dictionary containing:
+
+```python
+{
+    'title': 'Page Title',           # Extracted from og:title or title tag
+    'url': 'https://example.com',    # Canonical URL
+    'image': 'https://...',          # Featured image URL
+    'html': '<html>...</html>',      # Full HTML content
+    'text': 'Clean text content',    # Processed text without HTML
+    'media': 'example',              # Domain name extracted from URL
+    'published_date': datetime(...)  # Publication date if found
+}
+```

pocong-1.1.0/README.md

@@ -0,0 +1,175 @@
+<p align="center">
+  <img src="https://i.ibb.co.com/35P4Nq9x/Screenshot-2025-08-22-at-18-40-11.png?width=128" alt="POCONG Logo" width="128"/>
+</p>
+
+# POCONG 🪦
+**Python Oriented Crawling ON Going**
+
+POCONG is a lightweight web crawling framework built in Python.
+
+## Features
+
+- 🔒 **Get Free Proxy**: Automatic proxy fetching, validation, and rotation from free proxy sources
+- 🌐 **Dynamic Media Web Scraping**: Extract content, metadata, and media information from web pages with proxy support
+- 📱 **Social Media Scraping**: Extract data from social media platforms *(coming soon)*
+- 🛒 **E-commerce Scraping**: Extract product information from e-commerce websites *(coming soon)*
+
+## Installation
+```bash
+pip install pocong
+```
+
+## Usage: Get Proxy from proxy_spiders
+
+You can use the `get_proxy` and `get_proxy_random` methods from `proxy_spiders` to fetch working proxies.
+
+```python
+from pocong.proxy_spiders import GetProxy
+
+gp = GetProxy()
+
+# Get the first working proxy
+proxy = gp.get_proxy()
+print("First working proxy:", proxy)
+```
+```python
+from pocong.proxy_spiders import GetProxy
+
+gp = GetProxy()
+
+# Get a random working proxy
+random_proxy = gp.get_proxy_random()
+print("Random working proxy:", random_proxy)
+```
+
+Sample output:
+```
+First working proxy: {'ip': '123.45.67.89', 'port': '8080', 'https': 'yes', ...}
+Random working proxy: {'ip': '98.76.54.32', 'port': '3128', 'https': 'yes', ...}
+```
+
+You can use the returned proxy dictionary with the `requests` library, for example:
+
+```python
+import requests
+
+proxy = gp.get_proxy()
+if proxy:
+    proxies = {
+        'http': f"http://{proxy['ip']}:{proxy['port']}",
+        'https': f"http://{proxy['ip']}:{proxy['port']}"
+    }
+    response = requests.get('https://httpbin.org/ip', proxies=proxies)
+    print(response.json())
+else:
+    print("No working proxy found.")
+```
+
+- `get_proxy()` will return the first working proxy found.
+- `get_proxy_random()` will return a random working proxy (with up to 20 retries).
+
+Both methods return a dictionary with proxy details (e.g., `{ 'ip': '...', 'port': '...', ... }`) or `None` if no working proxy is found.
+
+## Usage: Dynamic Media Web Scraping
+
+The `DynamicScrapingNews` class provides comprehensive web scraping capabilities with built-in proxy support for extracting content, metadata, and media information from web pages.
+
+### Basic Usage
+
+```python
+from pocong.media_spiders import DynamicScrapingNews
+
+# Simple scraping without proxy
+scraper = DynamicScrapingNews("https://example.com", use_proxy=False)
+result = scraper.scrape()
+
+# Extract specific information
+print(f"Title: {result['title']}")
+print(f"URL: {result['url']}")
+print(f"Media: {result['media']}")
+print(f"Published: {result['published_date']}")
+print(f"Text content: {result['text'][:200]}...")  # First 200 chars
+```
+
+### Proxy Configuration Options
+
+#### 1. Automatic Proxy (Default)
+```python
+# Uses automatic proxy fetching
+scraper = DynamicScrapingNews("https://example.com")
+result = scraper.scrape()
+```
+
+#### 2. Manual Proxy Configuration
+```python
+# Method 1: IP:Port format
+scraper = DynamicScrapingNews("https://example.com", 
+                              manual_proxy="192.168.1.1:8080")
+
+# Method 2: Full URL format
+scraper = DynamicScrapingNews("https://example.com", 
+                              manual_proxy="http://192.168.1.1:8080")
+
+# Method 3: Dictionary format
+scraper = DynamicScrapingNews("https://example.com", 
+                              manual_proxy={"ip": "192.168.1.1", "port": "8080"})
+
+result = scraper.scrape()
+```
+
+#### 3. No Proxy
+```python
+# Disable proxy completely
+scraper = DynamicScrapingNews("https://example.com", use_proxy=False)
+result = scraper.scrape()
+```
+
+#### 4. Manual Proxy Override
+```python
+# Manual proxy overrides use_proxy setting
+scraper = DynamicScrapingNews("https://example.com", 
+                              use_proxy=False, 
+                              manual_proxy="192.168.1.1:8080")
+result = scraper.scrape()
+```
+
+### Complete Example with Proxy Integration
+
+```python
+from pocong.proxy_spiders import GetProxy
+from pocong.media_spiders import DynamicScrapingNews
+
+# Get a working proxy
+proxy = GetProxy().get_proxy()
+print(f"Using proxy: {proxy}")
+
+# Use automatic proxy (default behavior)
+scraper = DynamicScrapingNews("https://example.com")
+result = scraper.scrape()
+
+# Use manual proxy with ip:port format
+scraper = DynamicScrapingNews("https://example.com", 
+                              manual_proxy=f"{proxy['ip']}:{proxy['port']}")
+result = scraper.scrape()
+
+# Use manual proxy with dictionary format
+scraper = DynamicScrapingNews("https://example.com", 
+                              manual_proxy={"ip": proxy['ip'], "port": proxy['port']})
+result = scraper.scrape()
+```
+
+### Extracted Data Structure
+
+The `scrape()` method returns a dictionary containing:
+
+```python
+{
+    'title': 'Page Title',           # Extracted from og:title or title tag
+    'url': 'https://example.com',    # Canonical URL
+    'image': 'https://...',          # Featured image URL
+    'html': '<html>...</html>',      # Full HTML content
+    'text': 'Clean text content',    # Processed text without HTML
+    'media': 'example',              # Domain name extracted from URL
+    'published_date': datetime(...)  # Publication date if found
+}
+```

{pocong-1.0.0 → pocong-1.1.0}/setup.cfg

@@ -46,7 +46,14 @@ package_dir =
 packages = find:
 python_requires = >=3.8
 install_requires = 
+	Scrapy>=2.5.0
+	pandas>=1.3.0
+	requests>=2.25.0
 	Click>=7.0
+	mechanize>=0.4.0
+	html2text>=2020.1.16
+	fake-useragent>=1.1.0
+	beautifulsoup4>=4.9.0
 
 [options.extras_require]
 dev = 

{pocong-1.0.0 → pocong-1.1.0}/setup.py

@@ -17,6 +17,10 @@ install_requires = [
     "pandas>=1.3.0",
     "requests>=2.25.0",
     "Click>=7.0",
+    "mechanize>=0.4.0",
+    "html2text>=2020.1.16",
+    "fake-useragent>=1.1.0",
+    "beautifulsoup4>=4.9.0",
 ]
 
 extras_require = {

{pocong-1.0.0 → pocong-1.1.0}/src/pocong/_version.py

@@ -8,11 +8,11 @@ import json
 
 version_json = '''
 {
- "date": "2025-09-21T17:59:23+0700",
+ "date": "2025-10-11T12:21:52+0700",
  "dirty": false,
  "error": null,
- "full-revisionid": "9a2d0f58b89a546044fc52948ce274767aa450d4",
- "version": "1.0.0"
+ "full-revisionid": "4578ac0583c995b27ae2ec3e0297768410ee0bbc",
+ "version": "1.1.0"
 }
 '''  # END VERSION_JSON
 

pocong-1.1.0/src/pocong/media_spiders/__init__.py

@@ -0,0 +1,231 @@
+import datetime as dt
+import re
+
+import mechanize
+import requests
+from bs4 import BeautifulSoup
+from fake_useragent import UserAgent
+from html2text import html2text
+
+try:
+    from pocong.proxy_spiders import GetProxy
+    PROXY_AVAILABLE = True
+except ImportError:
+    PROXY_AVAILABLE = False
+
+
+class DynamicScrapingNews():
+    def __init__(self, url, use_proxy=True, manual_proxy=None):
+        self.url = url
+        self.use_proxy = use_proxy
+        self.proxy = None
+
+        # Use manual proxy if provided
+        if manual_proxy:
+            self.proxy = self._format_proxy(manual_proxy)
+        # Otherwise, initialize proxy if available and requested
+        elif self.use_proxy and PROXY_AVAILABLE:
+            try:
+                proxy_getter = GetProxy()
+                proxy_data = proxy_getter.get_proxy_random()
+                if proxy_data:
+                    self.proxy = f"http://{proxy_data['ip']}:{proxy_data['port']}"
+            except Exception:
+                # If proxy initialization fails, continue without proxy
+                self.proxy = None
+
+    def _format_proxy(self, proxy):
+        """
+        Format proxy to ensure it has the correct format.
+        Accepts formats like:
+        - "ip:port"
+        - "http://ip:port"
+        - "https://ip:port"
+        - {"ip": "x.x.x.x", "port": "xxxx"}
+        """
+        if isinstance(proxy, dict):
+            # If proxy is a dict with ip and port
+            if 'ip' in proxy and 'port' in proxy:
+                return f"http://{proxy['ip']}:{proxy['port']}"
+            else:
+                raise ValueError("Manual proxy dict must contain 'ip' and 'port' keys")
+        elif isinstance(proxy, str):
+            # If proxy is a string
+            if proxy.startswith(('http://', 'https://')):
+                return proxy
+            else:
+                # Assume it's in ip:port format
+                return f"http://{proxy}"
+        else:
+            raise ValueError("Manual proxy must be a string or dict")
+
+    def _remove_html_tags(self, text):
+        # This regular expression will match any HTML tag and capture its contents.
+        html_tags_pattern = r'<.*?>'
+        # Use re.sub to replace all matches with an empty string.
+        clean_text = re.sub(html_tags_pattern, '', text)
+        return clean_text
+
+    def _get_metadata(self, html, list_metadata=['title', 'url', 'image']):
+        result = dict()
+        for metadata in list_metadata:
+            # Define the regular expression pattern
+            pattern = r'property="og:{}" content="([^"]+)"'.format(metadata)
+
+            # Search for the pattern in the HTML content
+            match = re.search(pattern, html)
+
+            if match:
+                # Extract the content from the matched group
+                og_content = match.group(1)
+                result[metadata] = og_content if '?' not in og_content else og_content.split('?')[0]
+            else:
+                if metadata == 'url':
+                    result[metadata] = self.url if '?' not in self.url else self.url.split('?')[0]
+                else:
+                    result[metadata] = "Pattern not found in the HTML content."
+            result[metadata] = self._remove_html_tags(BeautifulSoup(result[metadata], 'html.parser').get_text())
+            result[metadata] = re.sub(r"&amp;", "&", result[metadata])
+        return result
+
+    def _clean_html_to_text(self, html):
+        # First we remove inline JavaScript/CSS:
+        cleaned = re.sub(r"(?is)<(script|style).*?>.*?(</\1>)", "", html.strip())
+        # Then we remove html comments. This has to be done before removing regular
+        # tags since comments can contain '>' characters.
+        cleaned = re.sub(r"(?s)<!--(.*?)-->[\n]?", "", cleaned)
+        # Next we can remove the remaining tags:
+        cleaned = re.sub(r"(?s)<.*?>", " ", cleaned)
+        # Finally, we deal with whitespace
+        cleaned = re.sub(r"&nbsp;", " ", cleaned)
+        cleaned = re.sub(r"  ", " ", cleaned)
+        cleaned = re.sub(r"  ", " ", cleaned)
+        text = html2text(cleaned).format('utf-8')
+        spe_char = [
+            '\\u0621', '\\u0622', '\\u0625', '\\u0627', '\\u0629', '\\u062a', '\\u062b', '\\u062c', '\\u062f',
+            '\\u0631', '\\u0632', '\\u0633', '\\u0634', '\\u0636', '\\u0637', '\\u0639', '\\u063a', '\\u0641',
+            '\\u0643', '\\u0644', '\\u0645', '\\u0646', '\\u0647', '\\u0648', '\\u064a'
+        ]
+        for char in spe_char:
+            text = text.replace(char, '')
+        return text.strip()
+
+    def _get_media(self, url):
+        # Define a regular expression pattern to match the main domain (excluding "sport" and subdomains)
+        pattern = r"https?://(?:www\.)?(?:[^./]+\.)*([^.]+\.\w+)"
+
+        # Use re.search to find the first match
+        match = re.search(pattern, url.replace('.co.', '.'))
+
+        # Extract the matched domain
+        if match:
+            domain = match.group(1)
+            return domain.split('.')[0]
+        else:
+            return None
+
+    def _get_pubdate(self, html):
+        # Define a regular expression pattern to match the content attribute value
+        pattern = r'content="(\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2})"'
+
+        # Use re.search to find the first match
+        match = re.search(pattern, html)
+
+        # Extract the matched content attribute value
+        if match:
+            content_value = match.group(1)
+
+            # Convert the content value to a datetime format
+            datetime_format = "%Y/%m/%d %H:%M:%S"
+            parsed_datetime = dt.datetime.strptime(content_value, datetime_format)
+
+            return parsed_datetime
+        else:
+            # Define a regular expression pattern to match the content attribute value
+            pattern = r'content="(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})"'
+
+            # Use re.search to find the first match
+            match = re.search(pattern, html)
+
+            # Extract the matched content attribute value
+            if match:
+                content_value = match.group(1)
+
+                # Convert the content value to a datetime format
+                datetime_format = "%Y-%m-%d %H:%M:%S"
+                parsed_datetime = dt.datetime.strptime(content_value, datetime_format)
+
+                return parsed_datetime
+            else:
+                # Define a regular expression pattern to match the content attribute value
+                pattern = r'content="(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\+\d{2}:\d{2})"'
+
+                # Use re.search to find the first match
+                match = re.search(pattern, html)
+
+                # Extract the matched content attribute value
+                if match:
+                    content_value = match.group(1)
+
+                    # Convert the content value to a datetime format
+                    datetime_format = "%Y-%m-%dT%H:%M:%S%z"
+                    parsed_datetime = dt.datetime.strptime(content_value, datetime_format)
+
+                    return parsed_datetime
+                else:
+                    return None
+
+    def _get_html(self, url):
+        # random useragent
+        ua = UserAgent()
+        user_agent = ua.random
+        headers = {'User-Agent': user_agent}
+
+        # Try with mechanize first (with proxy if available)
+        try:
+            br = mechanize.Browser()
+            br.set_handle_robots(False)
+            br.addheaders = [('User-Agent', user_agent)]
+
+            # Set proxy for mechanize if available
+            if self.proxy:
+                br.set_proxies({'http': self.proxy, 'https': self.proxy})
+
+            html = br.open(url).read().decode('utf-8')
+            return html
+        except Exception:
+            # Fallback to requests (with proxy if available)
+            try:
+                proxies = {'http': self.proxy, 'https': self.proxy} if self.proxy else None
+                response = requests.get(url, headers=headers, proxies=proxies, timeout=30)
+                html = response.content.decode('utf-8')
+                return html
+            except Exception:
+                # Final fallback without proxy
+                response = requests.get(url, headers=headers, timeout=30)
+                html = response.content.decode('utf-8')
+                return html
+
+    def scrape(self):
+        # get html from url
+        html = self._get_html(self.url)
+
+        # get metadata
+        metadata = self._get_metadata(html)
+
+        # convert html to text
+        text = self._clean_html_to_text(html)
+
+        # get media from url
+        media = self._get_media(self.url)
+
+        # get published_date from html
+        published_date = self._get_pubdate(html)
+
+        # combine result
+        metadata['html'] = html
+        metadata['text'] = text
+        metadata['media'] = media
+        metadata['published_date'] = published_date
+
+        return metadata