telegram-pm 0.1.0__tar.gz → 0.1.2__tar.gz

{telegram_pm-0.1.0 → telegram_pm-0.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: telegram-pm
-Version: 0.1.0
+Version: 0.1.2
 Summary: Telegram preview page parser
 Author: Your Name
 Author-email: you@example.com
@@ -32,15 +32,20 @@ Tool for monitoring public Telegram channels available in WEB preview mode
 4. [x] Support for forwarded messages and replies
 5. [x] Configurable data collection parameters
 
+
 ## 🛠 Installation
 1. Ensure Python 3.12+ is installed (recommendation)
 2. Clone repository
 ```bash
    git clone 'https://github.com/aIligat0r/tpm.git'
 ```
+or
+```bash
+pip install telegram-pm
+```
 
 ## ⚙️ Configuration
-Configurations (file `.env` or `tpm/config.py`)
+Configurations (file `.env` or `telegram_pm/config.py`)
 
 Parsing configurations:
 * `TELEGRAM_PARSE_REPEAT_COUNT` - Number of requests (default `5`). 20 messages per request. (1 iter - last 20 messages)
@@ -58,7 +63,7 @@ HTTP configurations:
 
 Build docker image:
 ```bash
-docker build -t telegram_pm .
+docker build -t tpm .
 ```
 Create poetry env:
 * Install poetry:
@@ -80,17 +85,17 @@ poetry install
 | `--verbose`/`--v`                 | Verbose mode                                                          | ➖                                                              |
 | `--help`/`--h`                    | Help information                                                      | ➖                                                              |
 
-**Poetry**:
+**Poetry:**
 ```bash
-poetry run telegram_pm --ch freegaza --ch BREAKINGNewsTG --db-path .\tg.db --v
+poetry run tpm --ch freegaza --ch BREAKINGNewsTG --db-path .\tg.db --v
 ```
 or
 ```bash
-poetry run telegram_pm --channels-filepath /path/to/monitoring_usernames.txt --db-path .\tg.db
+poetry run tpm --channels-filepath /path/to/monitoring_usernames.txt --db-path .\tg.db
 ```
-**Docker**:
+**Docker:**
 ```bash
-docker run -it --rm telegram_pm --ch freegaza --db-path test_tg.db --v
+docker run -it --rm tpm --ch freegaza --db-path test_tg.db --v
 ```
 or (if you want to transfer usernames in a file, then you need to mount the paths)
 ```bash
@@ -104,6 +109,29 @@ docker run -it --rm \
     -v ~/tpm_data_dir/usernames.txt:/data/usernames.txt \
     telegram_pm --db-path /data/telegram_messages.sqlite --chf /data/usernames.txt
 ```
+**Python:**
+```python
+from telegram_pm.run import run_tpm
+
+
+run_tpm(
+    db_path="tg.db",                    # Path to sqlite database
+    channels=["channel1", "channel2"],  # Channels list
+    verbose=True,                       # Verbose mode
+
+    # Configuration (optional)
+    tg_iteration_in_preview_count=5,    # Number of requests (default 5). 20 messages per request. (1 iter - last 20 messages)
+    tg_sleep_time_seconds=60,           # Number of seconds after which the next process of receiving data from channels will begin (default 60 seconds)
+    tg_sleep_after_error_request=30,    # Waiting after a failed requests (default 30)
+    http_retries=3,                     # Number of repeated request attempts (default 3)
+    http_backoff=3,                     # Delay between attempts for failed requests (default 3 seconds)
+    http_timeout=60,                    # Waiting for a response (default 30 seconds)
+    http_headers={                      # HTTP headers
+        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36"
+    }
+)
+```
+
 ## 🗃️ Database Structure
 
 The tables will be named as usernames. Each table is a username that was passed in the running parameters.

{telegram_pm-0.1.0 → telegram_pm-0.1.2}/README.md

@@ -12,15 +12,20 @@ Tool for monitoring public Telegram channels available in WEB preview mode
 4. [x] Support for forwarded messages and replies
 5. [x] Configurable data collection parameters
 
+
 ## 🛠 Installation
 1. Ensure Python 3.12+ is installed (recommendation)
 2. Clone repository
 ```bash
    git clone 'https://github.com/aIligat0r/tpm.git'
 ```
+or
+```bash
+pip install telegram-pm
+```
 
 ## ⚙️ Configuration
-Configurations (file `.env` or `tpm/config.py`)
+Configurations (file `.env` or `telegram_pm/config.py`)
 
 Parsing configurations:
 * `TELEGRAM_PARSE_REPEAT_COUNT` - Number of requests (default `5`). 20 messages per request. (1 iter - last 20 messages)
@@ -38,7 +43,7 @@ HTTP configurations:
 
 Build docker image:
 ```bash
-docker build -t telegram_pm .
+docker build -t tpm .
 ```
 Create poetry env:
 * Install poetry:
@@ -60,17 +65,17 @@ poetry install
 | `--verbose`/`--v`                 | Verbose mode                                                          | ➖                                                              |
 | `--help`/`--h`                    | Help information                                                      | ➖                                                              |
 
-**Poetry**:
+**Poetry:**
 ```bash
-poetry run telegram_pm --ch freegaza --ch BREAKINGNewsTG --db-path .\tg.db --v
+poetry run tpm --ch freegaza --ch BREAKINGNewsTG --db-path .\tg.db --v
 ```
 or
 ```bash
-poetry run telegram_pm --channels-filepath /path/to/monitoring_usernames.txt --db-path .\tg.db
+poetry run tpm --channels-filepath /path/to/monitoring_usernames.txt --db-path .\tg.db
 ```
-**Docker**:
+**Docker:**
 ```bash
-docker run -it --rm telegram_pm --ch freegaza --db-path test_tg.db --v
+docker run -it --rm tpm --ch freegaza --db-path test_tg.db --v
 ```
 or (if you want to transfer usernames in a file, then you need to mount the paths)
 ```bash
@@ -84,6 +89,29 @@ docker run -it --rm \
     -v ~/tpm_data_dir/usernames.txt:/data/usernames.txt \
     telegram_pm --db-path /data/telegram_messages.sqlite --chf /data/usernames.txt
 ```
+**Python:**
+```python
+from telegram_pm.run import run_tpm
+
+
+run_tpm(
+    db_path="tg.db",                    # Path to sqlite database
+    channels=["channel1", "channel2"],  # Channels list
+    verbose=True,                       # Verbose mode
+
+    # Configuration (optional)
+    tg_iteration_in_preview_count=5,    # Number of requests (default 5). 20 messages per request. (1 iter - last 20 messages)
+    tg_sleep_time_seconds=60,           # Number of seconds after which the next process of receiving data from channels will begin (default 60 seconds)
+    tg_sleep_after_error_request=30,    # Waiting after a failed requests (default 30)
+    http_retries=3,                     # Number of repeated request attempts (default 3)
+    http_backoff=3,                     # Delay between attempts for failed requests (default 3 seconds)
+    http_timeout=60,                    # Waiting for a response (default 30 seconds)
+    http_headers={                      # HTTP headers
+        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36"
+    }
+)
+```
+
 ## 🗃️ Database Structure
 
 The tables will be named as usernames. Each table is a username that was passed in the running parameters.

{telegram_pm-0.1.0 → telegram_pm-0.1.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "telegram-pm"
-version = "0.1.0"
+version = "0.1.2"
 description = "Telegram preview page parser"
 authors = [{name = "Your Name",email = "you@example.com"}]
 readme = "README.md"

{telegram_pm-0.1.0 → telegram_pm-0.1.2}/telegram_pm/config.py

@@ -12,6 +12,9 @@ class HttpClientConfig(BaseConfig):
     retries: int = int(environ.get("HTTP_RETRIES", 3))
     backoff: int = int(environ.get("HTTP_BACKOFF", 3))
     timeout: int = int(environ.get("HTTP_TIMEOUT", 30))
+    headers: dict[str, str] = {
+        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36"
+    }
 
 
 class TelegramConfig(BaseConfig):

telegram_pm-0.1.2/telegram_pm/http_client/client.py

@@ -0,0 +1,36 @@
+import httpx
+from retry import retry
+
+from telegram_pm.utils.logger import logger
+from telegram_pm.config import HttpClientConfig
+
+
+class HttpClient:
+    def __init__(
+        self,
+        retries: int = HttpClientConfig.retries,
+        timeout: int = HttpClientConfig.timeout,
+        backoff: int = HttpClientConfig.backoff,
+        headers: dict[str, str] = HttpClientConfig.headers,
+    ):
+        self._headers = headers
+        self._backoff = backoff
+        self._retries = retries
+        self.client = httpx.AsyncClient(
+            transport=httpx.AsyncHTTPTransport(
+                verify=False,
+                retries=retries,
+            ),
+            timeout=timeout,
+            verify=False,
+        )
+
+    async def request(self, url: str, method: str = "GET", **kwargs) -> httpx.Response:
+        @retry(backoff=self._backoff, logger=logger)  # type: ignore[arg-type]
+        async def nested_request() -> httpx.Response:
+            response = await self.client.request(
+                method=method, url=url, headers=self._headers, **kwargs
+            )
+            return response
+
+        return await nested_request()

{telegram_pm-0.1.0 → telegram_pm-0.1.2}/telegram_pm/parsers/preview.py

@@ -4,10 +4,9 @@ import httpx
 from bs4 import BeautifulSoup
 from structlog.contextvars import bound_contextvars
 
-from telegram_pm import utils
+from telegram_pm import utils, config
 from telegram_pm.entities import Post
 from telegram_pm.utils.logger import logger
-from telegram_pm.config import TelegramConfig
 from telegram_pm.parsers.base import BaseParser
 from telegram_pm.parsers.post import PostsParser
 from telegram_pm.http_client.client import HttpClient
@@ -24,9 +23,40 @@ class PreviewParser(BaseParser):
         channels: list[str],
         db_path: str,
         verbose: bool = False,
+        tg_before_param_size: int = config.TelegramConfig.before_param_size,
+        tg_iteration_in_preview_count: int = config.TelegramConfig.iteration_in_preview_count,
+        tg_sleep_time_seconds: int = config.TelegramConfig.sleep_time_seconds,
+        tg_sleep_after_error_request: int = config.TelegramConfig.sleep_after_error_request,
+        http_retries: int = config.HttpClientConfig.retries,
+        http_backoff: int = config.HttpClientConfig.backoff,
+        http_timeout: int = config.HttpClientConfig.timeout,
+        http_headers: dict[str, str] = config.HttpClientConfig.headers,
     ):
+        """
+        :param db_path: Path to sqlite database
+        :param channels: Channels list
+        :param verbose: Verbose mode
+        :param tg_before_param_size: 20 messages per request. (1 iter - last 20 messages)
+        :param tg_iteration_in_preview_count: Number of requests (default 5). 20 messages per request. (1 iter - last 20 messages)
+        :param tg_sleep_time_seconds:  Number of seconds after which the next process of receiving data from channels will begin (default 60 seconds)
+        :param tg_sleep_after_error_request: Waiting after a failed requests (default 30)
+        :param http_retries: Number of repeated request attempts (default 3)
+        :param http_backoff: Delay between attempts for failed requests (default 3 seconds)
+        :param http_timeout: Waiting for a response (default 30 seconds)
+        :param http_headers: HTTP headers
+        """
+        self._tg_sleep_after_error_request = tg_sleep_after_error_request
+        self._tg_sleep_time_seconds = tg_sleep_time_seconds
+        self._tg_iteration_in_preview_count = tg_iteration_in_preview_count
+        self._tg_before_param_size = tg_before_param_size
+
         self.channels: list[str] = channels
-        self.http_client = HttpClient()
+        self.http_client = HttpClient(
+            retries=http_retries,
+            backoff=http_backoff,
+            timeout=http_timeout,
+            headers=http_headers,
+        )
         self.post_parser = PostsParser(verbose=verbose)
         self.db = DatabaseProcessor(db_path=db_path)
         self._db_initialized = False
@@ -89,7 +119,7 @@ class PreviewParser(BaseParser):
             posts_result = []
             should_break = False
 
-            for parse_repeat in range(TelegramConfig.iteration_in_preview_count):
+            for parse_repeat in range(self._tg_iteration_in_preview_count):
                 if should_break:
                     await logger.ainfo("No new posts yet")
                     break
@@ -98,7 +128,7 @@ class PreviewParser(BaseParser):
                     response = await self._get_preview_page(preview_url=preview_url)
                     if not response:
                         await logger.awarning("Can not get preview page")
-                        await asyncio.sleep(TelegramConfig.sleep_after_error_request)
+                        await asyncio.sleep(self._tg_sleep_after_error_request)
                         continue
 
                     if self.__forbidden_parse_preview(response=response):
@@ -111,7 +141,7 @@ class PreviewParser(BaseParser):
                     if not parsed_posts:
                         await logger.awarning("No posts parsed from preview page")  # type: ignore
                         await self.db.drop_table_if_empty(channel_username)
-                        await asyncio.sleep(TelegramConfig.sleep_after_error_request)
+                        await asyncio.sleep(self._tg_sleep_after_error_request)
                         break
 
                     first_post_exists = await self.db.post_exists(
@@ -127,11 +157,11 @@ class PreviewParser(BaseParser):
                     before_param_number = self.__parse_before_param_value(
                         post_url=parsed_posts[-1].url
                     )
-                    if before_param_number <= TelegramConfig.before_param_size:
-                        before_param_number -= TelegramConfig.before_param_size
+                    if before_param_number <= self._tg_before_param_size:
+                        before_param_number -= self._tg_before_param_size
                     else:
                         before_param_number = (
-                            before_param_number - TelegramConfig.before_param_size
+                            before_param_number - self._tg_before_param_size
                         )
                         if before_param_number <= 0:
                             break

telegram_pm-0.1.2/telegram_pm/run.py

@@ -0,0 +1,110 @@
+import sys
+import signal
+import asyncio
+
+from telegram_pm import config
+from telegram_pm.utils.logger import logger
+from telegram_pm.config import TelegramConfig
+from telegram_pm.parsers.preview import PreviewParser
+
+
+class ParserRunner:
+    def __init__(
+        self,
+    ):
+        self._shutdown = False
+
+        # Setup signal handlers
+        signal.signal(signal.SIGINT, self.handle_signal)
+        signal.signal(signal.SIGTERM, self.handle_signal)
+
+    def handle_signal(self, signum, frame):
+        logger.info(f"Received signal {signum}, shutting down...")
+        self._shutdown = True
+        sys.exit(0)
+
+    async def run(
+        self,
+        db_path: str,
+        channels: list[str],
+        verbose: bool = False,
+        tg_before_param_size: int = config.TelegramConfig.before_param_size,
+        tg_iteration_in_preview_count: int = config.TelegramConfig.iteration_in_preview_count,
+        tg_sleep_time_seconds: int = config.TelegramConfig.sleep_time_seconds,
+        tg_sleep_after_error_request: int = config.TelegramConfig.sleep_after_error_request,
+        http_retries: int = config.HttpClientConfig.retries,
+        http_backoff: int = config.HttpClientConfig.backoff,
+        http_timeout: int = config.HttpClientConfig.timeout,
+        http_headers: dict[str, str] = config.HttpClientConfig.headers,
+    ):
+        """
+        :param db_path: Path to sqlite database
+        :param channels: Channels list
+        :param verbose: Verbose mode
+        :param tg_before_param_size: 20 messages per request. (1 iter - last 20 messages)
+        :param tg_iteration_in_preview_count: Number of requests (default 5). 20 messages per request. (1 iter - last 20 messages)
+        :param tg_sleep_time_seconds:  Number of seconds after which the next process of receiving data from channels will begin (default 60 seconds)
+        :param tg_sleep_after_error_request: Waiting after a failed requests (default 30)
+        :param http_retries: Number of repeated request attempts (default 3)
+        :param http_backoff: Delay between attempts for failed requests (default 3 seconds)
+        :param http_timeout: Waiting for a response (default 30 seconds)
+        :param http_headers: HTTP headers
+        """
+        parser = PreviewParser(
+            channels=channels,
+            verbose=verbose,
+            db_path=db_path,
+            tg_before_param_size=tg_before_param_size,
+            tg_iteration_in_preview_count=tg_iteration_in_preview_count,
+            tg_sleep_time_seconds=tg_sleep_time_seconds,
+            tg_sleep_after_error_request=tg_sleep_after_error_request,
+            http_retries=http_retries,
+            http_backoff=http_backoff,
+            http_timeout=http_timeout,
+            http_headers=http_headers,
+        )
+        try:
+            while not self._shutdown:
+                try:
+                    await parser.parse()
+                    logger.info(
+                        f"💤 Sleep {TelegramConfig.sleep_time_seconds} seconds ... 💤"
+                    )
+                    await asyncio.sleep(TelegramConfig.sleep_time_seconds)
+                except Exception as e:
+                    logger.error(f"Error during parsing: {e}")
+                    await asyncio.sleep(TelegramConfig.sleep_after_error_request)
+        finally:
+            if parser:
+                await parser.close()
+
+
+def run_tpm(
+    db_path: str,
+    channels: list[str],
+    verbose: bool = False,
+    tg_before_param_size: int = config.TelegramConfig.before_param_size,
+    tg_iteration_in_preview_count: int = config.TelegramConfig.iteration_in_preview_count,
+    tg_sleep_time_seconds: int = config.TelegramConfig.sleep_time_seconds,
+    tg_sleep_after_error_request: int = config.TelegramConfig.sleep_after_error_request,
+    http_retries: int = config.HttpClientConfig.retries,
+    http_backoff: int = config.HttpClientConfig.backoff,
+    http_timeout: int = config.HttpClientConfig.timeout,
+    http_headers: dict[str, str] = config.HttpClientConfig.headers,
+):
+    runner = ParserRunner()
+    asyncio.run(
+        runner.run(
+            channels=channels,
+            verbose=verbose,
+            db_path=db_path,
+            tg_before_param_size=tg_before_param_size,
+            tg_iteration_in_preview_count=tg_iteration_in_preview_count,
+            tg_sleep_time_seconds=tg_sleep_time_seconds,
+            tg_sleep_after_error_request=tg_sleep_after_error_request,
+            http_retries=http_retries,
+            http_backoff=http_backoff,
+            http_timeout=http_timeout,
+            http_headers=http_headers,
+        )
+    )

telegram_pm-0.1.0/telegram_pm/http_client/client.py

@@ -1,22 +0,0 @@
-import httpx
-from retry import retry
-
-from telegram_pm.utils.logger import logger
-from telegram_pm.config import HttpClientConfig
-
-
-class HttpClient:
-    def __init__(self):
-        self.client = httpx.AsyncClient(
-            transport=httpx.AsyncHTTPTransport(
-                verify=False,
-                retries=HttpClientConfig.retries,
-            ),
-            timeout=HttpClientConfig.timeout,
-            verify=False,
-        )
-
-    @retry(backoff=HttpClientConfig.backoff, logger=logger)  # type: ignore[arg-type]
-    async def request(self, url: str, method: str = "GET", **kwargs) -> httpx.Response:
-        response = await self.client.request(method=method, url=url, **kwargs)
-        return response

telegram_pm-0.1.0/telegram_pm/run.py

@@ -1,49 +0,0 @@
-import sys
-import signal
-import asyncio
-
-from telegram_pm.parsers.preview import PreviewParser
-from telegram_pm.utils.logger import logger
-from telegram_pm.config import TelegramConfig
-
-
-class ParserRunner:
-    def __init__(self, db_path: str, channels: list[str], verbose: bool = False):
-        self.db_path = db_path
-        self.channels = channels
-        self.verbose = verbose
-
-        self._shutdown = False
-
-        # Setup signal handlers
-        signal.signal(signal.SIGINT, self.handle_signal)
-        signal.signal(signal.SIGTERM, self.handle_signal)
-
-    def handle_signal(self, signum, frame):
-        logger.info(f"Received signal {signum}, shutting down...")
-        self._shutdown = True
-        sys.exit(0)
-
-    async def run(self):
-        parser = PreviewParser(
-            channels=self.channels, verbose=self.verbose, db_path=self.db_path
-        )
-        try:
-            while not self._shutdown:
-                try:
-                    await parser.parse()
-                    logger.info(
-                        f"💤 Sleep {TelegramConfig.sleep_time_seconds} seconds ... 💤"
-                    )
-                    await asyncio.sleep(TelegramConfig.sleep_time_seconds)
-                except Exception as e:
-                    logger.error(f"Error during parsing: {e}")
-                    await asyncio.sleep(TelegramConfig.sleep_after_error_request)
-        finally:
-            if parser:
-                await parser.close()
-
-
-def run_parser(db_path: str, channels: list[str], verbose: bool = False):
-    runner = ParserRunner(channels=channels, verbose=verbose, db_path=db_path)
-    asyncio.run(runner.run())