warp-beacon 2.3.38__tar.gz → 2.3.40__tar.gz

{warp_beacon-2.3.38/warp_beacon.egg-info → warp_beacon-2.3.40}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: warp_beacon
-Version: 2.3.38
+Version: 2.3.40
 Summary: Telegram bot for expanding external media links
 Home-page: https://github.com/sb0y/warp_beacon
 Author: Andrey Bagrintsev
@@ -237,10 +237,13 @@ Requires-Dist: pillow
 Requires-Dist: pymongo
 Requires-Dist: instagrapi==2.0.0
 Requires-Dist: bs4
+Requires-Dist: yt_dlp
 Dynamic: author
 Dynamic: home-page
+Dynamic: license-file
 
 # warp_beacon [![Upload Python Package](https://github.com/sb0y/warp_beacon/actions/workflows/python-publish.yml/badge.svg)](https://github.com/sb0y/warp_beacon/actions/workflows/python-publish.yml) [![Docker Image CI](https://github.com/sb0y/warp_beacon/actions/workflows/docker-image.yml/badge.svg)](https://github.com/sb0y/warp_beacon/actions/workflows/docker-image.yml) [![Build DEB package](https://github.com/sb0y/warp_beacon/actions/workflows/build-deb.yml/badge.svg)](https://github.com/sb0y/warp_beacon/actions/workflows/build-deb.yml)
+> because content should travel freely
 
 Telegram bot for external social networks media links expanding.
 Works with a links in personal messages and also with group chats.
@@ -253,6 +256,24 @@ Just send a media link to the chat with bot and get a video or audio reply.
 |<img width="700" alt="Instagram Photo post usage example" src="https://github.com/user-attachments/assets/29324b94-7314-4a38-8790-3483011d355d"> Instagram Photo post usage example|<img width="700" alt="Instagram Photo Carousel usage" src="https://github.com/user-attachments/assets/2598e329-e16e-455e-91e9-a027e8994283"> Instagram Photo Carousel usage example|<img width="757" alt="Instagram Photo bulk Strories download usage example" src="https://github.com/user-attachments/assets/2c8c91ac-6ade-4d1d-a677-2b36bb40ff39"> Instagram Photo bulk Strories download usage example|
 |<img width="700" alt="Instagram specific Story download usage example" src="https://github.com/user-attachments/assets/03dc70c5-6933-4122-9c7c-5f7d734d117b"> Instagram specific Story download usage example|<img width="700" alt="Group chat usage example" src="https://github.com/user-attachments/assets/649fcb1e-785b-4efd-9153-69644c6d898b"> Group chat usage example|
 
+### **Warp Beacon manifesto**
+
+> Once, the Internet was built as a borderless network — a space where knowledge, culture, and ideas could flow freely across the globe.  
+> But over time, freedom gave way to artificial walls, anti-bot shields, and region locks.
+>
+> **warp_beacon** is our answer to that shift.
+>
+> This is a tool for those who refuse to accept "access denied by geolocation" or "content unavailable in your region."  
+> It is a bridge over ML filters, CAPTCHAs, and man-made barriers.
+>
+> We don’t break the rules — we restore the original spirit of the Internet:  
+> 📡 **free exchange of information**,  
+> 🌍 **unrestricted access to global content**,  
+> 🤖 **tools that serve the user**, not the platform.
+>
+> **warp_beacon** — the freedom to deliver content where it’s needed most.
+
+## Configuration example ##
 
 In order to setup your own instance, you will need:
 
@@ -261,8 +282,6 @@ In order to setup your own instance, you will need:
 
 All bot configuration stored in [warp_beacon.conf](https://github.com/sb0y/warp_beacon/blob/main/etc/warp_beacon.conf) file.
 
-## Configuration example ##
-
 ```env
 TG_TOKEN="you telegram token received from @BotFather"
 # these 3 settings should be obtained at https://my.telegram.org/apps

{warp_beacon-2.3.38 → warp_beacon-2.3.40}/README.md

@@ -1,4 +1,5 @@
 # warp_beacon [![Upload Python Package](https://github.com/sb0y/warp_beacon/actions/workflows/python-publish.yml/badge.svg)](https://github.com/sb0y/warp_beacon/actions/workflows/python-publish.yml) [![Docker Image CI](https://github.com/sb0y/warp_beacon/actions/workflows/docker-image.yml/badge.svg)](https://github.com/sb0y/warp_beacon/actions/workflows/docker-image.yml) [![Build DEB package](https://github.com/sb0y/warp_beacon/actions/workflows/build-deb.yml/badge.svg)](https://github.com/sb0y/warp_beacon/actions/workflows/build-deb.yml)
+> because content should travel freely
 
 Telegram bot for external social networks media links expanding.
 Works with a links in personal messages and also with group chats.
@@ -11,6 +12,24 @@ Just send a media link to the chat with bot and get a video or audio reply.
 |<img width="700" alt="Instagram Photo post usage example" src="https://github.com/user-attachments/assets/29324b94-7314-4a38-8790-3483011d355d"> Instagram Photo post usage example|<img width="700" alt="Instagram Photo Carousel usage" src="https://github.com/user-attachments/assets/2598e329-e16e-455e-91e9-a027e8994283"> Instagram Photo Carousel usage example|<img width="757" alt="Instagram Photo bulk Strories download usage example" src="https://github.com/user-attachments/assets/2c8c91ac-6ade-4d1d-a677-2b36bb40ff39"> Instagram Photo bulk Strories download usage example|
 |<img width="700" alt="Instagram specific Story download usage example" src="https://github.com/user-attachments/assets/03dc70c5-6933-4122-9c7c-5f7d734d117b"> Instagram specific Story download usage example|<img width="700" alt="Group chat usage example" src="https://github.com/user-attachments/assets/649fcb1e-785b-4efd-9153-69644c6d898b"> Group chat usage example|
 
+### **Warp Beacon manifesto**
+
+> Once, the Internet was built as a borderless network — a space where knowledge, culture, and ideas could flow freely across the globe.  
+> But over time, freedom gave way to artificial walls, anti-bot shields, and region locks.
+>
+> **warp_beacon** is our answer to that shift.
+>
+> This is a tool for those who refuse to accept "access denied by geolocation" or "content unavailable in your region."  
+> It is a bridge over ML filters, CAPTCHAs, and man-made barriers.
+>
+> We don’t break the rules — we restore the original spirit of the Internet:  
+> 📡 **free exchange of information**,  
+> 🌍 **unrestricted access to global content**,  
+> 🤖 **tools that serve the user**, not the platform.
+>
+> **warp_beacon** — the freedom to deliver content where it’s needed most.
+
+## Configuration example ##
 
 In order to setup your own instance, you will need:
 
@@ -19,8 +38,6 @@ In order to setup your own instance, you will need:
 
 All bot configuration stored in [warp_beacon.conf](https://github.com/sb0y/warp_beacon/blob/main/etc/warp_beacon.conf) file.
 
-## Configuration example ##
-
 ```env
 TG_TOKEN="you telegram token received from @BotFather"
 # these 3 settings should be obtained at https://my.telegram.org/apps

{warp_beacon-2.3.38 → warp_beacon-2.3.40}/pyproject.toml

@@ -25,7 +25,8 @@ dependencies = [
 	"pillow",
 	"pymongo",
 	"instagrapi == 2.0.0",
-	"bs4"
+	"bs4",
+	"yt_dlp"
 ]
 keywords = ["bot", "telegram", "instagram", "export"]
 classifiers = [

warp_beacon-2.3.40/warp_beacon/__version__.py

@@ -0,0 +1,2 @@
+__version__ = "2.3.40"
+

{warp_beacon-2.3.38 → warp_beacon-2.3.40}/warp_beacon/scheduler/instagram_human.py

@@ -80,6 +80,7 @@ class InstagramHuman(object):
 			if random.random() > 0.6:
 				logging.info("Simulation profile view ...")
 				self.profile_view()
+				self.random_pause()
 		except Exception as e:
 			logging.warning("Error in evening_routine")
 			logging.exception(e)
@@ -101,7 +102,9 @@ class InstagramHuman(object):
 
 	def profile_view(self) -> None:
 		try:
+			logging.info("profile_view ...")
 			my_user_id = self.scrapler.cl.user_id
+			logging.info("user_following ...")
 			friends = list(self.scrapler.download_hndlr(self.scrapler.cl.user_following, my_user_id, amount=random.randint(5, 50)).values())
 			time.sleep(random.uniform(2, 5))
 			if not friends:
@@ -111,15 +114,18 @@ class InstagramHuman(object):
 			target_user_id = ""
 			if isinstance(random_friend, UserShort):
 				target_user_id = random_friend.pk
+				logging.info("user_info with target_user_id = '%s' ...", target_user_id)
 				self.scrapler.download_hndlr(self.scrapler.cl.user_info, target_user_id)
 				time.sleep(random.uniform(2, 5))
 			elif isinstance(random_friend, str):
 				target_user_id = self.scrapler.download_hndlr(self.scrapler.cl.user_id_from_username, random_friend)
+				logging.info("user_info with target_user_id = '%s' ...", target_user_id)
 				self.scrapler.download_hndlr(self.scrapler.cl.user_info, target_user_id)
 
 			time.sleep(random.uniform(2, 5))
 
 			if random.random() > 0.7:
+				logging.info("user_medias with target_user_id = '%s' ...", target_user_id)
 				self.scrapler.download_hndlr(self.scrapler.cl.user_medias, target_user_id, amount=random.randint(1, 5))
 
 			self.random_pause()

{warp_beacon-2.3.38 → warp_beacon-2.3.40}/warp_beacon/scheduler/scheduler.py

@@ -38,6 +38,8 @@ class IGScheduler(object):
 			if os.path.exists(self.state_file):
 				with open(self.state_file, 'r', encoding="utf-8") as f:
 					self.state = json.loads(f.read())
+				if "remaining" in self.state:
+					logging.info("Next scheduler activity in '%s' seconds", self.state["remaining"])
 		except Exception as e:
 			logging.error("Failed to load Scheduler state!")
 			logging.exception(e)
@@ -89,7 +91,7 @@ class IGScheduler(object):
 					self.save_state()
 
 				if self.state["remaining"] <= 0:
-					self.state["remaining"] = randrange(4100, 22300)
+					self.state["remaining"] = randrange(4292, 8623)
 					logging.info("Next scheduler activity in '%s' seconds", self.state["remaining"])
 
 				start_time = time.time()

{warp_beacon-2.3.38 → warp_beacon-2.3.40}/warp_beacon/scraper/exceptions.py

@@ -54,6 +54,8 @@ def extract_exception_message(e: Exception) -> str:
 		msg = e.message
 	elif hasattr(e, "reason"):
 		msg = e.reason
+	elif hasattr(e, "msg"):
+		msg = e.msg
 	elif hasattr(e, "args"):
 		msg = str(e.args)
 		

{warp_beacon-2.3.38 → warp_beacon-2.3.40}/warp_beacon/scraper/youtube/abstract.py

@@ -7,26 +7,29 @@ import socket
 import ssl
 #from abc import abstractmethod
 from typing import Callable, Union, Optional
+import logging
 import json
 import urllib
 import http.client
+import pytubefix.exceptions
 import requests
 from PIL import Image
 import numpy as np
+from urllib.parse import urlparse, parse_qs
 
-from warp_beacon.jobs.download_job import DownloadJob
-from warp_beacon.scraper.abstract import ScraperAbstract
-#from warp_beacon.mediainfo.abstract import MediaInfoAbstract
-from warp_beacon.scraper.exceptions import TimeOut, Unavailable, extract_exception_message
-
+import pytubefix
 from pytubefix import YouTube
 from pytubefix.innertube import _default_clients
 from pytubefix.streams import Stream
 from pytubefix.innertube import InnerTube, _client_id, _client_secret
-from pytubefix.exceptions import VideoUnavailable, VideoPrivate, MaxRetriesExceeded
+#from pytubefix.exceptions import VideoUnavailable, VideoPrivate, MaxRetriesExceeded
 from pytubefix import request
+import yt_dlp
 
-import logging
+from warp_beacon.jobs.download_job import DownloadJob
+from warp_beacon.scraper.abstract import ScraperAbstract
+#from warp_beacon.mediainfo.abstract import MediaInfoAbstract
+from warp_beacon.scraper.exceptions import TimeOut, Unavailable, extract_exception_message
 
 def patched_fetch_bearer_token(self) -> None:
 	"""Fetch an OAuth token."""
@@ -101,13 +104,24 @@ class YoutubeAbstract(ScraperAbstract):
 		os.rename(filename, new_filepath)
 
 		return new_filepath
+	
+	def get_video_id(self, url: str) -> Optional[str]:
+		parsed_url = urlparse(url)
+		query = parse_qs(parsed_url.query)
+		return query.get('v', [None])[0]
 
 	def remove_tmp_files(self) -> None:
 		for i in os.listdir(self.DOWNLOAD_DIR):
 			if "yt_download_" in i:
 				os.unlink("%s/%s" % (self.DOWNLOAD_DIR, i))
 
-	def calculate_size_with_padding(self, image: Image, aspect_ratio_width: int, aspect_ratio_height: int, target_size: tuple=(320, 320), background_color: tuple=(0, 0, 0)) -> Image:
+	def calculate_size_with_padding(self,
+		image: Image,
+		aspect_ratio_width: int,
+		aspect_ratio_height: int,
+		target_size: tuple=(320, 320),
+		background_color: tuple=(0, 0, 0)
+	) -> Image:
 		aspect_ratio = aspect_ratio_width / aspect_ratio_height
 		target_width, target_height = target_size
 
@@ -194,7 +208,7 @@ class YoutubeAbstract(ScraperAbstract):
 				kwargs["timeout"] = timeout
 				ret_val = func(*args, **kwargs)
 				break
-			except MaxRetriesExceeded:
+			except pytubefix.exceptions.MaxRetriesExceeded:
 				# do noting, not interested
 				pass
 			#except http.client.IncompleteRead as e:
@@ -216,7 +230,21 @@ class YoutubeAbstract(ScraperAbstract):
 				retries += 1
 				timeout += timeout_increment
 				time.sleep(pause_secs)
-			except (VideoUnavailable, VideoPrivate) as e:
+			except (pytubefix.exceptions.VideoUnavailable, pytubefix.exceptions.VideoPrivate) as e:
+				raise Unavailable(extract_exception_message(e))
+			except yt_dlp.utils.DownloadError as e:
+				raise Unavailable(extract_exception_message(e))
+			except yt_dlp.utils.GeoRestrictedError:
+				raise Unavailable(extract_exception_message(e))
+			except yt_dlp.utils.PostProcessingError as e:
+				raise Unavailable(extract_exception_message(e))
+			except yt_dlp.utils.ExtractorError as e:
+				raise Unavailable(extract_exception_message(e))
+			except yt_dlp.utils.MaxDownloadsReached as e:
+				raise Unavailable(extract_exception_message(e))
+			except yt_dlp.utils.UnavailableVideoError as e:
+				raise Unavailable(extract_exception_message(e))
+			except yt_dlp.utils.ThrottledDownload as e:
 				raise Unavailable(extract_exception_message(e))
 
 		return ret_val
@@ -241,8 +269,8 @@ class YoutubeAbstract(ScraperAbstract):
 			yt_opts["token_file"] = self.YT_SESSION_FILE % self.account_index
 		if self.proxy:
 			proxy_dsn = self.proxy.get("dsn", "")
+			logging.info("Using proxy DSN '%s'", proxy_dsn)
 			if proxy_dsn:
-				logging.info("Using proxy DSN '%s'", proxy_dsn)
 				if "https://" in proxy_dsn:
 					yt_opts["proxies"] = {"https": proxy_dsn}
 				elif "http://" in proxy_dsn:
@@ -251,14 +279,44 @@ class YoutubeAbstract(ScraperAbstract):
 					logging.warning("Proxy DSN malformed!")
 		return YouTube(**yt_opts)
 	
+	def build_yt_dlp(self) -> yt_dlp.YoutubeDL:
+		auth_data = {}
+		with open(self.YT_SESSION_FILE % self.account_index, 'r', encoding="utf-8") as f:
+			auth_data = json.loads(f.read())
+		time_name = str(time.time()).replace('.', '_')
+		ydl_opts = {
+			'outtmpl': f'{self.DOWNLOAD_DIR}/{time_name}.%(ext)s',
+			'format': 'bestvideo+bestaudio/best',
+			'merge_output_format': 'mp4',
+			'noplaylist': True,
+			'tv_auth': auth_data
+		}
+
+		if self.proxy:
+			proxy_dsn = self.proxy.get("dsn", "")
+			logging.info("Using proxy DSN '%s'", proxy_dsn)
+			if proxy_dsn:
+				ydl_opts["proxy"] = proxy_dsn
+
+		return yt_dlp.YoutubeDL(ydl_opts)
+	
 	def _download(self, _: str) -> list:
 		raise NotImplementedError("You should to implement _download method")
+	
+	def _download_yt_dlp(self, _: str) -> list:
+		raise NotImplementedError("You should to implement _download_yt_dlp method")
 
 	def download(self, job: DownloadJob) -> list:
+		ret = []
 		try:
-			return self.download_hndlr(self._download, job.url, session=False)
-		except Unavailable:
-			logging.warning("Failed sessionless download attempt. Trying with session.")
-			time.sleep(2)
+			ret = self.download_hndlr(self._download, job.url, session=True)
+		except (Unavailable, TimeOut):
+			logging.warning("Download failed, trying to download with yt_dlp")
 		
-		return self.download_hndlr(self._download, job.url, session=True)
+		try:
+			ret = self.download_hndlr(self._download_yt_dlp, job.url)
+		except NotImplementedError:
+			logging.info("yt_dlp is not supported for this submodule yet")
+			raise Unavailable("Video unvailable")
+
+		return ret

{warp_beacon-2.3.38 → warp_beacon-2.3.40}/warp_beacon/scraper/youtube/youtube.py

@@ -1,10 +1,10 @@
-from warp_beacon.jobs.types import JobType
-from warp_beacon.scraper.youtube.abstract import YoutubeAbstract
-from warp_beacon.scraper.exceptions import YoutubeLiveError, NotFound, YotubeAgeRestrictedError
+import logging
 
 from pytubefix.exceptions import AgeRestrictedError
 
-import logging
+from warp_beacon.jobs.types import JobType
+from warp_beacon.scraper.youtube.abstract import YoutubeAbstract
+from warp_beacon.scraper.exceptions import YoutubeLiveError, NotFound, YotubeAgeRestrictedError
 
 class YoutubeScraper(YoutubeAbstract):
 	YT_MAX_RETRIES_DEFAULT = 8
@@ -32,14 +32,14 @@ class YoutubeScraper(YoutubeAbstract):
 		res = []
 		try:
 			thumbnail = None
+			video_id = self.get_video_id(url)
+			if video_id:
+				thumbnail = self.download_hndlr(self.download_thumbnail, video_id)
 			yt = self.build_yt(url, session=session)
 
 			if self.is_live(yt.initial_data):
 				raise YoutubeLiveError("Youtube Live is not supported")
 
-			if yt:
-				thumbnail = self.download_hndlr(self.download_thumbnail, yt.video_id)
-
 			stream = yt.streams.get_highest_resolution()
 
 			if not stream:
@@ -65,4 +65,24 @@ class YoutubeScraper(YoutubeAbstract):
 		except AgeRestrictedError as e:
 			raise YotubeAgeRestrictedError("Youtube Age Restricted error")
 
+		return res
+	
+	def _download_yt_dlp(self, url: str) -> list:
+		res = []
+		thumbnail = None
+		video_id = self.get_video_id(url)
+		if video_id:
+			thumbnail = self.download_hndlr(self.download_thumbnail, video_id)
+		with self.build_yt_dlp() as ydl:
+			info = ydl.extract_info(url, download=True)
+			local_file = ydl.prepare_filename(info)
+			logging.debug("Temp filename: '%s'", local_file)
+			res.append({
+				"local_media_path": local_file,
+				"performer": info.get("uploader", "Unknown"),
+				"thumb": thumbnail,
+				"canonical_name": info.get("title", ''),
+				"media_type": JobType.VIDEO
+			})
+
 		return res

{warp_beacon-2.3.38 → warp_beacon-2.3.40/warp_beacon.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: warp_beacon
-Version: 2.3.38
+Version: 2.3.40
 Summary: Telegram bot for expanding external media links
 Home-page: https://github.com/sb0y/warp_beacon
 Author: Andrey Bagrintsev
@@ -237,10 +237,13 @@ Requires-Dist: pillow
 Requires-Dist: pymongo
 Requires-Dist: instagrapi==2.0.0
 Requires-Dist: bs4
+Requires-Dist: yt_dlp
 Dynamic: author
 Dynamic: home-page
+Dynamic: license-file
 
 # warp_beacon [![Upload Python Package](https://github.com/sb0y/warp_beacon/actions/workflows/python-publish.yml/badge.svg)](https://github.com/sb0y/warp_beacon/actions/workflows/python-publish.yml) [![Docker Image CI](https://github.com/sb0y/warp_beacon/actions/workflows/docker-image.yml/badge.svg)](https://github.com/sb0y/warp_beacon/actions/workflows/docker-image.yml) [![Build DEB package](https://github.com/sb0y/warp_beacon/actions/workflows/build-deb.yml/badge.svg)](https://github.com/sb0y/warp_beacon/actions/workflows/build-deb.yml)
+> because content should travel freely
 
 Telegram bot for external social networks media links expanding.
 Works with a links in personal messages and also with group chats.
@@ -253,6 +256,24 @@ Just send a media link to the chat with bot and get a video or audio reply.
 |<img width="700" alt="Instagram Photo post usage example" src="https://github.com/user-attachments/assets/29324b94-7314-4a38-8790-3483011d355d"> Instagram Photo post usage example|<img width="700" alt="Instagram Photo Carousel usage" src="https://github.com/user-attachments/assets/2598e329-e16e-455e-91e9-a027e8994283"> Instagram Photo Carousel usage example|<img width="757" alt="Instagram Photo bulk Strories download usage example" src="https://github.com/user-attachments/assets/2c8c91ac-6ade-4d1d-a677-2b36bb40ff39"> Instagram Photo bulk Strories download usage example|
 |<img width="700" alt="Instagram specific Story download usage example" src="https://github.com/user-attachments/assets/03dc70c5-6933-4122-9c7c-5f7d734d117b"> Instagram specific Story download usage example|<img width="700" alt="Group chat usage example" src="https://github.com/user-attachments/assets/649fcb1e-785b-4efd-9153-69644c6d898b"> Group chat usage example|
 
+### **Warp Beacon manifesto**
+
+> Once, the Internet was built as a borderless network — a space where knowledge, culture, and ideas could flow freely across the globe.  
+> But over time, freedom gave way to artificial walls, anti-bot shields, and region locks.
+>
+> **warp_beacon** is our answer to that shift.
+>
+> This is a tool for those who refuse to accept "access denied by geolocation" or "content unavailable in your region."  
+> It is a bridge over ML filters, CAPTCHAs, and man-made barriers.
+>
+> We don’t break the rules — we restore the original spirit of the Internet:  
+> 📡 **free exchange of information**,  
+> 🌍 **unrestricted access to global content**,  
+> 🤖 **tools that serve the user**, not the platform.
+>
+> **warp_beacon** — the freedom to deliver content where it’s needed most.
+
+## Configuration example ##
 
 In order to setup your own instance, you will need:
 
@@ -261,8 +282,6 @@ In order to setup your own instance, you will need:
 
 All bot configuration stored in [warp_beacon.conf](https://github.com/sb0y/warp_beacon/blob/main/etc/warp_beacon.conf) file.
 
-## Configuration example ##
-
 ```env
 TG_TOKEN="you telegram token received from @BotFather"
 # these 3 settings should be obtained at https://my.telegram.org/apps

{warp_beacon-2.3.38 → warp_beacon-2.3.40}/warp_beacon.egg-info/requires.txt

@@ -11,3 +11,4 @@ pillow
 pymongo
 instagrapi==2.0.0
 bs4
+yt_dlp

warp_beacon-2.3.38/warp_beacon/__version__.py

@@ -1,2 +0,0 @@
-__version__ = "2.3.38"
-