tgdata 0.0.6__tar.gz → 0.0.8__tar.gz

{tgdata-0.0.6 → tgdata-0.0.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tgdata
-Version: 0.0.6
+Version: 0.0.8
 Summary: A production-grade Python library for extracting and processing Telegram group and channel messages
 Author: enes kuzucu
 Classifier: Development Status :: 3 - Alpha
@@ -14,7 +14,7 @@ Classifier: Programming Language :: Python :: 3.10
 Classifier: Operating System :: OS Independent
 Requires-Python: >=3.7
 Description-Content-Type: text/markdown
-Requires-Dist: Telethon<2.0,>=1.24
+Requires-Dist: Telethon<2.0,>=1.33
 Requires-Dist: pandas>=1.0
 Dynamic: author
 Dynamic: classifier
@@ -28,6 +28,14 @@ Dynamic: summary
 
 A production-grade Python library for extracting and processing Telegram group and channel messages. Designed for ETL pipelines, data analysis, and archival purposes.
 
+> **New in 0.0.8 — group discovery.** tgdata can now *find* groups and channels, not only read
+> them: Telegram's name search, its "similar channels" recommendations, and links mined from
+> posts, on the same persistent session, paced, flood-safe and budgeted. Four calls —
+> `search_groups`, `similar_groups`, `linked_groups`, `discover_groups` — return candidate rooms
+> in the `list_groups` row shape, already usable by `get_messages`. Quick start below under
+> [Group discovery](#group-discovery--finding-groups-you-are-not-in); the full guide to using it
+> properly is [`devdocs/guides/group_discovery.md`](devdocs/guides/group_discovery.md).
+
 ## Features
 
 - 🚀 **Production-Ready**: Built for reliability and scale in ETL pipelines
@@ -42,6 +50,7 @@ A production-grade Python library for extracting and processing Telegram group a
 - ⏱️ **Polling Support**: Poll for new messages at configurable intervals
 - 🎯 **Batch Processing**: Handle large groups with configurable batch sizes and delays
 - 🖼️ **Media Detection & Download**: Flag photos/videos per message (no download needed) and pull the files on demand or during the fetch
+- 🔎 **Group Discovery**: Find rooms you are not in — Telegram's name search, its "similar channels", and links mined from posts — paced, flood-safe, budgeted
 
 ## Installation
 
@@ -110,6 +119,78 @@ Outputs:
 Python Devs 10012312313
 ```
 
+### Group discovery — finding groups you are not in
+
+`list_groups` shows what the account already follows. To find *new* rooms there are three
+routes, all returning the same columns as `list_groups` plus `FoundVia` and `FoundBy`.
+This is the quick start; the rules for running discovery on a real account without earning
+a block — pacing, budgets, resumable runs, reading the output — are in
+[`devdocs/guides/group_discovery.md`](devdocs/guides/group_discovery.md).
+
+```python
+from tgdata import TgData, build_search_queries
+
+tg = TgData("config.ini")
+
+# 1. Telegram's global search — matches room NAMES and @usernames only, never post text
+df = await tg.search_groups("Анталия аренда")
+
+# 2. Telegram's "similar channels" — rooms whose subscribers overlap the seeds', no keyword needed
+df = await tg.similar_groups(["@antalyadaa", "@sprout_antalya"], rounds=2)
+
+# 3. t.me links found in a room's recent posts — usernames only (free), or resolved
+df = await tg.linked_groups(group_id, posts=100, resolve=False)
+
+# All three on one session, deduplicated by id, in cost order
+df = await tg.discover_groups(
+    seeds=["@antalyadaa"],
+    queries=build_search_queries(["Анталия", "Antalya"], topics=["чат", "аренда", "новости"],
+                                 prefixes=["Турция"]),
+    mine_links=True,
+    heartbeat=lambda phase: print(phase),      # liveness ticks: search / similar / links / pause / flood-wait Ns
+    found_callback=lambda row: rows.append(row),  # each room as it is found — persist as you go
+)
+```
+
+**The name-only rule.** Telegram's search matches names and @usernames, never what a room
+posts. A perfect room under an unrelated name is invisible to every query — that is what
+`similar_groups` and `linked_groups` are for. Rooms name themselves "place + topic", so
+`build_search_queries` combines your place spellings with topic words, in both word orders.
+
+| Column | Meaning |
+|---|---|
+| `GroupID` … `ParticipantsCount` | the `list_groups` columns; `ParticipantsCount` is present on search and similar-channel results, absent (`NA`) on rooms resolved from links — Telegram's lookup reply does not carry it |
+| `FoundVia` | `search` · `similar` · `link` — the cheapest route that found the room |
+| `FoundBy` | the query, the seed's `@name`, or the source room |
+
+Unresolved link rows carry only `Username`/`Identifier`; `GroupID` and the flags are `NA`
+(the columns are nullable, so dtypes never drift). Every row with an id is already usable
+by `get_messages(row['GroupID'])` — Telegram results land in the session cache with their
+access hash, no dialog sync needed.
+
+**Pacing and waits.** Every request is followed by `pace` seconds (default 2, the proven
+pace for a personal account). A rate-limit wait up to `max_flood_wait` is obeyed exactly,
+with heartbeat ticks so a watchdog sees life; a longer one raises `DiscoveryInterrupted`,
+which carries `.found` (everything collected so far) and `.retry_after`. A dropped connection
+is waited out up to `max_offline` and the same request retried. Nothing found is ever lost.
+
+**The budget.** Resolving a username to a room is the request Telegram punishes hardest —
+a production account lost 8 and 21 hours of lookups to it. So every call has `max_resolve`
+(default 100), `discover_groups` mines links from the first `link_sources` rooms only
+(default 50), `resolve_links` is off by default (mined names come back as text, which costs
+nothing), and a wait on a lookup is skipped or stopped, never waited out. A truncating cap
+logs a warning telling you which parameter lifts it.
+
+**Similar channels** work on channels and megagroups, not basic groups; two rounds by
+default (a third drifts off topic); the seeds themselves are never returned. Yield depends
+on the account — a Premium account got 64 recommendations from one seed, a normal account
+reportedly about 10.
+
+**Links** are read from the visible text, from hyperlinked text (where channels put nearly
+all of them) and from URL buttons. Accepted: `t.me/name`, `t.me/name/123`, `t.me/s/name`,
+`t.me/boost/name`, `telegram.me/name`. Skipped: invite links (`t.me/+…`), `t.me/c/…`,
+reserved paths, and bots (`…bot`). Discovery reads; it never joins.
+
 
 ### Get all messages of a chat 
 

{tgdata-0.0.6 → tgdata-0.0.8}/README.md

@@ -2,6 +2,14 @@
 
 A production-grade Python library for extracting and processing Telegram group and channel messages. Designed for ETL pipelines, data analysis, and archival purposes.
 
+> **New in 0.0.8 — group discovery.** tgdata can now *find* groups and channels, not only read
+> them: Telegram's name search, its "similar channels" recommendations, and links mined from
+> posts, on the same persistent session, paced, flood-safe and budgeted. Four calls —
+> `search_groups`, `similar_groups`, `linked_groups`, `discover_groups` — return candidate rooms
+> in the `list_groups` row shape, already usable by `get_messages`. Quick start below under
+> [Group discovery](#group-discovery--finding-groups-you-are-not-in); the full guide to using it
+> properly is [`devdocs/guides/group_discovery.md`](devdocs/guides/group_discovery.md).
+
 ## Features
 
 - 🚀 **Production-Ready**: Built for reliability and scale in ETL pipelines
@@ -16,6 +24,7 @@ A production-grade Python library for extracting and processing Telegram group a
 - ⏱️ **Polling Support**: Poll for new messages at configurable intervals
 - 🎯 **Batch Processing**: Handle large groups with configurable batch sizes and delays
 - 🖼️ **Media Detection & Download**: Flag photos/videos per message (no download needed) and pull the files on demand or during the fetch
+- 🔎 **Group Discovery**: Find rooms you are not in — Telegram's name search, its "similar channels", and links mined from posts — paced, flood-safe, budgeted
 
 ## Installation
 
@@ -84,6 +93,78 @@ Outputs:
 Python Devs 10012312313
 ```
 
+### Group discovery — finding groups you are not in
+
+`list_groups` shows what the account already follows. To find *new* rooms there are three
+routes, all returning the same columns as `list_groups` plus `FoundVia` and `FoundBy`.
+This is the quick start; the rules for running discovery on a real account without earning
+a block — pacing, budgets, resumable runs, reading the output — are in
+[`devdocs/guides/group_discovery.md`](devdocs/guides/group_discovery.md).
+
+```python
+from tgdata import TgData, build_search_queries
+
+tg = TgData("config.ini")
+
+# 1. Telegram's global search — matches room NAMES and @usernames only, never post text
+df = await tg.search_groups("Анталия аренда")
+
+# 2. Telegram's "similar channels" — rooms whose subscribers overlap the seeds', no keyword needed
+df = await tg.similar_groups(["@antalyadaa", "@sprout_antalya"], rounds=2)
+
+# 3. t.me links found in a room's recent posts — usernames only (free), or resolved
+df = await tg.linked_groups(group_id, posts=100, resolve=False)
+
+# All three on one session, deduplicated by id, in cost order
+df = await tg.discover_groups(
+    seeds=["@antalyadaa"],
+    queries=build_search_queries(["Анталия", "Antalya"], topics=["чат", "аренда", "новости"],
+                                 prefixes=["Турция"]),
+    mine_links=True,
+    heartbeat=lambda phase: print(phase),      # liveness ticks: search / similar / links / pause / flood-wait Ns
+    found_callback=lambda row: rows.append(row),  # each room as it is found — persist as you go
+)
+```
+
+**The name-only rule.** Telegram's search matches names and @usernames, never what a room
+posts. A perfect room under an unrelated name is invisible to every query — that is what
+`similar_groups` and `linked_groups` are for. Rooms name themselves "place + topic", so
+`build_search_queries` combines your place spellings with topic words, in both word orders.
+
+| Column | Meaning |
+|---|---|
+| `GroupID` … `ParticipantsCount` | the `list_groups` columns; `ParticipantsCount` is present on search and similar-channel results, absent (`NA`) on rooms resolved from links — Telegram's lookup reply does not carry it |
+| `FoundVia` | `search` · `similar` · `link` — the cheapest route that found the room |
+| `FoundBy` | the query, the seed's `@name`, or the source room |
+
+Unresolved link rows carry only `Username`/`Identifier`; `GroupID` and the flags are `NA`
+(the columns are nullable, so dtypes never drift). Every row with an id is already usable
+by `get_messages(row['GroupID'])` — Telegram results land in the session cache with their
+access hash, no dialog sync needed.
+
+**Pacing and waits.** Every request is followed by `pace` seconds (default 2, the proven
+pace for a personal account). A rate-limit wait up to `max_flood_wait` is obeyed exactly,
+with heartbeat ticks so a watchdog sees life; a longer one raises `DiscoveryInterrupted`,
+which carries `.found` (everything collected so far) and `.retry_after`. A dropped connection
+is waited out up to `max_offline` and the same request retried. Nothing found is ever lost.
+
+**The budget.** Resolving a username to a room is the request Telegram punishes hardest —
+a production account lost 8 and 21 hours of lookups to it. So every call has `max_resolve`
+(default 100), `discover_groups` mines links from the first `link_sources` rooms only
+(default 50), `resolve_links` is off by default (mined names come back as text, which costs
+nothing), and a wait on a lookup is skipped or stopped, never waited out. A truncating cap
+logs a warning telling you which parameter lifts it.
+
+**Similar channels** work on channels and megagroups, not basic groups; two rounds by
+default (a third drifts off topic); the seeds themselves are never returned. Yield depends
+on the account — a Premium account got 64 recommendations from one seed, a normal account
+reportedly about 10.
+
+**Links** are read from the visible text, from hyperlinked text (where channels put nearly
+all of them) and from URL buttons. Accepted: `t.me/name`, `t.me/name/123`, `t.me/s/name`,
+`t.me/boost/name`, `telegram.me/name`. Skipped: invite links (`t.me/+…`), `t.me/c/…`,
+reserved paths, and bots (`…bot`). Discovery reads; it never joins.
+
 
 ### Get all messages of a chat 
 

{tgdata-0.0.6 → tgdata-0.0.8}/setup.py

@@ -33,7 +33,9 @@ setup(
     install_requires=[
         # min_id is exclusive in Telethon 1.x (the off-by-one fix relies on it);
         # 2.0 is a breaking API rewrite, so cap below it. Verified on 1.40.
-        'Telethon>=1.24,<2.0',
+        # 1.33 is the first release carrying channels.GetChannelRecommendationsRequest
+        # (group discovery, scoped/10) — verified against the 1.33.1 sdist.
+        'Telethon>=1.33,<2.0',
         # Required, not optional: everything returns DataFrames. Nullable Int64
         # (used for GroupedId album ids > 2^53) needs pandas >= 1.0.
         'pandas>=1.0',

{tgdata-0.0.6 → tgdata-0.0.8}/tgdata/__init__.py

@@ -9,6 +9,8 @@ from .tgdata import TgData
 
 # Errors
 from .message_engine import GroupAccessError
+from .connection_engine import AuthRequiredError
+from .discovery_engine import DiscoveryInterrupted
 
 # Models
 from .models import (
@@ -27,13 +29,14 @@ from .utils import (
     filter_messages_by_content,
     get_message_statistics,
     save_profile_photos,
-    create_metrics_report
+    create_metrics_report,
+    build_search_queries
 )
 
 # Progress tracking
 from .progress import ProgressTracker
 
-__version__ = "0.0.6"
+__version__ = "0.0.8"
 
 __all__ = [
     # Main class
@@ -41,13 +44,15 @@ __all__ = [
 
     # Errors
     "GroupAccessError",
+    "AuthRequiredError",
+    "DiscoveryInterrupted",
 
     # Models
     "MessageData",
     "GroupInfo",
     "ConnectionConfig",
     "RateLimitInfo",
-    
+
     # Utilities
     "format_message_for_display",
     "export_to_json",
@@ -57,7 +62,8 @@ __all__ = [
     "get_message_statistics",
     "save_profile_photos",
     "create_metrics_report",
-    
+    "build_search_queries",
+
     # Progress
     "ProgressTracker"
-]
+]

{tgdata-0.0.6 → tgdata-0.0.8}/tgdata/connection_engine.py

@@ -67,6 +67,14 @@ class ConnectionPool:
         self.connections.clear()
 
 
+class AuthRequiredError(RuntimeError):
+    """The session is not authorized and this code path is non-interactive.
+
+    Raised by ephemeral_client() instead of Telethon's start() prompt —
+    a headless process must FAIL LOUDLY here, never sit on input().
+    Fix: run an interactive login for this account once, by hand."""
+
+
 class ConnectionEngine:
     """
     Manages Telegram connections with advanced features.
@@ -189,6 +197,34 @@ class ConnectionEngine:
         yield client
         # Intentionally NO disconnect here — the connection is persistent.
 
+    @asynccontextmanager
+    async def ephemeral_client(self):
+        """USE-AND-CLOSE (2026-08-05): a fresh client on the same resolved
+        session, connected WITHOUT start() (never interactive), disconnected
+        in finally — for short, human-triggered lookups (identity fetch,
+        access probe) that must not park a live transport next to a standing
+        loop sharing the session file. The persistent session()/pool path
+        above is untouched — it remains correct for the loop's own fetches,
+        where ONE process owns the session for its lifetime.
+
+        Raises AuthRequiredError when the session is not authorized: the
+        one situation start() would have prompted for a login code and hung
+        a headless process on stdin forever."""
+        config = self._load_config()
+        client = TelegramClient(config.session_file, config.api_id, config.api_hash)
+        try:
+            await client.connect()
+            if not await client.is_user_authorized():
+                raise AuthRequiredError(
+                    f"session {config.session_file!r} is not authorized — "
+                    f"run an interactive login for this account once, then retry")
+            yield client
+        finally:
+            try:
+                await client.disconnect()
+            except Exception:  # noqa: BLE001 — teardown must never mask the work
+                pass
+
     async def _init_primary_client(self):
         """Create and authenticate the primary client once (may be interactive)."""
         config = self._load_config()