tgdata 0.1__tar.gz → 0.1.1__tar.gz

{tgdata-0.1 → tgdata-0.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tgdata
-Version: 0.1
+Version: 0.1.1
 Summary: A production-grade Python library for extracting and processing Telegram group and channel messages
 Author: enes kuzucu
 Classifier: Development Status :: 3 - Alpha
@@ -59,6 +59,8 @@ A production-grade Python library for extracting and processing Telegram group a
 - 🩺 **Account Health Events**: Every wait, logout, ban, restriction or lost group is reported as a plain-data event — through one callback, a log line and a per-account summary — and "ok" when it ends
 - **Per-account read budgets**: An optional shared SQLite ledger limits message pulls over a rolling 24 hours, with persistent warm-up steps and recoverable partial results
 - **Versioned message batches**: Read bounded raw snapshots with canonical JSON, replayable batch IDs, safe continuation cursors and optional content-addressed media
+- **Album assembly across batches**: Combine observed album members into stable post revisions, preserving every caption and media reference with explicit caller-owned persistence
+- **Optional photo trails**: Associate a user's following photo-only posts in a provisional view, retaining all originals and explicit heuristic provenance
 - **Durable daily continuation**: Keep per-group progress and pending batches in a pluggable store, replay after restart, and advance only after explicit acknowledgment
 - **Reading worker**: Run a serial process that accepts explicit-account JSON jobs for batches, group lookup, access checks and joining, with correlated health replies
 
@@ -807,6 +809,70 @@ refuses cleanup may leave a private temporary file. The exact schema, limits,
 interruption rules
 and delivery protocol are in [Message batch v1](docs/message_batch_v1.md).
 
+### Albums across message batches
+
+Use the opt-in assembler to keep one logical post as more album members arrive:
+
+```python
+from tgdata import album_post_ids, assemble_albums
+
+# In-memory illustration for a new collection; retain these posts across calls.
+known_posts = {}
+for batch in batches:
+    previous = {key: known_posts.get(key) for key in album_post_ids(batch)}
+    for post in assemble_albums(batch, previous=previous):
+        known_posts[post.post_id] = post
+        print(post.post_id, post.revision, post.member_ids)
+```
+
+Album IDs include the chat; each post retains all original member records and
+captions. Identical replay keeps the same revision. Album membership remains
+`unconfirmed`: a batch limit, fixed window or history end cannot prove that every
+member was seen. Assembly performs no additional reads and changes no raw batch
+hash, cursor or acknowledgment.
+
+For restartable collection, commit the original records and input recognition in
+your destination **before** acknowledging the raw delivery. Confirm an already
+accepted input from those records without rebuilding a growing album. The SQLite
+example's `accept` returns no post payload; `load(post_id)` reads the current view
+separately. Do not replace lost destination state with an empty dictionary. The
+[album contract](docs/album_assembly.md) explains updates, limits and storage;
+`python examples/album_assembly.py --demo` demonstrates SQLite restart and lost-ack
+replay without Telegram.
+
+### Batch-reader liveness
+
+Pass a quick synchronous `heartbeat(phase)` to `get_message_batch`, `sync_group`
+or `prepare_backfill` (also through AccountPool) to observe message, media-chunk
+and request progress. This lets a caller distinguish a slow download from a
+silent read. Defaults, batch hashes and acknowledgments stay unchanged. See
+[heartbeat and watchdog guidance](docs/batch_heartbeat.md) for callback isolation,
+replay behavior and the SDK-wait limit.
+
+### Optional photo trails
+
+For a text or album followed by additional photo-only posts, explicitly enable
+the conservative heuristic over your current original observations:
+
+```python
+from tgdata import PhotoTrailPolicy, fold_photo_trails
+
+view = fold_photo_trails(current_posts, policy=PhotoTrailPolicy(enabled=True))
+print(view.groups)          # tuples of original post IDs, anchor first
+print(view.folded_posts, view.folded_photos)
+```
+
+The default policy is disabled. Every original `TelegramPost` and caption remains
+in `view.posts`. A later observation can withdraw an inferred association; replace
+the earlier view for your chosen scope instead of appending final posts. Counts
+describe the current view, not additional work on each retry.
+
+The helper makes no source requests or acknowledgments. The optional SQLite
+example reuses saved original records and refuses a range that cuts a known
+album, preserving intervening context. See [photo-trail views](docs/photo_trails.md)
+for policy, limits and restart/acknowledgment guidance, or run
+`python examples/photo_trails.py --demo` without Telegram.
+
 ### Daily continuation across restarts
 
 Use a progress store when tgdata should remember unfinished delivery and the last

{tgdata-0.1 → tgdata-0.1.1}/README.md

@@ -30,6 +30,8 @@ A production-grade Python library for extracting and processing Telegram group a
 - 🩺 **Account Health Events**: Every wait, logout, ban, restriction or lost group is reported as a plain-data event — through one callback, a log line and a per-account summary — and "ok" when it ends
 - **Per-account read budgets**: An optional shared SQLite ledger limits message pulls over a rolling 24 hours, with persistent warm-up steps and recoverable partial results
 - **Versioned message batches**: Read bounded raw snapshots with canonical JSON, replayable batch IDs, safe continuation cursors and optional content-addressed media
+- **Album assembly across batches**: Combine observed album members into stable post revisions, preserving every caption and media reference with explicit caller-owned persistence
+- **Optional photo trails**: Associate a user's following photo-only posts in a provisional view, retaining all originals and explicit heuristic provenance
 - **Durable daily continuation**: Keep per-group progress and pending batches in a pluggable store, replay after restart, and advance only after explicit acknowledgment
 - **Reading worker**: Run a serial process that accepts explicit-account JSON jobs for batches, group lookup, access checks and joining, with correlated health replies
 
@@ -778,6 +780,70 @@ refuses cleanup may leave a private temporary file. The exact schema, limits,
 interruption rules
 and delivery protocol are in [Message batch v1](docs/message_batch_v1.md).
 
+### Albums across message batches
+
+Use the opt-in assembler to keep one logical post as more album members arrive:
+
+```python
+from tgdata import album_post_ids, assemble_albums
+
+# In-memory illustration for a new collection; retain these posts across calls.
+known_posts = {}
+for batch in batches:
+    previous = {key: known_posts.get(key) for key in album_post_ids(batch)}
+    for post in assemble_albums(batch, previous=previous):
+        known_posts[post.post_id] = post
+        print(post.post_id, post.revision, post.member_ids)
+```
+
+Album IDs include the chat; each post retains all original member records and
+captions. Identical replay keeps the same revision. Album membership remains
+`unconfirmed`: a batch limit, fixed window or history end cannot prove that every
+member was seen. Assembly performs no additional reads and changes no raw batch
+hash, cursor or acknowledgment.
+
+For restartable collection, commit the original records and input recognition in
+your destination **before** acknowledging the raw delivery. Confirm an already
+accepted input from those records without rebuilding a growing album. The SQLite
+example's `accept` returns no post payload; `load(post_id)` reads the current view
+separately. Do not replace lost destination state with an empty dictionary. The
+[album contract](docs/album_assembly.md) explains updates, limits and storage;
+`python examples/album_assembly.py --demo` demonstrates SQLite restart and lost-ack
+replay without Telegram.
+
+### Batch-reader liveness
+
+Pass a quick synchronous `heartbeat(phase)` to `get_message_batch`, `sync_group`
+or `prepare_backfill` (also through AccountPool) to observe message, media-chunk
+and request progress. This lets a caller distinguish a slow download from a
+silent read. Defaults, batch hashes and acknowledgments stay unchanged. See
+[heartbeat and watchdog guidance](docs/batch_heartbeat.md) for callback isolation,
+replay behavior and the SDK-wait limit.
+
+### Optional photo trails
+
+For a text or album followed by additional photo-only posts, explicitly enable
+the conservative heuristic over your current original observations:
+
+```python
+from tgdata import PhotoTrailPolicy, fold_photo_trails
+
+view = fold_photo_trails(current_posts, policy=PhotoTrailPolicy(enabled=True))
+print(view.groups)          # tuples of original post IDs, anchor first
+print(view.folded_posts, view.folded_photos)
+```
+
+The default policy is disabled. Every original `TelegramPost` and caption remains
+in `view.posts`. A later observation can withdraw an inferred association; replace
+the earlier view for your chosen scope instead of appending final posts. Counts
+describe the current view, not additional work on each retry.
+
+The helper makes no source requests or acknowledgments. The optional SQLite
+example reuses saved original records and refuses a range that cuts a known
+album, preserving intervening context. See [photo-trail views](docs/photo_trails.md)
+for policy, limits and restart/acknowledgment guidance, or run
+`python examples/photo_trails.py --demo` without Telegram.
+
 ### Daily continuation across restarts
 
 Use a progress store when tgdata should remember unfinished delivery and the last

{tgdata-0.1 → tgdata-0.1.1}/tgdata/__init__.py

@@ -72,7 +72,7 @@ from .utils import (
 # Progress tracking
 from .progress import ProgressTracker
 
-__version__ = "0.1"
+__version__ = "0.1.1"
 
 __all__ = [
     # Explicit account onboarding
@@ -182,3 +182,9 @@ __all__ = [
     # Progress
     "ProgressTracker"
 ]
+
+from .album import TelegramPost, AlbumAssemblyError, album_post_ids, assemble_albums
+__all__ += ["TelegramPost", "AlbumAssemblyError", "album_post_ids", "assemble_albums"]
+
+from .photo_trails import PhotoTrailPolicy, PhotoTrailView, PhotoTrailError, fold_photo_trails
+__all__ += ["PhotoTrailPolicy", "PhotoTrailView", "PhotoTrailError", "fold_photo_trails"]

{tgdata-0.1 → tgdata-0.1.1}/tgdata/account_pool.py

@@ -16,6 +16,8 @@ from uuid import uuid4
 import telethon
 from telethon import errors as rpc_errors
 
+from .batch_heartbeat import heartbeat_kwargs
+
 from . import health
 from .batch_engine import _integer as _batch_integer
 from .batch_files import BatchStorageError, prepare_directory
@@ -468,15 +470,16 @@ class AccountPool(ProgressFacade):
             raise local from None
 
     async def _read(self, chat_id, *, after_id=0, limit=200, download_media_to=None,
-                    start_date=None, end_date=None, _capture=None):
+                    start_date=None, end_date=None, heartbeat=None, _capture=None):
         """Read once through the first successful eligible account, with provenance."""
+        progress = heartbeat_kwargs(heartbeat)
         chat_id = self._group(chat_id)
         after_id = _batch_integer(after_id, 'after_id', 0, MAX_MESSAGE_ID)
         limit = _batch_integer(limit, 'limit', 1, MAX_BATCH_MESSAGES)
         _HistoryWindow.from_dates(start_date, end_date)
         root = prepare_directory(download_media_to) if download_media_to is not None else None
         options = dict(after_id=after_id, limit=limit, download_media_to=root,
-                       start_date=start_date, end_date=end_date)
+                       start_date=start_date, end_date=end_date, **progress)
         with self._operation():
             raw, document = await self._load()
             attempts, tried = [], set()
@@ -520,33 +523,33 @@ class AccountPool(ProgressFacade):
             raise PoolUnavailable(statuses.accounts, attempts, retry_after) from None
 
     async def read(self, chat_id, *, capture, after_id=0, limit=200,
-                   download_media_to=None, start_date=None, end_date=None):
+                   download_media_to=None, start_date=None, end_date=None, heartbeat=None):
         """Read with a caller-retained, single-use data destination."""
         with _capture_call(capture) as writer:
             result = await self._read(chat_id, after_id=after_id, limit=limit,
                 download_media_to=download_media_to, start_date=start_date,
-                end_date=end_date, _capture=writer)
+                end_date=end_date, _capture=writer, **heartbeat_kwargs(heartbeat))
             writer.offer(result.batch)
             return result
 
     async def _read_batch(self, group_id, *, after_id=0, limit=200,
-                          download_media_to=None, start_date=None, end_date=None, _capture=None):
+                          download_media_to=None, start_date=None, end_date=None, heartbeat=None, _capture=None):
         result = await self._read(group_id, after_id=after_id, limit=limit,
             download_media_to=download_media_to, start_date=start_date,
-            end_date=end_date, _capture=_capture)
+            end_date=end_date, _capture=_capture, **heartbeat_kwargs(heartbeat))
         return result.batch
 
     async def get_message_batch(self, group_id, *, capture, after_id=0, limit=200,
-                                download_media_to=None, start_date=None, end_date=None):
+                                download_media_to=None, start_date=None, end_date=None, heartbeat=None):
         """Return a bounded batch; keep capture to inspect data after a stop."""
         with _capture_call(capture) as writer:
             batch = await self._read_batch(group_id, after_id=after_id, limit=limit,
                 download_media_to=download_media_to, start_date=start_date,
-                end_date=end_date, _capture=writer)
+                end_date=end_date, _capture=writer, **heartbeat_kwargs(heartbeat))
             writer.offer(batch)
             return batch
 
-    async def sync_group(self, chat_id, *, capture, limit=200, download_media_to=None):
+    async def sync_group(self, chat_id, *, capture, limit=200, download_media_to=None, heartbeat=None):
         """Prepare/replay daily delivery with a new caller-retained capture.
 
         Only the saved pending batch's receipt can acknowledge progress. The
@@ -554,22 +557,23 @@ class AccountPool(ProgressFacade):
         """
         with _capture_call(capture) as writer:
             batch = await self.sync_engine.prepare(chat_id, limit=limit,
-                download_media_to=download_media_to, _capture=writer)
+                download_media_to=download_media_to, _capture=writer, **heartbeat_kwargs(heartbeat))
             writer.offer(batch)
             return batch
 
     def _new_backfill_engine(self, collection):
         return BackfillEngine(self._backfill_store, collection, read_batch=self._read_batch)
 
-    async def prepare_backfill(self, context, *, capture, download_media_to=None):
+    async def prepare_backfill(self, context, *, capture, download_media_to=None, heartbeat=None):
         """Prepare/replay one historical turn with a new retained capture.
 
         Source closure and progress publication have separate lifetimes; this
         call finalizes its capture only after its own preparation ends.
         """
         with _capture_call(capture) as writer:
+            progress = heartbeat_kwargs(heartbeat)
             engine, context = self._backfill_for(context, BackfillPrepareContext)
-            turn = await engine.prepare(context, download_media_to=download_media_to, _capture=writer)
+            turn = await engine.prepare(context, download_media_to=download_media_to, _capture=writer, **progress)
             writer.offer(turn.batch)
             return turn
 

tgdata-0.1.1/tgdata/album.py

@@ -0,0 +1,246 @@
+"""Pure, opt-in assembly of observed Telegram album members.
+
+Posts describe observed data, never source receipts or globally complete albums.
+Callers retain prior posts and commit them before acknowledging raw deliveries.
+"""
+
+from collections.abc import Mapping
+from dataclasses import dataclass, field
+import hashlib
+import json
+
+from .message_batch import (
+    MessageBatch, BatchFormatError, MIN_LONG, _canonical, _digest, _identifier,
+    _invalid_constant, _json_pairs, _keys, _validate_record,
+)
+
+
+DEFAULT_MAX_MEMBERS = 10000
+DEFAULT_MAX_BYTES = 16 * 1024 * 1024
+_KEYS = frozenset({'schema', 'version', 'post_id', 'revision', 'chat_id',
+                   'grouped_id', 'media_mode', 'membership', 'members'})
+
+
+class AlbumAssemblyError(ValueError):
+    """Local format, conflict or resource refusal; never an account-health event."""
+
+    def __init__(self, message):
+        super().__init__(message)
+        self.__suppress_context__ = True
+
+
+def _limits(max_members, max_bytes):
+    for value, name in ((max_members, 'max_members'), (max_bytes, 'max_bytes')):
+        if type(value) is not int or value <= 0:
+            raise AlbumAssemblyError('{} must be a positive integer'.format(name))
+
+
+def _bytes(text, maximum):
+    try:
+        size = len(text.encode('utf-8')) if isinstance(text, str) else len(text)
+    except UnicodeError:
+        raise AlbumAssemblyError('post data must be valid UTF-8') from None
+    if size > maximum:
+        raise AlbumAssemblyError('max_bytes exceeded')
+    return size
+
+
+def _identity(chat_id, member):
+    grouped = member['grouped_id']
+    return 'tg:{}:{}:{}'.format(chat_id, 'album' if grouped is not None else 'message',
+                                grouped if grouped is not None else member['id'])
+
+
+def _revision(value):
+    body = {key: item for key, item in value.items() if key != 'revision'}
+    return hashlib.sha256(_canonical(body).encode('utf-8')).hexdigest()
+
+
+def _validate(value, max_members):
+    _keys(value, _KEYS, 'post')
+    if value['schema'] != 'tgdata.telegram-post' or type(value['version']) is not int or value['version'] != 1:
+        raise AlbumAssemblyError('unsupported post schema or version')
+    _identifier(value['chat_id'], 'post.chat_id', MIN_LONG, -1)
+    _identifier(value['grouped_id'], 'post.grouped_id', nullable=True)
+    _digest(value['revision'], 'post.revision')
+    if value['media_mode'] not in ('references', 'download'):
+        raise AlbumAssemblyError('unsupported post media_mode')
+    members = value['members']
+    if type(members) is not list or not members or len(members) > max_members:
+        raise AlbumAssemblyError('members must be nonempty and within max_members')
+    grouped = value['grouped_id'] is not None
+    if value['membership'] != ('unconfirmed' if grouped else 'single'):
+        raise AlbumAssemblyError('post membership must reflect observed album or singleton data')
+    if not grouped and len(members) != 1:
+        raise AlbumAssemblyError('an ungrouped post must have one member')
+    last = 0
+    for member in members:
+        ident = _validate_record(member, value['media_mode'])
+        if ident <= last:
+            raise AlbumAssemblyError('member IDs must increase strictly')
+        last = ident
+        if member['grouped_id'] != value['grouped_id'] or _identity(value['chat_id'], member) != value['post_id']:
+            raise AlbumAssemblyError('member does not match post identity')
+    if value['revision'] != _revision(value):
+        raise AlbumAssemblyError('post revision does not match content')
+
+
+@dataclass(frozen=True, init=False)
+class TelegramPost:
+    """Immutable observed post; getters return copies, albums stay unconfirmed.
+
+    The revision hashes canonical content, independently of raw batch partitions.
+    Loading validates the entire value. Limits can be raised explicitly; they
+    are a serialized-work policy, not a promise about total process memory.
+    """
+
+    _encoded: str = field(repr=False)
+
+    def __init__(self, document, *, max_members=DEFAULT_MAX_MEMBERS, max_bytes=DEFAULT_MAX_BYTES):
+        _limits(max_members, max_bytes)
+        if type(document) is not dict:
+            raise AlbumAssemblyError('post must be a JSON object')
+        members = document.get('members')
+        if type(members) is list and len(members) > max_members:
+            raise AlbumAssemblyError('max_members exceeded')
+        try:
+            encoded = _canonical(document)
+            _bytes(encoded, max_bytes)
+            owned = json.loads(encoded)
+            _validate(owned, max_members)
+        except BatchFormatError as error:
+            raise AlbumAssemblyError(str(error)) from None
+        object.__setattr__(self, '_encoded', encoded)
+
+    @classmethod
+    def from_json(cls, data, *, max_members=DEFAULT_MAX_MEMBERS, max_bytes=DEFAULT_MAX_BYTES):
+        _limits(max_members, max_bytes)
+        if type(data) not in (str, bytes):
+            raise AlbumAssemblyError('post JSON must be text or UTF-8 bytes')
+        _bytes(data, max_bytes)
+        try:
+            if isinstance(data, bytes):
+                data = data.decode('utf-8')
+            value = json.loads(data, object_pairs_hook=_json_pairs, parse_constant=_invalid_constant)
+        except (BatchFormatError, ValueError, UnicodeError, RecursionError):
+            raise AlbumAssemblyError('invalid post JSON') from None
+        return cls(value, max_members=max_members, max_bytes=max_bytes)
+
+    def to_json(self):
+        return self._encoded
+
+    def to_dict(self):
+        return json.loads(self._encoded)
+
+    @property
+    def post_id(self):
+        return self.to_dict()['post_id']
+
+    @property
+    def revision(self):
+        return self.to_dict()['revision']
+
+    @property
+    def chat_id(self):
+        return int(self.to_dict()['chat_id'])
+
+    @property
+    def grouped_id(self):
+        value = self.to_dict()['grouped_id']
+        return int(value) if value is not None else None
+
+    @property
+    def media_mode(self):
+        return self.to_dict()['media_mode']
+
+    @property
+    def membership(self):
+        return self.to_dict()['membership']
+
+    @property
+    def members(self):
+        return self.to_dict()['members']
+
+    @property
+    def member_ids(self):
+        return tuple(int(member['id']) for member in self.to_dict()['members'])
+
+
+def _batch(batch, max_bytes):
+    if type(batch) is not MessageBatch:
+        raise AlbumAssemblyError('batch must be a MessageBatch')
+    size = _bytes(batch.to_json(), max_bytes)
+    return batch.to_dict(), size
+
+
+def album_post_ids(batch, *, max_bytes=DEFAULT_MAX_BYTES):
+    """Return affected post keys in first occurrence order, without any I/O."""
+    _limits(DEFAULT_MAX_MEMBERS, max_bytes)
+    value, _ = _batch(batch, max_bytes)
+    return tuple(dict.fromkeys(_identity(value['chat_id'], row) for row in value['messages']))
+
+
+def assemble_albums(batch, *, previous, max_members=DEFAULT_MAX_MEMBERS, max_bytes=DEFAULT_MAX_BYTES):
+    """Return updated affected posts from a raw batch and authoritative prior data.
+
+    ``previous`` must map exactly every affected key to a TelegramPost or None.
+    None asserts successful lookup found no prior post; lookup errors must raise.
+    Load current prior values and save results in a caller-owned transaction/CAS.
+    Exact duplicates are idempotent; incompatible observations refuse entirely.
+
+    max_members bounds total returned members; max_bytes bounds the sum of raw
+    batch JSON, supplied post JSON and returned post JSON in UTF-8. No data is
+    silently evicted. This function never fetches, saves or acknowledges.
+    """
+    _limits(max_members, max_bytes)
+    value, size = _batch(batch, max_bytes)
+    groups = {}
+    for member in value['messages']:
+        groups.setdefault(_identity(value['chat_id'], member), []).append(member)
+    if not isinstance(previous, Mapping) or len(previous) != len(groups) or set(previous) != set(groups):
+        raise AlbumAssemblyError('previous must contain exactly the affected post IDs')
+
+    # Check encoded input sizes before decoding prior observations. Revalidate
+    # each saved value with the caller's policy, including deliberately raised limits.
+    prior_documents = {}
+    for key in groups:
+        post = previous[key]
+        if post is None:
+            continue
+        if type(post) is not TelegramPost:
+            raise AlbumAssemblyError('previous values must be TelegramPost or None')
+        size += _bytes(post.to_json(), max_bytes)
+        if size > max_bytes:
+            raise AlbumAssemblyError('max_bytes exceeded')
+        loaded = TelegramPost.from_json(post.to_json(), max_members=max_members, max_bytes=max_bytes).to_dict()
+        if loaded['post_id'] != key or loaded['media_mode'] != value['media_mode']:
+            raise AlbumAssemblyError('prior post identity or media mode differs')
+        prior_documents[key] = loaded
+
+    output, count, seen_ids = [], 0, set()
+    for key, incoming in groups.items():
+        prior = prior_documents.get(key)
+        members = {row['id']: row for row in prior['members']} if prior is not None else {}
+        for row in incoming:
+            if row['id'] in members and members[row['id']] != row:
+                raise AlbumAssemblyError('conflicting observations of a member')
+            members[row['id']] = row
+        count += len(members)
+        if count > max_members:
+            raise AlbumAssemblyError('max_members exceeded')
+        if seen_ids.intersection(members):
+            raise AlbumAssemblyError('a member appears in conflicting posts')
+        seen_ids.update(members)
+        ordered = sorted(members.values(), key=lambda row: int(row['id']))
+        document = dict(schema='tgdata.telegram-post', version=1, post_id=key,
+                        chat_id=value['chat_id'], grouped_id=incoming[0]['grouped_id'],
+                        media_mode=value['media_mode'],
+                        membership='unconfirmed' if incoming[0]['grouped_id'] is not None else 'single',
+                        members=ordered)
+        document['revision'] = _revision(document)
+        post = TelegramPost(document, max_members=max_members, max_bytes=max_bytes)
+        size += _bytes(post.to_json(), max_bytes)
+        if size > max_bytes:
+            raise AlbumAssemblyError('max_bytes exceeded')
+        output.append((int(ordered[0]['id']), key, post))
+    return tuple(item[2] for item in sorted(output))

{tgdata-0.1 → tgdata-0.1.1}/tgdata/backfill_engine.py

@@ -11,6 +11,8 @@ import re
 import time
 from uuid import uuid4
 
+from .batch_heartbeat import heartbeat_kwargs
+
 from .backfill import (
     BackfillRunRef, BackfillStartRequest, BackfillStartResult, BackfillRecoveryResult,
     BackfillControlResult,
@@ -384,13 +386,14 @@ class BackfillEngine:
                 return self._turn(state)
         raise BackfillConflictError('source settlement kept conflicting; inspect the authoritative attempt')
 
-    async def prepare(self, context, *, download_media_to=None, _capture=None):
+    async def prepare(self, context, *, download_media_to=None, heartbeat=None, _capture=None):
         """Prepare/replay one exact delivery using current run/cursor/control intent.
 
         Early pacing returns a wait without sleeping or reading. Pending replay
         remains local, including while paused or cancelled. The
         caller excludes other source readers in separate collections/daily jobs.
         """
+        progress = heartbeat_kwargs(heartbeat)
         if not isinstance(context, BackfillPrepareContext):
             raise BackfillConfigurationError('prepare requires BackfillPrepareContext')
         context = BackfillPrepareContext.from_dict(context.to_dict())
@@ -430,7 +433,7 @@ class BackfillEngine:
             options = dict(after_id=int(admitted['after_id']), limit=admitted['request']['batch_size'],
                            download_media_to=root,
                            start_date=_decode_date(admitted['window']['start_date']),
-                           end_date=_decode_date(admitted['window']['end_date']))
+                           end_date=_decode_date(admitted['window']['end_date']), **progress)
             if _capture is not None:
                 options['_capture'] = _capture
             try:

{tgdata-0.1 → tgdata-0.1.1}/tgdata/batch_engine.py

@@ -9,6 +9,7 @@ from telethon import utils as tl_utils
 from telethon.tl import types
 
 from .batch_files import download_blob, prepare_directory
+from .batch_heartbeat import heartbeat_scope
 from .message_batch import (
     BatchFormatError, MessageBatch, MAX_BATCH_MESSAGES, MAX_MESSAGE_ID, MAX_LONG,
     MIN_LONG, _identifier, _validate_record,
@@ -151,7 +152,15 @@ class BatchEngine:
         self._retain_cancelled_prefix = _retain_cancelled_prefix
 
     async def fetch_batch(self, group_id, *, after_id=0, limit=200, download_media_to=None,
-                          start_date=None, end_date=None, _capture=None):
+                          start_date=None, end_date=None, heartbeat=None, _capture=None):
+        with heartbeat_scope(heartbeat) as beat:
+            return await self._fetch_batch(group_id, after_id=after_id, limit=limit,
+                download_media_to=download_media_to, start_date=start_date,
+                end_date=end_date, _capture=_capture, _beat=beat)
+
+    async def _fetch_batch(self, group_id, *, after_id=0, limit=200, download_media_to=None,
+                           start_date=None, end_date=None, _capture=None, _beat=None):
+        beat = _beat if _beat is not None else lambda phase: None
         group_id = _group_argument(group_id)
         after_id = _integer(after_id, 'after_id', 0, MAX_MESSAGE_ID)
         limit = _integer(limit, 'limit', 1, MAX_BATCH_MESSAGES)
@@ -166,14 +175,18 @@ class BatchEngine:
         records = []
         scan_id = after_id
         try:
+            beat("connect")
             async with self.connection_engine.session() as client:
+                beat("connected")
+                beat("resolve")
                 if self._resolve_entity is not None:
                     entity = await self._resolve_entity(client, group_id)
                 else:
                     try:
                         entity = await client.get_entity(group_id)
                     except ValueError:
-                        entity = await MessageEngine._entity_after_dialog_sync(client, group_id)
+                        entity = await MessageEngine._entity_after_dialog_sync(client, group_id, beat)
+                beat("resolved")
                 if isinstance(entity, (types.ChatForbidden, types.ChannelForbidden)):
                     raise GroupAccessError('account has no access to the requested group')
                 if not isinstance(entity, (types.Chat, types.Channel)):
@@ -203,6 +216,7 @@ class BatchEngine:
                             raise BatchFormatError('message IDs did not advance after the cursor')
                         scan_id = ident
                         scanned += 1
+                        beat("message scanned")
                         if window is not None:
                             date = _utc(message.date, 'message.date', allow_naive=True)
                             if date >= window.end_date:
@@ -215,11 +229,13 @@ class BatchEngine:
                         media = record['media']
                         if media_directory is not None and media is not None and media['downloadable']:
                             async def writer(output):
-                                return await client.download_media(message, file=output)
+                                options = {} if _beat is None else {"progress_callback": lambda current, total: beat("media")}
+                                return await client.download_media(message, file=output, **options)
 
                             media['blob'] = await download_blob(media_directory, writer, expected_size=media['size'])
                         _validate_record(record, mode)
                         records.append(record)
+                        beat("messages")
                     if end_reached or scanned < remaining or scan_id == MAX_MESSAGE_ID:
                         break
                     # A full page of excluded lower-bound messages is not the