TeLLMgramBot 3.0.4__tar.gz → 3.2.0__tar.gz

{tellmgrambot-3.0.4 → tellmgrambot-3.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: TeLLMgramBot
-Version: 3.0.4
+Version: 3.2.0
 Summary: LLM-powered Telegram bot (OpenAI + Anthropic)
 Home-page: https://github.com/Digital-Heresy/TeLLMgramBot
 Author: Digital Heresy
@@ -43,10 +43,10 @@ The basic goal of this project is to create a bridge between a Telegram Bot and
   * This uses a separate model (configurable via `url_model`) to support more URL content with its higher token limit.
 * Tokens are used to measure the length of all conversation messages between the Telegram bot assistant and the user. This is useful to:
   * Ensure the length does not go over the model limit. If it does, prune oldest messages to fit within the limit.
-  * Remember past conversations when restarting: loads the user's full history across all chats (private and groups) plus all other participants' messages in the current chat, up to 50% of the token budget. This eliminates amnesia when users switch between contexts.
+  * Remember past conversations when restarting: loads the user's full history across all chats (private and groups) plus all other participants' messages in the current chat, up to 50% of the token budget. In private chats, shared group context (messages from groups where both user and bot are active) fills the remaining budget, enabling the bot to reference group conversations from a private context. This eliminates amnesia when users switch between contexts.
 * Users can manage privacy via two commands:
-  * `/forget` — In private chats, clears the full conversation (including bot replies). In group chats, removes only your messages; other participants' messages remain.
-  * `/private` — Toggles per-user private mode (private chats only). When ON, messages are excluded from group conversation contexts, enabling selective privacy even in shared groups.
+  * `/forget` — In private chats, clears the full conversation (including bot replies) and resets all of your active sessions (any sessions where your context was merged). In group chats, removes your messages across all chat types; other participants' messages and sessions remain.
+  * `/private` — Toggles per-user private mode (private chats only). When ON, messages are excluded from group conversation contexts, enabling selective privacy even in shared groups. If you have private mode ON and message a group, the bot will remind you once per session.
 
 ## Why Telegram?
 Using Telegram as the interface not only solves "exposing" the interface, but gives you boatloads of interactivity over a standard Command Line interface, or trying to create a website with input boxes and submit buttons to try to handle everything:
@@ -87,7 +87,7 @@ When initializing TeLLMgramBot, the following directories get created:
    * User will also get notified to contact the owner.
 * `data`
   * Contains `conversations.db` — a SQLite database storing all conversations between the bot and users across all chats.
-  * When a user messages in any chat, their full history is available for context: private messages appear in group contexts, group messages appear in private contexts. This creates seamless cross-context awareness.
+  * When a user messages in any chat, their full history is available for context: private messages appear in group contexts, group messages appear in private contexts. This creates seamless cross-context awareness. The bot dynamically refreshes each user's context during a session to pick up messages sent between chats. In private chats, the bot also loads shared group context (messages from groups where both user and bot are active) to provide group awareness.
   * Users can manage their context via `/forget` (private chat: clears full conversation; group chat: removes only your messages) or `/private` (toggles per-user privacy for group contexts).
 
 ### Environment Variables

{tellmgrambot-3.0.4 → tellmgrambot-3.2.0}/README.md

@@ -12,10 +12,10 @@ The basic goal of this project is to create a bridge between a Telegram Bot and
   * This uses a separate model (configurable via `url_model`) to support more URL content with its higher token limit.
 * Tokens are used to measure the length of all conversation messages between the Telegram bot assistant and the user. This is useful to:
   * Ensure the length does not go over the model limit. If it does, prune oldest messages to fit within the limit.
-  * Remember past conversations when restarting: loads the user's full history across all chats (private and groups) plus all other participants' messages in the current chat, up to 50% of the token budget. This eliminates amnesia when users switch between contexts.
+  * Remember past conversations when restarting: loads the user's full history across all chats (private and groups) plus all other participants' messages in the current chat, up to 50% of the token budget. In private chats, shared group context (messages from groups where both user and bot are active) fills the remaining budget, enabling the bot to reference group conversations from a private context. This eliminates amnesia when users switch between contexts.
 * Users can manage privacy via two commands:
-  * `/forget` — In private chats, clears the full conversation (including bot replies). In group chats, removes only your messages; other participants' messages remain.
-  * `/private` — Toggles per-user private mode (private chats only). When ON, messages are excluded from group conversation contexts, enabling selective privacy even in shared groups.
+  * `/forget` — In private chats, clears the full conversation (including bot replies) and resets all of your active sessions (any sessions where your context was merged). In group chats, removes your messages across all chat types; other participants' messages and sessions remain.
+  * `/private` — Toggles per-user private mode (private chats only). When ON, messages are excluded from group conversation contexts, enabling selective privacy even in shared groups. If you have private mode ON and message a group, the bot will remind you once per session.
 
 ## Why Telegram?
 Using Telegram as the interface not only solves "exposing" the interface, but gives you boatloads of interactivity over a standard Command Line interface, or trying to create a website with input boxes and submit buttons to try to handle everything:
@@ -56,7 +56,7 @@ When initializing TeLLMgramBot, the following directories get created:
    * User will also get notified to contact the owner.
 * `data`
   * Contains `conversations.db` — a SQLite database storing all conversations between the bot and users across all chats.
-  * When a user messages in any chat, their full history is available for context: private messages appear in group contexts, group messages appear in private contexts. This creates seamless cross-context awareness.
+  * When a user messages in any chat, their full history is available for context: private messages appear in group contexts, group messages appear in private contexts. This creates seamless cross-context awareness. The bot dynamically refreshes each user's context during a session to pick up messages sent between chats. In private chats, the bot also loads shared group context (messages from groups where both user and bot are active) to provide group awareness.
   * Users can manage their context via `/forget` (private chat: clears full conversation; group chat: removes only your messages) or `/private` (toggles per-user privacy for group contexts).
 
 ### Environment Variables

{tellmgrambot-3.0.4 → tellmgrambot-3.2.0}/TeLLMgramBot/TeLLMgramBot.py

@@ -92,21 +92,38 @@ class TelegramBot:
         """
         Remove conversation history from the database (behavior depends on chat type).
 
-        In private chats: deletes all messages in the chat (including bot replies), fully clearing the conversation.
-        In group chats: deletes only this user's rows; other participants' messages remain.
-        Also removes the chat's Conversation object from this bot session's memory if it exists.
+        In private chats: deletes all messages in the private chat (including bot replies).
+        In group chats: deletes the user's rows across all chats (private included).
+
+        In both cases, evicts only the in-memory Conversation objects that contain this
+        user's data — scoped to the current chat, the user's private chat, and any other
+        chats where their context was previously merged. Other users' active sessions are
+        not affected. All private-mode notices for this user are also cleared.
         """
         user_id = update.message.from_user.id
         chat_id = update.message.chat.id
         chat_type = update.message.chat.type
-        # Private chats: delete all messages in the chat (including bot replies)
-        # Group chats: delete only the user's rows (other participants' messages remain)
+
+        # Find all chat_ids where this user's context has been merged in memory,
+        # identified by user_id being present in the Conversation's _context_cursor.
+        user_chat_ids = {
+            gid for gid, conv in self.conversations.items() if user_id in conv._context_cursor
+        }
+        # Always include the private chat (chat_id == user_id in Telegram) and current chat.
+        user_chat_ids.update({chat_id, user_id})
+
         if chat_type == 'private':
             await delete_messages_for_chat(chat_id)
         else:
             await delete_messages_for_user(user_id)
-        if chat_id in self.conversations:
-            del self.conversations[chat_id]
+
+        # Evict only the Conversations that contain this user's data, not all sessions.
+        for evict_id in user_chat_ids:
+            self.conversations.pop(evict_id, None)
+            self.token_warning.pop(evict_id, None)
+        # Clear all private-mode notices for this user across all chats.
+        self._private_mode_warned = {k for k in self._private_mode_warned if k[1] != user_id}
+
         await update.message.reply_text("My memories of our conversations are wiped!")
 
     async def tele_private_command(self, update: Update, context: ContextTypes.DEFAULT_TYPE):
@@ -193,7 +210,7 @@ class TelegramBot:
         chat_type = msg.chat.type
         print(f"User {user_id} ({username}) in {chat_type} Chat {chat_id}")
 
-        # For a new session, create a Conversation keyed by chat_id and load past history
+        # For a new session, create a Conversation keyed by chat_id
         if chat_id not in self.conversations:
             self.conversations[chat_id] = Conversation(
                 chat_id,
@@ -201,14 +218,37 @@ class TelegramBot:
                 self.chatgpt['prompt'],
                 self.chatgpt['chat_model']
             )
-            await self.conversations[chat_id].get_past_interaction(
-                floor(self.chatgpt['prune_threshold'] / 2), user_id
-            )
+
+        conv = self.conversations[chat_id]
+        token_budget = floor(self.chatgpt['prune_threshold'] / 2)
+
+        if user_id not in conv._context_cursor:
+            # First appearance of this user in this session — load their cross-chat history.
+            # Pass bot_id so private chats can also load shared group context.
+            # If no history exists yet, the cursor stays unset so the next message retries.
+            await conv.get_past_interaction(token_budget, user_id, self.telegram['bot_id'])
+        else:
+            # Already loaded — check for new cross-chat messages since last load.
+            await conv.refresh_user_context(user_id, token_budget)
 
         # Add the user's message to our conversation, respecting private mode
         # Private mode only applies in private chats — group messages are never flagged private
-        is_private = (chat_type == 'private') and await get_private_mode(user_id)
-        await self.conversations[chat_id].add_user_message(text, user_id, username, is_private)
+        user_private_mode = await get_private_mode(user_id)
+        is_private = (chat_type == 'private') and user_private_mode
+        await conv.add_user_message(text, user_id, username, is_private)
+
+        # Warn once per group session if the user has private mode ON.
+        # Their private-chat messages are excluded from this group's context, which can be
+        # surprising. The notice is appended to the first reply in the session for this user.
+        private_mode_notice = ""
+        if chat_type in ('group', 'supergroup') and user_private_mode:
+            warn_key = (chat_id, user_id)
+            if warn_key not in self._private_mode_warned:
+                self._private_mode_warned.add(warn_key)
+                private_mode_notice = (
+                    "\n\n(Note: your private mode is ON — messages from our private chat "
+                    "are not included in this group's context. To disable, send /private off in our DMs.)"
+                )
 
         # Check if the user is asking about a [URL]
         url_match = re.search(r'\[http(s)?://\S+]', text)
@@ -226,8 +266,11 @@ class TelegramBot:
             # This is the transition point between quick Telegram replies and the LLM
             reply = await self.llm_completion(chat_id)
 
+        if private_mode_notice:
+            reply += private_mode_notice
+
         # Check token count before storing to determine if warning should be appended
-        token_count = await self.conversations[chat_id].get_message_token_count()
+        token_count = await conv.get_message_token_count()
         if token_count > self.chatgpt['prune_back_to'] and chat_id not in self.token_warning:
             reply += ("\n\n"
                 "By the way, our conversation will soon reach my token limit, so I may"
@@ -236,15 +279,17 @@ class TelegramBot:
             )
             self.token_warning[chat_id] = True
 
-        # Add the full reply (including any warning) to the conversation
-        await self.conversations[chat_id].add_assistant_message(
-            reply, self.telegram['bot_id'], self.telegram['username']
+        # Add the full reply (including any warning) to the conversation.
+        # Propagate is_private so bot replies in private-mode exchanges are also excluded
+        # from group context loading — otherwise the assistant's half of the conversation leaks.
+        await conv.add_assistant_message(
+            reply, self.telegram['bot_id'], self.telegram['username'], is_private
         )
 
         # Recompute token count after full exchange is stored, then prune if needed
-        token_count = await self.conversations[chat_id].get_message_token_count()
+        token_count = await conv.get_message_token_count()
         if token_count > self.chatgpt['prune_threshold']:
-            await self.conversations[chat_id].prune_conversation(self.chatgpt['prune_back_to'])
+            await conv.prune_conversation(self.chatgpt['prune_back_to'])
 
         return reply
 
@@ -396,6 +441,7 @@ class TelegramBot:
         self.error_log = generate_error_path()
         self.token_warning = {}  # Determines whether user has reached token limit by AI model
         self.conversations = {}  # Provides Conversation class per user based on bot response
+        self._private_mode_warned = set()  # (chat_id, user_id) pairs that received the private mode notice
         self.telegram = {
             'bot_id'       : 0,  # overwritten by _tele_info(); 0 is a safe sentinel
             'owner'        : bot_owner,

tellmgrambot-3.2.0/TeLLMgramBot/conversation.py

@@ -0,0 +1,325 @@
+"""
+Manages conversation history, token counting, and SQLite-backed persistence for any LLM provider.
+
+The Conversation class is keyed by chat_id (Telegram chat ID) and maintains an in-memory message
+history, delegating persistence to the database module. Supports async token counting via provider-
+specific APIs and automatic pruning when token limits are approached.
+"""
+from .database import (
+    insert_message,
+    load_full_user_context,
+    get_max_cross_chat_message_id,
+    load_new_cross_chat_context,
+    get_shared_group_chat_ids,
+    load_shared_group_context,
+    delete_messages_for_user,
+    delete_messages_for_chat,
+)
+from .models import TokenLimits
+from .utils import generate_error_path, log_error
+
+# Minimum remaining token headroom required before loading shared group context
+# in a private chat. Prevents loading group history when the private context
+# already fills nearly the entire budget.
+MIN_GROUP_CONTEXT_TOKENS = 500
+
+class Conversation:
+    """
+    Manages a chat conversation including message history, token counting, and SQLite persistence.
+
+    Keyed by chat_id (Telegram chat ID): negative for groups, positive (== user_id) for private chats.
+    Works with any LLM provider (OpenAI, Anthropic). When the token limit is approached, oldest
+    messages are pruned from memory; the database retains the full history.
+    """
+
+    def __init__(self, chat_id: int, chat_type: str, system_content: str, system_model: str = "gpt-4o-mini"):
+        """
+        Initialize a conversation keyed by chat_id.
+
+        Args:
+            chat_id: Telegram chat ID (negative for groups, positive/==user_id for private chats).
+            chat_type: Chat type ('private', 'group', 'supergroup', 'channel').
+            system_content: System prompt (bot persona).
+            system_model: LLM model to use for token counting (default: "gpt-4o-mini").
+        """
+        self.error_log = generate_error_path()
+
+        self.chat_id = chat_id
+        self.chat_type = chat_type
+        self.chat_print = f"{self.chat_type} Chat {self.chat_id}"
+
+        self.system_content = system_content
+        self.system_model = TokenLimits(system_model)
+        self.messages = [{"role": "system", "content": system_content}]
+
+        # Maps user_id → max message id at the time their cross-chat context was loaded.
+        # Used for both per-user idempotency (is the user loaded?) and mid-session refresh
+        # detection (have new cross-chat messages appeared since the last load?).
+        self._context_cursor: dict[int, int] = {}
+
+        # Maps user_id → len(self.messages) after their history was loaded.
+        # Marks the boundary between historical context and live session messages so that
+        # refresh_user_context() inserts delta rows at the right position (after history,
+        # before live messages) rather than at index 1 (before all existing history).
+        self._history_end: dict[int, int] = {}
+
+    def set_system_content(self, new_content: str):
+        """
+        Replace the system content (bot's personality prompt) in the conversation.
+
+        Updates both self.system_content and the first message in self.messages.
+
+        Args:
+            new_content: The new system prompt text.
+        """
+        self.system_content = new_content
+        self.messages[0] = {"role": "system", "content": new_content}
+
+    async def add_user_message(self, content: str, user_id: int, username: str | None, is_private: bool = False):
+        """
+        Add a user message to conversation memory and persist it to the database.
+
+        Args:
+            content: The message text.
+            user_id: Telegram user ID of the sender.
+            username: Telegram username (display only, may be None).
+            is_private: If True, excludes this message from group context loading (used for /forget or private commands).
+        """
+        self.messages.append({"role": "user", "content": content})
+        await insert_message(self.chat_id, user_id, username, "user", content, is_private)
+
+    async def add_assistant_message(self, content: str, bot_user_id: int, bot_username: str | None, is_private: bool = False):
+        """
+        Add an assistant message to conversation memory and persist it to the database.
+
+        Args:
+            content: The message text (LLM response).
+            bot_user_id: Telegram user ID of the bot account.
+            bot_username: Telegram username of the bot (display only, may be None).
+            is_private: If True, excludes this reply from group context loading. Should match
+                        the is_private flag of the user message it responds to, so that both
+                        sides of a private-mode exchange are excluded from shared contexts.
+        """
+        self.messages.append({"role": "assistant", "content": content})
+        await insert_message(self.chat_id, bot_user_id, bot_username, "assistant", content, is_private)
+
+    async def get_message_token_count(self) -> int:
+        """
+        Get the current token count of all messages in this conversation.
+
+        Delegates to the configured model's provider to count tokens accurately.
+        This is an async operation because it may contact the provider API (e.g., for Anthropic).
+
+        Returns:
+            The total token count for all messages (system, user, assistant).
+
+        Raises:
+            ProviderAuthError: If API credentials are missing or invalid.
+            ProviderConnectionError: If the provider API cannot be reached.
+        """
+        return await self.system_model.num_tokens_from_messages(self.messages)
+
+    async def prune_conversation(self, token_limit: int):
+        """
+        Remove oldest user-assistant messages until token count is below threshold.
+
+        Recursively removes messages starting from index 1 (preserving the system message at index 0)
+        until the conversation fits within the token limit. The database is not modified — pruning
+        only affects the in-memory context window for the current session.
+
+        Args:
+            token_limit: The maximum token count threshold. Messages are removed until
+                         the conversation is strictly below this limit.
+        """
+        try:
+            self.messages.pop(1)
+            if await self.get_message_token_count() > token_limit:
+                await self.prune_conversation(token_limit)
+        except Exception as e:
+            log_error(e, f"{type(e).__name__} pruning under {token_limit} tokens for {self.chat_print}", self.error_log)
+
+    async def get_past_interaction(self, token_limit: int, user_id: int, bot_id: int | None = None) -> bool:
+        """
+        Load past conversation messages from the database up to a token limit.
+
+        Implements bidirectional cross-pollination: loads all messages from the current chat
+        (all participants) plus all messages from the user's private chat (both user and
+        assistant turns), merged chronologically. This eliminates amnesia when a user switches
+        between private and group contexts.
+
+        For group and supergroup chats, messages marked as private (is_private=1) are excluded
+        so private-mode messages don't surface in shared contexts.
+
+        For private chats, if bot_id is provided and remaining token budget allows, also loads
+        full message history from any groups shared between the user and the bot. This lets the
+        bot answer questions about group conversations from a private chat context.
+
+        Per-user idempotent: once a user's cursor is set, subsequent calls return True immediately.
+        Calling for a new user_id merges that user's cross-chat context into the existing
+        conversation. Context is considered "found" only if messages were actually inserted
+        (rows_inserted > 0 or group_rows_inserted > 0).
+
+        Args:
+            token_limit: The maximum token count to load into the in-memory context.
+            user_id: Telegram user ID of the requesting user; used to fetch bidirectional context.
+            bot_id: Telegram user ID of the bot account. When provided in a private chat,
+                    enables shared group context loading into any remaining token budget.
+
+        Returns:
+            True if past interactions were found (either on this call or previously),
+            False if no past interactions exist for this user.
+
+        Side Effects:
+            - Modifies self.messages in-place by inserting historical messages after the system message
+              (attempting to fit messages starting from most recent, stopping when limit is reached).
+            - Sets self._context_cursor[user_id] to the max message id loaded once context is found.
+            - Sets self._history_end[user_id] to len(self.messages) after loading, marking the
+              boundary between historical context and live session messages for future refreshes.
+            - Prints to stdout showing token count and whether the token limit was reached.
+        """
+        if user_id in self._context_cursor:
+            print(f"Context already loaded for User {user_id} in {self.chat_print}")
+            return True
+
+        exclude_private = self.chat_type in ('group', 'supergroup')
+        rows = await load_full_user_context(user_id, self.chat_id, exclude_private=exclude_private)
+
+        token_limit_reached = False
+        token_count = 0
+        rows_inserted = 0
+
+        for row in reversed(rows):
+            self.messages.insert(1, row)
+            token_count = await self.get_message_token_count()
+            if token_count > token_limit:
+                self.messages.pop(1)
+                token_count = await self.get_message_token_count()
+                token_limit_reached = True
+                break
+            rows_inserted += 1
+
+        # Load shared group context for private chats when token budget allows.
+        # Private context fills the primary budget; group context fills the remainder.
+        group_rows_inserted = 0
+        group_limit_reached = False
+        if self.chat_type == 'private' and bot_id is not None:
+            if token_count == 0:
+                token_count = await self.get_message_token_count()
+            remaining = token_limit - token_count
+            if remaining > MIN_GROUP_CONTEXT_TOKENS:
+                shared_groups = await get_shared_group_chat_ids(user_id, bot_id)
+                if shared_groups:
+                    group_rows = await load_shared_group_context(shared_groups, exclude_private=True)
+                    insert_at = 1 + rows_inserted
+                    for row in reversed(group_rows):
+                        self.messages.insert(insert_at, row)
+                        token_count = await self.get_message_token_count()
+                        if token_count > token_limit:
+                            self.messages.pop(insert_at)
+                            token_count = await self.get_message_token_count()
+                            group_limit_reached = True
+                            break
+                        group_rows_inserted += 1
+
+        has_context = rows_inserted > 0 or group_rows_inserted > 0
+
+        if has_context:
+            self._context_cursor[user_id] = await get_max_cross_chat_message_id(
+                user_id, self.chat_id, exclude_private=exclude_private
+            )
+            self._history_end[user_id] = len(self.messages)
+            label = "Full" if not token_limit_reached and not group_limit_reached else "Partial"
+            group_note = f", plus {group_rows_inserted} group message(s)" if group_rows_inserted > 0 else ""
+            print(
+                f"{label} context loaded for User {user_id} in {self.chat_print} — "
+                f"storing {token_count} token(s){group_note}"
+            )
+        else:
+            print(f"No prior context found for User {user_id} in {self.chat_print}")
+
+        return has_context
+
+    async def refresh_user_context(self, user_id: int, token_limit: int) -> bool:
+        """
+        Merge any new cross-chat messages that arrived since this user's context was last loaded.
+
+        Compares the current max message id in the cross-chat context against the stored
+        cursor for this user. If new rows exist, loads only the delta and merges it into
+        the in-memory conversation, respecting the token budget.
+
+        This handles the mid-session staleness case: a user whose context was loaded
+        at session start (with no private history) will pick up new private messages they
+        send before returning to the group, without requiring a full reload.
+
+        Args:
+            user_id: Telegram user ID of the user to refresh.
+            token_limit: Maximum token budget; new messages are discarded if they would
+                         push the conversation over this limit.
+
+        Returns:
+            True if at least one new message was merged, False otherwise.
+
+        Side Effects:
+            - Modifies self.messages in-place, inserting new rows at self._history_end[user_id]
+              (after historical context, before any live session messages).
+            - Advances self._context_cursor[user_id] and self._history_end[user_id] on success.
+        """
+        if user_id not in self._context_cursor:
+            return False
+
+        exclude_private = self.chat_type in ('group', 'supergroup')
+        current_max = await get_max_cross_chat_message_id(user_id, self.chat_id, exclude_private)
+
+        if current_max <= self._context_cursor[user_id]:
+            return False
+
+        new_rows = await load_new_cross_chat_context(
+            user_id, self.chat_id, self._context_cursor[user_id], exclude_private
+        )
+
+        if not new_rows:
+            return False
+
+        # Insert at the tracked history boundary so delta rows land after existing historical
+        # context and before any live session messages, preserving chronological ordering.
+        insert_pos = self._history_end.get(user_id, 1)
+        rows_inserted = 0
+        for row in reversed(new_rows):
+            self.messages.insert(insert_pos, row)
+            token_count = await self.get_message_token_count()
+            if token_count > token_limit:
+                self.messages.pop(insert_pos)
+                break
+            rows_inserted += 1
+
+        if rows_inserted > 0:
+            self._context_cursor[user_id] = current_max
+            self._history_end[user_id] = insert_pos + rows_inserted
+            print(f"Refreshed {rows_inserted} new message(s) for User {user_id} in {self.chat_print}")
+
+        return rows_inserted > 0
+
+    async def clear_interaction(self, user_id: int):
+        """
+        Clear this user's messages from memory and from the database.
+
+        In private chats, all messages in the chat are deleted (including bot replies),
+        fully clearing the conversation. In group chats, only this user's rows are removed;
+        other members' messages remain. Resets both _context_cursor and _history_end.
+
+        Args:
+            user_id: Telegram user ID of the user requesting the forget.
+
+        Side Effects:
+            - Clears self.messages to contain only the system message (index 0).
+            - Resets self._context_cursor to empty dict.
+            - Resets self._history_end to empty dict.
+            - Deletes all messages for this chat (private) or this user (group) from database.
+        """
+        self.messages = [self.messages[0]]
+        self._context_cursor = {}
+        self._history_end = {}
+        if self.chat_type == 'private':
+            await delete_messages_for_chat(self.chat_id)
+        else:
+            await delete_messages_for_user(user_id)

{tellmgrambot-3.0.4 → tellmgrambot-3.2.0}/TeLLMgramBot/database.py

@@ -14,6 +14,10 @@ from .utils import execution_dir
 # Allows tests to override the DB path without touching env vars
 _DB_PATH: Optional[str] = None
 
+# Maximum bound parameters per SQLite statement (SQLite default limit is 999).
+# Defined at module level so tests can patch it to force batching with small inputs.
+_SQLITE_MAX_PARAMS = 999
+
 _DDL = """
 PRAGMA journal_mode=WAL;
 
@@ -149,13 +153,27 @@ async def load_full_user_context(
     Load bidirectional conversation context for a user across all chats.
 
     Implements bidirectional cross-pollination: returns all messages from the current
-    chat (all participants) plus only the requesting user's own messages from every
-    other chat, merged in chronological order. This eliminates amnesia when a user
-    switches between private and group contexts.
+    chat (all participants) plus all messages from the user's private chat (both user
+    and assistant turns), merged in chronological order. This eliminates amnesia when
+    a user switches between private and group contexts.
+
+    In Telegram, a user's private chat_id always equals their user_id. Three arms cover
+    all cross-context cases:
+
+      Arm 1 — chat_id = current_chat_id:
+          All messages (user and bot) in the current chat. Always included.
+
+      Arm 2 — chat_id = user_id AND chat_id != current_chat_id:
+          All messages (both sides) from the user's private chat, included when the
+          current chat is a group. Pulling both sides restores proper role alternation
+          that was broken when only the user's own rows were fetched.
+
+      Arm 3 — user_id = user_id AND chat_id != current_chat_id AND chat_id != user_id:
+          The requesting user's own messages from any other group chats (not the private
+          chat, not the current chat). Covers group-to-private and group-to-group context.
+          Only the user's own rows are fetched from other groups (not bot replies).
 
-    Query logic: loads ALL rows from current_chat_id (all participants), plus only the requesting
-    user's own rows from every other chat, merged chronologically. The exclude_private filter
-    is applied globally when the current chat is a group.
+    The exclude_private filter is applied globally when the current chat is a group.
 
     Args:
         user_id: Telegram user ID of the requesting user.
@@ -169,9 +187,76 @@ async def load_full_user_context(
     """
     query = (
         "SELECT role, content FROM messages "
-        "WHERE (chat_id = ? OR (user_id = ? AND chat_id != ?))"
+        "WHERE (chat_id = ? "
+        "OR (chat_id = ? AND chat_id != ?) "
+        "OR (user_id = ? AND chat_id != ? AND chat_id != ?))"
+    )
+    params: tuple = (current_chat_id, user_id, current_chat_id, user_id, current_chat_id, user_id)
+    if exclude_private:
+        query += " AND is_private = 0"
+    query += " ORDER BY created_at ASC, id ASC"
+    async with aiosqlite.connect(get_db_path()) as db:
+        async with db.execute(query, params) as cursor:
+            rows = await cursor.fetchall()
+    return [{"role": row[0], "content": row[1]} for row in rows]
+
+async def get_max_cross_chat_message_id(
+    user_id: int,
+    current_chat_id: int,
+    exclude_private: bool = False,
+) -> int:
+    """
+    Return the maximum message id from the cross-chat context for this user.
+
+    Uses the same three-arm query as load_full_user_context, returning only the
+    MAX(id) scalar. Used to establish a cursor for mid-session refresh detection.
+
+    Returns:
+        Maximum message id as an integer, or 0 if no rows exist.
+    """
+    query = (
+        "SELECT MAX(id) FROM messages "
+        "WHERE (chat_id = ? "
+        "OR (chat_id = ? AND chat_id != ?) "
+        "OR (user_id = ? AND chat_id != ? AND chat_id != ?))"
+    )
+    params: tuple = (current_chat_id, user_id, current_chat_id, user_id, current_chat_id, user_id)
+    if exclude_private:
+        query += " AND is_private = 0"
+    async with aiosqlite.connect(get_db_path()) as db:
+        async with db.execute(query, params) as cursor:
+            row = await cursor.fetchone()
+    return row[0] if row and row[0] is not None else 0
+
+async def load_new_cross_chat_context(
+    user_id: int,
+    current_chat_id: int,
+    since_id: int,
+    exclude_private: bool = False,
+) -> list[dict]:
+    """
+    Load cross-chat context rows with id strictly greater than since_id.
+
+    Uses the same three-arm query as load_full_user_context but restricted to rows
+    newer than the given cursor. Returns only the delta since the last context load.
+
+    Args:
+        user_id: Telegram user ID of the requesting user.
+        current_chat_id: The chat_id of the current conversation.
+        since_id: Exclusive lower bound — only rows with id > since_id are returned.
+        exclude_private: If True, rows with is_private=1 are excluded.
+
+    Returns:
+        List of {"role": ..., "content": ...} dicts in ascending chronological order.
+    """
+    query = (
+        "SELECT role, content FROM messages "
+        "WHERE (chat_id = ? "
+        "OR (chat_id = ? AND chat_id != ?) "
+        "OR (user_id = ? AND chat_id != ? AND chat_id != ?)) "
+        "AND id > ?"
     )
-    params: tuple = (current_chat_id, user_id, current_chat_id)
+    params: tuple = (current_chat_id, user_id, current_chat_id, user_id, current_chat_id, user_id, since_id)
     if exclude_private:
         query += " AND is_private = 0"
     query += " ORDER BY created_at ASC, id ASC"
@@ -180,6 +265,80 @@ async def load_full_user_context(
             rows = await cursor.fetchall()
     return [{"role": row[0], "content": row[1]} for row in rows]
 
+async def get_shared_group_chat_ids(user_id: int, bot_id: int) -> list[int]:
+    """
+    Return chat_ids of groups where both the user and the bot have sent messages.
+
+    Eligibility conditions:
+    1. The requesting user has sent at least one message in the group.
+    2. The bot has also sent at least one message in the group (bot is active there).
+
+    Condition 2 prevents surfacing history from groups the bot was removed from
+    or never properly joined.
+
+    Args:
+        user_id: Telegram user ID of the requesting user.
+        bot_id: Telegram user ID of the bot account.
+
+    Returns:
+        List of negative chat_ids (Telegram group IDs).
+    """
+    query = """
+        SELECT DISTINCT chat_id
+        FROM messages
+        WHERE user_id = ? AND chat_id < 0
+        AND EXISTS (
+            SELECT 1 FROM messages m2
+            WHERE m2.chat_id = messages.chat_id AND m2.user_id = ?
+        )
+    """
+    async with aiosqlite.connect(get_db_path()) as db:
+        async with db.execute(query, (user_id, bot_id)) as cursor:
+            rows = await cursor.fetchall()
+    return [row[0] for row in rows]
+
+async def load_shared_group_context(
+    group_chat_ids: list[int],
+    exclude_private: bool = True,
+) -> list[dict]:
+    """
+    Load all messages from multiple group chats, merged in chronological order.
+
+    Used to fill remaining token budget in a private chat with context from
+    shared groups (full group context in private chat).
+
+    Automatically batches the query if group_chat_ids exceeds _SQLITE_MAX_PARAMS
+    (999, SQLite's bound parameter limit) to handle users who share many groups
+    with the bot. Results are merged and sorted chronologically across batches.
+
+    Args:
+        group_chat_ids: List of group chat_ids (negative integers) to include.
+        exclude_private: If True, rows with is_private=1 are excluded (default True).
+
+    Returns:
+        List of {"role": ..., "content": ...} dicts in ascending chronological order.
+        Empty list if group_chat_ids is empty.
+    """
+    if not group_chat_ids:
+        return []
+    # SQLite has a max of 999 bound parameters per statement. Batch group_chat_ids to
+    # avoid exceeding this limit for users who share many groups with the bot.
+    all_rows: list[tuple] = []
+    async with aiosqlite.connect(get_db_path()) as db:
+        for i in range(0, len(group_chat_ids), _SQLITE_MAX_PARAMS):
+            chunk = group_chat_ids[i:i + _SQLITE_MAX_PARAMS]
+            placeholders = ','.join('?' * len(chunk))
+            query = (
+                f"SELECT role, content, created_at, id FROM messages "
+                f"WHERE chat_id IN ({placeholders})"
+            )
+            if exclude_private:
+                query += " AND is_private = 0"
+            async with db.execute(query, tuple(chunk)) as cursor:
+                all_rows.extend(await cursor.fetchall())
+    all_rows.sort(key=lambda r: (r[2], r[3]))
+    return [{"role": row[0], "content": row[1]} for row in all_rows]
+
 async def delete_messages_for_user(user_id: int) -> None:
     """
     Delete all message rows for a given user_id across all chats.

{tellmgrambot-3.0.4 → tellmgrambot-3.2.0}/TeLLMgramBot.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: TeLLMgramBot
-Version: 3.0.4
+Version: 3.2.0
 Summary: LLM-powered Telegram bot (OpenAI + Anthropic)
 Home-page: https://github.com/Digital-Heresy/TeLLMgramBot
 Author: Digital Heresy
@@ -43,10 +43,10 @@ The basic goal of this project is to create a bridge between a Telegram Bot and
   * This uses a separate model (configurable via `url_model`) to support more URL content with its higher token limit.
 * Tokens are used to measure the length of all conversation messages between the Telegram bot assistant and the user. This is useful to:
   * Ensure the length does not go over the model limit. If it does, prune oldest messages to fit within the limit.
-  * Remember past conversations when restarting: loads the user's full history across all chats (private and groups) plus all other participants' messages in the current chat, up to 50% of the token budget. This eliminates amnesia when users switch between contexts.
+  * Remember past conversations when restarting: loads the user's full history across all chats (private and groups) plus all other participants' messages in the current chat, up to 50% of the token budget. In private chats, shared group context (messages from groups where both user and bot are active) fills the remaining budget, enabling the bot to reference group conversations from a private context. This eliminates amnesia when users switch between contexts.
 * Users can manage privacy via two commands:
-  * `/forget` — In private chats, clears the full conversation (including bot replies). In group chats, removes only your messages; other participants' messages remain.
-  * `/private` — Toggles per-user private mode (private chats only). When ON, messages are excluded from group conversation contexts, enabling selective privacy even in shared groups.
+  * `/forget` — In private chats, clears the full conversation (including bot replies) and resets all of your active sessions (any sessions where your context was merged). In group chats, removes your messages across all chat types; other participants' messages and sessions remain.
+  * `/private` — Toggles per-user private mode (private chats only). When ON, messages are excluded from group conversation contexts, enabling selective privacy even in shared groups. If you have private mode ON and message a group, the bot will remind you once per session.
 
 ## Why Telegram?
 Using Telegram as the interface not only solves "exposing" the interface, but gives you boatloads of interactivity over a standard Command Line interface, or trying to create a website with input boxes and submit buttons to try to handle everything:
@@ -87,7 +87,7 @@ When initializing TeLLMgramBot, the following directories get created:
    * User will also get notified to contact the owner.
 * `data`
   * Contains `conversations.db` — a SQLite database storing all conversations between the bot and users across all chats.
-  * When a user messages in any chat, their full history is available for context: private messages appear in group contexts, group messages appear in private contexts. This creates seamless cross-context awareness.
+  * When a user messages in any chat, their full history is available for context: private messages appear in group contexts, group messages appear in private contexts. This creates seamless cross-context awareness. The bot dynamically refreshes each user's context during a session to pick up messages sent between chats. In private chats, the bot also loads shared group context (messages from groups where both user and bot are active) to provide group awareness.
   * Users can manage their context via `/forget` (private chat: clears full conversation; group chat: removes only your messages) or `/private` (toggles per-user privacy for group contexts).
 
 ### Environment Variables

{tellmgrambot-3.0.4 → tellmgrambot-3.2.0}/setup.py

@@ -5,7 +5,7 @@ with open("README.md", "r") as fh:
 
 setup(
     name='TeLLMgramBot',
-    version='3.0.4',
+    version='3.2.0',
     packages=find_packages(),
     license='MIT',
     author='Digital Heresy',

tellmgrambot-3.0.4/TeLLMgramBot/conversation.py

@@ -1,196 +0,0 @@
-"""
-Manages conversation history, token counting, and SQLite-backed persistence for any LLM provider.
-
-The Conversation class is keyed by chat_id (Telegram chat ID) and maintains an in-memory message
-history, delegating persistence to the database module. Supports async token counting via provider-
-specific APIs and automatic pruning when token limits are approached.
-"""
-from .database import insert_message, load_full_user_context, delete_messages_for_user, delete_messages_for_chat
-from .models import TokenLimits
-from .utils import generate_error_path, log_error
-
-class Conversation:
-    """
-    Manages a chat conversation including message history, token counting, and SQLite persistence.
-
-    Keyed by chat_id (Telegram chat ID): negative for groups, positive (== user_id) for private chats.
-    Works with any LLM provider (OpenAI, Anthropic). When the token limit is approached, oldest
-    messages are pruned from memory; the database retains the full history.
-    """
-
-    def __init__(self, chat_id: int, chat_type: str, system_content: str, system_model: str = "gpt-4o-mini"):
-        """
-        Initialize a conversation keyed by chat_id.
-
-        Args:
-            chat_id: Telegram chat ID (negative for groups, positive/==user_id for private chats).
-            chat_type: Chat type ('private', 'group', 'supergroup', 'channel').
-            system_content: System prompt (bot persona).
-            system_model: LLM model to use for token counting (default: "gpt-4o-mini").
-        """
-        self.error_log = generate_error_path()
-
-        self.chat_id = chat_id
-        self.chat_type = chat_type
-        self.chat_print = f"{self.chat_type} Chat {self.chat_id}"
-
-        self.system_content = system_content
-        self.system_model = TokenLimits(system_model)
-        self.messages = [{"role": "system", "content": system_content}]
-
-        self._has_past_interaction = False
-
-    def set_system_content(self, new_content: str):
-        """
-        Replace the system content (bot's personality prompt) in the conversation.
-
-        Updates both self.system_content and the first message in self.messages.
-
-        Args:
-            new_content: The new system prompt text.
-        """
-        self.system_content = new_content
-        self.messages[0] = {"role": "system", "content": new_content}
-
-    async def add_user_message(self, content: str, user_id: int, username: str | None, is_private: bool = False):
-        """
-        Add a user message to conversation memory and persist it to the database.
-
-        Args:
-            content: The message text.
-            user_id: Telegram user ID of the sender.
-            username: Telegram username (display only, may be None).
-            is_private: If True, excludes this message from group context loading (used for /forget or private commands).
-        """
-        self.messages.append({"role": "user", "content": content})
-        await insert_message(self.chat_id, user_id, username, "user", content, is_private)
-
-    async def add_assistant_message(self, content: str, bot_user_id: int, bot_username: str | None):
-        """
-        Add an assistant message to conversation memory and persist it to the database.
-
-        Args:
-            content: The message text (LLM response).
-            bot_user_id: Telegram user ID of the bot account.
-            bot_username: Telegram username of the bot (display only, may be None).
-        """
-        self.messages.append({"role": "assistant", "content": content})
-        await insert_message(self.chat_id, bot_user_id, bot_username, "assistant", content, False)
-
-    async def get_message_token_count(self) -> int:
-        """
-        Get the current token count of all messages in this conversation.
-
-        Delegates to the configured model's provider to count tokens accurately.
-        This is an async operation because it may contact the provider API (e.g., for Anthropic).
-
-        Returns:
-            The total token count for all messages (system, user, assistant).
-
-        Raises:
-            ProviderAuthError: If API credentials are missing or invalid.
-            ProviderConnectionError: If the provider API cannot be reached.
-        """
-        return await self.system_model.num_tokens_from_messages(self.messages)
-
-    async def prune_conversation(self, token_limit: int):
-        """
-        Remove oldest user-assistant messages until token count is below threshold.
-
-        Recursively removes messages starting from index 1 (preserving the system message at index 0)
-        until the conversation fits within the token limit. The database is not modified — pruning
-        only affects the in-memory context window for the current session.
-
-        Args:
-            token_limit: The maximum token count threshold. Messages are removed until
-                         the conversation is strictly below this limit.
-        """
-        try:
-            self.messages.pop(1)
-            if await self.get_message_token_count() > token_limit:
-                await self.prune_conversation(token_limit)
-        except Exception as e:
-            log_error(e, f"{type(e).__name__} pruning under {token_limit} tokens for {self.chat_print}", self.error_log)
-
-    async def get_past_interaction(self, token_limit: int, user_id: int) -> bool:
-        """
-        Load past conversation messages from the database up to a token limit.
-
-        Implements bidirectional cross-pollination: loads all messages from the current chat
-        (all participants) plus only the requesting user's own messages from other chats,
-        merged chronologically. This eliminates amnesia when a user switches between private
-        and group contexts.
-
-        For group and supergroup chats, messages marked as private (is_private=1) are excluded
-        so private-mode messages don't surface in shared contexts.
-
-        Idempotent: calling again returns True without reloading if messages were already loaded.
-
-        Args:
-            token_limit: The maximum token count to load into the in-memory context.
-            user_id: Telegram user ID of the requesting user; used to fetch bidirectional context.
-
-        Returns:
-            True if past interactions were found (either on this call or previously),
-            False if no past interactions exist.
-
-        Side Effects:
-            - Modifies self.messages in-place by inserting historical messages after the system message
-              (attempting to fit messages starting from most recent, stopping when limit is reached).
-            - Sets self._has_past_interaction to True if any messages are loaded.
-            - Prints to stdout showing token count and whether the token limit was reached.
-        """
-        if self._has_past_interaction:
-            print(f"Context already loaded for User {user_id} in {self.chat_print}")
-            return True
-
-        exclude_private = self.chat_type in ('group', 'supergroup')
-        rows = await load_full_user_context(user_id, self.chat_id, exclude_private=exclude_private)
-
-        token_limit_reached = False
-        token_count = 0
-
-        for row in reversed(rows):
-            self.messages.insert(1, row)
-            token_count = await self.get_message_token_count()
-            if token_count > token_limit:
-                self.messages.pop(1)
-                token_count = await self.get_message_token_count()
-                token_limit_reached = True
-                break
-
-        self._has_past_interaction = len(self.messages) > 1
-
-        if self._has_past_interaction:
-            if token_limit_reached:
-                print(
-                    f"Partial context loaded for User {user_id} in {self.chat_print} — "
-                    f"full history exceeded token limit ({token_limit}), storing {token_count} token(s)"
-                )
-            else:
-                print(
-                    f"Full context loaded for User {user_id} in {self.chat_print} — "
-                    f"storing {token_count} token(s)"
-                )
-        else:
-            print(f"No prior context found for User {user_id} in {self.chat_print}")
-
-        return self._has_past_interaction
-
-    async def clear_interaction(self, user_id: int):
-        """
-        Clear this user's messages from memory and from the database.
-
-        In private chats, all messages in the chat are deleted (including bot replies),
-        fully clearing the conversation. In group chats, only this user's rows are removed;
-        other members' messages remain.
-
-        Args:
-            user_id: Telegram user ID of the user requesting the forget.
-        """
-        self.messages = [self.messages[0]]
-        self._has_past_interaction = False
-        if self.chat_type == 'private':
-            await delete_messages_for_chat(self.chat_id)
-        else:
-            await delete_messages_for_user(user_id)