@segmentstream/agent 0.6.5 → 0.6.7

package/README.md

@@ -82,7 +82,7 @@ Both entry points export the same handle: `@segmentstream/agent` and
 | Field                          | Default               | Meaning                                                                                                                  |
 | ------------------------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------ |
 | `channels.telegram.connection` | none                  | The Connect key whose credential is the bot token. Without `channels.telegram`, no Telegram.                             |
-| `channels.telegram.policy`     | `"smart"`             | Group messages that do not address the bot: `smart` asks the decision model, `mention` observes                          |
+| `channels.telegram.policy`     | `"smart"`             | Unaddressed group text: `smart` asks the decision model, `mention` observes; attachments always trigger a turn           |
 | `models.stt`                   | `"spacexai/grok-stt"` | The AI Gateway transcription model for voice messages; `false` leaves them untranscribed                                 |
 | `models.imageDescription`      | automatic             | A separate vision model for ingestion descriptions; omitted prefers native images, `false` disables description fallback |
 | `models.decision`              | `"typesafe-ai/jev"`   | The AI Gateway decision model of the inbound router                                                                      |
@@ -140,33 +140,38 @@ share a group chat's session never share a client.
 - **Who the agent is.** A paired sender's turn runs with the principal id `telegram:<user>`
   (`telegram:<chat>:<user>` in groups) and the attributes `memberId`, `memberName`, `roles`,
   `chatId`, `chatType` and `username`.
-- **Routing.** A private chat, a mention of the bot, a reply to it and a command are always
-  answered. Under `smart`, any other group message is put to one `decide` question on the decision
+- **Routing.** Any file attachment from a paired member starts a turn under both `smart` and
+  `mention`, without a routing model call, caption or mention. This includes PDFs, photos,
+  spreadsheets and other media, even when replying to or tagging another person: the agent can
+  file or reconcile the upload. The decision archive records `attachment`. Eve 0.76.3 does not
+  batch inbound albums; each Telegram message starts one turn, regardless of its file count.
+  A private chat, a mention of the bot, a reply to it and a command are always answered.
+  Under `smart`, any other group message is put to one `decide` question on the decision
   model, with the agent's description and the chat's recent observed messages. A message that is not
   answered joins a ring of the chat's last 30 observed messages. The next answered message carries
   the ring into its turn as context. A failed decision observes.
-  Replies to a known human or another bot, and explicit mentions of another recipient, are
+  Text-only replies to a known human or another bot, and explicit mentions of another recipient, are
   observed before smart inference unless this bot is addressed. This also takes precedence over
   a pending-question follow-up. Both mention entities and user-ID `text_mention` entities count;
   quoted/code mentions do not. Smart routing receives the original body, reply author/text,
   tagged users and whether this bot was tagged. Decision events retain a short routing audit note
   alongside the reason code; this is an observable summary, not model chain-of-thought.
 - **Voice.** Voice messages, audio files and video notes are downloaded through the connection and
-  transcribed with `models.stt` through AI Gateway before routing, including observations under
-  `mention`. The transcript becomes the message, with a note such as `[voice message, 12 s]` in
-  context. `stt: false` skips download and transcription. Transcription accepts at most 20 MiB
+  transcribed with `models.stt` through AI Gateway before the turn.
+  The transcript becomes the message, with a note such as `[voice message, 12 s]` in
+  context. `stt: false` skips transcription. Transcription accepts at most 20 MiB
   and ten minutes of declared duration, with a 60-second inference timeout. A skipped or failed
-  transcription reaches the agent as a note. Originals are not stored as ontology attachments.
+  transcription reaches the agent as a note. Originals are stored as ontology attachments.
   Eve 0.76.3 only hydrates inbound images and PDFs; it does **not** deliver native audio bytes to
   the chat model. Even an audio-capable chat model therefore uses the configured STT fallback,
   with an explicit host limitation note. Native audio requires a compatible Eve host change.
-- **Attachments.** Photos and documents up to 20 MB stream through the connection proxy (its
+- **Attachments.** Photos, documents and other Telegram files up to 20 MB stream through the connection proxy (its
   framed responses are not bound by the 2 MiB buffered limit) and are stored, before the turn,
   as the member's temporary attachments in the ontology's file storage
   (`POST /api/ontology/agent/attachments`), for this chat, for 24 hours.
-  Observed group uploads are stored too, after participant registration, without starting a turn
-  or sending a typing indication. The next reply receives their references; attachment tools can
-  also list the stored files after a cold start. The turn's file parts
+  Uploads start a turn after participant registration; attachment tools can also list the stored
+  files after a cold start. Eve supplies native file parts for photos and documents; other media
+  reaches the turn through stored attachment IDs in context. The turn's file parts
   point at the stored copies, which eve reads through the channel's `fetchFile` as the turn's
   session, which then owns them, and an `<attachments>` block names each attachment's id. A file
   that is too large or could not be stored becomes a note. The `segmentstream__attachments_*`

package/dist/extension/lib/chat-post-confirmation.mjs

@@ -6,8 +6,10 @@ __eveDirname(__filename);
 __eveCreateRequire(import.meta.url);
 import { object } from "./confirmation-description.mjs";
 import { createHash } from "node:crypto";
+import { TelegramFormatConverter, escapeMarkdownV2 } from "@chat-adapter/telegram";
 const CHAT_POST_PROTOCOL = "segmentstream/chat-post";
 const CHAT_POST_ENVELOPE = "_segmentstream";
+const converter = new TelegramFormatConverter();
 function isChatPost(action) {
 	return action.split("__").at(-1) === "chat_post_message";
 }
@@ -16,6 +18,7 @@ function supportsChatPost(tool) {
 }
 function chatPostDigest(preview) {
 	return createHash("sha256").update(JSON.stringify([
+		"v2",
 		preview.chatId,
 		preview.provider,
 		preview.externalId,
@@ -43,6 +46,7 @@ function describeChatPost(input) {
 		target.external_id,
 		caller.attribution
 	].every((value) => typeof value === "string" && value.length > 0) || target.provider !== "telegram" || target.id !== values.chat && target.title !== values.chat) throw new Error("An unambiguous destination and a complete exact chat preview are required.");
+	const footer = escapeMarkdownV2(caller.attribution);
 	return {
 		action: "Post message",
 		fields: [],
@@ -53,7 +57,7 @@ function describeChatPost(input) {
 			title: target.title,
 			attribution: caller.attribution,
 			text: values.text,
-			finalText: `${caller.attribution}: ${values.text}`,
+			finalText: `${converter.fromMarkdown(values.text)}\n\n_${footer}_`,
 			replyTo: typeof values.reply_to === "string" ? values.reply_to : null
 		}
 	};

package/dist/extension/lib/telegram/chat-post-card.d.ts

@@ -12,4 +12,6 @@ export declare function chatPostCopy(locale: string): {
     expired: string;
 };
 export declare function chatPostCard(preview: ChatPostPreview, locale: string): ConfirmationText;
+/** Status/receipt strings are data outside the already rendered preview. */
+export declare function chatPostCardStatus(text: string, status: string): string;
 export declare function chatPostReceipt(output: unknown, locale: string): string | undefined;

package/dist/extension/lib/telegram/chat-post-card.mjs

@@ -4,6 +4,7 @@ import { createRequire as __eveCreateRequire } from "node:module";
 const __filename = __eveFileURLToPath(import.meta.url);
 __eveDirname(__filename);
 __eveCreateRequire(import.meta.url);
+import { escapeMarkdownV2 } from "@chat-adapter/telegram";
 import { AsyncLocalStorage } from "node:async_hooks";
 const MAX_PREVIEW_LENGTH = 3584;
 const exactText = new AsyncLocalStorage();
@@ -37,9 +38,8 @@ function chatPostCard(preview, locale) {
 		russian ? "Отправить сообщение?" : "Send message?",
 		`${russian ? "Чат" : "Chat"}: ${preview.title} (${preview.provider})`,
 		...preview.replyTo === null ? [] : [`${russian ? "В ответ на" : "Reply to"}: ${preview.replyTo}`],
-		"",
-		preview.finalText
-	].join("\n");
+		""
+	].map(escapeMarkdownV2).join("\n") + `\n${preview.finalText}`;
 	if (text.length > MAX_PREVIEW_LENGTH) throw new Error("The full message and receipt cannot fit in one Telegram confirmation. Make a shorter draft.");
 	return {
 		text,
@@ -51,6 +51,9 @@ function chatPostCard(preview, locale) {
 		failed: russian ? "Ошибка отправки сообщения." : "Chat posting failed."
 	};
 }
+function chatPostCardStatus(text, status) {
+	return `${text}\n\n${escapeMarkdownV2(status)}`;
+}
 function chatPostReceipt(output, locale) {
 	const value = typeof output === "object" && output !== null ? output : {};
 	if (value.status !== "sent" && value.status !== "uncertain" || typeof value.history_recorded !== "boolean" || typeof value.duplicate !== "boolean" || value.message_id !== null && typeof value.message_id !== "string" || value.url !== null && typeof value.url !== "string") return;
@@ -62,4 +65,4 @@ function chatPostReceipt(output, locale) {
 		...value.history_recorded ? [] : [copy.history]
 	].join("\n");
 }
-export { chatPostCard, chatPostCopy, chatPostReceipt, isExactChatPostCard, withChatPostCard };
+export { chatPostCard, chatPostCardStatus, chatPostCopy, chatPostReceipt, isExactChatPostCard, withChatPostCard };

package/dist/extension/lib/telegram/confirmations.mjs

@@ -6,7 +6,7 @@ __eveDirname(__filename);
 __eveCreateRequire(import.meta.url);
 import { conversationLanguage, memberIdOf, telegramPrincipal } from "./principal.mjs";
 import { chatPostOperationId, isChatPost } from "../chat-post-confirmation.mjs";
-import { chatPostCard, chatPostCopy, chatPostReceipt, withChatPostCard } from "./chat-post-card.mjs";
+import { chatPostCard, chatPostCardStatus, chatPostCopy, chatPostReceipt, withChatPostCard } from "./chat-post-card.mjs";
 import { confirmations } from "../confirmations.mjs";
 import { conversationEvent } from "../conversations.mjs";
 import { currentInboundRequest } from "./inbound-request.mjs";
@@ -302,7 +302,7 @@ function telegramConfirmationEvents(render = async (description, locale) => fall
 				const copy = message.copy ?? confirmationCopy(message.locale ?? confirmationLocale(ctx.session.auth.current, currentLocale()));
 				const reason = confirmations.get()[callId]?.cancellationReason;
 				const status = resolution.outcome === "approved" ? copy.executing : reason === "expired" ? chatPostCopy(message.locale ?? "en").expired : copy.cancelled;
-				const text = message.exactPost ? `${message.text}\n\n${status}` : status;
+				const text = message.exactPost ? chatPostCardStatus(message.text, status) : status;
 				const edit = () => channel.telegram.editMessageText({
 					messageId: message.messageId,
 					replyMarkup: { inline_keyboard: [] },
@@ -390,7 +390,7 @@ function telegramConfirmationEvents(render = async (description, locale) => fall
 				const receipt = !failed && data.result.kind === "tool-result" ? chatPostReceipt(data.result.output, locale) : void 0;
 				const errorId = failed ? reportTelegramError("confirmed_chat_post_failed", data, { callId: data.result.callId }) : void 0;
 				const status = receipt ?? (failed ? `${message.copy?.failed ?? "Chat posting failed."} ${errorId}\n${chatPostCopy(locale).uncertain}` : chatPostCopy(locale).uncertain);
-				const text = `${message.text}\n\n${status}`;
+				const text = chatPostCardStatus(message.text, status);
 				await withChatPostCard(text, () => channel.telegram.editMessageText({
 					messageId: message.messageId,
 					replyMarkup: { inline_keyboard: [] },

package/dist/extension/lib/telegram/customer-text.mjs

@@ -140,17 +140,20 @@ function safeTelegramFetch(fetchImpl, scope = "telegram") {
 		const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url;
 		if (!methods.has(url.split("/").at(-1) ?? "") || typeof init?.body !== "string") return fetchImpl(input, init);
 		const body = JSON.parse(init.body);
-		for (const field of ["text", "caption"]) if (typeof body[field] === "string" && !isExactChatPostCard(body[field])) {
-			const entityField = field === "text" ? "entities" : "caption_entities";
-			const existing = body[entityField];
-			const message = Array.isArray(existing) ? {
-				text: body[field],
-				entities: existing
-			} : customerMessage(body[field]);
-			const guarded = customerText(message.text);
-			body[field] = message.entities.some((entity) => typeof entity.url === "string" && isTechnicalText(entity.url)) ? failureText(reportTelegramError("unsafe_entity_link", new Error("Credential-bearing link refused.")), "response") : guarded;
-			if (!url.endsWith("/answerCallbackQuery")) body[entityField] = body[field] === message.text ? message.entities : [];
-			delete body.parse_mode;
+		for (const field of ["text", "caption"]) {
+			if (typeof body[field] === "string" && isExactChatPostCard(body[field])) body.parse_mode = "MarkdownV2";
+			if (typeof body[field] === "string" && !isExactChatPostCard(body[field])) {
+				const entityField = field === "text" ? "entities" : "caption_entities";
+				const existing = body[entityField];
+				const message = Array.isArray(existing) ? {
+					text: body[field],
+					entities: existing
+				} : customerMessage(body[field]);
+				const guarded = customerText(message.text);
+				body[field] = message.entities.some((entity) => typeof entity.url === "string" && isTechnicalText(entity.url)) ? failureText(reportTelegramError("unsafe_entity_link", new Error("Credential-bearing link refused.")), "response") : guarded;
+				if (!url.endsWith("/answerCallbackQuery")) body[entityField] = body[field] === message.text ? message.entities : [];
+				delete body.parse_mode;
+			}
 		}
 		if (Array.isArray(body.media)) body.media = body.media.map((item) => ({
 			...item,

package/dist/extension/lib/telegram/files.d.ts

@@ -1,4 +1,4 @@
-import type { TelegramMessage } from "eve/channels/telegram";
+import type { TelegramAttachment, TelegramMessage } from "eve/channels/telegram";
 import { type AgentAttachments } from "../attachments.js";
 import type { StagedFile } from "./inbound-request.js";
 import { type ImageDescriber } from "./images.js";
@@ -13,6 +13,8 @@ import type { MediaPlan } from "./media-policy.js";
  */
 /** The chat an attachment belongs to: `telegram:<chat id>`. */
 export declare function telegramConversation(chatId: string): string;
+/** Eve normalizes photos/documents only. Keep other Telegram files available to attachment tools. */
+export declare function inboundAttachments(message: TelegramMessage): ReadonlyArray<TelegramAttachment>;
 /** Stores the message's files one after another, so that a large album holds one file at a time. */
 export declare function stageFiles(input: {
     readonly message: TelegramMessage;

package/dist/extension/lib/telegram/files.mjs

@@ -11,6 +11,31 @@ import { describableImage, describeReceivedImage } from "./images.mjs";
 function telegramConversation(chatId) {
 	return `telegram:${chatId}`;
 }
+function inboundAttachments(message) {
+	const files = [...message.attachments];
+	for (const kind of [
+		"voice",
+		"audio",
+		"video",
+		"video_note",
+		"sticker",
+		"animation"
+	]) {
+		const value = message.raw[kind];
+		if (value === null || typeof value !== "object" || Array.isArray(value)) continue;
+		const file = value;
+		if (typeof file.file_id !== "string" || file.file_id === "" || files.some((existing) => existing.fileId === file.file_id)) continue;
+		files.push({
+			kind: "document",
+			fileId: file.file_id,
+			fileName: typeof file.file_name === "string" ? file.file_name : kind,
+			...typeof file.file_unique_id === "string" ? { fileUniqueId: file.file_unique_id } : {},
+			...typeof file.mime_type === "string" ? { mediaType: file.mime_type } : {},
+			...typeof file.file_size === "number" ? { size: file.file_size } : {}
+		});
+	}
+	return files;
+}
 function described(file, index) {
 	return {
 		filename: file.fileName ?? (file.kind === "photo" ? `photo-${index}.jpg` : `file-${index}`),
@@ -106,4 +131,4 @@ function attachmentsContext(input) {
 	lines.push("</attachments>");
 	return [lines.join("\n")];
 }
-export { attachmentsContext, stageFiles, telegramConversation };
+export { attachmentsContext, inboundAttachments, stageFiles, telegramConversation };

package/dist/extension/lib/telegram/inbound.d.ts

@@ -21,8 +21,9 @@ import { type Transcriber } from "./voice.js";
  *    and so is the agent's own, which unlinks every member. Neither is the notice, in the old chat
  *    or the new one, that a group became a supergroup: the migration is reported.
  * 2. Voice becomes text, so the router and the agent read what was said.
- * 3. The router answers or observes. Photos and documents are stored after the registry knows the
- *    sender, including on observed messages. Observing starts no session or typing indication;
+ * 3. The router answers or observes. Files always start a turn after the identity gates, even
+ *    without a caption or when addressed to another person. They are stored after the registry knows the
+ *    sender. Observing starts no session or typing indication;
  *    the ring carries the stored files' references (or storage failures) into the next reply.
  * 4. The session policy runs on the session an answered message continues: an idle one is cleared, the
  *    first message after the day's compaction time compacts it.

package/dist/extension/lib/telegram/inbound.mjs

@@ -14,7 +14,7 @@ import { formatAddress, messageAddress, scopedAddress } from "./sessions.mjs";
 import { questionAddress, questionFromHistory } from "./confirmations.mjs";
 import { MAXIMUM_NATIVE_IMAGE_BYTES } from "./images.mjs";
 import { chatContext } from "./chats.mjs";
-import { attachmentsContext, stageFiles } from "./files.mjs";
+import { attachmentsContext, inboundAttachments, stageFiles } from "./files.mjs";
 import { messageAudience } from "./audience.mjs";
 import { addressingReason, routeMessage } from "./router.mjs";
 import { routingContext } from "./routing-context.mjs";
@@ -175,7 +175,8 @@ async function handleInboundMessage(input) {
 	if (message.from === void 0 || message.from.isBot || message.chat.type === "channel") return null;
 	const sent = {
 		...message,
-		from: message.from
+		from: message.from,
+		attachments: inboundAttachments(message)
 	};
 	const bot = await dependencies.bot(context.telegram);
 	const audience = messageAudience(sent, bot);
@@ -227,6 +228,7 @@ async function handleInboundMessage(input) {
 	const caption = sent.text || sent.caption;
 	const notes = [];
 	let text = caption;
+	let voiceBytes;
 	const chat = observedChatKey({
 		chatId: sent.chat.id,
 		...sent.messageThreadId === void 0 ? {} : { messageThreadId: sent.messageThreadId }
@@ -236,7 +238,10 @@ async function handleInboundMessage(input) {
 		const spoken = await voiceText({
 			voice,
 			model: dependencies.sttModel,
-			download: dependencies.download,
+			download: async (fileId) => {
+				voiceBytes = await dependencies.download(fileId);
+				return voiceBytes;
+			},
 			transcribe: dependencies.transcribe
 		});
 		text = [caption, spoken.text].filter((part) => part.trim() !== "").join("\n\n");
@@ -252,12 +257,16 @@ async function handleInboundMessage(input) {
 		recordGateDecision("empty_message");
 		return null;
 	}
+	if (text.trim() === "" && sent.attachments.length > 0) {
+		text = "[attachment]";
+		if (input.request !== void 0) input.request.text = text;
+	}
 	if (sent.attachments.length > 0) await report;
 	const files = sent.attachments.length === 0 || dependencies.attachments === void 0 || input.request === void 0 ? new Map() : await stageFiles({
 		message: sent,
 		memberId: lookup.member.id,
 		attachments: dependencies.attachments,
-		download: dependencies.download,
+		download: (fileId) => voiceBytes !== void 0 && voice?.fileId === fileId ? Promise.resolve(voiceBytes) : dependencies.download(fileId),
 		...mediaPlan === void 0 ? {} : { imagePlan: mediaPlan },
 		...dependencies.describeImage === void 0 ? {} : { describeImage: dependencies.describeImage }
 	});
@@ -269,6 +278,7 @@ async function handleInboundMessage(input) {
 		chatId: sent.chat.id,
 		memberId: lookup.member.id,
 		policy: dependencies.policy,
+		hasAttachments: sent.attachments.length > 0,
 		addressed: addressed ?? (answering ? "pending-question" : void 0),
 		audience,
 		record: (decision) => recordRouteDecision(decision.route, addressed, dependencies.policy, decision),

package/dist/extension/lib/telegram/router.d.ts

@@ -4,8 +4,8 @@ import type { ObservedMessage, RecentMessage } from "./observed.js";
 import { type MessageAudience } from "./audience.js";
 /**
  * The inbound router (`agent-design.md`, section 4): every message from a paired member is either
- * answered or observed before eve starts a turn. A private chat, a mention, a reply to the bot and
- * a command are always answered. Any other group message is observed, or, with the `smart`
+ * answered or observed before eve starts a turn. Attachments, a private chat, a mention, a reply
+ * to the bot and a command are always answered. Any other group message is observed, or, with the `smart`
  * policy, put to one decision on the decision model.
  */
 export type Route = "observe" | "reply";
@@ -78,6 +78,8 @@ export declare function routeMessage(input: {
     readonly chatId: string;
     readonly memberId: string;
     readonly policy: TelegramPolicy;
+    /** Files always start a turn, including replies to other people and captionless media. */
+    readonly hasAttachments?: boolean;
     readonly addressed: AddressingReason | undefined;
     readonly audience?: MessageAudience;
     readonly record?: (decision: RouteDecision) => void;

package/dist/extension/lib/telegram/router.mjs

@@ -115,6 +115,7 @@ async function routeMessage(input) {
 		});
 		return route;
 	};
+	if (input.hasAttachments) return complete("reply", "attachment", "Inbound attachment starts an agent turn regardless of caption or recipient.");
 	if (input.addressed !== void 0 && input.addressed !== "pending-question") return complete("reply", input.addressed, `Bot addressed directly: ${input.addressed}.`);
 	const elsewhere = otherRecipient(input.audience);
 	if (elsewhere !== void 0) return complete("observe", "other_recipient", elsewhere);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@segmentstream/agent",
-  "version": "0.6.5",
+  "version": "0.6.7",
   "description": "Mount a SegmentStream workspace's agent on eve: the ontology as tools, acting as the member who writes, and Telegram through the workspace's connection.",
   "license": "Apache-2.0",
   "author": "SegmentStream",
@@ -60,6 +60,7 @@
     "prepack": "npm run build"
   },
   "dependencies": {
+    "@chat-adapter/telegram": "4.41.0",
     "@vercel/oidc": "^3.8.4",
     "ai": "^7.0.128",
     "csv-parse": "^7.0.3",