@segmentstream/agent 0.6.5 → 0.6.6

package/README.md

@@ -82,7 +82,7 @@ Both entry points export the same handle: `@segmentstream/agent` and
 | Field                          | Default               | Meaning                                                                                                                  |
 | ------------------------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------ |
 | `channels.telegram.connection` | none                  | The Connect key whose credential is the bot token. Without `channels.telegram`, no Telegram.                             |
-| `channels.telegram.policy`     | `"smart"`             | Group messages that do not address the bot: `smart` asks the decision model, `mention` observes                          |
+| `channels.telegram.policy`     | `"smart"`             | Unaddressed group text: `smart` asks the decision model, `mention` observes; attachments always trigger a turn           |
 | `models.stt`                   | `"spacexai/grok-stt"` | The AI Gateway transcription model for voice messages; `false` leaves them untranscribed                                 |
 | `models.imageDescription`      | automatic             | A separate vision model for ingestion descriptions; omitted prefers native images, `false` disables description fallback |
 | `models.decision`              | `"typesafe-ai/jev"`   | The AI Gateway decision model of the inbound router                                                                      |
@@ -140,33 +140,38 @@ share a group chat's session never share a client.
 - **Who the agent is.** A paired sender's turn runs with the principal id `telegram:<user>`
   (`telegram:<chat>:<user>` in groups) and the attributes `memberId`, `memberName`, `roles`,
   `chatId`, `chatType` and `username`.
-- **Routing.** A private chat, a mention of the bot, a reply to it and a command are always
-  answered. Under `smart`, any other group message is put to one `decide` question on the decision
+- **Routing.** Any file attachment from a paired member starts a turn under both `smart` and
+  `mention`, without a routing model call, caption or mention. This includes PDFs, photos,
+  spreadsheets and other media, even when replying to or tagging another person: the agent can
+  file or reconcile the upload. The decision archive records `attachment`. Eve 0.76.3 does not
+  batch inbound albums; each Telegram message starts one turn, regardless of its file count.
+  A private chat, a mention of the bot, a reply to it and a command are always answered.
+  Under `smart`, any other group message is put to one `decide` question on the decision
   model, with the agent's description and the chat's recent observed messages. A message that is not
   answered joins a ring of the chat's last 30 observed messages. The next answered message carries
   the ring into its turn as context. A failed decision observes.
-  Replies to a known human or another bot, and explicit mentions of another recipient, are
+  Text-only replies to a known human or another bot, and explicit mentions of another recipient, are
   observed before smart inference unless this bot is addressed. This also takes precedence over
   a pending-question follow-up. Both mention entities and user-ID `text_mention` entities count;
   quoted/code mentions do not. Smart routing receives the original body, reply author/text,
   tagged users and whether this bot was tagged. Decision events retain a short routing audit note
   alongside the reason code; this is an observable summary, not model chain-of-thought.
 - **Voice.** Voice messages, audio files and video notes are downloaded through the connection and
-  transcribed with `models.stt` through AI Gateway before routing, including observations under
-  `mention`. The transcript becomes the message, with a note such as `[voice message, 12 s]` in
-  context. `stt: false` skips download and transcription. Transcription accepts at most 20 MiB
+  transcribed with `models.stt` through AI Gateway before the turn.
+  The transcript becomes the message, with a note such as `[voice message, 12 s]` in
+  context. `stt: false` skips transcription. Transcription accepts at most 20 MiB
   and ten minutes of declared duration, with a 60-second inference timeout. A skipped or failed
-  transcription reaches the agent as a note. Originals are not stored as ontology attachments.
+  transcription reaches the agent as a note. Originals are stored as ontology attachments.
   Eve 0.76.3 only hydrates inbound images and PDFs; it does **not** deliver native audio bytes to
   the chat model. Even an audio-capable chat model therefore uses the configured STT fallback,
   with an explicit host limitation note. Native audio requires a compatible Eve host change.
-- **Attachments.** Photos and documents up to 20 MB stream through the connection proxy (its
+- **Attachments.** Photos, documents and other Telegram files up to 20 MB stream through the connection proxy (its
   framed responses are not bound by the 2 MiB buffered limit) and are stored, before the turn,
   as the member's temporary attachments in the ontology's file storage
   (`POST /api/ontology/agent/attachments`), for this chat, for 24 hours.
-  Observed group uploads are stored too, after participant registration, without starting a turn
-  or sending a typing indication. The next reply receives their references; attachment tools can
-  also list the stored files after a cold start. The turn's file parts
+  Uploads start a turn after participant registration; attachment tools can also list the stored
+  files after a cold start. Eve supplies native file parts for photos and documents; other media
+  reaches the turn through stored attachment IDs in context. The turn's file parts
   point at the stored copies, which eve reads through the channel's `fetchFile` as the turn's
   session, which then owns them, and an `<attachments>` block names each attachment's id. A file
   that is too large or could not be stored becomes a note. The `segmentstream__attachments_*`

package/dist/extension/lib/telegram/files.d.ts

@@ -1,4 +1,4 @@
-import type { TelegramMessage } from "eve/channels/telegram";
+import type { TelegramAttachment, TelegramMessage } from "eve/channels/telegram";
 import { type AgentAttachments } from "../attachments.js";
 import type { StagedFile } from "./inbound-request.js";
 import { type ImageDescriber } from "./images.js";
@@ -13,6 +13,8 @@ import type { MediaPlan } from "./media-policy.js";
  */
 /** The chat an attachment belongs to: `telegram:<chat id>`. */
 export declare function telegramConversation(chatId: string): string;
+/** Eve normalizes photos/documents only. Keep other Telegram files available to attachment tools. */
+export declare function inboundAttachments(message: TelegramMessage): ReadonlyArray<TelegramAttachment>;
 /** Stores the message's files one after another, so that a large album holds one file at a time. */
 export declare function stageFiles(input: {
     readonly message: TelegramMessage;

package/dist/extension/lib/telegram/files.mjs

@@ -11,6 +11,31 @@ import { describableImage, describeReceivedImage } from "./images.mjs";
 function telegramConversation(chatId) {
 	return `telegram:${chatId}`;
 }
+function inboundAttachments(message) {
+	const files = [...message.attachments];
+	for (const kind of [
+		"voice",
+		"audio",
+		"video",
+		"video_note",
+		"sticker",
+		"animation"
+	]) {
+		const value = message.raw[kind];
+		if (value === null || typeof value !== "object" || Array.isArray(value)) continue;
+		const file = value;
+		if (typeof file.file_id !== "string" || file.file_id === "" || files.some((existing) => existing.fileId === file.file_id)) continue;
+		files.push({
+			kind: "document",
+			fileId: file.file_id,
+			fileName: typeof file.file_name === "string" ? file.file_name : kind,
+			...typeof file.file_unique_id === "string" ? { fileUniqueId: file.file_unique_id } : {},
+			...typeof file.mime_type === "string" ? { mediaType: file.mime_type } : {},
+			...typeof file.file_size === "number" ? { size: file.file_size } : {}
+		});
+	}
+	return files;
+}
 function described(file, index) {
 	return {
 		filename: file.fileName ?? (file.kind === "photo" ? `photo-${index}.jpg` : `file-${index}`),
@@ -106,4 +131,4 @@ function attachmentsContext(input) {
 	lines.push("</attachments>");
 	return [lines.join("\n")];
 }
-export { attachmentsContext, stageFiles, telegramConversation };
+export { attachmentsContext, inboundAttachments, stageFiles, telegramConversation };

package/dist/extension/lib/telegram/inbound.d.ts

@@ -21,8 +21,9 @@ import { type Transcriber } from "./voice.js";
  *    and so is the agent's own, which unlinks every member. Neither is the notice, in the old chat
  *    or the new one, that a group became a supergroup: the migration is reported.
  * 2. Voice becomes text, so the router and the agent read what was said.
- * 3. The router answers or observes. Photos and documents are stored after the registry knows the
- *    sender, including on observed messages. Observing starts no session or typing indication;
+ * 3. The router answers or observes. Files always start a turn after the identity gates, even
+ *    without a caption or when addressed to another person. They are stored after the registry knows the
+ *    sender. Observing starts no session or typing indication;
  *    the ring carries the stored files' references (or storage failures) into the next reply.
  * 4. The session policy runs on the session an answered message continues: an idle one is cleared, the
  *    first message after the day's compaction time compacts it.

package/dist/extension/lib/telegram/inbound.mjs

@@ -14,7 +14,7 @@ import { formatAddress, messageAddress, scopedAddress } from "./sessions.mjs";
 import { questionAddress, questionFromHistory } from "./confirmations.mjs";
 import { MAXIMUM_NATIVE_IMAGE_BYTES } from "./images.mjs";
 import { chatContext } from "./chats.mjs";
-import { attachmentsContext, stageFiles } from "./files.mjs";
+import { attachmentsContext, inboundAttachments, stageFiles } from "./files.mjs";
 import { messageAudience } from "./audience.mjs";
 import { addressingReason, routeMessage } from "./router.mjs";
 import { routingContext } from "./routing-context.mjs";
@@ -175,7 +175,8 @@ async function handleInboundMessage(input) {
 	if (message.from === void 0 || message.from.isBot || message.chat.type === "channel") return null;
 	const sent = {
 		...message,
-		from: message.from
+		from: message.from,
+		attachments: inboundAttachments(message)
 	};
 	const bot = await dependencies.bot(context.telegram);
 	const audience = messageAudience(sent, bot);
@@ -227,6 +228,7 @@ async function handleInboundMessage(input) {
 	const caption = sent.text || sent.caption;
 	const notes = [];
 	let text = caption;
+	let voiceBytes;
 	const chat = observedChatKey({
 		chatId: sent.chat.id,
 		...sent.messageThreadId === void 0 ? {} : { messageThreadId: sent.messageThreadId }
@@ -236,7 +238,10 @@ async function handleInboundMessage(input) {
 		const spoken = await voiceText({
 			voice,
 			model: dependencies.sttModel,
-			download: dependencies.download,
+			download: async (fileId) => {
+				voiceBytes = await dependencies.download(fileId);
+				return voiceBytes;
+			},
 			transcribe: dependencies.transcribe
 		});
 		text = [caption, spoken.text].filter((part) => part.trim() !== "").join("\n\n");
@@ -252,12 +257,16 @@ async function handleInboundMessage(input) {
 		recordGateDecision("empty_message");
 		return null;
 	}
+	if (text.trim() === "" && sent.attachments.length > 0) {
+		text = "[attachment]";
+		if (input.request !== void 0) input.request.text = text;
+	}
 	if (sent.attachments.length > 0) await report;
 	const files = sent.attachments.length === 0 || dependencies.attachments === void 0 || input.request === void 0 ? new Map() : await stageFiles({
 		message: sent,
 		memberId: lookup.member.id,
 		attachments: dependencies.attachments,
-		download: dependencies.download,
+		download: (fileId) => voiceBytes !== void 0 && voice?.fileId === fileId ? Promise.resolve(voiceBytes) : dependencies.download(fileId),
 		...mediaPlan === void 0 ? {} : { imagePlan: mediaPlan },
 		...dependencies.describeImage === void 0 ? {} : { describeImage: dependencies.describeImage }
 	});
@@ -269,6 +278,7 @@ async function handleInboundMessage(input) {
 		chatId: sent.chat.id,
 		memberId: lookup.member.id,
 		policy: dependencies.policy,
+		hasAttachments: sent.attachments.length > 0,
 		addressed: addressed ?? (answering ? "pending-question" : void 0),
 		audience,
 		record: (decision) => recordRouteDecision(decision.route, addressed, dependencies.policy, decision),

package/dist/extension/lib/telegram/router.d.ts

@@ -4,8 +4,8 @@ import type { ObservedMessage, RecentMessage } from "./observed.js";
 import { type MessageAudience } from "./audience.js";
 /**
  * The inbound router (`agent-design.md`, section 4): every message from a paired member is either
- * answered or observed before eve starts a turn. A private chat, a mention, a reply to the bot and
- * a command are always answered. Any other group message is observed, or, with the `smart`
+ * answered or observed before eve starts a turn. Attachments, a private chat, a mention, a reply
+ * to the bot and a command are always answered. Any other group message is observed, or, with the `smart`
  * policy, put to one decision on the decision model.
  */
 export type Route = "observe" | "reply";
@@ -78,6 +78,8 @@ export declare function routeMessage(input: {
     readonly chatId: string;
     readonly memberId: string;
     readonly policy: TelegramPolicy;
+    /** Files always start a turn, including replies to other people and captionless media. */
+    readonly hasAttachments?: boolean;
     readonly addressed: AddressingReason | undefined;
     readonly audience?: MessageAudience;
     readonly record?: (decision: RouteDecision) => void;

package/dist/extension/lib/telegram/router.mjs

@@ -115,6 +115,7 @@ async function routeMessage(input) {
 		});
 		return route;
 	};
+	if (input.hasAttachments) return complete("reply", "attachment", "Inbound attachment starts an agent turn regardless of caption or recipient.");
 	if (input.addressed !== void 0 && input.addressed !== "pending-question") return complete("reply", input.addressed, `Bot addressed directly: ${input.addressed}.`);
 	const elsewhere = otherRecipient(input.audience);
 	if (elsewhere !== void 0) return complete("observe", "other_recipient", elsewhere);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@segmentstream/agent",
-  "version": "0.6.5",
+  "version": "0.6.6",
   "description": "Mount a SegmentStream workspace's agent on eve: the ontology as tools, acting as the member who writes, and Telegram through the workspace's connection.",
   "license": "Apache-2.0",
   "author": "SegmentStream",