pi-twitterapi.io 0.1.2 → 0.2.0

package/CHANGELOG.md

@@ -5,7 +5,60 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [0.2.0] - 2026-10-04
+
+Opt-in real video understanding. **Off by default:** without
+`enableVideoProcessing`, behaviour is unchanged and a video post is still
+represented by its poster frame.
+
+### Added
+
+- Real video processing (`enableVideoProcessing`, which also requires
+  `enableVideoUnderstanding`): native video through a Gemini endpoint
+  (`videoEndpointType: gemini-files`, `videoModel`, `videoApiKeyEnv`), and/or
+  frames via `ffmpegPath` plus a transcript from a remote STT endpoint
+  (`sttEndpoint`, `sttModel`, `sttApiKeyEnv`, `sttLanguage`) or local whisper.cpp
+  (`whisperCppBinary`, `whisperModelPath`).
+- Bounds: `maxVideoSeconds` (120), `maxVideoBytes` (32 MiB), `maxFrames` (8),
+  `maxVideosPerSearch` (1), `videoBudgetMs` (180 s, max 300 s).
+- The method actually used is disclosed in the answer: `gemini-native`,
+  `frames+stt`, `stt-only`, `frames-only` or `transcript-only`.
+
+### Security
+
+- Executable paths, endpoints and credential env names are read from **user
+  settings only**; project-level values for those keys are ignored and disclosed.
+- A custom `videoEndpoint` is used only when `videoApiKeyEnv` is set explicitly
+  and the URL is `https://`, and that authorization is re-checked at the adapter
+  boundary.
+- ffmpeg is never handed a URL (`-nostdin`, `-protocol_whitelist file`) and media
+  downloads stay on the SSRF allowlist, with redirects refused on authenticated
+  requests.
+
+### Behaviour notes
+
+- A native upload happens only when the clip is provably inside
+  `maxVideoSeconds`: it is trimmed locally first, and the native path is skipped
+  with a disclosure when it cannot be bounded. A trimming failure never falls
+  back to uploading the whole clip.
+- The native reply is requested as structured JSON. A reply that ignores that is
+  kept whole as visual evidence, so a transcript is never inferred from prose;
+  STT can still recover the speech.
+- Gemini Files uploads are deleted on a best-effort basis, including when
+  generation failed or was cancelled; a file that could not be deleted is
+  disclosed as possibly retained (Google keeps undeleted uploads for ~48 hours).
+- Worst case a single video call can take several minutes.
+
+### Fixed
+
+- Video variants are now selected from a real `HEAD` request instead of the
+  advertised bitrate. That bitrate is a target rather than an average and
+  overstates the file by roughly 3x, so the old estimate picked a needlessly low
+  resolution (640x360 where 1280x720 fitted) and could misjudge both caps.
+- The video-phase budget defaults to 180 s and is capped at 300 s, up from 90 s
+  and 120 s. Live measurement showed one provider call over a 65 s clip taking
+  33 s to ~71 s with run-to-run variance, so the old ceiling aborted long videos
+  and silently fell back to the poster frame.
 
 ## [0.1.1] - 2026-10-03
 

package/README.md

@@ -61,6 +61,51 @@ override — and set only the keys you need:
 | `maxPagesCeiling` | no | Hard cap that `maxPages` is clamped to (default 20). |
 | `minRequestIntervalMs` | no | Minimum spacing between upstream requests (default 5000). twitterapi.io allows 0.2 QPS on unpaid accounts; raise it if you are being throttled, lower it for a higher-QPS tier, or set it to 0 to disable pacing. |
 | `retryBaseDelayMs` | no | Base delay for retry backoff (default 5000). |
+| `enableVideoProcessing` | no | Run **real** video processing (native video and/or frames + transcript). Requires `enableVideoUnderstanding`. Off by default. |
+| `videoEndpointType` | no | Native-video wire format: `gemini-files` (default) or `openai-compatible` (unverified). |
+| `videoEndpoint` / `videoModel` / `videoApiKeyEnv` | no | Native-video endpoint, model, and the **env var name** holding the key (default `GOOGLE_API_KEY`). `gemini-files` targets Google unless `videoEndpoint` is set. |
+| `sttEndpoint` / `sttModel` / `sttApiKeyEnv` | no | OpenAI-compatible speech-to-text endpoint, model, and key env var (default `STT_API_KEY`). No hidden default provider. |
+| `sttLanguage` | no | ISO-639-1 language for STT, or `auto` (default). |
+| `ffmpegPath` | no | ffmpeg binary override; otherwise `ffmpeg` is searched on `PATH`. ffmpeg must be installed locally (no bundled binary). |
+| `whisperCppBinary` / `whisperModelPath` | no | Local whisper.cpp binary and GGML model (both user-installed). |
+| `maxVideoSeconds` / `maxVideoBytes` / `maxFrames` / `maxVideosPerSearch` / `videoBudgetMs` | no | Video bounds: duration guard (120), download cap (32 MiB), frames per video (8), videos per search (1), time budget (180 s, max 300 s). |
+
+> **Video processing is opt-in and local-tooling first.** It needs
+> `enableVideoUnderstanding: true` **and** `enableVideoProcessing: true`, plus a
+> locally installed `ffmpeg` (for frames/audio) and optionally whisper.cpp, or a
+> configured native-video / STT endpoint. In v1 native video goes to **Gemini
+> only** (`gemini-files`); frames are sent to your pi model; `openai-compatible`
+> video is **not** enabled pending verification (Grok cannot take video input at
+> all). Sending video/audio to a third-party endpoint is disclosed in the answer.
+> The native request asks for structured JSON (`responseMimeType:
+> application/json`), so the model names the visual/transcript sections itself. A
+> reply that ignores that is kept whole as visual evidence: the transcript is
+> never inferred from prose (an invented transcript would be published as
+> evidence), and STT still recovers the real speech when it is configured.
+> Variants are chosen from a real `HEAD` request rather than the advertised
+> bitrate, which overstates the file by roughly 3x (a nominal 2176 kbps clip
+> measured 6.16 MB where the bitrate suggests 19.6 MB).
+> Executable paths, endpoints and credential names are read from **user
+> settings only** — project `.pi/settings.json` values for those keys are
+> ignored and disclosed. A custom `videoEndpoint` is only honoured when
+> `videoApiKeyEnv` is set explicitly (and must be `https://`), so the default
+> key is never sent to another host.
+>
+> **Retention and duration.** When native video uses the Gemini Files API the
+> upload is deleted on a **best-effort** basis once the call finishes, with its
+> own short timeout so a cancelled request cannot skip it. If deletion fails — or
+> an upload happened but generation failed, returned nothing, or was cancelled —
+> the answer says the file may be retained. Google's Files API keeps undeleted
+> uploads for roughly **48 hours**, so treat such a file as readable by that
+> project for about that long. Only the first `maxVideoSeconds` (default 120 s)
+> are analysed: the video is trimmed locally when possible, and a clip that
+> **cannot** be trimmed is not uploaded whole — the native path is skipped and
+> disclosed, while frames and audio stay limited to that window. Worst case a
+> single video call can take several minutes (retrieval pacing + 60 s media phase
+> + up to `videoBudgetMs` video phase + synthesis), and the budget defaults to
+> 180 s and caps at 300 s. Provider video analysis is the slow part and its
+> latency varies: measured live, one call over a 65 s clip took 33 s once and
+> ~71 s another time, so expect a long tool call on a media-heavy query.
 
 > **Important:** the extension needs a pi version whose `ModelRegistry.complete`
 > exists — it is absent on pi 0.80.6, present from pi 0.99.2, and verified on
@@ -126,8 +171,11 @@ than silently ignored.
   dropped from `Sources` and counted in the `## Notes` section. Non-X links are
   outside the citation contract: they are neither published as sources nor
   counted as invented citations.
-- **Media is best-effort.** Video cannot be sent to a chat model, so video posts
-  are represented by their poster frame and this limitation is disclosed.
+- **Media is best-effort.** By default a video post is represented by its poster
+  frame and the limitation is disclosed, because a chat model cannot ingest
+  video. With `enableVideoProcessing` (see above) the video itself is analysed —
+  locally trimmed frames/audio, and/or the configured native-video or STT
+  endpoint — and the poster is kept only as the fallback when that yields nothing.
 - **Partial retrieval is disclosed.** If paging stops early (page cap, cursor
   cycle, or a missing cursor while more results remain), the answer carries a
   note saying the results may be incomplete.
@@ -204,7 +252,7 @@ version, including parameter mapping, lives in
 | Result order | chosen by the model | `queryType` (`Latest`/`Top`), `replySort` |
 | Item-count control | ❌ | ✅ `count`, `limit` |
 | Image understanding | ✅ `enable_image_understanding` | ✅ `enableImageUnderstanding` (attached when the model accepts images) |
-| Video understanding | ✅ `enable_video_understanding` | ⚠️ poster frame only (chat models cannot ingest video) |
+| Video understanding | ✅ `enable_video_understanding` | ⚠️ poster frame by default; opt-in `enableVideoProcessing` adds native video (Gemini) and/or frames + STT |
 | Answer generation | Grok (xAI) | any pi model: `twitter.synthesisModel`, else the session model |
 | Citations | xAI annotations/citations | derived from fetched permalinks; unmatched X links dropped and disclosed |
 | Cost | xAI tokens + per post/profile | twitterapi.io credits + your model's tokens — [cost comparison](docs/pricing-comparison.md) |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-twitterapi.io",
-  "version": "0.1.2",
+  "version": "0.2.0",
   "description": "twitterapi.io-backed X/Twitter search extension for pi coding agent",
   "author": "José Antonio Galiano Sandoval",
   "keywords": [
@@ -20,12 +20,16 @@
     "url": "https://github.com/jagaliano/pi-twitterapi.io/issues"
   },
   "type": "module",
+  "engines": {
+    "node": ">=22.19.0"
+  },
   "files": [
     "src",
     "!src/**/*.test.ts",
     "README.md",
     "CHANGELOG.md",
-    "docs"
+    "docs",
+    "!docs/video-spike.md"
   ],
   "pi": {
     "extensions": [

package/src/backend/media.ts

@@ -3,12 +3,18 @@ import { toBase64, type ImageAttachment } from "../synthesize.js";
 const MAX_MEDIA_BYTES = 8 * 1024 * 1024;
 const MEDIA_TIMEOUT_MS = 20_000;
 
-/** Read a body while enforcing a byte cap, so an oversized response is never fully buffered. */
-async function readCapped(response: Response, limit: number): Promise<Uint8Array | undefined> {
+/** Receives streamed body chunks; may return a promise to apply backpressure. */
+export type ChunkSink = (chunk: Uint8Array) => void | Promise<void>;
+
+/**
+ * Stream a response body into `onChunk`, enforcing a byte cap, so an oversized
+ * response is never fully buffered. Returns `false` (and cancels the body) when
+ * the cap is exceeded, `true` when the body was read to the end.
+ */
+export async function readCapped(response: Response, limit: number, onChunk: ChunkSink): Promise<boolean> {
   const body = response.body;
-  if (!body) return undefined;
+  if (!body) return false;
   const reader = body.getReader();
-  const chunks: Uint8Array[] = [];
   let total = 0;
   try {
     for (;;) {
@@ -18,13 +24,25 @@ async function readCapped(response: Response, limit: number): Promise<Uint8Array
       total += value.byteLength;
       if (total > limit) {
         await reader.cancel().catch(() => undefined);
-        return undefined;
+        return false;
       }
-      chunks.push(value);
+      await onChunk(value);
     }
   } finally {
     reader.releaseLock();
   }
+  return true;
+}
+
+/** Buffer a capped body into bytes (images). Returns undefined when the cap is exceeded. */
+async function readCappedBytes(response: Response, limit: number): Promise<Uint8Array | undefined> {
+  const chunks: Uint8Array[] = [];
+  const ok = await readCapped(response, limit, (chunk) => {
+    chunks.push(chunk);
+  });
+  if (!ok) return undefined;
+  let total = 0;
+  for (const chunk of chunks) total += chunk.byteLength;
   const bytes = new Uint8Array(total);
   let offset = 0;
   for (const chunk of chunks) {
@@ -88,7 +106,7 @@ export function createFetchMedia(fetcher: typeof fetch, callerSignal?: AbortSign
       if (!mimeType.startsWith("image/")) return undefined;
       const declared = Number(response.headers.get("content-length") ?? Number.NaN);
       if (Number.isFinite(declared) && declared > maxBytes) return undefined;
-      const bytes = await readCapped(response, maxBytes);
+      const bytes = await readCappedBytes(response, maxBytes);
       if (!bytes) return undefined;
       return { data: toBase64(bytes), mimeType };
     } catch {

package/src/backend/model.ts

@@ -1,5 +1,6 @@
 import type { TwitterConfig } from "../config.js";
 import type { SynthesisModel } from "../synthesize.js";
+import type { ExecFn } from "./video.js";
 
 /**
  * Minimal structural view of pi's ModelRegistry, so this module stays testable
@@ -37,6 +38,8 @@ export interface BackendOptions {
   fallbackModelIds?: string[];
   /** Override the retry delay between last-model attempts (tests). */
   synthesisSleep?: (ms: number, signal?: AbortSignal) => Promise<void>;
+  /** Injected local-process runner for video processing (tests). */
+  videoExec?: ExecFn;
 }
 
 /** Resolve a `provider/model` spec, or a bare model id, against the registry. */

package/src/backend/runs.ts

@@ -6,6 +6,7 @@ import {
   synthesizeDocument,
   synthesizeTrends,
   synthesizeUserAnswer,
+  type SynthesisDeps,
 } from "../synthesize.js";
 import {
   fetchCommunityTweets,
@@ -33,6 +34,7 @@ import {
 } from "../twitterapi.js";
 import { toSynthesisModel, type TwitterApiSynthesisOptions } from "./model.js";
 import { createFetchMedia } from "./media.js";
+import { createProcessVideo } from "./video.js";
 import { applyFallbackNote, resolveSynthesisBackend, type SynthesisBackend } from "./synthesis.js";
 
 export interface TwitterApiRunOptions extends TwitterApiSynthesisOptions {
@@ -81,6 +83,32 @@ function incompleteReason(stoppedBy: string | undefined, pages: number): string
   return undefined;
 }
 
+/** Build synthesis deps, wiring the optional bound video pre-processor (M5). */
+function mediaDeps(
+  backend: Pick<SynthesisBackend, "complete" | "fetcher">,
+  options: TwitterApiSynthesisOptions,
+): SynthesisDeps {
+  return {
+    complete: backend.complete,
+    fetchMedia: createFetchMedia(backend.fetcher, options.signal),
+    processVideo: options.config.enableVideoProcessing
+      ? createProcessVideo({
+          fetcher: options.fetcher ?? fetch,
+          env: options.env ?? {},
+          signal: options.signal,
+          exec: options.videoExec,
+        })
+      : undefined,
+  };
+}
+
+/** Append config-level disclosures (ignored project keys, switch warnings). */
+function appendConfigNotes(options: TwitterApiSynthesisOptions, details: TwitterSearchDetails): void {
+  if (options.config.configNotes.length > 0) {
+    details.notes = [...(details.notes ?? []), ...options.config.configNotes];
+  }
+}
+
 /**
  * Retrieve posts from twitterapi.io and synthesize the answer, returning the
  * shared `{ markdown, details }` shape.
@@ -112,10 +140,7 @@ export async function runTwitterApiSearch(
     model: toSynthesisModel(model),
     signal: options.signal,
     incomplete,
-    deps: {
-      complete,
-      fetchMedia: createFetchMedia(fetcher, options.signal),
-    },
+    deps: mediaDeps(backend, options),
   });
 
   if (search.window?.shortfallHours) {
@@ -150,6 +175,7 @@ export async function runTwitterApiSearch(
     ];
   }
   applyFallbackNote(backend, details);
+  appendConfigNotes(options, details);
 
   return { markdown: formatTwitterResults(details), details };
 }
@@ -202,6 +228,7 @@ export async function runTwitterApiUserSearch(
     ];
   }
   applyFallbackNote(backend, details);
+  appendConfigNotes(options, details);
   return { markdown: formatTwitterResults(details), details };
 }
 
@@ -234,7 +261,7 @@ export async function runTwitterApiThread(
     model: toSynthesisModel(model),
     signal: options.signal,
     incomplete,
-    deps: { complete, fetchMedia: createFetchMedia(fetcher, options.signal) },
+    deps: mediaDeps(backend, options),
   });
   details.notes = [
     ...(details.notes ?? []),
@@ -247,6 +274,7 @@ export async function runTwitterApiThread(
     ];
   }
   applyFallbackNote(backend, details);
+  appendConfigNotes(options, details);
   return { markdown: formatTwitterResults(details), details };
 }
 
@@ -265,10 +293,11 @@ async function completeTweetAnswer(
     model: toSynthesisModel(backend.model),
     signal: options.signal,
     incomplete: input.incomplete,
-    deps: { complete: backend.complete, fetchMedia: createFetchMedia(backend.fetcher, options.signal) },
+    deps: mediaDeps(backend, options),
   });
   details.notes = [...(details.notes ?? []), ...input.notes];
   applyFallbackNote(backend, details);
+  appendConfigNotes(options, details);
   return { markdown: formatTwitterResults(details), details };
 }
 
@@ -392,6 +421,7 @@ export async function runTwitterApiTrends(
     deps: { complete: backend.complete },
   });
   applyFallbackNote(backend, details);
+  appendConfigNotes(options, details);
   return { markdown: formatTwitterResults(details), details };
 }
 
@@ -414,6 +444,7 @@ async function completeUserAnswer(
   });
   details.notes = [...(details.notes ?? []), ...input.notes];
   applyFallbackNote(backend, details);
+  appendConfigNotes(options, details);
   return { markdown: formatTwitterResults(details), details };
 }
 
@@ -556,6 +587,7 @@ export async function runTwitterApiAbout(
     notes,
   });
   applyFallbackNote(backend, details);
+  appendConfigNotes(options, details);
   return { markdown: formatTwitterResults(details), details };
 }
 
@@ -683,5 +715,6 @@ export async function runTwitterApiSpace(
     notes,
   });
   applyFallbackNote(backend, details);
+  appendConfigNotes(options, details);
   return { markdown: formatTwitterResults(details), details };
 }

package/src/backend/synthesis.ts

@@ -36,6 +36,37 @@ export function completionText(message: unknown): string {
 }
 
 
+/** The only efforts a provider can legitimately name for a reasoning model. */
+const REASONING_EFFORTS = ["minimal", "low", "medium", "high", "xhigh", "max"] as const;
+
+/**
+ * Recover the cheapest accepted reasoning effort from a provider rejection.
+ *
+ * Some models cannot be called with reasoning off at all — pi sends
+ * `reasoning_effort: 'none'` by default, and an `openai-responses` model that
+ * requires reasoning answers 400 with the accepted list. Retrying once with the
+ * first supported value keeps such models usable instead of failing synthesis
+ * (measured live against `opencode-go/muse-spark-1.3-contributor`).
+ *
+ * The rejection and the list must belong to the *same* sentence mentioning the
+ * reasoning effort, and the recovered value must be a real effort name: an
+ * unrelated 400 that happens to list its own values (for example
+ * `Invalid response_format. Supported values: [json, text]`) must not trigger a
+ * repair that would mask the original cause.
+ */
+export function supportedReasoningEffort(error: unknown): string | undefined {
+  const message = error instanceof Error ? error.message : String(error);
+  const rejected = /reasoning[_ ]effort[^.;]{0,120}?(?:not supported|unsupported|invalid)/i.test(message)
+    || /(?:not supported|unsupported|invalid)[^.;]{0,120}?reasoning[_ ]effort/i.test(message);
+  if (!rejected) return undefined;
+  const list = /supported values?:?\s*\[([^\]]+)\]/i.exec(message)?.[1];
+  if (!list) return undefined;
+  return list
+    .split(",")
+    .map((value) => value.trim().replace(/^["']|["']$/g, "").toLowerCase())
+    .find((value) => (REASONING_EFFORTS as readonly string[]).includes(value));
+}
+
 /**
  * Build the synthesis completion for a resolved model. Shared by every
  * twitterapi.io path so failure handling cannot differ between them.
@@ -50,32 +81,42 @@ function createCompletion(
       request.mediaManifest && request.images.length > 0
         ? `${request.prompt}\n\nAttached images, in order:\n${request.mediaManifest}`
         : request.prompt;
-    const message = await run.call(
-      registry,
-      model as never,
-      {
-        systemPrompt: request.system,
-        messages: [
-          {
-            role: "user",
-            content:
-              request.images.length > 0
-                ? [
-                    { type: "text", text: promptText },
-                    ...request.images.map((image) => ({
-                      type: "image",
-                      data: image.data,
-                      mimeType: image.mimeType,
-                    })),
-                  ]
-                : request.prompt,
-            timestamp: Date.now(),
-          },
-        ],
-      } as never,
-      { signal: request.signal } as never,
-    );
-    return completionText(message);
+    const attempt = (reasoningEffort?: string): Promise<unknown> =>
+      run.call(
+        registry,
+        model as never,
+        {
+          systemPrompt: request.system,
+          messages: [
+            {
+              role: "user",
+              content:
+                request.images.length > 0
+                  ? [
+                      { type: "text", text: promptText },
+                      ...request.images.map((image) => ({
+                        type: "image",
+                        data: image.data,
+                        mimeType: image.mimeType,
+                      })),
+                    ]
+                  : request.prompt,
+              timestamp: Date.now(),
+            },
+          ],
+        } as never,
+        (reasoningEffort ? { signal: request.signal, reasoningEffort } : { signal: request.signal }) as never,
+      );
+    try {
+      return completionText(await attempt());
+    } catch (error) {
+      // Authoritative cancellation wins: never spend another registry call on a
+      // repair after the caller has gone away (P2-2).
+      if (classifySynthesisError(error, request.signal) === "cancelled") throw error;
+      const effort = supportedReasoningEffort(error);
+      if (!effort) throw error;
+      return completionText(await attempt(effort));
+    }
   };
 }