ugc-lens 0.0.0-stage → 0.0.2

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 MEI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,106 @@
-# Temporary Holding Version
+# ugc-lens
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+面向中文社媒用户互动内容（评论、回复、弹幕等）的轻量分析 Agent。给它一个搜集好的数据文件，生成带原文引用的 Markdown 分析报告，也可以继续追问。以 npm 包形式提供命令行和代码调用，MCP 服务后续提供。
+
+> 状态：0.0.x 试用版。目前只支持 JSON 文件，接口和报告格式还可能调整。
+
+## 安装
+
+需要 Node.js 22.18 或更高版本。
+
+```
+npm install -g ugc-lens
+# 或者不安装，直接用 npx ugc-lens ...
+```
+
+## 配置模型
+
+最简单的方式：
+
+```
+ugc-lens init
+```
+
+交互式地选提供商、从 `/models` 列表里选模型、填 API key 并发一次测试调用，写入全局配置 `~/.ugc-lens/config.json`（权限 600），所有目录下都能用。
+
+也可以手动配置：在运行命令的目录里放一个 `.env`（或设置同名环境变量），选一个提供商并填它的 API key：
+
+```
+UGC_LENS_PROVIDER=deepseek
+UGC_LENS_MODEL=deepseek-flash
+DEEPSEEK_API_KEY=你的 key
+```
+
+支持 DeepSeek、通义千问、Kimi、智谱、MiniMax、OpenAI、Claude、Gemini、Grok、OpenRouter 和本地模型，变量名和差异见 [模型提供商](docs/providers.md)。优先级：环境变量 > 当前目录的 `.env` / `.ugc-lens/config.json` > 全局配置。
+
+## 使用
+
+```
+ugc-lens init                                     # 初始化模型配置
+
+# 分析一个文件：报告打印到标准输出，同时保存到 .ugc-lens/reports/
+ugc-lens analyze bilibili_all.json > report.md
+ugc-lens analyze bilibili_all.json --focus 价格,售后     # 只看这些方向
+
+# 追问：回答里用 [ID] 标出依据，多个问题依次追问
+ugc-lens ask bilibili_all.json "评论主要在吐槽什么？" "点赞最多的那条下面怎么回复的？"
+
+# 监听目录：新出现的 *_all.json 写完后自动分析，每个文件只分析一次
+ugc-lens watch ./data
+ugc-lens watch ./data --pattern "*_all.json,*_comments.json" --existing
+
+ugc-lens <命令> --help    # 查看全部选项
+```
+
+- 文件结构第一次由模型识别，规则缓存在 `.ugc-lens/mappings/`，同样结构的文件之后不再调用模型。
+- 报告包括：概况、IP 属地、关键词、弹幕高峰、各方向（主要观点、吐槽点、正面评价、争议）的主题和原文引用，末尾是耗时和 token 用量。
+- 统计、去重、关键词、引用校验都由代码完成，模型只负责分块总结和合并，一个 10 万条评论的文件大约用 6～7 万 token。
+- `watch` 的任务状态存在 `.ugc-lens/tasks.json`，按 Ctrl+C 停止后再启动会继续没做完的任务。
+
+## 代码调用
+
+```ts
+import { analyzeDataset, createChatClient, createLlmModel, loadDataset, loadLlmConfig } from 'ugc-lens'
+
+const config = loadLlmConfig()
+const model = createLlmModel(createChatClient(config), { temperature: 0, disableThinking: true })
+const dataset = await loadDataset('bilibili_all.json', { model })
+const result = await analyzeDataset({ model, dataset })
+console.log(result.markdown)
+```
+
+还导出了 `askDataset`（追问）、`watchDirectory`（监听目录）、`computeStats`、`searchFeedback` 等，类型声明随包提供。
+
+## 文档
+
+- [架构设计](docs/architecture.md)：定位、分层、Agent 循环、工具规范、数据格式、分析流水线
+- [模型提供商](docs/providers.md)：支持的厂商、配置格式、各家差异
+- [开发路线](docs/roadmap.md)：按步骤实现，每一步的目标、要掌握的概念和完成标准
+- [更新记录](docs/changelog/)：每次更新一份，文件名为 `日期-大致功能.md`
+- [项目指令](AGENTS.md)：协作规则
+
+## 开发
+
+```
+bun install          # 安装依赖
+npm run typecheck    # 类型检查
+npm test             # 测试
+npm run build        # 编译到 dist/（发布用）
+node src/cli/index.ts analyze <文件>   # 开发时直接运行源码
+npm run loop         # 用假模型演示 Agent 循环
+npm run tools        # 真实模型 + 基础工具（需要 .env）
+npm run params       # 参数纠错实验（需要 .env）
+npm run retry        # 重试演示与 token 预算（budget 部分需要 .env）
+npm run stats        # 模型识别 JSON 文件结构，转换成统一格式并打印统计（需要 .env）
+npm run ask          # 用数据工具追问，回答带评论 ID 作为证据（需要 .env）
+npm run chunks       # 新词发现、关键词、弹幕高峰和分块结果，不调用模型做分析
+npm run summarize    # 分块总结：每块调用一次模型，打印各方向的观点、引用和 token 用量（需要 .env）
+npm run analyze      # 完整分析，Markdown 报告打印出来并保存到 .ugc-lens/reports/（需要 .env）
+npm run watch -- <目录> # 监听目录，新出现的 *_all.json 写完后自动分析（需要 .env）
+```
+
+发布：修改 `package.json` 的版本号并推送到 master，GitHub Actions（`release.yml`）会检查、编译并发布到 npm，然后打 `v版本号` 标签。
+
+## 许可
+
+MIT

package/dist/agent/events.d.ts

@@ -0,0 +1,60 @@
+import type { Usage } from '../llm/types.js';
+export type RunEvent = {
+    type: 'run_start';
+    run_id: string;
+    kind: 'agent' | 'pipeline';
+}
+/** final 为 true 表示收尾调用 */
+ | {
+    type: 'model_start';
+    run_id: string;
+    step: number;
+    final: boolean;
+} | {
+    type: 'model_delta';
+    run_id: string;
+    step: number;
+    text: string;
+} | {
+    type: 'model_end';
+    run_id: string;
+    step: number;
+    usage: Usage;
+    tool_calls: number;
+}
+/** 模型接口暂时性错误，等待 delay_ms 后进行第 attempt 次重试 */
+ | {
+    type: 'model_retry';
+    run_id: string;
+    step: number;
+    attempt: number;
+    delay_ms: number;
+    reason: string;
+} | {
+    type: 'tool_start';
+    run_id: string;
+    step: number;
+    call_id: string;
+    name: string;
+    args: unknown;
+} | {
+    type: 'tool_end';
+    run_id: string;
+    step: number;
+    call_id: string;
+    name: string;
+    ok: boolean;
+    duration_ms: number;
+} | {
+    type: 'pipeline_stage';
+    run_id: string;
+    stage: string;
+    progress?: number;
+} | {
+    type: 'run_end';
+    run_id: string;
+    stop_reason: string;
+    usage: Usage;
+    duration_ms: number;
+};
+export type RunEventListener = (event: RunEvent) => void;

package/dist/agent/events.js

@@ -0,0 +1 @@
+export {};

package/dist/agent/fake_model.d.ts

@@ -0,0 +1,31 @@
+import type { ChatMessage, ToolChoice, ToolDefinition } from '../llm/types.js';
+import type { AgentModel, AgentModelRequest } from './model.js';
+export interface FakeToolCall {
+    name: string;
+    args?: unknown;
+    /** 直接指定 arguments 字符串，用来模拟模型给出不合法 JSON 的情况 */
+    raw_arguments?: string;
+}
+export interface FakeReply {
+    content?: string;
+    tool_calls?: FakeToolCall[];
+}
+/** 数组：按调用顺序依次返回；函数：根据请求动态决定（call 从 1 开始） */
+export type FakeScript = FakeReply[] | ((request: AgentModelRequest, call: number) => FakeReply);
+/** 每次调用时请求内容的快照，测试用来检查循环发出了什么 */
+export interface FakeRequestRecord {
+    messages: ChatMessage[];
+    tools: ToolDefinition[];
+    toolChoice: ToolChoice;
+}
+export interface FakeModel extends AgentModel {
+    readonly requests: readonly FakeRequestRecord[];
+}
+export declare function estimateTokens(text: string): number;
+/**
+ * 按剧本返回结果的假模型。行为与真实模型的约束一致：
+ * tool_choice 为 none 时不能请求工具。
+ */
+export declare function createFakeModel(script: FakeScript, options?: {
+    delayMs?: number;
+}): FakeModel;

package/dist/agent/fake_model.js

@@ -0,0 +1,53 @@
+import { setTimeout as sleep } from 'node:timers/promises';
+export function estimateTokens(text) {
+    return Math.ceil(text.length / 2);
+}
+/**
+ * 按剧本返回结果的假模型。行为与真实模型的约束一致：
+ * tool_choice 为 none 时不能请求工具。
+ */
+export function createFakeModel(script, options = {}) {
+    const requests = [];
+    return {
+        requests,
+        async complete(request) {
+            const call = requests.length + 1;
+            requests.push({
+                messages: structuredClone([...request.messages]),
+                tools: structuredClone([...request.tools]),
+                toolChoice: request.toolChoice,
+            });
+            if (options.delayMs)
+                await sleep(options.delayMs, undefined, { signal: request.signal });
+            const reply = typeof script === 'function' ? script(request, call) : script[call - 1];
+            if (!reply)
+                throw new Error(`假模型剧本已用完：第 ${call} 次调用没有对应的回复`);
+            if (request.toolChoice === 'none' && reply.tool_calls?.length) {
+                throw new Error(`假模型剧本错误：第 ${call} 次调用的 tool_choice 为 none，却返回了工具调用`);
+            }
+            if (typeof request.toolChoice === 'object') {
+                const forced = request.toolChoice.function.name;
+                if (!reply.tool_calls?.length || reply.tool_calls.some((item) => item.name !== forced)) {
+                    throw new Error(`假模型剧本错误：第 ${call} 次调用指定了工具 ${forced}，返回的工具调用不符`);
+                }
+            }
+            const toolCalls = (reply.tool_calls ?? []).map((item, index) => ({
+                id: `call_${call}_${index + 1}`,
+                type: 'function',
+                function: { name: item.name, arguments: item.raw_arguments ?? JSON.stringify(item.args ?? {}) },
+            }));
+            const message = { role: 'assistant', content: reply.content ?? null };
+            if (toolCalls.length > 0)
+                message.tool_calls = toolCalls;
+            return {
+                message,
+                finishReason: toolCalls.length > 0 ? 'tool_calls' : 'stop',
+                usage: {
+                    input_tokens: estimateTokens(JSON.stringify(request.messages)),
+                    output_tokens: estimateTokens(JSON.stringify(message)),
+                },
+                model: 'fake',
+            };
+        },
+    };
+}

package/dist/agent/llm_model.d.ts

@@ -0,0 +1,9 @@
+import type { ChatClient } from '../llm/client.js';
+import type { AgentModel } from './model.js';
+export interface LlmModelOptions {
+    temperature?: number;
+    maxTokens?: number;
+    disableThinking?: boolean;
+}
+/** 把模型客户端包装成 Agent 循环使用的 AgentModel */
+export declare function createLlmModel(client: Pick<ChatClient, 'chat'>, options?: LlmModelOptions): AgentModel;

package/dist/agent/llm_model.js

@@ -0,0 +1,6 @@
+/** 把模型客户端包装成 Agent 循环使用的 AgentModel */
+export function createLlmModel(client, options = {}) {
+    return {
+        complete: ({ messages, tools, toolChoice, signal, onRetry }) => client.chat(messages, { ...options, tools, toolChoice, signal, onRetry }),
+    };
+}

package/dist/agent/loop.d.ts

@@ -0,0 +1,56 @@
+import type { ChatMessage, Usage } from '../llm/types.js';
+import { type Approver } from '../tools/approval.js';
+import { type AgentTool } from '../tools/types.js';
+import type { RunEventListener } from './events.js';
+import type { AgentModel } from './model.js';
+export type StopReason = 'completed' | 'max_steps' | 'token_budget' | 'timeout' | 'aborted' | 'tool_errors';
+export interface AgentLimits {
+    maxSteps: number;
+    /** 输入加输出的累计 token */
+    tokenBudget: number;
+    timeoutMs: number;
+    /** 连续多少步的工具调用全部失败后停止 */
+    maxConsecutiveToolErrors: number;
+}
+export declare const DEFAULT_LIMITS: AgentLimits;
+export interface ToolCallRecord {
+    call_id: string;
+    name: string;
+    args: unknown;
+    ok: boolean;
+    result: string;
+    duration_ms: number;
+}
+export interface AgentStep {
+    step: number;
+    /** 收尾调用 */
+    final: boolean;
+    content: string | null;
+    tool_calls: ToolCallRecord[];
+    usage: Usage;
+}
+export interface AgentRunResult {
+    run_id: string;
+    answer: string;
+    stop_reason: StopReason;
+    steps: AgentStep[];
+    /** 运行结束时的完整历史 */
+    messages: ChatMessage[];
+    usage: Usage;
+    duration_ms: number;
+}
+export interface RunAgentOptions {
+    model: AgentModel;
+    /** 初始消息，通常是 system + user。不会被修改 */
+    messages: readonly ChatMessage[];
+    tools?: readonly AgentTool[];
+    limits?: Partial<AgentLimits>;
+    signal?: AbortSignal;
+    onEvent?: RunEventListener;
+    /** 工具解析相对路径、执行命令的目录，默认 process.cwd() */
+    cwd?: string;
+    /** 有副作用的工具执行前的确认函数，默认一律拒绝 */
+    approve?: Approver;
+}
+export declare function finalizeNotice(reason: StopReason): string;
+export declare function runAgent(options: RunAgentOptions): Promise<AgentRunResult>;

package/dist/agent/loop.js

@@ -0,0 +1,193 @@
+import { randomUUID } from 'node:crypto';
+import { denyAll } from '../tools/approval.js';
+import { ToolArgumentError } from '../tools/define.js';
+import { toToolDefinition } from '../tools/types.js';
+export const DEFAULT_LIMITS = {
+    maxSteps: 10,
+    tokenBudget: 100_000,
+    timeoutMs: 5 * 60_000,
+    maxConsecutiveToolErrors: 3,
+};
+const FINALIZE_LABELS = {
+    max_steps: '最大步数',
+    token_budget: 'token 预算',
+    tool_errors: '连续工具错误上限',
+};
+export function finalizeNotice(reason) {
+    const label = FINALIZE_LABELS[reason] ?? reason;
+    return `【系统说明】已达到${label}，不能再调用工具。请基于以上已有信息直接回答用户的问题，并说明哪些信息还不足。`;
+}
+export async function runAgent(options) {
+    const limits = { ...DEFAULT_LIMITS, ...options.limits };
+    const run_id = randomUUID();
+    const startedAt = Date.now();
+    const emit = options.onEvent ?? (() => { });
+    const messages = [...options.messages];
+    const tools = options.tools ?? [];
+    const toolMap = new Map(tools.map((tool) => [tool.name, tool]));
+    const definitions = tools.map(toToolDefinition);
+    const deadline = AbortSignal.timeout(limits.timeoutMs);
+    const signal = options.signal ? AbortSignal.any([options.signal, deadline]) : deadline;
+    const toolContext = {
+        run_id,
+        signal,
+        cwd: options.cwd ?? process.cwd(),
+        approve: options.approve ?? denyAll,
+    };
+    const steps = [];
+    const usage = { input_tokens: 0, output_tokens: 0 };
+    let consecutiveToolErrors = 0;
+    let answer = '';
+    let stopReason;
+    const abortReason = () => {
+        if (options.signal?.aborted)
+            return 'aborted';
+        if (deadline.aborted)
+            return 'timeout';
+        return undefined;
+    };
+    // 停止检查放在工具执行之后、下一次调用模型之前
+    const limitReason = (step) => {
+        if (usage.input_tokens + usage.output_tokens >= limits.tokenBudget)
+            return 'token_budget';
+        if (consecutiveToolErrors >= limits.maxConsecutiveToolErrors)
+            return 'tool_errors';
+        if (step > limits.maxSteps)
+            return 'max_steps';
+        return undefined;
+    };
+    const callModel = async (step, toolChoice, final) => {
+        emit({ type: 'model_start', run_id, step, final });
+        const result = await options.model.complete({
+            messages,
+            tools: definitions,
+            toolChoice,
+            signal,
+            onRetry: ({ attempt, delayMs, error }) => emit({ type: 'model_retry', run_id, step, attempt, delay_ms: delayMs, reason: error.message.slice(0, 200) }),
+        });
+        usage.input_tokens += result.usage.input_tokens;
+        usage.output_tokens += result.usage.output_tokens;
+        emit({
+            type: 'model_end',
+            run_id,
+            step,
+            usage: result.usage,
+            tool_calls: result.message.tool_calls?.length ?? 0,
+        });
+        return result;
+    };
+    emit({ type: 'run_start', run_id, kind: 'agent' });
+    for (let step = 1;; step++) {
+        stopReason = abortReason() ?? limitReason(step);
+        if (stopReason)
+            break;
+        let result;
+        try {
+            result = await callModel(step, 'auto', false);
+        }
+        catch (error) {
+            stopReason = abortReason();
+            if (stopReason)
+                break;
+            throw error;
+        }
+        messages.push(result.message);
+        const calls = result.message.tool_calls ?? [];
+        const record = {
+            step,
+            final: false,
+            content: result.message.content,
+            tool_calls: [],
+            usage: result.usage,
+        };
+        steps.push(record);
+        if (calls.length === 0) {
+            answer = result.message.content ?? '';
+            stopReason = 'completed';
+            break;
+        }
+        let failed = 0;
+        for (const call of calls) {
+            const callRecord = await executeToolCall(call, step, toolMap, toolContext, emit);
+            record.tool_calls.push(callRecord);
+            // 每个 tool_call 都必须有对应的 tool 消息，出错也一样
+            messages.push({ role: 'tool', tool_call_id: call.id, content: callRecord.result });
+            if (!callRecord.ok)
+                failed++;
+        }
+        consecutiveToolErrors = failed === calls.length ? consecutiveToolErrors + 1 : 0;
+    }
+    if (FINALIZE_LABELS[stopReason] !== undefined) {
+        answer = await finalize(stopReason);
+    }
+    const duration_ms = Date.now() - startedAt;
+    emit({ type: 'run_end', run_id, stop_reason: stopReason, usage: { ...usage }, duration_ms });
+    return { run_id, answer, stop_reason: stopReason, steps, messages, usage, duration_ms };
+    async function finalize(reason) {
+        const last = messages.at(-1);
+        // 这条工具结果还没发给模型，附加说明不算修改历史，也不需要伪造一条 user 消息
+        if (last?.role === 'tool') {
+            messages[messages.length - 1] = { ...last, content: `${last.content}\n\n${finalizeNotice(reason)}` };
+        }
+        const step = steps.length + 1;
+        let result;
+        try {
+            result = await callModel(step, 'none', true);
+        }
+        catch (error) {
+            if (abortReason())
+                return '';
+            throw error;
+        }
+        messages.push(result.message);
+        steps.push({ step, final: true, content: result.message.content, tool_calls: [], usage: result.usage });
+        return result.message.content ?? '';
+    }
+}
+async function executeToolCall(call, step, toolMap, ctx, emit) {
+    const startedAt = Date.now();
+    const { name, arguments: rawArgs } = call.function;
+    let args = rawArgs;
+    let ok = false;
+    let result;
+    const tool = toolMap.get(name);
+    const parsed = parseArguments(rawArgs);
+    if (parsed.ok)
+        args = parsed.value;
+    emit({ type: 'tool_start', run_id: ctx.run_id, step, call_id: call.id, name, args });
+    if (!tool) {
+        const available = [...toolMap.keys()].join(', ') || '无';
+        result = `错误：不存在名为 "${name}" 的工具。可用工具：${available}`;
+    }
+    else if (!parsed.ok) {
+        result = `错误：${parsed.error}`;
+    }
+    else {
+        try {
+            result = await tool.execute(parsed.value, ctx);
+            ok = true;
+        }
+        catch (error) {
+            result =
+                error instanceof ToolArgumentError
+                    ? `错误：工具 ${name} 的参数不正确。\n${error.message}`
+                    : `错误：工具 ${name} 执行失败：${error.message}`;
+        }
+    }
+    const duration_ms = Date.now() - startedAt;
+    emit({ type: 'tool_end', run_id: ctx.run_id, step, call_id: call.id, name, ok, duration_ms });
+    return { call_id: call.id, name, args, ok, result, duration_ms };
+}
+function parseArguments(raw) {
+    let value;
+    try {
+        value = JSON.parse(raw.trim() === '' ? '{}' : raw);
+    }
+    catch {
+        return { ok: false, error: `参数不是合法的 JSON：${raw}` };
+    }
+    if (typeof value !== 'object' || value === null || Array.isArray(value)) {
+        return { ok: false, error: `参数必须是 JSON 对象：${raw}` };
+    }
+    return { ok: true, value: value };
+}

package/dist/agent/model.d.ts

@@ -0,0 +1,14 @@
+import type { RetryInfo } from '../llm/retry.js';
+import type { ChatMessage, ChatResult, ToolChoice, ToolDefinition } from '../llm/types.js';
+export interface AgentModelRequest {
+    messages: readonly ChatMessage[];
+    tools: readonly ToolDefinition[];
+    toolChoice: ToolChoice;
+    signal: AbortSignal;
+    /** 模型接口重试前调用，循环用它发出 model_retry 事件 */
+    onRetry?: (info: RetryInfo) => void;
+}
+/** Agent 循环只依赖这个接口，假模型和真实模型都实现它 */
+export interface AgentModel {
+    complete(request: AgentModelRequest): Promise<ChatResult>;
+}

package/dist/agent/model.js

@@ -0,0 +1 @@
+export {};

package/dist/analysis/danmaku.d.ts

@@ -0,0 +1,39 @@
+import type { FeedbackItem } from '../dataset/types.js';
+import { type DistinctiveTerm, type TermStat, type Tokenizer } from './keywords.js';
+import { type Peak } from './timeline.js';
+/** 说的是同一句话的弹幕 */
+export interface DanmakuGroup {
+    /** 第一次出现的那条的原文 */
+    text: string;
+    count: number;
+    /** 第一条的 ID，引用时用它 */
+    id: string;
+    /** 第一次出现的视频时间 */
+    first_sec?: number;
+}
+export interface DanmakuPeak extends Peak {
+    /** 这个时段里重复最多的弹幕 */
+    groups: DanmakuGroup[];
+    /** 这个时段相对其他时段特有的词 */
+    terms: DistinctiveTerm[];
+}
+export interface DanmakuSummary {
+    total: number;
+    /** 去重后的条数 */
+    unique: number;
+    /** 按重复次数从多到少 */
+    groups: DanmakuGroup[];
+    bin_sec: number;
+    /** 每段的弹幕数 */
+    timeline: number[];
+    peaks: DanmakuPeak[];
+    terms: TermStat[];
+}
+export declare function groupDanmaku(items: readonly FeedbackItem[]): DanmakuGroup[];
+export interface DanmakuOptions {
+    binSec?: number;
+    peaks?: number;
+    groupsPerPeak?: number;
+    terms?: number;
+}
+export declare function summarizeDanmaku(items: readonly FeedbackItem[], tokenizer: Tokenizer, options?: DanmakuOptions): DanmakuSummary;

package/dist/analysis/danmaku.js

@@ -0,0 +1,50 @@
+import { countTerms, distinctiveTerms, topTerms } from './keywords.js';
+import { dedupeKey } from './text.js';
+import { binCounts, chooseBinSeconds, detectPeaks } from './timeline.js';
+export function groupDanmaku(items) {
+    const groups = new Map();
+    for (const item of items) {
+        const key = dedupeKey(item.text);
+        if (!key)
+            continue;
+        const group = groups.get(key);
+        if (group) {
+            group.count++;
+            if (item.video_time_sec !== undefined && (group.first_sec === undefined || item.video_time_sec < group.first_sec)) {
+                group.first_sec = item.video_time_sec;
+            }
+        }
+        else {
+            groups.set(key, { text: item.text, count: 1, id: item.id, first_sec: item.video_time_sec });
+        }
+    }
+    return [...groups.values()].sort((a, b) => b.count - a.count || (a.first_sec ?? 0) - (b.first_sec ?? 0));
+}
+export function summarizeDanmaku(items, tokenizer, options = {}) {
+    const danmaku = items.filter((item) => item.kind === 'danmaku');
+    const groups = groupDanmaku(danmaku);
+    const timed = danmaku.filter((item) => item.video_time_sec !== undefined);
+    const duration = Math.max(0, ...timed.map((item) => item.video_time_sec));
+    const binSec = options.binSec ?? chooseBinSeconds(duration);
+    const timeline = binCounts(timed.map((item) => item.video_time_sec), binSec);
+    const docs = (list) => list.map((item) => ({ id: item.id, text: item.text }));
+    const peaks = detectPeaks(timeline, binSec, { limit: options.peaks ?? 5 }).map((peak) => {
+        const inside = (item) => item.video_time_sec >= peak.start_sec && item.video_time_sec < peak.end_sec;
+        const within = timed.filter(inside);
+        const outside = timed.filter((item) => !inside(item));
+        return {
+            ...peak,
+            groups: groupDanmaku(within).slice(0, options.groupsPerPeak ?? 8),
+            terms: distinctiveTerms(countTerms(docs(within), tokenizer), countTerms(docs(outside), tokenizer), { limit: options.terms ?? 8 }),
+        };
+    });
+    return {
+        total: danmaku.length,
+        unique: groups.length,
+        groups,
+        bin_sec: binSec,
+        timeline,
+        peaks,
+        terms: topTerms(countTerms(docs(danmaku), tokenizer), options.terms ?? 20),
+    };
+}