omnius 1.0.885 → 1.0.887

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "omnius",
-  "version": "1.0.885",
+  "version": "1.0.887",
   "description": "AI coding agent powered by open-source models (Ollama/vLLM) — interactive TUI with agentic tool-calling loop",
   "type": "module",
   "main": "./dist/library.js",
@@ -146,13 +146,14 @@
     "viem": "2.47.4"
   },
   "optionalDependencies": {
-    "moondream": "^0.2.0",
+    "moondream": "file:vendor/moondream",
     "neovim": "^5.4.0",
-    "node-pty": "^1.1.0",
+    "node-pty": "npm:@lydell/node-pty@1.2.0-beta.15",
     "systray2": "2.1.4",
     "transcribe-cli": "^2.0.1",
     "viem": "2.47.4",
-    "cli-highlight": "^2.1.11"
+    "cli-highlight": "^2.1.11",
+    "sharp": "0.35.5"
   },
   "readme": "# Omnius\n\nOmnius is a local-first runtime for AI-assisted software development and generative work. It brings a terminal interface, browser dashboard, automation API, agent execution, memory, and media tools into one workspace-oriented system.\n\nOmnius connects to local and remote model providers. Projects retain their own sessions and generated artifacts, while model weights and runtime dependencies are shared across workspaces.\n\n[![npm](https://img.shields.io/npm/v/omnius.svg)](https://www.npmjs.com/package/omnius)\n[![Node](https://img.shields.io/badge/node-%3E%3D22-brightgreen.svg)](https://nodejs.org/)\n[![License](https://img.shields.io/badge/license-CC--BY--NC--4.0-blue.svg)](https://creativecommons.org/licenses/by-nc/4.0/)\n\n## Get started\n\nRequires Node.js 22 or newer and npm 10 or newer.\n\n```bash\nnpm install -g omnius\nomnius\n```\n\nUse `/setup` to configure inference and `/help` to explore terminal commands. See [installation](docs/getting-started/install.md), [first run](docs/getting-started/first-run.md), and [model providers](docs/getting-started/model-providers.md) for setup details.\n\nTo open the browser dashboard and API:\n\n```bash\nomnius serve\n```\n\nThe default dashboard address is `http://127.0.0.1:11435/`, with interactive API documentation at `/docs`.\n\n## System overview\n\n| Layer | Responsibility |\n| --- | --- |\n| Interfaces | Terminal sessions, browser workspaces, command-line automation, and system tray integration |\n| Agent runtime | Task execution, tool use, delegated agents, verification, and resumable work |\n| Inference | Model selection and routing across local servers, configured providers, and peers |\n| Tools | Files, code, shell commands, web access, browser automation, and extensible capabilities |\n| Memory and retrieval | Project context, indexed material, persistent memory, and targeted retrieval |\n| Media and voice | Image, video, audio, music, speech recognition, and speech synthesis |\n| Integration | REST APIs, events, MCP, Telegram, and shared inference services |\n\nThe [architecture overview](docs/architecture/overview.md) and [agent system map](docs/architecture/agent-system-map.md) describe the runtime boundaries and state ownership.\n\n## Workspaces and sessions\n\nA project provides the working directory for files, tools, conversations, and generated media. Terminal and browser interfaces expose saved sessions and project artifacts. Operational details remain available alongside the conversation without replacing its authored content.\n\nProject state lives under `.omnius/`. Shared model weights and runtime dependencies are managed separately from project-generated media. See [terminal workflows](docs/guides/tui-workflows.md), the [dashboard guide](docs/guides/dashboard.md), and [media generation](docs/guides/media-generation.md).\n\n## Automation and discovery\n\nOmnius can run a task directly from the command line:\n\n```bash\nomnius \"inspect this repository and explain its main entrypoints\"\n```\n\nAgents and integrations can discover capabilities through the CLI or daemon:\n\n```bash\nomnius discover \"bring your own inference\"\nomnius show layer.orchestration\nomnius capabilities --json\n```\n\nThe daemon provides stateful chat, agent execution, tools, memory, media, and OpenAI/Ollama-compatible inference interfaces. Begin with [agent integration](docs/guides/agent-integration.md) and the [discovery guide](docs/DISCOVERY.md).\n\n<!-- BEGIN GENERATED REST INVENTORY -->\nThe complete API inventory is maintained in the [REST reference](docs/reference/rest-api.md). The running daemon serves its machine-readable contract at `/openapi.json`.\n<!-- END GENERATED REST INVENTORY -->\n\n## Documentation\n\n| Topic | Guide |\n| --- | --- |\n| Inference configuration | [Provider protocols and keys](docs/guides/bring-your-own-inference.md) |\n| Commands and tools | [Slash commands](docs/reference/slash-commands.md), [tool discovery](docs/guides/tools-and-web-search.md) |\n| Agent coordination | [Adjudication](docs/ADJUDICATION.md), [agent system map](docs/architecture/agent-system-map.md) |\n| Browser interface | [Dashboard](docs/guides/dashboard.md) |\n| Voice and media | [Realtime voice](docs/guides/realtime.md), [media generation](docs/guides/media-generation.md), [Omni analysis](docs/reference/omni-analysis.md) |\n| Connected services | [Telegram](docs/guides/telegram.md), [system tray](docs/guides/system-tray.md), [sponsor and peer inference](docs/guides/sponsor-and-cohere.md) |\n| Runtime administration | [Runtime hygiene](docs/operations/runtime-hygiene.md), [access control](docs/operations/security-and-remote-access.md) |\n| Agent-readable documentation | [Documentation index](docs/agent-memory/INDEX.md) |\n\n## Development\n\nThe repository is a pnpm workspace. `packages/cli` contains the interfaces and daemon; `packages/orchestrator` coordinates agent execution; `packages/execution` provides tools. Supporting packages cover memory, indexing, retrieval, schemas, prompts, and backend integration.\n\nRequires pnpm 9 or newer.\n\n```bash\npnpm install\npnpm -r build\npnpm docs:check\n```\n\nPublishing uses the generated `publish/` package. The build, verification, and release procedure is documented in [AGENTS.md](AGENTS.md).\n\n## License\n\nOmnius is released under [CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/) for non-commercial use. Commercial use, redistribution, hosted services, and enterprise deployment require a commercial license.\n"
 }

package/vendor/moondream/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) M87 Labs
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/vendor/moondream/README.md

@@ -0,0 +1,11 @@
+# Omnius Moondream client distribution
+
+This package preserves the runtime and public types of [moondream 0.2.0](https://github.com/m87-labs/moondream-node), published by M87 Labs <contact@moondream.ai> under the MIT license.
+
+Upstream npm tarball: `moondream@0.2.0`
+
+Integrity: `sha512-8w/4uyHqHnkNhnQ0xo1eS+TZfRygPXKNVK09Gw+7Zk6zAkQaGX77y+JMXjbtgst9g+3Bwa8sLBEdVIzwTvhkHw==`
+
+The runtime and declarations come from the upstream tarball's `dist/src/`. The sole runtime change adjusts its package-version import for this directory layout. The manifest pins Sharp 0.35.5, which supplies prebuilt native binaries through npm without an install lifecycle or GitHub libvips download. Development tooling and upstream integration tests are excluded.
+
+The publish builder copies this directory to `publish/vendor/moondream` and declares its Sharp dependency directly as well: npm global tarball installations can skip dependencies of local directory links. No inference protocol or SDK methods are changed.

package/vendor/moondream/index.cjs

@@ -0,0 +1,292 @@
+"use strict";
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.vl = void 0;
+const buffer_1 = require("buffer");
+const sharp_1 = __importDefault(require("sharp"));
+const http_1 = __importDefault(require("http"));
+const https_1 = __importDefault(require("https"));
+const package_json_1 = require("./package.json");
+const DEFAULT_ENDPOINT = 'https://api.moondream.ai/v1';
+class vl {
+    constructor(config) {
+        this.apiKey = config.apiKey || '';
+        this.endpoint = config.endpoint || DEFAULT_ENDPOINT;
+        if (this.apiKey === '' && this.endpoint === DEFAULT_ENDPOINT) {
+            throw new Error('An apiKey is required for cloud inference. ');
+        }
+    }
+    async encodeImage(image) {
+        if ('imageUrl' in image) {
+            return image;
+        }
+        try {
+            // Process image with Sharp
+            const metadata = await (0, sharp_1.default)(image).metadata();
+            if (!metadata.width || !metadata.height) {
+                throw new Error('Unable to get image dimensions');
+            }
+            let processedImage = (0, sharp_1.default)(image);
+            const buffer = await processedImage
+                .toFormat('jpeg', { quality: 95 })
+                .toBuffer();
+            const base64Image = buffer.toString('base64');
+            return {
+                imageUrl: `data:image/jpeg;base64,${base64Image}`,
+            };
+        }
+        catch (error) {
+            throw new Error(`Failed to convert image to JPEG: ${error.message}`);
+        }
+    }
+    makeRequest(path, body, stream = false) {
+        return new Promise((resolve, reject) => {
+            const url = new URL(this.endpoint + path);
+            const requestBody = JSON.stringify(body);
+            const options = {
+                method: 'POST',
+                headers: {
+                    'X-Moondream-Auth': this.apiKey,
+                    'Content-Type': 'application/json',
+                    'User-Agent': `moondream-node/${package_json_1.version}`,
+                    'Content-Length': buffer_1.Buffer.byteLength(requestBody)
+                }
+            };
+            const client = url.protocol === 'https:' ? https_1.default : http_1.default;
+            const req = client.request(url, options, (res) => {
+                if (stream) {
+                    resolve(res);
+                    return;
+                }
+                let data = '';
+                res.on('data', chunk => data += chunk);
+                res.on('end', () => {
+                    if (res.statusCode !== 200) {
+                        reject(new Error(`HTTP error! status: ${res.statusCode}`));
+                        return;
+                    }
+                    try {
+                        resolve(JSON.parse(data));
+                    }
+                    catch (error) {
+                        reject(new Error(`Failed to parse JSON response: ${error.message}`));
+                    }
+                });
+            });
+            req.on('error', (error) => {
+                reject(error);
+            });
+            req.write(requestBody);
+            req.end();
+        });
+    }
+    async *streamResponse(response) {
+        let buffer = '';
+        try {
+            for await (const chunk of response) {
+                buffer += chunk.toString();
+                const lines = buffer.split('\n');
+                buffer = lines.pop() || '';
+                for (const line of lines) {
+                    if (line.startsWith('data: ')) {
+                        try {
+                            const data = JSON.parse(line.slice(6));
+                            if ('chunk' in data) {
+                                yield data.chunk;
+                            }
+                            if (data.completed) {
+                                return;
+                            }
+                        }
+                        catch (error) {
+                            throw new Error(`Failed to parse JSON response from server: ${error.message}`);
+                        }
+                    }
+                }
+            }
+            // Handle any remaining data in the buffer
+            if (buffer) {
+                const lines = buffer.split('\n');
+                for (const line of lines) {
+                    if (line.startsWith('data: ')) {
+                        try {
+                            const data = JSON.parse(line.slice(6));
+                            if ('chunk' in data) {
+                                yield data.chunk;
+                            }
+                        }
+                        catch (error) {
+                            throw new Error(`Failed to parse JSON response from server: ${error.message}`);
+                        }
+                    }
+                }
+            }
+        }
+        catch (error) {
+            throw new Error(`Failed to stream response: ${error.message}`);
+        }
+    }
+    async caption(request) {
+        const encodedImage = await this.encodeImage(request.image);
+        const requestBody = {
+            image_url: encodedImage.imageUrl,
+            length: request.length,
+            stream: request.stream,
+        };
+        if (request.variant) {
+            requestBody.variant = request.variant;
+        }
+        const response = await this.makeRequest('/caption', requestBody, request.stream);
+        if (request.stream) {
+            return { caption: this.streamResponse(response) };
+        }
+        return { caption: response.caption };
+    }
+    async query(request) {
+        let requestBody = {
+            question: request.question,
+            stream: request.stream,
+        };
+        if (request.image) {
+            const encodedImage = await this.encodeImage(request.image);
+            requestBody.image_url = encodedImage.imageUrl;
+        }
+        if (request.reasoning !== undefined) {
+            requestBody.reasoning = request.reasoning;
+        }
+        if (request.variant) {
+            requestBody.variant = request.variant;
+        }
+        const response = await this.makeRequest('/query', requestBody, request.stream);
+        if (request.stream) {
+            return { answer: this.streamResponse(response) };
+        }
+        const result = { answer: response.answer };
+        if (response.reasoning) {
+            result.reasoning = response.reasoning;
+        }
+        return result;
+    }
+    async detect(request) {
+        const encodedImage = await this.encodeImage(request.image);
+        const requestBody = {
+            image_url: encodedImage.imageUrl,
+            object: request.object,
+        };
+        if (request.variant) {
+            requestBody.variant = request.variant;
+        }
+        const response = await this.makeRequest('/detect', requestBody);
+        return { objects: response.objects };
+    }
+    async point(request) {
+        const encodedImage = await this.encodeImage(request.image);
+        const requestBody = {
+            image_url: encodedImage.imageUrl,
+            object: request.object,
+        };
+        if (request.variant) {
+            requestBody.variant = request.variant;
+        }
+        const response = await this.makeRequest('/point', requestBody);
+        return { points: response.points };
+    }
+    async *streamSegmentResponse(response) {
+        let buffer = '';
+        try {
+            for await (const chunk of response) {
+                buffer += chunk.toString();
+                const lines = buffer.split('\n');
+                buffer = lines.pop() || '';
+                for (const line of lines) {
+                    if (line.startsWith('data: ')) {
+                        try {
+                            const data = JSON.parse(line.slice(6));
+                            const msgType = data.type || '';
+                            if (msgType === 'bbox') {
+                                yield { bbox: data.bbox };
+                            }
+                            else if (msgType === 'path_delta') {
+                                if (data.chunk) {
+                                    yield { chunk: data.chunk };
+                                }
+                            }
+                            else if (msgType === 'final') {
+                                yield {
+                                    path: data.path || '',
+                                    bbox: data.bbox,
+                                    completed: true,
+                                };
+                                return;
+                            }
+                        }
+                        catch (error) {
+                            throw new Error(`Failed to parse JSON response from server: ${error.message}`);
+                        }
+                    }
+                }
+            }
+            // Handle any remaining data in the buffer
+            if (buffer) {
+                const lines = buffer.split('\n');
+                for (const line of lines) {
+                    if (line.startsWith('data: ')) {
+                        try {
+                            const data = JSON.parse(line.slice(6));
+                            const msgType = data.type || '';
+                            if (msgType === 'bbox') {
+                                yield { bbox: data.bbox };
+                            }
+                            else if (msgType === 'path_delta') {
+                                if (data.chunk) {
+                                    yield { chunk: data.chunk };
+                                }
+                            }
+                            else if (msgType === 'final') {
+                                yield {
+                                    path: data.path || '',
+                                    bbox: data.bbox,
+                                    completed: true,
+                                };
+                            }
+                        }
+                        catch (error) {
+                            throw new Error(`Failed to parse JSON response from server: ${error.message}`);
+                        }
+                    }
+                }
+            }
+        }
+        catch (error) {
+            throw new Error(`Failed to stream response: ${error.message}`);
+        }
+    }
+    async segment(request) {
+        const encodedImage = await this.encodeImage(request.image);
+        const requestBody = {
+            image_url: encodedImage.imageUrl,
+            object: request.object,
+            stream: request.stream || false,
+        };
+        if (request.spatialRefs) {
+            requestBody.spatial_refs = request.spatialRefs;
+        }
+        if (request.settings) {
+            requestBody.settings = request.settings;
+        }
+        if (request.variant) {
+            requestBody.variant = request.variant;
+        }
+        const response = await this.makeRequest('/segment', requestBody, request.stream);
+        if (request.stream) {
+            return { stream: this.streamSegmentResponse(response) };
+        }
+        return {
+            path: response.path,
+            bbox: response.bbox,
+        };
+    }
+}
+exports.vl = vl;

package/vendor/moondream/index.d.ts

@@ -0,0 +1,19 @@
+import { CaptionOutput, QueryOutput, DetectOutput, PointOutput, SegmentOutput, SegmentStreamOutput, CaptionRequest, QueryRequest, DetectRequest, PointRequest, SegmentRequest } from './types';
+export interface MoondreamVLConfig {
+    apiKey?: string;
+    endpoint?: string;
+}
+export declare class vl {
+    private apiKey;
+    private endpoint;
+    constructor(config: MoondreamVLConfig);
+    private encodeImage;
+    private makeRequest;
+    private streamResponse;
+    caption(request: CaptionRequest): Promise<CaptionOutput>;
+    query(request: QueryRequest): Promise<QueryOutput>;
+    detect(request: DetectRequest): Promise<DetectOutput>;
+    point(request: PointRequest): Promise<PointOutput>;
+    private streamSegmentResponse;
+    segment(request: SegmentRequest): Promise<SegmentOutput | SegmentStreamOutput>;
+}

package/vendor/moondream/package.json

@@ -0,0 +1,26 @@
+{
+  "name": "moondream",
+  "version": "0.2.0-omnius.1",
+  "description": "Moondream 0.2.0 client with Omnius' compiler-free Sharp dependency.",
+  "main": "index.cjs",
+  "types": "index.d.ts",
+  "files": [
+    "index.cjs",
+    "index.d.ts",
+    "types.d.ts",
+    "README.md",
+    "LICENSE"
+  ],
+  "author": "M87 Labs <contact@moondream.ai>",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/m87-labs/moondream-node"
+  },
+  "engines": {
+    "node": ">=22.0.0"
+  },
+  "dependencies": {
+    "sharp": "0.35.5"
+  }
+}

package/vendor/moondream/types.d.ts

@@ -0,0 +1,208 @@
+/// <reference types="node" />
+/// <reference types="node" />
+/**
+ * Base interface for encoded images
+ */
+export interface Base64EncodedImage {
+    imageUrl: string;
+}
+/**
+ * Length options for caption generation
+ */
+export type Length = "normal" | "short" | "long";
+/**
+ * Settings for controlling the model's generation behavior
+ */
+export interface SamplingSettings {
+    maxTokens?: number;
+}
+/**
+ * Base request interface with common properties
+ */
+export interface BaseRequest {
+    variant?: string;
+    settings?: SamplingSettings;
+}
+/**
+ * Request structure for image caption requests
+ */
+export interface CaptionRequest extends BaseRequest {
+    image: Buffer | Base64EncodedImage;
+    length?: Length;
+    stream?: boolean;
+}
+/**
+ * Response structure for image caption requests
+ */
+export interface CaptionOutput {
+    caption: string | AsyncGenerator<string, void, unknown>;
+}
+/**
+ * Request structure for image query requests
+ */
+export interface QueryRequest extends BaseRequest {
+    image?: Buffer | Base64EncodedImage;
+    question: string;
+    reasoning?: boolean;
+    stream?: boolean;
+}
+/**
+ * Reasoning grounding structure
+ */
+export interface ReasoningGrounding {
+    start_idx: number;
+    end_idx: number;
+    points: [number, number][];
+}
+/**
+ * Reasoning structure
+ */
+export interface Reasoning {
+    text: string;
+    grounding: ReasoningGrounding[];
+}
+/**
+ * Response structure for image query requests
+ */
+export interface QueryOutput {
+    answer: string | AsyncGenerator<string, void, unknown>;
+    reasoning?: Reasoning;
+}
+/**
+ * Request structure for object detection requests
+ */
+export interface DetectRequest extends BaseRequest {
+    image: Buffer | Base64EncodedImage;
+    object: string;
+}
+/**
+ * Response structure for object detection requests
+ */
+export interface DetectOutput {
+    objects: DetectedObject[];
+}
+/**
+ * Object detection result
+ */
+export interface DetectedObject {
+    x_min: number;
+    y_min: number;
+    x_max: number;
+    y_max: number;
+}
+/**
+ * Response structure for object detection requests
+ */
+export interface DetectOutput {
+    objects: DetectedObject[];
+}
+/**
+ * Error response from the API
+ */
+export interface ApiError {
+    error: {
+        message: string;
+        code?: string;
+        details?: unknown;
+    };
+}
+/**
+ * Configuration options for the client
+ */
+export interface ClientConfig {
+    apiKey: string;
+    endpoint?: string;
+    timeout?: number;
+    retries?: number;
+}
+/**
+ * API response for streaming requests
+ */
+export interface StreamResponse {
+    chunk?: string;
+    completed?: boolean;
+    error?: string;
+}
+/**
+ * Options for image processing
+ */
+export interface ImageProcessingOptions {
+    maxSize?: number;
+    quality?: number;
+    format?: "jpeg" | "png" | "gif";
+}
+/**
+ * Common response type for all API endpoints
+ */
+export type ApiResponse<T> = {
+    success: boolean;
+    data?: T;
+    error?: ApiError;
+    timestamp?: string;
+    requestId?: string;
+};
+/**
+ * Pointing request structure
+ */
+export interface PointRequest extends BaseRequest {
+    image: Buffer | Base64EncodedImage;
+    object: string;
+}
+/**
+ * Point coordinates for object location
+ */
+export interface Point {
+    x: number;
+    y: number;
+}
+/**
+ * Response structure for point requests
+ */
+export interface PointOutput {
+    points: Point[];
+}
+/**
+ * Spatial reference for segmentation - either a point [x, y] or bbox [x1, y1, x2, y2]
+ * Values are normalized to [0, 1]
+ */
+export type SpatialRef = [number, number] | [number, number, number, number];
+/**
+ * Request structure for segmentation requests
+ */
+export interface SegmentRequest extends BaseRequest {
+    image: Buffer | Base64EncodedImage;
+    object: string;
+    spatialRefs?: SpatialRef[];
+    stream?: boolean;
+}
+/**
+ * Bounding box for segmentation result
+ */
+export interface SegmentBbox {
+    x_min: number;
+    y_min: number;
+    x_max: number;
+    y_max: number;
+}
+/**
+ * Response structure for non-streaming segmentation requests
+ */
+export interface SegmentOutput {
+    path: string;
+    bbox?: SegmentBbox;
+}
+/**
+ * Streaming update for segmentation
+ */
+export interface SegmentStreamChunk {
+    bbox?: SegmentBbox;
+    chunk?: string;
+    path?: string;
+    completed?: boolean;
+}
+/**
+ * Response structure for streaming segmentation requests
+ */
+export interface SegmentStreamOutput {
+    stream: AsyncGenerator<SegmentStreamChunk, void, unknown>;
+}