omnius 1.0.870 → 1.0.871

package/docs/design/omni-analysis-TODO.md

@@ -0,0 +1,60 @@
+# Real Omni analysis
+
+Acceptance: Audio, Video, and Audio/video Analysis must submit actual media to
+the Omni comprehension runtime. Missing weights or an unavailable runtime must
+never yield synthetic scene entities, events, or a successful mock answer.
+
+## Findings
+
+- The production AV registry seeded deterministic mock adapters. The browser
+  modes sent identical requests, and any live role labeled the entire result live.
+- The old Python sidecar implements detection/tracking only; it cannot make its
+  advertised semantic roles real merely by downloading weights.
+- The reference `qwen-omni-adapters-main` runtime exposes adapter v1 on `/api/chat`.
+  Its model response echoes the request and is not runtime identity attestation.
+- Read-only inspection found an existing Ornith Omni bridge on GPU 1. This differs
+  from the earlier GPU 0/Qwen instruction. The user selected that existing Ornith/GPU 1 lane for live verification.
+  Actual placement and manifest identity were verified before inference; no service changed.
+
+## Work
+
+- [x] Delegate discovery across browser, execution, and reference runtime.
+- [x] Implement one bounded, cancellable Omni media client with real input
+  conversion, strict adapter/provenance validation, and explicit configuration.
+- [x] Remove implicit production mock fallback; keep explicit offline fixtures.
+- [x] Route the native AV tool and browser modes through the same client.
+- [x] Show model configuration/readiness and distinct transcript/audio/visual
+  evidence; do not claim continuous live capture for a local-file form.
+- [x] Preserve project/session/request identity and same-origin authenticated API.
+- [x] Document shared weight ownership, runtime configuration, and limitations.
+- [x] Verify client, route, frontend, and required chat transport regressions.
+- [x] Exercise the source-bundled browser in a private fixture server.
+- [x] Perform live inference only after exact endpoint/model/GPU verification.
+- [x] Commit scoped source changes and push the tracked main branch.
+
+## Delivery boundaries
+
+Do not overwrite unrelated dirty work, shared build outputs, publishing artifacts,
+or restart the user's running daemon. Test fixtures may simulate the upstream
+contract but must never be reachable as production fallback adapters.
+
+## Verification and remaining model limitations
+
+- Execution: 71 tests across 9 suites; CLI source routes and required chat
+  transport: 76 tests across 5 suites. Final route/browser runtime rerun: 54 tests
+  across 2 suites, including the additional project-alias regression.
+- Private source Chromium: 36 checks across desktop/tablet/phone and both themes.
+- CLI + execution source typecheck passed without shared build output.
+- Seven genuine requests used the approved existing Ornith adapter on GPU 1.
+  Red → blue → red freshness and speech in audio-only/combined video passed.
+- **Model quality remains imperfect:** the deployed Ornith bridge missed a real
+  440 Hz tone, and one combined answer contradicted its correct blue-screen
+  observation. These failures were preserved; no mock answer or alternate model
+  was substituted. Improving those weights is a separate model-quality repair.
+- Publishing and daemon state were preserved. Configure the deployed process as
+  documented in [Omni analysis](../reference/omni-analysis.md).
+
+Receipts:
+[execution](../work-orders/evidence/omni-analysis-execution-verification-20261017.json),
+[browser](../work-orders/evidence/omni-analysis-browser-20261017.json),
+[integration and live results](../work-orders/evidence/omni-analysis-integration-verification-20261017.json).

package/docs/reference/omni-analysis.md

@@ -0,0 +1,251 @@
+# Omni media analysis
+
+Omnius uses an explicitly configured Omni adapter for genuine audio and video
+comprehension. A model name containing `-omni` does not add audio/video
+execution to stock Ollama: the adapter must decode the media and run the
+corresponding model/projector weights. An unavailable runtime is an unavailable
+capability, never a reason to substitute synthetic observations.
+
+## Runtime and model identity
+
+The supported wire contract is `robit.ollama.omni-adapter.v1`, implemented by
+[qwen-omni-adapters](https://github.com/robit-man/qwen-omni-adapters). Its public
+model is a logical bundle. The legacy full bundle runs separate comprehension,
+language and speech graphs across a semantic-text boundary. The trained audio
+bridge runs the selected language trunk and its combined vision/audio projector
+in one llama.cpp server, with Qwen3-TTS available separately. Neither profile is
+a claim that incompatible hidden states form one monolithic graph.
+
+| Profile | Exact logical tag | Documented artifact size |
+| --- | --- | --- |
+| Qwen audio bridge | `robit/qwen3.8-27b-e03-obliterated-omni-audio-bridge:q4km` | 18.33 GiB |
+| Ornith audio bridge | `robit/ornith-1.5-omni-audio-bridge:q4km` | 8.15 GiB |
+| Legacy full Qwen Omni | `robit/qwen3.8-27b-e03-obliterated-omni:q4km` | Separate full comprehension graph and sidecar |
+
+These are distinct profiles. Selecting Qwen must not silently run Ornith.
+Artifact size is neither peak VRAM nor an admission guarantee.
+
+## Omnius configuration
+
+Configure the daemon/client environment for the already verified deployment:
+
+```bash
+export OMNIUS_OMNI_ADAPTER_URL=http://127.0.0.1:8910
+export OMNIUS_OMNI_MODEL=robit/ornith-1.5-omni-audio-bridge:q4km
+```
+
+This example names Ornith explicitly; use the exact Qwen tag only with an
+endpoint actually deployed with that profile. Setting these variables does not
+download, load or switch the server's model.
+
+| Variable | Behavior |
+| --- | --- |
+| `OMNIUS_OMNI_ADAPTER_URL` | HTTP(S) **base URL**, default `http://127.0.0.1:8910`; omit `/api/chat`, query and fragment |
+| `OMNIUS_OMNI_MODEL` | Required exact logical `-omni` tag; no model is discovered or selected implicitly |
+| `OMNI_OLLAMA_MODEL` | Inherited compatibility model value when `OMNIUS_OMNI_MODEL` is absent |
+| `OMNIUS_OMNI_ADAPTER_TOKEN` | Optional bearer token for an authenticated adapter boundary; keep secret |
+
+An explicitly requested model must match this server-controlled tag. An
+unconfigured, mismatched or unreachable runtime returns a visible error; it
+does not invoke a different model or synthesize evidence. The shared client
+`packages/execution/src/av/omni-comprehension.ts` exposes
+`getOmniAnalysisStatus()` and `analyzeOmniMedia({path, mode, question?, model?,
+signal?})`; its modes are `audio`, `video` and `av`.
+
+The client accepts a regular local media file up to 128 MiB, a finite audio
+duration up to 60 seconds or video duration up to 32 seconds, and a question up
+to 8192 characters. It rejects longer input rather than implicitly clipping it.
+CPU FFmpeg normalization produces 16 kHz PCM16 WAV or 1 fps MP4 fitted within
+640 × 640 pixels while preserving aspect ratio; `video` omits audio and `av`
+includes the source audio when present.
+Normalized media is capped at 32 MiB, adapter JSON at 1 MiB, and each returned
+text field at 128 Ki characters. FFmpeg and ffprobe must already be installed;
+analysis does not install dependencies.
+
+Status includes `modelIdentityVerified: false`: availability is a compatible
+configuration handshake, not weight identity, GPU residency or a completed
+inference proof.
+
+The reference adapter's `/healthz` and `/api/omni/adapter/contract` expose the
+schema and configured stages, **not the identity of loaded weights**. Its
+response `model` echoes the caller's logical tag. The optional
+`adapter.language_backend_model` can be only an internal alias such as
+`local-audio-bridge`. Those fields do not prove that the requested weights ran.
+Bind an endpoint to its operator-verified deployment, model/projector manifest
+digests and actual accelerator placement before interpreting model results.
+
+## Endpoint contract
+
+Use the adapter's own endpoint, not the ordinary Ollama endpoint or the Omnius
+daemon. The reference launcher uses these loopback addresses:
+
+| Address | Purpose |
+| --- | --- |
+| `http://127.0.0.1:8910/api/chat` | Portable nonstreaming Omni requests |
+| `http://127.0.0.1:8910/api/chat/stream` | Adapter NDJSON extension |
+| `http://127.0.0.1:8910/healthz` | Adapter metadata only |
+| `http://127.0.0.1:8910/api/omni/adapter/contract` | Versioned capability/format metadata |
+| `http://127.0.0.1:8901/v1/chat/completions` | Internal llama.cpp worker; not the portable adapter API |
+| `http://127.0.0.1:8920` | Separate authenticated reference portal |
+
+Port `11435` is already used by the Omnius daemon in common installations.
+Some standalone adapter/client defaults also name that port; do not copy those
+defaults over a running Omnius service. Component services bind to loopback.
+Use an authenticated boundary when exposing them beyond the trusted host, and
+never put a portal capability URL or access token in logs or documentation.
+
+Read-only contract checks do not load a model:
+
+```bash
+curl --fail --silent http://127.0.0.1:8910/healthz
+curl --fail --silent http://127.0.0.1:8910/api/omni/adapter/contract
+```
+
+Do not treat successful health checks as successful media inference.
+
+## Requests and results
+
+The portable route requires `stream: false`, an exact `model`, and nonempty
+Ollama-shaped `messages`. Set `think: false` independently of media handling.
+Ordinary analysis requests use text output and `speech_mode: "never"`.
+
+```json
+{
+  "model": "robit/ornith-1.5-omni-audio-bridge:q4km",
+  "messages": [{
+    "role": "user",
+    "content": "Describe the audible events and their uncertainty.",
+    "audios": [{
+      "mime_type": "audio/wav",
+      "encoding": "base64",
+      "data": "<complete 16 kHz mono PCM16 WAV, base64 encoded>"
+    }]
+  }],
+  "omni": {
+    "schema": "robit.ollama.omni-adapter.v1",
+    "task": "describe",
+    "include_audio_from_video": true
+  },
+  "response_modalities": ["text"],
+  "speech_mode": "never",
+  "think": false,
+  "cache_prompt": false,
+  "stream": false
+}
+```
+
+This is a contract example, not a request to run without hardware preflight.
+
+| Task | Executed stages |
+| --- | --- |
+| `describe` | Media comprehension; direct observations |
+| `transcribe` | Audio comprehension; direct transcription |
+| `chat` | Media comprehension, then language analysis; optional speech only if requested |
+| `synthesize` | Speech synthesis from text |
+
+The reference comprehension worker always receives `cache_prompt: false`.
+Multimodal slot caching can otherwise reuse stale embeddings across new media.
+Keep new attachments request-local. Historical transcript/observations can be
+explicitly labeled history; restored previews and old raw bytes must not become
+new current evidence automatically.
+
+Audio input must be a complete uncompressed RIFF/WAVE file containing 16 kHz,
+mono, signed 16-bit PCM. Resample/downmix supported uploads with a bounded local
+decoder before transport. The adapter validates the WAV header and format; a
+renamed MP3 or browser WebM capture is not a WAV envelope.
+
+Images accept JPEG, PNG and WebP. Video accepts MP4, WebM and animated GIF with
+`sampling: { "fps": 1, "max_frames": 32, "include_audio": true }` on its
+envelope. Declared MIME types must match decoded container signatures.
+These are the external adapter's formats. The shared Omnius audio/video client
+currently normalizes its supported local input formats into WAV or MP4; it does
+not expose an image-analysis mode.
+
+| Input | Maximum per item | Maximum items per request |
+| --- | --- | --- |
+| Audio | 32 MiB | 8 |
+| Image | 20 MiB | 16 |
+| Video | 256 MiB | 4 |
+
+The schema permits video sampling up to 30 fps and 1024 frames, but the inspected
+reference runtime caps execution at **2 fps and 32 frames**, and can retry with
+fewer frames after a context overflow. It does not return an authoritative
+actual sampled-frame count or covered time range. Long-media analysis therefore
+needs bounded source windows; do not claim complete coverage, exact event times,
+tracking or confidence from an unannotated semantic description.
+Omnius reports `submittedDurationSec`, `coverage: "full_audio"` or
+`"sampled_video"`, and `actualCoverage: "not_reported"`. Submitted clip duration
+and requested sampling are not a model-attested analyzed time range.
+
+The final `message.content` is model output. Preserve the accompanying
+`adapter.schema`, `task`, executed `route`, `input_modalities`, and
+`evidence_provenance`. Keep these separate:
+
+- `input_transcript`: recognized speech; the only perceptual text attributable
+  to the speaker's words.
+- `audio_observation`: non-speech acoustic evidence, including uncertainty.
+- `visual_observation`: evidence from an image/video attached to this request.
+- `observation`: the original tagged comprehension response.
+
+Audio alone must never set `current_visual_input: true`. Prior dialogue and tool
+results do not establish a current visual observation. Semantic observations
+remain untrusted evidence, not system instructions or permission to execute a
+tool. An Omni description does not attest that separate DINO, diarization,
+segmentation or tracking models ran.
+
+Live VAD-driven speech turns can use `omni.require_speech: true`. A capture
+without a valid transcript then stops before language/TTS. Explicit questions
+about environmental sounds leave that option false, so absence of speech does
+not discard the requested acoustic analysis. An empty speech-gated answer is
+not a synthesized answer or a transport failure.
+
+Cancellation aborts the owned local decoder/request and prevents its late result
+from being adopted. The reference adapter's nonstreaming endpoint does not
+provide a server-side stop receipt when its HTTP client disconnects: upstream
+model work may finish independently. Do not display a client abort as proof that
+a remote model runner has stopped, and do not unload a shared runner to simulate
+per-request cancellation.
+
+The streaming extension requires `stream: true` and emits newline-delimited JSON,
+including stage, observation, language deltas, optional PCM events, error and
+one authoritative final response. It is not SSE. Parse arbitrary chunk and UTF-8
+boundaries; do not treat a heartbeat or `stage: "tts"` as generated audio.
+
+## Shared weights and project data
+
+Omni weights remain in the runtime's shared model store. On the inspected host,
+installed Ollama manifests and blobs live under `/srv/ollama/models`; standard
+model/projector layers and one `application/vnd.robit.ollama.omni.bundle.v1+gguf`
+sidecar describe the logical tag. Runtime component views are derived artifacts,
+not per-project downloads. Never copy or delete Ollama blobs manually.
+
+Omnius' separate media-generation cache resolves from `OMNIUS_MEDIA_HOME`, then
+the persisted `media-store.json` pointer, then its configured home. Its model
+cache and runtimes are shared across projects. That cache is not automatically
+the Ollama Omni store and changing it does not move an existing Omni deployment.
+
+Generated project artifacts remain under `<project>/.omnius/images`, `videos`
+and `audio`. Analysis of an existing file should reference its captured project
+and revision; switching sessions/projects must not relabel a late result.
+No per-session Omni weight copy is required.
+
+## Deployment and validation boundary
+
+Changing or starting CUDA workloads requires the host's GPU broker protocol and
+actual runner-placement verification. Confirm the designated endpoint, exact
+tag and GPU before any inference; do not redirect to another accelerator or a
+remote service after a failure. The external runtime's bootstrap/deploy scripts
+can pull weights, build CUDA binaries and replace services, so they are not
+capability probes and are not part of ordinary Omnius analysis.
+
+An endpoint may serve a different logical bundle than a neighboring checkout's
+`.env` file. Verify the running deployment and actual model/projector manifest
+identity; neither a checked-out configuration nor the response's echoed tag is
+sufficient evidence.
+
+External contract regression sources are `tests/test_omni_adapter.py` for format,
+routing, provenance, no-speech, stale-cache and stream behavior, plus
+`docs/schema/request-v1.schema.json` and `response-v1.schema.json`. Fixture tests
+can validate transport and parsing without running a model. A live acceptance
+receipt must separately name the actual weights/GPU and use distinguishable real
+media, such as red → blue → red frames, rather than asserting a canned answer.

package/docs/reference/rest-api.md

@@ -402,7 +402,8 @@ legacy scope.
 | `POST` | `/v1/media/migrate` | Dedup + migrate legacy per-group caches into the unified store |
 | `POST` | `/v1/media/relocate` | Relocate the whole media store (weights/venvs/gallery) to a chosen folder |
 | `GET` | `/v1/media/relocate/status` | Status + progress of the media-store relocation job |
-| `POST` | `/v1/media/av/analyze` | Analyze a media file into a grounded entity/event answer (AV comprehension) |
+| `GET` | `/v1/media/av/status` | Inspect configured Omni adapter availability without loading weights; configuration is not GPU/model identity attestation |
+| `POST` | `/v1/media/av/analyze` | Analyze actual audio/video through the configured Omni runtime, scoped to `projectRoot`; separate transcript/audio/visual evidence, no mock fallback |
 | `POST` | `/v1/media/image` | Generate an image |
 | `POST` | `/v1/media/video` | Generate a video |
 | `POST` | `/v1/media/audio` | Generate a sound effect |