insika 0.9.0 → 1.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +237 -2
- data/README.md +5 -3
- data/bin/insika +3 -0
- data/bin/insika-router +0 -1
- data/docs/AGENTS.md +90 -7
- data/docs/ARCHITECTURE.md +12 -7
- data/docs/ARTIFACTS.md +33 -3
- data/docs/BENCHMARK.md +50 -7
- data/docs/CONTEXT.md +63 -0
- data/docs/CROSS-HARNESS-BENCH.md +157 -0
- data/docs/DEPLOY.md +57 -11
- data/docs/EMBEDDING.md +10 -0
- data/docs/EVALS.md +33 -4
- data/docs/FACTS.md +8 -6
- data/docs/HARVEST.md +8 -3
- data/docs/KNOWLEDGE.md +7 -0
- data/docs/MEDIA.md +6 -2
- data/docs/OBSERVABILITY.md +143 -22
- data/docs/PERFORMANCE.md +110 -0
- data/docs/PLUGINS.md +4 -4
- data/docs/POLICY.md +22 -13
- data/docs/RELEASING.md +5 -0
- data/docs/ROUTER.md +7 -0
- data/docs/RUBYLLM_2_MIGRATION.md +307 -0
- data/docs/SECURITY.md +7 -0
- data/docs/SHARED-CONVERSATIONS.md +40 -0
- data/docs/SKILLS.md +5 -3
- data/docs/STUDIO.md +122 -0
- data/docs/TOOLS.md +111 -9
- data/docs/operate.md +3 -0
- data/lib/insika/agent_defaults.rb +140 -0
- data/lib/insika/agent_file_store.rb +12 -0
- data/lib/insika/agent_profile.rb +78 -8
- data/lib/insika/artifact_signing.rb +11 -0
- data/lib/insika/chat_builder.rb +79 -25
- data/lib/insika/checkpoint.rb +6 -1
- data/lib/insika/checkpoint_store.rb +59 -7
- data/lib/insika/commands/agent_payload.rb +3 -3
- data/lib/insika/commands/delete_kit.rb +26 -0
- data/lib/insika/commands/gate_harvest.rb +3 -0
- data/lib/insika/commands/review_guardrail_flag.rb +38 -0
- data/lib/insika/commands/run_harvest.rb +17 -1
- data/lib/insika/commands/seed_session.rb +11 -1
- data/lib/insika/commands/send_message.rb +9 -3
- data/lib/insika/commands/update_settings.rb +23 -1
- data/lib/insika/commands/write_kit.rb +27 -0
- data/lib/insika/compaction.rb +33 -7
- data/lib/insika/config_store.rb +87 -7
- data/lib/insika/context/priority.rb +3 -0
- data/lib/insika/context/provider.rb +2 -2
- data/lib/insika/context/providers/knowledge.rb +25 -2
- data/lib/insika/context/providers/memory.rb +53 -4
- data/lib/insika/context/providers/pending_confirmation.rb +52 -0
- data/lib/insika/context/providers/prompt.rb +17 -4
- data/lib/insika/context/providers/session.rb +39 -3
- data/lib/insika/context/reranker.rb +75 -0
- data/lib/insika/distill.rb +22 -13
- data/lib/insika/doctor.rb +34 -4
- data/lib/insika/dsl/runtime.rb +36 -4
- data/lib/insika/dsl/server_boot.rb +4 -12
- data/lib/insika/dsl.rb +13 -5
- data/lib/insika/edge_limiter.rb +23 -15
- data/lib/insika/egress_guard.rb +11 -3
- data/lib/insika/env_schema.rb +33 -73
- data/lib/insika/evals/assertions.rb +171 -16
- data/lib/insika/evals/golden.rb +141 -4
- data/lib/insika/evals/judge.rb +19 -10
- data/lib/insika/evals/moderator_compare.rb +47 -0
- data/lib/insika/evals/report.rb +61 -9
- data/lib/insika/evals/runner.rb +44 -4
- data/lib/insika/evals/simulator.rb +10 -1
- data/lib/insika/evals/transport.rb +200 -0
- data/lib/insika/event_stream.rb +63 -7
- data/lib/insika/evidence.rb +29 -8
- data/lib/insika/executor.rb +668 -141
- data/lib/insika/harvest.rb +26 -14
- data/lib/insika/harvest_engine.rb +1 -0
- data/lib/insika/harvest_store.rb +12 -0
- data/lib/insika/http_client.rb +87 -4
- data/lib/insika/kits.rb +54 -0
- data/lib/insika/knowledge.rb +34 -20
- data/lib/insika/llm_configurator.rb +46 -4
- data/lib/insika/llm_provider_store.rb +1 -3
- data/lib/insika/llm_trace_store.rb +54 -0
- data/lib/insika/mcp_client.rb +27 -63
- data/lib/insika/mcp_live_tool.rb +55 -9
- data/lib/insika/mcp_store.rb +32 -1
- data/lib/insika/mcp_tool_registry.rb +69 -26
- data/lib/insika/media.rb +38 -25
- data/lib/insika/memory_store.rb +4 -3
- data/lib/insika/memory_trim.rb +60 -0
- data/lib/insika/middleware.rb +12 -3
- data/lib/insika/model_metrics_store.rb +285 -0
- data/lib/insika/model_registry.rb +110 -0
- data/lib/insika/model_resolver.rb +1 -1
- data/lib/insika/model_selection.rb +14 -21
- data/lib/insika/model_visible.rb +17 -6
- data/lib/insika/onboarding.rb +1 -0
- data/lib/insika/outcome_store.rb +1 -1
- data/lib/insika/overlay_tool_registry.rb +25 -2
- data/lib/insika/pack_importer.rb +54 -9
- data/lib/insika/packaging.rb +14 -5
- data/lib/insika/pending_action_store.rb +45 -18
- data/lib/insika/plugin/loader.rb +4 -2
- data/lib/insika/profile_source.rb +4 -0
- data/lib/insika/provider_error_classifier.rb +18 -3
- data/lib/insika/queue_policy.rb +7 -2
- data/lib/insika/refinement/proposer.rb +42 -17
- data/lib/insika/reliability.rb +12 -3
- data/lib/insika/router/app.rb +14 -5
- data/lib/insika/router/session_key.rb +35 -5
- data/lib/insika/safety/config.rb +20 -3
- data/lib/insika/safety/corpus.rb +10 -3
- data/lib/insika/safety/factory.rb +39 -1
- data/lib/insika/safety/input_guardrail.rb +21 -1
- data/lib/insika/safety/judge_moderator.rb +70 -0
- data/lib/insika/safety/output_validator.rb +29 -5
- data/lib/insika/safety/prompt_echo.rb +89 -0
- data/lib/insika/schema_guard.rb +53 -76
- data/lib/insika/server/app.rb +17 -8
- data/lib/insika/server/responses.rb +19 -1
- data/lib/insika/session_actor.rb +58 -28
- data/lib/insika/session_store.rb +159 -10
- data/lib/insika/settings_store.rb +47 -1
- data/lib/insika/shared_conversations.rb +305 -0
- data/lib/insika/skill_catalog.rb +28 -2
- data/lib/insika/skill_store.rb +6 -0
- data/lib/insika/soak/runner.rb +1 -1
- data/lib/insika/steer_injector.rb +7 -48
- data/lib/insika/store.rb +29 -1
- data/lib/insika/stores/json_value.rb +37 -0
- data/lib/insika/stores/memory.rb +22 -35
- data/lib/insika/stores/postgres.rb +295 -0
- data/lib/insika/stores/sqlite.rb +257 -43
- data/lib/insika/stores/turn_counter.rb +25 -0
- data/lib/insika/studio/app.rb +1453 -161
- data/lib/insika/studio/assets/dist/application.css +1 -1
- data/lib/insika/studio/assets/dist/application.js +25 -25
- data/lib/insika/studio/assets/dist/fonts/ibm-plex-mono-latin-400-normal.woff2 +0 -0
- data/lib/insika/studio/assets/dist/fonts/ibm-plex-mono-latin-500-normal.woff2 +0 -0
- data/lib/insika/studio/assets/dist/fonts/ibm-plex-mono-latin-600-normal.woff2 +0 -0
- data/lib/insika/studio/assets/dist/fonts/ibm-plex-sans-latin-wght-normal.woff2 +0 -0
- data/lib/insika/studio/forms.rb +171 -9
- data/lib/insika/studio/nav_icons.rb +3 -0
- data/lib/insika/studio/views/_agent_config_retrieval.erb +73 -0
- data/lib/insika/studio/views/_agent_picker.erb +15 -0
- data/lib/insika/studio/views/_agent_tab_config.erb +78 -242
- data/lib/insika/studio/views/_agent_tab_history.erb +56 -1
- data/lib/insika/studio/views/_agent_tab_performance.erb +34 -0
- data/lib/insika/studio/views/_agent_tab_prompts.erb +7 -9
- data/lib/insika/studio/views/_agent_tab_skills.erb +16 -7
- data/lib/insika/studio/views/_agent_tab_tools.erb +90 -0
- data/lib/insika/studio/views/_agents_master.erb +8 -12
- data/lib/insika/studio/views/_cfg_alerts.erb +4 -0
- data/lib/insika/studio/views/_cfg_budget.erb +20 -0
- data/lib/insika/studio/views/_cfg_distill.erb +26 -0
- data/lib/insika/studio/views/_cfg_followup.erb +30 -0
- data/lib/insika/studio/views/_cfg_grounding.erb +21 -0
- data/lib/insika/studio/views/_cfg_guardrails.erb +47 -0
- data/lib/insika/studio/views/_cfg_harvest.erb +35 -0
- data/lib/insika/studio/views/_cfg_reliability.erb +32 -0
- data/lib/insika/studio/views/_chats_master.erb +66 -0
- data/lib/insika/studio/views/_customers_master.erb +37 -0
- data/lib/insika/studio/views/_empty.erb +15 -0
- data/lib/insika/studio/views/_facts_master.erb +42 -0
- data/lib/insika/studio/views/_harvest_candidate_detail.erb +14 -0
- data/lib/insika/studio/views/_identity.erb +16 -0
- data/lib/insika/studio/views/_kpi.erb +17 -0
- data/lib/insika/studio/views/_message.erb +40 -2
- data/lib/insika/studio/views/_model_chart.erb +82 -0
- data/lib/insika/studio/views/_refinement_master.erb +38 -0
- data/lib/insika/studio/views/{tool_edit.erb → _tool_edit.erb} +32 -9
- data/lib/insika/studio/views/_tool_users.erb +17 -0
- data/lib/insika/studio/views/_why.erb +53 -0
- data/lib/insika/studio/views/agent_detail.erb +17 -10
- data/lib/insika/studio/views/agents.erb +28 -16
- data/lib/insika/studio/views/approvals.erb +16 -11
- data/lib/insika/studio/views/artifact.erb +20 -7
- data/lib/insika/studio/views/artifacts.erb +19 -14
- data/lib/insika/studio/views/chats.erb +22 -29
- data/lib/insika/studio/views/customer.erb +88 -43
- data/lib/insika/studio/views/customers.erb +16 -26
- data/lib/insika/studio/views/evals.erb +49 -31
- data/lib/insika/studio/views/fact_detail.erb +114 -0
- data/lib/insika/studio/views/facts.erb +15 -116
- data/lib/insika/studio/views/followups.erb +63 -52
- data/lib/insika/studio/views/funnel.erb +104 -43
- data/lib/insika/studio/views/harvest.erb +85 -59
- data/lib/insika/studio/views/home.erb +138 -23
- data/lib/insika/studio/views/inbox.erb +36 -0
- data/lib/insika/studio/views/kits.erb +176 -0
- data/lib/insika/studio/views/knowledge.erb +92 -80
- data/lib/insika/studio/views/layout.erb +42 -11
- data/lib/insika/studio/views/live.erb +45 -0
- data/lib/insika/studio/views/mcp.erb +43 -13
- data/lib/insika/studio/views/models.erb +187 -0
- data/lib/insika/studio/views/parity.erb +38 -32
- data/lib/insika/studio/views/playground.erb +13 -9
- data/lib/insika/studio/views/refinement.erb +15 -224
- data/lib/insika/studio/views/refinement_detail.erb +243 -0
- data/lib/insika/studio/views/session.erb +152 -117
- data/lib/insika/studio/views/settings.erb +86 -22
- data/lib/insika/studio/views/skills.erb +53 -19
- data/lib/insika/studio/views/system_files.erb +7 -7
- data/lib/insika/studio/views/task.erb +45 -15
- data/lib/insika/studio/views/tasks.erb +35 -18
- data/lib/insika/studio/views/tools.erb +62 -73
- data/lib/insika/task_actor.rb +4 -1
- data/lib/insika/task_store.rb +67 -1
- data/lib/insika/telemetry/pricing.rb +16 -14
- data/lib/insika/telemetry/recorder.rb +103 -9
- data/lib/insika/telemetry/ruby_llm_instrumenter.rb +110 -0
- data/lib/insika/telemetry.rb +12 -2
- data/lib/insika/templates/daily-digest/README.md +13 -0
- data/lib/insika/templates/daily-digest/agent.rb +22 -11
- data/lib/insika/testing/store_contract.rb +65 -0
- data/lib/insika/text_tool_call.rb +66 -0
- data/lib/insika/tick.rb +7 -1
- data/lib/insika/token_store.rb +5 -0
- data/lib/insika/tool_batch.rb +1 -1
- data/lib/insika/tool_catalog.rb +12 -2
- data/lib/insika/tool_definition.rb +10 -4
- data/lib/insika/tool_envelope.rb +165 -28
- data/lib/insika/tool_manifest.rb +1 -0
- data/lib/insika/tool_store.rb +3 -0
- data/lib/insika/tool_trace_store.rb +15 -8
- data/lib/insika/tool_usage_report.rb +2 -2
- data/lib/insika/tools/a2a_remote.rb +1 -1
- data/lib/insika/tools/agent_enum.rb +1 -1
- data/lib/insika/tools/confirm_pending.rb +112 -0
- data/lib/insika/tools/data_defined_tool.rb +23 -15
- data/lib/insika/tools/generate_image.rb +2 -2
- data/lib/insika/tools/load_knowledge.rb +7 -3
- data/lib/insika/tools/load_skill.rb +3 -3
- data/lib/insika/tools/present.rb +1 -8
- data/lib/insika/tools/remember.rb +7 -2
- data/lib/insika/tools/run_persona_eval.rb +3 -5
- data/lib/insika/tools/save_artifact.rb +16 -6
- data/lib/insika/tools/schedule_followup.rb +3 -3
- data/lib/insika/tools/stuck_signal.rb +3 -3
- data/lib/insika/tools/subagent.rb +5 -5
- data/lib/insika/tools/subagents.rb +3 -3
- data/lib/insika/tools/tool_search.rb +7 -5
- data/lib/insika/tools/tts.rb +2 -2
- data/lib/insika/tools/update_briefing.rb +4 -4
- data/lib/insika/turn_output.rb +2 -2
- data/lib/insika/turn_state.rb +18 -0
- data/lib/insika/turn_timing.rb +51 -14
- data/lib/insika/version.rb +1 -1
- data/lib/insika/wiring/graph.rb +81 -16
- data/lib/insika/workflow.rb +25 -64
- data/lib/insika.rb +19 -3
- metadata +69 -13
- data/lib/insika/commands/import_mcp_tools.rb +0 -48
- data/lib/insika/mcp_http_client.rb +0 -67
- data/lib/insika/mcp_tool_ingestor.rb +0 -165
|
@@ -0,0 +1,307 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: RubyLLM 2 Migration
|
|
3
|
+
parent: Ship it
|
|
4
|
+
nav_order: 6
|
|
5
|
+
permalink: /rubyllm-2-migration/
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# RubyLLM 2 Migration
|
|
9
|
+
|
|
10
|
+
Status reviewed on 2026-09-24: implemented on `codex/rubyllm-2-migration`,
|
|
11
|
+
and **approved for merge and testing on the existing Railway staging service**.
|
|
12
|
+
Keep Insika's durable runtime and policy;
|
|
13
|
+
delegate provider protocols, tool schemas, and cache formatting to RubyLLM.
|
|
14
|
+
No Rails, package extraction or database migration. JSONSchemer is a direct
|
|
15
|
+
dependency for workflow and tool argument validation.
|
|
16
|
+
|
|
17
|
+
The follow-up ecosystem audit and execution status (kept with the internal plans)
|
|
18
|
+
records replacements and the concrete gates that still block others.
|
|
19
|
+
|
|
20
|
+
## Dependency
|
|
21
|
+
|
|
22
|
+
The Gemfile and lockfile pin RubyLLM commit
|
|
23
|
+
`099c381b44fec6f1d72b2b63b9a9eea7635755f1`, the merge of
|
|
24
|
+
[native MCP PR 959](https://github.com/crmne/ruby_llm/pull/959).
|
|
25
|
+
The external `ruby_llm-mcp` dependency and its Origin-header patch are removed.
|
|
26
|
+
The pinned source still reports version 2.0.0, but published 2.0.0 does not contain
|
|
27
|
+
this client. Before publishing Insika, require a released RubyLLM version with
|
|
28
|
+
native MCP in the gemspec and remove the Git pin. A gemspec cannot carry it.
|
|
29
|
+
|
|
30
|
+
Existing MCP records and cached tool descriptors remain readable. Discovery and
|
|
31
|
+
execution use native Streamable HTTP or stdio; Insika retains policy, evidence,
|
|
32
|
+
credential masking, the stdio opt-in, and the egress allowlist. Refresh preserves
|
|
33
|
+
the server's strict boolean read-only hint. Legacy SSE records fail with a request
|
|
34
|
+
to configure a Streamable HTTP endpoint; they are not silently rewritten. Native
|
|
35
|
+
MCP also requires HTTPS outside loopback, even with the HTTP egress opt-in.
|
|
36
|
+
Sampling, roots and server logging are not supported by the native client. Insika
|
|
37
|
+
does not yet expose native MCP input-request resumption or its OAuth UI; unanswered
|
|
38
|
+
input requests surface as tool errors through the existing wrapper.
|
|
39
|
+
|
|
40
|
+
Native MCP validation on 2026-09-23: **5,614 examples, zero failures**, seed 46560.
|
|
41
|
+
Local protocol fixtures cover real stdio and HTTP sockets, modern discovery,
|
|
42
|
+
legacy handshake/session headers, event-stream responses, authorization headers,
|
|
43
|
+
tool errors, structured content, attachments, and read-only metadata. No external
|
|
44
|
+
production MCP server was invoked for these fixture checks. The current-candidate
|
|
45
|
+
benchmark below was repeated after this dependency switch; it does not call MCP.
|
|
46
|
+
|
|
47
|
+
The official RubyLLM coding skill (now from 2.1.0) is copied into `.agents/skills/rubyllm`.
|
|
48
|
+
Refresh it from the installed gem when upgrading RubyLLM, not from upstream main.
|
|
49
|
+
|
|
50
|
+
## Preserved Behavior
|
|
51
|
+
|
|
52
|
+
- Native `step` drives the existing runtime; a terminal tool finishes its batch
|
|
53
|
+
and prevents another model request. Siblings, result IDs, steering, and output remain covered.
|
|
54
|
+
- Tools use `parameter` and `parameters_schema`; data-defined JSON schemas stay unchanged.
|
|
55
|
+
- Native cache markers preserve the stable/volatile Anthropic instruction boundary.
|
|
56
|
+
- Native tokens and model readers retain Insika's usage fields. Auxiliary billing
|
|
57
|
+
includes cache reads and writes without counting thinking twice.
|
|
58
|
+
- RubyLLM now handles operator approval gating for native chats. Insika retains
|
|
59
|
+
authorization, persisted decisions, customer confirmation, security checks and UI.
|
|
60
|
+
Decisions bind to task, turn, call ID, real tool name and arguments. Checkpoints
|
|
61
|
+
preserve pending calls, completed results, usage, evidence and filtered halt text.
|
|
62
|
+
Recorded side effects are skipped on replay; terminal results survive interruption.
|
|
63
|
+
Direct workflow/customer-confirmation paths retain the existing blocking gate.
|
|
64
|
+
Legacy tool-name decisions do not authorize newly generated native calls.
|
|
65
|
+
A crash between an external effect and its local marker still requires downstream
|
|
66
|
+
idempotency; this is not an exactly-once guarantee.
|
|
67
|
+
- Opt-in native
|
|
68
|
+
compaction now uses separate model-bound state and filtered stored history, with
|
|
69
|
+
original-prefix recovery on fallback. See [native compaction](CONTEXT.md#native-responses-compaction)
|
|
70
|
+
for its conservative budget and supported providers.
|
|
71
|
+
|
|
72
|
+
## Verification
|
|
73
|
+
|
|
74
|
+
Latest completed checks (2026-09-24, before merge): **5,616 RSpec
|
|
75
|
+
examples, zero failures**, seed 19184; **53 Studio tests passed**. Assets were
|
|
76
|
+
rebuilt during the preceding Studio implementation and remain unchanged.
|
|
77
|
+
Final diff review reran 200 focused runtime/MCP/contract examples with no failures
|
|
78
|
+
and found no actionable introduced regression in the reviewed scope.
|
|
79
|
+
Browser checks covered tool grouping by origin/server, filtering, collapsible
|
|
80
|
+
sections and section selection without changing other sections or denied tools.
|
|
81
|
+
Changes still require **Save tools**; filtering does not change permissions.
|
|
82
|
+
|
|
83
|
+
Local development validation also exercised a real provider reply and MCP tool
|
|
84
|
+
discovery. The user confirmed WhatsApp replies and execution through the Metabase
|
|
85
|
+
MCP. This is smoke-test evidence, not a comparative quality eval or a new benchmark.
|
|
86
|
+
An HTTP data tool and an MCP instance may use different credentials: successful
|
|
87
|
+
MCP discovery does not validate a separately configured HTTP tool.
|
|
88
|
+
|
|
89
|
+
### Earlier Verification
|
|
90
|
+
|
|
91
|
+
Native transport retries are enabled for reliable native chats, with isolated
|
|
92
|
+
per-chat configuration and no outer Insika retries. The approved behavior is
|
|
93
|
+
per-request retries, stopping once streaming starts. Insika still owns fallback,
|
|
94
|
+
model policy, tenant circuits and the overall per-model deadline. Circuits count
|
|
95
|
+
an exhausted native sequence once, not each internal HTTP failure. The native
|
|
96
|
+
request timeout is also configured from the profile. Completed tool results
|
|
97
|
+
survive fallback, and failed-stream redactor tails are discarded.
|
|
98
|
+
Native `with_fallbacks` adoption remains deferred: it restores the original model
|
|
99
|
+
before the next generation and requires adaptation for model-bound compaction.
|
|
100
|
+
|
|
101
|
+
After native compaction, operator approvals and native transport retries: **5,603 examples, zero failures** (seed 32098).
|
|
102
|
+
The execution record (kept with the internal plans) has the new contracts, remaining replacement gates and offline benchmark.
|
|
103
|
+
|
|
104
|
+
Full RSpec after the benchmark correction: **5,519 examples, zero failures**
|
|
105
|
+
(seed 55367). Studio: **50 tests passed** during the preceding migration validation.
|
|
106
|
+
The real-gem contract has 17 behavioral cases, with no missing-gem skip.
|
|
107
|
+
Synthetic SQLite records passed 1.16 write -> 2.0 read/write -> 1.16 read/write,
|
|
108
|
+
including history, pending/resolved approvals, and recorded side-effect markers.
|
|
109
|
+
An isolated installed gem passed reply, SQLite embed, Rack mount, actual HTTP/SSE,
|
|
110
|
+
Studio login, packaged assets/docs, and lazy-load guards using an offline provider.
|
|
111
|
+
That earlier install used the former external MCP Git dependency. On 2026-09-24,
|
|
112
|
+
the current native-MCP candidate was built and installed in a fresh GEM_HOME
|
|
113
|
+
outside the checkout and passed all those checks again. The synthetic SQLite
|
|
114
|
+
1.16 -> 2.0 -> 1.16 record rehearsal also passed again. A Linux ARM64 Docker
|
|
115
|
+
build with the deployment lockfile passed and loaded native MCP while preserving
|
|
116
|
+
the core lazy-load guard. The isolated gem install explicitly pins RubyLLM's Git
|
|
117
|
+
revision and resolves other allowed dependencies; it is not a public gem-install
|
|
118
|
+
proof or a substitute for verifying Railway's x86_64 deployment.
|
|
119
|
+
Browser checks passed for login, agent configuration, skill/provider creation,
|
|
120
|
+
Playground replies, completed tasks, and session transcripts. Approval/resume
|
|
121
|
+
flows were covered by automated tests, not separately exercised in the browser.
|
|
122
|
+
|
|
123
|
+
## Performance
|
|
124
|
+
|
|
125
|
+
### Current Candidate (2026-09-24)
|
|
126
|
+
|
|
127
|
+
Five alternating runs per version in each of three modes (stub, real Chat with
|
|
128
|
+
a synthetic model, and real Chat with catalog model `gpt-4o`), plus five SQLite
|
|
129
|
+
runs per version. The catalog experiment was subsequently corrected and rerun
|
|
130
|
+
five times per version, as described below. Ruby 4.0.6,
|
|
131
|
+
arm64-darwin25, YJIT off; 300 latency turns, 30 warmups, concurrency 16 and
|
|
132
|
+
20 waves per scenario. Native MCP and the latest Studio changes are present.
|
|
133
|
+
The same existing benchmark harnesses load each checkout's actual runtime.
|
|
134
|
+
|
|
135
|
+
All 40 benchmark processes succeeded: 58,500 engine turns including warmups,
|
|
136
|
+
matching provider/tool/history/chunk counts, and 300,000 SQLite writes with zero
|
|
137
|
+
reported lock errors. Automated suites ran after the measurement. Development
|
|
138
|
+
servers remained running, so workstation noise is not eliminated.
|
|
139
|
+
|
|
140
|
+
Profiling exposed a flaw in the earlier catalog experiment: its flag assertion
|
|
141
|
+
wrapped `RubyLLM.chat`, but `main` constructs chats through a context. Its
|
|
142
|
+
`assume_model_exists: false` override therefore did not reach the actual Chat.
|
|
143
|
+
Those earlier catalog numbers do not establish equivalent registry behavior.
|
|
144
|
+
The corrected local harness forces the option at `Chat#initialize` and asserts
|
|
145
|
+
that the resulting model is the exact registry object. The separate profiler
|
|
146
|
+
confirms one registry lookup per measured turn in both versions.
|
|
147
|
+
|
|
148
|
+
All ten corrected benchmark processes succeeded, with matching activity counts
|
|
149
|
+
across 19,500 additional turns including warmups. Median corrected measurements:
|
|
150
|
+
|
|
151
|
+
| Scenario | Main p50 / p95 ms | Migration p50 / p95 ms | Turns/s, before -> after |
|
|
152
|
+
| --- | --- | --- | --- |
|
|
153
|
+
| Greeting | 1.37 / 2.01 | 1.48 / 2.48 | 654.1 -> 603.8 |
|
|
154
|
+
| Two tool rounds | 1.46 / 2.10 | 1.80 / 2.42 | 603.5 -> 515.7 |
|
|
155
|
+
| History | 1.51 / 2.08 | 1.62 / 2.25 | 573.9 -> 558.2 |
|
|
156
|
+
|
|
157
|
+
Catalog p50 increases are **7.3-23.3%**, or **0.11-0.34 ms per turn**.
|
|
158
|
+
Peak RSS increased **13.9%** (87,769,088 -> 99,991,552 bytes), and tool
|
|
159
|
+
allocations increased 8.4%. Tool p50 ranges did not overlap (main 1.35-1.58 ms,
|
|
160
|
+
migration 1.65-1.85 ms); the other scenarios' ranges overlap. These are new runs,
|
|
161
|
+
so differences from the invalid comparison are not attributable solely to the
|
|
162
|
+
harness correction. Synthetic-model p50 increased 18.3-28.2% and peak
|
|
163
|
+
RSS 29.4%. Stub p50 changes were -5.3%, +4.5% and +9.1%, with peak RSS +2.3%.
|
|
164
|
+
SQLite median writes/s stayed within 2.3% of baseline in either direction.
|
|
165
|
+
|
|
166
|
+
The proposed 10% gate still fails numerically. This isolates an increase along the real-Chat
|
|
167
|
+
path, not its cause; it does not measure native HTTP, MCP calls, UI speed or
|
|
168
|
+
provider latency. No safety or accounting was disabled. Earlier measurements
|
|
169
|
+
below are retained as historical comparisons, not current acceptance evidence.
|
|
170
|
+
|
|
171
|
+
**Decision (2026-09-24):** The project owner accepted the measured performance
|
|
172
|
+
tradeoff for this migration and approved continuing with RubyLLM 2's native
|
|
173
|
+
runtime. The additional 0.34 ms in the two-tool benchmark does not block ongoing
|
|
174
|
+
development validation. This is an explicit exception, not a passing benchmark
|
|
175
|
+
or acceptance of future regressions. There are no real operational workflows
|
|
176
|
+
beyond the tests already executed. Continue validation with the existing
|
|
177
|
+
automated tests, controlled integration checks and offline benchmarks. Measure
|
|
178
|
+
live latency, errors and memory when actual usage exists; that is not a current
|
|
179
|
+
development prerequisite. Investigate
|
|
180
|
+
upstream improvements with isolated evidence; do not add custom caches or a
|
|
181
|
+
replacement tool loop without demonstrated waste. The performance decision alone
|
|
182
|
+
did not authorize deployment. The project owner subsequently authorized merge
|
|
183
|
+
to `main` and testing through its automatic Railway staging deployment. Gem
|
|
184
|
+
publication remains outside that authorization.
|
|
185
|
+
|
|
186
|
+
### Targeted Profiling (2026-09-24)
|
|
187
|
+
|
|
188
|
+
Three alternating instrumented runs per version, each with 1,001 measured
|
|
189
|
+
two-tool turns after 30 warmups, completed without errors. Both versions made
|
|
190
|
+
one registry lookup, three provider calls and two tool calls per measured turn.
|
|
191
|
+
The probe wraps selected methods using the monotonic clock and `GC.stat`; it is
|
|
192
|
+
local diagnostic tooling, not a runtime dependency.
|
|
193
|
+
|
|
194
|
+
Median inclusive time per turn, rounded, across those instrumented runs:
|
|
195
|
+
|
|
196
|
+
| Scope | Main ms | Migration ms |
|
|
197
|
+
| --- | --- | --- |
|
|
198
|
+
| `Executor#ask_on` (generation and tool cycle) | 0.402 | 0.659 |
|
|
199
|
+
| `Executor#create_chat` | 0.261 | 0.279 |
|
|
200
|
+
| `Executor#record_model_visible` | 0.011 | 0.042 |
|
|
201
|
+
| `Executor#persist_turn` | 0.326 | 0.322 |
|
|
202
|
+
| `ContextBuilder#call` | 0.251 | 0.267 |
|
|
203
|
+
|
|
204
|
+
The largest observed increase is inside the generation/tool cycle. Native
|
|
205
|
+
`instrumentation_payload`, `record_completion_event`, and
|
|
206
|
+
`record_out_of_band_usage` account for approximately 0.030, 0.022 and 0.038 ms
|
|
207
|
+
per turn respectively. The latter is the offline provider's fallback accounting
|
|
208
|
+
path, not the accounting path used by native HTTP providers. Message creation
|
|
209
|
+
also grows from approximately 0.063 to 0.079 ms for the same 55 messages/chunks
|
|
210
|
+
per turn. These nested times overlap other scopes and must not be added to them.
|
|
211
|
+
|
|
212
|
+
Registry lookup is slower (approximately 0.073 -> 0.111 ms), but native 2.0
|
|
213
|
+
creates one connection per turn instead of 1.16's two; the overall Chat creation
|
|
214
|
+
difference is much smaller. The trace increase includes corrected behavior:
|
|
215
|
+
`main` treats the tool Hash as pairs and misses schemas, whereas the migration
|
|
216
|
+
captures actual tool schemas and system instructions. Removing those records
|
|
217
|
+
would reintroduce a correctness problem, not provide an equivalent optimization.
|
|
218
|
+
|
|
219
|
+
Instrumented timings include probe overhead and cannot apportion the exact
|
|
220
|
+
uninstrumented regression. Instrumented allocation counts are not clean
|
|
221
|
+
allocation measurements. RSS growth remains unattributed; MCP and HTTP were not
|
|
222
|
+
exercised. The next investigation should isolate RubyLLM's per-round bookkeeping
|
|
223
|
+
without Insika before proposing an upstream optimization. Keep accounting,
|
|
224
|
+
approval checks and trace correctness intact; no custom tool loop or cache was
|
|
225
|
+
added during profiling.
|
|
226
|
+
|
|
227
|
+
### Earlier Catalog-Model Experiment (Invalid Flag Assertion)
|
|
228
|
+
|
|
229
|
+
These results used the ineffective `RubyLLM.chat` assertion described above.
|
|
230
|
+
They are historical only, not an equivalent-registry comparison. Five
|
|
231
|
+
alternating runs per version named `gpt-4o`. The offline provider is
|
|
232
|
+
registered under `openai` only inside the experiment; it does not make HTTP calls.
|
|
233
|
+
Its intended assertions were bypassed by context-created chats. No application
|
|
234
|
+
model-selection behavior was changed.
|
|
235
|
+
Settings match the earlier comparison: Ruby 4.0.6, YJIT off, 300 latency turns,
|
|
236
|
+
30 warmups, concurrency 16, 20 waves, and identical tool/history/chunk counts.
|
|
237
|
+
Every scenario in all ten runs completed without errors.
|
|
238
|
+
|
|
239
|
+
| Scenario | 1.16 p50 / p95 ms | 2.0 p50 / p95 ms | Turns/s, before -> after |
|
|
240
|
+
| --- | --- | --- | --- |
|
|
241
|
+
| Greeting | 1.23 / 1.65 | 1.32 / 2.06 | 549.2 -> 629.2 |
|
|
242
|
+
| Two tool rounds | 1.36 / 1.75 | 1.62 / 2.09 | 620.3 -> 539.8 |
|
|
243
|
+
| History | 1.28 / 1.78 | 1.44 / 1.95 | 590.6 -> 561.9 |
|
|
244
|
+
|
|
245
|
+
Median p50 changes are +7.3%, +19.1%, and +12.5%. Median peak RSS is
|
|
246
|
+
89,194,496 -> 105,971,712 bytes (+18.8%); tool allocations rise 4.8%.
|
|
247
|
+
These numbers cannot determine whether registry lookup avoidance explains the
|
|
248
|
+
difference. They are offline engine-plus-Chat results, not
|
|
249
|
+
live-provider latency or proof of a RubyLLM defect. Investigate the remaining
|
|
250
|
+
tool-loop and memory differences before making an upstream performance claim.
|
|
251
|
+
|
|
252
|
+
### Earlier Unknown-Model Experiment
|
|
253
|
+
|
|
254
|
+
The following results are retained for comparison, not generalized to catalog
|
|
255
|
+
models. The fake model used `assume_model_exists: true`, which bypassed registry
|
|
256
|
+
resolution in 1.16 but not 2.0.
|
|
257
|
+
|
|
258
|
+
Five interleaved runs per version, Ruby 4.0.6, YJIT off, same host and settings.
|
|
259
|
+
Real Chat with a synthetic provider, no network; provider/tool/chunk/history counts match.
|
|
260
|
+
|
|
261
|
+
| Scenario | 1.16 p50 / p95 ms | 2.0 p50 / p95 ms | Turns/s, before -> after |
|
|
262
|
+
| --- | --- | --- | --- |
|
|
263
|
+
| Greeting | 1.38 / 1.83 | 1.60 / 2.23 | 476.3 -> 571.2 |
|
|
264
|
+
| Two tool rounds | 1.35 / 1.95 | 1.79 / 2.44 | 619.8 -> 491.5 |
|
|
265
|
+
| History | 1.34 / 1.87 | 1.78 / 2.37 | 597.3 -> 498.2 |
|
|
266
|
+
|
|
267
|
+
These replace the initial measurements: the offline provider omitted request
|
|
268
|
+
`model_info`, unlike native `Accounting::Usage::Tracker#succeed`. Supplying it
|
|
269
|
+
removes three spurious pricing catalog lookups per two-tool turn. The benchmark
|
|
270
|
+
spec now rejects those lookups. No production runtime workaround was added.
|
|
271
|
+
|
|
272
|
+
Corrected real-Chat p50 increased 15.9-32.8%; peak RSS increased 24.0%
|
|
273
|
+
(87,310,336 -> 108,232,704 bytes); tool allocations increased 7.4%.
|
|
274
|
+
RSS also increased 11.3% in stub mode, so the entire memory difference cannot
|
|
275
|
+
be attributed to RubyLLM catalog loading. SQLite had zero lock errors, but
|
|
276
|
+
throughput varied from +1.6% to -17.2%; the earlier no-regression claim is withdrawn.
|
|
277
|
+
These are host-specific samples, not a controlled attribution of every difference.
|
|
278
|
+
|
|
279
|
+
The standalone `scripts/rubyllm_profile.rb` reproduces the fixture defect and
|
|
280
|
+
the remaining catalog lookup without Insika. With `--request-context --profile`,
|
|
281
|
+
1,000 two-tool turns make 1,000 `Models#find` calls in 2.0 versus none in 1.16.
|
|
282
|
+
RubyLLM 2 resolves catalog metadata even with `assume_model_exists`; 1.16 skipped
|
|
283
|
+
it for this non-local provider. In one instrumented run that lookup consumed
|
|
284
|
+
112 ms total; fallback usage accounting consumed 39 ms. Inclusive timings must
|
|
285
|
+
not be summed. All three requests, both tools, history and aggregate tokens are
|
|
286
|
+
checked. The fake provider bypasses native transport and uses fallback usage
|
|
287
|
+
accounting, so this is not an end-to-end native-provider performance claim.
|
|
288
|
+
|
|
289
|
+
For upstream discussion, run the script outside Bundler with `1.16.0 --profile`
|
|
290
|
+
and `2.0.0 --request-context --profile`. Omitting `--request-context` deliberately
|
|
291
|
+
recreates the old fixture defect, not native provider behavior. The useful upstream
|
|
292
|
+
question is whether repeated registry resolution can be cheaper while preserving
|
|
293
|
+
model metadata and refresh semantics. No upstream patch or issue was published;
|
|
294
|
+
remaining latency and memory attribution is open.
|
|
295
|
+
The proposed 10% performance gate **failed**. No accounting or safety was disabled.
|
|
296
|
+
Reproduce the candidate with the commands in [Benchmark](BENCHMARK.md).
|
|
297
|
+
|
|
298
|
+
## Release Gate
|
|
299
|
+
|
|
300
|
+
Public gem release still requires a published RubyLLM native-MCP dependency.
|
|
301
|
+
The measured performance tradeoff is accepted; current installed-artifact and
|
|
302
|
+
synthetic record-compatibility checks passed. Merge to `main` and testing of its
|
|
303
|
+
automatic Railway staging deployment are authorized. Confirm that deployment's
|
|
304
|
+
status and HTTP behavior before reporting success. No production rollout,
|
|
305
|
+
database schema migration or gem publication is part of this validation.
|
|
306
|
+
Record compatibility does not guarantee exactly-once external effects; keep
|
|
307
|
+
destination idempotency or reconciliation for an execution-before-checkpoint failure.
|
data/docs/SECURITY.md
CHANGED
|
@@ -246,6 +246,13 @@ resume path is durable: an approval that arrives after a restart still resumes t
|
|
|
246
246
|
turn from its checkpoint (see
|
|
247
247
|
[Architecture](ARCHITECTURE.md#durability-checkpoints-and-resume)).
|
|
248
248
|
|
|
249
|
+
There is a second approver: the customer. `customer_confirm: [tool names]` holds a
|
|
250
|
+
call the same way, but the turn *ends* with a question instead of suspending, and the
|
|
251
|
+
next message of the conversation decides it through `confirm_pending` /
|
|
252
|
+
`cancel_pending`. A hold nobody resolves expires when that next turn ends. Use it for
|
|
253
|
+
the write the person on the other end must agree to; use approval for the one a staff
|
|
254
|
+
member must. A tool cannot be on both lists. See [Tools](TOOLS.md#customer-confirmation-a-write-the-conversation-approves).
|
|
255
|
+
|
|
249
256
|
## Egress: the SSRF boundary
|
|
250
257
|
|
|
251
258
|
Every outbound HTTP call from a data tool passes through the **EgressGuard**, a
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
---
|
|
2
|
+
layout: default
|
|
3
|
+
title: Shared conversations
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Shared conversations
|
|
7
|
+
|
|
8
|
+
`shared_conversations: true` opts an agent into required central history. The
|
|
9
|
+
DSL accepts `shared_conversations true`; stored profiles preserve the flag.
|
|
10
|
+
Default is false. Configure `INSIKA_CONVERSATIONS_URL`, a scoped
|
|
11
|
+
`INSIKA_CONVERSATIONS_TOKEN`, and a durable native backend (`INSIKA_DB` or
|
|
12
|
+
`INSIKA_DATABASE_URL`). The client provisions the central conversation first.
|
|
13
|
+
|
|
14
|
+
The tenant-authenticated `/v1/responses` request adds `user_text` (original speech)
|
|
15
|
+
and `shared_conversation` with UUID `tenant_id`, `user_id`, `agent_id`,
|
|
16
|
+
`conversation_id`, `turn_id`, and `message_id`. Keep the
|
|
17
|
+
same turn/message IDs across retries. Existing `input` still contains the
|
|
18
|
+
presenter/context text used for this request. The authenticated tenant must match.
|
|
19
|
+
Omit `generation` to acquire a settled conversation automatically when admitting
|
|
20
|
+
the next turn. The engine reads and writes the acquired generation internally.
|
|
21
|
+
An explicit integer `generation` retains the legacy ownership check.
|
|
22
|
+
|
|
23
|
+
Upload incoming media to the central attachment endpoint first. Optional
|
|
24
|
+
`shared_conversation.content` carries canonical text/media parts with attachment
|
|
25
|
+
IDs; its text must match `user_text`. Generated image/audio bytes upload before
|
|
26
|
+
completion. Native checkpoints store attachment bytes and hydrate them at the
|
|
27
|
+
RubyLLM boundary. Full tool results retain secret-key masking without prompt caps.
|
|
28
|
+
|
|
29
|
+
Admission precedes model/tool work. Delivery follows central completion. A lost
|
|
30
|
+
completion acknowledgment reuses the native durable result; it never reruns tools.
|
|
31
|
+
An uncertain execution stays blocked for operator reconciliation. Only the same
|
|
32
|
+
native waiting approval can resume from its checkpoint under the current generation.
|
|
33
|
+
Unresolved customer confirmations block switching. Central deletion/outage fails
|
|
34
|
+
before cached history is used. Local native purge still requires an explicit cleanup.
|
|
35
|
+
|
|
36
|
+
Run the isolated HTTP proof (synthetic model/tools, temporary databases):
|
|
37
|
+
|
|
38
|
+
```sh
|
|
39
|
+
SHARED_CONVERSATIONS_SOURCE=/path/to/shared-memory-repo bundle exec rspec spec/insika/shared_conversations_http_spec.rb
|
|
40
|
+
```
|
data/docs/SKILLS.md
CHANGED
|
@@ -249,9 +249,11 @@ in the allowlist → the model never sees it in `<available_skills>`.
|
|
|
249
249
|
Two ways to satisfy both:
|
|
250
250
|
|
|
251
251
|
- **Via a definition/pack import.** The import writes each skill directory into
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
252
|
+
that agent's scope, without overwriting another agent's or a shared skill, and
|
|
253
|
+
sets the agent's `skills` allowlist **authoritatively** from the skills present.
|
|
254
|
+
Dropping a skill removes it from the allowlist, not storage. Keep the definition
|
|
255
|
+
complete. Packs previously imported into shared scope must be reimported from
|
|
256
|
+
their original sources; the overwritten shared body cannot recover each agent's version.
|
|
255
257
|
- **Directly (Studio / API / DSL).** Write the skill (upserts the row and reloads
|
|
256
258
|
the catalog atomically — live immediately), then attach it to the agent(s) by
|
|
257
259
|
adding its name to the `skills` allowlist.
|
data/docs/STUDIO.md
ADDED
|
@@ -0,0 +1,122 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Studio
|
|
3
|
+
parent: Operate
|
|
4
|
+
nav_order: 0
|
|
5
|
+
permalink: /studio/
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Studio — the operator UI
|
|
9
|
+
|
|
10
|
+
The Studio is the web UI at `/studio`. Sign in with `ADMIN_TOKEN` (see
|
|
11
|
+
[Deploy](DEPLOY.md#environment-variables)). This page covers the screens for
|
|
12
|
+
watching conversations, reviewing what waits on a person, and seeing what a
|
|
13
|
+
change did. Model metrics are on [Observability](OBSERVABILITY.md#studio-models).
|
|
14
|
+
|
|
15
|
+
Outside production, a banner above every page names the environment
|
|
16
|
+
(`INSIKA_ENV`, else `RACK_ENV`, else `local`): amber for staging, blue for
|
|
17
|
+
anything else. Production (`production` or `prod`) shows no banner.
|
|
18
|
+
|
|
19
|
+
## Navigation
|
|
20
|
+
|
|
21
|
+
The sidebar groups pages as Operate, Review, Build, Insights and Platform. Each
|
|
22
|
+
Review page shows a badge with how many items wait on it. `⌘K` (or Search in the
|
|
23
|
+
sidebar) opens a palette that jumps to any page or agent.
|
|
24
|
+
|
|
25
|
+
## Overview
|
|
26
|
+
|
|
27
|
+
`/studio/home`. The Overview shows the headline numbers for the agent filter and
|
|
28
|
+
a **Needs attention** row. The row lists the review queues, customers waiting for
|
|
29
|
+
a reply and tools failing in the last 24 hours, each linking to its page. Only
|
|
30
|
+
non-zero items show.
|
|
31
|
+
|
|
32
|
+
## Live
|
|
33
|
+
|
|
34
|
+
`/studio/live` shows one card per recent conversation: customer, agent, the last
|
|
35
|
+
message, a presence dot (activity in the last 5 minutes) and the guardrail flag.
|
|
36
|
+
Cards move with the event stream. While a turn runs, a card shows the tools it
|
|
37
|
+
calls and "agent is writing…". A turn paused for an approval stops writing until
|
|
38
|
+
it resumes. A ticker lists the latest events, without the streaming deltas.
|
|
39
|
+
|
|
40
|
+
`?agent=<id>` shows only that agent's conversations.
|
|
41
|
+
|
|
42
|
+
## Chats
|
|
43
|
+
|
|
44
|
+
`/studio/chats` lists conversations with three tabs: **All**, **Active now**
|
|
45
|
+
(activity in the last 5 minutes) and **Guardrail flagged**, each with its count.
|
|
46
|
+
The session page shows the customer (key, name, phone) and links to their other
|
|
47
|
+
conversations.
|
|
48
|
+
|
|
49
|
+
### Why this reply
|
|
50
|
+
|
|
51
|
+
Each agent reply has a collapsed **why this reply** panel for its own turn:
|
|
52
|
+
|
|
53
|
+
- context tokens by category, and the skills injected with the reason;
|
|
54
|
+
- the tool calls with status and timing;
|
|
55
|
+
- the guardrail flags on the customer's message.
|
|
56
|
+
|
|
57
|
+
From the panel, **Replay in Playground** opens the conversation in the
|
|
58
|
+
Playground, and **Save as eval
|
|
59
|
+
case** opens the Evals editor prefilled from the reply (the customer's turns,
|
|
60
|
+
the transcript as the reference, the tools the turn called). A reply that
|
|
61
|
+
streams in live offers the panel when its turn finishes.
|
|
62
|
+
|
|
63
|
+
### Tool calls
|
|
64
|
+
|
|
65
|
+
A turn with two or more tool calls draws a waterfall of when each one ran.
|
|
66
|
+
Tool calls are grouped by turn within each task, since the turn number restarts
|
|
67
|
+
at 1 for every task.
|
|
68
|
+
|
|
69
|
+
### Markup tags
|
|
70
|
+
|
|
71
|
+
Tags an agent or a host writes in the text, like `<carrossel/>`,
|
|
72
|
+
`<vitrine formato="lista">` or `</b>`, show as highlighted tokens: in replies,
|
|
73
|
+
customer messages, and the Live and Chats previews. They are escaped and never
|
|
74
|
+
rendered. The channel decides what a tag does; the Studio only shows it is
|
|
75
|
+
there. A tag inside inline code stays code.
|
|
76
|
+
|
|
77
|
+
## Guardrail false positives
|
|
78
|
+
|
|
79
|
+
On a guardrail flag in the session page, **Mark false positive** records the
|
|
80
|
+
verdict on the flag itself, inside the session, so retention and tenant purges
|
|
81
|
+
cover it. The flag can later be marked resolved, or the verdict undone.
|
|
82
|
+
Reviewing a flag does not count as conversation activity: the session keeps its
|
|
83
|
+
place in the recent lists. The command is `:review_guardrail_flag`.
|
|
84
|
+
|
|
85
|
+
## Inbox
|
|
86
|
+
|
|
87
|
+
`/studio/inbox` is one list, oldest first, of everything that waits on a person:
|
|
88
|
+
approvals, fact proposals, mined skills, refinement runs and reported guardrail
|
|
89
|
+
false positives. Tabs filter by type, with counts. Each row opens the page where
|
|
90
|
+
the item is decided.
|
|
91
|
+
|
|
92
|
+
## Agent History
|
|
93
|
+
|
|
94
|
+
The **History** tab of an agent shows outcomes per day over the last 30 days:
|
|
95
|
+
the funnel's primary outcome when the agent declares a funnel, else all
|
|
96
|
+
outcomes. A dashed line marks each change to a prompt file or to one of this
|
|
97
|
+
agent's own skills (shared skills change for every agent, so they are not
|
|
98
|
+
marked).
|
|
99
|
+
|
|
100
|
+
Under the chart, one card per change shows the file or skill, when, a short line
|
|
101
|
+
diff, and the outcomes in the 7 days after against the 7 days before. A window
|
|
102
|
+
that is not complete yet, or that starts before the outcome data, reads "not
|
|
103
|
+
enough data". A prompt card has the Restore form; a skill card links to the
|
|
104
|
+
skill.
|
|
105
|
+
|
|
106
|
+
## Funnel: where conversations stop
|
|
107
|
+
|
|
108
|
+
The Funnel page marks the biggest loss between two consecutive declared stages.
|
|
109
|
+
Each loss opens (`?drop=<stage>`) the conversations that reached the stage but
|
|
110
|
+
not the next one in the period, grouped by the last thing that happened:
|
|
111
|
+
|
|
112
|
+
| Reason | Meaning |
|
|
113
|
+
|--------|---------|
|
|
114
|
+
| Last tool call failed | the conversation's last tool call returned an error |
|
|
115
|
+
| A guardrail flagged a message | a guardrail flagged one of its messages |
|
|
116
|
+
| Customer waiting on a reply | the customer wrote last |
|
|
117
|
+
| Customer went quiet after the agent's reply | the agent wrote last |
|
|
118
|
+
| No message to read | the conversation has no message to classify |
|
|
119
|
+
| Conversation no longer stored | the conversation was deleted |
|
|
120
|
+
|
|
121
|
+
Only the newest 200 stuck conversations are classified; the page says how many
|
|
122
|
+
were left out.
|