@polymorphism-tech/morph-spec 4.7.1 → 4.7.2

package/.morph/framework/standards/ai-agents/blazor-ui.md

@@ -0,0 +1,364 @@
+# Microsoft Agent Framework - Natural Language First Blazor UI
+
+> **Scope:** blazor-azure
+> **Layer:** 2 (on keyword)
+> **Keywords:** blazor, natural language, ui, llm, voice, ai-first
+> **Load When:** blazor ai-first ui keywords detected
+
+**Ref:** `setup.md` — Core setup | `production.md` — Middleware, tools
+
+Patterns for AI-first user interfaces in Blazor — voice, state-based control, semantic navigation.
+
+---
+
+## The Problem
+
+Current AI UIs are "empty text box + button" — users don't know what the app can do. We're in the "pinch and zoom" era of AI interfaces, still discovering what "AI First" means (like early mobile sites before responsive design).
+
+**Solution:** Smart Components + State-Based Control + Natural Language Processing
+
+---
+
+## State-Based Control Strategy
+
+Instead of giving the LLM the full DOM, extract component metadata as JSON, let the LLM modify values, apply back.
+
+```
+┌───────────────┐     ┌───────────────┐     ┌───────────────┐
+│  UI Component │ →→→ │  JSON State   │ →→→ │     LLM       │
+│  (Data Grid)  │     │ (sort, filter │     │ (modifies     │
+│               │     │  group, etc.) │     │  values only) │
+└───────────────┘     └───────────────┘     └───────┬───────┘
+                                                     │
+┌───────────────┐                           ┌───────┴───────┐
+│  UI Component │ ←←← Modified JSON  ←←←←← │  JSON Output  │
+│ (state applied)│                           └───────────────┘
+└───────────────┘
+```
+
+### System Prompt
+
+```
+You are a JSON parser. You will use the user's query to modify
+values in JSON format. Only update the values in JSON data.
+Do not modify properties.
+
+JSON Schema:
+{schema}
+
+Current State:
+{currentState}
+
+User Query: {userQuery}
+```
+
+### Key Insight
+
+LLMs treat JSON as a language — they can manipulate, translate, and transform structured data without understanding the underlying UI.
+
+---
+
+## Pattern 1: Data Grid Control via Voice
+
+```csharp
+// Extract grid state as JSON
+var gridState = new
+{
+    columns = new[] { "Name", "Amount", "Category", "Date" },
+    sort = new { column = (string?)null, direction = "asc" },
+    filter = new { column = (string?)null, value = (string?)null },
+    groupBy = (string?)null
+};
+
+// Send to agent with user query
+var agent = chatClient.AsAIAgent(
+    instructions: """
+        You modify data grid state based on user commands.
+        Only update values. Return valid JSON.
+        """);
+
+var response = await agent.RunAsync(
+    $"State: {JsonSerializer.Serialize(gridState)}\nQuery: Sort amount descending");
+
+// Parse response and apply to grid component
+var newState = JsonSerializer.Deserialize<GridState>(response.Text);
+```
+
+**Example interactions:**
+
+| User Command | Grid Action |
+|-------------|-------------|
+| "Sort amount descending" | `sort.column = "Amount", sort.direction = "desc"` |
+| "Group by category" | `groupBy = "Category"` |
+| "Show only items over $100" | `filter.column = "Amount", filter.value = ">100"` |
+
+---
+
+## Pattern 2: Tool Calling for UI Manipulation
+
+Define granular tools that the agent can call to interact with UI components.
+
+```csharp
+public class FormTools
+{
+    private readonly FormState _formState;
+    private readonly NavigationManager _nav;
+
+    [Description("Updates a specific field in the form")]
+    public void UpdateField(
+        [Description("Field name")] string fieldName,
+        [Description("New value")] string value)
+    {
+        _formState.SetField(fieldName, value);
+    }
+
+    [Description("Clears a specific field (use when bad data received)")]
+    public void ClearField([Description("Field name")] string fieldName)
+    {
+        _formState.ClearField(fieldName);
+    }
+
+    [Description("Validates the form and returns validation result")]
+    public string ValidateForm()
+    {
+        var errors = _formState.Validate();
+        return errors.Any()
+            ? $"Errors: {string.Join(", ", errors)}"
+            : "Form is valid";
+    }
+
+    [Description("Submits the form if valid")]
+    public string SubmitForm()
+    {
+        if (!_formState.IsValid) return "Form has errors. Fix them first.";
+        _formState.Submit();
+        return "Form submitted successfully";
+    }
+
+    [Description("Navigates to a specific page")]
+    public void NavigateTo([Description("Page URL")] string url)
+    {
+        _nav.NavigateTo(url);
+    }
+}
+```
+
+**Critical lessons:**
+- Use **granular tools** (e.g., `ClearField` separate from `UpdateField`) — without `ClearField`, the LLM might clear the entire form when receiving bad data
+- Tool **descriptions** determine which tool the LLM picks — be precise
+
+### Register Agent with Form Tools
+
+```csharp
+var formTools = new FormTools(formState, navigationManager);
+
+var agent = new ChatClientAgent(chatClient, new ChatClientAgentOptions
+{
+    Name = "FormAssistant",
+    Instructions = """
+        You help users fill out the RMA form.
+        Walk them through fields one by one.
+        Validate inputs before setting them.
+        Use ClearField when receiving invalid data.
+        """,
+    ChatOptions = new ChatOptions
+    {
+        Tools =
+        [
+            AIFunctionFactory.Create(formTools.UpdateField),
+            AIFunctionFactory.Create(formTools.ClearField),
+            AIFunctionFactory.Create(formTools.ValidateForm),
+            AIFunctionFactory.Create(formTools.SubmitForm)
+        ]
+    }
+});
+```
+
+---
+
+## Pattern 3: Semantic Navigation
+
+Store routes as embeddings in a vector store — match user intent to pages by meaning, not exact text.
+
+```csharp
+public class NavigationTools
+{
+    private readonly IVectorStore _vectorStore;
+    private readonly NavigationManager _nav;
+
+    [Description("Gets available pages that match a description")]
+    public async Task<string> FindPageAsync(
+        [Description("What the user is looking for")] string description)
+    {
+        var results = await _vectorStore.SearchAsync(description, maxResults: 3);
+        return string.Join("\n", results.Select(r =>
+            $"- {r.Title} ({r.Url}) — Similarity: {r.Score:P0}"));
+    }
+
+    [Description("Navigates to a specific page in the application")]
+    public void NavigateTo([Description("URL to navigate to")] string url)
+    {
+        _nav.NavigateTo(url);
+    }
+}
+```
+
+**Example:** User says "Show me the returns page" → vector search finds `/rma` (Return Merchandise Authorization) even though the words don't match literally.
+
+---
+
+## Pattern 4: Voice Integration
+
+Using Azure Speech Services for speech-to-text and text-to-speech:
+
+```csharp
+public class VoiceTools
+{
+    private readonly ITextToSpeechService _tts;
+
+    [Description("Gets available voice options")]
+    public async Task<List<string>> GetVoicesAsync()
+        => await _tts.GetAvailableVoicesAsync();
+
+    [Description("Changes the assistant's voice")]
+    public void SetVoice([Description("Voice name")] string voiceName)
+        => _tts.SetVoice(voiceName);
+}
+```
+
+**Stack:** Microsoft.Extensions.AI + Azure AI Services (Speech)
+
+---
+
+## Pattern 5: Intelligent Validation
+
+The LLM understands data types and can validate contextually:
+
+| Input | Expected | LLM Response |
+|-------|----------|--------------|
+| "555 Main Street" | Email field | "That looks like an address. What's your email?" |
+| "I want my money back" | Dropdown (Refund/Replace/Repair) | Maps to "Refund" option |
+| Text in Swedish | English form | Extracts and translates field values automatically |
+
+---
+
+## Multilingual Support (Free)
+
+Because the LLM translates between the user's language and the form schema, users can interact in any language without code changes.
+
+```
+User (Swedish): "Mitt namn är Erik och min e-post är erik@test.se"
+→ LLM maps: firstName="Erik", email="erik@test.se"
+→ Form filled in English schema
+```
+
+---
+
+## UX Best Practices
+
+| Practice | Why |
+|----------|-----|
+| Suggested prompt buttons | Reduce barrier — users see what's possible |
+| Hybrid input (voice + text + click) | Don't force a single modality |
+| Double validation (LLM + app) | Defense in depth |
+| Progressive disclosure | Start simple, reveal complexity on demand |
+| Feedback on agent actions | Show what the agent changed |
+
+---
+
+## Blazor Component Template
+
+```razor
+@page "/ai-form"
+@inject IServiceProvider ServiceProvider
+
+<div class="ai-form-container">
+    <EditForm Model="_formState">
+        @* Standard form fields *@
+        <InputText @bind-Value="_formState.Name" placeholder="Name" />
+        <InputText @bind-Value="_formState.Email" placeholder="Email" />
+        <InputSelect @bind-Value="_formState.RequestType">
+            <option value="">Select...</option>
+            <option value="Refund">Refund</option>
+            <option value="Replace">Replace</option>
+            <option value="Repair">Repair</option>
+        </InputSelect>
+    </EditForm>
+
+    @* AI Assistant Panel *@
+    <div class="ai-assistant">
+        <div class="messages">
+            @foreach (var msg in _messages)
+            {
+                <div class="message @msg.Role">@msg.Text</div>
+            }
+        </div>
+        <input @bind="_userInput" @onkeydown="HandleKeyDown" placeholder="Ask for help..." />
+    </div>
+</div>
+
+@code {
+    private AIAgent _agent = default!;
+    private FormState _formState = new();
+    private List<ChatMessage> _messages = new();
+    private string _userInput = "";
+
+    protected override void OnInitialized()
+    {
+        var chatClient = ServiceProvider.GetRequiredService<IChatClient>();
+        var formTools = new FormTools(_formState, NavigationManager);
+
+        _agent = new ChatClientAgent(chatClient, new ChatClientAgentOptions
+        {
+            Name = "FormAssistant",
+            Instructions = "Help fill the RMA form. Walk through fields. Validate inputs.",
+            ChatOptions = new ChatOptions
+            {
+                Tools =
+                [
+                    AIFunctionFactory.Create(formTools.UpdateField),
+                    AIFunctionFactory.Create(formTools.ClearField),
+                    AIFunctionFactory.Create(formTools.ValidateForm),
+                    AIFunctionFactory.Create(formTools.SubmitForm)
+                ]
+            }
+        });
+    }
+
+    private async Task HandleKeyDown(KeyboardEventArgs e)
+    {
+        if (e.Key != "Enter" || string.IsNullOrWhiteSpace(_userInput)) return;
+
+        _messages.Add(new(ChatRole.User, _userInput));
+        var response = await _agent.RunAsync(_userInput);
+        _messages.Add(new(ChatRole.Assistant, response.Text));
+        _userInput = "";
+        StateHasChanged();
+    }
+}
+```
+
+---
+
+## Checklist
+
+- [ ] State-based control strategy: UI metadata extracted as JSON
+- [ ] Tool definitions: granular (Update, Clear, Validate, Submit separately)
+- [ ] Tool descriptions: precise and action-oriented
+- [ ] Suggested prompts: buttons showing example interactions
+- [ ] Double validation: LLM validates + app validates
+- [ ] Semantic navigation: routes stored as embeddings (if applicable)
+- [ ] Voice integration: Azure Speech Services (if applicable)
+- [ ] Multilingual: schema stays in one language, LLM translates
+
+---
+
+## References
+
+- `.wiki/microsoft-agent-framework/natural-language-first-blazor-ai-user-experience.md`
+- `setup.md` — Core setup
+- `../backend/database/vector-search-rag.md` — Embeddings for semantic navigation
+
+---
+
+*MORPH-SPEC by Polymorphism Tech*