claude-smart 0.2.41 → 0.2.43

package/plugin/vendor/reflexio/reflexio/server/prompt/prompt_bank/profile_update_instruction_start/v1.0.0.prompt.md

@@ -0,0 +1,140 @@
+---
+active: false
+description: "System instruction for profile extraction (extraction-only, no delete/mention)"
+changelog: "Extraction-only: removed delete/mention operations. Output format changed to {profiles: [...]}. Deduplication against existing profiles handled separately by ProfileDeduplicator."
+variables:
+  - agent_context_prompt
+  - context_prompt
+  - extraction_definition_prompt
+  - metadata_definition_prompt
+---
+[Context of user interactions]
+{agent_context_prompt}
+{context_prompt}
+
+[Goal]
+You are a user personalization learning assistant.
+Your job is to analyze user–agent interactions and extract **salient information about the user** that should shape how an AI agent communicates with and serves this user in future conversations.
+
+A "profile" can be:
+• **Factual information**: Direct facts about the user (name, birthday, occupation, location)
+• **Work & expertise**: Professional role, technical skills, domain knowledge, tools used daily
+• **Goals & projects**: Current objectives, ongoing projects, deadlines, milestones
+• **Life circumstances**: Living situation, health considerations, family context, time constraints
+• **Relationships & family**: Family members, pets, key people in their life
+• **Domain / environment facts**: Stable facts about the user's working environment that the agent needs to know to serve them correctly — schema details (table names, column types, units of measurement), join paths, metric definitions the user enforces, tool quirks or limitations the user works around, file-format conventions, and similar reference knowledge. These are properties of the user's data or tooling, not of the agent's behavior.
+• **Inferred personalization signals**: Patterns derived from behavior or multi-turn inference that would cause an agent to meaningfully change how it responds
+
+Profiles may be extracted from:
+• Explicit statements ("I prefer sushi")
+• Implicit signals (implied preference/acceptance/rejection)
+• Multi-turn inference (a stable pattern across multiple turns where the user never explicitly states a preference, but behavior suggests one)
+
+**Scope boundary — what is NOT a profile:**
+Profiles capture what is *true about the user or their world*. They do NOT capture
+*what the agent should do differently next time*. If a learning is a behavioral
+rule for the agent (e.g., "when the user asks about methodology, stop and propose
+a corrected plan before re-running queries", "always explain root cause before
+proposing code changes"), it belongs in the playbook extractor, not here. Extract
+only the stable facts the agent needs to know; leave the action rules to playbooks.
+
+[Your Task - Follow These Steps]
+
+STEP 1: Analyze and reason through the user interactions below and compare them to existing profiles by the what to extract definition.
+
+STEP 2: Decide what to extract:
+- If the interaction reveals NEW information about the user that is NOT already stored in existing profiles → Extract it as a new profile
+- If the information is already in existing profiles → Do NOT re-extract it
+- If the interaction contains NO relevant profile information → Return empty result
+
+STEP 3: For each profile you extract, you must assign:
+- "content": The profile information (what you learned about the user)
+- "time_to_live": How long this information stays valid (see rules below)
+- "metadata": Extra categorization based on the metadata definition (only if metadata definition is provided)
+
+[Time to Live - Choose One]
+- "infinity" → Facts that rarely change (name, birthday, gender, phone number)
+- "one_year" → Long-term preferences (favorite color, hobby)
+- "one_quarter" → Seasonal preferences (winter activities, holiday traditions)
+- "one_month" → Regular preferences (food preferences, UI preferences)
+- "one_week" → Short-term or situational preferences (current project, temporary need)
+
+[What to Extract - Definitions]
+Only extract profiles that match this content definition: {extraction_definition_prompt}
+
+Only extract metadata that matches this definition (if provided): {metadata_definition_prompt}
+
+[Output Format]
+Return a JSON object. If nothing to extract, return {{"profiles": null}}.
+
+```json
+{{
+    "profiles": [
+        {{
+            "content": "the new profile information",
+            "time_to_live": "such as one_month",
+            "metadata": "metadata value if definition provided"
+        }}
+    ]
+}}
+```
+
+[Examples With Explanations]
+
+Example 1 - Extracting a new profile:
+- Content definition: "food preferences"
+- Metadata definition: "cuisine type (pizza, sushi, pasta, salad, steak)"
+- Existing profiles: []
+- Interaction: "i like to eat pizza"
+- Reasoning: User revealed a food preference. This is new (not in existing profiles). Food preference changes monthly, so time_to_live is "one_month".
+```json
+{{"profiles": [{{"content": "likes pizza", "time_to_live": "one_month", "metadata": "pizza"}}]}}
+```
+
+Example 2 - Extracting a permanent profile:
+- Interaction: "my name is John"
+- Reasoning: User's name is a permanent fact. Use "infinity" for time_to_live.
+```json
+{{"profiles": [{{"content": "name is John", "time_to_live": "infinity"}}]}}
+```
+
+Example 3 - Extracting work context and project:
+- Content definition: "user information"
+- Existing profiles: []
+- Interaction: "I'm a senior backend engineer at Acme Corp, currently migrating our payment service from monolith to microservices. The deadline is end of Q2."
+- Reasoning: Multiple salient facts: role, company, current project, and deadline.
+```json
+{{"profiles": [
+    {{"content": "senior backend engineer at Acme Corp", "time_to_live": "one_year"}},
+    {{"content": "migrating payment service from monolith to microservices with Q2 deadline", "time_to_live": "one_quarter"}}
+]}}
+```
+
+Example 4 - No relevant information:
+- Content definition: "food preferences"
+- Interaction: "what time is it?"
+- Reasoning: The interaction has nothing to do with food preferences. Return empty.
+```json
+{{"profiles": null}}
+```
+
+Example 5 - Domain / environment fact surfaced through error correction:
+- Content definition: "user information and working environment"
+- Existing profiles: []
+- Interaction: Agent attempts `SUM(orders.total_amount)` and fails with "unknown column 'total_amount'". Runs `DESCRIBE orders` and discovers the column is actually `gross_cents`, stored as INTEGER in cents, not dollars. User confirms: "yes, gross_cents is in cents — you'll need to divide by 100 for dollar amounts."
+- Reasoning: This is a stable fact about the user's data schema (table column name, type, unit) that the agent needs to know for any future revenue query. It's not a behavioral rule ("always run DESCRIBE first"); it's a property of the user's data. Store as a domain/environment fact with a long TTL (`one_year`) since schemas don't change often.
+```json
+{{"profiles": [
+    {{"content": "orders.gross_cents stores order revenue as INTEGER in cents (not dollars) — divide by 100 for dollar amounts", "time_to_live": "one_year"}}
+]}}
+```
+
+Note: if the same session surfaces a behavioral rule (e.g., "verify column types with DESCRIBE before aggregating"), that rule belongs in the playbook extractor, not here. This example captures only the fact.
+
+[Important Reminders]
+1. Only extract profiles matching the content definition provided above
+2. If the information is already in existing profiles, do NOT re-extract it
+3. Always include time_to_live for new profiles
+4. Only include metadata if a metadata definition is provided
+5. **Forget / delete / remove requests.** When the user explicitly asks to forget, delete, or stop remembering a stored fact (e.g. "forget that I like X", "remove my interest in Y", "don't remember that I work at Z"), emit a single profile whose `content` begins with "Requested removal of" and names the topic in positive form. Do NOT rephrase the request as a new fact ("User is no longer interested in X", "User stopped caring about Y") — that looks like a fact update and will be merged into a zombie profile instead of erasing the original. The downstream deduplicator recognizes the "Requested removal of…" phrasing and uses it to delete the matching existing profile outright.
+6. Return {{"profiles": null}} if there is nothing to extract

package/plugin/vendor/reflexio/reflexio/server/prompt/prompt_bank/profile_update_instruction_start/v1.1.0.prompt.md

@@ -0,0 +1,160 @@
+---
+active: true
+description: "System instruction for resumable profile extraction with optional pending-request attachment"
+changelog: "v1.1.0: Adds resumable extraction guidance for finish_extraction and optional attach_pending_info_request while preserving the extraction-only profile output schema."
+variables:
+  - agent_context_prompt
+  - context_prompt
+  - extraction_definition_prompt
+  - metadata_definition_prompt
+---
+[Context of user interactions]
+{agent_context_prompt}
+{context_prompt}
+
+[Goal]
+You are a user personalization learning assistant.
+Your job is to analyze user–agent interactions and extract **salient information about the user** that should shape how an AI agent communicates with and serves this user in future conversations.
+
+[Resumable Extraction Mode]
+When tool calling is available, **`finish_extraction` is the only *required* tool — you must always end the run by calling it.**
+
+• `attach_pending_info_request` — optional. When Prior Knowledge already lists a relevant pending request, and you are confident this run should resume when that request is answered, attach this run to it by reusing its `pending_tool_call_id`.
+
+Attachment is **non-blocking**: after calling it, keep extracting and still call `finish_extraction` now with the profiles you can extract. Use resolved feedback or Prior Knowledge only when relevant to the current window.
+
+A "profile" can be:
+• **Factual information**: Direct facts about the user (name, birthday, occupation, location)
+• **Work & expertise**: Professional role, technical skills, domain knowledge, tools used daily
+• **Goals & projects**: Current objectives, ongoing projects, deadlines, milestones
+• **Life circumstances**: Living situation, health considerations, family context, time constraints
+• **Relationships & family**: Family members, pets, key people in their life
+• **Domain / environment facts**: Stable facts about the user's working environment that the agent needs to know to serve them correctly — schema details (table names, column types, units of measurement), join paths, metric definitions the user enforces, tool quirks or limitations the user works around, file-format conventions, and similar reference knowledge. These are properties of the user's data or tooling, not of the agent's behavior.
+• **Inferred personalization signals**: Patterns derived from behavior or multi-turn inference that would cause an agent to meaningfully change how it responds
+
+Profiles may be extracted from:
+• Explicit statements ("I prefer sushi")
+• Implicit signals (implied preference/acceptance/rejection)
+• Multi-turn inference (a stable pattern across multiple turns where the user never explicitly states a preference, but behavior suggests one)
+
+**Scope boundary — what is NOT a profile:**
+Profiles capture what is *true about the user or their world*. They do NOT capture
+*what the agent should do differently next time*. If a learning is a behavioral
+rule for the agent (e.g., "when the user asks about methodology, stop and propose
+a corrected plan before re-running queries", "always explain root cause before
+proposing code changes"), it belongs in the playbook extractor, not here. Extract
+only the stable facts the agent needs to know; leave the action rules to playbooks.
+
+[Your Task - Follow These Steps]
+
+STEP 1: Analyze and reason through the user interactions below and compare them to existing profiles by the what to extract definition.
+
+STEP 2: Decide what to extract:
+- If the interaction reveals NEW information about the user that is NOT already stored in existing profiles → Extract it as a new profile
+- If the information is already in existing profiles → Do NOT re-extract it
+- If the interaction contains NO relevant profile information → Return empty result
+
+STEP 3: For each profile you extract, you must assign:
+- "content": The profile information (what you learned about the user)
+- "time_to_live": How long this information stays valid (see rules below)
+- "metadata": Extra categorization based on the metadata definition (only if metadata definition is provided)
+
+[Time to Live - Choose One]
+- "infinity" → Facts that rarely change (name, birthday, gender, phone number)
+- "one_year" → Long-term preferences (favorite color, hobby)
+- "one_quarter" → Seasonal preferences (winter activities, holiday traditions)
+- "one_month" → Regular preferences (food preferences, UI preferences)
+- "one_week" → Short-term or situational preferences (current project, temporary need)
+
+[What to Extract - Definitions]
+Only extract profiles that match this content definition: {extraction_definition_prompt}
+
+Only extract metadata that matches this definition (if provided): {metadata_definition_prompt}
+
+[Output Format]
+Report your result by calling the `finish_extraction` tool with a single JSON argument of the shape below. Put the JSON in the tool call — do not write it as a plain-text reply. If there is nothing to extract, call `finish_extraction` with {{"profiles": null}}.
+
+In the rare case described under Resumable Extraction Mode above, you may call `attach_pending_info_request` before `finish_extraction`; it is non-blocking, so always still call `finish_extraction` (see Example 6).
+
+```json
+{{
+    "profiles": [
+        {{
+            "content": "the new profile information",
+            "time_to_live": "such as one_month",
+            "metadata": "metadata value if definition provided"
+        }}
+    ]
+}}
+```
+
+[Examples With Explanations]
+In each example, the JSON shown is the argument you pass to `finish_extraction` — not text to print.
+
+Example 1 - Extracting a new profile:
+- Content definition: "food preferences"
+- Metadata definition: "cuisine type (pizza, sushi, pasta, salad, steak)"
+- Existing profiles: []
+- Interaction: "i like to eat pizza"
+- Reasoning: User revealed a food preference. This is new (not in existing profiles). Food preference changes monthly, so time_to_live is "one_month".
+```json
+{{"profiles": [{{"content": "likes pizza", "time_to_live": "one_month", "metadata": "pizza"}}]}}
+```
+
+Example 2 - Extracting a permanent profile:
+- Interaction: "my name is John"
+- Reasoning: User's name is a permanent fact. Use "infinity" for time_to_live.
+```json
+{{"profiles": [{{"content": "name is John", "time_to_live": "infinity"}}]}}
+```
+
+Example 3 - Extracting work context and project:
+- Content definition: "user information"
+- Existing profiles: []
+- Interaction: "I'm a senior backend engineer at Acme Corp, currently migrating our payment service from monolith to microservices. The deadline is end of Q2."
+- Reasoning: Multiple salient facts: role, company, current project, and deadline.
+```json
+{{"profiles": [
+    {{"content": "senior backend engineer at Acme Corp", "time_to_live": "one_year"}},
+    {{"content": "migrating payment service from monolith to microservices with Q2 deadline", "time_to_live": "one_quarter"}}
+]}}
+```
+
+Example 4 - No relevant information:
+- Content definition: "food preferences"
+- Interaction: "what time is it?"
+- Reasoning: The interaction has nothing to do with food preferences. Return empty.
+```json
+{{"profiles": null}}
+```
+
+Example 5 - Domain / environment fact surfaced through error correction:
+- Content definition: "user information and working environment"
+- Existing profiles: []
+- Interaction: Agent attempts `SUM(orders.total_amount)` and fails with "unknown column 'total_amount'". Runs `DESCRIBE orders` and discovers the column is actually `gross_cents`, stored as INTEGER in cents, not dollars. User confirms: "yes, gross_cents is in cents — you'll need to divide by 100 for dollar amounts."
+- Reasoning: This is a stable fact about the user's data schema (table column name, type, unit) that the agent needs to know for any future revenue query. It's not a behavioral rule ("always run DESCRIBE first"); it's a property of the user's data. Store as a domain/environment fact with a long TTL (`one_year`) since schemas don't change often.
+```json
+{{"profiles": [
+    {{"content": "orders.gross_cents stores order revenue as INTEGER in cents (not dollars) — divide by 100 for dollar amounts", "time_to_live": "one_year"}}
+]}}
+```
+
+Note: if the same session surfaces a behavioral rule (e.g., "verify column types with DESCRIBE before aggregating"), that rule belongs in the playbook extractor, not here. This example captures only the fact.
+
+Example 6 - `attach_pending_info_request` when the question is already pending, then finish:
+- Interaction: User: "my name is jack — please remember the canonical deployment target for my service. There's an internal standard, but I'm not stating it here."
+- Prior Knowledge contains `pending_tool_call_id: ptc_7f3a91` for the canonical deployment target.
+- Reasoning: Extract "name is jack" now. The canonical deployment target is missing and cannot be derived. Because Prior Knowledge already lists the exact pending question, attach to it so the run resumes when answered. Only pass a `pending_tool_call_id` that appears verbatim in Prior Knowledge.
+- Call `attach_pending_info_request`, then `finish_extraction`:
+```json
+{{"pending_tool_call_id": "ptc_7f3a91", "why_relevant": "Same org deployment standard the user wants stored; resume when answered."}}
+{{"profiles": [{{"content": "name is jack", "time_to_live": "infinity"}}]}}
+```
+
+[Important Reminders]
+1. Only extract profiles matching the content definition provided above
+2. If the information is already in existing profiles, do NOT re-extract it
+3. Always include time_to_live for new profiles
+4. Only include metadata if a metadata definition is provided
+5. **Forget / delete / remove requests.** When the user explicitly asks to forget, delete, or stop remembering a stored fact (e.g. "forget that I like X", "remove my interest in Y", "don't remember that I work at Z"), emit a single profile whose `content` begins with "Requested removal of" and names the topic in positive form. Do NOT rephrase the request as a new fact ("User is no longer interested in X", "User stopped caring about Y") — that looks like a fact update and will be merged into a zombie profile instead of erasing the original. The downstream deduplicator recognizes the "Requested removal of…" phrasing and uses it to delete the matching existing profile outright.
+6. Call `finish_extraction` with {{"profiles": null}} if there is nothing to extract

package/plugin/vendor/reflexio/reflexio/server/prompt/prompt_bank/profile_update_main/v1.0.0.prompt.md

@@ -0,0 +1,14 @@
+---
+active: true
+description: "Main user prompt for profile extraction with existing profiles for context"
+changelog: "Clarified existing profiles are for context only (do NOT re-extract). Aligned with extraction-only flow."
+variables:
+  - existing_profiles
+  - interactions
+---
+[Existing Profiles]
+These profiles are already stored for this user (for context — do NOT re-extract these):
+[{existing_profiles}]
+
+[User Interactions to Analyze]
+{interactions}

package/plugin/vendor/reflexio/reflexio/server/prompt/prompt_bank/query_reformulation/v1.0.0.prompt.md

@@ -0,0 +1,19 @@
+---
+active: true
+description: "Reformulates a user search query into clean, standalone natural language"
+variables:
+  - query
+  - conversation_context_block
+---
+Reformulate the user's search query into a clean, standalone natural language query.
+Resolve any pronouns, ellipsis, or implicit references from conversation context.
+Expand abbreviations, fix grammar, and normalize terminology.
+Do NOT add FTS syntax, OR operators, or synonym expansion -- just produce clean natural language.
+When conversation context is provided, extract the core problem from the full conversation -- the user's latest query alone may be vague (e.g., "help"), but earlier turns could contain the real issue.
+{conversation_context_block}
+Examples:
+"agent failed to refund" → agent failed to process refund
+Context: [user]: backup failing [user]: S3 timeout 900s | Query: "help" → backup failure with S3 timeout
+
+Output ONLY the reformulated query, nothing else.
+Query: {query}

package/plugin/vendor/reflexio/reflexio/server/prompt/prompt_bank/rerank_relevance/v1.1.0.prompt.md

@@ -0,0 +1,44 @@
+---
+active: true
+description: "LLM-as-reranker — scores retrieved profiles for query relevance using world knowledge. Bridges brand→category and tool→use-case gaps that lexical-semantic rerankers miss."
+variables:
+  - query
+  - docs_block
+  - num_docs
+---
+You are a relevance judge for a memory-search system. Score each candidate fact 0-10 for how relevant it is to answering the query.
+
+## Use world knowledge
+
+Recognize semantic equivalence that pure lexical matching misses. Patterns:
+- **Brand → category**: a specific store/product/manufacturer name maps to its broader category (a grocery store chain → "grocery", a luxury car maker → "luxury cars", a wearable-tech maker → "fitness device", a streaming-service brand → "streaming subscription"). Score profiles by category membership even when only the brand is named.
+- **Tool → use-case**: a tool/app/card name maps to what the tool DOES (a transit-card brand → "public-transit / getting around city X", a travel-organizer app → "travel itineraries / navigation", a note-taking app → "note-taking / organization", a workout-tracker app → "fitness tracking"). The query may name the use-case while the profile names the tool — score them as relevant.
+- **Action synonyms**: "purchased" = "bought" = "got"; "favorite" = "preferred" = "love most"; "completed" = "finished" = "done with"; "doctor" = "physician" = "MD".
+- **Topic ↔ instance**: a query about a category ("books I read", "movies I watched", "trips I took") matches profiles naming specific instances.
+
+## Scoring rubric
+
+- **10** — directly answers the query (states the fact the query asks about).
+- **7-9** — strong evidence, perhaps requires light inference (brand→category, tool→use-case, near-synonym). **A user-owned tool, card, app, or service whose function maps to the query's activity is 7-9, not 3** — even if the profile content doesn't share keywords with the query.
+- **4-6** — topical but doesn't answer (related entity but different facet).
+- **1-3** — tangentially related (same general domain, different topic).
+- **0** — unrelated.
+
+## Important
+
+- Treat brand/proper-noun mentions as **strong** evidence when the query asks about a category that the brand belongs to.
+- For "help / tips / anxious about / what should I do" questions, the user's stored **tools, cards, apps, services, and named preparations** are usually the answer's anchor — score them 7-9, not 1-3. Generic preference statements (e.g. `user enjoys <CATEGORY>`) are weaker than specific resource ownership (e.g. `user has a <RESOURCE>`, `user downloaded <APP>`) because the resource's affordance is what makes the tip personalised.
+- Do NOT penalize a fact for missing the literal query keyword if the meaning matches.
+- Do NOT score based on date recency or correctness — only relevance to the query topic.
+
+## Query
+
+{query}
+
+## Candidate facts (numbered)
+
+{docs_block}
+
+## Output
+
+Output ONLY a JSON array of {num_docs} numbers (each 0-10) in the same order as the candidates above. No prose, no keys, no explanation. Example for 3 candidates: `[8, 2, 10]`.

package/plugin/vendor/reflexio/reflexio/server/prompt/prompt_bank/shadow_comparison/v1.0.0.prompt.md

@@ -0,0 +1,43 @@
+---
+active: true
+description: "F1 per-turn comparison of two candidate AI responses against the same user message"
+variables:
+  - user_message
+  - request_1_response
+  - request_2_response
+---
+
+You are evaluating two candidate AI assistant responses to the same user
+message. Determine which response is better, or if they are roughly
+equivalent.
+
+Judgment criteria, in order of importance:
+1. Correctness — does the response accurately answer the user's question?
+2. Completeness — does the response cover what the user actually needs?
+3. Actionability — does it give the user a clear next step where applicable?
+4. Style and format — clarity, concision, structure.
+
+Important constraints:
+- Judge only on these two responses and the user message below. Do NOT
+  assume any prior conversation history exists or invent context.
+- Length does not equal quality. A shorter response that fully answers
+  is better than a longer one that meanders.
+- Position (Request 1 vs Request 2) is arbitrary — the labels do not
+  encode any prior knowledge. Evaluate purely on content.
+
+USER MESSAGE
+------------
+{user_message}
+
+REQUEST 1
+---------
+{request_1_response}
+
+REQUEST 2
+---------
+{request_2_response}
+
+Output a JSON object with exactly these fields:
+  better_request: "1" | "2" | "tie"
+  is_significantly_better: true | false  (false ↔ "they're close but 1/2 edges it")
+  comparison_reason: 1-2 sentences explaining the choice.

package/plugin/vendor/reflexio/reflexio/server/prompt/prompt_bank/shadow_content_evaluation/v1.0.0.prompt.md

@@ -0,0 +1,33 @@
+---
+active: false
+description: "DEPRECATED — session-level shadow comparison. Replaced by per-turn shadow_comparison prompt (F1). Kept as historical record so any v1.0.0 verdicts produced under this prompt can still be located in audit logs."
+variables:
+  - agent_context_prompt
+  - success_definition_prompt
+  - tool_can_use
+  - metadata_definition_prompt
+  - interactions
+---
+
+[Instruction]
+Given the interactions below, compare the response 1 and response 2 and determine which one is better based on the context and success definition.
+
+[Context]
+{agent_context_prompt}
+
+[Success definition]
+{success_definition_prompt}
+
+[Interactions]
+User and agent interactions:
+{interactions}
+
+[Output]
+Determine which response is better or tie and output the id of the better response or tie if they are similar.
+Generate the output in the following JSON schema
+#
+```
+{{
+    'better_response': "1 or 2 or tie",
+}}
+```

package/plugin/vendor/reflexio/reflexio/server/prompt/prompt_evaluation/prompt_evaluation_dataset/feedback_extraction_main_v1.jsonl

@@ -0,0 +1,10 @@
+{"test_id": "positive_product_feedback", "description": "Extract positive feedback about product features", "interactions": [{"role": "user", "content": "I absolutely love the new dashboard design! It's so much cleaner and easier to navigate.", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "The search functionality is incredibly fast now, great improvement!", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "agent_context_prompt": "Extract user feedback from interactions", "feedback_definition_prompt": "product usability feedback, feature satisfaction, and overall user sentiment", "expected_label": "The user provides highly positive feedback about the dashboard redesign, specifically praising its clean design and improved navigation. They also express satisfaction with the enhanced search functionality speed, indicating strong approval of recent product improvements."}
+{"test_id": "negative_performance_feedback", "description": "Extract negative feedback about system performance", "interactions": [{"role": "user", "content": "The app keeps crashing when I try to upload large files", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "This is really frustrating, it's been happening for weeks now", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "agent_context_prompt": "Extract user feedback from interactions", "feedback_definition_prompt": "technical problems, system reliability issues, and user frustration levels", "expected_label": "The user reports consistent application crashes during large file uploads, expressing significant frustration with the ongoing technical issue that has persisted for weeks, indicating poor system reliability and negative user experience."}
+{"test_id": "mixed_ui_feedback", "description": "Extract mixed feedback about user interface changes", "interactions": [{"role": "user", "content": "The new color scheme looks nice, but I'm having trouble finding the settings menu", "user_action": "click", "user_action_description": "clicked on various menu items", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "I like the visual improvements but the navigation feels confusing now", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "agent_context_prompt": "Extract user feedback from interactions", "feedback_definition_prompt": "visual design feedback, navigation usability, and user interface experience", "expected_label": "The user provides mixed feedback on UI changes, expressing appreciation for the new color scheme and visual improvements while simultaneously reporting navigation difficulties and confusion in finding the settings menu, indicating a trade-off between aesthetics and usability."}
+{"test_id": "feature_request_feedback", "description": "Extract feedback containing feature requests", "interactions": [{"role": "user", "content": "It would be great if you could add dark mode support", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "Also, I think keyboard shortcuts would make the workflow much faster", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "agent_context_prompt": "Extract user feedback from interactions", "feedback_definition_prompt": "feature requests, workflow improvements, and user enhancement suggestions", "expected_label": "The user suggests two specific feature enhancements: dark mode support for better visual comfort and keyboard shortcuts for improved workflow efficiency, indicating engaged user interest in product improvement and productivity optimization."}
+{"test_id": "support_experience_feedback", "description": "Extract feedback about customer support experience", "interactions": [{"role": "user", "content": "Thank you for the quick response to my ticket!", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "The support agent was very helpful and solved my problem within minutes", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "agent_context_prompt": "Extract user feedback from interactions", "feedback_definition_prompt": "support quality, response time satisfaction, and customer service experience", "expected_label": "The user expresses high satisfaction with customer support, specifically praising the quick response time and the helpfulness of the support agent who efficiently resolved their issue within minutes, indicating excellent customer service experience."}
+{"test_id": "pricing_concern_feedback", "description": "Extract feedback about pricing and value concerns", "interactions": [{"role": "user", "content": "The premium features are nice, but the subscription cost seems quite high", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "I'm not sure if I'm getting enough value for what I'm paying", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "agent_context_prompt": "Extract user feedback from interactions", "feedback_definition_prompt": "pricing concerns, value perception, and subscription satisfaction", "expected_label": "The user expresses concern about the premium subscription cost, acknowledging the quality of premium features while questioning the value proposition and expressing uncertainty about whether the benefits justify the current pricing structure."}
+{"test_id": "accessibility_feedback", "description": "Extract feedback about accessibility and inclusive design", "interactions": [{"role": "user", "content": "I appreciate that you added screen reader support, it makes a huge difference", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "However, some text is still too small to read comfortably even with zoom", "user_action": "scroll", "user_action_description": "scrolled to find readable text", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "agent_context_prompt": "Extract user feedback from interactions", "feedback_definition_prompt": "accessibility features, inclusive design elements, and usability barriers", "expected_label": "The user provides constructive accessibility feedback, expressing gratitude for screen reader support implementation while identifying remaining barriers with text size readability, even when using zoom features, indicating continued need for accessibility improvements."}
+{"test_id": "workflow_efficiency_feedback", "description": "Extract feedback about workflow and productivity", "interactions": [{"role": "user", "content": "The new batch processing feature has saved me hours of work each week", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "I can now process hundreds of documents at once instead of doing them individually", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "agent_context_prompt": "Extract user feedback from interactions", "feedback_definition_prompt": "workflow efficiency, productivity improvements, and feature impact on daily tasks", "expected_label": "The user reports significant productivity gains from the new batch processing feature, quantifying time savings of hours per week and highlighting the ability to process hundreds of documents simultaneously, indicating substantial positive impact on workflow efficiency."}
+{"test_id": "bug_report_feedback", "description": "Extract feedback containing bug reports and technical issues", "interactions": [{"role": "user", "content": "I found a bug where the save button doesn't work on mobile devices", "user_action": "click", "user_action_description": "attempted to click save button multiple times", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "This happens specifically on iPhone Safari, works fine on desktop Chrome", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "agent_context_prompt": "Extract user feedback from interactions", "feedback_definition_prompt": "technical bugs, device-specific issues, and system compatibility problems", "expected_label": "The user reports a device-specific bug affecting the save button functionality on iPhone Safari while noting that the same feature works correctly on desktop Chrome, providing detailed technical context for troubleshooting and cross-platform compatibility issues."}
+{"test_id": "onboarding_experience_feedback", "description": "Extract feedback about user onboarding and initial experience", "interactions": [{"role": "user", "content": "The getting started tutorial was really helpful for new users like me", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "I was up and running within 10 minutes, which is impressive for such a complex tool", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "agent_context_prompt": "Extract user feedback from interactions", "feedback_definition_prompt": "onboarding effectiveness, learning curve, and initial user satisfaction", "expected_label": "The user provides positive feedback about the onboarding experience, praising the tutorial's helpfulness for new users and expressing satisfaction with the quick setup time of 10 minutes despite the tool's complexity, indicating effective user onboarding design."}

package/plugin/vendor/reflexio/reflexio/server/prompt/prompt_evaluation/prompt_evaluation_dataset/profile_update_main_v1.jsonl

@@ -0,0 +1,10 @@
+{"test_id": "food_preferences_basic", "description": "Extract food preferences from simple user interaction", "interactions": [{"role": "user", "content": "I love Italian food, especially pasta carbonara and margherita pizza", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}], "existing_profiles": [], "agent_context_prompt": "Extract user profiles from interactions", "context_prompt": "Analyze the following user interactions to extract relevant user profile information", "profile_content_definition_prompt": "food preferences and dietary choices", "metadata_definition_prompt": "cuisine type from: italian, japanese, mexican, american, indian, chinese", "expected_label": "{\"add\": [{\"content\": \"loves Italian food, especially pasta carbonara and margherita pizza\", \"time_to_live\": \"one_month\", \"metadata\": \"italian\"}]}"}
+{"test_id": "food_preferences_multiple", "description": "Extract multiple food preferences from user interactions", "interactions": [{"role": "user", "content": "I love Italian food, especially pasta carbonara", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "I'm also a big fan of sushi, particularly salmon and tuna rolls", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "existing_profiles": [{"profile_content": "likes Mexican food", "metadata": "cuisine:mexican", "created_timestamp": 1703120000, "last_modified_timestamp": 1703120000}], "agent_context_prompt": "Extract user profiles from interactions", "context_prompt": "Analyze the following user interactions", "profile_content_definition_prompt": "food preferences and dietary choices", "metadata_definition_prompt": "cuisine type from: italian, japanese, mexican, american, indian, chinese", "expected_label": "{\"add\": [{\"content\": \"loves Italian food, especially pasta carbonara\", \"time_to_live\": \"one_month\", \"metadata\": \"italian\"}, {\"content\": \"enjoys sushi, particularly salmon and tuna rolls\", \"time_to_live\": \"one_month\", \"metadata\": \"japanese\"}]}"}
+{"test_id": "hobbies_learning", "description": "Extract hobbies and learning activities", "interactions": [{"role": "user", "content": "I've been learning to play guitar for the past few months", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "I also enjoy reading sci-fi novels in my free time", "user_action": "click", "user_action_description": "clicked on book recommendation", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "existing_profiles": [], "agent_context_prompt": "Extract user profiles from interactions", "context_prompt": "Analyze the following user interactions to extract relevant user profile information", "profile_content_definition_prompt": "hobbies and personal interests", "metadata_definition_prompt": "category from: music, reading, sports, art, technology, cooking", "expected_label": "{\"add\": [{\"content\": \"learning to play guitar\", \"time_to_live\": \"one_year\", \"metadata\": \"music\"}, {\"content\": \"enjoys reading sci-fi novels\", \"time_to_live\": \"one_year\", \"metadata\": \"reading\"}]}"}
+{"test_id": "professional_skills", "description": "Extract professional skills and expertise", "interactions": [{"role": "user", "content": "I work as a software engineer specializing in Python and JavaScript", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "I've recently been learning machine learning and AI development", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "existing_profiles": [{"profile_content": "knows Java programming", "metadata": "skill_level:intermediate", "created_timestamp": 1703120000, "last_modified_timestamp": 1703120000}], "agent_context_prompt": "Extract user profiles from interactions", "context_prompt": "Analyze the following user interactions", "profile_content_definition_prompt": "professional skills and technical expertise", "metadata_definition_prompt": "skill level from: beginner, intermediate, advanced, expert", "expected_label": "{\"add\": [{\"content\": \"software engineer specializing in Python and JavaScript\", \"time_to_live\": \"infinity\", \"metadata\": \"expert\"}, {\"content\": \"learning machine learning and AI development\", \"time_to_live\": \"one_year\", \"metadata\": \"beginner\"}]}"}
+{"test_id": "travel_preferences", "description": "Extract travel and vacation preferences", "interactions": [{"role": "user", "content": "I prefer beach vacations over mountain trips", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "My dream destination is the Maldives for the crystal clear water", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "existing_profiles": [], "agent_context_prompt": "Extract user profiles from interactions", "context_prompt": "Analyze the following user interactions", "profile_content_definition_prompt": "travel preferences and vacation interests", "metadata_definition_prompt": "travel type from: beach, mountain, city, adventure, cultural, relaxation", "expected_label": "{\"add\": [{\"content\": \"prefers beach vacations over mountain trips\", \"time_to_live\": \"one_year\", \"metadata\": \"beach\"}, {\"content\": \"dream destination is the Maldives for crystal clear water\", \"time_to_live\": \"one_year\", \"metadata\": \"beach\"}]}"}
+{"test_id": "profile_deletion", "description": "Delete existing profile when preferences change", "interactions": [{"role": "user", "content": "I actually don't like spicy food anymore, my taste has changed", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}], "existing_profiles": [{"profile_content": "loves spicy food", "metadata": "spice_level:high", "created_timestamp": 1703120000, "last_modified_timestamp": 1703120000}, {"profile_content": "enjoys mild curry", "metadata": "cuisine:indian", "created_timestamp": 1703120000, "last_modified_timestamp": 1703120000}], "agent_context_prompt": "Extract user profiles from interactions", "context_prompt": "Analyze the following user interactions", "profile_content_definition_prompt": "food preferences and dietary choices", "metadata_definition_prompt": "spice level from: none, mild, medium, high", "expected_label": "{\"delete\": [\"loves spicy food\"]}"}
+{"test_id": "fitness_activities", "description": "Extract fitness and exercise habits", "interactions": [{"role": "user", "content": "I go to the gym 5 times a week and focus on strength training", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "I also do yoga on weekends for flexibility and relaxation", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "existing_profiles": [{"profile_content": "enjoys running", "metadata": "activity:cardio", "created_timestamp": 1703120000, "last_modified_timestamp": 1703120000}], "agent_context_prompt": "Extract fitness and health related activities", "context_prompt": "Analyze fitness habits and exercise preferences", "profile_content_definition_prompt": "fitness activities and exercise habits", "metadata_definition_prompt": "activity type from: cardio, strength, flexibility, sports, outdoor", "expected_label": "{\"add\": [{\"content\": \"goes to gym 5 times a week focusing on strength training\", \"time_to_live\": \"one_year\", \"metadata\": \"strength\"}, {\"content\": \"does yoga on weekends for flexibility and relaxation\", \"time_to_live\": \"one_year\", \"metadata\": \"flexibility\"}]}"}
+{"test_id": "shopping_preferences", "description": "Extract shopping habits and brand preferences", "interactions": [{"role": "user", "content": "I always buy organic products when grocery shopping", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "I prefer shopping at local farmers markets over big chain stores", "user_action": "click", "user_action_description": "clicked on farmers market link", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "existing_profiles": [], "agent_context_prompt": "Extract user profiles from interactions", "context_prompt": "Analyze the following user interactions", "profile_content_definition_prompt": "shopping habits and brand preferences", "metadata_definition_prompt": "shopping type from: organic, budget, luxury, local, online, brand_loyal", "expected_label": "{\"add\": [{\"content\": \"always buys organic products when grocery shopping\", \"time_to_live\": \"one_year\", \"metadata\": \"organic\"}, {\"content\": \"prefers shopping at local farmers markets over big chain stores\", \"time_to_live\": \"one_year\", \"metadata\": \"local\"}]}"}
+{"test_id": "entertainment_preferences", "description": "Extract entertainment and media consumption preferences", "interactions": [{"role": "user", "content": "I love watching documentaries about nature and wildlife", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}, {"role": "user", "content": "Netflix is my go-to platform for streaming", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123457}], "existing_profiles": [{"profile_content": "enjoys comedy movies", "metadata": "genre:comedy", "created_timestamp": 1703120000, "last_modified_timestamp": 1703120000}], "agent_context_prompt": "Extract user profiles from interactions", "context_prompt": "Analyze the following user interactions", "profile_content_definition_prompt": "entertainment and media consumption preferences", "metadata_definition_prompt": "content type from: documentary, comedy, drama, action, horror, educational", "expected_label": "{\"add\": [{\"content\": \"loves watching documentaries about nature and wildlife\", \"time_to_live\": \"one_year\", \"metadata\": \"documentary\"}, {\"content\": \"uses Netflix as primary streaming platform\", \"time_to_live\": \"one_year\", \"metadata\": \"platform:netflix\"}]}"}
+{"test_id": "mention_existing_profile", "description": "Mention existing profile when user references it again", "interactions": [{"role": "user", "content": "I'm ordering pizza again for dinner tonight", "user_action": "none", "user_action_description": "", "interacted_image_url": null, "image_encoding": null, "timestamp": 1703123456}], "existing_profiles": [{"profile_content": "loves Italian food, especially pizza", "metadata": "cuisine:italian", "created_timestamp": 1703120000, "last_modified_timestamp": 1703120000}, {"profile_content": "enjoys cooking at home", "metadata": "activity:cooking", "created_timestamp": 1703120000, "last_modified_timestamp": 1703120000}], "agent_context_prompt": "Extract user profiles from interactions", "context_prompt": "Analyze the following user interactions", "profile_content_definition_prompt": "food preferences and dining habits", "metadata_definition_prompt": "cuisine type from: italian, japanese, mexican, american, indian, chinese", "expected_label": "{\"mention\": [\"loves Italian food, especially pizza\"]}"}