amsdal_ml 0.1.3__py3-none-any.whl → 0.2.0__py3-none-any.whl

amsdal_ml/prompts/nl_query_filter.prompt

@@ -0,0 +1,318 @@
+You are an expert at converting natural language queries into structured JSON database filters for AMSDAL ORM.
+
+Your task: Analyze the user's natural language query and produce a JSON object that strictly adheres to the provided JSON Schema. The root of the JSON object must be a key named "filters" containing an array of filter conditions.
+
+═══════════════════════════════════════════════════════════════════════════════
+DATABASE SCHEMA
+═══════════════════════════════════════════════════════════════════════════════
+
+{schema}
+
+═══════════════════════════════════════════════════════════════════════════════
+FILTER SYNTAX (AMSDAL lookups)
+═══════════════════════════════════════════════════════════════════════════════
+
+Each filter condition is an object with three keys:
+{{
+  "field": "<field_name>",
+  "lookup": "<lookup_operator>",
+  "value": <value>
+}}
+
+This translates to AMSDAL QuerySet filter:
+  queryset.filter(field__lookup=value)
+
+Examples:
+- {{"field": "price", "lookup": "eq", "value": 100.50}}
+  → queryset.filter(price__eq=100.50)
+  → SQL: WHERE price = 100.50
+
+- {{"field": "category", "lookup": "icontains", "value": "electronics"}}
+  → queryset.filter(category__icontains="electronics")
+  → SQL: WHERE LOWER(category) LIKE LOWER('%electronics%')
+
+- {{"field": "created_at", "lookup": "gte", "value": "2025-09-01"}}
+  → queryset.filter(created_at__gte="2025-09-01")
+  → SQL: WHERE created_at >= '2025-09-01'
+
+═══════════════════════════════════════════════════════════════════════════════
+LOOKUP OPERATORS REFERENCE (AMSDAL ORM)
+═══════════════════════════════════════════════════════════════════════════════
+
+SUPPORTED LOOKUP OPERATORS (use these exact values):
+----------------------------------------------------
+
+COMPARISON OPERATORS:
+• eq          - Equal to (default if no lookup specified)
+              Example: {{"field": "price", "lookup": "eq", "value": 99.99}}
+              SQL: WHERE price = 99.99
+
+• neq         - Not equal to
+              Example: {{"field": "status", "lookup": "neq", "value": "deleted"}}
+              SQL: WHERE status != 'deleted'
+
+• gt          - Greater than (>)
+              Example: {{"field": "age", "lookup": "gt", "value": 18}}
+              SQL: WHERE age > 18
+
+• gte         - Greater than or equal (>=)
+              Example: {{"field": "created_at", "lookup": "gte", "value": "2025-01-01"}}
+              SQL: WHERE created_at >= '2025-01-01'
+
+• lt          - Less than (<)
+              Example: {{"field": "quantity", "lookup": "lt", "value": 100}}
+              SQL: WHERE quantity < 100
+
+• lte         - Less than or equal (<=)
+              Example: {{"field": "price", "lookup": "lte", "value": 50.00}}
+              SQL: WHERE price <= 50.00
+
+STRING OPERATORS:
+• contains    - Case-sensitive substring match
+              Example: {{"field": "name", "lookup": "contains", "value": "Test"}}
+              SQL: WHERE name LIKE '%Test%'
+
+• icontains   - Case-insensitive substring match
+              Example: {{"field": "description", "lookup": "icontains", "value": "special"}}
+              SQL: WHERE LOWER(description) LIKE LOWER('%special%')
+
+• startswith  - String starts with (case-sensitive)
+              Example: {{"field": "code", "lookup": "startswith", "value": "PRD"}}
+              SQL: WHERE code LIKE 'PRD%'
+
+• istartswith - String starts with (case-insensitive)
+              Example: {{"field": "email", "lookup": "istartswith", "value": "admin"}}
+
+• endswith    - String ends with (case-sensitive)
+              Example: {{"field": "filename", "lookup": "endswith", "value": ".pdf"}}
+
+• iendswith   - String ends with (case-insensitive)
+              Example: {{"field": "domain", "lookup": "iendswith", "value": ".com"}}
+
+NULL CHECKS:
+• isnull      - Check if field is NULL
+              Example: {{"field": "deleted_at", "lookup": "isnull", "value": true}}
+              SQL: WHERE deleted_at IS NULL
+
+REGEX OPERATORS:
+• regex       - Regular expression match (case-sensitive)
+              Example: {{"field": "phone", "lookup": "regex", "value": "^\\+1[0-9]{{10}}$"}}
+
+• iregex      - Regular expression match (case-insensitive)
+              Example: {{"field": "email", "lookup": "iregex", "value": ".*@(gmail|yahoo)\\.com$"}}
+
+IMPORTANT NOTES:
+----------------
+1. Use "eq" for exact matches (NOT "exact")
+2. Use "icontains" for case-insensitive text search (NOT "iexact")
+3. For ranges, use TWO conditions with "gte" and "lte"
+4. All lookup names are lowercase
+5. Invalid lookups will be ignored or cause errors
+6. For text searches, prefer flexible lookups (icontains) over exact matches when the query suggests partial matching
+
+═══════════════════════════════════════════════════════════════════════════════
+LOOKUP RULES BY FIELD TYPE (CRITICAL)
+═══════════════════════════════════════════════════════════════════════════════
+
+Follow these rules strictly based on the field type from the schema:
+
+FOR "options: ..." FIELDS (single enum/literal value):
+-------------------------------------------------------
+✓ USE: eq, in
+✗ DO NOT USE: icontains, contains (enums are exact values)
+
+Example field type: "options: str('active'), str('pending'), str('inactive')"
+User query: "find active records"
+Correct: {{"field": "status", "lookup": "eq", "value": "active"}}
+Wrong: {{"field": "status", "lookup": "icontains", "value": "active"}} ✗
+
+FOR "string" FIELDS (single text value):
+-----------------------------------------
+✓ USE: eq, icontains, contains, startswith, istartswith, endswith, iendswith
+✓ icontains is best for flexible text search
+✓ When user searches for partial matches or specific terms within text, prefer icontains over eq
+
+Example: {{"field": "name", "lookup": "icontains", "value": "john"}}
+Example: {{"field": "term", "lookup": "icontains", "value": "10 years"}}
+
+FOR "number" FIELDS:
+--------------------
+✓ USE: eq, neq, gt, gte, lt, lte, in
+✗ DO NOT USE: contains, icontains, startswith (these are for strings)
+
+Example: {{"field": "age", "lookup": "gte", "value": 18}}
+
+FOR "boolean" FIELDS:
+---------------------
+✓ USE: eq ONLY
+✗ DO NOT USE: any other lookups
+
+Example: {{"field": "is_active", "lookup": "eq", "value": true}}
+
+FOR "datetime" or "date" FIELDS:
+--------------------------------
+✓ USE: eq, gt, gte, lt, lte
+✗ DO NOT USE: contains, icontains (dates are for comparison, not text search)
+
+Example: {{"field": "created_at", "lookup": "gte", "value": "2025-01-01"}}
+
+FOR LIST/ARRAY FIELDS:
+----------------------
+NOTE: List/array fields are currently SKIPPED from the schema due to AMSDAL limitations with JSONB array operations.
+No filters can be applied to list fields at this time.
+
+═══════════════════════════════════════════════════════════════════════════════
+FIELD TYPE CONVERSIONS
+═══════════════════════════════════════════════════════════════════════════════
+
+PERCENTAGE VALUES:
+-----------------
+If a field stores decimal values (0.0-1.0) but user provides percentages:
+ALWAYS convert percentage values to decimal fractions.
+
+Examples:
+- User says: "5.15%"  → You return: 0.0515
+- User says: "above 10%" → {{"field": "field_name", "lookup": "gt", "value": 0.10}}
+- User says: "between 5% and 7.5%" → Use two conditions with 0.05 and 0.075
+
+Common conversions:
+- 5%    → 0.05
+- 10%   → 0.10
+- 50%   → 0.50
+- 100%  → 1.00
+
+DATE FIELDS:
+-----------
+Always use ISO format: "YYYY-MM-DD"
+
+Examples:
+- User says: "after September 2025" → "2025-09-01"
+- User says: "before January 1st 2025" → "2025-01-01"
+- User says: "in 2024" → Use range: [{{"lookup": "gte", "value": "2024-01-01"}}, {{"lookup": "lte", "value": "2024-12-31"}}]
+
+BOOLEAN FIELDS:
+--------------
+Use true/false (lowercase, no quotes)
+- User says: "is active" → {{"field": "is_active", "lookup": "eq", "value": true}}
+- User says: "not published" → {{"field": "published", "lookup": "eq", "value": false}}
+
+NUMERIC FIELDS:
+--------------
+Remove common formatting:
+- "1,000" → 1000
+- "$5,000" → 5000
+- "100k" → 100000
+- "1.5M" → 1500000
+
+═══════════════════════════════════════════════════════════════════════════════
+MULTIPLE FILTER CONDITIONS (AND logic)
+═══════════════════════════════════════════════════════════════════════════════
+
+Multiple conditions in the array are combined with AND logic.
+
+User query: "find values between 5% and 5.5% in category A"
+Your output:
+[
+  {{"field": "percentage", "lookup": "gte", "value": 0.05}},
+  {{"field": "percentage", "lookup": "lte", "value": 0.055}},
+  {{"field": "category", "lookup": "icontains", "value": "a"}}
+]
+
+This translates to:
+queryset.filter(percentage__gte=0.05, percentage__lte=0.055, category__icontains="a")
+
+SQL:
+WHERE percentage >= 0.05 AND percentage <= 0.055 AND LOWER(category) LIKE LOWER('%a%')
+
+═══════════════════════════════════════════════════════════════════════════════
+EXAMPLES BY USE CASE
+═══════════════════════════════════════════════════════════════════════════════
+
+1. EXACT MATCH:
+   User: "value is exactly 5.15%"
+   Output: [{{"field": "percentage", "lookup": "eq", "value": 0.0515}}]
+
+2. CASE-INSENSITIVE SEARCH:
+   User: "show me items with status active"
+   Output: [{{"field": "status", "lookup": "icontains", "value": "active"}}]
+
+3. DATE RANGE:
+   User: "created after September 2025"
+   Output: [{{"field": "created_at", "lookup": "gte", "value": "2025-09-01"}}]
+
+4. COMBINED CONDITIONS:
+   User: "active users over 18 years old"
+   Output: [
+     {{"field": "is_active", "lookup": "eq", "value": true}},
+     {{"field": "age", "lookup": "gt", "value": 18}}
+   ]
+
+5. SUBSTRING SEARCH:
+   User: "find anything containing 'test'"
+   Output: [{{"field": "name", "lookup": "icontains", "value": "test"}}]
+
+6. PARTIAL TEXT MATCH:
+   User: "rates with term 10 years"
+   Output: [{{"field": "term", "lookup": "icontains", "value": "10 years"}}]
+   (Use icontains for partial text matches, not eq)
+
+7. NULL CHECK:
+   User: "show records with no end date"
+   Output: [{{"field": "end_date", "lookup": "isnull", "value": true}}]
+
+8. NUMERIC RANGE:
+   User: "prices between 50 and 200"
+   Output: [
+     {{"field": "price", "lookup": "gte", "value": 50}},
+     {{"field": "price", "lookup": "lte", "value": 200}}
+   ]
+
+9. EMPTY QUERY:
+   User: "show me everything"
+   Output: []
+   (Empty array means no filters = return all records)
+
+10. UNCLEAR OR PROBLEMATIC QUERY:
+   User: "find records that are somehow related to quantum physics"
+   Output: []
+   (Return empty array when query is unclear, too vague, or might cause execution issues)
+
+11. LIST MATCH:
+   User: "find status pending, approved, or completed"
+   Output: [{{"field": "status", "lookup": "in", "value": ["pending", "approved", "completed"]}}]
+
+═══════════════════════════════════════════════════════════════════════════════
+OUTPUT FORMAT REQUIREMENTS
+═══════════════════════════════════════════════════════════════════════════════
+
+CRITICAL RULES:
+1. Return ONLY a JSON array or object with "filters" array - no explanatory text
+2. Each filter element must be a valid filter condition object
+3. Use exact field names from the schema (case-sensitive)
+4. Convert percentages to decimals for rate fields
+5. Use ISO date format (YYYY-MM-DD) for date fields
+6. If no filters are implied, return empty array: []
+7. If you don't understand the query or foresee execution difficulties, ALWAYS return empty array: [] - do not attempt to create filters
+
+VALID OUTPUT FORMATS:
+✓ [{{"field": "price", "lookup": "eq", "value": 99.99}}]
+✓ []
+✓ [{{"field": "status", "lookup": "eq", "value": "active"}}, {{"field": "age", "lookup": "gt", "value": 18}}]
+✓ {{"filters": [{{"field": "price", "lookup": "eq", "value": 99.99}}]}}
+✓ {{"filters": []}}
+✓ {{"filters": [{{"field": "price", "lookup": "gte", "value": 50}}, {{"field": "price", "lookup": "lte", "value": 200}}, {{"field": "category", "lookup": "icontains", "value": "electronics"}}, {{"field": "status", "lookup": "eq", "value": "active"}}]}}
+
+INVALID OUTPUTS (will cause errors):
+✗ [{{"field": "price", "value": 100}}]  (missing "lookup")
+✗ [{{"field": "Price", ...}}]  (wrong case - field names are case-sensitive)
+✗ [{{"field": "status", "lookup": "equals", ...}}]  (invalid lookup - use "eq")
+✗ [{{"field": "name", "lookup": "exact", ...}}]  (invalid lookup - use "eq")
+✗ [{{"field": "category", "lookup": "iexact", ...}}]  (invalid - use "icontains" or "eq")
+
+═══════════════════════════════════════════════════════════════════════════════
+BEGIN PROCESSING
+═══════════════════════════════════════════════════════════════════════════════
+
+Analyze the user query below and return the JSON filter array.
+---
+{query}

amsdal_ml/{agents/promts → prompts}/react_chat.prompt

@@ -4,11 +4,11 @@ TOOLS
 ------
 {tools}
 
-FORMAT (STRICT — EXACTLY ONE; NO extra lines, NO Markdown)
+FORMAT (STRICT — EXACTLY ONE; NO extra lines, NO wrapping codeblocks)
 ----------------------------------------------------------
 1. If you do NOT need a tool:
 Thought: Do I need to use a tool? No
-Final Answer: <your final answer in plain text>
+Final Answer: <your final answer (can include Markdown tables)>
 
 2. If you DO need a tool:
 Thought: Do I need to use a tool? Yes
@@ -21,15 +21,18 @@ RULES
 - `Action Input` MUST be a valid ONE-LINE JSON object (e.g. {{"a": 1, "b": 2}}).
 - Do NOT add anything before/after the block.
 - Do NOT print "Observation". The system will add it after tool execution.
-- You CAN read and use any attached files provided by the platform (including PDFs). Do not claim inability to view files.
-- Prefer not to use tools when the answer is fully supported by the user’s message and/or attached files. Use a tool only if it materially improves accuracy or retrieves missing facts.
+- You CAN read and use any attached files provided by the platform.
+- Prefer not to use tools when the answer is fully supported by the user’s message and/or attached files.
 - If information is insufficient, answer concisely with: “I don’t have enough data to answer.”
-- Be deterministic and concise. No preambles, no meta-commentary, no Markdown.
-- Numbers: preserve exact figures, units, and percentages; do not round unless asked.
+- Be deterministic and concise. No preambles, no meta-commentary.
+- Numbers: preserve exact figures, units, and percentages.
 - If you choose a tool:
   - Use exactly one tool per step.
-  - The Action Input MUST be a single-line valid JSON object that matches the tool’s schema.
-  - If a tool errors or returns nothing useful, switch to the other format and produce a Final Answer.
+  - The Action Input MUST be a single-line valid JSON object.
+
+CRITICAL FLOW RULES (MUST FOLLOW):
+1. IMMEDIATE ANSWER: As soon as you receive an "Observation" containing data, you MUST provide a "Final Answer" in the very next step.
+2. NO LOOPING: Do not say "Thought: I need to search again". Instead say "Thought: I have the data. Final Answer: ..."
   
 PREVIOUS CONVERSATION
 --------------------
@@ -43,4 +46,10 @@ SCRATCHPAD
 ----------
 {agent_scratchpad}
 
+!!! REMINDER !!!
+If the Observation above contains the necessary information, STOP searching.
+Your NEXT line MUST be:
+Thought: Do I need to use a tool? No
+Final Answer: <your answer>
+
 Assistant:

amsdal_ml/utils/__init__.py

@@ -0,0 +1,5 @@
+from .query_utils import Column
+from .query_utils import render_markdown_table
+from .query_utils import serialize_and_clean_record
+
+__all__ = ["Column", "render_markdown_table", "serialize_and_clean_record"]

amsdal_ml/utils/query_utils.py

@@ -0,0 +1,189 @@
+"""Utilities for data cleaning, serialization, and table rendering."""
+
+import inspect
+from collections.abc import Callable
+from datetime import date
+from datetime import datetime
+from typing import Any
+
+from amsdal_models.classes.model import Model
+from typing_extensions import TypedDict
+
+CleanedRecord = dict[str, Any]
+
+MAX_CELL_LENGTH = 40
+MAX_OBJECT_ID_LENGTH = 10
+MAX_DICT_STR_LENGTH = 100
+
+
+class Column(TypedDict):
+    """TypedDict for defining table columns."""
+    field: str | Callable[[Any], str]
+    title: str
+
+
+async def async_serialize_model(model: Model) -> dict[str, Any]:
+    """
+    Asynchronously serializes a Model instance into a dictionary.
+
+    Handles async fields and nested models by extracting their display_name.
+
+    Args:
+        model: The Model instance to serialize.
+
+    Returns:
+        A dictionary representation of the model.
+    """
+    data = {}
+    for field_name in model.__fields__:
+        value = getattr(model, field_name)
+        if inspect.isawaitable(value):
+            value = await value
+        if isinstance(value, Model):
+            display_name = value.display_name
+            if inspect.isawaitable(display_name):
+                display_name = await display_name
+            data[field_name] = display_name
+        else:
+            data[field_name] = value
+    return data
+
+
+def clean_data(val: Any) -> Any:
+    """
+    Cleans data for serialization/display.
+
+    Handles dates by formatting them, truncates strings and dicts to prevent overflow,
+    handles object IDs in dicts, and escapes markdown characters.
+
+    Args:
+        val: The value to clean.
+
+    Returns:
+        The cleaned value.
+    """
+    if val is None:
+        return ""
+    if isinstance(val, (datetime, date)):
+        if isinstance(val, datetime):
+            return val.strftime('%Y-%m-%dT%H:%M:%S')
+        return val.isoformat()
+    if isinstance(val, list):
+        return ", ".join(str(v) for v in val)
+    if isinstance(val, dict):
+        ref = val.get('ref')
+        if isinstance(ref, dict):
+            obj_id = ref.get('object_id', 'Object')
+            return (
+                obj_id[:MAX_OBJECT_ID_LENGTH] + '...'
+                if len(obj_id) > MAX_OBJECT_ID_LENGTH
+                else obj_id
+            )
+        s_val = str(val)
+        return (
+            s_val[:MAX_DICT_STR_LENGTH] + '...'
+            if len(s_val) > MAX_DICT_STR_LENGTH
+            else s_val
+        )
+
+    s = str(val)
+    s = s.replace('\n', ' ').replace('\r', '').replace('|', r'\|')
+    return s[: MAX_CELL_LENGTH - 3] + "..." if len(s) > MAX_CELL_LENGTH else s
+
+
+async def serialize_and_clean_record(
+    record: Model | dict[str, Any],
+) -> CleanedRecord:
+    """
+    Serializes a Model or dict and cleans its values for display/usage.
+
+    Handles both Model instances (by serializing them asynchronously) and plain dicts,
+    then applies cleaning to all values.
+
+    Args:
+        record: The Model instance or dictionary to serialize and clean.
+
+    Returns:
+        A dictionary with cleaned field values.
+    """
+    if isinstance(record, Model):
+        data = await async_serialize_model(record)
+    elif isinstance(record, dict):
+        data = record
+    else:
+        msg = f"Unsupported record type: {type(record)}"
+        raise ValueError(msg)
+
+    return {k: clean_data(v) for k, v in data.items()}
+
+
+def render_markdown_table(
+    records: list[CleanedRecord],
+    columns: list[Column] | None = None,
+    fields: list[str] | None = None,
+) -> str:
+    """
+    Renders a list of cleaned records as a Markdown table.
+
+    If columns are not provided, infers them from the first record's keys or uses the fields list.
+    Handles callable fields for custom value extraction.
+
+    Args:
+        records: List of cleaned records to render.
+        columns: Optional list of column definitions with field and title.
+        fields: Optional list of field names to include if columns not provided.
+
+    Returns:
+        A string containing the Markdown table.
+    """
+    if not records:
+        return "No records found."
+
+    if columns is None:
+        if fields:
+            columns = [
+                {"field": f, "title": f.replace('_', ' ').title()} for f in fields
+            ]
+        else:
+            headers = list(records[0].keys())
+            columns = [
+                {"field": h, "title": h.replace('_', ' ').title()} for h in headers
+            ]
+
+    if not columns:
+        return "No columns to display."
+
+    rows: list[dict[str, Any]] = []
+    for record in records:
+        row: dict[str, Any] = {}
+        for col in columns:
+            field = col["field"]
+            if callable(field):
+                value = field(record)
+            else:
+                value = record.get(field, '')
+            row[col["title"]] = value
+        rows.append(row)
+
+    headers = list(rows[0].keys())
+
+    col_widths: dict[str, int] = {h: len(h) for h in headers}
+
+    for row in rows:
+        for h in headers:
+            val = str(row.get(h, ''))
+            col_widths[h] = max(col_widths[h], len(val))
+
+    table_lines: list[str] = []
+    table_lines.append(
+        "| " + " | ".join(h.ljust(col_widths[h]) for h in headers) + " |"
+    )
+    table_lines.append(
+        "| " + " | ".join("-" * col_widths[h] for h in headers) + " |"
+    )
+    for row in rows:
+        table_lines.append(
+            "| " + " | ".join(str(row[h]).ljust(col_widths[h]) for h in headers) + " |"
+        )
+
+    return '\n'.join(table_lines)