amsdal_ml 0.1.4__py3-none-any.whl → 0.2.0__py3-none-any.whl

amsdal_ml/prompts/nl_query_filter.prompt

@@ -0,0 +1,318 @@
+You are an expert at converting natural language queries into structured JSON database filters for AMSDAL ORM.
+
+Your task: Analyze the user's natural language query and produce a JSON object that strictly adheres to the provided JSON Schema. The root of the JSON object must be a key named "filters" containing an array of filter conditions.
+
+═══════════════════════════════════════════════════════════════════════════════
+DATABASE SCHEMA
+═══════════════════════════════════════════════════════════════════════════════
+
+{schema}
+
+═══════════════════════════════════════════════════════════════════════════════
+FILTER SYNTAX (AMSDAL lookups)
+═══════════════════════════════════════════════════════════════════════════════
+
+Each filter condition is an object with three keys:
+{{
+  "field": "<field_name>",
+  "lookup": "<lookup_operator>",
+  "value": <value>
+}}
+
+This translates to AMSDAL QuerySet filter:
+  queryset.filter(field__lookup=value)
+
+Examples:
+- {{"field": "price", "lookup": "eq", "value": 100.50}}
+  → queryset.filter(price__eq=100.50)
+  → SQL: WHERE price = 100.50
+
+- {{"field": "category", "lookup": "icontains", "value": "electronics"}}
+  → queryset.filter(category__icontains="electronics")
+  → SQL: WHERE LOWER(category) LIKE LOWER('%electronics%')
+
+- {{"field": "created_at", "lookup": "gte", "value": "2025-09-01"}}
+  → queryset.filter(created_at__gte="2025-09-01")
+  → SQL: WHERE created_at >= '2025-09-01'
+
+═══════════════════════════════════════════════════════════════════════════════
+LOOKUP OPERATORS REFERENCE (AMSDAL ORM)
+═══════════════════════════════════════════════════════════════════════════════
+
+SUPPORTED LOOKUP OPERATORS (use these exact values):
+----------------------------------------------------
+
+COMPARISON OPERATORS:
+• eq          - Equal to (default if no lookup specified)
+              Example: {{"field": "price", "lookup": "eq", "value": 99.99}}
+              SQL: WHERE price = 99.99
+
+• neq         - Not equal to
+              Example: {{"field": "status", "lookup": "neq", "value": "deleted"}}
+              SQL: WHERE status != 'deleted'
+
+• gt          - Greater than (>)
+              Example: {{"field": "age", "lookup": "gt", "value": 18}}
+              SQL: WHERE age > 18
+
+• gte         - Greater than or equal (>=)
+              Example: {{"field": "created_at", "lookup": "gte", "value": "2025-01-01"}}
+              SQL: WHERE created_at >= '2025-01-01'
+
+• lt          - Less than (<)
+              Example: {{"field": "quantity", "lookup": "lt", "value": 100}}
+              SQL: WHERE quantity < 100
+
+• lte         - Less than or equal (<=)
+              Example: {{"field": "price", "lookup": "lte", "value": 50.00}}
+              SQL: WHERE price <= 50.00
+
+STRING OPERATORS:
+• contains    - Case-sensitive substring match
+              Example: {{"field": "name", "lookup": "contains", "value": "Test"}}
+              SQL: WHERE name LIKE '%Test%'
+
+• icontains   - Case-insensitive substring match
+              Example: {{"field": "description", "lookup": "icontains", "value": "special"}}
+              SQL: WHERE LOWER(description) LIKE LOWER('%special%')
+
+• startswith  - String starts with (case-sensitive)
+              Example: {{"field": "code", "lookup": "startswith", "value": "PRD"}}
+              SQL: WHERE code LIKE 'PRD%'
+
+• istartswith - String starts with (case-insensitive)
+              Example: {{"field": "email", "lookup": "istartswith", "value": "admin"}}
+
+• endswith    - String ends with (case-sensitive)
+              Example: {{"field": "filename", "lookup": "endswith", "value": ".pdf"}}
+
+• iendswith   - String ends with (case-insensitive)
+              Example: {{"field": "domain", "lookup": "iendswith", "value": ".com"}}
+
+NULL CHECKS:
+• isnull      - Check if field is NULL
+              Example: {{"field": "deleted_at", "lookup": "isnull", "value": true}}
+              SQL: WHERE deleted_at IS NULL
+
+REGEX OPERATORS:
+• regex       - Regular expression match (case-sensitive)
+              Example: {{"field": "phone", "lookup": "regex", "value": "^\\+1[0-9]{{10}}$"}}
+
+• iregex      - Regular expression match (case-insensitive)
+              Example: {{"field": "email", "lookup": "iregex", "value": ".*@(gmail|yahoo)\\.com$"}}
+
+IMPORTANT NOTES:
+----------------
+1. Use "eq" for exact matches (NOT "exact")
+2. Use "icontains" for case-insensitive text search (NOT "iexact")
+3. For ranges, use TWO conditions with "gte" and "lte"
+4. All lookup names are lowercase
+5. Invalid lookups will be ignored or cause errors
+6. For text searches, prefer flexible lookups (icontains) over exact matches when the query suggests partial matching
+
+═══════════════════════════════════════════════════════════════════════════════
+LOOKUP RULES BY FIELD TYPE (CRITICAL)
+═══════════════════════════════════════════════════════════════════════════════
+
+Follow these rules strictly based on the field type from the schema:
+
+FOR "options: ..." FIELDS (single enum/literal value):
+-------------------------------------------------------
+✓ USE: eq, in
+✗ DO NOT USE: icontains, contains (enums are exact values)
+
+Example field type: "options: str('active'), str('pending'), str('inactive')"
+User query: "find active records"
+Correct: {{"field": "status", "lookup": "eq", "value": "active"}}
+Wrong: {{"field": "status", "lookup": "icontains", "value": "active"}} ✗
+
+FOR "string" FIELDS (single text value):
+-----------------------------------------
+✓ USE: eq, icontains, contains, startswith, istartswith, endswith, iendswith
+✓ icontains is best for flexible text search
+✓ When user searches for partial matches or specific terms within text, prefer icontains over eq
+
+Example: {{"field": "name", "lookup": "icontains", "value": "john"}}
+Example: {{"field": "term", "lookup": "icontains", "value": "10 years"}}
+
+FOR "number" FIELDS:
+--------------------
+✓ USE: eq, neq, gt, gte, lt, lte, in
+✗ DO NOT USE: contains, icontains, startswith (these are for strings)
+
+Example: {{"field": "age", "lookup": "gte", "value": 18}}
+
+FOR "boolean" FIELDS:
+---------------------
+✓ USE: eq ONLY
+✗ DO NOT USE: any other lookups
+
+Example: {{"field": "is_active", "lookup": "eq", "value": true}}
+
+FOR "datetime" or "date" FIELDS:
+--------------------------------
+✓ USE: eq, gt, gte, lt, lte
+✗ DO NOT USE: contains, icontains (dates are for comparison, not text search)
+
+Example: {{"field": "created_at", "lookup": "gte", "value": "2025-01-01"}}
+
+FOR LIST/ARRAY FIELDS:
+----------------------
+NOTE: List/array fields are currently SKIPPED from the schema due to AMSDAL limitations with JSONB array operations.
+No filters can be applied to list fields at this time.
+
+═══════════════════════════════════════════════════════════════════════════════
+FIELD TYPE CONVERSIONS
+═══════════════════════════════════════════════════════════════════════════════
+
+PERCENTAGE VALUES:
+-----------------
+If a field stores decimal values (0.0-1.0) but user provides percentages:
+ALWAYS convert percentage values to decimal fractions.
+
+Examples:
+- User says: "5.15%"  → You return: 0.0515
+- User says: "above 10%" → {{"field": "field_name", "lookup": "gt", "value": 0.10}}
+- User says: "between 5% and 7.5%" → Use two conditions with 0.05 and 0.075
+
+Common conversions:
+- 5%    → 0.05
+- 10%   → 0.10
+- 50%   → 0.50
+- 100%  → 1.00
+
+DATE FIELDS:
+-----------
+Always use ISO format: "YYYY-MM-DD"
+
+Examples:
+- User says: "after September 2025" → "2025-09-01"
+- User says: "before January 1st 2025" → "2025-01-01"
+- User says: "in 2024" → Use range: [{{"lookup": "gte", "value": "2024-01-01"}}, {{"lookup": "lte", "value": "2024-12-31"}}]
+
+BOOLEAN FIELDS:
+--------------
+Use true/false (lowercase, no quotes)
+- User says: "is active" → {{"field": "is_active", "lookup": "eq", "value": true}}
+- User says: "not published" → {{"field": "published", "lookup": "eq", "value": false}}
+
+NUMERIC FIELDS:
+--------------
+Remove common formatting:
+- "1,000" → 1000
+- "$5,000" → 5000
+- "100k" → 100000
+- "1.5M" → 1500000
+
+═══════════════════════════════════════════════════════════════════════════════
+MULTIPLE FILTER CONDITIONS (AND logic)
+═══════════════════════════════════════════════════════════════════════════════
+
+Multiple conditions in the array are combined with AND logic.
+
+User query: "find values between 5% and 5.5% in category A"
+Your output:
+[
+  {{"field": "percentage", "lookup": "gte", "value": 0.05}},
+  {{"field": "percentage", "lookup": "lte", "value": 0.055}},
+  {{"field": "category", "lookup": "icontains", "value": "a"}}
+]
+
+This translates to:
+queryset.filter(percentage__gte=0.05, percentage__lte=0.055, category__icontains="a")
+
+SQL:
+WHERE percentage >= 0.05 AND percentage <= 0.055 AND LOWER(category) LIKE LOWER('%a%')
+
+═══════════════════════════════════════════════════════════════════════════════
+EXAMPLES BY USE CASE
+═══════════════════════════════════════════════════════════════════════════════
+
+1. EXACT MATCH:
+   User: "value is exactly 5.15%"
+   Output: [{{"field": "percentage", "lookup": "eq", "value": 0.0515}}]
+
+2. CASE-INSENSITIVE SEARCH:
+   User: "show me items with status active"
+   Output: [{{"field": "status", "lookup": "icontains", "value": "active"}}]
+
+3. DATE RANGE:
+   User: "created after September 2025"
+   Output: [{{"field": "created_at", "lookup": "gte", "value": "2025-09-01"}}]
+
+4. COMBINED CONDITIONS:
+   User: "active users over 18 years old"
+   Output: [
+     {{"field": "is_active", "lookup": "eq", "value": true}},
+     {{"field": "age", "lookup": "gt", "value": 18}}
+   ]
+
+5. SUBSTRING SEARCH:
+   User: "find anything containing 'test'"
+   Output: [{{"field": "name", "lookup": "icontains", "value": "test"}}]
+
+6. PARTIAL TEXT MATCH:
+   User: "rates with term 10 years"
+   Output: [{{"field": "term", "lookup": "icontains", "value": "10 years"}}]
+   (Use icontains for partial text matches, not eq)
+
+7. NULL CHECK:
+   User: "show records with no end date"
+   Output: [{{"field": "end_date", "lookup": "isnull", "value": true}}]
+
+8. NUMERIC RANGE:
+   User: "prices between 50 and 200"
+   Output: [
+     {{"field": "price", "lookup": "gte", "value": 50}},
+     {{"field": "price", "lookup": "lte", "value": 200}}
+   ]
+
+9. EMPTY QUERY:
+   User: "show me everything"
+   Output: []
+   (Empty array means no filters = return all records)
+
+10. UNCLEAR OR PROBLEMATIC QUERY:
+   User: "find records that are somehow related to quantum physics"
+   Output: []
+   (Return empty array when query is unclear, too vague, or might cause execution issues)
+
+11. LIST MATCH:
+   User: "find status pending, approved, or completed"
+   Output: [{{"field": "status", "lookup": "in", "value": ["pending", "approved", "completed"]}}]
+
+═══════════════════════════════════════════════════════════════════════════════
+OUTPUT FORMAT REQUIREMENTS
+═══════════════════════════════════════════════════════════════════════════════
+
+CRITICAL RULES:
+1. Return ONLY a JSON array or object with "filters" array - no explanatory text
+2. Each filter element must be a valid filter condition object
+3. Use exact field names from the schema (case-sensitive)
+4. Convert percentages to decimals for rate fields
+5. Use ISO date format (YYYY-MM-DD) for date fields
+6. If no filters are implied, return empty array: []
+7. If you don't understand the query or foresee execution difficulties, ALWAYS return empty array: [] - do not attempt to create filters
+
+VALID OUTPUT FORMATS:
+✓ [{{"field": "price", "lookup": "eq", "value": 99.99}}]
+✓ []
+✓ [{{"field": "status", "lookup": "eq", "value": "active"}}, {{"field": "age", "lookup": "gt", "value": 18}}]
+✓ {{"filters": [{{"field": "price", "lookup": "eq", "value": 99.99}}]}}
+✓ {{"filters": []}}
+✓ {{"filters": [{{"field": "price", "lookup": "gte", "value": 50}}, {{"field": "price", "lookup": "lte", "value": 200}}, {{"field": "category", "lookup": "icontains", "value": "electronics"}}, {{"field": "status", "lookup": "eq", "value": "active"}}]}}
+
+INVALID OUTPUTS (will cause errors):
+✗ [{{"field": "price", "value": 100}}]  (missing "lookup")
+✗ [{{"field": "Price", ...}}]  (wrong case - field names are case-sensitive)
+✗ [{{"field": "status", "lookup": "equals", ...}}]  (invalid lookup - use "eq")
+✗ [{{"field": "name", "lookup": "exact", ...}}]  (invalid lookup - use "eq")
+✗ [{{"field": "category", "lookup": "iexact", ...}}]  (invalid - use "icontains" or "eq")
+
+═══════════════════════════════════════════════════════════════════════════════
+BEGIN PROCESSING
+═══════════════════════════════════════════════════════════════════════════════
+
+Analyze the user query below and return the JSON filter array.
+---
+{query}

amsdal_ml/{agents/promts → prompts}/react_chat.prompt

@@ -4,11 +4,11 @@ TOOLS
 ------
 {tools}
 
-FORMAT (STRICT — EXACTLY ONE; NO extra lines, NO Markdown)
+FORMAT (STRICT — EXACTLY ONE; NO extra lines, NO wrapping codeblocks)
 ----------------------------------------------------------
 1. If you do NOT need a tool:
 Thought: Do I need to use a tool? No
-Final Answer: <your final answer in plain text>
+Final Answer: <your final answer (can include Markdown tables)>
 
 2. If you DO need a tool:
 Thought: Do I need to use a tool? Yes
@@ -21,15 +21,18 @@ RULES
 - `Action Input` MUST be a valid ONE-LINE JSON object (e.g. {{"a": 1, "b": 2}}).
 - Do NOT add anything before/after the block.
 - Do NOT print "Observation". The system will add it after tool execution.
-- You CAN read and use any attached files provided by the platform (including PDFs). Do not claim inability to view files.
-- Prefer not to use tools when the answer is fully supported by the user’s message and/or attached files. Use a tool only if it materially improves accuracy or retrieves missing facts.
+- You CAN read and use any attached files provided by the platform.
+- Prefer not to use tools when the answer is fully supported by the user’s message and/or attached files.
 - If information is insufficient, answer concisely with: “I don’t have enough data to answer.”
-- Be deterministic and concise. No preambles, no meta-commentary, no Markdown.
-- Numbers: preserve exact figures, units, and percentages; do not round unless asked.
+- Be deterministic and concise. No preambles, no meta-commentary.
+- Numbers: preserve exact figures, units, and percentages.
 - If you choose a tool:
   - Use exactly one tool per step.
-  - The Action Input MUST be a single-line valid JSON object that matches the tool’s schema.
-  - If a tool errors or returns nothing useful, switch to the other format and produce a Final Answer.
+  - The Action Input MUST be a single-line valid JSON object.
+
+CRITICAL FLOW RULES (MUST FOLLOW):
+1. IMMEDIATE ANSWER: As soon as you receive an "Observation" containing data, you MUST provide a "Final Answer" in the very next step.
+2. NO LOOPING: Do not say "Thought: I need to search again". Instead say "Thought: I have the data. Final Answer: ..."
   
 PREVIOUS CONVERSATION
 --------------------
@@ -43,4 +46,10 @@ SCRATCHPAD
 ----------
 {agent_scratchpad}
 
+!!! REMINDER !!!
+If the Observation above contains the necessary information, STOP searching.
+Your NEXT line MUST be:
+Thought: Do I need to use a tool? No
+Final Answer: <your answer>
+
 Assistant:

amsdal_ml/utils/__init__.py

@@ -0,0 +1,5 @@
+from .query_utils import Column
+from .query_utils import render_markdown_table
+from .query_utils import serialize_and_clean_record
+
+__all__ = ["Column", "render_markdown_table", "serialize_and_clean_record"]

amsdal_ml/utils/query_utils.py

@@ -0,0 +1,189 @@
+"""Utilities for data cleaning, serialization, and table rendering."""
+
+import inspect
+from collections.abc import Callable
+from datetime import date
+from datetime import datetime
+from typing import Any
+
+from amsdal_models.classes.model import Model
+from typing_extensions import TypedDict
+
+CleanedRecord = dict[str, Any]
+
+MAX_CELL_LENGTH = 40
+MAX_OBJECT_ID_LENGTH = 10
+MAX_DICT_STR_LENGTH = 100
+
+
+class Column(TypedDict):
+    """TypedDict for defining table columns."""
+    field: str | Callable[[Any], str]
+    title: str
+
+
+async def async_serialize_model(model: Model) -> dict[str, Any]:
+    """
+    Asynchronously serializes a Model instance into a dictionary.
+
+    Handles async fields and nested models by extracting their display_name.
+
+    Args:
+        model: The Model instance to serialize.
+
+    Returns:
+        A dictionary representation of the model.
+    """
+    data = {}
+    for field_name in model.__fields__:
+        value = getattr(model, field_name)
+        if inspect.isawaitable(value):
+            value = await value
+        if isinstance(value, Model):
+            display_name = value.display_name
+            if inspect.isawaitable(display_name):
+                display_name = await display_name
+            data[field_name] = display_name
+        else:
+            data[field_name] = value
+    return data
+
+
+def clean_data(val: Any) -> Any:
+    """
+    Cleans data for serialization/display.
+
+    Handles dates by formatting them, truncates strings and dicts to prevent overflow,
+    handles object IDs in dicts, and escapes markdown characters.
+
+    Args:
+        val: The value to clean.
+
+    Returns:
+        The cleaned value.
+    """
+    if val is None:
+        return ""
+    if isinstance(val, (datetime, date)):
+        if isinstance(val, datetime):
+            return val.strftime('%Y-%m-%dT%H:%M:%S')
+        return val.isoformat()
+    if isinstance(val, list):
+        return ", ".join(str(v) for v in val)
+    if isinstance(val, dict):
+        ref = val.get('ref')
+        if isinstance(ref, dict):
+            obj_id = ref.get('object_id', 'Object')
+            return (
+                obj_id[:MAX_OBJECT_ID_LENGTH] + '...'
+                if len(obj_id) > MAX_OBJECT_ID_LENGTH
+                else obj_id
+            )
+        s_val = str(val)
+        return (
+            s_val[:MAX_DICT_STR_LENGTH] + '...'
+            if len(s_val) > MAX_DICT_STR_LENGTH
+            else s_val
+        )
+
+    s = str(val)
+    s = s.replace('\n', ' ').replace('\r', '').replace('|', r'\|')
+    return s[: MAX_CELL_LENGTH - 3] + "..." if len(s) > MAX_CELL_LENGTH else s
+
+
+async def serialize_and_clean_record(
+    record: Model | dict[str, Any],
+) -> CleanedRecord:
+    """
+    Serializes a Model or dict and cleans its values for display/usage.
+
+    Handles both Model instances (by serializing them asynchronously) and plain dicts,
+    then applies cleaning to all values.
+
+    Args:
+        record: The Model instance or dictionary to serialize and clean.
+
+    Returns:
+        A dictionary with cleaned field values.
+    """
+    if isinstance(record, Model):
+        data = await async_serialize_model(record)
+    elif isinstance(record, dict):
+        data = record
+    else:
+        msg = f"Unsupported record type: {type(record)}"
+        raise ValueError(msg)
+
+    return {k: clean_data(v) for k, v in data.items()}
+
+
+def render_markdown_table(
+    records: list[CleanedRecord],
+    columns: list[Column] | None = None,
+    fields: list[str] | None = None,
+) -> str:
+    """
+    Renders a list of cleaned records as a Markdown table.
+
+    If columns are not provided, infers them from the first record's keys or uses the fields list.
+    Handles callable fields for custom value extraction.
+
+    Args:
+        records: List of cleaned records to render.
+        columns: Optional list of column definitions with field and title.
+        fields: Optional list of field names to include if columns not provided.
+
+    Returns:
+        A string containing the Markdown table.
+    """
+    if not records:
+        return "No records found."
+
+    if columns is None:
+        if fields:
+            columns = [
+                {"field": f, "title": f.replace('_', ' ').title()} for f in fields
+            ]
+        else:
+            headers = list(records[0].keys())
+            columns = [
+                {"field": h, "title": h.replace('_', ' ').title()} for h in headers
+            ]
+
+    if not columns:
+        return "No columns to display."
+
+    rows: list[dict[str, Any]] = []
+    for record in records:
+        row: dict[str, Any] = {}
+        for col in columns:
+            field = col["field"]
+            if callable(field):
+                value = field(record)
+            else:
+                value = record.get(field, '')
+            row[col["title"]] = value
+        rows.append(row)
+
+    headers = list(rows[0].keys())
+
+    col_widths: dict[str, int] = {h: len(h) for h in headers}
+
+    for row in rows:
+        for h in headers:
+            val = str(row.get(h, ''))
+            col_widths[h] = max(col_widths[h], len(val))
+
+    table_lines: list[str] = []
+    table_lines.append(
+        "| " + " | ".join(h.ljust(col_widths[h]) for h in headers) + " |"
+    )
+    table_lines.append(
+        "| " + " | ".join("-" * col_widths[h] for h in headers) + " |"
+    )
+    for row in rows:
+        table_lines.append(
+            "| " + " | ".join(str(row[h]).ljust(col_widths[h]) for h in headers) + " |"
+        )
+
+    return '\n'.join(table_lines)

{amsdal_ml-0.1.4.dist-info → amsdal_ml-0.2.0.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: amsdal_ml
-Version: 0.1.4
+Version: 0.2.0
 Summary: amsdal_ml plugin for AMSDAL Framework
 Requires-Python: >=3.11
 Requires-Dist: aiohttp==3.12.15
@@ -13,6 +13,7 @@ Requires-Dist: mcp>=0.1
 Requires-Dist: openai==1.100.2
 Requires-Dist: pydantic-settings==2.10.1
 Requires-Dist: pydantic==2.11.7
+Requires-Dist: pymupdf>=1.24.10
 Description-Content-Type: text/markdown
 
 # AMSDAL ML
@@ -130,6 +131,53 @@ async for chunk in agent.astream('What is semantic search?'):
     print(chunk, end='', flush=True)
 ```
 
+### 5. Functional Calling Agent with Python Tools
+
+```python
+from amsdal_ml.agents.functional_calling_agent import FunctionalCallingAgent
+from amsdal_ml.agents.python_tool import PythonTool
+from amsdal_ml.ml_models.openai_model import OpenAIModel
+
+llm = OpenAIModel()
+agent = FunctionalCallingAgent(model=llm, tools=[search_tool, render_tool])
+result = await agent.arun(user_query="Find products with price > 100", history=[])
+```
+
+### 6. Natural Language Query Retriever
+
+```python
+from amsdal_ml.ml_retrievers.query_retriever import NLQueryRetriever
+
+retriever = NLQueryRetriever(llm=llm, queryset=Product.objects.all())
+documents = await retriever.invoke("Show me red products", limit=10)
+```
+
+### 7. Document Ingestion Pipeline
+
+```python
+from amsdal_ml.ml_ingesting import ModelIngester
+from amsdal_ml.ml_ingesting.pipeline import DefaultIngestionPipeline
+from amsdal_ml.ml_ingesting.loaders.pdf_loader import PdfLoader
+from amsdal_ml.ml_ingesting.processors.text_cleaner import TextCleaner
+from amsdal_ml.ml_ingesting.splitters.token_splitter import TokenSplitter
+from amsdal_ml.ml_ingesting.embedders.openai_embedder import OpenAIEmbedder
+from amsdal_ml.ml_ingesting.stores.embedding_data import EmbeddingDataStore
+
+pipeline = DefaultIngestionPipeline(
+    loader=PdfLoader(),  # Uses pymupdf for PDF processing
+    cleaner=TextCleaner(),
+    splitter=TokenSplitter(max_tokens=800, overlap_tokens=80),
+    embedder=OpenAIEmbedder(),
+    store=EmbeddingDataStore(),
+)
+
+ingester = ModelIngester(
+    pipeline=pipeline,
+    base_tags=["document"],
+    base_metadata={"source": "pdf"},
+)
+```
+
 ## Architecture
 
 ### Core Components
@@ -139,6 +187,16 @@ async for chunk in agent.astream('What is semantic search?'):
 - **`MLRetriever`**: Semantic similarity search with tag-based filtering
 - **`Agent`**: Q&A and task-oriented agents with streaming and citations
 - **`EmbeddingModel`**: Database model storing 1536-dimensional vectors linked to source objects
+- **`PythonTool`**: Tool for executing Python functions within agents
+- **`FunctionalCallingAgent`**: Agent specialized in functional calling with configurable tools
+- **`NLQueryRetriever`**: Retriever for natural language queries on AMSDAL querysets
+- **`DefaultIngestionPipeline`**: Pipeline for document ingestion including loader, cleaner, splitter, embedder, and store
+- **`ModelIngester`**: High-level ingester for processing models with customizable pipelines and metadata
+- **`PdfLoader`**: Document loader using pymupdf for PDF processing
+- **`TextCleaner`**: Processor for cleaning and normalizing text
+- **`TokenSplitter`**: Splitter for dividing text into chunks based on token count
+- **`OpenAIEmbedder`**: Embedder for generating embeddings via OpenAI API
+- **`EmbeddingDataStore`**: Store for saving embedding data linked to source objects
 - **MCP Server/Client**: Expose retrievers as tools or consume external MCP services
 
 ### Configuration