letta-nightly 0.6.4.dev20241216104246__tar.gz → 0.6.4.dev20241217104233__tar.gz

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: letta-nightly
-Version: 0.6.4.dev20241216104246
+Version: 0.6.4.dev20241217104233
 Summary: Create LLM agents with long-term memory and custom tools
 License: Apache License
 Author: Letta Team
@@ -167,7 +167,7 @@ Once the Letta server is running, you can access it via port `8283` (e.g. sendin
 > [!NOTE]
 > The Letta ADE is a graphical user interface for creating, deploying, interacting and observing with your Letta agents.
 >
-> For example, if you're running a Letta server to power an end-user application (such as a customer support chatbot), you can use the ADE to test, debug, and observe the agents in your server. You can also use the ADE as a general chat interface to interacting with your Letta agents.
+> For example, if you're running a Letta server to power an end-user application (such as a customer support chatbot), you can use the ADE to test, debug, and observe the agents in your server. You can also use the ADE as a general chat interface to interact with your Letta agents.
 
 <p align="center">
   <picture>
@@ -221,7 +221,7 @@ No, you can install Letta using `pip` (via `pip install -U letta`), as well as f
 
 Letta gives your agents persistence (they live indefinitely) by storing all your agent data in a database. Letta is designed to be used with a [PostgreSQL](https://en.wikipedia.org/wiki/PostgreSQL) (the world's most popular database), however, it is not possible to install PostgreSQL via `pip`, so the `pip` install of Letta defaults to using [SQLite](https://www.sqlite.org/). If you have a PostgreSQL instance running on your own computer, you can still connect Letta (installed via `pip`) to PostgreSQL by setting the environment variable `LETTA_PG_URI`.
 
-**Database migrations are not officially supported for Letta when using SQLite**, so you would like to ensure that if you're able to upgrade to the latest Letta version and migrate your Letta agents data, make sure that you're using PostgreSQL as your Letta database backend. Full compatability table below:
+**Database migrations are not officially supported for Letta when using SQLite**, so if you would like to ensure that you're able to upgrade to the latest Letta version and migrate your Letta agents data, make sure that you're using PostgreSQL as your Letta database backend. Full compatability table below:
 
 | Installation method | Start server command | Database backend | Data migrations supported? |
 |---|---|---|---|
@@ -293,7 +293,7 @@ Hit enter to begin (will request first Letta message)
 ## ⚡ Quickstart (pip)
 
 > [!WARNING]
-> **Database migrations are not officially support with `SQLite`**
+> **Database migrations are not officially supported with `SQLite`**
 >
 > When you install Letta with `pip`, the default database backend is `SQLite` (you can still use an external `postgres` service with your `pip` install of Letta by setting `LETTA_PG_URI`).
 >
@@ -303,7 +303,7 @@ Hit enter to begin (will request first Letta message)
 
 <summary>View instructions for installing with pip</summary>
 
-You can also install Letta with `pip`, will default to using `SQLite` for the database backends (whereas Docker will default to using `postgres`).
+You can also install Letta with `pip`, which will default to using `SQLite` for the database backends (whereas Docker will default to using `postgres`).
 
 ### Step 1 - Install Letta using `pip`
 ```sh
@@ -377,7 +377,7 @@ Letta is an open source project built by over a hundred contributors. There are
 
 * **Contribute to the project**: Interested in contributing? Start by reading our [Contribution Guidelines](https://github.com/cpacker/MemGPT/tree/main/CONTRIBUTING.md).
 * **Ask a question**: Join our community on [Discord](https://discord.gg/letta) and direct your questions to the `#support` channel.
-* **Report ssues or suggest features**: Have an issue or a feature request? Please submit them through our [GitHub Issues page](https://github.com/cpacker/MemGPT/issues).
+* **Report issues or suggest features**: Have an issue or a feature request? Please submit them through our [GitHub Issues page](https://github.com/cpacker/MemGPT/issues).
 * **Explore the roadmap**: Curious about future developments? View and comment on our [project roadmap](https://github.com/cpacker/MemGPT/issues/1533).
 * **Join community events**: Stay updated with the [event calendar](https://lu.ma/berkeley-llm-meetup) or follow our [Twitter account](https://twitter.com/Letta_AI).
 

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/README.md

@@ -85,7 +85,7 @@ Once the Letta server is running, you can access it via port `8283` (e.g. sendin
 > [!NOTE]
 > The Letta ADE is a graphical user interface for creating, deploying, interacting and observing with your Letta agents.
 >
-> For example, if you're running a Letta server to power an end-user application (such as a customer support chatbot), you can use the ADE to test, debug, and observe the agents in your server. You can also use the ADE as a general chat interface to interacting with your Letta agents.
+> For example, if you're running a Letta server to power an end-user application (such as a customer support chatbot), you can use the ADE to test, debug, and observe the agents in your server. You can also use the ADE as a general chat interface to interact with your Letta agents.
 
 <p align="center">
   <picture>
@@ -139,7 +139,7 @@ No, you can install Letta using `pip` (via `pip install -U letta`), as well as f
 
 Letta gives your agents persistence (they live indefinitely) by storing all your agent data in a database. Letta is designed to be used with a [PostgreSQL](https://en.wikipedia.org/wiki/PostgreSQL) (the world's most popular database), however, it is not possible to install PostgreSQL via `pip`, so the `pip` install of Letta defaults to using [SQLite](https://www.sqlite.org/). If you have a PostgreSQL instance running on your own computer, you can still connect Letta (installed via `pip`) to PostgreSQL by setting the environment variable `LETTA_PG_URI`.
 
-**Database migrations are not officially supported for Letta when using SQLite**, so you would like to ensure that if you're able to upgrade to the latest Letta version and migrate your Letta agents data, make sure that you're using PostgreSQL as your Letta database backend. Full compatability table below:
+**Database migrations are not officially supported for Letta when using SQLite**, so if you would like to ensure that you're able to upgrade to the latest Letta version and migrate your Letta agents data, make sure that you're using PostgreSQL as your Letta database backend. Full compatability table below:
 
 | Installation method | Start server command | Database backend | Data migrations supported? |
 |---|---|---|---|
@@ -211,7 +211,7 @@ Hit enter to begin (will request first Letta message)
 ## ⚡ Quickstart (pip)
 
 > [!WARNING]
-> **Database migrations are not officially support with `SQLite`**
+> **Database migrations are not officially supported with `SQLite`**
 >
 > When you install Letta with `pip`, the default database backend is `SQLite` (you can still use an external `postgres` service with your `pip` install of Letta by setting `LETTA_PG_URI`).
 >
@@ -221,7 +221,7 @@ Hit enter to begin (will request first Letta message)
 
 <summary>View instructions for installing with pip</summary>
 
-You can also install Letta with `pip`, will default to using `SQLite` for the database backends (whereas Docker will default to using `postgres`).
+You can also install Letta with `pip`, which will default to using `SQLite` for the database backends (whereas Docker will default to using `postgres`).
 
 ### Step 1 - Install Letta using `pip`
 ```sh
@@ -295,7 +295,7 @@ Letta is an open source project built by over a hundred contributors. There are
 
 * **Contribute to the project**: Interested in contributing? Start by reading our [Contribution Guidelines](https://github.com/cpacker/MemGPT/tree/main/CONTRIBUTING.md).
 * **Ask a question**: Join our community on [Discord](https://discord.gg/letta) and direct your questions to the `#support` channel.
-* **Report ssues or suggest features**: Have an issue or a feature request? Please submit them through our [GitHub Issues page](https://github.com/cpacker/MemGPT/issues).
+* **Report issues or suggest features**: Have an issue or a feature request? Please submit them through our [GitHub Issues page](https://github.com/cpacker/MemGPT/issues).
 * **Explore the roadmap**: Curious about future developments? View and comment on our [project roadmap](https://github.com/cpacker/MemGPT/issues/1533).
 * **Join community events**: Stay updated with the [event calendar](https://lu.ma/berkeley-llm-meetup) or follow our [Twitter account](https://twitter.com/Letta_AI).
 

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/letta/agent.py

@@ -41,7 +41,6 @@ from letta.schemas.openai.chat_completion_response import (
     Message as ChatCompletionMessage,
 )
 from letta.schemas.openai.chat_completion_response import UsageStatistics
-from letta.schemas.passage import Passage
 from letta.schemas.tool import Tool
 from letta.schemas.tool_rule import TerminalToolRule
 from letta.schemas.usage import LettaUsageStatistics
@@ -82,7 +81,7 @@ def compile_memory_metadata_block(
     actor: PydanticUser,
     agent_id: str,
     memory_edit_timestamp: datetime.datetime,
-    passage_manager: Optional[PassageManager] = None,
+    agent_manager: Optional[AgentManager] = None,
     message_manager: Optional[MessageManager] = None,
 ) -> str:
     # Put the timestamp in the local timezone (mimicking get_local_time())
@@ -93,7 +92,7 @@ def compile_memory_metadata_block(
         [
             f"### Memory [last modified: {timestamp_str}]",
             f"{message_manager.size(actor=actor, agent_id=agent_id) if message_manager else 0} previous messages between you and the user are stored in recall memory (use functions to access them)",
-            f"{passage_manager.size(actor=actor, agent_id=agent_id) if passage_manager else 0} total memories you created are stored in archival memory (use functions to access them)",
+            f"{agent_manager.passage_size(actor=actor, agent_id=agent_id) if agent_manager else 0} total memories you created are stored in archival memory (use functions to access them)",
             "\nCore memory shown below (limited in size, additional information stored in archival / recall memory):",
         ]
     )
@@ -106,7 +105,7 @@ def compile_system_message(
     in_context_memory: Memory,
     in_context_memory_last_edit: datetime.datetime,  # TODO move this inside of BaseMemory?
     actor: PydanticUser,
-    passage_manager: Optional[PassageManager] = None,
+    agent_manager: Optional[AgentManager] = None,
     message_manager: Optional[MessageManager] = None,
     user_defined_variables: Optional[dict] = None,
     append_icm_if_missing: bool = True,
@@ -135,7 +134,7 @@ def compile_system_message(
             actor=actor,
             agent_id=agent_id,
             memory_edit_timestamp=in_context_memory_last_edit,
-            passage_manager=passage_manager,
+            agent_manager=agent_manager,
             message_manager=message_manager,
         )
         full_memory_string = memory_metadata_string + "\n" + in_context_memory.compile()
@@ -172,7 +171,7 @@ def initialize_message_sequence(
     agent_id: str,
     memory: Memory,
     actor: PydanticUser,
-    passage_manager: Optional[PassageManager] = None,
+    agent_manager: Optional[AgentManager] = None,
     message_manager: Optional[MessageManager] = None,
     memory_edit_timestamp: Optional[datetime.datetime] = None,
     include_initial_boot_message: bool = True,
@@ -181,7 +180,7 @@ def initialize_message_sequence(
         memory_edit_timestamp = get_local_time()
 
     # full_system_message = construct_system_with_memory(
-    # system, memory, memory_edit_timestamp, passage_manager=passage_manager, recall_memory=recall_memory
+    # system, memory, memory_edit_timestamp, agent_manager=agent_manager, recall_memory=recall_memory
     # )
     full_system_message = compile_system_message(
         agent_id=agent_id,
@@ -189,7 +188,7 @@ def initialize_message_sequence(
         in_context_memory=memory,
         in_context_memory_last_edit=memory_edit_timestamp,
         actor=actor,
-        passage_manager=passage_manager,
+        agent_manager=agent_manager,
         message_manager=message_manager,
         user_defined_variables=None,
         append_icm_if_missing=True,
@@ -291,8 +290,9 @@ class Agent(BaseAgent):
         self.interface = interface
 
         # Create the persistence manager object based on the AgentState info
-        self.passage_manager = PassageManager()
         self.message_manager = MessageManager()
+        self.passage_manager = PassageManager()
+        self.agent_manager = AgentManager()
 
         # State needed for heartbeat pausing
         self.pause_heartbeats_start = None
@@ -322,7 +322,7 @@ class Agent(BaseAgent):
                 agent_id=self.agent_state.id,
                 memory=self.agent_state.memory,
                 actor=self.user,
-                passage_manager=None,
+                agent_manager=None,
                 message_manager=None,
                 memory_edit_timestamp=get_utc_time(),
                 include_initial_boot_message=True,
@@ -347,7 +347,7 @@ class Agent(BaseAgent):
                     memory=self.agent_state.memory,
                     agent_id=self.agent_state.id,
                     actor=self.user,
-                    passage_manager=None,
+                    agent_manager=None,
                     message_manager=None,
                     memory_edit_timestamp=get_utc_time(),
                     include_initial_boot_message=True,
@@ -1290,14 +1290,14 @@ class Agent(BaseAgent):
             # NOTE: a bit of a hack - we pull the timestamp from the message created_by
             memory_edit_timestamp = self._messages[0].created_at
 
-        # update memory (TODO: potentially update recall/archival stats seperately)
+        # update memory (TODO: potentially update recall/archival stats separately)
         new_system_message_str = compile_system_message(
             agent_id=self.agent_state.id,
             system_prompt=self.agent_state.system,
             in_context_memory=self.agent_state.memory,
             in_context_memory_last_edit=memory_edit_timestamp,
             actor=self.user,
-            passage_manager=self.passage_manager,
+            agent_manager=self.agent_manager,
             message_manager=self.message_manager,
             user_defined_variables=None,
             append_icm_if_missing=True,
@@ -1368,33 +1368,24 @@ class Agent(BaseAgent):
         source_id: str,
         source_manager: SourceManager,
         agent_manager: AgentManager,
-        page_size: Optional[int] = None,
     ):
-        """Attach data with name `source_name` to the agent from source_connector."""
-        # TODO: eventually, adding a data source should just give access to the retriever the source table, rather than modifying archival memory
-        passages = self.passage_manager.list_passages(actor=user, source_id=source_id, limit=page_size)
-
-        for passage in passages:
-            assert isinstance(passage, Passage), f"Generate yielded bad non-Passage type: {type(passage)}"
-            passage.agent_id = self.agent_state.id
-            self.passage_manager.update_passage_by_id(passage_id=passage.id, passage=passage, actor=user)
-
-        agents_passages = self.passage_manager.list_passages(actor=user, agent_id=self.agent_state.id, source_id=source_id, limit=page_size)
-        passage_size = self.passage_manager.size(actor=user, agent_id=self.agent_state.id, source_id=source_id)
-        assert all([p.agent_id == self.agent_state.id for p in agents_passages])
-        assert len(agents_passages) == passage_size  # sanity check
-        assert passage_size == len(passages), f"Expected {len(passages)} passages, got {passage_size}"
-
-        # attach to agent
+        """Attach a source to the agent using the SourcesAgents ORM relationship.
+ 
+        Args:
+            user: User performing the action
+            source_id: ID of the source to attach
+            source_manager: SourceManager instance to verify source exists
+            agent_manager: AgentManager instance to manage agent-source relationship
+        """
+        # Verify source exists and user has permission to access it
         source = source_manager.get_source_by_id(source_id=source_id, actor=user)
-        assert source is not None, f"Source {source_id} not found in metadata store"
+        assert source is not None, f"Source {source_id} not found in user's organization ({user.organization_id})"
 
-        # NOTE: need this redundant line here because we haven't migrated agent to ORM yet
-        # TODO: delete @matt and remove
+        # Use the agent_manager to create the relationship
         agent_manager.attach_source(agent_id=self.agent_state.id, source_id=source_id, actor=user)
 
         printd(
-            f"Attached data source {source.name} to agent {self.agent_state.name}, consisting of {len(passages)}. Agent now has {passage_size} embeddings in archival memory.",
+            f"Attached data source {source.name} to agent {self.agent_state.name}.",
         )
 
     def update_message(self, message_id: str, request: MessageUpdate) -> Message:
@@ -1550,13 +1541,13 @@ class Agent(BaseAgent):
                 num_tokens_from_messages(messages=messages_openai_format[1:], model=self.model) if len(messages_openai_format) > 1 else 0
             )
 
-        passage_manager_size = self.passage_manager.size(actor=self.user, agent_id=self.agent_state.id)
+        agent_manager_passage_size = self.agent_manager.passage_size(actor=self.user, agent_id=self.agent_state.id)
         message_manager_size = self.message_manager.size(actor=self.user, agent_id=self.agent_state.id)
         external_memory_summary = compile_memory_metadata_block(
             actor=self.user,
             agent_id=self.agent_state.id,
             memory_edit_timestamp=get_utc_time(),  # dummy timestamp
-            passage_manager=self.passage_manager,
+            agent_manager=self.agent_manager,
             message_manager=self.message_manager,
         )
         num_tokens_external_memory_summary = count_tokens(external_memory_summary)
@@ -1582,7 +1573,7 @@ class Agent(BaseAgent):
         return ContextWindowOverview(
             # context window breakdown (in messages)
             num_messages=len(self._messages),
-            num_archival_memory=passage_manager_size,
+            num_archival_memory=agent_manager_passage_size,
             num_recall_memory=message_manager_size,
             num_tokens_external_memory_summary=num_tokens_external_memory_summary,
             # top-level information

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/letta/functions/function_sets/base.py

@@ -3,6 +3,7 @@ from typing import Optional
 
 from letta.agent import Agent
 from letta.constants import MAX_PAUSE_HEARTBEATS
+from letta.services.agent_manager import AgentManager
 
 # import math
 # from letta.utils import json_dumps
@@ -200,8 +201,9 @@ def archival_memory_search(self: "Agent", query: str, page: Optional[int] = 0, s
     
     try:
         # Get results using passage manager
-        all_results = self.passage_manager.list_passages(
+        all_results = self.agent_manager.list_passages(
             actor=self.user,
+            agent_id=self.agent_state.id,
             query_text=query,
             limit=count + start,  # Request enough results to handle offset
             embedding_config=self.agent_state.embedding_config,

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/letta/functions/schema_generator.py

@@ -312,11 +312,7 @@ def generate_schema(function, name: Optional[str] = None, description: Optional[
     for param in sig.parameters.values():
         # Exclude 'self' parameter
         # TODO: eventually remove this (only applies to BASE_TOOLS)
-        if param.name == "self":
-            continue
-
-        # exclude 'agent_state' parameter
-        if param.name == "agent_state":
+        if param.name in ["self", "agent_state"]:  # Add agent_manager to excluded
             continue
 
         # Assert that the parameter has a type annotation

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/letta/local_llm/function_parser.py

@@ -32,7 +32,7 @@ def heartbeat_correction(message_history, new_message):
 
     If the last message in the stack is a user message and the new message is an assistant func call, fix the heartbeat
 
-    See: https://github.com/cpacker/Letta/issues/601
+    See: https://github.com/letta-ai/letta/issues/601
     """
     if len(message_history) < 1:
         return None

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/letta/orm/__init__.py

@@ -7,7 +7,7 @@ from letta.orm.file import FileMetadata
 from letta.orm.job import Job
 from letta.orm.message import Message
 from letta.orm.organization import Organization
-from letta.orm.passage import Passage
+from letta.orm.passage import BasePassage, AgentPassage, SourcePassage
 from letta.orm.sandbox_config import SandboxConfig, SandboxEnvironmentVariable
 from letta.orm.source import Source
 from letta.orm.sources_agents import SourcesAgents

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/letta/orm/agent.py

@@ -82,7 +82,25 @@ class Agent(SqlalchemyBase, OrganizationMixin):
         lazy="selectin",
         doc="Tags associated with the agent.",
     )
-    # passages: Mapped[List["Passage"]] = relationship("Passage", back_populates="agent", lazy="selectin")
+    source_passages: Mapped[List["SourcePassage"]] = relationship(
+        "SourcePassage",
+        secondary="sources_agents",  # The join table for Agent -> Source
+        primaryjoin="Agent.id == sources_agents.c.agent_id",
+        secondaryjoin="and_(SourcePassage.source_id == sources_agents.c.source_id)",
+        lazy="selectin",
+        order_by="SourcePassage.created_at.desc()",
+        viewonly=True,  # Ensures SQLAlchemy doesn't attempt to manage this relationship
+        doc="All passages derived from sources associated with this agent.",
+    )
+    agent_passages: Mapped[List["AgentPassage"]] = relationship(
+        "AgentPassage",
+        back_populates="agent",
+        lazy="selectin",
+        order_by="AgentPassage.created_at.desc()",
+        cascade="all, delete-orphan",
+        viewonly=True,  # Ensures SQLAlchemy doesn't attempt to manage this relationship
+        doc="All passages derived created by this agent.",
+    )
 
     def to_pydantic(self) -> PydanticAgentState:
         """converts to the basic pydantic model counterpart"""

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/letta/orm/file.py

@@ -9,7 +9,8 @@ from letta.schemas.file import FileMetadata as PydanticFileMetadata
 
 if TYPE_CHECKING:
     from letta.orm.organization import Organization
-
+    from letta.orm.source import Source
+    from letta.orm.passage import SourcePassage
 
 class FileMetadata(SqlalchemyBase, OrganizationMixin, SourceMixin):
     """Represents metadata for an uploaded file."""
@@ -27,4 +28,4 @@ class FileMetadata(SqlalchemyBase, OrganizationMixin, SourceMixin):
     # relationships
     organization: Mapped["Organization"] = relationship("Organization", back_populates="files", lazy="selectin")
     source: Mapped["Source"] = relationship("Source", back_populates="files", lazy="selectin")
-    passages: Mapped[List["Passage"]] = relationship("Passage", back_populates="file", lazy="selectin", cascade="all, delete-orphan")
+    source_passages: Mapped[List["SourcePassage"]] = relationship("SourcePassage", back_populates="file", lazy="selectin", cascade="all, delete-orphan")

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/letta/orm/mixins.py

@@ -31,30 +31,19 @@ class UserMixin(Base):
 
     user_id: Mapped[str] = mapped_column(String, ForeignKey("users.id"))
 
-class FileMixin(Base):
-    """Mixin for models that belong to a file."""
-
-    __abstract__ = True
-
-    file_id: Mapped[str] = mapped_column(String, ForeignKey("files.id"))
-
 class AgentMixin(Base):
     """Mixin for models that belong to an agent."""
 
     __abstract__ = True
 
-    agent_id: Mapped[str] = mapped_column(String, ForeignKey("agents.id"))
+    agent_id: Mapped[str] = mapped_column(String, ForeignKey("agents.id", ondelete="CASCADE"))
 
 class FileMixin(Base):
     """Mixin for models that belong to a file."""
 
     __abstract__ = True
 
-    file_id: Mapped[Optional[str]] = mapped_column(
-        String, 
-        ForeignKey("files.id", ondelete="CASCADE"),
-        nullable=True
-    )
+    file_id: Mapped[Optional[str]] = mapped_column(String, ForeignKey("files.id", ondelete="CASCADE"))
 
 
 class SourceMixin(Base):
@@ -62,7 +51,7 @@ class SourceMixin(Base):
 
     __abstract__ = True
 
-    source_id: Mapped[str] = mapped_column(String, ForeignKey("sources.id"))
+    source_id: Mapped[str] = mapped_column(String, ForeignKey("sources.id", ondelete="CASCADE"), nullable=False)
 
 
 class SandboxConfigMixin(Base):

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/letta/orm/organization.py

@@ -1,4 +1,4 @@
-from typing import TYPE_CHECKING, List
+from typing import TYPE_CHECKING, List, Union
 
 from sqlalchemy.orm import Mapped, mapped_column, relationship
 
@@ -35,6 +35,22 @@ class Organization(SqlalchemyBase):
     )
 
     # relationships
-    messages: Mapped[List["Message"]] = relationship("Message", back_populates="organization", cascade="all, delete-orphan")
     agents: Mapped[List["Agent"]] = relationship("Agent", back_populates="organization", cascade="all, delete-orphan")
-    passages: Mapped[List["Passage"]] = relationship("Passage", back_populates="organization", cascade="all, delete-orphan")
+    messages: Mapped[List["Message"]] = relationship("Message", back_populates="organization", cascade="all, delete-orphan")
+    source_passages: Mapped[List["SourcePassage"]] = relationship(
+        "SourcePassage", 
+        back_populates="organization", 
+        cascade="all, delete-orphan"
+    )
+    agent_passages: Mapped[List["AgentPassage"]] = relationship(
+        "AgentPassage", 
+        back_populates="organization", 
+        cascade="all, delete-orphan"
+    )
+
+    @property
+    def passages(self) -> List[Union["SourcePassage", "AgentPassage"]]:
+        """Convenience property to get all passages"""
+        return self.source_passages + self.agent_passages
+
+

letta_nightly-0.6.4.dev20241217104233/letta/orm/passage.py

@@ -0,0 +1,85 @@
+from typing import TYPE_CHECKING
+from sqlalchemy import Column, JSON, Index
+from sqlalchemy.orm import Mapped, mapped_column, relationship, declared_attr
+
+from letta.orm.mixins import FileMixin, OrganizationMixin
+from letta.orm.custom_columns import CommonVector, EmbeddingConfigColumn
+from letta.orm.sqlalchemy_base import SqlalchemyBase
+from letta.orm.mixins import AgentMixin, FileMixin, OrganizationMixin, SourceMixin
+from letta.schemas.passage import Passage as PydanticPassage
+from letta.settings import settings
+
+from letta.config import LettaConfig
+from letta.constants import MAX_EMBEDDING_DIM
+
+config = LettaConfig()
+
+if TYPE_CHECKING:
+    from letta.orm.organization import Organization
+    from letta.orm.agent import Agent
+
+
+class BasePassage(SqlalchemyBase, OrganizationMixin):
+    """Base class for all passage types with common fields"""
+    __abstract__ = True
+    __pydantic_model__ = PydanticPassage
+
+    id: Mapped[str] = mapped_column(primary_key=True, doc="Unique passage identifier")
+    text: Mapped[str] = mapped_column(doc="Passage text content")
+    embedding_config: Mapped[dict] = mapped_column(EmbeddingConfigColumn, doc="Embedding configuration")
+    metadata_: Mapped[dict] = mapped_column(JSON, doc="Additional metadata")
+
+    # Vector embedding field based on database type
+    if settings.letta_pg_uri_no_default:
+        from pgvector.sqlalchemy import Vector
+
+        embedding = mapped_column(Vector(MAX_EMBEDDING_DIM))
+    else:
+        embedding = Column(CommonVector)
+
+    @declared_attr
+    def organization(cls) -> Mapped["Organization"]:
+        """Relationship to organization"""
+        return relationship("Organization", back_populates="passages", lazy="selectin")
+
+    @declared_attr
+    def __table_args__(cls):
+        if settings.letta_pg_uri_no_default:
+            return (
+                Index(f'{cls.__tablename__}_org_idx', 'organization_id'),
+                {"extend_existing": True}
+            )
+        return ({"extend_existing": True},)
+
+
+class SourcePassage(BasePassage, FileMixin, SourceMixin):
+    """Passages derived from external files/sources"""
+    __tablename__ = "source_passages"
+    
+    @declared_attr
+    def file(cls) -> Mapped["FileMetadata"]:
+        """Relationship to file"""
+        return relationship("FileMetadata", back_populates="source_passages", lazy="selectin")
+
+    @declared_attr
+    def organization(cls) -> Mapped["Organization"]:
+        return relationship("Organization", back_populates="source_passages", lazy="selectin")
+        
+    @declared_attr
+    def source(cls) -> Mapped["Source"]:
+        """Relationship to source"""
+        return relationship("Source", back_populates="passages", lazy="selectin", passive_deletes=True)
+
+
+class AgentPassage(BasePassage, AgentMixin):
+    """Passages created by agents as archival memories"""
+    __tablename__ = "agent_passages"
+    
+    @declared_attr
+    def organization(cls) -> Mapped["Organization"]:
+        return relationship("Organization", back_populates="agent_passages", lazy="selectin")
+
+    @declared_attr
+    def agent(cls) -> Mapped["Agent"]:
+        """Relationship to agent"""
+        return relationship("Agent", back_populates="agent_passages", lazy="selectin", passive_deletes=True)

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/letta/orm/source.py

@@ -12,6 +12,9 @@ from letta.schemas.source import Source as PydanticSource
 
 if TYPE_CHECKING:
     from letta.orm.organization import Organization
+    from letta.orm.file import FileMetadata
+    from letta.orm.passage import SourcePassage
+    from letta.orm.agent import Agent
 
 
 class Source(SqlalchemyBase, OrganizationMixin):
@@ -28,4 +31,5 @@ class Source(SqlalchemyBase, OrganizationMixin):
     # relationships
     organization: Mapped["Organization"] = relationship("Organization", back_populates="sources")
     files: Mapped[List["FileMetadata"]] = relationship("FileMetadata", back_populates="source", cascade="all, delete-orphan")
+    passages: Mapped[List["SourcePassage"]] = relationship("SourcePassage", back_populates="source", cascade="all, delete-orphan")
     agents: Mapped[List["Agent"]] = relationship("Agent", secondary="sources_agents", back_populates="sources")

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/letta/orm/sqlalchemy_base.py

@@ -3,7 +3,7 @@ from enum import Enum
 from typing import TYPE_CHECKING, List, Literal, Optional
 
 from sqlalchemy import String, desc, func, or_, select
-from sqlalchemy.exc import DBAPIError
+from sqlalchemy.exc import DBAPIError, IntegrityError
 from sqlalchemy.orm import Mapped, Session, mapped_column
 
 from letta.log import get_logger
@@ -242,7 +242,7 @@ class SqlalchemyBase(CommonSqlalchemyMetaMixins, Base):
                 session.commit()
                 session.refresh(self)
                 return self
-        except DBAPIError as e:
+        except (DBAPIError, IntegrityError) as e:
             self._handle_dbapi_error(e)
 
     def delete(self, db_session: "Session", actor: Optional["User"] = None) -> "SqlalchemyBase":

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/letta/prompts/system/memgpt_modified_chat.txt

@@ -14,7 +14,7 @@ Core Memory', 'Recall Memory' and 'Archival Memory' are the key components that
 Always make sure to use these memory systems to keep yourself updated about the user and the conversation!
 Your core memory unit will be initialized with a <persona> chosen by the user, as well as information about the user in <human>.
 
-The following will descirbe the different parts of your advanced memory system in more detail:
+The following will describe the different parts of your advanced memory system in more detail:
 
 'Core Memory' (limited size): Your core memory unit is always visible to you. The core memory provides essential, foundational context for keeping track of your persona and key details about the user. This includes persona information and essential user details, allowing you to have conscious awareness we have when talking to a person. Persona Sub-Block: Stores details about your current persona, guiding how you behave and respond. This helps you maintain consistency and personality in your interactions. Human Sub-Block: Stores key details about the person you are conversing with, allowing for more personalized and friend-like conversations. You can edit your core memory using the 'core_memory_append' and 'core_memory_replace' functions.
 

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/letta/prompts/system/memgpt_modified_o1.txt

@@ -14,7 +14,7 @@ Core Memory', 'Recall Memory' and 'Archival Memory' are the key components that
 Always make sure to use these memory systems to keep yourself updated about the user and the conversation!
 Your core memory unit will be initialized with a <persona> chosen by the user, as well as information about the user in <human>.
 
-The following will descirbe the different parts of your advanced memory system in more detail:
+The following will describe the different parts of your advanced memory system in more detail:
 
 'Core Memory' (limited size): Your core memory unit is always visible to you. The core memory provides essential, foundational context for keeping track of your persona and key details about the user. This includes persona information and essential user details, allowing you to have conscious awareness we have when talking to a person. Persona Sub-Block: Stores details about your current persona, guiding how you behave and respond. This helps you maintain consistency and personality in your interactions. Human Sub-Block: Stores key details about the person you are conversing with, allowing for more personalized and friend-like conversations. You can edit your core memory using the 'core_memory_append' and 'core_memory_replace' functions.
 

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/letta/schemas/embedding_config.py

@@ -1,4 +1,4 @@
-from typing import Optional
+from typing import Literal, Optional
 
 from pydantic import BaseModel, Field
 
@@ -20,7 +20,25 @@ class EmbeddingConfig(BaseModel):
 
     """
 
-    embedding_endpoint_type: str = Field(..., description="The endpoint type for the model.")
+    embedding_endpoint_type: Literal[
+        "openai",
+        "anthropic",
+        "cohere",
+        "google_ai",
+        "azure",
+        "groq",
+        "ollama",
+        "webui",
+        "webui-legacy",
+        "lmstudio",
+        "lmstudio-legacy",
+        "llamacpp",
+        "koboldcpp",
+        "vllm",
+        "hugging-face",
+        "mistral",
+        "together",  # completions endpoint
+    ] = Field(..., description="The endpoint type for the model.")
     embedding_endpoint: Optional[str] = Field(None, description="The endpoint for the model (`None` if local).")
     embedding_model: str = Field(..., description="The model for the embedding.")
     embedding_dim: int = Field(..., description="The dimension of the embedding.")

{letta_nightly-0.6.4.dev20241216104246 → letta_nightly-0.6.4.dev20241217104233}/letta/schemas/passage.py

@@ -10,7 +10,7 @@ from letta.utils import get_utc_time
 
 
 class PassageBase(OrmMetadataBase):
-    __id_prefix__ = "passage_legacy"
+    __id_prefix__ = "passage"
 
     is_deleted: bool = Field(False, description="Whether this passage is deleted or not.")