naas-abi-core 1.0.0__py3-none-any.whl

naas_abi_core/modules/templatablesparqlquery/README.md

@@ -0,0 +1,196 @@
+# Templatable SPARQL Query Module
+
+## Overview
+
+### Description
+
+The Templatable SPARQL Query Module provides a dynamic system for creating reusable SPARQL queries through ontology-based definitions. This module enables developers to define SPARQL query templates in RDF/TTL format with arguments and validation patterns, which are then automatically converted into executable workflows and tools at runtime.
+
+This module enables:
+- Dynamic SPARQL query generation from ontology definitions
+- Automatic argument validation using regex patterns and formats
+- Runtime tool creation for AI agents without writing Python boilerplate
+- Template-based query execution using Jinja2 syntax
+- Integration with knowledge graphs and triple stores
+
+### Requirements
+
+Triple Store Setup:
+1. Ensure you have a running triple store service configured in your ABI instance
+2. The module uses `services.triple_store_service` to query ontology definitions
+
+### TL;DR
+
+To get started with the Templatable SPARQL Query module:
+
+1. Define your SPARQL queries in TTL format using the provided ontology
+2. Load the TTL file into your triple store
+3. The module automatically creates tools from your query definitions
+
+Access tools using:
+```python
+from src.core.modules.templatablesparqlquery import get_tools
+tools = get_tools()
+```
+
+### Structure
+
+```
+src/core/templatablesparqlquery/
+
+├── ontologies/                         
+│   └── TemplatableSparqlQueryOntology.ttl
+├── workflows/                     
+│   ├── GenericWorkflow.py          
+│   └── TemplatableSparqlQuery.py           
+└── README.md                       
+```
+
+## Core Components
+
+The module provides a declarative approach to SPARQL query definition and execution through ontology-based configuration.
+
+### Workflows
+
+#### Templatable SPARQL Query Workflow
+Dynamically loads query definitions from the triple store and creates executable workflows with proper argument validation and Jinja2 template rendering.
+
+**Capabilities:**
+- Loads query definitions from triple store using SPARQL
+- Creates Pydantic models for argument validation
+- Renders SPARQL templates with user parameters
+- Executes queries against the triple store
+- Returns formatted results
+
+**Use Cases:**
+- Dynamic query generation without hardcoded Python workflows
+- Reusable query templates with parameter validation
+- AI agent tool creation from ontology definitions
+
+#### Generic Workflow
+A generic wrapper class that handles the execution of templated SPARQL queries with type-safe argument validation.
+
+**Configuration:**
+
+```python
+from src.core.modules.templatablesparqlquery.workflows.GenericWorkflow import GenericWorkflow
+from pydantic import BaseModel, Field
+
+# Define argument model
+class MyQueryArguments(BaseModel):
+    department_id: str = Field(..., description="Department identifier", pattern="^[A-Z0-9]{3,}$")
+    employee_name: str = Field(..., description="Employee name filter")
+
+# Create workflow
+workflow = GenericWorkflow[MyQueryArguments](
+    name="find_employees",
+    description="Find employees in a department",
+    sparql_template=sparql_query_template,
+    arguments_model=MyQueryArguments
+)
+```
+
+#### Run
+Execute workflows by calling the module's tool loading functions:
+```python
+from src.core.modules.templatablesparqlquery import get_tools
+tools = get_tools()
+# Tools are automatically created from ontology definitions
+```
+
+#### Testing
+Currently no dedicated test files are present in the module structure.
+
+### Ontologies
+
+#### Templatable SPARQL Query Ontology
+
+The core ontology defining the structure for templatable SPARQL queries and their arguments:
+
+**Key Classes:**
+- `abi:TemplatableSparqlQuery`: Represents a SPARQL query template with intent information
+- `abi:QueryArgument`: Represents an argument for query templating
+
+**Key Properties:**
+- `abi:intentDescription`: Natural language description of query purpose
+- `abi:sparqlTemplate`: The SPARQL query template with Jinja2 variables
+- `abi:hasArgument`: Links queries to their arguments
+- `abi:argumentName`: Argument name used in templates
+- `abi:validationPattern`: Regex pattern for argument validation
+- `abi:validationFormat`: Expected format description
+
+#### Example Query Definition
+
+```turtle
+@prefix abi: <http://ontology.naas.ai/abi/> .
+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+
+abi:findEmployeesQuery a abi:TemplatableSparqlQuery ;
+    rdfs:label "Find Employees"@en ;
+    abi:intentDescription "Find all employees working in a specific department" ;
+    abi:sparqlTemplate """
+        SELECT ?employee ?name
+        WHERE {
+            ?employee :worksIn :{{ department_id }} ;
+                    :hasName ?name .
+            {% if employee_name %}
+            FILTER(CONTAINS(LCASE(?name), LCASE("{{ employee_name }}")))
+            {% endif %}
+        }
+    """ ;
+    abi:hasArgument abi:departmentArg .
+
+abi:departmentArg a abi:QueryArgument ;
+    abi:argumentName "department_id" ;
+    abi:argumentDescription "The unique identifier of the department" ;
+    abi:validationPattern "^[A-Z0-9]{3,}$" ;
+    abi:validationFormat "department_id" .
+```
+
+## How to Add New Queries
+
+### Step 1: Define Query in TTL Format
+Create or update a TTL file with your query definition using the templatable SPARQL query ontology.
+
+### Step 2: Define Arguments
+Specify all required arguments with proper validation patterns and descriptions.
+
+### Step 3: Load into Triple Store
+Ensure your TTL file is loaded into the triple store during application initialization.
+
+### Step 4: Access Generated Tools
+The module automatically creates tools from your definitions:
+
+```python
+from src.core.modules.templatablesparqlquery import get_tools
+tools = get_tools()
+```
+
+## Dependencies
+
+### Python Libraries
+- `pydantic`: Data validation and serialization for argument models
+- `rdflib`: RDF graph processing and SPARQL query execution
+- `jinja2`: Template rendering for SPARQL queries
+- `langchain_core`: Tool integration for AI agents
+- `asyncio`: Asynchronous processing support
+
+### Internal Modules
+- `src.utils.SPARQL`: SPARQL result processing utilities
+- `src.services`: Triple store service integration
+
+### External Services
+- **Triple Store**: Required for storing and querying ontology definitions
+- **Knowledge Graph**: Source data for SPARQL query execution
+
+## Technical Implementation
+
+The module uses a sophisticated runtime generation process:
+
+1. **Query Discovery**: Scans the triple store for `TemplatableSparqlQuery` instances
+2. **Argument Resolution**: Retrieves associated `QueryArgument` definitions
+3. **Model Generation**: Creates Pydantic models with validation patterns
+4. **Tool Creation**: Generates LangChain tools for AI agent integration
+5. **Template Execution**: Uses Jinja2 to render SPARQL templates with parameters
+
+This approach eliminates the need for writing repetitive Python code for similar query patterns, allowing developers to focus on query logic and ontology design.

naas_abi_core/modules/templatablesparqlquery/__init__.py

@@ -0,0 +1,39 @@
+from naas_abi_core.module.Module import (
+    BaseModule,
+    ModuleConfiguration,
+    ModuleDependencies,
+)
+from naas_abi_core.modules.templatablesparqlquery.workflows.TemplatableSparqlQueryLoader import (
+    TemplatableSparqlQueryLoader,
+)
+from naas_abi_core.services.triple_store.TripleStoreService import TripleStoreService
+
+
+class ABIModule(BaseModule):
+    __workflows: list = []
+    __tools: list = []
+
+    dependencies: ModuleDependencies = ModuleDependencies(
+        modules=[], services=[TripleStoreService]
+    )
+
+    class Configuration(ModuleConfiguration):
+        pass
+
+    def on_initialized(self):
+        self.__templatable_sparql_query_loader = TemplatableSparqlQueryLoader(
+            self.engine.services.triple_store
+        )
+        self.__workflows = self.__templatable_sparql_query_loader.load_workflows()
+        self.__tools = [
+            tool for workflow in self.__workflows for tool in workflow.as_tools()
+        ]
+
+    def get_workflows(self):
+        return self.__workflows
+
+    def get_tools(self, tool_names: list[str] = []):
+        if len(tool_names) == 0:
+            return self.__tools
+        else:
+            return [tool for tool in self.__tools if tool.name in tool_names]

naas_abi_core/modules/templatablesparqlquery/ontologies/TemplatableSparqlQueryOntology.ttl

@@ -0,0 +1,116 @@
+@prefix owl: <http://www.w3.org/2002/07/owl#> .
+@prefix dc11: <http://purl.org/dc/elements/1.1/> .
+@prefix dc: <http://purl.org/dc/terms/> .
+@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
+@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
+@prefix bfo: <http://purl.obolibrary.org/obo/> .
+@prefix cco: <https://www.commoncoreontologies.org/> .
+@prefix abi: <http://ontology.naas.ai/abi/> .
+
+<http://ontology.naas.ai/abi/TemplatableSparqlQueryOntology> a owl:Ontology ;
+    owl:versionIRI <https://github.com/jupyter-naas/abi/tree/cli/src/core/modules/abi/ontologies/domain-level/TemplatableSparqlQueryOntology.ttl> ;
+    dc11:contributor "Jeremy Ravenel" , "Maxime Jublou" , "Florent Ravenel" ;
+    dc:description "Templatable Sparql Query Ontology."@en ;
+    dc:license "" ;
+    dc:title "Templatable Sparql Query Ontology" .
+
+#################################################################
+#    Classes
+#################################################################
+
+abi:TemplatableSparqlQuery a owl:Class ;
+    rdfs:subClassOf bfo:BFO_0000031 ;
+    rdfs:label "Templatable SPARQL Query"@en ;
+    rdfs:comment "A class representing a SPARQL query that can be templated with variables and includes intent information."@en .
+
+abi:QueryArgument a owl:Class ;
+    rdfs:subClassOf bfo:BFO_0000031 ;
+    rdfs:label "Query Argument"@en ;
+    rdfs:comment "A class representing an argument that can be used to template a SPARQL query."@en .
+
+#################################################################
+#    Object Properties
+#################################################################
+
+abi:hasArgument a owl:ObjectProperty ;
+    rdfs:label "has argument"@en ;
+    rdfs:domain abi:TemplatableSparqlQuery ;
+    rdfs:range abi:QueryArgument ;
+    rdfs:comment "Links a templatable SPARQL query to its arguments."@en .
+
+#################################################################
+#    Data Properties
+#################################################################
+
+abi:intentDescription a owl:DatatypeProperty ;
+    rdfs:label "intent description"@en ;
+    rdfs:domain abi:TemplatableSparqlQuery ;
+    rdfs:range xsd:string ;
+    rdfs:comment "A natural language description of the query's intent."@en .
+
+abi:sparqlTemplate a owl:DatatypeProperty ;
+    rdfs:label "SPARQL template"@en ;
+    rdfs:domain abi:TemplatableSparqlQuery ;
+    rdfs:range xsd:string ;
+    rdfs:comment "The SPARQL query template with variable placeholders."@en .
+
+abi:argumentName a owl:DatatypeProperty ;
+    rdfs:label "argument name"@en ;
+    rdfs:domain abi:QueryArgument ;
+    rdfs:range xsd:string ;
+    rdfs:comment "The name of the argument used in the template."@en .
+
+abi:argumentDescription a owl:DatatypeProperty ;
+    rdfs:label "argument description"@en ;
+    rdfs:domain abi:QueryArgument ;
+    rdfs:range xsd:string ;
+    rdfs:comment "A description of what the argument represents."@en .
+
+abi:validationPattern a owl:DatatypeProperty ;
+    rdfs:label "validation pattern"@en ;
+    rdfs:domain abi:QueryArgument ;
+    rdfs:range xsd:string ;
+    rdfs:comment "A regex pattern for validating the argument value."@en .
+
+abi:validationFormat a owl:DatatypeProperty ;
+    rdfs:label "validation format"@en ;
+    rdfs:domain abi:QueryArgument ;
+    rdfs:range xsd:string ;
+    rdfs:comment "The expected format of the argument value (e.g., date, number, URI)."@en .
+
+#################################################################
+#    Example Usage
+#################################################################
+
+# # Example of a query to find employees in a department with optional name filter:
+# abi:findEmployeesQuery a abi:TemplatableSparqlQuery ;
+#     rdfs:label "findEmployeesQuery"@en ;
+#     abi:intentDescription "Find all employees working in a specific department, optionally filtered by name" ;
+#     abi:sparqlTemplate """
+#         SELECT ?employee ?name
+#         WHERE {
+#             ?employee :worksIn :{{ department_id }} ;
+#                     :hasName ?name .
+#             {% if employee_name %}
+#             FILTER(CONTAINS(LCASE(?name), LCASE("{{ employee_name }}")))
+#             {% endif %}
+#         }
+#     """ ;
+#     abi:hasArgument abi:departmentArg, abi:nameArg .
+
+# abi:departmentArg a abi:QueryArgument ;
+#     abi:argumentName "department_id" ;
+#     abi:argumentDescription "The unique identifier of the department" ;
+#     abi:validationPattern "^[A-Z0-9]{3,}$" ;
+#     abi:validationFormat "department_id" .
+
+# abi:nameArg a abi:QueryArgument ;
+#     abi:argumentName "employee_name" ;
+#     abi:argumentDescription "Optional name to filter employees (case-insensitive partial match)" ;
+#     abi:validationPattern "^[a-zA-Z0-9\\s-]{2,50}$" ;
+#     abi:validationFormat "employee_name" .
+
+# #################################################################
+

naas_abi_core/modules/templatablesparqlquery/workflows/GenericWorkflow.py

@@ -0,0 +1,48 @@
+from typing import Generic, Type, TypeVar
+
+from langchain_core.tools import BaseTool, StructuredTool
+from naas_abi_core.services.triple_store.TripleStoreService import TripleStoreService
+from naas_abi_core.utils.SPARQL import SPARQLUtils
+from pydantic import BaseModel
+
+T = TypeVar("T", bound=BaseModel)
+
+
+class GenericWorkflow(Generic[T]):
+    def __init__(
+        self,
+        name: str,
+        description: str,
+        sparql_template: str,
+        arguments_model: Type[T],
+        triple_store_service: TripleStoreService,
+    ):
+        self.name = name
+        self.description = description
+        self.sparql_template = sparql_template
+        self.arguments_model = arguments_model
+        self.triple_store_service = triple_store_service
+
+    def run(self, parameters: T):
+        try:
+            # Template the sparql template with the parameters using jinja2
+            from jinja2 import Template
+
+            template = Template(self.sparql_template)
+            sparql_query = template.render(parameters.model_dump())
+            # print(sparql_query)
+            results = self.triple_store_service.query(sparql_query)
+
+            return SPARQLUtils(self.triple_store_service).results_to_list(results)
+        except Exception as e:
+            return [{"error": str(e)}]
+
+    def as_tools(self) -> list[BaseTool]:
+        return [
+            StructuredTool(
+                name=self.name,
+                description=self.description,
+                func=lambda **kwargs: self.run(self.arguments_model(**kwargs)),
+                args_schema=self.arguments_model,
+            )
+        ]

naas_abi_core/modules/templatablesparqlquery/workflows/TemplatableSparqlQueryLoader.py

@@ -0,0 +1,192 @@
+import asyncio
+
+from naas_abi_core import logger
+from naas_abi_core.services.triple_store.TripleStoreService import TripleStoreService
+from pydantic import Field, create_model
+from rdflib import RDF, Graph, URIRef
+
+from .GenericWorkflow import GenericWorkflow
+
+
+async def async_asyncio_thread_jobs(jobs):
+    tasks = []
+    for job in jobs:
+        task = asyncio.create_task(asyncio.to_thread(*job))
+        tasks.append(task)
+    return await asyncio.gather(*tasks)
+
+
+def asyncio_thread_job(jobs):
+    return asyncio.run(async_asyncio_thread_jobs(jobs))
+
+
+class TemplatableSparqlQueryLoader:
+    triple_store_service: TripleStoreService
+
+    def __init__(self, triple_store_service: TripleStoreService):
+        self.triple_store_service = triple_store_service
+
+    def templatable_queries(self):
+        results = self.triple_store_service.query("""
+            PREFIX intentMapping: <http://ontology.naas.ai/intentMapping/>
+            PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
+            SELECT ?query ?label ?description ?sparqlTemplate ?hasArgument
+            WHERE {
+                ?query a intentMapping:TemplatableSparqlQuery ;
+                    intentMapping:intentDescription ?description ;
+                    intentMapping:sparqlTemplate ?sparqlTemplate ;
+                    intentMapping:hasArgument ?hasArgument ;
+                    rdfs:label ?label .
+            }
+        """)
+
+        queries = {}
+
+        for result in results:
+            query, label, description, sparqlTemplate, hasArgument = result
+            queries[query] = {
+                "label": label,
+                "description": description,
+                "sparqlTemplate": sparqlTemplate,
+                "hasArgument": [hasArgument]
+                if (query not in queries or queries[query].get("hasArgument") is None)
+                else queries[query].get("hasArgument") + [hasArgument],
+            }
+
+        arguments = {}
+
+        argument_graph = Graph()
+        argument_graph.bind(
+            "intentMapping", URIRef("http://ontology.naas.ai/intentMapping/")
+        )
+        results = self.triple_store_service.query("""
+                    PREFIX intentMapping: <http://ontology.naas.ai/intentMapping/>
+                    
+                    SELECT ?argument ?name ?description ?validationPattern ?validationFormat
+                    WHERE {
+                        ?argument a intentMapping:QueryArgument ;
+                            intentMapping:argumentName ?name ;
+                            intentMapping:argumentDescription ?description ;
+                            intentMapping:validationPattern ?validationPattern ;
+                            intentMapping:validationFormat ?validationFormat .
+                    }
+                """)
+
+        for argument, name, description, validationPattern, validationFormat in results:
+            argument_graph.add(
+                (
+                    argument,
+                    RDF.type,
+                    URIRef("http://ontology.naas.ai/intentMapping/QueryArgument"),
+                )
+            )
+            argument_graph.add(
+                (
+                    argument,
+                    URIRef("http://ontology.naas.ai/intentMapping/argumentName"),
+                    name,
+                )
+            )
+            argument_graph.add(
+                (
+                    argument,
+                    URIRef("http://ontology.naas.ai/intentMapping/argumentDescription"),
+                    description,
+                )
+            )
+            argument_graph.add(
+                (
+                    argument,
+                    URIRef("http://ontology.naas.ai/intentMapping/validationPattern"),
+                    validationPattern,
+                )
+            )
+            argument_graph.add(
+                (
+                    argument,
+                    URIRef("http://ontology.naas.ai/intentMapping/validationFormat"),
+                    validationFormat,
+                )
+            )
+
+        arguments = {}
+        for templatableQuery in queries:
+            for argument in queries[templatableQuery].get("hasArgument"):
+                q = (
+                    """
+                    PREFIX intentMapping: <http://ontology.naas.ai/intentMapping/>
+                    
+                    SELECT ?argument ?name ?description ?validationPattern ?validationFormat
+                    WHERE {
+                        BIND(<"""
+                    + str(argument)
+                    + """> AS ?argument)
+                        ?argument a intentMapping:QueryArgument ;
+                            intentMapping:argumentName ?name ;
+                            intentMapping:argumentDescription ?description ;
+                            intentMapping:validationPattern ?validationPattern ;
+                            intentMapping:validationFormat ?validationFormat .
+                    }
+                """
+                )
+                # logger.debug(f"Query: {q}")
+                results = argument_graph.query(q)
+
+                for result in results:
+                    argument, name, description, validationPattern, validationFormat = (
+                        result
+                    )
+
+                    arguments[argument] = {
+                        "name": name,
+                        "description": description,
+                        "validationPattern": validationPattern,
+                        "validationFormat": validationFormat,
+                    }
+
+        return queries, arguments
+
+    def load_workflows(self):
+        workflows = []
+
+        queries, arguments = self.templatable_queries()
+
+        # workflows = asyncio.run(__load_queries(queries, arguments))
+
+        # Now for each query, we need to create a Pydantic BaseModel based on the arguments
+        for _query in queries:
+            try:
+                query = queries[_query]
+
+                # Arguments Model with validation patterns
+                arguments_model = create_model(
+                    f"{str(query['label']).capitalize()}Arguments",
+                    **{
+                        str(arguments[argument]["name"]): (
+                            str,
+                            Field(
+                                ...,
+                                description=str(arguments[argument]["description"]),
+                                pattern=str(arguments[argument]["validationPattern"]),
+                                # You could also add additional metadata from validationFormat if needed
+                                example=str(arguments[argument]["validationFormat"]),
+                            ),
+                        )
+                        for argument in query.get("hasArgument")
+                    },
+                )
+
+                p = GenericWorkflow[arguments_model](
+                    str(query["label"]),
+                    str(query["description"]),
+                    str(query["sparqlTemplate"]),
+                    arguments_model,
+                    self.triple_store_service,
+                )
+                workflows.append(p)
+            except Exception as e:
+                logger.warning(
+                    f"Error loading workflow for query {query['label']}\nMessage: {e}"
+                )
+
+        return workflows

naas_abi_core/pipeline/__init__.py

@@ -0,0 +1,6 @@
+from naas_abi_core.pipeline.pipeline import Graph as Graph
+from naas_abi_core.pipeline.pipeline import Pipeline as Pipeline
+from naas_abi_core.pipeline.pipeline import (
+    PipelineConfiguration as PipelineConfiguration,
+)
+from naas_abi_core.pipeline.pipeline import PipelineParameters as PipelineParameters

naas_abi_core/pipeline/pipeline.py

@@ -0,0 +1,70 @@
+from abc import abstractmethod
+from dataclasses import dataclass
+from typing import Any
+
+from pydantic import BaseModel
+from rdflib import Graph
+
+from naas_abi_core.services.triple_store.TripleStorePorts import OntologyEvent
+from naas_abi_core.utils.Expose import Expose
+
+
+@dataclass
+class PipelineConfiguration:
+    """Base configuration class for pipelines.
+
+    This class serves as the base configuration for all pipeline classes.
+    Concrete pipeline implementations should extend this class and add their
+    specific configuration attributes.
+    """
+
+    pass
+
+
+class PipelineParameters(BaseModel):
+    """Base parameters class for pipeline execution.
+
+    This class serves as the base parameters for all pipeline executions.
+    Concrete pipeline implementations should extend this class and add their
+    specific runtime parameters.
+    """
+
+    pass
+
+
+class Pipeline(Expose):
+    __configuration: PipelineConfiguration
+
+    def __init__(self, configuration: PipelineConfiguration):
+        self.__configuration = configuration
+
+    # TODO: Make this an abstract method
+    # @abstractmethod
+    def trigger(
+        self, event: OntologyEvent, ontology_name: str, triple: tuple[Any, Any, Any]
+    ) -> Graph:
+        """Trigger the pipeline with the given event and triple.
+
+        This method should be implemented by concrete pipeline classes to process data
+        according to their specific logic and configuration.
+        """
+        raise NotImplementedError()
+
+    @abstractmethod
+    def run(self, parameters: PipelineParameters) -> Graph:
+        """Execute the pipeline with the given parameters.
+
+        This method should be implemented by concrete pipeline classes to process data
+        according to their specific logic and configuration.
+
+        Args:
+            parameters (PipelineParameters): Runtime parameters for pipeline execution
+                that control how the pipeline processes data
+
+        Returns:
+            Graph: An RDF graph containing the processed data and relationships
+
+        Raises:
+            NotImplementedError: If the concrete class does not implement this method
+        """
+        raise NotImplementedError()