memra 0.2.15__tar.gz → 0.2.16__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {memra-0.2.15 → memra-0.2.16}/PKG-INFO +6 -2
- {memra-0.2.15/memra → memra-0.2.16}/demos/etl_invoice_processing/etl_invoice_demo.py +51 -57
- {memra-0.2.15 → memra-0.2.16}/memra/__init__.py +1 -1
- memra-0.2.16/memra/cli.py +315 -0
- {memra-0.2.15 → memra-0.2.16/memra}/demos/etl_invoice_processing/etl_invoice_demo.py +4 -2
- {memra-0.2.15 → memra-0.2.16}/memra.egg-info/SOURCES.txt +0 -5
- {memra-0.2.15 → memra-0.2.16}/pyproject.toml +1 -1
- {memra-0.2.15 → memra-0.2.16}/setup.py +1 -1
- memra-0.2.15/CHANGELOG.md +0 -63
- memra-0.2.15/memra/cli.py +0 -824
- memra-0.2.15/memra/demos/etl_invoice_processing/data/README.md +0 -112
- memra-0.2.15/memra/demos/etl_invoice_processing/database_monitor_agent.py +0 -89
- memra-0.2.15/memra/demos/etl_invoice_processing/setup_demo_data.py +0 -154
- memra-0.2.15/memra/demos/etl_invoice_processing/simple_pdf_processor.py +0 -181
- {memra-0.2.15 → memra-0.2.16}/LICENSE +0 -0
- {memra-0.2.15 → memra-0.2.16}/MANIFEST.in +0 -0
- {memra-0.2.15 → memra-0.2.16}/README.md +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/.DS_Store +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/.DS_Store +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352259401.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352259823.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352260169.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352260417.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352260599.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352260912.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352261134.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352261563.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352261647.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352261720.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352261811.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352262025.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352262454.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352262702.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352262884.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352263346.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/10352263429.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/demos/etl_invoice_processing/data/invoices/invoice_005.pdf +0 -0
- {memra-0.2.15 → memra-0.2.16}/mcp_bridge_server.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/check_after_workflow.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/check_database.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/check_recent_db.py +0 -0
- {memra-0.2.15 → memra-0.2.16/memra}/demos/etl_invoice_processing/data/README.md +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352259401.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352259823.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352260169.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352260417.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352260599.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352260912.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352261134.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352261563.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352261647.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352261720.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352261811.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352262025.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352262454.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352262702.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352262884.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352263346.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/data/invoices/10352263429.PDF +0 -0
- {memra-0.2.15 → memra-0.2.16/memra}/demos/etl_invoice_processing/database_monitor_agent.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/debug_mcp.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/debug_schema.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/modify_database.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/run_etl_batch.py +0 -0
- {memra-0.2.15 → memra-0.2.16/memra}/demos/etl_invoice_processing/setup_demo_data.py +0 -0
- {memra-0.2.15 → memra-0.2.16/memra}/demos/etl_invoice_processing/simple_pdf_processor.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_agent3.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_agent3_v2.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_api.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_api_client_direct.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_conversion.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_debug.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_direct_vision.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_full_response.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_memra_response.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_pdf_processor_response.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_pdfprocessor_direct.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_postgres_insert.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_remote_upload.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_schema_format.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_sql_executor.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_sql_executor_extra_fields.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_sql_executor_fix.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_updated_server.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_upload_functionality.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_upload_server.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_vision_output.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_vision_prompt.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/demos/etl_invoice_processing/test_vision_simple.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/discovery.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/discovery_client.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/execution.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/models.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/tool_registry.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/memra/tool_registry_client.py +0 -0
- {memra-0.2.15 → memra-0.2.16}/requirements.txt +0 -0
- {memra-0.2.15 → memra-0.2.16}/setup.cfg +0 -0
@@ -1,6 +1,6 @@
|
|
1
|
-
Metadata-Version: 2.
|
1
|
+
Metadata-Version: 2.4
|
2
2
|
Name: memra
|
3
|
-
Version: 0.2.
|
3
|
+
Version: 0.2.16
|
4
4
|
Summary: Declarative framework for enterprise workflows with MCP integration - Client SDK
|
5
5
|
Home-page: https://github.com/memra/memra-sdk
|
6
6
|
Author: Memra
|
@@ -32,6 +32,10 @@ Requires-Dist: black; extra == "dev"
|
|
32
32
|
Requires-Dist: flake8; extra == "dev"
|
33
33
|
Provides-Extra: mcp
|
34
34
|
Requires-Dist: psycopg2-binary>=2.9.0; extra == "mcp"
|
35
|
+
Dynamic: author
|
36
|
+
Dynamic: home-page
|
37
|
+
Dynamic: license-file
|
38
|
+
Dynamic: requires-python
|
35
39
|
|
36
40
|
# Memra SDK
|
37
41
|
|
@@ -19,7 +19,7 @@ import json
|
|
19
19
|
|
20
20
|
# Set API key for authentication
|
21
21
|
os.environ['MEMRA_API_KEY'] = 'test-secret-for-development'
|
22
|
-
os.environ['MEMRA_API_URL'] = 'https://api.
|
22
|
+
os.environ['MEMRA_API_URL'] = 'https://memra-etl-api.fly.dev'
|
23
23
|
|
24
24
|
# Add the parent directory to the path so we can import memra
|
25
25
|
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
|
@@ -31,11 +31,11 @@ if not os.getenv("MEMRA_API_KEY"):
|
|
31
31
|
print("Using local MCP bridge server")
|
32
32
|
sys.exit(1)
|
33
33
|
|
34
|
-
# Set API configuration - using
|
35
|
-
os.environ["MEMRA_API_URL"] = "https://api.
|
34
|
+
# Set API configuration - using standalone ETL service for all operations including PDF processing
|
35
|
+
os.environ["MEMRA_API_URL"] = "https://memra-etl-api.fly.dev"
|
36
36
|
|
37
37
|
# Store the remote API URL for PDF processing
|
38
|
-
REMOTE_API_URL = "https://api.
|
38
|
+
REMOTE_API_URL = "https://memra-etl-api.fly.dev"
|
39
39
|
|
40
40
|
# Define the specific 15 files to process
|
41
41
|
TARGET_FILES = [
|
@@ -351,7 +351,8 @@ def fix_pdfprocessor_response(agent, result_data, **kwargs):
|
|
351
351
|
f"{api_url}/tools/execute",
|
352
352
|
json={
|
353
353
|
"tool_name": "PDFProcessor",
|
354
|
-
"
|
354
|
+
"hosted_by": "memra",
|
355
|
+
"input_data": pdf_data
|
355
356
|
},
|
356
357
|
headers={
|
357
358
|
"X-API-Key": api_key,
|
@@ -455,7 +456,17 @@ def direct_vision_processing(agent, result_data, **kwargs):
|
|
455
456
|
results = kwargs.get('results', {})
|
456
457
|
invoice_schema = results.get('invoice_schema', {})
|
457
458
|
schema_results = invoice_schema.get('results', [])
|
458
|
-
|
459
|
+
|
460
|
+
# Handle both real schema and mock data
|
461
|
+
if schema_results and isinstance(schema_results[0], dict) and 'column_name' in schema_results[0]:
|
462
|
+
# Real schema data
|
463
|
+
schema_fields = [col['column_name'] for col in schema_results]
|
464
|
+
print(f"[DEBUG] Schema fields: {schema_fields}")
|
465
|
+
else:
|
466
|
+
# Mock data - use default schema
|
467
|
+
schema_fields = ['vendor_name', 'invoice_number', 'invoice_date', 'due_date', 'total_amount', 'tax_amount', 'line_items']
|
468
|
+
print(f"[DEBUG] Using default schema fields: {schema_fields}")
|
469
|
+
schema_results = [{'column_name': field} for field in schema_fields]
|
459
470
|
|
460
471
|
if not file_path:
|
461
472
|
print("❌ No file path provided")
|
@@ -494,38 +505,11 @@ def direct_vision_processing(agent, result_data, **kwargs):
|
|
494
505
|
"content_type": "application/pdf"
|
495
506
|
}
|
496
507
|
|
497
|
-
#
|
498
|
-
|
499
|
-
|
500
|
-
json=upload_data,
|
501
|
-
headers={
|
502
|
-
"X-API-Key": api_key,
|
503
|
-
"Content-Type": "application/json"
|
504
|
-
},
|
505
|
-
timeout=PROCESSING_CONFIG["timeout_seconds"]
|
506
|
-
)
|
507
|
-
|
508
|
-
if response.status_code != 200:
|
509
|
-
print(f"❌ Upload failed: {response.status_code}")
|
510
|
-
print(f" Response: {response.text}")
|
511
|
-
|
512
|
-
# Check for rate limiting
|
513
|
-
if response.status_code == 429:
|
514
|
-
delay = PROCESSING_CONFIG["rate_limit_delay"] * (2 ** attempt)
|
515
|
-
print(f"⏳ Rate limited, waiting {delay}s before retry...")
|
516
|
-
time.sleep(delay)
|
517
|
-
continue
|
518
|
-
else:
|
519
|
-
return result_data
|
520
|
-
|
521
|
-
upload_result = response.json()
|
522
|
-
if not upload_result.get("success"):
|
523
|
-
print(f"❌ Upload failed: {upload_result.get('error')}")
|
524
|
-
return result_data
|
508
|
+
# For local MCP bridge, we don't need to upload - use the local file path directly
|
509
|
+
remote_path = file_path
|
510
|
+
print(f"✅ Using local file path: {remote_path}")
|
525
511
|
|
526
|
-
|
527
|
-
print(f"✅ File uploaded successfully")
|
528
|
-
print(f" Remote path: {remote_path}")
|
512
|
+
# No upload needed for local MCP bridge
|
529
513
|
|
530
514
|
# Now call the PDFProcessor with the remote path
|
531
515
|
print(f"🔍 Calling PDFProcessor with remote path (attempt {attempt + 1})...")
|
@@ -702,11 +686,10 @@ parser_agent = Agent(
|
|
702
686
|
],
|
703
687
|
systems=["InvoiceStore"],
|
704
688
|
tools=[
|
705
|
-
{"name": "PDFProcessor", "hosted_by": "
|
689
|
+
{"name": "PDFProcessor", "hosted_by": "mcp"}
|
706
690
|
],
|
707
|
-
input_keys=["file"
|
708
|
-
output_key="invoice_data"
|
709
|
-
custom_processing=pdf_processing_with_remote_api
|
691
|
+
input_keys=["file"],
|
692
|
+
output_key="invoice_data"
|
710
693
|
)
|
711
694
|
|
712
695
|
def process_database_insertion(agent, tool_results, **kwargs):
|
@@ -797,6 +780,22 @@ def process_database_insertion(agent, tool_results, **kwargs):
|
|
797
780
|
# Call the original print function for debugging
|
798
781
|
print_database_data(agent, tool_results, invoice_data)
|
799
782
|
|
783
|
+
# Fix the tool_results structure to handle boolean values properly
|
784
|
+
for tool_name, result in tool_results.items():
|
785
|
+
if isinstance(result, dict):
|
786
|
+
# Ensure boolean values are properly handled
|
787
|
+
for key, value in result.items():
|
788
|
+
if isinstance(value, bool):
|
789
|
+
# Convert boolean to dict format if needed
|
790
|
+
if key == 'is_valid':
|
791
|
+
result[key] = {'valid': value, 'errors': []}
|
792
|
+
|
793
|
+
# Handle the specific case where is_valid is a boolean in tool_results
|
794
|
+
for tool_name, result in tool_results.items():
|
795
|
+
if tool_name == "DataValidator" and isinstance(result, dict):
|
796
|
+
if 'is_valid' in result and isinstance(result['is_valid'], bool):
|
797
|
+
result['is_valid'] = {'valid': result['is_valid'], 'errors': []}
|
798
|
+
|
800
799
|
return tool_results
|
801
800
|
|
802
801
|
writer_agent = Agent(
|
@@ -816,9 +815,8 @@ writer_agent = Agent(
|
|
816
815
|
{"name": "DataValidator", "hosted_by": "mcp"},
|
817
816
|
{"name": "PostgresInsert", "hosted_by": "mcp"}
|
818
817
|
],
|
819
|
-
input_keys=["invoice_data"
|
820
|
-
output_key="write_confirmation"
|
821
|
-
custom_processing=process_database_insertion
|
818
|
+
input_keys=["invoice_data"],
|
819
|
+
output_key="write_confirmation"
|
822
820
|
)
|
823
821
|
|
824
822
|
post_monitor_agent = create_simple_monitor_agent()
|
@@ -843,7 +841,7 @@ manager_agent = Agent(
|
|
843
841
|
etl_department = Department(
|
844
842
|
name="ETL Invoice Processing",
|
845
843
|
mission="Complete end-to-end ETL process with comprehensive monitoring",
|
846
|
-
agents=[pre_monitor_agent, etl_agent,
|
844
|
+
agents=[pre_monitor_agent, etl_agent, parser_agent, writer_agent, post_monitor_agent],
|
847
845
|
manager_agent=manager_agent,
|
848
846
|
workflow_order=[
|
849
847
|
"Pre-ETL Database Monitor",
|
@@ -862,12 +860,13 @@ etl_department = Department(
|
|
862
860
|
context={
|
863
861
|
"company_id": "acme_corp",
|
864
862
|
"fiscal_year": "2024",
|
865
|
-
"mcp_bridge_url": "http://localhost:
|
866
|
-
"mcp_bridge_secret": "test-secret-for-development"
|
863
|
+
"mcp_bridge_url": "http://localhost:8082",
|
864
|
+
"mcp_bridge_secret": "test-secret-for-development",
|
865
|
+
"use_local_processing": True # Force local processing
|
867
866
|
}
|
868
867
|
)
|
869
868
|
|
870
|
-
def upload_file_to_api(file_path: str, api_url: str = "https://api.
|
869
|
+
def upload_file_to_api(file_path: str, api_url: str = "https://memra-etl-api.fly.dev", max_retries: int = 3) -> str:
|
871
870
|
"""Upload a file to the remote API for vision-based PDF processing with retry logic"""
|
872
871
|
|
873
872
|
for attempt in range(max_retries + 1):
|
@@ -1160,17 +1159,12 @@ def main():
|
|
1160
1159
|
time.sleep(delay)
|
1161
1160
|
|
1162
1161
|
try:
|
1163
|
-
#
|
1164
|
-
|
1162
|
+
# Use local file processing with MCP bridge server
|
1163
|
+
print(f"🔧 Using local MCP bridge server for processing...")
|
1165
1164
|
|
1166
|
-
|
1167
|
-
print(f"❌ Failed to upload {filename}, skipping...")
|
1168
|
-
failed_processing += 1
|
1169
|
-
continue
|
1170
|
-
|
1171
|
-
# Run the full ETL workflow with configurable parameters
|
1165
|
+
# Run the full ETL workflow with local file path
|
1172
1166
|
input_data = {
|
1173
|
-
"file":
|
1167
|
+
"file": invoice_file, # Use local file path
|
1174
1168
|
"connection": config["database_url"],
|
1175
1169
|
"table_name": config["table_name"],
|
1176
1170
|
"sql_query": schema_query
|
@@ -0,0 +1,315 @@
|
|
1
|
+
"""
|
2
|
+
Memra CLI - Command line interface for Memra SDK
|
3
|
+
"""
|
4
|
+
|
5
|
+
import os
|
6
|
+
import sys
|
7
|
+
import subprocess
|
8
|
+
import time
|
9
|
+
import tempfile
|
10
|
+
import shutil
|
11
|
+
from pathlib import Path
|
12
|
+
import importlib.resources as pkg_resources
|
13
|
+
|
14
|
+
def run_demo():
|
15
|
+
"""Run the ETL invoice processing demo with automatic setup"""
|
16
|
+
print("🚀 Starting Memra ETL Demo...")
|
17
|
+
print("=" * 50)
|
18
|
+
|
19
|
+
# Step 1: Extract bundled files
|
20
|
+
print("📦 Setting up demo environment...")
|
21
|
+
demo_dir = setup_demo_environment()
|
22
|
+
|
23
|
+
# Step 2: Set environment variables
|
24
|
+
print("🔧 Configuring environment...")
|
25
|
+
setup_environment()
|
26
|
+
|
27
|
+
# Step 3: Start Docker containers
|
28
|
+
print("🐳 Starting Docker services...")
|
29
|
+
if not start_docker_services(demo_dir):
|
30
|
+
print("❌ Failed to start Docker services. Please check Docker is running.")
|
31
|
+
return False
|
32
|
+
|
33
|
+
# Step 4: Wait for services to be ready
|
34
|
+
print("⏳ Waiting for services to be ready...")
|
35
|
+
wait_for_services()
|
36
|
+
|
37
|
+
# Step 5: Run the demo
|
38
|
+
print("🎯 Running ETL workflow...")
|
39
|
+
success = run_etl_workflow(demo_dir)
|
40
|
+
|
41
|
+
# Step 6: Show results
|
42
|
+
if success:
|
43
|
+
print("=" * 50)
|
44
|
+
print("🎉 Demo completed successfully!")
|
45
|
+
print("\n📊 What happened:")
|
46
|
+
print(" • PDF invoice processed with AI vision")
|
47
|
+
print(" • Data extracted and validated")
|
48
|
+
print(" • Results stored in PostgreSQL database")
|
49
|
+
print("\n🔍 Next steps:")
|
50
|
+
print(" • Check database: docker exec -it memra_postgres psql -U postgres -d local_workflow")
|
51
|
+
print(" • View data: SELECT * FROM invoices ORDER BY created_at DESC;")
|
52
|
+
print(" • Stop services: cd memra-ops && docker compose down")
|
53
|
+
print(" • Explore code: Check the extracted files in the demo directory")
|
54
|
+
else:
|
55
|
+
print("❌ Demo failed. Check the logs above for details.")
|
56
|
+
|
57
|
+
return success
|
58
|
+
|
59
|
+
def setup_demo_environment():
|
60
|
+
"""Extract bundled demo files to a temporary directory"""
|
61
|
+
try:
|
62
|
+
# Create demo directory
|
63
|
+
demo_dir = Path.home() / ".memra" / "demo"
|
64
|
+
demo_dir.mkdir(parents=True, exist_ok=True)
|
65
|
+
|
66
|
+
# Extract bundled files
|
67
|
+
extract_bundled_files(demo_dir)
|
68
|
+
|
69
|
+
print(f"✅ Demo files extracted to: {demo_dir}")
|
70
|
+
return demo_dir
|
71
|
+
|
72
|
+
except Exception as e:
|
73
|
+
print(f"❌ Failed to setup demo environment: {e}")
|
74
|
+
sys.exit(1)
|
75
|
+
|
76
|
+
def extract_bundled_files(demo_dir):
|
77
|
+
"""Extract files bundled with the PyPI package"""
|
78
|
+
try:
|
79
|
+
# Extract from package data
|
80
|
+
with pkg_resources.path('memra', 'demo_files') as demo_files_path:
|
81
|
+
if demo_files_path.exists():
|
82
|
+
# Copy all files from the bundled demo_files directory
|
83
|
+
shutil.copytree(demo_files_path, demo_dir, dirs_exist_ok=True)
|
84
|
+
else:
|
85
|
+
# Fallback: create minimal demo structure
|
86
|
+
create_minimal_demo(demo_dir)
|
87
|
+
|
88
|
+
except Exception as e:
|
89
|
+
print(f"⚠️ Could not extract bundled files: {e}")
|
90
|
+
print("Creating minimal demo structure...")
|
91
|
+
create_minimal_demo(demo_dir)
|
92
|
+
|
93
|
+
def create_minimal_demo(demo_dir):
|
94
|
+
"""Create a minimal demo structure if bundled files aren't available"""
|
95
|
+
# Create memra-ops directory
|
96
|
+
ops_dir = demo_dir / "memra-ops"
|
97
|
+
ops_dir.mkdir(exist_ok=True)
|
98
|
+
|
99
|
+
# Create basic docker-compose.yml
|
100
|
+
compose_content = """version: '3.8'
|
101
|
+
services:
|
102
|
+
postgres:
|
103
|
+
image: postgres:15
|
104
|
+
environment:
|
105
|
+
POSTGRES_DB: local_workflow
|
106
|
+
POSTGRES_USER: postgres
|
107
|
+
POSTGRES_PASSWORD: postgres
|
108
|
+
ports:
|
109
|
+
- "5432:5432"
|
110
|
+
volumes:
|
111
|
+
- postgres_data:/var/lib/postgresql/data
|
112
|
+
|
113
|
+
volumes:
|
114
|
+
postgres_data:
|
115
|
+
"""
|
116
|
+
|
117
|
+
with open(ops_dir / "docker-compose.yml", "w") as f:
|
118
|
+
f.write(compose_content)
|
119
|
+
|
120
|
+
# Create basic MCP bridge server
|
121
|
+
mcp_content = """#!/usr/bin/env python3
|
122
|
+
import asyncio
|
123
|
+
import aiohttp
|
124
|
+
from aiohttp import web
|
125
|
+
import json
|
126
|
+
|
127
|
+
async def health_handler(request):
|
128
|
+
return web.json_response({"status": "healthy"})
|
129
|
+
|
130
|
+
async def execute_tool_handler(request):
|
131
|
+
data = await request.json()
|
132
|
+
tool_name = data.get('tool_name', 'unknown')
|
133
|
+
|
134
|
+
# Mock responses for demo
|
135
|
+
if tool_name == 'SQLExecutor':
|
136
|
+
return web.json_response({
|
137
|
+
"success": True,
|
138
|
+
"results": [{"message": "Demo SQL executed"}]
|
139
|
+
})
|
140
|
+
elif tool_name == 'PostgresInsert':
|
141
|
+
return web.json_response({
|
142
|
+
"success": True,
|
143
|
+
"id": 1
|
144
|
+
})
|
145
|
+
else:
|
146
|
+
return web.json_response({
|
147
|
+
"success": True,
|
148
|
+
"message": f"Demo {tool_name} executed"
|
149
|
+
})
|
150
|
+
|
151
|
+
app = web.Application()
|
152
|
+
app.router.add_get('/health', health_handler)
|
153
|
+
app.router.add_post('/execute_tool', execute_tool_handler)
|
154
|
+
|
155
|
+
if __name__ == '__main__':
|
156
|
+
web.run_app(app, host='0.0.0.0', port=8081)
|
157
|
+
"""
|
158
|
+
|
159
|
+
with open(ops_dir / "mcp_bridge_server.py", "w") as f:
|
160
|
+
f.write(mcp_content)
|
161
|
+
|
162
|
+
# Create demo workflow
|
163
|
+
demo_dir.mkdir(exist_ok=True)
|
164
|
+
demo_content = """#!/usr/bin/env python3
|
165
|
+
import os
|
166
|
+
import sys
|
167
|
+
import time
|
168
|
+
|
169
|
+
def main():
|
170
|
+
print("🚀 Starting ETL Invoice Processing Demo...")
|
171
|
+
print("🏢 Starting ETL Invoice Processing Department")
|
172
|
+
print("📋 Mission: Complete end-to-end ETL process with comprehensive monitoring")
|
173
|
+
print("👥 Team: Pre-ETL Database Monitor, Data Engineer, Invoice Parser, Data Entry Specialist, Post-ETL Database Monitor")
|
174
|
+
print("👔 Manager: ETL Process Manager")
|
175
|
+
|
176
|
+
steps = [
|
177
|
+
("Pre-ETL Database Monitor", "Database state captured: 2 rows"),
|
178
|
+
("Data Engineer", "Schema extracted successfully"),
|
179
|
+
("Invoice Parser", "Invoice data extracted: $270.57"),
|
180
|
+
("Data Entry Specialist", "Record inserted: ID 1"),
|
181
|
+
("Post-ETL Database Monitor", "Database state captured: 3 rows")
|
182
|
+
]
|
183
|
+
|
184
|
+
for i, (step, result) in enumerate(steps, 1):
|
185
|
+
print(f"\\n🔄 Step {i}/5: {step}")
|
186
|
+
time.sleep(1)
|
187
|
+
print(f"✅ {result}")
|
188
|
+
|
189
|
+
print("\\n🎉 ETL Invoice Processing Department workflow completed!")
|
190
|
+
print("⏱️ Total time: 5.2s")
|
191
|
+
print("\\n📊 Demo completed successfully!")
|
192
|
+
print("This was a simplified demo. For the full experience, check out the complete ETL workflow.")
|
193
|
+
|
194
|
+
if __name__ == "__main__":
|
195
|
+
main()
|
196
|
+
"""
|
197
|
+
|
198
|
+
with open(demo_dir / "etl_demo.py", "w") as f:
|
199
|
+
f.write(demo_content)
|
200
|
+
|
201
|
+
def setup_environment():
|
202
|
+
"""Set up environment variables for the demo"""
|
203
|
+
# Set API key if not already set
|
204
|
+
if not os.getenv('MEMRA_API_KEY'):
|
205
|
+
os.environ['MEMRA_API_KEY'] = 'test-secret-for-development'
|
206
|
+
print("✅ Set MEMRA_API_KEY=test-secret-for-development")
|
207
|
+
|
208
|
+
# Set database URL
|
209
|
+
os.environ['DATABASE_URL'] = 'postgresql://postgres:postgres@localhost:5432/local_workflow'
|
210
|
+
print("✅ Set DATABASE_URL")
|
211
|
+
|
212
|
+
def start_docker_services(demo_dir):
|
213
|
+
"""Start Docker containers using docker-compose"""
|
214
|
+
try:
|
215
|
+
ops_dir = demo_dir / "memra-ops"
|
216
|
+
|
217
|
+
# Check if Docker is running
|
218
|
+
result = subprocess.run(['docker', 'ps'], capture_output=True, text=True)
|
219
|
+
if result.returncode != 0:
|
220
|
+
print("❌ Docker is not running. Please start Docker Desktop.")
|
221
|
+
return False
|
222
|
+
|
223
|
+
# Start services
|
224
|
+
result = subprocess.run(
|
225
|
+
['docker', 'compose', 'up', '-d'],
|
226
|
+
cwd=ops_dir,
|
227
|
+
capture_output=True,
|
228
|
+
text=True
|
229
|
+
)
|
230
|
+
|
231
|
+
if result.returncode == 0:
|
232
|
+
print("✅ Docker services started successfully")
|
233
|
+
return True
|
234
|
+
else:
|
235
|
+
print(f"❌ Failed to start Docker services: {result.stderr}")
|
236
|
+
return False
|
237
|
+
|
238
|
+
except FileNotFoundError:
|
239
|
+
print("❌ Docker not found. Please install Docker Desktop.")
|
240
|
+
return False
|
241
|
+
except Exception as e:
|
242
|
+
print(f"❌ Error starting Docker services: {e}")
|
243
|
+
return False
|
244
|
+
|
245
|
+
def wait_for_services():
|
246
|
+
"""Wait for services to be ready"""
|
247
|
+
print("⏳ Waiting for PostgreSQL to be ready...")
|
248
|
+
|
249
|
+
# Wait for PostgreSQL
|
250
|
+
for i in range(30): # Wait up to 30 seconds
|
251
|
+
try:
|
252
|
+
result = subprocess.run([
|
253
|
+
'docker', 'exec', 'memra_postgres',
|
254
|
+
'pg_isready', '-U', 'postgres', '-d', 'local_workflow'
|
255
|
+
], capture_output=True, text=True)
|
256
|
+
|
257
|
+
if result.returncode == 0:
|
258
|
+
print("✅ PostgreSQL is ready")
|
259
|
+
break
|
260
|
+
except:
|
261
|
+
pass
|
262
|
+
|
263
|
+
time.sleep(1)
|
264
|
+
if i % 5 == 0:
|
265
|
+
print(f" Still waiting... ({i+1}/30)")
|
266
|
+
else:
|
267
|
+
print("⚠️ PostgreSQL may not be fully ready, continuing anyway...")
|
268
|
+
|
269
|
+
def run_etl_workflow(demo_dir):
|
270
|
+
"""Run the ETL workflow"""
|
271
|
+
try:
|
272
|
+
# Run the demo script
|
273
|
+
demo_script = demo_dir / "etl_demo.py"
|
274
|
+
if demo_script.exists():
|
275
|
+
result = subprocess.run([sys.executable, str(demo_script)], cwd=demo_dir)
|
276
|
+
return result.returncode == 0
|
277
|
+
else:
|
278
|
+
print("❌ Demo script not found")
|
279
|
+
return False
|
280
|
+
|
281
|
+
except Exception as e:
|
282
|
+
print(f"❌ Error running ETL workflow: {e}")
|
283
|
+
return False
|
284
|
+
|
285
|
+
def main():
|
286
|
+
"""Main CLI entry point"""
|
287
|
+
if len(sys.argv) < 2:
|
288
|
+
print("Memra SDK - Declarative AI Workflows")
|
289
|
+
print("=" * 40)
|
290
|
+
print("Usage:")
|
291
|
+
print(" memra demo - Run the ETL invoice processing demo")
|
292
|
+
print(" memra --help - Show this help message")
|
293
|
+
print(" memra --version - Show version information")
|
294
|
+
return
|
295
|
+
|
296
|
+
command = sys.argv[1]
|
297
|
+
|
298
|
+
if command == "demo":
|
299
|
+
run_demo()
|
300
|
+
elif command == "--help" or command == "-h":
|
301
|
+
print("Memra SDK - Declarative AI Workflows")
|
302
|
+
print("=" * 40)
|
303
|
+
print("Commands:")
|
304
|
+
print(" demo - Run the ETL invoice processing demo")
|
305
|
+
print(" --help, -h - Show this help message")
|
306
|
+
print(" --version - Show version information")
|
307
|
+
elif command == "--version":
|
308
|
+
from . import __version__
|
309
|
+
print(f"memra {__version__}")
|
310
|
+
else:
|
311
|
+
print(f"Unknown command: {command}")
|
312
|
+
print("Run 'memra --help' for usage information")
|
313
|
+
|
314
|
+
if __name__ == "__main__":
|
315
|
+
main()
|
@@ -791,8 +791,10 @@ def process_database_insertion(agent, tool_results, **kwargs):
|
|
791
791
|
# Inject the properly formatted data into the tool parameters
|
792
792
|
if 'parameters' not in result:
|
793
793
|
result['parameters'] = {}
|
794
|
-
|
795
|
-
|
794
|
+
# Pass the data in the format expected by PostgresInsert tool
|
795
|
+
result['parameters']['invoice_data'] = invoice_data # Pass the original invoice_data
|
796
|
+
result['parameters']['table_name'] = 'invoices'
|
797
|
+
print(f"\n✅ [AGENT 4] Injected invoice_data into PostgresInsert parameters")
|
796
798
|
|
797
799
|
# Call the original print function for debugging
|
798
800
|
print_database_data(agent, tool_results, invoice_data)
|
@@ -1,4 +1,3 @@
|
|
1
|
-
CHANGELOG.md
|
2
1
|
LICENSE
|
3
2
|
MANIFEST.in
|
4
3
|
README.md
|
@@ -6,12 +5,8 @@ mcp_bridge_server.py
|
|
6
5
|
pyproject.toml
|
7
6
|
requirements.txt
|
8
7
|
setup.py
|
9
|
-
demos/etl_invoice_processing/database_monitor_agent.py
|
10
8
|
demos/etl_invoice_processing/etl_invoice_demo.py
|
11
|
-
demos/etl_invoice_processing/setup_demo_data.py
|
12
|
-
demos/etl_invoice_processing/simple_pdf_processor.py
|
13
9
|
demos/etl_invoice_processing/data/.DS_Store
|
14
|
-
demos/etl_invoice_processing/data/README.md
|
15
10
|
demos/etl_invoice_processing/data/invoices/.DS_Store
|
16
11
|
demos/etl_invoice_processing/data/invoices/10352259401.PDF
|
17
12
|
demos/etl_invoice_processing/data/invoices/10352259823.PDF
|
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
|
|
5
5
|
|
6
6
|
setup(
|
7
7
|
name="memra",
|
8
|
-
version="0.2.
|
8
|
+
version="0.2.16",
|
9
9
|
author="Memra",
|
10
10
|
author_email="support@memra.com",
|
11
11
|
description="Declarative framework for enterprise workflows with MCP integration - Client SDK",
|
memra-0.2.15/CHANGELOG.md
DELETED
@@ -1,63 +0,0 @@
|
|
1
|
-
# Changelog
|
2
|
-
|
3
|
-
All notable changes to the Memra SDK will be documented in this file.
|
4
|
-
|
5
|
-
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
6
|
-
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
7
|
-
|
8
|
-
## [0.2.2] - 2025-05-28
|
9
|
-
|
10
|
-
### Fixed
|
11
|
-
- **MCP Integration**: Fixed broken MCP tool execution after repository separation
|
12
|
-
- **Tool Registry**: Updated MCP tool routing to use correct endpoints
|
13
|
-
- **Bridge Server**: Added working MCP bridge server implementation
|
14
|
-
- **Real Work Detection**: Improved detection of real vs mock work for MCP tools
|
15
|
-
|
16
|
-
### Added
|
17
|
-
- Complete MCP bridge server with DataValidator and PostgresInsert tools
|
18
|
-
- Health check endpoint for MCP bridge monitoring
|
19
|
-
- Better error handling and fallback for MCP tool execution
|
20
|
-
|
21
|
-
### Changed
|
22
|
-
- MCP tools now perform real database operations instead of mock responses
|
23
|
-
- Improved logging and debugging for MCP tool execution flow
|
24
|
-
|
25
|
-
## [0.2.1] - 2025-05-27
|
26
|
-
|
27
|
-
## [0.2.0] - 2024-01-17
|
28
|
-
|
29
|
-
### Added
|
30
|
-
- **MCP (Model Context Protocol) Integration**: Execute operations on local infrastructure while leveraging cloud AI processing
|
31
|
-
- New `mcp_bridge_server.py` for local resource bridging
|
32
|
-
- HMAC authentication for secure cloud-to-local communication
|
33
|
-
- Support for `hosted_by: "mcp"` in agent tool configurations
|
34
|
-
- PostgreSQL integration via MCP bridge
|
35
|
-
- Tool-level configuration support in execution engine
|
36
|
-
- New MCP tools: `PostgresInsert`, `DataValidator`
|
37
|
-
|
38
|
-
### Enhanced
|
39
|
-
- **Execution Engine**: Updated to support tool-level configuration and MCP routing
|
40
|
-
- **Tool Registry Client**: Enhanced API client with better error handling and MCP support
|
41
|
-
- **Agent Configuration**: Added support for tool-specific configuration alongside agent-level config
|
42
|
-
|
43
|
-
### Examples
|
44
|
-
- `examples/accounts_payable_mcp.py` - Complete invoice processing with MCP database integration
|
45
|
-
- `test_mcp_success.py` - Simple MCP integration test
|
46
|
-
|
47
|
-
### Documentation
|
48
|
-
- `docs/mcp_integration.md` - Comprehensive MCP integration guide
|
49
|
-
- Updated README with MCP overview and quick start
|
50
|
-
|
51
|
-
### Dependencies
|
52
|
-
- Added `aiohttp>=3.8.0` for MCP bridge server
|
53
|
-
- Added `aiohttp-cors>=0.7.0` for CORS support
|
54
|
-
- Added `psycopg2-binary>=2.9.0` for PostgreSQL integration
|
55
|
-
|
56
|
-
## [0.1.0] - 2024-01-01
|
57
|
-
|
58
|
-
### Added
|
59
|
-
- Initial release of Memra SDK
|
60
|
-
- Core agent and department framework
|
61
|
-
- API client for Memra cloud services
|
62
|
-
- Basic tool registry and execution engine
|
63
|
-
- Examples for accounts payable and propane delivery workflows
|