data-connectors-ai 0.1.0__tar.gz

data_connectors_ai-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Nagarjun Rajendran
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data_connectors_ai-0.1.0/PKG-INFO

@@ -0,0 +1,476 @@
+Metadata-Version: 2.4
+Name: data-connectors-ai
+Version: 0.1.0
+Summary: Production-ready data connector libraries for AI applications and data processing pipelines
+Author-email: Nagarjun Rajendran <nagarjunr@live.in>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/nagarjunr/data-connectors-for-ai-agents
+Project-URL: Repository, https://github.com/nagarjunr/data-connectors-for-ai-agents
+Project-URL: Issues, https://github.com/nagarjunr/data-connectors-for-ai-agents/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: boto3>=1.26.0
+Requires-Dist: google-cloud-storage>=2.10.0
+Requires-Dist: psycopg2-binary>=2.9.0
+Requires-Dist: sqlalchemy>=2.0.0
+Requires-Dist: requests>=2.31.0
+Requires-Dist: python-dateutil>=2.8.0
+Requires-Dist: pika>=1.3.0
+Provides-Extra: s3
+Requires-Dist: boto3>=1.26.0; extra == "s3"
+Provides-Extra: gcp
+Requires-Dist: google-cloud-storage>=2.10.0; extra == "gcp"
+Provides-Extra: postgres
+Requires-Dist: psycopg2-binary>=2.9.0; extra == "postgres"
+Requires-Dist: sqlalchemy>=2.0.0; extra == "postgres"
+Provides-Extra: sharepoint
+Requires-Dist: requests>=2.31.0; extra == "sharepoint"
+Requires-Dist: msal>=1.20.0; extra == "sharepoint"
+Provides-Extra: onedrive
+Requires-Dist: requests>=2.31.0; extra == "onedrive"
+Requires-Dist: msal>=1.20.0; extra == "onedrive"
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.0.0; extra == "mcp"
+Provides-Extra: all
+Requires-Dist: boto3>=1.26.0; extra == "all"
+Requires-Dist: google-cloud-storage>=2.10.0; extra == "all"
+Requires-Dist: psycopg2-binary>=2.9.0; extra == "all"
+Requires-Dist: sqlalchemy>=2.0.0; extra == "all"
+Requires-Dist: requests>=2.31.0; extra == "all"
+Requires-Dist: msal>=1.20.0; extra == "all"
+Requires-Dist: pika>=1.3.0; extra == "all"
+Requires-Dist: mcp>=1.0.0; extra == "all"
+Dynamic: license-file
+
+# Data Connectors
+
+Production-ready, reusable data connector libraries for AI applications and data processing pipelines.
+
+![Python](https://img.shields.io/badge/python-3.12%2B-blue.svg)
+![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)
+
+**Key Features:**
+- PostgreSQL, S3-compatible, GCP Storage, SharePoint, OneDrive connectors
+- Enterprise-ready (proxy support, CA certificates)
+- Type-hinted and well-documented
+- Explicit configuration injection (no hidden state)
+- Safe for batch jobs, CI/CD, containers, Airflow
+
+---
+
+## 🚀 Quick Start (5 Minutes)
+
+### 1. Install Package
+
+```bash
+# Local development (editable)
+cd /path/to/your-project
+pip install -e /path/to/data-connectors
+
+# Production (git SSH)
+pip install git+ssh://git@github.com/nagarjunr/data-connectors-for-ai-agents.git@v0.1.0
+
+# Install specific connectors only
+pip install -e /path/to/data-connectors[postgres,s3]
+```
+
+### 2. Create Connector Module
+
+**File:** `src/connectors/__init__.py`
+
+```python
+"""Centralized connector initialization (singleton pattern)."""
+from typing import Optional
+import os
+from dotenv import load_dotenv
+
+# Import connectors you need
+from modules.postgres import PostgresClient
+from modules.s3_bucket import S3Client
+from modules.gcp_bucket import GCPBucketClient
+from modules.sharepoint import SharePointClient
+from modules.onedrive import OneDriveClient
+
+load_dotenv()
+
+# Singleton instances
+_pg_client: Optional[PostgresClient] = None
+_s3_client: Optional[S3Client] = None
+_gcp_client: Optional[GCPBucketClient] = None
+_sharepoint_client: Optional[SharePointClient] = None
+_onedrive_client: Optional[OneDriveClient] = None
+
+
+def get_postgres_client() -> PostgresClient:
+    """Get PostgreSQL client."""
+    global _pg_client
+    if _pg_client is None:
+        _pg_client = PostgresClient(
+            host=os.getenv("POSTGRES_HOST", "localhost"),
+            port=int(os.getenv("POSTGRES_PORT", "5432")),
+            database=os.getenv("POSTGRES_DATABASE", "your_database"),
+            user=os.getenv("POSTGRES_USER"),
+            password=os.getenv("POSTGRES_PASSWORD"),
+        )
+    return _pg_client
+
+
+def get_s3_client() -> S3Client:
+    """Get S3-compatible client."""
+    global _s3_client
+    if _s3_client is None:
+        _s3_client = S3Client(
+            endpoint_url=os.getenv("S3_ENDPOINT"),
+            access_key=os.getenv("AWS_ACCESS_KEY_ID"),
+            secret_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
+            region=os.getenv("AWS_DEFAULT_REGION", "us-east-1"),
+        )
+    return _s3_client
+
+
+def get_gcp_client() -> GCPBucketClient:
+    """Get GCP Storage client."""
+    global _gcp_client
+    if _gcp_client is None:
+        _gcp_client = GCPBucketClient(
+            bucket_name=os.getenv("GCP_BUCKET_NAME"),
+        )
+    return _gcp_client
+
+
+def get_sharepoint_client() -> SharePointClient:
+    """Get SharePoint client."""
+    global _sharepoint_client
+    if _sharepoint_client is None:
+        _sharepoint_client = SharePointClient(
+            site_id=os.getenv("SITE_ID"),
+            client_id=os.getenv("CLIENT_ID"),
+            client_secret=os.getenv("CLIENT_SECRET"),
+            tenant_id=os.getenv("TENANT_ID"),
+        )
+    return _sharepoint_client
+
+
+def get_onedrive_client() -> OneDriveClient:
+    """Get OneDrive client."""
+    global _onedrive_client
+    if _onedrive_client is None:
+        _onedrive_client = OneDriveClient(
+            client_id=os.getenv("CLIENT_ID"),
+            client_secret=os.getenv("CLIENT_SECRET"),
+            tenant_id=os.getenv("TENANT_ID"),
+            directory=os.getenv("ONEDRIVE_DIRECTORY"),  # Optional
+        )
+    return _onedrive_client
+
+
+def close_all_connections():
+    """Close all connections on shutdown."""
+    global _pg_client, _s3_client, _gcp_client, _sharepoint_client, _onedrive_client
+    for client in [_pg_client, _s3_client, _gcp_client, _sharepoint_client, _onedrive_client]:
+        if client:
+            client.close()
+```
+
+### 3. Add Environment Variables
+
+**File:** `.env`
+
+```bash
+# PostgreSQL
+POSTGRES_HOST=localhost
+POSTGRES_PORT=5432
+POSTGRES_DATABASE=your_database
+POSTGRES_USER=your_user
+POSTGRES_PASSWORD=your_password
+
+# S3-Compatible Object Storage
+AWS_ACCESS_KEY_ID=your_access_key
+AWS_SECRET_ACCESS_KEY=your_secret_key
+S3_ENDPOINT=https://your-s3-endpoint.example.com:9021
+S3_NAMESPACE=your-namespace  # Optional, only needed for some S3-compatible providers
+AWS_DEFAULT_REGION=us-east-1
+
+# GCP Storage
+GCP_BUCKET_NAME=your-gcp-bucket-name
+
+# Azure AD / Microsoft Graph (for SharePoint & OneDrive)
+CLIENT_ID=your_client_id
+CLIENT_SECRET=your_client_secret
+TENANT_ID=your-tenant-id
+
+# SharePoint Specific
+SITE_ID=your-site-id
+
+# OneDrive Specific
+ONEDRIVE_DIRECTORY=Documents  # Optional: default directory
+```
+
+### 4. Use in Your Code
+
+```python
+from src.connectors import (
+    get_postgres_client,
+    get_s3_client,
+    get_gcp_client,
+    get_sharepoint_client,
+    get_onedrive_client,
+    close_all_connections,
+)
+
+# PostgreSQL
+pg = get_postgres_client()
+results = pg.query("SELECT * FROM users LIMIT 10")
+
+# S3-compatible
+s3 = get_s3_client()
+s3.upload_file("local.txt", "bucket-name", "remote/path/file.txt")
+files = s3.list_objects("bucket-name", prefix="documents/")
+
+# GCP Storage
+gcp = get_gcp_client()
+gcp.upload("/tmp/report.pdf", "reports/monthly.pdf")
+
+# SharePoint
+sp = get_sharepoint_client()
+sp.download("Shared Documents/file.pdf", "/tmp/file.pdf")
+
+# OneDrive
+od = get_onedrive_client()
+od.list("Documents")
+
+# Cleanup on shutdown (FastAPI example)
+# @app.on_event("shutdown")
+# async def shutdown():
+#     close_all_connections()
+```
+
+---
+
+## 📦 Available Connectors
+
+| Connector | Purpose | Environment Variables | Documentation |
+|-----------|---------|----------------------|---------------|
+| **PostgreSQL** | Database connectivity with SQLAlchemy ORM | `POSTGRES_HOST`, `POSTGRES_PORT`, `POSTGRES_DATABASE`, `POSTGRES_USER`, `POSTGRES_PASSWORD` | [README](modules/postgres/README.md) |
+| **S3 Bucket** | S3-compatible object storage (MinIO, AWS, etc.) | `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION` | [README](modules/s3_bucket/README.md) |
+| **GCP Storage** | Google Cloud Storage | `GCP_BUCKET_NAME` | [README](modules/gcp_bucket/README.md) |
+| **SharePoint** | SharePoint document libraries via Microsoft Graph | `CLIENT_ID`, `CLIENT_SECRET`, `TENANT_ID`, `SITE_ID` | [README](modules/sharepoint/README.md) |
+| **OneDrive** | OneDrive file storage via Microsoft Graph | `CLIENT_ID`, `CLIENT_SECRET`, `TENANT_ID`, `SITE_ID`, `ONEDRIVE_DIRECTORY` | [README](modules/onedrive/README.md) |
+| **RabbitMQ** | Send/receive with a durable quorum queue, auto-reconnect | `CONNECTION_STRING` or `RABBITMQ_HOST`, `RABBITMQ_PORT`, `RABBITMQ_USERNAME`, `RABBITMQ_PASSWORD` | [README](modules/rabbitmq/README.md) |
+
+---
+
+## 📚 Documentation
+
+### For Users
+- **[Deployment Guide](docs/DEPLOYMENT.md)** - Complete deployment guide from local dev to OpenShift/Kubernetes
+
+### For Contributors
+- **[Development Guide](docs/DEVELOPMENT.md)** - Development setup, code quality standards, and contributing guidelines
+
+### Quick Development Commands
+
+```bash
+# Format code
+make format
+
+# Lint and auto-fix
+make lint-fix
+
+# Type check
+make typecheck
+
+# Run all checks
+make check
+```
+
+---
+
+## 🎯 Common Use Cases
+
+### Use Case 1: Load Documents from S3
+
+```python
+from src.connectors import get_s3_client
+from pathlib import Path
+
+def load_documents():
+    s3 = get_s3_client()
+    bucket = os.getenv("S3_BUCKET_NAME")
+    files = s3.list_objects(bucket, prefix="documents/")
+
+    documents = []
+    for file_key in files:
+        local_path = f"/tmp/{Path(file_key).name}"
+        s3.download_file(bucket, file_key, local_path)
+        with open(local_path, 'r') as f:
+            documents.append(f.read())
+
+    return documents
+```
+
+### Use Case 2: Save Results to PostgreSQL
+
+```python
+from src.connectors import get_postgres_client
+
+def save_result(data: dict):
+    pg = get_postgres_client()
+
+    query = """
+        INSERT INTO results (category, answer, status)
+        VALUES (%s, %s, %s)
+        RETURNING id
+    """
+    result_id = pg.execute(query, (data['category'], data['answer'], data['status']))
+    return result_id
+```
+
+### Use Case 3: Backup to GCP
+
+```python
+from src.connectors import get_gcp_client
+from pathlib import Path
+
+def backup_files(source_dir: str):
+    gcp = get_gcp_client()
+
+    for file_path in Path(source_dir).rglob("*"):
+        if file_path.is_file():
+            gcp_path = f"backups/{file_path.name}"
+            gcp.upload(str(file_path), gcp_path)
+```
+
+---
+
+## 🏗️ Repository Structure
+
+```
+data-connectors/
+├── README.md              # This file - Quick start and overview
+├── docs/
+│   ├── INTEGRATION.md     # Integration guide for using connectors
+│   ├── DEPLOYMENT.md      # Complete deployment guide (local to OpenShift)
+│   └── DEVELOPMENT.md     # Development setup and guidelines
+├── pyproject.toml         # Python project configuration
+├── requirements-dev.txt   # Development dependencies
+├── Makefile               # Development commands (format, lint, typecheck)
+├── examples/              # Runnable reference examples
+│   ├── s3_bucket/
+│   ├── sharepoint/
+│   ├── onedrive/
+│   ├── gcp_bucket/
+│   └── postgres/
+└── modules/
+    ├── _common/           # Shared base classes and auth helpers
+    ├── s3_bucket/           # S3-compatible connector
+    ├── sharepoint/       # SharePoint connector
+    ├── onedrive/         # OneDrive connector
+    ├── gcp_bucket/       # GCP bucket connector
+    └── postgres/         # PostgreSQL connector
+```
+
+---
+
+## 🔐 OpenShift Quick Setup
+
+**Note:** Deploy keys may be disabled depending on your Git host policy. Use Personal Access Tokens instead.
+
+### Step 1: Create GitHub Personal Access Token
+
+1. Go to: [GitHub Settings - Personal Access Tokens](https://github.com/settings/tokens)
+2. Click **"Generate new token (classic)"**
+3. Select `repo` scope (Full control of private repositories)
+4. Set expiration (90 days recommended)
+5. **Copy token immediately** - you won't see it again
+
+### Step 2: Create Secret in OpenShift
+
+```bash
+oc login --server=https://api.your-openshift-cluster.example.com:6443 --token=<your-token>
+oc project your-project
+
+oc create secret generic data-connectors-github-pat \
+  --from-literal=username=<your-github-username> \
+  --from-literal=password=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
+  --type=kubernetes.io/basic-auth
+
+oc annotate secret data-connectors-github-pat \
+  'build.openshift.io/source-secret-match-uri-1=https://github.com/*'
+```
+
+For complete OpenShift deployment instructions, see [Deployment Guide](docs/DEPLOYMENT.md#production-deployment-openshift).
+
+---
+
+## 🐛 Troubleshooting
+
+### Import Error
+```bash
+# Problem: ModuleNotFoundError: No module named 'modules'
+# Solution:
+pip install -e /path/to/data-connectors
+pip list | grep data-connectors
+```
+
+### Connection Error
+```bash
+# Problem: Connection refused to PostgreSQL/S3
+# Solution:
+# 1. Check .env variables
+# 2. Verify services are running
+# 3. Test connection:
+psql -h localhost -U user -d db_name
+```
+
+### SSL Certificate Error
+```bash
+# Problem: SSLError: certificate verify failed
+# Solution:
+export SSL_CERT_FILE=/path/to/ca-bundle.pem
+export REQUESTS_CA_BUNDLE=$SSL_CERT_FILE
+```
+
+For more troubleshooting help, see the [Deployment Guide](docs/DEPLOYMENT.md#troubleshooting).
+
+---
+
+## 🎓 Design Principles
+
+- ✅ **Explicit configuration**: All settings injected via constructor parameters
+- ✅ **No hidden state**: No environment variable loading inside modules
+- ✅ **Deterministic behavior**: Same input = same output, always
+- ✅ **Easy testing**: Pure functions, dependency injection, no side effects
+- ✅ **Production-ready**: Designed for batch jobs, CI/CD, containers, Airflow
+- ✅ **Type-safe**: Full type hints for better IDE support and error detection
+
+---
+
+## 🆘 Getting Help
+
+- **Issues:** [GitHub Issues](https://github.com/nagarjunr/data-connectors-for-ai-agents/issues)
+- **Integration Guide:** [docs/INTEGRATION.md](docs/INTEGRATION.md)
+- **Deployment Guide:** [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md)
+- **Development Guide:** [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md)
+
+---
+
+## 📋 Checklist for New Projects
+
+- [ ] Install data-connectors package
+- [ ] Create `src/connectors/__init__.py`
+- [ ] Add environment variables to `.env`
+- [ ] Test connections locally
+- [ ] Add to `requirements.txt` / `requirements.prod.txt`
+- [ ] For OpenShift: Create PAT and secrets
+- [ ] Test deployment in staging
+- [ ] Deploy to production
+
+---
+
+## License
+
+MIT — see [LICENSE](LICENSE).