rail-engine-ingest 0.1.2__tar.gz

rail_engine_ingest-0.1.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Railtown AI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

rail_engine_ingest-0.1.2/MANIFEST.in

@@ -0,0 +1,7 @@
+# Include custom README files for both packages
+include README-INGEST.md
+include README-RETRIEVAL.md
+# Include standard README.md (for backward compatibility)
+include README.md
+# Include LICENSE
+include LICENSE

rail_engine_ingest-0.1.2/PKG-INFO

@@ -0,0 +1,112 @@
+Metadata-Version: 2.4
+Name: rail-engine-ingest
+Version: 0.1.2
+Summary: Python SDK for Railtown AI Rail Engine - Ingestion
+Author-email: Railtown AI <support@railtown.ai>
+License: MIT
+Project-URL: Homepage, https://github.com/railtownai/railengine-sdk
+Project-URL: Documentation, https://github.com/railtownai/railengine-sdk
+Project-URL: Repository, https://github.com/railtownai/railengine-sdk
+Keywords: railtown,rail-engine,ingestion,sdk
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: httpx>=0.24.0
+Requires-Dist: pydantic>=1.10.0
+Dynamic: license-file
+
+# Rail Engine Ingestion SDK
+
+Python SDK for ingesting data into Railtown AI Rail Engine - handles embeddings, storage, indexing, and webhook publishing.
+
+## Overview
+
+The `rail-engine-ingest` package provides a Pythonic interface for publishing data to Rail Engine. It supports async/await patterns, Pydantic model validation, and familiar configuration patterns.
+
+## Installation
+
+```bash
+uv pip install rail-engine-ingest
+```
+
+## Quick Start
+
+```python
+
+# Copy your ENGINE_TOKEN from https://cndr.railtown.ai Project Settings
+
+import asyncio
+from railtown.engine.ingest import RailEngineIngest
+import base64
+import json
+
+engine_token = os.getenv("ENGINE_TOKEN")
+
+async def main():
+    # Initialize client
+    async with RailEngineIngest(engine_token=engine_token) as client:
+        # Ingest data
+        data = {
+            "name": "Chicken Shawarma",
+            "meal": "lunch",
+            "calories": 95
+        }
+        response = await client.upsert(data)
+        print(f"Status: {response.status_code}")
+
+asyncio.run(main())
+```
+
+## Configuration
+
+### Environment Variable
+
+- `ENGINE_TOKEN` - Base64-encoded JSON string containing ingestion credentials
+
+### Constructor Parameters
+
+- `engine_token` (optional) - ENGINE_TOKEN string (if not provided, reads from env)
+- `model` (optional) - Pydantic model type for validating ingested data
+
+## Features
+
+- **Single `upsert()` method** - Unified interface for ingesting data
+- **Multiple data formats** - Accepts Pydantic models, dictionaries, or JSON strings
+- **Model validation** - Optional Pydantic model validation
+- **Direct JSON payload** - Sends data directly as JSON to the ingestion endpoint
+- **Async/await support** - Built for modern async Python applications
+
+## Error Handling
+
+The SDK provides custom exception classes:
+
+- `RailtownError` - Base exception
+- `RailtownBadRequestError` - 400 Bad Request
+- `RailtownUnauthorizedError` - 401 Unauthorized
+- `RailtownNotFoundError` - 404 Not Found
+- `RailtownConflictError` - 409 Conflict
+- `RailtownServerError` - 5xx Server errors
+
+Exceptions are raised on errors.
+
+## Requirements
+
+- uv
+- Python 3.10+
+- httpx >= 0.24.0
+- pydantic >= 1.10.0
+
+## Related Package
+
+For retrieving and searching data from Rail Engine, see [`rail-engine-ingest`](https://pypi.org/project/rail-engine-ingest/).
+
+## License
+
+MIT

rail_engine_ingest-0.1.2/README-INGEST.md

@@ -0,0 +1,88 @@
+# Rail Engine Ingestion SDK
+
+Python SDK for ingesting data into Railtown AI Rail Engine - handles embeddings, storage, indexing, and webhook publishing.
+
+## Overview
+
+The `rail-engine-ingest` package provides a Pythonic interface for publishing data to Rail Engine. It supports async/await patterns, Pydantic model validation, and familiar configuration patterns.
+
+## Installation
+
+```bash
+uv pip install rail-engine-ingest
+```
+
+## Quick Start
+
+```python
+
+# Copy your ENGINE_TOKEN from https://cndr.railtown.ai Project Settings
+
+import asyncio
+from railtown.engine.ingest import RailEngineIngest
+import base64
+import json
+
+engine_token = os.getenv("ENGINE_TOKEN")
+
+async def main():
+    # Initialize client
+    async with RailEngineIngest(engine_token=engine_token) as client:
+        # Ingest data
+        data = {
+            "name": "Chicken Shawarma",
+            "meal": "lunch",
+            "calories": 95
+        }
+        response = await client.upsert(data)
+        print(f"Status: {response.status_code}")
+
+asyncio.run(main())
+```
+
+## Configuration
+
+### Environment Variable
+
+- `ENGINE_TOKEN` - Base64-encoded JSON string containing ingestion credentials
+
+### Constructor Parameters
+
+- `engine_token` (optional) - ENGINE_TOKEN string (if not provided, reads from env)
+- `model` (optional) - Pydantic model type for validating ingested data
+
+## Features
+
+- **Single `upsert()` method** - Unified interface for ingesting data
+- **Multiple data formats** - Accepts Pydantic models, dictionaries, or JSON strings
+- **Model validation** - Optional Pydantic model validation
+- **Direct JSON payload** - Sends data directly as JSON to the ingestion endpoint
+- **Async/await support** - Built for modern async Python applications
+
+## Error Handling
+
+The SDK provides custom exception classes:
+
+- `RailtownError` - Base exception
+- `RailtownBadRequestError` - 400 Bad Request
+- `RailtownUnauthorizedError` - 401 Unauthorized
+- `RailtownNotFoundError` - 404 Not Found
+- `RailtownConflictError` - 409 Conflict
+- `RailtownServerError` - 5xx Server errors
+
+Exceptions are raised on errors.
+
+## Requirements
+
+- uv
+- Python 3.10+
+- httpx >= 0.24.0
+- pydantic >= 1.10.0
+
+## Related Package
+
+For retrieving and searching data from Rail Engine, see [`rail-engine-ingest`](https://pypi.org/project/rail-engine-ingest/).
+
+## License
+
+MIT

rail_engine_ingest-0.1.2/README-RETRIEVAL.md

@@ -0,0 +1,90 @@
+# Rail Engine Retrieval SDK
+
+Python SDK for retrieving and searching data from Railtown AI Rail Engine - handles embeddings, storage documents, and indexed content.
+
+## Overview
+
+The `rail-engine` package provides a Pythonic interface for retrieving and searching data from Rail Engine. It supports async/await patterns, client-side filtering, automatic pagination, and Pydantic model deserialization.
+
+## Installation
+
+```bash
+uv pip install rail-engine
+```
+
+## Quick Start
+
+```python
+import asyncio
+from railtown.engine import RailEngine
+
+async def main():
+    # Initialize client (reads from ENGINE_PAT and ENGINE_ID env vars)
+    async with RailEngine() as client:
+        # Search vector store
+        results = client.search_vector_store(
+            query="apple"
+        )
+        async for item in results:
+            print(item)
+
+asyncio.run(main())
+```
+
+## Configuration
+
+### Environment Variables
+
+- `ENGINE_PAT` (required) - Personal Access Token
+- `ENGINE_ID` (required) - Engine ID (can also be passed to constructor)
+- `RAILTOWN_API_URL` (optional) - Base API URL (defaults to `https://cndr.railtown.ai/api`)
+
+### Constructor Parameters
+
+- `pat` (optional) - PAT token (if not provided, reads from `ENGINE_PAT` env)
+- `engine_id` (optional) - Engine ID (if not provided, reads from `ENGINE_ID` env, required if not in env)
+- `api_url` (optional) - Base API URL (if not provided, reads from `RAILTOWN_API_URL` env or defaults to production)
+- `model` (optional) - Pydantic model type for deserializing retrieved data
+
+## Features
+
+- **Multiple retrieval methods**:
+  - `search_vector_store()` - Semantic search in vector stores
+  - `get_storage_document_by_id()` - Get document by ID
+  - `get_storage_document_by_customer_key()` - Get documents by customer key
+  - `query_storage_by_jsonpath()` - Query using JSONPath
+  - `list_storage_documents()` - List all documents with pagination
+  - `search_index()` - Full-text search using Azure Search
+- **Client-side filtering** - Filter results using `filter_fn` parameter
+- **Automatic pagination** - Handles pagination automatically
+- **Model deserialization** - Optional Pydantic model support with per-call override
+- **Graceful error handling** - Returns None or empty iterables on errors
+- **Async/await support** - Built for modern async Python applications
+
+## Error Handling
+
+The SDK provides custom exception classes:
+
+- `RailtownError` - Base exception
+- `RailtownBadRequestError` - 400 Bad Request
+- `RailtownUnauthorizedError` - 401 Unauthorized
+- `RailtownNotFoundError` - 404 Not Found
+- `RailtownConflictError` - 409 Conflict
+- `RailtownServerError` - 5xx Server errors
+
+Returns `None` or empty iterables on errors (graceful degradation).
+
+## Requirements
+
+- uv
+- Python 3.10+
+- httpx >= 0.24.0
+- pydantic >= 1.10.0
+
+## Related Package
+
+For ingesting data into Rail Engine, see [`rail-engine`](https://pypi.org/project/rail-engine/).
+
+## License
+
+MIT

rail_engine_ingest-0.1.2/README.md

@@ -0,0 +1,233 @@
+# Rail Engine Python SDK
+
+Python SDK for Railtown AI Rail Engine - providing a Pythonic interface for ingesting and retrieving data from Rail Engine.
+
+## Overview
+
+The Rail Engine Python SDK is split into two separate packages:
+
+- **`rail-engine-ingest`** - For publishing data to Rail Engine (handles embeddings, storage, indexing, and webhook publishing)
+- **`rail-engine`** - For retrieving and searching data (embeddings, storage documents, and indexed content)
+
+Both packages support:
+
+- Async/await patterns
+- Client-side filtering
+- Automatic pagination
+- Pydantic model support for type safety
+- Familiar configuration patterns (like OpenAI SDK)
+
+## Installation
+
+### Ingestion Package
+
+```bash
+uv pip install rail-engine-ingest
+```
+
+### Retrieval Package
+
+```bash
+uv pip install rail-engine
+```
+
+### Both Packages
+
+```bash
+uv pip install rail-engine-ingest rail-engine
+```
+
+## Quick Start
+
+### Ingestion
+
+```python
+import asyncio
+from railtown.engine.ingest import RailEngineIngest
+import base64
+import json
+
+# Create ENGINE_TOKEN (base64-encoded JSON)
+token_data = {
+    "IngestionUrl": "https://eng123.railtownlogs.com",
+    "IngestionApiToken": "your-auth-token",
+    "EngineId": "your-engine-guid"
+}
+engine_token = base64.b64encode(json.dumps(token_data).encode()).decode()
+
+async def main():
+    # Initialize client
+    async with RailEngineIngest(engine_token=engine_token) as client:
+        # Ingest data
+        data = {
+            "EventId": "event-123",
+            "ProjectId": "project-456",
+            "food_name": "Apple",
+            "calories": 95
+        }
+        response = await client.upsert(data)
+        print(f"Status: {response.status_code}")
+
+asyncio.run(main())
+```
+
+### Retrieval
+
+```python
+import asyncio
+from railtown.engine import RailEngine
+
+async def main():
+    # Initialize client (reads from ENGINE_PAT and ENGINE_ID env vars)
+    async with RailEngine() as client:
+        # Search vector store
+        results = client.search_vector_store(
+            engine_id=client.engine_id,
+            vector_store="VectorStore1",
+            query="apple"
+        )
+        async for item in results:
+            print(item)
+
+asyncio.run(main())
+```
+
+## Configuration
+
+### Ingestion (`rail-engine-ingest`)
+
+**Environment Variable:**
+
+- `ENGINE_TOKEN` - Base64-encoded JSON string containing ingestion credentials
+
+**Constructor Parameters:**
+
+- `engine_token` (optional) - ENGINE_TOKEN string (if not provided, reads from env)
+- `model` (optional) - Pydantic model type for validating ingested data
+
+### Retrieval (`rail-engine`)
+
+**Environment Variables:**
+
+- `ENGINE_PAT` (required) - Personal Access Token
+- `ENGINE_ID` (required) - Engine ID (can also be passed to constructor)
+- `RAILTOWN_API_URL` (optional) - Base API URL (defaults to `https://cndr.railtown.ai/api`)
+
+**Constructor Parameters:**
+
+- `pat` (optional) - PAT token (if not provided, reads from `ENGINE_PAT` env)
+- `engine_id` (optional) - Engine ID (if not provided, reads from `ENGINE_ID` env, required if not in env)
+- `api_url` (optional) - Base API URL (if not provided, reads from `RAILTOWN_API_URL` env or defaults to production)
+- `model` (optional) - Pydantic model type for deserializing retrieved data
+
+## Features
+
+### Ingestion Features
+
+- **Single `upsert()` method** - Unified interface for ingesting data
+- **Multiple data formats** - Accepts Pydantic models, dictionaries, or JSON strings
+- **Model validation** - Optional Pydantic model validation
+- **Direct JSON payload** - Sends data directly as JSON to the ingestion endpoint
+- **UTF-8 encoding** - All text operations use UTF-8
+
+### Retrieval Features
+
+- **Multiple retrieval methods**:
+  - `search_vector_store()` - Semantic search in vector stores
+  - `get_storage_document_by_id()` - Get document by ID
+  - `get_storage_document_by_customer_key()` - Get documents by customer key
+  - `query_storage_by_jsonpath()` - Query using JSONPath
+  - `list_storage_documents()` - List all documents with pagination
+  - `search_index()` - Full-text search using Azure Search
+- **Client-side filtering** - Filter results using `filter_fn` parameter
+- **Automatic pagination** - Handles pagination automatically
+- **Model deserialization** - Optional Pydantic model support with per-call override
+- **Graceful error handling** - Returns None or empty iterables on errors
+
+## Examples
+
+See the [`samples/`](samples/) directory for comprehensive examples:
+
+- [`samples/ingestion_example.py`](samples/ingestion_example.py) - Ingestion examples
+- [`samples/retrieval_example.py`](samples/retrieval_example.py) - Retrieval examples
+- [`samples/README.md`](samples/README.md) - Samples documentation
+
+## API Reference
+
+### Ingestion Client (`RailEngineIngest`)
+
+#### Methods
+
+- `upsert(data)` - Upsert data to Rail Engine
+  - Accepts: Pydantic model instance, dict, or JSON string
+  - Returns: HTTP response
+
+#### Properties
+
+- `ingestion_url` - Ingestion URL from decoded ENGINE_TOKEN
+- `ingestion_api_token` - API token from decoded ENGINE_TOKEN
+- `engine_id` - Engine ID from decoded ENGINE_TOKEN
+
+### Retrieval Client (`RailEngine`)
+
+#### Methods
+
+- `search_vector_store(engine_id, vector_store, query, filter_fn=None, model=None)` - Search vector store
+- `get_storage_document_by_id(engine_id, engine_document_id, filter_fn=None, model=None)` - Get document by ID
+- `get_storage_document_by_customer_key(engine_id, customer_key, page_number=1, page_size=25, filter_fn=None, model=None)` - Get documents by customer key
+- `query_storage_by_jsonpath(engine_id, json_path_query, filter_fn=None, model=None)` - Query by JSONPath
+- `list_storage_documents(engine_id, customer_key=None, page_number=1, page_size=100, filter_fn=None, model=None)` - List documents
+- `search_index(project_id, engine_id, query, filter_fn=None, model=None)` - Search index
+
+#### Properties
+
+- `pat` - PAT token
+- `engine_id` - Engine ID
+- `api_url` - Base API URL
+- `model` - Default model type
+
+## Error Handling
+
+The SDK provides custom exception classes:
+
+- `RailtownError` - Base exception
+- `RailtownBadRequestError` - 400 Bad Request
+- `RailtownUnauthorizedError` - 401 Unauthorized
+- `RailtownNotFoundError` - 404 Not Found
+- `RailtownConflictError` - 409 Conflict
+- `RailtownServerError` - 5xx Server errors
+
+**Ingestion**: Raises exceptions on errors
+**Retrieval**: Returns `None` or empty iterables on errors (graceful degradation)
+
+## Requirements
+
+- Python 3.10+
+- httpx >= 0.24.0
+- pydantic >= 1.10.0
+
+## Testing
+
+Run the test suite:
+
+```bash
+# Install development dependencies
+uv pip install -r requirements-dev.txt
+
+# Run all tests
+pytest
+
+# Run with coverage
+pytest --cov=railtown --cov-report=html
+
+# Run specific test file
+pytest tests/test_ingest_client.py
+```
+
+## License
+
+MIT
+
+## Support
+
+For issues, questions, or contributions, please visit the [GitHub repository](https://github.com/railtownai/railengine-sdk).

rail_engine_ingest-0.1.2/pyproject.toml

@@ -0,0 +1,40 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "rail-engine-ingest"
+version = "0.1.2"
+description = "Python SDK for Railtown AI Rail Engine - Ingestion"
+readme = "README-INGEST.md"
+requires-python = ">=3.10"
+authors = [
+    {name = "Railtown AI", email = "support@railtown.ai"}
+]
+license = {text = "MIT"}
+keywords = ["railtown", "rail-engine", "ingestion", "sdk"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+]
+dependencies = [
+    "httpx>=0.24.0",
+    "pydantic>=1.10.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/railtownai/railengine-sdk"
+Documentation = "https://github.com/railtownai/railengine-sdk"
+Repository = "https://github.com/railtownai/railengine-sdk"
+
+[tool.setuptools.packages.find]
+include = ["railtown.engine.ingest*", "railtown.engine.exceptions*"]
+namespaces = false
+
+[tool.setuptools.package-data]
+"*" = ["py.typed"]

rail_engine_ingest-0.1.2/rail_engine_ingest.egg-info/PKG-INFO

@@ -0,0 +1,112 @@
+Metadata-Version: 2.4
+Name: rail-engine-ingest
+Version: 0.1.2
+Summary: Python SDK for Railtown AI Rail Engine - Ingestion
+Author-email: Railtown AI <support@railtown.ai>
+License: MIT
+Project-URL: Homepage, https://github.com/railtownai/railengine-sdk
+Project-URL: Documentation, https://github.com/railtownai/railengine-sdk
+Project-URL: Repository, https://github.com/railtownai/railengine-sdk
+Keywords: railtown,rail-engine,ingestion,sdk
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: httpx>=0.24.0
+Requires-Dist: pydantic>=1.10.0
+Dynamic: license-file
+
+# Rail Engine Ingestion SDK
+
+Python SDK for ingesting data into Railtown AI Rail Engine - handles embeddings, storage, indexing, and webhook publishing.
+
+## Overview
+
+The `rail-engine-ingest` package provides a Pythonic interface for publishing data to Rail Engine. It supports async/await patterns, Pydantic model validation, and familiar configuration patterns.
+
+## Installation
+
+```bash
+uv pip install rail-engine-ingest
+```
+
+## Quick Start
+
+```python
+
+# Copy your ENGINE_TOKEN from https://cndr.railtown.ai Project Settings
+
+import asyncio
+from railtown.engine.ingest import RailEngineIngest
+import base64
+import json
+
+engine_token = os.getenv("ENGINE_TOKEN")
+
+async def main():
+    # Initialize client
+    async with RailEngineIngest(engine_token=engine_token) as client:
+        # Ingest data
+        data = {
+            "name": "Chicken Shawarma",
+            "meal": "lunch",
+            "calories": 95
+        }
+        response = await client.upsert(data)
+        print(f"Status: {response.status_code}")
+
+asyncio.run(main())
+```
+
+## Configuration
+
+### Environment Variable
+
+- `ENGINE_TOKEN` - Base64-encoded JSON string containing ingestion credentials
+
+### Constructor Parameters
+
+- `engine_token` (optional) - ENGINE_TOKEN string (if not provided, reads from env)
+- `model` (optional) - Pydantic model type for validating ingested data
+
+## Features
+
+- **Single `upsert()` method** - Unified interface for ingesting data
+- **Multiple data formats** - Accepts Pydantic models, dictionaries, or JSON strings
+- **Model validation** - Optional Pydantic model validation
+- **Direct JSON payload** - Sends data directly as JSON to the ingestion endpoint
+- **Async/await support** - Built for modern async Python applications
+
+## Error Handling
+
+The SDK provides custom exception classes:
+
+- `RailtownError` - Base exception
+- `RailtownBadRequestError` - 400 Bad Request
+- `RailtownUnauthorizedError` - 401 Unauthorized
+- `RailtownNotFoundError` - 404 Not Found
+- `RailtownConflictError` - 409 Conflict
+- `RailtownServerError` - 5xx Server errors
+
+Exceptions are raised on errors.
+
+## Requirements
+
+- uv
+- Python 3.10+
+- httpx >= 0.24.0
+- pydantic >= 1.10.0
+
+## Related Package
+
+For retrieving and searching data from Rail Engine, see [`rail-engine-ingest`](https://pypi.org/project/rail-engine-ingest/).
+
+## License
+
+MIT