airbyte-internal-ops 0.1.5__tar.gz → 0.1.7__tar.gz

airbyte_internal_ops-0.1.7/CONTRIBUTING.md

@@ -0,0 +1,110 @@
+# Contributing
+
+First clone the repo, then use something like the following MCP config.
+
+```json
+{
+  "mcpServers": {
+    "airbyte-ops-mcp": {
+      "type": "stdio",
+      "command": "uv",
+      "args": [
+        "run",
+        "--project=/Users/{my-user-id}/repos/airbyte-ops-mcp/",
+        "airbyte-ops-mcp"
+      ],
+      "env": {
+        "AIRBYTE_MCP_ENV_FILE": "/path/to/airbyte-ops-mcp/.env"
+      }
+    },
+    "airbyte-coral-mcp": {
+      "type": "stdio",
+      "command": "uvx",
+      "args": [
+        "--python=3.11",
+        "--from=airbyte@latest",
+        "airbyte-mcp"
+      ],
+      "env": {
+        "AIRBYTE_MCP_ENV_FILE": "/Users/{user-id}/.mcp/airbyte_mcp.env"
+      }
+    }
+  }
+}
+```
+
+## Internal Secrets for Live Tests
+
+The live tests feature can retrieve unmasked connection secrets from Airbyte Cloud's internal database. This requires:
+
+- **GCP_PROD_DB_ACCESS_CREDENTIALS** - Access to prod Cloud SQL and Google Secret Manager for DB connection details
+
+To test locally:
+
+1. Set up GCP Application Default Credentials: `gcloud auth application-default login`
+2. Ensure you have access to the `prod-ab-cloud-proj` project
+3. Connect to Tailscale (required for private network access)
+
+In CI, these secrets are available at the org level and a Cloud SQL Auth Proxy handles connectivity.
+
+## Authorizing Service Accounts
+
+To grant a service account access to the prod database query tools, you need to grant permissions on the `CONNECTION_RETRIEVER_PG_CONNECTION_DETAILS` secret in Google Secret Manager (see `CONNECTION_RETRIEVER_PG_CONNECTION_DETAILS_SECRET_ID` in `src/airbyte_ops_mcp/constants.py`). The service account needs both `secretAccessor` (to read the secret value) and `viewer` (to list secret versions) roles.
+
+### Required Permissions
+
+#### Secret Manager Permissions
+
+Grant both roles scoped to the specific secret:
+
+```bash
+# Grant secretAccessor role (for reading secret values)
+gcloud secrets add-iam-policy-binding projects/587336813068/secrets/CONNECTION_RETRIEVER_PG_CONNECTION_DETAILS \
+  --member="serviceAccount:YOUR_SERVICE_ACCOUNT@prod-ab-cloud-proj.iam.gserviceaccount.com" \
+  --role="roles/secretmanager.secretAccessor"
+
+# Grant viewer role (for listing secret versions)
+gcloud secrets add-iam-policy-binding projects/587336813068/secrets/CONNECTION_RETRIEVER_PG_CONNECTION_DETAILS \
+  --member="serviceAccount:YOUR_SERVICE_ACCOUNT@prod-ab-cloud-proj.iam.gserviceaccount.com" \
+  --role="roles/secretmanager.viewer"
+```
+
+#### Cloud SQL Permissions
+
+Grant Cloud SQL access at the project level:
+
+```bash
+# Grant Cloud SQL Client role (required for connecting via Cloud SQL Proxy or Python Connector)
+gcloud projects add-iam-policy-binding prod-ab-cloud-proj \
+  --member="serviceAccount:YOUR_SERVICE_ACCOUNT@prod-ab-cloud-proj.iam.gserviceaccount.com" \
+  --role="roles/cloudsql.client"
+
+# Grant Cloud SQL Instance User role (may be required depending on org policies)
+gcloud projects add-iam-policy-binding prod-ab-cloud-proj \
+  --member="serviceAccount:YOUR_SERVICE_ACCOUNT@prod-ab-cloud-proj.iam.gserviceaccount.com" \
+  --role="roles/cloudsql.instanceUser"
+```
+
+Note: `roles/cloudsql.client` is required for the Cloud SQL Python Connector and Cloud SQL Auth Proxy to establish connections. `roles/cloudsql.instanceUser` is typically needed for IAM database authentication; it may be redundant when using username/password authentication (as this codebase does), but some environments may still require it.
+
+### Verifying Permissions
+
+To check existing permissions on the secret:
+
+```bash
+gcloud secrets get-iam-policy projects/587336813068/secrets/CONNECTION_RETRIEVER_PG_CONNECTION_DETAILS
+```
+
+### Using Cloud SQL Proxy
+
+When running outside of the VPC (without Tailscale), use the Cloud SQL Auth Proxy:
+
+```bash
+# Start the proxy (use a non-conflicting port if 5432 is in use)
+cloud-sql-proxy prod-ab-cloud-proj:us-west3:prod-pgsql-replica --port=5433
+
+# Set environment variables for your application
+export USE_CLOUD_SQL_PROXY=1
+export DB_PORT=5433
+export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json
+```

{airbyte_internal_ops-0.1.5 → airbyte_internal_ops-0.1.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: airbyte-internal-ops
-Version: 0.1.5
+Version: 0.1.7
 Summary: MCP and API interfaces that let the agents do the admin work
 Author-email: Aaron Steers <aj@airbyte.io>
 Keywords: admin,airbyte,api,mcp
@@ -24,6 +24,7 @@ Requires-Dist: fastmcp<3.0,>=2.12.1
 Requires-Dist: gitpython<4.0,>=3.1.29
 Requires-Dist: google-cloud-logging<4.0,>=3.9.0
 Requires-Dist: google-cloud-secret-manager<3.0,>=2.18.0
+Requires-Dist: google-cloud-storage<3.0,>=2.8.0
 Requires-Dist: jinja2<4.0,>=3.1.2
 Requires-Dist: pydantic>=2.0.0
 Requires-Dist: python-dotenv<2.0,>=1.0.0
@@ -95,3 +96,71 @@ AIRBYTE_CLOUD_TEST_WORKSPACE_ID="..."
 Once configured, use the `test_my_tools` prompt by typing "/test" into your agent and selecting the auto-complete option for the `test_my_tools` prompt.
 
 This prompt will step through all the tools, demoing their capabilities.
+
+## Usage Examples
+
+### Testing MCP Tools Locally
+
+Use the `mcp-tool-test` poe task to test tools directly:
+
+```bash
+# List connectors in a repo
+poe mcp-tool-test list_connectors_in_repo '{"repo_path": "/path/to/airbyte"}'
+
+# Get cloud connector version
+poe mcp-tool-test get_cloud_connector_version '{"workspace_id": "...", "actor_id": "..."}'
+```
+
+### Using Cloud SQL Proxy for Database Tools
+
+Some tools (like `list_org_connections_by_source_type_db`) require access to the Airbyte Cloud Prod DB Replica. To test these locally:
+
+1. Authenticate with GCP:
+   ```bash
+   gcloud auth login
+   gcloud auth application-default login
+   ```
+
+2. Start Cloud SQL Proxy:
+   ```bash
+   cloud-sql-proxy prod-ab-cloud-proj:us-west3:prod-pgsql-replica --port=15432
+   ```
+
+3. Run the tool with proxy environment variables:
+   ```bash
+   USE_CLOUD_SQL_PROXY=1 DB_PORT=15432 poe mcp-tool-test list_org_connections_by_source_type_db \
+     '{"organization_id": "...", "connector_canonical_name": "source-youtube-analytics"}'
+   ```
+
+## Instructions for Agents
+
+When working with a user to set up and use this MCP server, agents should follow these steps for authentication:
+
+### GCP Authentication Flow
+
+For tools that require GCP access (database queries, secret manager, etc.), guide the user through authentication:
+
+1. **Check if gcloud is installed.** If not, install it:
+   ```bash
+   curl -sSL https://sdk.cloud.google.com | bash -s -- --disable-prompts
+   ```
+
+2. **Run gcloud auth login with `--no-launch-browser`** to get a verification URL:
+   ```bash
+   gcloud auth login --no-launch-browser
+   ```
+   Send the verification URL to the user and ask them to complete sign-in and provide the verification code.
+
+3. **Set up Application Default Credentials (ADC)** for library access:
+   ```bash
+   gcloud auth application-default login --no-launch-browser
+   ```
+   Again, send the URL to the user and collect the verification code.
+
+4. **For database tools**, start Cloud SQL Proxy on an available port and use the `USE_CLOUD_SQL_PROXY` and `DB_PORT` environment variables.
+
+### Airbyte Cloud Authentication
+
+For tools that interact with Airbyte Cloud API, ensure the user has configured:
+- `AIRBYTE_CLOUD_CLIENT_ID` and `AIRBYTE_CLOUD_CLIENT_SECRET` in their `.env` file
+- For admin operations: `AIRBYTE_INTERNAL_ADMIN_FLAG=airbyte.io` and `AIRBYTE_INTERNAL_ADMIN_USER`

airbyte_internal_ops-0.1.7/README.md

@@ -0,0 +1,127 @@
+# airbyte-ops-mcp
+
+MCP and API interfaces that let the agents do the admin work.
+
+## Installing Ops MCP in your Client
+
+This config example will help you add the MCP server to your client:
+
+```json
+{
+  "mcpServers": {
+    "airbyte-ops-mcp": {
+      "type": "stdio",
+      "command": "uv",
+      "args": [
+        "run",
+        "--project=/Users/aj.steers/repos/airbyte-ops-mcp/",
+        "airbyte-ops-mcp"
+      ],
+      "env": {
+        "AIRBYTE_MCP_ENV_FILE": "/Users/{user-id}/.mcp/airbyte_mcp.env"
+      }
+    },
+    "airbyte-coral-mcp": {
+      "type": "stdio",
+      "command": "uvx",
+      "args": [
+        "--python=3.11",
+        "--from=airbyte@latest",
+        "airbyte-mcp"
+      ],
+      "env": {
+        "AIRBYTE_MCP_ENV_FILE": "/Users/{user-id}/.mcp/airbyte_mcp.env"
+      }
+    }
+  }
+}
+```
+
+Your `.env` file should include the following values:
+
+```ini
+# Creds for Airbyte Cloud OAuth
+AIRBYTE_CLOUD_CLIENT_ID="..."
+AIRBYTE_CLOUD_CLIENT_SECRET="..."
+
+# Required for elevated admin operations
+AIRBYTE_INTERNAL_ADMIN_FLAG=airbyte.io
+AIRBYTE_INTERNAL_ADMIN_USER={my-id}@airbyte.io
+
+# Workspace ID for Testing
+AIRBYTE_CLOUD_TEST_WORKSPACE_ID="..."
+```
+
+## Getting Started
+
+Once configured, use the `test_my_tools` prompt by typing "/test" into your agent and selecting the auto-complete option for the `test_my_tools` prompt.
+
+This prompt will step through all the tools, demoing their capabilities.
+
+## Usage Examples
+
+### Testing MCP Tools Locally
+
+Use the `mcp-tool-test` poe task to test tools directly:
+
+```bash
+# List connectors in a repo
+poe mcp-tool-test list_connectors_in_repo '{"repo_path": "/path/to/airbyte"}'
+
+# Get cloud connector version
+poe mcp-tool-test get_cloud_connector_version '{"workspace_id": "...", "actor_id": "..."}'
+```
+
+### Using Cloud SQL Proxy for Database Tools
+
+Some tools (like `list_org_connections_by_source_type_db`) require access to the Airbyte Cloud Prod DB Replica. To test these locally:
+
+1. Authenticate with GCP:
+   ```bash
+   gcloud auth login
+   gcloud auth application-default login
+   ```
+
+2. Start Cloud SQL Proxy:
+   ```bash
+   cloud-sql-proxy prod-ab-cloud-proj:us-west3:prod-pgsql-replica --port=15432
+   ```
+
+3. Run the tool with proxy environment variables:
+   ```bash
+   USE_CLOUD_SQL_PROXY=1 DB_PORT=15432 poe mcp-tool-test list_org_connections_by_source_type_db \
+     '{"organization_id": "...", "connector_canonical_name": "source-youtube-analytics"}'
+   ```
+
+## Instructions for Agents
+
+When working with a user to set up and use this MCP server, agents should follow these steps for authentication:
+
+### GCP Authentication Flow
+
+For tools that require GCP access (database queries, secret manager, etc.), guide the user through authentication:
+
+1. **Check if gcloud is installed.** If not, install it:
+   ```bash
+   curl -sSL https://sdk.cloud.google.com | bash -s -- --disable-prompts
+   ```
+
+2. **Run gcloud auth login with `--no-launch-browser`** to get a verification URL:
+   ```bash
+   gcloud auth login --no-launch-browser
+   ```
+   Send the verification URL to the user and ask them to complete sign-in and provide the verification code.
+
+3. **Set up Application Default Credentials (ADC)** for library access:
+   ```bash
+   gcloud auth application-default login --no-launch-browser
+   ```
+   Again, send the URL to the user and collect the verification code.
+
+4. **For database tools**, start Cloud SQL Proxy on an available port and use the `USE_CLOUD_SQL_PROXY` and `DB_PORT` environment variables.
+
+### Airbyte Cloud Authentication
+
+For tools that interact with Airbyte Cloud API, ensure the user has configured:
+- `AIRBYTE_CLOUD_CLIENT_ID` and `AIRBYTE_CLOUD_CLIENT_SECRET` in their `.env` file
+- For admin operations: `AIRBYTE_INTERNAL_ADMIN_FLAG=airbyte.io` and `AIRBYTE_INTERNAL_ADMIN_USER`

{airbyte_internal_ops-0.1.5 → airbyte_internal_ops-0.1.7}/pyproject.toml

@@ -14,12 +14,14 @@ dependencies = [
     "asyncer>=0.0.4,<1.0",
     "click>=8.1.3,<9.0",
     "cloud-sql-python-connector[pg8000]>=1.7.0,<2.0",
+    "cyclopts>=4.0.0,<5.0",
     "docker>=6.0,<7.0",
     "dpath>=2.1.5,<3.0",
     "fastmcp>=2.12.1,<3.0",
     "gitpython>=3.1.29,<4.0",
     "google-cloud-logging>=3.9.0,<4.0",
     "google-cloud-secret-manager>=2.18.0,<3.0",
+    "google-cloud-storage>=2.8.0,<3.0",
     "jinja2>=3.1.2,<4.0",
     "pydantic>=2.0.0",
     "python-dotenv>=1.0.0,<2.0",
@@ -30,7 +32,6 @@ dependencies = [
     "slack-sdk>=3.36.0,<4.0",
     "sqlalchemy>=2.0.0,<3.0",
     "toml>=0.10.2,<1.0",
-    "cyclopts>=4.0.0,<5.0",
 ]
 keywords = ["airbyte", "mcp", "admin", "api"]
 classifiers = [

airbyte_internal_ops-0.1.7/src/airbyte_ops_mcp/__init__.py

@@ -0,0 +1,44 @@
+"""Airbyte Admin MCP - MCP and API interfaces that let the agents do the admin work.
+
+.. include:: ../../../README.md
+"""
+
+from airbyte_ops_mcp import (
+    airbyte_repo,
+    cli,
+    cloud_admin,
+    connection_config_retriever,
+    constants,
+    live_tests,
+    mcp,
+    prod_db_access,
+    registry,
+)
+
+__version__ = "0.1.0"
+
+
+def hello() -> str:
+    """Return a friendly greeting."""
+    return "Hello from airbyte-internal-ops!"
+
+
+def get_version() -> str:
+    """Return the current version."""
+    return __version__
+
+
+__all__ = [
+    "__version__",
+    "airbyte_repo",
+    "cli",
+    "cloud_admin",
+    "connection_config_retriever",
+    "constants",
+    "get_version",
+    "hello",
+    "live_tests",
+    "mcp",
+    "prod_db_access",
+    "registry",
+]

{airbyte_internal_ops-0.1.5 → airbyte_internal_ops-0.1.7}/src/airbyte_ops_mcp/_legacy/airbyte_ci/connector_pipelines/airbyte_ci/connectors/pipeline.py

@@ -16,7 +16,6 @@ from connector_ops.utils import ConnectorLanguage  # type: ignore
 from consts import ContextState
 from dagger import Config
 from pipelines.airbyte_ci.connectors.context import ConnectorContext
-from pipelines.airbyte_ci.connectors.publish.context import PublishConnectorContext
 from pipelines.airbyte_ci.connectors.test.context import ConnectorTestContext
 from pipelines.airbyte_ci.steps.no_op import NoOpStep
 from pipelines.dagger.actions.system import docker
@@ -55,7 +54,6 @@ async def context_to_step_result(context: PipelineContext) -> StepResult:
 async def run_report_complete_pipeline(
     dagger_client: dagger.Client,
     contexts: List[ConnectorContext]
-    | List[PublishConnectorContext]
     | List[PipelineContext]
     | List[ConnectorTestContext],
 ) -> None:
@@ -87,18 +85,14 @@ async def run_report_complete_pipeline(
 
 
 async def run_connectors_pipelines(
-    contexts: List[ConnectorContext]
-    | List[PublishConnectorContext]
-    | List[ConnectorTestContext],
+    contexts: List[ConnectorContext] | List[ConnectorTestContext],
     connector_pipeline: Callable,
     pipeline_name: str,
     concurrency: int,
     dagger_logs_path: Optional[Path],
     execute_timeout: Optional[int],
     *args: Any,
-) -> (
-    List[ConnectorContext] | List[PublishConnectorContext] | List[ConnectorTestContext]
-):
+) -> List[ConnectorContext] | List[ConnectorTestContext]:
     """Run a connector pipeline for all the connector contexts."""
 
     default_connectors_semaphore = anyio.Semaphore(concurrency)

{airbyte_internal_ops-0.1.5 → airbyte_internal_ops-0.1.7}/src/airbyte_ops_mcp/airbyte_repo/list_connectors.py

@@ -34,6 +34,22 @@ class ConnectorLanguage(StrEnum):
     MANIFEST_ONLY = "manifest-only"
 
 
+class ConnectorType(StrEnum):
+    """Connector types (source or destination)."""
+
+    SOURCE = "source"
+    DESTINATION = "destination"
+
+
+class ConnectorSubtype(StrEnum):
+    """Connector subtypes based on data source category."""
+
+    API = "api"
+    DATABASE = "database"
+    FILE = "file"
+    CUSTOM = "custom"
+
+
 def get_modified_connectors(
     repo_path: str | Path,
     base_ref: str = GIT_DEFAULT_BRANCH,
@@ -245,6 +261,106 @@ def get_connectors_by_language(
     return connectors_by_language
 
 
+def get_connectors_by_type(
+    repo_path: str | Path,
+    connector_type: ConnectorType,
+) -> set[str]:
+    """Get set of all connector IDs for a specific type (source or destination).
+
+    This function reads connector directories to determine which connectors
+    match the specified type based on their metadata.yaml or name prefix.
+
+    Args:
+        repo_path: Path to the Airbyte monorepo
+        connector_type: Type to filter by (source or destination)
+
+    Returns:
+        Set of connector technical names matching the specified type
+
+    Example:
+        >>> source_connectors = get_connectors_by_type(
+        ...     "/path/to/airbyte", ConnectorType.SOURCE
+        ... )
+        >>> "source-postgres" in source_connectors
+        True
+    """
+    repo_path = Path(repo_path)
+    connectors_dir = repo_path / CONNECTOR_PATH_PREFIX
+
+    if not connectors_dir.exists():
+        raise ValueError(f"Connectors directory not found: {connectors_dir}")
+
+    type_value = connector_type.value
+    connectors_by_type = set()
+
+    for connector_dir in connectors_dir.iterdir():
+        if not connector_dir.is_dir():
+            continue
+
+        connector_name = connector_dir.name
+
+        # First try to get type from metadata.yaml
+        metadata = get_connector_metadata(repo_path, connector_name)
+        if metadata:
+            metadata_type = metadata.get("connectorType")
+            if metadata_type == type_value:
+                connectors_by_type.add(connector_name)
+                continue
+
+        # Fallback to name prefix detection
+        if connector_name.startswith(f"{type_value}-"):
+            connectors_by_type.add(connector_name)
+
+    return connectors_by_type
+
+
+def get_connectors_by_subtype(
+    repo_path: str | Path,
+    connector_subtype: ConnectorSubtype,
+) -> set[str]:
+    """Get set of all connector IDs for a specific subtype (api, database, file, etc.).
+
+    This function reads connector metadata.yaml files to determine which connectors
+    match the specified subtype.
+
+    Args:
+        repo_path: Path to the Airbyte monorepo
+        connector_subtype: Subtype to filter by (api, database, file, custom)
+
+    Returns:
+        Set of connector technical names matching the specified subtype
+
+    Example:
+        >>> database_connectors = get_connectors_by_subtype(
+        ...     "/path/to/airbyte", ConnectorSubtype.DATABASE
+        ... )
+        >>> "source-postgres" in database_connectors
+        True
+    """
+    repo_path = Path(repo_path)
+    connectors_dir = repo_path / CONNECTOR_PATH_PREFIX
+
+    if not connectors_dir.exists():
+        raise ValueError(f"Connectors directory not found: {connectors_dir}")
+
+    subtype_value = connector_subtype.value
+    connectors_by_subtype = set()
+
+    for connector_dir in connectors_dir.iterdir():
+        if not connector_dir.is_dir():
+            continue
+
+        connector_name = connector_dir.name
+        metadata = get_connector_metadata(repo_path, connector_name)
+
+        if metadata:
+            metadata_subtype = metadata.get("connectorSubtype")
+            if metadata_subtype == subtype_value:
+                connectors_by_subtype.add(connector_name)
+
+    return connectors_by_subtype
+
+
 @lru_cache(maxsize=1024)
 def _detect_connector_language(connector_dir: Path, connector_name: str) -> str | None:
     """Detect the language of a connector based on its file structure.
@@ -401,6 +517,8 @@ def list_connectors(
     modified: bool | None = None,
     language_filter: set[str] | None = None,
     language_exclude: set[str] | None = None,
+    connector_type: str | None = None,
+    connector_subtype: str | None = None,
     base_ref: str | None = None,
     head_ref: str | None = None,
 ) -> ConnectorListResult:
@@ -417,6 +535,8 @@ def list_connectors(
             False=not-modified only, None=all)
         language_filter: Set of languages to include (python, java, low-code, manifest-only)
         language_exclude: Set of languages to exclude (mutually exclusive with language_filter)
+        connector_type: Filter by connector type (source, destination, or None for all)
+        connector_subtype: Filter by connector subtype (api, database, file, custom, or None for all)
         base_ref: Base git reference for modification detection (default: "origin/main")
         head_ref: Head git reference for modification detection (default: "HEAD")
 
@@ -475,6 +595,18 @@ def list_connectors(
             excluded = get_connectors_by_language(repo_path, ConnectorLanguage(lang))
             result -= excluded
 
+    # Apply connector type filter
+    if connector_type is not None:
+        type_set = get_connectors_by_type(repo_path, ConnectorType(connector_type))
+        result &= type_set
+
+    # Apply connector subtype filter
+    if connector_subtype is not None:
+        subtype_set = get_connectors_by_subtype(
+            repo_path, ConnectorSubtype(connector_subtype)
+        )
+        result &= subtype_set
+
     return ConnectorListResult(
         connectors=sorted(result),
         count=len(result),