glean-indexing-sdk 0.2.0__tar.gz → 0.3.0__tar.gz

glean_indexing_sdk-0.3.0/.claude/commands/release.md

@@ -0,0 +1,77 @@
+# Release Process
+
+Guide the user through releasing a new version of the Glean Indexing SDK.
+
+## Instructions
+
+Follow these steps to release a new version:
+
+### Step 1: Pre-flight Checks
+
+Run the following checks to ensure the codebase is ready for release:
+
+1. **Check git status**: Ensure working directory is clean and on `main` branch
+2. **Pull latest**: `git pull origin main`
+3. **Run tests**: `mise run test`
+4. **Run linting**: `mise run lint`
+
+If any checks fail, stop and fix the issues before proceeding.
+
+### Step 2: Preview the Release
+
+Run a dry-run to see what version bump will occur and what the changelog will look like:
+
+```bash
+DRY_RUN=true mise run release
+```
+
+Show the user:
+- The current version (from `pyproject.toml`)
+- The new version that will be created
+- The changelog entries that will be added
+
+Ask the user to confirm they want to proceed with the release.
+
+### Step 3: Create the Release
+
+If the user confirms, run the actual release:
+
+```bash
+mise run release
+```
+
+This will:
+- Bump the version in `pyproject.toml`
+- Update `CHANGELOG.md`
+- Create a git commit with the version bump
+- Create a git tag (e.g., `v0.2.1`)
+
+### Step 4: Push to Remote
+
+Push the commit and tag to the remote:
+
+```bash
+git push origin main --follow-tags
+```
+
+**Important:** Pushing the tag triggers the GitHub Actions workflow (`.github/workflows/publish.yml`) which automatically:
+- Builds the package
+- Creates a GitHub Release with changelog
+- Publishes to PyPI
+
+### Step 5: Verify the Release
+
+After pushing, guide the user to verify:
+
+1. Check the GitHub Actions workflow is running: `gh run list --limit 3`
+2. Watch the workflow: `gh run watch`
+3. Once complete, verify:
+   - GitHub Release exists: `gh release view vX.Y.Z`
+   - PyPI package is published: https://pypi.org/project/glean-indexing-sdk/
+
+## Summary
+
+At the end, provide a summary:
+- New version number
+- Link to the GitHub release: https://github.com/gleanwork/glean-indexing-sdk/releases/tag/vX.Y.Z
+- Link to PyPI: https://pypi.org/project/glean-indexing-sdk/X.Y.Z/

{glean_indexing_sdk-0.2.0 → glean_indexing_sdk-0.3.0}/.cz.toml

@@ -1,5 +1,5 @@
 [tool.commitizen]
 name = "cz_conventional_commits"
-version = "0.2.0"
+version = "0.3.0"
 tag_format = "v$version"
 version_files = ["pyproject.toml:version", "src/glean/indexing/__init__.py:__version__"] 

{glean_indexing_sdk-0.2.0 → glean_indexing_sdk-0.3.0}/.github/workflows/ci.yml

@@ -22,13 +22,13 @@ jobs:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Setup development environment
-        run: task setup
+        run: mise run setup
 
       - name: Run linters
-        run: task lint
+        run: mise run lint
 
       - name: Build package
-        run: task build
+        run: mise run build
 
   test:
     name: Test Python ${{ matrix.python-version }}
@@ -44,23 +44,20 @@ jobs:
         uses: jdx/mise-action@v2
         with:
           cache: true
-          mise_toml: |
-            [tools]
-            python = "${{ matrix.python-version }}"
-            task = "latest"
-            uv = "latest"
-
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
+      - name: Set Python version for matrix
+        run: mise use python@${{ matrix.python-version }}
+
       - name: Setup development environment
-        run: task setup
+        run: mise run setup
 
       - name: Run tests
-        run: task test
+        run: mise run test
 
       - name: Test package installation
         run: |
-          task build
+          mise run build
           uv run pip install dist/*.whl
           uv run python -c "import glean.indexing; print('Package installed successfully')" 

{glean_indexing_sdk-0.2.0 → glean_indexing_sdk-0.3.0}/CHANGELOG.md

@@ -1,3 +1,9 @@
+## v0.2.0 (2025-07-24)
+
+### Feat
+
+- Adds support for forced restarts of indexing uploads
+
 ## v0.1.0 (2025-07-23)
 
 ### Feat

glean_indexing_sdk-0.3.0/CLAUDE.md

@@ -0,0 +1,73 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Python SDK for building custom Glean indexing connectors. Provides base classes and utilities to create connectors that fetch data from external systems and upload to Glean's indexing APIs.
+
+## Commands
+
+Uses [mise](https://mise.jdx.dev/) (`brew install mise`) for toolchain and task management with `uv` for Python dependencies.
+
+```bash
+# Setup
+mise run setup                # Create venv and install all dependencies
+
+# Testing
+mise run test                 # Run all tests
+mise run test:watch           # Run tests in watch mode
+mise run test:cov             # Run tests with coverage
+
+# Linting
+mise run lint                 # Run all linters (ruff, pyright, readme)
+mise run lint:fix             # Auto-fix lint issues and format code
+
+# Building
+mise run build                # Build the package
+```
+
+Run a single test:
+```bash
+uv run pytest tests/unit_tests/test_base_connector.py -v
+uv run pytest tests/unit_tests/test_base_connector.py::TestClassName::test_method -v
+```
+
+## Architecture
+
+### Core Abstractions
+
+**Connector hierarchy** (`src/glean/indexing/connectors/`):
+- `BaseConnector` - Abstract base defining `get_data()`, `transform()`, `index_data()`
+- `BaseDatasourceConnector` - For document/content indexing (fits in memory)
+- `BaseStreamingDatasourceConnector` - For large/paginated datasets (yields data via sync generator)
+- `BaseAsyncStreamingDatasourceConnector` - For large datasets with async APIs (yields data via async generator)
+- `BasePeopleConnector` - For employee/identity indexing
+
+**Data clients** (`src/glean/indexing/connectors/`):
+- `BaseDataClient[T]` - Fetches all data at once, returns `Sequence[T]`
+- `BaseStreamingDataClient[T]` - Yields data incrementally via `Generator[T]`
+- `BaseAsyncStreamingDataClient[T]` - Yields data incrementally via `AsyncGenerator[T]`
+
+### Pattern: Implementing a Connector
+
+1. Define data type as `TypedDict`
+2. Create data client extending `BaseDataClient[YourType]`
+3. Create connector extending `BaseDatasourceConnector[YourType]`
+4. Set `configuration: CustomDatasourceConfig` class attribute
+5. Implement `transform()` to convert source data to `DocumentDefinition`
+
+### Key Modules
+
+- `models.py` - Type definitions, `IndexingMode`, `DocumentDefinition`, etc.
+- `common/glean_client.py` - API client wrapper (uses env vars `GLEAN_INSTANCE`, `GLEAN_INDEXING_API_TOKEN`)
+- `common/batch_processor.py` - Batches data for upload
+- `observability/` - Logging decorators and metrics tracking
+- `testing/` - `ConnectorTestHarness`, `MockGleanClient` for testing without API calls
+
+## Code Style
+
+- Line length: 160 characters
+- Docstrings: Google style
+- Type hints required (pyright basic mode)
+- Ruff for linting and formatting

{glean_indexing_sdk-0.2.0 → glean_indexing_sdk-0.3.0}/CONTRIBUTING.md

@@ -7,11 +7,11 @@ Thank you for your interest in contributing to the Glean Connector SDK! This doc
 1. Clone the repository
 2. Set up your environment:
    ```bash
-   # Install go-task if not already installed
-   brew install go-task
+   # Install mise if not already installed
+   brew install mise
 
    # Set up development environment
-   task setup
+   mise run setup
    ```
 
 ## Development Workflow
@@ -27,8 +27,8 @@ We use the following workflow for development:
 
 3. Run linting and tests:
    ```bash
-   task lint:fix
-   task test:all
+   mise run lint:fix
+   mise run test:all
    ```
 
 4. Commit your changes using commitizen:
@@ -43,7 +43,7 @@ We use the following workflow for development:
 We follow standard Python code styles:
 
 - Use [Ruff](https://github.com/astral-sh/ruff) for linting and formatting
-- Use [MyPy](https://mypy.readthedocs.io) for type checking
+- Use [Pyright](https://github.com/microsoft/pyright) for type checking
 - Follow [type hints](https://docs.python.org/3/library/typing.html) in all code
 
 ## Testing
@@ -57,10 +57,10 @@ We use [commitizen](https://commitizen-tools.github.io/commitizen/) for versioni
 
 ```bash
 # Perform a dry run
-task release DRY_RUN=true
+DRY_RUN=true mise run release
 
 # Create a new release
-task release
+mise run release
 ```
 
 ## Documentation

{glean_indexing_sdk-0.2.0 → glean_indexing_sdk-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: glean-indexing-sdk
-Version: 0.2.0
+Version: 0.3.0
 Summary: SDK for building custom Glean indexing integrations
 Project-URL: Source Code, https://github.com/glean-io/glean-indexing-sdk
 Author-email: Steve Calvert <steve.calvert@glean.com>
@@ -435,6 +435,7 @@ class LargeKnowledgeBaseClient(StreamingConnectorDataClient[ArticleData]):
 from typing import List, Sequence
 
 from glean.api_client.models.userreferencedefinition import UserReferenceDefinition
+
 from glean.indexing.connectors import BaseStreamingDatasourceConnector
 from glean.indexing.models import ContentDefinition, CustomDatasourceConfig, DocumentDefinition
 

{glean_indexing_sdk-0.2.0 → glean_indexing_sdk-0.3.0}/README.md

@@ -405,6 +405,7 @@ class LargeKnowledgeBaseClient(StreamingConnectorDataClient[ArticleData]):
 from typing import List, Sequence
 
 from glean.api_client.models.userreferencedefinition import UserReferenceDefinition
+
 from glean.indexing.connectors import BaseStreamingDatasourceConnector
 from glean.indexing.models import ContentDefinition, CustomDatasourceConfig, DocumentDefinition
 

glean_indexing_sdk-0.3.0/RELEASE.md

@@ -0,0 +1,84 @@
+# Release Process
+
+This document describes the release process for the Glean Connector SDK.
+
+## Quick Start
+
+Run the `/release` command in Claude Code for a guided release process.
+
+## Dependencies
+
+- [`mise`](https://mise.jdx.dev/) - Tool and task management
+- [`commitizen`](https://github.com/commitizen-tools/commitizen) - Conventional commits and versioning
+- [`uv`](https://github.com/astral-sh/uv) - Python package management
+
+## Versioning
+
+We follow [Semantic Versioning](https://semver.org/).
+
+- **MAJOR** version for incompatible API changes
+- **MINOR** version for new functionality in a backward compatible manner
+- **PATCH** version for backward compatible bug fixes
+
+Version bumps are determined automatically by commit message prefixes:
+- `feat:` → MINOR bump
+- `fix:` → PATCH bump
+- `feat!:` or `BREAKING CHANGE:` → MAJOR bump
+
+## Process
+
+### 1. Ensure everything is ready for release
+
+```bash
+git checkout main
+git pull origin main
+mise run test
+mise run lint
+```
+
+### 2. Preview the release
+
+```bash
+DRY_RUN=true mise run release
+```
+
+This will show you:
+- The version bump (e.g., 0.2.0 → 0.2.1)
+- The changelog entries that will be generated
+
+### 3. Run the release
+
+```bash
+mise run release
+```
+
+This will:
+- Bump the version in `pyproject.toml`
+- Update `CHANGELOG.md`
+- Create a git commit
+- Create a git tag (e.g., `v0.2.1`)
+
+### 4. Push to trigger automated release
+
+```bash
+git push origin main --follow-tags
+```
+
+**That's it!** Pushing the tag triggers the GitHub Actions workflow which automatically:
+- Builds the package
+- Creates a GitHub Release with changelog
+- Publishes to PyPI
+
+### 5. Verify the release
+
+```bash
+# Watch the workflow
+gh run watch
+
+# Verify the release was created
+gh release view v0.2.1
+```
+
+Check:
+- [GitHub Releases](https://github.com/gleanwork/glean-indexing-sdk/releases)
+- [PyPI Package](https://pypi.org/project/glean-indexing-sdk/)

glean_indexing_sdk-0.3.0/mise.toml

@@ -0,0 +1,159 @@
+[tools]
+python = "3.10"
+uv = "latest"
+node = "22"
+
+[tasks]
+
+[tasks.setup]
+description = "Set up local environment (install dependencies, etc.)"
+run = [
+  "uv venv .venv",
+  "uv pip install -e .[dev,test,lint,typing]",
+]
+
+[tasks."test:all"]
+description = "Run all tests and lint fixes"
+run = [
+  "mise run test",
+  "mise run lint:fix",
+  "mise run lint",
+]
+
+[tasks.test]
+description = "Run unit tests"
+run = "uv run pytest -v --tb=auto -rA --durations=10 -p no:logging tests/"
+
+[tasks."test:watch"]
+description = "Run tests in watch mode"
+run = "uv run ptw --now . -- -vv --tb=auto -rA --durations=10 -p no:logging tests/"
+
+[tasks."test:cov"]
+description = "Run tests with coverage"
+run = "uv run pytest --cov=glean --cov-report=term --cov-report=html -v tests/"
+
+[tasks.lint]
+description = "Run all linters"
+run = [
+  "mise run lint:ruff",
+  "mise run lint:format:check",
+  "mise run lint:pyright",
+  "mise run lint:readme",
+]
+
+[tasks."lint:diff"]
+description = "Run linters on changed files"
+run = [
+  "bash -lc \"PYTHON_FILES=$(git diff --name-only --diff-filter=d main | grep -E '.*\\.(py|ipynb)$' || true); [ -z \\\"$PYTHON_FILES\\\" ] || uv run ruff check $PYTHON_FILES\"",
+  "bash -lc \"PYTHON_FILES=$(git diff --name-only --diff-filter=d main | grep -E '.*\\.(py|ipynb)$' || true); [ -z \\\"$PYTHON_FILES\\\" ] || uv run ruff format $PYTHON_FILES --diff\"",
+  "bash -lc \"PYTHON_FILES=$(git diff --name-only --diff-filter=d main | grep -E '.*\\.(py|ipynb)$' || true); if [ -z \\\"$PYTHON_FILES\\\" ]; then uv run pyright; else uv run pyright $PYTHON_FILES; fi\"",
+]
+
+[tasks."lint:package"]
+description = "Run linters on package files"
+run = [
+  "uv run ruff check src/glean/indexing",
+  "uv run ruff format src/glean/indexing --diff",
+  "uv run pyright src/glean/indexing",
+]
+
+[tasks."lint:tests"]
+description = "Run linters on test files"
+run = [
+  "uv run ruff check tests",
+  "uv run ruff format tests --diff",
+  "uv run pyright tests",
+]
+
+[tasks."lint:fix"]
+description = "Run the lint autofixers"
+run = [
+  "mise run lint:fix:ruff",
+  "mise run format",
+]
+
+[tasks."lint:fix:diff"]
+description = "Run lint autofixers on changed files"
+run = [
+  "bash -lc \"PYTHON_FILES=$(git diff --name-only --diff-filter=d main | grep -E '.*\\.(py|ipynb)$' || true); [ -z \\\"$PYTHON_FILES\\\" ] || uv run ruff check $PYTHON_FILES --fix\"",
+  "mise run format:diff",
+]
+
+[tasks."lint:fix:package"]
+description = "Run lint autofixers on package files"
+run = [
+  "uv run ruff check src/glean/indexing --fix",
+  "mise run format",
+]
+
+[tasks."lint:fix:ruff"]
+description = "Run Ruff autofixer"
+run = "uv run ruff check . --fix --exclude .venv/ --exclude **/site-packages/"
+
+[tasks."lint:ruff"]
+description = "Run Ruff linter"
+run = "uv run ruff check . --exclude .venv/ --exclude **/site-packages/"
+
+[tasks."lint:format:check"]
+description = "Check code formatting"
+run = "uv run ruff format . --diff"
+
+[tasks."lint:pyright"]
+description = "Run Pyright type checker"
+run = "uv run pyright"
+
+[tasks."lint:readme"]
+description = "Lint the README.md file"
+run = "npx -y markdown-code check"
+
+[tasks."lint:readme:fix"]
+description = "Fix the README.md file"
+run = "npx -y markdown-code sync"
+
+[tasks.format]
+description = "Run code formatters"
+run = [
+  "mise run format:ruff",
+  "mise run format:imports",
+]
+
+[tasks."format:diff"]
+description = "Run formatters on changed files"
+run = [
+  "bash -lc \"PYTHON_FILES=$(git diff --name-only --diff-filter=d main | grep -E '.*\\.(py|ipynb)$' || true); [ -z \\\"$PYTHON_FILES\\\" ] || uv run ruff format $PYTHON_FILES\"",
+  "bash -lc \"PYTHON_FILES=$(git diff --name-only --diff-filter=d main | grep -E '.*\\.(py|ipynb)$' || true); [ -z \\\"$PYTHON_FILES\\\" ] || uv run ruff check --select I --fix $PYTHON_FILES\"",
+]
+
+[tasks."format:ruff"]
+description = "Run Ruff formatter"
+run = "uv run ruff format ."
+
+[tasks."format:imports"]
+description = "Fix imports"
+run = "uv run ruff check --select I --fix ."
+
+[tasks."spell:check"]
+description = "Check spelling"
+run = "uv run codespell --toml pyproject.toml --skip=docs/"
+
+[tasks."spell:fix"]
+description = "Fix spelling"
+run = "uv run codespell --toml pyproject.toml -w --skip=docs/"
+
+[tasks.clean]
+description = "Clean build artifacts"
+run = [
+  "rm -rf dist .venv build **/*.egg-info .pytest_cache .coverage htmlcov .ruff_cache",
+  "find . -name \"*.pyc\" -delete",
+  "find . -name \"__pycache__\" -delete",
+]
+
+[tasks.build]
+description = "Build the package"
+run = "uv run python -m build"
+
+[tasks.release]
+description = "Bump version and create a new tag (use DRY_RUN=true for preview)"
+run = [
+  "bash -lc 'if [ \"$DRY_RUN\" = \"true\" ]; then uv run python -m commitizen bump --dry-run && uv run python -m commitizen changelog --dry-run; else uv run python -m commitizen bump --yes && uv run python -m commitizen changelog; fi'",
+]

{glean_indexing_sdk-0.2.0 → glean_indexing_sdk-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "glean-indexing-sdk"
-version = "0.2.0"
+version = "0.3.0"
 description = "SDK for building custom Glean indexing integrations"
 authors = [{ name = "Steve Calvert", email = "steve.calvert@glean.com" }]
 readme = "README.md"
@@ -129,3 +129,9 @@ filterwarnings = [
 
 [tool.hatch.build.targets.wheel]
 packages = ["src/glean"] 
+
+[dependency-groups]
+dev = [
+    "pytest>=8.3.5",
+    "pytest-asyncio>=0.26.0",
+]

{glean_indexing_sdk-0.2.0 → glean_indexing_sdk-0.3.0}/snippets/streaming/article_connector.py

@@ -1,6 +1,7 @@
 from typing import List, Sequence
 
 from glean.api_client.models.userreferencedefinition import UserReferenceDefinition
+
 from glean.indexing.connectors import BaseStreamingDatasourceConnector
 from glean.indexing.models import ContentDefinition, CustomDatasourceConfig, DocumentDefinition
 

{glean_indexing_sdk-0.2.0 → glean_indexing_sdk-0.3.0}/src/glean/indexing/__init__.py

@@ -1,56 +1,56 @@
 """Glean Indexing SDK.
 
-A Python SDK for building custom Glean indexing solutions. This package provides 
+A Python SDK for building custom Glean indexing solutions. This package provides
 the base classes and utilities to create custom connectors for Glean's indexing APIs.
 """
 
-from importlib.metadata import version, PackageNotFoundError
+from importlib.metadata import PackageNotFoundError, version
+
+from glean.indexing import models
+from glean.indexing.common import BatchProcessor, ConnectorMetrics, ContentFormatter, MockGleanClient, api_client
 from glean.indexing.connectors import (
+    BaseAsyncStreamingDataClient,
+    BaseAsyncStreamingDatasourceConnector,
     BaseConnector,
+    BaseDataClient,
     BaseDatasourceConnector,
-    BaseStreamingDatasourceConnector,
     BasePeopleConnector,
-    BaseConnectorDataClient,
-    StreamingConnectorDataClient,
+    BaseStreamingDataClient,
+    BaseStreamingDatasourceConnector,
 )
-from glean.indexing.common import BatchProcessor, ContentFormatter, ConnectorMetrics, api_client, MockGleanClient
-from glean.indexing.observability.observability import ConnectorObservability
-from glean.indexing.testing import ConnectorTestHarness
 from glean.indexing.models import (
     DatasourceIdentityDefinitions,
     IndexingMode,
-    TSourceData,
     TIndexableEntityDefinition,
+    TSourceData,
 )
-from glean.indexing import models
+from glean.indexing.observability.observability import ConnectorObservability
+from glean.indexing.testing import ConnectorTestHarness
 
 __all__ = [
     "BaseConnector",
+    "BaseDataClient",
     "BaseDatasourceConnector",
     "BasePeopleConnector",
+    "BaseStreamingDataClient",
     "BaseStreamingDatasourceConnector",
-    
-    "BaseConnectorDataClient",
-    "StreamingConnectorDataClient",
-    
+    "BaseAsyncStreamingDataClient",
+    "BaseAsyncStreamingDatasourceConnector",
     "BatchProcessor",
     "ContentFormatter",
     "ConnectorMetrics",
     "ConnectorObservability",
     "ConnectorTestHarness",
-    
     "DatasourceIdentityDefinitions",
     "IndexingMode",
     "TSourceData",
     "TIndexableEntityDefinition",
-    
     "MockGleanClient",
     "api_client",
-
     "models",
 ]
 
 try:
     __version__ = version("glean-indexing-sdk")
 except PackageNotFoundError:
-    __version__ = "0.2.0"
+    __version__ = "0.3.0"

{glean_indexing_sdk-0.2.0 → glean_indexing_sdk-0.3.0}/src/glean/indexing/connectors/__init__.py

@@ -1,21 +1,23 @@
 """Connector implementations for Glean indexing."""
 
 from glean.indexing.connectors.base_connector import BaseConnector
-from glean.indexing.connectors.base_data_client import BaseDataClient, BaseConnectorDataClient
+from glean.indexing.connectors.base_data_client import BaseDataClient
 from glean.indexing.connectors.base_datasource_connector import BaseDatasourceConnector
 from glean.indexing.connectors.base_people_connector import BasePeopleConnector
-from glean.indexing.connectors.base_streaming_data_client import BaseStreamingDataClient, StreamingConnectorDataClient
+from glean.indexing.connectors.base_streaming_data_client import BaseStreamingDataClient
 from glean.indexing.connectors.base_streaming_datasource_connector import BaseStreamingDatasourceConnector
+from glean.indexing.connectors.base_async_streaming_data_client import BaseAsyncStreamingDataClient
+from glean.indexing.connectors.base_async_streaming_datasource_connector import BaseAsyncStreamingDatasourceConnector
 from glean.indexing.testing.connector_test_harness import ConnectorTestHarness
 
 __all__ = [
     "BaseConnector",
     "BaseDataClient",
-    "BaseConnectorDataClient",  # Backward compatibility alias
     "BaseDatasourceConnector",
     "BasePeopleConnector",
-    "BaseStreamingDataClient", 
-    "StreamingConnectorDataClient",  # Backward compatibility alias
+    "BaseStreamingDataClient",
     "BaseStreamingDatasourceConnector",
+    "BaseAsyncStreamingDataClient",
+    "BaseAsyncStreamingDatasourceConnector",
     "ConnectorTestHarness",
 ]