container-superposition 0.1.7 → 0.1.8

package/overlays/pgvector/README.md

@@ -0,0 +1,215 @@
+# pgvector Overlay
+
+Adds a PostgreSQL 16 database pre-loaded with the [pgvector](https://github.com/pgvector/pgvector) extension, enabling vector similarity search directly within a familiar SQL environment.
+
+> **Note:** This overlay conflicts with the `postgres` overlay because both provide a PostgreSQL service on port 5432. Choose `pgvector` when you need vector search, or `postgres` for a plain database.
+
+## Features
+
+- **pgvector extension** — `CREATE EXTENSION vector;` is available immediately; no manual installation required
+- **Full PostgreSQL 16** — All standard SQL features alongside vector operations
+- **`psql` client in devcontainer** — `postgresql-client` package installed for interactive SQL sessions
+- **Pre-configured environment** — `PGHOST`, `PGPORT`, `PGDATABASE`, and `PGUSER` set automatically
+- **Persistent storage** — Named Docker volume preserves data across container rebuilds
+- **Health check** — Compose readiness check using `pg_isready` before dependent services start
+
+## How It Works
+
+pgvector runs as a Docker Compose service (`pgvector`) alongside your devcontainer, using the official `pgvector/pgvector:pg16` image, which bundles PostgreSQL 16 with the pgvector extension pre-compiled and ready to use.
+
+**Service configuration:**
+
+- Image: `pgvector/pgvector:pg16`
+- Network: `devnet` (shared with the dev container)
+- Port: `5432` (customisable via `PGVECTOR_PORT`)
+- Volume: `pgvector-data` for persistent data
+
+The following environment variables are pre-set in the devcontainer for seamless `psql` usage:
+
+| Variable     | Value      |
+| ------------ | ---------- |
+| `PGHOST`     | `pgvector` |
+| `PGPORT`     | `5432`     |
+| `PGDATABASE` | `devdb`    |
+| `PGUSER`     | `postgres` |
+
+## Getting Started
+
+```bash
+# Connect to the database
+psql -h pgvector -U postgres -d devdb
+
+# Enable the vector extension (run once per database)
+psql -h pgvector -U postgres -d devdb -c "CREATE EXTENSION IF NOT EXISTS vector;"
+
+# Verify the extension is installed
+psql -h pgvector -U postgres -d devdb -c "SELECT * FROM pg_extension WHERE extname = 'vector';"
+```
+
+## Common Commands
+
+### Vector Operations in SQL
+
+```sql
+-- Enable the extension in your database
+CREATE EXTENSION IF NOT EXISTS vector;
+
+-- Create a table with a vector column (1536 dimensions for OpenAI embeddings)
+CREATE TABLE documents (
+    id       SERIAL PRIMARY KEY,
+    content  TEXT,
+    embedding vector(1536)
+);
+
+-- Insert a record with an embedding
+INSERT INTO documents (content, embedding)
+VALUES ('Hello, world!', '[0.1, 0.2, 0.3, ...]');
+
+-- Cosine similarity search (find closest neighbours)
+SELECT id, content, 1 - (embedding <=> '[0.1, 0.2, 0.3, ...]') AS similarity
+FROM documents
+ORDER BY embedding <=> '[0.1, 0.2, 0.3, ...]'
+LIMIT 5;
+
+-- Create an HNSW index for faster approximate search
+CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops);
+
+-- Create an IVFFlat index for large datasets
+CREATE INDEX ON documents USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
+```
+
+### Database Management
+
+```bash
+# Connect interactively
+psql
+
+# Run a SQL file
+psql -f schema.sql
+
+# Dump the database
+pg_dump -h pgvector -U postgres devdb > backup.sql
+
+# Restore from dump
+psql -h pgvector -U postgres devdb < backup.sql
+```
+
+### Python Integration
+
+```bash
+pip install psycopg2-binary pgvector
+
+python - <<'EOF'
+import psycopg2
+from pgvector.psycopg2 import register_vector
+import numpy as np
+
+conn = psycopg2.connect(
+    host="pgvector", port=5432,
+    dbname="devdb", user="postgres", password="postgres"
+)
+register_vector(conn)
+cur = conn.cursor()
+
+cur.execute("CREATE EXTENSION IF NOT EXISTS vector")
+cur.execute("CREATE TABLE IF NOT EXISTS items (id serial PRIMARY KEY, embedding vector(3))")
+cur.execute("INSERT INTO items (embedding) VALUES (%s)", (np.array([1, 2, 3]),))
+conn.commit()
+
+cur.execute("SELECT * FROM items ORDER BY embedding <-> %s LIMIT 5", (np.array([1, 2, 4]),))
+print(cur.fetchall())
+conn.close()
+EOF
+```
+
+## Configuration
+
+### Environment Variables
+
+| Variable            | Default    | Description                                |
+| ------------------- | ---------- | ------------------------------------------ |
+| `PGVECTOR_VERSION`  | `pg16`     | pgvector image tag (e.g. `pg16`, `pg17`)   |
+| `PGVECTOR_DB`       | `devdb`    | Name of the database to create             |
+| `PGVECTOR_USER`     | `postgres` | PostgreSQL superuser name                  |
+| `PGVECTOR_PASSWORD` | `postgres` | PostgreSQL superuser password              |
+| `PGVECTOR_PORT`     | `5432`     | Host port mapped to the PostgreSQL service |
+
+Copy `.devcontainer/.env.example` to `.devcontainer/.env` and customize.
+
+## pgvector vs Qdrant
+
+| Feature              | pgvector (this overlay)         | Qdrant                   |
+| -------------------- | ------------------------------- | ------------------------ |
+| **Storage model**    | ✅ Relational + vector          | Vector-only              |
+| **SQL support**      | ✅ Full PostgreSQL SQL          | ❌ REST/gRPC API only    |
+| **Familiar tooling** | ✅ psql, ORMs, existing schema  | New client SDKs required |
+| **Query language**   | ✅ SQL with `<->`, `<=>`, `<#>` | ⚠️ JSON query DSL        |
+| **Filtering**        | ✅ SQL WHERE clause             | ✅ Payload-based filters |
+| **Scale**            | ⚠️ Vertical scaling preferred   | ✅ Distributed by design |
+| **Index types**      | IVFFlat, HNSW                   | HNSW                     |
+
+**Choose pgvector when:**
+
+- You already use PostgreSQL and want to add vectors to existing tables
+- You need to join vector results with relational data
+- Your team is comfortable with SQL
+
+**Choose Qdrant when:**
+
+- Vectors are your primary data model
+- You need purpose-built vector search performance at scale
+- You want advanced filtering and payload storage
+
+## Use Cases
+
+- **RAG pipelines** — Store document embeddings alongside metadata; retrieve context with SQL joins
+- **Semantic search** — Add vector search to an existing PostgreSQL application without a second database
+- **Hybrid search** — Combine full-text search (`tsvector`) with vector similarity in a single query
+- **Recommendation systems** — Store and query user or item embeddings with familiar SQL tooling
+
+**Integrates well with:**
+
+- `ollama` — Generate embeddings locally with `nomic-embed-text` or similar models
+- `python` — `psycopg2` + `pgvector` Python package for seamless integration
+- `nodejs` — `pg` driver with raw SQL for vector operations
+
+## Troubleshooting
+
+### Extension Not Found
+
+If `CREATE EXTENSION vector` fails:
+
+```bash
+# Confirm pgvector image (not plain postgres) is in use
+docker compose ps pgvector
+
+# Check available extensions
+psql -h pgvector -U postgres -c "SELECT name FROM pg_available_extensions WHERE name = 'vector';"
+```
+
+### Port Conflict with postgres Overlay
+
+The `pgvector` and `postgres` overlays both bind port 5432 and conflict with each other. Select only one.
+
+### Connection Refused
+
+```bash
+# Check service health
+docker compose ps pgvector
+docker compose logs pgvector
+```
+
+## References
+
+- [pgvector GitHub](https://github.com/pgvector/pgvector)
+- [pgvector Docker Image](https://hub.docker.com/r/pgvector/pgvector)
+- [PostgreSQL Documentation](https://www.postgresql.org/docs/16/)
+- [pgvector Python package](https://github.com/pgvector/pgvector-python)
+- [pgvector Node.js package](https://github.com/pgvector/pgvector-node)
+
+**Related Overlays:**
+
+- [`postgres`](../postgres/README.md) — Plain PostgreSQL without pgvector (conflicts)
+- [`qdrant`](../qdrant/README.md) — Purpose-built vector database (alternative)
+- [`ollama`](../ollama/README.md) — Local embedding generation
+- [`python`](../python/README.md) — Python development environment

package/overlays/pgvector/devcontainer.patch.json

@@ -0,0 +1,23 @@
+{
+    "$schema": "https://raw.githubusercontent.com/devcontainers/spec/main/schemas/devContainer.base.schema.json",
+    "runServices": ["pgvector"],
+    "forwardPorts": [5432],
+    "portsAttributes": {
+        "5432": {
+            "label": "pgvector database",
+            "onAutoForward": "notify"
+        }
+    },
+    "features": {
+        "./features/cross-distro-packages": {
+            "apt": "postgresql-client",
+            "apk": "postgresql-client"
+        }
+    },
+    "containerEnv": {
+        "PGHOST": "pgvector",
+        "PGPORT": "5432",
+        "PGDATABASE": "devdb",
+        "PGUSER": "postgres"
+    }
+}

package/overlays/pgvector/docker-compose.yml

@@ -0,0 +1,32 @@
+version: '3.8'
+
+services:
+    pgvector:
+        image: pgvector/pgvector:${PGVECTOR_VERSION:-{{cs.PGVECTOR_VERSION}}}
+        restart: unless-stopped
+        volumes:
+            - pgvector-data:/var/lib/postgresql/data
+        environment:
+            POSTGRES_DB: ${PGVECTOR_DB:-{{cs.PGVECTOR_DB}}}
+            POSTGRES_USER: ${PGVECTOR_USER:-{{cs.PGVECTOR_USER}}}
+            POSTGRES_PASSWORD: ${PGVECTOR_PASSWORD:-{{cs.PGVECTOR_PASSWORD}}}
+        ports:
+            - '${PGVECTOR_PORT:-{{cs.PGVECTOR_PORT}}}:5432'
+        networks:
+            - devnet
+        healthcheck:
+            test:
+                [
+                    'CMD-SHELL',
+                    'pg_isready -U ${PGVECTOR_USER:-{{cs.PGVECTOR_USER}}} -d ${PGVECTOR_DB:-{{cs.PGVECTOR_DB}}}',
+                ]
+            interval: 10s
+            timeout: 5s
+            retries: 5
+            start_period: 10s
+
+volumes:
+    pgvector-data:
+
+networks:
+    devnet:

package/overlays/pgvector/overlay.yml

@@ -0,0 +1,44 @@
+id: pgvector
+name: pgvector (PostgreSQL + vector)
+description: PostgreSQL 16 with the pgvector extension for vector similarity search
+category: database
+supports:
+    - compose
+requires: []
+suggests:
+    - ollama
+    - python
+    - nodejs
+conflicts:
+    - postgres
+tags:
+    - database
+    - sql
+    - vector
+    - embeddings
+    - postgres
+    - pgvector
+ports:
+    - port: 5432
+      service: pgvector
+      protocol: tcp
+      description: pgvector database connection
+      onAutoForward: notify
+      connectionStringTemplate: 'postgresql://{pgvector_user}:{pgvector_password}@{host}:{port}/{pgvector_db}'
+parameters:
+    PGVECTOR_DB:
+        description: Name of the database to create
+        default: devdb
+    PGVECTOR_USER:
+        description: PostgreSQL superuser name
+        default: postgres
+    PGVECTOR_PASSWORD:
+        description: PostgreSQL superuser password
+        default: postgres
+        sensitive: true
+    PGVECTOR_PORT:
+        description: Host port mapped to pgvector (5432 inside container)
+        default: '5432'
+    PGVECTOR_VERSION:
+        description: pgvector/pgvector image tag (e.g. pg16, pg17)
+        default: pg16

package/overlays/postgres/.env.example

@@ -1,6 +1,6 @@
 # PostgreSQL Configuration
-POSTGRES_VERSION=16
-POSTGRES_DB=devdb
-POSTGRES_USER=postgres
-POSTGRES_PASSWORD=postgres
-POSTGRES_PORT=5432
+POSTGRES_VERSION={{cs.POSTGRES_VERSION}}
+POSTGRES_DB={{cs.POSTGRES_DB}}
+POSTGRES_USER={{cs.POSTGRES_USER}}
+POSTGRES_PASSWORD={{cs.POSTGRES_PASSWORD}}
+POSTGRES_PORT={{cs.POSTGRES_PORT}}

package/overlays/postgres/devcontainer.patch.json

@@ -13,9 +13,9 @@
     },
     "remoteEnv": {
         "POSTGRES_HOST": "postgres",
-        "POSTGRES_PORT": "5432",
-        "POSTGRES_DB": "devdb",
-        "POSTGRES_USER": "postgres",
-        "POSTGRES_PASSWORD": "postgres"
+        "POSTGRES_PORT": "{{cs.POSTGRES_PORT}}",
+        "POSTGRES_DB": "{{cs.POSTGRES_DB}}",
+        "POSTGRES_USER": "{{cs.POSTGRES_USER}}",
+        "POSTGRES_PASSWORD": "{{cs.POSTGRES_PASSWORD}}"
     }
 }

package/overlays/postgres/docker-compose.yml

@@ -2,20 +2,24 @@ version: '3.8'
 
 services:
     postgres:
-        image: postgres:${POSTGRES_VERSION:-16}-alpine
+        image: postgres:${POSTGRES_VERSION:-{{cs.POSTGRES_VERSION}}}-alpine
         restart: unless-stopped
         volumes:
             - postgres-data:/var/lib/postgresql/data
         environment:
-            POSTGRES_DB: ${POSTGRES_DB:-devdb}
-            POSTGRES_USER: ${POSTGRES_USER:-postgres}
-            POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-postgres}
+            POSTGRES_DB: ${POSTGRES_DB:-{{cs.POSTGRES_DB}}}
+            POSTGRES_USER: ${POSTGRES_USER:-{{cs.POSTGRES_USER}}}
+            POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-{{cs.POSTGRES_PASSWORD}}}
         ports:
-            - '${POSTGRES_PORT:-5432}:5432'
+            - '${POSTGRES_PORT:-{{cs.POSTGRES_PORT}}}:5432'
         networks:
             - devnet
         healthcheck:
-            test: ['CMD-SHELL', 'pg_isready -U ${POSTGRES_USER:-postgres} -d ${POSTGRES_DB:-devdb}']
+            test:
+                [
+                    'CMD-SHELL',
+                    'pg_isready -U ${POSTGRES_USER:-{{cs.POSTGRES_USER}}} -d ${POSTGRES_DB:-{{cs.POSTGRES_DB}}}',
+                ]
             interval: 10s
             timeout: 5s
             retries: 5

package/overlays/postgres/overlay.yml

@@ -6,7 +6,8 @@ supports:
     - compose
 requires: []
 suggests: []
-conflicts: []
+conflicts:
+    - pgvector
 tags:
     - database
     - sql
@@ -18,3 +19,20 @@ ports:
       description: PostgreSQL database connection
       onAutoForward: notify
       connectionStringTemplate: 'postgresql://{postgres_user}:{postgres_password}@{host}:{port}/{postgres_db}'
+parameters:
+    POSTGRES_DB:
+        description: Name of the database to create
+        default: devdb
+    POSTGRES_USER:
+        description: PostgreSQL superuser name
+        default: postgres
+    POSTGRES_PASSWORD:
+        description: PostgreSQL superuser password
+        default: postgres
+        sensitive: true
+    POSTGRES_PORT:
+        description: Host port mapped to PostgreSQL (5432 inside container)
+        default: '5432'
+    POSTGRES_VERSION:
+        description: PostgreSQL major version
+        default: '16'

package/overlays/qdrant/.env.example

@@ -0,0 +1,4 @@
+# Qdrant Configuration
+QDRANT_VERSION={{cs.QDRANT_VERSION}}
+QDRANT_PORT={{cs.QDRANT_PORT}}
+QDRANT_GRPC_PORT={{cs.QDRANT_GRPC_PORT}}

package/overlays/qdrant/README.md

@@ -0,0 +1,216 @@
+# Qdrant Overlay
+
+Adds [Qdrant](https://qdrant.tech) as a Docker Compose service, providing a high-performance vector database for similarity search and embedding-based retrieval.
+
+## Features
+
+- **Qdrant vector database** — Store, index, and query high-dimensional embedding vectors at scale
+- **REST API on port 6333** — Full-featured HTTP API for collections, points, and search
+- **gRPC API on port 6334** — High-throughput binary protocol for production workloads
+- **Persistent storage** — Named Docker volume preserves data across container rebuilds
+- **Pre-configured `QDRANT_URL`** — Container environment variable set for easy SDK usage
+- **Health check** — Compose readiness check against `/readyz` before dependent services start
+
+## How It Works
+
+Qdrant runs as a Docker Compose service alongside your devcontainer. The devcontainer connects to it using the hostname `qdrant` on port `6333` (REST) or `6334` (gRPC).
+
+**Service configuration:**
+
+- Image: `qdrant/qdrant:latest`
+- Network: `devnet` (shared with the dev container)
+- Ports: `6333` (REST API), `6334` (gRPC)
+- Volume: `qdrant-data` for persistent vector storage
+
+The following environment variables are pre-set in the devcontainer:
+
+| Variable      | Value                |
+| ------------- | -------------------- |
+| `QDRANT_HOST` | `qdrant`             |
+| `QDRANT_PORT` | `6333`               |
+| `QDRANT_URL`  | `http://qdrant:6333` |
+
+## Common Commands
+
+### REST API
+
+```bash
+# Health check
+curl http://qdrant:6333/healthz
+
+# Create a collection
+curl -X PUT http://qdrant:6333/collections/my_collection \
+  -H "Content-Type: application/json" \
+  -d '{
+    "vectors": {
+      "size": 1536,
+      "distance": "Cosine"
+    }
+  }'
+
+# Insert points
+curl -X PUT http://qdrant:6333/collections/my_collection/points \
+  -H "Content-Type: application/json" \
+  -d '{
+    "points": [
+      {
+        "id": 1,
+        "vector": [0.1, 0.2, 0.3],
+        "payload": {"text": "example document"}
+      }
+    ]
+  }'
+
+# Search for similar vectors
+curl -X POST http://qdrant:6333/collections/my_collection/points/search \
+  -H "Content-Type: application/json" \
+  -d '{
+    "vector": [0.1, 0.2, 0.3],
+    "limit": 5
+  }'
+
+# List collections
+curl http://qdrant:6333/collections
+```
+
+### Python SDK
+
+```bash
+pip install qdrant-client
+
+python - <<'EOF'
+from qdrant_client import QdrantClient
+from qdrant_client.models import Distance, VectorParams, PointStruct
+
+client = QdrantClient(url="http://qdrant:6333")
+
+# Create collection
+client.create_collection(
+    collection_name="my_collection",
+    vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
+)
+
+# Insert vectors
+client.upsert(
+    collection_name="my_collection",
+    points=[
+        PointStruct(id=1, vector=[0.1] * 1536, payload={"text": "hello"})
+    ],
+)
+
+# Search
+results = client.search(
+    collection_name="my_collection",
+    query_vector=[0.1] * 1536,
+    limit=5,
+)
+print(results)
+EOF
+```
+
+### Node.js SDK
+
+```bash
+npm install @qdrant/js-client-rest
+
+node - <<'EOF'
+import { QdrantClient } from "@qdrant/js-client-rest";
+
+const client = new QdrantClient({ url: "http://qdrant:6333" });
+
+await client.createCollection("my_collection", {
+  vectors: { size: 1536, distance: "Cosine" },
+});
+
+const results = await client.search("my_collection", {
+  vector: Array(1536).fill(0.1),
+  limit: 5,
+});
+console.log(results);
+EOF
+```
+
+## Configuration
+
+### Environment Variables
+
+| Variable           | Default  | Description             |
+| ------------------ | -------- | ----------------------- |
+| `QDRANT_VERSION`   | `latest` | Qdrant Docker image tag |
+| `QDRANT_PORT`      | `6333`   | Host port for REST API  |
+| `QDRANT_GRPC_PORT` | `6334`   | Host port for gRPC API  |
+
+### Port Customization
+
+```bash
+# .devcontainer/.env
+QDRANT_PORT=6335
+QDRANT_GRPC_PORT=6336
+```
+
+## Use Cases
+
+- **RAG (Retrieval-Augmented Generation)** — Store embeddings for document chunks; retrieve context for LLM queries
+- **Semantic search** — Find similar documents, images, or code snippets by embedding similarity
+- **Recommendation systems** — Surface similar items based on learned embeddings
+- **Anomaly detection** — Identify outliers in high-dimensional embedding spaces
+- **Local AI pipelines** — Pair with `ollama` for fully offline embedding + retrieval workflows
+
+**Integrates well with:**
+
+- `ollama` — Generate embeddings locally (`ollama pull nomic-embed-text`)
+- `python` — LangChain, LlamaIndex, or raw `qdrant-client` integrations
+- `nodejs` — `@qdrant/js-client-rest` for JavaScript embeddings workflows
+
+## Troubleshooting
+
+### Service Not Ready
+
+```bash
+# Check service status
+docker compose ps qdrant
+
+# View Qdrant logs
+docker compose logs qdrant
+
+# Test health endpoint
+curl http://qdrant:6333/healthz
+```
+
+### Port Conflict
+
+If ports 6333 or 6334 are already in use:
+
+```bash
+# .devcontainer/.env
+QDRANT_PORT=6335
+QDRANT_GRPC_PORT=6336
+```
+
+### Data Persistence
+
+Qdrant data is stored in the `qdrant-data` named Docker volume. It persists across container rebuilds but is scoped to the Docker environment:
+
+```bash
+# Inspect the volume
+docker volume inspect qdrant-data
+
+# Backup data
+docker run --rm -v qdrant-data:/data -v $(pwd):/backup alpine \
+  tar czf /backup/qdrant-backup.tar.gz /data
+```
+
+## References
+
+- [Qdrant Documentation](https://qdrant.tech/documentation/)
+- [Qdrant REST API Reference](https://qdrant.github.io/qdrant/redoc/index.html)
+- [Qdrant Docker Hub](https://hub.docker.com/r/qdrant/qdrant)
+- [Python SDK](https://python-client.qdrant.tech/)
+- [JavaScript SDK](https://github.com/qdrant/qdrant-js)
+
+**Related Overlays:**
+
+- [`pgvector`](../pgvector/README.md) — PostgreSQL-native vector storage (alternative; conflicts with `postgres`)
+- [`ollama`](../ollama/README.md) — Local LLM and embedding generation
+- [`python`](../python/README.md) — Python development environment
+- [`nodejs`](../nodejs/README.md) — Node.js development environment

package/overlays/qdrant/devcontainer.patch.json

@@ -0,0 +1,20 @@
+{
+    "$schema": "https://raw.githubusercontent.com/devcontainers/spec/main/schemas/devContainer.base.schema.json",
+    "runServices": ["qdrant"],
+    "forwardPorts": [6333, 6334],
+    "portsAttributes": {
+        "6333": {
+            "label": "Qdrant REST API",
+            "onAutoForward": "notify"
+        },
+        "6334": {
+            "label": "Qdrant gRPC",
+            "onAutoForward": "ignore"
+        }
+    },
+    "containerEnv": {
+        "QDRANT_HOST": "qdrant",
+        "QDRANT_PORT": "6333",
+        "QDRANT_URL": "http://qdrant:6333"
+    }
+}

package/overlays/qdrant/docker-compose.yml

@@ -0,0 +1,25 @@
+version: '3.8'
+
+services:
+    qdrant:
+        image: qdrant/qdrant:${QDRANT_VERSION:-{{cs.QDRANT_VERSION}}}
+        restart: unless-stopped
+        volumes:
+            - qdrant-data:/qdrant/storage
+        ports:
+            - '${QDRANT_PORT:-{{cs.QDRANT_PORT}}}:6333'
+            - '${QDRANT_GRPC_PORT:-{{cs.QDRANT_GRPC_PORT}}}:6334'
+        networks:
+            - devnet
+        healthcheck:
+            test: ['CMD-SHELL', 'curl -sf http://localhost:6333/readyz || exit 1']
+            interval: 10s
+            timeout: 5s
+            retries: 5
+            start_period: 10s
+
+volumes:
+    qdrant-data:
+
+networks:
+    devnet: