nc-db 0.1.3__tar.gz

nc_db-0.1.3/PKG-INFO

@@ -0,0 +1,174 @@
+Metadata-Version: 2.3
+Name: nc_db
+Version: 0.1.3
+Summary: A PostgresSql Database project with a fully generic infrastructure and design.
+License: Proprietary
+Author: Colt Hale
+Author-email: colt@pixelthin.com
+Requires-Python: >=3.12,<4.0
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: psycopg-pool (>=3.2.6,<4.0.0)
+Requires-Dist: psycopg[binary] (>=3.2.9,<4.0.0)
+Requires-Dist: pydantic (>=2.11.7,<3.0.0)
+Description-Content-Type: text/markdown
+
+# nc_db
+PostgreSQL schema, lightweight Python DB client, tests, and Kubernetes manifests for the Neural Context project.
+
+This repository provides:
+- A thin, typed Python client for JSONB-first tables (insert/update/select/delete) under a configurable schema.
+- Test suite that spins up isolated schemas to validate behavior and edge cases.
+- Kubernetes manifests (Kustomize) to run Postgres locally or in-cluster, plus a safe reset script.
+
+## Contents
+
+```
+src/db/           # Python client: Db, DbConfig, SQL helpers
+src/tests/        # Pytest suite and fixtures (isolated schemas)
+k8s/              # K8s StatefulSet, init SQL, and ops scripts
+pyproject.toml    # Poetry + pytest configuration
+.github/          # Issue and PR templates
+README.md         # This file (schema diagram below)
+```
+
+## Prerequisites
+
+- Python 3.12+
+- PostgreSQL 15+ available locally, or run via the provided K8s manifests
+
+Default development credentials used in examples/tests:
+`host=localhost, port=5432, db=nc-data, user=nc_admin, password=bob`.
+
+## Quick Start
+
+1) Install dependencies with Poetry:
+
+```bash
+poetry install
+```
+
+2) Ensure Postgres is reachable using the defaults above. You can run it via Kubernetes (see below) or your local Postgres.
+
+3) Smoke test the Python client:
+
+```bash
+poetry run python -c "from src.db.db import Db; print(Db().get('scg', ids=[]))"
+```
+
+## Running Tests
+
+```bash
+kubectl apply -k k8s/postgres/overlays/ephemeral
+kubectl -n nc_db wait --for=condition=ready pod -l app=nc-data --timeout=90s
+kubectl -n nc_db port-forward svc/postgres 5432:5432
+```
+
+```bash
+poetry run pytest -q
+```
+
+- To run a single test: `poetry run pytest -k test_update_merge -q`
+- Tests create per-test schemas and tables; they do not require pre-existing app tables.
+
+## Python Client Overview
+
+The client favors tables shaped as `(id BIGSERIAL PRIMARY KEY, body JSONB NOT NULL)` and enforces `body.id == id` on writes/reads.
+
+Basic usage:
+
+```python
+from nc_db.db.db import Db
+from nc_db.db.DbConfig import DbConfig
+
+db = Db(schema="nc", config=DbConfig(
+    user="my_user", password="my_password", host="localhost", port=5432, dbname="my_database"
+))
+
+# Insert with or without explicit id
+id1 = db.insert("scg", {"body": {"name": "alpha"}})
+id2 = db.insert("scg", {"body": {"id": 42, "name": "explicit"}})
+
+# Update (merge JSON into existing body or replace entirely)
+db.update("scg", ids=[id1], items=[{"body": {"name": "alpha-2"}}], mode="merge")
+
+# Read (optionally map rows to models)
+rows = db.get("scg")  # [{"id": 1, "body": {...}}, ...]
+
+# Delete
+db.delete("scg", [id2])
+```
+
+Key features:
+- Insert trusts a provided id if present; otherwise uses the table sequence and normalizes `body.id`.
+- Update supports `mode="merge"` (default) and `mode="replace"`, always ensuring `body.id` matches the row id.
+- Get returns rows as dictionaries or via a mapper function to your own model types.
+- Delete single or all rows, plus a helper to `TRUNCATE ... RESTART IDENTITY`.
+
+## Configuration
+
+Use `DbConfig` or environment variables to control connection pooling and DSN; defaults are suitable for local development.
+
+```python
+from nc_db.db.DbConfig import DbConfig
+
+config = DbConfig(
+    user="your_username",
+    password="your_password",
+    host="localhost",
+    port=5432,
+    dbname="your_database",
+)
+```
+OR
+
+```bash
+export PGUSER="your_username"
+export PGPASSWORD="your_password"
+export PGHOST=localhost
+export PGPORT=5432
+export PGDATABASE="youd_database"
+
+```
+
+## Kubernetes (Optional)
+
+Run a local Postgres via Kustomize overlays.
+
+Ephemeral (emptyDir) instance:
+
+```bash
+kubectl apply -k k8s/postgres/overlays/ephemeral
+kubectl -n nc_db wait --for=condition=ready pod -l app=nc-data --timeout=90s
+kubectl -n nc_db port-forward svc/postgres 5432:5432
+```
+
+Persistent storage overlay is also available under `k8s/postgres/overlays/persistent`.
+
+To safely reset a StatefulSet (DANGEROUS — wipes data), use:
+
+```bash
+python k8s/scripts/full_pod_reset.py --yes
+```
+
+Review the script before use; it scales down, deletes the PVC (and optionally PV), and scales back up.
+
+## Development Conventions
+
+- Python 3.12, type hints, PEP 8; keep imports sorted and code minimal.
+- Modules: snake_case; Classes: CamelCase; variables/functions: snake_case.
+- Tests live in `src/tests`; name files/functions `test_*` and focus on behavior.
+- See `.github/PULL_REQUEST_TEMPLATE.md` for PR expectations.
+
+## Schema Diagram
+
+Note: the init SQL currently provisions `scg` and `healthcheck`
+
+## Security & License
+
+- Do not commit real secrets. The defaults in code and K8s manifests are for local development only.
+- Use `DbConfig` or environment variables to specify connection details.
+- License: Proprietary (see `pyproject.toml`).
+

nc_db-0.1.3/nc_db/README.md

@@ -0,0 +1,156 @@
+# nc_db
+PostgreSQL schema, lightweight Python DB client, tests, and Kubernetes manifests for the Neural Context project.
+
+This repository provides:
+- A thin, typed Python client for JSONB-first tables (insert/update/select/delete) under a configurable schema.
+- Test suite that spins up isolated schemas to validate behavior and edge cases.
+- Kubernetes manifests (Kustomize) to run Postgres locally or in-cluster, plus a safe reset script.
+
+## Contents
+
+```
+src/db/           # Python client: Db, DbConfig, SQL helpers
+src/tests/        # Pytest suite and fixtures (isolated schemas)
+k8s/              # K8s StatefulSet, init SQL, and ops scripts
+pyproject.toml    # Poetry + pytest configuration
+.github/          # Issue and PR templates
+README.md         # This file (schema diagram below)
+```
+
+## Prerequisites
+
+- Python 3.12+
+- PostgreSQL 15+ available locally, or run via the provided K8s manifests
+
+Default development credentials used in examples/tests:
+`host=localhost, port=5432, db=nc-data, user=nc_admin, password=bob`.
+
+## Quick Start
+
+1) Install dependencies with Poetry:
+
+```bash
+poetry install
+```
+
+2) Ensure Postgres is reachable using the defaults above. You can run it via Kubernetes (see below) or your local Postgres.
+
+3) Smoke test the Python client:
+
+```bash
+poetry run python -c "from src.db.db import Db; print(Db().get('scg', ids=[]))"
+```
+
+## Running Tests
+
+```bash
+kubectl apply -k k8s/postgres/overlays/ephemeral
+kubectl -n nc_db wait --for=condition=ready pod -l app=nc-data --timeout=90s
+kubectl -n nc_db port-forward svc/postgres 5432:5432
+```
+
+```bash
+poetry run pytest -q
+```
+
+- To run a single test: `poetry run pytest -k test_update_merge -q`
+- Tests create per-test schemas and tables; they do not require pre-existing app tables.
+
+## Python Client Overview
+
+The client favors tables shaped as `(id BIGSERIAL PRIMARY KEY, body JSONB NOT NULL)` and enforces `body.id == id` on writes/reads.
+
+Basic usage:
+
+```python
+from nc_db.db.db import Db
+from nc_db.db.DbConfig import DbConfig
+
+db = Db(schema="nc", config=DbConfig(
+    user="my_user", password="my_password", host="localhost", port=5432, dbname="my_database"
+))
+
+# Insert with or without explicit id
+id1 = db.insert("scg", {"body": {"name": "alpha"}})
+id2 = db.insert("scg", {"body": {"id": 42, "name": "explicit"}})
+
+# Update (merge JSON into existing body or replace entirely)
+db.update("scg", ids=[id1], items=[{"body": {"name": "alpha-2"}}], mode="merge")
+
+# Read (optionally map rows to models)
+rows = db.get("scg")  # [{"id": 1, "body": {...}}, ...]
+
+# Delete
+db.delete("scg", [id2])
+```
+
+Key features:
+- Insert trusts a provided id if present; otherwise uses the table sequence and normalizes `body.id`.
+- Update supports `mode="merge"` (default) and `mode="replace"`, always ensuring `body.id` matches the row id.
+- Get returns rows as dictionaries or via a mapper function to your own model types.
+- Delete single or all rows, plus a helper to `TRUNCATE ... RESTART IDENTITY`.
+
+## Configuration
+
+Use `DbConfig` or environment variables to control connection pooling and DSN; defaults are suitable for local development.
+
+```python
+from nc_db.db.DbConfig import DbConfig
+
+config = DbConfig(
+    user="your_username",
+    password="your_password",
+    host="localhost",
+    port=5432,
+    dbname="your_database",
+)
+```
+OR
+
+```bash
+export PGUSER="your_username"
+export PGPASSWORD="your_password"
+export PGHOST=localhost
+export PGPORT=5432
+export PGDATABASE="youd_database"
+
+```
+
+## Kubernetes (Optional)
+
+Run a local Postgres via Kustomize overlays.
+
+Ephemeral (emptyDir) instance:
+
+```bash
+kubectl apply -k k8s/postgres/overlays/ephemeral
+kubectl -n nc_db wait --for=condition=ready pod -l app=nc-data --timeout=90s
+kubectl -n nc_db port-forward svc/postgres 5432:5432
+```
+
+Persistent storage overlay is also available under `k8s/postgres/overlays/persistent`.
+
+To safely reset a StatefulSet (DANGEROUS — wipes data), use:
+
+```bash
+python k8s/scripts/full_pod_reset.py --yes
+```
+
+Review the script before use; it scales down, deletes the PVC (and optionally PV), and scales back up.
+
+## Development Conventions
+
+- Python 3.12, type hints, PEP 8; keep imports sorted and code minimal.
+- Modules: snake_case; Classes: CamelCase; variables/functions: snake_case.
+- Tests live in `src/tests`; name files/functions `test_*` and focus on behavior.
+- See `.github/PULL_REQUEST_TEMPLATE.md` for PR expectations.
+
+## Schema Diagram
+
+Note: the init SQL currently provisions `scg` and `healthcheck`
+
+## Security & License
+
+- Do not commit real secrets. The defaults in code and K8s manifests are for local development only.
+- Use `DbConfig` or environment variables to specify connection details.
+- License: Proprietary (see `pyproject.toml`).

nc_db-0.1.3/nc_db/k8s/postgres/base/init/01-create-schema.sql

@@ -0,0 +1,8 @@
+CREATE SCHEMA IF NOT EXISTS nc;
+
+CREATE TABLE IF NOT EXISTS nc.scg (
+	id BIGSERIAL PRIMARY KEY,
+	body JSONB NOT NULL
+);
+
+CREATE INDEX IF NOT EXISTS idx_data_body_gin on nc.scg USING GIN (body);

nc_db-0.1.3/nc_db/k8s/postgres/base/init/02-grants.sql

@@ -0,0 +1,16 @@
+-- Revoke overly broad PUBLIC defaults and grant to the application role
+
+-- Revoke defaults for future objects
+ALTER DEFAULT PRIVILEGES IN SCHEMA "nc" REVOKE ALL ON TABLES FROM PUBLIC;
+ALTER DEFAULT PRIVILEGES IN SCHEMA "nc" REVOKE ALL ON SEQUENCES FROM PUBLIC;
+
+-- Revoke on existing objects (no-op if none)
+REVOKE ALL ON ALL TABLES IN SCHEMA nc FROM PUBLIC;
+REVOKE ALL ON ALL SEQUENCES IN SCHEMA nc FROM PUBLIC;
+
+-- Grant defaults to nc_app for future objects (owner-dependent)
+ALTER DEFAULT PRIVILEGES IN SCHEMA "nc"
+GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO nc_app;
+
+ALTER DEFAULT PRIVILEGES IN SCHEMA "nc"
+GRANT USAGE, SELECT ON SEQUENCES TO nc_app;

nc_db-0.1.3/nc_db/k8s/postgres/base/init/10-healthcheck.sql

@@ -0,0 +1,10 @@
+CREATE SCHEMA IF NOT EXISTS nc;
+
+CREATE TABLE IF NOT EXISTS nc.healthcheck (
+  id int PRIMARY KEY,
+  ok boolean NOT NULL
+);
+
+INSERT INTO nc.healthcheck (id, ok)
+VALUES (1, true)
+ON CONFLICT (id) DO NOTHING;

nc_db-0.1.3/nc_db/k8s/postgres/base/kustomization.yaml

@@ -0,0 +1,31 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: nc_db
+
+secretGenerator:
+  # Cluster/postgres superuser creds
+  - name: postgres-credentials-root
+    envs:
+      - secrets/postgres.root.env
+
+  # App/user creds (what your app uses to connect)
+  - name: postgres-credentials-app
+    envs:
+      - secrets/postgres.app.env
+
+configMapGenerator:
+  - name: pg-init-sql
+    files:
+      - init/00-create-app-role.sql
+      - init/01-create-schema.sql
+      - init/02-grants.sql
+      - init/10-healthcheck.sql
+
+# Keep the suffix hash enabled so rolling updates occur when content changes
+generatorOptions:
+  disableNameSuffixHash: false
+
+resources:
+  - namespace.yaml
+  - service.yaml
+  - statefulset.yaml

nc_db-0.1.3/nc_db/k8s/postgres/base/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: nc_db

nc_db-0.1.3/nc_db/k8s/postgres/base/pg-app-bootstrap.yaml

@@ -0,0 +1,50 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: pg-app-bootstrap
+  namespace: nc_db
+type: Opaque
+stringData:
+  20-create-app-user.sh: |
+    #!/bin/bash
+    set -euo pipefail
+
+    echo "[initdb] creating / ensuring application role ${APP_DB_USER}"
+
+    psql -v ON_ERROR_STOP=1 --username "${POSTGRES_USER}" --dbname "${POSTGRES_DB}" <<-EOSQL
+      DO \$\$
+      BEGIN
+          IF NOT EXISTS (
+              SELECT 1 FROM pg_roles WHERE rolname = '${APP_DB_USER}'
+          ) THEN
+              EXECUTE format(
+                  'CREATE ROLE %I LOGIN PASSWORD %L',
+                  '${APP_DB_USER}',
+                  '${APP_DB_PASSWORD}'
+              );
+          END IF;
+      END
+      \$\$;
+
+      -- ensure schema exists
+      CREATE SCHEMA IF NOT EXISTS ${APP_DB_SCHEMA};
+
+      -- make sure app user can actually use the DB
+      GRANT CONNECT ON DATABASE "${APP_DB_NAME}" TO ${APP_DB_USER};
+
+      GRANT USAGE ON SCHEMA ${APP_DB_SCHEMA} TO ${APP_DB_USER};
+
+      GRANT SELECT, INSERT, UPDATE, DELETE
+        ON ALL TABLES IN SCHEMA ${APP_DB_SCHEMA}
+        TO ${APP_DB_USER};
+
+      GRANT USAGE, SELECT
+        ON ALL SEQUENCES IN SCHEMA ${APP_DB_SCHEMA}
+        TO ${APP_DB_USER};
+
+      ALTER DEFAULT PRIVILEGES IN SCHEMA ${APP_DB_SCHEMA}
+        GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO ${APP_DB_USER};
+
+      ALTER DEFAULT PRIVILEGES IN SCHEMA ${APP_DB_SCHEMA}
+        GRANT USAGE, SELECT ON SEQUENCES TO ${APP_DB_USER};
+    EOSQL

nc_db-0.1.3/nc_db/k8s/postgres/base/service.yaml

@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: postgres
+spec:
+  type: ClusterIP
+  selector:
+    app: nc-data
+  ports:
+    - name: pg
+      port: 5432
+      targetPort: 5432

nc_db-0.1.3/nc_db/k8s/postgres/base/statefulset.yaml

@@ -0,0 +1,88 @@
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: postgres
+  namespace: nc_db
+spec:
+  serviceName: postgres
+  replicas: 1
+  selector:
+    matchLabels:
+      app: nc-data
+  template:
+    metadata:
+      labels:
+        app: nc-data
+    spec:
+      containers:
+        - name: postgres
+          image: postgres:15
+          imagePullPolicy: IfNotPresent
+          ports:
+            - containerPort: 5432
+              name: pg
+
+          # Load secrets into env so init script can read them
+          envFrom:
+            - secretRef:
+                name: postgres-credentials-root        # POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB
+            - secretRef:
+                name: postgres-credentials-app         # APP_DB_USER, APP_DB_PASSWORD, APP_DB_NAME, APP_DB_SCHEMA
+
+          # Non-secret config
+          env:
+            - name: PGDATA
+              value: /var/lib/postgresql/data/pgdata
+
+          # NOTE: lifecycle: postStart was intentionally removed
+
+          readinessProbe:
+            tcpSocket:
+              port: 5432
+            initialDelaySeconds: 5
+            periodSeconds: 5
+
+          livenessProbe:
+            tcpSocket:
+              port: 5432
+            initialDelaySeconds: 10
+            periodSeconds: 10
+
+          volumeMounts:
+            - name: data
+              mountPath: /var/lib/postgresql/data
+
+            # merged init dir: configmap SQL + secret script
+            - name: init-bootstrap
+              mountPath: /docker-entrypoint-initdb.d
+              readOnly: true
+
+      volumes:
+        - name: init-bootstrap
+          projected:
+            sources:
+              - configMap:
+                  name: pg-init-sql
+                  items:
+                    - key: 01-create-schema.sql
+                      path: 01-create-schema.sql
+                    - key: 02-grants.sql
+                      path: 02-grants.sql
+                    - key: 10-healthcheck.sql
+                      path: 10-healthcheck.sql
+              - secret:
+                  name: pg-app-bootstrap
+                  items:
+                    - key: 20-create-app-user.sh
+                      path: 20-create-app-user.sh
+
+  volumeClaimTemplates:
+    - metadata:
+        name: data
+      spec:
+        accessModes:
+          - ReadWriteOnce
+        storageClassName: microk8s-hostpath
+        resources:
+          requests:
+            storage: 5Gi

nc_db-0.1.3/nc_db/k8s/postgres/overlays/ephemeral/kustomization.yaml

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: nc_db
+resources:
+  - ../../base
+patches:
+  - path: patch-ephemeral-emptydir.yaml

nc_db-0.1.3/nc_db/k8s/postgres/overlays/ephemeral/patch-ephemeral-emptydir.yaml

@@ -0,0 +1,15 @@
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: postgres
+spec:
+  template:
+    spec:
+      volumes:
+        - name: init-sql
+          configMap:
+            name: pg-init-sql
+            defaultMode: 0444
+        - name: data
+          emptyDir: {}
+  volumeClaimTemplates: []  # remove PVCs for ephemeral runs

nc_db-0.1.3/nc_db/k8s/postgres/overlays/persistent/kustomization.yaml

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: nc_db
+resources:
+  - ../../base
+patches:
+  - path: patch-persistent-storageclass.yaml

nc_db-0.1.3/nc_db/k8s/postgres/overlays/persistent/patch-persistent-storageclass.yaml

@@ -0,0 +1,17 @@
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: postgres
+spec:
+  volumeClaimTemplates: []   # disable PVC template from base
+  template:
+    spec:
+      volumes:
+        - name: data
+          hostPath:
+            path: /home/chale/nc_db/postgress-store/
+            type: DirectoryOrCreate
+        - name: init-sql
+          configMap:
+            name: pg-init-sql
+            defaultMode: 0444