@talkpilot/core-db 1.2.1 → 1.2.2

package/.cursor/rules/development.mdc

@@ -1,65 +1,65 @@
----
-description: Development standards and conventions for the core-db package
-globs: src/**/*.ts
-alwaysApply: true
----
-
-# Development Standards
-
-This package provides a centralized database layer for multiple domains (TalkPilot and Municipal). Follow these rules to maintain consistency and reliability.
-
-## Project Structure
-
-- **`src/talkpilot/`**: All database logic, types, and getters related to the TalkPilot domain.
-- **`src/municipal/`**: All database logic, types, and getters related to the Municipal Data domain.
-- **`src/test-utils/`**: Shared testing infrastructure, including factories and database utilities.
-
-## Database Connection Pattern
-
-Each domain is isolated. They have their own `db` instance and must be connected independently.
-
-```typescript
-import { mongodbClient, municipalDataMongodbClient } from '@talkpilot/core-db';
-
-// TalkPilot
-await mongodbClient.connect(uri);
-
-// Municipal
-await municipalDataMongodbClient.connect(uri);
-```
-
-## Creating New Getters
-
-1. **Location**: Place getters in the relevant domain folder (e.g., `src/talkpilot/agents/agents.getters.ts`).
-2. **Naming**: Use `find...` for multiple results and `get...ById` for single results.
-3. **Domain isolation**: Always use the `getDb()` function from the current domain's `index.ts`.
-
-## Environment Validation
-
-Always validate configuration "on-demand" within the `connect` methods using the validation utilities.
-
-```typescript
-import { validateConfig, validateMongoUri } from '../utils/validation';
-
-async connect(uri?: string) {
-  const mongodbUri = uri || process.env.MONGO_URI;
-  validateConfig('MONGO_URI', mongodbUri);
-  validateMongoUri(mongodbUri!);
-  // ... connection logic
-}
-```
-
-## Testing Standards
-
-1. **In-Memory DB**: Use `mongodb-memory-server` for all tests. It is automatically initialized in `src/__tests__/setup.ts`.
-2. **Factories**: Use the Fishery factories in `src/test-utils/factories/` to generate test data.
-3. **Organization**: Place tests in a `__tests__` folder within the relevant domain.
-
-```typescript
-import { createAgent } from '../../../test-utils/factories';
-
-it('should find agents', async () => {
-  const agent = createAgent({ name: 'Test' });
-  // ... test logic
-});
-```
+---
+description: Development standards and conventions for the core-db package
+globs: src/**/*.ts
+alwaysApply: true
+---
+
+# Development Standards
+
+This package provides a centralized database layer for multiple domains (TalkPilot and Municipal). Follow these rules to maintain consistency and reliability.
+
+## Project Structure
+
+- **`src/talkpilot/`**: All database logic, types, and getters related to the TalkPilot domain.
+- **`src/municipal/`**: All database logic, types, and getters related to the Municipal Data domain.
+- **`src/test-utils/`**: Shared testing infrastructure, including factories and database utilities.
+
+## Database Connection Pattern
+
+Each domain is isolated. They have their own `db` instance and must be connected independently.
+
+```typescript
+import { mongodbClient, municipalDataMongodbClient } from '@talkpilot/core-db';
+
+// TalkPilot
+await mongodbClient.connect(uri);
+
+// Municipal
+await municipalDataMongodbClient.connect(uri);
+```
+
+## Creating New Getters
+
+1. **Location**: Place getters in the relevant domain folder (e.g., `src/talkpilot/agents/agents.getters.ts`).
+2. **Naming**: Use `find...` for multiple results and `get...ById` for single results.
+3. **Domain isolation**: Always use the `getDb()` function from the current domain's `index.ts`.
+
+## Environment Validation
+
+Always validate configuration "on-demand" within the `connect` methods using the validation utilities.
+
+```typescript
+import { validateConfig, validateMongoUri } from '../utils/validation';
+
+async connect(uri?: string) {
+  const mongodbUri = uri || process.env.MONGO_URI;
+  validateConfig('MONGO_URI', mongodbUri);
+  validateMongoUri(mongodbUri!);
+  // ... connection logic
+}
+```
+
+## Testing Standards
+
+1. **In-Memory DB**: Use `mongodb-memory-server` for all tests. It is automatically initialized in `src/__tests__/setup.ts`.
+2. **Factories**: Use the Fishery factories in `src/test-utils/factories/` to generate test data.
+3. **Organization**: Place tests in a `__tests__` folder within the relevant domain.
+
+```typescript
+import { createAgent } from '../../../test-utils/factories';
+
+it('should find agents', async () => {
+  const agent = createAgent({ name: 'Test' });
+  // ... test logic
+});
+```

package/DEVELOPMENT.md

@@ -1,98 +1,98 @@
-# Development Guide
-
-Welcome to the `core-db` development guide. This document explains how to set up, develop, and test this package.
-
-## Getting Started
-
-### Prerequisites
-
-- Node.js (v18 or later)
-- TypeScript
-
-### Setup
-
-1. Install dependencies:
-   ```bash
-   npm install
-   ```
-
-2. Build the project:
-   ```bash
-   npm run build
-   ```
-
-## Local Development
-
-If you want to use this package in another project locally without publishing it:
-
-1. In the `core-db` folder:
-   ```bash
-   npm link
-   ```
-
-2. In your PROJECT folder:
-   ```bash
-   npm link @talkpilot/core-db
-   ```
-
-## Adding a New Getter
-
-1. **Define Types**: Add your data types in the relevant domain's `types.ts` file.
-2. **Implement Getter**: Add the function in the `getters.ts` file.
-3. **Export**: Ensure the getter is exported from the domain's `index.ts` and finally from the main `src/index.ts`.
-4. **Test**: Create a test in the domain's `__tests__` folder.
-
-## Testing
-
-We use Jest with `mongodb-memory-server` for fast, isolated database tests.
-
-### Running Tests
-
-```bash
-npm test
-```
-
-### Using Factories
-
-Always use factories to generate test data to keep tests clean and maintainable.
-
-```typescript
-import { createCallDoc } from '../calls.getters';
-import { createOutGoingCallDoc } from '../../../test-utils/factories';
-
-it('should save a call', async () => {
-  const call = createOutGoingCallDoc({ callSid: 'CA123' });
-  await createCallDoc(call);
-  // ... assertions
-});
-```
-
-## Build Process
-
-The project is built using TypeScript (`tsc`). The output is generated in the `dist/` directory.
-
-- `main`: `dist/index.js`
-- `types`: `dist/index.d.ts`
-
-The `prepare` script in `package.json` ensures that the project is built automatically when installed via a Git URL.
-
-## Team Access & Authentication
-
-To allow the whole team to publish and install without adding individual npm accounts, we use a shared **npm Granular Access Token**.
-
-### One-time Local Setup
-
-Each developer needs to add the shared token to their local npm configuration. Do **NOT** add this to the project's `.npmrc` file, as it will be committed to Git.
-
-1. Get the shared **npm Automation Token**.
-2. Open (or create) your global npm configuration file:
-   ```bash
-   nano ~/.npmrc
-   ```
-3. Add the following line (replace `[TOKEN]` with the actual token):
-   ```text
-   //registry.npmjs.org/:_authToken=[TOKEN]
-   ```
-4. Save and exit.
-
-Now you can run `npm publish` and `npm install` for scoped `@talkpilot` packages without being prompted for credentials.
+# Development Guide
+
+Welcome to the `core-db` development guide. This document explains how to set up, develop, and test this package.
+
+## Getting Started
+
+### Prerequisites
+
+- Node.js (v18 or later)
+- TypeScript
+
+### Setup
+
+1. Install dependencies:
+   ```bash
+   npm install
+   ```
+
+2. Build the project:
+   ```bash
+   npm run build
+   ```
+
+## Local Development
+
+If you want to use this package in another project locally without publishing it:
+
+1. In the `core-db` folder:
+   ```bash
+   npm link
+   ```
+
+2. In your PROJECT folder:
+   ```bash
+   npm link @talkpilot/core-db
+   ```
+
+## Adding a New Getter
+
+1. **Define Types**: Add your data types in the relevant domain's `types.ts` file.
+2. **Implement Getter**: Add the function in the `getters.ts` file.
+3. **Export**: Ensure the getter is exported from the domain's `index.ts` and finally from the main `src/index.ts`.
+4. **Test**: Create a test in the domain's `__tests__` folder.
+
+## Testing
+
+We use Jest with `mongodb-memory-server` for fast, isolated database tests.
+
+### Running Tests
+
+```bash
+npm test
+```
+
+### Using Factories
+
+Always use factories to generate test data to keep tests clean and maintainable.
+
+```typescript
+import { createCallDoc } from '../calls.getters';
+import { createOutGoingCallDoc } from '../../../test-utils/factories';
+
+it('should save a call', async () => {
+  const call = createOutGoingCallDoc({ callSid: 'CA123' });
+  await createCallDoc(call);
+  // ... assertions
+});
+```
+
+## Build Process
+
+The project is built using TypeScript (`tsc`). The output is generated in the `dist/` directory.
+
+- `main`: `dist/index.js`
+- `types`: `dist/index.d.ts`
+
+The `prepare` script in `package.json` ensures that the project is built automatically when installed via a Git URL.
+
+## Team Access & Authentication
+
+To allow the whole team to publish and install without adding individual npm accounts, we use a shared **npm Granular Access Token**.
+
+### One-time Local Setup
+
+Each developer needs to add the shared token to their local npm configuration. Do **NOT** add this to the project's `.npmrc` file, as it will be committed to Git.
+
+1. Get the shared **npm Automation Token**.
+2. Open (or create) your global npm configuration file:
+   ```bash
+   nano ~/.npmrc
+   ```
+3. Add the following line (replace `[TOKEN]` with the actual token):
+   ```text
+   //registry.npmjs.org/:_authToken=[TOKEN]
+   ```
+4. Save and exit.
+
+Now you can run `npm publish` and `npm install` for scoped `@talkpilot` packages without being prompted for credentials.

package/README.md

@@ -1,160 +1,139 @@
-# @talkpilot/core-db
-
-[NPM Version](https://www.npmjs.com/package/@talkpilot/core-db)
-
-A TypeScript-based core database management package designed to provide centralized database connections, ORM integrations, and client utilities for other projects. This package manages connections to both **TalkPilot** and **Municipal** MongoDB databases.
-
-## Features
-
-- 🚀 **Multi-Domain Database Management**: Centralized handlers for both TalkPilot and Municipal Data domains.
-- 📦 **Reusable Getters**: Standardized functions for fetching data from various collections (calls, agents, leads, cities, etc.).
-- 🏗️ **Type-safe**: Built with TypeScript for full type safety across all your database interactions.
-- 🛠️ **Environment Aware**: Configurable via environment variables or explicit parameters.
-
-## Installation
-
-This package is intended to be used as a dependency in other projects. If you're working locally, you can install it using a file path:
-
-```bash
-# In your other project:
-npm install /path/to/core-db
-# or if published to a registry:
-npm install @talkpilot/core-db
-# or via GitHub
-npm install github:talkpilot/core-db
-```
-
-## Quick Start
-
-### 1. Initialize Database Connections
-
-You must initialize the database clients before using any of the getters.
-
-```typescript
-import { mongodbClient, municipalDataMongodbClient } from '@talkpilot/core-db';
-
-async function bootstrap() {
-  // Initialize TalkPilot DB
-  await mongodbClient.connect(process.env.TALKPILOT_MONGO_URI);
-
-  // Initialize Municipal Data DB (optional)
-  await municipalDataMongodbClient.connect(process.env.MUNICIPAL_MONGO_URI);
-}
-
-bootstrap().catch(console.error);
-```
-
-### 2. Using Getters
-
-Once initialized, you can import and use any of the exported database getters.
-
-```typescript
-import { findAgents, findCities } from '@talkpilot/core-db';
-
-async function getSummary() {
-  // Get all agents from TalkPilot DB
-  const agents = await findAgents();
-  console.log('Total Agents:', agents.length);
-
-  // Get all cities from Municipal Data DB
-  const cities = await findCities();
-  console.log('Total Cities:', cities.length);
-}
-```
-
-## Available Domains
-
-The package exports two main sets of tools:
-
-### TalkPilot Domain
-
-Includes access to:
-
-- `agents`, `calls`, `clients`, `flows`, `leads`, `phone_numbers`, `plans`, `results`, `sessions`, `subscriptions`, etc.
-- Exported as root level functions or via `mongodbClient`.
-
-### Municipal Domain
-
-Includes access to:
-
-- `cities`, `streets`, `departmentsSubjects`, `tickets`.
-- Functions are prefixed where necessary or accessible via `getMunicipalDataDb`.
-
-## Environment Variables
-
-The clients will automatically attempt to use the following environment variables if no URI is provided to the `connect()` method:
-
-- `MONGO_URI` or `MONGODB_URI`: Primary MongoDB connection string.
-- `MONGODB_DB_NAME`: Database name (defaults to 'municipal-data' for the municipal client if not specified).
-
-## Development
-
-### Setup
-
-1. Clone the repository and install dependencies:
-  ```bash
-   git clone https://github.com/talkpilot/core-db.git
-   cd core-db
-   npm install
-  ```
-2. Build the project:
-  ```bash
-   npm run build
-  ```
-
-This will generate the `dist/` directory with the compiled JavaScript and type definitions.
-
-### Publishing New Versions
-
-To publish a new version of the package:
-
-1. **Test your changes**:
-  ```bash
-   npm test
-  ```
-2. **Update the version**:
-  ```bash
-   # Bumps patch version (0.1.0 -> 0.1.1)
-   npm version patch
-   # Or for minor changes (0.1.0 -> 0.2.0)
-   # npm version minor
-   # Or for major changes (0.1.0 -> 1.0.0)
-   # npm version major
-  ```
-3. **Publish to npm**:
-  Ensure you have the shared team token in your `~/.npmrc` (see [Development Guide](./DEVELOPMENT.md#team-access--authentication)).
-4. **Update in other repos**:
-  ```bash
-   npm update @talkpilot/core-db
-  ```
-
-## License
-
-MIT
-
----
-
-**Interested in contributing?** Check out our [Development Guide](./DEVELOPMENT.md) for standards and setup instructions.
-
-## Release Notes
-
-### 1.1.9
-
-- `getPhoneNumbersForFlows(clientId)` — lists each phone number linked to a flow for that client (newest first), with `flowId`, `phoneNumber`, and `isPrimary`.
-### 1.1.5
-
-- `createPurchasedPhoneNumber` for API-bought numbers (Twilio/Telnyx metadata); optional `flowId` until linked to a flow. Types/schema updated accordingly.
-
-### 1.0.16
-
-- Fixed critical issues present in version 1.0.15.
-- **Warning**: Version 1.0.15 is bugged and should not be used. Please use at least version 1.0.16.  
-
-### 1.0.20
-
-- Added to Moked106Config this param: withNeedAttentionCalls
-- withNeedAttentionCalls?: boolean
-
-### 1.0.27
-- Added an optional `language` field to `ClientConfig` (e.g. Hebrew/English/Russian/French/Spanish) and updated tests.
-
+L1:# @talkpilot/core-db
+
+`@talkpilot/core-db` is the shared TypeScript database package that wires TalkPilot APIs, municipal CRM integrations, and internal tools to a single, type-safe MongoDB surface. Every repo (MIS, CIS, TalkPilot Server, etc.) imports this package to avoid re-implementing connections, collections, or validation helpers.
+
+## Purpose
+
+- Provide a reliable, multi-domain MongoDB layer for TalkPilot and municipal data.
+- Export typed getters, vector search helpers, and document factories so services can focus on behavior instead of schema wiring.
+- Manage connection lifecycles, environment configuration, and test helpers from one place so every repo reuses the same plumbing.
+
+## Main Concepts
+
+- **Multi-domain clients** – `src/connection.ts` exposes `mongodbClient` (TalkPilot) and `municipalDataMongodbClient`, each of which resolves `MONGO_URI`, DB overrides, and default names.
+- **Domain-specific getters** – `src/talkpilot/` and `src/municipal/` host typed getters (agents, calls, streets, tickets, etc.), vector-search helpers, and service-friendly adapters that keep caller code DRY.
+- **Product-specific clients (future)** – While the package currently exposes the shared TalkPilot + municipal clients, we expect each product (CIS, MIS, TalkPilot Server) to eventually get its own domain-specific client helpers or wrappers so the shared core can remain stable while new consumers add targeted extensions.
+- **Test helpers** – `src/test-utils/` plus `src/__tests__/` reuse `MongoMemoryServer` and shared factories so tests start with clean data regardless of the consuming repo.
+- **Utility layers** – `src/utils/` contains shared validation, pagination, and environment helpers that complement the getters.
+- **Environment awareness** – Defaults, fallbacks, and `process.env` lookups ensure local, CI, and Cloud Run clients all connect using the right URI/DB names.
+
+## Key Components
+
+- `src/connection.ts` – Central connection logic that resolves URIs/DB names from env vars (`MONGO_URI`, `MONGODB_URI`, `TALKPILOT_DB_NAME`, `MUNICIPAL_DB_NAME`) and reuses a single `MongoClient`.
+- `src/talkpilot/` – Call history, agents, flows, sessions, leads, subscriptions, and support helpers exposed as getters plus helper enums/types for each collection.
+- `src/municipal/` – Municipal-specific collections (`cities`, `streets`, `departmentsSubjects`, `tickets`, etc.) plus vector search helpers and Ash Bina helpers used by MIS.
+- `src/utils/` – Shared helpers such as `resolveConnection`, pagination utilities, and schema validation helper functions.
+- `src/test-utils/` and `src/__tests__/` – Utilities that bootstrap `MongoMemoryServer`, expose factories, and make sure Jest environments can stub database calls predictably.
+- `dist/` – Compiled output consumed by downstream repos (CJS + ESM + type defs).
+
+## Domain APIs
+
+- **TalkPilot domain** – Imports like `findAgents`, `getFlows`, `findCalls`, and `vectorSearchCalls` live in `src/talkpilot`. These functions are the canonical access pattern for call history, session metadata, and provider configs.
+- **Municipal domain** – Helpers such as `findStreets`, `getMunicipalCities`, `findDepartmentSubjects`, and `createTicket` live under `src/municipal` and feed MIS workflows (street hints, subject matching, Ash Bina tickets).
+
+## Environment variables
+
+| Variable                  | Purpose                                                                              | Required |
+|---------------------------|--------------------------------------------------------------------------------------|----------|
+| `MONGO_URI`               | Primary MongoDB connection string for every domain (overridden by `MONGODB_URI`).     | ✅        |
+| `MONGODB_URI`             | Alternate connection string used when Mongo needs a second URI parameter.             | ✅        |
+| `TALKPILOT_DB_NAME`        | Optional override for the TalkPilot database name (defaults from URI path).          | ❌        |
+| `MUNICIPAL_DB_NAME`        | Optional override for the municipal database name (defaults to `municipal-data`).     | ❌        |
+| `ENV`                     | Free-form label used in logs/validators (defaults to `unknown`).                      | ❌        |
+
+If you pass a `uri` directly to `mongodbClient.connect()` or `municipalDataMongodbClient.connect()`, the client will prefer that value over the env vars.
+
+## Getting Started
+
+### Prerequisites
+
+- Node.js 22.x+ (aligns with downstream services).
+- npm 11+ or Yarn.
+- MongoDB accessible from your environment or a `MongoMemoryServer` for tests.
+
+### Setup
+
+1. Clone the repo and install dependencies:
+
+   ```bash
+   git clone https://github.com/talkpilot/core-db.git
+   cd core-db
+   npm install
+   ```
+
+2. Build the package before using it locally:
+
+   ```bash
+   npm run build
+   ```
+
+3. Import `@talkpilot/core-db` from another project by pointing `package.json` at the local path during development or installing the published release.
+
+## Sample `.env`
+
+```
+MONGO_URI=mongodb://localhost:27017
+TALKPILOT_DB_NAME=talkpilot-dev
+MUNICIPAL_DB_NAME=municipal-dev
+ENV=development
+```
+
+Adjust `MONGO_URI` to match the running Mongo instance and configure `talkpilot`/`municipal` DB names if you want to keep them separate.
+
+## Local development
+
+1. Run `npm install`.
+2. Build the compiled output: `npm run build`.
+3. Execute tests: `npm run test`.
+4. Use `npm link` or `npm pack` to consume the freshly built package from other repos (`CIS`, `MIS`, `TalkPilot Server`).
+
+## Development guide
+
+`DEVELOPMENT.md` contains the tactical steps for contributors. At a glance:
+
+- Node 18+/TypeScript is required (aligns with downstream services).
+- Run `npm install` → `npm run build` after cloning.
+- Use `npm link`/`npm link @talkpilot/core-db` to test the package locally before publishing.
+- When adding getters, define types, implement the function, export it through the domain `index.ts`, and add a corresponding test under the domain’s `__tests__` folder.
+- Always rely on the provided test factories (`src/test-utils/factories`) to seed data so tests remain consistent.
+- Jest with `mongodb-memory-server` is the only execution path we have to verify this core utility—unit tests are the safety net for every change.
+
+Refer to `DEVELOPMENT.md` for the full walkthrough, token instructions, and factory samples.
+
+## 🧪 Testing
+
+- `npm run test` – Jest suite (factories, utils, integration mocks) powered by `mongodb-memory-server`.
+- Tests rely on `src/__tests__/setup.ts` to bootstrap the in-memory Mongo instances and wire shared factories/helpers before each run.
+- When adding getters, helpers, or domain logic, create focused coverage inside the consuming domain’s `__tests__/` folder and use the provided factories to keep fixtures consistent.
+
+`@talkpilot/core-db` does not run in a product UI or feature branch—unit tests are the *only* reliable execution path to ensure your changes work. Every change must ship with a unit test so downstream repos can upgrade without surprises; treat the test suite as the canonical safety net for this core utility package.
+
+## 🧹 Lint & build verification
+
+- `npm run lint` – Run ESLint over `src/**/*.{ts,tsx}`.
+- `npm run format` – Format the source files with Prettier.
+- `npm run build` – Compile TypeScript and emit `dist/` (used by downstream consumers).
+
+## ✅ Pre-push checklist
+
+1. `npm run build`.
+2. `npm run test`.
+3. `npm run format`.
+
+## Publishing & release notes
+
+- Releases are handled by `npm version <patch|minor|major>` followed by `npm publish`. The package is a **private `@talkpilot` dependency**, so every contributor must install the shared npm automation token into their global `~/.npmrc` before running publish or `npm install`.
+- The shared token is rotated periodically—if you see authentication failures, request the refreshed token, update your `~/.npmrc`, and retry. Never commit credentials to source control.
+- After publishing, downstream repos (`CIS`, `MIS`, `TalkPilot Server`, etc.) should run `npm update @talkpilot/core-db` so they receive the latest helpers/bug fixes.
+- Cloud Build & Cloud Run jobs that depend on this package pick up the new version the next time they rebuild their container; the services pull the compiled `dist/` output and type definitions when installing the dependency.
+
+## 🛠 CI/CD & deployment
+
+- This package is consumed by Cloud Build-based services (TalkPilot Server, MIS, CIS) as a dependency when Docker images are built. Keep `dist/` in sync with your builds because the compiled artifact is what downstream services install.
+- Releases require the shared npm token documented in `DEVELOPMENT.md`; consult that guide for contribution, linking, and token rotation procedures.
+
+## Resources
+
+- [DEVELOPMENT.md](./DEVELOPMENT.md) (setup, tooling, publishing, token management)
+- [src/test-utils](src/test-utils) and the `__tests__` folder for examples of `MongoMemoryServer` wiring.
+