@badger_badger/badger_hq 0.0.0-stage → 1.5.1

package/CHANGELOG.md

@@ -0,0 +1,101 @@
+# badger_hq
+
+## 1.4.0
+
+### Minor Changes
+
+- Badger Agent runtime contracts.
+
+  - New entities and collections: `AgentSession` (`agentSessions`), `AgentMessage` (`agentMessages`, with the `AgentCard` discriminated union), `Capture` (`captures`, screenshot + read-only DOM snapshot refs, focus region, retention `expiresAt`) and `LlmCall` (`llmCalls`).
+  - New wire contracts in `zod/agent.ts`: session views, `AgentSessionStartBody`, `AgentReplyBody`, `AgentCaptureFields`, `AgentCaptureResponse`, `AgentStreamEvent` (SSE), `CaptureDomSnapshot`; new `AgentErrorCode` values (`FORBIDDEN`, `CONFLICT`, `SESSION_NOT_ACTIVE`, `PAYLOAD_TOO_LARGE`, `BUDGET_EXCEEDED`, `LLM_UNAVAILABLE`, `UPSTREAM_UNAVAILABLE`).
+  - New `zod/internal.ts`: service-to-service contracts for hero-service `/internal/sources/intake`, `/internal/sources/:id/visit`, `/internal/data-items/priority`.
+  - New enums: `AGENT_SESSION_MODE`, `AGENT_SESSION_STATUS`, `AGENT_MESSAGE_ROLE`, `AGENT_MESSAGE_KIND`, `AGENT_CARD_KIND`, `AGENT_SOURCE_OUTCOME`, `CAPTURE_STATUS`, `CAPTURE_DOM_STATUS`, `LLM_CAPABILITY`, `LLM_PROVIDER`, `LLM_OUTCOME`, `SOURCE_INTAKE_OUTCOME`.
+  - `SOURCE_STATUS_VALUES` gains `access_problem` (monitor visit hit a login wall; parked for review).
+  - `DataItem.contentHash?` with a partial unique index `(orgId, sourceId, contentHash)`.
+
+## 1.2.0
+
+### Minor Changes
+
+- 3740d46: Add Badger Agent wire contracts (`src/zod/agent.ts`): `AgentRole`, `AgentMeSchema`, `AgentTaskSchema`,
+  `AgentAssignmentItemSchema`, `AgentAssignmentsResponseSchema`, `AgentAssignmentVersionsResponseSchema`,
+  `AgentErrorSchema`. These are read models shared by `agent-service` and the Chrome extension.
+- 404cbc3: Initial release (0.1.0) of the shared Harvester V2 schema package.
+
+  - Browser-safe entry `.`: Zod schemas, inferred types, enums, `COLLECTIONS`, pagination and `ApiError` helpers.
+  - Server entry `./mongo`: Mongoose schemas typed against the Zod types, index definitions, `registerModels(connection)`,
+    `incrementTaskVersion` hook and the `objectIdsToStrings` `toJSON` transform.
+  - Entities: Mission, Task, Assignment, HoursLog, ScoutProfile, Source, SourceImport, DataItem, Scraper, ScraperMetric,
+    Deployment, AuditLog. Identity (users, orgs, memberships, roles) is owned by Clerk and referenced by id strings only.
+
+- Add `fullName` to `ScoutProfile`: the scout's real name, System- and self-only (organizations keep seeing
+  `displayName`, which may be an alias). Optional on stored profiles, required in `ScoutProfileCreateSchema`,
+  editable through `ScoutSelfUpdateSchema`.
+
+## 1.0.0
+
+### Major Changes
+
+- Scouts and analysts as assignable people; multiple assignees per task.
+
+  **Changed**
+
+  - `ScoutProfile`: `availability { tz, windows }` became `timezone` (IANA string) + `availability` (array of weekly
+    windows); `platforms` became `platformAccess { all, platforms, special[] }`; the single `rating` became
+    `ratingSummary { avg, count }` (denormalized from `scoutRatings`). New fields: `email` (unique, lowercased),
+    `status` (`active | disabled`), `systemRecommendation { taskKinds, tags, note, updatedBy, updatedAt }`.
+    `ScoutProfileCreateSchema` now includes `email` and `vettingStatus`; `ScoutSelfUpdateSchema` lists what a scout
+    may edit about themselves.
+  - `Assignment`: the partial unique index on `{ taskId }` (status `active`) is **removed**; a task may now hold many
+    active assignees. New partial unique index `{ taskId, assigneeUserId }` (status `active`), new index
+    `{ assigneeUserId, status, orgId }` (replaces `{ assigneeUserId, status }`), new optional `revokedBy`.
+
+  **Added**
+
+  - `AnalystProfile` (`analystProfiles`, unique `{ orgId, userId }`), `OrgSettings` (`orgSettings`, unique `{ orgId }`,
+    `DEFAULT_ANALYST_TASK_LIMIT = 5`), `ScoutRating` (`scoutRatings`), `AssigneeLoad` (`assigneeLoads`, unique
+    `{ userId, scope }`).
+  - Shared matching fragments: `MatchingFieldsSchema`, `AvailabilityWindowSchema`, `PlatformAccessSchema`,
+    `SpecialAccessSchema`, `TimeZone`; `RatingScore`, `RatingSummarySchema`, `SystemRecommendationSchema`, `Email`,
+    `LoadScope` / `GLOBAL_LOAD_SCOPE`, `MAX_RATING_COMMENT_LENGTH`.
+  - Enums `ScoutStatus` (`SCOUT_STATUS_VALUES`) and `RaterType` (`RATER_TYPE_VALUES`).
+  - `ObjectIdKey` includes `assignmentId`.
+
+  **Migration**
+
+  1. `scoutProfiles`: for each document set `timezone = availability.tz`, `availability = availability.windows`,
+     `platformAccess = { all: false, platforms, special: [] }`, `ratingSummary = { avg: rating ?? null, count: rating ? 1 : 0 }`,
+     `status = 'active'`, `email` = the Clerk user's primary email (lowercased); unset `platforms` and `rating`.
+  2. `assignments`: drop the index on `{ taskId: 1 }` with `partialFilterExpression { status: 'active', deletedAt: null }`
+     and the index `{ assigneeUserId: 1, status: 1 }`, then `syncIndexes()`.
+  3. Seed `assigneeLoads` from active assignments: for scouts `{ userId, scope: 'global', active }`, for analysts
+     `{ userId, scope: orgId, active }`.
+  4. `orgSettings` and `analystProfiles` are created lazily by hero-service (Clerk webhooks); no backfill needed.
+
+### Minor Changes
+
+- d6e348c: Add the `Database` entity: one per organization, holding the `name` and `connectionString` of the org's Mongo
+  database (where `sources`, `dataItems`, … live). Ships `DatabaseSchema` / `DatabaseCreateSchema` /
+  `DatabaseUpdateSchema`, the `DatabaseStatus` and `DatabaseCollection` enums, `COLLECTIONS.databases`, the
+  `databaseSchema` Mongoose schema with a unique partial index on `orgId`, and the `Database` model in
+  `registerModels()`.
+- Add a server-managed `databaseId` to `Source` and `DataItem` (new `DatabaseId` scalar in `common`). All
+  organizations' collected data lives in the shared service database, partitioned by `(orgId, databaseId)`;
+  today `databaseId` equals the Clerk org id, and it becomes the id of a physical `Database` record once orgs are
+  split into their own databases. `databaseId` is omitted from the `*CreateSchema`s and listed in
+  `SERVER_MANAGED_KEYS`; the `sources` and `dataItems` collections gain an `{ orgId, databaseId }` index. No
+  `Database` records are created any more — the entity is kept for the future split.
+- 404cbc3: Initial release (0.1.0) of the shared Harvester V2 schema package.
+
+  - Browser-safe entry `.`: Zod schemas, inferred types, enums, `COLLECTIONS`, pagination and `ApiError` helpers.
+  - Server entry `./mongo`: Mongoose schemas typed against the Zod types, index definitions, `registerModels(connection)`,
+    `incrementTaskVersion` hook and the `objectIdsToStrings` `toJSON` transform.
+  - Entities: Mission, Task, Assignment, HoursLog, ScoutProfile, Source, SourceImport, DataItem, Scraper, ScraperMetric,
+    Deployment, AuditLog. Identity (users, orgs, memberships, roles) is owned by Clerk and referenced by id strings only.
+
+### Patch Changes
+
+- d6e348c: Extend the `Platform` enum (`PLATFORM_VALUES`) with `instagram`, `youtube`, `whatsapp`, `reddit`, `discord`,
+  `linkedin`, `threads`, `vk`, `snapchat` and `web` (open web: sites, forums, blogs). Existing values are unchanged.
+
+Release notes are generated from `.changeset/` by `changeset version`.

package/README.md

@@ -1,3 +1,276 @@
-# Temporary Holding Version
+# badger_hq
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Shared types and schemas for **Harvester V2** (Badger HQ). Every service imports it: the Next.js web app,
+the Chrome extension, the Fastify core API, the worker, the agent and the scrapers.
+
+It ships two entry points:
+
+| Entry | Import | Contents | Safe in |
+|---|---|---|---|
+| `.` | `badger_hq` | Zod schemas, inferred TS types, enums, `COLLECTIONS`, pagination / `ApiError` helpers | Browser **and** Node |
+| `./mongo` | `badger_hq/mongo` | Mongoose schemas, index definitions, `registerModels(connection)` | Node only |
+
+Nothing reachable from `badger_hq` imports `mongoose`, `bson` or a Node built-in; a test bundles it with esbuild
+for `platform: 'browser'` and fails if that ever changes.
+
+## Install
+
+The package is published publicly to the npm registry.
+
+```sh
+npm install badger_hq zod
+# server-side packages also need:
+npm install mongoose
+```
+
+`zod` (v3) is a required peer dependency. `mongoose` (v8) is an **optional** peer: browser consumers don't install it.
+
+## Entry point `.` — Zod + types
+
+```ts
+import {
+  MissionCreateSchema,
+  type MissionCreate,
+  type Mission,
+  Platform,
+  PLATFORM_VALUES,
+  PaginationQuerySchema,
+  type Paginated,
+  type ApiError,
+} from 'badger_hq';
+```
+
+Per entity (`Mission` shown; the same pattern applies to every entity):
+
+| Export | Meaning |
+|---|---|
+| `MissionSchema` | Full document as it comes back from the API (`_id`, `orgId`, timestamps, …). |
+| `MissionCreateSchema` | Body for `POST`. Omits `_id`, `orgId`, `createdAt`, `updatedAt`, `deletedAt` and server-set fields (e.g. `createdBy`). |
+| `MissionUpdateSchema` | Body for `PATCH`. `MissionCreateSchema.partial()`. |
+| `Mission`, `MissionCreate`, `MissionUpdate` | `z.infer` of the above. |
+| `MissionCreateInput` | `z.input` — the pre-default shape, useful for form state (only where Create has defaults). |
+
+Dates use `z.coerce.date()`, so JSON ISO strings parse to `Date`.
+
+### Example: React Hook Form resolver
+
+```tsx
+import { useForm } from 'react-hook-form';
+import { zodResolver } from '@hookform/resolvers/zod';
+import { MissionCreateSchema, type MissionCreateInput, MISSION_PRIORITY_VALUES } from 'badger_hq';
+
+export function NewMissionForm() {
+  const form = useForm<MissionCreateInput>({
+    resolver: zodResolver(MissionCreateSchema),
+    defaultValues: {
+      title: '',
+      description: '',
+      priority: 'medium',
+      keywords: [],
+      defaultCriteria: { languages: [], platforms: [], regions: [], regionMode: 'preferred', emergency: false },
+    },
+  });
+
+  return (
+    <form onSubmit={form.handleSubmit((values) => fetch('/api/missions', { method: 'POST', body: JSON.stringify(values) }))}>
+      <input {...form.register('title')} />
+      <select {...form.register('priority')}>
+        {MISSION_PRIORITY_VALUES.map((p) => <option key={p}>{p}</option>)}
+      </select>
+      {/* … */}
+    </form>
+  );
+}
+```
+
+### Example: parsing an API response
+
+```ts
+import { PaginatedSchema, TaskSchema } from 'badger_hq';
+
+const PageOfTasks = PaginatedSchema(TaskSchema);
+const page = PageOfTasks.parse(await res.json()); // page.items[0].createdAt is a Date
+```
+
+## Entry point `./mongo` — Mongoose
+
+```ts
+import { registerModels, COLLECTIONS, INDEXES, incrementTaskVersion, type HarvesterModels } from 'badger_hq/mongo';
+```
+
+The package **never connects**. You own the `mongoose.Connection`; `registerModels(conn)` registers the models on it
+(idempotently — it reuses `conn.models[name]` if present) and returns them fully typed:
+
+```ts
+{ Database, Mission, Task, Assignment, HoursLog, ScoutProfile, AnalystProfile, OrgSettings, ScoutRating, AssigneeLoad,
+  Source, SourceImport, DataItem, Scraper, ScraperMetric, Deployment, AuditLog }
+```
+
+People who can be assigned to tasks:
+
+- **`ScoutProfile`** (global, not org-scoped) — a freelancer shared by every organization. Identity is the Clerk user
+  (`userId`, `email` mirror); the profile holds the matching data (`languages`, `regions`, `timezone`, weekly
+  `availability` windows, `platformAccess { all, platforms, special[] }`, `taskLimit`), System-only fields
+  (`hourlyRateUsd`, `systemRecommendation`) and the denormalized `ratingSummary { avg, count }`.
+- **`AnalystProfile`** (org-scoped, one per `(orgId, userId)`) — the matching data of an org member with role
+  `org:analyst`. Same `MatchingFieldsSchema` as scouts, no rating or recommendation.
+- **`OrgSettings`** (one per org) — `defaultAnalystTaskLimit` (5) and `timezone`.
+- **`ScoutRating`** — one rating per `(assignmentId, raterUserId)`; System users may leave general ratings without
+  an assignment. `ScoutProfile.ratingSummary` is recomputed from this collection.
+- **`AssigneeLoad`** — an atomic counter of active assignments per `(userId, scope)` where `scope` is `'global'` for
+  scouts and the `orgId` for analysts; task limits are enforced with a conditional `$inc` (`active < limit`).
+- **`Assignment`** — a task may hold many active assignees; a person holds at most one active assignment per task
+  (unique partial index on `{ taskId, assigneeUserId }`).
+
+`Database` is the per-organization data store: exactly one live document per `orgId` (unique partial index), holding
+the `name` and `connectionString` of the Mongo database where the org's collected data (`sources`, `dataItems`, …)
+lives. It is created automatically when an organization is created.
+
+Each Mongoose schema is typed against its Zod type through `ToDoc<T>`, which swaps ObjectId-string fields for
+`Types.ObjectId` and leaves Clerk ids as strings. Every schema sets `timestamps: true`, `versionKey: false`, an explicit
+`collection` from `COLLECTIONS`, and a `toJSON`/`toObject` transform that converts ObjectIds to hex strings — so
+`doc.toJSON()` parses with the matching Zod schema.
+
+The Mongoose schema instances are exported in lowerCamelCase (`missionSchema`, `taskSchema`, …) so they never collide
+with the Zod `MissionSchema`, `TaskSchema`, … from the browser entry.
+
+### Example: Fastify service
+
+```ts
+import Fastify from 'fastify';
+import mongoose from 'mongoose';
+import { registerModels } from 'badger_hq/mongo';
+import { MissionCreateSchema, MissionSchema, type Mission } from 'badger_hq';
+
+const app = Fastify();
+const conn = mongoose.createConnection(process.env.MONGO_URL!);
+await conn.asPromise();
+const models = registerModels(conn);
+await Promise.all(Object.values(models).map((m) => m.syncIndexes())); // or run once in a migration step
+
+app.post<{ Body: unknown; Reply: Mission }>('/missions', async (req, reply) => {
+  const body = MissionCreateSchema.parse(req.body);
+  const orgId = req.auth.orgId;        // from @clerk/fastify
+  const createdBy = req.auth.userId;   // from @clerk/fastify
+  const doc = await models.Mission.create({ ...body, orgId, createdBy });
+  return reply.code(201).send(MissionSchema.parse(doc.toJSON()));
+});
+```
+
+### Indexes
+
+`INDEXES` (in `./mongo`) lists every index per collection and is applied to the schemas at import time. Create them
+with `Model.syncIndexes()` / `Model.createIndexes()` (your call — the package never talks to the database).
+
+`orgId` is the first field of every compound index on org-scoped collections. Unique indexes on soft-deletable
+collections use `partialFilterExpression: { deletedAt: null }` so soft-deleted rows never block re-creation (see
+*Decisions*).
+
+### `Task.version`
+
+`version` is bumped by `$inc` on every `findOneAndUpdate` / `updateOne` through the exported `incrementTaskVersion`
+pre-hook (skipped when the update already sets/incs `version`, or is a pipeline update). Writes that bypass Mongoose
+queries (raw driver, `updateMany`, `bulkWrite`) must `$inc: { version: 1 }` themselves.
+
+## Conventions
+
+### Clerk ids (identity lives in Clerk)
+
+Clerk is the source of truth for users, organizations, memberships, roles, permissions and sessions. This package
+contains **no** `User`, `Organization`, `Membership`, role or permission definitions and does not depend on `@clerk/*`.
+Documents refer to Clerk entities by **id strings only**:
+
+- `orgId` — Clerk org id, validated as `org_…` (`ClerkOrgId`).
+- Every user reference (`createdBy`, `assigneeUserId`, `scoutUserId`, `actorUserId`, `approvedBy`, `doneBy`,
+  `assignedBy`, `requestedBy`, `userId`, …) — Clerk user id, validated as `user_…` (`ClerkUserId`).
+
+They are plain `String` paths in Mongoose — never `ObjectId`, never `ref`.
+
+### Ids, timestamps, soft delete, org scoping
+
+- Badger documents use Mongo ObjectIds: `ObjectIdString` (24-char hex) in Zod/API, `Schema.Types.ObjectId` in Mongoose.
+- Every collection has `createdAt` / `updatedAt`.
+- Every collection except `dataItems`, `scraperMetrics`, `orgSettings`, `scoutRatings` and `assigneeLoads` has an
+  optional `deletedAt` (soft delete).
+- Every collection except the global ones — `scoutProfiles`, `scoutRatings` (optional `orgId`) and `assigneeLoads` —
+  has a required `orgId`.
+
+### Enums
+
+Each enum is defined once in `enums.ts` as `z.enum([...])` plus a `*_VALUES` const array (e.g. `Platform` /
+`PLATFORM_VALUES`). Mongoose schemas use the arrays for `enum:` validation, so Zod and Mongoose cannot drift.
+
+### Terminology
+
+- **Mission** — a priority (PIR) in plain words; the top-level unit of work.
+- **Task** — a unit of work under a mission, `source_discovery` or `monitor`; what gets assigned to a scout or analyst.
+  There is no "subtask" concept.
+
+## Versioning
+
+Semver, managed with [Changesets](https://github.com/changesets/changesets):
+
+| Change | Bump |
+|---|---|
+| Add an optional field, add an enum value, add an index | `minor` |
+| Remove or rename a field, change or remove an enum value, make an optional field required | `major` + a **Migration** note in the changeset |
+| Docs, internal refactors, tightened validation with no shape change | `patch` |
+
+Snapshot tests of the Zod shapes, Mongoose paths, enum values and index definitions fail the build on any unannounced
+shape change; update the snapshot in the same PR as the changeset.
+
+### Release flow
+
+1. Open a PR with your change and `npm run changeset`.
+2. CI (`.github/workflows/ci.yml`) runs lint, typecheck, tests and build.
+3. On merge to `main`, `.github/workflows/release.yml` opens a *Version Packages* PR; merging it publishes to
+   GitHub Packages.
+
+## Development
+
+```sh
+npm install
+npm test          # vitest (incl. mongodb-memory-server integration tests)
+npm run lint
+npm run typecheck
+npm run build     # tsup -> dist/ (ESM + CJS + .d.ts for both entry points)
+npm pack --dry-run
+```
+
+## Decisions
+
+Choices made where the spec left room, kept consistent with the rest of it:
+
+1. **Partial filter for soft-delete uniqueness.** MongoDB rejects `{ $exists: false }` in `partialFilterExpression`
+   (only `$exists: true`, equality, ranges, `$type`, `$and`/`$or`/`$in` are allowed). The equivalent supported form is
+   `{ deletedAt: null }`, which matches documents where `deletedAt` is missing or `null`. Used on `sources`,
+   `scoutProfiles`, `analystProfiles` and (combined with `status: 'active'`) `assignments`. Verified by integration test.
+2. **Server-set fields omitted from Create inputs**, per entity: Mission `createdBy`; Task `approvedBy/At`, `doneBy/At`,
+   `version`; Assignment `assignedBy`, `activatedAt`, `endedAt`, `revokedBy`; HoursLog `scoutUserId`; ScoutProfile
+   `userId`, `status`, `ratingSummary`, `systemRecommendation.updatedBy/At`; Source `discoveredBy`, `lastCollectedAt`;
+   SourceImport `status`, `summary`, `rowErrors`,
+   `createdBy`, `committedAt`; Scraper `sourceCount`, `status`, `deploymentId`; Deployment `status`, `steps`,
+   `requestedBy`; AuditLog `actorUserId`, `at`. Status transitions (approve, mark done, vet, revoke) are expected to be
+   dedicated endpoints rather than generic `PATCH`es.
+3. **Defaults** live on the base schemas so they apply to both parsing and inputs: statuses default to their initial
+   value (`draft`, `active`, `pending`, `validating`, `deploying`, `queued`), array fields default to `[]`,
+   `Task.version` to `1`, `TaskCriteria.emergency` and `AuditLog.actingAsSystem` to `false`.
+4. **`ScoutProfile.taskLimit`** is optional (absent ⇒ global default). The default is exported as
+   `DEFAULT_SCOUT_TASK_LIMIT = 3` instead of storing `null`; analysts fall back to
+   `OrgSettings.defaultAnalystTaskLimit` (`DEFAULT_ANALYST_TASK_LIMIT = 5`).
+5. **`Assignment.scoreBreakdown` / `AuditLog.diff`** are `Schema.Types.Mixed` in Mongoose (free-form), validated by Zod
+   as `Record<string, number>` / `unknown`.
+6. **Cross-org indexes.** `assignments { assigneeUserId, status, orgId }`, `auditLogs { actorUserId, at }`, the unique
+   `scraperMetrics { scraperId, hour }` and everything on the global `scoutProfiles`, `scoutRatings` and
+   `assigneeLoads` collections intentionally don't start with `orgId`: they are per-user / per-scraper lookups.
+   All other compound indexes start with `orgId`.
+7. **`ScraperMetric.errors`** is a reserved Mongoose pathname; the spec names the field so it is kept, with
+   `suppressReservedKeysWarning` set on that schema.
+8. **Mongoose schema export names** are lowerCamelCase (`missionSchema`) to avoid colliding with the Zod
+   `MissionSchema` when both entry points are imported in one file.
+9. **Package name** is `badger_hq` (set in `package.json`), published publicly to the npm registry.
+10. **Multiple assignees per task (1.0.0).** The "one active assignee per task" unique index was replaced by
+    `{ taskId, assigneeUserId }` (partial, `status: 'active'`). Limits are enforced by `assigneeLoads`, not by an index,
+    because a scout's limit spans organizations while an analyst's is per org.
+11. **`ScoutRating.score`** is a Zod literal union `1 | 2 | 3 | 4 | 5` (and a Mongoose `Number` enum) rather than
+    `int().min(1).max(5)`, so the type itself documents the scale.