@hartren/assistant-js 0.0.0-stage → 2.3.1

package/CHANGELOG.md

@@ -0,0 +1,140 @@
+# Changelog
+
+## 2.3.1 (unreleased)
+
+### Changed
+
+- First version for the public npm registry (`registry.npmjs.org`). 2.3.0 is skipped because tag
+  `v2.3.0` already exists in the repository. Built from source in CI; the pack's SHA-256 and file list
+  are kept as a CI artifact. Third-party notices are included.
+- `package.json` no longer has `homepage` or `bugs` fields.
+
+## 2.3.0 (2026-11-02)
+
+### Changed
+
+- **Renamed to `@hartren/assistant-js`** (GitHub Packages). Replace `@hartren/copilotagent-js` with
+  `@hartren/assistant-js` in your `package.json` and imports. The API is unchanged. Existing
+  `@hartren/copilotagent-js` versions stay published unchanged.
+
+## 2.2.0 (2026-10-25)
+
+### Added
+
+- **`browserTranscriptionFallback`** (boolean, default `false`). When `true`, dictation switches to
+  the browser's `SpeechRecognition` when the hosted transcription socket cannot connect or drops
+  mid-session while the browser is online, and returns to the hosted service on `online` or at the
+  next `startListening()`. Offline behaves as without the option (cloud Web Speech cannot work
+  offline either): no switch, and a fallback already running is paused, with no restarts and no
+  "network hiccup" errors, until `online`. New `onCopilotError` codes `transcription-fallback` and
+  `transcription-restored` (both `final: false`), and `agent.isUsingBrowserTranscriptionFallback()`.
+  This restores, behind an opt-in, the automatic Web Speech fallback removed in 2.0.0 (ce9e), now
+  with hand-back.
+  **Warning:** when on, dictation audio may be processed by the browser vendor's speech service (no
+  BAA). Only enable it where that is acceptable, never with PHI. With the default (`false`),
+  `SpeechRecognition` is never constructed and nothing changes.
+
+## 2.1.0 (2026-10-24)
+
+### Changed
+
+- **Renamed to `@hartren/copilotagent-js` and published to GitHub Packages** (`npm.pkg.github.com`)
+  instead of public npm. Replace `@radaid/copilotagent-js` with `@hartren/copilotagent-js` in
+  `package.json` and imports, and add the `@hartren` registry line to `.npmrc` (see the README). The
+  API is unchanged. Existing `@radaid/copilotagent-js` versions (up to 2.0.1) stay on public npm.
+
+### Fixed
+
+- Legacy PAT mode: when `/token/refresh` fails (expired, used or revoked refresh token, or a server
+  error), the library now falls back to one PAT re-exchange, shared by concurrent requests, instead
+  of failing every request until reload.
+- Errors are logged by name or code only (`errorLabel`), never with their message or response text.
+- HTTP failures throw `HttpError` (exported; `status`, `code` such as `HTTP_500`) and are logged by
+  that code. The error message is unchanged.
+- Browser speech start/restart errors log only the error name, and a closed speech socket logs only
+  its close code; the full error and the close reason are logged only with debug logging on.
+- Concurrent legacy token refreshes share one request, cleared once when it settles.
+
+### Added
+
+- **Host token provider is the supported way to authenticate.** Pass `getAccessToken` (for example
+  `() => auth.getAccessToken()` from `@hartren/auth`) and, optionally, `onUnauthorized`
+  (`(rejectedToken) => auth.handleUnauthorized(rejectedToken)`). The provider is asked on every API
+  request and for the speech socket, and no copy of the token is kept. On a `401` the request is
+  retried once, only with a different token from `onUnauthorized` (or, without it, from the provider).
+  New exported types `AccessTokenProvider` and `UnauthorizedHandler`.
+- With a provider and no PAT, a missing token rejects with `Not signed in` instead of falling through
+  to the PAT exchange.
+
+### Deprecated
+
+- `pat`, `setUserId`, the browser PAT exchange (`/token/exchange`) and the legacy refresh
+  (`/token/refresh`). They still work when a PAT is passed and the provider is missing or returns
+  nothing, and will be removed in a future major version.
+
+### Fixed
+
+- `setUserId` no longer starts a PAT exchange when there is no PAT (or when a provider is set), so it
+  no longer sends `Authorization: PAT undefined` or leaves an unhandled rejection.
+- In the PAT path, a `401` now drops the cached token and refreshes before the single retry (it used
+  to retry with the same token).
+
+## 2.0.1 (2026-10-18)
+
+### Fixed
+
+- **No user or AI text in the browser console by default.** Dictated transcripts, prompts, assistant
+  replies and raw API responses were logged with `console.log` / `console.info`. That text can be PHI.
+  These logs are now off unless a host turns on debug logging: `new CopilotAgent({ debug: true })` or
+  `setCopilotDebugLogging(true)` (new exports, with `isCopilotDebugLogging()`). Use it for local
+  debugging only. The gated logs are in `CopilotAgent` (API response, combined message, AI reply,
+  recognized text) and `ResultsGroupingService` (each transcript and the grouped results).
+- A browser speech-synthesis error logs only its error code (for example `interrupted`), not the
+  event: the event's `utterance.text` is the AI reply. A failed tool logs only the error name or code,
+  not its message, which can echo tool arguments. The full details are logged only with debug logging on.
+- `package.json` now has real `repository`, `homepage` and `bugs` fields
+
+### Changed
+
+- The package no longer ships `src/`. Only `dist/` is published, plus `CHANGELOG.md`, `README.md`,
+  `LICENSE` and `package.json`. Nothing imports from `src/`: `main` and `types` point into `dist/`.
+
+## 2.0.0 (2026-10-18)
+
+### BREAKING CHANGE
+
+- **The speech token is now sent as a WebSocket subprotocol, never in the URL.** The speech sockets
+  (`/ws/speech/transcribe` and `/ws/speech/speak`) offer `hartren.speech.v1` and `bearer.<token>` in
+  the `Sec-WebSocket-Protocol` header.
+- **This needs a backend that echoes `hartren.speech.v1`.** Against an
+  older backend, Chrome refuses the socket and dictation and voice replies fail. Don't upgrade a host that talks to such a
+  backend until the backend is updated.
+- **Deepgram (the hosted Hartren API speech service) is the default, and browser speech is opt-in
+  only.** The SDK no longer falls back to the browser's Web Speech (`SpeechRecognition` /
+  `speechSynthesis`) under any failure. A host gets it only with `speechProvider: 'browser'`. Don't use
+  that setting with PHI: browser speech can send audio and text to the browser or OS vendor.
+
+### Changed
+
+- The speech socket URL now defaults to the `copilotApi` origin (`https://api.example.com/api` →
+  `wss://api.example.com`). A relative `copilotApi` resolves against `window.location.origin`.
+- Hosted speech failures reconnect with exponential backoff (`speechRetry`, default 3 retries,
+  1 s base, 8 s cap). Each failure is reported through `onCopilotError` with a `code` from
+  `SpeechErrorCodes`.
+- The token is sent only over `wss://`, or plain `ws://` to `localhost`, `127.0.0.1` or `[::1]`. Any
+  other `ws://` URL is refused before the token is read (`transcription-not-configured` /
+  `tts-not-configured`, no retry).
+- A token that cannot be sent as a subprotocol fails once with `transcription-failed`,
+  `reason: 'invalid-token'`, with no retry.
+- Microphone errors (ported from #30): `audio-setup-failed` with `reason` (for example `mic-denied`,
+  `mic-dismissed`, `mic-policy-blocked`, `mic-no-device`, `vad-asset-failed`), `detail` (the browser
+  error name) and `environment`. A session failure (`reason: 'session-failed'`) is retried.
+- The token and token-bearing URLs are never logged.
+- New exports: `speechProvider`, `SpeechErrorCodes`, `DEFAULT_SPEECH_RETRY`, `SPEECH_WS_PROTOCOL`,
+  `SPEECH_RECOGNITION_FAILURES`, and the types `CopilotError`, `SpeechProvider`, `SpeechRetryOptions`,
+  `SpeechErrorCode`, `SpeechRecognitionFailure`, `MicrophoneEnvironment`.
+
+### Notes
+
+- 1.1.10 to 1.1.16 fell back to browser speech when the hosted socket failed. Don't roll back to
+  them in production.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Hartren LLC
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,274 @@
-# Temporary Holding Version
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+# @hartren/assistant-js
+
+Browser client for the Hartren Assistant: dictation (speech socket) and the AI requests behind the Assistant and Reporting panels in the Hartren viewer and report builder.
+
+## Installation
+
+Published on the public npm registry:
+
+```bash
+npm install @hartren/assistant-js
+```
+
+Previously published as `@hartren/copilotagent-js` (and before that `@radaid/copilotagent-js`). Use `@hartren/assistant-js` for 2.3.0 and later.
+
+
+## Development
+
+```bash
+# from a checkout of this repository
+npm ci
+npm test
+```
+
+## Build
+To compile and bundle the package into a single JavaScript file:
+```bash
+yarn run build
+```
+
+The output will be generated in the `dist/` directory.
+
+## Directory Structure
+```
+/
+├── src/                     # Source files
+│   └── index.ts             # Entry point for the package
+├── dist/                    # Bundled output
+├── webpack.config.js        # Webpack configuration
+├── package.json             # Package configuration
+├── tsconfig.json            # TypeScript configuration
+├── README.md                # This README file
+```
+
+## Usage
+```ts
+import CopilotAgent from '@hartren/assistant-js';
+```
+
+## Scripts
+- **Build**: Compile TypeScript and bundle files:
+  ```bash
+  yarn run build
+  ```
+
+
+## How to Configure Package Locally
+
+
+1. **Unlink the package** (if it's already linked)
+   ```bash
+   yarn unlink @hartren/assistant-js
+   ```
+
+2. **Link the package**
+   ```bash
+   yarn dev
+   yarn link
+   yarn watch
+   ```
+
+3. **Go to another repo** where you want to configure this package
+
+4. **Remove the package** from `package.json`
+
+5. **Link the package** in the new repo
+   ```bash
+   yarn link @hartren/assistant-js
+   yarn dev
+   ```
+
+## Releasing
+
+1. Merge a PR that sets the new `version` in `package.json` and dates its `CHANGELOG.md` entry.
+2. A maintainer runs **Actions > Publish > Run workflow** on main with that version.
+3. The workflow checks that the version matches `package.json`, runs the tests and the build, and
+   publishes to npmjs (`--access public`) with npm Trusted Publishing or the `NPM_TOKEN` environment secret in `npm-publish`.
+
+## License
+This project is licensed under the [MIT License](LICENSE).
+
+ 
+## Debug logging (off by default)
+
+The agent never writes dictated text, prompts, AI replies or raw API responses to the browser
+console unless you turn on debug logging. That text can be PHI, so do this for local debugging only:
+
+```js
+new CopilotAgent({ ...options, debug: true });
+// or, at any time:
+import { setCopilotDebugLogging } from '@hartren/assistant-js';
+setCopilotDebugLogging(true);
+```
+
+## Authentication
+
+Pass the user's token through a provider. The agent asks for it on every API request and for the
+speech socket, and keeps no copy. With `@hartren/auth`:
+
+```ts
+new CopilotAgent({
+  copilotApi: 'https://api.hartren.com/api',
+  commands,
+  getAccessToken: () => auth.getAccessToken(), // resolve null when signed out
+  onUnauthorized: rejected => auth.handleUnauthorized(rejected), // optional: a different token to retry once
+});
+```
+
+On a `401` the request is retried once, and only with a different token. Without `onUnauthorized` the
+agent asks `getAccessToken` again.
+
+`pat` and `setUserId` (browser PAT exchange plus `/token/refresh`) are deprecated. They still work
+when a PAT is passed and the provider is missing or returns nothing, and will be removed in a future major version.
+
+## Speech (dictation and voice replies)
+
+> **BREAKING CHANGE in 2.0.0.** The speech token is sent as the WebSocket subprotocol
+> `bearer.<token>`, next to `hartren.speech.v1`. This needs a backend that echoes
+> `hartren.speech.v1` (the current Hartren API does). Against older backends, Chrome refuses the socket and dictation fails. Deepgram (hosted speech) is the
+> default, and browser speech is opt-in only (`speechProvider: 'browser'`). See
+> [CHANGELOG.md](CHANGELOG.md).
+
+Dictation audio and spoken replies can contain PHI. By default the agent uses **only** the hosted
+Hartren API speech service (Deepgram): `/ws/speech/transcribe` for dictation and `/ws/speech/speak`
+for voice replies. Browsers cannot set headers on a WebSocket, so the user's access token is sent
+as a WebSocket subprotocol: the socket offers `hartren.speech.v1` and `bearer.<token>` in the
+`Sec-WebSocket-Protocol` header, and the server echoes only `hartren.speech.v1`. The token is never
+put in the URL, so it cannot end up in access logs. This needs a Hartren API version that reads and
+echoes the subprotocol (Chrome fails the handshake when a server does not echo one).
+
+```ts
+new CopilotAgent({
+  copilotApi: 'https://api.hartren.com/api',
+  getAccessToken,
+  // Optional. Defaults to the copilotApi origin with a ws/wss scheme (wss://api.hartren.com).
+  transcribeWssUrl: 'wss://api.hartren.com',
+  // Optional reconnect policy for the transcription socket (these are the defaults).
+  speechRetry: { maxRetries: 3, baseDelayMs: 1000, maxDelayMs: 8000 },
+});
+```
+
+If the hosted service fails (handshake rejected, socket dropped, missing token or session, no URL),
+the agent never falls back to browser speech by default (see
+[Browser transcription fallback](#browser-transcription-fallback-opt-in) for the opt-in). Instead it reports the failure through
+`onCopilotError` and reconnects with exponential backoff. Each reconnect raises
+`code: 'transcription-retrying'` with `final: false`. When the retries run out, it raises
+`code: 'transcription-failed'` with `final: true` and stops dictation. If hosted TTS fails it raises
+`code: 'tts-failed'`; with no hosted TTS configured it raises `code: 'tts-not-configured'`. If the
+transcription service sends an error on an open socket, it raises `code: 'transcription-error'` with
+`final: false`. The full list is exported as `SpeechErrorCodes`.
+
+**Log `code`, not `message`, from `onCopilotError`.** Most messages are fixed strings, but for
+`transcription-error` the `message` is the server's own text, which can echo what was dictated. The
+agent itself prints only a fixed line (the server text only with `debug: true`). Type the listener with
+the exported `CopilotError`:
+
+```ts
+import type { CopilotError } from '@hartren/assistant-js';
+agent.onCopilotError((e: CopilotError) => console.warn('[Copilot] error:', e.code ?? 'unknown'));
+```
+
+**The token is only sent over `wss://`.** A plain `ws://` speech URL is accepted only for
+`localhost`, `127.0.0.1` or `[::1]` (local development). Any other `ws://` URL, including one derived
+from an `http://` copilotApi, is refused before the token is read: no socket is opened, and the agent
+raises `transcription-not-configured` (or `tts-not-configured` for voice replies) with no retry.
+
+**Microphone problems** raise `code: 'audio-setup-failed'` with no retry. `reason` names the cause
+(`mic-denied`, `mic-dismissed`, `mic-os-blocked`, `mic-policy-blocked`, `mic-no-device`, `mic-busy`,
+`mic-constraints`, `mic-unsupported`, `mic-insecure-context`, `mic-failed`, `vad-asset-failed`),
+`detail` is the browser's error name (never its raw message), and `environment` says what the browser
+reported about microphone access. The messages are exported as `SPEECH_RECOGNITION_FAILURES`. A
+session that cannot be started is reported as `reason: 'session-failed'` and is retried like a
+connection failure.
+
+**Browser speech is opt-in only.** `speechProvider: 'browser'` uses the browser's `SpeechRecognition`
+and `speechSynthesis`. Those can send audio or text to the browser or OS vendor (for example Google
+or Microsoft), and there is no BAA with them. Do not use this setting with PHI.
+
+### Browser transcription fallback (opt-in)
+
+> **Warning.** When `browserTranscriptionFallback` is on, dictation audio may be processed by the
+> browser vendor's speech service (for example Google for Chrome, Microsoft for Edge). There is no BAA
+> with them. Only enable it where that is acceptable, never where dictation can contain PHI. Hartren's
+> own deployments leave it off.
+
+```ts
+new CopilotAgent({
+  copilotApi: 'https://api.example.com/api',
+  getAccessToken,
+  browserTranscriptionFallback: true, // default false
+});
+```
+
+`browserTranscriptionFallback` (boolean, default `false`) keeps dictation going on a spotty
+connection by switching to the browser's `SpeechRecognition` / `webkitSpeechRecognition`. Only the
+boolean `true` turns it on; any other value (including the string `'true'`) leaves it off. It applies
+to the hosted (default) provider and to dictation only; voice replies stay hosted-only.
+
+- **Off (default):** nothing changes. `SpeechRecognition` is never constructed, and hosted failures
+  retry and then raise `transcription-failed` as described above.
+- **When it kicks in:** while dictation is running, on a hosted connection failure that would
+  otherwise be retried: the socket cannot be prepared or opened, the handshake is refused, the socket
+  drops mid-session, or the session cannot be started. The browser recognizer is created on the first
+  switch, not before. The agent raises `code: 'transcription-fallback'` (`final: false`, `reason`
+  such as `socket-closed` or `connect-failed`)
+  through `onCopilotError`, and `agent.isUsingBrowserTranscriptionFallback()` returns `true`.
+- **Not for configuration or microphone errors:** a refused `ws://` URL, an unusable token and
+  `audio-setup-failed` are reported as before, with no fallback.
+- **Offline behaves as without the option.** Chrome and Edge run Web Speech in the cloud, so it
+  cannot work offline either. The `offline` event never switches, and a hosted failure while
+  `navigator.onLine` is `false` retries and then raises `transcription-failed` as usual.
+- **Going offline while the fallback is active pauses it.** The browser recognizer's `network`
+  errors are expected then, so it is not restarted and the "network hiccup" error is not raised
+  (one console warning, fixed text). Dictation resumes on the hosted service on `online`. While
+  online, a `network` error from browser speech keeps the normal restart-and-report behavior.
+- **Handing back:** when the browser fires `online` during the dictation, the agent stops browser
+  speech, reconnects to the hosted service and raises `code: 'transcription-restored'`
+  (`final: false`). Every new `startListening()` tries the hosted service first. If the hosted
+  service fails again, it falls back again.
+- **Unsupported browsers** (no `SpeechRecognition`, e.g. Firefox): no error from the fallback; the
+  hosted retries and the final `transcription-failed` happen as if the option were off.
+- Browser speech needs a network connection itself (Chrome sends audio to Google), so the fallback
+  helps when the hosted service or its route is unreachable or flaky but the browser vendor is not.
+
+## License
+This project is licensed under the [MIT License](LICENSE).
+
+## Authentication and the speech socket
+
+- Get the user's token with `@hartren/auth` (which uses `POST /api/token/user-exchange`; your backend
+  does the first exchange with its integration token, the browser only refreshes) and pass
+  `getAccessToken: () => auth.getAccessToken()` and `onUnauthorized: (t) => auth.handleUnauthorized(t)`.
+  Don't use the legacy `/api/token/exchange` with `userId`; it is denied by default (`403`).
+- Dictation opens `wss://{api host}/ws/speech/transcribe` offering two WebSocket subprotocols:
+  `hartren.speech.v1` and `bearer.<access token>`. The server echoes only `hartren.speech.v1`.
+  A handshake without a valid token gets `401`; a user who hasn't accepted the Terms gets
+  `403 tos_required`, and a user with report generation disabled gets `403`, both before the socket opens.
+
+## Server responses to handle (Hartren API)
+
+Every API error body is JSON. Token endpoints use `{ "error", "error_description" }`; the other
+endpoints use `application/problem+json` (`title`, `detail`, `status`).
+
+| Status | When | What the client should do |
+|---|---|---|
+| `204 No Content` | Success with no body (for example deletes and some updates) | Treat as success; don't parse a body |
+| `401` | Token missing, expired or revoked (and the speech handshake before the socket opens) | Get a fresh token once (`onUnauthorized` / `handleUnauthorized`), retry once, then sign the user in again |
+| `403 tos_required` | The user hasn't accepted the current Terms of Use in this tenant. Returned on every API call and on the speech handshake | Read `GET /api/legal/status`, show the Terms (`GET /api/legal/tos`), then `POST /api/legal/accept` with the version. Don't retry the original call until then |
+| `403` "Report generation is disabled for this user" | The organization turned report generation (and dictation) off for this user | Show the message; don't retry |
+| `403` "Report limit reached for your plan" | The user's plan limit is used up | Show the message; don't retry |
+| `403 access_denied` / `unauthorized_client` | `user-exchange`: the user can't get a token through your integration, or wrong token type / IP | Don't retry; check the user mapping or integration token |
+| `409 tos_version_stale` | `POST /api/legal/accept` with an old version: the Terms changed | Fetch the current Terms, show them again, accept the new version |
+| `409` (other endpoints) | A conflicting change (for example the item already exists or changed concurrently) | Reload, then retry once if it still applies |
+| `429 slow_down` | Too many requests | Wait `Retry-After` seconds |
+| `503 legal_storage_unavailable` | Terms status couldn't be read or written (fails closed) | Retry later |
+
+Never retry `400`, `401` (after the one refresh), `403` or `409` in a loop.
+
+### Protected npm publishing
+
+Store `NPM_TOKEN` as an environment secret in the main-only `npm-publish` GitHub
+environment, never as a repository secret. Configure the npm trusted publisher
+with environment name `npm-publish` (alongside this repository and `publish.yml`).
+The first publish requires the short-lived token; subsequent publishing can use OIDC.