@atlaskit/editor-plugin-autocomplete 8.0.0 → 9.1.0

package/CHANGELOG.md

@@ -1,5 +1,113 @@
 # @atlaskit/editor-plugin-autocomplete
 
+## 9.1.0
+
+### Minor Changes
+
+- [`a164c6d8e7de6`](https://bitbucket.org/atlassian/atlassian-frontend-monorepo/commits/a164c6d8e7de6) -
+  Offer the Confluence page the chat is open on to autocomplete as context, so words already written
+  on it carry an L1 session boost in the chat input. This reads the editor context the page editor
+  already publishes, so no host wiring is needed, and covers a page being edited; reaching a page
+  that is being viewed needs a host bridge that comes separately. The page text is held together
+  with the page it was read from, so navigating invalidates it in the same render that changes the
+  page rather than just after — the context store keeps the last published document until another
+  publisher replaces it, and that document does not say which page it came from.
+
+  Both the page and the chat transcript sit behind
+  `platform_editor_ai_autocomplete_rovo_chat_editor`, and the editor context subscription that reads
+  the page lives in a component the chat input only mounts once enrolled. An unenrolled session
+  therefore neither parses a page document for context it cannot use nor re-renders the input on a
+  publish, which the editor does on blur and on selection change.
+
+  Context text now waits for the vocabulary load to settle before it is applied. Session boosts were
+  previously spent against an empty trie whenever the page resolved first, which silently dropped
+  every word in it. What the session holds is readable from the console with
+  `__atlCtcDebug__.session()`.
+
+### Patch Changes
+
+- Updated dependencies
+
+## 9.0.0
+
+### Major Changes
+
+- [`94ddcfaca6b5d`](https://bitbucket.org/atlassian/atlassian-frontend-monorepo/commits/94ddcfaca6b5d) -
+  Improve client-only contextual typeahead confidence and responsiveness, and extend it to bigram
+  and phrase completions.
+
+  All of this ships inside the autocomplete plugin, which is only ever added to a preset behind an
+  experiment that defaults to off: `platform_editor_ai_autocomplete_rovo_chat_editor` for Rovo chat,
+  with a Confluence `productKey` check ahead of the read, and
+  `platform_editor_ai_autocomplete_conf_comments` for Confluence comments. The on-device model path
+  these changes concentrate on is additionally behind
+  `platform_editor_ai_autocomplete_conf_local_setup`, which selects the slow-lane client. With those
+  experiments off the plugin is never constructed, so nothing here runs.
+
+  **Confidence.** Candidates competing at the same word boundary are normalised into a posterior
+  over that shortlist, so a suggestion's confidence is the share of probability mass the model puts
+  on it rather than its distance from whichever candidate happens to lead. The comparison is made on
+  each surface's sequence log-likelihood rather than its per-token mean, which is not comparable
+  across surfaces of different token counts. A shortlist is normalised over its minimal members
+  only, so a surface and its own extension are no longer double-counted. A surface must additionally
+  clear an absolute plausibility floor, anchored at the uniform distribution over the model's
+  49,152-token vocabulary (about -10.8 nats) rather than read off a histogram, hold a clear lead
+  over the runner-up, and come from a shortlist that offered a real alternative. Completions shorter
+  than three characters are no longer offered, and multi-word terms are held to the same
+  previous-word grammar transition as single words.
+
+  **Responsiveness.** Candidate work starts at two typed characters while ghost text stays gated at
+  three and must arrive within a 100 ms decision window. Queued scoring work is drained in an order
+  that keeps the engine's decoded sequence alive, so continuing a prefix costs a single decode step
+  instead of a full prompt prefill, and work no live decision would accept is discarded rather than
+  run. Contexts keep their prefilled model state until 128 are resident rather than 32, which stops
+  contexts being evicted while still in use. Once displayed, a ghost is an immutable snapshot shared
+  by rendering, Tab insertion, analytics and cooldown, so a later model result cannot make the
+  visible completion disagree with the inserted one.
+
+  **Per-keystroke cost.** One keystroke runs `predict()` several times — once when the decision
+  opens and again for each async evidence signal that lands inside the budget — and only the
+  evidence differs between those runs. Trie recall and the canonical context derived per candidate
+  are computed once and reused across them, invalidated when an artifact load changes what the tries
+  can return. Within one derivation the position-dependent half of a candidate's context is shared
+  by every candidate starting at the same offset, rather than being rebuilt a couple of hundred
+  times. A distribution's log-partition is memoised per logits buffer, since it does not depend on
+  the token being scored and the two full-vocabulary passes behind it previously ran once per
+  candidate for an answer that could not change. None of this changes which suggestion is chosen.
+
+  **Multi-word completions.** Bigrams and phrases are surfaced from a first-word or in-progress
+  window, frequency is normalised per term type, and selection runs a per-type precision floor
+  followed by expected-value arbitration, so the longest confident unit wins and a shaky long unit
+  self-demotes to a safer shorter one. Phrases surface only once the on-device continuation LM has
+  vouched for them. A whole-surface repetition guard and a post-accept cooldown together prevent
+  echoes such as `end to end` becoming `end to end to end`.
+
+  **Artifacts.** `bigrams.json`, `phrases.json` and `phrase-continuation-tokens.json` are resolved
+  from the artifacts manifest alongside the vocabulary and grammar payloads, so they are served from
+  the CDN rather than bundled. Each is fetched independently, so an unpublished payload disables
+  only its own term type. Every artifact is shape-checked as it downloads, failing at the boundary
+  naming the artifact rather than degrading later into an empty trie.
+
+  **Diagnostics.** Debug output is unified under a single `[CTC]` namespace behind
+  `__atlCtcDebug__.enable()`, with one collapsed group per keystroke and an opt-in verbose level for
+  the scored candidate table. All informational console output is routed behind the flag.
+
+  **Breaking change.** `@atlaskit/editor-plugin-autocomplete` no longer exposes its pm-plugin
+  modules as subpaths. The `./src/pm-plugins/autocomplete-plugin`,
+  `./src/pm-plugins/slow-lane-client` and `./src/pm-plugins/text-predictor` entry points are
+  removed, along with the `CANONICAL_FIX__DO_NOT_USE_ME_A`, `_B` and `_C` subpaths that exposed the
+  same three modules in raw form. The matching `./autocomplete/*` re-export wrappers in
+  `@atlaskit/editor-plugins` are removed too. No product consumed any of them; they were reachable
+  only from this plugin's own test package and the generated aggregator.
+
+  Removing both sets together is deliberate. The curated subpaths were named after real file paths,
+  so each one shadowed the module it wrapped and left two files claiming the same bundler canonical
+  id — which is what the `CANONICAL_FIX` subpaths existed to disambiguate. Keeping either set alone
+  reintroduces the collision. With both gone, nothing shadows a source path, and because the
+  wrappers mirrored the raw modules under an exact export-parity test, `@atlaskit/editor-plugins` is
+  no longer coupled to this plugin's internal module layout. `./autocompletePlugin` and
+  `./autocompletePluginType` are unaffected.
+
 ## 8.0.0
 
 ### Patch Changes

package/dist/cjs/analytics/ufo.js

@@ -39,6 +39,7 @@ var EXPERIENCE_NAME = exports.EXPERIENCE_NAME = {
   SLOW_LANE_FETCH: 'slow-lane-fetch',
   LOAD_VOCABULARY: 'load-vocabulary',
   LOAD_VECTORS: 'load-vectors',
+  LOAD_PHRASES: 'load-phrases',
   LOAD_GRAMMAR: 'load-grammar'
 };
 var isUfoEnabled = function isUfoEnabled() {
@@ -57,7 +58,7 @@ var operationConfig = {
     })
   }
 };
-var experiences = (0, _defineProperty2.default)((0, _defineProperty2.default)((0, _defineProperty2.default)((0, _defineProperty2.default)({}, EXPERIENCE_NAME.SLOW_LANE_FETCH, new _ufo.ConcurrentExperience(EXPERIENCE_NAME.SLOW_LANE_FETCH, operationConfig)), EXPERIENCE_NAME.LOAD_VOCABULARY, new _ufo.ConcurrentExperience(EXPERIENCE_NAME.LOAD_VOCABULARY, operationConfig)), EXPERIENCE_NAME.LOAD_VECTORS, new _ufo.ConcurrentExperience(EXPERIENCE_NAME.LOAD_VECTORS, operationConfig)), EXPERIENCE_NAME.LOAD_GRAMMAR, new _ufo.ConcurrentExperience(EXPERIENCE_NAME.LOAD_GRAMMAR, operationConfig));
+var experiences = (0, _defineProperty2.default)((0, _defineProperty2.default)((0, _defineProperty2.default)((0, _defineProperty2.default)((0, _defineProperty2.default)({}, EXPERIENCE_NAME.SLOW_LANE_FETCH, new _ufo.ConcurrentExperience(EXPERIENCE_NAME.SLOW_LANE_FETCH, operationConfig)), EXPERIENCE_NAME.LOAD_VOCABULARY, new _ufo.ConcurrentExperience(EXPERIENCE_NAME.LOAD_VOCABULARY, operationConfig)), EXPERIENCE_NAME.LOAD_VECTORS, new _ufo.ConcurrentExperience(EXPERIENCE_NAME.LOAD_VECTORS, operationConfig)), EXPERIENCE_NAME.LOAD_PHRASES, new _ufo.ConcurrentExperience(EXPERIENCE_NAME.LOAD_PHRASES, operationConfig)), EXPERIENCE_NAME.LOAD_GRAMMAR, new _ufo.ConcurrentExperience(EXPERIENCE_NAME.LOAD_GRAMMAR, operationConfig));
 var startExp = exports.startExp = function startExp(name, id, metadata) {
   if (!isUfoEnabled()) {
     return;

package/dist/cjs/pm-plugins/artifact-loader.js

@@ -0,0 +1,135 @@
+"use strict";
+
+var _interopRequireDefault = require("@babel/runtime/helpers/interopRequireDefault");
+Object.defineProperty(exports, "__esModule", {
+  value: true
+});
+exports.fetchAutocompleteArtifactJson = exports.fetchAutocompleteArtifactBinary = exports.buildAutocompleteGatewayUrl = void 0;
+var _regenerator = _interopRequireDefault(require("@babel/runtime/regenerator"));
+var _asyncToGenerator2 = _interopRequireDefault(require("@babel/runtime/helpers/asyncToGenerator"));
+var _artifactsManifest = require("./artifacts-manifest");
+var _debugMode = require("./debug-mode");
+/**
+ * Validation and diagnostics layer over the artifacts manifest.
+ *
+ * `artifacts-manifest` owns *where* a payload comes from (one manifest request,
+ * then a presigned CDN download). This module owns *whether the payload is
+ * usable*: every loader declares the shape it expects, so a mis-published
+ * artifact fails at the boundary with the artifact's name rather than surfacing
+ * later as an empty trie or a silent no-op in scoring.
+ */
+
+var TRAILING_SLASH_REGEX = /\/$/;
+var DEFAULT_BASE_URL = '';
+var AUTOCOMPLETE_GATEWAY_BASE = '/gateway/api/v1/autocomplete';
+/**
+ * Build a URL for one of the autocomplete gateway's own endpoints (e.g. the
+ * network slow lane). Model artifacts do NOT go through here — they are served
+ * from the CDN via the manifest.
+ */
+var buildAutocompleteGatewayUrl = exports.buildAutocompleteGatewayUrl = function buildAutocompleteGatewayUrl(path) {
+  var baseUrl = arguments.length > 1 && arguments[1] !== undefined ? arguments[1] : DEFAULT_BASE_URL;
+  var normalizedBase = baseUrl.replace(TRAILING_SLASH_REGEX, '');
+  return "".concat(normalizedBase).concat(AUTOCOMPLETE_GATEWAY_BASE).concat(path);
+};
+
+/**
+ * Download a JSON artifact and assert its shape before handing it to a loader.
+ *
+ * :params:
+ *   artifactName: Manifest key for the artifact
+ *   options.label: Name used in diagnostics; defaults to the manifest key
+ *   options.summarize: Renders a short size/count summary for the debug log
+ *   options.validate: Type guard the payload must satisfy
+ * :returns:
+ *   The parsed, validated payload
+ */
+var fetchAutocompleteArtifactJson = exports.fetchAutocompleteArtifactJson = /*#__PURE__*/function () {
+  var _ref = (0, _asyncToGenerator2.default)( /*#__PURE__*/_regenerator.default.mark(function _callee(artifactName) {
+    var _options$label, _options$summarize;
+    var options,
+      label,
+      payload,
+      summary,
+      _args = arguments;
+    return _regenerator.default.wrap(function (_context) {
+      while (1) switch (_context.prev = _context.next) {
+        case 0:
+          options = _args.length > 1 && _args[1] !== undefined ? _args[1] : {};
+          label = (_options$label = options.label) !== null && _options$label !== void 0 ? _options$label : artifactName; // Presigned CDN URLs carry an auth token, so only the artifact name is logged.
+          (0, _debugMode.ctcTag)('init', "\u2193 loading ".concat(label));
+          _context.next = 1;
+          return (0, _artifactsManifest.fetchArtifactJson)(artifactName);
+        case 1:
+          payload = _context.sent;
+          if (!(options.validate && !options.validate(payload))) {
+            _context.next = 2;
+            break;
+          }
+          throw new Error("[autocomplete-artifacts] ".concat(label, " payload shape was invalid"));
+        case 2:
+          summary = (_options$summarize = options.summarize) === null || _options$summarize === void 0 ? void 0 : _options$summarize.call(options, payload);
+          (0, _debugMode.ctcTag)('init', "\u2713 loaded ".concat(label).concat(summary ? " (".concat(summary, ")") : ''));
+          return _context.abrupt("return", payload);
+        case 3:
+        case "end":
+          return _context.stop();
+      }
+    }, _callee);
+  }));
+  return function fetchAutocompleteArtifactJson(_x) {
+    return _ref.apply(this, arguments);
+  };
+}();
+
+/**
+ * Download a binary artifact (the word-vectors blob) as an ArrayBuffer.
+ *
+ * :params:
+ *   artifactName: Manifest key for the artifact
+ *   label: Name used in diagnostics; defaults to the manifest key
+ * :returns:
+ *   The raw bytes of the artifact
+ */
+var fetchAutocompleteArtifactBinary = exports.fetchAutocompleteArtifactBinary = /*#__PURE__*/function () {
+  var _ref2 = (0, _asyncToGenerator2.default)( /*#__PURE__*/_regenerator.default.mark(function _callee2(artifactName) {
+    var label,
+      url,
+      response,
+      payload,
+      _args2 = arguments;
+    return _regenerator.default.wrap(function (_context2) {
+      while (1) switch (_context2.prev = _context2.next) {
+        case 0:
+          label = _args2.length > 1 && _args2[1] !== undefined ? _args2[1] : artifactName;
+          (0, _debugMode.ctcTag)('init', "\u2193 loading ".concat(label));
+          _context2.next = 1;
+          return (0, _artifactsManifest.getArtifactUrl)(artifactName);
+        case 1:
+          url = _context2.sent;
+          _context2.next = 2;
+          return fetch(url);
+        case 2:
+          response = _context2.sent;
+          if (response.ok) {
+            _context2.next = 3;
+            break;
+          }
+          throw new Error("[autocomplete-artifacts] Failed to fetch ".concat(label, ": ").concat(response.status));
+        case 3:
+          _context2.next = 4;
+          return response.arrayBuffer();
+        case 4:
+          payload = _context2.sent;
+          (0, _debugMode.ctcTag)('init', "\u2713 loaded ".concat(label, " (").concat(payload.byteLength, " bytes)"));
+          return _context2.abrupt("return", payload);
+        case 5:
+        case "end":
+          return _context2.stop();
+      }
+    }, _callee2);
+  }));
+  return function fetchAutocompleteArtifactBinary(_x2) {
+    return _ref2.apply(this, arguments);
+  };
+}();

package/dist/cjs/pm-plugins/artifacts-manifest.js

@@ -28,10 +28,13 @@ var _debugMode = require("./debug-mode");
  * hyphenated, unlike the underscored filenames the artifacts are generated from.
  */
 var ARTIFACT_NAME = exports.ARTIFACT_NAME = {
+  BIGRAMS: 'bigrams.json',
   FIRST_TOKEN_TO_WORDS: 'first-token-to-words.json',
   GHOST_POS_TAGS: 'ghost-pos-tags.json',
   GRAMMAR_TRANSITIONS: 'grammar-transitions-10k.json',
   L3_VOCABULARY: 'l3-vocabulary.json',
+  PHRASE_CONTINUATION_TOKENS: 'phrase-continuation-tokens.json',
+  PHRASES: 'phrases.json',
   POS_TAGS: 'combined-l2-l3-pos-tags.json',
   VOCABULARY: 'vocabulary-10k.json',
   WORD_INDEX: 'word-index-10k.json',