@backstage/plugin-search-backend-module-stack-overflow-collator 0.0.0-nightly-20231020021216

package/CHANGELOG.md

@@ -0,0 +1,17 @@
+# @backstage/plugin-search-backend-module-stack-overflow-collator
+
+## 0.0.0-nightly-20231020021216
+
+### Minor Changes
+
+- 46f0f1700eb8: Extract a package for the Stack Overflow new backend system plugin.
+
+### Patch Changes
+
+- Updated dependencies
+  - @backstage/plugin-search-backend-node@0.0.0-nightly-20231020021216
+  - @backstage/backend-common@0.0.0-nightly-20231020021216
+  - @backstage/config@1.1.1
+  - @backstage/backend-tasks@0.0.0-nightly-20231020021216
+  - @backstage/backend-plugin-api@0.0.0-nightly-20231020021216
+  - @backstage/plugin-search-common@1.2.7

package/README.md

@@ -0,0 +1,95 @@
+# Stack Overflow Search Backend Module
+
+A plugin that provides stack overflow specific functionality that can be used in different ways (e.g. for search) to compose your Backstage App.
+
+## Getting started
+
+Before we begin, make sure:
+
+- You have created your own standalone Backstage app using @backstage/create-app and not using a fork of the backstage repository. If you haven't setup Backstage already, start [here](https://backstage.io/docs/getting-started/).
+
+To use any of the functionality this plugin provides, you need to start by configuring your App with the following config:
+
+```yaml
+stackoverflow:
+  baseUrl: https://api.stackexchange.com/2.2 # alternative: your internal stack overflow instance
+```
+
+### Stack Overflow for Teams
+
+If you have a private Stack Overflow instance and/or a private Stack Overflow Team you will need to supply an API key or Personal Access Token. You can read more about how to set this up by going to [Stack Overflow's Help Page](https://stackoverflow.help/en/articles/4385859-stack-overflow-for-teams-api).
+
+The existing API key approach remains the default, to support the new v2.3 API and PAT authentication model you need to pass the team name and the new PAT into the existing apiAccessToken parameter to the new URL. See [15770](https://github.com/backstage/backstage/issues/15770) for more details.
+
+```yaml
+stackoverflow:
+  baseUrl: https://api.stackexchange.com/2.2 # alternative: your internal stack overflow instance
+  apiKey: $STACK_OVERFLOW_API_KEY
+  apiAccessToken: $STACK_OVERFLOW_API_ACCESS_TOKEN
+```
+
+```yaml
+stackoverflow:
+  baseUrl: https://api.stackoverflowteams.com/2.3 # alternative: your internal stack overflow instance
+  teamName: $STACK_OVERFLOW_TEAM_NAME
+  apiAccessToken: $STACK_OVERFLOW_API_ACCESS_TOKEN
+```
+
+## Areas of Responsibility
+
+This stack overflow backend plugin is primarily responsible for the following:
+
+- Provides a `StackOverflowQuestionsCollatorFactory`, which can be used in the search backend to index stack overflow questions to your Backstage Search.
+
+### Index Stack Overflow Questions to search
+
+Before you are able to start index stack overflow questions to search, you need to go through the [search getting started guide](https://backstage.io/docs/features/search/getting-started).
+
+When you have your `packages/backend/src/plugins/search.ts` file ready to make modifications, add the following code snippet to add the `StackOverflowQuestionsCollatorFactory`. Note that you can optionally modify the `requestParams`, otherwise it will defaults to `{ order: 'desc', sort: 'activity', site: 'stackoverflow' }` as done in the `Try It` section on the [official Stack Overflow API documentation](https://api.stackexchange.com/docs/questions).
+
+> Note: if your `baseUrl` is set to the external stack overflow api `https://api.stackexchange.com/2.2`, you can find optional and required parameters under the official API documentation under [`Usage of /questions GET`](https://api.stackexchange.com/docs/questions)
+
+```ts
+indexBuilder.addCollator({
+  schedule,
+  factory: StackOverflowQuestionsCollatorFactory.fromConfig(env.config, {
+    logger: env.logger,
+    requestParams: {
+      tagged: ['backstage'],
+      site: 'stackoverflow',
+      pagesize: 100,
+    },
+  }),
+});
+```
+
+## New Backend System
+
+> DISCLAIMER: The new backend system is in alpha, and so are the search backend module support for the new backend system. We don't recommend you to migrate your backend installations to the new system yet. But if you want to experiment, you can find getting started guides below.
+
+This package exports a module that extends the search backend to also indexing the questions exposed by the [`Stack Overflow` API](https://api.stackexchange.com/docs/questions).
+
+### Installation
+
+Add the module package as a dependency:
+
+```bash
+# From your Backstage root directory
+yarn add --cwd packages/backend @backstage/plugin-search-backend-module-stack-overflow-collator
+```
+
+Add the collator to your backend instance, along with the search plugin itself:
+
+```tsx
+// packages/backend/src/index.ts
+import { createBackend } from '@backstage/backend-defaults';
+
+const backend = createBackend();
+backend.add(import('@backstage/plugin-search-backend/alpha'));
+backend.add(
+  import('@backstage/plugin-search-backend-module-stack-overflow-collator'),
+);
+backend.start();
+```
+
+You may also want to add configuration parameters to your app-config, for example for controlling the scheduled indexing interval. These parameters should be placed under the `stackoverflow` key. See [the config definition file](https://github.com/backstage/backstage/blob/master/plugins/search-backend-module-stack-overflow-collator/config.d.ts) for more details.

package/config.d.ts

@@ -0,0 +1,51 @@
+/*
+ * Copyright 2023 The Backstage Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+export interface Config {
+  /**
+   * Configuration options for the stack overflow plugin
+   */
+  stackoverflow?: {
+    /**
+     * The base url of the Stack Overflow API used for the plugin
+     */
+    baseUrl?: string;
+
+    /**
+     * The API key to authenticate to Stack Overflow API
+     * @visibility secret
+     */
+    apiKey?: string;
+
+    /**
+     * The name of the team for a Stack Overflow for Teams account
+     */
+    teamName?: string;
+
+    /**
+     * The API Access Token to authenticate to Stack Overflow API
+     * @visibility secret
+     */
+    apiAccessToken?: string;
+
+    /**
+     * Type representing the request parameters.
+     */
+    requestParams?: {
+      [key: string]: string | string[] | number;
+    };
+  };
+}

package/dist/index.cjs.js

@@ -0,0 +1,167 @@
+'use strict';
+
+Object.defineProperty(exports, '__esModule', { value: true });
+
+var stream = require('stream');
+var fetch = require('node-fetch');
+var qs = require('qs');
+var backendCommon = require('@backstage/backend-common');
+var backendTasks = require('@backstage/backend-tasks');
+var backendPluginApi = require('@backstage/backend-plugin-api');
+var alpha = require('@backstage/plugin-search-backend-node/alpha');
+
+function _interopDefaultLegacy (e) { return e && typeof e === 'object' && 'default' in e ? e : { 'default': e }; }
+
+var fetch__default = /*#__PURE__*/_interopDefaultLegacy(fetch);
+var qs__default = /*#__PURE__*/_interopDefaultLegacy(qs);
+
+var __defProp = Object.defineProperty;
+var __defNormalProp = (obj, key, value) => key in obj ? __defProp(obj, key, { enumerable: true, configurable: true, writable: true, value }) : obj[key] = value;
+var __publicField = (obj, key, value) => {
+  __defNormalProp(obj, typeof key !== "symbol" ? key + "" : key, value);
+  return value;
+};
+class StackOverflowQuestionsCollatorFactory {
+  constructor(options) {
+    __publicField(this, "requestParams");
+    __publicField(this, "baseUrl");
+    __publicField(this, "apiKey");
+    __publicField(this, "apiAccessToken");
+    __publicField(this, "teamName");
+    __publicField(this, "maxPage");
+    __publicField(this, "logger");
+    __publicField(this, "type", "stack-overflow");
+    var _a, _b;
+    this.baseUrl = options.baseUrl;
+    this.apiKey = options.apiKey;
+    this.apiAccessToken = options.apiAccessToken;
+    this.teamName = options.teamName;
+    this.maxPage = options.maxPage;
+    this.requestParams = (_b = options.requestParams) != null ? _b : {
+      order: "desc",
+      sort: "activity",
+      site: "stackoverflow",
+      ...(_a = options.requestParams) != null ? _a : {}
+    };
+    this.logger = options.logger.child({ documentType: this.type });
+  }
+  static fromConfig(config, options) {
+    var _a;
+    const apiKey = config.getOptionalString("stackoverflow.apiKey");
+    const apiAccessToken = config.getOptionalString(
+      "stackoverflow.apiAccessToken"
+    );
+    const teamName = config.getOptionalString("stackoverflow.teamName");
+    const baseUrl = config.getOptionalString("stackoverflow.baseUrl") || "https://api.stackexchange.com/2.3";
+    const maxPage = options.maxPage || 100;
+    const requestParams = (_a = config.getOptionalConfig("stackoverflow.requestParams")) == null ? void 0 : _a.get();
+    return new StackOverflowQuestionsCollatorFactory({
+      baseUrl,
+      maxPage,
+      apiKey,
+      apiAccessToken,
+      teamName,
+      requestParams,
+      ...options
+    });
+  }
+  async getCollator() {
+    return stream.Readable.from(this.execute());
+  }
+  async *execute() {
+    var _a;
+    if (!this.baseUrl) {
+      this.logger.debug(
+        `No stackoverflow.baseUrl configured in your app-config.yaml`
+      );
+    }
+    if (this.apiKey && this.teamName) {
+      this.logger.debug(
+        "Both stackoverflow.apiKey and stackoverflow.teamName configured in your app-config.yaml, apiKey must be removed before teamName will be used"
+      );
+    }
+    try {
+      if (Object.keys(this.requestParams).indexOf("key") >= 0) {
+        this.logger.warn(
+          "The API Key should be passed as a separate param to bypass encoding"
+        );
+        delete this.requestParams.key;
+      }
+    } catch (e) {
+      this.logger.error(`Caught ${e}`);
+    }
+    const params = qs__default["default"].stringify(this.requestParams, {
+      arrayFormat: "comma",
+      addQueryPrefix: true
+    });
+    const apiKeyParam = this.apiKey ? `${params ? "&" : "?"}key=${this.apiKey}` : "";
+    const teamParam = this.teamName ? `${params ? "&" : "?"}team=${this.teamName}` : "";
+    const requestUrl = this.apiKey ? `${this.baseUrl}/questions${params}${apiKeyParam}` : `${this.baseUrl}/questions${params}${teamParam}`;
+    let hasMorePages = true;
+    let page = 1;
+    while (hasMorePages) {
+      if (page === this.maxPage) {
+        this.logger.warn(
+          `Over ${this.maxPage} requests to the Stack Overflow API have been made, which may not have been intended. Either specify requestParams that limit the questions returned, or configure a higher maxPage if necessary.`
+        );
+        break;
+      }
+      const res = await fetch__default["default"](
+        `${requestUrl}&page=${page}`,
+        this.apiAccessToken ? {
+          headers: {
+            "X-API-Access-Token": this.apiAccessToken
+          }
+        } : void 0
+      );
+      const data = await res.json();
+      for (const question of (_a = data.items) != null ? _a : []) {
+        yield {
+          title: question.title,
+          location: question.link,
+          text: question.owner.display_name,
+          tags: question.tags,
+          answers: question.answer_count
+        };
+      }
+      hasMorePages = data.has_more;
+      page = page + 1;
+    }
+  }
+}
+
+const searchStackOverflowCollatorModule = backendPluginApi.createBackendModule({
+  moduleId: "stackOverflowCollator",
+  pluginId: "search",
+  register(env) {
+    env.registerInit({
+      deps: {
+        config: backendPluginApi.coreServices.rootConfig,
+        logger: backendPluginApi.coreServices.logger,
+        discovery: backendPluginApi.coreServices.discovery,
+        scheduler: backendPluginApi.coreServices.scheduler,
+        indexRegistry: alpha.searchIndexRegistryExtensionPoint
+      },
+      async init({ config, logger, scheduler, indexRegistry }) {
+        const defaultSchedule = {
+          frequency: { minutes: 10 },
+          timeout: { minutes: 15 },
+          initialDelay: { seconds: 3 }
+        };
+        const schedule = config.has("stackoverflow.schedule") ? backendTasks.readTaskScheduleDefinitionFromConfig(
+          config.getConfig("stackoverflow.schedule")
+        ) : defaultSchedule;
+        indexRegistry.addCollator({
+          schedule: scheduler.createScheduledTaskRunner(schedule),
+          factory: StackOverflowQuestionsCollatorFactory.fromConfig(config, {
+            logger: backendCommon.loggerToWinstonLogger(logger)
+          })
+        });
+      }
+    });
+  }
+});
+
+exports.StackOverflowQuestionsCollatorFactory = StackOverflowQuestionsCollatorFactory;
+exports["default"] = searchStackOverflowCollatorModule;
+//# sourceMappingURL=index.cjs.js.map

package/dist/index.cjs.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.cjs.js","sources":["../src/collators/StackOverflowQuestionsCollatorFactory.ts","../src/module/SearchStackOverflowCollatorModule.ts"],"sourcesContent":["/*\n * Copyright 2022 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport {\n  IndexableDocument,\n  DocumentCollatorFactory,\n} from '@backstage/plugin-search-common';\nimport { Config } from '@backstage/config';\nimport { Readable } from 'stream';\nimport fetch from 'node-fetch';\nimport qs from 'qs';\nimport { Logger } from 'winston';\n\n/**\n * Extended IndexableDocument with stack overflow specific properties\n *\n * @public\n */\nexport interface StackOverflowDocument extends IndexableDocument {\n  answers: number;\n  tags: string[];\n}\n\n/**\n * Type representing the request parameters accepted by the {@link StackOverflowQuestionsCollatorFactory}\n *\n * @public\n */\nexport type StackOverflowQuestionsRequestParams = {\n  [key: string]: string | string[] | number;\n};\n\n/**\n * Options for {@link StackOverflowQuestionsCollatorFactory}\n *\n * @public\n */\nexport type StackOverflowQuestionsCollatorFactoryOptions = {\n  baseUrl?: string;\n  maxPage?: number;\n  apiKey?: string;\n  apiAccessToken?: string;\n  teamName?: string;\n  requestParams?: StackOverflowQuestionsRequestParams;\n  logger: Logger;\n};\n\n/**\n * Search collator responsible for collecting stack overflow questions to index.\n *\n * @public\n */\nexport class StackOverflowQuestionsCollatorFactory\n  implements DocumentCollatorFactory\n{\n  protected requestParams: StackOverflowQuestionsRequestParams;\n  private readonly baseUrl: string | undefined;\n  private readonly apiKey: string | undefined;\n  private readonly apiAccessToken: string | undefined;\n  private readonly teamName: string | undefined;\n  private readonly maxPage: number | undefined;\n  private readonly logger: Logger;\n  public readonly type: string = 'stack-overflow';\n\n  private constructor(options: StackOverflowQuestionsCollatorFactoryOptions) {\n    this.baseUrl = options.baseUrl;\n    this.apiKey = options.apiKey;\n    this.apiAccessToken = options.apiAccessToken;\n    this.teamName = options.teamName;\n    this.maxPage = options.maxPage;\n    // Sets the same default request parameters as the official API documentation\n    // See https://api.stackexchange.com/docs/questions\n    this.requestParams = options.requestParams ?? {\n      order: 'desc',\n      sort: 'activity',\n      site: 'stackoverflow',\n      ...(options.requestParams ?? {}),\n    };\n    this.logger = options.logger.child({ documentType: this.type });\n  }\n\n  static fromConfig(\n    config: Config,\n    options: StackOverflowQuestionsCollatorFactoryOptions,\n  ) {\n    const apiKey = config.getOptionalString('stackoverflow.apiKey');\n    const apiAccessToken = config.getOptionalString(\n      'stackoverflow.apiAccessToken',\n    );\n    const teamName = config.getOptionalString('stackoverflow.teamName');\n    const baseUrl =\n      config.getOptionalString('stackoverflow.baseUrl') ||\n      'https://api.stackexchange.com/2.3';\n    const maxPage = options.maxPage || 100;\n    const requestParams = config\n      .getOptionalConfig('stackoverflow.requestParams')\n      ?.get<StackOverflowQuestionsRequestParams>();\n    return new StackOverflowQuestionsCollatorFactory({\n      baseUrl,\n      maxPage,\n      apiKey,\n      apiAccessToken,\n      teamName,\n      requestParams,\n      ...options,\n    });\n  }\n\n  async getCollator() {\n    return Readable.from(this.execute());\n  }\n\n  async *execute(): AsyncGenerator<StackOverflowDocument> {\n    if (!this.baseUrl) {\n      this.logger.debug(\n        `No stackoverflow.baseUrl configured in your app-config.yaml`,\n      );\n    }\n\n    if (this.apiKey && this.teamName) {\n      this.logger.debug(\n        'Both stackoverflow.apiKey and stackoverflow.teamName configured in your app-config.yaml, apiKey must be removed before teamName will be used',\n      );\n    }\n\n    try {\n      if (Object.keys(this.requestParams).indexOf('key') >= 0) {\n        this.logger.warn(\n          'The API Key should be passed as a separate param to bypass encoding',\n        );\n        delete this.requestParams.key;\n      }\n    } catch (e) {\n      this.logger.error(`Caught ${e}`);\n    }\n\n    const params = qs.stringify(this.requestParams, {\n      arrayFormat: 'comma',\n      addQueryPrefix: true,\n    });\n\n    const apiKeyParam = this.apiKey\n      ? `${params ? '&' : '?'}key=${this.apiKey}`\n      : '';\n\n    const teamParam = this.teamName\n      ? `${params ? '&' : '?'}team=${this.teamName}`\n      : '';\n\n    // PAT change requires team name as a parameter\n    const requestUrl = this.apiKey\n      ? `${this.baseUrl}/questions${params}${apiKeyParam}`\n      : `${this.baseUrl}/questions${params}${teamParam}`;\n\n    let hasMorePages = true;\n    let page = 1;\n    while (hasMorePages) {\n      if (page === this.maxPage) {\n        this.logger.warn(\n          `Over ${this.maxPage} requests to the Stack Overflow API have been made, which may not have been intended. Either specify requestParams that limit the questions returned, or configure a higher maxPage if necessary.`,\n        );\n        break;\n      }\n      const res = await fetch(\n        `${requestUrl}&page=${page}`,\n        this.apiAccessToken\n          ? {\n              headers: {\n                'X-API-Access-Token': this.apiAccessToken,\n              },\n            }\n          : undefined,\n      );\n\n      const data = await res.json();\n      for (const question of data.items ?? []) {\n        yield {\n          title: question.title,\n          location: question.link,\n          text: question.owner.display_name,\n          tags: question.tags,\n          answers: question.answer_count,\n        };\n      }\n      hasMorePages = data.has_more;\n      page = page + 1;\n    }\n  }\n}\n","/*\n * Copyright 2023 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { loggerToWinstonLogger } from '@backstage/backend-common';\nimport { readTaskScheduleDefinitionFromConfig } from '@backstage/backend-tasks';\nimport {\n  coreServices,\n  createBackendModule,\n} from '@backstage/backend-plugin-api';\nimport { searchIndexRegistryExtensionPoint } from '@backstage/plugin-search-backend-node/alpha';\nimport { StackOverflowQuestionsCollatorFactory } from '../collators';\n\n/**\n * @public\n * Search backend module for the Stack Overflow index.\n */\nexport const searchStackOverflowCollatorModule = createBackendModule({\n  moduleId: 'stackOverflowCollator',\n  pluginId: 'search',\n  register(env) {\n    env.registerInit({\n      deps: {\n        config: coreServices.rootConfig,\n        logger: coreServices.logger,\n        discovery: coreServices.discovery,\n        scheduler: coreServices.scheduler,\n        indexRegistry: searchIndexRegistryExtensionPoint,\n      },\n      async init({ config, logger, scheduler, indexRegistry }) {\n        const defaultSchedule = {\n          frequency: { minutes: 10 },\n          timeout: { minutes: 15 },\n          initialDelay: { seconds: 3 },\n        };\n\n        const schedule = config.has('stackoverflow.schedule')\n          ? readTaskScheduleDefinitionFromConfig(\n              config.getConfig('stackoverflow.schedule'),\n            )\n          : defaultSchedule;\n\n        indexRegistry.addCollator({\n          schedule: scheduler.createScheduledTaskRunner(schedule),\n          factory: StackOverflowQuestionsCollatorFactory.fromConfig(config, {\n            logger: loggerToWinstonLogger(logger),\n          }),\n        });\n      },\n    });\n  },\n});\n"],"names":["Readable","qs","fetch","createBackendModule","coreServices","searchIndexRegistryExtensionPoint","readTaskScheduleDefinitionFromConfig","loggerToWinstonLogger"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;AAiEO,MAAM,qCAEb,CAAA;AAAA,EAUU,YAAY,OAAuD,EAAA;AAT3E,IAAU,aAAA,CAAA,IAAA,EAAA,eAAA,CAAA,CAAA;AACV,IAAiB,aAAA,CAAA,IAAA,EAAA,SAAA,CAAA,CAAA;AACjB,IAAiB,aAAA,CAAA,IAAA,EAAA,QAAA,CAAA,CAAA;AACjB,IAAiB,aAAA,CAAA,IAAA,EAAA,gBAAA,CAAA,CAAA;AACjB,IAAiB,aAAA,CAAA,IAAA,EAAA,UAAA,CAAA,CAAA;AACjB,IAAiB,aAAA,CAAA,IAAA,EAAA,SAAA,CAAA,CAAA;AACjB,IAAiB,aAAA,CAAA,IAAA,EAAA,QAAA,CAAA,CAAA;AACjB,IAAA,aAAA,CAAA,IAAA,EAAgB,MAAe,EAAA,gBAAA,CAAA,CAAA;AA3EjC,IAAA,IAAA,EAAA,EAAA,EAAA,CAAA;AA8EI,IAAA,IAAA,CAAK,UAAU,OAAQ,CAAA,OAAA,CAAA;AACvB,IAAA,IAAA,CAAK,SAAS,OAAQ,CAAA,MAAA,CAAA;AACtB,IAAA,IAAA,CAAK,iBAAiB,OAAQ,CAAA,cAAA,CAAA;AAC9B,IAAA,IAAA,CAAK,WAAW,OAAQ,CAAA,QAAA,CAAA;AACxB,IAAA,IAAA,CAAK,UAAU,OAAQ,CAAA,OAAA,CAAA;AAGvB,IAAK,IAAA,CAAA,aAAA,GAAA,CAAgB,EAAQ,GAAA,OAAA,CAAA,aAAA,KAAR,IAAyB,GAAA,EAAA,GAAA;AAAA,MAC5C,KAAO,EAAA,MAAA;AAAA,MACP,IAAM,EAAA,UAAA;AAAA,MACN,IAAM,EAAA,eAAA;AAAA,MACN,GAAI,CAAA,EAAA,GAAA,OAAA,CAAQ,aAAR,KAAA,IAAA,GAAA,EAAA,GAAyB,EAAC;AAAA,KAChC,CAAA;AACA,IAAK,IAAA,CAAA,MAAA,GAAS,QAAQ,MAAO,CAAA,KAAA,CAAM,EAAE,YAAc,EAAA,IAAA,CAAK,MAAM,CAAA,CAAA;AAAA,GAChE;AAAA,EAEA,OAAO,UACL,CAAA,MAAA,EACA,OACA,EAAA;AAjGJ,IAAA,IAAA,EAAA,CAAA;AAkGI,IAAM,MAAA,MAAA,GAAS,MAAO,CAAA,iBAAA,CAAkB,sBAAsB,CAAA,CAAA;AAC9D,IAAA,MAAM,iBAAiB,MAAO,CAAA,iBAAA;AAAA,MAC5B,8BAAA;AAAA,KACF,CAAA;AACA,IAAM,MAAA,QAAA,GAAW,MAAO,CAAA,iBAAA,CAAkB,wBAAwB,CAAA,CAAA;AAClE,IAAA,MAAM,OACJ,GAAA,MAAA,CAAO,iBAAkB,CAAA,uBAAuB,CAChD,IAAA,mCAAA,CAAA;AACF,IAAM,MAAA,OAAA,GAAU,QAAQ,OAAW,IAAA,GAAA,CAAA;AACnC,IAAA,MAAM,aAAgB,GAAA,CAAA,EAAA,GAAA,MAAA,CACnB,iBAAkB,CAAA,6BAA6B,MAD5B,IAElB,GAAA,KAAA,CAAA,GAAA,EAAA,CAAA,GAAA,EAAA,CAAA;AACJ,IAAA,OAAO,IAAI,qCAAsC,CAAA;AAAA,MAC/C,OAAA;AAAA,MACA,OAAA;AAAA,MACA,MAAA;AAAA,MACA,cAAA;AAAA,MACA,QAAA;AAAA,MACA,aAAA;AAAA,MACA,GAAG,OAAA;AAAA,KACJ,CAAA,CAAA;AAAA,GACH;AAAA,EAEA,MAAM,WAAc,GAAA;AAClB,IAAA,OAAOA,eAAS,CAAA,IAAA,CAAK,IAAK,CAAA,OAAA,EAAS,CAAA,CAAA;AAAA,GACrC;AAAA,EAEA,OAAO,OAAiD,GAAA;AA7H1D,IAAA,IAAA,EAAA,CAAA;AA8HI,IAAI,IAAA,CAAC,KAAK,OAAS,EAAA;AACjB,MAAA,IAAA,CAAK,MAAO,CAAA,KAAA;AAAA,QACV,CAAA,2DAAA,CAAA;AAAA,OACF,CAAA;AAAA,KACF;AAEA,IAAI,IAAA,IAAA,CAAK,MAAU,IAAA,IAAA,CAAK,QAAU,EAAA;AAChC,MAAA,IAAA,CAAK,MAAO,CAAA,KAAA;AAAA,QACV,8IAAA;AAAA,OACF,CAAA;AAAA,KACF;AAEA,IAAI,IAAA;AACF,MAAI,IAAA,MAAA,CAAO,KAAK,IAAK,CAAA,aAAa,EAAE,OAAQ,CAAA,KAAK,KAAK,CAAG,EAAA;AACvD,QAAA,IAAA,CAAK,MAAO,CAAA,IAAA;AAAA,UACV,qEAAA;AAAA,SACF,CAAA;AACA,QAAA,OAAO,KAAK,aAAc,CAAA,GAAA,CAAA;AAAA,OAC5B;AAAA,aACO,CAAG,EAAA;AACV,MAAA,IAAA,CAAK,MAAO,CAAA,KAAA,CAAM,CAAU,OAAA,EAAA,CAAC,CAAE,CAAA,CAAA,CAAA;AAAA,KACjC;AAEA,IAAA,MAAM,MAAS,GAAAC,sBAAA,CAAG,SAAU,CAAA,IAAA,CAAK,aAAe,EAAA;AAAA,MAC9C,WAAa,EAAA,OAAA;AAAA,MACb,cAAgB,EAAA,IAAA;AAAA,KACjB,CAAA,CAAA;AAED,IAAM,MAAA,WAAA,GAAc,IAAK,CAAA,MAAA,GACrB,CAAG,EAAA,MAAA,GAAS,MAAM,GAAG,CAAA,IAAA,EAAO,IAAK,CAAA,MAAM,CACvC,CAAA,GAAA,EAAA,CAAA;AAEJ,IAAM,MAAA,SAAA,GAAY,IAAK,CAAA,QAAA,GACnB,CAAG,EAAA,MAAA,GAAS,MAAM,GAAG,CAAA,KAAA,EAAQ,IAAK,CAAA,QAAQ,CAC1C,CAAA,GAAA,EAAA,CAAA;AAGJ,IAAA,MAAM,aAAa,IAAK,CAAA,MAAA,GACpB,CAAG,EAAA,IAAA,CAAK,OAAO,CAAa,UAAA,EAAA,MAAM,CAAG,EAAA,WAAW,KAChD,CAAG,EAAA,IAAA,CAAK,OAAO,CAAa,UAAA,EAAA,MAAM,GAAG,SAAS,CAAA,CAAA,CAAA;AAElD,IAAA,IAAI,YAAe,GAAA,IAAA,CAAA;AACnB,IAAA,IAAI,IAAO,GAAA,CAAA,CAAA;AACX,IAAA,OAAO,YAAc,EAAA;AACnB,MAAI,IAAA,IAAA,KAAS,KAAK,OAAS,EAAA;AACzB,QAAA,IAAA,CAAK,MAAO,CAAA,IAAA;AAAA,UACV,CAAA,KAAA,EAAQ,KAAK,OAAO,CAAA,iMAAA,CAAA;AAAA,SACtB,CAAA;AACA,QAAA,MAAA;AAAA,OACF;AACA,MAAA,MAAM,MAAM,MAAMC,yBAAA;AAAA,QAChB,CAAA,EAAG,UAAU,CAAA,MAAA,EAAS,IAAI,CAAA,CAAA;AAAA,QAC1B,KAAK,cACD,GAAA;AAAA,UACE,OAAS,EAAA;AAAA,YACP,sBAAsB,IAAK,CAAA,cAAA;AAAA,WAC7B;AAAA,SAEF,GAAA,KAAA,CAAA;AAAA,OACN,CAAA;AAEA,MAAM,MAAA,IAAA,GAAO,MAAM,GAAA,CAAI,IAAK,EAAA,CAAA;AAC5B,MAAA,KAAA,MAAW,QAAY,IAAA,CAAA,EAAA,GAAA,IAAA,CAAK,KAAL,KAAA,IAAA,GAAA,EAAA,GAAc,EAAI,EAAA;AACvC,QAAM,MAAA;AAAA,UACJ,OAAO,QAAS,CAAA,KAAA;AAAA,UAChB,UAAU,QAAS,CAAA,IAAA;AAAA,UACnB,IAAA,EAAM,SAAS,KAAM,CAAA,YAAA;AAAA,UACrB,MAAM,QAAS,CAAA,IAAA;AAAA,UACf,SAAS,QAAS,CAAA,YAAA;AAAA,SACpB,CAAA;AAAA,OACF;AACA,MAAA,YAAA,GAAe,IAAK,CAAA,QAAA,CAAA;AACpB,MAAA,IAAA,GAAO,IAAO,GAAA,CAAA,CAAA;AAAA,KAChB;AAAA,GACF;AACF;;AC5KO,MAAM,oCAAoCC,oCAAoB,CAAA;AAAA,EACnE,QAAU,EAAA,uBAAA;AAAA,EACV,QAAU,EAAA,QAAA;AAAA,EACV,SAAS,GAAK,EAAA;AACZ,IAAA,GAAA,CAAI,YAAa,CAAA;AAAA,MACf,IAAM,EAAA;AAAA,QACJ,QAAQC,6BAAa,CAAA,UAAA;AAAA,QACrB,QAAQA,6BAAa,CAAA,MAAA;AAAA,QACrB,WAAWA,6BAAa,CAAA,SAAA;AAAA,QACxB,WAAWA,6BAAa,CAAA,SAAA;AAAA,QACxB,aAAe,EAAAC,uCAAA;AAAA,OACjB;AAAA,MACA,MAAM,IAAK,CAAA,EAAE,QAAQ,MAAQ,EAAA,SAAA,EAAW,eAAiB,EAAA;AACvD,QAAA,MAAM,eAAkB,GAAA;AAAA,UACtB,SAAA,EAAW,EAAE,OAAA,EAAS,EAAG,EAAA;AAAA,UACzB,OAAA,EAAS,EAAE,OAAA,EAAS,EAAG,EAAA;AAAA,UACvB,YAAA,EAAc,EAAE,OAAA,EAAS,CAAE,EAAA;AAAA,SAC7B,CAAA;AAEA,QAAA,MAAM,QAAW,GAAA,MAAA,CAAO,GAAI,CAAA,wBAAwB,CAChD,GAAAC,iDAAA;AAAA,UACE,MAAA,CAAO,UAAU,wBAAwB,CAAA;AAAA,SAE3C,GAAA,eAAA,CAAA;AAEJ,QAAA,aAAA,CAAc,WAAY,CAAA;AAAA,UACxB,QAAA,EAAU,SAAU,CAAA,yBAAA,CAA0B,QAAQ,CAAA;AAAA,UACtD,OAAA,EAAS,qCAAsC,CAAA,UAAA,CAAW,MAAQ,EAAA;AAAA,YAChE,MAAA,EAAQC,oCAAsB,MAAM,CAAA;AAAA,WACrC,CAAA;AAAA,SACF,CAAA,CAAA;AAAA,OACH;AAAA,KACD,CAAA,CAAA;AAAA,GACH;AACF,CAAC;;;;;"}

package/dist/index.d.ts

@@ -0,0 +1,65 @@
+/// <reference types="node" />
+import { IndexableDocument, DocumentCollatorFactory } from '@backstage/plugin-search-common';
+import { Config } from '@backstage/config';
+import { Readable } from 'stream';
+import { Logger } from 'winston';
+import * as _backstage_backend_plugin_api from '@backstage/backend-plugin-api';
+
+/**
+ * Extended IndexableDocument with stack overflow specific properties
+ *
+ * @public
+ */
+interface StackOverflowDocument extends IndexableDocument {
+    answers: number;
+    tags: string[];
+}
+/**
+ * Type representing the request parameters accepted by the {@link StackOverflowQuestionsCollatorFactory}
+ *
+ * @public
+ */
+type StackOverflowQuestionsRequestParams = {
+    [key: string]: string | string[] | number;
+};
+/**
+ * Options for {@link StackOverflowQuestionsCollatorFactory}
+ *
+ * @public
+ */
+type StackOverflowQuestionsCollatorFactoryOptions = {
+    baseUrl?: string;
+    maxPage?: number;
+    apiKey?: string;
+    apiAccessToken?: string;
+    teamName?: string;
+    requestParams?: StackOverflowQuestionsRequestParams;
+    logger: Logger;
+};
+/**
+ * Search collator responsible for collecting stack overflow questions to index.
+ *
+ * @public
+ */
+declare class StackOverflowQuestionsCollatorFactory implements DocumentCollatorFactory {
+    protected requestParams: StackOverflowQuestionsRequestParams;
+    private readonly baseUrl;
+    private readonly apiKey;
+    private readonly apiAccessToken;
+    private readonly teamName;
+    private readonly maxPage;
+    private readonly logger;
+    readonly type: string;
+    private constructor();
+    static fromConfig(config: Config, options: StackOverflowQuestionsCollatorFactoryOptions): StackOverflowQuestionsCollatorFactory;
+    getCollator(): Promise<Readable>;
+    execute(): AsyncGenerator<StackOverflowDocument>;
+}
+
+/**
+ * @public
+ * Search backend module for the Stack Overflow index.
+ */
+declare const searchStackOverflowCollatorModule: () => _backstage_backend_plugin_api.BackendFeature;
+
+export { StackOverflowDocument, StackOverflowQuestionsCollatorFactory, StackOverflowQuestionsCollatorFactoryOptions, StackOverflowQuestionsRequestParams, searchStackOverflowCollatorModule as default };

package/package.json

@@ -0,0 +1,52 @@
+{
+  "name": "@backstage/plugin-search-backend-module-stack-overflow-collator",
+  "description": "A module for the search backend that exports stack overflow modules",
+  "version": "0.0.0-nightly-20231020021216",
+  "main": "dist/index.cjs.js",
+  "types": "dist/index.d.ts",
+  "license": "Apache-2.0",
+  "publishConfig": {
+    "access": "public",
+    "main": "dist/index.cjs.js",
+    "types": "dist/index.d.ts"
+  },
+  "backstage": {
+    "role": "backend-plugin-module"
+  },
+  "homepage": "https://backstage.io",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/backstage/backstage",
+    "directory": "plugins/search-backend-module-stack-overflow"
+  },
+  "scripts": {
+    "start": "backstage-cli package start",
+    "build": "backstage-cli package build",
+    "lint": "backstage-cli package lint",
+    "test": "backstage-cli package test",
+    "prepack": "backstage-cli package prepack",
+    "postpack": "backstage-cli package postpack",
+    "clean": "backstage-cli package clean"
+  },
+  "dependencies": {
+    "@backstage/backend-common": "^0.0.0-nightly-20231020021216",
+    "@backstage/backend-plugin-api": "^0.0.0-nightly-20231020021216",
+    "@backstage/backend-tasks": "^0.0.0-nightly-20231020021216",
+    "@backstage/config": "^1.1.1",
+    "@backstage/plugin-search-backend-node": "^0.0.0-nightly-20231020021216",
+    "@backstage/plugin-search-common": "^1.2.7",
+    "node-fetch": "^2.6.7",
+    "qs": "^6.9.4",
+    "winston": "^3.2.1"
+  },
+  "devDependencies": {
+    "@backstage/backend-test-utils": "^0.0.0-nightly-20231020021216",
+    "@backstage/cli": "^0.0.0-nightly-20231020021216",
+    "msw": "^1.2.1"
+  },
+  "files": [
+    "dist",
+    "config.d.ts"
+  ],
+  "configSchema": "config.d.ts"
+}