@backstage/plugin-catalog-backend-module-incremental-ingestion 0.2.0-next.1 → 0.2.0

package/CHANGELOG.md

@@ -1,5 +1,49 @@
 # @backstage/plugin-catalog-backend-module-incremental-ingestion
 
+## 0.2.0
+
+### Minor Changes
+
+- 1ba120faa3: Added new mechanism to handle deltas in incremental providers
+
+### Patch Changes
+
+- c51efce2a0: Update docs to always use `yarn add --cwd` for app & backend
+- 407dc01fc9: Removing extra imports for `run` script as `TestBackend` auto loads the default factories
+- b7e36660d5: Return `EventSubscriber` from `addIncrementalEntityProvider` to hook up to `EventsBackend`
+- 5b7cd5580d: Moving the backend-test-utils to devDependencies.
+- 77c41b6924: Updated README to include newer API options for incremental entity providers
+- Updated dependencies
+  - @backstage/plugin-catalog-backend@1.7.2
+  - @backstage/backend-plugin-api@0.4.0
+  - @backstage/backend-common@0.18.2
+  - @backstage/catalog-model@1.2.0
+  - @backstage/plugin-events-node@0.2.3
+  - @backstage/plugin-catalog-node@1.3.3
+  - @backstage/backend-tasks@0.4.3
+  - @backstage/config@1.0.6
+  - @backstage/errors@1.1.4
+  - @backstage/plugin-permission-common@0.7.3
+
+## 0.2.0-next.2
+
+### Patch Changes
+
+- 407dc01fc9: Removing extra imports for `run` script as `TestBackend` auto loads the default factories
+- 77c41b6924: Updated README to include newer API options for incremental entity providers
+- Updated dependencies
+  - @backstage/backend-plugin-api@0.4.0-next.2
+  - @backstage/backend-test-utils@0.1.34-next.2
+  - @backstage/backend-common@0.18.2-next.2
+  - @backstage/plugin-catalog-backend@1.7.2-next.2
+  - @backstage/catalog-model@1.2.0-next.1
+  - @backstage/plugin-events-node@0.2.3-next.2
+  - @backstage/plugin-catalog-node@1.3.3-next.2
+  - @backstage/backend-tasks@0.4.3-next.2
+  - @backstage/config@1.0.6
+  - @backstage/errors@1.1.4
+  - @backstage/plugin-permission-common@0.7.3
+
 ## 0.2.0-next.1
 
 ### Patch Changes

package/README.md

@@ -4,7 +4,7 @@ The Incremental Ingestion catalog backend module provides an Incremental Entity
 
 ## Why did we create it?
 
-Backstage provides an [Entity Provider mechanism that has two kinds of mutations](https://backstage.io/docs/features/software-catalog/external-integrations#provider-mutations): `delta` and `full`. `delta` mutations tell Backstage Catalog which entities should be added and removed from the catalog. `full` mutation accepts a list of entities and automatically computes which entities must be removed by comparing the provided entities against existing entities to create a diff between the two sets. These two kinds of mutations are convenient for different kinds of data sources. A `delta` mutation can be used with a data source that emits UPDATE and DELETE events for its data. A `full` mutation is useful for APIs that produce fewer entities than can fit in Backstage processes' memory.
+Backstage provides an [Entity Provider mechanism that has two kinds of mutations](https://backstage.io/docs/features/software-catalog/external-integrations#provider-mutations): `delta` and `full`. `delta` mutations tell the Backstage Catalog which entities should be added and removed from the catalog. `full` mutations accept a list of entities and automatically computes which entities must be removed by comparing the provided entities against existing entities to create a diff between the two sets. These two kinds of mutations are convenient for different kinds of data sources. A `delta` mutation can be used with a data source that emits UPDATE and DELETE events for its data. A `full` mutation is useful for APIs that produce fewer entities than can fit in the Backstage processes' memory.
 
 Unfortunately, these two kinds of mutations are insufficient for very large data sources for the following reasons,
 
@@ -41,69 +41,81 @@ The Incremental Entity Provider backend is designed for data sources that provid
 
 ## Installation
 
-1. Install `@backstage/plugin-catalog-backend-module-incremental-ingestion` with `yarn workspace backend add @backstage/plugin-catalog-backend-module-incremental-ingestion`
-2. Import `IncrementalCatalogBuilder` from `@backstage/plugin-catalog-backend-module-incremental-ingestion` and instantiate it with `await IncrementalCatalogBuilder.create(env, builder)`. You have to pass `builder` into `IncrementalCatalogBuilder.create` function because `IncrementalCatalogBuilder` will convert an `IncrementalEntityProvider` into an `EntityProvider` and call `builder.addEntityProvider`.
+1. Install `@backstage/plugin-catalog-backend-module-incremental-ingestion` with `yarn add --cwd packages/backend @backstage/plugin-catalog-backend-module-incremental-ingestion` from the Backstage root directory.
+2. In your catalog.ts, import `IncrementalCatalogBuilder` from `@backstage/plugin-catalog-backend-module-incremental-ingestion` and instantiate it with `await IncrementalCatalogBuilder.create(env, builder)`. You have to pass `builder` into `IncrementalCatalogBuilder.create` function because `IncrementalCatalogBuilder` will convert an `IncrementalEntityProvider` into an `EntityProvider` and call `builder.addEntityProvider`.
 
-   ```ts
-   const builder = CatalogBuilder.create(env);
-   // incremental builder receives builder because it'll register
-   // incremental entity providers with the builder
-   const incrementalBuilder = await IncrementalCatalogBuilder.create(
-     env,
-     builder,
-   );
-   ```
-
-3. Last step, add `await incrementBuilder.build()` after `await builder.build()` to ensure that all `CatalogBuilder` migration run before running `incrementBuilder.build()` migrations.
-
-   ```ts
-   const { processingEngine, router } = await builder.build();
-
-   // this has to run after `await builder.build()` to ensure that catalog migrations are completed
-   // before incremental builder migrations are executed
-   await incrementalBuilder.build();
-   ```
+```ts
+const builder = CatalogBuilder.create(env);
+// incremental builder receives builder because it'll register
+// incremental entity providers with the builder
+const incrementalBuilder = await IncrementalCatalogBuilder.create(env, builder);
+```
 
-   The result should look something like this,
+3. After building the regular `CatalogBuilder`, build the incremental builder:
 
-   ```ts
-   import { CatalogBuilder } from '@backstage/plugin-catalog-backend';
-   import { ScaffolderEntitiesProcessor } from '@backstage/plugin-scaffolder-backend';
-   import { IncrementalCatalogBuilder } from '@backstage/plugin-catalog-backend-module-incremental-ingestion';
-   import { Router } from 'express';
-   import { Duration } from 'luxon';
-   import { PluginEnvironment } from '../types';
+```ts
+// Must be run first to ensure CatalogBuilder database migrations run before Incremental Entity Provider database migrations
+const { processingEngine, router } = await builder.build();
 
-   export default async function createPlugin(
-     env: PluginEnvironment,
-   ): Promise<Router> {
-     const builder = CatalogBuilder.create(env);
-     // incremental builder receives builder because it'll register
-     // incremental entity providers with the builder
-     const incrementalBuilder = await IncrementalCatalogBuilder.create(
-       env,
-       builder,
-     );
+// Returns an optional - but highly recommended - set of administrative routes
+const { incrementalAdminRouter } = await incrementBuilder.build();
+```
 
-     builder.addProcessor(new ScaffolderEntitiesProcessor());
+The final result should look something like this,
 
-     const { processingEngine, router } = await builder.build();
+```ts
+import { CatalogBuilder } from '@backstage/plugin-catalog-backend';
+import { ScaffolderEntitiesProcessor } from '@backstage/plugin-scaffolder-backend';
+import { IncrementalCatalogBuilder } from '@backstage/plugin-catalog-backend-module-incremental-ingestion';
+import { Router } from 'express';
+import { Duration } from 'luxon';
+import { PluginEnvironment } from '../types';
+
+export default async function createPlugin(
+  env: PluginEnvironment,
+): Promise<Router> {
+  const builder = CatalogBuilder.create(env);
+  // incremental builder receives builder because it'll register
+  // incremental entity providers with the builder
+  const incrementalBuilder = await IncrementalCatalogBuilder.create(
+    env,
+    builder,
+  );
+
+  builder.addProcessor(new ScaffolderEntitiesProcessor());
+
+  const { processingEngine, router } = await builder.build();
+  const { incrementalAdminRouter } = await incrementalBuilder.build();
+
+  router.use(incrementalAdminRouter);
+
+  await processingEngine.start();
+
+  return router;
+}
+```
 
-     // this has to run after `await builder.build()` so ensure that catalog migrations are completed
-     // before incremental builder migrations are executed
-     const { incrementalAdminRouter } = await incrementalBuilder.build();
+## Administrative Routes
 
-     router.use('/incremental', incrementalAdminRouter);
+If you want to manage your incremental entity providers via REST endpoints, the following endpoints are available:
 
-     await processingEngine.start();
+| Method | Path                                       | Description                                                                                                                 |
+| ------ | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------- |
+| GET    | `/incremental/health`                      | Checks the health of all incremental providers. Returns array of any unhealthy ones.                                        |
+| GET    | `/incremental/providers/:provider`         | Checks the status of an incremental provider (resting, interstitial, etc).                                                  |
+| POST   | `/incremental/providers/:provider/trigger` | Triggers a provider's next action immediately. E.g., if it's currently interstitial, it will trigger the next burst.        |
+| POST   | `/incremental/providers/:provider/start`   | Stop the current ingestion cycle and start a new one immediately.                                                           |
+| POST   | `/incremental/providers/:provider/cancel`  | Stop the current ingestion cycle and start a new one in 24 hours.                                                           |
+| DELETE | `/incremental/providers/:provider`         | Completely remove all records for the provider and schedule it to start again in 24 hours.                                  |
+| GET    | `/incremental/providers/:provider/marks`   | Retrieve a list of all ingestion marks for the current ingestion cycle.                                                     |
+| DELETE | `/incremental/providers/:provider/marks`   | Remove all ingestion marks for the current ingestion cycle.                                                                 |
+| POST   | `/incremental/cleanup`                     | Completely remove all records for ALL providers and schedule them to start again in 24 hours. (CAUTION! Can cause orphans!) |
 
-     return router;
-   }
-   ```
+In all cases, `:provider` is the name of the incremental entity provider.
 
 ## Writing an Incremental Entity Provider
 
-To create an Incremental Entity Provider, you need to know how to retrieve a single page of the data that you wish to ingest into the Backstage catalog. If the API has pagination and you know how to make a paginated request to that API, you'll be able to implement an Incremental Entity Provider for this API. For more information about compatibility, checkout <a href="#compatible-data-source">Compatible data sources</a> section on this page.
+To create an Incremental Entity Provider, you need to know how to retrieve a single page of the data that you wish to ingest into the Backstage catalog. If the API has pagination and you know how to make a paginated request to that API, you'll be able to implement an Incremental Entity Provider for this API. For more information about compatibility, check out the <a href="#requirements">requirements</a> section of this page.
 
 Here is the type definition for an Incremental Entity Provider.
 
@@ -114,33 +126,31 @@ interface IncrementalEntityProvider<TCursor, TContext> {
    * operating in the catalog.
    */
   getProviderName(): string;
-
-  /**
-   * Do any setup and teardown necessary in order to provide the
-   * context for fetching pages. This should always invoke `burst` in
-   * order to fetch the individual pages.
-   *
-   * @param burst - a function which performs a series of iterations
-   */
-  around(burst: (context: TContext) => Promise<void>): Promise<void>;
-
   /**
    * Return a single page of entities from a specific point in the
    * ingestion.
    *
    * @param context - anything needed in order to fetch a single page.
    * @param cursor - a unique value identifying the page to ingest.
-   * @returns the entities to be ingested, as well as the cursor of
-   * the the next page after this one.
+   * @returns The entities to be ingested, as well as the cursor of
+   * the next page after this one.
    */
   next(
     context: TContext,
     cursor?: TCursor,
   ): Promise<EntityIteratorResult<TCursor>>;
+  /**
+   * Do any setup and teardown necessary in order to provide the
+   * context for fetching pages. This should always invoke `burst` in
+   * order to fetch the individual pages.
+   *
+   * @param burst - a function which performs a series of iterations
+   */
+  around(burst: (context: TContext) => Promise<void>): Promise<void>;
 }
 ```
 
-For tutorial, we'll write an Incremental Entity Provider that will call an imaginary API. This imaginary API will return a list of imaginary services. This imaginary API has an imaginary API client with the following interface.
+For this tutorial, we'll write an Incremental Entity Provider that will call an imaginary API. This imaginary API will return a list of imaginary services. The imaginary API has an imaginary API client with the following interface.
 
 ```ts
 interface MyApiClient {
@@ -157,15 +167,12 @@ interface Service {
 }
 ```
 
-These are the only 3 methods that you need to implement. `getProviderName()` is pretty self explanatory and it's exactly same as on Entity Provider.
+These are the only 3 methods that you need to implement. `getProviderName()` is pretty self explanatory and it's identical to the `getProviderName()` method on a regular Entity Provider.
 
 ```ts
-import {
-  IncrementalEntityProvider,
-  EntityIteratorResult,
-} from '@backstage/plugin-catalog-backend-module-incremental-ingestion';
+import { IncrementalEntityProvider } from '@backstage/plugin-catalog-backend-module-incremental-ingestion';
 
-// this will include your pagination information, let's say our API accepts a `page` parameter.
+// This will include your pagination information, let's say our API accepts a `page` parameter.
 // In this case, the cursor will include `page`
 interface MyApiCursor {
   page: number;
@@ -200,7 +207,7 @@ export class MyIncrementalEntityProvider
 
     await burst({ apiClient });
 
-    // if you need to do any teardown, you can do it here
+    // If you need to do any teardown, you can do it here.
   }
 }
 ```
@@ -306,31 +313,45 @@ We'll assume you followed the <a href="#installation">Installation</a> instructi
 ```ts
 const incrementalBuilder = await IncrementalCatalogBuilder.create(env, builder);
 
-// I'm assuming you're going to get your token from config
+// Assuming the token for the API comes from config
 const token = config.getString('myApiClient.token');
 
-const myEntityProvider = new MyIncrementalEntityProvider(token)
-
-incrementalBuilder.addIncrementalEntityProvider(
-  myEntityProvider,
-  {
-    // how long should it attempt to read pages from the API
-    // keep this short. Incremental Entity Provider will attempt to
-    // read as many pages as it can in this time
-    burstLength: Duration.fromObject({ seconds: 3 }),
-    // how long should it wait between bursts?
-    burstInterval: Duration.fromObject({ seconds: 3 }),
-    // how long should it rest before re-ingesting again?
-    restLength: Duration.fromObject({ day: 1 })
-    // optional back-off configuration - how long should it wait to retry?
-    backoff: [
-      Duration.fromObject({ seconds: 5 }),
-      Duration.fromObject({ seconds: 30 }),
-      Duration.fromObject({ minutes: 10 }),
-      Duration.fromObject({ hours: 3 })
-    ]
-  }
-)
+const myEntityProvider = new MyIncrementalEntityProvider(token);
+
+incrementalBuilder.addIncrementalEntityProvider(myEntityProvider, {
+  // How long should it attempt to read pages from the API in a
+  // single burst? Keep this short. The Incremental Entity Provider
+  // will attempt to read as many pages as it can in this time
+  burstLength: Duration.fromObject({ seconds: 3 }),
+
+  // How long should it wait between bursts?
+  burstInterval: Duration.fromObject({ seconds: 3 }),
+
+  // How long should it rest before re-ingesting again?
+  restLength: Duration.fromObject({ day: 1 }),
+
+  // Optional back-off configuration - how long should it wait to retry
+  // in the event of an error?
+  backoff: [
+    Duration.fromObject({ seconds: 5 }),
+    Duration.fromObject({ seconds: 30 }),
+    Duration.fromObject({ minutes: 10 }),
+    Duration.fromObject({ hours: 3 }),
+  ],
+
+  // Optional. Use this to prevent removal of entities above a given
+  // percentage. This can be helpful if a data source is flaky and
+  // sometimes returns a successful status, but fewer than expected
+  // assets to add or maintain in the catalog.
+  rejectRemovalsAbovePercentage: 5,
+
+  // Optional. Similar to rejectRemovalsAbovePercentage, except it
+  // applies to complete, 100% failure of a data source. If true,
+  // a data source that returns a successful status but does not
+  // provide any assets to turn into entities will have its empty
+  // data set rejected.
+  rejectEmptySourceCollections: true,
+});
 ```
 
 That's it!!!

package/alpha/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@backstage/plugin-catalog-backend-module-incremental-ingestion",
-  "version": "0.2.0-next.1",
+  "version": "0.2.0",
   "main": "../dist/index.cjs.js",
   "types": "../dist/index.alpha.d.ts"
 }

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@backstage/plugin-catalog-backend-module-incremental-ingestion",
   "description": "An entity provider for streaming large asset sources into the catalog",
-  "version": "0.2.0-next.1",
+  "version": "0.2.0",
   "main": "dist/index.cjs.js",
   "types": "dist/index.d.ts",
   "license": "Apache-2.0",
@@ -33,16 +33,15 @@
     "postpack": "backstage-cli package postpack"
   },
   "dependencies": {
-    "@backstage/backend-common": "^0.18.2-next.1",
-    "@backstage/backend-plugin-api": "^0.3.2-next.1",
-    "@backstage/backend-tasks": "^0.4.3-next.1",
-    "@backstage/backend-test-utils": "^0.1.34-next.1",
-    "@backstage/catalog-model": "^1.1.6-next.0",
+    "@backstage/backend-common": "^0.18.2",
+    "@backstage/backend-plugin-api": "^0.4.0",
+    "@backstage/backend-tasks": "^0.4.3",
+    "@backstage/catalog-model": "^1.2.0",
     "@backstage/config": "^1.0.6",
     "@backstage/errors": "^1.1.4",
-    "@backstage/plugin-catalog-backend": "^1.7.2-next.1",
-    "@backstage/plugin-catalog-node": "^1.3.3-next.1",
-    "@backstage/plugin-events-node": "^0.2.3-next.1",
+    "@backstage/plugin-catalog-backend": "^1.7.2",
+    "@backstage/plugin-catalog-node": "^1.3.3",
+    "@backstage/plugin-events-node": "^0.2.3",
     "@backstage/plugin-permission-common": "^0.7.3",
     "@types/express": "^4.17.6",
     "@types/luxon": "^3.0.0",
@@ -55,9 +54,10 @@
     "winston": "^3.2.1"
   },
   "devDependencies": {
-    "@backstage/backend-app-api": "^0.3.2-next.1",
-    "@backstage/cli": "^0.22.2-next.0",
-    "@backstage/plugin-catalog-backend": "^1.7.2-next.1"
+    "@backstage/backend-app-api": "^0.4.0",
+    "@backstage/backend-test-utils": "^0.1.34",
+    "@backstage/cli": "^0.22.2",
+    "@backstage/plugin-catalog-backend": "^1.7.2"
   },
   "files": [
     "alpha",