@backstage/plugin-catalog-backend 3.6.1 → 3.7.0-next.2

package/CHANGELOG.md

@@ -1,10 +1,97 @@
 # @backstage/plugin-catalog-backend
 
-## 3.6.1
+## 3.7.0-next.2
+
+### Minor Changes
+
+- c2de113: **BREAKING**: When paginating entities with an order field via `/entities/by-query`, entities that lack the order field are now excluded from both the result set and the `totalItems` count. Previously these entities appeared at the end of the sorted result via `NULLS LAST`, but cursor-based pagination could not actually reach them past the first page — the count over-reported the number of navigable entities. The new behavior aligns the count with what is actually returned.
+
+  This also removes the `DISTINCT` deduplication from the sort-field CTE, which is a prerequisite for the planner to use the `(key, value, entity_id)` index in sort order and short-circuit on `LIMIT`. Installations with duplicate search rows should land the search-table deduplication migration before adopting this change.
 
 ### Patch Changes
 
-- 250778e: Fixed a performance regression in the `/entity-facets` endpoint when filters or permission conditions are applied, by routing the EXISTS-based filter through `final_entities` instead of correlating against the much larger `search` table.
+- ccbad9d: Improved the performance of the `catalog_entities_count` metric.
+
+  The legacy Prometheus and OpenTelemetry observable gauges previously each ran their own copy of the per-kind count query against the `search` table on every metrics scrape. On large catalogs this could pile up faster than the queries completed, contending for buffers and stalling the database.
+
+  The two callbacks now share a single query result with a short in-process TTL cache, and the underlying query reads from `final_entities` instead of `search`, avoiding the bitmap heap scans that dominated the previous form. The emitted labels and values are unchanged.
+
+- add5d1a: Restructured the entity listing endpoint so that, when a sort field is specified, the search-by-key index drives the query rather than being side-joined onto `final_entities`. This lets PostgreSQL walk the `(key, value, entity_id)` index in already-sorted order and short-circuit on `LIMIT`, reducing typical broad-filter paginated list times from seconds to milliseconds. Entities that lack the sort field still appear at the end of sorted results (NULLS LAST semantics preserved), ordered by `entity_id`.
+- 387ea7d: Simplified the entity facets aggregation from `COUNT(DISTINCT entity_id)` to `COUNT(*)`. The unique constraint on `(entity_id, key, value)` guarantees each entity appears at most once per search row group, making the `DISTINCT` unnecessary. This allows the database to use a simpler aggregation plan.
+- 3f55b73: Improved the performance of the entity facets endpoint when filters are applied. The filtered entity set is now combined with the search table through an inner join rather than a `WHERE entity_id IN (subquery)`. Results are unchanged; on large catalogs the query planner is able to choose dramatically cheaper plans, with measured improvements ranging from roughly 1.2× on already-fast cases to 7× or more on high-cardinality facets.
+- cde3643: Added missing description to the `type` parameter on the `unregister-entity` MCP action.
+- 7445f0f: Added a migration that removes duplicate rows from the `search` table, creates covering indices for improved query performance, and adds a `UNIQUE` constraint on `(entity_id, key, value)`.
+
+  This is a long-running migration on large catalogs. On PostgreSQL with millions of search rows, the index creation may take 5-15 minutes per index. During this time, other pods running the previous version will continue to serve traffic normally — the index creation does not block reads or writes. However, if a Kubernetes liveness probe kills the pod before the index build completes, the build is lost and the next startup will start over. On large tables this can repeat indefinitely.
+
+  **For large installations**, it is recommended to run the following SQL commands against your PostgreSQL database **before deploying** this version. Each index build takes a few minutes but does not block reads or writes. If these have already completed, the migration will detect the existing indices and skip all work — startup will be instant.
+
+  ```sql
+  -- Step 1: Remove duplicate search rows
+  WITH cte AS (
+    SELECT ctid, row_number() OVER (PARTITION BY entity_id, key, value) AS rn
+    FROM search
+  )
+  DELETE FROM search USING cte WHERE search.ctid = cte.ctid AND cte.rn > 1;
+
+  -- Step 2: Create new indices (run each separately)
+  CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS
+    search_entity_key_value_idx ON search (entity_id, key, value);
+  CREATE INDEX CONCURRENTLY IF NOT EXISTS
+    search_key_value_entity_idx ON search (key, value, entity_id);
+  CREATE INDEX CONCURRENTLY IF NOT EXISTS
+    search_facets_covering_idx ON search (key, original_value, entity_id)
+    WHERE original_value IS NOT NULL;
+
+  -- Step 3: Drop old indices that are no longer needed
+  DROP INDEX CONCURRENTLY IF EXISTS search_key_value_idx;
+  DROP INDEX CONCURRENTLY IF EXISTS search_key_original_value_idx;
+  ```
+
+  Also fixed `buildEntitySearch` to remove duplicate output for entities with duplicate array values, and added `ON CONFLICT DO UPDATE` to `syncSearchRows` so that concurrent stitching races are handled gracefully.
+
+- Updated dependencies
+  - @backstage/backend-plugin-api@1.9.1-next.1
+
+## 3.6.2-next.1
+
+### Patch Changes
+
+- e9b78e9: Removed the `uuid` dependency and replaced usage with the built-in `crypto.randomUUID()`.
+- Updated dependencies
+  - @backstage/catalog-model@1.8.1-next.1
+  - @backstage/plugin-catalog-node@2.2.1-next.1
+  - @backstage/plugin-permission-common@0.9.9-next.1
+
+## 3.6.1-next.0
+
+### Patch Changes
+
+- b33f845: Fixed several database migration `down` functions that were not properly reversible, causing the SQL report to show warnings:
+
+  - `20241003170511_alter_target_in_locations.js`: both `up` and `down` now include `.notNullable()` when altering the `locations.target` column, preventing the `NOT NULL` constraint from being accidentally dropped when widening the column type from `varchar(255)` to `text`.
+  - `20220116144621_remove_legacy.js`: the `down` function now properly recreates the three dropped legacy tables (`entities`, `entities_search`, `entities_relations`) with correct columns and indices.
+  - `20210302150147_refresh_state.js`: the `down` function now drops dependent tables in the correct order (avoiding a FK constraint violation) and fixes a typo where the table was referred to as `references` instead of `refresh_state_references`.
+  - `20201005122705_add_entity_full_name.js`: the `down` function now drops the `full_name` column from `entities` (not `entities_search`), and restores the `entities_unique_name` index with the correct column order `(kind, name, namespace)`.
+  - `20200702153613_entities.js`: the `down` function now uses `table.integer('generation')` instead of `table.string('generation')`, restoring the correct column type.
+
+- cf195de: Fixed a performance regression in the `/entity-facets` endpoint when filters or permission conditions are applied, by routing the EXISTS-based filter through `final_entities` instead of correlating against the much larger `search` table.
+- 744fa1f: Removed duplicated entries that appeared in both `dependencies` and `devDependencies`.
+- Updated dependencies
+  - @backstage/errors@1.3.1-next.0
+  - @backstage/integration@2.0.2-next.0
+  - @backstage/backend-openapi-utils@0.6.9-next.0
+  - @backstage/backend-plugin-api@1.9.1-next.0
+  - @backstage/catalog-client@1.15.1-next.0
+  - @backstage/catalog-model@1.8.1-next.0
+  - @backstage/config@1.3.8-next.0
+  - @backstage/filter-predicates@0.1.3-next.0
+  - @backstage/plugin-catalog-node@2.2.1-next.0
+  - @backstage/plugin-events-node@0.4.22-next.0
+  - @backstage/plugin-permission-common@0.9.9-next.0
+  - @backstage/plugin-permission-node@0.10.13-next.0
+  - @backstage/types@1.2.2
+  - @backstage/plugin-catalog-common@1.1.10-next.0
 
 ## 3.6.0
 

package/dist/actions/createUnregisterCatalogEntitiesAction.cjs.js

@@ -31,10 +31,10 @@ Once completed, all entities associated with the Location will be deleted from t
               `URL of the catalog-info.yaml file to unregister for example: https://github.com/backstage/demo/blob/master/catalog-info.yaml`
             )
           })
-        ])
-      }).describe(
-        "The type to the unregister-entity action. Either locationId or locationUrl must be provided."
-      ),
+        ]).describe(
+          "Identifies the entity to unregister. Provide either locationId or locationUrl."
+        )
+      }),
       output: (z) => z.object({})
     },
     action: async ({ input: { type }, credentials }) => {

package/dist/actions/createUnregisterCatalogEntitiesAction.cjs.js.map

@@ -1 +1 @@
-{"version":3,"file":"createUnregisterCatalogEntitiesAction.cjs.js","sources":["../../src/actions/createUnregisterCatalogEntitiesAction.ts"],"sourcesContent":["/*\n * Copyright 2025 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\nimport { ActionsRegistryService } from '@backstage/backend-plugin-api/alpha';\nimport { NotFoundError } from '@backstage/errors';\nimport { CatalogService } from '@backstage/plugin-catalog-node';\n\nexport const createUnregisterCatalogEntitiesAction = ({\n  catalog,\n  actionsRegistry,\n}: {\n  catalog: CatalogService;\n  actionsRegistry: ActionsRegistryService;\n}) => {\n  actionsRegistry.register({\n    name: 'unregister-entity',\n    title: 'Unregister entity from the Catalog',\n    attributes: {\n      destructive: true,\n      readOnly: false,\n      idempotent: true,\n    },\n    description: `Unregisters a Location entity and all entities it owns from the Backstage catalog.\n\nThis action is similar to the \"Unregister location\" function in the Backstage UI, where you provide the unique identifier (locationId) of a Location entity. Alternatively, you can provide the URL used to register the location. The action will remove the specified Location from the catalog as well as all entities that were created when the Location was imported.\n\nOnce completed, all entities associated with the Location will be deleted from the catalog.\n`,\n    schema: {\n      input: z =>\n        z\n          .object({\n            type: z.union([\n              z.object({\n                locationId: z\n                  .string()\n                  .describe(`Location ID of the Entity to unregister`),\n              }),\n              z.object({\n                locationUrl: z\n                  .string()\n                  .describe(\n                    `URL of the catalog-info.yaml file to unregister for example: https://github.com/backstage/demo/blob/master/catalog-info.yaml`,\n                  ),\n              }),\n            ]),\n          })\n          .describe(\n            'The type to the unregister-entity action. Either locationId or locationUrl must be provided.',\n          ),\n      output: z => z.object({}),\n    },\n    action: async ({ input: { type }, credentials }) => {\n      if ('locationId' in type) {\n        await catalog.removeLocationById(type.locationId, {\n          credentials,\n        });\n      } else {\n        const locations = await catalog\n          .getLocations(\n            {},\n            {\n              credentials,\n            },\n          )\n          .then(response =>\n            response.items.filter(\n              location =>\n                location.target.toLowerCase() ===\n                type.locationUrl.toLowerCase(),\n            ),\n          );\n\n        if (locations.length === 0) {\n          throw new NotFoundError(\n            `Location with URL ${type.locationUrl} not found`,\n          );\n        }\n\n        for (const location of locations) {\n          await catalog.removeLocationById(location.id, {\n            credentials,\n          });\n        }\n      }\n\n      return { output: {} };\n    },\n  });\n};\n"],"names":["NotFoundError"],"mappings":";;;;AAmBO,MAAM,wCAAwC,CAAC;AAAA,EACpD,OAAA;AAAA,EACA;AACF,CAAA,KAGM;AACJ,EAAA,eAAA,CAAgB,QAAA,CAAS;AAAA,IACvB,IAAA,EAAM,mBAAA;AAAA,IACN,KAAA,EAAO,oCAAA;AAAA,IACP,UAAA,EAAY;AAAA,MACV,WAAA,EAAa,IAAA;AAAA,MACb,QAAA,EAAU,KAAA;AAAA,MACV,UAAA,EAAY;AAAA,KACd;AAAA,IACA,WAAA,EAAa,CAAA;;AAAA;;AAAA;AAAA,CAAA;AAAA,IAMb,MAAA,EAAQ;AAAA,MACN,KAAA,EAAO,CAAA,CAAA,KACL,CAAA,CACG,MAAA,CAAO;AAAA,QACN,IAAA,EAAM,EAAE,KAAA,CAAM;AAAA,UACZ,EAAE,MAAA,CAAO;AAAA,YACP,UAAA,EAAY,CAAA,CACT,MAAA,EAAO,CACP,SAAS,CAAA,uCAAA,CAAyC;AAAA,WACtD,CAAA;AAAA,UACD,EAAE,MAAA,CAAO;AAAA,YACP,WAAA,EAAa,CAAA,CACV,MAAA,EAAO,CACP,QAAA;AAAA,cACC,CAAA,4HAAA;AAAA;AACF,WACH;AAAA,SACF;AAAA,OACF,CAAA,CACA,QAAA;AAAA,QACC;AAAA,OACF;AAAA,MACJ,MAAA,EAAQ,CAAA,CAAA,KAAK,CAAA,CAAE,MAAA,CAAO,EAAE;AAAA,KAC1B;AAAA,IACA,MAAA,EAAQ,OAAO,EAAE,KAAA,EAAO,EAAE,IAAA,EAAK,EAAG,aAAY,KAAM;AAClD,MAAA,IAAI,gBAAgB,IAAA,EAAM;AACxB,QAAA,MAAM,OAAA,CAAQ,kBAAA,CAAmB,IAAA,CAAK,UAAA,EAAY;AAAA,UAChD;AAAA,SACD,CAAA;AAAA,MACH,CAAA,MAAO;AACL,QAAA,MAAM,SAAA,GAAY,MAAM,OAAA,CACrB,YAAA;AAAA,UACC,EAAC;AAAA,UACD;AAAA,YACE;AAAA;AACF,SACF,CACC,IAAA;AAAA,UAAK,CAAA,QAAA,KACJ,SAAS,KAAA,CAAM,MAAA;AAAA,YACb,cACE,QAAA,CAAS,MAAA,CAAO,aAAY,KAC5B,IAAA,CAAK,YAAY,WAAA;AAAY;AACjC,SACF;AAEF,QAAA,IAAI,SAAA,CAAU,WAAW,CAAA,EAAG;AAC1B,UAAA,MAAM,IAAIA,oBAAA;AAAA,YACR,CAAA,kBAAA,EAAqB,KAAK,WAAW,CAAA,UAAA;AAAA,WACvC;AAAA,QACF;AAEA,QAAA,KAAA,MAAW,YAAY,SAAA,EAAW;AAChC,UAAA,MAAM,OAAA,CAAQ,kBAAA,CAAmB,QAAA,CAAS,EAAA,EAAI;AAAA,YAC5C;AAAA,WACD,CAAA;AAAA,QACH;AAAA,MACF;AAEA,MAAA,OAAO,EAAE,MAAA,EAAQ,EAAC,EAAE;AAAA,IACtB;AAAA,GACD,CAAA;AACH;;;;"}
+{"version":3,"file":"createUnregisterCatalogEntitiesAction.cjs.js","sources":["../../src/actions/createUnregisterCatalogEntitiesAction.ts"],"sourcesContent":["/*\n * Copyright 2025 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\nimport { ActionsRegistryService } from '@backstage/backend-plugin-api/alpha';\nimport { NotFoundError } from '@backstage/errors';\nimport { CatalogService } from '@backstage/plugin-catalog-node';\n\nexport const createUnregisterCatalogEntitiesAction = ({\n  catalog,\n  actionsRegistry,\n}: {\n  catalog: CatalogService;\n  actionsRegistry: ActionsRegistryService;\n}) => {\n  actionsRegistry.register({\n    name: 'unregister-entity',\n    title: 'Unregister entity from the Catalog',\n    attributes: {\n      destructive: true,\n      readOnly: false,\n      idempotent: true,\n    },\n    description: `Unregisters a Location entity and all entities it owns from the Backstage catalog.\n\nThis action is similar to the \"Unregister location\" function in the Backstage UI, where you provide the unique identifier (locationId) of a Location entity. Alternatively, you can provide the URL used to register the location. The action will remove the specified Location from the catalog as well as all entities that were created when the Location was imported.\n\nOnce completed, all entities associated with the Location will be deleted from the catalog.\n`,\n    schema: {\n      input: z =>\n        z.object({\n          type: z\n            .union([\n              z.object({\n                locationId: z\n                  .string()\n                  .describe(`Location ID of the Entity to unregister`),\n              }),\n              z.object({\n                locationUrl: z\n                  .string()\n                  .describe(\n                    `URL of the catalog-info.yaml file to unregister for example: https://github.com/backstage/demo/blob/master/catalog-info.yaml`,\n                  ),\n              }),\n            ])\n            .describe(\n              'Identifies the entity to unregister. Provide either locationId or locationUrl.',\n            ),\n        }),\n      output: z => z.object({}),\n    },\n    action: async ({ input: { type }, credentials }) => {\n      if ('locationId' in type) {\n        await catalog.removeLocationById(type.locationId, {\n          credentials,\n        });\n      } else {\n        const locations = await catalog\n          .getLocations(\n            {},\n            {\n              credentials,\n            },\n          )\n          .then(response =>\n            response.items.filter(\n              location =>\n                location.target.toLowerCase() ===\n                type.locationUrl.toLowerCase(),\n            ),\n          );\n\n        if (locations.length === 0) {\n          throw new NotFoundError(\n            `Location with URL ${type.locationUrl} not found`,\n          );\n        }\n\n        for (const location of locations) {\n          await catalog.removeLocationById(location.id, {\n            credentials,\n          });\n        }\n      }\n\n      return { output: {} };\n    },\n  });\n};\n"],"names":["NotFoundError"],"mappings":";;;;AAmBO,MAAM,wCAAwC,CAAC;AAAA,EACpD,OAAA;AAAA,EACA;AACF,CAAA,KAGM;AACJ,EAAA,eAAA,CAAgB,QAAA,CAAS;AAAA,IACvB,IAAA,EAAM,mBAAA;AAAA,IACN,KAAA,EAAO,oCAAA;AAAA,IACP,UAAA,EAAY;AAAA,MACV,WAAA,EAAa,IAAA;AAAA,MACb,QAAA,EAAU,KAAA;AAAA,MACV,UAAA,EAAY;AAAA,KACd;AAAA,IACA,WAAA,EAAa,CAAA;;AAAA;;AAAA;AAAA,CAAA;AAAA,IAMb,MAAA,EAAQ;AAAA,MACN,KAAA,EAAO,CAAA,CAAA,KACL,CAAA,CAAE,MAAA,CAAO;AAAA,QACP,IAAA,EAAM,EACH,KAAA,CAAM;AAAA,UACL,EAAE,MAAA,CAAO;AAAA,YACP,UAAA,EAAY,CAAA,CACT,MAAA,EAAO,CACP,SAAS,CAAA,uCAAA,CAAyC;AAAA,WACtD,CAAA;AAAA,UACD,EAAE,MAAA,CAAO;AAAA,YACP,WAAA,EAAa,CAAA,CACV,MAAA,EAAO,CACP,QAAA;AAAA,cACC,CAAA,4HAAA;AAAA;AACF,WACH;AAAA,SACF,CAAA,CACA,QAAA;AAAA,UACC;AAAA;AACF,OACH,CAAA;AAAA,MACH,MAAA,EAAQ,CAAA,CAAA,KAAK,CAAA,CAAE,MAAA,CAAO,EAAE;AAAA,KAC1B;AAAA,IACA,MAAA,EAAQ,OAAO,EAAE,KAAA,EAAO,EAAE,IAAA,EAAK,EAAG,aAAY,KAAM;AAClD,MAAA,IAAI,gBAAgB,IAAA,EAAM;AACxB,QAAA,MAAM,OAAA,CAAQ,kBAAA,CAAmB,IAAA,CAAK,UAAA,EAAY;AAAA,UAChD;AAAA,SACD,CAAA;AAAA,MACH,CAAA,MAAO;AACL,QAAA,MAAM,SAAA,GAAY,MAAM,OAAA,CACrB,YAAA;AAAA,UACC,EAAC;AAAA,UACD;AAAA,YACE;AAAA;AACF,SACF,CACC,IAAA;AAAA,UAAK,CAAA,QAAA,KACJ,SAAS,KAAA,CAAM,MAAA;AAAA,YACb,cACE,QAAA,CAAS,MAAA,CAAO,aAAY,KAC5B,IAAA,CAAK,YAAY,WAAA;AAAY;AACjC,SACF;AAEF,QAAA,IAAI,SAAA,CAAU,WAAW,CAAA,EAAG;AAC1B,UAAA,MAAM,IAAIA,oBAAA;AAAA,YACR,CAAA,kBAAA,EAAqB,KAAK,WAAW,CAAA,UAAA;AAAA,WACvC;AAAA,QACF;AAEA,QAAA,KAAA,MAAW,YAAY,SAAA,EAAW;AAChC,UAAA,MAAM,OAAA,CAAQ,kBAAA,CAAmB,QAAA,CAAS,EAAA,EAAI;AAAA,YAC5C;AAAA,WACD,CAAA;AAAA,QACH;AAAA,MACF;AAEA,MAAA,OAAO,EAAE,MAAA,EAAQ,EAAC,EAAE;AAAA,IACtB;AAAA,GACD,CAAA;AACH;;;;"}

package/dist/database/DefaultProviderDatabase.cjs.js

@@ -2,7 +2,7 @@
 
 var catalogModel = require('@backstage/catalog-model');
 var lodash = require('lodash');
-var uuid = require('uuid');
+var node_crypto = require('node:crypto');
 var conversion = require('./conversion.cjs.js');
 var deleteWithEagerPruningOfChildren = require('./operations/provider/deleteWithEagerPruningOfChildren.cjs.js');
 var refreshByRefreshKeys = require('./operations/provider/refreshByRefreshKeys.cjs.js');
@@ -59,7 +59,7 @@ class DefaultProviderDatabase {
           await tx.batchInsert(
             "refresh_state",
             chunk.map((item) => ({
-              entity_id: uuid.v4(),
+              entity_id: node_crypto.randomUUID(),
               entity_ref: catalogModel.stringifyEntityRef(item.deferred.entity),
               unprocessed_entity: JSON.stringify(item.deferred.entity),
               unprocessed_hash: item.hash,

package/dist/database/DefaultProviderDatabase.cjs.js.map

@@ -1 +1 @@
-{"version":3,"file":"DefaultProviderDatabase.cjs.js","sources":["../../src/database/DefaultProviderDatabase.ts"],"sourcesContent":["/*\n * Copyright 2021 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { stringifyEntityRef } from '@backstage/catalog-model';\nimport { DeferredEntity } from '@backstage/plugin-catalog-node';\nimport { Knex } from 'knex';\nimport lodash from 'lodash';\nimport { v4 as uuid } from 'uuid';\nimport { rethrowError } from './conversion';\nimport { deleteWithEagerPruningOfChildren } from './operations/provider/deleteWithEagerPruningOfChildren';\nimport { refreshByRefreshKeys } from './operations/provider/refreshByRefreshKeys';\nimport { checkLocationKeyConflict } from './operations/refreshState/checkLocationKeyConflict';\nimport { insertUnprocessedEntity } from './operations/refreshState/insertUnprocessedEntity';\nimport { updateUnprocessedEntity } from './operations/refreshState/updateUnprocessedEntity';\nimport { DbRefreshStateReferencesRow, DbRefreshStateRow } from './tables';\nimport {\n  ProviderDatabase,\n  RefreshByKeyOptions,\n  ReplaceUnprocessedEntitiesOptions,\n  Transaction,\n} from './types';\nimport { generateStableHash } from './util';\nimport {\n  LoggerService,\n  isDatabaseConflictError,\n} from '@backstage/backend-plugin-api';\n\n// The number of items that are sent per batch to the database layer, when\n// doing .batchInsert calls to knex. This needs to be low enough to not cause\n// errors in the underlying engine due to exceeding query limits, but large\n// enough to get the speed benefits.\nconst BATCH_SIZE = 50;\n\nexport class DefaultProviderDatabase implements ProviderDatabase {\n  private readonly options: {\n    database: Knex;\n    logger: LoggerService;\n  };\n\n  constructor(options: { database: Knex; logger: LoggerService }) {\n    this.options = options;\n  }\n\n  async transaction<T>(fn: (tx: Transaction) => Promise<T>): Promise<T> {\n    try {\n      let result: T | undefined = undefined;\n      await this.options.database.transaction(\n        async tx => {\n          // We can't return here, as knex swallows the return type in case the\n          // transaction is rolled back:\n          // https://github.com/knex/knex/blob/e37aeaa31c8ef9c1b07d2e4d3ec6607e557d800d/lib/transaction.js#L136\n          result = await fn(tx);\n        },\n        {\n          // If we explicitly trigger a rollback, don't fail.\n          doNotRejectOnRollback: true,\n        },\n      );\n      return result!;\n    } catch (e) {\n      this.options.logger.debug(`Error during transaction, ${e}`);\n      throw rethrowError(e);\n    }\n  }\n\n  async replaceUnprocessedEntities(\n    txOpaque: Knex | Transaction,\n    options: ReplaceUnprocessedEntitiesOptions,\n  ): Promise<void> {\n    const tx = txOpaque as Knex | Knex.Transaction;\n    const { toAdd, toUpsert, toRemove } = await this.createDelta(tx, options);\n\n    if (toRemove.length) {\n      const removedCount = await deleteWithEagerPruningOfChildren({\n        knex: tx,\n        entityRefs: toRemove,\n        sourceKey: options.sourceKey,\n      });\n      this.options.logger.debug(\n        `removed, ${removedCount} entities: ${JSON.stringify(toRemove)}`,\n      );\n    }\n\n    if (toAdd.length) {\n      // The reason for this chunking, rather than just massively batch\n      // inserting the entire payload, is that we fall back to the individual\n      // upsert mechanism below on conflicts. That path is massively slower than\n      // the fast batch path, so we don't want to end up accidentally having to\n      // for example item-by-item upsert tens of thousands of entities in a\n      // large initial delivery dump. The implication is that the size of these\n      // chunks needs to weigh the benefit of fast successful inserts, against\n      // the drawback of super slow but more rare fallbacks. There's quickly\n      // diminishing returns though with turning up this value way high.\n      for (const chunk of lodash.chunk(toAdd, 50)) {\n        try {\n          await tx.batchInsert(\n            'refresh_state',\n            chunk.map(item => ({\n              entity_id: uuid(),\n              entity_ref: stringifyEntityRef(item.deferred.entity),\n              unprocessed_entity: JSON.stringify(item.deferred.entity),\n              unprocessed_hash: item.hash,\n              errors: '',\n              location_key: item.deferred.locationKey,\n              next_update_at: tx.fn.now(),\n              last_discovery_at: tx.fn.now(),\n            })),\n            BATCH_SIZE,\n          );\n          await tx.batchInsert(\n            'refresh_state_references',\n            chunk.map(item => ({\n              source_key: options.sourceKey,\n              target_entity_ref: stringifyEntityRef(item.deferred.entity),\n            })),\n            BATCH_SIZE,\n          );\n        } catch (error) {\n          if (!isDatabaseConflictError(error)) {\n            throw error;\n          } else {\n            this.options.logger.debug(\n              `Fast insert path failed, falling back to slow path, ${error}`,\n            );\n            toUpsert.push(...chunk);\n          }\n        }\n      }\n    }\n\n    if (toUpsert.length) {\n      for (const {\n        deferred: { entity, locationKey },\n        hash,\n      } of toUpsert) {\n        const entityRef = stringifyEntityRef(entity);\n\n        try {\n          let ok = await updateUnprocessedEntity({\n            tx,\n            entity,\n            hash,\n            locationKey,\n          });\n          if (!ok) {\n            ok = await insertUnprocessedEntity({\n              tx,\n              entity,\n              hash,\n              locationKey,\n              logger: this.options.logger,\n            });\n          }\n          if (ok) {\n            await tx<DbRefreshStateReferencesRow>('refresh_state_references')\n              .where('target_entity_ref', entityRef)\n              .delete();\n\n            await tx<DbRefreshStateReferencesRow>(\n              'refresh_state_references',\n            ).insert({\n              source_key: options.sourceKey,\n              target_entity_ref: entityRef,\n            });\n          } else {\n            await tx<DbRefreshStateReferencesRow>('refresh_state_references')\n              .where('target_entity_ref', entityRef)\n              .andWhere({ source_key: options.sourceKey })\n              .delete();\n\n            const conflictingKey = await checkLocationKeyConflict({\n              tx,\n              entityRef,\n              locationKey,\n            });\n            if (conflictingKey) {\n              this.options.logger.warn(\n                `Source ${options.sourceKey} detected conflicting entityRef ${entityRef} already referenced by ${conflictingKey} and now also ${locationKey}`,\n              );\n            }\n          }\n        } catch (error) {\n          this.options.logger.error(\n            `Failed to add '${entityRef}' from source '${options.sourceKey}', ${error}`,\n          );\n        }\n      }\n    }\n  }\n\n  async listReferenceSourceKeys(txOpaque: Transaction): Promise<string[]> {\n    const tx = txOpaque as Knex | Knex.Transaction;\n\n    const rows = await tx<DbRefreshStateReferencesRow>(\n      'refresh_state_references',\n    )\n      .distinct('source_key')\n      .whereNotNull('source_key');\n\n    return rows\n      .map(row => row.source_key)\n      .filter((key): key is string => !!key);\n  }\n\n  async refreshByRefreshKeys(\n    txOpaque: Transaction,\n    options: RefreshByKeyOptions,\n  ) {\n    const tx = txOpaque as Knex.Transaction;\n    await refreshByRefreshKeys({ tx, keys: options.keys });\n  }\n\n  private async createDelta(\n    tx: Knex | Knex.Transaction,\n    options: ReplaceUnprocessedEntitiesOptions,\n  ): Promise<{\n    toAdd: { deferred: DeferredEntity; hash: string }[];\n    toUpsert: { deferred: DeferredEntity; hash: string }[];\n    toRemove: string[];\n  }> {\n    if (options.type === 'delta') {\n      const toAdd = new Array<{ deferred: DeferredEntity; hash: string }>();\n      const toUpsert = new Array<{ deferred: DeferredEntity; hash: string }>();\n      const toRemove = options.removed.map(e => e.entityRef);\n\n      for (const chunk of lodash.chunk(options.added, 1000)) {\n        const entityRefs = chunk.map(e => stringifyEntityRef(e.entity));\n        const rows = await tx<DbRefreshStateRow>('refresh_state')\n          .select(['entity_ref', 'unprocessed_hash', 'location_key'])\n          .whereIn('entity_ref', entityRefs);\n        const oldStates = new Map(\n          rows.map(row => [\n            row.entity_ref,\n            {\n              unprocessed_hash: row.unprocessed_hash,\n              location_key: row.location_key,\n            },\n          ]),\n        );\n\n        chunk.forEach((deferred, i) => {\n          const entityRef = entityRefs[i];\n          const newHash = generateStableHash(deferred.entity);\n          const oldState = oldStates.get(entityRef);\n          if (oldState === undefined) {\n            // Add any entity that does not exist in the database\n            toAdd.push({ deferred, hash: newHash });\n          } else if (\n            (deferred.locationKey ?? null) !== (oldState.location_key ?? null)\n          ) {\n            // Remove and then re-add any entity that exists, but with a different location key\n            toRemove.push(entityRef);\n            toAdd.push({ deferred, hash: newHash });\n          } else if (newHash !== oldState.unprocessed_hash) {\n            // Entities with modifications should be pushed through too\n            toUpsert.push({ deferred, hash: newHash });\n          }\n        });\n      }\n\n      return { toAdd, toUpsert, toRemove };\n    }\n\n    // Grab all of the existing references from the same source, and their locationKeys as well\n    const oldRefs = await tx<DbRefreshStateReferencesRow>(\n      'refresh_state_references',\n    )\n      .leftJoin<DbRefreshStateRow>('refresh_state', {\n        target_entity_ref: 'entity_ref',\n      })\n      .where({ source_key: options.sourceKey })\n      .select({\n        target_entity_ref: 'refresh_state_references.target_entity_ref',\n        location_key: 'refresh_state.location_key',\n        unprocessed_hash: 'refresh_state.unprocessed_hash',\n      });\n\n    const items = options.items.map(deferred => ({\n      deferred,\n      ref: stringifyEntityRef(deferred.entity),\n      hash: generateStableHash(deferred.entity),\n    }));\n\n    const oldRefsSet = new Map(\n      oldRefs.map(r => [\n        r.target_entity_ref,\n        {\n          locationKey: r.location_key,\n          oldEntityHash: r.unprocessed_hash,\n        },\n      ]),\n    );\n    const newRefsSet = new Set(items.map(item => item.ref));\n\n    const toAdd = new Array<{ deferred: DeferredEntity; hash: string }>();\n    const toUpsert = new Array<{ deferred: DeferredEntity; hash: string }>();\n    const toRemove = oldRefs\n      .map(row => row.target_entity_ref)\n      .filter(ref => !newRefsSet.has(ref));\n\n    for (const item of items) {\n      const oldRef = oldRefsSet.get(item.ref);\n      const upsertItem = { deferred: item.deferred, hash: item.hash };\n      if (!oldRef) {\n        // Add any entity that does not exist in the database\n        toAdd.push(upsertItem);\n      } else if (\n        (oldRef.locationKey ?? undefined) !==\n        (item.deferred.locationKey ?? undefined)\n      ) {\n        // Remove and then re-add any entity that exists, but with a different location key\n        toRemove.push(item.ref);\n        toAdd.push(upsertItem);\n      } else if (oldRef.oldEntityHash !== item.hash) {\n        // Entities with modifications should be pushed through too\n        toUpsert.push(upsertItem);\n      }\n    }\n\n    return { toAdd, toUpsert, toRemove };\n  }\n}\n"],"names":["rethrowError","deleteWithEagerPruningOfChildren","lodash","uuid","stringifyEntityRef","isDatabaseConflictError","updateUnprocessedEntity","insertUnprocessedEntity","checkLocationKeyConflict","refreshByRefreshKeys","toAdd","toUpsert","toRemove","generateStableHash"],"mappings":";;;;;;;;;;;;;;;;;;AA4CA,MAAM,UAAA,GAAa,EAAA;AAEZ,MAAM,uBAAA,CAAoD;AAAA,EAC9C,OAAA;AAAA,EAKjB,YAAY,OAAA,EAAoD;AAC9D,IAAA,IAAA,CAAK,OAAA,GAAU,OAAA;AAAA,EACjB;AAAA,EAEA,MAAM,YAAe,EAAA,EAAiD;AACpE,IAAA,IAAI;AACF,MAAA,IAAI,MAAA,GAAwB,KAAA,CAAA;AAC5B,MAAA,MAAM,IAAA,CAAK,QAAQ,QAAA,CAAS,WAAA;AAAA,QAC1B,OAAM,EAAA,KAAM;AAIV,UAAA,MAAA,GAAS,MAAM,GAAG,EAAE,CAAA;AAAA,QACtB,CAAA;AAAA,QACA;AAAA;AAAA,UAEE,qBAAA,EAAuB;AAAA;AACzB,OACF;AACA,MAAA,OAAO,MAAA;AAAA,IACT,SAAS,CAAA,EAAG;AACV,MAAA,IAAA,CAAK,OAAA,CAAQ,MAAA,CAAO,KAAA,CAAM,CAAA,0BAAA,EAA6B,CAAC,CAAA,CAAE,CAAA;AAC1D,MAAA,MAAMA,wBAAa,CAAC,CAAA;AAAA,IACtB;AAAA,EACF;AAAA,EAEA,MAAM,0BAAA,CACJ,QAAA,EACA,OAAA,EACe;AACf,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAM,EAAE,OAAO,QAAA,EAAU,QAAA,KAAa,MAAM,IAAA,CAAK,WAAA,CAAY,EAAA,EAAI,OAAO,CAAA;AAExE,IAAA,IAAI,SAAS,MAAA,EAAQ;AACnB,MAAA,MAAM,YAAA,GAAe,MAAMC,iEAAA,CAAiC;AAAA,QAC1D,IAAA,EAAM,EAAA;AAAA,QACN,UAAA,EAAY,QAAA;AAAA,QACZ,WAAW,OAAA,CAAQ;AAAA,OACpB,CAAA;AACD,MAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,KAAA;AAAA,QAClB,YAAY,YAAY,CAAA,WAAA,EAAc,IAAA,CAAK,SAAA,CAAU,QAAQ,CAAC,CAAA;AAAA,OAChE;AAAA,IACF;AAEA,IAAA,IAAI,MAAM,MAAA,EAAQ;AAUhB,MAAA,KAAA,MAAW,KAAA,IAASC,uBAAA,CAAO,KAAA,CAAM,KAAA,EAAO,EAAE,CAAA,EAAG;AAC3C,QAAA,IAAI;AACF,UAAA,MAAM,EAAA,CAAG,WAAA;AAAA,YACP,eAAA;AAAA,YACA,KAAA,CAAM,IAAI,CAAA,IAAA,MAAS;AAAA,cACjB,WAAWC,OAAA,EAAK;AAAA,cAChB,UAAA,EAAYC,+BAAA,CAAmB,IAAA,CAAK,QAAA,CAAS,MAAM,CAAA;AAAA,cACnD,kBAAA,EAAoB,IAAA,CAAK,SAAA,CAAU,IAAA,CAAK,SAAS,MAAM,CAAA;AAAA,cACvD,kBAAkB,IAAA,CAAK,IAAA;AAAA,cACvB,MAAA,EAAQ,EAAA;AAAA,cACR,YAAA,EAAc,KAAK,QAAA,CAAS,WAAA;AAAA,cAC5B,cAAA,EAAgB,EAAA,CAAG,EAAA,CAAG,GAAA,EAAI;AAAA,cAC1B,iBAAA,EAAmB,EAAA,CAAG,EAAA,CAAG,GAAA;AAAI,aAC/B,CAAE,CAAA;AAAA,YACF;AAAA,WACF;AACA,UAAA,MAAM,EAAA,CAAG,WAAA;AAAA,YACP,0BAAA;AAAA,YACA,KAAA,CAAM,IAAI,CAAA,IAAA,MAAS;AAAA,cACjB,YAAY,OAAA,CAAQ,SAAA;AAAA,cACpB,iBAAA,EAAmBA,+BAAA,CAAmB,IAAA,CAAK,QAAA,CAAS,MAAM;AAAA,aAC5D,CAAE,CAAA;AAAA,YACF;AAAA,WACF;AAAA,QACF,SAAS,KAAA,EAAO;AACd,UAAA,IAAI,CAACC,wCAAA,CAAwB,KAAK,CAAA,EAAG;AACnC,YAAA,MAAM,KAAA;AAAA,UACR,CAAA,MAAO;AACL,YAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,KAAA;AAAA,cAClB,uDAAuD,KAAK,CAAA;AAAA,aAC9D;AACA,YAAA,QAAA,CAAS,IAAA,CAAK,GAAG,KAAK,CAAA;AAAA,UACxB;AAAA,QACF;AAAA,MACF;AAAA,IACF;AAEA,IAAA,IAAI,SAAS,MAAA,EAAQ;AACnB,MAAA,KAAA,MAAW;AAAA,QACT,QAAA,EAAU,EAAE,MAAA,EAAQ,WAAA,EAAY;AAAA,QAChC;AAAA,WACG,QAAA,EAAU;AACb,QAAA,MAAM,SAAA,GAAYD,gCAAmB,MAAM,CAAA;AAE3C,QAAA,IAAI;AACF,UAAA,IAAI,EAAA,GAAK,MAAME,+CAAA,CAAwB;AAAA,YACrC,EAAA;AAAA,YACA,MAAA;AAAA,YACA,IAAA;AAAA,YACA;AAAA,WACD,CAAA;AACD,UAAA,IAAI,CAAC,EAAA,EAAI;AACP,YAAA,EAAA,GAAK,MAAMC,+CAAA,CAAwB;AAAA,cACjC,EAAA;AAAA,cACA,MAAA;AAAA,cACA,IAAA;AAAA,cACA,WAAA;AAAA,cACA,MAAA,EAAQ,KAAK,OAAA,CAAQ;AAAA,aACtB,CAAA;AAAA,UACH;AACA,UAAA,IAAI,EAAA,EAAI;AACN,YAAA,MAAM,GAAgC,0BAA0B,CAAA,CAC7D,MAAM,mBAAA,EAAqB,SAAS,EACpC,MAAA,EAAO;AAEV,YAAA,MAAM,EAAA;AAAA,cACJ;AAAA,cACA,MAAA,CAAO;AAAA,cACP,YAAY,OAAA,CAAQ,SAAA;AAAA,cACpB,iBAAA,EAAmB;AAAA,aACpB,CAAA;AAAA,UACH,CAAA,MAAO;AACL,YAAA,MAAM,EAAA,CAAgC,0BAA0B,CAAA,CAC7D,KAAA,CAAM,qBAAqB,SAAS,CAAA,CACpC,QAAA,CAAS,EAAE,UAAA,EAAY,OAAA,CAAQ,SAAA,EAAW,EAC1C,MAAA,EAAO;AAEV,YAAA,MAAM,cAAA,GAAiB,MAAMC,iDAAA,CAAyB;AAAA,cACpD,EAAA;AAAA,cACA,SAAA;AAAA,cACA;AAAA,aACD,CAAA;AACD,YAAA,IAAI,cAAA,EAAgB;AAClB,cAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,IAAA;AAAA,gBAClB,CAAA,OAAA,EAAU,QAAQ,SAAS,CAAA,gCAAA,EAAmC,SAAS,CAAA,uBAAA,EAA0B,cAAc,iBAAiB,WAAW,CAAA;AAAA,eAC7I;AAAA,YACF;AAAA,UACF;AAAA,QACF,SAAS,KAAA,EAAO;AACd,UAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,KAAA;AAAA,YAClB,kBAAkB,SAAS,CAAA,eAAA,EAAkB,OAAA,CAAQ,SAAS,MAAM,KAAK,CAAA;AAAA,WAC3E;AAAA,QACF;AAAA,MACF;AAAA,IACF;AAAA,EACF;AAAA,EAEA,MAAM,wBAAwB,QAAA,EAA0C;AACtE,IAAA,MAAM,EAAA,GAAK,QAAA;AAEX,IAAA,MAAM,OAAO,MAAM,EAAA;AAAA,MACjB;AAAA,KACF,CACG,QAAA,CAAS,YAAY,CAAA,CACrB,aAAa,YAAY,CAAA;AAE5B,IAAA,OAAO,IAAA,CACJ,GAAA,CAAI,CAAA,GAAA,KAAO,GAAA,CAAI,UAAU,CAAA,CACzB,MAAA,CAAO,CAAC,GAAA,KAAuB,CAAC,CAAC,GAAG,CAAA;AAAA,EACzC;AAAA,EAEA,MAAM,oBAAA,CACJ,QAAA,EACA,OAAA,EACA;AACA,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAMC,0CAAqB,EAAE,EAAA,EAAI,IAAA,EAAM,OAAA,CAAQ,MAAM,CAAA;AAAA,EACvD;AAAA,EAEA,MAAc,WAAA,CACZ,EAAA,EACA,OAAA,EAKC;AACD,IAAA,IAAI,OAAA,CAAQ,SAAS,OAAA,EAAS;AAC5B,MAAA,MAAMC,MAAAA,GAAQ,IAAI,KAAA,EAAkD;AACpE,MAAA,MAAMC,SAAAA,GAAW,IAAI,KAAA,EAAkD;AACvE,MAAA,MAAMC,YAAW,OAAA,CAAQ,OAAA,CAAQ,GAAA,CAAI,CAAA,CAAA,KAAK,EAAE,SAAS,CAAA;AAErD,MAAA,KAAA,MAAW,SAASV,uBAAA,CAAO,KAAA,CAAM,OAAA,CAAQ,KAAA,EAAO,GAAI,CAAA,EAAG;AACrD,QAAA,MAAM,aAAa,KAAA,CAAM,GAAA,CAAI,OAAKE,+BAAA,CAAmB,CAAA,CAAE,MAAM,CAAC,CAAA;AAC9D,QAAA,MAAM,IAAA,GAAO,MAAM,EAAA,CAAsB,eAAe,EACrD,MAAA,CAAO,CAAC,YAAA,EAAc,kBAAA,EAAoB,cAAc,CAAC,CAAA,CACzD,OAAA,CAAQ,cAAc,UAAU,CAAA;AACnC,QAAA,MAAM,YAAY,IAAI,GAAA;AAAA,UACpB,IAAA,CAAK,IAAI,CAAA,GAAA,KAAO;AAAA,YACd,GAAA,CAAI,UAAA;AAAA,YACJ;AAAA,cACE,kBAAkB,GAAA,CAAI,gBAAA;AAAA,cACtB,cAAc,GAAA,CAAI;AAAA;AACpB,WACD;AAAA,SACH;AAEA,QAAA,KAAA,CAAM,OAAA,CAAQ,CAAC,QAAA,EAAU,CAAA,KAAM;AAC7B,UAAA,MAAM,SAAA,GAAY,WAAW,CAAC,CAAA;AAC9B,UAAA,MAAM,OAAA,GAAUS,uBAAA,CAAmB,QAAA,CAAS,MAAM,CAAA;AAClD,UAAA,MAAM,QAAA,GAAW,SAAA,CAAU,GAAA,CAAI,SAAS,CAAA;AACxC,UAAA,IAAI,aAAa,MAAA,EAAW;AAE1B,YAAAH,OAAM,IAAA,CAAK,EAAE,QAAA,EAAU,IAAA,EAAM,SAAS,CAAA;AAAA,UACxC,YACG,QAAA,CAAS,WAAA,IAAe,IAAA,OAAW,QAAA,CAAS,gBAAgB,IAAA,CAAA,EAC7D;AAEA,YAAAE,SAAAA,CAAS,KAAK,SAAS,CAAA;AACvB,YAAAF,OAAM,IAAA,CAAK,EAAE,QAAA,EAAU,IAAA,EAAM,SAAS,CAAA;AAAA,UACxC,CAAA,MAAA,IAAW,OAAA,KAAY,QAAA,CAAS,gBAAA,EAAkB;AAEhD,YAAAC,UAAS,IAAA,CAAK,EAAE,QAAA,EAAU,IAAA,EAAM,SAAS,CAAA;AAAA,UAC3C;AAAA,QACF,CAAC,CAAA;AAAA,MACH;AAEA,MAAA,OAAO,EAAE,KAAA,EAAAD,MAAAA,EAAO,QAAA,EAAAC,SAAAA,EAAU,UAAAC,SAAAA,EAAS;AAAA,IACrC;AAGA,IAAA,MAAM,UAAU,MAAM,EAAA;AAAA,MACpB;AAAA,KACF,CACG,SAA4B,eAAA,EAAiB;AAAA,MAC5C,iBAAA,EAAmB;AAAA,KACpB,EACA,KAAA,CAAM,EAAE,YAAY,OAAA,CAAQ,SAAA,EAAW,CAAA,CACvC,MAAA,CAAO;AAAA,MACN,iBAAA,EAAmB,4CAAA;AAAA,MACnB,YAAA,EAAc,4BAAA;AAAA,MACd,gBAAA,EAAkB;AAAA,KACnB,CAAA;AAEH,IAAA,MAAM,KAAA,GAAQ,OAAA,CAAQ,KAAA,CAAM,GAAA,CAAI,CAAA,QAAA,MAAa;AAAA,MAC3C,QAAA;AAAA,MACA,GAAA,EAAKR,+BAAA,CAAmB,QAAA,CAAS,MAAM,CAAA;AAAA,MACvC,IAAA,EAAMS,uBAAA,CAAmB,QAAA,CAAS,MAAM;AAAA,KAC1C,CAAE,CAAA;AAEF,IAAA,MAAM,aAAa,IAAI,GAAA;AAAA,MACrB,OAAA,CAAQ,IAAI,CAAA,CAAA,KAAK;AAAA,QACf,CAAA,CAAE,iBAAA;AAAA,QACF;AAAA,UACE,aAAa,CAAA,CAAE,YAAA;AAAA,UACf,eAAe,CAAA,CAAE;AAAA;AACnB,OACD;AAAA,KACH;AACA,IAAA,MAAM,UAAA,GAAa,IAAI,GAAA,CAAI,KAAA,CAAM,IAAI,CAAA,IAAA,KAAQ,IAAA,CAAK,GAAG,CAAC,CAAA;AAEtD,IAAA,MAAM,KAAA,GAAQ,IAAI,KAAA,EAAkD;AACpE,IAAA,MAAM,QAAA,GAAW,IAAI,KAAA,EAAkD;AACvE,IAAA,MAAM,QAAA,GAAW,OAAA,CACd,GAAA,CAAI,CAAA,GAAA,KAAO,GAAA,CAAI,iBAAiB,CAAA,CAChC,MAAA,CAAO,CAAA,GAAA,KAAO,CAAC,UAAA,CAAW,GAAA,CAAI,GAAG,CAAC,CAAA;AAErC,IAAA,KAAA,MAAW,QAAQ,KAAA,EAAO;AACxB,MAAA,MAAM,MAAA,GAAS,UAAA,CAAW,GAAA,CAAI,IAAA,CAAK,GAAG,CAAA;AACtC,MAAA,MAAM,aAAa,EAAE,QAAA,EAAU,KAAK,QAAA,EAAU,IAAA,EAAM,KAAK,IAAA,EAAK;AAC9D,MAAA,IAAI,CAAC,MAAA,EAAQ;AAEX,QAAA,KAAA,CAAM,KAAK,UAAU,CAAA;AAAA,MACvB,YACG,MAAA,CAAO,WAAA,IAAe,aACtB,IAAA,CAAK,QAAA,CAAS,eAAe,MAAA,CAAA,EAC9B;AAEA,QAAA,QAAA,CAAS,IAAA,CAAK,KAAK,GAAG,CAAA;AACtB,QAAA,KAAA,CAAM,KAAK,UAAU,CAAA;AAAA,MACvB,CAAA,MAAA,IAAW,MAAA,CAAO,aAAA,KAAkB,IAAA,CAAK,IAAA,EAAM;AAE7C,QAAA,QAAA,CAAS,KAAK,UAAU,CAAA;AAAA,MAC1B;AAAA,IACF;AAEA,IAAA,OAAO,EAAE,KAAA,EAAO,QAAA,EAAU,QAAA,EAAS;AAAA,EACrC;AACF;;;;"}
+{"version":3,"file":"DefaultProviderDatabase.cjs.js","sources":["../../src/database/DefaultProviderDatabase.ts"],"sourcesContent":["/*\n * Copyright 2021 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { stringifyEntityRef } from '@backstage/catalog-model';\nimport { DeferredEntity } from '@backstage/plugin-catalog-node';\nimport { Knex } from 'knex';\nimport lodash from 'lodash';\nimport { randomUUID as uuid } from 'node:crypto';\nimport { rethrowError } from './conversion';\nimport { deleteWithEagerPruningOfChildren } from './operations/provider/deleteWithEagerPruningOfChildren';\nimport { refreshByRefreshKeys } from './operations/provider/refreshByRefreshKeys';\nimport { checkLocationKeyConflict } from './operations/refreshState/checkLocationKeyConflict';\nimport { insertUnprocessedEntity } from './operations/refreshState/insertUnprocessedEntity';\nimport { updateUnprocessedEntity } from './operations/refreshState/updateUnprocessedEntity';\nimport { DbRefreshStateReferencesRow, DbRefreshStateRow } from './tables';\nimport {\n  ProviderDatabase,\n  RefreshByKeyOptions,\n  ReplaceUnprocessedEntitiesOptions,\n  Transaction,\n} from './types';\nimport { generateStableHash } from './util';\nimport {\n  LoggerService,\n  isDatabaseConflictError,\n} from '@backstage/backend-plugin-api';\n\n// The number of items that are sent per batch to the database layer, when\n// doing .batchInsert calls to knex. This needs to be low enough to not cause\n// errors in the underlying engine due to exceeding query limits, but large\n// enough to get the speed benefits.\nconst BATCH_SIZE = 50;\n\nexport class DefaultProviderDatabase implements ProviderDatabase {\n  private readonly options: {\n    database: Knex;\n    logger: LoggerService;\n  };\n\n  constructor(options: { database: Knex; logger: LoggerService }) {\n    this.options = options;\n  }\n\n  async transaction<T>(fn: (tx: Transaction) => Promise<T>): Promise<T> {\n    try {\n      let result: T | undefined = undefined;\n      await this.options.database.transaction(\n        async tx => {\n          // We can't return here, as knex swallows the return type in case the\n          // transaction is rolled back:\n          // https://github.com/knex/knex/blob/e37aeaa31c8ef9c1b07d2e4d3ec6607e557d800d/lib/transaction.js#L136\n          result = await fn(tx);\n        },\n        {\n          // If we explicitly trigger a rollback, don't fail.\n          doNotRejectOnRollback: true,\n        },\n      );\n      return result!;\n    } catch (e) {\n      this.options.logger.debug(`Error during transaction, ${e}`);\n      throw rethrowError(e);\n    }\n  }\n\n  async replaceUnprocessedEntities(\n    txOpaque: Knex | Transaction,\n    options: ReplaceUnprocessedEntitiesOptions,\n  ): Promise<void> {\n    const tx = txOpaque as Knex | Knex.Transaction;\n    const { toAdd, toUpsert, toRemove } = await this.createDelta(tx, options);\n\n    if (toRemove.length) {\n      const removedCount = await deleteWithEagerPruningOfChildren({\n        knex: tx,\n        entityRefs: toRemove,\n        sourceKey: options.sourceKey,\n      });\n      this.options.logger.debug(\n        `removed, ${removedCount} entities: ${JSON.stringify(toRemove)}`,\n      );\n    }\n\n    if (toAdd.length) {\n      // The reason for this chunking, rather than just massively batch\n      // inserting the entire payload, is that we fall back to the individual\n      // upsert mechanism below on conflicts. That path is massively slower than\n      // the fast batch path, so we don't want to end up accidentally having to\n      // for example item-by-item upsert tens of thousands of entities in a\n      // large initial delivery dump. The implication is that the size of these\n      // chunks needs to weigh the benefit of fast successful inserts, against\n      // the drawback of super slow but more rare fallbacks. There's quickly\n      // diminishing returns though with turning up this value way high.\n      for (const chunk of lodash.chunk(toAdd, 50)) {\n        try {\n          await tx.batchInsert(\n            'refresh_state',\n            chunk.map(item => ({\n              entity_id: uuid(),\n              entity_ref: stringifyEntityRef(item.deferred.entity),\n              unprocessed_entity: JSON.stringify(item.deferred.entity),\n              unprocessed_hash: item.hash,\n              errors: '',\n              location_key: item.deferred.locationKey,\n              next_update_at: tx.fn.now(),\n              last_discovery_at: tx.fn.now(),\n            })),\n            BATCH_SIZE,\n          );\n          await tx.batchInsert(\n            'refresh_state_references',\n            chunk.map(item => ({\n              source_key: options.sourceKey,\n              target_entity_ref: stringifyEntityRef(item.deferred.entity),\n            })),\n            BATCH_SIZE,\n          );\n        } catch (error) {\n          if (!isDatabaseConflictError(error)) {\n            throw error;\n          } else {\n            this.options.logger.debug(\n              `Fast insert path failed, falling back to slow path, ${error}`,\n            );\n            toUpsert.push(...chunk);\n          }\n        }\n      }\n    }\n\n    if (toUpsert.length) {\n      for (const {\n        deferred: { entity, locationKey },\n        hash,\n      } of toUpsert) {\n        const entityRef = stringifyEntityRef(entity);\n\n        try {\n          let ok = await updateUnprocessedEntity({\n            tx,\n            entity,\n            hash,\n            locationKey,\n          });\n          if (!ok) {\n            ok = await insertUnprocessedEntity({\n              tx,\n              entity,\n              hash,\n              locationKey,\n              logger: this.options.logger,\n            });\n          }\n          if (ok) {\n            await tx<DbRefreshStateReferencesRow>('refresh_state_references')\n              .where('target_entity_ref', entityRef)\n              .delete();\n\n            await tx<DbRefreshStateReferencesRow>(\n              'refresh_state_references',\n            ).insert({\n              source_key: options.sourceKey,\n              target_entity_ref: entityRef,\n            });\n          } else {\n            await tx<DbRefreshStateReferencesRow>('refresh_state_references')\n              .where('target_entity_ref', entityRef)\n              .andWhere({ source_key: options.sourceKey })\n              .delete();\n\n            const conflictingKey = await checkLocationKeyConflict({\n              tx,\n              entityRef,\n              locationKey,\n            });\n            if (conflictingKey) {\n              this.options.logger.warn(\n                `Source ${options.sourceKey} detected conflicting entityRef ${entityRef} already referenced by ${conflictingKey} and now also ${locationKey}`,\n              );\n            }\n          }\n        } catch (error) {\n          this.options.logger.error(\n            `Failed to add '${entityRef}' from source '${options.sourceKey}', ${error}`,\n          );\n        }\n      }\n    }\n  }\n\n  async listReferenceSourceKeys(txOpaque: Transaction): Promise<string[]> {\n    const tx = txOpaque as Knex | Knex.Transaction;\n\n    const rows = await tx<DbRefreshStateReferencesRow>(\n      'refresh_state_references',\n    )\n      .distinct('source_key')\n      .whereNotNull('source_key');\n\n    return rows\n      .map(row => row.source_key)\n      .filter((key): key is string => !!key);\n  }\n\n  async refreshByRefreshKeys(\n    txOpaque: Transaction,\n    options: RefreshByKeyOptions,\n  ) {\n    const tx = txOpaque as Knex.Transaction;\n    await refreshByRefreshKeys({ tx, keys: options.keys });\n  }\n\n  private async createDelta(\n    tx: Knex | Knex.Transaction,\n    options: ReplaceUnprocessedEntitiesOptions,\n  ): Promise<{\n    toAdd: { deferred: DeferredEntity; hash: string }[];\n    toUpsert: { deferred: DeferredEntity; hash: string }[];\n    toRemove: string[];\n  }> {\n    if (options.type === 'delta') {\n      const toAdd = new Array<{ deferred: DeferredEntity; hash: string }>();\n      const toUpsert = new Array<{ deferred: DeferredEntity; hash: string }>();\n      const toRemove = options.removed.map(e => e.entityRef);\n\n      for (const chunk of lodash.chunk(options.added, 1000)) {\n        const entityRefs = chunk.map(e => stringifyEntityRef(e.entity));\n        const rows = await tx<DbRefreshStateRow>('refresh_state')\n          .select(['entity_ref', 'unprocessed_hash', 'location_key'])\n          .whereIn('entity_ref', entityRefs);\n        const oldStates = new Map(\n          rows.map(row => [\n            row.entity_ref,\n            {\n              unprocessed_hash: row.unprocessed_hash,\n              location_key: row.location_key,\n            },\n          ]),\n        );\n\n        chunk.forEach((deferred, i) => {\n          const entityRef = entityRefs[i];\n          const newHash = generateStableHash(deferred.entity);\n          const oldState = oldStates.get(entityRef);\n          if (oldState === undefined) {\n            // Add any entity that does not exist in the database\n            toAdd.push({ deferred, hash: newHash });\n          } else if (\n            (deferred.locationKey ?? null) !== (oldState.location_key ?? null)\n          ) {\n            // Remove and then re-add any entity that exists, but with a different location key\n            toRemove.push(entityRef);\n            toAdd.push({ deferred, hash: newHash });\n          } else if (newHash !== oldState.unprocessed_hash) {\n            // Entities with modifications should be pushed through too\n            toUpsert.push({ deferred, hash: newHash });\n          }\n        });\n      }\n\n      return { toAdd, toUpsert, toRemove };\n    }\n\n    // Grab all of the existing references from the same source, and their locationKeys as well\n    const oldRefs = await tx<DbRefreshStateReferencesRow>(\n      'refresh_state_references',\n    )\n      .leftJoin<DbRefreshStateRow>('refresh_state', {\n        target_entity_ref: 'entity_ref',\n      })\n      .where({ source_key: options.sourceKey })\n      .select({\n        target_entity_ref: 'refresh_state_references.target_entity_ref',\n        location_key: 'refresh_state.location_key',\n        unprocessed_hash: 'refresh_state.unprocessed_hash',\n      });\n\n    const items = options.items.map(deferred => ({\n      deferred,\n      ref: stringifyEntityRef(deferred.entity),\n      hash: generateStableHash(deferred.entity),\n    }));\n\n    const oldRefsSet = new Map(\n      oldRefs.map(r => [\n        r.target_entity_ref,\n        {\n          locationKey: r.location_key,\n          oldEntityHash: r.unprocessed_hash,\n        },\n      ]),\n    );\n    const newRefsSet = new Set(items.map(item => item.ref));\n\n    const toAdd = new Array<{ deferred: DeferredEntity; hash: string }>();\n    const toUpsert = new Array<{ deferred: DeferredEntity; hash: string }>();\n    const toRemove = oldRefs\n      .map(row => row.target_entity_ref)\n      .filter(ref => !newRefsSet.has(ref));\n\n    for (const item of items) {\n      const oldRef = oldRefsSet.get(item.ref);\n      const upsertItem = { deferred: item.deferred, hash: item.hash };\n      if (!oldRef) {\n        // Add any entity that does not exist in the database\n        toAdd.push(upsertItem);\n      } else if (\n        (oldRef.locationKey ?? undefined) !==\n        (item.deferred.locationKey ?? undefined)\n      ) {\n        // Remove and then re-add any entity that exists, but with a different location key\n        toRemove.push(item.ref);\n        toAdd.push(upsertItem);\n      } else if (oldRef.oldEntityHash !== item.hash) {\n        // Entities with modifications should be pushed through too\n        toUpsert.push(upsertItem);\n      }\n    }\n\n    return { toAdd, toUpsert, toRemove };\n  }\n}\n"],"names":["rethrowError","deleteWithEagerPruningOfChildren","lodash","uuid","stringifyEntityRef","isDatabaseConflictError","updateUnprocessedEntity","insertUnprocessedEntity","checkLocationKeyConflict","refreshByRefreshKeys","toAdd","toUpsert","toRemove","generateStableHash"],"mappings":";;;;;;;;;;;;;;;;;;AA4CA,MAAM,UAAA,GAAa,EAAA;AAEZ,MAAM,uBAAA,CAAoD;AAAA,EAC9C,OAAA;AAAA,EAKjB,YAAY,OAAA,EAAoD;AAC9D,IAAA,IAAA,CAAK,OAAA,GAAU,OAAA;AAAA,EACjB;AAAA,EAEA,MAAM,YAAe,EAAA,EAAiD;AACpE,IAAA,IAAI;AACF,MAAA,IAAI,MAAA,GAAwB,KAAA,CAAA;AAC5B,MAAA,MAAM,IAAA,CAAK,QAAQ,QAAA,CAAS,WAAA;AAAA,QAC1B,OAAM,EAAA,KAAM;AAIV,UAAA,MAAA,GAAS,MAAM,GAAG,EAAE,CAAA;AAAA,QACtB,CAAA;AAAA,QACA;AAAA;AAAA,UAEE,qBAAA,EAAuB;AAAA;AACzB,OACF;AACA,MAAA,OAAO,MAAA;AAAA,IACT,SAAS,CAAA,EAAG;AACV,MAAA,IAAA,CAAK,OAAA,CAAQ,MAAA,CAAO,KAAA,CAAM,CAAA,0BAAA,EAA6B,CAAC,CAAA,CAAE,CAAA;AAC1D,MAAA,MAAMA,wBAAa,CAAC,CAAA;AAAA,IACtB;AAAA,EACF;AAAA,EAEA,MAAM,0BAAA,CACJ,QAAA,EACA,OAAA,EACe;AACf,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAM,EAAE,OAAO,QAAA,EAAU,QAAA,KAAa,MAAM,IAAA,CAAK,WAAA,CAAY,EAAA,EAAI,OAAO,CAAA;AAExE,IAAA,IAAI,SAAS,MAAA,EAAQ;AACnB,MAAA,MAAM,YAAA,GAAe,MAAMC,iEAAA,CAAiC;AAAA,QAC1D,IAAA,EAAM,EAAA;AAAA,QACN,UAAA,EAAY,QAAA;AAAA,QACZ,WAAW,OAAA,CAAQ;AAAA,OACpB,CAAA;AACD,MAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,KAAA;AAAA,QAClB,YAAY,YAAY,CAAA,WAAA,EAAc,IAAA,CAAK,SAAA,CAAU,QAAQ,CAAC,CAAA;AAAA,OAChE;AAAA,IACF;AAEA,IAAA,IAAI,MAAM,MAAA,EAAQ;AAUhB,MAAA,KAAA,MAAW,KAAA,IAASC,uBAAA,CAAO,KAAA,CAAM,KAAA,EAAO,EAAE,CAAA,EAAG;AAC3C,QAAA,IAAI;AACF,UAAA,MAAM,EAAA,CAAG,WAAA;AAAA,YACP,eAAA;AAAA,YACA,KAAA,CAAM,IAAI,CAAA,IAAA,MAAS;AAAA,cACjB,WAAWC,sBAAA,EAAK;AAAA,cAChB,UAAA,EAAYC,+BAAA,CAAmB,IAAA,CAAK,QAAA,CAAS,MAAM,CAAA;AAAA,cACnD,kBAAA,EAAoB,IAAA,CAAK,SAAA,CAAU,IAAA,CAAK,SAAS,MAAM,CAAA;AAAA,cACvD,kBAAkB,IAAA,CAAK,IAAA;AAAA,cACvB,MAAA,EAAQ,EAAA;AAAA,cACR,YAAA,EAAc,KAAK,QAAA,CAAS,WAAA;AAAA,cAC5B,cAAA,EAAgB,EAAA,CAAG,EAAA,CAAG,GAAA,EAAI;AAAA,cAC1B,iBAAA,EAAmB,EAAA,CAAG,EAAA,CAAG,GAAA;AAAI,aAC/B,CAAE,CAAA;AAAA,YACF;AAAA,WACF;AACA,UAAA,MAAM,EAAA,CAAG,WAAA;AAAA,YACP,0BAAA;AAAA,YACA,KAAA,CAAM,IAAI,CAAA,IAAA,MAAS;AAAA,cACjB,YAAY,OAAA,CAAQ,SAAA;AAAA,cACpB,iBAAA,EAAmBA,+BAAA,CAAmB,IAAA,CAAK,QAAA,CAAS,MAAM;AAAA,aAC5D,CAAE,CAAA;AAAA,YACF;AAAA,WACF;AAAA,QACF,SAAS,KAAA,EAAO;AACd,UAAA,IAAI,CAACC,wCAAA,CAAwB,KAAK,CAAA,EAAG;AACnC,YAAA,MAAM,KAAA;AAAA,UACR,CAAA,MAAO;AACL,YAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,KAAA;AAAA,cAClB,uDAAuD,KAAK,CAAA;AAAA,aAC9D;AACA,YAAA,QAAA,CAAS,IAAA,CAAK,GAAG,KAAK,CAAA;AAAA,UACxB;AAAA,QACF;AAAA,MACF;AAAA,IACF;AAEA,IAAA,IAAI,SAAS,MAAA,EAAQ;AACnB,MAAA,KAAA,MAAW;AAAA,QACT,QAAA,EAAU,EAAE,MAAA,EAAQ,WAAA,EAAY;AAAA,QAChC;AAAA,WACG,QAAA,EAAU;AACb,QAAA,MAAM,SAAA,GAAYD,gCAAmB,MAAM,CAAA;AAE3C,QAAA,IAAI;AACF,UAAA,IAAI,EAAA,GAAK,MAAME,+CAAA,CAAwB;AAAA,YACrC,EAAA;AAAA,YACA,MAAA;AAAA,YACA,IAAA;AAAA,YACA;AAAA,WACD,CAAA;AACD,UAAA,IAAI,CAAC,EAAA,EAAI;AACP,YAAA,EAAA,GAAK,MAAMC,+CAAA,CAAwB;AAAA,cACjC,EAAA;AAAA,cACA,MAAA;AAAA,cACA,IAAA;AAAA,cACA,WAAA;AAAA,cACA,MAAA,EAAQ,KAAK,OAAA,CAAQ;AAAA,aACtB,CAAA;AAAA,UACH;AACA,UAAA,IAAI,EAAA,EAAI;AACN,YAAA,MAAM,GAAgC,0BAA0B,CAAA,CAC7D,MAAM,mBAAA,EAAqB,SAAS,EACpC,MAAA,EAAO;AAEV,YAAA,MAAM,EAAA;AAAA,cACJ;AAAA,cACA,MAAA,CAAO;AAAA,cACP,YAAY,OAAA,CAAQ,SAAA;AAAA,cACpB,iBAAA,EAAmB;AAAA,aACpB,CAAA;AAAA,UACH,CAAA,MAAO;AACL,YAAA,MAAM,EAAA,CAAgC,0BAA0B,CAAA,CAC7D,KAAA,CAAM,qBAAqB,SAAS,CAAA,CACpC,QAAA,CAAS,EAAE,UAAA,EAAY,OAAA,CAAQ,SAAA,EAAW,EAC1C,MAAA,EAAO;AAEV,YAAA,MAAM,cAAA,GAAiB,MAAMC,iDAAA,CAAyB;AAAA,cACpD,EAAA;AAAA,cACA,SAAA;AAAA,cACA;AAAA,aACD,CAAA;AACD,YAAA,IAAI,cAAA,EAAgB;AAClB,cAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,IAAA;AAAA,gBAClB,CAAA,OAAA,EAAU,QAAQ,SAAS,CAAA,gCAAA,EAAmC,SAAS,CAAA,uBAAA,EAA0B,cAAc,iBAAiB,WAAW,CAAA;AAAA,eAC7I;AAAA,YACF;AAAA,UACF;AAAA,QACF,SAAS,KAAA,EAAO;AACd,UAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,KAAA;AAAA,YAClB,kBAAkB,SAAS,CAAA,eAAA,EAAkB,OAAA,CAAQ,SAAS,MAAM,KAAK,CAAA;AAAA,WAC3E;AAAA,QACF;AAAA,MACF;AAAA,IACF;AAAA,EACF;AAAA,EAEA,MAAM,wBAAwB,QAAA,EAA0C;AACtE,IAAA,MAAM,EAAA,GAAK,QAAA;AAEX,IAAA,MAAM,OAAO,MAAM,EAAA;AAAA,MACjB;AAAA,KACF,CACG,QAAA,CAAS,YAAY,CAAA,CACrB,aAAa,YAAY,CAAA;AAE5B,IAAA,OAAO,IAAA,CACJ,GAAA,CAAI,CAAA,GAAA,KAAO,GAAA,CAAI,UAAU,CAAA,CACzB,MAAA,CAAO,CAAC,GAAA,KAAuB,CAAC,CAAC,GAAG,CAAA;AAAA,EACzC;AAAA,EAEA,MAAM,oBAAA,CACJ,QAAA,EACA,OAAA,EACA;AACA,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAMC,0CAAqB,EAAE,EAAA,EAAI,IAAA,EAAM,OAAA,CAAQ,MAAM,CAAA;AAAA,EACvD;AAAA,EAEA,MAAc,WAAA,CACZ,EAAA,EACA,OAAA,EAKC;AACD,IAAA,IAAI,OAAA,CAAQ,SAAS,OAAA,EAAS;AAC5B,MAAA,MAAMC,MAAAA,GAAQ,IAAI,KAAA,EAAkD;AACpE,MAAA,MAAMC,SAAAA,GAAW,IAAI,KAAA,EAAkD;AACvE,MAAA,MAAMC,YAAW,OAAA,CAAQ,OAAA,CAAQ,GAAA,CAAI,CAAA,CAAA,KAAK,EAAE,SAAS,CAAA;AAErD,MAAA,KAAA,MAAW,SAASV,uBAAA,CAAO,KAAA,CAAM,OAAA,CAAQ,KAAA,EAAO,GAAI,CAAA,EAAG;AACrD,QAAA,MAAM,aAAa,KAAA,CAAM,GAAA,CAAI,OAAKE,+BAAA,CAAmB,CAAA,CAAE,MAAM,CAAC,CAAA;AAC9D,QAAA,MAAM,IAAA,GAAO,MAAM,EAAA,CAAsB,eAAe,EACrD,MAAA,CAAO,CAAC,YAAA,EAAc,kBAAA,EAAoB,cAAc,CAAC,CAAA,CACzD,OAAA,CAAQ,cAAc,UAAU,CAAA;AACnC,QAAA,MAAM,YAAY,IAAI,GAAA;AAAA,UACpB,IAAA,CAAK,IAAI,CAAA,GAAA,KAAO;AAAA,YACd,GAAA,CAAI,UAAA;AAAA,YACJ;AAAA,cACE,kBAAkB,GAAA,CAAI,gBAAA;AAAA,cACtB,cAAc,GAAA,CAAI;AAAA;AACpB,WACD;AAAA,SACH;AAEA,QAAA,KAAA,CAAM,OAAA,CAAQ,CAAC,QAAA,EAAU,CAAA,KAAM;AAC7B,UAAA,MAAM,SAAA,GAAY,WAAW,CAAC,CAAA;AAC9B,UAAA,MAAM,OAAA,GAAUS,uBAAA,CAAmB,QAAA,CAAS,MAAM,CAAA;AAClD,UAAA,MAAM,QAAA,GAAW,SAAA,CAAU,GAAA,CAAI,SAAS,CAAA;AACxC,UAAA,IAAI,aAAa,MAAA,EAAW;AAE1B,YAAAH,OAAM,IAAA,CAAK,EAAE,QAAA,EAAU,IAAA,EAAM,SAAS,CAAA;AAAA,UACxC,YACG,QAAA,CAAS,WAAA,IAAe,IAAA,OAAW,QAAA,CAAS,gBAAgB,IAAA,CAAA,EAC7D;AAEA,YAAAE,SAAAA,CAAS,KAAK,SAAS,CAAA;AACvB,YAAAF,OAAM,IAAA,CAAK,EAAE,QAAA,EAAU,IAAA,EAAM,SAAS,CAAA;AAAA,UACxC,CAAA,MAAA,IAAW,OAAA,KAAY,QAAA,CAAS,gBAAA,EAAkB;AAEhD,YAAAC,UAAS,IAAA,CAAK,EAAE,QAAA,EAAU,IAAA,EAAM,SAAS,CAAA;AAAA,UAC3C;AAAA,QACF,CAAC,CAAA;AAAA,MACH;AAEA,MAAA,OAAO,EAAE,KAAA,EAAAD,MAAAA,EAAO,QAAA,EAAAC,SAAAA,EAAU,UAAAC,SAAAA,EAAS;AAAA,IACrC;AAGA,IAAA,MAAM,UAAU,MAAM,EAAA;AAAA,MACpB;AAAA,KACF,CACG,SAA4B,eAAA,EAAiB;AAAA,MAC5C,iBAAA,EAAmB;AAAA,KACpB,EACA,KAAA,CAAM,EAAE,YAAY,OAAA,CAAQ,SAAA,EAAW,CAAA,CACvC,MAAA,CAAO;AAAA,MACN,iBAAA,EAAmB,4CAAA;AAAA,MACnB,YAAA,EAAc,4BAAA;AAAA,MACd,gBAAA,EAAkB;AAAA,KACnB,CAAA;AAEH,IAAA,MAAM,KAAA,GAAQ,OAAA,CAAQ,KAAA,CAAM,GAAA,CAAI,CAAA,QAAA,MAAa;AAAA,MAC3C,QAAA;AAAA,MACA,GAAA,EAAKR,+BAAA,CAAmB,QAAA,CAAS,MAAM,CAAA;AAAA,MACvC,IAAA,EAAMS,uBAAA,CAAmB,QAAA,CAAS,MAAM;AAAA,KAC1C,CAAE,CAAA;AAEF,IAAA,MAAM,aAAa,IAAI,GAAA;AAAA,MACrB,OAAA,CAAQ,IAAI,CAAA,CAAA,KAAK;AAAA,QACf,CAAA,CAAE,iBAAA;AAAA,QACF;AAAA,UACE,aAAa,CAAA,CAAE,YAAA;AAAA,UACf,eAAe,CAAA,CAAE;AAAA;AACnB,OACD;AAAA,KACH;AACA,IAAA,MAAM,UAAA,GAAa,IAAI,GAAA,CAAI,KAAA,CAAM,IAAI,CAAA,IAAA,KAAQ,IAAA,CAAK,GAAG,CAAC,CAAA;AAEtD,IAAA,MAAM,KAAA,GAAQ,IAAI,KAAA,EAAkD;AACpE,IAAA,MAAM,QAAA,GAAW,IAAI,KAAA,EAAkD;AACvE,IAAA,MAAM,QAAA,GAAW,OAAA,CACd,GAAA,CAAI,CAAA,GAAA,KAAO,GAAA,CAAI,iBAAiB,CAAA,CAChC,MAAA,CAAO,CAAA,GAAA,KAAO,CAAC,UAAA,CAAW,GAAA,CAAI,GAAG,CAAC,CAAA;AAErC,IAAA,KAAA,MAAW,QAAQ,KAAA,EAAO;AACxB,MAAA,MAAM,MAAA,GAAS,UAAA,CAAW,GAAA,CAAI,IAAA,CAAK,GAAG,CAAA;AACtC,MAAA,MAAM,aAAa,EAAE,QAAA,EAAU,KAAK,QAAA,EAAU,IAAA,EAAM,KAAK,IAAA,EAAK;AAC9D,MAAA,IAAI,CAAC,MAAA,EAAQ;AAEX,QAAA,KAAA,CAAM,KAAK,UAAU,CAAA;AAAA,MACvB,YACG,MAAA,CAAO,WAAA,IAAe,aACtB,IAAA,CAAK,QAAA,CAAS,eAAe,MAAA,CAAA,EAC9B;AAEA,QAAA,QAAA,CAAS,IAAA,CAAK,KAAK,GAAG,CAAA;AACtB,QAAA,KAAA,CAAM,KAAK,UAAU,CAAA;AAAA,MACvB,CAAA,MAAA,IAAW,MAAA,CAAO,aAAA,KAAkB,IAAA,CAAK,IAAA,EAAM;AAE7C,QAAA,QAAA,CAAS,KAAK,UAAU,CAAA;AAAA,MAC1B;AAAA,IACF;AAEA,IAAA,OAAO,EAAE,KAAA,EAAO,QAAA,EAAU,QAAA,EAAS;AAAA,EACrC;AACF;;;;"}

package/dist/database/metrics.cjs.js

@@ -2,26 +2,68 @@
 
 var metrics = require('../util/metrics.cjs.js');
 
+const ENTITIES_COUNT_TTL_MS = 3e4;
+function createEntitiesCountByKind(knex, options) {
+  const ttlMs = ENTITIES_COUNT_TTL_MS;
+  let inflight;
+  let cached;
+  return () => {
+    if (inflight) {
+      return inflight;
+    }
+    if (cached && Date.now() - cached.at < ttlMs) {
+      return Promise.resolve(cached.data);
+    }
+    inflight = (async () => {
+      try {
+        const data = await queryEntitiesCountByKind(knex);
+        cached = { at: Date.now(), data };
+        return data;
+      } finally {
+        inflight = void 0;
+      }
+    })();
+    return inflight;
+  };
+}
+async function queryEntitiesCountByKind(knex) {
+  const kindExpr = entityRefKindExpression(knex);
+  const rows = await knex(
+    "final_entities"
+  ).whereNotNull("final_entity").select({ kind: kindExpr, count: knex.raw("count(*)") }).groupBy(kindExpr);
+  return new Map(rows.map((row) => [String(row.kind), Number(row.count)]));
+}
+function entityRefKindExpression(knex) {
+  const client = knex.client.config.client;
+  if (client.includes("pg")) {
+    return knex.raw(`split_part(entity_ref, ':', 1)`);
+  }
+  if (client.includes("mysql")) {
+    return knex.raw(`substring_index(entity_ref, ':', 1)`);
+  }
+  return knex.raw(`substr(entity_ref, 1, instr(entity_ref, ':') - 1)`);
+}
 function initDatabaseMetrics(knex, metrics$1) {
   const seenProm = /* @__PURE__ */ new Set();
   const seen = /* @__PURE__ */ new Set();
+  const getEntitiesCountByKind = createEntitiesCountByKind(knex);
   return {
     entities_count_prom: metrics.createGaugeMetric({
       name: "catalog_entities_count",
       help: "Total amount of entities in the catalog. DEPRECATED: Please use opentelemetry metrics instead.",
       labelNames: ["kind"],
       async collect() {
-        const results = await knex("search").where("key", "=", "kind").whereNotNull("value").select({ kind: "value", count: knex.raw("count(*)") }).groupBy("value");
-        results.forEach(({ kind, count }) => {
+        const results = await getEntitiesCountByKind();
+        for (const [kind, count] of results) {
           seenProm.add(kind);
-          this.set({ kind }, Number(count));
-        });
-        seenProm.forEach((kind) => {
-          if (!results.some((r) => r.kind === kind)) {
+          this.set({ kind }, count);
+        }
+        for (const kind of seenProm) {
+          if (!results.has(kind)) {
             this.set({ kind }, 0);
             seenProm.delete(kind);
           }
-        });
+        }
       }
     }),
     registered_locations_prom: metrics.createGaugeMetric({
@@ -47,17 +89,17 @@ function initDatabaseMetrics(knex, metrics$1) {
     entities_count: metrics$1.createObservableGauge("catalog_entities_count", {
       description: "Total amount of entities in the catalog"
     }).addCallback(async (gauge) => {
-      const results = await knex("search").where("key", "=", "kind").whereNotNull("value").select({ kind: "value", count: knex.raw("count(*)") }).groupBy("value");
-      results.forEach(({ kind, count }) => {
+      const results = await getEntitiesCountByKind();
+      for (const [kind, count] of results) {
         seen.add(kind);
-        gauge.observe(Number(count), { kind });
-      });
-      seen.forEach((kind) => {
-        if (!results.some((r) => r.kind === kind)) {
+        gauge.observe(count, { kind });
+      }
+      for (const kind of seen) {
+        if (!results.has(kind)) {
           gauge.observe(0, { kind });
           seen.delete(kind);
         }
-      });
+      }
     }),
     registered_locations: metrics$1.createObservableGauge("catalog_registered_locations_count", {
       description: "Total amount of registered locations in the catalog"
@@ -96,5 +138,7 @@ function initDatabaseMetrics(knex, metrics$1) {
   };
 }
 
+exports.createEntitiesCountByKind = createEntitiesCountByKind;
 exports.initDatabaseMetrics = initDatabaseMetrics;
+exports.queryEntitiesCountByKind = queryEntitiesCountByKind;
 //# sourceMappingURL=metrics.cjs.js.map

package/dist/database/metrics.cjs.js.map

@@ -1 +1 @@
-{"version":3,"file":"metrics.cjs.js","sources":["../../src/database/metrics.ts"],"sourcesContent":["/*\n * Copyright 2021 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { Knex } from 'knex';\nimport { createGaugeMetric } from '../util/metrics';\nimport { DbRelationsRow, DbLocationsRow, DbSearchRow } from './tables';\nimport { MetricsService } from '@backstage/backend-plugin-api/alpha';\n\nexport function initDatabaseMetrics(knex: Knex, metrics: MetricsService) {\n  const seenProm = new Set<string>();\n  const seen = new Set<string>();\n\n  return {\n    entities_count_prom: createGaugeMetric({\n      name: 'catalog_entities_count',\n      help: 'Total amount of entities in the catalog. DEPRECATED: Please use opentelemetry metrics instead.',\n      labelNames: ['kind'],\n      async collect() {\n        const results = await knex<DbSearchRow>('search')\n          .where('key', '=', 'kind')\n          .whereNotNull('value')\n          .select({ kind: 'value', count: knex.raw('count(*)') })\n          .groupBy('value');\n\n        results.forEach(({ kind, count }) => {\n          seenProm.add(kind);\n          this.set({ kind }, Number(count));\n        });\n\n        // Set all the entities that were not seenProm to 0 and delete them from the seenProm set.\n        seenProm.forEach(kind => {\n          if (!results.some(r => r.kind === kind)) {\n            this.set({ kind }, 0);\n            seenProm.delete(kind);\n          }\n        });\n      },\n    }),\n    registered_locations_prom: createGaugeMetric({\n      name: 'catalog_registered_locations_count',\n      help: 'Total amount of registered locations in the catalog. DEPRECATED: Please use opentelemetry metrics instead.',\n      async collect() {\n        const total = await knex<DbLocationsRow>('locations').count({\n          count: '*',\n        });\n        this.set(Number(total[0].count));\n      },\n    }),\n    relations_prom: createGaugeMetric({\n      name: 'catalog_relations_count',\n      help: 'Total amount of relations between entities. DEPRECATED: Please use opentelemetry metrics instead.',\n      async collect() {\n        const total = await knex<DbRelationsRow>('relations').count({\n          count: '*',\n        });\n        this.set(Number(total[0].count));\n      },\n    }),\n    entities_count: metrics\n      .createObservableGauge('catalog_entities_count', {\n        description: 'Total amount of entities in the catalog',\n      })\n      .addCallback(async gauge => {\n        const results = await knex<DbSearchRow>('search')\n          .where('key', '=', 'kind')\n          .whereNotNull('value')\n          .select({ kind: 'value', count: knex.raw('count(*)') })\n          .groupBy('value');\n\n        results.forEach(({ kind, count }) => {\n          seen.add(kind);\n          gauge.observe(Number(count), { kind });\n        });\n\n        // Set all the entities that were not seen to 0 and delete them from the seen set.\n        seen.forEach(kind => {\n          if (!results.some(r => r.kind === kind)) {\n            gauge.observe(0, { kind });\n            seen.delete(kind);\n          }\n        });\n      }),\n    registered_locations: metrics\n      .createObservableGauge('catalog_registered_locations_count', {\n        description: 'Total amount of registered locations in the catalog',\n      })\n      .addCallback(async gauge => {\n        if (knex.client.config.client === 'pg') {\n          // https://stackoverflow.com/questions/7943233/fast-way-to-discover-the-row-count-of-a-table-in-postgresql\n          const total = await knex.raw(`\n            SELECT reltuples::bigint AS estimate\n            FROM   pg_class\n            WHERE  oid = 'locations'::regclass;\n          `);\n          gauge.observe(Number(total.rows[0].estimate));\n        } else {\n          const total = await knex<DbLocationsRow>('locations').count({\n            count: '*',\n          });\n          gauge.observe(Number(total[0].count));\n        }\n      }),\n    relations: metrics\n      .createObservableGauge('catalog_relations_count', {\n        description: 'Total amount of relations between entities',\n      })\n      .addCallback(async gauge => {\n        if (knex.client.config.client === 'pg') {\n          // https://stackoverflow.com/questions/7943233/fast-way-to-discover-the-row-count-of-a-table-in-postgresql\n          const total = await knex.raw(`\n            SELECT reltuples::bigint AS estimate\n            FROM   pg_class\n            WHERE  oid = 'relations'::regclass;\n          `);\n          gauge.observe(Number(total.rows[0].estimate));\n        } else {\n          const total = await knex<DbRelationsRow>('relations').count({\n            count: '*',\n          });\n          gauge.observe(Number(total[0].count));\n        }\n      }),\n  };\n}\n"],"names":["metrics","createGaugeMetric"],"mappings":";;;;AAqBO,SAAS,mBAAA,CAAoB,MAAYA,SAAA,EAAyB;AACvE,EAAA,MAAM,QAAA,uBAAe,GAAA,EAAY;AACjC,EAAA,MAAM,IAAA,uBAAW,GAAA,EAAY;AAE7B,EAAA,OAAO;AAAA,IACL,qBAAqBC,yBAAA,CAAkB;AAAA,MACrC,IAAA,EAAM,wBAAA;AAAA,MACN,IAAA,EAAM,gGAAA;AAAA,MACN,UAAA,EAAY,CAAC,MAAM,CAAA;AAAA,MACnB,MAAM,OAAA,GAAU;AACd,QAAA,MAAM,OAAA,GAAU,MAAM,IAAA,CAAkB,QAAQ,CAAA,CAC7C,MAAM,KAAA,EAAO,GAAA,EAAK,MAAM,CAAA,CACxB,YAAA,CAAa,OAAO,EACpB,MAAA,CAAO,EAAE,IAAA,EAAM,OAAA,EAAS,KAAA,EAAO,IAAA,CAAK,GAAA,CAAI,UAAU,CAAA,EAAG,CAAA,CACrD,OAAA,CAAQ,OAAO,CAAA;AAElB,QAAA,OAAA,CAAQ,OAAA,CAAQ,CAAC,EAAE,IAAA,EAAM,OAAM,KAAM;AACnC,UAAA,QAAA,CAAS,IAAI,IAAI,CAAA;AACjB,UAAA,IAAA,CAAK,IAAI,EAAE,IAAA,EAAK,EAAG,MAAA,CAAO,KAAK,CAAC,CAAA;AAAA,QAClC,CAAC,CAAA;AAGD,QAAA,QAAA,CAAS,QAAQ,CAAA,IAAA,KAAQ;AACvB,UAAA,IAAI,CAAC,OAAA,CAAQ,IAAA,CAAK,OAAK,CAAA,CAAE,IAAA,KAAS,IAAI,CAAA,EAAG;AACvC,YAAA,IAAA,CAAK,GAAA,CAAI,EAAE,IAAA,EAAK,EAAG,CAAC,CAAA;AACpB,YAAA,QAAA,CAAS,OAAO,IAAI,CAAA;AAAA,UACtB;AAAA,QACF,CAAC,CAAA;AAAA,MACH;AAAA,KACD,CAAA;AAAA,IACD,2BAA2BA,yBAAA,CAAkB;AAAA,MAC3C,IAAA,EAAM,oCAAA;AAAA,MACN,IAAA,EAAM,4GAAA;AAAA,MACN,MAAM,OAAA,GAAU;AACd,QAAA,MAAM,KAAA,GAAQ,MAAM,IAAA,CAAqB,WAAW,EAAE,KAAA,CAAM;AAAA,UAC1D,KAAA,EAAO;AAAA,SACR,CAAA;AACD,QAAA,IAAA,CAAK,IAAI,MAAA,CAAO,KAAA,CAAM,CAAC,CAAA,CAAE,KAAK,CAAC,CAAA;AAAA,MACjC;AAAA,KACD,CAAA;AAAA,IACD,gBAAgBA,yBAAA,CAAkB;AAAA,MAChC,IAAA,EAAM,yBAAA;AAAA,MACN,IAAA,EAAM,mGAAA;AAAA,MACN,MAAM,OAAA,GAAU;AACd,QAAA,MAAM,KAAA,GAAQ,MAAM,IAAA,CAAqB,WAAW,EAAE,KAAA,CAAM;AAAA,UAC1D,KAAA,EAAO;AAAA,SACR,CAAA;AACD,QAAA,IAAA,CAAK,IAAI,MAAA,CAAO,KAAA,CAAM,CAAC,CAAA,CAAE,KAAK,CAAC,CAAA;AAAA,MACjC;AAAA,KACD,CAAA;AAAA,IACD,cAAA,EAAgBD,SAAA,CACb,qBAAA,CAAsB,wBAAA,EAA0B;AAAA,MAC/C,WAAA,EAAa;AAAA,KACd,CAAA,CACA,WAAA,CAAY,OAAM,KAAA,KAAS;AAC1B,MAAA,MAAM,OAAA,GAAU,MAAM,IAAA,CAAkB,QAAQ,CAAA,CAC7C,MAAM,KAAA,EAAO,GAAA,EAAK,MAAM,CAAA,CACxB,YAAA,CAAa,OAAO,EACpB,MAAA,CAAO,EAAE,IAAA,EAAM,OAAA,EAAS,KAAA,EAAO,IAAA,CAAK,GAAA,CAAI,UAAU,CAAA,EAAG,CAAA,CACrD,OAAA,CAAQ,OAAO,CAAA;AAElB,MAAA,OAAA,CAAQ,OAAA,CAAQ,CAAC,EAAE,IAAA,EAAM,OAAM,KAAM;AACnC,QAAA,IAAA,CAAK,IAAI,IAAI,CAAA;AACb,QAAA,KAAA,CAAM,QAAQ,MAAA,CAAO,KAAK,CAAA,EAAG,EAAE,MAAM,CAAA;AAAA,MACvC,CAAC,CAAA;AAGD,MAAA,IAAA,CAAK,QAAQ,CAAA,IAAA,KAAQ;AACnB,QAAA,IAAI,CAAC,OAAA,CAAQ,IAAA,CAAK,OAAK,CAAA,CAAE,IAAA,KAAS,IAAI,CAAA,EAAG;AACvC,UAAA,KAAA,CAAM,OAAA,CAAQ,CAAA,EAAG,EAAE,IAAA,EAAM,CAAA;AACzB,UAAA,IAAA,CAAK,OAAO,IAAI,CAAA;AAAA,QAClB;AAAA,MACF,CAAC,CAAA;AAAA,IACH,CAAC,CAAA;AAAA,IACH,oBAAA,EAAsBA,SAAA,CACnB,qBAAA,CAAsB,oCAAA,EAAsC;AAAA,MAC3D,WAAA,EAAa;AAAA,KACd,CAAA,CACA,WAAA,CAAY,OAAM,KAAA,KAAS;AAC1B,MAAA,IAAI,IAAA,CAAK,MAAA,CAAO,MAAA,CAAO,MAAA,KAAW,IAAA,EAAM;AAEtC,QAAA,MAAM,KAAA,GAAQ,MAAM,IAAA,CAAK,GAAA,CAAI;AAAA;AAAA;AAAA;AAAA,UAAA,CAI5B,CAAA;AACD,QAAA,KAAA,CAAM,QAAQ,MAAA,CAAO,KAAA,CAAM,KAAK,CAAC,CAAA,CAAE,QAAQ,CAAC,CAAA;AAAA,MAC9C,CAAA,MAAO;AACL,QAAA,MAAM,KAAA,GAAQ,MAAM,IAAA,CAAqB,WAAW,EAAE,KAAA,CAAM;AAAA,UAC1D,KAAA,EAAO;AAAA,SACR,CAAA;AACD,QAAA,KAAA,CAAM,QAAQ,MAAA,CAAO,KAAA,CAAM,CAAC,CAAA,CAAE,KAAK,CAAC,CAAA;AAAA,MACtC;AAAA,IACF,CAAC,CAAA;AAAA,IACH,SAAA,EAAWA,SAAA,CACR,qBAAA,CAAsB,yBAAA,EAA2B;AAAA,MAChD,WAAA,EAAa;AAAA,KACd,CAAA,CACA,WAAA,CAAY,OAAM,KAAA,KAAS;AAC1B,MAAA,IAAI,IAAA,CAAK,MAAA,CAAO,MAAA,CAAO,MAAA,KAAW,IAAA,EAAM;AAEtC,QAAA,MAAM,KAAA,GAAQ,MAAM,IAAA,CAAK,GAAA,CAAI;AAAA;AAAA;AAAA;AAAA,UAAA,CAI5B,CAAA;AACD,QAAA,KAAA,CAAM,QAAQ,MAAA,CAAO,KAAA,CAAM,KAAK,CAAC,CAAA,CAAE,QAAQ,CAAC,CAAA;AAAA,MAC9C,CAAA,MAAO;AACL,QAAA,MAAM,KAAA,GAAQ,MAAM,IAAA,CAAqB,WAAW,EAAE,KAAA,CAAM;AAAA,UAC1D,KAAA,EAAO;AAAA,SACR,CAAA;AACD,QAAA,KAAA,CAAM,QAAQ,MAAA,CAAO,KAAA,CAAM,CAAC,CAAA,CAAE,KAAK,CAAC,CAAA;AAAA,MACtC;AAAA,IACF,CAAC;AAAA,GACL;AACF;;;;"}
+{"version":3,"file":"metrics.cjs.js","sources":["../../src/database/metrics.ts"],"sourcesContent":["/*\n * Copyright 2021 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { Knex } from 'knex';\nimport { createGaugeMetric } from '../util/metrics';\nimport { DbRelationsRow, DbLocationsRow } from './tables';\nimport { MetricsService } from '@backstage/backend-plugin-api/alpha';\n\nconst ENTITIES_COUNT_TTL_MS = 30_000;\n\n/**\n * Returns a function that produces a Map of entity kind -> count, with\n * a single-flight cache that coalesces overlapping calls.\n *\n * The OpenTelemetry observable gauge and the legacy Prometheus gauge are\n * both registered to emit the same `catalog_entities_count` series, and\n * both fire on every metrics scrape. Without coalescing, that means two\n * identical heavy queries per scrape per pod, which can pile up against\n * the database faster than they complete.\n *\n * Concurrent callers share the same in-flight promise, so a query is\n * never overlapped by another instance of itself. The TTL is the\n * minimum age at which the next caller is allowed to start a fresh\n * query; if a query is still running when the TTL would have elapsed,\n * waiting callers continue to share that one rather than starting a\n * duplicate.\n *\n * @internal exported for testing\n */\nexport function createEntitiesCountByKind(\n  knex: Knex,\n  options?: { ttlMs?: number },\n): () => Promise<Map<string, number>> {\n  const ttlMs = options?.ttlMs ?? ENTITIES_COUNT_TTL_MS;\n  let inflight: Promise<Map<string, number>> | undefined;\n  let cached: { at: number; data: Map<string, number> } | undefined;\n  return () => {\n    if (inflight) {\n      return inflight;\n    }\n    if (cached && Date.now() - cached.at < ttlMs) {\n      return Promise.resolve(cached.data);\n    }\n    inflight = (async () => {\n      try {\n        const data = await queryEntitiesCountByKind(knex);\n        cached = { at: Date.now(), data };\n        return data;\n      } finally {\n        inflight = undefined;\n      }\n    })();\n    return inflight;\n  };\n}\n\n/**\n * Reads kind counts straight from `final_entities` (one row per entity)\n * rather than from `search` (one row per entity per indexed key, often\n * 20-30x larger). The kind is parsed out of `entity_ref`, which is the\n * canonical lowercased `kind:namespace/name` form -- producing the same\n * lowercased labels the previous search-based query did.\n *\n * Rows where `final_entity` is null are excluded. They represent entities\n * that haven't been stitched yet, or tombstones for in-progress deletions.\n * Counting them would over-report by including entities the rest of the\n * catalog API treats as not present.\n *\n * @internal exported for testing\n */\nexport async function queryEntitiesCountByKind(\n  knex: Knex,\n): Promise<Map<string, number>> {\n  const kindExpr = entityRefKindExpression(knex);\n\n  const rows: { kind: string; count: string | number }[] = await knex(\n    'final_entities',\n  )\n    .whereNotNull('final_entity')\n    .select({ kind: kindExpr, count: knex.raw('count(*)') })\n    .groupBy(kindExpr);\n\n  return new Map(rows.map(row => [String(row.kind), Number(row.count)]));\n}\n\nfunction entityRefKindExpression(knex: Knex): Knex.Raw {\n  const client = knex.client.config.client as string;\n  if (client.includes('pg')) {\n    return knex.raw(`split_part(entity_ref, ':', 1)`);\n  }\n  if (client.includes('mysql')) {\n    return knex.raw(`substring_index(entity_ref, ':', 1)`);\n  }\n  // sqlite (better-sqlite3, sqlite3)\n  return knex.raw(`substr(entity_ref, 1, instr(entity_ref, ':') - 1)`);\n}\n\nexport function initDatabaseMetrics(knex: Knex, metrics: MetricsService) {\n  const seenProm = new Set<string>();\n  const seen = new Set<string>();\n  const getEntitiesCountByKind = createEntitiesCountByKind(knex);\n\n  return {\n    entities_count_prom: createGaugeMetric({\n      name: 'catalog_entities_count',\n      help: 'Total amount of entities in the catalog. DEPRECATED: Please use opentelemetry metrics instead.',\n      labelNames: ['kind'],\n      async collect() {\n        const results = await getEntitiesCountByKind();\n\n        for (const [kind, count] of results) {\n          seenProm.add(kind);\n          this.set({ kind }, count);\n        }\n\n        // Set all the entities that were not seenProm to 0 and delete them from the seenProm set.\n        for (const kind of seenProm) {\n          if (!results.has(kind)) {\n            this.set({ kind }, 0);\n            seenProm.delete(kind);\n          }\n        }\n      },\n    }),\n    registered_locations_prom: createGaugeMetric({\n      name: 'catalog_registered_locations_count',\n      help: 'Total amount of registered locations in the catalog. DEPRECATED: Please use opentelemetry metrics instead.',\n      async collect() {\n        const total = await knex<DbLocationsRow>('locations').count({\n          count: '*',\n        });\n        this.set(Number(total[0].count));\n      },\n    }),\n    relations_prom: createGaugeMetric({\n      name: 'catalog_relations_count',\n      help: 'Total amount of relations between entities. DEPRECATED: Please use opentelemetry metrics instead.',\n      async collect() {\n        const total = await knex<DbRelationsRow>('relations').count({\n          count: '*',\n        });\n        this.set(Number(total[0].count));\n      },\n    }),\n    entities_count: metrics\n      .createObservableGauge('catalog_entities_count', {\n        description: 'Total amount of entities in the catalog',\n      })\n      .addCallback(async gauge => {\n        const results = await getEntitiesCountByKind();\n\n        for (const [kind, count] of results) {\n          seen.add(kind);\n          gauge.observe(count, { kind });\n        }\n\n        // Set all the entities that were not seen to 0 and delete them from the seen set.\n        for (const kind of seen) {\n          if (!results.has(kind)) {\n            gauge.observe(0, { kind });\n            seen.delete(kind);\n          }\n        }\n      }),\n    registered_locations: metrics\n      .createObservableGauge('catalog_registered_locations_count', {\n        description: 'Total amount of registered locations in the catalog',\n      })\n      .addCallback(async gauge => {\n        if (knex.client.config.client === 'pg') {\n          // https://stackoverflow.com/questions/7943233/fast-way-to-discover-the-row-count-of-a-table-in-postgresql\n          const total = await knex.raw(`\n            SELECT reltuples::bigint AS estimate\n            FROM   pg_class\n            WHERE  oid = 'locations'::regclass;\n          `);\n          gauge.observe(Number(total.rows[0].estimate));\n        } else {\n          const total = await knex<DbLocationsRow>('locations').count({\n            count: '*',\n          });\n          gauge.observe(Number(total[0].count));\n        }\n      }),\n    relations: metrics\n      .createObservableGauge('catalog_relations_count', {\n        description: 'Total amount of relations between entities',\n      })\n      .addCallback(async gauge => {\n        if (knex.client.config.client === 'pg') {\n          // https://stackoverflow.com/questions/7943233/fast-way-to-discover-the-row-count-of-a-table-in-postgresql\n          const total = await knex.raw(`\n            SELECT reltuples::bigint AS estimate\n            FROM   pg_class\n            WHERE  oid = 'relations'::regclass;\n          `);\n          gauge.observe(Number(total.rows[0].estimate));\n        } else {\n          const total = await knex<DbRelationsRow>('relations').count({\n            count: '*',\n          });\n          gauge.observe(Number(total[0].count));\n        }\n      }),\n  };\n}\n"],"names":["metrics","createGaugeMetric"],"mappings":";;;;AAqBA,MAAM,qBAAA,GAAwB,GAAA;AAqBvB,SAAS,yBAAA,CACd,MACA,OAAA,EACoC;AACpC,EAAA,MAAM,KAAA,GAA0B,qBAAA;AAChC,EAAA,IAAI,QAAA;AACJ,EAAA,IAAI,MAAA;AACJ,EAAA,OAAO,MAAM;AACX,IAAA,IAAI,QAAA,EAAU;AACZ,MAAA,OAAO,QAAA;AAAA,IACT;AACA,IAAA,IAAI,UAAU,IAAA,CAAK,GAAA,EAAI,GAAI,MAAA,CAAO,KAAK,KAAA,EAAO;AAC5C,MAAA,OAAO,OAAA,CAAQ,OAAA,CAAQ,MAAA,CAAO,IAAI,CAAA;AAAA,IACpC;AACA,IAAA,QAAA,GAAA,CAAY,YAAY;AACtB,MAAA,IAAI;AACF,QAAA,MAAM,IAAA,GAAO,MAAM,wBAAA,CAAyB,IAAI,CAAA;AAChD,QAAA,MAAA,GAAS,EAAE,EAAA,EAAI,IAAA,CAAK,GAAA,IAAO,IAAA,EAAK;AAChC,QAAA,OAAO,IAAA;AAAA,MACT,CAAA,SAAE;AACA,QAAA,QAAA,GAAW,MAAA;AAAA,MACb;AAAA,IACF,CAAA,GAAG;AACH,IAAA,OAAO,QAAA;AAAA,EACT,CAAA;AACF;AAgBA,eAAsB,yBACpB,IAAA,EAC8B;AAC9B,EAAA,MAAM,QAAA,GAAW,wBAAwB,IAAI,CAAA;AAE7C,EAAA,MAAM,OAAmD,MAAM,IAAA;AAAA,IAC7D;AAAA,IAEC,YAAA,CAAa,cAAc,CAAA,CAC3B,MAAA,CAAO,EAAE,IAAA,EAAM,QAAA,EAAU,KAAA,EAAO,IAAA,CAAK,IAAI,UAAU,CAAA,EAAG,CAAA,CACtD,QAAQ,QAAQ,CAAA;AAEnB,EAAA,OAAO,IAAI,GAAA,CAAI,IAAA,CAAK,GAAA,CAAI,SAAO,CAAC,MAAA,CAAO,GAAA,CAAI,IAAI,GAAG,MAAA,CAAO,GAAA,CAAI,KAAK,CAAC,CAAC,CAAC,CAAA;AACvE;AAEA,SAAS,wBAAwB,IAAA,EAAsB;AACrD,EAAA,MAAM,MAAA,GAAS,IAAA,CAAK,MAAA,CAAO,MAAA,CAAO,MAAA;AAClC,EAAA,IAAI,MAAA,CAAO,QAAA,CAAS,IAAI,CAAA,EAAG;AACzB,IAAA,OAAO,IAAA,CAAK,IAAI,CAAA,8BAAA,CAAgC,CAAA;AAAA,EAClD;AACA,EAAA,IAAI,MAAA,CAAO,QAAA,CAAS,OAAO,CAAA,EAAG;AAC5B,IAAA,OAAO,IAAA,CAAK,IAAI,CAAA,mCAAA,CAAqC,CAAA;AAAA,EACvD;AAEA,EAAA,OAAO,IAAA,CAAK,IAAI,CAAA,iDAAA,CAAmD,CAAA;AACrE;AAEO,SAAS,mBAAA,CAAoB,MAAYA,SAAA,EAAyB;AACvE,EAAA,MAAM,QAAA,uBAAe,GAAA,EAAY;AACjC,EAAA,MAAM,IAAA,uBAAW,GAAA,EAAY;AAC7B,EAAA,MAAM,sBAAA,GAAyB,0BAA0B,IAAI,CAAA;AAE7D,EAAA,OAAO;AAAA,IACL,qBAAqBC,yBAAA,CAAkB;AAAA,MACrC,IAAA,EAAM,wBAAA;AAAA,MACN,IAAA,EAAM,gGAAA;AAAA,MACN,UAAA,EAAY,CAAC,MAAM,CAAA;AAAA,MACnB,MAAM,OAAA,GAAU;AACd,QAAA,MAAM,OAAA,GAAU,MAAM,sBAAA,EAAuB;AAE7C,QAAA,KAAA,MAAW,CAAC,IAAA,EAAM,KAAK,CAAA,IAAK,OAAA,EAAS;AACnC,UAAA,QAAA,CAAS,IAAI,IAAI,CAAA;AACjB,UAAA,IAAA,CAAK,GAAA,CAAI,EAAE,IAAA,EAAK,EAAG,KAAK,CAAA;AAAA,QAC1B;AAGA,QAAA,KAAA,MAAW,QAAQ,QAAA,EAAU;AAC3B,UAAA,IAAI,CAAC,OAAA,CAAQ,GAAA,CAAI,IAAI,CAAA,EAAG;AACtB,YAAA,IAAA,CAAK,GAAA,CAAI,EAAE,IAAA,EAAK,EAAG,CAAC,CAAA;AACpB,YAAA,QAAA,CAAS,OAAO,IAAI,CAAA;AAAA,UACtB;AAAA,QACF;AAAA,MACF;AAAA,KACD,CAAA;AAAA,IACD,2BAA2BA,yBAAA,CAAkB;AAAA,MAC3C,IAAA,EAAM,oCAAA;AAAA,MACN,IAAA,EAAM,4GAAA;AAAA,MACN,MAAM,OAAA,GAAU;AACd,QAAA,MAAM,KAAA,GAAQ,MAAM,IAAA,CAAqB,WAAW,EAAE,KAAA,CAAM;AAAA,UAC1D,KAAA,EAAO;AAAA,SACR,CAAA;AACD,QAAA,IAAA,CAAK,IAAI,MAAA,CAAO,KAAA,CAAM,CAAC,CAAA,CAAE,KAAK,CAAC,CAAA;AAAA,MACjC;AAAA,KACD,CAAA;AAAA,IACD,gBAAgBA,yBAAA,CAAkB;AAAA,MAChC,IAAA,EAAM,yBAAA;AAAA,MACN,IAAA,EAAM,mGAAA;AAAA,MACN,MAAM,OAAA,GAAU;AACd,QAAA,MAAM,KAAA,GAAQ,MAAM,IAAA,CAAqB,WAAW,EAAE,KAAA,CAAM;AAAA,UAC1D,KAAA,EAAO;AAAA,SACR,CAAA;AACD,QAAA,IAAA,CAAK,IAAI,MAAA,CAAO,KAAA,CAAM,CAAC,CAAA,CAAE,KAAK,CAAC,CAAA;AAAA,MACjC;AAAA,KACD,CAAA;AAAA,IACD,cAAA,EAAgBD,SAAA,CACb,qBAAA,CAAsB,wBAAA,EAA0B;AAAA,MAC/C,WAAA,EAAa;AAAA,KACd,CAAA,CACA,WAAA,CAAY,OAAM,KAAA,KAAS;AAC1B,MAAA,MAAM,OAAA,GAAU,MAAM,sBAAA,EAAuB;AAE7C,MAAA,KAAA,MAAW,CAAC,IAAA,EAAM,KAAK,CAAA,IAAK,OAAA,EAAS;AACnC,QAAA,IAAA,CAAK,IAAI,IAAI,CAAA;AACb,QAAA,KAAA,CAAM,OAAA,CAAQ,KAAA,EAAO,EAAE,IAAA,EAAM,CAAA;AAAA,MAC/B;AAGA,MAAA,KAAA,MAAW,QAAQ,IAAA,EAAM;AACvB,QAAA,IAAI,CAAC,OAAA,CAAQ,GAAA,CAAI,IAAI,CAAA,EAAG;AACtB,UAAA,KAAA,CAAM,OAAA,CAAQ,CAAA,EAAG,EAAE,IAAA,EAAM,CAAA;AACzB,UAAA,IAAA,CAAK,OAAO,IAAI,CAAA;AAAA,QAClB;AAAA,MACF;AAAA,IACF,CAAC,CAAA;AAAA,IACH,oBAAA,EAAsBA,SAAA,CACnB,qBAAA,CAAsB,oCAAA,EAAsC;AAAA,MAC3D,WAAA,EAAa;AAAA,KACd,CAAA,CACA,WAAA,CAAY,OAAM,KAAA,KAAS;AAC1B,MAAA,IAAI,IAAA,CAAK,MAAA,CAAO,MAAA,CAAO,MAAA,KAAW,IAAA,EAAM;AAEtC,QAAA,MAAM,KAAA,GAAQ,MAAM,IAAA,CAAK,GAAA,CAAI;AAAA;AAAA;AAAA;AAAA,UAAA,CAI5B,CAAA;AACD,QAAA,KAAA,CAAM,QAAQ,MAAA,CAAO,KAAA,CAAM,KAAK,CAAC,CAAA,CAAE,QAAQ,CAAC,CAAA;AAAA,MAC9C,CAAA,MAAO;AACL,QAAA,MAAM,KAAA,GAAQ,MAAM,IAAA,CAAqB,WAAW,EAAE,KAAA,CAAM;AAAA,UAC1D,KAAA,EAAO;AAAA,SACR,CAAA;AACD,QAAA,KAAA,CAAM,QAAQ,MAAA,CAAO,KAAA,CAAM,CAAC,CAAA,CAAE,KAAK,CAAC,CAAA;AAAA,MACtC;AAAA,IACF,CAAC,CAAA;AAAA,IACH,SAAA,EAAWA,SAAA,CACR,qBAAA,CAAsB,yBAAA,EAA2B;AAAA,MAChD,WAAA,EAAa;AAAA,KACd,CAAA,CACA,WAAA,CAAY,OAAM,KAAA,KAAS;AAC1B,MAAA,IAAI,IAAA,CAAK,MAAA,CAAO,MAAA,CAAO,MAAA,KAAW,IAAA,EAAM;AAEtC,QAAA,MAAM,KAAA,GAAQ,MAAM,IAAA,CAAK,GAAA,CAAI;AAAA;AAAA;AAAA;AAAA,UAAA,CAI5B,CAAA;AACD,QAAA,KAAA,CAAM,QAAQ,MAAA,CAAO,KAAA,CAAM,KAAK,CAAC,CAAA,CAAE,QAAQ,CAAC,CAAA;AAAA,MAC9C,CAAA,MAAO;AACL,QAAA,MAAM,KAAA,GAAQ,MAAM,IAAA,CAAqB,WAAW,EAAE,KAAA,CAAM;AAAA,UAC1D,KAAA,EAAO;AAAA,SACR,CAAA;AACD,QAAA,KAAA,CAAM,QAAQ,MAAA,CAAO,KAAA,CAAM,CAAC,CAAA,CAAE,KAAK,CAAC,CAAA;AAAA,MACtC;AAAA,IACF,CAAC;AAAA,GACL;AACF;;;;;;"}

package/dist/database/operations/refreshState/insertUnprocessedEntity.cjs.js

@@ -1,7 +1,7 @@
 'use strict';
 
 var catalogModel = require('@backstage/catalog-model');
-var uuid = require('uuid');
+var node_crypto = require('node:crypto');
 var backendPluginApi = require('@backstage/backend-plugin-api');
 
 async function insertUnprocessedEntity(options) {
@@ -10,7 +10,7 @@ async function insertUnprocessedEntity(options) {
   const serializedEntity = JSON.stringify(entity);
   try {
     let query = tx("refresh_state").insert({
-      entity_id: uuid.v4(),
+      entity_id: node_crypto.randomUUID(),
       entity_ref: entityRef,
       unprocessed_entity: serializedEntity,
       unprocessed_hash: hash,

package/dist/database/operations/refreshState/insertUnprocessedEntity.cjs.js.map

@@ -1 +1 @@
-{"version":3,"file":"insertUnprocessedEntity.cjs.js","sources":["../../../../src/database/operations/refreshState/insertUnprocessedEntity.ts"],"sourcesContent":["/*\n * Copyright 2022 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { Entity, stringifyEntityRef } from '@backstage/catalog-model';\nimport { Knex } from 'knex';\nimport { DbRefreshStateRow } from '../../tables';\nimport { v4 as uuid } from 'uuid';\nimport {\n  LoggerService,\n  isDatabaseConflictError,\n} from '@backstage/backend-plugin-api';\n\n/**\n * Attempts to insert a new refresh state row for the given entity, returning\n * true if successful and false if there was a conflict.\n */\nexport async function insertUnprocessedEntity(options: {\n  tx: Knex | Knex.Transaction;\n  entity: Entity;\n  hash: string;\n  locationKey?: string;\n  logger: LoggerService;\n}): Promise<boolean> {\n  const { tx, entity, hash, logger, locationKey } = options;\n\n  const entityRef = stringifyEntityRef(entity);\n  const serializedEntity = JSON.stringify(entity);\n\n  try {\n    let query = tx<DbRefreshStateRow>('refresh_state').insert({\n      entity_id: uuid(),\n      entity_ref: entityRef,\n      unprocessed_entity: serializedEntity,\n      unprocessed_hash: hash,\n      errors: '',\n      location_key: locationKey,\n      next_update_at: tx.fn.now(),\n      last_discovery_at: tx.fn.now(),\n    });\n\n    // TODO(Rugvip): only tested towards MySQL, Postgres and SQLite.\n    // We have to do this because the only way to detect if there was a conflict with\n    // SQLite is to catch the error, while Postgres needs to ignore the conflict to not\n    // break the ongoing transaction.\n    if (tx.client.config.client.includes('pg')) {\n      query = query.onConflict('entity_ref').ignore() as any; // type here does not match runtime\n    }\n\n    // Postgres gives as an object with rowCount, SQLite gives us an array\n    const result: { rowCount?: number; length?: number } = await query;\n    return result.rowCount === 1 || result.length === 1;\n  } catch (error) {\n    // SQLite, or MySQL reached this rather than the rowCount check above\n    if (!isDatabaseConflictError(error)) {\n      throw error;\n    } else {\n      logger.debug(`Unable to insert a new refresh state row, ${error}`);\n      return false;\n    }\n  }\n}\n"],"names":["stringifyEntityRef","uuid","isDatabaseConflictError"],"mappings":";;;;;;AA6BA,eAAsB,wBAAwB,OAAA,EAMzB;AACnB,EAAA,MAAM,EAAE,EAAA,EAAI,MAAA,EAAQ,IAAA,EAAM,MAAA,EAAQ,aAAY,GAAI,OAAA;AAElD,EAAA,MAAM,SAAA,GAAYA,gCAAmB,MAAM,CAAA;AAC3C,EAAA,MAAM,gBAAA,GAAmB,IAAA,CAAK,SAAA,CAAU,MAAM,CAAA;AAE9C,EAAA,IAAI;AACF,IAAA,IAAI,KAAA,GAAQ,EAAA,CAAsB,eAAe,CAAA,CAAE,MAAA,CAAO;AAAA,MACxD,WAAWC,OAAA,EAAK;AAAA,MAChB,UAAA,EAAY,SAAA;AAAA,MACZ,kBAAA,EAAoB,gBAAA;AAAA,MACpB,gBAAA,EAAkB,IAAA;AAAA,MAClB,MAAA,EAAQ,EAAA;AAAA,MACR,YAAA,EAAc,WAAA;AAAA,MACd,cAAA,EAAgB,EAAA,CAAG,EAAA,CAAG,GAAA,EAAI;AAAA,MAC1B,iBAAA,EAAmB,EAAA,CAAG,EAAA,CAAG,GAAA;AAAI,KAC9B,CAAA;AAMD,IAAA,IAAI,GAAG,MAAA,CAAO,MAAA,CAAO,MAAA,CAAO,QAAA,CAAS,IAAI,CAAA,EAAG;AAC1C,MAAA,KAAA,GAAQ,KAAA,CAAM,UAAA,CAAW,YAAY,CAAA,CAAE,MAAA,EAAO;AAAA,IAChD;AAGA,IAAA,MAAM,SAAiD,MAAM,KAAA;AAC7D,IAAA,OAAO,MAAA,CAAO,QAAA,KAAa,CAAA,IAAK,MAAA,CAAO,MAAA,KAAW,CAAA;AAAA,EACpD,SAAS,KAAA,EAAO;AAEd,IAAA,IAAI,CAACC,wCAAA,CAAwB,KAAK,CAAA,EAAG;AACnC,MAAA,MAAM,KAAA;AAAA,IACR,CAAA,MAAO;AACL,MAAA,MAAA,CAAO,KAAA,CAAM,CAAA,0CAAA,EAA6C,KAAK,CAAA,CAAE,CAAA;AACjE,MAAA,OAAO,KAAA;AAAA,IACT;AAAA,EACF;AACF;;;;"}
+{"version":3,"file":"insertUnprocessedEntity.cjs.js","sources":["../../../../src/database/operations/refreshState/insertUnprocessedEntity.ts"],"sourcesContent":["/*\n * Copyright 2022 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { Entity, stringifyEntityRef } from '@backstage/catalog-model';\nimport { Knex } from 'knex';\nimport { DbRefreshStateRow } from '../../tables';\nimport { randomUUID as uuid } from 'node:crypto';\nimport {\n  LoggerService,\n  isDatabaseConflictError,\n} from '@backstage/backend-plugin-api';\n\n/**\n * Attempts to insert a new refresh state row for the given entity, returning\n * true if successful and false if there was a conflict.\n */\nexport async function insertUnprocessedEntity(options: {\n  tx: Knex | Knex.Transaction;\n  entity: Entity;\n  hash: string;\n  locationKey?: string;\n  logger: LoggerService;\n}): Promise<boolean> {\n  const { tx, entity, hash, logger, locationKey } = options;\n\n  const entityRef = stringifyEntityRef(entity);\n  const serializedEntity = JSON.stringify(entity);\n\n  try {\n    let query = tx<DbRefreshStateRow>('refresh_state').insert({\n      entity_id: uuid(),\n      entity_ref: entityRef,\n      unprocessed_entity: serializedEntity,\n      unprocessed_hash: hash,\n      errors: '',\n      location_key: locationKey,\n      next_update_at: tx.fn.now(),\n      last_discovery_at: tx.fn.now(),\n    });\n\n    // TODO(Rugvip): only tested towards MySQL, Postgres and SQLite.\n    // We have to do this because the only way to detect if there was a conflict with\n    // SQLite is to catch the error, while Postgres needs to ignore the conflict to not\n    // break the ongoing transaction.\n    if (tx.client.config.client.includes('pg')) {\n      query = query.onConflict('entity_ref').ignore() as any; // type here does not match runtime\n    }\n\n    // Postgres gives as an object with rowCount, SQLite gives us an array\n    const result: { rowCount?: number; length?: number } = await query;\n    return result.rowCount === 1 || result.length === 1;\n  } catch (error) {\n    // SQLite, or MySQL reached this rather than the rowCount check above\n    if (!isDatabaseConflictError(error)) {\n      throw error;\n    } else {\n      logger.debug(`Unable to insert a new refresh state row, ${error}`);\n      return false;\n    }\n  }\n}\n"],"names":["stringifyEntityRef","uuid","isDatabaseConflictError"],"mappings":";;;;;;AA6BA,eAAsB,wBAAwB,OAAA,EAMzB;AACnB,EAAA,MAAM,EAAE,EAAA,EAAI,MAAA,EAAQ,IAAA,EAAM,MAAA,EAAQ,aAAY,GAAI,OAAA;AAElD,EAAA,MAAM,SAAA,GAAYA,gCAAmB,MAAM,CAAA;AAC3C,EAAA,MAAM,gBAAA,GAAmB,IAAA,CAAK,SAAA,CAAU,MAAM,CAAA;AAE9C,EAAA,IAAI;AACF,IAAA,IAAI,KAAA,GAAQ,EAAA,CAAsB,eAAe,CAAA,CAAE,MAAA,CAAO;AAAA,MACxD,WAAWC,sBAAA,EAAK;AAAA,MAChB,UAAA,EAAY,SAAA;AAAA,MACZ,kBAAA,EAAoB,gBAAA;AAAA,MACpB,gBAAA,EAAkB,IAAA;AAAA,MAClB,MAAA,EAAQ,EAAA;AAAA,MACR,YAAA,EAAc,WAAA;AAAA,MACd,cAAA,EAAgB,EAAA,CAAG,EAAA,CAAG,GAAA,EAAI;AAAA,MAC1B,iBAAA,EAAmB,EAAA,CAAG,EAAA,CAAG,GAAA;AAAI,KAC9B,CAAA;AAMD,IAAA,IAAI,GAAG,MAAA,CAAO,MAAA,CAAO,MAAA,CAAO,QAAA,CAAS,IAAI,CAAA,EAAG;AAC1C,MAAA,KAAA,GAAQ,KAAA,CAAM,UAAA,CAAW,YAAY,CAAA,CAAE,MAAA,EAAO;AAAA,IAChD;AAGA,IAAA,MAAM,SAAiD,MAAM,KAAA;AAC7D,IAAA,OAAO,MAAA,CAAO,QAAA,KAAa,CAAA,IAAK,MAAA,CAAO,MAAA,KAAW,CAAA;AAAA,EACpD,SAAS,KAAA,EAAO;AAEd,IAAA,IAAI,CAACC,wCAAA,CAAwB,KAAK,CAAA,EAAG;AACnC,MAAA,MAAM,KAAA;AAAA,IACR,CAAA,MAAO;AACL,MAAA,MAAA,CAAO,KAAA,CAAM,CAAA,0CAAA,EAA6C,KAAK,CAAA,CAAE,CAAA;AACjE,MAAA,OAAO,KAAA;AAAA,IACT;AAAA,EACF;AACF;;;;"}

package/dist/database/operations/stitcher/buildEntitySearch.cjs.js

@@ -2,6 +2,7 @@
 
 var catalogModel = require('@backstage/catalog-model');
 var errors = require('@backstage/errors');
+var util = require('./util.cjs.js');
 
 const SPECIAL_KEYS = [
   "attachments",
@@ -113,7 +114,14 @@ function buildEntitySearch(entityId, entity) {
       `Entity has duplicate keys that vary only in casing, ${badKeys}`
     );
   }
-  return mapToRows(raw, entityId);
+  const rows = mapToRows(raw, entityId);
+  const seen = /* @__PURE__ */ new Set();
+  return rows.filter((row) => {
+    const k = `${row.key}\0${row.value === null ? util.NULL_SENTINEL : row.value}`;
+    if (seen.has(k)) return false;
+    seen.add(k);
+    return true;
+  });
 }
 
 exports.buildEntitySearch = buildEntitySearch;

package/dist/database/operations/stitcher/buildEntitySearch.cjs.js.map

@@ -1 +1 @@
-{"version":3,"file":"buildEntitySearch.cjs.js","sources":["../../../../src/database/operations/stitcher/buildEntitySearch.ts"],"sourcesContent":["/*\n * Copyright 2020 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { DEFAULT_NAMESPACE, Entity } from '@backstage/catalog-model';\nimport { InputError } from '@backstage/errors';\nimport { DbSearchRow } from '../../tables';\n\n// These are excluded in the generic loop, either because they do not make sense\n// to index, or because they are special-case always inserted whether they are\n// null or not\nconst SPECIAL_KEYS = [\n  'attachments',\n  'relations',\n  'status',\n  'metadata.name',\n  'metadata.namespace',\n  'metadata.uid',\n  'metadata.etag',\n];\n\n// The maximum length allowed for search values. These columns are indexed, and\n// database engines do not like to index on massive values. For example,\n// postgres will balk after 8191 byte line sizes.\nconst MAX_KEY_LENGTH = 200;\nconst MAX_VALUE_LENGTH = 200;\n\ntype Kv = {\n  key: string;\n  value: unknown;\n};\n\n// Helper for traversing through a nested structure and outputting a list of\n// path->value entries of the leaves.\n//\n// For example, this yaml structure\n//\n// a: 1\n// b:\n//   c: null\n//   e: [f, g]\n// h:\n//  - i: 1\n//    j: k\n//  - i: 2\n//    j: l\n//\n// will result in\n//\n// \"a\", 1\n// \"b.c\", null\n// \"b.e\": \"f\"\n// \"b.e.f\": true\n// \"b.e\": \"g\"\n// \"b.e.g\": true\n// \"h.i\": 1\n// \"h.j\": \"k\"\n// \"h.i\": 2\n// \"h.j\": \"l\"\nexport function traverse(root: unknown): Kv[] {\n  const output: Kv[] = [];\n  // Use a Set for O(1) case-insensitive duplicate detection of synthetic\n  // boolean path keys (e.g. \"metadata.tags.java\"), instead of the previous\n  // O(n) Array.some() linear scan which caused O(n²) overall complexity\n  // and severe event loop blocking for entities with large arrays.\n  const seenPathKeys = new Set<string>();\n\n  function visit(path: string, current: unknown) {\n    if (SPECIAL_KEYS.includes(path)) {\n      return;\n    }\n\n    // empty or scalar\n    if (\n      current === undefined ||\n      current === null ||\n      ['string', 'number', 'boolean'].includes(typeof current)\n    ) {\n      output.push({ key: path, value: current });\n      return;\n    }\n\n    // unknown\n    if (typeof current !== 'object') {\n      return;\n    }\n\n    // array\n    if (Array.isArray(current)) {\n      for (const item of current) {\n        // NOTE(freben): The reason that these are output in two different ways,\n        // is to support use cases where you want to express that MORE than one\n        // tag is present in a list. Since the EntityFilters structure is a\n        // record, you can't have several entries of the same key. Therefore\n        // you will have to match on\n        //\n        // { \"a.b\": [\"true\"], \"a.c\": [\"true\"] }\n        //\n        // rather than\n        //\n        // { \"a\": [\"b\", \"c\"] }\n        //\n        // because the latter means EITHER b or c has to be present.\n        visit(path, item);\n        if (typeof item === 'string') {\n          const pathKey = `${path}.${item}`;\n          const lowerKey = pathKey.toLocaleLowerCase('en-US');\n          if (!seenPathKeys.has(lowerKey)) {\n            seenPathKeys.add(lowerKey);\n            output.push({ key: pathKey, value: true });\n          }\n        }\n      }\n      return;\n    }\n\n    // object\n    for (const [key, value] of Object.entries(current!)) {\n      visit(path ? `${path}.${key}` : key, value);\n    }\n  }\n\n  visit('', root);\n\n  return output;\n}\n\n// Translates a number of raw data rows to search table rows\nexport function mapToRows(input: Kv[], entityId: string): DbSearchRow[] {\n  const result: DbSearchRow[] = [];\n\n  for (const { key: rawKey, value: rawValue } of input) {\n    const key = rawKey.toLocaleLowerCase('en-US');\n    if (key.length > MAX_KEY_LENGTH) {\n      continue;\n    }\n    if (rawValue === undefined || rawValue === null) {\n      result.push({\n        entity_id: entityId,\n        key,\n        original_value: null,\n        value: null,\n      });\n    } else {\n      const value = String(rawValue).toLocaleLowerCase('en-US');\n      if (value.length <= MAX_VALUE_LENGTH) {\n        result.push({\n          entity_id: entityId,\n          key,\n          original_value: String(rawValue),\n          value: value,\n        });\n      } else {\n        result.push({\n          entity_id: entityId,\n          key,\n          original_value: null,\n          value: null,\n        });\n      }\n    }\n  }\n\n  return result;\n}\n\n/**\n * Generates all of the search rows that are relevant for this entity.\n *\n * @param entityId - The uid of the entity\n * @param entity - The entity\n * @returns A list of entity search rows\n */\nexport function buildEntitySearch(\n  entityId: string,\n  entity: Entity,\n): DbSearchRow[] {\n  // Visit the base structure recursively\n  const raw = traverse(entity);\n\n  // Start with some special keys that are always present because you want to\n  // be able to easily search for null specifically\n  raw.push({ key: 'metadata.name', value: entity.metadata.name });\n  raw.push({ key: 'metadata.namespace', value: entity.metadata.namespace });\n  raw.push({ key: 'metadata.uid', value: entity.metadata.uid });\n\n  // Namespace not specified has the default value \"default\", so we want to\n  // match on that as well\n  if (!entity.metadata.namespace) {\n    raw.push({ key: 'metadata.namespace', value: DEFAULT_NAMESPACE });\n  }\n\n  // Visit relations\n  for (const relation of entity.relations ?? []) {\n    raw.push({\n      key: `relations.${relation.type}`,\n      value: relation.targetRef,\n    });\n  }\n\n  // This validates that there are no keys that vary only in casing, such\n  // as `spec.foo` and `spec.Foo`.\n  const keys = new Set(raw.map(r => r.key));\n  const lowerKeys = new Set(raw.map(r => r.key.toLocaleLowerCase('en-US')));\n  if (keys.size !== lowerKeys.size) {\n    const difference = [];\n    for (const key of keys) {\n      const lower = key.toLocaleLowerCase('en-US');\n      if (!lowerKeys.delete(lower)) {\n        difference.push(lower);\n      }\n    }\n    const badKeys = `'${difference.join(\"', '\")}'`;\n    throw new InputError(\n      `Entity has duplicate keys that vary only in casing, ${badKeys}`,\n    );\n  }\n\n  return mapToRows(raw, entityId);\n}\n"],"names":["DEFAULT_NAMESPACE","InputError"],"mappings":";;;;;AAuBA,MAAM,YAAA,GAAe;AAAA,EACnB,aAAA;AAAA,EACA,WAAA;AAAA,EACA,QAAA;AAAA,EACA,eAAA;AAAA,EACA,oBAAA;AAAA,EACA,cAAA;AAAA,EACA;AACF,CAAA;AAKA,MAAM,cAAA,GAAiB,GAAA;AACvB,MAAM,gBAAA,GAAmB,GAAA;AAkClB,SAAS,SAAS,IAAA,EAAqB;AAC5C,EAAA,MAAM,SAAe,EAAC;AAKtB,EAAA,MAAM,YAAA,uBAAmB,GAAA,EAAY;AAErC,EAAA,SAAS,KAAA,CAAM,MAAc,OAAA,EAAkB;AAC7C,IAAA,IAAI,YAAA,CAAa,QAAA,CAAS,IAAI,CAAA,EAAG;AAC/B,MAAA;AAAA,IACF;AAGA,IAAA,IACE,OAAA,KAAY,MAAA,IACZ,OAAA,KAAY,IAAA,IACZ,CAAC,QAAA,EAAU,QAAA,EAAU,SAAS,CAAA,CAAE,QAAA,CAAS,OAAO,OAAO,CAAA,EACvD;AACA,MAAA,MAAA,CAAO,KAAK,EAAE,GAAA,EAAK,IAAA,EAAM,KAAA,EAAO,SAAS,CAAA;AACzC,MAAA;AAAA,IACF;AAGA,IAAA,IAAI,OAAO,YAAY,QAAA,EAAU;AAC/B,MAAA;AAAA,IACF;AAGA,IAAA,IAAI,KAAA,CAAM,OAAA,CAAQ,OAAO,CAAA,EAAG;AAC1B,MAAA,KAAA,MAAW,QAAQ,OAAA,EAAS;AAc1B,QAAA,KAAA,CAAM,MAAM,IAAI,CAAA;AAChB,QAAA,IAAI,OAAO,SAAS,QAAA,EAAU;AAC5B,UAAA,MAAM,OAAA,GAAU,CAAA,EAAG,IAAI,CAAA,CAAA,EAAI,IAAI,CAAA,CAAA;AAC/B,UAAA,MAAM,QAAA,GAAW,OAAA,CAAQ,iBAAA,CAAkB,OAAO,CAAA;AAClD,UAAA,IAAI,CAAC,YAAA,CAAa,GAAA,CAAI,QAAQ,CAAA,EAAG;AAC/B,YAAA,YAAA,CAAa,IAAI,QAAQ,CAAA;AACzB,YAAA,MAAA,CAAO,KAAK,EAAE,GAAA,EAAK,OAAA,EAAS,KAAA,EAAO,MAAM,CAAA;AAAA,UAC3C;AAAA,QACF;AAAA,MACF;AACA,MAAA;AAAA,IACF;AAGA,IAAA,KAAA,MAAW,CAAC,GAAA,EAAK,KAAK,KAAK,MAAA,CAAO,OAAA,CAAQ,OAAQ,CAAA,EAAG;AACnD,MAAA,KAAA,CAAM,OAAO,CAAA,EAAG,IAAI,IAAI,GAAG,CAAA,CAAA,GAAK,KAAK,KAAK,CAAA;AAAA,IAC5C;AAAA,EACF;AAEA,EAAA,KAAA,CAAM,IAAI,IAAI,CAAA;AAEd,EAAA,OAAO,MAAA;AACT;AAGO,SAAS,SAAA,CAAU,OAAa,QAAA,EAAiC;AACtE,EAAA,MAAM,SAAwB,EAAC;AAE/B,EAAA,KAAA,MAAW,EAAE,GAAA,EAAK,MAAA,EAAQ,KAAA,EAAO,QAAA,MAAc,KAAA,EAAO;AACpD,IAAA,MAAM,GAAA,GAAM,MAAA,CAAO,iBAAA,CAAkB,OAAO,CAAA;AAC5C,IAAA,IAAI,GAAA,CAAI,SAAS,cAAA,EAAgB;AAC/B,MAAA;AAAA,IACF;AACA,IAAA,IAAI,QAAA,KAAa,MAAA,IAAa,QAAA,KAAa,IAAA,EAAM;AAC/C,MAAA,MAAA,CAAO,IAAA,CAAK;AAAA,QACV,SAAA,EAAW,QAAA;AAAA,QACX,GAAA;AAAA,QACA,cAAA,EAAgB,IAAA;AAAA,QAChB,KAAA,EAAO;AAAA,OACR,CAAA;AAAA,IACH,CAAA,MAAO;AACL,MAAA,MAAM,KAAA,GAAQ,MAAA,CAAO,QAAQ,CAAA,CAAE,kBAAkB,OAAO,CAAA;AACxD,MAAA,IAAI,KAAA,CAAM,UAAU,gBAAA,EAAkB;AACpC,QAAA,MAAA,CAAO,IAAA,CAAK;AAAA,UACV,SAAA,EAAW,QAAA;AAAA,UACX,GAAA;AAAA,UACA,cAAA,EAAgB,OAAO,QAAQ,CAAA;AAAA,UAC/B;AAAA,SACD,CAAA;AAAA,MACH,CAAA,MAAO;AACL,QAAA,MAAA,CAAO,IAAA,CAAK;AAAA,UACV,SAAA,EAAW,QAAA;AAAA,UACX,GAAA;AAAA,UACA,cAAA,EAAgB,IAAA;AAAA,UAChB,KAAA,EAAO;AAAA,SACR,CAAA;AAAA,MACH;AAAA,IACF;AAAA,EACF;AAEA,EAAA,OAAO,MAAA;AACT;AASO,SAAS,iBAAA,CACd,UACA,MAAA,EACe;AAEf,EAAA,MAAM,GAAA,GAAM,SAAS,MAAM,CAAA;AAI3B,EAAA,GAAA,CAAI,IAAA,CAAK,EAAE,GAAA,EAAK,eAAA,EAAiB,OAAO,MAAA,CAAO,QAAA,CAAS,MAAM,CAAA;AAC9D,EAAA,GAAA,CAAI,IAAA,CAAK,EAAE,GAAA,EAAK,oBAAA,EAAsB,OAAO,MAAA,CAAO,QAAA,CAAS,WAAW,CAAA;AACxE,EAAA,GAAA,CAAI,IAAA,CAAK,EAAE,GAAA,EAAK,cAAA,EAAgB,OAAO,MAAA,CAAO,QAAA,CAAS,KAAK,CAAA;AAI5D,EAAA,IAAI,CAAC,MAAA,CAAO,QAAA,CAAS,SAAA,EAAW;AAC9B,IAAA,GAAA,CAAI,KAAK,EAAE,GAAA,EAAK,oBAAA,EAAsB,KAAA,EAAOA,gCAAmB,CAAA;AAAA,EAClE;AAGA,EAAA,KAAA,MAAW,QAAA,IAAY,MAAA,CAAO,SAAA,IAAa,EAAC,EAAG;AAC7C,IAAA,GAAA,CAAI,IAAA,CAAK;AAAA,MACP,GAAA,EAAK,CAAA,UAAA,EAAa,QAAA,CAAS,IAAI,CAAA,CAAA;AAAA,MAC/B,OAAO,QAAA,CAAS;AAAA,KACjB,CAAA;AAAA,EACH;AAIA,EAAA,MAAM,IAAA,GAAO,IAAI,GAAA,CAAI,GAAA,CAAI,IAAI,CAAA,CAAA,KAAK,CAAA,CAAE,GAAG,CAAC,CAAA;AACxC,EAAA,MAAM,SAAA,GAAY,IAAI,GAAA,CAAI,GAAA,CAAI,GAAA,CAAI,CAAA,CAAA,KAAK,CAAA,CAAE,GAAA,CAAI,iBAAA,CAAkB,OAAO,CAAC,CAAC,CAAA;AACxE,EAAA,IAAI,IAAA,CAAK,IAAA,KAAS,SAAA,CAAU,IAAA,EAAM;AAChC,IAAA,MAAM,aAAa,EAAC;AACpB,IAAA,KAAA,MAAW,OAAO,IAAA,EAAM;AACtB,MAAA,MAAM,KAAA,GAAQ,GAAA,CAAI,iBAAA,CAAkB,OAAO,CAAA;AAC3C,MAAA,IAAI,CAAC,SAAA,CAAU,MAAA,CAAO,KAAK,CAAA,EAAG;AAC5B,QAAA,UAAA,CAAW,KAAK,KAAK,CAAA;AAAA,MACvB;AAAA,IACF;AACA,IAAA,MAAM,OAAA,GAAU,CAAA,CAAA,EAAI,UAAA,CAAW,IAAA,CAAK,MAAM,CAAC,CAAA,CAAA,CAAA;AAC3C,IAAA,MAAM,IAAIC,iBAAA;AAAA,MACR,uDAAuD,OAAO,CAAA;AAAA,KAChE;AAAA,EACF;AAEA,EAAA,OAAO,SAAA,CAAU,KAAK,QAAQ,CAAA;AAChC;;;;;;"}
+{"version":3,"file":"buildEntitySearch.cjs.js","sources":["../../../../src/database/operations/stitcher/buildEntitySearch.ts"],"sourcesContent":["/*\n * Copyright 2020 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { DEFAULT_NAMESPACE, Entity } from '@backstage/catalog-model';\nimport { InputError } from '@backstage/errors';\nimport { DbSearchRow } from '../../tables';\nimport { NULL_SENTINEL } from './util';\n\n// These are excluded in the generic loop, either because they do not make sense\n// to index, or because they are special-case always inserted whether they are\n// null or not\nconst SPECIAL_KEYS = [\n  'attachments',\n  'relations',\n  'status',\n  'metadata.name',\n  'metadata.namespace',\n  'metadata.uid',\n  'metadata.etag',\n];\n\n// The maximum length allowed for search values. These columns are indexed, and\n// database engines do not like to index on massive values. For example,\n// postgres will balk after 8191 byte line sizes.\nconst MAX_KEY_LENGTH = 200;\nconst MAX_VALUE_LENGTH = 200;\n\ntype Kv = {\n  key: string;\n  value: unknown;\n};\n\n// Helper for traversing through a nested structure and outputting a list of\n// path->value entries of the leaves.\n//\n// For example, this yaml structure\n//\n// a: 1\n// b:\n//   c: null\n//   e: [f, g]\n// h:\n//  - i: 1\n//    j: k\n//  - i: 2\n//    j: l\n//\n// will result in\n//\n// \"a\", 1\n// \"b.c\", null\n// \"b.e\": \"f\"\n// \"b.e.f\": true\n// \"b.e\": \"g\"\n// \"b.e.g\": true\n// \"h.i\": 1\n// \"h.j\": \"k\"\n// \"h.i\": 2\n// \"h.j\": \"l\"\nexport function traverse(root: unknown): Kv[] {\n  const output: Kv[] = [];\n  // Use a Set for O(1) case-insensitive duplicate detection of synthetic\n  // boolean path keys (e.g. \"metadata.tags.java\"), instead of the previous\n  // O(n) Array.some() linear scan which caused O(n²) overall complexity\n  // and severe event loop blocking for entities with large arrays.\n  const seenPathKeys = new Set<string>();\n\n  function visit(path: string, current: unknown) {\n    if (SPECIAL_KEYS.includes(path)) {\n      return;\n    }\n\n    // empty or scalar\n    if (\n      current === undefined ||\n      current === null ||\n      ['string', 'number', 'boolean'].includes(typeof current)\n    ) {\n      output.push({ key: path, value: current });\n      return;\n    }\n\n    // unknown\n    if (typeof current !== 'object') {\n      return;\n    }\n\n    // array\n    if (Array.isArray(current)) {\n      for (const item of current) {\n        // NOTE(freben): The reason that these are output in two different ways,\n        // is to support use cases where you want to express that MORE than one\n        // tag is present in a list. Since the EntityFilters structure is a\n        // record, you can't have several entries of the same key. Therefore\n        // you will have to match on\n        //\n        // { \"a.b\": [\"true\"], \"a.c\": [\"true\"] }\n        //\n        // rather than\n        //\n        // { \"a\": [\"b\", \"c\"] }\n        //\n        // because the latter means EITHER b or c has to be present.\n        visit(path, item);\n        if (typeof item === 'string') {\n          const pathKey = `${path}.${item}`;\n          const lowerKey = pathKey.toLocaleLowerCase('en-US');\n          if (!seenPathKeys.has(lowerKey)) {\n            seenPathKeys.add(lowerKey);\n            output.push({ key: pathKey, value: true });\n          }\n        }\n      }\n      return;\n    }\n\n    // object\n    for (const [key, value] of Object.entries(current!)) {\n      visit(path ? `${path}.${key}` : key, value);\n    }\n  }\n\n  visit('', root);\n\n  return output;\n}\n\n// Translates a number of raw data rows to search table rows\nexport function mapToRows(input: Kv[], entityId: string): DbSearchRow[] {\n  const result: DbSearchRow[] = [];\n\n  for (const { key: rawKey, value: rawValue } of input) {\n    const key = rawKey.toLocaleLowerCase('en-US');\n    if (key.length > MAX_KEY_LENGTH) {\n      continue;\n    }\n    if (rawValue === undefined || rawValue === null) {\n      result.push({\n        entity_id: entityId,\n        key,\n        original_value: null,\n        value: null,\n      });\n    } else {\n      const value = String(rawValue).toLocaleLowerCase('en-US');\n      if (value.length <= MAX_VALUE_LENGTH) {\n        result.push({\n          entity_id: entityId,\n          key,\n          original_value: String(rawValue),\n          value: value,\n        });\n      } else {\n        result.push({\n          entity_id: entityId,\n          key,\n          original_value: null,\n          value: null,\n        });\n      }\n    }\n  }\n\n  return result;\n}\n\n/**\n * Generates all of the search rows that are relevant for this entity.\n *\n * @param entityId - The uid of the entity\n * @param entity - The entity\n * @returns A list of entity search rows\n */\nexport function buildEntitySearch(\n  entityId: string,\n  entity: Entity,\n): DbSearchRow[] {\n  // Visit the base structure recursively\n  const raw = traverse(entity);\n\n  // Start with some special keys that are always present because you want to\n  // be able to easily search for null specifically\n  raw.push({ key: 'metadata.name', value: entity.metadata.name });\n  raw.push({ key: 'metadata.namespace', value: entity.metadata.namespace });\n  raw.push({ key: 'metadata.uid', value: entity.metadata.uid });\n\n  // Namespace not specified has the default value \"default\", so we want to\n  // match on that as well\n  if (!entity.metadata.namespace) {\n    raw.push({ key: 'metadata.namespace', value: DEFAULT_NAMESPACE });\n  }\n\n  // Visit relations\n  for (const relation of entity.relations ?? []) {\n    raw.push({\n      key: `relations.${relation.type}`,\n      value: relation.targetRef,\n    });\n  }\n\n  // This validates that there are no keys that vary only in casing, such\n  // as `spec.foo` and `spec.Foo`.\n  const keys = new Set(raw.map(r => r.key));\n  const lowerKeys = new Set(raw.map(r => r.key.toLocaleLowerCase('en-US')));\n  if (keys.size !== lowerKeys.size) {\n    const difference = [];\n    for (const key of keys) {\n      const lower = key.toLocaleLowerCase('en-US');\n      if (!lowerKeys.delete(lower)) {\n        difference.push(lower);\n      }\n    }\n    const badKeys = `'${difference.join(\"', '\")}'`;\n    throw new InputError(\n      `Entity has duplicate keys that vary only in casing, ${badKeys}`,\n    );\n  }\n\n  const rows = mapToRows(raw, entityId);\n\n  // Deduplicate by (key, value). Duplicate array values in the entity data\n  // (e.g. tags: ['java', 'java']) produce identical search rows which would\n  // violate the unique constraint on (entity_id, key, value).\n  const seen = new Set<string>();\n  return rows.filter(row => {\n    const k = `${row.key}\\0${row.value === null ? NULL_SENTINEL : row.value}`;\n    if (seen.has(k)) return false;\n    seen.add(k);\n    return true;\n  });\n}\n"],"names":["DEFAULT_NAMESPACE","InputError","NULL_SENTINEL"],"mappings":";;;;;;AAwBA,MAAM,YAAA,GAAe;AAAA,EACnB,aAAA;AAAA,EACA,WAAA;AAAA,EACA,QAAA;AAAA,EACA,eAAA;AAAA,EACA,oBAAA;AAAA,EACA,cAAA;AAAA,EACA;AACF,CAAA;AAKA,MAAM,cAAA,GAAiB,GAAA;AACvB,MAAM,gBAAA,GAAmB,GAAA;AAkClB,SAAS,SAAS,IAAA,EAAqB;AAC5C,EAAA,MAAM,SAAe,EAAC;AAKtB,EAAA,MAAM,YAAA,uBAAmB,GAAA,EAAY;AAErC,EAAA,SAAS,KAAA,CAAM,MAAc,OAAA,EAAkB;AAC7C,IAAA,IAAI,YAAA,CAAa,QAAA,CAAS,IAAI,CAAA,EAAG;AAC/B,MAAA;AAAA,IACF;AAGA,IAAA,IACE,OAAA,KAAY,MAAA,IACZ,OAAA,KAAY,IAAA,IACZ,CAAC,QAAA,EAAU,QAAA,EAAU,SAAS,CAAA,CAAE,QAAA,CAAS,OAAO,OAAO,CAAA,EACvD;AACA,MAAA,MAAA,CAAO,KAAK,EAAE,GAAA,EAAK,IAAA,EAAM,KAAA,EAAO,SAAS,CAAA;AACzC,MAAA;AAAA,IACF;AAGA,IAAA,IAAI,OAAO,YAAY,QAAA,EAAU;AAC/B,MAAA;AAAA,IACF;AAGA,IAAA,IAAI,KAAA,CAAM,OAAA,CAAQ,OAAO,CAAA,EAAG;AAC1B,MAAA,KAAA,MAAW,QAAQ,OAAA,EAAS;AAc1B,QAAA,KAAA,CAAM,MAAM,IAAI,CAAA;AAChB,QAAA,IAAI,OAAO,SAAS,QAAA,EAAU;AAC5B,UAAA,MAAM,OAAA,GAAU,CAAA,EAAG,IAAI,CAAA,CAAA,EAAI,IAAI,CAAA,CAAA;AAC/B,UAAA,MAAM,QAAA,GAAW,OAAA,CAAQ,iBAAA,CAAkB,OAAO,CAAA;AAClD,UAAA,IAAI,CAAC,YAAA,CAAa,GAAA,CAAI,QAAQ,CAAA,EAAG;AAC/B,YAAA,YAAA,CAAa,IAAI,QAAQ,CAAA;AACzB,YAAA,MAAA,CAAO,KAAK,EAAE,GAAA,EAAK,OAAA,EAAS,KAAA,EAAO,MAAM,CAAA;AAAA,UAC3C;AAAA,QACF;AAAA,MACF;AACA,MAAA;AAAA,IACF;AAGA,IAAA,KAAA,MAAW,CAAC,GAAA,EAAK,KAAK,KAAK,MAAA,CAAO,OAAA,CAAQ,OAAQ,CAAA,EAAG;AACnD,MAAA,KAAA,CAAM,OAAO,CAAA,EAAG,IAAI,IAAI,GAAG,CAAA,CAAA,GAAK,KAAK,KAAK,CAAA;AAAA,IAC5C;AAAA,EACF;AAEA,EAAA,KAAA,CAAM,IAAI,IAAI,CAAA;AAEd,EAAA,OAAO,MAAA;AACT;AAGO,SAAS,SAAA,CAAU,OAAa,QAAA,EAAiC;AACtE,EAAA,MAAM,SAAwB,EAAC;AAE/B,EAAA,KAAA,MAAW,EAAE,GAAA,EAAK,MAAA,EAAQ,KAAA,EAAO,QAAA,MAAc,KAAA,EAAO;AACpD,IAAA,MAAM,GAAA,GAAM,MAAA,CAAO,iBAAA,CAAkB,OAAO,CAAA;AAC5C,IAAA,IAAI,GAAA,CAAI,SAAS,cAAA,EAAgB;AAC/B,MAAA;AAAA,IACF;AACA,IAAA,IAAI,QAAA,KAAa,MAAA,IAAa,QAAA,KAAa,IAAA,EAAM;AAC/C,MAAA,MAAA,CAAO,IAAA,CAAK;AAAA,QACV,SAAA,EAAW,QAAA;AAAA,QACX,GAAA;AAAA,QACA,cAAA,EAAgB,IAAA;AAAA,QAChB,KAAA,EAAO;AAAA,OACR,CAAA;AAAA,IACH,CAAA,MAAO;AACL,MAAA,MAAM,KAAA,GAAQ,MAAA,CAAO,QAAQ,CAAA,CAAE,kBAAkB,OAAO,CAAA;AACxD,MAAA,IAAI,KAAA,CAAM,UAAU,gBAAA,EAAkB;AACpC,QAAA,MAAA,CAAO,IAAA,CAAK;AAAA,UACV,SAAA,EAAW,QAAA;AAAA,UACX,GAAA;AAAA,UACA,cAAA,EAAgB,OAAO,QAAQ,CAAA;AAAA,UAC/B;AAAA,SACD,CAAA;AAAA,MACH,CAAA,MAAO;AACL,QAAA,MAAA,CAAO,IAAA,CAAK;AAAA,UACV,SAAA,EAAW,QAAA;AAAA,UACX,GAAA;AAAA,UACA,cAAA,EAAgB,IAAA;AAAA,UAChB,KAAA,EAAO;AAAA,SACR,CAAA;AAAA,MACH;AAAA,IACF;AAAA,EACF;AAEA,EAAA,OAAO,MAAA;AACT;AASO,SAAS,iBAAA,CACd,UACA,MAAA,EACe;AAEf,EAAA,MAAM,GAAA,GAAM,SAAS,MAAM,CAAA;AAI3B,EAAA,GAAA,CAAI,IAAA,CAAK,EAAE,GAAA,EAAK,eAAA,EAAiB,OAAO,MAAA,CAAO,QAAA,CAAS,MAAM,CAAA;AAC9D,EAAA,GAAA,CAAI,IAAA,CAAK,EAAE,GAAA,EAAK,oBAAA,EAAsB,OAAO,MAAA,CAAO,QAAA,CAAS,WAAW,CAAA;AACxE,EAAA,GAAA,CAAI,IAAA,CAAK,EAAE,GAAA,EAAK,cAAA,EAAgB,OAAO,MAAA,CAAO,QAAA,CAAS,KAAK,CAAA;AAI5D,EAAA,IAAI,CAAC,MAAA,CAAO,QAAA,CAAS,SAAA,EAAW;AAC9B,IAAA,GAAA,CAAI,KAAK,EAAE,GAAA,EAAK,oBAAA,EAAsB,KAAA,EAAOA,gCAAmB,CAAA;AAAA,EAClE;AAGA,EAAA,KAAA,MAAW,QAAA,IAAY,MAAA,CAAO,SAAA,IAAa,EAAC,EAAG;AAC7C,IAAA,GAAA,CAAI,IAAA,CAAK;AAAA,MACP,GAAA,EAAK,CAAA,UAAA,EAAa,QAAA,CAAS,IAAI,CAAA,CAAA;AAAA,MAC/B,OAAO,QAAA,CAAS;AAAA,KACjB,CAAA;AAAA,EACH;AAIA,EAAA,MAAM,IAAA,GAAO,IAAI,GAAA,CAAI,GAAA,CAAI,IAAI,CAAA,CAAA,KAAK,CAAA,CAAE,GAAG,CAAC,CAAA;AACxC,EAAA,MAAM,SAAA,GAAY,IAAI,GAAA,CAAI,GAAA,CAAI,GAAA,CAAI,CAAA,CAAA,KAAK,CAAA,CAAE,GAAA,CAAI,iBAAA,CAAkB,OAAO,CAAC,CAAC,CAAA;AACxE,EAAA,IAAI,IAAA,CAAK,IAAA,KAAS,SAAA,CAAU,IAAA,EAAM;AAChC,IAAA,MAAM,aAAa,EAAC;AACpB,IAAA,KAAA,MAAW,OAAO,IAAA,EAAM;AACtB,MAAA,MAAM,KAAA,GAAQ,GAAA,CAAI,iBAAA,CAAkB,OAAO,CAAA;AAC3C,MAAA,IAAI,CAAC,SAAA,CAAU,MAAA,CAAO,KAAK,CAAA,EAAG;AAC5B,QAAA,UAAA,CAAW,KAAK,KAAK,CAAA;AAAA,MACvB;AAAA,IACF;AACA,IAAA,MAAM,OAAA,GAAU,CAAA,CAAA,EAAI,UAAA,CAAW,IAAA,CAAK,MAAM,CAAC,CAAA,CAAA,CAAA;AAC3C,IAAA,MAAM,IAAIC,iBAAA;AAAA,MACR,uDAAuD,OAAO,CAAA;AAAA,KAChE;AAAA,EACF;AAEA,EAAA,MAAM,IAAA,GAAO,SAAA,CAAU,GAAA,EAAK,QAAQ,CAAA;AAKpC,EAAA,MAAM,IAAA,uBAAW,GAAA,EAAY;AAC7B,EAAA,OAAO,IAAA,CAAK,OAAO,CAAA,GAAA,KAAO;AACxB,IAAA,MAAM,CAAA,GAAI,CAAA,EAAG,GAAA,CAAI,GAAG,CAAA,EAAA,EAAK,IAAI,KAAA,KAAU,IAAA,GAAOC,kBAAA,GAAgB,GAAA,CAAI,KAAK,CAAA,CAAA;AACvE,IAAA,IAAI,IAAA,CAAK,GAAA,CAAI,CAAC,CAAA,EAAG,OAAO,KAAA;AACxB,IAAA,IAAA,CAAK,IAAI,CAAC,CAAA;AACV,IAAA,OAAO,IAAA;AAAA,EACT,CAAC,CAAA;AACH;;;;;;"}

package/dist/database/operations/stitcher/markForStitching.cjs.js

@@ -1,7 +1,7 @@
 'use strict';
 
 var splitToChunks = require('lodash/chunk');
-var uuid = require('uuid');
+var node_crypto = require('node:crypto');
 var util = require('../../util.cjs.js');
 
 function _interopDefaultCompat (e) { return e && typeof e === 'object' && 'default' in e ? e : { default: e }; }
@@ -38,7 +38,7 @@ async function markForStitching(options) {
       }, knex);
     }
   } else if (mode === "deferred") {
-    const ticket = uuid.v4();
+    const ticket = node_crypto.randomUUID();
     for (const chunk of entityRefs) {
       await util.retryOnDeadlock(async () => {
         if (chunk.length > 0) {

package/dist/database/operations/stitcher/markForStitching.cjs.js.map

@@ -1 +1 @@
-{"version":3,"file":"markForStitching.cjs.js","sources":["../../../../src/database/operations/stitcher/markForStitching.ts"],"sourcesContent":["/*\n * Copyright 2023 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { Knex } from 'knex';\nimport splitToChunks from 'lodash/chunk';\nimport { v4 as uuid } from 'uuid';\nimport { StitchingStrategy } from '../../../stitching/types';\nimport {\n  DbFinalEntitiesRow,\n  DbRefreshStateRow,\n  DbStitchQueueRow,\n} from '../../tables';\nimport { retryOnDeadlock } from '../../util';\n\nconst UPDATE_CHUNK_SIZE = 100; // Smaller chunks reduce contention\n\n/**\n * Marks a number of entities for stitching some time in the near\n * future.\n *\n * @remarks\n */\nexport async function markForStitching(options: {\n  knex: Knex | Knex.Transaction;\n  strategy: StitchingStrategy;\n  entityRefs?: Iterable<string>;\n  entityIds?: Iterable<string>;\n}): Promise<void> {\n  const entityRefs = sortSplit(options.entityRefs);\n  const entityIds = sortSplit(options.entityIds);\n  const knex = options.knex;\n  const mode = options.strategy.mode;\n\n  if (mode === 'immediate') {\n    for (const chunk of entityRefs) {\n      await knex\n        .table<DbFinalEntitiesRow>('final_entities')\n        .update({\n          hash: 'force-stitching',\n        })\n        .whereIn('entity_ref', chunk);\n      await retryOnDeadlock(async () => {\n        await knex\n          .table<DbRefreshStateRow>('refresh_state')\n          .update({\n            result_hash: 'force-stitching',\n            next_update_at: knex.fn.now(),\n          })\n          .whereIn('entity_ref', chunk);\n      }, knex);\n    }\n\n    for (const chunk of entityIds) {\n      await knex\n        .table<DbFinalEntitiesRow>('final_entities')\n        .update({\n          hash: 'force-stitching',\n        })\n        .whereIn('entity_id', chunk);\n      await retryOnDeadlock(async () => {\n        await knex\n          .table<DbRefreshStateRow>('refresh_state')\n          .update({\n            result_hash: 'force-stitching',\n            next_update_at: knex.fn.now(),\n          })\n          .whereIn('entity_id', chunk);\n      }, knex);\n    }\n  } else if (mode === 'deferred') {\n    // It's OK that this is shared across stitch_queue rows; it just needs to\n    // be uniquely generated for every new stitch request.\n    const ticket = uuid();\n\n    for (const chunk of entityRefs) {\n      await retryOnDeadlock(async () => {\n        if (chunk.length > 0) {\n          await knex<DbStitchQueueRow>('stitch_queue')\n            .insert(\n              chunk.map(ref => ({\n                entity_ref: ref,\n                stitch_ticket: ticket,\n                next_stitch_at: knex.fn.now(),\n              })),\n            )\n            .onConflict('entity_ref')\n            .merge(['next_stitch_at', 'stitch_ticket']);\n        }\n      }, knex);\n    }\n\n    for (const chunk of entityIds) {\n      await retryOnDeadlock(async () => {\n        // Look up entity_refs from refresh_state by entity_id\n        const refreshStateRows = await knex<DbRefreshStateRow>('refresh_state')\n          .select('entity_ref')\n          .whereIn('entity_id', chunk);\n\n        if (refreshStateRows.length > 0) {\n          await knex<DbStitchQueueRow>('stitch_queue')\n            .insert(\n              refreshStateRows.map(row => ({\n                entity_ref: row.entity_ref,\n                stitch_ticket: ticket,\n                next_stitch_at: knex.fn.now(),\n              })),\n            )\n            .onConflict('entity_ref')\n            .merge(['next_stitch_at', 'stitch_ticket']);\n        }\n      }, knex);\n    }\n  } else {\n    throw new Error(`Unknown stitching strategy mode ${mode}`);\n  }\n}\n\nfunction sortSplit(input: Iterable<string> | undefined): string[][] {\n  if (!input) {\n    return [];\n  }\n  const array = Array.isArray(input) ? input.slice() : [...input];\n  array.sort();\n  return splitToChunks(array, UPDATE_CHUNK_SIZE);\n}\n"],"names":["retryOnDeadlock","uuid","splitToChunks"],"mappings":";;;;;;;;;;AA2BA,MAAM,iBAAA,GAAoB,GAAA;AAQ1B,eAAsB,iBAAiB,OAAA,EAKrB;AAChB,EAAA,MAAM,UAAA,GAAa,SAAA,CAAU,OAAA,CAAQ,UAAU,CAAA;AAC/C,EAAA,MAAM,SAAA,GAAY,SAAA,CAAU,OAAA,CAAQ,SAAS,CAAA;AAC7C,EAAA,MAAM,OAAO,OAAA,CAAQ,IAAA;AACrB,EAAA,MAAM,IAAA,GAAO,QAAQ,QAAA,CAAS,IAAA;AAE9B,EAAA,IAAI,SAAS,WAAA,EAAa;AACxB,IAAA,KAAA,MAAW,SAAS,UAAA,EAAY;AAC9B,MAAA,MAAM,IAAA,CACH,KAAA,CAA0B,gBAAgB,CAAA,CAC1C,MAAA,CAAO;AAAA,QACN,IAAA,EAAM;AAAA,OACP,CAAA,CACA,OAAA,CAAQ,YAAA,EAAc,KAAK,CAAA;AAC9B,MAAA,MAAMA,qBAAgB,YAAY;AAChC,QAAA,MAAM,IAAA,CACH,KAAA,CAAyB,eAAe,CAAA,CACxC,MAAA,CAAO;AAAA,UACN,WAAA,EAAa,iBAAA;AAAA,UACb,cAAA,EAAgB,IAAA,CAAK,EAAA,CAAG,GAAA;AAAI,SAC7B,CAAA,CACA,OAAA,CAAQ,YAAA,EAAc,KAAK,CAAA;AAAA,MAChC,GAAG,IAAI,CAAA;AAAA,IACT;AAEA,IAAA,KAAA,MAAW,SAAS,SAAA,EAAW;AAC7B,MAAA,MAAM,IAAA,CACH,KAAA,CAA0B,gBAAgB,CAAA,CAC1C,MAAA,CAAO;AAAA,QACN,IAAA,EAAM;AAAA,OACP,CAAA,CACA,OAAA,CAAQ,WAAA,EAAa,KAAK,CAAA;AAC7B,MAAA,MAAMA,qBAAgB,YAAY;AAChC,QAAA,MAAM,IAAA,CACH,KAAA,CAAyB,eAAe,CAAA,CACxC,MAAA,CAAO;AAAA,UACN,WAAA,EAAa,iBAAA;AAAA,UACb,cAAA,EAAgB,IAAA,CAAK,EAAA,CAAG,GAAA;AAAI,SAC7B,CAAA,CACA,OAAA,CAAQ,WAAA,EAAa,KAAK,CAAA;AAAA,MAC/B,GAAG,IAAI,CAAA;AAAA,IACT;AAAA,EACF,CAAA,MAAA,IAAW,SAAS,UAAA,EAAY;AAG9B,IAAA,MAAM,SAASC,OAAA,EAAK;AAEpB,IAAA,KAAA,MAAW,SAAS,UAAA,EAAY;AAC9B,MAAA,MAAMD,qBAAgB,YAAY;AAChC,QAAA,IAAI,KAAA,CAAM,SAAS,CAAA,EAAG;AACpB,UAAA,MAAM,IAAA,CAAuB,cAAc,CAAA,CACxC,MAAA;AAAA,YACC,KAAA,CAAM,IAAI,CAAA,GAAA,MAAQ;AAAA,cAChB,UAAA,EAAY,GAAA;AAAA,cACZ,aAAA,EAAe,MAAA;AAAA,cACf,cAAA,EAAgB,IAAA,CAAK,EAAA,CAAG,GAAA;AAAI,aAC9B,CAAE;AAAA,WACJ,CACC,WAAW,YAAY,CAAA,CACvB,MAAM,CAAC,gBAAA,EAAkB,eAAe,CAAC,CAAA;AAAA,QAC9C;AAAA,MACF,GAAG,IAAI,CAAA;AAAA,IACT;AAEA,IAAA,KAAA,MAAW,SAAS,SAAA,EAAW;AAC7B,MAAA,MAAMA,qBAAgB,YAAY;AAEhC,QAAA,MAAM,gBAAA,GAAmB,MAAM,IAAA,CAAwB,eAAe,CAAA,CACnE,OAAO,YAAY,CAAA,CACnB,OAAA,CAAQ,WAAA,EAAa,KAAK,CAAA;AAE7B,QAAA,IAAI,gBAAA,CAAiB,SAAS,CAAA,EAAG;AAC/B,UAAA,MAAM,IAAA,CAAuB,cAAc,CAAA,CACxC,MAAA;AAAA,YACC,gBAAA,CAAiB,IAAI,CAAA,GAAA,MAAQ;AAAA,cAC3B,YAAY,GAAA,CAAI,UAAA;AAAA,cAChB,aAAA,EAAe,MAAA;AAAA,cACf,cAAA,EAAgB,IAAA,CAAK,EAAA,CAAG,GAAA;AAAI,aAC9B,CAAE;AAAA,WACJ,CACC,WAAW,YAAY,CAAA,CACvB,MAAM,CAAC,gBAAA,EAAkB,eAAe,CAAC,CAAA;AAAA,QAC9C;AAAA,MACF,GAAG,IAAI,CAAA;AAAA,IACT;AAAA,EACF,CAAA,MAAO;AACL,IAAA,MAAM,IAAI,KAAA,CAAM,CAAA,gCAAA,EAAmC,IAAI,CAAA,CAAE,CAAA;AAAA,EAC3D;AACF;AAEA,SAAS,UAAU,KAAA,EAAiD;AAClE,EAAA,IAAI,CAAC,KAAA,EAAO;AACV,IAAA,OAAO,EAAC;AAAA,EACV;AACA,EAAA,MAAM,KAAA,GAAQ,KAAA,CAAM,OAAA,CAAQ,KAAK,CAAA,GAAI,MAAM,KAAA,EAAM,GAAI,CAAC,GAAG,KAAK,CAAA;AAC9D,EAAA,KAAA,CAAM,IAAA,EAAK;AACX,EAAA,OAAOE,8BAAA,CAAc,OAAO,iBAAiB,CAAA;AAC/C;;;;"}
+{"version":3,"file":"markForStitching.cjs.js","sources":["../../../../src/database/operations/stitcher/markForStitching.ts"],"sourcesContent":["/*\n * Copyright 2023 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { Knex } from 'knex';\nimport splitToChunks from 'lodash/chunk';\nimport { randomUUID as uuid } from 'node:crypto';\nimport { StitchingStrategy } from '../../../stitching/types';\nimport {\n  DbFinalEntitiesRow,\n  DbRefreshStateRow,\n  DbStitchQueueRow,\n} from '../../tables';\nimport { retryOnDeadlock } from '../../util';\n\nconst UPDATE_CHUNK_SIZE = 100; // Smaller chunks reduce contention\n\n/**\n * Marks a number of entities for stitching some time in the near\n * future.\n *\n * @remarks\n */\nexport async function markForStitching(options: {\n  knex: Knex | Knex.Transaction;\n  strategy: StitchingStrategy;\n  entityRefs?: Iterable<string>;\n  entityIds?: Iterable<string>;\n}): Promise<void> {\n  const entityRefs = sortSplit(options.entityRefs);\n  const entityIds = sortSplit(options.entityIds);\n  const knex = options.knex;\n  const mode = options.strategy.mode;\n\n  if (mode === 'immediate') {\n    for (const chunk of entityRefs) {\n      await knex\n        .table<DbFinalEntitiesRow>('final_entities')\n        .update({\n          hash: 'force-stitching',\n        })\n        .whereIn('entity_ref', chunk);\n      await retryOnDeadlock(async () => {\n        await knex\n          .table<DbRefreshStateRow>('refresh_state')\n          .update({\n            result_hash: 'force-stitching',\n            next_update_at: knex.fn.now(),\n          })\n          .whereIn('entity_ref', chunk);\n      }, knex);\n    }\n\n    for (const chunk of entityIds) {\n      await knex\n        .table<DbFinalEntitiesRow>('final_entities')\n        .update({\n          hash: 'force-stitching',\n        })\n        .whereIn('entity_id', chunk);\n      await retryOnDeadlock(async () => {\n        await knex\n          .table<DbRefreshStateRow>('refresh_state')\n          .update({\n            result_hash: 'force-stitching',\n            next_update_at: knex.fn.now(),\n          })\n          .whereIn('entity_id', chunk);\n      }, knex);\n    }\n  } else if (mode === 'deferred') {\n    // It's OK that this is shared across stitch_queue rows; it just needs to\n    // be uniquely generated for every new stitch request.\n    const ticket = uuid();\n\n    for (const chunk of entityRefs) {\n      await retryOnDeadlock(async () => {\n        if (chunk.length > 0) {\n          await knex<DbStitchQueueRow>('stitch_queue')\n            .insert(\n              chunk.map(ref => ({\n                entity_ref: ref,\n                stitch_ticket: ticket,\n                next_stitch_at: knex.fn.now(),\n              })),\n            )\n            .onConflict('entity_ref')\n            .merge(['next_stitch_at', 'stitch_ticket']);\n        }\n      }, knex);\n    }\n\n    for (const chunk of entityIds) {\n      await retryOnDeadlock(async () => {\n        // Look up entity_refs from refresh_state by entity_id\n        const refreshStateRows = await knex<DbRefreshStateRow>('refresh_state')\n          .select('entity_ref')\n          .whereIn('entity_id', chunk);\n\n        if (refreshStateRows.length > 0) {\n          await knex<DbStitchQueueRow>('stitch_queue')\n            .insert(\n              refreshStateRows.map(row => ({\n                entity_ref: row.entity_ref,\n                stitch_ticket: ticket,\n                next_stitch_at: knex.fn.now(),\n              })),\n            )\n            .onConflict('entity_ref')\n            .merge(['next_stitch_at', 'stitch_ticket']);\n        }\n      }, knex);\n    }\n  } else {\n    throw new Error(`Unknown stitching strategy mode ${mode}`);\n  }\n}\n\nfunction sortSplit(input: Iterable<string> | undefined): string[][] {\n  if (!input) {\n    return [];\n  }\n  const array = Array.isArray(input) ? input.slice() : [...input];\n  array.sort();\n  return splitToChunks(array, UPDATE_CHUNK_SIZE);\n}\n"],"names":["retryOnDeadlock","uuid","splitToChunks"],"mappings":";;;;;;;;;;AA2BA,MAAM,iBAAA,GAAoB,GAAA;AAQ1B,eAAsB,iBAAiB,OAAA,EAKrB;AAChB,EAAA,MAAM,UAAA,GAAa,SAAA,CAAU,OAAA,CAAQ,UAAU,CAAA;AAC/C,EAAA,MAAM,SAAA,GAAY,SAAA,CAAU,OAAA,CAAQ,SAAS,CAAA;AAC7C,EAAA,MAAM,OAAO,OAAA,CAAQ,IAAA;AACrB,EAAA,MAAM,IAAA,GAAO,QAAQ,QAAA,CAAS,IAAA;AAE9B,EAAA,IAAI,SAAS,WAAA,EAAa;AACxB,IAAA,KAAA,MAAW,SAAS,UAAA,EAAY;AAC9B,MAAA,MAAM,IAAA,CACH,KAAA,CAA0B,gBAAgB,CAAA,CAC1C,MAAA,CAAO;AAAA,QACN,IAAA,EAAM;AAAA,OACP,CAAA,CACA,OAAA,CAAQ,YAAA,EAAc,KAAK,CAAA;AAC9B,MAAA,MAAMA,qBAAgB,YAAY;AAChC,QAAA,MAAM,IAAA,CACH,KAAA,CAAyB,eAAe,CAAA,CACxC,MAAA,CAAO;AAAA,UACN,WAAA,EAAa,iBAAA;AAAA,UACb,cAAA,EAAgB,IAAA,CAAK,EAAA,CAAG,GAAA;AAAI,SAC7B,CAAA,CACA,OAAA,CAAQ,YAAA,EAAc,KAAK,CAAA;AAAA,MAChC,GAAG,IAAI,CAAA;AAAA,IACT;AAEA,IAAA,KAAA,MAAW,SAAS,SAAA,EAAW;AAC7B,MAAA,MAAM,IAAA,CACH,KAAA,CAA0B,gBAAgB,CAAA,CAC1C,MAAA,CAAO;AAAA,QACN,IAAA,EAAM;AAAA,OACP,CAAA,CACA,OAAA,CAAQ,WAAA,EAAa,KAAK,CAAA;AAC7B,MAAA,MAAMA,qBAAgB,YAAY;AAChC,QAAA,MAAM,IAAA,CACH,KAAA,CAAyB,eAAe,CAAA,CACxC,MAAA,CAAO;AAAA,UACN,WAAA,EAAa,iBAAA;AAAA,UACb,cAAA,EAAgB,IAAA,CAAK,EAAA,CAAG,GAAA;AAAI,SAC7B,CAAA,CACA,OAAA,CAAQ,WAAA,EAAa,KAAK,CAAA;AAAA,MAC/B,GAAG,IAAI,CAAA;AAAA,IACT;AAAA,EACF,CAAA,MAAA,IAAW,SAAS,UAAA,EAAY;AAG9B,IAAA,MAAM,SAASC,sBAAA,EAAK;AAEpB,IAAA,KAAA,MAAW,SAAS,UAAA,EAAY;AAC9B,MAAA,MAAMD,qBAAgB,YAAY;AAChC,QAAA,IAAI,KAAA,CAAM,SAAS,CAAA,EAAG;AACpB,UAAA,MAAM,IAAA,CAAuB,cAAc,CAAA,CACxC,MAAA;AAAA,YACC,KAAA,CAAM,IAAI,CAAA,GAAA,MAAQ;AAAA,cAChB,UAAA,EAAY,GAAA;AAAA,cACZ,aAAA,EAAe,MAAA;AAAA,cACf,cAAA,EAAgB,IAAA,CAAK,EAAA,CAAG,GAAA;AAAI,aAC9B,CAAE;AAAA,WACJ,CACC,WAAW,YAAY,CAAA,CACvB,MAAM,CAAC,gBAAA,EAAkB,eAAe,CAAC,CAAA;AAAA,QAC9C;AAAA,MACF,GAAG,IAAI,CAAA;AAAA,IACT;AAEA,IAAA,KAAA,MAAW,SAAS,SAAA,EAAW;AAC7B,MAAA,MAAMA,qBAAgB,YAAY;AAEhC,QAAA,MAAM,gBAAA,GAAmB,MAAM,IAAA,CAAwB,eAAe,CAAA,CACnE,OAAO,YAAY,CAAA,CACnB,OAAA,CAAQ,WAAA,EAAa,KAAK,CAAA;AAE7B,QAAA,IAAI,gBAAA,CAAiB,SAAS,CAAA,EAAG;AAC/B,UAAA,MAAM,IAAA,CAAuB,cAAc,CAAA,CACxC,MAAA;AAAA,YACC,gBAAA,CAAiB,IAAI,CAAA,GAAA,MAAQ;AAAA,cAC3B,YAAY,GAAA,CAAI,UAAA;AAAA,cAChB,aAAA,EAAe,MAAA;AAAA,cACf,cAAA,EAAgB,IAAA,CAAK,EAAA,CAAG,GAAA;AAAI,aAC9B,CAAE;AAAA,WACJ,CACC,WAAW,YAAY,CAAA,CACvB,MAAM,CAAC,gBAAA,EAAkB,eAAe,CAAC,CAAA;AAAA,QAC9C;AAAA,MACF,GAAG,IAAI,CAAA;AAAA,IACT;AAAA,EACF,CAAA,MAAO;AACL,IAAA,MAAM,IAAI,KAAA,CAAM,CAAA,gCAAA,EAAmC,IAAI,CAAA,CAAE,CAAA;AAAA,EAC3D;AACF;AAEA,SAAS,UAAU,KAAA,EAAiD;AAClE,EAAA,IAAI,CAAC,KAAA,EAAO;AACV,IAAA,OAAO,EAAC;AAAA,EACV;AACA,EAAA,MAAM,KAAA,GAAQ,KAAA,CAAM,OAAA,CAAQ,KAAK,CAAA,GAAI,MAAM,KAAA,EAAM,GAAI,CAAC,GAAG,KAAK,CAAA;AAC9D,EAAA,KAAA,CAAM,IAAA,EAAK;AACX,EAAA,OAAOE,8BAAA,CAAc,OAAO,iBAAiB,CAAA;AAC/C;;;;"}