@backstage/plugin-catalog-backend 3.9.2-next.0 → 4.0.0

package/CHANGELOG.md

@@ -1,5 +1,70 @@
 # @backstage/plugin-catalog-backend
 
+## 4.0.0
+
+### Major Changes
+
+- b0d8a53: **BREAKING**: Removed the deprecated `CodeOwnersProcessor` and `AnnotateScmSlugEntityProcessor` exports. To retain their behavior, install and register the corresponding community backend module:
+
+  - `CodeOwnersProcessor`: [`@backstage-community/plugin-catalog-backend-module-codeowners`](https://github.com/backstage/community-plugins/tree/main/workspaces/catalog/plugins/catalog-backend-module-codeowners)
+  - `AnnotateScmSlugEntityProcessor`: [`@backstage-community/plugin-catalog-backend-module-annotate-scm-slug`](https://github.com/backstage/community-plugins/tree/main/workspaces/catalog/plugins/catalog-backend-module-annotate-scm-slug)
+
+  Add the module to your backend with `backend.add(import('<module-package>'))` and remove the custom registration of the old processor. For explicit processor configuration, follow the replacement module's documentation.
+
+- b0d8a53: **BREAKING**: Removed the deprecated `catalog.stitchingStrategy.mode` configuration setting and its startup warnings. Remove this key from your configuration, whether it is set to `immediate` or `deferred`. Stitching continues to run asynchronously via the worker queue. The `catalog.stitchingStrategy.pollingInterval` and `catalog.stitchingStrategy.stitchTimeout` settings remain supported and should be kept if you use them.
+- d2cf99f: **BREAKING**: Removed the deprecated `catalog.enableRelationsCompatibility` config option and its associated compatibility layer. Entity relations are now always returned in the standard format with only `targetRef`. If you were relying on the `target` field in relations, update your code to use `targetRef` instead.
+
+### Patch Changes
+
+- 736d84e: Use locale-insensitive Unicode casing for consistent string handling across environments.
+- f5119cc: Improved PostgreSQL catalog query planning by correcting entity cardinality statistics for the search table.
+- 72c16f5: Use stable PostgreSQL array parameters for `$in` filter predicates.
+- 78656f8: Fixed catalog model relations to inherit the source entity namespace when no default namespace is configured, as documented. Explicit namespaces in entity references and explicitly configured default namespaces continue to take precedence.
+- 7761a50: Improved path validation for catalog entity placeholders.
+- 49b41f6: Fixed a bug where catalog entities could fail to process when catalog model sources are enabled and the catalog mixes `backstage.io/v1alpha1` and `backstage.io/v1beta1` for the same kind.
+- 4b922b3: The `catalog:query-catalog-entities` action now accepts `limit` and `offset` when they are passed as strings, coercing them to numbers before validation. Previously the action failed with a validation error when a client sent these pagination arguments as strings, which is common for MCP/LLM clients.
+- fda0ef1: Correct catalog property permission matching.
+- ff327e0: Apply catalog location analysis permissions consistently when using a custom location analyzer.
+- 05e2005: Fixed PostgreSQL catalog migrations sometimes failing during search index deduplication when using a database connection pool.
+- 7629e5e: Use stable PostgreSQL array parameters for batched refresh state lookups and updates.
+- c26a19b: Improved catalog database performance stability for large PostgreSQL installations with frequent entity updates.
+- e363ae2: Allowed location type restrictions are now applied consistently during catalog processing.
+- Updated dependencies
+  - @backstage/integration@2.1.2
+  - @backstage/plugin-catalog-common@1.2.0
+  - @backstage/backend-openapi-utils@0.7.2
+  - @backstage/catalog-client@1.16.2
+  - @backstage/catalog-model@1.10.1
+  - @backstage/filter-predicates@0.1.5
+  - @backstage/config@1.3.9
+  - @backstage/plugin-permission-node@0.11.4
+  - @backstage/backend-plugin-api@1.10.1
+  - @backstage/plugin-catalog-node@2.2.5
+  - @backstage/plugin-events-node@0.4.26
+  - @backstage/plugin-permission-common@0.9.11
+
+## 4.0.0-next.1
+
+### Major Changes
+
+- d2cf99f: **BREAKING**: Removed the deprecated `catalog.enableRelationsCompatibility` config option and its associated compatibility layer. Entity relations are now always returned in the standard format with only `targetRef`. If you were relying on the `target` field in relations, update your code to use `targetRef` instead.
+
+### Patch Changes
+
+- f5119cc: Improved PostgreSQL catalog query planning by correcting entity cardinality statistics for the search table.
+- 72c16f5: Use stable PostgreSQL array parameters for `$in` filter predicates.
+- 7629e5e: Use stable PostgreSQL array parameters for batched refresh state lookups and updates.
+- Updated dependencies
+  - @backstage/plugin-catalog-common@1.2.0-next.0
+  - @backstage/config@1.3.9-next.0
+  - @backstage/catalog-client@1.16.2-next.1
+  - @backstage/plugin-catalog-node@2.2.5-next.1
+  - @backstage/backend-plugin-api@1.10.1-next.1
+  - @backstage/filter-predicates@0.1.5-next.1
+  - @backstage/integration@2.1.2-next.1
+  - @backstage/plugin-permission-common@0.9.11-next.0
+  - @backstage/plugin-permission-node@0.11.4-next.1
+
 ## 3.9.2-next.0
 
 ### Patch Changes

package/README.md

@@ -69,6 +69,19 @@ you will not have any catalog entities loaded. See [Catalog Configuration](https
 for how to add locations, or copy the catalog locations from the [create-app template](https://github.com/backstage/backstage/blob/master/packages/create-app/templates/default-app/app-config.yaml.hbs)
 to get up and running quickly.
 
+## Migrating removed processors
+
+`CodeOwnersProcessor` and `AnnotateScmSlugEntityProcessor` are no longer exported
+by this package. Use the corresponding community modules to retain their behavior:
+
+- `CodeOwnersProcessor`: [`@backstage-community/plugin-catalog-backend-module-codeowners`](https://github.com/backstage/community-plugins/tree/main/workspaces/catalog/plugins/catalog-backend-module-codeowners)
+- `AnnotateScmSlugEntityProcessor`: [`@backstage-community/plugin-catalog-backend-module-annotate-scm-slug`](https://github.com/backstage/community-plugins/tree/main/workspaces/catalog/plugins/catalog-backend-module-annotate-scm-slug)
+
+Install the module you need in your backend package and register it with
+`backend.add(import('<module-package>'))`. Remove your custom registration of the
+old processor to avoid registering it twice. If you need to configure a processor
+explicitly, follow the replacement module's documentation.
+
 ## Development
 
 This backend plugin can be started in a standalone mode from directly in this

package/config.schema.json

@@ -110,10 +110,6 @@
           },
           "description": "A set of static locations that the catalog shall always keep itself up-to-date with. This is commonly used for large, permanent integrations that are defined by the Backstage operators at an organization, rather than individual things that users register dynamically.\n\nThese have (optional) rules of their own. These override what the global rules above specify. This way, you can prevent everybody from register e.g. User and Group entities, except for one or a few static locations that have those two kinds explicitly allowed.\n\nFor example:\n\n```yaml rules:   - allow: [Component, API, Template, Location] locations:   - type: url     target: https://github.com/org/repo/blob/master/users.yaml     rules:       - allow: [User, Group]   - type: url     target: https://github.com/org/repo/blob/master/systems.yaml     rules:       - allow: [System] ```"
         },
-        "enableRelationsCompatibility": {
-          "type": "boolean",
-          "description": "Enables the compatibility layer for relations in returned entities that ensures that all relations objects have both `target` and `targetRef`.\n\nEnabling this option significantly increases the memory usage of the catalog, and slightly reduces performance, but may avoid breaking consumers that rely on the existence of `target` in the relations objects."
-        },
         "disableDefaultProcessors": {
           "type": "boolean",
           "description": "Disables the default backstage processors.\n\nEnabling this option allows more complete control of which processors are included in the backstage processing loop."
@@ -137,10 +133,6 @@
         "stitchingStrategy": {
           "type": "object",
           "properties": {
-            "mode": {
-              "type": "string",
-              "deprecated": "Immediate mode has been removed. This field is ignored."
-            },
             "pollingInterval": {
               "anyOf": [
                 {

package/dist/actions/createQueryCatalogEntitiesAction.cjs.js

@@ -104,43 +104,52 @@ const createQueryCatalogEntitiesAction = ({
     },
     description: useExperimentalCatalogLayersDescriptions ? MODEL_REFERENCE_DESCRIPTION : INLINE_MODEL_DESCRIPTION,
     schema: {
-      input: (z) => z.object({
-        query: filterPredicates.createZodV3FilterPredicateSchema(z).optional().describe(
-          "Entity predicate query. Supports field matching, $all, $any, $not, $exists, $in, $contains, and $hasPrefix operators."
-        ),
-        fields: z.array(z.string()).optional().describe(
-          "Specific fields to include in the response. If not provided, all fields are returned. Each entry is a dot separated path into an entity, e.g. `spec.type`."
-        ),
-        limit: z.number().int().positive().optional().describe("Maximum number of entities to return at a time."),
-        offset: z.number().int().min(0).optional().describe("Number of entities to skip before returning results."),
-        orderFields: z.union([
-          z.object({
-            field: z.string().describe(
-              "Field to order by. The format is a dot separated path into an entity, e.g. `spec.type`."
-            ),
-            order: z.enum(["asc", "desc"]).describe("Sort order")
-          }),
-          z.array(
+      input: (z) => {
+        const numericFromString = (value) => {
+          if (typeof value === "string" && value.trim() !== "") {
+            const parsed = Number(value);
+            return Number.isFinite(parsed) ? parsed : value;
+          }
+          return value;
+        };
+        return z.object({
+          query: filterPredicates.createZodV3FilterPredicateSchema(z).optional().describe(
+            "Entity predicate query. Supports field matching, $all, $any, $not, $exists, $in, $contains, and $hasPrefix operators."
+          ),
+          fields: z.array(z.string()).optional().describe(
+            "Specific fields to include in the response. If not provided, all fields are returned. Each entry is a dot separated path into an entity, e.g. `spec.type`."
+          ),
+          limit: z.preprocess(numericFromString, z.number().int().positive()).optional().describe("Maximum number of entities to return at a time."),
+          offset: z.preprocess(numericFromString, z.number().int().min(0)).optional().describe("Number of entities to skip before returning results."),
+          orderFields: z.union([
             z.object({
               field: z.string().describe(
                 "Field to order by. The format is a dot separated path into an entity, e.g. `spec.type`."
               ),
               order: z.enum(["asc", "desc"]).describe("Sort order")
-            })
-          )
-        ]).optional().describe(
-          "Ordering criteria for the results. Can be a single order directive or an array for multi-field sorting."
-        ),
-        fullTextFilter: z.object({
-          term: z.string().describe("Full text search term"),
-          fields: z.array(z.string()).optional().describe(
-            "Fields to search within. Each entry is a dot separated path into an entity, e.g. `spec.type`."
+            }),
+            z.array(
+              z.object({
+                field: z.string().describe(
+                  "Field to order by. The format is a dot separated path into an entity, e.g. `spec.type`."
+                ),
+                order: z.enum(["asc", "desc"]).describe("Sort order")
+              })
+            )
+          ]).optional().describe(
+            "Ordering criteria for the results. Can be a single order directive or an array for multi-field sorting."
+          ),
+          fullTextFilter: z.object({
+            term: z.string().describe("Full text search term"),
+            fields: z.array(z.string()).optional().describe(
+              "Fields to search within. Each entry is a dot separated path into an entity, e.g. `spec.type`."
+            )
+          }).optional().describe("Full text search criteria"),
+          cursor: z.string().optional().describe(
+            "Cursor for pagination. This can be used only after the first request with a response containing a cursor. If a cursor is given it takes precedence over `offset`."
           )
-        }).optional().describe("Full text search criteria"),
-        cursor: z.string().optional().describe(
-          "Cursor for pagination. This can be used only after the first request with a response containing a cursor. If a cursor is given it takes precedence over `offset`."
-        )
-      }),
+        });
+      },
       output: (z) => z.object({
         items: z.array(z.object({}).passthrough()).describe("List of entities"),
         totalItems: z.number().describe("Total number of entities"),

package/dist/actions/createQueryCatalogEntitiesAction.cjs.js.map

@@ -1 +1 @@
-{"version":3,"file":"createQueryCatalogEntitiesAction.cjs.js","sources":["../../src/actions/createQueryCatalogEntitiesAction.ts"],"sourcesContent":["/*\n * Copyright 2025 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\nimport { ActionsRegistryService } from '@backstage/backend-plugin-api/alpha';\nimport { CatalogService } from '@backstage/plugin-catalog-node';\nimport { createZodV3FilterPredicateSchema } from '@backstage/filter-predicates';\n\nconst QUERY_SYNTAX = `\n## Query Syntax\n\nThe query uses predicate expressions with dot-notation field paths.\n\nSimple matching:\n  { query: { kind: \"Component\" } }\n  { query: { kind: \"Component\", \"spec.type\": \"service\" } }\n\nValue operators:\n  { query: { kind: { \"$in\": [\"API\", \"Component\"] } } }\n  { query: { \"metadata.annotations.backstage.io/techdocs-ref\": { \"$exists\": true } } }\n  { query: { \"metadata.tags\": { \"$contains\": \"java\" } } }\n  { query: { \"metadata.name\": { \"$hasPrefix\": \"team-\" } } }\n\nLogical operators:\n  { query: { \"$all\": [{ kind: \"Component\" }, { \"spec.lifecycle\": \"production\" }] } }\n  { query: { \"$any\": [{ \"spec.type\": \"service\" }, { \"spec.type\": \"website\" }] } }\n  { query: { \"$not\": { kind: \"Group\" } } }\n\nQuerying relations - find all entities owned by a specific group:\n  { query: { \"relations.ownedby\": \"group:default/team-alpha\" } }\n\nCombined example - find production services or websites with TechDocs:\n  { query: { \"$all\": [\n    { kind: \"Component\", \"spec.lifecycle\": \"production\" },\n    { \"$any\": [{ \"spec.type\": \"service\" }, { \"spec.type\": \"website\" }] },\n    { \"metadata.annotations.backstage.io/techdocs-ref\": { \"$exists\": true } }\n  ] } }\n\n## Other Options\n\nLimit returned fields: { fields: [\"kind\", \"metadata.name\", \"metadata.namespace\"] }\nSort results: { orderFields: { field: \"metadata.name\", order: \"asc\" } }\nFull text search: { fullTextFilter: { term: \"auth\", fields: [\"metadata.name\", \"metadata.title\"] } }\nPagination: Use limit (e.g. 20) and the returned nextPageCursor for subsequent requests via cursor.\n`;\n\nconst INLINE_MODEL_DESCRIPTION = `\nQuery entities from the Backstage Software Catalog using predicate filters.\n\n## Catalog Model\n\nThe catalog contains entities of different kinds. Every entity has \"kind\", \"apiVersion\", \"metadata\", and optionally \"spec\" and \"relations\". Fields use dot notation for querying.\n\nCommon metadata fields on all entities: name, namespace (default: \"default\"), title, description, labels, annotations, tags (string array), links.\n\nEntity references use the format \"kind:namespace/name\", e.g. \"component:default/my-service\" or \"user:default/jane.doe\".\n\n### Entity Kinds\n\n**Component** - A piece of software such as a service, website, or library.\n  spec fields: type (e.g. \"service\", \"website\", \"library\"), lifecycle (e.g. \"production\", \"experimental\", \"deprecated\"), owner (entity ref), system, subcomponentOf, providesApis, consumesApis, dependsOn, dependencyOf.\n\n**API** - An interface that components expose, such as REST APIs or event streams.\n  spec fields: type (e.g. \"openapi\", \"asyncapi\", \"graphql\", \"grpc\"), lifecycle, owner (entity ref), definition (the API spec content), system.\n\n**System** - A collection of components, APIs, and resources that together expose some functionality.\n  spec fields: owner (entity ref), domain, type.\n\n**Domain** - A grouping of systems that share terminology, domain models, and business purpose.\n  spec fields: owner (entity ref), subdomainOf, type.\n\n**Resource** - Infrastructure required to operate a component, such as databases or storage buckets.\n  spec fields: type, owner (entity ref), system, dependsOn, dependencyOf.\n\n**Group** - An organizational entity such as a team or business unit.\n  spec fields: type (e.g. \"team\", \"business-unit\"), children (entity refs), parent (entity ref), members (entity refs), profile (displayName, email, picture).\n\n**User** - A person, such as an employee or contractor.\n  spec fields: memberOf (entity refs), profile (displayName, email, picture).\n\n**Location** - A marker that references other catalog descriptor files to be ingested.\n  spec fields: type, target, targets, presence.\n\n### Relations\n\nEntities have bidirectional relations stored in the \"relations\" array. Common relation types: ownedBy/ownerOf, dependsOn/dependencyOf, providesApi/apiProvidedBy, consumesApi/apiConsumedBy, parentOf/childOf, memberOf/hasMember, partOf/hasPart.\n\nRelations can be queried via \"relations.<type>\" e.g. \"relations.ownedby: user:default/jane-doe\". The value there must always be a valid entity reference.\n\nWhen querying for entity relationships, prefer using relations over spec fields. For example, use \"relations.ownedby\" instead of \"spec.owner\" to find entities owned by a particular group or user.\n${QUERY_SYNTAX}`;\n\nconst MODEL_REFERENCE_DESCRIPTION = `\nQuery entities from the Backstage Software Catalog using predicate filters.\n\nFor a complete list of entity kinds, fields, relations, and other queryable attributes available in the catalog, use \\`get-catalog-model-description\\`.\n${QUERY_SYNTAX}`;\n\nexport const createQueryCatalogEntitiesAction = ({\n  catalog,\n  actionsRegistry,\n  useExperimentalCatalogLayersDescriptions,\n}: {\n  catalog: CatalogService;\n  actionsRegistry: ActionsRegistryService;\n  useExperimentalCatalogLayersDescriptions?: boolean;\n}) => {\n  actionsRegistry.register({\n    name: 'query-catalog-entities',\n    title: 'Query Catalog Entities',\n    attributes: {\n      destructive: false,\n      readOnly: true,\n      idempotent: true,\n    },\n    description: useExperimentalCatalogLayersDescriptions\n      ? MODEL_REFERENCE_DESCRIPTION\n      : INLINE_MODEL_DESCRIPTION,\n    schema: {\n      input: z =>\n        z.object({\n          query: createZodV3FilterPredicateSchema(z)\n            .optional()\n            .describe(\n              'Entity predicate query. Supports field matching, $all, $any, $not, $exists, $in, $contains, and $hasPrefix operators.',\n            ),\n          fields: z\n            .array(z.string())\n            .optional()\n            .describe(\n              'Specific fields to include in the response. If not provided, all fields are returned. Each entry is a dot separated path into an entity, e.g. `spec.type`.',\n            ),\n          limit: z\n            .number()\n            .int()\n            .positive()\n            .optional()\n            .describe('Maximum number of entities to return at a time.'),\n          offset: z\n            .number()\n            .int()\n            .min(0)\n            .optional()\n            .describe('Number of entities to skip before returning results.'),\n          orderFields: z\n            .union([\n              z.object({\n                field: z\n                  .string()\n                  .describe(\n                    'Field to order by. The format is a dot separated path into an entity, e.g. `spec.type`.',\n                  ),\n                order: z.enum(['asc', 'desc']).describe('Sort order'),\n              }),\n              z.array(\n                z.object({\n                  field: z\n                    .string()\n                    .describe(\n                      'Field to order by. The format is a dot separated path into an entity, e.g. `spec.type`.',\n                    ),\n                  order: z.enum(['asc', 'desc']).describe('Sort order'),\n                }),\n              ),\n            ])\n            .optional()\n            .describe(\n              'Ordering criteria for the results. Can be a single order directive or an array for multi-field sorting.',\n            ),\n          fullTextFilter: z\n            .object({\n              term: z.string().describe('Full text search term'),\n              fields: z\n                .array(z.string())\n                .optional()\n                .describe(\n                  'Fields to search within. Each entry is a dot separated path into an entity, e.g. `spec.type`.',\n                ),\n            })\n            .optional()\n            .describe('Full text search criteria'),\n          cursor: z\n            .string()\n            .optional()\n            .describe(\n              'Cursor for pagination. This can be used only after the first request with a response containing a cursor. If a cursor is given it takes precedence over `offset`.',\n            ),\n        }),\n      output: z =>\n        z.object({\n          items: z\n            .array(z.object({}).passthrough())\n            .describe('List of entities'),\n          totalItems: z.number().describe('Total number of entities'),\n          hasMoreEntities: z\n            .boolean()\n            .describe('Whether more entities are available'),\n          nextPageCursor: z\n            .string()\n            .optional()\n            .describe('Next page cursor used to fetch next page of entities'),\n        }),\n    },\n    action: async ({ input, credentials }) => {\n      const response = await catalog.queryEntities(\n        {\n          ...input,\n          query: input.query,\n        },\n        { credentials },\n      );\n\n      return {\n        output: {\n          items: response.items,\n          totalItems: response.totalItems,\n          hasMoreEntities: !!response.pageInfo.nextCursor,\n          nextPageCursor: response.pageInfo.nextCursor,\n        },\n      };\n    },\n  });\n};\n"],"names":["createZodV3FilterPredicateSchema"],"mappings":";;;;AAmBA,MAAM,YAAA,GAAe;AAAA;;AAAA;;AAAA;AAAA;AAAA;;AAAA;AAAA;AAAA;AAAA;AAAA;;AAAA;AAAA;AAAA;AAAA;;AAAA;AAAA;;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;;AAAA;;AAAA;AAAA;AAAA;AAAA;AAAA,CAAA;AAsCrB,MAAM,wBAAA,GAA2B;AAAA;;AAAA;;AAAA;;AAAA;;AAAA;;AAAA;;AAAA;AAAA;;AAAA;AAAA;;AAAA;AAAA;;AAAA;AAAA;;AAAA;AAAA;;AAAA;AAAA;;AAAA;AAAA;;AAAA;AAAA;;AAAA;;AAAA;;AAAA;;AAAA;AAAA,EA4C/B,YAAY,CAAA,CAAA;AAEd,MAAM,2BAAA,GAA8B;AAAA;;AAAA;AAAA,EAIlC,YAAY,CAAA,CAAA;AAEP,MAAM,mCAAmC,CAAC;AAAA,EAC/C,OAAA;AAAA,EACA,eAAA;AAAA,EACA;AACF,CAAA,KAIM;AACJ,EAAA,eAAA,CAAgB,QAAA,CAAS;AAAA,IACvB,IAAA,EAAM,wBAAA;AAAA,IACN,KAAA,EAAO,wBAAA;AAAA,IACP,UAAA,EAAY;AAAA,MACV,WAAA,EAAa,KAAA;AAAA,MACb,QAAA,EAAU,IAAA;AAAA,MACV,UAAA,EAAY;AAAA,KACd;AAAA,IACA,WAAA,EAAa,2CACT,2BAAA,GACA,wBAAA;AAAA,IACJ,MAAA,EAAQ;AAAA,MACN,KAAA,EAAO,CAAA,CAAA,KACL,CAAA,CAAE,MAAA,CAAO;AAAA,QACP,KAAA,EAAOA,iDAAA,CAAiC,CAAC,CAAA,CACtC,UAAS,CACT,QAAA;AAAA,UACC;AAAA,SACF;AAAA,QACF,MAAA,EAAQ,EACL,KAAA,CAAM,CAAA,CAAE,QAAQ,CAAA,CAChB,UAAS,CACT,QAAA;AAAA,UACC;AAAA,SACF;AAAA,QACF,KAAA,EAAO,CAAA,CACJ,MAAA,EAAO,CACP,GAAA,EAAI,CACJ,QAAA,EAAS,CACT,QAAA,EAAS,CACT,QAAA,CAAS,iDAAiD,CAAA;AAAA,QAC7D,MAAA,EAAQ,CAAA,CACL,MAAA,EAAO,CACP,GAAA,EAAI,CACJ,GAAA,CAAI,CAAC,CAAA,CACL,QAAA,EAAS,CACT,QAAA,CAAS,sDAAsD,CAAA;AAAA,QAClE,WAAA,EAAa,EACV,KAAA,CAAM;AAAA,UACL,EAAE,MAAA,CAAO;AAAA,YACP,KAAA,EAAO,CAAA,CACJ,MAAA,EAAO,CACP,QAAA;AAAA,cACC;AAAA,aACF;AAAA,YACF,KAAA,EAAO,EAAE,IAAA,CAAK,CAAC,OAAO,MAAM,CAAC,CAAA,CAAE,QAAA,CAAS,YAAY;AAAA,WACrD,CAAA;AAAA,UACD,CAAA,CAAE,KAAA;AAAA,YACA,EAAE,MAAA,CAAO;AAAA,cACP,KAAA,EAAO,CAAA,CACJ,MAAA,EAAO,CACP,QAAA;AAAA,gBACC;AAAA,eACF;AAAA,cACF,KAAA,EAAO,EAAE,IAAA,CAAK,CAAC,OAAO,MAAM,CAAC,CAAA,CAAE,QAAA,CAAS,YAAY;AAAA,aACrD;AAAA;AACH,SACD,CAAA,CACA,QAAA,EAAS,CACT,QAAA;AAAA,UACC;AAAA,SACF;AAAA,QACF,cAAA,EAAgB,EACb,MAAA,CAAO;AAAA,UACN,IAAA,EAAM,CAAA,CAAE,MAAA,EAAO,CAAE,SAAS,uBAAuB,CAAA;AAAA,UACjD,MAAA,EAAQ,EACL,KAAA,CAAM,CAAA,CAAE,QAAQ,CAAA,CAChB,UAAS,CACT,QAAA;AAAA,YACC;AAAA;AACF,SACH,CAAA,CACA,QAAA,EAAS,CACT,SAAS,2BAA2B,CAAA;AAAA,QACvC,MAAA,EAAQ,CAAA,CACL,MAAA,EAAO,CACP,UAAS,CACT,QAAA;AAAA,UACC;AAAA;AACF,OACH,CAAA;AAAA,MACH,MAAA,EAAQ,CAAA,CAAA,KACN,CAAA,CAAE,MAAA,CAAO;AAAA,QACP,KAAA,EAAO,CAAA,CACJ,KAAA,CAAM,CAAA,CAAE,MAAA,CAAO,EAAE,CAAA,CAAE,WAAA,EAAa,CAAA,CAChC,QAAA,CAAS,kBAAkB,CAAA;AAAA,QAC9B,UAAA,EAAY,CAAA,CAAE,MAAA,EAAO,CAAE,SAAS,0BAA0B,CAAA;AAAA,QAC1D,eAAA,EAAiB,CAAA,CACd,OAAA,EAAQ,CACR,SAAS,qCAAqC,CAAA;AAAA,QACjD,gBAAgB,CAAA,CACb,MAAA,GACA,QAAA,EAAS,CACT,SAAS,sDAAsD;AAAA,OACnE;AAAA,KACL;AAAA,IACA,MAAA,EAAQ,OAAO,EAAE,KAAA,EAAO,aAAY,KAAM;AACxC,MAAA,MAAM,QAAA,GAAW,MAAM,OAAA,CAAQ,aAAA;AAAA,QAC7B;AAAA,UACE,GAAG,KAAA;AAAA,UACH,OAAO,KAAA,CAAM;AAAA,SACf;AAAA,QACA,EAAE,WAAA;AAAY,OAChB;AAEA,MAAA,OAAO;AAAA,QACL,MAAA,EAAQ;AAAA,UACN,OAAO,QAAA,CAAS,KAAA;AAAA,UAChB,YAAY,QAAA,CAAS,UAAA;AAAA,UACrB,eAAA,EAAiB,CAAC,CAAC,QAAA,CAAS,QAAA,CAAS,UAAA;AAAA,UACrC,cAAA,EAAgB,SAAS,QAAA,CAAS;AAAA;AACpC,OACF;AAAA,IACF;AAAA,GACD,CAAA;AACH;;;;"}
+{"version":3,"file":"createQueryCatalogEntitiesAction.cjs.js","sources":["../../src/actions/createQueryCatalogEntitiesAction.ts"],"sourcesContent":["/*\n * Copyright 2025 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\nimport { ActionsRegistryService } from '@backstage/backend-plugin-api/alpha';\nimport { CatalogService } from '@backstage/plugin-catalog-node';\nimport { createZodV3FilterPredicateSchema } from '@backstage/filter-predicates';\n\nconst QUERY_SYNTAX = `\n## Query Syntax\n\nThe query uses predicate expressions with dot-notation field paths.\n\nSimple matching:\n  { query: { kind: \"Component\" } }\n  { query: { kind: \"Component\", \"spec.type\": \"service\" } }\n\nValue operators:\n  { query: { kind: { \"$in\": [\"API\", \"Component\"] } } }\n  { query: { \"metadata.annotations.backstage.io/techdocs-ref\": { \"$exists\": true } } }\n  { query: { \"metadata.tags\": { \"$contains\": \"java\" } } }\n  { query: { \"metadata.name\": { \"$hasPrefix\": \"team-\" } } }\n\nLogical operators:\n  { query: { \"$all\": [{ kind: \"Component\" }, { \"spec.lifecycle\": \"production\" }] } }\n  { query: { \"$any\": [{ \"spec.type\": \"service\" }, { \"spec.type\": \"website\" }] } }\n  { query: { \"$not\": { kind: \"Group\" } } }\n\nQuerying relations - find all entities owned by a specific group:\n  { query: { \"relations.ownedby\": \"group:default/team-alpha\" } }\n\nCombined example - find production services or websites with TechDocs:\n  { query: { \"$all\": [\n    { kind: \"Component\", \"spec.lifecycle\": \"production\" },\n    { \"$any\": [{ \"spec.type\": \"service\" }, { \"spec.type\": \"website\" }] },\n    { \"metadata.annotations.backstage.io/techdocs-ref\": { \"$exists\": true } }\n  ] } }\n\n## Other Options\n\nLimit returned fields: { fields: [\"kind\", \"metadata.name\", \"metadata.namespace\"] }\nSort results: { orderFields: { field: \"metadata.name\", order: \"asc\" } }\nFull text search: { fullTextFilter: { term: \"auth\", fields: [\"metadata.name\", \"metadata.title\"] } }\nPagination: Use limit (e.g. 20) and the returned nextPageCursor for subsequent requests via cursor.\n`;\n\nconst INLINE_MODEL_DESCRIPTION = `\nQuery entities from the Backstage Software Catalog using predicate filters.\n\n## Catalog Model\n\nThe catalog contains entities of different kinds. Every entity has \"kind\", \"apiVersion\", \"metadata\", and optionally \"spec\" and \"relations\". Fields use dot notation for querying.\n\nCommon metadata fields on all entities: name, namespace (default: \"default\"), title, description, labels, annotations, tags (string array), links.\n\nEntity references use the format \"kind:namespace/name\", e.g. \"component:default/my-service\" or \"user:default/jane.doe\".\n\n### Entity Kinds\n\n**Component** - A piece of software such as a service, website, or library.\n  spec fields: type (e.g. \"service\", \"website\", \"library\"), lifecycle (e.g. \"production\", \"experimental\", \"deprecated\"), owner (entity ref), system, subcomponentOf, providesApis, consumesApis, dependsOn, dependencyOf.\n\n**API** - An interface that components expose, such as REST APIs or event streams.\n  spec fields: type (e.g. \"openapi\", \"asyncapi\", \"graphql\", \"grpc\"), lifecycle, owner (entity ref), definition (the API spec content), system.\n\n**System** - A collection of components, APIs, and resources that together expose some functionality.\n  spec fields: owner (entity ref), domain, type.\n\n**Domain** - A grouping of systems that share terminology, domain models, and business purpose.\n  spec fields: owner (entity ref), subdomainOf, type.\n\n**Resource** - Infrastructure required to operate a component, such as databases or storage buckets.\n  spec fields: type, owner (entity ref), system, dependsOn, dependencyOf.\n\n**Group** - An organizational entity such as a team or business unit.\n  spec fields: type (e.g. \"team\", \"business-unit\"), children (entity refs), parent (entity ref), members (entity refs), profile (displayName, email, picture).\n\n**User** - A person, such as an employee or contractor.\n  spec fields: memberOf (entity refs), profile (displayName, email, picture).\n\n**Location** - A marker that references other catalog descriptor files to be ingested.\n  spec fields: type, target, targets, presence.\n\n### Relations\n\nEntities have bidirectional relations stored in the \"relations\" array. Common relation types: ownedBy/ownerOf, dependsOn/dependencyOf, providesApi/apiProvidedBy, consumesApi/apiConsumedBy, parentOf/childOf, memberOf/hasMember, partOf/hasPart.\n\nRelations can be queried via \"relations.<type>\" e.g. \"relations.ownedby: user:default/jane-doe\". The value there must always be a valid entity reference.\n\nWhen querying for entity relationships, prefer using relations over spec fields. For example, use \"relations.ownedby\" instead of \"spec.owner\" to find entities owned by a particular group or user.\n${QUERY_SYNTAX}`;\n\nconst MODEL_REFERENCE_DESCRIPTION = `\nQuery entities from the Backstage Software Catalog using predicate filters.\n\nFor a complete list of entity kinds, fields, relations, and other queryable attributes available in the catalog, use \\`get-catalog-model-description\\`.\n${QUERY_SYNTAX}`;\n\nexport const createQueryCatalogEntitiesAction = ({\n  catalog,\n  actionsRegistry,\n  useExperimentalCatalogLayersDescriptions,\n}: {\n  catalog: CatalogService;\n  actionsRegistry: ActionsRegistryService;\n  useExperimentalCatalogLayersDescriptions?: boolean;\n}) => {\n  actionsRegistry.register({\n    name: 'query-catalog-entities',\n    title: 'Query Catalog Entities',\n    attributes: {\n      destructive: false,\n      readOnly: true,\n      idempotent: true,\n    },\n    description: useExperimentalCatalogLayersDescriptions\n      ? MODEL_REFERENCE_DESCRIPTION\n      : INLINE_MODEL_DESCRIPTION,\n    schema: {\n      input: z => {\n        // MCP/LLM clients often send numeric arguments as strings. Convert numeric\n        // strings to numbers, but leave everything else untouched so Zod rejects the\n        // original value (e.g. reports \"received string\" for 'abc' rather than\n        // \"received NaN\", and doesn't silently turn true/null/'' into 1/0/0).\n        const numericFromString = (value: unknown) => {\n          if (typeof value === 'string' && value.trim() !== '') {\n            const parsed = Number(value);\n            return Number.isFinite(parsed) ? parsed : value;\n          }\n          return value;\n        };\n\n        return z.object({\n          query: createZodV3FilterPredicateSchema(z)\n            .optional()\n            .describe(\n              'Entity predicate query. Supports field matching, $all, $any, $not, $exists, $in, $contains, and $hasPrefix operators.',\n            ),\n          fields: z\n            .array(z.string())\n            .optional()\n            .describe(\n              'Specific fields to include in the response. If not provided, all fields are returned. Each entry is a dot separated path into an entity, e.g. `spec.type`.',\n            ),\n          limit: z\n            .preprocess(numericFromString, z.number().int().positive())\n            .optional()\n            .describe('Maximum number of entities to return at a time.'),\n          offset: z\n            .preprocess(numericFromString, z.number().int().min(0))\n            .optional()\n            .describe('Number of entities to skip before returning results.'),\n          orderFields: z\n            .union([\n              z.object({\n                field: z\n                  .string()\n                  .describe(\n                    'Field to order by. The format is a dot separated path into an entity, e.g. `spec.type`.',\n                  ),\n                order: z.enum(['asc', 'desc']).describe('Sort order'),\n              }),\n              z.array(\n                z.object({\n                  field: z\n                    .string()\n                    .describe(\n                      'Field to order by. The format is a dot separated path into an entity, e.g. `spec.type`.',\n                    ),\n                  order: z.enum(['asc', 'desc']).describe('Sort order'),\n                }),\n              ),\n            ])\n            .optional()\n            .describe(\n              'Ordering criteria for the results. Can be a single order directive or an array for multi-field sorting.',\n            ),\n          fullTextFilter: z\n            .object({\n              term: z.string().describe('Full text search term'),\n              fields: z\n                .array(z.string())\n                .optional()\n                .describe(\n                  'Fields to search within. Each entry is a dot separated path into an entity, e.g. `spec.type`.',\n                ),\n            })\n            .optional()\n            .describe('Full text search criteria'),\n          cursor: z\n            .string()\n            .optional()\n            .describe(\n              'Cursor for pagination. This can be used only after the first request with a response containing a cursor. If a cursor is given it takes precedence over `offset`.',\n            ),\n        });\n      },\n      output: z =>\n        z.object({\n          items: z\n            .array(z.object({}).passthrough())\n            .describe('List of entities'),\n          totalItems: z.number().describe('Total number of entities'),\n          hasMoreEntities: z\n            .boolean()\n            .describe('Whether more entities are available'),\n          nextPageCursor: z\n            .string()\n            .optional()\n            .describe('Next page cursor used to fetch next page of entities'),\n        }),\n    },\n    action: async ({ input, credentials }) => {\n      const response = await catalog.queryEntities(\n        {\n          ...input,\n          query: input.query,\n        },\n        { credentials },\n      );\n\n      return {\n        output: {\n          items: response.items,\n          totalItems: response.totalItems,\n          hasMoreEntities: !!response.pageInfo.nextCursor,\n          nextPageCursor: response.pageInfo.nextCursor,\n        },\n      };\n    },\n  });\n};\n"],"names":["createZodV3FilterPredicateSchema"],"mappings":";;;;AAmBA,MAAM,YAAA,GAAe;AAAA;;AAAA;;AAAA;AAAA;AAAA;;AAAA;AAAA;AAAA;AAAA;AAAA;;AAAA;AAAA;AAAA;AAAA;;AAAA;AAAA;;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;;AAAA;;AAAA;AAAA;AAAA;AAAA;AAAA,CAAA;AAsCrB,MAAM,wBAAA,GAA2B;AAAA;;AAAA;;AAAA;;AAAA;;AAAA;;AAAA;;AAAA;AAAA;;AAAA;AAAA;;AAAA;AAAA;;AAAA;AAAA;;AAAA;AAAA;;AAAA;AAAA;;AAAA;AAAA;;AAAA;AAAA;;AAAA;;AAAA;;AAAA;;AAAA;AAAA,EA4C/B,YAAY,CAAA,CAAA;AAEd,MAAM,2BAAA,GAA8B;AAAA;;AAAA;AAAA,EAIlC,YAAY,CAAA,CAAA;AAEP,MAAM,mCAAmC,CAAC;AAAA,EAC/C,OAAA;AAAA,EACA,eAAA;AAAA,EACA;AACF,CAAA,KAIM;AACJ,EAAA,eAAA,CAAgB,QAAA,CAAS;AAAA,IACvB,IAAA,EAAM,wBAAA;AAAA,IACN,KAAA,EAAO,wBAAA;AAAA,IACP,UAAA,EAAY;AAAA,MACV,WAAA,EAAa,KAAA;AAAA,MACb,QAAA,EAAU,IAAA;AAAA,MACV,UAAA,EAAY;AAAA,KACd;AAAA,IACA,WAAA,EAAa,2CACT,2BAAA,GACA,wBAAA;AAAA,IACJ,MAAA,EAAQ;AAAA,MACN,OAAO,CAAA,CAAA,KAAK;AAKV,QAAA,MAAM,iBAAA,GAAoB,CAAC,KAAA,KAAmB;AAC5C,UAAA,IAAI,OAAO,KAAA,KAAU,QAAA,IAAY,KAAA,CAAM,IAAA,OAAW,EAAA,EAAI;AACpD,YAAA,MAAM,MAAA,GAAS,OAAO,KAAK,CAAA;AAC3B,YAAA,OAAO,MAAA,CAAO,QAAA,CAAS,MAAM,CAAA,GAAI,MAAA,GAAS,KAAA;AAAA,UAC5C;AACA,UAAA,OAAO,KAAA;AAAA,QACT,CAAA;AAEA,QAAA,OAAO,EAAE,MAAA,CAAO;AAAA,UACd,KAAA,EAAOA,iDAAA,CAAiC,CAAC,CAAA,CACtC,UAAS,CACT,QAAA;AAAA,YACC;AAAA,WACF;AAAA,UACF,MAAA,EAAQ,EACL,KAAA,CAAM,CAAA,CAAE,QAAQ,CAAA,CAChB,UAAS,CACT,QAAA;AAAA,YACC;AAAA,WACF;AAAA,UACF,KAAA,EAAO,CAAA,CACJ,UAAA,CAAW,iBAAA,EAAmB,EAAE,MAAA,EAAO,CAAE,GAAA,EAAI,CAAE,UAAU,CAAA,CACzD,QAAA,EAAS,CACT,SAAS,iDAAiD,CAAA;AAAA,UAC7D,QAAQ,CAAA,CACL,UAAA,CAAW,iBAAA,EAAmB,CAAA,CAAE,QAAO,CAAE,GAAA,EAAI,CAAE,GAAA,CAAI,CAAC,CAAC,CAAA,CACrD,QAAA,EAAS,CACT,SAAS,sDAAsD,CAAA;AAAA,UAClE,WAAA,EAAa,EACV,KAAA,CAAM;AAAA,YACL,EAAE,MAAA,CAAO;AAAA,cACP,KAAA,EAAO,CAAA,CACJ,MAAA,EAAO,CACP,QAAA;AAAA,gBACC;AAAA,eACF;AAAA,cACF,KAAA,EAAO,EAAE,IAAA,CAAK,CAAC,OAAO,MAAM,CAAC,CAAA,CAAE,QAAA,CAAS,YAAY;AAAA,aACrD,CAAA;AAAA,YACD,CAAA,CAAE,KAAA;AAAA,cACA,EAAE,MAAA,CAAO;AAAA,gBACP,KAAA,EAAO,CAAA,CACJ,MAAA,EAAO,CACP,QAAA;AAAA,kBACC;AAAA,iBACF;AAAA,gBACF,KAAA,EAAO,EAAE,IAAA,CAAK,CAAC,OAAO,MAAM,CAAC,CAAA,CAAE,QAAA,CAAS,YAAY;AAAA,eACrD;AAAA;AACH,WACD,CAAA,CACA,QAAA,EAAS,CACT,QAAA;AAAA,YACC;AAAA,WACF;AAAA,UACF,cAAA,EAAgB,EACb,MAAA,CAAO;AAAA,YACN,IAAA,EAAM,CAAA,CAAE,MAAA,EAAO,CAAE,SAAS,uBAAuB,CAAA;AAAA,YACjD,MAAA,EAAQ,EACL,KAAA,CAAM,CAAA,CAAE,QAAQ,CAAA,CAChB,UAAS,CACT,QAAA;AAAA,cACC;AAAA;AACF,WACH,CAAA,CACA,QAAA,EAAS,CACT,SAAS,2BAA2B,CAAA;AAAA,UACvC,MAAA,EAAQ,CAAA,CACL,MAAA,EAAO,CACP,UAAS,CACT,QAAA;AAAA,YACC;AAAA;AACF,SACH,CAAA;AAAA,MACH,CAAA;AAAA,MACA,MAAA,EAAQ,CAAA,CAAA,KACN,CAAA,CAAE,MAAA,CAAO;AAAA,QACP,KAAA,EAAO,CAAA,CACJ,KAAA,CAAM,CAAA,CAAE,MAAA,CAAO,EAAE,CAAA,CAAE,WAAA,EAAa,CAAA,CAChC,QAAA,CAAS,kBAAkB,CAAA;AAAA,QAC9B,UAAA,EAAY,CAAA,CAAE,MAAA,EAAO,CAAE,SAAS,0BAA0B,CAAA;AAAA,QAC1D,eAAA,EAAiB,CAAA,CACd,OAAA,EAAQ,CACR,SAAS,qCAAqC,CAAA;AAAA,QACjD,gBAAgB,CAAA,CACb,MAAA,GACA,QAAA,EAAS,CACT,SAAS,sDAAsD;AAAA,OACnE;AAAA,KACL;AAAA,IACA,MAAA,EAAQ,OAAO,EAAE,KAAA,EAAO,aAAY,KAAM;AACxC,MAAA,MAAM,QAAA,GAAW,MAAM,OAAA,CAAQ,aAAA;AAAA,QAC7B;AAAA,UACE,GAAG,KAAA;AAAA,UACH,OAAO,KAAA,CAAM;AAAA,SACf;AAAA,QACA,EAAE,WAAA;AAAY,OAChB;AAEA,MAAA,OAAO;AAAA,QACL,MAAA,EAAQ;AAAA,UACN,OAAO,QAAA,CAAS,KAAA;AAAA,UAChB,YAAY,QAAA,CAAS,UAAA;AAAA,UACrB,eAAA,EAAiB,CAAC,CAAC,QAAA,CAAS,QAAA,CAAS,UAAA;AAAA,UACrC,cAAA,EAAgB,SAAS,QAAA,CAAS;AAAA;AACpC,OACF;AAAA,IACF;AAAA,GACD,CAAA;AACH;;;;"}

package/dist/database/DefaultCatalogDatabase.cjs.js

@@ -30,7 +30,7 @@ class DefaultCatalogDatabase {
     const tx = txOpaque;
     const { entityRef } = options;
     const entityRefs = new Array();
-    let currentRef = entityRef.toLocaleLowerCase("en-US");
+    let currentRef = entityRef.toLowerCase();
     for (let depth = 1; depth <= MAX_ANCESTOR_DEPTH; depth += 1) {
       const rows = await tx(
         "refresh_state_references"
@@ -57,7 +57,7 @@ class DefaultCatalogDatabase {
   async refresh(txOpaque, options) {
     const tx = txOpaque;
     const { entityRef } = options;
-    const updateResult = await tx("refresh_state").where({ entity_ref: entityRef.toLocaleLowerCase("en-US") }).update({ next_update_at: tx.fn.now() });
+    const updateResult = await tx("refresh_state").where({ entity_ref: entityRef.toLowerCase() }).update({ next_update_at: tx.fn.now() });
     if (updateResult === 0) {
       throw new errors.NotFoundError(`Failed to schedule ${entityRef} for refresh`);
     }

package/dist/database/DefaultCatalogDatabase.cjs.js.map

@@ -1 +1 @@
-{"version":3,"file":"DefaultCatalogDatabase.cjs.js","sources":["../../src/database/DefaultCatalogDatabase.ts"],"sourcesContent":["/*\n * Copyright 2020 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { NotFoundError } from '@backstage/errors';\nimport { Knex } from 'knex';\nimport {\n  CatalogDatabase,\n  ListAncestorsOptions,\n  ListAncestorsResult,\n  RefreshOptions,\n  Transaction,\n} from './types';\nimport { DbRefreshStateReferencesRow, DbRefreshStateRow } from './tables';\nimport { rethrowError } from './conversion';\nimport { LoggerService } from '@backstage/backend-plugin-api';\n\nconst MAX_ANCESTOR_DEPTH = 32;\n\nexport class DefaultCatalogDatabase implements CatalogDatabase {\n  constructor(\n    private readonly options: {\n      database: Knex;\n      logger: LoggerService;\n    },\n  ) {}\n\n  async transaction<T>(fn: (tx: Transaction) => Promise<T>): Promise<T> {\n    try {\n      let result: T | undefined = undefined;\n\n      await this.options.database.transaction(\n        async tx => {\n          // We can't return here, as knex swallows the return type in case the transaction is rolled back:\n          // https://github.com/knex/knex/blob/e37aeaa31c8ef9c1b07d2e4d3ec6607e557d800d/lib/transaction.js#L136\n          result = await fn(tx);\n        },\n        {\n          // If we explicitly trigger a rollback, don't fail.\n          doNotRejectOnRollback: true,\n        },\n      );\n\n      return result!;\n    } catch (e) {\n      this.options.logger.debug(`Error during transaction, ${e}`);\n      throw rethrowError(e);\n    }\n  }\n\n  async listAncestors(\n    txOpaque: Transaction,\n    options: ListAncestorsOptions,\n  ): Promise<ListAncestorsResult> {\n    const tx = txOpaque as Knex.Transaction;\n    const { entityRef } = options;\n    const entityRefs = new Array<string>();\n\n    let currentRef = entityRef.toLocaleLowerCase('en-US');\n    for (let depth = 1; depth <= MAX_ANCESTOR_DEPTH; depth += 1) {\n      const rows = await tx<DbRefreshStateReferencesRow>(\n        'refresh_state_references',\n      )\n        .where({ target_entity_ref: currentRef })\n        .select();\n\n      if (rows.length === 0) {\n        if (depth === 1) {\n          throw new NotFoundError(`Entity ${currentRef} not found`);\n        }\n        throw new NotFoundError(\n          `Entity ${entityRef} has a broken parent reference chain at ${currentRef}`,\n        );\n      }\n\n      const parentRef = rows.find(r => r.source_entity_ref)?.source_entity_ref;\n      if (!parentRef) {\n        // We've reached the top of the tree which is the entityProvider.\n        // In this case we refresh the entity itself.\n        return { entityRefs };\n      }\n      entityRefs.push(parentRef);\n      currentRef = parentRef;\n    }\n    throw new Error(\n      `Unable receive ancestors for ${entityRef}, reached maximum depth of ${MAX_ANCESTOR_DEPTH}`,\n    );\n  }\n\n  async refresh(txOpaque: Transaction, options: RefreshOptions): Promise<void> {\n    const tx = txOpaque as Knex.Transaction;\n    const { entityRef } = options;\n\n    const updateResult = await tx<DbRefreshStateRow>('refresh_state')\n      .where({ entity_ref: entityRef.toLocaleLowerCase('en-US') })\n      .update({ next_update_at: tx.fn.now() });\n    if (updateResult === 0) {\n      throw new NotFoundError(`Failed to schedule ${entityRef} for refresh`);\n    }\n  }\n}\n"],"names":["rethrowError","NotFoundError"],"mappings":";;;;;AA6BA,MAAM,kBAAA,GAAqB,EAAA;AAEpB,MAAM,sBAAA,CAAkD;AAAA,EAC7D,YACmB,OAAA,EAIjB;AAJiB,IAAA,IAAA,CAAA,OAAA,GAAA,OAAA;AAAA,EAIhB;AAAA,EAEH,MAAM,YAAe,EAAA,EAAiD;AACpE,IAAA,IAAI;AACF,MAAA,IAAI,MAAA,GAAwB,KAAA,CAAA;AAE5B,MAAA,MAAM,IAAA,CAAK,QAAQ,QAAA,CAAS,WAAA;AAAA,QAC1B,OAAM,EAAA,KAAM;AAGV,UAAA,MAAA,GAAS,MAAM,GAAG,EAAE,CAAA;AAAA,QACtB,CAAA;AAAA,QACA;AAAA;AAAA,UAEE,qBAAA,EAAuB;AAAA;AACzB,OACF;AAEA,MAAA,OAAO,MAAA;AAAA,IACT,SAAS,CAAA,EAAG;AACV,MAAA,IAAA,CAAK,OAAA,CAAQ,MAAA,CAAO,KAAA,CAAM,CAAA,0BAAA,EAA6B,CAAC,CAAA,CAAE,CAAA;AAC1D,MAAA,MAAMA,wBAAa,CAAC,CAAA;AAAA,IACtB;AAAA,EACF;AAAA,EAEA,MAAM,aAAA,CACJ,QAAA,EACA,OAAA,EAC8B;AAC9B,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAM,EAAE,WAAU,GAAI,OAAA;AACtB,IAAA,MAAM,UAAA,GAAa,IAAI,KAAA,EAAc;AAErC,IAAA,IAAI,UAAA,GAAa,SAAA,CAAU,iBAAA,CAAkB,OAAO,CAAA;AACpD,IAAA,KAAA,IAAS,KAAA,GAAQ,CAAA,EAAG,KAAA,IAAS,kBAAA,EAAoB,SAAS,CAAA,EAAG;AAC3D,MAAA,MAAM,OAAO,MAAM,EAAA;AAAA,QACjB;AAAA,QAEC,KAAA,CAAM,EAAE,mBAAmB,UAAA,EAAY,EACvC,MAAA,EAAO;AAEV,MAAA,IAAI,IAAA,CAAK,WAAW,CAAA,EAAG;AACrB,QAAA,IAAI,UAAU,CAAA,EAAG;AACf,UAAA,MAAM,IAAIC,oBAAA,CAAc,CAAA,OAAA,EAAU,UAAU,CAAA,UAAA,CAAY,CAAA;AAAA,QAC1D;AACA,QAAA,MAAM,IAAIA,oBAAA;AAAA,UACR,CAAA,OAAA,EAAU,SAAS,CAAA,wCAAA,EAA2C,UAAU,CAAA;AAAA,SAC1E;AAAA,MACF;AAEA,MAAA,MAAM,YAAY,IAAA,CAAK,IAAA,CAAK,CAAA,CAAA,KAAK,CAAA,CAAE,iBAAiB,CAAA,EAAG,iBAAA;AACvD,MAAA,IAAI,CAAC,SAAA,EAAW;AAGd,QAAA,OAAO,EAAE,UAAA,EAAW;AAAA,MACtB;AACA,MAAA,UAAA,CAAW,KAAK,SAAS,CAAA;AACzB,MAAA,UAAA,GAAa,SAAA;AAAA,IACf;AACA,IAAA,MAAM,IAAI,KAAA;AAAA,MACR,CAAA,6BAAA,EAAgC,SAAS,CAAA,2BAAA,EAA8B,kBAAkB,CAAA;AAAA,KAC3F;AAAA,EACF;AAAA,EAEA,MAAM,OAAA,CAAQ,QAAA,EAAuB,OAAA,EAAwC;AAC3E,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAM,EAAE,WAAU,GAAI,OAAA;AAEtB,IAAA,MAAM,YAAA,GAAe,MAAM,EAAA,CAAsB,eAAe,EAC7D,KAAA,CAAM,EAAE,YAAY,SAAA,CAAU,iBAAA,CAAkB,OAAO,CAAA,EAAG,EAC1D,MAAA,CAAO,EAAE,gBAAgB,EAAA,CAAG,EAAA,CAAG,GAAA,EAAI,EAAG,CAAA;AACzC,IAAA,IAAI,iBAAiB,CAAA,EAAG;AACtB,MAAA,MAAM,IAAIA,oBAAA,CAAc,CAAA,mBAAA,EAAsB,SAAS,CAAA,YAAA,CAAc,CAAA;AAAA,IACvE;AAAA,EACF;AACF;;;;"}
+{"version":3,"file":"DefaultCatalogDatabase.cjs.js","sources":["../../src/database/DefaultCatalogDatabase.ts"],"sourcesContent":["/*\n * Copyright 2020 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { NotFoundError } from '@backstage/errors';\nimport { Knex } from 'knex';\nimport {\n  CatalogDatabase,\n  ListAncestorsOptions,\n  ListAncestorsResult,\n  RefreshOptions,\n  Transaction,\n} from './types';\nimport { DbRefreshStateReferencesRow, DbRefreshStateRow } from './tables';\nimport { rethrowError } from './conversion';\nimport { LoggerService } from '@backstage/backend-plugin-api';\n\nconst MAX_ANCESTOR_DEPTH = 32;\n\nexport class DefaultCatalogDatabase implements CatalogDatabase {\n  constructor(\n    private readonly options: {\n      database: Knex;\n      logger: LoggerService;\n    },\n  ) {}\n\n  async transaction<T>(fn: (tx: Transaction) => Promise<T>): Promise<T> {\n    try {\n      let result: T | undefined = undefined;\n\n      await this.options.database.transaction(\n        async tx => {\n          // We can't return here, as knex swallows the return type in case the transaction is rolled back:\n          // https://github.com/knex/knex/blob/e37aeaa31c8ef9c1b07d2e4d3ec6607e557d800d/lib/transaction.js#L136\n          result = await fn(tx);\n        },\n        {\n          // If we explicitly trigger a rollback, don't fail.\n          doNotRejectOnRollback: true,\n        },\n      );\n\n      return result!;\n    } catch (e) {\n      this.options.logger.debug(`Error during transaction, ${e}`);\n      throw rethrowError(e);\n    }\n  }\n\n  async listAncestors(\n    txOpaque: Transaction,\n    options: ListAncestorsOptions,\n  ): Promise<ListAncestorsResult> {\n    const tx = txOpaque as Knex.Transaction;\n    const { entityRef } = options;\n    const entityRefs = new Array<string>();\n\n    let currentRef = entityRef.toLowerCase();\n    for (let depth = 1; depth <= MAX_ANCESTOR_DEPTH; depth += 1) {\n      const rows = await tx<DbRefreshStateReferencesRow>(\n        'refresh_state_references',\n      )\n        .where({ target_entity_ref: currentRef })\n        .select();\n\n      if (rows.length === 0) {\n        if (depth === 1) {\n          throw new NotFoundError(`Entity ${currentRef} not found`);\n        }\n        throw new NotFoundError(\n          `Entity ${entityRef} has a broken parent reference chain at ${currentRef}`,\n        );\n      }\n\n      const parentRef = rows.find(r => r.source_entity_ref)?.source_entity_ref;\n      if (!parentRef) {\n        // We've reached the top of the tree which is the entityProvider.\n        // In this case we refresh the entity itself.\n        return { entityRefs };\n      }\n      entityRefs.push(parentRef);\n      currentRef = parentRef;\n    }\n    throw new Error(\n      `Unable receive ancestors for ${entityRef}, reached maximum depth of ${MAX_ANCESTOR_DEPTH}`,\n    );\n  }\n\n  async refresh(txOpaque: Transaction, options: RefreshOptions): Promise<void> {\n    const tx = txOpaque as Knex.Transaction;\n    const { entityRef } = options;\n\n    const updateResult = await tx<DbRefreshStateRow>('refresh_state')\n      .where({ entity_ref: entityRef.toLowerCase() })\n      .update({ next_update_at: tx.fn.now() });\n    if (updateResult === 0) {\n      throw new NotFoundError(`Failed to schedule ${entityRef} for refresh`);\n    }\n  }\n}\n"],"names":["rethrowError","NotFoundError"],"mappings":";;;;;AA6BA,MAAM,kBAAA,GAAqB,EAAA;AAEpB,MAAM,sBAAA,CAAkD;AAAA,EAC7D,YACmB,OAAA,EAIjB;AAJiB,IAAA,IAAA,CAAA,OAAA,GAAA,OAAA;AAAA,EAIhB;AAAA,EAEH,MAAM,YAAe,EAAA,EAAiD;AACpE,IAAA,IAAI;AACF,MAAA,IAAI,MAAA,GAAwB,KAAA,CAAA;AAE5B,MAAA,MAAM,IAAA,CAAK,QAAQ,QAAA,CAAS,WAAA;AAAA,QAC1B,OAAM,EAAA,KAAM;AAGV,UAAA,MAAA,GAAS,MAAM,GAAG,EAAE,CAAA;AAAA,QACtB,CAAA;AAAA,QACA;AAAA;AAAA,UAEE,qBAAA,EAAuB;AAAA;AACzB,OACF;AAEA,MAAA,OAAO,MAAA;AAAA,IACT,SAAS,CAAA,EAAG;AACV,MAAA,IAAA,CAAK,OAAA,CAAQ,MAAA,CAAO,KAAA,CAAM,CAAA,0BAAA,EAA6B,CAAC,CAAA,CAAE,CAAA;AAC1D,MAAA,MAAMA,wBAAa,CAAC,CAAA;AAAA,IACtB;AAAA,EACF;AAAA,EAEA,MAAM,aAAA,CACJ,QAAA,EACA,OAAA,EAC8B;AAC9B,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAM,EAAE,WAAU,GAAI,OAAA;AACtB,IAAA,MAAM,UAAA,GAAa,IAAI,KAAA,EAAc;AAErC,IAAA,IAAI,UAAA,GAAa,UAAU,WAAA,EAAY;AACvC,IAAA,KAAA,IAAS,KAAA,GAAQ,CAAA,EAAG,KAAA,IAAS,kBAAA,EAAoB,SAAS,CAAA,EAAG;AAC3D,MAAA,MAAM,OAAO,MAAM,EAAA;AAAA,QACjB;AAAA,QAEC,KAAA,CAAM,EAAE,mBAAmB,UAAA,EAAY,EACvC,MAAA,EAAO;AAEV,MAAA,IAAI,IAAA,CAAK,WAAW,CAAA,EAAG;AACrB,QAAA,IAAI,UAAU,CAAA,EAAG;AACf,UAAA,MAAM,IAAIC,oBAAA,CAAc,CAAA,OAAA,EAAU,UAAU,CAAA,UAAA,CAAY,CAAA;AAAA,QAC1D;AACA,QAAA,MAAM,IAAIA,oBAAA;AAAA,UACR,CAAA,OAAA,EAAU,SAAS,CAAA,wCAAA,EAA2C,UAAU,CAAA;AAAA,SAC1E;AAAA,MACF;AAEA,MAAA,MAAM,YAAY,IAAA,CAAK,IAAA,CAAK,CAAA,CAAA,KAAK,CAAA,CAAE,iBAAiB,CAAA,EAAG,iBAAA;AACvD,MAAA,IAAI,CAAC,SAAA,EAAW;AAGd,QAAA,OAAO,EAAE,UAAA,EAAW;AAAA,MACtB;AACA,MAAA,UAAA,CAAW,KAAK,SAAS,CAAA;AACzB,MAAA,UAAA,GAAa,SAAA;AAAA,IACf;AACA,IAAA,MAAM,IAAI,KAAA;AAAA,MACR,CAAA,6BAAA,EAAgC,SAAS,CAAA,2BAAA,EAA8B,kBAAkB,CAAA;AAAA,KAC3F;AAAA,EACF;AAAA,EAEA,MAAM,OAAA,CAAQ,QAAA,EAAuB,OAAA,EAAwC;AAC3E,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAM,EAAE,WAAU,GAAI,OAAA;AAEtB,IAAA,MAAM,YAAA,GAAe,MAAM,EAAA,CAAsB,eAAe,EAC7D,KAAA,CAAM,EAAE,YAAY,SAAA,CAAU,WAAA,IAAe,CAAA,CAC7C,OAAO,EAAE,cAAA,EAAgB,GAAG,EAAA,CAAG,GAAA,IAAO,CAAA;AACzC,IAAA,IAAI,iBAAiB,CAAA,EAAG;AACtB,MAAA,MAAM,IAAIA,oBAAA,CAAc,CAAA,mBAAA,EAAsB,SAAS,CAAA,YAAA,CAAc,CAAA;AAAA,IACvE;AAAA,EACF;AACF;;;;"}

package/dist/database/DefaultProcessingDatabase.cjs.js

@@ -117,9 +117,11 @@ class DefaultProcessingDatabase {
         }
       });
       if (items2.length > 0) {
-        await tx("refresh_state").whereIn(
-          "entity_ref",
-          items2.map((i) => i.entity_ref)
+        await tx("refresh_state").where(
+          util.whereInArray(
+            "entity_ref",
+            items2.map((i) => i.entity_ref)
+          )
         ).update({
           next_update_at: nextUpdateAt(tx, interval)
         });

package/dist/database/DefaultProcessingDatabase.cjs.js.map

@@ -1 +1 @@
-{"version":3,"file":"DefaultProcessingDatabase.cjs.js","sources":["../../src/database/DefaultProcessingDatabase.ts"],"sourcesContent":["/*\n * Copyright 2021 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { Entity, stringifyEntityRef } from '@backstage/catalog-model';\nimport { ConflictError } from '@backstage/errors';\nimport { DeferredEntity } from '@backstage/plugin-catalog-node';\nimport { Knex } from 'knex';\nimport { ProcessingIntervalFunction } from '../processing/refresh';\nimport { rethrowError, timestampToDateTime } from './conversion';\nimport { initDatabaseMetrics } from './metrics';\nimport {\n  DbRefreshKeysRow,\n  DbRefreshStateReferencesRow,\n  DbRefreshStateRow,\n  DbRelationsRow,\n} from './tables';\nimport {\n  GetProcessableEntitiesResult,\n  ListParentsOptions,\n  ListParentsResult,\n  ProcessingDatabase,\n  RefreshStateItem,\n  Transaction,\n  UpdateEntityCacheOptions,\n  UpdateProcessedEntityOptions,\n} from './types';\nimport { checkLocationKeyConflict } from './operations/refreshState/checkLocationKeyConflict';\nimport { insertUnprocessedEntity } from './operations/refreshState/insertUnprocessedEntity';\nimport { updateUnprocessedEntity } from './operations/refreshState/updateUnprocessedEntity';\nimport { generateStableHash, generateTargetKey } from './util';\nimport {\n  syncRelations,\n  SyncRelationsResult,\n} from './operations/relations/syncRelations';\nimport { EventParams, EventsService } from '@backstage/plugin-events-node';\nimport { DateTime } from 'luxon';\nimport { CATALOG_CONFLICTS_TOPIC } from '../constants';\nimport { CatalogConflictEventPayload } from '../catalog/types';\nimport { LoggerService } from '@backstage/backend-plugin-api';\nimport { MetricsService } from '@backstage/backend-plugin-api/alpha';\n\n// The number of items that are sent per batch to the database layer, when\n// doing .batchInsert calls to knex. This needs to be low enough to not cause\n// errors in the underlying engine due to exceeding query limits, but large\n// enough to get the speed benefits.\nconst BATCH_SIZE = 50;\n\nexport class DefaultProcessingDatabase implements ProcessingDatabase {\n  private readonly options: {\n    database: Knex;\n    logger: LoggerService;\n    refreshInterval: ProcessingIntervalFunction;\n    events: EventsService;\n    metrics: MetricsService;\n  };\n\n  constructor(options: {\n    database: Knex;\n    logger: LoggerService;\n    refreshInterval: ProcessingIntervalFunction;\n    events: EventsService;\n    metrics: MetricsService;\n  }) {\n    this.options = options;\n    initDatabaseMetrics(options.database, options.metrics);\n  }\n\n  async updateProcessedEntity(\n    txOpaque: Transaction,\n    options: UpdateProcessedEntityOptions,\n  ): Promise<{ relationsChange: SyncRelationsResult }> {\n    const tx = txOpaque as Knex.Transaction;\n    const {\n      id,\n      processedEntity,\n      resultHash,\n      errors,\n      relations,\n      deferredEntities,\n      refreshKeys,\n      locationKey,\n    } = options;\n    const refreshResult = await tx<DbRefreshStateRow>('refresh_state')\n      .update({\n        processed_entity: JSON.stringify(processedEntity),\n        result_hash: resultHash,\n        errors,\n        location_key: locationKey,\n      })\n      .where('entity_id', id)\n      .andWhere(inner => {\n        if (!locationKey) {\n          return inner.whereNull('location_key');\n        }\n        return inner\n          .where('location_key', locationKey)\n          .orWhereNull('location_key');\n      });\n    if (refreshResult === 0) {\n      throw new ConflictError(\n        `Conflicting write of processing result for ${id} with location key '${locationKey}'`,\n      );\n    }\n    const sourceEntityRef = stringifyEntityRef(processedEntity);\n\n    // Schedule all deferred entities for future processing.\n    await this.addUnprocessedEntities(tx, {\n      entities: deferredEntities,\n      sourceEntityRef,\n    });\n\n    // Sync relations using a diff-based approach — only touches rows that\n    // actually changed, eliminating steady-state write churn.\n    const relationRows: DbRelationsRow[] = relations.map(\n      ({ source, target, type }) => ({\n        originating_entity_id: id,\n        source_entity_ref: stringifyEntityRef(source),\n        target_entity_ref: stringifyEntityRef(target),\n        type,\n      }),\n    );\n\n    const relationsResult = await syncRelations(tx, id, relationRows);\n\n    // Delete old refresh keys\n    await tx<DbRefreshKeysRow>('refresh_keys')\n      .where({ entity_id: id })\n      .delete();\n\n    // Insert the refresh keys for the processed entity\n    await tx.batchInsert(\n      'refresh_keys',\n      refreshKeys.map(k => ({\n        entity_id: id,\n        key: generateTargetKey(k.key),\n      })),\n      BATCH_SIZE,\n    );\n\n    return {\n      relationsChange: relationsResult,\n    };\n  }\n\n  async updateProcessedEntityErrors(\n    txOpaque: Transaction,\n    options: UpdateProcessedEntityOptions,\n  ): Promise<void> {\n    const tx = txOpaque as Knex.Transaction;\n    const { id, errors, resultHash } = options;\n\n    await tx<DbRefreshStateRow>('refresh_state')\n      .update({\n        errors,\n        result_hash: resultHash,\n      })\n      .where('entity_id', id);\n  }\n\n  async updateEntityCache(\n    txOpaque: Transaction,\n    options: UpdateEntityCacheOptions,\n  ): Promise<void> {\n    const tx = txOpaque as Knex.Transaction;\n    const { id, state } = options;\n\n    await tx<DbRefreshStateRow>('refresh_state')\n      .update({ cache: JSON.stringify(state ?? {}) })\n      .where('entity_id', id);\n  }\n\n  async getProcessableEntities(\n    maybeTx: Transaction | Knex,\n    request: { processBatchSize: number },\n  ): Promise<GetProcessableEntitiesResult> {\n    const knex = maybeTx as Knex.Transaction | Knex;\n    const useLocking = ['mysql', 'mysql2', 'pg'].includes(\n      knex.client.config.client,\n    );\n\n    const interval = this.options.refreshInterval();\n\n    const nextUpdateAt = (\n      tx: Knex | Knex.Transaction,\n      refreshInterval: number,\n    ) => {\n      if (tx.client.config.client.includes('sqlite3')) {\n        return tx.raw(`datetime('now', ?)`, [`${refreshInterval} seconds`]);\n      } else if (tx.client.config.client.includes('mysql')) {\n        return tx.raw(`now() + interval ${refreshInterval} second`);\n      }\n      return tx.raw(`now() + interval '${refreshInterval} seconds'`);\n    };\n\n    // The SELECT FOR UPDATE SKIP LOCKED + UPDATE must run inside a\n    // single transaction so that the row locks persist until\n    // next_update_at has been bumped.\n    const run = async (tx: Knex | Knex.Transaction) => {\n      const items: DbRefreshStateRow[] = await tx('refresh_state')\n        .select([\n          'entity_id',\n          'entity_ref',\n          'unprocessed_entity',\n          'result_hash',\n          'cache',\n          'errors',\n          'location_key',\n          'next_update_at',\n        ])\n        .where('next_update_at', '<=', tx.fn.now())\n        .limit(request.processBatchSize)\n        .orderBy('next_update_at', 'asc')\n        .modify(qb => {\n          if (useLocking) {\n            qb.forUpdate().skipLocked();\n          }\n        });\n\n      if (items.length > 0) {\n        await tx('refresh_state')\n          .whereIn(\n            'entity_ref',\n            items.map(i => i.entity_ref),\n          )\n          .update({\n            next_update_at: nextUpdateAt(tx, interval),\n          });\n      }\n\n      return items;\n    };\n\n    const items =\n      knex.isTransaction || !useLocking\n        ? await run(knex)\n        : await knex.transaction(run);\n\n    return {\n      items: items.map(\n        i =>\n          ({\n            id: i.entity_id,\n            entityRef: i.entity_ref,\n            unprocessedEntity: JSON.parse(i.unprocessed_entity) as Entity,\n            resultHash: i.result_hash || '',\n            nextUpdateAt: timestampToDateTime(i.next_update_at),\n            state: i.cache ? JSON.parse(i.cache) : undefined,\n            errors: i.errors,\n            locationKey: i.location_key,\n          } satisfies RefreshStateItem),\n      ),\n    };\n  }\n\n  async listParents(\n    txOpaque: Transaction,\n    options: ListParentsOptions,\n  ): Promise<ListParentsResult> {\n    const tx = txOpaque as Knex.Transaction;\n\n    const rows = await tx<DbRefreshStateReferencesRow>(\n      'refresh_state_references',\n    )\n      .whereIn('target_entity_ref', options.entityRefs)\n      .select();\n\n    const entityRefs = rows.map(r => r.source_entity_ref!).filter(Boolean);\n\n    return { entityRefs };\n  }\n\n  async transaction<T>(fn: (tx: Transaction) => Promise<T>): Promise<T> {\n    try {\n      let result: T | undefined = undefined;\n\n      await this.options.database.transaction(\n        async tx => {\n          // We can't return here, as knex swallows the return type in case the transaction is rolled back:\n          // https://github.com/knex/knex/blob/e37aeaa31c8ef9c1b07d2e4d3ec6607e557d800d/lib/transaction.js#L136\n          result = await fn(tx);\n        },\n        {\n          // If we explicitly trigger a rollback, don't fail.\n          doNotRejectOnRollback: true,\n        },\n      );\n\n      return result!;\n    } catch (e) {\n      this.options.logger.debug(`Error during transaction, ${e}`);\n      throw rethrowError(e);\n    }\n  }\n\n  /**\n   * Add a set of deferred entities for processing.\n   * The entities will be added at the front of the processing queue.\n   */\n  private async addUnprocessedEntities(\n    txOpaque: Transaction,\n    options: {\n      sourceEntityRef: string;\n      entities: DeferredEntity[];\n    },\n  ): Promise<void> {\n    const tx = txOpaque as Knex.Transaction;\n\n    // Keeps track of the entities that we end up inserting to update refresh_state_references afterwards\n    const stateReferences = new Array<string>();\n\n    // Upsert all of the unprocessed entities into the refresh_state table, by\n    // their entity ref.\n    for (const { entity, locationKey } of options.entities) {\n      const entityRef = stringifyEntityRef(entity);\n      const hash = generateStableHash(entity);\n\n      const updated = await updateUnprocessedEntity({\n        tx,\n        entity,\n        hash,\n        locationKey,\n      });\n      if (updated) {\n        stateReferences.push(entityRef);\n        continue;\n      }\n\n      const inserted = await insertUnprocessedEntity({\n        tx,\n        entity,\n        hash,\n        locationKey,\n        logger: this.options.logger,\n      });\n      if (inserted) {\n        stateReferences.push(entityRef);\n        continue;\n      }\n\n      // If the row can't be inserted, we have a conflict, but it could be either\n      // because of a conflicting locationKey or a race with another instance, so check\n      // whether the conflicting entity has the same entityRef but a different locationKey\n      const conflictingKey = await checkLocationKeyConflict({\n        tx,\n        entityRef,\n        locationKey,\n      });\n      if (conflictingKey) {\n        this.options.logger.warn(\n          `Detected conflicting entityRef ${entityRef} already referenced by ${conflictingKey} and now also ${locationKey}`,\n        );\n        if (locationKey) {\n          const eventParams: EventParams<CatalogConflictEventPayload> = {\n            topic: CATALOG_CONFLICTS_TOPIC,\n            eventPayload: {\n              unprocessedEntity: entity,\n              entityRef,\n              newLocationKey: locationKey,\n              existingLocationKey: conflictingKey,\n              lastConflictAt: DateTime.now().toISO()!,\n            },\n          };\n          await this.options.events.publish(eventParams);\n        }\n      }\n    }\n\n    // Lastly, replace refresh state references for the originating entity and any successfully added entities\n    await tx<DbRefreshStateReferencesRow>('refresh_state_references')\n      // Remove all existing references from the originating entity\n      .where({ source_entity_ref: options.sourceEntityRef })\n      // And remove any existing references to entities that we're inserting new references for\n      .orWhereIn('target_entity_ref', stateReferences)\n      .delete();\n    await tx.batchInsert(\n      'refresh_state_references',\n      stateReferences.map(entityRef => ({\n        source_entity_ref: options.sourceEntityRef,\n        target_entity_ref: entityRef,\n      })),\n      BATCH_SIZE,\n    );\n  }\n}\n"],"names":["initDatabaseMetrics","errors","ConflictError","stringifyEntityRef","syncRelations","generateTargetKey","items","timestampToDateTime","rethrowError","generateStableHash","updateUnprocessedEntity","insertUnprocessedEntity","checkLocationKeyConflict","CATALOG_CONFLICTS_TOPIC","DateTime"],"mappings":";;;;;;;;;;;;;;AA0DA,MAAM,UAAA,GAAa,EAAA;AAEZ,MAAM,yBAAA,CAAwD;AAAA,EAClD,OAAA;AAAA,EAQjB,YAAY,OAAA,EAMT;AACD,IAAA,IAAA,CAAK,OAAA,GAAU,OAAA;AACf,IAAAA,2BAAA,CAAoB,OAAA,CAAQ,QAAA,EAAU,OAAA,CAAQ,OAAO,CAAA;AAAA,EACvD;AAAA,EAEA,MAAM,qBAAA,CACJ,QAAA,EACA,OAAA,EACmD;AACnD,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAM;AAAA,MACJ,EAAA;AAAA,MACA,eAAA;AAAA,MACA,UAAA;AAAA,cACAC,QAAA;AAAA,MACA,SAAA;AAAA,MACA,gBAAA;AAAA,MACA,WAAA;AAAA,MACA;AAAA,KACF,GAAI,OAAA;AACJ,IAAA,MAAM,aAAA,GAAgB,MAAM,EAAA,CAAsB,eAAe,EAC9D,MAAA,CAAO;AAAA,MACN,gBAAA,EAAkB,IAAA,CAAK,SAAA,CAAU,eAAe,CAAA;AAAA,MAChD,WAAA,EAAa,UAAA;AAAA,cACbA,QAAA;AAAA,MACA,YAAA,EAAc;AAAA,KACf,CAAA,CACA,KAAA,CAAM,aAAa,EAAE,CAAA,CACrB,SAAS,CAAA,KAAA,KAAS;AACjB,MAAA,IAAI,CAAC,WAAA,EAAa;AAChB,QAAA,OAAO,KAAA,CAAM,UAAU,cAAc,CAAA;AAAA,MACvC;AACA,MAAA,OAAO,MACJ,KAAA,CAAM,cAAA,EAAgB,WAAW,CAAA,CACjC,YAAY,cAAc,CAAA;AAAA,IAC/B,CAAC,CAAA;AACH,IAAA,IAAI,kBAAkB,CAAA,EAAG;AACvB,MAAA,MAAM,IAAIC,oBAAA;AAAA,QACR,CAAA,2CAAA,EAA8C,EAAE,CAAA,oBAAA,EAAuB,WAAW,CAAA,CAAA;AAAA,OACpF;AAAA,IACF;AACA,IAAA,MAAM,eAAA,GAAkBC,gCAAmB,eAAe,CAAA;AAG1D,IAAA,MAAM,IAAA,CAAK,uBAAuB,EAAA,EAAI;AAAA,MACpC,QAAA,EAAU,gBAAA;AAAA,MACV;AAAA,KACD,CAAA;AAID,IAAA,MAAM,eAAiC,SAAA,CAAU,GAAA;AAAA,MAC/C,CAAC,EAAE,MAAA,EAAQ,MAAA,EAAQ,MAAK,MAAO;AAAA,QAC7B,qBAAA,EAAuB,EAAA;AAAA,QACvB,iBAAA,EAAmBA,gCAAmB,MAAM,CAAA;AAAA,QAC5C,iBAAA,EAAmBA,gCAAmB,MAAM,CAAA;AAAA,QAC5C;AAAA,OACF;AAAA,KACF;AAEA,IAAA,MAAM,eAAA,GAAkB,MAAMC,2BAAA,CAAc,EAAA,EAAI,IAAI,YAAY,CAAA;AAGhE,IAAA,MAAM,EAAA,CAAqB,cAAc,CAAA,CACtC,KAAA,CAAM,EAAE,SAAA,EAAW,EAAA,EAAI,CAAA,CACvB,MAAA,EAAO;AAGV,IAAA,MAAM,EAAA,CAAG,WAAA;AAAA,MACP,cAAA;AAAA,MACA,WAAA,CAAY,IAAI,CAAA,CAAA,MAAM;AAAA,QACpB,SAAA,EAAW,EAAA;AAAA,QACX,GAAA,EAAKC,sBAAA,CAAkB,CAAA,CAAE,GAAG;AAAA,OAC9B,CAAE,CAAA;AAAA,MACF;AAAA,KACF;AAEA,IAAA,OAAO;AAAA,MACL,eAAA,EAAiB;AAAA,KACnB;AAAA,EACF;AAAA,EAEA,MAAM,2BAAA,CACJ,QAAA,EACA,OAAA,EACe;AACf,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAM,EAAE,EAAA,EAAI,MAAA,EAAQ,UAAA,EAAW,GAAI,OAAA;AAEnC,IAAA,MAAM,EAAA,CAAsB,eAAe,CAAA,CACxC,MAAA,CAAO;AAAA,MACN,MAAA;AAAA,MACA,WAAA,EAAa;AAAA,KACd,CAAA,CACA,KAAA,CAAM,WAAA,EAAa,EAAE,CAAA;AAAA,EAC1B;AAAA,EAEA,MAAM,iBAAA,CACJ,QAAA,EACA,OAAA,EACe;AACf,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAM,EAAE,EAAA,EAAI,KAAA,EAAM,GAAI,OAAA;AAEtB,IAAA,MAAM,GAAsB,eAAe,CAAA,CACxC,MAAA,CAAO,EAAE,OAAO,IAAA,CAAK,SAAA,CAAU,KAAA,IAAS,EAAE,CAAA,EAAG,CAAA,CAC7C,KAAA,CAAM,aAAa,EAAE,CAAA;AAAA,EAC1B;AAAA,EAEA,MAAM,sBAAA,CACJ,OAAA,EACA,OAAA,EACuC;AACvC,IAAA,MAAM,IAAA,GAAO,OAAA;AACb,IAAA,MAAM,UAAA,GAAa,CAAC,OAAA,EAAS,QAAA,EAAU,IAAI,CAAA,CAAE,QAAA;AAAA,MAC3C,IAAA,CAAK,OAAO,MAAA,CAAO;AAAA,KACrB;AAEA,IAAA,MAAM,QAAA,GAAW,IAAA,CAAK,OAAA,CAAQ,eAAA,EAAgB;AAE9C,IAAA,MAAM,YAAA,GAAe,CACnB,EAAA,EACA,eAAA,KACG;AACH,MAAA,IAAI,GAAG,MAAA,CAAO,MAAA,CAAO,MAAA,CAAO,QAAA,CAAS,SAAS,CAAA,EAAG;AAC/C,QAAA,OAAO,GAAG,GAAA,CAAI,CAAA,kBAAA,CAAA,EAAsB,CAAC,CAAA,EAAG,eAAe,UAAU,CAAC,CAAA;AAAA,MACpE,WAAW,EAAA,CAAG,MAAA,CAAO,OAAO,MAAA,CAAO,QAAA,CAAS,OAAO,CAAA,EAAG;AACpD,QAAA,OAAO,EAAA,CAAG,GAAA,CAAI,CAAA,iBAAA,EAAoB,eAAe,CAAA,OAAA,CAAS,CAAA;AAAA,MAC5D;AACA,MAAA,OAAO,EAAA,CAAG,GAAA,CAAI,CAAA,kBAAA,EAAqB,eAAe,CAAA,SAAA,CAAW,CAAA;AAAA,IAC/D,CAAA;AAKA,IAAA,MAAM,GAAA,GAAM,OAAO,EAAA,KAAgC;AACjD,MAAA,MAAMC,MAAAA,GAA6B,MAAM,EAAA,CAAG,eAAe,EACxD,MAAA,CAAO;AAAA,QACN,WAAA;AAAA,QACA,YAAA;AAAA,QACA,oBAAA;AAAA,QACA,aAAA;AAAA,QACA,OAAA;AAAA,QACA,QAAA;AAAA,QACA,cAAA;AAAA,QACA;AAAA,OACD,CAAA,CACA,KAAA,CAAM,kBAAkB,IAAA,EAAM,EAAA,CAAG,GAAG,GAAA,EAAK,EACzC,KAAA,CAAM,OAAA,CAAQ,gBAAgB,CAAA,CAC9B,OAAA,CAAQ,kBAAkB,KAAK,CAAA,CAC/B,OAAO,CAAA,EAAA,KAAM;AACZ,QAAA,IAAI,UAAA,EAAY;AACd,UAAA,EAAA,CAAG,SAAA,GAAY,UAAA,EAAW;AAAA,QAC5B;AAAA,MACF,CAAC,CAAA;AAEH,MAAA,IAAIA,MAAAA,CAAM,SAAS,CAAA,EAAG;AACpB,QAAA,MAAM,EAAA,CAAG,eAAe,CAAA,CACrB,OAAA;AAAA,UACC,YAAA;AAAA,UACAA,MAAAA,CAAM,GAAA,CAAI,CAAA,CAAA,KAAK,CAAA,CAAE,UAAU;AAAA,UAE5B,MAAA,CAAO;AAAA,UACN,cAAA,EAAgB,YAAA,CAAa,EAAA,EAAI,QAAQ;AAAA,SAC1C,CAAA;AAAA,MACL;AAEA,MAAA,OAAOA,MAAAA;AAAA,IACT,CAAA;AAEA,IAAA,MAAM,KAAA,GACJ,IAAA,CAAK,aAAA,IAAiB,CAAC,UAAA,GACnB,MAAM,GAAA,CAAI,IAAI,CAAA,GACd,MAAM,IAAA,CAAK,WAAA,CAAY,GAAG,CAAA;AAEhC,IAAA,OAAO;AAAA,MACL,OAAO,KAAA,CAAM,GAAA;AAAA,QACX,CAAA,CAAA,MACG;AAAA,UACC,IAAI,CAAA,CAAE,SAAA;AAAA,UACN,WAAW,CAAA,CAAE,UAAA;AAAA,UACb,iBAAA,EAAmB,IAAA,CAAK,KAAA,CAAM,CAAA,CAAE,kBAAkB,CAAA;AAAA,UAClD,UAAA,EAAY,EAAE,WAAA,IAAe,EAAA;AAAA,UAC7B,YAAA,EAAcC,8BAAA,CAAoB,CAAA,CAAE,cAAc,CAAA;AAAA,UAClD,OAAO,CAAA,CAAE,KAAA,GAAQ,KAAK,KAAA,CAAM,CAAA,CAAE,KAAK,CAAA,GAAI,MAAA;AAAA,UACvC,QAAQ,CAAA,CAAE,MAAA;AAAA,UACV,aAAa,CAAA,CAAE;AAAA,SACjB;AAAA;AACJ,KACF;AAAA,EACF;AAAA,EAEA,MAAM,WAAA,CACJ,QAAA,EACA,OAAA,EAC4B;AAC5B,IAAA,MAAM,EAAA,GAAK,QAAA;AAEX,IAAA,MAAM,OAAO,MAAM,EAAA;AAAA,MACjB;AAAA,MAEC,OAAA,CAAQ,mBAAA,EAAqB,OAAA,CAAQ,UAAU,EAC/C,MAAA,EAAO;AAEV,IAAA,MAAM,UAAA,GAAa,KAAK,GAAA,CAAI,CAAA,CAAA,KAAK,EAAE,iBAAkB,CAAA,CAAE,OAAO,OAAO,CAAA;AAErE,IAAA,OAAO,EAAE,UAAA,EAAW;AAAA,EACtB;AAAA,EAEA,MAAM,YAAe,EAAA,EAAiD;AACpE,IAAA,IAAI;AACF,MAAA,IAAI,MAAA,GAAwB,KAAA,CAAA;AAE5B,MAAA,MAAM,IAAA,CAAK,QAAQ,QAAA,CAAS,WAAA;AAAA,QAC1B,OAAM,EAAA,KAAM;AAGV,UAAA,MAAA,GAAS,MAAM,GAAG,EAAE,CAAA;AAAA,QACtB,CAAA;AAAA,QACA;AAAA;AAAA,UAEE,qBAAA,EAAuB;AAAA;AACzB,OACF;AAEA,MAAA,OAAO,MAAA;AAAA,IACT,SAAS,CAAA,EAAG;AACV,MAAA,IAAA,CAAK,OAAA,CAAQ,MAAA,CAAO,KAAA,CAAM,CAAA,0BAAA,EAA6B,CAAC,CAAA,CAAE,CAAA;AAC1D,MAAA,MAAMC,wBAAa,CAAC,CAAA;AAAA,IACtB;AAAA,EACF;AAAA;AAAA;AAAA;AAAA;AAAA,EAMA,MAAc,sBAAA,CACZ,QAAA,EACA,OAAA,EAIe;AACf,IAAA,MAAM,EAAA,GAAK,QAAA;AAGX,IAAA,MAAM,eAAA,GAAkB,IAAI,KAAA,EAAc;AAI1C,IAAA,KAAA,MAAW,EAAE,MAAA,EAAQ,WAAA,EAAY,IAAK,QAAQ,QAAA,EAAU;AACtD,MAAA,MAAM,SAAA,GAAYL,gCAAmB,MAAM,CAAA;AAC3C,MAAA,MAAM,IAAA,GAAOM,wBAAmB,MAAM,CAAA;AAEtC,MAAA,MAAM,OAAA,GAAU,MAAMC,+CAAA,CAAwB;AAAA,QAC5C,EAAA;AAAA,QACA,MAAA;AAAA,QACA,IAAA;AAAA,QACA;AAAA,OACD,CAAA;AACD,MAAA,IAAI,OAAA,EAAS;AACX,QAAA,eAAA,CAAgB,KAAK,SAAS,CAAA;AAC9B,QAAA;AAAA,MACF;AAEA,MAAA,MAAM,QAAA,GAAW,MAAMC,+CAAA,CAAwB;AAAA,QAC7C,EAAA;AAAA,QACA,MAAA;AAAA,QACA,IAAA;AAAA,QACA,WAAA;AAAA,QACA,MAAA,EAAQ,KAAK,OAAA,CAAQ;AAAA,OACtB,CAAA;AACD,MAAA,IAAI,QAAA,EAAU;AACZ,QAAA,eAAA,CAAgB,KAAK,SAAS,CAAA;AAC9B,QAAA;AAAA,MACF;AAKA,MAAA,MAAM,cAAA,GAAiB,MAAMC,iDAAA,CAAyB;AAAA,QACpD,EAAA;AAAA,QACA,SAAA;AAAA,QACA;AAAA,OACD,CAAA;AACD,MAAA,IAAI,cAAA,EAAgB;AAClB,QAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,IAAA;AAAA,UAClB,CAAA,+BAAA,EAAkC,SAAS,CAAA,uBAAA,EAA0B,cAAc,iBAAiB,WAAW,CAAA;AAAA,SACjH;AACA,QAAA,IAAI,WAAA,EAAa;AACf,UAAA,MAAM,WAAA,GAAwD;AAAA,YAC5D,KAAA,EAAOC,iCAAA;AAAA,YACP,YAAA,EAAc;AAAA,cACZ,iBAAA,EAAmB,MAAA;AAAA,cACnB,SAAA;AAAA,cACA,cAAA,EAAgB,WAAA;AAAA,cAChB,mBAAA,EAAqB,cAAA;AAAA,cACrB,cAAA,EAAgBC,cAAA,CAAS,GAAA,EAAI,CAAE,KAAA;AAAM;AACvC,WACF;AACA,UAAA,MAAM,IAAA,CAAK,OAAA,CAAQ,MAAA,CAAO,OAAA,CAAQ,WAAW,CAAA;AAAA,QAC/C;AAAA,MACF;AAAA,IACF;AAGA,IAAA,MAAM,EAAA,CAAgC,0BAA0B,CAAA,CAE7D,KAAA,CAAM,EAAE,iBAAA,EAAmB,OAAA,CAAQ,eAAA,EAAiB,CAAA,CAEpD,SAAA,CAAU,mBAAA,EAAqB,eAAe,EAC9C,MAAA,EAAO;AACV,IAAA,MAAM,EAAA,CAAG,WAAA;AAAA,MACP,0BAAA;AAAA,MACA,eAAA,CAAgB,IAAI,CAAA,SAAA,MAAc;AAAA,QAChC,mBAAmB,OAAA,CAAQ,eAAA;AAAA,QAC3B,iBAAA,EAAmB;AAAA,OACrB,CAAE,CAAA;AAAA,MACF;AAAA,KACF;AAAA,EACF;AACF;;;;"}
+{"version":3,"file":"DefaultProcessingDatabase.cjs.js","sources":["../../src/database/DefaultProcessingDatabase.ts"],"sourcesContent":["/*\n * Copyright 2021 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { Entity, stringifyEntityRef } from '@backstage/catalog-model';\nimport { ConflictError } from '@backstage/errors';\nimport { DeferredEntity } from '@backstage/plugin-catalog-node';\nimport { Knex } from 'knex';\nimport { ProcessingIntervalFunction } from '../processing/refresh';\nimport { rethrowError, timestampToDateTime } from './conversion';\nimport { initDatabaseMetrics } from './metrics';\nimport {\n  DbRefreshKeysRow,\n  DbRefreshStateReferencesRow,\n  DbRefreshStateRow,\n  DbRelationsRow,\n} from './tables';\nimport {\n  GetProcessableEntitiesResult,\n  ListParentsOptions,\n  ListParentsResult,\n  ProcessingDatabase,\n  RefreshStateItem,\n  Transaction,\n  UpdateEntityCacheOptions,\n  UpdateProcessedEntityOptions,\n} from './types';\nimport { checkLocationKeyConflict } from './operations/refreshState/checkLocationKeyConflict';\nimport { insertUnprocessedEntity } from './operations/refreshState/insertUnprocessedEntity';\nimport { updateUnprocessedEntity } from './operations/refreshState/updateUnprocessedEntity';\nimport { generateStableHash, generateTargetKey, whereInArray } from './util';\nimport {\n  syncRelations,\n  SyncRelationsResult,\n} from './operations/relations/syncRelations';\nimport { EventParams, EventsService } from '@backstage/plugin-events-node';\nimport { DateTime } from 'luxon';\nimport { CATALOG_CONFLICTS_TOPIC } from '../constants';\nimport { CatalogConflictEventPayload } from '../catalog/types';\nimport { LoggerService } from '@backstage/backend-plugin-api';\nimport { MetricsService } from '@backstage/backend-plugin-api/alpha';\n\n// The number of items that are sent per batch to the database layer, when\n// doing .batchInsert calls to knex. This needs to be low enough to not cause\n// errors in the underlying engine due to exceeding query limits, but large\n// enough to get the speed benefits.\nconst BATCH_SIZE = 50;\n\nexport class DefaultProcessingDatabase implements ProcessingDatabase {\n  private readonly options: {\n    database: Knex;\n    logger: LoggerService;\n    refreshInterval: ProcessingIntervalFunction;\n    events: EventsService;\n    metrics: MetricsService;\n  };\n\n  constructor(options: {\n    database: Knex;\n    logger: LoggerService;\n    refreshInterval: ProcessingIntervalFunction;\n    events: EventsService;\n    metrics: MetricsService;\n  }) {\n    this.options = options;\n    initDatabaseMetrics(options.database, options.metrics);\n  }\n\n  async updateProcessedEntity(\n    txOpaque: Transaction,\n    options: UpdateProcessedEntityOptions,\n  ): Promise<{ relationsChange: SyncRelationsResult }> {\n    const tx = txOpaque as Knex.Transaction;\n    const {\n      id,\n      processedEntity,\n      resultHash,\n      errors,\n      relations,\n      deferredEntities,\n      refreshKeys,\n      locationKey,\n    } = options;\n    const refreshResult = await tx<DbRefreshStateRow>('refresh_state')\n      .update({\n        processed_entity: JSON.stringify(processedEntity),\n        result_hash: resultHash,\n        errors,\n        location_key: locationKey,\n      })\n      .where('entity_id', id)\n      .andWhere(inner => {\n        if (!locationKey) {\n          return inner.whereNull('location_key');\n        }\n        return inner\n          .where('location_key', locationKey)\n          .orWhereNull('location_key');\n      });\n    if (refreshResult === 0) {\n      throw new ConflictError(\n        `Conflicting write of processing result for ${id} with location key '${locationKey}'`,\n      );\n    }\n    const sourceEntityRef = stringifyEntityRef(processedEntity);\n\n    // Schedule all deferred entities for future processing.\n    await this.addUnprocessedEntities(tx, {\n      entities: deferredEntities,\n      sourceEntityRef,\n    });\n\n    // Sync relations using a diff-based approach — only touches rows that\n    // actually changed, eliminating steady-state write churn.\n    const relationRows: DbRelationsRow[] = relations.map(\n      ({ source, target, type }) => ({\n        originating_entity_id: id,\n        source_entity_ref: stringifyEntityRef(source),\n        target_entity_ref: stringifyEntityRef(target),\n        type,\n      }),\n    );\n\n    const relationsResult = await syncRelations(tx, id, relationRows);\n\n    // Delete old refresh keys\n    await tx<DbRefreshKeysRow>('refresh_keys')\n      .where({ entity_id: id })\n      .delete();\n\n    // Insert the refresh keys for the processed entity\n    await tx.batchInsert(\n      'refresh_keys',\n      refreshKeys.map(k => ({\n        entity_id: id,\n        key: generateTargetKey(k.key),\n      })),\n      BATCH_SIZE,\n    );\n\n    return {\n      relationsChange: relationsResult,\n    };\n  }\n\n  async updateProcessedEntityErrors(\n    txOpaque: Transaction,\n    options: UpdateProcessedEntityOptions,\n  ): Promise<void> {\n    const tx = txOpaque as Knex.Transaction;\n    const { id, errors, resultHash } = options;\n\n    await tx<DbRefreshStateRow>('refresh_state')\n      .update({\n        errors,\n        result_hash: resultHash,\n      })\n      .where('entity_id', id);\n  }\n\n  async updateEntityCache(\n    txOpaque: Transaction,\n    options: UpdateEntityCacheOptions,\n  ): Promise<void> {\n    const tx = txOpaque as Knex.Transaction;\n    const { id, state } = options;\n\n    await tx<DbRefreshStateRow>('refresh_state')\n      .update({ cache: JSON.stringify(state ?? {}) })\n      .where('entity_id', id);\n  }\n\n  async getProcessableEntities(\n    maybeTx: Transaction | Knex,\n    request: { processBatchSize: number },\n  ): Promise<GetProcessableEntitiesResult> {\n    const knex = maybeTx as Knex.Transaction | Knex;\n    const useLocking = ['mysql', 'mysql2', 'pg'].includes(\n      knex.client.config.client,\n    );\n\n    const interval = this.options.refreshInterval();\n\n    const nextUpdateAt = (\n      tx: Knex | Knex.Transaction,\n      refreshInterval: number,\n    ) => {\n      if (tx.client.config.client.includes('sqlite3')) {\n        return tx.raw(`datetime('now', ?)`, [`${refreshInterval} seconds`]);\n      } else if (tx.client.config.client.includes('mysql')) {\n        return tx.raw(`now() + interval ${refreshInterval} second`);\n      }\n      return tx.raw(`now() + interval '${refreshInterval} seconds'`);\n    };\n\n    // The SELECT FOR UPDATE SKIP LOCKED + UPDATE must run inside a\n    // single transaction so that the row locks persist until\n    // next_update_at has been bumped.\n    const run = async (tx: Knex | Knex.Transaction) => {\n      const items: DbRefreshStateRow[] = await tx('refresh_state')\n        .select([\n          'entity_id',\n          'entity_ref',\n          'unprocessed_entity',\n          'result_hash',\n          'cache',\n          'errors',\n          'location_key',\n          'next_update_at',\n        ])\n        .where('next_update_at', '<=', tx.fn.now())\n        .limit(request.processBatchSize)\n        .orderBy('next_update_at', 'asc')\n        .modify(qb => {\n          if (useLocking) {\n            qb.forUpdate().skipLocked();\n          }\n        });\n\n      if (items.length > 0) {\n        await tx('refresh_state')\n          .where(\n            whereInArray(\n              'entity_ref',\n              items.map(i => i.entity_ref),\n            ),\n          )\n          .update({\n            next_update_at: nextUpdateAt(tx, interval),\n          });\n      }\n\n      return items;\n    };\n\n    const items =\n      knex.isTransaction || !useLocking\n        ? await run(knex)\n        : await knex.transaction(run);\n\n    return {\n      items: items.map(\n        i =>\n          ({\n            id: i.entity_id,\n            entityRef: i.entity_ref,\n            unprocessedEntity: JSON.parse(i.unprocessed_entity) as Entity,\n            resultHash: i.result_hash || '',\n            nextUpdateAt: timestampToDateTime(i.next_update_at),\n            state: i.cache ? JSON.parse(i.cache) : undefined,\n            errors: i.errors,\n            locationKey: i.location_key,\n          } satisfies RefreshStateItem),\n      ),\n    };\n  }\n\n  async listParents(\n    txOpaque: Transaction,\n    options: ListParentsOptions,\n  ): Promise<ListParentsResult> {\n    const tx = txOpaque as Knex.Transaction;\n\n    const rows = await tx<DbRefreshStateReferencesRow>(\n      'refresh_state_references',\n    )\n      .whereIn('target_entity_ref', options.entityRefs)\n      .select();\n\n    const entityRefs = rows.map(r => r.source_entity_ref!).filter(Boolean);\n\n    return { entityRefs };\n  }\n\n  async transaction<T>(fn: (tx: Transaction) => Promise<T>): Promise<T> {\n    try {\n      let result: T | undefined = undefined;\n\n      await this.options.database.transaction(\n        async tx => {\n          // We can't return here, as knex swallows the return type in case the transaction is rolled back:\n          // https://github.com/knex/knex/blob/e37aeaa31c8ef9c1b07d2e4d3ec6607e557d800d/lib/transaction.js#L136\n          result = await fn(tx);\n        },\n        {\n          // If we explicitly trigger a rollback, don't fail.\n          doNotRejectOnRollback: true,\n        },\n      );\n\n      return result!;\n    } catch (e) {\n      this.options.logger.debug(`Error during transaction, ${e}`);\n      throw rethrowError(e);\n    }\n  }\n\n  /**\n   * Add a set of deferred entities for processing.\n   * The entities will be added at the front of the processing queue.\n   */\n  private async addUnprocessedEntities(\n    txOpaque: Transaction,\n    options: {\n      sourceEntityRef: string;\n      entities: DeferredEntity[];\n    },\n  ): Promise<void> {\n    const tx = txOpaque as Knex.Transaction;\n\n    // Keeps track of the entities that we end up inserting to update refresh_state_references afterwards\n    const stateReferences = new Array<string>();\n\n    // Upsert all of the unprocessed entities into the refresh_state table, by\n    // their entity ref.\n    for (const { entity, locationKey } of options.entities) {\n      const entityRef = stringifyEntityRef(entity);\n      const hash = generateStableHash(entity);\n\n      const updated = await updateUnprocessedEntity({\n        tx,\n        entity,\n        hash,\n        locationKey,\n      });\n      if (updated) {\n        stateReferences.push(entityRef);\n        continue;\n      }\n\n      const inserted = await insertUnprocessedEntity({\n        tx,\n        entity,\n        hash,\n        locationKey,\n        logger: this.options.logger,\n      });\n      if (inserted) {\n        stateReferences.push(entityRef);\n        continue;\n      }\n\n      // If the row can't be inserted, we have a conflict, but it could be either\n      // because of a conflicting locationKey or a race with another instance, so check\n      // whether the conflicting entity has the same entityRef but a different locationKey\n      const conflictingKey = await checkLocationKeyConflict({\n        tx,\n        entityRef,\n        locationKey,\n      });\n      if (conflictingKey) {\n        this.options.logger.warn(\n          `Detected conflicting entityRef ${entityRef} already referenced by ${conflictingKey} and now also ${locationKey}`,\n        );\n        if (locationKey) {\n          const eventParams: EventParams<CatalogConflictEventPayload> = {\n            topic: CATALOG_CONFLICTS_TOPIC,\n            eventPayload: {\n              unprocessedEntity: entity,\n              entityRef,\n              newLocationKey: locationKey,\n              existingLocationKey: conflictingKey,\n              lastConflictAt: DateTime.now().toISO()!,\n            },\n          };\n          await this.options.events.publish(eventParams);\n        }\n      }\n    }\n\n    // Lastly, replace refresh state references for the originating entity and any successfully added entities\n    await tx<DbRefreshStateReferencesRow>('refresh_state_references')\n      // Remove all existing references from the originating entity\n      .where({ source_entity_ref: options.sourceEntityRef })\n      // And remove any existing references to entities that we're inserting new references for\n      .orWhereIn('target_entity_ref', stateReferences)\n      .delete();\n    await tx.batchInsert(\n      'refresh_state_references',\n      stateReferences.map(entityRef => ({\n        source_entity_ref: options.sourceEntityRef,\n        target_entity_ref: entityRef,\n      })),\n      BATCH_SIZE,\n    );\n  }\n}\n"],"names":["initDatabaseMetrics","errors","ConflictError","stringifyEntityRef","syncRelations","generateTargetKey","items","whereInArray","timestampToDateTime","rethrowError","generateStableHash","updateUnprocessedEntity","insertUnprocessedEntity","checkLocationKeyConflict","CATALOG_CONFLICTS_TOPIC","DateTime"],"mappings":";;;;;;;;;;;;;;AA0DA,MAAM,UAAA,GAAa,EAAA;AAEZ,MAAM,yBAAA,CAAwD;AAAA,EAClD,OAAA;AAAA,EAQjB,YAAY,OAAA,EAMT;AACD,IAAA,IAAA,CAAK,OAAA,GAAU,OAAA;AACf,IAAAA,2BAAA,CAAoB,OAAA,CAAQ,QAAA,EAAU,OAAA,CAAQ,OAAO,CAAA;AAAA,EACvD;AAAA,EAEA,MAAM,qBAAA,CACJ,QAAA,EACA,OAAA,EACmD;AACnD,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAM;AAAA,MACJ,EAAA;AAAA,MACA,eAAA;AAAA,MACA,UAAA;AAAA,cACAC,QAAA;AAAA,MACA,SAAA;AAAA,MACA,gBAAA;AAAA,MACA,WAAA;AAAA,MACA;AAAA,KACF,GAAI,OAAA;AACJ,IAAA,MAAM,aAAA,GAAgB,MAAM,EAAA,CAAsB,eAAe,EAC9D,MAAA,CAAO;AAAA,MACN,gBAAA,EAAkB,IAAA,CAAK,SAAA,CAAU,eAAe,CAAA;AAAA,MAChD,WAAA,EAAa,UAAA;AAAA,cACbA,QAAA;AAAA,MACA,YAAA,EAAc;AAAA,KACf,CAAA,CACA,KAAA,CAAM,aAAa,EAAE,CAAA,CACrB,SAAS,CAAA,KAAA,KAAS;AACjB,MAAA,IAAI,CAAC,WAAA,EAAa;AAChB,QAAA,OAAO,KAAA,CAAM,UAAU,cAAc,CAAA;AAAA,MACvC;AACA,MAAA,OAAO,MACJ,KAAA,CAAM,cAAA,EAAgB,WAAW,CAAA,CACjC,YAAY,cAAc,CAAA;AAAA,IAC/B,CAAC,CAAA;AACH,IAAA,IAAI,kBAAkB,CAAA,EAAG;AACvB,MAAA,MAAM,IAAIC,oBAAA;AAAA,QACR,CAAA,2CAAA,EAA8C,EAAE,CAAA,oBAAA,EAAuB,WAAW,CAAA,CAAA;AAAA,OACpF;AAAA,IACF;AACA,IAAA,MAAM,eAAA,GAAkBC,gCAAmB,eAAe,CAAA;AAG1D,IAAA,MAAM,IAAA,CAAK,uBAAuB,EAAA,EAAI;AAAA,MACpC,QAAA,EAAU,gBAAA;AAAA,MACV;AAAA,KACD,CAAA;AAID,IAAA,MAAM,eAAiC,SAAA,CAAU,GAAA;AAAA,MAC/C,CAAC,EAAE,MAAA,EAAQ,MAAA,EAAQ,MAAK,MAAO;AAAA,QAC7B,qBAAA,EAAuB,EAAA;AAAA,QACvB,iBAAA,EAAmBA,gCAAmB,MAAM,CAAA;AAAA,QAC5C,iBAAA,EAAmBA,gCAAmB,MAAM,CAAA;AAAA,QAC5C;AAAA,OACF;AAAA,KACF;AAEA,IAAA,MAAM,eAAA,GAAkB,MAAMC,2BAAA,CAAc,EAAA,EAAI,IAAI,YAAY,CAAA;AAGhE,IAAA,MAAM,EAAA,CAAqB,cAAc,CAAA,CACtC,KAAA,CAAM,EAAE,SAAA,EAAW,EAAA,EAAI,CAAA,CACvB,MAAA,EAAO;AAGV,IAAA,MAAM,EAAA,CAAG,WAAA;AAAA,MACP,cAAA;AAAA,MACA,WAAA,CAAY,IAAI,CAAA,CAAA,MAAM;AAAA,QACpB,SAAA,EAAW,EAAA;AAAA,QACX,GAAA,EAAKC,sBAAA,CAAkB,CAAA,CAAE,GAAG;AAAA,OAC9B,CAAE,CAAA;AAAA,MACF;AAAA,KACF;AAEA,IAAA,OAAO;AAAA,MACL,eAAA,EAAiB;AAAA,KACnB;AAAA,EACF;AAAA,EAEA,MAAM,2BAAA,CACJ,QAAA,EACA,OAAA,EACe;AACf,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAM,EAAE,EAAA,EAAI,MAAA,EAAQ,UAAA,EAAW,GAAI,OAAA;AAEnC,IAAA,MAAM,EAAA,CAAsB,eAAe,CAAA,CACxC,MAAA,CAAO;AAAA,MACN,MAAA;AAAA,MACA,WAAA,EAAa;AAAA,KACd,CAAA,CACA,KAAA,CAAM,WAAA,EAAa,EAAE,CAAA;AAAA,EAC1B;AAAA,EAEA,MAAM,iBAAA,CACJ,QAAA,EACA,OAAA,EACe;AACf,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAM,EAAE,EAAA,EAAI,KAAA,EAAM,GAAI,OAAA;AAEtB,IAAA,MAAM,GAAsB,eAAe,CAAA,CACxC,MAAA,CAAO,EAAE,OAAO,IAAA,CAAK,SAAA,CAAU,KAAA,IAAS,EAAE,CAAA,EAAG,CAAA,CAC7C,KAAA,CAAM,aAAa,EAAE,CAAA;AAAA,EAC1B;AAAA,EAEA,MAAM,sBAAA,CACJ,OAAA,EACA,OAAA,EACuC;AACvC,IAAA,MAAM,IAAA,GAAO,OAAA;AACb,IAAA,MAAM,UAAA,GAAa,CAAC,OAAA,EAAS,QAAA,EAAU,IAAI,CAAA,CAAE,QAAA;AAAA,MAC3C,IAAA,CAAK,OAAO,MAAA,CAAO;AAAA,KACrB;AAEA,IAAA,MAAM,QAAA,GAAW,IAAA,CAAK,OAAA,CAAQ,eAAA,EAAgB;AAE9C,IAAA,MAAM,YAAA,GAAe,CACnB,EAAA,EACA,eAAA,KACG;AACH,MAAA,IAAI,GAAG,MAAA,CAAO,MAAA,CAAO,MAAA,CAAO,QAAA,CAAS,SAAS,CAAA,EAAG;AAC/C,QAAA,OAAO,GAAG,GAAA,CAAI,CAAA,kBAAA,CAAA,EAAsB,CAAC,CAAA,EAAG,eAAe,UAAU,CAAC,CAAA;AAAA,MACpE,WAAW,EAAA,CAAG,MAAA,CAAO,OAAO,MAAA,CAAO,QAAA,CAAS,OAAO,CAAA,EAAG;AACpD,QAAA,OAAO,EAAA,CAAG,GAAA,CAAI,CAAA,iBAAA,EAAoB,eAAe,CAAA,OAAA,CAAS,CAAA;AAAA,MAC5D;AACA,MAAA,OAAO,EAAA,CAAG,GAAA,CAAI,CAAA,kBAAA,EAAqB,eAAe,CAAA,SAAA,CAAW,CAAA;AAAA,IAC/D,CAAA;AAKA,IAAA,MAAM,GAAA,GAAM,OAAO,EAAA,KAAgC;AACjD,MAAA,MAAMC,MAAAA,GAA6B,MAAM,EAAA,CAAG,eAAe,EACxD,MAAA,CAAO;AAAA,QACN,WAAA;AAAA,QACA,YAAA;AAAA,QACA,oBAAA;AAAA,QACA,aAAA;AAAA,QACA,OAAA;AAAA,QACA,QAAA;AAAA,QACA,cAAA;AAAA,QACA;AAAA,OACD,CAAA,CACA,KAAA,CAAM,kBAAkB,IAAA,EAAM,EAAA,CAAG,GAAG,GAAA,EAAK,EACzC,KAAA,CAAM,OAAA,CAAQ,gBAAgB,CAAA,CAC9B,OAAA,CAAQ,kBAAkB,KAAK,CAAA,CAC/B,OAAO,CAAA,EAAA,KAAM;AACZ,QAAA,IAAI,UAAA,EAAY;AACd,UAAA,EAAA,CAAG,SAAA,GAAY,UAAA,EAAW;AAAA,QAC5B;AAAA,MACF,CAAC,CAAA;AAEH,MAAA,IAAIA,MAAAA,CAAM,SAAS,CAAA,EAAG;AACpB,QAAA,MAAM,EAAA,CAAG,eAAe,CAAA,CACrB,KAAA;AAAA,UACCC,iBAAA;AAAA,YACE,YAAA;AAAA,YACAD,MAAAA,CAAM,GAAA,CAAI,CAAA,CAAA,KAAK,CAAA,CAAE,UAAU;AAAA;AAC7B,UAED,MAAA,CAAO;AAAA,UACN,cAAA,EAAgB,YAAA,CAAa,EAAA,EAAI,QAAQ;AAAA,SAC1C,CAAA;AAAA,MACL;AAEA,MAAA,OAAOA,MAAAA;AAAA,IACT,CAAA;AAEA,IAAA,MAAM,KAAA,GACJ,IAAA,CAAK,aAAA,IAAiB,CAAC,UAAA,GACnB,MAAM,GAAA,CAAI,IAAI,CAAA,GACd,MAAM,IAAA,CAAK,WAAA,CAAY,GAAG,CAAA;AAEhC,IAAA,OAAO;AAAA,MACL,OAAO,KAAA,CAAM,GAAA;AAAA,QACX,CAAA,CAAA,MACG;AAAA,UACC,IAAI,CAAA,CAAE,SAAA;AAAA,UACN,WAAW,CAAA,CAAE,UAAA;AAAA,UACb,iBAAA,EAAmB,IAAA,CAAK,KAAA,CAAM,CAAA,CAAE,kBAAkB,CAAA;AAAA,UAClD,UAAA,EAAY,EAAE,WAAA,IAAe,EAAA;AAAA,UAC7B,YAAA,EAAcE,8BAAA,CAAoB,CAAA,CAAE,cAAc,CAAA;AAAA,UAClD,OAAO,CAAA,CAAE,KAAA,GAAQ,KAAK,KAAA,CAAM,CAAA,CAAE,KAAK,CAAA,GAAI,MAAA;AAAA,UACvC,QAAQ,CAAA,CAAE,MAAA;AAAA,UACV,aAAa,CAAA,CAAE;AAAA,SACjB;AAAA;AACJ,KACF;AAAA,EACF;AAAA,EAEA,MAAM,WAAA,CACJ,QAAA,EACA,OAAA,EAC4B;AAC5B,IAAA,MAAM,EAAA,GAAK,QAAA;AAEX,IAAA,MAAM,OAAO,MAAM,EAAA;AAAA,MACjB;AAAA,MAEC,OAAA,CAAQ,mBAAA,EAAqB,OAAA,CAAQ,UAAU,EAC/C,MAAA,EAAO;AAEV,IAAA,MAAM,UAAA,GAAa,KAAK,GAAA,CAAI,CAAA,CAAA,KAAK,EAAE,iBAAkB,CAAA,CAAE,OAAO,OAAO,CAAA;AAErE,IAAA,OAAO,EAAE,UAAA,EAAW;AAAA,EACtB;AAAA,EAEA,MAAM,YAAe,EAAA,EAAiD;AACpE,IAAA,IAAI;AACF,MAAA,IAAI,MAAA,GAAwB,KAAA,CAAA;AAE5B,MAAA,MAAM,IAAA,CAAK,QAAQ,QAAA,CAAS,WAAA;AAAA,QAC1B,OAAM,EAAA,KAAM;AAGV,UAAA,MAAA,GAAS,MAAM,GAAG,EAAE,CAAA;AAAA,QACtB,CAAA;AAAA,QACA;AAAA;AAAA,UAEE,qBAAA,EAAuB;AAAA;AACzB,OACF;AAEA,MAAA,OAAO,MAAA;AAAA,IACT,SAAS,CAAA,EAAG;AACV,MAAA,IAAA,CAAK,OAAA,CAAQ,MAAA,CAAO,KAAA,CAAM,CAAA,0BAAA,EAA6B,CAAC,CAAA,CAAE,CAAA;AAC1D,MAAA,MAAMC,wBAAa,CAAC,CAAA;AAAA,IACtB;AAAA,EACF;AAAA;AAAA;AAAA;AAAA;AAAA,EAMA,MAAc,sBAAA,CACZ,QAAA,EACA,OAAA,EAIe;AACf,IAAA,MAAM,EAAA,GAAK,QAAA;AAGX,IAAA,MAAM,eAAA,GAAkB,IAAI,KAAA,EAAc;AAI1C,IAAA,KAAA,MAAW,EAAE,MAAA,EAAQ,WAAA,EAAY,IAAK,QAAQ,QAAA,EAAU;AACtD,MAAA,MAAM,SAAA,GAAYN,gCAAmB,MAAM,CAAA;AAC3C,MAAA,MAAM,IAAA,GAAOO,wBAAmB,MAAM,CAAA;AAEtC,MAAA,MAAM,OAAA,GAAU,MAAMC,+CAAA,CAAwB;AAAA,QAC5C,EAAA;AAAA,QACA,MAAA;AAAA,QACA,IAAA;AAAA,QACA;AAAA,OACD,CAAA;AACD,MAAA,IAAI,OAAA,EAAS;AACX,QAAA,eAAA,CAAgB,KAAK,SAAS,CAAA;AAC9B,QAAA;AAAA,MACF;AAEA,MAAA,MAAM,QAAA,GAAW,MAAMC,+CAAA,CAAwB;AAAA,QAC7C,EAAA;AAAA,QACA,MAAA;AAAA,QACA,IAAA;AAAA,QACA,WAAA;AAAA,QACA,MAAA,EAAQ,KAAK,OAAA,CAAQ;AAAA,OACtB,CAAA;AACD,MAAA,IAAI,QAAA,EAAU;AACZ,QAAA,eAAA,CAAgB,KAAK,SAAS,CAAA;AAC9B,QAAA;AAAA,MACF;AAKA,MAAA,MAAM,cAAA,GAAiB,MAAMC,iDAAA,CAAyB;AAAA,QACpD,EAAA;AAAA,QACA,SAAA;AAAA,QACA;AAAA,OACD,CAAA;AACD,MAAA,IAAI,cAAA,EAAgB;AAClB,QAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,IAAA;AAAA,UAClB,CAAA,+BAAA,EAAkC,SAAS,CAAA,uBAAA,EAA0B,cAAc,iBAAiB,WAAW,CAAA;AAAA,SACjH;AACA,QAAA,IAAI,WAAA,EAAa;AACf,UAAA,MAAM,WAAA,GAAwD;AAAA,YAC5D,KAAA,EAAOC,iCAAA;AAAA,YACP,YAAA,EAAc;AAAA,cACZ,iBAAA,EAAmB,MAAA;AAAA,cACnB,SAAA;AAAA,cACA,cAAA,EAAgB,WAAA;AAAA,cAChB,mBAAA,EAAqB,cAAA;AAAA,cACrB,cAAA,EAAgBC,cAAA,CAAS,GAAA,EAAI,CAAE,KAAA;AAAM;AACvC,WACF;AACA,UAAA,MAAM,IAAA,CAAK,OAAA,CAAQ,MAAA,CAAO,OAAA,CAAQ,WAAW,CAAA;AAAA,QAC/C;AAAA,MACF;AAAA,IACF;AAGA,IAAA,MAAM,EAAA,CAAgC,0BAA0B,CAAA,CAE7D,KAAA,CAAM,EAAE,iBAAA,EAAmB,OAAA,CAAQ,eAAA,EAAiB,CAAA,CAEpD,SAAA,CAAU,mBAAA,EAAqB,eAAe,EAC9C,MAAA,EAAO;AACV,IAAA,MAAM,EAAA,CAAG,WAAA;AAAA,MACP,0BAAA;AAAA,MACA,eAAA,CAAgB,IAAI,CAAA,SAAA,MAAc;AAAA,QAChC,mBAAmB,OAAA,CAAQ,eAAA;AAAA,QAC3B,iBAAA,EAAmB;AAAA,OACrB,CAAE,CAAA;AAAA,MACF;AAAA,KACF;AAAA,EACF;AACF;;;;"}

package/dist/database/DefaultProviderDatabase.cjs.js

@@ -170,7 +170,7 @@ class DefaultProviderDatabase {
       const toRemove2 = options.removed.map((e) => e.entityRef);
       for (const chunk of lodash__default.default.chunk(options.added, 1e3)) {
         const entityRefs = chunk.map((e) => catalogModel.stringifyEntityRef(e.entity));
-        const rows = await tx("refresh_state").select(["entity_ref", "unprocessed_hash", "location_key"]).whereIn("entity_ref", entityRefs);
+        const rows = await tx("refresh_state").select(["entity_ref", "unprocessed_hash", "location_key"]).where(util.whereInArray("entity_ref", entityRefs));
         const oldStates = new Map(
           rows.map((row) => [
             row.entity_ref,

package/dist/database/DefaultProviderDatabase.cjs.js.map

@@ -1 +1 @@
-{"version":3,"file":"DefaultProviderDatabase.cjs.js","sources":["../../src/database/DefaultProviderDatabase.ts"],"sourcesContent":["/*\n * Copyright 2021 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { stringifyEntityRef } from '@backstage/catalog-model';\nimport { DeferredEntity } from '@backstage/plugin-catalog-node';\nimport { Knex } from 'knex';\nimport lodash from 'lodash';\nimport { randomUUID as uuid } from 'node:crypto';\nimport { rethrowError } from './conversion';\nimport { deleteWithEagerPruningOfChildren } from './operations/provider/deleteWithEagerPruningOfChildren';\nimport { refreshByRefreshKeys } from './operations/provider/refreshByRefreshKeys';\nimport { checkLocationKeyConflict } from './operations/refreshState/checkLocationKeyConflict';\nimport { insertUnprocessedEntity } from './operations/refreshState/insertUnprocessedEntity';\nimport { updateUnprocessedEntity } from './operations/refreshState/updateUnprocessedEntity';\nimport { DbRefreshStateReferencesRow, DbRefreshStateRow } from './tables';\nimport {\n  ProviderDatabase,\n  RefreshByKeyOptions,\n  ReplaceUnprocessedEntitiesOptions,\n  Transaction,\n} from './types';\nimport { generateStableHash, isDeadlockError, retryOnDeadlock } from './util';\nimport {\n  LoggerService,\n  isDatabaseConflictError,\n} from '@backstage/backend-plugin-api';\n\n// The number of items that are sent per batch to the database layer, when\n// doing .batchInsert calls to knex. This needs to be low enough to not cause\n// errors in the underlying engine due to exceeding query limits, but large\n// enough to get the speed benefits.\nconst BATCH_SIZE = 50;\n\nexport class DefaultProviderDatabase implements ProviderDatabase {\n  private readonly options: {\n    database: Knex;\n    logger: LoggerService;\n  };\n\n  constructor(options: { database: Knex; logger: LoggerService }) {\n    this.options = options;\n  }\n\n  /**\n   * Executes `fn` inside a database transaction, retrying automatically on\n   * deadlock (PostgreSQL error 40P01). Because the callback may be invoked\n   * more than once, it must not perform non-database side effects such as\n   * emitting events, mutating in-memory state, or calling external services.\n   */\n  async transaction<T>(fn: (tx: Transaction) => Promise<T>): Promise<T> {\n    return retryOnDeadlock(async () => {\n      try {\n        let result: T | undefined = undefined;\n        await this.options.database.transaction(\n          async tx => {\n            // We can't return here, as knex swallows the return type in case the\n            // transaction is rolled back:\n            // https://github.com/knex/knex/blob/e37aeaa31c8ef9c1b07d2e4d3ec6607e557d800d/lib/transaction.js#L136\n            result = await fn(tx);\n          },\n          {\n            // If we explicitly trigger a rollback, don't fail.\n            doNotRejectOnRollback: true,\n          },\n        );\n        return result!;\n      } catch (e) {\n        this.options.logger.debug(`Error during transaction, ${e}`);\n        throw rethrowError(e);\n      }\n    }, this.options.database);\n  }\n\n  async replaceUnprocessedEntities(\n    txOpaque: Knex | Transaction,\n    options: ReplaceUnprocessedEntitiesOptions,\n  ): Promise<void> {\n    const tx = txOpaque as Knex | Knex.Transaction;\n    const { toAdd, toUpsert, toRemove } = await this.createDelta(tx, options);\n\n    if (toRemove.length) {\n      const removedCount = await deleteWithEagerPruningOfChildren({\n        knex: tx,\n        entityRefs: toRemove,\n        sourceKey: options.sourceKey,\n      });\n      this.options.logger.debug(\n        `removed, ${removedCount} entities: ${JSON.stringify(toRemove)}`,\n      );\n    }\n\n    if (toAdd.length) {\n      // The reason for this chunking, rather than just massively batch\n      // inserting the entire payload, is that we fall back to the individual\n      // upsert mechanism below on conflicts. That path is massively slower than\n      // the fast batch path, so we don't want to end up accidentally having to\n      // for example item-by-item upsert tens of thousands of entities in a\n      // large initial delivery dump. The implication is that the size of these\n      // chunks needs to weigh the benefit of fast successful inserts, against\n      // the drawback of super slow but more rare fallbacks. There's quickly\n      // diminishing returns though with turning up this value way high.\n      for (const chunk of lodash.chunk(toAdd, 50)) {\n        try {\n          await tx.batchInsert(\n            'refresh_state',\n            chunk.map(item => ({\n              entity_id: uuid(),\n              entity_ref: stringifyEntityRef(item.deferred.entity),\n              unprocessed_entity: JSON.stringify(item.deferred.entity),\n              unprocessed_hash: item.hash,\n              errors: '',\n              location_key: item.deferred.locationKey,\n              next_update_at: tx.fn.now(),\n              last_discovery_at: tx.fn.now(),\n            })),\n            BATCH_SIZE,\n          );\n          await tx.batchInsert(\n            'refresh_state_references',\n            chunk.map(item => ({\n              source_key: options.sourceKey,\n              target_entity_ref: stringifyEntityRef(item.deferred.entity),\n            })),\n            BATCH_SIZE,\n          );\n        } catch (error) {\n          if (!isDatabaseConflictError(error)) {\n            throw error;\n          } else {\n            this.options.logger.debug(\n              `Fast insert path failed, falling back to slow path, ${error}`,\n            );\n            toUpsert.push(...chunk);\n          }\n        }\n      }\n    }\n\n    if (toUpsert.length) {\n      for (const {\n        deferred: { entity, locationKey },\n        hash,\n      } of toUpsert) {\n        const entityRef = stringifyEntityRef(entity);\n\n        try {\n          let ok = await updateUnprocessedEntity({\n            tx,\n            entity,\n            hash,\n            locationKey,\n          });\n          if (!ok) {\n            ok = await insertUnprocessedEntity({\n              tx,\n              entity,\n              hash,\n              locationKey,\n              logger: this.options.logger,\n            });\n          }\n          if (ok) {\n            await tx<DbRefreshStateReferencesRow>('refresh_state_references')\n              .where('target_entity_ref', entityRef)\n              .delete();\n\n            await tx<DbRefreshStateReferencesRow>(\n              'refresh_state_references',\n            ).insert({\n              source_key: options.sourceKey,\n              target_entity_ref: entityRef,\n            });\n          } else {\n            await tx<DbRefreshStateReferencesRow>('refresh_state_references')\n              .where('target_entity_ref', entityRef)\n              .andWhere({ source_key: options.sourceKey })\n              .delete();\n\n            const conflictingKey = await checkLocationKeyConflict({\n              tx,\n              entityRef,\n              locationKey,\n            });\n            if (conflictingKey) {\n              this.options.logger.warn(\n                `Source ${options.sourceKey} detected conflicting entityRef ${entityRef} already referenced by ${conflictingKey} and now also ${locationKey}`,\n              );\n            }\n          }\n        } catch (error) {\n          if (isDeadlockError(tx, error)) {\n            throw error;\n          }\n          this.options.logger.error(\n            `Failed to add '${entityRef}' from source '${options.sourceKey}', ${error}`,\n          );\n        }\n      }\n    }\n  }\n\n  async listReferenceSourceKeys(txOpaque: Transaction): Promise<string[]> {\n    const tx = txOpaque as Knex | Knex.Transaction;\n\n    const rows = await tx<DbRefreshStateReferencesRow>(\n      'refresh_state_references',\n    )\n      .distinct('source_key')\n      .whereNotNull('source_key');\n\n    return rows\n      .map(row => row.source_key)\n      .filter((key): key is string => !!key);\n  }\n\n  async refreshByRefreshKeys(\n    txOpaque: Transaction,\n    options: RefreshByKeyOptions,\n  ) {\n    const tx = txOpaque as Knex.Transaction;\n    await refreshByRefreshKeys({ tx, keys: options.keys });\n  }\n\n  private async createDelta(\n    tx: Knex | Knex.Transaction,\n    options: ReplaceUnprocessedEntitiesOptions,\n  ): Promise<{\n    toAdd: { deferred: DeferredEntity; hash: string }[];\n    toUpsert: { deferred: DeferredEntity; hash: string }[];\n    toRemove: string[];\n  }> {\n    if (options.type === 'delta') {\n      const toAdd = new Array<{ deferred: DeferredEntity; hash: string }>();\n      const toUpsert = new Array<{ deferred: DeferredEntity; hash: string }>();\n      const toRemove = options.removed.map(e => e.entityRef);\n\n      for (const chunk of lodash.chunk(options.added, 1000)) {\n        const entityRefs = chunk.map(e => stringifyEntityRef(e.entity));\n        const rows = await tx<DbRefreshStateRow>('refresh_state')\n          .select(['entity_ref', 'unprocessed_hash', 'location_key'])\n          .whereIn('entity_ref', entityRefs);\n        const oldStates = new Map(\n          rows.map(row => [\n            row.entity_ref,\n            {\n              unprocessed_hash: row.unprocessed_hash,\n              location_key: row.location_key,\n            },\n          ]),\n        );\n\n        chunk.forEach((deferred, i) => {\n          const entityRef = entityRefs[i];\n          const newHash = generateStableHash(deferred.entity);\n          const oldState = oldStates.get(entityRef);\n          if (oldState === undefined) {\n            // Add any entity that does not exist in the database\n            toAdd.push({ deferred, hash: newHash });\n          } else if (\n            (deferred.locationKey ?? null) !== (oldState.location_key ?? null)\n          ) {\n            // Remove and then re-add any entity that exists, but with a different location key\n            toRemove.push(entityRef);\n            toAdd.push({ deferred, hash: newHash });\n          } else if (newHash !== oldState.unprocessed_hash) {\n            // Entities with modifications should be pushed through too\n            toUpsert.push({ deferred, hash: newHash });\n          }\n        });\n      }\n\n      return { toAdd, toUpsert, toRemove };\n    }\n\n    // Grab all of the existing references from the same source, and their locationKeys as well\n    const oldRefs = await tx<DbRefreshStateReferencesRow>(\n      'refresh_state_references',\n    )\n      .leftJoin<DbRefreshStateRow>('refresh_state', {\n        target_entity_ref: 'entity_ref',\n      })\n      .where({ source_key: options.sourceKey })\n      .select({\n        target_entity_ref: 'refresh_state_references.target_entity_ref',\n        location_key: 'refresh_state.location_key',\n        unprocessed_hash: 'refresh_state.unprocessed_hash',\n      });\n\n    const items = options.items.map(deferred => ({\n      deferred,\n      ref: stringifyEntityRef(deferred.entity),\n      hash: generateStableHash(deferred.entity),\n    }));\n\n    const oldRefsSet = new Map(\n      oldRefs.map(r => [\n        r.target_entity_ref,\n        {\n          locationKey: r.location_key,\n          oldEntityHash: r.unprocessed_hash,\n        },\n      ]),\n    );\n    const newRefsSet = new Set(items.map(item => item.ref));\n\n    const toAdd = new Array<{ deferred: DeferredEntity; hash: string }>();\n    const toUpsert = new Array<{ deferred: DeferredEntity; hash: string }>();\n    const toRemove = oldRefs\n      .map(row => row.target_entity_ref)\n      .filter(ref => !newRefsSet.has(ref));\n\n    for (const item of items) {\n      const oldRef = oldRefsSet.get(item.ref);\n      const upsertItem = { deferred: item.deferred, hash: item.hash };\n      if (!oldRef) {\n        // Add any entity that does not exist in the database\n        toAdd.push(upsertItem);\n      } else if (\n        (oldRef.locationKey ?? undefined) !==\n        (item.deferred.locationKey ?? undefined)\n      ) {\n        // Remove and then re-add any entity that exists, but with a different location key\n        toRemove.push(item.ref);\n        toAdd.push(upsertItem);\n      } else if (oldRef.oldEntityHash !== item.hash) {\n        // Entities with modifications should be pushed through too\n        toUpsert.push(upsertItem);\n      }\n    }\n\n    return { toAdd, toUpsert, toRemove };\n  }\n}\n"],"names":["retryOnDeadlock","rethrowError","deleteWithEagerPruningOfChildren","lodash","uuid","stringifyEntityRef","isDatabaseConflictError","updateUnprocessedEntity","insertUnprocessedEntity","checkLocationKeyConflict","isDeadlockError","refreshByRefreshKeys","toAdd","toUpsert","toRemove","generateStableHash"],"mappings":";;;;;;;;;;;;;;;;;;AA4CA,MAAM,UAAA,GAAa,EAAA;AAEZ,MAAM,uBAAA,CAAoD;AAAA,EAC9C,OAAA;AAAA,EAKjB,YAAY,OAAA,EAAoD;AAC9D,IAAA,IAAA,CAAK,OAAA,GAAU,OAAA;AAAA,EACjB;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,EAQA,MAAM,YAAe,EAAA,EAAiD;AACpE,IAAA,OAAOA,qBAAgB,YAAY;AACjC,MAAA,IAAI;AACF,QAAA,IAAI,MAAA,GAAwB,KAAA,CAAA;AAC5B,QAAA,MAAM,IAAA,CAAK,QAAQ,QAAA,CAAS,WAAA;AAAA,UAC1B,OAAM,EAAA,KAAM;AAIV,YAAA,MAAA,GAAS,MAAM,GAAG,EAAE,CAAA;AAAA,UACtB,CAAA;AAAA,UACA;AAAA;AAAA,YAEE,qBAAA,EAAuB;AAAA;AACzB,SACF;AACA,QAAA,OAAO,MAAA;AAAA,MACT,SAAS,CAAA,EAAG;AACV,QAAA,IAAA,CAAK,OAAA,CAAQ,MAAA,CAAO,KAAA,CAAM,CAAA,0BAAA,EAA6B,CAAC,CAAA,CAAE,CAAA;AAC1D,QAAA,MAAMC,wBAAa,CAAC,CAAA;AAAA,MACtB;AAAA,IACF,CAAA,EAAG,IAAA,CAAK,OAAA,CAAQ,QAAQ,CAAA;AAAA,EAC1B;AAAA,EAEA,MAAM,0BAAA,CACJ,QAAA,EACA,OAAA,EACe;AACf,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAM,EAAE,OAAO,QAAA,EAAU,QAAA,KAAa,MAAM,IAAA,CAAK,WAAA,CAAY,EAAA,EAAI,OAAO,CAAA;AAExE,IAAA,IAAI,SAAS,MAAA,EAAQ;AACnB,MAAA,MAAM,YAAA,GAAe,MAAMC,iEAAA,CAAiC;AAAA,QAC1D,IAAA,EAAM,EAAA;AAAA,QACN,UAAA,EAAY,QAAA;AAAA,QACZ,WAAW,OAAA,CAAQ;AAAA,OACpB,CAAA;AACD,MAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,KAAA;AAAA,QAClB,YAAY,YAAY,CAAA,WAAA,EAAc,IAAA,CAAK,SAAA,CAAU,QAAQ,CAAC,CAAA;AAAA,OAChE;AAAA,IACF;AAEA,IAAA,IAAI,MAAM,MAAA,EAAQ;AAUhB,MAAA,KAAA,MAAW,KAAA,IAASC,uBAAA,CAAO,KAAA,CAAM,KAAA,EAAO,EAAE,CAAA,EAAG;AAC3C,QAAA,IAAI;AACF,UAAA,MAAM,EAAA,CAAG,WAAA;AAAA,YACP,eAAA;AAAA,YACA,KAAA,CAAM,IAAI,CAAA,IAAA,MAAS;AAAA,cACjB,WAAWC,sBAAA,EAAK;AAAA,cAChB,UAAA,EAAYC,+BAAA,CAAmB,IAAA,CAAK,QAAA,CAAS,MAAM,CAAA;AAAA,cACnD,kBAAA,EAAoB,IAAA,CAAK,SAAA,CAAU,IAAA,CAAK,SAAS,MAAM,CAAA;AAAA,cACvD,kBAAkB,IAAA,CAAK,IAAA;AAAA,cACvB,MAAA,EAAQ,EAAA;AAAA,cACR,YAAA,EAAc,KAAK,QAAA,CAAS,WAAA;AAAA,cAC5B,cAAA,EAAgB,EAAA,CAAG,EAAA,CAAG,GAAA,EAAI;AAAA,cAC1B,iBAAA,EAAmB,EAAA,CAAG,EAAA,CAAG,GAAA;AAAI,aAC/B,CAAE,CAAA;AAAA,YACF;AAAA,WACF;AACA,UAAA,MAAM,EAAA,CAAG,WAAA;AAAA,YACP,0BAAA;AAAA,YACA,KAAA,CAAM,IAAI,CAAA,IAAA,MAAS;AAAA,cACjB,YAAY,OAAA,CAAQ,SAAA;AAAA,cACpB,iBAAA,EAAmBA,+BAAA,CAAmB,IAAA,CAAK,QAAA,CAAS,MAAM;AAAA,aAC5D,CAAE,CAAA;AAAA,YACF;AAAA,WACF;AAAA,QACF,SAAS,KAAA,EAAO;AACd,UAAA,IAAI,CAACC,wCAAA,CAAwB,KAAK,CAAA,EAAG;AACnC,YAAA,MAAM,KAAA;AAAA,UACR,CAAA,MAAO;AACL,YAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,KAAA;AAAA,cAClB,uDAAuD,KAAK,CAAA;AAAA,aAC9D;AACA,YAAA,QAAA,CAAS,IAAA,CAAK,GAAG,KAAK,CAAA;AAAA,UACxB;AAAA,QACF;AAAA,MACF;AAAA,IACF;AAEA,IAAA,IAAI,SAAS,MAAA,EAAQ;AACnB,MAAA,KAAA,MAAW;AAAA,QACT,QAAA,EAAU,EAAE,MAAA,EAAQ,WAAA,EAAY;AAAA,QAChC;AAAA,WACG,QAAA,EAAU;AACb,QAAA,MAAM,SAAA,GAAYD,gCAAmB,MAAM,CAAA;AAE3C,QAAA,IAAI;AACF,UAAA,IAAI,EAAA,GAAK,MAAME,+CAAA,CAAwB;AAAA,YACrC,EAAA;AAAA,YACA,MAAA;AAAA,YACA,IAAA;AAAA,YACA;AAAA,WACD,CAAA;AACD,UAAA,IAAI,CAAC,EAAA,EAAI;AACP,YAAA,EAAA,GAAK,MAAMC,+CAAA,CAAwB;AAAA,cACjC,EAAA;AAAA,cACA,MAAA;AAAA,cACA,IAAA;AAAA,cACA,WAAA;AAAA,cACA,MAAA,EAAQ,KAAK,OAAA,CAAQ;AAAA,aACtB,CAAA;AAAA,UACH;AACA,UAAA,IAAI,EAAA,EAAI;AACN,YAAA,MAAM,GAAgC,0BAA0B,CAAA,CAC7D,MAAM,mBAAA,EAAqB,SAAS,EACpC,MAAA,EAAO;AAEV,YAAA,MAAM,EAAA;AAAA,cACJ;AAAA,cACA,MAAA,CAAO;AAAA,cACP,YAAY,OAAA,CAAQ,SAAA;AAAA,cACpB,iBAAA,EAAmB;AAAA,aACpB,CAAA;AAAA,UACH,CAAA,MAAO;AACL,YAAA,MAAM,EAAA,CAAgC,0BAA0B,CAAA,CAC7D,KAAA,CAAM,qBAAqB,SAAS,CAAA,CACpC,QAAA,CAAS,EAAE,UAAA,EAAY,OAAA,CAAQ,SAAA,EAAW,EAC1C,MAAA,EAAO;AAEV,YAAA,MAAM,cAAA,GAAiB,MAAMC,iDAAA,CAAyB;AAAA,cACpD,EAAA;AAAA,cACA,SAAA;AAAA,cACA;AAAA,aACD,CAAA;AACD,YAAA,IAAI,cAAA,EAAgB;AAClB,cAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,IAAA;AAAA,gBAClB,CAAA,OAAA,EAAU,QAAQ,SAAS,CAAA,gCAAA,EAAmC,SAAS,CAAA,uBAAA,EAA0B,cAAc,iBAAiB,WAAW,CAAA;AAAA,eAC7I;AAAA,YACF;AAAA,UACF;AAAA,QACF,SAAS,KAAA,EAAO;AACd,UAAA,IAAIC,oBAAA,CAAgB,EAAA,EAAI,KAAK,CAAA,EAAG;AAC9B,YAAA,MAAM,KAAA;AAAA,UACR;AACA,UAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,KAAA;AAAA,YAClB,kBAAkB,SAAS,CAAA,eAAA,EAAkB,OAAA,CAAQ,SAAS,MAAM,KAAK,CAAA;AAAA,WAC3E;AAAA,QACF;AAAA,MACF;AAAA,IACF;AAAA,EACF;AAAA,EAEA,MAAM,wBAAwB,QAAA,EAA0C;AACtE,IAAA,MAAM,EAAA,GAAK,QAAA;AAEX,IAAA,MAAM,OAAO,MAAM,EAAA;AAAA,MACjB;AAAA,KACF,CACG,QAAA,CAAS,YAAY,CAAA,CACrB,aAAa,YAAY,CAAA;AAE5B,IAAA,OAAO,IAAA,CACJ,GAAA,CAAI,CAAA,GAAA,KAAO,GAAA,CAAI,UAAU,CAAA,CACzB,MAAA,CAAO,CAAC,GAAA,KAAuB,CAAC,CAAC,GAAG,CAAA;AAAA,EACzC;AAAA,EAEA,MAAM,oBAAA,CACJ,QAAA,EACA,OAAA,EACA;AACA,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAMC,0CAAqB,EAAE,EAAA,EAAI,IAAA,EAAM,OAAA,CAAQ,MAAM,CAAA;AAAA,EACvD;AAAA,EAEA,MAAc,WAAA,CACZ,EAAA,EACA,OAAA,EAKC;AACD,IAAA,IAAI,OAAA,CAAQ,SAAS,OAAA,EAAS;AAC5B,MAAA,MAAMC,MAAAA,GAAQ,IAAI,KAAA,EAAkD;AACpE,MAAA,MAAMC,SAAAA,GAAW,IAAI,KAAA,EAAkD;AACvE,MAAA,MAAMC,YAAW,OAAA,CAAQ,OAAA,CAAQ,GAAA,CAAI,CAAA,CAAA,KAAK,EAAE,SAAS,CAAA;AAErD,MAAA,KAAA,MAAW,SAASX,uBAAA,CAAO,KAAA,CAAM,OAAA,CAAQ,KAAA,EAAO,GAAI,CAAA,EAAG;AACrD,QAAA,MAAM,aAAa,KAAA,CAAM,GAAA,CAAI,OAAKE,+BAAA,CAAmB,CAAA,CAAE,MAAM,CAAC,CAAA;AAC9D,QAAA,MAAM,IAAA,GAAO,MAAM,EAAA,CAAsB,eAAe,EACrD,MAAA,CAAO,CAAC,YAAA,EAAc,kBAAA,EAAoB,cAAc,CAAC,CAAA,CACzD,OAAA,CAAQ,cAAc,UAAU,CAAA;AACnC,QAAA,MAAM,YAAY,IAAI,GAAA;AAAA,UACpB,IAAA,CAAK,IAAI,CAAA,GAAA,KAAO;AAAA,YACd,GAAA,CAAI,UAAA;AAAA,YACJ;AAAA,cACE,kBAAkB,GAAA,CAAI,gBAAA;AAAA,cACtB,cAAc,GAAA,CAAI;AAAA;AACpB,WACD;AAAA,SACH;AAEA,QAAA,KAAA,CAAM,OAAA,CAAQ,CAAC,QAAA,EAAU,CAAA,KAAM;AAC7B,UAAA,MAAM,SAAA,GAAY,WAAW,CAAC,CAAA;AAC9B,UAAA,MAAM,OAAA,GAAUU,uBAAA,CAAmB,QAAA,CAAS,MAAM,CAAA;AAClD,UAAA,MAAM,QAAA,GAAW,SAAA,CAAU,GAAA,CAAI,SAAS,CAAA;AACxC,UAAA,IAAI,aAAa,MAAA,EAAW;AAE1B,YAAAH,OAAM,IAAA,CAAK,EAAE,QAAA,EAAU,IAAA,EAAM,SAAS,CAAA;AAAA,UACxC,YACG,QAAA,CAAS,WAAA,IAAe,IAAA,OAAW,QAAA,CAAS,gBAAgB,IAAA,CAAA,EAC7D;AAEA,YAAAE,SAAAA,CAAS,KAAK,SAAS,CAAA;AACvB,YAAAF,OAAM,IAAA,CAAK,EAAE,QAAA,EAAU,IAAA,EAAM,SAAS,CAAA;AAAA,UACxC,CAAA,MAAA,IAAW,OAAA,KAAY,QAAA,CAAS,gBAAA,EAAkB;AAEhD,YAAAC,UAAS,IAAA,CAAK,EAAE,QAAA,EAAU,IAAA,EAAM,SAAS,CAAA;AAAA,UAC3C;AAAA,QACF,CAAC,CAAA;AAAA,MACH;AAEA,MAAA,OAAO,EAAE,KAAA,EAAAD,MAAAA,EAAO,QAAA,EAAAC,SAAAA,EAAU,UAAAC,SAAAA,EAAS;AAAA,IACrC;AAGA,IAAA,MAAM,UAAU,MAAM,EAAA;AAAA,MACpB;AAAA,KACF,CACG,SAA4B,eAAA,EAAiB;AAAA,MAC5C,iBAAA,EAAmB;AAAA,KACpB,EACA,KAAA,CAAM,EAAE,YAAY,OAAA,CAAQ,SAAA,EAAW,CAAA,CACvC,MAAA,CAAO;AAAA,MACN,iBAAA,EAAmB,4CAAA;AAAA,MACnB,YAAA,EAAc,4BAAA;AAAA,MACd,gBAAA,EAAkB;AAAA,KACnB,CAAA;AAEH,IAAA,MAAM,KAAA,GAAQ,OAAA,CAAQ,KAAA,CAAM,GAAA,CAAI,CAAA,QAAA,MAAa;AAAA,MAC3C,QAAA;AAAA,MACA,GAAA,EAAKT,+BAAA,CAAmB,QAAA,CAAS,MAAM,CAAA;AAAA,MACvC,IAAA,EAAMU,uBAAA,CAAmB,QAAA,CAAS,MAAM;AAAA,KAC1C,CAAE,CAAA;AAEF,IAAA,MAAM,aAAa,IAAI,GAAA;AAAA,MACrB,OAAA,CAAQ,IAAI,CAAA,CAAA,KAAK;AAAA,QACf,CAAA,CAAE,iBAAA;AAAA,QACF;AAAA,UACE,aAAa,CAAA,CAAE,YAAA;AAAA,UACf,eAAe,CAAA,CAAE;AAAA;AACnB,OACD;AAAA,KACH;AACA,IAAA,MAAM,UAAA,GAAa,IAAI,GAAA,CAAI,KAAA,CAAM,IAAI,CAAA,IAAA,KAAQ,IAAA,CAAK,GAAG,CAAC,CAAA;AAEtD,IAAA,MAAM,KAAA,GAAQ,IAAI,KAAA,EAAkD;AACpE,IAAA,MAAM,QAAA,GAAW,IAAI,KAAA,EAAkD;AACvE,IAAA,MAAM,QAAA,GAAW,OAAA,CACd,GAAA,CAAI,CAAA,GAAA,KAAO,GAAA,CAAI,iBAAiB,CAAA,CAChC,MAAA,CAAO,CAAA,GAAA,KAAO,CAAC,UAAA,CAAW,GAAA,CAAI,GAAG,CAAC,CAAA;AAErC,IAAA,KAAA,MAAW,QAAQ,KAAA,EAAO;AACxB,MAAA,MAAM,MAAA,GAAS,UAAA,CAAW,GAAA,CAAI,IAAA,CAAK,GAAG,CAAA;AACtC,MAAA,MAAM,aAAa,EAAE,QAAA,EAAU,KAAK,QAAA,EAAU,IAAA,EAAM,KAAK,IAAA,EAAK;AAC9D,MAAA,IAAI,CAAC,MAAA,EAAQ;AAEX,QAAA,KAAA,CAAM,KAAK,UAAU,CAAA;AAAA,MACvB,YACG,MAAA,CAAO,WAAA,IAAe,aACtB,IAAA,CAAK,QAAA,CAAS,eAAe,MAAA,CAAA,EAC9B;AAEA,QAAA,QAAA,CAAS,IAAA,CAAK,KAAK,GAAG,CAAA;AACtB,QAAA,KAAA,CAAM,KAAK,UAAU,CAAA;AAAA,MACvB,CAAA,MAAA,IAAW,MAAA,CAAO,aAAA,KAAkB,IAAA,CAAK,IAAA,EAAM;AAE7C,QAAA,QAAA,CAAS,KAAK,UAAU,CAAA;AAAA,MAC1B;AAAA,IACF;AAEA,IAAA,OAAO,EAAE,KAAA,EAAO,QAAA,EAAU,QAAA,EAAS;AAAA,EACrC;AACF;;;;"}
+{"version":3,"file":"DefaultProviderDatabase.cjs.js","sources":["../../src/database/DefaultProviderDatabase.ts"],"sourcesContent":["/*\n * Copyright 2021 The Backstage Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the License.\n * You may obtain a copy of the License at\n *\n *     http://www.apache.org/licenses/LICENSE-2.0\n *\n * Unless required by applicable law or agreed to in writing, software\n * distributed under the License is distributed on an \"AS IS\" BASIS,\n * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions and\n * limitations under the License.\n */\n\nimport { stringifyEntityRef } from '@backstage/catalog-model';\nimport { DeferredEntity } from '@backstage/plugin-catalog-node';\nimport { Knex } from 'knex';\nimport lodash from 'lodash';\nimport { randomUUID as uuid } from 'node:crypto';\nimport { rethrowError } from './conversion';\nimport { deleteWithEagerPruningOfChildren } from './operations/provider/deleteWithEagerPruningOfChildren';\nimport { refreshByRefreshKeys } from './operations/provider/refreshByRefreshKeys';\nimport { checkLocationKeyConflict } from './operations/refreshState/checkLocationKeyConflict';\nimport { insertUnprocessedEntity } from './operations/refreshState/insertUnprocessedEntity';\nimport { updateUnprocessedEntity } from './operations/refreshState/updateUnprocessedEntity';\nimport { DbRefreshStateReferencesRow, DbRefreshStateRow } from './tables';\nimport {\n  ProviderDatabase,\n  RefreshByKeyOptions,\n  ReplaceUnprocessedEntitiesOptions,\n  Transaction,\n} from './types';\nimport {\n  generateStableHash,\n  isDeadlockError,\n  retryOnDeadlock,\n  whereInArray,\n} from './util';\nimport {\n  LoggerService,\n  isDatabaseConflictError,\n} from '@backstage/backend-plugin-api';\n\n// The number of items that are sent per batch to the database layer, when\n// doing .batchInsert calls to knex. This needs to be low enough to not cause\n// errors in the underlying engine due to exceeding query limits, but large\n// enough to get the speed benefits.\nconst BATCH_SIZE = 50;\n\nexport class DefaultProviderDatabase implements ProviderDatabase {\n  private readonly options: {\n    database: Knex;\n    logger: LoggerService;\n  };\n\n  constructor(options: { database: Knex; logger: LoggerService }) {\n    this.options = options;\n  }\n\n  /**\n   * Executes `fn` inside a database transaction, retrying automatically on\n   * deadlock (PostgreSQL error 40P01). Because the callback may be invoked\n   * more than once, it must not perform non-database side effects such as\n   * emitting events, mutating in-memory state, or calling external services.\n   */\n  async transaction<T>(fn: (tx: Transaction) => Promise<T>): Promise<T> {\n    return retryOnDeadlock(async () => {\n      try {\n        let result: T | undefined = undefined;\n        await this.options.database.transaction(\n          async tx => {\n            // We can't return here, as knex swallows the return type in case the\n            // transaction is rolled back:\n            // https://github.com/knex/knex/blob/e37aeaa31c8ef9c1b07d2e4d3ec6607e557d800d/lib/transaction.js#L136\n            result = await fn(tx);\n          },\n          {\n            // If we explicitly trigger a rollback, don't fail.\n            doNotRejectOnRollback: true,\n          },\n        );\n        return result!;\n      } catch (e) {\n        this.options.logger.debug(`Error during transaction, ${e}`);\n        throw rethrowError(e);\n      }\n    }, this.options.database);\n  }\n\n  async replaceUnprocessedEntities(\n    txOpaque: Knex | Transaction,\n    options: ReplaceUnprocessedEntitiesOptions,\n  ): Promise<void> {\n    const tx = txOpaque as Knex | Knex.Transaction;\n    const { toAdd, toUpsert, toRemove } = await this.createDelta(tx, options);\n\n    if (toRemove.length) {\n      const removedCount = await deleteWithEagerPruningOfChildren({\n        knex: tx,\n        entityRefs: toRemove,\n        sourceKey: options.sourceKey,\n      });\n      this.options.logger.debug(\n        `removed, ${removedCount} entities: ${JSON.stringify(toRemove)}`,\n      );\n    }\n\n    if (toAdd.length) {\n      // The reason for this chunking, rather than just massively batch\n      // inserting the entire payload, is that we fall back to the individual\n      // upsert mechanism below on conflicts. That path is massively slower than\n      // the fast batch path, so we don't want to end up accidentally having to\n      // for example item-by-item upsert tens of thousands of entities in a\n      // large initial delivery dump. The implication is that the size of these\n      // chunks needs to weigh the benefit of fast successful inserts, against\n      // the drawback of super slow but more rare fallbacks. There's quickly\n      // diminishing returns though with turning up this value way high.\n      for (const chunk of lodash.chunk(toAdd, 50)) {\n        try {\n          await tx.batchInsert(\n            'refresh_state',\n            chunk.map(item => ({\n              entity_id: uuid(),\n              entity_ref: stringifyEntityRef(item.deferred.entity),\n              unprocessed_entity: JSON.stringify(item.deferred.entity),\n              unprocessed_hash: item.hash,\n              errors: '',\n              location_key: item.deferred.locationKey,\n              next_update_at: tx.fn.now(),\n              last_discovery_at: tx.fn.now(),\n            })),\n            BATCH_SIZE,\n          );\n          await tx.batchInsert(\n            'refresh_state_references',\n            chunk.map(item => ({\n              source_key: options.sourceKey,\n              target_entity_ref: stringifyEntityRef(item.deferred.entity),\n            })),\n            BATCH_SIZE,\n          );\n        } catch (error) {\n          if (!isDatabaseConflictError(error)) {\n            throw error;\n          } else {\n            this.options.logger.debug(\n              `Fast insert path failed, falling back to slow path, ${error}`,\n            );\n            toUpsert.push(...chunk);\n          }\n        }\n      }\n    }\n\n    if (toUpsert.length) {\n      for (const {\n        deferred: { entity, locationKey },\n        hash,\n      } of toUpsert) {\n        const entityRef = stringifyEntityRef(entity);\n\n        try {\n          let ok = await updateUnprocessedEntity({\n            tx,\n            entity,\n            hash,\n            locationKey,\n          });\n          if (!ok) {\n            ok = await insertUnprocessedEntity({\n              tx,\n              entity,\n              hash,\n              locationKey,\n              logger: this.options.logger,\n            });\n          }\n          if (ok) {\n            await tx<DbRefreshStateReferencesRow>('refresh_state_references')\n              .where('target_entity_ref', entityRef)\n              .delete();\n\n            await tx<DbRefreshStateReferencesRow>(\n              'refresh_state_references',\n            ).insert({\n              source_key: options.sourceKey,\n              target_entity_ref: entityRef,\n            });\n          } else {\n            await tx<DbRefreshStateReferencesRow>('refresh_state_references')\n              .where('target_entity_ref', entityRef)\n              .andWhere({ source_key: options.sourceKey })\n              .delete();\n\n            const conflictingKey = await checkLocationKeyConflict({\n              tx,\n              entityRef,\n              locationKey,\n            });\n            if (conflictingKey) {\n              this.options.logger.warn(\n                `Source ${options.sourceKey} detected conflicting entityRef ${entityRef} already referenced by ${conflictingKey} and now also ${locationKey}`,\n              );\n            }\n          }\n        } catch (error) {\n          if (isDeadlockError(tx, error)) {\n            throw error;\n          }\n          this.options.logger.error(\n            `Failed to add '${entityRef}' from source '${options.sourceKey}', ${error}`,\n          );\n        }\n      }\n    }\n  }\n\n  async listReferenceSourceKeys(txOpaque: Transaction): Promise<string[]> {\n    const tx = txOpaque as Knex | Knex.Transaction;\n\n    const rows = await tx<DbRefreshStateReferencesRow>(\n      'refresh_state_references',\n    )\n      .distinct('source_key')\n      .whereNotNull('source_key');\n\n    return rows\n      .map(row => row.source_key)\n      .filter((key): key is string => !!key);\n  }\n\n  async refreshByRefreshKeys(\n    txOpaque: Transaction,\n    options: RefreshByKeyOptions,\n  ) {\n    const tx = txOpaque as Knex.Transaction;\n    await refreshByRefreshKeys({ tx, keys: options.keys });\n  }\n\n  private async createDelta(\n    tx: Knex | Knex.Transaction,\n    options: ReplaceUnprocessedEntitiesOptions,\n  ): Promise<{\n    toAdd: { deferred: DeferredEntity; hash: string }[];\n    toUpsert: { deferred: DeferredEntity; hash: string }[];\n    toRemove: string[];\n  }> {\n    if (options.type === 'delta') {\n      const toAdd = new Array<{ deferred: DeferredEntity; hash: string }>();\n      const toUpsert = new Array<{ deferred: DeferredEntity; hash: string }>();\n      const toRemove = options.removed.map(e => e.entityRef);\n\n      for (const chunk of lodash.chunk(options.added, 1000)) {\n        const entityRefs = chunk.map(e => stringifyEntityRef(e.entity));\n        const rows = await tx<DbRefreshStateRow>('refresh_state')\n          .select(['entity_ref', 'unprocessed_hash', 'location_key'])\n          .where(whereInArray('entity_ref', entityRefs));\n        const oldStates = new Map(\n          rows.map(row => [\n            row.entity_ref,\n            {\n              unprocessed_hash: row.unprocessed_hash,\n              location_key: row.location_key,\n            },\n          ]),\n        );\n\n        chunk.forEach((deferred, i) => {\n          const entityRef = entityRefs[i];\n          const newHash = generateStableHash(deferred.entity);\n          const oldState = oldStates.get(entityRef);\n          if (oldState === undefined) {\n            // Add any entity that does not exist in the database\n            toAdd.push({ deferred, hash: newHash });\n          } else if (\n            (deferred.locationKey ?? null) !== (oldState.location_key ?? null)\n          ) {\n            // Remove and then re-add any entity that exists, but with a different location key\n            toRemove.push(entityRef);\n            toAdd.push({ deferred, hash: newHash });\n          } else if (newHash !== oldState.unprocessed_hash) {\n            // Entities with modifications should be pushed through too\n            toUpsert.push({ deferred, hash: newHash });\n          }\n        });\n      }\n\n      return { toAdd, toUpsert, toRemove };\n    }\n\n    // Grab all of the existing references from the same source, and their locationKeys as well\n    const oldRefs = await tx<DbRefreshStateReferencesRow>(\n      'refresh_state_references',\n    )\n      .leftJoin<DbRefreshStateRow>('refresh_state', {\n        target_entity_ref: 'entity_ref',\n      })\n      .where({ source_key: options.sourceKey })\n      .select({\n        target_entity_ref: 'refresh_state_references.target_entity_ref',\n        location_key: 'refresh_state.location_key',\n        unprocessed_hash: 'refresh_state.unprocessed_hash',\n      });\n\n    const items = options.items.map(deferred => ({\n      deferred,\n      ref: stringifyEntityRef(deferred.entity),\n      hash: generateStableHash(deferred.entity),\n    }));\n\n    const oldRefsSet = new Map(\n      oldRefs.map(r => [\n        r.target_entity_ref,\n        {\n          locationKey: r.location_key,\n          oldEntityHash: r.unprocessed_hash,\n        },\n      ]),\n    );\n    const newRefsSet = new Set(items.map(item => item.ref));\n\n    const toAdd = new Array<{ deferred: DeferredEntity; hash: string }>();\n    const toUpsert = new Array<{ deferred: DeferredEntity; hash: string }>();\n    const toRemove = oldRefs\n      .map(row => row.target_entity_ref)\n      .filter(ref => !newRefsSet.has(ref));\n\n    for (const item of items) {\n      const oldRef = oldRefsSet.get(item.ref);\n      const upsertItem = { deferred: item.deferred, hash: item.hash };\n      if (!oldRef) {\n        // Add any entity that does not exist in the database\n        toAdd.push(upsertItem);\n      } else if (\n        (oldRef.locationKey ?? undefined) !==\n        (item.deferred.locationKey ?? undefined)\n      ) {\n        // Remove and then re-add any entity that exists, but with a different location key\n        toRemove.push(item.ref);\n        toAdd.push(upsertItem);\n      } else if (oldRef.oldEntityHash !== item.hash) {\n        // Entities with modifications should be pushed through too\n        toUpsert.push(upsertItem);\n      }\n    }\n\n    return { toAdd, toUpsert, toRemove };\n  }\n}\n"],"names":["retryOnDeadlock","rethrowError","deleteWithEagerPruningOfChildren","lodash","uuid","stringifyEntityRef","isDatabaseConflictError","updateUnprocessedEntity","insertUnprocessedEntity","checkLocationKeyConflict","isDeadlockError","refreshByRefreshKeys","toAdd","toUpsert","toRemove","whereInArray","generateStableHash"],"mappings":";;;;;;;;;;;;;;;;;;AAiDA,MAAM,UAAA,GAAa,EAAA;AAEZ,MAAM,uBAAA,CAAoD;AAAA,EAC9C,OAAA;AAAA,EAKjB,YAAY,OAAA,EAAoD;AAC9D,IAAA,IAAA,CAAK,OAAA,GAAU,OAAA;AAAA,EACjB;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,EAQA,MAAM,YAAe,EAAA,EAAiD;AACpE,IAAA,OAAOA,qBAAgB,YAAY;AACjC,MAAA,IAAI;AACF,QAAA,IAAI,MAAA,GAAwB,KAAA,CAAA;AAC5B,QAAA,MAAM,IAAA,CAAK,QAAQ,QAAA,CAAS,WAAA;AAAA,UAC1B,OAAM,EAAA,KAAM;AAIV,YAAA,MAAA,GAAS,MAAM,GAAG,EAAE,CAAA;AAAA,UACtB,CAAA;AAAA,UACA;AAAA;AAAA,YAEE,qBAAA,EAAuB;AAAA;AACzB,SACF;AACA,QAAA,OAAO,MAAA;AAAA,MACT,SAAS,CAAA,EAAG;AACV,QAAA,IAAA,CAAK,OAAA,CAAQ,MAAA,CAAO,KAAA,CAAM,CAAA,0BAAA,EAA6B,CAAC,CAAA,CAAE,CAAA;AAC1D,QAAA,MAAMC,wBAAa,CAAC,CAAA;AAAA,MACtB;AAAA,IACF,CAAA,EAAG,IAAA,CAAK,OAAA,CAAQ,QAAQ,CAAA;AAAA,EAC1B;AAAA,EAEA,MAAM,0BAAA,CACJ,QAAA,EACA,OAAA,EACe;AACf,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAM,EAAE,OAAO,QAAA,EAAU,QAAA,KAAa,MAAM,IAAA,CAAK,WAAA,CAAY,EAAA,EAAI,OAAO,CAAA;AAExE,IAAA,IAAI,SAAS,MAAA,EAAQ;AACnB,MAAA,MAAM,YAAA,GAAe,MAAMC,iEAAA,CAAiC;AAAA,QAC1D,IAAA,EAAM,EAAA;AAAA,QACN,UAAA,EAAY,QAAA;AAAA,QACZ,WAAW,OAAA,CAAQ;AAAA,OACpB,CAAA;AACD,MAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,KAAA;AAAA,QAClB,YAAY,YAAY,CAAA,WAAA,EAAc,IAAA,CAAK,SAAA,CAAU,QAAQ,CAAC,CAAA;AAAA,OAChE;AAAA,IACF;AAEA,IAAA,IAAI,MAAM,MAAA,EAAQ;AAUhB,MAAA,KAAA,MAAW,KAAA,IAASC,uBAAA,CAAO,KAAA,CAAM,KAAA,EAAO,EAAE,CAAA,EAAG;AAC3C,QAAA,IAAI;AACF,UAAA,MAAM,EAAA,CAAG,WAAA;AAAA,YACP,eAAA;AAAA,YACA,KAAA,CAAM,IAAI,CAAA,IAAA,MAAS;AAAA,cACjB,WAAWC,sBAAA,EAAK;AAAA,cAChB,UAAA,EAAYC,+BAAA,CAAmB,IAAA,CAAK,QAAA,CAAS,MAAM,CAAA;AAAA,cACnD,kBAAA,EAAoB,IAAA,CAAK,SAAA,CAAU,IAAA,CAAK,SAAS,MAAM,CAAA;AAAA,cACvD,kBAAkB,IAAA,CAAK,IAAA;AAAA,cACvB,MAAA,EAAQ,EAAA;AAAA,cACR,YAAA,EAAc,KAAK,QAAA,CAAS,WAAA;AAAA,cAC5B,cAAA,EAAgB,EAAA,CAAG,EAAA,CAAG,GAAA,EAAI;AAAA,cAC1B,iBAAA,EAAmB,EAAA,CAAG,EAAA,CAAG,GAAA;AAAI,aAC/B,CAAE,CAAA;AAAA,YACF;AAAA,WACF;AACA,UAAA,MAAM,EAAA,CAAG,WAAA;AAAA,YACP,0BAAA;AAAA,YACA,KAAA,CAAM,IAAI,CAAA,IAAA,MAAS;AAAA,cACjB,YAAY,OAAA,CAAQ,SAAA;AAAA,cACpB,iBAAA,EAAmBA,+BAAA,CAAmB,IAAA,CAAK,QAAA,CAAS,MAAM;AAAA,aAC5D,CAAE,CAAA;AAAA,YACF;AAAA,WACF;AAAA,QACF,SAAS,KAAA,EAAO;AACd,UAAA,IAAI,CAACC,wCAAA,CAAwB,KAAK,CAAA,EAAG;AACnC,YAAA,MAAM,KAAA;AAAA,UACR,CAAA,MAAO;AACL,YAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,KAAA;AAAA,cAClB,uDAAuD,KAAK,CAAA;AAAA,aAC9D;AACA,YAAA,QAAA,CAAS,IAAA,CAAK,GAAG,KAAK,CAAA;AAAA,UACxB;AAAA,QACF;AAAA,MACF;AAAA,IACF;AAEA,IAAA,IAAI,SAAS,MAAA,EAAQ;AACnB,MAAA,KAAA,MAAW;AAAA,QACT,QAAA,EAAU,EAAE,MAAA,EAAQ,WAAA,EAAY;AAAA,QAChC;AAAA,WACG,QAAA,EAAU;AACb,QAAA,MAAM,SAAA,GAAYD,gCAAmB,MAAM,CAAA;AAE3C,QAAA,IAAI;AACF,UAAA,IAAI,EAAA,GAAK,MAAME,+CAAA,CAAwB;AAAA,YACrC,EAAA;AAAA,YACA,MAAA;AAAA,YACA,IAAA;AAAA,YACA;AAAA,WACD,CAAA;AACD,UAAA,IAAI,CAAC,EAAA,EAAI;AACP,YAAA,EAAA,GAAK,MAAMC,+CAAA,CAAwB;AAAA,cACjC,EAAA;AAAA,cACA,MAAA;AAAA,cACA,IAAA;AAAA,cACA,WAAA;AAAA,cACA,MAAA,EAAQ,KAAK,OAAA,CAAQ;AAAA,aACtB,CAAA;AAAA,UACH;AACA,UAAA,IAAI,EAAA,EAAI;AACN,YAAA,MAAM,GAAgC,0BAA0B,CAAA,CAC7D,MAAM,mBAAA,EAAqB,SAAS,EACpC,MAAA,EAAO;AAEV,YAAA,MAAM,EAAA;AAAA,cACJ;AAAA,cACA,MAAA,CAAO;AAAA,cACP,YAAY,OAAA,CAAQ,SAAA;AAAA,cACpB,iBAAA,EAAmB;AAAA,aACpB,CAAA;AAAA,UACH,CAAA,MAAO;AACL,YAAA,MAAM,EAAA,CAAgC,0BAA0B,CAAA,CAC7D,KAAA,CAAM,qBAAqB,SAAS,CAAA,CACpC,QAAA,CAAS,EAAE,UAAA,EAAY,OAAA,CAAQ,SAAA,EAAW,EAC1C,MAAA,EAAO;AAEV,YAAA,MAAM,cAAA,GAAiB,MAAMC,iDAAA,CAAyB;AAAA,cACpD,EAAA;AAAA,cACA,SAAA;AAAA,cACA;AAAA,aACD,CAAA;AACD,YAAA,IAAI,cAAA,EAAgB;AAClB,cAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,IAAA;AAAA,gBAClB,CAAA,OAAA,EAAU,QAAQ,SAAS,CAAA,gCAAA,EAAmC,SAAS,CAAA,uBAAA,EAA0B,cAAc,iBAAiB,WAAW,CAAA;AAAA,eAC7I;AAAA,YACF;AAAA,UACF;AAAA,QACF,SAAS,KAAA,EAAO;AACd,UAAA,IAAIC,oBAAA,CAAgB,EAAA,EAAI,KAAK,CAAA,EAAG;AAC9B,YAAA,MAAM,KAAA;AAAA,UACR;AACA,UAAA,IAAA,CAAK,QAAQ,MAAA,CAAO,KAAA;AAAA,YAClB,kBAAkB,SAAS,CAAA,eAAA,EAAkB,OAAA,CAAQ,SAAS,MAAM,KAAK,CAAA;AAAA,WAC3E;AAAA,QACF;AAAA,MACF;AAAA,IACF;AAAA,EACF;AAAA,EAEA,MAAM,wBAAwB,QAAA,EAA0C;AACtE,IAAA,MAAM,EAAA,GAAK,QAAA;AAEX,IAAA,MAAM,OAAO,MAAM,EAAA;AAAA,MACjB;AAAA,KACF,CACG,QAAA,CAAS,YAAY,CAAA,CACrB,aAAa,YAAY,CAAA;AAE5B,IAAA,OAAO,IAAA,CACJ,GAAA,CAAI,CAAA,GAAA,KAAO,GAAA,CAAI,UAAU,CAAA,CACzB,MAAA,CAAO,CAAC,GAAA,KAAuB,CAAC,CAAC,GAAG,CAAA;AAAA,EACzC;AAAA,EAEA,MAAM,oBAAA,CACJ,QAAA,EACA,OAAA,EACA;AACA,IAAA,MAAM,EAAA,GAAK,QAAA;AACX,IAAA,MAAMC,0CAAqB,EAAE,EAAA,EAAI,IAAA,EAAM,OAAA,CAAQ,MAAM,CAAA;AAAA,EACvD;AAAA,EAEA,MAAc,WAAA,CACZ,EAAA,EACA,OAAA,EAKC;AACD,IAAA,IAAI,OAAA,CAAQ,SAAS,OAAA,EAAS;AAC5B,MAAA,MAAMC,MAAAA,GAAQ,IAAI,KAAA,EAAkD;AACpE,MAAA,MAAMC,SAAAA,GAAW,IAAI,KAAA,EAAkD;AACvE,MAAA,MAAMC,YAAW,OAAA,CAAQ,OAAA,CAAQ,GAAA,CAAI,CAAA,CAAA,KAAK,EAAE,SAAS,CAAA;AAErD,MAAA,KAAA,MAAW,SAASX,uBAAA,CAAO,KAAA,CAAM,OAAA,CAAQ,KAAA,EAAO,GAAI,CAAA,EAAG;AACrD,QAAA,MAAM,aAAa,KAAA,CAAM,GAAA,CAAI,OAAKE,+BAAA,CAAmB,CAAA,CAAE,MAAM,CAAC,CAAA;AAC9D,QAAA,MAAM,OAAO,MAAM,EAAA,CAAsB,eAAe,CAAA,CACrD,OAAO,CAAC,YAAA,EAAc,kBAAA,EAAoB,cAAc,CAAC,CAAA,CACzD,KAAA,CAAMU,iBAAA,CAAa,YAAA,EAAc,UAAU,CAAC,CAAA;AAC/C,QAAA,MAAM,YAAY,IAAI,GAAA;AAAA,UACpB,IAAA,CAAK,IAAI,CAAA,GAAA,KAAO;AAAA,YACd,GAAA,CAAI,UAAA;AAAA,YACJ;AAAA,cACE,kBAAkB,GAAA,CAAI,gBAAA;AAAA,cACtB,cAAc,GAAA,CAAI;AAAA;AACpB,WACD;AAAA,SACH;AAEA,QAAA,KAAA,CAAM,OAAA,CAAQ,CAAC,QAAA,EAAU,CAAA,KAAM;AAC7B,UAAA,MAAM,SAAA,GAAY,WAAW,CAAC,CAAA;AAC9B,UAAA,MAAM,OAAA,GAAUC,uBAAA,CAAmB,QAAA,CAAS,MAAM,CAAA;AAClD,UAAA,MAAM,QAAA,GAAW,SAAA,CAAU,GAAA,CAAI,SAAS,CAAA;AACxC,UAAA,IAAI,aAAa,MAAA,EAAW;AAE1B,YAAAJ,OAAM,IAAA,CAAK,EAAE,QAAA,EAAU,IAAA,EAAM,SAAS,CAAA;AAAA,UACxC,YACG,QAAA,CAAS,WAAA,IAAe,IAAA,OAAW,QAAA,CAAS,gBAAgB,IAAA,CAAA,EAC7D;AAEA,YAAAE,SAAAA,CAAS,KAAK,SAAS,CAAA;AACvB,YAAAF,OAAM,IAAA,CAAK,EAAE,QAAA,EAAU,IAAA,EAAM,SAAS,CAAA;AAAA,UACxC,CAAA,MAAA,IAAW,OAAA,KAAY,QAAA,CAAS,gBAAA,EAAkB;AAEhD,YAAAC,UAAS,IAAA,CAAK,EAAE,QAAA,EAAU,IAAA,EAAM,SAAS,CAAA;AAAA,UAC3C;AAAA,QACF,CAAC,CAAA;AAAA,MACH;AAEA,MAAA,OAAO,EAAE,KAAA,EAAAD,MAAAA,EAAO,QAAA,EAAAC,SAAAA,EAAU,UAAAC,SAAAA,EAAS;AAAA,IACrC;AAGA,IAAA,MAAM,UAAU,MAAM,EAAA;AAAA,MACpB;AAAA,KACF,CACG,SAA4B,eAAA,EAAiB;AAAA,MAC5C,iBAAA,EAAmB;AAAA,KACpB,EACA,KAAA,CAAM,EAAE,YAAY,OAAA,CAAQ,SAAA,EAAW,CAAA,CACvC,MAAA,CAAO;AAAA,MACN,iBAAA,EAAmB,4CAAA;AAAA,MACnB,YAAA,EAAc,4BAAA;AAAA,MACd,gBAAA,EAAkB;AAAA,KACnB,CAAA;AAEH,IAAA,MAAM,KAAA,GAAQ,OAAA,CAAQ,KAAA,CAAM,GAAA,CAAI,CAAA,QAAA,MAAa;AAAA,MAC3C,QAAA;AAAA,MACA,GAAA,EAAKT,+BAAA,CAAmB,QAAA,CAAS,MAAM,CAAA;AAAA,MACvC,IAAA,EAAMW,uBAAA,CAAmB,QAAA,CAAS,MAAM;AAAA,KAC1C,CAAE,CAAA;AAEF,IAAA,MAAM,aAAa,IAAI,GAAA;AAAA,MACrB,OAAA,CAAQ,IAAI,CAAA,CAAA,KAAK;AAAA,QACf,CAAA,CAAE,iBAAA;AAAA,QACF;AAAA,UACE,aAAa,CAAA,CAAE,YAAA;AAAA,UACf,eAAe,CAAA,CAAE;AAAA;AACnB,OACD;AAAA,KACH;AACA,IAAA,MAAM,UAAA,GAAa,IAAI,GAAA,CAAI,KAAA,CAAM,IAAI,CAAA,IAAA,KAAQ,IAAA,CAAK,GAAG,CAAC,CAAA;AAEtD,IAAA,MAAM,KAAA,GAAQ,IAAI,KAAA,EAAkD;AACpE,IAAA,MAAM,QAAA,GAAW,IAAI,KAAA,EAAkD;AACvE,IAAA,MAAM,QAAA,GAAW,OAAA,CACd,GAAA,CAAI,CAAA,GAAA,KAAO,GAAA,CAAI,iBAAiB,CAAA,CAChC,MAAA,CAAO,CAAA,GAAA,KAAO,CAAC,UAAA,CAAW,GAAA,CAAI,GAAG,CAAC,CAAA;AAErC,IAAA,KAAA,MAAW,QAAQ,KAAA,EAAO;AACxB,MAAA,MAAM,MAAA,GAAS,UAAA,CAAW,GAAA,CAAI,IAAA,CAAK,GAAG,CAAA;AACtC,MAAA,MAAM,aAAa,EAAE,QAAA,EAAU,KAAK,QAAA,EAAU,IAAA,EAAM,KAAK,IAAA,EAAK;AAC9D,MAAA,IAAI,CAAC,MAAA,EAAQ;AAEX,QAAA,KAAA,CAAM,KAAK,UAAU,CAAA;AAAA,MACvB,YACG,MAAA,CAAO,WAAA,IAAe,aACtB,IAAA,CAAK,QAAA,CAAS,eAAe,MAAA,CAAA,EAC9B;AAEA,QAAA,QAAA,CAAS,IAAA,CAAK,KAAK,GAAG,CAAA;AACtB,QAAA,KAAA,CAAM,KAAK,UAAU,CAAA;AAAA,MACvB,CAAA,MAAA,IAAW,MAAA,CAAO,aAAA,KAAkB,IAAA,CAAK,IAAA,EAAM;AAE7C,QAAA,QAAA,CAAS,KAAK,UAAU,CAAA;AAAA,MAC1B;AAAA,IACF;AAEA,IAAA,OAAO,EAAE,KAAA,EAAO,QAAA,EAAU,QAAA,EAAS;AAAA,EACrC;AACF;;;;"}

package/dist/database/operations/stitcher/buildEntitySearch.cjs.js

@@ -34,7 +34,7 @@ function traverse(root) {
         visit(path, item);
         if (typeof item === "string") {
           const pathKey = `${path}.${item}`;
-          const lowerKey = pathKey.toLocaleLowerCase("en-US");
+          const lowerKey = pathKey.toLowerCase();
           if (!seenPathKeys.has(lowerKey)) {
             seenPathKeys.add(lowerKey);
             output.push({ key: pathKey, value: true });
@@ -53,7 +53,7 @@ function traverse(root) {
 function mapToRows(input, entityId) {
   const result = [];
   for (const { key: rawKey, value: rawValue } of input) {
-    const key = rawKey.toLocaleLowerCase("en-US");
+    const key = rawKey.toLowerCase();
     if (key.length > MAX_KEY_LENGTH) {
       continue;
     }
@@ -65,7 +65,7 @@ function mapToRows(input, entityId) {
         value: null
       });
     } else {
-      const value = String(rawValue).toLocaleLowerCase("en-US");
+      const value = String(rawValue).toLowerCase();
       if (value.length <= MAX_VALUE_LENGTH) {
         result.push({
           entity_id: entityId,
@@ -100,11 +100,11 @@ function buildEntitySearch(entityId, entity) {
     });
   }
   const keys = new Set(raw.map((r) => r.key));
-  const lowerKeys = new Set(raw.map((r) => r.key.toLocaleLowerCase("en-US")));
+  const lowerKeys = new Set(raw.map((r) => r.key.toLowerCase()));
   if (keys.size !== lowerKeys.size) {
     const difference = [];
     for (const key of keys) {
-      const lower = key.toLocaleLowerCase("en-US");
+      const lower = key.toLowerCase();
       if (!lowerKeys.delete(lower)) {
         difference.push(lower);
       }