aiiinotate 0.3.2 → 0.4.1

package/docs/endpoints.md

@@ -1,8 +1,273 @@
 # Endpoints
 
+## Introductory notes
+
+**aiiinotate** is meant to be able to handle both IIIF presentation APIs: the most common [2.x](https://iiif.io/api/presentation/2.1) and the more recent [3.x](https://iiif.io/api/presentation/3.0). Both APIs define a data structure for manifests, annotations, lists of annotations and collections of manifests.
+
+**HOWEVER, in aiiinotate, IIIF Presentation v2 and v3 data are isolated**: they form two separate collections, and no conversion is done between IIIF 2.x and 3.x data. This means that:
+- **when communicating with aiiinotate**, you must specify a **IIIF presentation version in the query URL**. In the docs, this is described by the  `iiif_version` keyword.
+- **when inserting/updating data**, the data structure you provide must match the URL's `iiif_version`: you can't insert an annotation in v3 if your `iiif_version` is `2`.
+- **when searching for data**, if you inserted an annotation in v3, you must search for it with `iiif_version = 3`.
+- **TLDR**:
+    - your data must match the `iiif_version` argument
+    - if you insert an Annotation following the API v3.x, you can't search for it using `iiif_version=2`.
+
+This is because 
+- the IIIF standard is quite complex and there are breaking changes between v2 and v3
+- handling conversions between v2 and v3 is error prone, would increase calculations and slow the app down
+
+---
+
+## Generic routes
+
+### IIIF search API
+
+```
+GET /search-api/{iiif_version}/manifests/{manifest_short_id}/search
+```
+
+Implementation of the [IIIF Search API](https://iiif.io/api/search/2.0/), to search one or several annotations within a manifest.
+
+#### Request
+
+- Variables:
+    - `iiif_version` (`2 | 3`): the IIIF aearch API version. 2 is for IIIF Presentation API 3.x, 1 is for IIIF Presentation API 2.x
+    - `manifest_short_id` (`string`): the ID of the manifest. See the *IIIF URIs* section. 
+- Parameters:
+    - `q` (`string`): query string. 
+        - if `iiif_version=1`, `q` is searched in the fields: `@id`, `resource.@id` or `resource.chars` fields
+    - `motivation` (`painting | non-painting | commenting | describing | tagging | linking`): values for the `motivation` field of an annotation
+
+#### Response
+
+Returns a JSON. If `iiif_version` is `1`, an `AnnotationList` is returned. Otherwise, an `AnnotationPage` is returned.
+
+#### Notes
+
+- if `q` and `motivation` are unused, it will return all annotations for the manifest
+- only exact matches are allowed for `q` and `motivation`
+
+---
+
+### Delete an annotation or a manifest
+
+```
+DELETE /{collection_name}/{iiif_version}/delete
+```
+
+#### Request
+
+- Variables
+    - `collection_name` (`annotations | manifests`): delete an annotation or a manifest
+    - `iiif_version` (`2 | 3`): IIIF presentation version
+- Parameters:
+    - if `collection_name = manifests`:
+        - `uri`: the full URI of the manifest to delete
+        - `manifestShortId`: the manifest's identifier
+    - if `collection_name = annotation`:
+        - `uri`: the full URI of the annotation to delete
+        - `manifestShortId`: a manifest's identifier, to delete all annotations for a manifest
+        - `canvasUri`: the full URI to an annotation's target canvas, to delete all annotatons for the canvas
+
+#### Response
+
+```
+{ deletedCount: <integer> }
+```
+
+--- 
+
+## Manifests routes
+
+### Get an index of all manifests
+
+```
+GET /manifests/{iiif_version}
+```
+
+Returns a Collection of all manifests in your **aiiinotate** instance.
+
+#### Request
+
+- Variables:
+    - `iiif_version` (`2 | 3`): the IIIF Presentation API version
+
+#### Response
+
+A IIIF `Collection`, following the IIIF Presentation API 2 or 3, depending of the value of `iiif_version`.
+
+---
+
+### Insert a manifest
+
+```
+POST /manifests/{iiif_version}/create
+```
+
+#### Request
+
+- Variable: 
+    - `iiif_version` (`2 | 3`): the IIIF Presentation API version of your manifest
+- Body (`JSON`): the manifest  to index in the database
+
+#### Response
+
+```
+{ 
+    insertedIds: string[],
+    preExistingIds: string[],
+    rejectedIds: []
+}
+```
+
+- `insertedIds`: the list of IDs of inserted manifests
+- `preExisingIds`: the IDs of manifests that were aldready in the database
+- `rejectedIds`: the IDs of manifests on which an error occurred
+
+---
+
+## Annotation routes
+
+### Get all annotations for a canvas
+
+```
+GET /annotations/{iiif_version}/search
+```
+
+#### Request
+
+- Variables:
+    - `iiif_version` (`2 | 3`): the IIIF Presentation API of your manifests
+- Parameters:
+    - `uri` (`string`): the URI of the target canvas
+    - `asAnnotationList` (`true | false`):  format of the response
+
+#### Response
+
+`Object[] | Object`: if `true`, return an array of annotations. Otherwise, return an `AnnotationList`.
+
+---
+
+### Count annotations
+
+```
+GET /annotations/{iiif_version}/count
+```
+
+#### Request
+
+- Variables:
+    - `iiif_version` (`2 | 3`): the IIIF Presentation API of your manifests
+- Parameters:
+    - `uri` (`string`): the annotation's `@id`
+    - `canvasUri` (`string`): the annotation's target canvas (`on.full`)
+    - `manifestShortId` (`string`): the short ID of the annotation's target manifest (`on.manifestShortId`)
+
+#### Response
+
+```
+{ count: integer }
+```
+
+---
+
+### Get a single annotation
+
+```
+GET /data/{iiif_version}/{manifest_short_id}/annotation/{annotation_short_id}
+```
+
+This route allows to query an annotation by its ID by defering its `@id | id` field. This URL follows the IIIF specification
+
+#### Request
+
+- Variables:
+    - `iiif_version` (`2 | 3`): the IIIF version of the annotation
+    - `manifest_short_id` (`string`): the identifier of the manifest the annotation is related to
+    - `annotation_short_id`: the unique part of the annotation URL
+
+#### Response
+
+`Object`: the annotation. Its format follows the IIIF Presentation specification 2 or 3, based on the value of `iiif_version`.
+
+---
+
+### Create/update an annotation
+
+```
+POST /annotations/{iiif_version}/{action}
+```
+
+Create or update a single annotation
+
+#### Request
+
+- Variables:
+    - `iiif_version` (`2 | 3`): the IIIF version of the annotation
+    - `action` (`create | update`): the action to perform: create or update an annotation
+- Body (`Object`): a IIIF annotation that follows the IIIF Presentation API 2 or 3 (depending on the value of `iiif_version`)
+
+#### Response
+
+```
+{ 
+    insertedIds: string[],
+    preExistingIds: string[],
+    rejectedIds: []
+}
+```
+
+#### Notes
+
+- A side effect of inserting annotations is inserting the related manifests.
+- When inserting an annotation, the annotation's target manifest is also fetched and inserted in the database
+- Annotations in `aiiinotate` contain 3 nonstandard fields. In IIIF presentation 2.x,
+    - `annotation.on[0].manifestUri`: the URI of the manifest on which is an annotation
+    - `annotation.on[0].manifestShortId`: the unique identifier of the manifest on which is an annotation
+    - `annotation.on[0].canvasIdx`: the position of an annotation's target canvas within the target manifest, as an integer
+    - this depends on reconstructing an annotation's target manifest URL and fetching it. If this process fails, the fields above will be `undefined`.
+    - the annotation's target's manifest is fetched and inserted in the database, if possible, and stored in `annotation.on[0].manifestShortId`
+
+
+---
+
+### Insert several annotations
+
+```
+POST /annotations/{iiif_version}/createMany
+```
+
+Batch insert multiple annotations.
+
+#### Request
+
+- Parameters: 
+    - `iiif_version` (`2 | 3`): the IIIF version of the annotation
+- Body: either:
+    - a full `AnnotationList | AnnotationPage` embedded in the body (type must match `iiif_version`: AnnotationPage for IIIF 3, AnnotationList for IIIF 2).
+    - `AnnotationList[] | AnnotationPage[]` (type must match `iiif_version`): an array of annotation lists or pages
+    - `{ uri: string }`: an object containing a reference to an `AnnotationList` or `AnnotationPage`
+    - `{ uri: string }[]`: an array of objects containing a reference to an `AnnotationList` or `AnnotationPage`.
+
+#### Response
+
+```
+{ 
+    insertedIds: string[],
+    preExistingIds: string[],
+    rejectedIds: []
+}
+```
+
+#### Notes
+
+- Be wary of maximum body size, especially when sending AnnotationLists in your body. If possible, using `{ uri: string }` is better.
+- All annotations within a single AnnotationList/Page may have different target canvases or manifests.
+- See **Create/update an annotation**.
+
 ---
 
-## URL prefixes
+## Appending 1: App logic: URL prefixes
 
 URL anatomy is a mix of [SAS endpoints](./specifications/4_sas.md) and IIIF specifications. In turn, we define the following prefixes:
 
@@ -27,22 +292,23 @@ Where:
 
 There is an extra URL prefix: `schemas`. It is only used internally (not accessible to clients or accessible through HTTP) to define the IDs of all JsonSchemas, so we won't talk about it here.
 
+
 ---
 
-## IIIF URIs
+## Appendix 2: IIIF URIs
 
 IIIF URIs in the Presentation 2.1 API are:
 
 ```
 Collection 	         {scheme}://{host}/{prefix}/collection/{name}
-Manifest 	         {scheme}://{host}/{prefix}/{identifier}/manifest
-Sequence 	         {scheme}://{host}/{prefix}/{identifier}/sequence/{name}
-Canvas 	                 {scheme}://{host}/{prefix}/{identifier}/canvas/{name}
-Annotation (incl images) {scheme}://{host}/{prefix}/{identifier}/annotation/{name}
-AnnotationList           {scheme}://{host}/{prefix}/{identifier}/list/{name}
-Range 	                 {scheme}://{host}/{prefix}/{identifier}/range/{name}
-Layer 	                 {scheme}://{host}/{prefix}/{identifier}/layer/{name}
-Content 	         {scheme}://{host}/{prefix}/{identifier}/res/{name}.{format}
+Manifest 	         {scheme}://{host}/{prefix}/{manifest_short_id}/manifest
+Sequence 	         {scheme}://{host}/{prefix}/{manifest_short_id}/sequence/{name}
+Canvas 	                 {scheme}://{host}/{prefix}/{manifest_short_id}/canvas/{name}
+Annotation (incl images) {scheme}://{host}/{prefix}/{manifest_short_id}/annotation/{name}
+AnnotationList           {scheme}://{host}/{prefix}/{manifest_short_id}/list/{name}
+Range 	                 {scheme}://{host}/{prefix}/{manifest_short_id}/range/{name}
+Layer 	                 {scheme}://{host}/{prefix}/{manifest_short_id}/layer/{name}
+Content 	         {scheme}://{host}/{prefix}/{manifest_short_id}/res/{name}.{format}
 ```
 
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "aiiinotate",
-  "version": "0.3.2",
+  "version": "0.4.1",
   "description": "a fast IIIF-compliant annotation server",
   "main": "./cli/index.js",
   "type": "module",

package/scripts/get_version.py

@@ -0,0 +1,12 @@
+import re
+import sys
+import json
+import shutil
+import pathlib
+
+curdir = pathlib.Path(__file__).parent.resolve()
+pkg_file = curdir.parent.joinpath("package.json").resolve()
+
+with open(pkg_file, mode="r") as fh:
+    data = json.load(fh)
+    print(f"\naiiinotate current version: {data['version']}\n")

package/scripts/update_version.py

@@ -27,4 +27,4 @@ for fp in [pkg_file, pkg_lock_file]:
     with open(fp, mode="w") as fh:
         json.dump(data, fh, indent=2)
 
-print(f"\nUpdated NPM package to version: {version}.")
+print(f"\nUpdated NPM package to version: {version}.\n")

package/src/data/annotations/annotations2.js

@@ -58,6 +58,24 @@ class Annotations2 extends CollectionAbstract {
   ////////////////////////////////////////////////////////////////
   // utils
 
+  /**
+   * expand a pair of `filterKey`, `filterVal` following the schema `routeAnnotationFilter` into a proper filter for the `annotations2` collection.
+   * @param {string} filterKey
+   * @param {string} filterVal
+   * @returns
+   */
+  #expandRouteAnnotationFilter(filterKey, filterVal) {
+    const allowedFilterKeys = [ "uri", "manifestShortId", "canvasUri" ];
+    if ( !allowedFilterKeys.includes(filterKey) ) {
+      throw new Error(`${this.funcname(this.#expandRouteAnnotationFilter)}: expected one of ${allowedFilterKeys} for param 'deleteKey', got '${filterKey}'`)
+    }
+    return  filterKey==="uri"
+      ? { "@id": filterVal }
+      : filterKey==="canvasUri"
+        ? { "on.full": filterVal }
+        : { "on.manifestShortId": filterVal };
+  }
+
   /**
    * clean the body of an annotation (annotation.resource).
    * if `annotation.resource` is an array (there are several bodies associated to that annotation), this function must be called on each item of the array
@@ -183,7 +201,7 @@ class Annotations2 extends CollectionAbstract {
     // 1. get all distinct manifest URIs
     const manifestUris = [];
     annotationData.map((ann) => ann.on.map((target) => {
-      if ( target.manifestUri !== undefined && !manifestUris.includes(target.manifestUri) ) {
+      if ( target.manifestUri != null && !manifestUris.includes(target.manifestUri) ) {
         manifestUris.push(target.manifestUri);
       }
     }));
@@ -275,19 +293,12 @@ class Annotations2 extends CollectionAbstract {
    * @returns {Promise<DeleteResponseType>}
    */
   async deleteAnnotations(deleteKey, deleteVal) {
-
-    const allowedDeleteKey = [ "uri", "manifestShortId", "canvasUri" ];
-    if ( !allowedDeleteKey.includes(deleteKey) ) {
-      throw this.deleteError(`${this.funcName(this.deleteAnnotations)}: expected one of ${allowedDeleteKey} for param 'deleteKey', got '${deleteKey}'`)
+    try {
+      const deleteFilter = this.#expandRouteAnnotationFilter(deleteKey, deleteVal);
+      return this.delete(deleteFilter);
+    } catch (err) {
+      throw this.deleteError(`${this.funcName(this.deleteAnnotations)}: ${err.message}`)
     }
-
-    const deleteFilter =
-      deleteKey==="uri"
-        ? { "@id": deleteVal }
-        : deleteKey==="canvasUri"
-          ? { "on.full": deleteVal }
-          : { "on.manifestShortId": deleteVal };
-    return this.delete(deleteFilter);
   }
 
   ////////////////////////////////////////////////////////////////
@@ -340,7 +351,7 @@ class Annotations2 extends CollectionAbstract {
    *
    * NOTE:
    *  - only `motivation` and `q` search params are implemented
-   *  - to increase search execution, ONLY EXACT STRING MACHES are
+   *  - to increase search execution speed, ONLY EXACT STRING MACHES are
    *    implemented for `q` and `motivation` (in the IIIF specs, you can supply
    *    multiple space-separated values and the server should return all partial
    *    matches to any of those strings.)
@@ -410,6 +421,23 @@ class Annotations2 extends CollectionAbstract {
     return this.collection.findOne({ "@id": annotationUri })
   }
 
+  /**
+   * count number of annotations.
+   * @param {string} filterKey
+   * @param {string} filterVal
+   * @returns
+   */
+  async count(filterKey, filterVal) {
+    try {
+      const
+        countFilter = this.#expandRouteAnnotationFilter(filterKey, filterVal),
+        count = await this.collection.countDocuments(countFilter);
+      return { count: count }
+    } catch (err) {
+      throw this.readError(`${this.funcName(this.count)}: ${err.message}`)
+    }
+  }
+
 }
 
 export default fastifyPlugin((fastify, options, done) => {

package/src/data/annotations/routes.js

@@ -64,9 +64,11 @@ function annotationsRoutes(fastify, options, done) {
     annotations2 = fastify.annotations2,
     /** @type {Annotations3InstanceType} */
     annotations3 = fastify.annotations3,
-    iiifPresentationVersionSchema = fastify.schemasBase.getSchema("presentation"),
     routeAnnotation2Or3Schema = fastify.schemasRoutes.getSchema("routeAnnotation2Or3"),
     routeAnnotationCreateManySchema = fastify.schemasRoutes.getSchema("routeAnnotationCreateMany"),
+    routeAnnotationFilterSchema = fastify.schemasRoutes.getSchema("routeAnnotationFilter"),
+    routeResponseCountSchema = fastify.schemasRoutes.getSchema("routeResponseCount"),
+    iiifPresentationVersionSchema = fastify.schemasBase.getSchema("presentation"),
     iiifAnnotationListSchema = fastify.schemasPresentation2.getSchema("annotationList"),
     iiifAnnotation2ArraySchema = fastify.schemasPresentation2.getSchema("annotationArray"),
     iiifAnnotation2Schema = fastify.schemasPresentation2.getSchema("annotation"),
@@ -122,6 +124,36 @@ function annotationsRoutes(fastify, options, done) {
     }
   );
 
+  fastify.get(
+    "/annotations/:iiifPresentationVersion/count",
+    {
+      schema: {
+        params: {
+          type: "object",
+          properties: {
+            iiifPresentationVersion: iiifPresentationVersionSchema
+          }
+        },
+        querystring: routeAnnotationFilterSchema,
+        response: makeResponseSchema(
+          fastify, routeResponseCountSchema
+        )
+      }
+    },
+    async (request, reply) => {
+      const
+        { iiifPresentationVersion } = request.params,
+        [ filterKey, filterVal ] = getFirstNonEmptyPair(request.query);
+      try {
+        return iiifPresentationVersion === 2
+          ? await annotations2.count(filterKey, filterVal)
+          : annotations3.notImplementedError();
+      } catch (err) {
+        returnError(request, reply, err);
+      }
+    }
+  )
+
   /** retrieve a single annotation by its "@id"|"id". this route defers an annotation */
   fastify.get(
     "/data/:iiifPresentationVersion/:manifestShortId/annotation/:annotationShortId",

package/src/data/annotations/routes.test.js

@@ -176,5 +176,39 @@ test("test annotation Routes", async (t) => {
     )
   })
 
+  await t.test("test route /annotations/:iiifPresentationVersion/count", async (t) => {
+    const expectedOnCount = (annotationArray, onKey, expectedOnVal) =>
+      annotationArray.filter((anno) => anno.on.some(x => x[onKey] === expectedOnVal)).length
+
+    await injectTestAnnotations(fastify, t, annotationList);
+    const
+      annotationArray = await fastify.mongo.db.collection("annotations2").find().toArray(),
+      annotationId = getRandomItem(annotationArray)["@id"],
+      canvasUri = getRandomItem(annotationArray).on[0].full,
+      manifestShortId = getRandomItem(annotationArray).on[0].manifestShortId,
+      expectedAnnotationIdCount = 1,
+      expectedCanvasUriCount = expectedOnCount(annotationArray, "full", canvasUri),
+      expectedManifestShortIdCount = expectedOnCount(annotationArray, "manifestShortId", manifestShortId),
+      mapper = [
+        ["uri", annotationId, expectedAnnotationIdCount],
+        ["canvasUri", canvasUri, expectedCanvasUriCount],
+        ["manifestShortId", manifestShortId, expectedManifestShortIdCount]
+      ];
+
+    await Promise.all(
+      mapper.map(async ([filterKey, filterVal, expectedCount]) => {
+        const
+          r = await fastify.inject({
+            method: "GET",
+            url: `/annotations/2/count?${filterKey}=${filterVal}`
+          }),
+          body = await r.json();
+        t.assert.deepStrictEqual(r.statusCode, 200);
+        t.assert.deepStrictEqual(body.count, expectedCount);
+      })
+    )
+
+  })
+
   return
 })

package/src/data/manifests/manifests2.js

@@ -147,9 +147,11 @@ class Manifests2 extends CollectionAbstract {
     const
       mongoResponse =
         await this.collection.find(
-          { "@id": {
-            $in: cleanManifestArray.map((manifest) => manifest["@id"])
-          }},
+          {
+            "@id": {
+              $in: cleanManifestArray.map((manifest) => manifest["@id"])
+            }
+          },
           { projection: { "@id": 1 } }
         )
           .toArray(),
@@ -206,7 +208,10 @@ class Manifests2 extends CollectionAbstract {
     await Promise.all(
       manifestUriArray.map(async (manifestUri) => {
         try {
-          manifestArray.push(await this.#fetchManifestFromUri(manifestUri));
+          const r = await this.#fetchManifestFromUri(manifestUri);
+          if ( ! r.error ) {
+            manifestArray.push(r);
+          }
         } catch (err) {
           if ( throwOnError ) {
             throw err;

package/src/data/routes.js

@@ -29,10 +29,10 @@ function commonRoutes(fastify, options, done) {
     routeDeleteSchema = fastify.schemasRoutes.getSchema("routeDelete"),
     responsePostSchema = makeResponsePostSchema(fastify),
     validatorRouteAnnotationDeleteSchema = ajvCompile(fastify.schemasResolver(
-      fastify.schemasRoutes.getSchema("routeAnnotationDelete")
+      fastify.schemasRoutes.getSchema("routeAnnotationFilter")
     )),
     validatorRouteManifestDeleteSchema = ajvCompile(fastify.schemasResolver(
-      fastify.schemasRoutes.getSchema("routeManifestDelete")
+      fastify.schemasRoutes.getSchema("routeManifestFilter")
     ));
 
   fastify.get(

package/src/schemas/schemasRoutes.js

@@ -163,8 +163,9 @@ function addSchemas(fastify, options, done) {
     ]
   })
 
+  // for annotations: key-value pairs to filter a manifests collection by
   fastify.addSchema({
-    $id: makeSchemaUri("routeAnnotationDelete"),
+    $id: makeSchemaUri("routeAnnotationFilter"),
     oneOf: [
       {
         type: "object",
@@ -200,8 +201,9 @@ function addSchemas(fastify, options, done) {
     ]
   });
 
+  // for manifests: key-value pairs to filter a manifests collection by
   fastify.addSchema({
-    $id: makeSchemaUri("routeManifestDelete"),
+    $id: makeSchemaUri("routeManifestFilter"),
     type: "object",
     oneOf: [
       {
@@ -224,8 +226,8 @@ function addSchemas(fastify, options, done) {
     $id: makeSchemaUri("routeDelete"),
     type: "object",
     oneOf: [
-      { $ref: makeSchemaUri("routeAnnotationDelete") },
-      { $ref: makeSchemaUri("routeManifestDelete") },
+      { $ref: makeSchemaUri("routeAnnotationFilter") },
+      { $ref: makeSchemaUri("routeManifestFilter") },
     ]
   });
 
@@ -288,6 +290,18 @@ function addSchemas(fastify, options, done) {
     }
   });
 
+  fastify.addSchema({
+    $id: makeSchemaUri("routeResponseCount"),
+    type: "object",
+    required: [ "count" ],
+    properties: {
+      count: {
+        type: "integer",
+        minimum: 0
+      }
+    }
+  })
+
   ////////////////////////////////////////////////////////
 
   fastify.decorate("schemasRoutes", {