@paulhectork/aiiinotate 0.12.0

package/docker/docker.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+
+set -e;
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+# docker specific config
+ENV_DOCKER="$SCRIPT_DIR/.env"
+
+if [ ! -f "$ENV_DOCKER" ];
+then echo ".env file not found at '$ENV_DOCKER'. exiting..."; exit 1
+fi
+
+build_containers () {
+    docker compose  build;
+}
+
+start_containers() {
+    docker compose up -d;
+}
+
+stop_containers() {
+    docker compose down;
+}
+
+case "$1" in
+    start)
+        start_containers
+        ;;
+    stop)
+        stop_containers
+        ;;
+    build)
+        stop_containers
+        build_containers
+        start_containers
+        ;;
+    *)
+        echo "Usage: $0 {build|start|stop}"
+        exit 1
+        ;;
+esac;
+

package/docker/docker_aiiinotate_import.sh

@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+
+# this script imports AnnotationLists from a directory
+# in an aiiinotate instance. the directory must only contain
+# AnnotationLists, with all files at the root of the directory.
+#
+# USAGE: bash docker_aiiinotate_import.sh <"annotations"|"manifests"> <path/to/directory>
+
+# "annotation" or "manifest": the type of data to import.
+DATATYPE="$1"
+
+# directory containing all files we want to import into aiiinotate
+HOST_DIR=$(echo "$2" | sed -e "s~/$~~g")  # sed removes trailing "/"
+
+CONTAINER="docker-aiiinotate-1"
+
+EXEC="docker exec $CONTAINER"
+NODE_BIN=$($EXEC realpath node_modules/.bin)
+DOTENVX_BIN=$($EXEC realpath $NODE_BIN/dotenvx)
+AIIINOTATE_BIN=$($EXEC realpath $NODE_BIN/aiiinotate)
+
+# .env file in container.
+CONTAINER_ENV_FILE=$($EXEC realpath .env)
+# directory where files in $IMPORT_DIR will be stored in the container.
+CONTAINER_DIR="import_files"
+# file listing absolute paht to contents of $CONTAINER_DIR
+CONTAINER_INDEX_FILE=import_files.txt
+
+[[ "$DATATYPE" != "annotation" && "$DATATYPE" != "manifest" ]] && {
+  echo "error: DATATYPE must be 'annotation' or 'manifest' (got: '$DATATYPE')"
+  exit 1
+}
+
+[ ! -d "$HOST_DIR" ] && {
+    echo "error: import directory '$HOST_DIR' not found ! exiting";
+    exit 1
+}
+
+# recreate CONTAINER_DIR
+$EXEC bash -c "[ -d \"$CONTAINER_DIR\" ] && rm -rf \"$CONTAINER_DIR\"";
+$EXEC mkdir "$CONTAINER_DIR"
+
+# covert paths to abspath
+CONTAINER_DIR=$($EXEC realpath "$CONTAINER_DIR")
+CONTAINER_INDEX_FILE=$($EXEC realpath "$CONTAINER_INDEX_FILE")
+
+# copy all files in $HOST_DIR to the docker in $CONTAINER_DIR.
+# if a file in $HOST_DIR is not found, exit.
+for fp in "$HOST_DIR"/*; do
+    [ ! -f "$fp" ] && {
+        echo "file not found on host: '$fp'. exiting.";
+        exit 1
+    }
+    docker container cp "$fp" "$CONTAINER":"$CONTAINER_DIR"
+done
+
+$EXEC bash -c "echo \$(find \"$CONTAINER_DIR\"/* -type f | xargs readlink -f > import_files.txt)"
+$EXEC cat "$CONTAINER_INDEX_FILE"
+
+$EXEC bash -c \
+    "$DOTENVX_BIN run -f \"$CONTAINER_ENV_FILE\" -- \
+    \"$AIIINOTATE_BIN\" import \"$DATATYPE\" -i 2 -f \"$CONTAINER_INDEX_FILE\""

package/docs/api.md

@@ -0,0 +1,381 @@
+# API
+
+This document describes **aiiinotate**'s API.
+
+## Introductory notes
+
+### Terminology
+
+In the docs below,
+
+- `Parameters` describes route parameters (dynamic segments of a route's URL).
+- `Query` describes the query string in a key-value format
+
+### IIIF Version
+
+**aiiinotate** is meant to be able to handle both IIIF presentation APIs: the most common [2.x](https://iiif.io/api/presentation/2.1) and the more recent [3.x](https://iiif.io/api/presentation/3.0). Both APIs define a data structure for manifests, annotations, lists of annotations and collections of manifests.
+
+**HOWEVER, in aiiinotate, v2 and v3 data are isolated**: they form two separate collections, and no conversion is done between IIIF 2.x and 3.x data. This means that:
+- **when communicating with aiiinotate**, you must specify a **IIIF presentation version in the query URL**. In the docs, this is described by the  `iiif_version` keyword.
+- **when inserting/updating data**, the data structure you provide must match the URL's `iiif_version`: you can't insert an annotation in v3 if your `iiif_version` is `2`.
+- **TLDR**:
+    - your data must match the `iiif_version` argument
+    - if you insert an Annotation following the API v3.x, you can't search for it using `iiif_version=2`.
+
+This is because
+- the IIIF standard is quite complex and there are breaking changes between v2 and v3
+- handling conversions between v2 and v3 is error prone, would increase calculations and slow the app down
+
+---
+
+## Generic routes
+
+### IIIF search API
+
+```
+GET /search-api/{iiif_version}/manifests/{manifest_short_id}/search
+```
+
+Implementation of the [IIIF Search API](https://iiif.io/api/search/2.0/), to search one or several annotations within a manifest.
+
+#### Request
+
+- Parameters:
+    - `iiif_version` (`2 | 3`): the IIIF aearch API version. 2 is for IIIF Presentation API 3.x, 1 is for IIIF Presentation API 2.x
+    - `manifest_short_id` (`string`): the ID of the manifest. See the *IIIF URIs* section.
+- Query:
+    - `q` (`string`): query string.
+        - if `iiif_version=1`, `q` is searched in the fields: `@id`, `resource.@id` or `resource.chars` fields
+    - `motivation` (`painting | non-painting | commenting | describing | tagging | linking`): values for the `motivation` field of an annotation
+    - `canvasMin` (`number`): a positive integer
+    - `canvasMax` (`number`): a positive integer
+        - `canvasMax` must be greater than `canvasMin`
+        - if `canvasMax` is undefined, then we will only return the annotations that target a canvas at the `canvasMin` position in its manifest.
+    - `page` (`number`): results are paginated. This specifies the page number
+    - `pageSize` (`number`): number of annotations to display per page. Defaults to `process.env.AIIINOTATE_PAGE_SIZE`.
+    - `onlyIds` (`boolean`): return just the value of `@id` fields of matched annotations as a `string[]` instead of returning all the annotations. If `onlyIds=true`, there is no pagination, `page` and `pageSize` won't have any effect.
+
+#### Response
+
+```
+AnnotationList | AnnotationPage | string[]
+```
+
+- if `onlyIds=true`, return a `string[]` (array of the IDs of all matched annotations)
+- otherwise,
+    - if `iiif_version` is `1`, return an `AnnotationList`
+    - else, return an `AnnotationPage`.
+
+#### Notes
+
+- `canvasMin`, `canvasMax` and `onlyIds` are non-standard query parameters that are NOT part of the IIIF Search API specification.
+- If `q` and `motivation` are unused, it will return all annotations for the manifest
+- Only exact matches are allowed for `q` and `motivation`
+- About `canvasMin` and `canvasMax`:
+    - they are used to search for annotations that target a range of canvases: for example, fetch all anotations between pages 3 and 30 of a manuscript.
+    - they are **0-indexed**: the 1st canvas of a manifest is indexed `0`, the 2nd is indexed `1`...
+    - See section [Create/update an annotation](#createupdate-an-annotation) for more information and possible issues with canvas indexes.
+
+---
+
+### Delete an annotation or a manifest
+
+```
+DELETE /{collection_name}/{iiif_version}/delete
+```
+
+#### Request
+
+- Parameters:
+    - `collection_name` (`annotations | manifests`): delete an annotation or a manifest
+    - `iiif_version` (`2 | 3`): IIIF presentation version
+- Query:
+    - if `collection_name = manifests`:
+        - `uri`: the full URI of the manifest to delete
+        - `manifestShortId`: the manifest's identifier
+    - if `collection_name = annotation`:
+        - `uri`: the full URI of the annotation to delete
+        - `manifestShortId`: a manifest's identifier, to delete all annotations for a manifest
+        - `canvasUri`: the full URI to an annotation's target canvas, to delete all annotatons for the canvas
+        - `tag`: delete all annotations with a certain tag in a manifest
+            - MUST be used in conjunction with `manifestShortId` to indicate on which manifest to delete annotations by tag: `delete?tag=<your-tag>&manifestShortId=<your-manifest-id>`
+
+#### Response
+
+```
+{ deletedCount: <integer> }
+```
+
+---
+
+## Manifests routes
+
+### Get an index of all manifests
+
+```
+GET /manifests/{iiif_version}
+```
+
+Returns a Collection of all manifests in your **aiiinotate** instance.
+
+#### Request
+
+- Parameters:
+    - `iiif_version` (`2 | 3`): the IIIF Presentation API version
+
+#### Response
+
+A IIIF `Collection`, following the IIIF Presentation API 2 or 3, depending of the value of `iiif_version`.
+
+---
+
+### Insert a manifest
+
+```
+POST /manifests/{iiif_version}/create
+```
+
+#### Request
+
+- Parameters:
+    - `iiif_version` (`2 | 3`): the IIIF Presentation API version of your manifest
+- Body (`JSON`): the manifest  to index in the database
+
+#### Response
+
+```
+{
+    insertedIds: string[],
+    preExistingIds: string[],
+    rejectedIds: []
+}
+```
+
+- `insertedIds`: the list of IDs of inserted manifests
+- `preExisingIds`: the IDs of manifests that were aldready in the database
+- `rejectedIds`: the IDs of manifests on which an error occurred
+
+---
+
+## Annotation routes
+
+### Get all annotations for a canvas
+
+```
+GET /annotations/{iiif_version}/search
+```
+
+#### Request
+
+- Parameters:
+    - `iiif_version` (`2 | 3`): the IIIF Presentation API of your manifests
+- Query:
+    - `canvasUri` (`string`): the URI of the target canvas
+    - `page` (`number`): results are paginated. Specifies the page number.
+    - `pageSize` (`number`): number of items per page. Defaults to `process.env.AIIINOTATE_PAGE_SIZE`.
+
+#### Response
+
+Results are paginated.
+
+```
+AnnotationList | AnnotationPage
+```
+
+---
+
+### Count annotations
+
+```
+GET /annotations/{iiif_version}/count
+```
+
+#### Request
+
+- Parameters:
+    - `iiif_version` (`2 | 3`): the IIIF Presentation API of your manifests
+- Query:
+    - `uri` (`string`): the annotation's `@id`
+    - `canvasUri` (`string`): the annotation's target canvas (`on.full`)
+    - `manifestShortId` (`string`): the short ID of the annotation's target manifest (`on.manifestShortId`)
+
+#### Response
+
+```
+{ count: integer }
+```
+
+---
+
+### Get a single annotation
+
+```
+GET /data/{iiif_version}/{manifest_short_id}/annotation/{annotation_short_id}
+```
+
+This route allows to query an annotation by its ID by defering its `@id | id` field. This URL follows the IIIF specification
+
+#### Request
+
+- Parameters:
+    - `iiif_version` (`2 | 3`): the IIIF version of the annotation
+    - `manifest_short_id` (`string`): the identifier of the manifest the annotation is related to
+    - `annotation_short_id`: the unique part of the annotation URL
+
+#### Response
+
+`Object`: the annotation. Its format follows the IIIF Presentation specification 2 or 3, based on the value of `iiif_version`.
+
+---
+
+### Create/update an annotation
+
+```
+POST /annotations/{iiif_version}/{action}
+```
+
+Create or update a single annotation
+
+#### Request
+
+- Parameters:
+    - `iiif_version` (`2 | 3`): the IIIF version of the annotation
+    - `action` (`create | update`): the action to perform: create or update an annotation
+- Query:
+    - `throwOnCanvasIndexError` (`boolean`): throw an error if there's a problem fetching the annotation's target canvas index. See [Appendix 1](#appendix-1-annotation-canvas-index-and-bounding-box-calculation)
+    - `throwOnXywhError` (`boolean`): throw an error if target bounding box calculation fails. See [Appendix 1](#appendix-1-annotation-canvas-index-and-bounding-box-calculation)
+- Body (`Object`): a IIIF annotation that follows the IIIF Presentation API 2 or 3 (depending on the value of `iiif_version`)
+
+#### Response
+
+```
+{
+    insertedIds: string[],
+    preExistingIds: string[],
+    rejectedIds: string[],
+    fetchErrorIds: string[]
+}
+```
+
+---
+
+### Insert several annotations
+
+```
+POST /annotations/{iiif_version}/createMany
+```
+
+Batch insert multiple annotations.
+
+#### Request
+
+- Parameters:
+    - `iiif_version` (`2 | 3`): the IIIF version of the annotation
+- Query:
+    - `throwOnCanvasIndexError` (`boolean`): throw an error if there's a problem fetching the annotation's target canvas index. See [Appendix 1](#appendix-1-annotation-canvas-index-and-bounding-box-calculation)
+    - `throwOnXywhError` (`boolean`): throw an error if target bounding box calculation fails. See [Appendix 1](#appendix-1-annotation-canvas-index-and-bounding-box-calculation)
+- Body: either:
+    - a full `AnnotationList | AnnotationPage` embedded in the body (type must match `iiif_version`: AnnotationPage for IIIF 3, AnnotationList for IIIF 2).
+    - `AnnotationList[] | AnnotationPage[]` (type must match `iiif_version`): an array of annotation lists or pages
+    - `{ uri: string }`: an object containing a reference to an `AnnotationList` or `AnnotationPage`
+    - `{ uri: string }[]`: an array of objects containing a reference to an `AnnotationList` or `AnnotationPage`.
+
+#### Response
+
+```
+{
+    insertedIds: string[],
+    preExistingIds: string[],
+    rejectedIds: []
+}
+```
+
+#### Notes
+
+- Calling this route using parallel processes (i.e., `Promise.all`) can cause data races which will cause inserts to fail
+- Be wary of maximum body size, especially when sending AnnotationLists in your body. If possible, using `{ uri: string }` is better.
+- All annotations within a single AnnotationList/Page may have different target canvases or manifests.
+- See **Create/update an annotation**.
+
+---
+
+## Appendix 1: annotation canvas index and bounding box calculation
+
+Two non-standard side-effects happen when inserting an annotation:
+- the annotation's target manifest is also fetched and inserted in the database. The annotation's target canvas index is fetched and added to the annotation.
+- calculating the XYWH bounding box of an annotation.
+
+Annotations in `aiiinotate` contain nonstandard fields. In IIIF presentation 2.x,
+- `manifestUri`, `manifestShortId`, `canvasIdx`:
+    - `annotation.on.manifestUri`: the URI of the manifest on which is an annotation
+    - `annotation.on.manifestShortId`: the unique identifier of the manifest on which is an annotation
+    - `annotation.on.canvasIdx`: the position of an annotation's target canvas within the target manifest, as an integer. `canvasIdx` is **0-indexed**: the 1st canvas in a manifest is indexed from `0`
+    - this relies on reconstructing an annotation's target manifest URL and fetching it. If this process fails, the fields above will be `undefined`.
+    - fetching an annotation's target manfest is error prone: it depends on the manifest being available through HTTP, which is not in our control.
+    - **you can control error throwing** (throw an error if a canvas index can't be found) using:
+        - at route-level, the `throwOnCanvasIndexError` query parameter
+        - at app-level, the `AIIINOTATE_STRICT_MODE` env variable.
+        - if `AIIINOTATE_STRICT_MODE` or `throwOnCanvasIndexError`, an error will be thrown if an error appears anywhere in the process of fetching the target manifest and populating the `canvasIdx` field.
+- `annotation.on.xywh`: the annotation's bounding box on its target canvas.
+    - when inserting an annotation, aiiinotate will attempt to extract its XYWH bounding box and store it as an `[number,number,number,number]`.
+    - this only works for `oa:SvgSelectors`, `oa:FragmentSeletors`, `oa:Choice` containing either an `SvgSelector` or a `FragmentSelector`, or a string-target (an URI with a `#xywh=` fragment selector).
+    - **you can control error-throwing** (throw an error if a bounding-box can't be calculated) using:
+        - at route-level, the `throwOnXywhError` query parameter
+        - at app-level, the `AIIINOTATE_STRICT_MODE` env variable.
+
+About `throwOnCanvasIndexError`, `throwOnXywhError` and `AIIINOTATE_STRICT_MODE`:
+- as indicated above, fetching an annotation's target canvas index and calculating an annotation's target bounding boxc is error prone.
+- in turn, you can decide wether or not to throw errors using:
+    - at route-level, the `throwOnCanvasIndexError` (for canvas index) and `throwOnXywhError` (for bounding box) query parameters
+    - at app-level, the `AIIINOTATE_STRICT_MODE` env variable. Setting this to `true` will set the value of `throwOnCanvasIndexError` and `throwOnXywhError` to `true` by default.
+    - the value of `AIIINOTATE_STRICT_MODE` can be overridden by setting a value to query parameters `throwOnCanvasIndexError` and `throwOnXywhError`.
+- if those are set to `true`, an error will be thrown if a canvas index can't be found. Otherwise, the values will be set to `undefined`.
+- in controlled environments where you know your manifests will be available and where you rely heavily on the `canvasIdx` field (like AIKON), throwing an error will ensure that the `canvasIdx` field is always defined.
+
+---
+
+## Appendix 2: App logic: URL prefixes
+
+URL anatomy is a mix of [SAS endpoints](./specifications/4_sas.md) and IIIF specifications. In turn, we define the following prefixes:
+
+- `data`: for all IIIF URIs: URIs of annotations and annotation lists
+- `annotations`: operations on annotations
+- `manifests`: operations on manifests
+- `search-api`: endpoint to access the IIIF search API
+
+In turn, URL anatomy is:
+
+```
+{host}/{prefix}/{iiif_version}/{slug}
+```
+
+Where:
+- `host`: the host of your app
+- `prefix`: `data | annotations | manifests`
+- `iiif_version`:
+    - if `prefix` is `search-api`, `1 | 2`: IIIF Search API version used
+    - otherwise, `2 | 3`, the IIIF Presentation API version your data is in
+- `slug`: the rest of the qurty URI
+
+There is an extra URL prefix: `schemas`. It is only used internally (not accessible to clients or accessible through HTTP) to define the IDs of all JsonSchemas, so we won't talk about it here.
+
+
+---
+
+## Appendix 3: IIIF URIs
+
+IIIF URIs in the Presentation 2.1 API are:
+
+```
+Collection 	         {scheme}://{host}/{prefix}/collection/{name}
+Manifest 	         {scheme}://{host}/{prefix}/{manifest_short_id}/manifest
+Sequence 	         {scheme}://{host}/{prefix}/{manifest_short_id}/sequence/{name}
+Canvas 	                 {scheme}://{host}/{prefix}/{manifest_short_id}/canvas/{name}
+Annotation (incl images) {scheme}://{host}/{prefix}/{manifest_short_id}/annotation/{name}
+AnnotationList           {scheme}://{host}/{prefix}/{manifest_short_id}/list/{name}
+Range 	                 {scheme}://{host}/{prefix}/{manifest_short_id}/range/{name}
+Layer 	                 {scheme}://{host}/{prefix}/{manifest_short_id}/layer/{name}
+Content 	         {scheme}://{host}/{prefix}/{manifest_short_id}/res/{name}.{format}
+```
+
+

package/docs/cli.md

@@ -0,0 +1,132 @@
+# CLI DOCUMENTATION
+
+The CLI is the main interface to interact with aiiinotate: all aiiinotate commands are started from the CLI.
+
+---
+
+## Basics
+
+1. **Have your app set up**, either in dev or in prod. See [README](https://github.com/Aikon-platform/aiiinotate/blob/main/README.md) for more info
+
+2. **In prod, source your `.env`**. See README.
+
+3. **The base command is**:
+
+```bash
+# in prod
+aiiinotate
+
+# in dev
+npm run cli
+```
+
+**The command changes with a prod install**, because the package has been installed with `npm`. 
+- for the rest of this doc, we use `aiiinotate` instead of `npm run cli`. Use whichever is appropriate to your use case.
+- `npm run cli` runs a bash script that also manages environment sourcing.
+- when using `npm run cli`, you will sometimes need to add `--` to separate different arguments:
+    ```bash
+    npm run cli -- <command> <arguments>
+    ```
+
+---
+
+## `serve`: start the app
+
+```bash
+# serve the app in dev mode
+aiiinotate serve dev
+
+# serve the app in prod mode
+aiiinotate serve prod
+```
+
+---
+
+## `migrate`: run database migrations
+
+NOTE: **in dev**, run `npm run migrate`, which is a wrapper for `npm run cli migrate`.
+
+```bash
+# create a new migration
+aiiinotate migrate make --migration-name <your migration name>
+# the equivalent in dev is:
+npm run migrate make -- --migration-name <your migration name>
+
+# apply all pending migrations
+aiiinotate migrate apply
+
+# revert the last migration
+aiiinotate migrate revert
+
+# revert all migrations
+aiiinotate migrate revert-all
+```
+
+---
+
+## `import`: import data
+
+The CLI can be used to import IIIF annotations or to index IIIF manifests in aiiinotate.
+
+```bash
+aiiinotate import <manifests|annotations> -i 2 -f <path/to/import/file.txt>
+```
+
+Where:
+- `<manifests|annotations>` is the kind of data to import: manifests or annotations
+- `-i` `--iiif-version` (`2|3`) is the IIIF Presentation API version of the data to import:
+    - if `-i 3`, we import IIIF presentation 3.x data
+    - if `-i 2`, we import IIIF presentation 2.x data
+- `-f --file` is a relative or absolute path the the import file. This import file:
+    - contains a list of paths to annotation lists, annotation pages or manifests
+    - with 1 path per line
+    - paths can be relative or absolute
+
+NOTE that data to import **MUST MATCH the IIIF version**: if `-i 2`, you can only import `AnnotationLists` following the IIIF Presentation 2.x standard, and note annotations in 3.x format.
+
+NOTE that for the moment, import is only implemented for IIIF version 2.
+
+Here is an example of import file:
+
+```
+./annotations/wit91_pdf108_anno194.json
+./annotations/wit70_man70_anno206.json
+./annotations/wit71_man71_anno202.json
+./annotations/wit68_man68_anno198.json
+./annotations/wit72_man72_anno133.json
+./annotations/wit75_man74_anno130.json
+./annotations/wit2_man4_anno4.json
+```
+
+---
+
+## `export`: export data
+
+The CLI can be used to export data to JSON.
+
+```bash
+aiiinotate export <all|annotations2|manifests2|annotations3|manifests3> [-o /output/dir]?
+```
+
+Each collection will be written to its own JSON file.
+
+Exported JSON data can be imported back in a MongoDB instance using [mongoimport](https://www.mongodb.com/docs/database-tools/mongoimport/):
+
+```bash
+ mongoimport \
+   --host localhost \
+   --db <db name> \
+   --collection <collection name> \
+   --file <./path/to/export.json> \
+   --jsonArray
+```
+
+---
+
+## `xywhToInt`
+
+In versions of aiiinotate `>0.10.3`, the `xywh` bounding box extracted from incoming annotations and saved to database may contain `floats`. This causes errors when reconstructing Cantaloupe URLs from t`xywh`. This command converts all `xywh` containing floats to intgers. It is only implemented for collection `annotations2`.
+
+```bash
+aiiinotate run xywh-to-int
+```