aiiinotate 0.11.0 → 0.13.0

package/README.md

@@ -1,6 +1,11 @@
 # aiiinotate
 
-aiiinotate is a fast and lightweight annotation server for IIIF. It relies on `nodejs/fastify` and `mongodb` and provides an API to read/write/update/delete IIIF annotations and index manifests.
+aiiinotate is a **fast and lightweight annotation server for IIIF**.
+
+- **aiiinotate relies on** `nodejs/fastify` and `mongodb`
+- **provides a REST API** to read/write/update/delete IIIF annotations and index manifests.
+- **is distributed as an NPM package**, can be used through NPM or Docker
+- **is built for scalability and speed**: [in benchmarks](https://github.com/paulhectork/aiiinotate-benchmark) aiiinotate stores up to 10,000,000 annotations and its response times are always between $$\frac{1}{10}$$ and $$\frac{1}{100}$$ seconds
 
 NOTE: currently, only annotations following the IIIF presentation API 2.0 and 2.1 are supported.
 
@@ -8,7 +13,7 @@ NOTE: currently, only annotations following the IIIF presentation API 2.0 and 2.
 
 ## API
 
-See the [docs on the aiiinotate API](https://github.com/Aikon-platform/aiiinotate/blob/dev/docs/endpoints.md).
+See the [docs on the aiiinotate API](https://github.com/Aikon-platform/aiiinotate/blob/dev/docs/api.md).
 
 ---
 
@@ -16,11 +21,9 @@ See the [docs on the aiiinotate API](https://github.com/Aikon-platform/aiiinotat
 
 ### Install
 
-1. **Install mongodb**.
-    - see [dev installation script for help](./scripts/setup_mongodb.sh)
-    - checkout the [official installation guide](https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/#std-label-install-mdb-community-ubuntu)
+1. **install mongodb** (see [dev installation script for help](./scripts/setup_mongodb.sh) and checkout the [official installation guide](https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/#std-label-install-mdb-community-ubuntu))
 
-2. **Install aiiinotate**
+2. **install aiiinotate**
 ```bash
 npm install aiiinotate
 ```
@@ -33,37 +36,32 @@ Copy [`config/.env.template`](./config/.env.template) to `.env` and edit it.
 
 #### Runtime env sourcing
 
-Once the package is installed, it must access variables from the .env file. However, running `aiiinotate <commands>` creates a subscript which means **you can't `source` an `.env` file**.
+`aiiinotate` runs in a subproicess and won't inherit variables from a plain bash `source` call. Use either of these instead:
+
+1. **`dotenvx` (recommended)**:
 
 ```bash
-# THIS WILL FAIL: `aiiinotate` is executed in a subscript that doesn't inherit from the variables fetched in `source`.
-source /path/to/.env && aiiinotate <command>
+npx dotenvx run -f /path/to/.env -- aiiinotate <command>
 ```
 
-#### The solutions: do either:
-1. use `dotenvx` to inject variables:
-    ```bash
-    npx dotenvx run -f /path/to/.env -- aiiinotate <command>
-    ```
-2. manually export variables:
-    ```bash
-    set -a
-    source /path/to/.env
-    set +a
-    aiiinotate <command>
-    ```
+2. **manual export**:
+
+```bash
+set -a && source /path/to/.env && set +a
+aiiinotate <command>
+```
 
 For clarity, we omit env sourcing from the below commands.
 
 ### Setup the app
 
-1. **Start `mongod`**
+1. **start `mongod`**
 
 ```bash
 sudo systemctl start mongod
 ```
 
-2. **Create and configure the database**
+2. **create and configure the database**
 
 ```bash
 aiiinotate migrate apply
@@ -89,12 +87,20 @@ aiiinotate -- <command>
 
 It will give full access to the CLI interface of Aiiinotate. Run `aiiinotate --help` for more info.
 
+Use the CLI to:
+- import data
+- export data
+- apply and manage migrations
+
 For more information, see [the CLI docs](https://github.com/Aikon-platform/aiiinotate/blob/main/docs/cli.md).
 
-#### Import data
+---
+
+## DOCKER USAGE
 
-See [the CLI docs](https://github.com/Aikon-platform/aiiinotate/blob/main/docs/cli.md).
+See the docs [here](https://github.com/Aikon-platform/aiiinotate/blob/dev/docs/docker.md)
 
+For a Mirador integration, see [the reference implementation](github.com/paulhectork/mirador-aiiinotate/tree/main) (aiiinotate + MongoDB + Mirador 4 + MAE bundled in a single `docker-compose`)
 
 ---
 
@@ -123,15 +129,15 @@ npm i
 
 After installing, some setup must be done
 
-1. **Setup your `.env`** file after [`config/.env.template`](./config/.env.template) and place it at `./config/.env`.
+1. **setup your `.env`** file after [`config/.env.template`](./config/.env.template) and place it at `./config/.env`.
 
-2. **Start `mongod`**
+2. **start `mongod`**
 
 ```bash
 sudo systemctl start mongod
 ```
 
-3. **Configure the database**
+3. **configure the database**
 
 ```bash
 npm run migrate apply
@@ -141,40 +147,33 @@ npm run migrate apply
 
 Remember to have your `mongodb` service running: `sudo systemctl start mongod` !
 
-- **Start the app**
+#### Start the app
 
 ```bash
 # reload enabled
 npm run dev
 ```
 
-- **Test the app**. NOTE: the tests will probably fail if you set the env variable `AIIINOTATE_STRICT_MODE` to `true`.
+#### Test the app
+
+Note that the tests will probably fail if you set the env variable `AIIINOTATE_STRICT_MODE` to `true`.
 
 ```bash
 npm run test
 ```
 
-- **Run the CLI**. (see [the CLI docs](https://github.com/Aikon-platform/aiiinotate/blob/dev/docs/cli.md) for more info)
+#### Run the CLI
 
+See [the CLI docs](https://github.com/Aikon-platform/aiiinotate/blob/dev/docs/cli.md) for more info: 
 
 ```bash
 npm run cli
 ```
 
-- **Process migrations**. (see [the CLI docs](https://github.com/Aikon-platform/aiiinotate/blob/dev/docs/cli.md) for more info)
-
-```bash
-# NOTE: the `--` is necessary !
-npm run migrate -- <command> <arguments?>
-```
-
-
-- **Import data**. (see [the CLI docs](https://github.com/Aikon-platform/aiiinotate/blob/main/docs/cli.md) for more info)
-
-```bash
-# NOTE: the `--` is necessary !
-npm run cli -- import <arguments>
-```
+Use the CLI to:
+- import data
+- export data
+- apply and manage database migrations
 
 --- 
 
@@ -192,6 +191,16 @@ aiiinotate is well tested: **~90% test coverage** on all files !
 
 ---
 
+## Scalability
+
+In [benchmarks](https://github.com/paulhectork/aiiinotate-benchmark), aiiinotate response times are between 1/100th and 1/10th of a second with up to 10,000,000 annotations. 
+
+![benchmark results](./docs/includes/report_benchmark_aiiinotate_2026-05-28-02:50:48_7steps.png)
+
+See [scalability.md](./docs/scalability.md) for more information.
+
+---
+
 ## License
 
 GNU GPL 3.0.

package/cli/export.js

@@ -0,0 +1,118 @@
+import path from "node:path";
+
+import { Command, Option, Argument } from "commander";
+
+import loadMongoClient from "#cli/utils/mongoClient.js";
+import { dirOk, getCwd, writeCursorToJson } from "#cli/utils/io.js";
+
+/** @typedef {import("#types").MongoDbType} MongoDbType */
+/** @typedef {import("#types").MongoFindCursorType} MongoFindCursorType */
+/** @typedef {import("fs").PathLike} PathLike */
+
+
+const exportableCollections = [ "all", "annotations2", "annotations3", "manifests2", "manifests3" ];
+const actualCollections = exportableCollections.filter(c => c!=="all");
+
+/** @type {(outDir: string) => string} */
+const getOutDir = (outDir) => {
+  let success;
+  if (outDir) {
+    [ outDir, success ] = dirOk(outDir);
+    if (!success) {
+      console.error(`Error opening directory "${outDir}". Are you sure it exists ?`);
+      process.exit(1);
+    }
+  } else {
+    outDir = getCwd();
+  }
+  return outDir;
+}
+
+/** @type {() => (collectionName: string) => string|PathLike} */
+const outFileNameFunc = () => {
+  // we currify this function so that, if a user exports all collections,
+  // output file nmaes share the same timestamp.
+  const timestamp = (new Date).toISOString().slice(0,-2);
+  return (outDir, collectionName) =>
+    path.join(outDir, `${timestamp}-aiiinotate-${collectionName}.json`);
+}
+const outFileName = outFileNameFunc();
+
+/**
+ * if `count`, returns a function to count all documents in a collection.
+ * else, returns a function to find all documents in a collection.
+ * @type {(db:MongoClient) => (count: boolean) => (collectionName: string) => MongoFindCursorType|Promise<number>}
+ */
+const exportCmdFunc = (db) =>
+  (count) =>
+    (collectionName) => {
+      const collection = db.collection(collectionName);
+      return count
+        ? collection.countDocuments()
+        : collection.find().project({ _id: 0 })
+
+    }
+
+/**
+ * export a single collection to a file
+ * @param {MongoClient} db
+ * @param {string} outDir
+ * @returns {(collctionName: string) => Promise<void>}
+ */
+const exportCollectionFunc = (db, outDir) => {
+  const exportCmd = exportCmdFunc(db)(false);
+  const countFunc = exportCmdFunc(db)(true);
+
+  return async (collectionName) => {
+    const out = outFileName(outDir, collectionName);
+    const collectionCursor = exportCmd(collectionName);
+    const totalCount = await countFunc(collectionName);
+    try {
+      await writeCursorToJson(out, collectionCursor, totalCount);
+      console.log(`Wrote export of collection "${collectionName}" (${totalCount} documents) to "${out}"\n`)
+    } catch(e) {
+      console.error(`Error writing export of collection "${collectionName}" to "${out}": ${e}`);
+      process.exit(1);
+    }
+  }
+}
+
+/**
+ * @param {string} collection
+ * @param {object} options
+ */
+async function action(collection, options) {
+  const { client, db } = loadMongoClient();
+  const outDir = getOutDir(options.output);
+  const collectionNameArray =collection === "all" ? actualCollections : [ collection ];
+  const exportCollection = exportCollectionFunc(db, outDir);
+
+  console.log(`\nExporting collection(s): ${collectionNameArray}\n`)
+
+  // fetch and write to file
+  for (const collectionName of collectionNameArray) {
+    await exportCollection(collectionName);
+  }
+
+  console.log(`\nFinished exporting collection(s): ${collectionNameArray} to "${outDir}".`)
+  await client.close();
+  process.exit(0);
+}
+
+/** define the cli */
+function makeExportCommand() {
+
+  const collectionArg =
+    new Argument("collection", `collection to export import: one of ${exportableCollections}`)
+      .choices(exportableCollections);
+
+  const outOpt = new Option("-o, --output <directory>", "Output directory (defaults to current working directory");
+
+  return new Command("export")
+    .description("export aiiinotate data")
+    .addArgument(collectionArg)
+    .addOption(outOpt)
+    .action(async (collection, options, command) => await action(collection, options))
+}
+
+export default makeExportCommand;

package/cli/index.js

@@ -14,6 +14,7 @@ import { Command, Option } from "commander";
 // import dotenvx from "dotenvx";
 
 import makeImportCommand from "#cli/import.js";
+import makeExportCommand from "#cli/export.js";
 import makeMigrateCommand from "#cli/migrate.js";
 import makeServeCommand from "#cli/serve.js";
 import makeXywhToIntCommand from "#cli/xywhToInt.js";
@@ -34,6 +35,7 @@ function makeCli() {
     .description(desc)
     .addCommand(makeServeCommand())
     .addCommand(makeImportCommand())
+    .addCommand(makeExportCommand())
     .addCommand(makeMigrateCommand())
     .addCommand(makeXywhToIntCommand());
 

package/cli/utils/io.js

@@ -1,8 +1,12 @@
 import path from "path";
 import fs from "fs";
 
+import ProgressBar from "#cli/utils/progressbar.js";
+
+/** @typedef {import("fs").WriteStream} WriteStreamType */
 
 const cwd = process.cwd();  // directory the script is run from
+const getCwd = () => process.cwd();
 
 /**
  * convert a filepath to absolute if it is relative.
@@ -24,6 +28,17 @@ const fileOk = (f) => {
   }
 }
 
+/** @returns {[PathLike, boolean]} */
+const dirOk = (d) => {
+  d = toAbsPath(d);
+  try {
+    fs.accessSync(d, fs.constants.R_OK);  // will throw if there's an error
+    return [ d, fs.lstatSync(d).isDirectory() ]
+  } catch (e) {
+    return [ d, false ]
+  }
+}
+
 /**
  * @param {string} f
  * @return {string?}
@@ -79,9 +94,132 @@ async function parseImportInputFile(file) {
   return [ ...new Set(fileArrayValidate(fileArr)) ];
 }
 
+const fileWrite = (f, data) => {
+  try {
+    fs.writeFileSync(f, data, "utf-8");
+  } catch (e) {
+    throw new Error(`Error writing to file: ${f} because of error: ${e}`);
+  }
+}
+
+/**
+ * NOTE: this will NOT work on collections and huge objects. use streamWriteJson to write from Mongo Cursors.
+ * @param {string|import("fs").PathLike} f
+ * @param {Array|object} data
+ * @returns {void}
+ */
+const fileWriteJson = (f, data) => {
+  data = JSON.stringify(data)
+  fileWrite(f, data);
+}
+
+/**
+ * wrapper for writer.write(data) that respects writer backpressure
+ *
+ * we wrap the .write() in a Promise that checks on the rturn of `writer.write`
+ * and drains the stream if necessary: if `writer.write()` returns
+ * false, the write buffer is full  and must be drained by the OS
+ * before continuing.
+ *
+ * this avoids:
+ * - writes that are silently dropped or reordered
+ * - corrupt/truncated JSON with no error thrown
+ *
+ * @type {(writer: WriteStreamType) => (data: string) => Promise<void> }
+ */
+const writeOrWait = (writer) =>
+  (data) =>
+    new Promise((resolve, reject) => {
+      const ok = writer.write(data, (error) => {
+        if (error) return reject(error);
+      });
+      if (ok) {
+        resolve();  // buffer has room, keep going
+      } else {
+        writer.once("drain", resolve);  // buffer full, wait for drain
+      }
+    });
+
+/**
+ * WriteStream.write is actually asyncrhonous => for the last line,
+ * we need to use `writer.end` to ensure everything has been written to file.
+ *
+ * @returns {Promise<void>}
+ */
+const endStream = (writer, data) =>
+  new Promise((resolve, reject) => {
+    writer.end(data, (error) => {
+      if (error) return reject(error);
+      resolve();
+    });
+  });
+
+/**
+ * given a file path `fp` and a FindCursor `cursor`,
+ * write all documents in `cursor` to a file.
+ *
+ * if `totalCount` is defined, print a progress bar as well.
+ * else, just print the document number and update at each iteration.
+ *
+ * necessary to use a string when `cursor` stores a lot of documents
+ * (instead of a simple file-write):
+ * - cursor.toArray() uses TONS of memory and is slow
+ * - JSON.stringify(), used to write JSONs to file, has a maximum
+ *    size for arrays and will crash if stringifying huge arrays.
+ *
+ * to check that the output is a valid JSON, use:
+ * ```bash
+ * python3 -mjson.tool path/to/json > /dev/null && echo "ok" || echo "err"
+ * ````
+ *
+ * @param {string|PathLike} fp
+ * @param {MongoFindCursorType} cursor
+ * @param {number?} totalCount
+ * @returns {Promise}
+ */
+const writeCursorToJson = async (fp, cursor, totalCount) => {
+  const writer = fs.createWriteStream(fp);
+  const onceWriter = writeOrWait(writer);
+
+  let pb;
+  if (totalCount) {
+    pb = new ProgressBar({ desc: "writing documents to file", total: totalCount });
+  } else {
+    console.log("");
+  }
+
+  // we are writing an array => manually write the opening "[".
+  await onceWriter("[");
+  let i = 0
+  for await (const doc of cursor) {
+    i += 1;
+    // if previous items were written to file, write a "," separator.
+    if (i!=1) await onceWriter(", ");
+    if (pb) {
+      pb.update(i);
+    } else {
+      process.stdout.clearLine(0);
+      process.stdout.cursorTo(0);
+      process.stdout.write(`writing document #${i} to file`);
+    }
+    await onceWriter(JSON.stringify(doc));
+  }
+  // close the array with a "]" and end the stream.
+  await endStream(writer, "]");
+
+  if (!pb && i>0) console.log("");
+  return totalCount;
+}
+
+
 export {
   fileRead,
   fileOk,
+  dirOk,
+  fileWrite,
+  fileWriteJson,
+  getCwd,
   fileArrayValidate,
-  parseImportInputFile
+  parseImportInputFile,
+  writeCursorToJson
 }

package/docker/Dockerfile

@@ -5,23 +5,35 @@ FROM node:23.11
 ARG PORT
 ENV PORT=${PORT}
 
+# path to the .env file
+ARG ENV_PATH
+ENV ENV_PATH="$ENV_PATH"
+
 # set up environment
+ENV DIR=/aiiinotate
 ENV TERM=linux
+ENV NPM_BIN=/aiiinotate/node_modules/.bin
 SHELL ["/bin/bash", "-c"]
 
 # root of the app in the docker container
-WORKDIR /aiiinotate
+WORKDIR ${DIR}
 # copy the docker .env in the docker container
-COPY ./docker/.env /aiiinotate/.env
-# for debug: install `iproute2` which provides CLI `ss` to monitor active ports
-RUN apt update && apt install iproute2 -y
+COPY "$ENV_PATH" ${DIR}/.env
+
+# install iproute and curl for debug
+RUN apt-get update
+RUN apt install curl iproute2 -y
+
+# create a npm package in $DIR to run aiiinotate from
+RUN npm init -y
 # install the app as an NPM library. we do a global install as it saves us from issues with `$PATH`.
-RUN npm i -g aiiinotate
+RUN npm i aiiinotate
 
 # expose the used port
-EXPOSE $PORT
+EXPOSE ${PORT}
 
 # run the migrations and start the app.
+# ./node_modules/.bin/aiiinotate is to be able to use the `aiiinotate` cli without doing a global install of `aiiinotate`
 # NOTE migrations must be done in CMD: they need the mongo service to be running.
-CMD aiiinotate --env=/aiiinotate/.env -- migrate apply && \
-    aiiinotate --env=/aiiinotate/.env -- serve prod;
+CMD ${NPM_BIN}/dotenvx run -f ${DIR}/.env -- ${NPM_BIN}/aiiinotate migrate apply && \
+    ${NPM_BIN}/dotenvx run -f ${DIR}/.env -- ${NPM_BIN}/aiiinotate serve prod;

package/docker/README.md

@@ -1,62 +1,5 @@
-# Docker usage
+# Docker setup
 
----
+This folder contains a `docker-compose` for an aiiinotate+MongoDB integration, with volume persistence and a script to import data.
 
-## Usage
-
-Note that all `docker` commands must be run from the `docker/` directory !
-
-1. Clone the repo and `cd` in it
-
-```bash
-git clone git@github.com:Aikon-platform/aiiinotate.git
-cd aiiinotate
-```
-
-2. Create a `.env` file at `config/.env` (from the root of the project)
-
-```bash
-cp ./config/.env.template ./config/.env
-vim ./config/.env  # do your edits
-```
-
-3. Build the containers
-
-```bash
-cd docker
-bash docker.sh build
-```
-
-4. Start the containers
-
-```bash
-bash docker.sh start
-```
-
----
-
-## Troubleshooting and useful commands
-
-Access `mongosh` within the Mongo container:
-
-```bash
-sudo docker exec -it docker-mongo-1 mongosh
-```
-
-Check running ports in the Web container:
-
-```bash
-sudo docker exec -it docker-web-1 ss -tnl
-```
-
-View globally installed NPM packages in the Web container:
-
-```bash
-sudo docker exec -it docker-web-1 npm list -g --depth=0
-```
-
-CURL the Web container
-
-```bash
-sudo docker exec -it docker-web-1 curl http://0.0.0.0:4444   # change 4444 by your $AIIINOTATE_PORT
-```
+View the documentation for aiiinotate Docker setup [here](https://github.com/Aikon-platform/aiiinotate/blob/main/docs/docker.md)

package/docker/docker-compose.yaml

@@ -9,30 +9,41 @@ services:
     restart: always
     networks:
       - aiiinotate_network
+    volumes:
+      - mongodata:/data/db
     environment:
       <<: *proxy-settings
-    #   MONGO_INITDB_ROOT_USERNAME: root
-    #   MONGO_INITDB_ROOT_PASSWORD: example
+    healthcheck:
+      test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
 
-  web:
+  aiiinotate:
     build:
       context: ..
       dockerfile: docker/Dockerfile
       args:
         PORT: ${AIIINOTATE_PORT}
+        ENV_PATH: ../docker/.env
     # NOTE: be careful to point to docker/.env, not to config/.env, otherwise you will have very, very annoying errors.
     env_file:
       - ../docker/.env
     ports:
       - "${AIIINOTATE_PORT}:${AIIINOTATE_PORT}"
     depends_on:
-      - mongo
+      mongo:
+        condition: service_healthy
     networks:
       - aiiinotate_network
     environment:
       <<: *proxy-settings
     restart: always
 
+
+volumes:
+  mongodata:
+
 networks:
   aiiinotate_network:
     driver: bridge