aiiinotate 0.10.3 → 0.12.0

package/README.md

@@ -8,7 +8,7 @@ NOTE: currently, only annotations following the IIIF presentation API 2.0 and 2.
 
 ## API
 
-See the [docs on the aiiinotate API](https://github.com/Aikon-platform/aiiinotate/blob/dev/docs/endpoints.md).
+See the [docs on the aiiinotate API](https://github.com/Aikon-platform/aiiinotate/blob/dev/docs/api.md).
 
 ---
 
@@ -16,11 +16,9 @@ See the [docs on the aiiinotate API](https://github.com/Aikon-platform/aiiinotat
 
 ### Install
 
-1. **Install mongodb**.
-    - see [dev installation script for help](./scripts/setup_mongodb.sh)
-    - checkout the [official installation guide](https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/#std-label-install-mdb-community-ubuntu)
+1. **install mongodb** (see [dev installation script for help](./scripts/setup_mongodb.sh) and checkout the [official installation guide](https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/#std-label-install-mdb-community-ubuntu))
 
-2. **Install aiiinotate**
+2. **install aiiinotate**
 ```bash
 npm install aiiinotate
 ```
@@ -33,37 +31,32 @@ Copy [`config/.env.template`](./config/.env.template) to `.env` and edit it.
 
 #### Runtime env sourcing
 
-Once the package is installed, it must access variables from the .env file. However, running `aiiinotate <commands>` creates a subscript which means **you can't `source` an `.env` file**.
+`aiiinotate` runs in a subproicess and won't inherit variables from a plain bash `source` call. Use either of these instead:
+
+1. **`dotenvx` (recommended)**:
 
 ```bash
-# THIS WILL FAIL: `aiiinotate` is executed in a subscript that doesn't inherit from the variables fetched in `source`.
-source /path/to/.env && aiiinotate <command>
+npx dotenvx run -f /path/to/.env -- aiiinotate <command>
 ```
 
-#### The solutions: do either:
-1. use `dotenvx` to inject variables:
-    ```bash
-    npx dotenvx run -f /path/to/.env -- aiiinotate <command>
-    ```
-2. manually export variables:
-    ```bash
-    set -a
-    source /path/to/.env
-    set +a
-    aiiinotate <command>
-    ```
+2. **manual export**:
+
+```bash
+set -a && source /path/to/.env && set +a
+aiiinotate <command>
+```
 
 For clarity, we omit env sourcing from the below commands.
 
 ### Setup the app
 
-1. **Start `mongod`**
+1. **start `mongod`**
 
 ```bash
 sudo systemctl start mongod
 ```
 
-2. **Create and configure the database**
+2. **create and configure the database**
 
 ```bash
 aiiinotate migrate apply
@@ -89,12 +82,20 @@ aiiinotate -- <command>
 
 It will give full access to the CLI interface of Aiiinotate. Run `aiiinotate --help` for more info.
 
+Use the CLI to:
+- import data
+- export data
+- apply and manage migrations
+
 For more information, see [the CLI docs](https://github.com/Aikon-platform/aiiinotate/blob/main/docs/cli.md).
 
-#### Import data
+---
+
+## DOCKER USAGE
 
-See [the CLI docs](https://github.com/Aikon-platform/aiiinotate/blob/main/docs/cli.md).
+See the docs [here](https://github.com/Aikon-platform/aiiinotate/blob/dev/docs/docker.md)
 
+For a Mirador integration, see [the reference implementation](github.com/paulhectork/mirador-aiiinotate/tree/main) (aiiinotate + MongoDB + Mirador 4 + MAE bundled in a single `docker-compose`)
 
 ---
 
@@ -123,15 +124,15 @@ npm i
 
 After installing, some setup must be done
 
-1. **Setup your `.env`** file after [`config/.env.template`](./config/.env.template) and place it at `./config/.env`.
+1. **setup your `.env`** file after [`config/.env.template`](./config/.env.template) and place it at `./config/.env`.
 
-2. **Start `mongod`**
+2. **start `mongod`**
 
 ```bash
 sudo systemctl start mongod
 ```
 
-3. **Configure the database**
+3. **configure the database**
 
 ```bash
 npm run migrate apply
@@ -141,40 +142,33 @@ npm run migrate apply
 
 Remember to have your `mongodb` service running: `sudo systemctl start mongod` !
 
-- **Start the app**
+#### Start the app
 
 ```bash
 # reload enabled
 npm run dev
 ```
 
-- **Test the app**. NOTE: the tests will probably fail if you set the env variable `AIIINOTATE_STRICT_MODE` to `true`.
+#### Test the app
+
+Note that the tests will probably fail if you set the env variable `AIIINOTATE_STRICT_MODE` to `true`.
 
 ```bash
 npm run test
 ```
 
-- **Run the CLI**. (see [the CLI docs](https://github.com/Aikon-platform/aiiinotate/blob/dev/docs/cli.md) for more info)
+#### Run the CLI
 
+See [the CLI docs](https://github.com/Aikon-platform/aiiinotate/blob/dev/docs/cli.md) for more info: 
 
 ```bash
 npm run cli
 ```
 
-- **Process migrations**. (see [the CLI docs](https://github.com/Aikon-platform/aiiinotate/blob/dev/docs/cli.md) for more info)
-
-```bash
-# NOTE: the `--` is necessary !
-npm run migrate -- <command> <arguments?>
-```
-
-
-- **Import data**. (see [the CLI docs](https://github.com/Aikon-platform/aiiinotate/blob/main/docs/cli.md) for more info)
-
-```bash
-# NOTE: the `--` is necessary !
-npm run cli -- import <arguments>
-```
+Use the CLI to:
+- import data
+- export data
+- apply and manage database migrations
 
 --- 
 

package/cli/export.js

@@ -0,0 +1,118 @@
+import path from "node:path";
+
+import { Command, Option, Argument } from "commander";
+
+import loadMongoClient from "#cli/utils/mongoClient.js";
+import { dirOk, getCwd, writeCursorToJson } from "#cli/utils/io.js";
+
+/** @typedef {import("#types").MongoDbType} MongoDbType */
+/** @typedef {import("#types").MongoFindCursorType} MongoFindCursorType */
+/** @typedef {import("fs").PathLike} PathLike */
+
+
+const exportableCollections = [ "all", "annotations2", "annotations3", "manifests2", "manifests3" ];
+const actualCollections = exportableCollections.filter(c => c!=="all");
+
+/** @type {(outDir: string) => string} */
+const getOutDir = (outDir) => {
+  let success;
+  if (outDir) {
+    [ outDir, success ] = dirOk(outDir);
+    if (!success) {
+      console.error(`Error opening directory "${outDir}". Are you sure it exists ?`);
+      process.exit(1);
+    }
+  } else {
+    outDir = getCwd();
+  }
+  return outDir;
+}
+
+/** @type {() => (collectionName: string) => string|PathLike} */
+const outFileNameFunc = () => {
+  // we currify this function so that, if a user exports all collections,
+  // output file nmaes share the same timestamp.
+  const timestamp = (new Date).toISOString().slice(0,-2);
+  return (outDir, collectionName) =>
+    path.join(outDir, `${timestamp}-aiiinotate-${collectionName}.json`);
+}
+const outFileName = outFileNameFunc();
+
+/**
+ * if `count`, returns a function to count all documents in a collection.
+ * else, returns a function to find all documents in a collection.
+ * @type {(db:MongoClient) => (count: boolean) => (collectionName: string) => MongoFindCursorType|Promise<number>}
+ */
+const exportCmdFunc = (db) =>
+  (count) =>
+    (collectionName) => {
+      const collection = db.collection(collectionName);
+      return count
+        ? collection.countDocuments()
+        : collection.find().project({ _id: 0 })
+
+    }
+
+/**
+ * export a single collection to a file
+ * @param {MongoClient} db
+ * @param {string} outDir
+ * @returns {(collctionName: string) => Promise<void>}
+ */
+const exportCollectionFunc = (db, outDir) => {
+  const exportCmd = exportCmdFunc(db)(false);
+  const countFunc = exportCmdFunc(db)(true);
+
+  return async (collectionName) => {
+    const out = outFileName(outDir, collectionName);
+    const collectionCursor = exportCmd(collectionName);
+    const totalCount = await countFunc(collectionName);
+    try {
+      await writeCursorToJson(out, collectionCursor, totalCount);
+      console.log(`Wrote export of collection "${collectionName}" (${totalCount} documents) to "${out}"\n`)
+    } catch(e) {
+      console.error(`Error writing export of collection "${collectionName}" to "${out}": ${e}`);
+      process.exit(1);
+    }
+  }
+}
+
+/**
+ * @param {string} collection
+ * @param {object} options
+ */
+async function action(collection, options) {
+  const { client, db } = loadMongoClient();
+  const outDir = getOutDir(options.output);
+  const collectionNameArray =collection === "all" ? actualCollections : [ collection ];
+  const exportCollection = exportCollectionFunc(db, outDir);
+
+  console.log(`\nExporting collection(s): ${collectionNameArray}\n`)
+
+  // fetch and write to file
+  for (const collectionName of collectionNameArray) {
+    await exportCollection(collectionName);
+  }
+
+  console.log(`\nFinished exporting collection(s): ${collectionNameArray} to "${outDir}".`)
+  await client.close();
+  process.exit(0);
+}
+
+/** define the cli */
+function makeExportCommand() {
+
+  const collectionArg =
+    new Argument("collection", `collection to export import: one of ${exportableCollections}`)
+      .choices(exportableCollections);
+
+  const outOpt = new Option("-o, --output <directory>", "Output directory (defaults to current working directory");
+
+  return new Command("export")
+    .description("export aiiinotate data")
+    .addArgument(collectionArg)
+    .addOption(outOpt)
+    .action(async (collection, options, command) => await action(collection, options))
+}
+
+export default makeExportCommand;

package/cli/index.js

@@ -14,8 +14,10 @@ import { Command, Option } from "commander";
 // import dotenvx from "dotenvx";
 
 import makeImportCommand from "#cli/import.js";
+import makeExportCommand from "#cli/export.js";
 import makeMigrateCommand from "#cli/migrate.js";
 import makeServeCommand from "#cli/serve.js";
+import makeXywhToIntCommand from "#cli/xywhToInt.js";
 
 function makeCli() {
 
@@ -33,7 +35,9 @@ function makeCli() {
     .description(desc)
     .addCommand(makeServeCommand())
     .addCommand(makeImportCommand())
-    .addCommand(makeMigrateCommand());
+    .addCommand(makeExportCommand())
+    .addCommand(makeMigrateCommand())
+    .addCommand(makeXywhToIntCommand());
 
   cli.parse(process.argv);
 }

package/cli/utils/io.js

@@ -1,8 +1,12 @@
 import path from "path";
 import fs from "fs";
 
+import ProgressBar from "#cli/utils/progressbar.js";
+
+/** @typedef {import("fs").WriteStream} WriteStreamType */
 
 const cwd = process.cwd();  // directory the script is run from
+const getCwd = () => process.cwd();
 
 /**
  * convert a filepath to absolute if it is relative.
@@ -24,6 +28,17 @@ const fileOk = (f) => {
   }
 }
 
+/** @returns {[PathLike, boolean]} */
+const dirOk = (d) => {
+  d = toAbsPath(d);
+  try {
+    fs.accessSync(d, fs.constants.R_OK);  // will throw if there's an error
+    return [ d, fs.lstatSync(d).isDirectory() ]
+  } catch (e) {
+    return [ d, false ]
+  }
+}
+
 /**
  * @param {string} f
  * @return {string?}
@@ -79,9 +94,132 @@ async function parseImportInputFile(file) {
   return [ ...new Set(fileArrayValidate(fileArr)) ];
 }
 
+const fileWrite = (f, data) => {
+  try {
+    fs.writeFileSync(f, data, "utf-8");
+  } catch (e) {
+    throw new Error(`Error writing to file: ${f} because of error: ${e}`);
+  }
+}
+
+/**
+ * NOTE: this will NOT work on collections and huge objects. use streamWriteJson to write from Mongo Cursors.
+ * @param {string|import("fs").PathLike} f
+ * @param {Array|object} data
+ * @returns {void}
+ */
+const fileWriteJson = (f, data) => {
+  data = JSON.stringify(data)
+  fileWrite(f, data);
+}
+
+/**
+ * wrapper for writer.write(data) that respects writer backpressure
+ *
+ * we wrap the .write() in a Promise that checks on the rturn of `writer.write`
+ * and drains the stream if necessary: if `writer.write()` returns
+ * false, the write buffer is full  and must be drained by the OS
+ * before continuing.
+ *
+ * this avoids:
+ * - writes that are silently dropped or reordered
+ * - corrupt/truncated JSON with no error thrown
+ *
+ * @type {(writer: WriteStreamType) => (data: string) => Promise<void> }
+ */
+const writeOrWait = (writer) =>
+  (data) =>
+    new Promise((resolve, reject) => {
+      const ok = writer.write(data, (error) => {
+        if (error) return reject(error);
+      });
+      if (ok) {
+        resolve();  // buffer has room, keep going
+      } else {
+        writer.once("drain", resolve);  // buffer full, wait for drain
+      }
+    });
+
+/**
+ * WriteStream.write is actually asyncrhonous => for the last line,
+ * we need to use `writer.end` to ensure everything has been written to file.
+ *
+ * @returns {Promise<void>}
+ */
+const endStream = (writer, data) =>
+  new Promise((resolve, reject) => {
+    writer.end(data, (error) => {
+      if (error) return reject(error);
+      resolve();
+    });
+  });
+
+/**
+ * given a file path `fp` and a FindCursor `cursor`,
+ * write all documents in `cursor` to a file.
+ *
+ * if `totalCount` is defined, print a progress bar as well.
+ * else, just print the document number and update at each iteration.
+ *
+ * necessary to use a string when `cursor` stores a lot of documents
+ * (instead of a simple file-write):
+ * - cursor.toArray() uses TONS of memory and is slow
+ * - JSON.stringify(), used to write JSONs to file, has a maximum
+ *    size for arrays and will crash if stringifying huge arrays.
+ *
+ * to check that the output is a valid JSON, use:
+ * ```bash
+ * python3 -mjson.tool path/to/json > /dev/null && echo "ok" || echo "err"
+ * ````
+ *
+ * @param {string|PathLike} fp
+ * @param {MongoFindCursorType} cursor
+ * @param {number?} totalCount
+ * @returns {Promise}
+ */
+const writeCursorToJson = async (fp, cursor, totalCount) => {
+  const writer = fs.createWriteStream(fp);
+  const onceWriter = writeOrWait(writer);
+
+  let pb;
+  if (totalCount) {
+    pb = new ProgressBar({ desc: "writing documents to file", total: totalCount });
+  } else {
+    console.log("");
+  }
+
+  // we are writing an array => manually write the opening "[".
+  await onceWriter("[");
+  let i = 0
+  for await (const doc of cursor) {
+    i += 1;
+    // if previous items were written to file, write a "," separator.
+    if (i!=1) await onceWriter(", ");
+    if (pb) {
+      pb.update(i);
+    } else {
+      process.stdout.clearLine(0);
+      process.stdout.cursorTo(0);
+      process.stdout.write(`writing document #${i} to file`);
+    }
+    await onceWriter(JSON.stringify(doc));
+  }
+  // close the array with a "]" and end the stream.
+  await endStream(writer, "]");
+
+  if (!pb && i>0) console.log("");
+  return totalCount;
+}
+
+
 export {
   fileRead,
   fileOk,
+  dirOk,
+  fileWrite,
+  fileWriteJson,
+  getCwd,
   fileArrayValidate,
-  parseImportInputFile
+  parseImportInputFile,
+  writeCursorToJson
 }

package/cli/utils/mongoClient.js

@@ -4,13 +4,14 @@ import { MONGODB_CONNSTRING, MONGODB_DB } from "#constants";
 
 /**
  * load a mongo client and connect it to the database. exists if there's an error
- * @returns {import("mongodb").MongoClient}
+ * @returns {{ client: import("mongodb").MongoClient, db: import("mongodb").Db }}
  */
 function loadMongoClient() {
   try {
-    const client = new MongoClient(MONGODB_CONNSTRING);
-    client.db(MONGODB_DB);
-    return client;
+    const
+      client = new MongoClient(MONGODB_CONNSTRING),
+      db = client.db(MONGODB_DB);
+    return { client, db };
   } catch (err) {
     console.error(`mongoClient: could not connect to DB because of error ${err.message}`);
     process.exit(1);

package/cli/xywhToInt.js

@@ -0,0 +1,99 @@
+import { Command, Option, Argument } from "commander";
+
+import loadMongoClient from "#cli/utils/mongoClient.js";
+import ProgressBar from "#cli/utils/progressbar.js";
+import { maybeToArray, xywhToInt } from "#src/utils/utils.js";
+
+/** @typedef {import("mongodb").Db} Db */
+
+/**
+ * bash command to get faulty annotations from a docker aiiinotate instance:
+ * ```
+ * docker exec docker-mongo-1 mongosh vhs_aiiinotate --eval \
+ *      "JSON.stringify(db.annotations2.find({ 'on.xywh': { \$type: 'double' } }, { '_id': 0 }).toArray(), null, 2);" \
+ *      > mogno_export.json
+ * ```
+ *
+ * bash command to import annotations exported above in a local mongo instance:
+ * ```
+ * mongoimport \
+ *   --host localhost \
+ *   --db aiiinotate \
+ *   --collection annotations2 \
+ *   --file ./mogno_export.json \
+ *   --jsonArray
+ * ```
+ */
+
+/**
+ *
+ * @param {Db} db - the mongo database
+ * @param {object} annotation - the annotation to update
+ * @param {boolean} dryRun - if `false`, don't update the annotation in database
+ */
+async function updateAnnotation(db, annotation, dryRun) {
+  let [ annotationTargetArray, converted ] = maybeToArray(annotation.on, true);
+
+  annotationTargetArray = annotationTargetArray.map((target) => {
+    target.xywh = xywhToInt(target.xywh);
+    console.log(target.xywh);
+    return target;
+  });
+  annotation.on = converted
+    ? annotationTargetArray[0]
+    : annotationTargetArray;
+
+  // console.log("post: ", annotation.on);
+  // we need to do 1 update / document, since the @id filter can select 1 document at a time.
+  if (!dryRun) {
+    await db.collection("annotations2").updateOne(
+      { "@id": annotation["@id"] },
+      { $set: annotation },
+      { upsert: false }
+    );
+  }
+  return
+}
+
+// NOTE: annotations have been exported from vhs and imported in local aiiinotate instance.
+async function action(options) {
+  const dryRun = options.dryRun || false;
+
+  const summary = { total: 0, ok: 0, error: 0 };
+
+  const { client, db } = loadMongoClient();
+  const filter = { "on.xywh": { $type: "double" } };
+  const toUpdateCount = await db.collection("annotations2").countDocuments(filter);
+  const annotationsCursor = db.collection("annotations2").find(filter);
+  summary.total = toUpdateCount;
+
+  const pb = new ProgressBar({ desc: "updating annotations", total: toUpdateCount });
+  let i = 0;
+  while (await annotationsCursor.hasNext()) {
+    i += 1;
+    pb.update(i);
+    const annotation = await annotationsCursor.next();
+    try {
+      await updateAnnotation(db, annotation, dryRun);
+      summary.ok += 1;
+    } catch (e) {
+      console.error(e);
+      summary.error += 1
+    }
+  }
+
+  console.log(`converting annotation.on.xywh to int (dry run=${dryRun}). results:`, summary);
+  await client.close()
+  process.exit(0);
+}
+
+function makeXywhToIntCommand() {
+  const dryRunOpt = new Option("-d, --dry-run", "dry run (don't update data)");
+
+  return new Command("xywh-to-int")
+    .description("convert all `db.annotations2.on.xywh` values from float to int")
+    .addOption(dryRunOpt)
+    .action(async (options, command) => { return await action(options) });
+}
+
+export default makeXywhToIntCommand;

package/docker/Dockerfile

@@ -5,23 +5,35 @@ FROM node:23.11
 ARG PORT
 ENV PORT=${PORT}
 
+# path to the .env file
+ARG ENV_PATH
+ENV ENV_PATH="$ENV_PATH"
+
 # set up environment
+ENV DIR=/aiiinotate
 ENV TERM=linux
+ENV NPM_BIN=/aiiinotate/node_modules/.bin
 SHELL ["/bin/bash", "-c"]
 
 # root of the app in the docker container
-WORKDIR /aiiinotate
+WORKDIR ${DIR}
 # copy the docker .env in the docker container
-COPY ./docker/.env /aiiinotate/.env
-# for debug: install `iproute2` which provides CLI `ss` to monitor active ports
-RUN apt update && apt install iproute2 -y
+COPY "$ENV_PATH" ${DIR}/.env
+
+# install iproute and curl for debug
+RUN apt-get update
+RUN apt install curl iproute2 -y
+
+# create a npm package in $DIR to run aiiinotate from
+RUN npm init -y
 # install the app as an NPM library. we do a global install as it saves us from issues with `$PATH`.
-RUN npm i -g aiiinotate
+RUN npm i aiiinotate
 
 # expose the used port
-EXPOSE $PORT
+EXPOSE ${PORT}
 
 # run the migrations and start the app.
+# ./node_modules/.bin/aiiinotate is to be able to use the `aiiinotate` cli without doing a global install of `aiiinotate`
 # NOTE migrations must be done in CMD: they need the mongo service to be running.
-CMD aiiinotate --env=/aiiinotate/.env -- migrate apply && \
-    aiiinotate --env=/aiiinotate/.env -- serve prod;
+CMD ${NPM_BIN}/dotenvx run -f ${DIR}/.env -- ${NPM_BIN}/aiiinotate migrate apply && \
+    ${NPM_BIN}/dotenvx run -f ${DIR}/.env -- ${NPM_BIN}/aiiinotate serve prod;