@paulhectork/aiiinotate 0.12.0

package/docs/dev_documentation/dev_architecture.md

@@ -0,0 +1,88 @@
+# Architecture
+
+This is a high-level overview of the Aiiinotate and a good place to start if you want to work on the app
+
+---
+
+## Top level plugins
+
+Fastify uses a [**plugin system**](https://fastify.dev/docs/latest/Guides/Plugins-Guide/). In short, it means that,
+- to access the global `fastify` instance, your code needs to be a plugin
+- all plugins must export a single function that registers the plugin on the global fastify instance
+
+This can be quite convoluted and conflict with a "normal" module architecture, so we do a mix of the two:
+- **all modules at the root** of `src/` are plugins (except `config`)
+- **nested plugins** within each of the above are defined when they need to access the fastify instance.
+
+```
+./src/fixtures/ - module storing and serving data files for test fixtures
+./src/schemas/    - JsonSchemas definition and validation
+./src/db/         - connects the app to the mongojs database
+./src/data/       - routes and modules to read/write data
+
+```
+
+---
+
+## The `data` plugin
+
+### In general
+
+As you can see, the `data` plugin stores the vast majority of the app's logic. It defines:
+- **`route.js` files** HTTP routes for user interactions
+- **`collection classes`** (`abstractCollection.js`, `manifests(2|3).js`, `annotations(2|3).js`) for each MongoJS collection that handles all internal functionnalities for all collections (read/write/update/delete data)
+
+Each `route` and `class` is a fastify plugin. Since the `data` plugin is registered last, it can access functionalities from all other root plugins defined above.
+
+### Structure and inheritence
+
+```
+├── index.js               // `data` plugin root
+├── routes.js              // generic routes
+├── annotations
+│   ├── annotations2.js    // plugin for IIIF presentation 2 annotations
+│   ├── annotations3.js    // plugin for IIIF presentation 3 annotations
+│   ├── routes.js          // routes for both plugins
+├── collectionAbstract.js  // abstract class with functionnalities for all plugins
+├── manifests
+    ├── manifests2.js      // plugin for IIIF presentation 2 manifests
+    ├── manifests3.js      // plugin for IIIF presentation 3 manifests
+    ├── routes.js          // routes for both plugins
+```
+
+At a high level, `route` files receive data and relegate internal mongoJS interaction to the `collection classes`. Once these processes are finished, the `collection classes` return data to the `routes`, which send the response to the users. So:
+
+- the work of `routes` is to define input/output JsonSchemas and delegate to the proper `collection classes`
+- the work of `classes` is to do the database interaction, data formatting etc.
+
+Our 5 `collection classes` (`collectionAbstract`, `annotations2`, `annotations3`, `manifests2`, `manifests3`) use class inheritence: all classes inherit from `collectionAbstract`
+- `collectionAbstract` defines collection-agnostic processes that can be used by all other classes.
+- other classes implement functionnalities for a specific data type and collection.
+
+---
+
+## Details: why plugins are complicated ?
+
+If using a plugin-only architecture, **you should not do "local" imports** (imports from one part of your app to another): everything that you want to communicate between apps must be registered as a plugin on the global fastify instance, and then other files need to access the fastify instance:
+
+```js
+// here, we define a root plugin with 2 subplugins (1st is a decorator, 2nd a normal plugin).
+fastify.register((instance, opts, done) => {
+
+  // decorator
+  instance.decorate('util', (a, b) => a + b)
+  console.log(instance.util('that is ', 'awesome'))
+
+  // plugin
+  fastify.register((instance, opts, done) => {
+    console.log(instance.util('that is ', 'awesome'))
+    done()
+  })
+
+  done()
+})
+```
+
+In the above example, functionnalities defined in each plugin (except decorators) are encapsulated: a plugin is a scope and everything that's defined in a plugin must be accessed through the fastify instance. There is also the question of plugins vs. decorators, plugin definition order...
+
+In short, if using a plugin-only architecture, every single one of your files would have to be a plugin, registered in the proper order... It would be hell.

package/docs/dev_documentation/dev_db.md

@@ -0,0 +1,58 @@
+# DATABASE AND MIGRATIONS
+
+---
+
+## Collections
+
+- `annotations2`: a collection of annotations following the IIIF Presentation API 2.x (OpenAnnotation specification)
+- `annotations3`: a collection of annotations following the IIIF Presentation API 3.x (WebAnnotation specification)
+- `manifests2`: an index of manifests (manifest ID and array of canvas IDs of the manifest) following the IIIF Presentation API 2.x (OpenAnnotation specification)
+- `manifests3`: an index of manifests (manifest ID and array of canvas IDs of the manifest) following the IIIF Presentation API 3.x (WebAnnotation specification)
+
+Conversion between IIIF Presentation APIs 2 and 3 is not implemented.
+
+---
+
+## Validation rules
+
+Two levels of validation rules are implemented:
+- route-level validation
+- database-level validation
+
+At route-level, we ensure that the data has the absolutely necessary data to work with annotations.
+
+At database-level (see [`src/schema`](https://github.com/Aikon-platform/aiiinotate/blob/main/src/schemas/) module, we define a complete model (in JSONSchema) to describe annotations.
+
+**TLDR**: when inserting annotations (i.e., from an AnnotationList)
+1. at [route-level](https://github.com/Aikon-platform/aiiinotate/blob/main/src/data/annotations/routes.js), we ensure that the AnnotationList is generally valid (contains a `resource` with annotations, each annotations contains a `on`...)
+2. in the [`Annotations2`](https://github.com/Aikon-platform/aiiinotate/blob/main/src/data/annotations/annotations2.js) module, we do some extra checks and extra minimal reformatting to ensure that our annotations all have the same structure (i.e., convert `annotation.on` values to an array of `SpecificResources`). After finishing the cleanup, we are certain that an annotation can be inserted
+3. when inserting an annotation in the MongDB `annotations2`, MongoDB uses the `annotation` schema defined in [`src/schemas/schemasPresentation2`](https://github.com/Aikon-platform/aiiinotate/blob/main/src/schemas/schemasPresentation2.js) to validate annotations before inserting.
+
+---
+
+## Migrations
+
+Migrations are used to specify changes to the database's structure (creation and deletion of collections and indexes, changes to collection options such as validation rules etc.). Migration is done using [`migrate-mongo`](https://github.com/seppevs/migrate-mongo), a command-line interface for npm.
+
+*Note that here, migrations only concern changes to the structure, not changes to the data.*
+
+Where things get a bit complicated is that, while in use, our app uses a single database, in **dev mode, our app uses 2 databses**:
+- a `main` database (default database that will be put in production)
+- a `test` database (empty database to run tests, add dummy data to and so on).
+
+For consistency, `test` must mirror the structure of `main`: same collections, indexes, validation rules etc. So, we must manage our migrations so that **all changes to the structure of `main` are reflected in the structure of `test`**. This is not possible natively through `migrate-mongo`. Managing both databases in parrallel manually would risk a lot of inconsistencies.
+
+
+Our solution is to automate the management of both databases in parrallel:
+- there are **2 migration config files**, one for each database
+- both migration config files point to **the same migration scripts folder**, so that both database can apply the same migrations
+- execute all migrations by **wrapping `migrate-mongo` in a homemade script**: [`scripts/migrations.sh`](../scripts/migrations.sh).
+
+```
+root/
+  |__migrations/
+       |__migrationScripts              // folder containing all migration scripts. consumed by both migration config files.
+       |__baseConfig.js                 // base config file that both config files are derived from
+       |__migrate-mongo-config-main.js  // config file for the main db
+       |__migrate-mongo-config-test.js  // config file for the test db
+```

package/docs/dev_documentation/dev_iiif_compatibility.md

@@ -0,0 +1,43 @@
+# Development notes: IIIF compatibility and variations between IIIF prsentation APIxs 3.x and IIIF 2.x
+
+The annotation server uses an internal data model that can convert to and from annotation models defined in IIIF 2.x and 3.x presentation API. Respectively:
+- IIIF 3.x follows the [W3C Web Annotations standard (WA)](https://www.w3.org/TR/annotation-model/)
+- IIIF 2.x follows the [Open Annotations standard (OA)] (http://www.commonsemantics.com/oa/Open%20Annotation%20Data%20Model%20Primer.html#examples)
+
+In general, there are [breaking changes](https://iiif.io/api/presentation/3.0/change-log/#1-breaking-changes) between IIIF 2.x and 3.x. For annotations, the OA model allows things that are not possible in WA, and vice versa. Our internal data model tries to store a maximum amount of data from WA and OA annotations, and then do the necessary conversion to produce valid OA/WA annotations.
+
+To make things funnier, the original Open Annotations standard website is no longer available.
+
+## Annotations and the `motivation` attribute
+
+### The problem
+
+Both OA and WA have a `motivation` attribute that describes the function of the annotation in relation to the Manifest. Mainly, the use of `painting` (WA) / `sc:painting` (OA) indicates that the annotation is the primary content of the canvas and should be rendered.
+- WA allows 2 values: `painting`, `supplementing`
+- OA allows a more values and defines an ontology. [IIIF 2.1 specifies](https://iiif.io/api/presentation/2.1/#comment-annotations) that the value given to a non-painting annotation should be `oa:commenting`
+
+In practice, in SAS/Aikon, non-painting annotations have the `motivation`: `[ "oa:tagging", "oa:supplementing" ]`.
+
+### What we do
+
+In an annotation, the `motivation` field is a `string[]` array that accepts all values it receives.
+- **when writing data from IIIF to DB**, `motivation` values are connected to string arrays but are not verified.
+- **when converting data from DB to IIIF**, `motivation` values stored in the database are processed:
+    - converting to IIIF 2.x:
+        - if WA values are stored in the DB, they are converted to 2.x: `painting => sc:painting`, `supplementing => oa:commenting`.
+        - if OA values are stored in the DB, they are kept as is
+    - converting to IIIF 3.x:
+        - if WA values are in the DB, they are kept as is
+        - if OA values are in the DB, they are converted to `painting` (if `sc:painting` is in the array of motivations stored) or `commenting` otherwise.
+
+## DCTypes and the `body.type` attribute
+
+### Problem
+
+- WA allows : `Dataset`, `Image`, `Video`, `Sound`, `Text`
+- OA allows : `text`, `image`, `sound`, `dataset`, `software`, `interactive`, `event`, `physical`, `object`
+- In IIIF 2.x, values are prefixed by `dctypes:`: `dctypes:text`.
+
+### Solution
+
+Since there is a close mapping between the two, we convert to WA allowed types before saving to database.

package/docs/dev_documentation/dev_notes_quirks_and_troubleshooting.md

@@ -0,0 +1,143 @@
+# Notes, quirks and troubleshooting
+
+---
+
+## Notes
+
+### Uniqueness
+
+As of writing (09.10.25), there are no uniqueness constraints on annotations. There is only a uniqueness constraint on collection `manifest2` on field `manifest2.@id` (the ID of a manifest).
+
+Ideally, we would want to avoid having duplicate annotations in the database. This is more complicated in practice: at least for `annotions2`, an annotation's `@id` field is re-generated and a random part (an UUID) is generated at insert time. This means that, when trying to store the same annotation (with the same `@id`), the `@id` is changed, and so a different value is inserted.
+
+This means that we can't have a uniqueness constraint on `@id` or `id` fields of annotations. Another option would be to have a uniqueness constraint on annotation targets (no 2 annotations can annotate the same region), but this behaviour seems brittle in practice, so it's not yet implemented.
+
+### Concurrency
+
+For clients, concurrency/parrallelization (i.e., with JS `Promise.all()`) on insert/update should be avoided because it can cause a data race: several processes can attempt to write the same thing in parrallel.
+
+For example, when inserting annotations, the manifests related to each annotation are inserted in parrallel. Since this is a side effect, 2 processes may unknowingly try to insert the same manifest in the database, which causes a uniqueness constraint to fail. This error can be hard to debug, so it's best to avoid concurrency at write time.
+
+---
+
+## Dev quirks
+
+Sometimes, node/fastify can be weird. When scratching your head at dev errors, look here:)
+
+### Error swallowing at app build
+
+Errors that happen when a plugin is being registered will cause the app's startup to fail, without necessarily throwing an error.
+
+This is especially true for test cases that build the fastify app:
+
+```
+test
+=> build fastify app
+=> build step fails silently
+=> test fails
+```
+
+#### Troubleshooting
+
+- normal behaviour: when a runtime error happens, the failing test will log the error on the console
+- what this error looks like:
+    - tests fail **very quickly**,
+    - without throwing an error, seemingly without even running the test suite
+    - the proces doesn't exit, although all tests have failed
+- when `npm run test` fails like this, run `npm run start`. See if normal startup throws an error
+    - NOTE: normal startup not throwing does NOT mean that the build step necessarily worked
+
+#### Possible help/solutions
+
+- find a way to stisfyingly use `try...catch` at plugin registration.
+- look into these issues: [2694](https://github.com/fastify/fastify/issues/2694)
+    - in particular, it may be an issue specific to async plugins ?
+
+### Route response schema definition
+
+For some reason, route schema definition is much less flexible for responses than for queries.
+
+#### The problem
+
+**Query schemas**: In queries (`schema.params` or `schema.querystring`), fastify is very permissive. You can use unresolved schemas (with `$ref`), save entire schemas as JsonSchemas ...
+
+**Response schemas**: Response schemas have more constraints.
+- **response schemas are defined at route level** and cannot be stored as full JsonSchemas:
+    ```js
+    // this will be an invalid response schema: it is trying to store a complete response schema as a JsonSchema
+    fastify.addSchema({
+        $id: makeSchemaUri("routeResponsePost"),
+        properties: {
+            200: { ... },
+            500: { ... },
+        }
+    })
+
+    // this is a fixed version: an object containing schemas, not a full JsonSchema, to be used inside a Route definition
+    schema: {
+        response: {
+            200: { ... },
+            500: { ... }
+        }
+    }
+    // if you want to reuse a schema, store it as a JS object and import it.
+    ```
+- **unresolved response schemas (`$ref`) are forbidden**. You cannot use `$ref`. In the app, use `fastify.schemasResolver` to resolve the schema to a plain `JsonSchema` without `$ref`.
+
+#### The fix
+
+In short, here's **how to use shared schemas in responses**:
+1. Define payload schemas for different response cases:
+    ```js
+    // in case of a POST success
+    fastify.addSchema({
+        $id: makeSchemaUri("routeResponseInsert"),
+        type: "object",
+        // ...properties
+    });
+
+    // in case of a POST error
+    fastify.addSchema({
+        $id: makeSchemaUri("routeResponseError"),
+        type: "object",
+        // ...properties
+    });
+    ```
+2. Resolve schemas and use in responses
+    ```js
+    const routeResponseInsert = fastify.getSchema("...");
+    const routeResponseError = fastify.getSchema("...");
+    fastify.post(
+        "/annotations/:iiifPresentationVersion/create",
+        {
+            schema: {
+                body: routeAnnotations2Or3Schema,
+                response: {
+                    200: routeResponseInsert,
+                    500: routeResponseError
+               }
+            }
+        },
+        async (req, rep) => {}
+    )
+    ```
+
+### `migrate revert-all && migrate apply`
+
+#### The problem
+
+Sometimes, you want to remove all migrations and then rerun them. This will cause an error like:
+
+```bash
+{
+    ...,
+    errmsg: 'namespace aiiinotate.manifests2 already exists, but with different options: { uuid: UUID("65ca87c4-ddce-4c2d-a6f9-08ff6b70c0a7"), validationLevel: "strict", validationAction: "error" }',
+    code: 48,
+    codeName: 'NamespaceExists',
+    ...
+}
+```
+
+#### The fix
+
+I figure it's an error due to cache or something like that. Either wait or manually delete the collection or the entire database using `mongosh`.

package/docs/docker.md

@@ -0,0 +1,98 @@
+# Docker usage
+
+The `docker/` folder contains:
+- a `Dockerfile` for aiiinotate
+- a `docker-comose` to integrate aiiinotate with MongoDB, with volume persistance
+- a script to import data into a dockerized aiiinotate instance
+
+---
+
+## Docker setup
+
+1. [Install docker](https://docs.docker.com/engine/install/)
+2. On linux, add your user to the `docker` group (see [full docs](https://docs.docker.com/engine/install/linux-postinstall/))
+
+```bash
+sudo groupadd docker
+sudo usermod -aG docker $USER
+newgrp docker
+```
+
+---
+
+## Usage
+
+Note that all `docker` commands must be run from the `docker/` directory!
+
+1. **Clone the repo** and `cd` in it
+
+```bash
+git clone git@github.com:Aikon-platform/aiiinotate.git
+cd aiiinotate/
+```
+
+2. **Create a `.env` file** at `docker/.env` (from the root of the project):
+```bash
+cp ./config/.env.template ./docker/.env
+vim ./docker/.env  # do your edits
+```
+
+3. **Edit the `.env` file**. Pay attention to the following values:
+- `MONGODB_HOST=mongo` (container name)
+- `AIIINOTATE_HOST=aiiinotate` (container name)
+- `AIIINOTATE_SCHEME=http`
+- `AIIINOTATE_LOG_TARGET=stdout` (Docker aldready collects stdout logs, so setting log target to `file` is redundant)
+- set `AIIINOTATE_PUBLIC_URL` to the URL at which your aiiinotate instance will be accessible (e.g., `aiiinotate.enpc.fr`).
+
+4. **Build the containers**
+
+```bash
+cd docker
+bash docker.sh build
+```
+
+5. **Start the containers**
+
+```bash
+bash docker.sh start
+```
+
+6. (Optional) **import data** into the running aiiinotate instance. `path/to/dir` is a path to a folder containing AnnotationLists or manifests to import.
+
+```bash
+bash docker_aiiinotate_import.sh <manifests|annotations> path/to/dir
+```
+
+---
+
+## Troubleshooting and useful commands
+
+View logs:
+
+```bash
+docker logs docker-aiiinotate-1 -f
+```
+
+Access `mongosh` within the Mongo container:
+
+```bash
+docker exec -it docker-mongo-1 mongosh
+```
+
+Check running ports in the Web container:
+
+```bash
+docker exec -it docker-web-1 ss -tnl
+```
+
+View globally installed NPM packages in the Web container:
+
+```bash
+docker exec -it docker-web-1 npm list -g --depth=0
+```
+
+CURL the Web container
+
+```bash
+docker exec -it docker-web-1 curl http://0.0.0.0:4444   # change 4444 by your $AIIINOTATE_PORT
+```

package/docs/scalability.md

@@ -0,0 +1,34 @@
+# Scalability
+
+## Results
+
+In [benchmarks](https://github.com/paulhectork/aiiinotate-benchmark), aiiinotate response times are between 1/100th and 1/10th of a second with up to 10,000,000 annotations. 
+
+![benchmark results](./includes/report_benchmark_aiiinotate_2026-05-28-02:50:48_7steps.png)
+
+## Discussion
+
+### Write times
+
+Write times (`Write anno. list`, `Write anno.`) presented above are in all probability lower than in real world applications.
+
+Indeed, a big bottleneck when inserting annotations is to fetch an annotation's target manifest through HTTP, in ordrer to fill the `canvasIdx` field.
+
+When running the benchmark, all annotation URIs are on `https://localhost`, an URI on localhost that is inaccessible. The request thus fails instantly.
+
+The reason this was done is that HTTP requests are non-deterministic: if we make requests to an external server storing IIIF manifests, then we end up also benchmarking this server. By fetching manifests on an inaccessible `https://localhost`, this non-deterministic process becomes deterministic and the benchmark makes more sense. 
+
+###  >10M annotations
+
+After benchmarking for 10M annotations, we delete the database and attempt to insert 100M annotations to benchmark response times.
+
+With 100M annotations stored, the database size becomes an issue. These scalability limits are mostly caused by MongoDB, and not by the aiiinotate app itself: the larger the database, the more hard drive memory it uses, and the more RAM it uses (since indexes are stored in the RAM). To store >100M annotations, we should scale hardware:
+
+- **vertical scaling**: more RAM and disk space
+- **horizontal scaling**: use a Mongo [sharded cluster](https://www.mongodb.com/docs/manual/sharding/).
+
+It should be noted that the benchmark stress tests your machine, aiiinotate and Mongo at the same time:
+- results presented above are obtained by running the benchmark, aiiinotate and Mongo on a single machine. All 3 are thus competing for RAM access and CPU.
+- it uses mutliple threads, costly JSON-stringification, fast I/O and has a high throughput: it runs a lot of queries to aiiinotate, and to the MongoDB. 
+
+In real-world examples, with less throughput and when not running the benchmark itself, there should be less stress on your MongoDB server, thus less CPU usage, thus possibly more scalability.