imerss-bioinfo 1.0.0

package/README.md

@@ -0,0 +1,156 @@
+# IMERSS Bioinformatics Working Group Data Archive, Pipelines and Visualisations
+
+This repository houses the working data of the [IMERSS](https://imerss.org/) [biodiversity informatics working group](https://imerss.org/2019/01/01/biodiversity-informatics-working-group/)
+together with algorithms for transforming, reconciling and projecting observation and checklist data into formats
+suitable for publication, submission to global authorities such as [GBIF](https://www.gbif.org/), as well as
+map-based and graphical visualisations.
+
+# Mini-portal
+
+A [mini-portal](https://imerss.github.io/imerss-bioinfo/) presenting a showcase of the latest versions of all of
+IMERSS' user-facing visualisations is available in [GitHub Pages](https://imerss.github.io/imerss-bioinfo/).
+
+# Data Archive
+
+All data is stored in CSV files in subdirectories under [data](data) although some unprocessed checklist data is a
+variety of formats such as XLSX and PDF.
+
+Authoritative data resulting from the full reconciliation of upstream catalogues and curated summaries for the Galiano
+Data Paper part 1: Marine Zoology is held in [data/dataPaper-I](data/dataPaper-I) and [data/dataPaper-I-in](data/dataPaper-I-in).
+In particular, the reconciled and normalised observation data is at [data/dataPaper-I/reintegrated-obs.csv](data/dataPaper-I/reintegrated-obs.csv),
+reconciled checklists derived from this data is at [/data/dataPaper-I/reintegrated.csv](/data/dataPaper-I/reintegrated.csv).
+The corresponding checklists derived from the curated summaries is at [/data/dataPaper-I-in/reintegrated.csv](/data/dataPaper-I-in/reintegrated.csv).
+
+Full intructions for running the data paper part I pipeline are held in
+[Galiano Data Paper Vol I.md](Galiano%20Data%20Paper%20Vol%20I.md), but here follows an overview of the scripts and overall
+installation instructions.
+
+# Data Pipeline
+
+The full data pipeline for deriving the observation data and its checklists from the upstream raw catalogues for
+the data paper is held at [/data/dataPaper-I/fusion.json5](/data/dataPaper-I/fusion.json5). This is stored in
+the [JSON5](https://json5.org/) format.
+This file refers to the most up-to-date versions of the constituent catalogues in their various other subdirectories
+under [/data](/data). Many of these subdirectories contain their own "fusion" files for producing other summaries
+and visualisations as seen in the [mini-portal](https://imerss.github.io/imerss-bioinfo/).
+
+All of the code operating the data pipelines is written in JavaScript running either in [node.js](http://nodejs.org) or
+the browser.
+After [installing git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) to check out this repository via
+
+    git clone IMERSS/imerss-biodata
+
+, and then installing node.js from its [download](https://nodejs.org/en/download/), you can install the pipeline's
+dependencies by running
+
+    npm install
+
+in the checkout directory.
+
+The scripts in [src](src) will then be ready to run via various `node` commands. These scripts are currently of a
+basic quality and not easily usable without being in close contact with members of the IMERSS BIWG team. Please join us
+in our [Matrix](https://matrix.org/) channels [IMERSS general](https://matrix.to/#/#imerss-general:matrix.org) and
+[IMERSS tech](https://matrix.to/#/#imerss-tech:matrix.org).
+
+More detailed documentation for these scripts will be forthcoming, but the principal ones involved in the data paper
+pipeline are:
+
+    taxonomise.js [obs or summary CSV file] [--map map JSON file] [-o output file]
+
+Main point of ingesting a collection of catalogues or summary in some CSV form. Accepts a "fusion" JSON5 file laying out
+all of the constituent catalogues files as CSV together with accompanying "map" files mapping columns for ingestion and
+rendering. Produces one or two `reintegrated` files combining the catalogues. This operates several stages of
+normalisation, including normalising species names with respect to the internal ontology mapping file
+[taxon-swaps.json5](data/taxon-swaps.json5), filtering out unwanted taxa, georeferencing correction patches,
+and filtering with respect to a project boundary defined in GeoJSON. Details of all the capabilities of this
+"Swiss Army knife" data processor can be followed in the data paper fusion file [data/dataPaper-I/fusion.json5](data/dataPaper-I/fusion.json5).
+
+    inatObs.js
+
+Downloads all observations in an iNaturalist project, whilst remaining within the iNaturalist team's recommended request
+rate limit (one per second). Currently hardwired to download Animalia from the Galiano project. Produces a CSV file suitable
+to form part of the input to `taxonomise.js`.
+
+    materialise.js
+
+From the curated summaries end, downloads a collection of summaries mapping in a Google Sheets directory. Currently hardwired
+to download the "Animalia" at [Galiano Data Paper 2021/Marine Life/Animalia](https://drive.google.com/drive/folders/14gItR0p_4wYo4K1__tyPYeIuc2yLr6l_)
+in the author's drive mapping.
+
+    compile.js
+
+Given the folder of files output by `materialise.js`, combines them together - currently hardwired to produce a file `Animalia.csv`.
+
+    compare.js
+
+Given the results of two `taxonomise.js` outputs as `reintegrated.csv` file checklists, compares them for any discrepancies,
+after casting out any records for higher taxa which are trumped by a more specific species records. Outputs two CSV files
+`excess1.csv` and `excess2.csv`.
+
+    arphify.js
+
+Given a pipeline specification such as the one in [data/dataPaper-I-in/arpha-out.json5], accepts both a normalised
+observation file such as [data/dataPaper-I/reintegrated-obs.csv](data/dataPaper-I/reintegrated-obs.csv) and a normalised
+curated summary file such as [/data/dataPaper-I-in/reintegrated.csv](/data/dataPaper-I-in/reintegrated.csv) and emits a
+directory of XLSX spreadsheets in the form accepted by the [ARPHA writing tool](https://arpha.pensoft.net/) used for
+submission of biodiversity data papers as well as a Darwin Core CSV file
+[Materials.csv](data/dataPaper-I-In/arphified/Materials.csv) suitable for submission to GBIF.
+
+    wormify.js
+
+Accepts arguments as for `taxonomise.js`. Produces a scratch `reintegrated-WoRMS.csv` file after downloading and
+caching WoRMS taxon files into [data/WoRMS](data/WoRMS) which compares the `authority` value listed against the
+one found in the WoRMS API.
+
+We dream of turning these pipelines into easily usable pluralistic graphical pipelines deployed on public live infrastructure
+such as github and Google Sheets.
+
+# Visualisations
+
+## Sunburst visualisation and map view
+
+Observation and checklist data derived from condensed summaries such as, e.g. [data/dataPaper-I/reintegrated.csv](data/dataPaper-I/reintegrated.csv)
+is in a sunburst partition layout inspired by <https://bl.ocks.org/mbostock/4348373>,
+<https://www.jasondavies.com/coffee-wheel/>, as well as a map-based view rendered with [Leaflet](https://leafletjs.com/).
+
+## Preparing data for visualisation
+
+Data is compiled into a compressed JSON representation from CSV sources via a command-line script.
+
+To convert a CSV file, run `marmalise.js` e.g. via a line such as
+
+    node src/marmalise.js data/dataPaper-I/reintegrated.csv --map data/dataPaper-I/combinedOutMap.json
+
+By default this will produce a `Life.json.lz4` file which can be copied into a suitable location, e.g. in the </data>
+directories and then referred to in the JavaScript initialisation block seen, e.g. in the various [index.html](index.html)
+files in this root. You can supply a `-o` option to output a file of a chosen name at a chosen path.
+
+To preview the web UI, host this project via some suitable static web server and then access its `index.html`.
+
+Condensed versions of the visualisation source files suitable for production hosting (JS and CSS) can be output to the
+<build> directory via `node build.js`.
+
+You can see such visualisations running online at locations like
+
+https://biogaliano.org/map-prototype/
+
+https://biogaliano.org/galiano-data-paper-map-view/
+
+And also a gallery in our [visualisations mini-portal](https://imerss.github.io/imerss-bioinfo/).
+
+These visualisations are entirely static and so easy to host at any kind of site simply by uploading a folder of files
+and injecting an initialisation block into the markup such as
+
+````html
+<script>
+    hortis.sunburstLoader(".fl-imerss-container", {
+        colourCount: "undocumentedCount",
+        selectOnStartup: "Life",
+        vizFile: "data/Galiano/Galiano-Life.json.lz4",
+        phyloMap: "%resourceBase/json/emptyPhyloMap.json",
+        commonNames: false
+    });
+</script>
+````
+
+together with `<script>` and `<style>` references to the built files.

package/build.js

@@ -0,0 +1,196 @@
+/* eslint-env node */
+"use strict";
+
+const readline = require("readline"),
+    fs = require("fs-extra");
+
+const fluid = require("infusion");
+const terser = require("terser");
+
+const buildIndex = {
+    excludes: [
+        "jquery.js"
+    ],
+    localSource: [
+        "src/client/css/imerss-viz.css",
+        "src/auxBuild/restoreJQuery.js",
+        "src/lib/lz4.js",
+        "src/client/js/imerss-viz.js",
+        "src/client/js/autocomplete.js",
+        "src/client/js/colour.js",
+        "src/client/js/leafletMap.js",
+        "src/client/js/datasetControls.js",
+        "src/client/js/leafletMapWithGrid.js",
+        "src/client/js/leafletMapWithRegions.js",
+        "src/client/js/renderSVG.js",
+        "src/client/js/tabs.js",
+        "src/client/js/checklist.js",
+        "src/client/js/xetthecum.js"
+    ],
+    codeHeader: "",
+    codeFooter: "", // "\njQuery.noConflict()",
+    copy: [{
+        src: "node_modules/infusion/src/lib/jquery/core/js/jquery.js",
+        dest: "docs/js/jquery.js"
+    }, {
+        src: "src/client/img",
+        dest: "docs/img"
+    }, {
+        src: "src/client/json",
+        dest: "docs/json"
+    }, {
+        src: "src/client/html",
+        dest: "docs/html"
+    }, {
+        src: "src/client/css/imerss-viz-wp-overrides.css",
+        dest: "docs/css/imerss-viz-wp-overrides.css"
+    }, {
+        src: "src/client/css/xetthecum.css",
+        dest: "docs/css/xetthecum.css"
+    }, {
+        src: "src/client/css/xetthecum-shared.css",
+        dest: "docs/css/xetthecum-shared.css"
+    }, {
+        src: "src/client/css/xetthecum-external.css",
+        dest: "docs/css/xetthecum-external.css"
+    }, {
+        src: "src/client/css/pepiowelh.css",
+        dest: "docs/css/pepiowelh.css"
+    }, {
+        src: "src/buildSource/Galiano Life List.html",
+        dest: "docs/Galiano Life List.html"
+    }, {
+        src: "src/buildSource/Comprehensive Lists.html",
+        dest: "docs/Comprehensive Lists.html"
+    }, {
+        src: "src/buildSource/Squamish Life List.html",
+        dest: "docs/Squamish Life List.html"
+    }, {
+        src: "src/buildSource/dataPaperSunburstAndMap.html",
+        dest: "docs/dataPaperSunburstAndMap.html"
+    }, {
+        src: "src/buildSource/Data Paper Part I Visualisation.html",
+        dest: "docs/Data Paper Part I Visualisation.html"
+    }, {
+        src: "src/buildSource/Valdes Island Biodiversity.html",
+        dest: "docs/Valdes Island Biodiversity.html"
+    }, {
+        src: "src/buildSource/Xetthecum.html",
+        dest: "docs/Xetthecum.html"
+    }, {
+        src: "src/buildSource/Pe'pi'ow'elh.html",
+        dest: "docs/Pe'pi'ow'elh.html"
+    }, {
+        src: "src/buildSource/index.html",
+        dest: "docs/index.html"
+    }, {
+        src: "data/dataPaper-I/Life.json.lz4",
+        dest: "docs/data/dataPaper-I/Life.json.lz4"
+    }, {
+        src: "data/Valdes/Life.json.lz4",
+        dest: "docs/data/Valdes/Life.json.lz4"
+    }, {
+        src: "data/Galiano/Galiano-Life.json.lz4",
+        dest: "docs/data/Galiano/Galiano-Life.json.lz4"
+    }, {
+        src: "data/Comprehensive Lists/Comp-Life.json.lz4",
+        dest: "docs/data/Comprehensive Lists/Comp-Life.json.lz4"
+    }, {
+        src: "data/Galiano/Galiano_map_0.js",
+        dest: "docs/data/Galiano/Galiano_map_0.js"
+    }, {
+        src: "data/Galiano WoL/Life.json.lz4",
+        dest: "docs/data/Galiano WoL/Life.json.lz4"
+    }, {
+        src: "data/Xetthecum/Life.json.lz4",
+        dest: "docs/data/Xetthecum/Life.json.lz4"
+    }, {
+        src: "data/Squamish/Squamish-Life.json.lz4",
+        dest: "docs/data/Squamish/Squamish-Life.json.lz4"
+    }, {
+        src: "data/Pepiowelh/Life.json.lz4",
+        dest: "docs/data/Pepiowelh/Life.json.lz4"
+    }]
+};
+
+
+const readLines = function (filename) {
+    const lines = [];
+    const togo = fluid.promise();
+    const rl = readline.createInterface({
+        input: fs.createReadStream(filename),
+        terminal: false
+    });
+    rl.on("line", function (line) {
+        lines.push(line);
+    });
+    rl.on("close", function () {
+        togo.resolve(lines);
+    });
+    rl.on("error", function (error) {
+        togo.reject(error);
+    });
+    return togo;
+};
+
+const filesToContentHash = function (allFiles, extension) {
+    const extFiles = allFiles.filter(function (file) {
+        return file.endsWith(extension);
+    });
+    const hash = fluid.transform(fluid.arrayToHash(extFiles), function (troo, filename) {
+        return fs.readFileSync(filename, "utf8");
+    });
+    return hash;
+};
+
+const computeAllFiles = function (buildIndex, nodeFiles) {
+    const withExcludes = nodeFiles.filter(function (oneFile) {
+        return !buildIndex.excludes.some(function (oneExclude) {
+            return oneFile.indexOf(oneExclude) !== -1;
+        });
+    });
+    return withExcludes.concat(buildIndex.localSource);
+};
+
+const buildFromFiles = function (buildIndex, nodeFiles) {
+    const allFiles = computeAllFiles(buildIndex, nodeFiles);
+    console.log("allFiles " + allFiles);
+    nodeFiles.concat(buildIndex.localSource);
+
+    const jsHash = filesToContentHash(allFiles, ".js");
+    const fullJsHash = fluid.extend({header: buildIndex.codeHeader}, jsHash, {footer: buildIndex.codeFooter});
+    fluid.log("Minifying " + Object.keys(fullJsHash).length + " JS files ... ");
+    const promise = terser.minify(fullJsHash, {
+        mangle: false,
+        sourceMap: {
+            filename: "imerss-viz.js",
+            url: "imerss-viz.js.map"
+        }
+    });
+    promise.then(function (minified) {
+        fs.removeSync("docs");
+        fs.ensureDirSync("docs/js");
+        fs.writeFileSync("docs/js/imerss-viz-all.js", minified.code, "utf8");
+        fs.writeFileSync("docs/js/imerss-viz-all.js.map", minified.map);
+
+        const cssHash = filesToContentHash(allFiles, ".css");
+        const cssConcat = String.prototype.concat.apply("", Object.values(cssHash));
+
+        fs.ensureDirSync("docs/css");
+        fs.writeFileSync("docs/css/imerss-viz-all.css", cssConcat);
+        buildIndex.copy.forEach(function (oneCopy) {
+            fs.copySync(oneCopy.src, oneCopy.dest);
+        });
+        fluid.log("Copied " + (buildIndex.copy.length + 3) + " files to " + fs.realpathSync("docs"));
+    });
+};
+
+fluid.setLogging(true);
+
+const linesPromise = readLines("gh-pages-nm.txt");
+
+linesPromise.then(function (lines) {
+    buildFromFiles(buildIndex, lines);
+}, function (error) {
+    console.log(error);
+});