firestore-meilisearch 0.1.0

package/guides/IMPORT_EXISTING_DOCUMENTS.md

@@ -0,0 +1,74 @@
+The `firestore-meilisearch` script is for use with the official Firebase Extension [**Meilisearch**](https://github.com/firebase/extensions/tree/main/firestore-meilisearch).
+
+### Overview
+
+The import script reads all existing documents in a Cloud Firestore collection or in sub-collections and index them into an index in Meilisearch.
+
+#### Important notes
+
+- You must run the import script over the entire collection **_after_** installing the Meilisearch extension in Firebase; otherwise all the data written in the meantime or during the import in your database might not be exported to your `${param:MEILISEARCH_INDEX_NAME}` index.
+
+- The import script may take a while to complete if your collection is large. In which case we suggest setting a larger batch size in the configuration.
+
+- You cannot use wildcard notation in the collection path (i.e. `/collection/{document}/sub_collection}`). Instead, you can use a collectionGroup query. To use a collectionGroup query, provide the collection name value as `${COLLECTION_PATH}`, and set `${COLLECTION_GROUP_QUERY}` to true.
+
+Warning: A collectionGroup query will target every collection in your Firestore project with the provided `${COLLECTION_PATH}`. For example, if you have 10,000 documents with a sub-collection named: actors, the import script will query every document in 10,000 actors collections.
+
+### Run the script
+
+The import script uses several values from the configuration provided when you installed the extension:
+
+- `${PROJECT_ID}`: the project ID for the Firebase project in which you installed the extension
+- `${COLLECTION_PATH}`: the collection path that you specified during extension installation
+- `${COLLECTION_GROUP_QUERY}`: uses a collectionGroup query if this value is "true". For any other value, a collection query is used.
+- `${MEILISEARCH_INDEX_NAME}`: the UID of the Meilisearch index that you specified for your indexation during extension installation
+- `${BATCHSIZE}`: the number of documents you want to import into Meilisearch at once
+- `${MEILISEARCH_HOST}`: the URL of the host of the Meilisearch database that you specified during extension installation
+- `${MEILISEARCH_API_KEY}`: the Meilisearch API key with permission to perform actions on indexes you specified during extension installation
+
+Run the import script using [`npx` (the Node Package Runner)](https://www.npmjs.com/package/npx).
+
+1.  Make sure that you've installed the required tools to run the import script:
+
+    - To access the `npm` command tools, you need to install [Node.js](https://www.nodejs.org/).
+    - If you use `npm` v5.1 or earlier, you need to explicitly install `npx`. Run `npm install --global npx`.
+
+2.  Set up credentials. The import script uses Application Default Credentials to communicate with Firebase. Please follow the instructions to [create and use a service account](https://cloud.google.com/docs/authentication/production#obtaining_and_providing_service_account_credentials_manually).
+
+3.  Run the import script interactively via `npx` by running the following command:
+
+  - Run interactively:
+    ```bash
+    npx firestore-meilisearch
+    ```
+    When prompted, enter the Cloud Firestore collection path that you specified during extension installation, `${COLLECTION_PATH}` along with any other arguments.
+
+  - Run non-interactively with paramaters:
+    ```bash
+    npx firestore-meilisearch \
+      --project <project_id> \
+      --source-collection-path <collection_name> \
+      --index <index_uid> \
+      --batch-size <100/default=300> \
+      --host <host_address> \
+      --api-key <api_key> \
+      --non-interactive
+    ```
+    **Note**: The `--batch-size` and `--query-collection-group` arguments are optional. To see its usage, run the above command with `--help`.
+
+4. Check if the index and Document are imported in Meilisearch:
+
+  1. Check that the index has been created.:
+    ```bash
+    curl \
+      -X GET 'http://${MEILISEARCH_HOST}/indexes/${MEILISEARCH_INDEX_NAME}'
+    ```
+    **Note**:  This example must be launched in your terminal but if you are already using the Meilisearch SDK you can use the command related to the [latter](https://docs.meilisearch.com/reference/api/indexes.html#get-one-index)
+
+  2. Check that the documents has been added to your Meilisearch database:
+    ```bash
+    curl \
+      -X GET 'http://${MEILISEARCH_HOST}/indexes/${MEILISEARCH_INDEX_NAME}/stats'
+    ```
+    The response will contain the number of documents in Meilisearch index.
+    **Note**:  This example must be launched in your terminal but if you are already using the Meilisearch SDK you can use the command related to the [latter](https://docs.meilisearch.com/reference/api/stats.html#get-stat-of-an-index)

package/package.json

@@ -0,0 +1,21 @@
+{
+  "name": "firestore-meilisearch",
+  "version": "0.1.0",
+  "description": "Search in your Firestore content with Meilisearch",
+  "repository": "",
+  "author": "Meilisearch (https://meilisearch.com/)",
+  "license": "Apache-2.0",
+  "scripts": {
+    "emulator": "firebase ext:dev:emulators:start --test-params=test-params-example.env --import=dataset --project=meilisearch --config=./firebase.json",
+    "lint": "yarn --cwd functions lint",
+    "lint:fix": "yarn --cwd functions lint:fix",
+    "build": "yarn --cwd functions build",
+    "watch": "yarn --cwd functions watch",
+    "playground": "yarn --cwd functions playground",
+    "install:functions": "yarn --cwd functions",
+    "test": "yarn --cwd functions test",
+    "test:watch": "yarn --cwd functions test:watch",
+    "test:coverage": "yarn --cwd functions test:coverage",
+    "run-cli": "./functions/lib/import/index.js"
+  }
+}

package/script/version.sh

@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+
+## Setup
+
+set -e
+
+bold=$(tput bold)
+normal=$(tput sgr0)
+
+function isValidSemver() {
+  if [[ $1 =~ ^(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(-((0|[1-9][0-9]*|[0-9]*[a-zA-Z-][0-9a-zA-Z-]*)(\.(0|[1-9][0-9]*|[0-9]*[a-zA-Z-][0-9a-zA-Z-]*))*))?(\+([0-9a-zA-Z-]+(\.[0-9a-zA-Z-]+)*))?$ ]]; then
+    echo "$1"
+  else
+    echo ""
+  fi
+}
+
+## Version
+
+echo "replacing version number"
+
+version=$(isValidSemver $1)
+
+if [ -z $version ]; then
+  echo "$1 is not a valid semver version"
+  exit 1
+fi
+
+# extension.yaml
+sed -i '' "s/^version:.*/version: $version/" extension.yaml
+# package.json
+sed -i '' "s/\"version.*/\"version\": \"$version\",/" package.json ./functions/package.json
+# version.ts
+echo "export const version = '$version'" > ./functions/src/version.ts
+
+## Build
+
+echo "installing and building"
+
+npm install
+
+cd functions
+npm install
+npm run build
+cd ..
+
+## Commit files
+
+git add -u
+git commit -m "Update version for the next release (v$version)"
+echo "The updated files are ready to be pushed"

package/test-params-example.env

@@ -0,0 +1,9 @@
+# from firebase v9.20.0 the parsing of optional parameters does not work in command line
+# the MEILISEARCH_FIELDS_TO_INDEX must be defined to '' instead of empty
+
+LOCATION=us-central1
+COLLECTION_PATH=movies
+MEILISEARCH_FIELDS_TO_INDEX=''
+MEILISEARCH_INDEX_NAME=movies
+MEILISEARCH_HOST='http://127.0.0.1:7700'
+MEILISEARCH_API_KEY='masterKey'