@helloao/cli 0.0.4 → 0.0.5

package/README.md

@@ -1,37 +1,126 @@
-## Hello AO CLI
-
-A Command Line Interface (CLI) that makes it easy to generate and manage your own [Free Use Bible API](https://bible.helloao.org/).
-
-Additionally, it includes many functions and utilities that can make working with commonly formatted Bible data much easier.
-
-### Features
-
--   Supports [USFM](https://ubsicap.github.io/usfm/), [USX](https://ubsicap.github.io/usx/), and Codex (A JSON format).
--   Download over 1000 Bible translations from [fetch.bible](https://fetch.bible/).
--   Import Bible translations into a SQLite database.
--   Upload to S3, a zip file, or a local directory.
-
-### Usage
-
-```
-Usage: helloao [options] [command]
-
-A CLI for managing a Free Use Bible API.
-
-Options:
-  -V, --version                                         output the version number
-  -h, --help                                            display help for command
-
-Commands:
-  init [options] [path]                                 Initialize a new Bible API DB.
-  import-translation [options] <dir> [dirs...]          Imports a translation from the given directory into the database.
-  import-translations [options] <dir>                   Imports all translations from the given directory into the database.
-  generate-translation-files [options] <input> <dir>    Generates API files from the given input translation.
-  generate-translations-files [options] <input> <dir>   Generates API files from the given input translations.
-  upload-api-files [options] <dest>                     Uploads API files to the specified destination. For S3, use the format s3://bucket-name/path/to/folder.
-  fetch-translations [options] <dir> [translations...]  Fetches the specified translations from fetch.bible and places them in the given directory.
-  fetch-audio [options] <dir> [translations...]         Fetches the specified audio translations and places them in the given directory.
-                                                        Translations should be in the format "translationId/audioId". e.g. "BSB/gilbert"
-  fetch-bible-metadata <dir>                            Fetches the Theographic bible metadata and places it in the given directory.
-  help [command]                                        display help for command
+## Hello AO CLI
+
+A Command Line Interface (CLI) that makes it easy to generate and manage your own [Free Use Bible API](https://bible.helloao.org/).
+
+Additionally, it includes many functions and utilities that can make working with commonly formatted Bible data much easier.
+
+### Features
+
+-   Supports [USFM](https://ubsicap.github.io/usfm/), [USX](https://ubsicap.github.io/usx/), and Codex (A JSON format).
+-   Download over 1000 Bible translations from [fetch.bible](https://fetch.bible/).
+-   Import Bible translations into a SQLite database.
+-   Upload to S3, a zip file, or a local directory.
+
+### Usage
+
+```
+Usage: helloao [options] [command]
+
+A CLI for managing a Free Use Bible API.
+
+Options:
+  -V, --version                                         output the version number
+  -h, --help                                            display help for command
+
+Commands:
+  init [options] [path]                                 Initialize a new Bible API DB.
+  import-translation [options] <dir> [dirs...]          Imports a translation from the given directory into the database.
+  import-translations [options] <dir>                   Imports all translations from the given directory into the database.
+  generate-translation-files [options] <input> <dir>    Generates API files from the given input translation.
+  generate-translations-files [options] <input> <dir>   Generates API files from the given input translations.
+  upload-api-files [options] <dest>                     Uploads API files to the specified destination. For S3, use the format s3://bucket-name/path/to/folder.
+  fetch-translations [options] <dir> [translations...]  Fetches the specified translations from fetch.bible and places them in the given directory.
+  fetch-audio [options] <dir> [translations...]         Fetches the specified audio translations and places them in the given directory.
+                                                        Translations should be in the format "translationId/audioId". e.g. "BSB/gilbert"
+  fetch-bible-metadata <dir>                            Fetches the Theographic bible metadata and places it in the given directory.
+  help [command]                                        display help for command
+```
+
+The `@helloao/cli` package can also be used as a library.
+
+The library exports a variety of actions, utilities, and supporting classes designed to assist with generating and managing a Free Use Bible API.
+
+There are 6 main exports:
+
+- `actions` - This export contains function versions of the CLI commands. They make it easy to call a CLI command from a script.
+- `db` - This export contains functions that make working with a database easier. It supports operations like importing translations into a database, inserting chapters, verses, etc. and getting an updated database instance from a path.
+- `downloads` - This export contains functions that make downloading files easier.
+- `files` - This export contains functions that make working with files easier. It has functions to load files from a translation, discover translation metadata from the filesystem, and classes that support uploading API files to the local file system or to a zip archive.
+- `uploads` - This export contains functions that make it easy to upload an API to a destination like S3, the local filesystem, or a zip archive.
+- `s3` - This export contains a class that can upload files to S3.
+
+
+Here are some common operations that you might want to perform:
+
+#### Get a SQL Database
+
+```typescript
+import { db } from '@helloao/cli';
+
+const pathToDb = './bible-database.db';
+const database = await db.getDb(pathToDb);
+
+// do work on the database
+
+// Close it when you are done.
+database.close();
+```
+
+#### Import a translation into a database from a directory
+
+```typescript
+import { db } from '@helloao/cli';
+
+const pathToDb = './bible-database.db';
+const database = await db.getDb(pathToDb);
+
+// Get a DOMParser for parsing USX.
+// On Node.js, you may have to import jsdom or linkedom.
+const parser = new DOMParser();
+
+const pathToTranslation = './path/to/translation';
+
+// Whether to overwrite files that already exist in the database.
+// The system will automatically determine the hashes of the input files and overwrite changed files if needed, so this is only needed
+// when you know that they need to be overwritten.
+const overwrite = false;
+await db.importTranslations(database, pathToTranslation, parser, overwrite);
+```
+
+#### Generate an API from a translation
+
+```typescript
+import { files, uploads } from '@helloao/cli';
+import { generation } from '@helloao/tools';
+import { toAsyncIterable } from '@helloao/tools/parser/iterators';
+
+const translationPath = './path/to/translation';
+const translationFiles = await files.loadTranslationFiles(translationPath);
+
+// Used to parse XML
+const domParser = new DOMParser();
+
+// Generate a dataset from the files
+// Datasets organize all the files and their content
+// by translation, book, chapter, and verse
+const dataset = generation.dataset.generateDataset(files, parser);
+
+// You can optionally specifiy a prefix that should be added to all API
+// links
+const pathPrefix = '';
+
+// Generate an API representation from the files
+// This adds links between chapters and additional metadata.
+const api = generation.api.generateApiForDataset(dataset, {
+  pathPrefix
+});
+
+// Generate output files from the API representation.
+// This will give us a list of files and file paths that represent
+// the entire API.
+const outputFiles = generation.api.generateFilesForApi(api);
+
+// Optionally upload files by using:
+// const dest = 's3://my-bucket';
+// await uploads.serializeAndUploadDatasets(dest, toAsyncIterable(outputFiles));
 ```

package/actions.js

@@ -57,46 +57,46 @@ async function initDb(dbPath, options) {
             if (options.language) {
                 console.log('Copying only the following languages:', options.language);
                 const languages = `(${options.language.map((l) => `'${l}'`).join(', ')})`;
-                db.exec(`
-                    ATTACH DATABASE "${sourcePath}" AS source;
-
-                    CREATE TABLE "_prisma_migrations" AS SELECT * FROM source._prisma_migrations;
-                    
-                    CREATE TABLE "Translation" AS SELECT * FROM source.Translation
-                    WHERE language IN ${languages};
-
-                    CREATE TABLE "Book" AS SELECT * FROM source.Book
-                    INNER JOIN source.Translation ON source.Translation.id = source.Book.translationId
-                    WHERE source.Translation.language IN ${languages};
-
-                    CREATE TABLE "Chapter" AS SELECT * FROM source.Chapter
-                    INNER JOIN source.Translation ON source.Translation.id = source.Chapter.translationId
-                    WHERE source.Translation.language IN ${languages};
-
-                    CREATE TABLE "ChapterVerse" AS SELECT * FROM source.ChapterVerse
-                    INNER JOIN source.Translation ON source.Translation.id = source.ChapterVerse.translationId
-                    WHERE source.Translation.language IN ${languages};
-
-                    CREATE TABLE "ChapterFootnote" AS SELECT * FROM source.ChapterFootnote
-                    INNER JOIN source.Translation ON source.Translation.id = source.ChapterFootnote.translationId
-                    WHERE source.Translation.language IN ${languages};
-
-                    CREATE TABLE "ChapterAudioUrl" AS SELECT * FROM source.ChapterAudioUrl
-                    INNER JOIN source.Translation ON source.Translation.id = source.ChapterAudioUrl.translationId
-                    WHERE source.Translation.language IN ${languages};
+                db.exec(`
+                    ATTACH DATABASE "${sourcePath}" AS source;
+
+                    CREATE TABLE "_prisma_migrations" AS SELECT * FROM source._prisma_migrations;
+                    
+                    CREATE TABLE "Translation" AS SELECT * FROM source.Translation
+                    WHERE language IN ${languages};
+
+                    CREATE TABLE "Book" AS SELECT * FROM source.Book
+                    INNER JOIN source.Translation ON source.Translation.id = source.Book.translationId
+                    WHERE source.Translation.language IN ${languages};
+
+                    CREATE TABLE "Chapter" AS SELECT * FROM source.Chapter
+                    INNER JOIN source.Translation ON source.Translation.id = source.Chapter.translationId
+                    WHERE source.Translation.language IN ${languages};
+
+                    CREATE TABLE "ChapterVerse" AS SELECT * FROM source.ChapterVerse
+                    INNER JOIN source.Translation ON source.Translation.id = source.ChapterVerse.translationId
+                    WHERE source.Translation.language IN ${languages};
+
+                    CREATE TABLE "ChapterFootnote" AS SELECT * FROM source.ChapterFootnote
+                    INNER JOIN source.Translation ON source.Translation.id = source.ChapterFootnote.translationId
+                    WHERE source.Translation.language IN ${languages};
+
+                    CREATE TABLE "ChapterAudioUrl" AS SELECT * FROM source.ChapterAudioUrl
+                    INNER JOIN source.Translation ON source.Translation.id = source.ChapterAudioUrl.translationId
+                    WHERE source.Translation.language IN ${languages};
                 `);
             }
             else {
-                db.exec(`
-                    ATTACH DATABASE "${sourcePath}" AS source;
-
-                    CREATE TABLE "_prisma_migrations" AS SELECT * FROM source._prisma_migrations;
-                    CREATE TABLE "Translation" AS SELECT * FROM source.Translation;
-                    CREATE TABLE "Book" AS SELECT * FROM source.Book;
-                    CREATE TABLE "Chapter" AS SELECT * FROM source.Chapter;
-                    CREATE TABLE "ChapterVerse" AS SELECT * FROM source.ChapterVerse;
-                    CREATE TABLE "ChapterFootnote" AS SELECT * FROM source.ChapterFootnote;
-                    CREATE TABLE "ChapterAudioUrl" AS SELECT * FROM source.ChapterAudioUrl;
+                db.exec(`
+                    ATTACH DATABASE "${sourcePath}" AS source;
+
+                    CREATE TABLE "_prisma_migrations" AS SELECT * FROM source._prisma_migrations;
+                    CREATE TABLE "Translation" AS SELECT * FROM source.Translation;
+                    CREATE TABLE "Book" AS SELECT * FROM source.Book;
+                    CREATE TABLE "Chapter" AS SELECT * FROM source.Chapter;
+                    CREATE TABLE "ChapterVerse" AS SELECT * FROM source.ChapterVerse;
+                    CREATE TABLE "ChapterFootnote" AS SELECT * FROM source.ChapterFootnote;
+                    CREATE TABLE "ChapterAudioUrl" AS SELECT * FROM source.ChapterAudioUrl;
                 `);
             }
             console.log('Done.');

package/cli.js

@@ -14,6 +14,7 @@ const actions_1 = require("./actions");
 const files_1 = require("./files");
 const dataset_1 = require("@helloao/tools/generation/dataset");
 const iterators_1 = require("@helloao/tools/parser/iterators");
+const db_1 = require("./db");
 async function start() {
     const parser = new linkedom_1.DOMParser();
     globalThis.DOMParser = linkedom_1.DOMParser;
@@ -60,7 +61,7 @@ async function start() {
         globalThis.Node = linkedom_1.Node;
         const files = await (0, files_1.loadTranslationFiles)(path_1.default.resolve(input));
         const dataset = (0, dataset_1.generateDataset)(files, parser);
-        await (0, uploads_1.uploadApiFiles)(path_1.default.resolve(dest), options, (0, iterators_1.toAsyncIterable)([dataset]));
+        await (0, uploads_1.serializeAndUploadDatasets)(path_1.default.resolve(dest), (0, iterators_1.toAsyncIterable)([dataset]), options);
     });
     program.command('generate-translations-files <input> <dir>')
         .description('Generates API files from the given input translations.')
@@ -83,7 +84,7 @@ async function start() {
         for (let b of (0, iterators_1.batch)(dirs, batchSize)) {
             const files = await (0, files_1.loadTranslationsFiles)(b);
             const dataset = (0, dataset_1.generateDataset)(files, parser);
-            await (0, uploads_1.uploadApiFiles)(dest, options, (0, iterators_1.toAsyncIterable)([dataset]));
+            await (0, uploads_1.serializeAndUploadDatasets)(dest, (0, iterators_1.toAsyncIterable)([dataset]), options);
         }
     });
     program.command('upload-api-files')
@@ -99,7 +100,13 @@ async function start() {
         .option('--profile <profile>', 'The AWS profile to use for uploading to S3.')
         .option('--pretty', 'Whether to generate pretty-printed JSON files.')
         .action(async (dest, options) => {
-        await (0, uploads_1.uploadApiFilesFromDatabase)(dest, options);
+        const db = (0, db_1.getPrismaDbFromDir)(process.cwd());
+        try {
+            await (0, uploads_1.uploadApiFilesFromDatabase)(db, dest, options);
+        }
+        finally {
+            db.$disconnect();
+        }
     });
     program.command('fetch-translations <dir> [translations...]')
         .description('Fetches the specified translations from fetch.bible and places them in the given directory.')

package/db.d.ts

@@ -2,9 +2,9 @@ import { PrismaClient } from "./prisma-gen";
 import { Database } from 'better-sqlite3';
 import { DatasetOutput, DatasetTranslation, DatasetTranslationBook } from "@helloao/tools/generation/dataset";
 import { InputFile, TranslationBookChapter } from "@helloao/tools/generation";
-import { GenerateApiOptions } from "@helloao/tools/generation/api";
+import { SerializeApiOptions, SerializedFile } from "./files";
 import { DOMParser } from "linkedom";
-import { Readable } from "stream";
+import { GenerateApiOptions } from "@helloao/tools/generation/api";
 /**
  * Imports the translations from the given directories into the database.
  * @param db The database to import the translations into.
@@ -50,14 +50,6 @@ export declare function getPrismaDbFromDir(dir: string): PrismaClient<{
 }, never, import("prisma-gen/runtime/library").DefaultArgs>;
 export declare function getDbFromDir(dir: string): Promise<Database>;
 export declare function getDb(dbPath: string): Promise<Database>;
-export interface SerializedFile {
-    path: string;
-    content: string | Readable;
-    /**
-     * Gets the base64-encoded SHA256 hash of the content of the file.
-     */
-    sha256?(): string;
-}
 /**
  * Loads the datasets from the database in a series of batches.
  * @param db The database.
@@ -65,46 +57,23 @@ export interface SerializedFile {
  * @param translationsToLoad The list of translations to load. If not provided, all translations will be loaded.
  */
 export declare function loadDatasets(db: PrismaClient, translationsPerBatch?: number, translationsToLoad?: string[]): AsyncGenerator<DatasetOutput>;
-export interface SerializeApiOptions extends GenerateApiOptions {
-    /**
-     * Whether the output should be pretty-printed.
-     */
-    pretty?: boolean;
-}
+export type SerializeDatasetOptions = SerializeApiOptions & GenerateApiOptions;
 /**
- * Generates and serializes the API files for the dataset that is stored in the database.
+ * Generates and serializes the API files for the datasets that are stored in the database.
  * Yields each batch of serialized files.
  * @param db The database that the dataset should be loaded from.
- * @param options The options to use for generating the API.
+ * @param options The options to use for serializing the files.
+ * @param apiOptions The options to use for generating the API files.
  * @param translationsPerBatch The number of translations that should be loaded and written per batch.
  * @param translations The list of translations that should be loaded. If not provided, all translations will be loaded.
  */
-export declare function serializeFilesForDataset(db: PrismaClient, options: SerializeApiOptions, translationsPerBatch?: number, translations?: string[]): AsyncGenerator<SerializedFile[]>;
+export declare function serializeFilesFromDatabase(db: PrismaClient, options?: SerializeDatasetOptions, translationsPerBatch?: number, translations?: string[]): AsyncGenerator<SerializedFile[]>;
 /**
- * Serializes the API files for the given datasets.
- * @param datasets The dataasets to serialize.
- * @param options The options to use for serializing the files.
- */
-export declare function serializeFiles(datasets: AsyncIterable<DatasetOutput>, options: SerializeApiOptions): AsyncGenerator<SerializedFile[]>;
-/**
- * Defines an interface that contains information about a serialized file.
+ * Generates and serializes the API files for the given datasets.
+ * Yields each batch of serialized files.
+ *
+ * @param datasets The datasets to serialize.
+ * @param options The options to use for generating and serializing the files.
  */
-export interface Uploader {
-    /**
-     * Gets the ideal batch size for the uploader.
-     * Null if the uploader does not need batching.
-     */
-    idealBatchSize: number | null;
-    /**
-     * Uploads the given file.
-     * @param file The file to upload.
-     * @param overwrite Whether the file should be overwritten if it already exists.
-     * @returns True if the file was uploaded. False if the file was skipped due to already existing.
-     */
-    upload(file: SerializedFile, overwrite: boolean): Promise<boolean>;
-    /**
-     * Disposes resources that the uploader uses.
-     */
-    dispose?(): Promise<void>;
-}
+export declare function serializeDatasets(datasets: AsyncIterable<DatasetOutput>, options?: SerializeDatasetOptions): AsyncGenerator<SerializedFile[]>;
 //# sourceMappingURL=db.d.ts.map