@openstax/ts-utils 1.33.0 → 1.34.0

package/README.md

@@ -1,118 +1,90 @@
-# @openstax/ts-utils
+<!-- spell-checker: ignore creds lastsync -->
+## Project Template
 
-A collection of utilities typescript applications. browser and server support.
+SPA React frontend with a serverless lambda backend, supported by aws data stores. fullstack typescript.
 
-## Libraries
-overviews of the libraries provided and their purposes, for more specific information check out the comments in the source code.
+features:
+- monorepo for frontend, lambda functions, deployment scripts
+- strong fullstack typescript support even across api calls
+- one command development server script
+- one command deployment script
+- production ready with error handling, logging, alerting
+- established paradigms for common issues like routing, middleware, pagination, configs, dependency injection, plug-able service drivers per environment
+- service providers for versioned document storage, file storage, document search, openstax accounts auth
+- 100% test coverage using jest in both frontend and lambda modules
+- CI checks for unit tests, spell checking, code linting, code coverage
 
-### [types.ts](./types.ts)
-utility types, no javascript functionality in here. A dumping ground for commonly used utility types or sorcery copied of stack overflow.
+## Design Goals
 
-### [guards.ts](./guards.ts)
-generic [type guards](https://www.typescriptlang.org/docs/handbook/advanced-types.html#user-defined-type-guards). most of this isn't super interesting its just annoying to have to re-define these in every project.
+### Easy and Full Featured
 
-### [assertions.ts](./assertions.ts)
-like a guard but throws if the condition fails. you can use an assertion if you're extra sure that something is true but typescript isn't so sure, its possible that a bug might exist but normally your expected case should always be the case. these assertions allow you to have useful errors/messages for the exceptional bug, without generating conditional logic in your application code. this is almost entirely driven by not wanting to create code branches that will then need unit tests.
+Setting up a new project or microservice using raw libraries or tools like SAM is pretty easy, but leaves a lot of problems unsolved. developers end up having to solve a lot of problems over and over again, or cutting corners. There are, surprisingly, not a lot of software frameworks designed to work in a serverless environment, so every time you need to figure out how do error logging, or routing. maybe there is a lot of copy/paste there once you've done it a few times, but not everyone has done it a few times, and its still a project if you have. Sometimes you think you can quickly bang out a service that does xyz, and it doesn't need any frills, but then it turns out it sure would have been nice to have a data store, or an admin interface, and it doesn't send any alerts when it has errors, and that takes the project into a whole new problem space.
 
-### [index.ts](./index.ts)
-basic array/object access/manipulation utilities
+This project aims to provide a baseline level of production-readiness, in addition to easily accessible prebuilt functionality for common problems.
 
-### [errors.ts](./errors.ts)
-a set of common application errors for the other libraries to use
+### Cohesive
 
-### [fetch.ts](./fetch.ts)
-utilities for working with `fetch` and fetching data. mostly there are constructors and guards for working with response objects that capture both a request status and associated data. the actual fetch api only throws if there is a connection errors, so most data access utilities need a way to communicate application layer failures back to the client in a discernible way. you can _almost_ do this with regular response objects, but then you're forcing all of the response processing down into user-land, and this system provides an additional loading state for UIs.
+A big problem in the javascript world is lack of cohesive systems. for any given problem there are a thousand npm modules that could help you, but given a set of problems, finding solutions that can work well together is a challenge.
 
-### [config.ts](./config.ts)
-defines ConfigProvider as an arbitrarily nested object where leaves must be ConfigValueProviders. ConfigValueProviders are strings, promises of strings, functions without arguments that return strings, or promises of strings. the library has a utility to resolve a ConfigValueProvider to a string, and a few re-usable providers.
+This project aims to provide a full stack developer experience using similar tools and patterns across the project, and typescript support as strong as we can make it.
 
-### [middleware.ts](./middleware.ts)
-middleware are chains of functions, with shared access to a parent application or services, that modify or decorate a result. middleware chains are used to pre-process requests before they reach routing, post-process results after routing, and compose service providers for the routes to use.
 
-### [routing.ts](./routing.ts)
-a flexible routing library with utils for defining routes, matching requests to routes, and rendering urls for routes. strong typing for route params, request bodies, response bodies. reverse routing asserts not only that the correct route and params are given, but that the route is composed in the application. built to be implementation agnostic and support both browser SPA routing and server side API routing.
+## Getting Started with the Project Template
 
-### [pagination.ts](./pagination.ts)
-utilities for interpreting pagination requests and generating pagination responses for both 'load more' and 'page number' style pagination. included middleware for working with the routing library.
+### Set up your repo
+- clone this repo
+- delete packages/utils (it'll pull it from npm)
+- run `git rev-parse head > .lastsync` so that future updates can be patched in
+- reset the git history (delete .git and run `git init; git commit -am "initial commit"` again)
+- follow the [development](#development) instructions to start your local environment
 
-## Services
-Services differ from the rest of the libraries in that they have a single discrete interface and support multiple implementations of that interface.
+### Get into the Code
+- start checking out the api code from the [routes file](./packages/lambda/src/functions/serviceApi/versions/v0/example/routes.ts). this example provides basic entity data storage, there are lots of comments in there.
+- start checking out the frontend code from the [homepage](./packages/frontend/src/example/screens/Home.tsx), there are a few pages that integrate with the example api routes.
+- more info about the libraries and utils used in the [utils module](./packages/utils/README.md)
 
-### [apiGateway](./services/apiGateway/index.ts)
-dynamically generates an api client from route configuration. if you're using the routing library in the api, this allows you to import your route types directly into your client and have very strong typing on your api calls for free. includes error handling, filtering responses by status code, and narrowing the expected data type by given status code.
+### your first deployment
+- change the `APPLICATION` in [the deployment constants file](./deploy/constants.env). the first time you run the deployment it'll
+walk you through some one-time setup stuff. it'll output the deployed url when it finishes.
+- upload some secrets to the parameter store with `AccountsBase=SOMETHING CookieName=SOMETHING EncryptionPrivateKey=SOMETHING SignaturePublicKey=SOMETHING yarn ts-utils upload-params <environment-name>`.
+- (optional) set PagerDuty params with `yarn -s ts-utils upload-pager-duty-endpoints`
 
-### [authProvider](./services/authProvider)
-integrates with openstax.org/accounts, includes several implementations for both browser and server side integrations.
+## Development
 
-### [lrsGateway](./services/lrsGateway/index.ts)
-handles authentication and payload formatting for interacting with the LRS
+install [nvm](https://github.com/creationix/nvm#installation)
 
-### [versionedDocumentStore](./services/versionedDocumentStore)
-a key/value document store with integrated audit logging. all writes are done as new versions of documents and version history can be fetched. includes implementations for dynamodb and local file system.
-
-### [searchProvider](./services/searchProvider/index.ts)
-allows filtering and open text search for documents. has strong typing for filterable values based on the document type definition. interface is based on elasticsearch options, but currently we only have an in memory filtering implementation. works out of the box with versionedDocumentStore.
-
-## Notes
-
-### Service drivers and configuration
-Service providers are defined in a little bit of a weird nested way, this is supposed to help separate driver specific and instance specific configurations, and let you swap drivers in different application entry points without too many application dependencies.
-
-an entry point is the file you point to when you build or run the application. the idea is that you make a different entry point files for different environments or contexts. so you have one for launching local development and one for launching an all-in-one stack on a testing server and another one for launching a production environment. in order to reduce environment specific issues you reduce your entry point to be as small as possible and just assemble the required bits for that environment to work. in the recommended setup, one of the main things the entry point does is choose which service provider drivers to use. there are two main benefits of using entry points, its more declarative than using downstream conditional logic with some kind of `driver: fs` flag in the config, and it allows bundled javascript to only include the dependencies relevant for the drivers that are being used in that build.
-
-the idea is that you may have some entities you're putting in a VersionedDocumentStore and maybe some others in a RelationalEntityStore but you're for sure not putting some versioned entities in dynamo and some in a local json file. so in your development environment entry point you provide your config for `versionedDocumentStore: fileSystemVersionedDocumentStore` and in the deployed environment entry point you put `versionedDocumentStore: dynamoVersionedDocumentStore`.
+```
 
-there are two levels the drivers are configured at, one is the driver level and the other is the instance level. for the VersionedDocumentStore the fileSystem driver takes a path to the directory to store the files in. the instance level may vary based on the usage of that driver, for example if you're storing 4 different entities in VersionedDocumentStores, you need to provide different table names to the driver to configure each store to create separate instances of that store. it just happens that for VersionedDocumentStore the fileSystem and dynamo implementations use the same instance config, this isn't always the case. once you get into your application logic you won't know which entry point was used to bootstrap your application, so you'll just know the driver is _one of_ the possible versions, typescript will recognize this and throw an error unless you provide a config that would work for any of the possible drivers. in order to make this more legible (and separate key collisions), most of the services can be set up in the entry point with a config prefix, so when you get into application-land it looks more like you're providing two discrete configs.
+# use the right version of node
+nvm install
 
-for example:
+# install yarn, skip if you have it already
+npm install -g yarn
 
-```typescript
-// entry/anEntryPoint.ts
-export const lambdaServices = {
-  // in this entry we provide the dynamo driver
-  versionedDocumentStore: dynamoVersionedDocumentStore({configSpace: 'dynamodb'}),
-};
+# install dependencies
+yarn install
 
-// entry/anotherEntryPoint.ts
-export const localServices = {
-  // in this entry we provide the fileSystem driver, it has a different config prefix
-  versionedDocumentStore: fileSystemVersionedDocumentStore({dataDir, configSpace: 'fileSystem'}),
-};
+# start
+#  run this from the project root, it builds the utils and lambda packages,
+#  sets up build watchers for them, and then starts the react app.
+#  changes in any package are built and visible in the app immediately.
+yarn start
+```
 
-// core/types.ts
+### to fix scary untrusted cert warning in chrome
 
-// there is other wiring up that happens with the app services on the request responder, if
-// you made a new entrypoint and tried to use a different format of services without registering it
-// here it would yell at you.
-export type AppServices = LambdaServices | LocalServices;
+*note:* you must do this for login to work
 
-// myCoolDocumentStoreMiddleware.ts
+#### trust the certificate
+- run `./packages/frontend/script/trust-localhost.bash` after starting the server
 
-// this config satisfies both config drivers with separate sections for each one.
-// the envConfig is defined here, but it wouldn't try to actually resolve that value
-// unless it was being used.
-const config = {
-  fileSystem: {
-    tableName: 'coolDocument'
-  },
-  dynamodb: {
-    tableName: envConfig('TestDocumentTableName', 'runtime'),
-  }
-};
+## Deployment
 
-export const myCoolDocumentStoreMiddleware = <M extends {}>(app: AppServices) => {
-  // providing the document interface and the config prepares the driver to work for this entity in particular.
-  // the extra function call in this line is to work around some typescript quirkiness with generic type inference.
-  // this line is only run once when the app is wired up, doing this call here allows the driver to do internal memory caching
-  // if it wants to.
-  const makeCoolDocumentStore = app.versionedDocumentStore<CoolDocument>()(config);
+dependencies: aws-sdk, jq
 
-  return (middleware: M) => ({
-    ...middleware,
-    // this part is run for every api request processed. in this case there it could have been combined with the call above,
-    // but most of the time you'll want to pull in the authProvider and hook up automatic audit logging, and that will be different on 
-    // each request. look at the VersionedDocumentStore documentation for details.
-    coolDocumentStore: makeCoolDocumentStore('id'),
-  })
-};
+```
+# if you don't have https://github.com/openstax/ox-bin set up you can call this directly from
+# your local https://github.com/openstax/aws-access/blob/main/scripts/set_aws_creds
+. ox set_aws_creds -i assume-role -r sandbox:full-admin
+yarn -s ts-utils deploy my-cool-environment
 ```

package/app.json

@@ -0,0 +1,23 @@
+{
+  "name": "project-template",
+  "scripts": {
+  },
+  "env": {
+    "NODE_ENV": "development",
+    "REACT_APP_API_BASE_URL": "/"
+  },
+  "formation": {
+    "web": {
+      "size": "hobby"
+    }
+  },
+  "addons": [
+
+  ],
+  "buildpacks": [
+    {
+      "url": "heroku/nodejs"
+    }
+  ],
+  "stack": "heroku-22"
+}

package/cspell.json

@@ -0,0 +1,32 @@
+{
+  "version": "0.2",
+  "$schema": "https://raw.githubusercontent.com/streetsidesoftware/cspell/master/cspell.schema.json",
+  "language": "en",
+  "words": [
+    "buildpacks",
+    "cloudformation",
+    "cookieyes",
+    "herokuapp",
+    "Jwks",
+    "jwks",
+    "ltijs",
+    "openstax",
+    "signup",
+    "stax",
+    "subcontent",
+    "subrequest",
+    "subrequests",
+    "systemsmanager",
+    "unversioned",
+    "Xapi"
+  ],
+  "flagWords": [],
+  "ignorePaths": [
+    "*.spec.ts.snap",
+    "coverage",
+    "package.json",
+    "yarn.lock",
+    "node_modules/**",
+    "*.svg"
+  ]
+}

package/deploy/constants.env

@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+# spell-checker: ignore pipefail
+set -euo pipefail; if [ -n "${DEBUG-}" ]; then set -x; fi
+
+# These constants also get applied as tags to CloudFormation stacks and SSM parameters
+# APPLICATION should probably be lower-case, dash-separated, since it's used to create stack names
+export APPLICATION=project-template
+export OWNER=tom
+export PROJECT=DISCO
+
+# List all parameter names to upload here
+PARAMS=(
+  AccountsBase
+  CookieName # for decrypt authorization
+  EncryptionPrivateKey # for decrypt authorization
+  SignaturePublicKey # for decrypt authorization
+)
+export PARAMS
+
+export AWS_DEFAULT_REGION=us-east-1
+export AWS_ALT_REGION=us-east-2

package/deploy/deploy.bash

@@ -0,0 +1,157 @@
+#!/usr/bin/env bash
+# spell-checker: ignore pipefail
+set -euo pipefail; if [ -n "${DEBUG-}" ]; then set -x; fi
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+if [ -z "${ENVIRONMENT:-}" ]; then echo "run this command with 'yarn -s ts-utils deploy' instead of executing it directly" > /dev/stderr; exit 1; fi
+
+timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
+gitVersion=$(git rev-parse HEAD)
+stackName="$ENVIRONMENT-$APPLICATION"
+
+# =======
+# shared setup is one per aws account for this project, currently only used for
+# lambda code zip archives and SNS topics for PagerDuty notifications
+# =======
+if ! yarn -s ts-utils stack-exists "subdomain-$APPLICATION-dns"; then
+  echo 'DNS stack not found. create one by following the instructions here: https://github.com/openstax/subdomains' > /dev/stderr
+  exit 1
+fi
+if ! yarn -s ts-utils stack-exists "subdomain-$APPLICATION-cert"; then
+  echo 'SSL cert stack not found. create one by following the instructions here: https://github.com/openstax/subdomains' > /dev/stderr
+  exit 1
+fi
+
+pagerDutyAnytimeEndpoint=$(aws ssm get-parameter --name "/external/pager_duty/$APPLICATION/anytime_endpoint" \
+   --with-decryption --query 'Parameter.Value' --output text 2>/dev/null || true)
+pagerDutyWorkdayEndpoint=$(aws ssm get-parameter --name "/external/pager_duty/$APPLICATION/workday_endpoint" \
+   --with-decryption --query 'Parameter.Value' --output text 2>/dev/null || true)
+
+if [ -z "$pagerDutyAnytimeEndpoint" ] || [ -z "$pagerDutyWorkdayEndpoint" ]; then
+  echo "Warning: PagerDuty endpoint(s) not set. You can set them by running: 'yarn -s ts-utils upload-pager-duty-endpoints'" > /dev/stderr
+fi
+
+yarn --check-files
+
+if [ -n "$(git status --porcelain=v1 2>/dev/null)" ]; then
+  echo "please stash, commit, gitignore, or reset your changes before deploying" > /dev/stderr
+  exit 1
+fi
+
+bucketPrefix=
+if [ "$AWS_ACCOUNT" != "openstax" ]; then
+  bucketPrefix="$AWS_ACCOUNT-"
+fi
+
+aws cloudformation deploy \
+  --region "$AWS_DEFAULT_REGION" \
+  --no-fail-on-empty-changeset \
+  --template-file "$SCRIPT_DIR/shared.cfn.yml" \
+  --stack-name "$APPLICATION-shared" \
+  --parameter-overrides "BucketPrefix=$bucketPrefix" \
+    "PagerDutyAnytimeEndpoint=$pagerDutyAnytimeEndpoint" \
+    "PagerDutyWorkdayEndpoint=$pagerDutyWorkdayEndpoint" \
+  --tags "Project=$PROJECT" "Application=$APPLICATION" 'Environment=shared' "Owner=$OWNER"
+
+codeBucket=$(yarn -s ts-utils get-stack-param "$APPLICATION-shared" BucketName)
+
+# =======
+# build utils code if the package exists in this repo
+# =======
+if [ -d "$SCRIPT_DIR"/../packages/utils ]; then
+  cd "$SCRIPT_DIR"/../packages/utils
+  yarn build:clean
+fi
+
+# =======
+# build api code and upload to code bucket
+# =======
+cd "$SCRIPT_DIR"/../packages/lambda
+
+export CODE_VERSION="$gitVersion"
+export APPLICATION;
+
+yarn archive:clean
+
+apiCodeKey="api-$timestamp.zip"
+aws s3 cp "dist/serviceApi.zip" "s3://$codeBucket/$apiCodeKey"
+
+# =======
+# build frontend and upload to static site bucket
+# =======
+cd "$SCRIPT_DIR"/../packages/frontend
+
+export REACT_APP_CODE_VERSION="$CODE_VERSION"
+export REACT_APP_NAME="$APPLICATION"
+export PUBLIC_URL="/build"
+
+# =======
+# try to read domainName from the stack
+# =======
+if yarn -s ts-utils stack-exists "$stackName"; then
+  previouslyDeployed=1
+  bucketName=$(yarn -s ts-utils get-stack-param "$stackName" StaticBucketName)
+  domainName=$(yarn -s ts-utils get-stack-param "$stackName" DistributionDomainName)
+  export REACT_APP_API_BASE_URL="https://${domainName}"
+  yarn build:clean
+  aws s3 sync build "s3://${bucketName}${PUBLIC_URL}" --region "$AWS_DEFAULT_REGION"
+else
+  previouslyDeployed=0
+fi
+
+cd "$SCRIPT_DIR"
+
+# Get values from SSM parameters
+cookieName=$(yarn -s ts-utils get-env-param "$ENVIRONMENT" "CookieName")
+accountsBase=$(yarn -s ts-utils get-env-param "$ENVIRONMENT" "AccountsBase")
+signaturePublicKey=$(yarn -s ts-utils get-env-param "$ENVIRONMENT" "SignaturePublicKey")
+
+cd "$SCRIPT_DIR"/../packages/lambda
+
+# =======
+# main deployment includes alt region for fail over
+# =======
+aws cloudformation deploy \
+  --region "$AWS_ALT_REGION" \
+  --no-fail-on-empty-changeset \
+  --template-file "$SCRIPT_DIR/deployment-alt-region.cfn.yml" \
+  --stack-name "$stackName" \
+  --parameter-overrides "BucketPrefix=$bucketPrefix" \
+  --tags "Project=$PROJECT" "Application=$APPLICATION" "Environment=$ENVIRONMENT" "Owner=$OWNER"
+
+# cloudformation cannot reference exports across regions, so these are applied like this
+replicaBucketWebsiteURL=$(AWS_DEFAULT_REGION="$AWS_ALT_REGION" yarn -s ts-utils get-stack-param "$stackName" ReplicaBucketWebsiteURL)
+
+aws cloudformation deploy \
+  --region "$AWS_DEFAULT_REGION" \
+  --template-file "$SCRIPT_DIR/deployment.cfn.yml" \
+  --stack-name "$stackName" \
+  --capabilities CAPABILITY_NAMED_IAM \
+  --parameter-overrides "BucketPrefix=$bucketPrefix" "CodeBucket=$codeBucket" \
+    "EnvName=$ENVIRONMENT" "Application=$APPLICATION" "ReplicaBucketWebsiteURL=$replicaBucketWebsiteURL" \
+    "ApiCodeKey=$apiCodeKey" \
+    "AccountsBase=$accountsBase" "CookieName=$cookieName" "SignaturePublicKey=$signaturePublicKey" \
+  --tags "Project=$PROJECT" "Application=$APPLICATION" "Environment=$ENVIRONMENT" "Owner=$OWNER"
+
+bucketName=$(yarn -s ts-utils get-stack-param "$stackName" StaticBucketName)
+domainName=$(yarn -s ts-utils get-stack-param "$stackName" DistributionDomainName)
+distributionId=$(yarn -s ts-utils get-stack-param "$stackName" DistributionId)
+
+# =======
+# build frontend and upload to static site bucket
+# =======
+export REACT_APP_API_BASE_URL="https://${domainName}"
+
+cd "$SCRIPT_DIR"/../packages/frontend
+if [ $previouslyDeployed -eq 0 ]; then
+  yarn build:clean
+fi
+aws s3 sync build "s3://${bucketName}${PUBLIC_URL}" --delete --region "$AWS_DEFAULT_REGION"
+
+aws cloudfront create-invalidation --distribution-id "$distributionId" --paths "/*" --output text --query "Invalidation.Status"
+
+# =======
+# done
+# =======
+echo "deployed: $domainName";

package/deploy/deployment-alt-region.cfn.yml

@@ -0,0 +1,70 @@
+Metadata:
+  cfn-lint:
+    config:
+      ignore_checks:
+        - W6001
+        - E3031 # we do this on purpose
+        - E3033 # we do this on purpose
+
+Parameters:
+  BucketPrefix:
+    Description: A prefix to prevent collisions between buckets in different accounts, e.g. sandbox-
+    Type: String
+
+Conditions:
+  # Lambda@Edge functions need to be created in us-east-1; in order for this to be redundant it must be somewhere else
+  WrongRegion: !Equals [!Ref 'AWS::Region', us-east-1]
+
+Resources:
+  # ==============
+  # Region validation
+  # ==============
+  YouAreInTheWrongRegion:
+    Type: "AWS::SSM::Parameter"
+    Condition: WrongRegion
+    Properties:
+      Name: '' # Leave name empty to force a fail
+      Type: String
+      Value: ''
+  # ==============
+  # static frontend S3 / Cloudfront
+  # ==============
+  ReplicaBucket:
+    Type: "AWS::S3::Bucket"
+    Properties:
+      BucketName: !Sub ${BucketPrefix}${AWS::StackName}-ui-replica-bucket
+      VersioningConfiguration:
+        Status: "Enabled"
+      LifecycleConfiguration:
+        Rules:
+          - Id: MustHaveVersioningButDoNotWantOldVersions
+            NoncurrentVersionExpirationInDays: 1
+            Status: "Enabled"
+      PublicAccessBlockConfiguration:
+        BlockPublicPolicy: false
+      WebsiteConfiguration:
+        IndexDocument: "does-not-exist.html"
+      OwnershipControls:
+        Rules:
+          - ObjectOwnership: BucketOwnerEnforced
+
+  ReplicaBucketPolicy:
+    Type: AWS::S3::BucketPolicy
+    Properties:
+      PolicyDocument:
+        Version: "2012-10-17"
+        Statement:
+          - Sid: PublicReadForGetBucketObjects
+            Effect: Allow
+            Principal: '*'
+            Action: s3:GetObject
+            Resource: !Sub ${ReplicaBucket.Arn}/*
+
+      Bucket: !Ref ReplicaBucket
+
+Outputs:
+  ReplicaBucketName:
+    Value: !Ref ReplicaBucket
+
+  ReplicaBucketWebsiteURL:
+    Value: !GetAtt ReplicaBucket.WebsiteURL