ts-proto 1.90.1 → 1.92.2

package/CHANGELOG.md

@@ -1,3 +1,36 @@
+## [1.92.2](https://github.com/stephenh/ts-proto/compare/v1.92.1...v1.92.2) (2021-12-08)
+
+
+### Bug Fixes
+
+* noImplicitReturns error in Value.unwrap ([#436](https://github.com/stephenh/ts-proto/issues/436)) ([2d7a5d0](https://github.com/stephenh/ts-proto/commit/2d7a5d04c72ace58fa3a6745c3857f6cc0468543)), closes [#432](https://github.com/stephenh/ts-proto/issues/432)
+
+## [1.92.1](https://github.com/stephenh/ts-proto/compare/v1.92.0...v1.92.1) (2021-12-02)
+
+
+### Bug Fixes
+
+* Respect stringEnums option in wrap function ([#420](https://github.com/stephenh/ts-proto/issues/420)) ([7adf90c](https://github.com/stephenh/ts-proto/commit/7adf90c23c46950bcf457b317764393ff4af2bf2))
+
+# [1.92.0](https://github.com/stephenh/ts-proto/compare/v1.91.0...v1.92.0) (2021-11-28)
+
+
+### Features
+
+* Use exact types for fromPartial ([#412](https://github.com/stephenh/ts-proto/issues/412)) ([808f8a7](https://github.com/stephenh/ts-proto/commit/808f8a7a77d56f65dd4a4643dd66158f106ab755)), closes [#156](https://github.com/stephenh/ts-proto/issues/156)
+
+# [1.91.0](https://github.com/stephenh/ts-proto/compare/v1.90.1...v1.91.0) (2021-11-27)
+
+
+### Bug Fixes
+
+* use Long.fromValue instead of Long.fromString for improved robustness regarding already parsed objects ([#405](https://github.com/stephenh/ts-proto/issues/405)) ([7bdc3ee](https://github.com/stephenh/ts-proto/commit/7bdc3eee05ed1318e18e27aa7d5bb2680060f8b6))
+
+
+### Features
+
+* Include dockerized protoc ([#404](https://github.com/stephenh/ts-proto/issues/404)) ([7564a78](https://github.com/stephenh/ts-proto/commit/7564a7887ccd0bb80cac19d313ee9bd8daae778d))
+
 ## [1.90.1](https://github.com/stephenh/ts-proto/compare/v1.90.0...v1.90.1) (2021-11-27)
 
 

package/README.markdown

@@ -9,25 +9,29 @@
 
 ## Table of contents
 
+- [ts-proto](#ts-proto)
+  - [Table of contents](#table-of-contents)
+- [Overview](#overview)
 - [QuickStart](#quickstart)
 - [Goals](#goals)
 - [Example Types](#example-types)
 - [Highlights](#highlights)
 - [Auto-Batching / N+1 Prevention](#auto-batching--n1-prevention)
 - [Usage](#usage)
-  - [Supported options](#supported-options)
-  - [Only Types](#only-types)
-  - [NestJS Support](NESTJS.markdown)
-- [Building](#building)
+    - [Supported options](#supported-options)
+    - [Only Types](#only-types)
+    - [NestJS Support](#nestjs-support)
+    - [Watch Mode](#watch-mode)
+- [Sponsors](#sponsors)
+- [Development](#development)
 - [Assumptions](#assumptions)
 - [Todo](#todo)
 - [OneOf Handling](#oneof-handling)
-- [Default Values and Unset Fields](#default-values-and-unset-fields)
+- [Default values and unset fields](#default-values-and-unset-fields)
 - [Well-Known Types](#well-known-types)
   - [Wrapper Types](#wrapper-types)
   - [JSON Types (Struct Types)](#json-types-struct-types)
-  - [Timestamps](#timestamp)
-- [Wrapper Types](#wrapper-types)
+  - [Timestamp](#timestamp)
 - [Number Types](#number-types)
 - [Current Status of Optional Values](#current-status-of-optional-values)
 
@@ -329,7 +333,7 @@ Generated code will be placed in the Gradle build directory.
 
   Note that `addGrpcMetadata`, `addNestjsRestParameter` and `returnObservable` will still be false.
 
-- With `--ts_proto_opt=useDate=false`, fields of type `google.protobuf.Timestamp` will not be mapped to type `Date` in the generated types. See [Timestamps](#timestamps) for more details.
+- With `--ts_proto_opt=useDate=false`, fields of type `google.protobuf.Timestamp` will not be mapped to type `Date` in the generated types. See [Timestamp](#timestamp) for more details.
 
 - With `--ts_proto_opt=outputSchema=true`, meta typings will be generated that can later be used in other code generators.
 
@@ -337,7 +341,7 @@ Generated code will be placed in the Gradle build directory.
 
 - With `--ts_proto_opt=outputServices=grpc-js`, ts-proto will output service definitions and server / client stubs in [grpc-js](https://github.com/grpc/grpc-node/tree/master/packages/grpc-js) format.
 
-- With `--ts_proto_opt=outputServices=generic-definitions`, ts-proto will output generic (framework-agnostic) service definitions.
+- With `--ts_proto_opt=outputServices=generic-definitions`, ts-proto will output generic (framework-agnostic) service definitions. These definitions contain descriptors for each method with links to request and response types, which allows to generate server and client stubs at runtime, and also generate strong types for them at compile time. An example of a library that uses this approach is [nice-grpc](https://github.com/deeplay-io/nice-grpc).
 
 - With `--ts_proto_opt=outputServices=false`, or `=none`, ts-proto will output NO service definitions.
 
@@ -354,6 +358,15 @@ If you're looking for `ts-proto` to generate only types for your Protobuf types
 
 We have a great way of working together with [nestjs](https://docs.nestjs.com/microservices/grpc). `ts-proto` generates `interfaces` and `decorators` for you controller, client. For more information see the [nestjs readme](NESTJS.markdown).
 
+### Watch Mode
+
+If you want to run `ts-proto` on every change of a proto file, you'll need to use a tool like [chokidar-cli](https://www.npmjs.com/package/chokidar-cli) and use it as a script in `package.json`:
+
+```json
+"proto:generate": "protoc --ts_proto_out=. ./<proto_path>/<proto_name>.proto --ts_proto_opt=esModuleInterop=true",
+"proto:watch": "chokidar \"**/*.proto\" -c \"npm run proto:generate\""
+```
+
 # Sponsors
 
 Kudos to our sponsors:
@@ -362,15 +375,69 @@ Kudos to our sponsors:
 
 If you need ts-proto customizations or priority support for your company, you can ping me at [via email](mailto:stephen.haberman@gmail.com).
 
-# Building
+# Development
+
+**Requirements**
+
+- [Docker](https://www.docker.com) or [protoc](https://github.com/protocolbuffers/protobuf/releases) v3.19.1
+- `yarn` — `npm install -g yarn`
+
+**Setup**
+
+The commands below assume you have **Docker** installed. To use a **local** copy of `protoc` without docker, use commands suffixed with `:local`
+
+- Check out the [repository]() for the latest code.
+- Run `yarn install` to install the dependencies.
+- Run `yarn build:test` or `yarn build:test:local` to generate the test files.
+  > _This runs the following commands:_
+  >  - `proto2bin` — Converts integration test `.proto` files to `.bin`.
+  >  - `bin2ts` — Runs `ts-proto` on the `.bin` files to generate  `.ts`  files.
+  >  - `proto2pbjs` — Generates a reference implementation using `pbjs` for testing compatibility.
+- Run `yarn test`
+
+**Workflow**
 
-After running `yarn install`, run `./integration/pbjs.sh` to create the integration test types. These pbjs-generated files are not currently checked in.
+- Modifying the plugin implementation:
+  - Run `yarn bin2ts` or `yarn bin2ts:local`.  
+    _Since the proto files were not changed, you only need to regenerate the typescript files._
+  - Run `yarn test` to verify the typescript files are compatible with the reference implementation, and pass other tests.
+- Updating or adding `.proto` files in the integration directory:
+  - Run `yarn build:test` to regenerate the integration test files.
+  - Run `yarn test` to retest.
 
-After this, the tests should pass.
+**Contributing**
 
-After making changes to `ts-proto`, you can run `cd integration` and `./codegen.sh` to re-generate the test case `*.ts` output files that are in each `integration/<test-case>/` directory.
+- Run `yarn build:test` and `yarn test` to make sure everything works.
+- Run `yarn prettier` to format the typescript files.
+- Commit the changes:
+  - Also include the generated `.bin` files for the tests where you added or modified `.proto` files.  
+    > These are checked into git so that the test suite can run without having to invoke the `protoc` build chain.
+  - Also include the generated `.ts` files.
+- Create a pull request
+
+**Dockerized Protoc**
+
+The repository includes a dockerized version of `protoc`, which is configured in [docker-compose.yml](docker-compose.yml).
+
+It can be useful in case you want to manually invoke the plugin with a known version of `protoc`.
+
+Usage:
+
+```bash
+# Include the protoc alias in your shell.
+. aliases.sh
+
+# Run protoc as usual. The ts-proto directory is available in /ts-proto.
+protoc --plugin=/ts-proto/protoc-gen-ts_proto --ts_proto_out=./output -I=./protos ./protoc/*.proto
+
+# Or use the ts-protoc alias which specifies the plugin path for you.
+ts-protoc --ts_proto_out=./output -I=./protos ./protoc/*.proto
+```
 
-The test suite's proto files (i.e. `simple.proto`, `batching.proto`, etc.) currently have serialized/`.bin` copies checked into git (i.e. `simple.bin`, `batching.bin`, etc.), so that the test suite can run without having to invoke the `protoc` build chain. I.e. if you change the `simple.proto`/etc. files, you'll need to run `./integration/update-bins.sh`, which does require having the `protoc` executable available.
+- All paths must be relative paths _within_ the current working directory of the host. `../` is not allowed
+- Within the docker container, the absolute path to the project root is `/ts-proto`
+- The container mounts the current working directory in `/host`, and sets it as its working directory.
+- Once `aliases.sh` is sourced, you can use the `protoc` command in any folder.
 
 # Assumptions
 

package/build/generate-type-registry.js

@@ -13,7 +13,7 @@ function generateTypeRegistry(ctx) {
     chunks.push(ts_poet_1.code `
     export const messageTypeRegistry = new Map<string, MessageType>();
   `);
-    chunks.push(ts_poet_1.code `${ctx.utils.DeepPartial.ifUsed}`);
+    chunks.push(ts_poet_1.code ` ${ctx.utils.Builtin.ifUsed} ${ctx.utils.DeepPartial.ifUsed}`);
     return ts_poet_1.joinCode(chunks, { on: '\n\n' });
 }
 exports.generateTypeRegistry = generateTypeRegistry;

package/build/main.js

@@ -289,11 +289,20 @@ function makeDeepPartial(options, longs) {
     const maybeExport = options.exportCommonSymbols ? 'export' : '';
     // Allow passing longs as numbers or strings, nad we'll convert them
     const maybeLong = options.forceLong === options_1.LongOption.LONG ? ts_poet_1.code ` : T extends ${longs.Long} ? string | number | Long ` : '';
-    const keys = options.outputTypeRegistry ? ts_poet_1.code `Exclude<keyof T, '$type'>` : ts_poet_1.code `keyof T`;
+    const Builtin = ts_poet_1.conditionalOutput('Builtin', ts_poet_1.code `type Builtin = Date | Function | Uint8Array | string | number | boolean | undefined;`);
+    // Based on https://github.com/sindresorhus/type-fest/pull/259
+    const maybeExcludeType = options.outputTypeRegistry ? `| '$type'` : '';
+    const Exact = ts_poet_1.conditionalOutput('Exact', ts_poet_1.code `
+      type KeysOfUnion<T> = T extends T ? keyof T : never;
+      ${maybeExport} type Exact<P, I extends P> = P extends ${Builtin}
+        ? P
+        : P &
+        { [K in keyof P]: Exact<P[K], I[K]> } & Record<Exclude<keyof I, KeysOfUnion<P> ${maybeExcludeType}>, never>;
+    `);
     // Based on the type from ts-essentials
+    const keys = options.outputTypeRegistry ? ts_poet_1.code `Exclude<keyof T, '$type'>` : ts_poet_1.code `keyof T`;
     const DeepPartial = ts_poet_1.conditionalOutput('DeepPartial', ts_poet_1.code `
-      type Builtin = Date | Function | Uint8Array | string | number | boolean | undefined;
-      ${maybeExport} type DeepPartial<T> =  T extends Builtin
+      ${maybeExport} type DeepPartial<T> =  T extends ${Builtin}
         ? T
         ${maybeLong}
         : T extends Array<infer U>
@@ -304,7 +313,7 @@ function makeDeepPartial(options, longs) {
         ? { [K in ${keys}]?: DeepPartial<T[K]> }
         : Partial<T>;
     `);
-    return { DeepPartial };
+    return { Builtin, DeepPartial, Exact };
 }
 function makeTimestampMethods(options, longs) {
     const Timestamp = ts_poet_1.imp('Timestamp@./google/protobuf/timestamp');
@@ -823,7 +832,7 @@ function generateFromJson(ctx, fullName, messageDesc) {
                             }
                         }
                         else if (types_1.isLong(valueType) && options.forceLong === options_1.LongOption.LONG) {
-                            return ts_poet_1.code `Long.fromString(${from} as string)`;
+                            return ts_poet_1.code `Long.fromValue(${from} as Long | string)`;
                         }
                         else if (types_1.isEnum(valueType)) {
                             return ts_poet_1.code `${from} as number`;
@@ -1045,8 +1054,9 @@ function generateFromPartial(ctx, fullName, messageDesc) {
     const chunks = [];
     const Timestamp = ts_poet_1.imp('Timestamp@./google/protobuf/timestamp');
     // create the basic function declaration
+    const paramName = messageDesc.field.length > 0 ? 'object' : '_';
     chunks.push(ts_poet_1.code `
-    fromPartial(${messageDesc.field.length > 0 ? 'object' : '_'}: ${utils.DeepPartial}<${fullName}>): ${fullName} {
+    fromPartial<I extends ${utils.Exact}<${utils.DeepPartial}<${fullName}>, I>>(${paramName}: I): ${fullName} {
       const message = { ...base${fullName} } as ${fullName};
   `);
     // add a check for each incoming field
@@ -1122,7 +1132,7 @@ function generateFromPartial(ctx, fullName, messageDesc) {
             }
             else {
                 chunks.push(ts_poet_1.code `
-          message.${fieldName} = (object.${fieldName} ?? []).map((e) => ${readSnippet('e')});
+          message.${fieldName} = object.${fieldName}?.map((e) => ${readSnippet('e')}) || [];
         `);
             }
         }
@@ -1174,7 +1184,7 @@ function generateWrap(fullProtoTypeName) {
     if (types_1.isAnyValueTypeName(fullProtoTypeName)) {
         chunks.push(ts_poet_1.code `wrap(value: any): Value {
       if (value === null) {
-        return {nullValue: 0} as Value;
+        return {nullValue: NullValue.NULL_VALUE} as Value;
       } else if (typeof value === 'boolean') {
         return {boolValue: value} as Value;
       } else if (typeof value === 'number') {
@@ -1212,19 +1222,20 @@ function generateUnwrap(fullProtoTypeName) {
     }
     if (types_1.isAnyValueTypeName(fullProtoTypeName)) {
         chunks.push(ts_poet_1.code `unwrap(message: Value): string | number | boolean | Object | null | Array<any> | undefined {
-      if (message.stringValue !== undefined) {
+      if (message?.stringValue !== undefined) {
         return message.stringValue;
-      } else if (message.numberValue !== undefined) {
+      } else if (message?.numberValue !== undefined) {
         return message.numberValue;
-      } else if (message.boolValue !== undefined) {
+      } else if (message?.boolValue !== undefined) {
         return message.boolValue;
-      } else if (message.structValue !== undefined) {
+      } else if (message?.structValue !== undefined) {
         return message.structValue;
-      } else if (message.listValue !== undefined) {
+      } else if (message?.listValue !== undefined) {
           return message.listValue;
-      } else if (message.nullValue !== undefined) {
+      } else if (message?.nullValue !== undefined) {
         return null;
       }
+      return undefined;
     }`);
     }
     if (types_1.isListValueTypeName(fullProtoTypeName)) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ts-proto",
-  "version": "1.90.1",
+  "version": "1.92.2",
   "description": "",
   "main": "build/plugin.js",
   "repository": "github:stephenh/ts-proto",
@@ -9,10 +9,19 @@
   },
   "scripts": {
     "build": "yarn tsc",
-    "setup": "cd ./integration && ./pbjs.sh && ./update-bins.sh && ./codegen.sh",
+    "build:test": "yarn proto2bin && yarn proto2pbjs && yarn bin2ts",
+    "build:test:local": "yarn proto2bin:local && yarn proto2pbjs:local && yarn bin2ts:local",
+    "proto2bin": "docker-compose run --rm protoc update-bins.sh",
+    "proto2bin-node": "docker-compose run --rm node update-bins.sh",
+    "proto2pbjs": "docker-compose run --rm protoc pbjs.sh",
+    "bin2ts": "docker-compose run --rm protoc codegen.sh",
+    "proto2bin:local": "integration/update-bins.sh",
+    "proto2pbjs:local": "integration/pbjs.sh",
+    "bin2ts:local": "integration/codegen.sh",
     "test": "yarn jest -c jest.config.js --maxWorkers=2",
     "prettier": "prettier --write {src,tests}/**/*.ts",
-    "prettier:check": "prettier --list-different {src,tests}/**/*.ts"
+    "prettier:check": "prettier --list-different {src,tests}/**/*.ts",
+    "setup:docker": "docker-compose build"
   },
   "files": [
     "build"