@google-cloud/bigquery-storage-api 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +111 -0
- package/build/protos/google/cloud/bigquery/storage/v1/annotations.proto +28 -0
- package/build/protos/google/cloud/bigquery/storage/v1/arrow.proto +89 -0
- package/build/protos/google/cloud/bigquery/storage/v1/avro.proto +81 -0
- package/build/protos/google/cloud/bigquery/storage/v1/protobuf.proto +48 -0
- package/build/protos/google/cloud/bigquery/storage/v1/storage.proto +786 -0
- package/build/protos/google/cloud/bigquery/storage/v1/stream.proto +335 -0
- package/build/protos/google/cloud/bigquery/storage/v1/table.proto +200 -0
- package/build/protos/protos.d.ts +12699 -0
- package/build/protos/protos.js +1 -0
- package/build/protos/protos.json +1 -0
- package/build/src/index.d.ts +14 -0
- package/build/src/index.js +30 -0
- package/build/src/index.js.map +1 -0
- package/build/src/v1/big_query_read_client.d.ts +424 -0
- package/build/src/v1/big_query_read_client.js +689 -0
- package/build/src/v1/big_query_read_client.js.map +1 -0
- package/build/src/v1/big_query_read_client_config.json +44 -0
- package/build/src/v1/big_query_read_proto_list.json +9 -0
- package/build/src/v1/big_query_write_client.d.ts +486 -0
- package/build/src/v1/big_query_write_client.js +835 -0
- package/build/src/v1/big_query_write_client.js.map +1 -0
- package/build/src/v1/big_query_write_client_config.json +73 -0
- package/build/src/v1/big_query_write_proto_list.json +9 -0
- package/build/src/v1/gapic_metadata.json +117 -0
- package/build/src/v1/index.d.ts +2 -0
- package/build/src/v1/index.js +25 -0
- package/build/src/v1/index.js.map +1 -0
- package/package.json +66 -0
package/README.md
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
[//]: # "This README.md file is auto-generated, all changes to this file will be lost."
|
|
2
|
+
[//]: # "The comments you see below are used to generate those parts of the template in later states."
|
|
3
|
+
<img src="https://avatars2.githubusercontent.com/u/2810941?v=3&s=96" alt="Google Cloud Platform logo" title="Google Cloud Platform" align="right" height="96" width="96"/>
|
|
4
|
+
|
|
5
|
+
# [BigQuery Storage API: Nodejs Client][homepage]
|
|
6
|
+
|
|
7
|
+
> [!IMPORTANT]
|
|
8
|
+
> **Internal API Client**: This library (`@google-cloud/bigquery-storage-api`) is an auto-generated, lightweight GAPIC layer designed as a dependency for the official handwritten client library. End users should install and import **[@google-cloud/bigquery-storage](https://www.npmjs.com/package/@google-cloud/bigquery-storage)** instead.
|
|
9
|
+
|
|
10
|
+
[//]: # "releaseLevel"
|
|
11
|
+
|
|
12
|
+
[](https://www.npmjs.org/package/@google-cloud/bigquery-storage-api)
|
|
13
|
+
|
|
14
|
+
BigQuery Storage API client for Node.js
|
|
15
|
+
|
|
16
|
+
[//]: # "partials.introduction"
|
|
17
|
+
|
|
18
|
+
A comprehensive list of changes in each version may be found in
|
|
19
|
+
[the CHANGELOG][homepage_changelog].
|
|
20
|
+
|
|
21
|
+
* [BigQuery Storage API Nodejs Client API Reference](https://cloud.google.com/nodejs/docs/reference/storage/latest)
|
|
22
|
+
* [BigQuery Storage API Documentation](https://cloud.google.com/bigquery/docs/reference/storage/)
|
|
23
|
+
|
|
24
|
+
Read more about the client libraries for Cloud APIs, including the older
|
|
25
|
+
Google APIs Client Libraries, in [Client Libraries Explained][explained].
|
|
26
|
+
|
|
27
|
+
[explained]: https://cloud.google.com/apis/docs/client-libraries-explained
|
|
28
|
+
|
|
29
|
+
**Table of contents:**
|
|
30
|
+
|
|
31
|
+
* [Quickstart](#quickstart)
|
|
32
|
+
* [Before you begin](#before-you-begin)
|
|
33
|
+
* [Installing the client library](#installing-the-client-library)
|
|
34
|
+
|
|
35
|
+
* [Versioning](#versioning)
|
|
36
|
+
* [Contributing](#contributing)
|
|
37
|
+
* [License](#license)
|
|
38
|
+
|
|
39
|
+
## Quickstart
|
|
40
|
+
### Before you begin
|
|
41
|
+
|
|
42
|
+
1. [Select or create a Cloud Platform project][projects].
|
|
43
|
+
1. [Enable billing for your project][billing].
|
|
44
|
+
1. [Enable the BigQuery Storage API API][enable_api].
|
|
45
|
+
1. [Set up authentication][auth] so you can access the
|
|
46
|
+
API from your local workstation.
|
|
47
|
+
### Installing the client library
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
npm install @google-cloud/bigquery-storage-api
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
[//]: # "partials.body"
|
|
54
|
+
|
|
55
|
+
## Samples
|
|
56
|
+
|
|
57
|
+
Samples are in the [`samples/`][homepage_samples] directory. Each sample's `README.md` has instructions for running its sample.
|
|
58
|
+
|
|
59
|
+
[//]: # "samples"
|
|
60
|
+
|
|
61
|
+
## Supported Node.js Versions
|
|
62
|
+
|
|
63
|
+
Our client libraries follow the [Node.js release schedule](https://github.com/nodejs/release#release-schedule).
|
|
64
|
+
Libraries are compatible with all current _active_ and _maintenance_ versions of
|
|
65
|
+
Node.js.
|
|
66
|
+
If you are using an end-of-life version of Node.js, we recommend that you update
|
|
67
|
+
as soon as possible to an actively supported LTS version.
|
|
68
|
+
|
|
69
|
+
Google's client libraries support legacy versions of Node.js runtimes on a
|
|
70
|
+
best-efforts basis with the following warnings:
|
|
71
|
+
|
|
72
|
+
* Legacy versions are not tested in continuous integration.
|
|
73
|
+
* Some security patches and features cannot be backported.
|
|
74
|
+
* Dependencies cannot be kept up-to-date.
|
|
75
|
+
|
|
76
|
+
Client libraries targeting some end-of-life versions of Node.js are available, and
|
|
77
|
+
can be installed through npm [dist-tags](https://docs.npmjs.com/cli/dist-tag).
|
|
78
|
+
The dist-tags follow the naming convention `legacy-(version)`.
|
|
79
|
+
For example, `npm install @google-cloud/bigquery-storage-api@legacy-8` installs client libraries
|
|
80
|
+
for versions compatible with Node.js 8.
|
|
81
|
+
|
|
82
|
+
## Versioning
|
|
83
|
+
|
|
84
|
+
This library follows [Semantic Versioning](http://semver.org/).
|
|
85
|
+
|
|
86
|
+
More Information: [Google Cloud Platform Launch Stages][launch_stages]
|
|
87
|
+
|
|
88
|
+
[launch_stages]: https://cloud.google.com/terms/launch-stages
|
|
89
|
+
|
|
90
|
+
## Contributing
|
|
91
|
+
|
|
92
|
+
Contributions welcome! See the [Contributing Guide](https://github.com/googleapis/google-cloud-node/blob/main/CONTRIBUTING.md).
|
|
93
|
+
|
|
94
|
+
Please note that this `README.md`
|
|
95
|
+
and a variety of configuration files in this repository (including `.nycrc` and `tsconfig.json`)
|
|
96
|
+
are generated from a central template.
|
|
97
|
+
|
|
98
|
+
## License
|
|
99
|
+
|
|
100
|
+
Apache Version 2.0
|
|
101
|
+
|
|
102
|
+
See [LICENSE](https://github.com/googleapis/google-cloud-node/blob/main/LICENSE)
|
|
103
|
+
|
|
104
|
+
[shell_img]: https://gstatic.com/cloudssh/images/open-btn.png
|
|
105
|
+
[projects]: https://console.cloud.google.com/project
|
|
106
|
+
[billing]: https://support.google.com/cloud/answer/6293499#enable-billing
|
|
107
|
+
[enable_api]: https://console.cloud.google.com/flows/enableapi?apiid=bigquerystorage.googleapis.com
|
|
108
|
+
[auth]: https://cloud.google.com/docs/authentication/external/set-up-adc-local
|
|
109
|
+
[homepage_samples]: https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-bigquery-storage-api/samples
|
|
110
|
+
[homepage_changelog]: https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-bigquery-storage-api/CHANGELOG.md
|
|
111
|
+
[homepage]: https://github.com/googleapis/google-cloud-node/blob/main/packages/google-cloud-bigquery-storage-api
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
syntax = "proto3";
|
|
2
|
+
|
|
3
|
+
package google.cloud.bigquery.storage.v1;
|
|
4
|
+
|
|
5
|
+
import "google/protobuf/descriptor.proto";
|
|
6
|
+
|
|
7
|
+
option csharp_namespace = "Google.Cloud.BigQuery.Storage.V1";
|
|
8
|
+
option go_package = "cloud.google.com/go/bigquery/storage/apiv1/storagepb;storagepb";
|
|
9
|
+
option java_package = "com.google.cloud.bigquery.storage.v1";
|
|
10
|
+
option java_multiple_files = true;
|
|
11
|
+
option java_outer_classname = "AnnotationsProto";
|
|
12
|
+
option php_namespace = "Google\\Cloud\\BigQuery\\Storage\\V1";
|
|
13
|
+
|
|
14
|
+
extend google.protobuf.FieldOptions {
|
|
15
|
+
// Setting the column_name extension allows users to reference
|
|
16
|
+
// bigquery column independently of the field name in the protocol buffer
|
|
17
|
+
// message.
|
|
18
|
+
//
|
|
19
|
+
// The intended use of this annotation is to reference a destination column
|
|
20
|
+
// named using characters unavailable for protobuf field names (e.g. unicode
|
|
21
|
+
// characters).
|
|
22
|
+
//
|
|
23
|
+
// More details about BigQuery naming limitations can be found here:
|
|
24
|
+
// https://cloud.google.com/bigquery/docs/schemas#column_names
|
|
25
|
+
//
|
|
26
|
+
// This extension is currently experimental.
|
|
27
|
+
optional string column_name = 454943157;
|
|
28
|
+
}
|
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
// Copyright 2025 Google LLC
|
|
2
|
+
//
|
|
3
|
+
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
4
|
+
// you may not use this file except in compliance with the License.
|
|
5
|
+
// You may obtain a copy of the License at
|
|
6
|
+
//
|
|
7
|
+
// http://www.apache.org/licenses/LICENSE-2.0
|
|
8
|
+
//
|
|
9
|
+
// Unless required by applicable law or agreed to in writing, software
|
|
10
|
+
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
11
|
+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
12
|
+
// See the License for the specific language governing permissions and
|
|
13
|
+
// limitations under the License.
|
|
14
|
+
|
|
15
|
+
syntax = "proto3";
|
|
16
|
+
|
|
17
|
+
package google.cloud.bigquery.storage.v1;
|
|
18
|
+
|
|
19
|
+
option csharp_namespace = "Google.Cloud.BigQuery.Storage.V1";
|
|
20
|
+
option go_package = "cloud.google.com/go/bigquery/storage/apiv1/storagepb;storagepb";
|
|
21
|
+
option java_multiple_files = true;
|
|
22
|
+
option java_outer_classname = "ArrowProto";
|
|
23
|
+
option java_package = "com.google.cloud.bigquery.storage.v1";
|
|
24
|
+
option php_namespace = "Google\\Cloud\\BigQuery\\Storage\\V1";
|
|
25
|
+
|
|
26
|
+
// Arrow schema as specified in
|
|
27
|
+
// https://arrow.apache.org/docs/python/api/datatypes.html
|
|
28
|
+
// and serialized to bytes using IPC:
|
|
29
|
+
// https://arrow.apache.org/docs/format/Columnar.html#serialization-and-interprocess-communication-ipc
|
|
30
|
+
//
|
|
31
|
+
// See code samples on how this message can be deserialized.
|
|
32
|
+
message ArrowSchema {
|
|
33
|
+
// IPC serialized Arrow schema.
|
|
34
|
+
bytes serialized_schema = 1;
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
// Arrow RecordBatch.
|
|
38
|
+
message ArrowRecordBatch {
|
|
39
|
+
// IPC-serialized Arrow RecordBatch.
|
|
40
|
+
bytes serialized_record_batch = 1;
|
|
41
|
+
|
|
42
|
+
// [Deprecated] The count of rows in `serialized_record_batch`.
|
|
43
|
+
// Please use the format-independent ReadRowsResponse.row_count instead.
|
|
44
|
+
int64 row_count = 2 [deprecated = true];
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
// Contains options specific to Arrow Serialization.
|
|
48
|
+
message ArrowSerializationOptions {
|
|
49
|
+
// Compression codec's supported by Arrow.
|
|
50
|
+
enum CompressionCodec {
|
|
51
|
+
// If unspecified no compression will be used.
|
|
52
|
+
COMPRESSION_UNSPECIFIED = 0;
|
|
53
|
+
|
|
54
|
+
// LZ4 Frame (https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md)
|
|
55
|
+
LZ4_FRAME = 1;
|
|
56
|
+
|
|
57
|
+
// Zstandard compression.
|
|
58
|
+
ZSTD = 2;
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
// The precision of the timestamp value in the Avro message. This precision
|
|
62
|
+
// will **only** be applied to the column(s) with the `TIMESTAMP_PICOS` type.
|
|
63
|
+
enum PicosTimestampPrecision {
|
|
64
|
+
// Unspecified timestamp precision. The default precision is microseconds.
|
|
65
|
+
PICOS_TIMESTAMP_PRECISION_UNSPECIFIED = 0;
|
|
66
|
+
|
|
67
|
+
// Timestamp values returned by Read API will be truncated to microsecond
|
|
68
|
+
// level precision. The value will be encoded as Arrow TIMESTAMP type in a
|
|
69
|
+
// 64 bit integer.
|
|
70
|
+
TIMESTAMP_PRECISION_MICROS = 1;
|
|
71
|
+
|
|
72
|
+
// Timestamp values returned by Read API will be truncated to nanosecond
|
|
73
|
+
// level precision. The value will be encoded as Arrow TIMESTAMP type in a
|
|
74
|
+
// 64 bit integer.
|
|
75
|
+
TIMESTAMP_PRECISION_NANOS = 2;
|
|
76
|
+
|
|
77
|
+
// Read API will return full precision picosecond value. The value will be
|
|
78
|
+
// encoded as a string which conforms to ISO 8601 format.
|
|
79
|
+
TIMESTAMP_PRECISION_PICOS = 3;
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
// The compression codec to use for Arrow buffers in serialized record
|
|
83
|
+
// batches.
|
|
84
|
+
CompressionCodec buffer_compression = 2;
|
|
85
|
+
|
|
86
|
+
// Optional. Set timestamp precision option. If not set, the default precision
|
|
87
|
+
// is microseconds.
|
|
88
|
+
PicosTimestampPrecision picos_timestamp_precision = 3;
|
|
89
|
+
}
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
// Copyright 2025 Google LLC
|
|
2
|
+
//
|
|
3
|
+
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
4
|
+
// you may not use this file except in compliance with the License.
|
|
5
|
+
// You may obtain a copy of the License at
|
|
6
|
+
//
|
|
7
|
+
// http://www.apache.org/licenses/LICENSE-2.0
|
|
8
|
+
//
|
|
9
|
+
// Unless required by applicable law or agreed to in writing, software
|
|
10
|
+
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
11
|
+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
12
|
+
// See the License for the specific language governing permissions and
|
|
13
|
+
// limitations under the License.
|
|
14
|
+
|
|
15
|
+
syntax = "proto3";
|
|
16
|
+
|
|
17
|
+
package google.cloud.bigquery.storage.v1;
|
|
18
|
+
|
|
19
|
+
option csharp_namespace = "Google.Cloud.BigQuery.Storage.V1";
|
|
20
|
+
option go_package = "cloud.google.com/go/bigquery/storage/apiv1/storagepb;storagepb";
|
|
21
|
+
option java_multiple_files = true;
|
|
22
|
+
option java_outer_classname = "AvroProto";
|
|
23
|
+
option java_package = "com.google.cloud.bigquery.storage.v1";
|
|
24
|
+
option php_namespace = "Google\\Cloud\\BigQuery\\Storage\\V1";
|
|
25
|
+
|
|
26
|
+
// Avro schema.
|
|
27
|
+
message AvroSchema {
|
|
28
|
+
// Json serialized schema, as described at
|
|
29
|
+
// https://avro.apache.org/docs/1.8.1/spec.html.
|
|
30
|
+
string schema = 1;
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
// Avro rows.
|
|
34
|
+
message AvroRows {
|
|
35
|
+
// Binary serialized rows in a block.
|
|
36
|
+
bytes serialized_binary_rows = 1;
|
|
37
|
+
|
|
38
|
+
// [Deprecated] The count of rows in the returning block.
|
|
39
|
+
// Please use the format-independent ReadRowsResponse.row_count instead.
|
|
40
|
+
int64 row_count = 2 [deprecated = true];
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
// Contains options specific to Avro Serialization.
|
|
44
|
+
message AvroSerializationOptions {
|
|
45
|
+
// The precision of the timestamp value in the Avro message. This precision
|
|
46
|
+
// will **only** be applied to the column(s) with the `TIMESTAMP_PICOS` type.
|
|
47
|
+
enum PicosTimestampPrecision {
|
|
48
|
+
// Unspecified timestamp precision. The default precision is microseconds.
|
|
49
|
+
PICOS_TIMESTAMP_PRECISION_UNSPECIFIED = 0;
|
|
50
|
+
|
|
51
|
+
// Timestamp values returned by Read API will be truncated to microsecond
|
|
52
|
+
// level precision. The value will be encoded as Avro TIMESTAMP type in a
|
|
53
|
+
// 64 bit integer.
|
|
54
|
+
TIMESTAMP_PRECISION_MICROS = 1;
|
|
55
|
+
|
|
56
|
+
// Timestamp values returned by Read API will be truncated to nanosecond
|
|
57
|
+
// level precision. The value will be encoded as Avro TIMESTAMP type in a
|
|
58
|
+
// 64 bit integer.
|
|
59
|
+
TIMESTAMP_PRECISION_NANOS = 2;
|
|
60
|
+
|
|
61
|
+
// Read API will return full precision picosecond value. The value will be
|
|
62
|
+
// encoded as a string which conforms to ISO 8601 format.
|
|
63
|
+
TIMESTAMP_PRECISION_PICOS = 3;
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
// Enable displayName attribute in Avro schema.
|
|
67
|
+
//
|
|
68
|
+
// The Avro specification requires field names to be alphanumeric. By
|
|
69
|
+
// default, in cases when column names do not conform to these requirements
|
|
70
|
+
// (e.g. non-ascii unicode codepoints) and Avro is requested as an output
|
|
71
|
+
// format, the CreateReadSession call will fail.
|
|
72
|
+
//
|
|
73
|
+
// Setting this field to true, populates avro field names with a placeholder
|
|
74
|
+
// value and populates a "displayName" attribute for every avro field with the
|
|
75
|
+
// original column name.
|
|
76
|
+
bool enable_display_name_attribute = 1;
|
|
77
|
+
|
|
78
|
+
// Optional. Set timestamp precision option. If not set, the default precision
|
|
79
|
+
// is microseconds.
|
|
80
|
+
PicosTimestampPrecision picos_timestamp_precision = 2;
|
|
81
|
+
}
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
// Copyright 2025 Google LLC
|
|
2
|
+
//
|
|
3
|
+
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
4
|
+
// you may not use this file except in compliance with the License.
|
|
5
|
+
// You may obtain a copy of the License at
|
|
6
|
+
//
|
|
7
|
+
// http://www.apache.org/licenses/LICENSE-2.0
|
|
8
|
+
//
|
|
9
|
+
// Unless required by applicable law or agreed to in writing, software
|
|
10
|
+
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
11
|
+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
12
|
+
// See the License for the specific language governing permissions and
|
|
13
|
+
// limitations under the License.
|
|
14
|
+
|
|
15
|
+
syntax = "proto3";
|
|
16
|
+
|
|
17
|
+
package google.cloud.bigquery.storage.v1;
|
|
18
|
+
|
|
19
|
+
import "google/protobuf/descriptor.proto";
|
|
20
|
+
|
|
21
|
+
option csharp_namespace = "Google.Cloud.BigQuery.Storage.V1";
|
|
22
|
+
option go_package = "cloud.google.com/go/bigquery/storage/apiv1/storagepb;storagepb";
|
|
23
|
+
option java_multiple_files = true;
|
|
24
|
+
option java_outer_classname = "ProtoBufProto";
|
|
25
|
+
option java_package = "com.google.cloud.bigquery.storage.v1";
|
|
26
|
+
option php_namespace = "Google\\Cloud\\BigQuery\\Storage\\V1";
|
|
27
|
+
|
|
28
|
+
// ProtoSchema describes the schema of the serialized protocol buffer data rows.
|
|
29
|
+
message ProtoSchema {
|
|
30
|
+
// Descriptor for input message. The provided descriptor must be self
|
|
31
|
+
// contained, such that data rows sent can be fully decoded using only the
|
|
32
|
+
// single descriptor. For data rows that are compositions of multiple
|
|
33
|
+
// independent messages, this means the descriptor may need to be transformed
|
|
34
|
+
// to only use nested types:
|
|
35
|
+
// https://developers.google.com/protocol-buffers/docs/proto#nested
|
|
36
|
+
//
|
|
37
|
+
// For additional information for how proto types and values map onto BigQuery
|
|
38
|
+
// see: https://cloud.google.com/bigquery/docs/write-api#data_type_conversions
|
|
39
|
+
google.protobuf.DescriptorProto proto_descriptor = 1;
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
message ProtoRows {
|
|
43
|
+
// A sequence of rows serialized as a Protocol Buffer.
|
|
44
|
+
//
|
|
45
|
+
// See https://developers.google.com/protocol-buffers/docs/overview for more
|
|
46
|
+
// information on deserializing this field.
|
|
47
|
+
repeated bytes serialized_rows = 1;
|
|
48
|
+
}
|