@clickhouse/client 1.18.4 → 1.18.5-head.09b78f0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/config.d.ts +17 -0
- package/dist/config.js +1 -0
- package/dist/config.js.map +1 -1
- package/dist/connection/create_connection.d.ts +2 -1
- package/dist/connection/create_connection.js +4 -1
- package/dist/connection/create_connection.js.map +1 -1
- package/dist/connection/node_base_connection.d.ts +9 -0
- package/dist/connection/node_base_connection.js.map +1 -1
- package/dist/connection/node_custom_agent_connection.js +3 -0
- package/dist/connection/node_custom_agent_connection.js.map +1 -1
- package/dist/connection/node_http_connection.js +3 -0
- package/dist/connection/node_http_connection.js.map +1 -1
- package/dist/connection/node_https_connection.js +3 -0
- package/dist/connection/node_https_connection.js.map +1 -1
- package/dist/version.d.ts +1 -1
- package/dist/version.js +1 -1
- package/dist/version.js.map +1 -1
- package/package.json +6 -2
- package/skills/clickhouse-js-node-coding/SKILL.md +146 -0
- package/skills/clickhouse-js-node-coding/evals/evals.json +103 -0
- package/skills/clickhouse-js-node-coding/reference/async-insert.md +103 -0
- package/skills/clickhouse-js-node-coding/reference/client-configuration.md +159 -0
- package/skills/clickhouse-js-node-coding/reference/custom-json.md +149 -0
- package/skills/clickhouse-js-node-coding/reference/data-types.md +169 -0
- package/skills/clickhouse-js-node-coding/reference/insert-columns.md +113 -0
- package/skills/clickhouse-js-node-coding/reference/insert-formats.md +145 -0
- package/skills/clickhouse-js-node-coding/reference/insert-values.md +141 -0
- package/skills/clickhouse-js-node-coding/reference/ping.md +120 -0
- package/skills/clickhouse-js-node-coding/reference/query-parameters.md +152 -0
- package/skills/clickhouse-js-node-coding/reference/select-formats.md +111 -0
- package/skills/clickhouse-js-node-coding/reference/sessions.md +152 -0
- package/skills/clickhouse-js-node-troubleshooting/reference/socket-hangup.md +15 -3
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
# Async Inserts
|
|
2
|
+
|
|
3
|
+
> **Applies to:** all client versions; the relevant settings are server-side.
|
|
4
|
+
> See https://clickhouse.com/docs/en/optimize/asynchronous-inserts.
|
|
5
|
+
|
|
6
|
+
Backing example:
|
|
7
|
+
[`examples/node/coding/async_insert.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/async_insert.ts).
|
|
8
|
+
|
|
9
|
+
> **When to use async inserts:** when many small inserts arrive concurrently
|
|
10
|
+
> (e.g., one per HTTP request) and you don't want to maintain a client-side
|
|
11
|
+
> batching layer. ClickHouse will batch them server-side. This is also the
|
|
12
|
+
> recommended ingestion pattern for **ClickHouse Cloud**.
|
|
13
|
+
|
|
14
|
+
> **When _not_ to use async inserts:** when you already build large batches
|
|
15
|
+
> client-side (e.g., from a stream). Plain inserts are simpler and lower
|
|
16
|
+
> latency. For raw throughput tuning of large async-insert workloads, see
|
|
17
|
+
> [`examples/node/performance/`](https://github.com/ClickHouse/clickhouse-js/tree/main/examples/node/performance).
|
|
18
|
+
|
|
19
|
+
## Setup
|
|
20
|
+
|
|
21
|
+
Enable on the client (or per-request) via `clickhouse_settings`:
|
|
22
|
+
|
|
23
|
+
```ts
|
|
24
|
+
import { createClient, ClickHouseError } from '@clickhouse/client'
|
|
25
|
+
|
|
26
|
+
const client = createClient({
|
|
27
|
+
url: process.env.CLICKHOUSE_URL,
|
|
28
|
+
password: process.env.CLICKHOUSE_PASSWORD,
|
|
29
|
+
max_open_connections: 10,
|
|
30
|
+
clickhouse_settings: {
|
|
31
|
+
async_insert: 1,
|
|
32
|
+
wait_for_async_insert: 1, // wait for ack from server
|
|
33
|
+
async_insert_max_data_size: '1000000',
|
|
34
|
+
async_insert_busy_timeout_ms: 1000,
|
|
35
|
+
},
|
|
36
|
+
})
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
## Concurrent small inserts
|
|
40
|
+
|
|
41
|
+
Each call still uses the client's normal `insert()` API — the server merges
|
|
42
|
+
the batches.
|
|
43
|
+
|
|
44
|
+
```ts
|
|
45
|
+
const promises = [...new Array(10)].map(async () => {
|
|
46
|
+
const values = [...new Array(1000).keys()].map(() => ({
|
|
47
|
+
id: Math.floor(Math.random() * 100_000) + 1,
|
|
48
|
+
data: Math.random().toString(36).slice(2),
|
|
49
|
+
}))
|
|
50
|
+
|
|
51
|
+
await client
|
|
52
|
+
.insert({ table: 'async_insert_example', values, format: 'JSONEachRow' })
|
|
53
|
+
.catch((err) => {
|
|
54
|
+
if (err instanceof ClickHouseError) {
|
|
55
|
+
// err.code matches a row in system.errors
|
|
56
|
+
console.error(`ClickHouse error ${err.code}:`, err)
|
|
57
|
+
return
|
|
58
|
+
}
|
|
59
|
+
console.error('Insert failed:', err)
|
|
60
|
+
})
|
|
61
|
+
})
|
|
62
|
+
|
|
63
|
+
await Promise.all(promises)
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
## `wait_for_async_insert` — fire-and-forget vs ack
|
|
67
|
+
|
|
68
|
+
| `wait_for_async_insert` | Promise resolves when… | Trade-off |
|
|
69
|
+
| ----------------------- | ------------------------------------------------- | ------------------------------------------------------------------- |
|
|
70
|
+
| `1` (default) | Server has flushed the batch to the table | Slower per call; insert errors surface to the client |
|
|
71
|
+
| `0` | Server accepted the row into its in-memory buffer | Faster; flush errors won't surface — only validation/parsing errors |
|
|
72
|
+
|
|
73
|
+
With `wait_for_async_insert: 1`, expect each insert call to take roughly
|
|
74
|
+
`async_insert_busy_timeout_ms` to resolve when traffic is light, because the
|
|
75
|
+
server waits for more rows or for the timer to fire before flushing.
|
|
76
|
+
|
|
77
|
+
## Combining DDL with async inserts
|
|
78
|
+
|
|
79
|
+
When creating tables in scripts that immediately insert, ack the DDL with
|
|
80
|
+
`wait_end_of_query: 1` so the table is ready before the first insert:
|
|
81
|
+
|
|
82
|
+
```ts
|
|
83
|
+
await client.command({
|
|
84
|
+
query: `
|
|
85
|
+
CREATE OR REPLACE TABLE async_insert_example (id Int32, data String)
|
|
86
|
+
ENGINE MergeTree ORDER BY id
|
|
87
|
+
`,
|
|
88
|
+
clickhouse_settings: { wait_end_of_query: 1 },
|
|
89
|
+
})
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
## Common pitfalls
|
|
93
|
+
|
|
94
|
+
- **Setting `async_insert` per call but expecting client-side batching.**
|
|
95
|
+
The client still issues each `insert()` as a separate HTTP request — the
|
|
96
|
+
batching happens on the server.
|
|
97
|
+
- **Confusing `wait_for_async_insert` (async-insert ack) with
|
|
98
|
+
`wait_end_of_query` (DDL ack).** They are unrelated.
|
|
99
|
+
- **Treating a resolved insert under `wait_for_async_insert: 0` as
|
|
100
|
+
durably written.** It only means the server accepted the bytes; flush
|
|
101
|
+
failures will not surface to the client.
|
|
102
|
+
- **Not handling `ClickHouseError`.** It exposes `err.code`, which maps to
|
|
103
|
+
rows in the `system.errors` table — use it to decide whether to retry.
|
|
@@ -0,0 +1,159 @@
|
|
|
1
|
+
# Client Configuration
|
|
2
|
+
|
|
3
|
+
> **Applies to:** all versions, with these notable additions:
|
|
4
|
+
>
|
|
5
|
+
> - `pathname` config option: client `>= 1.0.0`.
|
|
6
|
+
> - `clickhouse_setting_*` / `ch_*` URL parameters: client `>= 1.0.0`.
|
|
7
|
+
> - `keep_alive.idle_socket_ttl` (Node-only): client `>= 1.0.0`.
|
|
8
|
+
|
|
9
|
+
Backing examples:
|
|
10
|
+
[`examples/node/coding/url_configuration.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/url_configuration.ts),
|
|
11
|
+
[`examples/node/coding/clickhouse_settings.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/clickhouse_settings.ts),
|
|
12
|
+
[`examples/node/coding/default_format_setting.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/default_format_setting.ts).
|
|
13
|
+
|
|
14
|
+
## Answer checklist
|
|
15
|
+
|
|
16
|
+
When answering configuration questions, include the relevant points:
|
|
17
|
+
|
|
18
|
+
- Show `createClient` from `@clickhouse/client` with explicit fields when the
|
|
19
|
+
user is writing code; this is easier to read and review than encoding
|
|
20
|
+
everything into a URL string.
|
|
21
|
+
- When mentioning the URL form for environment variables / DSNs: show a **Bash**
|
|
22
|
+
`export` with the literal URL value, and `createClient({ url: process.env.CLICKHOUSE_URL })`
|
|
23
|
+
in the Node code. **Never construct a URL in application code** — no string
|
|
24
|
+
concatenation, no template literals, no query-string builders.
|
|
25
|
+
- If URL parameters and object fields both set the same option, URL parameters
|
|
26
|
+
override the rest of the configuration object.
|
|
27
|
+
- If `clickhouse_settings` appear on `createClient`, explain that they are
|
|
28
|
+
defaults for every request and can be overridden on individual `query()`,
|
|
29
|
+
`insert()`, `command()`, or `exec()` calls.
|
|
30
|
+
- Remind long-running services to close the client during graceful shutdown.
|
|
31
|
+
- The `application` field sets the name that appears in `system.query_log`.
|
|
32
|
+
Do **not** mention any specific HTTP header name — the client handles header
|
|
33
|
+
mapping internally and the header names are an implementation detail.
|
|
34
|
+
|
|
35
|
+
## Minimal client
|
|
36
|
+
|
|
37
|
+
```ts
|
|
38
|
+
import { createClient } from '@clickhouse/client'
|
|
39
|
+
|
|
40
|
+
const client = createClient({
|
|
41
|
+
url: process.env.CLICKHOUSE_URL, // defaults to 'http://localhost:8123'
|
|
42
|
+
username: process.env.CLICKHOUSE_USER, // defaults to 'default'
|
|
43
|
+
password: process.env.CLICKHOUSE_PASSWORD, // defaults to ''
|
|
44
|
+
database: 'analytics', // defaults to 'default'
|
|
45
|
+
})
|
|
46
|
+
// ... your queries ...
|
|
47
|
+
await client.close()
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
`url` accepts a string or a `URL` object. The accepted string format is:
|
|
51
|
+
|
|
52
|
+
```
|
|
53
|
+
http[s]://[username:password@]hostname:port[/database][?param1=value1¶m2=value2]
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Configuration via URL parameters
|
|
57
|
+
|
|
58
|
+
A fixed allowlist of config fields can be set as URL query parameters
|
|
59
|
+
(plus any key prefixed with `clickhouse_setting_` / `ch_` / `http_header_`).
|
|
60
|
+
**Supported URL parameters override the corresponding values in the rest of
|
|
61
|
+
the configuration object** — when they do, the client logs a warning.
|
|
62
|
+
Unknown URL parameters cause `createClient` to throw
|
|
63
|
+
`Unknown URL parameters: ...`
|
|
64
|
+
(see [`packages/client-common/src/config.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/packages/client-common/src/config.ts) for the shared allowlist, and [`packages/client-node/src/config.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/packages/client-node/src/config.ts) for Node-specific URL parameters).
|
|
65
|
+
|
|
66
|
+
Supported non-prefixed keys parsed by `client-common`: `application`,
|
|
67
|
+
`session_id`, `pathname`, `access_token`, `request_timeout`,
|
|
68
|
+
`max_open_connections`, `compression_request`, `compression_response`,
|
|
69
|
+
`log_level`, `keep_alive_enabled`. Additionally, Node supports
|
|
70
|
+
`keep_alive_idle_socket_ttl` via the Node-specific config implementation.
|
|
71
|
+
Anything else must be passed via the config object on `createClient`.
|
|
72
|
+
|
|
73
|
+
Prefer explicit object fields in application code. Use the URL form when the
|
|
74
|
+
application receives one connection string from an environment variable, secret
|
|
75
|
+
manager, or config file. The URL value belongs in the environment, not in the
|
|
76
|
+
source code — show it as a shell export and read it in Node:
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
# In your shell environment / deployment config (e.g. .env, Kubernetes secret):
|
|
80
|
+
export CLICKHOUSE_URL='https://bob:secret@my.host:8124/analytics?application=my_analytics_app&ch_async_insert=1&ch_wait_for_async_insert=0'
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
```ts
|
|
84
|
+
// In your Node.js code — no URL construction needed:
|
|
85
|
+
const client = createClient({ url: process.env.CLICKHOUSE_URL })
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
The same connection can also be expressed as an explicit config object (useful when you want to document each field individually):
|
|
89
|
+
|
|
90
|
+
```ts
|
|
91
|
+
import { createClient } from '@clickhouse/client'
|
|
92
|
+
|
|
93
|
+
createClient({
|
|
94
|
+
url: 'https://my.host:8124',
|
|
95
|
+
username: 'bob',
|
|
96
|
+
password: 'secret',
|
|
97
|
+
database: 'analytics',
|
|
98
|
+
application: 'my_analytics_app',
|
|
99
|
+
clickhouse_settings: { async_insert: 1, wait_for_async_insert: 0 },
|
|
100
|
+
})
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
## Per-client vs per-request `clickhouse_settings` ⭐
|
|
104
|
+
|
|
105
|
+
> **Always mention this when discussing `clickhouse_settings`:** settings set
|
|
106
|
+
> on `createClient` are defaults; any individual call can override them.
|
|
107
|
+
|
|
108
|
+
Settings on `createClient` apply to every request. Settings on a single
|
|
109
|
+
operation (`query`, `insert`, `command`, `exec`) override the client defaults
|
|
110
|
+
for **that call only**.
|
|
111
|
+
|
|
112
|
+
```ts
|
|
113
|
+
const client = createClient({
|
|
114
|
+
clickhouse_settings: {
|
|
115
|
+
date_time_input_format: 'best_effort', // applied to every request
|
|
116
|
+
},
|
|
117
|
+
})
|
|
118
|
+
|
|
119
|
+
const rows = await client.query({
|
|
120
|
+
query: 'SELECT number FROM system.numbers LIMIT 2',
|
|
121
|
+
format: 'JSONEachRow',
|
|
122
|
+
clickhouse_settings: {
|
|
123
|
+
output_format_json_quote_64bit_integers: 1, // overrides client default for this call
|
|
124
|
+
},
|
|
125
|
+
})
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
## `default_format` for `exec()`
|
|
129
|
+
|
|
130
|
+
`client.exec()` runs an arbitrary statement and returns a stream. If your
|
|
131
|
+
query has no trailing `FORMAT …` clause, set `default_format` so the server
|
|
132
|
+
knows what to send back, then wrap the response in a `ResultSet`:
|
|
133
|
+
|
|
134
|
+
```ts
|
|
135
|
+
import { createClient, ResultSet } from '@clickhouse/client'
|
|
136
|
+
|
|
137
|
+
const client = createClient()
|
|
138
|
+
const format = 'JSONCompactEachRowWithNamesAndTypes'
|
|
139
|
+
const { stream, query_id } = await client.exec({
|
|
140
|
+
query: 'SELECT database, name, engine FROM system.tables LIMIT 5',
|
|
141
|
+
clickhouse_settings: { default_format: format },
|
|
142
|
+
})
|
|
143
|
+
const rs = new ResultSet(stream, format, query_id)
|
|
144
|
+
console.log(await rs.json())
|
|
145
|
+
await client.close()
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
For ordinary `SELECT`s prefer `client.query({ format })` — `default_format` is
|
|
149
|
+
only needed for raw `exec()`.
|
|
150
|
+
|
|
151
|
+
## Common pitfalls
|
|
152
|
+
|
|
153
|
+
- **Don't put a path in `url` and expect it to be the database name when
|
|
154
|
+
you're behind a proxy.** Use `pathname` for the proxy path and `database`
|
|
155
|
+
for the DB. (Symptom: "wrong database selected.") See the
|
|
156
|
+
troubleshooting skill for diagnosis.
|
|
157
|
+
- **Don't create a client per request.** `createClient` opens a connection
|
|
158
|
+
pool; share one client across the process and `close()` on shutdown.
|
|
159
|
+
- **`max_open_connections` must be `>= 1`** when set explicitly.
|
|
@@ -0,0 +1,149 @@
|
|
|
1
|
+
# Custom JSON `parse` / `stringify`
|
|
2
|
+
|
|
3
|
+
> **Requires:** client `>= 1.14.0` (configurable `json.parse` and
|
|
4
|
+
> `json.stringify`). Earlier versions cannot swap the JSON implementation.
|
|
5
|
+
|
|
6
|
+
Backing example:
|
|
7
|
+
[`examples/node/coding/custom_json_handling.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/custom_json_handling.ts).
|
|
8
|
+
|
|
9
|
+
## Answer checklist
|
|
10
|
+
|
|
11
|
+
When the user wants `UInt64`/`Int64` values back as `BigInt`:
|
|
12
|
+
|
|
13
|
+
- State that configurable `json.parse` / `json.stringify` requires
|
|
14
|
+
`@clickhouse/client >= 1.14.0`.
|
|
15
|
+
- Show the supported `createClient({ json: { parse, stringify } })` option,
|
|
16
|
+
usually with `json-bigint` and `useNativeBigInt: true`.
|
|
17
|
+
- Combine it with `output_format_json_quote_64bit_integers: 0` so the server
|
|
18
|
+
emits unquoted 64-bit integers that the parser can turn into `BigInt`.
|
|
19
|
+
- Mention that `output_format_json_quote_64bit_integers: 0` is the default
|
|
20
|
+
since ClickHouse `25.8`, but setting it explicitly is useful for older
|
|
21
|
+
servers or portable examples.
|
|
22
|
+
- Warn that casting to JavaScript `Number` / `parseInt` / `parseFloat` loses
|
|
23
|
+
precision above `Number.MAX_SAFE_INTEGER`.
|
|
24
|
+
|
|
25
|
+
## Why customize?
|
|
26
|
+
|
|
27
|
+
The default `JSON.stringify` / `JSON.parse`:
|
|
28
|
+
|
|
29
|
+
- Throws on `BigInt`.
|
|
30
|
+
- Calls `Date.prototype.toJSON()` (ISO string) — fine for `DateTime` with
|
|
31
|
+
`date_time_input_format: 'best_effort'`, surprising in some workflows.
|
|
32
|
+
- Loses precision for 64-bit integers returned as numbers (a separate
|
|
33
|
+
issue — covered in the troubleshooting skill).
|
|
34
|
+
|
|
35
|
+
A custom `{ parse, stringify }` lets you plug in `JSONBig`,
|
|
36
|
+
`safe-stable-stringify`, your own `BigInt`-aware serializer, etc.
|
|
37
|
+
|
|
38
|
+
## Recipe: BigInt-safe stringify, custom Date handling
|
|
39
|
+
|
|
40
|
+
```ts
|
|
41
|
+
import { createClient } from '@clickhouse/client'
|
|
42
|
+
|
|
43
|
+
const valueSerializer = (value: unknown): unknown => {
|
|
44
|
+
// Serialize Date as a UNIX millis number (instead of toJSON's ISO string)
|
|
45
|
+
if (value instanceof Date) {
|
|
46
|
+
return value.getTime()
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
// Serialize BigInt as a string so JSON.stringify won't throw
|
|
50
|
+
if (typeof value === 'bigint') {
|
|
51
|
+
return value.toString()
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
if (Array.isArray(value)) {
|
|
55
|
+
return value.map(valueSerializer)
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
if (typeof value === 'object' && value !== null) {
|
|
59
|
+
return Object.fromEntries(
|
|
60
|
+
Object.entries(value).map(([k, v]) => [k, valueSerializer(v)]),
|
|
61
|
+
)
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
return value
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
const client = createClient({
|
|
68
|
+
json: {
|
|
69
|
+
parse: JSON.parse,
|
|
70
|
+
stringify: (obj: unknown) => JSON.stringify(valueSerializer(obj)),
|
|
71
|
+
},
|
|
72
|
+
})
|
|
73
|
+
|
|
74
|
+
await client.command({
|
|
75
|
+
query: `
|
|
76
|
+
CREATE OR REPLACE TABLE inserts_custom_json_handling
|
|
77
|
+
(id UInt64, dt DateTime64(3, 'UTC'))
|
|
78
|
+
ENGINE MergeTree
|
|
79
|
+
ORDER BY id
|
|
80
|
+
`,
|
|
81
|
+
})
|
|
82
|
+
|
|
83
|
+
await client.insert({
|
|
84
|
+
table: 'inserts_custom_json_handling',
|
|
85
|
+
format: 'JSONEachRow',
|
|
86
|
+
values: [
|
|
87
|
+
{
|
|
88
|
+
id: BigInt(250000000000000200), // serialized as a string
|
|
89
|
+
dt: new Date(), // serialized as ms since epoch
|
|
90
|
+
},
|
|
91
|
+
],
|
|
92
|
+
})
|
|
93
|
+
|
|
94
|
+
const rows = await client.query({
|
|
95
|
+
query: 'SELECT * FROM inserts_custom_json_handling',
|
|
96
|
+
format: 'JSONEachRow',
|
|
97
|
+
})
|
|
98
|
+
console.info(await rows.json())
|
|
99
|
+
await client.close()
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
> The custom `valueSerializer` runs **before** `JSON.stringify`, so values
|
|
103
|
+
> are transformed before the standard hooks (`Date.prototype.toJSON`,
|
|
104
|
+
> object `toJSON()` methods, etc.) ever run.
|
|
105
|
+
|
|
106
|
+
## Recipe: BigInt-safe parsing for 64-bit integer columns
|
|
107
|
+
|
|
108
|
+
If you want `UInt64`/`Int64` to come back as `BigInt`s (instead of strings
|
|
109
|
+
or precision-lossy numbers), plug in a `BigInt`-aware parser such as
|
|
110
|
+
[`json-bigint`](https://www.npmjs.com/package/json-bigint):
|
|
111
|
+
|
|
112
|
+
```ts
|
|
113
|
+
import { createClient } from '@clickhouse/client'
|
|
114
|
+
import JSONBig from 'json-bigint'
|
|
115
|
+
|
|
116
|
+
const bigJson = JSONBig({ useNativeBigInt: true })
|
|
117
|
+
|
|
118
|
+
const client = createClient({
|
|
119
|
+
json: {
|
|
120
|
+
parse: bigJson.parse,
|
|
121
|
+
stringify: bigJson.stringify,
|
|
122
|
+
},
|
|
123
|
+
clickhouse_settings: {
|
|
124
|
+
output_format_json_quote_64bit_integers: 0,
|
|
125
|
+
},
|
|
126
|
+
})
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
This applies to **both** outgoing JSON bodies and incoming JSON-format
|
|
130
|
+
responses. Combine with `output_format_json_quote_64bit_integers: 0` (the
|
|
131
|
+
default since CH 25.8) so the server emits unquoted 64-bit integers that
|
|
132
|
+
`json-bigint` can parse to `BigInt`.
|
|
133
|
+
|
|
134
|
+
## Common pitfalls
|
|
135
|
+
|
|
136
|
+
- **Setting `json.parse` only.** That only affects reading JSON responses;
|
|
137
|
+
outgoing JSON bodies use `json.stringify`. If you want consistent custom
|
|
138
|
+
handling in both directions, generally provide a matching `stringify` too.
|
|
139
|
+
- **Forgetting `bigint` handling in `stringify`.** Default `JSON.stringify`
|
|
140
|
+
throws on `BigInt`; if your data ever contains one, the insert will fail
|
|
141
|
+
with `TypeError: Do not know how to serialize a BigInt`.
|
|
142
|
+
- **Targeting client `< 1.14.0`.** The `json` option doesn't exist; you'll
|
|
143
|
+
need to convert values manually before calling `insert()` / `query()` (or
|
|
144
|
+
upgrade).
|
|
145
|
+
- **Casting 64-bit integers to `Number`.** JavaScript's `number` type has
|
|
146
|
+
only 53 bits of mantissa — values above `Number.MAX_SAFE_INTEGER` (2^53 − 1)
|
|
147
|
+
are silently rounded. Do **not** try to fix precision loss by calling
|
|
148
|
+
`Number()`, `parseInt()`, or `parseFloat()` on the value. The correct fix
|
|
149
|
+
is a `BigInt`-aware parser (shown above), not a lossy cast.
|
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
# Modern Data Types: Dynamic, Variant, JSON, Time, Time64
|
|
2
|
+
|
|
3
|
+
> **Applies to** (server side):
|
|
4
|
+
>
|
|
5
|
+
> - `Variant`: ClickHouse `>= 24.1`.
|
|
6
|
+
> - `Dynamic`: ClickHouse `>= 24.5`.
|
|
7
|
+
> - New `JSON` (object) type: ClickHouse `>= 24.8`.
|
|
8
|
+
> - All three are **no longer experimental since `25.3`**; on older servers,
|
|
9
|
+
> you must enable the corresponding `allow_experimental_*_type` setting.
|
|
10
|
+
> - `Time` / `Time64`: ClickHouse `>= 25.6` and require
|
|
11
|
+
> `enable_time_time64_type: 1`.
|
|
12
|
+
|
|
13
|
+
Backing examples:
|
|
14
|
+
[`examples/node/coding/dynamic_variant_json.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/dynamic_variant_json.ts),
|
|
15
|
+
[`examples/node/coding/time_time64.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/time_time64.ts).
|
|
16
|
+
|
|
17
|
+
## Answer checklist
|
|
18
|
+
|
|
19
|
+
When answering about storing and reading JSON objects:
|
|
20
|
+
|
|
21
|
+
- Use the new `JSON` column type, introduced in ClickHouse `>= 24.8`.
|
|
22
|
+
- Say `JSON` is no longer experimental since ClickHouse `25.3`; on older
|
|
23
|
+
supported versions, enable `allow_experimental_json_type`.
|
|
24
|
+
- Insert real JS objects with `format: 'JSONEachRow'`; do not
|
|
25
|
+
`JSON.stringify()` the column value.
|
|
26
|
+
- Read with a JSON output format such as `JSONEachRow` and `resultSet.json()`;
|
|
27
|
+
`JSON` column values come back as parsed JS objects.
|
|
28
|
+
|
|
29
|
+
## `Dynamic`, `Variant(...)`, `JSON`
|
|
30
|
+
|
|
31
|
+
```ts
|
|
32
|
+
import { createClient } from '@clickhouse/client'
|
|
33
|
+
|
|
34
|
+
const client = createClient({
|
|
35
|
+
// Required only on ClickHouse < 25.3 — harmless to leave on
|
|
36
|
+
clickhouse_settings: {
|
|
37
|
+
allow_experimental_variant_type: 1,
|
|
38
|
+
allow_experimental_dynamic_type: 1,
|
|
39
|
+
allow_experimental_json_type: 1,
|
|
40
|
+
},
|
|
41
|
+
})
|
|
42
|
+
|
|
43
|
+
await client.command({
|
|
44
|
+
query: `
|
|
45
|
+
CREATE OR REPLACE TABLE chjs_dynamic_variant_json
|
|
46
|
+
(
|
|
47
|
+
id UInt64,
|
|
48
|
+
var Variant(Int64, String),
|
|
49
|
+
dynamic Dynamic,
|
|
50
|
+
json JSON
|
|
51
|
+
)
|
|
52
|
+
ENGINE MergeTree
|
|
53
|
+
ORDER BY id
|
|
54
|
+
`,
|
|
55
|
+
})
|
|
56
|
+
|
|
57
|
+
await client.insert({
|
|
58
|
+
table: 'chjs_dynamic_variant_json',
|
|
59
|
+
format: 'JSONEachRow',
|
|
60
|
+
values: [
|
|
61
|
+
{ id: 1, var: 42, dynamic: 'foo', json: { foo: 'x' } },
|
|
62
|
+
{ id: 2, var: 'str', dynamic: 144, json: { bar: 10 } },
|
|
63
|
+
],
|
|
64
|
+
})
|
|
65
|
+
|
|
66
|
+
const rs = await client.query({
|
|
67
|
+
query: `
|
|
68
|
+
SELECT *,
|
|
69
|
+
variantType(var),
|
|
70
|
+
dynamicType(dynamic),
|
|
71
|
+
dynamicType(json.foo),
|
|
72
|
+
dynamicType(json.bar)
|
|
73
|
+
FROM chjs_dynamic_variant_json
|
|
74
|
+
`,
|
|
75
|
+
format: 'JSONEachRow',
|
|
76
|
+
})
|
|
77
|
+
console.log(await rs.json())
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
### Notes
|
|
81
|
+
|
|
82
|
+
- The `JSON` column type accepts a real JS object on insert and returns one
|
|
83
|
+
on select — no need for `JSON.stringify` / `JSON.parse` in your app code.
|
|
84
|
+
- A JS number written into a `Dynamic` or `Variant` column defaults to
|
|
85
|
+
`Int64` on the server. In JSON formats, `output_format_json_quote_64bit_integers`
|
|
86
|
+
controls how 64-bit integers are returned: `1` returns them as JSON strings,
|
|
87
|
+
while `0` returns them as JSON numbers (and `0` is the default since CH `25.8`).
|
|
88
|
+
In JS, large 64-bit integers returned as numbers can lose precision, so use
|
|
89
|
+
quoted output if you need exact integer values in application code.
|
|
90
|
+
- Use `variantType(...)`, `dynamicType(...)` to introspect what the server
|
|
91
|
+
ended up storing.
|
|
92
|
+
|
|
93
|
+
## `Time` and `Time64(p)`
|
|
94
|
+
|
|
95
|
+
`Time` is signed seconds (`-999:59:59` … `999:59:59`). `Time64(p)` adds
|
|
96
|
+
sub-second precision (`p` digits, up to `9` for nanoseconds). Both require
|
|
97
|
+
`enable_time_time64_type: 1` on `>= 25.6`.
|
|
98
|
+
|
|
99
|
+
```ts
|
|
100
|
+
const client = createClient({
|
|
101
|
+
clickhouse_settings: { enable_time_time64_type: 1 },
|
|
102
|
+
})
|
|
103
|
+
|
|
104
|
+
await client.command({
|
|
105
|
+
query: `
|
|
106
|
+
CREATE OR REPLACE TABLE chjs_time_time64
|
|
107
|
+
(
|
|
108
|
+
id UInt64,
|
|
109
|
+
t Time,
|
|
110
|
+
t64_0 Time64(0),
|
|
111
|
+
t64_3 Time64(3),
|
|
112
|
+
t64_6 Time64(6),
|
|
113
|
+
t64_9 Time64(9),
|
|
114
|
+
)
|
|
115
|
+
ENGINE MergeTree
|
|
116
|
+
ORDER BY id
|
|
117
|
+
`,
|
|
118
|
+
})
|
|
119
|
+
|
|
120
|
+
await client.insert({
|
|
121
|
+
table: 'chjs_time_time64',
|
|
122
|
+
format: 'JSONEachRow',
|
|
123
|
+
values: [
|
|
124
|
+
{
|
|
125
|
+
id: 1,
|
|
126
|
+
t: '12:34:56',
|
|
127
|
+
t64_0: '12:34:56',
|
|
128
|
+
t64_3: '12:34:56.123',
|
|
129
|
+
t64_6: '12:34:56.123456',
|
|
130
|
+
t64_9: '12:34:56.123456789',
|
|
131
|
+
},
|
|
132
|
+
{
|
|
133
|
+
id: 2,
|
|
134
|
+
t: '999:59:59',
|
|
135
|
+
t64_0: '999:59:59',
|
|
136
|
+
t64_3: '999:59:59.999',
|
|
137
|
+
t64_6: '999:59:59.999999',
|
|
138
|
+
t64_9: '999:59:59.999999999',
|
|
139
|
+
},
|
|
140
|
+
{
|
|
141
|
+
id: 3,
|
|
142
|
+
t: '-999:59:59',
|
|
143
|
+
t64_0: '-999:59:59',
|
|
144
|
+
t64_3: '-999:59:59.999',
|
|
145
|
+
t64_6: '-999:59:59.999999',
|
|
146
|
+
t64_9: '-999:59:59.999999999',
|
|
147
|
+
},
|
|
148
|
+
],
|
|
149
|
+
})
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
### Notes
|
|
153
|
+
|
|
154
|
+
- Pass values as **strings** in the `HH:MM:SS[.fraction]` format. Negatives
|
|
155
|
+
are supported; the magnitude can exceed 24 hours.
|
|
156
|
+
- For `Time64(p)` with `p > 3`, do not use JS `Date` — it tops out at
|
|
157
|
+
millisecond precision and will silently truncate.
|
|
158
|
+
|
|
159
|
+
## Common pitfalls
|
|
160
|
+
|
|
161
|
+
- **Targeting old ClickHouse servers without the `allow_experimental_*`
|
|
162
|
+
setting.** On `< 25.3`, `CREATE TABLE` will fail without them.
|
|
163
|
+
- **Expecting `JSON`-column reads to be raw strings.** They come back as
|
|
164
|
+
parsed objects in JSON formats.
|
|
165
|
+
- **Inserting `Time64(9)` from JS `Date` and losing precision.** Use a
|
|
166
|
+
string instead.
|
|
167
|
+
- **Reading a `Variant`/`Dynamic` value of type `Int64` and being surprised
|
|
168
|
+
it's a string.** That's the standard 64-bit-integers-in-JSON behavior;
|
|
169
|
+
see the troubleshooting skill if you need to change it.
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
# Insert into Specific Columns / Other Databases
|
|
2
|
+
|
|
3
|
+
> **Applies to:** all versions. The `columns` option (both forms) and the
|
|
4
|
+
> `database` config field are universally supported.
|
|
5
|
+
|
|
6
|
+
Backing examples:
|
|
7
|
+
[`examples/node/coding/insert_specific_columns.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/insert_specific_columns.ts),
|
|
8
|
+
[`examples/node/coding/insert_exclude_columns.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/insert_exclude_columns.ts),
|
|
9
|
+
[`examples/node/coding/insert_ephemeral_columns.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/insert_ephemeral_columns.ts),
|
|
10
|
+
[`examples/node/coding/insert_into_different_db.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/insert_into_different_db.ts).
|
|
11
|
+
|
|
12
|
+
## Answer checklist
|
|
13
|
+
|
|
14
|
+
When explaining partial-column inserts:
|
|
15
|
+
|
|
16
|
+
- Show `columns: ['col_a', 'col_b']` for the allowlist form.
|
|
17
|
+
- Also mention the inverse `columns: { except: ['col_to_skip'] }` form so the
|
|
18
|
+
user knows both supported shapes.
|
|
19
|
+
- Explain that omitted columns receive their server-side defaults
|
|
20
|
+
(`DEFAULT`, `MATERIALIZED`, `ALIAS`, nullable/type defaults) and inserts can
|
|
21
|
+
still fail or produce surprising zero/empty values if the table definition
|
|
22
|
+
has no appropriate defaults.
|
|
23
|
+
|
|
24
|
+
## Insert into specific columns
|
|
25
|
+
|
|
26
|
+
Pass `columns: string[]` to limit the `INSERT` to a subset. Omitted columns
|
|
27
|
+
get their declared default.
|
|
28
|
+
|
|
29
|
+
```ts
|
|
30
|
+
await client.insert({
|
|
31
|
+
table: 'events',
|
|
32
|
+
format: 'JSONEachRow',
|
|
33
|
+
values: [{ message: 'foo' }],
|
|
34
|
+
columns: ['message'], // `id` will get its default (0 for UInt32)
|
|
35
|
+
})
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## Insert excluding columns
|
|
39
|
+
|
|
40
|
+
Use `columns: { except: string[] }` for the inverse. Useful when most columns
|
|
41
|
+
should default but you want to name only the few to skip.
|
|
42
|
+
|
|
43
|
+
```ts
|
|
44
|
+
await client.insert({
|
|
45
|
+
table: 'events',
|
|
46
|
+
format: 'JSONEachRow',
|
|
47
|
+
values: [{ message: 'bar' }],
|
|
48
|
+
columns: { except: ['id'] },
|
|
49
|
+
})
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
## Tables with EPHEMERAL columns
|
|
53
|
+
|
|
54
|
+
[Ephemeral columns](https://clickhouse.com/docs/en/sql-reference/statements/create/table#ephemeral)
|
|
55
|
+
are not stored — they only exist to drive `DEFAULT` expressions of other
|
|
56
|
+
columns. To trigger that default logic, **the ephemeral column must be in the
|
|
57
|
+
`columns` list**, even though no value will be persisted for it.
|
|
58
|
+
|
|
59
|
+
```ts
|
|
60
|
+
await client.command({
|
|
61
|
+
query: `
|
|
62
|
+
CREATE OR REPLACE TABLE events
|
|
63
|
+
(
|
|
64
|
+
id UInt64,
|
|
65
|
+
message String DEFAULT message_default,
|
|
66
|
+
message_default String EPHEMERAL
|
|
67
|
+
)
|
|
68
|
+
ENGINE MergeTree
|
|
69
|
+
ORDER BY id
|
|
70
|
+
`,
|
|
71
|
+
})
|
|
72
|
+
|
|
73
|
+
await client.insert({
|
|
74
|
+
table: 'events',
|
|
75
|
+
format: 'JSONEachRow',
|
|
76
|
+
values: [
|
|
77
|
+
{ id: '42', message_default: 'foo' },
|
|
78
|
+
{ id: '144', message_default: 'bar' },
|
|
79
|
+
],
|
|
80
|
+
// Including the ephemeral column name triggers the DEFAULT expression
|
|
81
|
+
columns: ['id', 'message_default'],
|
|
82
|
+
})
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
## Insert into a different database
|
|
86
|
+
|
|
87
|
+
If the client's default `database` is not the target, qualify the table name
|
|
88
|
+
with `db.table`:
|
|
89
|
+
|
|
90
|
+
```ts
|
|
91
|
+
const client = createClient({ database: 'system' })
|
|
92
|
+
|
|
93
|
+
await client.command({ query: 'CREATE DATABASE IF NOT EXISTS analytics' })
|
|
94
|
+
|
|
95
|
+
await client.insert({
|
|
96
|
+
table: 'analytics.events', // fully qualified
|
|
97
|
+
format: 'JSONEachRow',
|
|
98
|
+
values: [{ id: 42, message: 'foo' }],
|
|
99
|
+
})
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
There is no per-call `database` override on `insert()` / `query()` — qualify
|
|
103
|
+
the identifier, or create a second client with the desired `database`.
|
|
104
|
+
|
|
105
|
+
## Common pitfalls
|
|
106
|
+
|
|
107
|
+
- **Forgetting the ephemeral column in `columns`.** If you list only the
|
|
108
|
+
non-ephemeral columns, the `DEFAULT` expression that depends on the
|
|
109
|
+
ephemeral value won't fire and you'll get empty/zero defaults instead.
|
|
110
|
+
- **Hoping `client.insert({ database: '…' })` works.** It doesn't — qualify
|
|
111
|
+
the `table` instead.
|
|
112
|
+
- **Mixing the two `columns` forms.** Use either `string[]` _or_
|
|
113
|
+
`{ except: string[] }`, not both.
|