@clickhouse/client 1.18.4 → 1.18.5-head.360dc24.1

package/skills/clickhouse-js-node-coding/reference/custom-json.md

@@ -0,0 +1,149 @@
+# Custom JSON `parse` / `stringify`
+
+> **Requires:** client `>= 1.14.0` (configurable `json.parse` and
+> `json.stringify`). Earlier versions cannot swap the JSON implementation.
+
+Backing example:
+[`examples/node/coding/custom_json_handling.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/custom_json_handling.ts).
+
+## Answer checklist
+
+When the user wants `UInt64`/`Int64` values back as `BigInt`:
+
+- State that configurable `json.parse` / `json.stringify` requires
+  `@clickhouse/client >= 1.14.0`.
+- Show the supported `createClient({ json: { parse, stringify } })` option,
+  usually with `json-bigint` and `useNativeBigInt: true`.
+- Combine it with `output_format_json_quote_64bit_integers: 0` so the server
+  emits unquoted 64-bit integers that the parser can turn into `BigInt`.
+- Mention that `output_format_json_quote_64bit_integers: 0` is the default
+  since ClickHouse `25.8`, but setting it explicitly is useful for older
+  servers or portable examples.
+- Warn that casting to JavaScript `Number` / `parseInt` / `parseFloat` loses
+  precision above `Number.MAX_SAFE_INTEGER`.
+
+## Why customize?
+
+The default `JSON.stringify` / `JSON.parse`:
+
+- Throws on `BigInt`.
+- Calls `Date.prototype.toJSON()` (ISO string) — fine for `DateTime` with
+  `date_time_input_format: 'best_effort'`, surprising in some workflows.
+- Loses precision for 64-bit integers returned as numbers (a separate
+  issue — covered in the troubleshooting skill).
+
+A custom `{ parse, stringify }` lets you plug in `JSONBig`,
+`safe-stable-stringify`, your own `BigInt`-aware serializer, etc.
+
+## Recipe: BigInt-safe stringify, custom Date handling
+
+```ts
+import { createClient } from '@clickhouse/client'
+
+const valueSerializer = (value: unknown): unknown => {
+  // Serialize Date as a UNIX millis number (instead of toJSON's ISO string)
+  if (value instanceof Date) {
+    return value.getTime()
+  }
+
+  // Serialize BigInt as a string so JSON.stringify won't throw
+  if (typeof value === 'bigint') {
+    return value.toString()
+  }
+
+  if (Array.isArray(value)) {
+    return value.map(valueSerializer)
+  }
+
+  if (typeof value === 'object' && value !== null) {
+    return Object.fromEntries(
+      Object.entries(value).map(([k, v]) => [k, valueSerializer(v)]),
+    )
+  }
+
+  return value
+}
+
+const client = createClient({
+  json: {
+    parse: JSON.parse,
+    stringify: (obj: unknown) => JSON.stringify(valueSerializer(obj)),
+  },
+})
+
+await client.command({
+  query: `
+    CREATE OR REPLACE TABLE inserts_custom_json_handling
+    (id UInt64, dt DateTime64(3, 'UTC'))
+    ENGINE MergeTree
+    ORDER BY id
+  `,
+})
+
+await client.insert({
+  table: 'inserts_custom_json_handling',
+  format: 'JSONEachRow',
+  values: [
+    {
+      id: BigInt(250000000000000200), // serialized as a string
+      dt: new Date(), // serialized as ms since epoch
+    },
+  ],
+})
+
+const rows = await client.query({
+  query: 'SELECT * FROM inserts_custom_json_handling',
+  format: 'JSONEachRow',
+})
+console.info(await rows.json())
+await client.close()
+```
+
+> The custom `valueSerializer` runs **before** `JSON.stringify`, so values
+> are transformed before the standard hooks (`Date.prototype.toJSON`,
+> object `toJSON()` methods, etc.) ever run.
+
+## Recipe: BigInt-safe parsing for 64-bit integer columns
+
+If you want `UInt64`/`Int64` to come back as `BigInt`s (instead of strings
+or precision-lossy numbers), plug in a `BigInt`-aware parser such as
+[`json-bigint`](https://www.npmjs.com/package/json-bigint):
+
+```ts
+import { createClient } from '@clickhouse/client'
+import JSONBig from 'json-bigint'
+
+const bigJson = JSONBig({ useNativeBigInt: true })
+
+const client = createClient({
+  json: {
+    parse: bigJson.parse,
+    stringify: bigJson.stringify,
+  },
+  clickhouse_settings: {
+    output_format_json_quote_64bit_integers: 0,
+  },
+})
+```
+
+This applies to **both** outgoing JSON bodies and incoming JSON-format
+responses. Combine with `output_format_json_quote_64bit_integers: 0` (the
+default since CH 25.8) so the server emits unquoted 64-bit integers that
+`json-bigint` can parse to `BigInt`.
+
+## Common pitfalls
+
+- **Setting `json.parse` only.** That only affects reading JSON responses;
+  outgoing JSON bodies use `json.stringify`. If you want consistent custom
+  handling in both directions, generally provide a matching `stringify` too.
+- **Forgetting `bigint` handling in `stringify`.** Default `JSON.stringify`
+  throws on `BigInt`; if your data ever contains one, the insert will fail
+  with `TypeError: Do not know how to serialize a BigInt`.
+- **Targeting client `< 1.14.0`.** The `json` option doesn't exist; you'll
+  need to convert values manually before calling `insert()` / `query()` (or
+  upgrade).
+- **Casting 64-bit integers to `Number`.** JavaScript's `number` type has
+  only 53 bits of mantissa — values above `Number.MAX_SAFE_INTEGER` (2^53 − 1)
+  are silently rounded. Do **not** try to fix precision loss by calling
+  `Number()`, `parseInt()`, or `parseFloat()` on the value. The correct fix
+  is a `BigInt`-aware parser (shown above), not a lossy cast.

package/skills/clickhouse-js-node-coding/reference/data-types.md

@@ -0,0 +1,169 @@
+# Modern Data Types: Dynamic, Variant, JSON, Time, Time64
+
+> **Applies to** (server side):
+>
+> - `Variant`: ClickHouse `>= 24.1`.
+> - `Dynamic`: ClickHouse `>= 24.5`.
+> - New `JSON` (object) type: ClickHouse `>= 24.8`.
+> - All three are **no longer experimental since `25.3`**; on older servers,
+>   you must enable the corresponding `allow_experimental_*_type` setting.
+> - `Time` / `Time64`: ClickHouse `>= 25.6` and require
+>   `enable_time_time64_type: 1`.
+
+Backing examples:
+[`examples/node/coding/dynamic_variant_json.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/dynamic_variant_json.ts),
+[`examples/node/coding/time_time64.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/time_time64.ts).
+
+## Answer checklist
+
+When answering about storing and reading JSON objects:
+
+- Use the new `JSON` column type, introduced in ClickHouse `>= 24.8`.
+- Say `JSON` is no longer experimental since ClickHouse `25.3`; on older
+  supported versions, enable `allow_experimental_json_type`.
+- Insert real JS objects with `format: 'JSONEachRow'`; do not
+  `JSON.stringify()` the column value.
+- Read with a JSON output format such as `JSONEachRow` and `resultSet.json()`;
+  `JSON` column values come back as parsed JS objects.
+
+## `Dynamic`, `Variant(...)`, `JSON`
+
+```ts
+import { createClient } from '@clickhouse/client'
+
+const client = createClient({
+  // Required only on ClickHouse < 25.3 — harmless to leave on
+  clickhouse_settings: {
+    allow_experimental_variant_type: 1,
+    allow_experimental_dynamic_type: 1,
+    allow_experimental_json_type: 1,
+  },
+})
+
+await client.command({
+  query: `
+    CREATE OR REPLACE TABLE chjs_dynamic_variant_json
+    (
+      id      UInt64,
+      var     Variant(Int64, String),
+      dynamic Dynamic,
+      json    JSON
+    )
+    ENGINE MergeTree
+    ORDER BY id
+  `,
+})
+
+await client.insert({
+  table: 'chjs_dynamic_variant_json',
+  format: 'JSONEachRow',
+  values: [
+    { id: 1, var: 42, dynamic: 'foo', json: { foo: 'x' } },
+    { id: 2, var: 'str', dynamic: 144, json: { bar: 10 } },
+  ],
+})
+
+const rs = await client.query({
+  query: `
+    SELECT *,
+           variantType(var),
+           dynamicType(dynamic),
+           dynamicType(json.foo),
+           dynamicType(json.bar)
+    FROM chjs_dynamic_variant_json
+  `,
+  format: 'JSONEachRow',
+})
+console.log(await rs.json())
+```
+
+### Notes
+
+- The `JSON` column type accepts a real JS object on insert and returns one
+  on select — no need for `JSON.stringify` / `JSON.parse` in your app code.
+- A JS number written into a `Dynamic` or `Variant` column defaults to
+  `Int64` on the server. In JSON formats, `output_format_json_quote_64bit_integers`
+  controls how 64-bit integers are returned: `1` returns them as JSON strings,
+  while `0` returns them as JSON numbers (and `0` is the default since CH `25.8`).
+  In JS, large 64-bit integers returned as numbers can lose precision, so use
+  quoted output if you need exact integer values in application code.
+- Use `variantType(...)`, `dynamicType(...)` to introspect what the server
+  ended up storing.
+
+## `Time` and `Time64(p)`
+
+`Time` is signed seconds (`-999:59:59` … `999:59:59`). `Time64(p)` adds
+sub-second precision (`p` digits, up to `9` for nanoseconds). Both require
+`enable_time_time64_type: 1` on `>= 25.6`.
+
+```ts
+const client = createClient({
+  clickhouse_settings: { enable_time_time64_type: 1 },
+})
+
+await client.command({
+  query: `
+    CREATE OR REPLACE TABLE chjs_time_time64
+    (
+      id    UInt64,
+      t     Time,
+      t64_0 Time64(0),
+      t64_3 Time64(3),
+      t64_6 Time64(6),
+      t64_9 Time64(9),
+    )
+    ENGINE MergeTree
+    ORDER BY id
+  `,
+})
+
+await client.insert({
+  table: 'chjs_time_time64',
+  format: 'JSONEachRow',
+  values: [
+    {
+      id: 1,
+      t: '12:34:56',
+      t64_0: '12:34:56',
+      t64_3: '12:34:56.123',
+      t64_6: '12:34:56.123456',
+      t64_9: '12:34:56.123456789',
+    },
+    {
+      id: 2,
+      t: '999:59:59',
+      t64_0: '999:59:59',
+      t64_3: '999:59:59.999',
+      t64_6: '999:59:59.999999',
+      t64_9: '999:59:59.999999999',
+    },
+    {
+      id: 3,
+      t: '-999:59:59',
+      t64_0: '-999:59:59',
+      t64_3: '-999:59:59.999',
+      t64_6: '-999:59:59.999999',
+      t64_9: '-999:59:59.999999999',
+    },
+  ],
+})
+```
+
+### Notes
+
+- Pass values as **strings** in the `HH:MM:SS[.fraction]` format. Negatives
+  are supported; the magnitude can exceed 24 hours.
+- For `Time64(p)` with `p > 3`, do not use JS `Date` — it tops out at
+  millisecond precision and will silently truncate.
+
+## Common pitfalls
+
+- **Targeting old ClickHouse servers without the `allow_experimental_*`
+  setting.** On `< 25.3`, `CREATE TABLE` will fail without them.
+- **Expecting `JSON`-column reads to be raw strings.** They come back as
+  parsed objects in JSON formats.
+- **Inserting `Time64(9)` from JS `Date` and losing precision.** Use a
+  string instead.
+- **Reading a `Variant`/`Dynamic` value of type `Int64` and being surprised
+  it's a string.** That's the standard 64-bit-integers-in-JSON behavior;
+  see the troubleshooting skill if you need to change it.

package/skills/clickhouse-js-node-coding/reference/insert-columns.md

@@ -0,0 +1,113 @@
+# Insert into Specific Columns / Other Databases
+
+> **Applies to:** all versions. The `columns` option (both forms) and the
+> `database` config field are universally supported.
+
+Backing examples:
+[`examples/node/coding/insert_specific_columns.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/insert_specific_columns.ts),
+[`examples/node/coding/insert_exclude_columns.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/insert_exclude_columns.ts),
+[`examples/node/coding/insert_ephemeral_columns.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/insert_ephemeral_columns.ts),
+[`examples/node/coding/insert_into_different_db.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/insert_into_different_db.ts).
+
+## Answer checklist
+
+When explaining partial-column inserts:
+
+- Show `columns: ['col_a', 'col_b']` for the allowlist form.
+- Also mention the inverse `columns: { except: ['col_to_skip'] }` form so the
+  user knows both supported shapes.
+- Explain that omitted columns receive their server-side defaults
+  (`DEFAULT`, `MATERIALIZED`, `ALIAS`, nullable/type defaults) and inserts can
+  still fail or produce surprising zero/empty values if the table definition
+  has no appropriate defaults.
+
+## Insert into specific columns
+
+Pass `columns: string[]` to limit the `INSERT` to a subset. Omitted columns
+get their declared default.
+
+```ts
+await client.insert({
+  table: 'events',
+  format: 'JSONEachRow',
+  values: [{ message: 'foo' }],
+  columns: ['message'], // `id` will get its default (0 for UInt32)
+})
+```
+
+## Insert excluding columns
+
+Use `columns: { except: string[] }` for the inverse. Useful when most columns
+should default but you want to name only the few to skip.
+
+```ts
+await client.insert({
+  table: 'events',
+  format: 'JSONEachRow',
+  values: [{ message: 'bar' }],
+  columns: { except: ['id'] },
+})
+```
+
+## Tables with EPHEMERAL columns
+
+[Ephemeral columns](https://clickhouse.com/docs/en/sql-reference/statements/create/table#ephemeral)
+are not stored — they only exist to drive `DEFAULT` expressions of other
+columns. To trigger that default logic, **the ephemeral column must be in the
+`columns` list**, even though no value will be persisted for it.
+
+```ts
+await client.command({
+  query: `
+    CREATE OR REPLACE TABLE events
+    (
+      id              UInt64,
+      message         String DEFAULT message_default,
+      message_default String EPHEMERAL
+    )
+    ENGINE MergeTree
+    ORDER BY id
+  `,
+})
+
+await client.insert({
+  table: 'events',
+  format: 'JSONEachRow',
+  values: [
+    { id: '42', message_default: 'foo' },
+    { id: '144', message_default: 'bar' },
+  ],
+  // Including the ephemeral column name triggers the DEFAULT expression
+  columns: ['id', 'message_default'],
+})
+```
+
+## Insert into a different database
+
+If the client's default `database` is not the target, qualify the table name
+with `db.table`:
+
+```ts
+const client = createClient({ database: 'system' })
+
+await client.command({ query: 'CREATE DATABASE IF NOT EXISTS analytics' })
+
+await client.insert({
+  table: 'analytics.events', // fully qualified
+  format: 'JSONEachRow',
+  values: [{ id: 42, message: 'foo' }],
+})
+```
+
+There is no per-call `database` override on `insert()` / `query()` — qualify
+the identifier, or create a second client with the desired `database`.
+
+## Common pitfalls
+
+- **Forgetting the ephemeral column in `columns`.** If you list only the
+  non-ephemeral columns, the `DEFAULT` expression that depends on the
+  ephemeral value won't fire and you'll get empty/zero defaults instead.
+- **Hoping `client.insert({ database: '…' })` works.** It doesn't — qualify
+  the `table` instead.
+- **Mixing the two `columns` forms.** Use either `string[]` _or_
+  `{ except: string[] }`, not both.

package/skills/clickhouse-js-node-coding/reference/insert-formats.md

@@ -0,0 +1,145 @@
+# Insert Data Formats
+
+> **Applies to:** all versions. The `JSON` type column / new JSON family is a
+> ClickHouse feature; the JSON _formats_ listed here are universally supported
+> by the client.
+
+Backing examples:
+[`examples/node/coding/array_json_each_row.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/array_json_each_row.ts),
+[`examples/node/coding/insert_data_formats_overview.ts`](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/coding/insert_data_formats_overview.ts).
+
+> **Raw / binary formats (CSV, TSV, CustomSeparated, Parquet) require a Node
+> stream as input.** See
+> [`examples/node/performance/`](https://github.com/ClickHouse/clickhouse-js/tree/main/examples/node/performance)
+> — defer if the user wants to insert from a file or `Readable`.
+
+## Answer checklist
+
+When answering "what format/call should I use for an array of JS objects?":
+
+- Use `client.insert({ table, values, format: 'JSONEachRow' })`.
+- Say the array of plain objects can be passed directly as `values` for
+  ordinary in-memory batches such as a few thousand or tens of thousands of
+  rows.
+- Do not steer the user to streaming, Parquet, or file APIs unless their input
+  is already a stream/file or the task is explicitly about throughput.
+- Warn not to wrap `JSONEachRow` rows in a `{ data: [...] }` envelope; that
+  shape belongs to single-document formats.
+- Mention `JSONCompactEachRow*` as a denser alternative for larger payloads
+  when the caller can provide positional arrays or explicit names/types.
+
+## Default choice: `JSONEachRow` with an array of objects
+
+This is the right answer for ~90% of inserts.
+
+```ts
+import { createClient } from '@clickhouse/client'
+
+const client = createClient()
+
+await client.insert({
+  table: 'events',
+  format: 'JSONEachRow',
+  values: [
+    { id: 42, name: 'foo' },
+    { id: 43, name: 'bar' },
+  ],
+})
+
+await client.close()
+```
+
+The shape of `values` must match the chosen format.
+
+## Streamable JSON formats (pass an array)
+
+| Format                                       | `values` shape                                      |
+| -------------------------------------------- | --------------------------------------------------- |
+| `JSONEachRow`                                | `Array<{ col: value, ... }>`                        |
+| `JSONStringsEachRow`                         | `Array<{ col: stringifiedValue, ... }>`             |
+| `JSONCompactEachRow`                         | `Array<[v1, v2, ...]>`                              |
+| `JSONCompactStringsEachRow`                  | `Array<[stringV1, stringV2, ...]>`                  |
+| `JSONCompactEachRowWithNames`                | First row = column names, then data rows            |
+| `JSONCompactEachRowWithNamesAndTypes`        | Row 1 = names, row 2 = types, then data             |
+| `JSONCompactStringsEachRowWithNames`         | First row = names, then stringified data rows       |
+| `JSONCompactStringsEachRowWithNamesAndTypes` | Row 1 = names, row 2 = types, then stringified data |
+
+```ts
+await client.insert({
+  table: 'events',
+  format: 'JSONCompactEachRowWithNamesAndTypes',
+  values: [
+    ['id', 'name', 'sku'],
+    ['UInt32', 'String', 'Array(UInt32)'],
+    [11, 'foo', [1, 2, 3]],
+    [12, 'bar', [4, 5, 6]],
+  ],
+})
+```
+
+These formats can be **streamed** — pass a Node stream of rows instead of an
+array. See
+[`examples/node/performance/`](https://github.com/ClickHouse/clickhouse-js/tree/main/examples/node/performance)
+for streaming guidance.
+
+## Single-document JSON formats (pass an object)
+
+These cannot be streamed — the entire body is sent in one shot.
+
+| Format                    | `values` shape (typed via `InputJSON<T>` / `InputJSONObjectEachRow<T>`)                                                   |
+| ------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
+| `JSON`                    | `{ meta: [], data: Array<{ col: value, ... }> }` — for TypeScript/client usage, pass `meta: []` if metadata is not needed |
+| `JSONCompact`             | `{ meta: [{ name, type }, ...], data: Array<[v1, v2, ...]> }`                                                             |
+| `JSONColumnsWithMetadata` | `{ meta: [...], data: { col1: [v, ...], col2: [v, ...] } }`                                                               |
+| `JSONObjectEachRow`       | `Record<string, { col: value, ... }>` (the record key labels each row but is not stored)                                  |
+
+```ts
+import type { InputJSON, InputJSONObjectEachRow } from '@clickhouse/client'
+
+const meta: InputJSON['meta'] = [
+  { name: 'id', type: 'UInt32' },
+  { name: 'name', type: 'String' },
+]
+
+await client.insert({
+  table: 'events',
+  format: 'JSONCompact',
+  values: {
+    meta,
+    data: [
+      [19, 'foo'],
+      [20, 'bar'],
+    ],
+  },
+})
+
+await client.insert({
+  table: 'events',
+  format: 'JSONObjectEachRow',
+  values: {
+    row_1: { id: 23, name: 'foo' },
+    row_2: { id: 24, name: 'bar' },
+  } satisfies InputJSONObjectEachRow<{ id: number; name: string }>,
+})
+```
+
+## Quick chooser
+
+| Use case                                     | Format                                            |
+| -------------------------------------------- | ------------------------------------------------- |
+| Insert plain JS objects                      | `JSONEachRow` _(default)_                         |
+| Insert tuples / column-positional rows       | `JSONCompactEachRow`                              |
+| Insert with explicit column ordering / types | `JSONCompactEachRow*WithNames…`                   |
+| Insert a single document with metadata       | `JSON`, `JSONCompact`                             |
+| Insert from a CSV / TSV / Parquet file       | Raw format + Node stream → `examples/node/performance/` |
+
+## Common pitfalls
+
+- **Wrong shape for the format.** The most common cause of insert failures —
+  e.g., passing `Array<{...}>` to `JSONCompact` (which expects
+  `{ meta, data }`).
+- **Don't wrap a `JSONEachRow` array in a `{ data: [...] }` envelope.** That
+  envelope only belongs to single-document formats (`JSON` / `JSONCompact` /
+  `JSONColumnsWithMetadata`).
+- For type guidance (`Decimal` strings, `Date` objects, `BigInt`), see
+  `insert-values.md` and `custom-json.md`.