hf2vespa 0.1.0__tar.gz → 0.1.1__tar.gz

{hf2vespa-0.1.0 → hf2vespa-0.1.1}/.github/workflows/benchmark.yml

@@ -40,20 +40,18 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v4
 
-      - name: Set up Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: '3.12'
-          cache: 'pip'
+      - name: Install uv
+        uses: astral-sh/setup-uv@v6
+
+      - name: Install Python
+        run: uv python install 3.12
 
       - name: Install dependencies
-        run: |
-          python -m pip install --upgrade pip
-          pip install -e ".[dev]"
+        run: uv sync --dev
 
       - name: Run benchmarks
         run: |
-          pytest tests/benchmarks/ \
+          uv run pytest tests/benchmarks/ \
             --benchmark-enable \
             --benchmark-only \
             --benchmark-json=benchmark-results.json \

{hf2vespa-0.1.0 → hf2vespa-0.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hf2vespa
-Version: 0.1.0
+Version: 0.1.1
 Summary: Stream HuggingFace datasets to Vespa JSON format
 Author-email: Thomas Thoresen <thomas.h.thoresen@gmail.com>
 License: Apache-2.0
@@ -28,7 +28,7 @@ Requires-Dist: ruamel.yaml>=0.19.0
 
 Stream HuggingFace datasets to Vespa JSON format
 
-[![asciicast](https://asciinema.org/a/kdD2bsVNFUL51Era.svg)](https://asciinema.org/a/kdD2bsVNFUL51Era)
+[![asciicast](https://asciinema.org/a/VGODiMSiXua4FX7w.svg)](https://asciinema.org/a/VGODiMSiXua4FX7w?speed=2)
 
 ## Description
 
@@ -73,23 +73,23 @@ uv tool install .
 Stream a HuggingFace dataset to Vespa JSON format:
 
 ```bash
-hf2vespa feed glue --config ax --split test --limit 5
+hf2vespa feed mteb/msmarco-v2 --config corpus --split corpus --rename _id:id --limit 3
 ```
 
 **Output:**
 ```json
-{"put":"id:doc:doc::0","fields":{"premise":"The cat sat on the mat.","hypothesis":"The cat did not sit on the mat.","label":-1,"idx":0}}
-{"put":"id:doc:doc::1","fields":{"premise":"The cat did not sit on the mat.","hypothesis":"The cat sat on the mat.","label":-1,"idx":1}}
-{"put":"id:doc:doc::2","fields":{"premise":"When you've got no snow...","hypothesis":"When you've got snow...","label":-1,"idx":2}}
+{"put":"id:doc:doc::0","fields":{"id":"00_0","title":"0-60 Times - 0-60 | 0 to 60 Times & 1/4 Mile Times | Zero to 60 Car Reviews","text":"0-60 Times - 0-60 | 0 to 60 Times & 1/4 Mile Times | Zero to 60 Car Reviews."}}
+{"put":"id:doc:doc::1","fields":{"id":"00_172","title":"0-60 Times - 0-60 | 0 to 60 Times & 1/4 Mile Times | Zero to 60 Car Reviews","text":"This allow for a more accurate measure, as does running the test first in one direction and then in the exact opposite direction..."}}
+{"put":"id:doc:doc::2","fields":{"id":"00_587","title":"0-60 Times - 0-60 | 0 to 60 Times & 1/4 Mile Times | Zero to 60 Car Reviews","text":"Instead, some believe the measure should include a range of times rather than one finite mark..."}}
 ```
 
 ```
 --- Completion Statistics ---
-Total records processed: 5
-Successful: 5
+Total records processed: 3
+Successful: 3
 Errors: 0
-Throughput: 2.1 records/sec
-Elapsed time: 2.38s
+Throughput: 4.5 records/sec
+Elapsed time: 0.67s
 ```
 
 ### Preview Dataset Schema
@@ -97,33 +97,115 @@ Elapsed time: 2.38s
 Inspect a dataset and generate a YAML configuration template:
 
 ```bash
-hf2vespa init glue --config ax --split test --output config.yaml
+hf2vespa init Cohere/wikipedia-2023-11-embed-multilingual-v3 --config en -o cohere-config.yaml
 ```
 
-**Generated config.yaml:**
+**Generated cohere-config.yaml:**
 ```yaml
 namespace: doc
 doctype: doc
-id_column: # null = auto-increment
+id_column:
 
 mappings:
-  - source: premise
-    target: premise
+  - source: _id
+    target: _id
     type:  # string
-  - source: hypothesis
-    target: hypothesis
+  - source: url
+    target: url
+    type:  # string
+  - source: title
+    target: title
+    type:  # string
+  - source: text
+    target: text
     type:  # string
+  - source: emb
+    target: emb
+    type: tensor  # Sequence[float32] -> suggested: tensor
 ```
 
 ### Use Config File
 
-Apply the generated configuration:
+Edit the config to customize type conversions, then apply it:
+
+```bash
+hf2vespa feed Cohere/wikipedia-2023-11-embed-multilingual-v3 --config en --config-file cohere-config.yaml --limit 5
+```
+
+See [Two Modes of Operation](#two-modes-of-operation) below for a complete example with bfloat16 hex encoding.
+
+## Two Modes of Operation
+
+hf2vespa supports two modes depending on your needs:
+
+### CLI Mode (Quick & Simple)
+
+Use CLI arguments when you need:
+- Column renaming (`--rename old:new`)
+- Column filtering (`--include col1 --include col2`)
+- Custom namespace/doctype (`--namespace`, `--doctype`)
+- Preview data structure
+
+**Example:** Rename columns and stream MS MARCO corpus:
+
+```bash
+hf2vespa feed mteb/msmarco-v2 --config corpus --split corpus --rename _id:id --limit 5
+```
+
+**Output:**
+```json
+{"put":"id:doc:doc::0","fields":{"id":"00_0","title":"0-60 Times - 0-60 | 0 to 60 Times...","text":"0-60 Times - 0-60 | 0 to 60 Times..."}}
+{"put":"id:doc:doc::1","fields":{"id":"00_172","title":"0-60 Times...","text":"This allow for a more accurate measure..."}}
+```
+
+### Config File Mode (Advanced)
+
+Use `hf2vespa init` + YAML config when you need:
+- Type conversions (tensor, hex-encoded formats)
+- bfloat16/int8 quantized embeddings
+- Sparse or mixed tensors
+- Complex multi-field transformations
+
+**Example:** Convert Cohere embeddings to hex-encoded bfloat16:
+
+1. Generate a config template:
+```bash
+hf2vespa init Cohere/wikipedia-2023-11-embed-multilingual-v3 --config en --output cohere-config.yaml
+```
+
+2. Edit the config to use `tensor_bfloat16_hex` for the embedding field:
+```yaml
+# cohere-config.yaml
+namespace: doc
+doctype: doc
+id_column:
+
+mappings:
+  - source: _id
+    target: _id
+  - source: url
+    target: url
+  - source: title
+    target: title
+  - source: text
+    target: text
+  - source: emb
+    target: emb
+    type: tensor_bfloat16_hex  # Convert to hex-encoded bfloat16
+```
 
+3. Stream with the config file:
 ```bash
-hf2vespa feed glue --config ax --split test --config-file config.yaml --limit 5
+hf2vespa feed Cohere/wikipedia-2023-11-embed-multilingual-v3 --config en --config-file cohere-config.yaml --limit 2
+```
+
+**Output:**
+```json
+{"put":"id:doc:doc::0","fields":{"_id":"20231101.en_13194570_0","url":"https://en.wikipedia.org/wiki/British%20Arab%20Commercial%20Bank","title":"British Arab Commercial Bank","text":"The British Arab Commercial Bank PLC (BACB) is an international wholesale bank...","emb":{"values":"3aeabd253b963d1a3b833d8f3d8bbb16bc3e3b01..."}}}
+{"put":"id:doc:doc::1","fields":{"_id":"20231101.en_13194570_1","url":"https://en.wikipedia.org/wiki/British%20Arab%20Commercial%20Bank","title":"British Arab Commercial Bank","text":"BACB has a head office in London...","emb":{"values":"3baabcd7bc3c3d623cc13d853d94ba8dbb45bcb5..."}}}
 ```
 
-The config file defines field mappings, document IDs, and type conversions (e.g., converting lists to Vespa tensor format).
+The `emb` field is now hex-encoded bfloat16, reducing storage size by 50% compared to float32.
 
 ## YAML Configuration
 

{hf2vespa-0.1.0 → hf2vespa-0.1.1}/README.md

@@ -2,7 +2,7 @@
 
 Stream HuggingFace datasets to Vespa JSON format
 
-[![asciicast](https://asciinema.org/a/kdD2bsVNFUL51Era.svg)](https://asciinema.org/a/kdD2bsVNFUL51Era)
+[![asciicast](https://asciinema.org/a/VGODiMSiXua4FX7w.svg)](https://asciinema.org/a/VGODiMSiXua4FX7w?speed=2)
 
 ## Description
 
@@ -47,23 +47,23 @@ uv tool install .
 Stream a HuggingFace dataset to Vespa JSON format:
 
 ```bash
-hf2vespa feed glue --config ax --split test --limit 5
+hf2vespa feed mteb/msmarco-v2 --config corpus --split corpus --rename _id:id --limit 3
 ```
 
 **Output:**
 ```json
-{"put":"id:doc:doc::0","fields":{"premise":"The cat sat on the mat.","hypothesis":"The cat did not sit on the mat.","label":-1,"idx":0}}
-{"put":"id:doc:doc::1","fields":{"premise":"The cat did not sit on the mat.","hypothesis":"The cat sat on the mat.","label":-1,"idx":1}}
-{"put":"id:doc:doc::2","fields":{"premise":"When you've got no snow...","hypothesis":"When you've got snow...","label":-1,"idx":2}}
+{"put":"id:doc:doc::0","fields":{"id":"00_0","title":"0-60 Times - 0-60 | 0 to 60 Times & 1/4 Mile Times | Zero to 60 Car Reviews","text":"0-60 Times - 0-60 | 0 to 60 Times & 1/4 Mile Times | Zero to 60 Car Reviews."}}
+{"put":"id:doc:doc::1","fields":{"id":"00_172","title":"0-60 Times - 0-60 | 0 to 60 Times & 1/4 Mile Times | Zero to 60 Car Reviews","text":"This allow for a more accurate measure, as does running the test first in one direction and then in the exact opposite direction..."}}
+{"put":"id:doc:doc::2","fields":{"id":"00_587","title":"0-60 Times - 0-60 | 0 to 60 Times & 1/4 Mile Times | Zero to 60 Car Reviews","text":"Instead, some believe the measure should include a range of times rather than one finite mark..."}}
 ```
 
 ```
 --- Completion Statistics ---
-Total records processed: 5
-Successful: 5
+Total records processed: 3
+Successful: 3
 Errors: 0
-Throughput: 2.1 records/sec
-Elapsed time: 2.38s
+Throughput: 4.5 records/sec
+Elapsed time: 0.67s
 ```
 
 ### Preview Dataset Schema
@@ -71,33 +71,115 @@ Elapsed time: 2.38s
 Inspect a dataset and generate a YAML configuration template:
 
 ```bash
-hf2vespa init glue --config ax --split test --output config.yaml
+hf2vespa init Cohere/wikipedia-2023-11-embed-multilingual-v3 --config en -o cohere-config.yaml
 ```
 
-**Generated config.yaml:**
+**Generated cohere-config.yaml:**
 ```yaml
 namespace: doc
 doctype: doc
-id_column: # null = auto-increment
+id_column:
 
 mappings:
-  - source: premise
-    target: premise
+  - source: _id
+    target: _id
     type:  # string
-  - source: hypothesis
-    target: hypothesis
+  - source: url
+    target: url
+    type:  # string
+  - source: title
+    target: title
+    type:  # string
+  - source: text
+    target: text
     type:  # string
+  - source: emb
+    target: emb
+    type: tensor  # Sequence[float32] -> suggested: tensor
 ```
 
 ### Use Config File
 
-Apply the generated configuration:
+Edit the config to customize type conversions, then apply it:
+
+```bash
+hf2vespa feed Cohere/wikipedia-2023-11-embed-multilingual-v3 --config en --config-file cohere-config.yaml --limit 5
+```
+
+See [Two Modes of Operation](#two-modes-of-operation) below for a complete example with bfloat16 hex encoding.
+
+## Two Modes of Operation
+
+hf2vespa supports two modes depending on your needs:
+
+### CLI Mode (Quick & Simple)
+
+Use CLI arguments when you need:
+- Column renaming (`--rename old:new`)
+- Column filtering (`--include col1 --include col2`)
+- Custom namespace/doctype (`--namespace`, `--doctype`)
+- Preview data structure
+
+**Example:** Rename columns and stream MS MARCO corpus:
+
+```bash
+hf2vespa feed mteb/msmarco-v2 --config corpus --split corpus --rename _id:id --limit 5
+```
+
+**Output:**
+```json
+{"put":"id:doc:doc::0","fields":{"id":"00_0","title":"0-60 Times - 0-60 | 0 to 60 Times...","text":"0-60 Times - 0-60 | 0 to 60 Times..."}}
+{"put":"id:doc:doc::1","fields":{"id":"00_172","title":"0-60 Times...","text":"This allow for a more accurate measure..."}}
+```
+
+### Config File Mode (Advanced)
+
+Use `hf2vespa init` + YAML config when you need:
+- Type conversions (tensor, hex-encoded formats)
+- bfloat16/int8 quantized embeddings
+- Sparse or mixed tensors
+- Complex multi-field transformations
+
+**Example:** Convert Cohere embeddings to hex-encoded bfloat16:
+
+1. Generate a config template:
+```bash
+hf2vespa init Cohere/wikipedia-2023-11-embed-multilingual-v3 --config en --output cohere-config.yaml
+```
+
+2. Edit the config to use `tensor_bfloat16_hex` for the embedding field:
+```yaml
+# cohere-config.yaml
+namespace: doc
+doctype: doc
+id_column:
+
+mappings:
+  - source: _id
+    target: _id
+  - source: url
+    target: url
+  - source: title
+    target: title
+  - source: text
+    target: text
+  - source: emb
+    target: emb
+    type: tensor_bfloat16_hex  # Convert to hex-encoded bfloat16
+```
 
+3. Stream with the config file:
 ```bash
-hf2vespa feed glue --config ax --split test --config-file config.yaml --limit 5
+hf2vespa feed Cohere/wikipedia-2023-11-embed-multilingual-v3 --config en --config-file cohere-config.yaml --limit 2
+```
+
+**Output:**
+```json
+{"put":"id:doc:doc::0","fields":{"_id":"20231101.en_13194570_0","url":"https://en.wikipedia.org/wiki/British%20Arab%20Commercial%20Bank","title":"British Arab Commercial Bank","text":"The British Arab Commercial Bank PLC (BACB) is an international wholesale bank...","emb":{"values":"3aeabd253b963d1a3b833d8f3d8bbb16bc3e3b01..."}}}
+{"put":"id:doc:doc::1","fields":{"_id":"20231101.en_13194570_1","url":"https://en.wikipedia.org/wiki/British%20Arab%20Commercial%20Bank","title":"British Arab Commercial Bank","text":"BACB has a head office in London...","emb":{"values":"3baabcd7bc3c3d623cc13d853d94ba8dbb45bcb5..."}}}
 ```
 
-The config file defines field mappings, document IDs, and type conversions (e.g., converting lists to Vespa tensor format).
+The `emb` field is now hex-encoded bfloat16, reducing storage size by 50% compared to float32.
 
 ## YAML Configuration
 

{hf2vespa-0.1.0 → hf2vespa-0.1.1}/src/hf2vespa/cli.py

@@ -1,6 +1,8 @@
 # Suppress warnings that occur during cleanup. These are harmless but confusing to users.
 # 1. HuggingFace HTTP retry warnings: https://github.com/apache/arrow/issues/45214
 # 2. Multiprocessing resource tracker warnings (leaked semaphores from HF datasets)
+# 3. Python 3.14+ SyntaxWarning for 'return' in 'finally' blocks (multiprocess package)
+#    See: https://peps.python.org/pep-0765/
 #
 # IMPORTANT: Set PYTHONWARNINGS env var BEFORE any imports that might trigger multiprocessing.
 # The resource_tracker runs as a separate daemon process and inherits the env at spawn time.
@@ -8,18 +10,23 @@
 import os as _os
 
 _existing_warnings = _os.environ.get("PYTHONWARNINGS", "")
-_new_filter = "ignore::UserWarning:multiprocessing.resource_tracker"
+_new_filters = [
+    "ignore::UserWarning:multiprocessing.resource_tracker",
+    "ignore::SyntaxWarning:multiprocess",
+]
+_combined = ",".join(_new_filters)
 if _existing_warnings:
-    _os.environ["PYTHONWARNINGS"] = f"{_existing_warnings},{_new_filter}"
+    _os.environ["PYTHONWARNINGS"] = f"{_existing_warnings},{_combined}"
 else:
-    _os.environ["PYTHONWARNINGS"] = _new_filter
-del _os, _existing_warnings, _new_filter
+    _os.environ["PYTHONWARNINGS"] = _combined
+del _os, _existing_warnings, _new_filters, _combined
 
 import logging as _logging
 import warnings as _warnings
 
 _logging.getLogger("huggingface_hub.utils._http").setLevel(_logging.CRITICAL)
 _warnings.filterwarnings("ignore", message="resource_tracker:", category=UserWarning)
+_warnings.filterwarnings("ignore", category=SyntaxWarning, module="multiprocess")
 del _logging, _warnings
 
 """CLI for streaming HuggingFace datasets to Vespa JSON format."""

{hf2vespa-0.1.0 → hf2vespa-0.1.1}/src/hf2vespa.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hf2vespa
-Version: 0.1.0
+Version: 0.1.1
 Summary: Stream HuggingFace datasets to Vespa JSON format
 Author-email: Thomas Thoresen <thomas.h.thoresen@gmail.com>
 License: Apache-2.0
@@ -28,7 +28,7 @@ Requires-Dist: ruamel.yaml>=0.19.0
 
 Stream HuggingFace datasets to Vespa JSON format
 
-[![asciicast](https://asciinema.org/a/kdD2bsVNFUL51Era.svg)](https://asciinema.org/a/kdD2bsVNFUL51Era)
+[![asciicast](https://asciinema.org/a/VGODiMSiXua4FX7w.svg)](https://asciinema.org/a/VGODiMSiXua4FX7w?speed=2)
 
 ## Description
 
@@ -73,23 +73,23 @@ uv tool install .
 Stream a HuggingFace dataset to Vespa JSON format:
 
 ```bash
-hf2vespa feed glue --config ax --split test --limit 5
+hf2vespa feed mteb/msmarco-v2 --config corpus --split corpus --rename _id:id --limit 3
 ```
 
 **Output:**
 ```json
-{"put":"id:doc:doc::0","fields":{"premise":"The cat sat on the mat.","hypothesis":"The cat did not sit on the mat.","label":-1,"idx":0}}
-{"put":"id:doc:doc::1","fields":{"premise":"The cat did not sit on the mat.","hypothesis":"The cat sat on the mat.","label":-1,"idx":1}}
-{"put":"id:doc:doc::2","fields":{"premise":"When you've got no snow...","hypothesis":"When you've got snow...","label":-1,"idx":2}}
+{"put":"id:doc:doc::0","fields":{"id":"00_0","title":"0-60 Times - 0-60 | 0 to 60 Times & 1/4 Mile Times | Zero to 60 Car Reviews","text":"0-60 Times - 0-60 | 0 to 60 Times & 1/4 Mile Times | Zero to 60 Car Reviews."}}
+{"put":"id:doc:doc::1","fields":{"id":"00_172","title":"0-60 Times - 0-60 | 0 to 60 Times & 1/4 Mile Times | Zero to 60 Car Reviews","text":"This allow for a more accurate measure, as does running the test first in one direction and then in the exact opposite direction..."}}
+{"put":"id:doc:doc::2","fields":{"id":"00_587","title":"0-60 Times - 0-60 | 0 to 60 Times & 1/4 Mile Times | Zero to 60 Car Reviews","text":"Instead, some believe the measure should include a range of times rather than one finite mark..."}}
 ```
 
 ```
 --- Completion Statistics ---
-Total records processed: 5
-Successful: 5
+Total records processed: 3
+Successful: 3
 Errors: 0
-Throughput: 2.1 records/sec
-Elapsed time: 2.38s
+Throughput: 4.5 records/sec
+Elapsed time: 0.67s
 ```
 
 ### Preview Dataset Schema
@@ -97,33 +97,115 @@ Elapsed time: 2.38s
 Inspect a dataset and generate a YAML configuration template:
 
 ```bash
-hf2vespa init glue --config ax --split test --output config.yaml
+hf2vespa init Cohere/wikipedia-2023-11-embed-multilingual-v3 --config en -o cohere-config.yaml
 ```
 
-**Generated config.yaml:**
+**Generated cohere-config.yaml:**
 ```yaml
 namespace: doc
 doctype: doc
-id_column: # null = auto-increment
+id_column:
 
 mappings:
-  - source: premise
-    target: premise
+  - source: _id
+    target: _id
     type:  # string
-  - source: hypothesis
-    target: hypothesis
+  - source: url
+    target: url
+    type:  # string
+  - source: title
+    target: title
+    type:  # string
+  - source: text
+    target: text
     type:  # string
+  - source: emb
+    target: emb
+    type: tensor  # Sequence[float32] -> suggested: tensor
 ```
 
 ### Use Config File
 
-Apply the generated configuration:
+Edit the config to customize type conversions, then apply it:
+
+```bash
+hf2vespa feed Cohere/wikipedia-2023-11-embed-multilingual-v3 --config en --config-file cohere-config.yaml --limit 5
+```
+
+See [Two Modes of Operation](#two-modes-of-operation) below for a complete example with bfloat16 hex encoding.
+
+## Two Modes of Operation
+
+hf2vespa supports two modes depending on your needs:
+
+### CLI Mode (Quick & Simple)
+
+Use CLI arguments when you need:
+- Column renaming (`--rename old:new`)
+- Column filtering (`--include col1 --include col2`)
+- Custom namespace/doctype (`--namespace`, `--doctype`)
+- Preview data structure
+
+**Example:** Rename columns and stream MS MARCO corpus:
+
+```bash
+hf2vespa feed mteb/msmarco-v2 --config corpus --split corpus --rename _id:id --limit 5
+```
+
+**Output:**
+```json
+{"put":"id:doc:doc::0","fields":{"id":"00_0","title":"0-60 Times - 0-60 | 0 to 60 Times...","text":"0-60 Times - 0-60 | 0 to 60 Times..."}}
+{"put":"id:doc:doc::1","fields":{"id":"00_172","title":"0-60 Times...","text":"This allow for a more accurate measure..."}}
+```
+
+### Config File Mode (Advanced)
+
+Use `hf2vespa init` + YAML config when you need:
+- Type conversions (tensor, hex-encoded formats)
+- bfloat16/int8 quantized embeddings
+- Sparse or mixed tensors
+- Complex multi-field transformations
+
+**Example:** Convert Cohere embeddings to hex-encoded bfloat16:
+
+1. Generate a config template:
+```bash
+hf2vespa init Cohere/wikipedia-2023-11-embed-multilingual-v3 --config en --output cohere-config.yaml
+```
+
+2. Edit the config to use `tensor_bfloat16_hex` for the embedding field:
+```yaml
+# cohere-config.yaml
+namespace: doc
+doctype: doc
+id_column:
+
+mappings:
+  - source: _id
+    target: _id
+  - source: url
+    target: url
+  - source: title
+    target: title
+  - source: text
+    target: text
+  - source: emb
+    target: emb
+    type: tensor_bfloat16_hex  # Convert to hex-encoded bfloat16
+```
 
+3. Stream with the config file:
 ```bash
-hf2vespa feed glue --config ax --split test --config-file config.yaml --limit 5
+hf2vespa feed Cohere/wikipedia-2023-11-embed-multilingual-v3 --config en --config-file cohere-config.yaml --limit 2
+```
+
+**Output:**
+```json
+{"put":"id:doc:doc::0","fields":{"_id":"20231101.en_13194570_0","url":"https://en.wikipedia.org/wiki/British%20Arab%20Commercial%20Bank","title":"British Arab Commercial Bank","text":"The British Arab Commercial Bank PLC (BACB) is an international wholesale bank...","emb":{"values":"3aeabd253b963d1a3b833d8f3d8bbb16bc3e3b01..."}}}
+{"put":"id:doc:doc::1","fields":{"_id":"20231101.en_13194570_1","url":"https://en.wikipedia.org/wiki/British%20Arab%20Commercial%20Bank","title":"British Arab Commercial Bank","text":"BACB has a head office in London...","emb":{"values":"3baabcd7bc3c3d623cc13d853d94ba8dbb45bcb5..."}}}
 ```
 
-The config file defines field mappings, document IDs, and type conversions (e.g., converting lists to Vespa tensor format).
+The `emb` field is now hex-encoded bfloat16, reducing storage size by 50% compared to float32.
 
 ## YAML Configuration