bpp-format 0.3.0__tar.gz → 0.4.0__tar.gz

{bpp_format-0.3.0 → bpp_format-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: bpp-format
-Version: 0.3.0
+Version: 0.4.0
 Summary: A token-efficient text format for feeding structured data and plans to LLMs, with a lossless JSON/YAML/CSV converter.
 Author: Furkan
 License-Expression: MIT
@@ -39,6 +39,11 @@ language models read with far fewer tokens. You can convert it back without losi
   every example.
 * **Lossless round trip:** `decode(encode(x)) == x` for JSON and YAML, with types preserved.
   CSV cells come back byte for byte. Markdown plans keep their structure.
+* **Same accuracy as JSON in a first comprehension test.** 6 datasets, 10 auto-graded questions
+  each, same data in every format. We set the bar before running: within 2 points of JSON counts
+  as "same accuracy". gpt-oss-120b: **84.2% from bpp vs 85.8% from pretty JSON** (2 of 120
+  questions). qwen3.8-27b: **80.0% from bpp and from minified JSON**. Small open models, small
+  samples; the caveats are [below](#honest-caveats).
 * **Measured, not invented.** Every syntax choice was benchmarked on two tokenizers. There are no
   made-up symbols or binary tricks, only patterns models already know from YAML, CSV, JSON and
   TypeScript.
@@ -52,7 +57,7 @@ token counts, with no install needed.
 ```
 JSON, pretty-printed (146 tokens)                  bpp (53 tokens)
 
-{                                                  bpp3
+{                                                  bpp4
   "store": "Downtown Branch",                      store Downtown Branch
   "orders": [                                      orders[3]{id product qty price customer}
     {"id": 1, "customer": "Alice Johnson",         1 Headphones 2 79.9 Alice Johnson
@@ -69,11 +74,11 @@ Token counts per format for six example files (o200k tokenizer; fewer is better)
 |---|---:|---:|---:|---:|---:|---:|---|
 | 60×10 table (CSV) | 5538 | 3562 | 4388 | 2277 | – | **2042** | −6% vs CSV |
 | nested config (YAML) | 601 | 365 | 443 | 399 | – | **323** | −12% vs minified JSON |
-| project plan with deps (JSON) | 1609 | 990 | 1212 | 1228 | – | **636** | −36% vs minified JSON |
-| Markdown checklist plan | 1501 | 1025 | 1119 | 1093 | 837 | **792** | −5% vs Markdown |
+| project plan with deps (JSON) | 1609 | 990 | 1212 | 1228 | – | **634** | −36% vs minified JSON |
+| Markdown checklist plan | 1445 | 990 | 1083 | 1024 | 837 | **758** | −9% vs Markdown |
 | API response with nested objects | 3524 | 2231 | 2636 | 2276 | – | **1151** | −48% vs minified JSON |
 | repetitive logs | 5629 | 4109 | 4587 | 3348 | – | **1857** | −42% vs CSV |
-| **total** | 18402 | 12282 | 14385 | 10621 | | **6801** | **−63% vs JSON, −36% vs TOON** |
+| **total** | 18346 | 12247 | 14349 | 10552 | | **6765** | **−63% vs JSON, −36% vs TOON** |
 
 Anthropic's published Claude tokenizer (`claude2`) shows the same picture: −62% vs JSON, −36% vs
 TOON. The full tables, the method and the cases where bpp wins by less are in
@@ -86,18 +91,19 @@ TOON. The full tables, the method and the cases where bpp wins by less are in
 You need Python 3.9 or newer (check with `python --version`). Then run:
 
 ```bash
-pip install https://github.com/E7lektronXF/bpp/archive/HEAD.zip
+pip install bpp-format
 ```
 
-This installs straight from GitHub. You don't need git. If `pip` isn't found, use `python3 -m pip`
-on macOS/Linux or `py -m pip` on Windows.
+If `pip` isn't found, use `python3 -m pip` on macOS/Linux or `py -m pip` on Windows.
 
-> ⚠️ **Don't run `pip install bpp`.** The package called `bpp` on PyPI is an unrelated project.
+> ⚠️ The package is called **`bpp-format`**; the command and the Python import are `bpp`.
+> `pip install bpp` installs an unrelated project.
 
-Optional: `pip install tiktoken` to get exact token counts instead of estimates.
+Optional: `pip install "bpp-format[stats]"` also installs tiktoken, for exact token counts instead
+of estimates.
 
 ```bash
-bpp --version      # bpp 0.3.0
+bpp --version      # bpp 0.4.0
 ```
 
 ### 2. Convert a file
@@ -113,15 +119,16 @@ This works with `.json`, `.yaml`, `.csv` and `.md` files. Want a sample to try?
 ### 3. Give it to your LLM
 
 Open `quickstart.bpp`, then paste it into ChatGPT, Claude or your prompt template. If the model
-hasn't seen the format before, add `--primer`. It prepends a one-line explanation (~85 tokens):
+hasn't seen the format before, add `--primer`. It prepends a one-line explanation of the syntax
+the file uses (40 tokens here, at most ~110):
 
 ```bash
 bpp quickstart.json --primer -o -     # print to the terminal instead of writing a file
 ```
 
 ```
-bpp3
-# bpp3: JSON as 'key value' lines, 1-space indent nests. k[N]{a b}: N rows of values in column order, last column = rest of line; x? = optional ('-' if absent), x?= columns appear as x=v. "..." = JSON string, *n = &n.
+bpp4
+# bpp4: JSON as 'key value' lines, 1-space indent nests. k[N]{a b}: N rows, values in column order, last one = rest of line.
 store Downtown Branch
 ...
 ```
@@ -144,10 +151,10 @@ JSON (indent 2)    434    146      136            +0.0%              +0.0%
 JSON (minified)    271     82       81           -43.8%             -40.4%
 YAML               261    102       82           -30.1%             -39.7%
 bpp                163     53       48           -63.7%             -64.7%
-bpp + primer       402    137      137            -6.2%              +0.7%
+bpp + primer       288     93       91           -36.3%             -33.1%
 ```
 
-On a file this small the primer eats most of the savings. Use it on larger inputs.
+On a file this small the primer still costs more than minified JSON. Use it on larger inputs.
 
 ### From Python
 
@@ -159,12 +166,13 @@ data = bpp.loads(text)          # bpp text -> Python object
 
 data = bpp.load("config.yaml")  # reads .json .yaml .csv .md .bpp
 bpp.dump(data, "config.bpp")    # format chosen by extension
+text = bpp.encode_md(markdown)  # Markdown text -> bpp (or the source itself if that is shorter)
 ```
 
 ## The format in one minute
 
 ```
-bpp3
+bpp4
 server
  host 0.0.0.0
  port 8080
@@ -185,16 +193,18 @@ regions [TR,DE,NL]
   means the rows indented under a row are its `items`, with their own columns.
 * **Optional columns:** `x?` sits in place, with `-` when absent. `x?=` appears only when present,
   as `x=value`.
-* **Trees:** `>steps` means rows indented under a row are its children. This is how plans are
-  written.
+* **Trees:** `>steps` means rows indented under a row are its children, and a bare `>` means the
+  same key as the table (`steps[6]{...}>`). This is how plans are written.
+* **Multi-line text:** `|3` stands for the next 3 lines, taken as they are: no quotes, no `\n`
+  escapes.
 * **Dictionary:** a long value repeated many times is written once as `&0 value` and referenced
   as `*0`, in YAML anchor style. It is used only when it measurably saves tokens.
 * **Strings** are quoted JSON-style only when they would be ambiguous. Unicode stays raw UTF-8.
 
-A plan with dependencies (`examples/plan.json`: 1609 tokens as JSON, 636 as bpp):
+A plan with dependencies (`examples/plan.json`: 1609 tokens as JSON, 634 as bpp):
 
 ```
-steps[6]{id:str status priority deps:str owner?= note?= title}>steps
+steps[6]{id:str status priority deps:str owner?= note?= title}>
 1 done P1 [] owner=Ayşe Gereksinim analizi
  1.1 done P1 [] Paydaş görüşmeleri
  1.2 done P0 [] Regülasyon incelemesi (BDDK, PCI-DSS)
@@ -213,17 +223,22 @@ ORD-2026-00001 Ankara paid standard 2897.68 *4 Ayşe Arslan
  SKU-940 1 1890.88 Laptop Standı
 ```
 
-A Markdown checklist (`examples/project_plan.md`: 837 tokens as Markdown, 792 as bpp). Headings
-and nested lists become a tree; `[ ]` `[x]` `[/]` `[-]` become `todo` `done` `doing` `cancelled`:
+A Markdown checklist (`examples/project_plan.md`: 837 tokens as Markdown, 758 as bpp). Headings
+and nested lists become a tree; `[ ]` `[x]` `[/]` `[-]` become `todo` `done` `doing` `cancelled`,
+and wrapped lines are joined:
 
 ```
-steps[5]{status? note?= title}>steps
-- Keşif ve envanter
- done note="Toplam 312 Airflow DAG'i, 48 Spark işi ve 17 Hive veritabanı tespit edildi." Mevcut iş akışlarının envanteri
+steps[5]{title}>{status? note?= title}>
+Keşif ve envanter
+ done Mevcut iş akışlarının envanteri Toplam 312 Airflow DAG'i, 48 Spark işi ve 17 Hive veritabanı tespit edildi.
  done Veri sahipleriyle görüşmeler
   done Pazarlama analitiği
 ```
 
+When the tree would be longer than the Markdown itself (short or prose-only files), `bpp x.md`
+writes `bpp4 md` and then the source unchanged. By the encoder's token estimate, a Markdown file
+never grows by more than that header line.
+
 The full grammar and the measurement behind each rule are in [SPEC.md](https://github.com/E7lektronXF/bpp/blob/main/SPEC.md).
 
 ## Why it is smaller
@@ -238,6 +253,8 @@ encodings usually cost more tokens and hurt understanding. bpp saves tokens by:
    gain is positive.
 4. **Keeping structure explicit:** row counts (`[N]`), column names and indentation give the model
    a frame to read against.
+5. **Writing text as text.** Multi-line strings are raw lines (`|N`), not JSON strings full of
+   `\n` and `\"` escapes.
 
 ## Guarantees
 
@@ -245,24 +262,32 @@ encodings usually cost more tokens and hurt understanding. bpp saves tokens by:
 |---|---|
 | JSON / YAML | `loads(dumps(x)) == x`, types included (`1` ≠ `1.0`, `"42"` ≠ `42`, `null`). With `--keep-order`, the JSON text comes back identical, key order included. YAML comments and anchors are not data and are not kept. Dates stay strings. |
 | CSV | Cells and column order come back byte for byte. Numbers are typed only when writing them back gives the same text (`007` and `1.50` stay strings). |
-| Markdown | Structure is kept, formatting is not. Output is normalized Markdown that parses back to the same tree. |
+| Markdown | Structure is kept, formatting is not (soft line breaks are joined). Output is normalized Markdown that parses back to the same tree. A file kept as `bpp4 md` comes back byte for byte. |
 
-Backed by 217 tests: edge cases (empty containers, 80-level nesting, delimiters inside strings,
+Backed by 240 tests: edge cases (empty containers, 80-level nesting, delimiters inside strings,
 multi-line text, Unicode, number-like strings) and hypothesis property tests on random JSON, CSV,
 YAML and Markdown trees.
 
 ## Honest caveats
 
-* **Understanding is not measured yet.** Token savings are measured. Whether models answer
-  questions about bpp as accurately as about JSON is not yet known. A ready-to-run comprehension
-  benchmark (10 auto-graded questions per dataset, every format) is included:
-  `ANTHROPIC_API_KEY=... python bench/run_qa.py`. The biggest risk is the dictionary: the model
-  has to resolve `*3` to its definition.
+* **Comprehension is measured, but only a little.** Two open-weight models on Groq's free tier
+  (gpt-oss-120b, 2 rounds; qwen3.8-27b, 1 round), so one question moves accuracy by 0.8–1.7
+  points. bpp stayed within the 2-point bar we set in advance. Not yet run on Claude, GPT or
+  Gemini; `python bench/run_qa.py` reproduces it with any of them (full tables:
+  [BENCHMARK §3](BENCHMARK.md#3-comprehension-benchmark-first-results)).
+* **The logs example is hard for every format** (counting over 80 entries: 0–8 of 20 right).
+  It is also where the `*n` dictionary does its work; bpp got 5/20 there vs 8/20 for pretty JSON
+  on gpt-oss-120b. Output-limit cut-offs (7 of 66 requests on that model, counted as wrong) blur
+  the difference, so a `--no-refs` run is the next measurement.
+* **The primer made no consistent difference:** −6.7 points on gpt-oss-120b (mostly logs
+  cut-offs), +1.7 on qwen (one question). Without it, bpp already met the bar.
 * **Token counts are proxies.** They come from tiktoken `o200k_base` and Anthropic's older public
   Claude tokenizer. Current Claude models use a different tokenizer. Set `ANTHROPIC_API_KEY` and
   `bpp stats` adds real `count_tokens` numbers.
 * **Small gains in some cases:** plain flat tables are only ~6% smaller than CSV on o200k (17% on
-  claude2). The primer (~85 tokens) cancels the savings on documents of a few hundred tokens.
+  claude2). On prose-heavy Markdown the tree saves 0.4–2% (this repository's README, SPEC,
+  BENCHMARK and RELEASING); the savings are in lists and checklists. The primer (40–110 tokens)
+  cancels the savings on documents of a few hundred tokens.
 
 ## Command reference
 
@@ -279,7 +304,7 @@ bpp stats data.json [--markdown]
 
 | option | effect |
 |---|---|
-| `--primer` | Prepend a format explanation (`short` ~85, `long` ~135 tokens). |
+| `--primer` | Prepend a format explanation (`short`: only the syntax used, 40–110 tokens; `long`: ~155 tokens). |
 | `--no-refs` | Disable the `&n`/`*n` dictionary. |
 | `--keep-order` | Never reorder keys. By default a table may move a free-text column such as `title` to the end of each row, which changes JSON key order but not the data. Always on for CSV input. |
 | `-o -` | Write to stdout. |
@@ -291,19 +316,19 @@ bpp stats data.json [--markdown]
 |---|---|
 | `bpp: command not found` | Use `python -m bpp ...`, or open a new terminal. |
 | `pip: command not found` | `python3 -m pip install ...` (macOS/Linux) or `py -m pip install ...` (Windows). |
-| `externally-managed-environment` | `pipx install https://github.com/E7lektronXF/bpp/archive/HEAD.zip`, or install inside a virtualenv (`python3 -m venv .venv && . .venv/bin/activate`). |
+| `externally-managed-environment` | `pipx install bpp-format`, or install inside a virtualenv (`python3 -m venv .venv && . .venv/bin/activate`). |
 | `cannot infer format` | Use one of these extensions: `.json .yaml .yml .csv .md .bpp`. |
 | `... exists; use -o ...` | bpp refused to overwrite your original file. Pick another name with `-o`. |
 
-Upgrade with `pip install --upgrade https://github.com/E7lektronXF/bpp/archive/HEAD.zip`. Uninstall
-with `pip uninstall bpp`.
+Upgrade with `pip install --upgrade bpp-format`. Uninstall
+with `pip uninstall bpp-format`.
 
 ## Development
 
 ```bash
 git clone https://github.com/E7lektronXF/bpp.git && cd bpp
 pip install -e ".[dev]"        # + pytest, hypothesis, tiktoken
-pytest -q                      # 217 tests
+pytest -q                      # 240 tests
 python bench/run_tokens.py     # regenerate the token benchmark
 python bench/experiments.py    # the design experiments behind SPEC.md
 python bench/run_qa.py         # comprehension benchmark (needs ANTHROPIC_API_KEY)

{bpp_format-0.3.0 → bpp_format-0.4.0}/README.md

@@ -8,6 +8,11 @@ language models read with far fewer tokens. You can convert it back without losi
   every example.
 * **Lossless round trip:** `decode(encode(x)) == x` for JSON and YAML, with types preserved.
   CSV cells come back byte for byte. Markdown plans keep their structure.
+* **Same accuracy as JSON in a first comprehension test.** 6 datasets, 10 auto-graded questions
+  each, same data in every format. We set the bar before running: within 2 points of JSON counts
+  as "same accuracy". gpt-oss-120b: **84.2% from bpp vs 85.8% from pretty JSON** (2 of 120
+  questions). qwen3.8-27b: **80.0% from bpp and from minified JSON**. Small open models, small
+  samples; the caveats are [below](#honest-caveats).
 * **Measured, not invented.** Every syntax choice was benchmarked on two tokenizers. There are no
   made-up symbols or binary tricks, only patterns models already know from YAML, CSV, JSON and
   TypeScript.
@@ -21,7 +26,7 @@ token counts, with no install needed.
 ```
 JSON, pretty-printed (146 tokens)                  bpp (53 tokens)
 
-{                                                  bpp3
+{                                                  bpp4
   "store": "Downtown Branch",                      store Downtown Branch
   "orders": [                                      orders[3]{id product qty price customer}
     {"id": 1, "customer": "Alice Johnson",         1 Headphones 2 79.9 Alice Johnson
@@ -38,11 +43,11 @@ Token counts per format for six example files (o200k tokenizer; fewer is better)
 |---|---:|---:|---:|---:|---:|---:|---|
 | 60×10 table (CSV) | 5538 | 3562 | 4388 | 2277 | – | **2042** | −6% vs CSV |
 | nested config (YAML) | 601 | 365 | 443 | 399 | – | **323** | −12% vs minified JSON |
-| project plan with deps (JSON) | 1609 | 990 | 1212 | 1228 | – | **636** | −36% vs minified JSON |
-| Markdown checklist plan | 1501 | 1025 | 1119 | 1093 | 837 | **792** | −5% vs Markdown |
+| project plan with deps (JSON) | 1609 | 990 | 1212 | 1228 | – | **634** | −36% vs minified JSON |
+| Markdown checklist plan | 1445 | 990 | 1083 | 1024 | 837 | **758** | −9% vs Markdown |
 | API response with nested objects | 3524 | 2231 | 2636 | 2276 | – | **1151** | −48% vs minified JSON |
 | repetitive logs | 5629 | 4109 | 4587 | 3348 | – | **1857** | −42% vs CSV |
-| **total** | 18402 | 12282 | 14385 | 10621 | | **6801** | **−63% vs JSON, −36% vs TOON** |
+| **total** | 18346 | 12247 | 14349 | 10552 | | **6765** | **−63% vs JSON, −36% vs TOON** |
 
 Anthropic's published Claude tokenizer (`claude2`) shows the same picture: −62% vs JSON, −36% vs
 TOON. The full tables, the method and the cases where bpp wins by less are in
@@ -55,18 +60,19 @@ TOON. The full tables, the method and the cases where bpp wins by less are in
 You need Python 3.9 or newer (check with `python --version`). Then run:
 
 ```bash
-pip install https://github.com/E7lektronXF/bpp/archive/HEAD.zip
+pip install bpp-format
 ```
 
-This installs straight from GitHub. You don't need git. If `pip` isn't found, use `python3 -m pip`
-on macOS/Linux or `py -m pip` on Windows.
+If `pip` isn't found, use `python3 -m pip` on macOS/Linux or `py -m pip` on Windows.
 
-> ⚠️ **Don't run `pip install bpp`.** The package called `bpp` on PyPI is an unrelated project.
+> ⚠️ The package is called **`bpp-format`**; the command and the Python import are `bpp`.
+> `pip install bpp` installs an unrelated project.
 
-Optional: `pip install tiktoken` to get exact token counts instead of estimates.
+Optional: `pip install "bpp-format[stats]"` also installs tiktoken, for exact token counts instead
+of estimates.
 
 ```bash
-bpp --version      # bpp 0.3.0
+bpp --version      # bpp 0.4.0
 ```
 
 ### 2. Convert a file
@@ -82,15 +88,16 @@ This works with `.json`, `.yaml`, `.csv` and `.md` files. Want a sample to try?
 ### 3. Give it to your LLM
 
 Open `quickstart.bpp`, then paste it into ChatGPT, Claude or your prompt template. If the model
-hasn't seen the format before, add `--primer`. It prepends a one-line explanation (~85 tokens):
+hasn't seen the format before, add `--primer`. It prepends a one-line explanation of the syntax
+the file uses (40 tokens here, at most ~110):
 
 ```bash
 bpp quickstart.json --primer -o -     # print to the terminal instead of writing a file
 ```
 
 ```
-bpp3
-# bpp3: JSON as 'key value' lines, 1-space indent nests. k[N]{a b}: N rows of values in column order, last column = rest of line; x? = optional ('-' if absent), x?= columns appear as x=v. "..." = JSON string, *n = &n.
+bpp4
+# bpp4: JSON as 'key value' lines, 1-space indent nests. k[N]{a b}: N rows, values in column order, last one = rest of line.
 store Downtown Branch
 ...
 ```
@@ -113,10 +120,10 @@ JSON (indent 2)    434    146      136            +0.0%              +0.0%
 JSON (minified)    271     82       81           -43.8%             -40.4%
 YAML               261    102       82           -30.1%             -39.7%
 bpp                163     53       48           -63.7%             -64.7%
-bpp + primer       402    137      137            -6.2%              +0.7%
+bpp + primer       288     93       91           -36.3%             -33.1%
 ```
 
-On a file this small the primer eats most of the savings. Use it on larger inputs.
+On a file this small the primer still costs more than minified JSON. Use it on larger inputs.
 
 ### From Python
 
@@ -128,12 +135,13 @@ data = bpp.loads(text)          # bpp text -> Python object
 
 data = bpp.load("config.yaml")  # reads .json .yaml .csv .md .bpp
 bpp.dump(data, "config.bpp")    # format chosen by extension
+text = bpp.encode_md(markdown)  # Markdown text -> bpp (or the source itself if that is shorter)
 ```
 
 ## The format in one minute
 
 ```
-bpp3
+bpp4
 server
  host 0.0.0.0
  port 8080
@@ -154,16 +162,18 @@ regions [TR,DE,NL]
   means the rows indented under a row are its `items`, with their own columns.
 * **Optional columns:** `x?` sits in place, with `-` when absent. `x?=` appears only when present,
   as `x=value`.
-* **Trees:** `>steps` means rows indented under a row are its children. This is how plans are
-  written.
+* **Trees:** `>steps` means rows indented under a row are its children, and a bare `>` means the
+  same key as the table (`steps[6]{...}>`). This is how plans are written.
+* **Multi-line text:** `|3` stands for the next 3 lines, taken as they are: no quotes, no `\n`
+  escapes.
 * **Dictionary:** a long value repeated many times is written once as `&0 value` and referenced
   as `*0`, in YAML anchor style. It is used only when it measurably saves tokens.
 * **Strings** are quoted JSON-style only when they would be ambiguous. Unicode stays raw UTF-8.
 
-A plan with dependencies (`examples/plan.json`: 1609 tokens as JSON, 636 as bpp):
+A plan with dependencies (`examples/plan.json`: 1609 tokens as JSON, 634 as bpp):
 
 ```
-steps[6]{id:str status priority deps:str owner?= note?= title}>steps
+steps[6]{id:str status priority deps:str owner?= note?= title}>
 1 done P1 [] owner=Ayşe Gereksinim analizi
  1.1 done P1 [] Paydaş görüşmeleri
  1.2 done P0 [] Regülasyon incelemesi (BDDK, PCI-DSS)
@@ -182,17 +192,22 @@ ORD-2026-00001 Ankara paid standard 2897.68 *4 Ayşe Arslan
  SKU-940 1 1890.88 Laptop Standı
 ```
 
-A Markdown checklist (`examples/project_plan.md`: 837 tokens as Markdown, 792 as bpp). Headings
-and nested lists become a tree; `[ ]` `[x]` `[/]` `[-]` become `todo` `done` `doing` `cancelled`:
+A Markdown checklist (`examples/project_plan.md`: 837 tokens as Markdown, 758 as bpp). Headings
+and nested lists become a tree; `[ ]` `[x]` `[/]` `[-]` become `todo` `done` `doing` `cancelled`,
+and wrapped lines are joined:
 
 ```
-steps[5]{status? note?= title}>steps
-- Keşif ve envanter
- done note="Toplam 312 Airflow DAG'i, 48 Spark işi ve 17 Hive veritabanı tespit edildi." Mevcut iş akışlarının envanteri
+steps[5]{title}>{status? note?= title}>
+Keşif ve envanter
+ done Mevcut iş akışlarının envanteri Toplam 312 Airflow DAG'i, 48 Spark işi ve 17 Hive veritabanı tespit edildi.
  done Veri sahipleriyle görüşmeler
   done Pazarlama analitiği
 ```
 
+When the tree would be longer than the Markdown itself (short or prose-only files), `bpp x.md`
+writes `bpp4 md` and then the source unchanged. By the encoder's token estimate, a Markdown file
+never grows by more than that header line.
+
 The full grammar and the measurement behind each rule are in [SPEC.md](https://github.com/E7lektronXF/bpp/blob/main/SPEC.md).
 
 ## Why it is smaller
@@ -207,6 +222,8 @@ encodings usually cost more tokens and hurt understanding. bpp saves tokens by:
    gain is positive.
 4. **Keeping structure explicit:** row counts (`[N]`), column names and indentation give the model
    a frame to read against.
+5. **Writing text as text.** Multi-line strings are raw lines (`|N`), not JSON strings full of
+   `\n` and `\"` escapes.
 
 ## Guarantees
 
@@ -214,24 +231,32 @@ encodings usually cost more tokens and hurt understanding. bpp saves tokens by:
 |---|---|
 | JSON / YAML | `loads(dumps(x)) == x`, types included (`1` ≠ `1.0`, `"42"` ≠ `42`, `null`). With `--keep-order`, the JSON text comes back identical, key order included. YAML comments and anchors are not data and are not kept. Dates stay strings. |
 | CSV | Cells and column order come back byte for byte. Numbers are typed only when writing them back gives the same text (`007` and `1.50` stay strings). |
-| Markdown | Structure is kept, formatting is not. Output is normalized Markdown that parses back to the same tree. |
+| Markdown | Structure is kept, formatting is not (soft line breaks are joined). Output is normalized Markdown that parses back to the same tree. A file kept as `bpp4 md` comes back byte for byte. |
 
-Backed by 217 tests: edge cases (empty containers, 80-level nesting, delimiters inside strings,
+Backed by 240 tests: edge cases (empty containers, 80-level nesting, delimiters inside strings,
 multi-line text, Unicode, number-like strings) and hypothesis property tests on random JSON, CSV,
 YAML and Markdown trees.
 
 ## Honest caveats
 
-* **Understanding is not measured yet.** Token savings are measured. Whether models answer
-  questions about bpp as accurately as about JSON is not yet known. A ready-to-run comprehension
-  benchmark (10 auto-graded questions per dataset, every format) is included:
-  `ANTHROPIC_API_KEY=... python bench/run_qa.py`. The biggest risk is the dictionary: the model
-  has to resolve `*3` to its definition.
+* **Comprehension is measured, but only a little.** Two open-weight models on Groq's free tier
+  (gpt-oss-120b, 2 rounds; qwen3.8-27b, 1 round), so one question moves accuracy by 0.8–1.7
+  points. bpp stayed within the 2-point bar we set in advance. Not yet run on Claude, GPT or
+  Gemini; `python bench/run_qa.py` reproduces it with any of them (full tables:
+  [BENCHMARK §3](BENCHMARK.md#3-comprehension-benchmark-first-results)).
+* **The logs example is hard for every format** (counting over 80 entries: 0–8 of 20 right).
+  It is also where the `*n` dictionary does its work; bpp got 5/20 there vs 8/20 for pretty JSON
+  on gpt-oss-120b. Output-limit cut-offs (7 of 66 requests on that model, counted as wrong) blur
+  the difference, so a `--no-refs` run is the next measurement.
+* **The primer made no consistent difference:** −6.7 points on gpt-oss-120b (mostly logs
+  cut-offs), +1.7 on qwen (one question). Without it, bpp already met the bar.
 * **Token counts are proxies.** They come from tiktoken `o200k_base` and Anthropic's older public
   Claude tokenizer. Current Claude models use a different tokenizer. Set `ANTHROPIC_API_KEY` and
   `bpp stats` adds real `count_tokens` numbers.
 * **Small gains in some cases:** plain flat tables are only ~6% smaller than CSV on o200k (17% on
-  claude2). The primer (~85 tokens) cancels the savings on documents of a few hundred tokens.
+  claude2). On prose-heavy Markdown the tree saves 0.4–2% (this repository's README, SPEC,
+  BENCHMARK and RELEASING); the savings are in lists and checklists. The primer (40–110 tokens)
+  cancels the savings on documents of a few hundred tokens.
 
 ## Command reference
 
@@ -248,7 +273,7 @@ bpp stats data.json [--markdown]
 
 | option | effect |
 |---|---|
-| `--primer` | Prepend a format explanation (`short` ~85, `long` ~135 tokens). |
+| `--primer` | Prepend a format explanation (`short`: only the syntax used, 40–110 tokens; `long`: ~155 tokens). |
 | `--no-refs` | Disable the `&n`/`*n` dictionary. |
 | `--keep-order` | Never reorder keys. By default a table may move a free-text column such as `title` to the end of each row, which changes JSON key order but not the data. Always on for CSV input. |
 | `-o -` | Write to stdout. |
@@ -260,19 +285,19 @@ bpp stats data.json [--markdown]
 |---|---|
 | `bpp: command not found` | Use `python -m bpp ...`, or open a new terminal. |
 | `pip: command not found` | `python3 -m pip install ...` (macOS/Linux) or `py -m pip install ...` (Windows). |
-| `externally-managed-environment` | `pipx install https://github.com/E7lektronXF/bpp/archive/HEAD.zip`, or install inside a virtualenv (`python3 -m venv .venv && . .venv/bin/activate`). |
+| `externally-managed-environment` | `pipx install bpp-format`, or install inside a virtualenv (`python3 -m venv .venv && . .venv/bin/activate`). |
 | `cannot infer format` | Use one of these extensions: `.json .yaml .yml .csv .md .bpp`. |
 | `... exists; use -o ...` | bpp refused to overwrite your original file. Pick another name with `-o`. |
 
-Upgrade with `pip install --upgrade https://github.com/E7lektronXF/bpp/archive/HEAD.zip`. Uninstall
-with `pip uninstall bpp`.
+Upgrade with `pip install --upgrade bpp-format`. Uninstall
+with `pip uninstall bpp-format`.
 
 ## Development
 
 ```bash
 git clone https://github.com/E7lektronXF/bpp.git && cd bpp
 pip install -e ".[dev]"        # + pytest, hypothesis, tiktoken
-pytest -q                      # 217 tests
+pytest -q                      # 240 tests
 python bench/run_tokens.py     # regenerate the token benchmark
 python bench/experiments.py    # the design experiments behind SPEC.md
 python bench/run_qa.py         # comprehension benchmark (needs ANTHROPIC_API_KEY)

{bpp_format-0.3.0 → bpp_format-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "bpp-format"
-version = "0.3.0"
+version = "0.4.0"
 description = "A token-efficient text format for feeding structured data and plans to LLMs, with a lossless JSON/YAML/CSV converter."
 readme = "README.md"
 requires-python = ">=3.9"

{bpp_format-0.3.0 → bpp_format-0.4.0}/src/bpp/__init__.py

@@ -4,17 +4,19 @@
     text = bpp.dumps({"users": [{"id": 1, "name": "Ayşe"}]})   # -> .bpp text
     data = bpp.loads(text)                                      # -> Python objects
     data = bpp.load("config.yaml")          # any supported file (.json .yaml .csv .md .bpp)
+    text = bpp.encode_md(markdown_text)     # Markdown -> .bpp (keeps the source if that is shorter)
     bpp.dump(data, "config.bpp")            # format from the extension
 """
 
 from pathlib import Path
 
-from .decoder import decode
-from .encoder import encode
+from .decoder import decode, md_source
+from .encoder import encode, encode_md
 from .lexer import BppError
 
-__version__ = "0.3.0"
-__all__ = ["encode", "decode", "dumps", "loads", "load", "dump", "BppError", "__version__"]
+__version__ = "0.4.0"
+__all__ = ["encode", "encode_md", "decode", "md_source", "dumps", "loads", "load", "dump",
+           "BppError", "__version__"]
 
 dumps = encode
 loads = decode