llguidance 0.7.10__tar.gz → 0.7.11__tar.gz

llguidance-0.7.11/CHANGELOG.md

@@ -0,0 +1,113 @@
+### Changelog
+
+All notable changes to this project will be documented in this file. Dates are displayed in UTC.
+
+If a release doesn't introduce any interesting changes (build fixes etc.), it's skipped.
+
+#### [0.7.11](https://github.com/guidance-ai/llguidance/compare/v0.7.10...0.7.11) 2025-03-27
+
+- add StructTag python API; fixes [`#146`](https://github.com/guidance-ai/llguidance/issues/146)
+- fix handling of AddedToken.special (gemma tokenizer, fixes [`#147`](https://github.com/guidance-ai/llguidance/issues/147))
+- handle incomplete tokenizers (SmolLM); fixes [`#138`](https://github.com/guidance-ai/llguidance/issues/138)
+- fix `validate_token_raw([EOS])` bug
+
+#### [v0.7.10](https://github.com/guidance-ai/llguidance/compare/v0.7.9...v0.7.10) 2025-03-25
+
+- add `llg_matcher_*()` functions to C interface [`#145`](https://github.com/guidance-ai/llguidance/pull/145)
+- always pass validation of grammars with special tokens without tokenizer [`0892a2a`](https://github.com/guidance-ai/llguidance/commit/0892a2adb5c8d818c025fe554bd67f05a5770aa7)
+
+#### [v0.7.9](https://github.com/guidance-ai/llguidance/compare/v0.7.8...v0.7.9) 2025-03-24
+
+- improve Python `LLMatcher.validate_grammar()` [`6b5f5ed`](https://github.com/guidance-ai/llguidance/commit/6b5f5eda7ca85ae2ca9a76c3813a0162a8b99b45)
+
+#### [v0.7.6](https://github.com/guidance-ai/llguidance/compare/v0.7.5...v0.7.6) 2025-03-21
+
+- Stabilize JSON schema property order [`#134`](https://github.com/guidance-ai/llguidance/pull/134)
+
+#### [v0.7.5](https://github.com/guidance-ai/llguidance/compare/v0.7.4...v0.7.5) 2025-03-21
+
+- add toktrie_hf_downloader crate [`11bea00`](https://github.com/guidance-ai/llguidance/commit/11bea00ecd1ef3c4a8970c1748db829e0c8a14de)
+- use hf tokenizers library in python ext [`61728be`](https://github.com/guidance-ai/llguidance/commit/61728be47828525e959f6db226a0f17a783442bc)
+- add LLTokenizer.tokenize_partial [`893cedf`](https://github.com/guidance-ai/llguidance/commit/893cedf614e234bd86bf01a99772d846b6ea884b)
+
+#### [v0.7.4](https://github.com/guidance-ai/llguidance/compare/v0.7.3...v0.7.4) 2025-03-20
+
+- fix gbnf parsing [`e5828b8`](https://github.com/guidance-ai/llguidance/commit/e5828b8a7a2fffaa9cf1aa2619c603a3d4ec7e17)
+
+#### [v0.7.3](https://github.com/guidance-ai/llguidance/compare/v0.7.2...v0.7.3) 2025-03-19
+
+- add LLMatcher.validate_grammar(); make it never raise [`0f8ec60`](https://github.com/guidance-ai/llguidance/commit/0f8ec6088a28eda13c2dd3d537733c0648e00cb3)
+- add LLMatcher.reset() [`6a70aa7`](https://github.com/guidance-ai/llguidance/commit/6a70aa7efa8121fcd1865cefa9998926852eee25)
+
+#### [v0.7.2](https://github.com/guidance-ai/llguidance/compare/v0.7.1...v0.7.2) 2025-03-18
+
+- don't go into error state on final EOS [`1f0f21d`](https://github.com/guidance-ai/llguidance/commit/1f0f21d41fe88427d065b09414047d76b8b32041)
+
+#### [v0.7.1](https://github.com/guidance-ai/llguidance/compare/v0.7.0...v0.7.1) 2025-03-18
+
+- add `LLMatcher` interface in python
+- add  whitespace_pattern to JsonCompileOptions [`04a5491`](https://github.com/guidance-ai/llguidance/commit/04a54912cf6d082669674340833f06385f7b66f8)
+- enable mypy in CI [`#140`](https://github.com/guidance-ai/llguidance/pull/140)
+- add py.typed for annotations information [`#139`](https://github.com/guidance-ai/llguidance/pull/139)
+- fix clippy warnings
+
+#### [v0.7.0](https://github.com/guidance-ai/llguidance/compare/v0.6.31...v0.7.0) 2025-03-07
+
+- remove JSON-based grammar serialization [`#133`](https://github.com/guidance-ai/llguidance/pull/133)
+
+#### [v0.6.31](https://github.com/guidance-ai/llguidance/compare/v0.6.29...v0.6.31) 2025-03-05
+
+- fix https://github.com/guidance-ai/guidance/issues/1131 - backtracking+prompt healing [`#1131`](https://github.com/guidance-ai/guidance/issues/1131)
+- optimize substring [`9950600`](https://github.com/guidance-ai/llguidance/commit/9950600f46e433b4c42506f8816f61cee331774f)
+
+
+#### [v0.6.29](https://github.com/guidance-ai/llguidance/compare/v0.6.28...v0.6.29) 2025-02-25
+
+- [JSON] "x-guidance" JsonCompileOptions [`#130`](https://github.com/guidance-ai/llguidance/pull/130)
+
+#### [v0.6.28](https://github.com/guidance-ai/llguidance/compare/v0.6.27...v0.6.28) 2025-02-21
+
+- support for rollback() [`#126`](https://github.com/guidance-ai/llguidance/pull/126)
+- allow lexer to produce alternative lexemes [`#124`](https://github.com/guidance-ai/llguidance/pull/124)
+- make tokenize_with_greedy_fallback() handle invalid UTF not only at the end [`4762895`](https://github.com/guidance-ai/llguidance/commit/476289558d7d1edefe42eb87a093865debae8129)
+- rise default lexer state limit from 50k to 250k [`202d3d5`](https://github.com/guidance-ai/llguidance/commit/202d3d545c14c63a62017b228c424a603619eb2a)
+
+#### [v0.6.27](https://github.com/guidance-ai/llguidance/compare/v0.6.26...v0.6.27) 2025-02-18
+
+- fix #122: captures with nullable symbols [`#122`](https://github.com/guidance-ai/llguidance/issues/122)
+
+#### [v0.6.26](https://github.com/guidance-ai/llguidance/compare/v0.6.25...v0.6.26) 2025-02-14
+
+- Extend Token::Number to match floats in scientific notation [`#121`](https://github.com/guidance-ai/llguidance/pull/121)
+- native [suffix=...] support [`6d648c7`](https://github.com/guidance-ai/llguidance/commit/6d648c748bed4d83db28ed96ea87ad40ea51bc7e)
+
+#### [v0.6.25](https://github.com/guidance-ai/llguidance/compare/v0.6.16...v0.6.25) 2025-02-12
+
+- update referencing to 0.29.0 [`#118`](https://github.com/guidance-ai/llguidance/pull/118)
+- Allow passing string for `capture_name` in lark syntax [`#119`](https://github.com/guidance-ai/llguidance/pull/119)
+
+Plus a few releases messing with, deps, unsafe code cleanup.
+
+#### [v0.6.16](https://github.com/guidance-ai/llguidance/compare/v0.6.15...v0.6.16) 2025-02-06
+
+- Port over guidance's 'substring' [`#116`](https://github.com/guidance-ai/llguidance/pull/116)
+- add %regex {...} syntax in lark for substrings [`b5ab086`](https://github.com/guidance-ai/llguidance/commit/b5ab0861e819b6e9221ef0aed3fcc827d6bad316)
+
+#### [v0.6.15](https://github.com/guidance-ai/llguidance/compare/v0.6.14...v0.6.15) 2025-02-04
+
+- move gbnf_to_lark to llguidance python pkg [`53134f1`](https://github.com/guidance-ai/llguidance/commit/53134f1befc6b6019bc88406e21b51c901943b51)
+- add LLExecutor and fill_next_token_bitmask_par() [`ba4b917`](https://github.com/guidance-ai/llguidance/commit/ba4b9175b8d6c5445e1c0bcc8d5ef8e62b6cf73c)
+
+#### [v0.6.14](https://github.com/guidance-ai/llguidance/compare/v0.6.13...v0.6.14) 2025-02-03
+
+- add llguidance.numpy and llguidance.mlx submodules [`c627a39`](https://github.com/guidance-ai/llguidance/commit/c627a39689c9147fe7b072e5075960d16d43fc73)
+
+#### [v0.6.13](https://github.com/guidance-ai/llguidance/compare/v0.6.12...v0.6.13) 2025-02-03
+
+- add llguidance.torch and llguidance.hf submodules [`3fcdb1d`](https://github.com/guidance-ai/llguidance/commit/3fcdb1d93af076bbc8f1b3bef6fa9ead22b3e959)
+
+#### [v0.6.12](https://github.com/guidance-ai/llguidance/compare/v0.6.11...v0.6.12) 2025-01-31
+
+- fixes for numeric tokens [`b7c9970`](https://github.com/guidance-ai/llguidance/commit/b7c99709a9cb7f7a8a3c4716092e4d94fae2ff2c)
+- make capture explicit in lark syntax [`2a57678`](https://github.com/guidance-ai/llguidance/commit/2a57678d9397e8be54cb0c9f14c4270604f8e1a5)
+

{llguidance-0.7.10 → llguidance-0.7.11}/Cargo.lock

@@ -1177,7 +1177,7 @@ checksum = "4ee93343901ab17bd981295f2cf0026d4ad018c7c31ba84549a4ddbb47a45104"
 
 [[package]]
 name = "llguidance"
-version = "0.7.10"
+version = "0.7.11"
 dependencies = [
  "anyhow",
  "derivre",
@@ -1196,7 +1196,7 @@ dependencies = [
 
 [[package]]
 name = "llguidance_py"
-version = "0.7.10"
+version = "0.7.11"
 dependencies = [
  "anyhow",
  "bytemuck",
@@ -2356,7 +2356,7 @@ dependencies = [
 
 [[package]]
 name = "toktrie"
-version = "0.7.10"
+version = "0.7.11"
 dependencies = [
  "anyhow",
  "bytemuck",
@@ -2367,7 +2367,7 @@ dependencies = [
 
 [[package]]
 name = "toktrie_hf_downloader"
-version = "0.7.10"
+version = "0.7.11"
 dependencies = [
  "anyhow",
  "hf-hub",
@@ -2378,7 +2378,7 @@ dependencies = [
 
 [[package]]
 name = "toktrie_hf_tokenizers"
-version = "0.7.10"
+version = "0.7.11"
 dependencies = [
  "anyhow",
  "log",

{llguidance-0.7.10 → llguidance-0.7.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: llguidance
-Version: 0.7.10
+Version: 0.7.11
 License-File: LICENSE
 Summary: Bindings for the Low-level Guidance (llguidance) Rust library for use within Guidance
 Author: Michal Moskal

{llguidance-0.7.10 → llguidance-0.7.11}/docs/json_schema.md

@@ -56,6 +56,41 @@ Number features (for both integer and number):
 - order of object properties is fixed, see below
 - string `format` is enforced by default, with unrecognized or unimplemented formats returning errors
 
+## Whitespace handling
+
+By default any whitespace is allowed inside of the JSON object.
+Whitespace is not allowed before the first `{` or after the last `}`.
+You can modify your grammar easily to allow initial or trailing whitespace.
+
+You can set top-level `"x-guidance"` key to control this.
+Following keys are available inside of it:
+
+- `item_separator`, defaults to `":"`
+- `key_separator`, defaults to `","`
+- `whitespace_flexible`, defaults to `true`; set to `false` to enforce compact JSON representation
+- `whitespace_pattern`, optional string, overrides `whitespace_flexible`;
+  `whitespace_flexible: true` is equivalent to `whitespace_pattern: r"[\x20\x0A\x0D\x09]+"`
+- `coerce_one_of`, defaults to `false`; when set to `true`, the `"oneOf"` will be treated as `"anyOf"`
+
+For example:
+
+```json
+{
+   "x-guidance": {
+      "whitespace_flexible": false
+   },
+   "type": "object",
+   "properties": {
+      "a": {
+         "type": "string"
+      }
+   }
+}
+```
+
+The `"x-guidance"` key is only recognized at the top level of the schema.
+
+
 ## Property order
 
 ### TL;DR

{llguidance-0.7.10 → llguidance-0.7.11}/docs/syntax.md

@@ -116,8 +116,7 @@ Often, the chat format already includes initial `<think>\n` - in these cases
 you can use `start: /(.|\n)*/ </think> json` as the grammar.
 You can also use `/(.|\n){1000,3000}/` to place lower and upper bounds on the thinking amount.
 
-
-This assumes `<think>` is a special token. If it was just a string, you would need 
+This assumes `<think>` is a special token. If it was just a string, you would need
 to use [`suffix="</think>"`](#lazy-lexemes).
 
 ### Lexeme options
@@ -135,7 +134,7 @@ Example: `mygen[stop="\n", max_tokens=10, temperature=0.7]: /.*/`
 The `temperature` alters temperature while sampling tokens inside of the terminal,
 while `max_tokens` limits the number of tokens generated for the terminal.
 
-#### Lazy lexemes
+### Lazy lexemes
 
 Specifying `stop=""` will make the EOS token of the model act as the stop condition.
 This is only useful if there is some other rule following this rule (otherwise the model will stop on EOS anyways),
@@ -187,20 +186,36 @@ If it doesn't, the results may be surprising:
 for example, `foo[lazy]: /.*/` will match only the empty string,
 while `foo[lazy]: /[0-9]+/` will only match a single digit.
 
-Typical use of `suffix` or `lazy` is for models that were finetuned for reasoning or
-tool calling without special tokens, but with special strings. For example:
+### Tool calling
+
+Some models have been finetuned to use normal text, not special tokens, for tool calling.
+Here's an example how `lazy` can be used to allow any text interspersed with
+tool calls:
 
 ```lark
-start: "<tool_name>" name "<tool_data>" data "</tool_data>"
-name[capture, suffix="</tool_name>"]: /.*/
-data[capture]: %json {
-    "properties": {
-        "foo": { "type": "string" }
-    },
-    "required": ["foo"]
-}
+start: ( f_foo | f_bar )* f_end
+f_end: TEXT
+TEXT: /(.|\n)*/
+
+f_foo_hd[lazy]: TEXT "<function"
+f_foo: f_foo_hd "=foo>" %json { "type": "object" } "</function>"
+
+f_bar_hd[lazy]: TEXT "<function"
+f_bar: f_bar_hd "=bar>" /[0-9]+/ "</function>"
 ```
 
+The above allows for anything, until the text `<function` is generated.
+Then it will force the model to choose between `=foo>` or `=bar>`.
+Then we'll either enforce valid JSON or a number, followed by string `</function>`.
+Then again any text is allowed, etc.
+
+Note that the tool call can be missing algtogether, you can change `*` to `+` if you want to enforce at least one tool call.
+You can also change the definition of `TEXT` to limit the free-form text.
+
+See [Special tokens](#special-tokens) if the "trigger" for the tool call is a special token
+like `<|python_tag|>`, not a string like `<function`.
+
+The `llguidance.StructTag` API, [inspired](https://github.com/mlc-ai/xgrammar/blob/fd9ee31/python/xgrammar/grammar.py#L211) by XGrammar, just compiles down to the above.
 
 ### Structured %regex
 
@@ -221,8 +236,6 @@ Additionally, "structured" regex nodes can be defined using `%regex { ... }` syn
 
 #### Substring
 
-**The syntax is not stable yet!**
-
 `%regex { "substring_chunks": lst }` will match `lst[n:m].join("")` for some `n <= m <= len(lst)`.
 Additionally `substring_words` or `substring_chars` can be specified.
 For example: