llguidance 0.7.10__tar.gz → 0.7.12__tar.gz

llguidance-0.7.12/CHANGELOG.md

@@ -0,0 +1,118 @@
+### Changelog
+
+All notable changes to this project will be documented in this file. Dates are displayed in UTC.
+
+If a release doesn't introduce any interesting changes (build fixes etc.), it's skipped.
+
+#### [0.7.12](https://github.com/guidance-ai/llguidance/compare/v0.7.11...0.7.12) 2025-04-04
+
+- performance optimizations
+- use factory in C FFI (otherwise slicer was not used)
+- add some null checks and safety comments in C FFI
+- implement subgrammar lexeme class merging; fixes [`#113`](https://github.com/guidance-ai/llguidance/issues/113)
+
+#### [0.7.11](https://github.com/guidance-ai/llguidance/compare/v0.7.10...0.7.11) 2025-03-27
+
+- add StructTag python API; fixes [`#146`](https://github.com/guidance-ai/llguidance/issues/146)
+- fix handling of AddedToken.special (gemma tokenizer, fixes [`#147`](https://github.com/guidance-ai/llguidance/issues/147))
+- handle incomplete tokenizers (SmolLM); fixes [`#138`](https://github.com/guidance-ai/llguidance/issues/138)
+- fix `validate_token_raw([EOS])` bug
+
+#### [v0.7.10](https://github.com/guidance-ai/llguidance/compare/v0.7.9...v0.7.10) 2025-03-25
+
+- add `llg_matcher_*()` functions to C interface [`#145`](https://github.com/guidance-ai/llguidance/pull/145)
+- always pass validation of grammars with special tokens without tokenizer [`0892a2a`](https://github.com/guidance-ai/llguidance/commit/0892a2adb5c8d818c025fe554bd67f05a5770aa7)
+
+#### [v0.7.9](https://github.com/guidance-ai/llguidance/compare/v0.7.8...v0.7.9) 2025-03-24
+
+- improve Python `LLMatcher.validate_grammar()` [`6b5f5ed`](https://github.com/guidance-ai/llguidance/commit/6b5f5eda7ca85ae2ca9a76c3813a0162a8b99b45)
+
+#### [v0.7.6](https://github.com/guidance-ai/llguidance/compare/v0.7.5...v0.7.6) 2025-03-21
+
+- Stabilize JSON schema property order [`#134`](https://github.com/guidance-ai/llguidance/pull/134)
+
+#### [v0.7.5](https://github.com/guidance-ai/llguidance/compare/v0.7.4...v0.7.5) 2025-03-21
+
+- add toktrie_hf_downloader crate [`11bea00`](https://github.com/guidance-ai/llguidance/commit/11bea00ecd1ef3c4a8970c1748db829e0c8a14de)
+- use hf tokenizers library in python ext [`61728be`](https://github.com/guidance-ai/llguidance/commit/61728be47828525e959f6db226a0f17a783442bc)
+- add LLTokenizer.tokenize_partial [`893cedf`](https://github.com/guidance-ai/llguidance/commit/893cedf614e234bd86bf01a99772d846b6ea884b)
+
+#### [v0.7.4](https://github.com/guidance-ai/llguidance/compare/v0.7.3...v0.7.4) 2025-03-20
+
+- fix gbnf parsing [`e5828b8`](https://github.com/guidance-ai/llguidance/commit/e5828b8a7a2fffaa9cf1aa2619c603a3d4ec7e17)
+
+#### [v0.7.3](https://github.com/guidance-ai/llguidance/compare/v0.7.2...v0.7.3) 2025-03-19
+
+- add LLMatcher.validate_grammar(); make it never raise [`0f8ec60`](https://github.com/guidance-ai/llguidance/commit/0f8ec6088a28eda13c2dd3d537733c0648e00cb3)
+- add LLMatcher.reset() [`6a70aa7`](https://github.com/guidance-ai/llguidance/commit/6a70aa7efa8121fcd1865cefa9998926852eee25)
+
+#### [v0.7.2](https://github.com/guidance-ai/llguidance/compare/v0.7.1...v0.7.2) 2025-03-18
+
+- don't go into error state on final EOS [`1f0f21d`](https://github.com/guidance-ai/llguidance/commit/1f0f21d41fe88427d065b09414047d76b8b32041)
+
+#### [v0.7.1](https://github.com/guidance-ai/llguidance/compare/v0.7.0...v0.7.1) 2025-03-18
+
+- add `LLMatcher` interface in python
+- add whitespace_pattern to JsonCompileOptions [`04a5491`](https://github.com/guidance-ai/llguidance/commit/04a54912cf6d082669674340833f06385f7b66f8)
+- enable mypy in CI [`#140`](https://github.com/guidance-ai/llguidance/pull/140)
+- add py.typed for annotations information [`#139`](https://github.com/guidance-ai/llguidance/pull/139)
+- fix clippy warnings
+
+#### [v0.7.0](https://github.com/guidance-ai/llguidance/compare/v0.6.31...v0.7.0) 2025-03-07
+
+- remove JSON-based grammar serialization [`#133`](https://github.com/guidance-ai/llguidance/pull/133)
+
+#### [v0.6.31](https://github.com/guidance-ai/llguidance/compare/v0.6.29...v0.6.31) 2025-03-05
+
+- fix https://github.com/guidance-ai/guidance/issues/1131 - backtracking+prompt healing [`#1131`](https://github.com/guidance-ai/guidance/issues/1131)
+- optimize substring [`9950600`](https://github.com/guidance-ai/llguidance/commit/9950600f46e433b4c42506f8816f61cee331774f)
+
+#### [v0.6.29](https://github.com/guidance-ai/llguidance/compare/v0.6.28...v0.6.29) 2025-02-25
+
+- [JSON] "x-guidance" JsonCompileOptions [`#130`](https://github.com/guidance-ai/llguidance/pull/130)
+
+#### [v0.6.28](https://github.com/guidance-ai/llguidance/compare/v0.6.27...v0.6.28) 2025-02-21
+
+- support for rollback() [`#126`](https://github.com/guidance-ai/llguidance/pull/126)
+- allow lexer to produce alternative lexemes [`#124`](https://github.com/guidance-ai/llguidance/pull/124)
+- make tokenize_with_greedy_fallback() handle invalid UTF not only at the end [`4762895`](https://github.com/guidance-ai/llguidance/commit/476289558d7d1edefe42eb87a093865debae8129)
+- rise default lexer state limit from 50k to 250k [`202d3d5`](https://github.com/guidance-ai/llguidance/commit/202d3d545c14c63a62017b228c424a603619eb2a)
+
+#### [v0.6.27](https://github.com/guidance-ai/llguidance/compare/v0.6.26...v0.6.27) 2025-02-18
+
+- fix #122: captures with nullable symbols [`#122`](https://github.com/guidance-ai/llguidance/issues/122)
+
+#### [v0.6.26](https://github.com/guidance-ai/llguidance/compare/v0.6.25...v0.6.26) 2025-02-14
+
+- Extend Token::Number to match floats in scientific notation [`#121`](https://github.com/guidance-ai/llguidance/pull/121)
+- native [suffix=...] support [`6d648c7`](https://github.com/guidance-ai/llguidance/commit/6d648c748bed4d83db28ed96ea87ad40ea51bc7e)
+
+#### [v0.6.25](https://github.com/guidance-ai/llguidance/compare/v0.6.16...v0.6.25) 2025-02-12
+
+- update referencing to 0.29.0 [`#118`](https://github.com/guidance-ai/llguidance/pull/118)
+- Allow passing string for `capture_name` in lark syntax [`#119`](https://github.com/guidance-ai/llguidance/pull/119)
+
+Plus a few releases messing with, deps, unsafe code cleanup.
+
+#### [v0.6.16](https://github.com/guidance-ai/llguidance/compare/v0.6.15...v0.6.16) 2025-02-06
+
+- Port over guidance's 'substring' [`#116`](https://github.com/guidance-ai/llguidance/pull/116)
+- add %regex {...} syntax in lark for substrings [`b5ab086`](https://github.com/guidance-ai/llguidance/commit/b5ab0861e819b6e9221ef0aed3fcc827d6bad316)
+
+#### [v0.6.15](https://github.com/guidance-ai/llguidance/compare/v0.6.14...v0.6.15) 2025-02-04
+
+- move gbnf_to_lark to llguidance python pkg [`53134f1`](https://github.com/guidance-ai/llguidance/commit/53134f1befc6b6019bc88406e21b51c901943b51)
+- add LLExecutor and fill_next_token_bitmask_par() [`ba4b917`](https://github.com/guidance-ai/llguidance/commit/ba4b9175b8d6c5445e1c0bcc8d5ef8e62b6cf73c)
+
+#### [v0.6.14](https://github.com/guidance-ai/llguidance/compare/v0.6.13...v0.6.14) 2025-02-03
+
+- add llguidance.numpy and llguidance.mlx submodules [`c627a39`](https://github.com/guidance-ai/llguidance/commit/c627a39689c9147fe7b072e5075960d16d43fc73)
+
+#### [v0.6.13](https://github.com/guidance-ai/llguidance/compare/v0.6.12...v0.6.13) 2025-02-03
+
+- add llguidance.torch and llguidance.hf submodules [`3fcdb1d`](https://github.com/guidance-ai/llguidance/commit/3fcdb1d93af076bbc8f1b3bef6fa9ead22b3e959)
+
+#### [v0.6.12](https://github.com/guidance-ai/llguidance/compare/v0.6.11...v0.6.12) 2025-01-31
+
+- fixes for numeric tokens [`b7c9970`](https://github.com/guidance-ai/llguidance/commit/b7c99709a9cb7f7a8a3c4716092e4d94fae2ff2c)
+- make capture explicit in lark syntax [`2a57678`](https://github.com/guidance-ai/llguidance/commit/2a57678d9397e8be54cb0c9f14c4270604f8e1a5)

{llguidance-0.7.10 → llguidance-0.7.12}/Cargo.lock

@@ -401,9 +401,9 @@ dependencies = [
 
 [[package]]
 name = "derivre"
-version = "0.3.1"
+version = "0.3.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3a3c2606b3ffc46f91fd62d954d55659ba9fb391bb673311b70f50daf9c15e49"
+checksum = "310c9990c5a531352e274c8c929ca667a84b6bbaceb1e095c177e6a979807f57"
 dependencies = [
  "ahash",
  "anyhow",
@@ -1177,7 +1177,7 @@ checksum = "4ee93343901ab17bd981295f2cf0026d4ad018c7c31ba84549a4ddbb47a45104"
 
 [[package]]
 name = "llguidance"
-version = "0.7.10"
+version = "0.7.12"
 dependencies = [
  "anyhow",
  "derivre",
@@ -1196,7 +1196,7 @@ dependencies = [
 
 [[package]]
 name = "llguidance_py"
-version = "0.7.10"
+version = "0.7.12"
 dependencies = [
  "anyhow",
  "bytemuck",
@@ -2356,7 +2356,7 @@ dependencies = [
 
 [[package]]
 name = "toktrie"
-version = "0.7.10"
+version = "0.7.12"
 dependencies = [
  "anyhow",
  "bytemuck",
@@ -2367,7 +2367,7 @@ dependencies = [
 
 [[package]]
 name = "toktrie_hf_downloader"
-version = "0.7.10"
+version = "0.7.12"
 dependencies = [
  "anyhow",
  "hf-hub",
@@ -2378,7 +2378,7 @@ dependencies = [
 
 [[package]]
 name = "toktrie_hf_tokenizers"
-version = "0.7.10"
+version = "0.7.12"
 dependencies = [
  "anyhow",
  "log",

{llguidance-0.7.10 → llguidance-0.7.12}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: llguidance
-Version: 0.7.10
+Version: 0.7.12
 License-File: LICENSE
 Summary: Bindings for the Low-level Guidance (llguidance) Rust library for use within Guidance
 Author: Michal Moskal

{llguidance-0.7.10 → llguidance-0.7.12}/docs/json_schema.md

@@ -56,6 +56,41 @@ Number features (for both integer and number):
 - order of object properties is fixed, see below
 - string `format` is enforced by default, with unrecognized or unimplemented formats returning errors
 
+## Whitespace handling
+
+By default any whitespace is allowed inside of the JSON object.
+Whitespace is not allowed before the first `{` or after the last `}`.
+You can modify your grammar easily to allow initial or trailing whitespace.
+
+You can set top-level `"x-guidance"` key to control this.
+Following keys are available inside of it:
+
+- `item_separator`, defaults to `":"`
+- `key_separator`, defaults to `","`
+- `whitespace_flexible`, defaults to `true`; set to `false` to enforce compact JSON representation
+- `whitespace_pattern`, optional string, overrides `whitespace_flexible`;
+  `whitespace_flexible: true` is equivalent to `whitespace_pattern: r"[\x20\x0A\x0D\x09]+"`
+- `coerce_one_of`, defaults to `false`; when set to `true`, the `"oneOf"` will be treated as `"anyOf"`
+
+For example:
+
+```json
+{
+   "x-guidance": {
+      "whitespace_flexible": false
+   },
+   "type": "object",
+   "properties": {
+      "a": {
+         "type": "string"
+      }
+   }
+}
+```
+
+The `"x-guidance"` key is only recognized at the top level of the schema.
+
+
 ## Property order
 
 ### TL;DR

{llguidance-0.7.10 → llguidance-0.7.12}/docs/syntax.md

@@ -116,8 +116,7 @@ Often, the chat format already includes initial `<think>\n` - in these cases
 you can use `start: /(.|\n)*/ </think> json` as the grammar.
 You can also use `/(.|\n){1000,3000}/` to place lower and upper bounds on the thinking amount.
 
-
-This assumes `<think>` is a special token. If it was just a string, you would need 
+This assumes `<think>` is a special token. If it was just a string, you would need
 to use [`suffix="</think>"`](#lazy-lexemes).
 
 ### Lexeme options
@@ -135,7 +134,7 @@ Example: `mygen[stop="\n", max_tokens=10, temperature=0.7]: /.*/`
 The `temperature` alters temperature while sampling tokens inside of the terminal,
 while `max_tokens` limits the number of tokens generated for the terminal.
 
-#### Lazy lexemes
+### Lazy lexemes
 
 Specifying `stop=""` will make the EOS token of the model act as the stop condition.
 This is only useful if there is some other rule following this rule (otherwise the model will stop on EOS anyways),
@@ -187,20 +186,36 @@ If it doesn't, the results may be surprising:
 for example, `foo[lazy]: /.*/` will match only the empty string,
 while `foo[lazy]: /[0-9]+/` will only match a single digit.
 
-Typical use of `suffix` or `lazy` is for models that were finetuned for reasoning or
-tool calling without special tokens, but with special strings. For example:
+### Tool calling
+
+Some models have been finetuned to use normal text, not special tokens, for tool calling.
+Here's an example how `lazy` can be used to allow any text interspersed with
+tool calls:
 
 ```lark
-start: "<tool_name>" name "<tool_data>" data "</tool_data>"
-name[capture, suffix="</tool_name>"]: /.*/
-data[capture]: %json {
-    "properties": {
-        "foo": { "type": "string" }
-    },
-    "required": ["foo"]
-}
+start: ( f_foo | f_bar )* f_end
+f_end: TEXT
+TEXT: /(.|\n)*/
+
+f_foo_hd[lazy]: TEXT "<function"
+f_foo: f_foo_hd "=foo>" %json { "type": "object" } "</function>"
+
+f_bar_hd[lazy]: TEXT "<function"
+f_bar: f_bar_hd "=bar>" /[0-9]+/ "</function>"
 ```
 
+The above allows for anything, until the text `<function` is generated.
+Then it will force the model to choose between `=foo>` or `=bar>`.
+Then we'll either enforce valid JSON or a number, followed by string `</function>`.
+Then again any text is allowed, etc.
+
+Note that the tool call can be missing algtogether, you can change `*` to `+` if you want to enforce at least one tool call.
+You can also change the definition of `TEXT` to limit the free-form text.
+
+See [Special tokens](#special-tokens) if the "trigger" for the tool call is a special token
+like `<|python_tag|>`, not a string like `<function`.
+
+The `llguidance.StructTag` API, [inspired](https://github.com/mlc-ai/xgrammar/blob/fd9ee31/python/xgrammar/grammar.py#L211) by XGrammar, just compiles down to the above.
 
 ### Structured %regex
 
@@ -221,8 +236,6 @@ Additionally, "structured" regex nodes can be defined using `%regex { ... }` syn
 
 #### Substring
 
-**The syntax is not stable yet!**
-
 `%regex { "substring_chunks": lst }` will match `lst[n:m].join("")` for some `n <= m <= len(lst)`.
 Additionally `substring_words` or `substring_chars` can be specified.
 For example:
@@ -280,12 +293,13 @@ that llguidance should be used to process the grammar.
 ### Multiple grammars
 
 The input to LLGuidance consists of a list of grammars. This can be accessed via
-[LLGuidance API](../parser/src/api.rs). Each of these can be a Lark grammar, a JSON schema,
-or a grammar in the API format. With the introduction of `%json` in Lark syntax
+[LLGuidance API](../parser/src/api.rs). Each of these can be a Lark grammar or a JSON schema.
+With the introduction of `%json` in Lark syntax
 there is less need now for using multiple grammars, but it is still supported.
+We may add nested lark grammars in future.
 
 Inside of Lark grammar, you can reference other grammars using syntax like `@my_grammar`,
-refering to grammar with `"name": "my_grammar"` (numeric reference like `@17` are no longer supported).
+refering to grammar with `"name": "my_grammar"` (numeric reference like `@17` are **no longer supported**).
 The top-level grammar is at index 0.
 
 You can specify temperature for subgrammar by referencing it via
@@ -304,6 +318,42 @@ Example:
 }
 ```
 
+#### Subgrammar details
+
+Generally, subgrammars share the same context-free grammar but have a separate
+set of lexemes (lexeme class).
+The parser keeps track of a stack of lexeme classes, and considers the top one
+to be the current lexeme class.
+The `%ignore` is applied based on the top lexeme class.
+
+Temperature and `max_tokens` can be applied to the current lexeme class as well.
+
+There may be issues with subgrammars spanning LLM token boundaries.
+
+If two lexeme classes share the `%ignore` regex, and `max_tokens=` and `temperature=`
+are not used, the lexeme classes are merged, which generally allows for parsing
+of more grammars.
+For example, consider:
+
+```lark
+start: a | b
+a: %json { A }
+b: %json { B }
+```
+
+Normally, the parser would have to pick between lexeme class for either A or B
+at the first `{` (it would always pick A since it comes first in the grammar).
+However, if the classes for A and B are merged, the grammar will be equivalent to
+`start: %json { "anyOf": [A, B] }` which is generally what the
+[users expect](https://github.com/guidance-ai/llguidance/issues/113).
+
+
+### Features to avoid
+
+- `stop=...` - use `suffix=...` or just `lazy`
+- `max_tokens=...` - any use of `max_tokens` will disable rollback, which is needed for spec-decoding; it also makes the parser slower and prevents subgrammar merging
+- `temperature=...` - this is not supported in most server side integrations and prevents subgrammar merging
+
 ### Unsupported Lark features
 
 Following features of Lark syntax are currently not supported: