llguidance 0.7.12__tar.gz → 0.7.14__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (174) hide show
  1. {llguidance-0.7.12 → llguidance-0.7.14}/CHANGELOG.md +14 -0
  2. {llguidance-0.7.12 → llguidance-0.7.14}/Cargo.lock +286 -192
  3. {llguidance-0.7.12 → llguidance-0.7.14}/PKG-INFO +1 -1
  4. {llguidance-0.7.12 → llguidance-0.7.14}/SUPPORT.md +13 -13
  5. {llguidance-0.7.12 → llguidance-0.7.14}/json_stats/expected_maskbench.json +25 -57
  6. {llguidance-0.7.12 → llguidance-0.7.14}/json_stats/src/json_stats.rs +4 -12
  7. {llguidance-0.7.12 → llguidance-0.7.14}/json_stats/src/stats.rs +1 -7
  8. {llguidance-0.7.12 → llguidance-0.7.14}/parser/Cargo.toml +2 -2
  9. {llguidance-0.7.12 → llguidance-0.7.14}/parser/llguidance.h +8 -0
  10. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/api.rs +10 -2
  11. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/earley/from_guidance.rs +14 -6
  12. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/earley/grammar.rs +20 -18
  13. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/earley/lexer.rs +33 -1
  14. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/earley/lexerspec.rs +2 -1
  15. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/earley/parser.rs +33 -9
  16. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/earley/perf.rs +3 -0
  17. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/earley/regexvec.rs +5 -0
  18. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/factory.rs +8 -1
  19. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/ffi.rs +1 -1
  20. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/json/compiler.rs +1 -0
  21. llguidance-0.7.14/parser/src/json/formats.rs +57 -0
  22. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/json/schema.rs +3 -21
  23. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/lark/ast.rs +14 -10
  24. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/lark/compiler.rs +82 -81
  25. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/lark/lexer.rs +2 -0
  26. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/lark/mod.rs +1 -1
  27. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/lark/parser.rs +73 -17
  28. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/lib.rs +3 -0
  29. llguidance-0.7.14/parser/src/regex_rewrite.rs +130 -0
  30. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/tokenparser.rs +6 -1
  31. {llguidance-0.7.12 → llguidance-0.7.14}/pyproject.toml +1 -1
  32. {llguidance-0.7.12 → llguidance-0.7.14}/python/llguidance/__init__.py +4 -0
  33. {llguidance-0.7.12 → llguidance-0.7.14}/python/llguidance/_lib.pyi +118 -15
  34. {llguidance-0.7.12 → llguidance-0.7.14}/python/llguidance/cli.py +4 -2
  35. {llguidance-0.7.12 → llguidance-0.7.14}/python/llguidance/hf.py +4 -4
  36. {llguidance-0.7.12 → llguidance-0.7.14}/python/torch_tests/test_hf.py +17 -17
  37. {llguidance-0.7.12 → llguidance-0.7.14}/python/torch_tests/test_matcher.py +31 -7
  38. {llguidance-0.7.12 → llguidance-0.7.14}/python_ext/Cargo.toml +2 -2
  39. {llguidance-0.7.12 → llguidance-0.7.14}/python_ext/src/lib.rs +2 -0
  40. {llguidance-0.7.12 → llguidance-0.7.14}/python_ext/src/llinterpreter.rs +7 -4
  41. {llguidance-0.7.12 → llguidance-0.7.14}/python_ext/src/llmatcher.rs +12 -8
  42. llguidance-0.7.14/python_ext/src/parserlimits.rs +106 -0
  43. {llguidance-0.7.12 → llguidance-0.7.14}/python_ext/src/py.rs +11 -4
  44. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/src/lib.rs +15 -1
  45. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/src/sample_parser.rs +19 -0
  46. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/tests/test_lark.rs +106 -1
  47. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/tests/test_ll.rs +13 -0
  48. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/bump.py +14 -5
  49. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/test-guidance.sh +5 -1
  50. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie/Cargo.toml +1 -1
  51. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie/src/toktree.rs +1 -1
  52. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie_hf_downloader/Cargo.toml +1 -1
  53. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie_hf_tokenizers/Cargo.toml +1 -1
  54. llguidance-0.7.12/parser/src/json/formats.rs +0 -51
  55. {llguidance-0.7.12 → llguidance-0.7.14}/.github/workflows/rust.yml +0 -0
  56. {llguidance-0.7.12 → llguidance-0.7.14}/.github/workflows/wheels.yml +0 -0
  57. {llguidance-0.7.12 → llguidance-0.7.14}/.gitignore +0 -0
  58. {llguidance-0.7.12 → llguidance-0.7.14}/CODE_OF_CONDUCT.md +0 -0
  59. {llguidance-0.7.12 → llguidance-0.7.14}/Cargo.toml +0 -0
  60. {llguidance-0.7.12 → llguidance-0.7.14}/LICENSE +0 -0
  61. {llguidance-0.7.12 → llguidance-0.7.14}/README.md +0 -0
  62. {llguidance-0.7.12 → llguidance-0.7.14}/SECURITY.md +0 -0
  63. {llguidance-0.7.12 → llguidance-0.7.14}/c_sample/Makefile +0 -0
  64. {llguidance-0.7.12 → llguidance-0.7.14}/c_sample/README.md +0 -0
  65. {llguidance-0.7.12 → llguidance-0.7.14}/c_sample/c_sample.cpp +0 -0
  66. {llguidance-0.7.12 → llguidance-0.7.14}/docs/fast_forward.md +0 -0
  67. {llguidance-0.7.12 → llguidance-0.7.14}/docs/json_schema.md +0 -0
  68. {llguidance-0.7.12 → llguidance-0.7.14}/docs/mask_plot.png +0 -0
  69. {llguidance-0.7.12 → llguidance-0.7.14}/docs/optimizations.md +0 -0
  70. {llguidance-0.7.12 → llguidance-0.7.14}/docs/special_tokens.md +0 -0
  71. {llguidance-0.7.12 → llguidance-0.7.14}/docs/syntax.md +0 -0
  72. {llguidance-0.7.12 → llguidance-0.7.14}/docs/toktrie.md +0 -0
  73. {llguidance-0.7.12 → llguidance-0.7.14}/json_stats/Cargo.toml +0 -0
  74. {llguidance-0.7.12 → llguidance-0.7.14}/json_stats/jstats.sh +0 -0
  75. {llguidance-0.7.12 → llguidance-0.7.14}/json_stats/scripts/split-stats.sh +0 -0
  76. {llguidance-0.7.12 → llguidance-0.7.14}/json_stats/scripts/split_plot.py +0 -0
  77. {llguidance-0.7.12 → llguidance-0.7.14}/json_stats/src/lib.rs +0 -0
  78. {llguidance-0.7.12 → llguidance-0.7.14}/parser/LICENSE +0 -0
  79. {llguidance-0.7.12 → llguidance-0.7.14}/parser/README.md +0 -0
  80. {llguidance-0.7.12 → llguidance-0.7.14}/parser/build.rs +0 -0
  81. {llguidance-0.7.12 → llguidance-0.7.14}/parser/cbindgen.toml +0 -0
  82. {llguidance-0.7.12 → llguidance-0.7.14}/parser/grammars/character.json +0 -0
  83. {llguidance-0.7.12 → llguidance-0.7.14}/parser/grammars/json.json +0 -0
  84. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/constraint.rs +0 -0
  85. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/earley/mod.rs +0 -0
  86. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/earley/slicer.rs +0 -0
  87. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/ffi_par.rs +0 -0
  88. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/grammar_builder.rs +0 -0
  89. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/json/README.md +0 -0
  90. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/json/context_ref.rs +0 -0
  91. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/json/context_simple/context.rs +0 -0
  92. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/json/context_simple/draft.rs +0 -0
  93. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/json/context_simple/mod.rs +0 -0
  94. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/json/mod.rs +0 -0
  95. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/json/numeric.rs +0 -0
  96. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/json/shared_context.rs +0 -0
  97. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/json_validation.rs +0 -0
  98. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/lark/README.md +0 -0
  99. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/lark/common.rs +0 -0
  100. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/logging.rs +0 -0
  101. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/matcher.rs +0 -0
  102. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/output.rs +0 -0
  103. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/panic_utils.rs +0 -0
  104. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/stop_controller.rs +0 -0
  105. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/substring.rs +0 -0
  106. {llguidance-0.7.12 → llguidance-0.7.14}/parser/src/tokenizer_json.rs +0 -0
  107. {llguidance-0.7.12 → llguidance-0.7.14}/plan.md +0 -0
  108. {llguidance-0.7.12 → llguidance-0.7.14}/python/llguidance/_grammar_from.py +0 -0
  109. {llguidance-0.7.12 → llguidance-0.7.14}/python/llguidance/_struct_tag.py +0 -0
  110. {llguidance-0.7.12 → llguidance-0.7.14}/python/llguidance/_tokenizer.py +0 -0
  111. {llguidance-0.7.12 → llguidance-0.7.14}/python/llguidance/_util.py +0 -0
  112. {llguidance-0.7.12 → llguidance-0.7.14}/python/llguidance/gbnf_to_lark.py +0 -0
  113. {llguidance-0.7.12 → llguidance-0.7.14}/python/llguidance/mlx.py +0 -0
  114. {llguidance-0.7.12 → llguidance-0.7.14}/python/llguidance/numpy.py +0 -0
  115. {llguidance-0.7.12 → llguidance-0.7.14}/python/llguidance/py.typed +0 -0
  116. {llguidance-0.7.12 → llguidance-0.7.14}/python/llguidance/torch.py +0 -0
  117. {llguidance-0.7.12 → llguidance-0.7.14}/python/mypy.ini +0 -0
  118. {llguidance-0.7.12 → llguidance-0.7.14}/python/torch_tests/__init__.py +0 -0
  119. {llguidance-0.7.12 → llguidance-0.7.14}/python/torch_tests/test_bitmask.py +0 -0
  120. {llguidance-0.7.12 → llguidance-0.7.14}/python_ext/src/pyjson.rs +0 -0
  121. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/Cargo.toml +0 -0
  122. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/README.md +0 -0
  123. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/cli.sh +0 -0
  124. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/blog.sample.json +0 -0
  125. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/blog.schema.json +0 -0
  126. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/blog.schema.ll.json +0 -0
  127. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/from-llama.cpp/README.md +0 -0
  128. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/from-llama.cpp/arithmetic.gbnf +0 -0
  129. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/from-llama.cpp/c.gbnf +0 -0
  130. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/from-llama.cpp/chess.gbnf +0 -0
  131. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/from-llama.cpp/english.gbnf +0 -0
  132. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/from-llama.cpp/japanese.gbnf +0 -0
  133. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/from-llama.cpp/json.gbnf +0 -0
  134. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/from-llama.cpp/json_arr.gbnf +0 -0
  135. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/from-llama.cpp/list.gbnf +0 -0
  136. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/from-llama.cpp/vllm-sql.gbnf +0 -0
  137. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/lark.lark +0 -0
  138. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/rfc.lark +0 -0
  139. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/data/rfc.xml +0 -0
  140. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/gtest.sh +0 -0
  141. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/lark.sh +0 -0
  142. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/run.sh +0 -0
  143. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/src/minimal.rs +0 -0
  144. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/tests/test_raw_parser.rs +0 -0
  145. {llguidance-0.7.12 → llguidance-0.7.14}/sample_parser/tests/test_stop.rs +0 -0
  146. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/annotate_asm.js +0 -0
  147. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/cbindgen.sh +0 -0
  148. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/checklinks.py +0 -0
  149. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/checklinks.sh +0 -0
  150. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/ci-publish.py +0 -0
  151. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/disasm.sh +0 -0
  152. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/gbnf_to_lark.py +0 -0
  153. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/gen-testcase.py +0 -0
  154. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/git-version.sh +0 -0
  155. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/install-deps.sh +0 -0
  156. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/jsonschema-stats.js +0 -0
  157. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/remote-guidance-test.sh +0 -0
  158. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/rust-size.js +0 -0
  159. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/rust_size.py +0 -0
  160. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/tokenizer_test.py +0 -0
  161. {llguidance-0.7.12 → llguidance-0.7.14}/scripts/update-git.py +0 -0
  162. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie/LICENSE +0 -0
  163. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie/README.md +0 -0
  164. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie/src/bytes.rs +0 -0
  165. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie/src/lib.rs +0 -0
  166. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie/src/recognizer.rs +0 -0
  167. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie/src/rng.rs +0 -0
  168. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie/src/svob.rs +0 -0
  169. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie/src/tokenv.rs +0 -0
  170. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie/tests/test_svob.rs +0 -0
  171. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie_hf_downloader/LICENSE +0 -0
  172. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie_hf_downloader/src/lib.rs +0 -0
  173. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie_hf_tokenizers/LICENSE +0 -0
  174. {llguidance-0.7.12 → llguidance-0.7.14}/toktrie_hf_tokenizers/src/lib.rs +0 -0
{llguidance-0.7.12 → llguidance-0.7.14}/CHANGELOG.md RENAMED
@@ -4,6 +4,20 @@ All notable changes to this project will be documented in this file. Dates are d
4
4
 
5
5
  If a release doesn't introduce any interesting changes (build fixes etc.), it's skipped.
6
6
 
7
+ #### [0.7.14](https://github.com/guidance-ai/llguidance/compare/v0.7.13...0.7.14) 2025-04-11
8
+
9
+ - support %lark { ... } syntax for nested grammars [`#157`](https://github.com/guidance-ai/llguidance/pull/157)
10
+ - treat \d and \w in json schema as ASCII; fix ^$ anchors [`#158`](https://github.com/guidance-ai/llguidance/issues/158)
11
+ - make it build without "lark" feature again [`929b13f`](https://github.com/guidance-ai/llguidance/commit/929b13f1e523b4cfda6d842ef84cfabf8b99224f)
12
+ - bump derivre to 0.3.7 and use it for anchor handling in json schema (fixes a few testcases) [`bb228cb`](https://github.com/guidance-ai/llguidance/commit/bb228cbca080f1382bb992dd27bddc0223e9dd00)
13
+ - expose regex_to_lark() in Rust and Python; add \d\w\s replacement [`78fb32f`](https://github.com/guidance-ai/llguidance/commit/78fb32fe2745d30ca94a62b00e5a7299750d80b0)
14
+ - fix usage of / vs \* in python signatures [`ca73c2a`](https://github.com/guidance-ai/llguidance/commit/ca73c2abd44e75d569230b942f53c72b052ed2ab)
15
+
16
+ #### [0.7.13](https://github.com/guidance-ai/llguidance/compare/v0.7.12...0.7.13) 2025-04-05
17
+
18
+ - expose LLParserLimits in Python API [`598dc8f`](https://github.com/guidance-ai/llguidance/commit/598dc8f37f69f51244e54d9885445abf02a515a7)
19
+ - pre-compute lexer states for particularly large regexes (can be disabled in ParserLimits)
20
+
7
21
  #### [0.7.12](https://github.com/guidance-ai/llguidance/compare/v0.7.11...0.7.12) 2025-04-04
8
22
 
9
23
  - performance optimizations