typellm 0.7.0__tar.gz → 0.7.2__tar.gz

{typellm-0.7.0 → typellm-0.7.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: typellm
-Version: 0.7.0
+Version: 0.7.2
 Summary: Type-Safe Decoding for Autoregressive LLMs
 License-Expression: Apache-2.0
 Project-URL: Homepage, https://typellm.ai/
@@ -31,6 +31,9 @@ Dynamic: license-file
 
 ## Updates
 - 🚀 **[TypeLLM API](https://typellm.ai)** is live: typed outputs without serving a model, $5 free credit. [Try it](https://typellm.ai/dashboard/playground) · [Quick start](#typellm-api-cloud) · [Examples](https://typellm.ai/examples).
+- **[2026/10/15]** An [array of boxes](#boxes) finds every match in an image: every dog, every line item.
+- **[2026/10/15]** Added [boxes](#boxes): find something in an image, returned as its edges.
+- **[2026/10/14]** Added [tools and human input](#tools-and-human-input): a workflow pauses for answers from your own tools or a person, and resumes with them.
 - **[2026/10/08]** [Conditional fields](#conditional-fields) can test how sure an answer is: `"when": {"category": {"confidence": {"gte": 0.8}}}`.
 - **[2026/10/05]** Added [objects and arrays](#objects-and-arrays): group typed properties into one value, or return a variable number of typed items.
 - **[2026/10/01]** Added [conditional fields](#conditional-fields): a field with `when` is answered only if its dependencies meet conditions.
@@ -118,6 +121,23 @@ client = TypeLLMClient(
 )
 ```
 
+| Setting | Default | Purpose |
+| --- | --- | --- |
+| `base_url` | `"http://127.0.0.1:30000"` | SGLang server URL. |
+| `model` | `None` | Model ID; discovered from the server when omitted. |
+| `tokenizer` | `None` | Matching tokenizer ID or local path; when omitted, the server's paths, then its served model name. |
+| `text_max_tokens` | `128` | Text generation budget per field. |
+| `numeric_max_digits` | `32` | Digit limit for open numeric fields. |
+| `temperature` | `0` | 0 gives each field its most likely answer; above 0 samples at that temperature. |
+| `seed` | `None` | Client random-number seed. |
+| `timeout` | `120.0` | HTTP request timeout in seconds. |
+
+On your own server, no service limits apply: your server's context length,
+memory and timeouts are the only bounds. An enum takes up to 26 values, and a
+field that thinks without a `thinking_budget` may reason until the context is
+full. `label_pool` customizes the internal candidate labels; leave it unset for
+normal use. For debugging, `print_final_prompt=True` prints the prompts sent.
+
 ### Make a request
 
 With either client:
@@ -419,6 +439,44 @@ Each image can be a local file path, an http(s) URL, a `data:` URI, raw bytes,
 or a PIL image. Local files are read by the client, so the SGLang server does
 not need access to your filesystem.
 
+### Boxes
+
+A `box` field finds something in the image and returns where it is: `top`,
+`bottom`, `left` and `right`, each from 0 to 1 across the image, from its
+top-left corner.
+
+```python
+response = client.generate(
+    context="Read the receipt photo.",
+    images=["receipt.jpg"],
+    questions={"total_line": {"type": "box", "instructions": "The line that shows the total."}},
+)
+response.result  # {"total_line": {"top": 0.652, "bottom": 0.689, "left": 0.287, "right": 0.855}}
+```
+
+Multiply by the image's width and height for pixels. There is always a box: to
+check that the target is there, ask a boolean field and give the box a `when` on
+it, so the box is asked only when the target is there. A box cannot be null,
+sit inside an object's or an array's items, or be tested by a `when`.
+
+An array of boxes finds every match, from none to `maxItems` (at most 50):
+
+```python
+response = client.generate(
+    context="Read the receipt photo.",
+    images=["receipt.jpg"],
+    questions={"items": {"type": "array", "items": {"type": "box"},
+                         "instructions": "Every line item, each the whole line from its name to its price."}},
+)
+response.result  # {"items": [{"top": 0.421, "bottom": 0.461, "left": 0.12, "right": 0.837},
+                 #            {"top": 0.449, "bottom": 0.493, "left": 0.12, "right": 0.834},
+                 #            {"top": 0.48, "bottom": 0.521, "left": 0.118, "right": 0.829}]}
+```
+
+It returns `[]` when the image has no match. Its `items` take only
+`{"type": "box"}`: say what to find in the array's `instructions`. `minItems`
+and `maxItems` bound it; `continue_from` is not supported.
+
 ## Dependency-aware generation
 
 Fields run together by default, and each sees only the original context.
@@ -516,6 +574,40 @@ skipped field are skipped too, even when their other dependencies ran. A skipped
 field sends no requests; only its definition counts toward the call's input
 tokens.
 
+### Tools and human input
+
+Some answers come from outside the model: an order's status from your database,
+a result from your own tool, or a person's decision. On the hosted API, a field
+with `"source": "user"` is answered by your code. The call answers every field
+that does not wait on it, then stops with `status` `"needs_input"`, and
+`client.resume()` goes on with your answers:
+
+```python
+response = client.generate(context=email, questions={
+    "order_id": {"type": ["string", "null"], "instructions": "The order number in the email."},
+    "order_status": {"type": "string", "enum": ["shipped", "delivered", "lost"],
+                     "source": "user", "when": {"order_id": {"ne": None}}},
+    "reply": {"type": "string", "depends_on": ["order_status"]},
+})
+while response.status == "needs_input":   # response.needs_input names the fields
+    response = client.resume(response, {"order_status": lookup(response.result["order_id"])})
+```
+
+On your own server, pass answers you already have in `given`, as `result` gives
+them, and fields to leave out in `skipped`. The model is not asked those fields,
+and the fields that depend on them see them as answers:
+
+```python
+response = client.generate(context=email, questions=questions,
+                           given={"order_status": "lost"}, skipped=["refund"])
+```
+
+Passing a call's `result` and `skipped` back, with more fields, answers only the
+new ones. A field with `return_probabilities` is given as `result` has it, with
+its probabilities.
+
+[Docs](https://typellm.ai/docs/workflows#tools-and-human-input) · [Cookbook](https://typellm.ai/cookbook/tools-and-human-input)
+
 ## Probabilities and sampling
 
 Set `return_probabilities` on individual enum or boolean fields:

{typellm-0.7.0 → typellm-0.7.2}/README.md

@@ -16,6 +16,9 @@
 
 ## Updates
 - 🚀 **[TypeLLM API](https://typellm.ai)** is live: typed outputs without serving a model, $5 free credit. [Try it](https://typellm.ai/dashboard/playground) · [Quick start](#typellm-api-cloud) · [Examples](https://typellm.ai/examples).
+- **[2026/10/15]** An [array of boxes](#boxes) finds every match in an image: every dog, every line item.
+- **[2026/10/15]** Added [boxes](#boxes): find something in an image, returned as its edges.
+- **[2026/10/14]** Added [tools and human input](#tools-and-human-input): a workflow pauses for answers from your own tools or a person, and resumes with them.
 - **[2026/10/08]** [Conditional fields](#conditional-fields) can test how sure an answer is: `"when": {"category": {"confidence": {"gte": 0.8}}}`.
 - **[2026/10/05]** Added [objects and arrays](#objects-and-arrays): group typed properties into one value, or return a variable number of typed items.
 - **[2026/10/01]** Added [conditional fields](#conditional-fields): a field with `when` is answered only if its dependencies meet conditions.
@@ -103,6 +106,23 @@ client = TypeLLMClient(
 )
 ```
 
+| Setting | Default | Purpose |
+| --- | --- | --- |
+| `base_url` | `"http://127.0.0.1:30000"` | SGLang server URL. |
+| `model` | `None` | Model ID; discovered from the server when omitted. |
+| `tokenizer` | `None` | Matching tokenizer ID or local path; when omitted, the server's paths, then its served model name. |
+| `text_max_tokens` | `128` | Text generation budget per field. |
+| `numeric_max_digits` | `32` | Digit limit for open numeric fields. |
+| `temperature` | `0` | 0 gives each field its most likely answer; above 0 samples at that temperature. |
+| `seed` | `None` | Client random-number seed. |
+| `timeout` | `120.0` | HTTP request timeout in seconds. |
+
+On your own server, no service limits apply: your server's context length,
+memory and timeouts are the only bounds. An enum takes up to 26 values, and a
+field that thinks without a `thinking_budget` may reason until the context is
+full. `label_pool` customizes the internal candidate labels; leave it unset for
+normal use. For debugging, `print_final_prompt=True` prints the prompts sent.
+
 ### Make a request
 
 With either client:
@@ -404,6 +424,44 @@ Each image can be a local file path, an http(s) URL, a `data:` URI, raw bytes,
 or a PIL image. Local files are read by the client, so the SGLang server does
 not need access to your filesystem.
 
+### Boxes
+
+A `box` field finds something in the image and returns where it is: `top`,
+`bottom`, `left` and `right`, each from 0 to 1 across the image, from its
+top-left corner.
+
+```python
+response = client.generate(
+    context="Read the receipt photo.",
+    images=["receipt.jpg"],
+    questions={"total_line": {"type": "box", "instructions": "The line that shows the total."}},
+)
+response.result  # {"total_line": {"top": 0.652, "bottom": 0.689, "left": 0.287, "right": 0.855}}
+```
+
+Multiply by the image's width and height for pixels. There is always a box: to
+check that the target is there, ask a boolean field and give the box a `when` on
+it, so the box is asked only when the target is there. A box cannot be null,
+sit inside an object's or an array's items, or be tested by a `when`.
+
+An array of boxes finds every match, from none to `maxItems` (at most 50):
+
+```python
+response = client.generate(
+    context="Read the receipt photo.",
+    images=["receipt.jpg"],
+    questions={"items": {"type": "array", "items": {"type": "box"},
+                         "instructions": "Every line item, each the whole line from its name to its price."}},
+)
+response.result  # {"items": [{"top": 0.421, "bottom": 0.461, "left": 0.12, "right": 0.837},
+                 #            {"top": 0.449, "bottom": 0.493, "left": 0.12, "right": 0.834},
+                 #            {"top": 0.48, "bottom": 0.521, "left": 0.118, "right": 0.829}]}
+```
+
+It returns `[]` when the image has no match. Its `items` take only
+`{"type": "box"}`: say what to find in the array's `instructions`. `minItems`
+and `maxItems` bound it; `continue_from` is not supported.
+
 ## Dependency-aware generation
 
 Fields run together by default, and each sees only the original context.
@@ -501,6 +559,40 @@ skipped field are skipped too, even when their other dependencies ran. A skipped
 field sends no requests; only its definition counts toward the call's input
 tokens.
 
+### Tools and human input
+
+Some answers come from outside the model: an order's status from your database,
+a result from your own tool, or a person's decision. On the hosted API, a field
+with `"source": "user"` is answered by your code. The call answers every field
+that does not wait on it, then stops with `status` `"needs_input"`, and
+`client.resume()` goes on with your answers:
+
+```python
+response = client.generate(context=email, questions={
+    "order_id": {"type": ["string", "null"], "instructions": "The order number in the email."},
+    "order_status": {"type": "string", "enum": ["shipped", "delivered", "lost"],
+                     "source": "user", "when": {"order_id": {"ne": None}}},
+    "reply": {"type": "string", "depends_on": ["order_status"]},
+})
+while response.status == "needs_input":   # response.needs_input names the fields
+    response = client.resume(response, {"order_status": lookup(response.result["order_id"])})
+```
+
+On your own server, pass answers you already have in `given`, as `result` gives
+them, and fields to leave out in `skipped`. The model is not asked those fields,
+and the fields that depend on them see them as answers:
+
+```python
+response = client.generate(context=email, questions=questions,
+                           given={"order_status": "lost"}, skipped=["refund"])
+```
+
+Passing a call's `result` and `skipped` back, with more fields, answers only the
+new ones. A field with `return_probabilities` is given as `result` has it, with
+its probabilities.
+
+[Docs](https://typellm.ai/docs/workflows#tools-and-human-input) · [Cookbook](https://typellm.ai/cookbook/tools-and-human-input)
+
 ## Probabilities and sampling
 
 Set `return_probabilities` on individual enum or boolean fields:

{typellm-0.7.0 → typellm-0.7.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "typellm"
-version = "0.7.0"
+version = "0.7.2"
 description = "Type-Safe Decoding for Autoregressive LLMs"
 readme = "README.md"
 license = "Apache-2.0"

typellm-0.7.2/tests/test_box.py

@@ -0,0 +1,217 @@
+import unittest
+
+from typellm import SchemaError, TypeLLMClient
+from typellm.runtime import BOX_ITEM_TOKENS, BOX_MAX_TOKENS, BOX_PATTERN, box_list_pattern
+
+from tests.test_batching import FakeServer
+from tests.test_images import PNG
+
+
+class BoxServer(FakeServer):
+    """A FakeServer that answers a box's grammar with `coordinates`, as the model writes them."""
+
+    def __init__(self, coordinates="199, 166, 400, 500]}"):
+        super().__init__()
+        self.coordinates = coordinates
+
+    def _request(self, path, payload=None, *, allow_text=False):
+        response = super()._request(path, payload, allow_text=allow_text)
+        if path == "/generate" and not isinstance(payload["sampling_params"], dict):
+            for params, item in zip(payload["sampling_params"], response if isinstance(response, list) else [response]):
+                if params.get("regex") == BOX_PATTERN:
+                    item["text"] = self.coordinates
+        elif path == "/generate" and payload["sampling_params"].get("regex") == BOX_PATTERN:
+            response["text"] = self.coordinates
+        return response
+
+    def box_requests(self):
+        return [params for payload in self.payloads
+                for params in (payload["sampling_params"] if isinstance(payload["sampling_params"], list)
+                               else [payload["sampling_params"]])
+                if params.get("regex") == BOX_PATTERN]
+
+
+class BoxTests(unittest.TestCase):
+    def run_questions(self, questions, server=None, images=(PNG,), **options):
+        client = TypeLLMClient("http://127.0.0.1:30000", model="fake")
+        client.sglang = server or BoxServer()
+        done = client.generate(context="Read the receipt photo.", images=list(images), questions=questions, **options)
+        sent = [t for p in client.sglang.payloads
+                for t in ([p["text"]] if isinstance(p["text"], str) else p["text"])]
+        return done, sent, client.sglang
+
+    def test_a_box_comes_back_as_fractions_of_the_image(self):
+        done, sent, server = self.run_questions({"total": {"type": "box", "instructions": "The total line."}})
+        self.assertEqual(done.result, {"total": {"top": 0.166, "bottom": 0.5, "left": 0.199, "right": 0.4}})
+        [prompt] = [t for t in sent if t.endswith('{"bbox_2d": [')]
+        self.assertIn("Type: bounding box", prompt)
+        self.assertIn("Instructions: The total line.", prompt)
+        [params] = server.box_requests()
+        self.assertGreaterEqual(params["max_new_tokens"], BOX_MAX_TOKENS)
+
+    def test_edges_given_the_wrong_way_round_are_put_in_order(self):
+        done, _, _ = self.run_questions({"total": {"type": "box"}}, server=BoxServer("400, 500, 199, 166]}"))
+        self.assertEqual(done.result["total"], {"top": 0.166, "bottom": 0.5, "left": 0.199, "right": 0.4})
+
+    def test_a_box_without_instructions_is_asked_by_its_name(self):
+        _, sent, _ = self.run_questions({"signature": {"type": "box"}})
+        [prompt] = [t for t in sent if t.endswith('{"bbox_2d": [')]
+        self.assertIn('Instructions: Locate "signature" in the image.', prompt)
+
+    def test_fields_that_depend_on_a_box_see_it(self):
+        questions = {"total": {"type": "box"}, "legible": {"type": "boolean", "depends_on": ["total"]}}
+        done, sent, _ = self.run_questions(questions)
+        self.assertTrue(done.result["legible"])
+        [later] = [t for t in sent if 'Field: "legible"' in t[t.rindex("Field:"):]]
+        self.assertIn('{"bbox_2d": [199, 166, 400, 500]}', later)  # the box in history, as the model wrote it
+        self.assertIn('"total": {"top": 0.166, "bottom": 0.5, "left": 0.199, "right": 0.4}', later)
+
+    def test_boxes_and_numbers_decode_in_one_request(self):
+        done, _, server = self.run_questions({"total": {"type": "box"}, "amount": {"type": "number"}})
+        self.assertEqual(done.result["amount"], 7)
+        self.assertEqual(len([p for p in server.payloads if any(
+            "regex" in params for params in (p["sampling_params"] if isinstance(p["sampling_params"], list)
+                                             else [p["sampling_params"]]))]), 1)
+
+    def test_a_box_can_be_given(self):
+        box = {"top": 0.2, "bottom": 0.4, "left": 0.1, "right": 0.3}
+        done, sent, server = self.run_questions(
+            {"total": {"type": "box"}, "legible": {"type": "boolean", "depends_on": ["total"]}}, given={"total": box})
+        self.assertEqual(done.result["total"], box)
+        self.assertEqual(server.box_requests(), [])
+        [later] = [t for t in sent if 'Field: "legible"' in t[t.rindex("Field:"):]]
+        self.assertIn('{"bbox_2d": [100, 200, 300, 400]}', later)
+        with self.assertRaisesRegex(SchemaError, "must be a box"):
+            self.run_questions({"total": {"type": "box"}}, given={"total": {"top": 2}})
+
+    def test_what_a_box_cannot_be(self):
+        cases = [
+            ({"total": {"type": "box"}}, (), "needs images"),
+            ({"total": {"type": ["box", "null"]}}, (PNG,), "cannot be null"),
+            ({"total": {"type": "box", "enum": [1, 2]}}, (PNG,), "enum is not supported on a box"),
+            ({"total": {"type": "box", "return_probabilities": True}}, (PNG,), "return_probabilities is not supported"),
+            ({"rows": {"type": "array", "items": {"type": "object", "properties": {"at": {"type": "box"}}}}},
+             (PNG,), "inside an array"),
+            ({"total": {"type": "box"}, "ok": {"type": "boolean", "when": {"total": None}}}, (PNG,), "a box"),
+        ]
+        for questions, images, message in cases:
+            with self.subTest(message=message):
+                with self.assertRaisesRegex(SchemaError, message):
+                    self.run_questions(questions, images=images)
+
+
+
+class ListServer(BoxServer):
+    """A BoxServer that answers a list of boxes' grammar with `listed`, after its "[" as the model writes it,
+    and whether there is any match with `present`."""
+
+    def __init__(self, listed='{"bbox_2d": [100, 200, 300, 400]}, {"bbox_2d": [500, 600, 700, 800]}]', present=True):
+        super().__init__()
+        self.listed = listed
+        self.present = present
+
+    def _request(self, path, payload=None, *, allow_text=False):
+        response = super()._request(path, payload, allow_text=allow_text)
+        if path == "/generate" and "token_ids_logprob" in payload and not self.present:
+            texts = [payload["text"]] if isinstance(payload["text"], str) else payload["text"]
+            for text, item in zip(texts, response if isinstance(response, list) else [response]):
+                if "Is there at least one match" in text[text.rindex("<|im_start|>user"):]:
+                    scores = item["meta_info"]["output_token_ids_logprobs"][0]
+                    # The second label, false, over the first.
+                    scores[0][0], scores[1][0] = -9.0, 0.0
+        if path == "/generate":
+            params = payload["sampling_params"]
+            for one, item in zip(params if isinstance(params, list) else [params],
+                                 response if isinstance(response, list) else [response]):
+                if one.get("regex", "").startswith(("(?:" + r'\{"bbox_2d"', r'\{"bbox_2d"')):
+                    item["text"] = self.listed
+        return response
+
+    def list_requests(self):
+        return [params for payload in self.payloads
+                for params in (payload["sampling_params"] if isinstance(payload["sampling_params"], list)
+                               else [payload["sampling_params"]])
+                if params.get("regex", "").endswith(r"\]") and "bbox_2d" in params.get("regex", "")]
+
+
+class BoxListTests(unittest.TestCase):
+    def run_questions(self, questions, server=None, **options):
+        return BoxTests.run_questions(self, questions, server=server or ListServer(), **options)
+
+    def test_an_array_of_boxes_lists_every_match_in_one_answer(self):
+        questions = {"dogs": {"type": "array", "items": {"type": "box"}, "instructions": "Every dog."}}
+        done, sent, server = self.run_questions(questions)
+        self.assertEqual(done.result["dogs"], [{"top": 0.2, "bottom": 0.4, "left": 0.1, "right": 0.3},
+                                               {"top": 0.6, "bottom": 0.8, "left": 0.5, "right": 0.7}])
+        [prompt] = [t for t in sent if t.endswith("Answer [] if there is none.<|im_end|>\n<|im_start|>assistant\n[")
+                    or t.rstrip().endswith("[") and "list of bounding boxes" in t]
+        self.assertIn("Instructions: Every dog.", prompt)
+        [params] = server.list_requests()
+        # The hidden question said there is a match, so the list holds one at least.
+        self.assertEqual(params["regex"], box_list_pattern(1, 50))
+        self.assertGreaterEqual(params["max_new_tokens"], 50 * BOX_ITEM_TOKENS)
+        self.assertEqual(list(done.result), ["dogs"])  # the presence question is not in the result
+        self.assertEqual(done.skipped, [])
+
+    def test_bounds_shape_the_grammar(self):
+        questions = {"dogs": {"type": "array", "items": {"type": "box"}, "minItems": 1, "maxItems": 3}}
+        _, _, server = self.run_questions(questions)
+        [params] = server.list_requests()
+        self.assertEqual(params["regex"], box_list_pattern(1, 3))
+
+    def test_no_match_is_an_empty_list_and_asks_for_no_boxes(self):
+        questions = {"dogs": {"type": "array", "items": {"type": "box"}},
+                     "count": {"type": "integer", "depends_on": ["dogs"]}}
+        done, sent, server = self.run_questions(questions, server=ListServer(present=False))
+        self.assertEqual(done.result, {"dogs": [], "count": 7})
+        self.assertEqual(server.list_requests(), [])
+        [later] = [t for t in sent if 'Field: "count"' in t[t.rindex("Field:"):]]
+        self.assertIn('assistant\n[]', later)
+
+    def test_a_list_that_must_hold_a_box_asks_no_presence_question(self):
+        questions = {"dogs": {"type": "array", "items": {"type": "box"}, "minItems": 1}}
+        done, sent, _ = self.run_questions(questions, server=ListServer(present=False))
+        self.assertEqual(len(done.result["dogs"]), 2)
+        self.assertFalse(any("Is there at least one match" in t for t in sent))
+
+    def test_a_box_listed_twice_comes_back_once(self):
+        twice = '{"bbox_2d": [100, 200, 300, 400]}, {"bbox_2d": [300, 400, 100, 200]}]'
+        done, _, _ = self.run_questions({"dogs": {"type": "array", "items": {"type": "box"}}}, server=ListServer(twice))
+        self.assertEqual(done.result["dogs"], [{"top": 0.2, "bottom": 0.4, "left": 0.1, "right": 0.3}])
+
+    def test_dependents_see_the_list(self):
+        questions = {"dogs": {"type": "array", "items": {"type": "box"}},
+                     "count": {"type": "integer", "depends_on": ["dogs"]}}
+        done, sent, _ = self.run_questions(questions)
+        [later] = [t for t in sent if 'Field: "count"' in t[t.rindex("Field:"):]]
+        self.assertIn('[{"bbox_2d": [100, 200, 300, 400]}, {"bbox_2d": [500, 600, 700, 800]}]', later)
+
+    def test_a_list_can_be_given(self):
+        boxes = [{"top": 0.2, "bottom": 0.4, "left": 0.1, "right": 0.3}]
+        questions = {"dogs": {"type": "array", "items": {"type": "box"}},
+                     "count": {"type": "integer", "depends_on": ["dogs"]}}
+        done, sent, server = self.run_questions(questions, given={"dogs": boxes})
+        self.assertEqual(done.result["dogs"], boxes)
+        self.assertEqual(server.list_requests(), [])
+        self.assertFalse(any("Is there at least one match" in t for t in sent))
+        [later] = [t for t in sent if 'Field: "count"' in t[t.rindex("Field:"):]]
+        self.assertIn('[{"bbox_2d": [100, 200, 300, 400]}]', later)
+        with self.assertRaisesRegex(SchemaError, "list of 0 to 50 boxes"):
+            self.run_questions(questions, given={"dogs": [{"top": 2}]})
+
+    def test_what_an_array_of_boxes_cannot_be(self):
+        cases = [
+            ({"dogs": {"type": "array", "items": {"type": "box", "instructions": "A dog."}}}, "take only their type"),
+            ({"dogs": {"type": "array", "items": {"type": "box"}, "continue_from": []}}, "continue_from"),
+            ({"dogs": {"type": "array", "items": {"type": "box"}, "maxItems": 60}}, "at most 50"),
+            ({"dogs": {"type": "array", "items": {"type": "box"}},
+              "ok": {"type": "boolean", "when": {"dogs": None}}}, "a box"),
+        ]
+        for questions, message in cases:
+            with self.subTest(message=message):
+                with self.assertRaisesRegex(SchemaError, message):
+                    self.run_questions(questions)
+
+
+if __name__ == "__main__":
+    unittest.main()

{typellm-0.7.0 → typellm-0.7.2}/typellm/runtime.py

@@ -43,10 +43,61 @@ DEFAULT_SOCKET_TIMEOUT = 120.0
 
 
 def _closed_answer(decision: "Choice", value_json: str) -> str:
-    """The full answer an open field leaves in history, e.g. {"total": 174600}."""
+    """The full answer an open field leaves in history, e.g. {"total": 174600}; a box's, its
+    coordinates as the model writes them: {"bbox_2d": [199, 166, 400, 500]}."""
+    if decision.box_list:
+        return decision.answer_prefill + value_json  # the list after its "[", closing bracket included
+    if decision.box:
+        return decision.answer_prefill + value_json + "]}"
     return decision.answer_prefill + " " + value_json + "}" if decision.answer_prefill else value_json
 
 
+# A box is asked as the model grounds things itself: [x1, y1, x2, y2], each from 0 to 1000 across
+# the image, x from its left edge and y from its top. The answer is decoded to that grammar.
+BOX_COORDINATE = r"(?:1000|[1-9][0-9]{0,2}|0)"
+BOX_PATTERN = ", ".join([BOX_COORDINATE] * 4) + r"\]\}?"
+# Tokens for the four coordinates and their separators, one digit a token at worst.
+BOX_MAX_TOKENS = 32
+# An array of boxes is the list the model writes when it grounds several things, after its "[":
+# [{"bbox_2d": [x1, y1, x2, y2]}, ...], each box its own coordinates.
+_BOX_ITEM = r'\{"bbox_2d": \[' + ", ".join([BOX_COORDINATE] * 4) + r"\]\}"
+# Tokens for one box in the list: its key, coordinates and separators.
+BOX_ITEM_TOKENS = 28
+
+
+def box_list_pattern(min_items: int, max_items: int) -> str:
+    """The grammar of a list of boxes after its "[": from min_items to max_items boxes, then "]"."""
+    more = _BOX_ITEM + ("(?:, " + _BOX_ITEM + "){" + f"{max(min_items - 1, 0)},{max_items - 1}" + "}")
+    return (more if min_items else "(?:" + more + ")?") + r"\]"
+
+
+def _box_list_value(text: str) -> tuple[list[dict[str, float]], str]:
+    """A decoded list of boxes as its value, each as _box_value gives it, and as history keeps it,
+    after the "[". A box the list already holds is not added again."""
+    boxes = json.loads("[" + text)
+    values: list[dict[str, float]] = []
+    written: list[str] = []
+    for box in boxes:
+        value, coordinates = _box_value(", ".join(str(n) for n in box["bbox_2d"]))
+        if value not in values:
+            values.append(value)
+            written.append('{"bbox_2d": [' + coordinates + "]}")
+    return values, ", ".join(written) + "]"
+
+
+def _box_value(text: str) -> tuple[dict[str, float], str]:
+    """A decoded box as its value, {"top", "bottom", "left", "right"} from 0 to 1, and its
+    coordinates as history keeps them. Edges given the wrong way round are put in order."""
+    numbers = [int(part) for part in text.strip().rstrip("}").rstrip().rstrip("]").split(",")]
+    if len(numbers) != 4 or any(not 0 <= n <= 1000 for n in numbers):
+        raise ValueError(f"Generated an invalid box: {text!r}")
+    x1, y1, x2, y2 = numbers
+    left, right = sorted((x1, x2))
+    top, bottom = sorted((y1, y2))
+    value = {"top": top / 1000, "bottom": bottom / 1000, "left": left / 1000, "right": right / 1000}
+    return value, f"{left}, {top}, {right}, {bottom}"
+
+
 def _closed_label(decision: "Choice", label: str) -> str:
     """A choice answer as history keeps it, e.g. {"category": "A"}."""
     return decision.label_prefill + label + '"}' if decision.label_prefill else label
@@ -103,9 +154,19 @@ class Choice:
     # More choices than labels: the prompt lists values, the answer writes one, and the client's
     # choice scorer gives each its probability. choices is keyed "0", "1", ... in enum order.
     by_value: bool = False
+    # A box: where something is in the image (see BOX_PATTERN).
+    box: bool = False
+    box_list: tuple[int, int] | None = None
+    presence_from: str | None = None
+    presence_for: str | None = None
+
+    @property
+    def hidden(self) -> bool:
+        """A question the call asks for a field's sake, never in the result: an effort, a presence."""
+        return self.effort_for is not None or self.presence_for is not None
 
     def __post_init__(self) -> None:
-        if not self.choices and self.numeric_type is None and not self.text_type:
+        if not self.choices and self.numeric_type is None and not self.text_type and not self.box:
             raise ValueError("Choice.choices must not be empty")
         if self.text_type and (self.choices or self.numeric_type is not None):
             raise ValueError("Text choices cannot have enum values or a numeric type")
@@ -136,6 +197,12 @@ class Choice:
         Chat models tend to answer {"name": value}, so the value is decoded right
         where they would write it.
         """
+        if self.box_list:
+            return "["
+        if self.box:
+            # The key the model grounds with, whatever the field's name: a box's own name can
+            # pull the model to answer the thing itself, such as the total, not where it is.
+            return '{"bbox_2d": ['
         if self.field_name is None or not (self.text_type or self.numeric_type is not None):
             return ""
         # No trailing space: in {"name": 12} the space belongs to the value's first token.
@@ -160,6 +227,19 @@ class Choice:
         # A nullable field says "or null" in both its type and its answer: a bare
         # <number> reads as "a number is required" and pulls absent values to 0.
         or_null = " or null" if self.nullable else ""
+        if self.box_list:
+            lines.append("Type: list of bounding boxes")
+            lines.append(f"Instructions: {self.question}")
+            lines.append('Answer with a JSON list of every match, each as {"bbox_2d": [x1, y1, x2, y2]} with '
+                         "coordinates from 0 to 1000: x from the left edge, y from the top. "
+                         "Answer [] if there is none.")
+            return "\n".join(lines)
+        if self.box:
+            lines.append("Type: bounding box")
+            lines.append(f"Instructions: {self.question}")
+            lines.append('Answer with its bounding box in the image as {"bbox_2d": [x1, y1, x2, y2]}, each '
+                         "from 0 to 1000: x from the left edge, y from the top.")
+            return "\n".join(lines)
         if self.text_type:
             kind = f"string{or_null}"
         elif self.numeric_type is not None:
@@ -221,6 +301,9 @@ class ArrayChoice:
     dependency_groups: tuple[tuple[str, ...], ...] | None = None
     effort_for: None = None
     effort_from: None = None
+    presence_for: None = None
+    presence_from: None = None
+    hidden: bool = False
     return_probabilities: bool = False
     # Items the caller already has: the array starts from them (schema's continue_from).
     continue_from: tuple[Any, ...] = ()
@@ -475,9 +558,15 @@ class TypeLLMClient:
                     dependency_groups=item.dependency_groups,
                     descriptions=item.descriptions,
                     levels=item.levels,
+                    box=item.box,
+                    box_list=item.box_list,
+                    presence_from=item.presence_from,
+                    presence_for=item.presence_for,
                 )
 
             def bind_array(array):
+                if any(item.box for item in array.items):
+                    raise SchemaError(f"{array.name!r}: a box is not supported inside an array's items")
                 return ArrayChoice(
                     name=array.name,
                     question=array.question,
@@ -653,6 +742,9 @@ class TypeLLMClient:
         with call_scope(timeout, cancel) as scope:
             try:
                 decisions = self.compile_schema(schema)
+                boxes = [d.name for d in decisions if getattr(d, "box", False)]
+                if boxes and not encoded_images:
+                    raise SchemaError(f"{boxes!r} are boxes: a box locates something in an image, so the call needs images")
                 given_by_name = _given_by_name(decisions, given or {})
                 skip = _skip_names(decisions, skipped or ())
                 count = getattr(self.sglang, "count_tokens", None)
@@ -879,7 +971,7 @@ def _assemble(decisions: Sequence[Any], rows: Sequence[dict | None], prefix: str
 
     output: dict[str, Any] = {}
     for decision, row in zip(decisions, rows):
-        if decision.effort_for is None and row is not None:  # not a hidden effort question, not skipped
+        if not decision.hidden and row is not None:  # not a hidden question, not skipped
             _put(output, _path(decision), row["value"] if isinstance(decision, ArrayChoice)
                  else _answer(decision, row))
     thinking: dict[str, str] = {}
@@ -887,6 +979,8 @@ def _assemble(decisions: Sequence[Any], rows: Sequence[dict | None], prefix: str
     skipped: list[str] = []
     for decision, row in zip(decisions, rows):
         path = _path(decision)
+        if decision.presence_for is not None:
+            continue
         if decision.effort_for is not None:
             if row is not None:
                 effort[label(path[:-1])] = row["value"]
@@ -1010,6 +1104,14 @@ def _parse_numbers(
     """Read the grammar-generated numbers: (value, closed prompt, value text)."""
     outputs = []
     for (prompt, decision), raw in zip(items, texts):
+        if decision.box_list:
+            value, written = _box_list_value(raw)
+            outputs.append((value, prompt + written, written))
+            continue
+        if decision.box:
+            value, coordinates = _box_value(raw)
+            outputs.append((value, prompt + coordinates + "]}", coordinates))
+            continue
         text = raw.strip()
         if text.endswith("}"):
             text = text[:-1].rstrip()
@@ -1147,10 +1249,10 @@ def _given_value(given: Mapping[str, Any], path: Sequence[str]) -> Any:
 def _skip_names(decisions: Sequence[Any], skipped: Sequence[str]) -> set[str]:
     """The decisions skipped names, as Generation.skipped names them: a field, an object or a property."""
     paths = [tuple(name.split(".")) for name in skipped]
-    names = {d.name for d in decisions if d.effort_for is None and any(
+    names = {d.name for d in decisions if not d.hidden and any(
         _path(d)[:len(path)] == path for path in paths)}
     unknown = [name for name, path in zip(skipped, paths)
-               if not any(_path(d)[:len(path)] == path for d in decisions if d.effort_for is None)]
+               if not any(_path(d)[:len(path)] == path for d in decisions if not d.hidden)]
     if unknown:
         raise SchemaError(f"skipped names fields the schema does not have: {unknown!r}")
     return names
@@ -1160,12 +1262,12 @@ def _given_by_name(decisions: Sequence[Any], given: Mapping[str, Any]) -> dict[s
     """Each given decision's value, by decision name; a value is checked when its row is made."""
     by_name = {}
     for decision in decisions:
-        if decision.effort_for is not None:
+        if decision.hidden:
             continue
         value = _given_value(given, _path(decision))
         if value is not _MISSING:
             by_name[decision.name] = value
-    tops = {_path(d)[0] for d in decisions if d.effort_for is None}
+    tops = {_path(d)[0] for d in decisions if not d.hidden}
     unknown = [name for name in given if name not in tops]
     if unknown:
         raise SchemaError(f"given names fields the schema does not have: {unknown!r}")
@@ -1217,6 +1319,24 @@ def _given_row(client, context: str, decision: Any, given: Any, parent: str | No
     chosen = None
     if value is None and decision.nullable and (decision.text_type or decision.numeric_type is not None):
         closed = _closed_answer(decision, "null")
+    elif decision.box_list:
+        low, high = decision.box_list
+        edges = ("top", "bottom", "left", "right")
+        if (not isinstance(value, list) or not low <= len(value) <= high
+                or not all(isinstance(box, Mapping) and set(box) == set(edges)
+                           and all(_is_number(box[edge]) and 0 <= box[edge] <= 1 for edge in edges)
+                           for box in value)):
+            raise SchemaError(f"given {label} must be a list of {low} to {high} boxes: "
+                              "top, bottom, left and right from 0 to 1")
+        closed = _closed_answer(decision, ", ".join(
+            '{"bbox_2d": [' + ", ".join(str(round(box[edge] * 1000)) for edge in ("left", "top", "right", "bottom"))
+            + "]}" for box in value) + "]")
+    elif decision.box:
+        edges = ("top", "bottom", "left", "right")
+        if (not isinstance(value, Mapping) or set(value) != set(edges)
+                or not all(_is_number(value[edge]) and 0 <= value[edge] <= 1 for edge in edges)):
+            raise SchemaError(f"given {label} must be a box: top, bottom, left and right from 0 to 1")
+        closed = _closed_answer(decision, ", ".join(str(round(value[edge] * 1000)) for edge in ("left", "top", "right", "bottom")))
     elif decision.text_type:
         if not isinstance(value, str):
             raise SchemaError(f"given {label} must be a string")
@@ -1274,11 +1394,12 @@ def _execute_dependency_decisions(
     rows_by_name = {}
     prompts_by_name = {}
     ancestors = {}
-    hidden = {d.name for d in decisions if d.effort_for is not None}
+    hidden = {d.name for d in decisions if d.hidden}
     by_name = {d.name: d for d in decisions}
     given = given or {}
-    # A given or skipped field asks no effort question.
-    skip = set(skip) | {d.name for d in decisions if d.effort_for in given or d.effort_for in skip}
+    # A given or skipped field asks no effort or presence question.
+    skip = set(skip) | {d.name for d in decisions
+                        if d.hidden and (d.effort_for or d.presence_for) in set(given) | set(skip)}
 
     def confidence(name: str) -> float | None:
         """A dependency's confidence, for a "when" that tests it; None without probabilities."""
@@ -1335,6 +1456,13 @@ def _execute_dependency_decisions(
             if isinstance(decision, ArrayChoice):
                 arrays.append(decision)
                 continue
+            if decision.presence_from is not None:
+                if not rows_by_name[decision.presence_from]["value"]:
+                    # No match in the image: the list is empty, and no box is asked for.
+                    rows_by_name[decision.name], prompts_by_name[decision.name] = _given_row(
+                        client, context, decision, [], parent_prefixes.get(decision.name), values, image_count, None)
+                    continue
+                decision = replace(decision, box_list=(1, decision.box_list[1]))
             if decision.effort_from is not None:
                 budget = THINKING_EFFORTS[rows_by_name[decision.effort_from]["value"]]
                 decision = replace(decision, thinking=budget is not None, thinking_budget=budget)
@@ -1670,7 +1798,8 @@ def _execute_batch_decisions(
         if decision.text_type:
             text_pending.append((index, decision, messages))
             continue
-        if decision.numeric_type is not None:
+        # A box decodes beside the numbers, to its own grammar.
+        if decision.numeric_type is not None or decision.box:
             numeric_pending.append((index, decision, messages))
             continue
         if decision.by_value:
@@ -1735,7 +1864,7 @@ def _execute_batch_decisions(
 
     open_pending = [(index, decision, messages, ready[decision_slots[index]])
                     for index, decision, messages in numeric_pending + text_pending]
-    numeric_pending = [item for item in open_pending if item[1].numeric_type is not None]
+    numeric_pending = [item for item in open_pending if item[1].numeric_type is not None or item[1].box]
     text_pending = [item for item in open_pending if item[1].text_type]
 
     # A layer's numbers and strings decode side by side in one request, and its choices are
@@ -1749,9 +1878,14 @@ def _execute_batch_decisions(
         text_seed = rng.randrange(2**31) if text_pending else 0
         fields = (
             [prompt for prompt, _ in number_items],
-            [numeric_pattern(d.numeric_type or "", numeric_max_digits, d.nullable) for _, d in number_items],
-            # Digits, sign, point and the closing brace; tokens hold one or more characters.
-            numeric_max_digits + 4,
+            [box_list_pattern(*d.box_list) if d.box_list else BOX_PATTERN if d.box
+             else numeric_pattern(d.numeric_type or "", numeric_max_digits, d.nullable)
+             for _, d in number_items],
+            # Digits, sign, point and the closing brace; tokens hold one or more characters. A box
+            # needs its four coordinates, and a list of boxes each of its boxes.
+            max([numeric_max_digits + 4]
+                + [BOX_MAX_TOKENS for _, d in number_items if d.box]
+                + [d.box_list[1] * BOX_ITEM_TOKENS + 4 for _, d in number_items if d.box_list]),
             # From {"name": the model writes the string's first token, quote included.
             [prompt + decision.answer_prefill for _, decision, _, prompt in text_pending],
         )

{typellm-0.7.0 → typellm-0.7.2}/typellm/schema.py

@@ -59,6 +59,14 @@ class Decision:
     # A score's ordered levels, lowest first: its choices are their labels, and its value is the
     # probability-weighted average of their indices. Empty for every other field.
     levels: tuple[str, ...] = ()
+    # A box: where something is in the image, as {"top", "bottom", "left", "right"} from 0 to 1.
+    box: bool = False
+    # An array of boxes: every match in the image, listed in one answer; (minItems, maxItems).
+    box_list: tuple[int, int] | None = None
+    # An array of boxes that may be empty asks first whether there is any match: presence_from on
+    # the list names that hidden question, presence_for on it names the list.
+    presence_from: str | None = None
+    presence_for: str | None = None
 
 
 @dataclass(frozen=True)
@@ -97,6 +105,9 @@ MAX_ARRAY_ITEMS = 50
 # context and arrays the model produces alike: "would another item improve the array" stopped empty
 # arrays early, and "does the context mention another" stopped produced ones.
 CONTINUE_QUESTION = "Should a new item be appended to the current array?"
+# Asked before an array of boxes that may be empty: a model listing boxes lists some whether or
+# not the image has the target, and answers this yes or no reliably.
+PRESENCE_QUESTION = "Is there at least one match for the question below in the image?\n\nThe question:\n{question}"
 # The name the question is asked under, so its answer starts as a field's does.
 CONTINUE_FIELD = "append_item"
 # What opens the prompt of an item, or of each property of an object item.
@@ -126,7 +137,11 @@ EFFORT_QUESTION = (
 def _describe(decision: Decision) -> str:
     """A field's type, instructions and choices, as its own prompt states them."""
     or_null = " or null" if decision.nullable else ""
-    if decision.text_type:
+    if decision.box_list:
+        kind = "list of bounding boxes"
+    elif decision.box:
+        kind = "bounding box"
+    elif decision.text_type:
         kind = "string" + or_null
     elif decision.numeric_type is not None:
         kind = decision.numeric_type + or_null
@@ -197,6 +212,11 @@ def _compile(schema: Mapping[str, Any]) -> list[Decision | ArrayField]:
             entries.extend(members)
             scope[name] = ("object", tuple(entry.decision.name for entry in members))
             order.append(name)
+        elif kind == "array" and _boxes(field):
+            # Every box is found in one answer, as the model lists them: one field, not an array's turns.
+            entries.append(_Entry(_box_list(name, field), field, None))
+            scope[name] = ("scalar", name)
+            order.append(name)
         elif kind == "array":
             scope[name] = ("array", name)
             order.append(_array(name, field))
@@ -349,14 +369,8 @@ def _object_entries(path: tuple[str, ...], field: Mapping[str, Any], label: str,
     return entries
 
 
-def _array(name: str, field: Mapping[str, Any]) -> ArrayField:
-    """An array field, its items compiled and linked; its own depends_on and when come later."""
-    question = _question(name, field, f'All the items of "{name}".')
-    if "items" not in field:
-        raise SchemaError(f"array {name!r} needs items")
-    items = field["items"]
-    if not isinstance(items, Mapping):
-        raise SchemaError(f"items for {name!r} must be a schema object")
+def _item_bounds(name: str, field: Mapping[str, Any]) -> tuple[int, int | None]:
+    """An array's minItems and maxItems, checked."""
     min_items = field.get("minItems", 0)
     if type(min_items) is not int or min_items < 0:
         raise SchemaError(f"minItems for {name!r} must be a non-negative integer")
@@ -367,6 +381,37 @@ def _array(name: str, field: Mapping[str, Any]) -> ArrayField:
         raise SchemaError(f"minItems for {name!r} is more than its maxItems")
     if min_items > MAX_ARRAY_ITEMS or (max_items or 0) > MAX_ARRAY_ITEMS:
         raise SchemaError(f"an array holds at most {MAX_ARRAY_ITEMS} items (on {name!r})")
+    return min_items, max_items
+
+
+def _boxes(field: Mapping[str, Any]) -> bool:
+    """Whether an array's items are boxes."""
+    items = field.get("items")
+    return isinstance(items, Mapping) and items.get("type") == "box"
+
+
+def _box_list(name: str, field: Mapping[str, Any]) -> Decision:
+    """An array of boxes: every match for the instructions in the image, from none to maxItems."""
+    if set(field["items"]) != {"type"}:
+        raise SchemaError(f"the items of {name!r} take only their type, box: "
+                          "say what to find in the array's instructions")
+    if "continue_from" in field:
+        raise SchemaError(f"continue_from is not supported on an array of boxes ({name!r})")
+    min_items, max_items = _item_bounds(name, field)
+    question = _question(name, field, f'Every "{name}" in the image.')
+    return Decision(name, question, (), "Box", box=True,
+                    box_list=(min_items, MAX_ARRAY_ITEMS if max_items is None else max_items))
+
+
+def _array(name: str, field: Mapping[str, Any]) -> ArrayField:
+    """An array field, its items compiled and linked; its own depends_on and when come later."""
+    question = _question(name, field, f'All the items of "{name}".')
+    if "items" not in field:
+        raise SchemaError(f"array {name!r} needs items")
+    items = field["items"]
+    if not isinstance(items, Mapping):
+        raise SchemaError(f"items for {name!r} must be a schema object")
+    min_items, max_items = _item_bounds(name, field)
     label = f"{name}.items"
     kind = _kind(label, items)
     if kind == "array":
@@ -550,6 +595,16 @@ def _link(entries: Sequence[_Entry]) -> list[Decision]:
             compiled.append(replace(decision, depends_on=(dependencies or ()) + (effort,), effort_from=effort,
                                     when=when, dependency_groups=(groups_t or ()) + ((effort,),)))
             continue
+        if decision.box_list and decision.box_list[0] == 0:
+            present = f"{decision.name}.present"
+            # Asked with the list's own dependencies and condition, one layer before it; no: the list is empty.
+            compiled.append(Decision(present, PRESENCE_QUESTION.format(question=_describe(decision)),
+                                     (True, False), "Bool", depends_on=dependencies, thinking=False,
+                                     presence_for=decision.name, when=when, dependency_groups=groups_t,
+                                     path=(decision.name, "present")))
+            compiled.append(replace(decision, depends_on=(dependencies or ()) + (present,), presence_from=present,
+                                    when=when, dependency_groups=(groups_t or ()) + ((present,),)))
+            continue
         compiled.append(replace(decision, depends_on=dependencies, thinking=thinking, thinking_budget=budget,
                                 when=when, dependency_groups=groups_t))
     return compiled
@@ -576,6 +631,8 @@ def _scalar(name: str, label: str, field: Mapping[str, Any], limit: int | None =
         if len(field_type) != 2 or len(kinds) != 1 or not isinstance(kinds[0], str):
             raise SchemaError(f'type for {label!r} must be one type or [type, "null"]')
         field_type, nullable = kinds[0], True
+    if field_type == "box":
+        return _box(name, label, question, field, nullable)
     if "levels" in field:
         return _score(name, label, question, field, field_type, nullable)
     enum = field.get("enum")
@@ -682,6 +739,20 @@ def _scalar(name: str, label: str, field: Mapping[str, Any], limit: int | None =
                     permutations=permutations, nullable=nullable, descriptions=descriptions)
 
 
+def _box(name: str, label: str, question: str, field: Mapping[str, Any], nullable: bool) -> Decision:
+    """A box: where the instructions' target is in the image. There is always one; a boolean field
+    asks whether the target is there."""
+    if nullable:
+        raise SchemaError(f"a box ({label!r}) cannot be null: it always returns a box; "
+                          "ask a boolean field whether the target is there")
+    for keyword in ("enum", "choices", "levels", "return_probabilities", "permutations"):
+        if keyword in field:
+            raise SchemaError(f"{keyword} is not supported on a box ({label!r})")
+    if "instructions" not in field and "description" not in field:
+        question = f'Locate "{label}" in the image.'
+    return Decision(name, question, (), "Box", box=True)
+
+
 def _score(name: str, label: str, question: str, field: Mapping[str, Any], field_type: Any,
            nullable: bool) -> Decision:
     """A score: a number from ordered "levels", [{"label": ..., "description": ...}, ...], lowest first.
@@ -786,6 +857,9 @@ def _condition(name: str, field: Mapping[str, Any], by_name: Mapping[str, Decisi
         if parent not in by_name:
             raise SchemaError(f"when for {name!r} names {parent!r}, which is not a field")
         source = by_name[parent]
+        if source.box:
+            raise SchemaError(f"when for {name!r} names {parent!r}, a box; a condition can test a box's presence "
+                              "with a boolean field instead")
         if not isinstance(test, Mapping):
             test = {"in": test if isinstance(test, list) else [test]}
         if not test:

{typellm-0.7.0 → typellm-0.7.2}/typellm.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: typellm
-Version: 0.7.0
+Version: 0.7.2
 Summary: Type-Safe Decoding for Autoregressive LLMs
 License-Expression: Apache-2.0
 Project-URL: Homepage, https://typellm.ai/
@@ -31,6 +31,9 @@ Dynamic: license-file
 
 ## Updates
 - 🚀 **[TypeLLM API](https://typellm.ai)** is live: typed outputs without serving a model, $5 free credit. [Try it](https://typellm.ai/dashboard/playground) · [Quick start](#typellm-api-cloud) · [Examples](https://typellm.ai/examples).
+- **[2026/10/15]** An [array of boxes](#boxes) finds every match in an image: every dog, every line item.
+- **[2026/10/15]** Added [boxes](#boxes): find something in an image, returned as its edges.
+- **[2026/10/14]** Added [tools and human input](#tools-and-human-input): a workflow pauses for answers from your own tools or a person, and resumes with them.
 - **[2026/10/08]** [Conditional fields](#conditional-fields) can test how sure an answer is: `"when": {"category": {"confidence": {"gte": 0.8}}}`.
 - **[2026/10/05]** Added [objects and arrays](#objects-and-arrays): group typed properties into one value, or return a variable number of typed items.
 - **[2026/10/01]** Added [conditional fields](#conditional-fields): a field with `when` is answered only if its dependencies meet conditions.
@@ -118,6 +121,23 @@ client = TypeLLMClient(
 )
 ```
 
+| Setting | Default | Purpose |
+| --- | --- | --- |
+| `base_url` | `"http://127.0.0.1:30000"` | SGLang server URL. |
+| `model` | `None` | Model ID; discovered from the server when omitted. |
+| `tokenizer` | `None` | Matching tokenizer ID or local path; when omitted, the server's paths, then its served model name. |
+| `text_max_tokens` | `128` | Text generation budget per field. |
+| `numeric_max_digits` | `32` | Digit limit for open numeric fields. |
+| `temperature` | `0` | 0 gives each field its most likely answer; above 0 samples at that temperature. |
+| `seed` | `None` | Client random-number seed. |
+| `timeout` | `120.0` | HTTP request timeout in seconds. |
+
+On your own server, no service limits apply: your server's context length,
+memory and timeouts are the only bounds. An enum takes up to 26 values, and a
+field that thinks without a `thinking_budget` may reason until the context is
+full. `label_pool` customizes the internal candidate labels; leave it unset for
+normal use. For debugging, `print_final_prompt=True` prints the prompts sent.
+
 ### Make a request
 
 With either client:
@@ -419,6 +439,44 @@ Each image can be a local file path, an http(s) URL, a `data:` URI, raw bytes,
 or a PIL image. Local files are read by the client, so the SGLang server does
 not need access to your filesystem.
 
+### Boxes
+
+A `box` field finds something in the image and returns where it is: `top`,
+`bottom`, `left` and `right`, each from 0 to 1 across the image, from its
+top-left corner.
+
+```python
+response = client.generate(
+    context="Read the receipt photo.",
+    images=["receipt.jpg"],
+    questions={"total_line": {"type": "box", "instructions": "The line that shows the total."}},
+)
+response.result  # {"total_line": {"top": 0.652, "bottom": 0.689, "left": 0.287, "right": 0.855}}
+```
+
+Multiply by the image's width and height for pixels. There is always a box: to
+check that the target is there, ask a boolean field and give the box a `when` on
+it, so the box is asked only when the target is there. A box cannot be null,
+sit inside an object's or an array's items, or be tested by a `when`.
+
+An array of boxes finds every match, from none to `maxItems` (at most 50):
+
+```python
+response = client.generate(
+    context="Read the receipt photo.",
+    images=["receipt.jpg"],
+    questions={"items": {"type": "array", "items": {"type": "box"},
+                         "instructions": "Every line item, each the whole line from its name to its price."}},
+)
+response.result  # {"items": [{"top": 0.421, "bottom": 0.461, "left": 0.12, "right": 0.837},
+                 #            {"top": 0.449, "bottom": 0.493, "left": 0.12, "right": 0.834},
+                 #            {"top": 0.48, "bottom": 0.521, "left": 0.118, "right": 0.829}]}
+```
+
+It returns `[]` when the image has no match. Its `items` take only
+`{"type": "box"}`: say what to find in the array's `instructions`. `minItems`
+and `maxItems` bound it; `continue_from` is not supported.
+
 ## Dependency-aware generation
 
 Fields run together by default, and each sees only the original context.
@@ -516,6 +574,40 @@ skipped field are skipped too, even when their other dependencies ran. A skipped
 field sends no requests; only its definition counts toward the call's input
 tokens.
 
+### Tools and human input
+
+Some answers come from outside the model: an order's status from your database,
+a result from your own tool, or a person's decision. On the hosted API, a field
+with `"source": "user"` is answered by your code. The call answers every field
+that does not wait on it, then stops with `status` `"needs_input"`, and
+`client.resume()` goes on with your answers:
+
+```python
+response = client.generate(context=email, questions={
+    "order_id": {"type": ["string", "null"], "instructions": "The order number in the email."},
+    "order_status": {"type": "string", "enum": ["shipped", "delivered", "lost"],
+                     "source": "user", "when": {"order_id": {"ne": None}}},
+    "reply": {"type": "string", "depends_on": ["order_status"]},
+})
+while response.status == "needs_input":   # response.needs_input names the fields
+    response = client.resume(response, {"order_status": lookup(response.result["order_id"])})
+```
+
+On your own server, pass answers you already have in `given`, as `result` gives
+them, and fields to leave out in `skipped`. The model is not asked those fields,
+and the fields that depend on them see them as answers:
+
+```python
+response = client.generate(context=email, questions=questions,
+                           given={"order_status": "lost"}, skipped=["refund"])
+```
+
+Passing a call's `result` and `skipped` back, with more fields, answers only the
+new ones. A field with `return_probabilities` is given as `result` has it, with
+its probabilities.
+
+[Docs](https://typellm.ai/docs/workflows#tools-and-human-input) · [Cookbook](https://typellm.ai/cookbook/tools-and-human-input)
+
 ## Probabilities and sampling
 
 Set `return_probabilities` on individual enum or boolean fields:

{typellm-0.7.0 → typellm-0.7.2}/typellm.egg-info/SOURCES.txt

@@ -2,6 +2,7 @@ LICENSE
 README.md
 pyproject.toml
 tests/test_batching.py
+tests/test_box.py
 tests/test_choices.py
 tests/test_concurrency.py
 tests/test_conditions.py