vision-agent 0.0.44__tar.gz → 0.0.46__tar.gz

{vision_agent-0.0.44 → vision_agent-0.0.46}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: vision-agent
-Version: 0.0.44
+Version: 0.0.46
 Summary: Toolset for Vision Agent
 Author: Landing AI
 Author-email: dev@landing.ai
@@ -71,8 +71,8 @@ You can interact with the agents as you would with any LLM or LMM model:
 ```python
 >>> import vision_agent as va
 >>> agent = VisionAgent()
->>> agent("How many apples are in this image?", image="apples.jpg")
-"There are 2 apples in the image."
+>>> agent("What percentage of the area of this jar is filled with coffee beans?", image="jar.jpg")
+"The percentage of area of the jar filled with coffee beans is 25%."
 ```
 
 To better understand how the model came up with it's answer, you can also run it in
@@ -86,22 +86,22 @@ You can also have it return the workflow it used to complete the task along with
 the individual steps and tools to get the answer:
 
 ```python
->>> resp, workflow = agent.chat_with_workflow([{"role": "user", "content": "How many apples are in this image?"}], image="apples.jpg")
+>>> resp, workflow = agent.chat_with_workflow([{"role": "user", "content": "What percentage of the area of this jar is filled with coffee beans?"}], image="jar.jpg")
 >>> print(workflow)
-[{"task": "Count the number of apples using 'grounding_dino_'.",
-  "tool": "grounding_dino_",
-  "parameters": {"prompt": "apple", "image": "apples.jpg"},
+[{"task": "Segment the jar using 'grounding_sam_'.",
+  "tool": "grounding_sam_",
+  "parameters": {"prompt": "jar", "image": "jar.jpg"},
   "call_results": [[
     {
-      "labels": ["apple", "apple"],
-      "scores": [0.99, 0.95],
+      "labels": ["jar"],
+      "scores": [0.99],
       "bboxes": [
         [0.58, 0.2, 0.72, 0.45],
-        [0.94, 0.57, 0.98, 0.66],
-      ]
+      ],
+      "masks": "mask.png"
     }
   ]],
-  "answer": "There are 2 apples in the image.",
+  "answer": "The jar is located at [0.58, 0.2, 0.72, 0.45].",
 }]
 ```
 
@@ -113,13 +113,12 @@ you. For example:
 ```python
 >>> import vision_agent as va
 >>> llm = va.llm.OpenAILLM()
->>> detector = llm.generate_detector("Can you build an apple detector for me?")
->>> detector("apples.jpg")
-[{"labels": ["apple", "apple"],
-  "scores": [0.99, 0.95],
+>>> detector = llm.generate_detector("Can you build a jar detector for me?")
+>>> detector("jar.jpg")
+[{"labels": ["jar",],
+  "scores": [0.99],
   "bboxes": [
     [0.58, 0.2, 0.72, 0.45],
-    [0.94, 0.57, 0.98, 0.66],
   ]
 }]
 ```
@@ -133,7 +132,9 @@ you. For example:
 | Crop | Crop crops an image given a bounding box and returns a file name of the cropped image. |
 | BboxArea | BboxArea returns the area of the bounding box in pixels normalized to 2 decimal places. |
 | SegArea | SegArea returns the area of the segmentation mask in pixels normalized to 2 decimal places. |
-| ExtractFrames | ExtractFrames extracts image frames from the input video. |
+| BboxIoU | BboxIoU returns the intersection over union of two bounding boxes normalized to 2 decimal places. |
+| SegIoU | SegIoU returns the intersection over union of two segmentation masks normalized to 2 decimal places. |
+| ExtractFrames | ExtractFrames extracts frames with motion from a video. |
 
 
 It also has a basic set of calculate tools such as add, subtract, multiply and divide.

{vision_agent-0.0.44 → vision_agent-0.0.46}/README.md

@@ -42,8 +42,8 @@ You can interact with the agents as you would with any LLM or LMM model:
 ```python
 >>> import vision_agent as va
 >>> agent = VisionAgent()
->>> agent("How many apples are in this image?", image="apples.jpg")
-"There are 2 apples in the image."
+>>> agent("What percentage of the area of this jar is filled with coffee beans?", image="jar.jpg")
+"The percentage of area of the jar filled with coffee beans is 25%."
 ```
 
 To better understand how the model came up with it's answer, you can also run it in
@@ -57,22 +57,22 @@ You can also have it return the workflow it used to complete the task along with
 the individual steps and tools to get the answer:
 
 ```python
->>> resp, workflow = agent.chat_with_workflow([{"role": "user", "content": "How many apples are in this image?"}], image="apples.jpg")
+>>> resp, workflow = agent.chat_with_workflow([{"role": "user", "content": "What percentage of the area of this jar is filled with coffee beans?"}], image="jar.jpg")
 >>> print(workflow)
-[{"task": "Count the number of apples using 'grounding_dino_'.",
-  "tool": "grounding_dino_",
-  "parameters": {"prompt": "apple", "image": "apples.jpg"},
+[{"task": "Segment the jar using 'grounding_sam_'.",
+  "tool": "grounding_sam_",
+  "parameters": {"prompt": "jar", "image": "jar.jpg"},
   "call_results": [[
     {
-      "labels": ["apple", "apple"],
-      "scores": [0.99, 0.95],
+      "labels": ["jar"],
+      "scores": [0.99],
       "bboxes": [
         [0.58, 0.2, 0.72, 0.45],
-        [0.94, 0.57, 0.98, 0.66],
-      ]
+      ],
+      "masks": "mask.png"
     }
   ]],
-  "answer": "There are 2 apples in the image.",
+  "answer": "The jar is located at [0.58, 0.2, 0.72, 0.45].",
 }]
 ```
 
@@ -84,13 +84,12 @@ you. For example:
 ```python
 >>> import vision_agent as va
 >>> llm = va.llm.OpenAILLM()
->>> detector = llm.generate_detector("Can you build an apple detector for me?")
->>> detector("apples.jpg")
-[{"labels": ["apple", "apple"],
-  "scores": [0.99, 0.95],
+>>> detector = llm.generate_detector("Can you build a jar detector for me?")
+>>> detector("jar.jpg")
+[{"labels": ["jar",],
+  "scores": [0.99],
   "bboxes": [
     [0.58, 0.2, 0.72, 0.45],
-    [0.94, 0.57, 0.98, 0.66],
   ]
 }]
 ```
@@ -104,7 +103,9 @@ you. For example:
 | Crop | Crop crops an image given a bounding box and returns a file name of the cropped image. |
 | BboxArea | BboxArea returns the area of the bounding box in pixels normalized to 2 decimal places. |
 | SegArea | SegArea returns the area of the segmentation mask in pixels normalized to 2 decimal places. |
-| ExtractFrames | ExtractFrames extracts image frames from the input video. |
+| BboxIoU | BboxIoU returns the intersection over union of two bounding boxes normalized to 2 decimal places. |
+| SegIoU | SegIoU returns the intersection over union of two segmentation masks normalized to 2 decimal places. |
+| ExtractFrames | ExtractFrames extracts frames with motion from a video. |
 
 
 It also has a basic set of calculate tools such as add, subtract, multiply and divide.

{vision_agent-0.0.44 → vision_agent-0.0.46}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"
 
 [tool.poetry]
 name = "vision-agent"
-version = "0.0.44"
+version = "0.0.46"
 description = "Toolset for Vision Agent"
 authors = ["Landing AI <dev@landing.ai>"]
 readme = "README.md"