pixeltable 0.2.17__tar.gz → 0.2.18__tar.gz

{pixeltable-0.2.17 → pixeltable-0.2.18}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pixeltable
-Version: 0.2.17
+Version: 0.2.18
 Summary: Pixeltable: The Multimodal AI Data Plane
 Author: Pixeltable, Inc.
 Author-email: contact@pixeltable.com
@@ -16,7 +16,6 @@ Requires-Dist: cloudpickle (>=2.2.1,<3.0.0)
 Requires-Dist: ftfy (>=6.2.0,<7.0.0)
 Requires-Dist: jinja2 (>=3.1.3,<4.0.0)
 Requires-Dist: jmespath (>=1.0.1,<2.0.0)
-Requires-Dist: mistune (>=3.0.2,<4.0.0)
 Requires-Dist: more-itertools (>=10.2,<11.0)
 Requires-Dist: numpy (>=1.25)
 Requires-Dist: opencv-python-headless (>=4.7.0.68,<5.0.0.0)
@@ -26,19 +25,18 @@ Requires-Dist: pillow (>=9.3.0)
 Requires-Dist: pixeltable-pgserver (==0.2.7)
 Requires-Dist: psutil (>=5.9.5,<6.0.0)
 Requires-Dist: psycopg[binary] (==3.1.18)
+Requires-Dist: puremagic (>=1.20)
 Requires-Dist: pymupdf (>=1.24.1,<2.0.0)
 Requires-Dist: pyyaml (>=6.0.1,<7.0.0)
 Requires-Dist: requests (>=2.31.0,<3.0.0)
-Requires-Dist: setuptools (==69.1.1)
-Requires-Dist: sqlalchemy[mypy] (>=2.0.23,<3.0.0)
+Requires-Dist: sqlalchemy (>=2.0.23,<3.0.0)
 Requires-Dist: tenacity (>=8.2,<9.0)
 Requires-Dist: tqdm (>=4.64)
 Description-Content-Type: text/markdown
 
 <div align="center">
-<img src="https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/release/pixeltable-banner.png" alt="Pixeltable" width="45%" />
-
-# Unifying Data, Models, and Orchestration for AI Products
+<img src="https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/source/data/pixeltable-logo-large.png" alt="Pixeltable" width="50%" />
+<br></br>
 
 [![License](https://img.shields.io/badge/License-Apache%202.0-darkblue.svg)](https://opensource.org/licenses/Apache-2.0)
 ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pixeltable?logo=python&logoColor=white)
@@ -49,22 +47,17 @@ Description-Content-Type: text/markdown
 [Installation](https://pixeltable.github.io/pixeltable/getting-started/) | [Documentation](https://pixeltable.readme.io/) | [API Reference](https://pixeltable.github.io/pixeltable/) | [Code Samples](https://pixeltable.readme.io/recipes) | [Examples](https://github.com/pixeltable/pixeltable/tree/release/docs/release/tutorials)
 </div>
 
-Pixeltable is a Python library that lets ML Engineers and Data Scientists focus on exploration, modeling, and app development without dealing with the customary data plumbing.
-
-### What problems does Pixeltable solve?
-
-Today’s solutions for AI app development require extensive custom coding and infrastructure plumbing. Tracking lineage and versions between and across data transformations, models, and deployment is cumbersome.
+Pixeltable is a Python library providing a declarative interface for multimodal data (text, images, audio, video). It features built-in versioning, lineage tracking, and incremental updates, enabling users to store, transform, index, and iterate on data for their ML workflows. Data transformations, model inference, and custom logic are embedded as computed columns.
 
 ## 💾 Installation
 
 ```python
 pip install pixeltable
 ```
-> [!IMPORTANT]
-> Pixeltable is persistent. Unlike in-memory Python libraries such as Pandas, Pixeltable is a database. When working locally or against an hosted version of Pixeltable, use [get_table](https://pixeltable.github.io/pixeltable/api/pixeltable/#pixeltable.get_table) at any time to retrieve an existing table.
+**Pixeltable is persistent. Unlike in-memory Python libraries such as Pandas, Pixeltable is a database.**
 
 ## 💡 Getting Started
-Learn how to create tables, populate them with data, and enhance them with built-in or user-defined transformations and AI operations.
+Learn how to create tables, populate them with data, and enhance them with built-in or user-defined transformations.
 
 | Topic | Notebook | Topic | Notebook |
 |:----------|:-----------------|:-------------------------|:---------------------------------:|
@@ -91,9 +84,32 @@ v.insert({'video': prefix + p} for p in paths)
 ```
 Learn how to [work with data in Pixeltable](https://pixeltable.readme.io/docs/working-with-external-files).
 
-### Add an object detection model to your workflow
+### Object detection in images using DETR model
 ```python
-table['detections'] = huggingface.detr_for_object_detection(table.input_image, model_id='facebook/detr-resnet-50')
+import pixeltable as pxt
+from pixeltable.functions import huggingface
+
+# Create a table to store data persistently
+t = pxt.create_table('image', {'image': pxt.ImageType()})
+
+# Insert some images
+prefix = 'https://upload.wikimedia.org/wikipedia/commons'
+paths = [
+    '/1/15/Cat_August_2010-4.jpg',
+    '/e/e1/Example_of_a_Dog.jpg',
+    '/thumb/b/bf/Bird_Diversity_2013.png/300px-Bird_Diversity_2013.png'
+]
+t.insert({'image': prefix + p} for p in paths)
+
+# Add a computed column for image classification
+t['classification'] = huggingface.detr_for_object_detection(
+    (t.image), model_id='facebook/detr-resnet-50'
+    )
+
+# Retrieve the rows where cats have been identified
+t.select(animal = t.image,
+         classification = t.classification.label_text[0]) \
+.where(t.classification.label_text[0]=='cat').head()
 ```
 Learn about computed columns and object detection: [Comparing object detection models](https://pixeltable.readme.io/docs/object-detection-in-videos).
 
@@ -109,9 +125,9 @@ def draw_boxes(img: PIL.Image.Image, boxes: list[list[float]]) -> PIL.Image.Imag
 ```
 Learn more about user-defined functions: [UDFs in Pixeltable](https://pixeltable.readme.io/docs/user-defined-functions-udfs).
 
-### Automate data operations with views
+### Automate data operations with views, e.g., split documents into chunks
 ```python
-# In this example, the view is defined by iteration over the chunks of a DocumentSplitter.
+# In this example, the view is defined by iteration over the chunks of a DocumentSplitter
 chunks_table = pxt.create_view(
     'rag_demo.chunks',
     documents_table,
@@ -124,7 +140,7 @@ Learn how to leverage views to build your [RAG workflow](https://pixeltable.read
 
 ### Evaluate model performance
 ```python
-# The computation of the mAP metric can simply become a query over the evaluation output, aggregated with the mean_ap() function.
+# The computation of the mAP metric can become a query over the evaluation output
 frames_view.select(mean_ap(frames_view.eval_yolox_tiny), mean_ap(frames_view.eval_yolox_m)).show()
 ```
 Learn how to leverage Pixeltable for [Model analytics](https://pixeltable.readme.io/docs/object-detection-in-videos).
@@ -136,7 +152,7 @@ chat_table = pxt.create_table('together_demo.chat', {'input': pxt.StringType()})
 # The chat-completions API expects JSON-formatted input:
 messages = [{'role': 'user', 'content': chat_table.input}]
 
-# This example shows how additional parameters from the Together API can be used in Pixeltable to customize the model behavior.
+# This example shows how additional parameters from the Together API can be used in Pixeltable
 chat_table['output'] = chat_completions(
     messages=messages,
     model='mistralai/Mixtral-8x7B-Instruct-v0.1',
@@ -160,12 +176,54 @@ chat_table.select(chat_table.input, chat_table.response).head()
 ```
 Learn how to interact with inference services such as [Together AI](https://pixeltable.readme.io/docs/together-ai) in Pixeltable.
 
+### Text and image similarity search on video frames with embedding indexes
+```python
+import pixeltable as pxt
+from pixeltable.functions.huggingface import clip_image, clip_text
+from pixeltable.iterators import FrameIterator
+import PIL.Image
+
+video_table = pxt.create_table('videos', {'video': pxt.VideoType()})
+
+video_table.insert([{'video': '/video.mp4'}])
+
+frames_view = pxt.create_view(
+    'frames', video_table, iterator=FrameIterator.create(video=video_table.video))
+
+@pxt.expr_udf
+def embed_image(img: PIL.Image.Image):
+    return clip_image(img, model_id='openai/clip-vit-base-patch32')
+
+@pxt.expr_udf
+def str_embed(s: str):
+    return clip_text(s, model_id='openai/clip-vit-base-patch32')
+
+# Create an index on the 'frame' column that allows text and image search
+frames_view.add_embedding_index('frame', string_embed=str_embed, image_embed=embed_image)
+
+# Now we will retrieve images based on a sample image
+sample_image = '/image.jpeg'
+sim = frames_view.frame.similarity(sample_image)
+frames_view.order_by(sim, asc=False).limit(5).select(frames_view.frame, sim=sim).collect()
+
+# Now we will retrieve images based on a string
+sample_text = 'red truck'
+sim = frames_view.frame.similarity(sample_text)
+frames_view.order_by(sim, asc=False).limit(5).select(frames_view.frame, sim=sim).collect()
+
+```
+Learn how to work with [Embedding and Vector Indexes](https://docs.pixeltable.com/docs/embedding-vector-indexes).
+
 ## ❓ FAQ
 
 ### What is Pixeltable?
 
 Pixeltable unifies data storage, versioning, and indexing with orchestration and model versioning under a declarative table interface, with transformations, model inference, and custom logic represented as computed columns.
 
+### What problems does Pixeltable solve?
+
+Today's solutions for AI app development require extensive custom coding and infrastructure plumbing. Tracking lineage and versions between and across data transformations, models, and deployments is cumbersome. Pixeltable lets ML Engineers and Data Scientists focus on exploration, modeling, and app development without dealing with the customary data plumbing.
+
 ### What does Pixeltable provide me with? Pixeltable provides:
 
 - Data storage and versioning

{pixeltable-0.2.17 → pixeltable-0.2.18}/README.md

@@ -1,7 +1,6 @@
 <div align="center">
-<img src="https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/release/pixeltable-banner.png" alt="Pixeltable" width="45%" />
-
-# Unifying Data, Models, and Orchestration for AI Products
+<img src="https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/source/data/pixeltable-logo-large.png" alt="Pixeltable" width="50%" />
+<br></br>
 
 [![License](https://img.shields.io/badge/License-Apache%202.0-darkblue.svg)](https://opensource.org/licenses/Apache-2.0)
 ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pixeltable?logo=python&logoColor=white)
@@ -12,22 +11,17 @@
 [Installation](https://pixeltable.github.io/pixeltable/getting-started/) | [Documentation](https://pixeltable.readme.io/) | [API Reference](https://pixeltable.github.io/pixeltable/) | [Code Samples](https://pixeltable.readme.io/recipes) | [Examples](https://github.com/pixeltable/pixeltable/tree/release/docs/release/tutorials)
 </div>
 
-Pixeltable is a Python library that lets ML Engineers and Data Scientists focus on exploration, modeling, and app development without dealing with the customary data plumbing.
-
-### What problems does Pixeltable solve?
-
-Today’s solutions for AI app development require extensive custom coding and infrastructure plumbing. Tracking lineage and versions between and across data transformations, models, and deployment is cumbersome.
+Pixeltable is a Python library providing a declarative interface for multimodal data (text, images, audio, video). It features built-in versioning, lineage tracking, and incremental updates, enabling users to store, transform, index, and iterate on data for their ML workflows. Data transformations, model inference, and custom logic are embedded as computed columns.
 
 ## 💾 Installation
 
 ```python
 pip install pixeltable
 ```
-> [!IMPORTANT]
-> Pixeltable is persistent. Unlike in-memory Python libraries such as Pandas, Pixeltable is a database. When working locally or against an hosted version of Pixeltable, use [get_table](https://pixeltable.github.io/pixeltable/api/pixeltable/#pixeltable.get_table) at any time to retrieve an existing table.
+**Pixeltable is persistent. Unlike in-memory Python libraries such as Pandas, Pixeltable is a database.**
 
 ## 💡 Getting Started
-Learn how to create tables, populate them with data, and enhance them with built-in or user-defined transformations and AI operations.
+Learn how to create tables, populate them with data, and enhance them with built-in or user-defined transformations.
 
 | Topic | Notebook | Topic | Notebook |
 |:----------|:-----------------|:-------------------------|:---------------------------------:|
@@ -54,9 +48,32 @@ v.insert({'video': prefix + p} for p in paths)
 ```
 Learn how to [work with data in Pixeltable](https://pixeltable.readme.io/docs/working-with-external-files).
 
-### Add an object detection model to your workflow
+### Object detection in images using DETR model
 ```python
-table['detections'] = huggingface.detr_for_object_detection(table.input_image, model_id='facebook/detr-resnet-50')
+import pixeltable as pxt
+from pixeltable.functions import huggingface
+
+# Create a table to store data persistently
+t = pxt.create_table('image', {'image': pxt.ImageType()})
+
+# Insert some images
+prefix = 'https://upload.wikimedia.org/wikipedia/commons'
+paths = [
+    '/1/15/Cat_August_2010-4.jpg',
+    '/e/e1/Example_of_a_Dog.jpg',
+    '/thumb/b/bf/Bird_Diversity_2013.png/300px-Bird_Diversity_2013.png'
+]
+t.insert({'image': prefix + p} for p in paths)
+
+# Add a computed column for image classification
+t['classification'] = huggingface.detr_for_object_detection(
+    (t.image), model_id='facebook/detr-resnet-50'
+    )
+
+# Retrieve the rows where cats have been identified
+t.select(animal = t.image,
+         classification = t.classification.label_text[0]) \
+.where(t.classification.label_text[0]=='cat').head()
 ```
 Learn about computed columns and object detection: [Comparing object detection models](https://pixeltable.readme.io/docs/object-detection-in-videos).
 
@@ -72,9 +89,9 @@ def draw_boxes(img: PIL.Image.Image, boxes: list[list[float]]) -> PIL.Image.Imag
 ```
 Learn more about user-defined functions: [UDFs in Pixeltable](https://pixeltable.readme.io/docs/user-defined-functions-udfs).
 
-### Automate data operations with views
+### Automate data operations with views, e.g., split documents into chunks
 ```python
-# In this example, the view is defined by iteration over the chunks of a DocumentSplitter.
+# In this example, the view is defined by iteration over the chunks of a DocumentSplitter
 chunks_table = pxt.create_view(
     'rag_demo.chunks',
     documents_table,
@@ -87,7 +104,7 @@ Learn how to leverage views to build your [RAG workflow](https://pixeltable.read
 
 ### Evaluate model performance
 ```python
-# The computation of the mAP metric can simply become a query over the evaluation output, aggregated with the mean_ap() function.
+# The computation of the mAP metric can become a query over the evaluation output
 frames_view.select(mean_ap(frames_view.eval_yolox_tiny), mean_ap(frames_view.eval_yolox_m)).show()
 ```
 Learn how to leverage Pixeltable for [Model analytics](https://pixeltable.readme.io/docs/object-detection-in-videos).
@@ -99,7 +116,7 @@ chat_table = pxt.create_table('together_demo.chat', {'input': pxt.StringType()})
 # The chat-completions API expects JSON-formatted input:
 messages = [{'role': 'user', 'content': chat_table.input}]
 
-# This example shows how additional parameters from the Together API can be used in Pixeltable to customize the model behavior.
+# This example shows how additional parameters from the Together API can be used in Pixeltable
 chat_table['output'] = chat_completions(
     messages=messages,
     model='mistralai/Mixtral-8x7B-Instruct-v0.1',
@@ -123,12 +140,54 @@ chat_table.select(chat_table.input, chat_table.response).head()
 ```
 Learn how to interact with inference services such as [Together AI](https://pixeltable.readme.io/docs/together-ai) in Pixeltable.
 
+### Text and image similarity search on video frames with embedding indexes
+```python
+import pixeltable as pxt
+from pixeltable.functions.huggingface import clip_image, clip_text
+from pixeltable.iterators import FrameIterator
+import PIL.Image
+
+video_table = pxt.create_table('videos', {'video': pxt.VideoType()})
+
+video_table.insert([{'video': '/video.mp4'}])
+
+frames_view = pxt.create_view(
+    'frames', video_table, iterator=FrameIterator.create(video=video_table.video))
+
+@pxt.expr_udf
+def embed_image(img: PIL.Image.Image):
+    return clip_image(img, model_id='openai/clip-vit-base-patch32')
+
+@pxt.expr_udf
+def str_embed(s: str):
+    return clip_text(s, model_id='openai/clip-vit-base-patch32')
+
+# Create an index on the 'frame' column that allows text and image search
+frames_view.add_embedding_index('frame', string_embed=str_embed, image_embed=embed_image)
+
+# Now we will retrieve images based on a sample image
+sample_image = '/image.jpeg'
+sim = frames_view.frame.similarity(sample_image)
+frames_view.order_by(sim, asc=False).limit(5).select(frames_view.frame, sim=sim).collect()
+
+# Now we will retrieve images based on a string
+sample_text = 'red truck'
+sim = frames_view.frame.similarity(sample_text)
+frames_view.order_by(sim, asc=False).limit(5).select(frames_view.frame, sim=sim).collect()
+
+```
+Learn how to work with [Embedding and Vector Indexes](https://docs.pixeltable.com/docs/embedding-vector-indexes).
+
 ## ❓ FAQ
 
 ### What is Pixeltable?
 
 Pixeltable unifies data storage, versioning, and indexing with orchestration and model versioning under a declarative table interface, with transformations, model inference, and custom logic represented as computed columns.
 
+### What problems does Pixeltable solve?
+
+Today's solutions for AI app development require extensive custom coding and infrastructure plumbing. Tracking lineage and versions between and across data transformations, models, and deployments is cumbersome. Pixeltable lets ML Engineers and Data Scientists focus on exploration, modeling, and app development without dealing with the customary data plumbing.
+
 ### What does Pixeltable provide me with? Pixeltable provides:
 
 - Data storage and versioning

{pixeltable-0.2.17 → pixeltable-0.2.18}/pixeltable/__version__.py

@@ -1,3 +1,3 @@
 # These version placeholders will be replaced during build.
-__version__ = "0.2.17"
-__version_tuple__ = (0, 2, 17)
+__version__ = "0.2.18"
+__version_tuple__ = (0, 2, 18)

{pixeltable-0.2.17 → pixeltable-0.2.18}/pixeltable/catalog/catalog.py

@@ -1,8 +1,9 @@
 from __future__ import annotations
-from typing import Optional, List, Any, Dict, Tuple
-from uuid import UUID
+
 import dataclasses
 import logging
+from typing import Optional
+from uuid import UUID
 
 import sqlalchemy as sql
 import sqlalchemy.orm as orm
@@ -10,8 +11,8 @@ import sqlalchemy.orm as orm
 from .table_version import TableVersion
 from .table_version_path import TableVersionPath
 from .table import Table
-from .named_function import NamedFunction
 from .path_dict import PathDict
+
 import pixeltable.env as env
 import pixeltable.metadata.schema as schema
 
@@ -39,10 +40,10 @@ class Catalog:
         # key: [id, version]
         # - mutable version of a table: version == None (even though TableVersion.version is set correctly)
         # - snapshot versions: records the version of the snapshot
-        self.tbl_versions: Dict[Tuple[UUID, Optional[int]], TableVersion] = {}
+        self.tbl_versions: dict[tuple[UUID, Optional[int]], TableVersion] = {}
 
-        self.tbls: Dict[UUID, Table] = {}  # don't use a defaultdict here, it doesn't cooperate with the debugger
-        self.tbl_dependents: Dict[UUID, List[Table]] = {}
+        self.tbls: dict[UUID, Table] = {}  # don't use a defaultdict here, it doesn't cooperate with the debugger
+        self.tbl_dependents: dict[UUID, list[Table]] = {}
 
         self._init_store()
         self.paths = PathDict()  # do this after _init_catalog()
@@ -133,7 +134,7 @@ class Catalog:
                         base_path=base_path if not is_snapshot else None)
                     view_path = TableVersionPath(tbl_version, base=base_path)
 
-                tbl = View(
+                tbl: Table = View(
                     tbl_record.id, tbl_record.dir_id, tbl_md.name, view_path, base_tbl_id,
                     snapshot_only=snapshot_only)
                 self.tbl_dependents[base_tbl_id].append(tbl)

{pixeltable-0.2.17 → pixeltable-0.2.18}/pixeltable/catalog/column.py

@@ -1,15 +1,19 @@
 from __future__ import annotations
 
 import logging
-from typing import Any, Callable, Optional, Union
+from typing import TYPE_CHECKING, Any, Callable, Optional, Union
 
 import sqlalchemy as sql
 
 import pixeltable.exceptions as excs
 import pixeltable.type_system as ts
+from pixeltable import exprs
 
 from .globals import is_valid_identifier
 
+if TYPE_CHECKING:
+    from .table_version import TableVersion
+
 _logger = logging.getLogger('pixeltable')
 
 class Column:
@@ -20,7 +24,7 @@ class Column:
     """
     def __init__(
             self, name: Optional[str], col_type: Optional[ts.ColumnType] = None,
-            computed_with: Optional[Union['Expr', Callable]] = None,
+            computed_with: Optional[Union[exprs.Expr, Callable]] = None,
             is_pk: bool = False, stored: bool = True,
             col_id: Optional[int] = None, schema_version_add: Optional[int] = None,
             schema_version_drop: Optional[int] = None, sa_col_type: Optional[sql.sqltypes.TypeEngine] = None,
@@ -57,15 +61,14 @@ class Column:
         if col_type is None and computed_with is None:
             raise excs.Error(f'Column `{name}`: col_type is required if computed_with is not specified')
 
-        self._value_expr: Optional['Expr'] = None
+        self._value_expr: Optional[exprs.Expr] = None
         self.compute_func: Optional[Callable] = None
         self.value_expr_dict = value_expr_dict
-        from pixeltable import exprs
         if computed_with is not None:
             value_expr = exprs.Expr.from_object(computed_with)
             if value_expr is None:
                 # computed_with needs to be a Callable
-                if not isinstance(computed_with, Callable):
+                if not callable(computed_with):
                     raise excs.Error(
                         f'Column {name}: computed_with needs to be either a Pixeltable expression or a Callable, '
                         f'but it is a {type(computed_with)}')
@@ -103,7 +106,7 @@ class Column:
         self.tbl: Optional[TableVersion] = None  # set by owning TableVersion
 
     @property
-    def value_expr(self) -> Optional['Expr']:
+    def value_expr(self) -> Optional[exprs.Expr]:
         """Instantiate value_expr on-demand"""
         # TODO: instantiate expr in the c'tor and add an Expr.prepare() that can create additional state after the
         # catalog has been fully loaded; that way, we encounter bugs in the serialization/deserialization logic earlier
@@ -112,7 +115,7 @@ class Column:
             self._value_expr = exprs.Expr.from_dict(self.value_expr_dict)
         return self._value_expr
 
-    def set_value_expr(self, value_expr: 'Expr') -> None:
+    def set_value_expr(self, value_expr: exprs.Expr) -> None:
         self._value_expr = value_expr
         self.value_expr_dict = None
 
@@ -130,7 +133,7 @@ class Column:
         l = list(self.value_expr.subexprs(filter=lambda e: isinstance(e, exprs.FunctionCall) and e.is_window_fn_call))
         return len(l) > 0
 
-    def get_idx_info(self) -> dict[str, 'pixeltable.catalog.TableVersion.IndexInfo']:
+    def get_idx_info(self) -> dict[str, 'TableVersion.IndexInfo']:
         assert self.tbl is not None
         return {name: info for name, info in self.tbl.idxs_by_name.items() if info.col == self}
 

{pixeltable-0.2.17 → pixeltable-0.2.18}/pixeltable/catalog/insertable_table.py

@@ -82,7 +82,7 @@ class InsertableTable(Table):
     @overload
     def insert(self, *, print_stats: bool = False, fail_on_exception: bool = True, **kwargs: Any) -> UpdateStatus: ...
 
-    def insert(
+    def insert(  # type: ignore[misc]
             self, rows: Optional[Iterable[dict[str, Any]]] = None, /, *, print_stats: bool = False,
             fail_on_exception: bool = True, **kwargs: Any
     ) -> UpdateStatus:

{pixeltable-0.2.17 → pixeltable-0.2.18}/pixeltable/catalog/path_dict.py

@@ -2,7 +2,7 @@ from __future__ import annotations
 
 import copy
 import logging
-from typing import Optional, List, Dict, Type
+from typing import Optional
 from uuid import UUID
 
 import sqlalchemy.orm as orm
@@ -10,6 +10,7 @@ import sqlalchemy.orm as orm
 from pixeltable import exceptions as excs
 from pixeltable.env import Env
 from pixeltable.metadata import schema
+
 from .dir import Dir
 from .path import Path
 from .schema_object import SchemaObject
@@ -19,8 +20,8 @@ _logger = logging.getLogger('pixeltable')
 class PathDict:
     """Keep track of all paths in a Db instance"""
     def __init__(self):
-        self.dir_contents: Dict[UUID, Dict[str, SchemaObject]] = {}
-        self.schema_objs: Dict[UUID, SchemaObject] = {}
+        self.dir_contents: dict[UUID, dict[str, SchemaObject]] = {}
+        self.schema_objs: dict[UUID, SchemaObject] = {}
 
         # load dirs
         with orm.Session(Env.get().engine, future=True) as session:
@@ -36,7 +37,8 @@ class PathDict:
         self.root_dir = root_dirs[0]
 
         # build dir_contents
-        def record_dir(dir: Dir) -> None:
+        def record_dir(dir: SchemaObject) -> None:
+            assert isinstance(dir, Dir)
             if dir._id in self.dir_contents:
                 return
             else:
@@ -99,7 +101,7 @@ class PathDict:
         assert to_path.name not in self.dir_contents[to_dir._id]
         self.dir_contents[to_dir._id][to_path.name] = obj
 
-    def check_is_valid(self, path: Path, expected: Optional[Type[SchemaObject]]) -> None:
+    def check_is_valid(self, path: Path, expected: Optional[type[SchemaObject]]) -> None:
         """Check that path is valid and that the object at path has the expected type.
 
         Args:
@@ -124,7 +126,7 @@ class PathDict:
                 obj = self.dir_contents[parent_obj._id][path.name]
                 raise excs.Error(f"{type(obj)._display_name()} '{str(path)}' already exists")
 
-    def get_children(self, parent: Path, child_type: Optional[Type[SchemaObject]], recursive: bool) -> List[Path]:
+    def get_children(self, parent: Path, child_type: Optional[type[SchemaObject]], recursive: bool) -> list[Path]:
         dir = self._resolve_path(parent)
         if not isinstance(dir, Dir):
             raise excs.Error(f'{str(parent)} is a {type(dir)._display_name()}, not a directory')

{pixeltable-0.2.17 → pixeltable-0.2.18}/pixeltable/catalog/table.py

@@ -1,14 +1,16 @@
 from __future__ import annotations
 
 import abc
+import builtins
 import itertools
 import json
 import logging
 from pathlib import Path
-from typing import Any, Callable, Iterable, Literal, Optional, Set, Tuple, Type, Union, overload
+from typing import TYPE_CHECKING, Any, Callable, Iterable, Literal, Optional, Set, Tuple, Type, Union, overload
 from uuid import UUID
 
 import pandas as pd
+import pandas.io.formats.style
 import sqlalchemy as sql
 
 import pixeltable
@@ -26,6 +28,9 @@ from .schema_object import SchemaObject
 from .table_version import TableVersion
 from .table_version_path import TableVersionPath
 
+if TYPE_CHECKING:
+    import torch.utils.data
+
 _logger = logging.getLogger('pixeltable')
 
 class Table(SchemaObject):
@@ -211,23 +216,24 @@ class Table(SchemaObject):
         })
         return df
 
-    def _description_html(self) -> pd.DataFrame:
+    def _description_html(self) -> pandas.io.formats.style.Styler:
         pd_df = self._description()
         # white-space: pre-wrap: print \n as newline
         # th: center-align headings
-        return pd_df.style.set_properties(**{'white-space': 'pre-wrap', 'text-align': 'left'}) \
-            .set_table_styles([dict(selector='th', props=[('text-align', 'center')])]) \
+        return (
+            pd_df.style.set_properties(None, **{'white-space': 'pre-wrap', 'text-align': 'left'})
+            .set_table_styles([dict(selector='th', props=[('text-align', 'center')])])
             .hide(axis='index')
+        )
 
     def describe(self) -> None:
         """
         Print the table schema.
         """
-        try:
-            __IPYTHON__
+        if getattr(builtins, '__IPYTHON__', False):
             from IPython.display import display
             display(self._description_html())
-        except NameError:
+        else:
             print(self.__repr__())
 
     # TODO: Display comments in _repr_html()
@@ -240,7 +246,7 @@ class Table(SchemaObject):
         return f'{self._display_name()} \'{self._name}\'\n{comment}{description_str}'
 
     def _repr_html_(self) -> str:
-        return self._description_html()._repr_html_()
+        return self._description_html()._repr_html_()  # type: ignore[attr-defined]
 
     def _drop(self) -> None:
         self._check_is_dropped()
@@ -282,7 +288,7 @@ class Table(SchemaObject):
             raise excs.Error(f'Column name must be a string, got {type(col_name)}')
         if not isinstance(spec, (ts.ColumnType, exprs.Expr)):
             raise excs.Error(f'Column spec must be a ColumnType or an Expr, got {type(spec)}')
-        self.add_column(**{col_name: spec})
+        self.add_column(type=None, stored=None, print_stats=False, **{col_name: spec})
 
     def add_column(
             self,
@@ -368,7 +374,7 @@ class Table(SchemaObject):
             col_schema['stored'] = stored
 
         new_col = self._create_columns({col_name: col_schema})[0]
-        self._verify_column(new_col, set(self._schema.keys()), self._query_names)
+        self._verify_column(new_col, set(self._schema.keys()), set(self._query_names))
         return self._tbl_version.add_column(new_col, print_stats=print_stats)
 
     @classmethod
@@ -395,7 +401,7 @@ class Table(SchemaObject):
             value_expr = exprs.Expr.from_object(value_spec)
             if value_expr is None:
                 # needs to be a Callable
-                if not isinstance(value_spec, Callable):
+                if not callable(value_spec):
                     raise excs.Error(
                         f'Column {name}: value needs to be either a Pixeltable expression or a Callable, '
                         f'but it is a {type(value_spec)}')
@@ -427,7 +433,7 @@ class Table(SchemaObject):
             elif isinstance(spec, exprs.Expr):
                 # create copy so we can modify it
                 value_expr = spec.copy()
-            elif isinstance(spec, Callable):
+            elif callable(spec):
                 raise excs.Error((
                     f'Column {name} computed with a Callable: specify using a dictionary with '
                     f'the "value" and "type" keys (e.g., "{name}": {{"value": <Callable>, "type": IntType()}})'
@@ -546,6 +552,7 @@ class Table(SchemaObject):
             metric: str = 'cosine'
     ) -> None:
         """Add an index to the table.
+
         Args:
             col_name: name of column to index
             idx_name: name of index, which needs to be unique for the table; if not provided, a name will be generated
@@ -656,7 +663,7 @@ class Table(SchemaObject):
     @overload
     def insert(self, *, print_stats: bool = False, fail_on_exception: bool = True, **kwargs: Any) -> UpdateStatus: ...
 
-    @abc.abstractmethod
+    @abc.abstractmethod  # type: ignore[misc]
     def insert(
             self, rows: Optional[Iterable[dict[str, Any]]] = None, /, *, print_stats: bool = False,
             fail_on_exception: bool = True, **kwargs: Any