pixeltable 0.2.6__tar.gz → 0.2.7__tar.gz

pixeltable-0.2.7/PKG-INFO

@@ -0,0 +1,137 @@
+Metadata-Version: 2.1
+Name: pixeltable
+Version: 0.2.7
+Summary: Pixeltable: The Multimodal AI Data Plane
+Author: Marcel Kornacker
+Author-email: marcelk@gmail.com
+Requires-Python: >=3.9,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: av (>=10.0.0)
+Requires-Dist: beautifulsoup4 (>=4.0.0,<5.0.0)
+Requires-Dist: cloudpickle (>=2.2.1,<3.0.0)
+Requires-Dist: ftfy (>=6.2.0,<7.0.0)
+Requires-Dist: jinja2 (>=3.1.3,<4.0.0)
+Requires-Dist: jmespath (>=1.0.1,<2.0.0)
+Requires-Dist: mistune (>=3.0.2,<4.0.0)
+Requires-Dist: more-itertools (>=10.2,<11.0)
+Requires-Dist: numpy (>=1.25)
+Requires-Dist: opencv-python-headless (>=4.7.0.68,<5.0.0.0)
+Requires-Dist: pandas (>=2.0,<3.0)
+Requires-Dist: pgserver (==0.1.3)
+Requires-Dist: pgvector (>=0.2.1,<0.3.0)
+Requires-Dist: pillow (>=9.3.0)
+Requires-Dist: psutil (>=5.9.5,<6.0.0)
+Requires-Dist: psycopg2-binary (>=2.9.5,<3.0.0)
+Requires-Dist: pymupdf (>=1.24.1,<2.0.0)
+Requires-Dist: pyyaml (>=6.0.1,<7.0.0)
+Requires-Dist: requests (>=2.31.0,<3.0.0)
+Requires-Dist: setuptools (==69.1.1)
+Requires-Dist: sqlalchemy[mypy] (>=2.0.23,<3.0.0)
+Requires-Dist: tenacity (>=8.2,<9.0)
+Requires-Dist: tqdm (>=4.64)
+Description-Content-Type: text/markdown
+
+<div align="center">
+<img src="https://raw.githubusercontent.com/pixeltable/pixeltable/master/docs/release/pixeltable-banner.png" alt="Pixeltable" width="45%" />
+
+# Unifying Data, Models, and Orchestration for AI Products
+
+[![License](https://img.shields.io/badge/License-Apache%202.0-darkblue.svg)](https://opensource.org/licenses/Apache-2.0)
+![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pixeltable?logo=python&logoColor=white)
+  [![Platform Support](https://img.shields.io/badge/platform-Linux%20%7C%20macOS%20%7C%20Windows-8A2BE2)]()
+[![pytest status](https://github.com/pixeltable/pixeltable/actions/workflows/pytest.yml/badge.svg)](https://github.com/pixeltable/pixeltable/actions)
+[![PyPI Package](https://img.shields.io/pypi/v/pixeltable?color=darkorange)](https://pypi.org/project/pixeltable/)
+
+[Installation](https://pixeltable.github.io/pixeltable/getting-started/) | [Documentation](https://pixeltable.readme.io/) | [API Reference](https://pixeltable.github.io/pixeltable/) | [Code Samples](https://pixeltable.readme.io/recipes) | [Examples](https://github.com/pixeltable/pixeltable/tree/master/docs/release/tutorials)
+</div>
+
+Pixeltable is a Python library that lets AI engineers and data scientists focus on exploration, modeling, and app development without dealing with the customary data plumbing.
+
+## What problems does Pixeltable solve?
+
+Today’s solutions for AI app development require extensive custom coding and infrastructure plumbing. Tracking lineage and versions between and across data transformations, models, and deployment is cumbersome. With Pixeltable you can store, transform, index, and iterate on your data within the same table interface, whether it's text, images, embeddings, or even video. Built-in lineage and versioning ensure transparency and reproducibility, while the development-to-production mirror streamlines deployment.
+
+## 💾 Installation
+
+```python
+%pip install pixeltable
+```
+
+To verify that it's working:
+
+```python
+import pixeltable as pxt
+pxt.init()
+```
+> [!NOTE]
+> Check out the [Pixeltable Basics](https://pixeltable.readme.io/docs/pixeltable-basics) tutorial for a tour of its most important features.
+
+## 💡 Get Started
+Learn how to create tables, populate them with data, and enhance them with built-in or user-defined transformations and AI operations.
+
+| Topic      |                                                                                Notebook                                                                                |                                                                                API                                                                                |
+|:--------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------:|
+| Get Started    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/master/docs/tutorials/pixeltable-basics.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a> |                 [![API](https://img.shields.io/badge/api-reference-blue.svg)](https://pixeltable.github.io/pixeltable/api/pixeltable/)                 |
+| User-Defined Functions (UDFs)    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/master/docs/release/howto/udfs-in-pixeltable.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a> |                 [![API](https://img.shields.io/badge/api-reference-blue.svg)](https://pixeltable.github.io/pixeltable/api/iterators/document-splitter/)                 |
+| Comparing Object Detection Models | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/master/docs/release/tutorials/object-detection-in-videos.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>            | [![API](https://img.shields.io/badge/api-reference-blue.svg)](https://pixeltable.github.io/pixeltable/api-cheat-sheet/#frame-extraction-for-video-data) |
+| Experimenting with Chunking (RAG) | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/master/docs/release/tutorials/rag-operations.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>            | [![API](https://img.shields.io/badge/api-reference-blue.svg)](https://pixeltable.github.io/pixeltable/api/iterators/document-splitter/) |
+| Working with External Files    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/master/docs/release/howto/working-with-external-files.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a> |                 [![API](https://img.shields.io/badge/api-reference-blue.svg)](https://pixeltable.github.io/pixeltable/api-cheat-sheet/#inserting-data-into-a-table)                 |
+
+## ❓ FAQ
+
+### What does Pixeltable provide me with? Pixeltable provides:
+
+- Data storage and versioning
+- Combined Data and Model Lineage
+- Indexing (e.g. embedding vectors) and Data Retrieval
+- Orchestration of multimodal workloads
+- Incremental updates
+- Code is automatically production-ready
+
+### Why should you use Pixeltable?
+
+- **It gives you transparency and reproducibility**
+  - All generated data is automatically recorded and versioned
+  - You will never need to re-run a workload because you lost track of the input data
+- **It saves you money**
+  - All data changes are automatically incremental
+  - You never need to re-run pipelines from scratch because you’re adding data
+- **It integrates with any existing Python code or libraries**
+  - Bring your ever-changing code and workloads
+  - You choose the models, tools, and AI practices (e.g., your embedding model for a vector index); Pixeltable orchestrates the data
+ 
+### What is Pixeltable not providing?
+
+- Pixeltable is not a low-code, prescriptive AI solution. We empower you to use the best frameworks and techniques for your specific needs.
+- We do not aim to replace your existing AI toolkit, but rather enhance it by streamlining the underlying data infrastructure and orchestration.
+
+> [!TIP]
+> Check out the [Integrations](https://pixeltable.readme.io/docs/working-with-openai) section, and feel free to submit a request for additional ones.
+
+## 📙 Example of Use Cases
+
+- **Interact with video data at the frame level** without having to think about frame extraction, intermediate file storage, or storage space explosion.
+- **Augment your data incrementally and interactively with built-in functions and UDFs**, such as image transformations, model inference, and visualizations, without having to think about data pipelines, incremental updates, or capturing function output.
+- **Interact with all the data relevant to your AI application** (video, images, documents, audio, structured data, JSON) through a simple dataframe-style API directly in Python. This includes:
+  - similarity search on embeddings, supported by high-dimensional vector indexing;
+  - path expressions and transformations on JSON data;
+  - PIL and OpenCV image operations;
+  - assembling frames into videos.
+- **Perform keyword and image similarity search at the video frame level** without having to worry about frame storage.
+- **Access all Pixeltable-resident data directly as a PyTorch dataset** in your training scripts.
+- **Understand the compute and storage costs of your data at the granularity** of individual augmentations and get cost projections before adding new data and new augmentations.
+- **Rely on Pixeltable's automatic versioning and snapshot functionality** to protect against regressions and to ensure reproducibility.
+
+## 🐛 Contributions & Feedback
+
+Are you experiencing issues or bugs with Pixeltable? File an [Issue](https://github.com/pixeltable/pixeltable/issues).
+</br>Do you want to contribute? Feel free to open a [PR](https://github.com/pixeltable/pixeltable/pulls).
+
+## :classical_building: License
+
+This library is licensed under the Apache 2.0 License.
+

pixeltable-0.2.7/README.md

@@ -0,0 +1,99 @@
+<div align="center">
+<img src="https://raw.githubusercontent.com/pixeltable/pixeltable/master/docs/release/pixeltable-banner.png" alt="Pixeltable" width="45%" />
+
+# Unifying Data, Models, and Orchestration for AI Products
+
+[![License](https://img.shields.io/badge/License-Apache%202.0-darkblue.svg)](https://opensource.org/licenses/Apache-2.0)
+![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pixeltable?logo=python&logoColor=white)
+  [![Platform Support](https://img.shields.io/badge/platform-Linux%20%7C%20macOS%20%7C%20Windows-8A2BE2)]()
+[![pytest status](https://github.com/pixeltable/pixeltable/actions/workflows/pytest.yml/badge.svg)](https://github.com/pixeltable/pixeltable/actions)
+[![PyPI Package](https://img.shields.io/pypi/v/pixeltable?color=darkorange)](https://pypi.org/project/pixeltable/)
+
+[Installation](https://pixeltable.github.io/pixeltable/getting-started/) | [Documentation](https://pixeltable.readme.io/) | [API Reference](https://pixeltable.github.io/pixeltable/) | [Code Samples](https://pixeltable.readme.io/recipes) | [Examples](https://github.com/pixeltable/pixeltable/tree/master/docs/release/tutorials)
+</div>
+
+Pixeltable is a Python library that lets AI engineers and data scientists focus on exploration, modeling, and app development without dealing with the customary data plumbing.
+
+## What problems does Pixeltable solve?
+
+Today’s solutions for AI app development require extensive custom coding and infrastructure plumbing. Tracking lineage and versions between and across data transformations, models, and deployment is cumbersome. With Pixeltable you can store, transform, index, and iterate on your data within the same table interface, whether it's text, images, embeddings, or even video. Built-in lineage and versioning ensure transparency and reproducibility, while the development-to-production mirror streamlines deployment.
+
+## 💾 Installation
+
+```python
+%pip install pixeltable
+```
+
+To verify that it's working:
+
+```python
+import pixeltable as pxt
+pxt.init()
+```
+> [!NOTE]
+> Check out the [Pixeltable Basics](https://pixeltable.readme.io/docs/pixeltable-basics) tutorial for a tour of its most important features.
+
+## 💡 Get Started
+Learn how to create tables, populate them with data, and enhance them with built-in or user-defined transformations and AI operations.
+
+| Topic      |                                                                                Notebook                                                                                |                                                                                API                                                                                |
+|:--------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------:|
+| Get Started    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/master/docs/tutorials/pixeltable-basics.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a> |                 [![API](https://img.shields.io/badge/api-reference-blue.svg)](https://pixeltable.github.io/pixeltable/api/pixeltable/)                 |
+| User-Defined Functions (UDFs)    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/master/docs/release/howto/udfs-in-pixeltable.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a> |                 [![API](https://img.shields.io/badge/api-reference-blue.svg)](https://pixeltable.github.io/pixeltable/api/iterators/document-splitter/)                 |
+| Comparing Object Detection Models | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/master/docs/release/tutorials/object-detection-in-videos.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>            | [![API](https://img.shields.io/badge/api-reference-blue.svg)](https://pixeltable.github.io/pixeltable/api-cheat-sheet/#frame-extraction-for-video-data) |
+| Experimenting with Chunking (RAG) | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/master/docs/release/tutorials/rag-operations.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>            | [![API](https://img.shields.io/badge/api-reference-blue.svg)](https://pixeltable.github.io/pixeltable/api/iterators/document-splitter/) |
+| Working with External Files    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/master/docs/release/howto/working-with-external-files.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a> |                 [![API](https://img.shields.io/badge/api-reference-blue.svg)](https://pixeltable.github.io/pixeltable/api-cheat-sheet/#inserting-data-into-a-table)                 |
+
+## ❓ FAQ
+
+### What does Pixeltable provide me with? Pixeltable provides:
+
+- Data storage and versioning
+- Combined Data and Model Lineage
+- Indexing (e.g. embedding vectors) and Data Retrieval
+- Orchestration of multimodal workloads
+- Incremental updates
+- Code is automatically production-ready
+
+### Why should you use Pixeltable?
+
+- **It gives you transparency and reproducibility**
+  - All generated data is automatically recorded and versioned
+  - You will never need to re-run a workload because you lost track of the input data
+- **It saves you money**
+  - All data changes are automatically incremental
+  - You never need to re-run pipelines from scratch because you’re adding data
+- **It integrates with any existing Python code or libraries**
+  - Bring your ever-changing code and workloads
+  - You choose the models, tools, and AI practices (e.g., your embedding model for a vector index); Pixeltable orchestrates the data
+ 
+### What is Pixeltable not providing?
+
+- Pixeltable is not a low-code, prescriptive AI solution. We empower you to use the best frameworks and techniques for your specific needs.
+- We do not aim to replace your existing AI toolkit, but rather enhance it by streamlining the underlying data infrastructure and orchestration.
+
+> [!TIP]
+> Check out the [Integrations](https://pixeltable.readme.io/docs/working-with-openai) section, and feel free to submit a request for additional ones.
+
+## 📙 Example of Use Cases
+
+- **Interact with video data at the frame level** without having to think about frame extraction, intermediate file storage, or storage space explosion.
+- **Augment your data incrementally and interactively with built-in functions and UDFs**, such as image transformations, model inference, and visualizations, without having to think about data pipelines, incremental updates, or capturing function output.
+- **Interact with all the data relevant to your AI application** (video, images, documents, audio, structured data, JSON) through a simple dataframe-style API directly in Python. This includes:
+  - similarity search on embeddings, supported by high-dimensional vector indexing;
+  - path expressions and transformations on JSON data;
+  - PIL and OpenCV image operations;
+  - assembling frames into videos.
+- **Perform keyword and image similarity search at the video frame level** without having to worry about frame storage.
+- **Access all Pixeltable-resident data directly as a PyTorch dataset** in your training scripts.
+- **Understand the compute and storage costs of your data at the granularity** of individual augmentations and get cost projections before adding new data and new augmentations.
+- **Rely on Pixeltable's automatic versioning and snapshot functionality** to protect against regressions and to ensure reproducibility.
+
+## 🐛 Contributions & Feedback
+
+Are you experiencing issues or bugs with Pixeltable? File an [Issue](https://github.com/pixeltable/pixeltable/issues).
+</br>Do you want to contribute? Feel free to open a [PR](https://github.com/pixeltable/pixeltable/pulls).
+
+## :classical_building: License
+
+This library is licensed under the Apache 2.0 License.

{pixeltable-0.2.6 → pixeltable-0.2.7}/pixeltable/__init__.py

@@ -1,5 +1,7 @@
 from .catalog import Column, Table, InsertableTable, View
 from .dataframe import DataFrame
+from .datatransfer import Remote
+from .catalog import Column, Table, InsertableTable, View
 from .exceptions import Error, Error
 from .exprs import RELATIVE_PATH_ROOT
 from .func import Function, udf, uda, Aggregator, expr_udf
@@ -21,7 +23,7 @@ from .type_system import (
 from .utils.help import help
 
 # noinspection PyUnresolvedReferences
-from . import functions, io
+from . import functions, io, iterators
 from .__version__ import __version__, __version_tuple__
 
 __all__ = [

{pixeltable-0.2.6 → pixeltable-0.2.7}/pixeltable/__version__.py

@@ -1,3 +1,3 @@
 # These version placeholders will be replaced during build.
-__version__ = "0.2.6"
-__version_tuple__ = (0, 2, 6)
+__version__ = "0.2.7"
+__version_tuple__ = (0, 2, 7)

{pixeltable-0.2.6 → pixeltable-0.2.7}/pixeltable/catalog/column.py

@@ -22,7 +22,8 @@ class Column:
             computed_with: Optional[Union['Expr', Callable]] = None,
             is_pk: bool = False, stored: Optional[bool] = None,
             col_id: Optional[int] = None, schema_version_add: Optional[int] = None,
-            schema_version_drop: Optional[int] = None, sa_col_type: Optional[sql.sqltypes.TypeEngine] = None
+            schema_version_drop: Optional[int] = None, sa_col_type: Optional[sql.sqltypes.TypeEngine] = None,
+            records_errors: Optional[bool] = None
     ):
         """Column constructor.
 
@@ -80,12 +81,19 @@ class Column:
         assert self.col_type is not None
 
         self.stored = stored
-        self.dependent_cols: Set[Column] = set()  # cols with value_exprs that reference us; set by TableVersion
+        self.dependent_cols: set[Column] = set()  # cols with value_exprs that reference us; set by TableVersion
         self.id = col_id
         self.is_pk = is_pk
         self.schema_version_add = schema_version_add
         self.schema_version_drop = schema_version_drop
 
+        # stored_proxy may be set later if this is a non-stored column.
+        # if col1.stored_proxy == col2, then also col1 == col2.proxy_base.
+        self.stored_proxy: Optional[Column] = None
+        self.proxy_base: Optional[Column] = None
+
+        self._records_errors = records_errors
+
         # column in the stored table for the values of this Column
         self.sa_col: Optional[sql.schema.Column] = None
         self.sa_col_type = sa_col_type
@@ -93,6 +101,7 @@ class Column:
         # computed cols also have storage columns for the exception string and type
         self.sa_errormsg_col: Optional[sql.schema.Column] = None
         self.sa_errortype_col: Optional[sql.schema.Column] = None
+
         from .table_version import TableVersion
         self.tbl: Optional[TableVersion] = None  # set by owning TableVersion
 
@@ -131,6 +140,9 @@ class Column:
     @property
     def records_errors(self) -> bool:
         """True if this column also stores error information."""
+        # default: record errors for computed and media columns
+        if self._records_errors is not None:
+            return self._records_errors
         return self.is_stored and (self.is_computed or self.col_type.is_media_type())
 
     def source(self) -> None:

{pixeltable-0.2.6 → pixeltable-0.2.7}/pixeltable/catalog/insertable_table.py

@@ -60,25 +60,29 @@ class InsertableTable(Table):
             return tbl
 
     @overload
-    def insert(self, rows: Iterable[Dict[str, Any]], /, print_stats: bool = False, fail_on_exception: bool = True): ...
+    def insert(
+            self, rows: Iterable[Dict[str, Any]], /, *, print_stats: bool = False, fail_on_exception: bool = True
+    ) -> UpdateStatus: ...
 
     @overload
-    def insert(self, print_stats: bool = False, fail_on_exception: bool = True, **kwargs: Any): ...
+    def insert(self, *, print_stats: bool = False, fail_on_exception: bool = True, **kwargs: Any) -> UpdateStatus: ...
 
-    def insert(self, *args, **kwargs) -> UpdateStatus:
-        """Insert rows into table.
+    def insert(
+            self, rows: Optional[Iterable[dict[str, Any]]] = None, /, *, print_stats: bool = False,
+            fail_on_exception: bool = True, **kwargs: Any
+    ) -> UpdateStatus:
+        """Inserts rows into this table. There are two mutually exclusive call patterns:
 
         To insert multiple rows at a time:
-
-        ``insert(rows: List[Dict[str, Any]], print_stats: bool = False, fail_on_exception: bool = True)``
+        ``insert(rows: Iterable[dict[str, Any]], /, *, print_stats: bool = False, fail_on_exception: bool = True)``
 
         To insert just a single row, you can use the more convenient syntax:
-        ``insert(print_stats: bool = False, fail_on_exception: bool = True, **kwargs: Any)``
+        ``insert(*, print_stats: bool = False, fail_on_exception: bool = True, **kwargs: Any)``
 
         Args:
             rows: (if inserting multiple rows) A list of rows to insert, each of which is a dictionary mapping column
                 names to values.
-            kwargs: (if inserting a single row) keyword-argument pairs representing column names and values.
+            kwargs: (if inserting a single row) Keyword-argument pairs representing column names and values.
             print_stats: If ``True``, print statistics about the cost of computed columns.
             fail_on_exception:
                 Determines how exceptions in computed columns and invalid media files (e.g., corrupt images)
@@ -102,16 +106,27 @@ class InsertableTable(Table):
 
             >>> tbl.insert(a=1, b=1, c=1)
         """
-        print_stats = kwargs.pop('print_stats', False)
-        fail_on_exception = kwargs.pop('fail_on_exception', True)
-        if len(args) > 0:
-            # There's a positional argument; this means `rows` is expressed as a
-            # list of dicts (multi-insert)
-            rows = list(args[0])
-        else:
-            # No positional argument; this means we're inserting a single row
-            # using kwargs syntax
+        # The commented code is the intended implementation, with signature (*args, **kwargs).
+        # That signature cannot be used currently, due to a present limitation in mkdocs.
+        # See: https://github.com/mkdocstrings/mkdocstrings/issues/669
+
+        # print_stats = kwargs.pop('print_stats', False)
+        # fail_on_exception = kwargs.pop('fail_on_exception', True)
+        # if len(args) > 0:
+        #     # There's a positional argument; this means `rows` is expressed as a
+        #     # list of dicts (multi-insert)
+        #     rows = list(args[0])
+        # else:
+        #     # No positional argument; this means we're inserting a single row
+        #     # using kwargs syntax
+        #     rows = [kwargs]
+
+        if rows is None:
             rows = [kwargs]
+        else:
+            rows = list(rows)
+            if len(kwargs) > 0:
+                raise excs.Error('`kwargs` cannot be specified unless `rows is None`.')
 
         if not isinstance(rows, list):
             raise excs.Error('rows must be a list of dictionaries')