pixeltable 0.2.21__tar.gz → 0.2.22__tar.gz

{pixeltable-0.2.21 → pixeltable-0.2.22}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pixeltable
-Version: 0.2.21
+Version: 0.2.22
 Summary: Pixeltable: The Multimodal AI Data Plane
 Author: Pixeltable, Inc.
 Author-email: contact@pixeltable.com
@@ -16,9 +16,9 @@ Requires-Dist: cloudpickle (>=2.2.1,<3.0.0)
 Requires-Dist: ftfy (>=6.2.0,<7.0.0)
 Requires-Dist: jinja2 (>=3.1.3,<4.0.0)
 Requires-Dist: jmespath (>=1.0.1,<2.0.0)
+Requires-Dist: lxml (>=5.0)
 Requires-Dist: more-itertools (>=10.2,<11.0)
 Requires-Dist: numpy (>=1.25,<2.0)
-Requires-Dist: opencv-python-headless (>=4.7.0.68,<5.0.0.0)
 Requires-Dist: pandas (>=2.0,<3.0)
 Requires-Dist: pgvector (>=0.2.1,<0.3.0)
 Requires-Dist: pillow (>=9.3.0)
@@ -39,6 +39,8 @@ Description-Content-Type: text/markdown
 <img src="https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/source/data/pixeltable-logo-large.png" alt="Pixeltable" width="50%" />
 <br></br>
 
+<h2>AI Data Insfrastructure — Declarative, Multimodal, and Incremental</h2>
+
 [![License](https://img.shields.io/badge/License-Apache%202.0-0530AD.svg)](https://opensource.org/licenses/Apache-2.0)
 ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pixeltable?logo=python&logoColor=white&)
 ![Platform Support](https://img.shields.io/badge/platform-Linux%20%7C%20macOS%20%7C%20Windows-E5DDD4)
@@ -46,9 +48,10 @@ Description-Content-Type: text/markdown
 [![tests status](https://github.com/pixeltable/pixeltable/actions/workflows/pytest.yml/badge.svg)](https://github.com/pixeltable/pixeltable/actions/workflows/pytest.yml)
 [![tests status](https://github.com/pixeltable/pixeltable/actions/workflows/nightly.yml/badge.svg)](https://github.com/pixeltable/pixeltable/actions/workflows/nightly.yml)
 [![PyPI Package](https://img.shields.io/pypi/v/pixeltable?color=4D148C)](https://pypi.org/project/pixeltable/)
-<a target="_blank" href="https://huggingface.co/Pixeltable"> <img src="https://img.shields.io/badge/🤗-HF Space-F25022" alt="Visit our Hugging Face space"/></a>
+[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fpixeltable%2Fpixeltable.svg?type=shield&issueType=security)](https://app.fossa.com/projects/git%2Bgithub.com%2Fpixeltable%2Fpixeltable?ref=badge_shield&issueType=security)
+<a target="_blank" href="https://huggingface.co/Pixeltable"> <img src="https://img.shields.io/badge/🤗-HF Space-FF7D04" alt="Visit our Hugging Face space"/></a>
 
-[Installation](https://pixeltable.github.io/pixeltable/getting-started/) | [Documentation](https://pixeltable.readme.io/) | [API Reference](https://pixeltable.github.io/pixeltable/) | [Code Samples](https://github.com/pixeltable/pixeltable?tab=readme-ov-file#-code-samples) | [Computer Vision](https://docs.pixeltable.com/docs/object-detection-in-videos) | [LLM](https://docs.pixeltable.com/docs/document-indexing-and-rag)
+[Installation](https://docs.pixeltable.com/docs/installation) | [Documentation](https://pixeltable.readme.io/) | [API Reference](https://pixeltable.github.io/pixeltable/) | [Code Samples](https://github.com/pixeltable/pixeltable?tab=readme-ov-file#-code-samples) | [Computer Vision](https://docs.pixeltable.com/docs/object-detection-in-videos) | [LLM](https://docs.pixeltable.com/docs/document-indexing-and-rag)
 </div>
 
 Pixeltable is a Python library providing a declarative interface for multimodal data (text, images, audio, video). It features built-in versioning, lineage tracking, and incremental updates, enabling users to **store**, **transform**, **index**, and **iterate** on data for their ML workflows.
@@ -74,8 +77,9 @@ Learn how to create tables, populate them with data, and enhance them with built
 |:----------|:-----------------|:-------------------------|:---------------------------------:|
 | 10-Minute Tour of Pixeltable    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/tutorials/pixeltable-basics.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a> | Tables and Data Operations    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/fundamentals/tables-and-data-operations.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>
 | User-Defined Functions (UDFs)    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/howto/udfs-in-pixeltable.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a> | Object Detection Models | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/tutorials/object-detection-in-videos.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>
-| Experimenting with Chunking (RAG) | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/tutorials/rag-operations.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> | Working with External Files    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/howto/working-with-external-files.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>
-| Integrating with Label Studio    | <a target="_blank" href="https://pixeltable.readme.io/docs/label-studio"> <img src="https://img.shields.io/badge/Docs-Label Studio-blue" alt="Visit our documentation"/></a> | Audio/Video Transcript Indexing    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/tutorials/audio-transcriptions.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>
+| Incremental Prompt Engineering | <a target="_blank" href="https://colab.research.google.com/github/mistralai/cookbook/blob/main/third_party/Pixeltable/incremental_prompt_engineering_and_model_comparison.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Github"/> | Working with External Files    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/howto/working-with-external-files.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>
+| Integrating with Label Studio    | <a target="_blank" href="https://pixeltable.readme.io/docs/label-studio"> <img src="https://img.shields.io/badge/Documentation-013056" alt="Visit our documentation"/></a> | Audio/Video Transcript Indexing    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/tutorials/audio-transcriptions.ipynb">  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+| Multimodal Application    | <a target="_blank" href="https://huggingface.co/spaces/Pixeltable/Multimodal-Powerhouse"> <img src="https://img.shields.io/badge/Hugging Face-FF7D04" alt="Visit our documentation"/></a> | Document Indexing and RAG    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/tutorials/rag-demo.ipynb">  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
 
 ## 🧱 Code Samples
 
@@ -83,7 +87,7 @@ Learn how to create tables, populate them with data, and enhance them with built
 ```python
 import pixeltable as pxt
 
-v = pxt.create_table('external_data.videos', {'video': pxt.VideoType()})
+v = pxt.create_table('external_data.videos', {'video': pxt.Video})
 
 prefix = 's3://multimedia-commons/'
 paths = [
@@ -101,7 +105,7 @@ import pixeltable as pxt
 from pixeltable.functions import huggingface
 
 # Create a table to store data persistently
-t = pxt.create_table('image', {'image': pxt.ImageType()})
+t = pxt.create_table('image', {'image': pxt.Image})
 
 # Insert some images
 prefix = 'https://upload.wikimedia.org/wikipedia/commons'
@@ -158,7 +162,7 @@ Learn how to leverage Pixeltable for [Model analytics](https://pixeltable.readme
 
 ### Working with inference services
 ```python
-chat_table = pxt.create_table('together_demo.chat', {'input': pxt.StringType()})
+chat_table = pxt.create_table('together_demo.chat', {'input': pxt.String})
 
 # The chat-completions API expects JSON-formatted input:
 messages = [{'role': 'user', 'content': chat_table.input}]
@@ -194,7 +198,7 @@ from pixeltable.functions.huggingface import clip_image, clip_text
 from pixeltable.iterators import FrameIterator
 import PIL.Image
 
-video_table = pxt.create_table('videos', {'video': pxt.VideoType()})
+video_table = pxt.create_table('videos', {'video': pxt.Video})
 
 video_table.insert([{'video': '/video.mp4'}])
 
@@ -225,6 +229,38 @@ frames_view.order_by(sim, asc=False).limit(5).select(frames_view.frame, sim=sim)
 ```
 Learn how to work with [Embedding and Vector Indexes](https://docs.pixeltable.com/docs/embedding-vector-indexes).
 
+## 🔄 AI Stack Comparison
+
+### 🎯 Computer Vision Workflows
+
+| Requirement | Traditional | Pixeltable |
+|-------------|---------------------|------------|
+| Frame Extraction | ffmpeg + custom code | Automatic via FrameIterator |
+| Object Detection | Multiple scripts + caching | Single computed column |
+| Video Indexing | Custom pipelines + Vector DB | Native similarity search |
+| Annotation Management | Separate tools + custom code | Label Studio integration |
+| Model Evaluation | Custom metrics pipeline | Built-in mAP computation |
+
+### 🤖 LLM Workflows
+
+| Requirement | Traditional | Pixeltable |
+|-------------|---------------------|------------|
+| Document Chunking | Tool + custom code | Native DocumentSplitter |
+| Embedding Generation | Separate pipeline + caching | Computed columns |
+| Vector Search | External vector DB | Built-in vector indexing |
+| Prompt Management | Custom tracking solution | Version-controlled columns |
+| Chain Management | Tool + custom code | Computed column DAGs |
+
+### 🎨 Multimodal Workflows
+
+| Requirement | Traditional | Pixeltable |
+|-------------|---------------------|------------|
+| Data Types | Multiple storage systems | Unified table interface |
+| Cross-Modal Search | Complex integration | Native similarity support |
+| Pipeline Orchestration | Multiple tools (Airflow, etc.) | Single declarative interface |
+| Asset Management | Custom tracking system | Automatic lineage |
+| Quality Control | Multiple validation tools | Computed validation columns |
+
 ## ❓ FAQ
 
 ### What is Pixeltable?

{pixeltable-0.2.21 → pixeltable-0.2.22}/README.md

@@ -2,6 +2,8 @@
 <img src="https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/source/data/pixeltable-logo-large.png" alt="Pixeltable" width="50%" />
 <br></br>
 
+<h2>AI Data Insfrastructure — Declarative, Multimodal, and Incremental</h2>
+
 [![License](https://img.shields.io/badge/License-Apache%202.0-0530AD.svg)](https://opensource.org/licenses/Apache-2.0)
 ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pixeltable?logo=python&logoColor=white&)
 ![Platform Support](https://img.shields.io/badge/platform-Linux%20%7C%20macOS%20%7C%20Windows-E5DDD4)
@@ -9,9 +11,10 @@
 [![tests status](https://github.com/pixeltable/pixeltable/actions/workflows/pytest.yml/badge.svg)](https://github.com/pixeltable/pixeltable/actions/workflows/pytest.yml)
 [![tests status](https://github.com/pixeltable/pixeltable/actions/workflows/nightly.yml/badge.svg)](https://github.com/pixeltable/pixeltable/actions/workflows/nightly.yml)
 [![PyPI Package](https://img.shields.io/pypi/v/pixeltable?color=4D148C)](https://pypi.org/project/pixeltable/)
-<a target="_blank" href="https://huggingface.co/Pixeltable"> <img src="https://img.shields.io/badge/🤗-HF Space-F25022" alt="Visit our Hugging Face space"/></a>
+[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fpixeltable%2Fpixeltable.svg?type=shield&issueType=security)](https://app.fossa.com/projects/git%2Bgithub.com%2Fpixeltable%2Fpixeltable?ref=badge_shield&issueType=security)
+<a target="_blank" href="https://huggingface.co/Pixeltable"> <img src="https://img.shields.io/badge/🤗-HF Space-FF7D04" alt="Visit our Hugging Face space"/></a>
 
-[Installation](https://pixeltable.github.io/pixeltable/getting-started/) | [Documentation](https://pixeltable.readme.io/) | [API Reference](https://pixeltable.github.io/pixeltable/) | [Code Samples](https://github.com/pixeltable/pixeltable?tab=readme-ov-file#-code-samples) | [Computer Vision](https://docs.pixeltable.com/docs/object-detection-in-videos) | [LLM](https://docs.pixeltable.com/docs/document-indexing-and-rag)
+[Installation](https://docs.pixeltable.com/docs/installation) | [Documentation](https://pixeltable.readme.io/) | [API Reference](https://pixeltable.github.io/pixeltable/) | [Code Samples](https://github.com/pixeltable/pixeltable?tab=readme-ov-file#-code-samples) | [Computer Vision](https://docs.pixeltable.com/docs/object-detection-in-videos) | [LLM](https://docs.pixeltable.com/docs/document-indexing-and-rag)
 </div>
 
 Pixeltable is a Python library providing a declarative interface for multimodal data (text, images, audio, video). It features built-in versioning, lineage tracking, and incremental updates, enabling users to **store**, **transform**, **index**, and **iterate** on data for their ML workflows.
@@ -37,8 +40,9 @@ Learn how to create tables, populate them with data, and enhance them with built
 |:----------|:-----------------|:-------------------------|:---------------------------------:|
 | 10-Minute Tour of Pixeltable    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/tutorials/pixeltable-basics.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a> | Tables and Data Operations    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/fundamentals/tables-and-data-operations.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>
 | User-Defined Functions (UDFs)    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/howto/udfs-in-pixeltable.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a> | Object Detection Models | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/tutorials/object-detection-in-videos.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>
-| Experimenting with Chunking (RAG) | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/tutorials/rag-operations.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> | Working with External Files    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/howto/working-with-external-files.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>
-| Integrating with Label Studio    | <a target="_blank" href="https://pixeltable.readme.io/docs/label-studio"> <img src="https://img.shields.io/badge/Docs-Label Studio-blue" alt="Visit our documentation"/></a> | Audio/Video Transcript Indexing    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/tutorials/audio-transcriptions.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>
+| Incremental Prompt Engineering | <a target="_blank" href="https://colab.research.google.com/github/mistralai/cookbook/blob/main/third_party/Pixeltable/incremental_prompt_engineering_and_model_comparison.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Github"/> | Working with External Files    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/howto/working-with-external-files.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>
+| Integrating with Label Studio    | <a target="_blank" href="https://pixeltable.readme.io/docs/label-studio"> <img src="https://img.shields.io/badge/Documentation-013056" alt="Visit our documentation"/></a> | Audio/Video Transcript Indexing    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/tutorials/audio-transcriptions.ipynb">  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+| Multimodal Application    | <a target="_blank" href="https://huggingface.co/spaces/Pixeltable/Multimodal-Powerhouse"> <img src="https://img.shields.io/badge/Hugging Face-FF7D04" alt="Visit our documentation"/></a> | Document Indexing and RAG    | <a target="_blank" href="https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/tutorials/rag-demo.ipynb">  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
 
 ## 🧱 Code Samples
 
@@ -46,7 +50,7 @@ Learn how to create tables, populate them with data, and enhance them with built
 ```python
 import pixeltable as pxt
 
-v = pxt.create_table('external_data.videos', {'video': pxt.VideoType()})
+v = pxt.create_table('external_data.videos', {'video': pxt.Video})
 
 prefix = 's3://multimedia-commons/'
 paths = [
@@ -64,7 +68,7 @@ import pixeltable as pxt
 from pixeltable.functions import huggingface
 
 # Create a table to store data persistently
-t = pxt.create_table('image', {'image': pxt.ImageType()})
+t = pxt.create_table('image', {'image': pxt.Image})
 
 # Insert some images
 prefix = 'https://upload.wikimedia.org/wikipedia/commons'
@@ -121,7 +125,7 @@ Learn how to leverage Pixeltable for [Model analytics](https://pixeltable.readme
 
 ### Working with inference services
 ```python
-chat_table = pxt.create_table('together_demo.chat', {'input': pxt.StringType()})
+chat_table = pxt.create_table('together_demo.chat', {'input': pxt.String})
 
 # The chat-completions API expects JSON-formatted input:
 messages = [{'role': 'user', 'content': chat_table.input}]
@@ -157,7 +161,7 @@ from pixeltable.functions.huggingface import clip_image, clip_text
 from pixeltable.iterators import FrameIterator
 import PIL.Image
 
-video_table = pxt.create_table('videos', {'video': pxt.VideoType()})
+video_table = pxt.create_table('videos', {'video': pxt.Video})
 
 video_table.insert([{'video': '/video.mp4'}])
 
@@ -188,6 +192,38 @@ frames_view.order_by(sim, asc=False).limit(5).select(frames_view.frame, sim=sim)
 ```
 Learn how to work with [Embedding and Vector Indexes](https://docs.pixeltable.com/docs/embedding-vector-indexes).
 
+## 🔄 AI Stack Comparison
+
+### 🎯 Computer Vision Workflows
+
+| Requirement | Traditional | Pixeltable |
+|-------------|---------------------|------------|
+| Frame Extraction | ffmpeg + custom code | Automatic via FrameIterator |
+| Object Detection | Multiple scripts + caching | Single computed column |
+| Video Indexing | Custom pipelines + Vector DB | Native similarity search |
+| Annotation Management | Separate tools + custom code | Label Studio integration |
+| Model Evaluation | Custom metrics pipeline | Built-in mAP computation |
+
+### 🤖 LLM Workflows
+
+| Requirement | Traditional | Pixeltable |
+|-------------|---------------------|------------|
+| Document Chunking | Tool + custom code | Native DocumentSplitter |
+| Embedding Generation | Separate pipeline + caching | Computed columns |
+| Vector Search | External vector DB | Built-in vector indexing |
+| Prompt Management | Custom tracking solution | Version-controlled columns |
+| Chain Management | Tool + custom code | Computed column DAGs |
+
+### 🎨 Multimodal Workflows
+
+| Requirement | Traditional | Pixeltable |
+|-------------|---------------------|------------|
+| Data Types | Multiple storage systems | Unified table interface |
+| Cross-Modal Search | Complex integration | Native similarity support |
+| Pipeline Orchestration | Multiple tools (Airflow, etc.) | Single declarative interface |
+| Asset Management | Custom tracking system | Automatic lineage |
+| Quality Control | Multiple validation tools | Computed validation columns |
+
 ## ❓ FAQ
 
 ### What is Pixeltable?

{pixeltable-0.2.21 → pixeltable-0.2.22}/pixeltable/__version__.py

@@ -1,3 +1,3 @@
 # These version placeholders will be replaced during build.
-__version__ = "0.2.21"
-__version_tuple__ = (0, 2, 21)
+__version__ = "0.2.22"
+__version_tuple__ = (0, 2, 22)

{pixeltable-0.2.21 → pixeltable-0.2.22}/pixeltable/catalog/__init__.py

@@ -1,7 +1,7 @@
 from .catalog import Catalog
 from .column import Column
 from .dir import Dir
-from .globals import UpdateStatus, is_valid_identifier, is_valid_path
+from .globals import UpdateStatus, is_valid_identifier, is_valid_path, MediaValidation
 from .insertable_table import InsertableTable
 from .named_function import NamedFunction
 from .path import Path

{pixeltable-0.2.21 → pixeltable-0.2.22}/pixeltable/catalog/column.py

@@ -8,24 +8,43 @@ import sqlalchemy as sql
 import pixeltable.exceptions as excs
 import pixeltable.type_system as ts
 from pixeltable import exprs
-
-from .globals import is_valid_identifier
+from .globals import is_valid_identifier, MediaValidation
 
 if TYPE_CHECKING:
     from .table_version import TableVersion
 
 _logger = logging.getLogger('pixeltable')
 
+
 class Column:
     """Representation of a column in the schema of a Table/DataFrame.
 
     A Column contains all the metadata necessary for executing queries and updates against a particular version of a
     table/view.
     """
+    name: str
+    id: Optional[int]
+    col_type: ts.ColumnType
+    stored: bool
+    is_pk: bool
+    _media_validation: Optional[MediaValidation]  # if not set, TableVersion.media_validation applies
+    schema_version_add: Optional[int]
+    schema_version_drop: Optional[int]
+    _records_errors: Optional[bool]
+    sa_col: Optional[sql.schema.Column]
+    sa_col_type: Optional[sql.sqltypes.TypeEngine]
+    sa_errormsg_col: Optional[sql.schema.Column]
+    sa_errortype_col: Optional[sql.schema.Column]
+    compute_func: Optional[Callable]
+    _value_expr: Optional[exprs.Expr]
+    value_expr_dict: Optional[dict[str, Any]]
+    dependent_cols: set[Column]
+    tbl: Optional[TableVersion]
+
     def __init__(
             self, name: Optional[str], col_type: Optional[ts.ColumnType] = None,
             computed_with: Optional[Union[exprs.Expr, Callable]] = None,
-            is_pk: bool = False, stored: bool = True,
+            is_pk: bool = False, stored: bool = True, media_validation: Optional[MediaValidation] = None,
             col_id: Optional[int] = None, schema_version_add: Optional[int] = None,
             schema_version_drop: Optional[int] = None, sa_col_type: Optional[sql.sqltypes.TypeEngine] = None,
             records_errors: Optional[bool] = None, value_expr_dict: Optional[dict[str, Any]] = None,
@@ -61,8 +80,8 @@ class Column:
         if col_type is None and computed_with is None:
             raise excs.Error(f'Column `{name}`: col_type is required if computed_with is not specified')
 
-        self._value_expr: Optional[exprs.Expr] = None
-        self.compute_func: Optional[Callable] = None
+        self._value_expr = None
+        self.compute_func = None
         self.value_expr_dict = value_expr_dict
         if computed_with is not None:
             value_expr = exprs.Expr.from_object(computed_with)
@@ -86,24 +105,24 @@ class Column:
         assert self.col_type is not None
 
         self.stored = stored
-        self.dependent_cols: set[Column] = set()  # cols with value_exprs that reference us; set by TableVersion
+        self.dependent_cols = set()  # cols with value_exprs that reference us; set by TableVersion
         self.id = col_id
         self.is_pk = is_pk
+        self._media_validation = media_validation
         self.schema_version_add = schema_version_add
         self.schema_version_drop = schema_version_drop
 
         self._records_errors = records_errors
 
         # column in the stored table for the values of this Column
-        self.sa_col: Optional[sql.schema.Column] = None
+        self.sa_col = None
         self.sa_col_type = sa_col_type
 
         # computed cols also have storage columns for the exception string and type
-        self.sa_errormsg_col: Optional[sql.schema.Column] = None
-        self.sa_errortype_col: Optional[sql.schema.Column] = None
+        self.sa_errormsg_col = None
+        self.sa_errortype_col = None
 
-        from .table_version import TableVersion
-        self.tbl: Optional[TableVersion] = None  # set by owning TableVersion
+        self.tbl = None  # set by owning TableVersion
 
     @property
     def value_expr(self) -> Optional[exprs.Expr]:
@@ -160,6 +179,13 @@ class Column:
         assert self.tbl is not None
         return f'{self.tbl.name}.{self.name}'
 
+    @property
+    def media_validation(self) -> MediaValidation:
+        if self._media_validation is not None:
+            return self._media_validation
+        assert self.tbl is not None
+        return self.tbl.media_validation
+
     def source(self) -> None:
         """
         If this is a computed col and the top-level expr is a function call, print the source, if possible.

{pixeltable-0.2.21 → pixeltable-0.2.22}/pixeltable/catalog/globals.py

@@ -1,8 +1,12 @@
+from __future__ import annotations
 import dataclasses
+import enum
 import itertools
 import logging
 from typing import Optional
 
+import pixeltable.exceptions as excs
+
 _logger = logging.getLogger('pixeltable')
 
 # name of the position column in a component view
@@ -34,6 +38,20 @@ class UpdateStatus:
         self.cols_with_excs = list(dict.fromkeys(self.cols_with_excs + other.cols_with_excs))
         return self
 
+
+class MediaValidation(enum.Enum):
+    ON_READ = 0
+    ON_WRITE = 1
+
+    @classmethod
+    def validated(cls, name: str, error_prefix: str) -> MediaValidation:
+        try:
+            return cls[name.upper()]
+        except KeyError:
+            val_strs = ', '.join(f'{s.lower()!r}' for s in cls.__members__.keys())
+            raise excs.Error(f'{error_prefix} must be one of: [{val_strs}]')
+
+
 def is_valid_identifier(name: str) -> bool:
     return name.isidentifier() and not name.startswith('_')
 

{pixeltable-0.2.21 → pixeltable-0.2.22}/pixeltable/catalog/insertable_table.py

@@ -13,7 +13,7 @@ from pixeltable.env import Env
 from pixeltable.utils.filecache import FileCache
 
 from .catalog import Catalog
-from .globals import UpdateStatus
+from .globals import UpdateStatus, MediaValidation
 from .table import Table
 from .table_version import TableVersion
 from .table_version_path import TableVersionPath
@@ -35,8 +35,8 @@ class InsertableTable(Table):
     # MODULE-LOCAL, NOT PUBLIC
     @classmethod
     def _create(
-            cls, dir_id: UUID, name: str, schema: dict[str, ts.ColumnType], df: Optional[pxt.DataFrame], primary_key: List[str],
-            num_retained_versions: int, comment: str
+        cls, dir_id: UUID, name: str, schema: dict[str, ts.ColumnType], df: Optional[pxt.DataFrame],
+        primary_key: List[str], num_retained_versions: int, comment: str, media_validation: MediaValidation
     ) -> InsertableTable:
         columns = cls._create_columns(schema)
         cls._verify_schema(columns)
@@ -50,7 +50,9 @@ class InsertableTable(Table):
             col.is_pk = True
 
         with orm.Session(Env.get().engine, future=True) as session:
-            _, tbl_version = TableVersion.create(session, dir_id, name, columns, num_retained_versions, comment)
+            _, tbl_version = TableVersion.create(
+                session, dir_id, name, columns, num_retained_versions=num_retained_versions, comment=comment,
+                media_validation=media_validation)
             tbl = cls(dir_id, tbl_version)
             # TODO We need to commit before doing the insertion, in order to avoid a primary key (version) collision
             #   when the table metadata gets updated. Once we have a notion of user-defined transactions in

{pixeltable-0.2.21 → pixeltable-0.2.22}/pixeltable/catalog/table.py

@@ -24,7 +24,7 @@ import pixeltable.type_system as ts
 from pixeltable.utils.filecache import FileCache
 
 from .column import Column
-from .globals import _ROWID_COLUMN_NAME, UpdateStatus, is_system_column_name, is_valid_identifier
+from .globals import _ROWID_COLUMN_NAME, UpdateStatus, is_system_column_name, is_valid_identifier, MediaValidation
 from .schema_object import SchemaObject
 from .table_version import TableVersion
 from .table_version_path import TableVersionPath
@@ -91,6 +91,7 @@ class Table(SchemaObject):
                     'num_retained_versions': 10,
                     'is_view': False,
                     'is_snapshot': False,
+                    'media_validation': 'on_write',
                 }
                 ```
         """
@@ -101,6 +102,7 @@ class Table(SchemaObject):
         md['schema_version'] = self._tbl_version.schema_version
         md['comment'] = self._comment
         md['num_retained_versions'] = self._num_retained_versions
+        md['media_validation'] = self._media_validation.name.lower()
         return md
 
     @property
@@ -244,6 +246,10 @@ class Table(SchemaObject):
     def _num_retained_versions(self):
         return self._tbl_version.num_retained_versions
 
+    @property
+    def _media_validation(self) -> MediaValidation:
+        return self._tbl_version.media_validation
+
     def _description(self) -> pd.DataFrame:
         cols = self._tbl_version_path.columns()
         df = pd.DataFrame({
@@ -422,7 +428,7 @@ class Table(SchemaObject):
         (on account of containing Python Callables or Exprs).
         """
         assert isinstance(spec, dict)
-        valid_keys = {'type', 'value', 'stored'}
+        valid_keys = {'type', 'value', 'stored', 'media_validation'}
         has_type = False
         for k in spec.keys():
             if k not in valid_keys:
@@ -449,6 +455,9 @@ class Table(SchemaObject):
                 if 'type' in spec:
                     raise excs.Error(f'Column {name}: "type" is redundant if value is a Pixeltable expression')
 
+        if 'media_validation' in spec:
+            _ = catalog.MediaValidation.validated(spec['media_validation'], f'Column {name}: media_validation')
+
         if 'stored' in spec and not isinstance(spec['stored'], bool):
             raise excs.Error(f'Column {name}: "stored" must be a bool, got {spec["stored"]}')
         if not has_type:
@@ -462,6 +471,7 @@ class Table(SchemaObject):
             col_type: Optional[ts.ColumnType] = None
             value_expr: Optional[exprs.Expr] = None
             primary_key: Optional[bool] = None
+            media_validation: Optional[catalog.MediaValidation] = None
             stored = True
 
             if isinstance(spec, (ts.ColumnType, type, _GenericAlias)):
@@ -484,9 +494,15 @@ class Table(SchemaObject):
                     value_expr = value_expr.copy()
                 stored = spec.get('stored', True)
                 primary_key = spec.get('primary_key')
+                media_validation_str = spec.get('media_validation')
+                media_validation = (
+                    catalog.MediaValidation[media_validation_str.upper()] if media_validation_str is not None
+                    else None
+                )
 
             column = Column(
-                name, col_type=col_type, computed_with=value_expr, stored=stored, is_pk=primary_key)
+                name, col_type=col_type, computed_with=value_expr, stored=stored, is_pk=primary_key,
+                media_validation=media_validation)
             columns.append(column)
         return columns
 

{pixeltable-0.2.21 → pixeltable-0.2.22}/pixeltable/catalog/table_version.py

@@ -26,7 +26,7 @@ from pixeltable.utils.media_store import MediaStore
 
 from ..func.globals import resolve_symbol
 from .column import Column
-from .globals import _POS_COLUMN_NAME, _ROWID_COLUMN_NAME, UpdateStatus, is_valid_identifier
+from .globals import _POS_COLUMN_NAME, _ROWID_COLUMN_NAME, UpdateStatus, is_valid_identifier, MediaValidation
 
 if TYPE_CHECKING:
     from pixeltable import exec, store
@@ -53,6 +53,7 @@ class TableVersion:
     name: str
     version: int
     comment: str
+    media_validation: MediaValidation
     num_retained_versions: int
     schema_version: int
     view_md: Optional[schema.ViewMd]
@@ -109,6 +110,7 @@ class TableVersion:
         self.view_md = tbl_md.view_md  # save this as-is, it's needed for _create_md()
         is_view = tbl_md.view_md is not None
         self.is_snapshot = (is_view and tbl_md.view_md.is_snapshot) or bool(is_snapshot)
+        self.media_validation = MediaValidation[schema_version_md.media_validation.upper()]
         # a mutable TableVersion doesn't have a static version
         self.effective_version = self.version if self.is_snapshot else None
 
@@ -182,7 +184,7 @@ class TableVersion:
     @classmethod
     def create(
             cls, session: orm.Session, dir_id: UUID, name: str, cols: list[Column], num_retained_versions: int,
-            comment: str, base_path: Optional[pxt.catalog.TableVersionPath] = None,
+            comment: str, media_validation: MediaValidation, base_path: Optional[pxt.catalog.TableVersionPath] = None,
             view_md: Optional[schema.ViewMd] = None
     ) -> tuple[UUID, Optional[TableVersion]]:
         # assign ids
@@ -214,11 +216,17 @@ class TableVersion:
             tbl_id=tbl_record.id, version=0, md=dataclasses.asdict(table_version_md))
 
         # create schema.TableSchemaVersion
-        schema_col_md = {col.id: schema.SchemaColumn(pos=pos, name=col.name) for pos, col in enumerate(cols)}
+        schema_col_md: dict[int, schema.SchemaColumn] = {}
+        for pos, col in enumerate(cols):
+            md = schema.SchemaColumn(
+                pos=pos, name=col.name,
+                media_validation=col._media_validation.name.lower() if col._media_validation is not None else None)
+            schema_col_md[col.id] = md
 
         schema_version_md = schema.TableSchemaVersionMd(
             schema_version=0, preceding_schema_version=None, columns=schema_col_md,
-            num_retained_versions=num_retained_versions, comment=comment)
+            num_retained_versions=num_retained_versions, comment=comment,
+            media_validation=media_validation.name.lower())
         schema_version_record = schema.TableSchemaVersion(
             tbl_id=tbl_record.id, schema_version=0, md=dataclasses.asdict(schema_version_md))
 
@@ -285,10 +293,15 @@ class TableVersion:
         self.cols_by_name = {}
         self.cols_by_id = {}
         for col_md in tbl_md.column_md.values():
-            col_name = schema_version_md.columns[col_md.id].name if col_md.id in schema_version_md.columns else None
+            schema_col_md = schema_version_md.columns[col_md.id] if col_md.id in schema_version_md.columns else None
+            col_name = schema_col_md.name if schema_col_md is not None else None
+            media_val = (
+                MediaValidation[schema_col_md.media_validation.upper()]
+                if schema_col_md is not None and schema_col_md.media_validation is not None else None
+            )
             col = Column(
                 col_id=col_md.id, name=col_name, col_type=ts.ColumnType.from_dict(col_md.col_type),
-                is_pk=col_md.is_pk, stored=col_md.stored,
+                is_pk=col_md.is_pk, stored=col_md.stored, media_validation=media_val,
                 schema_version_add=col_md.schema_version_add, schema_version_drop=col_md.schema_version_drop,
                 value_expr_dict=col_md.value_expr)
             col.tbl = self
@@ -349,7 +362,8 @@ class TableVersion:
             self.store_tbl = StoreTable(self)
 
     def _update_md(
-            self, timestamp: float, conn: sql.engine.Connection, update_tbl_version: bool = True, preceding_schema_version: Optional[int] = None
+        self, timestamp: float, conn: sql.engine.Connection, update_tbl_version: bool = True,
+        preceding_schema_version: Optional[int] = None
     ) -> None:
         """Writes table metadata to the database.
 
@@ -710,20 +724,22 @@ class TableVersion:
 
         if conn is None:
             with Env.get().engine.begin() as conn:
-                return self._insert(plan, conn, time.time(), print_stats=print_stats, rowids=rowids())
+                return self._insert(
+                    plan, conn, time.time(), print_stats=print_stats, rowids=rowids(), abort_on_exc=fail_on_exception)
         else:
-            return self._insert(plan, conn, time.time(), print_stats=print_stats, rowids=rowids())
+            return self._insert(
+                plan, conn, time.time(), print_stats=print_stats, rowids=rowids(), abort_on_exc=fail_on_exception)
 
     def _insert(
         self, exec_plan: 'exec.ExecNode', conn: sql.engine.Connection, timestamp: float, *,
-        rowids: Optional[Iterator[int]] = None, print_stats: bool = False,
+        rowids: Optional[Iterator[int]] = None, print_stats: bool = False, abort_on_exc: bool = False
     ) -> UpdateStatus:
         """Insert rows produced by exec_plan and propagate to views"""
         # we're creating a new version
         self.version += 1
         result = UpdateStatus()
         num_rows, num_excs, cols_with_excs = self.store_tbl.insert_rows(
-            exec_plan, conn, v_min=self.version, rowids=rowids)
+            exec_plan, conn, v_min=self.version, rowids=rowids, abort_on_exc=abort_on_exc)
         result.num_rows = num_rows
         result.num_excs = num_excs
         result.num_computed_values += exec_plan.ctx.num_computed_exprs * num_rows
@@ -1203,7 +1219,8 @@ class TableVersion:
             name=self.name, current_version=self.version, current_schema_version=self.schema_version,
             next_col_id=self.next_col_id, next_idx_id=self.next_idx_id, next_row_id=self.next_rowid,
             column_md=self._create_column_md(self.cols), index_md=self.idx_md,
-            external_stores=self._create_stores_md(self.external_stores.values()), view_md=self.view_md)
+            external_stores=self._create_stores_md(self.external_stores.values()), view_md=self.view_md,
+        )
 
     def _create_version_md(self, timestamp: float) -> schema.TableVersionMd:
         return schema.TableVersionMd(created_at=timestamp, version=self.version, schema_version=self.schema_version)
@@ -1211,11 +1228,14 @@ class TableVersion:
     def _create_schema_version_md(self, preceding_schema_version: int) -> schema.TableSchemaVersionMd:
         column_md: dict[int, schema.SchemaColumn] = {}
         for pos, col in enumerate(self.cols_by_name.values()):
-            column_md[col.id] = schema.SchemaColumn(pos=pos, name=col.name)
+            column_md[col.id] = schema.SchemaColumn(
+                pos=pos, name=col.name,
+                media_validation=col._media_validation.name.lower() if col._media_validation is not None else None)
         # preceding_schema_version to be set by the caller
         return schema.TableSchemaVersionMd(
             schema_version=self.schema_version, preceding_schema_version=preceding_schema_version,
-            columns=column_md, num_retained_versions=self.num_retained_versions, comment=self.comment)
+            columns=column_md, num_retained_versions=self.num_retained_versions, comment=self.comment,
+            media_validation=self.media_validation.name.lower())
 
     def as_dict(self) -> dict:
         return {'id': str(self.id), 'effective_version': self.effective_version}