pixeltable 0.2.19__tar.gz → 0.2.20__tar.gz

{pixeltable-0.2.19 → pixeltable-0.2.20}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pixeltable
-Version: 0.2.19
+Version: 0.2.20
 Summary: Pixeltable: The Multimodal AI Data Plane
 Author: Pixeltable, Inc.
 Author-email: contact@pixeltable.com
@@ -31,6 +31,7 @@ Requires-Dist: pyyaml (>=6.0.1,<7.0.0)
 Requires-Dist: requests (>=2.31.0,<3.0.0)
 Requires-Dist: sqlalchemy (>=2.0.23,<3.0.0)
 Requires-Dist: tenacity (>=8.2,<9.0)
+Requires-Dist: toml (>=0.10)
 Requires-Dist: tqdm (>=4.64)
 Description-Content-Type: text/markdown
 
@@ -46,10 +47,17 @@ Description-Content-Type: text/markdown
 [![tests status](https://github.com/pixeltable/pixeltable/actions/workflows/nightly.yml/badge.svg)](https://github.com/pixeltable/pixeltable/actions/workflows/nightly.yml)
 [![PyPI Package](https://img.shields.io/pypi/v/pixeltable?color=darkorange)](https://pypi.org/project/pixeltable/)
 
-[Installation](https://pixeltable.github.io/pixeltable/getting-started/) | [Documentation](https://pixeltable.readme.io/) | [API Reference](https://pixeltable.github.io/pixeltable/) | [Code Samples](https://pixeltable.readme.io/recipes) | [Examples](https://github.com/pixeltable/pixeltable/tree/release/docs/release/tutorials)
+[Installation](https://pixeltable.github.io/pixeltable/getting-started/) | [Documentation](https://pixeltable.readme.io/) | [API Reference](https://pixeltable.github.io/pixeltable/) | [Code Samples](https://github.com/pixeltable/pixeltable?tab=readme-ov-file#-code-samples) | [Computer Vision](https://docs.pixeltable.com/docs/object-detection-in-videos) | [LLM](https://docs.pixeltable.com/docs/document-indexing-and-rag)
 </div>
 
-Pixeltable is a Python library providing a declarative interface for multimodal data (text, images, audio, video). It features built-in versioning, lineage tracking, and incremental updates, enabling users to store, transform, index, and iterate on data for their ML workflows. Data transformations, model inference, and custom logic are embedded as computed columns.
+Pixeltable is a Python library providing a declarative interface for multimodal data (text, images, audio, video). It features built-in versioning, lineage tracking, and incremental updates, enabling users to **store**, **transform**, **index**, and **iterate** on data for their ML workflows.
+
+Data transformations, model inference, and custom logic are embedded as **computed columns**.
+- **Load/Query all data types**: Interact with [video data](https://github.com/pixeltable/pixeltable?tab=readme-ov-file#import-media-data-into-pixeltable-videos-images-audio) at the [frame level](https://github.com/pixeltable/pixeltable?tab=readme-ov-file#text-and-image-similarity-search-on-video-frames-with-embedding-indexes) and documents at the [chunk level](https://github.com/pixeltable/pixeltable?tab=readme-ov-file#automate-data-operations-with-views-eg-split-documents-into-chunks)
+- **Incremental updates for data transformation**: Maintain an [embedding index](https://docs.pixeltable.com/docs/embedding-vector-indexes) colocated with your data
+- **Lazy evaluation and cache management**: Eliminates the need for [manual frame extraction](https://docs.pixeltable.com/docs/object-detection-in-videos)
+- **Integrates with any Python libraries**: Use [built-in and custom functions (UDFs)](https://docs.pixeltable.com/docs/user-defined-functions-udfs) without complex pipelines
+- **Data format agnostic and extensibility**: Access tables as Parquet files, [PyTorch datasets](https://pixeltable.github.io/pixeltable/api/data-frame/#pixeltable.DataFrame.to_pytorch_dataset), or [COCO annotations](https://pixeltable.github.io/pixeltable/api/table/#pixeltable.Table.to_coco_dataset)
 
 ## 💾 Installation
 

{pixeltable-0.2.19 → pixeltable-0.2.20}/README.md

@@ -10,10 +10,17 @@
 [![tests status](https://github.com/pixeltable/pixeltable/actions/workflows/nightly.yml/badge.svg)](https://github.com/pixeltable/pixeltable/actions/workflows/nightly.yml)
 [![PyPI Package](https://img.shields.io/pypi/v/pixeltable?color=darkorange)](https://pypi.org/project/pixeltable/)
 
-[Installation](https://pixeltable.github.io/pixeltable/getting-started/) | [Documentation](https://pixeltable.readme.io/) | [API Reference](https://pixeltable.github.io/pixeltable/) | [Code Samples](https://pixeltable.readme.io/recipes) | [Examples](https://github.com/pixeltable/pixeltable/tree/release/docs/release/tutorials)
+[Installation](https://pixeltable.github.io/pixeltable/getting-started/) | [Documentation](https://pixeltable.readme.io/) | [API Reference](https://pixeltable.github.io/pixeltable/) | [Code Samples](https://github.com/pixeltable/pixeltable?tab=readme-ov-file#-code-samples) | [Computer Vision](https://docs.pixeltable.com/docs/object-detection-in-videos) | [LLM](https://docs.pixeltable.com/docs/document-indexing-and-rag)
 </div>
 
-Pixeltable is a Python library providing a declarative interface for multimodal data (text, images, audio, video). It features built-in versioning, lineage tracking, and incremental updates, enabling users to store, transform, index, and iterate on data for their ML workflows. Data transformations, model inference, and custom logic are embedded as computed columns.
+Pixeltable is a Python library providing a declarative interface for multimodal data (text, images, audio, video). It features built-in versioning, lineage tracking, and incremental updates, enabling users to **store**, **transform**, **index**, and **iterate** on data for their ML workflows.
+
+Data transformations, model inference, and custom logic are embedded as **computed columns**.
+- **Load/Query all data types**: Interact with [video data](https://github.com/pixeltable/pixeltable?tab=readme-ov-file#import-media-data-into-pixeltable-videos-images-audio) at the [frame level](https://github.com/pixeltable/pixeltable?tab=readme-ov-file#text-and-image-similarity-search-on-video-frames-with-embedding-indexes) and documents at the [chunk level](https://github.com/pixeltable/pixeltable?tab=readme-ov-file#automate-data-operations-with-views-eg-split-documents-into-chunks)
+- **Incremental updates for data transformation**: Maintain an [embedding index](https://docs.pixeltable.com/docs/embedding-vector-indexes) colocated with your data
+- **Lazy evaluation and cache management**: Eliminates the need for [manual frame extraction](https://docs.pixeltable.com/docs/object-detection-in-videos)
+- **Integrates with any Python libraries**: Use [built-in and custom functions (UDFs)](https://docs.pixeltable.com/docs/user-defined-functions-udfs) without complex pipelines
+- **Data format agnostic and extensibility**: Access tables as Parquet files, [PyTorch datasets](https://pixeltable.github.io/pixeltable/api/data-frame/#pixeltable.DataFrame.to_pytorch_dataset), or [COCO annotations](https://pixeltable.github.io/pixeltable/api/table/#pixeltable.Table.to_coco_dataset)
 
 ## 💾 Installation
 

{pixeltable-0.2.19 → pixeltable-0.2.20}/pixeltable/__version__.py

@@ -1,3 +1,3 @@
 # These version placeholders will be replaced during build.
-__version__ = "0.2.19"
-__version_tuple__ = (0, 2, 19)
+__version__ = "0.2.20"
+__version_tuple__ = (0, 2, 20)

{pixeltable-0.2.19 → pixeltable-0.2.20}/pixeltable/catalog/insertable_table.py

@@ -10,6 +10,7 @@ import pixeltable as pxt
 import pixeltable.type_system as ts
 from pixeltable import exceptions as excs
 from pixeltable.env import Env
+from pixeltable.utils.filecache import FileCache
 
 from .catalog import Catalog
 from .globals import UpdateStatus
@@ -101,21 +102,22 @@ class InsertableTable(Table):
             if not isinstance(row, dict):
                 raise excs.Error('rows must be a list of dictionaries')
         self._validate_input_rows(rows)
-        result = self._tbl_version.insert(rows, None, print_stats=print_stats, fail_on_exception=fail_on_exception)
+        status = self._tbl_version.insert(rows, None, print_stats=print_stats, fail_on_exception=fail_on_exception)
 
-        if result.num_excs == 0:
+        if status.num_excs == 0:
             cols_with_excs_str = ''
         else:
             cols_with_excs_str = \
-                f' across {len(result.cols_with_excs)} column{"" if len(result.cols_with_excs) == 1 else "s"}'
-            cols_with_excs_str += f' ({", ".join(result.cols_with_excs)})'
+                f' across {len(status.cols_with_excs)} column{"" if len(status.cols_with_excs) == 1 else "s"}'
+            cols_with_excs_str += f' ({", ".join(status.cols_with_excs)})'
         msg = (
-            f'Inserted {result.num_rows} row{"" if result.num_rows == 1 else "s"} '
-            f'with {result.num_excs} error{"" if result.num_excs == 1 else "s"}{cols_with_excs_str}.'
+            f'Inserted {status.num_rows} row{"" if status.num_rows == 1 else "s"} '
+            f'with {status.num_excs} error{"" if status.num_excs == 1 else "s"}{cols_with_excs_str}.'
         )
         print(msg)
         _logger.info(f'InsertableTable {self._name}: {msg}')
-        return result
+        FileCache.get().emit_eviction_warnings()
+        return status
 
     def _validate_input_rows(self, rows: List[Dict[str, Any]]) -> None:
         """Verify that the input rows match the table schema"""

{pixeltable-0.2.19 → pixeltable-0.2.20}/pixeltable/catalog/table.py

@@ -20,6 +20,7 @@ import pixeltable.exprs as exprs
 import pixeltable.index as index
 import pixeltable.metadata.schema as schema
 import pixeltable.type_system as ts
+from pixeltable.utils.filecache import FileCache
 
 from .column import Column
 from .globals import _ROWID_COLUMN_NAME, UpdateStatus, is_system_column_name, is_valid_identifier
@@ -33,7 +34,12 @@ if TYPE_CHECKING:
 _logger = logging.getLogger('pixeltable')
 
 class Table(SchemaObject):
-    """Base class for table objects (base tables, views, snapshots)."""
+    """
+    Base class for table objects (base tables, views, snapshots).
+
+    Every user-invoked operation that runs an ExecNode tree (directly or indirectly) needs to call
+    FileCache.emit_eviction_warnings() at the end of the operation.
+    """
 
     def __init__(self, id: UUID, dir_id: UUID, name: str, tbl_version_path: TableVersionPath):
         super().__init__(id, name, dir_id)
@@ -374,7 +380,10 @@ class Table(SchemaObject):
 
         new_col = self._create_columns({col_name: col_schema})[0]
         self._verify_column(new_col, set(self._schema.keys()), set(self._query_names))
-        return self._tbl_version.add_column(new_col, print_stats=print_stats)
+        status = self._tbl_version.add_column(new_col, print_stats=print_stats)
+        FileCache.get().emit_eviction_warnings()
+        return status
+
 
     @classmethod
     def _validate_column_spec(cls, name: str, spec: dict[str, Any]) -> None:
@@ -587,6 +596,7 @@ class Table(SchemaObject):
         idx = EmbeddingIndex(col, metric=metric, string_embed=string_embed, image_embed=image_embed)
         status = self._tbl_version.add_index(col, idx_name=idx_name, idx=idx)
         # TODO: how to deal with exceptions here? drop the index and raise?
+        FileCache.get().emit_eviction_warnings()
 
     def drop_embedding_index(self, *, column_name: Optional[str] = None, idx_name: Optional[str] = None) -> None:
         """Drop an embedding index from the table.
@@ -732,7 +742,9 @@ class Table(SchemaObject):
             >>> tbl.update({'int_col': tbl.int_col + 1}, where=tbl.int_col == 0)
         """
         self._check_is_dropped()
-        return self._tbl_version.update(value_spec, where, cascade)
+        status = self._tbl_version.update(value_spec, where, cascade)
+        FileCache.get().emit_eviction_warnings()
+        return status
 
     def batch_update(
             self, rows: Iterable[dict[str, Any]], cascade: bool = True,
@@ -789,9 +801,11 @@ class Table(SchemaObject):
                     missing_cols = pk_col_names - set(col.name for col in col_vals.keys())
                     raise excs.Error(f'Primary key columns ({", ".join(missing_cols)}) missing in {row_spec}')
             row_updates.append(col_vals)
-        return self._tbl_version.batch_update(
+        status = self._tbl_version.batch_update(
             row_updates, rowids, error_if_not_exists=if_not_exists == 'error',
             insert_if_not_exists=if_not_exists == 'insert', cascade=cascade)
+        FileCache.get().emit_eviction_warnings()
+        return status
 
     def delete(self, where: Optional['pixeltable.exprs.Expr'] = None) -> UpdateStatus:
         """Delete rows in this table.

{pixeltable-0.2.19 → pixeltable-0.2.20}/pixeltable/env.py

@@ -8,6 +8,7 @@ import importlib.util
 import inspect
 import logging
 import os
+import shutil
 import subprocess
 import sys
 import threading
@@ -15,12 +16,12 @@ import uuid
 import warnings
 from dataclasses import dataclass
 from pathlib import Path
-from typing import TYPE_CHECKING, Any, Callable, Optional
+from typing import TYPE_CHECKING, Any, Callable, Optional, TypeVar
 from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
 
 import pixeltable_pgserver
 import sqlalchemy as sql
-import yaml
+import toml
 from tqdm import TqdmWarning
 
 import pixeltable.exceptions as excs
@@ -64,7 +65,7 @@ class Env:
     _log_to_stdout: bool
     _module_log_level: dict[str, int]  # module name -> log level
     _config_file: Optional[Path]
-    _config: Optional[dict[str, Any]]
+    _config: Optional[Config]
     _stdout_handler: logging.StreamHandler
     _initialized: bool
 
@@ -110,6 +111,7 @@ class Env:
         self._log_to_stdout = False
         self._module_log_level = {}  # module name -> log level
 
+        # config
         self._config_file = None
         self._config = None
 
@@ -119,7 +121,8 @@ class Env:
         self._initialized = False
 
     @property
-    def config(self):
+    def config(self) -> Config:
+        assert self._config is not None
         return self._config
 
     @property
@@ -227,30 +230,13 @@ class Env:
         home = Path(os.environ.get('PIXELTABLE_HOME', str(Path.home() / '.pixeltable')))
         assert self._home is None or self._home == home
         self._home = home
-        self._config_file = Path(os.environ.get('PIXELTABLE_CONFIG', str(self._home / 'config.yaml')))
+        self._config_file = Path(os.environ.get('PIXELTABLE_CONFIG', str(self._home / 'config.toml')))
         self._media_dir = self._home / 'media'
         self._file_cache_dir = self._home / 'file_cache'
         self._dataset_cache_dir = self._home / 'dataset_cache'
         self._log_dir = self._home / 'logs'
         self._tmp_dir = self._home / 'tmp'
 
-        # Read in the config
-        if os.path.isfile(self._config_file):
-            with open(self._config_file, 'r') as stream:
-                try:
-                    self._config = yaml.safe_load(stream)
-                except yaml.YAMLError as exc:
-                    self._logger.error(f'Could not read config file: {self._config_file}')
-                    self._config = {}
-        else:
-            self._config = {}
-
-        # Disable spurious warnings
-        warnings.simplefilter('ignore', category=TqdmWarning)
-        if 'hide_warnings' in self._config and self._config['hide_warnings']:
-            # Disable more warnings
-            warnings.simplefilter('ignore', category=UserWarning)
-
         if self._home.exists() and not self._home.is_dir():
             raise RuntimeError(f'{self._home} is not a directory')
 
@@ -274,6 +260,22 @@ class Env:
         if not self._tmp_dir.exists():
             self._tmp_dir.mkdir()
 
+        # Read in the config
+        self._config = Config.from_file(self._config_file)
+        self._file_cache_size_g = self._config.get_float_value('file_cache_size_g')
+        if self._file_cache_size_g is None:
+            raise excs.Error(
+                'pixeltable/file_cache_size_g is missing from configuration\n'
+                f'(either add a `file_cache_size_g` entry to the `pixeltable` section of {self._config_file},\n'
+                'or set the PIXELTABLE_FILE_CACHE_SIZE_G environment variable)'
+            )
+
+        # Disable spurious warnings
+        warnings.simplefilter('ignore', category=TqdmWarning)
+        if self._config.get_bool_value('hide_warnings'):
+            # Disable more warnings
+            warnings.simplefilter('ignore', category=UserWarning)
+
         # configure _logger to log to a file
         self._logfilename = datetime.datetime.now().strftime('%Y%m%d_%H%M%S') + '.log'
         fh = logging.FileHandler(self._log_dir / self._logfilename, mode='w')
@@ -313,7 +315,7 @@ class Env:
         self._db_server = pixeltable_pgserver.get_server(self._pgdata_dir, cleanup_mode=None)
         self._db_url = self._db_server.get_uri(database=self._db_name, driver='psycopg')
 
-        tz_name = os.environ.get('PXT_TIME_ZONE', self._config.get('pxt_time_zone', None))
+        tz_name = self.config.get_string_value('time_zone')
         if tz_name is not None:
             # Validate tzname
             if not isinstance(tz_name, str):
@@ -440,21 +442,18 @@ class Env:
         if cl.client_obj is not None:
             return cl.client_obj  # Already initialized
 
-        # Construct a client. For each client parameter, first check if the parameter is in the environment;
-        # if not, look in Pixeltable config from `config.yaml`.
+        # Construct a client, retrieving each parameter from config.
 
         init_kwargs: dict[str, str] = {}
         for param in cl.param_names:
-            environ = f'{name.upper()}_{param.upper()}'
-            if environ in os.environ:
-                init_kwargs[param] = os.environ[environ]
-            elif name.lower() in self._config and param in self._config[name.lower()]:
-                init_kwargs[param] = self._config[name.lower()][param.lower()]
-            if param not in init_kwargs or init_kwargs[param] == '':
+            arg = self._config.get_string_value(param, section=name)
+            if arg is not None and len(arg) > 0:
+                init_kwargs[param] = arg
+            else:
                 raise excs.Error(
                     f'`{name}` client not initialized: parameter `{param}` is not configured.\n'
-                    f'To fix this, specify the `{environ}` environment variable, or put `{param.lower()}` in '
-                    f'the `{name.lower()}` section of $PIXELTABLE_HOME/config.yaml.'
+                    f'To fix this, specify the `{name.upper()}_{param.upper()}` environment variable, or put `{param.lower()}` in '
+                    f'the `{name.lower()}` section of $PIXELTABLE_HOME/config.toml.'
                 )
 
         cl.client_obj = cl.init_fn(**init_kwargs)
@@ -506,7 +505,6 @@ class Env:
         self.__register_package('spacy')
         self.__register_package('tiktoken')
         self.__register_package('together')
-        self.__register_package('toml')
         self.__register_package('torch')
         self.__register_package('torchvision')
         self.__register_package('transformers')
@@ -643,7 +641,7 @@ def register_client(name: str) -> Callable:
     Pixeltable will attempt to load the client parameters from config. For each
     config parameter:
     - If an environment variable named MY_CLIENT_API_KEY (for example) is set, use it;
-    - Otherwise, look for 'api_key' in the 'my_client' section of config.yaml.
+    - Otherwise, look for 'api_key' in the 'my_client' section of config.toml.
 
     If all config parameters are found, Pixeltable calls the initialization function;
     otherwise it throws an exception.
@@ -660,6 +658,79 @@ def register_client(name: str) -> Callable:
     return decorator
 
 
+class Config:
+    """
+    The (global) Pixeltable configuration, as loaded from `config.toml`. Provides methods for retrieving
+    configuration values, which can be set in the config file or as environment variables.
+    """
+    __config: dict[str, Any]
+
+    T = TypeVar('T')
+
+    @classmethod
+    def from_file(cls, path: Path) -> Config:
+        """
+        Loads configuration from the specified TOML file. If the file does not exist, it will be
+        created and populated with the default configuration.
+        """
+        if os.path.isfile(path):
+            with open(path, 'r') as stream:
+                try:
+                    config_dict = toml.load(stream)
+                except Exception as exc:
+                    raise excs.Error(f'Could not read config file: {str(path)}') from exc
+        else:
+            config_dict = cls.__create_default_config(path)
+            with open(path, 'w') as stream:
+                try:
+                    toml.dump(config_dict, stream)
+                except Exception as exc:
+                    raise excs.Error(f'Could not write config file: {str(path)}') from exc
+            logging.getLogger('pixeltable').info(f'Created default config file at: {str(path)}')
+        return cls(config_dict)
+
+    @classmethod
+    def __create_default_config(cls, config_path: Path) -> dict[str, Any]:
+        free_disk_space_bytes = shutil.disk_usage(config_path.parent).free
+        # Default cache size is 1/5 of free disk space
+        file_cache_size_g = free_disk_space_bytes / 5 / (1 << 30)
+        return {
+            'pixeltable': {
+                'file_cache_size_g': round(file_cache_size_g, 1),
+                'hide_warnings': False,
+            }
+        }
+
+    def __init__(self, config: dict[str, Any]) -> None:
+        self.__config = config
+
+    def get_value(self, key: str, expected_type: type[T], section: str = 'pixeltable') -> Optional[T]:
+        env_var = f'{section.upper()}_{key.upper()}'
+        if env_var in os.environ:
+            value = os.environ[env_var]
+        elif section in self.__config and key in self.__config[section]:
+            value = self.__config[section][key]
+        else:
+            return None
+
+        try:
+            return expected_type(value)  # type: ignore[call-arg]
+        except ValueError:
+            raise excs.Error(f'Invalid value for configuration parameter {section}.{key}: {value}')
+
+    def get_string_value(self, key: str, section: str = 'pixeltable') -> Optional[str]:
+        return self.get_value(key, str, section)
+
+    def get_int_value(self, key: str, section: str = 'pixeltable') -> Optional[int]:
+        return self.get_value(key, int, section)
+
+    def get_float_value(self, key: str, section: str = 'pixeltable') -> Optional[float]:
+        return self.get_value(key, float, section)
+
+    def get_bool_value(self, key: str, section: str = 'pixeltable') -> Optional[bool]:
+        return self.get_value(key, bool, section)
+
+
 _registered_clients: dict[str, ApiClient] = {}
 
 

{pixeltable-0.2.19 → pixeltable-0.2.20}/pixeltable/exceptions.py

@@ -1,6 +1,9 @@
-from typing import List, Any
-from types import TracebackType
 from dataclasses import dataclass
+from types import TracebackType
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+    from pixeltable import exprs
 
 
 class Error(Exception):
@@ -9,11 +12,11 @@ class Error(Exception):
 
 @dataclass
 class ExprEvalError(Exception):
-    expr: Any  # exprs.Expr, but we're not importing pixeltable.exprs to avoid circular imports
+    expr: 'exprs.Expr'
     expr_msg: str
     exc: Exception
     exc_tb: TracebackType
-    input_vals: List[Any]
+    input_vals: list[Any]
     row_num: int
 
 

pixeltable-0.2.20/pixeltable/exec/component_iteration_node.py

@@ -0,0 +1,100 @@
+import inspect
+from typing import Iterator, Optional
+
+import pixeltable.catalog as catalog
+import pixeltable.exceptions as excs
+import pixeltable.exprs as exprs
+
+from .data_row_batch import DataRowBatch
+from .exec_node import ExecNode
+
+
+class ComponentIterationNode(ExecNode):
+    """Expands each row from a base table into one row per component returned by an iterator
+
+    Returns row batches of OUTPUT_BATCH_SIZE size.
+    """
+    __OUTPUT_BATCH_SIZE = 1024
+
+    def __init__(self, view: catalog.TableVersion, input: ExecNode):
+        assert view.is_component_view()
+        super().__init__(input.row_builder, [], [], input)
+        self.view = view
+        iterator_args = [view.iterator_args.copy()]
+        self.row_builder.set_slot_idxs(iterator_args)
+        self.iterator_args = iterator_args[0]
+        assert isinstance(self.iterator_args, exprs.InlineDict)
+        self.iterator_args_ctx = self.row_builder.create_eval_ctx([self.iterator_args])
+        self.iterator_output_schema, self.unstored_column_names = (
+            self.view.iterator_cls.output_schema(**self.iterator_args.to_kwargs())
+        )
+        self.iterator_output_fields = list(self.iterator_output_schema.keys())
+        self.iterator_output_cols = {
+            field_name: self.view.cols_by_name[field_name] for field_name in self.iterator_output_fields
+        }
+        # referenced iterator output fields
+        self.refd_output_slot_idxs = {
+            e.col.name: e.slot_idx for e in self.row_builder.unique_exprs
+            if isinstance(e, exprs.ColumnRef) and e.col.name in self.iterator_output_fields
+        }
+        self.__output: Optional[Iterator[DataRowBatch]] = None
+
+    def __output_batches(self) -> Iterator[DataRowBatch]:
+        output_batch = DataRowBatch(self.view, self.row_builder)
+        for input_batch in self.input:
+            for input_row in input_batch:
+                self.row_builder.eval(input_row, self.iterator_args_ctx)
+                iterator_args = input_row[self.iterator_args.slot_idx]
+                assert isinstance(iterator_args, dict)
+                # We need to ensure that all of the required (non-nullable) parameters of the iterator are
+                # specified and are not null. If any of them are null, then we skip this row (i.e., we emit 0
+                # output rows for this input row).
+                if self.__non_nullable_args_specified(iterator_args):
+                    iterator = self.view.iterator_cls(**iterator_args)
+                    for pos, component_dict in enumerate(iterator):
+                        output_row = output_batch.add_row()
+                        input_row.copy(output_row)
+                        # we're expanding the input and need to add the iterator position to the pk
+                        self.__populate_output_row(output_row, pos, component_dict)
+                        if len(output_batch) == self.__OUTPUT_BATCH_SIZE:
+                            yield output_batch
+                            output_batch = DataRowBatch(self.view, self.row_builder)
+
+        if len(output_batch) > 0:
+            yield output_batch
+
+    def __non_nullable_args_specified(self, iterator_args: dict) -> bool:
+        """
+        Returns true if all non-nullable iterator arguments are not `None`.
+        """
+        input_schema = self.view.iterator_cls.input_schema()
+        for arg_name, arg_value in iterator_args.items():
+            col_type = input_schema[arg_name]
+            if arg_value is None and not col_type.nullable:
+                return False
+        return True
+
+    def __populate_output_row(self, output_row: exprs.DataRow, pos: int, component_dict: dict) -> None:
+        pk = output_row.pk[:-1] + (pos,) + output_row.pk[-1:]
+        output_row.set_pk(pk)
+        # verify and copy component_dict fields to their respective slots in output_row
+        for field_name, field_val in component_dict.items():
+            if field_name not in self.iterator_output_fields:
+                raise excs.Error(
+                    f'Invalid field name {field_name} in output of {self.view.iterator_cls.__name__}')
+            if field_name not in self.refd_output_slot_idxs:
+                # we can ignore this
+                continue
+            output_col = self.iterator_output_cols[field_name]
+            output_col.col_type.validate_literal(field_val)
+            output_row[self.refd_output_slot_idxs[field_name]] = field_val
+        if len(component_dict) != len(self.iterator_output_fields):
+            missing_fields = set(self.refd_output_slot_idxs.keys()) - set(component_dict.keys())
+            raise excs.Error(
+                f'Invalid output of {self.view.iterator_cls.__name__}: '
+                f'missing fields {", ".join(missing_fields)}')
+
+    def __next__(self) -> DataRowBatch:
+        if self.__output is None:
+            self.__output = self.__output_batches()
+        return next(self.__output)

{pixeltable-0.2.19 → pixeltable-0.2.20}/pixeltable/exprs/json_path.py

@@ -105,12 +105,9 @@ class JsonPath(Expr):
         return JsonPath(self._anchor, self.path_elements + [name])
 
     def __getitem__(self, index: object) -> 'JsonPath':
-        if isinstance(index, str):
-            if index != '*':
-                raise excs.Error(f'Invalid json list index: {index}')
-        elif not isinstance(index, (int, slice)):
-                raise excs.Error(f'Invalid json list index: {index}')
-        return JsonPath(self._anchor, self.path_elements + [index])
+        if isinstance(index, (int, slice, str)):
+            return JsonPath(self._anchor, self.path_elements + [index])
+        raise excs.Error(f'Invalid json list index: {index}')
 
     def __rshift__(self, other: object) -> 'JsonMapper':
         rhs_expr = Expr.from_object(other)

{pixeltable-0.2.19 → pixeltable-0.2.20}/pixeltable/functions/together.py

@@ -7,13 +7,15 @@ the [Working with Together AI](https://pixeltable.readme.io/docs/together-ai) tu
 
 import base64
 import io
-from typing import TYPE_CHECKING, Callable, Optional
+from typing import TYPE_CHECKING, Callable, Optional, TypeVar
 
 import numpy as np
 import PIL.Image
+import requests
 import tenacity
 
 import pixeltable as pxt
+import pixeltable.exceptions as excs
 from pixeltable import env
 from pixeltable.func import Batch
 from pixeltable.utils.code import local_public_names
@@ -32,7 +34,10 @@ def _together_client() -> 'together.Together':
     return env.Env.get().get_client('together')
 
 
-def _retry(fn: Callable) -> Callable:
+T = TypeVar('T')
+
+
+def _retry(fn: Callable[..., T]) -> Callable[..., T]:
     import together
     return tenacity.retry(
         retry=tenacity.retry_if_exception_type(together.error.RateLimitError),
@@ -249,20 +254,29 @@ def image_generations(
         The generated image.
 
     Examples:
-        Add a computed column that applies the model `runwayml/stable-diffusion-v1-5`
+        Add a computed column that applies the model `stabilityai/stable-diffusion-xl-base-1.0`
         to an existing Pixeltable column `tbl.prompt` of the table `tbl`:
 
-        >>> tbl['response'] = image_generations(tbl.prompt, model='runwayml/stable-diffusion-v1-5')
+        >>> tbl['response'] = image_generations(tbl.prompt, model='stabilityai/stable-diffusion-xl-base-1.0')
     """
-    # TODO(aaron-siegel): Decompose CPU/GPU ops into separate functions
     result = _retry(_together_client().images.generate)(
         prompt=prompt, model=model, steps=steps, seed=seed, height=height, width=width, negative_prompt=negative_prompt
     )
-    b64_str = result.data[0].b64_json
-    b64_bytes = base64.b64decode(b64_str)
-    img = PIL.Image.open(io.BytesIO(b64_bytes))
-    img.load()
-    return img
+    if result.data[0].b64_json is not None:
+        b64_bytes = base64.b64decode(result.data[0].b64_json)
+        img = PIL.Image.open(io.BytesIO(b64_bytes))
+        img.load()
+        return img
+    if result.data[0].url is not None:
+        try:
+            resp = requests.get(result.data[0].url)
+            with io.BytesIO(resp.content) as fp:
+                image = PIL.Image.open(fp)
+                image.load()
+                return image
+        except Exception as exc:
+            raise excs.Error('Failed to download generated image from together.ai.') from exc
+    raise excs.Error('Response does not contain a generated image.')
 
 
 __all__ = local_public_names(__name__)

{pixeltable-0.2.19 → pixeltable-0.2.20}/pixeltable/globals.py

@@ -16,6 +16,7 @@ from pixeltable.dataframe import DataFrameResultSet
 from pixeltable.env import Env
 from pixeltable.iterators import ComponentIterator
 from pixeltable.metadata import schema
+from pixeltable.utils.filecache import FileCache
 
 _logger = logging.getLogger('pixeltable')
 
@@ -193,6 +194,7 @@ def create_view(
     )
     Catalog.get().paths[path] = view
     _logger.info(f'Created view `{path_str}`.')
+    FileCache.get().emit_eviction_warnings()
     return view
 
 

{pixeltable-0.2.19 → pixeltable-0.2.20}/pixeltable/io/globals.py

@@ -43,7 +43,7 @@ def create_label_studio_project(
     The API key and URL for a valid Label Studio server must be specified in Pixeltable config. Either:
 
     * Set the `LABEL_STUDIO_API_KEY` and `LABEL_STUDIO_URL` environment variables; or
-    * Specify `api_key` and `url` fields in the `label-studio` section of `$PIXELTABLE_HOME/config.yaml`.
+    * Specify `api_key` and `url` fields in the `label-studio` section of `$PIXELTABLE_HOME/config.toml`.
 
     __Requirements:__