oracle-ads 2.13.9rc0__py3-none-any.whl → 2.13.10rc0__py3-none-any.whl

ads/feature_store/docs/source/feature_store.rst

@@ -0,0 +1,122 @@
+Feature Store
+*************
+
+Feature store is the top-level entity for a Feature Store.
+
+Define
+======
+
+In an ADS Feature Store module, you can use the Python API or a .yaml file to define a Feature Store.
+
+
+The following example defines a Feature Store and gives it a name. A ``FeatureStore`` instance is created.
+
+.. tabs::
+
+  .. code-tab:: Python3
+    :caption: Python
+
+    from ads.feature_store.feature_store import FeatureStore
+
+    feature_store = (
+            FeatureStore().
+            with_description(<feature_store_description>).
+            with_compartment_id("ocid1.compartment..<unique_id>").
+            with_name(<feature_store_name>).
+            with_offline_config(
+                metastore_id=metastoreId
+            )
+    )
+
+  .. code-tab:: Python3
+    :caption: YAML
+
+    from ads.feature_store.feature_store import FeatureStore
+
+    yaml_string = """
+    kind: feature_store
+    spec:
+      compartmentId: ocid1.compartment..<unique_id>
+      description: <feature_store_description>
+      name: <feature_store_name>
+      featureStoreId: <feature_store_id>
+    type: feature_store
+    """
+
+    feature_store = FeatureStore.from_yaml(yaml_string)
+
+
+Create
+======
+
+Use the ``create()`` method of the ``FeatureStore`` instance to create a Feature Store.
+
+.. code-block:: python3
+
+  # Create an Feature Store
+  feature_store.create()
+
+
+Load
+====
+
+Use the ``from_id()`` method from the ``FeatureStore`` class to load an existing Feature Store by specifying its OCID. A ``FeatureStore`` instance is returned.
+
+.. code-block:: python3
+
+  from ads.feature_store.feature_store import FeatureStore
+
+  feature_store = FeatureStore.from_id("<unique_id>")
+
+Delete
+======
+
+Use the ``.delete()`` method on the ``FeatureStore`` instance to delete a Feature Store. A Feature Store can only be deleted when its associated entities are all deleted.
+
+.. code-block:: python3
+
+  feature_store.delete()
+
+SQL
+===
+Use the ``sql()`` method of the ``FeatureStore`` instance to query a Feature Store.
+
+The following example queries a Feature Store using SQL:
+
+.. code-block:: python3
+
+  # Fetch the entity id. Entity id is used as database name in Feature Store
+  entity_id = entity.id
+
+  # Form a query with entity id and fetch the results
+  sql = (f"SELECT feature_group_a.* "
+       f"FROM {entity_id}.feature_group_a "
+       f"JOIN {entity_id}.feature_group_b "
+       f"ON {entity_id}.feature_group_a.col_1={entity_id}.feature_group_b.col_2 "
+       f"JOIN {entity_id}.feature_group_a.col_1={entity_id}.feature_group_b.col_3 ")
+
+  # Run the sql query and fetch the results as data-frame
+  df = feature_store.sql(sql)
+
+Create an Entity
+================
+Use the ``create_entity()`` method of the ``FeatureStore`` instance to create an ``Entity``.
+
+.. code-block:: python3
+
+  # Create a Feature Store entity
+  feature_store.create_entity(name="<ENTITY_NAME>")
+
+Create a Transformation
+=======================
+Transformations in a Feature Store are the operations and processes applied to raw data to create, modify, or derive new features for use as inputs for machine learning models. These transformations are necessary for improving the quality, relevance, and usefulness of features that then enhance the performance of models.
+You can call the ``create_transformation()`` method of the FeatureStore instance to create a ``Transformation``.
+
+.. code-block:: python3
+
+  # Create a Feature Store entity
+  feature_store.create_transformation(
+      source_code_func="<FUNCTION>",
+      transformation_mode="SQL|PANDAS"
+      name="<TRANSFORMATION NAME>"
+  )

ads/feature_store/docs/source/feature_store_class.rst

@@ -0,0 +1,123 @@
+feature_store package
+*********************
+
+Submodules
+==========
+
+ads.feature_store.query package
+###############################
+
+.. automodule:: ads.feature_store.query.filter
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.query.join
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.query.query
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+ads.feature_store package
+#########################
+.. automodule:: ads.feature_store.dataset
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.dataset_job
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.entity
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.feature
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.feature_group
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.feature_group_expectation
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.feature_group_job
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.feature_option_details
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.feature_store
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.feature_store_registrar
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.input_feature_detail
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.model_details
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.statistics_config
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.transformation
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.validation_output
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+ads.feature_store.statistics package
+####################################
+
+.. automodule:: ads.feature_store.statistics.statistics
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+ads.feature_store.common package
+################################
+
+.. automodule:: ads.feature_store.common.enums
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. automodule:: ads.feature_store.common.spark_session_singleton
+   :members:
+   :undoc-members:
+   :show-inheritance:

ads/feature_store/docs/source/feature_validation.rst

@@ -0,0 +1,66 @@
+.. _Feature Validation:
+
+Feature Validation
+******************
+
+Feature validation is the process of checking the quality and accuracy of the features used in a machine learning model. This is important because features that aren't accurate or reliable can lead to poor model performance. Feature Store lets you define expectations on the data that is being materialised into feature groups and datasets. The Great Expectations open source library is used to define expectations.
+
+There are two types of expectations possible for expectation suite
+
+- ``ExpectationType.STRICT``: Requires meeting set expectations. The job fails if criteria aren't met.
+- ``ExpectationType.LENIENT``: Permits unmet expectations. The job continues despite criteria not being fully satisfied.
+
+
+.. note::
+  `Great Expectations <https://docs.greatexpectations.io/docs/0.15.50/>`_  is an open source Python-based library that validates, documents, and profiles data. It automates testing, which is essential for managing complex code bases.
+
+.. image:: figures/data_validation.png
+
+Expectations
+============
+An expectation is a verifiable assertion about your data.
+
+The following example defines an expectation:
+
+.. code-block:: python3
+
+    from great_expectations.core import ExpectationSuite, ExpectationConfiguration
+    from ads.feature_store.common.enums import ExpectationType
+    from ads.feature_store.feature_group import FeatureGroup
+
+    feature_group = FeatureGroup.from_id("<unique_id>")
+
+    # Create an Expectation
+    expectation_suite = ExpectationSuite(
+        expectation_suite_name="test_suite"
+    )
+    expectation_suite.add_expectation(
+        ExpectationConfiguration(
+            expectation_type="expect_column_values_to_not_be_null",
+            kwargs={"column": "COL1"},
+        )
+    )
+    expectation_suite.add_expectation(
+        ExpectationConfiguration(
+            expectation_type="expect_column_values_to_be_between",
+            kwargs={"column": "COL2", "min_value": 0, "max_value": 30},
+        )
+    )
+
+    feature_group.with_expectation_suite(expectation_suite, expectation_type = ExpectationType.STRICT)
+    feature_group.update()
+
+Expectations Suite
+===================
+
+An expectation suite is a collection of verifiable assertions. For example, expectations about your data.
+
+The following example defines an expectation suite:
+
+.. code-block:: python3
+
+    # Create an Expectation Suite
+    expectation_suite = ExpectationSuite(
+        expectation_suite_name=<expectation_suite_name>
+    )
+    expectation_suite.add_expectation(expect_config)

ads/feature_store/docs/source/index.rst

@@ -0,0 +1,81 @@
+=============================================
+Welcome to oci-feature-store's documentation!
+=============================================
+
+Feature Store
+=============
+
+|PyPI|_ |PysparkConda|_ |Notebook Examples|_ |Delta|_ |PySpark|_ |Great Expectations|_ |Pandas|_ |PyArrow|_
+
+.. |PyPI| image:: https://img.shields.io/badge/python-3.8-blue?style=for-the-badge&logo=pypi&logoColor=white
+..  _PyPI: https://pypi.org/project/oracle-ads/
+.. |PysparkConda| image:: https://img.shields.io/badge/fspyspark32_p38_cpu_v2-1.0-blue?style=for-the-badge&logo=pypi&logoColor=white
+..  _PysparkConda: https://docs.oracle.com/en-us/iaas/data-science/using/conda-pyspark-fam.htm
+.. |Notebook Examples| image:: https://img.shields.io/badge/docs-notebook--examples-blue?style=for-the-badge&logo=pypi&logoColor=white
+..  _Notebook Examples: https://github.com/oracle-samples/oci-data-science-ai-samples/tree/master/notebook_examples
+.. |Delta| image:: https://img.shields.io/badge/delta-2.0.1-blue?style=for-the-badge&logo=pypi&logoColor=white
+..  _Delta: https://delta.io/
+.. |PySpark| image:: https://img.shields.io/badge/pyspark-3.2.1-blue?style=for-the-badge&logo=pypi&logoColor=white
+..  _PySpark: https://spark.apache.org/docs/3.2.1/api/python/index.html
+.. |Great Expectations| image:: https://img.shields.io/badge/greatexpectations-0.17.19-blue?style=for-the-badge&logo=pypi&logoColor=white
+..  _Great Expectations: https://greatexpectations.io/
+.. |Pandas| image:: https://img.shields.io/badge/pandas-1.5.3-blue?style=for-the-badge&logo=pypi&logoColor=white
+..  _Pandas: https://pandas.pydata.org/
+.. |PyArrow| image:: https://img.shields.io/badge/pyarrow-11.0.0-blue?style=for-the-badge&logo=pypi&logoColor=white
+..  _PyArrow: https://arrow.apache.org/docs/python/index.html
+
+.. toctree::
+    :maxdepth: 2
+    :caption: Getting started:
+
+    overview
+    setup_feature_store
+    quickstart
+
+.. toctree::
+    :maxdepth: 2
+    :caption: Feature store entities:
+
+    feature_store
+    entity
+    transformation
+    feature_group
+    feature_group_job
+    dataset
+    dataset_job
+
+.. toctree::
+    :maxdepth: 2
+    :caption: Feature store concepts:
+
+    cicd
+    statistics
+    feature_validation
+    data_versioning
+    ui
+
+.. toctree::
+    :maxdepth: 2
+    :caption: Demo and examples:
+
+    demo
+    notebook
+
+
+.. toctree::
+    :maxdepth: 2
+    :caption: Release notes:
+
+    release_notes
+
+.. toctree::
+    :hidden:
+    :maxdepth: 5
+    :caption: Classes:
+
+    module
+
+License
++++++++
+
+Copyright (c) 2020, 2022 Oracle and/or its affiliates. Licensed under the `Universal Permissive License v1.0 <https://oss.oracle.com/licenses/upl/>`_

ads/feature_store/docs/source/module.rst

@@ -0,0 +1,8 @@
+===================
+Class Documentation
+===================
+
+.. toctree::
+   :maxdepth: 3
+
+   feature_store_class

ads/feature_store/docs/source/notebook.rst

@@ -0,0 +1,94 @@
+.. _Notebook Examples:
+
+
+Notebook Examples
+*****************
+
+Below is a compilation of tutorials focused on understanding and utilizing Feature Stores. You can find the raw notebook files in our `tutorials repository <https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/feature_store/tutorials/README.md>`_.
+
+Quick start examples
+####################
+
+.. admonition:: Notebook Examples
+  :class: note
+
+  .. list-table::
+    :widths: 50 50
+    :header-rows: 1
+
+    * - Jupyter Notebook
+      - Description
+
+    * - `Feature store querying <https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/notebook_examples/feature_store_querying.ipynb>`__
+      - | * Ingestion of data.
+        | * Querying and exploration of data.
+
+    * - `Feature store quickstart <https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/notebook_examples/feature_store_quickstart.ipynb>`__
+      - | * Ingestion of data.
+        | * Querying and exploration of data.
+
+    * - `Schema enforcement and schema evolution <https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/notebook_examples/feature_store_schema_evolution.ipynb>`__
+      - | * ``Schema evolution`` allows you to easily change a table's current schema to accommodate data that is changing over time.
+        | * ``Schema enforcement``, also known as schema validation, is a safeguard in Delta Lake that ensures data quality by rejecting writes to a table that don't match the table's schema.
+
+    * - `Storage of medical records in feature store <https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/notebook_examples/feature_store_ehr_data.ipynb>`__
+      - | Example to demonstrate storage of medical records in Feature Store.
+
+Big data operations using OCI DataFlow
+######################################
+
+.. admonition:: Notebook Examples
+  :class: note
+
+  .. list-table::
+    :widths: 50 50
+    :header-rows: 1
+
+    * - Jupyter Notebook
+      - Description
+
+    * - `Big data operations with feature store <https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/notebook_examples/feature_store_spark_magic.ipynb>`__
+      - | * Ingestion of data using Spark Magic.
+        | * Querying and exploration of data using Spark Magic.
+
+Streaming operations using Spark Streaming
+##########################################
+
+.. admonition:: Notebook Examples
+  :class: note
+
+  .. list-table::
+    :widths: 50 50
+    :header-rows: 1
+
+    * - Jupyter Notebook
+      - Description
+
+    * - `Streaming operations with feature store <https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/notebook_examples/feature_store_streaming_data_frame.ipynb>`__
+      - | * Ingestion of data using spark streaming.
+        | * Modes of ingestion: ``COMPLETE`` and ``APPEND``.
+
+LLM Use cases
+#############
+
+.. admonition:: Notebook Examples
+  :class: note
+
+  .. list-table::
+    :widths: 50 50
+    :header-rows: 1
+
+    * - Jupyter Notebook
+      - Description
+
+    * - `Embeddings in Feature Store <https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/notebook_examples/feature_store_embeddings.ipynb>`__
+      - | * ``Embedding feature stores`` are optimized for fast and efficient retrieval of embeddings. This is important because embeddings can be high-dimensional and computationally expensive to calculate. By storing them in a dedicated store, you can avoid the need to recalculate embeddings for the same data repeatedly.
+
+    * - `Synthetic data generation in feature store using OpenAI and FewShotPromptTemplate <https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/notebook_examples/feature_store_medical_synthetic_data_openai.ipynb>`__
+      - | * ``Synthetic data`` is artificially generated data, rather than data collected from real-world events. It's used to simulate real data without compromising privacy or encountering real-world limitations.
+
+    * - `PII Data redaction, Summarise Content and Translate content using doctran and open AI <https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/notebook_examples/feature_store_pii_redaction_and_transformation.ipynb>`__
+      - | * One way to think of Doctran is a LLM-powered black box where messy strings go in and nice, clean, labelled strings come out. Another way to think about it is a modular, declarative wrapper over OpenAI's functional calling feature that significantly improves the developer experience.
+
+    * - `OpenAI embeddings in feature store <https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/notebook_examples/feature_store_embeddings_openai.ipynb>`__
+      - | ``Embedding feature stores`` are optimized for fast and efficient retrieval of embeddings. This is important because embeddings can be high-dimensional and computationally expensive to calculate. By storing them in a dedicated store, you can avoid the need to recalculate embeddings for the same data repeatedly.

ads/feature_store/docs/source/overview.rst

@@ -0,0 +1,47 @@
+Overview
+********
+
+Managing many datasets, data sources, and transformations for machine learning is complex and costly. Poorly cleaned data, data issues, bugs in transformations, data drift, and training serving skew all lead to increased model development time and poor model performance. Feature Store solves many of the problems because it is a centralized way to transform and access data for training and serving time, Feature Stores help define a Feature Store pipeline for ingestion of data and querying of data.
+
+.. image:: figures/feature_store_overview.png
+
+ADS Feature Store is a stack-based solution that is deployed in your tenancy using OCI Resource Manager.
+
+The following are brief descriptions of key concepts and the main components of ADS Feature Store.
+
+- ``Feature Vector``:  A set of feature values for any one primary and identifier key. For example, all and a subset of features of customer ID 2536 can be called as one feature vector .
+- ``Feature``: A feature is an individual measurable property or characteristic of an event being observed.
+- ``Entity``: An entity is a group of semantically related features. The first step a consumer of features would typically do when accessing the Feature Store service is to list the entities and the entities associated with features. Another way to look at it is that an entity is an object or concept that's described by its features. Examples of entities are customer, product, transaction, review, image, and document.
+- ``Feature Group``: A feature group in a Feature Store is a collection of related features that are often used together in ML models. It serves as an organizational unit in the Feature Store for users to manage, version, and share features across different ML projects. By organizing features into groups, data scientists and ML engineers can efficiently discover, reuse, and collaborate on features reducing the redundant work and ensuring consistency in feature engineering.
+- ``Feature Group Job``: Feature group jobs are the processing instance of a feature group. Each feature group job includes validation results and statistics results.
+- ``Dataset``: 	A dataset is a collection of features that are used together to either train a model or perform model inference.
+- ``Dataset Job``: A dataset job is the processing instance of a dataset. Each dataset job includes validation results and statistics results.
+
+.. important::
+
+      Prerequisite : Contact #oci-feature-store_early-preview to get your tenancy whitelisted for early access of ADS Feature Store.
+
+.. important::
+
+  The ADS Feature Store supports and requires the following versions:
+
+    .. list-table::
+      :widths: 25 75
+      :header-rows: 1
+
+      * - Package Name
+        - Latest Version
+      * - python
+        - .. image:: https://img.shields.io/badge/python-3.8-blue?style=for-the-badge&logo=pypi&logoColor=white
+      * - fspyspark32_p38_cpu_v2
+        - .. image:: https://img.shields.io/badge/fspyspark32_p38_cpu_v2-1.0-blue?style=for-the-badge&logo=pypi&logoColor=white
+      * - delta-spark
+        - .. image:: https://img.shields.io/badge/delta-2.0.1-blue?style=for-the-badge&logo=pypi&logoColor=white
+      * - pyspark
+        - .. image:: https://img.shields.io/badge/pyspark-3.2.1-blue?style=for-the-badge&logo=pypi&logoColor=white
+      * - great-expectations
+        - .. image:: https://img.shields.io/badge/greatexpectations-0.17.19-blue?style=for-the-badge&logo=pypi&logoColor=white
+      * - pandas
+        - .. image:: https://img.shields.io/badge/pandas-1.5.3-blue?style=for-the-badge&logo=pypi&logoColor=white
+      * - pyarrow
+        - .. image:: https://img.shields.io/badge/pyarrow-11.0.0-blue?style=for-the-badge&logo=pypi&logoColor=white