oracle-ads 2.13.9rc0__py3-none-any.whl → 2.13.10__py3-none-any.whl

ads/feature_store/docs/source/cicd.rst

@@ -0,0 +1,137 @@
+CI/CD: Small and Large Data Use Cases
+**************************************
+
+Introduction
+============
+
+This section elaborates on different data ingestion modes (Append, Overwrite, Upsert) used within a feature store context. Understanding these modes is crucial for efficient data management and integration.
+
+
+.. image:: figures/cicd.png
+
+
+Feature Store with ML Jobs (Small Data Use Cases)
+=================================================
+
+Utilizing ML jobs within the Feature Store caters to scenarios dealing with small data volumes.
+
+**Usage Scenario**:
+
+- **ML Job Integration**: Integrate ML jobs into the CI/CD pipeline for feature extraction, transformation, and loading using small datasets.
+- **Automation with CI/CD**: Automate the deployment and execution of ML job scripts through CI/CD pipelines for feature management in the Feature Store environment.
+
+.. seealso::
+   `Feature Store Creation and Ingestion with Jobs <https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/feature_store/tutorials/feature_store_creation_ingestion_with_jobs>`_
+
+Feature Store with Dataflow (Large Data Use Cases)
+==================================================
+
+Dataflow is a robust solution when handling large-scale data processing within the Feature Store.
+
+**Usage Scenario**:
+
+- **Dataflow Integration**: Incorporate Dataflow capabilities within the CI/CD pipeline for efficient feature extraction, transformation, and loading with large datasets.
+- **Scalable Processing**: Leverage Dataflow's scalable processing power to handle and manage extensive data volumes effectively.
+
+.. seealso::
+   `Feature Store Creation and Ingestion with Dataflow <https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/feature_store/tutorials/feature_store_creation_ingestion_with_dataflow>`_
+
+Modes of ingestion
+===================
+
+Append Mode
+###########
+
+In Append mode, new data is added to the existing dataset or table. This process involves appending new records to the end of the existing dataset without altering or removing pre-existing data. It's suitable for continuous addition of new records while preserving the integrity of the existing dataset.
+
+.. code-block:: python3
+
+   from ads.feature_store.feature_group_job import BatchIngestionMode
+   from ads.feature_store.feature_group import FeatureGroup
+
+   feature_group = FeatureGroup.from_id("<unique_id>")
+   feature_group.materialise(airports_df, ingestion_mode=BatchIngestionMode.APPEND)
+
+Overwrite Mode
+##############
+
+Overwrite mode replaces the entire existing dataset with the new data being saved. If the target table or dataset already exists, it is completely dropped and recreated with the new data. This mode is beneficial for refreshing the dataset entirely with the most recent data, discarding all previous records.
+
+.. code-block:: python3
+
+   from ads.feature_store.feature_group_job import BatchIngestionMode
+   from ads.feature_store.feature_group import FeatureGroup
+
+   feature_group = FeatureGroup.from_id("<unique_id>")
+   feature_group.materialise(airports_df, ingestion_mode=BatchIngestionMode.OVERWRITE)
+
+Upsert Mode (Merge Mode)
+########################
+
+Upsert mode, also known as merge mode, updates existing records in the dataset based on a specified condition or primary key. If a record with the same key exists, it is updated with the new data; otherwise, a new record is inserted. This mode helps in maintaining data consistency and avoiding duplicates.
+
+.. code-block:: python3
+
+   from ads.feature_store.feature_group_job import BatchIngestionMode
+   from ads.feature_store.feature_group import FeatureGroup
+
+   feature_group = FeatureGroup.from_id("<unique_id>")
+   feature_group.materialise(airports_df, ingestion_mode=BatchIngestionMode.UPSERT)
+
+Complete Mode
+#############
+
+Complete Mode involves rewriting the full output for each batch interval. It provides the complete result set for the computation performed on that batch interval. This mode is suitable when requiring a comprehensive, updated result set for every batch interval, regardless of whether it's a complete rewrite of the entire output.
+
+.. code-block:: python3
+
+   from ads.feature_store.feature_group_job import StreamingIngestionMode
+   from ads.feature_store.feature_group import FeatureGroup
+
+   feature_group = FeatureGroup.from_id("<unique_id>")
+   feature_group.materialise_stream(df, ingestion_mode=StreamingIngestionMode.COMPLETE)
+
+Update Mode
+###########
+
+Update Mode is specifically designed to capture only the changed records in the output. It outputs the rows that have been updated since the last batch, maintaining intermediate state and providing insights into the delta or changes between different batches. This mode is ideal for scenarios involving aggregations or incremental updates to the output data.
+
+.. code-block:: python3
+
+   from ads.feature_store.feature_group_job import StreamingIngestionMode
+   from ads.feature_store.feature_group import FeatureGroup
+
+   feature_group = FeatureGroup.from_id("<unique_id>")
+   feature_group.materialise_stream(df, ingestion_mode=StreamingIngestionMode.UPDATE)
+
+Append Mode
+###########
+
+Append Mode appends newly generated results from each micro-batch to the output sink. It works when the computation produces only new records and does not modify or update existing data in the output. This mode suits scenarios where the result set continuously grows, with each batch contributing new records without altering existing ones.
+
+.. code-block:: python3
+
+   from ads.feature_store.feature_group_job import StreamingIngestionMode
+   from ads.feature_store.feature_group import FeatureGroup
+
+   feature_group = FeatureGroup.from_id("<unique_id>")
+   feature_group.materialise_stream(df, ingestion_mode=StreamingIngestionMode.APPEND)
+
+
+Considerations for Usage
+########################
+
+- **Performance Implications:** Each mode has its performance considerations. Append tends to be faster as it merely adds new records, while Overwrite involves dropping and recreating the entire dataset, potentially impacting performance and resource usage.
+- **Data Consistency:** Upsert mode helps in maintaining data consistency by updating existing records and inserting new ones, ensuring the integrity of the dataset.
+
+Use Cases
+###########
+
+- **Append:** Real-time streaming data, continuous addition of new records without altering existing data.
+- **Overwrite:** Periodic data refreshes or complete replacement of the dataset with the latest information.
+- **Upsert:** Synchronizing datasets, updating existing records, and avoiding duplicates, commonly used in databases and data warehouses.
+
+Conclusion
+###########
+
+Understanding the nuances of these ingestion modes is crucial for choosing the appropriate method based on specific requirements and use cases. It ensures efficient data management and integration within the feature store environment.

ads/feature_store/docs/source/conf.py

@@ -0,0 +1,86 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*--
+
+# Copyright (c) 2022, 2023 Oracle and/or its affiliates.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
+
+import os
+import sys
+
+sys.path.insert(0, os.path.abspath("../../"))
+
+version = "1.0.4"
+release = version
+
+
+# -- Project information -----------------------------------------------------
+
+project = "oci_feature_store"
+copyright = "Oracle and/or its affiliates 2022, 2023"
+author = "Oracle Corporation"
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    "sphinx.ext.napoleon",
+    "sphinx.ext.autodoc",
+    "sphinx.ext.doctest",
+    "sphinx.ext.mathjax",
+    "sphinx.ext.ifconfig",
+    "sphinx.ext.autodoc",
+    "sphinx.ext.todo",
+    "sphinx.ext.extlinks",
+    "sphinx.ext.intersphinx",
+    "sphinx.ext.graphviz",
+    "nbsphinx",
+    "sphinx_code_tabs",
+    "sphinx_copybutton",
+    "sphinx.ext.duration",
+    "sphinx.ext.autosummary",
+    "sphinx.ext.intersphinx",
+    "sphinx.ext.viewcode",
+    "sphinx_autorun",
+]
+
+intersphinx_mapping = {
+    "python": ("https://docs.python.org/3/", None),
+    "sphinx": ("https://www.sphinx-doc.org/en/master/", None),
+}
+intersphinx_disabled_domains = ["std"]
+
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates"]
+
+# Unless we want to expose real buckets and namespaces
+nbsphinx_allow_errors = True
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ["build", "**.ipynb_checkpoints", "Thumbs.db", ".DS_Store"]
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = "furo"
+language = "en"
+
+html_theme_options = {
+    "light_logo": "logo-light-mode.png",
+    "dark_logo": "logo-dark-mode.png",
+    "collapse_navigation": True,
+    "sticky_navigation": True,
+    "navigation_depth": 4,
+}
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ["figures"]

ads/feature_store/docs/source/data_versioning.rst

@@ -0,0 +1,33 @@
+Data Versioning
+****************
+
+Data versioning is a practice aimed at recording the various data commits integrated into a particular feature group and dataset. This involves tracking changes in data over time while maintaining consistent schema structures and feature definitions within a shared schema version. Note that, in Feature Store, data versioning features are available only for offline feature groups.
+
+.. image:: figures/data_versioning.png
+
+
+As Of
+======
+
+You can call the ``as_of()`` method of the ``FeatureGroup`` or ``Dataset`` instance to get the specified point in time and time traveled data.
+
+The ``.as_of()`` method takes the following optional parameters:
+
+- ``commit_timestamp: date-time``. The commit timestamp for the feature group.
+- ``version_number: int``. The version number for the feature group.
+
+.. code-block:: python3
+
+  # as_of feature group
+  df = feature_group.as_of(version_number=1)
+
+
+History
+=======
+
+You can call the ``history()`` method of the ``FeatureGroup`` or ``Dataset`` instance to show the history of the feature group.
+
+.. code-block:: python3
+
+  # Show history of feature group
+  df = feature_group.history()

ads/feature_store/docs/source/dataset.rst

@@ -0,0 +1,388 @@
+Dataset
+********
+
+A dataset is a collection of features that are used together to train a model or perform model inference.
+
+.. image:: figures/dataset.png
+
+Define
+======
+
+In an ADS Feature Store module, you can use the Python API or a yaml file to define a dataset.
+
+
+The following example defines a dataset and gives it a name. A ``Dataset`` instance is created.
+
+.. tabs::
+
+  .. code-tab:: Python3
+    :caption: Python
+
+    from ads.feature_store.dataset import Dataset
+
+    dataset = (
+        Dataset
+        .with_name("<dataset_name>")
+        .with_entity_id(<entity_id>)
+        .with_feature_store_id("<feature_store_id>")
+        .with_description("<dataset_description>")
+        .with_compartment_id("<compartment_id>")
+        .with_query('SELECT col FROM <entity_id>.<feature_group_name>')
+    )
+
+  .. code-tab:: Python3
+    :caption: YAML
+
+    from ads.feature_store.dataset import Dataset
+
+    yaml_string = """
+    kind: dataset
+    spec:
+      compartmentId: ocid1.compartment..<unique_id>
+      description: <dataset_description>
+      name: <dataset_name>
+      featureStoreId: <feature_store_id>
+    type: dataset
+    """
+
+    dataset = Dataset.from_yaml(yaml_string)
+
+
+Create
+======
+
+Use the the ``create()`` method of the ``Dataset`` instance to create a dataset.
+
+.. important::
+
+ This method does not persist any metadata or feature data in the Feature Store. To persist the dataset and save feature data including the metadata in the Feature Store, use the ``materialise()`` method with a dataframe. For simple queries with only one level of nesting, users do not need to define ``with_feature_groups``. However, in complex queries involving more than one level of nesting, users are required to define ``with_feature_groups``.
+
+
+.. tabs::
+
+  .. code-tab:: Python3
+    :caption: Simple SQL
+
+    from ads.feature_store.dataset import Dataset
+
+    dataset = (
+        Dataset
+        .with_name("<dataset_name>")
+        .with_entity_id(<entity_id>)
+        .with_feature_store_id("<feature_store_id>")
+        .with_description("<dataset_description>")
+        .with_compartment_id("<compartment_id>")
+        .with_query('SELECT col FROM <entity_id>.<feature_group_name>')
+    )
+
+    dataset.create()
+
+
+  .. code-tab:: Python3
+    :caption: Complex SQL
+
+    from ads.feature_store.dataset import Dataset
+    from ads.feature_store.feature_group import FeatureGroup
+
+    feature_group = FeatureGroup.from_id("<unique_id>")
+
+    dataset = (
+        Dataset
+        .with_name("<dataset_name>")
+        .with_entity_id(<entity_id>)
+        .with_feature_store_id("<feature_store_id>")
+        .with_description("<dataset_description>")
+        .with_compartment_id("<compartment_id>")
+        .with_query('SELECT col FROM (SELECT col FROM <entity_id>.<feature_group_name> WHERE condition = 'some_condition') AS nested_table;')
+        .with_feature_groups([feature_group])
+    )
+
+    # Create an dataset
+    dataset.create()
+
+
+Load
+====
+
+Use the ``from_id()`` method from the ``Dataset`` class to load an existing dataset by specifying the OCID. A ``Dataset`` instance is returned.
+
+.. code-block:: python3
+
+  from ads.feature_store.dataset import Dataset
+
+  dataset = Dataset.from_id("<unique_id>")
+
+Materialise
+===========
+
+Use the the ``materialise() -> DatasetJob`` method of the ``Dataset`` instance to load the data to dataset. To persist the dataset and save dataset data, including the metadata in the Feature Store, use ``materialise()``.
+
+The ``.materialise()`` method has the following parameters:
+
+- ``input_dataframe: Union[DataFrame, pd.DataFrame]``. The Spark dataframe or Pandas dataframe.
+- ``from_timestamp: str(Optional)``. The from timestamp of the dataset.
+- ``to_timestamp: str(Optional)``. The to timestamp of the dataset.
+- ``feature_option_details: FeatureOptionDetails(Optional)``. The feature option details for the materialise operation.
+    - ``write_config_details: (merge_schema: bool, overwrite_schema: bool)``. The write configuration details for the feature option details.
+    - ``read_config_details: (version_as_of: int, timestamp_as_of: datetime)``. The read configuration details for the feature option details.
+
+.. code-block:: python3
+
+  from ads.feature_store.dataset_job import DatasetJob
+
+  dataset_job: DatasetJob = dataset.materialise(dataframe)
+
+.. seealso::
+   :ref:`Dataset Job`
+
+
+Delete
+======
+
+Use the ``.delete()`` method on the ``Dataset`` instance to delete a dataset. A dataset can only be deleted when its associated entities are all deleted.
+
+.. code-block:: python3
+
+  dataset.delete()
+
+Get Last Dataset Job
+====================
+A dataset job is the processing instance of a dataset. Each dataset job includes validation and statistics results.
+
+With a ``Dataset`` instance, you can get the last dataset job details using ``get_last_job()``.
+
+.. code-block:: python3
+
+  dataset_job = dataset.get_last_job()
+
+Save Expectation Entity
+=======================
+Feature Store allows you to define expectations on data being materialised into a dataset instance. With a ``Dataset`` instance, save the expectation details using ``with_expectation_suite()`` with the following parameters:
+
+- ``expectation_suite: ExpectationSuite``. ``ExpectationSuite`` of the great expectation library.
+- ``expectation_type: ExpectationType``. The type of expectation.
+        - ``ExpectationType.STRICT``: Fail the job if the expectation isn't met.
+        - ``ExpectationType.LENIENT``: Pass the job even if the expectation isn't met.
+
+.. note::
+
+  `Great Expectations <https://docs.greatexpectations.io/docs/0.15.50/>`_  is an open source Python-based library that validates, documents, and profiles data. It automates testing, which is essential for managing complex code bases.
+
+.. image:: figures/validation.png
+
+.. code-block:: python3
+
+    from great_expectations.core import ExpectationSuite, ExpectationConfiguration
+    from ads.feature_store.common.enums import ExpectationType
+    from ads.feature_store.dataset import Dataset
+
+    dataset = Dataset.from_id("<unique_id>")
+
+    expectation_suite = ExpectationSuite(
+        expectation_suite_name="expectation_suite_name"
+    )
+    expectation_suite.add_expectation(
+        ExpectationConfiguration(
+            expectation_type="expect_column_values_to_not_be_null",
+            kwargs={"column": "<column>"},
+        )
+
+    dataset_resource = (
+            Dataset()
+            .with_description("dataset description")
+            .with_compartment_id(<compartment_id>)
+            .with_name(<name>)
+            .with_entity_id(entity_id)
+            .with_feature_store_id(feature_store_id)
+            .with_query(f"SELECT * FROM `{entity_id}`.{feature_group_name}")
+            .with_expectation_suite(
+                expectation_suite=expectation_suite,
+                expectation_type=ExpectationType.STRICT,
+            )
+        )
+
+Use the the ``get_validation_output()`` method of the dataset instance to fetch validation results for a specific ingestion job.
+The ``get_validation_output()`` method has the following optional parameters:
+
+- ``job_id: string``. The ID of the dataset job.
+
+``get_validation_output().to_pandas()`` Outputs the validation results for each expectation as a Pandas dataframe.
+
+.. image:: figures/dataset_validation_results.png
+
+``get_validation_output().to_summary()`` Outputs the overall summary of the validation as a Pandas dataframe.
+
+.. image:: figures/dataset_validation_summary.png
+
+.. seealso::
+
+    :ref:`Feature Validation`
+
+Statistics Computation
+========================
+During the materialisation, Feature Store performs computation of statistical metrics for all the features by default. The computation is configured using a ``StatisticsConfig`` object, which is passed at the creation of the dataset, or it can be updated later.
+
+.. code-block:: python3
+
+  # Define statistics configuration for selected features
+  from ads.feature_store.statistics_config import StatisticsConfig
+  stats_config = StatisticsConfig().with_is_enabled(True).with_columns(["column1", "column2"])
+
+This is used with ``Dataset`` instance.
+
+.. code-block:: python3
+
+  from ads.feature_store.dataset import Dataset
+
+  dataset = (
+        Dataset
+        .with_name("<dataset_name>")
+        .with_entity_id(<entity_id>)
+        .with_feature_store_id("<feature_store_id>")
+        .with_description("<dataset_description>")
+        .with_compartment_id("<compartment_id>")
+        .with_dataset_ingestion_mode(DatasetIngestionMode.SQL)
+        .with_query('SELECT col FROM <entity_id>.<feature_group_name>')
+        .with_statistics_config(stats_config)
+  )
+
+Use the the ``get_statistics()`` method of the ``Dataset`` instance to fetch the metrics for a specific ingestion job.
+
+The ``get_statistics()`` method has the following optional parameter:
+
+- ``job_id: string``. The ID of the dataset job.
+
+.. code-block:: python3
+
+  # Fetch stats results for a dataset job
+  df = dataset.get_statistics(job_id).to_pandas()
+
+.. image:: figures/dataset_statistics.png
+
+.. code-block:: python3
+
+  # Fetch and visualize stats for a dataset job
+  df = dataset.get_statistics(job_id).to_viz()
+
+.. image:: figures/dataset_statistics_viz.png
+
+
+.. seealso::
+
+    :ref:`Statistics`
+
+
+Get features
+============
+Use the the ``get_features_df()`` method of the ``Dataset`` instance to fetch features in a dataset.
+
+.. code-block:: python3
+
+  # Fetch features for a dataset
+  df = dataset.get_features_df()
+  df.show()
+
+
+Preview
+========
+
+.. deprecated:: 1.0.3
+   Use :func:`as_of` instead.
+
+Use the the ``preview()`` method of the ``Dataset`` instance to preview the dataset.
+
+The ``.preview()`` method has the following optional parameters:
+- ``timestamp: date-time``. The commit timestamp for the dataset.
+- ``version_number: int``. The version number for the dataset.
+- ``row_count: int``. Defaults to 10. The total number of rows to return.
+
+.. code-block:: python3
+
+  # Preview dataset
+  df = dataset.preview(row_count=50)
+  df.show()
+
+As Of
+=======
+
+Use the the ``as_of()`` method of the ``Dataset`` instance to get a specified point in time and time traveled data.
+
+The ``.as_of()`` method has the following optional parameters:
+
+- ``commit_timestamp: date-time``. The commit timestamp for the dataset.
+- ``version_number: int``. The version number for the dataset.
+
+.. code-block:: python3
+
+  # as_of feature group
+  df = dataset.as_of(version_number=1)
+
+
+Restore
+=======
+Use the the ``restore()`` method of the ``Dataset`` instance to restore the dataset to a particular version and timestamp.
+
+The ``.restore()`` method has the following optional parameters:
+- ``timestamp: date-time``. The commit timestamp for the dataset.
+- ``version_number: int``. The version number for the dataset.
+
+.. code-block:: python3
+
+  # Restore the dataset to a particular version and timestamp
+  df = feature_group.restore(version_number=2)
+  df.show()
+
+
+Profile
+=======
+Use the the ``profile()`` method of the ``Dataset`` instance to profile the dataset.
+
+.. code-block:: python3
+
+  # Profile dataset
+  df = dataset.profile()
+  df.show()
+
+
+History
+=======
+Use the the ``history()`` method of the ``Dataset`` instance to show the history of the dataset.
+
+.. code-block:: python3
+
+  # Show history of dataset
+  df = dataset.history()
+  df.show()
+
+
+Visualize Lineage
+=================
+
+Use the ``show()`` method on the ``Dataset`` instance to visualize the lineage of the dataset.
+
+The ``show()`` method has the following optional parameter:
+
+- ``rankdir: (str, optional)``. Defaults to ``LR``. The allowed values are ``TB`` or ``LR``. This parameter is applicable only for ``graph`` mode. It renders the direction of the graph as either top to bottom (TB) or left to right (LR).
+
+
+.. code-block:: python3
+
+  dataset.show()
+
+The following is an example of the output:
+
+.. figure:: figures/dataset_lineage.png
+  :width: 400
+
+
+Add Model Details
+=================
+
+Use the the ``add_models()`` method of the ``Dataset`` instance to add model IDs to the dataset.
+The ``.add_models()`` method has the following parameter:
+
+- ``model_details: ModelDetails``.  Provide ``items: List[str]`` as parameter and model IDs are passed as items.
+
+.. code-block:: python3
+
+  dataset.add_models(ModelDetails().with_items([<ocid1.datasciencemodel..<unique_id>]))

ads/feature_store/docs/source/dataset_job.rst

@@ -0,0 +1,27 @@
+.. _Dataset Job:
+
+Dataset Job
+***********
+
+A dataset job is the processing instance of a dataset. Each dataset job includes validation and statistics results.
+
+
+Load
+====
+
+Use the ``from_id()`` method from the ``DatasetJob`` class to load an existing dataset job by specifying its OCID. A``DatasetJob`` instance is returned.
+
+.. code-block:: python3
+
+  from ads.feature_store.dataset_job import DatasetJob
+
+  dataset_job = DatasetJob.from_id("<unique_id>")
+
+Delete
+======
+
+Use the ``.delete()`` method on the ``DatasetJob`` instance to delete a dataset job. A dataset job can only be deleted when its associated entities are all deleted.
+
+.. code-block:: python3
+
+  dataset_job.delete()