oracle-ads 2.13.9rc0__py3-none-any.whl → 2.13.10rc0__py3-none-any.whl

ads/feature_store/docs/source/quickstart.rst

@@ -0,0 +1,176 @@
+Quick start
+************
+1. Create a `Data Science notebook session <https://docs.oracle.com/en-us/iaas/data-science/using/create-notebook-sessions.htm>`__ to access jupyterlab interface.
+
+2. Open a terminal in the notebook session, and then install the ``fspyspark32_p38_cpu_v2`` plugin:
+
+  ..  code-block:: shell
+
+     odsc conda install -s fspyspark32_p38_cpu_v3
+
+3. Download the notebook examples from the example notebook section.
+
+.. seealso::
+   Refer to :ref:`Notebook Examples` which contains more examples for using feature store.
+
+4. Upload the notebook in the notebook session, and then run the notebook after replacing the required variables.
+
+
+
+**Feature Store and Data Science Concepts:**
+
+- Getting started with  `OCI Data Science Jobs <https://docs.oracle.com/en-us/iaas/data-science/using/jobs-about.htm>`__
+- Getting started with  `Oracle Accelerated Data Science SDK <https://accelerated-data-science.readthedocs.io/en/latest/index.html>`__ to simplify `creating <https://accelerated-data-science.readthedocs.io/en/latest/user_guide/jobs/data_science_job.html#define-a-job>`__ and `running <https://accelerated-data-science.readthedocs.io/en/latest/user_guide/jobs/data_science_job.html#run-a-job-and-monitor-outputs>`__ Jobs
+- Getting started with  `Data Science Environments <https://docs.oracle.com/en-us/iaas/data-science/using/conda_environ_list.htm>`__
+- Getting started with  `Custom Conda Environments <https://docs.oracle.com/en-us/iaas/data-science/using/conda_create_conda_env.htm>`__
+
+**Authentication and Policies:**
+
+- Getting started with `OCI Data Science Policies <https://docs.oracle.com/en-us/iaas/data-science/using/policies.htm>`__
+- `API Key-Based Authentication <https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_authentication_methods.htm#sdk_authentication_methods_api_key>`__ - ``api_key``
+- `Resource Principal Authentication <https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_authentication_methods.htm#sdk_authentication_methods_resource_principal>`__ - ``resource_principal``
+- `Instance Principal Authentication <https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_authentication_methods.htm#sdk_authentication_methods_instance_principaldita>`__ - ``instance_principal``
+
+.. seealso::
+
+   Review the `Terraform section <https://objectstorage.us-ashburn-1.oraclecloud.com/p/hh2NOgFJbVSg4amcLM3G3hkTuHyBD-8aE_iCsuZKEvIav1Wlld-3zfCawG4ycQGN/n/ociodscdev/b/oci-feature-store/o/beta/index.html#document-terraform>`__ for setting up feature store server.
+
+.. warning::
+
+   1. Feature store doesn’t allow parallel execution of similar logical constructs. Creation is sequential.
+   2. If a failure occurs, processing stops, and rollback can’t happen. Retrying the operation isn’t supported.
+   3. Define exactly one feature store construct in the YAML file. Creation of multiple feature store constructs in the YAML file causes a failure.
+   4. To allow references in the feature store definition, the name of the defined logical constructs in the YAML file must be unique.
+
+.. tabs::
+
+  .. code-tab:: Python3
+    :caption: Python
+
+    from ads.feature_store.feature_group_expectation import Expectation, Rule, ExpectationType, ValidationEngineType
+    from ads.feature_store.feature_store import FeatureStore
+    from ads.feature_store.input_feature_detail import FeatureDetail, FeatureType
+    from ads.feature_store.transformation import TransformationMode
+    import ads
+
+    compartment_id = "ocid1.compartment.<unique_id>"
+    metastore_id = "ocid1.datacatalogmetastore.oc1.iad.<unique_id>"
+    api_gateway_endpoint = "https://**.{region}.oci.customer-oci.com/20230101"
+    os.environ["OCI_FS_SERVICE_ENDPOINT"] = api_gateway_endpoint
+
+    ads.set_auth(auth="api_key")
+
+    # step1: Create the feature store
+    feature_store_resource = (
+        FeatureStore()
+        .with_description("<feature_store_description>")
+        .with_compartment_id(compartment_id)
+        .with_name("<name>")
+        .with_offline_config(metastore_id=metastore_id)
+    )
+
+    feature_store = feature_store_resource.create()
+    entity = feature_store.create_entity(name="product")
+
+
+    # step2: Create the feature store
+    def transactions_df(dataframe, **kwargs):
+        columns = kwargs.get('columns', '*')  # The default is to select all columns if 'columns' isn't provided
+        where_clause = kwargs.get('where_clause', '')  # The default is to an empty where clause if 'where_clause' not provided
+
+        sql_query = f"""
+                    SELECT
+                        {columns}
+                    FROM
+                        {table_name}
+                    {where_clause}
+                """
+        return sql_query
+
+    transformation = feature_store.create_transformation(
+        transformation_mode=TransformationMode.SQL,
+        source_code_func=transactions_df
+    )
+
+
+    # step3: Create the expectation
+    expectation_suite = ExpectationSuite(expectation_suite_name="feature_definition")
+    expectation_suite.add_expectation(
+        ExpectationConfiguration(
+            expectation_type="expect_column_values_to_not_be_null",
+            kwargs={"column": "date"}
+        )
+    )
+
+    input_feature_details = [FeatureDetail("rule_name").with_feature_type(FeatureType.STRING).with_order_number(1)]
+
+    # step4: Create the statistics configuration
+    stats_config = StatisticsConfig().with_is_enabled(False)
+
+    # step5: Create the feature group
+    transformation_args = {"columns": "col1, col2", "where_clause": "col3 > 100"}
+    feature_group = entity.create_feature_group(
+                        primary_keys=["name"],
+                        partition_keys=["name"],
+                        input_feature_details,
+                        expectation_suite=expectation_suite,
+                        expectation_type=ExpectationType.LENIENT,
+                        statistics_config=stats_config,
+                        name="<feature_group_name>",
+                        transformation_id=transformation.id,
+                        transformation_kwargs=transformation_args
+                    )
+
+
+  .. code-tab:: Python3
+    :caption: YAML
+
+    from ads.feature_store.feature_store_registrar import FeatureStoreRegistrar
+
+    yaml_string = """
+    apiVersion: 20230101
+    kind: featureStore
+    spec:
+      name: *feature_store_name
+      offlineConfig:
+        metastoreId: *metastore_id
+
+      entity: &entity
+        - kind: entity
+          spec:
+            name: *entity_name
+
+
+      transformation: &transformation
+        - kind: transformation
+          spec:
+            name: *transformation_name
+            transformationMode: *transformation_mode
+            sourceCode: *source_code
+
+      featureGroup:
+        - kind: featureGroup
+          spec:
+            name: *feature_group_name
+            dataSource: *ds
+            description: *feature_group_desc
+            transformation: *transformation
+            entity: *entity
+            primaryKeys:
+              *fg_primary_key
+            inputFeatureDetails:
+              - name: *feature_name
+                featureType: *feature_type
+                orderNumber: 1
+
+      dataset:
+        - kind: dataset
+          spec:
+            name: *dataset_name
+            entity: *entity
+            datasetIngestionMode: *ingestion_mode
+            description: *dataset_description
+            query: *query_statement
+    """
+
+    feature_registrar = FeatureStoreRegistrar.from_yaml(yaml_string)

ads/feature_store/docs/source/release_notes.rst

@@ -0,0 +1,194 @@
+.. _Release Notes:
+
+=============
+Release Notes
+=============
+
+1.1.0
+-----
+.. note::
+
+    .. list-table::
+      :header-rows: 1
+
+      * - Package Name
+        - Latest Version
+        - Notes
+      * - Conda pack
+        - `fs_pyspark32_p38_cpu_v3`
+        -
+      * - ADS_VERSION
+        - oracle-ads==2.9.2
+        - `https://github.com/oracle/accelerated-data-science/releases/tag/v2.9.2`
+
+Release notes: January 5, 2024
+
+* [FEATURE] Onboarding feature store to OCI marketplace
+* [MAINTENANCE] Upgrade to helidon based server
+* [MAINTENANCE] Upgrade of ``mlm`` version to ``1.0.3``
+* [UI] Addition of drift detection tab in UI
+* [CONDA] Release of feature store v2 conda pack ``fspyspark32_p38_cpu_v3``
+* [DOCS] Addition of examples to demonstrate Ci/Cd using ``Ml Jobs`` and ``OCI DataFlow``
+
+1.0.4
+-----
+.. note::
+
+    .. list-table::
+      :header-rows: 1
+
+      * - Package Name
+        - Latest Version
+        - Notes
+      * - Conda pack
+        - `fspyspark32_p38_cpu_v2`
+        -
+      * - SERVICE_VERSION
+        - 0.1.371.master
+        -
+      * - ADS_VERSION
+        - oracle-ads==2.9.0
+        - `https://github.com/oracle/accelerated-data-science/releases/tag/v2.9.0`
+      * - Terraform Stack
+        - `link <https://objectstorage.us-ashburn-1.oraclecloud.com/p/vZogtXWwHqbkGLeqyKiqBmVxdbR4MK4nyOBqDsJNVE4sHGUY5KFi4T3mOFGA3FOy/n/idogsu2ylimg/b/oci-feature-store/o/beta/terraform/feature-store-terraform.zip>`__
+        -
+
+
+Release notes: November 15, 2023
+
+* [FEATURE] Streaming data-frame support in ``FeatureGroup`` and ``Dataset`` construct
+* [FEATURE] Support for custom spark transformations using ``Transformation``  construct
+* [MAINTENANCE] Decouple feature store client with oci client
+* [MAINTENANCE] Upgrade of ``mlm`` version to ``1.0.2``
+* [MAINTENANCE] Upgrade of ``great-expectations`` version to ``0.17.19``
+* [UI] Addition of dataset jobs UI tab
+* [UI] Addition of feature group jobs UI tab
+* [UI] Addition of feature store landing page
+* [DOCS] Addition of class documentation for feature store
+* [CONDA] Release of feature store v2 conda pack ``fspyspark32_p38_cpu_v2``
+
+1.0.3
+-----
+.. note::
+
+    .. list-table::
+      :header-rows: 1
+
+      * - Package Name
+        - Latest Version
+        - Notes
+      * - Conda pack
+        - `fspyspark32_p38_cpu_v1`
+        -
+      * - SERVICE_VERSION
+        - 0.1.256.master
+        -
+      * - ADS_VERSION
+        - oracle-ads==2.9.0rc0
+        - `https://github.com/oracle/accelerated-data-science/releases/tag/v2.9.0rc0`
+      * - Terraform Stack
+        - `link <https://objectstorage.us-ashburn-1.oraclecloud.com/p/vZogtXWwHqbkGLeqyKiqBmVxdbR4MK4nyOBqDsJNVE4sHGUY5KFi4T3mOFGA3FOy/n/idogsu2ylimg/b/oci-feature-store/o/beta/terraform/feature-store-terraform.zip>`__
+        -
+
+
+Release notes: September 22, 2023
+
+* [FEATURE] Addition of ``featurestore_dataset``  as optional parameter in GenericModel ``save`` function.
+* [FEATURE] Addition of ``transformation_kwargs`` in ``Transformation`` entity.
+* [FEATURE] Addition of partition keys in ``FeatureGroup`` and ``Dataset``
+* [FEATURE] as_of interface for time travel
+* [FEATURE] Manual association of feature groups in ``Dataset`` construct and support for complex queries
+* [FEATURE] Simplify the ads init experience without users need to specify the feature store endpoint
+* [FEATURE] Visualisation of feature statistics with ``to_viz`` in ``Statistics`` entity
+* [FIX] Validation of model ids when associated with ``Dataset``
+* [UI] Stats visualisation of feature group and dataset.
+* [UI] Transformation listing in the transformation tab
+* [UI] Global search for feature store entities.
+* [UI] Addition of onboarding page for feature store.
+* [UI] Redirection of entities within the lineage tab of feature group and dataset.
+
+1.0.2
+-----
+.. note::
+
+    .. list-table::
+      :header-rows: 1
+
+      * - Package Name
+        - Latest Version
+        - Notes
+      * - Conda pack
+        - `fspyspark32_p38_cpu_v1`
+        -
+      * - SERVICE_VERSION
+        - 0.1.225.master
+        -
+      * - Terraform Stack
+        - `link <https://objectstorage.us-ashburn-1.oraclecloud.com/p/vZogtXWwHqbkGLeqyKiqBmVxdbR4MK4nyOBqDsJNVE4sHGUY5KFi4T3mOFGA3FOy/n/idogsu2ylimg/b/oci-feature-store/o/beta/terraform/feature-store-terraform.zip>`__
+        - Par link expires Jan 5, 2026
+
+Release notes: July 18, 2023
+
+* [FEATURE] Supporting for deployment in ``us-ashburn`` and ``uk-london`` region.
+* [FEATURE] For ``ValidationOutput`` instance, addition of ``to_summary()`` method  for validation summary details.
+* [FEATURE] For ``ValidationOutput`` instance, addition of ``to_pandas()`` method  for validation detailed report.
+* [FIX] Fixed unit test integration to support the merging of ADS into the main branch.
+* [DOCS] For ``ValidationOutput`` instance, addition of ``to_summary()`` method  for validation summary details.
+* [DOCS] For ``ValidationOutput`` instance, addition of ``to_pandas()`` method  for validation detailed report.
+
+1.0.1
+-----
+
+.. note::
+
+    .. list-table::
+      :header-rows: 1
+
+      * - Package Name
+        - Latest Version
+        - Notes
+      * - Conda pack
+        - `fspyspark32_p38_cpu_v1`
+        -
+      * - SERVICE_VERSION
+        - 0.1.218.master
+        -
+      * - Terraform Stack
+        - `link <https://objectstorage.us-ashburn-1.oraclecloud.com/p/vZogtXWwHqbkGLeqyKiqBmVxdbR4MK4nyOBqDsJNVE4sHGUY5KFi4T3mOFGA3FOy/n/idogsu2ylimg/b/oci-feature-store/o/beta/terraform/feature-store-terraform.zip>`__
+        - Par link expires Jan 5, 2026
+
+
+Release notes: July 5, 2023
+
+* [FEATURE] Supporting Offline Feature Type COMPLEX
+* [FEATURE] Added k8 default version as v1.25.4
+* [FEATURE] Improvements in logging during materialisation of feature group and dataset and showcasing validation results during materialisation
+* [FIX] Fixed creation of singleton spark session without metastore id
+* [DOCS] Data Type update for Offline Feature Type COMPLEX
+* [DOCS] Updated terraform default version as 1.1.x
+
+1.0.0
+-----
+
+.. note::
+
+    .. list-table::
+      :header-rows: 1
+
+      * - Package Name
+        - Latest Version
+        - Notes
+      * - Conda pack
+        - `fspyspark32_p38_cpu_v1`
+        -
+      * - SERVICE_VERSION
+        - 0.1.209.master
+        -
+      * - Terraform Stack
+        - `link <https://objectstorage.us-ashburn-1.oraclecloud.com/p/vZogtXWwHqbkGLeqyKiqBmVxdbR4MK4nyOBqDsJNVE4sHGUY5KFi4T3mOFGA3FOy/n/idogsu2ylimg/b/oci-feature-store/o/beta/terraform/feature-store-terraform.zip>`__
+        - Par link expires Jan 5, 2026
+
+Release notes: June 15, 2023
+
+* [FEATURE] Included ``FeatureStore``, ``FeatureGroup``, ``Dataset``, ``Entity`` and ``Transformation`` concepts for feature store.
+* [DOCS] Included documentation for ``FeatureStore``, ``FeatureGroup``, ``Dataset``, ``Entity`` and ``Transformation`` constructs

ads/feature_store/docs/source/setup_feature_store.rst

@@ -0,0 +1,81 @@
+====================
+Setup Feature Store
+====================
+
+Feature store is being provided by OCI as a stack based offering in user's own tenancy via OCI marketplace. It can be configured by the user primarily in two ways:
+
+:doc:`Setup via Feature Store Marketplace Operator (Recommended) <./user_guides.setup.feature_store_operator>`
+__________________________________________________________
+
+The feature store marketplace operator can be used to setup the feature store api server in an existing OKE cluster while also utilising an existing MySQL database. It will also help setup authentication and authorization using OCI. For more details, see :doc:`Marketplace operator <./user_guides.setup.feature_store_operator>`. Optionally, we can also setup `Feature Store API Gateway stack <https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/feature_store/README.md>`_ for authentication and authorization.
+
+:doc:`Setup via Helm Charts <./user_guides.setup.helm_chart>`
+_______________________________
+
+We can manually export images to OCIR using Marketplace UI and then deploy the obtained Helm Chart. Optionally, we can also setup `Feature Store API Gateway stack <https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/feature_store/README.md>`_ for authentication and authorization.
+
+.. _Database configuration:
+
+Database configuration
+-----------------------
+
+Feature Store can be configured to use your existing MySQL database. It supports two types of authentication:
+
+1.  Basic (Not recommended): The password is stored as plaintext in the API server.
+2.  Vault (Recommended): The password is stored in an encrypted format inside `OCI Vault <https://docs.oracle.com/en-us/iaas/Content/KeyManagement/Concepts/keyoverview.htm>`_.
+
+Storing the password in Vault:
+
+1. (Optional) Create a new Vault.
+2. (Required) Create a secret of plain-text type containing the db password.
+3. (Required) Additional policies for the Feature Store API dynamic group to allow reading the secret from Vault:
+    - ``Allow dynamic-group <feature-store-dynamic-group> to use secret-family in tenancy``
+
+Here ``feature-store-dynamic-group`` is the dynamic group corresponding to the instances of the OKE nodepool where the server is deployed.
+
+.. seealso::
+    `Dynamic groups <https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm#:~:text=Dynamic%20groups%20allow%20you%20to,against%20Oracle%20Cloud%20Infrastructure%20services.>`_
+
+
+.. _Policies:
+Policies
+---------
+
+Policies required by the user deploying:
+
+.. code-block:: text
+
+    allow group <user-group> to manage clusters in compartment <compartment-name>
+    allow group <user-group> to use repos in compartment <compartment-name>
+    allow group <user-group> to manage marketplace-listings in compartment <compartment-name>
+    allow group <user-group> to read compartments in compartment <compartment-name>
+    allow group <user-group> to manage app-catalog-listing in compartment <compartment-name>
+    allow group <user-group> to read object-family in tenancy
+    allow group <user-group> to read marketplace-workrequests in compartment FeatureStoreDeployment
+
+The policies required by the Feature Store API server are:
+
+.. code-block:: text
+
+    allow dynamic-group <feature-store-dynamic-group> to read compartments in tenancy
+
+    allow dynamic-group <feature-store-dynamic-group> to manage data-catalog-family in tenancy
+
+    allow dynamic-group <feature-store-dynamic-group> to inspect data-science-models in tenancy
+
+
+Here ``feature-store-dynamic-group`` is the dynamic group corresponding to the instances of the OKE nodepool where the server is deployed. `Dynamic groups <https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm#:~:text=Dynamic%20groups%20allow%20you%20to,against%20Oracle%20Cloud%20Infrastructure%20services.>`_
+
+.. _Known Issues:
+
+Known Issues
+-------------
+
+1. Deployment doesn't work in Virtual Nodepool as the Feature Store API server relies on Instance Principal authentication.
+
+
+.. toctree::
+  :maxdepth: 1
+
+  ./user_guides.setup.feature_store_operator
+  ./user_guides.setup.helm_chart

ads/feature_store/docs/source/statistics.rst

@@ -0,0 +1,58 @@
+.. _Statistics:
+
+Statistics
+*************
+
+Feature Store provides functionality to compute statistics for feature groups and datasets, and then persists them including the metadata. These statistics can help you
+derive insights about the data quality. These statistical metrics are computed during materialisation time and persisting with other metadata.
+
+.. note::
+
+  Feature Store uses MLM Insights, which is a Python API that helps evaluate and monitor data for entire ML observability lifecycle. It performs data summarization, which reduces a dataset into a set of descriptive statistics.
+
+The statistical metrics that are computed by feature store depend on the feature type.
+
++------------------------+-----------------------+
+| Numerical Metrics      | Categorical Metrics   |
++========================+=======================+
+| Skewness               | Count                 |
++------------------------+-----------------------+
+| StandardDeviation      | TopKFrequentElements  |
++------------------------+-----------------------+
+| Min                    | TypeMetric            |
++------------------------+-----------------------+
+| IsConstantFeature      | DuplicateCount        |
++------------------------+-----------------------+
+| IQR                    | Mode                  |
++------------------------+-----------------------+
+| Range                  | DistinctCount         |
++------------------------+-----------------------+
+| ProbabilityDistribution|                       |
++------------------------+-----------------------+
+| Variance               |                       |
++------------------------+-----------------------+
+| FrequencyDistribution  |                       |
++------------------------+-----------------------+
+| Count                  |                       |
++------------------------+-----------------------+
+| Max                    |                       |
++------------------------+-----------------------+
+| DistinctCount          |                       |
++------------------------+-----------------------+
+| Sum                    |                       |
++------------------------+-----------------------+
+| IsQuasiConstantFeature |                       |
++------------------------+-----------------------+
+| Quartiles              |                       |
++------------------------+-----------------------+
+| Mean                   |                       |
++------------------------+-----------------------+
+| Kurtosis               |                       |
++------------------------+-----------------------+
+
+Drift Monitoring
+================
+
+Models can fail silently. Over and over we see the root cause of model issues in production can be traced back to the data itself, not the model. By applying data monitoring to the feature store, practitioners can automatically catch data issues like missing values, change in data format or unexpected values (change in data cardinality), and data drift upstream before the models are impacted
+
+.. image:: figures/drift_monitoring.png