oracle-ads 2.11.14__py3-none-any.whl → 2.11.15__py3-none-any.whl

ads/opctl/operator/lowcode/recommender/README.md

@@ -0,0 +1,206 @@
+# Recommender Operator
+
+Recommender Systems are designed to suggest relevant items, products, or content to users based on their preferences and behaviors. These systems are widely used in various industries such as e-commerce, entertainment, and social media to enhance user experience by providing personalized recommendations. They help in increasing user engagement, satisfaction, and sales by predicting what users might like or need based on their past interactions and the preferences of similar users.
+
+
+Below are the steps to configure and run the Recommender Operator on different resources.
+
+## 1. Prerequisites
+
+Follow the [CLI Configuration](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/cli/opctl/configure.html) steps from the ADS documentation. This step is mandatory as it sets up default values for different options while running the Recommender Operator on OCI Data Science jobs or OCI Data Flow applications. If you have previously done this and used a flexible shape, make sure to adjust `ml_job_config.ini` with shape config details and `docker_registry` information.
+
+- ocpus = 1
+- memory_in_gbs = 16
+- docker_registry = `<iad.ocir.io/namespace/>`
+
+## 2. Generating configs
+
+To generate starter configs, run the command below. This will create a list of YAML configs and place them in the `output` folder.
+
+```bash
+ads operator init -t recommender --overwrite --output ~/recommender/
+```
+
+The most important files expected to be generated are:
+
+- `recommender.yaml`: Contains recommender related configuration.
+- `backend_operator_local_python_config.yaml`: This includes a local backend configuration for running recommender in a local environment. The environment should be set up manually before running the operator.
+- `backend_operator_local_container_config.yaml`: This includes a local backend configuration for running recommender within a local container. The container should be built before running the operator. Please refer to the instructions below for details on how to accomplish this.
+- `backend_job_container_config.yaml`: Contains Data Science job-related config to run recommender in a Data Science job within a container (BYOC) runtime. The container should be built and published before running the operator. Please refer to the instructions below for details on how to accomplish this.
+- `backend_job_python_config.yaml`: Contains Data Science job-related config to run recommender in a Data Science job within a conda runtime. The conda should be built and published before running the operator.
+
+All generated configurations should be ready to use without the need for any additional adjustments. However, they are provided as starter kit configurations that can be customized as needed.
+
+## 3. Running recommender on the local conda environment
+
+To run recommender locally, create and activate a new conda environment (`ads-recommender`). Install all the required libraries listed in the `environment.yaml` file.
+
+```yaml
+- report-creator
+- cerberus
+- "git+https://github.com/oracle/accelerated-data-science.git@feature/recommender#egg=oracle-ads"
+```
+
+Please review the previously generated `recommender.yaml` file using the `init` command, and make any necessary adjustments to the input and output file locations. By default, it assumes that the files should be located in the same folder from which the `init` command was executed.
+
+Use the command below to verify the recommender config.
+
+```bash
+ads operator verify -f ~/recommender/recommender.yaml
+```
+
+Use the following command to run the recommender within the `ads-recommender` conda environment.
+
+```bash
+ads operator run -f ~/recommender/recommender.yaml -b local
+```
+
+The operator will run in your local environment without requiring any additional modifications.
+
+## 4. Running recommender on the local container
+
+To run the recommender operator within a local container, follow these steps:
+
+Use the command below to build the recommender container.
+
+```bash
+ads operator build-image -t recommender
+```
+
+This will create a new `recommender:v1` image, with `/etc/operator` as the designated working directory within the container.
+
+
+Check the `backend_operator_local_container_config.yaml` config file. By default, it should have a `volume` section with the `.oci` configs folder mounted.
+
+```yaml
+volume:
+  - "/Users/<user>/.oci:/root/.oci"
+```
+
+Mounting the OCI configs folder is only required if an OCI Object Storage bucket will be used to store the input recommender data or output recommender result. The input/output folders can also be mounted to the container.
+
+```yaml
+volume:
+  - /Users/<user>/.oci:/root/.oci
+  - /Users/<user>/recommender/data:/etc/operator/data
+  - /Users/<user>/recommender/result:/etc/operator/result
+```
+
+The full config can look like:
+```yaml
+kind: operator.local
+spec:
+  image: recommender:v1
+  volume:
+  - /Users/<user>/.oci:/root/.oci
+  - /Users/<user>/recommender/data:/etc/operator/data
+  - /Users/<user>/recommender/result:/etc/operator/result
+type: container
+version: v1
+```
+
+Run the recommender within a container using the command below:
+
+```bash
+ads operator run -f ~/recommender/recommender.yaml --backend-config ~/recommender/backend_operator_local_container_config.yaml
+```
+
+## 5. Running recommender in the Data Science job within container runtime
+
+To execute the recommender operator within a Data Science job using container runtime, please follow the steps outlined below:
+
+You can use the following command to build the recommender container. This step can be skipped if you have already done this for running the operator within a local container.
+
+```bash
+ads operator build-image -t recommender
+```
+
+This will create a new `recommender:v1` image, with `/etc/operator` as the designated working directory within the container.
+
+Publish the `recommender:v1` container to the [Oracle Container Registry](https://docs.public.oneportal.content.oci.oraclecloud.com/en-us/iaas/Content/Registry/home.htm). To become familiar with OCI, read the documentation links posted below.
+
+- [Access Container Registry](https://docs.public.oneportal.content.oci.oraclecloud.com/en-us/iaas/Content/Registry/Concepts/registryoverview.htm#access)
+- [Create repositories](https://docs.public.oneportal.content.oci.oraclecloud.com/en-us/iaas/Content/Registry/Tasks/registrycreatingarepository.htm#top)
+- [Push images](https://docs.public.oneportal.content.oci.oraclecloud.com/en-us/iaas/Content/Registry/Tasks/registrypushingimagesusingthedockercli.htm#Pushing_Images_Using_the_Docker_CLI)
+
+To publish `recommender:v1` to OCR, use the command posted below:
+
+```bash
+ads operator publish-image recommender:v1 --registry <iad.ocir.io/tenancy/>
+```
+
+After the container is published to OCR, it can be used within Data Science jobs service. Check the `backend_job_container_config.yaml` config file. It should contain pre-populated infrastructure and runtime sections. The runtime section should contain an image property, something like `image: iad.ocir.io/<tenancy>/recommender:v1`. More details about supported options can be found in the ADS Jobs documentation - [Run a Container](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/jobs/run_container.html).
+
+Adjust the `recommender.yaml` config with proper input/output folders. When the recommender is run in the Data Science job, it will not have access to local folders. Therefore, input data and output folders should be placed in the Object Storage bucket. Open the `recommender.yaml` and adjust the following fields:
+
+```yaml
+input_data:
+  url: oci://bucket@namespace/recommender/input_data/data.csv
+output_directory:
+  url: oci://bucket@namespace/recommender/result/
+```
+
+Run the recommender on the Data Science jobs using the command posted below:
+
+```bash
+ads operator run -f ~/recommender/recommender.yaml --backend-config ~/recommender/backend_job_container_config.yaml
+```
+
+The logs can be monitored using the `ads opctl watch` command.
+
+```bash
+ads opctl watch <OCID>
+```
+
+## 6. Running recommender in the Data Science job within conda runtime
+
+To execute the recommender operator within a Data Science job using conda runtime, please follow the steps outlined below:
+
+You can use the following command to build the recommender conda environment.
+
+```bash
+ads operator build-conda -t recommender
+```
+
+This will create a new `recommender_v1` conda environment and place it in the folder specified within `ads opctl configure` command.
+
+Use the command below to Publish the `recommender_v1` conda environment to the Object Storage bucket.
+
+```bash
+ads operator publish-conda -t recommender
+```
+More details about configuring CLI can be found here - [Configuring CLI](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/cli/opctl/configure.html)
+
+
+After the conda environment is published to Object Storage, it can be used within Data Science jobs service. Check the `backend_job_python_config.yaml` config file. It should contain pre-populated infrastructure and runtime sections. The runtime section should contain a `conda` section.
+
+```yaml
+conda:
+  type: published
+  uri: oci://bucket@namespace/conda_environments/cpu/recommender/1/recommender_v1
+```
+
+More details about supported options can be found in the ADS Jobs documentation - [Run a Python Workload](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/jobs/run_python.html).
+
+Adjust the `recommender.yaml` config with proper input/output folders. When the recommender is run in the Data Science job, it will not have access to local folders. Therefore, input data and output folders should be placed in the Object Storage bucket. Open the `recommender.yaml` and adjust the following fields:
+
+```yaml
+input_data:
+  url: oci://bucket@namespace/recommender/input_data/data.csv
+output_directory:
+  url: oci://bucket@namespace/recommender/result/
+test_data:
+  url: oci://bucket@namespace/recommender/input_data/test.csv
+```
+
+Run the recommender on the Data Science jobs using the command posted below:
+
+```bash
+ads operator run -f ~/recommender/recommender.yaml --backend-config ~/recommender/backend_job_python_config.yaml
+```
+
+The logs can be monitored using the `ads opctl watch` command.
+
+```bash
+ads opctl watch <OCID>
+```

ads/opctl/operator/lowcode/recommender/__init__.py

@@ -0,0 +1,5 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*--
+
+# Copyright (c) 2023 Oracle and/or its affiliates.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/

ads/opctl/operator/lowcode/recommender/__main__.py

@@ -0,0 +1,82 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*--
+
+# Copyright (c) 2024 Oracle and/or its affiliates.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
+
+import json
+import os
+import sys
+from typing import Dict, List
+
+import yaml
+
+from ads.opctl import logger
+from ads.opctl.operator.common.const import ENV_OPERATOR_ARGS
+from ads.opctl.operator.common.utils import _parse_input_args
+
+from .model.recommender_dataset import RecommenderDatasets
+from .operator_config import RecommenderOperatorConfig
+from .model.factory import RecommenderOperatorModelFactory
+
+def operate(operator_config: RecommenderOperatorConfig) -> None:
+    """Runs the recommender operator."""
+
+    datasets = RecommenderDatasets(operator_config)
+    RecommenderOperatorModelFactory.get_model(
+        operator_config, datasets
+    ).generate_report()
+
+
+def verify(spec: Dict, **kwargs: Dict) -> bool:
+    """Verifies the recommender detection operator config."""
+    operator = RecommenderOperatorConfig.from_dict(spec)
+    msg_header = (
+        f"{'*' * 50} The operator config has been successfully verified {'*' * 50}"
+    )
+    print(msg_header)
+    print(operator.to_yaml())
+    print("*" * len(msg_header))
+
+
+def main(raw_args: List[str]):
+    """The entry point of the recommender the operator."""
+    args, _ = _parse_input_args(raw_args)
+    if not args.file and not args.spec and not os.environ.get(ENV_OPERATOR_ARGS):
+        logger.info(
+            "Please specify -f[--file] or -s[--spec] or "
+            f"pass operator's arguments via {ENV_OPERATOR_ARGS} environment variable."
+        )
+        return
+
+    logger.info("-" * 100)
+    logger.info(
+        f"{'Running' if not args.verify else 'Verifying'} the recommender detection operator."
+    )
+
+    yaml_string = ""
+    if args.spec or os.environ.get(ENV_OPERATOR_ARGS):
+        operator_spec_str = args.spec or os.environ.get(ENV_OPERATOR_ARGS)
+        try:
+            yaml_string = yaml.safe_dump(json.loads(operator_spec_str))
+        except json.JSONDecodeError:
+            yaml_string = yaml.safe_dump(yaml.safe_load(operator_spec_str))
+        except:
+            yaml_string = operator_spec_str
+
+    operator_config = RecommenderOperatorConfig.from_yaml(
+        uri=args.file,
+        yaml_string=yaml_string,
+    )
+
+    logger.info(operator_config.to_yaml())
+
+    # run operator
+    if args.verify:
+        verify(operator_config)
+    else:
+        operate(operator_config)
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])

ads/opctl/operator/lowcode/recommender/cmd.py

@@ -0,0 +1,33 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*--
+
+# Copyright (c) 2023 Oracle and/or its affiliates.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
+
+from typing import Dict
+
+from ads.opctl.operator.common.operator_yaml_generator import YamlGenerator
+from ads.opctl.operator.common.utils import _load_yaml_from_uri
+
+
+def init(**kwargs: Dict) -> str:
+    """
+    Generates operator config by the schema.
+
+    Properties
+    ----------
+    kwargs: (Dict, optional).
+        Additional key value arguments.
+
+        - type: str
+            The type of the operator.
+
+    Returns
+    -------
+    str
+        The YAML specification generated based on the schema.
+    """
+
+    return YamlGenerator(
+        schema=_load_yaml_from_uri(__file__.replace("cmd.py", "schema.yaml"))
+    ).generate_example_dict(values={"type": kwargs.get("type")})

ads/opctl/operator/lowcode/recommender/constant.py

@@ -0,0 +1,25 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*--
+
+# Copyright (c) 2023 Oracle and/or its affiliates.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
+
+from ads.common.extended_enum import ExtendedEnumMeta
+
+DEFAULT_SHOW_ROWS = 25
+DEFAULT_REPORT_FILENAME = "report.html"
+
+class OutputColumns(str, metaclass=ExtendedEnumMeta):
+    """output columns for recommender operator"""
+    USER_COL = "user"
+    ITEM_COL = "item"
+    SCORE = "score"
+
+class SupportedMetrics(str, metaclass=ExtendedEnumMeta):
+    """Supported recommender metrics."""
+    RMSE = "RMSE"
+    MAE = "MAE"
+
+class SupportedModels(str, metaclass=ExtendedEnumMeta):
+    """Supported recommender models."""
+    SVD = "svd"

ads/opctl/operator/lowcode/recommender/environment.yaml

@@ -0,0 +1,11 @@
+name: recommender
+channels:
+  - conda-forge
+dependencies:
+  - python=3.9
+  - pip
+  - pip:
+      - report-creator
+      - oracle_ads[opctl]
+      - plotly
+      - scikit-surprise

ads/opctl/operator/lowcode/recommender/model/base_model.py

@@ -0,0 +1,198 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*--
+
+# Copyright (c) 2023, 2024 Oracle and/or its affiliates.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
+
+import os
+import tempfile
+import time
+from abc import ABC, abstractmethod
+from typing import Tuple, Dict
+
+import fsspec
+import pandas as pd
+import report_creator as rc
+
+from ads.common.object_storage_details import ObjectStorageDetails
+from ads.opctl import logger
+from ads.opctl.operator.lowcode.common.utils import default_signer
+from ads.opctl.operator.lowcode.common.utils import (
+    human_time_friendly,
+    enable_print,
+    disable_print,
+    write_data,
+)
+from .factory import SupportedModels
+from .recommender_dataset import RecommenderDatasets
+from ..operator_config import RecommenderOperatorConfig
+from plotly import graph_objects as go
+import matplotlib.pyplot as plt
+
+
+class RecommenderOperatorBaseModel(ABC):
+    """The base class for the recommender detection operator models."""
+
+    def __init__(self, config: RecommenderOperatorConfig, datasets: RecommenderDatasets):
+        self.config = config
+        self.spec = self.config.spec
+        self.datasets = datasets
+
+    def generate_report(self):
+        item_col = self.spec.item_column
+        user_col = self.spec.user_column
+        interaction_col = self.spec.interaction_column
+        start_time = time.time()
+        result_df, metrics = self._build_model()
+        elapsed_time = time.time() - start_time
+        logger.info("Building the models completed in %s seconds", elapsed_time)
+
+        if self.spec.generate_report:
+            # build the report
+            (
+                model_description,
+                other_sections,
+            ) = self._generate_report()
+
+            header_section = rc.Block(
+                rc.Heading("Recommender Report", level=1),
+                rc.Text(
+                    f"The recommendations was generated using {SupportedModels.SVD.upper()}. {model_description}"
+                ),
+                rc.Group(
+                    rc.Metric(
+                        heading="Recommendations was generated in ",
+                        value=human_time_friendly(elapsed_time),
+                    ),
+                    rc.Metric(
+                        heading="Num users",
+                        value=len(self.datasets.users),
+                    ),
+                    rc.Metric(
+                        heading="Num items",
+                        value=len(self.datasets.items),
+                    )
+                ),
+            )
+
+        summary = rc.Block(
+            header_section,
+        )
+        # user and item distributions in interactions
+        user_title = rc.Heading("User Statistics", level=2)
+        user_rating_counts = self.datasets.interactions[user_col].value_counts()
+        fig_user = go.Figure(data=[go.Histogram(x=user_rating_counts, nbinsx=100)])
+        fig_user.update_layout(
+            title=f'Distribution of the number of interactions by {user_col}',
+            xaxis_title=f'Number of {interaction_col}',
+            yaxis_title=f'Number of {user_col}',
+            bargap=0.2
+        )
+        item_title = rc.Heading("Item Statistics", level=2)
+        item_rating_counts = self.datasets.interactions[item_col].value_counts()
+        fig_item = go.Figure(data=[go.Histogram(x=item_rating_counts, nbinsx=100)])
+        fig_item.update_layout(
+            title=f'Distribution of the number of interactions by {item_col}',
+            xaxis_title=f'Number of {interaction_col}',
+            yaxis_title=f'Number of {item_col}',
+            bargap=0.2
+        )
+        result_heatmap_title = rc.Heading("Sample Recommendations", level=2)
+        sample_items = result_df[item_col].head(100).index
+        filtered_df = result_df[result_df[item_col].isin(sample_items)]
+        data = filtered_df.pivot(index=user_col, columns=item_col, values=interaction_col)
+        fig = go.Figure(data=go.Heatmap(
+            z=data.values,
+            x=data.columns,
+            y=data.index,
+            colorscale='Viridis'
+        ))
+        fig.update_layout(
+            title='Recommendation heatmap of User-Item Interactions (sample)',
+            width=1500,
+            height=800,
+            xaxis_title=item_col,
+            yaxis_title=user_col,
+            coloraxis_colorbar=dict(title=interaction_col)
+        )
+        plots = [user_title, rc.Widget(fig_user),
+                 item_title, rc.Widget(fig_item),
+                 result_heatmap_title, rc.Widget(fig)]
+
+        test_metrics_sections = [rc.DataTable(pd.DataFrame(metrics, index=[0]))]
+        yaml_appendix_title = rc.Heading("Reference: YAML File", level=2)
+        yaml_appendix = rc.Yaml(self.config.to_dict())
+        report_sections = (
+                [summary]
+                + plots
+                + test_metrics_sections
+                + other_sections
+                + [yaml_appendix_title, yaml_appendix]
+        )
+
+        # save the report and result CSV
+        self._save_report(
+            report_sections=report_sections,
+            result_df=result_df
+        )
+
+    def _evaluation_metrics(self):
+        pass
+
+    def _test_data_evaluate_metrics(self):
+        pass
+
+    def _save_report(self, report_sections: Tuple, result_df: pd.DataFrame):
+        """Saves resulting reports to the given folder."""
+
+        unique_output_dir = self.spec.output_directory.url
+
+        if ObjectStorageDetails.is_oci_path(unique_output_dir):
+            storage_options = default_signer()
+        else:
+            storage_options = dict()
+
+            # report-creator html report
+            if self.spec.generate_report:
+                with tempfile.TemporaryDirectory() as temp_dir:
+                    report_local_path = os.path.join(temp_dir, "___report.html")
+                    disable_print()
+                    with rc.ReportCreator("My Report") as report:
+                        report.save(rc.Block(*report_sections), report_local_path)
+                    enable_print()
+
+                    report_path = os.path.join(unique_output_dir, self.spec.report_filename)
+                    with open(report_local_path) as f1:
+                        with fsspec.open(
+                                report_path,
+                                "w",
+                                **storage_options,
+                        ) as f2:
+                            f2.write(f1.read())
+
+        # recommender csv report
+        write_data(
+            data=result_df,
+            filename=os.path.join(unique_output_dir, self.spec.recommendations_filename),
+            format="csv",
+            storage_options=storage_options,
+        )
+
+        logger.info(
+            f"The outputs have been successfully "
+            f"generated and placed into the directory: {unique_output_dir}."
+        )
+
+    @abstractmethod
+    def _generate_report(self):
+        """
+        Generates the report for the particular model.
+        The method that needs to be implemented on the particular model level.
+        """
+
+    @abstractmethod
+    def _build_model(self) -> [pd.DataFrame, Dict]:
+        """
+        Build the model.
+        The method that needs to be implemented on the particular model level.
+        """

ads/opctl/operator/lowcode/recommender/model/factory.py

@@ -0,0 +1,58 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*--
+
+# Copyright (c) 2023 Oracle and/or its affiliates.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
+
+from ..constant import SupportedModels
+from ..operator_config import RecommenderOperatorConfig
+from .base_model import RecommenderOperatorBaseModel
+from .recommender_dataset import RecommenderDatasets
+from .svd import SVDOperatorModel
+
+class UnSupportedModelError(Exception):
+    def __init__(self, model_type: str):
+        super().__init__(
+            f"Model: `{model_type}` "
+            f"is not supported. Supported models: {SupportedModels.values}"
+        )
+
+
+class RecommenderOperatorModelFactory:
+    """
+    The factory class helps to instantiate proper model operator based on the model type.
+    """
+
+    _MAP = {
+        SupportedModels.SVD: SVDOperatorModel
+    }
+
+    @classmethod
+    def get_model(
+        cls, operator_config: RecommenderOperatorConfig, datasets: RecommenderDatasets
+    ) -> RecommenderOperatorBaseModel:
+        """
+        Gets the operator model based on the model type.
+
+        Parameters
+        ----------
+        operator_config: RecommenderOperatorConfig
+            The recommender detection operator config.
+
+        datasets: RecommenderDatasets
+            Datasets for finding recommender
+
+        Returns
+        -------
+        RecommenderOperatorBaseModel
+            The recommender detection operator model.
+
+        Raises
+        ------
+        UnSupportedModelError
+            In case of not supported model.
+        """
+        model_type = SupportedModels.SVD
+        if model_type not in cls._MAP:
+            raise UnSupportedModelError(model_type)
+        return cls._MAP[model_type](config=operator_config, datasets=datasets)

ads/opctl/operator/lowcode/recommender/model/recommender_dataset.py

@@ -0,0 +1,25 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*--
+
+# Copyright (c) 2023 Oracle and/or its affiliates.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
+
+import pandas as pd
+
+from ads.opctl.operator.lowcode.common.utils import load_data
+from ..operator_config import RecommenderOperatorConfig
+
+
+class RecommenderDatasets:
+    def __init__(self, config: RecommenderOperatorConfig):
+        """Instantiates the DataIO instance.
+
+        Properties
+        ----------
+        spec: RecommenderOperatorSpec
+            The recommender operator spec.
+        """
+        spec = config.spec
+        self.interactions: pd.DataFrame = load_data(getattr(spec, "interactions_data"))
+        self.users: pd.DataFrame = load_data(getattr(spec, "user_data"))
+        self.items: pd.DataFrame = load_data(getattr(spec, "item_data"))