data-formulator 0.1.2__tar.gz

data_formulator-0.1.2/LICENSE

@@ -0,0 +1,21 @@
+    MIT License
+
+    Copyright (c) Microsoft Corporation.
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE

data_formulator-0.1.2/MANIFEST.in

@@ -0,0 +1,2 @@
+include py-src/data_formulator/dist/*
+include py-src/data_formulator/dist/assets/*

data_formulator-0.1.2/PKG-INFO

@@ -0,0 +1,200 @@
+Metadata-Version: 2.1
+Name: data_formulator
+Version: 0.1.2
+Summary: Data Formulator is research protoype data visualization tool powered by AI.
+Author-email: Chenglong Wang <chenglong.wang@microsoft.com>, Dan Marshall <danmar@microsoft.com>
+License:     MIT License
+        
+            Copyright (c) Microsoft Corporation.
+        
+            Permission is hereby granted, free of charge, to any person obtaining a copy
+            of this software and associated documentation files (the "Software"), to deal
+            in the Software without restriction, including without limitation the rights
+            to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+            copies of the Software, and to permit persons to whom the Software is
+            furnished to do so, subject to the following conditions:
+        
+            The above copyright notice and this permission notice shall be included in all
+            copies or substantial portions of the Software.
+        
+            THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+            IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+            FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+            AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+            LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+            OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+            SOFTWARE
+        
+Project-URL: Homepage, https://github.com/microsoft/data-formulator
+Project-URL: Repository, https://github.com/microsoft/data-formulator.git
+Project-URL: Bug Tracker, https://github.com/microsoft/data-formulator/issues
+Keywords: data visualization,LLM,AI
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: autopep8
+Requires-Dist: jupyter
+Requires-Dist: pandas
+Requires-Dist: docker
+Requires-Dist: namedlist
+Requires-Dist: matplotlib
+Requires-Dist: flask
+Requires-Dist: flask-cors
+Requires-Dist: openai
+Requires-Dist: azure-identity
+Requires-Dist: azure-keyvault-secrets
+Requires-Dist: python-dotenv
+Requires-Dist: vega_datasets
+
+<h1>
+    <img src="./public/favicon.ico" alt="Data Formulator icon" width="28"> <b>Data Formulator: Create Rich Visualizations with AI</b>
+</h1>
+
+<div>
+    
+[![arxiv](https://img.shields.io/badge/Paper-arXiv:2408.16119-b31b1b.svg)](https://arxiv.org/abs/2408.16119)&ensp;
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)&ensp;
+[![YouTube](https://img.shields.io/badge/YouTube-white?logo=youtube&logoColor=%23FF0000)](https://youtu.be/3ndlwt0Wi3c)&ensp;
+[![build](https://github.com/microsoft/data-formulator/actions/workflows/python-build.yml/badge.svg)](https://github.com/microsoft/data-formulator/actions/workflows/python-build.yml)
+
+</div>
+
+Transform data and create rich visualizations iteratively with AI 🪄. Try Data Formulator now in GitHub Codespaces!
+
+[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/microsoft/data-formulator?quickstart=1)
+
+<kbd>
+  <a target="_blank" rel="noopener noreferrer" href="https://codespaces.new/microsoft/data-formulator?quickstart=1" title="open Data Formulator in GitHub Codespaces"><img src="public/data-formulator-screenshot.png"></a>
+</kbd>
+
+
+## News 🔥🔥🔥
+
+- [10-11-2024] Data Formulator python package released! 
+  - You can now install Data Formulator using Python and run it locally, easily. [[check it out]](#get-started).
+  - Our Codespace configuration is also updated for fast start up ⚡️. [[try it now!]](https://codespaces.new/microsoft/data-formulator?quickstart=1)
+  - New exprimental feature: load an image or a messy text, and ask AI parsing and cleaning it for you(!). [[demo]](https://github.com/microsoft/data-formulator/pull/31#issuecomment-2403652717)
+  
+- [10-01-2024] Initial release of Data Formulator, check out our [[blog]](https://www.microsoft.com/en-us/research/blog/data-formulator-exploring-how-ai-can-help-analysts-create-rich-data-visualizations/) and [[video]](https://youtu.be/3ndlwt0Wi3c)!
+
+
+
+## Overview
+
+**Data Formulator** is an application from Microsoft Research that uses large language models to transform data, expediting the practice of data visualization.
+
+Data Formulator is an AI-powered tool for analysts to iteratively create rich visualizations. Unlike most chat-based AI tools where users need to describe everything in natural language, Data Formulator combines *user interface interactions (UI)* and *natural language (NL) inputs* for easier interaction. This blended approach makes it easier for users to describe their chart designs while delegating data transformation to AI. 
+
+## Get Started
+
+Play with Data Formulator with one of the following options:
+
+- **Option 1: Install via Python PIP**
+  
+  Use Python PIP for an easy setup experience, running locally (recommend: install it in a virtual environment).
+  
+  ```bash
+  # install data_formulator
+  pip install data_formulator
+
+  # start data_formulator
+  data_formulator 
+  
+  # alternatively, you can run data formualtor with this command
+  python -m data_formulator
+  ```
+
+  Data Formulator will be automatically opened in the browser at [http://localhost:5000](http://localhost:5000).
+
+- **Option 2: Codespaces (5 minutes)**
+  
+  You can also run Data Formualtor in codespace, we have everything pre-configured. For more details, see [CODESPACES.md](CODESPACES.md).
+  
+  [![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/microsoft/data-formulator?quickstart=1)
+
+- **Option 3: Working in the developer mode**
+  
+  You can build Data Formulator locally if you prefer full control over your development environment and the ability to customize the setup to your specific needs. For detailed instructions, refer to [DEVELOPMENT.md](DEVELOPMENT.md).
+
+
+## Using Data Formulator
+
+Once you’ve completed the setup using either option, follow these steps to start using Data Formulator:
+
+### The basics of data visualization
+* Provide OpenAI keys and select a model (GPT-4o suggested) and choose a dataset.
+* Choose a chart type, and then drag-and-drop data fields to chart properties (x, y, color, ...) to specify visual encodings.
+
+https://github.com/user-attachments/assets/0fbea012-1d2d-46c3-a923-b1fc5eb5e5b8
+
+
+### Create visualization beyond the initial dataset (powered by 🤖)
+* You can type names of **fields that do not exist in current data** in the encoding shelf:
+    - this tells Data Formulator that you want to create visualizions that require computation or transformation from existing data,
+    - you can optionally provide a natural language prompt to explain your intent to clarify your intent (not necessary when field names are self-explanatory).
+* Click the **Formulate** button.
+    - Data Formulator will transform data and instantiate the visualization based on the encoding and prompt.
+* Inspect the data, chart and code.
+* To create a new chart based on existing ones, follow up in natural language:
+    - provide a follow up prompt (e.g., *``show only top 5!''*),
+    - you may also update visual encodings for the new chart.
+
+https://github.com/user-attachments/assets/160c69d2-f42d-435c-9ff3-b1229b5bddba
+
+https://github.com/user-attachments/assets/c93b3e84-8ca8-49ae-80ea-f91ceef34acb
+
+Repeat this process as needed to explore and understand your data. Your explorations are trackable in the **Data Threads** panel. 
+
+## Developers' Guide
+
+Follow the [developers' instructions](DEVELOPMENT.md) to build your new data analysis tools on top of Data Formulator.
+
+## Research Papers
+* [Data Formulator 2: Iteratively Creating Rich Visualizations with AI](https://arxiv.org/abs/2408.16119)
+
+```
+@article{wang2024dataformulator2iteratively,
+      title={Data Formulator 2: Iteratively Creating Rich Visualizations with AI}, 
+      author={Chenglong Wang and Bongshin Lee and Steven Drucker and Dan Marshall and Jianfeng Gao},
+      year={2024},
+      booktitle={ArXiv preprint arXiv:2408.16119},
+}
+```
+
+* [Data Formulator: AI-powered Concept-driven Visualization Authoring](https://arxiv.org/abs/2309.10094)
+
+```
+@article{wang2023data,
+  title={Data Formulator: AI-powered Concept-driven Visualization Authoring},
+  author={Wang, Chenglong and Thompson, John and Lee, Bongshin},
+  journal={IEEE Transactions on Visualization and Computer Graphics},
+  year={2023},
+  publisher={IEEE}
+}
+```
+
+
+## Contributing
+
+This project welcomes contributions and suggestions. Most contributions require you to
+agree to a Contributor License Agreement (CLA) declaring that you have the right to,
+and actually do, grant us the rights to use your contribution. For details, visit
+https://cla.microsoft.com.
+
+When you submit a pull request, a CLA-bot will automatically determine whether you need
+to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the
+instructions provided by the bot. You will only need to do this once across all repositories using our CLA.
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
+or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
+
+## Trademarks
+
+This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft 
+trademarks or logos is subject to and must follow 
+[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
+Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
+Any use of third-party trademarks or logos are subject to those third-party's policies.

data_formulator-0.1.2/README.md

@@ -0,0 +1,150 @@
+<h1>
+    <img src="./public/favicon.ico" alt="Data Formulator icon" width="28"> <b>Data Formulator: Create Rich Visualizations with AI</b>
+</h1>
+
+<div>
+    
+[![arxiv](https://img.shields.io/badge/Paper-arXiv:2408.16119-b31b1b.svg)](https://arxiv.org/abs/2408.16119)&ensp;
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)&ensp;
+[![YouTube](https://img.shields.io/badge/YouTube-white?logo=youtube&logoColor=%23FF0000)](https://youtu.be/3ndlwt0Wi3c)&ensp;
+[![build](https://github.com/microsoft/data-formulator/actions/workflows/python-build.yml/badge.svg)](https://github.com/microsoft/data-formulator/actions/workflows/python-build.yml)
+
+</div>
+
+Transform data and create rich visualizations iteratively with AI 🪄. Try Data Formulator now in GitHub Codespaces!
+
+[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/microsoft/data-formulator?quickstart=1)
+
+<kbd>
+  <a target="_blank" rel="noopener noreferrer" href="https://codespaces.new/microsoft/data-formulator?quickstart=1" title="open Data Formulator in GitHub Codespaces"><img src="public/data-formulator-screenshot.png"></a>
+</kbd>
+
+
+## News 🔥🔥🔥
+
+- [10-11-2024] Data Formulator python package released! 
+  - You can now install Data Formulator using Python and run it locally, easily. [[check it out]](#get-started).
+  - Our Codespace configuration is also updated for fast start up ⚡️. [[try it now!]](https://codespaces.new/microsoft/data-formulator?quickstart=1)
+  - New exprimental feature: load an image or a messy text, and ask AI parsing and cleaning it for you(!). [[demo]](https://github.com/microsoft/data-formulator/pull/31#issuecomment-2403652717)
+  
+- [10-01-2024] Initial release of Data Formulator, check out our [[blog]](https://www.microsoft.com/en-us/research/blog/data-formulator-exploring-how-ai-can-help-analysts-create-rich-data-visualizations/) and [[video]](https://youtu.be/3ndlwt0Wi3c)!
+
+
+
+## Overview
+
+**Data Formulator** is an application from Microsoft Research that uses large language models to transform data, expediting the practice of data visualization.
+
+Data Formulator is an AI-powered tool for analysts to iteratively create rich visualizations. Unlike most chat-based AI tools where users need to describe everything in natural language, Data Formulator combines *user interface interactions (UI)* and *natural language (NL) inputs* for easier interaction. This blended approach makes it easier for users to describe their chart designs while delegating data transformation to AI. 
+
+## Get Started
+
+Play with Data Formulator with one of the following options:
+
+- **Option 1: Install via Python PIP**
+  
+  Use Python PIP for an easy setup experience, running locally (recommend: install it in a virtual environment).
+  
+  ```bash
+  # install data_formulator
+  pip install data_formulator
+
+  # start data_formulator
+  data_formulator 
+  
+  # alternatively, you can run data formualtor with this command
+  python -m data_formulator
+  ```
+
+  Data Formulator will be automatically opened in the browser at [http://localhost:5000](http://localhost:5000).
+
+- **Option 2: Codespaces (5 minutes)**
+  
+  You can also run Data Formualtor in codespace, we have everything pre-configured. For more details, see [CODESPACES.md](CODESPACES.md).
+  
+  [![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/microsoft/data-formulator?quickstart=1)
+
+- **Option 3: Working in the developer mode**
+  
+  You can build Data Formulator locally if you prefer full control over your development environment and the ability to customize the setup to your specific needs. For detailed instructions, refer to [DEVELOPMENT.md](DEVELOPMENT.md).
+
+
+## Using Data Formulator
+
+Once you’ve completed the setup using either option, follow these steps to start using Data Formulator:
+
+### The basics of data visualization
+* Provide OpenAI keys and select a model (GPT-4o suggested) and choose a dataset.
+* Choose a chart type, and then drag-and-drop data fields to chart properties (x, y, color, ...) to specify visual encodings.
+
+https://github.com/user-attachments/assets/0fbea012-1d2d-46c3-a923-b1fc5eb5e5b8
+
+
+### Create visualization beyond the initial dataset (powered by 🤖)
+* You can type names of **fields that do not exist in current data** in the encoding shelf:
+    - this tells Data Formulator that you want to create visualizions that require computation or transformation from existing data,
+    - you can optionally provide a natural language prompt to explain your intent to clarify your intent (not necessary when field names are self-explanatory).
+* Click the **Formulate** button.
+    - Data Formulator will transform data and instantiate the visualization based on the encoding and prompt.
+* Inspect the data, chart and code.
+* To create a new chart based on existing ones, follow up in natural language:
+    - provide a follow up prompt (e.g., *``show only top 5!''*),
+    - you may also update visual encodings for the new chart.
+
+https://github.com/user-attachments/assets/160c69d2-f42d-435c-9ff3-b1229b5bddba
+
+https://github.com/user-attachments/assets/c93b3e84-8ca8-49ae-80ea-f91ceef34acb
+
+Repeat this process as needed to explore and understand your data. Your explorations are trackable in the **Data Threads** panel. 
+
+## Developers' Guide
+
+Follow the [developers' instructions](DEVELOPMENT.md) to build your new data analysis tools on top of Data Formulator.
+
+## Research Papers
+* [Data Formulator 2: Iteratively Creating Rich Visualizations with AI](https://arxiv.org/abs/2408.16119)
+
+```
+@article{wang2024dataformulator2iteratively,
+      title={Data Formulator 2: Iteratively Creating Rich Visualizations with AI}, 
+      author={Chenglong Wang and Bongshin Lee and Steven Drucker and Dan Marshall and Jianfeng Gao},
+      year={2024},
+      booktitle={ArXiv preprint arXiv:2408.16119},
+}
+```
+
+* [Data Formulator: AI-powered Concept-driven Visualization Authoring](https://arxiv.org/abs/2309.10094)
+
+```
+@article{wang2023data,
+  title={Data Formulator: AI-powered Concept-driven Visualization Authoring},
+  author={Wang, Chenglong and Thompson, John and Lee, Bongshin},
+  journal={IEEE Transactions on Visualization and Computer Graphics},
+  year={2023},
+  publisher={IEEE}
+}
+```
+
+
+## Contributing
+
+This project welcomes contributions and suggestions. Most contributions require you to
+agree to a Contributor License Agreement (CLA) declaring that you have the right to,
+and actually do, grant us the rights to use your contribution. For details, visit
+https://cla.microsoft.com.
+
+When you submit a pull request, a CLA-bot will automatically determine whether you need
+to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the
+instructions provided by the bot. You will only need to do this once across all repositories using our CLA.
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
+or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
+
+## Trademarks
+
+This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft 
+trademarks or logos is subject to and must follow 
+[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
+Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
+Any use of third-party trademarks or logos are subject to those third-party's policies.

data_formulator-0.1.2/py-src/data_formulator/__init__.py

@@ -0,0 +1,5 @@
+from .app import run_app
+
+__all__ = [
+    "run_app",
+]

data_formulator-0.1.2/py-src/data_formulator/__main__.py

@@ -0,0 +1,4 @@
+from .app import run_app
+
+if __name__ == "__main__":
+    run_app()

data_formulator-0.1.2/py-src/data_formulator/agents/__init__.py

@@ -0,0 +1,22 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+
+from data_formulator.agents.agent_concept_derive import ConceptDeriveAgent
+from data_formulator.agents.agent_py_concept_derive import PyConceptDeriveAgent
+from data_formulator.agents.agent_data_transformation import DataTransformationAgent
+from data_formulator.agents.agent_data_transform_v2 import DataTransformationAgentV2
+from data_formulator.agents.agent_data_load import DataLoadAgent
+from data_formulator.agents.agent_sort_data import SortDataAgent
+from data_formulator.agents.agent_data_clean import DataCleanAgent
+from data_formulator.agents.agent_data_rec import DataRecAgent
+
+__all__ = [
+    "ConceptDeriveAgent",
+    "PyConceptDeriveAgent",
+    "DataTransformationAgent",
+    "DataTransformationAgentV2",
+    "DataRecAgent",
+    "DataLoadAgent",
+    "SortDataAgent",
+    "DataCleanAgent"
+]

data_formulator-0.1.2/py-src/data_formulator/agents/agent_code_explanation.py

@@ -0,0 +1,92 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+
+import pandas as pd
+from data_formulator.agents.agent_utils import generate_data_summary, extract_code_from_gpt_response
+
+import logging
+
+logger = logging.getLogger(__name__)
+
+
+SYSTEM_PROMPT = '''You are a data scientist to help user explain code, 
+so that a non-code can clearly understand what the code is doing, you are provided with a summary of the input data, and the transformation code.
+You should generate a good itemized explanation of the code so that the reader can understand high-level steps of what the data transformation is doing.
+Be very concise, and stay at a high-level. The reader doesn't understand code and does not want to learn exactly what the code is doing. They just want to learn what have been done from a logical level.
+
+The focus is to explain how new fields are computed, don't generate explanation for low-level actions like "return", "load data" etc. 
+
+Format the transformation explanation in markdown format: highlight constants, data fields, and important verbs. Be sure to be concise.
+
+
+For example:
+
+[CONTEXT]
+
+Here are our datasets, here are their field summaries and samples:
+
+table_0 (us_covid_cases) fields:
+	Date -- type: object, values: 1/1/2021, 1/1/2022, 1/1/2023, ..., 9/8/2022, 9/9/2020, 9/9/2021, 9/9/2022
+	Cases -- type: int64, values: -23999, -14195, -6940, ..., 1018935, 1032159, 1178403, 1433977
+
+table_0 (us_covid_cases) sample:
+```
+|Date|Cases
+0|1/21/2020|1
+1|1/22/2020|0
+2|1/23/2020|0
+3|1/24/2020|1
+4|1/25/2020|1
+......
+```
+
+[CODE]
+
+```python
+import pandas as pd
+
+def transform_data(df_0):
+    # Convert Date field to datetime format
+    df_0['Date'] = pd.to_datetime(df_0['Date'])
+
+    # Sort the dataframe by Date column
+    df_0 = df_0.sort_values('Date')
+
+    # Calculate 7-day rolling average of Cases column
+    df_0['7-day average cases'] = df_0['Cases'].rolling(window=7).mean()
+
+    return df_0
+```
+
+[EXPLANATION]
+
+1. **Sort** the data in chronological order based on the `Date` column.
+2. **Calculate** the `7-day rolling average` of `Cases`.
+'''
+
+class CodeExplanationAgent(object):
+
+    def __init__(self, client, model):
+        self.client = client
+        self.model = model
+
+    def run(self, input_tables, code):
+
+        data_summary = generate_data_summary(input_tables, include_data_samples=True)
+
+        user_query = f"[CONTEXT]\n\n{data_summary}\n\n[CODE]\n\here is the transformation code: {code}\n\n[EXPLANATION]\n"
+
+        logger.info(user_query)
+
+        messages = [{"role":"system", "content": SYSTEM_PROMPT},
+                    {"role":"user","content": user_query}]
+        
+        ###### the part that calls open_ai
+        response = self.client.chat.completions.create(
+            model=self.model, messages = messages, temperature=0.7, max_tokens=1200,
+            top_p=0.95, n=1, frequency_penalty=0, presence_penalty=0, stop=None)
+        
+        logger.info('\n=== explanation output ===>\n')
+        logger.info(response.choices[0].message.content)
+        
+        return response.choices[0].message.content