edsl 0.1.27.dev2__tar.gz → 0.1.29__tar.gz

{edsl-0.1.27.dev2 → edsl-0.1.29}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: edsl
-Version: 0.1.27.dev2
+Version: 0.1.29
 Summary: Create and analyze LLM-based surveys
 Home-page: https://www.expectedparrot.com/
 License: MIT
@@ -30,6 +30,7 @@ Requires-Dist: pandas (>=2.1.4,<3.0.0)
 Requires-Dist: pydot (>=2.0.0,<3.0.0)
 Requires-Dist: pygments (>=2.17.2,<3.0.0)
 Requires-Dist: pymupdf (>=1.24.4,<2.0.0)
+Requires-Dist: pyreadstat (>=1.2.7,<2.0.0)
 Requires-Dist: python-docx (>=1.1.0,<2.0.0)
 Requires-Dist: python-dotenv (>=1.0.0,<2.0.0)
 Requires-Dist: restrictedpython (>=7.1,<8.0)
@@ -45,28 +46,54 @@ Description-Content-Type: text/markdown
   <img src="https://github.com/expectedparrot/edsl/blob/main/static/logo.png?raw=true" alt="edsl.png" width="100"/>
 </p>
 
-The Expected Parrot Domain-Specific Language (EDSL) package lets you conduct computational social science and market research with AI. Use it to design surveys and experiments, simulate responses with large language models, and perform data labeling and other research tasks. EDSL comes with built-in methods for analyzing, visualizing and sharing your results. 
+The Expected Parrot Domain-Specific Language (EDSL) package lets you conduct computational social science and market research with AI. Use it to design surveys and experiments, simulate responses with large language models, and perform data labeling and other research tasks. Results are formatted as specified datasets and come with built-in methods for analyzing, visualizing, and sharing. 
 
 ## 🔗 Links
-- PyPI: https://pypi.org/project/edsl/
-- Documentation: https://docs.expectedparrot.com
-- Getting started: https://docs.expectedparrot.com/en/latest/starter_tutorial.html
-- Discord: https://discord.com/invite/mxAYkjfy9m
+- [PyPI](https://pypi.org/project/edsl/)
+- [Documentation](https://docs.expectedparrot.com)
+- [Getting started](https://docs.expectedparrot.com/en/latest/starter_tutorial.html) 
+- [Discord](https://discord.com/invite/mxAYkjfy9m)
+- [Twitter](https://x.com/ExpectedParrot)
+- [LinkedIn](https://www.linkedin.com/company/expectedparrot/)
+- [Blog](https://blog.expectedparrot.com)
 
+## 🌎 Hello, World!
+A quick example:
 
-## 💡 Contributions, Feature Requests & Bugs
-Interested in contributing? Want us to add a new feature? Found a nasty bug that you would like us to squash? Please send us an email at info@expectedparrot.com or message us at our Discord server.
+```python
+# Import a question type
+from edsl.questions import QuestionMultipleChoice
 
+# Construct a question using the question type template
+q = QuestionMultipleChoice(
+    question_name="example_question",
+    question_text="How do you feel today?",
+    question_options=["Bad", "OK", "Good"]
+)
 
-## 💻 Getting started
-EDSL is compatible with Python 3.9 - 3.12.
+# Run it with the default language model
+results = q.run()
+
+# Inspect the results in a dataset
+results.select("example_question").print()
 ```
-pip install edsl
+
+Output:
+```python
+┏━━━━━━━━━━━━━━━━━━━┓
+┃ answer            ┃
+┃ .example_question ┃
+┡━━━━━━━━━━━━━━━━━━━┩
+│ Good              │
+└───────────────────┘
 ```
 
-- Read the docs at http://docs.expectedparrot.com.
-- See https://docs.expectedparrot.com/en/latest/starter_tutorial.html for examples and tutorials.
+## 💻 Requirements
+* EDSL is compatible with Python 3.9 - 3.12.
+* API keys for large language models that you want to use, stored in a `.env` file.
+See instructions on [storing API keys](https://docs.expectedparrot.com/en/latest/api_keys.html).
 
-## 🔧 Dependencies
-API keys for LLMs that you want to use, stored in a `.env` file
+## 💡 Contributions, feature requests & bugs
+Interested in contributing? Want us to add a new feature? Found a bug for us to squash? 
+Please send us an email at [info@expectedparrot.com](mailto:info@expectedparrot.com) or message us at our [Discord channel](https://discord.com/invite/mxAYkjfy9m).
 

edsl-0.1.29/README.md

@@ -0,0 +1,55 @@
+# Expected Parrot Domain-Specific Language 
+<p align="center">
+  <img src="https://github.com/expectedparrot/edsl/blob/main/static/logo.png?raw=true" alt="edsl.png" width="100"/>
+</p>
+
+The Expected Parrot Domain-Specific Language (EDSL) package lets you conduct computational social science and market research with AI. Use it to design surveys and experiments, simulate responses with large language models, and perform data labeling and other research tasks. Results are formatted as specified datasets and come with built-in methods for analyzing, visualizing, and sharing. 
+
+## 🔗 Links
+- [PyPI](https://pypi.org/project/edsl/)
+- [Documentation](https://docs.expectedparrot.com)
+- [Getting started](https://docs.expectedparrot.com/en/latest/starter_tutorial.html) 
+- [Discord](https://discord.com/invite/mxAYkjfy9m)
+- [Twitter](https://x.com/ExpectedParrot)
+- [LinkedIn](https://www.linkedin.com/company/expectedparrot/)
+- [Blog](https://blog.expectedparrot.com)
+
+## 🌎 Hello, World!
+A quick example:
+
+```python
+# Import a question type
+from edsl.questions import QuestionMultipleChoice
+
+# Construct a question using the question type template
+q = QuestionMultipleChoice(
+    question_name="example_question",
+    question_text="How do you feel today?",
+    question_options=["Bad", "OK", "Good"]
+)
+
+# Run it with the default language model
+results = q.run()
+
+# Inspect the results in a dataset
+results.select("example_question").print()
+```
+
+Output:
+```python
+┏━━━━━━━━━━━━━━━━━━━┓
+┃ answer            ┃
+┃ .example_question ┃
+┡━━━━━━━━━━━━━━━━━━━┩
+│ Good              │
+└───────────────────┘
+```
+
+## 💻 Requirements
+* EDSL is compatible with Python 3.9 - 3.12.
+* API keys for large language models that you want to use, stored in a `.env` file.
+See instructions on [storing API keys](https://docs.expectedparrot.com/en/latest/api_keys.html).
+
+## 💡 Contributions, feature requests & bugs
+Interested in contributing? Want us to add a new feature? Found a bug for us to squash? 
+Please send us an email at [info@expectedparrot.com](mailto:info@expectedparrot.com) or message us at our [Discord channel](https://discord.com/invite/mxAYkjfy9m).

edsl-0.1.29/edsl/Base.py

@@ -0,0 +1,288 @@
+"""Base class for all classes in the package. It provides rich printing and persistence of objects."""
+
+from abc import ABC, abstractmethod, ABCMeta
+import gzip
+import io
+import json
+from typing import Any, Optional, Union
+from uuid import UUID
+
+
+class RichPrintingMixin:
+    """Mixin for rich printing and persistence of objects."""
+
+    def _for_console(self):
+        """Return a string representation of the object for console printing."""
+        from rich.console import Console
+
+        with io.StringIO() as buf:
+            console = Console(file=buf, record=True)
+            table = self.rich_print()
+            console.print(table)
+            return console.export_text()
+
+    def __str__(self):
+        """Return a string representation of the object for console printing."""
+        return self._for_console()
+
+    def print(self):
+        """Print the object to the console."""
+        from edsl.utilities.utilities import is_notebook
+
+        if is_notebook():
+            from IPython.display import display
+
+            display(self.rich_print())
+        else:
+            from rich.console import Console
+
+            console = Console()
+            console.print(self.rich_print())
+
+
+class PersistenceMixin:
+    """Mixin for saving and loading objects to and from files."""
+
+    def push(
+        self,
+        description: Optional[str] = None,
+        visibility: Optional[str] = "unlisted",
+    ):
+        """Post the object to coop."""
+        from edsl.coop import Coop
+
+        c = Coop()
+        return c.create(self, description, visibility)
+
+    @classmethod
+    def pull(cls, uuid: Optional[Union[str, UUID]] = None, url: Optional[str] = None):
+        """Pull the object from coop."""
+        from edsl.coop import Coop
+        from edsl.coop.utils import ObjectRegistry
+
+        object_type = ObjectRegistry.get_object_type_by_edsl_class(cls)
+        coop = Coop()
+        return coop.get(uuid, url, object_type)
+
+    @classmethod
+    def delete(cls, uuid: Optional[Union[str, UUID]] = None, url: Optional[str] = None):
+        """Delete the object from coop."""
+        from edsl.coop import Coop
+
+        coop = Coop()
+        return coop.delete(uuid, url)
+
+    @classmethod
+    def patch(
+        cls,
+        uuid: Optional[Union[str, UUID]] = None,
+        url: Optional[str] = None,
+        description: Optional[str] = None,
+        value: Optional[Any] = None,
+        visibility: Optional[str] = None,
+    ):
+        """
+        Patch an uploaded objects attributes.
+        - `description` changes the description of the object on Coop
+        - `value` changes the value of the object on Coop. **has to be an EDSL object**
+        - `visibility` changes the visibility of the object on Coop
+        """
+        from edsl.coop import Coop
+
+        coop = Coop()
+        return coop.patch(uuid, url, description, value, visibility)
+
+    @classmethod
+    def search(cls, query):
+        """Search for objects on coop."""
+        from edsl.coop import Coop
+
+        c = Coop()
+        return c.search(cls, query)
+
+    def save(self, filename, compress=True):
+        """Save the object to a file as zippped JSON.
+
+        >>> obj.save("obj.json.gz")
+
+        """
+        if filename.endswith("json.gz"):
+            import warnings
+
+            warnings.warn(
+                "Do not apply the file extensions. The filename should not end with 'json.gz'."
+            )
+            filename = filename[:-7]
+        if filename.endswith("json"):
+            filename = filename[:-4]
+            warnings.warn(
+                "Do not apply the file extensions. The filename should not end with 'json'."
+            )
+
+        if compress:
+            with gzip.open(filename + ".json.gz", "wb") as f:
+                f.write(json.dumps(self.to_dict()).encode("utf-8"))
+        else:
+            with open(filename + ".json", "w") as f:
+                f.write(json.dumps(self.to_dict()))
+
+    @staticmethod
+    def open_compressed_file(filename):
+        with gzip.open(filename, "rb") as f:
+            file_contents = f.read()
+            file_contents_decoded = file_contents.decode("utf-8")
+            d = json.loads(file_contents_decoded)
+        return d
+
+    @staticmethod
+    def open_regular_file(filename):
+        with open(filename, "r") as f:
+            d = json.loads(f.read())
+        return d
+
+    @classmethod
+    def load(cls, filename):
+        """Load the object from a file.
+
+        >>> obj = cls.load("obj.json.gz")
+
+        """
+
+        if filename.endswith("json.gz"):
+            d = cls.open_compressed_file(filename)
+        elif filename.endswith("json"):
+            d = cls.open_regular_file(filename)
+        else:
+            try:
+                d = cls.open_compressed_file(filename)
+            except:
+                d = cls.open_regular_file(filename)
+            finally:
+                raise ValueError("File must be a json or json.gz file")
+
+        return cls.from_dict(d)
+
+
+class RegisterSubclassesMeta(ABCMeta):
+    """Metaclass for registering subclasses."""
+
+    _registry = {}
+
+    def __init__(cls, name, bases, nmspc):
+        """Register the class in the registry upon creation."""
+        super(RegisterSubclassesMeta, cls).__init__(name, bases, nmspc)
+        if cls.__name__ != "Base":
+            RegisterSubclassesMeta._registry[cls.__name__] = cls
+
+    @staticmethod
+    def get_registry():
+        """Return the registry of subclasses."""
+        return dict(RegisterSubclassesMeta._registry)
+
+
+class DiffMethodsMixin:
+    def __sub__(self, other):
+        """Return the difference between two objects."""
+        from edsl.BaseDiff import BaseDiff
+
+        return BaseDiff(self, other)
+
+
+class Base(
+    RichPrintingMixin,
+    PersistenceMixin,
+    DiffMethodsMixin,
+    ABC,
+    metaclass=RegisterSubclassesMeta,
+):
+    """Base class for all classes in the package."""
+
+    # def __getitem__(self, key):
+    #     return getattr(self, key)
+
+    # @abstractmethod
+    # def _repr_html_(self) -> str:
+    #     raise NotImplementedError("This method is not implemented yet.")
+
+    # @abstractmethod
+    # def _repr_(self) -> str:
+    #     raise NotImplementedError("This method is not implemented yet.")
+
+    def keys(self):
+        """Return the keys of the object."""
+        _keys = list(self.to_dict().keys())
+        if "edsl_version" in _keys:
+            _keys.remove("edsl_version")
+        if "edsl_class_name" in _keys:
+            _keys.remove("edsl_class_name")
+        return _keys
+
+    def values(self):
+        """Return the values of the object."""
+        data = self.to_dict()
+        keys = self.keys()
+        return {data[key] for key in keys}
+
+    def _repr_html_(self):
+        from edsl.utilities.utilities import data_to_html
+
+        return data_to_html(self.to_dict())
+
+    # def html(self):
+    #     html_string = self._repr_html_()
+    #     import tempfile
+    #     import webbrowser
+
+    #     with tempfile.NamedTemporaryFile("w", delete=False, suffix=".html") as f:
+    #         # print("Writing HTML to", f.name)
+    #         f.write(html_string)
+    #         webbrowser.open(f.name)
+
+    def __eq__(self, other):
+        """Return whether two objects are equal."""
+        import inspect
+
+        if not isinstance(other, self.__class__):
+            return False
+        if "sort" in inspect.signature(self._to_dict).parameters:
+            return self._to_dict(sort=True) == other._to_dict(sort=True)
+        else:
+            return self._to_dict() == other._to_dict()
+
+    @abstractmethod
+    def example():
+        """This method should be implemented by subclasses."""
+        raise NotImplementedError("This method is not implemented yet.")
+
+    @abstractmethod
+    def rich_print():
+        """This method should be implemented by subclasses."""
+        raise NotImplementedError("This method is not implemented yet.")
+
+    @abstractmethod
+    def to_dict():
+        """This method should be implemented by subclasses."""
+        raise NotImplementedError("This method is not implemented yet.")
+
+    @abstractmethod
+    def from_dict():
+        """This method should be implemented by subclasses."""
+        raise NotImplementedError("This method is not implemented yet.")
+
+    @abstractmethod
+    def code():
+        """This method should be implemented by subclasses."""
+        raise NotImplementedError("This method is not implemented yet.")
+
+    def show_methods(self, show_docstrings=True):
+        """Show the methods of the object."""
+        public_methods_with_docstrings = [
+            (method, getattr(self, method).__doc__)
+            for method in dir(self)
+            if callable(getattr(self, method)) and not method.startswith("_")
+        ]
+        if show_docstrings:
+            for method, documentation in public_methods_with_docstrings:
+                print(f"{method}: {documentation}")
+        else:
+            return [x[0] for x in public_methods_with_docstrings]

edsl-0.1.29/edsl/BaseDiff.py

@@ -0,0 +1,260 @@
+import difflib
+import json
+from typing import Any, Dict, Tuple
+from collections import UserList
+import inspect
+
+
+class BaseDiffCollection(UserList):
+    def __init__(self, diffs=None):
+        if diffs is None:
+            diffs = []
+        super().__init__(diffs)
+
+    def apply(self, obj: Any):
+        for diff in self:
+            obj = diff.apply(obj)
+        return obj
+
+    def add_diff(self, diff) -> "BaseDiffCollection":
+        self.append(diff)
+        return self
+
+
+class DummyObject:
+    def __init__(self, object_dict):
+        self.object_dict = object_dict
+
+    def _to_dict(self):
+        return self.object_dict
+
+
+class BaseDiff:
+    def __init__(
+        self, obj1: Any, obj2: Any, added=None, removed=None, modified=None, level=0
+    ):
+        self.level = level
+
+        self.obj1 = obj1
+        self.obj2 = obj2
+
+        if "sort" in inspect.signature(obj1._to_dict).parameters:
+            self._dict1 = obj1._to_dict(sort=True)
+            self._dict2 = obj2._to_dict(sort=True)
+        else:
+            self._dict1 = obj1._to_dict()
+            self._dict2 = obj2._to_dict()
+        self._obj_class = type(obj1)
+
+        self.added = added
+        self.removed = removed
+        self.modified = modified
+
+    def __bool__(self):
+        return bool(self.added or self.removed or self.modified)
+
+    @property
+    def added(self):
+        if self._added is None:
+            self._added = self._find_added()
+        return self._added
+
+    def __add__(self, other):
+        return self.apply(other)
+
+    @added.setter
+    def added(self, value):
+        self._added = value if value is not None else self._find_added()
+
+    @property
+    def removed(self):
+        if self._removed is None:
+            self._removed = self._find_removed()
+        return self._removed
+
+    @removed.setter
+    def removed(self, value):
+        self._removed = value if value is not None else self._find_removed()
+
+    @property
+    def modified(self):
+        if self._modified is None:
+            self._modified = self._find_modified()
+        return self._modified
+
+    @modified.setter
+    def modified(self, value):
+        self._modified = value if value is not None else self._find_modified()
+
+    def _find_added(self) -> Dict[Any, Any]:
+        return {k: self._dict2[k] for k in self._dict2 if k not in self._dict1}
+
+    def _find_removed(self) -> Dict[Any, Any]:
+        return {k: self._dict1[k] for k in self._dict1 if k not in self._dict2}
+
+    def _find_modified(self) -> Dict[Any, Tuple[Any, Any, str]]:
+        modified = {}
+        for k in self._dict1:
+            if k in self._dict2 and self._dict1[k] != self._dict2[k]:
+                if isinstance(self._dict1[k], str) and isinstance(self._dict2[k], str):
+                    diff = self._diff_strings(self._dict1[k], self._dict2[k])
+                    modified[k] = (self._dict1[k], self._dict2[k], diff)
+                elif isinstance(self._dict1[k], dict) and isinstance(
+                    self._dict2[k], dict
+                ):
+                    diff = self._diff_dicts(self._dict1[k], self._dict2[k])
+                    modified[k] = (self._dict1[k], self._dict2[k], diff)
+                elif isinstance(self._dict1[k], list) and isinstance(
+                    self._dict2[k], list
+                ):
+                    d1 = dict(zip(range(len(self._dict1[k])), self._dict1[k]))
+                    d2 = dict(zip(range(len(self._dict2[k])), self._dict2[k]))
+                    diff = BaseDiff(
+                        DummyObject(d1), DummyObject(d2), level=self.level + 1
+                    )
+                    modified[k] = (self._dict1[k], self._dict2[k], diff)
+                else:
+                    modified[k] = (self._dict1[k], self._dict2[k], "")
+        return modified
+
+    @staticmethod
+    def is_json(string_that_could_be_json: str) -> bool:
+        try:
+            json.loads(string_that_could_be_json)
+            return True
+        except json.JSONDecodeError:
+            return False
+
+    def _diff_dicts(self, dict1: Dict[str, Any], dict2: Dict[str, Any]) -> str:
+        diff = BaseDiff(DummyObject(dict1), DummyObject(dict2), level=self.level + 1)
+        return diff
+
+    def _diff_strings(self, str1: str, str2: str) -> str:
+        if self.is_json(str1) and self.is_json(str2):
+            diff = self._diff_dicts(json.loads(str1), json.loads(str2))
+            return diff
+        diff = difflib.ndiff(str1.splitlines(), str2.splitlines())
+        return diff
+
+    def apply(self, obj: Any):
+        """Apply the diff to the object."""
+
+        new_obj_dict = obj._to_dict()
+        for k, v in self.added.items():
+            new_obj_dict[k] = v
+        for k in self.removed.keys():
+            del new_obj_dict[k]
+        for k, (v1, v2, diff) in self.modified.items():
+            new_obj_dict[k] = v2
+
+        return obj.from_dict(new_obj_dict)
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "added": self.added,
+            "removed": self.removed,
+            "modified": self.modified,
+            "obj1": self._dict1,
+            "obj2": self._dict2,
+            "obj_class": self._obj_class.__name__,
+            "level": self.level,
+        }
+
+    @classmethod
+    def from_dict(cls, diff_dict: Dict[str, Any], obj1: Any, obj2: Any):
+        return cls(
+            obj1=obj1,
+            obj2=obj2,
+            added=diff_dict["added"],
+            removed=diff_dict["removed"],
+            modified=diff_dict["modified"],
+            level=diff_dict["level"],
+        )
+
+    class Results(UserList):
+        def __init__(self, prepend=" ", level=0):
+            super().__init__()
+            self.prepend = prepend
+            self.level = level
+
+        def append(self, item):
+            super().append(self.prepend * self.level + item)
+
+    def __str__(self):
+        prepend = " "
+        result = self.Results(level=self.level, prepend="\t")
+        if self.added:
+            result.append("Added keys and values:")
+            for k, v in self.added.items():
+                result.append(prepend + f"  {k}: {v}")
+        if self.removed:
+            result.append("Removed keys and values:")
+            for k, v in self.removed.items():
+                result.append(f"  {k}: {v}")
+        if self.modified:
+            result.append("Modified keys and values:")
+            for k, (v1, v2, diff) in self.modified.items():
+                result.append(f"Key: {k}:")
+                result.append(f"    Old value: {v1}")
+                result.append(f"    New value: {v2}")
+                if diff:
+                    result.append(f"    Diff:")
+                    try:
+                        for line in diff:
+                            result.append(f"      {line}")
+                    except:
+                        result.append(f"      {diff}")
+        return "\n".join(result)
+
+    def __repr__(self):
+        return (
+            f"BaseDiff(obj1={self.obj1!r}, obj2={self.obj2!r}, added={self.added!r}, "
+            f"removed={self.removed!r}, modified={self.modified!r})"
+        )
+
+    def add_diff(self, diff) -> "BaseDiffCollection":
+        return BaseDiffCollection([self, diff])
+
+
+if __name__ == "__main__":
+    from edsl import Question
+
+    q_ft = Question.example("free_text")
+    q_mc = Question.example("multiple_choice")
+
+    diff1 = q_ft - q_mc
+    assert q_ft == q_mc + diff1
+    assert q_ft == diff1.apply(q_mc)
+    # new_q_mc = diff1.apply(q_ft)
+    # assert new_q_mc == q_mc
+
+    # new_q_mc = q_ft + diff1
+    # assert new_q_mc == q_mc
+
+    # new_q_mc = diff1 + q_ft
+    # assert new_q_mc == q_mc
+
+    # ## Test chain of diffs
+    q0 = Question.example("free_text")
+    q1 = q0.copy()
+    q1.question_text = "Why is Buzzard's Bay so named?"
+    diff1 = q1 - q0
+    q2 = q1.copy()
+    q2.question_name = "buzzard_bay"
+    diff2 = q2 - q1
+
+    diff_chain = diff1.add_diff(diff2)
+
+    new_q2 = diff_chain.apply(q0)
+    assert new_q2 == q2
+
+    new_q2 = diff_chain + q0
+    assert new_q2 == q2
+
+    # new_diffs = diff1.add_diff(diff1).add_diff(diff1)
+    # assert len(new_diffs) == 3
+
+    # q0 = Question.example("free_text")
+    # q1 = Question.example("free_text")
+    # q1.question_text = "Why is Buzzard's Bay so named?"
+    # q2 = q1.copy()