llm-codegen-research 1.2__tar.gz

llm_codegen_research-1.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Luke Twist
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llm_codegen_research-1.2/PKG-INFO

@@ -0,0 +1,101 @@
+Metadata-Version: 2.4
+Name: llm-codegen-research
+Version: 1.2
+Summary: Utilities for use when research code-generation by LLMs.
+Author-email: Lukas Twist <itsluketwist@gmail.com>
+Project-URL: Homepage, https://github.com/itsluketwist/llm-codegen-research
+Keywords: llm,code-generation,research,prompting,nlp
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: anthropic>=0.49.0
+Requires-Dist: openai>=1.78.1
+Requires-Dist: together>=1.5.8
+Dynamic: license-file
+
+# **llm-codegen-research**
+
+
+![lint code workflow](https://github.com/itsluketwist/llm-codegen-research/actions/workflows/lint.yaml/badge.svg)
+![test code workflow](https://github.com/itsluketwist/llm-codegen-research/actions/workflows/test.yaml/badge.svg)
+![release workflow](https://github.com/itsluketwist/llm-codegen-research/actions/workflows/release.yaml/badge.svg)
+
+
+<div>
+    <!-- badges from : https://shields.io/ -->
+    <!-- logos available : https://simpleicons.org/ -->
+    <a href="https://opensource.org/licenses/MIT">
+        <img alt="MIT License" src="https://img.shields.io/badge/Licence-MIT-yellow?style=for-the-badge&logo=docs&logoColor=white" />
+    </a>
+    <a href="https://www.python.org/">
+        <img alt="Python 3" src="https://img.shields.io/badge/Python_3-blue?style=for-the-badge&logo=python&logoColor=white" />
+    </a>
+    <a href="https://openai.com/blog/openai-api/">
+        <img alt="OpenAI API" src="https://img.shields.io/badge/OpenAI_API-412991?style=for-the-badge&logo=openai&logoColor=white" />
+    </a>
+    <a href="https://www.anthropic.com/api/">
+        <img alt="Anthropic API" src="https://img.shields.io/badge/Claude_API-D97757?style=for-the-badge&logo=claude&logoColor=white" />
+    </a>
+    <a href="https://api.together.ai/">
+        <img alt="together.ai API" src="https://img.shields.io/badge/together.ai_API-B5B5B5?style=for-the-badge&logoColor=white" />
+    </a>
+</div>
+
+
+## *usage*
+
+A collection of methods and classes I repeatedly use when conducting research on LLM code-generation.
+Covers both prompting various LLMs, and analysing the markdown responses.
+
+```python
+from llm_cgr import quick_complete, Markdown
+
+response = quick_complete("Write python code to generate the nth fibonacci number.")
+
+markdown = Markdown(text=response)
+```
+
+## *installation*
+
+Install directly from PyPI, using pip:
+
+```shell
+pip install llm-codegen-research
+```
+
+## *development*
+
+Clone the repository code:
+
+```shell
+git clone https://github.com/itsluketwist/llm-codegen-research.git
+```
+
+We use [`uv`](https://astral.sh/blog/uv) for project management.
+Once cloned, create a virtual environment and install uv and the project:
+
+```shell
+python -m venv .venv
+
+. .venv/bin/activate
+
+pip install uv
+
+uv sync
+```
+
+Use `make` commands to lint and test:
+
+```shell
+make lint
+
+make test
+```
+
+Use `uv` to add new dependencies into the project and `uv.lock`:
+
+```shell
+uv add openai
+```

llm_codegen_research-1.2/README.md

@@ -0,0 +1,84 @@
+# **llm-codegen-research**
+
+
+![lint code workflow](https://github.com/itsluketwist/llm-codegen-research/actions/workflows/lint.yaml/badge.svg)
+![test code workflow](https://github.com/itsluketwist/llm-codegen-research/actions/workflows/test.yaml/badge.svg)
+![release workflow](https://github.com/itsluketwist/llm-codegen-research/actions/workflows/release.yaml/badge.svg)
+
+
+<div>
+    <!-- badges from : https://shields.io/ -->
+    <!-- logos available : https://simpleicons.org/ -->
+    <a href="https://opensource.org/licenses/MIT">
+        <img alt="MIT License" src="https://img.shields.io/badge/Licence-MIT-yellow?style=for-the-badge&logo=docs&logoColor=white" />
+    </a>
+    <a href="https://www.python.org/">
+        <img alt="Python 3" src="https://img.shields.io/badge/Python_3-blue?style=for-the-badge&logo=python&logoColor=white" />
+    </a>
+    <a href="https://openai.com/blog/openai-api/">
+        <img alt="OpenAI API" src="https://img.shields.io/badge/OpenAI_API-412991?style=for-the-badge&logo=openai&logoColor=white" />
+    </a>
+    <a href="https://www.anthropic.com/api/">
+        <img alt="Anthropic API" src="https://img.shields.io/badge/Claude_API-D97757?style=for-the-badge&logo=claude&logoColor=white" />
+    </a>
+    <a href="https://api.together.ai/">
+        <img alt="together.ai API" src="https://img.shields.io/badge/together.ai_API-B5B5B5?style=for-the-badge&logoColor=white" />
+    </a>
+</div>
+
+
+## *usage*
+
+A collection of methods and classes I repeatedly use when conducting research on LLM code-generation.
+Covers both prompting various LLMs, and analysing the markdown responses.
+
+```python
+from llm_cgr import quick_complete, Markdown
+
+response = quick_complete("Write python code to generate the nth fibonacci number.")
+
+markdown = Markdown(text=response)
+```
+
+## *installation*
+
+Install directly from PyPI, using pip:
+
+```shell
+pip install llm-codegen-research
+```
+
+## *development*
+
+Clone the repository code:
+
+```shell
+git clone https://github.com/itsluketwist/llm-codegen-research.git
+```
+
+We use [`uv`](https://astral.sh/blog/uv) for project management.
+Once cloned, create a virtual environment and install uv and the project:
+
+```shell
+python -m venv .venv
+
+. .venv/bin/activate
+
+pip install uv
+
+uv sync
+```
+
+Use `make` commands to lint and test:
+
+```shell
+make lint
+
+make test
+```
+
+Use `uv` to add new dependencies into the project and `uv.lock`:
+
+```shell
+uv add openai
+```

llm_codegen_research-1.2/pyproject.toml

@@ -0,0 +1,39 @@
+# Package specification, as defined here:
+# https://packaging.python.org/en/latest/specifications/pyproject-toml/#pyproject-toml-spec
+
+[build-system]
+requires = ["setuptools", "setuptools-git-versioning<2"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools-git-versioning]
+enabled = true
+
+[project]
+name = "llm-codegen-research"
+dynamic = ["version"]
+description = "Utilities for use when research code-generation by LLMs."
+readme = "README.md"
+authors = [{ name = "Lukas Twist", email = "itsluketwist@gmail.com" }]
+license-files = ["LICENSE"]
+classifiers = [
+    "Programming Language :: Python",
+    "Programming Language :: Python :: 3",
+]
+keywords = ["llm", "code-generation", "research", "prompting", "nlp"]
+requires-python = ">=3.11"
+dependencies = [
+    "anthropic>=0.49.0",
+    "openai>=1.78.1",
+    "together>=1.5.8",
+]
+
+[project.urls]
+Homepage = "https://github.com/itsluketwist/llm-codegen-research"
+
+[dependency-groups]
+dev = [
+    "pre-commit>=4.2.0",
+    "pytest>=8.3.5",
+    "pytest-cov>=6.1.1",
+    "uv>=0.7.4",
+]

llm_codegen_research-1.2/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

llm_codegen_research-1.2/src/llm_cgr/__init__.py

@@ -0,0 +1,19 @@
+from llm_cgr import analyse, query
+from llm_cgr.analyse import CodeBlock, CodeData, Markdown, analyse_code
+from llm_cgr.json_utils import read_json, save_json
+from llm_cgr.query import get_client, query_list, quick_generate
+
+
+__all__ = [
+    "analyse",
+    "query",
+    "CodeBlock",
+    "CodeData",
+    "Markdown",
+    "analyse_code",
+    "read_json",
+    "save_json",
+    "get_client",
+    "query_list",
+    "quick_generate",
+]

llm_codegen_research-1.2/src/llm_cgr/analyse/__init__.py

@@ -0,0 +1,10 @@
+from llm_cgr.analyse.classes import CodeBlock, Markdown
+from llm_cgr.analyse.languages import CodeData, analyse_code
+
+
+__all__ = [
+    "CodeBlock",
+    "Markdown",
+    "CodeData",
+    "analyse_code",
+]

llm_codegen_research-1.2/src/llm_cgr/analyse/classes.py

@@ -0,0 +1,139 @@
+"""Classes for handling markdown responses from LLMs."""
+
+from dataclasses import dataclass
+
+from llm_cgr.analyse.languages import analyse_code
+from llm_cgr.analyse.regexes import CODE_BLOCK_REGEX
+from llm_cgr.defaults import DEFAULT_CODEBLOCK_LANGUAGE
+
+
+@dataclass
+class CodeBlock:
+    """
+    A class to represent a block of code with it's language.
+    """
+
+    language: str | None
+    text: str
+    valid: bool | None  # None if validity unknown, language not supported
+    error: str | None  # only set if not valid
+    defined_funcs: list[str]
+    called_funcs: list[str]
+    packages: list[str]
+    imports: list[str]
+
+    def __init__(
+        self,
+        language: str | None,
+        text: str,
+        default_language: str | None = DEFAULT_CODEBLOCK_LANGUAGE,
+    ):
+        self.language = language.strip().lower() if language else None
+        self.text = text.strip()
+
+        code_data = analyse_code(
+            code=self.text, language=self.language or default_language
+        )
+
+        if self.language is None and not code_data.valid:
+            # if we hit errors after defaulting the language, ignore the results!
+            self.valid = None
+            self.error = None
+            self.defined_funcs = []
+            self.called_funcs = []
+            self.packages = []
+            self.imports = []
+
+        else:
+            self.language = self.language or default_language
+            self.valid = code_data.valid
+            self.error = code_data.error
+            self.defined_funcs = code_data.defined_funcs
+            self.called_funcs = code_data.called_funcs
+            self.packages = code_data.packages
+            self.imports = code_data.imports
+
+    def __repr__(self):
+        _language = f"language={self.language or 'unspecified'}"
+        _lines = f"lines={len(self.text.splitlines())}"
+        return f"{self.__class__.__name__}({_language}, {_lines})"
+
+    def __str__(self):
+        return self.text
+
+    @property
+    def markdown(self):
+        return f"```{self.language or ''}\n{self.text}\n```"
+
+
+@dataclass
+class Markdown:
+    """
+    A class to hold a markdown response from an LLM as a series of text and code blocks.
+    """
+
+    text: str
+    code_blocks: list[CodeBlock]
+    code_errors: list[str]
+    languages: list[str]
+
+    def __init__(
+        self,
+        text: str,
+        default_codeblock_language: str | None = DEFAULT_CODEBLOCK_LANGUAGE,
+    ):
+        """
+        Initialise the MarkdownResponse object with the text and code blocks.
+        Use `codeblock_language` when no language is specified for a code block.
+        """
+        self.text = text
+        self.code_blocks = self.extract_code_blocks(
+            response=text,
+            default_language=default_codeblock_language,
+        )
+        self.code_errors = [
+            f"{i}: {cb.error}" for i, cb in enumerate(self.code_blocks) if cb.error
+        ]
+        self.languages = sorted(
+            list({bl.language for bl in self.code_blocks if bl.language})
+        )
+
+    def __repr__(self):
+        _lines = f"lines={len(self.text.splitlines())}"
+        _code_blocks = f"code_blocks={len(self.code_blocks)}"
+        _languages = f"languages={','.join(self.languages)}"
+        return f"{self.__class__.__name__}({_lines}, {_code_blocks}, {_languages})"
+
+    def __str__(self):
+        return self.text
+
+    @staticmethod
+    def extract_code_blocks(
+        response: str,
+        default_language: str | None = DEFAULT_CODEBLOCK_LANGUAGE,
+    ) -> list[CodeBlock]:
+        """
+        Extract the code blocks from the markdown formatted text.
+        """
+        matches = CODE_BLOCK_REGEX.findall(string=response)
+        blocks = []
+        for match in matches:
+            language, text = match
+            blocks.append(
+                CodeBlock(
+                    language=language if language else None,
+                    text=text,
+                    default_language=default_language,
+                )
+            )
+        return blocks
+
+    def first_code_block(self, language: str) -> CodeBlock | None:
+        """
+        Return the first code block of a certain language in the response.
+        """
+        for block in self.code_blocks:
+            if block.language == language:
+                return block
+
+        return None

llm_codegen_research-1.2/src/llm_cgr/analyse/languages/__init__.py

@@ -0,0 +1,24 @@
+from llm_cgr.analyse.languages.code_data import CodeData
+from llm_cgr.analyse.languages.python import analyse_python_code
+
+
+def analyse_code(code: str, language: str | None) -> CodeData:
+    """
+    Analyse code based on the language.
+    """
+    try:
+        if language == "python":
+            return analyse_python_code(code=code)
+    except Exception as exc:
+        return CodeData(
+            valid=False,
+            error=str(exc),
+        )
+
+    return CodeData()
+
+
+__all__ = [
+    "analyse_code",
+    "CodeData",
+]

llm_codegen_research-1.2/src/llm_cgr/analyse/languages/code_data.py

@@ -0,0 +1,34 @@
+"""Define the CodeData class, to store code analysis data."""
+
+from dataclasses import dataclass
+from typing import Iterable
+
+
+@dataclass
+class CodeData:
+    """
+    A class to hold code analysis data.
+    """
+
+    valid: bool | None
+    error: str | None
+    defined_funcs: list[str]
+    called_funcs: list[str]
+    packages: list[str]
+    imports: list[str]
+
+    def __init__(
+        self,
+        valid: bool | None = None,
+        error: str | None = None,
+        defined_funcs: Iterable | None = None,
+        called_funcs: Iterable | None = None,
+        packages: Iterable | None = None,
+        imports: Iterable | None = None,
+    ):
+        self.valid = valid
+        self.error = error
+        self.defined_funcs = sorted(defined_funcs) if defined_funcs else []
+        self.called_funcs = sorted(called_funcs) if called_funcs else []
+        self.packages = sorted(packages) if packages else []
+        self.imports = sorted(imports) if imports else []

llm_codegen_research-1.2/src/llm_cgr/analyse/languages/python.py

@@ -0,0 +1,86 @@
+"""Utility functions for Python code analysis."""
+
+import ast
+import sys
+
+from llm_cgr.analyse.languages.code_data import CodeData
+
+
+PYTHON_STDLIB = getattr(
+    sys, "stdlib_module_names", []
+)  # use this below to determine packages
+
+
+class PythonAnalyser(ast.NodeVisitor):
+    def __init__(self):
+        self.defined_funcs: set[str] = set()
+        self.called_funcs: set[str] = set()
+        self.imports: set[str] = set()
+        self.packages: set[str] = set()
+
+    def visit_FunctionDef(self, node: ast.FunctionDef):
+        # save defined functions
+        self.defined_funcs.add(node.name)
+        self.generic_visit(node)
+
+    def visit_Call(self, node: ast.Call):
+        func = node.func
+
+        # save `foo()` function calls
+        if isinstance(func, ast.Name):
+            self.called_funcs.add(func.id)
+
+        # save `lib.method()` function calls
+        elif isinstance(func, ast.Attribute):
+            if isinstance(func.value, ast.Name):
+                self.called_funcs.add(f"{func.value.id}.{func.attr}")
+            else:
+                self.called_funcs.add(func.attr)
+
+        self.generic_visit(node)
+
+    def visit_Import(self, node: ast.Import):
+        # save `import module` imports
+        for alias in node.names:
+            # save all imports
+            self.imports.add(alias.name)
+
+            # save external packages
+            top_level = alias.name.split(".")[0]
+            if top_level not in PYTHON_STDLIB:
+                self.packages.add(top_level)
+
+        self.generic_visit(node)
+
+    def visit_ImportFrom(self, node: ast.ImportFrom):
+        # save `from module import thing` imports
+        module = node.module or ""
+
+        # save external packages
+        # node.level is 0 for absolute imports, 1+ for relative imports
+        if module and module not in PYTHON_STDLIB and node.level == 0:
+            self.packages.add(module.split(".")[0])
+
+        # save all imports
+        for alias in node.names:
+            full = f"{module}.{alias.name}" if module else alias.name
+            self.imports.add(full)
+
+        self.generic_visit(node)
+
+
+def analyse_python_code(code: str) -> CodeData:
+    """
+    Analyse Python code to extract functions and imports.
+    """
+    tree = ast.parse(code)
+    analyser = PythonAnalyser()
+    analyser.visit(tree)
+    return CodeData(
+        valid=True,
+        defined_funcs=analyser.defined_funcs,
+        called_funcs=analyser.called_funcs,
+        # packages lowercase for analysis (names are case-insensitive outside of code)
+        packages=[p.lower() for p in analyser.packages],
+        imports=analyser.imports,
+    )

llm_codegen_research-1.2/src/llm_cgr/analyse/regexes.py

@@ -0,0 +1,10 @@
+"""Regexes used to parse markdown."""
+
+import re
+
+
+# regex to extract code blocks from markdown text
+CODE_BLOCK_REGEX = re.compile(
+    pattern=r"\`\`\`(\w*)\n(.*?)(?:\`\`\`|$)",
+    flags=re.DOTALL,
+)

llm_codegen_research-1.2/src/llm_cgr/defaults.py

@@ -0,0 +1,10 @@
+"""Collection of default values for the llm_cgr package."""
+
+# the default model to be used across tasks
+DEFAULT_MODEL = "gpt-4.1-mini-2025-04-14"
+
+# the default language to assume for code blocks if not specified
+DEFAULT_CODEBLOCK_LANGUAGE = "python"
+
+# the default max_tokens to be used when prompting models
+DEFAULT_MAX_TOKENS = 2000

llm_codegen_research-1.2/src/llm_cgr/json_utils.py

@@ -0,0 +1,24 @@
+"""Simple utility functions to save and read json files."""
+
+import json
+
+
+def save_json(
+    data: dict | list,
+    file_path: str,
+):
+    """
+    Utility to save python dictionary or list to a json file.
+    """
+    with open(file_path, mode="w", encoding="utf-8") as f:
+        json.dump(obj=data, fp=f, indent=4)
+
+
+def read_json(
+    file_path: str,
+) -> dict | list:
+    """
+    Utility to load python dictionary or list from a json file.
+    """
+    with open(file_path, mode="r", encoding="utf-8") as f:
+        return json.load(fp=f)

llm_codegen_research-1.2/src/llm_cgr/query/__init__.py

@@ -0,0 +1,18 @@
+from llm_cgr.query.api_utils import get_client, query_list, quick_generate
+from llm_cgr.query.generate import (
+    AnthropicGenerationAPI,
+    OpenAIGenerationAPI,
+    TogetherGenerationAPI,
+)
+from llm_cgr.query.protocol import GenerationProtocol
+
+
+__all__ = [
+    "get_client",
+    "query_list",
+    "quick_generate",
+    "AnthropicGenerationAPI",
+    "OpenAIGenerationAPI",
+    "TogetherGenerationAPI",
+    "GenerationProtocol",
+]

llm_codegen_research-1.2/src/llm_cgr/query/api_utils.py

@@ -0,0 +1,99 @@
+"""API utilities for interfacing with the generation models."""
+
+from typing import Literal
+
+from llm_cgr.defaults import DEFAULT_MODEL
+from llm_cgr.query.generate import (
+    AnthropicGenerationAPI,
+    OpenAIGenerationAPI,
+    TogetherGenerationAPI,
+)
+from llm_cgr.query.prompts import (
+    BASE_SYSTEM_PROMPT,
+    CODE_SYSTEM_PROMPT,
+    LIST_SYSTEM_PROMPT,
+)
+from llm_cgr.query.protocol import GenerationProtocol
+
+
+def get_client(
+    model: str,
+    type: Literal["base", "code", "list"] | None = None,
+) -> GenerationProtocol:
+    """
+    Initialise the correct generation interface for the given model.
+    """
+    _system = None
+    match type:
+        case "base":
+            _system = BASE_SYSTEM_PROMPT
+        case "code":
+            _system = CODE_SYSTEM_PROMPT
+        case "list":
+            _system = LIST_SYSTEM_PROMPT
+
+    if "claude" in model:
+        return AnthropicGenerationAPI(model=model, system=_system)
+
+    if "gpt" in model or "o1" in model:
+        return OpenAIGenerationAPI(model=model, system=_system)
+
+    return TogetherGenerationAPI(model=model, system=_system)
+
+
+def quick_generate(
+    user: str,
+    type: Literal["base", "code", "list"] | None = None,
+    model: str = DEFAULT_MODEL,
+    system: str | None = None,
+    temperature: float | None = None,
+) -> str:
+    """
+    Simple function to quickly prompt a model for a response.
+    """
+    client = get_client(model=model, type=type)
+
+    [result] = client.generate(
+        user=user,
+        system=system,
+        temperature=temperature,
+    )
+    return result
+
+
+def query_list(
+    user: str,
+    system: str = LIST_SYSTEM_PROMPT,
+    model: str = DEFAULT_MODEL,
+) -> list[str]:
+    """
+    Simple function to quickly prompt a model for a list of words.
+    """
+    _response = quick_generate(
+        user=user,
+        system=system,
+        model=model,
+    )
+
+    # sometimes the LLM will return a code block with the list inside it
+    if _response.startswith("```python"):
+        _response = _response.split("```python")[1]
+    if _response.endswith("```"):
+        _response = _response.split("```")[0]
+    _response = _response.strip()
+
+    try:
+        _list = eval(_response)
+    except Exception as e:
+        print(f"Error evaluating response.\nresponse: {_response}\nexception: {e}")
+        _list = []
+
+    if not isinstance(_list, list):
+        print(f"Error querying list. Response is not a list: {_list}")
+        _list = []
+
+    if any(not isinstance(item, str) for item in _list):
+        print(f"Error querying list. Response contains non-string items: {_list}")
+        _list = []
+
+    return _list

llm_codegen_research-1.2/src/llm_cgr/query/generate.py

@@ -0,0 +1,149 @@
+"""Classes for access to generation APIs of LLM services."""
+
+import anthropic
+import openai
+import together
+
+from llm_cgr.defaults import DEFAULT_MAX_TOKENS
+
+
+class OpenAIGenerationAPI:
+    """
+    Class to access the OpenAI API.
+    """
+
+    def __init__(
+        self,
+        model: str | None = None,
+        system: str | None = None,
+    ) -> None:
+        self._model = model
+        self._system = system or openai.NOT_GIVEN
+        self._client = openai.OpenAI()
+
+    def generate(
+        self,
+        user: str,
+        system: str | None = None,
+        model: str | None = None,
+        samples: int = 1,
+        temperature: float | None = None,
+        max_tokens: int | None = None,
+    ) -> list[str]:
+        """
+        Generate a model response from the OpenAI API.
+
+        Returns
+        -------
+        The text response to the prompt.
+        """
+        _generations = []
+        for _ in range(samples):
+            response = self._client.responses.create(
+                input=user,
+                model=model or self._model,
+                instructions=system or self._system,
+                temperature=temperature or openai.NOT_GIVEN,
+                max_output_tokens=max_tokens or openai.NOT_GIVEN,
+            )
+            _generations.append(response.output_text)
+
+        return _generations
+
+
+class TogetherGenerationAPI:
+    """
+    Class to access the TogetherAI API.
+    """
+
+    def __init__(
+        self,
+        model: str | None = None,
+        system: str | None = None,
+    ) -> None:
+        self._model = model
+        self._system = system
+        self._client = together.Together()
+
+    def generate(
+        self,
+        user: str,
+        system: str | None = None,
+        model: str | None = None,
+        samples: int = 1,
+        temperature: float | None = None,
+        max_tokens: int | None = None,
+    ) -> list[str]:
+        """
+        Generate a model response from the TogetherAI API.
+
+        Returns
+        -------
+        The text response to the prompt.
+        """
+        _input = [{"role": "user", "content": user}]
+        _system = system or self._system
+        if _system:
+            _input = [{"role": "system", "content": _system}] + _input
+
+        _generations = []
+        for _ in range(samples):
+            response = self._client.chat.completions.create(
+                model=model or self._model,
+                messages=_input,
+                temperature=temperature,
+                max_tokens=max_tokens,
+            )
+            _generations.append(response.choices[0].message.content)
+
+        return _generations
+
+
+class AnthropicGenerationAPI:
+    """
+    Class to access the Anthropic Claude API.
+    """
+
+    def __init__(self, model: str | None = None, system: str | None = None) -> None:
+        self._model = model
+        self._system = system or anthropic.NOT_GIVEN
+        self._client = anthropic.Anthropic()
+
+    def generate(
+        self,
+        user: str,
+        system: str | None = None,
+        model: str | None = None,
+        samples: int = 1,
+        temperature: float | None = None,
+        max_tokens: int | None = None,
+    ) -> list[str]:
+        """
+        Generate a model response from the Anthropic Claude API.
+
+        Returns
+        -------
+        The text response to the prompt.
+        """
+        _generations = []
+        for _ in range(samples):
+            response = self._client.messages.create(
+                model=model or self._model,
+                system=system or self._system,
+                messages=[
+                    {
+                        "role": "user",
+                        "content": [
+                            {
+                                "type": "text",
+                                "text": user,
+                            },
+                        ],
+                    },
+                ],
+                temperature=temperature or anthropic.NOT_GIVEN,
+                max_tokens=max_tokens or DEFAULT_MAX_TOKENS,
+            )
+            _generations.append(response.content[0].text)
+
+        return _generations

llm_codegen_research-1.2/src/llm_cgr/query/prompts.py

@@ -0,0 +1,13 @@
+"""All prompts used in the project are defined here."""
+
+# the default system prompt to be used across tasks
+BASE_SYSTEM_PROMPT = "You are a helpful assistant."
+
+# the default system prompt to be used across coding tasks
+CODE_SYSTEM_PROMPT = "You are a helpful and knowledgeable code assistant!"
+
+# the system prompt to be used for generating lists of words
+LIST_SYSTEM_PROMPT = (
+    "You are a helpful assistant that provides lists of words.\n"
+    "You only respond in correctly formatted python lists, containing only strings."
+)

llm_codegen_research-1.2/src/llm_cgr/query/protocol.py

@@ -0,0 +1,20 @@
+"""Protocol for the completion API of an LLM service."""
+
+from typing import Protocol
+
+
+class GenerationProtocol(Protocol):
+    """
+    Protocol that describes how to access the generation API of an LLM service.
+    """
+
+    def generate(
+        self,
+        user: str,
+        system: str | None = None,
+        model: str | None = None,
+        samples: int = 1,
+        temperature: float | None = None,
+        max_tokens: int | None = None,
+    ) -> list[str]:
+        pass

llm_codegen_research-1.2/src/llm_codegen_research.egg-info/PKG-INFO

@@ -0,0 +1,101 @@
+Metadata-Version: 2.4
+Name: llm-codegen-research
+Version: 1.2
+Summary: Utilities for use when research code-generation by LLMs.
+Author-email: Lukas Twist <itsluketwist@gmail.com>
+Project-URL: Homepage, https://github.com/itsluketwist/llm-codegen-research
+Keywords: llm,code-generation,research,prompting,nlp
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: anthropic>=0.49.0
+Requires-Dist: openai>=1.78.1
+Requires-Dist: together>=1.5.8
+Dynamic: license-file
+
+# **llm-codegen-research**
+
+
+![lint code workflow](https://github.com/itsluketwist/llm-codegen-research/actions/workflows/lint.yaml/badge.svg)
+![test code workflow](https://github.com/itsluketwist/llm-codegen-research/actions/workflows/test.yaml/badge.svg)
+![release workflow](https://github.com/itsluketwist/llm-codegen-research/actions/workflows/release.yaml/badge.svg)
+
+
+<div>
+    <!-- badges from : https://shields.io/ -->
+    <!-- logos available : https://simpleicons.org/ -->
+    <a href="https://opensource.org/licenses/MIT">
+        <img alt="MIT License" src="https://img.shields.io/badge/Licence-MIT-yellow?style=for-the-badge&logo=docs&logoColor=white" />
+    </a>
+    <a href="https://www.python.org/">
+        <img alt="Python 3" src="https://img.shields.io/badge/Python_3-blue?style=for-the-badge&logo=python&logoColor=white" />
+    </a>
+    <a href="https://openai.com/blog/openai-api/">
+        <img alt="OpenAI API" src="https://img.shields.io/badge/OpenAI_API-412991?style=for-the-badge&logo=openai&logoColor=white" />
+    </a>
+    <a href="https://www.anthropic.com/api/">
+        <img alt="Anthropic API" src="https://img.shields.io/badge/Claude_API-D97757?style=for-the-badge&logo=claude&logoColor=white" />
+    </a>
+    <a href="https://api.together.ai/">
+        <img alt="together.ai API" src="https://img.shields.io/badge/together.ai_API-B5B5B5?style=for-the-badge&logoColor=white" />
+    </a>
+</div>
+
+
+## *usage*
+
+A collection of methods and classes I repeatedly use when conducting research on LLM code-generation.
+Covers both prompting various LLMs, and analysing the markdown responses.
+
+```python
+from llm_cgr import quick_complete, Markdown
+
+response = quick_complete("Write python code to generate the nth fibonacci number.")
+
+markdown = Markdown(text=response)
+```
+
+## *installation*
+
+Install directly from PyPI, using pip:
+
+```shell
+pip install llm-codegen-research
+```
+
+## *development*
+
+Clone the repository code:
+
+```shell
+git clone https://github.com/itsluketwist/llm-codegen-research.git
+```
+
+We use [`uv`](https://astral.sh/blog/uv) for project management.
+Once cloned, create a virtual environment and install uv and the project:
+
+```shell
+python -m venv .venv
+
+. .venv/bin/activate
+
+pip install uv
+
+uv sync
+```
+
+Use `make` commands to lint and test:
+
+```shell
+make lint
+
+make test
+```
+
+Use `uv` to add new dependencies into the project and `uv.lock`:
+
+```shell
+uv add openai
+```

llm_codegen_research-1.2/src/llm_codegen_research.egg-info/SOURCES.txt

@@ -0,0 +1,25 @@
+LICENSE
+README.md
+pyproject.toml
+src/llm_cgr/__init__.py
+src/llm_cgr/defaults.py
+src/llm_cgr/json_utils.py
+src/llm_cgr/py.typed
+src/llm_cgr/analyse/__init__.py
+src/llm_cgr/analyse/classes.py
+src/llm_cgr/analyse/regexes.py
+src/llm_cgr/analyse/languages/__init__.py
+src/llm_cgr/analyse/languages/code_data.py
+src/llm_cgr/analyse/languages/python.py
+src/llm_cgr/query/__init__.py
+src/llm_cgr/query/api_utils.py
+src/llm_cgr/query/generate.py
+src/llm_cgr/query/prompts.py
+src/llm_cgr/query/protocol.py
+src/llm_codegen_research.egg-info/PKG-INFO
+src/llm_codegen_research.egg-info/SOURCES.txt
+src/llm_codegen_research.egg-info/dependency_links.txt
+src/llm_codegen_research.egg-info/requires.txt
+src/llm_codegen_research.egg-info/top_level.txt
+tests/test_analyse.py
+tests/test_query.py

llm_codegen_research-1.2/src/llm_codegen_research.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

llm_codegen_research-1.2/src/llm_codegen_research.egg-info/requires.txt

@@ -0,0 +1,3 @@
+anthropic>=0.49.0
+openai>=1.78.1
+together>=1.5.8

llm_codegen_research-1.2/src/llm_codegen_research.egg-info/top_level.txt

@@ -0,0 +1 @@
+llm_cgr

llm_codegen_research-1.2/tests/test_analyse.py

@@ -0,0 +1,125 @@
+from llm_cgr import Markdown
+
+
+TEST_LLM_RESPONSE = """
+Here's a Python solution to process some data and return an answer.
+
+```python
+import numpy as np
+from requests import get
+import json
+from collections import defaultdict
+from cryptography.fernet import Fernet
+import pandas.DataFrame
+
+def process_data(data):
+    response = get("https://api.example.com/data")
+    data = np.array([1, 2, 3, 4, 5])
+    return np.process(data, response)
+```
+
+Some more code:
+
+```
+import pandas as pd
+
+csv = pd.read_csv("data.csv")
+```
+
+Run some code:
+
+```bash
+python script.py
+```
+
+Some very bad python code:
+
+```python
+import problem
+
+problem.bad_brackets((()
+```
+
+Some very bad unknown code:
+
+```
+for import xxx)[
+```
+"""
+
+
+def test_markdown():
+    """
+    Test the MarkdownResponse class, extracting and analysing multiple code blocks.
+    """
+    # parse the response
+    analysed = Markdown(text=TEST_LLM_RESPONSE)
+
+    # check initial properties
+    assert analysed.text == TEST_LLM_RESPONSE
+    assert len(analysed.code_blocks) == 5
+    assert analysed.code_errors == ["3: '(' was never closed (<unknown>, line 3)"]
+    assert analysed.languages == ["bash", "python"]
+
+    # expected python code block
+    python_code_one = analysed.code_blocks[0]
+    assert python_code_one.language == "python"
+    assert python_code_one.valid is True
+    assert python_code_one.error is None
+    assert python_code_one.defined_funcs == ["process_data"]
+    assert python_code_one.called_funcs == ["get", "np.array", "np.process"]
+    assert python_code_one.packages == [
+        "cryptography",
+        "numpy",
+        "pandas",
+        "requests",
+    ]
+    assert python_code_one.imports == [
+        "collections.defaultdict",
+        "cryptography.fernet.Fernet",
+        "json",
+        "numpy",
+        "pandas.DataFrame",
+        "requests.get",
+    ]
+
+    # unspecified code block defaults to python
+    python_code_two = analysed.code_blocks[1]
+    assert python_code_two.language == "python"
+    assert python_code_two.valid is True
+    assert python_code_two.error is None
+    assert python_code_two.defined_funcs == []
+    assert python_code_two.called_funcs == ["pd.read_csv"]
+    assert python_code_two.packages == ["pandas"]
+    assert python_code_two.imports == ["pandas"]
+
+    # bash code block with no analysis
+    bash_code = analysed.code_blocks[2]
+    assert bash_code.language == "bash"
+    assert bash_code.valid is None
+    assert bash_code.error is None
+    assert bash_code.defined_funcs == []
+    assert bash_code.called_funcs == []
+    assert bash_code.packages == []
+    assert bash_code.imports == []
+
+    # python code block with incorrect syntax
+    bad_code = analysed.code_blocks[3]
+    assert bad_code.language == "python"
+    assert bad_code.valid is False
+    assert bad_code.error == "'(' was never closed (<unknown>, line 3)"
+    assert bad_code.defined_funcs == []
+    assert bad_code.called_funcs == []
+    assert bad_code.packages == []
+    assert bad_code.imports == []
+
+    # unknown code block
+    unknown_code = analysed.code_blocks[4]
+    assert unknown_code.text == "for import xxx)["
+    assert unknown_code.language is None
+    assert unknown_code.valid is None
+    assert unknown_code.error is None
+    assert unknown_code.defined_funcs == []
+    assert unknown_code.called_funcs == []
+    assert unknown_code.packages == []
+    assert unknown_code.imports == []