compressa-perf 0.1.0__tar.gz

compressa_perf-0.1.0/PKG-INFO

@@ -0,0 +1,208 @@
+Metadata-Version: 2.1
+Name: compressa-perf
+Version: 0.1.0
+Summary: Performance Measurement tool by Compresss
+Author: Gleb Morgachev
+Author-email: morgachev.g@gmail.com
+Requires-Python: >=3.9,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: openai (>=1.47.1,<2.0.0)
+Requires-Dist: python-dotenv (>=1.0.1,<2.0.0)
+Requires-Dist: tabulate (>=0.9.0,<0.10.0)
+Description-Content-Type: text/markdown
+
+# Compressa Performance Measurement Tool
+
+This tool is designed to measure the performance of Compressa models.  
+It uses the OpenAI API to run inference tasks and stores the results in a SQLite database.
+
+## Installation
+
+```bash
+git clone https://github.com/compressa-ai/compressa-perf.git
+cd compressa-perf
+poetry install
+```
+
+## Install with Pip
+
+```bash
+pip install compressa-perf
+```
+
+## Usage
+
+### 1. Run experiment with prompts from a file
+
+```bash
+❯ compressa-perf measure \
+    --openai_url https://some-api-url.ru/chat-2/v1/ \
+    --api_key "${OPENAI_API_KEY}" \
+    --model_name Compressa-Qwen2.5-14B-Instruct \
+    --experiment_name "File Prompts Run" \
+    --prompts_file resources/prompts.csv \
+    --num_tasks 1000 \
+    --num_runners 100
+```
+
+### 2. Run experiment with generated prompts
+
+```bash
+❯ compressa-perf measure \
+    --openai_url https://some-api-url.ru/chat-2/v1/ \
+    --api_key "${OPENAI_API_KEY}" \
+    --model_name Compressa-Qwen2.5-14B-Instruct \
+    --experiment_name "Generated Prompts Run" \
+    --num_tasks 2 \
+    --num_runners 2 \
+    --generate_prompts \
+    --num_prompts 1000 \
+    --prompt_length 5000
+```
+
+Full parameter list can be obtained with `compressa-perf measure -h`.
+
+### 3. Run set of experiments from YAML file
+
+You can describe set of experiments in YAML file and run them on different services in one command:
+
+```bash
+❯ compressa-perf measure-from-yaml experiments.yaml
+```
+
+Example of YAML file:
+
+```yaml
+- openai_url: https://some-api-url/v1/
+  model_name: Compressa-Qwen2-72B-Instruct
+  experiment_name: "File Prompts Run 1"
+  description: "Experiment using prompts from a file with 10 tasks and 5 runners"
+  prompts_file: resources/prompts.csv
+  num_tasks: 10
+  num_runners: 5
+  generate_prompts: false
+  num_prompts: 0
+  prompt_length: 0
+  max_tokens: 1000
+
+- openai_url: https://some-api-url/v1/
+  model_name: Compressa-Qwen2-72B-Instruct
+  experiment_name: "Qwen2-72B Long Input / Short Output"
+  description: "Experiment using prompts from a file with 20 tasks and 10 runners"
+  prompts_file: resources/prompts.csv
+  num_tasks: 20
+  num_runners: 10
+  generate_prompts: true
+  num_prompts: 10
+  prompt_length: 10000
+  max_tokens: 100
+```
+
+### 4. List experiments
+
+You can select experiments by name, parameters or metrics (or substrings in these fields) via `compressa-perf list` command.
+
+For example:
+```
+❯ compressa-perf list \
+    --show-metrics \
+    --param-filter openai_url=chat-2 \
+    --param-filter avg_n_input=30
+
+List of Experiments:
++----+----------------------------------------------------------------------------+---------------------+--------+-----------------------+
+|    | ID                                                                         | Name                | Date   | Description           |
++====+============================================================================+=====================+========+=======================+
+| 25 | Compressa-Qwen2.5-14B-Instruct-Int4 Long Input / Short Output | 5 runners  | 2024-10-03 06:21:45 |        | ttft: 25.0960         |
+|    |                                                                            |                     |        | latency: 52.5916      |
+|    |                                                                            |                     |        | tpot: 0.5530          |
+|    |                                                                            |                     |        | throughput: 2891.0323 |
++----+----------------------------------------------------------------------------+---------------------+--------+-----------------------+
+| 23 | Compressa-Qwen2.5-14B-Instruct-Int4 Long Input / Short Output | 4 runners  | 2024-10-03 06:14:57 |        | ttft: 17.1862         |
+|    |                                                                            |                     |        | latency: 37.9612      |
+|    |                                                                            |                     |        | tpot: 0.3954          |
+|    |                                                                            |                     |        | throughput: 3230.8769 |
++----+----------------------------------------------------------------------------+---------------------+--------+-----------------------+
+```
+
+Full parameter list:
+```bash
+❯ compressa-perf list -h
+usage: compressa-perf list [-h] [--db DB] [--show-parameters] [--show-metrics] [--name-filter NAME_FILTER] [--param-filter PARAM_FILTER]
+
+options:
+  -h, --help            show this help message and exit
+  --db DB               Path to the SQLite database
+  --show-parameters     Show all parameters for each experiment
+  --show-metrics        Show metrics for each experiment
+  --name-filter NAME_FILTER
+                        Filter experiments by substring in the name
+  --param-filter PARAM_FILTER
+                        Filter experiments by parameter value (e.g., paramkey=value_substring)
+```
+
+
+### 5. Generate a report for an experiment
+
+```bash
+❯ compressa-perf report <EXPERIMENT_ID>
+```
+
+Output example:
+
+```
+❯ compressa-perf report 3
+
+Experiment Details:
+ID: 3
+Name: My First Run
+Date: 2024-09-24 07:10:39
+Description: None
+
+Experiment Parameters:
+╒══════════════╤═══════════════════════════════════════════╕
+│    Parameter │                                     Value │
+╞══════════════╪═══════════════════════════════════════════╡
+│  num_workers │                                         2 │
+├──────────────┼───────────────────────────────────────────┤
+│    num_tasks │                                         2 │
+├──────────────┼───────────────────────────────────────────┤
+│   openai_url │ https://some-api-url.ru/chat-2/v1/ │
+├──────────────┼───────────────────────────────────────────┤
+│  avg_n_input │                                        32 │
+├──────────────┼───────────────────────────────────────────┤
+│  std_n_input │                                    2.8284 │
+├──────────────┼───────────────────────────────────────────┤
+│ avg_n_output │                                  748.5000 │
+├──────────────┼───────────────────────────────────────────┤
+│ std_n_output │                                    2.1213 │
+╘══════════════╧═══════════════════════════════════════════╛
+
+Experiment Metrics:
+╒═══════════════════════╤══════════╕
+│ Metric                │    Value │
+╞═══════════════════════╪══════════╡
+│ MetricName.TTFT       │   0.7753 │
+├───────────────────────┼──────────┤
+│ MetricName.LATENCY    │   7.5016 │
+├───────────────────────┼──────────┤
+│ MetricName.TPOT       │   0.01   │
+├───────────────────────┼──────────┤
+│ MetricName.THROUGHPUT │ 207.84   │
+╘═══════════════════════╧══════════╛
+```
+
+For more information on available commands and options, run:
+
+```bash
+compressa-perf --help
+```
+
+## License
+
+This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.
+

compressa_perf-0.1.0/README.md

@@ -0,0 +1,190 @@
+# Compressa Performance Measurement Tool
+
+This tool is designed to measure the performance of Compressa models.  
+It uses the OpenAI API to run inference tasks and stores the results in a SQLite database.
+
+## Installation
+
+```bash
+git clone https://github.com/compressa-ai/compressa-perf.git
+cd compressa-perf
+poetry install
+```
+
+## Install with Pip
+
+```bash
+pip install compressa-perf
+```
+
+## Usage
+
+### 1. Run experiment with prompts from a file
+
+```bash
+❯ compressa-perf measure \
+    --openai_url https://some-api-url.ru/chat-2/v1/ \
+    --api_key "${OPENAI_API_KEY}" \
+    --model_name Compressa-Qwen2.5-14B-Instruct \
+    --experiment_name "File Prompts Run" \
+    --prompts_file resources/prompts.csv \
+    --num_tasks 1000 \
+    --num_runners 100
+```
+
+### 2. Run experiment with generated prompts
+
+```bash
+❯ compressa-perf measure \
+    --openai_url https://some-api-url.ru/chat-2/v1/ \
+    --api_key "${OPENAI_API_KEY}" \
+    --model_name Compressa-Qwen2.5-14B-Instruct \
+    --experiment_name "Generated Prompts Run" \
+    --num_tasks 2 \
+    --num_runners 2 \
+    --generate_prompts \
+    --num_prompts 1000 \
+    --prompt_length 5000
+```
+
+Full parameter list can be obtained with `compressa-perf measure -h`.
+
+### 3. Run set of experiments from YAML file
+
+You can describe set of experiments in YAML file and run them on different services in one command:
+
+```bash
+❯ compressa-perf measure-from-yaml experiments.yaml
+```
+
+Example of YAML file:
+
+```yaml
+- openai_url: https://some-api-url/v1/
+  model_name: Compressa-Qwen2-72B-Instruct
+  experiment_name: "File Prompts Run 1"
+  description: "Experiment using prompts from a file with 10 tasks and 5 runners"
+  prompts_file: resources/prompts.csv
+  num_tasks: 10
+  num_runners: 5
+  generate_prompts: false
+  num_prompts: 0
+  prompt_length: 0
+  max_tokens: 1000
+
+- openai_url: https://some-api-url/v1/
+  model_name: Compressa-Qwen2-72B-Instruct
+  experiment_name: "Qwen2-72B Long Input / Short Output"
+  description: "Experiment using prompts from a file with 20 tasks and 10 runners"
+  prompts_file: resources/prompts.csv
+  num_tasks: 20
+  num_runners: 10
+  generate_prompts: true
+  num_prompts: 10
+  prompt_length: 10000
+  max_tokens: 100
+```
+
+### 4. List experiments
+
+You can select experiments by name, parameters or metrics (or substrings in these fields) via `compressa-perf list` command.
+
+For example:
+```
+❯ compressa-perf list \
+    --show-metrics \
+    --param-filter openai_url=chat-2 \
+    --param-filter avg_n_input=30
+
+List of Experiments:
++----+----------------------------------------------------------------------------+---------------------+--------+-----------------------+
+|    | ID                                                                         | Name                | Date   | Description           |
++====+============================================================================+=====================+========+=======================+
+| 25 | Compressa-Qwen2.5-14B-Instruct-Int4 Long Input / Short Output | 5 runners  | 2024-10-03 06:21:45 |        | ttft: 25.0960         |
+|    |                                                                            |                     |        | latency: 52.5916      |
+|    |                                                                            |                     |        | tpot: 0.5530          |
+|    |                                                                            |                     |        | throughput: 2891.0323 |
++----+----------------------------------------------------------------------------+---------------------+--------+-----------------------+
+| 23 | Compressa-Qwen2.5-14B-Instruct-Int4 Long Input / Short Output | 4 runners  | 2024-10-03 06:14:57 |        | ttft: 17.1862         |
+|    |                                                                            |                     |        | latency: 37.9612      |
+|    |                                                                            |                     |        | tpot: 0.3954          |
+|    |                                                                            |                     |        | throughput: 3230.8769 |
++----+----------------------------------------------------------------------------+---------------------+--------+-----------------------+
+```
+
+Full parameter list:
+```bash
+❯ compressa-perf list -h
+usage: compressa-perf list [-h] [--db DB] [--show-parameters] [--show-metrics] [--name-filter NAME_FILTER] [--param-filter PARAM_FILTER]
+
+options:
+  -h, --help            show this help message and exit
+  --db DB               Path to the SQLite database
+  --show-parameters     Show all parameters for each experiment
+  --show-metrics        Show metrics for each experiment
+  --name-filter NAME_FILTER
+                        Filter experiments by substring in the name
+  --param-filter PARAM_FILTER
+                        Filter experiments by parameter value (e.g., paramkey=value_substring)
+```
+
+
+### 5. Generate a report for an experiment
+
+```bash
+❯ compressa-perf report <EXPERIMENT_ID>
+```
+
+Output example:
+
+```
+❯ compressa-perf report 3
+
+Experiment Details:
+ID: 3
+Name: My First Run
+Date: 2024-09-24 07:10:39
+Description: None
+
+Experiment Parameters:
+╒══════════════╤═══════════════════════════════════════════╕
+│    Parameter │                                     Value │
+╞══════════════╪═══════════════════════════════════════════╡
+│  num_workers │                                         2 │
+├──────────────┼───────────────────────────────────────────┤
+│    num_tasks │                                         2 │
+├──────────────┼───────────────────────────────────────────┤
+│   openai_url │ https://some-api-url.ru/chat-2/v1/ │
+├──────────────┼───────────────────────────────────────────┤
+│  avg_n_input │                                        32 │
+├──────────────┼───────────────────────────────────────────┤
+│  std_n_input │                                    2.8284 │
+├──────────────┼───────────────────────────────────────────┤
+│ avg_n_output │                                  748.5000 │
+├──────────────┼───────────────────────────────────────────┤
+│ std_n_output │                                    2.1213 │
+╘══════════════╧═══════════════════════════════════════════╛
+
+Experiment Metrics:
+╒═══════════════════════╤══════════╕
+│ Metric                │    Value │
+╞═══════════════════════╪══════════╡
+│ MetricName.TTFT       │   0.7753 │
+├───────────────────────┼──────────┤
+│ MetricName.LATENCY    │   7.5016 │
+├───────────────────────┼──────────┤
+│ MetricName.TPOT       │   0.01   │
+├───────────────────────┼──────────┤
+│ MetricName.THROUGHPUT │ 207.84   │
+╘═══════════════════════╧══════════╛
+```
+
+For more information on available commands and options, run:
+
+```bash
+compressa-perf --help
+```
+
+## License
+
+This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.

compressa_perf-0.1.0/pyproject.toml

@@ -0,0 +1,29 @@
+[tool.poetry]
+name = "compressa-perf"
+version = "0.1.0"
+description = "Performance Measurement tool by Compresss"
+authors = ["Gleb Morgachev <morgachev.g@gmail.com>"]
+readme = "README.md"
+packages = [{include = "compressa", from = "src"}]
+
+[tool.poetry.dependencies]
+python = "^3.9"
+python-dotenv = "^1.0.1"
+openai = "^1.47.1"
+tabulate = "^0.9.0"
+
+[tool.poetry.group.dev.dependencies]
+jupyterlab = "^4.2.4"
+black = "^24.8.0"
+datasets = "^3.0.1"
+transformers = "^4.45.1"
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]
+
+[tool.poetry.scripts]
+compressa-perf = "compressa.perf.cli.__main__:main"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"

compressa_perf-0.1.0/src/compressa/perf/cli/__main__.py

@@ -0,0 +1,228 @@
+import argparse
+from typing import List
+from compressa.perf.cli.tools import (
+    run_experiment,
+    report_experiment,
+    list_experiments,
+    run_experiments_from_yaml,
+    DEFAULT_DB_PATH,
+)
+
+
+def run_experiment_args(args):
+    run_experiment(
+        db=args.db,
+        openai_api_key=args.openai_api_key,
+        openai_url=args.openai_url,
+        model_name=args.model_name,
+        experiment_name=args.experiment_name,
+        description=args.description,
+        prompts_file=args.prompts_file,
+        num_tasks=args.num_tasks,
+        num_runners=args.num_runners,
+        generate_prompts=args.generate_prompts,
+        num_prompts=args.num_prompts,
+        prompt_length=args.prompt_length,
+        max_tokens=args.max_tokens
+    )
+
+
+def report_experiment_args(args):
+    report_experiment(
+        experiment_id=args.experiment_id,
+        db=args.db,
+        recompute=args.recompute
+    )
+
+
+def list_experiments_args(args):
+    list_experiments(
+        db=args.db,
+        show_parameters=args.show_parameters,
+        show_metrics=args.show_metrics,
+        name_filter=args.name_filter,
+        param_filters=args.param_filter,
+    )
+
+
+def run_experiments_from_yaml_args(args):
+    run_experiments_from_yaml(
+        yaml_file=args.yaml_file,
+        db=args.db,
+        openai_api_key=args.openai_api_key
+    )
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="CLI tool for running and analyzing experiments",
+        epilog="""
+Examples:
+1. Run experiment with prompts from a file:
+    ```
+    compressa-perf measure \\
+        --openai_url https://api.qdrant.mil-team.ru/chat-2/v1/ \\
+        --api_key "${OPENAI_API_KEY}" \\
+        --model_name Compressa-Qwen2.5-14B-Instruct \\
+        --experiment_name "File Prompts Run" \\
+        --prompts_file resources/prompts.csv \\
+        --num_tasks 1000 \\
+        --num_runners 100
+    ```
+2. Run experiment with generated prompts:
+    ```
+    compressa-perf measure \\
+        --openai_url https://api.qdrant.mil-team.ru/chat-2/v1/ \\
+        --api_key "${OPENAI_API_KEY}" \\
+        --model_name Compressa-Qwen2.5-14B-Instruct \\
+        --experiment_name "Generated Prompts Run" \\
+        --num_tasks 2 \\
+        --num_runners 2 \\
+        --generate_prompts \\
+        --num_prompts 1000 \\
+        --prompt_length 5000
+    ```
+
+3. List all experiments:
+    ```
+    compressa-perf list
+    ```
+
+4. Generate a report for an experiment:
+    ```
+    compressa-perf report <EXPERIMENT_ID>
+    ```
+        """,
+        formatter_class=argparse.RawTextHelpFormatter
+    )
+    subparsers = parser.add_subparsers()
+
+    parser_run = subparsers.add_parser(
+        "measure",
+        help="Run an experiment",
+    )
+    parser_run.add_argument(
+        "--db",
+        type=str,
+        default=DEFAULT_DB_PATH,
+        help="Path to the SQLite database",
+    )
+    parser_run.add_argument(
+        "--openai_url", type=str, required=True, help="OpenAI-compatible API URL"
+    )
+    parser_run.add_argument(
+        "--model_name", type=str, required=True, help="Model name"
+    )
+    parser_run.add_argument(
+        "--experiment_name", type=str, required=True, help="Name of the experiment"
+    )
+    parser_run.add_argument(
+        "--description", type=str, help="Description of the experiment"
+    )
+    parser_run.add_argument(
+        "--prompts_file", type=str, help="Path to the file containing prompts (separated by newlines)"
+    )
+    parser_run.add_argument(
+        "--num_tasks", type=int, default=100, help="Number of requests to send"
+    )
+    parser_run.add_argument(
+        "--num_runners", type=int, default=10, help="Number of concurrent runners"
+    )
+    parser_run.add_argument(
+        "--api_key", type=str, required=True, help="API key"
+    )
+    parser_run.add_argument(
+        "--generate_prompts", action="store_true", help="Generate random prompts instead of using a file"
+    )
+    parser_run.add_argument(
+        "--num_prompts", type=int, default=100, help="Number of prompts to generate (if --generate_prompts is used)"
+    )
+    parser_run.add_argument(
+        "--prompt_length", type=int, default=100, help="Length of each generated prompt (if --generate_prompts is used)"
+    )
+    parser_run.add_argument(
+        "--max_tokens", type=int, default=1000, help="Maximum number of tokens for the model to generate"
+    )
+    parser_run.set_defaults(func=run_experiment_args)
+
+    parser_report = subparsers.add_parser(
+        "report",
+        help="Generate a report for an experiment",
+    )
+    parser_report.add_argument(
+        "experiment_id", type=int, help="ID of the experiment to report on"
+    )
+    parser_report.add_argument(
+        "--db",
+        type=str,
+        default=DEFAULT_DB_PATH,
+        help="Path to the SQLite database",
+    )
+    parser_report.add_argument(
+        "--recompute",
+        action="store_true",
+        help="Recompute metrics before generating the report",
+    )
+    parser_report.set_defaults(func=report_experiment_args)
+
+    parser_list = subparsers.add_parser(
+        "list",
+        help="List all experiments",
+    )
+    parser_list.add_argument(
+        "--db",
+        type=str,
+        default=DEFAULT_DB_PATH,
+        help="Path to the SQLite database",
+    )
+    parser_list.add_argument(
+        "--show-parameters",
+        action="store_true",
+        help="Show all parameters for each experiment"
+    )
+    parser_list.add_argument(
+        "--show-metrics",
+        action="store_true",
+        help="Show metrics for each experiment"
+    )
+    parser_list.add_argument(
+        "--name-filter",
+        type=str,
+        help="Filter experiments by substring in the name"
+    )
+    parser_list.add_argument(
+        "--param-filter",
+        type=str,
+        action="append",
+        help="Filter experiments by parameter value (e.g., paramkey=value_substring)"
+    )
+    parser_list.set_defaults(func=list_experiments_args)
+
+    parser_yaml = subparsers.add_parser(
+        "measure-from-yaml",
+        help="Run experiments from a YAML configuration file",
+    )
+    parser_yaml.add_argument(
+        "yaml_file",
+        help="YAML configuration file for experiments",
+    )
+    parser_yaml.add_argument(
+        "--db",
+        type=str,
+        default=DEFAULT_DB_PATH,
+        help="Path to the SQLite database",
+    )
+    parser_yaml.add_argument(
+        "--api_key",
+        type=str,
+        required=True,
+        help="OpenAI API key",
+    )
+    parser_yaml.set_defaults(func=run_experiments_from_yaml_args)
+
+    args = parser.parse_args()
+    args.func(args)
+
+
+if __name__ == "__main__":
+    main()