arize-phoenix 4.1.1__py3-none-any.whl → 4.1.2__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Potentially problematic release.
This version of arize-phoenix might be problematic. Click here for more details.
- arize_phoenix-4.1.2.dist-info/METADATA +177 -0
- {arize_phoenix-4.1.1.dist-info → arize_phoenix-4.1.2.dist-info}/RECORD +7 -7
- phoenix/trace/dsl/helpers.py +32 -20
- phoenix/version.py +1 -1
- arize_phoenix-4.1.1.dist-info/METADATA +0 -520
- {arize_phoenix-4.1.1.dist-info → arize_phoenix-4.1.2.dist-info}/WHEEL +0 -0
- {arize_phoenix-4.1.1.dist-info → arize_phoenix-4.1.2.dist-info}/licenses/IP_NOTICE +0 -0
- {arize_phoenix-4.1.1.dist-info → arize_phoenix-4.1.2.dist-info}/licenses/LICENSE +0 -0
|
@@ -0,0 +1,177 @@
|
|
|
1
|
+
Metadata-Version: 2.3
|
|
2
|
+
Name: arize-phoenix
|
|
3
|
+
Version: 4.1.2
|
|
4
|
+
Summary: AI Observability and Evaluation
|
|
5
|
+
Project-URL: Documentation, https://docs.arize.com/phoenix/
|
|
6
|
+
Project-URL: Issues, https://github.com/Arize-ai/phoenix/issues
|
|
7
|
+
Project-URL: Source, https://github.com/Arize-ai/phoenix
|
|
8
|
+
Author-email: Arize AI <phoenix-devs@arize.com>
|
|
9
|
+
License: Elastic-2.0
|
|
10
|
+
License-File: IP_NOTICE
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Keywords: Explainability,Monitoring,Observability
|
|
13
|
+
Classifier: Programming Language :: Python
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.8
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Requires-Python: <3.13,>=3.8
|
|
20
|
+
Requires-Dist: aioitertools
|
|
21
|
+
Requires-Dist: aiosqlite
|
|
22
|
+
Requires-Dist: alembic<2,>=1.3.0
|
|
23
|
+
Requires-Dist: cachetools
|
|
24
|
+
Requires-Dist: grpcio
|
|
25
|
+
Requires-Dist: hdbscan>=0.8.33
|
|
26
|
+
Requires-Dist: jinja2
|
|
27
|
+
Requires-Dist: numpy
|
|
28
|
+
Requires-Dist: openinference-instrumentation
|
|
29
|
+
Requires-Dist: openinference-instrumentation-langchain>=0.1.12
|
|
30
|
+
Requires-Dist: openinference-instrumentation-llama-index>=1.2.0
|
|
31
|
+
Requires-Dist: openinference-instrumentation-openai>=0.1.4
|
|
32
|
+
Requires-Dist: openinference-semantic-conventions>=0.1.5
|
|
33
|
+
Requires-Dist: opentelemetry-exporter-otlp
|
|
34
|
+
Requires-Dist: opentelemetry-proto>=1.12.0
|
|
35
|
+
Requires-Dist: opentelemetry-sdk
|
|
36
|
+
Requires-Dist: opentelemetry-semantic-conventions
|
|
37
|
+
Requires-Dist: pandas
|
|
38
|
+
Requires-Dist: protobuf<6.0,>=3.20
|
|
39
|
+
Requires-Dist: psutil
|
|
40
|
+
Requires-Dist: pyarrow
|
|
41
|
+
Requires-Dist: requests
|
|
42
|
+
Requires-Dist: scikit-learn
|
|
43
|
+
Requires-Dist: scipy
|
|
44
|
+
Requires-Dist: sqlalchemy[asyncio]<3,>=2.0.4
|
|
45
|
+
Requires-Dist: sqlean-py>=3.45.1
|
|
46
|
+
Requires-Dist: starlette
|
|
47
|
+
Requires-Dist: strawberry-graphql==0.227.2
|
|
48
|
+
Requires-Dist: tqdm
|
|
49
|
+
Requires-Dist: typing-extensions>=4.5; python_version < '3.12'
|
|
50
|
+
Requires-Dist: typing-extensions>=4.6; python_version >= '3.12'
|
|
51
|
+
Requires-Dist: umap-learn
|
|
52
|
+
Requires-Dist: uvicorn
|
|
53
|
+
Requires-Dist: wrapt
|
|
54
|
+
Provides-Extra: container
|
|
55
|
+
Requires-Dist: opentelemetry-exporter-otlp; extra == 'container'
|
|
56
|
+
Requires-Dist: opentelemetry-instrumentation-grpc; extra == 'container'
|
|
57
|
+
Requires-Dist: opentelemetry-instrumentation-sqlalchemy; extra == 'container'
|
|
58
|
+
Requires-Dist: opentelemetry-instrumentation-starlette; extra == 'container'
|
|
59
|
+
Requires-Dist: opentelemetry-proto>=1.12.0; extra == 'container'
|
|
60
|
+
Requires-Dist: opentelemetry-sdk; extra == 'container'
|
|
61
|
+
Requires-Dist: opentelemetry-semantic-conventions; extra == 'container'
|
|
62
|
+
Requires-Dist: prometheus-client; extra == 'container'
|
|
63
|
+
Requires-Dist: py-grpc-prometheus; extra == 'container'
|
|
64
|
+
Requires-Dist: strawberry-graphql[opentelemetry]==0.227.2; extra == 'container'
|
|
65
|
+
Requires-Dist: uvloop; (platform_system != 'Windows') and extra == 'container'
|
|
66
|
+
Provides-Extra: dev
|
|
67
|
+
Requires-Dist: anthropic; extra == 'dev'
|
|
68
|
+
Requires-Dist: arize[autoembeddings,llm-evaluation]; extra == 'dev'
|
|
69
|
+
Requires-Dist: asyncpg; extra == 'dev'
|
|
70
|
+
Requires-Dist: gcsfs; extra == 'dev'
|
|
71
|
+
Requires-Dist: google-cloud-aiplatform>=1.3; extra == 'dev'
|
|
72
|
+
Requires-Dist: hatch; extra == 'dev'
|
|
73
|
+
Requires-Dist: jupyter; extra == 'dev'
|
|
74
|
+
Requires-Dist: langchain>=0.0.334; extra == 'dev'
|
|
75
|
+
Requires-Dist: litellm>=1.0.3; extra == 'dev'
|
|
76
|
+
Requires-Dist: llama-index>=0.10.3; extra == 'dev'
|
|
77
|
+
Requires-Dist: nbqa; extra == 'dev'
|
|
78
|
+
Requires-Dist: pandas-stubs<=2.0.2.230605; extra == 'dev'
|
|
79
|
+
Requires-Dist: pre-commit; extra == 'dev'
|
|
80
|
+
Requires-Dist: prometheus-client; extra == 'dev'
|
|
81
|
+
Requires-Dist: psycopg[binary]; extra == 'dev'
|
|
82
|
+
Requires-Dist: pytest-asyncio; extra == 'dev'
|
|
83
|
+
Requires-Dist: pytest-cov; extra == 'dev'
|
|
84
|
+
Requires-Dist: pytest-lazy-fixture; extra == 'dev'
|
|
85
|
+
Requires-Dist: pytest-postgresql; extra == 'dev'
|
|
86
|
+
Requires-Dist: pytest==7.4.4; extra == 'dev'
|
|
87
|
+
Requires-Dist: ruff==0.3.0; extra == 'dev'
|
|
88
|
+
Requires-Dist: strawberry-graphql[debug-server,opentelemetry]==0.227.2; extra == 'dev'
|
|
89
|
+
Provides-Extra: evals
|
|
90
|
+
Requires-Dist: arize-phoenix-evals>=0.3.0; extra == 'evals'
|
|
91
|
+
Provides-Extra: experimental
|
|
92
|
+
Requires-Dist: tenacity; extra == 'experimental'
|
|
93
|
+
Provides-Extra: llama-index
|
|
94
|
+
Requires-Dist: llama-index-callbacks-arize-phoenix>=0.1.2; extra == 'llama-index'
|
|
95
|
+
Requires-Dist: llama-index==0.10.3; extra == 'llama-index'
|
|
96
|
+
Requires-Dist: openinference-instrumentation-llama-index>=1.2.0; extra == 'llama-index'
|
|
97
|
+
Provides-Extra: pg
|
|
98
|
+
Requires-Dist: asyncpg; extra == 'pg'
|
|
99
|
+
Description-Content-Type: text/markdown
|
|
100
|
+
|
|
101
|
+
<p align="center">
|
|
102
|
+
<a target="_blank" href="https://phoenix.arize.com" style="background:none">
|
|
103
|
+
<img alt="phoenix banner" src="https://github.com/Arize-ai/phoenix-assets/blob/main/images/socal/github-large-banner-phoenix.jpg?raw=true" width="auto" height="auto"></img>
|
|
104
|
+
</a>
|
|
105
|
+
<br/>
|
|
106
|
+
<br/>
|
|
107
|
+
<a href="https://docs.arize.com/phoenix/">
|
|
108
|
+
<img src="https://img.shields.io/static/v1?message=Docs&logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAIAAAACACAYAAADDPmHLAAAG4ElEQVR4nO2d4XHjNhCFcTf+b3ZgdWCmgmMqOKUC0xXYrsBOBVEqsFRB7ApCVRCygrMriFQBM7h5mNlwKBECARLg7jeDscamSQj7sFgsQfBL27ZK4MtXsT1vRADMEQEwRwTAHBEAc0QAzBEBMEcEwBwRAHNEAMwRATBnjAByFGE+MqVUMcYOY24GVUqpb/h8VErVKAf87QNFcEcbd4WSw+D6803njHscO5sATmGEURGBiCj6yUlv1uX2gv91FsDViArbcA2RUKF8QhAV8RQc0b15DcOt0VaTE1oAfWj3dYdCBfGGsmSM0XX5HsP3nEMAXbqCeCdiOERQPx9og5exGJ0S4zRQN9KrUupfpdQWjZciure/YIj7K0bjqwTyAHdovA805iqCOg2xgnB1nZ97IvaoSCURdIPG/IHGjTH/YAz/A8KdJai7lBQzgbpx/0Hg6DT18UzWMXxSjMkDrElPNEmKfAbl6znwI3IMU/OCa0/1nfckwWaSbvWYYDnEsvCMJDNckhqu7GCMKWYOBXp9yPGd5kvqUAKf6rkAk7M2SY9QDXdEr9wEOr9x96EiejMFnixBNteDISsyNw7hHRqc22evWcP4vt39O85bzZH30AKg4+eo8cQRI4bHAJ7hyYM3CNHrG9RrimSXuZmUkZjN/O6nAPpcwCcJNmipAle2QM/1GU3vITCXhvY91u9geN/jOY27VuTnYL1PCeAcRhwh7/Bl8Ai+IuxPiOCShtfX/sPDtY8w+sZjby86dw6dBeoigD7obd/Ko6fI4BF8DA9HnGdrcU0fLt+n4dfE6H5jpjYcVdu2L23b5lpjHoo+18FDbcszddF1rUee/4C6ZiO+80rHZmjDoIQUQLdRtm3brkcKIUPjjqVPBIUHgW1GGN4YfawAL2IqAVB8iEE31tvIelARlCPPVaFOLoIupzY6xVcM4MoRUyHXyHhslH6PaPl5RP1Lh4UsOeKR2e8dzC0Aiuvc2Nx3fwhfxf/hknouUYbWUk5GTAIwmOh5e+H0cor8vEL91hfOdEqINLq1AV+RKImJ6869f9tFIBVc6y7gd3lHfWyNX0LEr7EuDElhRdAlQjig0e/RU31xxDltM4pF7IY3pLIgxAhhgzF/iC2M0Hi4dkOGlyGMd/g7dsMbUlsR9ICe9WhxbA3DjRkSdjiHzQzlBSKNJsCzIcUlYdfI0dcWS8LMkPDkcJ0n/O+Qyy/IAtDkSPnp4Fu4WpthQR/zm2VcoI/51fI28iYld9/HEh4Pf7D0Bm845pwIPnHMUJSf45pT5x68s5T9AW6INzhHDeP1BYcNMew5SghkinWOwVnaBhHGG5ybMn70zBDe8buh8X6DqV0Sa/5tWOIOIbcWQ8KBiGBnMb/P0OuTd/lddCrY5jn/VLm3nL+fY4X4YREuv8vS9wh6HSkAExMs0viKySZRd44iyOH2FzPe98Fll7A7GNMmjay4GF9BAKGXesfCN0sRsDG+YrhP4O2ACFgZXzHdKPL2RMJoxc34ivFOod3AMMNUj5XxFfOtYrUIXvB5MandS+G+V/AzZ+MrEcBPlpoFtUIEwBwRAG+OIgDe1CIA5ogAmCMCYI4IgDkiAOaIAJgjAmCOCIA5IgDmiACYIwJgjgiAOSIA5ogAmCMCYI4IgDkiAOaIAJgjAmCOCIA5IgDmiACYIwJgjgiAOSIA5ogAmCMCYI4IgDkiAOaIAJgjAmDOVYBXvwvxQV8NWJOd0esvJ94babZaz7B5ovldxnlDpYhp0JFr/KTlLKcEMMQKpcDPXIQxGXsYmhZnXAXQh/EWBQrr3bc80mATyyrEvs4+BdBHgbdxFOIhrDkSg1/6Iu2LCS0AyoqI4ftUF00EY/Q3h1fRj2JKAVCMGErmnsH1lfnemEsAlByvgl0z2qx5B8OPCuB8EIMADBlEEOV79j1whNE3c/X2PmISAGUNr7CEmUSUhjfEKgBDAY+QohCiNrwhdgEYzPv7UxkadvBg0RrekMrNoAozh3vLN4DPhc7S/WL52vkoSO1u4BZC+DOCulC0KJ/gqWaP7C8hlSGgjxyCmDuPsEePT/KuasrrAcyr4H+f6fq01yd7Sz1lD0CZ2hs06PVJufs+lrIiyLwufjfBtXYpjvWnWIoHoJSYe4dIK/t4HX1ULFEACkPCm8e8wXFJvZ6y1EWhJkDcWxw7RINzLc74auGrgg8e4oIm9Sh/CA7LwkvHqaIJ9pLI6Lmy1BigDy2EV8tjdzh+8XB6MGSLKH4INsZXDJ8MGhIBK+Mrpo+GnRIBO+MrZjFAFxoTNBwCvj6u4qvSZJiM3iNX4yvmHoA9Sh4PF0QAzBEBMEcEwBwRAHNEAMwRAXBGKfUfr5hKvglRfO4AAAAASUVORK5CYII=&labelColor=grey&color=blue&logoColor=white&label=%20"/>
|
|
109
|
+
</a>
|
|
110
|
+
<a target="_blank" href="https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q">
|
|
111
|
+
<img src="https://img.shields.io/static/v1?message=Community&logo=slack&labelColor=grey&color=blue&logoColor=white&label=%20"/>
|
|
112
|
+
</a>
|
|
113
|
+
<a target="_blank" href="https://twitter.com/ArizePhoenix">
|
|
114
|
+
<img src="https://img.shields.io/badge/-ArizePhoenix-blue.svg?color=blue&labelColor=gray&logo=twitter">
|
|
115
|
+
</a>
|
|
116
|
+
<a target="_blank" href="https://pypi.org/project/arize-phoenix/">
|
|
117
|
+
<img src="https://img.shields.io/pypi/v/arize-phoenix?color=blue">
|
|
118
|
+
</a>
|
|
119
|
+
<a target="_blank" href="https://anaconda.org/conda-forge/arize-phoenix">
|
|
120
|
+
<img src="https://img.shields.io/conda/vn/conda-forge/arize-phoenix.svg?color=blue">
|
|
121
|
+
</a>
|
|
122
|
+
<a target="_blank" href="https://pypi.org/project/arize-phoenix/">
|
|
123
|
+
<img src="https://img.shields.io/pypi/pyversions/arize-phoenix">
|
|
124
|
+
</a>
|
|
125
|
+
<a target="_blank" href="https://hub.docker.com/r/arizephoenix/phoenix/tags">
|
|
126
|
+
<img src="https://img.shields.io/docker/v/arizephoenix/phoenix?sort=semver&logo=docker&label=image&color=blue">
|
|
127
|
+
</a>
|
|
128
|
+
</p>
|
|
129
|
+
|
|
130
|
+
Phoenix is an open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. It provides:
|
|
131
|
+
|
|
132
|
+
- **_Tracing_** - Trace your LLM application's runtime using using OpenTelemetry-based instrumentation.
|
|
133
|
+
- **_Evaluation_** - Leverage LLMs to benchmark your application's performance using response and retrieval evals.
|
|
134
|
+
- **_Inference Analysis_** - Visualize inferences and embeddings using dimensionality reduction and clustering to identify drift and performance degradation.
|
|
135
|
+
|
|
136
|
+
Phoenix is vendor and language agnostic with out-of-the-box support for popular frameworks (🦙LlamaIndex, 🦜⛓LangChain, 🧩DSPy) and LLM providers (OpenAI, Bedrock, and more). For details on auto-instrumentation, check out the [OpenInference](https://github.com/Arize-ai/openinference) project.
|
|
137
|
+
|
|
138
|
+
Phoenix runs practically anywhere, including your Jupyter notebook, local machine, containerized deployment, or in the cloud.
|
|
139
|
+
|
|
140
|
+
## Installation
|
|
141
|
+
|
|
142
|
+
Install Phoenix via `pip` or `conda` along with extra dependencies for running evals:
|
|
143
|
+
|
|
144
|
+
```shell
|
|
145
|
+
pip install 'arize-phoenix[evals]'
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
Phoenix container images are available via [Docker Hub](https://hub.docker.com/r/arizephoenix/phoenix) and can be deployed using Docker or Kubernetes.
|
|
149
|
+
|
|
150
|
+
## Community
|
|
151
|
+
|
|
152
|
+
Join our community to connect with thousands of AI builders.
|
|
153
|
+
|
|
154
|
+
- 🌍 Join our [Slack community](https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q).
|
|
155
|
+
- 💡 Ask questions and provide feedback in the _#phoenix-support_ channel.
|
|
156
|
+
- 🌟 Leave a star on our [GitHub](https://github.com/Arize-ai/phoenix).
|
|
157
|
+
- 🐞 Report bugs with [GitHub Issues](https://github.com/Arize-ai/phoenix/issues).
|
|
158
|
+
- 𝕏 Follow us on [𝕏](https://twitter.com/ArizePhoenix).
|
|
159
|
+
- 💌️ Sign up for our [mailing list](https://phoenix.arize.com/#updates).
|
|
160
|
+
- 🗺️ Check out our [roadmap](https://github.com/orgs/Arize-ai/projects/45) to see where we're heading next.
|
|
161
|
+
|
|
162
|
+
## Thanks
|
|
163
|
+
|
|
164
|
+
- [UMAP](https://github.com/lmcinnes/umap) For unlocking the ability to visualize and reason about embeddings
|
|
165
|
+
- [HDBSCAN](https://github.com/scikit-learn-contrib/hdbscan) For providing a clustering algorithm to aid in the discovery of drift and performance degradation
|
|
166
|
+
|
|
167
|
+
## Breaking Changes
|
|
168
|
+
|
|
169
|
+
See the [migration guide](./MIGRATION.md) for a list of breaking changes.
|
|
170
|
+
|
|
171
|
+
## Copyright, Patent, and License
|
|
172
|
+
|
|
173
|
+
Copyright 2023 Arize AI, Inc. All Rights Reserved.
|
|
174
|
+
|
|
175
|
+
Portions of this code are patent protected by one or more U.S. Patents. See [IP_NOTICE](https://github.com/Arize-ai/phoenix/blob/main/IP_NOTICE).
|
|
176
|
+
|
|
177
|
+
This software is licensed under the terms of the Elastic License 2.0 (ELv2). See [LICENSE](https://github.com/Arize-ai/phoenix/blob/main/LICENSE).
|
|
@@ -5,7 +5,7 @@ phoenix/exceptions.py,sha256=n2L2KKuecrdflB9MsCdAYCiSEvGJptIsfRkXMoJle7A,169
|
|
|
5
5
|
phoenix/py.typed,sha256=AbpHGcgLb-kRsJGnwFEktk7uzpZOCcBY74-YBdrKVGs,1
|
|
6
6
|
phoenix/services.py,sha256=haZUWKuxfI5lm0o61HgW_4B6RTBg9CUbXC8y92ZMsw4,5121
|
|
7
7
|
phoenix/settings.py,sha256=cO-qgis_S27nHirTobYI9hHPfZH18R--WMmxNdsVUwc,273
|
|
8
|
-
phoenix/version.py,sha256=
|
|
8
|
+
phoenix/version.py,sha256=AyLvA6hLK9mzQgiHyEbmvYQ9t6-oye0gm_KltlPS_oc,22
|
|
9
9
|
phoenix/core/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
|
10
10
|
phoenix/core/embedding_dimension.py,sha256=zKGbcvwOXgLf-yrJBpQyKtd-LEOPRKHnUToyAU8Owis,87
|
|
11
11
|
phoenix/core/model.py,sha256=SBO8BZg_CeQuH5LrSgzlQfqMModzirkQePdhnbLw7dE,4756
|
|
@@ -166,7 +166,7 @@ phoenix/trace/utils.py,sha256=7LurVGXn245cjj4MJsc7v6jq4DSJkpK6YGBfIaSywuw,1307
|
|
|
166
166
|
phoenix/trace/dsl/README.md,sha256=ihmP9zGUC5V-TDbzKla76LuyDqPDQIBUH2BORwxNI68,2902
|
|
167
167
|
phoenix/trace/dsl/__init__.py,sha256=WIQIjJg362XD3s50OsPJJ0xbDsGp41bSv7vDllLrPuA,144
|
|
168
168
|
phoenix/trace/dsl/filter.py,sha256=w4jHpzykdcpuk-3Z7diI6hXQjBKIuJ7vuS-bDIC_7Ck,31900
|
|
169
|
-
phoenix/trace/dsl/helpers.py,sha256=
|
|
169
|
+
phoenix/trace/dsl/helpers.py,sha256=ULAhqWULPqYWCSNX7y50DVKIqfySx86nqb6hDvZPnVk,3896
|
|
170
170
|
phoenix/trace/dsl/query.py,sha256=W0t-tiXh2WIVb96lzFAGQOQ-U46uKux78d4KL3rW-PE,30316
|
|
171
171
|
phoenix/trace/langchain/__init__.py,sha256=F37GfD1pd5Kuw7R7iRUM1zXXpO8xEcycNZh5dwqBXNk,109
|
|
172
172
|
phoenix/trace/langchain/instrumentor.py,sha256=isW3KRRP55t3JeaipAJ_aS7N_3HtwScPCO67ymSLr0Y,1372
|
|
@@ -183,8 +183,8 @@ phoenix/utilities/error_handling.py,sha256=7b5rpGFj9EWZ8yrZK1IHvxB89suWk3lggDayU
|
|
|
183
183
|
phoenix/utilities/logging.py,sha256=lDXd6EGaamBNcQxL4vP1au9-i_SXe0OraUDiJOcszSw,222
|
|
184
184
|
phoenix/utilities/project.py,sha256=qWsvKnG1oKhOFUowXf9qiOL2ia7jaFe_ijFFHEt8GJo,431
|
|
185
185
|
phoenix/utilities/span_store.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
|
186
|
-
arize_phoenix-4.1.
|
|
187
|
-
arize_phoenix-4.1.
|
|
188
|
-
arize_phoenix-4.1.
|
|
189
|
-
arize_phoenix-4.1.
|
|
190
|
-
arize_phoenix-4.1.
|
|
186
|
+
arize_phoenix-4.1.2.dist-info/METADATA,sha256=tzT6A4WMGsmhlKlHqZgkEoEMvv3DES0rG_lE4sy-aK8,11148
|
|
187
|
+
arize_phoenix-4.1.2.dist-info/WHEEL,sha256=zEMcRr9Kr03x1ozGwg5v9NQBKn3kndp6LSoSlVg-jhU,87
|
|
188
|
+
arize_phoenix-4.1.2.dist-info/licenses/IP_NOTICE,sha256=JBqyyCYYxGDfzQ0TtsQgjts41IJoa-hiwDrBjCb9gHM,469
|
|
189
|
+
arize_phoenix-4.1.2.dist-info/licenses/LICENSE,sha256=HFkW9REuMOkvKRACuwLPT0hRydHb3zNg-fdFt94td18,3794
|
|
190
|
+
arize_phoenix-4.1.2.dist-info/RECORD,,
|
phoenix/trace/dsl/helpers.py
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import warnings
|
|
2
2
|
from datetime import datetime
|
|
3
|
-
from typing import List, Optional, Protocol, Union, cast
|
|
3
|
+
from typing import List, Optional, Protocol, Tuple, Union, cast
|
|
4
4
|
|
|
5
5
|
import pandas as pd
|
|
6
6
|
from openinference.semconv.trace import DocumentAttributes, SpanAttributes
|
|
@@ -75,7 +75,7 @@ def get_qa_with_reference(
|
|
|
75
75
|
project_name: Optional[str] = None,
|
|
76
76
|
# Deprecated
|
|
77
77
|
stop_time: Optional[datetime] = None,
|
|
78
|
-
) -> pd.DataFrame:
|
|
78
|
+
) -> Optional[pd.DataFrame]:
|
|
79
79
|
project_name = project_name or get_env_project_name()
|
|
80
80
|
if stop_time:
|
|
81
81
|
# Deprecated. Raise a warning
|
|
@@ -84,23 +84,35 @@ def get_qa_with_reference(
|
|
|
84
84
|
DeprecationWarning,
|
|
85
85
|
)
|
|
86
86
|
end_time = end_time or stop_time
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
87
|
+
separator = "\n\n"
|
|
88
|
+
qa_query = SpanQuery().select("span_id", **IO).where(IS_ROOT).with_index("trace_id")
|
|
89
|
+
docs_query = (
|
|
90
|
+
SpanQuery()
|
|
91
|
+
.where(IS_RETRIEVER)
|
|
92
|
+
.concat(RETRIEVAL_DOCUMENTS, reference=DOCUMENT_CONTENT)
|
|
93
|
+
.with_concat_separator(separator=separator)
|
|
94
|
+
.with_index("trace_id")
|
|
95
|
+
)
|
|
96
|
+
df_qa, df_docs = cast(
|
|
97
|
+
Tuple[pd.DataFrame, pd.DataFrame],
|
|
98
|
+
obj.query_spans(
|
|
99
|
+
qa_query,
|
|
100
|
+
docs_query,
|
|
101
|
+
start_time=start_time,
|
|
102
|
+
end_time=end_time,
|
|
103
|
+
project_name=project_name,
|
|
103
104
|
),
|
|
104
|
-
axis=1,
|
|
105
|
-
join="inner",
|
|
106
105
|
)
|
|
106
|
+
if df_qa is None or df_qa.empty:
|
|
107
|
+
print("No spans found.")
|
|
108
|
+
return None
|
|
109
|
+
if df_docs is None or df_docs.empty:
|
|
110
|
+
print("No retrieval documents found.")
|
|
111
|
+
return None
|
|
112
|
+
# Consolidate duplicate rows via concatenation. This can happen if there are multiple
|
|
113
|
+
# retriever spans in the same trace. We simply concatenate all of them (in no particular
|
|
114
|
+
# order) into a single row.
|
|
115
|
+
ref = df_docs.groupby("context.trace_id")["reference"].apply(lambda x: separator.join(x))
|
|
116
|
+
df_ref = pd.DataFrame({"reference": ref})
|
|
117
|
+
df_qa_ref = pd.concat([df_qa, df_ref], axis=1, join="inner").set_index("context.span_id")
|
|
118
|
+
return df_qa_ref
|
phoenix/version.py
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
__version__ = "4.1.
|
|
1
|
+
__version__ = "4.1.2"
|
|
@@ -1,520 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.3
|
|
2
|
-
Name: arize-phoenix
|
|
3
|
-
Version: 4.1.1
|
|
4
|
-
Summary: AI Observability and Evaluation
|
|
5
|
-
Project-URL: Documentation, https://docs.arize.com/phoenix/
|
|
6
|
-
Project-URL: Issues, https://github.com/Arize-ai/phoenix/issues
|
|
7
|
-
Project-URL: Source, https://github.com/Arize-ai/phoenix
|
|
8
|
-
Author-email: Arize AI <phoenix-devs@arize.com>
|
|
9
|
-
License: Elastic-2.0
|
|
10
|
-
License-File: IP_NOTICE
|
|
11
|
-
License-File: LICENSE
|
|
12
|
-
Keywords: Explainability,Monitoring,Observability
|
|
13
|
-
Classifier: Programming Language :: Python
|
|
14
|
-
Classifier: Programming Language :: Python :: 3.8
|
|
15
|
-
Classifier: Programming Language :: Python :: 3.9
|
|
16
|
-
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
-
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
-
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
-
Requires-Python: <3.13,>=3.8
|
|
20
|
-
Requires-Dist: aioitertools
|
|
21
|
-
Requires-Dist: aiosqlite
|
|
22
|
-
Requires-Dist: alembic<2,>=1.3.0
|
|
23
|
-
Requires-Dist: cachetools
|
|
24
|
-
Requires-Dist: grpcio
|
|
25
|
-
Requires-Dist: hdbscan>=0.8.33
|
|
26
|
-
Requires-Dist: jinja2
|
|
27
|
-
Requires-Dist: numpy
|
|
28
|
-
Requires-Dist: openinference-instrumentation
|
|
29
|
-
Requires-Dist: openinference-instrumentation-langchain>=0.1.12
|
|
30
|
-
Requires-Dist: openinference-instrumentation-llama-index>=1.2.0
|
|
31
|
-
Requires-Dist: openinference-instrumentation-openai>=0.1.4
|
|
32
|
-
Requires-Dist: openinference-semantic-conventions>=0.1.5
|
|
33
|
-
Requires-Dist: opentelemetry-exporter-otlp
|
|
34
|
-
Requires-Dist: opentelemetry-proto>=1.12.0
|
|
35
|
-
Requires-Dist: opentelemetry-sdk
|
|
36
|
-
Requires-Dist: opentelemetry-semantic-conventions
|
|
37
|
-
Requires-Dist: pandas
|
|
38
|
-
Requires-Dist: protobuf<6.0,>=3.20
|
|
39
|
-
Requires-Dist: psutil
|
|
40
|
-
Requires-Dist: pyarrow
|
|
41
|
-
Requires-Dist: requests
|
|
42
|
-
Requires-Dist: scikit-learn
|
|
43
|
-
Requires-Dist: scipy
|
|
44
|
-
Requires-Dist: sqlalchemy[asyncio]<3,>=2.0.4
|
|
45
|
-
Requires-Dist: sqlean-py>=3.45.1
|
|
46
|
-
Requires-Dist: starlette
|
|
47
|
-
Requires-Dist: strawberry-graphql==0.227.2
|
|
48
|
-
Requires-Dist: tqdm
|
|
49
|
-
Requires-Dist: typing-extensions>=4.5; python_version < '3.12'
|
|
50
|
-
Requires-Dist: typing-extensions>=4.6; python_version >= '3.12'
|
|
51
|
-
Requires-Dist: umap-learn
|
|
52
|
-
Requires-Dist: uvicorn
|
|
53
|
-
Requires-Dist: wrapt
|
|
54
|
-
Provides-Extra: container
|
|
55
|
-
Requires-Dist: opentelemetry-exporter-otlp; extra == 'container'
|
|
56
|
-
Requires-Dist: opentelemetry-instrumentation-grpc; extra == 'container'
|
|
57
|
-
Requires-Dist: opentelemetry-instrumentation-sqlalchemy; extra == 'container'
|
|
58
|
-
Requires-Dist: opentelemetry-instrumentation-starlette; extra == 'container'
|
|
59
|
-
Requires-Dist: opentelemetry-proto>=1.12.0; extra == 'container'
|
|
60
|
-
Requires-Dist: opentelemetry-sdk; extra == 'container'
|
|
61
|
-
Requires-Dist: opentelemetry-semantic-conventions; extra == 'container'
|
|
62
|
-
Requires-Dist: prometheus-client; extra == 'container'
|
|
63
|
-
Requires-Dist: py-grpc-prometheus; extra == 'container'
|
|
64
|
-
Requires-Dist: strawberry-graphql[opentelemetry]==0.227.2; extra == 'container'
|
|
65
|
-
Requires-Dist: uvloop; (platform_system != 'Windows') and extra == 'container'
|
|
66
|
-
Provides-Extra: dev
|
|
67
|
-
Requires-Dist: anthropic; extra == 'dev'
|
|
68
|
-
Requires-Dist: arize[autoembeddings,llm-evaluation]; extra == 'dev'
|
|
69
|
-
Requires-Dist: asyncpg; extra == 'dev'
|
|
70
|
-
Requires-Dist: gcsfs; extra == 'dev'
|
|
71
|
-
Requires-Dist: google-cloud-aiplatform>=1.3; extra == 'dev'
|
|
72
|
-
Requires-Dist: hatch; extra == 'dev'
|
|
73
|
-
Requires-Dist: jupyter; extra == 'dev'
|
|
74
|
-
Requires-Dist: langchain>=0.0.334; extra == 'dev'
|
|
75
|
-
Requires-Dist: litellm>=1.0.3; extra == 'dev'
|
|
76
|
-
Requires-Dist: llama-index>=0.10.3; extra == 'dev'
|
|
77
|
-
Requires-Dist: nbqa; extra == 'dev'
|
|
78
|
-
Requires-Dist: pandas-stubs<=2.0.2.230605; extra == 'dev'
|
|
79
|
-
Requires-Dist: pre-commit; extra == 'dev'
|
|
80
|
-
Requires-Dist: prometheus-client; extra == 'dev'
|
|
81
|
-
Requires-Dist: psycopg[binary]; extra == 'dev'
|
|
82
|
-
Requires-Dist: pytest-asyncio; extra == 'dev'
|
|
83
|
-
Requires-Dist: pytest-cov; extra == 'dev'
|
|
84
|
-
Requires-Dist: pytest-lazy-fixture; extra == 'dev'
|
|
85
|
-
Requires-Dist: pytest-postgresql; extra == 'dev'
|
|
86
|
-
Requires-Dist: pytest==7.4.4; extra == 'dev'
|
|
87
|
-
Requires-Dist: ruff==0.3.0; extra == 'dev'
|
|
88
|
-
Requires-Dist: strawberry-graphql[debug-server,opentelemetry]==0.227.2; extra == 'dev'
|
|
89
|
-
Provides-Extra: evals
|
|
90
|
-
Requires-Dist: arize-phoenix-evals>=0.3.0; extra == 'evals'
|
|
91
|
-
Provides-Extra: experimental
|
|
92
|
-
Requires-Dist: tenacity; extra == 'experimental'
|
|
93
|
-
Provides-Extra: llama-index
|
|
94
|
-
Requires-Dist: llama-index-callbacks-arize-phoenix>=0.1.2; extra == 'llama-index'
|
|
95
|
-
Requires-Dist: llama-index==0.10.3; extra == 'llama-index'
|
|
96
|
-
Requires-Dist: openinference-instrumentation-llama-index>=1.2.0; extra == 'llama-index'
|
|
97
|
-
Provides-Extra: pg
|
|
98
|
-
Requires-Dist: asyncpg; extra == 'pg'
|
|
99
|
-
Description-Content-Type: text/markdown
|
|
100
|
-
|
|
101
|
-
<p align="center">
|
|
102
|
-
<a target="_blank" href="https://phoenix.arize.com" style="background:none">
|
|
103
|
-
<img alt="phoenix logo" src="https://storage.googleapis.com/arize-assets/phoenix/assets/phoenix-logo-light.svg" width="auto" height="200"></img>
|
|
104
|
-
</a>
|
|
105
|
-
<br/>
|
|
106
|
-
<br/>
|
|
107
|
-
<a href="https://docs.arize.com/phoenix/">
|
|
108
|
-
<img src="https://img.shields.io/static/v1?message=Docs&logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAIAAAACACAYAAADDPmHLAAAG4ElEQVR4nO2d4XHjNhCFcTf+b3ZgdWCmgmMqOKUC0xXYrsBOBVEqsFRB7ApCVRCygrMriFQBM7h5mNlwKBECARLg7jeDscamSQj7sFgsQfBL27ZK4MtXsT1vRADMEQEwRwTAHBEAc0QAzBEBMEcEwBwRAHNEAMwRATBnjAByFGE+MqVUMcYOY24GVUqpb/h8VErVKAf87QNFcEcbd4WSw+D6803njHscO5sATmGEURGBiCj6yUlv1uX2gv91FsDViArbcA2RUKF8QhAV8RQc0b15DcOt0VaTE1oAfWj3dYdCBfGGsmSM0XX5HsP3nEMAXbqCeCdiOERQPx9og5exGJ0S4zRQN9KrUupfpdQWjZciure/YIj7K0bjqwTyAHdovA805iqCOg2xgnB1nZ97IvaoSCURdIPG/IHGjTH/YAz/A8KdJai7lBQzgbpx/0Hg6DT18UzWMXxSjMkDrElPNEmKfAbl6znwI3IMU/OCa0/1nfckwWaSbvWYYDnEsvCMJDNckhqu7GCMKWYOBXp9yPGd5kvqUAKf6rkAk7M2SY9QDXdEr9wEOr9x96EiejMFnixBNteDISsyNw7hHRqc22evWcP4vt39O85bzZH30AKg4+eo8cQRI4bHAJ7hyYM3CNHrG9RrimSXuZmUkZjN/O6nAPpcwCcJNmipAle2QM/1GU3vITCXhvY91u9geN/jOY27VuTnYL1PCeAcRhwh7/Bl8Ai+IuxPiOCShtfX/sPDtY8w+sZjby86dw6dBeoigD7obd/Ko6fI4BF8DA9HnGdrcU0fLt+n4dfE6H5jpjYcVdu2L23b5lpjHoo+18FDbcszddF1rUee/4C6ZiO+80rHZmjDoIQUQLdRtm3brkcKIUPjjqVPBIUHgW1GGN4YfawAL2IqAVB8iEE31tvIelARlCPPVaFOLoIupzY6xVcM4MoRUyHXyHhslH6PaPl5RP1Lh4UsOeKR2e8dzC0Aiuvc2Nx3fwhfxf/hknouUYbWUk5GTAIwmOh5e+H0cor8vEL91hfOdEqINLq1AV+RKImJ6869f9tFIBVc6y7gd3lHfWyNX0LEr7EuDElhRdAlQjig0e/RU31xxDltM4pF7IY3pLIgxAhhgzF/iC2M0Hi4dkOGlyGMd/g7dsMbUlsR9ICe9WhxbA3DjRkSdjiHzQzlBSKNJsCzIcUlYdfI0dcWS8LMkPDkcJ0n/O+Qyy/IAtDkSPnp4Fu4WpthQR/zm2VcoI/51fI28iYld9/HEh4Pf7D0Bm845pwIPnHMUJSf45pT5x68s5T9AW6INzhHDeP1BYcNMew5SghkinWOwVnaBhHGG5ybMn70zBDe8buh8X6DqV0Sa/5tWOIOIbcWQ8KBiGBnMb/P0OuTd/lddCrY5jn/VLm3nL+fY4X4YREuv8vS9wh6HSkAExMs0viKySZRd44iyOH2FzPe98Fll7A7GNMmjay4GF9BAKGXesfCN0sRsDG+YrhP4O2ACFgZXzHdKPL2RMJoxc34ivFOod3AMMNUj5XxFfOtYrUIXvB5MandS+G+V/AzZ+MrEcBPlpoFtUIEwBwRAG+OIgDe1CIA5ogAmCMCYI4IgDkiAOaIAJgjAmCOCIA5IgDmiACYIwJgjgiAOSIA5ogAmCMCYI4IgDkiAOaIAJgjAmCOCIA5IgDmiACYIwJgjgiAOSIA5ogAmCMCYI4IgDkiAOaIAJgjAmDOVYBXvwvxQV8NWJOd0esvJ94babZaz7B5ovldxnlDpYhp0JFr/KTlLKcEMMQKpcDPXIQxGXsYmhZnXAXQh/EWBQrr3bc80mATyyrEvs4+BdBHgbdxFOIhrDkSg1/6Iu2LCS0AyoqI4ftUF00EY/Q3h1fRj2JKAVCMGErmnsH1lfnemEsAlByvgl0z2qx5B8OPCuB8EIMADBlEEOV79j1whNE3c/X2PmISAGUNr7CEmUSUhjfEKgBDAY+QohCiNrwhdgEYzPv7UxkadvBg0RrekMrNoAozh3vLN4DPhc7S/WL52vkoSO1u4BZC+DOCulC0KJ/gqWaP7C8hlSGgjxyCmDuPsEePT/KuasrrAcyr4H+f6fq01yd7Sz1lD0CZ2hs06PVJufs+lrIiyLwufjfBtXYpjvWnWIoHoJSYe4dIK/t4HX1ULFEACkPCm8e8wXFJvZ6y1EWhJkDcWxw7RINzLc74auGrgg8e4oIm9Sh/CA7LwkvHqaIJ9pLI6Lmy1BigDy2EV8tjdzh+8XB6MGSLKH4INsZXDJ8MGhIBK+Mrpo+GnRIBO+MrZjFAFxoTNBwCvj6u4qvSZJiM3iNX4yvmHoA9Sh4PF0QAzBEBMEcEwBwRAHNEAMwRAXBGKfUfr5hKvglRfO4AAAAASUVORK5CYII=&labelColor=grey&color=blue&logoColor=white&label=%20"/>
|
|
109
|
-
</a>
|
|
110
|
-
<a target="_blank" href="https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q">
|
|
111
|
-
<img src="https://img.shields.io/static/v1?message=Community&logo=slack&labelColor=grey&color=blue&logoColor=white&label=%20"/>
|
|
112
|
-
</a>
|
|
113
|
-
<a target="_blank" href="https://twitter.com/ArizePhoenix">
|
|
114
|
-
<img src="https://img.shields.io/badge/-ArizePhoenix-blue.svg?color=blue&labelColor=gray&logo=twitter">
|
|
115
|
-
</a>
|
|
116
|
-
<a target="_blank" href="https://pypi.org/project/arize-phoenix/">
|
|
117
|
-
<img src="https://img.shields.io/pypi/v/arize-phoenix?color=blue">
|
|
118
|
-
</a>
|
|
119
|
-
<a target="_blank" href="https://anaconda.org/conda-forge/arize-phoenix">
|
|
120
|
-
<img src="https://img.shields.io/conda/vn/conda-forge/arize-phoenix.svg?color=blue">
|
|
121
|
-
</a>
|
|
122
|
-
<a target="_blank" href="https://pypi.org/project/arize-phoenix/">
|
|
123
|
-
<img src="https://img.shields.io/pypi/pyversions/arize-phoenix">
|
|
124
|
-
</a>
|
|
125
|
-
<a target="_blank" href="https://hub.docker.com/r/arizephoenix/phoenix/tags">
|
|
126
|
-
<img src="https://img.shields.io/docker/v/arizephoenix/phoenix?sort=semver&logo=docker&label=image&color=blue">
|
|
127
|
-
</a>
|
|
128
|
-
</p>
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
Phoenix provides MLOps and LLMOps insights at lightning speed with zero-config observability. Phoenix provides a notebook-first experience for monitoring your models and LLM Applications by providing:
|
|
133
|
-
|
|
134
|
-
- **LLM Traces** - Trace through the execution of your LLM Application to understand the internals of your LLM Application and to troubleshoot problems related to things like retrieval and tool execution.
|
|
135
|
-
- **LLM Evals** - Leverage the power of large language models to evaluate your generative model or application's relevance, toxicity, and more.
|
|
136
|
-
- **Embedding Analysis** - Explore embedding point-clouds and identify clusters of high drift and performance degradation.
|
|
137
|
-
- **RAG Analysis** - Visualize your generative application's search and retrieval process to identify problems and improve your RAG pipeline.
|
|
138
|
-
- **Structured Data Analysis** - Statistically analyze your structured data by performing A/B analysis, temporal drift analysis, and more.
|
|
139
|
-
|
|
140
|
-
**Table of Contents**
|
|
141
|
-
|
|
142
|
-
- [Installation](#installation)
|
|
143
|
-
- [LLM Traces](#llm-traces)
|
|
144
|
-
- [Tracing with LlamaIndex](#tracing-with-llamaindex)
|
|
145
|
-
- [Tracing with LangChain](#tracing-with-langchain)
|
|
146
|
-
- [LLM Evals](#llm-evals)
|
|
147
|
-
- [Embedding Analysis](#embedding-analysis)
|
|
148
|
-
- [UMAP-based Exploratory Data Analysis](#umap-based-exploratory-data-analysis)
|
|
149
|
-
- [Cluster-driven Drift and Performance Analysis](#cluster-driven-drift-and-performance-analysis)
|
|
150
|
-
- [Exportable Clusters](#exportable-clusters)
|
|
151
|
-
- [Retrieval-Augmented Generation Analysis](#retrieval-augmented-generation-analysis)
|
|
152
|
-
- [Structured Data Analysis](#structured-data-analysis)
|
|
153
|
-
- [Deploying Phoenix](#deploying-phoenix)
|
|
154
|
-
- [Breaking Changes](#breaking-changes)
|
|
155
|
-
- [Community](#community)
|
|
156
|
-
- [Thanks](#thanks)
|
|
157
|
-
- [Copyright, Patent, and License](#copyright-patent-and-license)
|
|
158
|
-
|
|
159
|
-
## Installation
|
|
160
|
-
|
|
161
|
-
Install Phoenix via `pip` or or `conda` as well as any of its subpackages.
|
|
162
|
-
|
|
163
|
-
```shell
|
|
164
|
-
pip install arize-phoenix[evals]
|
|
165
|
-
```
|
|
166
|
-
|
|
167
|
-
> [!NOTE]
|
|
168
|
-
> The above will install Phoenix and its `evals` subpackage. To just install phoenix's evaluation package, you can run `pip install arize-phoenix-evals` instead.
|
|
169
|
-
|
|
170
|
-
## LLM Traces
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
With the advent of powerful LLMs, it is now possible to build LLM Applications that can perform complex tasks like summarization, translation, question and answering, and more. However, these applications are often difficult to debug and troubleshoot as they have an extensive surface area: search and retrieval via vector stores, embedding generation, usage of external tools and so on. Phoenix provides a tracing framework that allows you to trace through the execution of your LLM Application hierarchically. This allows you to understand the internals of your LLM Application and to troubleshoot the complex components of your applicaition. Phoenix is built on top of the OpenInference tracing standard and uses it to trace, export, and collect critical information about your LLM Application in the form of `spans`. For more details on the OpenInference tracing standard, see the [OpenInference Specification](https://github.com/Arize-ai/openinference)
|
|
175
|
-
|
|
176
|
-
### Tracing with LlamaIndex
|
|
177
|
-
|
|
178
|
-
[](https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/tracing/llama_index_tracing_tutorial.ipynb) [](https://github.com/Arize-ai/phoenix/blob/main/tutorials/tracing/llama_index_tracing_tutorial.ipynb)
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
To extract traces from your LlamaIndex application, you will have to add Phoenix's `OpenInferenceTraceCallback` to your LlamaIndex application. A callback (in this case an OpenInference `Tracer`) is a class that automatically accumulates `spans` that trac your application as it executes. The OpenInference `Tracer` is a tracer that is specifically designed to work with Phoenix and by default exports the traces to a locally running phoenix server.
|
|
183
|
-
|
|
184
|
-
```shell
|
|
185
|
-
# Install phoenix as well as llama_index and your LLM of choice
|
|
186
|
-
pip install "arize-phoenix[evals]" "openai>=1" "llama-index>=0.10.3" "openinference-instrumentation-llama-index>=1.0.0" "llama-index-callbacks-arize-phoenix>=0.1.2" llama-index-llms-openai
|
|
187
|
-
```
|
|
188
|
-
|
|
189
|
-
Launch Phoenix in a notebook and view the traces of your LlamaIndex application in the Phoenix UI.
|
|
190
|
-
|
|
191
|
-
```python
|
|
192
|
-
import os
|
|
193
|
-
import phoenix as px
|
|
194
|
-
from llama_index.core import (
|
|
195
|
-
Settings,
|
|
196
|
-
VectorStoreIndex,
|
|
197
|
-
SimpleDirectoryReader,
|
|
198
|
-
set_global_handler,
|
|
199
|
-
)
|
|
200
|
-
from llama_index.embeddings.openai import OpenAIEmbedding
|
|
201
|
-
from llama_index.llms.openai import OpenAI
|
|
202
|
-
|
|
203
|
-
os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY"
|
|
204
|
-
|
|
205
|
-
# To view traces in Phoenix, you will first have to start a Phoenix server. You can do this by running the following:
|
|
206
|
-
session = px.launch_app()
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
# Once you have started a Phoenix server, you can start your LlamaIndex application and configure it to send traces to Phoenix. To do this, you will have to add configure Phoenix as the global handler
|
|
210
|
-
|
|
211
|
-
set_global_handler("arize_phoenix")
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
# LlamaIndex application initialization may vary
|
|
215
|
-
# depending on your application
|
|
216
|
-
Settings.llm = OpenAI(model="gpt-4-turbo-preview")
|
|
217
|
-
Settings.embed_model = OpenAIEmbedding(model="text-embedding-ada-002")
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
# Load your data and create an index. Note you usually want to store your index in a persistent store like a database or the file system
|
|
221
|
-
documents = SimpleDirectoryReader("YOUR_DATA_DIRECTORY").load_data()
|
|
222
|
-
index = VectorStoreIndex.from_documents(documents)
|
|
223
|
-
|
|
224
|
-
query_engine = index.as_query_engine()
|
|
225
|
-
|
|
226
|
-
# Query your LlamaIndex application
|
|
227
|
-
query_engine.query("What is the meaning of life?")
|
|
228
|
-
query_engine.query("Why did the cow jump over the moon?")
|
|
229
|
-
|
|
230
|
-
# View the traces in the Phoenix UI
|
|
231
|
-
px.active_session().url
|
|
232
|
-
```
|
|
233
|
-
|
|
234
|
-
### Tracing with LangChain
|
|
235
|
-
|
|
236
|
-
[](https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/tracing/langchain_tracing_tutorial.ipynb) [](https://github.com/Arize-ai/phoenix/blob/main/tutorials/tracing/langchain_tracing_tutorial.ipynb)
|
|
237
|
-
|
|
238
|
-
To extract traces from your LangChain application, you will have to add Phoenix's OpenInference Tracer to your LangChain application. A tracer is a class that automatically accumulates traces as your application executes. The OpenInference Tracer is a tracer that is specifically designed to work with Phoenix and by default exports the traces to a locally running phoenix server.
|
|
239
|
-
|
|
240
|
-
```shell
|
|
241
|
-
# Install phoenix as well as langchain and your LLM of choice
|
|
242
|
-
pip install arize-phoenix langchain openai
|
|
243
|
-
|
|
244
|
-
```
|
|
245
|
-
|
|
246
|
-
Launch Phoenix in a notebook and view the traces of your LangChain application in the Phoenix UI.
|
|
247
|
-
|
|
248
|
-
```python
|
|
249
|
-
import phoenix as px
|
|
250
|
-
import pandas as pd
|
|
251
|
-
import numpy as np
|
|
252
|
-
|
|
253
|
-
# Launch phoenix
|
|
254
|
-
session = px.launch_app()
|
|
255
|
-
|
|
256
|
-
# Once you have started a Phoenix server, you can start your LangChain application with the OpenInferenceTracer as a callback. To do this, you will have to instrument your LangChain application with the tracer:
|
|
257
|
-
|
|
258
|
-
from phoenix.trace.langchain import LangChainInstrumentor
|
|
259
|
-
|
|
260
|
-
# By default, the traces will be exported to the locally running Phoenix server.
|
|
261
|
-
LangChainInstrumentor().instrument()
|
|
262
|
-
|
|
263
|
-
# Initialize your LangChain application
|
|
264
|
-
from langchain.chains import RetrievalQA
|
|
265
|
-
from langchain.chat_models import ChatOpenAI
|
|
266
|
-
from langchain.embeddings import OpenAIEmbeddings
|
|
267
|
-
from langchain.retrievers import KNNRetriever
|
|
268
|
-
|
|
269
|
-
embeddings = OpenAIEmbeddings(model="text-embedding-ada-002")
|
|
270
|
-
documents_df = pd.read_parquet(
|
|
271
|
-
"http://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/llm/context-retrieval/langchain-pinecone/database.parquet"
|
|
272
|
-
)
|
|
273
|
-
knn_retriever = KNNRetriever(
|
|
274
|
-
index=np.stack(documents_df["text_vector"]),
|
|
275
|
-
texts=documents_df["text"].tolist(),
|
|
276
|
-
embeddings=OpenAIEmbeddings(),
|
|
277
|
-
)
|
|
278
|
-
chain_type = "stuff" # stuff, refine, map_reduce, and map_rerank
|
|
279
|
-
chat_model_name = "gpt-3.5-turbo"
|
|
280
|
-
llm = ChatOpenAI(model_name=chat_model_name)
|
|
281
|
-
chain = RetrievalQA.from_chain_type(
|
|
282
|
-
llm=llm,
|
|
283
|
-
chain_type=chain_type,
|
|
284
|
-
retriever=knn_retriever,
|
|
285
|
-
)
|
|
286
|
-
|
|
287
|
-
# Instrument the execution of the runs with the tracer. By default the tracer uses an HTTPExporter
|
|
288
|
-
query = "What is euclidean distance?"
|
|
289
|
-
response = chain.run(query, callbacks=[tracer])
|
|
290
|
-
|
|
291
|
-
# By adding the tracer to the callbacks of LangChain, we've created a one-way data connection between your LLM application and Phoenix.
|
|
292
|
-
|
|
293
|
-
# To view the traces in Phoenix, simply open the UI in your browser.
|
|
294
|
-
session.url
|
|
295
|
-
```
|
|
296
|
-
|
|
297
|
-
## LLM Evals
|
|
298
|
-
|
|
299
|
-
[](https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/evals/evaluate_relevance_classifications.ipynb) [](https://github.com/Arize-ai/phoenix/blob/main/tutorials/evals/evaluate_relevance_classifications.ipynb)
|
|
300
|
-
|
|
301
|
-
Phoenix provides tooling to evaluate LLM applications, including tools to determine the relevance or irrelevance of documents retrieved by retrieval-augmented generation (RAG) application, whether or not the response is toxic, and much more.
|
|
302
|
-
|
|
303
|
-
Phoenix's approach to LLM evals is notable for the following reasons:
|
|
304
|
-
|
|
305
|
-
- Includes pre-tested templates and convenience functions for a set of common Eval “tasks”
|
|
306
|
-
- Data science rigor applied to the testing of model and template combinations
|
|
307
|
-
- Designed to run as fast as possible on batches of data
|
|
308
|
-
- Includes benchmark datasets and tests for each eval function
|
|
309
|
-
|
|
310
|
-
Here is an example of running the RAG relevance eval on a dataset of Wikipedia questions and answers:
|
|
311
|
-
|
|
312
|
-
```shell
|
|
313
|
-
# Install phoenix as well as the evals subpackage
|
|
314
|
-
pip install 'arize-phoenix[evals]' ipython matplotlib openai pycm scikit-learn
|
|
315
|
-
```
|
|
316
|
-
|
|
317
|
-
```python
|
|
318
|
-
from phoenix.evals import (
|
|
319
|
-
RAG_RELEVANCY_PROMPT_TEMPLATE,
|
|
320
|
-
RAG_RELEVANCY_PROMPT_RAILS_MAP,
|
|
321
|
-
OpenAIModel,
|
|
322
|
-
download_benchmark_dataset,
|
|
323
|
-
llm_classify,
|
|
324
|
-
)
|
|
325
|
-
from sklearn.metrics import precision_recall_fscore_support, confusion_matrix, ConfusionMatrixDisplay
|
|
326
|
-
|
|
327
|
-
# Download the benchmark golden dataset
|
|
328
|
-
df = download_benchmark_dataset(
|
|
329
|
-
task="binary-relevance-classification", dataset_name="wiki_qa-train"
|
|
330
|
-
)
|
|
331
|
-
# Sample and re-name the columns to match the template
|
|
332
|
-
df = df.sample(100)
|
|
333
|
-
df = df.rename(
|
|
334
|
-
columns={
|
|
335
|
-
"query_text": "input",
|
|
336
|
-
"document_text": "reference",
|
|
337
|
-
},
|
|
338
|
-
)
|
|
339
|
-
model = OpenAIModel(
|
|
340
|
-
model="gpt-4",
|
|
341
|
-
temperature=0.0,
|
|
342
|
-
)
|
|
343
|
-
rails =list(RAG_RELEVANCY_PROMPT_RAILS_MAP.values())
|
|
344
|
-
df[["eval_relevance"]] = llm_classify(df, model, RAG_RELEVANCY_PROMPT_TEMPLATE, rails)
|
|
345
|
-
#Golden dataset has True/False map to -> "irrelevant" / "relevant"
|
|
346
|
-
#we can then scikit compare to output of template - same format
|
|
347
|
-
y_true = df["relevant"].map({True: "relevant", False: "irrelevant"})
|
|
348
|
-
y_pred = df["eval_relevance"]
|
|
349
|
-
|
|
350
|
-
# Compute Per-Class Precision, Recall, F1 Score, Support
|
|
351
|
-
precision, recall, f1, support = precision_recall_fscore_support(y_true, y_pred)
|
|
352
|
-
```
|
|
353
|
-
|
|
354
|
-
To learn more about LLM Evals, see the [Evals documentation](https://docs.arize.com/phoenix/concepts/llm-evals/).
|
|
355
|
-
|
|
356
|
-
## Embedding Analysis
|
|
357
|
-
|
|
358
|
-
[](https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/image_classification_tutorial.ipynb) [](https://github.com/Arize-ai/phoenix/blob/main/tutorials/image_classification_tutorial.ipynb)
|
|
359
|
-
|
|
360
|
-
Explore UMAP point-clouds at times of high drift and performance degredation and identify clusters of problematic data.
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
|
|
364
|
-
Embedding analysis is critical for understanding the behavior of you NLP, CV, and LLM Apps that use embeddings. Phoenix provides an A/B testing framework to help you understand how your embeddings are changing over time and how they are changing between different versions of your model (`prod` vs `train`, `champion` vs `challenger`).
|
|
365
|
-
|
|
366
|
-
```python
|
|
367
|
-
# Import libraries.
|
|
368
|
-
from dataclasses import replace
|
|
369
|
-
import pandas as pd
|
|
370
|
-
import phoenix as px
|
|
371
|
-
|
|
372
|
-
# Download curated datasets and load them into pandas DataFrames.
|
|
373
|
-
train_df = pd.read_parquet(
|
|
374
|
-
"https://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/cv/human-actions/human_actions_training.parquet"
|
|
375
|
-
)
|
|
376
|
-
prod_df = pd.read_parquet(
|
|
377
|
-
"https://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/cv/human-actions/human_actions_production.parquet"
|
|
378
|
-
)
|
|
379
|
-
|
|
380
|
-
# Define schemas that tell Phoenix which columns of your DataFrames correspond to features, predictions, actuals (i.e., ground truth), embeddings, etc.
|
|
381
|
-
train_schema = px.Schema(
|
|
382
|
-
prediction_id_column_name="prediction_id",
|
|
383
|
-
timestamp_column_name="prediction_ts",
|
|
384
|
-
prediction_label_column_name="predicted_action",
|
|
385
|
-
actual_label_column_name="actual_action",
|
|
386
|
-
embedding_feature_column_names={
|
|
387
|
-
"image_embedding": px.EmbeddingColumnNames(
|
|
388
|
-
vector_column_name="image_vector",
|
|
389
|
-
link_to_data_column_name="url",
|
|
390
|
-
),
|
|
391
|
-
},
|
|
392
|
-
)
|
|
393
|
-
prod_schema = replace(train_schema, actual_label_column_name=None)
|
|
394
|
-
|
|
395
|
-
# Define your production and training datasets.
|
|
396
|
-
prod_inf = px.Inferences(prod_df, prod_schema)
|
|
397
|
-
train_inf = px.Inferences(train_df, train_schema)
|
|
398
|
-
|
|
399
|
-
# Launch Phoenix.
|
|
400
|
-
session = px.launch_app(prod_inf, train_inf)
|
|
401
|
-
|
|
402
|
-
# View the Phoenix UI in the browser
|
|
403
|
-
session.url
|
|
404
|
-
```
|
|
405
|
-
|
|
406
|
-
### UMAP-based Exploratory Data Analysis
|
|
407
|
-
|
|
408
|
-
Color your UMAP point-clouds by your model's dimensions, drift, and performance to identify problematic cohorts.
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
### Cluster-driven Drift and Performance Analysis
|
|
413
|
-
|
|
414
|
-
Break-apart your data into clusters of high drift or bad performance using HDBSCAN
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
### Exportable Clusters
|
|
419
|
-
|
|
420
|
-
Export your clusters to `parquet` files or dataframes for further analysis and fine-tuning.
|
|
421
|
-
|
|
422
|
-
## Retrieval-Augmented Generation Analysis
|
|
423
|
-
|
|
424
|
-
[](https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/llama_index_search_and_retrieval_tutorial.ipynb) [](https://github.com/Arize-ai/phoenix/blob/main/tutorials/llama_index_search_and_retrieval_tutorial.ipynb)
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
Search and retrieval is a critical component of many LLM Applications as it allows you to extend the LLM's capabilities to encompass knowledge about private data. This process is known as RAG (retrieval-augmented generation) and often times a vector store is leveraged to store chunks of documents encoded as embeddings so that they can be retrieved at inference time.
|
|
429
|
-
|
|
430
|
-
To help you better understand your RAG application, Phoenix allows you to upload a corpus of your knowledge base along with your LLM application's inferences to help you troubleshoot hard to find bugs with retrieval.
|
|
431
|
-
|
|
432
|
-
## Structured Data Analysis
|
|
433
|
-
|
|
434
|
-
[](https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/credit_card_fraud_tutorial.ipynb) [](https://github.com/Arize-ai/phoenix/blob/main/tutorials/credit_card_fraud_tutorial.ipynb)
|
|
435
|
-
|
|
436
|
-
Phoenix provides a suite of tools for analyzing structured data. These tools allow you to perform A/B analysis, temporal drift analysis, and more.
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
```python
|
|
441
|
-
import pandas as pd
|
|
442
|
-
import phoenix as px
|
|
443
|
-
|
|
444
|
-
# Perform A/B analysis on your training and production datasets
|
|
445
|
-
train_df = pd.read_parquet(
|
|
446
|
-
"http://storage.googleapis.com/arize-assets/phoenix/datasets/structured/credit-card-fraud/credit_card_fraud_train.parquet",
|
|
447
|
-
)
|
|
448
|
-
prod_df = pd.read_parquet(
|
|
449
|
-
"http://storage.googleapis.com/arize-assets/phoenix/datasets/structured/credit-card-fraud/credit_card_fraud_production.parquet",
|
|
450
|
-
)
|
|
451
|
-
|
|
452
|
-
# Describe the data for analysis
|
|
453
|
-
schema = px.Schema(
|
|
454
|
-
prediction_id_column_name="prediction_id",
|
|
455
|
-
prediction_label_column_name="predicted_label",
|
|
456
|
-
prediction_score_column_name="predicted_score",
|
|
457
|
-
actual_label_column_name="actual_label",
|
|
458
|
-
timestamp_column_name="prediction_timestamp",
|
|
459
|
-
feature_column_names=feature_column_names,
|
|
460
|
-
tag_column_names=["age"],
|
|
461
|
-
)
|
|
462
|
-
|
|
463
|
-
# Define your production and training datasets.
|
|
464
|
-
prod_inf = px.Inferences(dataframe=prod_df, schema=schema, name="production")
|
|
465
|
-
train_inf = px.Inferences(dataframe=train_df, schema=schema, name="training")
|
|
466
|
-
|
|
467
|
-
# Launch Phoenix for analysis
|
|
468
|
-
session = px.launch_app(primary=prod_inf, reference=train_inf)
|
|
469
|
-
```
|
|
470
|
-
|
|
471
|
-
## Deploying Phoenix
|
|
472
|
-
|
|
473
|
-
<a target="_blank" href="https://hub.docker.com/repository/docker/arizephoenix/phoenix/general">
|
|
474
|
-
<img src="https://img.shields.io/docker/v/arizephoenix/phoenix?sort=semver&logo=docker&label=image&color=blue">
|
|
475
|
-
</a>
|
|
476
|
-
|
|
477
|
-
<img src="https://storage.googleapis.com/arize-assets/phoenix/assets/images/deployment.png" title="How phoenix can collect traces from an LLM application"/>
|
|
478
|
-
|
|
479
|
-
Phoenix's notebook-first approach to observability makes it a great tool to utilize during experimentation and pre-production. However at some point you are going to want to ship your application to production and continue to monitor your application as it runs. Phoenix is made up of two components that can be deployed independently:
|
|
480
|
-
|
|
481
|
-
- **Trace Instrumentation**: These are a set of plugins that can be added to your application's startup process. These plugins (known as instrumentations) automatically collect spans for your application and export them for collection and visualization. For phoenix, all the instrumentors are managed via a single repository called [OpenInference](https://github.com/Arize-ai/openinference)
|
|
482
|
-
- **Trace Collector**: The Phoenix server acts as a trace collector and application that helps you troubleshoot your application in real time. You can pull the latest images of Phoenix from the [Docker Hub](https://hub.docker.com/repository/docker/arizephoenix/phoenix/general)
|
|
483
|
-
|
|
484
|
-
In order to run Phoenix tracing in production, you will have to follow these following steps:
|
|
485
|
-
|
|
486
|
-
- **Setup a Server**: your LLM application to run on a server ([examples](https://github.com/Arize-ai/openinference/tree/main/python/examples))
|
|
487
|
-
- **Instrument**: Add [OpenInference](https://github.com/Arize-ai/openinference) Instrumentation to your server
|
|
488
|
-
- **Observe**: Run the Phoenix server as a side-car or a standalone instance and point your tracing instrumentation to the phoenix server
|
|
489
|
-
|
|
490
|
-
For more information on deploying Phoenix, see the [Phoenix Deployment Guide](https://docs.arize.com/phoenix/deployment/deploying-phoenix).
|
|
491
|
-
|
|
492
|
-
## Breaking Changes
|
|
493
|
-
|
|
494
|
-
see the [migration guide](./MIGRATION.md) for a list of breaking changes.
|
|
495
|
-
|
|
496
|
-
## Community
|
|
497
|
-
|
|
498
|
-
Join our community to connect with thousands of machine learning practitioners and ML observability enthusiasts.
|
|
499
|
-
|
|
500
|
-
- 🌍 Join our [Slack community](https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q).
|
|
501
|
-
- 💡 Ask questions and provide feedback in the _#phoenix-support_ channel.
|
|
502
|
-
- 🌟 Leave a star on our [GitHub](https://github.com/Arize-ai/phoenix).
|
|
503
|
-
- 🐞 Report bugs with [GitHub Issues](https://github.com/Arize-ai/phoenix/issues).
|
|
504
|
-
- 🐣 Follow us on [twitter](https://twitter.com/ArizePhoenix).
|
|
505
|
-
- 💌️ Sign up for our [mailing list](https://phoenix.arize.com/#updates).
|
|
506
|
-
- 🗺️ Check out our [roadmap](https://github.com/orgs/Arize-ai/projects/45) to see where we're heading next.
|
|
507
|
-
- 🎓 Learn the fundamentals of ML observability with our [introductory](https://arize.com/ml-observability-fundamentals/) and [advanced](https://arize.com/blog-course/) courses.
|
|
508
|
-
|
|
509
|
-
## Thanks
|
|
510
|
-
|
|
511
|
-
- [UMAP](https://github.com/lmcinnes/umap) For unlocking the ability to visualize and reason about embeddings
|
|
512
|
-
- [HDBSCAN](https://github.com/scikit-learn-contrib/hdbscan) For providing a clustering algorithm to aid in the discovery of drift and performance degradation
|
|
513
|
-
|
|
514
|
-
## Copyright, Patent, and License
|
|
515
|
-
|
|
516
|
-
Copyright 2023 Arize AI, Inc. All Rights Reserved.
|
|
517
|
-
|
|
518
|
-
Portions of this code are patent protected by one or more U.S. Patents. See [IP_NOTICE](https://github.com/Arize-ai/phoenix/blob/main/IP_NOTICE).
|
|
519
|
-
|
|
520
|
-
This software is licensed under the terms of the Elastic License 2.0 (ELv2). See [LICENSE](https://github.com/Arize-ai/phoenix/blob/main/LICENSE).
|
|
File without changes
|
|
File without changes
|
|
File without changes
|