eth-portfolio-temp 0.0.20.dev0__tar.gz → 0.0.22.dev0__tar.gz

{eth_portfolio_temp-0.0.20.dev0 → eth_portfolio_temp-0.0.22.dev0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: eth_portfolio_temp
-Version: 0.0.20.dev0
+Version: 0.0.22.dev0
 Summary: eth-portfolio makes it easy to analyze your portfolio.
 Home-page: https://github.com/BobTheBuidler/eth-portfolio
 Author: BobTheBuidler

{eth_portfolio_temp-0.0.20.dev0 → eth_portfolio_temp-0.0.22.dev0}/README.md

@@ -1,17 +1,21 @@
 # eth-portfolio
 Use `eth-portfolio` to output information about your portfolio in a streamlined, speed-optimized way.
 
+### Requirements
+- Python 3.9 or higher.
+- At least 16GB of RAM.
+
+### Prerequisites
+- First, you will need to bring your own archive node. This can be one you run yourself, or one from one of the common providers (Tenderly, Alchemy, QuickNode, etc.)
+- You must configure a [brownie network](https://eth-brownie.readthedocs.io/en/stable/network-management.html) to use your RPC.
+- You will also need an auth token for [Etherscan](https://etherscan.io/)'s API. Follow their [guide](https://docs.etherscan.io/etherscan-v2/getting-an-api-key) to get your key, and set env var `ETHERSCAN_TOKEN` with its value.
+
 ### Installation:
 - Make sure you are using Python >= 3.9 and < 3.13. You should start with a fresh virtual environment, and then just...
 ```
 pip install git+https://github.com/BobTheBuidler/eth-portfolio
 ```
 
-### Prerequisites
-
-- First, you will need to bring your own archive node. This can be one you run yourself, or one from one of the common providers (Tenderly, Alchemy, QuickNode, etc.)
-- You must configure a [brownie network](https://eth-brownie.readthedocs.io/en/stable/network-management.html) to use your RPC.
-
 ### Usage:
 For basic use, input each of your addresses as environment variables using the following pattern:
 ```

eth_portfolio_temp-0.0.22.dev0/docs/exporter.rst

@@ -0,0 +1,131 @@
+Portfolio Exporter
+==================
+
+Overview
+--------
+The Portfolio Exporter is a command-line tool provided by eth-portfolio. It is designed to extract and compile portfolio data – such as token balances, transaction history, and other relevant financial information – into exportable formats that can be further analyzed or integrated with external services.
+
+Requirements
+------------
+- Python 3.10 or higher.
+- At least 16GB of RAM.
+- All dependencies installed as specified in the project’s ``requirements.txt`` file.
+- Proper configuration settings (e.g., API keys, network endpoints) must be set up as required by the portfolio and balance loaders. More info below.
+
+Usage
+-----
+The Portfolio Exporter should be used via the CLI. Include each of your wallets in the command like so:
+  
+  .. code-block:: bash
+  
+      eth-portfolio export --wallet 0x123 --wallet 0x234 --wallet 0x345
+
+Command-Line Options
+--------------------
+The Portfolio Exporter supports several command-line options to customize its behavior:
+
+- ``--wallet <address>``
+  Specify a wallet to be included in your portfolio. You can pass this flag multiple times for multiple wallets.
+
+- ``--network <network-id>``
+  The brownie network identifier for the rpc you wish to use. default: mainnet
+
+- ``--interval <interval>``
+  The time interval between datapoints. default: 6h
+
+- ``--first-tx-block <block>``
+  The block of your portfolio's first transaction, if known. This value, if provided, allows us to speed up processing of your data by limiting the block range we need to query. If not provided, the whole blockchain will be scanned. default: 0
+  
+- ``--export-start-block <block>``
+  The first block in the range you wish to export. default: 0
+
+Docker Containers
+-----------------
+When you run the Portfolio Exporter, ETH Portfolio will build and start 3 required Docker containers on your system, as well as up to 1 optional container depending on your configuration. These containers work together to collect, store, and visualize your portfolio data:
+
+- ``grafana``
+  The Grafana container provides a web-based dashboard for visualizing your portfolio metrics and balances. It is pre-configured with a "Balances" dashboard and uses victoria-metrics as its Prometheus-compatible data source. Grafana is accessible locally (default port 3000) and supports dashboard image rendering via the renderer container. Anonymous access is enabled for convenience.
+
+- ``renderer`` (optional)
+  The renderer container runs the official grafana-image-renderer service, enabling Grafana to export dashboards as images. This is useful for generating reports or sharing visualizations. It is tightly integrated with the Grafana container and operates on port 8091 by default. **This container is only started if you pass the ``--start-renderer`` flag to the infra command.**
+
+- ``victoria-metrics``
+  victoria-metrics is a high-performance, long-retention time-series database that stores all portfolio and balance metrics. Data is persisted to a local volume for durability, with a default retention period of 10 years. It receives metrics from vmagent and serves as the main data source for Grafana (default port 8428).
+
+- ``vmagent``
+  vmagent is responsible for collecting and forwarding metrics to victoria-metrics. It is configured to remote write all scraped metrics to the victoria-metrics container. vmagent ensures that all relevant portfolio data is reliably collected and stored for visualization and analysis.
+
+Data Flow Summary
+-----------------
+The containers interact as follows:
+
+1. **eth-portfolio** and/or related exporters expose metrics endpoints.
+2. **vmagent** scrapes these endpoints and forwards the data to **victoria-metrics**.
+3. **victoria-metrics** stores the time-series data for long-term analysis.
+4. **Grafana** is pre-configured to use victoria-metrics as its data source and provides dashboards (such as "Balances") for visualization.
+5. The **renderer** container enables Grafana to export dashboards as images for reporting or sharing (if started with ``--start-renderer``).
+
+.. mermaid::
+
+   graph TD
+     A[eth-portfolio metrics/exporters] --> B(vmagent)
+     B --> C(victoria-metrics)
+     C --> D[Grafana]
+     D --> E[renderer]
+     D --> F[User Dashboard]
+
+Additional Information
+----------------------
+- All containers are orchestrated via Docker Compose and are automatically started as needed.
+- Grafana provisioning files ensure dashboards and data sources are set up out-of-the-box.
+- Metrics are stored with a long retention period, enabling historical analysis.
+- Dashboard images can be generated for reporting via the renderer (if started).
+
+Data Extraction and Portfolio Calculations
+------------------------------------------
+
+The Portfolio Exporter performs a comprehensive extraction and calculation process to ensure accurate and detailed portfolio data. The workflow is as follows:
+
+**1. Data Extraction Process**
+
+- **Per-Wallet, Per-Timestamp:**  
+  For each wallet specified via the CLI, the exporter iterates over a series of timestamps (based on the `--interval` argument). At each timestamp, it determines the corresponding Ethereum block and extracts a snapshot of the portfolio at that point in time.
+
+- **Asynchronous Data Loading:**  
+  The exporter leverages highly concurrent, asynchronous routines to efficiently load data for all wallets and tokens. It uses specialized loader functions to fetch:
+  - Token balances (including ERC-20 and protocol-specific tokens)
+  - Transaction history
+  - Token transfers
+  - Protocol positions (e.g., lending, borrowing, staking)
+
+- **Block and Transaction Resolution:**  
+  For each timestamp, the exporter resolves the closest block and loads all relevant on-chain data for the specified wallets. This includes querying balances, fetching transaction receipts, and aggregating protocol positions.
+
+**2. Portfolio Calculations**
+
+- **Balance and Value Computation:**  
+  For each token held by a wallet, the exporter calculates:
+  - The raw token balance
+  - The USD value of the balance (using price oracles or on-chain data)
+  - Protocol-specific metrics (e.g., supplied/borrowed amounts, rewards)
+
+- **Aggregation and Structuring:**  
+  The exporter organizes data into logical sections, such as "assets", "debts", and "external" protocol positions. It aggregates balances across all wallets and protocols, ensuring a unified view of the portfolio.
+
+- **Metric Formatting:**  
+  Each data point is formatted as a Prometheus metric, including metadata such as wallet address, token address, token symbol, protocol name, and a logical "bucket" for grouping. Both the raw balance and USD value are exported for each token.
+
+**3. Export and Storage**
+
+- **Deduplication:**  
+  Before exporting, the system checks if data for a given timestamp already exists in the time-series database to avoid redundant processing.
+
+- **Export to VictoriaMetrics:**  
+  The processed metrics are sent to the VictoriaMetrics time-series database, where they are stored for long-term analysis and visualization in Grafana.
+
+- **Error Handling and Robustness:**  
+  The exporter is designed to handle network errors, missing data, and protocol-specific quirks gracefully, ensuring reliable and repeatable exports.
+
+**Summary**
+
+This architecture allows the Portfolio Exporter to provide a high-fidelity, time-resolved view of all wallet balances, protocol positions, and historical portfolio values. The modular, asynchronous design ensures scalability and performance, even for large portfolios or long time ranges.

{eth_portfolio_temp-0.0.20.dev0 → eth_portfolio_temp-0.0.22.dev0}/docs/index.rst

@@ -12,6 +12,7 @@ Welcome to eth-portfolio's documentation!
 
    source/modules.rst
    exporter
+   new-protocols
 
 Indices and tables
 ==================

eth_portfolio_temp-0.0.22.dev0/docs/new-protocols.rst

@@ -0,0 +1,132 @@
+Contributing: Adding New Protocols
+==================================
+
+This guide explains how to add support for new staking pools, lending/borrowing protocols, and other DeFi integrations to eth-portfolio. The system is designed for extensibility, using abstract base classes (ABCs) and a modular protocol architecture.
+
+Choosing the Appropriate Base Class
+-----------------------------------
+
+eth-portfolio supports several types of protocols, each with a corresponding abstract base class. To add a new protocol, first determine which category your integration fits into:
+
+- **Generic Protocol** (`ProtocolABC`):  
+  Use this for most DeFi protocols that do not fit a more specific category. This is the most flexible base class and is suitable for lending, borrowing, or any protocol with custom logic.  
+  See: `Adding a New Lending/Borrowing or Generic Protocol`_
+
+- **Staking Pool** (`StakingPoolABC`):  
+  Use this for protocols where users stake tokens to earn rewards, and the pool may have multiple tokens or custom reward logic.  
+  See: `Adding a New Staking Pool`_
+
+- **Single-Token Staking Pool** (`SingleTokenStakingPoolABC`):  
+  Use this for staking pools that only accept a single token and have a straightforward staking/reward structure.  
+  See: `Adding a New Single-Token Staking Pool`_
+
+- **Protocol with Staking** (`ProtocolWithStakingABC`):  
+  Use this if your protocol combines lending/borrowing or other DeFi features with an integrated staking component.  
+  See: `Adding a Protocol with Staking`_
+
+If you are unsure, start with `ProtocolABC` for lending/borrowing or custom protocols, and use the staking pool base classes for staking integrations. See the subsections below for details and examples.
+
+Adding a New Staking Pool
+-------------------------
+
+Subclass `StakingPoolABC` from `eth_portfolio/protocols/_base.py` for pools with multiple tokens or custom logic.
+
+**Steps:**
+1. Subclass `StakingPoolABC`.
+2. Implement the `_balances` method to return a `TokenBalances` mapping for a given address and block.
+3. Implement any required methods such as `contract`, `deploy_block`, etc.
+4. Register your pool in the appropriate protocol module.
+
+**Example:**
+
+.. code-block:: python
+
+   from eth_portfolio.protocols._base import StakingPoolABC
+
+   class MyStakingPool(StakingPoolABC):
+       def contract(self):
+           # Return a web3 contract instance for the staking pool
+           ...
+
+       async def _balances(self, address, block=None):
+           # Query the staked balance and rewards for the address
+           ...
+
+See `eth_portfolio/protocols/liquity.py` for real-world examples.
+
+Adding a New Single-Token Staking Pool
+--------------------------------------
+
+Subclass `SingleTokenStakingPoolABC` from `eth_portfolio/protocols/_base.py` for pools that only accept a single token.
+
+**Example:**
+
+.. code-block:: python
+
+   from eth_portfolio.protocols._base import SingleTokenStakingPoolABC
+
+   class MySingleTokenStakingPool(SingleTokenStakingPoolABC):
+       def contract(self):
+           ...
+
+       async def _balances(self, address, block=None):
+           ...
+
+See `eth_portfolio/protocols/liquity.py` and `eth_portfolio/protocols/convex.py` for examples.
+
+Adding a New Lending/Borrowing or Generic Protocol
+--------------------------------------------------
+
+Subclass `ProtocolABC` from `eth_portfolio/protocols/_base.py` for lending, borrowing, or other custom DeFi protocols.
+
+**Steps:**
+1. Subclass `ProtocolABC`.
+2. Implement the `_balances` method to return a `TokenBalances` mapping, including supplied, borrowed, and reward tokens.
+3. Register your protocol in the appropriate module.
+
+**Example:**
+
+.. code-block:: python
+
+   from eth_portfolio.protocols._base import ProtocolABC
+
+   class MyLendingProtocol(ProtocolABC):
+       async def _balances(self, address, block=None):
+           # Query supplied and borrowed balances for the address
+           ...
+
+See `eth_portfolio/protocols/dsr.py` and `eth_portfolio/protocols/lending/` for examples.
+
+Adding a Protocol with Staking
+------------------------------
+
+If your protocol combines lending/borrowing with staking, subclass `ProtocolWithStakingABC` from `eth_portfolio/protocols/_base.py`.
+
+**Example:**
+
+.. code-block:: python
+
+   from eth_portfolio.protocols._base import ProtocolWithStakingABC
+
+   class MyProtocolWithStaking(ProtocolWithStakingABC):
+       async def _balances(self, address, block=None):
+           # Query balances and staking positions
+           ...
+
+See `eth_portfolio/protocols/convex.py` for an example.
+
+Best Practices and Tips
+-----------------------
+
+- **Use Async Methods:** All balance queries should be asynchronous for performance.
+- **Handle Edge Cases:** Account for protocol-specific quirks, such as non-standard tokens or special reward logic.
+- **Testing:** Add tests for your protocol to ensure correctness and compatibility.
+- **Documentation:** Document your protocol’s behavior and any configuration requirements.
+
+References
+----------
+
+- Abstract base classes: `eth_portfolio/protocols/_base.py`
+- Example staking pools: `eth_portfolio/protocols/liquity.py`, `eth_portfolio/protocols/convex.py`
+- Example lending protocols: `eth_portfolio/protocols/dsr.py`, `eth_portfolio/protocols/lending/`
+- Protocol registration: `eth_portfolio/protocols/__init__.py`

{eth_portfolio_temp-0.0.20.dev0 → eth_portfolio_temp-0.0.22.dev0}/eth_portfolio/__init__.py

@@ -1,3 +1,12 @@
+import warnings
+
+warnings.filterwarnings(
+    "ignore",
+    message="Changing decimals precision could have unintended side effects!",
+    category=UserWarning,
+)
+
+
 import a_sync._smart
 
 a_sync._smart.set_smart_task_factory()

{eth_portfolio_temp-0.0.20.dev0 → eth_portfolio_temp-0.0.22.dev0}/eth_portfolio/_loaders/token_transfer.py

@@ -148,7 +148,7 @@ async def _insert_to_db(transfer: TokenTransfer, load_prices: bool) -> None:
                 raise
         except decimal.InvalidOperation as e:
             # TODO: debug why this happens sometimes
-            logger.exception(e, transfer)
+            logger.exception("%s %s", e, transfer)
             pass
 
 

{eth_portfolio_temp-0.0.20.dev0 → eth_portfolio_temp-0.0.22.dev0}/eth_portfolio/_shitcoins.py

@@ -89,6 +89,7 @@ shitcoins: Final = {
         "0xF01f7A348681776c1FC9A066c6973882B693cdC6",
         "0x0e1CD6d2715432e4DBedFE969b0Eb2867FF61d5b",
         # Generally looks like shit
+        "0x7CD6143B8781dC7e0667e50DB02Eb6539799722F",
         "0x174Cd3359C6a4E6B64D2995Da4E2E4631379526E",
         "0x85E199607eb8aEef775Ae6424b42d20616267852",
         "0x4D57c67C8Bab0Fb3A0a0A35B904FBff4E5450521",

{eth_portfolio_temp-0.0.20.dev0 → eth_portfolio_temp-0.0.22.dev0}/eth_portfolio_scripts/docker/__init__.py

@@ -1,11 +1,12 @@
 from eth_portfolio_scripts.docker.check import *
-from eth_portfolio_scripts.docker.docker_compose import build, down, ensure_containers, up
+from eth_portfolio_scripts.docker.docker_compose import build, down, ensure_containers, up, stop
 
 __all__ = [
     # commands
     "build",
     "up",
     "down",
+    "stop",
     # decorators
     "ensure_containers",
     # checks

{eth_portfolio_temp-0.0.20.dev0 → eth_portfolio_temp-0.0.22.dev0}/eth_portfolio_scripts/docker/docker_compose.py

@@ -14,19 +14,28 @@ logger = logging.getLogger(__name__)
 compose_file = path.join(path.dirname(path.abspath(__file__)), "docker-compose.yaml")
 
 
-def up() -> None:
-    build()
+def up(*services: str) -> None:
+    build(*services)
     print("starting the infra containers...")
-    _exec_command(["up", "-d"])
+    _exec_command(["up", "-d", *services])
 
 
-def down(*_) -> None:
+def down() -> None:
     _exec_command(["down"])
 
 
-def build() -> None:
+def build(*services: str) -> None:
     print("building the grafana containers")
-    _exec_command(["build"])
+    _exec_command(["build", *services])
+
+
+def stop(container_name: str) -> None:
+    """
+    Stop the specified container if it is running.
+    Defaults to stopping the 'renderer' container.
+    """
+    print(f"stopping the {container_name} container...")
+    _exec_command(["stop", container_name])
 
 
 _P = ParamSpec("_P")

{eth_portfolio_temp-0.0.20.dev0 → eth_portfolio_temp-0.0.22.dev0}/eth_portfolio_scripts/main.py

@@ -23,6 +23,11 @@ infra_parser = subparsers.add_parser(
 )
 infra_parser.add_argument("cmd", help="What do you want to do?")
 add_infra_port_args(infra_parser)
+infra_parser.add_argument(
+    "--start-renderer",
+    action="store_true",
+    help="If set, starts the renderer container in addition to the default containers. By default, only grafana, victoria-metrics, and vmagent are started.",
+)
 
 export_parser.add_argument(
     "--wallet",
@@ -87,7 +92,10 @@ def main():
     if command == "infra":
         if args.cmd == "start":
             # Start the backend containers
-            docker.up()
+            if getattr(args, "start_renderer", False):
+                docker.up()
+            else:
+                docker.up("grafana", "victoria-metrics", "vmagent")
         elif args.cmd == "stop":
             docker.down()
         else:

{eth_portfolio_temp-0.0.20.dev0 → eth_portfolio_temp-0.0.22.dev0}/eth_portfolio_temp.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: eth-portfolio-temp
-Version: 0.0.20.dev0
+Version: 0.0.22.dev0
 Summary: eth-portfolio makes it easy to analyze your portfolio.
 Home-page: https://github.com/BobTheBuidler/eth-portfolio
 Author: BobTheBuidler

{eth_portfolio_temp-0.0.20.dev0 → eth_portfolio_temp-0.0.22.dev0}/eth_portfolio_temp.egg-info/SOURCES.txt

@@ -16,6 +16,7 @@ docs/conf.py
 docs/exporter.rst
 docs/index.rst
 docs/make.bat
+docs/new-protocols.rst
 docs/_build/html/_static/alabaster.css
 docs/_build/html/_static/basic.css
 docs/_build/html/_static/custom.css

eth_portfolio_temp-0.0.20.dev0/docs/exporter.rst

@@ -1,44 +0,0 @@
-Portfolio Exporter
-==================
-
-Overview
---------
-The Portfolio Exporter is a command-line tool provided by eth-portfolio. It is designed to extract and compile portfolio data – such as token balances, transaction history, and other relevant financial information – into exportable formats that can be further analyzed or integrated with external services.
-
-Requirements
-------------
-- Python 3.10 or higher.
-- At least 16GB of RAM.
-- All dependencies installed as specified in the project’s ``requirements.txt`` file.
-- Proper configuration settings (e.g., API keys, network endpoints) must be set up as required by the portfolio and balance loaders. More info below.
-
-Usage
------
-The Portfolio Exporter should be used via the CLI. Include each of your wallets in the command like so:
-  
-  .. code-block:: bash
-  
-      eth-portfolio export --wallet 0x123 --wallet 0x234 --wallet 0x345
-
-Command-Line Options
---------------------
-The Portfolio Exporter supports several command-line options to customize its behavior:
-
-- ``--wallet <address>``
-  Specify a wallet to be included in your portfolio. You can pass this flag multiple times for multiple wallets.
-
-- ``--network <network-id>``
-  The brownie network identifier for the rpc you wish to use. default: mainnet
-
-- ``--interval <interval>``
-  The time interval between datapoints. default: 6h
-
-- ``--first-tx-block <block>``
-  The block of your portfolio's first transaction, if known. This value, if provided, allows us to speed up processing of your data by limiting the block range we need to query. If not provided, the whole blockchain will be scanned. default: 0
-  
-- ``--export-start-block <block>``
-  The first block in the range you wish to export. default: 0
-
-Additional Information
-----------------------
-TODO: add more details on the underlying data extraction process and portfolio calculations