ethspecify 0.3.0__tar.gz → 0.3.2__tar.gz

ethspecify-0.3.2/PKG-INFO

@@ -0,0 +1,237 @@
+Metadata-Version: 2.4
+Name: ethspecify
+Version: 0.3.2
+Summary: A utility for processing Ethereum specification tags.
+Home-page: https://github.com/jtraglia/ethspecify
+Author: Justin Traglia
+Author-email: jtraglia@pm.me
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.6
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests==2.32.3
+Requires-Dist: PyYAML>=6.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# ethspecify
+
+A tool for referencing the Ethereum specifications in clients.
+
+The idea is that ethspecify will help developers keep track of when the specification changes. It
+will also help auditors verify that the client implementations match the specifications. Ideally,
+this is configured as a CI check which notifies client developers when the specification changes.
+When that happens, they can update the implementations appropriately.
+
+## Getting started
+
+### Installation
+
+```
+pipx install ethspecify
+```
+
+### Initialize specrefs
+
+From the root of the client source directory, initialize a directory for specrefs:
+
+```
+$ ethspecify init v1.6.0-beta.0
+Initializing specrefs directory: v1.6.0-beta.0
+Successfully created specrefs directory at: specrefs
+```
+
+This creates a `specrefs` directory with `.ethspecify.yml` and YAML files for each spec category
+(constants, configs, presets, functions, containers, dataclasses, types).
+
+### Map sources
+
+Edit the YAML files to add sources for where each spec item is implemented.
+
+If it's the entire file:
+
+```yaml
+- name: BlobParameters
+  sources:
+    - file: ethereum/spec/src/main/java/tech/pegasys/teku/spec/logic/versions/fulu/helpers/BlobParameters.java
+  spec: |
+    <spec dataclass="BlobParameters" fork="fulu" hash="a4575aa8">
+    class BlobParameters:
+        epoch: Epoch
+        max_blobs_per_block: uint64
+    </spec>
+```
+
+If it's multiple entire files:
+
+```yaml
+- name: BlobsBundleDeneb
+  sources:
+    - file: ethereum/spec/src/main/java/tech/pegasys/teku/spec/datastructures/execution/BlobsBundle.java
+    - file: ethereum/spec/src/main/java/tech/pegasys/teku/spec/datastructures/builder/BlobsBundleSchema.java
+    - file: ethereum/spec/src/main/java/tech/pegasys/teku/spec/datastructures/builder/versions/deneb/BlobsBundleDeneb.java
+    - file: ethereum/spec/src/main/java/tech/pegasys/teku/spec/datastructures/builder/versions/deneb/BlobsBundleSchemaDeneb.java
+  spec: |
+    <spec dataclass="BlobsBundle" fork="deneb" hash="8d6e7be6">
+    class BlobsBundle(object):
+        commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+        proofs: List[KZGProof, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+        blobs: List[Blob, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+    </spec>
+```
+
+If it's a specific part of a file:
+
+```yaml
+- name: EFFECTIVE_BALANCE_INCREMENT
+  sources:
+    - file: ethereum/spec/src/main/resources/tech/pegasys/teku/spec/config/presets/mainnet/phase0.yaml
+      search: "EFFECTIVE_BALANCE_INCREMENT:"
+  spec: |
+    <spec preset_var="EFFECTIVE_BALANCE_INCREMENT" fork="phase0" hash="23dfe52c">
+    EFFECTIVE_BALANCE_INCREMENT: Gwei = 1000000000
+    </spec>
+```
+
+You can also use regex in the searches if that is necessary:
+
+```yaml
+- name: ATTESTATION_DUE_BPS
+  sources:
+    - file: ethereum/spec/src/main/resources/tech/pegasys/teku/spec/config/configs/mainnet.yaml
+      search: "^ATTESTATION_DUE_BPS:"
+      regex: true
+  spec: |
+    <spec config_var="ATTESTATION_DUE_BPS" fork="phase0" hash="929dd1c9">
+    ATTESTATION_DUE_BPS: uint64 = 3333
+    </spec>
+```
+
+### Check specrefs
+
+Run the check command in CI to verify all spec items are properly mapped:
+
+```
+$ ethspecify check --path=specrefs
+MISSING: constants.BLS_MODULUS#deneb
+```
+
+### Add exceptions
+
+Some spec items may not be implemented in your client. Add them to the exceptions list in
+`specrefs/.ethspecify.yml`:
+
+```yaml
+specrefs:
+  files:
+    - containers.yml
+    - functions.yml
+    # ...
+
+exceptions:
+  containers:
+    # Not defined, unnecessary
+    - Eth1Block
+
+  functions:
+    # No light client support
+    - is_valid_light_client_header
+    - process_light_client_update
+```
+
+## Style Options
+
+This attribute can be used to change how the specification content is shown.
+
+### `hash` (default)
+
+This style adds a hash of the specification content to the spec tag, without showing the content.
+
+```
+<spec fn="apply_deposit" fork="electra" hash="c723ce7b" />
+```
+
+> [!NOTE]
+> The hash is the first 8 characters of the specification content's SHA256 digest.
+
+### `full`
+
+This style displays the whole content of this specification item, including comments.
+
+```
+<spec fn="is_fully_withdrawable_validator" fork="deneb" style="full">
+def is_fully_withdrawable_validator(validator: Validator, balance: Gwei, epoch: Epoch) -> bool:
+    """
+    Check if ``validator`` is fully withdrawable.
+    """
+    return (
+        has_eth1_withdrawal_credential(validator)
+        and validator.withdrawable_epoch <= epoch
+        and balance > 0
+    )
+</spec>
+```
+
+### `link`
+
+This style displays a GitHub link to the specification item.
+
+```
+<spec fn="apply_pending_deposit" fork="electra" style="link" hash="83ee9126">
+https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#new-apply_pending_deposit
+</spec>
+```
+
+### `diff`
+
+This style displays a diff with the previous fork's version of the specification.
+
+```
+<spec ssz_object="BeaconState" fork="electra" style="diff">
+--- deneb
++++ electra
+@@ -27,3 +27,12 @@
+     next_withdrawal_index: WithdrawalIndex
+     next_withdrawal_validator_index: ValidatorIndex
+     historical_summaries: List[HistoricalSummary, HISTORICAL_ROOTS_LIMIT]
++    deposit_requests_start_index: uint64
++    deposit_balance_to_consume: Gwei
++    exit_balance_to_consume: Gwei
++    earliest_exit_epoch: Epoch
++    consolidation_balance_to_consume: Gwei
++    earliest_consolidation_epoch: Epoch
++    pending_deposits: List[PendingDeposit, PENDING_DEPOSITS_LIMIT]
++    pending_partial_withdrawals: List[PendingPartialWithdrawal, PENDING_PARTIAL_WITHDRAWALS_LIMIT]
++    pending_consolidations: List[PendingConsolidation, PENDING_CONSOLIDATIONS_LIMIT]
+</spec>
+```
+
+> [!NOTE]
+> Comments are stripped from the specifications when the `diff` style is used. We do this because
+> these complicate the diff; the "[Modified in Fork]" comments aren't valuable here.
+
+This can be used with any specification item, like functions too:
+
+```
+<spec fn="is_eligible_for_activation_queue" fork="electra" style="diff">
+--- phase0
++++ electra
+@@ -4,5 +4,5 @@
+     """
+     return (
+         validator.activation_eligibility_epoch == FAR_FUTURE_EPOCH
+-        and validator.effective_balance == MAX_EFFECTIVE_BALANCE
++        and validator.effective_balance >= MIN_ACTIVATION_BALANCE
+     )
+</spec>
+```

ethspecify-0.3.2/README.md

@@ -0,0 +1,211 @@
+# ethspecify
+
+A tool for referencing the Ethereum specifications in clients.
+
+The idea is that ethspecify will help developers keep track of when the specification changes. It
+will also help auditors verify that the client implementations match the specifications. Ideally,
+this is configured as a CI check which notifies client developers when the specification changes.
+When that happens, they can update the implementations appropriately.
+
+## Getting started
+
+### Installation
+
+```
+pipx install ethspecify
+```
+
+### Initialize specrefs
+
+From the root of the client source directory, initialize a directory for specrefs:
+
+```
+$ ethspecify init v1.6.0-beta.0
+Initializing specrefs directory: v1.6.0-beta.0
+Successfully created specrefs directory at: specrefs
+```
+
+This creates a `specrefs` directory with `.ethspecify.yml` and YAML files for each spec category
+(constants, configs, presets, functions, containers, dataclasses, types).
+
+### Map sources
+
+Edit the YAML files to add sources for where each spec item is implemented.
+
+If it's the entire file:
+
+```yaml
+- name: BlobParameters
+  sources:
+    - file: ethereum/spec/src/main/java/tech/pegasys/teku/spec/logic/versions/fulu/helpers/BlobParameters.java
+  spec: |
+    <spec dataclass="BlobParameters" fork="fulu" hash="a4575aa8">
+    class BlobParameters:
+        epoch: Epoch
+        max_blobs_per_block: uint64
+    </spec>
+```
+
+If it's multiple entire files:
+
+```yaml
+- name: BlobsBundleDeneb
+  sources:
+    - file: ethereum/spec/src/main/java/tech/pegasys/teku/spec/datastructures/execution/BlobsBundle.java
+    - file: ethereum/spec/src/main/java/tech/pegasys/teku/spec/datastructures/builder/BlobsBundleSchema.java
+    - file: ethereum/spec/src/main/java/tech/pegasys/teku/spec/datastructures/builder/versions/deneb/BlobsBundleDeneb.java
+    - file: ethereum/spec/src/main/java/tech/pegasys/teku/spec/datastructures/builder/versions/deneb/BlobsBundleSchemaDeneb.java
+  spec: |
+    <spec dataclass="BlobsBundle" fork="deneb" hash="8d6e7be6">
+    class BlobsBundle(object):
+        commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+        proofs: List[KZGProof, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+        blobs: List[Blob, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+    </spec>
+```
+
+If it's a specific part of a file:
+
+```yaml
+- name: EFFECTIVE_BALANCE_INCREMENT
+  sources:
+    - file: ethereum/spec/src/main/resources/tech/pegasys/teku/spec/config/presets/mainnet/phase0.yaml
+      search: "EFFECTIVE_BALANCE_INCREMENT:"
+  spec: |
+    <spec preset_var="EFFECTIVE_BALANCE_INCREMENT" fork="phase0" hash="23dfe52c">
+    EFFECTIVE_BALANCE_INCREMENT: Gwei = 1000000000
+    </spec>
+```
+
+You can also use regex in the searches if that is necessary:
+
+```yaml
+- name: ATTESTATION_DUE_BPS
+  sources:
+    - file: ethereum/spec/src/main/resources/tech/pegasys/teku/spec/config/configs/mainnet.yaml
+      search: "^ATTESTATION_DUE_BPS:"
+      regex: true
+  spec: |
+    <spec config_var="ATTESTATION_DUE_BPS" fork="phase0" hash="929dd1c9">
+    ATTESTATION_DUE_BPS: uint64 = 3333
+    </spec>
+```
+
+### Check specrefs
+
+Run the check command in CI to verify all spec items are properly mapped:
+
+```
+$ ethspecify check --path=specrefs
+MISSING: constants.BLS_MODULUS#deneb
+```
+
+### Add exceptions
+
+Some spec items may not be implemented in your client. Add them to the exceptions list in
+`specrefs/.ethspecify.yml`:
+
+```yaml
+specrefs:
+  files:
+    - containers.yml
+    - functions.yml
+    # ...
+
+exceptions:
+  containers:
+    # Not defined, unnecessary
+    - Eth1Block
+
+  functions:
+    # No light client support
+    - is_valid_light_client_header
+    - process_light_client_update
+```
+
+## Style Options
+
+This attribute can be used to change how the specification content is shown.
+
+### `hash` (default)
+
+This style adds a hash of the specification content to the spec tag, without showing the content.
+
+```
+<spec fn="apply_deposit" fork="electra" hash="c723ce7b" />
+```
+
+> [!NOTE]
+> The hash is the first 8 characters of the specification content's SHA256 digest.
+
+### `full`
+
+This style displays the whole content of this specification item, including comments.
+
+```
+<spec fn="is_fully_withdrawable_validator" fork="deneb" style="full">
+def is_fully_withdrawable_validator(validator: Validator, balance: Gwei, epoch: Epoch) -> bool:
+    """
+    Check if ``validator`` is fully withdrawable.
+    """
+    return (
+        has_eth1_withdrawal_credential(validator)
+        and validator.withdrawable_epoch <= epoch
+        and balance > 0
+    )
+</spec>
+```
+
+### `link`
+
+This style displays a GitHub link to the specification item.
+
+```
+<spec fn="apply_pending_deposit" fork="electra" style="link" hash="83ee9126">
+https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#new-apply_pending_deposit
+</spec>
+```
+
+### `diff`
+
+This style displays a diff with the previous fork's version of the specification.
+
+```
+<spec ssz_object="BeaconState" fork="electra" style="diff">
+--- deneb
++++ electra
+@@ -27,3 +27,12 @@
+     next_withdrawal_index: WithdrawalIndex
+     next_withdrawal_validator_index: ValidatorIndex
+     historical_summaries: List[HistoricalSummary, HISTORICAL_ROOTS_LIMIT]
++    deposit_requests_start_index: uint64
++    deposit_balance_to_consume: Gwei
++    exit_balance_to_consume: Gwei
++    earliest_exit_epoch: Epoch
++    consolidation_balance_to_consume: Gwei
++    earliest_consolidation_epoch: Epoch
++    pending_deposits: List[PendingDeposit, PENDING_DEPOSITS_LIMIT]
++    pending_partial_withdrawals: List[PendingPartialWithdrawal, PENDING_PARTIAL_WITHDRAWALS_LIMIT]
++    pending_consolidations: List[PendingConsolidation, PENDING_CONSOLIDATIONS_LIMIT]
+</spec>
+```
+
+> [!NOTE]
+> Comments are stripped from the specifications when the `diff` style is used. We do this because
+> these complicate the diff; the "[Modified in Fork]" comments aren't valuable here.
+
+This can be used with any specification item, like functions too:
+
+```
+<spec fn="is_eligible_for_activation_queue" fork="electra" style="diff">
+--- phase0
++++ electra
+@@ -4,5 +4,5 @@
+     """
+     return (
+         validator.activation_eligibility_epoch == FAR_FUTURE_EPOCH
+-        and validator.effective_balance == MAX_EFFECTIVE_BALANCE
++        and validator.effective_balance >= MIN_ACTIVATION_BALANCE
+     )
+</spec>
+```

{ethspecify-0.3.0 → ethspecify-0.3.2}/ethspecify/cli.py

@@ -3,7 +3,7 @@ import json
 import os
 import sys
 
-from .core import grep, replace_spec_tags, get_pyspec, get_latest_fork, get_spec_item_history, load_config, run_checks, sort_specref_yaml
+from .core import grep, replace_spec_tags, get_pyspec, get_latest_fork, get_spec_item_history, load_config, run_checks, sort_specref_yaml, generate_specref_files, get_yaml_filename_for_spec_attr, get_spec_attr_and_name, add_missing_entries_to_yaml
 
 
 def process(args):
@@ -16,20 +16,37 @@ def process(args):
     # Load config once from the project directory
     config = load_config(project_dir)
 
-    # Process spec tags in files
-    for f in grep(project_dir, r"<spec\b.*?>", args.exclude):
-        print(f"Processing file: {f}")
-        replace_spec_tags(f, config)
-
-    # Sort specref YAML files if they exist in config
+    # Check if auto_add_missing_entries and auto_standardize_names are enabled
     specrefs_config = config.get('specrefs', {})
     if isinstance(specrefs_config, dict):
+        auto_add_missing = specrefs_config.get('auto_add_missing_entries', False)
+        auto_standardize_names = specrefs_config.get('auto_standardize_names', False)
         specrefs_files = specrefs_config.get('files', [])
     elif isinstance(specrefs_config, list):
+        auto_add_missing = False
+        auto_standardize_names = False
         specrefs_files = specrefs_config
     else:
+        auto_add_missing = False
+        auto_standardize_names = False
         specrefs_files = []
 
+    # Process spec tags in files
+    for f in grep(project_dir, r"<spec\b.*?>", args.exclude):
+        print(f"Processing file: {f}")
+        replace_spec_tags(f, config)
+
+    # Add missing spec items to YAML files if enabled
+    if auto_add_missing:
+        from .core import add_missing_spec_items_to_yaml_files
+        add_missing_spec_items_to_yaml_files(project_dir, config, specrefs_files)
+
+    # Update entry names to <spec_item>#<fork> format if enabled
+    if auto_standardize_names:
+        from .core import update_entry_names_in_yaml_files
+        update_entry_names_in_yaml_files(project_dir, specrefs_files)
+
+    # Sort specref YAML files if they exist in config
     for yaml_file in specrefs_files:
         yaml_path = os.path.join(project_dir, yaml_file)
         if os.path.exists(yaml_path):
@@ -182,6 +199,28 @@ def list_forks(args):
     return 0
 
 
+def init(args):
+    """Initialize a specrefs directory with basic configuration and empty source mappings."""
+    output_dir = args.path or "specrefs"
+    version = args.version
+
+    # Check if output directory already exists
+    if os.path.exists(output_dir):
+        print(f"Error: directory {repr(output_dir)} already exists.")
+        print("Please specify a different directory or remove the existing one.")
+        return 1
+
+    try:
+        # Generate the specref files
+        print(f"Initializing specrefs directory: {version}")
+        generate_specref_files(output_dir, version, "mainnet")
+        print(f"Successfully created specrefs directory at: {output_dir}")
+        return 0
+    except Exception as e:
+        print(f"Error: {e}")
+        return 1
+
+
 def main():
     parser = argparse.ArgumentParser(
         description="Process files containing <spec> tags."
@@ -256,6 +295,21 @@ def main():
         help="Output format (text or json)",
     )
 
+    # Parser for 'init' command
+    init_parser = subparsers.add_parser("init", help="Initialize a specrefs directory")
+    init_parser.set_defaults(func=init)
+    init_parser.add_argument(
+        "version",
+        type=str,
+        help="Specification version (e.g., 'nightly' or 'v1.6.0-alpha.5')",
+    )
+    init_parser.add_argument(
+        "--path",
+        type=str,
+        help="Output directory for specrefs (default: specrefs)",
+        default="specrefs",
+    )
+
     # Default to 'process' if no args are provided
     if len(sys.argv) == 1:
         sys.argv.insert(1, "process")