exness-data-preprocess 0.8.0__tar.gz → 1.0.0__tar.gz

{exness_data_preprocess-0.8.0 → exness_data_preprocess-1.0.0}/.gitignore

@@ -100,3 +100,10 @@ uv.lock
 # Pre-commit
 # .pre-commit-config.yaml should be tracked, not ignored
 .token-info.md
+
+# Node.js (semantic-release)
+node_modules/
+
+# Link checking (lychee)
+.lychee*
+.lycheecache

exness_data_preprocess-1.0.0/.releaserc.yml

@@ -0,0 +1,22 @@
+# semantic-release configuration for Python project
+# Uses @semantic-release/exec instead of @semantic-release/npm
+branches:
+  - main
+
+plugins:
+  - "@semantic-release/commit-analyzer"
+  - "@semantic-release/release-notes-generator"
+  - "@semantic-release/changelog"
+  - - "@semantic-release/exec"
+    - prepareCmd: |
+        # Update version in pyproject.toml
+        sed -i '' 's/^version = ".*"/version = "${nextRelease.version}"/' pyproject.toml
+        # Update version in __init__.py
+        sed -i '' 's/^__version__ = ".*"/__version__ = "${nextRelease.version}"/' src/exness_data_preprocess/__init__.py
+  - - "@semantic-release/git"
+    - assets:
+        - pyproject.toml
+        - src/exness_data_preprocess/__init__.py
+        - CHANGELOG.md
+      message: "chore(release): bump version ${lastRelease.version} → ${nextRelease.version} [skip ci]"
+  - "@semantic-release/github"

{exness_data_preprocess-0.8.0 → exness_data_preprocess-1.0.0}/CHANGELOG.md

@@ -1,9 +1,33 @@
-# [0.8.0](https://github.com/terrylica/exness-data-preprocess/compare/v0.7.2...v0.8.0) (2025-12-10)
+# [1.0.0](https://github.com/terrylica/exness-data-preprocess/compare/v0.8.0...v1.0.0) (2025-12-10)
+
+
+* feat!: prune codebase - remove unused deps, archive docs, extract mixin ([e0d48b8](https://github.com/terrylica/exness-data-preprocess/commit/e0d48b8fe7ba44d2c3bc326661102a0b3d3b33b7))
+
+
+### BREAKING CHANGES
+
+* Remove deprecated api.py and cli.py modules
 
+Dependency cleanup:
+- Remove httpx, pyarrow, polars (unused)
+- Move tqdm to optional [examples] extra
+- Consolidate dev dependencies to [dependency-groups]
+
+Documentation archival:
+- Archive 20 planning/research docs to docs/archive/
+- Update .gitignore for node_modules, lychee patterns
+
+Code refactoring:
+- Extract ClickHouseClientMixin from 4 modules (~45 lines saved)
+- Delete api.py (304 lines) and cli.py (252 lines)
+
+ADR: docs/adr/2025-12-09-codebase-pruning.md
+
+# [0.8.0](https://github.com/terrylica/exness-data-preprocess/compare/v0.7.2...v0.8.0) (2025-12-10)
 
 ### Features
 
-* add ClickHouse backend for cloud deployment ([c9b51a3](https://github.com/terrylica/exness-data-preprocess/commit/c9b51a3b374eea74375cc86afdd8cb9391fe73e8))
+- add ClickHouse backend for cloud deployment ([c9b51a3](https://github.com/terrylica/exness-data-preprocess/commit/c9b51a3b374eea74375cc86afdd8cb9391fe73e8))
 
 # Changelog
 

{exness_data_preprocess-0.8.0 → exness_data_preprocess-1.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: exness-data-preprocess
-Version: 0.8.0
+Version: 1.0.0
 Summary: Professional Exness forex tick data preprocessing with optimal compression (Parquet Zstd-22) and DuckDB OHLC generation. Provides efficient storage (9% smaller than ZIP) with lossless precision and direct queryability.
 Project-URL: Homepage, https://github.com/terrylica/exness-data-preprocess
 Project-URL: Documentation, https://github.com/terrylica/exness-data-preprocess#readme
@@ -12,7 +12,7 @@ Author-email: Eon Labs <terry@eonlabs.com>
 Maintainer-email: Terry Li <terry@eonlabs.com>
 License: MIT
 License-File: LICENSE
-Keywords: backtesting,compression,data-engineering,duckdb,efficient-storage,eurusd,exness,financial-data,forex,forex-data,lossless,microstructure,ohlc,pandas,parquet,preprocessing,pyarrow,quantitative-finance,queryable,tick-data,time-series,zstd
+Keywords: backtesting,compression,data-engineering,duckdb,efficient-storage,eurusd,exness,financial-data,forex,forex-data,lossless,microstructure,ohlc,pandas,parquet,preprocessing,quantitative-finance,queryable,tick-data,time-series,zstd
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Financial and Insurance Industry
@@ -29,21 +29,11 @@ Requires-Python: >=3.11
 Requires-Dist: clickhouse-connect>=0.7.0
 Requires-Dist: duckdb>=0.9.0
 Requires-Dist: exchange-calendars>=4.5.6
-Requires-Dist: httpx>=0.25.0
 Requires-Dist: pandas>=2.0.0
-Requires-Dist: pyarrow>=16.0.0
 Requires-Dist: pydantic>=2.0.0
 Requires-Dist: pyyaml>=6.0.0
-Requires-Dist: tqdm>=4.65.0
-Provides-Extra: dev
-Requires-Dist: black>=23.0.0; extra == 'dev'
-Requires-Dist: mypy>=1.5.0; extra == 'dev'
-Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
-Requires-Dist: pytest>=7.0.0; extra == 'dev'
-Requires-Dist: ruff>=0.1.0; extra == 'dev'
-Provides-Extra: test
-Requires-Dist: pytest-cov>=4.0.0; extra == 'test'
-Requires-Dist: pytest>=7.0.0; extra == 'test'
+Provides-Extra: examples
+Requires-Dist: tqdm>=4.65.0; extra == 'examples'
 Description-Content-Type: text/markdown
 
 # Exness Data Preprocess v2.0.0

{exness_data_preprocess-0.8.0 → exness_data_preprocess-1.0.0}/docs/README.md

@@ -55,11 +55,6 @@
 
 ### Implementation Architecture v1.7.0
 
-**Refactoring Plans**:
-
-- [`plans/PHASE7_v1.6.0_REFACTORING_PROGRESS.md`](plans/PHASE7_v1.6.0_REFACTORING_PROGRESS.md) - Complete refactoring progress
-- [`plans/REFACTORING_CHECKLIST.md`](plans/REFACTORING_CHECKLIST.md) - Quick reference checklist
-
 **Pattern**: Facade with 7 specialized modules
 
 **Key Modules**:
@@ -261,5 +256,11 @@ curl -s "https://ticks.ex2archive.com/ticks/" | jq -r '.[] | .name' | grep -i "E
 
 ---
 
-**Last Updated**: 2025-10-18
+## Archived Documentation
+
+Historical planning, research, and completed phase documents are archived in [`archive/`](archive/).
+
+---
+
+**Last Updated**: 2025-12-09
 **Maintainer**: Terry Li <terry@eonlabs.com>

exness_data_preprocess-1.0.0/docs/adr/2025-12-09-codebase-pruning.md

@@ -0,0 +1,150 @@
+---
+status: accepted
+date: 2025-12-09
+decision-maker: Terry Li
+consulted:
+  [Code-Pruning-Agent, Dependency-Pruning-Agent, Documentation-Pruning-Agent]
+research-method: multi-agent
+clarification-iterations: 1
+perspectives: [Maintainability, Technical-Debt, Developer-Experience]
+---
+
+# Codebase Pruning: Dependencies, Documentation, and Code Cleanup
+
+**Design Spec**: [Implementation Spec](/docs/design/2025-12-09-codebase-pruning/spec.md)
+
+## Context and Problem Statement
+
+After the ClickHouse migration (v0.8.0), the codebase has accumulated:
+
+1. **Unused dependencies**: httpx (never imported), polars (abandoned experiment), pyarrow (research-only)
+2. **Orphaned documentation**: 18 planning documents from completed Oct 2025 phases
+3. **Code duplication**: 60+ lines of repeated client lifecycle code across ClickHouse modules
+4. **Deprecated API layer**: 304-line api.py marked for removal
+
+This technical debt increases install size by 80+ MB, confuses navigation, and violates DRY principles.
+
+## Decision Drivers
+
+- **Install size**: 200 MB → 120 MB (40% reduction)
+- **Codebase clarity**: Remove orphaned files that confuse AI assistants
+- **DRY compliance**: Eliminate duplicate ClickHouse client lifecycle code
+- **Breaking change acceptance**: User approved api.py removal
+
+## Considered Options
+
+1. **Quick wins only**: Dependencies + documentation archival
+2. **Quick wins + code refactoring**: Above + ClickHouseClientMixin
+3. **Full pruning** (SELECTED): All above + remove deprecated api.py
+
+## Decision Outcome
+
+**Chosen option**: Full pruning including api.py removal
+
+### Consequences
+
+**Good**:
+
+- 80+ MB smaller install size
+- 400+ lines of code removed
+- Cleaner documentation structure
+- DRY-compliant ClickHouse modules
+
+**Bad**:
+
+- api.py removal is breaking change for any external consumers
+- Research scripts in docs/research/ will need manual pyarrow install
+
+## Architecture
+
+### Before/After Diagram
+
+<!-- graph-easy source:
+[ pyproject.toml ] -- httpx (unused) --> [ 1 MB waste ]
+
+[ pyproject.toml ] -- polars (unused) --> [ 50 MB waste ]
+[ pyproject.toml ] -- pyarrow (research) --> [ 30 MB waste ]
+[ docs/plans/ ] -- 15 orphaned files --> [ 7,405 lines ]
+[ clickhouse_*.py (4) ] -- duplicate lifecycle --> [ 60 lines ]
+[ api.py ] -- deprecated --> [ 304 lines ]
+-->
+
+```
++------------------+     httpx (unused)      +-------------+
+| pyproject.toml   | ----------------------> | 1 MB waste  |
++------------------+                         +-------------+
+        |
+        |            polars (unused)         +-------------+
+        +------------------------------->    | 50 MB waste |
+        |                                    +-------------+
+        |
+        |           pyarrow (research)       +-------------+
+        +------------------------------->    | 30 MB waste |
+                                             +-------------+
+
++------------------+   15 orphaned files     +-------------+
+|   docs/plans/    | ----------------------> | 7,405 lines |
++------------------+                         +-------------+
+
++------------------+  duplicate lifecycle    +-------------+
+|clickhouse_*.py(4)| ----------------------> |  60 lines   |
++------------------+                         +-------------+
+
++------------------+     deprecated          +-------------+
+|     api.py       | ----------------------> | 304 lines   |
++------------------+                         +-------------+
+```
+
+### After Pruning
+
+<!-- graph-easy source:
+[ pyproject.toml ] -- clean deps --> [ 120 MB install ]
+
+[ docs/archive/ ] -- preserved --> [ Historical context ]
+[ clickhouse_base.py ] -- mixin --> [ DRY code ]
+[ api.py ] -- removed --> [ Breaking change logged ]
+-->
+
+```
++------------------+     clean deps          +----------------+
+| pyproject.toml   | ----------------------> | 120 MB install |
++------------------+                         +----------------+
+
++------------------+     preserved           +------------------+
+|  docs/archive/   | ----------------------> | Historical ctxt  |
++------------------+                         +------------------+
+
++--------------------+      mixin            +----------+
+| clickhouse_base.py | --------------------> | DRY code |
++--------------------+                       +----------+
+
++------------------+     removed             +-------------------+
+|     api.py       | ----------------------> | Breaking chg log  |
++------------------+                         +-------------------+
+```
+
+## Implementation Details
+
+### Phase A: Dependency Cleanup
+
+- Remove: httpx, pyarrow, polars
+- Move: tqdm to `[project.optional-dependencies] examples`
+- Consolidate: dev dependencies to `[dependency-groups]`
+
+### Phase B: Documentation Cleanup
+
+- Update .gitignore: node_modules/, .lychee\*, .token-info.md
+- Archive: 18 planning documents to docs/archive/
+- Update: docs/README.md
+
+### Phase C: Code Cleanup
+
+- Create: `clickhouse_base.py` with `ClickHouseClientMixin`
+- Refactor: 4 ClickHouse modules to use mixin
+- Delete: `api.py` (304 lines)
+- Update: `__init__.py` exports
+
+## More Information
+
+- Related: [ClickHouse Migration ADR](/docs/adr/2025-12-09-exness-clickhouse-migration.md)
+- Source Plan: `~/.claude/plans/lively-splashing-steele.md`

exness_data_preprocess-1.0.0/docs/design/2025-12-09-codebase-pruning/spec.md

@@ -0,0 +1,93 @@
+---
+adr: 2025-12-09-codebase-pruning
+source: ~/.claude/plans/lively-splashing-steele.md
+implementation-status: in_progress
+phase: phase-1
+last-updated: 2025-12-09
+---
+
+# Codebase Pruning Implementation Spec
+
+**ADR**: [Codebase Pruning ADR](/docs/adr/2025-12-09-codebase-pruning.md)
+
+## Overview
+
+Full pruning of exness-data-preprocess codebase:
+
+- 80+ MB dependency reduction
+- 18 files archived
+- 400+ lines of code removed
+
+## Implementation Tasks
+
+### Phase A: Dependency Cleanup ✅
+
+**File**: `pyproject.toml`
+
+- [x] Remove `httpx>=0.25.0` from dependencies
+- [x] Remove `pyarrow>=16.0.0` from dependencies
+- [x] Remove `polars>=1.34.0` from dependency-groups
+- [x] Move `tqdm>=4.65.0` to `[project.optional-dependencies] examples`
+- [x] Consolidate dev dependencies (remove conflicting `[project.optional-dependencies] dev`)
+- [x] Verify: `uv sync` succeeds
+- [x] Verify: `pytest tests/` passes (48/48)
+
+### Phase B: Documentation Cleanup ✅
+
+**Files**: `.gitignore`, `docs/archive/`, `docs/README.md`
+
+- [x] Add to `.gitignore`: `node_modules/`, `.lychee*`, `.lycheecache`, `.token-info.md`
+- [x] Create `docs/archive/` directory if not exists
+- [x] Move 17 planning docs from `docs/plans/` to `docs/archive/`
+- [x] Move 2 research docs from `docs/research/` to `docs/archive/`
+- [x] Move `docs/phases/PHASE4_DRY_RUN_CONFIG_PLAN.yaml` to `docs/archive/`
+- [x] Update `docs/README.md` to reflect archival
+
+### Phase C: Code Cleanup ✅
+
+**Files**: `clickhouse_base.py` (NEW), `clickhouse_*.py` (4), `api.py` (DELETE), `cli.py` (DELETE)
+
+- [x] Create `src/exness_data_preprocess/clickhouse_base.py` with `ClickHouseClientMixin`
+- [x] Refactor `clickhouse_manager.py` to use mixin
+- [x] Refactor `clickhouse_gap_detector.py` to use mixin
+- [x] Refactor `clickhouse_query_engine.py` to use mixin
+- [x] Refactor `clickhouse_ohlc_generator.py` to use mixin
+- [x] Delete `src/exness_data_preprocess/api.py` (304 lines)
+- [x] Delete `src/exness_data_preprocess/cli.py` (252 lines, dependent on api.py)
+- [x] Remove CLI entry point from `pyproject.toml`
+- [x] Verify: `pytest tests/` passes (48/48)
+
+## Success Criteria
+
+| Metric                   | Before  | After   | Target          |
+| ------------------------ | ------- | ------- | --------------- |
+| Dependencies (install)   | ~200 MB | ~120 MB | 40% reduction   |
+| docs/plans/ files        | 17      | 2       | 15 archived     |
+| api.py lines             | 304     | 0       | Deleted         |
+| Duplicate lifecycle code | 60      | 15      | ~45 lines saved |
+| Test count               | 48      | 48      | No regression   |
+
+## Files to Modify
+
+| File                                            | Action                                    |
+| ----------------------------------------------- | ----------------------------------------- |
+| `pyproject.toml`                                | Remove 3 deps, move tqdm, consolidate dev |
+| `.gitignore`                                    | Add node_modules, lychee patterns         |
+| `src/exness_data_preprocess/api.py`             | **DELETE**                                |
+| `src/exness_data_preprocess/__init__.py`        | Remove api.py exports                     |
+| `src/exness_data_preprocess/clickhouse_base.py` | **CREATE** mixin                          |
+| `src/exness_data_preprocess/clickhouse_*.py`    | Refactor to use mixin                     |
+| `docs/plans/*.md` (15 files)                    | Move to archive                           |
+| `docs/research/GAP_*.md` (2 files)              | Move to archive                           |
+| `docs/README.md`                                | Update after archival                     |
+
+## Risk Assessment
+
+| Item           | Risk   | Mitigation                              |
+| -------------- | ------ | --------------------------------------- |
+| Remove httpx   | None   | Zero imports confirmed                  |
+| Remove pyarrow | Low    | Research scripts need manual install    |
+| Remove polars  | None   | Zero imports confirmed                  |
+| Archive docs   | Low    | git mv preserves history                |
+| Remove api.py  | Medium | User approved; document breaking change |
+| Mixin refactor | Low    | Tests verify no regression              |