exness-data-preprocess 0.8.0__tar.gz → 1.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/.gitignore +7 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/.lint-relative-paths-results.txt +2 -2
- exness_data_preprocess-1.1.0/.mise.toml +472 -0
- exness_data_preprocess-1.1.0/.releaserc.yml +22 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/CHANGELOG.md +33 -2
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/PKG-INFO +4 -14
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/README.md +7 -6
- exness_data_preprocess-1.1.0/docs/adr/2025-12-09-codebase-pruning.md +150 -0
- exness_data_preprocess-1.1.0/docs/adr/2025-12-10-clickhouse-e2e-validation-pipeline.md +332 -0
- exness_data_preprocess-1.1.0/docs/design/2025-12-09-codebase-pruning/spec.md +93 -0
- exness_data_preprocess-1.1.0/docs/design/2025-12-10-clickhouse-e2e-validation-pipeline/spec.md +127 -0
- exness_data_preprocess-1.1.0/package-lock.json +6272 -0
- exness_data_preprocess-1.1.0/package.json +9 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/pyproject.toml +4 -19
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/__init__.py +15 -12
- exness_data_preprocess-1.1.0/src/exness_data_preprocess/clickhouse_base.py +71 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/clickhouse_client.py +0 -1
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/clickhouse_gap_detector.py +3 -18
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/clickhouse_manager.py +3 -17
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/clickhouse_ohlc_generator.py +3 -18
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/clickhouse_query_engine.py +187 -21
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/gap_detector.py +0 -1
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/models.py +53 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/processor.py +1 -2
- exness_data_preprocess-0.8.0/.lycheecache +0 -0
- exness_data_preprocess-0.8.0/src/exness_data_preprocess/api.py +0 -303
- exness_data_preprocess-0.8.0/src/exness_data_preprocess/cli.py +0 -251
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/.cz.toml +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/.github/workflows/ci.yml +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/.github/workflows/pre-commit-autoupdate.yml +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/.github/workflows/publish.yml +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/.pre-commit-config.yaml +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/CLAUDE.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/CONTRIBUTING.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/LICENSE +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/Makefile +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/README.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/cliff-release-notes.toml +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/cliff.toml +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/DATABASE_SCHEMA.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/EXNESS_DATA_SOURCES.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/MODULE_ARCHITECTURE.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/PRIORITY3_CLEANUP_PLAN.yaml +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/REPOSITORY_REFACTORING_PLAN.yaml +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/RESEARCH_PATTERNS.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/TRADING_HOURS_RESEARCH.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/UNIFIED_DUCKDB_PLAN_v2.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/adr/2025-12-09-exness-clickhouse-migration.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/CLAUDE_MD_AUDIT_2025-10-16_PASS2.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/CLAUDE_MD_VERIFICATION_CHECKLIST_2025-10-16.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/CLAUDE_MD_VERIFICATION_FINDINGS_2025-10-16.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/DOCUMENTATION_AUDIT_2025-10-15.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/DOCUMENTATION_AUDIT_PASS2_2025-10-15.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/DOCUMENTATION_BEST_PRACTICES_LINE_COUNTS.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/E2E_TESTING_PLAN_v1.0.0.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/EXCHANGE_SESSION_AUDIT_2025-10-17.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/FUNCTIONALITY_VALIDATION_REPORT_2025-10-15.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/FUNCTIONALITY_VALIDATION_REPORT_2025-10-16.md +0 -0
- {exness_data_preprocess-0.8.0/docs/research → exness_data_preprocess-1.1.0/docs/archive}/GAP_DETECTION_COMPARISON.md +0 -0
- {exness_data_preprocess-0.8.0/docs/research → exness_data_preprocess-1.1.0/docs/archive}/GAP_DETECTION_SQL_APPROACH.md +0 -0
- {exness_data_preprocess-0.8.0/docs/phases → exness_data_preprocess-1.1.0/docs/archive}/PHASE4_DRY_RUN_CONFIG_PLAN.yaml +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/PHASE7_v1.5.0_REFACTORING_PLAN.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/PHASE7_v1.6.0_REFACTORING_PROGRESS.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/archive/PYDANTIC_REFACTORING_PLAN_v0.2.0_COMPLETE.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/archive/PYDANTIC_REFACTORING_STATUS_v0.2.0_COMPLETE.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/REFACTORING_CHECKLIST.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/RESEARCH_PATTERNS_CANONICALIZATION_2025-10-17.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/SCHEMA_v1.6.0_AUDIT_FINDINGS.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/SCHEMA_v1.6.0_MIGRATION_GUIDE.md +0 -0
- {exness_data_preprocess-0.8.0/docs/plans → exness_data_preprocess-1.1.0/docs/archive}/SESSION_2025-10-15_SUMMARY.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/archive/UNIFIED_DUCKDB_PLAN_v1.0.0_LEGACY.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/archive/implementation-plan-2024-11-24.yaml +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/design/2025-12-09-exness-clickhouse-migration/spec.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/optimization/OPTIMIZATION_OPPORTUNITIES_2025-10-18.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/releases/RELEASE_NOTES.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/releases/RELEASE_NOTES_SHORT.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/HYBRID_SESSION_DETECTION_ANALYSIS.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/compression-benchmarks/README.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/compression-benchmarks/test_all_compression_methods.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/compression-benchmarks/test_delta_encoding_properly.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/compression-benchmarks/test_parquet_compression_methods.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-spread-analysis/01-methodology.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-spread-analysis/02-mode-truncated-analysis.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-spread-analysis/03-hurdle-model-analysis.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-spread-analysis/04-temporal-patterns.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-spread-analysis/05-final-recommendations.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-spread-analysis/README.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-spread-analysis/scripts/reproduction_guide.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-spread-analysis/visualizations/hurdle-model/hurdle_decomposition.png +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-spread-analysis/visualizations/mode-truncated/cv_mode_comparison.png +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-spread-analysis/visualizations/mode-truncated/distribution_overlay.png +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-spread-analysis/visualizations/mode-truncated/signal_quality_comparison.png +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-spread-analysis/visualizations/mode-truncated/variance_decomposition.png +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-spread-analysis/visualizations/temporal/concentration_curve.png +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-spread-analysis/visualizations/temporal/percentile_analysis.png +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/01-methodology.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/02-baseline-sep2024.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/03-multiperiod-validation.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/04-discoveries-and-plan-evolution.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/README.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/findings/baseline-sep2024-comprehensive-summary.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/findings/phase2-mean-reversion-report.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/findings/phase3-volatility-model-report.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/findings/phase4-5-statistical-tests-survivorship-report.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/baseline-sep2024/01_mean_reversion_analysis.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/baseline-sep2024/02_volatility_model_simple.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/baseline-sep2024/03_liquidity_crisis_detection.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/baseline-sep2024/04_regime_detection_analysis.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/multiperiod-validation/phase1_data_validation.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/multiperiod-validation/phase2_mean_reversion.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/multiperiod-validation/phase3_volatility_model.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/multiperiod-validation/phase4_statistical_tests.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/multiperiod-validation/phase5_survivorship_bias.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/multiperiod-validation/phase5_survivorship_bias_v1.0.1.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/multiperiod-validation/phase6a_interval_analysis.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/multiperiod-validation/phase6b_price_clustering.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/multiperiod-validation/phase6c_combined_pattern.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/reproduction_guide.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/visualization/diagnostic_burst_decay.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/visualization/generate_interactive_dashboard.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/visualization/generate_lightweight_dashboard.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/eurusd-zero-spread-deviations/scripts/visualization/prepare_visualization_data.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/research/gap_detection_implementation_example.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/setup/GITHUB_PYPI_SETUP.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/validation/ARCHITECTURE_AUDIT_2025-10-17.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/validation/ARCHITECTURE_EFFICIENCY_AUDIT_2025-10-18.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/validation/DOCUMENTATION_SURVEY_v1.7.0.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/validation/E2E_VALIDATION_RESULTS_v1.6.0.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/validation/WEEKEND_GAP_VALIDATION_SUMMARY.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/docs/validation/WORKSPACE_SURVEY_2025-10-17.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/examples/basic_usage.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/examples/batch_processing.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/config.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/database_manager.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/downloader.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/exchanges.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/ohlc_generator.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/query_engine.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/schema.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/session_detector.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/src/exness_data_preprocess/tick_loader.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/tests/README.md +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/tests/__init__.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/tests/conftest.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/tests/test_basic.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/tests/test_functional_regression.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/tests/test_models.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/tests/test_processor_pydantic.py +0 -0
- {exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/tests/test_types.py +0 -0
{exness_data_preprocess-0.8.0 → exness_data_preprocess-1.1.0}/.lint-relative-paths-results.txt
RENAMED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
🔍 Scanning for non-repo-relative paths in: /Users/terryli/eon/exness-data-preprocess
|
|
2
2
|
Violations: paths not starting with / or http(s)
|
|
3
3
|
|
|
4
|
-
❌ Found
|
|
4
|
+
❌ Found 140 violation(s):
|
|
5
5
|
|
|
6
6
|
/Users/terryli/eon/exness-data-preprocess/README.md:112:- **Column Definitions**: See [`schema.py`](src/exness_data_preprocess/schema.py) - Single source of truth
|
|
7
7
|
/Users/terryli/eon/exness-data-preprocess/README.md:113:- **Comprehensive Reference**: See [`DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md) - Query examples and usage patterns
|
|
@@ -54,7 +54,7 @@
|
|
|
54
54
|
/Users/terryli/eon/exness-data-preprocess/docs/DATABASE_SCHEMA.md:244:**Column Definitions**: See [`../src/exness_data_preprocess/schema.py`](../src/exness_data_preprocess/schema.py) for complete column definitions with types and descriptions.
|
|
55
55
|
/Users/terryli/eon/exness-data-preprocess/docs/DATABASE_SCHEMA.md:258:**Architecture**: Exchange Registry Pattern (v1.6.0) - Session columns dynamically generated from centralized EXCHANGES dict in [`exchanges.py`](../src/exness_data_preprocess/exchanges.py)
|
|
56
56
|
|
|
57
|
-
... and
|
|
57
|
+
... and 90 more (showing first 50)
|
|
58
58
|
|
|
59
59
|
📝 Fix: Convert relative paths to repo-relative paths starting with /
|
|
60
60
|
WRONG: [Link](../docs/foo.md)
|
|
@@ -0,0 +1,472 @@
|
|
|
1
|
+
# mise configuration for exness-data-preprocess
|
|
2
|
+
# E2E validation workflow for codebase pruning verification
|
|
3
|
+
|
|
4
|
+
[env]
|
|
5
|
+
# Python version for validation
|
|
6
|
+
PYTHON_VERSION = "3.14"
|
|
7
|
+
# Package name
|
|
8
|
+
PACKAGE_NAME = "exness_data_preprocess"
|
|
9
|
+
# Validation temp directory
|
|
10
|
+
VALIDATION_TMP = "./tmp/validation"
|
|
11
|
+
|
|
12
|
+
[tools]
|
|
13
|
+
python = "3.14"
|
|
14
|
+
|
|
15
|
+
# =============================================================================
|
|
16
|
+
# Validation Tasks - E2E Testing Pipeline
|
|
17
|
+
# =============================================================================
|
|
18
|
+
|
|
19
|
+
[tasks.validate]
|
|
20
|
+
description = "Run full E2E validation pipeline"
|
|
21
|
+
# ADR: /docs/adr/2025-12-10-clickhouse-e2e-validation-pipeline.md
|
|
22
|
+
depends = [
|
|
23
|
+
"validate:imports",
|
|
24
|
+
"validate:lint",
|
|
25
|
+
"validate:typecheck",
|
|
26
|
+
"validate:test",
|
|
27
|
+
"validate:build",
|
|
28
|
+
"validate:install",
|
|
29
|
+
"validate:mixin",
|
|
30
|
+
]
|
|
31
|
+
run = '''
|
|
32
|
+
echo ""
|
|
33
|
+
echo "=============================================="
|
|
34
|
+
echo "✅ ALL VALIDATIONS PASSED"
|
|
35
|
+
echo "=============================================="
|
|
36
|
+
echo ""
|
|
37
|
+
echo "💡 To run ClickHouse validation (requires ClickHouse running):"
|
|
38
|
+
echo " mise run clickhouse:start"
|
|
39
|
+
echo " mise run validate:clickhouse"
|
|
40
|
+
'''
|
|
41
|
+
|
|
42
|
+
# -----------------------------------------------------------------------------
|
|
43
|
+
# Individual Validation Tasks
|
|
44
|
+
# -----------------------------------------------------------------------------
|
|
45
|
+
|
|
46
|
+
[tasks."validate:imports"]
|
|
47
|
+
description = "Validate all module imports (catch circular imports, missing deps)"
|
|
48
|
+
run = '''
|
|
49
|
+
#!/usr/bin/env bash
|
|
50
|
+
set -e
|
|
51
|
+
echo "🔍 Validating module imports..."
|
|
52
|
+
|
|
53
|
+
# Test package import
|
|
54
|
+
uv run python -c "import exness_data_preprocess" || { echo "❌ Package import failed"; exit 1; }
|
|
55
|
+
|
|
56
|
+
# Test all public exports
|
|
57
|
+
uv run python -c "
|
|
58
|
+
from exness_data_preprocess import (
|
|
59
|
+
ExnessDataProcessor,
|
|
60
|
+
UpdateResult,
|
|
61
|
+
CoverageInfo,
|
|
62
|
+
DryRunResult,
|
|
63
|
+
PairType,
|
|
64
|
+
TimeframeType,
|
|
65
|
+
VariantType,
|
|
66
|
+
supported_pairs,
|
|
67
|
+
supported_timeframes,
|
|
68
|
+
supported_variants,
|
|
69
|
+
# ClickHouse exports
|
|
70
|
+
ClickHouseManager,
|
|
71
|
+
ClickHouseGapDetector,
|
|
72
|
+
ClickHouseQueryEngine,
|
|
73
|
+
ClickHouseOHLCGenerator,
|
|
74
|
+
ClickHouseConnectionError,
|
|
75
|
+
ClickHouseQueryError,
|
|
76
|
+
get_clickhouse_client,
|
|
77
|
+
)
|
|
78
|
+
print('✅ All public exports importable')
|
|
79
|
+
" || { echo "❌ Public exports import failed"; exit 1; }
|
|
80
|
+
|
|
81
|
+
# Test all internal modules
|
|
82
|
+
uv run python -c "
|
|
83
|
+
modules = [
|
|
84
|
+
'exness_data_preprocess.clickhouse_base',
|
|
85
|
+
'exness_data_preprocess.clickhouse_client',
|
|
86
|
+
'exness_data_preprocess.clickhouse_gap_detector',
|
|
87
|
+
'exness_data_preprocess.clickhouse_manager',
|
|
88
|
+
'exness_data_preprocess.clickhouse_ohlc_generator',
|
|
89
|
+
'exness_data_preprocess.clickhouse_query_engine',
|
|
90
|
+
'exness_data_preprocess.config',
|
|
91
|
+
'exness_data_preprocess.database_manager',
|
|
92
|
+
'exness_data_preprocess.downloader',
|
|
93
|
+
'exness_data_preprocess.exchanges',
|
|
94
|
+
'exness_data_preprocess.gap_detector',
|
|
95
|
+
'exness_data_preprocess.models',
|
|
96
|
+
'exness_data_preprocess.ohlc_generator',
|
|
97
|
+
'exness_data_preprocess.processor',
|
|
98
|
+
'exness_data_preprocess.query_engine',
|
|
99
|
+
'exness_data_preprocess.schema',
|
|
100
|
+
'exness_data_preprocess.session_detector',
|
|
101
|
+
'exness_data_preprocess.tick_loader',
|
|
102
|
+
]
|
|
103
|
+
import importlib
|
|
104
|
+
for mod in modules:
|
|
105
|
+
importlib.import_module(mod)
|
|
106
|
+
print(f' ✓ {mod}')
|
|
107
|
+
print('✅ All 18 modules importable')
|
|
108
|
+
"
|
|
109
|
+
echo "✅ Import validation passed"
|
|
110
|
+
'''
|
|
111
|
+
|
|
112
|
+
[tasks."validate:lint"]
|
|
113
|
+
description = "Run ruff linting"
|
|
114
|
+
run = '''
|
|
115
|
+
#!/usr/bin/env bash
|
|
116
|
+
set -e
|
|
117
|
+
echo "🔍 Running ruff check..."
|
|
118
|
+
uv run ruff check src/ tests/ --output-format=concise
|
|
119
|
+
echo "✅ Lint validation passed"
|
|
120
|
+
'''
|
|
121
|
+
|
|
122
|
+
[tasks."validate:typecheck"]
|
|
123
|
+
description = "Run mypy type checking"
|
|
124
|
+
run = '''
|
|
125
|
+
#!/usr/bin/env bash
|
|
126
|
+
set -e
|
|
127
|
+
echo "🔍 Running mypy..."
|
|
128
|
+
uv run mypy src/exness_data_preprocess/ --ignore-missing-imports --no-error-summary 2>&1 | head -50
|
|
129
|
+
echo "✅ Type check validation passed (warnings may exist)"
|
|
130
|
+
'''
|
|
131
|
+
|
|
132
|
+
[tasks."validate:test"]
|
|
133
|
+
description = "Run pytest test suite"
|
|
134
|
+
run = '''
|
|
135
|
+
#!/usr/bin/env bash
|
|
136
|
+
set -e
|
|
137
|
+
echo "🔍 Running pytest..."
|
|
138
|
+
uv run pytest tests/ -v --tb=short -q
|
|
139
|
+
echo "✅ Test validation passed"
|
|
140
|
+
'''
|
|
141
|
+
|
|
142
|
+
[tasks."validate:test-coverage"]
|
|
143
|
+
description = "Run pytest with coverage report"
|
|
144
|
+
run = '''
|
|
145
|
+
#!/usr/bin/env bash
|
|
146
|
+
set -e
|
|
147
|
+
echo "🔍 Running pytest with coverage..."
|
|
148
|
+
uv run pytest tests/ --cov=exness_data_preprocess --cov-report=term-missing --cov-report=html:tmp/coverage
|
|
149
|
+
echo ""
|
|
150
|
+
echo "📊 Coverage report: tmp/coverage/index.html"
|
|
151
|
+
'''
|
|
152
|
+
|
|
153
|
+
[tasks."validate:build"]
|
|
154
|
+
description = "Validate package builds correctly"
|
|
155
|
+
run = '''
|
|
156
|
+
#!/usr/bin/env bash
|
|
157
|
+
set -e
|
|
158
|
+
echo "🔍 Building package..."
|
|
159
|
+
rm -rf dist/
|
|
160
|
+
uv build 2>&1 | tail -5
|
|
161
|
+
echo ""
|
|
162
|
+
ls -la dist/
|
|
163
|
+
echo "✅ Build validation passed"
|
|
164
|
+
'''
|
|
165
|
+
|
|
166
|
+
[tasks."validate:install"]
|
|
167
|
+
description = "Test fresh install in isolated environment"
|
|
168
|
+
depends = ["validate:build"]
|
|
169
|
+
run = '''
|
|
170
|
+
#!/usr/bin/env bash
|
|
171
|
+
set -e
|
|
172
|
+
echo "🔍 Testing fresh install..."
|
|
173
|
+
|
|
174
|
+
# Create isolated temp venv
|
|
175
|
+
TMPDIR=$(mktemp -d)
|
|
176
|
+
trap "rm -rf $TMPDIR" EXIT
|
|
177
|
+
|
|
178
|
+
# Create fresh venv and install wheel using uv
|
|
179
|
+
uv venv "$TMPDIR/venv" --python 3.14 -q
|
|
180
|
+
WHEEL=$(ls dist/*.whl | head -1)
|
|
181
|
+
uv pip install "$WHEEL" --python "$TMPDIR/venv/bin/python" -q
|
|
182
|
+
|
|
183
|
+
# Test import in fresh environment
|
|
184
|
+
"$TMPDIR/venv/bin/python" -c "
|
|
185
|
+
import exness_data_preprocess as edp
|
|
186
|
+
print(f'Version: {edp.__version__}')
|
|
187
|
+
print(f'Processor: {edp.ExnessDataProcessor}')
|
|
188
|
+
print(f'ClickHouseManager: {edp.ClickHouseManager}')
|
|
189
|
+
print('✅ Fresh install works')
|
|
190
|
+
"
|
|
191
|
+
echo "✅ Install validation passed"
|
|
192
|
+
'''
|
|
193
|
+
|
|
194
|
+
# -----------------------------------------------------------------------------
|
|
195
|
+
# Mixin Validation
|
|
196
|
+
# -----------------------------------------------------------------------------
|
|
197
|
+
|
|
198
|
+
[tasks."validate:mixin"]
|
|
199
|
+
description = "Validate ClickHouseClientMixin inheritance"
|
|
200
|
+
run = '''
|
|
201
|
+
#!/usr/bin/env bash
|
|
202
|
+
set -e
|
|
203
|
+
echo "🔍 Validating ClickHouseClientMixin..."
|
|
204
|
+
|
|
205
|
+
uv run python -c "
|
|
206
|
+
from exness_data_preprocess.clickhouse_base import ClickHouseClientMixin
|
|
207
|
+
from exness_data_preprocess.clickhouse_manager import ClickHouseManager
|
|
208
|
+
from exness_data_preprocess.clickhouse_gap_detector import ClickHouseGapDetector
|
|
209
|
+
from exness_data_preprocess.clickhouse_query_engine import ClickHouseQueryEngine
|
|
210
|
+
from exness_data_preprocess.clickhouse_ohlc_generator import ClickHouseOHLCGenerator
|
|
211
|
+
|
|
212
|
+
# Verify inheritance
|
|
213
|
+
classes = [
|
|
214
|
+
('ClickHouseManager', ClickHouseManager),
|
|
215
|
+
('ClickHouseGapDetector', ClickHouseGapDetector),
|
|
216
|
+
('ClickHouseQueryEngine', ClickHouseQueryEngine),
|
|
217
|
+
('ClickHouseOHLCGenerator', ClickHouseOHLCGenerator),
|
|
218
|
+
]
|
|
219
|
+
|
|
220
|
+
for name, cls in classes:
|
|
221
|
+
assert issubclass(cls, ClickHouseClientMixin), f'{name} should inherit from ClickHouseClientMixin'
|
|
222
|
+
print(f' ✓ {name} inherits ClickHouseClientMixin')
|
|
223
|
+
|
|
224
|
+
# Verify mixin methods exist at class level (properties/methods are on class, not instance)
|
|
225
|
+
for name, cls in classes:
|
|
226
|
+
assert hasattr(cls, 'client'), f'{name} missing client property'
|
|
227
|
+
assert hasattr(cls, 'close'), f'{name} missing close method'
|
|
228
|
+
assert hasattr(cls, '_init_client'), f'{name} missing _init_client method'
|
|
229
|
+
print(f' ✓ {name} has mixin methods')
|
|
230
|
+
|
|
231
|
+
print('✅ Mixin validation passed')
|
|
232
|
+
"
|
|
233
|
+
'''
|
|
234
|
+
|
|
235
|
+
# -----------------------------------------------------------------------------
|
|
236
|
+
# Quick Tasks
|
|
237
|
+
# -----------------------------------------------------------------------------
|
|
238
|
+
|
|
239
|
+
[tasks.test]
|
|
240
|
+
description = "Run tests (alias for validate:test)"
|
|
241
|
+
alias = "t"
|
|
242
|
+
run = "uv run pytest tests/ -v --tb=short"
|
|
243
|
+
|
|
244
|
+
[tasks.lint]
|
|
245
|
+
description = "Run linting (alias for validate:lint)"
|
|
246
|
+
alias = "l"
|
|
247
|
+
run = "uv run ruff check src/ tests/"
|
|
248
|
+
|
|
249
|
+
[tasks.fix]
|
|
250
|
+
description = "Auto-fix linting issues"
|
|
251
|
+
run = "uv run ruff check --fix src/ tests/ && uv run ruff format src/ tests/"
|
|
252
|
+
|
|
253
|
+
# -----------------------------------------------------------------------------
|
|
254
|
+
# CI Simulation
|
|
255
|
+
# -----------------------------------------------------------------------------
|
|
256
|
+
|
|
257
|
+
[tasks.ci]
|
|
258
|
+
description = "Simulate CI pipeline locally"
|
|
259
|
+
depends = ["validate"]
|
|
260
|
+
depends_post = ["validate:test-coverage"]
|
|
261
|
+
run = '''
|
|
262
|
+
echo ""
|
|
263
|
+
echo "=============================================="
|
|
264
|
+
echo "✅ CI SIMULATION COMPLETE"
|
|
265
|
+
echo "=============================================="
|
|
266
|
+
'''
|
|
267
|
+
|
|
268
|
+
# =============================================================================
|
|
269
|
+
# ClickHouse Server Management
|
|
270
|
+
# ADR: /docs/adr/2025-12-10-clickhouse-e2e-validation-pipeline.md
|
|
271
|
+
# =============================================================================
|
|
272
|
+
|
|
273
|
+
[tasks."clickhouse:start"]
|
|
274
|
+
description = "Start mise-installed ClickHouse server"
|
|
275
|
+
run = '''
|
|
276
|
+
#!/usr/bin/env bash
|
|
277
|
+
set -euo pipefail
|
|
278
|
+
|
|
279
|
+
SHIM="${HOME}/.local/share/mise/shims/clickhouse"
|
|
280
|
+
|
|
281
|
+
# Check mise ClickHouse installation
|
|
282
|
+
if [[ ! -x "${SHIM}" ]]; then
|
|
283
|
+
echo "ERROR: mise ClickHouse not found. Install: mise install clickhouse" >&2
|
|
284
|
+
exit 1
|
|
285
|
+
fi
|
|
286
|
+
|
|
287
|
+
# Check if already running
|
|
288
|
+
if nc -z localhost 8123 2>/dev/null; then
|
|
289
|
+
echo "✅ ClickHouse already running on port 8123"
|
|
290
|
+
"${SHIM}" client --query "SELECT version()"
|
|
291
|
+
exit 0
|
|
292
|
+
fi
|
|
293
|
+
|
|
294
|
+
# Start server in daemon mode
|
|
295
|
+
echo "Starting ClickHouse server..."
|
|
296
|
+
"${SHIM}" server --daemon
|
|
297
|
+
|
|
298
|
+
# Wait for server to be ready
|
|
299
|
+
for i in $(seq 1 10); do
|
|
300
|
+
if nc -z localhost 8123 2>/dev/null; then
|
|
301
|
+
echo "✅ ClickHouse server started"
|
|
302
|
+
"${SHIM}" client --query "SELECT version()"
|
|
303
|
+
exit 0
|
|
304
|
+
fi
|
|
305
|
+
sleep 1
|
|
306
|
+
done
|
|
307
|
+
|
|
308
|
+
echo "ERROR: ClickHouse failed to start" >&2
|
|
309
|
+
exit 1
|
|
310
|
+
'''
|
|
311
|
+
|
|
312
|
+
[tasks."clickhouse:stop"]
|
|
313
|
+
description = "Stop mise-installed ClickHouse server"
|
|
314
|
+
run = '''
|
|
315
|
+
#!/usr/bin/env bash
|
|
316
|
+
if pkill -f "clickhouse server" 2>/dev/null; then
|
|
317
|
+
echo "✅ ClickHouse server stopped"
|
|
318
|
+
else
|
|
319
|
+
echo "ClickHouse was not running"
|
|
320
|
+
fi
|
|
321
|
+
'''
|
|
322
|
+
|
|
323
|
+
[tasks."clickhouse:status"]
|
|
324
|
+
description = "Check ClickHouse connection via Python client"
|
|
325
|
+
run = '''
|
|
326
|
+
#!/usr/bin/env bash
|
|
327
|
+
set -e
|
|
328
|
+
|
|
329
|
+
if ! nc -z localhost 8123 2>/dev/null; then
|
|
330
|
+
echo "❌ ClickHouse not running on port 8123"
|
|
331
|
+
exit 1
|
|
332
|
+
fi
|
|
333
|
+
|
|
334
|
+
uv run python -c "
|
|
335
|
+
from exness_data_preprocess import get_clickhouse_client
|
|
336
|
+
from exness_data_preprocess.clickhouse_client import check_connection
|
|
337
|
+
client = get_clickhouse_client()
|
|
338
|
+
check_connection(client)
|
|
339
|
+
print('✅ ClickHouse connection OK')
|
|
340
|
+
client.close()
|
|
341
|
+
"
|
|
342
|
+
'''
|
|
343
|
+
|
|
344
|
+
# =============================================================================
|
|
345
|
+
# ClickHouse E2E Validation Tasks
|
|
346
|
+
# ADR: /docs/adr/2025-12-10-clickhouse-e2e-validation-pipeline.md
|
|
347
|
+
# =============================================================================
|
|
348
|
+
|
|
349
|
+
[tasks."validate:clickhouse"]
|
|
350
|
+
description = "Run all ClickHouse E2E validations with real data"
|
|
351
|
+
depends = ["clickhouse:status"]
|
|
352
|
+
run = '''
|
|
353
|
+
#!/usr/bin/env bash
|
|
354
|
+
set -e
|
|
355
|
+
echo "🔍 Running ClickHouse E2E validation..."
|
|
356
|
+
mise run validate:clickhouse-schema
|
|
357
|
+
mise run validate:clickhouse-data
|
|
358
|
+
mise run validate:clickhouse-pagination
|
|
359
|
+
mise run validate:clickhouse-ohlc
|
|
360
|
+
echo "✅ All ClickHouse validations passed"
|
|
361
|
+
'''
|
|
362
|
+
|
|
363
|
+
[tasks."validate:clickhouse-schema"]
|
|
364
|
+
description = "Validate ClickHouse schema creation"
|
|
365
|
+
depends = ["clickhouse:status"]
|
|
366
|
+
run = '''
|
|
367
|
+
#!/usr/bin/env bash
|
|
368
|
+
set -e
|
|
369
|
+
echo "🔍 Validating ClickHouse schema..."
|
|
370
|
+
uv run python -c "
|
|
371
|
+
from exness_data_preprocess import ClickHouseManager
|
|
372
|
+
|
|
373
|
+
manager = ClickHouseManager()
|
|
374
|
+
manager.ensure_schema()
|
|
375
|
+
print(' ✓ ticks table created')
|
|
376
|
+
print(' ✓ ohlc table created')
|
|
377
|
+
manager.close()
|
|
378
|
+
print('✅ Schema validation passed')
|
|
379
|
+
"
|
|
380
|
+
'''
|
|
381
|
+
|
|
382
|
+
[tasks."validate:clickhouse-data"]
|
|
383
|
+
description = "Validate ClickHouse data queries with real Exness data"
|
|
384
|
+
depends = ["validate:clickhouse-schema"]
|
|
385
|
+
run = '''
|
|
386
|
+
#!/usr/bin/env bash
|
|
387
|
+
set -e
|
|
388
|
+
echo "🔍 Validating ClickHouse data operations..."
|
|
389
|
+
uv run python -c "
|
|
390
|
+
from exness_data_preprocess import ClickHouseQueryEngine
|
|
391
|
+
|
|
392
|
+
engine = ClickHouseQueryEngine()
|
|
393
|
+
|
|
394
|
+
# Query with limit to verify functionality
|
|
395
|
+
df = engine.query_ticks('EURUSD', limit=1000)
|
|
396
|
+
print(f' ✓ query_ticks returned {len(df)} rows')
|
|
397
|
+
|
|
398
|
+
# Verify columns
|
|
399
|
+
expected_cols = ['timestamp', 'bid', 'ask']
|
|
400
|
+
for col in expected_cols:
|
|
401
|
+
assert col in df.columns, f'Missing column: {col}'
|
|
402
|
+
print(' ✓ All expected columns present')
|
|
403
|
+
|
|
404
|
+
engine.close()
|
|
405
|
+
print('✅ Data validation passed')
|
|
406
|
+
"
|
|
407
|
+
'''
|
|
408
|
+
|
|
409
|
+
[tasks."validate:clickhouse-pagination"]
|
|
410
|
+
description = "Validate ClickHouse pagination (LIMIT/OFFSET/cursor/batch)"
|
|
411
|
+
depends = ["validate:clickhouse-schema"]
|
|
412
|
+
run = '''
|
|
413
|
+
#!/usr/bin/env bash
|
|
414
|
+
set -e
|
|
415
|
+
echo "🔍 Validating ClickHouse pagination..."
|
|
416
|
+
uv run python -c "
|
|
417
|
+
from exness_data_preprocess import ClickHouseQueryEngine
|
|
418
|
+
|
|
419
|
+
engine = ClickHouseQueryEngine()
|
|
420
|
+
|
|
421
|
+
# Test LIMIT
|
|
422
|
+
df1 = engine.query_ticks('EURUSD', limit=100)
|
|
423
|
+
assert len(df1) <= 100, f'LIMIT failed: got {len(df1)} rows'
|
|
424
|
+
print(f' ✓ LIMIT=100 returned {len(df1)} rows')
|
|
425
|
+
|
|
426
|
+
# Test OFFSET
|
|
427
|
+
df2 = engine.query_ticks('EURUSD', limit=100, offset=100)
|
|
428
|
+
assert len(df2) <= 100, f'OFFSET failed: got {len(df2)} rows'
|
|
429
|
+
print(f' ✓ LIMIT=100 OFFSET=100 returned {len(df2)} rows')
|
|
430
|
+
|
|
431
|
+
# Test cursor-based pagination
|
|
432
|
+
result = engine.query_ticks_paginated('EURUSD', page_size=1000)
|
|
433
|
+
print(f' ✓ Cursor pagination: {len(result.data)} rows, has_more={result.has_more}')
|
|
434
|
+
|
|
435
|
+
# Test batch iterator
|
|
436
|
+
batches = list(engine.query_ticks_batches('EURUSD', batch_size=1000, max_batches=3))
|
|
437
|
+
print(f' ✓ Batch iterator: {len(batches)} batches')
|
|
438
|
+
|
|
439
|
+
engine.close()
|
|
440
|
+
print('✅ Pagination validation passed')
|
|
441
|
+
"
|
|
442
|
+
'''
|
|
443
|
+
|
|
444
|
+
[tasks."validate:clickhouse-ohlc"]
|
|
445
|
+
description = "Validate ClickHouse OHLC queries"
|
|
446
|
+
depends = ["validate:clickhouse-schema"]
|
|
447
|
+
run = '''
|
|
448
|
+
#!/usr/bin/env bash
|
|
449
|
+
set -e
|
|
450
|
+
echo "🔍 Validating ClickHouse OHLC queries..."
|
|
451
|
+
uv run python -c "
|
|
452
|
+
from exness_data_preprocess import ClickHouseQueryEngine
|
|
453
|
+
|
|
454
|
+
engine = ClickHouseQueryEngine()
|
|
455
|
+
|
|
456
|
+
# Query OHLC with limit
|
|
457
|
+
df = engine.query_ohlc('EURUSD', timeframe='1h', limit=100)
|
|
458
|
+
print(f' ✓ query_ohlc(1h) returned {len(df)} bars')
|
|
459
|
+
|
|
460
|
+
# Verify OHLC columns exist (if data present)
|
|
461
|
+
if len(df) > 0:
|
|
462
|
+
ohlc_cols = ['open', 'high', 'low', 'close']
|
|
463
|
+
for col in ohlc_cols:
|
|
464
|
+
assert col in df.columns, f'Missing OHLC column: {col}'
|
|
465
|
+
print(' ✓ All OHLC columns present')
|
|
466
|
+
else:
|
|
467
|
+
print(' ⚠ No OHLC data available (expected if no data loaded)')
|
|
468
|
+
|
|
469
|
+
engine.close()
|
|
470
|
+
print('✅ OHLC validation passed')
|
|
471
|
+
"
|
|
472
|
+
'''
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# semantic-release configuration for Python project
|
|
2
|
+
# Uses @semantic-release/exec instead of @semantic-release/npm
|
|
3
|
+
branches:
|
|
4
|
+
- main
|
|
5
|
+
|
|
6
|
+
plugins:
|
|
7
|
+
- "@semantic-release/commit-analyzer"
|
|
8
|
+
- "@semantic-release/release-notes-generator"
|
|
9
|
+
- "@semantic-release/changelog"
|
|
10
|
+
- - "@semantic-release/exec"
|
|
11
|
+
- prepareCmd: |
|
|
12
|
+
# Update version in pyproject.toml
|
|
13
|
+
sed -i '' 's/^version = ".*"/version = "${nextRelease.version}"/' pyproject.toml
|
|
14
|
+
# Update version in __init__.py
|
|
15
|
+
sed -i '' 's/^__version__ = ".*"/__version__ = "${nextRelease.version}"/' src/exness_data_preprocess/__init__.py
|
|
16
|
+
- - "@semantic-release/git"
|
|
17
|
+
- assets:
|
|
18
|
+
- pyproject.toml
|
|
19
|
+
- src/exness_data_preprocess/__init__.py
|
|
20
|
+
- CHANGELOG.md
|
|
21
|
+
message: "chore(release): bump version ${lastRelease.version} → ${nextRelease.version} [skip ci]"
|
|
22
|
+
- "@semantic-release/github"
|
|
@@ -1,9 +1,40 @@
|
|
|
1
|
-
# [
|
|
1
|
+
# [1.1.0](https://github.com/terrylica/exness-data-preprocess/compare/v1.0.0...v1.1.0) (2025-12-10)
|
|
2
|
+
|
|
3
|
+
|
|
4
|
+
### Features
|
|
5
|
+
|
|
6
|
+
* add ClickHouse E2E validation pipeline with pagination API ([2ec7325](https://github.com/terrylica/exness-data-preprocess/commit/2ec7325905459f3073e591af4f800b942de9a099))
|
|
7
|
+
|
|
8
|
+
# [1.0.0](https://github.com/terrylica/exness-data-preprocess/compare/v0.8.0...v1.0.0) (2025-12-10)
|
|
9
|
+
|
|
10
|
+
|
|
11
|
+
* feat!: prune codebase - remove unused deps, archive docs, extract mixin ([e0d48b8](https://github.com/terrylica/exness-data-preprocess/commit/e0d48b8fe7ba44d2c3bc326661102a0b3d3b33b7))
|
|
2
12
|
|
|
3
13
|
|
|
14
|
+
### BREAKING CHANGES
|
|
15
|
+
|
|
16
|
+
* Remove deprecated api.py and cli.py modules
|
|
17
|
+
|
|
18
|
+
Dependency cleanup:
|
|
19
|
+
- Remove httpx, pyarrow, polars (unused)
|
|
20
|
+
- Move tqdm to optional [examples] extra
|
|
21
|
+
- Consolidate dev dependencies to [dependency-groups]
|
|
22
|
+
|
|
23
|
+
Documentation archival:
|
|
24
|
+
- Archive 20 planning/research docs to docs/archive/
|
|
25
|
+
- Update .gitignore for node_modules, lychee patterns
|
|
26
|
+
|
|
27
|
+
Code refactoring:
|
|
28
|
+
- Extract ClickHouseClientMixin from 4 modules (~45 lines saved)
|
|
29
|
+
- Delete api.py (304 lines) and cli.py (252 lines)
|
|
30
|
+
|
|
31
|
+
ADR: docs/adr/2025-12-09-codebase-pruning.md
|
|
32
|
+
|
|
33
|
+
# [0.8.0](https://github.com/terrylica/exness-data-preprocess/compare/v0.7.2...v0.8.0) (2025-12-10)
|
|
34
|
+
|
|
4
35
|
### Features
|
|
5
36
|
|
|
6
|
-
|
|
37
|
+
- add ClickHouse backend for cloud deployment ([c9b51a3](https://github.com/terrylica/exness-data-preprocess/commit/c9b51a3b374eea74375cc86afdd8cb9391fe73e8))
|
|
7
38
|
|
|
8
39
|
# Changelog
|
|
9
40
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: exness-data-preprocess
|
|
3
|
-
Version:
|
|
3
|
+
Version: 1.1.0
|
|
4
4
|
Summary: Professional Exness forex tick data preprocessing with optimal compression (Parquet Zstd-22) and DuckDB OHLC generation. Provides efficient storage (9% smaller than ZIP) with lossless precision and direct queryability.
|
|
5
5
|
Project-URL: Homepage, https://github.com/terrylica/exness-data-preprocess
|
|
6
6
|
Project-URL: Documentation, https://github.com/terrylica/exness-data-preprocess#readme
|
|
@@ -12,7 +12,7 @@ Author-email: Eon Labs <terry@eonlabs.com>
|
|
|
12
12
|
Maintainer-email: Terry Li <terry@eonlabs.com>
|
|
13
13
|
License: MIT
|
|
14
14
|
License-File: LICENSE
|
|
15
|
-
Keywords: backtesting,compression,data-engineering,duckdb,efficient-storage,eurusd,exness,financial-data,forex,forex-data,lossless,microstructure,ohlc,pandas,parquet,preprocessing,
|
|
15
|
+
Keywords: backtesting,compression,data-engineering,duckdb,efficient-storage,eurusd,exness,financial-data,forex,forex-data,lossless,microstructure,ohlc,pandas,parquet,preprocessing,quantitative-finance,queryable,tick-data,time-series,zstd
|
|
16
16
|
Classifier: Development Status :: 3 - Alpha
|
|
17
17
|
Classifier: Intended Audience :: Developers
|
|
18
18
|
Classifier: Intended Audience :: Financial and Insurance Industry
|
|
@@ -29,21 +29,11 @@ Requires-Python: >=3.11
|
|
|
29
29
|
Requires-Dist: clickhouse-connect>=0.7.0
|
|
30
30
|
Requires-Dist: duckdb>=0.9.0
|
|
31
31
|
Requires-Dist: exchange-calendars>=4.5.6
|
|
32
|
-
Requires-Dist: httpx>=0.25.0
|
|
33
32
|
Requires-Dist: pandas>=2.0.0
|
|
34
|
-
Requires-Dist: pyarrow>=16.0.0
|
|
35
33
|
Requires-Dist: pydantic>=2.0.0
|
|
36
34
|
Requires-Dist: pyyaml>=6.0.0
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
Requires-Dist: black>=23.0.0; extra == 'dev'
|
|
40
|
-
Requires-Dist: mypy>=1.5.0; extra == 'dev'
|
|
41
|
-
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
|
|
42
|
-
Requires-Dist: pytest>=7.0.0; extra == 'dev'
|
|
43
|
-
Requires-Dist: ruff>=0.1.0; extra == 'dev'
|
|
44
|
-
Provides-Extra: test
|
|
45
|
-
Requires-Dist: pytest-cov>=4.0.0; extra == 'test'
|
|
46
|
-
Requires-Dist: pytest>=7.0.0; extra == 'test'
|
|
35
|
+
Provides-Extra: examples
|
|
36
|
+
Requires-Dist: tqdm>=4.65.0; extra == 'examples'
|
|
47
37
|
Description-Content-Type: text/markdown
|
|
48
38
|
|
|
49
39
|
# Exness Data Preprocess v2.0.0
|
|
@@ -55,11 +55,6 @@
|
|
|
55
55
|
|
|
56
56
|
### Implementation Architecture v1.7.0
|
|
57
57
|
|
|
58
|
-
**Refactoring Plans**:
|
|
59
|
-
|
|
60
|
-
- [`plans/PHASE7_v1.6.0_REFACTORING_PROGRESS.md`](plans/PHASE7_v1.6.0_REFACTORING_PROGRESS.md) - Complete refactoring progress
|
|
61
|
-
- [`plans/REFACTORING_CHECKLIST.md`](plans/REFACTORING_CHECKLIST.md) - Quick reference checklist
|
|
62
|
-
|
|
63
58
|
**Pattern**: Facade with 7 specialized modules
|
|
64
59
|
|
|
65
60
|
**Key Modules**:
|
|
@@ -261,5 +256,11 @@ curl -s "https://ticks.ex2archive.com/ticks/" | jq -r '.[] | .name' | grep -i "E
|
|
|
261
256
|
|
|
262
257
|
---
|
|
263
258
|
|
|
264
|
-
|
|
259
|
+
## Archived Documentation
|
|
260
|
+
|
|
261
|
+
Historical planning, research, and completed phase documents are archived in [`archive/`](archive/).
|
|
262
|
+
|
|
263
|
+
---
|
|
264
|
+
|
|
265
|
+
**Last Updated**: 2025-12-09
|
|
265
266
|
**Maintainer**: Terry Li <terry@eonlabs.com>
|