sec-core 0.1.0__tar.gz

sec_core-0.1.0/.gitignore

@@ -0,0 +1,138 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/

sec_core-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 chonglie
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sec_core-0.1.0/PKG-INFO

@@ -0,0 +1,42 @@
+Metadata-Version: 2.4
+Name: sec-core
+Version: 0.1.0
+Summary: Shared securities market-data protocol contracts
+License-File: LICENSE
+Requires-Python: >=3.11
+Requires-Dist: msgpack>=1.1.2
+Description-Content-Type: text/markdown
+
+# sec-core
+
+`sec-core` 是证券行情消息协议核心包，提供 subject 构造/解析、schema 常量、payload 校验、`Nats-Msg-Id` 生成和 MessagePack 编解码。
+
+安装：
+
+```bash
+uv add sec-core
+pip install sec-core
+```
+
+示例：
+
+```python
+from sec_core.market_data.subjects import format_tick_subject
+
+print(format_tick_subject("000001.SZ"))
+```
+
+输出：
+
+```text
+tick.v2.sz.000001
+```
+
+第一版支持：
+
+- `tick.v2.{market}.{code}`
+- `backfill.tick.v2.{market}.{code}`
+- `kl.v1.1m.{market}.{code}`
+- `backfill.kl.v1.1m.{market}.{code}`
+
+本包不依赖 `xtquant`、`nats-py` 或 `pyarrow`。

sec_core-0.1.0/README.md

@@ -0,0 +1,33 @@
+# sec-core
+
+`sec-core` 是证券行情消息协议核心包，提供 subject 构造/解析、schema 常量、payload 校验、`Nats-Msg-Id` 生成和 MessagePack 编解码。
+
+安装：
+
+```bash
+uv add sec-core
+pip install sec-core
+```
+
+示例：
+
+```python
+from sec_core.market_data.subjects import format_tick_subject
+
+print(format_tick_subject("000001.SZ"))
+```
+
+输出：
+
+```text
+tick.v2.sz.000001
+```
+
+第一版支持：
+
+- `tick.v2.{market}.{code}`
+- `backfill.tick.v2.{market}.{code}`
+- `kl.v1.1m.{market}.{code}`
+- `backfill.kl.v1.1m.{market}.{code}`
+
+本包不依赖 `xtquant`、`nats-py` 或 `pyarrow`。

sec_core-0.1.0/docs/superpowers/plans/2026-06-23-sec-core-extraction-publish-integration.md

@@ -0,0 +1,384 @@
+# sec-core Extraction, Publish, and Integration Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Build and publish the lightweight `sec-core` market-data protocol package, then migrate `securities_miniqmt` and `securities_lab` to consume it and verify real backfill E2E flows.
+
+**Architecture:** `sec-core` owns protocol contracts only: subject formatting/parsing, schema constants, payload validation, message-id generation, and MessagePack codec. `securities_miniqmt` keeps xtdata download and NATS publishing, but imports protocol helpers from `sec_core`. `securities_lab` keeps Arrow conversion and storage, but imports subject parsing, required fields, schema validation, and codec helpers from `sec_core`.
+
+**Tech Stack:** Python 3.11+, uv, hatchling/setuptools, msgpack, pytest, NATS JetStream, MiniQMT xtdata.
+
+## Global Constraints
+
+- `sec-core` PyPI project name is `sec-core`.
+- Python import name is `sec_core`.
+- Repository path is `D:\Workspace\python_projects\securities_core`.
+- `sec-core` first version is `0.1.0`.
+- `sec-core` must not depend on `xtquant`, `nats-py`, or `pyarrow`.
+- `sec-core` first release covers only protocol core, not Arrow schema or storage conversion.
+- Consumer-facing docs are written in Chinese; code comments may be Chinese where useful; log/output text remains English in existing services.
+- Real E2E evidence must include `securities_miniqmt` tick backfill, `securities_miniqmt` 1m backfill, and `securities_lab` consuming backfill prefixes for tick and 1m.
+
+---
+
+## File Structure
+
+### `D:\Workspace\python_projects\securities_core`
+
+- Create `pyproject.toml`: package metadata, hatchling backend, runtime dependency `msgpack`.
+- Create `README.md`: install, quick examples, supported schemas.
+- Create `src/sec_core/__init__.py`: package version export.
+- Create `src/sec_core/market_data/__init__.py`: public market-data API exports.
+- Create `src/sec_core/market_data/constants.py`: dataset/source/version constants.
+- Create `src/sec_core/market_data/subjects.py`: stock-code parsing, subject formatting, prefix helpers, subject parsing.
+- Create `src/sec_core/market_data/schemas.py`: field-type enum, field maps, required fields, schema version lookup.
+- Create `src/sec_core/market_data/validators.py`: payload validation.
+- Create `src/sec_core/market_data/message_ids.py`: `Nats-Msg-Id` helpers.
+- Create `src/sec_core/market_data/codec.py`: MessagePack encode/decode.
+- Create tests under `tests/` for every module above.
+- Create `docs/reports/2026-06-23-sec-core-release-report.md`: final release and integration report.
+
+### `D:\Workspace\python_projects\securities_miniqmt`
+
+- Modify `pyproject.toml`: add `sec-core`.
+- Modify protocol-adjacent code to import from `sec_core`:
+  - `src/miniqmt_nats/core/schema_version.py`
+  - `src/miniqmt_nats/core/subject_formatter.py`
+  - `src/miniqmt_nats/core/serializer.py`
+  - `src/miniqmt_nats/publisher/tick_publisher.py`
+  - `src/miniqmt_nats/kline/bar_publisher.py`
+- Keep xtdata normalizers in this repo.
+- Run unit tests and real `tests/integration/test_backfill_tick_e2e.py`, `tests/integration/test_backfill_kline_e2e.py`.
+
+### `D:\Workspace\python_projects\securities_lab`
+
+- Modify `pyproject.toml`: add `sec-core`.
+- Modify `src/sec_lab/tick_msg_handlers/transformer.py`:
+  - Use `sec_core.market_data.subjects.parse_market_data_subject`.
+  - Use `sec_core.market_data.schemas` for required fields and field type constants where compatible.
+  - Use `sec_core.market_data.validators.validate_payload`.
+  - Keep Arrow schema and Arrow row conversion local.
+- Modify message handlers that call `msgpack.unpackb` directly to use `sec_core.market_data.codec.decode_msgpack` where it improves consistency.
+- Add integration tests proving `backfill.tick.v2` and `backfill.kl.v1.1m` prefixes are accepted and converted.
+
+---
+
+## Task 1: Implement `sec-core` Package
+
+**Files:**
+- Create: `pyproject.toml`
+- Create: `README.md`
+- Create: `src/sec_core/__init__.py`
+- Create: `src/sec_core/market_data/*.py`
+- Create: `tests/test_*.py`
+
+- [ ] **Step 1: Create package scaffold**
+
+Create a src-layout project using hatchling:
+
+```toml
+[project]
+name = "sec-core"
+version = "0.1.0"
+description = "Shared securities market-data protocol contracts"
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+    "msgpack>=1.1.2",
+]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[dependency-groups]
+dev = [
+    "pytest>=9.0.3",
+]
+```
+
+- [ ] **Step 2: Add subject tests**
+
+Test:
+
+```python
+def test_format_subjects():
+    assert format_tick_subject("000001.SZ") == "tick.v2.sz.000001"
+    assert format_backfill_tick_subject("000001.SZ") == "backfill.tick.v2.sz.000001"
+    assert format_kline_subject("000001.SZ") == "kl.v1.1m.sz.000001"
+    assert format_backfill_kline_subject("000001.SZ") == "backfill.kl.v1.1m.sz.000001"
+```
+
+Run:
+
+```powershell
+uv run pytest tests/test_subjects.py -q
+```
+
+Expected before implementation: fails with missing modules.
+
+- [ ] **Step 3: Implement subject helpers**
+
+Implement:
+
+```python
+parse_stock_code()
+format_tick_subject()
+format_backfill_tick_subject()
+format_kline_subject()
+format_backfill_kline_subject()
+normalize_subject_prefix()
+wildcard_for_prefix()
+subject_for_prefix()
+parse_market_data_subject()
+```
+
+- [ ] **Step 4: Add schema and validator tests**
+
+Cover required fields, schema version mismatch, period mismatch, and missing fields.
+
+- [ ] **Step 5: Implement schema and validators**
+
+Implement `FieldType`, field maps, required fields, `validate_tick_v2_payload()`, `validate_kline_1m_v1_payload()`, and `validate_payload()`.
+
+- [ ] **Step 6: Add message-id and codec tests**
+
+Cover all four message-id formats and MessagePack Decimal-to-string behavior.
+
+- [ ] **Step 7: Implement message-id and codec helpers**
+
+Use `msgpack.packb(payload, use_bin_type=True, default=str)` and `msgpack.unpackb(data, raw=False)`.
+
+- [ ] **Step 8: Run package tests and build**
+
+```powershell
+uv run pytest -q
+uv build
+```
+
+Expected: tests pass and `dist/*.whl`, `dist/*.tar.gz` are created.
+
+- [ ] **Step 9: Commit**
+
+```powershell
+git add pyproject.toml README.md src tests
+git commit -m "feat: add market data protocol core"
+```
+
+---
+
+## Task 2: Publish `sec-core`
+
+**Files:**
+- Modify: release report after publishing.
+
+- [ ] **Step 1: Verify PyPI project name**
+
+Open or query `https://pypi.org/project/sec-core/`.
+
+Expected before first release: 404 or no existing project owned by another account.
+
+- [ ] **Step 2: Publish**
+
+```powershell
+uv publish
+```
+
+If PyPI token is not configured, stop and report the exact missing credential error.
+
+- [ ] **Step 3: Verify install from a clean temp directory**
+
+```powershell
+$tmp = Join-Path $env:TEMP "sec-core-install-check"
+Remove-Item -LiteralPath $tmp -Recurse -Force -ErrorAction SilentlyContinue
+New-Item -ItemType Directory -Force -Path $tmp | Out-Null
+Set-Location $tmp
+uv init --bare
+uv add sec-core
+uv run python -c "from sec_core.market_data.subjects import format_tick_subject; print(format_tick_subject('000001.SZ'))"
+```
+
+Expected:
+
+```text
+tick.v2.sz.000001
+```
+
+---
+
+## Task 3: Integrate `sec-core` into `securities_miniqmt`
+
+**Files:**
+- Modify: `pyproject.toml`
+- Modify: `src/miniqmt_nats/core/schema_version.py`
+- Modify: `src/miniqmt_nats/core/subject_formatter.py`
+- Modify: `src/miniqmt_nats/core/serializer.py`
+- Modify: `src/miniqmt_nats/publisher/tick_publisher.py`
+- Modify: `src/miniqmt_nats/kline/bar_publisher.py`
+- Modify tests only where imports or expected helper ownership changes.
+
+- [ ] **Step 1: Add dependency**
+
+```powershell
+uv add sec-core
+```
+
+- [ ] **Step 2: Replace duplicated helpers**
+
+Use `sec_core.market_data` for:
+
+```python
+format_tick_subject
+format_tick_versioned_subject
+format_backfill_tick_subject
+format_kline_subject
+format_backfill_kline_subject
+encode_msgpack
+DEFAULT_TICK_SCHEMA_VERSION
+DEFAULT_KLINE_SCHEMA_VERSION
+validate_tick_v2_payload
+```
+
+- [ ] **Step 3: Keep producer-only normalization local**
+
+Do not move MiniQMT raw tick normalization or K-line row normalization into `sec-core` in this task.
+
+- [ ] **Step 4: Run unit tests**
+
+```powershell
+uv run pytest tests/unit -q
+```
+
+Expected: all unit tests pass.
+
+- [ ] **Step 5: Run real backfill E2E tests**
+
+```powershell
+uv run pytest tests/integration/test_backfill_tick_e2e.py --run-e2e -v -s
+uv run pytest tests/integration/test_backfill_kline_e2e.py --run-e2e -v -s
+```
+
+Expected: both tests pass, writing to `BACKFILL_STREAM` and consuming `backfill.tick.v2.sz.000001` / `backfill.kl.v1.1m.sz.000001`.
+
+- [ ] **Step 6: Commit**
+
+```powershell
+git add pyproject.toml uv.lock src tests
+git commit -m "refactor: use sec-core market data contracts"
+```
+
+---
+
+## Task 4: Integrate `sec-core` into `securities_lab`
+
+**Files:**
+- Modify: `pyproject.toml`
+- Modify: `src/sec_lab/tick_msg_handlers/transformer.py`
+- Modify or add tests under `tests/`
+
+- [ ] **Step 1: Add dependency**
+
+```powershell
+uv add sec-core
+```
+
+- [ ] **Step 2: Replace duplicated subject and validation logic**
+
+Use `sec_core.market_data` for:
+
+```python
+parse_market_data_subject
+normalize_subject_prefix
+subject_for_prefix
+wildcard_for_prefix
+validate_payload
+TICK_V2_REQUIRED_FIELDS
+KLINE_1M_V1_REQUIRED_FIELDS
+decode_msgpack
+```
+
+- [ ] **Step 3: Keep Arrow conversion local**
+
+`securities_lab` keeps `pyarrow` schemas, `rows_to_arrow_table()`, and storage path logic.
+
+- [ ] **Step 4: Add backfill prefix tests**
+
+Add tests proving:
+
+```python
+convert_payload("backfill.tick.v2.sz.000001", tick_payload, tick_subject_prefix="backfill.tick.v2")
+convert_payload("backfill.kl.v1.1m.sz.000001", kline_payload, kline_1m_subject_prefix="backfill.kl.v1.1m")
+```
+
+both succeed and produce the same dataset names as realtime prefixes.
+
+- [ ] **Step 5: Add or run E2E backfill-prefix message test**
+
+Use real or existing integration harness to consume messages by configured prefixes:
+
+```powershell
+$env:NATS_SUBJECT_PREFIXES="backfill.tick.v2,backfill.kl.v1.1m"
+$env:NATS_TICK_SUBJECT_PREFIX="backfill.tick.v2"
+$env:NATS_KLINE_1M_SUBJECT_PREFIX="backfill.kl.v1.1m"
+uv run pytest tests/integration -q
+```
+
+If the existing integration suite is not safe to run globally, create a focused test for receiving and converting one tick and one 1m backfill message.
+
+- [ ] **Step 6: Commit**
+
+```powershell
+git add pyproject.toml uv.lock src tests
+git commit -m "refactor: use sec-core market data contracts"
+```
+
+---
+
+## Task 5: Final Verification Report
+
+**Files:**
+- Create: `D:\Workspace\python_projects\securities_core\docs\reports\2026-06-23-sec-core-release-report.md`
+
+- [ ] **Step 1: Collect evidence**
+
+Record exact outputs for:
+
+```powershell
+# securities_core
+uv run pytest -q
+uv build
+uv publish
+uv add sec-core install check
+
+# securities_miniqmt
+uv run pytest tests/unit -q
+uv run pytest tests/integration/test_backfill_tick_e2e.py --run-e2e -v -s
+uv run pytest tests/integration/test_backfill_kline_e2e.py --run-e2e -v -s
+
+# securities_lab
+uv run pytest focused backfill-prefix tests
+```
+
+- [ ] **Step 2: Write report**
+
+Report must include:
+
+- Published `sec-core` version.
+- PyPI URL.
+- Commit ids for all modified repositories.
+- MiniQMT tick backfill E2E result.
+- MiniQMT 1m backfill E2E result.
+- Securities Lab backfill tick prefix validation result.
+- Securities Lab backfill 1m prefix validation result.
+- Known limitations or skipped checks, if any.
+
+- [ ] **Step 3: Commit report**
+
+```powershell
+git add docs/reports/2026-06-23-sec-core-release-report.md
+git commit -m "docs: add sec-core release verification report"
+```
+