komachi 0.1.0__tar.gz

komachi-0.1.0/LICENSE.md

@@ -0,0 +1,228 @@
+# License
+
+Komachi is licensed under the **Apache License, Version 2.0**.
+
+Komachi は **Apache License 2.0** のもとで公開しています。
+
+Copyright 2026 Kamakura Quant Lab / 鎌倉クオンツラボ
+
+## Why Apache 2.0 rather than MIT
+
+Two clauses matter for a tool like this one, and MIT has neither.
+
+**Section 6 withholds trademark rights explicitly.** Komachi is published so a
+buyer can read what it does with their purchase token before running it, which
+only works if the name means something. The licence grants the code and not the
+name.
+
+**Section 4(b) requires a modified file to say it was modified.** A fork that
+changes what this tool does with a credential has to declare it. That is worth
+more here than the brevity MIT would buy.
+
+Neither clause restricts ordinary use. Fork it, ship it, sell something built
+on it; the terms below are the standard Apache 2.0 text.
+
+---
+
+
+                                     Apache License
+                               Version 2.0, January 2004
+                            http://www.apache.org/licenses/
+
+       TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+       1. Definitions.
+
+          "License" shall mean the terms and conditions for use, reproduction,
+          and distribution as defined by Sections 1 through 9 of this document.
+
+          "Licensor" shall mean the copyright owner or entity authorized by
+          the copyright owner that is granting the License.
+
+          "Legal Entity" shall mean the union of the acting entity and all
+          other entities that control, are controlled by, or are under common
+          control with that entity. For the purposes of this definition,
+          "control" means (i) the power, direct or indirect, to cause the
+          direction or management of such entity, whether by contract or
+          otherwise, or (ii) ownership of fifty percent (50%) or more of the
+          outstanding shares, or (iii) beneficial ownership of such entity.
+
+          "You" (or "Your") shall mean an individual or Legal Entity
+          exercising permissions granted by this License.
+
+          "Source" form shall mean the preferred form for making modifications,
+          including but not limited to software source code, documentation
+          source, and configuration files.
+
+          "Object" form shall mean any form resulting from mechanical
+          transformation or translation of a Source form, including but
+          not limited to compiled object code, generated documentation,
+          and conversions to other media types.
+
+          "Work" shall mean the work of authorship, whether in Source or
+          Object form, made available under the License, as indicated by a
+          copyright notice that is included in or attached to the work
+          (an example is provided in the Appendix below).
+
+          "Derivative Works" shall mean any work, whether in Source or Object
+          form, that is based on (or derived from) the Work and for which the
+          editorial revisions, annotations, elaborations, or other modifications
+          represent, as a whole, an original work of authorship. For the purposes
+          of this License, Derivative Works shall not include works that remain
+          separable from, or merely link (or bind by name) to the interfaces of,
+          the Work and Derivative Works thereof.
+
+          "Contribution" shall mean any work of authorship, including
+          the original version of the Work and any modifications or additions
+          to that Work or Derivative Works thereof, that is intentionally
+          submitted to Licensor for inclusion in the Work by the copyright owner
+          or by an individual or Legal Entity authorized to submit on behalf of
+          the copyright owner. For the purposes of this definition, "submitted"
+          means any form of electronic, verbal, or written communication sent
+          to the Licensor or its representatives, including but not limited to
+          communication on electronic mailing lists, source code control systems,
+          and issue tracking systems that are managed by, or on behalf of, the
+          Licensor for the purpose of discussing and improving the Work, but
+          excluding communication that is conspicuously marked or otherwise
+          designated in writing by the copyright owner as "Not a Contribution."
+
+          "Contributor" shall mean Licensor and any individual or Legal Entity
+          on behalf of whom a Contribution has been received by Licensor and
+          subsequently incorporated within the Work.
+
+       2. Grant of Copyright License. Subject to the terms and conditions of
+          this License, each Contributor hereby grants to You a perpetual,
+          worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+          copyright license to reproduce, prepare Derivative Works of,
+          publicly display, publicly perform, sublicense, and distribute the
+          Work and such Derivative Works in Source or Object form.
+
+       3. Grant of Patent License. Subject to the terms and conditions of
+          this License, each Contributor hereby grants to You a perpetual,
+          worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+          (except as stated in this section) patent license to make, have made,
+          use, offer to sell, sell, import, and otherwise transfer the Work,
+          where such license applies only to those patent claims licensable
+          by such Contributor that are necessarily infringed by their
+          Contribution(s) alone or by combination of their Contribution(s)
+          with the Work to which such Contribution(s) was submitted. If You
+          institute patent litigation against any entity (including a
+          cross-claim or counterclaim in a lawsuit) alleging that the Work
+          or a Contribution incorporated within the Work constitutes direct
+          or contributory patent infringement, then any patent licenses
+          granted to You under this License for that Work shall terminate
+          as of the date such litigation is filed.
+
+       4. Redistribution. You may reproduce and distribute copies of the
+          Work or Derivative Works thereof in any medium, with or without
+          modifications, and in Source or Object form, provided that You
+          meet the following conditions:
+
+          (a) You must give any other recipients of the Work or
+              Derivative Works a copy of this License; and
+
+          (b) You must cause any modified files to carry prominent notices
+              stating that You changed the files; and
+
+          (c) You must retain, in the Source form of any Derivative Works
+              that You distribute, all copyright, patent, trademark, and
+              attribution notices from the Source form of the Work,
+              excluding those notices that do not pertain to any part of
+              the Derivative Works; and
+
+          (d) If the Work includes a "NOTICE" text file as part of its
+              distribution, then any Derivative Works that You distribute must
+              include a readable copy of the attribution notices contained
+              within such NOTICE file, excluding those notices that do not
+              pertain to any part of the Derivative Works, in at least one
+              of the following places: within a NOTICE text file distributed
+              as part of the Derivative Works; within the Source form or
+              documentation, if provided along with the Derivative Works; or,
+              within a display generated by the Derivative Works, if and
+              wherever such third-party notices normally appear. The contents
+              of the NOTICE file are for informational purposes only and
+              do not modify the License. You may add Your own attribution
+              notices within Derivative Works that You distribute, alongside
+              or as an addendum to the NOTICE text from the Work, provided
+              that such additional attribution notices cannot be construed
+              as modifying the License.
+
+          You may add Your own copyright statement to Your modifications and
+          may provide additional or different license terms and conditions
+          for use, reproduction, or distribution of Your modifications, or
+          for any such Derivative Works as a whole, provided Your use,
+          reproduction, and distribution of the Work otherwise complies with
+          the conditions stated in this License.
+
+       5. Submission of Contributions. Unless You explicitly state otherwise,
+          any Contribution intentionally submitted for inclusion in the Work
+          by You to the Licensor shall be under the terms and conditions of
+          this License, without any additional terms or conditions.
+          Notwithstanding the above, nothing herein shall supersede or modify
+          the terms of any separate license agreement you may have executed
+          with Licensor regarding such Contributions.
+
+       6. Trademarks. This License does not grant permission to use the trade
+          names, trademarks, service marks, or product names of the Licensor,
+          except as required for reasonable and customary use in describing the
+          origin of the Work and reproducing the content of the NOTICE file.
+
+       7. Disclaimer of Warranty. Unless required by applicable law or
+          agreed to in writing, Licensor provides the Work (and each
+          Contributor provides its Contributions) on an "AS IS" BASIS,
+          WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+          implied, including, without limitation, any warranties or conditions
+          of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+          PARTICULAR PURPOSE. You are solely responsible for determining the
+          appropriateness of using or redistributing the Work and assume any
+          risks associated with Your exercise of permissions under this License.
+
+       8. Limitation of Liability. In no event and under no legal theory,
+          whether in tort (including negligence), contract, or otherwise,
+          unless required by applicable law (such as deliberate and grossly
+          negligent acts) or agreed to in writing, shall any Contributor be
+          liable to You for damages, including any direct, indirect, special,
+          incidental, or consequential damages of any character arising as a
+          result of this License or out of the use or inability to use the
+          Work (including but not limited to damages for loss of goodwill,
+          work stoppage, computer failure or malfunction, or any and all
+          other commercial damages or losses), even if such Contributor
+          has been advised of the possibility of such damages.
+
+       9. Accepting Warranty or Additional Liability. While redistributing
+          the Work or Derivative Works thereof, You may choose to offer,
+          and charge a fee for, acceptance of support, warranty, indemnity,
+          or other liability obligations and/or rights consistent with this
+          License. However, in accepting such obligations, You may act only
+          on Your own behalf and on Your sole responsibility, not on behalf
+          of any other Contributor, and only if You agree to indemnify,
+          defend, and hold each Contributor harmless for any liability
+          incurred by, or claims asserted against, such Contributor by reason
+          of your accepting any such warranty or additional liability.
+
+       END OF TERMS AND CONDITIONS
+
+       APPENDIX: How to apply the Apache License to your work.
+
+          To apply the Apache License to your work, attach the following
+          boilerplate notice, with the fields enclosed by brackets "[]"
+          replaced with your own identifying information. (Don't include
+          the brackets!)  The text should be enclosed in the appropriate
+          comment syntax for the file format. We also recommend that a
+          file or class name and description of purpose be included on the
+          same "printed page" as the copyright notice for easier
+          identification within third-party archives.
+
+       Copyright [yyyy] [name of copyright owner]
+
+       Licensed under the Apache License, Version 2.0 (the "License");
+       you may not use this file except in compliance with the License.
+       You may obtain a copy of the License at
+
+           http://www.apache.org/licenses/LICENSE-2.0
+
+       Unless required by applicable law or agreed to in writing, software
+       distributed under the License is distributed on an "AS IS" BASIS,
+       WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+       See the License for the specific language governing permissions and
+       limitations under the License.

komachi-0.1.0/PKG-INFO

@@ -0,0 +1,314 @@
+Metadata-Version: 2.4
+Name: komachi
+Version: 0.1.0
+Summary: Kamakura Quant Lab command line client for historical market data
+Author-email: Kamakura Quant Lab <support@kamakuraquantlab.jp>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://kamakuraquantlab.jp
+Project-URL: Documentation, https://kamakuraquantlab.jp/support/
+Project-URL: Source, https://github.com/kamakuraquantlab/Komachi
+Project-URL: The data, https://kamakuraquantlab.jp/data/
+Keywords: cryptocurrency,market-data,order-book,japan,quantitative-finance
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Financial and Insurance Industry
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Office/Business :: Financial :: Investment
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE.md
+Requires-Dist: httpx>=0.27
+Provides-Extra: duckdb
+Requires-Dist: duckdb>=1.0.0; extra == "duckdb"
+Requires-Dist: pyarrow>=15.0; extra == "duckdb"
+Requires-Dist: pandas; extra == "duckdb"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: duckdb>=1.0.0; extra == "dev"
+Requires-Dist: pyarrow>=15.0; extra == "dev"
+Requires-Dist: pandas; extra == "dev"
+Dynamic: license-file
+
+# Komachi
+
+**鎌倉クオンツラボ**のコマンドラインクライアントです。
+ヒストリカル市場データを取得し、DuckDB からそのまま参照できる構成で保存します。
+
+*[English README](README-en.md)*
+
+[Kamakura Quant Lab](https://kamakuraquantlab.jp) のデータは、2 つの方法で取得できます。
+収録範囲、スキーマ、品質基準は[データのページ](https://kamakuraquantlab.jp/data/)に記載しています。
+
+**ブラウザ** — [tsurugaoka](https://kamakuraquantlab.jp/tsurugaoka/)
+
+数日分を確認する用途に適しています。1 ファイルずつ、ブラウザの保存先に保存されます。
+
+**コマンドライン** — Komachi（本リポジトリ）
+
+Yukinoshita API（`yukinoshita.kamakuraquantlab.jp`）を呼び出すクライアントです。
+まとまった量を扱う場合はこちらを使用します。
+
+| 機能 | 内容 |
+|---|---|
+| 残高と配信状況の確認 | 利用できるマーケット、日付ごとの収録状況と品質を、取得前に確認できます |
+| 一括取得 | 範囲を指定してまとめて取得します。中断後は同じコマンドで再開し、チェックサムで検証します |
+| Hive 形式での保存 | 収集時と同じディレクトリ構成で保存するため、DuckDB から取り込み処理なしに参照できます |
+| 無償公開データの取り込み | Binance Vision と GMO コインの公開データを、日本時間の日付へ再分割して取り込みます。配信データと同一の条件で比較できます |
+| ローカル参照 | 手元のファイルの一覧、約定と気配の表示、DuckDB ビューの作成 |
+
+権限の判断と URL の署名は Yukinoshita が行います。
+ファイルの実体はオブジェクトストレージから直接取得するため、
+API がデータ本体を中継することはありません。
+
+## 1 クイックスタート
+
+28 market-day を使う場合の例
+
+### 1.1 サインイン
+
+トークンは
+[kamakuraquantlab.jp/tsurugaoka/](https://kamakuraquantlab.jp/tsurugaoka/)
+で発行します。
+手元にある注文番号とメールアドレスでサインインし、「トークンを表示」を押してください。
+表示はその 1 回だけですが、必要になればいつでも発行し直せます。
+発行し直すと、それまでのトークンは使えなくなります。
+
+```bash
+pip install 'komachi[duckdb]'
+komachi token set --token hk_...
+```
+
+最小構成は `pip install komachi`（`httpx` のみ）です。
+取り込み、DuckDB 連携、ファイル検査を使う場合は上記の `duckdb` エクストラを指定します。
+
+初回はデータの保存先を尋ねられ、実行ディレクトリの `.env` に記録されます。
+トークンは `.env` ではなく `~/.komachi/config.json`（パーミッション 0600）に保存されます。
+データディレクトリをそのままバージョン管理下に置いても、資格情報が混入しません。
+
+### 1.2 残高と期限の確認
+
+```bash
+komachi token status
+```
+
+```text
+Product        starter-4w
+Token status   active
+Access         active
+First used     2026-09-11T05:54:23+00:00
+Access until   2026-09-25T05:54:23+00:00
+Allowance      28 market-days  (4 market-weeks)
+Remaining      28 market-days  (4 market-weeks)
+
+Markets        BITBANK:BTC_SPOT, BITBANK:ETH_SPOT, ... COINCHECK:XRP_SPOT
+
+Active until 2026-09-25. Until then you can unlock market-days and re-download
+anything already unlocked as often as you like, at no further cost.
+```
+
+期限は 2 つあり、順番に効きます。
+
+| | 期間 | 起点 | 過ぎると |
+|---|---|---|---|
+| 利用開始期限 | 1 か月 | トークンの発行時 | 利用期間が始まらず、以後は使えません |
+| 利用期間 | 14 日 | **最初に使った時** | 新たなアンロックも配信も行われません |
+
+**利用開始期限**までに、一度 tsurugaoka にサインインする必要があります。
+この最初のサインインでトークンが発行され、同時に利用期間が始まります。
+期限を過ぎてからサインインしても利用期間は始まりません。
+
+**利用期間**はその最初の 1 回から数えます。
+受け取ってすぐに作業を始められなくても不利にならないよう、
+発行時ではなく初回利用時を起点にしています。
+
+利用期間の中では、アンロックした market-day を何度でも取得できます。
+ファイルを削除しても、ダウンロードに失敗しても、取り直しに残高は消費しません。
+回数の上限はありません。
+ダウンロードのたびに新しい URL を発行し、その URL は 1 時間で失効します。
+`komachi download` は未アンロックの日付をその場でアンロックしてから取得するため、
+アンロックのための操作は必要ありません。
+
+利用期間を過ぎると、新しい market-day のアンロックも、ダウンロード用 URL の発行も停止します。
+残高が残っていても使えません。
+`komachi download` は取得を始める前に、その旨を表示して終了します。
+tsurugaoka にはサインインでき、利用状況と終了日を確認できます。
+消費済みの market-day は消費済みのまま残り、返還はありません。
+期間内に必要なファイルを保存してください。
+
+実際には、ここに来た時点ですでに利用期間が始まっています。
+`komachi token set` がトークンを確認するために行う通信も「使った」1 回に数えるためです。
+そのため `komachi token status` には、期限の日付と残り日数が表示されます。
+
+日付も残り日数もサーバーが計算して返したものをそのまま表示しています。
+残り日数の計算には時計が必要ですが、その時計は手元のものではないからです。
+このツールが独自に期限を判断することはありません。
+
+### 1.3 収録状況の確認
+
+どのマーケットが使えるかを確認します。
+
+```bash
+komachi markets
+komachi catalog --market COINCHECK:BTC_SPOT --days 3
+```
+
+```text
+date         datasets                     rows      size    gap  note
+2026-09-01   OrderBook,Trade           264,447    24.8MB     8m
+2026-09-02   OrderBook,Trade           274,792    26.7MB     1m
+2026-09-03   OrderBook,Trade           280,067    27.9MB     9m
+
+3 market-day(s). 'gap' is minutes with no record in the JST day.
+```
+
+`gap` は、その日のうち記録のない分数です。取得前に品質を確認できます。
+
+### 1.4 取得
+
+```bash
+komachi download --market COINCHECK:BTC_SPOT --start 2025-07-01 --days 28
+```
+
+実行前に、対象期間・ファイル数・容量・消費する market-day 数を表示して確認を求めます。
+中断した場合は同じコマンドを再実行すれば、取得済みのファイルは飛ばして続きから再開します。
+すでに手元にあるファイルに対して残高を再消費することはありません。
+
+### 1.5 無償公開データの取り込み
+
+同じ期間の Binance と GMO コインの約定データを、無償公開元から取り込みます。
+残高は消費しません。
+
+```bash
+komachi binance-import --symbol BTC_USDT --start 2025-07-01 --end 2025-07-28
+komachi gmo-import     --symbol BTC_JPY  --start 2025-07-01 --end 2025-07-28
+```
+
+取り込み時に日本時間の日付へ再分割するため、配信データと同じ基準で比較できます。
+
+### 1.6 保存先の確認
+
+```bash
+komachi local
+```
+
+```text
+market                   dataset     days  range
+COINCHECK:BTC_SPOT       OrderBook     28  2025-07-01 .. 2025-07-28
+COINCHECK:BTC_SPOT       Trade         28  2025-07-01 .. 2025-07-28
+BINANCE:BTC_USDT         Trade         28  2025-07-01 .. 2025-07-28
+GMO:BTC_JPY              Trade         28  2025-07-01 .. 2025-07-28
+```
+
+### 1.7 中身の確認
+
+```bash
+komachi trades --market COINCHECK:BTC_SPOT --date 2025-07-01 --rows 3
+komachi book   --market COINCHECK:BTC_SPOT --date 2025-07-01 --rows 3
+```
+
+```text
+time (JST)    side             price            size
+00:00:00.000  buy    15,456,978.0000      0.03000000
+00:00:01.000  buy    15,458,282.0000      0.03102218
+
+time (JST)            best bid        best ask      spread
+00:00:00.000   15,456,905.0000 15,456,978.0000     73.0000
+00:00:00.000   15,453,307.0000 15,458,283.0000  4,976.0000
+```
+
+### 1.8 分析へ
+
+```bash
+komachi duckdb
+duckdb ~/kql-data/kql.duckdb
+```
+
+```sql
+SELECT date, count(*) AS trades, avg(price) AS vwap
+FROM trade
+WHERE exchange = 'COINCHECK' AND symbol = 'BTC_SPOT'
+GROUP BY date ORDER BY date;
+```
+
+作図と派生データの作成は Komachi の範囲外です。
+分析ツールキットの Hase が担当します（公開準備中）。
+
+## 2 コマンド一覧
+
+### 2.1 リモート操作
+
+Yukinoshita API または無償公開元との通信を伴うコマンドです。
+
+| コマンド | 内容 | 残高の消費 |
+|---|---|---|
+| `komachi token set --token TOKEN` | トークンを検証して保存 | なし |
+| `komachi token status` | 付与数、使用済み、残高、2 つの期限 | なし |
+| `komachi markets` | 利用できるマーケットと、無償公開元の案内 | なし |
+| `komachi calendar --market MARKET` | 公開済みの日付と、取得済みの日付 | なし |
+| `komachi catalog --market MARKET` | 日ごとの収録状況と品質 | なし |
+| `komachi download --market MARKET --start DATE [--days N]` | 範囲を指定して取得（中断後は再開） | **あり** |
+| `komachi binance-import --symbol SYMBOL --start DATE --end DATE` | Binance Vision から取り込み | なし |
+| `komachi gmo-import --symbol SYMBOL --start DATE --end DATE` | GMO コインの公開データから取り込み | なし |
+
+残高を消費するのは `download` のみです。
+取得前に、対象期間・ファイル数・容量・消費数を表示して確認を求めます。
+market-day は 1 マーケットの 1 日分で、同じ日の板と約定を合わせて 1 と数えます。
+
+### 2.2 ローカル操作
+
+手元のファイルだけを参照します。通信も残高の消費もありません。
+
+| コマンド | 内容 |
+|---|---|
+| `komachi local` | 保存済みファイルの一覧 |
+| `komachi trades --market MARKET --date DATE` | 約定データの表示 |
+| `komachi book --market MARKET --date DATE` | 最良気配とスプレッドの表示 |
+| `komachi stats --path PATH` | 行数と時間範囲 |
+| `komachi decode --path PATH` | スキーマと先頭数行 |
+| `komachi duckdb` | DuckDB のビューを作成・更新 |
+| `komachi sql` | ビュー定義の SQL を出力 |
+
+## 3 保存先の構成
+
+データは、収集時と同じ Hive 形式のディレクトリに保存されます。
+
+```text
+$KQL_ROOT_PATH/bronze/dataset=Trade/exchange=GMO/symbol=BTC_JPY/date=2026-01-15/data.parquet
+```
+
+DuckDB、PyArrow、Spark、Athena のいずれも `dataset`・`exchange`・`symbol`・`date` を
+パスから列として認識します。1 年分でも次の 1 行で読み込めます。
+
+```sql
+SELECT * FROM read_parquet('~/kql-data/bronze/dataset=Trade/**/*.parquet', hive_partitioning = 1);
+```
+
+## 4 日付の扱い
+
+`date=` は**日本時間の 1 日**です。`date=2026-01-15` は 2026-01-15 00:00〜23:59 JST、
+UTC では 2026-01-14 15:00〜2026-01-15 14:59 にあたります。
+ファイル内のタイムスタンプは UTC エポック秒のため、実行環境のタイムゾーンに依存しません。
+
+取り込み元の日付区切りは取引所ごとに異なります。
+Binance Vision は 00:00 UTC、GMO コインは取引日の切り替えである 21:00 UTC を基準としています。
+Komachi は取り込み時に日本時間の日付へ再分割するため、
+配信データと無償公開データを同じ基準で比較できます。
+
+このため、ある 1 日を取り込むには前日分の元ファイルも必要です。
+どちらか一方しか取得できない日は、不完全なまま書き出さずに保留します。
+
+## 5 Komachi が行わないこと
+
+1. 取引所 API への直接接続。接続先は Yukinoshita API、Binance Vision、GMO コインの公開データのみです。
+2. データの加工・導出。silver 以降は [Hase](https://github.com/kamakuraquantlab) が担当します。
+3. Kamakura Quant Lab の API 以外に対する資格情報の保持。
+
+## 6 ライセンス
+
+Apache License 2.0。[LICENSE.md](LICENSE.md)