echoss-db 2.3.1__tar.gz → 2.4.0__tar.gz

{echoss_db-2.3.1/echoss_db.egg-info → echoss_db-2.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: echoss-db
-Version: 2.3.1
+Version: 2.4.0
 Summary: echoss AI Bigdata Solution - Database Query Package
 Author-email: ckkim <ckkim@12cm.co.kr>
 License-Expression: Apache-2.0
@@ -16,7 +16,7 @@ License-File: LICENSE
 Requires-Dist: pandas>=3.0
 Requires-Dist: sqlalchemy>=2.0.0
 Requires-Dist: PyMySQL>=1.0.2
-Requires-Dist: opensearch-py<3.0.0,>=2.8.0
+Requires-Dist: opensearch-py<4,>=3
 Requires-Dist: echoss-common<3,>=2.0.0
 Requires-Dist: psycopg[binary]<4.0.0,>=3.2
 Requires-Dist: qdrant-client[fastembed]<2.0.0,>=1.14.1
@@ -447,6 +447,34 @@ elastic.ping()
 elastic.info()
 ```
 
+### Chunk Output (2.4.0)
+
+---
+
+Large results as fixed-schema chunks: `res.schema` comes from DB metadata (MySQL/Postgres `information_schema`, OpenSearch mapping) and can be read before iteration; iterating yields `(rows, resume_token)`. Errors are raised, never swallowed.
+
+```python
+# MySQL / Postgres — keyset on a unique, sortable key column (integer or string)
+res = mysql.select_chunks('orders', key='id', columns=['id', 'status', 'created'],
+                          where='status = %s', params=('paid',), chunk_size=10000)
+res.schema            # e.g. {'id': 'int64', 'status': 'string', 'created': 'timestamp[us]'}
+for rows, token in res:
+    ...               # token = last key of the chunk; pass resume_token=token to continue
+
+# OpenSearch — PIT + search_after on a unique sort field
+res = elastic.search_chunks('shopby-log-v1-dev', sort_key='received_at', source_fields=['received_at', 'items'])
+
+# write with echoss-fileformat (overrides / type errors are handled there)
+from echoss_fileformat import FileUtil
+FileUtil.dump_chunks(res, 'out_dir')
+```
+
+- Values are in the declared type: OpenSearch `date` → `timestamp[ms,UTC]` (epoch-ms ints kept as is, sub-ms digits truncated), `nested`/`knn_vector` → `json_array`, `object` → `json_object`; SQL `json`/`jsonb` → JSON text (`string`). `date_nanos` is not supported (company policy: ms/us timestamps only).
+- One shape per column: values that do not match the schema (e.g. a list in a `keyword` field) are passed through and rejected by echoss-fileformat (`ChunkTypeError`); use its `overrides` to read them differently.
+- MySQL `DATETIME`: set `time_zone` (IANA name, e.g. `Asia/Manila`) in the `mysql` config section to get `timestamp[us,UTC]`; without it `timestamp[us]`.
+- Resuming re-reads from the token: rows changed in the meantime are reflected. `key`/`sort_key` must be unique, or rows may be skipped or repeated.
+- `where` is a caller-written SQL fragment (same trust level as `select()`); bind values via `params`.
+
 ### Qdrant
 
 ---
@@ -634,3 +662,4 @@ v2.2.0 qdrant generalized dict filter (breaking behavior: previously ignored key
 v2.2.2 qdrant admin/read-only API key support (`role` param, `api_key`/`read_only_api_key` config fields), config.yaml credential fields (mysql/postgres/elastic `passwd`, qdrant `api_key`/`read_only_api_key`) now accept `${VAR_NAME}` placeholders resolved from `.env`/OS env (new `python-dotenv` dependency)
 v2.3.0 config loading and logger now imported from `echoss-common` (`>=1.2.0,<3`) instead of `echoss-fileformat`; `echoss-fileformat` removed from install dependencies (declare it directly if you use it); raise pandas floor to 3.0. XML config files are now read by `echoss-common` (result keys may differ from the previous `echoss-fileformat` XML reader)
 v2.3.1 raise `echoss-common` floor to `>=2.0.0,<3` (no code change). With `echoss-common` 2.0: no default `logs/echoss.log` file (console only), and a missing config file path raises `FileNotFoundError` (with `echoss-common` 1.x: `ValueError`/`TypeError`)
+v2.4.0 chunk output: `MysqlQuery.select_chunks`, `PostgresQuery.select_chunks`, `ElasticSearch.search_chunks` (schema from DB metadata + chunks + resume token, OpenSearch via PIT); `faster_select_generator` and `next_scroll_chunk` now raise errors instead of ending silently (a scroll context that expired raises `NotFoundError` instead of returning `[]`); `opensearch-py>=3,<4`; MySQL config `time_zone`

{echoss_db-2.3.1 → echoss_db-2.4.0}/README.md

@@ -415,6 +415,34 @@ elastic.ping()
 elastic.info()
 ```
 
+### Chunk Output (2.4.0)
+
+---
+
+Large results as fixed-schema chunks: `res.schema` comes from DB metadata (MySQL/Postgres `information_schema`, OpenSearch mapping) and can be read before iteration; iterating yields `(rows, resume_token)`. Errors are raised, never swallowed.
+
+```python
+# MySQL / Postgres — keyset on a unique, sortable key column (integer or string)
+res = mysql.select_chunks('orders', key='id', columns=['id', 'status', 'created'],
+                          where='status = %s', params=('paid',), chunk_size=10000)
+res.schema            # e.g. {'id': 'int64', 'status': 'string', 'created': 'timestamp[us]'}
+for rows, token in res:
+    ...               # token = last key of the chunk; pass resume_token=token to continue
+
+# OpenSearch — PIT + search_after on a unique sort field
+res = elastic.search_chunks('shopby-log-v1-dev', sort_key='received_at', source_fields=['received_at', 'items'])
+
+# write with echoss-fileformat (overrides / type errors are handled there)
+from echoss_fileformat import FileUtil
+FileUtil.dump_chunks(res, 'out_dir')
+```
+
+- Values are in the declared type: OpenSearch `date` → `timestamp[ms,UTC]` (epoch-ms ints kept as is, sub-ms digits truncated), `nested`/`knn_vector` → `json_array`, `object` → `json_object`; SQL `json`/`jsonb` → JSON text (`string`). `date_nanos` is not supported (company policy: ms/us timestamps only).
+- One shape per column: values that do not match the schema (e.g. a list in a `keyword` field) are passed through and rejected by echoss-fileformat (`ChunkTypeError`); use its `overrides` to read them differently.
+- MySQL `DATETIME`: set `time_zone` (IANA name, e.g. `Asia/Manila`) in the `mysql` config section to get `timestamp[us,UTC]`; without it `timestamp[us]`.
+- Resuming re-reads from the token: rows changed in the meantime are reflected. `key`/`sort_key` must be unique, or rows may be skipped or repeated.
+- `where` is a caller-written SQL fragment (same trust level as `select()`); bind values via `params`.
+
 ### Qdrant
 
 ---
@@ -602,3 +630,4 @@ v2.2.0 qdrant generalized dict filter (breaking behavior: previously ignored key
 v2.2.2 qdrant admin/read-only API key support (`role` param, `api_key`/`read_only_api_key` config fields), config.yaml credential fields (mysql/postgres/elastic `passwd`, qdrant `api_key`/`read_only_api_key`) now accept `${VAR_NAME}` placeholders resolved from `.env`/OS env (new `python-dotenv` dependency)
 v2.3.0 config loading and logger now imported from `echoss-common` (`>=1.2.0,<3`) instead of `echoss-fileformat`; `echoss-fileformat` removed from install dependencies (declare it directly if you use it); raise pandas floor to 3.0. XML config files are now read by `echoss-common` (result keys may differ from the previous `echoss-fileformat` XML reader)
 v2.3.1 raise `echoss-common` floor to `>=2.0.0,<3` (no code change). With `echoss-common` 2.0: no default `logs/echoss.log` file (console only), and a missing config file path raises `FileNotFoundError` (with `echoss-common` 1.x: `ValueError`/`TypeError`)
+v2.4.0 chunk output: `MysqlQuery.select_chunks`, `PostgresQuery.select_chunks`, `ElasticSearch.search_chunks` (schema from DB metadata + chunks + resume token, OpenSearch via PIT); `faster_select_generator` and `next_scroll_chunk` now raise errors instead of ending silently (a scroll context that expired raises `NotFoundError` instead of returning `[]`); `opensearch-py>=3,<4`; MySQL config `time_zone`

echoss_db-2.4.0/echoss_db/chunk_output.py

@@ -0,0 +1,204 @@
+"""
+조각 출력 — 정답 스키마 + 그 형식의 값 조각 + 재개 표지
+
+Design Ref: docs/02-design/features/db-raw-result-chunk-output.design.md §3·§4.1
+- 스키마는 DB 메타데이터(OpenSearch 매핑, MySQL·Postgres information_schema)로 만든다
+- 값은 DB 가 선언한 형식의 값: 드라이버 표현이 선언 형식과 다를 때만 맞추고 값의 뜻은 바꾸지 않는다
+- 값 형식 검사·덮어쓰기는 echoss-fileformat(ChunkTypeError·overrides) 몫 — 여기서는 검사하지 않는다(§3.4)
+- 형식 이름은 echoss-fileformat chunk_schema.py 의 이름(문자열). pyarrow·echoss-fileformat 은 import 하지 않는다
+"""
+from __future__ import annotations
+
+import base64
+import datetime
+import uuid
+from typing import Iterable, Optional, Tuple
+from zoneinfo import ZoneInfo
+
+
+class ChunkResult:
+    """새 조각 함수의 결과. schema 는 반복 전에 읽을 수 있고, 반복하면 (rows, resume_token) 을 낸다"""
+
+    def __init__(self, schema: dict, chunks: Iterable[Tuple[list, object]]):
+        self.schema = schema
+        self.chunks = chunks
+
+    def __iter__(self):
+        return iter(self.chunks)
+
+
+# §3.1 OpenSearch — 공식 필드 형식 목록 전체 (docs.opensearch.org supported-field-types, 2026-10-10 조회)
+OPENSEARCH_TYPES = {
+    'boolean': 'bool', 'binary': 'binary',
+    'text': 'string', 'match_only_text': 'string', 'keyword': 'string', 'constant_keyword': 'string',
+    'icu_collation_keyword': 'string', 'wildcard': 'string', 'version': 'string',
+    'search_as_you_type': 'string', 'semantic': 'string', 'ip': 'string',
+    'token_count': 'int32', 'byte': 'int8', 'short': 'int16', 'integer': 'int32', 'long': 'int64',
+    'unsigned_long': 'uint64', 'half_float': 'float32', 'float': 'float32', 'double': 'float64',
+    'scaled_float': 'float64', 'rank_feature': 'float32',
+    'date': 'timestamp[ms,UTC]',
+    'geo_point': 'json_object', 'geo_shape': 'json_object', 'xy_point': 'json_object', 'xy_shape': 'json_object',
+    'integer_range': 'json_object', 'long_range': 'json_object', 'double_range': 'json_object',
+    'float_range': 'json_object', 'ip_range': 'json_object', 'date_range': 'json_object',
+    'object': 'json_object', 'flat_object': 'json_object', 'join': 'json_object',
+    'completion': 'json_object', 'percolator': 'json_object', 'rank_features': 'json_object',
+    'sparse_vector': 'json_object',
+    'nested': 'json_array', 'knn_vector': 'json_array',
+}
+OPENSEARCH_EXCLUDED = {'alias', 'star_tree', 'derived'}   # _source 에 없거나 문서 필드가 아님
+
+
+def opensearch_type_name(name: str, mapping_field: dict) -> Optional[str]:
+    """매핑 한 필드 → 형식 이름. None 이면 스키마에서 뺀다. 옮길 수 없는 형식은 ValueError"""
+    type_ = mapping_field.get('type', 'object' if 'properties' in mapping_field else None)
+    if type_ in OPENSEARCH_EXCLUDED:
+        return None
+    if type_ == 'date_nanos':
+        raise ValueError(f"[OpenSearch] field '{name}': date_nanos is not supported (company policy: ms/us timestamps only)")
+    if type_ not in OPENSEARCH_TYPES:
+        raise ValueError(f"[OpenSearch] field '{name}': unsupported mapping type {type_!r}")
+    return OPENSEARCH_TYPES[type_]
+
+
+# §3.2 MySQL
+MYSQL_INTS = {'tinyint': 'int8', 'smallint': 'int16', 'mediumint': 'int32', 'int': 'int32', 'bigint': 'int64'}
+MYSQL_STRINGS = {'char', 'varchar', 'tinytext', 'text', 'mediumtext', 'longtext', 'enum', 'set', 'json'}
+MYSQL_BINARIES = {'bit', 'binary', 'varbinary', 'tinyblob', 'blob', 'mediumblob', 'longblob', 'geometry', 'point',
+                  'linestring', 'polygon', 'multipoint', 'multilinestring', 'multipolygon', 'geometrycollection'}
+MYSQL_OTHERS = {'float': 'float32', 'double': 'float64', 'date': 'date32', 'timestamp': 'timestamp[us]',
+                'time': 'duration[us]', 'year': 'int16'}
+
+
+def mysql_type_name(column: dict, time_zone: Optional[str] = None) -> str:
+    """information_schema.columns 한 행(DATA_TYPE·COLUMN_TYPE·NUMERIC_PRECISION·NUMERIC_SCALE·COLUMN_NAME) → 형식 이름"""
+    name = column['COLUMN_NAME']
+    data_type = column['DATA_TYPE'].lower()
+    if data_type in MYSQL_INTS:
+        type_ = MYSQL_INTS[data_type]
+        return 'u' + type_ if 'unsigned' in column['COLUMN_TYPE'].lower() else type_
+    if data_type == 'decimal':
+        return decimal_type_name('MySQL', name, column['NUMERIC_PRECISION'], column['NUMERIC_SCALE'])
+    if data_type in MYSQL_STRINGS:
+        return 'string'   # json 은 JSON 텍스트 그대로(D13)
+    if data_type in MYSQL_BINARIES:
+        return 'binary'
+    if data_type == 'datetime':
+        return 'timestamp[us,UTC]' if time_zone else 'timestamp[us]'
+    if data_type in MYSQL_OTHERS:
+        return MYSQL_OTHERS[data_type]
+    raise ValueError(f"[MySQL] column '{name}': unsupported data type {data_type!r}")
+
+
+# §3.3 Postgres
+POSTGRES_TYPES = {
+    'smallint': 'int16', 'integer': 'int32', 'bigint': 'int64', 'real': 'float32', 'double precision': 'float64',
+    'boolean': 'bool', 'character': 'string', 'character varying': 'string', 'text': 'string', 'uuid': 'string',
+    'json': 'string', 'jsonb': 'string', 'bytea': 'binary', 'ARRAY': 'json_array', 'date': 'date32',
+    'timestamp without time zone': 'timestamp[us]', 'timestamp with time zone': 'timestamp[us,UTC]',
+    'time without time zone': 'duration[us]',
+}
+
+
+def postgres_type_name(column: dict) -> str:
+    """information_schema.columns 한 행(data_type·numeric_precision·numeric_scale·column_name) → 형식 이름"""
+    name = column['column_name']
+    data_type = column['data_type']
+    if data_type == 'numeric':
+        if column['numeric_precision'] is None:
+            raise ValueError(f"[Postgres] column '{name}': numeric without precision has no fixed type (cast it in a view)")
+        return decimal_type_name('Postgres', name, column['numeric_precision'], column['numeric_scale'])
+    if data_type in POSTGRES_TYPES:
+        return POSTGRES_TYPES[data_type]
+    raise ValueError(f"[Postgres] column '{name}': unsupported data type {data_type!r}")
+
+
+def decimal_type_name(db: str, name: str, precision, scale) -> str:
+    if int(precision) > 38:
+        raise ValueError(f"[{db}] column '{name}': decimal precision {precision} > 38")
+    return f"decimal({int(precision)},{int(scale or 0)})"
+
+
+# §3.4 값 맞추기 — 검사하지 않는다
+def convert_rows(rows: list, schema: dict, chunk_no: int, db: str, time_zone: Optional[str] = None) -> list:
+    """드라이버 표현을 선언 형식으로 맞춘 행. 변환 실패(날짜 해석, MySQL 0 날짜)만 ValueError"""
+    zone = ZoneInfo(time_zone) if time_zone else None
+    for row in rows:
+        for column, type_name in schema.items():
+            value = row.get(column)
+            if value is None:
+                continue
+            try:
+                row[column] = convert_value(value, type_name, db, zone)
+            except ValueError as e:
+                raise ValueError(f"[{db}] column '{column}' chunk {chunk_no}: {e}") from e
+    return rows
+
+
+def convert_value(value, type_name: str, db: str, zone):
+    if db == 'MySQL' and isinstance(value, str) and value.startswith('0000-00-00') \
+            and type_name in ('date32', 'timestamp[us]', 'timestamp[us,UTC]'):
+        raise ValueError(f"zero date {value!r}")
+    if type_name == 'timestamp[ms,UTC]':          # OpenSearch date (D6)
+        if isinstance(value, str):
+            return opensearch_date(value)
+        return value                               # epoch 밀리초 정수는 그대로
+    if type_name == 'timestamp[us,UTC]' and isinstance(value, datetime.datetime):
+        if value.tzinfo is None:                   # MySQL DATETIME: 서비스 현지 시각 + 연결 설정 시간대 (D7)
+            value = value.replace(tzinfo=zone)
+        return value.astimezone(datetime.timezone.utc)
+    if type_name == 'duration[us]' and isinstance(value, datetime.time):   # Postgres time: 자정부터의 경과(§3.3)
+        return datetime.timedelta(hours=value.hour, minutes=value.minute, seconds=value.second,
+                                  microseconds=value.microsecond)
+    if type_name == 'binary' and isinstance(value, str) and db == 'OpenSearch':
+        return base64.b64decode(value)
+    if type_name == 'string' and isinstance(value, uuid.UUID):
+        return str(value)
+    return value
+
+
+def opensearch_date(value: str) -> datetime.datetime:
+    """ISO 문자열 → UTC, 밀리초 아래는 자른다(OpenSearch 색인값과 같음). 날짜만 적힌 값은 그날 0시 UTC.
+    시간대 표기가 없으면 UTC 로 본다(OpenSearch date 해석 — 공식 문서 기준 지식)"""
+    try:
+        parsed = datetime.datetime.fromisoformat(value)
+    except ValueError as e:
+        raise ValueError(f"cannot parse date {value!r}") from e
+    if parsed.tzinfo is None:
+        parsed = parsed.replace(tzinfo=datetime.timezone.utc)
+    parsed = parsed.astimezone(datetime.timezone.utc)
+    return parsed.replace(microsecond=parsed.microsecond // 1000 * 1000)
+
+
+# §4.2 SQL keyset — MySQL·Postgres 공통 반복
+KEY_TYPES = {'int8', 'int16', 'int32', 'int64', 'uint8', 'uint16', 'uint32', 'uint64', 'string'}
+
+
+def keyset_chunks(fetch_page, key: str, chunk_size: int, resume_token, convert, drop_key: bool):
+    """fetch_page(last, n) → 행 목록(key 오름차순, key > last). 조각마다 (rows, 마지막 key 값) 을 낸다.
+    오류는 잡지 않는다 — 끝과 오류를 구분하게 다시 던진다(계약 7)"""
+    last = resume_token
+    chunk_no = 0
+    while True:
+        rows = fetch_page(last, chunk_size)
+        if not rows:
+            return
+        last = rows[-1][key]
+        if isinstance(last, uuid.UUID):            # 재개 표지도 선언 형식(string)으로 — echoss-fileformat 이 JSON 으로 기록(Check 게이트 R2)
+            last = str(last)
+        if drop_key:
+            for row in rows:
+                row.pop(key)
+        yield convert(rows, chunk_no), last
+        chunk_no += 1
+        if len(rows) < chunk_size:
+            return
+
+
+def sql_select_list(columns: list, quote, text_columns: set) -> str:
+    """식별자를 인용한 SELECT 목록. text_columns 는 ::text 로 읽는다(Postgres json·jsonb, D13)"""
+    return ", ".join(f"{quote(c)}::text AS {quote(c)}" if c in text_columns else quote(c) for c in columns)
+
+
+def check_key(db: str, key: str, schema_types: dict):
+    if schema_types[key] not in KEY_TYPES:
+        raise ValueError(f"[{db}] key '{key}' must be an integer or string column, got {schema_types[key]}")

{echoss_db-2.3.1 → echoss_db-2.4.0}/echoss_db/elastic_search.py

@@ -7,6 +7,7 @@ from typing import Any, List, Tuple, Dict, Optional, Union
 from echoss_common import dict_load, get_logger, set_logger_level
 
 from .env_config import resolve_credential
+from .chunk_output import ChunkResult, convert_rows, opensearch_type_name
 
 logger = get_logger("echoss_query")
 
@@ -121,6 +122,104 @@ class ElasticSearch:
         except Exception as e:
             raise ValueError("Connection failed by config. Please check config data")
 
+    def search_chunks(self, index: str, sort_key: str, query: dict = None, source_fields: List[str] = None,
+                      chunk_size: int = 10000, keep_alive: str = '5m', resume_token: list = None) -> ChunkResult:
+        """대량 조회를 「정답 스키마 + 값 조각 + 재개 표지」로 낸다 (Design Ref: §3.1·§4.2, PIT + search_after).
+
+        Args:
+            index: 인덱스 이름(별칭·패턴이면 묶인 인덱스들의 스키마가 같아야 한다)
+            sort_key: 정렬·재개에 쓸 유일한 필드. 유일하지 않으면 재개 때 문서가 빠지거나 겹칠 수 있다
+            query: 질의 본문의 query 부분(기본 match_all)
+            source_fields: 낼 필드 목록(점 경로 가능), 없으면 매핑 최상위 필드 전체
+            keep_alive: PIT 유지 시간
+            resume_token: 이전 조각의 재개 표지(search_after 정렬값). 새 PIT 를 열므로 그사이 바뀐 자료는 반영된다
+        Returns:
+            ChunkResult — schema 는 매핑에서, 값은 _source 에서(색인값 docvalue_fields 는 쓰지 않음)
+        """
+        try:
+            mappings = self.conn.indices.get_mapping(index=index)
+        except NotFoundError as e:
+            raise ValueError(f"[OpenSearch] index '{index}' not found") from e
+        schemas = [self.mapping_schema(m['mappings'].get('properties', {}), source_fields) for m in mappings.values()]
+        if not schemas:
+            raise ValueError(f"[OpenSearch] index '{index}' not found")
+        if any(s != schemas[0] for s in schemas[1:]):
+            raise ValueError(f"[OpenSearch] indices behind '{index}' have different mappings for the requested fields")
+        schema = schemas[0]
+        for m in mappings.values():   # 다중 필드(name.keyword)는 정렬 키로 받지 않는다(설계 §4.2)
+            if self.mapping_field(m['mappings'].get('properties', {}), sort_key) is None:
+                raise ValueError(f"[OpenSearch] sort_key '{sort_key}' not found in mapping of '{index}' (multi-fields are not supported)")
+
+        def chunks():
+            pit_id = self.conn.create_pit(index=index, keep_alive=keep_alive)['pit_id']
+            try:
+                last = resume_token
+                chunk_no = 0
+                while True:
+                    body = {"size": chunk_size, "query": query or {"match_all": {}}, "_source": list(schema),
+                            "sort": [{sort_key: "asc"}], "pit": {"id": pit_id, "keep_alive": keep_alive}}
+                    if last is not None:
+                        body["search_after"] = last
+                    response = self.conn.search(body=body)
+                    pit_id = response.get('pit_id', pit_id)
+                    hits = response['hits']['hits']
+                    if not hits:
+                        return
+                    try:
+                        rows = [{f: self.source_value(h.get('_source', {}), f) for f in schema} for h in hits]
+                    except ValueError as e:
+                        raise ValueError(f"[OpenSearch] chunk {chunk_no}: {e}") from e
+                    last = hits[-1]['sort']
+                    yield convert_rows(rows, schema, chunk_no, 'OpenSearch'), last
+                    chunk_no += 1
+                    if len(hits) < chunk_size:
+                        return
+            finally:
+                self.conn.delete_pit(body={"pit_id": [pit_id]})
+
+        return ChunkResult(schema, chunks())
+
+    @staticmethod
+    def mapping_field(properties: dict, path: str) -> Optional[dict]:
+        """점 경로(a.b)의 매핑 한 필드. 없으면 None"""
+        node = {'properties': properties}
+        walked = []
+        for part in path.split('.'):
+            if node.get('type') == 'nested':   # nested 하위 값은 문서마다 배열 — 한 열 한 모양이 안 됨(D13)
+                raise ValueError(f"[OpenSearch] field '{path}' is inside nested field '{'.'.join(walked)}' — select the nested field itself (json_array)")
+            node = node.get('properties', {}).get(part)
+            if node is None:
+                return None
+            walked.append(part)
+        return node
+
+    @classmethod
+    def mapping_schema(cls, properties: dict, source_fields: List[str] = None) -> dict:
+        """매핑 → {필드: 형식 이름}. 스키마에서 빼는 형식(alias 등)은 넣지 않는다"""
+        schema = {}
+        for name in (source_fields or list(properties)):
+            field = cls.mapping_field(properties, name)
+            if field is None:
+                raise ValueError(f"[OpenSearch] field '{name}' not found in mapping")
+            type_name = opensearch_type_name(name, field)
+            if type_name is not None:
+                schema[name] = type_name
+        return schema
+
+    @staticmethod
+    def source_value(source: dict, path: str):
+        """_source 의 점 경로 값. 없으면 None. 경로가 배열을 지나면 값을 고를 수 없어 ValueError(값을 버리지 않는다)"""
+        if path in source:             # 점이 든 키를 그대로 저장한 문서
+            return source[path]
+        value = source
+        for part in path.split('.'):
+            if value is None:
+                return None
+            if not isinstance(value, dict):
+                raise ValueError(f"field '{path}' crosses a {type(value).__name__} value — select its parent field")
+            value = value.get(part)
+        return value
+
     def _clear_scroll_context(self):
         """내부 스크롤 컨텍스트를 정리합니다."""
         if self._scroll_id:
@@ -168,7 +267,8 @@ class ElasticSearch:
         """
         준비된 스크롤에서 다음 문서 청크를 가져옵니다.
         첫 호출 시에는 초기 검색을, 이후 호출 시에는 스크롤 ID를 사용하여 데이터를 가져옵니다.
-        더 이상 문서가 없거나 에러 발생 시 빈 리스트를 반환하고 컨텍스트를 정리합니다.
+        더 이상 문서가 없으면 빈 리스트를 반환하고 컨텍스트를 정리합니다.
+        오류(scroll 문맥 만료 NotFoundError 포함)는 컨텍스트를 정리한 뒤 다시 던집니다 — 끝과 오류를 구분(2.4.0)
 
         :return: 문서 _source (dict) 리스트. 더 이상 문서가 없으면 빈 리스트.
         """
@@ -208,11 +308,11 @@ class ElasticSearch:
         except NotFoundError:
             logger.warning(f"Scroll context {self._scroll_id} not found or expired. Clearing context.")
             self._clear_scroll_context()
-            return []
+            raise
         except Exception as e:
             logger.error(f"An error occurred during scroll operation: {e}", exc_info=True)
             self._clear_scroll_context()  # 에러 발생 시 컨텍스트 정리
-            return []
+            raise
 
     def _bulk_operations(self, actions: list, chunk_size: int = 500, raise_on_error: bool = True,
                         max_retries: int = 3, request_timeout: int = 120) -> tuple[int, list]:
@@ -362,7 +462,7 @@ class ElasticSearch:
         """
         if index is None:
             index = self.index_name
-        return self.conn.exists(index, id)
+        return self.conn.exists(index=index, id=id)
 
     def search(self, body: dict = None, index=None) -> dict:
         """
@@ -568,7 +668,7 @@ class ElasticSearch:
         """
         if index is None:
             index = self.index_name
-        return self.conn.get_source(index, id)
+        return self.conn.get_source(index=index, id=id)
 
     def index(self, index: str, body: dict, id: str or int = None) -> Any:
         """
@@ -581,7 +681,7 @@ class ElasticSearch:
         Returns:
             생성 결과
         """
-        return self.conn.index(index, body, id=id)
+        return self.conn.index(index=index, body=body, id=id)
 
     def update(self, id: str or int, body: dict, index=None) -> Any:
         """
@@ -602,7 +702,7 @@ class ElasticSearch:
             doc_body = {
                 'doc' : body
             }
-        return self.conn.update(index, id, doc_body)
+        return self.conn.update(index=index, id=id, body=doc_body)
 
     def delete(self, id: str or int, index=None) -> Any:
         """
@@ -615,7 +715,7 @@ class ElasticSearch:
         """
         if index is None:
             index = self.index_name
-        return self.conn.delete(index, id)
+        return self.conn.delete(index=index, id=id)
 
     def delete_index(self, index):
         """
@@ -625,7 +725,7 @@ class ElasticSearch:
         Returns:
             result(str) : 처리 결과
         """
-        return self.conn.indices.delete(index)
+        return self.conn.indices.delete(index=index)
 
 
     def bulk(self, actions:list, index=None) -> Tuple[int, int]:

{echoss_db-2.3.1 → echoss_db-2.4.0}/echoss_db/mysql_query.py

@@ -12,6 +12,7 @@ from sqlalchemy.exc import SQLAlchemyError, DBAPIError
 from echoss_common import dict_load, get_logger
 from .env_config import resolve_credential
 from .sql_transaction import SQLTransaction
+from .chunk_output import ChunkResult, check_key, convert_rows, keyset_chunks, mysql_type_name, sql_select_list
 
 logger = get_logger("echoss_db")
 
@@ -149,6 +150,7 @@ class MysqlQuery:
             self.port = m.get('port', 3306)
             self.db = m['db']
             self.charset = m.get('charset', 'utf8mb4')
+            self.time_zone = m.get('time_zone')  # DATETIME 이 적힌 서비스 현지 시간대(IANA 이름) — select_chunks (Design §3.2)
         else:
             logger.error(f'[MySQL] config info not exist or any required keys are missing {required_keys}')
             raise ValueError("invalid conn_info")
@@ -447,7 +449,7 @@ class MysqlQuery:
     @parse_query('SELECT')
     def faster_select_generator(self, query_str: str, params=None, fetch_size=1000):
         """
-        대량 조회의 제너레이터 버전: DataFrame 청크를 yield 합니다.
+        대량 조회의 제너레이터 버전: list[dict] 청크를 yield 합니다. 오류는 다시 던집니다(2.4.0).
         - 메모리 피크를 최소화하고, 소비자 측에서 스트리밍 처리 가능
         - 사용 예:
             for chunk_list in mq.faster_select_generator("SELECT ... WHERE id>%s", (1000,), fetch_size=5000):
@@ -471,9 +473,65 @@ class MysqlQuery:
                     yield rows
 
         except SQLAlchemyError as e:
-            logger.debug(f"[MySQL] faster_select_generotor Exception : {e}")
-            # 제너레이터 내부 예외는 호출측에서 캐치 가능하도록 재전파하지 않음
-            return
+            logger.debug(f"[MySQL] faster_select_generator Exception : {e}")
+            raise
+
+    def select_chunks(self, table: str, key: str, columns: Optional[list] = None, where: Optional[str] = None,
+                      params=None, chunk_size: int = 10000, resume_token=None) -> ChunkResult:
+        """대량 조회를 「정답 스키마 + 값 조각 + 재개 표지」로 낸다 (Design Ref: §4.2, keyset).
+
+        Args:
+            table: 테이블 이름, 'db.table' 도 된다(없으면 연결한 db)
+            key: 유일하고 정렬 가능한 열(정수·문자열). 조각마다 그 조각 마지막 행의 key 값이 재개 표지
+            columns: 낼 열 목록, 없으면 전체
+            where: 호출자가 쓴 SQL 조건 조각 — select() 의 query_str 과 같은 신뢰 수준. 값은 params 로 바인드
+            params: where 의 바인드 값(use_percent_param 에 맞는 형식)
+            resume_token: 이전 조각의 재개 표지. 그사이 바뀐 행은 반영된다
+        Returns:
+            ChunkResult — schema 는 information_schema 에서. DATETIME 은 연결 설정 time_zone 이 있으면
+            그 시간대로 보고 UTC 로(timestamp[us,UTC]), 없으면 timestamp[us]. json 은 JSON 텍스트(string)
+        """
+        db_name, table_name = table.split('.', 1) if '.' in table else (self.db, table)
+        with self.engine.connect() as conn:
+            meta = conn.execute(text(
+                "SELECT COLUMN_NAME, DATA_TYPE, COLUMN_TYPE, NUMERIC_PRECISION, NUMERIC_SCALE "
+                "FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = :s AND TABLE_NAME = :t ORDER BY ORDINAL_POSITION"),
+                {"s": db_name, "t": table_name}).mappings().all()
+        by_name = {m['COLUMN_NAME']: dict(m) for m in meta}
+        if not by_name:
+            raise ValueError(f"[MySQL] table '{table}' not found")
+        names = list(columns) if columns else list(by_name)
+        missing = [c for c in names + [key] if c not in by_name]
+        if missing:
+            raise ValueError(f"[MySQL] columns not found in '{table}': {missing}")
+        types = {c: mysql_type_name(by_name[c], self.time_zone) for c in set(names) | {key}}
+        check_key('MySQL', key, types)
+        schema = {c: types[c] for c in names}
+        selected = names if key in names else names + [key]
+
+        def quote(name):
+            return '`' + name.replace('`', '``') + '`'
+
+        base = f"SELECT {sql_select_list(selected, quote, set())} FROM {quote(db_name)}.{quote(table_name)} " \
+               f"WHERE ({where or 'TRUE'})"
+
+        def chunks():
+            with self.engine.connect() as conn:
+                def fetch_page(last, n):
+                    if self.use_percent_param:
+                        q = base + (f" AND {quote(key)} > %s" if last is not None else "") + f" ORDER BY {quote(key)} LIMIT %s"
+                        p = tuple(params or ()) + ((last,) if last is not None else ()) + (n,)
+                    else:
+                        q = base + (f" AND {quote(key)} > :chunk_last" if last is not None else "") \
+                            + f" ORDER BY {quote(key)} LIMIT :chunk_limit"
+                        p = dict(params or {}, chunk_limit=n, **({'chunk_last': last} if last is not None else {}))
+                    return [dict(r) for r in self._execute_query(conn, q, p).mappings()]
+
+                yield from keyset_chunks(fetch_page, key, chunk_size, resume_token,
+                                         lambda rows, no: convert_rows(rows, schema, no, 'MySQL', self.time_zone),
+                                         drop_key=key not in names)
+
+        return ChunkResult(schema, chunks())
 
     # ----------------------------------------------------------------------------------
     # INSERT / UPDATE / DELETE