AeroViz 0.1.21__py3-none-any.whl

AeroViz/rawDataReader/FLOW.md

@@ -0,0 +1,138 @@
+# RawDataReader 處理流程
+
+## 流程圖
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                            RawDataReader(inst, path, start, end, qc=True)   │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                        │
+                                        ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  1. _raw_reader()                                                           │
+│     ├─ 讀取原始檔案 (*.dat, *.txt, *.csv)                                    │
+│     ├─ 時間解析度: 儀器原生 (1min/5min/6min/1h)                              │
+│     └─ 欄位: 原始量測值                                                      │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                        │
+                                        ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  2. _timeIndex_process()                                                    │
+│     └─ 對齊標準時間索引                                                      │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                        │
+                                        ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  3. _QC()                                                                   │
+│     ├─ QCFlagBuilder 根據規則標記資料                                        │
+│     ├─ 新增 QC_Flag 欄位 ("Valid" / "Status Error" / "Insufficient" / ...)  │
+│     └─ 暫存 QC Summary (待 _process 輸出)                                    │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                        │
+                                        ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  4. _process()                                                              │
+│     ├─ 計算衍生參數 (BC, Abs, AAE, GMD, GSD, etc.)                          │
+│     ├─ 額外 QC 驗證 (如 Invalid AAE) 並更新 QC_Flag                          │
+│     └─ 輸出完整 QC Summary (含 _QC 規則 + _process 驗證結果)                  │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                        │
+                                        ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  5. _save_data()                                                            │
+│     ├─ 儲存 raw_data.pkl / raw_data.csv                                     │
+│     └─ 儲存 qc_data.pkl / qc_data.csv                                       │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                        │
+                                        ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  6. __call__() - QC 處理                                                    │
+│     ├─ 根據 QC_Flag 將無效資料設為 NaN                                       │
+│     └─ 移除 QC_Flag 欄位                                                    │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                        │
+                                        ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  7. _generate_report()                                                      │
+│     ├─ calculate_rates(raw_data, qc_flag)                                   │
+│     │   ├─ Acquisition Rate: 有資料的時段 / 期望時段                         │
+│     │   ├─ Yield Rate: 通過 QC 的時段 / 有資料的時段                         │
+│     │   └─ Total Rate: 通過 QC 的時段 / 期望時段                             │
+│     └─ process_rates_report() - 生成週/月報告                                │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                        │
+                                        ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  8. 最終輸出                                                                 │
+│     ├─ resample(mean_freq) - 重採樣至使用者指定頻率 (預設 1h)                 │
+│     ├─ 儲存 output_{inst}.csv                                               │
+│     ├─ 儲存 output_{inst}_dN/dS/dVdlogDp.csv (SMPS/APS)                     │
+│     ├─ 儲存 report.json                                                     │
+│     └─ 返回 DataFrame                                                       │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+## 資料欄位變化
+
+| 階段 | 欄位範例 (AE33) | 欄位範例 (APS) |
+|-----|----------------|---------------|
+| `_raw_reader` | BC1-7, ATN, Flow, Status... | 粒徑 bins (0.5-20 μm) |
+| `_QC` | + `QC_Flag` | + `QC_Flag` |
+| `_process` | + Abs_370-950, AAE | + total, GMD, GSD, mode (num/surf/vol) |
+| `__call__` | 無效資料 → NaN，移除 QC_Flag | 無效資料 → NaN，移除 QC_Flag |
+| 最終輸出 | BC, Abs, AAE | 統計參數 (無 bins) |
+
+## QC_Flag 的角色
+
+```
+QC_Flag 貫穿整個流程：
+  _QC()              → 建立 QC_Flag，暫存 Summary
+  _process()         → 更新 QC_Flag (如 Invalid AAE)，輸出完整 QC Summary
+  _generate_report() → 使用 QC_Flag 計算 rates
+  __call__()         → 使用 QC_Flag 標記無效資料為 NaN，然後移除
+```
+
+## QC Summary 輸出格式
+
+```
+AE33 QC Summary:
+  Status Error: 24312 (4.9%)
+  Invalid BC: 29265 (5.9%)
+  Insufficient: 105481 (21.1%)
+  Invalid AAE: 25948 (5.2%)
+  Valid: 356025 (71.2%)
+```
+
+## 各儀器原生時間解析度
+
+| 儀器 | 原生頻率 | 主要輸出參數 |
+|-----|---------|-------------|
+| **NEPH** | 5 min | 散射係數 (G), sca_550, SAE |
+| **Aurora** | 1 min | 散射係數 (G), sca_550, SAE |
+| **SMPS** | 6 min | total, GMD, GSD, mode (num/surf/vol) |
+| **APS** | 6 min | total (1μm/2.5μm/all), GMD, GSD, mode |
+| **GRIMM** | 6 min | 粒徑分佈統計 |
+| **AE33** | 1 min | BC, Abs coef, AAE |
+| **AE43** | 1 min | BC, Abs coef, AAE |
+| **BC1054** | 1 min | BC, Abs coef, AAE |
+| **MA350** | 1 min | BC, Abs coef, AAE |
+| **BAM1020** | 1 h | PM 質量濃度 |
+| **TEOM** | 6 min | PM_Total, PM_NV |
+| **OCEC** | 1 h | Thermal/Optical OC & EC |
+| **IGAC** | 1 h | 離子濃度 (9種) |
+| **XRF** | 1 h | 元素濃度 |
+| **VOC** | 1 h | VOC 濃度 |
+| **EPA** | 1 h | 環保署監測數據 |
+
+## Rate 計算說明
+
+- **Acquisition Rate**: `有資料的時段數 / 期望的時段數 × 100%`
+  - 反映儀器的資料取得率
+
+- **Yield Rate**: `通過 QC 的時段數 / 有資料的時段數 × 100%`
+  - 反映資料的品質良率
+
+- **Total Rate**: `通過 QC 的時段數 / 期望的時段數 × 100%`
+  - 反映整體有效資料比例
+
+計算時使用 QC_Flag，每個時段內若超過 50% 的資料點為 "Valid"，則該時段視為有效。

AeroViz/rawDataReader/__init__.py

@@ -0,0 +1,220 @@
+from datetime import datetime
+from pathlib import Path
+from typing import Literal
+
+from pandas import Grouper, Timedelta
+
+from AeroViz.rawDataReader.config.supported_instruments import meta
+from AeroViz.rawDataReader.script import *
+
+__all__ = ['RawDataReader']
+
+
+def RawDataReader(instrument: str,
+                  path: Path | str,
+                  reset: bool | str = False,
+                  qc: bool | str = True,
+                  start: datetime | str = None,
+                  end: datetime | str = None,
+                  mean_freq: str = '1h',
+                  size_range: tuple[float, float] | None = None,
+                  suppress_warnings: bool = False,
+                  log_level: Literal['DEBUG', 'INFO', 'WARNING', 'ERROR'] = 'INFO',
+                  **kwargs):
+    """
+    Factory function to instantiate the appropriate reader module for a given instrument and
+    return the processed data over the specified time range.
+
+    Parameters
+    ----------
+    instrument : str
+        The instrument name for which to read data, must be a valid key in the meta dictionary
+
+    path : Path or str
+        The directory where raw data files for the instrument are stored
+
+    reset : bool or str
+        Data processing control mode:
+        False (default) - Use existing processed data if available
+        True - Force reprocess all data from raw files
+        'append' - Add new data to existing processed data
+
+    qc : bool or str
+        Quality control and rate calculation mode:
+        True (default) - Apply QC and calculate overall rates
+        False - Skip QC and return raw data only
+        str - Calculate rates at specified intervals:
+            'W' - Weekly rates
+            'MS' - Month start rates
+            'QS' - Quarter start rates
+            'YS' - Year start rates
+            Can add number prefix (e.g., '2MS' for bi-monthly)
+
+    start : datetime
+        Start time for filtering the data
+
+    end : datetime
+        End time for filtering the data
+
+    mean_freq : str
+        Resampling frequency for averaging the data (e.g., '1h' for hourly mean)
+
+    size_range : tuple[float, float], optional
+        Size range in nanometers (min_size, max_size) for SMPS/APS data filtering
+
+    suppress_warnings : bool, optional
+        Whether to suppress warning messages (default: False)
+
+    log_level : {'DEBUG', 'INFO', 'WARNING', 'ERROR'}
+        Logging level (default: 'INFO')
+
+    **kwargs
+        Additional arguments to pass to the reader module
+
+    Returns
+    -------
+    pd.DataFrame
+        Processed data with specified QC and time range
+
+    Raises
+    ------
+    ValueError
+        If QC mode or mean_freq format is invalid
+    TypeError
+        If parameters are of incorrect type
+    KeyError
+        If instrument name is not found in the supported instruments list
+    FileNotFoundError
+        If path does not exist or cannot be accessed
+
+    See Also
+    --------
+    AeroViz.rawDataReader.core.AbstractReader
+        A abstract reader class for reading raw data from different instruments
+
+    Examples
+    --------
+    >>> from AeroViz import RawDataReader
+    >>>
+    >>> # Using string inputs
+    >>> df_ae33 = RawDataReader(
+    ...     instrument='AE33',
+    ...     path='/path/to/your/data/folder',
+    ...     reset=True,
+    ...     qc='1MS',
+    ...     start='2024-01-01',
+    ...     end='2024-06-30',
+    ...     mean_freq='1h',
+    ... )
+
+    >>> # Using Path and datetime objects
+    >>> from pathlib import Path
+    >>> from datetime import datetime
+    >>>
+    >>> df_ae33 = RawDataReader(
+    ...     instrument='AE33',
+    ...     path=Path('/path/to/your/data/folder'),
+    ...     reset=True,
+    ...     qc='1MS',
+    ...     start=datetime(2024, 1, 1),
+    ...     end=datetime(2024, 6, 30),
+    ...     mean_freq='1h',
+    ... )
+    """
+
+    # Dynamically build instrument class map from meta configuration
+    # This avoids hardcoding the list and automatically includes new instruments
+    import AeroViz.rawDataReader.script as script_module
+    instrument_class_map = {}
+    for instrument_name in meta.keys():
+        if hasattr(script_module, instrument_name):
+            instrument_class_map[instrument_name] = getattr(script_module, instrument_name)
+
+    # Check if the instrument name is in the map
+    if instrument not in instrument_class_map:
+        raise KeyError(f"Instrument name '{instrument}' is not valid. \nMust be one of: {list(instrument_class_map.keys())}")
+
+    # Check if path exists and is a directory
+    if not isinstance(path, Path):
+        path = Path(path)
+    if not path.exists() or not path.is_dir():
+        raise FileNotFoundError(f"The specified path '{path}' does not exist or is not a directory.")
+
+    # Validate the QC frequency
+    if isinstance(qc, str):
+        try:
+            Grouper(freq=qc)
+        except (ValueError, TypeError):
+            raise ValueError(f"Invalid frequency: {qc}. Must be one of: "
+                             f"W (week), MS (month start), QS (quarter start), YS (year start)")
+
+    # Convert and verify input times
+    if not (start and end):
+        raise ValueError("Both start and end times must be provided.")
+
+    # Convert start time if it's a string
+    if isinstance(start, str):
+        try:
+            start = datetime.fromisoformat(start.replace('Z', '+00:00'))
+        except ValueError as e:
+            raise ValueError(
+                f"Invalid start time format. Please use ISO format (YYYY-MM-DD or YYYY-MM-DD HH:MM:SS): {e}")
+
+    # Convert end time if it's a string
+    if isinstance(end, str):
+        try:
+            end = datetime.fromisoformat(end.replace('Z', '+00:00'))
+        except ValueError as e:
+            raise ValueError(
+                f"Invalid end time format. Please use ISO format (YYYY-MM-DD or YYYY-MM-DD HH:MM:SS): {e}")
+
+    if end <= start:
+        raise ValueError(f"Invalid time range: start {start} is after end {end}")
+
+    end = end.replace(hour=23, minute=59, second=59) if end.hour == 0 and end.minute == 0 else end
+
+    # Verify that mean_freq format
+    try:
+        Timedelta(mean_freq)
+    except ValueError:
+        raise ValueError(
+            f"Invalid mean_freq: '{mean_freq}'. It should be a valid frequency string (e.g., '1h', '30min', '1D').")
+
+    # Validate size range
+    if size_range is not None:
+        SIZE_RANGE_INSTRUMENTS = ['SMPS', 'APS', 'GRIMM']
+        if instrument not in SIZE_RANGE_INSTRUMENTS:
+            raise ValueError(f"Size range filtering is only supported for {SIZE_RANGE_INSTRUMENTS}")
+
+        min_size, max_size = size_range
+        if not isinstance(min_size, (int, float)) or not isinstance(max_size, (int, float)):
+            raise ValueError("Size range values must be numeric")
+        if min_size >= max_size:
+            raise ValueError("Minimum size must be less than maximum size")
+
+        if instrument == 'SMPS':
+            if not (1 <= min_size <= 1000) or not (1 <= max_size <= 1000):
+                raise ValueError("SMPS size range must be between 1 and 1000 nm")
+        elif instrument == 'APS':
+            if not (500 <= min_size <= 20000) or not (500 <= max_size <= 20000):
+                raise ValueError("APS size range must be between 500 and 20000 nm")
+
+        kwargs.update({'size_range': size_range})
+
+    kwargs.update({
+        'suppress_warnings': suppress_warnings,
+        'log_level': log_level
+    })
+
+    # Instantiate the class and return the instance
+    reader_module = instrument_class_map[instrument].Reader(
+        path=path,
+        reset=reset,
+        qc=qc,
+        **kwargs
+    )
+    return reader_module(
+        start=start,
+        end=end,
+        mean_freq=mean_freq,
+    )

AeroViz/rawDataReader/config/supported_instruments.py

@@ -0,0 +1,135 @@
+# Description: Configuration file for rawDataReader
+
+meta = {
+    "NEPH": {
+        "pattern": ["*.dat"],
+        "freq": "5min",
+    },
+
+    "Aurora": {
+        "pattern": ["*.csv"],
+        "freq": "1min",
+    },
+
+    "SMPS": {
+        "pattern": ["*.txt", "*.csv"],
+        "freq": "6min",
+    },
+
+    "GRIMM": {
+        "pattern": ["*.dat"],
+        "freq": "6min",
+    },
+
+    "APS": {
+        "pattern": ["*.txt"],
+        "freq": "6min",
+    },
+
+    "AE33": {
+        "pattern": ["[!ST|!CT|!FV]*[!log]_AE33*.dat"],
+        "freq": "1min",
+    },
+
+    "AE43": {
+        "pattern": ["[!ST|!CT|!FV]*[!log]_AE43*.dat"],
+        "freq": "1min",
+    },
+
+    "BC1054": {
+        "pattern": ["*.csv"],
+        "freq": "1min",
+    },
+
+    "MA350": {
+        "pattern": ["*.csv"],
+        "freq": "1min",
+    },
+
+    "BAM1020": {
+        "pattern": ["*.csv"],
+        "freq": "1h",
+    },
+
+    "TEOM": {
+        "pattern": ["*.csv"],
+        "freq": "6min",
+    },
+
+    "OCEC": {
+        "pattern": ["*LCRes.csv"],
+        "freq": "1h",
+    },
+
+    "IGAC": {
+        "pattern": ["*.csv"],
+        "freq": "1h",
+
+        # https://www.yangyao-env.com/web/product/product_in2.jsp?pd_id=PD1640151884502
+        # HF: 0.08, F-: 0.08, PO43-: None is not measured
+        "MDL": {
+            'HF': None, 'HCl': 0.05, 'HNO2': 0.01, 'HNO3': 0.05, 'G-SO2': 0.05, 'NH3': 0.1,
+            'Na+': 0.05, 'NH4+': 0.08, 'K+': 0.08, 'Mg2+': 0.05, 'Ca2+': 0.05,
+            'F-': None, 'Cl-': 0.05, 'NO2-': 0.05, 'NO3-': 0.01, 'PO43-': None, 'SO42-': 0.05,
+        },
+
+        "MR": {
+            'HF': 200, 'HCl': 200, 'HNO2': 200, 'HNO3': 200, 'G-SO2': 200, 'NH3': 300,
+            'Na+': 300, 'NH4+': 300, 'K+': 300, 'Mg2+': 300, 'Ca2+': 300,
+            'F-': 300, 'Cl-': 300, 'NO2-': 300, 'NO3-': 300, 'PO43-': None, 'SO42-': 300,
+        }
+    },
+
+    "Xact": {
+        "pattern": ["*.csv"],
+        "freq": "1h",
+
+        # base on Xact 625i Minimum Decision Limit (MDL) for XRF in ng/m3, 60 min sample time
+        "MDL": {
+            'Al': 100, 'Si': 18, 'P': 5.2, 'S': 3.2, 'Cl': 1.7,
+            'K': 1.2, 'Ca': 0.3, 'Ti': 1.6, 'V': 0.12, 'Cr': 0.12,
+            'Mn': 0.14, 'Fe': 0.17, 'Co': 0.14, 'Ni': 0.096, 'Cu': 0.079,
+            'Zn': 0.067, 'Ga': 0.059, 'Ge': 0.056, 'As': 0.063, 'Se': 0.081,
+            'Br': 0.1, 'Rb': 0.19, 'Sr': 0.22, 'Y': 0.28, 'Zr': 0.33,
+            'Nb': 0.41, 'Mo': 0.48, 'Pd': 2.2, 'Ag': 1.9, 'Cd': 2.5,
+            'In': 3.1, 'Sn': 4.1, 'Sb': 5.2, 'Te': 0.6, 'Cs': 0.37,
+            'Ba': 0.39, 'La': 0.36, 'Ce': 0.3, 'W': 0.0001, 'Pt': 0.12,
+            'Au': 0.1, 'Hg': 0.12, 'Tl': 0.12, 'Pb': 0.13, 'Bi': 0.13
+        }
+    },
+
+    "Q-ACSM": {
+        "pattern": ["*.csv"],
+        "freq": "30min",
+    },
+
+    "VOC": {
+        "pattern": ["*.csv"],
+        "freq": "1h",
+        "key": [
+            'Benzene', 'Toluene', 'Ethylbenzene', 'm/p-Xylene', 'o-Xylene', 'Ethane', 'Propane', 'Isobutane',
+            'n-Butane', 'Isopentane', 'n-Pentane', 'n-Hexane', 'n-Heptane', 'n-Octane', 'n-Nonane', 'n-Decane',
+            'n-Undecane', 'n-Dodecane', 'Ethylene', 'Propylene', '1-Butene', 't-2-Butene', 'cis-2-Butene',
+            '1-Pentene', 't-2-Pentene', 'cis-2-Pentene', '1-Hexene', 'Acetylene', 'Cyclopentane', 'Methylcyclopentane',
+            'Cyclohexane', 'Methylcyclohexane', 'Isoprene', '2,2-Dimethylbutane', '2,3-Dimethylbutane',
+            '2-Methylpentane', '3-Methylpentane', '2,4-Dimethylpentane', '2-Methylhexane', '2,3-Dimethylpentane',
+            '3-Methylheptane', '2,2,4-Trimethylpentane', '2,3,4-Trimethylpentane', '2-Methylheptane', '3-Methylhexane',
+            'Styrene', 'Isopropylbenzene', 'n-Propylbenzene', 'm-Ethyltoluene', 'p-Ethyltoluene', 'm-Diethylbenzene',
+            'p-Diethylbenzene', '1,3,5-Trimethylbenzene', 'o-Ethyltoluene', '1,2,4-Trimethylbenzene',
+            '1,2,3-Trimethylbenzene',
+            '1.2-DCB', '1.4-DCB', '1.3-Butadiene', '1-Octene', '2-Ethyltoluene', '3.4-Ethyltoluene', 'Acetaldehyde',
+            'Acetone', 'Butyl Acetate', 'Ethanol', 'Ethyl Acetate', 'Hexane', 'IPA', 'Iso-Propylbenzene',
+            'PCE', 'Propene', 'TCE', 'VCM',
+        ],
+    },
+
+    "EPA": {
+        "pattern": ["*.csv"],
+        "freq": "1h",
+    },
+
+    "Minion": {
+        "pattern": ["*.csv", "*.xlsx"],
+        "freq": "1h",
+    },
+}