fm35-decoder 1.1__tar.gz → 1.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (24) hide show
  1. {fm35_decoder-1.1 → fm35_decoder-1.2}/.gitignore +4 -1
  2. {fm35_decoder-1.1 → fm35_decoder-1.2}/PKG-INFO +5 -3
  3. {fm35_decoder-1.1 → fm35_decoder-1.2}/README.md +4 -2
  4. {fm35_decoder-1.1 → fm35_decoder-1.2}/pyproject.toml +1 -1
  5. {fm35_decoder-1.1 → fm35_decoder-1.2}/src/fm35_decoder/decoder.py +105 -42
  6. fm35_decoder-1.2/src/fm35_decoder/table_codes/CL_0513.csv +12 -0
  7. fm35_decoder-1.2/src/fm35_decoder/table_codes/Nh_2700.csv +12 -0
  8. fm35_decoder-1.2/src/fm35_decoder/table_codes/Sr_3849.csv +10 -0
  9. fm35_decoder-1.2/src/fm35_decoder/table_codes/h_1600.csv +12 -0
  10. {fm35_decoder-1.1 → fm35_decoder-1.2}/src/fm35_decoder/table_codes/rara_3685.csv +47 -103
  11. fm35_decoder-1.2/src/fm35_decoder/table_codes/sasa_3872.csv +102 -0
  12. fm35_decoder-1.2/src/fm35_decoder/tables.py +383 -0
  13. fm35_decoder-1.1/src/fm35_decoder/table_codes/CL_0513.csv +0 -12
  14. fm35_decoder-1.1/src/fm35_decoder/table_codes/Nh_2700.csv +0 -12
  15. fm35_decoder-1.1/src/fm35_decoder/table_codes/Sr_3849.csv +0 -10
  16. fm35_decoder-1.1/src/fm35_decoder/table_codes/h_1600.csv +0 -12
  17. fm35_decoder-1.1/src/fm35_decoder/table_codes/sasa_3872.csv +0 -27
  18. fm35_decoder-1.1/src/fm35_decoder/tables.py +0 -291
  19. {fm35_decoder-1.1 → fm35_decoder-1.2}/LICENSE +0 -0
  20. {fm35_decoder-1.1 → fm35_decoder-1.2}/src/fm35_decoder/__init__.py +0 -0
  21. {fm35_decoder-1.1 → fm35_decoder-1.2}/src/fm35_decoder/table_codes/CH_0509.csv +0 -0
  22. {fm35_decoder-1.1 → fm35_decoder-1.2}/src/fm35_decoder/table_codes/CM_0515.csv +0 -0
  23. {fm35_decoder-1.1 → fm35_decoder-1.2}/src/fm35_decoder/table_codes/D_0777.csv +0 -0
  24. {fm35_decoder-1.1 → fm35_decoder-1.2}/src/fm35_decoder/table_codes/T_3931.csv +0 -0
@@ -29,4 +29,7 @@ htmlcov/
29
29
 
30
30
  # OS files
31
31
  .DS_Store
32
- Thumbs.db
32
+ Thumbs.db
33
+
34
+ lib/
35
+ .gitignore
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: fm35_decoder
3
- Version: 1.1
3
+ Version: 1.2
4
4
  Summary: A Python tool to decode standard FM35 TEMP weather messages from upper-air observations.
5
5
  Project-URL: Homepage, https://github.com/luciodaou/fm35_decoder
6
6
  Project-URL: Bug Tracker, https://github.com/luciodaou/fm35_decoder/issues
@@ -115,6 +115,8 @@ Missing geopotential heights are computed using the Hypsometric Equation:
115
115
  $$Z_2 - Z_1 = \frac{R_d \cdot \bar{T}_v}{g_0} \ln\left(\frac{P_1}{P_2}\right)$$
116
116
  The decoder incorporates **virtual temperature ($T_v$)** when moisture data is available, accounting for air density variations in warm and humid tropical boundary layers.
117
117
 
118
+ A level without a temperature takes one for this calculation only: interpolated in $\ln P$ between observed temperatures, and the top temperature above the last one. Above the last temperature a height is computed only within the limits of Regulation 35.2.2.4 (25 hPa, and a quarter of the level's pressure); further up a level keeps no height unless the report gives it. Under the same rule, a standard level just above the top of the sounding that the report does not hold is added with its geopotential only (`Source` = `Extrapolated`).
119
+
118
120
  ---
119
121
 
120
122
  ## Performance & Architecture
@@ -149,9 +151,9 @@ The decoder incorporates **virtual temperature ($T_v$)** when moisture data is a
149
151
  CL Cloud Low Type Stratus nebulosus and/or Stratus fractus TTBB
150
152
  CM Cloud Mid Type No CM clouds TTBB
151
153
  CH Cloud High Type No CH clouds TTBB
152
- sr Solar/Inst Solar Corr NOAA solar corrected TTAA
154
+ sr Solar/Inst Solar Corr Solar and infrared corrected automatically by radiosonde system TTAA
153
155
  rara Solar/Inst Sonde Type Vaisala RS41/DigiCORA MW41 (Finland) TTAA
154
- sasa Solar/Inst Tracking Radar (5 cm) TTAA
156
+ sasa Solar/Inst Tracking Automatic satellite navigation TTAA
155
157
  8GGgg Solar/Inst Time 11:31 TTAA
156
158
  PtPtPt Tropopause Pressure 90.6hPa TTCC
157
159
  TtTtTt Tropopause Temperature -77.1C TTCC
@@ -97,6 +97,8 @@ Missing geopotential heights are computed using the Hypsometric Equation:
97
97
  $$Z_2 - Z_1 = \frac{R_d \cdot \bar{T}_v}{g_0} \ln\left(\frac{P_1}{P_2}\right)$$
98
98
  The decoder incorporates **virtual temperature ($T_v$)** when moisture data is available, accounting for air density variations in warm and humid tropical boundary layers.
99
99
 
100
+ A level without a temperature takes one for this calculation only: interpolated in $\ln P$ between observed temperatures, and the top temperature above the last one. Above the last temperature a height is computed only within the limits of Regulation 35.2.2.4 (25 hPa, and a quarter of the level's pressure); further up a level keeps no height unless the report gives it. Under the same rule, a standard level just above the top of the sounding that the report does not hold is added with its geopotential only (`Source` = `Extrapolated`).
101
+
100
102
  ---
101
103
 
102
104
  ## Performance & Architecture
@@ -131,9 +133,9 @@ The decoder incorporates **virtual temperature ($T_v$)** when moisture data is a
131
133
  CL Cloud Low Type Stratus nebulosus and/or Stratus fractus TTBB
132
134
  CM Cloud Mid Type No CM clouds TTBB
133
135
  CH Cloud High Type No CH clouds TTBB
134
- sr Solar/Inst Solar Corr NOAA solar corrected TTAA
136
+ sr Solar/Inst Solar Corr Solar and infrared corrected automatically by radiosonde system TTAA
135
137
  rara Solar/Inst Sonde Type Vaisala RS41/DigiCORA MW41 (Finland) TTAA
136
- sasa Solar/Inst Tracking Radar (5 cm) TTAA
138
+ sasa Solar/Inst Tracking Automatic satellite navigation TTAA
137
139
  8GGgg Solar/Inst Time 11:31 TTAA
138
140
  PtPtPt Tropopause Pressure 90.6hPa TTCC
139
141
  TtTtTt Tropopause Temperature -77.1C TTCC
@@ -1,6 +1,6 @@
1
1
  [project]
2
2
  name = "fm35_decoder"
3
- version = "1.1"
3
+ version = "1.2"
4
4
  description = "A Python tool to decode standard FM35 TEMP weather messages from upper-air observations."
5
5
  readme = "README.md"
6
6
  requires-python = ">=3.12"
@@ -1,6 +1,7 @@
1
1
  """
2
2
  FM 35-XII TEMP Decoder.
3
- Decodes standard WMO FM 35 / FM 36 / FM 38 upper-air radiosonde observations.
3
+ Decodes WMO FM 35 TEMP upper-air radiosonde observations from fixed land stations
4
+ (the ship and mobile headers of FM 36 / FM 38 are not read).
4
5
  """
5
6
 
6
7
  from __future__ import annotations
@@ -79,7 +80,7 @@ def decode_temperature(ttt_str: str, tables: Optional[Dict[str, Any]] = None) ->
79
80
  tt = int(ttt_str[:2])
80
81
  ta_code = ttt_str[2]
81
82
 
82
- t_table = (tables or WMO_TABLES).get("T_3931", TABLE_T_3931 if "TABLE_T_3931" in globals() else {})
83
+ t_table = (tables or WMO_TABLES).get("T_3931", {})
83
84
  entry = t_table.get(ta_code)
84
85
 
85
86
  if entry:
@@ -135,7 +136,8 @@ def decode_wind(dff_str: str) -> Tuple[Optional[float], Optional[float]]:
135
136
 
136
137
  Returns (direction_degrees, speed_knots).
137
138
  For variable wind (dd=99), direction is returned as np.nan while speed is preserved.
138
- For calm wind (dd=00, fff=000), returns (0.0, 0.0).
139
+ For calm wind (dd=00, fff=000), returns (0.0, 0.0). dd=00 with a speed (north is 36) is not a
140
+ direction: the speed is kept and the direction is np.nan.
139
141
  """
140
142
  if not dff_str or len(dff_str) != 5 or "/////" in dff_str or "/" in dff_str:
141
143
  return None, None
@@ -161,6 +163,8 @@ def decode_wind(dff_str: str) -> Tuple[Optional[float], Optional[float]]:
161
163
  # Calm wind
162
164
  if dd == 0 and speed == 0:
163
165
  return 0.0, 0.0
166
+ if (dd == 0 and direction_unit == 0) or (dd == 36 and direction_unit == 5):
167
+ return np.nan, float(speed)
164
168
 
165
169
  direction = dd * 10 + direction_unit
166
170
  return float(direction), float(speed)
@@ -219,9 +223,14 @@ def decode_height(pressure: float, h_str: str) -> Optional[int]:
219
223
  # --- Atmospheric Thermodynamics & Physics ---
220
224
 
221
225
 
222
- def interpolate_data(df: pd.DataFrame) -> pd.DataFrame:
226
+ def interpolate_data(
227
+ df: pd.DataFrame, missing_layers: Optional[List[Tuple[str, float, float]]] = None
228
+ ) -> pd.DataFrame:
223
229
  """
224
230
  Interpolates missing Temperature, DewPoint, and Wind data vertically.
231
+ - missing_layers: (kind, bottom, top) layers reported as missing (nn/// /////), kind being
232
+ "Temperature" or "Wind" and the pressures those of the boundary levels; nothing is filled
233
+ strictly inside them.
225
234
  - Temperature/DewPoint: Linear interpolation in Log-Pressure space.
226
235
  - Physical bound: Enforces DewPoint <= Temp (no super-saturation).
227
236
  - Wind: Vector interpolation (U/V components) in Log-Pressure space.
@@ -244,6 +253,13 @@ def interpolate_data(df: pd.DataFrame) -> pd.DataFrame:
244
253
  df["log_p"] = np.log(df["Pressure"].astype(float))
245
254
  df = df.set_index("log_p")
246
255
 
256
+ def in_missing_layer(kind: str) -> pd.Series:
257
+ inside = pd.Series(False, index=df.index)
258
+ for layer_kind, bottom, top in missing_layers or ():
259
+ if layer_kind == kind:
260
+ inside |= (df["Pressure"] < bottom) & (df["Pressure"] > top)
261
+ return inside
262
+
247
263
  # --- Temperature & Dewpoint Interpolation ---
248
264
  for col in ["Temp", "DewPoint"]:
249
265
  if col in df.columns:
@@ -252,6 +268,7 @@ def interpolate_data(df: pd.DataFrame) -> pd.DataFrame:
252
268
  # only between observed levels: nothing is made up above the last or below the first
253
269
  df[col] = df[col].interpolate(method="index", limit_area="inside")
254
270
  df.loc[mask, col] = df.loc[mask, col].round(1)
271
+ df.loc[mask & in_missing_layer("Temperature"), col] = np.nan
255
272
 
256
273
  # Physical constraint: Dewpoint cannot exceed dry temperature
257
274
  if "Temp" in df.columns and "DewPoint" in df.columns:
@@ -291,8 +308,9 @@ def interpolate_data(df: pd.DataFrame) -> pd.DataFrame:
291
308
 
292
309
  # Fill missing values in original columns; a variable wind (speed without direction) keeps
293
310
  # its NaN direction
294
- missing_speed = df["WindSpeed"].isna() & df["u"].notna()
295
- missing_dir = df["WindDir"].isna() & df["WindSpeed"].isna() & df["u"].notna()
311
+ fillable = df["u"].notna() & ~in_missing_layer("Wind")
312
+ missing_speed = df["WindSpeed"].isna() & fillable
313
+ missing_dir = df["WindDir"].isna() & df["WindSpeed"].isna() & fillable
296
314
 
297
315
  df.loc[missing_speed, "WindSpeed"] = reconstructed_speed[missing_speed].round(1)
298
316
  df.loc[missing_dir, "WindDir"] = pd.Series(reconstructed_dir, index=df.index)[missing_dir].round(0)
@@ -308,7 +326,13 @@ def calculate_geopotential(df: pd.DataFrame) -> pd.DataFrame:
308
326
  Calculates missing Geopotential Height values using the Hypsometric Equation.
309
327
  Formula: Z2 = Z1 + (R * Tv_avg / g) * ln(P1 / P2)
310
328
  Uses virtual temperature Tv where humidity is available to ensure physical accuracy.
311
- Includes WMO Extrapolation Rule 35.2.2.4 upward to the next standard level.
329
+ Includes WMO Extrapolation Rule 35.2.2.4 upward to the next standard level: a level the
330
+ report does not hold gets a row with its geopotential only (the extrapolated temperature
331
+ serves that calculation and is not given), and a level the report holds is left as reported.
332
+ A level without a temperature takes one for this calculation only: interpolated in ln(P)
333
+ between observed temperatures, and the top temperature above the last one, there within the
334
+ limits of the same rule (25 hPa and a quarter of the level's pressure); a level further up
335
+ keeps no height unless the report gives it.
312
336
  """
313
337
  if df.empty or "Pressure" not in df.columns or "Temp" not in df.columns:
314
338
  return df
@@ -324,12 +348,17 @@ def calculate_geopotential(df: pd.DataFrame) -> pd.DataFrame:
324
348
  dewpoints = pd.to_numeric(df.get("DewPoint", pd.Series(dtype=float)), errors="coerce").to_numpy(dtype=float)
325
349
  heights = pd.to_numeric(df.get("Height", pd.Series(dtype=float)), errors="coerce").to_numpy(dtype=float)
326
350
 
327
- # Linear interpolation for temperature to ensure continuous profile for hypsometric calculation
328
- valid_t = ~np.isnan(temps)
351
+ # Temperature in ln(P) between observed levels, for a continuous profile in the hypsometric
352
+ # calculation (the top and bottom temperatures are held beyond the observed ones)
353
+ valid_t = ~np.isnan(temps) & (pressures > 0)
329
354
  if not valid_t.any():
330
355
  return df
356
+ # Lowest pressure with an observed temperature: heights above it are bounded (35.2.2.4)
357
+ p_top_t = float(pressures[valid_t].min())
331
358
  if not valid_t.all():
332
- temps = np.interp(np.arange(len(temps)), np.where(valid_t)[0], temps[valid_t])
359
+ with np.errstate(divide="ignore", invalid="ignore"):
360
+ height_coord = -np.log(pressures)
361
+ temps = np.interp(height_coord, height_coord[valid_t], temps[valid_t])
333
362
 
334
363
  # Virtual temperature: Tv = T_K * (1 + 0.608 * w)
335
364
  temp_k = temps + 273.15
@@ -348,6 +377,9 @@ def calculate_geopotential(df: pd.DataFrame) -> pd.DataFrame:
348
377
  for i in range(1, n):
349
378
  if np.isnan(heights[i]) and not np.isnan(heights[i - 1]):
350
379
  p1, p2 = pressures[i - 1], pressures[i]
380
+ beyond = p_top_t - p2
381
+ if beyond > 25.0 or beyond > 0.25 * p2:
382
+ continue
351
383
  if p1 > 0 and p2 > 0 and p1 > p2:
352
384
  avg_tv = (tv_k[i - 1] + tv_k[i]) / 2.0
353
385
  dz = (R * avg_tv / g) * np.log(p1 / p2)
@@ -379,7 +411,8 @@ def calculate_geopotential(df: pd.DataFrame) -> pd.DataFrame:
379
411
  1000, 925, 850, 700, 500, 400, 300, 250, 200, 150,
380
412
  100, 70, 50, 30, 20, 10, 7, 5, 3, 2, 1,
381
413
  ]
382
- targets = [sl for sl in standard_levels if sl < p_min]
414
+ reported = set(pd.to_numeric(df["Pressure"], errors="coerce").round(1))
415
+ targets = [sl for sl in standard_levels if sl < p_min and float(sl) not in reported]
383
416
 
384
417
  new_rows = []
385
418
  for p_target in targets:
@@ -387,8 +420,8 @@ def calculate_geopotential(df: pd.DataFrame) -> pd.DataFrame:
387
420
  if delta_p <= 25 and delta_p <= 0.25 * p_target:
388
421
  try:
389
422
  p_base = p_min + delta_p
390
- all_p = df["Pressure"].to_numpy(dtype=float)
391
- all_t = (df["Temp"].to_numpy(dtype=float) + 273.15)
423
+ all_p = valid_rows["Pressure"].to_numpy(dtype=float)
424
+ all_t = valid_rows["Temp"].to_numpy(dtype=float) + 273.15
392
425
  log_p = np.log(all_p)
393
426
  log_p_base = np.log(p_base)
394
427
 
@@ -410,8 +443,6 @@ def calculate_geopotential(df: pd.DataFrame) -> pd.DataFrame:
410
443
  {
411
444
  "Pressure": int(p_target),
412
445
  "Height": round(z_target),
413
- "Temp": round(t_target - 273.15, 1),
414
- "DewPoint": None,
415
446
  "Source": "Extrapolated",
416
447
  }
417
448
  )
@@ -502,11 +533,12 @@ PART_NAMES = ("TTAA", "TTBB", "TTCC", "TTDD")
502
533
 
503
534
  KT_PER_MS = 3600.0 / 1852.0
504
535
 
505
- # Groups that cannot be a TTTDD group: "not observed" markers, and section markers whose digits
506
- # give an impossible temperature or an unused depression code. The other markers are valid TTTDD
507
- # groups (21212 = 21.2 C, 31313 = -31.3 C, 51515 = -51.5 C, ...), read as data where the position
508
- # in the message says so; temperatures of -66 and -77 C likewise start like the 66/77 indicators.
509
- _NOT_TEMP = frozenset(["88999", "77999", "66999", "41414", "52525", "54545", "55555", "56565", "58585"])
536
+ # Groups that cannot be a TTTDD group: "not observed" markers, 41414 (41.4 C with a dew point of
537
+ # 40 C) and 55555 (depression code 55 is not used). The other markers are valid TTTDD groups
538
+ # (21212 = 21.2 C, 31313 = -31.3 C, 52525 = -52.5 C, 56565 = -56.5 C with a depression of 15 C,
539
+ # ...), read as data where the position in the message says so; temperatures of -66 and -77 C
540
+ # likewise start like the 66/77 indicators.
541
+ _NOT_TEMP = frozenset(["88999", "77999", "66999", "41414", "55555"])
510
542
 
511
543
  # Regional (51515-59595) and national (61616-69696) sections come last in a part: no level data
512
544
  # follows them. 55555 and 66666 are not in the list: in parts B and D 55555 is a significant level
@@ -556,16 +588,17 @@ def wind_top_pressure(id_char: str, part_c: bool) -> Optional[float]:
556
588
  Id (code table 1734): pressure of the last standard level that carries a wind group, in
557
589
  hundreds of hPa in Part A (0 = 1000, 9 = 925, 8 = 850) and tens of hPa in Part C. Levels
558
590
  above it have no wind group (Id 2 covers 250 and 200 hPa, Id 1 covers 150 and 100 hPa).
559
- "/" (no wind at any standard level) gives infinity; anything else gives None (unknown).
591
+ "/" (no wind at any standard level) gives infinity; a figure the table does not define for
592
+ the part (6 in Part A; 0, 4, 6, 8, 9 in Part C) or anything else gives None (unknown).
560
593
  """
561
594
  if id_char == "/":
562
595
  return math.inf
563
- if len(id_char) != 1 or id_char not in "0123456789":
564
- return None
565
- d = int(id_char)
566
596
  if part_c:
567
- return 100.0 if d == 0 else d * 10.0
568
- return {0: 1000.0, 9: 925.0, 8: 850.0}.get(d, d * 100.0)
597
+ return {"7": 70.0, "5": 50.0, "3": 30.0, "2": 20.0, "1": 10.0}.get(id_char)
598
+ return {
599
+ "0": 1000.0, "9": 925.0, "8": 850.0, "7": 700.0, "5": 500.0,
600
+ "4": 400.0, "3": 300.0, "2": 200.0, "1": 100.0,
601
+ }.get(id_char)
569
602
 
570
603
 
571
604
  def split_bulletin(text: Optional[str]) -> Dict[str, str]:
@@ -695,9 +728,9 @@ def parse_ttaa_ttcc(
695
728
  continue
696
729
  pressure = special_pressure(g[2:])
697
730
  i += 1
698
- if pressure is None:
699
- continue
700
- special_data.append(
731
+ # Without a readable pressure the level is not kept, but its groups are still its own
732
+ found: List[Dict[str, Any]] = []
733
+ found.append(
701
734
  {"Symbol": "PtPtPt", "Subject": "Tropopause", "Description": "Pressure", "Value": f"{pressure}hPa"}
702
735
  )
703
736
  t_grp = group(i)
@@ -705,12 +738,12 @@ def parse_ttaa_ttcc(
705
738
  t = decode_temperature(t_grp[:3], tables=cloud_tables)
706
739
  d = decode_dewpoint_depression(t_grp[3:], tables=cloud_tables)
707
740
  if t is not None:
708
- special_data.append(
741
+ found.append(
709
742
  {"Symbol": "TtTtTt", "Subject": "Tropopause", "Description": "Temperature", "Value": f"{t}C"}
710
743
  )
711
744
  if t is not None and d is not None:
712
745
  dw = calculate_dewpoint(t, d)
713
- special_data.append(
746
+ found.append(
714
747
  {"Symbol": "DtDt", "Subject": "Tropopause", "Description": "Dewpoint", "Value": f"{dw}C"}
715
748
  )
716
749
  i += 1
@@ -718,7 +751,7 @@ def parse_ttaa_ttcc(
718
751
  if _is_wind_group(w_grp):
719
752
  wd, ws = decode_wind(w_grp)
720
753
  if ws is not None:
721
- special_data.append(
754
+ found.append(
722
755
  {
723
756
  "Symbol": "dtdtftft",
724
757
  "Subject": "Tropopause",
@@ -727,6 +760,8 @@ def parse_ttaa_ttcc(
727
760
  }
728
761
  )
729
762
  i += 1
763
+ if pressure is not None:
764
+ special_data.extend(found)
730
765
  continue
731
766
 
732
767
  # Maximum Wind Level (77PmPmPm or 66PmPmPm, dmdmfmfmfm, optional 4vbvbvava)
@@ -736,16 +771,14 @@ def parse_ttaa_ttcc(
736
771
  continue
737
772
  pressure = special_pressure(g[2:])
738
773
  i += 1
739
- if pressure is None:
740
- continue
741
- special_data.append(
774
+ found = [
742
775
  {"Symbol": "PmPmPm", "Subject": "Max Wind", "Description": "Pressure", "Value": f"{pressure}hPa"}
743
- )
776
+ ]
744
777
  wind_grp = group(i)
745
778
  if _is_wind_group(wind_grp):
746
779
  wd, ws = decode_wind(wind_grp)
747
780
  if ws is not None:
748
- special_data.append(
781
+ found.append(
749
782
  {
750
783
  "Symbol": "dmdmfmfmfm",
751
784
  "Subject": "Max Wind",
@@ -758,13 +791,15 @@ def parse_ttaa_ttcc(
758
791
  shear_grp = group(i)
759
792
  if shear_grp is not None and len(shear_grp) == 5 and shear_grp.startswith("4"):
760
793
  vb, va = shear_grp[1:3], shear_grp[3:5]
761
- special_data.append(
794
+ found.append(
762
795
  {"Symbol": "vbvb", "Subject": "Max Wind", "Description": "Shear Below", "Value": f"{vb}{unit}"}
763
796
  )
764
- special_data.append(
797
+ found.append(
765
798
  {"Symbol": "vava", "Subject": "Max Wind", "Description": "Shear Above", "Value": f"{va}{unit}"}
766
799
  )
767
800
  i += 1
801
+ if pressure is not None:
802
+ special_data.extend(found)
768
803
  continue
769
804
 
770
805
  # Regional / Instrumentation group 31313
@@ -856,6 +891,8 @@ def parse_ttbb_ttdd(
856
891
  mode = "TEMP"
857
892
  is_ttdd = "TTDD" in message
858
893
  last_p: Optional[float] = None
894
+ # A layer of missing data (nn/// /////) is open until the level that closes it
895
+ layer_open = False
859
896
  unit = "kt"
860
897
 
861
898
  i = 0
@@ -867,6 +904,7 @@ def parse_ttbb_ttdd(
867
904
  is_ttdd = (g == "TTDD")
868
905
  mode = "TEMP"
869
906
  last_p = None
907
+ layer_open = False
870
908
  i += 1
871
909
  if i < len(groups) and re.match(r"^[0-9]{4}[0-9/]$", groups[i]):
872
910
  unit = speed_unit(groups[i])
@@ -879,6 +917,7 @@ def parse_ttbb_ttdd(
879
917
  if g == "21212":
880
918
  mode = "WIND"
881
919
  last_p = None
920
+ layer_open = False
882
921
  i += 1
883
922
  continue
884
923
 
@@ -916,6 +955,12 @@ def parse_ttbb_ttdd(
916
955
  i += 1
917
956
  continue
918
957
 
958
+ # Layer of missing data (35.3.1.6, 35.3.2.2): nn/// /////, between its boundary levels
959
+ if g[2:] == "///":
960
+ layer_open = True
961
+ i += 2 if i + 1 < len(groups) and groups[i + 1] == "/////" else 1
962
+ continue
963
+
919
964
  try:
920
965
  ppp_part = int(g[2:])
921
966
  if nn == "00":
@@ -930,11 +975,24 @@ def parse_ttbb_ttdd(
930
975
  if is_ttdd:
931
976
  pressure = pressure / 10.0
932
977
 
978
+ if layer_open and last_p is not None and last_p > pressure:
979
+ special_data.append(
980
+ {
981
+ "Symbol": "nn///",
982
+ "Subject": "Missing Data",
983
+ "Description": "Temperature" if mode == "TEMP" else "Wind",
984
+ "Value": f"{last_p:g}-{pressure:g}hPa",
985
+ "Bottom": last_p,
986
+ "Top": pressure,
987
+ }
988
+ )
989
+ layer_open = False
933
990
  last_p = pressure
934
991
 
935
992
  if mode == "TEMP":
993
+ # The group after nnPPP is the temperature group, 21212 (21.2 C) included
936
994
  t_group = groups[i + 1] if i + 1 < len(groups) else None
937
- if t_group and len(t_group) == 5 and t_group != "21212":
995
+ if t_group and len(t_group) == 5:
938
996
  t = decode_temperature(t_group[:3], tables=cloud_tables)
939
997
  d = decode_dewpoint_depression(t_group[3:], tables=cloud_tables)
940
998
  levels_data.append(
@@ -1015,6 +1073,7 @@ def decode_full(
1015
1073
 
1016
1074
  data_frames: List[pd.DataFrame] = []
1017
1075
  special_frames: List[pd.DataFrame] = []
1076
+ missing_layers: List[Tuple[str, float, float]] = []
1018
1077
 
1019
1078
  for name, parser in [
1020
1079
  ("TTAA", parse_ttaa_ttcc),
@@ -1027,13 +1086,16 @@ def decode_full(
1027
1086
  lvls, spcls = parser(msg, cloud_tables=cloud_tables)
1028
1087
  if lvls:
1029
1088
  data_frames.append(pd.DataFrame(lvls))
1089
+ missing_layers += [
1090
+ (s["Description"], s["Bottom"], s["Top"]) for s in spcls if s["Subject"] == "Missing Data"
1091
+ ]
1030
1092
  if spcls:
1031
1093
  special_frames.append(pd.DataFrame(spcls).assign(Part=name))
1032
1094
 
1033
1095
  df_main = merge_data(data_frames)
1034
1096
  if not df_main.empty and df_main["Temp"].notna().sum() < 2:
1035
1097
  raise ValueError("TEMP message without temperatures above the surface")
1036
- df_main = interpolate_data(df_main)
1098
+ df_main = interpolate_data(df_main, missing_layers)
1037
1099
  df_main = calculate_geopotential(df_main)
1038
1100
 
1039
1101
  df_special = pd.concat(special_frames, ignore_index=True) if special_frames else pd.DataFrame()
@@ -1078,7 +1140,8 @@ def decode(
1078
1140
  Returns:
1079
1141
  tuple[pd.DataFrame, pd.DataFrame]:
1080
1142
  - df_main: Vertical profile with Pressure, Height, Temp, DewPoint, WindDir, WindSpeed.
1081
- - df_special: Tropopause, Maximum Wind, Cloud layers, and Radiosonde/Solar metadata,
1143
+ - df_special: Tropopause, Maximum Wind, Cloud layers, layers of missing data, and
1144
+ Radiosonde/Solar metadata,
1082
1145
  with the part (TTAA, TTBB, TTCC, TTDD) each entry came from in the Part column.
1083
1146
  """
1084
1147
  # Defensive type validation
@@ -0,0 +1,12 @@
1
+ "Code","Description"
2
+ "0","No CL clouds"
3
+ "1","Cumulus humilis and/or Cumulus fractus other than of bad weather"
4
+ "2","Cumulus mediocris or congestus, with or without Cumulus of species in code 1"
5
+ "3","Cumulonimbus calvus, with or without Cumulus, Stratocumulus or Stratus"
6
+ "4","Stratocumulus cumulogenitus"
7
+ "5","Stratocumulus other than cumulogenitus"
8
+ "6","Stratus nebulosus and/or Stratus fractus other than of bad weather"
9
+ "7","Stratus fractus and/or Cumulus fractus of bad weather"
10
+ "8","Cumulus and Stratocumulus other than cumulogenitus, with bases at different levels"
11
+ "9","Cumulonimbus capillatus, with or without Cumulonimbus calvus, Cumulus, Stratocumulus, Stratus"
12
+ "/","CL clouds invisible owing to darkness, fog, blowing dust or sand, or other similar phenomena"
@@ -0,0 +1,12 @@
1
+ "Code","Description"
2
+ "0","No clouds"
3
+ "1","1 okta or less, but not zero"
4
+ "2","2 oktas"
5
+ "3","3 oktas"
6
+ "4","4 oktas"
7
+ "5","5 oktas"
8
+ "6","6 oktas"
9
+ "7","7 oktas or more, but not 8 oktas"
10
+ "8","8 oktas"
11
+ "9","Sky obscured by fog and/or other meteorological phenomena"
12
+ "/","Cloud cover indiscernible for reasons other than fog or other meteorological phenomena, or observation not made"
@@ -0,0 +1,10 @@
1
+ "Code","Description"
2
+ "0","No correction"
3
+ "1","CIMO solar corrected and CIMO infrared corrected"
4
+ "2","CIMO solar corrected and infrared corrected"
5
+ "3","CIMO solar corrected only"
6
+ "4","Solar and infrared corrected automatically by radiosonde system"
7
+ "5","Solar corrected automatically by radiosonde system"
8
+ "6","Solar and infrared corrected as specified by country"
9
+ "7","Solar corrected as specified by country"
10
+ "/","Missing value"
@@ -0,0 +1,12 @@
1
+ "Code","Description"
2
+ "0","< 50 m (< 150 ft)"
3
+ "1","50-100 m (150-300 ft)"
4
+ "2","100-200 m (300-600 ft)"
5
+ "3","200-300 m (600-1000 ft)"
6
+ "4","300-600 m (1000-2000 ft)"
7
+ "5","600-1000 m (2000-3300 ft)"
8
+ "6","1000-1500 m (3300-5000 ft)"
9
+ "7","1500-2000 m (5000-6500 ft)"
10
+ "8","2000-2500 m (6500-8000 ft)"
11
+ "9","> 2500 m (> 8000 ft), or no clouds"
12
+ "/","Height of base of cloud not known, or base of clouds below and tops above the station"