Water vapor first guess retrieval fails with recently acquired EnMAP data
As reported in EnMAP/GFZ_Tools_EnMAP_BOX/EnPT#92 (closed) and EnMAP/GFZ_Tools_EnMAP_BOX/EnPT#90 (closed), the water vapor first guess retrieval seems to fail sometimes with newly acquired EnMAP data. I therefore disabled it in EnMAP/GFZ_Tools_EnMAP_BOX/EnPT!69 (merged) for now. However, this might lead to a decreased WV retrieval quality in case of scenes with spatially variable water vapor. Therefore, it would be nice to find the actual reason of this.
Here is the full log and traceback without the multiprocessing stuff from EnMAP/GFZ_Tools_EnMAP_BOX/EnPT#92 (closed):
log
test_controller.py::Test_EnPT_Controller_DLR_testdata::test_run_all_processors EnPT Controller received the following configuration:
{'general_opts': {'CPUs': 1,
'average_elevation': 0,
'create_logfile': True,
'disable_progress_bars': False,
'drop_bad_bands': True,
'is_dummy_dataformat': False,
'log_level': 'DEBUG',
'n_lines_to_append': 50,
'path_dem': '',
'path_l1b_enmap_image': '/home/gfz-fe/scheffler/python/EnPT/tests/data/EnMAP_Level_1B/ENMAP01-____L1B-DT0000003706_20220921T111402Z_005_V010110_20221104T234042Z.ZIP',
'path_l1b_enmap_image_gapfill': '',
'path_l1b_snr_model': '/home/gfz-fe/scheffler/python/EnPT/enpt/resources/EnMAP_Sensor/EnMAP_Level_1B_SNR.zip',
'working_dir': None},
'output': {'output_dir': '/home/gfz-fe/scheffler/python/EnPT/enpt/../tests/data/test_outputs',
'output_format': 'ENVI',
'output_interleave': 'pixel'},
'processors': {'atmospheric_correction': {'auto_download_ecmwf': False,
'blocksize': 100,
'enable_ac': True,
'mode_ac': 'land',
'polymer_root': '',
'scale_factor_boa_ref': 10000,
'threads': -1},
'dead_pixel': {'algorithm': 'spectral',
'interp_method_spatial': 'linear',
'interp_method_spectral': 'linear',
'run_processor': True},
'geometry': {'enable_absolute_coreg': False,
'enable_keystone_correction': False,
'enable_vnir_swir_coreg': False,
'path_reference_image': ''},
'orthorectification': {'resamp_alg': 'nearest',
'target_coord_grid': {'x': array([ 0, 30]),
'y': array([ 0, 30])},
'target_epsg': None,
'target_projection_type': 'UTM',
'vswir_overlap_algorithm': 'swir_only'},
'smile': {'run_processor': False},
'toa_ref': {'path_earthSunDist': '/home/gfz-fe/scheffler/python/EnPT/enpt/resources/earth_sun_distance/Earth_Sun_distances_per_day_edited__1980_2030.csv',
'path_solar_irr': '/home/gfz-fe/scheffler/python/EnPT/enpt/resources/solar_irradiance/SUNp1fontenla__350-2500nm_@0.1nm_converted.txt',
'scale_factor_toa_ref': 10000}}}
2022/12/09 11:58:49: Reading metadata for VNIR detector...
2022/12/09 11:58:49: Calculating solar irradiance...
2022/12/09 11:58:49: Reading metadata for SWIR detector...
2022/12/09 11:58:50: Calculating solar irradiance...
2022/12/09 11:58:50: Reading image masks in VNIR sensor geometry.
2022/12/09 11:58:55: Converting DN values to radiance [mW/m^2/sr/nm] for VNIR detector...
2022/12/09 11:58:58: Converting DN values to radiance [mW/m^2/sr/nm] for SWIR detector...
2022/12/09 11:59:01: Computing SNR from VNIR TOA radiance.
2022/12/09 11:59:11: Computing SNR from SWIR TOA radiance.
2022/12/09 11:59:15: Computing VNIR geolayer...
2022/12/09 11:59:18: Computing SWIR geolayer...
2022/12/09 12:00:54: Correcting dead pixels of VNIR detector...
Used algorithm: linear interpolation in the spectral domain
2022/12/09 12:00:57: Percentage of defective pixels: 0.69
2022/12/09 12:00:57: Percentage of pixels with a defect in any band: 30.90
2022/12/09 12:01:01: Correcting dead pixels of SWIR detector...
Used algorithm: linear interpolation in the spectral domain
2022/12/09 12:01:05: Percentage of defective pixels: 4.41
2022/12/09 12:01:05: Percentage of pixels with a defect in any band: 79.19
2022/12/09 12:03:12: No DEM for the VNIR detector provided. Falling back to an average elevation of 0 meters.
2022/12/09 12:03:12: No DEM for the SWIR detector provided. Falling back to an average elevation of 0 meters.
2022/12/09 12:03:16: Transforming the 'mask_landwater' attribute from VNIR to SWIR sensor geometry.
2022/12/09 12:03:22: Starting atmospheric correction for VNIR and SWIR detector in 'land' mode. Source radiometric unit code is 'TOARad'.
2022/12/09 12:03:22: Running atmospheric correction in 'land' mode, i.e., SICOR is applied to ALL surfaces. Uncertainty is expected for water surfaces because SICOR is designed for land only.
2022/12/09 12:03:23: SICOR AC configuration:
{'processing': {'Exception': None,
'Exception_type': '',
'clear_fraction': None,
'interface': {'args': (), 'kwargs': {}},
'status': 1,
'tIO': 0.0,
'tRT': 0.0,
'uncertainties': {}},
'retrieval': {'cpu': 1,
'default_aot_value': 0.31,
'disable_progressbars': False,
'fn_LUT': '',
'inversion': {'eps': 0.01,
'full': False,
'gnform': 'n',
'maxiter': 35},
'land_only': False,
'n_pca': 5,
'segmentation': True,
'segs': 200,
'sol_model': 'fontenla',
'state_vector': {'ice': {'ll': 0.001,
'prior_mean': 0.02,
'prior_sigma': 1000.0,
'ul': 0.499,
'use_prior_mean': False},
'intercept': {'ll': 0.001,
'prior_mean': 0.3,
'prior_sigma': 1000.0,
'ul': 0.999,
'use_prior_mean': False},
'liquid_water': {'ll': 0.001,
'prior_mean': 0.02,
'prior_sigma': 1000.0,
'ul': 0.499,
'use_prior_mean': False},
'slope': {'ll': -0.0004,
'prior_mean': 0.0002,
'prior_sigma': 1000.0,
'ul': 0.0004,
'use_prior_mean': False},
'water_vapor': {'ll': 0.001,
'prior_mean': 2.5,
'prior_sigma': 1000.0,
'ul': 4.999,
'use_prior_mean': False}},
'unknowns': {'ice_absorption_coefficients': {'sigma': 2.39e-07},
'liquid_water_absorption_coefficients': {'sigma': 2.43e-07},
'skyview': {'sigma': 0.1},
'water_vapor_absorption_coefficients': {'sigma': 0.01}}},
'sensor': {'fit': {'idx': [14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26,
27,
28,
29,
30,
31],
'snr': [322.94254006,
313.45752342,
313.32464722,
306.19455173,
281.35641378,
227.9759974,
157.22390595,
148.93620623,
154.40197388,
182.60431866,
232.89644996,
250.1244036,
252.2267179,
272.16592448,
299.71964816,
316.11909184,
326.33202741,
312.28461288]},
'name': 'EnMAP',
'resamp_alg': 'nearest'}}
2022/12/09 12:03:23: Setting up forward operator...
2022/12/09 12:03:26: SICOR is applied to land AND water pixels.
2022/12/09 12:03:26: Transforming VNIR data to SWIR sensor geometry to enable first guess retrievals for liquid water and ice...
2022/12/09 12:03:35: Getting observation metadata...
2022/12/09 12:03:35: Checking if observation metadata values are within LUT value ranges...
2022/12/09 12:03:35: Loading solar irradiance model...
2022/12/09 12:03:35: Loading RT LUT...
2022/12/09 12:03:35: LUT file was properly downloaded and is available for AC!
2022/12/09 12:03:36: Converting LUT to Fontenla solar model...
2022/12/09 12:03:36: Resampling LUT to EnMAP wavelengths...
2022/12/09 12:04:01: Reducing grid dimensionality of LUT to increase interpolation speed...
2022/12/09 12:04:01: Calculating absorption coefficients of liquid water and ice...
2022/12/09 12:04:02: Performing first guess water vapor retrieval based on a common band ratio using VNIR data...
0%| | 4781/1024000 [00:06<22:03, 769.98it/s]
FAILED
tests/test_controller.py:67 (Test_EnPT_Controller_DLR_testdata.test_run_all_processors)
self = <tests.test_controller.Test_EnPT_Controller_DLR_testdata testMethod=test_run_all_processors>
def test_run_all_processors(self):
> self.CTR.run_all_processors()
test_controller.py:69:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../enpt/execution/controller.py:184: in run_all_processors
self.run_atmospheric_correction()
../enpt/execution/controller.py:152: in run_atmospheric_correction
self.L1_obj.run_AC()
../enpt/model/images/images_sensorgeo.py:858: in run_AC
AC.run_ac(self)
../enpt/processors/atmospheric_correction/atmospheric_correction.py:269: in run_ac
self._run_AC__land_mode(enmap_ImageL1)
../enpt/processors/atmospheric_correction/atmospheric_correction.py:130: in _run_AC__land_mode
sicor_ac_enmap(enmap_l1b=enmap_ImageL1,
../../../../mambaforge/envs/enpt_full/lib/python3.9/site-packages/sicor/sicor_enmap.py:87: in sicor_ac_enmap
fo_enmap = Fo(enmap_l1b=enmap_l1b, options=options, logger=logger)
../../../../mambaforge/envs/enpt_full/lib/python3.9/site-packages/sicor/AC/RtFo_3_phases.py:753: in __init__
self.cwv_fg = wv_band_ratio(data=self.data_vnir, water_msk=self.water_mask_vnir, fn_table=self.fn_table,
../../../../mambaforge/envs/enpt_full/lib/python3.9/site-packages/sicor/Tools/EnMAP/first_guess.py:166: in wv_band_ratio
_compute_wv(ind)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ind = 4781
def _compute_wv(ind):
if _globs["water_msk_flat"][ind] != 1:
pass
else:
dem_fac_pix = _globs["dem_fac"][ind]
if _globs["ll_cal"][ind] <= _globs["rhos"][0]:
_globs["ll_cal"][ind] = _globs["rhos"][0] + 0.01 * _globs["rhos"][0]
ll_cal_low = _globs["ll_cal"][ind] >= _globs["rhos"]
ll_cal_high = _globs["ll_cal"][ind] <= _globs["rhos"]
idx_low = np.where(ll_cal_low)
idx_high = np.where(ll_cal_high)
rfl_fac = (_globs["ll_cal"][ind] - _globs["rhos"][idx_low[0][-1]]) / (
> _globs["rhos"][idx_high[0][0]] - _globs["rhos"][idx_low[0][-1]])
E IndexError: index 0 is out of bounds for axis 0 with size 0
../../../../mambaforge/envs/enpt_full/lib/python3.9/site-packages/sicor/Tools/EnMAP/first_guess.py:205: IndexError@nbohn I will send you the dataset to reproduce it.
Edited by Daniel Scheffler