SICOR usable for multispectral data without having water vapour aux data?

My current implementation of SICOR in gms_preprocessing fails in case there are no ECMWF CWV data available. Since ECMWF data are not available for (i think) 2010 and earlier, it is not possible to apply the AC to some older multispectral data.

For example, here is log from gms_preprocessing for a Landsat-5 dataset from 2008:

2020/11/20 18:25:22 [9273108]: Atmospheric correction started. 2020/11/20 18:25:22 [9273108]: Check if AC tables are available. 2020/11/20 18:25:22 [9273108]: No valid path to AC table for scattering type aerosol_1 is provided by the user. 2020/11/20 18:25:22 [9273108]: Instead, AC table for scattering type aerosol_1 is available at default tables path, proceed. 2020/11/20 18:25:22 [9273108]: No valid path to AC table for scattering type aerosol_0 is provided by the user. 2020/11/20 18:25:22 [9273108]: Instead, AC table for scattering type aerosol_0 is available at default tables path, proceed. 2020/11/20 18:25:22 [9273108]: No valid path to AC table for scattering type cirrus is provided by the user. 2020/11/20 18:25:22 [9273108]: Instead, AC table for scattering type cirrus is available at default tables path, proceed. 2020/11/20 18:25:22 [9273108]: No valid path to AC table for scattering type aerosol_2 is provided by the user. 2020/11/20 18:25:22 [9273108]: Instead, AC table for scattering type aerosol_2 is available at default tables path, proceed. 2020/11/20 18:25:22 [9273108]: No valid path to AC table for scattering type aerosol_3 is provided by the user. 2020/11/20 18:25:22 [9273108]: Instead, AC table for scattering type aerosol_3 is available at default tables path, proceed. 2020/11/20 18:25:22 [9273108]: Check if cloud mask persistence file is available. 2020/11/20 18:25:22 [9273108]: No valid path to cloud mask persistence file is provided by the user. 2020/11/20 18:25:22 [9273108]: Instead, cloud mask persistence file is available at default tables path, proceed. 2020/11/20 18:25:22 [9273108]: Check if novelty detector is available. 2020/11/20 18:25:22 [9273108]: No valid path to novelty detector is provided by the user. 2020/11/20 18:25:22 [9273108]: Instead, novelty detector is available at default tables path, proceed. 2020/11/20 18:25:23 [9273108]: Clear fraction: 0.8482 2020/11/20 18:25:23 [9273108]: Derive surface pressure from NWP model. 2020/11/20 18:25:23 [9273108]: Look 0 day back in time to find suitable ECMWF data. 2020/11/20 18:25:23 [9273108]: Look 1 day back in time to find suitable ECMWF data. 2020/11/20 18:25:23 [9273108]: Look 2 days back in time to find suitable ECMWF data. 2020/11/20 18:25:23 [9273108]: Look 3 days back in time to find suitable ECMWF data. 2020/11/20 18:25:23 [9273108]: Look 4 days back in time to find suitable ECMWF data. 2020/11/20 18:25:23 [9273108]: Look 5 days back in time to find suitable ECMWF data. 2020/11/20 18:25:23 [9273108]: Look 6 days back in time to find suitable ECMWF data. 2020/11/20 18:25:23 [9273108]: Look 7 days back in time to find suitable ECMWF data. 2020/11/20 18:25:23 [9273108]: Look 8 days back in time to find suitable ECMWF data. 2020/11/20 18:25:23 [9273108]: Look 9 days back in time to find suitable ECMWF data. 2020/11/20 18:25:23 [9273108]: ECMWF NWP data is missing -> proceed with standard values. 2020/11/20 18:25:23 [9273108]: Error message was: FileNotFoundError('ECMWF data not found after looking back 9 days.') 2020/11/20 18:25:26 [9273108]: Ozone missing in metadata -> fallback to ECMWF. 2020/11/20 18:25:26 [9273108]: Look 0 day back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 1 day back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 2 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 3 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 4 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 5 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 6 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 7 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 8 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 9 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Error with ECMWF data -> fallback to default ozone value: 500.00 2020/11/20 18:25:26 [9273108]: ECMWF Ozone value missing -> proceed with default value. 2020/11/20 18:25:26 [9273108]: Error message was: FileNotFoundError('ECMWF data not found after looking back 9 days.') 2020/11/20 18:25:26 [9273108]: Look 0 day back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 1 day back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 2 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 3 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 4 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 5 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 6 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 7 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 8 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: Look 9 days back in time to find suitable ECMWF data. 2020/11/20 18:25:26 [9273108]: ECMWF AOT value missing -> proceed with default aerosol value and type. 2020/11/20 18:25:26 [9273108]: Error message was: FileNotFoundError('ECMWF data not found after looking back 9 days.') 2020/11/20 18:25:26 [9273108]: Aerosol type: aerosol_2 100%|██████████| 2/2 [00:00<00:00, 13.16it/s] 100%|██████████| 2/2 [00:00<00:00, 13.26it/s] 100%|██████████| 2/2 [00:00<00:00, 13.34it/s] 100%|██████████| 2/2 [00:00<00:00, 13.39it/s] 2020/11/20 18:38:19 [9273108]: Look 0 day back in time to find suitable ECMWF data. 2020/11/20 18:38:19 [9273108]: Look 1 day back in time to find suitable ECMWF data. 2020/11/20 18:38:19 [9273108]: Look 2 days back in time to find suitable ECMWF data. 2020/11/20 18:38:19 [9273108]: Look 3 days back in time to find suitable ECMWF data. 2020/11/20 18:38:19 [9273108]: Look 4 days back in time to find suitable ECMWF data. 2020/11/20 18:38:19 [9273108]: Look 5 days back in time to find suitable ECMWF data. 2020/11/20 18:38:19 [9273108]: Look 6 days back in time to find suitable ECMWF data. 2020/11/20 18:38:19 [9273108]: Look 7 days back in time to find suitable ECMWF data. 2020/11/20 18:38:19 [9273108]: Look 8 days back in time to find suitable ECMWF data. 2020/11/20 18:38:19 [9273108]: Look 9 days back in time to find suitable ECMWF data. 2020/11/20 18:38:19 [9273108]: ECMWF CWV value missing -> Unable to proceed. 2020/11/20 18:38:19 [9273108]: Error message was: FileNotFoundError('ECMWF data not found after looking back 9 days.') 2020/11/20 18:38:19 [9273108]: An error occurred during atmospheric correction. BE AWARE THAT THE SCENE 9273108 (ENTITY ID LT50450322008196PAC01) HAS NOT BEEN ATMOSPHERICALLY CORRECTED! Error message was: FileNotFoundError('ECMWF data not found after looking back 9 days.') 2020/11/20 18:38:19 [9273108]: Traceback (most recent call last): File "/home/gfz-fe/scheffler/python/gms_preprocessing/gms_preprocessing/algorithms/L1C_P.py", line 905, in run_atmospheric_correction self.results = ac_gms(rs_image, self.options, logger=self.logger, script=script) File "/home/gfz-fe/miniconda3/envs/gms/lib/python3.8/site-packages/sicor/sicor_ac.py", line 1074, in ac_gms s2cwv = {options["ECMWF"]["target_resolution"]: get_ecmwf_data( File "/home/gfz-fe/miniconda3/envs/gms/lib/python3.8/site-packages/sicor/sicor.py", line 62, in get_ecmwf_data raise FileNotFoundError("ECMWF data not found after looking back %i days." % delta_day) FileNotFoundError: ECMWF data not found after looking back 9 days. 2020/11/20 18:38:19 [9273108]: Deleting input arrays for atmospheric correction... 2020/11/20 18:38:19 [9273108]: Acquired lock 'IOLock', slot #1 (closed). 2020/11/20 18:38:19 [9273108]: Writing L1C atm. corrected reflectance data. 2020/11/20 18:38:21 [9273108]: Writing L1C masks. 2020/11/20 18:38:22 [9273108]: Writing L1C cloud mask. 2020/11/20 18:38:24 [9273108]: L1C data successfully saved.

Do you see any chance to get that working?

These are my thoughts:

• SICOR currently has default values for surface pressure, ozone and aerosol values, but not for column water vapour (I guess, they should be here). Would it be possible and meaningful to implement a CWV default?
• With a look to the code (here), SICOR seems to have a water vapour retrieval implemented for multispectral data. However, there is no documentation about that. Is it principally possible to apply that to Landsat-5 and Landsat-7?
• If I remember it right, André wrote some code to backcast ECMWF climatologies to make SICOR work for older datasets. I think the respective repository was this one here. However, except of a Jupyter notebook, there is no documention.. Do you know how to use it and if that could help?