ECMWF data crawler only works for acquisition dates between 07/2012 and 06/2018 for the default variables.

I need to process some Landsat data acquired in 2008 in SICOR and I noticed that the ECMWF crawler does not download most of the variables for this year. I took a closer look into the code and I think I found the reason for it.

The ECMWF_download class downloads the individual variables (T2M, O3, SLP, TCWV, Ozone, AOT, ...) from different ECMWF datasets whereas each dataset only covers a specific time interval:

dataset	dataset slug in the code	interval
ERA Interim, Daily	interim	01/1979 - 08/2019
CAMS Near-real-time	cams_nrealtime	07/2012 - present
MACC Near-real-time	macc_nrealtime	07/2012 - 06/2018

Since only a few variables use the ERA interim dataset in SICOR, the download fails for most of them.

I already created a new branch and will try to fix that. My plan is to implement a check that takes the date_from and date_to parameters that are passed to download_variables() and automatically selects a suitable ECMWF dataset.

The question is only if we should prefer specific ECMWF datasets over others. As I am not familiar with these data I would just use the datasets with the newer time interval first if available. Or do you have a better idea?

Another issue is that the ERA interim dataset has phased out in 08/2019 and it is recommended to switch to ERA 5. However, this seems to use another API.