Cope with ACCESS CHANGE to CAMS Global atmospheric composition forecast data
In GTS2 we use SICOR ECMWF downloader. It current attempts to download the following products:
fc_black_carbon_AOT_550nm
fc_GMES_ozone
fc_organic_matter_AOT_550nm
fc_SLP
fc_T2M
fc_total_AOT_550nm
fc_dust_AOT_550nm
fc_O3
fc_sea_salt_AOT_550nm
fc_sulphate_AOT_550nm
fc_TCWV
Since July 1st 2021, we are not getting data for:
fc_O3
fc_SLP
fc_T2M
In the past weeks we got few email from ECMWF announcing ACCESS CHANGE to CAMS Global atmospheric composition forecast data
. Here is part of the email content:
This message is a final reminder that if you currently retrieve CAMS Near-real-time (July 2012 - present) data through the ECMWF public web API service, this facility is being stopped on 30 June 2021.
The new access method is through the CAMS Atmosphere Data Store (ADS). You can now access the available data interactively through the ADS web interface (CAMS global atmospheric composition forecasts) and/or programmatically using the API.
Please follow our guidelines on How to migrate to CDS API on the Atmosphere Data Store (ADS). Some CAMS global atmospheric composition forecasts data access specifics are also provided in the new consolidated CAMS global atmospheric composition forecasts data documentation.
That can easily be seen if we do a quick test for the data product fc_O3
, this is, we get the notification that we need to port:
{'grib': True, 'output_file_name': '/tmp/tmp07v6vpu2/fc_O3/2013/01/20130101_fc_O3.h5', 'parameter': 'fc_O3', 'year': 2013, 'month': 1, 'day_start': 1, 'day_end': 1, 'status': 'error', 'error': "APIException('ecmwf.API error 1: ERROR 193 (DATASET_MIGRATED): This dataset is only available via the ADS')"}
For one of the other ones (fc_TCWV
) we were able to get data:
{'grib': True, 'output_file_name': '/tmp/tmp6_af8k30/fc_TCWV/2013/01/20130101_fc_TCWV.h5', 'parameter': 'fc_TCWV', 'year': 2013, 'month': 1, 'day_start': 1, 'day_end': 1, 'status': 'OK'}
In attempt to get data for the ones missing we have followed the instructions to do the port and in branch port_to_ads
there is an attempt to get it done. We have replaced the ecmwf-api-client
by the cdsapi
and changed the code accordingly. However, we start to realize that we need both services to download all the products and that the api request might also differ depending on the service. Before someone moves forward and do a proper implementation, we have the following questions:
Questions:
- For
fc_O3
,fc_SLP
, andfc_T2M
from which data resource should we download it? The new service has the following resources:
0 "cams-europe-air-quality-forecasts"
1 "cams-global-atmospheric-composition-forecasts"
2 "cams-global-atmospheric-composition-forecasts-external"
3 "cams-global-atmospheric-composition-forecasts-internal"
4 "cams-global-emission-inventories"
5 "cams-global-ghg-reanalysis-egg4"
6 "cams-global-ghg-reanalysis-egg4-monthly"
7 "cams-global-greenhouse-gas-inversion"
8 "cams-global-radiative-forcing-auxilliary-variables"
9 "cams-global-radiative-forcings"
10 "cams-global-reanalysis-eac4"
11 "cams-global-reanalysis-eac4-external"
12 "cams-global-reanalysis-eac4-internal"
13 "cams-global-reanalysis-eac4-monthly"
14 "cams-solar-radiation-timeseries"
More info at: https://ads.atmosphere.copernicus.eu/api/v2/resources/. We tried cams-global-atmospheric-composition-forecasts-external
and cams-global-reanalysis-eac4-external
but we always got:
2021-07-14 17:19:40,184 ERROR ValueError: too many values to unpack (expected 2)
Here a small test (the request dictionary was obtained via a print from SICOR), the example comes from here (to test locally you need to create ~/.cdsapirc
credentials file):
import cdsapi
cds = cdsapi.Client()
cds.retrieve("cams-global-atmospheric-composition-forecasts-external", {'class': 'mc', 'dataset': 'cams_nrealtime', 'expver': '0001', 'levtype': 'sfc', 'param': '206.210', 'step': '0/3/6/9/12/15/18/21/24/27/30/33/36/39/42/45/48/51/54/57/60/63/66/69/72/75/78/81/84/87/90/93/96/99/102/105/108/111/114/117/120', 'stream': 'oper', 'time': '00:00:00', 'type': 'fc', 'target': '/tmp/tmpkib72ftk/fc_O3/2021/07/20210701_fc_O3.h5.grb', 'date': '2021-07-01/to/2021-07-01'})
- Are
fc_O3
,fc_SLP
, andfc_T2M
really required? If not, it might happen that we do not need to update SICOR ECMWF downloader. In case it is needed for other data products at GFZ, do you think GTS2 needs it?
@nbohn what are your thoughts? @choh anything you know about this topic from the previous developers of GTS2?