EnPT issueshttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues2024-01-04T17:30:58+01:00https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/134Missing water-leaving reflectance data at coast line in case EnPT runs in com...2024-01-04T17:30:58+01:00Daniel SchefflerMissing water-leaving reflectance data at coast line in case EnPT runs in combined AC modeLooks like this:
![grafik](/uploads/993ad8fdf6db85f01690e35ef17c5d20/grafik.png)
Pixel values are set to no-data.Looks like this:
![grafik](/uploads/993ad8fdf6db85f01690e35ef17c5d20/grafik.png)
Pixel values are set to no-data.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/110Polymer>4.14 incompatible with EnPT2023-05-26T23:16:24+02:00Daniel SchefflerPolymer>4.14 incompatible with EnPTAs reported by Dagmar Müller, the Polymer AC fails with the following error when polymer>4.14 is installed:
```python
The atmospheric correction mode was set to 'water' but Polymer cannot be imported (Error was: cannot import name 'Leve...As reported by Dagmar Müller, the Polymer AC fails with the following error when polymer>4.14 is installed:
```python
The atmospheric correction mode was set to 'water' but Polymer cannot be imported (Error was: cannot import name 'Level2' from 'polymer.main' (/home/dagmar/EnPT/polymer-v4.15/polymer/main.py))
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/109calc_solar_irradiance_CWL_FWHM_per_band delivers too smooth irradiances2023-04-04T14:07:55+02:00Maximilian Brellcalc_solar_irradiance_CWL_FWHM_per_band delivers too smooth irradiancesI do not understand what happens here (calc_solar_irradiance_CWL_FWHM_per_band(self)), but it does not look like it is convolved correctly to EnMAP CWL and FWHM...
` def calc_solar_irradiance_CWL_FWHM_per_band(self) -> np.array:
...I do not understand what happens here (calc_solar_irradiance_CWL_FWHM_per_band(self)), but it does not look like it is convolved correctly to EnMAP CWL and FWHM...
` def calc_solar_irradiance_CWL_FWHM_per_band(self) -> np.array:
from ...io.reader import Solar_Irradiance_reader
self.logger.debug('Calculating solar irradiance...')
sol_irr = Solar_Irradiance_reader(path_solar_irr_model=self.cfg.path_solar_irr, wvl_min_nm=350, wvl_max_nm=2500)
irr_bands = []
for band in self.srf.bands:
WVL_band = self.srf.srfs_wvl if self.srf.wvl_unit == 'nanometers' else self.srf.srfs_wvl * 1000
RSP_band = self.srf.srfs_norm01[band]
sol_irr_at_WVL = np.interp(WVL_band, sol_irr[:, 0], sol_irr[:, 1], left=0, right=0)
irr_bands.append(np.round(np.sum(sol_irr_at_WVL * RSP_band) / np.sum(RSP_band), 2))
return np.array(irr_bands)
`Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/105ACwater shows progress bars although `disable_progress_bars` was set to True2024-01-02T19:59:28+01:00Daniel SchefflerACwater shows progress bars although `disable_progress_bars` was set to TrueACwater shows progress bars although `disable_progress_bars` was set to True:
```python
Input parameters:
{ 'CPUs' : 32, 'auto_download_ecmwf' : False, 'average_elevation' : 0, 'blocksize' : 100, 'deadpix_P_algorithm' : 1, 'deadpix_P_in...ACwater shows progress bars although `disable_progress_bars` was set to True:
```python
Input parameters:
{ 'CPUs' : 32, 'auto_download_ecmwf' : False, 'average_elevation' : 0, 'blocksize' : 100, 'deadpix_P_algorithm' : 1, 'deadpix_P_interp_spatial' : 0, 'deadpix_P_interp_spectral' : 0, 'disable_progress_bars' : True, 'drop_bad_bands' : True, 'enable_ac' : True, 'json_config' : '', 'mode_ac' : 2, 'n_lines_to_append' : None, 'ortho_resampAlg' : 2, 'output_dir' : '/home/gfz-fe/EnPT/Output/20230228__1', 'output_format' : 1, 'output_interleave' : 0, 'path_dem' : '/home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_elevation.tif', 'path_earthSunDist' : '', 'path_l1b_enmap_image' : '/home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/ENMAP01-____L1B-DT0000001867_20220724T104521Z_007_V010111_20230228T092319Z.ZIP', 'path_l1b_enmap_image_gapfill' : '', 'path_reference_image' : '/home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_ref.tif', 'path_solar_irr' : '', 'polymer_root' : '/home/gfz-fe/scheffler/python/polymer', 'run_deadpix_P' : False, 'scale_factor_boa_ref' : 10000, 'scale_factor_toa_ref' : 10000, 'target_epsg' : None, 'target_projection_type' : 0, 'threads' : -1, 'vswir_overlap_algorithm' : 3, 'working_dir' : '' }
```
```python
2023/02/28 23:19:52: Running polymer...
2023/02/28 23:19:52: Currently only tested with ERA5 data, i.e. set True for "auto_download_ecmwf"
Initializing ENMAP product of size (1024, 1000)
2023/02/28 23:19:52: Landmask is EnMAP L1B
Download 2022-07-24 10:00:00 -> /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_100000.nc
2023-02-28 23:19:53,012 INFO Welcome to the CDS
2023-02-28 23:19:53,012 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
2023-02-28 23:19:53,266 INFO Request is queued
2023-02-28 23:19:54,310 INFO Request is running
2023-02-28 23:19:55,854 INFO Request is completed
2023-02-28 23:19:55,855 INFO Downloading https://download-0000-clone.copernicus-climate.eu/cache-compute-0000/cache/data6/adaptor.mars.internal-1677622794.747088-20076-11-6bbabe83-8d5a-4b0e-8963-a6bb1eeb68f9.nc to /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_100000.nc.tmp (9.9M)
0%| | 0.00/9.91M [00:00<?, ?B/s] 1%| | 78.0k/9.91M [00:00<00:14, 726kB/s] 7%|▋ | 754k/9.91M [00:00<00:02, 3.84MB/s] 32%|███■| 3.14M/9.91M [00:00<00:00, 12.9MB/s] 62%|██████■| 6.15M/9.91M [00:00<00:00, 19.9MB/s] 87%|████████▋ | 8.58M/9.91M [00:00<00:00, 21.9MB/s] 2023-02-28 23:19:56,694 INFO Download rate 11.8M/s
Download 2022-07-24 11:00:00 -> /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_110000.nc
2023-02-28 23:19:56,780 INFO Welcome to the CDS
2023-02-28 23:19:56,780 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
2023-02-28 23:19:56,897 INFO Request is queued
2023-02-28 23:19:57,942 INFO Request is running
2023-02-28 23:19:59,487 INFO Request is completed
2023-02-28 23:19:59,488 INFO Downloading https://download-0001-clone.copernicus-climate.eu/cache-compute-0001/cache/data2/adaptor.mars.internal-1677622797.8498647-11507-3-392d1195-e895-4600-b6db-3147f22fd704.nc to /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_110000.nc.tmp (9.9M)
0%| | 0.00/9.91M [00:00<?, ?B/s] 1%| | 78.0k/9.91M [00:00<00:13, 740kB/s] 7%|▋ | 751k/9.91M [00:00<00:02, 4.04MB/s] 35%|███▌ | 3.48M/9.91M [00:00<00:00, 14.8MB/s] 70%|███████ | 6.95M/9.91M [00:00<00:00, 23.1MB/s] 2023-02-28 23:20:00,244 INFO Download rate 13.1M/s
Starting processing at 2023-02-28 23:20:00.527176
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/104`auto_download_ecmwf` flag not working2023-12-21T18:44:26+01:00Daniel Scheffler`auto_download_ecmwf` flag not workingThe ECMWF download is executed although `auto_download_ecmwf` was set to False:
```python
Input parameters:
{ 'CPUs' : 32, 'auto_download_ecmwf' : False, 'average_elevation' : 0, 'blocksize' : 100, 'deadpix_P_algorithm' : 1, 'deadpix_P_...The ECMWF download is executed although `auto_download_ecmwf` was set to False:
```python
Input parameters:
{ 'CPUs' : 32, 'auto_download_ecmwf' : False, 'average_elevation' : 0, 'blocksize' : 100, 'deadpix_P_algorithm' : 1, 'deadpix_P_interp_spatial' : 0, 'deadpix_P_interp_spectral' : 0, 'disable_progress_bars' : True, 'drop_bad_bands' : True, 'enable_ac' : True, 'json_config' : '', 'mode_ac' : 2, 'n_lines_to_append' : None, 'ortho_resampAlg' : 2, 'output_dir' : '/home/gfz-fe/EnPT/Output/20230228__1', 'output_format' : 1, 'output_interleave' : 0, 'path_dem' : '/home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_elevation.tif', 'path_earthSunDist' : '', 'path_l1b_enmap_image' : '/home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/ENMAP01-____L1B-DT0000001867_20220724T104521Z_007_V010111_20230228T092319Z.ZIP', 'path_l1b_enmap_image_gapfill' : '', 'path_reference_image' : '/home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_ref.tif', 'path_solar_irr' : '', 'polymer_root' : '/home/gfz-fe/scheffler/python/polymer', 'run_deadpix_P' : False, 'scale_factor_boa_ref' : 10000, 'scale_factor_toa_ref' : 10000, 'target_epsg' : None, 'target_projection_type' : 0, 'threads' : -1, 'vswir_overlap_algorithm' : 3, 'working_dir' : '' }
```
```python
2023/02/28 23:19:52: Running polymer...
2023/02/28 23:19:52: Currently only tested with ERA5 data, i.e. set True for "auto_download_ecmwf"
Initializing ENMAP product of size (1024, 1000)
2023/02/28 23:19:52: Landmask is EnMAP L1B
Download 2022-07-24 10:00:00 -> /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_100000.nc
2023-02-28 23:19:53,012 INFO Welcome to the CDS
2023-02-28 23:19:53,012 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
2023-02-28 23:19:53,266 INFO Request is queued
2023-02-28 23:19:54,310 INFO Request is running
2023-02-28 23:19:55,854 INFO Request is completed
2023-02-28 23:19:55,855 INFO Downloading https://download-0000-clone.copernicus-climate.eu/cache-compute-0000/cache/data6/adaptor.mars.internal-1677622794.747088-20076-11-6bbabe83-8d5a-4b0e-8963-a6bb1eeb68f9.nc to /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_100000.nc.tmp (9.9M)
0%| | 0.00/9.91M [00:00<?, ?B/s] 1%| | 78.0k/9.91M [00:00<00:14, 726kB/s] 7%|▋ | 754k/9.91M [00:00<00:02, 3.84MB/s] 32%|███■| 3.14M/9.91M [00:00<00:00, 12.9MB/s] 62%|██████■| 6.15M/9.91M [00:00<00:00, 19.9MB/s] 87%|████████▋ | 8.58M/9.91M [00:00<00:00, 21.9MB/s] 2023-02-28 23:19:56,694 INFO Download rate 11.8M/s
Download 2022-07-24 11:00:00 -> /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_110000.nc
2023-02-28 23:19:56,780 INFO Welcome to the CDS
2023-02-28 23:19:56,780 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
2023-02-28 23:19:56,897 INFO Request is queued
2023-02-28 23:19:57,942 INFO Request is running
2023-02-28 23:19:59,487 INFO Request is completed
2023-02-28 23:19:59,488 INFO Downloading https://download-0001-clone.copernicus-climate.eu/cache-compute-0001/cache/data2/adaptor.mars.internal-1677622797.8498647-11507-3-392d1195-e895-4600-b6db-3147f22fd704.nc to /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_110000.nc.tmp (9.9M)
0%| | 0.00/9.91M [00:00<?, ?B/s] 1%| | 78.0k/9.91M [00:00<00:13, 740kB/s] 7%|▋ | 751k/9.91M [00:00<00:02, 4.04MB/s] 35%|███▌ | 3.48M/9.91M [00:00<00:00, 14.8MB/s] 70%|███████ | 6.95M/9.91M [00:00<00:00, 23.1MB/s] 2023-02-28 23:20:00,244 INFO Download rate 13.1M/s
Starting processing at 2023-02-28 23:20:00.527176
```
Also `auto_download_ecmwf` could be set to True by default.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/103"ValueError: cannot convert float NaN to integer" within dead pixel correction2023-04-03T16:31:57+02:00Daniel Scheffler"ValueError: cannot convert float NaN to integer" within dead pixel correctionEnPT was executed within the GUI on Linux:
```python
QGIS version: 3.28.3-Firenze
Qt version: 5.15.8
Python version: 3.10.8
GDAL version: 3.6.2
GEOS version: 3.11.1-CAPI-1.17.1
PROJ version: Rel. 9.1.1, December 1st, 2022
PDAL version: ...EnPT was executed within the GUI on Linux:
```python
QGIS version: 3.28.3-Firenze
Qt version: 5.15.8
Python version: 3.10.8
GDAL version: 3.6.2
GEOS version: 3.11.1-CAPI-1.17.1
PROJ version: Rel. 9.1.1, December 1st, 2022
PDAL version: 2.5.0 (git-version: d98e35)
Algorithm started at: 2023-02-28T23:16:10
Algorithm 'EnPT - EnMAP Processing Tool (v0.7.4)' starting…
Input parameters:
{ 'CPUs' : 32, 'auto_download_ecmwf' : False, 'average_elevation' : 0, 'blocksize' : 100, 'deadpix_P_algorithm' : 1, 'deadpix_P_interp_spatial' : 0, 'deadpix_P_interp_spectral' : 0, 'disable_progress_bars' : True, 'drop_bad_bands' : True, 'enable_ac' : True, 'json_config' : '', 'mode_ac' : 2, 'n_lines_to_append' : None, 'ortho_resampAlg' : 2, 'output_dir' : '/home/gfz-fe/EnPT/Output/20230228__1', 'output_format' : 1, 'output_interleave' : 0, 'path_dem' : '/home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_elevation.tif', 'path_earthSunDist' : '', 'path_l1b_enmap_image' : '/home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/ENMAP01-____L1B-DT0000001867_20220724T104521Z_007_V010111_20230228T092319Z.ZIP', 'path_l1b_enmap_image_gapfill' : '', 'path_reference_image' : '/home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_ref.tif', 'path_solar_irr' : '', 'polymer_root' : '/home/gfz-fe/scheffler/python/polymer', 'run_deadpix_P' : False, 'scale_factor_boa_ref' : 10000, 'scale_factor_toa_ref' : 10000, 'target_epsg' : None, 'target_projection_type' : 0, 'threads' : -1, 'vswir_overlap_algorithm' : 3, 'working_dir' : '' }
Calling EnPT with the following command:
enpt --CPUs 32 --auto_download_ecmwf False --average_elevation 0 --blocksize 100 --deadpix_P_algorithm spatial --deadpix_P_interp_spatial linear --deadpix_P_interp_spectral linear --disable_progress_bars True --drop_bad_bands True --enable_ac True --mode_ac combined --ortho_resampAlg gauss --output_dir /home/gfz-fe/EnPT/Output/20230228__1 --output_format ENVI --output_interleave band --path_dem /home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_elevation.tif --path_l1b_enmap_image /home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/ENMAP01-____L1B-DT0000001867_20220724T104521Z_007_V010111_20230228T092319Z.ZIP --path_reference_image /home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_ref.tif --polymer_root /home/gfz-fe/scheffler/python/polymer --run_deadpix_P False --scale_factor_boa_ref 10000 --scale_factor_toa_ref 10000 --target_projection_type UTM --threads -1 --vswir_overlap_algorithm swir_only
The log messages of the EnMAP processing tool are written to the *.log file in the specified output folder.
2023/02/28 23:16:18: Reading metadata for VNIR detector...
2023/02/28 23:16:19: Reading metadata for SWIR detector...
2023/02/28 23:16:19: Reading image masks in VNIR sensor geometry.
2023/02/28 23:16:22: Converting DN values to radiance [mW/m^2/sr/nm] for VNIR detector...
2023/02/28 23:16:23: Converting DN values to radiance [mW/m^2/sr/nm] for SWIR detector...
2023/02/28 23:16:25: Computing SNR from VNIR TOA radiance.
2023/02/28 23:16:35: Computing SNR from SWIR TOA radiance.
2023/02/28 23:16:37: Computing VNIR geolayer...
2023/02/28 23:16:40: Computing SWIR geolayer...
2023/02/28 23:17:00: Statically using band 40 for co-registration.
2023/02/28 23:17:00: Temporarily transforming EnMAP band 40 to map geometry for co-registration.
2023/02/28 23:17:09: Temporarily transforming EnMAP water mask to map geometry for co-registration.
2023/02/28 23:17:12: Preparing reference image for co-registration.
2023/02/28 23:17:14: Computing tie points between the EnMAP image and the given spatial reference image.
Calculating footprint polygon and actual data corner coordinates for reference image...
Bounding box of calculated footprint for reference image:
(365670.0, 5812395.0, 403785.0, 5849100.0)
Bounding box of calculated footprint for image to be shifted:
(365694.83596754586, 5812421.986557503, 403766.8580051669, 5849070.490111086)
Matching window position (X,Y): 384728.9599848737/5830747.846876609
Initializing tie points grid...
With respect to the provided bad data mask(s) 26 points of initially 2589 have been excluded.
Calculating tie point grid (2563 points) using 32 CPU cores...
Found 2392 matches.
Performing validity checks...
230 tie points flagged by level 1 filtering (reliability).
45 tie points flagged by level 2 filtering (SSIM).
237 tie points flagged by level 3 filtering (RANSAC)
1912 valid tie points remain after filtering.
2023/02/28 23:17:26: Generating misregistration array.
interpolation runtime: 12.19s
interpolation runtime: 12.24s
2023/02/28 23:17:51: Transforming spatial optimization results back to sensor geometry.
/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/site-packages/pyresample/kd_tree.py:384: UserWarning: Possible more than 8 neighbours within 30 m for some data points
warnings.warn(('Possible more than %s neighbours '
/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/site-packages/pyresample/kd_tree.py:384: UserWarning: Possible more than 8 neighbours within 30 m for some data points
warnings.warn(('Possible more than %s neighbours '
2023/02/28 23:18:00: Applying results of spatial optimization to geolayer.
2023/02/28 23:18:03: Pre-processing DEM for VNIR...
2023/02/28 23:18:03: Transforming DEM to VNIR sensor geometry (using first band of VNIR geolayer)...
2023/02/28 23:18:04: Pre-processing DEM for SWIR...
2023/02/28 23:18:04: Transforming DEM to SWIR sensor geometry (using first band of SWIR geolayer)...
2023/02/28 23:18:06: Transforming the 'mask_landwater' attribute from VNIR to SWIR sensor geometry.
2023/02/28 23:18:11: Starting atmospheric correction for VNIR and SWIR detector in 'combined' mode. Source radiometric unit code is 'TOARad'.
2023/02/28 23:18:11: Setting up forward operator...
2023/02/28 23:18:13: SICOR is applied to land pixels only. This may result in edge effects, e.g., at coastlines...
2023/02/28 23:18:18: Transforming VNIR data to SWIR sensor geometry to enable first guess retrievals for liquid water and ice...
2023/02/28 23:18:27: Getting observation metadata...
2023/02/28 23:18:27: Checking if observation metadata values are within LUT value ranges...
2023/02/28 23:18:27: Loading solar irradiance model...
2023/02/28 23:18:27: Loading RT LUT...
2023/02/28 23:18:27: LUT file was properly downloaded and is available for AC!
2023/02/28 23:18:27: Converting LUT to Fontenla solar model...
2023/02/28 23:18:28: Resampling LUT to EnMAP wavelengths...
2023/02/28 23:18:55: Reducing grid dimensionality of LUT to increase interpolation speed...
2023/02/28 23:18:55: Calculating absorption coefficients of liquid water and ice...
2023/02/28 23:18:56: Transforming CWV first guess map to SWIR sensor geometry to enable segmentation and 3 phases of water retrieval...
2023/02/28 23:19:00: Calculating first guess for intercept and slope of absorption feature continuum...
2023/02/28 23:19:00: Segmenting SWIR L1B spectra to enhance processing speed...
2023/02/28 23:19:04: Preparing segmented SWIR L1B data cube...
2023/02/28 23:19:12: Performing 3 phases of water retrieval...
2023/02/28 23:19:12: Setting up multiprocessing...
2023/02/28 23:19:12: Multiprocessing on 32 cpus
2023/02/28 23:19:12: Preparing optimal estimation input...
2023/02/28 23:19:12: Optimization...
2023/02/28 23:19:20: Done!
2023/02/28 23:19:20: Runtime: 8.47 s
2023/02/28 23:19:20: Extrapolating CWV retrieval map to full SWIR data cube...
2023/02/28 23:19:21: Transforming CWV retrieval map and segmentation labels to VNIR sensor geometry to enable VNIR segmentation...
2023/02/28 23:19:29: Segmenting VNIR data cube according to transformed SWIR segmentation labels...
2023/02/28 23:19:31: Starting surface reflectance retrieval...
2023/02/28 23:19:31: Calculating surface reflectance for vnir detector...
2023/02/28 23:19:31: Calculating surface reflectance for swir detector...
2023/02/28 23:19:31: Applying empirical line solution to extrapolate L2A data pixelwise...
2023/02/28 23:19:31: Extrapolating L2A spectra for vnir detector...
2023/02/28 23:19:38: Extrapolating L2A spectra for swir detector...
2023/02/28 23:19:45: Extrapolating liquid water and ice maps...
2023/02/28 23:19:49: VNIR L2A data contain negative values indicating an overcorrection. Please check for errors in the input data, the options file, or the processing code.
2023/02/28 23:19:49: VNIR L2A data contain values exceeding 1 indicating a saturation. Please check for errors in the input data, the options file, or the processing code.
2023/02/28 23:19:51: SWIR L2A data contain negative values indicating an overcorrection. Please check for errors in the input data, the options file, or the processing code.
2023/02/28 23:19:52: SWIR L2A data contain values exceeding 1 indicating a saturation. Please check for errors in the input data, the options file, or the processing code.
2023/02/28 23:19:52: SICOR atmospheric correction for EnMAP in combined mode successfully finished!
vnir
2023/02/28 23:19:52: Running polymer...
2023/02/28 23:19:52: Currently only tested with ERA5 data, i.e. set True for "auto_download_ecmwf"
Initializing ENMAP product of size (1024, 1000)
2023/02/28 23:19:52: Landmask is EnMAP L1B
Download 2022-07-24 10:00:00 -> /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_100000.nc
2023-02-28 23:19:53,012 INFO Welcome to the CDS
2023-02-28 23:19:53,012 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
2023-02-28 23:19:53,266 INFO Request is queued
2023-02-28 23:19:54,310 INFO Request is running
2023-02-28 23:19:55,854 INFO Request is completed
2023-02-28 23:19:55,855 INFO Downloading https://download-0000-clone.copernicus-climate.eu/cache-compute-0000/cache/data6/adaptor.mars.internal-1677622794.747088-20076-11-6bbabe83-8d5a-4b0e-8963-a6bb1eeb68f9.nc to /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_100000.nc.tmp (9.9M)
0%| | 0.00/9.91M [00:00<?, ?B/s] 1%| | 78.0k/9.91M [00:00<00:14, 726kB/s] 7%|▋ | 754k/9.91M [00:00<00:02, 3.84MB/s] 32%|███■| 3.14M/9.91M [00:00<00:00, 12.9MB/s] 62%|██████■| 6.15M/9.91M [00:00<00:00, 19.9MB/s] 87%|████████▋ | 8.58M/9.91M [00:00<00:00, 21.9MB/s] 2023-02-28 23:19:56,694 INFO Download rate 11.8M/s
Download 2022-07-24 11:00:00 -> /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_110000.nc
2023-02-28 23:19:56,780 INFO Welcome to the CDS
2023-02-28 23:19:56,780 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
2023-02-28 23:19:56,897 INFO Request is queued
2023-02-28 23:19:57,942 INFO Request is running
2023-02-28 23:19:59,487 INFO Request is completed
2023-02-28 23:19:59,488 INFO Downloading https://download-0001-clone.copernicus-climate.eu/cache-compute-0001/cache/data2/adaptor.mars.internal-1677622797.8498647-11507-3-392d1195-e895-4600-b6db-3147f22fd704.nc to /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_110000.nc.tmp (9.9M)
0%| | 0.00/9.91M [00:00<?, ?B/s] 1%| | 78.0k/9.91M [00:00<00:13, 740kB/s] 7%|▋ | 751k/9.91M [00:00<00:02, 4.04MB/s] 35%|███▌ | 3.48M/9.91M [00:00<00:00, 14.8MB/s] 70%|███████ | 6.95M/9.91M [00:00<00:00, 23.1MB/s] 2023-02-28 23:20:00,244 INFO Download rate 13.1M/s
Starting processing at 2023-02-28 23:20:00.527176
Processing block: size (100, 1000), offset (0, 0)
Processing block: size (100, 1000), offset (100, 0)
Processing block: size (100, 1000), offset (200, 0)
Processing block: size (100, 1000), offset (300, 0)
Processing block: size (100, 1000), offset (400, 0)
Processing block: size (100, 1000), offset (500, 0)
Processing block: size (100, 1000), offset (600, 0)
Processing block: size (100, 1000), offset (700, 0)
Processing block: size (100, 1000), offset (800, 0)
Processing block: size (100, 1000), offset (900, 0)
Processing block: size (24, 1000), offset (1000, 0)
Done in 0:00:53.595420
2023/02/28 23:20:59: Joining results of atmospheric correction.
2023/02/28 23:21:01: Re-applying dead pixel correction to correct for spectral spikes due to fringe effect.
2023/02/28 23:21:01: Correcting dead pixels of VNIR detector...
Used algorithm: linear interpolation in the linear domain
2023/02/28 23:21:04: Percentage of defective pixels: 0.18
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/home/gfz-fe/scheffler/python/EnPT/enpt/processors/dead_pixel_correction/dead_pixel_correction.py", line 349, in interp_nodata_spatially_2d
data_2d[np.isnan(data_2d)] = fill_value
ValueError: cannot convert float NaN to integer
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/gfz-fe/mambaforge/envs/enpt_full/bin/enpt", line 33, in <module>
sys.exit(load_entry_point('enpt', 'console_scripts', 'enpt')())
File "/home/gfz-fe/scheffler/python/EnPT/enpt/cli.py", line 227, in main
parsed_args.func(get_config(parsed_args))
File "/home/gfz-fe/scheffler/python/EnPT/enpt/cli.py", line 195, in run_job
CTR.run_all_processors()
File "/home/gfz-fe/scheffler/python/EnPT/enpt/execution/controller.py", line 189, in run_all_processors
self.L1_obj.correct_dead_pixels()
File "/home/gfz-fe/scheffler/python/EnPT/enpt/model/images/images_sensorgeo.py", line 753, in correct_dead_pixels
self.vnir.correct_dead_pixels()
File "/home/gfz-fe/scheffler/python/EnPT/enpt/model/images/images_sensorgeo.py", line 132, in correct_dead_pixels
.correct(self.data, self.deadpixelmap)
File "/home/gfz-fe/scheffler/python/EnPT/enpt/processors/dead_pixel_correction/dead_pixel_correction.py", line 182, in correct
return self._interpolate_nodata_spatially(image2correct, deadpixel_map)
File "/home/gfz-fe/scheffler/python/EnPT/enpt/processors/dead_pixel_correction/dead_pixel_correction.py", line 127, in _interpolate_nodata_spatially
image_sub_interp = interp_nodata_spatially_3d(image_sub, axis=1, nodata=deadpixel_map_sub, **kw)
File "/home/gfz-fe/scheffler/python/EnPT/enpt/processors/dead_pixel_correction/dead_pixel_correction.py", line 387, in interp_nodata_spatially_3d
results = pool.starmap(interp_nodata_spatially_2d, args)
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/multiprocessing/pool.py", line 774, in get
raise self._value
ValueError: cannot convert float NaN to integer
============================================================
An exception occurred. Processing failed.
The output folder '/home/gfz-fe/EnPT/Output/20230228__1' contains:
ENMAP01-____L1B-DT0000001867_20220724T104521Z_007_V010111_20230228T092319Z.log
No output raster was written.
Execution completed in 300.17 seconds (5 minutes 0 seconds)
Results:
{'success': False}
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/102Output writer raises warnings that Polymer results cannot be written although...2023-03-01T01:47:32+01:00Daniel SchefflerOutput writer raises warnings that Polymer results cannot be written although AC mode is 'land'These warnings do not make sense if Polymer is intentionally not running.These warnings do not make sense if Polymer is intentionally not running.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/101Lower case raster file extensions are incompatible with EnMAP-Box product imp...2023-03-01T01:13:28+01:00Daniel SchefflerLower case raster file extensions are incompatible with EnMAP-Box product import dialog.In case of `.bsq` file extensions:
![grafik](/uploads/22b1e9d9478add32c385565f273d4d06/grafik.png)In case of `.bsq` file extensions:
![grafik](/uploads/22b1e9d9478add32c385565f273d4d06/grafik.png)Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/100`run_deadpix_P=False` does not disable second run of dead pixel correction af...2023-03-01T00:00:29+01:00Daniel Scheffler`run_deadpix_P=False` does not disable second run of dead pixel correction after AC```python
[...]
2023/02/28 23:20:59: Joining results of atmospheric correction.
2023/02/28 23:21:01: Re-applying dead pixel correction to correct for spectral spikes due to fringe effect.
2023/02/28 23:21:01: Correcting dead pixels of VN...```python
[...]
2023/02/28 23:20:59: Joining results of atmospheric correction.
2023/02/28 23:21:01: Re-applying dead pixel correction to correct for spectral spikes due to fringe effect.
2023/02/28 23:21:01: Correcting dead pixels of VNIR detector...
Used algorithm: linear interpolation in the linear domain
2023/02/28 23:21:04: Percentage of defective pixels: 0.18
[...]
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/99Unexpected progress bar within co-registration.2023-02-28T23:06:19+01:00Daniel SchefflerUnexpected progress bar within co-registration.When resampling the spatial reference image during co-registration, a progress bar is shown although the `disable_progress_bars` parameter is set to True.When resampling the spatial reference image during co-registration, a progress bar is shown although the `disable_progress_bars` parameter is set to True.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/98Co-Registration computes way to many tie points in case of high resolution re...2023-02-17T16:17:21+01:00Daniel SchefflerCo-Registration computes way to many tie points in case of high resolution reference imageUsing a 3m resolution reference image, EnPT computes more than 80.000 tie points which takes hours to complete. However, it should not compute more than 5.000 tie points.Using a 3m resolution reference image, EnPT computes more than 80.000 tie points which takes hours to complete. However, it should not compute more than 5.000 tie points.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/97RuntimeError appears in combination with oversized DEM and gapfill options2023-03-02T18:41:55+01:00Maximilian BrellRuntimeError appears in combination with oversized DEM and gapfill optionsThe RuntimeError appears in combination with oversized DEM and gabfill options.
A possible solution could be to clip the DEM to the extent of a single tile before the. Coordinate normalization
```python
Traceback (most recent call last):...The RuntimeError appears in combination with oversized DEM and gabfill options.
A possible solution could be to clip the DEM to the extent of a single tile before the. Coordinate normalization
```python
Traceback (most recent call last):
File "/misc/hy14/Daten/EnMAP/EnPT/enpt/execution/controller.py", line 179, in run_all_processors
self.read_L1B_data()
File "/misc/hy14/Daten/EnMAP/EnPT/enpt/execution/controller.py", line 127, in read_L1B_data
self.L1_obj = RD.read_inputdata(root_dir_main=path_enmap_image, root_dir_ext=path_enmap_image_gapfill,
File "/misc/hy14/Daten/EnMAP/EnPT/enpt/io/reader.py", line 99, in read_inputdata
l1b_main_obj.meta.vnir.compute_geolayer_for_cube()
File "/misc/hy14/Daten/EnMAP/EnPT/enpt/model/metadata/metadata_sensorgeo.py", line 435, in compute_geolayer_for_cube
.compute_geolayer()
File "/misc/hy14/Daten/EnMAP/EnPT/enpt/processors/spatial_transform/spatial_transform.py", line 650, in compute_geolayer
lons_oneband, lats_oneband = self._compute_geolayer_for_unique_coeffgroup(kwargs_list[0])[:2]
File "/misc/hy14/Daten/EnMAP/EnPT/enpt/processors/spatial_transform/spatial_transform.py", line 626, in _compute_geolayer_for_unique_coeffgroup
).compute_geolayer()
File "/misc/hy14/Daten/EnMAP/EnPT/enpt/processors/spatial_transform/spatial_transform.py", line 512, in compute_geolayer
rows, cols = self.transform_LonLatHeight_to_RowCol(lon=lons, lat=lats, height=heights)
File "/misc/hy14/Daten/EnMAP/EnPT/enpt/processors/spatial_transform/spatial_transform.py", line 430, in transform_LonLatHeight_to_RowCol
self._normalize_map_coordinates(lon=lon, lat=lat, height=height)
File "/misc/hy14/Daten/EnMAP/EnPT/enpt/processors/spatial_transform/spatial_transform.py", line 345, in _normalize_map_coordinates
raise RuntimeError((llh.min(), llh.max()), msg % name)
RuntimeError: ((-1.3684170724625786, 1.034636410949972), 'Coordinate normalization yields significantly out-of-range values for longitudes. Check the coordinates and RPC coefficients.')
```https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/92[SICOR AC] IndexError: index 0 is out of bounds for axis 0 with size 02022-12-09T15:29:14+01:00Daniel Scheffler[SICOR AC] IndexError: index 0 is out of bounds for axis 0 with size 0@nbohn It seems like we currently have an issue with the first guess WV retrieval implemented in SICOR. This was reported by Dagmar Müller, I guess with a recently acquired EnMAP scene. Might be related to the latest changes in the L1 pr...@nbohn It seems like we currently have an issue with the first guess WV retrieval implemented in SICOR. This was reported by Dagmar Müller, I guess with a recently acquired EnMAP scene. Might be related to the latest changes in the L1 processing of the ground segment which masked a few more bands. Probably, it is also related to #90.
```python
2022/12/08 15:25:15: Performing first guess water vapor retrieval based on a common band ratio using VNIR data...
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/sicor/Tools/EnMAP/first_guess.py", line 205, in _compute_wv
_globs["rhos"][idx_high[0][0]] - _globs["rhos"][idx_low[0][-1]])
IndexError: index 0 is out of bounds for axis 0 with size 0
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/dagmar/micromamba/envs/enpt_full2/bin/enpt", line 10, in <module>
sys.exit(main())
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/enpt/cli.py", line 225, in main
parsed_args.func(get_config(parsed_args))
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/enpt/cli.py", line 193, in run_job
CTR.run_all_processors()
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/enpt/execution/controller.py", line 184, in run_all_processors
self.run_atmospheric_correction()
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/enpt/execution/controller.py", line 152, in run_atmospheric_correction
self.L1_obj.run_AC()
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/enpt/model/images/images_sensorgeo.py", line 858, in run_AC
AC.run_ac(self)
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/enpt/processors/atmospheric_correction/atmospheric_correction.py", line 250, in run_ac
self._run_AC__land_mode(enmap_ImageL1)
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/enpt/processors/atmospheric_correction/atmospheric_correction.py", line 129, in _run_AC__land_mode
sicor_ac_enmap(enmap_l1b=enmap_ImageL1,
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/sicor/sicor_enmap.py", line 87, in sicor_ac_enmap
fo_enmap = Fo(enmap_l1b=enmap_l1b, options=options, logger=logger)
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/sicor/AC/RtFo_3_phases.py", line 753, in __init__
self.cwv_fg = wv_band_ratio(data=self.data_vnir, water_msk=self.water_mask_vnir, fn_table=self.fn_table,
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/sicor/Tools/EnMAP/first_guess.py", line 177, in wv_band_ratio
results.get()
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
IndexError: index 0 is out of bounds for axis 0 with size 0
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/91"NameError: name 'polymer_ac_enmap' is not defined" in case acwater is instal...2022-12-06T16:33:21+01:00Daniel Scheffler"NameError: name 'polymer_ac_enmap' is not defined" in case acwater is installed but polymer is missingIf ACWater is installed but polymer is missing, the expected behavior should be to raise a warning and continue with the atmospheric correction in the "land" mode. Instead, a NameError is raised:
```python
../enpt/execution/controller.p...If ACWater is installed but polymer is missing, the expected behavior should be to raise a warning and continue with the atmospheric correction in the "land" mode. Instead, a NameError is raised:
```python
../enpt/execution/controller.py:184: in run_all_processors
self.run_atmospheric_correction()
../enpt/execution/controller.py:152: in run_atmospheric_correction
self.L1_obj.run_AC()
../enpt/model/images/images_sensorgeo.py:857: in run_AC
AC = AtmosphericCorrector(config=self.cfg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <enpt.processors.atmospheric_correction.atmospheric_correction.AtmosphericCorrector object at 0x7fbf90744220>
config = {'general_opts': {'CPUs': 32,
'average_elevation': 0,
'create_logfile': True,
...rradiance/SUNp1fontenla__350-2500nm_@0.1nm_converted.txt',
'scale_factor_toa_ref': 10000}}}
def __init__(self, config: EnPTConfig = None):
"""Create an instance of AtmosphericCorrector."""
self.cfg = config
> self.is_polymer_installed = polymer_ac_enmap is not None
E NameError: name 'polymer_ac_enmap' is not defined
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/89ValueError due to capital letters in EnMAP Zipfile extension2022-11-15T16:19:43+01:00Daniel SchefflerValueError due to capital letters in EnMAP Zipfile extension```python
Traceback (most recent call last):
File "/home/expedition/miniconda3/envs/enpt_full/bin/enpt", line 10, in <module>
sys.exit(main())
File "/home/expedition/miniconda3/envs/enpt_full/lib/python3.10/site-packages/enpt/cli.py", li...```python
Traceback (most recent call last):
File "/home/expedition/miniconda3/envs/enpt_full/bin/enpt", line 10, in <module>
sys.exit(main())
File "/home/expedition/miniconda3/envs/enpt_full/lib/python3.10/site-packages/enpt/cli.py", line 225, in main
parsed_args.func(get_config(parsed_args))
File "/home/expedition/miniconda3/envs/enpt_full/lib/python3.10/site-packages/enpt/cli.py", line 193, in run_job
CTR.run_all_processors()
File "/home/expedition/miniconda3/envs/enpt_full/lib/python3.10/site-packages/enpt/execution/controller.py", line 176, in run_all_processors
self.read_L1B_data()
File "/home/expedition/miniconda3/envs/enpt_full/lib/python3.10/site-packages/enpt/execution/controller.py", line 114, in read_L1B_data
raise ValueError("The parameter 'path_enmap_image' must be a directory or the path to an existing zip "
ValueError: The parameter 'path_enmap_image' must be a directory or the path to an existing zip archive. Received /home/expedition/Documents/EnMAP_data/ENMAP01-____L1B-DT0000001567_20220709T105740Z_032_V010110_20221114T120244Z.ZIP.
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/84Orthorectification fails if L1B data are pseudo-projected/rotated2022-03-22T16:26:43+01:00Daniel SchefflerOrthorectification fails if L1B data are pseudo-projected/rotatedIf the sensor geometry input data contain a map info with a rotation, the orthorectification fails:
```python
2022/03/16 20:50:59: Starting orthorectification...
2022/03/16 20:50:59: Computed common target extent of orthorectified i...If the sensor geometry input data contain a map info with a rotation, the orthorectification fails:
```python
2022/03/16 20:50:59: Starting orthorectification...
2022/03/16 20:50:59: Computed common target extent of orthorectified image (xmin, ymin, xmax, ymax in EPSG 32632): (620070.0, 5226540.0, 655080.0, 5262900.0)
2022/03/16 21:10:23: Orthorectifying VNIR data using 'nearest' resampling algorithm...
2022/03/16 21:11:46: Orthorectifying SWIR data using 'nearest' resampling algorithm...
2022/03/16 21:26:54: Merging VNIR and SWIR data...
2022/03/16 21:26:55: Orthorectifying 'mask_landwater' attribute...
FAILED
tests/test_orthorectification.py:110 (Test_Orthorectifier_DLR.test_run_transformation)
self = <tests.test_orthorectification.Test_Orthorectifier_DLR testMethod=test_run_transformation>
def test_run_transformation(self):
OR = Orthorectifier(config=self.config)
> L2_obj = OR.run_transformation(self.L1_obj)
test_orthorectification.py:113:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../enpt/processors/orthorectification/orthorectification.py:142: in run_transformation
attr_ortho = GeoArray(*GT_2D.to_map_geometry(attr, **kw_trafo), nodata=attr.nodata)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <enpt.processors.spatial_transform.spatial_transform.Geometry_Transformer object at 0x7f88f58d52b0>
path_or_geoarray_sensorgeo = <geoarray.baseclasses.GeoArray object at 0x7f88f75ed670>
tgt_prj = 32632, tgt_extent = (620070.0, 5226540.0, 655080.0, 5262900.0)
tgt_res = None, tgt_coordgrid = (array([ 0, 30]), array([ 0, 30]))
area_definition = None
def to_map_geometry(self,
path_or_geoarray_sensorgeo: Union[str, GeoArray, np.ndarray],
tgt_prj: Union[str, int] = None,
tgt_extent: Tuple[float, float, float, float] = None,
tgt_res: Tuple[float, float] = None,
tgt_coordgrid: Tuple[Tuple, Tuple] = None,
area_definition: AreaDefinition = None):
data_sensorgeo = GeoArray(path_or_geoarray_sensorgeo)
if data_sensorgeo.is_map_geo:
> raise RuntimeError('The dataset to be transformed into map geometry already represents map geometry.')
E RuntimeError: The dataset to be transformed into map geometry already represents map geometry.
../enpt/processors/spatial_transform/spatial_transform.py:87: RuntimeError
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/81Coverage raises warnings when running in multiprocessing2022-03-10T18:51:10+01:00Daniel SchefflerCoverage raises warnings when running in multiprocessing```
coverage data file: cannot unpack non-iterable NoneType object
Coverage.py warning: Data file '/home/gfz-fe/scheffler/python/findr_homo/.coverage.geoms.1279091.080330' doesn't seem to be a coverage data file: Couldn't use data file '...```
coverage data file: cannot unpack non-iterable NoneType object
Coverage.py warning: Data file '/home/gfz-fe/scheffler/python/findr_homo/.coverage.geoms.1279091.080330' doesn't seem to be a coverage data file: Couldn't use data file '/home/gfz-fe/scheffler/python/findr_homo/.coverage.geoms.1279091.080330': no such table: coverage_schema
Coverage.py warning: Couldn't use data file '/home/gfz-fe/scheffler/python/findr_homo/.coverage.geoms.1274268.063192': database disk image is malformed
```
For example, in [this pipeline](https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/jobs/123288#L847).
Solution:
Properly close and join multiprocessing.Pool like this:
```python
with multiprocessing.Pool() as pool:
pool.map(func, args)
pool.close()
pool.join()
```
This has to be done in the EnPT code (https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/merge_requests/60/diffs?commit_id=6ffec51a18bc764e20a85b1ef087f4ccd6933c11) AND ALL EXTERNAL LIBRARIES where multiprocessing.Pool is called, i.e., in arosics, spechomo, sicor, sensormapgeo, ...
Another way is to make coverage exclude external packages by adding this to the .coveragerc:
```bash
[run]
omit = */site-packages/*,*/tests/*,*/.eggs/*
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/79FileNotFoundError when using the official EnMAP test data downloaded from enm...2021-06-18T17:24:09+02:00Daniel SchefflerFileNotFoundError when using the official EnMAP test data downloaded from enmap.org.When using the data downloaded from https://www.enmap.org/data_tools/testdata/, EnPT raises a FileNotFoundError.
```python
Traceback (most recent call last):
File "/home/gfz-fe/miniconda3/envs/enpt_full/bin/enpt", line 10, in <module>
s...When using the data downloaded from https://www.enmap.org/data_tools/testdata/, EnPT raises a FileNotFoundError.
```python
Traceback (most recent call last):
File "/home/gfz-fe/miniconda3/envs/enpt_full/bin/enpt", line 10, in <module>
sys.exit(main())
File "/home/gfz-fe/miniconda3/envs/enpt_full/lib/python3.9/site-packages/enpt/cli.py", line 222, in main
parsed_args.func(get_config(parsed_args))
File "/home/gfz-fe/miniconda3/envs/enpt_full/lib/python3.9/site-packages/enpt/cli.py", line 190, in run_job
CTR.run_all_processors()
File "/home/gfz-fe/miniconda3/envs/enpt_full/lib/python3.9/site-packages/enpt/execution/controller.py", line 166, in run_all_processors
self.read_L1B_data()
File "/home/gfz-fe/miniconda3/envs/enpt_full/lib/python3.9/site-packages/enpt/execution/controller.py", line 117, in read_L1B_data
self.L1_obj = RD.read_inputdata(root_dir_main=path_enmap_image, root_dir_ext=path_enmap_image_gapfill,
File "/home/gfz-fe/miniconda3/envs/enpt_full/lib/python3.9/site-packages/enpt/io/reader.py", line 82, in read_inputdata
self.validate_input(root_dir_main, root_dir_ext)
File "/home/gfz-fe/miniconda3/envs/enpt_full/lib/python3.9/site-packages/enpt/io/reader.py", line 114, in validate_input
self._validate_enmap_l1b_rootdir(root_dir_main)
File "/home/gfz-fe/miniconda3/envs/enpt_full/lib/python3.9/site-packages/enpt/io/reader.py", line 149, in _validate_enmap_l1b_rootdir
raise FileNotFoundError('The root directory of the EnMAP image %s misses a file with the pattern %s.'
FileNotFoundError: The root directory of the EnMAP image /tmp/tmpz7iw_5vg/image_main misses a file with the pattern *-HISTORY.XML.
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/78Spatial artifacts in EnMAP L2A after running EnPT in combined AC mode.2022-08-12T17:18:06+02:00Daniel SchefflerSpatial artifacts in EnMAP L2A after running EnPT in combined AC mode.When running EnPT in the 'combined' AC mode, we have a yellowish area as pointed out in https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/merge_requests/54#note_57492 in the L2A results of EnPT.
From my point of view, this can...When running EnPT in the 'combined' AC mode, we have a yellowish area as pointed out in https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/merge_requests/54#note_57492 in the L2A results of EnPT.
From my point of view, this cannot come from ACwater as the spectral information over land is coming from SICOR. Probably, there is something wrong with the configuration of the software test (see [here](https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/merge_requests/54/diffs#876b55ca4274f27b0138fc581dc1153b0ccb9b79_65_71)) which is a different one than we use for the 'land' mode. This could also explain why you don´t see this effect when only SICOR is running ('land' mode), @nbohn. I looked a bit deeper into this and found out that mainly the shorter wavelenghts are are affected by these anomalies. To me, it looks like it is somehow related to the segmentation in SICOR.
What do you think?
This is EnMAP L2A band 1 as it comes out of EnPT in case of the 'combined' AC mode. Spectra jump from positive values to slightly negative values at the edge of the artifact:
![image](/uploads/4da4edead0c41c568626665169674547/image.png)Niklas BohnNiklas Bohnhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/73Unexpected pixel value in the land/water mask passed to SICOR.2021-03-15T17:35:00+01:00Daniel SchefflerUnexpected pixel value in the land/water mask passed to SICOR.See https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/47#note_45547
There should be only 3 different pixel values in the land/water mask: 0=background, 1=land, 2=water. The pixel value 3 might be a resampling artifact.See https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/47#note_45547
There should be only 3 different pixel values in the land/water mask: 0=background, 1=land, 2=water. The pixel value 3 might be a resampling artifact.Daniel SchefflerDaniel Scheffler