EnPT issueshttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues2019-10-15T22:47:19+02:00https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/28[Orthorectification] Incorrect L2A output extent.2019-10-15T22:47:19+02:00Daniel Scheffler[Orthorectification] Incorrect L2A output extent.L2A output is currently cut off like this:
![image](/uploads/12a8a1e162ea3c94a2b62919e4f87766/image.png)L2A output is currently cut off like this:
![image](/uploads/12a8a1e162ea3c94a2b62919e4f87766/image.png)Add geometry processorsDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/134Missing water-leaving reflectance data at coast line in case EnPT runs in com...2024-01-04T17:30:58+01:00Daniel SchefflerMissing water-leaving reflectance data at coast line in case EnPT runs in combined AC modeLooks like this:
![grafik](/uploads/993ad8fdf6db85f01690e35ef17c5d20/grafik.png)
Pixel values are set to no-data.Looks like this:
![grafik](/uploads/993ad8fdf6db85f01690e35ef17c5d20/grafik.png)
Pixel values are set to no-data.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/110Polymer>4.14 incompatible with EnPT2023-05-26T23:16:24+02:00Daniel SchefflerPolymer>4.14 incompatible with EnPTAs reported by Dagmar Müller, the Polymer AC fails with the following error when polymer>4.14 is installed:
```python
The atmospheric correction mode was set to 'water' but Polymer cannot be imported (Error was: cannot import name 'Leve...As reported by Dagmar Müller, the Polymer AC fails with the following error when polymer>4.14 is installed:
```python
The atmospheric correction mode was set to 'water' but Polymer cannot be imported (Error was: cannot import name 'Level2' from 'polymer.main' (/home/dagmar/EnPT/polymer-v4.15/polymer/main.py))
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/103"ValueError: cannot convert float NaN to integer" within dead pixel correction2023-04-03T16:31:57+02:00Daniel Scheffler"ValueError: cannot convert float NaN to integer" within dead pixel correctionEnPT was executed within the GUI on Linux:
```python
QGIS version: 3.28.3-Firenze
Qt version: 5.15.8
Python version: 3.10.8
GDAL version: 3.6.2
GEOS version: 3.11.1-CAPI-1.17.1
PROJ version: Rel. 9.1.1, December 1st, 2022
PDAL version: ...EnPT was executed within the GUI on Linux:
```python
QGIS version: 3.28.3-Firenze
Qt version: 5.15.8
Python version: 3.10.8
GDAL version: 3.6.2
GEOS version: 3.11.1-CAPI-1.17.1
PROJ version: Rel. 9.1.1, December 1st, 2022
PDAL version: 2.5.0 (git-version: d98e35)
Algorithm started at: 2023-02-28T23:16:10
Algorithm 'EnPT - EnMAP Processing Tool (v0.7.4)' starting…
Input parameters:
{ 'CPUs' : 32, 'auto_download_ecmwf' : False, 'average_elevation' : 0, 'blocksize' : 100, 'deadpix_P_algorithm' : 1, 'deadpix_P_interp_spatial' : 0, 'deadpix_P_interp_spectral' : 0, 'disable_progress_bars' : True, 'drop_bad_bands' : True, 'enable_ac' : True, 'json_config' : '', 'mode_ac' : 2, 'n_lines_to_append' : None, 'ortho_resampAlg' : 2, 'output_dir' : '/home/gfz-fe/EnPT/Output/20230228__1', 'output_format' : 1, 'output_interleave' : 0, 'path_dem' : '/home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_elevation.tif', 'path_earthSunDist' : '', 'path_l1b_enmap_image' : '/home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/ENMAP01-____L1B-DT0000001867_20220724T104521Z_007_V010111_20230228T092319Z.ZIP', 'path_l1b_enmap_image_gapfill' : '', 'path_reference_image' : '/home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_ref.tif', 'path_solar_irr' : '', 'polymer_root' : '/home/gfz-fe/scheffler/python/polymer', 'run_deadpix_P' : False, 'scale_factor_boa_ref' : 10000, 'scale_factor_toa_ref' : 10000, 'target_epsg' : None, 'target_projection_type' : 0, 'threads' : -1, 'vswir_overlap_algorithm' : 3, 'working_dir' : '' }
Calling EnPT with the following command:
enpt --CPUs 32 --auto_download_ecmwf False --average_elevation 0 --blocksize 100 --deadpix_P_algorithm spatial --deadpix_P_interp_spatial linear --deadpix_P_interp_spectral linear --disable_progress_bars True --drop_bad_bands True --enable_ac True --mode_ac combined --ortho_resampAlg gauss --output_dir /home/gfz-fe/EnPT/Output/20230228__1 --output_format ENVI --output_interleave band --path_dem /home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_elevation.tif --path_l1b_enmap_image /home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/ENMAP01-____L1B-DT0000001867_20220724T104521Z_007_V010111_20230228T092319Z.ZIP --path_reference_image /home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_ref.tif --polymer_root /home/gfz-fe/scheffler/python/polymer --run_deadpix_P False --scale_factor_boa_ref 10000 --scale_factor_toa_ref 10000 --target_projection_type UTM --threads -1 --vswir_overlap_algorithm swir_only
The log messages of the EnMAP processing tool are written to the *.log file in the specified output folder.
2023/02/28 23:16:18: Reading metadata for VNIR detector...
2023/02/28 23:16:19: Reading metadata for SWIR detector...
2023/02/28 23:16:19: Reading image masks in VNIR sensor geometry.
2023/02/28 23:16:22: Converting DN values to radiance [mW/m^2/sr/nm] for VNIR detector...
2023/02/28 23:16:23: Converting DN values to radiance [mW/m^2/sr/nm] for SWIR detector...
2023/02/28 23:16:25: Computing SNR from VNIR TOA radiance.
2023/02/28 23:16:35: Computing SNR from SWIR TOA radiance.
2023/02/28 23:16:37: Computing VNIR geolayer...
2023/02/28 23:16:40: Computing SWIR geolayer...
2023/02/28 23:17:00: Statically using band 40 for co-registration.
2023/02/28 23:17:00: Temporarily transforming EnMAP band 40 to map geometry for co-registration.
2023/02/28 23:17:09: Temporarily transforming EnMAP water mask to map geometry for co-registration.
2023/02/28 23:17:12: Preparing reference image for co-registration.
2023/02/28 23:17:14: Computing tie points between the EnMAP image and the given spatial reference image.
Calculating footprint polygon and actual data corner coordinates for reference image...
Bounding box of calculated footprint for reference image:
(365670.0, 5812395.0, 403785.0, 5849100.0)
Bounding box of calculated footprint for image to be shifted:
(365694.83596754586, 5812421.986557503, 403766.8580051669, 5849070.490111086)
Matching window position (X,Y): 384728.9599848737/5830747.846876609
Initializing tie points grid...
With respect to the provided bad data mask(s) 26 points of initially 2589 have been excluded.
Calculating tie point grid (2563 points) using 32 CPU cores...
Found 2392 matches.
Performing validity checks...
230 tie points flagged by level 1 filtering (reliability).
45 tie points flagged by level 2 filtering (SSIM).
237 tie points flagged by level 3 filtering (RANSAC)
1912 valid tie points remain after filtering.
2023/02/28 23:17:26: Generating misregistration array.
interpolation runtime: 12.19s
interpolation runtime: 12.24s
2023/02/28 23:17:51: Transforming spatial optimization results back to sensor geometry.
/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/site-packages/pyresample/kd_tree.py:384: UserWarning: Possible more than 8 neighbours within 30 m for some data points
warnings.warn(('Possible more than %s neighbours '
/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/site-packages/pyresample/kd_tree.py:384: UserWarning: Possible more than 8 neighbours within 30 m for some data points
warnings.warn(('Possible more than %s neighbours '
2023/02/28 23:18:00: Applying results of spatial optimization to geolayer.
2023/02/28 23:18:03: Pre-processing DEM for VNIR...
2023/02/28 23:18:03: Transforming DEM to VNIR sensor geometry (using first band of VNIR geolayer)...
2023/02/28 23:18:04: Pre-processing DEM for SWIR...
2023/02/28 23:18:04: Transforming DEM to SWIR sensor geometry (using first band of SWIR geolayer)...
2023/02/28 23:18:06: Transforming the 'mask_landwater' attribute from VNIR to SWIR sensor geometry.
2023/02/28 23:18:11: Starting atmospheric correction for VNIR and SWIR detector in 'combined' mode. Source radiometric unit code is 'TOARad'.
2023/02/28 23:18:11: Setting up forward operator...
2023/02/28 23:18:13: SICOR is applied to land pixels only. This may result in edge effects, e.g., at coastlines...
2023/02/28 23:18:18: Transforming VNIR data to SWIR sensor geometry to enable first guess retrievals for liquid water and ice...
2023/02/28 23:18:27: Getting observation metadata...
2023/02/28 23:18:27: Checking if observation metadata values are within LUT value ranges...
2023/02/28 23:18:27: Loading solar irradiance model...
2023/02/28 23:18:27: Loading RT LUT...
2023/02/28 23:18:27: LUT file was properly downloaded and is available for AC!
2023/02/28 23:18:27: Converting LUT to Fontenla solar model...
2023/02/28 23:18:28: Resampling LUT to EnMAP wavelengths...
2023/02/28 23:18:55: Reducing grid dimensionality of LUT to increase interpolation speed...
2023/02/28 23:18:55: Calculating absorption coefficients of liquid water and ice...
2023/02/28 23:18:56: Transforming CWV first guess map to SWIR sensor geometry to enable segmentation and 3 phases of water retrieval...
2023/02/28 23:19:00: Calculating first guess for intercept and slope of absorption feature continuum...
2023/02/28 23:19:00: Segmenting SWIR L1B spectra to enhance processing speed...
2023/02/28 23:19:04: Preparing segmented SWIR L1B data cube...
2023/02/28 23:19:12: Performing 3 phases of water retrieval...
2023/02/28 23:19:12: Setting up multiprocessing...
2023/02/28 23:19:12: Multiprocessing on 32 cpus
2023/02/28 23:19:12: Preparing optimal estimation input...
2023/02/28 23:19:12: Optimization...
2023/02/28 23:19:20: Done!
2023/02/28 23:19:20: Runtime: 8.47 s
2023/02/28 23:19:20: Extrapolating CWV retrieval map to full SWIR data cube...
2023/02/28 23:19:21: Transforming CWV retrieval map and segmentation labels to VNIR sensor geometry to enable VNIR segmentation...
2023/02/28 23:19:29: Segmenting VNIR data cube according to transformed SWIR segmentation labels...
2023/02/28 23:19:31: Starting surface reflectance retrieval...
2023/02/28 23:19:31: Calculating surface reflectance for vnir detector...
2023/02/28 23:19:31: Calculating surface reflectance for swir detector...
2023/02/28 23:19:31: Applying empirical line solution to extrapolate L2A data pixelwise...
2023/02/28 23:19:31: Extrapolating L2A spectra for vnir detector...
2023/02/28 23:19:38: Extrapolating L2A spectra for swir detector...
2023/02/28 23:19:45: Extrapolating liquid water and ice maps...
2023/02/28 23:19:49: VNIR L2A data contain negative values indicating an overcorrection. Please check for errors in the input data, the options file, or the processing code.
2023/02/28 23:19:49: VNIR L2A data contain values exceeding 1 indicating a saturation. Please check for errors in the input data, the options file, or the processing code.
2023/02/28 23:19:51: SWIR L2A data contain negative values indicating an overcorrection. Please check for errors in the input data, the options file, or the processing code.
2023/02/28 23:19:52: SWIR L2A data contain values exceeding 1 indicating a saturation. Please check for errors in the input data, the options file, or the processing code.
2023/02/28 23:19:52: SICOR atmospheric correction for EnMAP in combined mode successfully finished!
vnir
2023/02/28 23:19:52: Running polymer...
2023/02/28 23:19:52: Currently only tested with ERA5 data, i.e. set True for "auto_download_ecmwf"
Initializing ENMAP product of size (1024, 1000)
2023/02/28 23:19:52: Landmask is EnMAP L1B
Download 2022-07-24 10:00:00 -> /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_100000.nc
2023-02-28 23:19:53,012 INFO Welcome to the CDS
2023-02-28 23:19:53,012 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
2023-02-28 23:19:53,266 INFO Request is queued
2023-02-28 23:19:54,310 INFO Request is running
2023-02-28 23:19:55,854 INFO Request is completed
2023-02-28 23:19:55,855 INFO Downloading https://download-0000-clone.copernicus-climate.eu/cache-compute-0000/cache/data6/adaptor.mars.internal-1677622794.747088-20076-11-6bbabe83-8d5a-4b0e-8963-a6bb1eeb68f9.nc to /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_100000.nc.tmp (9.9M)
0%| | 0.00/9.91M [00:00<?, ?B/s] 1%| | 78.0k/9.91M [00:00<00:14, 726kB/s] 7%|▋ | 754k/9.91M [00:00<00:02, 3.84MB/s] 32%|███■| 3.14M/9.91M [00:00<00:00, 12.9MB/s] 62%|██████■| 6.15M/9.91M [00:00<00:00, 19.9MB/s] 87%|████████▋ | 8.58M/9.91M [00:00<00:00, 21.9MB/s] 2023-02-28 23:19:56,694 INFO Download rate 11.8M/s
Download 2022-07-24 11:00:00 -> /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_110000.nc
2023-02-28 23:19:56,780 INFO Welcome to the CDS
2023-02-28 23:19:56,780 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
2023-02-28 23:19:56,897 INFO Request is queued
2023-02-28 23:19:57,942 INFO Request is running
2023-02-28 23:19:59,487 INFO Request is completed
2023-02-28 23:19:59,488 INFO Downloading https://download-0001-clone.copernicus-climate.eu/cache-compute-0001/cache/data2/adaptor.mars.internal-1677622797.8498647-11507-3-392d1195-e895-4600-b6db-3147f22fd704.nc to /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_110000.nc.tmp (9.9M)
0%| | 0.00/9.91M [00:00<?, ?B/s] 1%| | 78.0k/9.91M [00:00<00:13, 740kB/s] 7%|▋ | 751k/9.91M [00:00<00:02, 4.04MB/s] 35%|███▌ | 3.48M/9.91M [00:00<00:00, 14.8MB/s] 70%|███████ | 6.95M/9.91M [00:00<00:00, 23.1MB/s] 2023-02-28 23:20:00,244 INFO Download rate 13.1M/s
Starting processing at 2023-02-28 23:20:00.527176
Processing block: size (100, 1000), offset (0, 0)
Processing block: size (100, 1000), offset (100, 0)
Processing block: size (100, 1000), offset (200, 0)
Processing block: size (100, 1000), offset (300, 0)
Processing block: size (100, 1000), offset (400, 0)
Processing block: size (100, 1000), offset (500, 0)
Processing block: size (100, 1000), offset (600, 0)
Processing block: size (100, 1000), offset (700, 0)
Processing block: size (100, 1000), offset (800, 0)
Processing block: size (100, 1000), offset (900, 0)
Processing block: size (24, 1000), offset (1000, 0)
Done in 0:00:53.595420
2023/02/28 23:20:59: Joining results of atmospheric correction.
2023/02/28 23:21:01: Re-applying dead pixel correction to correct for spectral spikes due to fringe effect.
2023/02/28 23:21:01: Correcting dead pixels of VNIR detector...
Used algorithm: linear interpolation in the linear domain
2023/02/28 23:21:04: Percentage of defective pixels: 0.18
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/home/gfz-fe/scheffler/python/EnPT/enpt/processors/dead_pixel_correction/dead_pixel_correction.py", line 349, in interp_nodata_spatially_2d
data_2d[np.isnan(data_2d)] = fill_value
ValueError: cannot convert float NaN to integer
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/gfz-fe/mambaforge/envs/enpt_full/bin/enpt", line 33, in <module>
sys.exit(load_entry_point('enpt', 'console_scripts', 'enpt')())
File "/home/gfz-fe/scheffler/python/EnPT/enpt/cli.py", line 227, in main
parsed_args.func(get_config(parsed_args))
File "/home/gfz-fe/scheffler/python/EnPT/enpt/cli.py", line 195, in run_job
CTR.run_all_processors()
File "/home/gfz-fe/scheffler/python/EnPT/enpt/execution/controller.py", line 189, in run_all_processors
self.L1_obj.correct_dead_pixels()
File "/home/gfz-fe/scheffler/python/EnPT/enpt/model/images/images_sensorgeo.py", line 753, in correct_dead_pixels
self.vnir.correct_dead_pixels()
File "/home/gfz-fe/scheffler/python/EnPT/enpt/model/images/images_sensorgeo.py", line 132, in correct_dead_pixels
.correct(self.data, self.deadpixelmap)
File "/home/gfz-fe/scheffler/python/EnPT/enpt/processors/dead_pixel_correction/dead_pixel_correction.py", line 182, in correct
return self._interpolate_nodata_spatially(image2correct, deadpixel_map)
File "/home/gfz-fe/scheffler/python/EnPT/enpt/processors/dead_pixel_correction/dead_pixel_correction.py", line 127, in _interpolate_nodata_spatially
image_sub_interp = interp_nodata_spatially_3d(image_sub, axis=1, nodata=deadpixel_map_sub, **kw)
File "/home/gfz-fe/scheffler/python/EnPT/enpt/processors/dead_pixel_correction/dead_pixel_correction.py", line 387, in interp_nodata_spatially_3d
results = pool.starmap(interp_nodata_spatially_2d, args)
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/multiprocessing/pool.py", line 774, in get
raise self._value
ValueError: cannot convert float NaN to integer
============================================================
An exception occurred. Processing failed.
The output folder '/home/gfz-fe/EnPT/Output/20230228__1' contains:
ENMAP01-____L1B-DT0000001867_20220724T104521Z_007_V010111_20230228T092319Z.log
No output raster was written.
Execution completed in 300.17 seconds (5 minutes 0 seconds)
Results:
{'success': False}
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/102Output writer raises warnings that Polymer results cannot be written although...2023-03-01T01:47:32+01:00Daniel SchefflerOutput writer raises warnings that Polymer results cannot be written although AC mode is 'land'These warnings do not make sense if Polymer is intentionally not running.These warnings do not make sense if Polymer is intentionally not running.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/101Lower case raster file extensions are incompatible with EnMAP-Box product imp...2023-03-01T01:13:28+01:00Daniel SchefflerLower case raster file extensions are incompatible with EnMAP-Box product import dialog.In case of `.bsq` file extensions:
![grafik](/uploads/22b1e9d9478add32c385565f273d4d06/grafik.png)In case of `.bsq` file extensions:
![grafik](/uploads/22b1e9d9478add32c385565f273d4d06/grafik.png)Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/100`run_deadpix_P=False` does not disable second run of dead pixel correction af...2023-03-01T00:00:29+01:00Daniel Scheffler`run_deadpix_P=False` does not disable second run of dead pixel correction after AC```python
[...]
2023/02/28 23:20:59: Joining results of atmospheric correction.
2023/02/28 23:21:01: Re-applying dead pixel correction to correct for spectral spikes due to fringe effect.
2023/02/28 23:21:01: Correcting dead pixels of VN...```python
[...]
2023/02/28 23:20:59: Joining results of atmospheric correction.
2023/02/28 23:21:01: Re-applying dead pixel correction to correct for spectral spikes due to fringe effect.
2023/02/28 23:21:01: Correcting dead pixels of VNIR detector...
Used algorithm: linear interpolation in the linear domain
2023/02/28 23:21:04: Percentage of defective pixels: 0.18
[...]
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/99Unexpected progress bar within co-registration.2023-02-28T23:06:19+01:00Daniel SchefflerUnexpected progress bar within co-registration.When resampling the spatial reference image during co-registration, a progress bar is shown although the `disable_progress_bars` parameter is set to True.When resampling the spatial reference image during co-registration, a progress bar is shown although the `disable_progress_bars` parameter is set to True.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/98Co-Registration computes way to many tie points in case of high resolution re...2023-02-17T16:17:21+01:00Daniel SchefflerCo-Registration computes way to many tie points in case of high resolution reference imageUsing a 3m resolution reference image, EnPT computes more than 80.000 tie points which takes hours to complete. However, it should not compute more than 5.000 tie points.Using a 3m resolution reference image, EnPT computes more than 80.000 tie points which takes hours to complete. However, it should not compute more than 5.000 tie points.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/92[SICOR AC] IndexError: index 0 is out of bounds for axis 0 with size 02022-12-09T15:29:14+01:00Daniel Scheffler[SICOR AC] IndexError: index 0 is out of bounds for axis 0 with size 0@nbohn It seems like we currently have an issue with the first guess WV retrieval implemented in SICOR. This was reported by Dagmar Müller, I guess with a recently acquired EnMAP scene. Might be related to the latest changes in the L1 pr...@nbohn It seems like we currently have an issue with the first guess WV retrieval implemented in SICOR. This was reported by Dagmar Müller, I guess with a recently acquired EnMAP scene. Might be related to the latest changes in the L1 processing of the ground segment which masked a few more bands. Probably, it is also related to #90.
```python
2022/12/08 15:25:15: Performing first guess water vapor retrieval based on a common band ratio using VNIR data...
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/sicor/Tools/EnMAP/first_guess.py", line 205, in _compute_wv
_globs["rhos"][idx_high[0][0]] - _globs["rhos"][idx_low[0][-1]])
IndexError: index 0 is out of bounds for axis 0 with size 0
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/dagmar/micromamba/envs/enpt_full2/bin/enpt", line 10, in <module>
sys.exit(main())
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/enpt/cli.py", line 225, in main
parsed_args.func(get_config(parsed_args))
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/enpt/cli.py", line 193, in run_job
CTR.run_all_processors()
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/enpt/execution/controller.py", line 184, in run_all_processors
self.run_atmospheric_correction()
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/enpt/execution/controller.py", line 152, in run_atmospheric_correction
self.L1_obj.run_AC()
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/enpt/model/images/images_sensorgeo.py", line 858, in run_AC
AC.run_ac(self)
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/enpt/processors/atmospheric_correction/atmospheric_correction.py", line 250, in run_ac
self._run_AC__land_mode(enmap_ImageL1)
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/enpt/processors/atmospheric_correction/atmospheric_correction.py", line 129, in _run_AC__land_mode
sicor_ac_enmap(enmap_l1b=enmap_ImageL1,
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/sicor/sicor_enmap.py", line 87, in sicor_ac_enmap
fo_enmap = Fo(enmap_l1b=enmap_l1b, options=options, logger=logger)
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/sicor/AC/RtFo_3_phases.py", line 753, in __init__
self.cwv_fg = wv_band_ratio(data=self.data_vnir, water_msk=self.water_mask_vnir, fn_table=self.fn_table,
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/site-packages/sicor/Tools/EnMAP/first_guess.py", line 177, in wv_band_ratio
results.get()
File "/home/dagmar/micromamba/envs/enpt_full2/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
IndexError: index 0 is out of bounds for axis 0 with size 0
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/91"NameError: name 'polymer_ac_enmap' is not defined" in case acwater is instal...2022-12-06T16:33:21+01:00Daniel Scheffler"NameError: name 'polymer_ac_enmap' is not defined" in case acwater is installed but polymer is missingIf ACWater is installed but polymer is missing, the expected behavior should be to raise a warning and continue with the atmospheric correction in the "land" mode. Instead, a NameError is raised:
```python
../enpt/execution/controller.p...If ACWater is installed but polymer is missing, the expected behavior should be to raise a warning and continue with the atmospheric correction in the "land" mode. Instead, a NameError is raised:
```python
../enpt/execution/controller.py:184: in run_all_processors
self.run_atmospheric_correction()
../enpt/execution/controller.py:152: in run_atmospheric_correction
self.L1_obj.run_AC()
../enpt/model/images/images_sensorgeo.py:857: in run_AC
AC = AtmosphericCorrector(config=self.cfg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <enpt.processors.atmospheric_correction.atmospheric_correction.AtmosphericCorrector object at 0x7fbf90744220>
config = {'general_opts': {'CPUs': 32,
'average_elevation': 0,
'create_logfile': True,
...rradiance/SUNp1fontenla__350-2500nm_@0.1nm_converted.txt',
'scale_factor_toa_ref': 10000}}}
def __init__(self, config: EnPTConfig = None):
"""Create an instance of AtmosphericCorrector."""
self.cfg = config
> self.is_polymer_installed = polymer_ac_enmap is not None
E NameError: name 'polymer_ac_enmap' is not defined
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/89ValueError due to capital letters in EnMAP Zipfile extension2022-11-15T16:19:43+01:00Daniel SchefflerValueError due to capital letters in EnMAP Zipfile extension```python
Traceback (most recent call last):
File "/home/expedition/miniconda3/envs/enpt_full/bin/enpt", line 10, in <module>
sys.exit(main())
File "/home/expedition/miniconda3/envs/enpt_full/lib/python3.10/site-packages/enpt/cli.py", li...```python
Traceback (most recent call last):
File "/home/expedition/miniconda3/envs/enpt_full/bin/enpt", line 10, in <module>
sys.exit(main())
File "/home/expedition/miniconda3/envs/enpt_full/lib/python3.10/site-packages/enpt/cli.py", line 225, in main
parsed_args.func(get_config(parsed_args))
File "/home/expedition/miniconda3/envs/enpt_full/lib/python3.10/site-packages/enpt/cli.py", line 193, in run_job
CTR.run_all_processors()
File "/home/expedition/miniconda3/envs/enpt_full/lib/python3.10/site-packages/enpt/execution/controller.py", line 176, in run_all_processors
self.read_L1B_data()
File "/home/expedition/miniconda3/envs/enpt_full/lib/python3.10/site-packages/enpt/execution/controller.py", line 114, in read_L1B_data
raise ValueError("The parameter 'path_enmap_image' must be a directory or the path to an existing zip "
ValueError: The parameter 'path_enmap_image' must be a directory or the path to an existing zip archive. Received /home/expedition/Documents/EnMAP_data/ENMAP01-____L1B-DT0000001567_20220709T105740Z_032_V010110_20221114T120244Z.ZIP.
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/84Orthorectification fails if L1B data are pseudo-projected/rotated2022-03-22T16:26:43+01:00Daniel SchefflerOrthorectification fails if L1B data are pseudo-projected/rotatedIf the sensor geometry input data contain a map info with a rotation, the orthorectification fails:
```python
2022/03/16 20:50:59: Starting orthorectification...
2022/03/16 20:50:59: Computed common target extent of orthorectified i...If the sensor geometry input data contain a map info with a rotation, the orthorectification fails:
```python
2022/03/16 20:50:59: Starting orthorectification...
2022/03/16 20:50:59: Computed common target extent of orthorectified image (xmin, ymin, xmax, ymax in EPSG 32632): (620070.0, 5226540.0, 655080.0, 5262900.0)
2022/03/16 21:10:23: Orthorectifying VNIR data using 'nearest' resampling algorithm...
2022/03/16 21:11:46: Orthorectifying SWIR data using 'nearest' resampling algorithm...
2022/03/16 21:26:54: Merging VNIR and SWIR data...
2022/03/16 21:26:55: Orthorectifying 'mask_landwater' attribute...
FAILED
tests/test_orthorectification.py:110 (Test_Orthorectifier_DLR.test_run_transformation)
self = <tests.test_orthorectification.Test_Orthorectifier_DLR testMethod=test_run_transformation>
def test_run_transformation(self):
OR = Orthorectifier(config=self.config)
> L2_obj = OR.run_transformation(self.L1_obj)
test_orthorectification.py:113:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../enpt/processors/orthorectification/orthorectification.py:142: in run_transformation
attr_ortho = GeoArray(*GT_2D.to_map_geometry(attr, **kw_trafo), nodata=attr.nodata)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <enpt.processors.spatial_transform.spatial_transform.Geometry_Transformer object at 0x7f88f58d52b0>
path_or_geoarray_sensorgeo = <geoarray.baseclasses.GeoArray object at 0x7f88f75ed670>
tgt_prj = 32632, tgt_extent = (620070.0, 5226540.0, 655080.0, 5262900.0)
tgt_res = None, tgt_coordgrid = (array([ 0, 30]), array([ 0, 30]))
area_definition = None
def to_map_geometry(self,
path_or_geoarray_sensorgeo: Union[str, GeoArray, np.ndarray],
tgt_prj: Union[str, int] = None,
tgt_extent: Tuple[float, float, float, float] = None,
tgt_res: Tuple[float, float] = None,
tgt_coordgrid: Tuple[Tuple, Tuple] = None,
area_definition: AreaDefinition = None):
data_sensorgeo = GeoArray(path_or_geoarray_sensorgeo)
if data_sensorgeo.is_map_geo:
> raise RuntimeError('The dataset to be transformed into map geometry already represents map geometry.')
E RuntimeError: The dataset to be transformed into map geometry already represents map geometry.
../enpt/processors/spatial_transform/spatial_transform.py:87: RuntimeError
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/81Coverage raises warnings when running in multiprocessing2022-03-10T18:51:10+01:00Daniel SchefflerCoverage raises warnings when running in multiprocessing```
coverage data file: cannot unpack non-iterable NoneType object
Coverage.py warning: Data file '/home/gfz-fe/scheffler/python/findr_homo/.coverage.geoms.1279091.080330' doesn't seem to be a coverage data file: Couldn't use data file '...```
coverage data file: cannot unpack non-iterable NoneType object
Coverage.py warning: Data file '/home/gfz-fe/scheffler/python/findr_homo/.coverage.geoms.1279091.080330' doesn't seem to be a coverage data file: Couldn't use data file '/home/gfz-fe/scheffler/python/findr_homo/.coverage.geoms.1279091.080330': no such table: coverage_schema
Coverage.py warning: Couldn't use data file '/home/gfz-fe/scheffler/python/findr_homo/.coverage.geoms.1274268.063192': database disk image is malformed
```
For example, in [this pipeline](https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/jobs/123288#L847).
Solution:
Properly close and join multiprocessing.Pool like this:
```python
with multiprocessing.Pool() as pool:
pool.map(func, args)
pool.close()
pool.join()
```
This has to be done in the EnPT code (https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/merge_requests/60/diffs?commit_id=6ffec51a18bc764e20a85b1ef087f4ccd6933c11) AND ALL EXTERNAL LIBRARIES where multiprocessing.Pool is called, i.e., in arosics, spechomo, sicor, sensormapgeo, ...
Another way is to make coverage exclude external packages by adding this to the .coveragerc:
```bash
[run]
omit = */site-packages/*,*/tests/*,*/.eggs/*
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/79FileNotFoundError when using the official EnMAP test data downloaded from enm...2021-06-18T17:24:09+02:00Daniel SchefflerFileNotFoundError when using the official EnMAP test data downloaded from enmap.org.When using the data downloaded from https://www.enmap.org/data_tools/testdata/, EnPT raises a FileNotFoundError.
```python
Traceback (most recent call last):
File "/home/gfz-fe/miniconda3/envs/enpt_full/bin/enpt", line 10, in <module>
s...When using the data downloaded from https://www.enmap.org/data_tools/testdata/, EnPT raises a FileNotFoundError.
```python
Traceback (most recent call last):
File "/home/gfz-fe/miniconda3/envs/enpt_full/bin/enpt", line 10, in <module>
sys.exit(main())
File "/home/gfz-fe/miniconda3/envs/enpt_full/lib/python3.9/site-packages/enpt/cli.py", line 222, in main
parsed_args.func(get_config(parsed_args))
File "/home/gfz-fe/miniconda3/envs/enpt_full/lib/python3.9/site-packages/enpt/cli.py", line 190, in run_job
CTR.run_all_processors()
File "/home/gfz-fe/miniconda3/envs/enpt_full/lib/python3.9/site-packages/enpt/execution/controller.py", line 166, in run_all_processors
self.read_L1B_data()
File "/home/gfz-fe/miniconda3/envs/enpt_full/lib/python3.9/site-packages/enpt/execution/controller.py", line 117, in read_L1B_data
self.L1_obj = RD.read_inputdata(root_dir_main=path_enmap_image, root_dir_ext=path_enmap_image_gapfill,
File "/home/gfz-fe/miniconda3/envs/enpt_full/lib/python3.9/site-packages/enpt/io/reader.py", line 82, in read_inputdata
self.validate_input(root_dir_main, root_dir_ext)
File "/home/gfz-fe/miniconda3/envs/enpt_full/lib/python3.9/site-packages/enpt/io/reader.py", line 114, in validate_input
self._validate_enmap_l1b_rootdir(root_dir_main)
File "/home/gfz-fe/miniconda3/envs/enpt_full/lib/python3.9/site-packages/enpt/io/reader.py", line 149, in _validate_enmap_l1b_rootdir
raise FileNotFoundError('The root directory of the EnMAP image %s misses a file with the pattern %s.'
FileNotFoundError: The root directory of the EnMAP image /tmp/tmpz7iw_5vg/image_main misses a file with the pattern *-HISTORY.XML.
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/78Spatial artifacts in EnMAP L2A after running EnPT in combined AC mode.2022-08-12T17:18:06+02:00Daniel SchefflerSpatial artifacts in EnMAP L2A after running EnPT in combined AC mode.When running EnPT in the 'combined' AC mode, we have a yellowish area as pointed out in https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/merge_requests/54#note_57492 in the L2A results of EnPT.
From my point of view, this can...When running EnPT in the 'combined' AC mode, we have a yellowish area as pointed out in https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/merge_requests/54#note_57492 in the L2A results of EnPT.
From my point of view, this cannot come from ACwater as the spectral information over land is coming from SICOR. Probably, there is something wrong with the configuration of the software test (see [here](https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/merge_requests/54/diffs#876b55ca4274f27b0138fc581dc1153b0ccb9b79_65_71)) which is a different one than we use for the 'land' mode. This could also explain why you don´t see this effect when only SICOR is running ('land' mode), @nbohn. I looked a bit deeper into this and found out that mainly the shorter wavelenghts are are affected by these anomalies. To me, it looks like it is somehow related to the segmentation in SICOR.
What do you think?
This is EnMAP L2A band 1 as it comes out of EnPT in case of the 'combined' AC mode. Spectra jump from positive values to slightly negative values at the edge of the artifact:
![image](/uploads/4da4edead0c41c568626665169674547/image.png)Niklas BohnNiklas Bohnhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/73Unexpected pixel value in the land/water mask passed to SICOR.2021-03-15T17:35:00+01:00Daniel SchefflerUnexpected pixel value in the land/water mask passed to SICOR.See https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/47#note_45547
There should be only 3 different pixel values in the land/water mask: 0=background, 1=land, 2=water. The pixel value 3 might be a resampling artifact.See https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/47#note_45547
There should be only 3 different pixel values in the land/water mask: 0=background, 1=land, 2=water. The pixel value 3 might be a resampling artifact.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/68Deadlock within the orthorectification when using multiprocessing and 'gauss'...2020-09-16T11:04:23+02:00Daniel SchefflerDeadlock within the orthorectification when using multiprocessing and 'gauss' resampling.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/67Packages installed from PyPI cannot depend on packages which are not also hos...2020-10-14T15:20:37+02:00Niklas BohnPackages installed from PyPI cannot depend on packages which are not also hosted on PyPI.When executing `pip install enpt` in a fresh conda environment following the installation instructions in the EnPT documentation, I get the following error:
![Screenshot_2020-08-28_at_11.34.45](/uploads/53762625516aeb5ff303a3d9ec74f44c/...When executing `pip install enpt` in a fresh conda environment following the installation instructions in the EnPT documentation, I get the following error:
![Screenshot_2020-08-28_at_11.34.45](/uploads/53762625516aeb5ff303a3d9ec74f44c/Screenshot_2020-08-28_at_11.34.45.png)
The problem seems to be that SICOR is not hosted on PyPI. Which solution would you recommend to solve this issue? I could either start to prepare SICOR for being suited for hosting on PyPI or you could insert some information in the EnPT documentation that SICOR needs to be installed via cloning the repository prior to installing EnPT.Daniel SchefflerNiklas BohnDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/66The number of CPU cores is not respected during geolayer computation.2020-08-12T21:32:50+02:00Daniel SchefflerThe number of CPU cores is not respected during geolayer computation.Tested with 8 cores but all available cores were used during geolayer computation.Tested with 8 cores but all available cores were used during geolayer computation.Daniel SchefflerDaniel Scheffler