EnPT issueshttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues2020-04-20T18:47:54+02:00https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/49Provide a method to transform L1B masks from VNIR to SWIR sensor geometry.2020-04-20T18:47:54+02:00Daniel SchefflerProvide a method to transform L1B masks from VNIR to SWIR sensor geometry.Since SICOR runs all retrievals on the SWIR and all L1B masks are provided in VNIR sensor geometry (see below), a method is needed to convert the masks into SWIR sensor geometry. This is needed to apply SICOR to land pixels only.
![imag...Since SICOR runs all retrievals on the SWIR and all L1B masks are provided in VNIR sensor geometry (see below), a method is needed to convert the masks into SWIR sensor geometry. This is needed to apply SICOR to land pixels only.
![image](/uploads/b798c97af3ae928a980555b5e014f91a/image.png)Add separate AC for water surfacesDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/47Use Polymer-based atmospheric correction for water surfaces.2021-06-15T13:13:59+02:00Daniel SchefflerUse Polymer-based atmospheric correction for water surfaces.AWI Bremerhaven is developing an atmospheric correction for water surfaces based on Polymer. This has to be implemented into EnPT. So far, SICOR is running for all pixels of the input image, regardless of whether they contain land or wat...AWI Bremerhaven is developing an atmospheric correction for water surfaces based on Polymer. This has to be implemented into EnPT. So far, SICOR is running for all pixels of the input image, regardless of whether they contain land or water surfaces.
Steps to be done:
* [x] implement water AC provided as an external Python package
* [x] check ability of SICOR / water-AC to apply the algorithm only to pixels selected by a binary mask
* [x] use water masks provided by the ground segment to decide which AC has to be applied where
* [x] implement an algorithm to fuse the results of SICOR and the water AC to a single reflectance product
* [x] provide config parameters to run EnPT in 3 modes: water+land, land-only, water-only
* [x] include the new config parameters into the EnPT GUI
* [x] decide which radiometric unit will be contained in the L2A product for water surfaces (water leaving radiance/reflectance, surface reflectance) -> ground segment L2A product contains surface reflectance but this is not a commonly used rad. unit in the water community
* [x] add documentation on water AC (and possibly a radiometric output unit different to the L2A product of the ground segment)Add separate AC for water surfacesDaniel SchefflerNiklas BohnBrenner SilvaDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/33Add correction of spatial misregistrations2020-09-21T13:41:29+02:00Daniel SchefflerAdd correction of spatial misregistrationsNeeded functionality:
* [x] add CLI parameters allowing to provide a spatial reference dataset (e.g., a Sentinel-2 band or a vegetation index)
* [x] implement AROSICS to compute spatial misregistration between EnMAP and the user provide...Needed functionality:
* [x] add CLI parameters allowing to provide a spatial reference dataset (e.g., a Sentinel-2 band or a vegetation index)
* [x] implement AROSICS to compute spatial misregistration between EnMAP and the user provided reference
* [x] add an algorithm for interpolating the spatial misregistrations in space
* [x] update the geolayer based on the computed pixelwise misregistration
We should only allow shifts up to a few pixels. We should recommend sensors such as Sentinel-2 as reference image. Detection of misregistrations should use a red or NIR band. Karl suggested 710 nm.Add geometry processorsDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/32Add keystone correction2024-02-09T17:20:06+01:00Daniel SchefflerAdd keystone correctionAdd geometry processorsDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/28[Orthorectification] Incorrect L2A output extent.2019-10-15T22:47:19+02:00Daniel Scheffler[Orthorectification] Incorrect L2A output extent.L2A output is currently cut off like this:
![image](/uploads/12a8a1e162ea3c94a2b62919e4f87766/image.png)L2A output is currently cut off like this:
![image](/uploads/12a8a1e162ea3c94a2b62919e4f87766/image.png)Add geometry processorsDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/27Add support for dead pixel correction based on 3D dead pixel map.2019-01-23T14:05:20+01:00Daniel SchefflerAdd support for dead pixel correction based on 3D dead pixel map.DLR provides the dead pixel maps as 3D arrays (rows x colmns x bands of each detector). The current version of DeadPixelCorrector is only compatible with 2D dead pixel maps (columns x bands).DLR provides the dead pixel maps as 3D arrays (rows x colmns x bands of each detector). The current version of DeadPixelCorrector is only compatible with 2D dead pixel maps (columns x bands).Provide a working L1B -> L2A processing chain for official DLR test data.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/26Add support for geolayer generation based on RPCs.2019-01-23T14:05:08+01:00Daniel SchefflerAdd support for geolayer generation based on RPCs.Needed functionality:
- generation of geolayers from RPC coefficientsNeeded functionality:
- generation of geolayers from RPC coefficientsProvide a working L1B -> L2A processing chain for official DLR test data.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/25Update L1B reader to match DLR L1B format.2019-01-23T14:04:08+01:00Daniel SchefflerUpdate L1B reader to match DLR L1B format.The current L1B reader is not yet compatible with the final L1B format as provided by DLR. We need to update it to make it read the DLR L1B test data.The current L1B reader is not yet compatible with the final L1B format as provided by DLR. We need to update it to make it read the DLR L1B test data.Provide a working L1B -> L2A processing chain for official DLR test data.Daniel SchefflerStephane Guillasostephane@gfz-potsdam.deDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/24Add output writer for EnMAP Level-2 data.2019-01-23T14:04:56+01:00Daniel SchefflerAdd output writer for EnMAP Level-2 data.L2 data specification needs to be discussed.
* Which image files will be included?
* DLR L2 test data as template for metadata?L2 data specification needs to be discussed.
* Which image files will be included?
* DLR L2 test data as template for metadata?Provide a working L1B -> L2A processing chain for official DLR test data.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/23Add metadata class for EnMAP Level-2 data.2019-01-23T14:04:30+01:00Daniel SchefflerAdd metadata class for EnMAP Level-2 data.Needed for generating Level-2 data with Orthorectifier module.Needed for generating Level-2 data with Orthorectifier module.Provide a working L1B -> L2A processing chain for official DLR test data.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/22Add support for 3D geometry layers.2019-01-23T13:45:27+01:00Daniel SchefflerAdd support for 3D geometry layers.So far, the transformation between sensor and map geometry uses 2D coordinate layers. The geolayers provided by DLR will have band-wise 3D coordinate layers.
Those are currently not supported but needed for
* spatial optimization (c...So far, the transformation between sensor and map geometry uses 2D coordinate layers. The geolayers provided by DLR will have band-wise 3D coordinate layers.
Those are currently not supported but needed for
* spatial optimization (co-registration and keystone correction)
* orthorectification.Add geometry processorsDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/20Add orthorectification module2020-03-17T18:29:35+01:00Daniel SchefflerAdd orthorectification module* [x] specify EnPT L2 output format
* metadata format according to official DLR L2A data
* [x] add image class for EnMAP L2 data in map geometry
* [x] add metadata class for EnMAP L2, map geometry metadata (DLR test data as templa...* [x] specify EnPT L2 output format
* metadata format according to official DLR L2A data
* [x] add image class for EnMAP L2 data in map geometry
* [x] add metadata class for EnMAP L2, map geometry metadata (DLR test data as template)
* [x] add output writer module for EnMAP L2, map geometry data
* [x] define a workflow for fusing the overlapping VNIR/SWIR bands
* gradually weighting of both sensors?
* [x] add Orthorectifier class for transformation of EnMAP L1 sensor geometry objects to L2 map geometry
(includes array transformation and metadata adjustment)Add geometry processorsDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/19Add DEM processor2018-12-13T23:36:31+01:00Daniel SchefflerAdd DEM processorWe need a processor for DEM data.
**Needed functionalities:**
* [x] input validation (geographic datum, overlap)
* [x] transformation from map to sensor geometry (comes in map geometry, AC need sensor geometry)
* [ ] computation of s...We need a processor for DEM data.
**Needed functionalities:**
* [x] input validation (geographic datum, overlap)
* [x] transformation from map to sensor geometry (comes in map geometry, AC need sensor geometry)
* [ ] computation of slopes?Add geometry processorsDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/18Add module for geometric transformation between sensor and map geometry.2018-12-13T23:35:49+01:00Daniel SchefflerAdd module for geometric transformation between sensor and map geometry.We need a module for transforming sensor geometry data to map geometry and vice versa.We need a module for transforming sensor geometry data to map geometry and vice versa.Add geometry processorsDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/12Correct for 20px shift between VNIR and SWIR before passing TOA radiance to S...2019-10-17T19:48:02+02:00Daniel SchefflerCorrect for 20px shift between VNIR and SWIR before passing TOA radiance to SICOR.VNIR and SWIR do not match geometrically but have an along-track shift of 20px and an across-track shift of ~1px depending on the geographic position. This has to be corrected before passing the TOA radiance to SICOR AC. Otherwise SICOR ...VNIR and SWIR do not match geometrically but have an along-track shift of 20px and an across-track shift of ~1px depending on the geographic position. This has to be corrected before passing the TOA radiance to SICOR AC. Otherwise SICOR will applies the retrieved atmospheric parameters (retrieved in SWIR) to the wrong VNIR pixels.
The shift is included in the RPC-coefficients, so we need to:
* [ ] determine the along-/across-track shift between VNIR and SWIR based on the RPC-coefficients
* [ ] crop the VNIR and SWIR data in sensor geometry so that the spatial shift is corrected (avoid resampling)
* [ ] respect along-/across-track shift in the geolayers used in downstream products
Add geometry processorsDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/137Replace legacy scipy.interpolation.interp1d.2024-03-05T15:38:57+01:00Daniel SchefflerReplace legacy scipy.interpolation.interp1d.`scipy.interpolation.interp1d` is considered as legacy code in the scipy docs and won't receive updates anymore. In https://github.com/EnMAP-Box/enmap-box/issues/804, importing it on a Mac seems to cause a QGIS crash. Thus, it should be ...`scipy.interpolation.interp1d` is considered as legacy code in the scipy docs and won't receive updates anymore. In https://github.com/EnMAP-Box/enmap-box/issues/804, importing it on a Mac seems to cause a QGIS crash. Thus, it should be replaced.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/135POLYMER SWIR output is not 0 but 12024-01-04T19:04:58+01:00Daniel SchefflerPOLYMER SWIR output is not 0 but 1POLYMER should set SWIR pixel values to 0 instead of 1. Probably, 1 was used so far, because EnPT used 0 as no-data value which changed in !98, so 0 is now a valid pixel value.
This is related to https://gitlab.awi.de/phytooptics/acwate...POLYMER should set SWIR pixel values to 0 instead of 1. Probably, 1 was used so far, because EnPT used 0 as no-data value which changed in !98, so 0 is now a valid pixel value.
This is related to https://gitlab.awi.de/phytooptics/acwater/-/issues/23.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/134Missing water-leaving reflectance data at coast line in case EnPT runs in com...2024-01-04T17:30:58+01:00Daniel SchefflerMissing water-leaving reflectance data at coast line in case EnPT runs in combined AC modeLooks like this:
![grafik](/uploads/993ad8fdf6db85f01690e35ef17c5d20/grafik.png)
Pixel values are set to no-data.Looks like this:
![grafik](/uploads/993ad8fdf6db85f01690e35ef17c5d20/grafik.png)
Pixel values are set to no-data.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/133"RuntimeWarning: invalid value encountered in cast" when joining POLYMER AC r...2024-01-03T20:09:27+01:00Daniel Scheffler"RuntimeWarning: invalid value encountered in cast" when joining POLYMER AC results```python
[...]
Starting processing at 2023-12-21 12:06:41.127358
Processing block: size (100, 1000), offset (0, 0)
Processing block: size (100, 1000), offset (100, 0)
Processing block: size (100, 1000), offset (200, 0)
Processing block:...```python
[...]
Starting processing at 2023-12-21 12:06:41.127358
Processing block: size (100, 1000), offset (0, 0)
Processing block: size (100, 1000), offset (100, 0)
Processing block: size (100, 1000), offset (200, 0)
Processing block: size (100, 1000), offset (300, 0)
Processing block: size (100, 1000), offset (400, 0)
Processing block: size (100, 1000), offset (500, 0)
Processing block: size (100, 1000), offset (600, 0)
Processing block: size (100, 1000), offset (700, 0)
Processing block: size (100, 1000), offset (800, 0)
Processing block: size (100, 1000), offset (900, 0)
Processing block: size (24, 1000), offset (1000, 0)
Done in 0:04:10.474827
2023/12/21 12:10:54: Joining results of atmospheric correction.
==> /home/gfz-fe/mambaforge/envs/enpt_full_dev/lib/python3.11/site-packages/enpt/processors/atmospheric_correction/atmospheric_correction.py:319: RuntimeWarning: invalid value encountered in cast
in_detector.data = (out_detector * self.cfg.scale_factor_boa_ref).astype(np.int16)
2023/12/21 12:10:55: Starting orthorectification...
[...]
```
The full log + input and output data was posted [here](https://github.com/GFZ/enpt/issues/7#issuecomment-1866122575).
This should be checked behind the background of some weird-looking water spectra, e.g. here:
![grafik](/uploads/a0d44b8a445af1b0be81ea9c6cbf385b/grafik.png)
The adjacent water pixels have more resonable signatures:
![grafik](/uploads/af7f042b1f1b2539f76c4669cef2de73/grafik.png)Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT/-/issues/132POLYMER compilation fails if Cython 3 is installed.2024-01-04T18:40:06+01:00Daniel SchefflerPOLYMER compilation fails if Cython 3 is installed.Polymer compilation fails when running `make` as documented [here](https://enmap.git-pages.gfz-potsdam.de/GFZ_Tools_EnMAP_BOX/EnPT/doc/installation.html#optional-install-polymer-for-advanced-atmospheric-correction-over-water-surfaces), s...Polymer compilation fails when running `make` as documented [here](https://enmap.git-pages.gfz-potsdam.de/GFZ_Tools_EnMAP_BOX/EnPT/doc/installation.html#optional-install-polymer-for-advanced-atmospheric-correction-over-water-surfaces), see https://forum.hygeos.com/viewtopic.php?f=7&t=221&start=10:
```python
polymer) [sc73004@ln32%bscc-a2 polymer-v4.16.1]$ make
python setup.py build_ext --inplace
Compiling polymer/clut.pyx because it changed.
Compiling polymer/neldermead.pyx because it changed.
Compiling polymer/polymer_main.pyx because it changed.
Compiling polymer/water.pyx because it changed.
[1/4] Cythonizing polymer/clut.pyx
/public3/home/sc73004/.conda/envs/polymer/lib/python3.11/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /public3/home/sc73004/software-sc73004/polymer-v4.16.1.tar/polymer-v4.16.1/polymer-v4.16.1/polymer/clut.pxd
tree = Parsing.p_module(s, pxd, full_module_name)
Error compiling Cython file:
------------------------------------------------------------
...
cdef CLUT A = CLUT(np.array([[1, 2, 3], [4, 5, 6]]), axes=[[1, 2], [10, 11, 12]])
assert A.lookup(0, 1.5) == 0
assert A.lookup(1, 11.5) == 0
print A.interp()
^
------------------------------------------------------------
polymer/clut.pyx:293:10: Syntax error in simple statement list
Traceback (most recent call last):
File "/public3/home/sc73004/software-sc73004/polymer-v4.16.1.tar/polymer-v4.16.1/polymer-v4.16.1/setup.py", line 26, in <module>
EXTENSIONS = cythonize([SRC_DIR + '/*.pyx'],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/public3/home/sc73004/.conda/envs/polymer/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
cythonize_one(*args)
File "/public3/home/sc73004/.conda/envs/polymer/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: polymer/clut.pyx
make: *** [main] Error 1
```Daniel SchefflerDaniel Scheffler