Skip to content

Provide a Windows and macOS compatible way to use multiprocessing in SICOR

Since the QGIS GUI for EnPT is now able to run the SICOR code from the QGIS GUI, we need to run SICOR on Windows and macOS.

This first arising in that context is the way how SICOR implements multiprocessing. This causes SICOR to crash under Windows and macOS:

(enpt) D:\Daten\Code\python\enpt_enmapboxapp\enpt_enmapboxapp>python C:\ProgramData\Anaconda3\envs\enpt\Scripts\enpt_cli.py --CPUs 12 --auto_download_ecmwf False --deadpix_P_algorithm spectral --deadpix_P_interp_spatial linear --deadpix_P_interp_spectral linear --disable_progress_bars False --enable_cloud_screening False --enable_keystone_correction False --enable_vnir_swir_coreg False --ortho_resampAlg bilinear --output_dir TEMPORARY_OUTPUT --path_dem D:\Daten\Code\python\EnPT\tests\data\DLR_L2A_DEM_UTM32.bsq --path_l1b_enmap_image D:\Daten\Code\python\EnPT\tests\data\EnMAP_Level_1B\ENMAP01-____L1B-DT000000987_20130205T105307Z_001_V000101_20190426T143700Z__rows0-99.zip --run_deadpix_P True --run_smile_P False --scale_factor_boa_ref 10000 --scale_factor_toa_ref 10000 
2020/01/10 21:13:26:   Reading metadata for VNIR detector...
2020/01/10 21:13:26:   Reading metadata for SWIR detector...
2020/01/10 21:13:26:   Converting DN values to radiance [mW/m^2/sr/nm] for VNIR detector...
2020/01/10 21:13:27:   Converting DN values to radiance [mW/m^2/sr/nm] for SWIR detector...
2020/01/10 21:13:27:   Computing SNR for VNIR using C:\Users\danschef\AppData\Local\Temp\tmptmeybnny\SNR_D1.bsq
2020/01/10 21:13:27:   Computing SNR for SWIR using C:\Users\danschef\AppData\Local\Temp\tmptmeybnny\SNR_D2.bsq
2020/01/10 21:13:28:   Computing VNIR geolayer...
2020/01/10 21:14:07:   Computing SWIR geolayer...
2020/01/10 21:14:48:   Correcting dead pixels of VNIR detector...
Used algorithm: spectral interpolation in the linear domain
2020/01/10 21:14:49:   Percentage of defective pixels: 0.49
2020/01/10 21:14:49:   Correcting dead pixels of SWIR detector...
Used algorithm: spectral interpolation in the linear domain
2020/01/10 21:14:49:   Percentage of defective pixels: 3.23
2020/01/10 21:14:50:   Pre-processing DEM for VNIR...
2020/01/10 21:14:50:   Transforming DEM to VNIR sensor geometry (using first band of VNIR geolayer)...
2020/01/10 21:14:52:   Pre-processing DEM for SWIR...
2020/01/10 21:14:52:   Transforming DEM to SWIR sensor geometry (using first band of SWIR geolayer)...
2020/01/10 21:14:55:   Starting atmospheric correction for VNIR and SWIR detector. Source radiometric unit code is 'TOARad'.
2020/01/10 21:14:55:   set up forward operator...
2020/01/10 21:14:55:   get observation metadata...
2020/01/10 21:14:58:   check if observation metadata values are within LUT value ranges...
2020/01/10 21:14:58:   get solar irradiances for absorption feature shoulders wavelengths...
2020/01/10 21:14:59:   load RT LUT...
2020/01/10 21:14:59:   resampling LUT to EnMAP wavelengths...
2020/01/10 21:15:00:   load imaginary parts of refractive indexes of liquid water and ice...
2020/01/10 21:15:00:   calculate mean solar exoatmospheric irradiances...
2020/01/10 21:15:00:   set up forward operator function...
2020/01/10 21:15:00:   estimate first guess for water vapor field...
2020/01/10 21:15:00:   start 3 phases of water retrieval...
2020/01/10 21:15:00:   set up multiprocessing...
2020/01/10 21:15:00:   multiprocessing is conducted on 12 cpu's
2020/01/10 21:15:00:   inversion method: 3 water phases optimal estimation (vapor, liquid and ice)
2020/01/10 21:15:00:   start optimization...
Processing(2): Traceback (most recent call last):
  File "D:/Daten/Code/python/enpt_enmapboxapp/enpt_enmapboxapp/enpt_enmapboxapp.py", line 452, in processAlgorithm
    raise Exception(err.decode('latin-1'))
Exception: 2020/01/10 21:13:26:   DLR test data provide multiple cloud masks. Added only *QUALITY_CLOUD.GEOTIFF!
2020/01/10 21:13:26:   DLR test data provide multiple cloud masks. Added only *QUALITY_CLOUD.GEOTIFF!
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\enpt\Scripts\enpt_cli.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "D:\Daten\Code\python\EnPT\bin\enpt_cli.py", line 170, in <module>
    parsed_args.func(get_config(parsed_args))
  File "D:\Daten\Code\python\EnPT\bin\enpt_cli.py", line 165, in run_job
    CTR.run_all_processors()
  File "d:\daten\code\python\enpt\enpt\execution\controller.py", line 149, in run_all_processors
    self.run_atmospheric_correction()
  File "d:\daten\code\python\enpt\enpt\execution\controller.py", line 126, in run_atmospheric_correction
    self.L1_obj.run_AC()
  File "d:\daten\code\python\enpt\enpt\model\images.py", line 869, in run_AC
    AC.run_ac(self)
  File "d:\daten\code\python\enpt\enpt\processors\atmospheric_correction\atmospheric_correction.py", line 94, in run_ac
    sicor_ac_enmap(enmap_l1b=enmap_ImageL1, options=options, logger=enmap_ImageL1.logger)
  File "d:\daten\code\python\sicor\sicor\sicor_enmap.py", line 129, in sicor_ac_enmap
    ["Retrieval"]["ice"])
  File "d:\daten\code\python\sicor\sicor\AC\RtFo_3_phases.py", line 835, in __minimize__
    initargs=(globals(), globs,))) as pl:
  File "C:\ProgramData\Anaconda3\envs\enpt\lib\multiprocessing\context.py", line 119, in Pool
    context=self.get_context())
  File "C:\ProgramData\Anaconda3\envs\enpt\lib\multiprocessing\pool.py", line 176, in __init__
    self._repopulate_pool()
  File "C:\ProgramData\Anaconda3\envs\enpt\lib\multiprocessing\pool.py", line 241, in _repopulate_pool
    w.start()
  File "C:\ProgramData\Anaconda3\envs\enpt\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\ProgramData\Anaconda3\envs\enpt\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\ProgramData\Anaconda3\envs\enpt\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\ProgramData\Anaconda3\envs\enpt\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle PyCapsule objects
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\envs\enpt\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\ProgramData\Anaconda3\envs\enpt\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

As a workaround, you could simply disable multiprocessing on Windows and macOS as long as this is not fixed. However, we have to provide a working version of EnPT soon - currently I can only turn off the AC completely.

Edited by Niklas Bohn