Fixed UnicodeEncodeError while writing ENVI headers. Moved scipy imports from module level to function level to avoid 'static TLS' ImportError. Fixed gdal_warp call within DEM_Creator class. Fixed issues while overwriting projections.

Signed-off-by: Daniel Scheffler <danschef@gfz-potsdam.de>

Fixed copy/paste error in license headers. Added .h5 files to GitLFS. Fixed missing bracket. Updated HISTORY.rst and version info.

Signed-off-by: Daniel Scheffler <danschef@gfz-potsdam.de>

Signed-off-by: Daniel Scheffler <danschef@gfz-potsdam.de>

Signed-off-by: Daniel Scheffler <danschef@gfz-potsdam.de>

Signed-off-by: Daniel Scheffler <danschef@gfz-potsdam.de>

Updated HISTORY.rst

Daniel Scheffler authored Mar 20, 2018 and Mathias Peters committed Mar 20, 2018

Bugfix '.fuse_hidden' files.
Removed ASCII_writer (deprecated).

Former-commit-id: ea66f0bd
Former-commit-id: a2792c94

Former-commit-id: 64594529
Former-commit-id: 7df60bbb

Former-commit-id: 4c2a7803
Former-commit-id: e599ca13

Refactored GEOPROCESSING, METADATA, Input_reader, Output_writer and SpatialIndexMediator. Deleted a lot of deprecated/unused code:
GEOPROCESSING:
- deleted many unused functions written by Robert Behling
L1A_P:
- deleted decompress()
- deleted Layerstacking()
L1B_P:
- deleted functions related with SIFT/Orfeo toolbox co-registration
L2A_P:
- deleted get_DESHIFTER_configs()
- deleted envifilehandling.py
- deleted SRF_DB_2_PKL.py (content moved to output_writer
gms_object:
- deleted calc_mask_nodataOLD()
METADATA:
- deleted some unused functions written by Robert Behling
- deleted sandbox module
Former-commit-id: 36ab28d1
Former-commit-id: 7d82a28a

Former-commit-id: b6a2f1b6
Former-commit-id: f867b21f

Former-commit-id: cd487679
Former-commit-id: cc4b7724

Former-commit-id: 1a5c6148
Former-commit-id: bd1801cf

Former-commit-id: f2eb8b47
Former-commit-id: 0f59e4d3

Former-commit-id: 10090c7d
Former-commit-id: ea75e756

algorithms.gms_object.GMS_object:
- GMS_identifier: is only a getter now, not a singleton
- added property 'MetaObj' / 'meta_odict' -> self-synchronizing
- added LayerBandsAssignment.setter
- revised arr.setter
- revised mask_nodata.setter
- mask_clouds.getter: bugfix
- revised mask_clouds.setter
- added property 'mask_clouds_confidence'
- revised masks.setter
- revised dem.setter
- ac_options: number of CPUs are now passed (temporarily disabled)
- revised ac_errors.setter
- added LBA2bandnames
- attributes2dict: bugfix for not addin meta_odict
- revised from_tiles()
- added from_sensor_subsystems(): a function for merging multiple subsystems of the same sensor (needed for Sentinel-2 and ASTER)
- revised build_combined_masks_array()
- get_subset_obj(): multiple bugfixes
- to_GMS_file(): updated in the context of new property architecture
- delete_previous_proc_level_results: added functionality to delete subsystem products after subsystems have been merged
algorithms.gms_object.failed_GMS_object:
- removed logger creation
algorithms.L1A_P.L1A_object:
- import_metadata(): updated METADATA call
- calc_TOARadRefTemp: added warning when using Sentinel-2 Gains
- calc_corner_positions(): added UTC timezone to datetime object
algorithms.L1B_P.L1B_object:
- _get_reference_image_params_pgSQL(): replaced temp_logger by self.logger
- revised correct_spatial_shifts()
algorithms.L1C_P:
- L1C_object:
    - removed deprecated atm_corr()
- AtmCorr:
    - __init__: added warning
    - revised data property
    - _meta_get_aux_data(): changed lonlat_arr to float16
    - run_atmospheric_correction(): added 'dump' keyword
    - revised _join_results_to_inObjs()
algorithms.L2B_P.L2B_object:
- spectral_homogenization(): bugfix
algorithms.METADATA:
- METADATA:
    - __init__(): is now initialized by GMS_identifier; does not directly run metadata reader
    - added read_meta()
    - Read_Sentinel2A_xmls(): updated setting of Gains
    - refactored to_meta_odict() to to_odict() and revised the function
    - added from_odict()
    - revised filter_layerdependent_metadata()
    - added 'map_odictKeys_objAttrnames'
    - get_LayerBandsAssignment():
        - added functionality to return full LBA for Sentinel-2 and ASTER
        - now properly handles bands removed after L1C and L2B
io.Output_writer:
- mask_to_ENVI_Classification(): bugfix
misc.definition_dicts:
- get_GMS_sensorcode(): added codes for Sentinel-2 full and ASTER full
- get_outFillZeroSaturated(): added bool dtype
- is_dataset_provided_as_fullScene(): added ASTER full and Sentinel-2 full
misc.exception_handler:
- log_uncaught_exceptions(): bugfix for 'disable_exception_handler'
misc.helper_functions:
- cut_GMS_obj_into_blocks(): bugfix
misc.logging.GMS_logger:
- added scene ID to formatter
misc.mgrs_tile:
- replaced deprecated reference
misc.path_generator:
- get_baseN(): bugfix
- added get_path_ac_input_dump()
processing.multiproc:
- MAP: added functionality to disable multiprocessing
processing.pipeline:
- revised L2A_map(): new L2A calls
processing.process_controller:
- add_local_availability(): changed get_LayerBandsAssignment call
- get_DB_objects(): bugfix for copied memory link during GMS object init
- L1C_processing() raises NotImplementedError in tiled mode
- L2A_processing(): added grouping of subsystems
- updated __version__

algorithms.GEOPROCESSING:
- GEOPROCESSING: conversion_type_optical value renamed from 'Ref' to 'TOA_Ref' and 'BOA_Ref'
- added keyword 'meshwidth' to the following functions (allows much faster processing):
    - zoom_2Darray_to_shapeFullArr()
    - adjust_acquisArrProv_to_shapeFullArr()
    - get_lonlat_coord_array():revised calculation of meshgrid
    - calc_VZA_array()
    - calc_AcqTime_array()
    - calc_SZA_SAA_array()
- calc_RAA_array(): now receives a VAA_array instead of VAA_mean
algorithms.gms_object:
- added attributes 'fullSceneCornerPos' and 'fullSceneCornerPos'
- logger: added assertion
- added property 'log'
- added dem deleter
- revised property 'ac_options'
- added property 'ac_errors'
- added property 'subset'
- refactored attribute 'acquisition_date' to 'acq_datetime' containing a full datetime timestamp in UTC time zone
- to_GMS_file(): updated timestamp format
algorithms.L1A_P.L1A_object:
- get_MetaObj(): now also updates 'acq_datetime'
- refactored get_MetaObj() to import_metadata()
- included set_arr_desc_from_MetaObj() in import_metadata()
- calc_TOARadRefTemp(): updated in the context of "conversion_type_optical" value change to 'TOA_Ref'
- calc_cloud_mask(): temporarily excluded Sentinel-2 here; added code draft of S2A cloud mask calculation
- calc_corner_positions(): major revision -> now calculates trueDataCornerPos/-LonLat AND fullSceneCornerPos/-LonLat
- calc_center_AcqTime() now also updates 'acq_datetime'
algorithms.L1B_P.ref_Scene:
- moved _get_reference_image_params_pgSQL() and _sceneIDList_to_filt_overlap_scenes() to L1B_object
algorithms.L1C_P.L1C_object:   -> major revision
- added properties 'lonlat_arr', 'VZA_arr', 'VAA_arr', 'SZA_arr', 'SAA_arr', 'RAA_arr' based on get_lonlat_coord_array() and calc_acquisition_illumination_geometry()
- removed deprecated functions get_lonlat_coord_array() and calc_acquisition_illumination_geometry()
- revised delete_ac_input_arrays()
algorithms.L1C_P.AtmCorr:
- revised property 'logger'
- metadata: added some tests
- revised _meta_get_viewing_zenith(), _meta_get_viewing_azimuth(), _meta_get_relative_viewing_azimuth()
- added _meta_get_aux_data()
- added _get_dem()
- added dummy version of _get_srf()
- run_atmospheric_correction(): added docstring; some minor revisions
- _join_results_to_inObjs(): now working
algorithms.METADATA:
- added property AcqDateTime: returns a full datetime object with UTC timezone
- revised setters for AcqDate, AcqTime and AcqDateTime -> timezone now properly handled
- refactored 'Meta2ODict' to 'to_meta_odict'
- calc_center_acquisition_time(): now also sets AcqDateTime
- get_LayerBandsAssignment(): processing level is now properly handled (in the context of missing bands after atmospheric correction)
misc.database_tools:
- renamed keyword 'trueDataCornerLonLat' to 'tgt_corners_lonlat' in the following functions:
    - get_pgSQL_geospatial_query_cond()
    - get_overlapping_scenes_from_postgreSQLdb()
    - get_overlapping_MGRS_tiles()
    - get_overlapping_MGRS_tiles2()
misc.definitions_dicts:
- added is_dataset_provided_as_fullScene()
misc.exception_handler:
- log_uncaught_exceptions:  exception handling is now optional and can be turned off via config
misc.helper_functions:
- modified some docstrings
misc.logging.GMS_logger:
- added property 'captured_stream' (not yet working)
- added draft of StringIO handler (not yet working)
misc.logging.path-generator:
- get_path_srf_file: bugfix
processing.pipeline:
- L1A_map(): updated calls
- L1A_map_1(): updated calls
- revised L1C_map()
processing.process_controller:
- add_local_availability(): added comments and revised structure
config:
- Job:
    - added attribute 'disable_exception_handler'
    - added attribute 'scale_factor_BOARef'
    - added attribute 'scale_factor_errors_ac'
    - added assertion
pg_SQLdb:
- table config:
    - added fields 'scale_factor_BOARef' and 'scale_factor_errors_ac'
    - changed value of 'conversion_type_optical' to 'BOA_Ref'
- updated __version__

algorithms.GEOPROCESSING:
- revised imports
algorithms.gms_object:
- gms_object:
    - revised imports
    - added property 'dem': gms_object can now directly provide a corresponding SRTM DEM
    - arr: bandnames of property 'arr' are now in the form [B01, .., B8A,] and correspond to LayerBandsAssignment
    - added property 'ac_options': getter or options dictionary needed for atmospheric correction
    - from_disk(): added time zone to 'acquisition_date' datetime object
- added class failed_GMS_object (based on earlier version from helper functions)
algorithms.L1A_P.L1A_object:
- revised imports
- calc_TOARadRefTemp(): bugfix for wrong nodata value in out returned array in case of Sentinel-2
- update_spec_vals_according_to_dtype: bugfix for not updating L1A_object.arr.nodata
algorithms.L1B_P:
- revised imports
algorithms.L1C_P:
- L1C_object:
    - get_lonlat_coord_array(): changed handling of return values
    - calc_acquisition_illumination_geometry(): changed handling of return values
- AtmCorr:
    -  added attribute 'ac_input' containing input args/kwargs of atmospheric correction
    - data: now uses L1C_obj.arr.bandnames for lopping over bands
    - added property 'nodata'
    - added property 'tile_name'
    - added property 'band_spatial_sampling'
    - added property 'nodata'
    - added property 'nodata'
    - revised property 'metadata'
    - added _meta_get_spatial_samplings()
    - added _meta_get_solar_irradiance()
    - added _meta_get_viewing_zenith()
    - added _meta_get_viewing_azimuth()
    - added _meta_get_relative_viewing_azimuth()
    - revised run_atmospheric_correction()
    - revised join_results_to_inObjs()
algorithms.METADATA
- revised imports
- added 'ScaleFactor' to meta_odict
io.Input_reader:
- fixed some bad type hints
- SRF_reader(): moved path generator functionality to path_generator
- added open_specific_file_within_archive() (moved)
- added get_dem_by_extent(): new function for reading SRTM DEM data and warping to a given pixel grid
io.Output_writer:
- revised imports
- fixed some bad type hints
- added 'ScaleFactor' to enviHdr_keyOrder
misc.__init__:
- added __all__
misc.database_tools:
- fixed some bad type hints
- get_overlapping_scenes_from_postgreSQLdb(): bugfix for wrong indexing
misc.definition_dicts:
- new module, consisting of earlier functions from helper_functions
misc.exception_handler:
- new module, consisting of earlier functions from helper_functions
misc.helper_functions:
- moved trace_unhandled_exceptions(), log_uncaught_exceptions() to misc.exception_handler
- moved failed_GMS_object to gms_object
- moved get_job_summary to process_controller
- fixed some bad type hints
- moved get_GMS_sensorcode(), get_mask_classdefinition(), get_outFillZeroSaturated(), get_mask_colormap() to misc.definition_dicts
- moved open_specific_file_within_archive() to Input_reader
misc.path_generator:
- path_generator:
    - revised get_path_rawdata()
    - revised get_local_archive_path_baseN(): 'image_type' is not used anymore; removed deprecated warning
    - get_path_cloud_class_obj(): added cloud classificator files for Sentinel-2
    - added get_path_srf_file()
    - added get_path_snr_model()
    - added get_path_ac_options()
    - added get_path_ac_table()
processing.multiproc:
- revised MAP(): added new keyword 'flatten_ouput'
processing.pipeline:
- updated imports
- revised L1C_map(): input represents one OR multiple L1B_objects belonging to the same scene_ID (atmospheric correction has to be applied to ALL subsystems of a scene at once)
processing.process_controller:
- changed some map calls due to new keyword  'flatten_ouput' of processing.multiproc.MAP
- revised L1C_processing(): added grouping of L1B_objects by scene_ID
- revised create_job_summary(): bugfix for emtpy input list of get_job_summary()
- added get_job_summary(): moved from helper_functions
config.Job:
- added new attributes 'path_ac_options', 'path_ac_tables', 'path_SNR_models', 'path_dem_proc_srtm_90m', 'path_ECMWF_db'
pgSGL_db table 'config':
- added keys 'path_SNR_models', 'path_ac_options', 'path_dem_proc_srtm_90m', 'path_ECMWF_db',  'path_ac_tables'
Fileserver:
- added input datasets for atmospheric correction
- updated __version__

major revision of previous commit; module gms_object now fully implemented; fixed differences in results between Flink and Python execution mode
algorithms.GEOPROCESSING.GEOPROCESSING:
- Layerstacking(): added possible bug hint; update_dataset_related_attributes() now always executed
- get_lonlat_coord_array(): adjusted to handle GeoArray instances
- calc_VZA_array(): adjusted to handle GeoArray instances
- calc_SZA_SAA_array(): adjusted to handle GeoArray instances
- calc_RAA_array(): adjusted to handle GeoArray instances
algorithms.gms_object:
- __init__(): added attribute 'MGRS_info'
- renamed attribute mask_1bit to 'mask_nodata'
- revised set_pathes()
- revised __getstate__()
- revised __deepcopy__()
- revised coreg_needed()
- added property 'resamp_needed'
- added property 'arr'
- added property 'mask_nodata'
- added property 'mask_clouds'
- added property 'masks'
- added property 'pathGen'
- revised attributes2dict()
- revised from_disk()
- revised calc_mask_nodata()
- revised build_combined_masks_array()
- combine_tiles_to_ObjAttr(): added DeprecationWarning
- get_subset_obj(): adjusted to properly handle GeoArray instances
- added to_ENVI(), based on Obj2ENVI from OUT_W
algorithms.L1A_P.L1A_object:
- import_rasterdata() now also sets shape_fullArr
- archive_to_rasObj(): bugfix for overwriting Layerstacking result
- revised get_MetaObj()
- removed deprecated get_shape_fullArr()
- calc_TOARadRefTemp(): revised to properly handle GeoArray instances
- reference_data(): revised to properly handle GeoArray instances
- calc_cloud_mask(): revised to properly handle GeoArray instances
- calc_corner_positions(): added assertion; now robust to different WKT string formats
- revised add_rasterInfo_to_MetaObj()
- update_spec_vals_according_to_dtype(): now does not set nodata value anymore (property)
algorithms.L1B_P:
- Scene_finder:
    - ensured Python 2.7 compatibility
- L1B_object:
    - correct_spatial_shifts(): major revision fixing some bugs
    - removed deprecated join_deshift_results()
    - removed deprecated apply_deshift_results()
algorithms.L1C_P.L1C_Object:
- get_lonlat_coord_array(): commented deprecated code out
- calc_acquisition_illumination_geometry(): commented deprecated code out
algorithms.L2A_P.L2A_Object:
- get_DESHIFTER_configs(): some simplifications
- removed deprecated class DESHIFTER (now fully imported from external library CoReg_Sat)
algorithms.L2B_P.L2B_Object:
- revised __init__()
io.Output_writer:
- moved Obj2ENVI to gms_object
misc.helper_functions:
- get_job_summary(): bugfix for counting inputs given as tiles of the same sceneid multiple times
misc.path_generator:
- added __getstate__() that automatically closes loggers
- added __setstate__()
processing.pipeline:
- adjusted output writer calls
- L1A_map(): removed calls to get_shape_fullArr() and calc_mask_nodata()
- L1A_map_1(): removed calls to get_shape_fullArr() and calc_mask_nodata()
- L2A_map(): added overwrite kwarg to calc_mask_nodata()
- removed deprecated L2A_map_1()
- removed deprecated L2A_map_2()
- renamed L2B_map_1 to L2B_map
- renamed L2C_map_1 to L2C_map
processing.process_controller.process_controller:
- added keyword 'db_host'
- simplified L2A_processing()
- create_job_summary(): bugfix for always passing L2C objects to get_job_summary() instead of highest requested processing level
config:
- set_config(): added keyword 'db_host'
- Job:
    - added keyword 'db_host'
    - added attribute 'allow_subMultiprocessing'
    - database connection string is now dynamically adjusted to database host
- updated __version__

implemented "GMS_object" as parent object for all GMS objects; dissolved L0A_P to config, and META; deleted L0B_P
algorithms:
- added new module 'gms_object', based on parts of L1A_object
    - GMS_object now provides the complete basic functionality of all GMS objects in later processing levels
- L0A_P has been dissolved/moved to config and METADATA
- L0B_P removed (deprecated)
algorithms.L1A_P.L1A_object:
- revised __init__() (L1A_object now inherits from GMS_object)
- added import_rasterdata()
- moved functions providing basic GMS functionality to GMS_object
- refactored fill_from_disk() to from_disk()
algorithms.L1B_P - L2C-P:
- updated __init__() functions due to new object structure
io.Output_writer:
- Obj2ENVI: data_DB_updater now receives a dictionary including current state of all properties
misc.database_tools.GMS_JOB:
- added __repr__()
- bugfix for using CFG.job.conn_database instead of self.conn
- revised from_job_ID(): self.dataframe is now properly filled
misc.helper_functions:
- failed_GMS_object now inherits from GMS_object
- get_GMS_sensorcode() now does not need a logger anymore
processing.pipeline:
- refactored L0A_L1A_map to L1A_map and L0B_L1A_map_1 to L1A_map_1
- revised L1A_map and L1A_map_1 according to new structure of L1A_object
processing.process_controller:
- added attribute '_DB_job_record'
- added property 'DB_job_record'
- removed get_data_list() (deprecated)
- stop(): added docstring
- L1A_processing(): edited docstring and added logger output; changed reduce call (!)
- L1B_processing(): edited docstring and added logger output
- L1C_processing(): edited docstring and added logger output; changed reduce call (!)
- L2A_processing(): edited docstring and added logger output
- L2B_processing(): edited docstring and added logger output; changed reduce call (!)
- L2C_processing(): edited docstring and added logger output
config:
- Job: added attribute 'DB_job_record'
- Usecase:
    - added attributes '_job' and 'data_list'
    - moved get_usecase_coord_grid() from top level of module here
    - added get_entity_IDs_within_AOI() from L0A_P
    - added get_data_list_of_current_jobID() from L0A_P
- updated __version__

Daniel Scheffler authored Dec 01, 2016 and Mathias Peters committed Dec 01, 2016

summary:
- revised config -> database must be queried once per machine
- all modules of the package are now directly importable
- process controller is now a Python class containing a couple of methods for running the job
all modules:
- revised import statements
- replaced functions 'CFG.get_job()' and 'CFG.get_usecase()' by property 'CFG.job' and 'CFG.usecase'
misc.helper_functions:
- moved MAP() to new module processing.multiproc
misc.SpatialIndexMediator.SpatialIndexMediatorServer:
- bugfix for returning 'port' instead of 'process_id'
added new package 'processing'
- added __init__
- added module 'multiproc'
- added new module 'process_controller' containing new class 'process_controller' with several methods for running the GMS job
__init__:
- revised
config:
- added set_config() replacing unified_config.set_config()
- added class GMS_configuration containing properties for 'job' and 'usecase' that replace unified_config.get_job() and unified_config.get_usecase()
- Job:
    - added attributes 'end_time' and 'computation_time'
    - revised class structure
- Usecase:
    - revised class structure
- removed deprecated code
copied old version of config to config_old (still used by deprecated process_controller_for_testing)
added run_gms.py for running GeoMultiSens process controller from console
- updated __version__

Daniel Scheffler authored Oct 14, 2016 and Mathias Peters committed Oct 14, 2016

Bugfix for wrong no data value in case of cloud classification masks that have no no-data values inside
OUT_W:
- mask_to_ENVI_Classification(): Bugfix for wrong no data value in case of cloud classification masks that have no no-data values inside
DB_T:
removed deprecated import statement
PC:
- deleted deprecated versions of process_controller modules

Daniel Scheffler authored Oct 14, 2016 and Mathias Peters committed Oct 14, 2016

Bugfix for unexpected pixel values in cloud masks (bug from last commit); bugfix for wrong colors in some cloud classification files
L1A_P:
- L1A_object.logger: bugfix for not creating new logger if there is a logger without handlers
L1B_P:
- correct_spatial_shifts(): bugfix for wrong resampling algorithm in case of masks attribute
OUT_W:
- mask_to_ENVI_Classification(): bugfix for returning array with wrong pixel values in case of missing no data value

Daniel Scheffler authored Oct 13, 2016 and Mathias Peters committed Oct 13, 2016

revised logging practices of whole GMS package (fixes Permission errors during logging); added direct call of external DeShifter module
L0B_P:
- updated calls for logger closing
L1A_P.L1A_object:
- converted L1A_object.logger to property 'logger'
- added close_GMS_loggers()
- updated __getstate__ and __setstate__
- log_for_fullArr_or_firstTile(): added 'close logger'-calls
- MetaObj2ODict(): bugfix for not deleting MetaObj
- added to_GMS_file() on the basis of OUT_W.ASCII_writer() which is now deprecated
- delete_tempFiles(): added 'close logger'-calls
L1B_P:
- reordered imports
- updated logger getters and 'close logger'-statements
- L1B_object:
    - removed its own logger setup
    - correct_spatial_shifts(): added working version
L2A_P:
- DESHIFTER: updated logger getters and 'close logger'-statements
INP_R:
- pickle_SRF_DB(): updated logger getters and 'close logger'-statements
OUT_W:
- ASCII_writer(): now deprecated; added 'unclosed logfile' warnings
- Obj2ENVI: updated writer calls for GMS file
HLP_F:
- removed setup_logger()
- moved close_logger() to new module 'logging'
- moved GMS_logger() to new module 'logging'
- failed_GMS_object: updated logger getter
- find_nearest() is now imported from external package 'py_tools_ds'
- removed duplicate version of corner_coord_to_minmax()
logging:
- added new module logging, based on previous logging functions from HLP_F
MGRS_tile:
- added higher timeout for getting geometry from pgSQL database
SpatialIndexMediator:
- SpatialIndexMediatorServer:
    - status: revised return value
    - start(): revised warnings
    - stop(): revised return value
PC:
- job.logger is now an instance of GMS_logger
- added new version of L2A_map()
- added shutdown_loggers statement

removed deprecated code in L1B_P; bugfix for wrong pixel size in reprojected MGRS tiles (fixed in 'py_tools_ds'-package)
L1B_P:
- refactored Scene_getter to Scene_finder
- moved COREG_OLD.get_reference_image_params_pgSQL() and COREG_OLD.sceneIDList_to_filt_overlap_scenes() to Scene_finder
- removed deprecated class COREG_OLD

- GeoMultiSens now depends on CoReg_Sat and py_tools_ds!
GEOP:
- moved warp_ndarray to external package 'py_tools_ds'
- updated warp_ndarray calls
L1A_P:
- added property coreg_needed
- added cubic resampling for MGRS tiles that have to be reprojected to the next UTM zone
- to_MGRS_tiles(): added verbose mode
L1B_P:
- COREG and DESHIFTER are now imported from external package CoReg_Sat
- deleted deprecated draft of COREG_GMS
- added class Scene_getter(): a class used to find a proper geospatial reference scene for a given target scene
- added class ref_Scene
- L1B_object:
    - revised the whole class (it now generates its coreg_info and deshift_results on its own)
    - added property spatRef_available
    - added get_spatial_reference_scene()
    - added get_opt_bands4matching (based on an earlier version from COREG)
    - added coregister_spatially()
    - added correct_spatial_shifts(): not yet working
L1C_P:
- L1C_object: updated __init__() args of super class L1B_object
L2A_P:
- updated warp_ndarray calls
L2B_P:
- spectral homogenization is now only executed if target CWLs are different to source CWLs
INP_R:
- fixed an unclosed file within GMSfile2dict
OUT_W:
- fixed an unclosed file within Obj2ENVI
PC:
- added a new version of L1B_map_1()

GeoArray:
- GeoArray: improved error handling
OUT_W:
- added 'file compression' key to enviHdr_keyOrder
- added write_ENVI_compressed()
- Obj2ENVI: added calls of write_ENVI_compressed
HLP_F:
- log_uncaught_exceptions: replaced a hard coded path reference
PC:
- L2C_map_1: enabled file compression for L2C MGRS tiles output

OUT_W:
- Obj2ENVI: Bugfix for corrupted ENVI headers within cloud classification image

Bugfix for "missing L2B mask files" during continued processing that earlier stopped in L2B processing; implemented proper MGRS tiling scheme according to project agreements
GEOP:
- updated some deprecated 3rd-party-library references
- added transform_any_prj(): a function fort transforming XY-coordinates from any source to any target projection
- warp_ndarray():
    - implemented extra option 'outExtent_within' allowing to output arrays with a larger geographical extent than the input image (needed for MGRS tiling scheme)
- improved some docstrings
- added prj_equal(): a function to quickly check if two projections are equal
- added snap_bounds_to_pixGrid(): a function to snap map bounds to a given pixel grid
- added clip_array_using_mapBounds(): a function for clipping arrays with a map info using a given bounding box
L0A_P:
- add_local_availability():
    - replaced a hard coded list of processing levels
    - bugfix for returning an invalid processing level if processing is already done
L1A_P:
- fill_from_disk(): bugfix for copying memory addresses  when fill_from_disk() is called in multiprocessing
- log_for_fullArr_or_firstTile(): bugfix for not handling empty arr_pos
- added get_subset_obj(), based on older code from helper_functions: a function to generate subsets of GMS objects based on bounding box that can contain image AND map coordinates whereas map coordinates can also have a different projection (needed for proper generation of MGRS tiles)
 - added to_MGRS_tiles(, based on older code from helper_functions: a function for cutting a GMS object into MGRS tiles
 L1B_P:
 - L1B_object.apply_deshift_results():
     - bugfix for not updating geoinformations of 'masks' attribute after having applied spatial shift corrections
META:
- added 'data ignore value' to metadata
IO:
- refactored module 'gms_io' to 'io'
ARR:
- added submodule 'GeoArray' to io:
    - added class 'GeoArray': a class for simplifying array access, regardless to their actual memory location (in memory or on disk)
    - added _clip_array_at_mapPos(): a function for clipping a geocoded array using a given bounding box in the same projection like the array itself
    - added  get_array_at_mapPos(): a function for clipping a geocoded array using a given bounding box that can have any projection
OUT_W:
- added 'data ignore value' to enviHdr_keyOrder
- added 'data ignore value' to mask_to_ENVI_Classification()
- added set_output_nodataVal(): a function for adding a data ignore value to an already written file
- Obj2ENVI: bugfix for not handling empty arr_pos
DB_T:
- delete_processing_results(), GMS_JOB.__delete_procdata(), GMS_JOB.delete_procdata_of_entire_job(), GMS_JOB.delete_procdata_of_failed_sceneIDs(), added functionality to delete a specific processing level from disk
HLP_F:
- revised find_nearest(): it now supports an automatic rounding algorithm that rounds a value to the nearest neighbour
- replaced a deprecated 3rd-party function reference
- get_arrSubsetBounds_from_shapelyPolyLonLat(): added flag 'ensure_valid_coords'
MRGS_tile:
- added submodule "mgrs_tile" for easily retrieving informations about MGRS tiles (bounding box, projection conversions, buffering, etc.)
SpatialIndexMediator:
- replaced wrong host address
PC:
- updated calls for MGRS tiling within L2C_map_1()

HLP_F:
- added MAP() dummy function
PC:
. added job 26185275

added handling of unexpected exceptions within all GMS mappers;  added creation of job success summary
L1A_P:
- removed deprecated attributes path_Outfile_L1A and path_Outfile_L1A_masks
- calc_cloud_mask(): bugfix - wrong arguments passed to log_for_fullArr_or_firstTile
L1B_P:
- revised get_reference_image_params()
     -> spatial query works now but it still does not handle the case that no reference scene could be found or reference scene has a different projection
L1C_P:
- removed deprecated import
INP_R:
- get_list_GMSfiles(): bugfix for ignoring the given target processing level if job.call_type=='webapp'
OUT_W:
- Obj2ENVI(): bugfix for not assigning the correct output path to attribute 'mask_1bit' in case job.exec_mode=='Python'
 DB_T:
 - moved SpatialIndexMediator to a separate module within misc
 - simplified datetime reference
 - GMS_JOB: added comment keyword in order to allow short descriptions and comments for the job
HLP_F:
- added trace_unhandled_exceptions(): a dummy decorator function for handling unexpected exceptions
- added class 'failed_GMS_object': a class to represent failed GMS objects (allows to get information from them - even after processing failed)
- added log_uncaught_exceptions(): a decorator function for logging unhandled exceptions within GMS mapper functions
- added get_job_summary(): a function that returns a detailed and a less detailed pandas dataframe summarizing job processing success
- cut_GMS_obj_into_MGRS_tiles(): added logging
- get_mask_classdefinition(): added mask_1bit class definition
SpatialIndexMediator (SIM):
- added new module SpatialIndexMediator for performing fast spatial queries targeting the 'scenes' table of postgreSQL database
PC:
- added decorators to all GMS mapper functions that enable handling of unexpected exceptions
- added new list "failed_objects" in order to collect information about all GMS objects where processing failed
- added job database entry updates regarding failed scenes and finish time
- added creation and logging of job summary
VC:
- added cloud classifier objects for Landsat-5/7/8 to version control

Landsat-7 processing now available in exec_mode='Python' and parallelization_level='scenes'; bugfixes in MGRS tile creation; database query for spatial reference not fully operable yet
GEOP:
- calc_FullDataset_corner_positions(): bugfix for returning wrong data corner positions for MGRS tiles
- transform_utm_to_wgs84(): bugfix for returning "wrong UTM zone" error in case of negative zone number (south)
- geotransform2mapinfo(): added NotImplementedError in case of rotated datasets
- mapinfo2geotransform(): bugfix for improper handling of UL coordinate if UL coordinate is not 1/1
- moved find_line_intersection_point() from calc_FullDataset_corner_positions() to top level
L1A_P:
- log_for_fullArr_or_firstTile(): added check of  MGRS_tile_obj.logAtThisTile in case of MGRS tile -> needed for proper logging for MGRS tiles if xmin=0/ymin=0 not exists
- delete_tempFiles() bugfix for running delete_previous_proc_level_results() for the wrong processing level
- added method __getstate__(): for defining custom pickling settings for logger
- added method __setstate__(): for defining custom unpickling settings for logger
- replaced all temp_logger-calls by self.logger calls
- log_for_fullArr_or_firstTile(self, log_msg, subset=None, logger=None): deleted deprecated logger argument
L1B_P:
- calculate_spatial_shifts(): updated a warning if calculated shifts are out of given threshold
- calculate_spatial_shifts(): bugfix for returning an updated map info although shift calculation did not pass validity test
- L1B_obj: replaced all temp_logger-calls by self.logger calls
- added a new version of get_reference_image_params() -> not fully operable yet
- renamed old version of get_reference_image_params() to get_reference_image_params_pgSQL()
- added editor fold to class GMS_COREG and commented out imports
- commented out deprecated imports and extensions of sys.path
L1C_P:
- L1C_obj: replaced all temp_logger-calls by self.logger calls
L2A_P:
- correct_shifts(): bugfix for not processing scenes if no reference scene was available
- correcr_shifts(): out_gsd is now properly set
- L2A_obj: replaced all temp_logger-calls by self.logger calls
 META:
 - added docstring to metaDict_to_metaODict()
 OUT_W:
- Obj2ENVI():
     - better documentation
     - replaced all temp_logger-calls by InObj.logger calls
- ASCII_writer(): proper exclusion of loggers from JSON serialization
 DB_T:
 - updated GMS_JOB.jobs_table_columns due to two deleted columns in 'jobs' table
 - updated data_DB_updater() due to changes in attribute namings og GMS object
 - added class SpatialIndexMediator(): for fast geospatial queries targeting 'scenes' table of postgreSQL database
 HLP_F:
 - get_subset_GMS_obj(): major revision
    - some renamed object attributes -> new attributes: data_corners_LonLat, data_corners_utm, fullScene_corner_lonlat, bounds_LonLat, bounds_utm, corner_utm, corner_lonlat
    - bugfix for wrong map info of the returned subsetted GMS objects
    - much better documentation
- cut_GMS_obj_into_MGRS_tiles():
    - better documentation
    - moved calculation of tile attributes to get_subset_GMS_obj()
    - added MGRS_tile_obj,logAtThisTile attribute for proper logging for MGRS tiles
- corner_coord_to_minmax():
    - added documentation
    - updated first timestamp of Landsat-7 SLC-off data
PG:
- added job 26185268  # 25x L7 SLC off

First version that is able to produce MGRS cubes in mass production (so far only fully operable for Landsat-8)
GEOP:
- merged calc_FullDataset_corner_positionsOLD() and calc_FullDataset_corner_positions() to calc_FullDataset_corner_positions()
	- the function now features two algorithms: 'numpy' (for Landsat-7 ETM+ SLC-off) and 'shapely' (for all remaining scenes)
- deleted calc_FullDataset_corner_positionsOLD()
- added is_point_on_grid(): Checks if a given point is exactly on the given coordinate grid. Needed to check if scene has to be coregistered or not.
- added is_coord_grid_equal(): Needed to check if scene has to be coregistered or not.
L0A_P:
- updated name of jobs table in postgreSQL database
L1A_P:
- added attribute dataset_ID -> Needed to check if scene has to be coregistered or not.
- added attributes scenes_proc_ID and mgrs_tiles_proc_ID -> needed for database entries
- archive_to_rasObj() now asserts that the read archive contains any files
- bugfix for Landsat-7 SLC-off scenes: archive_to_rasObj() now expects more files in the archive due to SLC off masks
- bugfix calc_mask_nodata(): data had been read from wrong source if job.exec_mode=='Python'
- revised calc_cloud_mask() -> the resulting array is now contains correct pixel values and is properly included in GMS_obj.masks
- bugfix for Landsat-7 SLC-off scenes: calc_corner_positions() now calls a different algorithm that can deal with SLC-off gaps
- bugfix calc_corner_positions(): incorrect getter for projection
- bugfix build_combined_masks_array(): it now out always outputs unsigned integer 8bit
- added delete_previous_proc_level_results(): needed to save disk space by deleting results of previous processing levels that are not needed anymore
- delete_tempFiles() now also deletes results of previous processing levels on demand (depending on job.exec_L1AP[2])
L1B_P:
- added class COREG_GMS (not yet working) -> will replace class COREG soon and will synchronize its changes with external library CoReg_Sat
- COREG.__init__():
	- added attributes dataset_ID and coreg_needed -> Needed to check if scene has to be coregistered or not.
	-> the class now only runs coregistration if its really needed. otherwise its skipped
- COREG.get_updated_map_info(): now no output stream anymore in quiet mode
- L1B_object:
	- slighty changed logging behavior
	- apply_deshift_results(): - bugfix for not properly updated map info and projection
	- apply_deshift_results(): - bugfix for not properly updated dimensions of the mask attributes
L1C_P:
- delete_tempfiles() now also deletes results of previous processing levels  on demand
L2A_P:
- bugfix in get_DESHIFTER_configs(): not properly closed gdal dataset
- added attributes shift_dataset_ID, x_shift_px, y_shift_px and deshift_needed in order to check if deshifting is really needed or not
- correct_shifts():
	- added proper logging
	- bugfix for missing updated_projection in case coregistration has been skipped
	- bugfix for not properly closed gdal dataset
	- bugfix for wrong resampling method for cloud mask (cubic results in wrong output pixel values) -> now nearest neighbour
	- added 'is clipped to extent' to deshift_results
L1C_P:
- bugfix spectral_homogenization(): now it also accepts string type array attributes if exec_mode=='Python'
- revised interpolate_cube_linear()
META:
- CS_DATUM, CS_UTM_ZONE, CS_TYPE are now properly written to gms-files
- added metaDict_to_metaODict() needed by output writer
INP_R:
- read_mask_subset(): added missing 'MGRS_tile' subset type
OUT_W:
- added silent_envi_write_image(): a monkey patch in order to silence spectral.io.envi._write_image
- added write_ordered_envi_header(): in order to write properly sorted ENVI header files
- revised Tiles_Writer(): previous version caused output errors for mask attributes of GMS objects
- revised mask_to_ENVI_Classification(): bugfixes for wrong conversions of numpy arrays to ENVI classdification ready array and metadata
- added check_header_not_empty()
- major revision of Obj2ENVI(): now it properly writes MRGS tiles if exec_mode=='Python'
- added database update for MGRS tiles (new records created in new database table 'mgrs_tiles_proc')
DB_T:
- splitted get_postgreSQL_matchingExp() to get_postgreSQL_value() and get_postgreSQL_matchingExp()
	-> much more python variable types can now be converted to postgreSQL types
- added create_record_in_postgreSQLdb(): needed to create new records for MGRS tiles
- added class GMS_JOB: a job manager for GMS jobs
	-> current features:
		- create a job in the postgreSQL database, based on a list of archive files
		- get the details about an existing job
- added add_missing_filenames_in_pgSQLdb(): useful for finding and fixing database errors
- import_shapefile_into_postgreSQL_database() updated name of postgreSQL database
- major revision of data_DB_updater(): now it properly updates both tables 'scenes_proc' and 'mgrs_scenes_proc'
HLP_F:
- added attribute proc_chain
- get_subset_GMS_obj(): bugfix for clipping the wrong tile dimension
- get_subset_GMS_obj(): added updated tile corners and data corners to MGRS tiles
- cut_GMS_obj_into_MGRS_tiles(): added calculation of tile bounds as well as data corner positions
- _numba_array_merger(): added a type assertion
- get_valid_arrSubsetBounds(): bugfix for returning wrong tile dimensions (1 row/column too big)
- sceneID_to_trueDataCornerLonLat(): added valid polygon assertion
- added class Landsat_entityID_decrypter: needed by class JOB_GMS for finding correct sensor parameters for Landsat data
PG:
- added get_path_rawdata()
- get_path_procdata(): changed returned path structure for MGRS tiles saved to disk
- get_baseN(): changed returned basename for GMS objects on disk
CFG:
- updated postgreSQL database connection credentials
- added a "delete output of previous porcessing levels"-flag to exec__L**P
- usecase: added datasetid_spatial_ref, virtual_sensor_name, target_gsd, EPSG, spatial_ref_gridx, spatial_ref_gridy
PC:
- added multiple job IDs containing pre-defined database jobs
- updated L1B_map_1(): shift calculation is now skipped if no reference scene is needed or available
- added L2A_map(): needed to execute processing in mass production, controlled by Python process controller
- revised tempfile deletion within L1C_map_1(), L2A_map_1(), L2B_map_1()
- updated L2A mapper calls -> exec_mode=='Python' is now supported
- added hardcoded thread count restriction to L2B mapper call due to memory errors in Python execution mode with parallelization level 'scenes'
pgSQLdb:
- renamed database from usgs_cache to geomultisens
- added table 'mrgs_tiles_proc'
- renamed table 'scenes_jobs' to 'jobs'
- updated jobs table with new columns
- fixed missing filenames in scenes table
- ...

- warp_ndarray(): fixed a bug in the validation of the output dimensions calculated by rasterio that always expected a quadratic input array for performing a correction of out_cols and out_rows
- added reproject_shapelyPoly()
- added mapXY2imXY()
- added imXY2mapXY()
- added proj4_to_dict()
L1A_P:
- fixed a bug in calc_cloud_mask()
- fixed a bug in calc_corner_positions()
L1B_P:
- fixed a bug in get_reference_image_params()
- fixed a bug in apply_deshift_results() when applying results to a 2D array
L2A_P:
- changed titlebar
L2B_P:
- changed titlebar
L2C_P:
- changed titlebar
- added a dummy version of L2C_object
L2D_P:  Level 2D processor has been deleted due to changes in the L2 processing architecture
META:
- added META.CS_EPSG attribute
OUT_W:
- Obj2ENVI(): added functionality to write MGRS tiles as compressed ENVI files (compression mode not yet working)
- dirty hack in mask_to_ENVI_Classification() -> still not working
- added get_dtypeStr()
- added new version of write_shp()
DB_T:
- moved some code from get_scene_and_dataset_infos_from_postgreSQLdb() to new function execute_pgSQL_query()
- added get_pgSQL_geospatial_query_cond()
- added get_overlapping_MGRS_tiles()
- added get_overlapping_MGRS_tiles2()
- added pdDataFrame_to_sql_k()
- added import_shapefile_into_postgreSQL_database()
- added postgreSQL_table_to_csv() (not yet working)
HLP_F:
- added docstring to setup_logger()
- added gzipfile()
- added ENVIfile_to_ENVIcompressed()
- added get_subset_GMS_obj()
- added cut_GMS_obj_into_MGRS_tiles()
- added an extra assertion to postgreSQL_poly_to_cornerLonLat()
- added some type hints
- added postgreSQL_geometry_to_shapelyPolygon()
- added shapelyPolygon_to_postgreSQL_geometry()
- added get_imageCoords_from_shapelyPoly()
- added get_valid_arrSubsetBounds()
- added get_arrSubsetBounds_from_shapelyPolyLonLat() -> needed to get image coordinates of MGRS tiles
- added get_UL_LR_from_shapefile_features()
PG:
- added functionality to get pathes for MGRS-tiled products
CFG:
- added additional output folder for MGRS-tiled products
PC:
- removed L2D_P import
- added cloud mask and corner coordinates calculation to L2A_map_2
- added L2C_map_1()
- adjusted L2B calls
- added first version of L2C calls

First prototype of algorithms for geometrical and spectral resolution working in map-reduce context (L2A, L2B).
GEOP:
    - renamed GEOPROCESSING_BD.py to GEOPROCESSING.py
    - moved get_prjLonLat(), get_proj4info(), corner_coord_to_minmax() to GEOP
    - added docstrings to DN2Rad(), DN2TOARef(), TOARad2Kelvin_fastforward(), DN2DegreesCelsius_fastforward()
L1A_P:
    - L1A_object.fill_arr_from_disk(): further silencing of console outputs
L1B_P:
    - moved get_DESHIFTER_configs() and class DESHIFTER() to L2A_P
    - adjusted initial values for COREG attributes related to reference image (not None anymore in order to make L2A_P work if shift calculation failed)
    - increased database statement timeouts for queries within get_reference_image_params() to 25sek
    - L1B_object():
        - added attribute "deshift_results"
        - removed deprecated code
        - added join_deshift_results()
        - revised apply_deshift_results()
L2A_P:
    -  added get_DESHIFTER_configs() and class DESHIFTER() from L1B_P
    - fixed two bugs in DESHIFTER.correct_shifts() where DESHIFTER.band2process was not respected and whole image cube was read instead of only one band
    - added class L2A_object()
L2B_P:
    - added class L2B_object()
    - L2B_object():
        - added interpolate_cube_linear()
        - added spectral_homogenization()
META:
    - renamed METADATA_BD.py to METADATA.py
INP_R:
    - added quiet mode to read_ENVIfile()
OUT_W:
    - added enviHdr_keyOrder using list from reorder_envi_header()
    - fixed a bug in reorder_ENVI_header() that caused repetitions of header keys
    - adjusted print_dict within mask_to_ENVI_Classification() in order to also support L2A and L2B
HLP_F:
    - added parent objects for L2A and L2B in parentObjDict
    - added type hints to cut_GMS_obj_into_blocks() and merge_GMS_tiles_to_GMS_obj()
GITIGNORE:
    - updated .gitignore file
CFG:
    - added virtual_sensor_id, datasetid_spectral_ref, target_CWL, target_FWHM to usecase class by querying the database
PC:
    - added type hints to mapper functions
    - revised L2A_map_2()
    - added L2B_map_1()
    - revised/added L2A algorithm calls (only Flink mode is supported so far)
    - added L2B algorithm calls (only Flink mode is supported so far)
pgDB:
    - added Sentinel-2A virtual sensors to virtual_sensors table (different spatial resolutions)
    - added wavelengths positions and band widths to virtual_sensors table