Fixed buggy handling of acq_datetime when recreating GMS_object from disk.
Fixed 'str' object has no attribute 'month'.

-> required to revise metadata storage for layer dependent metadata (they are now stored as dicts instead of lists)

Daniel Scheffler authored Feb 08, 2018 and Mathias Peters committed Feb 08, 2018

Added timeout to ECMWF download.
Bugfix process_controller.shutdown().
Added dependency timeout_decorator.
Commented tests out that fail due to missing ECMWF data.

Former-commit-id: 2bdabd85
Former-commit-id: 5c389f43

Daniel Scheffler authored Jan 29, 2018 and Mathias Peters committed Jan 29, 2018

Former-commit-id: 63a9f352
Former-commit-id: 32c982dc

Spectral homogenization via Linear Regression now working with proper handling of LayerBandsAssignments.
Former-commit-id: 7a78bdb3
Former-commit-id: 4888d8a7

Former-commit-id: ff5dad9c
Former-commit-id: 2da69a73

Added Sentinel-2B compatibility. Added Sentinel-2B test. Added Sentinel-2B test data. Replaced Sentinel-2A new style test data.
Former-commit-id: cfb10541
Former-commit-id: b4b6ea2e

Former-commit-id: 405fe519
Former-commit-id: 449af678

Renamed option conversion_type_optical to target_radunit_optical and conversion_type_thermal to target_radunit_thermal. Keys in provided JSON file are now optional. Bugfixes.
Former-commit-id: 981de81d
Former-commit-id: 68ff63b7

Former-commit-id: ea66f0bd
Former-commit-id: a2792c94

Former-commit-id: 64594529
Former-commit-id: 7df60bbb

Former-commit-id: 74cad5e3
Former-commit-id: 068e26cb

Fixed AssertionError 'exactly 4 image corners must be present within the dataset'. Updated version info.
Former-commit-id: e75b3615
Former-commit-id: e4bf232e

Refactored GEOPROCESSING, METADATA, Input_reader, Output_writer and SpatialIndexMediator. Deleted a lot of deprecated/unused code:
GEOPROCESSING:
- deleted many unused functions written by Robert Behling
L1A_P:
- deleted decompress()
- deleted Layerstacking()
L1B_P:
- deleted functions related with SIFT/Orfeo toolbox co-registration
L2A_P:
- deleted get_DESHIFTER_configs()
- deleted envifilehandling.py
- deleted SRF_DB_2_PKL.py (content moved to output_writer
gms_object:
- deleted calc_mask_nodataOLD()
METADATA:
- deleted some unused functions written by Robert Behling
- deleted sandbox module
Former-commit-id: 36ab28d1
Former-commit-id: 7d82a28a

Former-commit-id: b6a2f1b6
Former-commit-id: f867b21f

Former-commit-id: 2b5d577f
Former-commit-id: 7bec6542

Former-commit-id: 684ce544
Former-commit-id: 383682c2

Former-commit-id: cd487679
Former-commit-id: cc4b7724

Former-commit-id: f2eb8b47
Former-commit-id: 0f59e4d3

Fix for job creation issues in case of Landsat ETM+ SLC-ON; Fix for exception during reading of AC options file.
- removed installer of ecmwf-api-client since this is now done in SICOR directly
updated __version__ and __versioninfo__
Former-commit-id: f4702bf9
Former-commit-id: a7cbc08b

Former-commit-id: 7f9b01f8
Former-commit-id: 71fc1d2b

Updated deprecated import statements. Updated deprecated link to controller file of SpatialIndexMediator.
updated __version__ and __versionalias__
Former-commit-id: c25ef78d
Former-commit-id: 2354d9d1

Added first TestCase, updated run-file, moved METADATA, gms_object and dataset to new submodule 'model'
bin.run_gms:
- added run_from_filenames() dummy
geomultisens.model.METADATA:
- get_EarthSunDistance(): fixed unclosed File warning
tests.data:
- added LC80010702013141LGN01.tar.gz
- test_geomultisens:
    - added class BaseTestCases
    - added class Test_Landsat8_PreCollectionData
updated __version__
updated __versionalias__
Former-commit-id: 2b978035
Former-commit-id: e41b2f4a

Former-commit-id: 10090c7d
Former-commit-id: ea75e756

- added submodule 'model' including the module 'dataset'
- moved all GMS specific attributes and functions of GMS_object to new class /models/dataset/Dataset
updated __version__
Former-commit-id: b0333947
Former-commit-id: ed000ded

config:
- Job: now accepts keyword arguments for most of the available options
updated __version__
Former-commit-id: cd4c82d2
Former-commit-id: de56a803

- updated __version__
Former-commit-id: 2edcbc23
Former-commit-id: 8d45362d

 algorithms.L1A_PL1A_object:
 - archive_to_rasObj(): revised matching expression to match S2A files within archive
 algorithms.METADATA.METADATA:
 - Read_Sentinel2A_xmls(): revised matching expressions to find correct xml data
 io.Input_reader:
 - open_specific_file_within_archive(): added assertions, docstring and revised function style
- updated __version__
Former-commit-id: 0497218d
Former-commit-id: 7cdf1a80

algorithms.L1C_P.AtmCorr:
- options: bugfix for not excluding specific bands from AC output
- _join_results_to_inObjs():
    - bugfix for stacking data_ac bands in the wrong order
    - NaN values withon data_ac are now set to outFill value
- updated __version__
Former-commit-id: 577c72b9
Former-commit-id: 1460d9b2

algorithms.GEOPROCESSING:
- calc_VAA_using_fullSceneCornerLonLat():
    - implemented keyword 'orbit_params' as fallback; added docstring
    -added 90 degrees to VAA
- calc_VZA_array(): added some notes
- calc_RAA_array(): added docstring
algorithms.L1A_P.L1A_object():
- calc_mean_VAA(): updated calc_VAA_using_fullSceneCornerLonLat call
algorithms.L1C_P.L1C_object():
- VZA_arr.getter: added parameter 'meshwidth'
- revised VAA_arr.getter
algorithms.METADATA:
- get_orbit_params(): added some notes
- updated __version__
Former-commit-id: b8a055bf
Former-commit-id: d08a8a57

- from_tiles(): bugfix for not updating array-dependent attributes
- from_sensor_subsystems(): bugfix for not recreating loggers after subsystem merging
- removed deprecated MetaObj2ODict()
- get_subset_obj(): now operable from imBounds; fixed inconsistent attribute names
- added to_tiles()
algorithms.L2B_P.L2B_object:
- added log message
algorithms.METADATA:
- get_LayerbandsAssignment: bugfix
misc.helper_functions:
- cut_GMS_obj_into_blocks(): now calls GMS_object.to_tiles()
- revised cornerLonLat_to_postgreSQL_poly()
misc.mgrs_tile:
- EPSG: bugfix
processing.pipeline:
- L1A_map_1() and L2A_map(): changed call for object conversion into tiles
- removed /database/cloud_classifier from version controlling -> now external resource
- updated __version__
Former-commit-id: c9f38cb9
Former-commit-id: 9b805282

- added ac_options for Sentinel-2
- added cloud classier for Sentinel-2 (not included in repository)
- added SNR model for Sentinel-2
config:
- reactivated exception handler
- updated __version__
Former-commit-id: 5d596a79
Former-commit-id: 85650583

algorithms.gms_object.GMS_object:
- GMS_identifier: is only a getter now, not a singleton
- added property 'MetaObj' / 'meta_odict' -> self-synchronizing
- added LayerBandsAssignment.setter
- revised arr.setter
- revised mask_nodata.setter
- mask_clouds.getter: bugfix
- revised mask_clouds.setter
- added property 'mask_clouds_confidence'
- revised masks.setter
- revised dem.setter
- ac_options: number of CPUs are now passed (temporarily disabled)
- revised ac_errors.setter
- added LBA2bandnames
- attributes2dict: bugfix for not addin meta_odict
- revised from_tiles()
- added from_sensor_subsystems(): a function for merging multiple subsystems of the same sensor (needed for Sentinel-2 and ASTER)
- revised build_combined_masks_array()
- get_subset_obj(): multiple bugfixes
- to_GMS_file(): updated in the context of new property architecture
- delete_previous_proc_level_results: added functionality to delete subsystem products after subsystems have been merged
algorithms.gms_object.failed_GMS_object:
- removed logger creation
algorithms.L1A_P.L1A_object:
- import_metadata(): updated METADATA call
- calc_TOARadRefTemp: added warning when using Sentinel-2 Gains
- calc_corner_positions(): added UTC timezone to datetime object
algorithms.L1B_P.L1B_object:
- _get_reference_image_params_pgSQL(): replaced temp_logger by self.logger
- revised correct_spatial_shifts()
algorithms.L1C_P:
- L1C_object:
    - removed deprecated atm_corr()
- AtmCorr:
    - __init__: added warning
    - revised data property
    - _meta_get_aux_data(): changed lonlat_arr to float16
    - run_atmospheric_correction(): added 'dump' keyword
    - revised _join_results_to_inObjs()
algorithms.L2B_P.L2B_object:
- spectral_homogenization(): bugfix
algorithms.METADATA:
- METADATA:
    - __init__(): is now initialized by GMS_identifier; does not directly run metadata reader
    - added read_meta()
    - Read_Sentinel2A_xmls(): updated setting of Gains
    - refactored to_meta_odict() to to_odict() and revised the function
    - added from_odict()
    - revised filter_layerdependent_metadata()
    - added 'map_odictKeys_objAttrnames'
    - get_LayerBandsAssignment():
        - added functionality to return full LBA for Sentinel-2 and ASTER
        - now properly handles bands removed after L1C and L2B
io.Output_writer:
- mask_to_ENVI_Classification(): bugfix
misc.definition_dicts:
- get_GMS_sensorcode(): added codes for Sentinel-2 full and ASTER full
- get_outFillZeroSaturated(): added bool dtype
- is_dataset_provided_as_fullScene(): added ASTER full and Sentinel-2 full
misc.exception_handler:
- log_uncaught_exceptions(): bugfix for 'disable_exception_handler'
misc.helper_functions:
- cut_GMS_obj_into_blocks(): bugfix
misc.logging.GMS_logger:
- added scene ID to formatter
misc.mgrs_tile:
- replaced deprecated reference
misc.path_generator:
- get_baseN(): bugfix
- added get_path_ac_input_dump()
processing.multiproc:
- MAP: added functionality to disable multiprocessing
processing.pipeline:
- revised L2A_map(): new L2A calls
processing.process_controller:
- add_local_availability(): changed get_LayerBandsAssignment call
- get_DB_objects(): bugfix for copied memory link during GMS object init
- L1C_processing() raises NotImplementedError in tiled mode
- L2A_processing(): added grouping of subsystems
- updated __version__

algorithms.GEOPROCESSING:
- GEOPROCESSING: conversion_type_optical value renamed from 'Ref' to 'TOA_Ref' and 'BOA_Ref'
- added keyword 'meshwidth' to the following functions (allows much faster processing):
    - zoom_2Darray_to_shapeFullArr()
    - adjust_acquisArrProv_to_shapeFullArr()
    - get_lonlat_coord_array():revised calculation of meshgrid
    - calc_VZA_array()
    - calc_AcqTime_array()
    - calc_SZA_SAA_array()
- calc_RAA_array(): now receives a VAA_array instead of VAA_mean
algorithms.gms_object:
- added attributes 'fullSceneCornerPos' and 'fullSceneCornerPos'
- logger: added assertion
- added property 'log'
- added dem deleter
- revised property 'ac_options'
- added property 'ac_errors'
- added property 'subset'
- refactored attribute 'acquisition_date' to 'acq_datetime' containing a full datetime timestamp in UTC time zone
- to_GMS_file(): updated timestamp format
algorithms.L1A_P.L1A_object:
- get_MetaObj(): now also updates 'acq_datetime'
- refactored get_MetaObj() to import_metadata()
- included set_arr_desc_from_MetaObj() in import_metadata()
- calc_TOARadRefTemp(): updated in the context of "conversion_type_optical" value change to 'TOA_Ref'
- calc_cloud_mask(): temporarily excluded Sentinel-2 here; added code draft of S2A cloud mask calculation
- calc_corner_positions(): major revision -> now calculates trueDataCornerPos/-LonLat AND fullSceneCornerPos/-LonLat
- calc_center_AcqTime() now also updates 'acq_datetime'
algorithms.L1B_P.ref_Scene:
- moved _get_reference_image_params_pgSQL() and _sceneIDList_to_filt_overlap_scenes() to L1B_object
algorithms.L1C_P.L1C_object:   -> major revision
- added properties 'lonlat_arr', 'VZA_arr', 'VAA_arr', 'SZA_arr', 'SAA_arr', 'RAA_arr' based on get_lonlat_coord_array() and calc_acquisition_illumination_geometry()
- removed deprecated functions get_lonlat_coord_array() and calc_acquisition_illumination_geometry()
- revised delete_ac_input_arrays()
algorithms.L1C_P.AtmCorr:
- revised property 'logger'
- metadata: added some tests
- revised _meta_get_viewing_zenith(), _meta_get_viewing_azimuth(), _meta_get_relative_viewing_azimuth()
- added _meta_get_aux_data()
- added _get_dem()
- added dummy version of _get_srf()
- run_atmospheric_correction(): added docstring; some minor revisions
- _join_results_to_inObjs(): now working
algorithms.METADATA:
- added property AcqDateTime: returns a full datetime object with UTC timezone
- revised setters for AcqDate, AcqTime and AcqDateTime -> timezone now properly handled
- refactored 'Meta2ODict' to 'to_meta_odict'
- calc_center_acquisition_time(): now also sets AcqDateTime
- get_LayerBandsAssignment(): processing level is now properly handled (in the context of missing bands after atmospheric correction)
misc.database_tools:
- renamed keyword 'trueDataCornerLonLat' to 'tgt_corners_lonlat' in the following functions:
    - get_pgSQL_geospatial_query_cond()
    - get_overlapping_scenes_from_postgreSQLdb()
    - get_overlapping_MGRS_tiles()
    - get_overlapping_MGRS_tiles2()
misc.definitions_dicts:
- added is_dataset_provided_as_fullScene()
misc.exception_handler:
- log_uncaught_exceptions:  exception handling is now optional and can be turned off via config
misc.helper_functions:
- modified some docstrings
misc.logging.GMS_logger:
- added property 'captured_stream' (not yet working)
- added draft of StringIO handler (not yet working)
misc.logging.path-generator:
- get_path_srf_file: bugfix
processing.pipeline:
- L1A_map(): updated calls
- L1A_map_1(): updated calls
- revised L1C_map()
processing.process_controller:
- add_local_availability(): added comments and revised structure
config:
- Job:
    - added attribute 'disable_exception_handler'
    - added attribute 'scale_factor_BOARef'
    - added attribute 'scale_factor_errors_ac'
    - added assertion
pg_SQLdb:
- table config:
    - added fields 'scale_factor_BOARef' and 'scale_factor_errors_ac'
    - changed value of 'conversion_type_optical' to 'BOA_Ref'
- updated __version__

algorithms.GEOPROCESSING:
- revised imports
algorithms.gms_object:
- gms_object:
    - revised imports
    - added property 'dem': gms_object can now directly provide a corresponding SRTM DEM
    - arr: bandnames of property 'arr' are now in the form [B01, .., B8A,] and correspond to LayerBandsAssignment
    - added property 'ac_options': getter or options dictionary needed for atmospheric correction
    - from_disk(): added time zone to 'acquisition_date' datetime object
- added class failed_GMS_object (based on earlier version from helper functions)
algorithms.L1A_P.L1A_object:
- revised imports
- calc_TOARadRefTemp(): bugfix for wrong nodata value in out returned array in case of Sentinel-2
- update_spec_vals_according_to_dtype: bugfix for not updating L1A_object.arr.nodata
algorithms.L1B_P:
- revised imports
algorithms.L1C_P:
- L1C_object:
    - get_lonlat_coord_array(): changed handling of return values
    - calc_acquisition_illumination_geometry(): changed handling of return values
- AtmCorr:
    -  added attribute 'ac_input' containing input args/kwargs of atmospheric correction
    - data: now uses L1C_obj.arr.bandnames for lopping over bands
    - added property 'nodata'
    - added property 'tile_name'
    - added property 'band_spatial_sampling'
    - added property 'nodata'
    - added property 'nodata'
    - revised property 'metadata'
    - added _meta_get_spatial_samplings()
    - added _meta_get_solar_irradiance()
    - added _meta_get_viewing_zenith()
    - added _meta_get_viewing_azimuth()
    - added _meta_get_relative_viewing_azimuth()
    - revised run_atmospheric_correction()
    - revised join_results_to_inObjs()
algorithms.METADATA
- revised imports
- added 'ScaleFactor' to meta_odict
io.Input_reader:
- fixed some bad type hints
- SRF_reader(): moved path generator functionality to path_generator
- added open_specific_file_within_archive() (moved)
- added get_dem_by_extent(): new function for reading SRTM DEM data and warping to a given pixel grid
io.Output_writer:
- revised imports
- fixed some bad type hints
- added 'ScaleFactor' to enviHdr_keyOrder
misc.__init__:
- added __all__
misc.database_tools:
- fixed some bad type hints
- get_overlapping_scenes_from_postgreSQLdb(): bugfix for wrong indexing
misc.definition_dicts:
- new module, consisting of earlier functions from helper_functions
misc.exception_handler:
- new module, consisting of earlier functions from helper_functions
misc.helper_functions:
- moved trace_unhandled_exceptions(), log_uncaught_exceptions() to misc.exception_handler
- moved failed_GMS_object to gms_object
- moved get_job_summary to process_controller
- fixed some bad type hints
- moved get_GMS_sensorcode(), get_mask_classdefinition(), get_outFillZeroSaturated(), get_mask_colormap() to misc.definition_dicts
- moved open_specific_file_within_archive() to Input_reader
misc.path_generator:
- path_generator:
    - revised get_path_rawdata()
    - revised get_local_archive_path_baseN(): 'image_type' is not used anymore; removed deprecated warning
    - get_path_cloud_class_obj(): added cloud classificator files for Sentinel-2
    - added get_path_srf_file()
    - added get_path_snr_model()
    - added get_path_ac_options()
    - added get_path_ac_table()
processing.multiproc:
- revised MAP(): added new keyword 'flatten_ouput'
processing.pipeline:
- updated imports
- revised L1C_map(): input represents one OR multiple L1B_objects belonging to the same scene_ID (atmospheric correction has to be applied to ALL subsystems of a scene at once)
processing.process_controller:
- changed some map calls due to new keyword  'flatten_ouput' of processing.multiproc.MAP
- revised L1C_processing(): added grouping of L1B_objects by scene_ID
- revised create_job_summary(): bugfix for emtpy input list of get_job_summary()
- added get_job_summary(): moved from helper_functions
config.Job:
- added new attributes 'path_ac_options', 'path_ac_tables', 'path_SNR_models', 'path_dem_proc_srtm_90m', 'path_ECMWF_db'
pgSGL_db table 'config':
- added keys 'path_SNR_models', 'path_ac_options', 'path_dem_proc_srtm_90m', 'path_ECMWF_db',  'path_ac_tables'
Fileserver:
- added input datasets for atmospheric correction
- updated __version__

algorithms.__init__:
- added some imports
algorithms.gms_object:
 - removed deprecated code
algorithms.L1C_P:
- removed deprecated code
- added class 'AtmCorr': a wrapper class for atmospheric correction
misc.database_tools:
- fixed some broken type hints
misc.helper_functions
- fixed broken type hint
- replaced deprecated function call
processing.pipeline:
- fixed some broken type hints
- updated __version__

implemented "GMS_object" as parent object for all GMS objects; dissolved L0A_P to config, and META; deleted L0B_P
algorithms:
- added new module 'gms_object', based on parts of L1A_object
    - GMS_object now provides the complete basic functionality of all GMS objects in later processing levels
- L0A_P has been dissolved/moved to config and METADATA
- L0B_P removed (deprecated)
algorithms.L1A_P.L1A_object:
- revised __init__() (L1A_object now inherits from GMS_object)
- added import_rasterdata()
- moved functions providing basic GMS functionality to GMS_object
- refactored fill_from_disk() to from_disk()
algorithms.L1B_P - L2C-P:
- updated __init__() functions due to new object structure
io.Output_writer:
- Obj2ENVI: data_DB_updater now receives a dictionary including current state of all properties
misc.database_tools.GMS_JOB:
- added __repr__()
- bugfix for using CFG.job.conn_database instead of self.conn
- revised from_job_ID(): self.dataframe is now properly filled
misc.helper_functions:
- failed_GMS_object now inherits from GMS_object
- get_GMS_sensorcode() now does not need a logger anymore
processing.pipeline:
- refactored L0A_L1A_map to L1A_map and L0B_L1A_map_1 to L1A_map_1
- revised L1A_map and L1A_map_1 according to new structure of L1A_object
processing.process_controller:
- added attribute '_DB_job_record'
- added property 'DB_job_record'
- removed get_data_list() (deprecated)
- stop(): added docstring
- L1A_processing(): edited docstring and added logger output; changed reduce call (!)
- L1B_processing(): edited docstring and added logger output
- L1C_processing(): edited docstring and added logger output; changed reduce call (!)
- L2A_processing(): edited docstring and added logger output
- L2B_processing(): edited docstring and added logger output; changed reduce call (!)
- L2C_processing(): edited docstring and added logger output
config:
- Job: added attribute 'DB_job_record'
- Usecase:
    - added attributes '_job' and 'data_list'
    - moved get_usecase_coord_grid() from top level of module here
    - added get_entity_IDs_within_AOI() from L0A_P
    - added get_data_list_of_current_jobID() from L0A_P
- updated __version__