SpatialIndexMediator.getFullSceneDataForDataset() now retries the query 10 times in case of a struct.error (relates to issue #103

).

Signed-off-by: Daniel Scheffler <danschef@gfz-potsdam.de>

Fixed copy/paste error in license headers. Added .h5 files to GitLFS. Fixed missing bracket. Updated HISTORY.rst and version info.

Signed-off-by: Daniel Scheffler <danschef@gfz-potsdam.de>

Signed-off-by: Daniel Scheffler <danschef@gfz-potsdam.de>

Signed-off-by: Daniel Scheffler <danschef@gfz-potsdam.de>

Signed-off-by: Daniel Scheffler <danschef@gfz-potsdam.de>

Daniel Scheffler authored Mar 20, 2018 and Mathias Peters committed Mar 20, 2018

Bugfix '.fuse_hidden' files.
Removed ASCII_writer (deprecated).

Daniel Scheffler authored Feb 09, 2018 and Mathias Peters committed Feb 09, 2018

Moved retry functionality of spatial query within L1B to SpatialIndexMediator.

Added config options 'spatial_index_server_host' and 'spatial_index_server_port'. Catched ConnectionRefusedError during connection to index server. Bugfix SpatialIndexMediatorServer.status. Added tempdir deletion to controller shutdown.

Daniel Scheffler authored Jan 17, 2018 and Mathias Peters committed Jan 17, 2018

Added code to check proper activation of GDAL. Removed hardcoded database host from tests. Bugfix for test_spectral_resampler.
Former-commit-id: 4fda9e47
Former-commit-id: 8547efe6

Former-commit-id: 07c096f0
Former-commit-id: edc62020

Former-commit-id: a73f14de
Former-commit-id: 56f3f29c

Former-commit-id: 2d13fb28
Former-commit-id: 8b6a7759

Former-commit-id: 3ad3db23
Former-commit-id: dfacdeb0

Former-commit-id: 550648f5
Former-commit-id: 6a125fe7

Revised environment and spatial_index_mediator modules. Added config parameter to skip coregistration.
Former-commit-id: 9d2cbe5c
Former-commit-id: abf5c744

Former-commit-id: c5515d7c
Former-commit-id: 2c2146d5

Refactored GEOPROCESSING, METADATA, Input_reader, Output_writer and SpatialIndexMediator. Deleted a lot of deprecated/unused code:
GEOPROCESSING:
- deleted many unused functions written by Robert Behling
L1A_P:
- deleted decompress()
- deleted Layerstacking()
L1B_P:
- deleted functions related with SIFT/Orfeo toolbox co-registration
L2A_P:
- deleted get_DESHIFTER_configs()
- deleted envifilehandling.py
- deleted SRF_DB_2_PKL.py (content moved to output_writer
gms_object:
- deleted calc_mask_nodataOLD()
METADATA:
- deleted some unused functions written by Robert Behling
- deleted sandbox module
Former-commit-id: 36ab28d1
Former-commit-id: 7d82a28a

Former-commit-id: 5107ea13
Former-commit-id: b3925713

Former-commit-id: cd487679
Former-commit-id: cc4b7724

Updated deprecated import statements. Updated deprecated link to controller file of SpatialIndexMediator.
updated __version__ and __versionalias__
Former-commit-id: c25ef78d
Former-commit-id: 2354d9d1

Former-commit-id: 10090c7d
Former-commit-id: ea75e756

updated imports of GeoArray; fixed some broken indentations in SpatialIndexMediator; updated environment variables within run script
- updated __version__
Former-commit-id: 5d0aef9a
Former-commit-id: ee42a0a5

Former-commit-id: 165205fd
Former-commit-id: c80ba350

Daniel Scheffler authored Dec 01, 2016 and Mathias Peters committed Dec 01, 2016

summary:
- revised config -> database must be queried once per machine
- all modules of the package are now directly importable
- process controller is now a Python class containing a couple of methods for running the job
all modules:
- revised import statements
- replaced functions 'CFG.get_job()' and 'CFG.get_usecase()' by property 'CFG.job' and 'CFG.usecase'
misc.helper_functions:
- moved MAP() to new module processing.multiproc
misc.SpatialIndexMediator.SpatialIndexMediatorServer:
- bugfix for returning 'port' instead of 'process_id'
added new package 'processing'
- added __init__
- added module 'multiproc'
- added new module 'process_controller' containing new class 'process_controller' with several methods for running the GMS job
__init__:
- revised
config:
- added set_config() replacing unified_config.set_config()
- added class GMS_configuration containing properties for 'job' and 'usecase' that replace unified_config.get_job() and unified_config.get_usecase()
- Job:
    - added attributes 'end_time' and 'computation_time'
    - revised class structure
- Usecase:
    - revised class structure
- removed deprecated code
copied old version of config to config_old (still used by deprecated process_controller_for_testing)
added run_gms.py for running GeoMultiSens process controller from console
- updated __version__

Daniel Scheffler authored Oct 13, 2016 and Mathias Peters committed Oct 13, 2016

revised logging practices of whole GMS package (fixes Permission errors during logging); added direct call of external DeShifter module
L0B_P:
- updated calls for logger closing
L1A_P.L1A_object:
- converted L1A_object.logger to property 'logger'
- added close_GMS_loggers()
- updated __getstate__ and __setstate__
- log_for_fullArr_or_firstTile(): added 'close logger'-calls
- MetaObj2ODict(): bugfix for not deleting MetaObj
- added to_GMS_file() on the basis of OUT_W.ASCII_writer() which is now deprecated
- delete_tempFiles(): added 'close logger'-calls
L1B_P:
- reordered imports
- updated logger getters and 'close logger'-statements
- L1B_object:
    - removed its own logger setup
    - correct_spatial_shifts(): added working version
L2A_P:
- DESHIFTER: updated logger getters and 'close logger'-statements
INP_R:
- pickle_SRF_DB(): updated logger getters and 'close logger'-statements
OUT_W:
- ASCII_writer(): now deprecated; added 'unclosed logfile' warnings
- Obj2ENVI: updated writer calls for GMS file
HLP_F:
- removed setup_logger()
- moved close_logger() to new module 'logging'
- moved GMS_logger() to new module 'logging'
- failed_GMS_object: updated logger getter
- find_nearest() is now imported from external package 'py_tools_ds'
- removed duplicate version of corner_coord_to_minmax()
logging:
- added new module logging, based on previous logging functions from HLP_F
MGRS_tile:
- added higher timeout for getting geometry from pgSQL database
SpatialIndexMediator:
- SpatialIndexMediatorServer:
    - status: revised return value
    - start(): revised warnings
    - stop(): revised return value
PC:
- job.logger is now an instance of GMS_logger
- added new version of L2A_map()
- added shutdown_loggers statement

Daniel Scheffler authored Oct 12, 2016 and Mathias Peters committed Oct 12, 2016

Bugfix for not properly closed logfiles; added environment checker for Spatial Index Mediator Server
 L1A_P:
 - L1A_Obj.__getstate__(): bugfix for not properly closed logfiles
L1B_P:
- get_reference_image_params_pgSQL(): bugfix for not properly closed logfile
L2A_P:
- correct_shifts(): bugfix for not properly closed logfile
ENV:
- check_dependencies(): added functionality to check current status of Spatial Index Mediator Server and to start the server if needed
- added _log_or_print()
HLP_F:
- added close_logger()
- added draft of class GMS_logger (not usable so far)
- subcall_with_output(): added keywords no_stdout and no_stderr
SpatialIndexMediator:
- added class SpatialIndexMediatorServer: a class for interacting with java server (status, start, stop, restart, is_running, port, ...)
CFG:
- added job.path_spatIdxSrv
PC:
- added calls of environment checkers
pgSQL_DB:
- added 'path_spatial_index_mediator_server' to config table

- GeoMultiSens now depends on CoReg_Sat and py_tools_ds!
GEOP:
- moved warp_ndarray to external package 'py_tools_ds'
- updated warp_ndarray calls
L1A_P:
- added property coreg_needed
- added cubic resampling for MGRS tiles that have to be reprojected to the next UTM zone
- to_MGRS_tiles(): added verbose mode
L1B_P:
- COREG and DESHIFTER are now imported from external package CoReg_Sat
- deleted deprecated draft of COREG_GMS
- added class Scene_getter(): a class used to find a proper geospatial reference scene for a given target scene
- added class ref_Scene
- L1B_object:
    - revised the whole class (it now generates its coreg_info and deshift_results on its own)
    - added property spatRef_available
    - added get_spatial_reference_scene()
    - added get_opt_bands4matching (based on an earlier version from COREG)
    - added coregister_spatially()
    - added correct_spatial_shifts(): not yet working
L1C_P:
- L1C_object: updated __init__() args of super class L1B_object
L2A_P:
- updated warp_ndarray calls
L2B_P:
- spectral homogenization is now only executed if target CWLs are different to source CWLs
INP_R:
- fixed an unclosed file within GMSfile2dict
OUT_W:
- fixed an unclosed file within Obj2ENVI
PC:
- added a new version of L1B_map_1()

Bugfix for "missing L2B mask files" during continued processing that earlier stopped in L2B processing; implemented proper MGRS tiling scheme according to project agreements
GEOP:
- updated some deprecated 3rd-party-library references
- added transform_any_prj(): a function fort transforming XY-coordinates from any source to any target projection
- warp_ndarray():
    - implemented extra option 'outExtent_within' allowing to output arrays with a larger geographical extent than the input image (needed for MGRS tiling scheme)
- improved some docstrings
- added prj_equal(): a function to quickly check if two projections are equal
- added snap_bounds_to_pixGrid(): a function to snap map bounds to a given pixel grid
- added clip_array_using_mapBounds(): a function for clipping arrays with a map info using a given bounding box
L0A_P:
- add_local_availability():
    - replaced a hard coded list of processing levels
    - bugfix for returning an invalid processing level if processing is already done
L1A_P:
- fill_from_disk(): bugfix for copying memory addresses  when fill_from_disk() is called in multiprocessing
- log_for_fullArr_or_firstTile(): bugfix for not handling empty arr_pos
- added get_subset_obj(), based on older code from helper_functions: a function to generate subsets of GMS objects based on bounding box that can contain image AND map coordinates whereas map coordinates can also have a different projection (needed for proper generation of MGRS tiles)
 - added to_MGRS_tiles(, based on older code from helper_functions: a function for cutting a GMS object into MGRS tiles
 L1B_P:
 - L1B_object.apply_deshift_results():
     - bugfix for not updating geoinformations of 'masks' attribute after having applied spatial shift corrections
META:
- added 'data ignore value' to metadata
IO:
- refactored module 'gms_io' to 'io'
ARR:
- added submodule 'GeoArray' to io:
    - added class 'GeoArray': a class for simplifying array access, regardless to their actual memory location (in memory or on disk)
    - added _clip_array_at_mapPos(): a function for clipping a geocoded array using a given bounding box in the same projection like the array itself
    - added  get_array_at_mapPos(): a function for clipping a geocoded array using a given bounding box that can have any projection
OUT_W:
- added 'data ignore value' to enviHdr_keyOrder
- added 'data ignore value' to mask_to_ENVI_Classification()
- added set_output_nodataVal(): a function for adding a data ignore value to an already written file
- Obj2ENVI: bugfix for not handling empty arr_pos
DB_T:
- delete_processing_results(), GMS_JOB.__delete_procdata(), GMS_JOB.delete_procdata_of_entire_job(), GMS_JOB.delete_procdata_of_failed_sceneIDs(), added functionality to delete a specific processing level from disk
HLP_F:
- revised find_nearest(): it now supports an automatic rounding algorithm that rounds a value to the nearest neighbour
- replaced a deprecated 3rd-party function reference
- get_arrSubsetBounds_from_shapelyPolyLonLat(): added flag 'ensure_valid_coords'
MRGS_tile:
- added submodule "mgrs_tile" for easily retrieving informations about MGRS tiles (bounding box, projection conversions, buffering, etc.)
SpatialIndexMediator:
- replaced wrong host address
PC:
- updated calls for MGRS tiling within L2C_map_1()

added handling of unexpected exceptions within all GMS mappers;  added creation of job success summary
L1A_P:
- removed deprecated attributes path_Outfile_L1A and path_Outfile_L1A_masks
- calc_cloud_mask(): bugfix - wrong arguments passed to log_for_fullArr_or_firstTile
L1B_P:
- revised get_reference_image_params()
     -> spatial query works now but it still does not handle the case that no reference scene could be found or reference scene has a different projection
L1C_P:
- removed deprecated import
INP_R:
- get_list_GMSfiles(): bugfix for ignoring the given target processing level if job.call_type=='webapp'
OUT_W:
- Obj2ENVI(): bugfix for not assigning the correct output path to attribute 'mask_1bit' in case job.exec_mode=='Python'
 DB_T:
 - moved SpatialIndexMediator to a separate module within misc
 - simplified datetime reference
 - GMS_JOB: added comment keyword in order to allow short descriptions and comments for the job
HLP_F:
- added trace_unhandled_exceptions(): a dummy decorator function for handling unexpected exceptions
- added class 'failed_GMS_object': a class to represent failed GMS objects (allows to get information from them - even after processing failed)
- added log_uncaught_exceptions(): a decorator function for logging unhandled exceptions within GMS mapper functions
- added get_job_summary(): a function that returns a detailed and a less detailed pandas dataframe summarizing job processing success
- cut_GMS_obj_into_MGRS_tiles(): added logging
- get_mask_classdefinition(): added mask_1bit class definition
SpatialIndexMediator (SIM):
- added new module SpatialIndexMediator for performing fast spatial queries targeting the 'scenes' table of postgreSQL database
PC:
- added decorators to all GMS mapper functions that enable handling of unexpected exceptions
- added new list "failed_objects" in order to collect information about all GMS objects where processing failed
- added job database entry updates regarding failed scenes and finish time
- added creation and logging of job summary
VC:
- added cloud classifier objects for Landsat-5/7/8 to version control