Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • gms_preprocessing gms_preprocessing
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 29
    • Issues 29
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 1
    • Merge requests 1
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Container Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • geomultisensgeomultisens
  • gms_preprocessinggms_preprocessing
  • Issues
  • #103
Closed
Open
Issue created Nov 25, 2020 by Daniel Scheffler@danschefOwner

'struct.error: unpack requires a buffer of 4 bytes' within SpatialIndexMediator in case of various parallel database accesses

The SpatialIndexMediator crashes with the exception below if many database queries are executed in parallel (here, GMS was running on 32 CPUs):

2020/11/25 22:33:34 [9290467]:   Generating DEM...
2020/11/25 22:33:34 [13512282]:   Generating DEM...
2020/11/25 22:33:34 [9290467]:   Acquired lock 'DatabaseLock', slot #1.
2020/11/25 22:33:34 [9290467]:   Released lock 'DatabaseLock', slot #1.
2020/11/25 22:33:34 [9290467]:   A static elevation is assumed during atmospheric correction due to an error during creation of the DEM corresponding to scene 9290467 (entity ID: LT50460322008283PAC01). Error message was: 
error('unpack requires a buffer of 4 bytes')

2020/11/25 22:33:34 [9290467]:   Print traceback in case you care:
2020/11/25 22:33:34 [9290467]:   Traceback (most recent call last):
  File "/home/gfz-fe/scheffler/python/gms_preprocessing/gms_preprocessing/algorithms/L1C_P.py", line 689, in _get_dem
    dem = inObj4dem.dem[:].astype(np.float32)
  File "/home/gfz-fe/scheffler/python/gms_preprocessing/gms_preprocessing/model/gms_object.py", line 551, in dem
    self._dem = DC.from_extent(*DC_args)
  File "/home/gfz-fe/scheffler/python/gms_preprocessing/gms_preprocessing/io/input_reader.py", line 476, in from_extent
    pathes = self._get_DEMPathes_to_include(tgt_corner_coord_lonlat)
  File "/home/gfz-fe/scheffler/python/gms_preprocessing/gms_preprocessing/io/input_reader.py", line 413, in _get_DEMPathes_to_include
    sceneIDs_srtm = self.get_overlapping_DEM_tiles(tgt_corner_coord_lonlat, timeout_sec)
  File "/home/gfz-fe/scheffler/python/gms_preprocessing/gms_preprocessing/io/input_reader.py", line 382, in get_overlapping_DEM_tiles
    scenes = SpIM.getFullSceneDataForDataset(envelope=corner_coord_to_minmax(tgt_corner_coord_lonlat),
  File "/home/gfz-fe/scheffler/python/gms_preprocessing/gms_preprocessing/misc/spatial_index_mediator.py", line 253, in getFullSceneDataForDataset
    numBytes = con.recvInt()
  File "/home/gfz-fe/scheffler/python/gms_preprocessing/gms_preprocessing/misc/spatial_index_mediator.py", line 364, in recvInt
    return struct.unpack('>i', self.socket.recv(4))[0]
struct.error: unpack requires a buffer of 4 bytes
Assignee
Assign to
Time tracking