Skip to content

Introduced shared memory and parallel computation for external data

Karsten Prehn requested to merge feature/shared-memory into main

This assembles the loose pieces that so far have been introduced (e.g. parallel module) to now generate the geo-data for input quadkeys (e.g. land and water polygons) in parallel (object tile_polygons_gdf); or, as before, as a single call to the respective function (discretizer.tile_polygons_in_quadkey) when there is only one quadkey to calculate.

Additionally, external land, water and coast polygons are now kept in a shared memory block that is accessible by all (parallel) processes. This is meant to reduce per-process memory consumption.

For instance:

In tile_polygons_gdf, the water polygon parts of level 18 coastal tiles in a single level 15 tile (e.g. 101) look like so (as shown before):

10_water_per_tile_gdf

And when calculating above level 15 polygon (now in the upper right corner) and its three complements (e.g. 100, 101, 102, 103) in parallel, above information looks like so:

water_poly_parts

\approve @ds @marius

Merge request reports