Skip to content

Changed parallelisation strategy

Cecilia Nievas requested to merge feature/change_parallelisation into master

This MR changes the parallelisation strategy. Up to now, parallelisation occurred when reaching the calculations at the data-unit tile level. However, when put to run on the server, the database connection eventually crashed, likely due to an overload of connections-disconnections in very short intervals of time.

This new version of the code parallelises at the data unit level instead, which is the same approach followed in the gde-importer. Apart from the unit/integration tests, I have tested it locally on my laptop and my desktop, running with 3 and 6 cores in each case, and it ran within a reasonable amount of time (much faster than in previous attempts) and without crashing. The final test will be to run it on the server, once merged and updated in the corresponding docker container.

Merge request reports