Initializer runs for large countries create huge memory demand on the database server
At the end of the country processing, Initializer is searching for tiles with built area but without reference entities previously distributed over the districts. This happens when the known districts do not fully cover the countries built area or when the country boundary covers a larger area than the union of the districts (e.g. country boundary extending into the sea).
The query identifying these tiles first searches for all tiles with built area within the country and then left joins these tiles with the reference entity table to find the tiles without reference entities assigned. This left join seems expensive and is done on the entire EntityReference
table.
I can see two approached to lower the memory demand on the server side:
- Divide the country into level-12 (or similar) tiles and run the same query for these tiles separately to keep the size of the query results small.
- Introduce a subquery to first select all reference entities within the country of interest and then left join it with the query
A
(see lines 537-549 inexposureinitializer.py
)
The second option seems easier to implement but may still use too much memory for countries like China or Russia.