Libraries issueshttps://git.gfz-potsdam.de/groups/globaldynamicexposure/libraries/-/issues2024-01-16T12:13:59+01:00https://git.gfz-potsdam.de/globaldynamicexposure/libraries/database-lib/-/issues/24Determine if spatial metadata needs to be initialized and do so automatically2024-01-16T12:13:59+01:00Danijel SchorlemmerDetermine if spatial metadata needs to be initialized and do so automaticallyDanijel SchorlemmerDanijel Schorlemmerhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/exposure-lib/-/issues/43Change the filename of `database.py`2022-08-30T13:07:23+02:00Danijel SchorlemmerChange the filename of `database.py`The main Python file of the exposure-lib should be called `exposurelib.py`.The main Python file of the exposure-lib should be called `exposurelib.py`.Danijel SchorlemmerDanijel Schorlemmerhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/exposure-lib/-/issues/50Limit area size for import PostGIS exposure database to local SpatiaLite data...2023-02-03T02:36:56+01:00Tara Evaz ZadehLimit area size for import PostGIS exposure database to local SpatiaLite databaseUsing the function `Import_from_postgis` in `exposure-lib`, one can import exposure database from server in PostGIS format locally in SpatiaLite format. There will be a problem if the area for import is too large. We should limit the are...Using the function `Import_from_postgis` in `exposure-lib`, one can import exposure database from server in PostGIS format locally in SpatiaLite format. There will be a problem if the area for import is too large. We should limit the area size for which a user can download exposure database at once. This is not implemented now because we may need to import in large scales for now for investigation purposes.
For more information, please check https://git.gfz-potsdam.de/dynamicexposure/globaldynamicexposure/exposure-lib/-/merge_requests/47#note_128507Tara Evaz ZadehTara Evaz Zadehhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/exposure-lib/-/issues/62Include the standard string in the Taxonomy table2022-09-30T10:32:59+02:00Danijel SchorlemmerInclude the standard string in the Taxonomy tableCurrently, only the extended string is stored in the Taxonomy table. Although it is much more suitable for quick parsing and also contains the occupancy type for more efficient filtering, it would be good to have the standard string as u...Currently, only the extended string is stored in the Taxonomy table. Although it is much more suitable for quick parsing and also contains the occupancy type for more efficient filtering, it would be good to have the standard string as used in GEM available too. Given that taxonomies are only stored once globally, Storing two strings per taxonomy is not a problem. With the standard string, the databases with exposure models that we plan to deliver will be more easily usable for others.
\rfc @tara @laurens @shindeDanijel SchorlemmerDanijel Schorlemmerhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/rule-lib/-/issues/5Implement a connection to a rule database2023-04-13T11:41:12+02:00Laurens OostwegelImplement a connection to a rule databaseEventually, rules need to be dynamically loaded into a rule database and called when needed. Therefore, a rule database should include the necessary definitions of the Rule class, such as the function, inputs and outputs.Eventually, rules need to be dynamically loaded into a rule database and called when needed. Therefore, a rule database should include the necessary definitions of the Rule class, such as the function, inputs and outputs.Laurens OostwegelLaurens Oostwegelhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/rule-lib/-/issues/7Explore if .eval should be changed into __call__2023-08-18T13:51:02+02:00Laurens OostwegelExplore if .eval should be changed into __call__See https://git.gfz-potsdam.de/globaldynamicexposure/building-initializer/-/merge_requests/3#note_160154See https://git.gfz-potsdam.de/globaldynamicexposure/building-initializer/-/merge_requests/3#note_160154Laurens OostwegelLaurens Oostwegelhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/taxonomy-lib/-/issues/12Bug in function `get_number_of_stories()`2024-02-06T12:29:59+01:00Tara Evaz ZadehBug in function `get_number_of_stories()`The function `get_number_of_stories()` is supposed to output the average, minimum and the maximum of the height section of taxonomy strings.
but if the range of number of stories is 4-7, the average will be 5.5!
This needs to be fix...The function `get_number_of_stories()` is supposed to output the average, minimum and the maximum of the height section of taxonomy strings.
but if the range of number of stories is 4-7, the average will be 5.5!
This needs to be fixed. Doesn't it?
@ds @laurensDanijel SchorlemmerDanijel Schorlemmer2024-01-17https://git.gfz-potsdam.de/globaldynamicexposure/libraries/rule-lib/-/issues/9Make a RuleHandler subclass for rules with databases2023-09-20T12:58:41+02:00Laurens OostwegelMake a RuleHandler subclass for rules with databasesLaurens OostwegelLaurens Oostwegelhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/database-lib/-/issues/20Do we need helper functions in database-lib?2024-02-06T12:23:13+01:00Laurens OostwegelDo we need helper functions in database-lib?Something like a formatter for None values for example. These could be for example that if a value is `None`, it automatically formats it to a NULL, while if it is a string, we should put quotation marks
\rfc @tara @dsSomething like a formatter for None values for example. These could be for example that if a value is `None`, it automatically formats it to a NULL, while if it is a string, we should put quotation marks
\rfc @tara @dsDanijel SchorlemmerDanijel Schorlemmerhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/exposure-lib/-/issues/103Integrate additional building datasets2023-09-25T18:45:32+02:00Danijel SchorlemmerIntegrate additional building datasetsTo include the building datasets of Google and Microsoft, the exposure-lib needs to be changed. The major change is the extension of the `Entity` to include a field indicating the building source. The field `osm_id` should be renamed to ...To include the building datasets of Google and Microsoft, the exposure-lib needs to be changed. The major change is the extension of the `Entity` to include a field indicating the building source. The field `osm_id` should be renamed to `building_id` as it now may point to any building dataset as specified in the `building_source` field. Also, a new table containing the description of the building sources needs to be added.
\rfc @tara @laurens @gislarsDanijel SchorlemmerDanijel Schorlemmerhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/exposure-lib/-/issues/107Create table configuration that only creates a certain selection of tables2024-01-26T11:03:02+01:00Laurens OostwegelCreate table configuration that only creates a certain selection of tablesCurrently, the `create_tables` function creates all tables, but this may be confusing. Something like a typed dict can solve this, where we can specify a few use cases, where certain tables can be created. We should also be able to creat...Currently, the `create_tables` function creates all tables, but this may be confusing. Something like a typed dict can solve this, where we can specify a few use cases, where certain tables can be created. We should also be able to create custom configurations.
Something like this in a config.py:
```python
from typing import TypedDict
class TableConfig(TypedDict):
entity: bool = False
asset: bool = False
taxonomy: bool = False
assetreference: bool = False
entityreference: bool = False
building: bool = False
damage: bool = False
STANDARD_CONFIG = TableConfig(entity=True, asset=True, taxonomy=True)
BASELINE_AS_STANDARD_CONFIG = TableConfig(entity=True, asset=True, taxonomy=True)
LOSS_CALCULATION = TableConfig(entity=True, asset=True, taxonomy=True, damage=True)
...
```
In the `__init__.py` it should be:
```python
...
from .config.py import TableConfig, STANDARD_CONFIG, BASELINE_CONFIG, LOSS_CALCULATION
__all__ = [..., "TableConfig", "STANDARD_CONFIG", "BASELINE_CONFIG", "LOSS_CALCULATION"]
```
We should be able to call the `create_tables` function like:
```python
import exposurelib
exposure_db = exposurelib.SpatiaLiteExposure('my_exposure.db')
exposure_db.create_tables(exposurelib.BASELINE_AS_STANDARD_CONFIG)
```
\\fyi @ds @taraDanijel SchorlemmerDanijel Schorlemmerhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/exposure-lib/-/issues/111Possible optimization of code2023-12-20T16:01:38+01:00Laurens OostwegelPossible optimization of codeThe function `get_tiles_without_reference_entities` may do an unnecessary filtering on the EntityReference table in the query (used by Initializer):
```
sql_statement = f"""
SELECT A.quadkey, built_are...The function `get_tiles_without_reference_entities` may do an unnecessary filtering on the EntityReference table in the query (used by Initializer):
```
sql_statement = f"""
SELECT A.quadkey, built_area_size
FROM
(
SELECT quadkey, built_area_size
FROM {self.tile_view}
WHERE quadkey IN ({quadkeys_in_batch}) AND built_area_size IS NOT NULL
) AS A
LEFT JOIN
(
SELECT quadkey FROM EntityReference
WHERE quadkey IN ({quadkeys_in_batch})
) AS E
ON E.quadkey = A.quadkey
WHERE E.quadkey IS NULL
"""
```
This could be simplified to a simple `LEFT JOIN EntityReference AS E`, but it needs to be checked if this runs faster or at the same speed as before for bigger countries. There may also be other SQL queries that can be optimized.
\rfc @tara @laurens @gislarshttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/exposure-lib/-/issues/116[Discussion] - Should we retire the backwards compatibility to versions < 3.3...2024-02-15T13:50:11+01:00Danijel Schorlemmer[Discussion] - Should we retire the backwards compatibility to versions < 3.35 of SQLiteVersions of SQLite prior to 3.35 do not support the `RETURNING` statement which we often use to return a newly created ID during an `INSERT` statement. We have accommodated earlier versions in exposure-lib by inserting a dataset and then...Versions of SQLite prior to 3.35 do not support the `RETURNING` statement which we often use to return a newly created ID during an `INSERT` statement. We have accommodated earlier versions in exposure-lib by inserting a dataset and then retrieving the ID separately if the SQLite installation doesn’t support the `RETURNING` statement.
This works nicely when the inserted dataset can easily be identified using integer values. However, I see three reasons to retire this backwards compatibility:
1. Versions prior to 3.35 are really old and I doubt that many people still use them.
2. We can simplify the codes that rely on the `RETURNING` statement.
3. A new function in exposure-lib is inserting geographic points into a table and retrieving the ID would require identifying the dataset by the geometry. I am afraid that this may not work properly as floating-point values are used to define geometries. This can cause rounding problems that would cause problems in finding the respective dataset to retrieve the ID.
\rfc @tara @laurens @gislarsLars LingnerLars Lingnerhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/exposure-lib/-/issues/118[Bug] - The indentation of the statement in line 1615 is wrong2024-01-08T16:19:10+01:00Danijel Schorlemmer[Bug] - The indentation of the statement in line 1615 is wrongIn function `create_damage_view` the indentation in line 1615 (`sql_statement = f"""`) is wrong. The statement is called multiple times for no reason. Also, the SQL code within this statement is wrongly indented.
```
for damage_...In function `create_damage_view` the indentation in line 1615 (`sql_statement = f"""`) is wrong. The statement is called multiple times for no reason. Also, the SQL code within this statement is wrongly indented.
```
for damage_state_id, damage_state_level in damage_states:
damage_state_tables += f"""
INNER JOIN (
SELECT damage_probability, assessment_id
FROM Damage
WHERE Damage.damage_state_id = {damage_state_id}
) AS {damage_state_level}
ON {damage_state_level}.assessment_id = Assessment.id
"""
sql_statement = f"""
CREATE VIEW {name} AS
SELECT
Entity.id AS entityid,
Assessment.id AS assessmentid,
{geom_table}.geom AS geometry,
Entity.quadkey,
ST_Area(SumNum.geom, True)/1000000 AS area_in_sqkm,
SumNum.number_buildings,
MostProbable.damage_state_name AS most_probable_damage,
{damage_state_selection}
FROM Entity
{damage_ratio_query}
INNER JOIN Assessment ON Assessment.entity_id = Entity.id
{damage_state_tables}
INNER JOIN (
SELECT MAX(damage_probability), damage_state_id,
DamageState.damage_state_name, assessment_id
FROM Damage
INNER JOIN DamageState
ON Damage.damage_state_id = DamageState.id
GROUP BY assessment_id
) AS MostProbable ON MostProbable.assessment_id = Assessment.id
WHERE Assessment.assessment_source_id = {assessment_source_id}
"""
# Expand view creation sql_statement for tile entities because
# of aggregation of buildings and residuals over quad-keys
if entity_type == "tile":
sql_statement += " GROUP BY Entity.quadkey"
# Creating the view
self.connection.execute(sql_statement)
```
\fyi @laurensTara Evaz ZadehTara Evaz Zadehhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/taxonomy-lib/-/issues/22Update the HAZUS Taxonomy string import for the 24.01 release2024-01-15T11:32:24+01:00Danijel SchorlemmerUpdate the HAZUS Taxonomy string import for the 24.01 releaseBecause the 23.07 release only distinguishes between `RES`, `COM`, and `IND` occupancy types, the occupancy mapping from the HAZUS Taxonomy to the GEM Taxonomy is simplified. The correct mapping would be:
```
# Dictionary mapping the HA...Because the 23.07 release only distinguishes between `RES`, `COM`, and `IND` occupancy types, the occupancy mapping from the HAZUS Taxonomy to the GEM Taxonomy is simplified. The correct mapping would be:
```
# Dictionary mapping the HAZUS Taxonomy entries for occupancy to the ones of the GEM Taxonomy
HAZUS_TO_GEM_OCCUPANCY_MAP = {
"COM1": "COM1",
"COM2": "COM2",
"COM3": "COM3",
"COM4": "COM3",
"COM5": "COM3",
"COM6": "COM4",
"COM7": "COM4",
"COM8": "COM11",
"COM9": "COM5",
"COM10": "COM7",
"IND1": "IND1",
"IND2": "IND2",
"IND3": "IND3",
"IND4": "IND4",
"IND5": "IND5",
"IND6": "IND6",
"RES1": "RES1",
"RES2": "RES5",
"RES3A": "RES2A",
"RES3B": "RES2B",
"RES3C": "RES2C",
"RES3D": "RES2D",
"RES3E": "RES2E",
"RES3F": "RES2F",
"RES4": "RES3",
"RES5": "RES4",
"RES6": "RES4",
}
```
Also, in the `HAZUS_TO_GEM_BUILDING_TYPE_MAP`, we need to change the entry for mobile homes (`MH`) to:
```
"MH": {
"material": "W+WLI",
"llrs": "LWAL",
"height": "HBET:1-2",
"occupancy": "RES5",
"shape": "BPD",
},
```
\fyi @tara @laurensDanijel SchorlemmerDanijel Schorlemmerhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/exposure-lib/-/issues/121Add function `retrieve_buildings` that can retrieve buildings for any dataset2024-01-15T14:10:34+01:00Laurens OostwegelAdd function `retrieve_buildings` that can retrieve buildings for any dataset`retrieve_buildings` is now implemented in the finalizer for only OpenStreetMap buildings. It should be implemented in exposure-lib for any dataset`retrieve_buildings` is now implemented in the finalizer for only OpenStreetMap buildings. It should be implemented in exposure-lib for any datasetLaurens OostwegelLaurens Oostwegelhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/taxonomy-lib/-/issues/23Expand taxonomy-lib to cover the extended sections of GED4ALL2024-01-19T16:08:23+01:00Danijel SchorlemmerExpand taxonomy-lib to cover the extended sections of GED4ALLThe Japan model of the Global Exposure Model contains an attribute `FII` for fire insulation that is not covered in the 12 section of the general GEM Taxonomy. It is provided by the GED4ALL Taxonomy. Therefore, taxonomy-lib should be ext...The Japan model of the Global Exposure Model contains an attribute `FII` for fire insulation that is not covered in the 12 section of the general GEM Taxonomy. It is provided by the GED4ALL Taxonomy. Therefore, taxonomy-lib should be extended to cover the additional sections.
\rfc @tara @laurensDanijel SchorlemmerDanijel Schorlemmerhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/taxonomy-lib/-/issues/25Add the capability to process the occupancy type `NONRES` for New Zealand2024-01-19T17:09:11+01:00Danijel SchorlemmerAdd the capability to process the occupancy type `NONRES` for New ZealandThe exposure data for the New Zealand part of the Global Exposure Model contains only files for residential and non-residential occupancy. The tag for the non-residential occupancy is `NONRES` and is not in accordance with the GEM Taxono...The exposure data for the New Zealand part of the Global Exposure Model contains only files for residential and non-residential occupancy. The tag for the non-residential occupancy is `NONRES` and is not in accordance with the GEM Taxonomy. However, in order to be able to process the New Zealand model, we need to introduce this tag in the occupancy section.
\rfc @tara @laurensDanijel SchorlemmerDanijel Schorlemmerhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/exposure-lib/-/issues/130Introduce logging output in the `_import_exposure` function2024-02-14T16:52:26+01:00Danijel SchorlemmerIntroduce logging output in the `_import_exposure` functionIt is difficult in exposure-share to estimate the remaining run time when exporting large countries because no progress indicator exists. Adding the progress in Quadkeys being exported would help to make exposure-share more user-friendly...It is difficult in exposure-share to estimate the remaining run time when exporting large countries because no progress indicator exists. Adding the progress in Quadkeys being exported would help to make exposure-share more user-friendly.
\rfc @tara @laurens
\fyi @kasraDanijel SchorlemmerDanijel Schorlemmerhttps://git.gfz-potsdam.de/globaldynamicexposure/libraries/exposure-lib/-/issues/132[Bug] - Function `aggregate_exposure_entities` does not aggregate building as...2024-03-25T17:26:58+01:00Danijel Schorlemmer[Bug] - Function `aggregate_exposure_entities` does not aggregate building assets into tile assets if `min_number_buildings` is exceededThe function `aggregate_exposure_entities` only aggregates building assets into tile assets if the aggregation condition is met, which is a tile containing fewer buildings than the given `min_number_buildings` parameter. This causes all ...The function `aggregate_exposure_entities` only aggregates building assets into tile assets if the aggregation condition is met, which is a tile containing fewer buildings than the given `min_number_buildings` parameter. This causes all tiles with more than `min_number_buildings` to keep their building entities. This defeats the notion of aggregating the dataset.
\rfc @laurens @tara @kasraDanijel SchorlemmerDanijel Schorlemmer