README.md 5.52 KB
Newer Older
Cecilia Nievas's avatar
Cecilia Nievas committed
1
2
# gde-importer

Cecilia Nievas's avatar
Cecilia Nievas committed
3
4
5
Importer of aggregated exposure models to serve as input for the Global Dynamic Exposure (GDE)
model.

6

7
8
## Installing gde-importer

9
10
11
12
13
14
15
16
17
18
19
20
### Software dependencies

- Python >= 3.7

### Python libraries

- `numpy`
- `pandas`
- `pyyaml`
- `openpyxl`

### Install
21

22
23
24
25
26
27
```bash
git clone https://git.gfz-potsdam.de/dynamicexposure/globaldynamicexposure/gde-importer.git
cd gde-importer
pip3 install -e .
```

28
## Preparation
29

30
### Create necessary databases
31

32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
1. If it does not already exist, create the GDE Tiles database as shown [here](https://git.gfz-potsdam.de/dynamicexposure/globaldynamicexposure/database-gdetiles).
2. If it does not already exist, create the OBM Tiles database as shown [here](https://git.gfz-potsdam.de/dynamicexposure/openbuildingmap/database-obmtiles) and populate the `obm_built_area_assessments` table (using, for example, [obmgapanalysis](https://git.gfz-potsdam.de/dynamicexposure/openbuildingmap/obmgapanalysis)).

### Obtain data for the European Seismic Risk Model 2020 (ESRM20)

Currently the `gde-importer` only supports the European Seismic Risk Model 2020 (ESRM20). The data for ESRM20 needs to be retrieved and placed in appropriate directories before running `gde-importer`.

1. Download main ESRM20 data. 
   You can do this in two alternative ways:
   1. In your shell: $ `wget --recursive --no-parent https://datasources.dynamicexposure.org/ESRM20/data/`
   2. Clone the ESRM20 repository to a local path of your choice following these 
[instructions](https://git.gfz-potsdam.de/dynamicexposure/datasources/-/tree/master/ESRM20).
2. Obtain ESRM20-compatible boundaries (geodata files of administrative divisions):
   $ `wget --http-user=USERNAME --http-password=PASSWORD --recursive --no-parent https://datasources.dynamicexposure.org/private/ESRM20_boundaries/data/`
   *(Unfortunately we are not allowed to redistribute the data and you'll need a password to access the sources. [Read here](https://git.gfz-potsdam.de/dynamicexposure/datasources/-/tree/master/ESRM20_boundaries) for more information about the data being used.)*
3. Place the downloaded data into paths and directories of your preference.

### Configuration
50

51

52
53
54
55
56
57
58
59
60
61
#### Quickstart:

Copy the file `config_example.yml` to your working directory as `config.yml` and provide the necessary parameters:
  - `exposure_format: esrm20`
  - `data_pathname: /path/to/downloaded/data/ESRM20`
  - `boundaries_pathname: /path/to/downloaded/ESRM20_boundaries`

#### Configuration file

The following configuration options are available in the `config.yml`:
62
63
64
65
66
67
68
69
70
71

- `model_name`: Name of the input aggregated exposure model (only relevant for the user).
- `exposure_format`: Format of the input aggregated exposure model. Currently supported values: esrm20.
- `data_pathname`: Path to directory that contains the model data.
- `boundaries_pathname`: Path to directory that contains the boundary geodata files.
- `occupancies_to_run`: List of occupancies for which the code will be run, separated by ", " (comma and space). They need to exist for the indicated `exposure format`. Currently supported values: residential, commercial.
- `exposure_entities_to_run`: List of names of exposure entities for which the code will be run. Currently supported options:
  - "all": The list of names will be retrieved from the metadata of the input aggregated exposure model.
  - A comma-space-separated list of entity names: This list of names will be used.
  - A full path to a .txt or .csv file:  The list of names will be retrieved from the indicated .txt/.csv file.
72
73
74
75
76
77
78
79
80
81
82
83
- `exposure_entities_code`: Either "ISO3" or a nested structure with exposure entities names and 3-character codes. For running ESRM20, "ISO3" is the preferred option.
- `number_cores`: Number of cores used for parallelisation.
- `database_built_up`: Credentials for the [database](https://git.gfz-potsdam.de/dynamicexposure/openbuildingmap/database-obmtiles#obm_built_area_assessments-completeness-assessments-information) where the built-up areas per quadtile are stored.
- `database_gde_tiles`: Credentials for the [database](https://git.gfz-potsdam.de/dynamicexposure/globaldynamicexposure/database-gdetiles) where information on the GDE tiles is stored.

## Running gde-importer

From the working directory (where you placed `config.yml`), run the code by typing:

```
gdeimporter
```
84

Cecilia Nievas's avatar
Cecilia Nievas committed
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
## Copyright and copyleft

Copyright (C) 2021

* Helmholtz-Zentrum Potsdam Deutsches GeoForschungsZentrum GFZ

This program is free software: you can redistribute it and/or modify it
under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or (at
your option) any later version.

This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero
General Public License for more details.

You should have received a copy of the GNU Affero General Public License
along with this program. If not, see http://www.gnu.org/licenses/.

Also add information on how to contact you by electronic and paper mail.

If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive
of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the
specific requirements.

See the [LICENSE](./LICENSE) for the full license text.