Skip to content
Snippets Groups Projects
README.md 16.8 KiB
Newer Older
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
# EWRICA
Administrator's avatar
Administrator committed

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
The Early-Warning and Rapid Impact Assessment with real-time GNSS in the Mediterranean (EWRICA) is a federal Ministry of Education and Research funded project (funding period: 2020-2023) that aims to develop fast kinematic and point source inversion and modeling tools combining GNSS-based near field data with traditional broadband ground velocity and accelerometer data.

Fast and robust estimates of seismic source parameters are essential for reliable hazard estimates, e.g. in the frame of tsunami early warning. Hence, EWRICA aims for the development and testing of new real time seismic source inversion techniques based on local surface displacements. The resulting methods shall be applied for tsunami early warning purposes in the Mediterranean area. 


In this framework, this repository is a suite of four packages that can be used and combined in different ways and are [ewricacore](https://git.gfz-potsdam.de/ewrica/ewricacore), [ewricasiria](https://git.gfz-potsdam.de/ewrica/ewricasiria), [ewricagm](https://git.gfz-potsdam.de/ewrica/ewricagm) and [ewricawebapp](https://git.gfz-potsdam.de/ewrica/ewricawebapp). These four packages can be deployed in a docker container (see instructions below) to demonstrate a possible output of Early-Warning and Rapid Impact Assessment. 
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
The suite is composed of a probabilistic earthquake source inversion report (*ewricasiria*) and a Neural network-based Shake map (*ewricagm*) which are triggered upon event detection by *ewricacore*, a central unit of work that first fetches the waveform data via a configurable seedlink protocol and event data via event bus or FDSN web service, then collects and cuts waveforms segments according to a custom configuration, and eventually triggers custom processing (ewricasiria and ewricagm in the current workflow, but any processing can be implemented) whenever configurable conditions are met. 
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
The final package in this suite, *ewricawebapp*, is a web-based graphical user interface that has been used mainly for testing purposes but that can be deployed also to visualize and check in your browser all output produced by workflow in form of HTML pges, images and data in various formats (e.g., JSON, log text files).

The whole workflow has been tested for two past earthquakes whose data (as of 2023) are continuously served by [GEOFON](https://geofon.gfz-potsdam.de/) at regualr intervals to produce and test a real case scenario. These two test cases can also be reproduced locally following the instructions below.
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
Citation (software):
        
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
    Zaccarelli, Riccardo; Metz, Malte; Lehman, Lukas; Heinloo, Andres (2023): EWRICA - Docker container for Early-Warning and Rapid Impact Assessment with real-time GNSS data. GFZ Data Services. https://doi.org/10.5880/GFZ.2.6.2023.003
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
|Jump to: | [Docker installation and connection](#docker-installation-and-connection) |  [Docker services](#docker-services) |  [Docker source code](#docker-source-code) | [Inspect generated processing files in Docker](#inspect-generated-processing-files-in-docker) | [Webapp (GUI) in Docker](#webapp-gui-in-docker) | [Git workflow (for developers)](#git-workflow) |
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
|---|---|---|---|---|---|---|
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
**NOTE** For any problem/question [open an issue](https://git.gfz-potsdam.de/ewrica/readme/-/issues)
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed

## Docker installation and connection

### Installation

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
1. Make sure you have Docker installed, eg.:

```console
apt-get install docker.io
```

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
<details>
<summary>Get ewrica snapshot (outdated, click for details)</summary>

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
2. Get the EWRICA container snapshot from https://nextcloud.gfz-potsdam.de/s/ifJRJ3P9FoFdDj7

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
   Create the container from snapshot:
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
   ```console
   docker load -i ewrica_snapshot_20220707.tar.gz
   docker create --name ewrica --hostname ewrica ewrica_snapshot:20220707
   ```

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
   Start the container:
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
    ```console
    docker start ewrica
    ```
</details>
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
2. Download this repository and rename Dockerfile to `ewrica` (or copy it wherever you want). The latter is the dockerfile (or docker image) that you can use to start a container following the official documentation here: https://docs.docker.com/language/nodejs/run-containers/
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
3. Show the IP address of running container:
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed

```console
docker inspect -f "{{ .NetworkSettings.IPAddress }}" ewrica
Andres Heinloo's avatar
Andres Heinloo committed
172.17.0.2
Andres Heinloo's avatar
Andres Heinloo committed
If the container started successfully, you can open http://172.17.0.2/ewrica-realtime/ or http://172.17.0.2/ewrica-playback/ in your browser.
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
#### Further useful commands:

Enter the container (eg., run a shell in container):

```console
docker exec -ti ewrica bash
```

Show latest logs (see also /var/log in container):

```console
docker logs ewrica 2>&1 | tail
```

Start and stop the container:

```console
docker start ewrica
docker stop ewrica
```

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
### Connection / logout
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed

```Console
ssh root@ewrica2
```

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
```Console
logout
```
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
**NOTE on local deploys**: The docker container might be cut of the internet for any reason. This will result both in an unavailable Seedlink Connection `obspy.clients.seedlink.seedlinkexception.SeedLinkException: 'cannot connect to SeedLink server: [Errno 110] Connection timed out'` and/or not available remote git servers. Then logout of the docker, restart docker with

```console
service docker restart
```

or 

```console
systemctl restart docker
```
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
## Docker services
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
The services in the docker container are simple Unix services that are setup to run ewrica source code continuously in
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
the background. This means that any time source code is changed, the service must be restarted to put changes into effect:  
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
```terminal
    sv stop <service>
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
```
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
```terminal
    sv start <service>
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
Or, in one command:
```terminal
    sv restart <service>
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
where <service> is one of the following:

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
* `ewricacore` background service acquiring real time data and triggering custom process
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
* `ewricacore-playback` same as above, but listening for played back Norcia event continuously
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
* `ewricawebapp` web service showing `ewricacore` data and processed output (mainly used as browser GUI)
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
* `ewricawebapp-playback` same as above, but shows `ewricacore-playback` data and processed output
* `nginx` web service for all static content (images and so on) of both `ewricawebapp` and `ewricawebapp-playback`

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
**IMPORTANT: the commands above DO NOT STOP all Python processes attached to the service, this can cause memory leaks in the long run. Therefore, after stopping a service, please run `ps aux` and `kill -9 pid1 pid2 ...`, where each `pid` are all process id related to `/root/ewrica/pyenv3.9.5` (ewricacore\* services) or `/root/ewrica/pyenv3.9.1` (ewricawebapp\* services)**
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
### Edit a service

`ll /etc/service/` will list all services. Inside each directory, the service command is implemented in the `run` file. So to edit the service `ewricacore`  you can type:

```console
vim /etc/service/ewricacore/run
```

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
To edit any other service, just replace `ewricacore` with the service name, **with the exception of nginx**. In this case, edit the nginx config file:
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed

```console
vim /etc/nginx/sites-available/ewricawebapp
```

(note that in this case you might need to ask site maintainers for changes to take effect, e.g., if you want to change the site URL).
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
## Docker source code
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
The source code that the services run is located in the container inside `/root/ewrica`:
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed

```console
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
root@ewrica2:~/ewrica# ll /root/ewrica
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
total 60
drwxr-xr-x 13 root root 4096 Oct 24 14:16 ./
drwxr-xr-x  1 root root 4096 Oct 25 10:54 ../
drwxr-xr-x  8 root root 4096 Oct 24 14:32 ewricacore/
drwxr-xr-x  4 root root 4096 Oct 19 17:35 ewrica-data/
drwxr-xr-x  5 root root 4096 Oct 19 17:35 ewrica-data-playback/
drwxr-xr-x  5 root root 4096 Oct 24 14:20 ewricagm/
drwxr-xr-x  8 root root 4096 Oct 20 13:46 ewricasiria/
drwxr-xr-x  5 root root 4096 Oct 24 14:41 ewricawebapp/
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
drwxr-xr-x 11 root root 4096 Jul  4 14:30 grond/
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
drwxr-xr-x  6 root root 4096 Oct 17 14:58 .old_stuff/
drwxr-xr-x  3 root root 4096 Sep 23 10:40 processing-data/
drwxr-xr-x  6 root root 4096 Mar  1  2022 pyenv3.9.1/
drwxr-xr-x  6 root root 4096 Oct 24 14:16 pyenv3.9.5/
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
drwxr-xr-x 16 root root 4096 Jan 16 14:31 pyrocko/
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
```
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
<!-- -rwxr-xr-x  1 root root  164 Oct 10 12:55 restart-services*
-rwxr-xr-x  1 root root  166 Oct 11 12:37 restart-web-services* -->

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
where:
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
- `ewricacore`: real time data acquisition and custom process trigger
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
- `ewrica-data*`: the directories where `ewricacore` data is stored (each directory is relative to a different scenario. The directory paths are configured in `ewricacore/configs`)
- `ewricasiria`: source inversion code
- `ewricawebapp`: web app or GUI of ewricacore
- `grond`: A probabilistic earthquake source inversion framework required by `ewricasiria`
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
- `pyenv3.9.1`: the Python virtual environment of `ewricawebapp` (it has also some redundant packages becuase it was previously used to run also `ewricacore`. Note that the Python version is actually 3.9.5)
- `pyenv3.9.5`: the Python virtual environment of `ewricacore`
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
- `pyrocko`: needed for installing [pyrocko from source](https://pyrocko.org/docs/current/install/system/linux/index.html#user-installation-from-source-into-isolated-environment)
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
### Pyenv Installation notes (for docker maintainers only)
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
The packages installed in `pyenv3.9.5` (type `pip freeze` for details) were obtained by installing all required packages in the following order:
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed

```
cd /root/ewrica
python3.9 -m venv ./pyenv3.9.5
source pyenv3.9.5/bin/activate
pip install --upgrade pip setuptools
(cd grond && pip install -e .)
(cd ewricasiria/ && pip install -r ./requirements.txt && pip install -e .)
(cd ewricagm && pip install -r ./requirements.txt && pip install -e .)
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
(cd pyrocko && pip install .)
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
Note: as of January 2023, Pyrocko has to be (re)installed from source because of a small bugfix not available through `pip` (see last line above)

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
<!--
```
cd /root/ewrica
python3.9 -m venv ./pyenv3.9.5
source pyenv3.9.5/bin/activate
pip install --upgrade pip setuptools
cd grond
pip install -e .
cd ../ewricasiria/
pip install -r ./requirements.txt 
pip install -e .
cd ../ewricagm
pip install -r ./requirements.txt 
pip install -e .
```
-->
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
### Update git repository code
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
Once changes are performed (see [Git workflow](#git-workflow)), you `pull` the last changes:
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed

```console
cd /root/ewrica/ewricasiria  # ewricasiria is just an example, it can be any other git repo
git pull
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
```

and then restart *all* services:
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed

```console
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
sv ewricacore restart
... # and so on (see services list above)
```

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
Note: When updating only `ewricawebapp`, you can restart only the two services `ewricawebapp-*` and `nginx`
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
### Edit ewricacore config:
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed

To change the background watcher and trigger service config:

```
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
vim /root/ewrica/ewricacore/configs/deploy-playback.yaml
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
```

Or, for real time (should not be the case for the moment)
```
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
vim /root/ewrica/ewricacore/configs/deploy.yaml
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
Then push changes (see [Git workflow](#git-workflow)) and then restart **all** docker services
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
### Edit ewricacore processing function or function call:

```console
vim /root/ewrica/ewricacore/processing/functions.py
```

Note that if you change the name of the processing function, or implement a new one, you must set the new name in the config file(s) (see section above) in the YAML property 'function_name'. 

When finished, restart the service(s)

Note: everything you put in the directory `/root/ewrica/ewricacore/processing` is ignored by `git` (with the exception of the example file `functions.example.py`), so you can add / remove wehatever you want inside with no risk of committing and pushing huge files, which would make the repository unmaintainable.


Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
## Inspect generated processing files in Docker

The output files generated by the whole processing pipeline are under `/root/ewrica/ewrica-data-playback` (or `ewrica-data` for realtime).
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
Therein, there is a set of directories organizing events per year:
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed

```console
root@ewrica2:/etc/service# ll /root/ewrica/ewrica-data-playback/
total 24
drwxr-xr-x  5 root root 4096 Jul  5 08:04 ./
drwxr-xr-x 10 root root 4096 Jul  5 12:21 ../
drwxr-xr-x  3 root root 4096 Jul  5 08:04 2022/
-rw-r--r--  1 root root 1306 Jul  5 16:49 __current_running_process_info__.log
drwxr-xr-x  2 root root 4096 Jun 23 13:19 playedback-events/
drwxr-xr-x  2 root root 4096 Jun 25 14:42 stations/
```

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
(here in the playback we obviously have only one year, 2022). Inside each year you find the processed event(s) for that year. E.g., for the playback, we can inspect the  only processed event `gfz2016vicq`:
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed

```console
root@ewrica2:/etc/service# ll /root/ewrica/ewrica-data-playback/2022/gfz2016vicq/
total 24
drwxr-xr-x 4 root root 4096 Jul  5 08:15 ./
drwxr-xr-x 3 root root 4096 Jul  5 08:04 ../
-rw-r--r-- 1 root root  130 Jul  6 06:04 gfz2016vicq.json
-rw-r--r-- 1 root root 2245 Jul  6 06:04 gfz2016vicq.xml
drwxr-xr-x 3 root root 4096 Jul  6 05:57 process_execution_01/
drwxr-xr-x 3 root root 4096 Jul  6 06:01 process_execution_02/
```

we can see that there is the event file (XML and JSON), and several `process_execution_**` directories where we store the produced output and the input waveform data. For instance:

```console
root@ewrica2:/etc/service# ll /root/ewrica/ewrica-data-playback/2022/gfz2016vicq/process_execution_01
total 16
drwxr-xr-x 3 root root 4096 Jul  6 05:57 ./
drwxr-xr-x 4 root root 4096 Jul  5 08:15 ../
drwxr-xr-x 2 root root 4096 Jul  5 09:12 __input-data/
-rw-r--r-- 1 root root  127 Jul  6 05:57 __processing-info__.log
```

In the example above for instance, the execution finished successfully (there is no `__processing-error__.log` file) but it did not produce any output. The directory `__input-data` is where we store the miniSEED waveforms used by the process execution. 
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
## Webapp (GUI) in Docker

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
### Change web GUI config

Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed
Similar to `ewricacore`, config files are located in `ewricawebapp/ewrica/configs`. Remeber that you need to restart only web related services after changing the config(s)
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed


### Run GUI in local browser:

Execute `wsgi` or `wsgi-debug` inside `/root/ewrica/ewricawebapp` (`cd /root/ewrica/ewricawebapp` first):
```
export CONFIG_PATH="/root/ewrica/ewricawebapp/ewrica/configs/example-playback.yaml; /root/ewrica/pyenv3.9.1/bin/python ./wsgi-debug.py
```
(change `example-playback.yaml` with `example.yaml` if you want to test realtime, although there is nothing interesting to test that you cannot do with the playback)
Riccardo Zaccarelli's avatar
Riccardo Zaccarelli committed


## Git workflow

This section is intended for developers contributing to any of the Ewrica repositories. Remember: keep `main/master` branch **always production ready**. You can commit and push directly on the `main` branch only for small fixes (e.g. bugfixes) *and* if you have access to the remote docker (`git pull` from there). Otherwise, try whatever you want on separate branches: 

1. create a new branch (let's call it `dev` as example, but you can call it whatever you want): 
   
    `git checkout -b dev`
   
    Or move to an already existing branch:
   
    `git checkout dev` (without `-b`). In this case, the branch might be behind `main` so update it: `git merge main`

2. Work on that branch as you always do: `git add`, `git commit` and `git push -u origin dev` (`-u` only the first time, to add the upstream branch, i.e. on `gitlab`). 

3. When ready for merging, you can merge `dev` into `main/master`:

    - When working with several contributors, one usually issues merge request ([full details here](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)), aka "pull requests" in `github`. **With merge requests other developers are notified and can review and/or deploy your code**: 
    
        I. In the browser go to the project `gitlab`'s page
    
        II. Above the file list, on the right side, select `Create merge request`. A merge request is created. The default branch (`main`) is the target
    
        III. Fill out the fields and select `Create merge request` 

        If you put the docker maintainer(s) in the list of reviewers, they will be notified, so that they can merge and deploy automativally without emails exchange

    - Alternatively you can merge the "quick and dirty way" (but this should be done only for quick bugfixes and if you have docker remote access for deploy):

        ```console
        git checkout main
        git merge dev
        git push
        ```

        Then update (`git pull`) the remote docker.

   
### Merge conflicts

This should not be a big problem here, because we are not 100 developers working simultaneously and regularly on the same code. Howerver, sometimes merging is not possible due to conflicts. For instance, somebody changed or pushed to the `main` branch while you were working in `dev`, and the branches have diverged, which basically means that `git` cannot resolve conflicts.

You will need to fix your code before merging. You will find several instructions on how to do (e.g. [here](https://www.atlassian.com/git/tutorials/using-branches/merge-conflicts)). In case of merge requests, you are notified beforehand on the web page. You can also `git merge main` inside the `dev` regularly so that you can fix merge conflicts when still small and easily fixable.