README.md 7.18 KB
Newer Older
Hannes Diedrich's avatar
Hannes Diedrich committed
1
# GTS2 Sentinel-2 downloader
Hannes Diedrich's avatar
Hannes Diedrich committed
2
3
Status
------
Hannes Diedrich's avatar
Hannes Diedrich committed
4
5
[![pipeline status](https://gitext.gfz-potsdam.de/gts2/gts2_client/badges/master/pipeline.svg)](https://gitext.gfz-potsdam.de/gts2/gts2_client/commits/master)
[![coverage report](https://gitext.gfz-potsdam.de/gts2/gts2_client/badges/master/coverage.svg)](https://gitext.gfz-potsdam.de/gts2/gts2_client/commits/master)
Hannes Diedrich's avatar
Hannes Diedrich committed
6
[![AUR](https://img.shields.io/aur/license/yaourt.svg)](https://gitext.gfz-potsdam.de/gts2/gts2_client/blob/master/LICENCE)
Hannes Diedrich's avatar
Hannes Diedrich committed
7
8


Hannes Diedrich's avatar
Hannes Diedrich committed
9
10
[Coverage report](http://gts2.gitext.gfz-potsdam.de/gts2_client/coverage/)

Hannes Diedrich's avatar
Hannes Diedrich committed
11
## Release notes
12
* **2020-07-20:** Added timeout parameter to avoid freezing of client and changed versioning, making this version 1.0.0
13
* **2019-05-23:** Switched to AC version 0.13
14
* **2018-11-13:** Changed default port to 443
Niklas Bohn's avatar
Niklas Bohn committed
15
* **2018-02-20:** Improved output to netCDF file
16
* **2018-02-16:** Added installation information and installation script for miniconda and all needed packages
Hannes Diedrich's avatar
Hannes Diedrich committed
17
18
* **2018-01-18:** Added docker file and script for building an compatible image and running a container for the client
* **2018-01-17:** Added option for mosaicing/merging tifs and RGBs on client side
Hannes Diedrich's avatar
Hannes Diedrich committed
19
20
* **2018-01-10:** Added output of RGB images into jpg or png, nc-output still in progress

Hannes Diedrich's avatar
Hannes Diedrich committed
21
22
23
24
## Description
This program package downloads Sentinel-2 data from the GTS2
(GFZ Time Series System for Sentinel-2) cloud for a area of interest,
time of interest and wanted band combination and saves them to geotiff (.tiff)
Niklas Bohn's avatar
Niklas Bohn committed
25
 or alternatively as .json file or as netcdf (.nc) file.
Hannes Diedrich's avatar
Hannes Diedrich committed
26

Hannes Diedrich's avatar
Hannes Diedrich committed
27
## Requirements
Hannes Diedrich's avatar
Hannes Diedrich committed
28
29
1. Access to GTS2 API (username, password, port)
1. Python3
Hannes Diedrich's avatar
Hannes Diedrich committed
30
31
32
33
34
1. Python packages:
  * numpy
  * gdal
  * scipy
  * json
Niklas Bohn's avatar
Niklas Bohn committed
35
  * netCDF4
Hannes Diedrich's avatar
Hannes Diedrich committed
36
37
  * requests
  * scikit-image
Hannes Diedrich's avatar
Hannes Diedrich committed
38

39
40
## Download package:
Clone the repository with:
Niklas Bohn's avatar
Niklas Bohn committed
41
<code>git clone https://gitext.gfz-potsdam.de/gts2/gts2_client.git
Hannes Diedrich's avatar
Hannes Diedrich committed
42
</code>
43

Hannes Diedrich's avatar
Hannes Diedrich committed
44
## Installation
Hannes Diedrich's avatar
Hannes Diedrich committed
45
46
### ... is easy (as long your python is compatible):
* Make sure your system meets all the requirements (see above)
Hannes Diedrich's avatar
Hannes Diedrich committed
47
* Install the package by running:
48
49
50
51
```
make install
make clean
```
Hannes Diedrich's avatar
Hannes Diedrich committed
52

Hannes Diedrich's avatar
Hannes Diedrich committed
53
### Install a compatible python:
Hannes Diedrich's avatar
Hannes Diedrich committed
54
The following instruction is only valid for Linux distributions but can be adapted on Windows machines.
Hannes Diedrich's avatar
Hannes Diedrich committed
55
56
57
Everything is based on bash shell.

#### EITHER: Use install script and follow instructions:
58
59
60
61
Run:
```
make install_conda_gts2_client
```
Hannes Diedrich's avatar
Hannes Diedrich committed
62

63
After providing the installation path, miniconda and all necessary packages as well as the gts2_client will be installed.
64
At the script will provide instructions which PATHs to add to your .bashrc.
Hannes Diedrich's avatar
Hannes Diedrich committed
65

Hannes Diedrich's avatar
Hannes Diedrich committed
66
#### OR: Per hand (expert modus):
Hannes Diedrich's avatar
Hannes Diedrich committed
67
* Download and install at least Miniconda with default settings:
Hannes Diedrich's avatar
Hannes Diedrich committed
68
69
```
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
Hannes Diedrich's avatar
Hannes Diedrich committed
70
71
72
73
chmod 755 Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh -b
rm -f Miniconda3-latest-Linux-x86_64.sh
```
Hannes Diedrich's avatar
Hannes Diedrich committed
74
* Install necessary packages for gts2_client using **gts2_client_conda_install.yml**:
Hannes Diedrich's avatar
Hannes Diedrich committed
75
76
77
`conda env update -f gts2_client_conda_install.yml`


Hannes Diedrich's avatar
Hannes Diedrich committed
78
### Finally:
Hannes Diedrich's avatar
Hannes Diedrich committed
79
80
Create a credentials file in your home directory **credentials_gts2_client**
that contains the following structure:
Hannes Diedrich's avatar
Hannes Diedrich committed
81

Hannes Diedrich's avatar
Hannes Diedrich committed
82
```bash
Hannes Diedrich's avatar
Hannes Diedrich committed
83
{"user": "",
84
"password": "",
Hannes Diedrich's avatar
Hannes Diedrich committed
85
"port": ""}
86
```
Hannes Diedrich's avatar
Hannes Diedrich committed
87
88

Please fill in your credentials (provided by GFZ Potsdam, please contact gts2@gfz-potsdam.de).
Hannes Diedrich's avatar
Hannes Diedrich committed
89

Hannes Diedrich's avatar
Hannes Diedrich committed
90
### If you want to use docker
Hannes Diedrich's avatar
Hannes Diedrich committed
91
92
93
94
95
96
97
98
**!! Attention: Expert mode !! root access to computer is needed.**
You only have to run **docker_gts2_client/build_run_gts2_client_docker.sh**
It creates an image that is based on a Centos:7, performs all needed installation steps and starts a container.
You end up with a shell where you can run `gts2_client`.
The building of the image can take some time (up to 30 Minutes), but once it is done,
**docker_gts2_client/build_run_gts2_client_docker.sh** skips the build and only starts the container.


Hannes Diedrich's avatar
Hannes Diedrich committed
99
100
## Usage

101
### As command line tool:
Hannes Diedrich's avatar
Hannes Diedrich committed
102
103
104
105
106
```bash
gts2_client.py required_arguments [optional_arguments]
```

The list of arguments including their default values can be called from the command line with:
Hannes Diedrich's avatar
Hannes Diedrich committed
107
`gts2_client --help`
Hannes Diedrich's avatar
Hannes Diedrich committed
108

109
110
#### Arguments:
##### Required:
Hannes Diedrich's avatar
Hannes Diedrich committed
111
112
  * -o OUT_DIR, --out_dir OUT_DIR
    output_directory
Hannes Diedrich's avatar
Hannes Diedrich committed
113
  * -l LAT_LL, --lat_ll LAT_LL
Hannes Diedrich's avatar
Hannes Diedrich committed
114
    latitude lower left corner
Hannes Diedrich's avatar
Hannes Diedrich committed
115
  * -k LON_LL, --lon_ll LON_LL
Hannes Diedrich's avatar
Hannes Diedrich committed
116
    longitude lower left corner
Hannes Diedrich's avatar
Hannes Diedrich committed
117
  * -i LAT_UR, --lat_ur LAT_UR
Hannes Diedrich's avatar
Hannes Diedrich committed
118
    latitude upper right corner
Hannes Diedrich's avatar
Hannes Diedrich committed
119
  * -j LON_UR, --lon_ur LON_UR
Hannes Diedrich's avatar
Hannes Diedrich committed
120
     longitude upper right corner
Hannes Diedrich's avatar
Hannes Diedrich committed
121
  * -s START_DATE, --start_date START_DATE
Hannes Diedrich's avatar
Hannes Diedrich committed
122
     Startdate (e.g. 20160701)
Hannes Diedrich's avatar
Hannes Diedrich committed
123
  * -e END_DATE, --end_date END_DATE
Hannes Diedrich's avatar
Hannes Diedrich committed
124
     Enddate (e.g. 20160701)
Hannes Diedrich's avatar
Hannes Diedrich committed
125

126
##### Optional
127
128
129
130
  * -r OUT_PREFIX, --out_prefix OUT_PREFIX
    output_prefix for naming the output files
  * -b BANDS, --bands BANDS
     list of Bands (e.g. -b B02_B03)
Hannes Diedrich's avatar
Hannes Diedrich committed
131
132
  * -m OUT_MODE, --out_mode OUT_MODE
    output_mode, use 'json' for json-file,'single' for
133
    single tiffs or 'stack' for tiff band stack, 'nc' for netcdf-file
Hannes Diedrich's avatar
Hannes Diedrich committed
134
135
136
137
138
139
  * -a SENSOR, --sensor SENSOR
     sensor name  (e.g. S2A)
  * -t LEVEL, --level LEVEL
     processing level (e.g. L2A)
  * -v VERSION, --version VERSION
     version of atmospheric correction (e.g. 0.12)
Hannes Diedrich's avatar
Hannes Diedrich committed
140
  * -c COREG, --coreg COREG
Hannes Diedrich's avatar
Hannes Diedrich committed
141
142
143
     get coregistrated data with corrected pixel shifts (e.g. True)
     [pixel shifts between time steps were eliminated by coregistrating
     images with a reference image (currently only avalaible for Germany)]
Hannes Diedrich's avatar
Hannes Diedrich committed
144
  * -z MAX_CLOUDY, --max_cloudy MAX_CLOUDY
Hannes Diedrich's avatar
Hannes Diedrich committed
145
     maximal percentage of cloudyness of requested scene (e.g. 0.2)
Hannes Diedrich's avatar
Hannes Diedrich committed
146
  * -f MINIMUM_FILL, --minimum_fill MINIMUM_FILL
Hannes Diedrich's avatar
Hannes Diedrich committed
147
     minimal percentage of data in scene (e.g. 0.5)
148
149
  * -d UTM_ZONE, --utm_zone UTM_ZONE
    only return data for specific utm-zone (MGRS-tile, e.g. 33UUV)
150
151
  * -g STACK_RESOLUTION, --stack_resolution STACK_RESOLUTION
    spatial sampling [in meters] of the output stack file,
152
                        choose from [10,20])
Hannes Diedrich's avatar
Hannes Diedrich committed
153
154
155
156
157
158
  * -n RGB_EXTENSION, --rgb_extension RGB_EXTENSION
                        file extension of rgb files e.g.[jpg, png], default:
                        jpg (default: jpg)
  * -q RGB_BANDS_SELECTION, --rgb_bands_selection RGB_BANDS_SELECTION
                        band selection for rgb production, choose from:
                        realistic, nice_looking, vegetation,
159
                        healthy_vegetation_urban, snow, agriculture
Hannes Diedrich's avatar
Hannes Diedrich committed
160
                        (default: realistic)
Hannes Diedrich's avatar
Hannes Diedrich committed
161
162
163
164
165
166
  * -w MERGE_TIFS, --merge_tifs MERGE_TIFS
                        Merge tifs and RGBs if area in two or more MGRS tiles
                        per time step (True or False). (default: False)
  * -x MERGE_TILE, --merge_tile MERGE_TILE
                        Choose MGRS tile into which the merge of files is
                        performed (e.g. 33UUV). (default: None)
167

Hannes Diedrich's avatar
Hannes Diedrich committed
168
169
170
171
172
173
174
175
176
177
178
179
**Legend of RGB band selection:**


| Name | band selection |
| :---------  | :----------- |
| realistic | B04, B03, B02 |
| nice_looking | B11, B08, B03 |
| vegetation | B8A, B04, B03 |
| healthy_vegetation_urban | B8A, B11, B02 |
| snow |  B08, B11, B04|
| agriculture | B11, B08, B02 |

180
### As python package
Hannes Diedrich's avatar
Hannes Diedrich committed
181

182
183
184
185
186
187
Set out_mode to "to_python" in order to include the results of the client (dictionary) into
the local namespace:
```
from gts2_client import gts2_client
result = gts2_client.client(out_mode="python", **kwargs)
```
Hannes Diedrich's avatar
Hannes Diedrich committed
188

189
## Limitations:
190
  * Bands with spatial resolution 60m (Band 1) are not stacked.
Hannes Diedrich's avatar
Hannes Diedrich committed
191
192
193
  * Requests with areas larger than 0.2°x0.2° will probably not be processed
     if you also request a large time range this threshold can also be smaller