README.md 6.65 KB
Newer Older
Hannes Diedrich's avatar
Hannes Diedrich committed
1
# GTS2 Sentinel-2 downloader
Hannes Diedrich's avatar
Hannes Diedrich committed
2
3
Status
------
Hannes Diedrich's avatar
Hannes Diedrich committed
4
5
[![pipeline status](https://gitext.gfz-potsdam.de/gts2/gts2_client/badges/master/pipeline.svg)](https://gitext.gfz-potsdam.de/gts2/gts2_client/commits/master)
[![coverage report](https://gitext.gfz-potsdam.de/gts2/gts2_client/badges/master/coverage.svg)](https://gitext.gfz-potsdam.de/gts2/gts2_client/commits/master)
Hannes Diedrich's avatar
Hannes Diedrich committed
6
[![AUR](https://img.shields.io/aur/license/yaourt.svg)](https://gitext.gfz-potsdam.de/gts2/gts2_client/blob/master/LICENCE)
Hannes Diedrich's avatar
Hannes Diedrich committed
7
8


Hannes Diedrich's avatar
Hannes Diedrich committed
9
10
[Coverage report](http://gts2.gitext.gfz-potsdam.de/gts2_client/coverage/)

Hannes Diedrich's avatar
Hannes Diedrich committed
11
## Release notes
12
* **2018-02-16:** Added installation information and installation script for miniconda and all needed packages
Hannes Diedrich's avatar
Hannes Diedrich committed
13
14
* **2018-01-18:** Added docker file and script for building an compatible image and running a container for the client
* **2018-01-17:** Added option for mosaicing/merging tifs and RGBs on client side
Hannes Diedrich's avatar
Hannes Diedrich committed
15
16
* **2018-01-10:** Added output of RGB images into jpg or png, nc-output still in progress

Hannes Diedrich's avatar
Hannes Diedrich committed
17
18
19
20
## Description
This program package downloads Sentinel-2 data from the GTS2
(GFZ Time Series System for Sentinel-2) cloud for a area of interest,
time of interest and wanted band combination and saves them to geotiff (.tiff)
Niklas Bohn's avatar
Niklas Bohn committed
21
 or alternatively as .json file or as netcdf (.nc) file.
Hannes Diedrich's avatar
Hannes Diedrich committed
22

Hannes Diedrich's avatar
Hannes Diedrich committed
23
## Requirements
Hannes Diedrich's avatar
Hannes Diedrich committed
24
25
1. Access to GTS2 API (username, password, port)
1. Python3
Hannes Diedrich's avatar
Hannes Diedrich committed
26
27
28
29
30
1. Python packages:
  * numpy
  * gdal
  * scipy
  * json
Niklas Bohn's avatar
Niklas Bohn committed
31
  * netCDF4
Hannes Diedrich's avatar
Hannes Diedrich committed
32
33
  * requests
  * scikit-image
Hannes Diedrich's avatar
Hannes Diedrich committed
34

35
36
## Download package:
Clone the repository with:
Niklas Bohn's avatar
Niklas Bohn committed
37
<code>git clone https://gitext.gfz-potsdam.de/gts2/gts2_client.git
Hannes Diedrich's avatar
Hannes Diedrich committed
38
</code>
39

Hannes Diedrich's avatar
Hannes Diedrich committed
40
## Installation
Hannes Diedrich's avatar
Hannes Diedrich committed
41
42
### ... is easy (as long your python is compatible):
* Make sure your system meets all the requirements (see above)
Hannes Diedrich's avatar
Hannes Diedrich committed
43
* Install the package by running:
Hannes Diedrich's avatar
Hannes Diedrich committed
44
45
`python gts2_client/setup.py install`

Hannes Diedrich's avatar
Hannes Diedrich committed
46
### Install a compatible python:
Hannes Diedrich's avatar
Hannes Diedrich committed
47
The following instruction is only valid for Linux distributions but can be adapted on Windows machines.
Hannes Diedrich's avatar
Hannes Diedrich committed
48
49
50
51
Everything is based on bash shell.

#### EITHER: Use install script and follow instructions:
Run: `bash install_py_gts2_client.sh`
Hannes Diedrich's avatar
Hannes Diedrich committed
52

Hannes Diedrich's avatar
Hannes Diedrich committed
53
54
After providing the installation path the script will install miniconda
and all necessary packages as well as the gts2_client.
55
At the script will provide instructions which PATHs to add to your .bashrc.
Hannes Diedrich's avatar
Hannes Diedrich committed
56

Hannes Diedrich's avatar
Hannes Diedrich committed
57
#### OR: Per hand (expert modus):
Hannes Diedrich's avatar
Hannes Diedrich committed
58
* Download and install at least Miniconda with default settings:
Hannes Diedrich's avatar
Hannes Diedrich committed
59
60
```
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
Hannes Diedrich's avatar
Hannes Diedrich committed
61
62
63
64
chmod 755 Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh -b
rm -f Miniconda3-latest-Linux-x86_64.sh
```
Hannes Diedrich's avatar
Hannes Diedrich committed
65
* Install necessary packages for gts2_client using **gts2_client_conda_install.yml**:
Hannes Diedrich's avatar
Hannes Diedrich committed
66
67
68
`conda env update -f gts2_client_conda_install.yml`


Hannes Diedrich's avatar
Hannes Diedrich committed
69
### Finally:
Hannes Diedrich's avatar
Hannes Diedrich committed
70
71
Create a credentials file in your home directory **credentials_gts2_client**
that contains the following structure:
Hannes Diedrich's avatar
Hannes Diedrich committed
72

Hannes Diedrich's avatar
Hannes Diedrich committed
73
```bash
Hannes Diedrich's avatar
Hannes Diedrich committed
74
{"user": "",
Hannes Diedrich's avatar
Hannes Diedrich committed
75
76
"password": ""
"port": ""}
77
```
Hannes Diedrich's avatar
Hannes Diedrich committed
78
79

Please fill in your credentials (provided by GFZ Potsdam, please contact gts2@gfz-potsdam.de).
Hannes Diedrich's avatar
Hannes Diedrich committed
80

Hannes Diedrich's avatar
Hannes Diedrich committed
81
### If you want to use docker
Hannes Diedrich's avatar
Hannes Diedrich committed
82
83
84
85
86
87
88
89
**!! Attention: Expert mode !! root access to computer is needed.**
You only have to run **docker_gts2_client/build_run_gts2_client_docker.sh**
It creates an image that is based on a Centos:7, performs all needed installation steps and starts a container.
You end up with a shell where you can run `gts2_client`.
The building of the image can take some time (up to 30 Minutes), but once it is done,
**docker_gts2_client/build_run_gts2_client_docker.sh** skips the build and only starts the container.


Hannes Diedrich's avatar
Hannes Diedrich committed
90
91
## Usage

92
### As command line tool:
Hannes Diedrich's avatar
Hannes Diedrich committed
93
94
95
96
97
```bash
gts2_client.py required_arguments [optional_arguments]
```

The list of arguments including their default values can be called from the command line with:
Hannes Diedrich's avatar
Hannes Diedrich committed
98
`gts2_client --help`
Hannes Diedrich's avatar
Hannes Diedrich committed
99

100
101
#### Arguments:
##### Required:
Hannes Diedrich's avatar
Hannes Diedrich committed
102
103
  * -o OUT_DIR, --out_dir OUT_DIR
    output_directory
Hannes Diedrich's avatar
Hannes Diedrich committed
104
  * -l LAT_LL, --lat_ll LAT_LL
Hannes Diedrich's avatar
Hannes Diedrich committed
105
    latitude lower left corner
Hannes Diedrich's avatar
Hannes Diedrich committed
106
  * -k LON_LL, --lon_ll LON_LL
Hannes Diedrich's avatar
Hannes Diedrich committed
107
    longitude lower left corner
Hannes Diedrich's avatar
Hannes Diedrich committed
108
  * -i LAT_UR, --lat_ur LAT_UR
Hannes Diedrich's avatar
Hannes Diedrich committed
109
    latitude upper right corner
Hannes Diedrich's avatar
Hannes Diedrich committed
110
  * -j LON_UR, --lon_ur LON_UR
Hannes Diedrich's avatar
Hannes Diedrich committed
111
     longitude upper right corner
Hannes Diedrich's avatar
Hannes Diedrich committed
112
  * -s START_DATE, --start_date START_DATE
Hannes Diedrich's avatar
Hannes Diedrich committed
113
     Startdate (e.g. 20160701)
Hannes Diedrich's avatar
Hannes Diedrich committed
114
  * -e END_DATE, --end_date END_DATE
Hannes Diedrich's avatar
Hannes Diedrich committed
115
     Enddate (e.g. 20160701)
Hannes Diedrich's avatar
Hannes Diedrich committed
116

117
##### Optional
118
119
120
121
  * -r OUT_PREFIX, --out_prefix OUT_PREFIX
    output_prefix for naming the output files
  * -b BANDS, --bands BANDS
     list of Bands (e.g. -b B02_B03)
Hannes Diedrich's avatar
Hannes Diedrich committed
122
123
  * -m OUT_MODE, --out_mode OUT_MODE
    output_mode, use 'json' for json-file,'single' for
124
    single tiffs or 'stack' for tiff band stack, 'nc' for netcdf-file
Hannes Diedrich's avatar
Hannes Diedrich committed
125
126
127
128
129
130
  * -a SENSOR, --sensor SENSOR
     sensor name  (e.g. S2A)
  * -t LEVEL, --level LEVEL
     processing level (e.g. L2A)
  * -v VERSION, --version VERSION
     version of atmospheric correction (e.g. 0.12)
Hannes Diedrich's avatar
Hannes Diedrich committed
131
  * -c COREG, --coreg COREG
Hannes Diedrich's avatar
Hannes Diedrich committed
132
133
134
     get coregistrated data with corrected pixel shifts (e.g. True)
     [pixel shifts between time steps were eliminated by coregistrating
     images with a reference image (currently only avalaible for Germany)]
Hannes Diedrich's avatar
Hannes Diedrich committed
135
  * -z MAX_CLOUDY, --max_cloudy MAX_CLOUDY
Hannes Diedrich's avatar
Hannes Diedrich committed
136
     maximal percentage of cloudyness of requested scene (e.g. 0.2)
Hannes Diedrich's avatar
Hannes Diedrich committed
137
  * -f MINIMUM_FILL, --minimum_fill MINIMUM_FILL
Hannes Diedrich's avatar
Hannes Diedrich committed
138
     minimal percentage of data in scene (e.g. 0.5)
139
140
  * -d UTM_ZONE, --utm_zone UTM_ZONE
    only return data for specific utm-zone (MGRS-tile, e.g. 33UUV)
141
142
  * -g STACK_RESOLUTION, --stack_resolution STACK_RESOLUTION
    spatial sampling [in meters] of the output stack file,
143
                        choose from [10,20])
Hannes Diedrich's avatar
Hannes Diedrich committed
144
145
146
147
148
149
  * -n RGB_EXTENSION, --rgb_extension RGB_EXTENSION
                        file extension of rgb files e.g.[jpg, png], default:
                        jpg (default: jpg)
  * -q RGB_BANDS_SELECTION, --rgb_bands_selection RGB_BANDS_SELECTION
                        band selection for rgb production, choose from:
                        realistic, nice_looking, vegetation,
150
                        healthy_vegetation_urban, snow, agriculture
Hannes Diedrich's avatar
Hannes Diedrich committed
151
                        (default: realistic)
Hannes Diedrich's avatar
Hannes Diedrich committed
152
153
154
155
156
157
  * -w MERGE_TIFS, --merge_tifs MERGE_TIFS
                        Merge tifs and RGBs if area in two or more MGRS tiles
                        per time step (True or False). (default: False)
  * -x MERGE_TILE, --merge_tile MERGE_TILE
                        Choose MGRS tile into which the merge of files is
                        performed (e.g. 33UUV). (default: None)
158

159
### As python package
Hannes Diedrich's avatar
Hannes Diedrich committed
160

161
162
163
164
165
166
Set out_mode to "to_python" in order to include the results of the client (dictionary) into
the local namespace:
```
from gts2_client import gts2_client
result = gts2_client.client(out_mode="python", **kwargs)
```
Hannes Diedrich's avatar
Hannes Diedrich committed
167

168
## Limitations:
169
  * Bands with spatial resolution 60m (Band 1) are not stacked.
Hannes Diedrich's avatar
Hannes Diedrich committed
170
171
172
  * Requests with areas larger than 0.2°x0.2° will probably not be processed
     if you also request a large time range this threshold can also be smaller