Merge branch 'feature/merge_tifs' into 'master'

Feature/merge tifs

See merge request !6

README.md

@@ -7,6 +7,9 @@ Status
 [Coverage report](http://gts2.gitext.gfz-potsdam.de/gts2_client/coverage/)
 
 ## Release notes
+* **2018-02-16:** Added installation information and installation script for miniconda and all needed packages
+* **2018-01-18:** Added docker file and script for building an compatible image and running a container for the client
+* **2018-01-17:** Added option for mosaicing/merging tifs and RGBs on client side
 * **2018-01-10:** Added output of RGB images into jpg or png, nc-output still in progress
 
 ## Description
@@ -16,8 +19,8 @@ time of interest and wanted band combination and saves them to geotiff (.tiff)
  or alternatively as .json file or as netcdf (.nc) file.
 
 ## Requirements
-1. Access to GTS2 API (username and password)
-1. Python 3
+1. Access to GTS2 API (username, password, port)
+1. Python3
 1. Python packages:
   * numpy
   * gdal
@@ -33,21 +36,55 @@ Clone the repository with:
 </code>
 
 ## Installation
-1. Make sure your system meets all the Requirements (see above)
-  * For GFZ users: you can achieve that by using the gfz python.
-   For that please run: <code>module load pygfz</code>
-2. Install the package by first going into the repository folder and then run:
-<code> python gts2_client/setup.py install </code>
-3. Create a credentials file in your home directory **credentials_gts2_client**
-   that contains the following structure:
+### ... is easy (as long your python is compatible):
+* Make sure your system meets all the requirements (see above)
+* Install the package by running:
+`python gts2_client/setup.py install`
+
+### Install a compatible python:
+The following instruction is only valid for Linux distributions but can be adapted on Windows machines.
+Everything is based on bash shell.
+
+#### EITHER: Use install script and follow instructions:
+Run: `bash install_py_gts2_client.sh`
+
+After providing the installation path the script will install miniconda
+and all necessary packages as well as the gts2_client.
+At the script will provide instructions which PATHs to add to your .bashrc.
+
+#### OR: Per hand (expert modus):
+* Download and install at least Miniconda with default settings:
+```
+wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
+chmod 755 Miniconda3-latest-Linux-x86_64.sh
+./Miniconda3-latest-Linux-x86_64.sh -b
+rm -f Miniconda3-latest-Linux-x86_64.sh
+```
+* Install necessary packages for gts2_client using **gts2_client_conda_install.yml**:
+`conda env update -f gts2_client_conda_install.yml`
+
+
+### Finally:
+Create a credentials file in your home directory **credentials_gts2_client**
+that contains the following structure:
 
 ```bash
 {"user": "",
-"password": ""}
+"password": ""
+"port": ""}
 ```
 
 Please fill in your credentials (provided by GFZ Potsdam, please contact gts2@gfz-potsdam.de).
 
+### If you want to use docker
+**!! Attention: Expert mode !! root access to computer is needed.**
+You only have to run **docker_gts2_client/build_run_gts2_client_docker.sh**
+It creates an image that is based on a Centos:7, performs all needed installation steps and starts a container.
+You end up with a shell where you can run `gts2_client`.
+The building of the image can take some time (up to 30 Minutes), but once it is done,
+**docker_gts2_client/build_run_gts2_client_docker.sh** skips the build and only starts the container.
+
+
 ## Usage
 
 ### As command line tool:
@@ -56,7 +93,7 @@ gts2_client.py required_arguments [optional_arguments]
 ```
 
 The list of arguments including their default values can be called from the command line with:
-<code>gts2_client --help</code>
+`gts2_client --help`
 
 #### Arguments:
 ##### Required:
@@ -108,8 +145,14 @@ The list of arguments including their default values can be called from the comm
   * -q RGB_BANDS_SELECTION, --rgb_bands_selection RGB_BANDS_SELECTION
                         band selection for rgb production, choose from:
                         realistic, nice_looking, vegetation,
-                        healthy_vegetation_urban, water, snow, agriculture
+                        healthy_vegetation_urban, snow, agriculture
                         (default: realistic)
+  * -w MERGE_TIFS, --merge_tifs MERGE_TIFS
+                        Merge tifs and RGBs if area in two or more MGRS tiles
+                        per time step (True or False). (default: False)
+  * -x MERGE_TILE, --merge_tile MERGE_TILE
+                        Choose MGRS tile into which the merge of files is
+                        performed (e.g. 33UUV). (default: None)
 
 ### As python package
 
@@ -122,3 +165,6 @@ result = gts2_client.client(out_mode="python", **kwargs)
 
 ## Limitations:
   * Bands with spatial resolution 60m (Band 1) are not stacked.
+  * Requests with areas larger than 0.2°x0.2° will probably not be processed
+     if you also request a large time range this threshold can also be smaller
+

credentials_gts2_client

 {
   "user": "",
-  "password": ""
+  "password": "",
+  "port": 80
 }
\ No newline at end of file

docker_gts2_client/build_run_gts2_client_docker.sh

+#!/usr/bin/env bash
+
+context_dir="./context"
+dockerfile="gts2_client.docker"
+runner_os="centos"
+runner_iname="gts2_client_runner"
+runner_tag="${runner_os}:${runner_iname}"
+container_name="gts2_client"
+cred_file="$HOME/credentials_gts2_client"
+out_data_folder="/tmp/gts2_client"
+
+#Check if image exists
+if [ $(sudo docker images | grep ${runner_iname} | wc -l) == 0 ]
+then
+
+    # Copying credentials file into context dir, if exists
+    if [ -e ${cred_file} ]
+    then
+        echo "cp ${cred_file} ./context/"
+        cp ${cred_file} ./context/
+    else
+        echo "read"
+        read -p "Please enter GTS2 username: " username
+        read -p "Please enter GTS2 password: " password
+        read -p "Please enter GTS2 port (default 80): " port
+        cat > ${cred_file} <<EOL
+    {
+      "user": "${username}",
+      "password": "${password}",
+      "port": ${port}
+    }
+EOL
+    fi
+
+    # build docker image
+    sudo docker build -f ${context_dir}/${dockerfile} -m 20G -t ${runner_tag} ${context_dir}
+
+fi
+
+#run container and start a shell where you can run the gts2_client
+sudo docker rm -f ${container_name}
+mkdir -p ${out_data_folder}
+echo "Starting gts2_container, please write files to ${out_data_folder} (setting the -o option of client accordingly)"
+sudo docker run -it --name ${container_name} -v ${out_data_folder}:${out_data_folder} ${runner_tag} \
+bash -i -c "cd /home/gts2_client; git pull origin master; source ~/anaconda3/bin/activate; python setup.py install;
+echo "";echo "";echo "";echo "";echo '#######';echo 'This is a shell were you can run the gts2_client :::';echo '#######';  bash"
\ No newline at end of file

docker_gts2_client/context/gts2_client.docker

+FROM centos:7
+RUN yum update -y && \
+    yum install -y wget vim bzip2 git
+ENV anaconda_dl='Anaconda3-5.0.1-Linux-x86_64.sh'
+RUN /bin/bash -i -c "wget https://repo.continuum.io/archive/$anaconda_dl && \
+    bash ./$anaconda_dl -b && \
+    rm -f /root/$anaconda_dl"
+RUN /bin/bash -i -c "source ~/anaconda3/bin/activate && \
+    conda install --yes -q -c conda-forge gdal 'icu=58.*' lxml pyqt && \
+    pip install netCDF4 && \
+    conda update -q --all"
+RUN /bin/bash -i -c "cd /home/ && \
+    git clone https://gitext.gfz-potsdam.de/gts2/gts2_client.git"
+COPY credentials_gts2_client /root/credentials_gts2_client
\ No newline at end of file

gts2_client_conda_install.yml

+name: root
+channels:
+  - conda-forge
+dependencies:
+  - scipy
+  - requests
+  - scikit-image
+  - gdal
+  - ipython
+  - pip
+  - libssh2
+  - pip:
+    - netcdf4
+
+

install_py_gts2_client.sh

+#!/usr/bin/env bash
+
+echo "###################################"
+echo "This is the installation routine for installing Miniconda and all necessary packages for gts2_client."
+echo "Depending on the load of your machine, this process can take up to an hour, so lean back and relax!"
+echo "All files are stored into your installation path which you will provide in the following."
+echo "Please make sure that you have sufficient free space in ths installation path (at least 3 GB)."
+echo "###################################"
+current_path=$(pwd)
+
+read -p "Please enter path for the python installation: " inst_path
+cd ${inst_path}
+
+conda_vers="miniconda3"
+url="https://repo.continuum.io/miniconda/"
+conda_inst_name="Miniconda3-latest-Linux-x86_64.sh"
+echo "#### Downloading ${conda_vers}.... ####"
+wget ${url}${conda_inst_name}
+chmod 755 ${conda_inst_name}
+echo "#### Installing ${conda_vers}.... ####"
+./${conda_inst_name} -p ${inst_path}/${conda_vers} -b
+rm -f ${conda_inst_name}
+echo "... Ready"
+
+env_name=gts2_client
+echo "Creating a conda environment named: ${env_name} ..."
+export PATH=${inst_path}/${conda_vers}/bin/:$PATH
+export PYTHONPATH=${inst_path}/${conda_vers}/lib/python3.6/site-packages/
+echo "... Ready"
+
+echo "Installing necessary packages ..."
+conda env update -f ${current_path}/gts2_client_conda_install.yml
+echo "... Ready"
+
+echo "Installing gts2_client ..."
+cd ${current_path}
+python setup.py install
+echo "... Ready"
+
+echo "###############################"
+echo "Congratulations! Miniconda and all necessary packages are installed"
+echo " "
+echo "###############################"
+echo "In order to load your environment and run gts2_client, run the following commands in your bash shell:"
+echo "###############################"
+echo "### 1. ###"
+echo "Set the paths right (you can also add the following lines to your $HOME/.bashrc [recommended]) :"
+echo "--------"
+echo "export PATH=${inst_path}/${conda_vers}/bin/:$PATH"
+echo "export PYTHONPATH=${inst_path}/${conda_vers}/lib/python3.6/site-packages/:$PYTHONPATH"
+echo "export LD_LIBRARY_PATH=${inst_path}/${conda_vers}/lib"
+echo "--------"
+echo "### 2. ###"
+echo "Activate your conda environment:"
+echo "source activate ${env_name}"
+echo " "
+echo "... Now you can use the gts2_client ..."
+echo " "

setup.py

 from setuptools import setup, find_packages
 from importlib import util
 
-requirements = ["numpy", "scipy", "netCDF4", "requests", "scikit-image"]
+requirements = ["numpy", "scipy", "netCDF4", "requests", "scikit-image", "gdal"]
 other_requirements = ["gdal"]
 test_requirements = requirements + ["coverage"]
 
@@ -17,7 +17,7 @@ if not_installed != []:
             ', '.join(not_installed)))
 
 setup(name='gts2_client',
-      version='0.4',
+      version='0.5',
       packages=find_packages(exclude=['tests*']),
       url='https://gitext.gfz-potsdam.de/gts2/gts2_client.git',
       license='GNU LESSER GENERAL PUBLIC LICENSE - Version 3, 29 June 2007',
@@ -28,4 +28,4 @@ setup(name='gts2_client',
       install_requires=requirements,
       test_suite='tests',
       tests_require=test_requirements
-      )
\ No newline at end of file
+      )

tests/test_gts2_client.py

@@ -62,11 +62,44 @@ class TestGts2Client(unittest.TestCase):
                     minimum_fill=minimum_fill,
                     stack_resolution=stack_res)
 
-                list_of_files = glob(os.path.join(tmpdir, "*{mode}*.{form}".format(form=out_formats[ii],
-                                                                                   mode=out_mode)))
+                list_of_files = glob(os.path.join(tmpdir, "*.{form}".format(form=out_formats[ii])))
                 if len(list_of_files) == 0:
-                    raise FileNotFoundError("Could not find files matching: *{mode}*.{form}".format(
-                        form=out_formats[ii], mode=out_mode))
+                    raise FileNotFoundError("Could not find files matching: *.{form}".format(form=out_formats[ii]))
+                else:
+                    print("Test OK.")
+
+    def test_gts2_client_merge(self):
+        """
+        Test the merge tifs and rgbs if different tiles are retrieved.
+        :return:
+        """
+        out_modes = ["single", "stack", "rgb", "stack"]
+        out_formats = ["tif", "tif", "jpg", "tif"]
+        levels = ["L2A", "L2A", "L2A", "L1C"]
+        merge_tile = ["32UQD", "32UQD", None, None]
+
+        for ii, out_mode in enumerate(out_modes):
+            with tempfile.TemporaryDirectory() as tmpdir:
+                gts2_client.client(outpath=tmpdir,
+                                   out_prefix="krh",
+                                   out_mode=out_mode,
+                                   geo_ll=(12.559433, 53.036066),
+                                   geo_ur=(12.737961, 53.238058),
+                                   sensor="S2A",
+                                   level=levels[ii],
+                                   version="0.12",
+                                   bands="B04_B03_B02",
+                                   max_cloudy="0.2",
+                                   suffix="",
+                                   start_date="20170501",
+                                   end_date="20170530",
+                                   minimum_fill="0.9",
+                                   merge_tifs=True,
+                                   merge_tile=merge_tile[ii])
+
+                list_of_files = glob(os.path.join(tmpdir, "*.{form}".format(form=out_formats[ii])))
+                if len(list_of_files) == 0:
+                    raise FileNotFoundError("Could not find files matching: *.{form}".format(form=out_formats[ii]))
                 else:
                     print("Test OK.")
 
@@ -162,6 +195,45 @@ class TestGts2Client(unittest.TestCase):
         :return:
         """
 
+        try:
+            out_mode = "stack"
+            print("#### Testing '_' in prefix ")
+            gts2_client.client(
+                out_prefix="test_",
+                out_mode=out_mode,
+                geo_ll=geo_ll,
+                geo_ur=geo_ur,
+                bands="B05",
+                start_date=start_date,
+                end_date=end_date,
+                version=version,
+                level=level,
+                max_cloudy=max_cloudy,
+                minimum_fill=minimum_fill,
+                stack_resolution=stack_res,
+                quiet=False)
+        except ValueError:
+            print("Test OK.")
+
+        try:
+            out_mode = "stack"
+            print("#### Testing too large area")
+            gts2_client.client(
+                out_mode=out_mode,
+                geo_ll=geo_ll,
+                geo_ur=(12.737961+0.4, 53.238058+0.4),
+                bands="B05",
+                start_date=start_date,
+                end_date=end_date,
+                version=version,
+                level=level,
+                max_cloudy=max_cloudy,
+                minimum_fill=minimum_fill,
+                stack_resolution=stack_res,
+                quiet=False)
+        except ValueError:
+            print("Test OK.")
+
         try:
             out_mode = "python"
             print("#### Testing stack_resolution=60")
@@ -201,7 +273,7 @@ class TestGts2Client(unittest.TestCase):
             print("Test OK.")
 
         try:
-            out_mode = "dowsnotwork"
+            out_mode = "doesnotwork"
             print("#### Testing wrong out_mode")
             gts2_client.client(
                 out_mode=out_mode,