Commit 00065288 authored by Romulo Pereira Goncalves's avatar Romulo Pereira Goncalves
Browse files

Merge branch 'dev' into 'master'

New release v1.1.0

See merge request !13
parents 7f66bf05 3d2297d6
Pipeline #13045 passed with stages
in 18 minutes and 27 seconds
...@@ -7,6 +7,7 @@ test_gts2_client: ...@@ -7,6 +7,7 @@ test_gts2_client:
script: script:
- source ~/anaconda3/bin/activate - source ~/anaconda3/bin/activate
- export GDAL_DATA=/home/gitlab-runner/anaconda3/share/gdal - export GDAL_DATA=/home/gitlab-runner/anaconda3/share/gdal
- export PROJ_LIB=/root/anaconda3/share/proj
- make coverage - make coverage
artifacts: artifacts:
paths: paths:
...@@ -22,4 +23,4 @@ pages: ...@@ -22,4 +23,4 @@ pages:
artifacts: artifacts:
paths: paths:
- public - public
expire_in: 600 days expire_in: 600 days
\ No newline at end of file
## Release notes
* **2020-09-29:** New scipy, add CHANGELOG.md, fix issues #24, #25, #29 and switched to AC version 0.14. Make this release 1.1.0.
* **2020-07-20:** Added timeout parameter to avoid freezing of client and changed versioning, making this version 1.0.0
* **2019-05-23:** Switched to AC version 0.13
* **2018-11-13:** Changed default port to 443
* **2018-02-20:** Improved output to netCDF file
* **2018-02-16:** Added installation information and installation script for miniconda and all needed packages
* **2018-01-18:** Added docker file and script for building an compatible image and running a container for the client
* **2018-01-17:** Added option for mosaicing/merging tifs and RGBs on client side
* **2018-01-10:** Added output of RGB images into jpg or png, nc-output still in progress
...@@ -8,16 +8,6 @@ Status ...@@ -8,16 +8,6 @@ Status
[Coverage report](http://gts2.gitext.gfz-potsdam.de/gts2_client/coverage/) [Coverage report](http://gts2.gitext.gfz-potsdam.de/gts2_client/coverage/)
## Release notes
* **2020-07-20:** Added timeout parameter to avoid freezing of client and changed versioning, making this version 1.0.0
* **2019-05-23:** Switched to AC version 0.13
* **2018-11-13:** Changed default port to 443
* **2018-02-20:** Improved output to netCDF file
* **2018-02-16:** Added installation information and installation script for miniconda and all needed packages
* **2018-01-18:** Added docker file and script for building an compatible image and running a container for the client
* **2018-01-17:** Added option for mosaicing/merging tifs and RGBs on client side
* **2018-01-10:** Added output of RGB images into jpg or png, nc-output still in progress
## Description ## Description
This program package downloads Sentinel-2 data from the GTS2 This program package downloads Sentinel-2 data from the GTS2
(GFZ Time Series System for Sentinel-2) cloud for a area of interest, (GFZ Time Series System for Sentinel-2) cloud for a area of interest,
...@@ -26,15 +16,8 @@ time of interest and wanted band combination and saves them to geotiff (.tiff) ...@@ -26,15 +16,8 @@ time of interest and wanted band combination and saves them to geotiff (.tiff)
## Requirements ## Requirements
1. Access to GTS2 API (username, password, port) 1. Access to GTS2 API (username, password, port)
1. Python3 2. Python3
1. Python packages: 3. Python packages - check the requirements in [setup.py](./setup.py)
* numpy
* gdal
* scipy
* json
* netCDF4
* requests
* scikit-image
## Download package: ## Download package:
Clone the repository with: Clone the repository with:
...@@ -100,7 +83,7 @@ The building of the image can take some time (up to 30 Minutes), but once it is ...@@ -100,7 +83,7 @@ The building of the image can take some time (up to 30 Minutes), but once it is
### As command line tool: ### As command line tool:
```bash ```bash
gts2_client.py required_arguments [optional_arguments] gts2_client required_arguments [optional_arguments]
``` ```
The list of arguments including their default values can be called from the command line with: The list of arguments including their default values can be called from the command line with:
...@@ -136,7 +119,7 @@ The list of arguments including their default values can be called from the comm ...@@ -136,7 +119,7 @@ The list of arguments including their default values can be called from the comm
* -t LEVEL, --level LEVEL * -t LEVEL, --level LEVEL
processing level (e.g. L2A) processing level (e.g. L2A)
* -v VERSION, --version VERSION * -v VERSION, --version VERSION
version of atmospheric correction (e.g. 0.12) version of atmospheric correction (e.g. 0.14 - recommended version)
* -c COREG, --coreg COREG * -c COREG, --coreg COREG
get coregistrated data with corrected pixel shifts (e.g. True) get coregistrated data with corrected pixel shifts (e.g. True)
[pixel shifts between time steps were eliminated by coregistrating [pixel shifts between time steps were eliminated by coregistrating
......
...@@ -4,7 +4,8 @@ context_dir="./context" ...@@ -4,7 +4,8 @@ context_dir="./context"
dockerfile="gts2_client.docker" dockerfile="gts2_client.docker"
runner_os="centos" runner_os="centos"
runner_iname="gts2_client_runner" runner_iname="gts2_client_runner"
runner_tag="${runner_os}:${runner_iname}" runner_version="latest"
runner_tag="${runner_os}_${runner_iname}:${runner_version}"
container_name="gts2_client" container_name="gts2_client"
cred_file="$HOME/credentials_gts2_client" cred_file="$HOME/credentials_gts2_client"
out_data_folder="/tmp/gts2_client" out_data_folder="/tmp/gts2_client"
...@@ -43,4 +44,4 @@ mkdir -p ${out_data_folder} ...@@ -43,4 +44,4 @@ mkdir -p ${out_data_folder}
echo "Starting gts2_container, please write files to ${out_data_folder} (setting the -o option of client accordingly)" echo "Starting gts2_container, please write files to ${out_data_folder} (setting the -o option of client accordingly)"
sudo docker run -it --name ${container_name} -v ${out_data_folder}:${out_data_folder} ${runner_tag} \ sudo docker run -it --name ${container_name} -v ${out_data_folder}:${out_data_folder} ${runner_tag} \
bash -i -c "cd /home/gts2_client; git pull origin master; source ~/anaconda3/bin/activate; python setup.py install; bash -i -c "cd /home/gts2_client; git pull origin master; source ~/anaconda3/bin/activate; python setup.py install;
echo "";echo "";echo "";echo "";echo '#######';echo 'This is a shell were you can run the gts2_client :::';echo '#######'; bash" echo "";echo "";echo "";echo "";echo '#######';echo 'This is a shell were you can run the gts2_client :::';echo '#######'; bash"
\ No newline at end of file
FROM centos:7 FROM centos:7
RUN yum update -y && \ RUN yum update -y && \
yum install -y wget vim bzip2 git yum install -y wget vim bzip2 gcc gcc-c++ make libgl1-mesa-glx mesa-libGL qt5-qtbase-gui git texlive
ENV anaconda_dl='Anaconda3-5.0.1-Linux-x86_64.sh' ENV anaconda_dl='Anaconda3-2019.07-Linux-x86_64.sh'
RUN /bin/bash -i -c "wget https://repo.continuum.io/archive/$anaconda_dl && \ RUN /bin/bash -i -c "wget https://repo.continuum.io/archive/$anaconda_dl && \
bash ./$anaconda_dl -b && \ bash ./$anaconda_dl -b && \
rm -f /root/$anaconda_dl" rm -f /root/$anaconda_dl"
RUN /bin/bash -i -c "source ~/anaconda3/bin/activate && \ RUN /bin/bash -i -c "source ~/anaconda3/bin/activate && \
conda install --yes -q -c conda-forge gdal 'icu=58.*' lxml pyqt && \ conda config --add channels conda-forge && \
pip install netCDF4 && \ conda install --yes -c conda-forge proj gdal=3.1.0 lxml pyqt netcdf4 coverage libiconv imageio"
conda update -q --all"
RUN /bin/bash -i -c "cd /home/ && \ RUN /bin/bash -i -c "cd /home/ && \
git clone https://gitext.gfz-potsdam.de/gts2/gts2_client.git" git clone https://gitext.gfz-potsdam.de/gts2/gts2_client.git"
COPY credentials_gts2_client /root/credentials_gts2_client COPY credentials_gts2_client /root/credentials_gts2_client
\ No newline at end of file
...@@ -23,7 +23,7 @@ from os.path import join ...@@ -23,7 +23,7 @@ from os.path import join
from os.path import expanduser from os.path import expanduser
from scipy.ndimage.interpolation import zoom from scipy.ndimage.interpolation import zoom
from skimage.exposure import rescale_intensity, adjust_gamma from skimage.exposure import rescale_intensity, adjust_gamma
from scipy.misc import imsave from imageio import imwrite
from subprocess import Popen, PIPE from subprocess import Popen, PIPE
band_settings = {"realistic": ("B04", "B03", "B02"), band_settings = {"realistic": ("B04", "B03", "B02"),
...@@ -361,7 +361,7 @@ def mk_rgb(basedir, outdir, rgb_comb=("B04", "B03", "B02"), rgb_gamma=(1.0, 1.0, ...@@ -361,7 +361,7 @@ def mk_rgb(basedir, outdir, rgb_comb=("B04", "B03", "B02"), rgb_gamma=(1.0, 1.0,
del data del data
del ds del ds
imsave(fn_out, S2_rgb) imwrite(fn_out, S2_rgb)
fnout_list.append(fn_out) fnout_list.append(fn_out)
return fnout_list return fnout_list
...@@ -615,9 +615,9 @@ def json_to_netcdf(out_mode, api_result, outpath, out_prefix, geo_ll, geo_ur, st ...@@ -615,9 +615,9 @@ def json_to_netcdf(out_mode, api_result, outpath, out_prefix, geo_ll, geo_ur, st
for attr in api_result[key][tile][band][data_info].keys()] for attr in api_result[key][tile][band][data_info].keys()]
if data_info == 'data': if data_info == 'data':
band_arr = np.asarray(api_result[key][tile][band][data_info]) band_arr = np.asarray(api_result[key][tile][band][data_info])
band_group.createDimension('x', band_arr.shape[2]) band_group.createDimension('x', band_arr.shape[0])
band_group.createDimension('y', band_arr.shape[1]) band_group.createDimension('y', band_arr.shape[1])
band_group.createDimension('t', band_arr.shape[0]) band_group.createDimension('t', band_arr.shape[2])
fill = api_result[key][tile][band]['fill_value'] fill = api_result[key][tile][band]['fill_value']
data = band_group.createVariable('Data', 'i4', ('x', 'y', 't'), fill_value=fill) data = band_group.createVariable('Data', 'i4', ('x', 'y', 't'), fill_value=fill)
data.units = "None" data.units = "None"
...@@ -666,9 +666,9 @@ def json_to_netcdf(out_mode, api_result, outpath, out_prefix, geo_ll, geo_ur, st ...@@ -666,9 +666,9 @@ def json_to_netcdf(out_mode, api_result, outpath, out_prefix, geo_ll, geo_ur, st
api_result[key][tile][msk][data_info]['time']) api_result[key][tile][msk][data_info]['time'])
if data_info == 'data': if data_info == 'data':
mask_arr = np.asarray(api_result[key][tile][msk][data_info]) mask_arr = np.asarray(api_result[key][tile][msk][data_info])
mask_group.createDimension('x', mask_arr.shape[2]) mask_group.createDimension('x', mask_arr.shape[0])
mask_group.createDimension('y', mask_arr.shape[1]) mask_group.createDimension('y', mask_arr.shape[1])
mask_group.createDimension('t', mask_arr.shape[0]) mask_group.createDimension('t', mask_arr.shape[2])
fill = api_result[key][tile][msk]['fill_value'] fill = api_result[key][tile][msk]['fill_value']
data = mask_group.createVariable('Data', 'i4', ('x', 'y', 't'), fill_value=fill) data = mask_group.createVariable('Data', 'i4', ('x', 'y', 't'), fill_value=fill)
data.units = "None" data.units = "None"
......
...@@ -5,11 +5,12 @@ dependencies: ...@@ -5,11 +5,12 @@ dependencies:
- scipy - scipy
- requests - requests
- scikit-image - scikit-image
- gdal - proj
- gdal=3.1.0
- ipython - ipython
- pip - pip
- libssh2 - libssh2
- pip: - netcdf4
- netcdf4 - imageio
from setuptools import setup, find_packages from setuptools import setup, find_packages
from importlib import util from importlib import util
requirements = ["numpy", "scipy", "netCDF4", "requests", "scikit-image", "gdal"] requirements = ["proj", "numpy", "imageio", "scipy", "netCDF4", "requests", "scikit-image", "gdal"]
other_requirements = ["gdal"] other_requirements = ["gdal"]
test_requirements = requirements + ["coverage"] test_requirements = requirements + ["coverage"]
...@@ -17,14 +17,18 @@ if not_installed != []: ...@@ -17,14 +17,18 @@ if not_installed != []:
', '.join(not_installed))) ', '.join(not_installed)))
setup(name='gts2_client', setup(name='gts2_client',
version='1.0.0', version='1.1.0',
packages=find_packages(exclude=['tests*']), packages=find_packages(exclude=['tests*']),
url='https://gitext.gfz-potsdam.de/gts2/gts2_client.git', url='https://gitext.gfz-potsdam.de/gts2/gts2_client.git',
license='GNU LESSER GENERAL PUBLIC LICENSE - Version 3, 29 June 2007', license='GNU LESSER GENERAL PUBLIC LICENSE - Version 3, 29 June 2007',
author='Hannes Diedrich, Niklas Bohn, Andre Hollstein', author='Hannes Diedrich, Niklas Bohn, Andre Hollstein',
author_email='hannes.diedrich@gfz-potsdam.de', author_email='hannes.diedrich@gfz-potsdam.de',
description='Downloads Sentinel-2 data from GTS2 cloud', description='Downloads Sentinel-2 data from GTS2 cloud',
scripts=['gts2_client/gts2_client.py'], entry_points={
'console_scripts': [
'gts2_client = gts2_client.gts2_client:main'
],
},
install_requires=requirements, install_requires=requirements,
test_suite='tests', test_suite='tests',
tests_require=test_requirements tests_require=test_requirements
......
...@@ -2,7 +2,8 @@ ...@@ -2,7 +2,8 @@
context_dir="./context" context_dir="./context"
dockerfile="build_runner_image.docker" dockerfile="build_runner_image.docker"
runner_tag="gts2_client_ci" runner_version="latest"
runner_tag="gts2_client_ci:$runner_version"
gitlab_runner="gts2_client_gitlab_CI_runner" gitlab_runner="gts2_client_gitlab_CI_runner"
echo "Preparing gts2 credentials file" echo "Preparing gts2 credentials file"
...@@ -37,5 +38,6 @@ sudo docker exec -it ${gitlab_runner} /bin/bash -c "export RUNNER_EXECUTOR=docke ...@@ -37,5 +38,6 @@ sudo docker exec -it ${gitlab_runner} /bin/bash -c "export RUNNER_EXECUTOR=docke
--run-untagged=true \ --run-untagged=true \
--locked=true \ --locked=true \
--tag-list gts2_client \ --tag-list gts2_client \
--docker-pull-policy='never' \
--description '${runner_name}' \ --description '${runner_name}' \
--docker-image '${runner_tag}:latest' " --docker-image '${runner_tag}' "
FROM centos:7 FROM centos:7
RUN yum update -y && \ RUN yum update -y && \
yum install -y wget vim bzip2 gcc gcc-c++ make libgl1-mesa-glx mesa-libGL qt5-qtbase-gui git texlive yum install -y wget vim bzip2 gcc gcc-c++ make libgl1-mesa-glx mesa-libGL qt5-qtbase-gui git texlive
RUN /bin/bash -i -c "wget https://repo.continuum.io/archive/Anaconda3-4.3.1-Linux-x86_64.sh && \ ENV anaconda_dl='Anaconda3-2019.07-Linux-x86_64.sh'
bash ./Anaconda3-4.3.1-Linux-x86_64.sh -b && \ RUN /bin/bash -i -c "wget https://repo.continuum.io/archive/$anaconda_dl && \
rm -f /root/Anaconda3-4.3.1-Linux-x86_64.sh" bash ./$anaconda_dl -b && \
rm -f /root/$anaconda_dl"
RUN /bin/bash -i -c "source ~/anaconda3/bin/activate && \ RUN /bin/bash -i -c "source ~/anaconda3/bin/activate && \
conda install --yes -c conda-forge gdal 'icu=58.*' lxml pyqt coverage && \ conda config --add channels conda-forge && \
pip install netCDF4" conda install --yes -c conda-forge proj gdal=3.1.0 lxml pyqt netcdf4 coverage libiconv imageio"
RUN mkdir -p /home/tmp/ RUN mkdir -p /home/tmp/
COPY credentials_gts2_client /root/ COPY credentials_gts2_client /root/
\ No newline at end of file
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment