Commit 3d2297d6 authored by Romulo Pereira Goncalves's avatar Romulo Pereira Goncalves
Browse files

The recent version of scipy does not support anymore imsave. We need to use...

The recent version of scipy does not support anymore imsave. We need to use imwrite from Python's imageio module.
parent 7f66bf05
...@@ -7,6 +7,7 @@ test_gts2_client: ...@@ -7,6 +7,7 @@ test_gts2_client:
script: script:
- source ~/anaconda3/bin/activate - source ~/anaconda3/bin/activate
- export GDAL_DATA=/home/gitlab-runner/anaconda3/share/gdal - export GDAL_DATA=/home/gitlab-runner/anaconda3/share/gdal
- export PROJ_LIB=/root/anaconda3/share/proj
- make coverage - make coverage
artifacts: artifacts:
paths: paths:
......
## Release notes
* **2020-09-29:** New scipy, add CHANGELOG.md, fix issues #24, #25, #29 and switched to AC version 0.14. Make this release 1.1.0.
* **2020-07-20:** Added timeout parameter to avoid freezing of client and changed versioning, making this version 1.0.0
* **2019-05-23:** Switched to AC version 0.13
* **2018-11-13:** Changed default port to 443
* **2018-02-20:** Improved output to netCDF file
* **2018-02-16:** Added installation information and installation script for miniconda and all needed packages
* **2018-01-18:** Added docker file and script for building an compatible image and running a container for the client
* **2018-01-17:** Added option for mosaicing/merging tifs and RGBs on client side
* **2018-01-10:** Added output of RGB images into jpg or png, nc-output still in progress
...@@ -8,16 +8,6 @@ Status ...@@ -8,16 +8,6 @@ Status
[Coverage report](http://gts2.gitext.gfz-potsdam.de/gts2_client/coverage/) [Coverage report](http://gts2.gitext.gfz-potsdam.de/gts2_client/coverage/)
## Release notes
* **2020-07-20:** Added timeout parameter to avoid freezing of client and changed versioning, making this version 1.0.0
* **2019-05-23:** Switched to AC version 0.13
* **2018-11-13:** Changed default port to 443
* **2018-02-20:** Improved output to netCDF file
* **2018-02-16:** Added installation information and installation script for miniconda and all needed packages
* **2018-01-18:** Added docker file and script for building an compatible image and running a container for the client
* **2018-01-17:** Added option for mosaicing/merging tifs and RGBs on client side
* **2018-01-10:** Added output of RGB images into jpg or png, nc-output still in progress
## Description ## Description
This program package downloads Sentinel-2 data from the GTS2 This program package downloads Sentinel-2 data from the GTS2
(GFZ Time Series System for Sentinel-2) cloud for a area of interest, (GFZ Time Series System for Sentinel-2) cloud for a area of interest,
...@@ -26,15 +16,8 @@ time of interest and wanted band combination and saves them to geotiff (.tiff) ...@@ -26,15 +16,8 @@ time of interest and wanted band combination and saves them to geotiff (.tiff)
## Requirements ## Requirements
1. Access to GTS2 API (username, password, port) 1. Access to GTS2 API (username, password, port)
1. Python3 2. Python3
1. Python packages: 3. Python packages - check the requirements in [setup.py](./setup.py)
* numpy
* gdal
* scipy
* json
* netCDF4
* requests
* scikit-image
## Download package: ## Download package:
Clone the repository with: Clone the repository with:
...@@ -100,7 +83,7 @@ The building of the image can take some time (up to 30 Minutes), but once it is ...@@ -100,7 +83,7 @@ The building of the image can take some time (up to 30 Minutes), but once it is
### As command line tool: ### As command line tool:
```bash ```bash
gts2_client.py required_arguments [optional_arguments] gts2_client required_arguments [optional_arguments]
``` ```
The list of arguments including their default values can be called from the command line with: The list of arguments including their default values can be called from the command line with:
...@@ -136,7 +119,7 @@ The list of arguments including their default values can be called from the comm ...@@ -136,7 +119,7 @@ The list of arguments including their default values can be called from the comm
* -t LEVEL, --level LEVEL * -t LEVEL, --level LEVEL
processing level (e.g. L2A) processing level (e.g. L2A)
* -v VERSION, --version VERSION * -v VERSION, --version VERSION
version of atmospheric correction (e.g. 0.12) version of atmospheric correction (e.g. 0.14 - recommended version)
* -c COREG, --coreg COREG * -c COREG, --coreg COREG
get coregistrated data with corrected pixel shifts (e.g. True) get coregistrated data with corrected pixel shifts (e.g. True)
[pixel shifts between time steps were eliminated by coregistrating [pixel shifts between time steps were eliminated by coregistrating
......
...@@ -4,7 +4,8 @@ context_dir="./context" ...@@ -4,7 +4,8 @@ context_dir="./context"
dockerfile="gts2_client.docker" dockerfile="gts2_client.docker"
runner_os="centos" runner_os="centos"
runner_iname="gts2_client_runner" runner_iname="gts2_client_runner"
runner_tag="${runner_os}:${runner_iname}" runner_version="latest"
runner_tag="${runner_os}_${runner_iname}:${runner_version}"
container_name="gts2_client" container_name="gts2_client"
cred_file="$HOME/credentials_gts2_client" cred_file="$HOME/credentials_gts2_client"
out_data_folder="/tmp/gts2_client" out_data_folder="/tmp/gts2_client"
......
FROM centos:7 FROM centos:7
RUN yum update -y && \ RUN yum update -y && \
yum install -y wget vim bzip2 git yum install -y wget vim bzip2 gcc gcc-c++ make libgl1-mesa-glx mesa-libGL qt5-qtbase-gui git texlive
ENV anaconda_dl='Anaconda3-5.0.1-Linux-x86_64.sh' ENV anaconda_dl='Anaconda3-2019.07-Linux-x86_64.sh'
RUN /bin/bash -i -c "wget https://repo.continuum.io/archive/$anaconda_dl && \ RUN /bin/bash -i -c "wget https://repo.continuum.io/archive/$anaconda_dl && \
bash ./$anaconda_dl -b && \ bash ./$anaconda_dl -b && \
rm -f /root/$anaconda_dl" rm -f /root/$anaconda_dl"
RUN /bin/bash -i -c "source ~/anaconda3/bin/activate && \ RUN /bin/bash -i -c "source ~/anaconda3/bin/activate && \
conda install --yes -q -c conda-forge gdal 'icu=58.*' lxml pyqt && \ conda config --add channels conda-forge && \
pip install netCDF4 && \ conda install --yes -c conda-forge proj gdal=3.1.0 lxml pyqt netcdf4 coverage libiconv imageio"
conda update -q --all"
RUN /bin/bash -i -c "cd /home/ && \ RUN /bin/bash -i -c "cd /home/ && \
git clone https://gitext.gfz-potsdam.de/gts2/gts2_client.git" git clone https://gitext.gfz-potsdam.de/gts2/gts2_client.git"
COPY credentials_gts2_client /root/credentials_gts2_client COPY credentials_gts2_client /root/credentials_gts2_client
...@@ -23,7 +23,7 @@ from os.path import join ...@@ -23,7 +23,7 @@ from os.path import join
from os.path import expanduser from os.path import expanduser
from scipy.ndimage.interpolation import zoom from scipy.ndimage.interpolation import zoom
from skimage.exposure import rescale_intensity, adjust_gamma from skimage.exposure import rescale_intensity, adjust_gamma
from scipy.misc import imsave from imageio import imwrite
from subprocess import Popen, PIPE from subprocess import Popen, PIPE
band_settings = {"realistic": ("B04", "B03", "B02"), band_settings = {"realistic": ("B04", "B03", "B02"),
...@@ -361,7 +361,7 @@ def mk_rgb(basedir, outdir, rgb_comb=("B04", "B03", "B02"), rgb_gamma=(1.0, 1.0, ...@@ -361,7 +361,7 @@ def mk_rgb(basedir, outdir, rgb_comb=("B04", "B03", "B02"), rgb_gamma=(1.0, 1.0,
del data del data
del ds del ds
imsave(fn_out, S2_rgb) imwrite(fn_out, S2_rgb)
fnout_list.append(fn_out) fnout_list.append(fn_out)
return fnout_list return fnout_list
...@@ -615,9 +615,9 @@ def json_to_netcdf(out_mode, api_result, outpath, out_prefix, geo_ll, geo_ur, st ...@@ -615,9 +615,9 @@ def json_to_netcdf(out_mode, api_result, outpath, out_prefix, geo_ll, geo_ur, st
for attr in api_result[key][tile][band][data_info].keys()] for attr in api_result[key][tile][band][data_info].keys()]
if data_info == 'data': if data_info == 'data':
band_arr = np.asarray(api_result[key][tile][band][data_info]) band_arr = np.asarray(api_result[key][tile][band][data_info])
band_group.createDimension('x', band_arr.shape[2]) band_group.createDimension('x', band_arr.shape[0])
band_group.createDimension('y', band_arr.shape[1]) band_group.createDimension('y', band_arr.shape[1])
band_group.createDimension('t', band_arr.shape[0]) band_group.createDimension('t', band_arr.shape[2])
fill = api_result[key][tile][band]['fill_value'] fill = api_result[key][tile][band]['fill_value']
data = band_group.createVariable('Data', 'i4', ('x', 'y', 't'), fill_value=fill) data = band_group.createVariable('Data', 'i4', ('x', 'y', 't'), fill_value=fill)
data.units = "None" data.units = "None"
...@@ -666,9 +666,9 @@ def json_to_netcdf(out_mode, api_result, outpath, out_prefix, geo_ll, geo_ur, st ...@@ -666,9 +666,9 @@ def json_to_netcdf(out_mode, api_result, outpath, out_prefix, geo_ll, geo_ur, st
api_result[key][tile][msk][data_info]['time']) api_result[key][tile][msk][data_info]['time'])
if data_info == 'data': if data_info == 'data':
mask_arr = np.asarray(api_result[key][tile][msk][data_info]) mask_arr = np.asarray(api_result[key][tile][msk][data_info])
mask_group.createDimension('x', mask_arr.shape[2]) mask_group.createDimension('x', mask_arr.shape[0])
mask_group.createDimension('y', mask_arr.shape[1]) mask_group.createDimension('y', mask_arr.shape[1])
mask_group.createDimension('t', mask_arr.shape[0]) mask_group.createDimension('t', mask_arr.shape[2])
fill = api_result[key][tile][msk]['fill_value'] fill = api_result[key][tile][msk]['fill_value']
data = mask_group.createVariable('Data', 'i4', ('x', 'y', 't'), fill_value=fill) data = mask_group.createVariable('Data', 'i4', ('x', 'y', 't'), fill_value=fill)
data.units = "None" data.units = "None"
......
...@@ -5,11 +5,12 @@ dependencies: ...@@ -5,11 +5,12 @@ dependencies:
- scipy - scipy
- requests - requests
- scikit-image - scikit-image
- gdal - proj
- gdal=3.1.0
- ipython - ipython
- pip - pip
- libssh2 - libssh2
- pip:
- netcdf4 - netcdf4
- imageio
from setuptools import setup, find_packages from setuptools import setup, find_packages
from importlib import util from importlib import util
requirements = ["numpy", "scipy", "netCDF4", "requests", "scikit-image", "gdal"] requirements = ["proj", "numpy", "imageio", "scipy", "netCDF4", "requests", "scikit-image", "gdal"]
other_requirements = ["gdal"] other_requirements = ["gdal"]
test_requirements = requirements + ["coverage"] test_requirements = requirements + ["coverage"]
...@@ -17,14 +17,18 @@ if not_installed != []: ...@@ -17,14 +17,18 @@ if not_installed != []:
', '.join(not_installed))) ', '.join(not_installed)))
setup(name='gts2_client', setup(name='gts2_client',
version='1.0.0', version='1.1.0',
packages=find_packages(exclude=['tests*']), packages=find_packages(exclude=['tests*']),
url='https://gitext.gfz-potsdam.de/gts2/gts2_client.git', url='https://gitext.gfz-potsdam.de/gts2/gts2_client.git',
license='GNU LESSER GENERAL PUBLIC LICENSE - Version 3, 29 June 2007', license='GNU LESSER GENERAL PUBLIC LICENSE - Version 3, 29 June 2007',
author='Hannes Diedrich, Niklas Bohn, Andre Hollstein', author='Hannes Diedrich, Niklas Bohn, Andre Hollstein',
author_email='hannes.diedrich@gfz-potsdam.de', author_email='hannes.diedrich@gfz-potsdam.de',
description='Downloads Sentinel-2 data from GTS2 cloud', description='Downloads Sentinel-2 data from GTS2 cloud',
scripts=['gts2_client/gts2_client.py'], entry_points={
'console_scripts': [
'gts2_client = gts2_client.gts2_client:main'
],
},
install_requires=requirements, install_requires=requirements,
test_suite='tests', test_suite='tests',
tests_require=test_requirements tests_require=test_requirements
......
...@@ -2,7 +2,8 @@ ...@@ -2,7 +2,8 @@
context_dir="./context" context_dir="./context"
dockerfile="build_runner_image.docker" dockerfile="build_runner_image.docker"
runner_tag="gts2_client_ci" runner_version="latest"
runner_tag="gts2_client_ci:$runner_version"
gitlab_runner="gts2_client_gitlab_CI_runner" gitlab_runner="gts2_client_gitlab_CI_runner"
echo "Preparing gts2 credentials file" echo "Preparing gts2 credentials file"
...@@ -37,5 +38,6 @@ sudo docker exec -it ${gitlab_runner} /bin/bash -c "export RUNNER_EXECUTOR=docke ...@@ -37,5 +38,6 @@ sudo docker exec -it ${gitlab_runner} /bin/bash -c "export RUNNER_EXECUTOR=docke
--run-untagged=true \ --run-untagged=true \
--locked=true \ --locked=true \
--tag-list gts2_client \ --tag-list gts2_client \
--docker-pull-policy='never' \
--description '${runner_name}' \ --description '${runner_name}' \
--docker-image '${runner_tag}:latest' " --docker-image '${runner_tag}' "
FROM centos:7 FROM centos:7
RUN yum update -y && \ RUN yum update -y && \
yum install -y wget vim bzip2 gcc gcc-c++ make libgl1-mesa-glx mesa-libGL qt5-qtbase-gui git texlive yum install -y wget vim bzip2 gcc gcc-c++ make libgl1-mesa-glx mesa-libGL qt5-qtbase-gui git texlive
RUN /bin/bash -i -c "wget https://repo.continuum.io/archive/Anaconda3-4.3.1-Linux-x86_64.sh && \ ENV anaconda_dl='Anaconda3-2019.07-Linux-x86_64.sh'
bash ./Anaconda3-4.3.1-Linux-x86_64.sh -b && \ RUN /bin/bash -i -c "wget https://repo.continuum.io/archive/$anaconda_dl && \
rm -f /root/Anaconda3-4.3.1-Linux-x86_64.sh" bash ./$anaconda_dl -b && \
rm -f /root/$anaconda_dl"
RUN /bin/bash -i -c "source ~/anaconda3/bin/activate && \ RUN /bin/bash -i -c "source ~/anaconda3/bin/activate && \
conda install --yes -c conda-forge gdal 'icu=58.*' lxml pyqt coverage && \ conda config --add channels conda-forge && \
pip install netCDF4" conda install --yes -c conda-forge proj gdal=3.1.0 lxml pyqt netcdf4 coverage libiconv imageio"
RUN mkdir -p /home/tmp/ RUN mkdir -p /home/tmp/
COPY credentials_gts2_client /root/ COPY credentials_gts2_client /root/
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment