README.rst 8.56 KB
Newer Older
Daniel Scheffler's avatar
Daniel Scheffler committed
1
2
3
=========================================================================================
gms_preprocessing - Spatial and spectral homogenization of satellite remote sensing data.
=========================================================================================
4

Daniel Scheffler's avatar
Daniel Scheffler committed
5
6
7
The goal of the gms_preprocessing Python library is to provide a fully automatic
pre-precessing pipeline for spatial and spectral fusion (i.e., homogenization)
of multispectral satellite image data. Currently it offers compatibility to
Daniel Scheffler's avatar
Daniel Scheffler committed
8
Landsat-5, Landsat-7, Landsat-8, Sentinel-2A and Sentinel-2B.
9

10
* Free software: GNU General Public License v3 or later (GPLv3+) (`license details <https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/blob/master/LICENSE>`_)
11
* Documentation: https://geomultisens.gitext-pages.gfz-potsdam.de/gms_preprocessing/doc/
Daniel Scheffler's avatar
Daniel Scheffler committed
12
* Code history: Release notes for the current and earlier versions of gms_preprocessing can be found `here <./HISTORY.rst>`_.
Daniel Scheffler's avatar
Daniel Scheffler committed
13
* OS compatibility: Linux
Daniel Scheffler's avatar
Daniel Scheffler committed
14
15
16
17
18


Status
------

19
20
.. .. image:: https://img.shields.io/travis/geomultisens/gms_preprocessing.svg
        :target: https://travis-ci.org/geomultisens/gms_preprocessing
21

22
23
.. .. image:: https://readthedocs.org/projects/gms_preprocessing/badge/?version=latest
        :target: https://gms_preprocessing.readthedocs.io/en/latest/?badge=latest
24
25
        :alt: Documentation Status

26
27
.. .. image:: https://pyup.io/repos/github/geomultisens/gms_preprocessing/shield.svg
     :target: https://pyup.io/repos/github/geomultisens/gms_preprocessing/
28
29
     :alt: Updates

Daniel Scheffler's avatar
Daniel Scheffler committed
30
.. image:: https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/badges/master/pipeline.svg
31
32
        :target: https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/commits/master
.. image:: https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/badges/master/coverage.svg
33
        :target: https://geomultisens.gitext-pages.gfz-potsdam.de/gms_preprocessing/coverage/
Daniel Scheffler's avatar
Daniel Scheffler committed
34
.. .. image:: https://img.shields.io/pypi/v/gms_preprocessing.svg
Daniel Scheffler's avatar
Daniel Scheffler committed
35
        :target: https://pypi.python.org/pypi/gms_preprocessing
Daniel Scheffler's avatar
Daniel Scheffler committed
36
.. .. image:: https://img.shields.io/pypi/l/gms_preprocessing.svg
Daniel Scheffler's avatar
Daniel Scheffler committed
37
        :target: https://gitext.gfz-potsdam.de/danschef/gms_preprocessing/blob/master/LICENSE
Daniel Scheffler's avatar
Daniel Scheffler committed
38
.. .. image:: https://img.shields.io/pypi/pyversions/gms_preprocessing.svg
Daniel Scheffler's avatar
Daniel Scheffler committed
39
        :target: https://img.shields.io/pypi/pyversions/gms_preprocessing.svg
40

41
See also the latest coverage_ report and the nosetests_ HTML report.
42
43
44
45

Features
--------

Daniel Scheffler's avatar
Daniel Scheffler committed
46
47
48
49
50
Level-1 processing:
^^^^^^^^^^^^^^^^^^^

* data import and  metadata homogenization (compatibility: Landsat-5/7/8, Sentinel-2A/2B)
* equalization of acquisition- and illumination geometry
Daniel Scheffler's avatar
Daniel Scheffler committed
51
52
* atmospheric correction (using `SICOR <https://gitext.gfz-potsdam.de/EnMAP/sicor>`_)
* correction of geometric errors (using `AROSICS <https://gitext.gfz-potsdam.de/danschef/arosics>`_)
Daniel Scheffler's avatar
Daniel Scheffler committed
53
54
55
56
57

Level-2 processing:
^^^^^^^^^^^^^^^^^^^

* spatial homogenization
58
* spectral homogenization (using `SpecHomo <https://gitext.gfz-potsdam.de/geomultisens/spechomo>`_)
Daniel Scheffler's avatar
Daniel Scheffler committed
59
60
61
62
63
64
65
66
67
68
69
* estimation of accuracy layers

=> application oriented analysis dataset


Getting started
---------------

Usage via WebApp
^^^^^^^^^^^^^^^^

Daniel Scheffler's avatar
Daniel Scheffler committed
70
71
72
73
The recommended way to use gms_preprocessing is to setup the WebApp (see the
gms-vis_ repository) providing a UI for GeoMultiSens. Using this UI, existing
satellite data can be explored, filtered and selected for processing. New data
homogenization jobs (using gms_preprocessing) can be defined and started. All
Daniel Scheffler's avatar
Daniel Scheffler committed
74
75
76
configuration parameters of gms_preprocessing are accessible in the UI.


77
.. image:: https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/-/blob/master/docs/images/webapp_screenshot_900x497.png
Daniel Scheffler's avatar
Daniel Scheffler committed
78
79
80
    :width: 900 px
    :height: 497 px
    :scale: 100 %
81
    :alt: WebApp Screenshot
Daniel Scheffler's avatar
Daniel Scheffler committed
82
83
84
85
86


Usage via console interface
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Daniel Scheffler's avatar
Daniel Scheffler committed
87
Homogenization jobs can also be created and started using the command line
88
interface. Documentation can be found `here <https://geomultisens.gitext-pages.gfz-potsdam.de/gms_preprocessing/doc/usage.html#gms-preprocessing-command-line-interface>`_.
Daniel Scheffler's avatar
Daniel Scheffler committed
89

Daniel Scheffler's avatar
Daniel Scheffler committed
90
91
Here is a small example:

92
.. code:: bash
Daniel Scheffler's avatar
Daniel Scheffler committed
93

Daniel Scheffler's avatar
Daniel Scheffler committed
94
95
    # start the job with the ID 123456 and override default configuration with the given one.
    >>> run_gms.py jobid 123456 --json_config /path/to/my/config.json
Daniel Scheffler's avatar
Daniel Scheffler committed
96
97
98

There is a default configuration file, called `options_default.json <https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/blob/master/gms_preprocessing/options/options_default.json>`_.
This file contains the documentation for all the available configuration
Daniel Scheffler's avatar
Daniel Scheffler committed
99
100
101
102
103
104
parameters.


Usage via Python API
^^^^^^^^^^^^^^^^^^^^

Daniel Scheffler's avatar
Daniel Scheffler committed
105
There is also a Python API that allows to setup and start homogenization jobs
Daniel Scheffler's avatar
Daniel Scheffler committed
106
107
108
109
110
111
112
by a Python function call.

This is an example:

.. code:: python

    from gms_preprocessing import ProcessController
Daniel Scheffler's avatar
Daniel Scheffler committed
113

Daniel Scheffler's avatar
Daniel Scheffler committed
114
115
116
117
    configuration = dict(
        db_host='localhost',
        CPUs=20
        )
Daniel Scheffler's avatar
Daniel Scheffler committed
118

Daniel Scheffler's avatar
Daniel Scheffler committed
119
120
    PC = ProcessController(jobID=123456, **configuration)
    PC.run_all_processors()
Daniel Scheffler's avatar
Daniel Scheffler committed
121

Daniel Scheffler's avatar
Daniel Scheffler committed
122
123
Possible configuration arguments can be found `here <https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/blob/master/gms_preprocessing/options/options_schema.py>`_.

124

Daniel Scheffler's avatar
Daniel Scheffler committed
125
126
127
128

Installation
------------

129
130
131
132
133
gms_preprocessing depends on some open source packages which are usually installed without problems by the automatic install
routine. However, for some projects, we strongly recommend resolving the dependency before the automatic installer
is run. This approach avoids problems with conflicting versions of the same software.
Using conda_, the recommended approach is:

134
.. code:: bash
135

Daniel Scheffler's avatar
Daniel Scheffler committed
136
  # create virtual environment for gms_preprocessing, this is optional
137
  conda create -c conda-forge --name gms_preprocessing python=3
138
  conda activate gms_preprocessing
139

140
  # install some dependencies that cause trouble when installed via pip
141
  conda install -c conda-forge numpy gdal scikit-image pyproj geopandas ipython matplotlib cartopy scikit-learn=0.23.2 shapely pyhdf python-fmask holoviews
142

143
144
  # install not pip-installable deps of arosics
  conda install -c conda-forge pyfftw pykrige
145

146
147
  # install not pip-installable deps of sicor
  conda install -c conda-forge glymur pygrib cachetools pyhdf h5py pytables llvmlite numba
Daniel Scheffler's avatar
Daniel Scheffler committed
148

149
150
151
152
  # install gms_preprocessing
  git clone https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing.git
  cd gms_preprocessing
  pip install .
Daniel Scheffler's avatar
Daniel Scheffler committed
153
154


Daniel Scheffler's avatar
Daniel Scheffler committed
155
156
To enable lock functionality (needed for CPU / memory / disk IO management), install redis-server_:

157
.. code-block:: bash
Daniel Scheffler's avatar
Daniel Scheffler committed
158
159
160

    sudo apt-get install redis-server

Daniel Scheffler's avatar
Daniel Scheffler committed
161

162
163
164
165
166
167
168
History / Changelog
-------------------

You can find the protocol of recent changes in the gms_preprocessing package
`here <https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/-/blob/master/HISTORY.rst>`__.


Daniel Scheffler's avatar
Daniel Scheffler committed
169
170
License
-------
Daniel Scheffler's avatar
Daniel Scheffler committed
171

Daniel Scheffler's avatar
Daniel Scheffler committed
172
173
gms_preprocessing - Spatial and spectral homogenization of satellite remote sensing data.

174
Copyright 2020 Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences, Potsdam, Germany
Daniel Scheffler's avatar
Daniel Scheffler committed
175

Daniel Scheffler's avatar
Daniel Scheffler committed
176
177
178
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
Daniel Scheffler's avatar
Daniel Scheffler committed
179
180
(at your option) any later version. This program is distributed in the hope
that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
Daniel Scheffler's avatar
Daniel Scheffler committed
181
warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
Daniel Scheffler's avatar
Daniel Scheffler committed
182
183
GNU General Public License for more details. You should have received a copy
of the GNU General Public License along with this program.
Daniel Scheffler's avatar
Daniel Scheffler committed
184
If not, see <http://www.gnu.org/licenses/>.
185
186


Daniel Scheffler's avatar
Daniel Scheffler committed
187
188
189
190
191
Contact
-------

.. line-block::

Daniel Scheffler's avatar
Daniel Scheffler committed
192
193
    Daniel Scheffler
    eMail: daniel.scheffler@gfz-potsdam.de
Daniel Scheffler's avatar
Daniel Scheffler committed
194

Daniel Scheffler's avatar
Daniel Scheffler committed
195
196
197
198
199
    Helmholtz Centre Potsdam GFZ German Research Centre for Geoscienes
    Section 1.4 Remote Sensing
    Telegrafenberg
    14473 Potsdam
    Germany
Daniel Scheffler's avatar
Daniel Scheffler committed
200
201


202
Credits
203
-------
204

205
206
207
208
The development of the gms_preprocessing package was funded by the German Federal Ministry of Education and Research
(BMBF, project grant code: 01 IS 14 010 A-C).

The package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.
209

210
Landsat-5/7/8 satellite data and SRTM/ASTER digital elevation models have been provided by the US Geological
Daniel Scheffler's avatar
Daniel Scheffler committed
211
212
Survey. Sentinel-2 data have been provided by ESA.

213
214
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
215
216
.. _coverage: https://geomultisens.gitext-pages.gfz-potsdam.de/gms_preprocessing/coverage/
.. _nosetests: https://geomultisens.gitext-pages.gfz-potsdam.de/gms_preprocessing/nosetests_reports/nosetests.html
217
.. _conda: https://conda.io/docs/
Daniel Scheffler's avatar
Daniel Scheffler committed
218
.. _redis-server: https://www.rosehosting.com/blog/how-to-install-configure-and-use-redis-on-ubuntu-16-04/
Daniel Scheffler's avatar
Daniel Scheffler committed
219
.. _gms-vis: https://gitext.gfz-potsdam.de/geomultisens/gms-vis