py_tools_ds issueshttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues2022-02-10T17:54:21+01:00https://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/20Wrong output of raster2polygon in case exact=False.2022-02-10T17:54:21+01:00Daniel SchefflerWrong output of raster2polygon in case exact=False.```python
from geoarray import GeoArray
from py_tools_ds.geo.raster.conversion import raster2polygon
gA = GeoArray('mask.tif')
raster2polygon(gA.mask_nodata.astype(np.uint8), gA.gt, gA.prj, exact=False)
```
returns:
![grafik](/uploads...```python
from geoarray import GeoArray
from py_tools_ds.geo.raster.conversion import raster2polygon
gA = GeoArray('mask.tif')
raster2polygon(gA.mask_nodata.astype(np.uint8), gA.gt, gA.prj, exact=False)
```
returns:
![grafik](/uploads/1be48a1b052d7b3500a9c1a57623016b/grafik.png)
where the input image clearly has valid values on the right:
![grafik](/uploads/bb7235ff6697809f0cb08cc3d2b04c83/grafik.png)Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/19raster2polygon slow for input masks that contain many small raster polygons.2021-12-03T22:13:35+01:00Daniel Schefflerraster2polygon slow for input masks that contain many small raster polygons.Might be fixed by [GDALSieveFilter](https://gdal.org/api/gdal_alg.html#_CPPv415GDALSieveFilter15GDALRasterBandH15GDALRasterBandH15GDALRasterBandHiiPPc16GDALProgressFuncPv) like:
```python
from osgeo import gdal
Image = gdal.Open('SomeI...Might be fixed by [GDALSieveFilter](https://gdal.org/api/gdal_alg.html#_CPPv415GDALSieveFilter15GDALRasterBandH15GDALRasterBandH15GDALRasterBandHiiPPc16GDALProgressFuncPv) like:
```python
from osgeo import gdal
Image = gdal.Open('SomeImageName.tif', 1) # open image in read-write mode
Band = Image.GetRasterBand(1)
gdal.SieveFilter(srcBand=Band, maskBand=None, dstBand=Band, threshold=100, connectedness=8, callback=gdal.TermProgress_nocb)
del Image, Band # close the datasets.
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/18Move tests to pytest.2021-12-02T21:26:45+01:00Daniel SchefflerMove tests to pytest.Nose is not maintained anymore and now becomes incompatible with recent Python versions, see https://git.gfz-potsdam.de/danschef/py_tools_ds/-/jobs/104240. Pytest should be a good alternative.Nose is not maintained anymore and now becomes incompatible with recent Python versions, see https://git.gfz-potsdam.de/danschef/py_tools_ds/-/jobs/104240. Pytest should be a good alternative.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/17initially set up SQA2021-09-28T13:24:42+02:00Maximilian Dollinginitially set up SQA- [ ] create config
- [ ] add to CI- [ ] create config
- [ ] add to CIMaximilian DollingMaximilian Dolling2021-06-30https://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/16WKT2EPSG() returns None in case of bound CRS.2020-12-15T16:45:53+01:00Daniel SchefflerWKT2EPSG() returns None in case of bound CRS.```python
wkt = 'PROJCS["ETRS89-extended / LAEA Europe",GEOGCS["ETRS89",DATUM["European_Terrestrial_Reference_System_1989",SPHEROID["GRS 1980",6378137,298.257222101,AUTHORITY["EPSG","7019"]],TOWGS84[0,0,0,0,0,0,0],AUTHORITY["EPSG","6258"...```python
wkt = 'PROJCS["ETRS89-extended / LAEA Europe",GEOGCS["ETRS89",DATUM["European_Terrestrial_Reference_System_1989",SPHEROID["GRS 1980",6378137,298.257222101,AUTHORITY["EPSG","7019"]],TOWGS84[0,0,0,0,0,0,0],AUTHORITY["EPSG","6258"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4258"]],PROJECTION["Lambert_Azimuthal_Equal_Area"],PARAMETER["latitude_of_center",52],PARAMETER["longitude_of_center",10],PARAMETER["false_easting",4321000],PARAMETER["false_northing",3210000],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Northing",NORTH],AXIS["Easting",EAST],AUTHORITY["EPSG","3035"]]'
WKT2EPSG(wkt)
```
`None`Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/15boxObj.get_coordArray_MapXY() returns faulty y-coordinate array2020-12-07T22:23:24+01:00Daniel SchefflerboxObj.get_coordArray_MapXY() returns faulty y-coordinate arrayE.g., for an array with GeoTransform (365985.0, 30.0, 0.0, 5916615.0, 0.0, -30.0) and EPSG 32633, the following y-coordinate array is returned:
![image](/uploads/5352c523ee1117468557bf0b907f97f5/image.png)
The upper left coordinate sho...E.g., for an array with GeoTransform (365985.0, 30.0, 0.0, 5916615.0, 0.0, -30.0) and EPSG 32633, the following y-coordinate array is returned:
![image](/uploads/5352c523ee1117468557bf0b907f97f5/image.png)
The upper left coordinate should be equal to the coordinates in the GeoTransform and the array is y-flipped.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/14calc_ssim() does not work for input data with completely different data range...2020-11-02T18:35:11+01:00Daniel Schefflercalc_ssim() does not work for input data with completely different data ranges or data ranges exceeding np.int16.In such cases, the output values of calc_ssim are wrong.
Proposed solution:
* [ ] remove datatype downcasting
* [ ] add data normalization
* [ ] mask nodata valuesIn such cases, the output values of calc_ssim are wrong.
Proposed solution:
* [ ] remove datatype downcasting
* [ ] add data normalization
* [ ] mask nodata valuesDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/13TypeError when a MultiPolygon instance is passed to fill_holes_within_poly()2020-10-30T16:57:17+01:00Daniel SchefflerTypeError when a MultiPolygon instance is passed to fill_holes_within_poly()```python
from shapely.geometry import Polygon, MultiPolygon
from py_tools_ds.geo.vector.topology import fill_holes_within_poly
polygon_1 = Polygon([(0, 0), (5, 0), (5, 5), (0, 5)]) # large poly
polygon_2 = Polygon([(1, 1), (1, 2), (2,...```python
from shapely.geometry import Polygon, MultiPolygon
from py_tools_ds.geo.vector.topology import fill_holes_within_poly
polygon_1 = Polygon([(0, 0), (5, 0), (5, 5), (0, 5)]) # large poly
polygon_2 = Polygon([(1, 1), (1, 2), (2, 2), (2, 1)]) # small poly within large poly
multipolygon = MultiPolygon([polygon_1, polygon_2])
fill_holes_within_poly(multipolygon)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-11-8ecef912bc33> in <module>
6 multipolygon = MultiPolygon([polygon_1, polygon_2])
7
----> 8 fill_holes_within_poly(multipolygon)
~/miniconda3/envs/arosics/lib/python3.8/site-packages/py_tools_ds/geo/vector/topology.py in fill_holes_within_poly(poly)
187 lambda GDF_row: Polygon(np.swapaxes(np.array(GDF_row.geometry.exterior.coords.xy), 0, 1)).area, axis=1)
188
--> 189 largest_poly_filled = gdf.loc[gdf['area_filled'].idxmax()]['geometry']
190
191 # return the outer boundary of the largest polygon
~/miniconda3/envs/arosics/lib/python3.8/site-packages/pandas/core/series.py in idxmax(self, axis, skipna, *args, **kwargs)
2166 """
2167 skipna = nv.validate_argmax_with_skipna(skipna, args, kwargs)
-> 2168 i = nanops.nanargmax(self._values, skipna=skipna)
2169 if i == -1:
2170 return np.nan
~/miniconda3/envs/arosics/lib/python3.8/site-packages/pandas/core/nanops.py in _f(*args, **kwargs)
64 if any(self.check(obj) for obj in obj_iter):
65 f_name = f.__name__.replace("nan", "")
---> 66 raise TypeError(
67 f"reduction operation '{f_name}' not allowed for this dtype"
68 )
TypeError: reduction operation 'argmax' not allowed for this dtype
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/12DeprecationWarning: pyproj.transformer.transform() is deprecated.2020-05-08T00:53:56+02:00Daniel SchefflerDeprecationWarning: pyproj.transformer.transform() is deprecated.pyproj.transformer.transform() since pyproj v2.0.0.
See here how to replace it:
https://pyproj4.github.io/pyproj/stable/gotchas.html#upgrading-to-pyproj-2-from-pyproj-1pyproj.transformer.transform() since pyproj v2.0.0.
See here how to replace it:
https://pyproj4.github.io/pyproj/stable/gotchas.html#upgrading-to-pyproj-2-from-pyproj-1Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/11DeprecationWarning: compare_ssim moved2020-10-27T12:05:28+01:00Daniel SchefflerDeprecationWarning: compare_ssim moved/opt/anaconda3/envs/arosics/lib/python3.8/site-packages/py_tools_ds/similarity/raster.py:44: UserWarning:
DEPRECATED: skimage.measure.compare_ssim has been moved to skimage.metrics.structural_similarity. It will be removed from skimage..../opt/anaconda3/envs/arosics/lib/python3.8/site-packages/py_tools_ds/similarity/raster.py:44: UserWarning:
DEPRECATED: skimage.measure.compare_ssim has been moved to skimage.metrics.structural_similarity. It will be removed from skimage.measure in version 0.18.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/10round_shapelyPoly_coords() is incompatible with shapely 1.7.02020-03-26T23:30:10+01:00Daniel Schefflerround_shapelyPoly_coords() is incompatible with shapely 1.7.0Cause for issue https://gitext.gfz-potsdam.de/danschef/arosics/-/issues/34.Cause for issue https://gitext.gfz-potsdam.de/danschef/arosics/-/issues/34.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/9Install some non pip-installable packages on demand.2020-01-07T18:54:18+01:00Daniel SchefflerInstall some non pip-installable packages on demand.This may be useful to to avoid complicated installation processes in downstream packages that not even need these packages.This may be useful to to avoid complicated installation processes in downstream packages that not even need these packages.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/8Exception: dlopen: cannot load any more object with static TLS2020-09-25T17:13:35+02:00Daniel SchefflerException: dlopen: cannot load any more object with static TLSIt seems like there is a maximum number of objects with static TLS. In case py_tools_ds.geo.raster.reproject is imported from sicor, this number is exceeded. This causes the following exception during import of sicor:
```python
Tracebac...It seems like there is a maximum number of objects with static TLS. In case py_tools_ds.geo.raster.reproject is imported from sicor, this number is exceeded. This causes the following exception during import of sicor:
```python
Traceback (most recent call last):
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/unittest/case.py", line 59, in testPartExecutor
yield
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/unittest/case.py", line 615, in run
testMethod()
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/site-packages/nose/failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/imp.py", line 234, in load_module
return load_source(name, filename, file)
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/imp.py", line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 696, in _load
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/gfz-fe/scheffler/python/EnPT/tests/test_controller.py", line 14, in <module>
from enpt.execution.controller import EnPT_Controller
File "/home/gfz-fe/scheffler/python/EnPT/enpt/execution/controller.py", line 14, in <module>
from ..options.config import EnPTConfig
File "/home/gfz-fe/scheffler/python/EnPT/enpt/options/config.py", line 20, in <module>
import sicor
File "/home/gfz-fe/scheffler/python_deployed/sicor/sicor/__init__.py", line 6, in <module>
from .sicor_ac import ac
File "/home/gfz-fe/scheffler/python_deployed/sicor/sicor/sicor_ac.py", line 46, in <module>
from .sensors.S2MSI import S2Image, GranuleDEM
File "/home/gfz-fe/scheffler/python_deployed/sicor/sicor/sensors/S2MSI/__init__.py", line 3, in <module>
from .S2Image import S2Image
File "/home/gfz-fe/scheffler/python_deployed/sicor/sicor/sensors/S2MSI/S2Image/__init__.py", line 1, in <module>
from .S2Image import S2Image
File "/home/gfz-fe/scheffler/python_deployed/sicor/sicor/sensors/S2MSI/S2Image/S2Image.py", line 25, in <module>
from geoarray import GeoArray
File "/home/gfz-fe/scheffler/python_deployed/geoarray/geoarray/__init__.py", line 7, in <module>
from .baseclasses import GeoArray # noqa: E402
File "/home/gfz-fe/scheffler/python_deployed/geoarray/geoarray/baseclasses.py", line 28, in <module>
from py_tools_ds.geo.coord_calc import get_corner_coordinates
File "/home/gfz-fe/scheffler/python_deployed/py_tools_ds/py_tools_ds/geo/__init__.py", line 1, in <module>
from . import raster
File "/home/gfz-fe/scheffler/python_deployed/py_tools_ds/py_tools_ds/geo/raster/__init__.py", line 5, in <module>
from . import reproject
File "/home/gfz-fe/scheffler/python_deployed/py_tools_ds/py_tools_ds/geo/raster/reproject.py", line 23, in <module>
from pyresample.bilinear import resample_bilinear
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/site-packages/pyresample/__init__.py", line 27, in <module>
from pyresample import image # noqa
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/site-packages/pyresample/image.py", line 24, in <module>
from pyresample import geometry, grid, kd_tree, bilinear
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/site-packages/pyresample/kd_tree.py", line 29, in <module>
from pykdtree.kdtree import KDTree
Exception: dlopen: cannot load any more object with static TLS
```
So far, I investigated pyresample and rasterio as libs related to that (with static TLS?).
Possible solutions:
* put a `from pykdtree.kdtree import KDTree` before the imports of pyresample and rasterio
* import pyresample BEFORE rasterio
* move rasterio imports to function locals to avoid rasterio import if it is not neededDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/7geotransform2mapinfo() and mapinfo2geotransform() incompatible with ETRS/LAEA...2019-02-27T13:10:07+01:00Daniel Schefflergeotransform2mapinfo() and mapinfo2geotransform() incompatible with ETRS/LAEA projection.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/5[Enhancement] Add possibility to use SensorMapGeometryTransformer for 3D arra...2019-01-23T14:09:04+01:00Daniel Scheffler[Enhancement] Add possibility to use SensorMapGeometryTransformer for 3D array data with per-band coordinate layers.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/4py_tools_ds.geo.raster.warp_ndarray() freezes / crashes in case of very large...2018-09-17T14:08:33+02:00Daniel Schefflerpy_tools_ds.geo.raster.warp_ndarray() freezes / crashes in case of very large input rastersCrashes at line 430:
```python
res_ds = gdal_Warp(
'', in_ds, format='MEM',
dstSRS=get_SRS(out_prj),
outputType=get_GDT(out_dtype) if out_dtype else get_GDT(in_dtype_np),
xRes=out_gsd[0],
yRes=out...Crashes at line 430:
```python
res_ds = gdal_Warp(
'', in_ds, format='MEM',
dstSRS=get_SRS(out_prj),
outputType=get_GDT(out_dtype) if out_dtype else get_GDT(in_dtype_np),
xRes=out_gsd[0],
yRes=out_gsd[1],
outputBounds=out_bounds,
outputBoundsSRS=get_SRS(out_bounds_prj),
width=out_XYdims[0],
height=out_XYdims[1],
resampleAlg=rspAlg,
srcNodata=in_nodata,
dstNodata=out_nodata,
srcAlpha=in_alpha,
dstAlpha=out_alpha,
options=options if options else [],
warpOptions=warpOptions or [],
transformerOptions=transformerOptions or [],
targetAlignedPixels=targetAlignedPixels,
tps=True if gcpList else False,
polynomialOrder=polynomialOrder,
warpMemoryLimit=warpMemoryLimit,
callback=ProgressBar(prefix='Warping progress ', timeout=None) if progress and not q else None,
callback_data=[0],
errorThreshold=0.125, # this is needed to get exactly the same output like the console version of GDAL warp
)
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/3TypeError: type() argument 1 must be str, not type2017-09-25T18:16:39+02:00Daniel SchefflerTypeError: type() argument 1 must be str, not type```
Traceback (most recent call last):
File "/tmp/flink_plan504207454/plan.py", line 10, in <module>
from gms.io.GMSPreprocessInputFormat import
GmsPreprocessInputFormat, SpillInputFormat
File
"/home/hub-inf/python_GFZ/python/lib...```
Traceback (most recent call last):
File "/tmp/flink_plan504207454/plan.py", line 10, in <module>
from gms.io.GMSPreprocessInputFormat import
GmsPreprocessInputFormat, SpillInputFormat
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/gms/io/GMSPreprocessInputFormat.py",
line 7, in <module>
import gms_preprocessing.config as CFG
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/gms_preprocessing/__init__.py",
line 9, in <module>
from . import algorithms
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/gms_preprocessing/algorithms/__init__.py",
line 10, in <module>
from . import GEOPROCESSING
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/gms_preprocessing/algorithms/GEOPROCESSING.py",
line 54, in <module>
from geoarray import GeoArray
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/geoarray/__init__.py",
line 7, in <module>
from .baseclasses import GeoArray # noqa: E402
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/geoarray/baseclasses.py",
line 27, in <module>
from py_tools_ds.geo.coord_calc import get_corner_coordinates
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/py_tools_ds/geo/__init__.py",
line 2, in <module>
from . import vector
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/py_tools_ds/geo/vector/__init__.py",
line 3, in <module>
from . import geometry
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/py_tools_ds/geo/vector/geometry.py",
line 6, in <module>
from .conversion import shapelyImPoly_to_shapelyMapPoly,
get_boxImXY_from_shapelyPoly, \
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/py_tools_ds/geo/vector/conversion.py",
line 23, in <module>
_T_polygon = TypeVar(Polygon)
File "/home/hub-inf/python_GFZ/python/lib/python3.5/typing.py", line
406, in __new__
self = super().__new__(cls, name, (Final,), {}, _root=True)
File "/home/hub-inf/python_GFZ/python/lib/python3.5/typing.py", line
107, in __new__
return super().__new__(cls, name, bases, namespace)
TypeError: type() argument 1 must be str, not type
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/2ERROR:rasterio._gdal:CPLE_UserInterrupt in b'User terminated' in case polygon...2021-09-28T13:27:35+02:00Daniel SchefflerERROR:rasterio._gdal:CPLE_UserInterrupt in b'User terminated' in case polygonizing process exceeds timeoutThis error is raised as a warning but should be handled internally instead.
Context:
Calculating actual data corner coordinates for image to be shifted...
Polygonize progress |==========================------------------------| 52.0...This error is raised as a warning but should be handled internally instead.
Context:
Calculating actual data corner coordinates for image to be shifted...
Polygonize progress |==========================------------------------| 52.0% Complete => 0:00:02
ERROR:rasterio._gdal:CPLE_UserInterrupt in b'User terminated'Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/1Some metadata in the xml files is not parsed to the corrected files.2017-03-31T18:01:50+02:00Hannes DiedrichSome metadata in the xml files is not parsed to the corrected files.We should talk about that at some point (easer than write it down here, maybe todo on my side)We should talk about that at some point (easer than write it down here, maybe todo on my side)Daniel SchefflerDaniel Scheffler