py_tools_ds issueshttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues2020-12-07T22:23:24+01:00https://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/15boxObj.get_coordArray_MapXY() returns faulty y-coordinate array2020-12-07T22:23:24+01:00Daniel SchefflerboxObj.get_coordArray_MapXY() returns faulty y-coordinate arrayE.g., for an array with GeoTransform (365985.0, 30.0, 0.0, 5916615.0, 0.0, -30.0) and EPSG 32633, the following y-coordinate array is returned:
![image](/uploads/5352c523ee1117468557bf0b907f97f5/image.png)
The upper left coordinate sho...E.g., for an array with GeoTransform (365985.0, 30.0, 0.0, 5916615.0, 0.0, -30.0) and EPSG 32633, the following y-coordinate array is returned:
![image](/uploads/5352c523ee1117468557bf0b907f97f5/image.png)
The upper left coordinate should be equal to the coordinates in the GeoTransform and the array is y-flipped.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/14calc_ssim() does not work for input data with completely different data range...2020-11-02T18:35:11+01:00Daniel Schefflercalc_ssim() does not work for input data with completely different data ranges or data ranges exceeding np.int16.In such cases, the output values of calc_ssim are wrong.
Proposed solution:
* [ ] remove datatype downcasting
* [ ] add data normalization
* [ ] mask nodata valuesIn such cases, the output values of calc_ssim are wrong.
Proposed solution:
* [ ] remove datatype downcasting
* [ ] add data normalization
* [ ] mask nodata valuesDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/8Exception: dlopen: cannot load any more object with static TLS2020-09-25T17:13:35+02:00Daniel SchefflerException: dlopen: cannot load any more object with static TLSIt seems like there is a maximum number of objects with static TLS. In case py_tools_ds.geo.raster.reproject is imported from sicor, this number is exceeded. This causes the following exception during import of sicor:
```python
Tracebac...It seems like there is a maximum number of objects with static TLS. In case py_tools_ds.geo.raster.reproject is imported from sicor, this number is exceeded. This causes the following exception during import of sicor:
```python
Traceback (most recent call last):
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/unittest/case.py", line 59, in testPartExecutor
yield
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/unittest/case.py", line 615, in run
testMethod()
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/site-packages/nose/failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/imp.py", line 234, in load_module
return load_source(name, filename, file)
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/imp.py", line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 696, in _load
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/gfz-fe/scheffler/python/EnPT/tests/test_controller.py", line 14, in <module>
from enpt.execution.controller import EnPT_Controller
File "/home/gfz-fe/scheffler/python/EnPT/enpt/execution/controller.py", line 14, in <module>
from ..options.config import EnPTConfig
File "/home/gfz-fe/scheffler/python/EnPT/enpt/options/config.py", line 20, in <module>
import sicor
File "/home/gfz-fe/scheffler/python_deployed/sicor/sicor/__init__.py", line 6, in <module>
from .sicor_ac import ac
File "/home/gfz-fe/scheffler/python_deployed/sicor/sicor/sicor_ac.py", line 46, in <module>
from .sensors.S2MSI import S2Image, GranuleDEM
File "/home/gfz-fe/scheffler/python_deployed/sicor/sicor/sensors/S2MSI/__init__.py", line 3, in <module>
from .S2Image import S2Image
File "/home/gfz-fe/scheffler/python_deployed/sicor/sicor/sensors/S2MSI/S2Image/__init__.py", line 1, in <module>
from .S2Image import S2Image
File "/home/gfz-fe/scheffler/python_deployed/sicor/sicor/sensors/S2MSI/S2Image/S2Image.py", line 25, in <module>
from geoarray import GeoArray
File "/home/gfz-fe/scheffler/python_deployed/geoarray/geoarray/__init__.py", line 7, in <module>
from .baseclasses import GeoArray # noqa: E402
File "/home/gfz-fe/scheffler/python_deployed/geoarray/geoarray/baseclasses.py", line 28, in <module>
from py_tools_ds.geo.coord_calc import get_corner_coordinates
File "/home/gfz-fe/scheffler/python_deployed/py_tools_ds/py_tools_ds/geo/__init__.py", line 1, in <module>
from . import raster
File "/home/gfz-fe/scheffler/python_deployed/py_tools_ds/py_tools_ds/geo/raster/__init__.py", line 5, in <module>
from . import reproject
File "/home/gfz-fe/scheffler/python_deployed/py_tools_ds/py_tools_ds/geo/raster/reproject.py", line 23, in <module>
from pyresample.bilinear import resample_bilinear
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/site-packages/pyresample/__init__.py", line 27, in <module>
from pyresample import image # noqa
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/site-packages/pyresample/image.py", line 24, in <module>
from pyresample import geometry, grid, kd_tree, bilinear
File "/home/gfz-fe/pygfz/envs/enpt/lib/python3.7/site-packages/pyresample/kd_tree.py", line 29, in <module>
from pykdtree.kdtree import KDTree
Exception: dlopen: cannot load any more object with static TLS
```
So far, I investigated pyresample and rasterio as libs related to that (with static TLS?).
Possible solutions:
* put a `from pykdtree.kdtree import KDTree` before the imports of pyresample and rasterio
* import pyresample BEFORE rasterio
* move rasterio imports to function locals to avoid rasterio import if it is not neededDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/7geotransform2mapinfo() and mapinfo2geotransform() incompatible with ETRS/LAEA...2019-02-27T13:10:07+01:00Daniel Schefflergeotransform2mapinfo() and mapinfo2geotransform() incompatible with ETRS/LAEA projection.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/4py_tools_ds.geo.raster.warp_ndarray() freezes / crashes in case of very large...2018-09-17T14:08:33+02:00Daniel Schefflerpy_tools_ds.geo.raster.warp_ndarray() freezes / crashes in case of very large input rastersCrashes at line 430:
```python
res_ds = gdal_Warp(
'', in_ds, format='MEM',
dstSRS=get_SRS(out_prj),
outputType=get_GDT(out_dtype) if out_dtype else get_GDT(in_dtype_np),
xRes=out_gsd[0],
yRes=out...Crashes at line 430:
```python
res_ds = gdal_Warp(
'', in_ds, format='MEM',
dstSRS=get_SRS(out_prj),
outputType=get_GDT(out_dtype) if out_dtype else get_GDT(in_dtype_np),
xRes=out_gsd[0],
yRes=out_gsd[1],
outputBounds=out_bounds,
outputBoundsSRS=get_SRS(out_bounds_prj),
width=out_XYdims[0],
height=out_XYdims[1],
resampleAlg=rspAlg,
srcNodata=in_nodata,
dstNodata=out_nodata,
srcAlpha=in_alpha,
dstAlpha=out_alpha,
options=options if options else [],
warpOptions=warpOptions or [],
transformerOptions=transformerOptions or [],
targetAlignedPixels=targetAlignedPixels,
tps=True if gcpList else False,
polynomialOrder=polynomialOrder,
warpMemoryLimit=warpMemoryLimit,
callback=ProgressBar(prefix='Warping progress ', timeout=None) if progress and not q else None,
callback_data=[0],
errorThreshold=0.125, # this is needed to get exactly the same output like the console version of GDAL warp
)
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/10round_shapelyPoly_coords() is incompatible with shapely 1.7.02020-03-26T23:30:10+01:00Daniel Schefflerround_shapelyPoly_coords() is incompatible with shapely 1.7.0Cause for issue https://gitext.gfz-potsdam.de/danschef/arosics/-/issues/34.Cause for issue https://gitext.gfz-potsdam.de/danschef/arosics/-/issues/34.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/13TypeError when a MultiPolygon instance is passed to fill_holes_within_poly()2020-10-30T16:57:17+01:00Daniel SchefflerTypeError when a MultiPolygon instance is passed to fill_holes_within_poly()```python
from shapely.geometry import Polygon, MultiPolygon
from py_tools_ds.geo.vector.topology import fill_holes_within_poly
polygon_1 = Polygon([(0, 0), (5, 0), (5, 5), (0, 5)]) # large poly
polygon_2 = Polygon([(1, 1), (1, 2), (2,...```python
from shapely.geometry import Polygon, MultiPolygon
from py_tools_ds.geo.vector.topology import fill_holes_within_poly
polygon_1 = Polygon([(0, 0), (5, 0), (5, 5), (0, 5)]) # large poly
polygon_2 = Polygon([(1, 1), (1, 2), (2, 2), (2, 1)]) # small poly within large poly
multipolygon = MultiPolygon([polygon_1, polygon_2])
fill_holes_within_poly(multipolygon)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-11-8ecef912bc33> in <module>
6 multipolygon = MultiPolygon([polygon_1, polygon_2])
7
----> 8 fill_holes_within_poly(multipolygon)
~/miniconda3/envs/arosics/lib/python3.8/site-packages/py_tools_ds/geo/vector/topology.py in fill_holes_within_poly(poly)
187 lambda GDF_row: Polygon(np.swapaxes(np.array(GDF_row.geometry.exterior.coords.xy), 0, 1)).area, axis=1)
188
--> 189 largest_poly_filled = gdf.loc[gdf['area_filled'].idxmax()]['geometry']
190
191 # return the outer boundary of the largest polygon
~/miniconda3/envs/arosics/lib/python3.8/site-packages/pandas/core/series.py in idxmax(self, axis, skipna, *args, **kwargs)
2166 """
2167 skipna = nv.validate_argmax_with_skipna(skipna, args, kwargs)
-> 2168 i = nanops.nanargmax(self._values, skipna=skipna)
2169 if i == -1:
2170 return np.nan
~/miniconda3/envs/arosics/lib/python3.8/site-packages/pandas/core/nanops.py in _f(*args, **kwargs)
64 if any(self.check(obj) for obj in obj_iter):
65 f_name = f.__name__.replace("nan", "")
---> 66 raise TypeError(
67 f"reduction operation '{f_name}' not allowed for this dtype"
68 )
TypeError: reduction operation 'argmax' not allowed for this dtype
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/3TypeError: type() argument 1 must be str, not type2017-09-25T18:16:39+02:00Daniel SchefflerTypeError: type() argument 1 must be str, not type```
Traceback (most recent call last):
File "/tmp/flink_plan504207454/plan.py", line 10, in <module>
from gms.io.GMSPreprocessInputFormat import
GmsPreprocessInputFormat, SpillInputFormat
File
"/home/hub-inf/python_GFZ/python/lib...```
Traceback (most recent call last):
File "/tmp/flink_plan504207454/plan.py", line 10, in <module>
from gms.io.GMSPreprocessInputFormat import
GmsPreprocessInputFormat, SpillInputFormat
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/gms/io/GMSPreprocessInputFormat.py",
line 7, in <module>
import gms_preprocessing.config as CFG
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/gms_preprocessing/__init__.py",
line 9, in <module>
from . import algorithms
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/gms_preprocessing/algorithms/__init__.py",
line 10, in <module>
from . import GEOPROCESSING
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/gms_preprocessing/algorithms/GEOPROCESSING.py",
line 54, in <module>
from geoarray import GeoArray
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/geoarray/__init__.py",
line 7, in <module>
from .baseclasses import GeoArray # noqa: E402
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/geoarray/baseclasses.py",
line 27, in <module>
from py_tools_ds.geo.coord_calc import get_corner_coordinates
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/py_tools_ds/geo/__init__.py",
line 2, in <module>
from . import vector
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/py_tools_ds/geo/vector/__init__.py",
line 3, in <module>
from . import geometry
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/py_tools_ds/geo/vector/geometry.py",
line 6, in <module>
from .conversion import shapelyImPoly_to_shapelyMapPoly,
get_boxImXY_from_shapelyPoly, \
File
"/home/hub-inf/python_GFZ/python/lib/python3.5/site-packages/py_tools_ds/geo/vector/conversion.py",
line 23, in <module>
_T_polygon = TypeVar(Polygon)
File "/home/hub-inf/python_GFZ/python/lib/python3.5/typing.py", line
406, in __new__
self = super().__new__(cls, name, (Final,), {}, _root=True)
File "/home/hub-inf/python_GFZ/python/lib/python3.5/typing.py", line
107, in __new__
return super().__new__(cls, name, bases, namespace)
TypeError: type() argument 1 must be str, not type
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/16WKT2EPSG() returns None in case of bound CRS.2020-12-15T16:45:53+01:00Daniel SchefflerWKT2EPSG() returns None in case of bound CRS.```python
wkt = 'PROJCS["ETRS89-extended / LAEA Europe",GEOGCS["ETRS89",DATUM["European_Terrestrial_Reference_System_1989",SPHEROID["GRS 1980",6378137,298.257222101,AUTHORITY["EPSG","7019"]],TOWGS84[0,0,0,0,0,0,0],AUTHORITY["EPSG","6258"...```python
wkt = 'PROJCS["ETRS89-extended / LAEA Europe",GEOGCS["ETRS89",DATUM["European_Terrestrial_Reference_System_1989",SPHEROID["GRS 1980",6378137,298.257222101,AUTHORITY["EPSG","7019"]],TOWGS84[0,0,0,0,0,0,0],AUTHORITY["EPSG","6258"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4258"]],PROJECTION["Lambert_Azimuthal_Equal_Area"],PARAMETER["latitude_of_center",52],PARAMETER["longitude_of_center",10],PARAMETER["false_easting",4321000],PARAMETER["false_northing",3210000],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Northing",NORTH],AXIS["Easting",EAST],AUTHORITY["EPSG","3035"]]'
WKT2EPSG(wkt)
```
`None`Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/py_tools_ds/-/issues/20Wrong output of raster2polygon in case exact=False.2022-02-10T17:54:21+01:00Daniel SchefflerWrong output of raster2polygon in case exact=False.```python
from geoarray import GeoArray
from py_tools_ds.geo.raster.conversion import raster2polygon
gA = GeoArray('mask.tif')
raster2polygon(gA.mask_nodata.astype(np.uint8), gA.gt, gA.prj, exact=False)
```
returns:
![grafik](/uploads...```python
from geoarray import GeoArray
from py_tools_ds.geo.raster.conversion import raster2polygon
gA = GeoArray('mask.tif')
raster2polygon(gA.mask_nodata.astype(np.uint8), gA.gt, gA.prj, exact=False)
```
returns:
![grafik](/uploads/1be48a1b052d7b3500a9c1a57623016b/grafik.png)
where the input image clearly has valid values on the right:
![grafik](/uploads/bb7235ff6697809f0cb08cc3d2b04c83/grafik.png)Daniel SchefflerDaniel Scheffler