arosics issueshttps://git.gfz-potsdam.de/danschef/arosics/-/issues2024-03-28T21:16:46+01:00https://git.gfz-potsdam.de/danschef/arosics/-/issues/102Memory sharing in multiprocessing2024-03-28T21:16:46+01:00Roelof van DijkMemory sharing in multiprocessingAs far as I understand, global variables can only be shared between multiprocessing processes when using `multiprocessing.set_start_method('fork')`.
A different way to share the memory would be to back the cached arrays in the `GeoArray...As far as I understand, global variables can only be shared between multiprocessing processes when using `multiprocessing.set_start_method('fork')`.
A different way to share the memory would be to back the cached arrays in the `GeoArray`s using `multiprocessing.shared_memory`, there is a numpy example on https://docs.python.org/3/library/multiprocessing.shared_memory.html.https://git.gfz-potsdam.de/danschef/arosics/-/issues/101gdal.AllRegister() is called on COREG construction2024-03-26T17:11:22+01:00Roelof van Dijkgdal.AllRegister() is called on COREG construction`gdal.AllRegister()` is called during the construction of a `COREG` class, it can be moved to the top of the file to save some time.`gdal.AllRegister()` is called during the construction of a `COREG` class, it can be moved to the top of the file to save some time.https://git.gfz-potsdam.de/danschef/arosics/-/issues/100lru_cache can speed up many functions calls to py_tools_ds2024-03-26T17:05:40+01:00Roelof van Dijklru_cache can speed up many functions calls to py_tools_dsMany lower-level functions within `arosics` and `py_tools_ds` get called thousands of times with the same input, since all GCP tiles have the same projection.
One example is `prj_equal` in `py_tools_ds.geo.projection`.
A simple way to ...Many lower-level functions within `arosics` and `py_tools_ds` get called thousands of times with the same input, since all GCP tiles have the same projection.
One example is `prj_equal` in `py_tools_ds.geo.projection`.
A simple way to speed up those calls is by using a `lru_cache` decorator.
```
import functools
@functools.lru_cache
def prj_equal(prj1, prj2):
```
A more fundamental improvement would be to remove some of these checks for the individual tile `COREG` classes, similar to the `nodata` check.https://git.gfz-potsdam.de/danschef/arosics/-/issues/99scipy.fft.fft2 is 2-3x faster than np.fft.fft22024-03-26T17:26:02+01:00Roelof van Dijkscipy.fft.fft2 is 2-3x faster than np.fft.fft2One of the bottlenecks of the local coregistration is the fourier transform. If `pyfftw` is not available (which is everywhere, because the most recent version is currently disabled), `numpy` is used.
In my testing (M3 Mac), the `scipy....One of the bottlenecks of the local coregistration is the fourier transform. If `pyfftw` is not available (which is everywhere, because the most recent version is currently disabled), `numpy` is used.
In my testing (M3 Mac), the `scipy.fft.fft2` implementation is 2-3x faster than `np.fft.fft2` within `arosics`.
```
Time per function call
479822.5 fft_arr0 = np.fft.fft2(in_arr0)
350634.3 fft_arr1 = np.fft.fft2(in_arr1)
171587.7 fft_arr0 = scipy.fft.fft2(in_arr0)
154358.3 fft_arr1 = scipy.fft.fft2(in_arr1)
```
It's a drop-in replacement in `CoReg.py`, `scipy` is already used elsewhere.https://git.gfz-potsdam.de/danschef/arosics/-/issues/98find_nearest in py_tools_ds can be much faster2024-03-26T15:15:51+01:00Roelof van Dijkfind_nearest in py_tools_ds can be much fasterSeveral issues:
* The array sorting is slow and only required if the input array is not already sorted. In actual usages, the arrays are mostly created using np.arange. Way faster with an `is_sorted` flag in those cases.
```
if isins...Several issues:
* The array sorting is slow and only required if the input array is not already sorted. In actual usages, the arrays are mostly created using np.arange. Way faster with an `is_sorted` flag in those cases.
```
if isinstance(array, list):
array = np.array(array)
if array.ndim > 1:
array = array.flatten()
if not is_sorted:
array = np.sort(array)
minimum, maximum = array[0], array[-1]
```
* The array is already sorted, yet the code uses min(array) and max(array). That's unnecessary, those are just the first and last elements
* The array is already sorted, the `collections.Counter` use can be replaced by something like
```
neighbours = diffs[max(minIdx-1,0):min(minIdx+2, len(diffs)-1)]
isMiddleVal = len(neighbours) > len(np.unique(neighbours))
```https://git.gfz-potsdam.de/danschef/arosics/-/issues/972x performance for local coreg: disable nodata check of complete image for ea...2024-03-26T14:53:17+01:00Roelof van Dijk2x performance for local coreg: disable nodata check of complete image for each GCPDuring local coregistration, the check if there is any valid (non-nodata) pixel is ran on the complete image for each GCP:
https://git.gfz-potsdam.de/danschef/arosics/-/blob/main/arosics/CoReg.py?ref_type=heads#L109
Through the instanti...During local coregistration, the check if there is any valid (non-nodata) pixel is ran on the complete image for each GCP:
https://git.gfz-potsdam.de/danschef/arosics/-/blob/main/arosics/CoReg.py?ref_type=heads#L109
Through the instantiation here:
https://git.gfz-potsdam.de/danschef/arosics/-/blob/main/arosics/Tie_Point_Grid.py?ref_type=heads#L275
This is not necessary, since the image should already be checked once in the instantiation of `COREG_LOCAL`.
It is also very slow, especially for very large images and many points.
Something like this halves the total coregistration time in our use-case:
```
diff --git a/CoReg.py b/CoReg.py
index 22cec69..66c428b 100644
--- a/CoReg.py
+++ b/CoReg.py
@@ -103,11 +103,11 @@ class GeoArray_CoReg(GeoArray):
% (self.imName, self.bands, 'bands' if self.bands > 1 else
'band', 'between 1 and ' if self.bands > 1 else '', self.bands, self.band4match)
- # compute nodata mask and validate that it is not completely filled with nodata
- self.calc_mask_nodata(fromBand=self.band4match) # this avoids that all bands have to be read
-
- if True not in self.mask_nodata[:]:
- raise RuntimeError(f'The {self.imName} passed to AROSICS only contains nodata values.')
+ if CoReg_params["validate_nonempty"]:
+ # compute nodata mask and validate that it is not completely filled with nodata
+ self.calc_mask_nodata(fromBand=self.band4match) # this avoids that all bands have to be read
+ if not self.mask_nodata.any():
+ raise RuntimeError(f'The {self.imName} passed to AROSICS only contains nodata values.')
# set footprint_poly
given_footprint_poly = CoReg_params['footprint_poly_%s' % ('ref' if imID == 'ref' else 'tgt')]
@@ -180,6 +180,7 @@ class COREG(object):
data_corners_ref: list = None,
data_corners_tgt: list = None,
nodata: Tuple = (None, None),
+ validate_nonempty: bool = True,
calc_corners: bool = True,
binary_ws: bool = True,
mask_baddata_ref: Union[GeoArray, str] = None,
@@ -359,7 +360,7 @@ class COREG(object):
warnings.warn("The resampling algorithm 'average' causes sinus-shaped patterns in fft images that will "
"affect the precision of the calculated spatial shifts! It is highly recommended to "
"choose another resampling algorithm.")
-
+ self.validate_nonempty = validate_nonempty
self.path_out = path_out # updated by self.set_outpathes
self.fmt_out = fmt_out
self.out_creaOpt = out_crea_options
@@ -1414,7 +1415,7 @@ class COREG(object):
band2process=otherFull.band4match + 1,
clipextent=self.matchBox.mapPoly.bounds,
target_xyGrid=matchFull.xygrid_specs,
- q=True
+ q=True,
).correct_shifts()
return ds_results['GeoArray_shifted']
```
```
diff --git a/Tie_Point_Grid.py b/Tie_Point_Grid.py
index 260ca3a..8222444 100644
--- a/Tie_Point_Grid.py
+++ b/Tie_Point_Grid.py
@@ -360,6 +360,7 @@ class Tie_Point_Grid(object):
nodata=(self.COREG_obj.ref.nodata, self.COREG_obj.shift.nodata),
force_quadratic_win=self.COREG_obj.force_quadratic_win,
binary_ws=self.COREG_obj.bin_ws,
+ validate_nonempty=False,
v=False, # True leads to massive STDOUT
q=True, # True leads to massive STDOUT
ignore_errors=True
```https://git.gfz-potsdam.de/danschef/arosics/-/issues/96conda package for Python 3.122024-03-26T16:55:17+01:00Michael Ayeconda package for Python 3.12When you got time, would be nice to get a conda package for 3.12 going. at the moment the requirement says "<3.12a"When you got time, would be nice to get a conda package for 3.12 going. at the moment the requirement says "<3.12a"https://git.gfz-potsdam.de/danschef/arosics/-/issues/95Verbose controls functionality (don't think it should)2024-02-22T19:06:26+01:00Michael AyeVerbose controls functionality (don't think it should)This line:
https://git.gfz-potsdam.de/danschef/arosics/-/blob/main/arosics/CoReg.py#L444
is performing a write_shapefile or something similar ONLY if verbose is set.
So, as by default it's off I never saw this issue and as I'm using no...This line:
https://git.gfz-potsdam.de/danschef/arosics/-/blob/main/arosics/CoReg.py#L444
is performing a write_shapefile or something similar ONLY if verbose is set.
So, as by default it's off I never saw this issue and as I'm using non-standard planetary projections, I cannot use all the things that arosics offers.
But I was very surprised when a working pipeline broke, just because I switched on "verbose=True" at the beginning.
I haven't checked if this happens more than once, but IMHO, functional code should not depend on a setting that is usually used for deciding how much feedback is given on everything.https://git.gfz-potsdam.de/danschef/arosics/-/issues/94SSIM input array shapes could not be equalized2024-02-06T11:15:03+01:00Maximilian BrellSSIM input array shapes could not be equalized/misc/soft/local/pygfz/mambaforge/envs/enval/lib/python3.10/site-packages/arosics/CoReg.py:1491: UserWarning: SSIM input array shapes could not be equalized. SSIM calculation failed. SSIM of the de-shifted target image is set to 0.
war.../misc/soft/local/pygfz/mambaforge/envs/enval/lib/python3.10/site-packages/arosics/CoReg.py:1491: UserWarning: SSIM input array shapes could not be equalized. SSIM calculation failed. SSIM of the de-shifted target image is set to 0.
warnings.warn('SSIM input array shapes could not be equalized. SSIM calculation failed. '
Testdatensat: enval1\EnVAL_ID_2\ENMAP01-____L2A-DT0000001226_20220623T025609Z_011_V010201_20230331T142658Z.ZIPhttps://git.gfz-potsdam.de/danschef/arosics/-/issues/92Tie_Point_Grid creates FutureWarning2024-01-30T18:50:22+01:00Michael AyeTie_Point_Grid creates FutureWarningIt's only a warning now but needs to be dealt with sooner or later:
```
/home/ayek72/mambaforge/envs/py311/lib/python3.11/site-packages/arosics/Tie_Point_Grid.py:433: FutureWarning: Downcasting behavior in `replace` is deprecated and wi...It's only a warning now but needs to be dealt with sooner or later:
```
/home/ayek72/mambaforge/envs/py311/lib/python3.11/site-packages/arosics/Tie_Point_Grid.py:433: FutureWarning: Downcasting behavior in `replace` is deprecated and will be removed in a future version. To retain the old behavior, explicitly call `result.infer_objects(copy=False)`. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
GDF = GDF.replace([np.nan, None], int(self.outFillVal)) # fillna fails with geopandas==0.6.0
/home/ayek72/mambaforge/envs/py311/lib/python3.11/site-packages/arosics/Tie_Point_Grid.py:1220: FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
fullGDF = fullGDF.fillna(False) # NaNs are due to exclude_previous_outliers
```
My geopandas version is 0.14.2
Arosics is 1.10.1https://git.gfz-potsdam.de/danschef/arosics/-/issues/84Move random selection of 7000 tie points to image warping to allow feature tr...2023-08-10T12:28:06+02:00Daniel SchefflerMove random selection of 7000 tie points to image warping to allow feature tracking with more shift vectorsAs discussed in https://github.com/GFZ/arosics/issues/28, it might make sense to move the automatic sub-setting of the originally computed tie point list to the image warping instead of doing this directly when creating the GCP list. Cur...As discussed in https://github.com/GFZ/arosics/issues/28, it might make sense to move the automatic sub-setting of the originally computed tie point list to the image warping instead of doing this directly when creating the GCP list. Currently, at most 7000 tie points are auto-selected when creating the GCP list within the Tie_Point_Grid module. However, moving this to a later stage would enable to use AROSICS for feature tracking with more than 7000 tie points, where the correction of image displacements is not needed at all.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/issues/79Apply detected shifts to different resolution images2023-04-19T13:05:25+02:00Philippe VerleyApply detected shifts to different resolution imagesDear Daniel,
We're using AROSICS to coregister tropical canopy drone images, 5cm resolution.
We noticed that both global and local shift perform better in our case with lower res images (~1m).
So I intended the following chain:
* detec...Dear Daniel,
We're using AROSICS to coregister tropical canopy drone images, 5cm resolution.
We noticed that both global and local shift perform better in our case with lower res images (~1m).
So I intended the following chain:
* detect local shift on 1m-res image with COREG_LOCAL.correct_shifts()
* apply these shifts to the 5cm-res image with DESHIFTER
but results are erratic. Should it work out of the box? If so could you please guide us in parametrizing correctly both COREG_LOCAL and DESHIFTER?
[Documentation on the topic](https://danschef.git-pages.gfz-potsdam.de/arosics/doc/usage/global_coreg.html#apply-detected-shifts-to-multiple-images) says that deshifting should be applied to images derived from the same satellite dataset. Does it imply images with same resolution?
Kind regards,
Philippe Verley (AMAP lab, Montpellier, France)https://git.gfz-potsdam.de/danschef/arosics/-/issues/76Extra columns in CoRegPoints_table2023-05-04T17:31:16+02:00Michael AyeExtra columns in CoRegPoints_tableHello! Thanks for your package!
I am looking for how to make use out of these extra undocumented columns in the COREG_*LOCAL.CoRegPoints_table, what do these columns contain?
![image](/uploads/a428ccbfb02f80d5224f292704302ab8/image.png...Hello! Thanks for your package!
I am looking for how to make use out of these extra undocumented columns in the COREG_*LOCAL.CoRegPoints_table, what do these columns contain?
![image](/uploads/a428ccbfb02f80d5224f292704302ab8/image.png)
I am also wondering/worried about the GeoDataFrame.dtype for the outlier columns? They all are `object`, because the are a mix of booleans and -9999?
What is the difference between False and -9999 in the table?
My OS is Linux, arosics version 1.7.8https://git.gfz-potsdam.de/danschef/arosics/-/issues/74Scale and offset metadata of input dataset get lost in the output.2023-05-03T13:05:14+02:00Daniel SchefflerScale and offset metadata of input dataset get lost in the output.See [here](https://bitbucket.org/hu-geomatics/enmap-box/issues/1315/a-large-difference-in-reflectance-values#comment-62248636).See [here](https://bitbucket.org/hu-geomatics/enmap-box/issues/1315/a-large-difference-in-reflectance-values#comment-62248636).Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/issues/69Local co-registration - automatically determine a meaningful grid resolution2021-11-20T02:06:18+01:00Daniel SchefflerLocal co-registration - automatically determine a meaningful grid resolutionThis would be quite useful, also see here: https://github.com/SMByC/Coregistration-Qgis-processing/issues/2#issuecomment-974558482This would be quite useful, also see here: https://github.com/SMByC/Coregistration-Qgis-processing/issues/2#issuecomment-974558482Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/issues/57Clarification on rotation tolerance2021-07-01T11:47:56+02:00RobinClarification on rotation toleranceMy apologies as this is not an issue as such, although might be a clarification to add in the documentation. I understand that only XY translations can be corrected in both local and global mode. Can you advise/clarify the requirement on...My apologies as this is not an issue as such, although might be a clarification to add in the documentation. I understand that only XY translations can be corrected in both local and global mode. Can you advise/clarify the requirement on rotational alignment? e.g. must be better than 2 degrees for examplehttps://git.gfz-potsdam.de/danschef/arosics/-/issues/54Allow to enable or disable the filtering techniques individually.2021-05-04T22:35:59+02:00Daniel SchefflerAllow to enable or disable the filtering techniques individually.Currectly, the user may set a filtering level between 1 and 3 but it is not possible to, e.g., disable SSIM filtering without disabling RANSAC.Currectly, the user may set a filtering level between 1 and 3 but it is not possible to, e.g., disable SSIM filtering without disabling RANSAC.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/issues/53Implement a filtering technique that flags tie points with largely differing ...2021-05-04T22:35:29+02:00Daniel SchefflerImplement a filtering technique that flags tie points with largely differing shift direction compared to adjacent tie points.Such largely differing shift directions appear especially in case of high resolution input images and may cause warping artifacts. To prevent this, it may be useful to add a filtering technique that detects such cases and flags them befo...Such largely differing shift directions appear especially in case of high resolution input images and may cause warping artifacts. To prevent this, it may be useful to add a filtering technique that detects such cases and flags them before warping.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/issues/52Disable SSIM filtering for shifts in the subpixel range or add a tolerance.2021-05-04T22:35:59+02:00Daniel SchefflerDisable SSIM filtering for shifts in the subpixel range or add a tolerance.For shifts in the subpixel range, it may happen that the SSIM decreases after correcting the detected misregistration. In many cases this is only caused by the resampling and there is no reason to flag the respective tie point as an outl...For shifts in the subpixel range, it may happen that the SSIM decreases after correcting the detected misregistration. In many cases this is only caused by the resampling and there is no reason to flag the respective tie point as an outlier.
A solution could be to disable the SSIM filtering for all tie points with shifts well in the subpixel range or to add some tolerance when comparing the pre- and post-coregistration SSIM. This way, the RANSAC filtering can still be turned on - otherwise it is also turned off when disabling SSIM filtering via the input parameters.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/issues/39Automatically crop the reference image to the target image dimensions + margi...2023-01-31T23:11:24+01:00Daniel SchefflerAutomatically crop the reference image to the target image dimensions + margin to reduce memory load in case of much larger reference images.Currently AROSICS gets stuck when the input images have very large pixel dimensions (depending on the host technical resources). This should be handled somehow. As a first step, those parts of the reference image that are not needed for ...Currently AROSICS gets stuck when the input images have very large pixel dimensions (depending on the host technical resources). This should be handled somehow. As a first step, those parts of the reference image that are not needed for co-registration should also not be read into memory.Daniel SchefflerDaniel Scheffler