arosics merge requestshttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests2024-02-13T17:24:09+01:00https://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/51Use joblib with mem-mapping for multiprocessing2024-02-13T17:24:09+01:00Daniel SchefflerUse joblib with mem-mapping for multiprocessingThis replaces the current multiprocessing implementation with parallelization based on joblib including mem-mapping.
Fixes https://github.com/GFZ/arosics/issues/36.
Here are some benchmark results with 2.5k tie points and 10k tie point...This replaces the current multiprocessing implementation with parallelization based on joblib including mem-mapping.
Fixes https://github.com/GFZ/arosics/issues/36.
Here are some benchmark results with 2.5k tie points and 10k tie points. The table contains processing times for tie point computation only and the total computation for the entire workflow. The Linux machine has 32 cores, the Windows machine 12 cores. RAM usage is much higher with the former plain multiprocessing implementation on Windows than with joblib. Bold fonts indicate changes in this merge request:
| Backend | OS/Branch | 10k | total 10k | 2.5k | total 2.5k |
|----------------------------------|--------------|-------|-----------|-------|-----------|
| multiprocessing with progress | Linux main | 00:51 | 01:16 | 00:13 | 00:36 |
| Loky with progress | Linux new | 01:02 | **01:26** | 00:18 | **00:41** |
| multiprocessing without progress | Linux new | 00:55 | **01:19** | 00:15 | **00:37** |
| multiprocessing with progress | Windows main | 03:40 | 04:19 | 00:55 | 01:33 |
| Loky with progress | Windows new | 04:09 | **04:25** | 01:05 | **01:23** |
| multiprocessing without progress | Windows new | 04:12 | 04:28 | 01:18 | 01:33 |
So the new implementation is slightly slower than the old one but requires much less memory on Windows and seems to fix deadlocks on Windows. Processing can also be interrupted with Ctrl-C and tracebacks are more useful.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/50Maintenance/replace http links2024-01-02T19:21:20+01:00Daniel SchefflerMaintenance/replace http linksDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/49Maintenance/update copyright2024-01-02T18:50:06+01:00Daniel SchefflerMaintenance/update copyrightUpdates the copyright.Updates the copyright.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/48Enhancement/ignore runtime warning2023-12-22T14:36:52+01:00Daniel SchefflerEnhancement/ignore runtime warningFix #91 by ignoring the warning.Fix #91 by ignoring the warning.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/47Fix some code style and language issues.2023-12-16T00:13:07+01:00Daniel SchefflerFix some code style and language issues.Signed-off-by: Daniel Scheffler <danschef@gfz-potsdam.de>Signed-off-by: Daniel Scheffler <danschef@gfz-potsdam.de>Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/46Maintenance/remove Tie_Point_Grid.to_Raster_using_Kriging2023-12-15T15:04:52+01:00Daniel SchefflerMaintenance/remove Tie_Point_Grid.to_Raster_using_KrigingThis removed Tie_Point_Grid.to_Raster_using_Kriging() which was replaced by Tie_Point_Grid.to_interpolated_raster().This removed Tie_Point_Grid.to_Raster_using_Kriging() which was replaced by Tie_Point_Grid.to_interpolated_raster().Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/45Enhancement/use pytest plain assertions2023-12-15T13:10:26+01:00Daniel SchefflerEnhancement/use pytest plain assertionsThis replaced the unittest assertions with pytest plain assertions to simplify the code and improve the test output.This replaced the unittest assertions with pytest plain assertions to simplify the code and improve the test output.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/44Enhancement/drop deprecated to_PointShapefile2023-12-15T12:46:10+01:00Daniel SchefflerEnhancement/drop deprecated to_PointShapefileThis drops Tie_Point_Grid.Tie_Point_Grid._to_PointShapefile() which was replaced by Tie_Point_Grid.Tie_Point_Grid.to_PointShapefile().This drops Tie_Point_Grid.Tie_Point_Grid._to_PointShapefile() which was replaced by Tie_Point_Grid.Tie_Point_Grid.to_PointShapefile().Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/43Enhancement/add outlier removal to shapefile2023-12-15T14:20:32+01:00Daniel SchefflerEnhancement/add outlier removal to shapefileThis adds a parameter to exclude outliers from the point shapefile written by `Tie_Point_Grid.Tie_Point_Grid.to_PointShapefile()`.
Closes #90.This adds a parameter to exclude outliers from the point shapefile written by `Tie_Point_Grid.Tie_Point_Grid.to_PointShapefile()`.
Closes #90.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/42Bugfix/fix typeerror mask2023-11-23T00:52:36+01:00Daniel SchefflerBugfix/fix typeerror maskFix for https://github.com/GFZ/arosics/issues/33 ('TypeError: ufunc 'invert' not supported for the input types' when using input mask).
Related to https://git.gfz-potsdam.de/danschef/geoarray/-/issues/44 and https://git.gfz-potsdam.de/d...Fix for https://github.com/GFZ/arosics/issues/33 ('TypeError: ufunc 'invert' not supported for the input types' when using input mask).
Related to https://git.gfz-potsdam.de/danschef/geoarray/-/issues/44 and https://git.gfz-potsdam.de/danschef/geoarray/-/issues/45.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/41Bugfix/fix show matchwin2023-10-16T16:28:23+02:00Daniel SchefflerBugfix/fix show matchwinFixes #86 and improves the interactive output.Fixes #86 and improves the interactive output.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/40Fix warning during package building.2023-08-11T14:12:10+02:00Daniel SchefflerFix warning during package building.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/39Maintenance/fix ransac warning2023-08-11T12:06:29+02:00Daniel SchefflerMaintenance/fix ransac warningFixes #83.Fixes #83.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/38Update CI.2023-06-16T14:33:33+02:00Daniel SchefflerUpdate CI.Signed-off-by: Daniel Scheffler <danschef@gfz-potsdam.de>Signed-off-by: Daniel Scheffler <danschef@gfz-potsdam.de>Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/37Bugfix/fix indexerror2023-06-16T14:17:31+02:00Daniel SchefflerBugfix/fix indexerrorFixes #82.Fixes #82.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/35Enhancement/fix ransac random state2023-05-04T18:16:59+02:00Daniel SchefflerEnhancement/fix ransac random stateThis sets the RANSAC random state to 0 by default to ensure reproducable filtering results.
Fixes https://github.com/GFZ/arosics/issues/18.This sets the RANSAC random state to 0 by default to ensure reproducable filtering results.
Fixes https://github.com/GFZ/arosics/issues/18.Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/34Enhancement/fix gdal warning2023-05-04T17:34:08+02:00Daniel SchefflerEnhancement/fix gdal warningFixes the following warning:
```python
Warning 1: PROJ: proj_create_from_database: Open of /home/gfz-fe/mambaforge/envs/arosics/share/proj failed
Warning 1: The definition of projected CRS EPSG:32633 got from GeoTIFF keys is not the sam...Fixes the following warning:
```python
Warning 1: PROJ: proj_create_from_database: Open of /home/gfz-fe/mambaforge/envs/arosics/share/proj failed
Warning 1: The definition of projected CRS EPSG:32633 got from GeoTIFF keys is not the same as the one from the EPSG registry, which may cause issues during reprojection operations. Set GTIFF_SRS_SOURCE configuration option to EPSG to use official parameters (overriding the ones from GeoTIFF keys), or to GEOKEYS to use custom values from GeoTIFF keys and drop the EPSG code.
Warning 1: PROJ: proj_create_from_database: Open of /home/gfz-fe/mambaforge/envs/arosics/share/proj failed
Warning 1: The definition of projected CRS EPSG:32633 got from GeoTIFF keys is not the same as the one from the EPSG registry, which may cause issues during reprojection operations. Set GTIFF_SRS_SOURCE configuration option to EPSG to use official parameters (overriding the ones from GeoTIFF keys), or to GEOKEYS to use custom values from GeoTIFF keys and drop the EPSG code.
```Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/33Maintenance/update copyright2023-03-10T17:59:47+01:00Daniel SchefflerMaintenance/update copyrightDaniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/32Bugfix/fix crea options2023-03-10T18:44:21+01:00Daniel SchefflerBugfix/fix crea optionsThis should fix https://github.com/GFZ/arosics/issues/20 (out_crea_options has no effect).This should fix https://github.com/GFZ/arosics/issues/20 (out_crea_options has no effect).Daniel SchefflerDaniel Schefflerhttps://git.gfz-potsdam.de/danschef/arosics/-/merge_requests/31Enhancement/update ci system2022-12-02T17:09:19+01:00Daniel SchefflerEnhancement/update ci systemReplaces the CentOS docker image with Ubuntu.Replaces the CentOS docker image with Ubuntu.Daniel SchefflerDaniel Scheffler