obmgapanalysis merge requestshttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests2021-08-12T08:42:09+02:00https://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/23Added exception for rasters intersecting quadtile sides2021-08-12T08:42:09+02:00Nicolas Garcia OspinaAdded exception for rasters intersecting quadtile sidesThis small MR includes an exception for a case when a quadkey searches for pixels on a raster that is not really intersecting the quadkey but just a border effect (Raster 1 in the included diagram). I found this for the quadkeys at Green...This small MR includes an exception for a case when a quadkey searches for pixels on a raster that is not really intersecting the quadkey but just a border effect (Raster 1 in the included diagram). I found this for the quadkeys at Greenwich for instance. With this exception the code will log this but will still process the right raster as well.
\approve @fd @ds
![IMG_20210809_104510_862](/uploads/f0f16b85201a96561f64d0a413805e99/IMG_20210809_104510_862.jpg)Nicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/22Improved config and import behaviour.2021-07-15T10:53:47+02:00Nicolas Garcia OspinaImproved config and import behaviour.This MR has 3 changes:
- It now allows the usage of a custom config filepath with the `-conf` argument.
- when import is launched, it lists files in both `output_pathname` and `obm_output_pathname`. With this the config file must not be...This MR has 3 changes:
- It now allows the usage of a custom config filepath with the `-conf` argument.
- when import is launched, it lists files in both `output_pathname` and `obm_output_pathname`. With this the config file must not be changed between commands.
- I included a new parameter in the config file to distinguish parallelization while processing and while importing. This is important because too many parallel processes during the SQL imports result in connection errors.
\approve @fdNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/21Improved memory usage and removed geometry sotrage2021-06-28T10:53:26+02:00Nicolas Garcia OspinaImproved memory usage and removed geometry sotrageThis MR focuses on the removal of geometries in the output and makes better use of RAM during execution on large areas.
\approve @fdThis MR focuses on the removal of geometries in the output and makes better use of RAM during execution on large areas.
\approve @fdNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/20Included new csv importer and Albers calculator2021-06-11T07:54:20+02:00Nicolas Garcia OspinaIncluded new csv importer and Albers calculatorThis MR includes 2 changes that produce a better behavior of the program:
- Adoption of a better definition of the Albers Equal Area transformation.
- CSV import done completely using `psycopg2` with custom import statements that support...This MR includes 2 changes that produce a better behavior of the program:
- Adoption of a better definition of the Albers Equal Area transformation.
- CSV import done completely using `psycopg2` with custom import statements that support entry updates.
\approve @fd @dsDanijel SchorlemmerDanijel Schorlemmerhttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/19Included obm_built_up program and debug2021-05-03T13:34:26+02:00Nicolas Garcia OspinaIncluded obm_built_up program and debugWith this MR I included a working program that extracts the built-up areas from the the OBM database, the output matches the `obm_completeneness_assessments` table now.
\approve @fd @dsWith this MR I included a working program that extracts the built-up areas from the the OBM database, the output matches the `obm_completeneness_assessments` table now.
\approve @fd @dsNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/18Fixed attribute names and geometry issues2021-04-13T15:40:54+02:00Nicolas Garcia OspinaFixed attribute names and geometry issuesThis MR includes the debugging of the program and it is working properly. I used this version to create the tiles for Greece.
\approve @fd @dsThis MR includes the debugging of the program and it is working properly. I used this version to create the tiles for Greece.
\approve @fd @dsNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/17Changed size_built_area to built_area_size2021-03-30T11:45:16+02:00Nicolas Garcia OspinaChanged size_built_area to built_area_sizeThis MR changes the variable name from `size_built_area` to `built_area_size` to fit the `obm_tiles` schema.
\approve @ds @fdThis MR changes the variable name from `size_built_area` to `built_area_size` to fit the `obm_tiles` schema.
\approve @ds @fdNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/16Improved config file and added documentation2021-03-29T08:34:19+02:00Nicolas Garcia OspinaImproved config file and added documentationThis is a small MR where I improved the`config.yml` (added an argument for road buffering). I also included some docs about this file and included 2 minor fixes.
\approve @fd @dsThis is a small MR where I improved the`config.yml` (added an argument for road buffering). I also included some docs about this file and included 2 minor fixes.
\approve @fd @dsNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/15Included program to push CSV into SQL2021-03-23T09:46:03+01:00Nicolas Garcia OspinaIncluded program to push CSV into SQLIn this MR I created a second program that handles the CSV files and push them into a `target_database` which in this case would be obm_tiles, it also works on a multiprocessing frame since it is meant to be scaled up. I also did a small...In this MR I created a second program that handles the CSV files and push them into a `target_database` which in this case would be obm_tiles, it also works on a multiprocessing frame since it is meant to be scaled up. I also did a small fix in the dictionary creation.
\approve @fd @dsNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/14Included build_dictionary and fixed main program2021-03-18T10:56:18+01:00Nicolas Garcia OspinaIncluded build_dictionary and fixed main programWith this MR I introduced a function that organizes the output tile information in a dictionary that is later used in groups by the main program to write csv files.
I found out that managing single pickle files would be much slower tha...With this MR I introduced a function that organizes the output tile information in a dictionary that is later used in groups by the main program to write csv files.
I found out that managing single pickle files would be much slower than pushing big batches in csv files.
The main program is written in a way that the reusable objects (Datasource, Database connection ...) are created outside everything so in the multiprocessing they can be consulted without creating them over and over.
\approve @fd @dsNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/13Included roads processing and area calculation2021-03-08T11:48:19+01:00Nicolas Garcia OspinaIncluded roads processing and area calculationThis merge request focuses on refining the built areas with the OBM_roads database, I included 4 new functions:
- `reproject_polygon`: manages reprojecting polygons between different crs (generic to be used among other functions)
- `pro...This merge request focuses on refining the built areas with the OBM_roads database, I included 4 new functions:
- `reproject_polygon`: manages reprojecting polygons between different crs (generic to be used among other functions)
- `process_dataframe_with_tile`: Process any dataframe with the tile specs (meant to be used with roads)
- `polygon_difference`: Simple difference between 2 geometries
- `albers_area_calculation`: does the albers transformation and calculates the area
I also included a Jupyter notebook for documentation of how a tile is being processed. I think it is important to have this explicit documentation with graphs.
\approve @fd @dshttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/12Included the DataBase class with proper connections and data retrieval functi...2021-03-01T12:12:16+01:00Nicolas Garcia OspinaIncluded the DataBase class with proper connections and data retrieval functionalityIn this merge request a new functionality is added. The DataBase class is included, so far it can take credentials and create a connection to a postgis database, also it can be point to a given table and do spatial queries for buildings ...In this merge request a new functionality is added. The DataBase class is included, so far it can take credentials and create a connection to a postgis database, also it can be point to a given table and do spatial queries for buildings and roads in a tile.
\approve @fd @dsNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/11Pixel poligonization and database2021-02-25T10:49:40+01:00Nicolas Garcia OspinaPixel poligonization and databaseIn this MR I included a step that transforms specific pixel values (found in `built_pixel_values` attribute for `DataSource`) into a polygon.
Also introduced the `DataBase` class that shall be further used to request roads data and also...In this MR I included a step that transforms specific pixel values (found in `built_pixel_values` attribute for `DataSource`) into a polygon.
Also introduced the `DataBase` class that shall be further used to request roads data and also pushing results from this program into the final DB (I could not get the mocking to work so there are no tests here)
\approve @fd @ds
PD. sorry for the delay on creating the MR, I discovered that polygons are created with different point orders on different machines.Nicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/10Included Processor with two basic functions.2021-02-25T10:49:40+01:00Nicolas Garcia OspinaIncluded Processor with two basic functions.Hi!
In this merge request I introduce the `Processor` class, this class works as a toolbox that makes use of Tile and Datasource classes to complete settlement data processing at a tile level.
The functions included in this class make ...Hi!
In this merge request I introduce the `Processor` class, this class works as a toolbox that makes use of Tile and Datasource classes to complete settlement data processing at a tile level.
The functions included in this class make the following taks:
- find raster files with relevant information for a tile.
- load raster files and extract relevant pixels.
\rfc \approve @fd @dsNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/9Included DataSource.py,2021-02-25T10:49:40+01:00Nicolas Garcia OspinaIncluded DataSource.py,With this MR I want to start introducing the `DataSource` class.
The __init__ is builds paths and characteristics of the dataset that is intended to be used for the tile processing.
Im looking forward for comments, having this correct...With this MR I want to start introducing the `DataSource` class.
The __init__ is builds paths and characteristics of the dataset that is intended to be used for the tile processing.
Im looking forward for comments, having this correct will help on having cleaner functions here as well.
\approve @ds @fdNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/8Wip: Switched to debian:bullseye-slim image2021-01-21T10:20:15+01:00Marius Kriegerowskimarius@gfz-potsdam.deWip: Switched to debian:bullseye-slim imagehttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/7Added class based cell_processor as well as 1 test for the module2021-02-25T10:49:40+01:00Nicolas Garcia OspinaAdded class based cell_processor as well as 1 test for the moduleI added a the cell_processor module which includes the cell processing routines to be used. The module has an improvement in comparison with the previous prototypes:
- Make use of the `Babelgrid` library to generate the cell geometry ba...I added a the cell_processor module which includes the cell processing routines to be used. The module has an improvement in comparison with the previous prototypes:
- Make use of the `Babelgrid` library to generate the cell geometry based on the `quadkey`, before the input needed to be a pandas DataFrame, now it only needs `quadkey` codes.
I also added one test that runs fine in my computer but not in gitlab.
Do I need to fix the `.gitlab-ci.yml` or the `setup.py`?
\fyi @all
\rfc \approve @ds @fdNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/6Added License2021-01-08T11:57:34+01:00Nicolas Garcia OspinaAdded LicenseHere I add a `ICENSE` file, included copyright statements in skeleton files as well as `README.md`.
@ds @prehnHere I add a `ICENSE` file, included copyright statements in skeleton files as well as `README.md`.
@ds @prehnNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/5Added CI pipeline2021-03-01T12:06:57+01:00Nicolas Garcia OspinaAdded CI pipelineAdded a CI pipeline that includes checking of code formatting and runs the tests created (dummy test so far).
cc @ds @prehnAdded a CI pipeline that includes checking of code formatting and runs the tests created (dummy test so far).
cc @ds @prehnNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/4Added code formatting2021-03-01T12:07:16+01:00Nicolas Garcia OspinaAdded code formattingAdded a `Makefile`, modified the `setup.py` and reformated one file
cc @ds @prehnAdded a `Makefile`, modified the `setup.py` and reformated one file
cc @ds @prehnNicolas Garcia OspinaNicolas Garcia Ospina