obmgapanalysis merge requestshttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests2020-12-01T18:23:17+01:00https://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/1Database connections merge request2020-12-01T18:23:17+01:00Nicolas Garcia OspinaDatabase connections merge requestThese 2 modules are used to connect to the running databases obm_buildings and planet_osm_line in the legacy database (for now). These functions have an important role in the collection of built areas per cell for the automatic completen...These 2 modules are used to connect to the running databases obm_buildings and planet_osm_line in the legacy database (for now). These functions have an important role in the collection of built areas per cell for the automatic completeness assesment. functions in the 2 modules are very similar but work with different default parameters for the 2 mentioned sql tables.BacklogMarius Kriegerowskimarius@gfz-potsdam.deDanijel SchorlemmerMarius Kriegerowskimarius@gfz-potsdam.dehttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/3Added very basic project structure2021-03-01T12:08:02+01:00Nicolas Garcia OspinaAdded very basic project structureAccording to #5 the basic structure of the project is added including:
setup.py
Readme
.gitignore
skeleton files
test directory with dummy test
cc @ds, @prehnAccording to #5 the basic structure of the project is added including:
setup.py
Readme
.gitignore
skeleton files
test directory with dummy test
cc @ds, @prehnNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/4Added code formatting2021-03-01T12:07:16+01:00Nicolas Garcia OspinaAdded code formattingAdded a `Makefile`, modified the `setup.py` and reformated one file
cc @ds @prehnAdded a `Makefile`, modified the `setup.py` and reformated one file
cc @ds @prehnNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/5Added CI pipeline2021-03-01T12:06:57+01:00Nicolas Garcia OspinaAdded CI pipelineAdded a CI pipeline that includes checking of code formatting and runs the tests created (dummy test so far).
cc @ds @prehnAdded a CI pipeline that includes checking of code formatting and runs the tests created (dummy test so far).
cc @ds @prehnNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/6Added License2021-01-08T11:57:34+01:00Nicolas Garcia OspinaAdded LicenseHere I add a `ICENSE` file, included copyright statements in skeleton files as well as `README.md`.
@ds @prehnHere I add a `ICENSE` file, included copyright statements in skeleton files as well as `README.md`.
@ds @prehnNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/7Added class based cell_processor as well as 1 test for the module2021-02-25T10:49:40+01:00Nicolas Garcia OspinaAdded class based cell_processor as well as 1 test for the moduleI added a the cell_processor module which includes the cell processing routines to be used. The module has an improvement in comparison with the previous prototypes:
- Make use of the `Babelgrid` library to generate the cell geometry ba...I added a the cell_processor module which includes the cell processing routines to be used. The module has an improvement in comparison with the previous prototypes:
- Make use of the `Babelgrid` library to generate the cell geometry based on the `quadkey`, before the input needed to be a pandas DataFrame, now it only needs `quadkey` codes.
I also added one test that runs fine in my computer but not in gitlab.
Do I need to fix the `.gitlab-ci.yml` or the `setup.py`?
\fyi @all
\rfc \approve @ds @fdNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/9Included DataSource.py,2021-02-25T10:49:40+01:00Nicolas Garcia OspinaIncluded DataSource.py,With this MR I want to start introducing the `DataSource` class.
The __init__ is builds paths and characteristics of the dataset that is intended to be used for the tile processing.
Im looking forward for comments, having this correct...With this MR I want to start introducing the `DataSource` class.
The __init__ is builds paths and characteristics of the dataset that is intended to be used for the tile processing.
Im looking forward for comments, having this correct will help on having cleaner functions here as well.
\approve @ds @fdNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/10Included Processor with two basic functions.2021-02-25T10:49:40+01:00Nicolas Garcia OspinaIncluded Processor with two basic functions.Hi!
In this merge request I introduce the `Processor` class, this class works as a toolbox that makes use of Tile and Datasource classes to complete settlement data processing at a tile level.
The functions included in this class make ...Hi!
In this merge request I introduce the `Processor` class, this class works as a toolbox that makes use of Tile and Datasource classes to complete settlement data processing at a tile level.
The functions included in this class make the following taks:
- find raster files with relevant information for a tile.
- load raster files and extract relevant pixels.
\rfc \approve @fd @dsNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/11Pixel poligonization and database2021-02-25T10:49:40+01:00Nicolas Garcia OspinaPixel poligonization and databaseIn this MR I included a step that transforms specific pixel values (found in `built_pixel_values` attribute for `DataSource`) into a polygon.
Also introduced the `DataBase` class that shall be further used to request roads data and also...In this MR I included a step that transforms specific pixel values (found in `built_pixel_values` attribute for `DataSource`) into a polygon.
Also introduced the `DataBase` class that shall be further used to request roads data and also pushing results from this program into the final DB (I could not get the mocking to work so there are no tests here)
\approve @fd @ds
PD. sorry for the delay on creating the MR, I discovered that polygons are created with different point orders on different machines.Nicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/12Included the DataBase class with proper connections and data retrieval functi...2021-03-01T12:12:16+01:00Nicolas Garcia OspinaIncluded the DataBase class with proper connections and data retrieval functionalityIn this merge request a new functionality is added. The DataBase class is included, so far it can take credentials and create a connection to a postgis database, also it can be point to a given table and do spatial queries for buildings ...In this merge request a new functionality is added. The DataBase class is included, so far it can take credentials and create a connection to a postgis database, also it can be point to a given table and do spatial queries for buildings and roads in a tile.
\approve @fd @dsNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/13Included roads processing and area calculation2021-03-08T11:48:19+01:00Nicolas Garcia OspinaIncluded roads processing and area calculationThis merge request focuses on refining the built areas with the OBM_roads database, I included 4 new functions:
- `reproject_polygon`: manages reprojecting polygons between different crs (generic to be used among other functions)
- `pro...This merge request focuses on refining the built areas with the OBM_roads database, I included 4 new functions:
- `reproject_polygon`: manages reprojecting polygons between different crs (generic to be used among other functions)
- `process_dataframe_with_tile`: Process any dataframe with the tile specs (meant to be used with roads)
- `polygon_difference`: Simple difference between 2 geometries
- `albers_area_calculation`: does the albers transformation and calculates the area
I also included a Jupyter notebook for documentation of how a tile is being processed. I think it is important to have this explicit documentation with graphs.
\approve @fd @dshttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/14Included build_dictionary and fixed main program2021-03-18T10:56:18+01:00Nicolas Garcia OspinaIncluded build_dictionary and fixed main programWith this MR I introduced a function that organizes the output tile information in a dictionary that is later used in groups by the main program to write csv files.
I found out that managing single pickle files would be much slower tha...With this MR I introduced a function that organizes the output tile information in a dictionary that is later used in groups by the main program to write csv files.
I found out that managing single pickle files would be much slower than pushing big batches in csv files.
The main program is written in a way that the reusable objects (Datasource, Database connection ...) are created outside everything so in the multiprocessing they can be consulted without creating them over and over.
\approve @fd @dsNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/15Included program to push CSV into SQL2021-03-23T09:46:03+01:00Nicolas Garcia OspinaIncluded program to push CSV into SQLIn this MR I created a second program that handles the CSV files and push them into a `target_database` which in this case would be obm_tiles, it also works on a multiprocessing frame since it is meant to be scaled up. I also did a small...In this MR I created a second program that handles the CSV files and push them into a `target_database` which in this case would be obm_tiles, it also works on a multiprocessing frame since it is meant to be scaled up. I also did a small fix in the dictionary creation.
\approve @fd @dsNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/16Improved config file and added documentation2021-03-29T08:34:19+02:00Nicolas Garcia OspinaImproved config file and added documentationThis is a small MR where I improved the`config.yml` (added an argument for road buffering). I also included some docs about this file and included 2 minor fixes.
\approve @fd @dsThis is a small MR where I improved the`config.yml` (added an argument for road buffering). I also included some docs about this file and included 2 minor fixes.
\approve @fd @dsNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/17Changed size_built_area to built_area_size2021-03-30T11:45:16+02:00Nicolas Garcia OspinaChanged size_built_area to built_area_sizeThis MR changes the variable name from `size_built_area` to `built_area_size` to fit the `obm_tiles` schema.
\approve @ds @fdThis MR changes the variable name from `size_built_area` to `built_area_size` to fit the `obm_tiles` schema.
\approve @ds @fdNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/18Fixed attribute names and geometry issues2021-04-13T15:40:54+02:00Nicolas Garcia OspinaFixed attribute names and geometry issuesThis MR includes the debugging of the program and it is working properly. I used this version to create the tiles for Greece.
\approve @fd @dsThis MR includes the debugging of the program and it is working properly. I used this version to create the tiles for Greece.
\approve @fd @dsNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/19Included obm_built_up program and debug2021-05-03T13:34:26+02:00Nicolas Garcia OspinaIncluded obm_built_up program and debugWith this MR I included a working program that extracts the built-up areas from the the OBM database, the output matches the `obm_completeneness_assessments` table now.
\approve @fd @dsWith this MR I included a working program that extracts the built-up areas from the the OBM database, the output matches the `obm_completeneness_assessments` table now.
\approve @fd @dsNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/20Included new csv importer and Albers calculator2021-06-11T07:54:20+02:00Nicolas Garcia OspinaIncluded new csv importer and Albers calculatorThis MR includes 2 changes that produce a better behavior of the program:
- Adoption of a better definition of the Albers Equal Area transformation.
- CSV import done completely using `psycopg2` with custom import statements that support...This MR includes 2 changes that produce a better behavior of the program:
- Adoption of a better definition of the Albers Equal Area transformation.
- CSV import done completely using `psycopg2` with custom import statements that support entry updates.
\approve @fd @dsDanijel SchorlemmerDanijel Schorlemmerhttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/21Improved memory usage and removed geometry sotrage2021-06-28T10:53:26+02:00Nicolas Garcia OspinaImproved memory usage and removed geometry sotrageThis MR focuses on the removal of geometries in the output and makes better use of RAM during execution on large areas.
\approve @fdThis MR focuses on the removal of geometries in the output and makes better use of RAM during execution on large areas.
\approve @fdNicolas Garcia OspinaNicolas Garcia Ospinahttps://git.gfz-potsdam.de/globaldynamicexposure/openbuildingmap/obmgapanalysis/-/merge_requests/22Improved config and import behaviour.2021-07-15T10:53:47+02:00Nicolas Garcia OspinaImproved config and import behaviour.This MR has 3 changes:
- It now allows the usage of a custom config filepath with the `-conf` argument.
- when import is launched, it lists files in both `output_pathname` and `obm_output_pathname`. With this the config file must not be...This MR has 3 changes:
- It now allows the usage of a custom config filepath with the `-conf` argument.
- when import is launched, it lists files in both `output_pathname` and `obm_output_pathname`. With this the config file must not be changed between commands.
- I included a new parameter in the config file to distinguish parallelization while processing and while importing. This is important because too many parallel processes during the SQL imports result in connection errors.
\approve @fdNicolas Garcia OspinaNicolas Garcia Ospina