exposure-japan issueshttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues2024-01-16T13:13:53+01:00https://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/46Get all the boundaries for Japan2024-01-16T13:13:53+01:00Simantini ShindeGet all the boundaries for JapanThe boundaries for Japan on the e-stat website are can be downloaded in two ways:
1. Prefactures (this also includes the municipalities in each prefecture)
2. Each municipality
The problem noticed was that the prefectures sometimes didn...The boundaries for Japan on the e-stat website are can be downloaded in two ways:
1. Prefactures (this also includes the municipalities in each prefecture)
2. Each municipality
The problem noticed was that the prefectures sometimes didn't include some of the municipalities which were given separately to be downloaded.
- [x] This means that each municipality has to be downloaded separately (maybe webscrapping can be used here).
- [x] Set up this functionality in datasources
- [x] Add `unzip` functionality in the download.sh for boundaries
- [x] The boundaries have to be checked if they are doubled and if yes one of them to be deleted (assuming they are the same for both)
- [x] The boundaries have to be combined into one
- [x] Add a admin level to it.
- [ ] Fix admin names (use the arealist file)
\fyi @ds @gislars @laurens @taraSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/24Preprocess the commercial data2024-01-16T12:44:49+01:00Simantini ShindePreprocess the commercial dataThe data (prefectures, designated cities and government office cities) is mixed and in Japanese and has to be preprocessed:
- [x] Replace - with 0
- [ ] Rename columns (https://git.gfz-potsdam.de/dynamicexposure/globaldynamicexposure/ex...The data (prefectures, designated cities and government office cities) is mixed and in Japanese and has to be preprocessed:
- [x] Replace - with 0
- [ ] Rename columns (https://git.gfz-potsdam.de/dynamicexposure/globaldynamicexposure/exposure-japan/-/issues/29) (https://git.gfz-potsdam.de/dynamicexposure/globaldynamicexposure/exposure-japan/-/merge_requests/80)
- [ ] Translate the column values (https://git.gfz-potsdam.de/dynamicexposure/globaldynamicexposure/exposure-japan/-/issues/29) (https://git.gfz-potsdam.de/dynamicexposure/globaldynamicexposure/exposure-japan/-/merge_requests/80)
- [ ] Translate the construction year (https://git.gfz-potsdam.de/dynamicexposure/globaldynamicexposure/exposure-japan/-/issues/29) (https://git.gfz-potsdam.de/dynamicexposure/globaldynamicexposure/exposure-japan/-/merge_requests/80)
- [x] Make the data uniform and readable by pandas.Simantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/47Add unidentified values in the individual attribute tables2024-01-16T12:44:38+01:00Simantini ShindeAdd unidentified values in the individual attribute tablesThese unidentified values are missing in the individual attribute table, this needs to be added.
\fyi @laurens @ds @taraThese unidentified values are missing in the individual attribute table, this needs to be added.
\fyi @laurens @ds @taraSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/45Rules to be applied during future working2024-01-16T12:43:39+01:00Simantini ShindeRules to be applied during future workingThis issues collects a list of logics and rules that need to be applied while creating the full exposure.
\fyi @ds @tara @laurensThis issues collects a list of logics and rules that need to be applied while creating the full exposure.
\fyi @ds @tara @laurensSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/44Find relationship between main usage, current usage and corporate industry2024-01-16T12:43:20+01:00Simantini ShindeFind relationship between main usage, current usage and corporate industryThe Japan data is given for two main commercial types: by usage and by corporate industry. The usage is further divided into current and main.
Here is the mapping between the current and the main usage:
| Main Usage ...The Japan data is given for two main commercial types: by usage and by corporate industry. The usage is further divided into current and main.
Here is the mapping between the current and the main usage:
| Main Usage | Current Usages | | | | |
|-----------------------------------------|-----------------------------------------|-------------------|----------------------|--------------------|-------------------|
| total | total | | | | |
| Office | office_only | office_and_stores | office_and_warehouse | office_and_housing | |
| Store | only_stores | office_and_stores | | | store_and_housing |
| Warehouse | warehouse_only | | office_and_warehouse | | |
| factory_plant | factory_only | | | | |
| Housing | house_only | | | office_and_housing | store_and_housing |
| Welfare_facilities | welfare_facility_only | | | | |
| Hotels_and_inns | hotel_inns_only | | | | |
| Educational_facilities | only_the_sector_facilities | | | | |
| Religious_facilities | religious_facilities_only | | | | |
| Building_type_parking_lot | building_type_parking_lot_only | | | | |
| Other_buildings | only_other_buildings | | | | |
| Unusable_buildings_abandoned_houses_etc | Unusable_buildings_abandoned_houses_etc | | | | |
The corporate industries are the same for main and current usage as follows:
| Type of corporate industry Japan Standard Standard Industry Category (13th revision) | |
|--------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Large Class | Middle Class |
| academic research, specialty and technical services | Professional / technical service industry, academic and Development Research Institute, Advertising business |
| accommodation industry, food and drink service | Accommodation business, Restaurant, Take-out / delivery food service industry |
| agriculture, forestry, fisheries (excluding fishery aquaculture) | (excluding fishery aquaculture) |
| compound (complex) service business | Post office, Cooperatives (not classified elsewhere) |
| construction industry | Comprehensive construction business, Job-specific construction business (excluding equipment construction business), Equipment work business (Additional understanding of the terms in the link at the bottom) |
| education, learning support business | School education,Other education and learning support businesses |
| electricity, gas, heat supply, water supply industry | Electricity, Gas industry, Heat supply industry, Water services |
| financial industry, insurance industry | Banking, Cooperative financial industry, Money lending business, credit card business, etc. Non-deposit credit institution, Financial Commodity Trading Business, Commodity Futures Trading Business, Auxiliary financial industry, etc., Insurance business (i |
| information (service) and communication industry | Internet-related service industry, broadcasting industry, Video / audio / text information system work |
| life related service industry | Laundry / Barber / Beauty / Bath business, Other life-related service industry, Entertainment |
| manufacturing | Food manufacturing industry, Beverage / tobacco / feed manufacturing industry, Textile industry, Wood / wood product manufacturing industry (excluding furniture), Pulp / paper / paper processed product manufacturing industry, Printing and related business |
| medical, welfare | Medical industry, Health and hygiene, Social insurance / social welfare / long-term care business |
| mining, quarry business, gravel collection | Mining, quarrying, gravel pitting |
| real estate business, goods leasing business | Real estate transaction business , Real estate leasing and management business, Goods leasing business |
| service industry (Those not classified elsewhere) | Waste disposal industry, Car maintenance industry, Machinery repair business (excluding separate listings), Employment agency / worker dispatching business, Other Business Services, , Religion, Political, economic and cultural organizations, Other service |
| transportation, postal business | Railway industry, Road passenger transportation business, Road freight forwarding business, Water transportation industry, Air transportation industry, Warehouse business, Service industry incidental to transportation, Postal business (including letter s |
| unknown | |
| wholesale business, retail | Various product wholesale business, Wholesale of textiles, clothing, etc., Food and beverage wholesale business, Wholesale business of building materials, minerals, metal materials, etc., Machinery and equipment wholesale business, Other wholesale busines |
- [ ] Here I am trying to figure out their relationship in terms of number of buildings.
- [ ] Figure out if and how the taxonomic strings change if main usage is used instead of current. Is there an improvement ?
\fyi @ds @laurens @taraSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/38Apply suggestions on the albers area code2024-01-16T12:39:52+01:00Simantini ShindeApply suggestions on the albers area codeThere were some suggestions by @tara in this https://git.gfz-potsdam.de/dynamicexposure/globaldynamicexposure/exposure-japan/-/merge_requests/79, apply these to the albers code in utils.py.
\fyi @ds @tara @laurensThere were some suggestions by @tara in this https://git.gfz-potsdam.de/dynamicexposure/globaldynamicexposure/exposure-japan/-/merge_requests/79, apply these to the albers code in utils.py.
\fyi @ds @tara @laurensSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/37Fix the repeat of code for commercial, industrial and residential codes2024-01-16T12:39:24+01:00Simantini ShindeFix the repeat of code for commercial, industrial and residential codesThere are some pieces of code (or functionalities) which are repeated in the residential, commercial and industrial code. This code as @ds pointed out can be inherited from another file . Here is the list of functions and tables which ov...There are some pieces of code (or functionalities) which are repeated in the residential, commercial and industrial code. This code as @ds pointed out can be inherited from another file . Here is the list of functions and tables which overlap right now. I suggest starting with this, and as and when we move on with the industrial exposure, we can find more tables which overlap (I am sure there are more functions which overlap, but I am not entirely clear which ones do as of now)
Tables:
1. `District` table (Overlaps for residential, commercial and industrial exposure).
2. `import_districts_and_boundaries` (Overlaps for residential, commercial and industrial exposure).
\rfc @ds \fyi @taraSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/41Add code to download all boundary files and combine them2024-01-16T12:39:15+01:00Simantini ShindeAdd code to download all boundary files and combine themThe boundaries in E-stat are given for each prefacture and neighborhoods. These boundaries have to be downloaded and combined to form boundaries for municipalities and districts.
\fyi @ds @taraThe boundaries in E-stat are given for each prefacture and neighborhoods. These boundaries have to be downloaded and combined to form boundaries for municipalities and districts.
\fyi @ds @taraSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/29Add code for translation of document2024-01-16T12:38:42+01:00Simantini ShindeAdd code for translation of documentAdd a code to translate the datasets from the e-Stat website.
\rfc @tara @dsAdd a code to translate the datasets from the e-Stat website.
\rfc @tara @dsSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/39Add `_jp` in the filename of the industrial data2024-01-16T12:38:35+01:00Simantini ShindeAdd `_jp` in the filename of the industrial dataMake sure to add `_jp` to the filenames of the Japanese files of the industrial data.
\fyi @ds @taraMake sure to add `_jp` to the filenames of the Japanese files of the industrial data.
\fyi @ds @taraSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/31Add functionality to import datasets from datasources2024-01-16T12:36:26+01:00Simantini ShindeAdd functionality to import datasets from datasourcesAdd a code which imports the data directly from datasources or downloads the data from data sources and then imports this data.
\rfc @ds @LaurensJN @taraAdd a code which imports the data directly from datasources or downloads the data from data sources and then imports this data.
\rfc @ds @LaurensJN @taraSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/26Re-do some code in residential.py to match the same code in commercial.py2024-01-16T12:34:39+01:00Simantini ShindeRe-do some code in residential.py to match the same code in commercial.py- [ ] Remove `reproject_polygon`, `albers_area_calculation` and import it from utils.py
- [ ] Add `add_element_and_get_index` into utils.py and import it from there.
- [ ] Changes in the comments/docstring in the commercial to be added ...- [ ] Remove `reproject_polygon`, `albers_area_calculation` and import it from utils.py
- [ ] Add `add_element_and_get_index` into utils.py and import it from there.
- [ ] Changes in the comments/docstring in the commercial to be added into residential.py.
\fyi @tara @dsSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/21Find structural replacement costs for Japan2024-01-16T12:30:19+01:00Simantini ShindeFind structural replacement costs for JapanCurrently the `structural` value in the BuildingAsset table is kept to 1. Structural replacement costs for each building class in Japan have to be found and inserted into the table.
\fyi @ds @taraCurrently the `structural` value in the BuildingAsset table is kept to 1. Structural replacement costs for each building class in Japan have to be found and inserted into the table.
\fyi @ds @taraSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/17Delete not required constants2024-01-16T12:29:57+01:00Simantini ShindeDelete not required constantsThere are some constants in the constants.py file which are not required, delete them.
\fyi @dsThere are some constants in the constants.py file which are not required, delete them.
\fyi @dsSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/15Some districts missing from Districts table2024-01-16T12:29:45+01:00Simantini ShindeSome districts missing from Districts tableThere are some districts like `47350: Okinawa` is missing from the districts table. This would mean the boundary file has a problem.
FYI: This issue means that we have less districts but it doesn't badly affect our calculation in findin...There are some districts like `47350: Okinawa` is missing from the districts table. This would mean the boundary file has a problem.
FYI: This issue means that we have less districts but it doesn't badly affect our calculation in finding out the districts which are not surveyed in the Japan statistics.
\fyi @ds @kevinSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/9Districts with same district area sizes2024-01-16T12:29:36+01:00Simantini ShindeDistricts with same district area sizesSome districts have same area sizes in the Districts table.
\fyi @dsSome districts have same area sizes in the Districts table.
\fyi @dsSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/35Add a new column with difference of values and update the total2022-08-16T08:45:07+02:00Simantini ShindeAdd a new column with difference of values and update the totalThere is a difference in the total values for `construction material` and `basement presence` for different types of `main usage`.
To solve this, we add an additional `type i.e. unknown` into `basement presence`. The number of building...There is a difference in the total values for `construction material` and `basement presence` for different types of `main usage`.
To solve this, we add an additional `type i.e. unknown` into `basement presence`. The number of buildings in this type is the difference between the `totals`.
i.e. current version:
| main usage | construction material | number_buildings_cm |presence of basement | number_buildings_bp|
| ------ | ------ |------ | ------ | ------ |
| office | total | 12000 | unknwon | - |
| office | unknwon | - | total | 10000 |
Adding additional type
| main usage | construction material | presence of basement | number_buildings|
| ------ | ------ |------ | ------ |
| office | total | total | 12000 |
| office | total | unknown | 2000 |
Also, updates the total for `basement presence` to the ones for `construction material` (here considering that the numbers coming from `construction material` are the correct ones.
\rfc @ds @taraSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/43Remove * in Select statements in commercial.py2022-08-09T15:35:31+02:00Simantini ShindeRemove * in Select statements in commercial.pyRemove * and replace it with the required column names in commercial.py
\fyi @ds @tara @laurensRemove * and replace it with the required column names in commercial.py
\fyi @ds @tara @laurensSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/42Add constants in commercial.py functions2022-08-09T11:25:16+02:00Simantini ShindeAdd constants in commercial.py functions
\fyi @ds @laurens @tara
\fyi @ds @laurens @taraSimantini ShindeSimantini Shindehttps://git.gfz-potsdam.de/globaldynamicexposure/exposure-japan/-/issues/13Fix import_building_numbers2022-06-16T13:18:12+02:00Simantini ShindeFix import_building_numbersFix the repetition of for loop in import_building_numbers function. The second for loop was added to separate building numbers with the same district_id which we don't need anymore.
\fyi @ds @taraFix the repetition of for loop in import_building_numbers function. The second for loop was added to separate building numbers with the same district_id which we don't need anymore.
\fyi @ds @taraSimantini ShindeSimantini Shinde