Commit 79a3d54d authored by Knut Behrends's avatar Knut Behrends
Browse files

improve clarity of USAGE.md; alter some constants in .js files

parent 00e527dd
......@@ -7,7 +7,7 @@ See also [README.md](README.md).
### Configfile
Take a look at file `src./config.js`.
Take a look at file `src/config.js`.
#### baseURL
......@@ -15,9 +15,9 @@ Update the value for `baseURL` by setting it to the URL of your mDIS that you wa
#### Credentials
Login via the web interface once.
Using a Browser, Login via the web interface once.
After successfull login, use the Browser's developer tools (F11 or CTRL+Shift+I) to find the value of the `Authorization` header for your login.
After successfull login, use the Browser's developer tools (F11 or CTRL+Shift+I) to find the value of the `Authorization` HTTP-header for your login request.
This will look like this:
`Authorization: "Bearer eQv809NUzP4447eD3Ws4dBwCOFvf0E8I`
......@@ -29,6 +29,7 @@ Enter this into line 1 of `src/config.js`, or export it as a shell environment v
See also "Configuration" section at [bottom of page](#configuration)
TODO: enable login from command-line
## USAGE
Written in Javascript for the Node Platform v14+. You *must* use node v14+, because the JS code internally uses the `?.` 'optional chaining' operator as a shortcut for definedness of nested object properties.
......@@ -52,9 +53,11 @@ To debug with `.js` files directly, call the node binary on the command line.
### Inserting fake data
> It is usually not necessary to insert lots of expeditions, sites and holes. So do it manually. You must edit the `.js` files.
#### Inserting a Fake Program
Usually the `program` table is not necessary to automate. This can be done by hand. The table has only 3 columns.
Usually it is not necessary to fill the `program` table automatically. Rather, this can be done by hand. The table has only 3 columns.
-----
......@@ -69,6 +72,12 @@ Change the line
`num_expeditions=1`
accordingly, if you want to add more than 1 expedition.
TODO: make this a command-line param. (But it is not really necessary.)
The fake-data generation script will invent some Project-acronyms (e.g. "MWTF") and a fake expedition location, etc.
Note, however, that some columns (e.g. geo-coordinates) will not necessarily consistent with similar inserted into dependent tables. That means, geo-coordinates of *expedition* will not match (= in the vicinity of ) geocoordinates of *sites*, *holes*, etc.
-----
#### Inserting a Fake Site
......@@ -83,7 +92,7 @@ Do not change the line
Instead, you must change file `src/site-inserter.js`.
Change the lines
`let num_sites_to_insert = 1` and
`let num_sites_to_insert = 1`
`let expedition_id = 6`
accordingly, if you want to add any sites to that expedition.
......@@ -105,7 +114,7 @@ Instead, you must change file `src/hole-inserter.js`. In there, change the lines
`let site_id = 6`
accordingly, if you want to add any holes to that site.
You can get the site id from the site form or from an SQL query.
You can get the valuue of the site-id from the mDIS site-form, or from an SQL query.
-----
......@@ -115,40 +124,53 @@ To insert a single new core, call the script with arguments `-n`, `-e`, `-s`, `-
`bin/api-caller-cores.sh -n 2 -e 2`
Alternatively, change the file `src/core-inserter.js`. In there, change the `declare` lines
- `-n` is the number of cores to insert
- `-e` is the expedition id
- `-s` is the site id
- `-h` is the hole id
Alternatively, change the file `src/core-inserter.js`. In there, change the `declare` lines, e.g.
`declare e=11`
accordingly, if you want to add any cores to that expedition.
You can get the ids from the respective forms or from an SQL query.
You can get the ids from the respective mDIS form, or from an SQL query.
##### Troubleshooting
Error `Data Validation Failed`:
Copy the returned JSON structure and insert it into Postman, test the API response. The HTTP response will be easier to work with, and the error message will be more verbose.
- Invalid combination of -e, -s, -h.
- Something else is wrong (mandatory fields missing?).
Copy the returned JSON structure and insert it into Postman, test the API response. The HTTP response will be easier to work with, and the error message will be more verbose, and easier to understand.
-----
#### Inserting a Fake Section
Creating new sections is a complex operation.
There must not be any sections for a core defined previously (the scripts does not check for uplicate entries).
Note: There must not be any sections for a core defined previously (the scripts do not check for duplicate entries). Manually delete fake-data from previous runs, perhaps with an SQL GUI tool.
There are two scripts to execute,
1. `api-caller-core-section-find.sh`,
1. `api-caller-core-section-insert.sh`.
The first writes a JSON file with fake data. The second script reads that file and uploads it to mDIS. The benefit of this two-step approach is that you can have a look at the data before it actually gets created inside mDIS.
The first writes a JSON file with fake data. The second script reads that file and uploads it to mDIS. The benefit of this two-step approach is that you can have a look at the data before it actually gets created inside mDIS. Maybe manually remove known duplicates or preexisting values, and continue inserting.
The two scipts at a glance:
```sh
# at a glance
cd bin # important
./api-caller-core-section-find.sh
./api-caller-core-section-insert.sh
./api-caller-core-section-find.sh # Script 1
./api-caller-core-section-insert.sh # Script 2
```
Script 1:
Core Insertion Scipts - Example
> Script 1:
```sh
./api-caller-core-section-find.sh -e 2 -s 2 -h 2 -f exp-2-manycores-sections.json
......@@ -156,10 +178,10 @@ Finding cores and respective section counts
22 cores found
JSON data file 'section_counts.json' overwritten as requested,
please use this with the 'section-insert.sh' script now
please use this with the './api-caller-core-section-insert.sh' script now
```
Script 2:
> Script 2:
```sh
# looks for file section_counts.json in same directory, and processes it.
......@@ -194,9 +216,9 @@ core 1: 4 sections
'1': '{"core_id":228,"section":1,"top_depth":0.57,"section_length":1.58,"bottom_depth":2.7,"analyst":"CK","section_state":"discontinuous","curated_length":1.45,"comment":"c.1 s.1/4, cL: 2.29m (0.57 - 3.04m)"}'
}
{
'2': {...}, // omitted
'2': {...}, // object details omitted
'3': {...},
'4': {...}
'4': {...} // and many many more....
}
```
......@@ -204,7 +226,9 @@ core 1: 4 sections
#### Inserting a Fake Split
Splits get inserted core-by-core. (write a bash for-loop of you need to insert more)
Splits get inserted core-by-core.
> Single Core:
Command for splitting all sections of a single core:
......@@ -216,6 +240,11 @@ Command for splitting all sections of a single core:
```
> Many Cores:
(write a bash for-loop of you need to insert splits for more than one core:
`for i in $(seq 1 10);do ./api-caller-sectionsplit-insert.sh -c $i; done )`
or see `bin/api-caller-sectionsplit-insert-loop.sh`.
Expected outputs:
Core was _not_ split before:
......@@ -248,6 +277,7 @@ Map(2) {
#### Inserting a Fake scientist
> Before you can insert *samples*, you need to insert *scientists* and their *requests*.
> Before you can insert *scientists*, you need to insert *splits*.
Scientists issue _sample requests_ to express their intention to sample a certain core. The actual sample is taken from a section-split.
......
#!/usr/bin/env bash
# knb 20200714
# add respective fake sections to mDIS
OPTIND=1
declare from=2
declare to=20
########## options and error handling
while getopts ':f:t:' opt
do
case "$opt" in
f) from="${OPTARG}"
;;
t) to=${OPTARG}
;;
*) echo "Usage: $0 [-f|-t] from / to core number )" && exit 0 ;;
esac
done
for c in $(seq $from $to); do
echo "core $c"
. api-caller-sectionsplit-insert.sh -c $c
done
......@@ -4,7 +4,6 @@
OPTIND=1
declare core=""
declare debug=""
declare infile=section_counts.json
########## options and error handling
......
ALTER TABLE
`core_core`
ADD
CONSTRAINT `core_core__project_hole__parent`
FOREIGN KEY (`hole_id`)
REFERENCES `project_hole` (`id`) ON DELETE RESTRICT ON UPDATE CASCADE
\ No newline at end of file
{"splits":[{"split_id":440,"combined_id":"5363_A_AB_10_2_A","expedition_id":10,"scientist":"Dillon Kunze","first_name":"Dillon","last_name":"Kunze"},{"split_id":439,"combined_id":"5363_A_AB_10_2_W","expedition_id":10,"scientist":"Sophia Morar","first_name":"Sophia","last_name":"Morar"},{"split_id":438,"combined_id":"5363_A_AB_10_1_A","expedition_id":10,"scientist":"Yessenia Witting","first_name":"Yessenia","last_name":"Witting"},{"split_id":445,"combined_id":"5363_A_AB_10_1_W","expedition_id":10,"scientist":"Virginia Mueller","first_name":"Virginia","last_name":"Mueller"},{"split_id":443,"combined_id":"5363_A_AB_10_3_A","expedition_id":10,"scientist":"Erick Hansen","first_name":"Erick","last_name":"Hansen"},{"split_id":442,"combined_id":"5363_A_AB_10_3_W","expedition_id":10,"scientist":"Cordia Gislason","first_name":"Cordia","last_name":"Gislason"},{"split_id":444,"combined_id":"5363_A_AB_10_4_A","expedition_id":10,"scientist":"Cynthia Hoeger","first_name":"Cynthia","last_name":"Hoeger"},{"split_id":441,"combined_id":"5363_A_AB_10_4_W","expedition_id":10,"scientist":"Keegan Fadel","first_name":"Keegan","last_name":"Fadel"},{"split_id":416,"combined_id":"5363_A_AB_11_1_A","expedition_id":10,"scientist":"Beulah Lockman","first_name":"Beulah","last_name":"Lockman"},{"split_id":414,"combined_id":"5363_A_AB_11_1_W","expedition_id":10,"scientist":"Ashtyn Wintheiser","first_name":"Ashtyn","last_name":"Wintheiser"},{"split_id":415,"combined_id":"5363_A_AB_11_2_A","expedition_id":10,"scientist":"Shanny Hessel","first_name":"Shanny","last_name":"Hessel"},{"split_id":417,"combined_id":"5363_A_AB_11_2_W","expedition_id":10,"scientist":"Burnice Jerde","first_name":"Burnice","last_name":"Jerde"},{"split_id":419,"combined_id":"5363_A_AB_12_1_A","expedition_id":10,"scientist":"Margaret Quigley","first_name":"Margaret","last_name":"Quigley"},{"split_id":418,"combined_id":"5363_A_AB_12_1_W","expedition_id":10,"scientist":"Jaylen Strosin","first_name":"Jaylen","last_name":"Strosin"},{"split_id":421,"combined_id":"5363_A_AB_13_1_A","expedition_id":10,"scientist":"Lula Wisozk","first_name":"Lula","last_name":"Wisozk"},{"split_id":422,"combined_id":"5363_A_AB_13_1_W","expedition_id":10,"scientist":"Johnnie Hamill","first_name":"Johnnie","last_name":"Hamill"},{"split_id":420,"combined_id":"5363_A_AB_13_3_A","expedition_id":10,"scientist":"Roxane Eichmann","first_name":"Roxane","last_name":"Eichmann"},{"split_id":424,"combined_id":"5363_A_AB_13_3_W","expedition_id":10,"scientist":"Katherine Pacocha","first_name":"Katherine","last_name":"Pacocha"},{"split_id":423,"combined_id":"5363_A_AB_13_2_A","expedition_id":10,"scientist":"Wilfrid Heidenreich","first_name":"Wilfrid","last_name":"Heidenreich"},{"split_id":425,"combined_id":"5363_A_AB_13_2_W","expedition_id":10,"scientist":"Jamar Jacobson","first_name":"Jamar","last_name":"Jacobson"},{"split_id":428,"combined_id":"5363_A_AB_14_1_A","expedition_id":10,"scientist":"Alivia Lesch","first_name":"Alivia","last_name":"Lesch"},{"split_id":427,"combined_id":"5363_A_AB_14_1_W","expedition_id":10,"scientist":"Wanda Bernier","first_name":"Wanda","last_name":"Bernier"},{"split_id":426,"combined_id":"5363_A_AB_14_2_A","expedition_id":10,"scientist":"Amely Hyatt","first_name":"Amely","last_name":"Hyatt"},{"split_id":429,"combined_id":"5363_A_AB_14_2_W","expedition_id":10,"scientist":"Dean Bernhard","first_name":"Dean","last_name":"Bernhard"},{"split_id":431,"combined_id":"5363_A_AB_15_2_A","expedition_id":10,"scientist":"Jazmin Blick","first_name":"Jazmin","last_name":"Blick"},{"split_id":432,"combined_id":"5363_A_AB_15_2_W","expedition_id":10,"scientist":"Vernie Maggio","first_name":"Vernie","last_name":"Maggio"},{"split_id":430,"combined_id":"5363_A_AB_15_1_A","expedition_id":10,"scientist":"Jimmy Toy","first_name":"Jimmy","last_name":"Toy"},{"split_id":434,"combined_id":"5363_A_AB_15_1_W","expedition_id":10,"scientist":"Noemi Hamill","first_name":"Noemi","last_name":"Hamill"},{"split_id":433,"combined_id":"5363_A_AB_15_4_A","expedition_id":10,"scientist":"Vilma Sipes","first_name":"Vilma","last_name":"Sipes"},{"split_id":435,"combined_id":"5363_A_AB_15_4_W","expedition_id":10,"scientist":"Aliya Bogan","first_name":"Aliya","last_name":"Bogan"},{"split_id":436,"combined_id":"5363_A_AB_15_3_A","expedition_id":10,"scientist":"Leopold Christiansen","first_name":"Leopold","last_name":"Christiansen"},{"split_id":437,"combined_id":"5363_A_AB_15_3_W","expedition_id":10,"scientist":"Katelyn Kessler","first_name":"Katelyn","last_name":"Kessler"}]}
\ No newline at end of file
{
"splits": [
{
"split_id": 440,
"combined_id": "5363_A_AB_10_2_A",
"expedition_id": 10,
"scientist": "Dillon Kunze",
"first_name": "Dillon",
"last_name": "Kunze"
},
{
"split_id": 439,
"combined_id": "5363_A_AB_10_2_W",
"expedition_id": 10,
"scientist": "Sophia Morar",
"first_name": "Sophia",
"last_name": "Morar"
},
{
"split_id": 438,
"combined_id": "5363_A_AB_10_1_A",
"expedition_id": 10,
"scientist": "Yessenia Witting",
"first_name": "Yessenia",
"last_name": "Witting"
},
{
"split_id": 445,
"combined_id": "5363_A_AB_10_1_W",
"expedition_id": 10,
"scientist": "Virginia Mueller",
"first_name": "Virginia",
"last_name": "Mueller"
},
{
"split_id": 443,
"combined_id": "5363_A_AB_10_3_A",
"expedition_id": 10,
"scientist": "Erick Hansen",
"first_name": "Erick",
"last_name": "Hansen"
},
{
"split_id": 442,
"combined_id": "5363_A_AB_10_3_W",
"expedition_id": 10,
"scientist": "Cordia Gislason",
"first_name": "Cordia",
"last_name": "Gislason"
},
{
"split_id": 444,
"combined_id": "5363_A_AB_10_4_A",
"expedition_id": 10,
"scientist": "Cynthia Hoeger",
"first_name": "Cynthia",
"last_name": "Hoeger"
},
{
"split_id": 441,
"combined_id": "5363_A_AB_10_4_W",
"expedition_id": 10,
"scientist": "Keegan Fadel",
"first_name": "Keegan",
"last_name": "Fadel"
},
{
"split_id": 416,
"combined_id": "5363_A_AB_11_1_A",
"expedition_id": 10,
"scientist": "Beulah Lockman",
"first_name": "Beulah",
"last_name": "Lockman"
},
{
"split_id": 414,
"combined_id": "5363_A_AB_11_1_W",
"expedition_id": 10,
"scientist": "Ashtyn Wintheiser",
"first_name": "Ashtyn",
"last_name": "Wintheiser"
},
{
"split_id": 415,
"combined_id": "5363_A_AB_11_2_A",
"expedition_id": 10,
"scientist": "Shanny Hessel",
"first_name": "Shanny",
"last_name": "Hessel"
},
{
"split_id": 417,
"combined_id": "5363_A_AB_11_2_W",
"expedition_id": 10,
"scientist": "Burnice Jerde",
"first_name": "Burnice",
"last_name": "Jerde"
},
{
"split_id": 419,
"combined_id": "5363_A_AB_12_1_A",
"expedition_id": 10,
"scientist": "Margaret Quigley",
"first_name": "Margaret",
"last_name": "Quigley"
},
{
"split_id": 418,
"combined_id": "5363_A_AB_12_1_W",
"expedition_id": 10,
"scientist": "Jaylen Strosin",
"first_name": "Jaylen",
"last_name": "Strosin"
},
{
"split_id": 421,
"combined_id": "5363_A_AB_13_1_A",
"expedition_id": 10,
"scientist": "Lula Wisozk",
"first_name": "Lula",
"last_name": "Wisozk"
},
{
"split_id": 422,
"combined_id": "5363_A_AB_13_1_W",
"expedition_id": 10,
"scientist": "Johnnie Hamill",
"first_name": "Johnnie",
"last_name": "Hamill"
},
{
"split_id": 420,
"combined_id": "5363_A_AB_13_3_A",
"expedition_id": 10,
"scientist": "Roxane Eichmann",
"first_name": "Roxane",
"last_name": "Eichmann"
},
{
"split_id": 424,
"combined_id": "5363_A_AB_13_3_W",
"expedition_id": 10,
"scientist": "Katherine Pacocha",
"first_name": "Katherine",
"last_name": "Pacocha"
},
{
"split_id": 423,
"combined_id": "5363_A_AB_13_2_A",
"expedition_id": 10,
"scientist": "Wilfrid Heidenreich",
"first_name": "Wilfrid",
"last_name": "Heidenreich"
},
{
"split_id": 425,
"combined_id": "5363_A_AB_13_2_W",
"expedition_id": 10,
"scientist": "Jamar Jacobson",
"first_name": "Jamar",
"last_name": "Jacobson"
},
{
"split_id": 428,
"combined_id": "5363_A_AB_14_1_A",
"expedition_id": 10,
"scientist": "Alivia Lesch",
"first_name": "Alivia",
"last_name": "Lesch"
},
{
"split_id": 427,
"combined_id": "5363_A_AB_14_1_W",
"expedition_id": 10,
"scientist": "Wanda Bernier",
"first_name": "Wanda",
"last_name": "Bernier"
},
{
"split_id": 426,
"combined_id": "5363_A_AB_14_2_A",
"expedition_id": 10,
"scientist": "Amely Hyatt",
"first_name": "Amely",
"last_name": "Hyatt"
},
{
"split_id": 429,
"combined_id": "5363_A_AB_14_2_W",
"expedition_id": 10,
"scientist": "Dean Bernhard",
"first_name": "Dean",
"last_name": "Bernhard"
},
{
"split_id": 431,
"combined_id": "5363_A_AB_15_2_A",
"expedition_id": 10,
"scientist": "Jazmin Blick",
"first_name": "Jazmin",
"last_name": "Blick"
},
{
"split_id": 432,
"combined_id": "5363_A_AB_15_2_W",
"expedition_id": 10,
"scientist": "Vernie Maggio",
"first_name": "Vernie",
"last_name": "Maggio"
},
{
"split_id": 430,
"combined_id": "5363_A_AB_15_1_A",
"expedition_id": 10,
"scientist": "Jimmy Toy",
"first_name": "Jimmy",
"last_name": "Toy"
},
{
"split_id": 434,
"combined_id": "5363_A_AB_15_1_W",
"expedition_id": 10,
"scientist": "Noemi Hamill",
"first_name": "Noemi",
"last_name": "Hamill"
},
{
"split_id": 433,
"combined_id": "5363_A_AB_15_4_A",
"expedition_id": 10,
"scientist": "Vilma Sipes",
"first_name": "Vilma",
"last_name": "Sipes"
},
{
"split_id": 435,
"combined_id": "5363_A_AB_15_4_W",
"expedition_id": 10,
"scientist": "Aliya Bogan",
"first_name": "Aliya",
"last_name": "Bogan"
},
{
"split_id": 436,
"combined_id": "5363_A_AB_15_3_A",
"expedition_id": 10,
"scientist": "Leopold Christiansen",
"first_name": "Leopold",
"last_name": "Christiansen"
},
{
"split_id": 437,
"combined_id": "5363_A_AB_15_3_W",
"expedition_id": 10,
"scientist": "Katelyn Kessler",
"first_name": "Katelyn",
"last_name": "Kessler"
}
]
}
let token = process.env.token || "GIzxYfRCLXSEegaLm_CbEgEbvl1qPUuK"
let token = process.env.token || "Y2riMvNQIO8VNj4Cubv6X4DN-V85bmto"
let config = {
token: token,
endpoint: {
// baseURL: "https://internal.rundis.com/",
//baseURL: "http://172.17.0.1:8888/",
baseURL: "https://data.icdp-online.org/mdis/jet/",
baseURL: "https://data.icdp-online.org/mdis/internal/",
timeout: 3000,
headers: {
"Authorization": `Bearer ${token}`,
......
......@@ -39,7 +39,7 @@ const api_url_frag = `/api/v1/form`
const qs_splits = {
name: "split",
"per-page": -1,
"per-page": 100,
page: 1,
sort: "id",
"filter[type]": "[^W$]",
......
......@@ -14,7 +14,7 @@ const util = require("./classes/fake-util.js")
// this URL gets holes sorted descending, (newest holes first)
let num_holes_to_insert = 1
let program_id = 1
let site_id = 6
let site_id = 7
//let expedition_id = irrelevant here
const api_url_frag = `/api/v1/form?`
let qs_hole = {
......
......@@ -23,7 +23,7 @@ const options = getopts(process.argv.slice(2), {
//infile: "i", // alternative
},
default: {
expedition: 10, //CAIR
expedition: 2, //
},
})
......@@ -48,7 +48,7 @@ const qs_request = {
}
const qs_splits = {
name: "split",
"per-page": -1,
"per-page": 200,
page: 1,
sort: "id",
"filter[type]": "[^A]",
......@@ -212,7 +212,7 @@ Promise.all(queries_map.values())
})
.then(() => {
axutil
.get_columns(ax, "curation_sample_2")
.get_columns(ax, "curation_sample")
.then((curation_sample) => {
let fake_sample_data = []
let i = 0
......
......@@ -28,7 +28,6 @@ let outfile = options.outfile
// filter[hole_id]=${hole_id}
//&filter[expedition_id]=${expedition_id}&filter[site_id]=${site_id}&
let split_url = `/api/v1/form?name=split&&per-page=1000&page=1&sort=core&filter[hole_id]=${hole_id}`
let section_count = 0
let ax = axios.create(config.endpoint)
//const core = new Core()
......@@ -43,7 +42,6 @@ ax.get(split_url)
split_id: element.id,
combined_id: element.combined_id,
// core_id: element.core,
// section_count: element.last_section,
// recovered_length: element.core_recovery,
expedition_id: element.expedition_id,
//site_id: element.site_id,
......
......@@ -12,7 +12,7 @@ const Site = require("./classes/fake-sites.js")
// this URL gets sites sorted descending, (newest sites first)
let num_sites_to_insert = 1
let program_id = 1
let expedition_id = 6
let expedition_id = 5
const api_url_frag = `/api/v1/form?`
let qs_site = {
"per-page": 100,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment