Commit fcb27086 authored by Knut Behrends's avatar Knut Behrends
Browse files

Better document shell scripts

parent cccc33c8
......@@ -20,3 +20,4 @@ scientists-hole-21-enty.json
scientists-hole-8-samp.json
scientists-hole-9-samp.json
.vscode
scientists-hole-4-NLPDP.json
......@@ -2,12 +2,21 @@
## README
A set of scripts generating fake-data and testdata for a new mDIS instance.
### Synopsis
A set of scripts generating fake-data and testdata for a new mDIS instance.
Written in Javascript for the Node Platform (v14+).
For details: read [USAGE.md](USAGE.md)
### .sh and .js Scripts available
There is a set of shellscripts in the `bin/` subdirectory. They are just simple wrappers for the js Files. Each Shell scripts load the node v14 for you.
You can call the JavaScript files directly as well, but then you have to make sure that you are using Node 14+.
### See also
[USAGE.md](USAGE.md)
knb@gfz-potsdam.de
ICDP-OSG 2020
© ICDP-OSG 2020
# dis-datagen
## USAGE
A set of scripts generating fake-data and testdata for a new mDIS instance.
See also [README.md](README.md).
## Setup
Take a look at file `src./config.js`.
### baseURL
Update the value for `baseURL` by setting it to the URL of your mDIS that you want to fill with fake-data.
### Credentials
A set of scripts generating fake-data and testdata for a new mDIS instance.
Login via the web interface once.
After successfull login, use the Browser's developer tools (F11 or CTRL+Shift+I) to find the value of the `Authorization` header for your login.
This will look like this:
`Authorization: "Bearer eQv809NUzP4447eD3Ws4dBwCOFvf0E8I`
Copy the value after `Bearer`.
Here you need `eQv809NUzP4447eD3Ws4dBwCOFvf0E8I`.
Enter this into line 1 of config.js, or export it as a shell environment variable `token`:
`export token=eQv809NUzP4447eD3Ws4dBwCOFvf0E8I`.
## USAGE
Written in Javascript for the Node Platform v14+
You must use node v14 because the JS code internaly uses the `?.` 'optional chaining' operator as a shortcut for definedness of nested object properties.
### Inserting fake data
First, look at the `bin/` subdirectory to execute Shell scripts, or enter the `src/`subdirectory to execute Node scripts.
For regular usage, call the scripts from the parent directory.
### Debugging
First, cd into the `bin/` subdirectory to execute Shell scripts,
or enter the `src/`subdirectory to execute Node scripts.
To enable debugging with the Chrome Developer Tools, open any shellscript `bin/api_caller*.sh`, comment out the line
`node src/expedition-inserter.js 1;`
and uncomment line
`#node --inspect-brk --inspect ../src/expedition-inserter.js 1;`
### Inserting fake data
#### Inserting a Fake Program
Usually the `program` table is not necessary to automate. This can be done by hand. The table has only 3 columns.
#### Inserting a Fake Expedition
To insert a single new expedition, call the script without arguments:
`bin/api-caller-expedition.sh`
The script does not take arguments.
Change the line
`num_expeditions=1`
accordingly, if you want to add more than 1 expedition.
#### Inserting a Fake Site
To insert a single new site, call the script without arguments:
`bin/api-caller-sites.sh`
The script does not take arguments.
Do not change the line
`num_sites=1` if you want to add more than 1 site.
Instead, you must change file `src/site-inserter.js`.
Change the lines
`let num_sites_to_insert = 1` and
`let expedition_id = 6`
accordingly, if you want to add any sites to that expedition.
You can get the expedition id from the expedition form or from an SQL query.
#### Inserting a Fake Hole
To insert a single new hole, call the script without arguments:
`bin/api-caller-hole.sh`
Do *not* change the line
`num_holes=1` if you want to add more than 1 site.
Instead, you must change file `src/hole-inserter.js`. In there, change the lines
`let num_holes_to_insert = 1` and
`let site_id = 6`
accordingly, if you want to add any holes to that site.
You can get the site id from the site form or from an SQL query.
#### Inserting a Fake Core
To insert a single new core, call the script with arguments `-n`, `-e`, `-s`, `-h`:
`bin/api-caller-cores.sh -n 2 -e 2`
Alternatively, change the file `src/core-inserter.js`. In there, change the `declare` lines
`declare e=11`
accordingly, if you want to add any cores to that expedition.
You can get the ids from the respective forms or from an SQL query.
##### Troubleshooting
Error `Data Validation Failed`:
Copy the returned JSON structure and insert it into Postman, test the API response. The HTTP response will be easier to work with, and the error message will be more verbose.
#### Inserting a Fake Section
Creating new sections is a complex operation.
......@@ -32,10 +120,10 @@ There are two scripts to execute,
1. `api-caller-core-section-find.sh`,
1. `api-caller-core-section-insert.sh`.
The first writes a JSON file with fake data. The second script reads that file and uploads it to mDIS. The benefit of this two-step approach is tha you can have a look at the data before it actually gets created inside mDIS.
The first writes a JSON file with fake data. The second script reads that file and uploads it to mDIS. The benefit of this two-step approach is that you can have a look at the data before it actually gets created inside mDIS.
```sh
cd bin
cd bin # important
./api-caller-core-section-find.sh
./api-caller-core-section-insert.sh
......@@ -56,7 +144,7 @@ Script 2:
```sh
# looks for file section_counts.json in same directory, and processes it.
./api-caller-core-section-insert.sh
./api-caller-core-section-insert.sh
# use -i to process a different infile.
# ./api-caller-core-section-insert.sh -i infile.json
......@@ -95,7 +183,9 @@ core 1: 4 sections
#### Inserting a Fake Split
Command:
Splits get inserted core-by-core. (write a bash for-loop of you need to insert more)
Command for splitting all sections of a single core:
```sh
./api-caller-sectionsplit-insert.sh -c 226 # core 226
......@@ -134,11 +224,15 @@ Map(2) {
#### Inserting a Fake scientist
> Before you can insert *samples*, you need to insert *scientists* and their *requests*.
Scientists issue _sample requests_ to express their intention to sample a certain core. The actual sample is taken from a section-split.
For now, assign scientists on a per-hole-level.
These scientists will then later express interest in splits from _all_ sites and holes from that expedition.
> No shellscripts available. Call `.js` files.
`# write new science team to a JSON file`
`node src/scientist-finder.js -h 20 > scientists-hole-20-enty.json`
......
......@@ -3,10 +3,10 @@
# add some fake core data to mdis
OPTIND=1
declare debug=""
declare e=11
declare s=4
declare h=11
declare n=1
declare e=6
declare s=6
declare h=4
declare n=2
while getopts ':d:e:s:h:n:' opt
do
case "$opt" in
......@@ -37,6 +37,6 @@ for i in $(seq 1 $num_cores) ; do
# insert 1 core synchronously,
# as a copy of the "previous" core
# n must be 1 here for now
node $debug ../src/core-inserter.js -n 1 -e ${e} -s ${s} -h ${h}
node $debug src/core-inserter.js -n 1 -e ${e} -s ${s} -h ${h}
done
\ No newline at end of file
......@@ -2,6 +2,8 @@
# knb 20200714
# add some fake expedition data to mdis
[[ "well" == $(uname -n) ]] && export NODE_PATH=~/code/git/_my/work/dis-datagen/node_modules
# how many expeditions to add
num_expeditions=1 #20
export NVM_DIR="$HOME/.nvm"
......
......@@ -2,8 +2,11 @@
# knb 20200714
# add some fake hole data to mdis
[[ "well" == $(uname -n) ]] && export NODE_PATH=~/code/git/_my/work/dis-datagen/node_modules
# how many holes to add
num_holes=1 #20
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" # This loads nvm
nvm use 14
......@@ -14,6 +17,6 @@ for i in $(seq 1 $num_holes) ; do
echo hole $i of $num_holes;
# insert 1 hole synchronously,
# as a copy of the "previous" hole
node --inspect-brk --inspect ../src/hole-inserter.js 1;
# node src/hole-inserter.js 1;
#node --inspect-brk --inspect ../src/hole-inserter.js 1;
node src/hole-inserter.js 1;
done
\ No newline at end of file
......@@ -2,8 +2,10 @@
# knb 20200714
# add some fake site data to mdis
[[ "well" == $(uname -n) ]] && export NODE_PATH=~/code/git/_my/work/dis-datagen/node_modules
# how many sites to add
num_sites=3 #20
num_sites=1 #20
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" # This loads nvm
nvm use 14
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment