I am calling python command which will return data is JSON key-value pair.
I have put python command and other command in one shell script named as - a.sh
Code (a.sh):
cd /home/drg/Code/dth
a=$(python3 main.py -z shell -y droub -i 56)
echo "$a"
When I am calling this script I am getting output as:
{'password': 'XYZ', 'name': 'Stguy', 'port': '5412', 'host': 'igtet', 'db_name': 'test3'}
And after getting this output I want to pass the output value like password, name to psql command to run postgresql query.
So, what I want is that I should be able to store password value in one variable, name in one variable like:
a= xyz
b=Stguy
p= port
So, that I can use this variables to pass in psql query as:
psql -h $a -p $p -U $b -d $db -c "CREATE SCHEMA IF NOT EXISTS ${sname,,};"
Can someone please help me with this?
Note: Env is linux(Centos 8)
Thanks in advance!
One way of solving this could be a combination of jq for value extraction and shell-builtin read for multiple variable assignment:
JSON='{"name": "Stguy", "port": 5412, "host": "igtet", "db_name": "test3"}'
read -r a b c <<<$( echo $JSON | jq -r '"\(.host) \(.port) \(.name)"' )
echo "a: $a, b: $b, c: $c"
doing jq string interpolation "\( )" to print result in one line
You can aslo go with sed or awk:
PSQL="$( python3 main.py -z shell -y droub -i 56 | sed "s/^[^:]*: *'\([^']*\)'[^:]*: *'\([^']*\)'[^:]*: *'\([^']*\)'[^:]*: *'\([^']*\)'[^:]*: *'\([^']*\)'}/psql -h '\4' -p '\1' -U '\2' -d '\5'/")"
[ "${PSQL:0:5}" = "psql " ] && ${PSQL} -c "CREATE SCHEMA IF NOT EXISTS ${sname,,};"
For security consideration, i urge you anyway to avoid passing account data (user passwd) through environment variables.
It would be better if your python script had an option to directly launch psql with required parameters.
Related
I am writing a bash script that reads a JSON string then loop based on the JSON values to execute a CLI command.
#!/usr/bin/env bash
jq --version > /dev/null 2>&1 || { echo >&2 "jq is required but it's not installed. Aborting."; exit 1; }
read -r -d '' USER_ACTIONS << EOM
{
"user1": [
"action1"
],
"user2": [
"action2",
"action3"
]
}
EOM
USERS= #TODO
for user in USERS; do
ACTIONS= #TODO
for action in ACTIONS; do
echo "Executing ${command} ${user}-${action}"
done
done
If jq is present in the server, how do I populate the USERS and ACTIONS variable?
Depending on what command you want to execute, if it can be performed from within jq, it's easier to also move the loop inside. There are several ways to accomplish that. Here are some examples, all yielding the same output:
jq -r 'to_entries[] | "Executing command \(.key)-\(.value[])"' <<< "$USER_ACTIONS"
jq -r 'keys[] as $user | "Executing command \($user)-\(.[$user][])"' <<< "$USER_ACTIONS"
jq -r --stream '"Executing command \(.[0][0])-\(.[1]? // empty)"' <<< "$USER_ACTIONS"
Output:
Executing command user1-action1
Executing command user2-action2
Executing command user2-action3
It seems better to play with vanilla js (nodejs):
const myvar={
"user1": [
"action1"
],
"user2": [
"action2",
"action3"
]
};
for (let user in myvar) {
myvar[user].forEach((action) => {
console.log("Executing command " + user + "-" + action);
});
}
Output
Executing command user1-action1
Executing command user2-action2
Executing command user2-action3
Usage
node script.js
Usage with bash
You can remove Executing string, then:
node script.js | bash
Would you please try the following:
#!/bin/bash
declare -a users # array of users
declare -A actions # array of actions indexed by user
read -r -d '' user_actions << EOM
{
"user1": [
"action1"
],
"user2": [
"action2",
"action3"
]
}
EOM
while IFS=$'\t' read -r key val; do
users+=( "$key" ) # add the user to the array "users"
actions[$key]="$val" # associate the actions with the user
done < <(jq -r 'to_entries[] | [.key, .value[]] | #tsv' <<< "$user_actions")
for user in "${users[#]}"; do # loop over "users"
IFS=$'\t' read -r -a vals <<< "${actions[$user]}"
for action in "${vals[#]}"; do
echo yourcommand "$user" "$action"
done
done
Output:
yourcommand user1 action1
yourcommand user2 action2
yourcommand user2 action3
[Explanations]
The jq command outputs TSV which looks like:
user1\taction1
user2\taction2\taction3
where \t represents a tab character used as a field delimiter.
The first read builtin command assigns key to the first field and val
to the remaining field(s). If val contains two or more fields,
it will be splitted with tne next read builtin in the next loop.
if the output looks good, drop echo and replace the string yourcommand with the command name.
SHELL code as follow:
#!/bin/sh
..// etc
ssh root#127.0.0.1 << EOF
KEYS=$(mysql -h ${DB_IP} -u ${USERNAME} -p${PASSWORD} -P ${DB_PORT} --database bosdb -e "select a from B where id=1");
echo "123";
echo $KEYS;
... // etc
EOF
When I run this script, the output text is
123
(this is a NULL line, that means KEYS is null)
I have tried if I log in to the machine of MYSQL directly, I can get the result of the "MYSQL" command, and the echo $KEYS also shows a string of values. But when I used subcommand as input, it didn't work.
So, how can I get the value of KEYS correctly? I'd really appreciate if someone can help me
I am trying to store MySQL result into a global bash array variable but I don't know how to do it.
Should I save the MySQL command result in a file and read the file line by line in my for loop for my other treatment?
Example:
user password
Pierre aaa
Paul bbb
Command:
$results = $( mysql –uroot –ppwd –se « SELECT * from users );
I want that results contains the two rows.
Mapfile for containing whole table into one bash variable
You could try this:
mapfile result < <(mysql –uroot –ppwd –se "SELECT * from users;")
Than
echo ${result[0]%$'\t'*}
echo ${result[0]#*$'\t'}
or
for row in "${result[#]}";do
echo Name: ${row%$'\t'*} pass: ${row#*$'\t'}
done
Nota This will work fine while there is only 2 fields by row. More is possible but become tricky
Read for reading table row by row
while IFS=$'\t' read name pass ;do
echo name:$name pass:$pass
done < <(mysql -uroot –ppwd –se "SELECT * from users;")
Read and loop to hold whole table into many variables:
i=0
while IFS=$'\t' read name[i] pass[i++];do
:;done < <(mysql -uroot –ppwd –se "SELECT * from users;")
echo ${name[0]} ${pass[0]}
echo ${name[1]} ${pass[1]}
New (feb 2018) shell connector
There is a little tool (on github) or on my own site: (shellConnector.sh you could use:
Some preparation:
cd /tmp/
wget -q http://f-hauri.ch/vrac/shell_connector.sh
. shell_connector.sh
newSqlConnector /usr/bin/mysql '–uroot –ppwd'
Following is just for demo, skip until test for quick run
Thats all. Now, creating temporary table for demo:
echo $SQLIN
3
cat >&3 <<eof
CREATE TEMPORARY TABLE users (
id bigint(20) unsigned NOT NULL PRIMARY KEY AUTO_INCREMENT,
name VARCHAR(30), date DATE)
eof
myMysql myarray ';'
declare -p myarray
bash: declare: myarray: not found
The command myMysql myarray ';' will send ; then execute inline command,
but as mysql wont anwer anything, variable $myarray wont exist.
cat >&3 <<eof
INSERT INTO users VALUES (1,'alice','2015-06-09 22:15:01'),
(2,'bob','2016-08-10 04:13:21'),(3,'charlie','2017-10-21 16:12:11')
eof
myMysql myarray ';'
declare -p myarray
bash: declare: myarray: not found
Operational Test:
Ok, then now:
myMysql myarray "SELECT * from users;"
printf "%s\n" "${myarray[#]}"
1 alice 2015-06-09
2 bob 2016-08-10
3 charlie 2017-10-21
declare -p myarray
declare -a myarray=([0]=$'1\talice\t2015-06-09' [1]=$'2\tbob\t2016-08-10' [2]=$'3\tcharlie\t2017-10-21')
This tool are in early step of built... You have to manually clear your variable before re-using them:
unset myarray
myMysql myarray "SELECT name from users where id=2;"
echo $myarray
bob
declare -p myarray
declare -a myarray=([0]="bob")
If you're looking to get a global variable inside your script you can simply assign a value to a varname:
VARNAME=('var' 'name') # no space between the variable name and value
Doing this you'll be able to access VARNAME's value anywhere in your script after you initialize it.
If you want your variable to be shared between multiple scripts you have to use export:
script1.sh:
export VARNAME=('var' 'name')
echo ${VARNAME[0]} # will echo 'var'
script2.sh
echo ${VARNAME[1]} # will echo 'name', provided that
# script1.sh was executed prior to this one
NOTE that export will work only when running scripts in the same shell instance. If you want it to work cross-instance you would have to put the export variable code somewhere in .bashrc or .bash_profile
The answer from #F. Hauri seems really complicated.
https://stackoverflow.com/a/38052768/470749 helped me realize that I needed to use parentheses () wrapped around the query result to treat is as an array.
#You can ignore this function since you'll do something different.
function showTbl {
echo $1;
}
MOST_TABLES=$(ssh -vvv -t -i ~/.ssh/myKey ${SERVER_USER_AND_IP} "cd /app/ && docker exec laradock_mysql_1 mysql -u ${DB} -p${REMOTE_PW} -e 'SELECT table_name FROM information_schema.tables WHERE table_schema = \"${DB}\" AND table_name NOT LIKE \"pma_%\" AND table_name NOT IN (\"mail_webhooks\");'")
#Do some string replacement to get rid of the query result header and warning. https://stackoverflow.com/questions/13210880/replace-one-substring-for-another-string-in-shell-script
warningToIgnore="mysql\: \[Warning\] Using a password on the command line interface can be insecure\."
MOST_TABLES=${MOST_TABLES/$warningToIgnore/""}
headerToIgnore="table_name"
MOST_TABLES=${MOST_TABLES/$headerToIgnore/""}
#HERE WAS THE LINE THAT I NEEDED TO ADD! Convert the string to array:
MOST_TABLES=($MOST_TABLES)
for i in ${MOST_TABLES[#]}; do
if [[ $i = *[![:space:]]* ]]
then
#Remove whitespace from value https://stackoverflow.com/a/3232433/470749
i="$(echo -e "${i}" | tr -d '[:space:]')"
TBL_ARR+=("$i")
fi
done
for t in ${TBL_ARR[#]}; do
showTbl $t
done
This successfully shows me that ${TBL_ARR[#]} has all the values from the query result.
results=($( mysql –uroot –ppwd –se "SELECT * from users" ))
if [ "$?" -ne 0 ]
then
echo fail
exit
fi
I was wondering how to parse the CURL JSON output from the server into variables.
Currently, I have -
curl -X POST -H "Content: agent-type: application/x-www-form-urlencoded" https://www.toontownrewritten.com/api/login?format=json -d username="$USERNAME" -d password="$PASSWORD" | python -m json.tool
But it only outputs the JSON from the server and then have it parsed, like so:
{
"eta": "0",
"position": "0",
"queueToken": "6bee9e85-343f-41c7-a4d3-156f901da615",
"success": "delayed"
}
But how do I put - for example the success value above returned from the server into a variable $SUCCESS and have the value as delayed & have queueToken as a variable $queueToken and 6bee9e85-343f-41c7-a4d3-156f901da615 as a value?
Then when I use-
echo "$SUCCESS"
it shows this as the output -
delayed
And when I use
echo "$queueToken"
and the output as
6bee9e85-343f-41c7-a4d3-156f901da615
Thanks!
Find and install jq (https://stedolan.github.io/jq/). jq is a JSON parser. JSON is not reliably parsed by line-oriented tools like sed because, like XML, JSON is not a line-oriented data format.
In terms of your question:
source <(
curl -X POST -H "$content_type" "$url" -d username="$USERNAME" -d password="$PASSWORD" |
jq -r '. as $h | keys | map(. + "=\"" + $h[.] + "\"") | .[]'
)
The jq syntax is a bit weird, I'm still working on it. It's basically a series of filters, each pipe taking the previous input and transforming it. In this case, the end result is some lines that look like variable="value"
This answer uses bash's "process substitution" to take the results of the jq command, treat it like a file, and source it into the current shell. The variables will then be available to use.
Here's an example of Extract a JSON value from a BASH script
#!/bin/bash
function jsonval {
temp=`echo $json | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w $prop`
echo ${temp##*|}
}
json=`curl -s -X GET http://twitter.com/users/show/$1.json`
prop='profile_image_url'
picurl=`jsonval`
`curl -s -X GET $picurl -o $1.png`
A bash script which demonstrates parsing a JSON string to extract a
property value. The script contains a jsonval function which operates
on two variables, json and prop. When the script is passed the name of
a twitter user it attempts to download the user's profile picture.
You could use perl module on command line:
1st, ensure they is installed, under debian based, you could
sudo apt-get install libjson-xs-perl
But for other OS, you could install perl modules via CPAN (the Comprehensive Perl Archive Network):
cpan App::cpanminus
cpan JSON::XS
Note: You may have to run this with superuser privileges.
then:
curlopts=(-X POST -H
"Content: apent-type: application/x-www-form-urlencoded"
-d username="$USERNAME" -d password="$PASSWORD")
curlurl=https://www.toontownrewritten.com/api/login?format=json
. <(
perl -MJSON::XS -e '
$/=undef;my $a=JSON::XS::decode_json <> ;
printf "declare -A Json=\047(%s)\047\n", join " ",map {
"[".$_."]=\"".$a->{$_}."\""
} qw|queueToken success eta position|;
' < <(
curl "${curlopts[#]}" $curlurl
)
)
The line qw|...| let you precise which variables you want to be driven... This could be replaced by keys $a, but could have to be debugged as some characters is forbiden is associative arrays values names.
echo ${Json[queueToken]}
6bee9e85-343f-41c7-a4d3-156f901da615
echo ${Json[eta]}
0
This is a second part of Read from file into variable - Bash Script
I have a bash script that reads strings in a file parses and assigns it to a variable. The file looks like this (file.txt):
database1 table1
database1 table4
database2
database3 table2
Using awk in the script:
s=$(awk '{$1=$1}1' OFS='.' ORS='|' file.txt)
LIST="${s%|}"
echo "$LIST"
database1.table1|database1.table4|database2|database3.table2
But I need to add some wildcards at the end of each substring. I need this result:
database1.table1.*|database1.table4.*|database2*.*|database3.table2.*
The conditions are: if we read database2 the output should be database2*.* and if we read a database and a table the output should be database1.table1.*
Use this awk with ORS='.*|':
s=$(awk '$0=="database2"{$0=$0 "*.*";print;next} {$2=$2 ".*"}1' OFS='.' ORS='|' file.txt)
LIST="${s%|}"
echo "$LIST"
database1.table1.*|database1.table4.*|database2*.*|database3.table2.*
Assuming the (slightly odd) regex is correct the following awk script works for me on your example input.
BEGIN {OFS="."; ORS="|"}
!$2 {$1=$1"*"}
{$(NF+1)="*"}
1
Set OFS and ORS.
If we do not have a second field add a * to our first field.
Add a * as a final field.
Print the line.
Run as awk -f script.awk inputfile where the above script is in the script.awk (or whatever) file.
I'd do it like this.
script.sh containing the following code:
#!/bin/bash
while IFS='' read -r line ;do
database=$(awk '{print $1}' <<< $line)
table=$(awk '{print $2}' <<< $line)
if [ "${table}" == '' ] ;then
list=(${list}\|${database}\*\.\*)
else
list=(${list}\|${database}\.${table}\.\*)
fi
done <file.txt
list=$(cut -c 2- <<< ${list})
echo ${list}
exit 0
file.txt containing the following data:
database1 table1
database1 table4
database2
database3 table2
Script output is the following:
database1.table1.*|database1.table4.*|database2*.*|database3.table2.*
Tested in BASH version:
GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)