Pass JSON via command line whilst command uses both sets of quotes - json

I am trying to send this query to my command line:
docker exec cardano-node sh -c 'echo {
"type": "PaymentExtendedSigningKeyShelley_ed25519_bip32",
"description": "Payment Signing Key",
"cborHex": "xxxxx"
}
> /tmp/payment.skey '
The issue with this command is that it is malformed and I need to wrap the JSON in quotes, but single quotes are already being used by the parent command. I don't have access to change the JSON. How can I escape some of the quotes or change this query to pass this command to my docker container?

If Your Container Uses Bash
Let the shell do the quoting for you. If the copy of sh you're dealing with is provided by bash (the easiest case) and your host has bash 5.0 or newer, that looks like:
json='{
"type": "PaymentExtendedSigningKeyShelley_ed25519_bip32",
"description": "Payment Signing Key",
"cborHex": "xxxxx"
}'
docker exec cardano-node bash -c "echo ${json#Q} >/tmp/payment.skey"
...or, compatible with older versions of bash, you can ship a function into the container:
writeFile() { cat >/tmp/payment.skey <<'EOF'
{
"type": "PaymentExtendedSigningKeyShelley_ed25519_bip32",
"description": "Payment Signing Key",
"cborHex": "xxxxx"
}
EOF
}
docker-exec cardano-node sh -c "$(declare -f writeFile); writeFile"
Without Bash: Feeding Data Through Stdin
Without requiring bash, you can take advantage of stdin being plumbed through, to do the echo outside the container:
docker exec -i cardano-node sh -c 'cat >/tmp/payment.skey' <<'EOF'
{
"type": "PaymentExtendedSigningKeyShelley_ed25519_bip32",
"description": "Payment Signing Key",
"cborHex": "xxxxx"
}
EOF
...or, equivalently...
echo '{
"type": "PaymentExtendedSigningKeyShelley_ed25519_bip32",
"description": "Payment Signing Key",
"cborHex": "xxxxx"
}' | docker exec -i cardano-node sh -c 'cat >/tmp/payment.skey'
Using Bash Extensions Only On The Host
You can put single quotes inside a single-quoted string using $'', as follows:
docker exec cardano-node sh -c $'echo \'{
"type": "PaymentExtendedSigningKeyShelley_ed25519_bip32",
"description": "Payment Signing Key",
"cborHex": "xxxxx"
}\' >/tmp/payment.skey'

You can put single quotes inside single quotes in bash:
sh -c $'echo \'{ "foo": "bar" }\''
However, it's much easier to store it in variable and echo that:
json='{ "foo": "bar" }'
sh -c "echo '$json'"
But it's generally better to feed the commands through the shell's stdin, so:
echo "echo '$json'" | sh
Or use cat to output the stdin inside sh:
echo "$json" | sh -c cat
If the JSON is inside a file, it's even easier:
sh -c cat < file.json
If you need to use a big string, you can use a heredoc instead:
sh -c 'cat' <<EOF
{ "foo": "bar" }
EOF
So your command would be:
docker exec cardano-node sh -c 'cat > /tmp/payment.skey' <<EOF
{
"type": "PaymentExtendedSigningKeyShelley_ed25519_bip32",
"description": "Payment Signing Key",
"cborHex": "xxxxx"
}
EOF
But there's probably no need to use sh, since you can use tee to write to the file:
docker exec cardano-node tee /tmp/payment.skey
But that would output to stdout, if you don't want to see the output:
docker exec cardano-node tee /tmp/payment.skey > /dev/null
So your final command would be:
docker exec cardano-node tee /tmp/payment.skey > /dev/null <<EOF
{
"type": "PaymentExtendedSigningKeyShelley_ed25519_bip32",
"description": "Payment Signing Key",
"cborHex": "xxxxx"
}
EOF

Related

Replace a keyword with the content of the file

I have a templatized json file called template.json as below:
{
"subject": "Some subject line",
"content": $CONTENT,
}
I have another file called sample.json with the json content as below:
{
"status": "ACTIVE",
"id": 217,
"type": "TEXT",
"name": "string",
"subject": "string",
"url": "contenttemplates/217",
"content": {
"text": "hello ${user_name}",
"variables": [{
"key": "${user_name}",
"value": null
}]
},
"content_footer": null,
"audit": {
"creator": "1000",
"timestamp": 1548613800000,
"product": "2",
"channel": "10",
"party": null,
"event": {
"type": null,
"type_id": "0",
"txn_id": "0"
},
"client_key": "pk6781gsfr5"
}
}
I want to replace $CONTENT from template.json with the content under the tag "content" from the content.json file . I have tried with below sed commands:
sed -i 's/$CONTENT/'$(jq -c '.content' sample.json)'/' template.json
I am getting below error:
sed: -e expression #1, char 15: unterminated `s' command
Can someone please help me to get the right sed command (or any other alternative)?
The jq Cookbook has a section on using jq with templates: https://github.com/stedolan/jq/wiki/Cookbook#using-jq-as-a-template-engine
In the present case, the first technique ("Using jq variables as template variables") matches the already-defined template file (except for the dangling comma), so you could for example write:
jq -n --arg CONTENT "$(jq -c .content sample.json)" '
{"subject": "Some subject line", "content": $CONTENT}'
or use the format:
jq -n --arg CONTENT "$(jq -c .content sample.json)" -f template.jq
(I'd only use the .json suffix for files that hold JSON or JSON streams.)
The output from jq contains spaces, you need to quote them to prevent the shell from tokenizing them.
sed -i 's/$CONTENT/'"$(jq -c '.content' sample.json)/" template.json
See further When to wrap quotes around a shell variable?
With GNU sed:
sed '/$CONTENT/{s/.*/jq -c ".content" sample.json/e}'
Replace the entire line with your command and e (GNU only) to execute the command and replace sed's pattern space with the output of the command.

How to parse JSON format output of "kubectl get pods" using jq and create an array

The JSON output returned to me after running this command
kubectl get pods -o json | jq '.items[].spec.containers[].env'
on my kuberntes cluster is this
[
{
"name": "USER_NAME",
"value": "USER_NAME_VALUE_A"
},
{
"name": "USER_ADDRESS",
"value": "USER_ADDRESS_VALUE_A"
}
]
[
{
"name": "USER_NAME",
"value": "USER_NAME_VALUE_B"
},
{
"name": "USER_ADDRESS",
"value": "USER_ADDRESS_VALUE_B"
}
]
I'd like to create a unified array/dictionary (Using Bash script) which looks like the example below and how can I get the value of each key?
[
{
"USER_NAME": "USER_NAME_VALUE_A",
"USER_ADDRESS": "USER_ADDRESS_VALUE_A"
},
{
"USER_NAME": "USER_NAME_VALUE_B",
"USER_ADDRESS": "USER_ADDRESS_VALUE_B"
}
]
use the jsonpath
C02W84XMHTD5:~ iahmad$ kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{.metadata.name}{"\n"}'
coredns-c4cffd6dc-nsd2k
etcd-minikube
kube-addon-manager-minikube
kube-apiserver-minikube
kube-controller-manager-minikube
kube-dns-86f4d74b45-d5njm
kube-proxy-pg89s
kube-scheduler-minikube
kubernetes-dashboard-6f4cfc5d87-b7n7v
storage-provisioner
tiller-deploy-778f674bf5-vt4mj
https://kubernetes.io/docs/reference/kubectl/jsonpath/
it can output key values as well
C02W84XMHTD5:~ iahmad$ kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.startTime}{"\n"}{end}'
coredns-c4cffd6dc-nsd2k 2018-10-16T21:44:19Z
etcd-minikube 2018-10-29T17:30:56Z
kube-addon-manager-minikube 2018-10-29T17:30:56Z
kube-apiserver-minikube 2018-10-29T17:30:56Z
kube-controller-manager-minikube 2018-10-29T17:30:56Z
kube-dns-86f4d74b45-d5njm 2018-10-16T21:44:16Z
kube-proxy-pg89s 2018-10-29T17:32:05Z
kube-scheduler-minikube 2018-10-29T17:30:56Z
kubernetes-dashboard-6f4cfc5d87-b7n7v 2018-10-16T21:44:19Z
storage-provisioner 2018-10-16T21:44:19Z
tiller-deploy-778f674bf5-vt4mj 2018-11-01T13:45:23Z
then you can split those by space and form json or list
This will do it in bash. You'd be surprised how much you can do with bash:
#!/bin/bash
NAMES=`kubectl get pods -o=jsonpath='{range .items[*]}{.spec.containers[*].env[*].name}{"\n"}' | tr -d '\011\012\015'`
VALUES=`kubectl get pods -o=jsonpath='{range .items[*]}{.spec.containers[*].env[*].value}{"\n"}' | tr -d '\011\012\015'`
IFS=' ' read -ra NAMESA <<< "$NAMES"
IFS=' ' read -ra VALUESA <<< "$VALUES"
MAXINDEX=`expr ${#NAMESA[#]} - 1`
printf "[\n"
for i in "${!NAMESA[#]}"; do
printf " {\n"
printf " \"USER_NAME\": \"${NAMESA[$i]}\",\n"
printf " \"USER_ADDRESS\": \"${VALUESA[$i]}\"\n"
if [ "$i" == "${MAXINDEX}" ]; then
printf " }\n"
else
printf " },\n"
fi
done
printf "]\n"
While you are using jq as a filter, why not use it as a transformer, too?
kubectl get pods -o json | \
jq '.items|map(.spec.containers|map(.env|map({key: .name, value})|from_entries)|add)'
I know this is totally a necromancer badge, but still ;)

Parse json into array in bash

{
"db_status": {
"sysa": {
"taskname": "AB",
"state": "Running",
"status": "System ATTENTION",
"updated": "0727",
"version": "5"
},
"sysb": {
"taskname": "null",
"state": "Standby",
"status": "System OK",
"updated": "0727",
"version": "6"
}
}
}
CURL command returns json object. Trying to get both state variables in an array i.e. running and standby. So far i have tried
curl -s http://localhost:9099/api | grep state | sed 's/"//g' | awk -F ": " '/state/ {print $2}' | tr '\n' ' ' | sed s'/..$//'
FYI you don't need a pipeline of 20 different commands when you're using awk. The command line you provided can be written as just one command:
awk -F'"' '$2=="state"{printf "%s%s", (++c>1?", ":""),$4}'
but all you really need is:
awk -F'"' '$2=="state"{print $4}'
and to save the output in a shell array (assuming your json is really always formatted as you show in your question) would be:
$ arr=( $(cat file | awk -F'"' '$2=="state"{print $4}') )
$ echo "${arr[0]}"
Running
$ echo "${arr[1]}"
Standby
Replace cat file with your curl command.
You could use jq
$ curl -s http://localhost:9099/api | jq '.db_status.sysa.state, .db_status.sysb.state'
"Running"
"Standby"
Or if you want all the entries no matter how many
$ curl -s http://localhost:9099/api | jq '.db_status[].state'
"Running"
"Standby"
Alternatively, if you don't have jq (or just looking for ways of feeling pain) you can parse most json in bash using ticktick -- which is written in 250 lines of bash

Send data with curl in Bash script: positional argument not working

Total noob in BASH. Trying to learn.
I have the following bash script to make an API request:
#!/bin/bash
if [ $1 = "new_event" ]; then
a='https://www.googleapis.com/calendar/v3/calendars/'
b=$2
c='/events?access_token='
d=$3
path=$a$b$c$d
echo $4
OUTPUT="$(curl -s -H "Content-Type: application/json" $path -d $4 )"
echo "${OUTPUT}"
fi
Positional arguments are 'new_event', calendarId, access token and json string.
If I run the script I get:
first echo is the json string I pass as parameter in the call:
' {"guestsCanSeeOtherGuests": false, "location": "", "description": "TEST", "reminders": {"useDefault": false}, "start": {"dateTime": "2017-07-06T14:00:00", "timeZone": "America/Sao_Paulo"}, "end": {"dateTime": "2017-07-06T15:00:00", "timeZone": "America/Sao_Paulo"}, "guestsCanInviteOthers": false, "summary": "TEST", "status": "tentative", "attendees": []} '
second echo gives me parsing error.
BUT, if I copy the echoed json string and replace $4 for it, everything works.
OUTPUT="$(curl -s -H "Content-Type: application/json" $path -d ' {"guestsCanSeeOtherGuests": false, "location": "", "description": "TEST", "reminders": {"useDefault": false}, "start": {"dateTime": "2017-07-06T14:00:00", "timeZone": "America/Sao_Paulo"}, "end": {"dateTime": "2017-07-06T15:00:00", "timeZone": "America/Sao_Paulo"}, "guestsCanInviteOthers": false, "summary": "TEST", "status": "tentative", "attendees": []} ' )"
Any hint why is it not working with the positional argument while it works if I paste it's content?
Thanks!
When you pass the JSON string as a parameter, it assigns the content contained inside the single quote as a whole string but when you pass it to curl it was subjected to word splitting.
To show what it looks like here's a sample script to demonstrate it.
This script will receive the string and pass it to the second script.
#!/bin/bash
./params $1
This second script simulates what curl sees. It will print the number of parameters it receives.
#!/bin/bash
echo $#
Guess what the output is:
27
To fix your issue and make it simpler, drop the outermost quote and quote everything inside the $().
OUTPUT=$(curl -s -H "Content-Type: application/json" "$path" -d "$4" )

Find the value of key from JSON

I'd like to extract the "id" key from this single line of JSON.
I believe this can be accomplished with grep, but I am not sure on the correct way.
If there is a better way that does not have dependencies, I would be interested.
Here is my example output:
{
"data": {
"name": "test",
"id": "4dCYd4W9i6gHQHvd",
"domains": ["www.test.domain.com", "test.domain.com"],
"serverid": "bbBdbbHF8PajW221",
"ssl": null,
"runtime": "php5.6",
"sysuserid": "4gm4K3lUerbSPfxz",
"datecreated": 1474597357
},
"actionid": "WXVAAHQDCSILMYTV"
}
If you have a grep that can do Perl compatible regular expressions (PCRE):
$ grep -Po '"id": *\K"[^"]*"' infile.json
"4dCYd4W9i6gHQHvd"
-P enables PCRE
-o retains nothing but the match
"id": * matches "id" and an arbitrary amount of spaces
\K throws away everything to its left ("variable size positive look-behind")
"[^"]*" matches two quotes and all the non-quotes between them
If your grep can't do that, you an use
$ grep -o '"id": *"[^"]*"' infile.json | grep -o '"[^"]*"$'
"4dCYd4W9i6gHQHvd"
This uses grep twice. The result of the first command is "id": "4dCYd4W9i6gHQHvd"; the second command removes everything but a pair of quotes and the non-quotes between them, anchored at the end of the string ($).
But, as pointed out, you shouldn't use grep for this, but a tool that can parse JSON – for example jq:
$ jq '.data.id' infile.json
"4dCYd4W9i6gHQHvd"
This is just a simple filter for the id key in the data object. To get rid of the double quotes, you can use the -r ("raw output") option:
$ jq -r '.data.id' infile.json
4dCYd4W9i6gHQHvd
jq can also neatly pretty print your JSON:
$ jq . infile.json
{
"data": {
"name": "test",
"id": "4dCYd4W9i6gHQHvd",
"domains": [
"www.test.domain.com",
"test.domain.com"
],
"serverid": "bbBdbbHF8PajW221",
"ssl": null,
"runtime": "php5.6",
"sysuserid": "4gm4K3lUerbSPfxz",
"datecreated": 1474597357
},
"actionid": "WXVAAHQDCSILMYTV"
}
Just pipe your data to jq and select by keys
"data": {
"name": "test",
"id": "4dCYd4W9i6gHQHvd",
"domains": [
"www.test.domain.com",
"test.domain.com"
],
"serverid": "bbBdbbHF8PajW221",
"ssl": null,
"runtime": "php5.6",
"sysuserid": "4gm4K3lUerbSPfxz",
"datecreated": 1474597357
},
"actionid": "WXVAAHQDCSILMYTV"
} | jq '.data.id'
# 4dCYd4W9i6gHQHvd
Tutorial Here
I found myself that the best way is to use python, as it handles JSON natively and is preinstalled on most systems these days, unlike jq:
$ python -c 'import sys, json; print(json.load(sys.stdin)["data"]["id"])' < infile.json
4dCYd4W9i6gHQHvd
No python ,jq, awk, sed just GNU grep:
#!/bin/bash
json='{"data": {"name": "test", "id": "4dCYd4W9i6gHQHvd", "domains": ["www.test.domain.com", "test.domain.com"], "serverid": "bbBdbbHF8PajW221", "ssl": null, "runtime": "php5.6", "sysuserid": "4gm4K3lUerbSPfxz", "datecreated": 1474597357}, "actionid": "WXVAAHQDCSILMYTV"}'
echo $json | grep -o '"id": "[^"]*' | grep -o '[^"]*$'
Tested & working here: https://ideone.com/EG7fv7
source: https://brianchildress.co/parse-json-using-grep
$ grep -oP '"id": *"\K[^"]*' infile.json
4dCYd4W9i6gHQHvd
Hopefully it will work for all. As this will work for me to print without quotes.