I am looking easy ways to clean my json in linux.
Sample Input:
[{"name":"Student1","value":"John"},{"name":"Student2","value":"Jack"},{"name":"Student3","value":"Nick"},{"name":"Student4","value":"Karen"},{"name":"Student5","value":"Jonas"}]
Expected Output:
I want to save below expected output in txt/yaml file in linux.
Student1:John
Student2:Jack
Student3:Nick
Student4:Karen
Student5:Jonas
My Attempt:
jq '. | map([.name, .value] | join(": ")) | join("\n")' file.json >> all_param.yml;
You are only missing the additional flag --raw-output for jq.
The command should look like
jq --raw-output '.[] | [.name, .value] | join(":")' file.json
The --raw-output argument makes jq print the output directly instead of printing it as a JSON string, i.e., escaping it in a string.
In your expected output you have no space after the colon, but you have it in your jq command.
Related
I have written a script to retrieve certain value from file.json. It works if I provide the value to jq select, but the variable doesn't seem to work (or I don't know how to use it).
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
#this does not work *** no value is printed
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="$EMAILID") | .id')
echo "$projectID"
Consider also passing in the shell variable (EMAILID) as a jq variable (here also EMAILID, for the sake of illustration):
projectID=$(jq -r --arg EMAILID "$EMAILID" '
.resource[]
| select(.username==$EMAILID)
| .id' file.json)
Postscript
For the record, another possibility would be to use jq's env function for accessing environment variables. For example, consider this sequence of bash commands:
EMAILID=foo#bar.com # not exported
EMAILID="$EMAILID" jq -n 'env.EMAILID'
The output is a JSON string:
"foo#bar.com"
I resolved this issue by escaping the inner double quotes
projectID=$(cat file.json | jq -r ".resource[] | select(.username==\"$EMAILID\") | .id")
Little unrelated but I will still put it here,
For other practical purposes shell variables can be used as -
value=10
jq '."key" = "'"$value"'"' file.json
Posting it here as it might help others. In string it might be necessary to pass the quotes to jq. To do the following with jq:
.items[] | select(.name=="string")
in bash you could do
EMAILID=$1
projectID=$(cat file.json | jq -r '.resource[] | select(.username=='\"$EMAILID\"') | .id')
essentially escaping the quotes and passing it on to jq
It's a quote issue, you need :
projectID=$(
cat file.json | jq -r ".resource[] | select(.username=='$EMAILID') | .id"
)
If you put single quotes to delimit the main string, the shell takes $EMAILID literally.
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words
Jq now have better way to access environment variables, you can use env.EMAILID:
projectID=$(cat file.json | jq -r ".resource[] | select(.username==env.EMAILID) | .id")
Another way to accomplish this is with the jq "--arg" flag.
Using the original example:
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] |
select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
# Use --arg to pass the variable to jq. This should work:
projectID=$(cat file.json | jq --arg EMAILID $EMAILID -r '.resource[]
| select(.username=="$EMAILID") | .id')
echo "$projectID"
See here, which is where I found this solution:
https://github.com/stedolan/jq/issues/626
I know is a bit later to reply, sorry. But that works for me.
export K8S_public_load_balancer_url="$(kubectl get services -n ${TENANT}-production -o wide | grep "ingress-nginx-internal$" | awk '{print $4}')"
And now I am able to fetch and pass the content of the variable to jq
export TF_VAR_public_load_balancer_url="$(aws elbv2 describe-load-balancers --region eu-west-1 | jq -r '.LoadBalancers[] | select (.DNSName == "'$K8S_public_load_balancer_url'") | .LoadBalancerArn')"
In my case I needed to use double quote and quote to access the variable value.
Cheers.
I also faced same issue of variable substitution with jq. I found that --arg is the option which must be used with square bracket [] otherwise it won't work.. I am giving you sample example below:
RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id $SECRET_ID | jq '.SecretString|fromjson' | jq --arg kt $SECRET_KEY -r '.[$kt]' | tr -d '"')
In case where we want to append some string to the variable value and we are using the escaped double quotes, for example appending .crt to a variable CERT_TYPE; the following should work:
$ CERT_TYPE=client.reader
$ cat certs.json | jq -r ".\"${CERT_TYPE}\".crt" #### This will *not* work #####
$ cat certs.json | jq -r ".\"${CERT_TYPE}.crt\""
I have written a script to retrieve certain value from file.json. It works if I provide the value to jq select, but the variable doesn't seem to work (or I don't know how to use it).
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
#this does not work *** no value is printed
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="$EMAILID") | .id')
echo "$projectID"
Consider also passing in the shell variable (EMAILID) as a jq variable (here also EMAILID, for the sake of illustration):
projectID=$(jq -r --arg EMAILID "$EMAILID" '
.resource[]
| select(.username==$EMAILID)
| .id' file.json)
Postscript
For the record, another possibility would be to use jq's env function for accessing environment variables. For example, consider this sequence of bash commands:
EMAILID=foo#bar.com # not exported
EMAILID="$EMAILID" jq -n 'env.EMAILID'
The output is a JSON string:
"foo#bar.com"
I resolved this issue by escaping the inner double quotes
projectID=$(cat file.json | jq -r ".resource[] | select(.username==\"$EMAILID\") | .id")
Little unrelated but I will still put it here,
For other practical purposes shell variables can be used as -
value=10
jq '."key" = "'"$value"'"' file.json
Posting it here as it might help others. In string it might be necessary to pass the quotes to jq. To do the following with jq:
.items[] | select(.name=="string")
in bash you could do
EMAILID=$1
projectID=$(cat file.json | jq -r '.resource[] | select(.username=='\"$EMAILID\"') | .id')
essentially escaping the quotes and passing it on to jq
It's a quote issue, you need :
projectID=$(
cat file.json | jq -r ".resource[] | select(.username=='$EMAILID') | .id"
)
If you put single quotes to delimit the main string, the shell takes $EMAILID literally.
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words
Jq now have better way to access environment variables, you can use env.EMAILID:
projectID=$(cat file.json | jq -r ".resource[] | select(.username==env.EMAILID) | .id")
Another way to accomplish this is with the jq "--arg" flag.
Using the original example:
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] |
select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
# Use --arg to pass the variable to jq. This should work:
projectID=$(cat file.json | jq --arg EMAILID $EMAILID -r '.resource[]
| select(.username=="$EMAILID") | .id')
echo "$projectID"
See here, which is where I found this solution:
https://github.com/stedolan/jq/issues/626
I know is a bit later to reply, sorry. But that works for me.
export K8S_public_load_balancer_url="$(kubectl get services -n ${TENANT}-production -o wide | grep "ingress-nginx-internal$" | awk '{print $4}')"
And now I am able to fetch and pass the content of the variable to jq
export TF_VAR_public_load_balancer_url="$(aws elbv2 describe-load-balancers --region eu-west-1 | jq -r '.LoadBalancers[] | select (.DNSName == "'$K8S_public_load_balancer_url'") | .LoadBalancerArn')"
In my case I needed to use double quote and quote to access the variable value.
Cheers.
I also faced same issue of variable substitution with jq. I found that --arg is the option which must be used with square bracket [] otherwise it won't work.. I am giving you sample example below:
RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id $SECRET_ID | jq '.SecretString|fromjson' | jq --arg kt $SECRET_KEY -r '.[$kt]' | tr -d '"')
In case where we want to append some string to the variable value and we are using the escaped double quotes, for example appending .crt to a variable CERT_TYPE; the following should work:
$ CERT_TYPE=client.reader
$ cat certs.json | jq -r ".\"${CERT_TYPE}\".crt" #### This will *not* work #####
$ cat certs.json | jq -r ".\"${CERT_TYPE}.crt\""
I have written a script to retrieve certain value from file.json. It works if I provide the value to jq select, but the variable doesn't seem to work (or I don't know how to use it).
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
#this does not work *** no value is printed
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="$EMAILID") | .id')
echo "$projectID"
Consider also passing in the shell variable (EMAILID) as a jq variable (here also EMAILID, for the sake of illustration):
projectID=$(jq -r --arg EMAILID "$EMAILID" '
.resource[]
| select(.username==$EMAILID)
| .id' file.json)
Postscript
For the record, another possibility would be to use jq's env function for accessing environment variables. For example, consider this sequence of bash commands:
EMAILID=foo#bar.com # not exported
EMAILID="$EMAILID" jq -n 'env.EMAILID'
The output is a JSON string:
"foo#bar.com"
I resolved this issue by escaping the inner double quotes
projectID=$(cat file.json | jq -r ".resource[] | select(.username==\"$EMAILID\") | .id")
Little unrelated but I will still put it here,
For other practical purposes shell variables can be used as -
value=10
jq '."key" = "'"$value"'"' file.json
Posting it here as it might help others. In string it might be necessary to pass the quotes to jq. To do the following with jq:
.items[] | select(.name=="string")
in bash you could do
EMAILID=$1
projectID=$(cat file.json | jq -r '.resource[] | select(.username=='\"$EMAILID\"') | .id')
essentially escaping the quotes and passing it on to jq
It's a quote issue, you need :
projectID=$(
cat file.json | jq -r ".resource[] | select(.username=='$EMAILID') | .id"
)
If you put single quotes to delimit the main string, the shell takes $EMAILID literally.
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words
Jq now have better way to access environment variables, you can use env.EMAILID:
projectID=$(cat file.json | jq -r ".resource[] | select(.username==env.EMAILID) | .id")
Another way to accomplish this is with the jq "--arg" flag.
Using the original example:
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] |
select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
# Use --arg to pass the variable to jq. This should work:
projectID=$(cat file.json | jq --arg EMAILID $EMAILID -r '.resource[]
| select(.username=="$EMAILID") | .id')
echo "$projectID"
See here, which is where I found this solution:
https://github.com/stedolan/jq/issues/626
I know is a bit later to reply, sorry. But that works for me.
export K8S_public_load_balancer_url="$(kubectl get services -n ${TENANT}-production -o wide | grep "ingress-nginx-internal$" | awk '{print $4}')"
And now I am able to fetch and pass the content of the variable to jq
export TF_VAR_public_load_balancer_url="$(aws elbv2 describe-load-balancers --region eu-west-1 | jq -r '.LoadBalancers[] | select (.DNSName == "'$K8S_public_load_balancer_url'") | .LoadBalancerArn')"
In my case I needed to use double quote and quote to access the variable value.
Cheers.
I also faced same issue of variable substitution with jq. I found that --arg is the option which must be used with square bracket [] otherwise it won't work.. I am giving you sample example below:
RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id $SECRET_ID | jq '.SecretString|fromjson' | jq --arg kt $SECRET_KEY -r '.[$kt]' | tr -d '"')
In case where we want to append some string to the variable value and we are using the escaped double quotes, for example appending .crt to a variable CERT_TYPE; the following should work:
$ CERT_TYPE=client.reader
$ cat certs.json | jq -r ".\"${CERT_TYPE}\".crt" #### This will *not* work #####
$ cat certs.json | jq -r ".\"${CERT_TYPE}.crt\""
I'm using jq to parse a JSON file as shown here. However, the results for string values contain the "double-quotes" as expected, as shown below:
$ cat json.txt | jq '.name'
"Google"
How can I pipe this into another command to remove the ""? so I get
$ cat json.txt | jq '.name' | some_other_command
Google
What some_other_command can I use?
Use the -r (or --raw-output) option to emit raw strings as output:
jq -r '.name' <json.txt
So for a file containing just {"name": "Google"} then yes
sample='{"name":"Google"}'
echo $sample| jq '.name'
"Google"
using --raw-input helps
echo $sample| jq --raw-output '.name'
Google
But I stumbled upon this question because I was using --raw-output on a json array like this
sample='[{"name":"Yahoo"},{"name":"Google"}]'
echo $sample | jq --raw-output 'map(.name)'
[
"Yahoo",
"Google"
]
And I didn't understand why the quotes remained. I came across this post, and now I know adding | .[] does the trick!
echo $sample | jq --raw-output 'map(.name)| .[]'
Yahoo
Google
I have written a script to retrieve certain value from file.json. It works if I provide the value to jq select, but the variable doesn't seem to work (or I don't know how to use it).
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
#this does not work *** no value is printed
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="$EMAILID") | .id')
echo "$projectID"
Consider also passing in the shell variable (EMAILID) as a jq variable (here also EMAILID, for the sake of illustration):
projectID=$(jq -r --arg EMAILID "$EMAILID" '
.resource[]
| select(.username==$EMAILID)
| .id' file.json)
Postscript
For the record, another possibility would be to use jq's env function for accessing environment variables. For example, consider this sequence of bash commands:
EMAILID=foo#bar.com # not exported
EMAILID="$EMAILID" jq -n 'env.EMAILID'
The output is a JSON string:
"foo#bar.com"
I resolved this issue by escaping the inner double quotes
projectID=$(cat file.json | jq -r ".resource[] | select(.username==\"$EMAILID\") | .id")
Little unrelated but I will still put it here,
For other practical purposes shell variables can be used as -
value=10
jq '."key" = "'"$value"'"' file.json
Posting it here as it might help others. In string it might be necessary to pass the quotes to jq. To do the following with jq:
.items[] | select(.name=="string")
in bash you could do
EMAILID=$1
projectID=$(cat file.json | jq -r '.resource[] | select(.username=='\"$EMAILID\"') | .id')
essentially escaping the quotes and passing it on to jq
It's a quote issue, you need :
projectID=$(
cat file.json | jq -r ".resource[] | select(.username=='$EMAILID') | .id"
)
If you put single quotes to delimit the main string, the shell takes $EMAILID literally.
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words
Jq now have better way to access environment variables, you can use env.EMAILID:
projectID=$(cat file.json | jq -r ".resource[] | select(.username==env.EMAILID) | .id")
Another way to accomplish this is with the jq "--arg" flag.
Using the original example:
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] |
select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
# Use --arg to pass the variable to jq. This should work:
projectID=$(cat file.json | jq --arg EMAILID $EMAILID -r '.resource[]
| select(.username=="$EMAILID") | .id')
echo "$projectID"
See here, which is where I found this solution:
https://github.com/stedolan/jq/issues/626
I know is a bit later to reply, sorry. But that works for me.
export K8S_public_load_balancer_url="$(kubectl get services -n ${TENANT}-production -o wide | grep "ingress-nginx-internal$" | awk '{print $4}')"
And now I am able to fetch and pass the content of the variable to jq
export TF_VAR_public_load_balancer_url="$(aws elbv2 describe-load-balancers --region eu-west-1 | jq -r '.LoadBalancers[] | select (.DNSName == "'$K8S_public_load_balancer_url'") | .LoadBalancerArn')"
In my case I needed to use double quote and quote to access the variable value.
Cheers.
I also faced same issue of variable substitution with jq. I found that --arg is the option which must be used with square bracket [] otherwise it won't work.. I am giving you sample example below:
RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id $SECRET_ID | jq '.SecretString|fromjson' | jq --arg kt $SECRET_KEY -r '.[$kt]' | tr -d '"')
In case where we want to append some string to the variable value and we are using the escaped double quotes, for example appending .crt to a variable CERT_TYPE; the following should work:
$ CERT_TYPE=client.reader
$ cat certs.json | jq -r ".\"${CERT_TYPE}\".crt" #### This will *not* work #####
$ cat certs.json | jq -r ".\"${CERT_TYPE}.crt\""