I need to prepare JSON which contains apostrophe to be sent via CURL.
Example of JSON:
{"myField":"Apos'test"}
Example of JSON I need as an output:
{"myField":"Apos'\''test"}
What I have tried:
sed -e "s/'/'\\\''/g" <<< {"myField":"Apos'test"}
which outputs:
{myField:Apos'\''test}
And I do not understand why it removes double quotes.
P.S. it is not obligatory to use sed, any other standard linux tool would work.
try this:
#/bin/bash
replacement=$((cat << EOT
{"myField":"Apos'test"}
EOT
) | sed "s|'|'\\\''|")
echo $replacement
output:
{"myField":"Apos'\''test"}
It doesn't
If it was because you used <<< , here, of which "" pair was parsed, expanded and dropped by the shell you're in
$ cat d
{"myField":"Apos'test"}
$ sed -E "s/'/'\\\''/g" d
{"myField":"Apos'\''test"}
Related
I would like to replace the fortawesome string (if it is not followed by the /fontawesome-common-type string) by the stephane string.
sed -e 's,"#fortawesome(/^fontawesome-common-types+),"#stephaneeybert\1,g'
sed: -e expression #1, char 65: invalid reference \1 on `s' command's RHS
An example input:
"#fortawesome/fontawesome-common-types": "^0.2.32"
"name": "#fortawesome/pro-duotone-svg-icons",
And its expected output:
"#fortawesome/fontawesome-common-types": "^0.2.32"
"name": "#stephane/pro-duotone-svg-icons",
UPDATE: I went with the simple alternative of using an intermediate variable:
EXCLUDE=fontawesome-common-types
BUFFER=EkSkLUdE
cat package/package.json \
| sed -e "s,\"#$REPO_SOURCE/$EXCLUDE,\"#$BUFFER,g" \
| sed -e "s,\"#$REPO_SOURCE,\"#$REPO_DEST,g" \
| sed -e "s,\"#$BUFFER,\"#$REPO_SOURCE/$EXCLUDE,g" \
> package/package.out.json;
sed doesn't support negative lookahead functionality. Other than the obvious perl fallback that supports lookaheads, uou may use this awk as a work-around:
awk -F 'fortawesome' -v OFS='stephane' 'NF > 1 {
s = ""
for (i=1; i<NF; ++i)
s = s $i ($(i+1) ~ /^\/fontawesome-common-type/ ? FS : OFS)
$0 = s $i
} 1' file
This awk uses fortawesome as input field separator and stephane as OFS
NF > 1 will be true when we have fortawesome in a line
we loop through fields split by fortawesome and keep track of next field
if next field starts with /fontawesome-common-type then we keep same FS otherwise use OFS
Use temporary values:
exclude='fortawesome/fontawesome-common-type';
match='fortawesome';
repl='stephane';
tmpvar='EkSkLUdE';
sed "s#$exclude#$tmpvar#g;s#$match#$repl#g;s#$tmpvar#$exclude#g" file > newfile
All cases of exclude are replaced with tmpvars, then real expected matches are replaced with repls, and then tmpvars are changed back to excludes.
I'm trying to extract a specific string value from a text file and then remove the backslash from him. the name of the value is "display_url"
My script:
url=$(cat /var/scripts/string.txt | grep -oP '(?<=display_url":")[^"]+')
for link in $url; do
echo 'https://'$link
done
output:
https://pastebin.com\/WRv5ir4Y
https://reddit.com\/r\/IBO\/comments\u2026
The desired output:
https://pastebin.com/WRv5ir4Y
https://reddit.com/r/IBO/comments/u2026
text file:
{"created_at":"Thu Dec 13 08:43:38 +0000 2018","id":1073136349845303297,"id_str":"1073136349845303297","text":"https:\/\/t.co\/aPu5ln7yjO\nhttps:\/\/t.co\/pBvevjSCc9\n\n#osectraining","source":"\u003ca href=\"http:\/\/twitter.com\" rel=\"nofollow\"\u003eTwitter Web Client\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":961508561217052675,"id_str":"961508561217052675","name":"Online Security","screen_name":"osectraining","location":"Israel","url":"https:\/\/www.onlinesecurity.co.il","description":"OnlineSecurity provides online cyber-security training courses and certification, from beginner to advanced with the most advanced virtual labs in the field.","translator_type":"none","protected":false,"verified":false,"followers_count":2,"friends_count":51,"listed_count":0,"favourites_count":0,"statuses_count":1,"created_at":"Thu Feb 08 07:54:39 +0000 2018","utc_offset":null,"time_zone":null,"geo_enabled":false,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"000000","profile_background_image_url":"http:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_image_url_https":"https:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_tile":false,"profile_link_color":"1B95E0","profile_sidebar_border_color":"000000","profile_sidebar_fill_color":"000000","profile_text_color":"000000","profile_use_background_image":false,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/961510231346958336\/d_KhBeTD_normal.jpg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/961510231346958336\/d_KhBeTD_normal.jpg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/961508561217052675\/1518076913","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"quote_count":0,"reply_count":0,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[{"text":"osectraining","indices":[49,62]}],"urls":[{"url":"https:\/\/t.co\/aPu5ln7yjO","expanded_url":"https:\/\/pastebin.com\/WRv5ir4Y","display_url":"pastebin.com\/WRv5ir4Y","indices":[0,23]},{"url":"https:\/\/t.co\/pBvevjSCc9","expanded_url":"https:\/\/www.reddit.com\/r\/IBO\/comments\/9ragj7\/ioc_in_10_hours\/","display_url":"reddit.com\/r\/IBO\/comments\u2026","indices":[24,47]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"und","timestamp_ms":"1544690618369"}
any ideas?
Change your script into:
url=$(grep -oP '(?<=display_url":")[^"]+' /var/scripts/string.txt )
sed 's/\\//g;s#^#https://#' <<< "$url"
should help.
removed useless cat
make use of sed to do the substitution
The reason you are needing both that grep and a separate sed to parse it is because the grep is able to use Perl regexes (power...) but sed can't. You probably have Perl available - use it if you do.
perl -pe '
s/\\//g;
s{.*?display_url":"}{https://};
s{",".*display_url":"}{\nhttps://} while /display_url/;
s/",".*/\n/;
' /var/scripts/string.txt
https://pastebin.com/WRv5ir4Y
https://reddit.com/r/IBO/commentsu2026
And there's virtually always awk.
awk '{
gsub("\\\\","",$0);
split($0, chnk, "display_url.:.");
for (x=2; x<=length(chnk); x++) {
gsub("\".*","", chnk[x]);
printf "https://%s\n", chnk[x];
}
}' /var/scripts/string.txt
https://pastebin.com/WRv5ir4Y
https://reddit.com/r/IBO/commentsu2026
But if you can't use either of those, then one sed to strip the backslashes and some basic shell string processing in a loop, because it's fun. :D
$: txt=$(sed 's/\\//g' i)
$: while [[ "$txt" =~ display_url ]]
do txt=${txt#*display_url?:?}
echo https://${txt%%?,*}
done
https://pastebin.com/WRv5ir4Y
https://reddit.com/r/IBO/commentsu2026
Trickiest part is getting around the double-quotes in the shell parsing, but I'm sure someone can suggest a better way.
I would use the JSON command line parser jq:
jq -r '"https://" + .entities.urls[].display_url' /var/scripts/string.txt
-r stands for "return raw inputs" (strings without quotes)
"https://" + concat ...
.entities.urls[] ... for each item of the array .entities.urls ...
.display_url ... the value of display_url member"
Result:
https://pastebin.com/WRv5ir4Y
https://reddit.com/r/IBO/comments
I was wondering how to parse the CURL JSON output from the server into variables.
Currently, I have -
curl -X POST -H "Content: agent-type: application/x-www-form-urlencoded" https://www.toontownrewritten.com/api/login?format=json -d username="$USERNAME" -d password="$PASSWORD" | python -m json.tool
But it only outputs the JSON from the server and then have it parsed, like so:
{
"eta": "0",
"position": "0",
"queueToken": "6bee9e85-343f-41c7-a4d3-156f901da615",
"success": "delayed"
}
But how do I put - for example the success value above returned from the server into a variable $SUCCESS and have the value as delayed & have queueToken as a variable $queueToken and 6bee9e85-343f-41c7-a4d3-156f901da615 as a value?
Then when I use-
echo "$SUCCESS"
it shows this as the output -
delayed
And when I use
echo "$queueToken"
and the output as
6bee9e85-343f-41c7-a4d3-156f901da615
Thanks!
Find and install jq (https://stedolan.github.io/jq/). jq is a JSON parser. JSON is not reliably parsed by line-oriented tools like sed because, like XML, JSON is not a line-oriented data format.
In terms of your question:
source <(
curl -X POST -H "$content_type" "$url" -d username="$USERNAME" -d password="$PASSWORD" |
jq -r '. as $h | keys | map(. + "=\"" + $h[.] + "\"") | .[]'
)
The jq syntax is a bit weird, I'm still working on it. It's basically a series of filters, each pipe taking the previous input and transforming it. In this case, the end result is some lines that look like variable="value"
This answer uses bash's "process substitution" to take the results of the jq command, treat it like a file, and source it into the current shell. The variables will then be available to use.
Here's an example of Extract a JSON value from a BASH script
#!/bin/bash
function jsonval {
temp=`echo $json | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w $prop`
echo ${temp##*|}
}
json=`curl -s -X GET http://twitter.com/users/show/$1.json`
prop='profile_image_url'
picurl=`jsonval`
`curl -s -X GET $picurl -o $1.png`
A bash script which demonstrates parsing a JSON string to extract a
property value. The script contains a jsonval function which operates
on two variables, json and prop. When the script is passed the name of
a twitter user it attempts to download the user's profile picture.
You could use perl module on command line:
1st, ensure they is installed, under debian based, you could
sudo apt-get install libjson-xs-perl
But for other OS, you could install perl modules via CPAN (the Comprehensive Perl Archive Network):
cpan App::cpanminus
cpan JSON::XS
Note: You may have to run this with superuser privileges.
then:
curlopts=(-X POST -H
"Content: apent-type: application/x-www-form-urlencoded"
-d username="$USERNAME" -d password="$PASSWORD")
curlurl=https://www.toontownrewritten.com/api/login?format=json
. <(
perl -MJSON::XS -e '
$/=undef;my $a=JSON::XS::decode_json <> ;
printf "declare -A Json=\047(%s)\047\n", join " ",map {
"[".$_."]=\"".$a->{$_}."\""
} qw|queueToken success eta position|;
' < <(
curl "${curlopts[#]}" $curlurl
)
)
The line qw|...| let you precise which variables you want to be driven... This could be replaced by keys $a, but could have to be debugged as some characters is forbiden is associative arrays values names.
echo ${Json[queueToken]}
6bee9e85-343f-41c7-a4d3-156f901da615
echo ${Json[eta]}
0
I have a question about json and parsing with sed:
Here is what I get in json:
response='{"found":"true","downloadLink":"http:\/\/www.addic7ed.com\/updated\/1\/86593\/2"}'
If I use this:
downloadLink=`echo $response | sed -e 's/^.*"downloadLink"[ ]*:[ ]*"//' -e 's/".*//'`
then downloadLink will contain http:\/\/www.addic7ed.com\/updated\/1\/86593\/2.
I tried to put a variable instead of downloadLink:
downloadLink=`echo $response | sed -e 's/^.*"$value"[ ]*:[ ]*"//' -e 's/".*//'`
But it doesn't seem to work properly. Do you know how to do it?
The single quotes are not expanded in bash. Use double quotes and escape those already used: - like this:
echo $response | sed -e "s/^.*\"$value\"[ ]*:[ ]*\"//" -e 's/".*//'
Rather than using two sed commands, you can capture the value you are interested like this:
echo "$response" | sed -e "s/^.*\"$value\"\s*:\s*\"\([^\"]*\)\".*$/\1/"
The contents of the \( \) are captured into the variable \1. I have chosen to capture [^\"]* (any number of characters that are not a double quote), which works for your example.
I am also using the \s "whitespace" character class rather than [ ], as I believe it is clearer.
Testing it out:
$ echo "$response"
{"found":"true","downloadLink":"http:\/\/www.addic7ed.com\/updated\/1\/86593\/2"}
$ value=downloadLink
$ echo "$response" | sed -e "s/^.*\"$value\"\s*:\s*\"\([^\"]*\)\".*$/\1/"
http:\/\/www.addic7ed.com\/updated\/1\/86593\/2
$ value=found
$ echo "$response" | sed -e "s/^.*\"$value\"\s*:\s*\"\([^\"]*\)\".*$/\1/"
true
By the way, if you're using bash, you can avoid echo $var | sed by using <<<:
sed -e "s/^.*\"$value\"\s*:\s*\"\([^\"]*\)\".*$/\1/" <<<"$response"
Variables are not expanded inside single quotes. You could use double quotes insted like
sed "s/$variable/newlaue/g" ...
but then you should be extra careful with the contents of $variable, since sed will interpret any special characters in the variable (like the slash /) in this specific example.
I have the json as below, i need to get only the mail from the above json in bash script
value={"count":5,"users":[{"username":"asa","name":"asa
Tran","mail":"asa#xyz.com"},{"username":"qq","name":"qq
Morris","mail":"qq#xyz.com"},{"username":"qwe","name":"qwe
Org","mail":"qwe#xyz.com"}]}
Output can be as
mail=asa#xyz.com,qq#xyz.com,qwe#xyz.com
All the above need to be done in the bash script (.sh)
I have already tried with the array iteration as but of no use
for key in "${!value[#]}"
do
#echo "key = $key"
echo "value = ${value[$key]}"
done
Even i have tried with the array conversion as
alias json-decode="php -r
'print_r(json_decode(file_get_contents(\"php://stdin\"),1));'"
value=$(curl --user $credentials -k $endPoint | json-decode)
Still i was not able to get the specific output.
jq is the tool to iterate through a json. In your case:
while read user; do
jq -r '.mail' <<< $user
done <<< $(jq -c '.users[]' users.json)
would give:
asa#xyz.com
qq#xyz.com
qwe#xyz.com
NOTE: I removed "value=" because that is not valid json. Users.json contains:
{"count":5,"users":[{"username":"asa","name":"asa Tran","mail":"asa#xyz.com"},{"username":"qq","name":"qq Morris","mail":"qq#xyz.com"},{"username":"qwe","name":"qwe Org","mail":"qwe#xyz.com"}]}
If this is valid json and the email field is the only one containing a # character, you can do something like this:
echo $value | tr '"' '\n' | grep #
It replaces double-quotes by new line character and only keeps lines containing #. It is really not json parsing, but it works.
You can store the result in a bash array
emails=($(echo $value | tr '"' '\n' | grep #))
and iterate on them
for email in ${emails[#]}
do
echo $email
done
You should use json_pp tool (in debian, it is part of the libjson-pp-perl package)
One would use it like this :
cat file.json | json_pp
And get a pretty print for your json.
So in your case, you could do :
#!/bin/bash
MAILS=""
LINES=`cat test.json | json_pp | grep '"mail"' | sed 's/.* : "\(.*\)".*/\1/'`
for LINE in $LINES ; do
MAILS="$LINE,$MAILS"
done
echo $MAILS | sed 's/.$//'
Output :
qwe#xyz.com,qq#xyz.com,asa#xyz.com
Using standard unix toolbox : sed command
cat so.json | sed "s/},/\n/g" | sed 's/.*"mail":"\([^"]*\)".*/\1/'
With R you could do this as follows:
$ value={"count":5,"users":[{"username":"asa","name":"asa Tran","mail":"asa#xyz.com"},{"username":"qq","name":"qq Morris","mail":"qq#xyz.com"},{"username":"qwe","name":"qwe Org","mail":"qwe#xyz.com"}]}
$ echo $value | R path users | R map path mail
["asa#xyz.com", "qq#xyz.com", "qwe#gyz.com"]