I can't really figure this out - the SQL query outputed is not valid
key="test"
payload=$(gzip -ckqd ./temp.json.gz | jq -c . | sed 's/"/\\"/g')
printf 'INSERT INTO my_table VALUES ("%s", "%s")' "$key" "$payload" | sqlite3 ./temp.db
Obiously the $payload variable is a json string (can have single and double quotes etc)
In SQL, strings are delimited not with double quotes but with single quotes.
Inside a string, the only special character is the single quote itself, and it must be escaped not with a backslash, but with another quote:
... sed "s/'/''/g"
Related
I have a simple file with this test line:
mmm#gmail.com 31460 147557432
My goal is to send as json data.
In my while loop I can echo the variables in the second line of my code example.
However, when I attempt to assign them to jsonstring and echo, the values are not populated.
What do I need to do to pass these values to my json string?
while read emailvar idvar expirevar; do
echo "$emailvar:$expirevar:$idvar"
jsonstring=$idvar $emailvar $expirevar
echo "$jsonstring"
#jsonstring='{"user_id":"$idvar","email":"$emailvar","custom_attributes":{"Program_Expires_at":"$expirevar"}}'
done < "tempdata.txt"
#!/bin/bash
while read line;
do
line_array=($line)
emailvar=${line_array[0]}
expirevar=${line_array[1]}
idvar=${line_array[2]}
jsonstring='{"user_id": "'$idvar'", "email": "'$emailvar'", "custom_attributes":{"Program_Expires_at": "'$expirevar'"}'
echo $jsonstring
done < 'tempdata.txt'
Output:
You have to escape the whitespace to make it part of the string, rather than creating a simple command with some pre-command assignments.
jsonstring=$idvar\ $emailvar\ $expirevar
more commonly written as
jsonstring="$idvar $emailvar $expirevar"
In your commented assignment, you used single quotes, which prevent parameter expansion. You need to use double quotes, which requires manually escaping the interior double quotes. More robust, though, is to use a tool like jq to generate the JSON for you: it will take care of escaping any characters in your variables to generate valid JSON.
jsonstring=$(jq -n \
--arg id "$idvar" \
--arg email "$emailvar" \
--arg expire "$expirevar" \
'{user_id: $id,
email: $email,
custom_attributes: {Program_Expires_at: $expire}}'
)
It seems this is essentially a problem with how bash handles variables and parameter expansion. I believe the solution here basically adding bunch of double quotes.
Double quotes can be used to enable parameter expansion for multiple variables. For JSON output in this bash script, we'll need to use nested double-quotes.
To fix this, we can:
put double quotes (") surrounding the value for jsonstring
escape double quotes surrounding strings used within the value for jsonstring with \
If you'd like $idvar and $expirevar to be interpreted as numbers instead of strings, you don't need escaped double-quotes around these values.
For example:
#!/bin/bash
while read emailvar idvar expirevar; do
jsonstring="{\"user_id\":$idvar,\"email\":\"$emailvar\",\"custom_attributes\":{\"Program_Expires_at\":$expirevar}}"
echo "$jsonstring"
done < "tempdata.txt"
Example output:
user#pc: bash ./script.sh
{"user_id":31460,"email":"mmm#gmail.com","custom_attributes":{"Program_Expires_at":147557432}}
user#pc: bash ./script.sh | jq .
{
"user_id": 31460,
"email": "mmm#gmail.com",
"custom_attributes": {
"Program_Expires_at": 147557432
}
}
I have a JSON where I am storing it in a variable in my Bash script like this:
RAW_JSON="{"secretKey": "ADFGHJKGBNJK"}"
I wanted to store the value of secretKey in another variable called secretValue. How can I do that?
To preserve double quotation (") marks around the key and value, the syntax is:
$ RAW_JSON='{"secretKey": "ADFGHJKGBNJK"}'
$ echo "$RAW_JSON"
{"secretKey": "ADFGHJKGBNJK"}
Then jq can be used to return the value:
$ secretValue=$(echo "$RAW_JSON" | jq -r .secretKey)
$ echo "$secretValue"
ADFGHJKGBNJK
The -r parameter removes the quotes from the result.
I'm trying to get jq to parse a JSON structure like:
{
"a" : 1,
"b" : 2,
"c" : "{\"id\":\"9ee ...\",\"parent\":\"abc...\"}\n"
}
That is, an element in the JSON is a string with escaped json.
So, I have something along the lines of
$ jq [.c] myFile.json | jq [.id]
But that crashes with jq: error: Cannot index string with string
This is because the output of .c is a string, not more JSON.
How do I get jq to parse this string?
My initial solution is to use sed to replace all the escape chars (\":\", \",\" and \") but that's messy, I assume there's a way built into jq to do this?
Thanks!
edit:
Also, the jq version available here is:
$ jq --version
jq version 1.3
I guess I could update it if required.
jq has the fromjson builtin for this:
jq '.c | fromjson | .id' myFile.json
fromjson was added in version 1.4.
You can use the raw output (-r) that will unescape characters:
jq -r .c myfile.json | jq .id
ADDENDUM: This has the advantage that it works in jq 1.3 and up; indeed, it should work in every version of jq that has the -r option.
Motivation: you want to parse JSON string - you want to escape a JSON object that's wrapped with quotes and represented as a String buffer, and convert it to a valid JSON object. For example:
some JSON unescaped string :
"{\"name\":\"John Doe\",\"position\":\"developer\"}"
the expected result ( a JSON object ):
{"name":"John Doe","position":"developer"}
Solution: In order to escape a JSON string and convert it into a valid JSON object use the sed tool in command line and use regex expressions to remove/replace specific characters:
cat current_json.txt | sed -e 's/\\\"/\"/g' -e 's/^.//g' -e 's/.$//g'
s/\\\"/\"/g replacing all backslashes and quotes ( \" ) into quotes only (")
s/^.//g replacing the first character in the stream to none character
s/.$//g replacing the last character in the stream to none character
I have a JSON string without the separator ,. How do I parse it using jq?
$echo '{"access_token":"XXXX""expires_in":300"token_type":"Bearer"}' | jq -r .access_token
The above line gives me the below error:
parse error: Expected separator between values at line 1
I understand that the issue is because the JSON string provided is not comma-separated. But this is what I am getting as a response from the server. How do I parse such a string? I want to retrive the value for key "access_token".
You can use a regular expression with sed if you know the accesss token never contains quotes.
echo '{"access_token":"XXXX""expires_in":300"token_type":"Bearer"}' |
sed 's/"access_token":"\([^"]*\)/\1/'
The capture group between \( and \) captures the string between the quotes, and \1 in the replacement string extracts it.
Here are two just-jq solutions, each with its own degree of brittleness. The first one attempts to convert each entire input line into valid JSON:
Using fromjson
echo '{"access_token":"XXXX""expires_in":300"token_type":"Bearer"}' |
jq -rR 'gsub("(?<k>\"[^\"]*\")"; "," + .k )
| gsub("{,\"";"{\"") | gsub(":,\""; ":\"")
| fromjson | .access_token'
XXXX
Assume the value is a string on the same line
jq -rR 'sub(".*\"access_token\" *: *\"(?<v>[^\"]*)\".*"; .v )'
I am trying to parse json files that contain sequences of slashes and backslashes in some of their strings like this:
echo '{"tag_string":"/\/\/\ test"}' | jq
which gives me:
parse error: Invalid escape at line 1, column 27
I have tried escaping with backslashes at different positions, but I can't seem to find a correct way. How do I output the string as it is, without removing any character or getting errors?
This only works on bash, but not sh (or zsh):
echo '{"tag_string":"/\\/\\/\\ test"}' | jq -r '.tag_string'
/\/\/\ test
A forward slash character is legal, but a single backslash character is not. According to json.org char description, the valid chars are:
char
any-Unicode-character-
except-"-or-\-or-
control-character
\"
\\
\/
\b
\f
\n
\r
\t
\u four-hex-digits
So in your example, the single backslashes are not legal, you need either "\\" which is interpreted as double backslashes, or you need to remove them entirely.
If you are trying to include literal backslashes:
(bash)
echo '{"tag_string":"/\\/\\/\\ test"}' | jq
{
"tag_string": "/\\/\\/\\ test"
}
echo '{"tag_string":"/\\/\\/\\ test"}' | jq -r '.["tag_string"]'
/\/\/\ test
(sh)
echo '{"tag_string":"/\\\\/\\\\/\\\\ test"}' | jq -r '.["tag_string"]'
/\/\/\ test
printf "%s" '{"tag_string":"/\\/\\/\\ test"}' | jq -r '.["tag_string"]'
/\/\/\ test
If you are trying to convert a file with non-JSON strings, then consider a tool such as any-json. Using the "cson-to-json" mode, "\/" will be interpreted as "/":
$ any-json -format=cson
Input:
{"tag_string":"/\/\/\ test"}
Output:
{
"tag_string": "/// test"
}