I have some info store in a MySQL database, something like: AHmmgZq\n/+AH+G4
We get that using an API, so when I read it in my python I get: AHmmgZq\\n/+AH+G4 The backslash is doubled!
Now I need to put that into a JSON file, how can I remove the extra backslash?
EDIT: let me show my full code:
json_dict = {
"private_key": "AHmmgZq\\n/+AH+G4"
}
print(json_dict)
print(json_dict['private_key'])
with open(file_name, "w", encoding="utf-8") as f:
json.dump(json_dict, f, ensure_ascii=False, indent=2)
In the first print I have the doubled backslash, but in the second one there's only one. When I dump it to the json file it gives me doubled.
"AHmmgZq\\n/+AH+G4" in python is equivalent to the literal string "AHmmgZq\n/+AH+G4". print("AHmmgZq\\n/+AH+G4") => "AHmmgZq\n/+AH+G4"
\n is a new line character in python. So to represent \n literally it needs to be escaped with a \. I would first try to convert to json as is and see if that works.
Otherwise for removing extra backslashs:
string_to_json.replace("\\\\","\\")
Remember that \\ = escaped \ = \
But in the above string that will not help you, because python reads "AHmmgZq\\n/+AH+G4" as "AHmmgZq\n/+AH+G4" and so finds no double backslash.
What solved my problem was this:
string_to_json.replace("\\n","\n")
Thanks to everybody!
Related
I have a simple file with this test line:
mmm#gmail.com 31460 147557432
My goal is to send as json data.
In my while loop I can echo the variables in the second line of my code example.
However, when I attempt to assign them to jsonstring and echo, the values are not populated.
What do I need to do to pass these values to my json string?
while read emailvar idvar expirevar; do
echo "$emailvar:$expirevar:$idvar"
jsonstring=$idvar $emailvar $expirevar
echo "$jsonstring"
#jsonstring='{"user_id":"$idvar","email":"$emailvar","custom_attributes":{"Program_Expires_at":"$expirevar"}}'
done < "tempdata.txt"
#!/bin/bash
while read line;
do
line_array=($line)
emailvar=${line_array[0]}
expirevar=${line_array[1]}
idvar=${line_array[2]}
jsonstring='{"user_id": "'$idvar'", "email": "'$emailvar'", "custom_attributes":{"Program_Expires_at": "'$expirevar'"}'
echo $jsonstring
done < 'tempdata.txt'
Output:
You have to escape the whitespace to make it part of the string, rather than creating a simple command with some pre-command assignments.
jsonstring=$idvar\ $emailvar\ $expirevar
more commonly written as
jsonstring="$idvar $emailvar $expirevar"
In your commented assignment, you used single quotes, which prevent parameter expansion. You need to use double quotes, which requires manually escaping the interior double quotes. More robust, though, is to use a tool like jq to generate the JSON for you: it will take care of escaping any characters in your variables to generate valid JSON.
jsonstring=$(jq -n \
--arg id "$idvar" \
--arg email "$emailvar" \
--arg expire "$expirevar" \
'{user_id: $id,
email: $email,
custom_attributes: {Program_Expires_at: $expire}}'
)
It seems this is essentially a problem with how bash handles variables and parameter expansion. I believe the solution here basically adding bunch of double quotes.
Double quotes can be used to enable parameter expansion for multiple variables. For JSON output in this bash script, we'll need to use nested double-quotes.
To fix this, we can:
put double quotes (") surrounding the value for jsonstring
escape double quotes surrounding strings used within the value for jsonstring with \
If you'd like $idvar and $expirevar to be interpreted as numbers instead of strings, you don't need escaped double-quotes around these values.
For example:
#!/bin/bash
while read emailvar idvar expirevar; do
jsonstring="{\"user_id\":$idvar,\"email\":\"$emailvar\",\"custom_attributes\":{\"Program_Expires_at\":$expirevar}}"
echo "$jsonstring"
done < "tempdata.txt"
Example output:
user#pc: bash ./script.sh
{"user_id":31460,"email":"mmm#gmail.com","custom_attributes":{"Program_Expires_at":147557432}}
user#pc: bash ./script.sh | jq .
{
"user_id": 31460,
"email": "mmm#gmail.com",
"custom_attributes": {
"Program_Expires_at": 147557432
}
}
I have a client sending smb user info as json object. The user has two backslashes - for example: {"user":"AD.EXAMPLE\\stack", "password": "gamma"}
I have a golang app that unmarshalls as below:
type smb_cred struct {
User string `json:"user"`
Password string `json:"password"`
}
var cred smb_cred
err = json.Unmarshal(input, &cred)
After unmarshal, it strips off one backslash, so the resulting string contains only one slash. I am trying to split the string with slash separator to separate user and AD domain to pass onto cifs mounting as : mount -t cifs -o username=stack,password=gamma,domain=AD.EXAMPLE.
How to split string with baskslash as separator?
----update with the fix----
strings.Split(cred.User, "\\") fixed my issue.
Backslashes are the escape character in JSON. The character after a backslash has special meaning.
\\ is a single \
\" is a single "
\n is a newline.
\u2603 is a snowman ☃.
So "AD.EXAMPLE\\stack" is the string AD.EXAMPLE\stack. That is correct JSON, Go is doing the correct thing by interpreting \\ as \.
If you truly want AD.EXAMPLE\\stack you need to add another escaped \. "AD.EXAMPLE\\\\stack".
I have an array which I want to convert to a JSON string. One of the elements has a backtick. This would cause an error when I try to run the command in the shell:
data = [["305", "John Smith", "Amy Smith`", "10/11/2008", "Medical", {"page_count"=>4}]]
json_str = data.to_json.gsub('"','\"')
cmd = "node myscript.js #{json_str}"
Open3.popen3(cmd) do |stdin, stdout, stderr, wait_thr|
output = [stdout.read, stderr.read]
end
Error retrieving data: sh: 1: Syntax error: EOF in backquote substitution
An obvious solution is to escape the backtick:
json_str = data.to_json.gsub('"','\"').gsub('`','\\\`')
But I want to escape all special shell characters that could raise an isuse. Ruby's shellescape escapes a string so that it can be safely used in a Bourne shell command line. Here's an example:
argv = "It's better to give than to receive".shellescape
argv #=> "It\\'s\\ better\\ to\\ give\\ than\\ to\\ receive"
But look what happens when I apply it to JSON string:
data = [["305", "John Smith", "Amy Smith`", "10/11/2008", "Medical", {"page_count"=>4}]]
data = data.to_json
=> "[[\"305\",\"John Smith\",\"Amy Smith`\",\"10/11/2008\",\"Medical\",{\"page_count\":4}]]"
data = data.to_json.shellescape
=> "\\"\\\\"\[\[\\\\\\\\"305\\\\\\\\",\\\\\\\\"John\ Smith\\\\\\\\",\\\\\\\\"Amy\ Smith\`\\\\\\\\",\\\\\\\\"10/11/2008\\\\\\\\",\\\\\\\\"Medical\\\\\\\\",\{\\\\\\\\"page_count\\\\\\\\":4\}\]\]\\\\"\\""
Clearly, this would raise an error like:
SyntaxError: Unexpected token \ in JSON at position 0
What happens is that shellescape will also escape spaces, since the shell requires spaces to be escaped. But having spaces is valid and necessary JSON. So how could I escape shell characters that would cause an error in my command without it breaking the JSON?
Shells are for humans, not for machines. Having a machine produce shell commands is a code smell indicating that you're automating at the wrong layer.
Skip the shell, and just run your program with the required arguments:
data = [["305", "John Smith", "Amy Smith`", "10/11/2008", "Medical", {"page_count"=>4}]]
json_str = data.to_json
Open3.popen3("node", "myscript.js", json_str) do |stdin, stdout, stderr, wait_thr|
output = [stdout.read, stderr.read]
end
Since there is no shell involved, there's no human silliness like escaping to care about.
When trying to pass the user defined value to the body content, I am getting error "message": "Bad JSON escape sequence: \S. ---- \r\nUnexpected character encountered while parsing value".
When passing complete raw payload through body data, I am not getting this error.
With User defined Variable,
"customerBillingAddress":"26 Chestnut St\Suite 2B\Andover, MA 01810",
convert as " "customerBillingAddress":"26 Chestnut St\Suite 2B\Andover, MA 01810","
"\" is throwing error.
When testing with raw data, I am getting as it is in the payload.
"customerBillingAddress":"26 Chestnut St\\Suite 2B\\Andover, MA 01810",
Please advise
You need to escape \ with \\ and double quote " with \" as per JSON format guideline. Here I think you only need to escape \ in your JSON payload like below .
"customerBillingAddress":"26 Chestnut St\\Suite 2B\\Andover, MA 01810"
You need to escape the following characters in JSON:
\b Backspace (ascii code 08)
\f Form feed (ascii code 0C)
\n New line
\r Carriage return
\t Tab
\" Double quote
\\ Backslash character
In order to do this automatically you can use __groovy() function available since JMeter 3.1 like:
${__groovy(org.apache.commons.lang3.StringEscapeUtils.escapeJson(vars.get('json')),)}
Demo:
Here is my code:
import json
a = "\1"
print json.dumps(a)
It returns "\u0001", instead of desired "\1".
Is there any way to get the exact character after passing with json dumps.
In Python, the string literal "\1" represents just one character of which the character code is 1. The backslash functions here as an escape to provide the character code as an octal value.
So either escape the backslash like this:
a = "\\1"
Or use the raw string literal notation with the r prefix:
a = r"\1"
Both will assign exactly the same string: print a produces:
\1
The output of json.dumps(a) will be:
"\\1"
... as also in JSON format, a literal backslash (reverse solidus) needs to be escaped by another backslash. But it truly represents \1.
The following prints True:
print a == json.loads(json.dumps(a))