I have a string that needs to read from the database and sent as JSON. How can I make sure all quotes are escaped properly and handle any other characters that might make the String invalid JSON?
So for example I have the following code..
def jsonFormatted = new groovy.json.JsonBuilder(products:finalList).toPrettyString()
And finalList variable is an array of HashMaps, where each Map has a key value pair, for example..
"Product Id" -> "555"
"Product Name" -> "32" Flat Screen TV"
Because that open quote in product name is not escaped, the program which reads the JSON on the other end breaks.
Any solution that uses JsonBuilder or JsonSlurper libraries would be optimal.
It seems the only problem was I was calling toPrettyString() instead of toString() of JsonBuilder. Calling toString made JsonBuilder to escape that open quote. Special thanks to Gerd Castan for pointing this out.
Related
I'm calling an API from Go and trying to push json string data from another api call into it.
I can hand craft the calls using a payload like
payload := strings.NewReader('[{"value1":333, "value2":444}]'
and everything is happy.
I'm now trying to covert this to take the json string {"value1":333, "value2":444} as an input parameter of type string to a function, but when I try and use that as the payload, the api is responding with
expected type: JSONArray, found: JSONObject
I naively tried setting the input to the function as []string and appending the data to an array as the input, but then strings.NewReader complained that it was being fed an array.. which is was.
I'm at a loss to work out how to convert a string of json into a json array that the api will be happy with.
I tried just surrounding the string with [] but the compiler threw a fit about incorrect line termination.
Must have been doing something wrong with the string, surrounding the {} with [] let the function pass the data, but there must be a better way than this.
Any ideas, or am I making this harder than it should be?
You were on the right track with the brackets, but you actually need to append the characters to the string. For example:
str := `{"value1":333, "value2":444}`
str = "[" + str + "]"
// [{"value1":333, "value2":444}]
https://play.golang.org/p/rWHCLDCAngd
If you use brackets outside a string or rune literal, then it is parsed as Go language syntax.
We use Freemarker to transform one JSON to another. The input JSON is something like this:
{"k1": "a", "k2":"line1. \n line2"}
Post using the Freemarker template, the JSON is converted to:
{ \n\n "p1": "a", \n\n "p2": "line1. \n line2"}
Here is the logic we use to do the transformation
final Map<String, Object> input = JsonConverter.convertFromJson(input, Map.class);
final Template template = freeMarkerConfiguration.getTemplate("Template1.ftl");
final Writer out = new StringWriter();
template.process(input, out);
out.flush();
final String newlineFilteredResult = new JSONObject(out.toString).toString();
The conversion to JSON object fails due to a newline character inside a string for key k2 and gives the following exception:
Caused by: org.json.JSONException: Unterminated string at ...
I tried using the following but nothing works:
1. JSONObject.quote
2. JSONValue.escape
3. out.toString().replaceAll("[\n\r]+", "\\n");
I get the following exception due to the newline characters at the beginning as well:
Caused by: org.json.JSONException: Missing value at 1 [character 2 line 1]
Could someone please point me in the correct direction.
Edit
After further clarification from OP he had "${key}": "${value}" in his freemarker template and ${value} could contain line brakes. The solution in this case is to use ${value?json_string}.
Starting from FreeMarker 2.3.32 you can write "${key}": ${value?c} instead of "${key}": "${value}", because if the left-side of ?c is a string, now instead of failing, it quotes and escapes the string. Thus you don't even have to know if the left-side is a number/boolean, which must not be quoted (and ?c won't quote them), or a string, which must be quoted, as it's automatic.
Also, if the left-value is known to be missing/null sometimes, them ?cn will handle that case by printing a null literal.
Also, check out the c_format setting for best results, but by default string formatting is JSON compatible, so using ?c will be an improvement even without setting that.
I am trying to invoke my Lambda function by passing parameters as below. it contains apostrophe(').
https://invoke_url?param=[["kurlo jack's book","Adventure Books",8.8,1]]
Stringifyed to be 'https://invoke_url?param=%5B%5B%229780786706211%22s....`
I used the mapping below to pass parameter to lambda
"query": {
#foreach($queryParam in $input.params().querystring.keySet())
"$queryParam": "$util.escapeJavaScript($input.params().querystring.get($queryParam))" #if($foreach.hasNext),#end
#end
}
I got following error
{"message": "Could not parse request body into json: Unrecognized character escape \'\'\' (code 39)\n at [Source: [B#5b70c341; line: 29, column: 65]"}
i have also tried after removing double quotes from mapping template. But did't work.
Be sure to add .replaceAll("\\'","'") to your request body passthrough template after .escapeJavaScript(data)
I found this bit from AWS's documentation to be very helpful for this issue:
$util.escapeJavaScript()
Escapes the characters in a string using JavaScript string rules.
Note This function will turn any regular single quotes (') into
escaped ones (\'). However, the escaped single quotes are not valid in
JSON. Thus, when the output from this function is used in a JSON
property, you must turn any escaped single quotes (\') back to regular
single quotes ('). This is shown in the following example:
$util.escapeJavaScript(data).replaceAll("\\'","'")
I don't have a solution but I have narrowed the root cause. Lambda does not seem to like single quotes to be escaped with a single slash.
If you hardcode your mapping template to look like this:
{
"query-fixed": {
"param": "[[\"kurlo jack\\'s book\",\"Adventure Books\",8.8,1]]"
}
}
my test Lambda invocation succeeds. However, if you hardcode the template to this:
{
"query-fixed": {
"param": "[[\"kurlo jack\'s book\",\"Adventure Books\",8.8,1]]"
}
}
I get the same error message that you got above. Unfortunately, the second variation is what API Gateway produces for the Lambda invocation.
A workaround might involve using the template to replace single quotes escaped with slash to two slashes. See Replace a Substring of a String in Velocity Template Language
I'll follow up with Lambda internally and update if I hear anything or have a functional workaround.
Try changing your encoding of ' to %27 as per what is is defined in this W3Schools page (ironically their example does not encodes the single quote either, I guess its because it belongs to the "supported" ASSCII set of characters)
The "query string" (the part in the hyperlink after ?) must be a string. Whatever you have constructing that must be appended to it like: https://invoke_url?a=x&b=y
In your Lambda code put:
if( event.hasOwnProperty( 'params' ) )
if( event.params.hasOwnProperty( 'querystring' ) )
params = event.params.querystring;
(obviously some extraneous checks, probably unnecessary but ehh)
In your API Gateway go to:
APIs -> api_name -> Resources -> invoke_url -> GET -> Method Execution
Under URL Query String Parameters "Add query string" a and b (or whatever)
When you hit www.com/invoke_url?a=x&b=y you can now access them with:
...
params = event.params.querystring;
console.log( params.a, params.b );
...
So I have some json that looks like this, which I got after taking it out of some other json by doing response.body.to_json:
{\n \"access_token\": \"<some_access_token>\",\n \"token_type\": \"Bearer\",\n \"expires_in\": 3600,\n \"id_token\": \<some_token>\"\n}\n"
I want to pull out the access_token, so I do
to_return = {token: responseJson[:access_token]}
but this gives me a
TypeError: no implicit conversion of Symbol into Integer
Why? How do I get my access token out? Why are there random backslashes everywhere?
to_json doesn't parse JSON - it does the complete opposite: it turns a ruby object into a string containing the JSON representation of that object is.
It's not clear from your question what response.body is. It could be a string, or depending on your http library it might have already been parsed for you.
If the latter then
response.body["access_token"]
Will be your token, if the former then try
JSON.parse(response.body)["access_token"]
Use with double quotes when calling access_token. Like below:
to_return = {token: responseJson["access_token"]}
Or backslashes are escaped delimiters and make sure you first parse JSON.
I'm attempting to read the following JSON file ("my_file.json") into R, which contains the following:
[{"id":"484","comment":"They call me "Bruce""}]
using the jsonlite package (0.9.12), the following fails:
library(jsonlite)
fromJSON(readLines('~/my_file.json'))
receiving an error:
"Error in parseJSON(txt) : lexical error: invalid char in json text.
84","comment":"They call me "Bruce""}]
(right here) ------^"
Here is the output from R escaping of the file:
readLines('~/my_file.json')
"[{\"id\":\"484\",\"comment\":\"They call me \"Bruce\"\"}]"
Removing the quotes around "Bruce" solves the problem, as in:
my_file.json
[{"id":"484","comment":"They call me Bruce"}]
But what is the issue with the escapement?
In R strings literals can be defined using single or double quotes.
e.g.
s1 <- 'hello'
s2 <- "world"
Of course, if you want to include double quotes inside a string literal defined using double quotes you need to escape (using backslash) the inner quotes, otherwise the R code parser won't be able to detect the end of the string correctly (the same holds for single quote).
e.g.
s1 <- "Hello, my name is \"John\""
If you print (using cat¹) this string on the console, or you write this string on a file you will get the actual "face" of the string, not the R literal representation, that is :
> cat("Hello, my name is \"John\"")
Hello, my name is "John"
The json parser, reads the actual "face" of the string, so, in your case json reads :
[{"id":"484","comment":"They call me "Bruce""}]
not (the R literal representation) :
"[{\"id\":\"484\",\"comment\":\"They call me \"Bruce\"\"}]"
That being said, also the json parser needs double-quotes escaping when you have quotes inside strings.
Hence, your string should be modified in this way :
[{"id":"484","comment":"They call me \"Bruce\""}]
If you simply modify your file by adding the backslashes you will be perfectly able to read the json.
Note that the corresponding R literal representation of that string would be :
"[{\"id\":\"484\",\"comment\":\"They call me \\\"Bruce\\\"\"}]"
in fact, this works :
> fromJSON("[{\"id\":\"484\",\"comment\":\"They call me \\\"Bruce\\\"\"}]")
id comment
1 484 They call me "Bruce"
¹
the default R print function (invoked also when you simply press ENTER on a value) returns the corresponding R string literal. If you want to print the actual string, you need to use print(quote=F,stringToPrint), or cat function.
EDIT (on #EngrStudent comment on the possibility to automatize quotes escaping) :
Json parser cannot do quotes escaping automatically.
I mean, try to put yourself in the computer's shoes and image you should parse this (unescaped) string as json: { "foo1" : " : "foo2" : "foo3" }
I see at least three possible escaping giving a valid json:
{ "foo1" : " : \"foo2\" : \"foo3" }
{ "foo1\" : " : "foo2\" : \"foo3" }
{ "foo1\" : \" : \"foo2" : "foo3" }
As you can see from this small example, escaping is really necessary to avoid ambiguities.
Maybe, if the string you want to escape has a really particular structure where you can recognize (without uncertainty) the double-quotes needing to be escaped, you can create your own automatic escaping procedure, but you need to start from scratch, because there's nothing built-in.