I am trying to submit a spark job in spark job server with input in json format. However in my case one of values contains '!' character, which is not allowing me to parse it.Here is my input and response.
Input
curl -d "{"test.input1"="abc", "test.input2"="def!"}" 'http://localhost:8090/jobs?appName=my_spark_job&classPath=com.example.spark.job.MySparkJob'
Response
"result": "Cannot parse config: String: 1: Reserved character '!' is not allowed outside quotes (if you intended '!' (Reserved character '!' is not allowed outside quotes) to be part of the value for 'test.input2', try enclosing the value in double quotes, or you may be able to rename the file .properties rather than .conf)"
The value of "test.input2" is already in double quotes. I tried adding single/double quotes but still didnt work. Any thoughts how can i parse it.
Thanks
Put everything in a json file and do the following
curl --data-binary #/path/to/json/file 'http://localhost:8090/jobs?appName=my_spark_job&classPath=com.example.spark.job.MySparkJob'
Related
I'm getting
/json/common.rb:156:in `parse': 783: unexpected token at '' (JSON::ParserError) while trying to parse a json file in ruby. The problem was because there were some single quotes in one of the strings:
parsed = JSON.parse("{
\"key1\":\"value1\",
\"key2\":\"value2\",
\"key3\":12345,
\"key4\":\"''value4''\",
}")
Is there a way to escape the single quotes in the strings without affect words like don't? The json is read from a file using JSON.parse(file.get_input_stream.read) that's why there are \.
The single quotes aren't your problem, your problem is that you have a stray trailing comma:
parsed = JSON.parse("{
\"key1\":\"value1\",
\"key2\":\"value2\",
\"key3\":12345,
\"key4\":\"''value4''\",
}") #--------------------^ This should not be there.
JSON doesn't allow that comma so you don't actually have a JSON file.
You should figure out where the file came from and fix that tool to produce real JSON rather than the "looks mostly like JSON" that is currently being written to the file.
We are loading data into Snowflake using a JavaScript procedure.
The script will loop over an array of objects to load some data. These objects contain string that may have special characters.
i.e.:
"Description": "This file contain "sensitive" information."
The double quotes on sensitive word will become:
"Description": "This file contain \"sensitive\" information."
Which broke the loading script.
The same issue happened when we used HTML tags within description key:
"Description": "Please use <b>specific fonts</b> to update the file".
This is another example on the Snowflake community site.
Also this post recommended setting FIELD_OPTIONALLY_ENCLOSED_BY equal to the special characters, but I am handling large data set which might have all the special characters.
How can we escape special characters automatically without updating the script and use JavaScript to loop over the whole array to anticipate and replace each special character with something else?
EDIT
I tried using JSON_EXTRACT_PATH_TEXT:
select JSON_EXTRACT_PATH_TEXT(parse_json('{
"description": "Please use \"Custom\" fonts"
}'), 'description');
and got the following error:
Error parsing JSON: missing comma, line 2, pos 33.
I think the escape characters generated by the JS procedure are escaped when passing to SQL functions.
'{"description": "Please use \"Custom\" fonts"}'
becomes
'{"description": "Please use "Custom" fonts"}'
Therefore parsing them as JSON/fetching a field from JSON fails. To avoid error, the JavaScript procedure should generate a double backslash instead of a backslash:
'{"description": "Please use \\"Custom\\" fonts"}'
I do not think there is a way to prevent this error without modifying the JavaScript procedure.
I came across this today, Gokhan is right you need the double backslashes to properly escape the quote.
Here are a couple links that explain it a little more:
https://community.snowflake.com/s/article/Escaping-new-line-character-in-JSON-to-avoid-data-loading-errors
https://community.snowflake.com/s/article/Unable-to-Insert-Data-Containing-Back-Slash-from-Stored-Procedure
For my case I found that I could address this challenge by disabling the escaping and then manually replacing the using replace function.
For your example the replace is not necessary.
select parse_json($${"description": "Please use \"Custom\" fonts"}$$);
select parse_json($${"description": "Please use \"Custom\" fonts"}$$):description;
I am trying to escape delimiter character that appears inside data. Is there a way to do it by passing option parameters? I can do it from udf, but I am hoping it is possible using options.
val df = Seq((8, "test,me\nand your", "other")).toDF("number", "test", "t")
df.coalesce(1).write.mode("overwrite").format("csv").option("quote", "\u0000").option("delimiter", ",").option("escape", "\\").save("testcsv1")
But the escape is not working. The output file is written as
8,test,me
and your,other
I want the output file to be written as.
8,test\,me\\nand your,other
I'm not certain, but I think if you had your sequence as
Seq((8, "test\\,me\\\\nand your", "other"))
and did not specify a custom escape character, it would behave as you are expecting and give you 8,test\,me\\nand your,other as the output. This is because \\ acts simply as the character '\' rather than an escape, so they are printed where you want and the n immediately after is not interpreted as part of a newline character.
I have the following two lines of code:
json_str = _cases.to_json
path += " #{USER} #{PASS} #{json_str}"
When I use the debugger, I noticed that json_str appears to be formatted as JSON:
"[["FMCE","Wiltone","Wiltone","04/10/2018","Marriage + - DOM"]]"
However, when I interpolate it into another string, the quotes are removed:
"node superuser 123456 [["FMCE","Wiltone","Wiltone","04/10/2018","Marriage + - DOM"]]"
Why does string interpolation remove the quotes from JSON string and how can I resolve this?
I did find one solution to the problem, which was manually escaping the string:
json_str = _cases.to_json.gsub('"','\"')
path += " #{USER} #{PASS} \"#{json_str}\""
So basically I escape the double quotes generated in the to_json call. Then I manually add two escaped quotes around the interpolated variable. This will produce a desired result:
node superuser 123456 "[[\"FMCE\",\"Wiltone\",\"Wiltone\",\"04/10/2018\",\"Marriage + - DOM\"]]"
Notice how the outer quotes around the collection are not escaped, but the strings inside the collection are escaped. That will enable JavaScript to parse it with JSON.parse.
It is important to note that in this part:
json_str = _cases.to_json.gsub('"','\"')
it is adding a LITERAL backslash. Not an escape sequence.
But in this part:
path += " #{USER} #{PASS} \"#{json_str}\""
The \" wrapping the interpolated variable is an escape sequence and NOT a literal backslash.
Why do you think the first and last quote marks are part of the string? They do not belong to the JSON format. Your program’s behavior looks correct to me.
(Or more precisely, your program seems to be doing exactly what you told it to. Whether your instructions are any good is a question I can’t answer without more context.)
It's hard to tell with the small sample, but it looks like you might be getting quotes from your debugger output. assuming the output of .to_json is a string (usually is), then "#{json_str}" should be exactly equal to json_str. If it isn't, that's a bug in ruby somehow (doubtful).
If you need the quotes, you need to either add them manually or escape the string using whatever escape function is appropriate for your use case. You could use .to_json as your escape function even ("#{json_str.to_json}", for example).
When I click finish import, Parse.com says "invalid character '\'' looking for beginning of value". However, there is not a single character "\" in my entire file. You can check it below.
Apparently, this is because of using single quotes instead of double quotes.
Can I use "name": 'Samat', instead of "name": "Samat"?
https://gist.github.com/samatdav/61db29a676da21dc4bbd
The JSON format specification is very clear about this: String values must be enclosed in double quotes. Single quotes or unquoted values (other than true, false, null, numbers or nested object/array definitions) are not allowed.
JavaScript's internal object notation is much less strict in that regard, as it generally allows single-quoted strings. However, JSON is only a subset of the original JavaScript object notation syntax.
For anyone that might need this later.
As ipfs daemon --help suggests, the cors domain can be set via
>ipfs config --json API.HTTPHeaders.Access-Control-Allow-Origin '["*"]'
which in Windows yields
Error: failed to unmarshal json. invalid character '\'' looking for beginning of value
The correct version should be
>ipfs config --json API.HTTPHeaders.Access-Control-Allow-Origin "[\"*\"]"
The error itself is telling you the invalid character is ' the single quote. It's just represented as a \' since they are using single quotes to enclose the invalid character the character must be escaped.
"invalid character '\'' looking for beginning of value"
^ ^ notice the single quotes.
The issue in your gist is that single quotes are not valid representation of strings in JSON.
Note
{
"foo": 'bar'
}
Yields the following error on JSONLint
Parse error on line 2:
{ "foo": 'bar'}
------------^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['