I have a script which uses a variable which contains a value in the form of '11/12/2014 07:58:08.157'. The script contains a curl command with this variable forming part of the http body (JSON request). The issue is the server is throwing a JSON parser error when this value is passed in the body. Replacing this value with '11/12/2014' seems to work fine, so it seems the time part of this string is causing issues. Using an echo command I have seen the curl command have this value enclosed in double quotes. (I have even tried single quotes.) Either way, I am still getting a JSON parser error. Removing the time component leads to another functional error, so I am hard pressed to pass both the date and time value to the server in the JSON body.
Related
We are loading data into Snowflake using a JavaScript procedure.
The script will loop over an array of objects to load some data. These objects contain string that may have special characters.
i.e.:
"Description": "This file contain "sensitive" information."
The double quotes on sensitive word will become:
"Description": "This file contain \"sensitive\" information."
Which broke the loading script.
The same issue happened when we used HTML tags within description key:
"Description": "Please use <b>specific fonts</b> to update the file".
This is another example on the Snowflake community site.
Also this post recommended setting FIELD_OPTIONALLY_ENCLOSED_BY equal to the special characters, but I am handling large data set which might have all the special characters.
How can we escape special characters automatically without updating the script and use JavaScript to loop over the whole array to anticipate and replace each special character with something else?
EDIT
I tried using JSON_EXTRACT_PATH_TEXT:
select JSON_EXTRACT_PATH_TEXT(parse_json('{
"description": "Please use \"Custom\" fonts"
}'), 'description');
and got the following error:
Error parsing JSON: missing comma, line 2, pos 33.
I think the escape characters generated by the JS procedure are escaped when passing to SQL functions.
'{"description": "Please use \"Custom\" fonts"}'
becomes
'{"description": "Please use "Custom" fonts"}'
Therefore parsing them as JSON/fetching a field from JSON fails. To avoid error, the JavaScript procedure should generate a double backslash instead of a backslash:
'{"description": "Please use \\"Custom\\" fonts"}'
I do not think there is a way to prevent this error without modifying the JavaScript procedure.
I came across this today, Gokhan is right you need the double backslashes to properly escape the quote.
Here are a couple links that explain it a little more:
https://community.snowflake.com/s/article/Escaping-new-line-character-in-JSON-to-avoid-data-loading-errors
https://community.snowflake.com/s/article/Unable-to-Insert-Data-Containing-Back-Slash-from-Stored-Procedure
For my case I found that I could address this challenge by disabling the escaping and then manually replacing the using replace function.
For your example the replace is not necessary.
select parse_json($${"description": "Please use \"Custom\" fonts"}$$);
select parse_json($${"description": "Please use \"Custom\" fonts"}$$):description;
This is how I am executing my python from Execute process in SSIS:
I have the same command in .bat file and I can execute .bat from SSIS execute task but if I put the whole command like above it doesn't work. Any help is appreciated.
Error I get is
The process exit code was "2" while the expected was "0".I have attached the screenshot of my SSIS executeprocess taskenter image description here
C:/Users/datadude/AppData/Local/Continuum/miniconda3/envs/ra_platform-201909/python.exe e:/Source/Scripts/Python/rapc/kozuchi/core/qc/solactive_returns_qc.py --datasource solactive
As comment from Nick.McDemaid says - to get proper help you need to provide better diagnostics. Try running it in Debug mode in SSIS as Execute container first and show the errors output? Here is a shot at an answer though, maybe it will help others.
I think the problem is in the way you use quotes in the Arguments field. The Arguments field is treated somewhat surprisingly by SSIS. In my tests it was always quoted (extra surrounding quotes added by SSIS) when passed as an argument to the command (Executable) of the execute process task.
I had similar problem when I was trying to evaluate a variable to be passed as an extra argument to a python script (i.e. part of the arguments field). The whole Arguments string was quoted and variable not evaluated.
The solution (to my problem) was to use Expressions (third option on the left in the dialog), and select Arguments as an expression. Then the Arguments expression was fully flexible. In your case I think you do not need the quotes, or should try building the Arguments as an expression.
Error sending JSON structure using aws-cli in Powershell. Specifically a call to put an item into an existing DynamoDB table.
The problem seems to be that the lack of double quotes around keys and values in the JSON object I'm attempting to send. I've read that Powershell is finicky with outputting double quotes, especially when leveraging external APIs.
Unfortunately, since my org uses okta for authenticating AWS requests, I have to use Powershell.
I've tried everything that I've seen here:
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/convertto-json?view=powershell-6
...here:
Error parsing parameter '--expression-attribute-values': Invalid JSON: Expecting property name enclosed in double quotes: line 1 column 3 (char 2)
...here:
https://github.com/aws/aws-cli/issues/1326
...and here:
PowerShell: best way to escape double quotes in string passed to an external program? E.g., a JSON string
WHAT I'VE TRIED:
This is the basic first attempt:
okta-aws dh dynamodb put-item --table-name AlexaRoomLookup-dev --item '{"deviceId": {"S":"amzn1.ask.device.AEH2LHYGV7GSPP5THMR
5H56AI2OOMAQ7MF54CZ3E6WR433WGS6QAOCYCKJWRJ3TQY5IE76NWR2IKCANB6TJNKLDEZOO2YN6ACUVT33MKSS4CO6R7GJI6GDFLOBOPUA2IXX7RI732UXJ6PDST5KYC7CSQK
634K4APEBRNVOKVZIDECOCBBIFB4"},"roomNumber": {"N":9110}}' --return-consumed-capacity TOTAL
Then I tried escaping with backslash:
okta-aws dh dynamodb put-item --table-name AlexaRoomLookup-dev --item {\"deviceId\":{\"S\":\"amzn1.ask.device.AEH2LHYGV7GSPP5THMR5H56AI2OOMAQ7MF54CZ3E6WR433WGS6QAOCYCKJWRJ3TQY5IE76NWR2IKCANB6TJNKLDEZOO2YN6ACUVT33MKSS4CO6R7GJI6GDFLOBOPUA2IXX7RI732UXJ6PDST5KYC7CSQK634K4APEBRNVOKVZIDECOCBBIFB4\"},\"roomNumber\": {\"N\":9110}} --return-consumed-capacity TOTAL
Then esacping with backtick (which i've replaced here with an asterisk so SO would read it as code) and backslash:
{*"deviceId*": {*"S*":*"amzn1.ask.device.AEH2LHYGV7GSPP
5THMR5H56AI2OOMAQ7MF54CZ3E6WR433WGS6QAOCYCKJWRJ3TQY5IE76NWR2IKCANB6TJNKLDEZOO2YN6ACUVT33MKSS4CO6R7GJI6GDFLOBOPUA2IXX7RI732UXJ6PDST5KYC
7CSQK634K4APEBRNVOKVZIDECOCBBIFB4*"},*"roomNumber*": {*"N*":9110}}" --return- consumed-capacity TOTAL
I then tried a "here string" to no avail.
EXPECTATIONS and RESULTS:
I would expect a method of escaping that's in the microsoft documentation to work.
Each of the above gave this error with a variation of the problematic "JSON received" based on the escape method, but it never had double quotes around keys and values:
Error parsing parameter '--item': Invalid JSON: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
JSON received: {deviceId: {S:amzn1.ask.device.AEH2LHYGV7GSPP5THMR5H56AI2OOMAQ7MF54CZ3E6WR433WGS6QAOCYCKJWRJ3TQY5IE76NWR2IKCANB6TJNKLDEZOO2YN6ACUVT33MKSS4CO6R7GJI6GDFLOBOPUA2IXX7RI732UXJ6PDST5KYC7CSQK634K4APEBRNVOKVZIDECOCBBIFB4},roomNumber: {N:9110}}
The only thing that seemed to work was using "file://file.json" as the input to --item, which I couldn't find documented anywhere... I think it was on that github thread I linked. However, I'd rather not have to edit a file every time I want to send JSON with an AWS API call... Here it is:
okta-aws dh dynamodb put-item --table-name AlexaRoomLookup-dev --item file://file.json --return-consumed-capacity TOTAL
Can anyone provide info other than what's listed here as to why the above methods wouldn't work? Have I just implemented them incorrectly?
Thanks.
I was having the same issue while trying to send a message to Amazon SQS with JSON body using PowerShell. After trying different escape characters, the following worked for me.
aws sqs send-message --queue-url "<queue-url>" --message-body '{\""key1\"": \""value1\"",\""key2\"": \""value2\"",\""key3\"": \""value3\"" }'
OS: Windows 10 Pro (Version 1803)
AWS CLI version: 1.16.180
For further information, see the official documentation.
Using quotation marks with strings in the AWS CLI
You could try the PowerShell "stop parsing symbol" (i.e. "--%") at the start of the command. This tells PowerShell to use the rest of the parameters verbatim.
PS> okta-aws --% dh dynamodb put-item --table-name AlexaRoomLookup-dev --item '{"deviceId": {"S":"amzn1.ask.device.AEH...etc...FB4"},"roomNumber": {"N":9110}}' --return-consumed-capacity TOTAL
See about_parsing for more details...
It won't help if your json is in a variable, but if it's hard-coded like your example above it might work.
How I can load OSM data to ArangoDB?
I loaded data sed named luxembourg-latest.osm.pbf from OSM, than converted it to JSON with OSMTOGEOJSON, after I tried to load result geojson to ArangoDB with next command: arangoimp --file out.json --collection lux1 --server.database geodb and got hude list of errors:
...
2017-03-17T12:44:28Z [7712] WARNING at position 719386: invalid JSON type (expecting object, probably parse error), offending context: ],
2017-03-17T12:44:28Z [7712] WARNING at position 719387: invalid JSON type (expecting object, probably parse error), offending context: [
2017-03-17T12:44:28Z [7712] WARNING at position 719388: invalid JSON type (expecting object, probably parse error), offending context: 5.867441,
...
What I am doing wrong?
upd: it's seems that converter osm2json converter should be run with option osmtogeojson --ndjson that produce items not as single Json, but in line by line mode.
As #dmitry-bubnenkov already found out, --ndjson is required to produce the right input for ArangoImp.
One has to know here, that ArangoImp expects a JSON-Subset (since it doesn't parse the json on its own) dubbed as JSONL.
Thus, Each line of the JSON-File is expected to become one json document in the collection after the import. To maximize performance and simplify the implementation, The json is not completely parsed before sending it to the server.
It tries to chop the JSON into chunks with the maximum request size that the server permits. It leans on the JSONL-line endings to isolate possible chunks.
However, the server expects valid JSON for sure. Sending the chopped part to the server with possibly incomplete JSON documents will lead to parse errors on the server, which is the error message you saw in your output.
I am trying to submit a spark job in spark job server with input in json format. However in my case one of values contains '!' character, which is not allowing me to parse it.Here is my input and response.
Input
curl -d "{"test.input1"="abc", "test.input2"="def!"}" 'http://localhost:8090/jobs?appName=my_spark_job&classPath=com.example.spark.job.MySparkJob'
Response
"result": "Cannot parse config: String: 1: Reserved character '!' is not allowed outside quotes (if you intended '!' (Reserved character '!' is not allowed outside quotes) to be part of the value for 'test.input2', try enclosing the value in double quotes, or you may be able to rename the file .properties rather than .conf)"
The value of "test.input2" is already in double quotes. I tried adding single/double quotes but still didnt work. Any thoughts how can i parse it.
Thanks
Put everything in a json file and do the following
curl --data-binary #/path/to/json/file 'http://localhost:8090/jobs?appName=my_spark_job&classPath=com.example.spark.job.MySparkJob'