I am trying to insert JSON payload into memsql JSON type column but it is failing due to the following reason.
My JSON content having '?' character.
I tried to escape '?' by using the following ways, but it doesn't worked for me.
The Exception i am getting is:
Root Exception stack trace:
java.lang.IndexOutOfBoundsException: Index: 0
Ex payload: "question mark content?"
1. #[org.mule.util.StringUtils.replace(payload,"?","\\?")]
Result: "question mark content\?"
2. #[org.mule.util.StringUtils.replace(payload,"?","\?")]
Result: not allowed to use the above expression
If i use the payload "question mark content" then it is inserted successfully.
Please help me how can I escape '?' in my JSON content while saving it into memsql?
'\?' itself is an escape sequence, so achive this you have to use "\\?" which produce "\?" which should work with memsql.
#[org.mule.util.StringUtils.replace(payload,"?","\\\\?")]
Hope this helps.
From the looks of your exception it looks like you are calling for it to replace the payload, but you're not assigning it to anything.
Going off of the documentation at:
http://grepcode.com/file/repo1.maven.org/maven2/commons-lang/commons-lang/2.4/org/apache/commons/lang/StringUtils.java#3457
It basically says that it's trying to replace the items in a string, and the method itself returns a string. Based on what I can tell in the stack trace, it seems as though you are passing a null or uninitialized variable to something that's trying to parse str[0], which is returning an array out of bounds error.
The way to correct this would be to do something like:
payload = org.mule.util.StringUtils.replace(payload,"?","\\\\?")
Which should replace any instance of ? with \? and re-write it to the payload variable. That said, it sounds like payload may actually be null when you're evaluating it later in your program, which could be indicative of a larger issue.
Related
consider a msg on kafka topic, event: { "x":1, "y":2, "c": "abc"}. I would like to convert the event object to event: "{\"x\": 1, \"y\":2, \"c\":\"abc\"}.
Looking to use a regex transformation to capture everything between the curly brace.
Apparently I get an error Invalid value "{.*?}" for configuration regex: Invalid regex: Illegal repetition near index 0 "{.*?}".
Not sure how to solve this.
This is the connector config so far:
transforms=escapeBrace
transforms.escapeBrace.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.escapeBrace.regex="\{.*?\}"
transforms.escapeBrace.replacement="\"$1\""
RegexRouter is only for modifying the topic name, not the data within any record. (Topic names cannot contain braces, slashes, or quotes.)
You'll want to try using Kafka Streams, or some other consumer to modify the data before sending to a new topic. Writing your own SMT is another option.
I have a server written in Rust, this server gets a request in JSON, the JSON the server is getting is a string and sometimes users write quotes inside the value. For example when making a new forum thread.
The only thing I really need to do is to escape the quotes inside the value.
So this:
"{"name":""test"", "username":"tomdrc1", "date_created":"07/12/2019", "category":"Developer", "content":"awdawdasdwd"}"
Needs to be turned into this:
"{"name":"\"test\"", "username":"tomdrc1", "date_created":"07/12/2019", "category":"Developer", "content":"awdawdasdwd"}"
I tried to replace:
let data = let data = "{"name":""test"", "username":"tomdrc1", "date_created":"07/12/2019", "category":"Developer", "content":"awdawdasdwd"}".to_string().replace("\"", "\\\"");
let res: serde_json::Value = serde_json::from_str(&data).unwrap();
But it results in the following error:
thread '' panicked at 'called Result::unwrap() on an Err value: Error("key must be a string", line: 1, column: 2)
I suspect because it transforms the string to the following:
let data = "{\"name\":\"\"test\"\", \"username\":\"tomdrc1\", \"date_created\":\"07/12/2019\", \"category\":\"Developer\", \"content\":\"awdawdasdwd\"}"
If I understand your question right, the issue is that you are receiving strings which should be JSON but are in fact malformed (perhaps generated by concatenating strings).
If you are unable to fix the source of those non-JSON strings the only solution I can think of involves a lot of heavy lifting with caveat:
Writing a custom "malformed-JSON" parser
Careful inspection/testing/analysis of how the broken client is broken
Using the brokenness information to fix the "malformed-JSON"
Using the fixed JSON to do normal request processing
I would recommend not to do that except for maybe a training excercise. Fixing the client will be done in minutes but implementing this perfectly on the server will take days or weeks. The next time this one problematic client has been changed you'll have to redo all the hard work.
The real answer:
Return "400 Bad Request" with some additional "malformed json" hint
Fix the client if you have access to it
Additional notes:
Avoid unwrapping in a server
Look for ways to propagate the Result::Err to caller and use it to trigger a "400 Bad Request" response
Check out error handling chapter in the Rust book for more
I am not able to extract quoted strings using JSON_EXTRACT function in MySQL 5.7.
Sample input:
{
"email": "d'souza#email.com",
"body": "I may have "random quotes '(single)/(double)" " in my source data"
}
Tried using,
SELECT
#valid_source := JSON_VALID(jsonString), NULL
IF(#valid_source, JSON_UNQUOTE(JSON_EXTRACT(jsonString, "$.email")), NULL)
I get an error stating: Invalid JSON text in argument 1 to function json_extract: "Missing a closing quotation mark in string." at position xxx
Any help will be appreciated, thanks!
Here is the fix that worked for me:
I used the operator "-->" instead of JSON_UNQUOTE(JSON_EXTRACT("jsonString")) and it did not throw me any error for any kind of quotes in my input string. Please note that the above Sample JSON was only one of the use cases that I was expecting in my input. I have around 4 million records, with all different combinations of characters and quotes since it contains the email bodies and using the operators instead of actual commands worked perfectly fine, it's weird since both are essentially the same but never the less I'm happy that I could resolve it using a small fix.
{
#valid_json := JSON_VALID(inputString),
IF(#valid_json, inputString ->> '$.email', NULL) AS EMAIL,
}
You are getting this error because the contents in the field, 'inputString' are invalid JSONs for some record. I think it's a mistake to blindly handle this error without persisting any error message for downstream / future users.
This approach may be acceptable (although I wouldn't accept this from a data engineer). A better solution would be to put a more informative error in the IF statement rather than a null, perhaps something like
CONCAT_WS(" ", 'INCORRECTLY FORMED JSON :', inputString), so that your overall function becomes
IF(valid_json(inputString), input_string ->> '$.email', CONCAT_WS(" ", 'INCORRECTLY FORMED JSON :', inputString)) as EMAIL.
That way any downstream users will be able to identify any underlying data quality issues that may be affecting the validity of conclusions. Along those lines, it may be worth having a completely separate field like
JSON_VALID(inputString) AS INPUT_JSON_VALID that you can use for easier downstream filtering.
I am trying to send the following JSON object and I get the error shown below,I checked #http://jsonviewer.stack.hu ,the JSON format seems to be correct,what am I missing and how to fix this?
{"component":{"name":"Company tech (New Bugs)", "version":"B"},"assignee":1234456,"milestone":"chiHW","priority":2,"state":"Analyze","substate":"Nominate","title":"[please ignore]CS\:4355C1\,4364B2\:WDI\:DHD\:HLK\(16299\)\-\>\"DF\ \-\ Sleep\ Tests\"\-\>Assert\-\>bcmpciedhd63\.sys\(dhd\_os\_ioctl\_resp\_wait\)\-\>dhd\_ndis\.c\#4449"}
Error:-
{"message":"An invalid JSON Object was passed in the request body. Please pass a valid JSON object.","help":"View documentation at http://bugs.company.com/","title":"Invalid Request","status":"400 Bad Request"}
You could have pasted your expression into https://jsonlint.com. It tells you where the problem is, and you can experiment until the JSON is no longer invalid. In your case, the problem is those backslashes in the last dictionary value (the one that starts "please ignore").
You could use jq to get a more specific parse error
watson:~$ cat >j
{"component":{"name":"Company tech (New Bugs)", "version":"B"},"assignee":1234456,"milestone":"chiHW","priority":2,"state":"Analyze","substate":"Nominate","title":"[please ignore]CS\:4355C1\,4364B2\:WDI\:DHD\:HLK\(16299\)\-\>\"DF\ \-\ Sleep\ Tests\"\-\>Assert\-\>bcmpciedhd63\.sys\(dhd\_os\_ioctl\_resp\_wait\)\-\>dhd\_ndis\.c\#4449"}
^D
watson:~$ jq <j
parse error: Invalid escape at line 1, column 333
I am getting text from MySQL via a laravel model and converting the content to JSON. Unfortunately the text contains new lines and carriage returns, is there any way to get the contents properly escaped?
Article::find(1)->toJSON();
Gives me an Ajax/JSON error in my view so I was wondering where the problem is. Am I either storing the content the wrong way or am I retrieving it the wrong way?
Thanks.
This is the JSON string I am getting for a test article:
{"id":22,"short_title":"Another test article","long_title":"Longer title for the test article mention in the short title","description":"This article describes a computer classroom where strange things happen and stuf","content":"This is a test article and I think this will generate the error I am looking for. <\/p>\r\n\r\n Maybe or maybe not. <\/p>\r\n","deleted_at":null,"created_at":"2014-04-25 09:10:45","updated_at":"2014-04-25 09:10:45","category_id":1,"zenra_link":"","published_on":null,"published":1,"source_url":"http:\/\/blog.livedoor.jp\/morisitakurumi\/archives\/51856280.html","source_title":"Some source article","source_date":null,"slug":"another-test-article","view_count":0}
The content portion is generated by a textarea that's running CKEditor and it gets saved to a MySQL MEDIUMTEXT field.
Next thing I do is that I inject this into my view to populate my backbone views like this:
<script>var initialData = JSON.parse('{{ $cJson }}');</script>
And that's where the console tells me: Uncaught SyntaxError: Unexpected Token. As suggested I tested the above string on jsonlint.com and it comes back as valid.
I figured out the problem:
<script>var initialData = JSON.parse('{{ $cJson }}');</script>
This is where the error happens. JSON.parse freaks out on the string I supply. If I remove JSON.parse it works like this:
<script>var initialData = {{ $cJson }};</script>
No more unexpected token errors. I was on the wrong track with the problem because I read on so many places that the carriage returns and new lines produce errors in the JSON format.