I inserting data from mysql to postgresql using Talend and I get invalid byte sequence error.
I used trim(both CHAR(0x00) from column_name) to remove invalid characters.
It removed most of the errors but I am guessing there are invalid characters in the middle of string. How can I find and remove them using sql?
Related
We keep some JSON's in our database as NVARCHAR. Before inserting, we validate JSON with "IsJSON" function. The problem is we have a JSON with Accent character in the keys like "Malārd". IsJson returns 1 which means it is a valid JSON but when we try to use it in a path such as '$.Malārd' it will cause below error:
-- QUERY
SELECT JSON_VALUE(N'{"Malārd":1000}', N'$.Malārd')
-- OUTPUT
-- JSON path is not properly formatted. Unexpected character 'ā' is found at position 5
If I have to, I will prevent inserting accent characters in keys, but I hope to find a better solution for that.
I know that I can replace accent character and use trick like COLLATE LATIN1_GENERAL_CS_AI but we have some infrastructure tools and API's in our project which allowed client developer insert and grabs a variety of JSON models.
You need to surround path expression with quotes if the key name starts with a dollar sign or contains special characters:
SELECT JSON_VALUE(N'{"Malārd":1000}', N'$."Malārd"')
i have a json field in my table(MySQL Database).
the following is the structure:
{
"article":{
"Key's 1":{
"value":"24"
}
"Key's of the something's 2":{
"value":"55"
}
}
}
i am trying to extract the "value" field of "Key's 1".
Due special character such as single quote and space, i am unable use JSON_EXTRACT function.
It give the error: Invalid JSON path expression.This error is around character position no : 10
My Query:
select
JSON_EXTRACT(analytics_json,'$.article.Key\'s 1.value')As value
from
tbl_json_data;
Even after place a backslash, i am getting the error.
You may escape each JSON path component in double quotes to handle special characters as well as spaces. The following works:
SELECT
JSON_EXTRACT(analytics_json,'$.article."Key''s 1".value') AS value
FROM tbl_json_data;
Demo
Note that your key name actually has two problems. First, it contains a literal single quote. We can handle that by just doubling up two single quotes. The key name also contains whitespace. By escaping in double quotes we may workaround this problem, but it would probably be best to avoid using JSON keys which have whitespace.
I'm encrypting some user data (no passwords) using Crypto.Cipher AES.
The return is a AESCipher of the form:
b'o\xab\xdd\x19\xaat\xfcIAN\xd2\x00\xe9'
sometimes it produces spaces and non hex representations.
b"N%?\x91\xe8'J\xc0\x10 p"
b'QV8>K\xd8\xfa\x9a\x05%\xe8LJp\xd0gf'
When I try to insert these values into the SQL table created as:
data_encrypted VARBINARY(40)
I get the following warning:
Warning: (1300, "Invalid utf8 character string: 'ABDD19'")
Seems like it's trimming the binary array, when I Query the inserted rows in the table, the row was effectively inserted but only the first bytes of the array, from b'o\xab\xdd\x19\xaat\xfcIAN\xd2\x00\xe9' it inserts only the 'o'.
Do I have to specify something else in the format?
Thanks
Adding a _binary solved the warning
self.cur.execute("INSERT IGNORE INTO members VALUES(%s,_binary %s...
And I'm also formatting the array to be hex represented, it looks better in the MySQL table and the decryption still works ok binascii.b2a_hex
I am having an issue on inserting this string
input: GyYu4$%acV
posted string: GyYu4$�V
Query error: Incorrect string value: '\xACV' for column.
It seems the best I get is str_replace.
I got the best escape for the issue its about escaping the non-utf8 character that #developerCK mentioned. its utf8_encode
I tried running this query:
SELECT column FROM table WHERE column REGEXP '[^A-Za-z\-\']'
but this returns
#1139 - Got error 'invalid character range' from regexp
which seems to me like the - in the character class is not being escaped, and instead read as an invalid range. Is there some other way that it's suppose to be escaped for mysql to be the literal -?
This regex works as expected outside of mysql, https://regex101.com/r/wE8vY5/1.
I came up with an alternative to that regex which is
SELECT column FROM table WHERE column NOT REGEXP '([:alpha:]|-|\')'
so the question isn't how do I get this to work. The question is why doesn't the first regex work?
Here's a SQL fiddle of the issue, http://sqlfiddle.com/#!9/f8a006/1.
Also, there is no language being used here, query is being run at DB level.
Regex in PHP: http://sandbox.onlinephpfunctions.com/code/10f5fe2939bdbbbebcc986c171a97c0d63d06e55
Regex in JS: https://jsfiddle.net/6ay4zmrb/
Just change the order.
SELECT column FROM table WHERE column REGEXP '[^-A-Za-z\']'
#Avinash Raj is correct the - must be first (or last). The \ is not an escape character in POSIX, which is what mysql uses, https://dev.mysql.com/doc/refman/5.1/en/regexp.html.
One key syntactic difference is that the backslash is NOT a metacharacter in a POSIX bracket expression.
-http://www.regular-expressions.info/posixbrackets.html
What special characters must be escaped in regular expressions?
Inside character classes, the backslash is a literal character in POSIX regular expressions. You cannot use it to escape anything. You have to use "clever placement" if you want to include character class metacharacters as literals. Put the ^ anywhere except at the start, the ] at the start, and the - at the start or the end of the character class to match these literally