I'm trying to insert an emoji to my MySQL database with a querystring and I don´t know what Im doing wrong?
I have a MySQL database that is set to utf8mb4 and I can insert and emoji to a table with the Mysqlworkbench and it shows as a real emoji. So the database is working as it should. And if I just try to show that on a .asp page then I shows an emoji as it should.
I have set the database, table, and column to be utf8mb4_general_ci, is this right?
And my db connection (koppling.asp):
strConn = "driver={MySQL ODBC 5.2 Unicode Driver};server=localhost;uid=xxx;pwd=xxx;database=emojtest;option=3;charset=utf8mb4;"
set conn = Server.CreateObject("ADODB.Connection")
conn.Open strConn
To my insertemoji.asp page I'm sending the querystring/variable like this.
www.somepage.se/emojitest/insertemoji.asp?text=😜😀😊😃
And my insertemoji.asp looks like this.
<!--#include file="koppling.asp" -->
<%
Response.charset="utf-8"
thetext = request.querystring("text")
sql= "INSERT INTO news (text) VALUES ('"&thetext&"');"
conn.Execute (sql)
response.write thetext (this is showing 😜😀😊😃)
%>
But the emoji is not stored as an emoji, only like this 😜😀😊😃 and if it was encoded right it would be something like this I think \xF0\x9F\x98\x82
So what am I missing? Where do I go wrong?
If it should be stored like \xF0\x9F\x98\x82, how do I then convert/encode it to a real emoji when I display it on my .asp page?
How is an emoji suppose to be stored in the DB?
Should it be like a visible real emoji?
Or like this 😜😀😊😃
Or like this \xF0\x9F\x98\x82
Solved
I finally figured out the problem, I used the MySQL ODBC 5.2 Unicode Driver and when I changed to MySQL ODBC 3.51 Driver it worked!
😜😀😊😃, when Mojibaked, becomes 😜😀😊😃. This because of confusion over the hex F09F989CF09F9880F09F988AF09F9883, which is treated as Emoji in utf8mb4, but that gibberish by latin1.
To see what probably went wrong in the code, see Trouble with UTF-8 characters; what I see is not what I stored
Related
I've read many threads on Stack and other forums, but still can't figure how to make it work.
The configuration
I have a MySQL 8.0 database named test, and configured it with default CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_as_ci.
Inside there's a table creatively named table, this too configured as utf8mb4 and utf8mb4_0900_as_ci.
The only field is named field and is a blob type, where I write and read encoded data using AES_ENCRYPT/DECRYPT
VBScript classic connect to it using the last ODBC 8 UNICODE driver
I've tried the SET NAMES statement but is not supported in this ODBC version. Instead the charset is included directly in the connection string.
Following other threads, I made a test file
without any external include
saved as UTF8
specified all the possible Vbscript Codepage and Charset at the beginning of the file
the connection string specifies a charset, I've tried both ucs2 and utf8mb4
the html segment specifies the utf charset both in the Content Type and in the form
when reading from the database I specify to CONVERT USING utf8mb4
The page is setup to be the simplest test possible, with a form on top, and the results listed below.
All the simple text is processed correctly, while any emoji is read as ?
For example, if this is the text passed through the form -> hello 😀 😃 😄 😁 😆 😅 😂
this is what is returned -> hello ? ? ? ? ? ? ?
Executing the same SELECT query in Workbench 8, shows the correct text with the emoji, so it's not a problem directly related to the Codepage or Charset, or when it writes the data, but only when it read them.
VarType report the RS("Field") as a simple string / 8.
I've spent the last days studying and testing all the possible solutions, but can't solve it :/
<%#Language="VBScript" CodePage="65001"%>
<% Option Explicit %>
<%
' THE PAGE ENCODING
Response.ContentType = "text/html;charset=UTF-8"
Session.CodePage = 65001
Response.CodePage = 65001
Response.CharSet = "UTF-8"
'----------------------------------------------------------------
dim dbConn, sql, RS
' SIMPLE CONNECTION STRING
' NOTE THAT I'VE TRIED BOTH charset=ucs2 AND charset=utf8mb4
Set dbConn = Server.CreateObject ("ADODB.Connection")
dbConn.Open "DRIVER={MySQL ODBC 8.0 UNICODE Driver}; SERVER=127.0.0.1; PORT=3306; DATABASE=test; Uid=user; Pwd=password; charset=ucs2;"
'----------------------------------------------------------------
%>
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>test MySQL UTF-8</title>
</head>
<body>
<form action="?action=insert" method="post" accept-charset="utf-8">
<textarea name="text"></textarea>
<input type="submit" value="Insert">
</form>
<%
'----------------------------------------------------------------
' INSERT THE STRING WHEN THE FORM IS SUBMITTED
If Request.Querystring("action") = "insert" Then
sql = "INSERT INTO table (Field) VALUES(AES_ENCRYPT('" & Request.Form("text") & "', 'AES_Key'))"
Response.Write(sql & "</br>")
dbConn.execute(sql)
End If
'----------------------------------------------------------------
' LIST ALL THE RECORDS FROM THE TABLE
sql = "SELECT CONVERT(AES_DECRYPT(Field, 'AES_Key') USING utf8mb4) AS Field FROM table"
Set RS = dbconn.execute(sql)
Do Until RS.EOF
Response.Write(RS("Field") & "</br>")
RS.MoveNext
Loop
RS.close
dbConn.close
%>
</body>
</html>
Good Morning!
I've been trying to implement emojis in my applications for a while, but I'm having a lot of difficulties. I have already tested several internet solutions, but none has been effective.
I will try to detail as much as I am doing:
I'm using Classic ASP and MySQL in versions
5.6.40-84.0-log and 5.6.26-log.
The application is hosted on a Plesk Windows (I tried to run locally
and the same goes for the database). Notepad ++ pages have already
been tested in UTF-8 and UTF8 without BOM.
In the HTML <head> there is <meta charset = "utf-8">.
The <form> has the tag accept-charset="UTF-8".
The ASP has Response.AddHeader "Content-Type", "text/html;charset=UTF-8", Response.CodePage=65001,Response.LCID=1060 and Response.Charset="utf-8".
Now for the problems:
In both versions of MySQL, when I change COLLATE to utf8mb4_unicode_* or utf8mb4_bin, it returns to utf8mb4_0900_ai_ci automatically.
In version 5.6.40-84.0-log, if I register the emojis directly in the database, they are like ??????????????.
In version 5.6.26-log, if I register directly with the database, the error returns:
Executing:
INSERT INTO `db`.`table` (` emoji`) VALUES ('tion 123 😀😉😙😐🤐 ão');
Operation failed: There was an error while applying the SQL script to the database.
ERROR 1366: 1366: Incorrect string value: '\ xF0 \ x9F \ x98 \ x80 \ xF0 \ x9F ...' for column 'emoji' at row 1
SQL Statement:
INSERT INTO 'db'. 'Table' ('emoji') VALUES ('tion 123 😀😉😙😐🤐 ão')
In this same version, registering through the of the page, using SET NAMES 'utf8mb4' or SET NAMES' utf8' in the ODBC connection string, the following error is returned:
Microsoft OLE DB Provider for ODBC Drivers error '80004005'
[MySQL] [ODBC 3.51 Driver] [mysqld-5.6.22] Incorrect string value: '\ xE7 \ xE3o' for column 'field' at row 1
EDIT 1:
When I consult SHOW SESSION VARIABLES LIKE 'character_set%'; and SHOW SESSION VARIABLES LIKE "%collation%"; the database it returns the following results:
Thanks!
Translation by Google Translate (rsrs)
For Emoji and some Chinese, you need utf8mb4, not utf8 in MySQL.
Since you are running the old 5.6, you may stumble over another problem. http://mysql.rjweb.org/doc.php/limits#767_limit_in_innodb_indexes
If that E7E3 should have been çã, then you have some latin1-encoded text, too.
Do not mix encodings in a single column.
Your output for SHOW SESSION VARIABLES LIKE 'char%' shows that the connection is just utf8; it needs to be utf8mb4. The main difference between them is in Emoji.
I don't think title could be more described better as tl;dr, because problem is a bit deeper.
I've got two databases (finnish language):
MSSQL (collation: SQL_Latin1_General_CP437_CI_AI)
MySQL (collation: utf_general_ci)
I've created BI project in vs2017, connected two databases and transfered tables from one to another, no problem. Except for 1 letter: "å" - instead it was "?". I cannot change any database collation so I am trying to find a way to transfer words with this letter.
What I've tried:
OLD DB Source -> ODBC Destination
Point "1" with "Data Conversion" block in between (with code page 1252)
Script Component, in which I have tried:
Insert with "_latin"
sql= "INSERT INTO db.words(Name) VALUES(_latin1'å')";
byte[] b = Encoding.UTF8.GetBytes(sql);
odbcCmd = new OdbcCommand(Encoding.UTF8.GetString(b), odbcConn);
odbcCmd.ExecuteNonQuery();
Insert without it
sql= "INSERT INTO db.words(Name) VALUES('å')";
byte[] b = Encoding.UTF8.GetBytes(sql);
odbcCmd = new OdbcCommand(Encoding.UTF8.GetString(b), odbcConn);
odbcCmd.ExecuteNonQuery();
Diferent ways of encoding
byte[] bytes = Encoding.GetEncoding(1252).GetBytes("å");
var myString = Encoding.GetEncoding(1252).GetString(bytes);
byte[] bytes2 = Encoding.Default.GetBytes("å");
var myString2 = Encoding.Default.GetString(bytes2);
Insert with COLLATE which got me error
insert into db.words(Name) values ("å" COLLATE latin1_swedish_ci) ;
and error:
System.Data.Odbc.OdbcException: „ERROR [HY000] [MySQL][ODBC 5.3(a) Driver][mysqld-5.7.21-log]COLLATION 'latin1_swedish_ci' is not valid for CHARACTER SET 'cp1250'”
Here is interesting part:
I can make insert with this letter in MySQL Workbench without a problem, and it will be inserted, but when I try to pass it from one database to another it is lost. I've set Data Viewers between Data Conversion and the letter was still there, and also when debugging script it was after encoding in string that were inserted to database.
Maybe someone got any idea what else I can try, because I feel like I have tried everything, and feel that the resolve of this problem is really close, but I just don't see it.
CP1250 does not include å; CP437 and utf8 do include it.
COLLATE is irrelevant -- it applies only to comparing and sorting.
Don't use any encode/conversion functions; instead, specify how the data is encoded.
I see 'code' -- but what is the encoding for the source in that language and/or editor?
Show us the hex of any strings in question.
Which direction are you trying to transfer?
What are the connection parameters for each database?
My old website in ASP started generating this error:
Microsoft Cursor Engine error '80040e21'
Multiple-step operation generated errors. Check each status value.
I haven't changed recently anything - perhaps my hosting provider made a change but the error is pointing to this line of code:
rs.Open SQL, adoCon
I checked: rs, adoCon and SQL are all set and it used to work for years before.
Set rs = Server.CreateObject("ADODB.Recordset")
adoCon.Open "DRIVER={MySQL ODBC 3.51 Driver}; SERVER=<SERVER_NAME>; PORT=<PORT_NUMBER>;" &_
"DATABASE=<DBNAME>; USER=<USERNAME>; PASSWORD=<PASS>; OPTION=3;"
SQL is correct - I directly injected it into mySQL web interface and it generated needed results.
When I handle this error in ASP the Error.Description prints
"Object required"
What is it complaining about?
The answer to my question is that I had to convert decimal field to string. The fix looked like this:
SELECT Convert(Price,char) as Price FROM Table;
This is very strange but perhaps something was changed on my hosting provider site that caused this issue. I actually found similar solution here (strange mysql results in asp) which helped me to discover the issue. As I quote from link above "ASP can't recognize the results from MySQL and thinks it's EOF" without conversion from decimal to string.
For me it happened years ago With Oracle's Date/Time field having corrupt values causing this exact error when trying to select them.
Guess it's the same issue behind the scenes, what worked for me was manually deleting the corrupt rows using Oracle tools.
Another tip that has small chance to work is having such code instead:
adoCon.Open "DRIVER={MySQL ODBC 3.51 Driver}; SERVER=<SERVER_NAME>; PORT=<PORT_NUMBER>;" &_
"DATABASE=<DBNAME>; USER=<USERNAME>; PASSWORD=<PASS>; OPTION=3;"
Set rs = adoCon.Execute(SQL)
Hopefully the cursor used in this method won't crash.
http://www.sqlfiddle.com/#!2/82f65/1
I tried this:
create table x(y varchar(100) character set utf8);
insert into x(y) values('爱');
But the chinese character doesn't appear:
select y from x;
Output:
Y
?
I'm the author of sqlfiddle.com. The problem was that I didn't have my connection string and default database encoding for mysql setup to properly handle UTF8. I have fixed this now, but because the fiddle you posted is still using the obsolete settings, you'll have to see it working here on my slightly-modified version of your fiddle:
http://www.sqlfiddle.com/#!2/e79e8/1
Your link might start working eventually, it just needs to clear out of the running memory and be reset. After no one hits it for a while it should be harvested and then ready to be built back up cleanly. Thanks!
FYI, the changes I had to make to get it to work were found here: http://www.compoundtheory.com/?action=displayPost&ID=421
The relavent bits where adding this to my connection string from java:
useUnicode=true&characterEncoding=UTF-8
And adding this to my create database statement:
create database my_new_database default CHARACTER SET = utf8 default COLLATE = utf8_general_ci;
It is working fine in mysql on my localhost. it may be due to mysql charset or some setting please check it.
If you have to run this query via program like php then
run query before select query
"SET NAMES utf8"
It will be return result properly
thanks
The Chinese character is not displaying in fiddle but in actual mysql database it is working fine. Kindly check your mysql version