I am storing emojis in a MySQL database expressed in UTF8 Bytes, like "\xf0\x9f\x98\x80", which is the Unicode character U+1F600 GRINNING FACE
It is fine if I copy and paste it in and test it like this
print MAIL "Subject: \xf0\x9f\x98\x80\n";
It works and sends me the emoji.
But if I tell the script to get it from the database and plug it in like this:
print MAIL "Subject: $subject\n";
It will give me the subject: \xf0\x9f\x98\x80
What do I need to do? I thought if I was storing it in bytes it would see it as plain text and it would work.
It seems most likely that you have added the value to the database wrongly.
If you use Perl code and write the string '\xf0\x9f\x98\x80' to the database (note the single quotes) then you will get exactly the symptoms you describe. Your database will contain the sixteen-character ASCII string \xf0\x9f\x98\x80 and it will be displayed as such.
You shouldn't be involved with the UTF-8 encoded bytes; it is far better to specify the Unicode code point either by name or number
All of these produce the same Perl UTF-8-encoded string
$s = "\N{U+1F600}";
$s = "\N{GRINNING FACE}";
$s = "\x{1F600}";
The corresponding encoded bytes are irrelevant to the programmer, but if you must you can use the Encode module like this
use Encode 'decode_utf8';
$s = decode_utf8 "\xf0\x9f\x98\x80";
Another way is to enter the character directly into your code. You will need use utf8 to indicate to the compiler that the source contains non-ASCII UTF-8-encoded characters, like this
use utf8;
$s = "😀";
All of these assignments to $s will produce exactly the same result, and the values will compare as being equal using eq
On the database side you need a MySQL column with a four-byte UTF-8 character set, for instance
column VARCHAR(50) CHARACTER SET utf8mb4
Note that the character set must be utf8mb4 as if you use the earlier utf8 then you would be restricted to three-byte encoding, whereas emoji characters are all four bytes
Related
I have a perl script that reads from a web service and saves in a mysql table. this table uses latin1. from the web service there are coming some wrong characters and need to remove them before saving them in the database, otherwise they get saved as '?'
wanted to do something similar as:
$desc=~s///gsi;
but is not removing them.
the webservice that has the wrong characters is: https://jobvacancies.services.businesslink.gov.uk:8443/vacancy/26653478
using a user agent to get the data, seems coming in utf8 but the characters need to be removed:
my $ua = LWP::UserAgent->new ();
$ua->default_headers->push_header ('Accept' =>
"text/html,application/xhtml" .
"+xml,application/xml");
$ua->default_headers->push_header ('Accept-Charset' => "utf-8");
my $doc = $ua->get ("https://jobvacancies.services.businesslink.gov.uk:8443/vacancy/26653478")
If you just want to remove the characters outside the 7-bit ascii set (which are sufficient to display messages in english), you can you do this:
$desc=~s/[^\x00-\x7f]//g
Edit: If you want something more elaborate that supports the entire latin-1 set, you can do this:
use Encode;
$desc=encode('latin-1',$desc,sub {''});
This will remove exactly the characters that cannot be represented by latin-1. Note that this line expects that the utf-8 flag is on for the string $desc and that the resulting string will have the utf-8 flag is off.
Finally, if you want to preserve the euro sign (€), please note that you cannot do that with latin-1 because it is not part of that encoding. You will have to use a different encoding, such as ISO-8859-15.
The content sent by the web service is XML that contains HTML in the Description tag. If this is that content that worries you, another option than deleting non-Latin-1 character is to encode characters using HTML encoding:
$desc =~ s/([^\x00-\x7f])/sprintf("&%d;", ord $1)/ge
Here is an example:
$ echo 'é' | perl -C -pE 's/([^\x00-\x7f])/sprintf("&%d;", ord $1)/ge'
&233;
Change your column definition to CHARACTER SET utf8mb4 so that the naughty character does not need to be removed, and can actually be stored.
I have this string in Excel (I've UTF encoded It) when I save as CSV and import to MySql I get only the below, I know it's probably a charset issue but could you explain why as I'm having difficulty understanding it.
In Excel Cell:
PARTY HARD PAYDAY SPECIAL â UPTO £40 OFF EVENT PACKAGES INCLUDING HOTTEST EVENTS! MUST END SUNDAY! http://bit.ly/1Gzrw9H
Ends up in DB:
PARTY HARD PAYDAY SPECIAL
The field is structured to be utf8_general_ci encoded and VARCHAR(10000)
Mysql does not support full unicode utf8. There are some 4 byte characters that cannot be processed and, I guess, stored properly in regular utf8. I am assuming that upon import it is truncating the value after SPECIAL since mysql does not know how to process or store the character in the string that comes after that.
In order to handle full utf8 with 4 byte characters you will have to switch over to the utf8mb4.
This is from the mysql documentation:
The character set named utf8 uses a maximum of three bytes per character and contains only BMP characters. The utf8mb4 character set uses a maximum of four bytes per character supports supplemental characters...
You can read more here #dev.mysql
Also, Here is a great detailed explanation on reg-utf8 issues in mysql and how to switch to utf8mb4.
Since JSON 2.xx i need to set the latin1 flag in order to get umlauts safe to the html document:
my $obj_with_umlauts = {
title => 'geändert',
}
my $json = JSON->new()->latin1(1)->encode($obj_with_umlauts);
This was not necessary using JSON 1.xx :
my $json = JSON->new()->objToJson($obj_with_umlauts);
The html document is in iso-8559-1 (meta-tag).
Can anybody explain to me why?
This is such a huge can of worms that you're opening here.
I suspect that the answer is something along the lines of "a bug was fixed in the character handling of JSON.pm". But it's hard to know what is going on without a lot more information about your situation.
How is $string_with_umlauts being set? How are you encoding the data that you write to the HTML document?
Do you want to handle utf8 data correctly (you really should) or are you happy assuming that you live in a Latin1 world?
It's important to realise that if you completely ignore Unicode considerations then it can often seem that your programs are working correctly as errors often cancel each other out. When you start to address Unicode issues, it can seem that your programs are getting worse until you address all of the issues.
The Perl Unicode Tutorial is a good place to start learning about these things.
P.S. It's "Perl", not "PERL".
What are you talking about?
$ perl -MJSON -E'
say $JSON::VERSION;
my $json = JSON->new()->objToJson(["\xE4"]);
say sprintf "%v02X", $json;
'
1.15
5B.22.E4.22.5D # Unicode code points for ["ä"]
$ perl -MJSON -E'
say $JSON::VERSION;
my $json = JSON->new()->encode(["\xE4"]);
say sprintf "%v02X", $json;
'
2.59
5B.22.E4.22.5D # Unicode code points for ["ä"]
Those two strings are identical! In fact, adding ->latin1() doesn't change anything because the iso-8859-1 encoding of Unicode code point U+00E4 is E4.
$ perl -MJSON -E'
say $JSON::VERSION;
my $json = JSON->new()->latin1()->encode(["\xE4"]);
say sprintf "%v02X", $json;
'
2.59
5B.22.E4.22.5D # iso-8859-1 encoding of ["ä"]
There is one difference between the last two: it's stored differently in the scalar. That should make absolutely no difference. If code treats them differently, then that code is incorrectly reading the data in the scalar, and that code is buggy.
$string_with_umlauts definetly is a string in winLatin
Well, that's error number one.
JSON expects strings of decoded text (strings of Unicode code points), not encoded text.
That said, there happens to be no difference between a string encoded using iso-8859-1 and a string of Unicode code points. For example, when encoded using iso-8859-1, "ä" is byte E4, and it's Unicode code point U+00E4, two different notation for the same number.
If the string is encoded using cp1252, though, you'll have problems with characters €‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ (the characters in cp1252 but not in iso-8859-1). For example, when encoded using cp1252, "€" is byte 80, but it's Unicode code point U+20AC. 0x80 != 0x20AC.
The html document is in iso-8559-1 (meta-tag).
Then at some point, you'll have to encode the output into iso-8859-1. You can do it using an :encoding layer, or using Encode's encode or using JSON's ->latin1 directive. The advantage of using this final option is that it will cause JSON to escape any character outside of the iso-8859-1 character set before attempting to encode it.
Can anybody explain to me why?
You have a code (an XS module) that reads the underlying string buffer of the scalar and incorrectly treats that as the content of the string. There is a bug is in that module.
I have a MySQL 'articles' table and I am trying to make the following insert using SQLyog.
insert into articles (id,title) values (2356606,'Jérôme_Lejeune');
This works fine and the data shows fine when I do a select query.
The problem is that when I do the same insert query through my perl script, the name shows up with some junk characters in place of é and ô in the database. I need to know how to properly store the name through my script. The part of code that does the insert is like this.
$sql_insert = "insert into articles (id,title) values (?,?)";
$sth_insert = $dbh->prepare($sql_insert);
$sth_insert->execute($id,$title);
$id and $title have the correct required data which I have checked by print before I am inserting them. Please assist.
You have opened up the character encoding can of worms, and you have a lot to learn before you will solve this problem and have it stay solved.
You are probably already used to thinking of how a character of text can be encoded as a string of bits. Under the ASCII encoding, for example, the 8-bit string 01000001 (65) is used to indicate the A character. When you start to think about how many different languages there are and how many different kinds of characters there are, you quickly realize that an 8-bit encoding is not going to get you very far. So a number of other character encodings have proliferated. Some of the most popular are latin1 (ISO-8859-1) and UTF-8. Both of these encodings can render the é and ô characters, but they use quite different bit strings to represent them. As you write to a file (or to the terminal) or add a row to a database, Perl and MySQL have a notion of what the character encoding of the output stream is. An encoding is also used when you read data. If you don't know what this encoding is, then it doesn't make any sense to say that the data looks good/looks bad when you store it and retrieve it.
Perl and MySQL can, with the right settings, handle both of these encodings and several more. Which encoding you choose to use is not as important as making sure that all the pieces of your application are using the same encoding. But you should choose an encoding that
can encode all of the characters you will need (for this problem, you mention é and ô, but will there be others? what about in the future?)
is supported by all the pieces of your application (front-end, database, back-end)
Here's some suggested reading to get you headed in the right direction:
The Encode module for Perl
character sets in MySQL
(others should feel free to recommend additional links)
I can't speak to MySQL so much, but character encoding support in Perl is rapidly evolving (which isn't to say that it ain't damn good). The latest versions of Perl will have the best support (for the most obscure character sets) and the best features (for example, regular expressions and character classes) for characters beyond ASCII.
There are few things to follow.
First you have to make sure, that Perl understands that data which is moving between your program and DB is encoded as UTF-8 (i expect your databases and tables are set properly). For this you need to say it loud out on connecting to database, like this:
my($dbh) = DBI->connect(
'dbi:mysql:test',
'user',
'password',
{
mysql_enable_utf8 => 1,
}
);
Next, you need send data to output and you must set it to decaode data as UTF-8. For this i like pretty good module:
use utf8::all;
But this module is not in core, so you may want to set it with binmode yourself too:
binmode STDIN, ":utf8";
binmode STDOUT, ":utf8";
And if you deal with webpages, you have to make sure, that browser understoods that you are sending your data encoded as UTF-8. For that you should make sure your HTTP-headers include encoding:
Content-Type: text/html; charset=utf-8;
and set it with HTML META-tag too:
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
Now you should get your road covered.
I've got a problem with the following string:
$str="this is \321\213\321\213\321\213\321\213\321\213 \321\201\320\277\320\260\321\200\321\202\320\260\321\200";
This string is located in an ascii text file and I want to store in a Mysql db (utf8). \321\231 ... are cyrillic symbols.
What can I do to make \321\213 look like cyrillic characters in Mysql db
This should be described in RFC2047, end look like it was utf7 to utf8 conversion.. dont know excatly.
its "unicode escape"
working variant:
use Encode::Escape;
$var1='\321\213';
print decode 'unicode-escape', $var1;
#correct mysql view in phpmyadmin
$dbh = DBI->connect('DBI:mysql:database=test', 'testuser', 'testpass', { mysql_enable_utf8 => 1});
This is not quoted-printable at all. This is Perl quoted string representation, also know as PERLQQ, of a series of octets. The numbers are octal.
These bytes encode UTF-8 for the most part, but the data contain two errors. Looks like one half of a character each somehow fell off. I have marked it with arrows just below.
my $octets = "this is \321\213\321\213\321\213\321\213\321 \321\201\320\277\320\260\321\200\321\202\320\260\321";
# ↑↑↑↑ ↑↑↑↑
This invalid in UTF-8, but can be repaired. We put the Unicode replacement character.
use Encode qw(decode);
my $characters = decode 'UTF-8', $octets, Encode::FB_DEFAULT | Encode::LEAVE_SRC;
# this is ыыыы� спарта�
This character string can now be simply inserted into the database as usual. The DSN in the connect call for DBI or DBIx::Class must include the attribute mysql_enable_utf8.
connect('DBI:mysql:foobar;mysql_enable_utf8=1', …, …);
You need to convert explicitly the codes to characters. For that you need to know what's the input encoding. I suppose it's iso-8859-5, but it could be windows-1252 or something else.
use Encode qw( decode );
my $str="this is \321\213\321\213\321\213\321\213\321 \321\201\320\277\320\260\321\200\321\202\320\260\321";
my $out .= from_to( "iso-8859-5","utf-8", $str );
I've just seen that your source string is indeed QP, so you need to convert from QP to bytes; that's easy, simply use MIME::QuotedPrint:
use MIME::QuotedPrint ();
my $out = MIME::QuotedPrint::decode($str);
Problem is: perl does not know that the string is UTF-8, so you must turn flag explicitly on.
Encode::_utf8_on($str);