I have a table, in joomla I'm getting the following data:
1. fc k��ln - vfl wolfsburg
germany 1. bundesliga
so I created a test page, I set up the header as utf-8, and everything was fine, I received the wanted data:
Array
(
[0] => Array
(
[id] => 4e36e64eb34d2
[team1] => 1. FC Köln
[team2] => VFL Wolfsburg
[league] => Germany 1. Bundesliga
[sport] => Soccer
[time] => 2011-08-06 15:30:00
)
)
I also check the joomla header and it's contains the charset:
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
What am I doing wrong?
your database connection has to be set UTF8
set the mysql connection charset in includes/database.php file at about line 102 (second line below)
$this->_table_prefix = $table_prefix;
//#mysql_query("SET NAMES 'utf8'", $this->_resource); // THIS IS THE LINE TO UNCOMMENT
$this->_ticker = 0;
$this->_og = array();
Ofcourse your data in table has to be utf-8, check this.
Other possibible thing to check is if your browser sees this page as utf8 (in firefox: View->character encoding)
Maybe your files are in wrong encoding? I was having similar problem whan my files were in CP-1251, all data were in UTF-8 and I needed to show danish symbols. Changing file-encoding has solved this problem. Try, maybe it will be helpfull for you too.
Related
I am trying to fetch UTF-8 accentuated characters "é" "ê" from mysql and convert them to UCS-2 when sending over SMPP. The data is stored as utf8_general_ci and I perform the following when opening the DB connection:
$dbh->{'mysql_enable_utf8'}=1;
$dbh->do("set NAMES 'utf8'");
If I test the sending part by hard coding the string value with "é" "ê" using data_encoding=8, it goes through perfectly. However if I comment out the first line and just use what comes from the DB, it fails. Also, if I try to send the characters using the DB and setting data_encoding=3, it also works fine, but then the "ê" would not appear, which is also expected. Here is what I use:
$fred = 'éêcole'; <-- If I comment out this line, the SMPP call fails
$fred = decode('utf-8', $fred);
$fred = encode('UCS-2', $fred);
$resp_pdu = $short_smpp->submit_sm(
source_addr_ton => 0x00,
source_addr_npi => 0x01,
source_addr => $didnb,
dest_addr_ton => 0x01,
dest_addr_npi => 0x01,
destination_addr => $number,
data_coding => 0x08,
short_message => $fred
) or do {
Log("ERROR: submit_sm indicated error: " . $resp_pdu->explain_status());
$success = 0;
};
The different values for the data_coding fields are the following:
Meaning of "data_coding" field in SMPP
00000000 (0) - usually GSM7
00000011 (3) for standard ISO-8859-1
00001000 (8) for the universal character set -- de facto UTF-16
The SMPP provider's documentation also mentions that special characters should be handled via UCS-2:
https://community.sinch.com/t5/SMS-365-enterprise-service/Handling-Special-Characters/ta-p/1137
How should I prepare the data that is coming out of the DB to make this SMPP call work?
I am using Perl v5.10.1
Thanks !
$dbh->{'mysql_enable_utf8'} = 1; is used to decode the values returned from the database, causing queries to return decoded text (strings of Unicode Code Points). It makes no sense to decode such a string. Go straight to the encode.
my $s_ucp = "\xE9\xEA\x63\x6F\x6C\x65"; # éêcole
# -or-
use utf8; # Script is encoded using UTF-8.
my $s_ucp = "éêcole";
printf "%vX\n", $s_ucp; # E9.EA.63.6F.6C.65
my $s_ucs2be = encode('UCS-2', $s_ucp);
printf "%vX\n", $s_ucs2be; # 0.E9.0.EA.0.63.0.6F.0.6C.0.65
SET NAMES says the encoding you have/want in the client. That is, regardless of the encoding in the table, MySQL will convert it to whatever SET NAMES says during a SELECT.
So, feed what comes from the SELECT directly to SMPP. (It won't be readable by most other clients.)
SET NAMES ucs2
(The collation is irrelevant to the encoding.)
You could ask the SELECT to convert with something like
CONVERT(col_name, CHAR UNICODE)
https://dev.mysql.com/doc/refman/8.0/en/cast-functions.html
Seems like I opened another chapter of the encoding from hell book. I seek help with a problem I encounter when pulling and writing data from\to a MySQL data base with R. After a good amount of time I was able to write my data back but still don't understand what exactly is going on.
library(RMySQL)
library(dbplyr)
con <- dbConnect(MySQL(),
host = "localhost",
user = "root",
dbname="test",
password = rstudioapi::askForPassword("Database password"))
address <- as_tibble(tbl(con, "address"))
The pulled address dataframe looks like
address <- structure(list(address_id = c(1809463, 2213341, 2614879, 4536353
), street = c("5, RUE DU GRAND CORMORAN APPT. C15", "14, PLACE EGLISE",
"1058 TENNESSEE", "38 ALLEE GERARD DE NERVAL"), city = c("31240 L A°NION",
"85140 L AÂIE", "ELK GROVE VILLAGE A¨LLINOIS 60007", "F-69360 SAINT-
SYPHORIEN D AÂZON"
)), .Names = c("address_id", "street", "city"), row.names = c(NA,
-4L), class = c("tbl_df", "tbl", "data.frame"))
You can see right away that there is some encoding issues in address$city so I run
address$city <- iconv(address$city, from = "UTF-8", "windows-1252")
which seems to fix it as everything looks fine now but as soon as I want to write the file back to the MySQL I run into problems with the encoding again getting following error
dbWriteTable(con, value =address, name = "address_cleaned", overwrite=TRUE ,rownames = FALSE )
Error in .local(conn, statement, ...) :
could not run statement: Invalid utf8 character string: '31240 L A'
What I do now fixes the problem but I don't really understand what is going on.
Encoding(address$city) <- 'UTF-8'
address$city <- iconv(address$city, from = "windows-1252","UTF-8")
address$city <- iconv(address$city, from = "latin1","UTF-8")
While this code works it seems more like a work around than a real solution. I'm sure it has to do with the encoding of the MySQL data as well as Windows as my OS but I wonder if there is a more elegant solution to this.
Additional info
dbGetQuery(con, "SHOW VARIABLES LIKE 'character_set_%';")
Variable_name Value
1 character_set_client utf8
2 character_set_connection utf8
3 character_set_database utf8
4 character_set_filesystem binary
5 character_set_results utf8
6 character_set_server utf8
7 character_set_system utf8
8 character_sets_dir C:\\Program Files\\MySQL\\MySQL Server 5.7\\share\\charsets\\
and
Sys.getlocale()
[1] "LC_COLLATE=English_United States.1252;LC_CTYPE=English_United States.1252;LC_MONETARY=English_United States.1252;LC_NUMERIC=C;LC_TIME=English_United States.1252"
Edit 1. hex
1809463 31240 L A°NION 3331323430204C2041C2B04E494F4E
2213341 85140 L AIE 3835313430204C2041C2904945
2614879 ELK GROVE VILLAGE A¨LLINOIS 60007 454C4B2047524F56452056494C4C4147452041C2A84C4C494E4F4953203630303037
4536353 F-69360 SAINT-SYPHORIEN D AZON 462D3639333630205341494E542D535950484F5249454E20442041C2905A4F4E
Do not use any conversion functions, it will probably make things worse.
¨ is Mojibake for ¨ and ° for °. Since I see A before each of those, I guess you are trying to enter an accented A by first typing the A, then the accent. However, your data entry tool is failing to combine those. What editor are you using?
(Yes, you have 'opened another chapter of the encoding from hell book' -- I have seen a lot of character set problems, but not this one until now.)
I am getting the Exception when I attempt to update the record with "tableGateway" object:
Zend\Db\Adapter\Exception\InvalidQueryException
Statement could not be executed
(HY000 - 1300 - Invalid utf8 character string: 'C`\xC3`\xB3`digo')
I have the following table structure with data in mySQL:
CREATE TABLE `clientes` (
`Código` int,
`Nome` varchar(50),
`Descricao` varchar(150)
....
);
INSERT INTO `clientes` (`Código`, `Nome`, `Descricao`)
VALUES (1, 'Test Nome', 'Test Descricao');
The database encoding is 'latin1', but the database configuration is as shown:
'mycnn' => array(
'driver' => 'pdo',
'dsn' => 'mysql:dbname={$mydb};host={$myhost}',
'username' => '{$myuser}',
'password' => '{$mypassword}',
'driver_options' => array(
PDO::MYSQL_ATTR_INIT_COMMAND => 'SET NAMES \'UTF8\''
),
)
As you can see the above scenario, I have setup the driver for "UTF-8", the column name "Código" has a special character and renaming this column is not an option.
The syntax that I am using for updating in the model is:
$set = array("Nome" => "Edited Test");
$where = array("Código" => 1);
$this->tableGateway->update($set, $where);
After that, the ZF is parsing the SQL throwing the Exception:
UPDATE "clientes" SET "Nome" = 'Edited Test' WHERE "C`\xC3`\xB3`digo" = 1
I have also removed the UTF-8 option, since the catalog is "latin1_swedish_ci" without success.
I would appreciate anyone who gives me a hint how to face this issue.
Thanks in advance.
Make sure your database encoding type is UTF-8.
'driver_options' => array(
PDO::MYSQL_ATTR_INIT_COMMAND => 'SET NAMES \'UTF8\''
),
Make sure fields have utf8_general_ci.
In your layout phtml head has
<meta charset='utf-8'>
Updated
As you said you are not able to change encoding to utf-8 so use one of the following commands using driver_options
'SET NAMES \'latin1\'' or 'SET CHARACTER SET \'latin1\''
For more details check out the doc please!
When problem with column name which has latin1 characters
Just pass the condition as string not an array as the second argument into TableGateway's update() method. But you must set this 'SET NAMES \'UTF8\'' in your driver_options.
$id = 1;
$where = "Código = {$id}";
// Or use this way
$where = "Código = 1";
$this->tableGateway->update($set, $where);
I have a legacy MySQL database with latin1 encoding, and new MySQL database with utf-8 encoding. When I migrate the data from the legacy to new one, the strings returned and rendered in webpages contains weird characters.
example: â¥Chocolate biscuitâ¢
There's a model that connects to the old database to get the records, and use a rake task to map the old records and create new records
class LegacyDatabase < ActiveRecord::Base
self.abstract_class = true
establish_connection(
adapter: "mysql2",
database: "old_database",
encoding: "latin1"
)
end
I referred to this website to solve the encoding problems, but the fallback hash is hardcoded. I need a dynamic one so that I don't need to add the fallbacks each time when a loop that fixes encoding is called.
def fix_encoding(str)
# Referring to the solution from the website
new_str = str.encode('cp1252',:fallback => {
"\u0080" => "\x80".force_encoding("cp1252"),
"\u0081" => "\x81".force_encoding("cp1252"),
"\u008D" => "\x8D".force_encoding("cp1252"),
"\u008F" => "\x8F".force_encoding("cp1252"),
"\u0090" => "\x90".force_encoding("cp1252"),
"\u009D" => "\x9D".force_encoding("cp1252"),
"\u0098" => "\x98".force_encoding("cp1252"),
"\u0099" => "\x99".force_encoding("cp1252")
}).force_encoding("utf-8")
return new_str
end
I want to change it to be dynamic, but I failed to convert it.
# How to do it in dynamic?
new_str = str.encode('cp1252', :fallback => Proc.new { |v| "#{v[4..5]}".force_encoding("cp1252") }).force_encoding("utf-8")
Or is there any other solution to fix the encoding problem?
I'm having some trouble with running a query on a database using utf8 encoding.
Everything works fine until I try a string with special characters.
I can't really make a lot of changes to the database since I'm adding some separate pages to an existing prestashop template.
This is my code (sorry, but some of it is in Romanian)
$acces_server=mysql_connect($db_host, $db_user, $db_pass);
if(!$acces_server) die ("Server-ul nu este disponibil momentan !");
mysql_set_charset('utf8');
echo mysql_client_encoding();
$acces_bd = mysql_select_db($db_name);
if (!$acces_bd) die ("Ne pare rau, dar baza de date nu poate fi accesata momentan !");
mysql_query('SET NAMES utf8') or die(mysql_error());
mysql_query('SET CHARACTER SET utf8') or die(mysql_error());
mysql_query('SET COLLATION_CONNECTION="utf8_general_ci" ') or die(mysql_error());
If I try to make a query like this I get 0 rows:
$experienta= $_GET['name'];
$rez=mysql_query("SELECT nume_cutie FROM rez_experiente WHERE nume = '".$experienta."'");
I've used echo to display the query and then typed it in phpMyAdmin and it worked .
Also the head section has this line :
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
I really don't know what else to do , all the solutions I've found so far don't seem to work. I have even tried using mysqli instead and the result was the same.
Please help me , and thank you for your time.
*Edit
I tried using stmt->prepare with mysqli and that didn't work.
I've also tried doing this :
$experienta2=str_replace("?","%",utf8_decode($experienta));
$rez=mysql_query("SELECT nume_cutie FROM rez_experiente WHERE nume LIKE '".$experienta2."'");
Wich turns this :
SELECT nume_cutie FROM rez_experiente WHERE nume LIKE '5th Avenue Beauty Center - Răsfăţ'
Into this:
SELECT nume_cutie FROM rez_experiente WHERE nume LIKE '5th Avenue Beauty Center - R%sf%%'
And again the query works in phpMyAdmin but mysql_query() returns nothing
I switched to utf8_unicode_ci and nothing changed.
After the database connection, run these queries:
SET character_set_client=utf8;
SET character_set_connection=utf8;
SET character_set_database=utf8;
SET character_set_results=utf8;
SET character_set_server=utf8;