Im having a strange problem with some data that is saved in mysql.
im saving a certain value as text in my db but for some reason some of the text is transformed to this "ＡＴＥＬＯＰＨＯＢＩＡ" kind of symbols.
the code:
$data = mysql_real_escape_string($_GET['data']);
$id = $_GET["id"];
mysql_query("INSERT INTO table (ID, name, pos,data) VALUES ('$id', '$regionname','$regionps[0]','$data')");
table structure:
CREATE TABLE IF NOT EXISTS `sim_scanner` (
`ID` varchar(64) COLLATE latin1_general_ci NOT NULL,
`simname` varchar(36) COLLATE latin1_general_ci NOT NULL,
`simpos` varchar(25) COLLATE latin1_general_ci NOT NULL,
`time` varchar(25) COLLATE latin1_general_ci NOT NULL,
`data` text COLLATE latin1_general_ci NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci;
the data that is transforming itself to that is a value like this , <151.25,255.54,1254.12>(vector), as the data is generated by a script i don't know the exact value its basicly a characters position once it moves the data is another vector
thank you for your time
your mysql character encoding is different to the data character encoding that's why ...
for example try use the utf-8_general_ci in DB character set..
UTF-8 definition for the characters they support, usually UTF8 supports most of the characters
Related
I'm having an issue with emojis not being transferred correctly to my staging site. Some emojis are replaced by a question mark, some don't. It's mainly affecting the blog posts. I'm not sure what is causing this and was hoping someone could point me in the right direction to solving this problem! Could it be something with the character encoding in the database?
For example:
These work: ☀️ ☔️❤️✨
These get replaced by a question mark: 🇸🇪💛🎉😍😘🌸😎
and show this Error:
1366 Incorrect string value: '\xF0\x9F\x98\x83\xF0\x9F...' for column 'text' at row 1
MY SQL Code:
CREATE TABLE `posts` (
`id` int(11) UNSIGNED NOT NULL,
`user_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '0',
`text` longtext COLLATE utf8mb4_unicode_ci NOT NULL,
`status` varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '0',
`timestamp` varchar(50) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT '0'
) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
Have run and tried this SQL Codes:
ALTER TABLE posts CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin
ALTER DATABASE test CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
The table definition is correct.
Where are you trying to insert from? Make sure the connection is using utf8mb4-charset. From command line use the --default-character-set=utf8mb4 option.
See db-fiddle
I am trying to add Turkish names on my table but then when displayed it gives me ? instead of any of them. Any help what I am missing here? This is my table:
CREATE TABLE IF NOT EXISTS `offerings` (
`dep` varchar(5) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`grade` varchar(4) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`section` varchar(3) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`name` varchar(100) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`teacher` varchar(50) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`quota` varchar(2) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`lec1` varchar(35) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`lec2` varchar(35) CHARACTER SET utf16 COLLATE utf16_turkish_ci DEFAULT NULL,
`lec3` varchar(35) DEFAULT NULL,
`lec4` varchar(35) DEFAULT NULL,
`lec5` varchar(35) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf16;
As suggested from the answer I choose here is the solution to the problem for whoever googles this topic. Special thanks to all who contributed in the solution of my problem.
CREATE TABLE IF NOT EXISTS `offerings` (
`dep` varchar(5) NOT NULL,
`grade` varchar(4) COLLATE utf8_unicode_ci NOT NULL,
`section` varchar(3) COLLATE utf8_unicode_ci NOT NULL,
`name` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`teacher` varchar(50) COLLATE utf8_unicode_ci NOT NULL,
`quota` varchar(2) COLLATE utf8_unicode_ci,
`lec1` varchar(35) DEFAULT NULL,
`lec2` varchar(35) DEFAULT NULL,
`lec3` varchar(35) DEFAULT NULL,
`lec4` varchar(35) DEFAULT NULL,
`lec5` varchar(35) DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
(Beginnings of an answer...)
Please don't use utf16; there is virtually no reason for such in a MySQL table.
So, assuming you switch to utf8, let's see if we can get rid of the ? problems.
utf8 needs to be established in about 4 places.
The column(s) in the database -- Use SHOW CREATE TABLE to verify that they are explicitly set to utf8, or defaulted from the table definition. (It is not enough to change the database default.)
The connection between the client and the server. See SET NAMES utf8.
The bytes you have. (This is probably the case.)
If you are displaying the text in a web page, check the <meta> tag.
What probably happened:
you had utf8-encoded data (good)
SET NAMES latin1 was in effect (default, but wrong)
the column was declared CHARACTER SET latin1 (default, but wrong)
Since the CHARACTER SET disagrees with what you have shown, the problem is possibly more complex. Please provide
SELECT col, HEX(col) FROM tbl WHERE ...
for some simple cell with Turkish characters. With this, I may be able to figure out what happened.
Also, Reference notes on encodings.
VARCHARs are character strings, while NVARCHARS are Unicode character strings. NVARCHARS require more bits per character to store, but have a greater range. Try updating your data types. This should fix your problem.
EDIT This answer is wrong. The OP clearly asked for a MySQL solution, but the above applies only to SQL Server.
I need some help getting a MySQL table to store and output characters from the following languages:
English
French
Russian
Turkish
German
These are the languages I know of in the data. It also uses mathematical symbols such as this:
b ∈ A. Define s(A):= supn≥0 r A (n) for each A ⊆ ? ∪ {0}.
I'm using htmlentities to encode the text. The ? above is meant to display as a ℕ.
It displays this way when I look at the data in PhpMyAdmin. The other characters are encoded as expected.
The table is set to utf8_unicode_ci and all aspects of the website have been set to UTF-8 (including via the .htaccess file, a PHP header and a meta tag).
Please help?
Additional info:
Hosting environment:
Linux, Apache
Mysql 5.5.38
PHP Version 5.4.4-14
Connection string :
ini_set('default_charset', 'UTF-8');
$mysqli = new mysqli($DB_host , $DB_username, $DB_password);
$mysqli->set_charset("utf8");
$mysqli->select_db($DB_name);
SHOW CREATE TABLE mydatabase.mytable output:
CREATE TABLE `tablename` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`created` datetime NOT NULL,
`updated` datetime NOT NULL,
`product` int(11) NOT NULL,
`ppub` tinytext COLLATE utf8_unicode_ci NOT NULL,
`pubdate` date NOT NULL,
`numerous_other_tinytext_cols` tinytext COLLATE utf8_unicode_ci NOT NULL,
`numerous_other_tinytext_cols` tinytext COLLATE utf8_unicode_ci NOT NULL,
`text` text COLLATE utf8_unicode_ci NOT NULL,
`keywords` tinytext COLLATE utf8_unicode_ci NOT NULL,
`active` int(11) NOT NULL DEFAULT '1',
`orderid` int(11) NOT NULL,
`src` tinytext CHARACTER SET latin1 NOT NULL,
`views` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=17780 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
SELECT DEFAULT_CHARACTER_SET_NAME FROM information_schema.SCHEMATA output:
DEFAULT_CHARACTER_SET_NAME
utf8 [->UTF-8 Unicode]
utf8mb4 [->UTF-8 Unicode]
Fonts used:
Arial
Sample of text in the database:
Let <em>A</em> be a subset of the set of nonnegative integers ℕ ∪ {0}, and let <em>r</em><sub><em>A</em></sub> (<em>n</em>) be the number of representations of <em>n</em> ≥ 0 by the sum <em>a</em> + <em>b</em> with <em>a, b</em> ∈ <em>A</em>.
Output on web page:
Let <em>A</em> be a subset of the set of nonnegative integers ? ∪ {0}, and let <em>r</em><sub><em>A</em></sub> (<em>n</em>) be the number of representations of <em>n</em> ≥ 0 by the sum <em>a</em> + <em>b</em> with <em>a, b</em> ∈ <em>A</em>.
Which becomes
Let A be a subset of the set of nonnegative integers ? ∪ {0}, and let rA (n) be the number of representations of n ≥ 0 by the sum a + b with a, b ∈ A.
While your database and table are configured to use UTF-8, one of your columns still isn't:
CREATE TABLE `tablename` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`created` datetime NOT NULL,
`updated` datetime NOT NULL,
`product` int(11) NOT NULL,
`ppub` tinytext COLLATE utf8_unicode_ci NOT NULL,
`pubdate` date NOT NULL,
`numerous_other_tinytext_cols` tinytext COLLATE utf8_unicode_ci NOT NULL,
`numerous_other_tinytext_cols` tinytext COLLATE utf8_unicode_ci NOT NULL,
`text` text COLLATE utf8_unicode_ci NOT NULL,
`keywords` tinytext COLLATE utf8_unicode_ci NOT NULL,
`active` int(11) NOT NULL DEFAULT '1',
`orderid` int(11) NOT NULL,
`src` tinytext CHARACTER SET latin1 NOT NULL, <--------- This one
`views` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=17780 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
Since all the other symbols got HTML-encoded, they will survive all charsets, but not ℕ, which doesn't have a named entity reference.
You need to convert your column:
ALTER TABLE tablename MODIFY src TINYTEXT CHARACTER SET utf8;
NOTE: I noticed you like mathematical symbols. Some of them are outside of the Basic Multilingual Plane, ie. have codepoints > 0xFFFF, for example mathematical letter variants (fraktur, double-struck, semantic italic etc.).
If you want to support them, you need to switch encoding in MySQL everywhere (table, columns, connection) to utf8mb4, which is the true UTF-8 (utf8 in MySQL means the subset of UTF-8 with BMP only), with utf8mb4_unicode_ci collation. Here is how to do the migration.
Also, I have noticed that you are HTML-encoding HTML. Maybe you have a reason, but in my opinion storing this doesn't make sense:
<em>A</em>
If you want to put it into an HTML document, now you need to HTML-decode it at least once, sometimes twice. I'd rather store what almost everybody else does:
<em>A</em>
This way, you will store Unicode characters natively, in optimal way.
I'm working on a website in which we save Japanese characters in a text field.
The table looks like this:
| contactlogs | CREATE TABLE `contactlogs` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`contact_email` varchar(128) CHARACTER SET latin1 NOT NULL,
`name` varchar(128) CHARACTER SET latin1 DEFAULT NULL,
`company_name` varchar(128) CHARACTER SET latin1 DEFAULT NULL,
`email` varchar(128) CHARACTER SET latin1 DEFAULT NULL,
`telephone` varchar(30) CHARACTER SET latin1 DEFAULT NULL,
`fax` varchar(30) CHARACTER SET latin1 DEFAULT NULL,
`subject` text COLLATE utf8_unicode_ci NOT NULL,
`message` text COLLATE utf8_unicode_ci NOT NULL,
`created` datetime DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=4 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
When I retrieve the data that was saved, the message field is often cut off, with a garbage character at the end (I assume because it might not have saved all the data of the last character.
The website itself is in cakephp. The data is being saved by just doing a $this->model->save($data) (Haven't changed anything about the way it's saving, the model itself is empty). There's no special database settings. Just set host, login, persistent => false, driver => mysql, database, prefix.
Your DB and tables are probably under latin1 default collation, so without 'encoding' => 'utf8' in your database config cake is gonna be talking to the DB in latin1. The transcoding of UTF to latin1 and back leaves space for data to be lost. This is not really cake's problem.
So go on and change the connection to UTF and all default collations in the DB too. Newly saved data will be displayed fine, for old ones it will depend, I won't go into details cause it's out of context.
I have a .sql file and I want to load it into MySQL database. I don't know from which database (MySQL or MS-SQL) it was created.
Now, I am trying to import that file into MySQL database. It is showing errors while importing and executing that file.
Q1. So, my question is whether the .sql file generated from MySQL and MS-SQL are different?
Note: I am using SQLYog software (graphical interface for MySQL) for importing the file.
Here is the ERROR:
Query:
CREATE TABLE ads (
id bigint(20) NOT NULL auto_increment,
city_id int(11) NOT NULL,
type text collate utf8_bin NOT NULL,
town text collate utf8_bin NOT NULL,
address text collate utf8_bin NOT NULL,
price text collate utf8_bin NOT NULL,
info text collate utf8_bin NOT NULL,
link text collate utf8_bin NOT NULL,
hasImage int(11) NOT NULL,
language varchar(2) collate utf8_bin NOT NULL,
time_added varchar(255) collate utf8_bin NOT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=1
Error occured at:2009-09-08 17:41:01
Line no.:35
Error Code: 1050 - Table 'ads' already exists
Query:
CREATE TABLE ads (
id bigint(20) NOT NULL auto_increment,
city_id int(11) NOT NULL,
type text collate utf8_bin NOT NULL,
town text collate utf8_bin NOT NULL,
address text collate utf8_bin NOT NULL,
price text collate utf8_bin NOT NULL,
info text collate utf8_bin NOT NULL,
link text collate utf8_bin NOT NULL,
hasImage int(11) NOT NULL,
language varchar(2) collate utf8_bin NOT NULL,
time_added varchar(255) collate utf8_bin NOT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=1
Error occured at:2009-09-08 17:41:21
Line no.:35
Error Code: 1050 - Table 'ads' already exists
Query:
CREATE TABLE ads (
id bigint(20) NOT NULL auto_increment,
city_id int(11) NOT NULL,
type text collate utf8_bin NOT NULL,
town text collate utf8_bin NOT NULL,
address text collate utf8_bin NOT NULL,
price text collate utf8_bin NOT NULL,
info text collate utf8_bin NOT NULL,
link text collate utf8_bin NOT NULL,
hasImage int(11) NOT NULL,
language varchar(2) collate utf8_bin NOT NULL,
time_added varchar(255) collate utf8_bin NOT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=1
Error occured at:2009-09-08 17:41:35
Line no.:35
Error Code: 1050 - Table 'ads' already exists
Query:
CREATE TABLE ads (
id bigint(20) NOT NULL auto_increment,
city_id int(11) NOT NULL,
type text collate utf8_bin NOT NULL,
town text collate utf8_bin NOT NULL,
address text collate utf8_bin NOT NULL,
price text collate utf8_bin NOT NULL,
info text collate utf8_bin NOT NULL,
link text collate utf8_bin NOT NULL,
hasImage int(11) NOT NULL,
language varchar(2) collate utf8_bin NOT NULL,
time_added varchar(255) collate utf8_bin NOT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=1
Error occured at:2009-09-08 17:42:07
Line no.:35
Error Code: 1050 - Table 'ads' already exists
Query:
1 Stanford University 6700 http://www.orkut.co.in
.
.
.
The file extension ".sql" is essentially meaningless: it's just there so that you know what the file is. It will just be a plain text file containing SQL, which you can open in Notepad. Therefore, there's no special "mysql" or "sql server" extensions.
The errors you're getting there "Table 'ads' already exists" are because you're trying to create a table which already exists in the database. (Did you read the error?) You have a few options:
Change the SQL to this:
CREATE TABLE IF NOT EXISTS ads ( id bigint(20) ...
Change the SQL to this:
DROP TABLE IF EXISTS ads;
CREATE TABLE ads (id bigint(20) ...
Clear out all the tables in the DB first.
MySQL and Microsoft SQL Server do, unfortunately, implement different dialects of SQL. So, the answer to your question :
Q1. So, my question is whether the
.sql file generated from MySQL and
MS-SQL are different?
is "yes, quite possiby". However, the specific error you're showing appears to be due strictly to a "Create table" statement on line 35 being executed over and over and over again, and that can't depend on such dialect differences, so the causes have to be other ones (such as bugs or subtle format differences in what that "SQLYog" program, which I'm not familiar with, expects as its input).
It depends on which software exported this file. Also check in the file's header which software did it and analyze the syntax to determine where this file belongs to. Note also that SQL is microsoft sql server.
Yes. SQL varies greatly from database to database. While there is a SQL standard which most databases support much of, every database has numerous, incompatible nonstandard features and functions above and beyond that specified by the standard.
There are also many migration docs avaiable:
MSSQL => MySQL:
http://dev.mysql.com/tech-resources/articles/migrating-from-microsoft.html
MySQL => MSSQL:
http://technet.microsoft.com/en-us/library/cc966396.aspx
There are two obvious things in your CREATE TABLE statement that MSSQL doesn't support which jump out immediately:
ENGINE=InnoDB
AUTO_INCREMENT=1
Those are both MySQL-specific statements.
It looks like your script is being executed multiple times (as the line number where the error occurs is the same each time). The error also indicates that you're trying to define a table that already exists, which may be from the first time this script ran. You may want to look into how the script is being run to see why it may be executing multiple times.
The SQL itself looks fine for MySQL. It wouldn't run with SQL Server.