SQL Server Migration Assistant for MySQL (Chinese display error)
I am using SQL Server Migration Assistant for MySQL to migrate from Mysql to SQL Server
After the migration data is completed Chinese display ?? Error display!
迁移工具
Our data is available in four languages
zh_TW
zh_CN
pt_PT
en
I finally solved the problem completely
When I link SQL server data, I set charset = utf8, but there are still individual displays in Chinese Simplified "?"
At last, we set the field type to nvarchar (max). At first, I knew that modifying the field type could solve the problem, but we were migrating data and could not modify the field type. Using symfony schema update to generate SQL statements also did not support nvarchar (max). At last, we modified the source code of generating SQL statements. Using text type, we could compile nvarchar (max)
enter image description here
Related
I have a .NET 6 application that is currently backed by MSSQL database, that is maintained by Entity Framework using a Model First approach. I am trying to migrate it to use a MySQL database backend, for a variety of reasons.
I have installed MySQL Locally (Windows) to start exploring and getting it working. I can migrate the schema easily enough (With either MySQL Workbench or using EF) but migrating the data is proving to be a little tricky.
Around half of the tables migrated fine, but the other half, relating to string data, are failing due to errors which look a little like this - the column obviously differs from table to table. The source data is nvarchar in SQL, and the destination is type `varchar'
Statement execution failed: Incorrect string value: '\xF0\x9F\x8E\xB1' for column 'AwayNote'
Does anyone know how I can get the Migration to run successfully?
The research I have read has said to ensure server and table character sets are aligned as per the below.
I have set up my Source as SQL using the ODBC FreeTDS
The data import screen is set up like this - the check box doesn;t seem to affect things especially.
I have MySQL setup with this too, which I have also read is important.
[mysql]
default-character-set = utf8mb4
Final Update
I was able to easily migrate the data with Talend. No errors, and it worked perfectly the first time with no special settings. This shows what an utter piece of garbage the MySQL Workbench Migration tool is. While the learning curve of Talend is rough (it's not intuitive at all), it appears to be one of the best data migration solutions out there. I recommend using it. Note I never figured out why the migration failed (as seen below). I'm just walking away from the utter garbage Oracle has pushed on the community. Oh, and Talend migrated the data to utf8mb4/utf8_general_ci without a hitch.
Please note there are updates at the bottom.
We have to migrate an export from TrackerRMS (which luckily doesn't have FK constraints, but the data is a total mess) to MySQL. Restoring the backup of the TrackerRMS data to SQL Server is cake; no issues. The problem is copying the data from SQL Server to MySQL.
MySQL Workbench Migration can handle all but 4 of the tables; but those 4 tables are the key problem. They have crazy content in their fields which causes the migration tool to choke. I attempted to export the data as .sql from HeidiSQL and it chokes as well.
The source table problem fields are NVARCHAR(MAX) and SQL_Latin1_General_CP1_CI_AS collation.
Note I've tried changing the collation of the source SQL Server table columns to Latin1_General_100_BIN2_UTF8 and Latin1_General_100_CI_AI_SC_UTF8 and there is no effect.
The errors are:
ERROR: `Backup_EmpowerAssociates`.`BACKUP_documents`:Inserting Data: Incorrect string value: '\xF0\x9F\x93\x8A x...' for column 'filepath' at row 13
ERROR: `Backup_EmpowerAssociates`.`BACKUP_activities`:Inserting Data: Incorrect string value: '\xF0\x9F\x91\x80' for column 'subject' at row 42
ERROR: `Backup_EmpowerAssociates`.`BACKUP_resourcehistory`:Inserting Data: Incorrect string value: '\xF0\x9D\x91\x82(\xF0...' for column 'jobdescription' at row 80
This tells me the source data has 4-byte character details (which is beyond the standard utf8). Note the destination database in MySQL is utf8mb4 and utf8mb4_unicode_ci collated, and has the default settings as such. No connection settings override this.
When migrating I use Microsoft SQL Server and ODBC (native) for localhost (SQL Server) with default options. I've also tried turning ANSI off, but it has no impact. Note the ODBC configuration for SQL Server has no charset or collation settings or options. For target, I use the localhost stored connection which I use for general access.
Note the MySQL Workbench migration tool defines the receiving table columns (for the above problem columns) as LONGTEXT CHARACTER SET 'utf8mb4'.
Could the issue be the migration proxy (ODBC?) is somehow converting it to utf8 (even though I don't have that selected)? But if that was the case, wouldn't the incoming data not be erroring out in the migration process as a UTF8MB4 solution (4-byte vs less)?
Note I tried creating and adjusting the destination MySQL table (by adjusting the SQL in the migration tool) as CHARSET latin1 and latin1_general_ci collation. Same issue.
Migration simply does not want to work (this is with SQL Server source being SQL_Latin1_General_CP1_CI_AS). And I've tried it with UTF8 both on and off for driver. No effect.
Does anyone with migration experience recognize this issue, or have recommendations on how to resolve the problem? I'm fine with scrubbing the source data in SQL Server before I migrate - I just don't know the best method to do that (or if it's necessary).
Thanks!
===
UPDATE 1
This is very strange; using the below technique to show values that won't convert, this is the result:
SELECT filepath, CONVERT(varchar,filepath) FROM BACKUP_documents WHERE filepath <> CONVERT(varchar, Filepath);
Why on earth is the data being truncated upon convert with a simple filename at the "c" in documents?
Here's a capture that might also help resolve this issue.
But the strange part is MSSQL is showing normal text (without special characters) as being non-ASCII. I'm wondering if the folks at TrackerRMS are running code written in another country/language and it's messing up the data, but it's something that's not visible?
UPDATE 2
So to make things clear, here's what one of the characters that is messing up the data looks like.
I was able to easily migrate the data with Talend. No errors, and it worked perfectly the first time with no special settings. This shows what an utter piece of garbage the MySQL Workbench Migration tool is. While the learning curve of Talend is rough (it's not intuitive at all), it appears to be one of the best data migration solutions out there. I recommend using it. Note I never figured out why the migration failed (as seen below). I'm just walking away from the utter garbage Oracle has pushed on the community. Oh, and Talend migrated the data to utf8mb4/utf8_general_ci without a hitch.
When migrating from SQL Server to MySQL, I'm having the error "Incorrect String Value" when the string has any Portuguese specific character (like in "Olá", which means hello).
The problem is that this occur in same tables, but not on others. I have a tableA which have this characters, but imports correctly, but another tableB gives me the error. Both tables have the same codepage.
I'm using the script generated my MySQL Migration Tool, which calls wbcopytables.exe.
Does anyone has any clue on what could be the problem?
I have a live MySQL database with UTF-8 data in it. I'm trying to migrate it to SQL Server 2005.
All table fields and all data are in UTF8.
The destination database will have SQL_Latin1_General_CP1_CI_AS collation .
What I did so far was to create an ODBC connection to my MySQL database and do the following:
EXEC master.dbo.sp_addlinkedserver #server = N'MYSQL2', #srvproduct=N'MySQL', #provider=N'MSDASQL', #provstr=N'DRIVER={MySQL ODBC 5.1 Driver}; SERVER=SERVERHOST; DATABASE=mysqldb; USER=mysqluser;PASSWORD=PASSWORD; OPTION=3;charset:utf8;';
select * into mssql.dbo.members from openquery(MySQL, 'select * from mysqldb.members');
the collation that my SQL Server has is Greek_CI_AS.
the destination database has SQL_Latin1_General_CP1_CI_AS collation and I created another one with Greek_CI_AS collation just to see if that was the problem.
The import went well and all table fields in SQL Server were created correctly BUT the data that was stored had encoding ISO-8859-7 (which i presume comes from the Greek_CI_AS collation).
this is a problem since our current site has UTF-8 pages and it should stay that way.
If i use Firefox and MDDB2 to try and get the data I get confused characters. If I change the View-> encoding to ISO-8859-7 I see the data correctly .. but this is not the solution
so how to do this migration and have the correct encoding on the destination SQL Server?
any help appreciated
thanks
you need to tell us what types of data you're using on the MS SQL side.
Are you using VARCHAR or NVARCHAR?
You need to use NVARCHAR if you want to store unicode data.
I want to convert a database from access to SQL Server using SSIS. I cannot convert the date/time columns of the access db. SSIS says something like:
conversion between DT_Date and DT_DBTIMESTAMP is not supported.
(Its translated from my German version, might be different in English version). In Access I have Date/Time column, in SQL Server I have datetime. In the dataflow chart of the SSIS I have a OLE DB source for the access db, an sql server target and a data conversion. In the data conversion I convert the columns to date[DT_DATE]. They are connected like this:
AccessDB -> conversion -> SQL DB
What am I doing wrong? How can I convert the Access date columns to SQL Server date columns?
Looks like you'll need to add a specific conversion to convert this column to the correct type. It's the DT_DBTIMESTAMP that you need to convert to, not DT_DATE. Using a Derived Column Transformation for instance, you could use an expression like:
(DT_DBTIMESTAMP)YourDateColumn
You use another Provider on the source side. I used the JET provider. I am doing this on a computer where no Access is installed. When I try to create a data connection using the Office 12.0 Access Database Engine OLE DB Provider I get an error message that the workgroup information file is missing. I get this message on the connection test.
The question is if the conversion depends on the db driver you use to get the data. I cannot install Access on the server I am working, so I cannot try it. Could this be the cause of the problem?
Casting D_Date column to DT_DBTIMESTAMP solve the same problem for me.