I'd like to insert HTML data in SQL Server using ASP.NET and a stored procedure but I'm not able to insert all the tags and double quotes. Any help would be appreciated.
You can use base64 encoding to store and decoding when reading the data
First try to strip down your problem to the basics:
SQL-Server has a datetype nvarchar(max). This typ can store a lot of data. All characters that "arive" at the database can be inserted. You shoul first check, what data arives at the server. An easy way is SQL prifiler. You should see the data in Profiler, that "arives" at the database. If things are broken here (i.e.wrong encoding) you should the way up to you application.
Which way comes the data? You only mention ASP.NET. You can connect to the database using ODBC, JDBC, OLEDB or ADO-driver (there might be more). Data can be converterd on this way. You can set up a Trace an see what happens. The easy way normaly is the fastest: Check the options of the driver/try other settings. Most drivers have only a small number of options. If you can not find the problem here...
Your application. Strip it down to the minimal functionality: Establish a connection to the database an insert some data into a table. Does this work? Does it work with different collation? Does it word with unicode data?
Maybe a better question could better be answered.
Related
Final Update
I was able to easily migrate the data with Talend. No errors, and it worked perfectly the first time with no special settings. This shows what an utter piece of garbage the MySQL Workbench Migration tool is. While the learning curve of Talend is rough (it's not intuitive at all), it appears to be one of the best data migration solutions out there. I recommend using it. Note I never figured out why the migration failed (as seen below). I'm just walking away from the utter garbage Oracle has pushed on the community. Oh, and Talend migrated the data to utf8mb4/utf8_general_ci without a hitch.
Please note there are updates at the bottom.
We have to migrate an export from TrackerRMS (which luckily doesn't have FK constraints, but the data is a total mess) to MySQL. Restoring the backup of the TrackerRMS data to SQL Server is cake; no issues. The problem is copying the data from SQL Server to MySQL.
MySQL Workbench Migration can handle all but 4 of the tables; but those 4 tables are the key problem. They have crazy content in their fields which causes the migration tool to choke. I attempted to export the data as .sql from HeidiSQL and it chokes as well.
The source table problem fields are NVARCHAR(MAX) and SQL_Latin1_General_CP1_CI_AS collation.
Note I've tried changing the collation of the source SQL Server table columns to Latin1_General_100_BIN2_UTF8 and Latin1_General_100_CI_AI_SC_UTF8 and there is no effect.
The errors are:
ERROR: `Backup_EmpowerAssociates`.`BACKUP_documents`:Inserting Data: Incorrect string value: '\xF0\x9F\x93\x8A x...' for column 'filepath' at row 13
ERROR: `Backup_EmpowerAssociates`.`BACKUP_activities`:Inserting Data: Incorrect string value: '\xF0\x9F\x91\x80' for column 'subject' at row 42
ERROR: `Backup_EmpowerAssociates`.`BACKUP_resourcehistory`:Inserting Data: Incorrect string value: '\xF0\x9D\x91\x82(\xF0...' for column 'jobdescription' at row 80
This tells me the source data has 4-byte character details (which is beyond the standard utf8). Note the destination database in MySQL is utf8mb4 and utf8mb4_unicode_ci collated, and has the default settings as such. No connection settings override this.
When migrating I use Microsoft SQL Server and ODBC (native) for localhost (SQL Server) with default options. I've also tried turning ANSI off, but it has no impact. Note the ODBC configuration for SQL Server has no charset or collation settings or options. For target, I use the localhost stored connection which I use for general access.
Note the MySQL Workbench migration tool defines the receiving table columns (for the above problem columns) as LONGTEXT CHARACTER SET 'utf8mb4'.
Could the issue be the migration proxy (ODBC?) is somehow converting it to utf8 (even though I don't have that selected)? But if that was the case, wouldn't the incoming data not be erroring out in the migration process as a UTF8MB4 solution (4-byte vs less)?
Note I tried creating and adjusting the destination MySQL table (by adjusting the SQL in the migration tool) as CHARSET latin1 and latin1_general_ci collation. Same issue.
Migration simply does not want to work (this is with SQL Server source being SQL_Latin1_General_CP1_CI_AS). And I've tried it with UTF8 both on and off for driver. No effect.
Does anyone with migration experience recognize this issue, or have recommendations on how to resolve the problem? I'm fine with scrubbing the source data in SQL Server before I migrate - I just don't know the best method to do that (or if it's necessary).
Thanks!
===
UPDATE 1
This is very strange; using the below technique to show values that won't convert, this is the result:
SELECT filepath, CONVERT(varchar,filepath) FROM BACKUP_documents WHERE filepath <> CONVERT(varchar, Filepath);
Why on earth is the data being truncated upon convert with a simple filename at the "c" in documents?
Here's a capture that might also help resolve this issue.
But the strange part is MSSQL is showing normal text (without special characters) as being non-ASCII. I'm wondering if the folks at TrackerRMS are running code written in another country/language and it's messing up the data, but it's something that's not visible?
UPDATE 2
So to make things clear, here's what one of the characters that is messing up the data looks like.
I was able to easily migrate the data with Talend. No errors, and it worked perfectly the first time with no special settings. This shows what an utter piece of garbage the MySQL Workbench Migration tool is. While the learning curve of Talend is rough (it's not intuitive at all), it appears to be one of the best data migration solutions out there. I recommend using it. Note I never figured out why the migration failed (as seen below). I'm just walking away from the utter garbage Oracle has pushed on the community. Oh, and Talend migrated the data to utf8mb4/utf8_general_ci without a hitch.
I am migrating a database from PostgresSql to MySql.
We were saving files in the database as PostgreSQL bytea columns. I wrote a script to export the bytea data and then insert the data into a new MySql database as a blob. The data is inserting into Mysql fine, but it is not working at the application level. However, the application should not care, as the data is exactly the same. I am not sure what is wrong, but I feel like it is some difference between MySql vs. PostgreSQL. Any help would be greatly appreciated.
This could really be a number of issues, but I can provide some tips in regards to converting binary data between sql vendors.
The first thing you need to be aware of is that each sql database vendor uses different escape characters. I suspect that your binary data export is using hex and you most likely have unwanted escape characters when you import to your new database.
I recently had to do this. The exported binary data was in hex and vendor specific escape characters were included.
In your new database, check if the text value of the binary data starts with an 'x' or unusual encoding. If it does you need to get rid of this. Since you already have the data inserting properly, to test, you can just write an sql script to remove any unwanted vendor specific escape characters from each imported binary data record in your new database. Finally, you may need to unhex each each new record.
So, something like this worked for me:
UPDATE my_example_table
SET my_blob_column = UNHEX(SUBSTRING(my_blob_column, 2, CHAR_LENGTH(my_blob_column)))
Note: The 2 in the SUBSTRING function is because the export script
was using hex and prepending '\x' as a vendor specific escape character.
I am not sure that will work for you, but it maybe worth a try.
My requirement is to create a stored procedure that joins two tables and inserts it into a remote server table.
How can I solve this without using ssis and any import/export data?
please
use the search before asking your question
Be more specific, stackoverflow is not here to write your whole program, people assist you
Use punctations marks and have a look at your format
Your question
Have a look at this, it shows you how to do that. You can first add a linked server(thats the destionation in your case) and then use it for your insert statement.
I am using SQL server 2008 and have the column in my table set to nvarchar. Data with special characters are getting stored wrongly in this table. Eg: this is one entry
Need to check if doesn’t comes as doesn’t itself and don’t comes asdon’t itself and ensure closure of issues.
The garbage ’ should actually be an apostrophe ('). I have checked my collation string. At database level it is SQL_Latin1_General_CP850_BIN2 and at server level it is SQL_Latin1_General_CP1_CI_AS.
I know for sure the encoding set everywhere else in my application is UTF-8.
How do I store the data correctly in my table? Do I need to change my SQL queries or any settings in the database?
Please advise.
You need to make sure that you're observing two things:
Always use NVARCHAR as datatype for your columns
Always make sure to use the N'....' prefix when dealing with string literals (for example in your INSERT or UPDATE statements)
With those two things in place, SQL Server has no trouble at all storing all Unicode characters you might throw at it...
This is not exactly the same as passing a huge string parameter to stored procedure ....
I have a SQL Server 2008 sproc that takes an id int and a large string and inserts it into a table. The sproc is called by a .NET 4.0 code, which does a File.ReadAllText into a string and then sends it to the sproc. The source of the string is a text file. The thought of future reading many 100mb files all day into an immutable "use once" strings and then sending it to SQL Server sounds memory-wasteful on the C# side.
What is a smarter way to stream the text from disk to the sproc? I can change the current Varchar(Max) parameter to anything that makes better sense.
All ideas appreciated.
Thanks.
I see three options for your situation:
Keep your existing design
Use OPENROWSET in the SP
Filestream
Item one is best if the files are remote from the server and have unique names.
Item two will take the work off the c# side but you will have to deal with read permissions for SQL on the file, naming convention and file cleanup.
Item three is the current best pratice for SQL Server 2008. There are numerous HOW2 articles to follow. This choice will allow SQL to manage the file while keeping the file on NTFS storage