I have recently encountered an error while working with bcp.
Here is the error.
SQLState = 22001, NativeError = 0 Error = [Microsoft][SQL Server
Native Client 10.0]String data, right truncation
I'm trying to unpack the data into a staging table which does not have any constraints and the datatypes are also fairly large when compared to the data. I have about 11 files from different tables being bcp'd and zipped out of which only one file when unpacking errors out.
This is the command which I have been using succesfully. Very recently(when trying to make a copy of the current WH and settign up the process) I have been facing issues.
bcp.exe employee_details in employee_details.dat -n -E -S "servername"
-U sa -P "Password"
I have tried changing the commands to -C -T -S which worked when I gave the format manually. This is a very big and important packet I need to load in to my WH.
I don't know if I see a format file here or not.
Any help is needed.
Thanks
Cinnamon girl.
We also faced same issue while doing BCP and it turned out to be an issue with new line character in .dat file.
View the file in Notepad++ and click on "Show All Characters" to see the new line character.
BCP throws following error with -r "\r\n" option i.e. with below command
bcp dbo.Test in C:\Test.dat -c -t "|" -r "\r\n" -S "DBServerName" -T -E
" SQLState = 22001, NativeError = 0 Error = [Microsoft][SQL Server
Native Client 10.0]String data, right truncation "
BCP treat all rows in file as a single row with -r "\n" or -r "\r" option i.e. with below command
bcp dbo.Test in C:\Test.dat -c -t "|" -r "\n" -S "DBServerName" -T -E
Issue was resolved when we used the Haxadecimal value (0x0a) for New Line character in BCP command
bcp dbo.Test in C:\Test.dat -c -t "|" -r "0x0a" -S "DBServerName" -T -E
bcp right truncation error occurs when there is too much data that can be fitted into a single column.
This can be caused by improper format files(if any being used) or delimiter.
The line terminator (Windows has CRLF or '\r\n' and UNIX has '\n') can also cause this error. Example Your format file contains Windows CRLF ie, '\r\n' as the row terminator but the file contains '\n' as line endings. This would mean fitting the whole file in 1 row(rather 1 column) which leads to right truncation error.
I was also getting the truncation message. After hours of searching forums and trying suggested solutions I finally got my load to work.
The reason for the truncation message was because I was gullible enough to think that putting the column name in the format file actually mattered. It's the preceding numeric that appears to dictate where the data gets loaded.
My input file did not have data for the third column in the table. So this is how my format file looked.
... "," 1 Cust_Name SQL_Latin1...
... "," 2 Cust_Ref SQL_Latin1...
... "," 3 Cust_Amount SQL_Latin1...
... "\r\n" 4 Cust_notes SQL_Latin1...
My input file looked like this:
Jones,ABC123,200.67,New phone
Smith,XYZ564,10.23,New SIM
The table looked like
Cust_Name Varchar(20)
Cust_Ref Varchar(10)
Cust_Type Varchar(3)
Cust_amount Decimal(10,2)
Cust_Notes Varchar (50)
Cust_Tel Varchar(15)
Cust......
I'd assumed by giving the column name in the format file that the data would go into the appropriate column on the table.
This however works as the column number is important and the column name is noise.
... "," 1 A SQL_Latin1...
... "," 2 B SQL_Latin1...
... "," 4 C SQL_Latin1...
... "\r\n" 5 D SQL_Latin1...
For us it turned out that the file we were trying to upload was in Unicode instead of ANSI format.
There is a -N switch, but our tables didn't have any NVARCHAR data.
We just saved the file in ANSI format and it worked, but if you have NVARCHAR data or you may need to use the -N switch
See TechNet - Using Unicode Native Format to Import or Export Data
I know this is old - but I just came across an instance where I was getting this error, turns out one of my numeric fields had more decimals that was allowed by the schema.
In my case the reason was that in one field there was written "|" = chr$(124) and the separator was in my case "|" = chr$(179).
MS SQL to not make a difference between both characters. I eliminated the chr$(124) and then the import by BCP works fine.
Open the files in notepad++. GO to View tab->show symbols->show all characters. I was also facing the same issue in .tsv files.one tab was misplaced.
Late, but still: In my case I exactly got this one
SQLState = 22001, NativeError = 0
Error = [Microsoft][ODBC Driver 11 for SQL Server]String data, right truncation
And the problem was that the schema changed. The target database had two new fields. Once I installed the previous schema, the import succeeded.
After spending 4 hrs, doing a ton of trail and error, I found that the solution can be as simple as the table where you are importing the data to should have a suitable schema for the file that you trying to import.
ex: In my case. I was importing a .csv with 667,aaa,bbb into a table that has a schema of
int(4),char(2),char(2) causing String Data, Right Truncation.
My bcp was ignoring any of those newline characters like \r, \n, \r\n\, 0x0d, 0x0a, 0x0d0x0a, etc.. The only sulution I found was to include "real" newline directly into the bcp command. I think this works because the csv was generated on the same server as the bcp is running on. When I transfer the csv to the mssql server manually then 0x0a works as well inside BULK INSERT.
Please note, that nl1=^ must be followed by two new blank lines.
my_script.bat:
#echo off
setlocal enableDelayedExpansion
set nl=^
set cmd=bcp db_name.db_schema.my_table in stats.csv -w -t, -r "!nl!" -S my_server -U my_username -P password123
!cmd!
Related
I've been trying to import a text file into mysql, but with no luck. I get the following error when running this command
LOAD DATA LOCAL INFILE '/file' INTO TABLE test COLUMNS TERMINATED BY ':';
ERROR 1300 (HY000): Invalid utf8mb4 character string: ''
I found a command that was shared on here that shows me the lines with the bad characters.
grep -axv '.*' filename
11241321:▒oy▒_moley#zmal.com
11241771:▒om▒_shead#zmal.com
11241811:▒eva▒_gould#zmal.com
Now that I can tell which lines the bad characters are on then I could go delete the bad characters manually, but that's a pain because these files are potentially huge. I wonder if there is a way to find those bad characters and then delete them using the linux command line. I would like to keep the line intact, but just remove the bad character.
Remove characters that are not within the ascii table (11,12,40-176)
\11 = tab
\12 = new line
\40-176 = ( to ~ this range includes all letters and symbols present in the keyboard
cat test.txt | tr -cd '\11\12\40-\176' > temp && mv temp test.txt
NOTE: If your data has special characters that are not in the ascii table, they might be removed as well
After trying for a full day, I'm hoping someone here can help me make below script work. I've combined information from multiple threads (example) and websites, but can't get it to work.
What I'm trying to do:
I'm trying to get a MariaDB10 database called 'stock_db' on my Synology NAS to load all *.csv files from a specific folder (where I save downloaded historical prices of stocks) and add these to a table called 'prices'. The files are all equally named "price_history_'isin'.csv".
Below SQL statement works when running it individually from HeidiSQL on my Windows machine:
Working SQL
LOAD DATA LOW_PRIORITY LOCAL INFILE 'D:\\Downloads\\price_history_NL0010366407.csv'
IGNORE INTO TABLE `stock_db`.`prices`
CHARACTER SET utf8
FIELDS TERMINATED BY ';'
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 2 LINES
(#vdate, #vprice)
SET
isin = 'NL0010366407',
date = STR_TO_DATE(#vdate, '%d-%m-%Y'),
price = #vprice
;
The issue
Unfortunately, when I try to batch loading all csv's from a folder on my NAS through below script, I keep getting the same error.
#!/bin/bash
for filename in ./price_history/*.csv; do
echo $filename
isin=${filename:30:12}
echo $isin
/volume1/#appstore/MariaDB10/usr/local/mariadb10/bin/mysql -u root -p \
"LOAD DATA LOW_PRIORITY LOCAL INFILE '$filename'\
IGNORE INTO TABLE 'stock_db.prices'\
CHARACTER SET utf8\
FIELDS TERMINATED BY ';'\
OPTIONALLY ENCLOSED BY '"'"'"'\
ESCAPED BY '"'"'"'\
LINES TERMINATED BY '\r\n'\
IGNORE 2 LINES (#vdate, #vprice)\
SET\
isin = '$isin',\
date = STR_TO_DATE(#vdate, '%d-%m-%Y'),\
price = #vprice;"
done
ERROR 1102 (42000): Incorrect database name
What I've tried
Took the database name out of stock_db.prices and mentioned it separately as [database] outside of the quoted SQL statement - Doesn't work
Changed quotes around 'stock_db.prices' in many different ways - Doesn't work
Separated the SQL into a separate file and referenced it '< stmt.sql' - Complicates things even further and couldn't get it to work at all (although probably preferred)
Considered (or even preferred) using a PREPARE statement, but seems I can't use this in combination with LOAD DATA (reference)
Bonus Question
If someone can help me do this without having to re-enter the user's password or putting the password in the script, this would be really nice bonus!
Update
Got the 'Incorrect Database Error' resolved by adding '-e' option
Now I have a new error on the csv files:
ERROR 13 "Permission Denied"
While the folder and files are full access for everyone.
Anyone any thoughts to this?
Thanks a lot!
Try to set database using -D option: change the first line to
/volume1/#appstore/MariaDB10/usr/local/mariadb10/bin/mysql -D stock_db -u root -p \ ...
You may have an error in this line IGNORE INTO TABLE 'stock_db.prices'\ - try to remove the single quotes.
Create file .my.cnf in your user's home directory and put the following information into it:
[client]
password="my password"
Info about option files.
'stock_db.prices'
Incorrect quoting. This will work since neither are keywords:
stock_db.prices
This will also work:
`stock_db`.`prices`
Note that the db name and the table name are quoted separately, using backtics.
I can't predict what will happen with this nightmare:
'"'"'"'
I have a strange problem. I am using the below code to import a bunch of csv files into MySQL database.
#ECHO OFF
setlocal enabledelayedexpansion
IF /I "%CD%\" NEQ "%~dp0" PUSHD "%~dp0"
mysql -e "DELETE FROM software_it.hardware" -u root
FOR %%f IN ("*.csv") DO (
set old=%%~dpnxf
set new=!old:\=\\!
mysql -e "load data local infile '"!new!"' into table software_it.hardware COLUMNS TERMINATED BY ',' IGNORE 1 ROWS" -u root
echo %%~nxf DONE
)
To start off with, the csv file is coming from a CMD output.
I have a field called "UserName" and I have observed the problem only in that field, starngely.
The username has the format :
Trilegal\{username}
Now, firstly after the import, the "\" is missing and some of the names have the first charecter missing after the "\".
I am guessing something is wrong because of the presence of this "\". I thought the problem was with my table and hence I used the GUI way of importing. The GUI way imported fine. I used the same table to import using the above code and the same problem is back again.
I am kind of new to MySQL so if anyone can help me understand the problem, it'd be great.
EDIT: Changed to "\" from "/"
Thanks,
SS.
The answer is escape character, by default mySQL treats "\" as an escape character.
Add a line which says ESCAPED BY '' in your code and it will not use the default escape parameters.
P.S: Yes I know I am answering myself. Came to this answer after some reading. :D
What I'm trying to do is export a view/table from Sybase ASE 12.0 into a CSV file, but I'm having a lot of difficulty in it.
We want to import it into IDEA or MS-Access. The way that these programs operate is with the text-field encapsulation character and a field separator character, along with new lines being the record separator (without being able to modify this).
Well, using bcp to export it is ultimately fruitless with its built in options. It doesn't allow you to define a text field encapsulation character (as far as I can tell). So we tried to create another view that reads from the other view/table that concatenates the fields that have new lines in them (text fields), however, you may not do that without losing some of the precision because it forces the field into a varchar of 8000 characters/bytes, of which our max field used is 16000 (so there's definitely some truncation).
So, we decided to create columns in the new view that had the text field delimiters. However, that put our column count for the view at 320 -- 70 more than the 250 column limit in ASE 12.0.
bcp can only work on existing tables and views, so what can we do to export this data? We're pretty much open to anything.
If its only the new line char that is causing problems can you not just do a replace
create new view as
select field1, field2, replace(text_field_with_char, 'new line char,' ' ')
from old_view
You may have to consider exporting as 2 files, importing into your target as 2 tables and then combining them again in the target. If both files have a primary key this is simple.
That sounds like bcp's right, but process the output via awk or perl.
But are those things you have and know? That might be a little overhead for you.
If you're on Windows you can get Active Perl free and it could be quick.
something like:
perl -F, -lane 'print "\"$F[0]\",$F[1],\"$F[2]\",$F[3]\n" ;' bcp-output-file
how's that? $F is an array of fields. The text ones you encircle with \"
You can use BCP format files for this.
bcp .... -f XXXX.fmt
BCP can also produce this format files interactively if you don't state
any of -c -n -f flags. Then you can save the format file and experiment with it, editing it and runnign BCP.
To safe time while exporting and debugging, use -F -L flags like "-F 1 -L 10" -- this gets only first 10 lines.
What's the easiest way to get the data for a single table, delete a single table or break up the whole dump file into files each containing individual tables? I usually end up doing a lot of vi regex munging, but I bet there are easier ways to do these things with awk/perl, etc. The first page of Google results brings back a bunch of non-working perl scripts.
When I need to pull a single table from an sql dump, I use a combination of grep, head and tail.
Eg:
grep -n "CREATE TABLE" dump.sql
This then gives you the line numbers for each one, so if your table is on line 200 and the one after is on line 269, I do:
head -n 268 dump.sql > tophalf.sql
tail -n 69 tophalf.sql > yourtable.sql
I would imagine you could extend upon those principles to knock up a script that would split the whole thing down into one file per table.
Anyone want a go doing it here?
Another bit that might help start a bash loop going:
grep -n "CREATE TABLE " dump.sql | tr ':`(' ' ' | awk '{print $1, $4}'
That gives you a nice list of line numbers and table names like:
200 FooTable
269 BarTable
Save yourself a lot of hassle and use mysqldump -T if you can.
From the documentation:
--tab=path, -T path
Produce tab-separated data files. For each dumped table, mysqldump
creates a tbl_name.sql file that contains the CREATE TABLE statement
that creates the table, and a tbl_name.txt file that contains its
data. The option value is the directory in which to write the files.
By default, the .txt data files are formatted using tab characters
between column values and a newline at the end of each line. The
format can be specified explicitly using the --fields-xxx and
--lines-terminated-by options.
Note This option should be used only when mysqldump is run on the
same machine as the mysqld server. You must have the FILE privilege,
and the server must have permission to write files in the directory
that you specify.
This shell script will grab the tables you want and pass them to splitted.sql.
It’s capable of understanding regular expressions as I’ve added a sed -r option.
Also MyDumpSplitter can split the dump into individual table dumps.
Maatkit seems quite appropriate for this with mk-parallel-dump and mk-parallel-restore.
I am a bit late on that one, but if it can help anyone, I had to split a huge SQL dump file in order to import the data to another Mysql server.
what I ended up doing was splitting the dump file using the system command.
split -l 1000 import.sql splited_file
The above will split the sql file every 1000 lines.
Hope this helps someone