I have been trying to load csv data into Snowflake using COPY INTO command
This is the sample data
4513194677~"DELL - ULTRASHARP 32\" MONITOR 4K U3223QE"~""~""
I have tried using below COPY INTO syntax
file_format =
type = 'csv'
field_delimiter = '~'
skip_header = 1
record_delimiter = '\\n'
field_optionally_enclosed_by = '"'
ESCAPE = 'NONE'
ESCAPE_UNENCLOSED_FIELD = 'NONE'
However, getting this error "Found character 'M' instead of field delimiter '~'"
How can I escape the " and load the columns data as DELL - ULTRASHARP 32 " MONITOR 4K U3223QE
If I try to use ESCAPE, I get below error when running the COPY command
[ERROR] ProgrammingError: 001003 (42000): 01a8e01d-3201-36a9-0050-4502537cfc7f: SQL compilation error:
syntax error line 15 at position 43 unexpected '''.
syntax error line 20 at position 20 unexpected ')'.
file_format =
type = 'csv'
field_delimiter = '~'
skip_header = 1
record_delimiter = '\\n'
field_optionally_enclosed_by = '"'
ESCAPE = '\\'
ESCAPE_UNENCLOSED_FIELD = '\\'
Try using two double quotes in the data instead of one without trying to escape the double quote
Data similar to "sample"
You can have your csv formated like below
"Data similar to ""sample"""
Related
df <- data.frame(category = c("A","B","A","D","E"),
date = c("5/10/2005","6/10/2005","7/10/2005","8/10/2005","9/10/2005"),
col1 = c(1,NA,2,NA,3),
col2 = c(1,2,NA,4,5),
col3 = c(2,3,NA,NA,4))
I have to insert a data frame that is created in R to mysql server.
I have tried these methods(Efficient way to insert data frame from R to SQL). However, my data also has NA which are fails the whole process of exporting.
Is there a way around to faster upload to data.
dbWriteTable(cn,name ="table_name",value = df,overwrite=TRUE, row.names = FALSE)
The above works but is very slow to upload
The method that I have to use is this :
before = Sys.time()
chunksize = 1000000 # arbitrary chunk size
for (i in 1:ceiling(nrow(df)/chunksize)) {
query = paste0('INSERT INTO dashboard_file_new_rohan_testing (',paste0(colnames(df),collapse = ','),') VALUES ')
vals = NULL
for (j in 1:chunksize) {
k = (i-1)*chunksize+j
if (k <= nrow(df)) {
vals[j] = paste0('(', paste0(df[k,],collapse = ','), ')')
}
}
query = paste0(query, paste0(vals,collapse=','))
dbExecute(cn, query)
}
time_chunked = Sys.time() - before
Error Encountered:
Error in .local(conn, statement, ...) :
could not run statement: Unknown column 'NA' in 'field list'
One of the fastest ways to load data into MySQL is to use its LOAD DATA command line tool. You may try first writing your R data frame to a CSV file, then using MySQL's LOAD DATA to load it:
write.csv(df, "output.csv", row.names=FALSE)
Then from your command line, use:
LOAD DATA INFILE 'output.csv' INTO TABLE table_name
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES;
Note that this assumes the CSV file is already on the same machine as MySQL. If not, and you have it still locally, then use LOAD DATA LOCAL INFILE instead.
You may read MYSQL import data from csv using LOAD DATA INFILE for more help using LOAD DATA.
Edit:
To deal with the issue of NA values, which should represent NULL in MySQL, you may take the approach of first casting the entire data frame to text, and then replacing the NA values with empty string. LOAD DATA will interpret a missing value in a CSV column as being NULL. Consider this:
df <- data.frame(lapply(df, as.character), stringsAsFactors=FALSE)
df[is.na(df)] <- ""
Then, use write.csv along with LOAD DATA as described above.
I was trying to load trans_dt into cmdels table. but it throws syntax error. How to add local variable into table. The Local variable name is newDate
import mysql.connector
config = {
'user':'root',
'password':'password',
'host':'127.0.0.1',
'database':'nse'
}
conn= mysql.connector.connect(**config)
c=conn.cursor()
def insertRows(fileName,c):
delimiter=r','
dateString=r'%d-%b-%Y'
file=fileName.split("/")[-1]
if file.startswith("MTO"):
newDate=new_Date(file)
c.execute("Load data local infile %s into table cmdels fields terminated by %s
ignore 4 lines(recno,srno,symbol,series,qtytrd,qtydel,qtyper,trans_dt)
SET trans_dt=(#trans_dt,%s)", (fileName, delimiter,newDate))
localExtractFilePath="/Users/data/nse"
import os
def new_Date(s):
return s[4:-4]
for file in os.listdir(localExtractFilePath):
if file.endswith(".DAT"):
insertRows(localExtractFilePath+"/"+file,c)
print "Loaded file "+file+" into database"
conn.commit()
c.close()
conn.close()
Single quoted string literals can't span newlines. You can use triple-quoted strings ("""Load data...""") but you end up with extra whitespace and newlines in the string, which can cause their own problems. A good option is to end each line with an end quote and let python concatenate them in the compilation step
c.execute("Load data local infile %s into table cmdels fields terminated by %s"
" ignore 4 lines(recno,srno,symbol,series,qtytrd,qtydel,qtyper,trans_dt)"
" SET trans_dt=(#trans_dt,%s)", (fileName, delimiter,newDate))
I'm trying to import data from a .csv file and I'm getting and error code 1193 unknown system variable. I'm utilizing MySQL 5.5.34.
LOAD DATA LOCAL INFILE 'path to the file/student_2.csv'
INTO TABLE STUDENT
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 2 LINES
(S_ID, S_LAST, S_FIRST, S_MI, S_ADDRESS, S_CITY, S_STATE, S_ZIP, S_PHONE, S_CLASS, #S_DOB, S_PIN, F_ID, #DATE_ENROLLED);
SET S_DOB = STR_TO_DATE(#S_DOB, '%m/%d/%Y'),
DATE_ENROLLED = STR_TO_DATE(#DATE_ENROLLED, '%m/%d/%Y');
The csv file's data is as follows:
S_ID,S_LAST,S_FIRST,S_MI,S_ADDRESS,S_CITY,S_STATE,S_ZIP,S_PHONE,S_CLASS,S_DOB,S_PIN,F_ID,DATE_ENROLLED
Number,String,String,String,String,String,String,String,String,String,Date/Time,String,Number,String
1,Joffs,Tami,R,1817 Eagldge Cle,Houston,TX,74027,356487654,SR,7/14/88,8891,1,1/3/13
2,Petez,Jimmge,C,951 Drainbow Place,Absail,TX,76901,3253945432,SR,18/09/76,1230,1,11/10/02
3,Marks,Johannes,A,1015 Wild St,Dallas,TX,71012,3251454321,JR,08/13/83,1613,1,8/24/03
4,Smyth,Mark,,428 EN 16 Plaza,Arsehole,TX,7012,3221143210,SO,1/14/88,1841,2,8/23/04
I also change the year format from %Y to %y and did not work either.
It is something wrong with the script?
Hm - I can't try it out and I did not dive deep into your script, but are you sure about the ; before the set-commands?
....
(S_ID, S_LAST, S_FIRST, S_MI, S_ADDRESS, S_CITY,
S_STATE, S_ZIP, S_PHONE, S_CLASS, #S_DOB, S_PIN, F_ID, #DATE_ENROLLED);
SET S_DOB = STR_TO_DATE(#S_DOB, '%m/%d/%Y'),
DATE_ENROLLED = STR_TO_DATE(#DATE_ENROLLED, '%m/%d/%Y');
In the syntax of load infile data i saw that the fields and line clauses are optional. So I used only character set clause for utf8
Here my sql:
cmd = new MySqlCommand("LOAD DATA INFILE " + filename + " INTO TABLE " + tblname + " CHARACTER SET 'UTF8'", conn);
filename is the addresse it's format is: "E:\Macdata\20131228\atelier.sql"
table name is directly taken from database is as : "atelier"
But I get the error : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'E:\Macdata\20131228\atelier.sql INTO TABLE atelier CHARACTER SET 'UTF8'' at line 1
What is the mistake in my query command ?
MYSQLversion is 5.0.10 with XAMPP
After changing the query I begin to receive fatal error number 0 (enclosed filename with ')
cmd = new MySqlCommand("LOAD DATA LOCAL INFILE '" + filename + "' IGNORE INTO TABLE " + tblname + " CHARACTER SET UTF8", conn);
My data file has this form which works on phpmyadmin
INSERT INTO `atelier` VALUES(1, 'Chateau Carbonnieux -1', '2013-12-26', 23, 10, 0, '4 macarons differents', 'mamie', '2013-12-15 11:09:14', 'sabrina', '2013-12-18 05:29:26');
As the error says, your statements is wrong. Quotes are missing in your first statement (see second statement). Check the syntax here:
http://dev.mysql.com/doc/refman/5.6/en/load-data.html
Some sparse notes:
0 is not a fatal error, it's the code for success.
IGNORE handles duplicate rows, not syntax errors.
I am writing a script to read data from a file by lines and insert each line into MySQL database. I use mysql.connector to do this. Here is a piece of script.
def insert_query(data):
return ("INSERT INTO " + tblname + " (`log`) " + "VALUES " + "(" + "'" + data + "'" + ")")
with open('data.txt', 'r') as f:
lines = f.readlines()
for line in lines:
add_line = insert_query(line)
cursor.execute(add_line)
cursor.commit()
File data.txt has size is 5Mbyte, but it has about 10000 lines.
tblname has 2 field: ID - INT (11) (auto-increment) , log - TEXT
When i run this script, it add to database about 100 lines and crashed. It report a error:
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an
error in your SQL syntax; check the manual that corresponds to your
MySQL server version for ')'
MySQL version : 5.5.27
How to solve this problem ? Thanks.
Your insert statement is incorrect: 'tblname' is undefined in your function, hence the syntax error. However, there are bigger issues with how you solve it: what if a line in the log has quotes and brackets?
The following code show how to read data from a file and insert it line by line:
stmt = "INSERT INTO {table} (c1) VALUES (%s)".format(table=table_name)
with open('data.txt', 'r') as fp:
for line in fp:
data = (line.strip(),)
cur.execute(stmt, (line.strip(),))
cnx.commit()
Or using executemany(), which would be faster:
with open('data.txt', 'r') as fp:
cur.executemany(stmt, [(line.strip(),) for line in fp])
cnx.commit()