I have this data in my file (sometimes fields could be in double quotes) as shown below:
1, "ala", johnes, 2017-09-01, 100
2, dqwdqdq,, 2017-09-01, 101
Data is loaded to database however quotes are also inserted into column so instead to see ala i see "ala" in the column.
My code:
Using exclecon As New MySqlConnection("Server=Localhost;DataBase=mobily;user=root;password=897;")
Dim uploadQry As String = "Load DATA LOCAL INFILE 'C:/Users/Robert/Desktop/test.csv' INTO TABLE Test FIELDS TERMINATED BY ',' ENCLOSED BY '""' ESCAPED BY '' LINES TERMINATED BY '\r\n' IGNORE 0 LINES;"
Dim myCUpload As New MySqlCommand(uploadQry, exclecon)
exclecon.Open()
myCUpload.ExecuteNonQuery()
End Using
Related
I have been trying to load csv data into Snowflake using COPY INTO command
This is the sample data
4513194677~"DELL - ULTRASHARP 32\" MONITOR 4K U3223QE"~""~""
I have tried using below COPY INTO syntax
file_format =
type = 'csv'
field_delimiter = '~'
skip_header = 1
record_delimiter = '\\n'
field_optionally_enclosed_by = '"'
ESCAPE = 'NONE'
ESCAPE_UNENCLOSED_FIELD = 'NONE'
However, getting this error "Found character 'M' instead of field delimiter '~'"
How can I escape the " and load the columns data as DELL - ULTRASHARP 32 " MONITOR 4K U3223QE
If I try to use ESCAPE, I get below error when running the COPY command
[ERROR] ProgrammingError: 001003 (42000): 01a8e01d-3201-36a9-0050-4502537cfc7f: SQL compilation error:
syntax error line 15 at position 43 unexpected '''.
syntax error line 20 at position 20 unexpected ')'.
file_format =
type = 'csv'
field_delimiter = '~'
skip_header = 1
record_delimiter = '\\n'
field_optionally_enclosed_by = '"'
ESCAPE = '\\'
ESCAPE_UNENCLOSED_FIELD = '\\'
Try using two double quotes in the data instead of one without trying to escape the double quote
Data similar to "sample"
You can have your csv formated like below
"Data similar to ""sample"""
df <- data.frame(category = c("A","B","A","D","E"),
date = c("5/10/2005","6/10/2005","7/10/2005","8/10/2005","9/10/2005"),
col1 = c(1,NA,2,NA,3),
col2 = c(1,2,NA,4,5),
col3 = c(2,3,NA,NA,4))
I have to insert a data frame that is created in R to mysql server.
I have tried these methods(Efficient way to insert data frame from R to SQL). However, my data also has NA which are fails the whole process of exporting.
Is there a way around to faster upload to data.
dbWriteTable(cn,name ="table_name",value = df,overwrite=TRUE, row.names = FALSE)
The above works but is very slow to upload
The method that I have to use is this :
before = Sys.time()
chunksize = 1000000 # arbitrary chunk size
for (i in 1:ceiling(nrow(df)/chunksize)) {
query = paste0('INSERT INTO dashboard_file_new_rohan_testing (',paste0(colnames(df),collapse = ','),') VALUES ')
vals = NULL
for (j in 1:chunksize) {
k = (i-1)*chunksize+j
if (k <= nrow(df)) {
vals[j] = paste0('(', paste0(df[k,],collapse = ','), ')')
}
}
query = paste0(query, paste0(vals,collapse=','))
dbExecute(cn, query)
}
time_chunked = Sys.time() - before
Error Encountered:
Error in .local(conn, statement, ...) :
could not run statement: Unknown column 'NA' in 'field list'
One of the fastest ways to load data into MySQL is to use its LOAD DATA command line tool. You may try first writing your R data frame to a CSV file, then using MySQL's LOAD DATA to load it:
write.csv(df, "output.csv", row.names=FALSE)
Then from your command line, use:
LOAD DATA INFILE 'output.csv' INTO TABLE table_name
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES;
Note that this assumes the CSV file is already on the same machine as MySQL. If not, and you have it still locally, then use LOAD DATA LOCAL INFILE instead.
You may read MYSQL import data from csv using LOAD DATA INFILE for more help using LOAD DATA.
Edit:
To deal with the issue of NA values, which should represent NULL in MySQL, you may take the approach of first casting the entire data frame to text, and then replacing the NA values with empty string. LOAD DATA will interpret a missing value in a CSV column as being NULL. Consider this:
df <- data.frame(lapply(df, as.character), stringsAsFactors=FALSE)
df[is.na(df)] <- ""
Then, use write.csv along with LOAD DATA as described above.
I'm trying to import a csv file that is over 5 GB into mysql in a created table but it isn't importing despise showing success message:
LOAD DATA LOCAL INFILE
"path/to/this/file/StockEtablissement_utf8.csv"
INTO TABLE entreprises COLUMNS TERMINATED BY ',' LINES TERMINATED BY '\n'(
siren,
nic,
siret,
statutDiffusionEtablissement,
dateCreationEtablissement,
trancheEffectifsEtablissement,
anneeEffectifsEtablissement,
activitePrincipaleRegistreMetiersEtablissement,
dateDernierTraitementEtablissement,
etablissementSiege,
nombrePeriodesEtablissement,
complementAdresseEtablissement,
numeroVoieEtablissement,
indiceRepetitionEtablissement,
typeVoieEtablissement,
libelleVoieEtablissement,
codePostalEtablissement,
libelleCommuneEtablissement,
libelleCommuneEtrangerEtablissement,
distributionSpecialeEtablissement,
codeCommuneEtablissement,
codeCedexEtablissement,
libelleCedexEtablissement,
codePaysEtrangerEtablissement,
libellePaysEtrangerEtablissement,
complementAdresse2Etablissement,
numeroVoie2Etablissement,
indiceRepetition2Etablissement,
typeVoie2Etablissement,
libelleVoie2Etablissement,
codePostal2Etablissement,
libelleCommune2Etablissement,
libelleCommuneEtranger2Etablissement,
distributionSpeciale2Etablissement,
codeCommune2Etablissement,
codeCedex2Etablissement,
libelleCedex2Etablissement,
codePaysEtranger2Etablissement,
libellePaysEtranger2Etablissement,
dateDebut,
etatAdministratifEtablissement,
enseigne1Etablissement,
enseigne2Etablissement,
enseigne3Etablissement,
denominationUsuelleEtablissement,
activitePrincipaleEtablissement,
nomenclatureActivitePrincipaleEtablissement,
caractereEmployeurEtablissement
);
screenshot of the success message :mysql
the mysql version is mysql-5.7.24-winx64
the csv is located here http://files.data.gouv.fr/insee-sirene/StockEtablissement_utf8.zip
I'm trying to import data from a .csv file and I'm getting and error code 1193 unknown system variable. I'm utilizing MySQL 5.5.34.
LOAD DATA LOCAL INFILE 'path to the file/student_2.csv'
INTO TABLE STUDENT
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 2 LINES
(S_ID, S_LAST, S_FIRST, S_MI, S_ADDRESS, S_CITY, S_STATE, S_ZIP, S_PHONE, S_CLASS, #S_DOB, S_PIN, F_ID, #DATE_ENROLLED);
SET S_DOB = STR_TO_DATE(#S_DOB, '%m/%d/%Y'),
DATE_ENROLLED = STR_TO_DATE(#DATE_ENROLLED, '%m/%d/%Y');
The csv file's data is as follows:
S_ID,S_LAST,S_FIRST,S_MI,S_ADDRESS,S_CITY,S_STATE,S_ZIP,S_PHONE,S_CLASS,S_DOB,S_PIN,F_ID,DATE_ENROLLED
Number,String,String,String,String,String,String,String,String,String,Date/Time,String,Number,String
1,Joffs,Tami,R,1817 Eagldge Cle,Houston,TX,74027,356487654,SR,7/14/88,8891,1,1/3/13
2,Petez,Jimmge,C,951 Drainbow Place,Absail,TX,76901,3253945432,SR,18/09/76,1230,1,11/10/02
3,Marks,Johannes,A,1015 Wild St,Dallas,TX,71012,3251454321,JR,08/13/83,1613,1,8/24/03
4,Smyth,Mark,,428 EN 16 Plaza,Arsehole,TX,7012,3221143210,SO,1/14/88,1841,2,8/23/04
I also change the year format from %Y to %y and did not work either.
It is something wrong with the script?
Hm - I can't try it out and I did not dive deep into your script, but are you sure about the ; before the set-commands?
....
(S_ID, S_LAST, S_FIRST, S_MI, S_ADDRESS, S_CITY,
S_STATE, S_ZIP, S_PHONE, S_CLASS, #S_DOB, S_PIN, F_ID, #DATE_ENROLLED);
SET S_DOB = STR_TO_DATE(#S_DOB, '%m/%d/%Y'),
DATE_ENROLLED = STR_TO_DATE(#DATE_ENROLLED, '%m/%d/%Y');
i want to import csv file. my script is :
#echo off
set numid=2015092510524361378197540100
sqlldr USER#db/PSW data=csv\2015092510524361378197540100.csv control=ctl\control.ctl log=log\2015092510524361378197540100.log bad=bad\id.bad
pause
my table is :
CREATE TABLE SV (NO1 VARCHAR2(255),NAMA VARCHAR2(255),ALAMAT VARCHAR2(255),id VARCHAR2(20),JAB VARCHAR2(50),numid VARCHAR(55));
my control.ctl is :
OPTIONS (SKIP=43, errors=12000) LOAD DATA APPEND INTO TABLE sv when NAMA <> '' FIELDS TERMINATED BY ',' optionally enclosed by '"' TRAILING NULLCOLS (no filler,no1 "TRIM (:no1)",nama "TRIM (:nama)", alamat "TRIM (:alamat)",id "TRIM (:id)",jab "TRIM (:jab)",numid "%numid%")
error is :
Record 10: Rejected - Error on table sV, column numid.
ORA-00911: invalid character
please let me know which one is wrong. thanks All
Since numid is a VARCHAR2(55), change the table datatype to be VARCHAR2 and the control file to read:
numid "TRIM(:numid)"