I have a dataset as follows,
Date,Time,ID,Name,Count
01-MAY-2009,00:00,4,Town Hall (West),209
01-MAY-2009,02:00,17,Collins Place (South),28
01-MAY-2009,23:00,18,Collins Place (North),36
For this I have created table with following schema,
CREATE TABLE table_name(date DATE, time TIME, id int, name VARCHAR(50), count int);
And for loading the table from ".csv" file as,
LOAD DATA INFILE '/home/cloudera/dataset.csv'
INTO TABLE table_name
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(#var1, #var2, id, name, count)
SET date = STR_TO_DATE(#var1, '%d-%b-%Y')
SET time = TIME(#var2, "%H:%i");
But I get an error as,
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'SET time = TIME(#var2, "%H:%i")' at line 1
I can't understand what the error is. I have tried going through MySQL website and documentation, but can't make out what the correct format is. Can anyone please me. Thanks in advance.
I don't think that you even need to be using the TIME function here. Your current hour:minute string time literals should be good enough, q.v. the documentation:
Be careful about assigning abbreviated values to a TIME column. MySQL interprets abbreviated TIME values with colons as time of the day. That is, '11:12' means '11:12:00', not '00:11:12'
As valid literals, your times would be interpreted as having a zero second component. So, if you just insert those time strings as is, I think it should work. Try the following code:
LOAD DATA INFILE '/home/cloudera/dataset.csv'
INTO TABLE table_name
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(#var1, Time, id, name, count)
SET date = STR_TO_DATE(#var1, '%d-%b-%Y');
Tim is right in pointing out that you do not need to convert your time data in the load data infile statement.
Just to answer why you get a syntax error: load data infile can only have a single set clause, in which assignments to multiple columns are separated by comma. Your code has multiple set clauses, hence it fails.
Also, the time() function does not have a 2nd parameter for a pattern. The function you need to use is called str_to_date().
So, it should look like as follows:
...
SET date = STR_TO_DATE(#var1, '%d-%b-%Y'), time = TIME(str_to_date(#var2, "%H:%i"));
Related
I've been trying to learn SQL using python to update a db and am trying to do something simple. Iterate through a csv file that includes the fortune 500 with their revenue info and push into an SQL db. I've run it a few times and it's working great, the only issue is I'm getting duplicates because I've run the same file a few times.
In the future, I'm assuming it's good to learn how to avoid duplicates. After looking around this is what I've found for a proposed solution using WHERE NOT EXISTS but am getting an error. Any advice is welcome as I'm totally new.
Note - I do know I should be updating more than one row at a time, that's my next lesson
import pymysql
import csv
with open('companies.csv','rU') as f:
reader = csv.DictReader(f)
for i in reader:
conn = pymysql.connect(host='host', user='user', passwd='pw', db='db_test')
cur = conn.cursor()
query1 = "INSERT INTO companies (Name, Revenue, Profit, Stock_Price) VALUES (\'{}\',{},{},{})".format(str(i['Standard']),float(i['Revenues']),float(i['Profits']),float(i['Rank']))
query2 = 'WHERE NOT EXISTS (SELECT Name FROM companies WHERE Name = \'{}\')'.format(str(i['Standard']))
query = query1+' '+query2
cur.execute(query)
conn.commit()
cur.close()
OUTPUT:
INSERT INTO companies (Name, Revenue, Profit, Stock_Price) VALUES ('WalMart Stores',469.2,16999.0,1.0) WHERE NOT EXISTS (SELECT Name FROM companies WHERE Name = 'WalMart Stores')
ERROR:
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'WHERE NOT EXISTS (SELECT Name FROM companies WHERE Name = 'WalMart Stores')' at line 1")
Ok. First of all, congratulations on self-learning!
Now, to the point.
When you use insert ... values, you can't define a where condition for the table on which you're inserting values. insert statement is only used to insert (When you use insert... select, you can define a where condition on the select, not on the table on which you're about to insert values).
So, there are two ways to do what you want:
Create a unique index on the column that you want to test, and then use insert ignore...
In your code, check if the value is already there, and if it's not, then insert it.
I'll tell you how to work with the first suggestion, because it'll teach you a couple of things. As for suggestion 2, I'll leave that for you as homework ;-)
First, you need to add a unique index to your table. If you want to avoid duplicates on the Name column, then:
alter table companies
add unique index idx_dedup_name(Name);
Check the syntax for ALTER TABLE.
And now, let's say that Companies already has a row with name 'XCorp'. If you try a normal INSERT... VALUES statement here, you'll get an error, because you're trying to add a duplicate value. If you want to avoid that error, you can use something like this:
insert ignore into companies(name) values ('XCorp');
This will execute as a normal insert, but, since you're trying to insert a duplicate value, it will fail, but silently (it wil throw a warning instead of an error).
As for suggestion 2, as I told you, I leave it to you as homework.
Hints:
Count the rows where the name matches a value.
Read the count to a variable in your python program
Test the value... if there's zero entries, then perform the insert.
I am trying to load data from a CSV into a database in MySQL workbench. The table I am loading into has an auto increment ID column. I am trying to get the query to recognize that I want it to keep the first column as Null, and I put NULL as the value in the CSV, but I cannot get the SET ... NULL command to recognize the name of the ID column. Here is the SQL I am using:
load data infile 'filenam.csv'
INTO TABLE table_name
fields Terminated By ','
LINES TERMINATED BY ',,'
SET column_name = null
I suspect I am making a simple syntax error that is causing the problem. But I cannot figure out what it is.
If you put NULL as the value in the CSV file then you shouldn't need the "SET column_name = null" in the statement. AFAIK, the SET value should be used to supply values not derived from the input file or to perform calculations to the value before insertion. This statement should work fine since you said you specified NULL in the CSV. However, make sure you specified NULL "properly" according to the documentation. I always use \N in my import files.
LOAD DATA INFILE 'filename.csv'
INTO TABLE table_name
FIELDS TERMINATED BY ','
LINES TERMINATED BY ',,'
Secondly, you can discard the NULL specified in the CSV file by assigning it to a user variable and then specifying the column value with SET. However, you need to specify a column list
LOAD DATA INFILE 'filename.csv'
INTO TABLE table_name (#dummy, column_2, column_3)
FIELDS TERMINATED BY ','
LINES TERMINATED BY ',,'
SET column_name = NULL
I have one other thought based on the MySQL docs and dependent upon how your server is configured. Comment if this does not work and I will provide more options.
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
I've looked at the other questions regarding STR_TO_DATE() and their solutions aren't working for my case. I've also read up and searched elsewhere to no avail.
I'm importing many rows of the following XML:
<Game>
<id>21</id>
<GameTitle>Final Fantasy XIII</GameTitle>
<ReleaseDate>03/09/2010</ReleaseDate>
</Game>
<Game>
<id>31</id>
<GameTitle>Heavenly Sword</GameTitle>
<ReleaseDate>09/12/2007</ReleaseDate>
</Game>
With the following SQL statement:
USE test;
LOAD XML LOCAL INFILE 'C:\\xampp\\mysql\\data\\test.xml'
REPLACE
INTO TABLE games
ROWS IDENTIFIED BY '<Game>'
SET ReleaseDate = STR_TO_DATE(#ReleaseDate, '%c/%e/%Y');
GameTitle and ID are importing perfectly. I've tried using %m and %d and have ensured that the ReleaseDate column is the DATE datatype. All rows are writing NULL in the ReleaseDate column.
MySql is returning
2466 row(s) affected, 64 warning(s): 1265 Data truncated for column 'ReleaseDate'
at row 1 1265
Thank you in advance for you help. StackOverflow has solved every problem I've really ever had without even having to ask.
You'd need to assign something to the variable you're using with STR_TO_DATE, something like
LOAD XML LOCAL INFILE 'C:\\xampp\\mysql\\data\\test.xml'
REPLACE
INTO TABLE games (id, GameTitle, #var1)
ROWS IDENTIFIED BY '<Game>'
SET ReleaseDate = STR_TO_DATE(#var1, '%c/%e/%Y');
Note that the syntax for SET in LOAD XML is described in the syntax for LOAD DATA
I'm importing a CSV file with dotted german dates into a MySQL database. I want the dates in the CSV to automatically be formatted correctly to the correct data type fields used by MySQL.
I'm using Sequel Pro for the import. I gather I'm supposed to use the STR_TO_DATE function, but I just can't wrap my head around how to use Add Value or Expression in the program.
German date
Here are the dates in the CSV file:
DD.MM.YYYY e.g.: 28.01.1978
MySQL date
Here is what I want to end up with in the database:
YYYY-MM-DD
e.g.: 1978-01-28
Here's what I've tried
I put in STR_TO_DATE('$5', '%d.%m.%Y'); into Add Value or Expression, but this only gives the following error message:
[ERROR in row 1] You have an error in your SQL syntax; check the manual that corresponds to your
MySQL server version for the right syntax to use near '06.04.1997'', '%d.%m.
%Y');,'2KMX','43354997')' at line 2
Any ideas?
You need import the date field in a varchar fields (temp_varchar_field) first, after that, you can use something like:
update table_name set final_date=STR_TO_DATE(temp_varchar_field,'%d.%m.%Y');
You should do something like:
Create a temporary field: alter table table_name add column temp_varchar_field varchar(10);
Import, using Sequel Pro, the CVS file but using the temp_varchar_field for the date.
update table_name set final_date=STR_TO_DATE(temp_varchar_field,'%d.%m.%Y');
Delete the temp field if everything was imported properly. Using: alter table_name drop column temp_varchar_field;
I just got it to work with this piece of SQL-code:
load data local infile 'myfile.csv' into table `mytable`
fields terminated by ','
enclosed by '"'
lines terminated by '\n'
(surname, name, #germandate, telephone, etc)
set birthyear = STR_TO_DATE(#germandate , "%d.%m.%Y")
;
The clue here being the #germandate variable which is turned into the default MySQL date by setting the respective column with STR_TO_DATE(). No hacks needed! :)
It is easier if your CSV import would contain a date as a MySQL string, but it can be done otherwise too:
Step 1:
Define a varchar(10) for your german dates and import the data.
Step 2:
Add another field to your table:
ALTER TABLE `yourtable`
ADD COLUMN `your_mysql_date` DATE NULL;
Step 3:
Move the data:
UPDATE yourtable
SET your_mysql_date = CONCAT(
RIGHT(your_german_date,4),
'-',
MID(your_german_date,4,2),
'-',
LEFT(your_german_date,2)
);
...done!
There might be an easier way to solve this, but this way you have alot of control over the data and the formatting.
I am attempting a LOAD DATA INFILE and getting the above error.
LOAD DATA INFILE '$file'
REPLACE INTO TABLE $custom_parts
FIELDS TERMINATED BY ',' ESCAPED BY '\\\\'
LINES TERMINATED BY '\\r\\n'
IGNORE 1 LINES
(`partsno`, `mfg`, `cond`, `price`, `is_deleted`, #date_added)
SET `date_added` = STR_TO_DATE(#date_added, '%c/%e/%Y'),
`prtky` = REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(partsno, ' ', '' )
, '\\\\', '' ) , '/', '' ) , '_', '' ) , '.', '' ) , '-', '' )
The columns of the file are so
Part Number,MFR,Condition,price,is_deleted,date_added
Whenever I run the PHP to load this, I get the error. I am entrigued as why this is occuring. I believe that it is an issue with the user variable not being assigned and I am just looking for verification.
You cannot use variables for dynamic table names.
You can only do this using prepared statements,
However
You cannot use load data infile inside a stored procedure and I'm not sure you can use it in a prepared statement either.
If you use MySQL from a higher level program (php, pascal, whatever) you can
resolve the variable before constructing the query;
Check the columnname against a whitelist to prevent SQL-injection
Feed MySQL the expanded statement.
See this question for sample code: How to prevent SQL injection with dynamic tablenames?
Also, if you are using dynamic tablenames, escape them using ` backticks. This prevents MySQL from bombing if the tablenames contains funny chars or is a reserved word.
I answered my own question. ---- User variables may be used in most contexts where expressions are permitted. This does not currently include contexts that explicitly require a literal value, such as in the LIMIT clause of a SELECT statement, or the IGNORE N LINES clause of a LOAD DATA statement.