I have a column in a MySQL table that consists of comma-delimited strings. I would like to convert this column into a set of distinct strings that occur in the column (for any row in the table -- the set includes strings that occur in any row of the table in this column). What is the easiest way to accomplish this?
The solution doesn't need to be pure MySQL. It could involve unix, perl, etc.
You could probably get a quick-and-dirty list of distinct strings from a comma-delimited column using SELECT INTO OUTFILE, sed, and LOAD DATA INFILE.
Basically you want to dump the text data to a flat file, using a comma as the line delimiter so each string will be treated as a separate row when you load it back into the database. Then load the extracted into a new table and select the distinct values.
Off the top of my head, the code would look something like this:
select str
into outfile
'/tmp/your_table_data.txt'
lines terminated by ','
from your_table;
sed -e 's/\\,/,/g' -e 's/,$//' /tmp/your_table_data.txt > /tmp/commadelimited.txt
create table your_table_parsed(str text);
load data infile '/tmp/commadelimited.txt'
ignore into table your_table_parsed
fields terminated by ','
lines terminated by ',';
select distinct str from your_table_parsed;
The way I chose was to run the select mysql command outside of the mysql shell and pipe the results into tr and sort --uniq
mysql my_db [-p -u -h] -se "select my_column from my_table;" | tr ',' '\n' | sort -u
This is pretty simple and seems to give the correct results as far as I can tell.
Related
I am trying to insert some data using Load data infile into a mysql table that already has some data. The table contains id and name. My csv file contains three fields: id, name and code. The table schema also has these three fields, but currently, the table has NULL for the code field. I am trying to insert the code from csv to an existing row in the table if it matches the name, else I am trying to insert a complete new row.
The code I have tried is as follows:
LOAD DATA INFILE 'table1.csv'
INTO TABLE table1
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(#code, #name, #other_columns)
IF EXISTS (SELECT * FROM table1 where name=#name);
BEGIN
set code=#Code;
END
ELSE
BEGIN
set code=#Code, name=#name;
END
By doing so, I am getting a mysql syntax error, but am unable to figure it out. Can anyone point me in the right direction, or suggest to me another approach? I have thousands of new rows to insert and thousands of existing rows to modify based on the certain field, (name in this case).
MySQL does not allow the LOAD DATA INFILE statement inside a stored program, which is where the IF statement appears. Break up your task into two parts. First, LOAD DATA INFILE into a temporary table. Then create a stored program that replaces the loaded data into your table1 from the temporary table.
i am using bash to load a file in mysql, and i have:
mysql --local-infile=1 -u user data_base_1 < file.sql
and file.sql is :
..$ cat file.sql
load data local infile '/folder/load.csv' into table table_1 fields terminated by '|'
The code works fine.
The problem is that if the PK of one row in the file exist, the row is not inserted, and i need if the row exist insert and replace the row in the table. How can i do it?
Who can help me?
Thanks
You can specify REPLACE with LOAD DATA:
LOAD DATA LOCAL INFILE '/folder/load.csv' REPLACE INTO TABLE table_1 FIELDS TERMINATED BY '|'
Or else use the mysqlimport --replace option.
http://dev.mysql.com/doc/refman/5.6/en/mysqlimport.html#option_mysqlimport_replace
You could load into a temporary table and then execute two SQL statements:
UPDATE table
WHERE ... (match found)
;
INSERT into table(...)
SELECT ...
FROM temp_table
WHERE NOT EXISTS(...)
I am trying to load data from a CSV into a database in MySQL workbench. The table I am loading into has an auto increment ID column. I am trying to get the query to recognize that I want it to keep the first column as Null, and I put NULL as the value in the CSV, but I cannot get the SET ... NULL command to recognize the name of the ID column. Here is the SQL I am using:
load data infile 'filenam.csv'
INTO TABLE table_name
fields Terminated By ','
LINES TERMINATED BY ',,'
SET column_name = null
I suspect I am making a simple syntax error that is causing the problem. But I cannot figure out what it is.
If you put NULL as the value in the CSV file then you shouldn't need the "SET column_name = null" in the statement. AFAIK, the SET value should be used to supply values not derived from the input file or to perform calculations to the value before insertion. This statement should work fine since you said you specified NULL in the CSV. However, make sure you specified NULL "properly" according to the documentation. I always use \N in my import files.
LOAD DATA INFILE 'filename.csv'
INTO TABLE table_name
FIELDS TERMINATED BY ','
LINES TERMINATED BY ',,'
Secondly, you can discard the NULL specified in the CSV file by assigning it to a user variable and then specifying the column value with SET. However, you need to specify a column list
LOAD DATA INFILE 'filename.csv'
INTO TABLE table_name (#dummy, column_2, column_3)
FIELDS TERMINATED BY ','
LINES TERMINATED BY ',,'
SET column_name = NULL
I have one other thought based on the MySQL docs and dependent upon how your server is configured. Comment if this does not work and I will provide more options.
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
I'm importing a CSV file with dotted german dates into a MySQL database. I want the dates in the CSV to automatically be formatted correctly to the correct data type fields used by MySQL.
I'm using Sequel Pro for the import. I gather I'm supposed to use the STR_TO_DATE function, but I just can't wrap my head around how to use Add Value or Expression in the program.
German date
Here are the dates in the CSV file:
DD.MM.YYYY e.g.: 28.01.1978
MySQL date
Here is what I want to end up with in the database:
YYYY-MM-DD
e.g.: 1978-01-28
Here's what I've tried
I put in STR_TO_DATE('$5', '%d.%m.%Y'); into Add Value or Expression, but this only gives the following error message:
[ERROR in row 1] You have an error in your SQL syntax; check the manual that corresponds to your
MySQL server version for the right syntax to use near '06.04.1997'', '%d.%m.
%Y');,'2KMX','43354997')' at line 2
Any ideas?
You need import the date field in a varchar fields (temp_varchar_field) first, after that, you can use something like:
update table_name set final_date=STR_TO_DATE(temp_varchar_field,'%d.%m.%Y');
You should do something like:
Create a temporary field: alter table table_name add column temp_varchar_field varchar(10);
Import, using Sequel Pro, the CVS file but using the temp_varchar_field for the date.
update table_name set final_date=STR_TO_DATE(temp_varchar_field,'%d.%m.%Y');
Delete the temp field if everything was imported properly. Using: alter table_name drop column temp_varchar_field;
I just got it to work with this piece of SQL-code:
load data local infile 'myfile.csv' into table `mytable`
fields terminated by ','
enclosed by '"'
lines terminated by '\n'
(surname, name, #germandate, telephone, etc)
set birthyear = STR_TO_DATE(#germandate , "%d.%m.%Y")
;
The clue here being the #germandate variable which is turned into the default MySQL date by setting the respective column with STR_TO_DATE(). No hacks needed! :)
It is easier if your CSV import would contain a date as a MySQL string, but it can be done otherwise too:
Step 1:
Define a varchar(10) for your german dates and import the data.
Step 2:
Add another field to your table:
ALTER TABLE `yourtable`
ADD COLUMN `your_mysql_date` DATE NULL;
Step 3:
Move the data:
UPDATE yourtable
SET your_mysql_date = CONCAT(
RIGHT(your_german_date,4),
'-',
MID(your_german_date,4,2),
'-',
LEFT(your_german_date,2)
);
...done!
There might be an easier way to solve this, but this way you have alot of control over the data and the formatting.
Say I have a view in my database, and I want to send a file to someone to create that view's output as a table in their database.
mysqldump of course only exports the 'create view...' statement (well, okay, it includes the create table, but no data).
What I have done is simply duplicate the view as a real table and dump that. But for a big table it's slow and wasteful:
create table tmptable select * from myview
Short of creating a script that mimics the behaviour of mysqldump and does this, is there a better way?
One option would be to do a query into a CSV file and import that. To select into a CSV file:
From http://www.tech-recipes.com/rx/1475/save-mysql-query-results-into-a-text-or-csv-file/
SELECT order_id,product_name,qty
FROM orders
INTO OUTFILE '/tmp/orders.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
OK, so based on your CSV failure comment, start with Paul's answer. Make the following change to it:
- FIELDS TERMINATED BY ','
+ FIELDS TERMINATED BY ',' ESCAPED BY '\'
When you're done with that, on the import side you'll do a "load data infile" and use the same terminated / enclosed / escaped statements.
Same problem here my problem is that I want to export view definition (84 fields and millions of records) as a "create table" statement, because view can variate along time and I want an automatic process. So that's what I did:
Create table from view but with no records
mysql -uxxxx -pxxxxxx my_db -e "create table if not exists my_view_def as select * from my_view limit 0;"
Export new table definition. I'm adding a sed command to change table name my_view_def to match original view name ("my_view")
mysqldump -uxxxx -pxxxxxx my_db my_view_def | sed s/my_view_def/my_view/g > /tmp/my_view.sql
drop temporary table
mysql -uxxxx -pxxxxxx my_db -e "drop table my_view_def;"
Export data as a CSV file
SELECT * from my_view into outfile "/tmp/my_view.csv" fields terminated BY ";" ENCLOSED BY '"' LINES TERMINATED BY '\n';
Then you'll have two files, one with the definition and another with the data in CSV format.