This is similar to another question (http://stackoverflow.com/questions/935556/mysql-dump-by-query) but I hope different enough.
I want to export a specific items from a db table so I can back it up for possible future restoration.
I'm already using something like this from another table...
mysqldump --user="user" --password="password" --opt -w"id=1" databasebname tablename
But now I need something more complex.
I have the following query that I need to use to generate the export data...
SELECT tbl2.*
FROM tbl1, tbl2
WHERE tbl2.parent = tbl1.child
AND tbl1.id = 1
Can I do this with mysqldump?
Or do I need to think of a different approach?
(If it helps, this is all being done from within a bash script)
I think this will accomplish what you're looking for:
SELECT tbl2.*
FROM tbl1, tbl2
WHERE tbl2.parent = tbl1.child
AND tbl1.id = 1
INTO OUTFILE '/path/to/file.csv'
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
This will save your data into a CSV file. You can also save into other formats. I found a helpful tutorial on this topic a while back from here
You can do your select statement as normal and then add to the end of it
INTO OUTPUT FILE 'path/to/file'
That file can later be used with the LOAD DATA command as a backup.
Of course, if it were me, I'd feel better just dumping the whole table.
Related
My situation: a webshop running Shopware6, database quite big (34GB total) but most of it is the logs (table log_entry = 28GB) and the saved shopping carts (table cart = 3GB).
I would like to do a mysqldump but for 2 tables log_entry and cart, I would like to save only the schema.
I know how to do only the schema for all tables with the --no-data flag or the data only with the --no-create-info flag and to ignore a table with the --ignore-table=[tablename].
Is my best option to do 2 dumps, one with the schema only and a second one with data only where I ignore the 2 tables?
that would then give
mysqldump -u user -p $dbname --no-data > backup_schema.sql
mysqldump -u user -p $dbname --no-create-info --ignore-table=$dbname.cart --ignore-table=$dbname.log_entry > backup_data.sql
If you want to use native mysqldump, you cannot avoid to make two calls as already mentioned by yourself.
We use the GDPRdump tool by SmileSA for such jobs, where you can leave out (truncate) and even anonymize data during the dump.
There is already a Shopware 6 template for this on GitHub
https://github.com/portaltech-reply/gdpr-dump-shopware
A less sophisticated solution which basically does what you already tried in a bit more flexible way and into one dump-file, is https://github.com/amenk/SelfScripts/blob/master/mysql-stripped-dump (self-link)
If it works, it might be your best bet. Although, is it possible to send it SQL statements directly in your environment? Another way might be to export the data into CSV format using an SQL statement that gets the exact data you want. This code would get just the data (username, email and state):
SELECT username, email, state
FROM TABLENAME
INTO OUTFILE '/temp/yourdata.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n';
This one will get the Columns & Data together:
SELECT 'username', 'email', 'state'
UNION ALL
SELECT username, email, state
FROM TABLENAME
INTO OUTFILE '/temp/yourdatafull.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n';
I know this might not be the exact answer you were looking for, but it might give you an alternative idea for a secondary or alternative backup method. Or at the very least it is handy to drop a file that will load easy into excel that you can play around with to do some manual calculations or data mining. Although a drawback is I believe if you do this, it will use the first datatype when you do a join so if you have dates or numbers after it might confuse it a bit. Also if you have the --secure-file-priv set to ON you will only be able to output to the specific directly specified in the MySQL settings.
Of course if you have an environment where you are saving integers and dates as strings, I think you should be fine. Will need some testing on that for sure, just stumbled across that over here if you want more information on this method:
https://www.databasestar.com/mysql-output-file/
I have tables that are on different mysql instances. I want to export some data as csv from a mysql instance, and perform a left join on a table with the exported csv data. How can I achieve this?
Quite surprisingly that is possible with MySQL, there are several steps that you need to go through.
First create a template table using CSV engine and desired table layout. This is the table into which you will import your CSV file. Use CREATE TABLE yourcsvtable (field1 INT NOT NULL, field2 INT NOT NULL) ENGINE=CSV for example. Please note that NULL values are not supported by CSV engine.
Perform you SELECT to extract the CSV file. E.g. SELECT * FROM anothertable INTO OUTFILE 'temp.csv' FIELDS TERMINATED BY ',';
Copy temp.csv into your target MySQL data directory as yourcsvtable.CSV. Location and exact name of this file depends on your MySQL setup. You cannot perform the SELECT in step 2 directly into this file as it is already open - you need to handle this in your script.
Use FLUSH TABLE yourcsvtable; to reload/import the CSV table.
Now you can execute your query against the CSV file as expected.
Depending on your data you need to ensure that the data is correctly enclosed by quotation marks or escaped - this needs to be taken into account in step 2.
CSV file can be created by MySQL on some another server or by some other application as long as it is well-formed.
If you export it as CSV, it's no longer SQL, it's just plain row data. Suggest you export as SQL, and import into the second database.
I am using PHP scripts on my mySQL database. What I have is an inventory of products that is being pulled for a search page. I have this working fine, and my search page is working flawlessly. The only thing I cannot figure out is how to update the stock of each product in the database. I have a new file that needs to match up with the product number and then replace the data in one column of the mysql database. Much like this below
mysql database:
ProductNumber..................ProductStock
12345678....................................1
New file:
12345678..................5
Basically I need the new file to match with the product number on the mysql and replace the product stock with the new number. All of this in PHP.
You don't say how many rows you got in that database. Usually individual UPDATEs are pretty slow.
You can
load your "new stocks" file into a temporary table using LOAD DATA INFILE
make an UPDATE using a JOIN to set the values in your products table
This will be a few orders of magnitude faster.
Update with the syntax to use on mysql :
UPDATE products p JOIN temp_products t ON (p.id=t.id) SET p.stock = t.stock;
Note that you need special privileges for LOAD DATA INFILE.
You can also use LOAD DATA INFILE LOCAL (check the docs).
Or, you can parse the file with PHP, generate an INSERT with multi-values in a temp table, and do a joined update. This will be a little slower than LOAD DATA INFILE, but much much faster than doing 27000 queries.
$sql = 'UPDATE table_products SET ProductStock=5 WHERE ProductNumber=12345678';
mysql_query($sql);
So basically: use an UPDATE statement for the database.
How does the file look?
Well you will have to open the file, use explode() and then use preg_match as per the requirements.
You can use mysql's load data local infile to do this. Write a small import script and run it on a cron daily. Is the stock file in csv format?
An example script would be as follows
$dir = dirname(__FILE__) . DIRECTORY_SEPARATOR;
$stock_file = $dir . "files". DIRECTORY_SEPARATOR."stock.csv";
$stock_query = "
load data local infile \"{$stock_file}\"
replace into table products
fields terminated by ',' enclosed by '\"'
lines terminated by '\n'
ignore 1 lines
(ProductNumber,ProductStock);
";
$m->query($stock_query);
Things to consider would be what the fields are terminated by and enclosed by. Also the ignore 1 lines is there to ignore the header row if the file has one.
I want to do that using a code and not using a tool like "MySQL Migration Toolkit". The easiest way I know is to open a connection (using MySQL connectors) to DB1 and read its data. Open connection to DB2 and write the data to it. Is there a better/easiest way ?
First I'm going to assume you aren't in a position to just copy the data/ directory, because if you are then using your existing snapshot/backup/restore will probably suffice (and test your backup/restore procedures into the bargain).
In which case, if the two tables have the same structure generally the quickest, and ironically the easiest approach will be to use SELECT...INTO OUTFILE... on one end, and LOAD DATA INFILE... on the other.
See http://dev.mysql.com/doc/refman/5.1/en/load-data.html and .../select.html for definitive details.
For trivial tables the following will work:
SELECT * FROM mytable INTO OUTFILE '/tmp/mytable.csv'
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '\\\\'
LINES TERMINATED BY '\\n' ;
LOAD DATA INFILE '/tmp/mytable.csv' INTO TABLE mytable
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '\\\\'
LINES TERMINATED BY '\\n' ;
We have also used FIFO's to great effect to avoid the overhead of actually writing to disk, or if we do need to write to disk for some reason, to pipe it through gzip.
ie.
mkfifo /tmp/myfifo
gzip -c /tmp/myfifo > /tmp/mytable.csv.gz &
... SEL
ECT... INTO OUTFILE '/tmp/myfifo' .....
wait
gunzip -c /tmp/mytable.csv.gz > /tmp/myfifo &
... LOAD DATA INFILE /tmp/myfifo .....
wait
Basically, one you direct the table data to a FIFO you can compress it, munge it, or tunnel it across a network to your hearts content.
The FEDERATED storage engine? Not the fastest one in the bunch, but for one time, incidental, or small amounts of data it'll do. That is assuming you're talking about 2 SERVERS. With 2 databases on one and the same server it'll simply be:
INSERT INTO databasename1.tablename SELECT * FROM databasename2.tablename;
You can use mysqldump and mysql (the command line client). These are command line tools and in the question you write you don't want to use them, but still using them (even by running them from your code) is the easiest way; mysqldump solves a lot of problems.
You can make selects from one database and insert to the other, which is pretty easy. But if you need also to transfer the database schema (create tables etc.), it gets little bit more complicated, which is the reason I recommend mysqldump. But lot of PHP-MySQL-admin tools also does this, so you can use them or look at their code.
Or maybe you can use MySQL replication.
from http://dev.mysql.com/doc/refman/5.0/en/rename-table.html:
As long as two databases are on the same file system, you can use RENAME TABLE to move a table from one database to another:
RENAME TABLE current_db.tbl_name TO other_db.tbl_name;
If you enabled binary logging on your current server (and have all the bin logs) you can setup replication for the second server
Is there a way to create a dump/export/save a temporary MySQL table into a file on disk(.sql file that is, similar to one that is created by mysqldump)?
Sorry, I did not read the question properly the first time around... at any rate, the best I can think of is using the SELECT ... INTO OUTFILE statement, like this:
SELECT * INTO OUTFILE 'result.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM temp_table;
This does have many limitations thought, for instance, it only dumps the raw data without including the field headers. The other thing I found that may or may not be of use is the SHOW CREATE TABLE statement. If you can find some way of combining the output from these two statements, you may be able to get a proper "dump" file as produced by my command below.
You should be able to use the mysqldump application:
mysqldump --databases temptable > file.sql
This will dump the table with CREATE decelerations.