My situation: a webshop running Shopware6, database quite big (34GB total) but most of it is the logs (table log_entry = 28GB) and the saved shopping carts (table cart = 3GB).
I would like to do a mysqldump but for 2 tables log_entry and cart, I would like to save only the schema.
I know how to do only the schema for all tables with the --no-data flag or the data only with the --no-create-info flag and to ignore a table with the --ignore-table=[tablename].
Is my best option to do 2 dumps, one with the schema only and a second one with data only where I ignore the 2 tables?
that would then give
mysqldump -u user -p $dbname --no-data > backup_schema.sql
mysqldump -u user -p $dbname --no-create-info --ignore-table=$dbname.cart --ignore-table=$dbname.log_entry > backup_data.sql
If you want to use native mysqldump, you cannot avoid to make two calls as already mentioned by yourself.
We use the GDPRdump tool by SmileSA for such jobs, where you can leave out (truncate) and even anonymize data during the dump.
There is already a Shopware 6 template for this on GitHub
https://github.com/portaltech-reply/gdpr-dump-shopware
A less sophisticated solution which basically does what you already tried in a bit more flexible way and into one dump-file, is https://github.com/amenk/SelfScripts/blob/master/mysql-stripped-dump (self-link)
If it works, it might be your best bet. Although, is it possible to send it SQL statements directly in your environment? Another way might be to export the data into CSV format using an SQL statement that gets the exact data you want. This code would get just the data (username, email and state):
SELECT username, email, state
FROM TABLENAME
INTO OUTFILE '/temp/yourdata.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n';
This one will get the Columns & Data together:
SELECT 'username', 'email', 'state'
UNION ALL
SELECT username, email, state
FROM TABLENAME
INTO OUTFILE '/temp/yourdatafull.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n';
I know this might not be the exact answer you were looking for, but it might give you an alternative idea for a secondary or alternative backup method. Or at the very least it is handy to drop a file that will load easy into excel that you can play around with to do some manual calculations or data mining. Although a drawback is I believe if you do this, it will use the first datatype when you do a join so if you have dates or numbers after it might confuse it a bit. Also if you have the --secure-file-priv set to ON you will only be able to output to the specific directly specified in the MySQL settings.
Of course if you have an environment where you are saving integers and dates as strings, I think you should be fine. Will need some testing on that for sure, just stumbled across that over here if you want more information on this method:
https://www.databasestar.com/mysql-output-file/
Related
I got a problem, and after some hours of research I just want to die.
Is there a way to import lots of CSV data into one MySQL database but creating new tables with the file name of the CSV data?
Example: If I import data1.csv into db the table should be named data1 with all the data from data1.csv.
Thanks for your suggestions and answers.
There is no built in tool/method/command/query to accomplish what you desire within MySQL alone.
What will be required is 2 parts.
1st. of course your MySQL DB where the table will be created.
2nd. some 3rd party program that can interact with your DB. Eg. (Java, JavaScript, Python, even Unix shell scripting)
Following is a sudo example of what will be needed.
What this program will have to do is relatively simple.
It will require a couple inputs:
DataBase IP, Username, Password (these can be parameters passed into your program, or for simplicity of testing hard coded directly into the program)
The next input will be your file name. data1.csv
Using the inputs the program will harvest the 'data1' name as well as the first row of the data1.csv file to name each column.
Once the program collects this info, it can Connect to the DB and run the MySQL statement for CREATE TABLE TableName (RowName1 VARCHAR(255), RowName2 VARCHAR(255), ect...)
Finally it can do a MySQL command to import he *.csv file into the newly created table. eg.
LOAD DATA LOCAL INFILE 'C:/Stuff/csvFiles/Data1.csv'
INTO TABLE `SchemaName`.`Data1`
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
Hope this helps clear up your options an approach a little.
Using: MySQL Workbench
I am trying to write in to my MySQL Query to have the file automatically save using the current date (orders_20150214.csv), however even the basic save(no date) is not working, I keep getting a server refusal.
Code:
USE `databasename`;
Select -- complex query from multiple tables --
-- Get Order data --
INTO OUTFILE 'E:\My Documents\2014-2015 Accs\ccccc\orders.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
ESCAPED BY '\\'
LINES TERMINATED BY '\n'
Error:
Error Code: 1045. Access denied for user 'dddddd'#'%.dddddddd' (using password: YES)
I have also tried a number of variations of the above
USE `databasename`;
Select -- complex query from multiple tables --
-- Get Order data --
FROM table_name -- multiple tables --
INTO OUTFILE 'E:\My Documents\2014-2015 Accs\ccccc\orders.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
I know MySQL Workbench has an inbuilt "save as" and I use it, however there are days where I want to run the query, and it autosaves. I have a macro that retrieves data from .csv files and compiles to a running order book, I'd like to automate it further.
I have also tried:
INTO OUTFILE 'E:\\My Documents\\2014-2015 Accs\\dddd\\orders.csv'
INTO OUTFILE 'E:/My Documents/2014-2015 Accs/dddd/orders.csv'
-- http://dev.mysql.com/doc/refman/5.1/en/select-into.html
I've read mixed messages as to whether I can do this programmatically to save on my local machine. So it may not be possible directly through MySQL Workbench.
Please help :)
I am pretty sure you're getting the 1045 error because your user account needs to be granted the FILE privilege in the MySQL server in order to even attempt INTO OUTFILE operations.
You, or your friendly local admin, can do that with a query like this.
GRANT FILE ON *.* TO 'user'#'host';
The stuff Gordon mentioned is still true: The MySQL server will write the output file into its own file system, and the file it writes will be owned by it. You'll need to copy it.
If you're on a Windows system, you can try using the path to which the %TEMP% environment variable expands... for example
INTO OUTFILE 'C:\\Users\\MrsAdmin\\AppData\\Local\\Temp\\orders.csv'
On a *nix system,
INTO OUTFILE '/tmp/orders.csv'
will do the same thing. Then you can copy the file to wherever you need it written.
Keep in mind that your MySQL server runs usually under a different account than your personal one (on Windows usually "Network Services"). This account doesn't have write access to "normal" folders. You can simply create an export folder and add "Users" with read/write access to it's ACL. That will allow the server to write files there then.
This is similar to another question (http://stackoverflow.com/questions/935556/mysql-dump-by-query) but I hope different enough.
I want to export a specific items from a db table so I can back it up for possible future restoration.
I'm already using something like this from another table...
mysqldump --user="user" --password="password" --opt -w"id=1" databasebname tablename
But now I need something more complex.
I have the following query that I need to use to generate the export data...
SELECT tbl2.*
FROM tbl1, tbl2
WHERE tbl2.parent = tbl1.child
AND tbl1.id = 1
Can I do this with mysqldump?
Or do I need to think of a different approach?
(If it helps, this is all being done from within a bash script)
I think this will accomplish what you're looking for:
SELECT tbl2.*
FROM tbl1, tbl2
WHERE tbl2.parent = tbl1.child
AND tbl1.id = 1
INTO OUTFILE '/path/to/file.csv'
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
This will save your data into a CSV file. You can also save into other formats. I found a helpful tutorial on this topic a while back from here
You can do your select statement as normal and then add to the end of it
INTO OUTPUT FILE 'path/to/file'
That file can later be used with the LOAD DATA command as a backup.
Of course, if it were me, I'd feel better just dumping the whole table.
I want to do that using a code and not using a tool like "MySQL Migration Toolkit". The easiest way I know is to open a connection (using MySQL connectors) to DB1 and read its data. Open connection to DB2 and write the data to it. Is there a better/easiest way ?
First I'm going to assume you aren't in a position to just copy the data/ directory, because if you are then using your existing snapshot/backup/restore will probably suffice (and test your backup/restore procedures into the bargain).
In which case, if the two tables have the same structure generally the quickest, and ironically the easiest approach will be to use SELECT...INTO OUTFILE... on one end, and LOAD DATA INFILE... on the other.
See http://dev.mysql.com/doc/refman/5.1/en/load-data.html and .../select.html for definitive details.
For trivial tables the following will work:
SELECT * FROM mytable INTO OUTFILE '/tmp/mytable.csv'
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '\\\\'
LINES TERMINATED BY '\\n' ;
LOAD DATA INFILE '/tmp/mytable.csv' INTO TABLE mytable
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '\\\\'
LINES TERMINATED BY '\\n' ;
We have also used FIFO's to great effect to avoid the overhead of actually writing to disk, or if we do need to write to disk for some reason, to pipe it through gzip.
ie.
mkfifo /tmp/myfifo
gzip -c /tmp/myfifo > /tmp/mytable.csv.gz &
... SEL
ECT... INTO OUTFILE '/tmp/myfifo' .....
wait
gunzip -c /tmp/mytable.csv.gz > /tmp/myfifo &
... LOAD DATA INFILE /tmp/myfifo .....
wait
Basically, one you direct the table data to a FIFO you can compress it, munge it, or tunnel it across a network to your hearts content.
The FEDERATED storage engine? Not the fastest one in the bunch, but for one time, incidental, or small amounts of data it'll do. That is assuming you're talking about 2 SERVERS. With 2 databases on one and the same server it'll simply be:
INSERT INTO databasename1.tablename SELECT * FROM databasename2.tablename;
You can use mysqldump and mysql (the command line client). These are command line tools and in the question you write you don't want to use them, but still using them (even by running them from your code) is the easiest way; mysqldump solves a lot of problems.
You can make selects from one database and insert to the other, which is pretty easy. But if you need also to transfer the database schema (create tables etc.), it gets little bit more complicated, which is the reason I recommend mysqldump. But lot of PHP-MySQL-admin tools also does this, so you can use them or look at their code.
Or maybe you can use MySQL replication.
from http://dev.mysql.com/doc/refman/5.0/en/rename-table.html:
As long as two databases are on the same file system, you can use RENAME TABLE to move a table from one database to another:
RENAME TABLE current_db.tbl_name TO other_db.tbl_name;
If you enabled binary logging on your current server (and have all the bin logs) you can setup replication for the second server
Is there a way to create a dump/export/save a temporary MySQL table into a file on disk(.sql file that is, similar to one that is created by mysqldump)?
Sorry, I did not read the question properly the first time around... at any rate, the best I can think of is using the SELECT ... INTO OUTFILE statement, like this:
SELECT * INTO OUTFILE 'result.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM temp_table;
This does have many limitations thought, for instance, it only dumps the raw data without including the field headers. The other thing I found that may or may not be of use is the SHOW CREATE TABLE statement. If you can find some way of combining the output from these two statements, you may be able to get a proper "dump" file as produced by my command below.
You should be able to use the mysqldump application:
mysqldump --databases temptable > file.sql
This will dump the table with CREATE decelerations.