MySQL purging shell script - mysql

I have a mysql table with 120 million rows and I want to write a shell script to start purging some of that useless information in that table that's not needed. Problem is I'm super new to shell scripting.
I have a datetime column with a unix timestamp in it. I want to delete every row that's not within the last 2 months since I've recently enacted a data retention policy that will allow me to only keep 2 months of certain data.
TL;DR Need to build a shell script that deletes all rows up until the last 2 months of data by using the unix timestamp in the datetime column.
UPDATE: Here's my new shell script
#!/bin/sh
i=1
while [ "$i" -ne 0 ]
do
i=mysql -h 127.0.0.1 -u halo_ylh -pCa5d8a88 halo_twittstats < mysqlpurge.sql
sleep 10
done


Wouldn't it be easier to just use the current_timestamp and unix_timestamp function to execute:
DELETE FROM Table1
WHERE datetime < unix_timestamp(current_timestamp - interval 2 month)
To run it from the shell you could put that command in a file script1.sql and run it using the mysql command line tool:
mysql db_name < script1.sql


mysql --host ? -user ? -password ? {DB_NAME} < dump_script.sql > data.dat
dump_script.sql will have your SELECT statememt to retrieve the data you want archived. it will store the output in data.dat
then
mysql --host ? -user ? -password ? {DB_NAME} < delete_script.sql
delete_script.sql will cotain the DELETE statement with the same WHERE clause as in dump_script.sql.
Make sure you lock the table so that nothing writes in between the two script exections that could make it into the WHERE clause to avoid phantom inserts that would be deleted by the delete script yet not be included in the dump script.

Related

Backup mysql database daily based on day name

I'm using ubuntu 16.04 with nginx installed, currently i run daily backup using cron like this:
#crontab -u root -e
0 2 * * * mysqldump -u username -p"password" production | gzip -c > production.gz
this will backup my database everydate at 2am, the problem here is i need to backup database based on day name, so the backup database name will be suited based on dayname, for example the file name will look like this:
production_monday.gz
production_tuesday.gz
production_wednesday.gz
production_thursday.gz
production_friday.gz
production_saturday.gz
production_sunday.gz
how can i set the cron to produce the file like above ? cron schedule will auto rewrite the file based on the day name

My suggestion would be to create a shell script that finds the current day of the week (date +%A), then write the mysqldump output to a file formatted as "prefix"_"dayofweek" for you to zip. Then from cron, just execute this shell script rather than the mysqldump directly.
You may also find this answer helpful.

Delete query execution is very slow when run inside bash script

I'm trying to run following query inside a bash script.
When it is executed from mysql command promt, execution time was 0.06sec.
mysql> delete from assign_history where offer_id not in
->('7','8','9','10','11','12','13','14','32','157','211','240','241','242','273',
->'274','275','310','312','313','314','326','328','329','333','334','335','336',
->'337','342','343','355','362','374','375','376','378','379','383','384','409','411')
->and date(action_date) < "2015-06-25" order by id limit 1000;
Query OK, 1000 rows affected (0.06 sec)
But when run it inside a bash script, it takes more than 2 minutes.
[root#localhost umap]# cat ./history_del.sh
#! /bin/bash
echo $(date)
mysql -uroot -ppassword db_offers -e "delete from assign_history where offer_id not in ('7','8','9','10','11','12','13','14','32','157','211','240','241','242','273','274','275','310','312','313','314','326','328','329','333','334','335','336','337','342','343','355','362','374','375','376','378','379','383','384','409','411') and date(action_date) < "2015-06-25" limit 1000;"
echo $(date)
[root#localhost umap]# ./history_del.sh
Wed Aug 26 19:08:45 IST 2015
Wed Aug 26 19:10:48 IST 2015
I also tried with "mysql -Bse" options. No improvement. Any ideas?

Any ideas?
First, you need to escape double-quotes inside the query string: \"2015-06-25\" (try to output your query with echo and you'll see, why ). I dont know, how your request works without properly specified quotes...
Second, it is better and preferred to place your long-line-request in the file, so your command-line will look like this:
mysql -uroot -ppassword db_offers <YOUR_FILE
Request in YOUR_FILE will be the same as in the mysql prompt (of course, you dont need to escape double-quotes here).
And yes, when you call mysql utility - it can take unpredictably long time to connect to MySQL server, so 2 minutes include this time (but 0.06 sec in mysql prompt doesnt!), so you cant say, how much time does it take to connect to server and how much - to send and execute your query.
To know, how much time does it take to connect to mysql server, try to execute (wait several seconds after previous run of the mysql utility) any empty query, such as:
time mysql -u user -ppassword -Bs <<<'select null'

Cron job using database results

I need help in creating a cron job script. Basically, I want to grab the next scheduled item and run it through ffmpeg to stream. This would be the mysql query (I'm using PHP variables to indicate what should go there - I don't actually know how variables work in cron jobs):
SELECT show.file FROM show, schedule
WHERE channel = 1 AND start_time <= $current_time;
This would be the ffmpeg command:
ffmpeg -re -i $file http://127.0.0.1:8090/feed.ffm
How would I create a cron job to execute these commands?

First of all IMHO you don't need to pass current time to your select statement, just use CURRENT_TIME
SELECT `show.file`
FROM show, schedule
WHERE channel = 1 AND start_time <= CURRENT_TIME;
Depending on your actual table's DDL you might need to do some conversion to correctly compare time values.
Assuming that your query correct and returns ONLY ONE filename you can execute the query with mysql, output the result into predefined file and use && to chain ffmpeg command
mysql -u user -ppassword dbname -sN -e \
"SELECT show.file \
FROM show, schedule \
WHERE channel = 1 AND start_time <= CURRENT_TIME" > /tmp/cur_show_file && \
ffmpeg -re -i /tmp/cur_show_file http://127.0.0.1:8090/feed.ffm

Require Cron syntax for MySQL maintenance

I'm trying to set up a Cron job for deleting MySQL records where a date field is older than three weeks, but I can't figure out what the string is that goes in the box.
Here's a pic of the Cron management screen. Can anyone help please?
http://i46.tinypic.com/id4nsj.jpg

If you know the query you want to run, you can use the -e argument for mysql at the command line for your script. So the "Command to Run" in your cron management tool would be:
mysql -u <username> -p<password> -h <name-of-mysql-server> <databasename>
-e "<YOUR-QUERY-HERE>"
The general structure of a query to delete records older than a date is:
DELETE FROM [table] WHERE [column] < DATE_SUB(NOW(), INTERVAL 3 WEEK);

MySQL command-line tool: How to find out number of rows affected by a DELETE?

I'm trying to run a script that deletes a bunch of rows in a MySQL (innodb) table in batches, by executing the following in a loop:
mysql --user=MyUser --password=MyPassword MyDatabase < SQL_FILE
where SQL_FILE contains a DELETE FROM ... LIMIT X command.
I need to keep running this loop until there's no more matching rows. But unlike running in the mysql shell, the above command does not return the number of rows affected. I've tried -v and -t but neither works. How can I find out how many rows the batch script affected?
Thanks!

You can add SELECT ROW_COUNT(); at the end of the batch script.

If you add option -vv it will produce more verbose output, which also contain an information about number of affected rows;
mysql -vv --user=MyUser --password=MyPassword MyDatabase < SQL_FILE