I am running an ETL script that loads data from mysql into teradata. The script aims to select all rows later than the timestamp of the last successful run of the bash script. Since I do not have write access to the mysql database, I need to store the last run timestamp with the bash script. Is there an easy way to store the timestamp of a successful run? I was thinking I could have a file that I would touch at the end of the script and then check its mtime, or just parse out the timestamp from a log file. What are some better strategies to do this?
Within your script, use set -e1 so that the script exits immediately if any command within the script fails. Then, at the end, log successful completion with a unix timestamp date +%s.
You can then use SELECT FROM_UNIXTIME(<YOUR TIMESTAMP>, <YOUR MYSQL DATE FORMAT>)2 to pull rows that are newer than the last successful completion.
One big caveat: I would not rely solely on timestamps to approach this problem. I would pull from MySQL with some time overlap and check primary keys for each insert into teradata to avoid inserting duplicates. To follow this approach, just subtract 1800 from <YOUR TIMESTAMP> to ensure a 30 minute overlap.
Related
I need to be able to copy data from one table to another on a specific date. I have a list of names that can be altered by a user. However, on a certain date, these need to be locked in and copied to a uneditable table and then mailed to key users.
How would I do this?
Cheers
mysql has trigger.
trigger is like functions which is called on a specific events and cannot be called explicitly. If the event specified occurs, database automatically calls that trigger and does what it supposed to do.
So to do your job, you can write a trigger which will work on your tables and set the date when you want to do your task. Then use if to check the system date is your desired date and if yes do your job.
I used trigger on oracle and never used on mysql but as triggers are using pl/sql, there should be no difference between triggers of oracle and mysql.
PHP has Mail sending options. You can send mail using php.
Let me know if this helps you.
First step would be to create a script in any language, let's say PHP, this script will just connect to the database, and execute the copy queries (grab and insert queries) whenever called.
Then, create a cronjob using Linux to execute the script on the needed schedule.
Sending Emails would be carried out by a similar technique.
Configure a cronjob: setup a cronjob
Copying data using PHP: copying data using php
MySQL implements an event scheduler which can execute mysql stored procedures either once after a specific interval, once at a specific time or repeatedly at a specific interval; that might be of some use to you.
I have a debian machine with a mysql server.
On mysql I have a table that contains a number of rows with a datetime field.
How can I execute a php script when the date and time of the machine match with those specified in any mysql record?
a) Mysql triggers ?
b) Deamon that runs in background and checks time every n seconds ?
c) Cron?
Let me know!
Polling the MySQL will work but surely is not efficient, especially when there are a lot of "triggers".
So use a cronjob. Note: MySQL-Queries cannot schedule or cancel cronjobs. So the (i guess) PHP-Script updating that date field also has to schedule the cronjobs. Also, every time the column changes you will have to cancel the previously scheduled cronjob and schedule a new one.
Since you can't run shell command from within Mysql database the MySql Triggers are no good. The MySQL Scheduler is therefore useless too. So you need an external script to help.
I'd suggest to create a (php) script and add it to the crontab to run every minute. Its task would be to check for matching dates in the database and run whatever command you need.
I think that the easiest solution is to use , if possible , the at command.
Basically after inserting the data into mysql do the following:
1)Create (using PHP) a BASH script that runs the php file
script_1.sh should look like this:
!#/bin/bash
php /path/to/file.php
and make it executable with :
exec("chmod +x script_1.sh");
2)create a second BASH script that executes the first at the desired time:
script_2.sh should look like this:
!#/bin/bash
at H:M Y:M:D < script_1.sh
Make script_2 executable and run it with the exec command.
I have a mysql table which has an auto increment id field(id), videoid(int) and userid(int). The site is live now. I forgot to put a date/time field there. now there are some data here. Is it possible to get the insert time for all the existing data?
I have another table which gets reset every week by a cron job. Something wrong happened last week and now I badly need those data. Is there any option by which I can get any kind of backup from a certain date? Does mysql has auto backup or something like that?
If you have access to the binary log, you can get the insert statements by using mysqlbinlog. Read more about it in the manual.
The output from mysqlbinlog can be re-executed (for example, by using it as input to mysql) to redo the statements in the log. This is useful for recovery operations after a server crash.
The goal is to make a graph (gnuplot) every month, containing the last 13 months of rows per month (MySQL)[so the 2-D graph will have 13 x-axis entries], one graph per customer.
Stepping back, is bash the right answer? I'd say I'm an advanced noob, but have not done for statements before, so I'm not sure how to approach this.
MySQL- I have a single mysql CLI line that I can get to run in a bash script with a variable (the time period) - it runs just fine:
PER003="'2013-08-01 00:00:00' AND '2013-08-31 23:59:59'"
mysql -uroot -password -h 192.168.x.x dbname -se "SELECT COUNT(*) FROM tablename WHERE study.created_time BETWEEN $PER003;"
I know all the periods, as they are months of time, but in the specified format. So I could make PER004, PER005, etc. as variables.
I'm not quite worried about the gnuplot issue yet, but am wondering if am I approaching this the right way? Should I make a script per customer(each customer has their own db). Should the time periods be listed in the script or be called from an external file?(because if I have to add time periods manually each month, I'd like to do it once, rather than have to edit each customer script). Also, could I make another external file that contains each customer db and IP for the mysql command, and do a loop calling those variables for that as well?
I'm just trying to wrap my head around what all is needed and the correct approach before I dive in..... THANKS!
I would like to ask a question for connoisseurs of SQL (MySQL, to be specific).
I have a table reservation schedules. And when a customer makes a reservation there is a time to let the client to use my service. Therefore, the reservation that he did have to leave the table reservations.
Once the time limit of use is reached, there is some method (trigger,
I believe), which automatically erase the record of this book on the
table?
If so, can someone give me some idea of how to start my search for it, or it is also totally welcome some help as some more advanced lines of code.
There is also the possibility that this only be possible to be implemented via Server-Side (PHP, ASP ...), which does not believe is so true because SQL is a language very complete (to my knowledge).
Edit1: The problem is that I believe this is a task of the DBMS, so I wanted to leave this responsibility to the MySQL The problem is: how?
A trigger is triggered by either before or after an insert , update or delete event (at least in MySQL according to the docs)
What you want is some sort of scheduled job either through your application be it php, asp.net, etc.. or cron job that calls some sort of SQL script.
So to answer, it can't be done purely with triggers.
You can use SQL jobs, but if the removal logic is to complex to manage it with queries I suggest you to use a PHP script that does all that work for you.
Just write down the data check/remove logic in PHP and set up a simple cron operation for it.
The advantage of this solution is that you can access to your scripts/classes/db providers and save your time and your can log all the operations separately (instead of logging to MySQL logs, no matter what script language you are relying on).
If you have a full control of your server the scheduled operation will look like this (if you want to check your DB entries every day at 00:01):
cat /etc/cron.d/php5
0 1 * * * php /path/to/your/script.php >> /path/to/your_script.log
..otherwise you will have to check the control panel of your hosting account and figure out how to manage
You can create one more column in your table where you will create the expiration date. Then you can on your sql server create the job that will erase all records that have expiration date less than curent date.
CREATE EVENT db_name
ON SCHEDULE EVERY 10 SECOND
DO
DELETE FROM myschema.mytable WHERE expiration_date < NOW()
I hope that will help.