The goal is to make a graph (gnuplot) every month, containing the last 13 months of rows per month (MySQL)[so the 2-D graph will have 13 x-axis entries], one graph per customer.
Stepping back, is bash the right answer? I'd say I'm an advanced noob, but have not done for statements before, so I'm not sure how to approach this.
MySQL- I have a single mysql CLI line that I can get to run in a bash script with a variable (the time period) - it runs just fine:
PER003="'2013-08-01 00:00:00' AND '2013-08-31 23:59:59'"
mysql -uroot -password -h 192.168.x.x dbname -se "SELECT COUNT(*) FROM tablename WHERE study.created_time BETWEEN $PER003;"
I know all the periods, as they are months of time, but in the specified format. So I could make PER004, PER005, etc. as variables.
I'm not quite worried about the gnuplot issue yet, but am wondering if am I approaching this the right way? Should I make a script per customer(each customer has their own db). Should the time periods be listed in the script or be called from an external file?(because if I have to add time periods manually each month, I'd like to do it once, rather than have to edit each customer script). Also, could I make another external file that contains each customer db and IP for the mysql command, and do a loop calling those variables for that as well?
I'm just trying to wrap my head around what all is needed and the correct approach before I dive in..... THANKS!
Related
I am working at a company and we are in a situation where we have to do mysql queries each day(for the previous day) to check some numbers. I was assigned to it and have about 10 queries to run each morning. I want to know if there is a way so that I can automate these queries each day. I don't mind logging in to workbench to run all of then at once? Only tricky part is we are using clarion date and since I have to run the queries for the previous day the clarion date will have to increase each day.
An example of one of the queries.
select count(*) from item_sales
where code = 12
and date = 79686;
This is an older clariondate but that would have to increase with each day.
The expected outcome would preferably be receiving an email everyday.
You could use a simple script which would :
get the timestamp for the day before
convert it to clarion date format
run the SQL queries, fetch the results
send an email with the results
When it is tested and works, create a simple crontab specifying that you want your script to be ran every day.
You can write below query which takes care of clarion date
query="select count(*) from item_sales
where code = 12
and date = DATEDIFF(now(),'1800-12-28')";
And you can write shell script or php script and set cronjob to run this query and get result and send mail with result.
Shell example:
data=`mysql -u $user --password=$password -e $query`;
echo "Total Count :$data" | mail -s "subject" someone#somewhere.com
I am running an ETL script that loads data from mysql into teradata. The script aims to select all rows later than the timestamp of the last successful run of the bash script. Since I do not have write access to the mysql database, I need to store the last run timestamp with the bash script. Is there an easy way to store the timestamp of a successful run? I was thinking I could have a file that I would touch at the end of the script and then check its mtime, or just parse out the timestamp from a log file. What are some better strategies to do this?
Within your script, use set -e1 so that the script exits immediately if any command within the script fails. Then, at the end, log successful completion with a unix timestamp date +%s.
You can then use SELECT FROM_UNIXTIME(<YOUR TIMESTAMP>, <YOUR MYSQL DATE FORMAT>)2 to pull rows that are newer than the last successful completion.
One big caveat: I would not rely solely on timestamps to approach this problem. I would pull from MySQL with some time overlap and check primary keys for each insert into teradata to avoid inserting duplicates. To follow this approach, just subtract 1800 from <YOUR TIMESTAMP> to ensure a 30 minute overlap.
echo -e "job create --name $completeJobName --definition \"filejdbc --resources=file:/var/lib/wwhs/eligibility/processing/complete/*
--names=file_id,client_id,member_record_number,membe r_first_name,member_middle_initial,member_last_name
,group_id,member_id,date_of_birth,created_by,created_date,modified_by
,modified_date
--tableName=eligibility.eligibility_file_staging_complete\" --deploy" > $completeJobCmdFile
/var/lib/spring_xd/shell/bin/xd-shell script $completeJobCmdFile # Call XD to create the complete job
/var/lib/spring_xd/shell/bin/xd-shell job launch $completeJobName # Run XD processing for simple input files
rm $completeJobCmdFile
In the above code, we are pushing the data via spring XD from a .csv file to a MySQL table. We are then reading the count(*) from the table. Once the count matches the count in the file header, we are initiating the next SQL to process rows from this staging table.
The problem I am facing is that the count(*) is varying. So, when i match it in the if-else condition with header value, it matches.
Later, the count is much lower. The table-type is InnoDB, can that be the issue ?
Please guide me.
While I'm not sure what would cause the count(*) query to be varying in it's results, since you're using a job to import the data, why not launch the next process once the job is complete? In Spring XD, job related events can be listened to so that you can execute dependent processes.
You can read more about the job related notifications here: http://docs.spring.io/spring-xd/docs/1.0.3.RELEASE/reference/html/#_retrieve_job_notifications
I have a debian machine with a mysql server.
On mysql I have a table that contains a number of rows with a datetime field.
How can I execute a php script when the date and time of the machine match with those specified in any mysql record?
a) Mysql triggers ?
b) Deamon that runs in background and checks time every n seconds ?
c) Cron?
Let me know!
Polling the MySQL will work but surely is not efficient, especially when there are a lot of "triggers".
So use a cronjob. Note: MySQL-Queries cannot schedule or cancel cronjobs. So the (i guess) PHP-Script updating that date field also has to schedule the cronjobs. Also, every time the column changes you will have to cancel the previously scheduled cronjob and schedule a new one.
Since you can't run shell command from within Mysql database the MySql Triggers are no good. The MySQL Scheduler is therefore useless too. So you need an external script to help.
I'd suggest to create a (php) script and add it to the crontab to run every minute. Its task would be to check for matching dates in the database and run whatever command you need.
I think that the easiest solution is to use , if possible , the at command.
Basically after inserting the data into mysql do the following:
1)Create (using PHP) a BASH script that runs the php file
script_1.sh should look like this:
!#/bin/bash
php /path/to/file.php
and make it executable with :
exec("chmod +x script_1.sh");
2)create a second BASH script that executes the first at the desired time:
script_2.sh should look like this:
!#/bin/bash
at H:M Y:M:D < script_1.sh
Make script_2 executable and run it with the exec command.
I'm using php with MySQL on macOS.
I would like to select a large amount of emails from a database and perform a dns lookup for each email in my selection using a dig command from the terminal/shell, something like: "dig gmail.com" .
Of course, I can loop this select through php but it will be very slow compared to looping cursor on MySQL.
How to send terminal commands from mysql to the terminal and retrieve answer on macOS?
You can't execute shell commands from within an SQL query (thank god), or else it would be a horrible security vulnerability... You would have to do it from php.
P.S. It is however possible to execute shell commands from the MySQL command line utility
\! ls
...but if I understand your question, it won't help solve your current problem.
(I'm assuming that you really mean ADDR_SPEC when you're talking about email addresses)
but it will be very slow in compare with looping cursor on mysql
No not really. The only difference is that depending on how you implement this the PHP approach requires that you retrieve the entire result set before you start iterating through it. However breaking this up into smaller result sets is trivial.
Also, the limitation on the performance of your algorithm is the speed of DNS lookups - and that's all about latency - if your objective is to make this go faster then you should be running multple requests in parallel.
The next thing you should consider is that you've probably got multiple mailboxes for each MX, e.g. user1#gmail.com, user2#gmail.com.... While if you've got DNS caching setup properly there will be less overhead than going to the source each time, if you're working with a very large data set or will be doing this more than once, it makes a lot more sense to just work with unique MX host values, e.g.
SELECT DISTINCT SUBSTR(addr_spec FROM LOCATE('#', addr_spec)) AS mx2chk
FROM yourtable
WHERE addr_spec LIKE '%#%'
AND (email_checked IS NULL
OR email_checked<NOW() - INTERVAL 300 DAY )
;
Indeed, if your flagging the data then you can use your own database to verify the MX.
using a dig command from terminal/shell
Please don't tell me that you're running a shell from a PHP controlling process to do a DNS lookup?