I have a python app that has an admin dashboard.
There I have a button called "Update DB".
(The app uses MySQL and SQLAlchemy)
Once it's clicked it makes an API call and gets a list of data and writes that to the DB, and if there are new records returned by the API call it adds them and does not duplicate currently existing records.
However if API call returns less items, it does not delete them.
Since I don't even have a "starting to google" point I need some guidance of what type of SQL query should my app be making.
Like once button is clicked ,it needs to go through all the rows:
do the changes to the updated records that existed
add new ones if there are any returned by the API call
delete ones that API call did not return.
What is this operation called or how can I accomplish this in mysql?
Once I find out about this I'll see how can I do that in SQLAlchemy.
You may want to set a timestamp column to the time of latest action on the table and have a background thread remove old rows as another action. I don't know of any atomic action that may perform the desired data reformation. Another option might be satisfactory is to write the replacement batch to a staging table, rename both versions (swap) and drop the old table. HTH
Related
We would like to perform incremental loading in DataStage (in parallel environement). Exactly load only the delta between the previous load and the new one (for create, update, delete the records in DWH).
We would like to store the last key recovered during the previous load to be able to restart the request from the next record on a new loading.
We have already successfully used a parameter to filter the SQL load query at runtime. Unfortunately, we have not yet found the possibility to retrieve the last key (max (Key) - Aggregator?) And to store it in this parameter.
Which stage to use, to output a single value in the same parallel job, and then store to parameter ?
Any ideas ?
Thanks for your help.
Think about getting the max value from your target - it is most probably a database and a max() is easy to do.
Check out my post about getting some data from the "stream" to a parameter
Thank's Michael,
I've found Head stage to get the max(LastRowId) in the same job, with 'All rows (after skip) = False), and 'Number of Rows (Per partition)=1. And I run the job in sequential mode...
That's worked fine.
The title nearly says it all.
I would like to write a trigger that:
Uses a table called "audit_trail" with fields table_name, by, timestamp, operation, contents where contents is in JSON format
The trigger listens for update or insert on each table
If the table has a column called last_modified_by, then:
Make up a JSON version of the record updated/inserted
Add a record to the audit_trail table, with all relevant fields including contents which would have the JSON representation of the record updated/inserted
Is this technically possible with MySql? I really don't want to code this into the application itself, as it would be messy.
Please note that I am fully aware about the limitation about recording this info as JSON (hard to query, etc.). The only requirement my app has is that an admin must be able to see the "history" of a record, of when/who modified it.
While this is quite trivial, there are things I just cannot work out:
Things I can't work out:
How do you write a trigger that will get triggered on insert or update on ANY table
How to get the JSON version of a record
How to get the trigger to store the JSON onto the contents column
Ideas?
I am currently working on a j2ee web application. The application features a way for users to reset their passwords if they forget them.
I have a database table with 3 columns: username, key, and timestamp.
When the user requests a password change, I add an entry in that table with their username and a random key (making sure that their are no duplicate keys in the table, also that a user can only appear once in the table). I also add the current time. I then send them an e-mail with a link to the application that contains their key, something like:
mysite.com/app/reset?key=abcxyz123
The servlet that handles this request looks at the key in the url to find the matching entry in the reset table to determine which user the key belongs to. If the key doesn't match an entry, I show an error page, if it does, I show the password reset screen. Once the user changes their password, I manually delete the entry from that reset table.
I am trying to implement the equivalent of a time to live for the password reset links, so that I don't have entries loitering in the table unnecessarily, and I thought of 2 options, the first of which I have implemented:
1) Create an EJB Timer that fires every minute that will delete entries in the reset table where the timestamp is older than 30 minutes. This is a manual process in that I am using hibernate as my jpa implementation, so I retrieve all the entries from the table, examine their timestamps, and delete the old ones.
2) Create a database job that deletes rows over a certain age?
My question is, does anyone see any drawbacks to the first approach, and second, is the 2nd option even possible with mysql? I figure that if I can use the 2nd approach, I can get rid of the timer, and let the database handle the time to live aspect of the password reset links, and that may be more efficient.
I haven't been doing j2ee development for that long, but based on the knowledge that I have, these seemed like 2 logical approaches. I welcome any input.
3) Create script that will connect to db, execute delete, disconnect. Then you can schedule this script via operating system e.g. crontab.
Regarding option 1 - Drawback of that solution is that it uses application server resources for stuff that can be done on database only and is not dependent/uses any application logic.
Benefit is that whole app is self contained and you don't need any additional installation/setup task on database as with 2 and 3.
I am working on a project and would like a notification each time a new record is added to a specific table in a MySQL DB. I would like a small popup to be displayed each time a new record is added, but without refreshing the page. I heard AJAX was the way to go, but am not very familiar with it.
Nope it won't. The database won't tell you when a record is inserted. You can use AJAX to send a request to the server. Then, that server can query for changes. It can send a response indicating whether there is a change. Then, the AJAX request's response handler can show a message accordingly.
But implementing this will cause quite a load on both the webserver and the database server. So if you do this, choose the timing wisely. Don't execute this procedure 10 times a second or you'll kill your server as soon as you hit a 100 visitors.
To solve your problem, break it up in two pieces:
1. Get the actual AJAX request to work. Let the server return dummy values and try to handle them correctly. Hint: Use JQuery.ajax (or even JQuery.get) to ease your life.
2. Get the server to query for changes. If you want to monitor a single table this can easily be done. Add a timestamp column to the table if you don't already have one. You can configure it so that it will be updated each time the table is updates. Then, query for the highest timestamp. Don't forget to add an index to that column!
You can experiment with other solutions too. Add a trigger that alters a date/time in a different table. That way, the polling only needs to query that single column instead of the 'max' query.
To handle the change correctly, I think its best to let Javascript hold on to the last timestamp. Send the timestamp back in the response. Javascript can compare the timestamp to the last timestamp and show a message if needed. This way, you won't need to keep the timestamps in the session.
Is it possible for MySQL database to invoke an external exe file when a new row is added to one of the tables in the database?
I need to monitor the changes in the database, so when a relevant change is made, I need to do some batch jobs outside the database.
Chad Birch has a good idea with using MySQL triggers and a user-defined function. You can find out more in the MySQL CREATE TRIGGER Syntax reference.
But are you sure that you need to call an executable right away when the row is inserted? It seems like that method will be prone to failure, because MySQL might spawn multiple instances of the executable at the same time. If your executable fails, then there will be no record of which rows have been processed yet and which have not. If MySQL is waiting for your executable to finish, then inserting rows might be very slow. Also, if Chad Birch is right, then will have to recompile MySQL, so it sounds difficult.
Instead of calling the executable directly from MySQL, I would use triggers to simply record the fact that a row got INSERTED or UPDATED: record that information in the database, either with new columns in your existing tables or with a brand new table called say database_changes. Then make an external program that regularly reads the information from the database, processes it, and marks it as done.
Your specific solution will depend on what parameters the external program actually needs.
If your external program needs to know which row was inserted, then your solution could be like this: Make a new table called database_changes with fields date, table_name, and row_id, and for all the other tables, make a trigger like this:
CREATE TRIGGER `my_trigger`
AFTER INSERT ON `table_name`
FOR EACH ROW BEGIN
INSERT INTO `database_changes` (`date`, `table_name`, `row_id`)
VALUES (NOW(), "table_name", NEW.id)
END;
Then your batch script can do something like this:
Select the first row in the database_changes table.
Process it.
Remove it.
Repeat 1-3 until database_changes is empty.
With this approach, you can have more control over when and how the data gets processed, and you can easily check to see whether the data actually got processed (just check to see if the database_changes table is empty).
you could do what replication does: hang on the 'binary log'. setup your server as a 'master server', and instead of adding a 'slave server', run mysqlbinlog. you'll get a stream of every command that modifies your database.
step in 'between' the client and server: check MySQLProxy. you point it to your server, and point your client(s) to the proxy. it lets you interpose Lua scripts to monitor, analyze or transform any SQL command.
I think it's going to require adding a User-Defined Function, which I believe requires recompilation:
MySQL FAQ - Triggers: Can triggers call an external application through a UDF?
I think it's really a MUCH better idea to have some external process poll changes to the table and execute the external program - you could also have a column which contains the status of this external program run (e.g. "pending", "failed", "success") - and just select rows where that column is "pending".
It depends how soon the batch job needs to be run. If it's something which needs to be run "sooner or later" and can fail and need to be retried, definitely have an app polling the table and running them as necessary.