I'm not an expert but I do have a web front processing orders that have data needing to be input for further logins. Instead of using that database, I created another one with an extra column called status. Initially when orders are processed, they are set to 0. The cron job runs every 3 minutes polling this database for all users with status 0. When run, the cron sets the status of all currently processed users to status 1 (so if there are any that do get input during runtime of the script, it will be processed next time which is only 3 minutes).
After the status of all new users is set to 1, just the password and email fields are dumped to a file and then loaded via "LOAD DATA INFILE" back into the real database that users need to log in with their client. there is no web log in form. It is for emails, just using the IMAP client. However, I do use the root account for the cron since I realized I needed to grant all privs to a user for the dumping of data and if that is going to be it, I might as well just use root to update the status column first, then dump the new data to a file, then load it into the new db and go back and delete all users with status 1. It is a simple 4 line script running mysql from the command line.
Is this a safe bet or am I risking something running a root cron every 3 min? I don't see how I can possibly have an issue since I never use root to process the web stuff. I use a separate mysql user with only INSERT privs for the web front to process new orders. Any comments? I feel like this way I can avoid sql injection even though my mysql user still has limited privs, there always might be something I don't know about.
Is this a safe bet or am I risking something
As long as it's simple LOAD DATA INFILE query - no. However,
Instead of using that database, I created another one with an extra column called status.
Such a flying circus is absolutely unnecessary.
It doesn't protect you from injection anyway.
Instead, you have to use prepared statements for ALL the queries in your application.
Related
I have a big amount of data in a mysql database. I want to poll data from database and push them in a activemq in camel. the connection between database and queue will be lost every 15 minutes. some of the messages are lost during connection interruption. I need to know which messages are lost to poll them again from database. the messages should not be send more that one time. and this should be done without any changes in database schema.(i can not add any Boolean status field to my database).
any suggestion is welcomed.
Essentially, you need to have some unique identifier in the data you pull from the source database. Maybe it is whatever has already been defined as the primary key. Or, maybe the table has some timestamp field. Or, maybe some combination of fields will be unique.
Once you identify that, when you are putting the data into the target, reject any key that is already in the target. You could use Camel's "idempotency" features, but if you are able to check for the key in the target database, you probably won't need anything else.
If you have to make the decision about what to send, but do not have access to your remote database from App #1, you'll need to keep a record on the other side of the firewall.
You would need to do this, even if the connection did not break every 15 minutes...because you could have failures for other reasons.
If you can have an Idempotency database for App#1, another approach could be to transfer data from the local database to some other local table, and read from this. Then you poll this other table, and delete whenever the send is successful.
Example:
It looks like you're using MySql. If both databases are on MySql, you could look into MySql data-replication, rather than using your own app, with Camel.
I have this problem: I'm using a SQL Server 2008R2 backend and MS Access 2000 frontend where some tables are connected via ODBC.
Following Structure (Tables all on SQL-Server):
Import (not connected to Access)
Products (connected via ODBC to Access)
Pricing (connected via ODBC to Access)
I want to fill the Pricing table automatically with some data from Products and Import. This is supposed to run as a SQL Agent job with a T-SQL script.
I want to insert the data from "Products" with following command:
INSERT INTO Pricing (Productnr, Manufacturernr)
(SELECT Productnr, Manufacturernr
FROM Products
WHERE Valid = 1
AND Productnr NOT IN (SELECT Productnr FROM Pricing ));
Right after that the inserted rows are locked for Access, I can't change anything. If I execute sql queries with SQL Server Management Suite or if i start queries as SQL Agent jobs everything works fine.
Why are the rows locked in ms access after the query ran (even if it finished successfully)? and how can I unlock them or make it unlock itself right after the query/job ran?
Thanks
When SQL Server inserts new rows, those new rows are in fact exclusively locked to prevent other transactions from reading or manipulating them - that's by design, and it's a good thing! And it's something you cannot change - you cannot insert without those locks.
You can unlock them by committing the transaction that they're being inserted under - once they're committed to SQL Server, you can access them again normally.
The error message i get says, that the dataset has been changed by another user and if i save it, i would undo the changes of the other user. (and asks me for copying into clipboard).
This is different from "locked", and completely normal.
If you have a ODBC linked table (or form based on the table) open, and change data in the backend, Access doesn't know about the change.
You need to do a full requery (Shift+F9) in Access to reload the data, afterwards all records can be edited again.
Got the solution for my Problem now.
I had to add a timestamp row into the pricing table, so access could recognize the change.
Access loads the data into the front end when the table is first accessed. If something in the backend changes the data, you need Access to refresh it first, before you can edit it from the front end (or see the changes).
Do this by (in Access) by closing and reopening the table, or switching to the table and pressing shift-F9 as Andre suggested, or programmatically using a requery statement. You must requery, not refresh, for it to release the locks and register the changes made in SQL.
I am currently working on a j2ee web application. The application features a way for users to reset their passwords if they forget them.
I have a database table with 3 columns: username, key, and timestamp.
When the user requests a password change, I add an entry in that table with their username and a random key (making sure that their are no duplicate keys in the table, also that a user can only appear once in the table). I also add the current time. I then send them an e-mail with a link to the application that contains their key, something like:
mysite.com/app/reset?key=abcxyz123
The servlet that handles this request looks at the key in the url to find the matching entry in the reset table to determine which user the key belongs to. If the key doesn't match an entry, I show an error page, if it does, I show the password reset screen. Once the user changes their password, I manually delete the entry from that reset table.
I am trying to implement the equivalent of a time to live for the password reset links, so that I don't have entries loitering in the table unnecessarily, and I thought of 2 options, the first of which I have implemented:
1) Create an EJB Timer that fires every minute that will delete entries in the reset table where the timestamp is older than 30 minutes. This is a manual process in that I am using hibernate as my jpa implementation, so I retrieve all the entries from the table, examine their timestamps, and delete the old ones.
2) Create a database job that deletes rows over a certain age?
My question is, does anyone see any drawbacks to the first approach, and second, is the 2nd option even possible with mysql? I figure that if I can use the 2nd approach, I can get rid of the timer, and let the database handle the time to live aspect of the password reset links, and that may be more efficient.
I haven't been doing j2ee development for that long, but based on the knowledge that I have, these seemed like 2 logical approaches. I welcome any input.
3) Create script that will connect to db, execute delete, disconnect. Then you can schedule this script via operating system e.g. crontab.
Regarding option 1 - Drawback of that solution is that it uses application server resources for stuff that can be done on database only and is not dependent/uses any application logic.
Benefit is that whole app is self contained and you don't need any additional installation/setup task on database as with 2 and 3.
I am very new to this and a good friend is in a bind. I am at my wits end. I have used gui's like navicat and sqlyog to do this but, only manually.
His band info data (schedules and whatnot) is in a MYSQL database on a server (admin server).
I am putting together a basic site for him written in Perl that grabs data from a database that resides on my server (public server) and displays schedule info, previous gig newsletters and some fan interaction.
He uses an administrative interface, which he likes and desires to keep, to manage the data on the admin server.
The admin server db has a bunch of tables and even table data the public db does not need.
So, I created tables on the public side that only contain relevant data.
I basically used a gui to export the data, then insert to the public side whenever he made updates to the admin db (copy and paste).
(FYI I am using DBI module to access the data in/via my public db perl script.)
I could access the admin server directly to grab only the data I need but, the whole purpose of this is to "mirror" the data not access the admin server on every query. Also, some tables are THOUSANDS of rows and parsing every row in a loop seemed too "bulky" to me. There is however a "time" column which could be utilized to compare to.
I cannot "sync" due to the fact that the structures are different, I only need the relevant table data from only three tables.
SO...... I desire to automate!
I read "copy" was a fast way but, my findings in how to implement were too advanced for my level.
I do not have the luxury of placing a script on the admin server to notify when there was an update.
1- I would like to set up a script to check a table to see if a row was updated or added on the admin servers db.
I would then desire to update or insert the new or changed data to the public servers db.
This "check" could be set up in a cron job I guess or triggered when a specific page loads on the public side. (the same sub routine called by the cron I would assume).
This data does not need to be "real time" but, if he updates something it would be nice to have it appear as quickly as possible.
I have done much reading, module research and experimenting but, here I am again at stackoverflow where I always get great advice and examples.
Much of the terminology is still quite over my head so verbose examples with explanations really help me learn quicker.
Thanks in advance.
The two terms you are looking for are either "replication" or "ETL".
First, replication approach.
Let's assume your admin server has tables T1, T2, T3 and your public server has tables TP1, TP2.
So, what you want to do (since you have different table structres as you said) is:
Take the tables from public server, and create exact copies of those tables on the admin server (TP1 and TP2).
Create a trigger on the admin server's original tables to populate the data from T1/T2/T3 into admin server's copy of TP1/TP2.
You will also need to do initial data population from T1/T2/T3 into admin server's copy of TP1/TP2. Duh.
Set up the "replication" from admin server's TP1/TP2 to public server's TP1/TP2
A different approach is to write a program (such programs are called ETL - Extract-Transform-Load) which will extract the data from T1/T2/T3 on admin server (the "E" part of "ETL"), massage the data into format suitable for loading into TP1/TP2 tables (the "T" part of "ETL"), transfer (via ftp/scp/whatnot) those files to public server, and the second half of the program (the "L") part will load the files into the tables TP1/TP2 on public server. Both halfs of the program would be launched by cron or your scheduler of choice.
There's an article with a very good example of how to start building Perl/MySQL ETL: http://oreilly.com/pub/a/databases/2007/04/12/building-a-data-warehouse-with-mysql-and-perl.html?page=2
If you prefer not to build your own, here's a list of open source ETL systems, never used any of them so no opinions on their usability/quality: http://www.manageability.org/blog/stuff/open-source-etl
I think you've misunderstood ETL as a problem domain, which is complicated, versus ETL as a one-off solution, which is often not much harder than writing a report. Unless I've totally misunderstood your problem, you don't need a general ETL solution, you need a one-off solution that works on a handful of tables and a few thousand rows. ETL and Schema mapping sound scarier than they are for a single job. (The generalization, scaling, change-management, and OLTP-to-OLAP support of ETL are where it gets especially difficult.) If you can use Perl to write a report out of a SQL database, you probably know enough to handle the ETL involved here.
1- I would like to set up a script to check a table to see if a row was updated or added on the admin servers db. I would then desire to update or insert the new or changed data to the public servers db.
If every table you need to pull from has an update timestamp column, then your cron job includes some SELECT statements with WHERE clauses based on the last time the cron job ran to get only the updates. Tables without an update timestamp will probably need a full dump.
I'd use a one-to-one table mapping unless normalization was required... just simpler to my opinion. Why complicate it with "big" schema changes if you don't have to?
some tables are THOUSANDS of rows and parsing every row in a loop seemed too "bulky" to me.
Limit your queries to only the columns you need (and if there are no BLOBs or exceptionally big columns in what you need) a few thousand rows should not be a problem via DBI with a FETCHALL method. Loop all you want locally, just make as few trips to the remote database as possible.
If a row is has a newer date, update it. I will also have to check for new rows for insertion.
Each table needs one SELECT ... WHERE updated_timestamp_columnname > last_cron_run_timestamp. That result set will contain all rows with newer timestamps, which contains newly inserted rows (if the timestamp column behaves like I'd expect). For updating your local database, check out MySQL's ON DUPLICATE KEY UPDATE syntax... this will let you do it in one step.
... how to implement were too advanced for my level ...
Yes, I have actually done this already but, I have to manually update...
Some questions to help us understand your level... Are you hitting the database from the mysql client command-line or from a GUI? Have you gotten to the point where you've wrapped your SQL queries in Perl and DBI, yet?
If the two databases have different, you'll need an ETL solution to map from one schema to another.
If the schemas are the same, all you have to do is replicate the data from one to the other.
Why not just create identical structure on the 'slave' server to the master server. Then create a small table that keeps track of the last timestamp or id for the updated tables.
Then select from the master all rows changed since the last timestamp or greater than the id. Insert them into the matching table on the slave server.
You will need to be careful of updated rows. If a row on the master is updated but the timestamp doesn't change then how will you tell which rows to fetch? If that's not an issue the process is quite simple.
If it is an issue then you need to be more sophisticated, but without knowing the data structure and update mechanism its a goose chase to give pointers on it.
The script could be called by cron every so often to update the changes.
if the database structures must be different on the two servers then a simple translation step may need to be added, but most of the time that can be done within the sql select statement and maybe a join or two.
I am wanting to create a VB.Net app that checks a database and then prints out the latest orders, the database is mysql and i need it to periodatically check and only print new orders i.e. not print ones it has already printed out.
Any advice would be greatly appriciated.
Kyle
My suggestion is to add a printed flag to your database, i.e. add a printed field in the database which stores 0 by default denoting the entry has not been printed and a 1 that denotes the entry has been printed.
I would then create an application which checks the database table for all entries where flag = 0.
These entries are the ones that need printed.
Print each entry an set its flag to 1 in the database. Setting the flag to 1 ensures that the next time you poll the database the entries will be ignored.
I'd recommend breaking the problem down into pieces:
"checks a database" - you'll need to create a connection to a relational database. You'll have to set up either an ADO.NET or ODBC connection to MySQL.
You don't say if the database is running on the same machine as the app. If it's not, you'll have to be able to connect to it on the network.
Make sure that you GRANT the appropriate permissions in MySQL. Don't just use the admin credentials to log in. Set up a separate ID for your app.
"then prints out" - your app will have to find a way to acquire information about a printer. You don't say if it's connected directly to the server on which the job will run or if it's a networked printer. Be sure your app can see the networked printer if it's the latter.
"the latest orders, the database is mysql" - try writing a separate query to test getting only the latest orders.
Does your schema include a way to mark which orders have been printed and when? Your app should INSERT a record noting the timestamp when each order is printed. You can JOIN on this table to figure out what the latest, unprinted orders are.
"and i need it to periodically" - your app needs to be a cron job that's run on a schedule that you specify. Every weekday, hourly? Once a night? You decide when you set up the cron job.
You could try using LINQ and dbLINQ for connecting with MySQL. If you are using .NET Framework 3.5 you may as well take advantage of something like LINQ to speed up development time.
If using WPF for the client application you could use a FlowDocument to layout the order details and then handle printing of the orders. Once they are printed you can update the database or store details locally to mark that order as printed. It depends on your database schema.