I'm doing a web application and i want to do is if user don't like the changes or he makes a mistake, he could rollback the changes, and if he likes, save it. I'm using Perl with DBI module and MySQL.
First I send the data to update to a other Perl file, in that page I perform the update and i return the flow to the first page and show the changes to the user.
So I am wondering if its possible to persist or keep alive the transaction through HTTP request or how to do the transaction?
I did the following:
$dbh->{AutoCommit} = 0;
$dbh-do("update ...")
I'm a beginner with Perl and DBI so any answer will be appreciated
How complex a transaction is it? One table, or multiple tables and complex relationships?
If it's a single table, it might be a lot simpler for the confirmation page to show the before (DBI) values and the after (form) values, and perform the transaction following a 'commit' from there.
Apache::DBI and other ORM modules do exist that attempt to persist database connections, but given each web-server process has its own memory-space, you quickly get into some pretty hairy problems. Not for the noob, I would suggest.
I would also recommend that before you go too far with hand-crafted DBI, have a look at some of the object-relational mapping modules out there. DBIx::Class is the most popular/actively maintained one.
Related
I have an Android frontend.
The Android client makes a request to my NodeJS backend server and waits for a reply.
The NodeJS reads a value in a MySQL database record (without send it back to the client) and waits that its value changes (an other Android client changes it with a different request in less than 20 seconds), then when it happens the NodeJS server replies to client with that new value.
Now, my approach was to create a MySQL trigger and when there is an update in that table it notifies the NodeJS server, but I don't know how to do it.
I thought two easiers ways with busy waiting for give you an idea:
the client sends a request every 100ms and the server replies with the SELECT of that value, then when the client gets a different reply it means that the value changed;
the client sends a request and the server every 100ms makes a SELECT query until it gets a different value, then it replies with value to the client.
Both are bruteforce approach, I would like to don't use them for obvious reasons. Any idea?
Thank you.
Welcome to StackOverflow. Your question is very broad and I don't think I can give you a very detailed answer here. However, I think I can give you some hints and ideas that may help you along the road.
Mysql has no internal way to running external commands as a trigger action. To my knowledge there exists a workaround in form of external plugin (UDF) that allowes mysql to do what you want. See Invoking a PHP script from a MySQL trigger and https://patternbuffer.wordpress.com/2012/09/14/triggering-shell-script-from-mysql/
However, I think going this route is a sign of using the wrong architecture or wrong design patterns for what you want to achieve.
First idea that pops into my mind is this: Would it not be possible to introduce some sort of messaging from the second nodjs request (the one that changes the DB) to the first one (the one that needs an update when the DB value changes)? That way the the first nodejs "process" only need to query the DB upon real changes when it receives a message.
Another question would be, if you actually need to use mysql, or if some other datastore might be better suited. Redis comes to my mind, since with redis you could implement the messaging to the nodejs at the same time...
In general polling is not always the wrong choice. Especially for high load environments where you expect in each poll to collect some data. Polling makes impossible to overload the processing capacity for the data retrieving side, since this process controls the maximum throughput. With pushing you give that control to the pushing side and if there is many such pushing sides, control is hard to achieve.
If I was you I would look into redis and learn how elegantly its publish/subscribe mechanism can be used as messaging system in your context. See https://redis.io/topics/pubsub
I have the following architechture:
I Import data from a SQL datbase into a mongodb. I use the importer to migrate data into the mongodb that provides the data to a website via an API.
The importing can take a couple of minutes and if it fails I would like to be able to either rollback (it would be awsome to be able to rollback multiple imports) or drop the database/collections of the uncommited rows (if you think of it as SQL transactions).
I tried to import everything into a transactions collection that, on success, moved the data into the correct collection. This took way to much time to be performant. I also tried the solution of importing into a temp db and then swapping them. But then I run into problems if someone e.g. registers a new user on the website after the db-copy but before the importing is done (that user will be lost when swapping).
How can a perform an import in a safe way and not have the most basic concurrency problems?
EDIT:
To clarify the solution:
I will run the importer in a cron job, at least once a day. I currently keep a timestamp for the latest synchronization and select everything that is newer than that from the SQL-db. Things will automagically appear in the SQL-db over time.
At the end of the importing I run a downloader that downloads all the images from urls in the SQL db.
I don't want to start a new sync before the images are downloaded since that could result in strange behaviour.
In cases like this, where you need to move data between very different types of databases, you're going to tend to want something really reliable, robust, and, most-importantly, with its primary concern being transferring data and doing it well. Node.js is wonderful, but I would highly recommend you find some tool out there that is focused on only doing the transfer/mapping/etc. and use that regardless of what language/technology it uses. There's nothing about node.js, as much as I love it, that particularly recommends it for this sort of thing (e.g., it's best characteristics don't necessarily make it good for this sort of transfer/migration).
Better to find a mature, well-developed library that handles this sort of thing :)
Some tools/resources that turned up in my research:
mongify
SQL to MongoDB Mapping Chart
Would love if people could suggest more in comments :)
I am writing a multilingual CMS where admin can add and delete languages. When they add a new language, I would copy all the rows from multiple table with language_id = 1 and insert them with the newly created language_id .
I'm using PHP so the database copying and inserting process would probably be done asynchronously. The problem is the user might add new content during the process and there is a chance both language would not have the same number of rows in the end.
I could probably lock all the table involved but since the user of the CMS is not tech savvy, I don't want them to see a generic error message when they try to create or update record.
I would much prefer to show them a customize message notifying them the system is converting language. But doing so require me to know that tables are being locked.
I should add that CRUD are mostly done by one person at any time so there should be less difficulties.
Any help would be greatly appreciate.
I would just lock the table where new rows are inserted while doing the "creating new language" operation. That way any new insert would automatic wait until the "creating new language" operation was finished, and all the user would see was a slight delay.
This is assuming that the "creating new language" can be done i a few seconds but I can't imagine this being a problem.
What do you mean by I should add that CRUD are mostly done by one person at any time so there should be less difficulties.? if your assumption is correct, how in the world developers are making concurrent requests, same time ?
Generally you will only have one DAO (Data access object) and all your code will come to that which will make a single connection to database and it will be synchronized all the time. Did you follow this practice ? php asynchronous means while pinging the server, ajax request will be made. Not while communicating with DB. If there are so many clients connecting, there will be a pattern of connection pooling. Think of it, does this make any sense ?
Does it make sense to use a combination of MySQL and MongoDB. What im trying to do basically is use MySQl as a "raw data backup" type thing where all the data is being stored there but not being read from there.
The Data is also stored at the same time in MongoDB and the reads happen only from mongoDB because I dont have to do joins and stuff.
For example assume in building NetFlix
in mysql i have a table for Comments and Movies. Then when a comment is made In mySQL i just add it to the table, and in MongoDB i update the movies document to hold this new comment.
And then when i want to get movies and comments i just grab the document from mongoDb.
My main concern is because of how "new" mongodb is compared to MySQL. In the case where something unexpected happens in Mongo, we have a MySQL backup where we can quickly get the app fallback to mysql and memcached.
On paper it may sound like a good idea, but there are a lot of things you will have to take into account. This will make your application way more complex than you may think. I'll give you some examples.
Two different systems
You'll be dealing with two different systems, each with its own behavior. These different behaviors will make it quite hard to keep everything synchronized.
What will happen when a write in MongoDB fails, but succeeds in MySQL?
Or the other way around, when a column constraint in MySQL is violated, for example?
What if a deadlock occurs in MySQL?
What if your schema changes? One migration is painful, but you'll have to do two migrations.
You'd have to deal with some of these scenarios in your application code. Which brings me to the next point.
Two data access layers
Your application needs to interact with two external systems, so you'll need to write two data access layers.
These layers both have to be tested.
Both have to be maintained.
The rest of your application needs to communicate with both layers.
Abstracting away both layers will introduce another layer, which will further increase complexity.
Chance of cascading failure
Should MongoDB fail, the application will fall back to MySQL and memcached. But at this point memcached will be empty. So each request right after MongoDB fails will hit the database. If you have a high-traffic site, this can easily take down MySQL as well.
Word of advice
Identify all possible ways in which you think 'something unexpected' can happen with MongoDB. Then use the most simple solution for each individual case. For example, if it's data loss you're worried about, use replication. If it's data corruption, use delayed replication.
I'm creating a Twitter application, and every time user updates the page it reloads the newest messages from Twitter and saves them to local database, unless they have already been created before. This works well in development environment (database: sqlite3), but in production environment (mysql) it always creates messages again, even though they already have been created.
Message creation is checked by twitter_id, that each message has:
msg = Message.find_by_twitter_id(message_hash['id'].to_i)
if msg.nil?
# creates new message from message_hash (and possibly new user too)
end
msg.save
Apparently, in production environment it's unable to find the messages by twitter id for some reason (when I look at the database it has saved all the attributes correctly before).
With this long introduction, I guess my main question is how do I debug this? (unless you already have an answer to the main problem, of course :) When I look in the production.log, it only shows something like:
Processing MainPageController#feeds (for 91.154.7.200 at 2010-01-16 14:35:36) [GET]
Rendering template within layouts/application
Rendering main_page/feeds
Completed in 9774ms (View: 164, DB: 874) | 200 OK [http://www.tweets.vidious.net/]
...but not the database requests, logger.debug texts, or anything that could help me find the problem.
You can change the log level in production by setting the log level in config/environment/production.rb
config.log_level = :debug
That will log the sql and everything else you are used to seeing in dev - it will slow down the app a bit, and your logs will be large, so use judiciously.
But as to the actual problem behind the question...
Could it be because of multiple connections accessing mysql?
If the twitter entries have not yet been committed, then a query for them from another connection will not return them, so if your query for them is called before the commit, then you won't find them, and will instead insert the same entries again. This is much more likely to happen in a production environment with many users than with you alone testing on sqlite.
Since you are using mysql, you could use a unique key on the twitter id to prevent dupes, then catch the ActiveRecord exception if you try to insert a dupe. But this means handling an error, which is not a pretty way to handle this (though I recommend doing it as a back up means of prevent dupes - mysql is good for this, use it).
You should also prevent the attempt to insert the dupes. One way is to use a lock on a common record, say the User record which all the tweets are related to, so that another process cannot try to add tweets for the user until it can get that lock (which you will only free once the transaction is done), and so prevent simultaneous commits of the same info.
I ran into a similar issue while saving emails to a database, I agree with Andrew, set the log level to debug for more information on what exactly is happening.
As for the actual problem, you can try adding a unique index to the database that will prevent two items from being saved with the same parameters. This is like the validates_uniqueness but at the database level, and is very effective: Mysql Constraign Database Entries in Rails.
For example if you wanted no message objects in your database that had a duplicate body of text, and a duplicate twitter id (which would mean the same person tweeted the same text). Then you can add this to your migration:
add_index( :message, [:twitter_id, :body] , :unique => true)
It takes a small amount of time after you tell an object in Rails to save, before it actually gets in the database, thats maybe why the query for the id doesn't find anything yet.
For your production server, I would recommend setting up a rollbar to report you all of the unhandled errors and exceptions in your production servers.
You can also store a bunch of useful information, like http request, requested users, code which invoked an error and many more or sends email notifications each time some unhandled exceptions happened on your production server.
Here is a simple article about debugging in rails that could help you out.