I have a database that is read-only in nature. So it seems to make sense to boost performance by setting it to readonly. Then I get updates for this database once a month. At that point I would like to switch to readwrite, do the updates, and switch back to readonly. I've looked around and haven't seen too much mention of this. Are there any problems I'm going to run into with this approach or is this as safe and basic as you get?
thanks
No problems with this approach.
Remember to rebuild index and update stats before making it read only again.
Then again, keep it simple and read write always...?
Related
I am using Octopus gem to handle database sharding in my application. I have a master and a slave. The insert query always hits the master and the read goes to slave.
But I am facing a weird issue like, after inserting a record and when I try to fetch it, record is not found. This is affecting my whole application.
I tried to resolve this issue by the following code.
Model.using(:master).where(id: 250)
This will force the model to fetch record from master rather than from slave. But if we add this everywhere in the application there is no point of sharding.
Any solution for this?
Thanks in advance.
Welcome to fun world of asynchronous replication.
Generally, when updating data to your master database, the data is replicated asynchronously to the slaves, meaning it will arrive there at any later point in time. Unfortunately, you can't known when that will happen as the only thing typically guaranteed is the order of updates to the slave, not when they will happen.
Often, you'll try to keep the replication delay rather small but you can't ignore it. Generally, when using asynchronous replication, you have to think critically about your data access strategies to avoid presenting unwanted stale data.
Sharding and replication definitely doesn't come for free. Database systems try hard to implement strongly defined levels of atomicity via transactions but due to CAP, things get more complicated (or sometimes impossible) when introducing a distributed system.
There isn't a generally correct answer for this issue as it is not directly clear, which data can be stale and which doesn't. Think about your access patterns and chose the appropriate server. Often, the simplest answer is to get rid of sharding and replication completely and to simply use a bigger server.
We have a separate RDS Instance to handle session state tables, however found that the session DB load is very low. if we can convert the instance handling session as a Read Replica of the main DB, then we can use it for read-only tasks that are safe even with a large lag in the copy.
Has anyone done something like this on RDS (Is it possible and safe)? Should I watch out for any serious side effects? Any links or help in understanding this better would help.
http://aws.amazon.com/rds/faqs/#95 attempts to answer the question but am looking for more insights.
Yes, it is possible. I am using it with success using RDS, for a specific case of local cache.
You need to set the read_only parameter on your replica to 0. I've had to reboot my server in order for that parameter to work.
It's going to work nicely if use different table names, as RDS doesn't allow you to set: replicate-ignore-table parameter.
Remember there musn't be any data collision between master<>slave. If there is a statement which works ok on MASTER, but fails on SLAVE, then you've just broke your replication. That might happen e.g. when you've created table on SLAVE first then after some time you've added that table to MASTER. The CREATE statement will work clean on MASTER, but fail on SLAVE, as table already exist.
Assuming, you need to be really careful, allowing your application to write to SLAVE. If you forget / or make a mistake and start writing to read replica for some of your other data, in the end you might lose data or experience hard to debug issues.
There's not a lot to add -- the only normal scenario that really makes sense on a pure read replica is things like adding a few indexes and the like if its used primarily for reporting or something else read-intensive.
If you're trying to pre-calculate a lot of data and otherwise modify what's on the read replica you need to be really careful you're not changing data -- if the read is no longer consistent then you're in trouble :)
If you're curious about what happens if you change data on the slave and the master tries to update it, you're already heading down the wrong path IMHO.
TL;DR Don't do it unless you really know what you're doing and you understand all the ramifications.
And bluntly, MySQL replication can be quirky in my experience, so even knowing what is supposed to happen and what does happen if there's as the master tries to write updated data to slave you've also updated.... who knows.
I have a script that is part of my deployment process to push DB changes to the production server. If the script corrupts my data for some reason (a bad update), it is tough to recover.
One way to solve this is to shut down the application to users while updating, so if a problem occurs, just go back to the backup I made before deploying.
But I have heard of others who deploy and keep their site live... how would you go about doing this, and if you failed, how could you recover the data that came in since you took your backup before deploying?
This is a tricky problem in general, like with many things in database administration. There are basically three ways to approach this:
Avoid failure at all costs.
Lock everything down (and make the upgrade really fast).
It's OK to lose data.
If you have a complex system, isolate your components according to these or similar categories.
Have a staging system to test upgrades. The staging system is more or less a copy of the production system; it's separate from the test system. Another thing is to have an audit or logging system that you can refer to if you need to replay data.
The real problem is if you notice much later that your upgrade was faulty. Then you're quite screwed.
How big is your database? Can you afford to lose data that was updated while the customer were using it and before you had to go to the backup? Every plan for deployment involves some compromises somewhere, and you've got to decide which compromises are the least painful for what you want to do.
For simple websites running just pgsql, you can disconnect clients, and run the entire update in one big transaction. If any part fails, the whole thing rolls back and it's like you never did anything. Sadly this doesn't work exactly the same for other dbs, but with flashback or whatever oracle calls it you can get something similar.
For bigger, complex websites running on top of a replicated db server set, things get much more complex much more quickly. Where I've worked we've used Slony, and it doesn't take nicely to playing with others when you're deploying DDL changes, and you pretty much HAVE to take all the customers offline while deploying DDL. However, the downtime is measured in minutes for us, even with databases approaching 1TB in size.
I've been doing a lot of research, reading on replication, etc but just not sure as to what mysql solution would work.
This is what I'm looking at:
when my mysql fails for some reason or there are certain queries that are taking really long to execute and locking some tables, I want the other insert/update/select queries to still function at normal speed without having to wait for locks to be released or for the main database to be back up. I'm thinking there should be a second mysql server for this to happen, but is what I mentioned possible even if there is and would it involve a lot of change in my existing programming logic?
when my database is being backed up, I would still like my site to function normally, all inserts/selects/updates should function as normal.
when I need to alter a large table, I wouldn't like it to affect my application, there should be a backup server to work from.
So what do I need to do to get all this done and also would it require changing plenty of existing coding to suit the new set up? [My site has a lot of reads and writes]
There's no easy way. You're asking for a highly-available MySQL-based setup, and that requires a lot of work at the server and client ends.
Some issues, for example:
when I need to alter a large table, I wouldn't like it to affect my application, there should be a backup server to work from.
If you're altering the table, you can't trivially create a copy to work from during the update. What about the changes that are made to your copy while the first update is taking place?
Have a search for "High Availability MySQL". It's mostly a solved problem, but the solution depends heavily on your exact requirements. You cannot just ask for "I want my SQL server to run at full speed always forever no matter what I throw at it".
Not a MySQL specific answer, but a general one. Have a read only copy of your DB for site to render, which will be synced to the master DB regularly. This way, you can keep your site working even if the master DB is under load/lock due to insert/delete. For efficiency, keep this copy as denormalized as you can.
I'm rewriting a PHP+MySQL site that averages 40-50 hits a day using Django.
Is SQLite a suitable database to use here? Are there any advantages/disadvantages between them?
I'm just using the db to store a blog and the users who can edit it. I am using fulltext search for the blog search, but no complex joins anywhere.
40-50 hits per day is very small and SQLLite can be used without any problem.
MySql might be better once you will get more hit because it handles in a better way multiple connexion (lock isn't the same with MySql and SqlLite).
The major problem with sqlite is concurrency. If you expect 40-50 hits a day, that's probably a non-issue. However, if that load increases you should be ready to migrate to a database daemon such as MySQL - better abstract your database specific code to make such a switch as painless as possible.
The performance section of the SQLite wiki might be of use to you.
Since you're already using an adequate database, I don't see a reason to migrate to a smaller one.
While sqlite might be perfectly adequate, too - changing to a less-capable platform from a more-capable one doesn't seem the best choice :)
SQLite will work just fine for you. It sounds as though you're largely using the database as read-only (with occasional writes to update the content). SQLite excels at this kind of access pattern. The only place where SQLite chokes is when you have a lot of writes to a database, because once a process attempts to write the file is locked until the write is complete. Also, if you do lots of writes (like updating rows in a loop) you should look into putting all those writes into a transaction - while the file is locked once the transaction hits a write query, the updates themselves take much less time because they're written to the file at once and not individually.
SQLite would be fine for this level of traffic. It actually performs quite well, the only thing that it is lacking is caching of data and queries because it needs to be spun up every time your page is accessed. That said, it is still very quick and it shouldn't be too hard to migrate to MySQL later if need be.