I have been noticing quite a bit of latancy with my Magento store, we are running a Dedicated Virtual server with 4GB of RAM. When I asked my host about these problems I was told the below statement. Could you offer assistance on what specifically needs to be tweaked in MySQL for optimal performance?
However, it appears that using Magento is causing a problem. Magento
requires significant MySQL tuning to keep running properly. I suggest
you work with a sysadmin to tune the server to suit Magentos very
large resource requirements.
Magento has a few white papers that cover the basics.
There's an old thread on the Magento forums that, while containing out of date information, will point you towards the MySQL configuration fields that you'd need to tweak.
Seriously consider using a hosting partner that specializes in Magento hosting. If you read between the lines your host is telling you they don't want your business.
Related
I identified an issue with an infrastructure I created on the Google Cloud Platform and would like to ask the community for help.
A charge was made that I did not expect, as theoretically it would be almost impossible for me to pass the limits of the free tier. But I noticed that my database is huge, with 1gb and growing, and there are some very heavy buckets.
My database is managed by a Django APP, and accessing the tool's admin panel there are only 2 records in production. There were several backups and things like that, but it wasn't me.
Can anyone give me a guide on how to solve this?
I would assume that you manage the database yourself, i.e. it is not Cloud SQL. Databases pre-allocate files in larger chunks in order to be efficient on writes. You can check this yourself - write additional 100k records, most likely size will not change.
Go to Cloud SQL and it will say how many sql instances you have. If you see the "create instance" window that means that you don't have any Google managed SQL instances and the one we're talking about is a standalone one.
You can also check Deployment Manger to see if you deployed one from the marketplace or it's a standalone VM with MySQL installed manually.
I made a simple experiment and deployed (from Marketplace) a MySQL 5.6 on top of Ubuntu 14.04 LTS. Initial DB size was over 110MB - but there are some records in it initially (schema, permissions etc) so it's not "empty" at all.
It is still weird that your DB is 1GB already. Try to deploy new one (if your use case permits that). Maybe this is some old DB that was used for some purpose and all the content deleted afterwards but some "trash" still remains and takes a lot of space.
Or - it may well be exactly as NeatNerd said - that DB will allocate some space due to performance issues and optimisation.
If I have more details I can give you better explanation.
I am currently creating a site(php,css,html,ajax,MySql) which will have heavy user usage of space(regarding data). These data are quite essential and the can NOT be lost, it is really essential.
I am looking for tips on servers, languages and everything else(even theory) about distributed database systems. Any help would be really appreciated. Also it would be great if the system used mysql.
Thank you
P.S. dont link Google.com. I have done that and reached nothing but a wall :(
My guess is that you're Googling for the wrong terms. If you searched for MySql Replication you might run into this article
Database replication is what enables a "distributed database system". You should also look into clustering to see if that type of distributio/replicationn might meet your needs.
Also, you didn't specify if you were running LAMP or WAMP but here's a how-to on setting up a MySQL and Apache cluster.
I've been doing some work on Wordpress and just stumbled across database scaling.
I was wondering if it was at all possible to implement this with Magento as we have an increasingly large database and thousands of products still to add (with various options for configurable products!!)
You can setup master slave replication for mysql so that you have a customer facing server on the master database and a admin only server on the slave database. You can edit /app/etc/local.xml on the slave so that it reads from the slave but writes to the master. In that way you can upload loads of products, do loads of reports and the live system only has to slow down for the occasional write to the db. You can also have your slave system in your office on the end of an ADSL line.
For this to work standard mysql master slave settings work fine, it also does rather well in catching up with itself if the slave needs rebooting. The only gotcha of note is that mysql log files get big and your mysql my.cnf needs to have the setting on to delete the log files after n days.
Magento has recently certified ClustrixDB to be the scale-out database solution: http://www.marketwired.com/press-release/clustrix-selected-as-magento-technology-partner-2010010.htm
Disclaimer: I work at Clustrix and I know this is just a press release, but I have been also involved in the testing and it really works!
Disclaimer - I work at Clustrix.
Magento solution architect and certified developer, Kevin Bortnick, recently gave a webinar entiteld "Scaling Techniques to Increase Magento Capacity" at Meet Magento NY. Kevin currently works for clustrix but prior to joining this year, he worked on large Magento sites such as The Home Depot and Dr. Jays. In the webinar he shares his practical experiences. He covers 3 front end strategies and 8 RDBMS scaling strategies including:
Percona
Faster hardware
Master with read slaves
Master/Master, Store/Admin
True Multi-master
Partitioning (Magento 2 feature CQRS)
NoSQL
ClustrixDB
I hope this webinar helps you find the right solution for your use case.
For a heavy user content website with user profiles, live feeds, photo /video/content sharing, etc what DB to use, and what DB engine? Ofcourse Oracle/Microsoft sql are out because they are not free (or cheap). I am using MySQL with MyISAM but that will run into performance issues on a social site. Even using InnoDB may not help performance. So firstly which DB is better to use and which engine - starting from Free to paid?
Main concern is with live feeds performance, plus there is lots of data tracking and mining with analytic.
Since this is a startup - hardware costs are limited so no hardware farm here to support performance, hence seeking a decent alternative for the first 2 years atleast.
If you are a start-up, you may be able to afford Microsoft's stuff after all. Check out their BizSPark program which gives you software for free for a few years.
I am not sure that I agree that MySQL is not your best option though. Doesn't FaceBook use MySQL? Are you expecting to be bigger?
If you think that InnoDB will be too slow for you, there are other storage engine options on MySQL. For example, have you investigated TokuDB?
Have you considered a hosting service like Linode.com instead of your own hardware? This might be a better fit for your cashflow profile. If you really feel the DB load is onerous, you could have dedicated or multiple servers. Start with a dedicated DB box and scale up from there. You could also go with a cloud service like Amazon EC2.
EDIT: I just realized that Tokutek have a social networking case study:
http://tokutek.com/customers/a-social-networking-case-study/
It sounds to me like you are unlikely to outgrow MySQL any time soon. If you grow busy enough to need anything like TokuDB you can thank me with a few shares. :-)
If your userbase is large enough to require a high-performance DB, your bandwidth and storage charges will not be cheap; why skimp on the database?
MS SQL has free and/or cheap editions which are performance-inhibited. I always start with them, and if the site takes off, I upgrade. Probably Oracle offers something similar.
Edit: I should have said at the start that you're unlikely to outgrow MySQL anytime soon, and that this whole question might be a case of premature optimization.
I would recommend getting a cloud account for the files and potentially a dedicated server for the DB (MySQL and MyISAM).
You can limit your cost on the cloud storing the files, but enhance performance by having a dedicated server for the DB by increasing processing power and memory.
I'm working on a SaaS project and mysql is our main database. Our applications is written on c# .net and runs under an windows 2003 server.
Considering maintainance, cost, options and performance, which server plattaform can I decide for MySQL hosting, windows or Unix/Linux/Ubuntu/Debian?
The scenario is as following:
The server I run today has a modarate transaction volume. Databases increase 5MB daily and we expect to increase 50MB in couple of months and it is mission critical.
I don't know how big the database is going to be. We rent a VPS to host application and database server.
Most of our queries are simple but our ORM Tool makes constantly use of subqueries. Also we run reports simple and heavy ones. Some them runs after user click, but most runs in order to the queue.
Buy an extra co-lo space will be nice as we got more clients. That's SaaS project after all.
When developing, you can use your Windows box to also run a MySQL server. If and when you
want to have your DBMS in a separate server it can be in either a Windows or Linux server.
MySql and supporting tools for backup etc probably have more choices in Linux.
There are also 3rd party suppliers who will host your MySQL database on their servers. The benefit is they will handle backups, maintenance etc.
Also: look into phpMyAdmin for use as a great admin tool.
Larry
I think you need more information to make an informed decision. It's hard to just pull out a "best" answer based on no specific information.
What is your expected transaction volume?
How big will the database get?
How complex are your queries, ie are they long running or relatively quick?
Are you hosting the application on your own server at your own location? If you have to buy extra co-lo space maybe an extra server isn't the best option.
How "mission critical" is this database? Ie maybe you need replicated servers to ensure stability.
There is a server sizing tool online at http://www.sizinglounge.com/, so you should check that out. It sounds like your server could be smaller than their smallest tier, but it should be a good place to start.
If this is a mission critical application you need to do some kind of replication to an extra server in case the primary one fails, so you are definitely looking at two systems. This has to be in addition to a good backup plan.
Given that you are uncertain about how big it could get you might just continue renting a server. For your backup one idea would be to look at running MySQL on an Amazon EC2 instance. BTW it is important to have a remote replicated server. If you have two systems next to each other and an environmental problem comes up, they could both be out of commission at the same time. But with a remote copy your options are open to potentially working around it.
If you run a lot of read-only queries locally and have your site hosted somewhere, it might make sense to set up a local replicated database copy to query against. That could potentially improve both your website and local performance quite a bit. Plus it would give you some good piece of mind having a local copy under your control.
HTH,
Brandon