We had an applicationg running using MySql. We found MySql was not suitable for our app after we found that it didnt support some of the GIS capability that PostGIS has (note: mysql only supports minimum-bounding rectangle GIS search).
So we changed our DB to PostgreSQL. We then found out that Postgresql 8.2 running on Windows is so much slower compared to Mysql 5.1. By slower, I mean at roughly 4-5 times slower.
Why is this? Is there something in the configuration that we need to change?
I found some comments from other websites such as this:
UPDATE: We found that the cause of the slowness is due to the BLOB that we are inserting into the DB. We need to be able to insert BLOB at a sustained rate of 10-15 MB/s. We are using libpq's lo_read and lo_write for each BLOB we are inserting/reading. Is that the best way? Has anyone used Pgsql for inserting large BLOB at a high rate before?
EDIT: I heard that PgSql just recently got ported to Windows. Could this be one of the reasons?
There are cases where PostgreSQL on Windows pays an additional overhead compared to other solutions, due to tradeoffs made when we ported it.
For example, PostgreSQL uses a process per connection, MySQL uses a thread. On Unix, this is usually not a noticeable performance difference, but on Windows creating new processes is very expensive (due to the lack of the fork() system call). For this reason, using persistent connections or a connection pooler is much more important on Windows when using PostgreSQL.
Another issue I've seen is that early PostgreSQL on Windows will by default make sure that it's writes are going through the write cache - even if it's battery backed. AFAIK, MySQL does not do this, and it will greatly affect write performance. Now, this is actually required if you have a non-safe hardware, such as a cheap drive. But if you have a battery-backed write cache, you want to change this to regular fsync. Modern versions of PostgreSQL (certainly 8.3) will default to open_datasync instead, which should remove this difference.
You also mention nothing about how you have tuned the configuration of the database. By default, the configuration file shipped with PostgreSQL is very conservative. If you haven't changed anything there, you definitely need to take a look at it. There is some tuning advice available on the PostgreSQL wiki.
To give any more details, you will have to provide a lot more details about exactly what runs slow, and how you have tuned your database. I'd suggest an email to the pgsql-general mailinglist.
While the Windows port of PostgreSQL is relatively recent, my understanding is that it performs about as well as the other versions. But it's definitely a port; almost all developers work primarily or exclusively on Unix/Linux/BSD.
You really shouldn't be running 8.2 on Windows. In my opinion, 8.3 was the first Windows release that was truly production-ready; 8.4 is better yet. 8.2 is rather out of date anyway, and you'll reap several benefits if you can manage to upgrade.
Another thing to consider is tuning. PostgreSQL requires more tuning than MySQL to achieve optimal performance. You may want to consider posting to one of the mailing lists for help with more than basic tweaking.
PostgreSQL is already slower than MySQL up to a certain point (it is actually faster when you have a ridiculously large database). Just FYI, this isn't causing your problem but keep that in mind.
Related
All,
Our product uses Mysql enterprise db. currently at 5.5.42 but planning to upgrade to 5.7.latest. What changes should I expect to see ?
I'm interested in :
1) Performance impacts
2) Broken behavior
3) Changed behavior
4) Improvements -- functional and performance-wise
Is it advised to 1st upgrade our product to use 5.6.x and then next to 5.7.x OR directly to 5.7.x and then test/QA from there ? Any inputs will be valuable.
Performance
Performance gains in 5.7 can be achieved only on high loaded systems. If you have less than 40-60 parallel executions, then expect small performance fall around 10%. This is very system- and load-dependent. Always test yourself!
Here is good MySQL 5.7 performance test done by Percona: https://www.percona.com/blog/2016/05/17/mysql-5-7-read-write-benchmarks/
Changed and broken behaviour, Improvements
There were a lot of changes in 5.7. Check this link for complete list: http://www.thecompletelistoffeatures.com/
I would mention the following changes, that one should think of during upgrade:
SQL_MODE was changed. It is good for new installations, but for old systems most probably you will have to set it back to the old value.
First GA release had password expiry, but after complaints it was disabled in 5.7.11. So nothing to worry about now.
SQL optimizer was changed a lot. Queries should run faster now, but I expect, that some could run slower. The most time should be spent on queries testing on the new version!
Here is good overview of all changes affecting upgrade to 5.7:
http://dev.mysql.com/doc/refman/5.7/en/upgrading-from-previous-series.html
Upgrade
There are two ways to upgrade: using mysql_upgrade and mysqldump.
With mysql_upgrade go with one version at a time. There is no need to test intermediary version. Test only final version.
With mysqldump go directly to 5.7. It is advisable to skip mysql database. Users can not be migrated with mysqldump, use grant statements.
For further information read this: http://dev.mysql.com/doc/refman/5.7/en/upgrading.html
There are tremendous amount of change between 5.5 and 5.7. It's simply not practical to enlist them all here. I'll try to highlight the most important ones. Obviously there are also a lot of new features introduced (GTID, semi-sync replication, JSON support, virtual columns etc.).
1) Performance
You should expect performance gains since 5.6 and 5.7 improved a lot in mutex handling therefore less contention is expected.
Also scalability improvements were done so you can scale your servers up further and performance/query capacity follows for longer than before.
There were a lot of change in the optimizer. So make sure to check your queries.
At the end of the day you should always check your own case with the your own application.
2+3) Changed and broken behaviour
Password expiry. In 5.7 passwords can have expire time. Just something to be aware of.
Password management commands have changed so make sure you check them before you get surprised by having your automation broken.
Since optimizer and the cost model changed a lot some queries might get executed differently.
Two parameters I would mention specifically that caused some headaches for us: Look into the optimizer_switch and decide what you turn on and off. range_index_dive_limit default value was raised from 10 to 200 by default in 5.7.4+. Since I see this parameter is rarely being set in any my.cnf this new default might affect you big time.
4) Improvements
A lot. As I wrote above there are plenty of new features and scalability, performance improvements that were meant to server you better.
5) Upgrade path
Yes, it's best to do one major upgrade at a time, test and when everything's working you can do the second major upgrade.
Compatibility - The issue that would be your greatest concern when upgrading. Because your codes/syntaxes are made with lower version, expect them to behave differently for some of them will be invalid or would not be recognized with the newer version (great impact will be occurred if you jump to a very far higher version).
Normally, you'll get improved performance, security, etc with newer versions but you need to bend your codes/syntaxes to cope with new functions and features.
I have an application that I will soon be doing a significant rewrite of that stores a significant quantity of transient blobs in a database. The application will insert and delete a large number of blobs (up to 5mb in size each) during the course of each day. Currently the application uses a version of PostgreSQL that is very old (7.3.x). With this version of PostgreSQL we had to routinely run the external vacuum process to keep the database size under control, additionally this process required the application to be shutdown to function correctly.
We were looking at either upgrading to the newest PostgreSQL or migrating to another database. Specifically we were interested in moving to MySQL. I was wondering if anyone here was familiar with the blob handling support of the newest versions of these servers, and provide any suggestions on which one might perform best for an application that will constantly be inserting and removing blobs. Other feature differences between the two servers aren't an issue for us.
I did some research and found plenty of feature comparisons between MySQL and PostgreSQL, but nothing that really addressed this issue. I'm hoping someone here might have some experience with this aspect of one or both database systems.
Thanks
Postgres 7.x indeed was a major PITA when it comes to vacuuming. 9.0 is a lot better in this area. The autovacuum daemon can be configured on a per-table level since I think 8.3 and for the described scenario you would probably make it very aggressive for that table (or tables if more than one is involved).
I don't think it matters whether you delete rows with BLOBs (i.e bytea) column or not. Especially because the blobs are stored out-of-line anyway (you might need to configure the auto-vaccuum daemon for the so called TOAST table as well, but I'm not sure)
The question is rather how many rows (in percent of the total rows) you delete/update in the table, rather than how big each blob is.
As much as I like PostgreSQL I do have to admit that the whole vacuum topic (even though getting easier and easier which each release) is still one of its weakest points (and the source of a lot of trouble).
I can't say anything about MySQL as I have never used in a production environment. In contrast to you the other features (beside blobs) are important enough for me to stay away from MySQL - and if it's only for the license.
I'm building a web application right now for my company and for whatever reason, can't get mySQL working on my Mac w/ my Ruby install (OSX 10.5). SQLite works fine though, so would it be a problem to use SQLite for now so I can get to work on this and then just change up my database.yml file to point to a mySQL database when I deploy (assuming I rerun migrations and such)?
Also, what are the benefits/drawbacks of using mySQL over SQLite in a RoR application? I've always used mySQL by default in the past, but never learned SQL directly (always through ActiveRecord) and never thought too much about the difference.
Benefits of MySQL/PostrgreSQL/etc
Pros
Stronger data typing, which means cleaner data
Ability to store more data
Scale better to larger data sets
Spatial support (think GPS)
Full Text Search (FTS)
Cons
Stronger data typing means data will be validated, bad data will cause errors
Not a good candidate (if even possible) for devices with limited resources (iPhone, Blackberry, iPad, etc)
I would pick PostgreSQL v8.4+ over MySQL given the choice. MySQL's features lag behind the rest of the major SQL database alternatives.
THe biggest performance issue you may run into is table locks. SQLite unfortunately does not have row level locking. So if your app is going to run multiple processes / threads (as with multiple web users) its likely some threads will not be able to perform an SQL op. For this reason i would go with MySQL - or perhaps Postgresql.
Should be no problems, as MySQL should have a superset of SQLite capabilities, and as #Sean pointed out, performance should only increase. Just try to make sure you're not using anything too SQLite specific (I'm mainly a SQL Server and Oracle guy, so don't know what that would be, if anything). Remember, the "S" in SQL stands for Structured, not Standard ;)
Paul.
SQLite is perfect for a desktop or smartphone application ("embedded" usage). However, if you plan to build a web-application, you are highly encouraged to make use of a non-embedded DMS like MySQL. The benefits are countless, such as 3rd party design and analysis apps, performance etc ...
Currently have approximately 2000 simultaneouse connections. We average approximately 425 reads and writes per second. We have a read to write ration of 3:1. All of our tables are myisam. Can we expect better or worse performance when we go from mysql 4.1.22 to 5.0?
There's no way for anyone here to tell you without the schema, queries and test data.
Why not setup a dev environment on 5.0 and testing it out?
The main concern should be that the 5.0 Information Schemas, are a HUGE vulnerability and can be used to very easily gain access to the SQL server from remote locations simply by printing off the schema using injection will let an unwanted viewer, view all of the tables and capitalize off the knowledge to get passwords using the same schema for its columns.
The MySQL source tree includes a set of benchmark tests written as Perl scripts. See The MySQL Benchmark Suite for some information. You can download the source distribution for MySQL 5.0.91 at the archives.
Source distribution of MySQL 4.1 doesn't seem to be easily available anymore. You might have to check it old sources from LaunchPad unless you can find a copy of an old source distribution elsewhere on the internet.
However, the comparison that these benchmarks show is only of general interest. It may be irrelevant to how your application performs. For instance, your usage of the database may not take advantage of some performance improvements in MySQL 5.0, but it may run into some regressions in MySQL 5.0 that were necessary.
The only way to get an answer that is relevant to your application is to try the new software with a test instance of your application, using a sample of data that is a realistic model of the type and volume of data your application typically deals with. As #BenS says, no one on a site like StackOverflow can give an answer specific to your application.
You say in a comment that you're very concerned about performance, but if you don't have an instance of your application and database that you can run tests on, you aren't doing the work necessary to satisfy this concern.
I would strongly suggest moving straight to 5.1.45 with Innodb Support. Percona provides an excellent version with XtraDB that provides a number of performance related improvements. Moving off of your MyISAM tables and onto Innodb will provide a huge performance increase in almost all cases. If you are going to burn the QA/Testing time to move, do a full move now, not a half-way step.
I've used both and I've found MySql to have several frustrating bugs, limited support for: IDE integration, profiling, integration services, reporting, and even lack of a decent manager. Total cost of ownership of MSSQL Server is touted to be less than MySQL too (.net environment), but maintaining an open mind could someone point out any killer features of MySql?
I've used MySQL in the past and I'm using MSSQL lately but I can't remember anything that MySQL has and MSSQL can't do.
I think the most killer feature of MySQL it's the simplicity. For some projects you just don't need all the power you can have with a huge system like MSSQL. I have an UNIX heritage and find the simple configuration file like my.ini a killer feature of MySQL.
Also the security system of MySQL is much less robust but it makes the job right for most of applications. I believe MySQL it's killer itself from this point of view, and should stay that way, letting young users being introduced to RDBMS with a simple view first. If your project gets big enough that you are considering switch to a more robust system, then MSSQL can pop as a possibility.
That's what happened to me.
The only thing I can think of, off hand, is locking. SQLServer has traditionally had poor locking strategy that has tripped many people up.
You should use what you prefer, ultimately. Its not as if MySQL is not good enough to compete with MS SQL, eg. Slashdot uses MySQL, so its hardly got problems with high-scalability performance.
Its killer feature though, is that it is free - you can deploy as many of them without worrying one fig about licensing issues. That's more important for the spread of software than anyone could imagine.
(TCO is a difficult thing to calculate - and is advice only ever given from paid consultants and other vested interests. Ignore that. MSSQL is expensive and MySQL is free.)
About 6 years ago I developed a custom e-commernce website using ASP and MySQL for the database. At the time MySQL was clearly a better choice than MSDE which had built in throttling which concerned me enough to use MySQL. Also the difference in coding between using MySQL and MSDE/SQL was not that different or much of a concern.
Now all these years later I'm trying to get the code converted to .NET and even after purchasing commercial MySQL drivers from CRLab. I found that, as you hinted, the IDE integration is just not up to par.
I will say that MySQL is doing a great job even with our database tables approaching 4GB. So when I switch to MSSQL I have to go ahead and get SQL Workstation or higher ($$$), and not use SQL Express which has a 4gb limit.
All of my experience has changed the way I develop new websites. Now, unless it is expected to have a lot of traffic. I use VistaDB and then upgrade to SQL Server if needed. VistaDB is syntax and datasource compatible with SQL Server. And the best part is it is only a single file for the database and a dll for your bin folder.
That's my two cents based on my personal experience with using MySQL in ASP and now .NET.
I work with MSSQL, MySql and PostGres regularly (using .net, java and PHP). One of my favorite things about about MySQL (esp. compared to MSSQL) is the ease with which you can run and restore full database backups.
MSSQL's model of using .bak files is really ugly and time-consuming (topic for another post.) But if you want to do somethign like automated testing, or automated build processes (that include building a db from scratch), MySQL can be a bit easier to deal with.
A few other points:
The management tools have gotten a lot better since the early days.
If you are interested in transactions, constraints, etc.. be sure you are defining your tables to use the InnoDB storage engine (instead of MyISAM which is designed for speed.)
I do miss MSSQL's schema generating tool, but I think there are equivalent tools out there.
We've used a Linux database server and a window's web server (for .net apps) with great success.
If you are using something like NHibernate or some other non-MS data abstraction layer, the case to look beyond MSSQL is stronger too...
Three points to consider; unfortunately the first two are contradictory:
1) .NET and MySQL were not designed to interact with one another, and there is no official support from either side. You're invariably going to encounter issues trying to use them together.
2) If portability off of Windows may ever be an issue (much .NET code runs quite nicely on other platforms via Mono), you'll want to avoid locking yourself too deeply to MSSQL. That doesn't mean not using it, but being careful that you don't rely on its particular quirks too much.
3) TCO is just a buzzword. It's complete nonsense when it's calculated by anyone other than you. Nobody can make such a calculation and honestly claim that it has any relevance outside their particular environment. There are too many factors, most of which have absolutely nothing to do with things like tool availability.
I've been using the community version of MySQL for alsmost 99% of my project. I like MySQL is that I can deploy via Xcopy and is powerful compare to other "xcopy-able" database server. I also wrote a wrapper to start and stop MySQL & Apache (like LAMP), but with my own implemetation and addon capability
MySQL probably has a lower TCO, since administration and configuration is more simple and straightforward than the Spaghetti GUI that MS SQL makes you do most of the configuration through, having to dig through hundreds of obscure properties dialogs to accomplish even basic administration tasks.
There is one area where MS SQL clearly excels over MySQL in my experience:
Integration with other technologies. MS SQL allows you to replicate back and forth with Oracle and MySQL databases, and provides SSIS for executing scheduled data transformations from other database servers.
There may be others, but I don't have experience with them.