I'm trying to test the performance of using memcached on a MySQL server to improve performance.
I want to be able to use the normal MySQL command line, but I can't seem to get it to connect to memcached, even when I specify the right port.
I'm running the MySQL command on the same machine as both the memcached process and the MySQL server.
I've looked around online, but I can't seem to find anything about using memcached other than with program APIs. Any ideas?
Memcached has its own protocol. The MySQL client cannot connect directly to a memcached server.
You may be thinking of the MySQL 5.6 feature that allows MySQL server to respond to connections using a memcached-compatible protocol, and read and write directly to InnoDB tables. See http://dev.mysql.com/doc/refman/5.6/en/innodb-memcached.html
But this does not allow MySQL clients to connect to memcached -- it's the opposite, allowing memcached clients to connect to mysqld.
Re your comment:
The InnoDB memcached interface is not really a caching solution per se, it's a solution for using a familiar key/value API for persistent data in InnoDB tables. InnoDB does do transparent caching of data pages in its buffer pool, but this is no different from conventional data reads with SQL. InnoDB also commits all changes to its transaction log synchronously on commit.
Here's a blog from my colleague at Percona. He tested whether the MySQL 5.6 memcached API could be used as a caching layer, and found that actually using memcached is still superior.
http://www.mysqlperformanceblog.com/2013/03/29/mysql-5-6-innodb-memcached-plugin-as-a-caching-layer/
Here's one conclusion from that blog:
As expected, there is a slowdown for write operations when using the InnoDB version. But there is also a slight increase in the average fetch time.
Related
I have been working on the server migration of a legacy ecommerce application using PHP 5.6.
The switch involved two Dedicated 32 servers from Linode.
One server is for NginX + PHP and the other is for MySQL only.
The legacy application leverages memcached.
After the switch, I can see a heavy internal traffic caused due to private inbound and outbound connections.
So far this element didn't cause any problem on performance.
However, I was under the impression that the queries would be cached on the local machine, and not on the remote.
Because if the query is cached on the remote host, it sill has to transmit the result set over the private network, instead of retrieving from RAM or the local SSD.
Am I assuming this wrong?
It may be that I am missing the point where the private inbound traffic is more beneficial for overall performance when compared to a local cache.
MySQL has a feature called the Query Cache, but this caches query result sets in the mysqld server process, not on the client. If you run the exact same query again after the result has been cached in the Query Cache, it will copy the result from the Query Cache and avoid the cost of running the query again. But this will not avoid the time to transfer the result across the network from mysqld to your PHP application.
Also keep in mind that the MySQL Query Cache is being deprecated and retired.
Alternatively, your application may store data from query results in memcached, but typically this would be done by the application code (I know there are UDF's to read and write memcached from MySQL triggers, but this is a bad idea).
If your memcached service is not on the same host as your PHP code, it would result in network transfer twice: Once when querying the data from MySQL the first time, then again transferring the data into memcached, then later every time you fetch the cached data out of memcached.
PHP also has some features to do in-memory caching, such as APCu. I don't have any experience with this, and it's not clear from a brief scan of the documentation where it stores cached data.
PHP is designed to be a "shared nothing" language. Every PHP request has its own data, and data doesn't normally last until the next request. This is why a cache is typically not kept in PHP memory. Applications rely on either memcached or the database itself, because those will hold data longer than a single PHP request.
If you have a fast enough network, it shouldn't be a high cost to fetch items out of a cache over a network. The performance architects at a past job of mine developed this wisdom:
"Remote memory is faster than local storage."
They meant that if the data is in RAM on a server, then reading it from RAM even with the additional overhead of transferring it across a network is usually better than reading the data from persistent (disk) storage on the local host.
I am designing a data replication solution across timezones and have run into the issue where I can only run and old version of MySQL (5.6) in one location, whereas the other two have MariaDB 10.2.
Now, I have read the information about Replication Compatibility over at MariaDB. Clearly I can't use MariaDB as a master and MySQL as a slave.
Intermediate solution based on Bash scripts
Yet, I want to use my EU server as a master and that is running MariaDB. So I'm now contemplating a way around the limitation. So far I have come up with an intermediate data storage solution in the overseas server, where data is shuffled periodically using my own Bash data migration scripts over a low bandwidth link.
MariaDB is required in the primary location
I have to use MariaDB in my primary location because I'm using the ColumnStore database there. That is unconditional as part of the application design.
Does this situation ring a bell?
Do you have experience with similar situations and would you mind sharing some inspiration as for how you did it?
My best solution so far is with Bash scripts that are cronned, where MySQL data is dumped (mysqldump) and transferred over a low bandwidth link, then merged with the master (ColumnStore storage engine). I'm looking at a T+1 data lag between my primary location and the secondary location that is running MySQL.
Any high level design thoughts or shared experience is highly appreciated.
Best regards
Currently we have a website running on a Centos 6.5 webserver with Direct Admin. APC was configured in the past and is working nicely, but some reading suggested to also implement memcached to cache some static tables (like for instance menu's).
As of MySQL 5.6, the innoDB tables are compatible with a mysql memcache deamon, so I started off following this guide: http://dev.mysql.com/doc/refman/5.6/en/innodb-memcached-installing.html.
The config script is ran and the deamon is installed.
However the Drupal memcache plugin does not see memcache is running. That module is obviously checking for the php memcached deamon, while my deamon is already running in mysql.
Since both extensions are called memcached.so, we strongly have the feeling this is the same thing. Are there actually two different things and does Drupal not support the InnoDB memcached deamon, do I need both (php extension to access the mysql extension?), or was this supposed to be working and did we something wrong?
Update
The status report showed something like "not running", but one of my colleagues has installed the PHP PECL extension now and it seems to be working. But then I still don't understand what the MySQL innodb deamon plugin does. Is it not needed, or does it improve database access even more then the PHP extension would?
The Memcached interface to InnoDB is a feature of MySQL to support the memcached protocol, yet with InnoDB as the back-end storage. It seems like lot of people have been confused by what this means, so I'll try to explain.
Whereas a real memcached daemon stores data in memory, MySQL stores data persistently in an InnoDB table. PHP applications can read and write data using the memcached extension, as if they're using a standard memcached in-memory store. However, they are really reading and writing rows from the InnoDB table.
This is somewhat slower than standard memcached, because it has the overhead of writing to disk is greater than accessing memory. But it's somewhat faster than using SQL to read and write those rows, because it skips the complexity of the SQL parser and query optimizer.
That's really the new feature in MySQL: to bypass SQL, and give access directly to the InnoDB storage engine through a simple, but familiar interface. They chose memcached on the theory that many developers would be familiar with it and have tools and language support for it already.
The InnoDB memcached interface is similar to the earlier experimental plugin called HandlerSocket developed in 2010. http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as-nosql-story-for.html
Here's a Percona blog that shows tests of the relative speeds of InnoDB memcached versus SQL queries: http://www.percona.com/blog/2013/03/29/mysql-5-6-innodb-memcached-plugin-as-a-caching-layer/
Re your question in comments:
You might be misunderstanding. The data is never in Memcached. There is no automatic synchronization between MySQL and Memcached. The only thing is that MySQL is mimicking the API and protocol of Memcached. There's no reason that they did this, except to make the API familiar to developers.
When you use the "Memcached API for InnoDB" you're connecting your application to a port listened to by the mysqld daemon process. Your requests on this connections read and write rows directly in the InnoDB storage engine. There's no Memcached instance in between.
The InnoDB Memcached Plugin is a feature in MySQL >= 5.6 that runs a Memcached daemon in the existing mysqld process supporting the Memcached API listening on a different port (11211 by default). Because it runs in the same process space as InnoDB, you get low-latency access to data stored in InnoDB tables and through existing, widely-available Memcached clients.
This has a few interesting use-cases:
Transparent support for adding InnoDB as a high-performance persistence layer to existing memcached clients/applications
Improved performance over SQL queries for interacting with InnoDB tables directly (especially for inserting new key/value pairs), since the simple Memcached protocol bypasses the overhead of SQL parsing and query-plan optimization
High-performance 'MySQL + caching layer' architecture, where the memcached daemon fetches data from from the underlying InnoDB table and serves cached data directly from local memory
Case #3 is particularly noteworthy: Although the Memcached plugin is configured by default to read/write directly to the underlying InnoDB table (innodb_only cache policy), it can be configured to use its own, separate local memory cache just like a standalone Memcached instance would, either without using InnoDB storage at all (cache-only), or using InnoDB as a backing store (caching). (Note that the currently-accepted answer is incorrect on this point.)
Refer to the architecture diagram from the documentation (note the 'local cache (optional)' component, which is used by the cache-only or caching cache policies):
(source: mysql.com)
In terms of setup/installation, if you're using the Memcached plugin for anything other than a standalone key-value cache (case #1), you will need to create a mapping from Memcached keys/values to your InnoDB tables/columns by writing a row to the special innodb_memcache.containers table. See Creating a New Table and Column Mapping for details.
A bit of a side-track here, but nowadays you're better of using the memcache_storage module. The module page has plenty of good pointers how to use the module and with what other modules it integrates nicely to have a better and faster caching for your site.
I'm trying to implement a proxy layer in front of MySQL server, that will catch redundant SQL queries and send them only once to the server. In other words, I have many clients (in PHP, Perl, on different web nodes) that talk to the MySQL and very often repeat the same SELECT queries. When traffic goes up MySQL, very often, goes down.
The question is - are you aware of any open source (or commercial) tool that can help? I tried MySQL Proxy, but looks like it can't help.
Two suggestions:
MySQL Proxy
This is a front end proxy from MySQL which does what you want as far as I know
vtocc
From the vitess project, used in the YouTube mysql environment, also does a similar thing. Query consolidation: The ability to reuse the results of an in-flight query to any subsequent requests that were received while the query was still executing.
You may want to look into HAProxy and how it works.
Here two additional suggestions
SUGGESTION #1 Setup a Cluster
If your data is all InnoDB, you should try Percona XtraDB Cluster and use HAProxy in conjunction with it. You can load balance across all server in the Cluster including the Write Master.
SUGGESTION #2 Setup a Cluster via MySQL Replication to 1 or more DB Servers
Use HAProxy to load balance your reads across the Read Slaves
If you are on a budget and your data is relatively small, setup multiple MySQL Instances on one server
I have a MySQL database running on our server at this location.
However, the internet connection at this location is slow (Especially when several users are connected remotely).
We also have a remote web server on a very fast internet connection.
Can I run another MySQL server on the remote server and still be able to run queries and updates on it?
I want to have two servers because
- Users at this location can connect via lan (fast)
- Users working remotely can connect to synced remote server (fast)
Is this possible? From what I understand replication does not work this way. What is replication used for then? Backups?
Thanks for your help!
[Edit]
After doing some more reading, I am a little worried about setting up multi-master replication due to the fact that I had not considered multi-master when designing the database and conflicts could be an issue.
The good news though is that most time consuming operations are queries not updates.
And, I found out that there is a driver that handles master-slave connections.
http://dev.mysql.com/doc/refman/5.1/en/connector-j-reference-replication-connection.html
That way writes will be sent to the master and reads can come from the faster connection.
Has anyone tried doing this before? My one concern is that if I update to the master, then run a query expecting to see the update on the slave, will it be there right away? Or will the slow connection make this solution just as slow as using the master for both read and write?
What you're asking, I believe, is called Multi-Master Replication, by which both servers serve as replication masters to each other. Changes on either server become replicated back to the other as soon as possible. MySQL can be configured to do it, however I'm not sure how the differences in speed would affect your performance and data integrity.
http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-replication-multi-master.html