Currently we have a website running on a Centos 6.5 webserver with Direct Admin. APC was configured in the past and is working nicely, but some reading suggested to also implement memcached to cache some static tables (like for instance menu's).
As of MySQL 5.6, the innoDB tables are compatible with a mysql memcache deamon, so I started off following this guide: http://dev.mysql.com/doc/refman/5.6/en/innodb-memcached-installing.html.
The config script is ran and the deamon is installed.
However the Drupal memcache plugin does not see memcache is running. That module is obviously checking for the php memcached deamon, while my deamon is already running in mysql.
Since both extensions are called memcached.so, we strongly have the feeling this is the same thing. Are there actually two different things and does Drupal not support the InnoDB memcached deamon, do I need both (php extension to access the mysql extension?), or was this supposed to be working and did we something wrong?
Update
The status report showed something like "not running", but one of my colleagues has installed the PHP PECL extension now and it seems to be working. But then I still don't understand what the MySQL innodb deamon plugin does. Is it not needed, or does it improve database access even more then the PHP extension would?
The Memcached interface to InnoDB is a feature of MySQL to support the memcached protocol, yet with InnoDB as the back-end storage. It seems like lot of people have been confused by what this means, so I'll try to explain.
Whereas a real memcached daemon stores data in memory, MySQL stores data persistently in an InnoDB table. PHP applications can read and write data using the memcached extension, as if they're using a standard memcached in-memory store. However, they are really reading and writing rows from the InnoDB table.
This is somewhat slower than standard memcached, because it has the overhead of writing to disk is greater than accessing memory. But it's somewhat faster than using SQL to read and write those rows, because it skips the complexity of the SQL parser and query optimizer.
That's really the new feature in MySQL: to bypass SQL, and give access directly to the InnoDB storage engine through a simple, but familiar interface. They chose memcached on the theory that many developers would be familiar with it and have tools and language support for it already.
The InnoDB memcached interface is similar to the earlier experimental plugin called HandlerSocket developed in 2010. http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as-nosql-story-for.html
Here's a Percona blog that shows tests of the relative speeds of InnoDB memcached versus SQL queries: http://www.percona.com/blog/2013/03/29/mysql-5-6-innodb-memcached-plugin-as-a-caching-layer/
Re your question in comments:
You might be misunderstanding. The data is never in Memcached. There is no automatic synchronization between MySQL and Memcached. The only thing is that MySQL is mimicking the API and protocol of Memcached. There's no reason that they did this, except to make the API familiar to developers.
When you use the "Memcached API for InnoDB" you're connecting your application to a port listened to by the mysqld daemon process. Your requests on this connections read and write rows directly in the InnoDB storage engine. There's no Memcached instance in between.
The InnoDB Memcached Plugin is a feature in MySQL >= 5.6 that runs a Memcached daemon in the existing mysqld process supporting the Memcached API listening on a different port (11211 by default). Because it runs in the same process space as InnoDB, you get low-latency access to data stored in InnoDB tables and through existing, widely-available Memcached clients.
This has a few interesting use-cases:
Transparent support for adding InnoDB as a high-performance persistence layer to existing memcached clients/applications
Improved performance over SQL queries for interacting with InnoDB tables directly (especially for inserting new key/value pairs), since the simple Memcached protocol bypasses the overhead of SQL parsing and query-plan optimization
High-performance 'MySQL + caching layer' architecture, where the memcached daemon fetches data from from the underlying InnoDB table and serves cached data directly from local memory
Case #3 is particularly noteworthy: Although the Memcached plugin is configured by default to read/write directly to the underlying InnoDB table (innodb_only cache policy), it can be configured to use its own, separate local memory cache just like a standalone Memcached instance would, either without using InnoDB storage at all (cache-only), or using InnoDB as a backing store (caching). (Note that the currently-accepted answer is incorrect on this point.)
Refer to the architecture diagram from the documentation (note the 'local cache (optional)' component, which is used by the cache-only or caching cache policies):
(source: mysql.com)
In terms of setup/installation, if you're using the Memcached plugin for anything other than a standalone key-value cache (case #1), you will need to create a mapping from Memcached keys/values to your InnoDB tables/columns by writing a row to the special innodb_memcache.containers table. See Creating a New Table and Column Mapping for details.
A bit of a side-track here, but nowadays you're better of using the memcache_storage module. The module page has plenty of good pointers how to use the module and with what other modules it integrates nicely to have a better and faster caching for your site.
Related
I have been working on the server migration of a legacy ecommerce application using PHP 5.6.
The switch involved two Dedicated 32 servers from Linode.
One server is for NginX + PHP and the other is for MySQL only.
The legacy application leverages memcached.
After the switch, I can see a heavy internal traffic caused due to private inbound and outbound connections.
So far this element didn't cause any problem on performance.
However, I was under the impression that the queries would be cached on the local machine, and not on the remote.
Because if the query is cached on the remote host, it sill has to transmit the result set over the private network, instead of retrieving from RAM or the local SSD.
Am I assuming this wrong?
It may be that I am missing the point where the private inbound traffic is more beneficial for overall performance when compared to a local cache.
MySQL has a feature called the Query Cache, but this caches query result sets in the mysqld server process, not on the client. If you run the exact same query again after the result has been cached in the Query Cache, it will copy the result from the Query Cache and avoid the cost of running the query again. But this will not avoid the time to transfer the result across the network from mysqld to your PHP application.
Also keep in mind that the MySQL Query Cache is being deprecated and retired.
Alternatively, your application may store data from query results in memcached, but typically this would be done by the application code (I know there are UDF's to read and write memcached from MySQL triggers, but this is a bad idea).
If your memcached service is not on the same host as your PHP code, it would result in network transfer twice: Once when querying the data from MySQL the first time, then again transferring the data into memcached, then later every time you fetch the cached data out of memcached.
PHP also has some features to do in-memory caching, such as APCu. I don't have any experience with this, and it's not clear from a brief scan of the documentation where it stores cached data.
PHP is designed to be a "shared nothing" language. Every PHP request has its own data, and data doesn't normally last until the next request. This is why a cache is typically not kept in PHP memory. Applications rely on either memcached or the database itself, because those will hold data longer than a single PHP request.
If you have a fast enough network, it shouldn't be a high cost to fetch items out of a cache over a network. The performance architects at a past job of mine developed this wisdom:
"Remote memory is faster than local storage."
They meant that if the data is in RAM on a server, then reading it from RAM even with the additional overhead of transferring it across a network is usually better than reading the data from persistent (disk) storage on the local host.
Does MySQL use fread, read, mmap, or another file system when saving database data to the disk on a Linux OS? Or is MySQL doing a test to see which one to use? This is not in reference to saving config data. I'm interested in the actual database, preferably InnoDB.
Thanks for any help.
Edit: To be more specific, I'm interested in the c/c++ source code in MySQL that does the actual calls that saves data to a InnoDB database. Possible options are fread, read, mmap, among others.
What file system does MySQL use?
The MySQL access method code (InnoDB, MyISAM, AriaDB and the rest) uses the native file system of the host volume on the host operating system. NTFS on Windows, ext4fs on U**X systems, etc. The competent platform ports use a variety of I/O techniques including memory mapping, scatter/gather and ordinary read and write system calls, and integrate with the file systems' journaling features. The exact techniques used depend on the kind of query, the access method, and the state of caches.
Pro tip: Don't worry about this for performance reasons unless your server is running on an old 32-bit 486 machine you found in a storeroom (or unless you have millions of users and billions of rows of data).
On Linux systems all POSIX fileystems will work. fread is a libc construct that will translate to underlying syscalls like read, mmap, write etc.
The read, mmap, write operations are implemented in a Linux VFS (virtual file system) layer before those map to specific operations in the filesystem code. So any POSIX filesystem will work with MySQL.
The only filesystem test I've seen in the MySQL code is a fallocate syscall which isn't implemented on all filesystems (especially when it was first added, its probably significantly available now). There is an implementation workaround when fallocate isn't available.
I am designing a data replication solution across timezones and have run into the issue where I can only run and old version of MySQL (5.6) in one location, whereas the other two have MariaDB 10.2.
Now, I have read the information about Replication Compatibility over at MariaDB. Clearly I can't use MariaDB as a master and MySQL as a slave.
Intermediate solution based on Bash scripts
Yet, I want to use my EU server as a master and that is running MariaDB. So I'm now contemplating a way around the limitation. So far I have come up with an intermediate data storage solution in the overseas server, where data is shuffled periodically using my own Bash data migration scripts over a low bandwidth link.
MariaDB is required in the primary location
I have to use MariaDB in my primary location because I'm using the ColumnStore database there. That is unconditional as part of the application design.
Does this situation ring a bell?
Do you have experience with similar situations and would you mind sharing some inspiration as for how you did it?
My best solution so far is with Bash scripts that are cronned, where MySQL data is dumped (mysqldump) and transferred over a low bandwidth link, then merged with the master (ColumnStore storage engine). I'm looking at a T+1 data lag between my primary location and the secondary location that is running MySQL.
Any high level design thoughts or shared experience is highly appreciated.
Best regards
I have a Rails 4.x application running on server A and MySQL on server B.
Using ab to do a load test of my API calls I notice that the MySQL server is showing CPU activity. So I go back to the code and check, but no SQL statements are triggered, to be sure I also deactivate all before filters, but still the MySQL server shows CPU load.
I went to MySQL and run
show processlist;
but that also shows no active SQL statements
Why would there be load on the DB server?
A Rails application initializes connection pools to the configured database on app load and also loads basic schema data for each ActiveModel defined to populate runtime mappings from the DB to instances of that model.
These connections/queries will happen as soon as you have loaded the application and running traffic.
If this is not what is responsible for the activity on your database server, you will need to use other tools to see what is responsible. For example, NewRelic's system monitoring tools are great for snapshotting CPU/memory usage over time correlated to what processes were running. This will help you rule out MySQL itself using resources vs. other things running on the DB server.
According to this article, storage engines like Innodb may have their own per thread and/or global memory allocations which is probably accounting for the CPU overhead. If this is a stock (non-tuned) MySql install, you're probably just seeing baseline CPU activity. The article mentions a number of places to look that might indicate areas that can be tuned to reduce this footprint.
I'm trying to test the performance of using memcached on a MySQL server to improve performance.
I want to be able to use the normal MySQL command line, but I can't seem to get it to connect to memcached, even when I specify the right port.
I'm running the MySQL command on the same machine as both the memcached process and the MySQL server.
I've looked around online, but I can't seem to find anything about using memcached other than with program APIs. Any ideas?
Memcached has its own protocol. The MySQL client cannot connect directly to a memcached server.
You may be thinking of the MySQL 5.6 feature that allows MySQL server to respond to connections using a memcached-compatible protocol, and read and write directly to InnoDB tables. See http://dev.mysql.com/doc/refman/5.6/en/innodb-memcached.html
But this does not allow MySQL clients to connect to memcached -- it's the opposite, allowing memcached clients to connect to mysqld.
Re your comment:
The InnoDB memcached interface is not really a caching solution per se, it's a solution for using a familiar key/value API for persistent data in InnoDB tables. InnoDB does do transparent caching of data pages in its buffer pool, but this is no different from conventional data reads with SQL. InnoDB also commits all changes to its transaction log synchronously on commit.
Here's a blog from my colleague at Percona. He tested whether the MySQL 5.6 memcached API could be used as a caching layer, and found that actually using memcached is still superior.
http://www.mysqlperformanceblog.com/2013/03/29/mysql-5-6-innodb-memcached-plugin-as-a-caching-layer/
Here's one conclusion from that blog:
As expected, there is a slowdown for write operations when using the InnoDB version. But there is also a slight increase in the average fetch time.