I wrote some functions that work with SQL. I test the functions using testthat and an in memory SQLite database. However, some functions cannot be tested using SQLite because SQLite does not support the ALTER TABLE command.
Is there some way to simulate a mySQL database in memory the same way that one can simulate a SQLite?
> DBI::dbConnect(RSQLite::SQLite(), ":memory:")
<SQLiteConnection>
Path: :memory:
Extensions: TRUE
> DBI::dbConnect(RMySQL::MySQL(), ":memory:")
Error in .local(drv, ...) :
Failed to connect to database: Error: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)
If not, how does one write automatic tests for mySQL functions?
You can't make a whole MySQL instance run in memory like the SQLite :memory: option. MySQL is designed to be a persistent database server, not an ephemeral embedded database like SQLite.
However, you can use MySQL's MEMORY Storage Engine for individual tables:
CREATE TABLE MyTable ( ...whatever... ) ENGINE=MEMORY;
The equivalent in RMySQL seems to be the dbWriteTable() method, but as far as I can tell from documentation, you can't specify the ENGINE when creating a table with this method. You'll have to create the table manually in the MySQL client if you want to use the MEMORY engine.
But you should be aware that every storage engine has some subtle different behavior. If any of your tests depend on features of InnoDB, you won't be able to simulate them with the MEMORY storage engine (e.g. row-level locking, foreign keys, fulltext search).
Read the manual on the MEMORY storage engine: https://dev.mysql.com/doc/refman/5.7/en/memory-storage-engine.html
P.S.: Ignore the suggestions on that manual page to use NDB Cluster. It may be faster, but it requires multiple servers and special database design to achieve that performance. It's much harder to set up.
Related
I am using '5.7.mysql_aurora.2.10.2' version of AWS Aurora MySQL instance and I have trying to create Log table with innoDB engine so I can Log things (INSERT query) combined with SIGNAL command. Something like this:
INSERT INTO Log(type, info, date) VALUES("ERROR", "Error happened...!", CURRENT_TIMESTAMP());
SIGNAL CUSTOM_EXCEPTION SET MESSAGE_TEXT = "Error happened...";
But as I found out, SIGNAL basically rolls back everything including my INSERT statement. I have been trying to figure out the workaround and I stumbled upon the DB tables with engine MyISAM which should solve my problem. So I decided to create a table for testing:
CREATE TABLE t (i INT) ENGINE = MYISAM;
And for some reason, the engine keeps being InnoDB. I have tested on my local instance and it works fine but as soon as I try it on my RDS database, it keeps changing back. I have tried to use ALTER TABLE but it doesn't work.
Is there a possibility that RDS has some configuration that doesn't let me use any other engine other than InnoDB?
Amazon Aurora only supports InnoDB.
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Reference.html says:
Aurora MySQL clusters use the InnoDB storage engine for all of your data.
This is by design. They implemented their distributed storage by modifying the InnoDB storage engine. Aurora simply doesn't work with any other storage engine, so they disabled the capability to specify the storage engine for a table.
Aurora is not MySQL.
In our project we are planning to introduce MySQL NDB cluster to have 99% uptime for our multiple applications dependent on MySQL.
So MySQL is being deployed in two machines. In both the machines Data Node, Management Server and SQL node is deployed and configured to form a cluster as shown in the below snippet.
Based on my understanding replication of data will be done for the data stored in Data Nodes. But can we restrict the replication only to a set of tables or database?
Reason for this query is, there are two applications that are dependent on MySQL, where only one application needs this replication and the other doesn't need this feature because it should connect to a standalone instance of MySQL to store it's local data which shouldn't be replicated as it would cause problem to the application running in another machine.
Please share your thoughts on this.
Though we have deployed MySQL NDB Cluster, the mysqld still supports INNODB storage type. So in order to achieve the above requested need, we created tables explicitly with Storage Engine by mentioning in CREATE TABLE statement like below. This overrode the storage configuration mentioned in my.cnf.
CREATE TABLE IF NOT EXISTS `CDS` (
`CD_ID` bigint(20) NOT NULL,
PRIMARY KEY (`CD_ID`)
) ENGINE=innodb
I'm developing a application that needs two diferent databases, this is because one of this databases is per client and the other one is a generic database.
I'm thinking in make a MySQL NDB Cluster and i need to know if it's possible to uses some Federated tables in the Cluster or all must use ndbcluster engine.
If this is not posible, how can i make joins with tables that are in other host using ndbcluster?
Please forget about, why i need this schema (one databse per client and one generic database) because i spent a lot of time thinking which should be the best schema for my application i choosed this one.
Thanks in for your help!!!
MySQL Cluster uses a full version of mysqld (slightly modified), which includes all storage engines included on a standalone version. So the question for your answer is YES, you can have some tables in FEDERATED storage engine, or any other storage engine.
However, only tables with storage engine=ndbcluster will be replicated to all the api nodes connected to the cluster.
The federated approach with a cluster of two api nodes could work, but keep in mind that only those tables with the same storage engine can have referential integrity (FK) between them.
Which version of MySQL Cluster are you using? It is recommended using always the latest GA release (now 7.4.12)
Regards
Currently we have a website running on a Centos 6.5 webserver with Direct Admin. APC was configured in the past and is working nicely, but some reading suggested to also implement memcached to cache some static tables (like for instance menu's).
As of MySQL 5.6, the innoDB tables are compatible with a mysql memcache deamon, so I started off following this guide: http://dev.mysql.com/doc/refman/5.6/en/innodb-memcached-installing.html.
The config script is ran and the deamon is installed.
However the Drupal memcache plugin does not see memcache is running. That module is obviously checking for the php memcached deamon, while my deamon is already running in mysql.
Since both extensions are called memcached.so, we strongly have the feeling this is the same thing. Are there actually two different things and does Drupal not support the InnoDB memcached deamon, do I need both (php extension to access the mysql extension?), or was this supposed to be working and did we something wrong?
Update
The status report showed something like "not running", but one of my colleagues has installed the PHP PECL extension now and it seems to be working. But then I still don't understand what the MySQL innodb deamon plugin does. Is it not needed, or does it improve database access even more then the PHP extension would?
The Memcached interface to InnoDB is a feature of MySQL to support the memcached protocol, yet with InnoDB as the back-end storage. It seems like lot of people have been confused by what this means, so I'll try to explain.
Whereas a real memcached daemon stores data in memory, MySQL stores data persistently in an InnoDB table. PHP applications can read and write data using the memcached extension, as if they're using a standard memcached in-memory store. However, they are really reading and writing rows from the InnoDB table.
This is somewhat slower than standard memcached, because it has the overhead of writing to disk is greater than accessing memory. But it's somewhat faster than using SQL to read and write those rows, because it skips the complexity of the SQL parser and query optimizer.
That's really the new feature in MySQL: to bypass SQL, and give access directly to the InnoDB storage engine through a simple, but familiar interface. They chose memcached on the theory that many developers would be familiar with it and have tools and language support for it already.
The InnoDB memcached interface is similar to the earlier experimental plugin called HandlerSocket developed in 2010. http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as-nosql-story-for.html
Here's a Percona blog that shows tests of the relative speeds of InnoDB memcached versus SQL queries: http://www.percona.com/blog/2013/03/29/mysql-5-6-innodb-memcached-plugin-as-a-caching-layer/
Re your question in comments:
You might be misunderstanding. The data is never in Memcached. There is no automatic synchronization between MySQL and Memcached. The only thing is that MySQL is mimicking the API and protocol of Memcached. There's no reason that they did this, except to make the API familiar to developers.
When you use the "Memcached API for InnoDB" you're connecting your application to a port listened to by the mysqld daemon process. Your requests on this connections read and write rows directly in the InnoDB storage engine. There's no Memcached instance in between.
The InnoDB Memcached Plugin is a feature in MySQL >= 5.6 that runs a Memcached daemon in the existing mysqld process supporting the Memcached API listening on a different port (11211 by default). Because it runs in the same process space as InnoDB, you get low-latency access to data stored in InnoDB tables and through existing, widely-available Memcached clients.
This has a few interesting use-cases:
Transparent support for adding InnoDB as a high-performance persistence layer to existing memcached clients/applications
Improved performance over SQL queries for interacting with InnoDB tables directly (especially for inserting new key/value pairs), since the simple Memcached protocol bypasses the overhead of SQL parsing and query-plan optimization
High-performance 'MySQL + caching layer' architecture, where the memcached daemon fetches data from from the underlying InnoDB table and serves cached data directly from local memory
Case #3 is particularly noteworthy: Although the Memcached plugin is configured by default to read/write directly to the underlying InnoDB table (innodb_only cache policy), it can be configured to use its own, separate local memory cache just like a standalone Memcached instance would, either without using InnoDB storage at all (cache-only), or using InnoDB as a backing store (caching). (Note that the currently-accepted answer is incorrect on this point.)
Refer to the architecture diagram from the documentation (note the 'local cache (optional)' component, which is used by the cache-only or caching cache policies):
(source: mysql.com)
In terms of setup/installation, if you're using the Memcached plugin for anything other than a standalone key-value cache (case #1), you will need to create a mapping from Memcached keys/values to your InnoDB tables/columns by writing a row to the special innodb_memcache.containers table. See Creating a New Table and Column Mapping for details.
A bit of a side-track here, but nowadays you're better of using the memcache_storage module. The module page has plenty of good pointers how to use the module and with what other modules it integrates nicely to have a better and faster caching for your site.
I'm trying to test the performance of using memcached on a MySQL server to improve performance.
I want to be able to use the normal MySQL command line, but I can't seem to get it to connect to memcached, even when I specify the right port.
I'm running the MySQL command on the same machine as both the memcached process and the MySQL server.
I've looked around online, but I can't seem to find anything about using memcached other than with program APIs. Any ideas?
Memcached has its own protocol. The MySQL client cannot connect directly to a memcached server.
You may be thinking of the MySQL 5.6 feature that allows MySQL server to respond to connections using a memcached-compatible protocol, and read and write directly to InnoDB tables. See http://dev.mysql.com/doc/refman/5.6/en/innodb-memcached.html
But this does not allow MySQL clients to connect to memcached -- it's the opposite, allowing memcached clients to connect to mysqld.
Re your comment:
The InnoDB memcached interface is not really a caching solution per se, it's a solution for using a familiar key/value API for persistent data in InnoDB tables. InnoDB does do transparent caching of data pages in its buffer pool, but this is no different from conventional data reads with SQL. InnoDB also commits all changes to its transaction log synchronously on commit.
Here's a blog from my colleague at Percona. He tested whether the MySQL 5.6 memcached API could be used as a caching layer, and found that actually using memcached is still superior.
http://www.mysqlperformanceblog.com/2013/03/29/mysql-5-6-innodb-memcached-plugin-as-a-caching-layer/
Here's one conclusion from that blog:
As expected, there is a slowdown for write operations when using the InnoDB version. But there is also a slight increase in the average fetch time.