I heard that a lot of big project administrators caches results. I wonder how do I do it? Thanks
I'm using PHP.
Pseudo-logic:
Test if query result exists in cache
If so,
retrieve from cache
return result
else
execute query against database
transfer results to an array, list or object(s) [dependent on language]
store results in cache (serialize if necessary)
return result
Often memcache or APC (if you're using PHP) can be used to store the cache, but files can be used at a pinch. Code will vary depending on what medium you're using for cacheing
In a tipical mysql - php environment memcached can be a good choice.
memcached FAQ
Related
problem
My question is I am developing a system. I click a query caching concept for fast response time. now I want to find which kind of traffic on system on web application is better for query caching and which is not. and what is the downside of query caching.
Whether Query Cache is good for you? Depends on
what MySQL version are you using
what is the scale of your application
what kind of queries you want to cache
How it works
If MySQL Query Cache is used, MySQL won't go to the trouble of parsing the query every time the query is hit. MySQL will look for a identical query in the query cache whenever a query is hit and if it finds the query, it won't need to parse it again, it will just send it to the server and fetch the results.
Issues & Limitations
Please do remember that the cache doesn't store data of your query. You will not receive old/stale data from a cached query. It just stores the parsed query. But a point to be made here is that if the underlying tables (of the cached query) undergo any change, all the tables being used in the cache will be invalidated.
Among other things, there are serious limitations to the Query Cache. Cached queries cannot be used for stored procedures, functions and triggers. They're also not used for queries which are subqueries of an outer query.
It was once considered a great tool for speeding up the queries, but recently MySQL development team has decided to retire this feature as they found some scalability issues with the query cache.
Do read this this article on MySQL Server Team's blog about retiring the Query Cache in MySQL 8.0
I've created a very simple development view on my production Couchbase server
function (doc, meta) {
if (meta.id.indexOf("user:") == 0) emit(meta.id, doc);
}
This view doesn't return any results. Testing the same view on my local couchbase server works fine.
Publishing this view as a production view works fine.
What could be wrong? I'm using Couchbase version: 2.2.0 community edition (build-837)
When developing views, Couchbase will only use a subset of data in your bucket for the results. IIRC this is one vBucket's worth of data. It uses a subset of data as some views can be quite large and when developing a new view you do not want to bog down the whole cluster. So while rare, it is possible that when you are running your development view, the subset of data picked up on that one cluster has no matching results. Once that index is promoted to be a production view, i.e. you are done developing it and are ready to use it in prod, then it'll use the whole data set in that bucket.
On a related note, it looks like you are emitting the entire document if I am not mistaken. Depending on your data set and the selectivity of this view on that data, that could get quite expensive as your data grows. So you will have to watch this. Make sure you have enough disk space for this and enough OS RAM (not taken up by Couchbase) to keep as much of that index in memory since it is treated differently than Couchbase objects. Emitting a lot of data can also slow down your rebalances, especially if you are using the default internal settings like index aware rebalances. I do not know your use case obviously, but it might be better to just emit the IDs and then bulk get those IDs or something. This might be one of those situations where can views do this, yes, but should they and is there a better way to solve the same thing. Anyhow, just something to think about.
I just started using couchbase and hoping to use it as my data store.
One of my requirements in performing a query that will return a certain field about all the documents in the store. This query is done once at the server startup.
For this purpose I need all the documents that exist and can't miss any of them.
I understand that views in couchbase are eventually consistent but I still hope this query can be done (at the cost of performance).
Notes about my configurating:
I have only one couchbase server instance (I dont need sharding or
replication)
I am using the java client (1.4.1)
What I have tried to do is saving my documents this way:
client.set(key, value, PersistTo.ONE).get();
And querying using:
query.setStale(Stale.FALSE);
Adding the PersistTo parameter caused the following exception:
Cause by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: <unknown>
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:167)
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:140)
So I guess I am actually asking 3 questions:
Is it possible to get the consistent results I need?
If so, is what I suggested the correct way of doing that?
How can I prevent those exceptions?
The mapping I'm using:
function (doc,meta) {
if (doc.doc_type && doc.doc_type == "MyType" && doc.myField) {
emit(meta.id,null);
}
}
Thank you
Is it possible to get the consistent results I need?
Yes it is possible to set Couchbase views to be consistent by setting the STALE flag to false as you've done. However there are performance impacts with this, so dependent on your data size the query may be slow, if you are only going to be doing it once a day then it should be ok.
Couchbase is designed to be a distributed system comprising of more than node, it's not really suitable for single node deployments. I have read (but can't find the link) that view performances are much better in larger clusters.
You are also forcing more of a sync processing model onto a system that shines with async requests, PersistTo is ok to use for some requests but not system wide on every call (personal opinion), it'll definitely throttle throughput and performance.
If so, is what I suggested the correct way of doing that?
You say the query is done after your application server is running, is this once per day or more? If once a day then your application should work (I'd consider upping the nodes ;)), if you have to do this query a lot and you are 'hammering' the node over and over with sets then I'd expect to see what you are currently experiencing.
How can I prevent those exceptions?
It could be a variety of reasons, what are the specs of your computer, RAM,CPU,DISK? How much ram is allocated to Couchbase, how much to your bucket, what % of the bucket ram is used?
I've personally seen this when I've hammered some lower end AWS instances on some not so amazing networks. What version of Couchbase are you using? It could be a whole variety of factors that and deserves to be a separate question.
Hope that helps!
EDIT regarding more information on the Stale = false parameter (from official docs)
http://docs.couchbase.com/couchbase-manual-2.2/#couchbase-views-writing-stale
The index is updated before the query is executed. This ensures that any documents updated (and persisted to disk) are included in the view. The client will wait until the index has been updated before the query has executed, and therefore the response will be delayed until the updated index is available.
I just installed yum install memcached on MySQL Server. But then as service memcached status is Running.., what to do more as i do not think nothing is happening in performance. I think i still need to configure MySQL to work with Memcached.
What configuration need at MySQL end to talk with Memcached? (How to do?)
Or even still need to configuration from the Web Server end also?
Please help i'm stuck.
Memcached is not a MySQL performance plugin - it's commonly used as a write-though or write-back caching system - often where you need to store commonly used key/value pairs without hitting your database unless they change - thus reducing load on your DB.
Visiting the Memcached website provides useful information:
Free & open source, high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.
Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.
Ultra-simplistic pseudo-code example:
Read:
if data in memcache {
// Retrieve and use
} else {
// Pull from DB, use and write to memcache
}
Or, here, Memcached describes a simple example for caching results:
Cache Results:
function get_foo(foo_id)
foo = memcached_get("foo:" . foo_id)
return foo if defined foo
foo = fetch_foo_from_database(foo_id)
memcached_set("foo:" . foo_id, foo)
return foo
end
It doesn't do anything unless you write your software/codebase to use it (usually, you write your DB functions to check the cache before hitting your DB). It's worth reading the Memcached Story of caching to understand a basic use-case scenario.
Historically, some have compared it to using HEAP or memory tables in MySQL, but it's not the same and can be distributed on many machines in a cluster and/or over a network where you have free/unallocated memory available for use - even on one server.
If you want to understand it's value at scale, then look no-further than the list of companies that use it.
Are you searching for this?
Mysql5.6 now support memcache integrated version plugin.
If in that case, you don't need to install another memcached server.
Simply install mysql 5.6. and memcached plugin.
Then.. as you can see in that page....
you can send memacached command to memcached server and memcached plugin automatically write to backup table which you want.
One thing that I think there should be more improvement is table field type.
Only varchar/text/blob type can be possible( backup table field which you want to interact with memcached server... ).
PS)
I heard that memcached plugin DML throughput is amazing.
but when I tried, I've only? got 30% improvement...
Is there an implementation of database (mysql) query caching written purely in Node.js?
I'm writing a Node web app and was planning on caching queries with memcached, but while considering this I realised it's probably possible to do the caching through a separate Node.js layer instead
To explain:
You could query the database through a node server on a separate port, returning data from memory where available and loading it into memory where it isn't.
Anyone know how Node.js would compare to memcache in terms of return speed on hashed arrays? Is this a pipe-dream or something I should look at?
I went ahead and wrote a caching solution for private use that stored the data in a shared object. This wasn't really query caching, it stores specific results instead of raw sql results ordered by hashes, but it kept what I needed in memory and was ridiculously easy to write.
Since I originally asked this question a number of node caching solutions have emmerged:
ptarjan/node-cache
tcs-de/nodecache
vxtindia/node-cache
mape/node-caching
I haven't used any of these but one of them might well be of use to someone else.
There are now also redis and memcached clients for node.
You can definitely implement something like this in node, and it could be an interesting project, but it depends on your needs. If you're just doing this for a hobby project, by all means, build a caching layer in node and try it out. Let us know how it goes!
If this is for production use, then I would recommend sticking to the established caching layers (memcached, redis, etc) as they have already gone through all of the growing pains associated with building a scalable caching system.
I have written a node.js module that performs MySQL query caching using memcached.
The module is named Memento and is available at https://www.npmjs.com/package/memento-mysql
Enjoy!