I am going to build a website which will provide to the clients the ability to live chat user to user and also implement a notification system (for new updates, messages, events e.t.c). The primary database is going to be MySQL.
But i was thinking to release some "pressure" from the main database MySQL and instead of saving the chat messages and notifications into MySQL, create a personal SQLite for each client where the messages and notifications will be stored which as a result will leave MySQL do other more important queries. Each client can see only his/her messages and notifications so only he/she will open/read/write/update the corresponding SQLite database.
I think creating individual SQLite databases for each client is like partitioning the tables (messages, notifications) by user_id if they were stored in MySQL. So f.e: instead of quering each time through all the clients notifications in order to find the one which are for user with id:5, we just open his personal SQLite database which will contain ofc much fewer records (only his records).
Now to the question part:
As i said the queries will be simple. Just some selects with 1 or 2 where clauses (mostly on indexes) and some ordering. Since this is the first time i am going to use SQLite i am wondering:
1) Will this method (individual SQLite instead of MySQL) practically work in case of perfomance?
2) Which method will still perform better after some time when they will begin getting bigger and bigger? SQLite or MySQL?
3) Does SQLite have any memory limits which will make it run slower in time?
Whats your opinions on this method?
Related
I’ve been trying to optimize for performance one behemoth software based on php and mysql. I have gone through caching in Apache and indexes in MySQL but it is not enough.
Since all forms within this software are built and printed dynamically from configuration in the database the software sends huge number of SQL’s and does a lot of joins which slows the whole thing when there are many concurrent users connected (on average 200-300).
Since we cannot touch the code, I have seen that mysql-proxy can be placed between application server and database server and over there query results can be cached accessing redis o memchached via lua. My idea is to cache everything. However, the problems is invalidating the cache. Once record is updated how do I invalidate all cached result sets?
One of the ideas was convert SQL query into md5 and store result as a key of a set. But also do analysis of a query and store the same md5 key and references to the table. For example:
Query:
select * from products left join users on products.user_id = user.id
Cache Instance A
3b98ab273f45af78849db563df6598d1– {result set}
Cache Instance B
products - 3b98ab273f45af78849db563df6598d1
users - 3b98ab273f45af78849db563df6598d1
So once UPDATE or INSERT or DELETE is issued on of these tables it invalidates all result sets where the particular table was queried.
I see quite a lot of work with it and I was wondering if there are any simpler methods to achieve this.
I'm trying to find the best candidate for storing the follower/following user data,
I was thinking initially to store it in Redis in a set where user -> set of user ids but then I thought about scenario where there is over 1 million or even 10 million followers for a user, how Redis would handle such a huge set? also there is no way I can do pagination on a set in redis, I have to retrieve the whole set which will not work if a user wants to browse who following after him.
If I store it in MySQL I can definitely do pagination but it might take a long time fetching 10 million records from the database whenever I have to build the user feed, I can do this in old batch fashion but it still sounds pretty painful whenever a user who has many followers will post something and then processing these 10 million records would take forever just for fetching his followers.
Would it be worthwhile to store it in MySQL for pagination (mainly Frontend) and in Redis for the event-driven messaging which builds the activity feed?
It's a personal decision whether to use redis or mysql for this task. Both will have no problem with those 10 million records.
MySQL has the LIMIT x,y command for getting a subset of the followers from the database.
For redis you can use sorted sets and use the userid of the follower or the time user started following as score for an sorted set. And like MySQL redis supports getting a subset of the large sorted set.
Long story short - I am dealing with a largish database where basic user details (userid (index), username, password, parent user, status) are stored in one database and extended user details (same userid (index), full name, address etc. etc.) are stored in another database on another server.
I need to do a query where I select all users owned by a particular user (via the parent user field from basic user details database), sorted by their full name (from the extended user details field) and just 25 at a time (there are thousands, maybe tens of thousands for any one user).
As far as I can work out there are three possible solutions;
No JOIN - get all the user IDs in one query and run the second query based on those IDs. This would be fine, except the number of user IDs could get so high that it would exceed the maximum query length, or be horribly inefficient.
Replicate the database table with the basic user details onto the server with the extended details so I can do a JOIN
Use a federated storage engine table to achieve the same results as #2
It seems that 3 is the best option, but I have been able to find little information about performance and I also found one comment to be careful using this on production databases.
I would appreciate any suggestions on what would be the best implementation.
Thanks!
FEDERATED tables are a nice feature .. but they do not support indexes, which would slow down your application dramatically.
if (!) you do read only from the users database on the remote server.
replication would be more effective and also faster.
Talking in terms of performance or limitations, Federated Engine has a lot of limitations. It doesn't support for transactions, Performance on a FEDERATED table when performing bulk inserts is slower than with other table types etc..
Replication and Federated engines are not meant to do same things. First of all, did you try both?
Is it possible to cache recently inserted data in MySQL database internally?
I looked at query cache etc (http://dev.mysql.com/doc/refman/5.1/en/query-cache.html) but thats not what I am looking for. I know that 'SELECT' query will be cached.
Details:
I am inserting lots of data to MySQL DB every second.
I have two kind of users for this Data.
Users who query any random data
Users who query recently inserted data
For 2nd kind of users, my table has primary key as unix time-stamp which tells me how new the data is. Is there any way to cache the data at the time of insert?
One option is to write my own caching module which cache data and then 'INSERT'.
Users can query this module before going to MySQL DB.
I was just wondering if something similar is available.
PS: I am open to other database providing similar feature.
Usually you get the best performance from MySQL if you allow a big index cache (config setting key_buffer_size), at least for MyISAM tables.
If latency is really an issue (as it seems in your case) have a look at Sphinx which has recently introduced real-time indexes.
I need to log all post and get requests on web site in the database.
There will be two tables:
requests with time stamp, user id and requested URI
request parameters with name, value and request id
I will use it only for analytical reports once per month. No regular usage of this data.
I have about one million requests a day and the request parameters table will be very huge.
Can I handle such a large table in MySQL with no problems?
I'd avoid writing to the db on each request or you'll be vulnerable to slashdot effect. Parse your web logs during quiet times to update the db.
The usual solution of this type of problem is to write a program that parses the logs from the whole month. If You don't need sophisticated MySQL capabilities, You should consider this approach.
If You really need the database, then consider parsing logs offline. Otherwise, if Your database goes down, You will loose data. Logs are know to be pretty safe.
Table indexes are not free. The more indexes You have, the faster the queries run, but the more indexes You have, the slower inserting data becomes.
Yes, mysql will handle millions of rows normally, but depending on what you wanna do with your data later and on indexes on those tables perfomance may be not very high.
PS. In my project we have a huge pricelist with a few millions of products in it and it works without any problems.