I'm trying to find the best candidate for storing the follower/following user data,
I was thinking initially to store it in Redis in a set where user -> set of user ids but then I thought about scenario where there is over 1 million or even 10 million followers for a user, how Redis would handle such a huge set? also there is no way I can do pagination on a set in redis, I have to retrieve the whole set which will not work if a user wants to browse who following after him.
If I store it in MySQL I can definitely do pagination but it might take a long time fetching 10 million records from the database whenever I have to build the user feed, I can do this in old batch fashion but it still sounds pretty painful whenever a user who has many followers will post something and then processing these 10 million records would take forever just for fetching his followers.
Would it be worthwhile to store it in MySQL for pagination (mainly Frontend) and in Redis for the event-driven messaging which builds the activity feed?
It's a personal decision whether to use redis or mysql for this task. Both will have no problem with those 10 million records.
MySQL has the LIMIT x,y command for getting a subset of the followers from the database.
For redis you can use sorted sets and use the userid of the follower or the time user started following as score for an sorted set. And like MySQL redis supports getting a subset of the large sorted set.
Related
I'm currently working on a query that gets data from two tables on two independent MySQL servers. Table A keeps game user data such as player GUID, game-server. And table B stores logs such as game-currency consumption log with player GUID. The number of logs in table B is around 6 millions.
What I did so far is SELECT and CONCAT user IDs from table A to a very long string, and pass it to a WHERE CLAUSE to query logs from table B. But in some special cases the number of target IDs is more than thousands, and made my query very slow cause I'm not able to perform an INNER JOIN for filtering rows I need. And worse, sometimes the concated ID string would be too long that crashes my stored procedure.
I've googled for this problem for days, but the only solution I get is to use federated tables. However, I found it may be slow when the data is in a large amount and it has security risks, so I'm not really sure whether I should use federate tables.
Is there any other solution to efficiently query data from two independent MySQL servers? Thanks for your helps.
I am developing a multi-tenant application where for each tenant I create separate set of 50 tables in a single MySQL database in LAMP environment.
In each set average table size is 10 MB with the exception of about 10 tables having size between 50 to 200MB.
MySQL InnoDB creates 2 files(.frm & .ibd) for each table.
For 100 tenants there will be 100 x 50 = 5000 Tables x 2 Files = 10,000 Files
It looks too high to me. Am I doing it in a wrong way or its common in this kind of scenario. What other options I should consider ?
I also read this question but this question was closed by moderators so it did not attract many thoughts.
Have one database per tenant. That would be 100 directories, each with 2*50 = 100 files. 100 is reasonable; 10,000 items in a directory is dangerously high in most operating systems.
Addenda
If you have 15 tables that are used by all tenants, put them in one extra database. If you call that db Common, then consider these snippits:
USE Tenant; -- Customer starts in his own db
SELECT ... FROM table1 ...; -- Accesses `table1` for that tenant
SELECT a.this, b.blah
FROM table1 AS a -- tenant's table
JOIN Common.foo AS b ON ... -- common table
Note on grants...
GRANT ALL PRIVILEGES ON Tenant_123.* TO tenant_123#'%' IDENTIFIED BY ...;
GRANT SELECT ON Common.* TO tenant_123#'%';
That is, it is probably OK to 'grant' everything to his own database. But he show have very limited access to the Common data.
If, instead, you manage the logins and all accesses go through, say, a PHP API, then you probably have only one mysql 'user' for all accesses. In this case, my notes above about GRANTs are not relevant.
Do not let the Tenants have access to everything. Your entire system will quickly be hacked and possibly destroyed.
Typically, this has little to do with which way to do it versus which way you've basically sold your customers on how it's to be done, or in some cases having no choice due to the type of data.
For example, does your application have a policy or similar that defines isolation of user generated data? Does your application store HIPAA or PCI type data? If so, you may not even have a choice, and if the customer is expecting that sort of privacy, that normally comes at a premium due to the potential overhead of creating the separation.
If the separation/isolation of data is not required, then adding a field to tables indicating which application owns the data would be most ideal from a performance perspective, and you would just need to update your queries to filter based on that.
Using MySQL or MariaDB I prefer to use a single database for all tenants and restrict access to data by using a different database user per tenant which only has permission to their data.
You can accomplish by using an tenant_id column that stores the database username of the tenant that owns the data. I use a trigger to populate this column automatically when new rows are added. I then use Views to filter the tables where the tenant_id = current_database_user. Then I restrict the tenant database users to only have access to the views, not the real tables.
I was able to convert a large single-tenant application to a multi-tenant application over a weekend using this technique because I only needed to modify the database and my database connection code.
I've written a blog post fully describing this approach. https://opensource.io/it/mysql-multi-tenant/
I am going to build a website which will provide to the clients the ability to live chat user to user and also implement a notification system (for new updates, messages, events e.t.c). The primary database is going to be MySQL.
But i was thinking to release some "pressure" from the main database MySQL and instead of saving the chat messages and notifications into MySQL, create a personal SQLite for each client where the messages and notifications will be stored which as a result will leave MySQL do other more important queries. Each client can see only his/her messages and notifications so only he/she will open/read/write/update the corresponding SQLite database.
I think creating individual SQLite databases for each client is like partitioning the tables (messages, notifications) by user_id if they were stored in MySQL. So f.e: instead of quering each time through all the clients notifications in order to find the one which are for user with id:5, we just open his personal SQLite database which will contain ofc much fewer records (only his records).
Now to the question part:
As i said the queries will be simple. Just some selects with 1 or 2 where clauses (mostly on indexes) and some ordering. Since this is the first time i am going to use SQLite i am wondering:
1) Will this method (individual SQLite instead of MySQL) practically work in case of perfomance?
2) Which method will still perform better after some time when they will begin getting bigger and bigger? SQLite or MySQL?
3) Does SQLite have any memory limits which will make it run slower in time?
Whats your opinions on this method?
I'm pulling match data from an online game API and saving details to a locally hosted mysql database. Each call to the API returns about 100 matches and I'm inserting 15 matches at a time. For each match i'm inserting anywhere from 150-250 rows in 5 tables.
I've used the optimizations described here: Fastest Way of Inserting in Entity Framework
I've been able to insert about 9 matches/sec but now that I've saved 204,000 matches the insertion time has slowed to 2.5 matches/sec. I hope to save all matches since inception which is probably around 300+ million matches.
I can't use SqlBulkCopy because this is a mysql database.
Is there any further optimizations that I can do? I'd like to parallelize, but I suppose I'll still be blocked on DB.
Thanks.
I need to log all post and get requests on web site in the database.
There will be two tables:
requests with time stamp, user id and requested URI
request parameters with name, value and request id
I will use it only for analytical reports once per month. No regular usage of this data.
I have about one million requests a day and the request parameters table will be very huge.
Can I handle such a large table in MySQL with no problems?
I'd avoid writing to the db on each request or you'll be vulnerable to slashdot effect. Parse your web logs during quiet times to update the db.
The usual solution of this type of problem is to write a program that parses the logs from the whole month. If You don't need sophisticated MySQL capabilities, You should consider this approach.
If You really need the database, then consider parsing logs offline. Otherwise, if Your database goes down, You will loose data. Logs are know to be pretty safe.
Table indexes are not free. The more indexes You have, the faster the queries run, but the more indexes You have, the slower inserting data becomes.
Yes, mysql will handle millions of rows normally, but depending on what you wanna do with your data later and on indexes on those tables perfomance may be not very high.
PS. In my project we have a huge pricelist with a few millions of products in it and it works without any problems.