We have a web app which requires to send lots of notifications to users (over 1,000,000 notifications per day). We use Laravel and MySQL for the database.
I loop through a group of users send notifications and save it to the database. Let's say if I want to send a group of 1000 users. The data will be written to the DB 1000 time. As I said, we have over 1,000,000 notifications per day, which take so many resources.
What's the appropriate way to approach this problem?
I should change to a new database system like MongoDB or I should redesign the notification table schema and the way I save to DB?
The following is my notification table schema.
PS:: I need to save every notification to DB and need to show to users. Can't skip this part.
If you're sending the same content for all users, I would suggest the following approach.
DATABASE SCHEMA
notification_contents (id, content, created_by, created_at, updated_at)
user_notifications (id, user_id, notification_id, status, created_at,
updated_at)
When there is a new notification is to be sent,
Insert it into the notification_contents table. You can use placeholders in your content if you want to customize the content. For example Username or email
Then loop through the users and insert them into the user_notifications table with the user_id and notification_id. Initially set a status flag as zero if you want to implement a queuing mechanism for this.
Then write a cron job to take x number of users from the user_notifications table and send the content to them. You can join user_notifications with notification_contents to get the content for email.
Set the status flag to one once the notification is sent.
Shrink the dataset.
Use ascii for UUIDs. Do you really need a UUID?
Compress UUIDs into BINARY(16). (This won't completely solve the scalability nightmare that UUIDs lead to, but it will help some.)
Use ENUM for types and status, not long VARCHARs.
What is the difference between the two "types"?
Do you need 8-byte BIGINT?
There are 3 TIMESTAMPs, do you need more than one?
I don't see anything like user_id??
Related
Let's say that I have a website and I want to know all the users that logged in during a certain time interval.
Would it be a good idea to create a new table in the database for this purpose and add a new entry whenever a users logs in?
The table would contain 2 columns: the id of the user and the login date.
My main concern is that the number of entries from the table will become extremely large.
Can this be considered a good idea? Do you know if this method is being applied for other websites?
Thanks in advance!
The number of records in a table can be controlled via external script, which is put on cron/scheduler. If it becomes too big, old records can be removed
if it is not possible, as a workaround there could be a check of the number of records on each insert
just do not forget to set an index on the date field...
Yes, you can create a table that logs all the login time of each user. If there are millions of users you might want to store the recent login time instead. If space is not a problem then it will be good to store the login time each time a user is authenticated or authorized. Like this you can archive the data in this table periodically.
The general answer to this question is 'depends'.
You can:
Add user to the table on login. You hit the disk for each user, so be careful with a big amount of users.
You store a bunch of users in memory and write all the group at a certain size or time. This way you hit the disk fewer times.
Depending on how many users you expect you can think of a no-SQL solution.
Depending on your system, I advise the 2nd o 3rd approach
Read this for more info: Fast write performance, even if reads are very slow
I want design a inside system message for about 10000 user!
Steps seem easy:
1- create a DB table. for example: Message
2- create an user table. for example : user
3- the search Message DB and return all messages to user.
but you know for all 10000 user is too much time with one table to access messages.
My plan is create a independent table for each user for keep his messages. (not one table for all users)
but after that,Mybe I have 10000 tables in my DB. it is too much and administration is too heavy.
Is there a better way to save messages?
First , 10000 users is not so much data for mysql. In my system, I have more than 10,000,000 data in one table, and it works well.
Second, whether user and message should be stored in independent tables, this depends on the query. If you make a lot of join, I suggest You store them in one table, But if the join action is not that frequent, you can seperate them.
I am working on building a social network application similar to twitter where users have newsfeeds, followers, posts ect...
I am trying to implement a feature which would make posts (a post in my application is equivalent to a post on facebook) EXPIRE after a certain amount of time.
What do I mean by expire?
1. Post disappears from news feed
2. User whose post expires, relieves a notification alerting them that the post
has expired. On a programmatic level this is just a insert statement being executed when the post expires.
What have I done so far?
Making posts disappear from the newsfeed was simple, I just adjusted the query which return the newsfeed by checking the date_of_expiration column and compare it to NOW().
Creating notifications when the post expired was trickier.
My initial approach was to make a mysql CRON job which ran every 2 minutes, triggering an event which would select all posts where NOW() > date_of_expiration and use the selected data to insert a notification entry into my notification table.
This works, however, I do not want to use a CRON job. The 2 minute delay means a user might have to wait a full 2 minutes after the post actually expired before receiving the notification telling the user their post expired. I'm assuming if the table had many entries this wait time could be even greater depending on how long the it takes to run the select and insert statements.
What am I looking for?
Another solution to inserting a notification into the notification table when a users post expires.
I was thinking that if there was a way to create some kind of event that would trigger when the expiration date value for each row (in the posts table) is greater than NOW(), it would be a very good solution to my problem. Is something like this possible? What is commonly done in this scenario?
FYI my stack is: MYSQL, JAVA with an Android+IOS front end, but I don't mind going out of my stack to accomplish this feature
I am not sure how your application works. But here is a though, I have done in an application that interact with a telephone system where each second count.
I implemented a server-sent event where a script will keep checking for new updates every second. Then the script will update the client with any new/expired notifications.
I am not sure if this is what you are looking for but it is worth sharing.
EDITED
Since you are leaning more toward having a table for the notification why now create the notification at run time with in a transaction?
START TRANSACTION;
INSERT INTO posts(comment, createdBy....)Values('My new comment',123);
SELECT #lastID := LAST_INSERT_ID();
-- Create temporary table with all the friends to notify
-- "this will help you with performance" Hint then engine type
-- Make sure the final userId list is unique otherwise you will be
-- inserting duplicate notifications which I am sure you want to avoid
CREATE TEMPORARY TABLE myFriends (KEY(user_id)) ENGINE=MEMORY
SELECT 1 FROM users AS s
INNER JOIN friends AS f ON f.friedId = s.userId
WHERE s.userID = su.userID
-- insert the notifications all at once
-- This will also help you with the performance a little
INSERT INTO notifications(userID, postId, isRead)
SELECT userID, #lastID AS postId,'0' AS isRead
FROM users AS su
INNER JOIN myFriends AS f ON f.userId = su.userId;
-- commit the transaction if everything passed
commit;
-- if something fails
rollback;
more thoughts, depending how busy you application will be things to consider
Make sure your server is built with good hardware. lots of RAM 64GB+ and a good hard drives, SSD will be great if possible,
Also, you may consider using GTID replication to have more sources for read.
This is hard to answer, since i don't understand well enough your database schema or the access pattern of the clients. However, I have some ideas that might help you:
What about marking the posts table as expired with a separate "expired" column? If you do that, you could select the posts that are to be sent to the client by getting all posts that are not marked as expired. This of course will include also the messages that are newly expired (NOW() > date_of_expiration) but are not marked yet. Let your java program sort the freshly expired posts out before sending the reply. At this point in your program you already have the posts that need to be marked and these are the exact same ones that need to be inserted into the notification table. You can just do that at this place in your Java program.
Advantage
No need for EVENTS or Cron jobs at all. This should be fairly efficient if you set indexes correctly in your tables. No need for a JOIN with the notification table.
Disadvantage
You need to store the expired info extra in a column, that may require a schema change.
What could be the best way to fetch records from a MySQL table for more than one clients connected, which are retrieving records concurrently and periodically.
So everyone gets the new messages as the new record enters the table but old messages should not retrieve again.
Current Table Structure:
MessageId, Message, DatePosted, MessageFromID
Thanks
Your problem can be translated to: How can each client know, which records to read and which records not.
There are two completly different approaches to that, with very different properties.
Let the client care for that
Let the server care for it
Model #1 would quite simply require, that you
Use something like an AUTO_INCREMENT on some field, if your MessageID is not guaranteed to be incrementing
On the server give each client not only the messages, but also the ID
Have the client keep this ID and use it as a filter for the next poll
Model #2 needs you to
Have another table with 'ClientID and MessageID'
Whenever a client gets a message, create a record there
Use non-existance of such a record as a polling filter
i wanna create a 2 level status message system. Which is the best way to create a tables ?
Scope:
User sets a Status Message
Users Reply to the status message
this is a picture showing it
Tables i have created
users (id, name .... )
status_messages (id, message, time, user_id)
status_message_replies (id, message, time, status_message_id, user_d)
Some one suggested this can be done in a single table format
status_messages (id, pid, message, time, user_id)
where pid = selfId or ParentId of the status.
I wanna know which is the best method to create the system ?
As long as the original messages and the responses have the same structure (set of attributes, or columns) then you can use the single table approach. It has the advantage that you can search over original messages and responses with a single query.
The set of original messages can be found where pid = selfid and the responses where pid <> selfid. If it's important to be able to see the original and response messages separately (without knowledge of the storage mechanism) you can encapsulate the above conditions in two VIEWs: OriginalMessages and Responses.
If the originals and responses have different attributes (for instance, if you want the original to allow links to URLs, photos, etc) you might consider using two separate tables. But even there, I'd probably argue for the one table structure with a separate, extender table for the additional attributes. That means you don't have to store often-empty columns for those original messages that don't use the extended attributes, and you can later easily add the extended attributes to the response messages as well (if desired).
A classical IS-A relationship: every reply is a message with an extra attribute (the message it is a reply to).
This is probably not the best way to model it. You'll be running the risk of having to write a lot of UNION queries over those two tables.
Alternatives:
just one table: status_messages (id, message, time, status_message_id, user_id), and allowing status_message_id to be NULL
use a HAS-A: one table status_messages (id, message, time, user_id) and one table replies (reply_id, replies_to_id
The former has the disadvantage that working with NULL is tricky in SQL.
The latter will necessitate joins when you want to query replies specifically.
BTW it's much clearer (IMO) to name columns after the relationship they stand for, not the table they refer to.