We are implementing a chat service, and I'd like some input on table design. Our service uses MySQL, and our DB has 2 tables, Threads and Messages. Threads table stores all the chat threads, and Messages table stores all the messages. A chat thread can have multiple messages, while a message belong to only one thread. Each message is identified by a column in Messages table called messageId.
We need to get the messageId of the last message of each thread from time to time in our service. I can see 2 options:
1 add a column called lastMessageId to Threads to keep track of the last message; each time a message is inserted into Messages table, we need to update Threads table as well;
2 each time we need the last message's id, perform a query on Messages table to find the last message;
Which option should I take, and why?
I would suggest to go for option 2, below are the reason.
You said that u need last message id time to time which means not so frequent.
Making a query time to time is less intensive than making an Update operation on every insert.
You can further fine tune your query by creating indexes on Messages Table.
Related
I have a central database containing millions of IDs. And I have a group of users (50-100 users), all being able to request extraction of IDs from this big database.
Atm what I do is when a user sends a GET request, I SELECT 100 ids then update them with the flag USED and return the 100. The problem is, if I get too many requests at the same time, multiple users will receive the same ids (because I dont lock the db when doing select and then update)
If I lock the database my problem will be solved, but it will also be slower.
What other alternative I have?
Thanks!
Look ahead another step... What if a "user" gets 100 rows, then keels over dead. Do you have a way to release those 100 for someone else to work on?
You need an extra table to handle "check out" and "check in". Also, use that table to keep track of the "next" 100 to assign to a user.
When a user checks out the 100, a record of that is stored in the table, together with a timestamp and "who" checked them out. If they don't "check them back in within, say, an hour, then you assign that 100 to another user.
Back on something more mundane... How to pick 100. If there is an auto_increment id with no gaps, then use simple math to chunk up the list. If there are a lot of gaps, then use SELECT id FROM tbl WHERE id > $leftoff ORDER BY id LIMIT 100, 1 to get the end of the next 100.
If each user has their own key, you could pull from the millions of IDs starting from their key*10000. For example, user #9 would first get IDs #90000 to #90099, then #90100 to #90199 next time.
You could set the IDs as "Used" before they get sent back, so one user requesting IDs multiple times will never get duplicates. This needn't lock the database for other users.
If they don't request keys more than 100 times before the database can update, this should avoid collisions. You might need to add logic to allow users who request often not to run out, like by having a pool of IDs that can repopulate their supply, but that depends on particulars that aren't clear from the original question.
I want design a inside system message for about 10000 user!
Steps seem easy:
1- create a DB table. for example: Message
2- create an user table. for example : user
3- the search Message DB and return all messages to user.
but you know for all 10000 user is too much time with one table to access messages.
My plan is create a independent table for each user for keep his messages. (not one table for all users)
but after that,Mybe I have 10000 tables in my DB. it is too much and administration is too heavy.
Is there a better way to save messages?
First , 10000 users is not so much data for mysql. In my system, I have more than 10,000,000 data in one table, and it works well.
Second, whether user and message should be stored in independent tables, this depends on the query. If you make a lot of join, I suggest You store them in one table, But if the join action is not that frequent, you can seperate them.
I am working on building a social network application similar to twitter where users have newsfeeds, followers, posts ect...
I am trying to implement a feature which would make posts (a post in my application is equivalent to a post on facebook) EXPIRE after a certain amount of time.
What do I mean by expire?
1. Post disappears from news feed
2. User whose post expires, relieves a notification alerting them that the post
has expired. On a programmatic level this is just a insert statement being executed when the post expires.
What have I done so far?
Making posts disappear from the newsfeed was simple, I just adjusted the query which return the newsfeed by checking the date_of_expiration column and compare it to NOW().
Creating notifications when the post expired was trickier.
My initial approach was to make a mysql CRON job which ran every 2 minutes, triggering an event which would select all posts where NOW() > date_of_expiration and use the selected data to insert a notification entry into my notification table.
This works, however, I do not want to use a CRON job. The 2 minute delay means a user might have to wait a full 2 minutes after the post actually expired before receiving the notification telling the user their post expired. I'm assuming if the table had many entries this wait time could be even greater depending on how long the it takes to run the select and insert statements.
What am I looking for?
Another solution to inserting a notification into the notification table when a users post expires.
I was thinking that if there was a way to create some kind of event that would trigger when the expiration date value for each row (in the posts table) is greater than NOW(), it would be a very good solution to my problem. Is something like this possible? What is commonly done in this scenario?
FYI my stack is: MYSQL, JAVA with an Android+IOS front end, but I don't mind going out of my stack to accomplish this feature
I am not sure how your application works. But here is a though, I have done in an application that interact with a telephone system where each second count.
I implemented a server-sent event where a script will keep checking for new updates every second. Then the script will update the client with any new/expired notifications.
I am not sure if this is what you are looking for but it is worth sharing.
EDITED
Since you are leaning more toward having a table for the notification why now create the notification at run time with in a transaction?
START TRANSACTION;
INSERT INTO posts(comment, createdBy....)Values('My new comment',123);
SELECT #lastID := LAST_INSERT_ID();
-- Create temporary table with all the friends to notify
-- "this will help you with performance" Hint then engine type
-- Make sure the final userId list is unique otherwise you will be
-- inserting duplicate notifications which I am sure you want to avoid
CREATE TEMPORARY TABLE myFriends (KEY(user_id)) ENGINE=MEMORY
SELECT 1 FROM users AS s
INNER JOIN friends AS f ON f.friedId = s.userId
WHERE s.userID = su.userID
-- insert the notifications all at once
-- This will also help you with the performance a little
INSERT INTO notifications(userID, postId, isRead)
SELECT userID, #lastID AS postId,'0' AS isRead
FROM users AS su
INNER JOIN myFriends AS f ON f.userId = su.userId;
-- commit the transaction if everything passed
commit;
-- if something fails
rollback;
more thoughts, depending how busy you application will be things to consider
Make sure your server is built with good hardware. lots of RAM 64GB+ and a good hard drives, SSD will be great if possible,
Also, you may consider using GTID replication to have more sources for read.
This is hard to answer, since i don't understand well enough your database schema or the access pattern of the clients. However, I have some ideas that might help you:
What about marking the posts table as expired with a separate "expired" column? If you do that, you could select the posts that are to be sent to the client by getting all posts that are not marked as expired. This of course will include also the messages that are newly expired (NOW() > date_of_expiration) but are not marked yet. Let your java program sort the freshly expired posts out before sending the reply. At this point in your program you already have the posts that need to be marked and these are the exact same ones that need to be inserted into the notification table. You can just do that at this place in your Java program.
Advantage
No need for EVENTS or Cron jobs at all. This should be fairly efficient if you set indexes correctly in your tables. No need for a JOIN with the notification table.
Disadvantage
You need to store the expired info extra in a column, that may require a schema change.
What could be the best way to fetch records from a MySQL table for more than one clients connected, which are retrieving records concurrently and periodically.
So everyone gets the new messages as the new record enters the table but old messages should not retrieve again.
Current Table Structure:
MessageId, Message, DatePosted, MessageFromID
Thanks
Your problem can be translated to: How can each client know, which records to read and which records not.
There are two completly different approaches to that, with very different properties.
Let the client care for that
Let the server care for it
Model #1 would quite simply require, that you
Use something like an AUTO_INCREMENT on some field, if your MessageID is not guaranteed to be incrementing
On the server give each client not only the messages, but also the ID
Have the client keep this ID and use it as a filter for the next poll
Model #2 needs you to
Have another table with 'ClientID and MessageID'
Whenever a client gets a message, create a record there
Use non-existance of such a record as a polling filter
We are to design a architecture storing conversations (one to many)...
we are using the following tables (and columns)
messages_table - message_id, sender_id (user_id), timestamp, reference_id
recipients - message_id, recipient_id (user_id)
unread_status_table - message_id, user_id
we are storing messages in messages table, and reference_id stores the message id of starting thread.
unread status table saves only messages that are unread.
I am not sure if we should use a separate table for unread messages, advantage is, if all the messages are read, the table is empty.
please help me :)
I don't think there is one single answer to a question like this. It all depends on how many users, how many conversations, the hardware you have, requirements for how responsive the system should be to users, and so forth. If you can get adequate performance by not having a second table for unread messages, then that's probably the better route. Always simplify when possible.
My thought is that, if I had a second table, it probably wouldn't be for unread messages. Instead, in a very high-concurrency situation, that table would have no indexes apart from a primary key, and new messages would be pushed into it. It should allow for very fast operations even when lots of new messages come in. Then a job could run periodically to read a small number of records and push them into the main table, where the operation is slower, and then delete them from the second table. This means that users can continue to create new messages and the system stays responsive, but it might be a little while until recipients see the messages.
But again, this approach is needed only when you have seen that performance requires it.