I've my messenger app which sends request to server for group creation, server process the request(making a database entry of group) and send back response, but sometimes it happens due to weak connection, response is not received in particular time instant, as a result client sends request again for the same group.
The fault which occurs in this case the server processes both these request and makes two entries (or more in case of more requests) in the database with different group_id for the same group.
How can I avoid multiple entries in database and make it consistent?
Due to multiple entries, when client reinstall app, if there are three entries of a group in database, all three will be loaded in app.
One solution which I thought of is that check if the group with given name already exist, but this is not the accepted solution, since client can create more one group with same name.
Note:
I'm using MYSQL Enterprise edition for storing entries on server.
You can think of group creation as same as groups are created in WhatsApp messenger.
Packet Id is unique for such repeating JSON requests being sent to server. Use that as a filter and discard the duplicate packet Ids. Same as done with message packets and other requests.
Related
I am reading this writing: https://medium.com/#narengowda/system-design-dropbox-or-google-drive-8fd5da0ce55b. In the Synchronization Service part, it writes that:
The Response Queues that correspond to individual subscribed clients are responsible for delivering the update messages to each client. Since a message will be deleted from the queue once received by a client, we need to create separate Response Queues for each client to be able to share an update message which should be sent to multiple subscribed clients.
The context is that we need a response queue to send the file updates from one client to other clients. I am confused by this statement. If Dropbox has 100 million clients, we need to create 100 million queues, based on the statement. It is unimaginable to me. For example, a Kafka cluster can support up to 5K topics (https://stackoverflow.com/questions/32950503/can-i-have-100s-of-thousands-of-topics-in-a-kafka-cluster#:~:text=The%20rule%20of%20thumb%20is,5K%20topics%20should%20be%20fine.). We need 20K Kafka clusters in this case. Which queuing system can do 100 million "topics"?
Not sure but I expect such notification to clients via web-sockets only.
Additionally as this medium blog states that if client is not online then messages might have to be persisted in DB. After that when client comes online they can request for all updates after certain timestamp after which web sockets can be setup to facilitate future communication.
Happy to know your thoughts on this.
P.S : Most of dropbox system design blogs/vlogs have just copied from each other without going into low level detail.
What is the best way to store user's socket.id across multiple servers? Take for example a simple chat app, if two users on different servers are sending messages to each other, the two servers must store each user's socket id somewhere so they can send the message from one user to another.
Currently I am using a redis hash to store each user's socket id (if they are online) but this doesn't work if a user has two connections (for example they have two tabs of the chat app open). Is the best approach to continue using redis but restructure the data structure in a way that makes it work when a user is connected twice, or would it be better to move to something like mongodb or mysql?
I would also like a way to expire data, for example if a socket id is stored for more than 24h then it should be automatically deleted. I have looked into doing this with redis but it doesn't seem possible to delete only one pair inside a hash. Is expiring data something that can be done in mysql or mongodb?
Did you try socket rooms?
check this link for rooms and namespaces
for example, if a user has multiple connections join them in a room with a unique name(maybe the userId or something)
I'm building a mobile app where a pull-down gesture on the UI initiates an update of existing data/posts (also retrieves new posts if there are any, but that's not the point here). The server is stateless meaning there is no sessions.
If the posts have been updated in the database, how do I let the front-end know which posts need to be updated? Only way I could think of is to send a list of ids of all retrieved posts to the server, and have it check if any of the posts have been modified since the time fetched.
This however seems quiet inefficient as the users might have stacked up hundreds of posts in some extreme cases, and it's most likely that only few or none of the posts need to be updated. Issuing hundreds of db requests could be a huge overhead.
There are at least 2 ways of doing this
Long Polling
client requests server for new information.
the server keeps the connection open until there is some new data to send.
once the server gets data, it sends this to the client and connection is closed.
the client then sends a new request for information.
This is a continuous process.
WebSockets
Create a websocket connection, keep it open.
The server pushes any updates as and when they come.
Problems with both situations
May take a significant amount of time to have production ready implementations.
Both them will require the server to be aware of any change in the database. This can be tricky as well
I'm trying to understand whether it is possible to achieve the following:
I have multiple instances of an application server running behind a round-robin load balancer. The client expects GET after POST/PUT semantics, in particular the client will make a POST request, wait for the response and immediately make a GET request expecting the response to reflect the change made by the POST request, e.g:
> Request: POST /some/endpoint
< Response: 201 CREATED
< Location: /some/endpoint/123
> Request: GET /some/endpoint/123
< Response must not be 404 Not Found
It is not guaranteed that both requests are handled by the same application server. Each application server has a pool of connections to the DB. Each request will commit a transaction before responding to the client.
Thus the database will on one connection see an INSERT statement, followed by a COMMIT. One another connection, it will see a SELECT statement. Temporally, the SELECT will be strictly after the commit, however there may only be a tiny delay in the order of milliseconds.
The application server I have in mind uses Java, Spring, and Hibernate. The database is MySQL 5.7.11 managed by Amazon RDS in a multiple availability zone setup.
I'm trying to understand whether this behavior can be achieved and how so. There is a similar question, but the answer suggesting to lock the table does not seem right for an application that must handle concurrent requests.
Under ordinary circumstances, you will not have any issue with this sequence of requests, since your MySQL will have committed the changes to the database by the time the 201 response has been sent back. Therefore, any subsequent statements will see the created / updated record.
What could be the extraordinary circumstances under which the subsequent select will not find the updated / inserted record?
Another process commits an update or delete statement that changes or removes the given record. There is not too much you can do about this, since it is part of the normal operation. If you do not want such thing to happen, then you have to implement application level locking of data.
The subsequent GET request is routed not only to a different application server, but that one uses (or is forced to use) a different database instance, which does not have the most updated state of that record. I would envisage this to happen if either application or database server level there is a severe failure, or routing of the request goes really bad (routed to a data center at a different geographical location). These should not happen too frequently.
If you're using MyISAM tables, you might be seeing the effects of 'concurrent inserts' (see 8.11.3 in the mysql manual). You can avoid them by either setting the concurrent_insert system variable to 0, or by using the HIGH_PRIORITY keyword on the INSERT.
I'm developing an app using socket.io where users send and receive data to users who are present in there channels/rooms. Here, I need your suggestion for storing data that is passed from user to a channel. So when some one enters that channel he should be able to get the data from that particular channel he entered.
So how will I save the data to the particular channel?
I had planned for storing data in MySQL database, which will have channel id, channel name, and channel message columns.
But I think it will be a problem if number of users increases and inserting each message as a new row into database?
Please help me the best way for these query.
Until you have thousands of simultaneous users, it hardly matters. Just use whatever you are most comfortable with. When you get thousands of users you can change the architecture, if necessary.