What could be the best way to fetch records from a MySQL table for more than one clients connected, which are retrieving records concurrently and periodically.
So everyone gets the new messages as the new record enters the table but old messages should not retrieve again.
Current Table Structure:
MessageId, Message, DatePosted, MessageFromID
Thanks
Your problem can be translated to: How can each client know, which records to read and which records not.
There are two completly different approaches to that, with very different properties.
Let the client care for that
Let the server care for it
Model #1 would quite simply require, that you
Use something like an AUTO_INCREMENT on some field, if your MessageID is not guaranteed to be incrementing
On the server give each client not only the messages, but also the ID
Have the client keep this ID and use it as a filter for the next poll
Model #2 needs you to
Have another table with 'ClientID and MessageID'
Whenever a client gets a message, create a record there
Use non-existance of such a record as a polling filter
Related
I have a central database containing millions of IDs. And I have a group of users (50-100 users), all being able to request extraction of IDs from this big database.
Atm what I do is when a user sends a GET request, I SELECT 100 ids then update them with the flag USED and return the 100. The problem is, if I get too many requests at the same time, multiple users will receive the same ids (because I dont lock the db when doing select and then update)
If I lock the database my problem will be solved, but it will also be slower.
What other alternative I have?
Thanks!
Look ahead another step... What if a "user" gets 100 rows, then keels over dead. Do you have a way to release those 100 for someone else to work on?
You need an extra table to handle "check out" and "check in". Also, use that table to keep track of the "next" 100 to assign to a user.
When a user checks out the 100, a record of that is stored in the table, together with a timestamp and "who" checked them out. If they don't "check them back in within, say, an hour, then you assign that 100 to another user.
Back on something more mundane... How to pick 100. If there is an auto_increment id with no gaps, then use simple math to chunk up the list. If there are a lot of gaps, then use SELECT id FROM tbl WHERE id > $leftoff ORDER BY id LIMIT 100, 1 to get the end of the next 100.
If each user has their own key, you could pull from the millions of IDs starting from their key*10000. For example, user #9 would first get IDs #90000 to #90099, then #90100 to #90199 next time.
You could set the IDs as "Used" before they get sent back, so one user requesting IDs multiple times will never get duplicates. This needn't lock the database for other users.
If they don't request keys more than 100 times before the database can update, this should avoid collisions. You might need to add logic to allow users who request often not to run out, like by having a pool of IDs that can repopulate their supply, but that depends on particulars that aren't clear from the original question.
I am working on a little package using PHP and MySQL to handle entries for events. After completing an entry form the user will see all his details on a page called something like website.com/entrycomplete.php?entry_id=15 where the entry_id is a sequential number. Obviously it will be laughably easy for a nosey person to change the entry_id number and look at other people's entries.
Is there a simple way of camouflaging the entry_id? Obviously I'm not looking to secure the Bank of England so something simple and easy will do the job. I thought of using MD5 but that produces quite a long string so perhaps there is something better.
Security through obscurity is no security at all.
Even if the id's are random, that doesn't prevent a user from requesting a few thousand random id's until they find one that matches an entry that exists in your database.
Instead, you need to secure the access privileges of users, and disallow them from viewing data they shouldn't be allowed to view.
Then it won't matter if the id's are sequential.
If the users do have some form of authentication/login, use that to determine if they are allowed to see a particular entry id.
If not, instead of using a url parameter for the id, store it in and read it from a cookie. And be aware that this is still not secure. An additional step you could take (short of requiring user authentication) is to cryptographically sign the cookie.
A better way to implement this is to show only the records that belong to that user. Say the id is the unique identifier for each user. Now store both entry_id and id in your table (say table name is entries).
Now when the user requests for record, add another condition in the mysql query like this
select * from entries where entry_id=5 and id=30;
So if entry_id 5 does not belong to this user, it will not have any result at all.
Coming towards restricting the user to not change his own id, you can implement jwt tokens. You can give a token on login and add it to every call. You can then decrypt the token in the back end and get the user's actual id out of it.
We have a web app which requires to send lots of notifications to users (over 1,000,000 notifications per day). We use Laravel and MySQL for the database.
I loop through a group of users send notifications and save it to the database. Let's say if I want to send a group of 1000 users. The data will be written to the DB 1000 time. As I said, we have over 1,000,000 notifications per day, which take so many resources.
What's the appropriate way to approach this problem?
I should change to a new database system like MongoDB or I should redesign the notification table schema and the way I save to DB?
The following is my notification table schema.
PS:: I need to save every notification to DB and need to show to users. Can't skip this part.
If you're sending the same content for all users, I would suggest the following approach.
DATABASE SCHEMA
notification_contents (id, content, created_by, created_at, updated_at)
user_notifications (id, user_id, notification_id, status, created_at,
updated_at)
When there is a new notification is to be sent,
Insert it into the notification_contents table. You can use placeholders in your content if you want to customize the content. For example Username or email
Then loop through the users and insert them into the user_notifications table with the user_id and notification_id. Initially set a status flag as zero if you want to implement a queuing mechanism for this.
Then write a cron job to take x number of users from the user_notifications table and send the content to them. You can join user_notifications with notification_contents to get the content for email.
Set the status flag to one once the notification is sent.
Shrink the dataset.
Use ascii for UUIDs. Do you really need a UUID?
Compress UUIDs into BINARY(16). (This won't completely solve the scalability nightmare that UUIDs lead to, but it will help some.)
Use ENUM for types and status, not long VARCHARs.
What is the difference between the two "types"?
Do you need 8-byte BIGINT?
There are 3 TIMESTAMPs, do you need more than one?
I don't see anything like user_id??
Currently I am having a hard time deciding/weighing the pros/cons of tracking login information for a member website.
Currently
I have two tables, login_i and login_d.
login_i contains the member's id, password, last login datetime, and total count of logins. (member id is primary key and obviously unique so one row per member)
login_d contains a list of all login data in history which tracks each and every time a login occurs. It contains member's id, datetime of login, ip_address of login. This table's primary key is simply an auto-incremented INT field, really purposeless but need a primary and the only unique single field (an index on the otherhand is different but still not concerned).
In many ways I see these tables as being very similar but the benefit of having the latter is to view exactly when a member logged in, how many times, and which IP it came from. All of the information in login_i (last login and count) truthfully exists in login_d but in a more concise form without ever needing to calculate a COUNT(*) on the latter table.
Does anybody have advice on which method is preferred? Two tables will exist regardless but should I keep record of last_login and count in login_i at all if login_d exists?
added thought/question
good comment made below - what about also tracking login attempts based on a username/email/ip? Should this ALSO be stored in a table (a 3rd table I assume).
this is called denormalization.
you ideally would never denormalize.
it is sometimes done anyway to save on computationally expensive results - possibly like your total login count value.
the downside is that you may at some point get into a situation where the value in one table does not match the values in the other table(s). of course you will try your best to keep them properly up to date, but sometimes things happen. In this case, you will possibly generate bugs in application logic if they receive an incorrect value from one of the sources.
In this specific case, a count of logins is probably not that critical to the successful running of the app - so not a big risk - although you will still have the overhead of maintaining the value.
Do you often need last login and count? If Yes, then you should store it in login_i aswell. If it's rarely used then you can take your time process the query in the giant table of all logins instead of storing duplicated data.
i am scrubbing my head now for hours to solve thw following situation:
Several Html Forms on a webpage are identified by an id. Users can create forms on the clients side themselves and fill in data. How can I guarantee that the id of the form the user generates is unique and that there doesnt occure any collision in the saving process because the same id was generated by the client of someone else.
The problems/questions:
A random function on the client side could return identical id's on two clients
Looking up the SQL table for free id wouldnt solve the problem
Autoincrement a new id would complicate the whole process because DOM id and SQL id differ so we come to the next point:
A "left join" to combine dom_id and user_id to identify the forms in the database looks like a performance killer because i expect these tables will be huge
The question (formed as simple as i can):
Is there a way that the client can create/fetch a unique id which will be later used as the primary key for a database entry without any collisions? Whats the best practice?
My current solution (bad):
No unique id's at all to identify the forms. Always a combination through a left join to identify the forms generated by the specific user. But what happens if the user says: Delete my account (and my user_id) but leave the data on the server. I would loose the user id and this query qouldn't work anymore...
I am really sorry that i couldn't explain it in another way. But i hope someone understood what i am faced with and could give me at least a hint
THANK YOU VERY MUCH!
GUIDs (Globally Unique IDentifiers) might help. See http://en.wikipedia.org/wiki/GUID
For each form the client could generate a new GUID. Theoretically it should be unique.
I just don't show IDs to the user until they've submitted something, at which point they get to see the generated auto-increment id. It keeps things simple. If you however really need it, you could use a sequence table, but it has some caveats which make me advise against it:
CREATE TABLE sequence (id integer default 0, sequencename varchar(32));
Incrementing:
UPDATE sequence
SET id = #generated := id + 1
WHERE sequencename = 'yoursequencename';
Getting:
SELECT #generated;