I've kinda silly question. I have a small community website. I'm thinking to make specific pages which can be viewed only by the members who have permission. So I suppose i will add each member ID in the database and when a member will try to access the page then i will first check if the member is logged in and then i will check the user ID, if it exists in the database table of users which have permission to view that content. Now Im just wondering if the database grows up, wont it take a long time to check everythng before loading the page?
Premature optimization is the root of all evil (Donald Knuth)
You can easily handle several millions of users with a single database, so that won't be a problem until your community is huge. When you reach that step, you can switch to more scalable DB solutions like Cassandra.
Having that said, take Brad Christie's comment into account, and use a reasonable identity management that won't thrash your database unnecessarily.
"a long time" is subjective and depends on many factors. For a small community website, you will likely not run into any issues with the method you've described. Still, it is considered best practice, and will speed up queries significantly, if you make use of proper indexes. Columns that will be queried against, such as the user ID, should be indexed. Not using an index means that MySQL has to read every record in your table and check to see if it matches your criteria.
This article may be of use to you:
http://www.databasejournal.com/features/mysql/article.php/1382791/Optimizing-MySQL-Queries-and-Indexes.htm
Also, if you are concerned about how your site will perform when your dataset grows, consider populating it with a bunch of dummy data and running a few tests. This site will help you generate a bunch of data to put in your database.
http://www.generatedata.com/#about
Lastly, if pages are not specific to a particular person or small group of people, consider using more general buckets for access control. For example, if only admins can view a page, tie that page to an "admin" permission and note which users are admins. Then, you can do a quick check to see what type or types of user a particular person is, and decide to show them the page or not. This type of system is typically refered to as an Access Control List (ACL).
http://en.wikipedia.org/wiki/Access_control_list
Related
So currently I'm working on a web application for a game and stumbled upon the following "problem"/question: Is it actually safer to have an individual table for the players/users and administrators?
Personally I think it's not and storing them both in one table would also be easier and more efficient, since every administrator is considered a player as well, but some people tell me it's safer to separate them without a clear reason why.
As for now I have them both stored in one table and am using a role based permission system.
By separating them you could control permissions to the tables with more granularity, for example limiting access to the administrators table to only certain database user accounts (so that it cannot be accessed by "game" code/servers at all).
But outside of those kinds of scenarios, I can't really think of one and agree with the second comment to your question.
I have a website on a shared host, where I expect a lot of visitors. I don't need a database for reading (everything presented on the pages is hardcoded in PHP) but I would like to store data that my users enter, so for writing only. In fact, I only store this to do a statistical analysis on it afterwards (on my local computer, after downloading it).
So my two questions:
Is MySQL a viable option for this? It is meant to run on shared hosting, with PHP/MySQL available, so I cannot really use much other fancy packages, but if e.g. writing to a file would be better for this purpose, that's possible too. As far as I understood, adding a line to a file is a relatively complex operation for huge files. On the other hand, 100+ users simultaneously connecting to a MySQL database is probably also a huge load, even if it's just for doing 1 cheap INSERT query.
If MySQL is a good option, how should the table best be configured? Currently I have one InnoDB table, with a primary key id that auto-increments (next to of course the columns storing the data). This is general-purpose configuration, so maybe there are more optimized ways given that I only need to write to the table, and not read from it?
Edit: I mainly fear that the website will go viral as soon as it's released, so I expect the users to visit in a very short timeframe. And of course I would not like to lose the data that they enter due to an overloaded database.
MySQL is a perfectly reasonable choice for this. Probably much better than a flat file, since you say you want to aggregate and analyze this data later. Doing so with a flat file might take a long time, especially if the file is large. Additionally, RDBMS are for aggregation and dataset manipulation. Ideal for creating report data.
Put whatever data columns you want in your table, and some kind of identifier to track a user, in addition to your existing row key. IP address is a logical choice for user tracking, or a magic cookie value could potentially work. It's only a single table, you don't need to think too hard about it. You may want to add nonclustered indexes on columns you'll frequently filter on for reports, e.g. IP address, access date, etc.
I mainly fear that the website will go viral as soon as it's released, so I expect the users to visit in a very short timeframe. And of course I would not like to lose the data that they enter due to an overloaded database.
RDBMS such as MySQL are explicitly designed to handle heavy loads, assuming appropriate hardware backing. Don't sweat it.
I need professional programmers/DBAs to bounce my idea off of and to know if it would/could even work. Please read below and give me any information that may break this theory. Thanks.
Overview of Website Idea:
The website will be used by sports card collectors to chat, answer questions on forums, showcase their cards/box breaks, trade/sell to/with other users, and keep a collection of their cards.
Design Issue:
A user can have an unlimited number of cards. This could make for some very large tables.
Design Question:
I do not want to limit the users on how many cards they can have in their collection on the site. If they have 5 copies of one card, and would rather have 5 records, one for each card, then that is their prerogative. This may also be necessary as each of the cards may be in a different condition. However, by allowing this to happen, this means that having only one table to store all records for all users is not even close to an option. I know sports card collectors with over 1,000,000 cards.
I was thinking that by either creating a table or a database for each user, it would allow for faster queries. All databases would be on the same server (I don't know who my host will be yet, only in design phase currently). There would be a main database with data that everyone would need (the base item while the user table/database would have a reference to the base item). I do see that it is possible for a field to be a foreign key from another database, so I know my idea in that aspect is possible, but overall I'm not sure what the best idea is.
I see most hosts say "unlimited number of databases" which is what got me to thinking about a database for each user. I could use this for that users posts on threads, their collection items, their preferences, and other information. Also, by having each user have a different table/database, if someone's table needed to be reindexed for whatever reason, it wouldn't affect the other users.
However, my biggest concern in either fashion would be additions/deletions to the structure of the tables/databases. I'm pretty sure a script could be written to make the necessary changes, but it seems like a pretty high risk. For instance, I'm pretty sure that I could write a script to add a field to a specific table in each database, or all of the like tables, but then to verify them it could prove difficult.
Any ideas you can throw out there for me would be greatly appreciated. I've been trying to work on this site for over a year now and keep getting stuck on the database design because of my worry of too large of tables, slow response time, and if the number of users grow, breaking some constraints set by phpmyadmin/MySQL. I also don't want to get half way through the database building and then think that there's a better way to do it. I know there may be multiple ways to do it, but what is the most common practice for it? Thank you all very much.
I was thinking that by either creating a table or a database for each user, it would allow for faster queries.
That's false. A single data base will be faster.
1,000,000 cards per user isn't really a very large number unless you have 1,000,000 users.
Multiple databases is an administration nightmare. A single database is always preferred.
my worry of too large of tables, slow response time, and if the number of users grow, breaking some constraints set by phpmyadmin/MySQL
You'll be hard-pressed to exceed MySQL limits.
Slow response is part of your application and details of your SQL queries more than anything else.
Finally. And Most Important.
All technology goes out of date. Eventually, you must replace something. In order to get to the point where you're forced to upgrade, you must first get something running.
Don't worry about "large database" until you have numbers of rows in the billions.
Don't worry about "long-term" solutions because all software technology expires. Quickly.
Regarding number of users.
Much of web interaction is time spent interacting with the browser through JavaScript. Or reading a page. Clicks are actually sort of rare. MySQL on a reasonably large server should handle 30 or more nearly concurrent queries with sub-second response. Your application will probably take very little time to format and start sending an HTML page. Things can rip along at a very, very good clip on a typical server.
If your database design avoids the dreaded full-table scan.
You must have proper indexes for the most common queries.
Now. What are the odds of 30 nearly concurrent requests? If a user only clicks once every 10 seconds (they have to read the page, fill in the form, re-read the page, think, drink their beer) then the odds of 30 clicks in a single second means you have to have 300 concurrent users. Considering that people have other things to do in their lives, that means you must have 50,000 or so users (figuring they're spending 1 hour each week on your site.)
I wouldn't go down the path of creating a database for every user... that will create countless headaches for you: data integrity issues, referential integrity issues, administrative issues...
As long as your table is well normalized and indexed, I don't think a table with hundreds of millions of rows is prohibitively large.
Instead, I would just start with a simple table design. If your site is wildly successful, it wouldn't be any extra effort to implement partitioning or sharding in MySql down the road as opposed to scaling out right off the bat.
If I where in your shoes I would start with one database and one table and not worry too much about the possible size of the table. If you ever get so successful and reach the size you imagine you would probably have a lot more resources and knowledge of your domain to make a better informed decision. Once that happens, you can also consider noSql solution such as HBase, Mondgodb and others that allow for horizontal scaling(unlimited size) with some limitations that businesses that deal with big data are bound to face. You can also use mysql partitions or other sharding solutions. So, go build your product with one table and don't sweat this problem until you absolutely need to. Good luck!
I am developing a layered web app. In brief it has:
UI: html, javascript and jquery
Domain logic: Java and servlets
Business logic: MySQL
I have large amounts of records in the database containing info about books. In addition the application will be used by a lot of users at the same time.
I want to enable users to input a book's "name" in a search text field
, say "book1" and display a drop down list using jquery autocomplete.
The records in database are not updatable since they will never change.
Considering solid design patterns,which is better (performance and speed wise) :
Preloading these database records into a cache object at the domain logic and let the users search (query) them from this object? Or querying directly from
the database using something like MySQL full-text search?
If using MySQL full-text search, I am concerned about having lots of calls to the database by many users at the same time.
As for preloading into a cache object, i am not sure if this is generally a good software practice, does anyone recommend it? Should
i put a timer for records to remain cached in memory?
Which of these 2 methods is preferable? are there other better methods for such scenarios?
I found a solution and hope this answer will help the ones who are dealing with a similar situation:
I will be using a software engineering design pattern, which I used frequently in the past, called Identity Map.
Only this time, since records are not updatable (i.e. not changeable), I will be using only the caching functionality of the identity map. So I will will be loading the records from the database into this identity map object once the server starts. This way a user will query them directly from the domain layer, thus faster and less calls to the database.
One issue to consider, and it is when adding new records by the administration, for this situation I will be using another design pattern called "observer pattern" (you can learn about it in this book).
UPDATE:
In case you are dealing with a similar situation, a good idea is to use MySQL indexing. I used it on the column of the "book's name" to enable faster loading in the cache object, because in my case the cache object will contain only book(s) name(s) since using the search field in the UI, the user is only concerned about a book's name. PS: other book details will be loaded only when user clicks on a book's name in the drop down list, and at this point u might want to use another Identity map holding the details . The reason behind this architecture is that, logically speaking never will u have all the books (with their details) from your database searched (loaded) by all the users of your application at the same time... thus to minimize server memory and bandwidth usage you first compromise loading the whole column of book names in memory for faster searching of all available book names BUT their details (ie more space in memory) will be loaded only when needed by a user and will be kept in another identity map to be used by another user searching for the same book with same details. In my opinion this minimizes memory usage on the server and less calls to the database to get details that have already been fetched by other users before.
In Mongo my understanding is that you can have databases and collections. I'm working on a social-type app that will have blogs and comments (among other things) and had previously be using MySQL and pretty heavy partitioning in an attempt to limit possible concurrency issues.
With MySQL I've stuffed all my user data into a _user database with several tables to further partition the data (blogs, pages, etc).
My immediate reaction with Mongo would be to create a 'users' database with one collection per user. In this way user 'zach' blog entries would go into the 'zach' collection with associated comments and such becoming sub-objects in the same collection. Basically like dynamically creating one table per user in MySQL, but apparently without the complexity and limitations that might impose.
Of course since I haven't really used Mongo before I'm having trouble gauging the (ahem..) quality of this idea and the potential problems it might cause down the road.
I'd like user data to be treated a lot like a users directory in a *nix environment where user created/non-shared (mostly) gets put into one place (currently with MySQL that would be the appname_users as mentioned above).
Most of the users data will be specific to the users page(s). Some of the user data which is queried across all site users (searchable user profiles) is currently kept in a separate database/table and I expect things like this could be put into a appname_system database and be broken up into collections and/or application specific databases (appname_profiles).
Anyway, since the available documentation on this is currently a little thin and my experience is extremely limited I thought I might find a little guidance from someone with a better working understanding of the system.
On the plus side I'd really already been attempting to treat MySQL as a schema-less document-store and doing this with Mongo seems much more intuitive/sane/rational so I'm really looking forward to getting started.
Thanks,
Zach
I have the same kind of application.
Some things to consider: you can cross query between collection bu not between databases.
So It's probably better to have a database with all you data and then a collection for each Object.
Then each document can contain any kind and number of fields.
I tried to avoid embedding arrays b/c I had trouble query properly my object (it was working fine, but the architecture of my system was designed for this use)
And a database can be shared between several sever automatically so space is not an issue (if you have more than 1 server)