This is more of a philosophical question, than a technical question. I'm an Access noob and am running into a philosophical conundrum.
I've run some queries from my base tables. I have them pretty much how I want them, don't really foresee making additional changes.
So, my question is this: is there an advantage to keeping my data in queries? If they're going to be static queries, should I just make them into tables? Have I already made too many tables/is there such a thing?
I'm working with computer scans. The scans are looking at different things on the computer--these are 2 tables. I also have a master list of stuff that I put together. And then I have a list of printers.
Then, I have like 7 queries. They're things like looking for intersects between the different scans, comparing the results of scans to lists of printers, etc. etc.
So, yeah. Do I keep them as queries, does it not matter, or should I make them into tables if they're just going to be static?
welcome to stack overflow.
You are going to need to provide much more detail if you want specific help in these forums. Showing us details of what you've already accomplished helps a lot. Also showing more details on where you are trying to get to is helpful. You might want to read the site rules for how to post a well formed question.
As for databases... there are many ways to construct a database depending on the amount of data fields as well as amount of data in those fields.
If you have a relatively small database that doesn't change, you can break with conventions and clump data into rough tables. But it's very advisable to NOT do that. Because if you ever need to change the tables or add new data... it starts to become a nightmare fairly quickly.
Which brings up the question: what are well formed tables in a database and how are they connected?
The answer usually takes going to some database classes. But you can start by looking up what Third Normal Form is. This will give you an idea of how to break down data into tables that are manageable and easy to expand upon.
But Third Normal Form is not always the best way to store data. Sometimes for reporting purposes, it's better to have tables in second normal form or lower to give you more speed on retrieval. (Mind you, this is usually for databases with massive amounts of data.)
Anyway, it's worth looking up articles on database design or taking a class. The more you understand about how data is retrieved and stored, the better you will be at deciding what the best structure is for you.
If you post more details, I'm sure people from stack overflow will help give you more pointers.
Best of luck! :)
Related
I am developing a Discussion Forum for my University. For this to manipulate the data i m using CouchDB as database.
I m finding difficulty in designing the structure of my db, in order to maximize the performance of my db.
I want to discuss what is the good practice of designing a document database.
Either we should make only one database as SQL and make 'n' no. of documents in the database.
Or we can make more no of database in order to flatten my db structure.This also reduce the more no. of documents to be developed.
The questions you need to ask are simply this: "How do you want to get data out of your database?"
Database design hinges around the queries to be made, not what is available to be stored.
This is especially important for Document DBs like Couch, since, while it does have a flexible schema, it does not have flexible indexing. By that I mean that because of the granularity of the data, it's quite like that later on, when you need to ask a question that it was not designed to answer, answering that question may well be very expensive. It's much, much cheaper to design your views and other constructs early, when there is little data in the data base rather than later after you have thousands or millions of rows.
RDBMS's, since they tend to have a finer granularity of data, tend to be more nimble to new queries and such later in life. Document DBs, not so much.
So think through your use cases up front, and design around those, and design those early on, it's much less painless now than later.
It's hard to tell the right way to approach modeling your data since you don't give much information. Generally though you want to keep as much data as possible in one database as this allows you to index it together (indexes cannot span more than one database).
Also, since there is no schema enforcement in the database, you can create different types of records in each database. For example, there is nothing wrong with have both user information and forum entries in the same database.
Last, you will most likely want to keep messages and their replies in different records. This is an old but still relevant discussion on this topic: http://www.cmlenz.net/archives/2007/10/couchdb-joins
Cheers.
I'm building a part onto a WP site that is similar to a "bidding" board. Basically it will have items with price, time, expiration, etc. Not too many fields.
Before I begin, I'm curious about the pros and cons of building these bid "items" as custom post types, allowing them to be viewable through the WP backend - but then they are all in the wp_posts table, meaning they are mixed with everything else.
Is there a big speed hit on this? In other words, should I create another wp_bids table and store ONLY bids in there? The con to this would be that the end user wouldn't have a built in way to see this through the backend (I'd have to build that system) and would take me a long time... Can someone offer some insight on this? Thanks!
Yes, there is going to be a performance difference. Doing it through WordPress' tables and interface will most likely be slower. How much slower depends on many things, and how slow it's allowed to be is only something you can decide. You may never notice the difference.
The only ways to find out would be to do it both ways and compare, or make some representative examples and see how they perform.
I need professional programmers/DBAs to bounce my idea off of and to know if it would/could even work. Please read below and give me any information that may break this theory. Thanks.
Overview of Website Idea:
The website will be used by sports card collectors to chat, answer questions on forums, showcase their cards/box breaks, trade/sell to/with other users, and keep a collection of their cards.
Design Issue:
A user can have an unlimited number of cards. This could make for some very large tables.
Design Question:
I do not want to limit the users on how many cards they can have in their collection on the site. If they have 5 copies of one card, and would rather have 5 records, one for each card, then that is their prerogative. This may also be necessary as each of the cards may be in a different condition. However, by allowing this to happen, this means that having only one table to store all records for all users is not even close to an option. I know sports card collectors with over 1,000,000 cards.
I was thinking that by either creating a table or a database for each user, it would allow for faster queries. All databases would be on the same server (I don't know who my host will be yet, only in design phase currently). There would be a main database with data that everyone would need (the base item while the user table/database would have a reference to the base item). I do see that it is possible for a field to be a foreign key from another database, so I know my idea in that aspect is possible, but overall I'm not sure what the best idea is.
I see most hosts say "unlimited number of databases" which is what got me to thinking about a database for each user. I could use this for that users posts on threads, their collection items, their preferences, and other information. Also, by having each user have a different table/database, if someone's table needed to be reindexed for whatever reason, it wouldn't affect the other users.
However, my biggest concern in either fashion would be additions/deletions to the structure of the tables/databases. I'm pretty sure a script could be written to make the necessary changes, but it seems like a pretty high risk. For instance, I'm pretty sure that I could write a script to add a field to a specific table in each database, or all of the like tables, but then to verify them it could prove difficult.
Any ideas you can throw out there for me would be greatly appreciated. I've been trying to work on this site for over a year now and keep getting stuck on the database design because of my worry of too large of tables, slow response time, and if the number of users grow, breaking some constraints set by phpmyadmin/MySQL. I also don't want to get half way through the database building and then think that there's a better way to do it. I know there may be multiple ways to do it, but what is the most common practice for it? Thank you all very much.
I was thinking that by either creating a table or a database for each user, it would allow for faster queries.
That's false. A single data base will be faster.
1,000,000 cards per user isn't really a very large number unless you have 1,000,000 users.
Multiple databases is an administration nightmare. A single database is always preferred.
my worry of too large of tables, slow response time, and if the number of users grow, breaking some constraints set by phpmyadmin/MySQL
You'll be hard-pressed to exceed MySQL limits.
Slow response is part of your application and details of your SQL queries more than anything else.
Finally. And Most Important.
All technology goes out of date. Eventually, you must replace something. In order to get to the point where you're forced to upgrade, you must first get something running.
Don't worry about "large database" until you have numbers of rows in the billions.
Don't worry about "long-term" solutions because all software technology expires. Quickly.
Regarding number of users.
Much of web interaction is time spent interacting with the browser through JavaScript. Or reading a page. Clicks are actually sort of rare. MySQL on a reasonably large server should handle 30 or more nearly concurrent queries with sub-second response. Your application will probably take very little time to format and start sending an HTML page. Things can rip along at a very, very good clip on a typical server.
If your database design avoids the dreaded full-table scan.
You must have proper indexes for the most common queries.
Now. What are the odds of 30 nearly concurrent requests? If a user only clicks once every 10 seconds (they have to read the page, fill in the form, re-read the page, think, drink their beer) then the odds of 30 clicks in a single second means you have to have 300 concurrent users. Considering that people have other things to do in their lives, that means you must have 50,000 or so users (figuring they're spending 1 hour each week on your site.)
I wouldn't go down the path of creating a database for every user... that will create countless headaches for you: data integrity issues, referential integrity issues, administrative issues...
As long as your table is well normalized and indexed, I don't think a table with hundreds of millions of rows is prohibitively large.
Instead, I would just start with a simple table design. If your site is wildly successful, it wouldn't be any extra effort to implement partitioning or sharding in MySql down the road as opposed to scaling out right off the bat.
If I where in your shoes I would start with one database and one table and not worry too much about the possible size of the table. If you ever get so successful and reach the size you imagine you would probably have a lot more resources and knowledge of your domain to make a better informed decision. Once that happens, you can also consider noSql solution such as HBase, Mondgodb and others that allow for horizontal scaling(unlimited size) with some limitations that businesses that deal with big data are bound to face. You can also use mysql partitions or other sharding solutions. So, go build your product with one table and don't sweat this problem until you absolutely need to. Good luck!
I want to know what is better.
save all data (for example skype,icq,facebook,stackoverflow,etc,etc,etc) profiles in one column
save data in separated columns (separated column for skype, icq, facebook, stackoverflow, etc,etc)
What is better and easy for mysql server?
Saving them all in one column would violate first normal form and make querying and aggregating the data very difficult.
Use one column to store each piece of information.
If you are storing profile data for each service, you might even be better of with a separate table per service, with a foreign key to the user id.
Beyond discussions with respect to normalization which might seem abstract, asking the right questions can help make it clear why one approach might be superior to another.
Would you want to query for specific accounts? E.g., find all Skype accounts or find a specific Skype account. Of your two approaches, which makes that easier?
Which approach do you suppose would be easier to correct if there was a problem with the storage of one of the account types?
Which approach do you suppose would be easier to expand the types of accounts? E.g., today you build for Skype, ICQ, Facebook and StackOverflow. Tomorrow, you realize you want to store Twitter accounts. Which approach do you suppose would make that easier?
Which approach do you think would be easier for another developer to grasp, maintain and expand? E.g., your system becomes so popular that you hire developers to take over your work. The approach that is more inline with industry best practices will be easier for other developers to understand.
When you analyze the problem like this, it should be clear that storing the information into separate columns or perhaps a table with accounts where each user can have multiple rows that represents multiple accounts is the superior approach.
You will gain much more flexibility if you use multiple columns.
If you would like to query the database based on one of these fields, it would be very difficult if it is all packed together into one column. It also makes maintaining the database more difficult for yourself and whoever comes after you.
I've spent a few days researching the pros and cons of mysql against nosql solutions (specifically mongodb) for my project.
The project needs to be able to eventually scale to handle tens of thousands of simultaneous users - millions of users in total. The site is heavily user focussed and will interact with the database as much if not more than a site like facebook - it is very relational, all functionality is dependant on the relation to the user and their relationship with other users. It's also data heavy - lots of files, images, audio, messaging, personal news feed etc.
I like the look on mongodb a lot, I like the way it works, and I like how it scales - but can't get my head around how this would work for a site such as I describe. Would all interactions for a specific user have to be stored in a single document?
I am however very comfortable using mysql and like the relational aspect of it. I am just worried without a lot of work there will be scalability issues with this project - although perhaps with memcached and sharding this won't be an issue?
I'd like to know from those with experience with the two databases on large projects, out of mysql and mongodb which is the right tool for this particular job?
If the data is highly relational, use a relational database. If it's not, don't. NoSQL is great, don't get me wrong, but it's not suited to all tasks. It may be suited to your task, but the only way to find out is for you to build some tests for your specific usecase. Add a bunch of dummy data (millions if not hundreds of millions of rows). And then load test it.
As far as scaling, that's more of a component of how you build your application than the backend you choose. Do you have a solid schema? Do you have a strong cache layer with write-through caching? Do you access the backend as efficiently as possible (queries and such)? Can you shard based upon your application?
Those are the kind of questions which are appropriate here. Not "which will scale for me better". And not "which is the right tool". Both can do the job fine. Which is best is up to you...
Obviously, there's no silver bullet here. However, I would like to challenge this one assumption you've made:
... it is very relational, all functionality is dependant on the relation to the user and their relationship with other users...
OK, I'd like you to picture having 100M users in a relational database and start building this model. Let's try something simple, grab the names of a user's friends.
How do you get a user's friends? Well you go to the users_friends table. If each user has even just 10 friends, that table contains a billion rows. If users have a more reasonable 100 friends, you now have 10B rows.
So now you have a user and a list of their friends IDs. How do we get their friend's names? Well you go through the list of 100 IDs and pull down each of the friends. Perfect.
So now, if you want to show one user the names of all of their friends, all you have to do is join the 100M record table to the 10B record table. This is not a simple task. Scaling joins becomes exponentially harder and more expensive as the dataset grows.
So, to make this easier, you're probably going to run a for loop and manually collect the records for each friend. You have to do this because the friends are scattered across multiple servers, so each "lookup" has to be done individually.
Already you've broken your "relational model".
What about the friends list? Is keeping a table of 10B records really practical? Why not just keep a list of friend IDs with each user? Why do an extra query.
If you notice the pattern here, we've basically broken down the "very relational" model into something that's effectively key-value lookups. Of course, the key-value model will scale much better. And so, MongoDB seems like a good fit here.
Don't get me wrong, there are lots of good uses for relational databases. But when you're talking about handling millions of individual key-value style requests, you probably want to look at a NoSQL database.
There is no law that you have to build an application with exactly one database. It is often common practice having dedicated backends for particular tasks. E.g. in the context of a Facebook-like application it may make sense to work with a graph-database for storing relations between users - every database has its pros and cons and only would fools implement large backends with only a RDBMS or only a NoSQL db just because they don't know better.