I am about to start developing a classified web application. but I am facing certain difficulties
while building the DB design:
User can post their advertisements under different types of pre-existing catagories (like Vehicles, Real Estate, Computers, Education etc). But each category have their certain specific fields/properties as well as some common fields/properties.
My difficulties is like after filling a post Ad, say for Bike, I am saving the common i/p values to my advertisement_t table (as below).
but where will I store other fields values specific to Bike like: Bike Manufacture, Bike Model, Bike Year etc?
same way, if a user wanted to post an advertisement for House(rent), I can save the common i/p values to my advertisement_t table.
but where will I store other fields values specific to House For Rent like: total area(sq), No of Rooms, etc. and how will I map.
My Tables are below:
category_id
------------
(catId, parentCatId, cat_title, cat_desc)
advertisement_t
----------------
(adId, catId, userId, ad_title, ad_desc, photoId, postDate, statusId, price, ad_address, adValidFrom, adValidTo, adIsDeleted)
Photo_t
--------
(photoId, primary_photo, sec_photo1, sec_photo_2, sec_photo_3)
user_t
-------
(userId, username, password, email, activationkey, isvalidated, joinedDate, activatedDate, statusId)
I dont have any idea how other classified sites(like olx, freeadds) used to maintain(persist & display) their posted advertisement data on their database.
If any one has any idea, please help.
Thanks in advance.
I usually like classify these data into 2 groups (1) searchable data which will be saved in some column/mapped to column and (2) junk data( i call that additional_info) saved in a text column in json_encoded format.
For searchable columns, you can add another table category_values table, which has
(category_id, category_value, ad_id)
Again, if your DB is expected to grow huge with category type repeating often, you can break the category_values table to two as
(value_id, Category_id, category_value) and (value_id, ad_id). This results in more joins and need not be introduced in you case I believe.
I did something close to this Once.
And I can see 2 options:
Something you shouldn't do - Implement lots of tables for the required Goods (Houses, Cars, Bikes,..) which is not a very good idea because work would never end and it wouldn't be extensible.
My suggestion:
Add 3 more tables:
CustomField (To set a list of possible fields used by categories)
(CFid, Title)
CategoryCField (Associative relation to know what fields to display when user enters a new ad)
(catId, CFid)
StoredCFData (Actually store the Data)
(id, adId, CFid, value)
You can find however 2 problems doing it this way. Making a "relation circle" which is not very good, but if carefully done will do the work. And having to reduce the DataType of costum fields to text/numbers to make it simple. You'll have troubles applying it to Radio buttons or Combo Boxes.
Best Luck,
Raul
Related
I'm creating a real-estate website and i was wondering if there was a better way of organizing my columns or tables, not sure what would be the best way to go about it, i currently have a lot of columns and im worried about performance issues.
The columns are as follows
5 for things like property id, add date, duration, owner/user id.
35 columns for things like title, description, price, energy rating, location, etc.
40 columns for features like swimming-pool, central heating, river front, garage, well, etc.
15 for image locations which are stored on server
15 for the image descriptions
Is 110+ columns bad practice in MySQL? Everything is lightning fast but i'm in localhost at the mo, wont the monstrous size of the tables slow queries? Especially if I have a couple hundred properties?
Am i ok with my current setup? What would best practice be? How do e-commerce websites that have many feature options go about this?
It is not a good practice since the data can be stored in separate tables. What would help you most would be to create an ERD to visualize how you can organize your tables. Even if you do not understand the ins and outs of ERDs, you can still use it to at least organize your thoughts.
It seems that you already have your tables separated based on the bullet points that you made within your question. One thing that I would add to your bullets is maybe breaking down your features into categories and creating a table for each.
For example, swimming pools and riverfront can be placed in a table called
LandscapeFeatures or OutdoorFeatures.
Most likely, the property features would be better stored in a separate table, with one row per property feature, rather than as columns in your main table. I understand this as a many-to-many relationship between a propery and its features, so this suggest two more tables:
properties (property_id (pk), date_added, title, description)
features (feature_id (pk), description)
property_features (property_id (fk), feature_id (fk))
Such structure is much more flexible and easier to query than having one column per feature. As examples:
easy to add features by creating new rows in the features table (while in the old structure you had to create a new column)
easy to aggregate the features, and answer a question like: count how many features each property has
As for images, they should have their ow table too. If an image maby belong to several user, then it's a many-to-many relationship, and you can follow the above pattern. If each image belongs to a single user, one more table is enough:
properties (property_id (pk), date_added, title, description)
features (feature_id (pk), description)
property_features (property_id (fk), feature_id (fk))
images (image_id, location, description, property_id (fk))
One table:
Columns for the dozen or so values that you are most likely to search on.
Devise several composite indexes that involve those columns, starting with the more commonly searched columns.
Devise a TEXT column and put "words" in it for a FULLTEXT index. If this is home sales, consider words like "swimming pool septic tank gazebo Eichler". This will help with certain "boolean" type queries. (If you like this idea, let's discuss how to make use of filtering with indexes and/or fulltext; it gets tricky.)
Put the rest into a JSON (or TEXT column). Do not plan on searching it; instead bring the row(s) into your app code for further filtering after searching by the actual INDEXes
So I have this application that I'm drawing up and I start to think about my users. Well, My initial thought was to create a table for each group type. I've been thinking this over though and I'm not sure that this is the best way.
Example:
// Users
Users [id, name, email, age, etc]
// User Groups
Player [id, years playing, etc]
Ref [id, certified, etc]
Manufacturer Rep [id, years employed, etc]
So everyone would be making an account, but each user would have a different group. They can also be in multiple different groups. Each group has it's own list of different columns. So what is the best way to do this? Lets say I have 5 groups. Do I need 8 tables + a relational table connecting each one to the user table?
I just want to be sure that this is the best way to organize it before I build it.
Edit:
A player would have columns regarding the gear that they use to play, the teams they've played with, events they've gone to.
A ref would have info regarding the certifications they have and the events they've reffed.
Manufacturer reps would have info regarding their position within the company they rep.
A parent would have information regarding how long they've been involved with the sport, perhaps relations with the users they are parent of.
Just as an example.
Edit 2:
**Player Table
id
user id
started date
stopped date
rank
**Ref Table
id
user id
started date
stopped date
is certified
certified by
verified
**Photographer / Videographer / News Reporter Table
id
user id
started date
stopped date
worked under name
website / channel link
about
verified
**Tournament / Big Game Rep Table
id
user id
started date
stopped date
position
tourney id
verified
**Store / Field / Manufacturer Rep Table
id
user id
started date
stopped date
position
store / field / man. id
verified
This is what I planned out so far. I'm still new to this so I could be doing it completely wrong. And it's only five groups. It was more until I condensed it some.
Although I find it weird having so many entities which are different from each other, but I will ignore this and get to the question.
It depends on the group criteria you need, in the case you described where each group has its own columns and information I guess your design is a good one, especially if you need the information in a readable form in the database. If you need all groups in a single table you will have to save the group relevant information in a kind of object, either a blob, XML string or any other form, but then you will lose the ability to filter on these criteria using the database.
In a relational Database I would do it using the design you described.
The design of your tables greatly depends on the requirements of your software.
E.g. your description of users led me in a wrong direction, I was at first thinking about a "normal" user of a software. Basically name, login-information and stuff like that. This I would never split over different tables as it really makes tasks like login, session handling, ... really complicated.
Another point which surprised me, was that you want to store the equipment in columns of those user's tables. Usually the relationship between a person and his equipment is not 1 to 1 and in most cases the amount of different equipment varies. Thus you usually have a relationship between users and their equipment (1:n). Thus you would design an equipment table and there refer to the owner's user id.
But after you have an idea of which data you have in your application and which relationships exist between your data, the design of the tables and so on is rather straitforward.
The good news is, that your data model and database design will develop over time. Try to start with a basic model, covering the majority of your use cases. Then slowly add more use cases / aspects.
As long as you are in the stage of planning and early implementation phasis, it is rather easy to change your database design.
I'm trying to figure out the best practices for storing user data on a php/mysql site.
let's say the website will host a service of saving people's input for items they have in their house.
I have set up tables that includes: kitchen, bathroom, bedroom, etc.
Sally adds her 6 kitchen items.
John adds his 3 kitchen items.
etc.
I'm just wondering what may be the common practice on storing other user information in the mysql database. I've taken a class on databases, so i'm thinking relationally linking by foreign key, john with his items in the lists, and sally too..
does that sound about right? or is there a better way? I can see the list getting really large quite quickly.
would it be possible to set up a different table to each user? is that possible? or would it be silly?
I would not set up a table for each user.
Definitely go relational. I am not sure I follow you completely around "john with his items.." and so on. So I interpret this as
user table
room table
item table
relational user->item (id, user_id, item_id, room_id) OR:
relational item->room
So you can pull a user, list the rooms they have related to them, then list the items in that room. Additionally, like this you do not need a new item entry for common things like tables, stoves, spatulas, etc.
Your list could get large, but if you scale properly and plan a back end based update migration when you absolutely need to (like millions of users) then you should be fine. Consider how many relations sites like facebook and ebay have to maintain. Large relations are normal for databases so I wouldn't let a couple million rows scare you.
I would use three tables:
rooms (id, room), to store values kitchen, bathroom, bedroom, etc.
users
items: assuming you have a common structure your your current kitchen, bathroom, bedroom tables, one table could replace all of them. This table should also contain two foreign keys, user_id and room_id.
With that structure, you can easily retrieve and filter your data.
In Meetup.com, when you join a meetup group, you are usually required to complete a profile for that particular group. For example, if you join a movie meetup group, you may need to list the genres of movies you enjoy, etc.
I'm building a similar application, wherein users can join various groups and complete different profile details for each group. Assume the 2 possibilities:
Users can create their own groups and define what details to ask users that join that group (so, something a bit dynamic -- perhaps suggesting that at least an EAV design is required)
The developer decides now which groups to create and specify what details to ask users who join that group (meaning that the profile details will be predefined and "hard coded" into the system)
What's the best way to model such data?
More elaborate example:
The "Movie Goers" group request their members to specify the following:
Name
Birthdate (to be used to compute member's age)
Gender (must select from "male" or "female")
Favorite Genres (must select 1 or more from a list of specified genres)
The "Extreme Sports" group request their member to specify the following:
Name
Description of Activities Enjoyed (narrative form)
Postal Code
The bottom line is that each group may require different details from members joining their group. Ideally, I would like anyone to create a group (ala MeetUp.com). However, I also need the ability to query for members fairly well (e.g. find all women movie goers between the ages of 25 and 30).
For something like this....you'd want maximum normalization, so you wouldn't have duplicate data anywhere. Because your user-defined tables could possibly contain the same type of record, I think that you might have to go above 3NF for this.
My suggestion would be this - explode your tables so that you have something close to 6NF with EAV, so that each question that users must answer will have its own table. Then, your user-created tables will all reference one of your question tables. This avoids the duplication of data issue. (For instance, you don't want an entry in the "MovieGoers" group with the name "John Brown" and one in the "Extreme Sports" group with the name "Johnny B." for the same user; you also don't want his "what is your favorite color" answer to be "Blue" in one group and "Red" in another. Any data that can span across groups, like common questions, would be normalized in this form.)
The main drawback to this is that you'd end up with a lot of tables, and you'd probably want to create views for your statistical queries. However, in terms of pure data integrity, this would work well.
Note that you could probably get away with only factoring out the common fields, if you really wanted to. Examples of common fields would include Name, Location, Gender, and others; you could also do the same for common questions, like "what is your favorite color" or "do you have pets" or something to that extent. Group-specific questions that don't span across groups could be stored in a separate table for that group, un-exploded. I wouldn't advise this because it wouldn't be as flexible as the pure 6NF option and you run the risk of duplication (how do you predetermine which questions won't be common questions?) but if you really wanted to, you could do this.
There's a good question about 6NF here: Would like to Understand 6NF with an Example
I hope that made some sense and I hope it helps. If you have any questions, leave a comment.
Really, this is exactly a problem for which SQL is not a right solution. Forget normalization. This is exactly the job for NoSQL document stores. Every user as a document, having some essential fields like id, name, pwd etc. And every group adds possibility to add some fields. Unique fields can have names group-id-prefixed, shared fields (that grasp some more general concept) can have that field name free.
Except users (and groups) then you will have field descriptions with name, type, possible values, ... which is also very good for a document store.
If you use key-value document store from the beginning, you gain this freeform possibility of structuring your data plus querying them (though not by SQL, but by the means this or that NoSQL database provides).
First i'd like to note that the following structure is just a basis to your DB and you will need to expand/reduce it.
There are the following entities in DB:
user (just user)
group (any group)
template (list of requirement united into template to simplify assignment)
requirement (single requirement. For example: date of birth, gender, favorite sport)
"Modeling":
**User**
user_id
user_name
**Group**
name
group_id
user_group
user_id (FK)
group_id (FK)
**requirement**:
requirement_id
requirement_name
requirement_type (FK) (means the type: combo, free string, date) - should refers to dictionary)
**template**
template_id
template_name
**template_requirement**
r_id (FK)
t_id (FK)
The next step is to model appropriate schema for storing restrictions, i.e. validating rule for any requirement in any template. We have to separate it because for different groups the same restrictions can be different (for example: "age"). You can use the following table:
**restrictions**
group_id
template_id
requirement_id (should be here as template_id because the same requirement can exists in different templates and any group can consists of many templates)
restriction_type (FK) (points to another dict: value, length, regexp, at_least_one_value_choosed and so on)
So, as i said it is the basis. You can feel free to simplify this schema (wipe out tables, multiple templates for group). Or you can make it more general adding opportunity to create and publish temaplate, requirements and so on.
Hope you find this idea useful
You could save such data as JSON or XML (Structure, Data)
User Table
Userid
Username
Password
Groups -> JSON Array of all Groups
GroupStructure Table
Groupid
Groupname
Groupstructure -> JSON Structure (with specified Fields)
GroupData Table
Userid
Groupid
Groupdata -> JSON Data
I think this covers most of your constraints:
users
user_id, user_name, password, birth_date, gender
1, Robert Jones, *****, 2011-11-11, M
group
group_id, group_name
1, Movie Goers
2, Extreme Sports
group_membership
user_id, group_id
1, 1
1, 2
group_data
group_data_id, group_id, group_data_name
1, 1, Favorite Genres
2, 2, Favorite Activities
group_data_value
id, group_data_id, group_data_value
1,1,Comedy
2,1,Sci-Fi
3,1,Documentaries
4,2,Extreme Cage Fighting
5,2,Naked Extreme Bike Riding
user_group_data
user_id, group_id, group_data_id, group_data_value_id
1,1,1,1
1,1,1,2
1,2,2,4
1,2,2,5
I've had similar issues to this. I'm not sure if this would be the best recommendation for your specific situation but consider this.
Provide a means of storing data as XML, or JSON, or some other format that delimits the data, but basically stores it in field that has no specific format.
Provide a way to store the definition of that data
Provide a lookup/index table for the data.
This is a combination of techniques indicated already.
Essentially, you would create some interface to your clients to create a "form" for what they want saved. This form would indicated what pieces of information they want from the user. It would also indicate what pieces of information you want to search on.
Save this information to the definition table.
The definition table is then used to describe the user interface for entering data.
Once user data is entered, save the data (as xml or whatever) to one table with a unique id. At the same time, another table will be populated as an index with
id where the xml data was saved
name of field data is stored in
value of field data stored.
id of data definition.
now when a search commences, there should be no issue in searching for the information in the index table by name, value and definition id and getting back the id of the xml/json (or whatever) data you stored in the table that the data form was stored.
That data should be transformable once it is retrieved.
I was seriously sketchy on the details here, I hope this is enough of an answer to get you started. If you would like any explanation or additional details, let me know and I'll be happy to help.
if you're not stuck to mysql, i suggest you to use postgresql which provides build-in array datatypes.
you can define a define an array of varchar field to store group specific fields, in your groups table. to store values you can do the same in the membership table.
comparing to string parsing based xml types, this array approach will be really fast.
if you dont like array approach you can check out xml datatypes and an optional hstore datatype which is a key-value store.
I am developing a database to store test data. Each piece of data has 11 tags of metadata. Currently I have a separate table for each of the metadata options. I have seen a few questions on here regarding best practices for numerous small tables, but I thought I'd pose the question for my own project because I didn't get a clear answer from the other questions asked.
Here is my table list, with the fields in each table:
Source Type - id, name, description
For Flight - id, name, description
Site - id, name, abrv, description
Stand - id, site (FK site table), name, abrv, descrition
Sensor Type - id, name, channels, descrition
Vehicle - id, name, abrv, descrition
Zone - id, vehicle (FK vehicle table), name, abrv, description
Event Type - id, name, description
Event - id, event type (FK to event type Table), name, descrition
Analysis - id, name, descrition
Bandwidth - id, name, descrition
You can see the fields are more or less the same in each of these tables. There are three tables that reference another table.
Would it be better to have just one large table called something like Meta with the following fields:
Meta: id, metavalue, name, abrv, FK, value, descrition
where metavalue = one of the above table names
and FK = a reference to another row in the Meta table in place of a foreign key?
I am new to databases and multiple tables seems most intuitive, but one table makes the programming easier.
So questions are:
Is it good practice to reduce the number of tables and put all static values in one table.
Is it bad to have a self referencing table.
FYI I am making this web database using django and mysql on a windows server with NTFS formatting.
Tips and best practices appreciate.
thanks.
"Would it be better to have just one large table" - emphatically and categorically, NO!
This anti-pattern is sometimes referred to as 'The one table to rule them all"!
Ten Common Database Design Mistakes: One table to hold all domain values.
Using the data in a query is much easier
Data can be validated using foreign key constraints very naturally,
something not feasible for the other
solution unless you implement ranges
of keys for every table – a terrible
mess to maintain.
If it turns out that you need to keep more information about a
ShipViaCarrier than just the code,
'UPS', and description, 'United Parcel
Service', then it is as simple as
adding a column or two. You could even
expand the table to be a full blown
representation of the businesses that
are carriers for the item.
All of the smaller domain tables will fit on a single page of disk.
This ensures a single read (and likely
a single page in cache). If the other
case, you might have your domain table
spread across many pages, unless you
cluster on the referring table name,
which then could cause it to be more
costly to use a non-clustered index if
you have many values.
You can still have one editor for all rows, as most domain tables will
likely have the same base
structure/usage. And while you would
lose the ability to query all domain
values in one query easily, why would
you want to? (A union query could
easily be created of the tables easily
if needed, but this would seem an
unlikely need.)
Most of these look like they won't do anything but expand codes into descriptions. Do you even need the tables? Just define a bunch of constants, or codes, and then have a dictionary of long descriptions for the codes.
The field in the referring table just stores the code. eg: "SRC_FOO", "EVT_BANG" etc.
This is also often known as the One True Lookup Table (OTLT) - see my old blog entry OTLT and EAV: the two big design mistakes all beginners make.