I want to add dynamic columns in a mysql table but I don't know exactly how.
I want to let the user add some columns (fields) in a thread, eg. let him add a integer field and a value (eg. price: 199) or a string field and a value (eg. name: teddybear).
The user can add as many field/value-pairs as he wants.
I thought I could create a many-to-many table:
thread <-> thread_field <-> field
thread: id, title
thread_field: field_id, thread_id, value
field: id, name
is this a good structure?
But in this way I have to set a specific column type of thread_field.value. either it's an integer or a string. I want to have the possibility to have it dynamic, let the user choose.
How can I do this?
Thanks!
The ugly way:
thread_field: field_id, thread_id, value_text, value_int
where value_text is declared TEXT and value_int is declared INT.
For any given entry, you use only one of the two fields.
This problem comes up very frequently with regards to relational databases. It's part of the definition of a relation that it has a fixed set of attributes, not dynamic attributes.
If you need to allow user-defined attributes in a relational database, the simplest solution is to store a Serialized BLOB such as XML or other semi-structured format (JSON, YAML). You lose the ability to query this efficiently using SQL (unless you use an RDBMS that extends SQL with XML functions and indexes), but you're going to sacrifice many features of the RDBMS no matter how you solve this problem.
Another alternative is to use one of the new non-relational data stores like CouchDB or MongoDB, which makes it easy to extend entities with dynamic, strucutred attributes while remaining somewhat efficient.
You can add the type of data to the 'fields' table and store either a string or a binary object and then convert it to the proper type in your code, you would have to do all the validation in code though.
You could also add other properties to the field such as Optional vs Required value, visibility, etc.
If you intend your user to enter arbitrary key=>value pairs, you might want to look at a vertical table. Something like:
post_id, keyname, keyvalue
1, 'name', 'teddybear'
1, 'price', '1.99'
2, 'colour', 'fuchsia'
78, 'mother', 'Diana'
78, 'father', 'Bob'
78, 'pet', 'Fido'
Each of these key/values is linked the the record (post_id) they were created in. This also works reasonably well when you have a lot of options for the user, but very few will ever be chosen. The cons of this solution include a) you can no longer take advantage of automatic typing and b) index is not as useful since the value types are mixed.
If you have a system where the options are well defined, you should design your tables to fit those options. The pros of automatic typing and indexing usually outweigh the flexibility of changing your apparent data structure on the fly.
Finally it's here MySQL select where JSON field property has value .
Save your data in json column and let mysql do the magic
You shouldn't do it.
Site users has absolutely nothing to do with database structure.
You have to learn how to design database structure properly.
There is not a single site around who uses user-defined dynamic columns in the database.
And I am sure you can manage without it somehow.
Related
I wanted to use a relational database(MySql) to store my data as key-value pair.
I would be getting no. of key-value pairs dynamically.
I can create a simple table to store them in separate columns.
Values can be of type- int, varchar, text or date.
The problem which I am facing is:
When I need to run a query on key whose value should be an integer and I need to use and greater than or less than query with it. Same case when I need to use between query with date fields.
How can I achieve it?
------------------------------------------------Edit---------------------------------------------------
For greater clarity, I am providing the background for this question which I have divided into three parts:
1. Data 2: Use Case 3. Possible Designs
1. Data
Suppose I'm creating data store for census of a country**(Just an example)**. Fields for storing data would be different for male, female, boy or girl and also it will vary according to the person's profession. The number of fields depends on the requirement which can increase up to 500 or more.
2. Use Case
Show a paginated list of persons whose monthly income is between $7000 to $10000. User can click on any page number and the database should directly fetch the data for that page number. For example, if we are showing 10 results in a page and user clicks on the 5th page then we should show him the list of the person's from 40 to 50.
Some of the values belonging to a particular group store description which can have large data. So they should be stored as TEXT.
3. Possible Designs
I can create a separate table for each different type and store their data in respective fields. But the problem I'm thinking about this approach is that MySQL table has a maximum row size limit of 65,535 bytes. Going by this approach and storing all data horizontally might cross the max size limit. As the number of fields are not fixed and can change as per requirement.
Instead of storing data horizontally I can store them vertically using Entity Attribute Value design(key-value pair). For now, the increase in the number of rows due to this design is not a problem. Using this I can store data of all male, female or child in the same table. But the problem with this approach is:
I will lose the Datatype of certain important fields. I can not query and get the list of persons whose income is more than 1000.
For storing data or all fields in single Value type, I need to make it varchar. But some fields store large data which requires TEXT as the type.
Considering the above problem, I thought that instead of creating only one value field, I will create multiple value fields like value_int, value_varchar, value_date or value_text.
DB structure
For this problem, I will be using MySQL and cannot change the DB due to certain restrictions. So I am looking for a design with MySQL only.
Going by key-value approach is a good idea or not? Or any other possible design which can be used?
In very general terms, if you know the entities and attributes of your problem domain, and the data is relational, I'd use a relational schema (your "possible design 1"). If you actually encounter problems with maximum row width, your problem domain might contain logical subgroupings of attributes, so you can split them into separate table.
For instance:
Person (id, name, ...)
Person_demographics (person_id, age, location, ...)
Person_finance (person_id, income, wealth...)
If you don't know the entities and attributes in advance, I recommend using MySQL's JSON support. or XML support. This gives you access to much better query options than EAV.
The problem with EAV-like solutions in your scenario is that any non-trivial queries end up being incredibly complicated - "find all responses where salary is between x and y, and the age is z, in locations (a, b, c)" turns into a horrible mess of SQL, but with XPath this is pretty straightforward.
I'm trying to do it like this:
Every single user can choose fields (like structures on MySQL) where this fields can handle their respective value, it's like doing a DB inside a DB.
But how can I do it using a single table?
(not talking about user accounts etc where I should be able to use a pointer to his own "structure")
Do something like: varchar Key where register something like "Name:asd" where PHP explode : to get the respective structure ('name' in this case) and the respective value? ('asd')
Use BLOB? can someone turn the light on for me? I don't know how to do something where works better than my current explanation...
I know my text is confuse and sorry for any bad english.
EDIT:
Also, they could add multiple keys/"structures" where accepts a new value
And they are not able to see the Database or Tables, they still normal users
My server does not support Postogre
In my opinion you should create two tables.
with the user info
with 3 fields (userid, key and value)
Each user has 1 record in the first table. Each user can have 0 or more records in the second table. This will ensure you can still search the data and that users can easily add more key/value pairs when needed.
Don't start building a database in a database. In this case, since the user makes the field by himself there is no relation between the fields as I understand? In that case it would make sense to take a look at the NoSQL databases since they seem to fit very good for this kind of situations.
Another thing to check is something like:
http://www.postgresql.org/docs/8.4/static/hstore.html
Do not try to build tables like: records, fields, field types etc. That's a bad practice and should not be needed.
For a more specific answer on your wishes we need a bit more info about the data the user is storing.
While i think the rational answer to this question is the one given by PeeHaa, if you really want the data to fit into one table you could try saving a serialized PHP array in one of the fields. Check out serialize and unserialize
Generates a storable representation of a value
This is useful for storing or passing PHP values around without losing
their type and structure.
This method is discouraged as it is not at all scalable.
Use a table with key-value pairs. So three columns:
user id
key ("name")
value ("asd")
Add an index on user id, so that you can query a user's attributes easily. If you wanted to query all users with the same properties, then you could add a second index on key and/or value.
Hope you are using a programming language also to get the data and present them.
You can have a single table which has a varchar field. Then you store the serialized data of the field structure and their value in that field. When you want to get the structure, query the data and De-serialize that varchar field data.
As per my knowledge every programming language supports serialization and De-serialization.
Edited : This is not a scalable option.
I am currently building a small crm application. I need each user to be able to define their own custom fields. I am currently building this crm using php and mysql.
Example: I have a "customer" table which has the standard fields: name, phone, address, email, etc. But i want to allow the user (unique session) to add fields that are custom to his/her business which are only accessible to him (not other users). I then want these custom fields to function just like all the other fields in the table (ability to search, send and received data). I am hoping i can accomplish this in mysql and php but am open to any technology or solution that is considered best practice. Thank you for your help.
This can be done by creating a table called "customfields" with the elements "id, fieldname, company_id", then another table that would associate those custom fields with data, eg "customercustomdata: id, customfields_id, customer_id". Associate "ownership" of a field the same way
To create a new custom field, "insert into customfields (fieldname,company_id) values ('Birthday',companyid);"
Does that help?
#Matt H: Is this method considered AEV or just standard relational db?
So because i will have many users in many dif industries that will want to add their own custom fields to a number of different tables (contacts, transactions, events, etc) i am assuming that i would need the customfield table to have a user_fk/id or company fk/id, a related table fk/id, an id, and a field name? Am i on the right track? Then in the need to create a 2nd table to hold the data for each custom field buy having a customfield fk/id, customer fk/id, id and a data field to hold the actual data. Is this correct?
Ok so once i build those two additional tables how do I add them to the contacts table so it looks like one big table for the user, instead of the 3 tables?
Thanks again for you help.
Answer
after much research i have found that most people who wish to accomplish this are using document databases not relational databases.
You could place an extra column for storing string data and store an array describing the contents for custom cells. For example:
$custom = array(
array("field" => "bikesOwned", "value" => 4),
array("field" => "travelled", "value" => 14)
);
then use something like PHPs json_encode to store that data in the extra cell. Then all you would need to do is decode and process the array.
Some people suggesting using the Entity-Attribute-Value design, but before you do, please read Bad CaRMa, a story about an EAV-like design that nearly destroyed a company because it was unmaintainable.
To solve this better, read How FriendFeed uses MySQL to store schema-less data. You can lump all the custom columns into a single BLOB, and store it that way. Then if you want individual attributes to be searchable, create a table for that attribute, that maps values back to the customers table.
I wonder if anyone could offer their advice on this one.
I have some customer data whereby certain fields remain consistent. For example:
'Firstname', 'Lastname', 'Postcode'
These are the important fields and the ones that would be searched against. Over time we may have some additional data for a customer, however it is guaranteed that the fields will differ, therefore we can't indefinitely create more fields in the table to accommodate every possible new field.
I wondered what the options would be of storing the auxiliary date fields, for example would creating one additional field 'AuxData' maybe as a VARCHAR that used a JSON array?
(I am thinking of what I have seen in wordpress in the past)
I'd really appreciate anyone's thoughts on this one!
Many thanks
JAson
You can create an extra column with XML data, usually I would recommend JSON over XML, but MySQL has some special functionality to search trough XML data in fields. See this article for a few examples: http://www.informit.com/articles/article.aspx?p=1019623
There is also another possibility, you could create a new table with 3 columns: [Person_ID, Property, Value] where the primary key would span (Person_ID, Property). This way you can more easily search trough data, and you keep it MySQL instead of XML/JSON. However both options are valid.
So on my older work, I had always used the 'text' data type to store items, like so:
0=4151:54;1=995:5000;2=521:1;
So basically: slot=item:amount;
I've been looking into finding the best ways of storing information in a sql database, and everywhere i go, it says that using text is a big performance hit.
I was thinking of doing something else, like having a table with the following columns:
id, owner_id, slot_id, item_id, amount
Where as now i can just insert a row for each item a character allocates. But i have no clue how to save them, since the slot's item can change, etc. A character has 28 inventory slots, and 500 bank slots, should i insert them all at registration? or is there a smarter way to save the items
Yes use that structure. Using text to store relational data defeats the purpose of a relational database.
I don't see what you mean by insert them all at registration. Can you not insert them as you need to?
Edit
Based on your previous comment I would recommend only inserting a slot as it is needed (if I understand your problem). It may be an idea to keep the ID of the slot in the application, if need be.
If I understand you correctly, and that the slot's item can change, then you want to further abstract the mapping between item_id and the item:
entry_tbl.item_id->item_rel_realitems_tbl.real_id->items_tbl
This way, all entries with an itemid point to a table that maps those ids to a mutable item. When you UPDATE an item in 'items_tbl' then the mapping automatically updates the entry_tbl.
Another JOIN is needed however. I would also use stored procedures in any case to abstract the mechanism from semantics.
I am not sure I understand the wording of your question however.