I have a form with about 500+ fields (it's a 10 page form, different data types). Can you guys please advise me on the best way to store the data from the form? I can create 500 fields in multiple, logically divided tables but that seems a lot (or maybe that's the best way?!) since I have a few of these forms. I am looking into serializing the data and storing in longtext mysql field. That will have its drawbacks (the one I am thinking of is if the customer wants to search individual fields in the future) but it does seem like a pretty fast solution. I will appreciate if you would share you experience with a similar situation.
Presumibly you dont expect the user to fill the form in in a single sitting! So you will need some sort of work flow to store drafts and amend previous copies etc.
Also assuming some parts of the form are optional.
You could either define a set of database tables with a master table to track status, user name etc, and a child table for each optional part of the form.
Or you could define an XML schema which contains all the possible fields in the form etc plus some status information.
If you always process the entire form and dont want to search through your collection of forms then the XML soiution is slightly better as there are some nifty tricks for moving data from XML to HTML forms and back again. If you need to search based on values inside the form then the SQL based solution is preferable.
You may need 500 columns - unless they can be placed in other tables. It can be hard to tell without seeing your requirements.
Serialising it would make one of the advantages of using a database impossible - querying against certain column values.
create table profile_details (
user_id number,
field_name varchar,
field_value varchar
);
Now you are not only not limited by number of fields, you also pretty free to add and remove them as you keep developing and maintaining your app.
select firstname, lastname, zipcode
from profiles p
join profile_details d1 on (p.user_id=d1.user_id)
join profile_details d2 on (p.user_id=d2.user_id)
where d1.field_name='hobby' and d1.field_value='fishing'
and d2.field_name='income' and d2.field_value>cast(250000 as number);
Related
Apologies if this is redundant, and it probably is, I gave it a look but couldn't find a question here that fell in with what I wanted to know.
Basically we have a table with about ~50000 rows, and it's expected to grow much bigger than that. We need to be able to allow admin users to add in custom data to an item based on its category, and users can just pick which fields defined by the administrators they want to add info to.
Initially I had gone with an item_categories_fields table which pairs up entries from item_fields to item_categories, so admins can add custom fields and reuse them across categories for consistency. item_fields has a relationship to item_field_values which links values with fields, which is how we handled things in .NET. The project is using CAKEPHP though, and we're just learning as we go, so it can get a bit annoying at times.
I'm however thinking of maybe just adding an item_custom_fields table that is essentially the item_id and a text field that stores XMLish formatted data. This is just for the values of the custom fields.
No problems if I want to fetch the item by its id as the required data is stored in the items table, but what if I wanted to do a search based on a custom field? Would a
SELECT * FROM item_custom_fields
WHERE custom_data LIKE '%<material>Plastic</material>%'
(user input related issues aside) be practical if I wanted to fetch items made of plastic in this case? Like how slow would that be?
Thanks.
Edit: I was afraid of that as realistically this thing will be around 400k rows for that one table at launch, thanks guys.
Any LIKE query that starts with % will not use any indexes you have on the column, so the query will scan the whole table to find the result.
The response time for that depends highly on your machine and the size of the table, but it definitely won't be efficient in any shape or form.
Your previous/existing solution (if well indexed) should be quite a bit faster.
I'm trying to do it like this:
Every single user can choose fields (like structures on MySQL) where this fields can handle their respective value, it's like doing a DB inside a DB.
But how can I do it using a single table?
(not talking about user accounts etc where I should be able to use a pointer to his own "structure")
Do something like: varchar Key where register something like "Name:asd" where PHP explode : to get the respective structure ('name' in this case) and the respective value? ('asd')
Use BLOB? can someone turn the light on for me? I don't know how to do something where works better than my current explanation...
I know my text is confuse and sorry for any bad english.
EDIT:
Also, they could add multiple keys/"structures" where accepts a new value
And they are not able to see the Database or Tables, they still normal users
My server does not support Postogre
In my opinion you should create two tables.
with the user info
with 3 fields (userid, key and value)
Each user has 1 record in the first table. Each user can have 0 or more records in the second table. This will ensure you can still search the data and that users can easily add more key/value pairs when needed.
Don't start building a database in a database. In this case, since the user makes the field by himself there is no relation between the fields as I understand? In that case it would make sense to take a look at the NoSQL databases since they seem to fit very good for this kind of situations.
Another thing to check is something like:
http://www.postgresql.org/docs/8.4/static/hstore.html
Do not try to build tables like: records, fields, field types etc. That's a bad practice and should not be needed.
For a more specific answer on your wishes we need a bit more info about the data the user is storing.
While i think the rational answer to this question is the one given by PeeHaa, if you really want the data to fit into one table you could try saving a serialized PHP array in one of the fields. Check out serialize and unserialize
Generates a storable representation of a value
This is useful for storing or passing PHP values around without losing
their type and structure.
This method is discouraged as it is not at all scalable.
Use a table with key-value pairs. So three columns:
user id
key ("name")
value ("asd")
Add an index on user id, so that you can query a user's attributes easily. If you wanted to query all users with the same properties, then you could add a second index on key and/or value.
Hope you are using a programming language also to get the data and present them.
You can have a single table which has a varchar field. Then you store the serialized data of the field structure and their value in that field. When you want to get the structure, query the data and De-serialize that varchar field data.
As per my knowledge every programming language supports serialization and De-serialization.
Edited : This is not a scalable option.
I am currently building a small crm application. I need each user to be able to define their own custom fields. I am currently building this crm using php and mysql.
Example: I have a "customer" table which has the standard fields: name, phone, address, email, etc. But i want to allow the user (unique session) to add fields that are custom to his/her business which are only accessible to him (not other users). I then want these custom fields to function just like all the other fields in the table (ability to search, send and received data). I am hoping i can accomplish this in mysql and php but am open to any technology or solution that is considered best practice. Thank you for your help.
This can be done by creating a table called "customfields" with the elements "id, fieldname, company_id", then another table that would associate those custom fields with data, eg "customercustomdata: id, customfields_id, customer_id". Associate "ownership" of a field the same way
To create a new custom field, "insert into customfields (fieldname,company_id) values ('Birthday',companyid);"
Does that help?
#Matt H: Is this method considered AEV or just standard relational db?
So because i will have many users in many dif industries that will want to add their own custom fields to a number of different tables (contacts, transactions, events, etc) i am assuming that i would need the customfield table to have a user_fk/id or company fk/id, a related table fk/id, an id, and a field name? Am i on the right track? Then in the need to create a 2nd table to hold the data for each custom field buy having a customfield fk/id, customer fk/id, id and a data field to hold the actual data. Is this correct?
Ok so once i build those two additional tables how do I add them to the contacts table so it looks like one big table for the user, instead of the 3 tables?
Thanks again for you help.
Answer
after much research i have found that most people who wish to accomplish this are using document databases not relational databases.
You could place an extra column for storing string data and store an array describing the contents for custom cells. For example:
$custom = array(
array("field" => "bikesOwned", "value" => 4),
array("field" => "travelled", "value" => 14)
);
then use something like PHPs json_encode to store that data in the extra cell. Then all you would need to do is decode and process the array.
Some people suggesting using the Entity-Attribute-Value design, but before you do, please read Bad CaRMa, a story about an EAV-like design that nearly destroyed a company because it was unmaintainable.
To solve this better, read How FriendFeed uses MySQL to store schema-less data. You can lump all the custom columns into a single BLOB, and store it that way. Then if you want individual attributes to be searchable, create a table for that attribute, that maps values back to the customers table.
I wonder if anyone could offer their advice on this one.
I have some customer data whereby certain fields remain consistent. For example:
'Firstname', 'Lastname', 'Postcode'
These are the important fields and the ones that would be searched against. Over time we may have some additional data for a customer, however it is guaranteed that the fields will differ, therefore we can't indefinitely create more fields in the table to accommodate every possible new field.
I wondered what the options would be of storing the auxiliary date fields, for example would creating one additional field 'AuxData' maybe as a VARCHAR that used a JSON array?
(I am thinking of what I have seen in wordpress in the past)
I'd really appreciate anyone's thoughts on this one!
Many thanks
JAson
You can create an extra column with XML data, usually I would recommend JSON over XML, but MySQL has some special functionality to search trough XML data in fields. See this article for a few examples: http://www.informit.com/articles/article.aspx?p=1019623
There is also another possibility, you could create a new table with 3 columns: [Person_ID, Property, Value] where the primary key would span (Person_ID, Property). This way you can more easily search trough data, and you keep it MySQL instead of XML/JSON. However both options are valid.
I'm planning a database who has a couple of tables who contain plenty of address information, city, zip code, email address, phone #, fax #, and so on (about 11 columns worth of it), a table is an organizations table containing (up to) 2 addresses (legal contacts and contacts they should actually be used), plus every user has the same information tied to him.
We are going to have to run some geolocation stuff on those addresses too (like every address that's within X Kilometers from another address).
I have a bunch of options, each with its own problem:
I could put all the information inside every table but that would make for tables with a very large amount of columns which I'd have problems indexing, and if I change my address format it'll take a while to fix it.
I could put all the information inside an array and serialize it, then store the serialized information in one field, same problem with the previous method with a little less columns and much less availability through mysql queries
I could create a separate table with address information and link it to the other tables either by
putting an address_id column in the users and organizations table
putting a related_id and related_table columns in the addresses table
That should keep stuff tidier, but it might create some unforeseen problems with excessive joining or whatever.
Personally I think that solution 3.2 is the best, but I'm not too confident about it, so I'm asking for opinions.
Option 2 is definitely out as it would put the filtering logic into your codes instead of letting the DBMS handle them.
Option 1 or 3 will depend on your need.
if you need fast access to all the data, and you usually access both addresses along with the organization information, then you might consider option 1. But this will make it difficult to query out (i.e. slow) if the table get too big in mysql.
option 3 is good provided you index the tables correctly.