In my MySQL database I have a table of products which contains almost 625k rows. The table has 162 columns.
Now there is a search box on my home page where you can search for anything and, if your search term is matched from any of my product titles, it give you a list of 15 products. This is similar to Amazon and other e-commerce websites.
What I did so far was to create a JSON file with all the product ID's and title names. When user inputs a minimum of 3 chars into the search field, an AJAX request is made and gets the list. But my issue is that the JSON file is almost 12MB in size, and the ajax calls it whenever user write's a char or removes a char. It was working fine until I was on local Machine and now as soon as I made it live it doesn't work for users, having lower then 5 MBPS internet connection. So I am looking for some advice, how do I create it fast as Amazon. I mean the search with auto suggestion from 625K products.
I am really sorry, but there is nothing more to give as an advice here then "go do some reading on database design and schema normalization".
If you have 162 columns in a table you will never be able to do an efficient search. The database (especially MySQL) will not hold the table in memory and indexes will not help either. Yes, you can throw it all into an ElasticSearch instance and it will fix some of your problems. But, honestly, this solution does not clean up the mess you have.
You should have a table with relevant information (titles, names, etc.) in one column (or also a numeric column for prices, etc). This metadata should reference the main table, the column should be fulltext-indexed. This way you ask for matches, filter results and JOIN relevant lines from the main table. This will work quickly with very little resources used.
Prob: I have a table which contains list of toys(dolls).while purchasing user can choose options(black dress or white dress) and accessories(earring and anklet).
options 1: First i thought of adding 2 different table for options and accessories.
http://imgur.com/a/FnE6O
But as i am never going to filter/search on these table so i thought of putting these options as JSON in seperate column.So that frontend can easily render JSON as options to user
QUES: As i have to add total price from options and accessories
Is this ok to store these type of details as JSON format?
I am also open for other suggestions which is easily maintainable
If you are using an sql-database (and it looks like you are) I really advise you against keeping the data as JSON inside.
Even though MySQL 5.7.8 gives you a good way to access data inside json types, you never know when you are going to need to filter/search/group based in that data.
A quick example that jumps here is that you want to give a user that buys a new doll (productid == 1) a list of all the add-on that people that also bought that doll (productid == 1) purchased in the last week (or month). If you are going to save that data as JSON it will be very hard to query.
I know that your example shows only the meta-data of the add-ons and options, but it's "easy" to go and also save the actual purchases that way (just as add-on to the purchase-row itself).
I advise against it.
Being new to relational database design, I am trying to clarify one piece of information to properly design this database. Although I am using Filemaker as the platform, I believe this is a universal question.
Using the logic of ideally having all one to many relationships, and using separate tables or join tables to solve these.
I have a database with multiple products, made by multiple brands, in multiple product categories. I also want this to be as scale-able as possible when it comes to reporting, being able to slice and dice the data in as many ways as possible since the needs of the users are constantly changing.
So when I ask the question "Does each Brand have multiple products" I get a yes, and "Does each product have multiple brands" the answer is no. So this is a one to many relationship, but it also seems that a self-join table might give me everything that I need.
This methodology also seems to go down a rabbit hole for other "product related" information such as product category, each product is tied to one product category, but only one product category is related to a product.
So I see 2 possibilities, make three tables and join them with primary and foreign keys, one for Brand, one for Product Category, and one for Products.
Or the second possibility is to create one table that has the brand and product category and product info all in one table (since they are all product related) and simply do self-joins and other query based tables to give me the future reporting requirements that will be changing over time.
I am looking for input from experiences that might point me in the right direction.
Thanks in advance!
Could you ever want to store additional information about a brand (company URL, phone number, etc.) or about a product category (description, etc.)?
If the answer is yes, you definitely want to use three tables. If you don't, you'll be repeating all that information for every single item that belongs to the same brand or same category.
If the answer is no, there is still an advantage to using three tables - it will prevent typos or other spelling inconsistencies from getting into your database. For example, it would prevent you from writing a brand as "Coca Cola" for some items and as "Coca-Cola" for other items. These inconsistencies get harder and harder to find and correct as your database grows. By having each brand only listed once in it's own table, it will always be written the same way.
The disadvantage of multiple tables is the SQL for your queries is more complicated. There's definitely a tradeoff, but when in doubt, normalize into multiple tables. You'll learn when it's better to de-normalize with more experience.
I am not sure where do you see a room for a self-join here. It seems to me you are saying: I have a table of products; each product has one brand and one (?) category. If that's the case then you need either three tables:
Brands -< Products >- Categories
or - in Filemaker only - you can replace either or both the Brands and the Categories tables with a value list (assuming you won't be renaming brands/categories and at the expense of some reporting capabilities). So really it depends on what type of information you want to get out in the end.
If you truly want your solution to be scalable you need to parse and partition your data now. Otherwise you will be faced with the re-structuring of the solution down the road when the solution grows in size. You will also be faced with parsing and relocating the data to new tables. Since you've also included the SQL and MySQL tags if you plan on connecting Filemaker to an external data source then you will definitely need to up your game structurally.
Building everything in one table is essentially using Filemaker to do Excel work and it won't cut it if you are connecting to SQL, MySQL, etc.
Self join tables are a great tool. However, they should really only be used for calculating small data points and should not be used as pivot points or foundations for your reporting features. It can grow out of control as time goes on and you need to keep your backend clean.
Use summary and sub-summary reporting features to slice product based data.
For retail and general product management solutions, whether it's Filemaker/SQL/or whatever the "Brand" or "Vendor" is it's own table. Then you would have a "Products" table (the match key being the "Brand ID").
The "Product Category" field should be a field in the "Products" table. You can manage the category values by building a standard value list or building a value list based on a "Product Category" table. The second scenario is better for long term administration.
I´m working on a joomla website (Virtuemart 2.0.6 webshop) and the plan is to make 2 or maybe even 3 seperate webshops with different domain names and databases but the owner uses the same warehouse for all the shops.
So I´m searching to find a way to syncronize the product stock in multiple Virtuemart shops.
Basically I´m looking for way to only share one specific table field between the shops.
Specifically in Virtuemart 2.xx the product stock data is stored in the table:
"jos_virtuemart_products"
field name: "product_in_stock"
I have looked everywhere for the answer to this one, but I haven´t found anything I could use to solve this problem.
You are going to have to code a custom plugin to do this if you want to have different product names, descriptions, and prices. In any case, it is going to take a custom plugin to achieve your synchronization.
Probably the best way to do it is write a plugin that updates the other sites when an order is completed.
How to import a CSV product catalog from my supplier(s) using Magento Community Version: 1.7.0.2?
I have 2 suppliers:
The first with 200 products, that I have inserted manually.
The second one (is a new supplier) with over 1000 products.
I have just received a CSV file with all the products from that supplier.
I would like this time to make full use of Magento import function in order to save a lot of time. But I don't know how to proceed.
My first issue is that the CSV file from my new supplier have the following attributes:
products_id, products_master, products_model, products_ean_code, products_name, products_description, products_image, products_image_extra1, products_image_extra2, products_image_extra3, products_image_extra4, products_image_extra5, products_image_extra6, products_quantity, products_price, products_manufacturer, products_status
While my Magento store have these attributes:
sku,_store,_attribute_set,_type,_category,_root_category,_product_websites,activation_information,color,colors,computer_manufacturers,contrast_ratio,cost,country_of_manufacture,country_orgin,cpu_speed,created_at,custom_design,custom_design_from,custom_design_to,custom_layout_update,description,dimension,enable_googlecheckout,farge,featured,finish,gallery,gender,gift_message_available,harddrive_speed,hardrive,has_options,image,image_label,in_depth,manufacturer,max_resolution,media_gallery,megapixels,memory,meta_description,meta_keyword,meta_title,minimal_price,model,msrp,msrp_display_actual_price_type,msrp_enabled,name,navn_eller_initialer,news_from_date,news_to_date,options_container,page_layout,personalisering,price,processor,ram_size,required_options,response_time,room,screensize,shape,shirt_size,shoe_size,shoe_type,short_description,sko_size,sko_size_m,small_image,small_image_label,special_from_date,special_price,special_to_date,status,tax_class_id,thumbnail,thumbnail_label,updated_at,url_key,url_path,visibility,weight,qty,min_qty,use_config_min_qty,is_qty_decimal,backorders,use_config_backorders,min_sale_qty,use_config_min_sale_qty,max_sale_qty,use_config_max_sale_qty,is_in_stock,notify_stock_qty,use_config_notify_stock_qty,manage_stock,use_config_manage_stock,stock_status_changed_auto,use_config_qty_increments,qty_increments,use_config_enable_qty_inc,enable_qty_increments,is_decimal_divided,_links_related_sku,_links_related_position,_links_crosssell_sku,_links_crosssell_position,_links_upsell_sku,_links_upsell_position,_associated_sku,_associated_default_qty,_associated_position,_tier_price_website,_tier_price_customer_group,_tier_price_qty,_tier_price_price,_group_price_website,_group_price_customer_group,_group_price_price,_media_attribute_id,_media_image,_media_lable,_media_position,_media_is_disabled,_super_products_sku,_super_attribute_code,_super_attribute_option,_super_attribute_price_corr
How should I proceed step by step to make Magento accept the CSV file from my supplier?
I need a solution as simple and as specific as possible because I am new at Magento and even if I have built my store all by myself, I am not a programmer.
What you're looking to do isn't hugely difficult, but careful work needs to be considered. I'm only going to talk about simple products and not other product types, or extra attributes based on the assumption that you don't need those.
What you need to do is to "map" your supplier's attributes to magento's attributes. Some of them should be very easy to see: "products_name" from your supplier will be "name" in the magento table.
What I'd recommend you do is to create a dummy product using the manage products page first, fill in all the details with the supplier's table's title instead of the data - so for instance products_name for Name. Fill in all other necessary information as well, and choosing the appropriate options for the product.
Note that some of them may not be relevant at all and you can leave them out. Leave out the images as well, we'll deal with that another time.
Once you've done that, use System > Import/Export > Export to export a .csv file - you now have a .csv file for mapping purposes, where the relevant supplier's titles match up against your magneto table. Using that, copy and paste your data over from your supplier's file. Rinse and repeat for different suppliers.
Important Note: products_id from your supplier may not make a good sku, since different suppliers may actually use the same code for their own products (we have two suppliers that use 5 digit numbers), so you should create your own sku accordingly using excel formulas or something.
Import this file now and you should have all the products setup, sans images.
I'll come back to images tomorrow but it's unfortunately seriously complicated due to various shortfalls in magento's data profile when handling .csv files. Here's a link to give you an idea:
http://ashsmith.co/2012/05/importing-product-images-via-csv-in-magento-1-7-0-0/