Related
Below are the Table Schema and Data in EVENT_DETAILS is 1.76 Million Records
CREATE TABLE EVENT_DETAILS
(
ORG_ID CHAR(32) NOT NULL,
CONFIG_KEY CHAR(32) NOT NULL,
COMPONENT_NAME VARCHAR(512) NOT NULL,
REQUEST_ID CHAR(32) NOT NULL,
UPDATE_DATE INT NOT NULL,
UPDATE_TIME INT NOT NULL,
CLIENT CHAR(16) NOT NULL,
STATUS INT NOT NULL,
EVENT_DATA VARCHAR(1024),
MESSAGE VARCHAR(512),
INSERTED_TIME TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL
);
CREATE INDEX MYINDEX1 ON EVENT_DETAILS (ORG_ID,CONFIG_KEY,COMPONENT_NAME,UPDATE_DATE);
CREATE INDEX MYINDEX2 ON EVENT_DETAILS (UPDATE_DATE,COMPONENT_NAME);
CREATE TABLE DOMAIN_KEYS (
DOMAIN_KEY char(32) COLLATE utf8_unicode_ci NOT NULL,
DOMAIN_IDENTITY varchar(256) COLLATE utf8_unicode_ci NOT NULL,
DOMAIN_DISPLAY_NAME varchar(256) COLLATE utf8_unicode_ci NOT NULL,
ORG_ID char(32) COLLATE utf8_unicode_ci NOT NULL,
LOCATION_KEY int(11) DEFAULT '0',
DESCRIPTION varchar(4000) COLLATE utf8_unicode_ci DEFAULT NULL,
CATEGORY int(11) NOT NULL DEFAULT '1',
ENVIRONMENT_TYPE int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (DOMAIN_KEY)
);
CREATE TABLE ORGANISATIONS (
ORG_ID char(32) NOT NULL,
ORG_IDENTITY varchar(256) NOT NULL,
ORG_NAME varchar(256) NOT NULL,
ORG_DESC varchar(256) DEFAULT NULL,
PARENT_ORG_ID char(32) NOT NULL,
PRIMARY KEY (ORG_ID)
);
Below is the Query and on average it takes around 90 Seconds to fetch the Required data and Optimizer is not using any Index I created, Data is fetched for last 5 days data which is around 40% of complete data. Please suggest some Optimizations and any Indexes if required
SELECT ED.ORG_ID,
ORG.ORG_IDENTITY,
ORG.ORG_NAME,
ED.CONFIG_KEY,
REPLACE(REPLACE(ED.COMPONENT_NAME,'-CURRENT',''),'-HISTORICAL','') AS COMP_NAME,
DO.DOMAIN_IDENTITY,
DO.DOMAIN_DISPLAY_NAME,
GROUP_CONCAT(ED.STATUS ORDER BY ED.INSERTED_TIME DESC) AS ALL_STATUS
FROM EVENT_DETAILS ED ,
ORGANISATIONS ORG ,
DOMAIN_KEYS DO
WHERE UPDATE_DATE > '20191204'
AND ORG.ORG_ID = ED.ORG_ID
AND DO.DOMAIN_KEY=ED.CONFIG_KEY
AND DO.LOCATION_KEY <> 0
GROUP BY ED.ORG_ID,
ORG.ORG_IDENTITY,
ORG.ORG_NAME,
ED.CONFIG_KEY,
DO.DOMAIN_IDENTITY,
DO.DOMAIN_DISPLAY_NAME,
COMP_NAME
HAVING ALL_STATUS LIKE '0,0,0%';
I have 3 tables:
CREATE TABLE `UrunPaketDetay`(
`UrunPaketDetay` int NOT NULL AUTO_INCREMENT,
`UrunPaketNo` int NULL,
`Tarih` date NULL,
`Barkod` varchar(14) NOT NULL,
`Urun` varchar(50) NULL,
`SunumSekli` varchar(50) NULL,
`IslemeSekli` varchar(50) NULL,
`PaketlemeSekli` varchar(50) NULL,
`Kalibre` varchar(50) NULL,
`Kilo` double NULL,
`GlazeOran` varchar(50) NULL,
`Uretici` varchar(190) NULL,
`PaletKod` varchar(50) NULL,
PRIMARY KEY (`UrunPaketDetay`)
)CHARACTER SET utf8;
CREATE TABLE `CkisEks`(
`CikId` int NOT NULL AUTO_INCREMENT,
`Tarih` date NULL,
`Musteri` varchar(190) NULL,
`TeslimatYer` varchar(50) NULL,
`CikisSaati` time NULL,
`AracPlakasi` varchar(18) NULL,
`AracTel` varchar(16) NULL,
`KonteynirNo` varchar(50) NULL,
`PaletKod` varchar(12) NULL,
`Kilo` double NULL,
PRIMARY KEY (`CikId`)
)CHARACTER SET utf8;
CREATE TABLE `Ckis_Detay`(
`CD_Id` int NOT NULL AUTO_INCREMENT,
`CikId` int NULL,
`Barkod` varchar(50) NULL,
`Urun` varchar(50) NULL,
`Kalibre` varchar(50) NULL,
`Kilo` double NULL,
`Uretici` varchar(50) NULL,
`Musteri` varchar(190) NULL,
`PaletKod` varchar(50) NULL,
`Tarih` date NULL,
PRIMARY KEY (`CD_Id`)
)CHARACTER SET utf8;
I fill up my first table. After on 2nd table I call PaletKod row. PaletKod row autofill Kilo Row on 2nd table from 1st table.
But I need when I fill PaletKod to my 2nd table how can I select Paletkod from first table and fill my 3rd table values (Uretici, Urun, Kilo, Kalibre )?
I need trigger code. Can you help?
From My Understanding to do this,
DELIMITER $$
CREATE TRIGGER Insert_UrunPaketDetay
AFTER INSERT ON UrunPaketDetay
FOR EACH ROW
INSERT INTO Ckis_Detay (Uretici,Urun,Kilo,Kalibre)
VALUES
(UrunPaketDetay.Uretici,UrunPaketDetay.Urun
,UrunPaketDetay.Kilo,UrunPaketDetay.Kalibre);
END$$
DELIMITER ;
Try this code
Please could anyone help with the following query? (180352 rows)
SELECT COUNT(p.stock_id) AS num_products,
p.master_photo, p.product_photo, p.stock_id, p.master, p.title, p.price, p.stock_level, p.on_order, p.location, p.supplier, p.category, p.sub_category, p.reorder
FROM products AS p
WHERE p.sub_category != 'Subscriptions'
GROUP BY p.master
ORDER BY p.stock_id ASC
LIMIT 0, 20
It's running at 6 seconds.
When I remove the order by it run's at 0.0023 seconds.
And also the same when I remove the group by.
The stock_id (unique) and sub_category are indexed.
I can't think of another way to approach a query like this as it is vital that I group by the master to get the count of product variations and also vital that they can be ordered (not necessarily by stock_id but that's the default).
Thank you
As requested by e4c5 below is the result of the explain with the order by
id: 1
select_type: SIMPLE
table: p
type: range
possible_keys: sub_category
key: sub_category
key_len: 52
ref: NULL
rows: 181691
Extra: Using where; Using temporary; Using filesort
and then without the order by
id: 1
select_type: SIMPLE
table: p
type: index
possible_keys: sub_category
key: master
key_len: 52
ref: NULL
rows: 21
Extra: Using where
and then below is the create table
CREATE TABLE IF NOT EXISTS `products` (
`stock_id` varchar(50) NOT NULL,
`conv_stock_id` varchar(100) NOT NULL,
`conv_quantity` decimal(10,2) NOT NULL,
`master` varchar(50) NOT NULL,
`master_photo` varchar(255) NOT NULL,
`free_guide_photo` varchar(255) NOT NULL,
`product_var_photo` varchar(255) NOT NULL,
`master_title` varchar(255) NOT NULL,
`master_slug` varchar(255) NOT NULL,
`master_page_title` varchar(255) NOT NULL,
`product_photo` varchar(255) NOT NULL,
`original_product_photo` varchar(255) NOT NULL,
`title` varchar(255) NOT NULL,
`orig_title` varchar(255) NOT NULL,
`page_title` varchar(255) NOT NULL,
`description` longtext NOT NULL,
`slug` varchar(255) NOT NULL,
`custom_url` varchar(255) NOT NULL,
`location` varchar(255) NOT NULL,
`supplier` varchar(50) NOT NULL,
`supplier_stock_id` varchar(50) NOT NULL,
`supplier_discount` int(11) NOT NULL,
`category` varchar(50) NOT NULL,
`sub_category` varchar(50) NOT NULL,
`cost_price` decimal(10,2) NOT NULL,
`discount_cost_price` decimal(10,2) NOT NULL,
`price` decimal(10,2) NOT NULL,
`sale_price` decimal(10,2) NOT NULL,
`sale_price_startdate` date NOT NULL,
`sale_price_enddate` date NOT NULL,
`orig_price_trail` int(3) NOT NULL,
`price_trail` varchar(50) NOT NULL,
`price_rule` int(1) NOT NULL,
`pack_size` int(11) NOT NULL,
`parcel_size` int(1) NOT NULL,
`packaging_rule` int(11) NOT NULL,
`cut_tear` int(1) NOT NULL,
`oversized_parcel` int(1) NOT NULL,
`print_label` int(1) NOT NULL,
`stock_level` decimal(10,1) NOT NULL,
`stock_level_group` varchar(50) NOT NULL,
`stock_level_increment` decimal(10,2) NOT NULL,
`stock_check_date` datetime NOT NULL,
`reorder` int(1) NOT NULL,
`reorder_level` decimal(10,1) NOT NULL,
`reorder_quantity` decimal(10,1) NOT NULL,
`reorder_attempts` int(1) NOT NULL,
`unit_size` decimal(10,1) NOT NULL,
`on_order` decimal(10,1) NOT NULL,
`date_ordered` datetime NOT NULL,
`back_order` decimal(10,1) NOT NULL,
`uom` decimal(10,1) NOT NULL,
`uom_value` varchar(100) NOT NULL,
`stock_estimate` int(1) NOT NULL,
`due_date` datetime NOT NULL,
`quantity` varchar(255) NOT NULL,
`colour` varchar(255) NOT NULL,
`colour_family` varchar(255) NOT NULL,
`type` varchar(255) NOT NULL,
`style` varchar(255) NOT NULL,
`pattern` varchar(255) NOT NULL,
`shape` varchar(255) NOT NULL,
`design` varchar(255) NOT NULL,
`fibre` varchar(255) NOT NULL,
`material` varchar(255) NOT NULL,
`pattern_for` varchar(255) NOT NULL,
`difficulty` varchar(255) NOT NULL,
`fabric_count` varchar(255) NOT NULL,
`yarn_thickness` varchar(255) NOT NULL,
`suggested_needle_size` varchar(255) NOT NULL,
`tension` varchar(255) NOT NULL,
`collections` varchar(255) NOT NULL,
`product_features` varchar(255) NOT NULL,
`size` varchar(255) NOT NULL,
`actual_size` varchar(255) NOT NULL,
`length` varchar(255) NOT NULL,
`width` varchar(255) NOT NULL,
`weight` varchar(255) NOT NULL,
`weight_gsm` varchar(255) NOT NULL,
`brand` varchar(255) NOT NULL,
`designer` varchar(255) NOT NULL,
`composition` varchar(255) NOT NULL,
`washing_instructions` varchar(255) NOT NULL,
`matching_thread` varchar(50) NOT NULL,
`sample` varchar(50) NOT NULL,
`fat_quarter` varchar(50) NOT NULL,
`barcode` varchar(13) NOT NULL,
`list_international` int(1) NOT NULL,
`token` varchar(50) NOT NULL,
`create_sample` int(1) NOT NULL,
`create_fatquarter` int(1) NOT NULL,
`create_listing_type` int(1) NOT NULL,
`create_listing_size` int(11) NOT NULL,
`create_listing_price` decimal(10,2) NOT NULL,
`create_listing_price_rule` int(11) NOT NULL,
`create_listing_sale_price` decimal(10,2) NOT NULL,
`create_listing_parcelsize` int(1) NOT NULL,
`create_listing_barcode` varchar(13) NOT NULL,
`auto_listing` int(1) NOT NULL,
`custom_bridal` int(1) NOT NULL,
`pickwave_assign` int(1) NOT NULL,
`kit_product` int(11) NOT NULL,
`fatquarter_product` int(1) NOT NULL,
`sample_product` int(1) NOT NULL,
`grouped_product` int(1) NOT NULL,
`grouped_product_quantity` decimal(10,1) NOT NULL,
`multiple_product` int(1) NOT NULL,
`freepost_product` int(1) NOT NULL,
`status` int(1) NOT NULL,
`update_stock_level` int(1) NOT NULL,
`force_product_photo` int(1) NOT NULL,
`created_master_photo` int(1) NOT NULL,
`force_master_photo` int(1) NOT NULL,
`created_free_guide_photo` int(1) NOT NULL,
`force_free_guide_photo` int(1) NOT NULL,
`created_product_var_photo` int(1) NOT NULL,
`force_product_var_photo` int(1) NOT NULL,
`force_additional_photo` int(1) NOT NULL,
`created_price_levelling` int(1) NOT NULL,
`created_grouped_product` int(1) NOT NULL,
`updated_stock_level` int(1) NOT NULL,
`create_multiple_listing` int(1) NOT NULL,
`create_freepost_listing` int(1) NOT NULL,
`create_freeguide_info` int(1) NOT NULL,
`created_by` int(11) NOT NULL,
`date_created` datetime NOT NULL,
UNIQUE KEY `stock_id` (`stock_id`),
KEY `token` (`token`),
KEY `title` (`title`),
KEY `stock_level_group` (`stock_level_group`),
KEY `sub_category` (`sub_category`),
KEY `stock_level` (`stock_level`),
KEY `category` (`category`),
KEY `conv_stock_id` (`conv_stock_id`),
KEY `conv_quantity` (`conv_quantity`),
KEY `created_price_levelling` (`created_price_levelling`),
KEY `master` (`master`),
KEY `colour` (`colour`),
KEY `auto_listing` (`auto_listing`),
KEY `multiple_product` (`multiple_product`),
KEY `status` (`status`),
KEY `ebay_master` (`ebay_master`),
KEY `parcel_size` (`parcel_size`),
KEY `grouped_product` (`grouped_product`),
KEY `sample_product` (`sample_product`),
KEY `fatquarter_product` (`fatquarter_product`),
KEY `created_grouped_product` (`created_grouped_product`),
KEY `price` (`price`),
KEY `freepost_product` (`freepost_product`),
KEY `master_title` (`master_title`),
KEY `c_sub_category_master` (`sub_category`,`master`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
You haven't provided the output from explain, however based on your query it would seem that ORDER BY forces a full table scan. That would make the query very slow.
When you don't use the ORDER BY, the db reads the results for the first 20 master values (there maybe quite a few of them) and groups them together and returns the result.
When you order by stock_id the whole table needs to be looked at to find which masters are associated with the lowest values stock_ids
It maybe possible to improve performance with a composite index on sub_category,master however a conclusion cannot be made unless you share your SHOW CREATE TABLES, EXPLAIN output.
Update
Based on your CREATE TABLE statements, I see that your database isn't normalized. For example Why do I get the feeling that the following columns should in a table of their own?
supplier varchar(50) NOT NULL,
supplier_stock_id varchar(50) NOT NULL,
supplier_discount int(11) NOT NULL,
You should only have a supplier_stock_id in your products table (foreign key to the suppliers table). There are similar sets of columns which really should be moved out.
When you do so you can create leaner and meaner indexes on this table. But that's not all the table becomes narrower. Which in turn means the worst case scenario of a full table scan actually becomes a lot faster.
I also noticed that the table does not have a primary key. Which is a big no-no. The stock_id if it's numeric should be primary key. If it's not numeric it might stil be the best candidate for primary key but this is something you need to decide.
Try adding an Index on stock_id in the products table... that should help.
Im trying to create a table using the following query.phpmyadmin does not allow it.What im i doing wrong
CREATE TABLE main (
id INT NOT NULL AUTO_INCREMENT PRIMARY KEY(id),
dealername VARCHAR(32) NOT NULL,
product VARCHAR(32) NOT NULL,
type VARCHAR(16) NOT NULL,
date VARCHAR(16) NOT NULL,
desc VARCHAR(255) NOT NULL,
location VARCHAR(32) NOT NULL,
sublocation VARCHAR(32) NOT NULL,
address VARCHAR(255) NOT NULL,
phone1 VARCHAR(16) NOT NULL,
phone2 VARCHAR(16) NOT NULL,
auth VARCHAR(10) NOT NULL,
brands VARCHAR(255) NOT NULL,
lat VARCHAR(32) NOT NULL,
lon VARCHAR(32) NOT NULL
);
ERROR:
#1064 - You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '(id), dealername VARCHAR(32) NOT NULL, product VARCHAR(32) NOT NULL, type VAR' at line 2
You have to remove (id)
CREATE TABLE main (
id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
dealername VARCHAR(32) NOT NULL,
product VARCHAR(32) NOT NULL,
type VARCHAR(16) NOT NULL,
date VARCHAR(16) NOT NULL,
`desc` VARCHAR(255) NOT NULL,
location VARCHAR(32) NOT NULL,
sublocation VARCHAR(32) NOT NULL,
address VARCHAR(255) NOT NULL,
phone1 VARCHAR(16) NOT NULL,
phone2 VARCHAR(16) NOT NULL,
auth VARCHAR(10) NOT NULL,
brands VARCHAR(255) NOT NULL,
lat VARCHAR(32) NOT NULL,
lon VARCHAR(32) NOT NULL
);
The syntax of the table create statement is wrong. See the documentation.
The name of the column after Primary key is not expected.
And desc is a reserved word in sql so you have to escape it or better take an other column name.
remove (id)
desc is mysql keyword.
put it in backticks like this, if you still want to use desc as your column name.
CREATE TABLE main (
id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
dealername VARCHAR(32) NOT NULL,
product VARCHAR(32) NOT NULL,
type VARCHAR(16) NOT NULL,
date VARCHAR(16) NOT NULL,
`desc` VARCHAR(255) NOT NULL,
location VARCHAR(32) NOT NULL,
sublocation VARCHAR(32) NOT NULL,
address VARCHAR(255) NOT NULL,
phone1 VARCHAR(16) NOT NULL,
phone2 VARCHAR(16) NOT NULL,
auth VARCHAR(10) NOT NULL,
brands VARCHAR(255) NOT NULL,
lat VARCHAR(32) NOT NULL,
lon VARCHAR(32) NOT NULL
);
ive run into a bit of a snag here, it seems that my pages are loading slower and slower the most content is being added to the database.
i have already added indexes on the ID and ID_* fields, that fixed the problem a while ago, but is not working anymore.
i've also run OPTIMIZE on all tables.
here is my query:
select
l.id, l.id_infusionsoft, l.name_first, l.name_last, l.postcode, a.website as a_website, l.website as l_website, l.date_added,
a.id_dealership, a.id_lead, a.id as id_assign, a.date_assigned, a.manual_or_auto, a.assigned_by,
d.name as dealership,
COALESCE(a.date_assigned, l.date_added) AS date_sort
from `leads` as l
left join `assignments` as a on (a.id_lead = l.id)
left join `dealerships` as d on (d.id = a.id_dealership)
order by date_sort desc
here are the table structures:
CREATE TABLE assignments (
id int(11) NOT NULL auto_increment,
id_dealership int(11) NOT NULL,
id_lead int(11) NOT NULL,
date_assigned int(11) NOT NULL,
website varchar(255) NOT NULL default '',
make varchar(255) NOT NULL default '',
model varchar(255) NOT NULL default '',
ip_address varchar(255) NOT NULL default '',
is_reassign varchar(255) NOT NULL default 'no',
manual_or_auto varchar(255) NOT NULL default 'N/A',
assigned_by varchar(255) NOT NULL default 'N/A',
PRIMARY KEY (id),
KEY id_dealership (id_dealership),
KEY id_lead (id_lead),
KEY date_assigned (date_assigned)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=24569 ;
CREATE TABLE dealerships (
id int(11) NOT NULL auto_increment,
province varchar(255) NOT NULL default '',
city varchar(255) NOT NULL default '',
`name` varchar(255) NOT NULL,
email text NOT NULL,
email_subject varchar(255) NOT NULL default 'Car Lead',
`type` varchar(255) NOT NULL,
make varchar(255) NOT NULL,
leads int(11) NOT NULL default '0',
`status` varchar(255) NOT NULL,
low_notif int(11) NOT NULL,
reassign_id int(11) NOT NULL,
reassign_days int(11) NOT NULL,
salesman varchar(255) NOT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=176 ;
CREATE TABLE leads (
id int(11) NOT NULL auto_increment,
id_infusionsoft int(11) NOT NULL,
name_first varchar(255) NOT NULL,
name_last varchar(255) NOT NULL,
email varchar(255) NOT NULL,
phone_home varchar(255) NOT NULL,
phone_cell varchar(255) NOT NULL,
phone_work varchar(255) NOT NULL,
postcode varchar(255) NOT NULL,
website varchar(255) NOT NULL,
address varchar(255) NOT NULL,
province varchar(255) NOT NULL,
employer varchar(255) NOT NULL,
city varchar(255) NOT NULL,
rentorown varchar(255) NOT NULL,
emp_months varchar(255) NOT NULL,
emp_years varchar(255) NOT NULL,
sin varchar(255) NOT NULL,
occupation varchar(255) NOT NULL,
monthly_income varchar(255) NOT NULL,
bankruptcy varchar(255) NOT NULL,
tradein varchar(255) NOT NULL,
cosign varchar(255) NOT NULL,
monthly_payment varchar(255) NOT NULL,
residence_years varchar(255) NOT NULL,
residence_months varchar(255) NOT NULL,
birthday varchar(255) NOT NULL,
make varchar(255) NOT NULL,
model varchar(255) NOT NULL,
date_added int(11) NOT NULL,
ip_address varchar(255) NOT NULL default '',
time_to_call varchar(255) NOT NULL,
referrer varchar(255) NOT NULL default '',
`source` varchar(255) NOT NULL,
PRIMARY KEY (id),
KEY id_infusionsoft (id_infusionsoft),
KEY date_added (date_added)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=20905 ;
what can i do to make my query run faster?
EDIT: here is the explain select output:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE l ALL NULL NULL NULL NULL 15381 Using temporary; Using filesort
1 SIMPLE a ref id_lead id_lead 4 infu.l.id 1
1 SIMPLE d eq_ref PRIMARY PRIMARY 4 infu.a.id_dealership 1
Use InnoDB instead of MyISAM. InnoDB caches both data and indexes, whereas MyISAM only caches indexes. InnoDB has other benefits too.
Do you really need all 15381 leads to be returned by this query? Or are you trying to query only a subset of leads, such as those that have assignments and dealerships?
Make sure you understand the type of join you're using. I suspect you're using left join when you need an inner join. It's probably causing the result set to be larger than it needs to be. Adding overhead for sorting, memory use, etc.
Choose data types more appropriately. Does every string have to be varchar(255)? Are you aware that varchars pad out to their maximum length in memory? You're even using varchar(255) to store an IP address. Also, you're using a signed integer to store dates?
Change the following index:
ALTER TABLE assignments ADD KEY assgn_lead_dealership (id_lead, id_dealership),
DROP KEY id_lead;