I have a table in a mysql DB with about 6 mil rows of data. Structure below. Most of my queries are searching for specific "customer" fields and display a value for each customer according to the value in column "value". The query searches the whole Table to match those customers specified in the query. This table started rather small but now it's gotten too big and my queries are taking quite some time to retrieve results. My questions is the following: If i create a separate table with just the customer field, along with an index, will that make my customer queries faster?
TABLE `data` (
`id` bigint(20) UNSIGNED NOT NULL,
`entry_id` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`date` date DEFAULT NULL,
`media_name` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`media_type` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`rate` decimal(8,2) DEFAULT NUCREATELL,
`value` decimal(8,2) DEFAULT NULL,
`page` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`type` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`sector` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`category` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`customer` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`product` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`description` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`image_id` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`city` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`address` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`supplier` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`time` time DEFAULT NULL,
`duration` time DEFAULT NULL,
`promoted_on` datetime NOT NULL,
`hasimage` tinyint(4) NOT NULL DEFAULT '0'
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
You want an index.
If you are searching for customers using in or = (the most common methods), then you can use a standard index on customer.
If you have more complex searches -- say using like with a leading wildcard -- then this does not work. A full text index might help. Or it might not, depending on the nature of the query and the data.
The "separate table" you're thinking about should be an index on your main table.
CREATE INDEX index_name
ON data(customer,value);
This will speed up the queries, and even prevent access to the table itself, at the cost of slightly slower INSERT and UPDATE operations.
Related
When I run this query:
select b.email,
concat('[AAA]: Xin chao QK ', a.client_name, ' HD ',a.account_no, ' vua khoi tao') as subject,
replace( (select email_content from xxff_message where refid=1 and status='O'),'#LOAN_ID#', a.account_no) as message, -- cp init
a.loan_id
from interface.vw_loan_full a
left join interface.xxff_client b
on a.client_id=b.client_id
where 1=1
and a.status=100
and (a.loan_id, email) not in ( select loan_id, receiver from xxff_loan_message_sent where stage='INIT100_MSG_EMAIL')
and date(a.value_date) between DATE_ADD(CURDATE(), INTERVAL -30 day) and date(now());
And I get this error:
Err] 1270 - Illegal mix of collations (utf8_unicode_ci,IMPLICIT),
(utf8_unicode_ci,COERCIBLE), (utf8_general_ci,IMPLICIT) for operation
'replace'
When I research, this cause is different utf8_unicode_ci and utf8_general_ci. But I checked all table, database are using utf8_unicode_ci .
But view table vw_loan_full is utf8_general_ci. Please help me change collation of view table.
Because I run command:
ALTER TABLE vw_loan_full CONVERT TO CHARACTER SET utf8 COLLATE
utf8_unicode_ci;
And get Error:
[Err] 1347 - 'interface.vw_loan_full' is not BASE TABLE
My result from SHOW CREATE TABLE xxff_client:
CREATE TABLE xxff_client ( client_id bigint(20) NOT NULL,
client_name varchar(150) COLLATE utf8_unicode_ci DEFAULT NULL,
national_id varchar(15) COLLATE utf8_unicode_ci DEFAULT NULL,
issue_date date DEFAULT NULL, issue_place varchar(120) COLLATE
utf8_unicode_ci DEFAULT NULL, Gender varchar(15) COLLATE
utf8_unicode_ci DEFAULT NULL, dob date DEFAULT NULL, phone
varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL, email
varchar(150) COLLATE utf8_unicode_ci DEFAULT NULL, address
varchar(250) COLLATE utf8_unicode_ci DEFAULT NULL, status
varchar(1) COLLATE utf8_unicode_ci DEFAULT NULL, creation_date
date DEFAULT NULL, USER_ID varchar(20) COLLATE utf8_unicode_ci
DEFAULT NULL, company_name varchar(150) COLLATE utf8_unicode_ci
DEFAULT NULL, company_address varchar(255) COLLATE utf8_unicode_ci
DEFAULT NULL, title varchar(50) COLLATE utf8_unicode_ci DEFAULT
NULL, dept_name varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
join_date date DEFAULT NULL, salary_date int(11) DEFAULT NULL,
remain_of_laborcontract int(11) DEFAULT NULL, salary_recently
varchar(150) COLLATE utf8_unicode_ci DEFAULT NULL, eco_userid
bigint(20) DEFAULT NULL, otp_count tinyint(20) DEFAULT '0',
PRIMARY KEY (client_id) ) ENGINE=InnoDB DEFAULT CHARSET=utf8
COLLATE=utf8_unicode_ci ROW_FORMAT=DYNAMIC
My result from SHOW CREATE TABLE xxff_loan_message_sent:
CREATE TABLE xxff_loan_message_sent ( refid bigint(20) NOT NULL
AUTO_INCREMENT, loan_id int(11) NOT NULL, stage varchar(20)
COLLATE utf8_unicode_ci DEFAULT NULL, sent_status varchar(6)
COLLATE utf8_unicode_ci DEFAULT NULL, created_by varchar(15)
COLLATE utf8_unicode_ci DEFAULT NULL, created_date date DEFAULT
NULL, receiver varchar(150) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (refid) ) ENGINE=InnoDB AUTO_INCREMENT=280 DEFAULT
CHARSET=utf8 COLLATE=utf8_unicode_ci ROW_FORMAT=DYNAMIC
My result from SHOW CREATE TABLE vw_loan_full:
CREATE ALGORITHM=UNDEFINED DEFINER=root#% SQL SECURITY DEFINER
VIEW vw_loan_full AS select a.id AS loan_id,a.client_id AS
client_id,b.display_name AS client_name,b.account_no AS
cif_no,a.product_id AS product_id,c.short_name AS
product_name,a.currency_code AS
ccy_code,a.annual_nominal_interest_rate AS
rate,a.term_frequency AS term,a.approved_principal AS
approved_principal,a.principal_amount AS
disbursed_amount,a.principal_outstanding_derived AS
pos,a.loan_status_id AS status,a.submittedon_date AS
created_date,a.submittedon_userid AS
created_by,a.expected_disbursedon_date AS
value_date,a.maturedon_date AS
maturity_date,a.approvedon_date AS
approvedon_date,a.disbursedon_date AS
disbursedon_date,a.expected_maturedon_date AS
expected_maturedon_date,a.loanpurpose_cv_id AS
loan_purpose,a.account_no AS account_no from
((mifostenant-default.m_loan a left join
mifostenant-default.m_client b on((a.client_id = b.id)))
left join mifostenant-default.m_product_loan c
on((a.product_id = c.id)))
My result from SHOW CREATE TABLE vw_loan_full:
CREATE TABLE xxff_message ( refid bigint(20) NOT NULL
AUTO_INCREMENT, msg_type varchar(8) COLLATE utf8_unicode_ci
DEFAULT NULL, sms_content varchar(500) COLLATE utf8_unicode_ci
DEFAULT NULL, email_content varchar(2000) COLLATE utf8_unicode_ci
DEFAULT NULL, email_subject varchar(250) COLLATE utf8_unicode_ci
DEFAULT NULL, status varchar(4) COLLATE utf8_unicode_ci DEFAULT
NULL, created_by varchar(15) COLLATE utf8_unicode_ci DEFAULT NULL,
created_date datetime DEFAULT NULL, updated_by varchar(15)
COLLATE utf8_unicode_ci DEFAULT NULL, updated_date datetime
DEFAULT NULL, stage_code varchar(35) COLLATE utf8_unicode_ci
DEFAULT NULL, email_receiver varchar(120) COLLATE utf8_unicode_ci
DEFAULT NULL, PRIMARY KEY (refid) ) ENGINE=InnoDB
AUTO_INCREMENT=16 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
ROW_FORMAT=DYNAMIC
How I can change collation for view table in MySQL? Thanks for your helpfull !!!
MySQL 5.5 in Windows
I hope someone can help me with this.
I am saving a personal data into MYSQL by encrypting using Yii2 encrypt/decrypt function, testing without saving the content into the DB works.
However saving the content into the DB and retrieving it back, seems something goes wrong because I am not able to decrypt it.
My MYSQL scheme is this
CREATE TABLE IF NOT EXISTS `customer` (
`customer_id` int(11) NOT NULL,
`customerHash` varchar(32) COLLATE utf8_bin DEFAULT NULL,
`customerDate` date DEFAULT NULL,
`customerTime` time DEFAULT NULL,
`customerUpdated` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`customerEmail` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`customerPassword` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`customerToken` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`customerAddress` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`customerPhone` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`customerPostcode` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`customerName` blob,
`customerSurname` varchar(255) COLLATE utf8_bin DEFAULT NULL
) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
Yii code
Encrypt
$request=Yii::$app->request;
$hash=Yii::$app->getSecurity()->generateRandomString();
$model->customerHash=$hash;
$model->customerName=Yii::$app->getSecurity()->encryptByKey($request->post('customerName'), $hash);
Decrypt
<? echo"Decode : ".Yii::$app->getSecurity()->decryptByKey($model->customerName, $model->customerHash);?>
Any help welcome.
Thanks
So i get an error when i try and use
SELECT views, keywords, title, url, thumbnail,
MATCH(keywords,title) AGAINST ('%$search_value%') AS relevance
FROM straight
WHERE MATCH (keywords,title) AGAINST ('%$search_value%')
ORDER BY relevance DESC
This is due to me not having FULLtext search enabled, but i cant seem to enable it. when i run the sql below:
ALTER TABLE straight ADD FULLTEXT(keywords, title)
i get this response:
MySQL returned an empty result set (i.e. zero rows). (Query took 3.8022 sec)
Then when trying to run the first query again i get the failed
#1191 - Can't find FULLTEXT index matching the column list
I can't tell why it's not registering. Any help would be great.
Thanks!
Edit:
My tabel:
CREATE TABLE `straight` (
`url` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
`title` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
`keywords` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
`production` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
`categories` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
`views` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
`likes` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
`length` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
`thumbnail` varchar(200) COLLATE utf8_unicode_ci DEFAULT NULL,
`date` varchar(12) COLLATE utf8_unicode_ci DEFAULT NULL,
UNIQUE KEY `url` (`url`),
FULLTEXT KEY `url_2` (`url`,`title`,`keywords`,`production`,
`categories`,`views`,`likes`,`length`,`thumbnail`,`date`
), ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
You need a FULLTEXT index that matches, exactly, the columns upon which you want to search. The FULLTEXT index you have has more columns than you need.
Try adding the one you mentioned.
ALTER TABLE straight ADD FULLTEXT(keywords, title)
Then look at the table definition and make sure it's there.
I am struggling with a MySQL query. Here's the problem:
SELECT * from sf_forecasts WHERE contentid='1234' AND fulldate='2012-12-13'
There is one record in my table that matches this criteria, and this query finds it with no difficulty. However, when I move to update it using this query...
UPDATE sf_forecasts
SET listingTitle ='Some Snow Resort', location ='49.23, -115.22'
WHERE contentid='1234' AND fulldate='2012-12-13'
...I always get a notice of "0 rows affected." Strangely, the query is actually updating the record, but it's only touching fields that are NULL. If my example record has a 'location' value of NULL, the query will update the data, but if it already contains a non-NULL value, the query will not change it, even though the query clearly contains new information.
What gives?
P.S. I am using a PRIMARY key of contentid+fulldate (b-tree, unique) if this matters.
P.S.S. Here is the 'SHOW CREATE TABLE sf_forecasts' results as requested below:
sf_forecasts CREATE TABLE `sf_forecasts` (
`contentid` int(11) NOT NULL,
`fulldate` varchar(10) COLLATE utf8_unicode_ci NOT NULL,
`listingTitle` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`location` varchar(10) COLLATE utf8_unicode_ci NOT NULL,
`dayofmonth` smallint(2) NOT NULL,
`dayofweek` varchar(10) COLLATE utf8_unicode_ci NOT NULL,
`minTempF` smallint(4) NOT NULL,
`maxTempF` smallint(4) NOT NULL,
`minTempC` smallint(4) NOT NULL,
`maxTempC` smallint(4) NOT NULL,
`skyCondition` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`precip` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`snowPotential` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`wind` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`minTempF_custom` smallint(4) DEFAULT NULL,
`maxTempF_custom` smallint(4) DEFAULT NULL,
`minTempC_custom` smallint(4) DEFAULT NULL,
`maxTempC_custom` smallint(4) DEFAULT NULL,
`baseH_custom` smallint(4) DEFAULT NULL,
`baseL_custom` smallint(4) DEFAULT NULL,
`topH_custom` smallint(4) DEFAULT NULL,
`topL_custom` smallint(4) DEFAULT NULL,
`skyCondition_custom` varchar(200) COLLATE utf8_unicode_ci DEFAULT NULL,
`precip_custom` varchar(200) COLLATE utf8_unicode_ci DEFAULT NULL,
`snowPotential_custom` varchar(200) COLLATE utf8_unicode_ci DEFAULT NULL,
`wind_custom` varchar(200) COLLATE utf8_unicode_ci DEFAULT NULL,
`icon` varchar(30) COLLATE utf8_unicode_ci NOT NULL,
`icon_custom` varchar(30) COLLATE utf8_unicode_ci DEFAULT NULL,
`shortrange_custom` varchar(5000) COLLATE utf8_unicode_ci DEFAULT NULL,
`longrange_custom` varchar(5000) COLLATE utf8_unicode_ci DEFAULT NULL,
`scCode` varchar(10) COLLATE utf8_unicode_ci DEFAULT NULL,
`scUpdate` varchar(10) COLLATE utf8_unicode_ci DEFAULT NULL,
`lastUpdate` varchar(10) COLLATE utf8_unicode_ci DEFAULT NULL,
`skys` varchar(30) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`contentid`,`fulldate`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
The issue is with difference in datatype length defined and length of data being updated
Your location is varchar(10) and you are trying to insert 14 characters
You can always check if there are errors after query execution:
SHOW ERRORS;
SHOW WARNINGS;
I'm trying to import a csv (that is a data extract from a SQL Server db) into MySQL.
I'm using this command to load the file:
LOAD DATA INFILE '/Users/Tyler/Desktop/playersToTeams.txt' INTO TABLE players_to_teams FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n';
And I get this error:
Error Code: 1062. Duplicate entry '124547' for key 'PRIMARY'
When I run: grep 124547 playersToTeams.txt:
119683,True,True,124547,1493,2011-08-31 02:22:56.630000000,,,,,http://bucknellbison.cstv.com///sports/m-wrestl/mtt/stolfi_joe00.html,,,,,,
124547,True,True,129411,14726,2011-08-31 02:22:56.630000000,Free/Breast,,,,http://usctrojans.collegesports.com/sports/m-swim/mtt/walling_emmett00.html,,,,,,
I can see that the 4th column of the first entry has the same number as the first column (pk, id), but the 4th column doesn't have any sort of index on it.
Here's the create schema created by sql workbench:
CREATE TABLE `players_to_teams` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`IsActive` tinyint(1) DEFAULT NULL,
`IsVisible` tinyint(1) DEFAULT NULL,
`PlayerId` int(11) DEFAULT NULL,
`TeamId` int(11) DEFAULT NULL,
`CreationDate` datetime DEFAULT NULL,
`Position` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`Number` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`Club` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`BT` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`BioLink` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`OtherBioLink` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`StartYear` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`EndYear` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`NeulionPlayerID` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`SeasonYear` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`GamesPlayed` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=124549 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci$$
If I rerun the LOAD DATA command, without changing anything, I get a different error, this time the number is 2 more than the previous time (124549 vs. 124547). Running it again skips to 124552.
Any ideas what is causing this duplicate error?
Thanks
The file was ill-formatted. Props to #Vapire for thinking of that.