does text variable effect queries speed ? mysql - mysql

I have table that contain 2 long text columns , when I fetch 100 rows it takes 5 seconds, this is long time right ?
maybe this happen because I have this 2 long text columns ?
here it is the table structure :
CREATE TABLE `tempBusiness2` (
`bussId` int(11) NOT NULL AUTO_INCREMENT,
`nameHe` varchar(200) COLLATE utf8_bin NOT NULL,
`nameAr` varchar(200) COLLATE utf8_bin NOT NULL,
`nameEn` varchar(200) COLLATE utf8_bin NOT NULL,
`addressHe` varchar(200) COLLATE utf8_bin NOT NULL,
`addressAr` varchar(200) COLLATE utf8_bin NOT NULL,
`addressEn` varchar(200) COLLATE utf8_bin NOT NULL,
`x` varchar(200) COLLATE utf8_bin NOT NULL,
`y` varchar(200) COLLATE utf8_bin NOT NULL,
`categoryId` int(11) NOT NULL,
`subcategoryId` int(11) NOT NULL,
`cityId` int(11) NOT NULL,
`cityName` varchar(200) COLLATE utf8_bin NOT NULL,
`phone` varchar(200) COLLATE utf8_bin NOT NULL,
`userDetails` text COLLATE utf8_bin NOT NULL,
`selectedIDFace` tinyint(4) NOT NULL,
`alluserDetails` longtext COLLATE utf8_bin NOT NULL,
`details` varchar(500) COLLATE utf8_bin NOT NULL,
`picture` varchar(200) COLLATE utf8_bin NOT NULL,
`imageUrl` varchar(200) COLLATE utf8_bin NOT NULL,
`fax` varchar(200) COLLATE utf8_bin NOT NULL,
`email` varchar(200) COLLATE utf8_bin NOT NULL,
`facebook` varchar(200) COLLATE utf8_bin NOT NULL,
`trash` tinyint(4) NOT NULL,
`subCategories` varchar(500) COLLATE utf8_bin NOT NULL,
`openHours` varchar(500) COLLATE utf8_bin NOT NULL,
`lastCheckedDuplications` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`bussStatus` tinyint(4) NOT NULL,
`approveStatus` tinyint(4) NOT NULL,
`steps` tinyint(4) NOT NULL DEFAULT '0',
`allDetails` longtext COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`bussId`),
KEY `bussId` (`allDetails`(5),`bussId`),
KEY `face` (`alluserDetails`(5),`userDetails`(5),`bussId`)
) ENGINE=InnoDB AUTO_INCREMENT=2515926 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
my query = SELECT * FROM tempBusiness2 LIMIT 100

If SELECT * FROM tempBusiness2 LIMIT 100 is really your query, then no INDEX is involved, and no INDEX would make it run any faster.
What that statement does:
Start at the beginning of the "data". (In InnoDB the PRIMARY KEY and the data are 'clustered' together. So, you are coincidentally starting with the first value of the PK.)
Read that row.
Move to the next row -- This is easy and efficient because the PK & Data are stored in a B+Tree structure.
Repeat until finished with the 100 or the table.
But... Because of lots of TEXT and VARCHAR fields, it is not that efficient. No more than 8K of a row is stored in the B+Tree mentioned above; the rest is sitting in extra blocks that are linked to. (I do not know how many extra blocks, but I fear it is more than one.) Each extra block is another disk hit.
Now, let's try to "count the disk hits". If you run this query a second time (and have a reasonably large innodb_buffer_pool_size), there would be any disk hits. Instead, let's focus on a "cold cache" and count the data blocks that are touched.
If there is only one row per block (as derived from the 8KB comment), that's 100 blocks to read. Plus the extra blocks -- hundred(s) more.
Ordinary disks can handle 100 reads per second. So that is a total of several seconds -- possibly the 5 that you experienced!.
Now... What can be done?
Don't do SELECT * unless you really want all the columns. By avoiding some of the bulky column, you can avoid some of the disk hits.
innodb_buffer_pool_size should be about 70% of available RAM.
"Vertical partitioning" may help. This is where you split off some columns into a 'parallel' table. This is handy if some subset of the columns are really a chunk of related stuff, and especially handy if it is "optional" in some sense. The JOIN to "put the data back together" is likely to be no worse than what you are experiencing now.
Do you really need (200) in all those fields?
You have what looks like a 3-element array of names and addresses. That might be better as another table with up to 3 rows per bussId.
On another note: If you run EXPLAIN SELECT ... on all your queries, you will probably find that the "prefix indexes" are never used:
KEY `bussId` (`allDetails`(5),`bussId`),
KEY `face` (`alluserDetails`(5),`userDetails`(5),`bussId`)
What were you hoping for in them? Consider FULLTEXT index(es), instead.
Why do you have both city_id and city_name in this table? It sounds like normalization gone berserk.

Yes, this kind of column take a lot of time, return this column only when you need.
And in addiction you will have to do a index on your table.

Related

I need to optimize a large data table in a mysql database

I have a table in a mysql DB with about 6 mil rows of data. Structure below. Most of my queries are searching for specific "customer" fields and display a value for each customer according to the value in column "value". The query searches the whole Table to match those customers specified in the query. This table started rather small but now it's gotten too big and my queries are taking quite some time to retrieve results. My questions is the following: If i create a separate table with just the customer field, along with an index, will that make my customer queries faster?
TABLE `data` (
`id` bigint(20) UNSIGNED NOT NULL,
`entry_id` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`date` date DEFAULT NULL,
`media_name` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`media_type` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`rate` decimal(8,2) DEFAULT NUCREATELL,
`value` decimal(8,2) DEFAULT NULL,
`page` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`type` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`sector` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`category` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`customer` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`product` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`description` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`image_id` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`city` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`address` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`supplier` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`time` time DEFAULT NULL,
`duration` time DEFAULT NULL,
`promoted_on` datetime NOT NULL,
`hasimage` tinyint(4) NOT NULL DEFAULT '0'
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
You want an index.
If you are searching for customers using in or = (the most common methods), then you can use a standard index on customer.
If you have more complex searches -- say using like with a leading wildcard -- then this does not work. A full text index might help. Or it might not, depending on the nature of the query and the data.
The "separate table" you're thinking about should be an index on your main table.
CREATE INDEX index_name
ON data(customer,value);
This will speed up the queries, and even prevent access to the table itself, at the cost of slightly slower INSERT and UPDATE operations.

Error while I'm trying to partitioning a table

Here is my posts table:
CREATE TABLE `posts` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`date` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`img` varchar(255) COLLATE utf8_croatian_ci NOT NULL,
`vid` varchar(255) COLLATE utf8_croatian_ci NOT NULL,
`title` varchar(255) COLLATE utf8_croatian_ci NOT NULL,
`subtitle` varchar(255) COLLATE utf8_croatian_ci NOT NULL,
`auth` varchar(54) COLLATE utf8_croatian_ci NOT NULL,
`story` longtext COLLATE utf8_croatian_ci NOT NULL,
`tags` varchar(255) COLLATE utf8_croatian_ci NOT NULL,
`status` varchar(100) COLLATE utf8_croatian_ci NOT NULL,
`moder` varchar(50) COLLATE utf8_croatian_ci NOT NULL,
`rec` varchar(50) COLLATE utf8_croatian_ci NOT NULL,
`pos` varchar(50) COLLATE utf8_croatian_ci NOT NULL,
`inde` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=117 DEFAULT CHARSET=utf8 COLLATE=utf8_croatian_ci
I want to make two partitions in order to improve query performances.
First partition should contain all non-archive rows.
Second partition - all archive rows.
ALTER TABLE posts
PARTITION BY LIST COLUMNS (status)
(
PARTITION P1 VALUES IN ('admin', 'moder', 'public', 'rec'),
PARTITION P2 VALUES IN ('archive')
);
phpmyadmin error:
Static analysis:
1 errors were found during analysis.
Unrecognized alter operation. (near "" at position 0)
MySQL said:
#1503 - A PRIMARY KEY must include all columns in the table's partitioning function
Any help?
What queries are you trying to speed up? Since the only index you currently have, WHERE id=... or WHERE id BETWEEN ... AND ... are the only queries that will be fast. And the partitioning you suggest will not help much for other queries.
You seem to have only dozens of rows; don't consider partitioning unless you expect to have at least a million rows.
status has only 5 values? Then make it ENUM('archive', 'admin', 'moder', 'public', 'rec') NOT NULL. That will take 1 byte instead of lots.
If you will be querying on date and/or status and/or auth, then let's talk about indexes, especially 'composite' indexes on such. And, to achieve the "archive" split you envision, put status as the first column in the index.

MySQL indexing optimization

I have following table - 'element'
CREATE TABLE `element` (
`eid` bigint(22) NOT NULL AUTO_INCREMENT,
`tag_name` varchar(45) COLLATE utf8_bin DEFAULT NULL,
`text` text COLLATE utf8_bin,
`depth` tinyint(2) DEFAULT NULL,
`classes` tinytext COLLATE utf8_bin,
`webarchiver_uniqueid` int(11) DEFAULT NULL,
`created` datetime DEFAULT NULL,
`updated` datetime DEFAULT NULL,
`rowstatus` char(1) COLLATE utf8_bin DEFAULT 'A',
PRIMARY KEY (`eid`)
) ENGINE=InnoDB AUTO_INCREMENT=12090 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
Column details and current index details are given above. Almost 90% of queries on this table are like:
select * from element
where tag_name = 'XXX'
and text = 'YYYY'
and depth = 20
and classes = 'ZZZZZ'
and rowstatus = 'A'
What would be the most optimal way to create index on this table? The table has around 60k rows.
Change classes from TINYTEXT to VARCHAR(255) (or some more reasonable size), then have
INDEX(tag_name, depth, classes)
with the columns in any order. I left out rowstatus because it smells like a column that is likely to change. (Anyway, a flag does not add much to an index.)
You can't include TEXT or BLOB columns in an index. And it is not worth it to do a 'prefix' index.
Since a PRIMARY KEY is a UNIQUE key, DROP INDEX eid_UNIQUE.
Is there some reason for picking "binary" / "utf8_bin" for all the character fields?

FULLTEXT SEARCH not working mysql

So i get an error when i try and use
SELECT views, keywords, title, url, thumbnail,
MATCH(keywords,title) AGAINST ('%$search_value%') AS relevance
FROM straight
WHERE MATCH (keywords,title) AGAINST ('%$search_value%')
ORDER BY relevance DESC
This is due to me not having FULLtext search enabled, but i cant seem to enable it. when i run the sql below:
ALTER TABLE straight ADD FULLTEXT(keywords, title)
i get this response:
MySQL returned an empty result set (i.e. zero rows). (Query took 3.8022 sec)
Then when trying to run the first query again i get the failed
#1191 - Can't find FULLTEXT index matching the column list
I can't tell why it's not registering. Any help would be great.
Thanks!
Edit:
My tabel:
CREATE TABLE `straight` (
 `url` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
 `title` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
 `keywords` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
 `production` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
 `categories` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
 `views` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
 `likes` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
 `length` varchar(80) COLLATE utf8_unicode_ci DEFAULT NULL,
 `thumbnail` varchar(200) COLLATE utf8_unicode_ci DEFAULT NULL,
 `date` varchar(12) COLLATE utf8_unicode_ci DEFAULT NULL,
 UNIQUE KEY `url` (`url`),
 FULLTEXT KEY `url_2` (`url`,`title`,`keywords`,`production`,
`categories`,`views`,`likes`,`length`,`thumbnail`,`date`
), ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
You need a FULLTEXT index that matches, exactly, the columns upon which you want to search. The FULLTEXT index you have has more columns than you need.
Try adding the one you mentioned.
ALTER TABLE straight ADD FULLTEXT(keywords, title)
Then look at the table definition and make sure it's there.

Filesize: CSV vs MySQL

I am trying to optimise my MySQL table structure for a 3GB CSV file. So far, I have managed to import 60% of the 19m+ rows, with a MySQL table size of 5.5GB. How could I optimise my table structure to reduce the database table size? (as I am running out of disk space!)
A sample line from the CSV file is
"{0C7ADEF5-878D-4066-B785-0000003ED74A}","163000","2003-02-21 00:00","UB5 4PJ","T","N","F","106","","READING ROAD","NORTHOLT","NORTHOLT","EALING","GREATER LONDON","A"
...and my database structure is:
(
`transaction_id` int(10) unsigned NOT NULL,
`reference` varchar(100) COLLATE utf32_unicode_ci NOT NULL,
`price` int(10) unsigned NOT NULL,
`sale_date` date COLLATE utf32_unicode_ci NOT NULL,
`postcode` varchar(8) COLLATE utf32_unicode_ci NOT NULL,
`type` varchar(1) COLLATE utf32_unicode_ci NOT NULL,
`new_build` varchar(1) COLLATE utf32_unicode_ci NOT NULL,
`tenure` varchar(1) COLLATE utf32_unicode_ci NOT NULL,
`property_number` varchar(10) COLLATE utf32_unicode_ci NOT NULL,
`property_name` varchar(100) COLLATE utf32_unicode_ci NOT NULL,
`street` varchar(100) COLLATE utf32_unicode_ci NOT NULL,
`area` varchar(100) COLLATE utf32_unicode_ci NOT NULL,
`city` varchar(100) COLLATE utf32_unicode_ci NOT NULL,
`county1` varchar(100) COLLATE utf32_unicode_ci NOT NULL,
`county2` varchar(100) COLLATE utf32_unicode_ci NOT NULL,
`unknown` varchar(1) COLLATE utf32_unicode_ci NOT NULL
)
Let's look at the size of the fields.
Your database structure consists primarily of varchars. Which, under normal circumstances, should be about one byte per character in the CSV file. With the overhead for length, these should be about the same size or a little bigger (two bytes for length versus one for the comma). You might throw in a 10% fudge factor for storing in the database.
Integers could go either way. They could be a single digit in the CSV file (two characters with the comma) or several digits. They will occupy 4 bytes in MySQL. Dates are probably smaller in MySQL than in the CSV file.
There is additional overhead for indexes, particularly if you have a fill factor that leaves space on a data page for additional storage. There is additional overhead for other stuff on a data page. But, your tables seem much larger than would be expected.
My guess is that your table is much bigger because of the utf32 considerations. If you have no really good reason for this, switch to utf8.
As a note: normally varchar(1) not null can be replaced by char(1) or char(1) not null. This saves you on the encoding of the length, which is a big savings for such small fields. This is also a savings for other fields If you know the postal code is 8 characters, then define it as char(8) rather than varchar(8).
Two suggestions:
(1) Your fields
You might ask MySQL itself regarding your data! Try
SELECT * FROM yourtable PROCEDURE ANALYSE;
and have a look at the result.
(2) Your charset
You're using utf32. If you don't really need it due to other parts of your tables/applications, go for utf8 instead.