We have a mysql data table which has 10 years old. For each query, it will search data from 2005 to 2015. However, the most useful data for me will be the recently days or weeks. Is that possible using some command to ask the query search from the end of table.
For example,
select *from Portions where Date="20151126" limit 5;
This simply query takes ages to run.
First off, check the structure of your database. It may be that you've saved dates as a VARCHAR whereas they should be being saved as DATE.
The next important part is ensuring you have a primary key on your table which should be of INT type and is your primary key. You should also make this auto increment so that it keeps unique IDs of your table. Here's the structure:
--
-- Table structure for table `portions`
--
CREATE TABLE IF NOT EXISTS `portions` (
`id` int(11) NOT NULL,
`date` date NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
--
-- Indexes for table `portions`
--
ALTER TABLE `portions`
ADD PRIMARY KEY (`id`);
--
-- AUTO_INCREMENT for table `portions`
--
ALTER TABLE `portions`
MODIFY `id` int(11) NOT NULL AUTO_INCREMENT;
Ensure this structure is correct and query using the correct syntax:
SELECT * FROM `portions` WHERE `date` = '2015-11-26' LIMIT 5;
Now, you should find your queries are much faster.
You should consider using materialized view in this scenario. See here if you want to use it.
Related
more information added 2022-09-05.
I have a problem with a sql query in a mariadb environment. I have a query that runs a number of tests in the database to identify unprocessed/unverified records, or records with contradicting data. The record identifiers that need to be checked are collected in one field (named fam_list) using a group_concat. The use of group_concat properly collects all record identifiers if no group by function is used. Issue is when group by is added in the query. I want to group the record identifiers by person responsible and project code. After adding the group by instruction, I retrieve several rows but the field that should collect all record identifiers is empty in all rows.
Additionally surprising is the fact that the column that counts the number of family identifiers (named fam_count), which is also an aggregate function, works correct in all attempts.
The query functions ok in my development machine, but fails in my production machine.
extra information 2022-09-05
Removing parts of the query suggests that the origin of the problem lies in the combination of the table fam_proj_link and the view Team_curr_members.
Building up from left to right continues to produce the expected result until Team_curr_members is added in the join. From that moment onwards the group_concat column is empty. The table fam_proj_link can be joined with Team_curr_members directly. A query running in the table fam_proj_link joined to the view Team_curr_members still produces the empty group_concat column.
The simplified query that still fails is:
select 'uninsp' as checkname, l.proj_id, count(l.pb_fam_id), group_concat(l.pb_fam_id)
from PLS.fam_proj_link l left join Projects_2012.Team_curr_members t on(l.proj_id=t.proj_id and t.func_id=1)
where rank_id=-1
group by l.proj_id
I have tried modifying the join condition by doing a straight join, but still an empty group_concat field. I have tried redefyning the join to 'using(proj_id), but still an empty group_concat field.
In none of these cases does the mariadb server produce any error information and all the queries run as expected in the development server.
I checked the site for other usage of the group_concat function and found irregular behaviour in more pages. Including pages that used to work correctly until shortly before my holidays.
I have run optimize tables on the entire database and restarted the mariadb server package, but without improvement.
Could it be that there are damages in some database related files and if so, where do I have to look and what do I have to look for?
end of extra added information
Does anybody have any idea what could be the reason or what I can try to find out what goes wrong?
Query (for one of the tests):
This works in development and production:
select 'uninsp' as checkname, coalesce(user_id,0) as user_id, coalesce(firstsur,' none') as firstsur, l.proj_id, label, stat_name, stat_class, count(distinct pb_fam_id) as fam_count, group_concat(distinct pb_fam_id) as fam_list
from PLS.fam_proj_link l join Projects_2012.Proj_list_2018 p using(proj_id) join Projects_2012.Proj_status_reflist using(stat_id)
left join Projects_2012.Team_curr_members t on(p.proj_id=t.proj_id and t.func_id=1) left join UserData.people_view using(user_id)
where ref_code and stat_name!='cancelled' and rank_id=-1;
This works in development only, not in production:
select 'uninsp' as checkname, coalesce(user_id,0) as user_id, coalesce(firstsur,' none') as firstsur, l.proj_id, label, stat_name, stat_class, count(distinct pb_fam_id) as fam_count, group_concat(distinct pb_fam_id) as fam_list
from PLS.fam_proj_link l join Projects_2012.Proj_list_2018 p using(proj_id) join Projects_2012.Proj_status_reflist using(stat_id)
left join Projects_2012.Team_curr_members t on(p.proj_id=t.proj_id and t.func_id=1) left join UserData.people_view using(user_id)
where ref_code and stat_name!='cancelled' and rank_id=-1
group by user_id, proj_id;
Production machine:
Debian linux running mariadb 10.1.41
Development machine:
Manjaro linux running mariadb 10.8.3
Of course, I can solve the issue by abandoning the group_concat in mariadb and store the record identifiers in arrays in php and implode the array before generating the output, but I assume that concatenation in mariadb is faster than concatenation via arrays in php.
Relevant table definitions:
fam_proj_link
--
-- Table structure for table `fam_proj_link`
--
CREATE TABLE `fam_proj_link` (
`proj_id` smallint(5) UNSIGNED NOT NULL,
`pb_fam_id` int(10) UNSIGNED NOT NULL,
`originator` tinyint(1) NOT NULL DEFAULT '0',
`rank_id` tinyint(3) NOT NULL DEFAULT '-1',
`factsheet` tinyint(1) NOT NULL DEFAULT '0',
`cust_title` varchar(255) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
--
-- RELATIONS FOR TABLE `fam_proj_link`:
-- `rank_id`
-- `def_ranks` -> `rank_id`
-- `pb_fam_id`
-- `pb_fams` -> `pb_fam_id`
-- `proj_id`
-- `Projects` -> `proj_id`
--
--
-- Indexes for dumped tables
--
--
-- Indexes for table `fam_proj_link`
--
ALTER TABLE `fam_proj_link`
ADD PRIMARY KEY (`proj_id`,`pb_fam_id`) USING BTREE,
ADD KEY `originator` (`originator`),
ADD KEY `rank_id` (`rank_id`),
ADD KEY `pb_fam_id` (`pb_fam_id`),
ADD KEY `factsheet` (`factsheet`);
--
-- Constraints for dumped tables
--
--
-- Constraints for table `fam_proj_link`
--
ALTER TABLE `fam_proj_link`
ADD CONSTRAINT `fam_proj_link_ibfk_3` FOREIGN KEY (`rank_id`) REFERENCES `def_ranks` (`rank_id`) ON DELETE CASCADE ON UPDATE CASCADE,
ADD CONSTRAINT `fam_proj_link_ibfk_4` FOREIGN KEY (`pb_fam_id`) REFERENCES `pb_fams` (`pb_fam_id`) ON DELETE CASCADE ON UPDATE CASCADE,
ADD CONSTRAINT `fam_proj_link_ibfk_5` FOREIGN KEY (`proj_id`) REFERENCES `Projects_2012`.`Projects` (`proj_id`) ON DELETE CASCADE ON UPDATE CASCADE;
Proj_list_2018 is a view with the below stand-in structure:
all fields are indexed in the parent tables, except for the label which is a combination of a group_concat and a normal concat over fields in three different tables.
--
-- Stand-in structure for view `Proj_list_2018`
-- (See below for the actual view)
--
CREATE TABLE `Proj_list_2018` (
`proj_id` smallint(6) unsigned
,`priority` tinyint(4)
,`stat_id` tinyint(4) unsigned
,`ref_code` bigint(20)
,`label` mediumtext
);
Proj_status_reflist:
--
-- Table structure for table `Proj_status_reflist`
--
CREATE TABLE `Proj_status_reflist` (
`stat_id` tinyint(3) UNSIGNED NOT NULL,
`stat_group_id` tinyint(3) NOT NULL DEFAULT '1',
`stat_name` varchar(20) NOT NULL DEFAULT '-',
`stat_class` varchar(25) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
--
-- RELATIONS FOR TABLE `Proj_status_reflist`:
-- `stat_group_id`
-- `Proj_status_groups` -> `stat_group_id`
--
--
-- Indexes for dumped tables
--
--
-- Indexes for table `Proj_status_reflist`
--
ALTER TABLE `Proj_status_reflist`
ADD PRIMARY KEY (`stat_id`),
ADD UNIQUE KEY `stat_name_UNIQUE` (`stat_name`),
ADD KEY `stat_group_id` (`stat_group_id`);
--
-- AUTO_INCREMENT for dumped tables
--
--
-- AUTO_INCREMENT for table `Proj_status_reflist`
--
ALTER TABLE `Proj_status_reflist`
MODIFY `stat_id` tinyint(3) UNSIGNED NOT NULL AUTO_INCREMENT;
--
-- Constraints for dumped tables
--
--
-- Constraints for table `Proj_status_reflist`
--
ALTER TABLE `Proj_status_reflist`
ADD CONSTRAINT `Proj_status_reflist_ibfk_1` FOREIGN KEY (`stat_group_id`) REFERENCES `Proj_status_groups` (`stat_group_id`) ON UPDATE CASCADE;
Team_curr_members is a view with the below stand-in structure:
all fields are indexed in the parent table.
--
-- Stand-in structure for view `Team_curr_members`
-- (See below for the actual view)
--
CREATE TABLE `Team_curr_members` (
`proj_id` smallint(5) unsigned
,`func_id` tinyint(3) unsigned
,`user_id` smallint(5) unsigned
);
people_view is a view with the below stand-in structure:
--
-- Stand-in structure for view `people_view`
-- (See below for the actual view)
--
CREATE TABLE `people_view` (
`user_id` smallint(5) unsigned
,`synth_empl` int(1)
,`firstsur` varchar(77)
,`surfirst` varchar(78)
);
After trying many things, I noticed that the problem appeared in more group_concat queries, many of them had been running without problems for several years. Rebuilding the indexes of the database tables did not solve the issue. Since the software was not altered during the last three years, I concluded that the database engine might be damaged.
I asked IT to create a new virtual server. After building a complete new server environment and migration of the databases and web interface to the new server all problems were solved.
Apparently the problem was not in the queries, but in the server enviroment. I have not discovered which part of the old server environment caused the actual issue, but possibly one or more mariadb/php related files were damaged.
Thanks to those that took the effort to come with suggestions.
MySQL seems to be very slow for updates.
A simple update statement is taking more time than MS SQL for same update call.
Ex:
UPDATE ValuesTbl SET value1 = #value1,
value2 = #value2
WHERE co_id = #co_id
AND sel_date = #sel_date
I have changed some config settings as below
innodb_flush_log_at_trx_commit=2
innodb_buffer_pool_size=10G
innodb_log_file_size=2G
log-bin="foo-bin"
skip-log-bin
This is the create table query
CREATE TABLE `valuestbl` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`sel_date` datetime NOT NULL,
`co_id` int(11) NOT NULL,
`value1` decimal(10,2) NOT NULL,
`value2` decimal(10,2) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=21621889 DEFAULT CHARSET=latin1;
MySQL version: 8.0 on Windows
The update query takes longer time to update when compared to MS SQL, anything else I need to do to make it faster?
There are no indices, the ValuesTbl tables has a PK, not using for anything. the id column is a Primary key from another table, the sel_date is a date field and 2 decimal columns
If there are no indexes on ValuesTbl then the update has to scan the entire table which will be slow if the table is large. No amount of server tuning will fix this.
A simple update statement is taking more time than MS SQL for same update call.
The MS SQL server probably has an index on either co_id or sel_date. Or it has fewer rows in the table.
You need to add indexes, like the index of a book, so the database doesn't have to search the whole table. At minimum an index on co_id will vastly help performance. If there are many columns with different sel_date per ID, a compound index on (co_id, sel_date) would help further.
See Use The Index, Luke for an extensive tutorial on indexes.
I am using mysql 5.7.18.
I tried:
CREATE TABLE t1 (
`UID` int(10) auto_increment,
day int(10),
PRIMARY KEY (`UID`)
) ENGINE=MyISAM AUTO_INCREMENT=0;
When i looked at the table info in workbench, it shows the autoinc value is 1.
The weird thing is that I managed to set it to 0 in two other tables!
I also tried:
alter table t1 auto_increment=0;
I did this before any data was written into the table.
I'm at a bit of a loss.
Any recommendations?
Update:
CREATE TABLE `sname`.`z` (
`Uid` INT NOT NULL DEFAULT -1,
PRIMARY KEY (`Uid`))
ENGINE = MyISAM
AUTO_INCREMENT = 0;
This creates the table with the autoincrement start at 0
I do not understand why the top example does not* do what I want it to and the bottom one does.
After even more research, it would appear that you cannot have 0 for an initial value in an autoincrement field.
I'm trying to keep a record of the earnings of the users of my website and I'm stuck at which of the following designs is best regarding performance and overall usability:
• First way:
In this way a single database will be created containing a table for each year. Each table will have 13 columns, the user ID and the 12 months. The value for each field will be a stringified array with the values of all the days of the month, like so: [12.5, 28.3, 9.75, ...].
Code:
-- Create a database to keep record of the earnings of all users.
CREATE DATABASE IF NOT EXISTS `Earnings_Record`;
-- Create a table for each year containing 13 columns, the user ID and the 12 months.
CREATE TABLE IF NOT EXISTS `Earnings_Record`.`Earnings_2017` (
`ID` INT(11) NOT NULL,
`January` VARCHAR(250) NOT NULL,
`February` VARCHAR(250) NOT NULL,
...
`December` VARCHAR(250) NOT NULL,
PRIMARY KEY (`ID`)
) ENGINE = InnoDB CHARACTER SET utf8 COLLATE utf8_general_ci;
• Second way:
In this way multiple databases will be created, one for each year, containing a table for each month. Each table will have 28-31 + 1 columns, the user ID and the 28-31 days. The value for each field will be a decimal, like so: 12.5.
Code:
-- Create a database to keep record of the earnings of all users for 2017.
CREATE DATABASE IF NOT EXISTS `Earnings_2017`;
-- Create a table for each month with 28-31 + 1 columns, the user ID and the 28-31 days.
CREATE TABLE IF NOT EXISTS `Earnings_2017`.`January` (
`ID` INT(11) NOT NULL,
`Day 1` DECIMAL(10, 2) NOT NULL,
`Day 2` DECIMAL(10, 2) NOT NULL,
...
`Day 31` DECIMAL(10, 2) NOT NULL,
PRIMARY KEY (`ID`)
) ENGINE = InnoDB CHARACTER SET utf8 COLLATE utf8_general_ci;
Since the website will hopefully be running for 5-10 years, which is the best way to design this when it comes to overall performance and long-term maintainability?
(One thing to keep in mind is that the earnings of each user will be updated multiple times every day for active users)
Third way:
Create a single database. Create a single table for each entity:
CREATE DATABASE IF NOT EXISTS Earnings_Record;
CREATE TABLE IF NOT EXISTS Earnings_Record.Users (
UsersId INT AUTO_INCREMENT PRIMARY KEY,
. . .
);
CREATE TABLE IF NOT EXISTS Earnings_Record.Earnings (
EarningsID INT AUTO_INCREMENT PRIMARY KEY,
UserId INT NOT NULL,
EarningsDate DATE,
Amount DECIMAL(10, 2) -- or whatever
CONSTRAINT fk_Earnings_UserId FOREIGN KEY (UserId) REFERENCES Users(UserId)
) ;
Simple. Easy-to-query. Just what you need. This is the SQL way to represent data.
It's hard to answer this - but any solution that requires multiple databases or tables is probably not maintainable, or scalable, or fast.
I really don't understand your business domain - you say you want to maintain earnings per user, but your tables don't have any reference to a user.
To design the database, it would really help to understand typical queries - do you want to find out total earnings for a period? Do you want to find days with high and/or low earnings? Do you want to aggregate earnings over a group of dates, e.g. "every monday"?
I'd start with:
table earnings
-------------
earnings_date (date) (pk)
earnings_amount (decimal 10,2)
Multiple databases -- NO
Splaying the months across 12 columns -- NO
Stringifying -- Only if you never need MySQL to filter on the data or do arithmetic on the data.
All of these are discussed in various ways in this forum.
There is no problem having thousands, even millions, of rows in a table. The other approaches are headache-causers.
I am using this query:
insert ignore into CategoryLinks (article_id,category_id) values ('$art_id','$id')
$art_id and $id are two integers.
CategoryLinks has one unique index (both columns).
Unfortunately the query is very slow sometimes and sometimes it's fast and I don't know why!
The table has around 100,000 data records. The query needs between 1*10^(-5) seconds and over two seconds.
And it's strange that PHPMyAdmin shows: Index usage: 0B.
show create table CategoryLinks
CREATE TABLE `CategoryLinks` (
`article_id` int(10) NOT NULL,
`category_id` int(7) NOT NULL,
UNIQUE KEY `Unique` (`article_id`,`category_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
Btw: Is it possible to check whether the index is used?
MyISAM is faster!