This is my SQL Server query for pagination.
declare #PageNumber INT;
declare #PageSize INT;
set #PageNumber = 1600000;
set #PageSize = 10;
SELECT
q.Id, q.QuestionTitle,
SUBSTRING(q.QuestionContent,0,200) as QuestionContent,
q.QuestionVote, q.QuestionView,
q.Tag1, q.Tag2, q.Tag3, q.Tag4, q.Tag5,
q.CreatedDate,q.ModifiedDate
FROM
Question AS q
ORDER BY
q.Id DESC
OFFSET #PageNumber * #PageSize ROWS
FETCH NEXT #PageSize ROWS ONLY;
When page number is low there is no problem. But when page number is high, then query takes about 15-25 seconds.
Id column is a clustered index. There are 18 million records in the table.
How can I speed up this query?
Related
I am currently using mysql and I have to split the data into 10 groups.
For example, if the total count of data is 90, it should go like
1~9,
10~18,
19~27,
28~36,
37~45,
46~54,
55~63,
64~72,
73~81,
82~90.
if the total count of data is 100, it should go like
1~10,
11~20,
21~30,
31~40,
41~50,
51~60,
61~70,
71~80,
81~90,
91~100.
Can anyone give me a clue to split the data into 10 groups. I used rownum, but it did not work....
select total.row_num,
total.name,
total.reg,
total.id,
total.motspd
from(
select
(#row_num:=#row_num+1) AS row_num,
cg.group_name as name,
td.reg_date as reg,
td.car_id as id,
td.mcu_motspd as motspd
from
cartracker.tracker_data td
left join car c on (c.car_device_no = td.car_id)
left join car_group cg on (c.car_group_no = cg.car_group_no)
where cg.car_group_no = "1"
group by DATE_FORMAT(td.reg_date, "%Y%M%d%h%m")
)total
This is a result of a query, but it shows wrong row numbers.I want row_num goes from 0 to the end number of the data. but, in the picture, it starts from 44,713. can anyone help me to fix row num as it starts from 0 to the end number of the data.
attached image
I am going by the question in the title:
I am currently using mysql and I have to split the data into 10 groups.
That is exactly what the function ntile() does. So:
select t.*,
ntile(10) over (order by <whatever>) as tile
from t;
I have no idea what the query has to do with this question.
Haven't realize OP want with dynamic table BEFORE re-edit, this is not cleaver and require a lot of time to process though. switch a with OP table name and create rownumber first.
DECLARE #TotalNum INT;
DECLARE #Num INT;
DECLARE #NUm2 INT;
DECLARE #COUNTS INT;
SET #COUNTS = (select count(rownum) from a)/10
SET #TotalNum = 10
SET #Num =1
SET #Num2 =0
WHILE #Num <= #TotalNum
BEGIN
update a
set a.flag = #Num
where a.rownum >= (#COUNTS)*#NUM2+1 and a.rownum <= #NUM*(#COUNTS)
SET #Num = #NUM + 1
SET #Num2 = #NUM2 + 1
END
I have a requirement to write a SP where -
1. Select the rows count which match the given where clause.
2. Loop on the row counts and execute the delete query with limit n, till the time row count is not 0.
3. Decrease the row count value by n.
SP written -
BEGIN
DECLARE current_timestamp_millis BIGINT;
DECLARE RETENTION_DAYS SMALLINT;
DECLARE numRows BIGINT DEFAULT 0;
-- No. of days to retain data for
SET RETENTION_DAYS = 1;
-- Current epoch timestamp in millis
SET current_timestamp_millis = UNIX_TIMESTAMP(NOW())*1000;
-- SQL query to get the count of rows ,eligible to get deleted.
select count(*) as numRows from table1 where state = 2 AND end_time < (UNIX_TIMESTAMP(NOW())*1000 - (1 * 24 * 60 * 60 * 1000)) ;
-- Loop on the num of rows from above select query and delete the rows in chunks of 100
WHILE(numRows >=0)
DO
Insert into test_t values(current_time_millis);
Delete from table1 where end_time < ((UNIX_TIMESTAMP(NOW())*1000 - (1 * 24 * 60 * 60 * 1000))) limit 100;
SET numRows = numRows - 100;
DO SLEEP(2);
END WHILE;
END;
You are not setting up numrows try select..into numrows.
We have a SQL Server Scalar Function and part of the process is to take one of the input values and do the following
'inputvalue'
Create a table variable and populate with the following rows
inputvalue
inputvalu
inputval
inputva
inputv
input
inpu
inp
Then this table is joined to a query, ordered by len of the inputvalue desc and returns the top 1. The actual code is here
DECLARE #Result NVARCHAR(20);
DECLARE #tempDialCodes TABLE (tempDialCode NVARCHAR(20));
DECLARE #counter INT = LEN(#PhoneNumber);
WHILE #counter > 2
BEGIN
INSERT INTO #tempDialCodes(tempDialCode) VALUES(#PhoneNumber);
SET #PhoneNumber = SUBSTRING(#PhoneNumber, 1, #counter - 1);
SET #counter = #counter - 1;
END
SET #Result = (SELECT TOP 1 [DialCodeID]
FROM DialCodes dc JOIN #tempDialCodes s
ON dc.DialCode = s.tempDialCode
ORDER BY LEN(DialCode) DESC);
RETURN #Result
It works fine but I am asking if there is a way to replace the while loop and somehow joining to the inputvalue to get the same result. When I say it works fine, it's too dam slow but it does work.
I'm stumped on how to break up this string without using a loop and to a table variable but my warning light tells me this is not efficient for running against a table with a million rows.
Are you familiar with tally tables? The speed difference can be incredible. I try to replace every loop with a tally table if possible. The only time I haven't been able to so far is when calling a proc from within a cursor. If using this solution I would recommend a permanent dbo.Tally table with a sufficiently large size rather than recreating every time in the function. You will find other uses for it!
declare #PhoneNumber nvarchar(20) = 'inputvalue';
declare #tempDialCodes table (tempDialCode nvarchar(20));
--create and populate tally table if you don't already a permanent one
--arbitrary 1000 rows for demo...you should figure out if that is enough
--this a 1-based tally table - you will need to tweak if you make it 0-based
declare #Tally table (N int primary key);
insert #Tally
select top (1000) row_number() over (order by o1.object_id) from sys.columns o1, sys.columns o2 order by 1;
--select * from #Tally order by N;
insert #tempDialCodes
select substring(#PhoneNumber, 1, t.N)
from #Tally t
where t.N between 3 and len(#PhoneNumber)
order by t.N desc;
select *
from #tempDialCodes
order by len(tempDialCode) desc;
I'm new in sql, so i need your help! I need a stored procedure to get a certain number of articles with some offset and total number of them to make pagination...
is this a correct code or there is a better way? How much queries will be executed, 1 for ctid, 1 for total and 1 for content data??
DELIMITER $$
CREATE PROCEDURE `getArticles`(`offset` INT, `count` INT)
BEGIN
DECLARE ctid, total INT;
SET ctid = (SELECT id FROM content_types WHERE name='article');
SET total = (SELECT COUNT(*) FROM content WHERE content_type = ctid);
SELECT *, total FROM content
WHERE content_type = ctid
LIMIT count
OFFSET offset;
END $$
DELIMITER ;
You can try to leverage SQL_CALC_FOUND_ROWS option and FOUND_ROWS() function
SQL_CALC_FOUND_ROWS and FOUND_ROWS() can be useful in situations when
you want to restrict the number of rows that a query returns, but also
determine the number of rows in the full result set without running
the query again. An example is a Web script that presents a paged
display containing links to the pages that show other sections of a
search result. Using FOUND_ROWS() enables you to determine how many
other pages are needed for the rest of the result.
That being said your procedure might look like
DELIMITER $$
CREATE PROCEDURE get_articles(IN _offset INT, IN _count INT, OUT _total INT)
BEGIN
SELECT SQL_CALC_FOUND_ROWS *
FROM content c JOIN content_types t
ON c.content_type = t.id
WHERE t.name = 'article'
LIMIT _offset, _count;
SET _total = FOUND_ROWS();
END$$
DELIMITER ;
Sample usage:
SET #total = 0;
CALL get_articles(0, 3, #total);
SELECT #total;
Here is SQLFiddle demo
I guess your lookin a query like this. Try to do everything in the same query, if it doesn't get too complex.
SELECT c.*,
(SELECT COUNT(*) FROM content AS c1 WHERE c1.content_type = ct.id) AS total
FROM content AS c
INNER JOIN content_type AS ct ON c.content_type = ct.id
WHERE ct.name = 'article'
LIMIT offset, count;
I am currently running the following stored procedure. While it is a lot more efficient than my original procedure it is still taking an excessive amount of time. I'm not actually sure what the slow down is as the first 10k-30k records happened fast, but it has grown slower and slower as it gets further in. I'm expecting to update about 250k rows of about 1.7 million. Once this is complete I will then be doing something similar to insert records into each "Solar System".
To give you an example of the time this is taking. It has now been running for a little over 24 hours and it is only on iteration 786 of the 1716 it has to do. The reason for the changing limit on the selects is that there are 1000 possible rows per a sector in my table. I don't personally see any slow downs, but then I don't understand the inner workings of MySQL that well.
This work is being done on my local computer, no it is not slow, but there is always the possibility that there are changes that need to be done at the server level that would make these queries more efficient. If need be I can change the server settings so that is a possibility also. FYI I'm using the stock configuration from MySQL on a Windows 7.
DECLARE CurrentOffset int; -- Current offset limit to only deal with one
DECLARE CurrentOffsetMultiplier int;
DECLARE RandRow int; -- Random Row to make a Solar System with
DECLARE CheckSystemExists int; -- Used to insure RandRow is not already a Solar System Row
DECLARE TotalSystemLoops int; -- Total number of loops so each Galaxy gets it's systems.
DECLARE RandomSolarSystemCount int; -- This is the number of Solar Systems that will be in each Galaxy;
DECLARE UpdateSolarCount int;
DECLARE NumberOfOffsets int;
SET CurrentOffsetMultiplier = 0;
SET NumberOfOffsets = 1716;
SET CurrentOffset = 0;
OffsetLoop: LOOP
SET UpdateSolarCount = 0;
/*Sets the amount of Solary Systems going in a Galaxy*/
CheckRandomSolarSystemCount: LOOP
SET RandomSolarSystemCount = FLOOR(125 + RAND() * (175 - 125) + 1);
IF RandomSolarSystemCount >= 125 THEN
IF RandomSolarSystemCount <= 175 THEN
LEAVE CheckRandomSolarSystemCount;
END IF;
END IF;
END LOOP;
UpdateGalaxyWithSolarSystems: LOOP
SET UpdateSolarCount = UpdateSolarCount + 1;
IF UpdateSolarCount > RandomSolarSystemCount THEN
LEAVE UpdateGalaxyWithSolarSystems;
END IF;
/*Sets RandRow and CheckSystemExists*/
CheckExistsLoop: Loop
SET RandRow = FLOOR(0 + RAND() * (1000)+ 1);
SET CheckSystemExists = (SELECT COUNT(*)
FROM
(SELECT * FROM
(SELECT * FROM galaxies2 LIMIT CurrentOffset, 1000) AS LimitedTable
LIMIT RandRow ,1) AS RandTable
WHERE SolarSystemName IS NULL);
IF CheckSystemExists THEN
LEAVE CheckExistsLoop;
END IF;
END LOOP;
/*Updates the tables SolarSystemName column with a default system name*/
UPDATE galaxies2
SET SolarSystemName = CONCAT("Solar System ", RandRow)
WHERE galaxies2.idGalaxy =
(SELECT LimitedTable.idGalaxy AS GalaxyID FROM
(SELECT galaxies2.idGalaxy FROM galaxies2 LIMIT CurrentOffset, 1000) AS LimitedTable
LIMIT RandRow ,1)
;
END LOOP;
SET CurrentOffsetMultiplier = CurrentOffsetMultiplier + 1;
SET CurrentOffset = CurrentOffsetMultiplier * 1000;
IF CurrentOffsetMultiplier = 1717 THEN
LEAVE OffsetLoop;
END IF;
END LOOP;
It's getting slower and slower because you are "walking" through the galaxies2 table.
SELECT * FROM galaxies2 LIMIT CurrentOffset, 1000
As the CurrentOffset value increases, MySQL has to "walk" through more and more records to get to the starting point. You may actually be able to get a speed boost by specifying an ORDER BY on the primary key. You would want to have an ORDER BY anyway since MySQL just reads records randomly if no order is specified. It does not read the records in any particular order so you could (though unlikely) get the same set of records in different offsets.
It would be better to specify a range on the auto increment field. Assuming you have one. Then the first and last queries should perform about the same. It's not as ideal since there could be gaps from deleted records.
SELECT * FROM galaxies2 WHERE auto_incr_field BETWEEN CurrentOffset AND CurrentOffset+1000