How I can in MySQL fetch any row by index from result set as it possible with arrays or collections in most programming languages ?
array[index]
Or:
collection.getElementByIndex(index)
Update:
I have a result set of dates, me need to check whether the 90 days between each date
You have two alternatives:
Use a a sub-select.
Use the ability for MySQL to iterate over the returned rows.
First alternative looks like:
SELECT BIT_AND(IFNULL(DATEDIFF((SELECT dt FROM foo WHERE dt > a.dt ORDER BY dt LIMIT 1), a.dt) >= 90, 1)) AS all_larger
FROM foo a;
Update: To handle a table where a date is duplicated, it is necessary to add a second sub-select to see if there are duplicates for the date, as follows:
SELECT BIT_AND(larger && ! duplicates) AS all_larger
FROM (SELECT a.dt
, IFNULL(DATEDIFF((SELECT dt FROM foo WHERE dt > a.dt ORDER BY dt LIMIT 1), a.dt) >= 90, 1) AS larger
, (SELECT COUNT(*) FROM foo WHERE dt = a.dt) > 1 AS duplicates
FROM foo a) AS x;
Second alternative looks like:
SET #prev = NULL;
SELECT BIT_AND(a.larger) AS all_larger
FROM (SELECT IFNULL(DATEDIFF(dt, #prev) >= 90, 1) AS larger
, #prev := dt
FROM foo ORDER BY dt) a;
Both give the following result set when run on a table where the difference between the dates are more than 90 days:
+------------+
| all_larger |
+------------+
| 1 |
+------------+
The second one is probably faster, but I haven't measured on larger sets.
Intrinsically you cannot. A relational database doesn't preserve record order (or at least you can't rely on it, even if it temporarily stores record order). In this way it acts more like a hashmap or List than an array.
However if you want, you can add a field in the table - let's call it RowNum - that stores a row number, and you can query on that.
select * from Table where RowNum = %index%;
Related
I have a table which contains thousands of rows and I would like to calculate the 90th percentile for one of the fields, called 'round'.
For example, select the value of round which is at the 90th percentile.
I don't see a straightforward way to do this in MySQL.
Can somebody provide some suggestions as to how I may start this sort of calculation?
Thank you!
First, lets assume that you have a table with a value column. You want to get the row with 95th percentile value. In other words, you are looking for a value that is bigger than 95 percent of all values.
Here is a simple answer:
SELECT * FROM
(SELECT t.*, #row_num :=#row_num + 1 AS row_num FROM YOUR_TABLE t,
(SELECT #row_num:=0) counter ORDER BY YOUR_VALUE_COLUMN)
temp WHERE temp.row_num = ROUND (.95* #row_num);
Compare solutions:
Number of seconds it took on my server to get 99 percentile of 1.3 million rows:
LIMIT x,y with index and no where: 0.01 seconds
LIMIT x,y with no where: 0.7 seconds
LIMIT x,y with where: 2.3 seconds
Full scan with no where: 1.6 seconds
Full scan with where: 5.7 seconds
Fastest solution for large tables using LIMIT x,y ():
Get count of values: SELECT COUNT(*) AS cnt FROM t
Get nth value, where n = (cnt - 1) * (1 - 0.95) : SELECT k FROM t ORDER BY k DESC LIMIT n,1
This solution requires two queries, because mysql does not support specifying variables in LIMIT clause, except for stored procedures (can be optimized with stored procedure). Usually additional query overhead is very low
This solution can be further optimized if you add index to k column and do not use complex where clauses (like 0.01 second for table with 1 million rows, because sorting is not needed).
Implementation example in PHP (can calculate percentile not only of columns, but also of expressions):
function get_percentile($table, $where, $expr, $percentile) {
if ($where) $subq = "WHERE $where";
else $subq = "";
$r = query("SELECT COUNT(*) AS cnt FROM $table $subq");
$w = mysql_fetch_assoc($r);
$num = abs(round(($w['cnt'] - 1) * (100 - $percentile) / 100.0));
$q = "SELECT ($expr) AS prcres FROM $table $subq ORDER BY ($expr) DESC LIMIT $num,1";
$r = query($q);
if (!mysql_num_rows($r)) return null;
$w = mysql_fetch_assoc($r);
return $w['prcres'];
}
// Usage example
$time = get_percentile(
"state", // table
"service='Time' AND cnt>0 AND total>0", // some filter
"total/cnt", // expression to evaluate
80); // percentile
The SQL standard supports the PERCENTILE_DISC and PERCENTILE_CONT inverse distribution functions for precisely this job. Implementations are available in at least Oracle, PostgreSQL, SQL Server, Teradata. Unfortunately not in MySQL. But you can emulate PERCENTILE_DISC in MySQL 8 as follows:
SELECT DISTINCT first_value(my_column) OVER (
ORDER BY CASE WHEN p <= 0.9 THEN p END DESC /* NULLS LAST */
) x,
FROM (
SELECT
my_column,
percent_rank() OVER (ORDER BY my_column) p,
FROM my_table
) t;
This calculates the PERCENT_RANK for each row given your my_column ordering, and then finds the last row for which the percent rank is less or equal to the 0.9 percentile.
This only works on MySQL 8+, which has window function support.
I was trying to solve this for quite some time and then I found the following answer. Honestly brilliant. Also quite fast even for big tables (the table where I used it contained approx 5 mil records and needed a couple of seconds).
SELECT
CAST(SUBSTRING_INDEX(SUBSTRING_INDEX( GROUP_CONCAT(field_name ORDER BY
field_name SEPARATOR ','), ',', 95/100 * COUNT(*) + 1), ',', -1) AS DECIMAL)
AS 95th Per
FROM table_name;
As you can imagine just replace table_name and field_name with your table's and column's names.
For further information check Roland Bouman's original post
In MySQL 8 there is the ntile window function you can use:
SELECT SomeTable.ID, SomeTable.Round
FROM SomeTable
JOIN (
SELECT SomeTable, (NTILE(100) OVER w) AS Percentile
FROM SomeTable
WINDOW w AS (ORDER BY Round)
) AS SomeTablePercentile ON SomeTable.ID = SomeTablePercentile.ID
WHERE Percentile = 90
LIMIT 1
https://dev.mysql.com/doc/refman/8.0/en/window-function-descriptions.html#function_ntile
http://www.artfulsoftware.com/infotree/queries.php#68
SELECT
a.film_id ,
ROUND( 100.0 * ( SELECT COUNT(*) FROM film AS b WHERE b.length <= a.length ) / total.cnt, 1 )
AS percentile
FROM film a
CROSS JOIN (
SELECT COUNT(*) AS cnt
FROM film
) AS total
ORDER BY percentile DESC;
This can be slow for very large tables
As pert Tony_Pets answer, but as I noted on a similar question: I had to change the calculation slightly, for example the 90th percentile - "90/100 * COUNT(*) + 0.5" instead of "90/100 * COUNT(*) + 1". Sometimes it was skipping two values past the percentile point in the ordered list, instead of picking the next higher value for the percentile. Maybe the way integer rounding works in mysql.
ie:
.... SUBSTRING_INDEX(SUBSTRING_INDEX( GROUP_CONCAT(fieldValue ORDER BY fieldValue SEPARATOR ','), ',', 90/100 * COUNT(*) + 0.5), ',', -1) as 90thPercentile ....
The most common definition of a percentile is a number where a certain percentage of scores fall below that number. You might know that you scored 67 out of 90 on a test. But that figure has no real meaning unless you know what percentile you fall into. If you know that your score is in the 95th percentile, that means you scored better than 95% of people who took the test.
This solution works also with the older MySQL 5.7.
SELECT *, #row_num as numRows, 100 - (row_num * 100/(#row_num + 1)) as percentile
FROM (
select *, #row_num := #row_num + 1 AS row_num
from (
SELECT t.subject, pt.score, p.name
FROM test t, person_test pt, person p, (
SELECT #row_num := 0
) counter
where t.id=pt.test_id
and p.id=pt.person_id
ORDER BY score desc
) temp
) temp2
-- optional: filter on a minimal percentile (uncomment below)
-- having percentile >= 80
An alternative solution that works in MySQL 8: generate a histogram of your data:
ANALYZE TABLE my_table UPDATE HISTOGRAM ON my_column WITH 100 BUCKETS;
And then just select the 95th record from information_schema.column_statistics:
SELECT v,c FROM information_schema.column_statistics, JSON_TABLE(histogram->'$.buckets',
'$[*]' COLUMNS(v VARCHAR(60) PATH '$[0]', c double PATH '$[1]')) hist
WHERE column_name='my_column' LIMIT 95,1
And voila! You will still need to decide whether you take the lower or upper limit of the percentile, or perhaps take an average - but that is a small task now. Most importantly - this is very quick, once the histogram object is built.
Credit for this solution: lefred's blog.
Given the following table, I want to select the value of each ID at exactly 00:00:00. When there's an entry at this exact time, return it, otherwise calculate it with linear interpolation (an imaginary graph line between the nearest values before and after 00:00:00). If there's no value after the given time yet, return the last value, or use linear interpolation from the last two points.
ID|Timestamp|Value
1|2015-01-01 23:00:00|90
1|2015-01-02 01:00:00|110
2|2015-01-01 23:00:00|210
2|2015-01-02 01:00:00|190
3|2015-01-02 00:00:00|50
4|2015-01-01 23:00:00|100
5|2015-01-01 22:00:00|80
5|2015-01-01 23:00:00|90
Result:
ID|Value
1|100
2|200
3|50
4|100
5|100
Is this possible with MySQL only and how?
First, let's split it into single entries like #3 versus double entries (the rest):
( SELECT ID, value FROM tbl GROUP BY ID HAVING COUNT(*) = 1 )
UNION ALL
( ... ) -- This needs interpolation code, below
ORDER BY ID;
To separate the pair of rows is tricky since there is no good way to do "groupwise-max". So, instead, I will work with #variables and walk through the table in order.
To round to the nearest midnight might be ROUND(... / 86400) * 86400 . The potential problem is the time_zone you are in. I don't feel like fixing that.
SELECT ID, val FROM (
SELECT ID,
IF(ID != #prevID, '1st', '2nd') AS picker, -- See WHERE filter, below
#ts = timestamp, -- Need an INT here, not sure that is what I have
#dts = #ts - #prev_ts,
#dval = value - #prev_val,
#midnight := ROUND(#ts / 86400) * 86400, -- DST issues?
value + (#midnight - #ts) * (#dval / #dts) AS val, -- interpolate
#prev_id = ID,
#prev_ts = #ts,
#prev_val = value
FROM ( SELECT #prevID := 0 ) AS Initialize
JOIN tbl
ORDER BY ID, timestamp
) AS x
WHERE picker = '2nd'
ORDER BY ID
Put that monster as the second part of the UNION.
Here is my data structure :
name value date_received
foo 100 2013-09-19 10:00:00
bar 200 2013-09-19 10:00:00
foo 100 2013-09-19 10:05:00 //no change
bar 200 2013-09-19 10:05:00 //no change
foo 110 2013-09-19 10:08:00 // foo changed
bar 200 2013-09-19 10:08:00 // no change
......
Question:
I want a query (mysql) which can do something like:
select date_received where anyOf(foo, bar) changed from the previous
specified value in the past N hours.
There could be other names in the table but we are only interested in foo and bar.
Any pointers. To me it looks like we'll need a self join - but don't know how.
EDIT: looks like the below query is just a good starting point.
select date_received from (SELECT DISTINCT name, value from data) a
INNER JOIN (select DISTINCT name, value, date_received from data)b
on (a.name=b.name and a.value=b.value)
update Looks like below query works - easier than I thought it would be.
SELECT DISTINCT a.tr FROM (
SELECT name, value, MAX(date_received) dr from data
where date_received > now() - INTERVAL 2 hour
GROUP BY name, value order by dr desc)a;
I do not see how your edited query solves the problem. Where does the "last N hours" come in, for instance?
I would approach this by looking at the previous value, then using logic around the datetime constraints and value changes to see if there has been a change. Your question is ambiguous: Are you looking for changes only in the last N hours? Are you looking for a change from the last value before N hours? What happens if the value changes back?
All of these, though, could be answered by having the previous value and previous time on each row. Here is an example of how to get this:
select t.*,
(select t.date_received
from t t2
where t2.date_received < t.date_received and
t2.name = t.name
order by t2.date_received desc
limit 1
) as prev_date_received,
(select t.value
from t t2
where t2.date_received < t.date_received and
t2.name = t.name
order by t2.date_received desc
limit 1
) as prev_value
from t
having <your logic goes here for the date time and changes you care about>;
This is using the having clause instead of a subquery, just out of convenience (this is not supported by other databases).
For instance, if you want any changes in the last N hours:
having date_received > now() - interval N hour and prev_value <> value
I'm having trouble with calculating the median of a list of values, not the average.
I found this article
Simple way to calculate median with MySQL
It has a reference to the following query which I don't understand properly.
SELECT x.val from data x, data y
GROUP BY x.val
HAVING SUM(SIGN(1-SIGN(y.val-x.val))) = (COUNT(*)+1)/2
If I have a time column and I want to calculate the median value, what do the x and y columns refer to?
I propose a faster way.
Get the row count:
SELECT CEIL(COUNT(*)/2) FROM data;
Then take the middle value in a sorted subquery:
SELECT max(val) FROM (SELECT val FROM data ORDER BY val limit #middlevalue) x;
I tested this with a 5x10e6 dataset of random numbers and it will find the median in under 10 seconds.
This will find an arbitrary percentile by replacing the COUNT(*)/2 with COUNT(*)*n where n is the percentile (.5 for median, .75 for 75th percentile, etc).
val is your time column, x and y are two references to the data table (you can write data AS x, data AS y).
EDIT:
To avoid computing your sums twice, you can store the intermediate results.
CREATE TEMPORARY TABLE average_user_total_time
(SELECT SUM(time) AS time_taken
FROM scores
WHERE created_at >= '2010-10-10'
and created_at <= '2010-11-11'
GROUP BY user_id);
Then you can compute median over these values which are in a named table.
EDIT: Temporary table won't work here. You could try using a regular table with "MEMORY" table type. Or just have your subquery that computes the values for the median twice in your query. Apart from this, I don't see another solution. This doesn't mean there isn't a better way, maybe somebody else will come with an idea.
First try to understand what the median is: it is the middle value in the sorted list of values.
Once you understand that, the approach is two steps:
sort the values in either order
pick the middle value (if not an odd number of values, pick the average of the two middle values)
Example:
Median of 0 1 3 7 9 10: 5 (because (7+3)/2=5)
Median of 0 1 3 7 9 10 11: 7 (because 7 is the middle value)
So, to sort dates you need a numerical value; you can get their time stamp (as seconds elapsed from epoch) and use the definition of median.
Finding median in mysql using group_concat
Query:
SELECT
IF(count%2=1,
SUBSTRING_INDEX(substring_index(data_str,",",pos),",",-1),
(SUBSTRING_INDEX(substring_index(data_str,",",pos),",",-1)
+ SUBSTRING_INDEX(substring_index(data_str,",",pos+1),",",-1))/2)
as median
FROM (SELECT group_concat(val order by val) data_str,
CEILING(count(*)/2) pos,
count(*) as count from data)temp;
Explanation:
Sorting is done using order by inside group_concat function
Position(pos) and Total number of elements (count) is identified. CEILING to identify position helps us to use substring_index function in the below steps.
Based on count, even or odd number of values is decided.
Odd values: Directly choose the element belonging to the pos using substring_index.
Even values: Find the element belonging to the pos and pos+1, then add them and divide by 2 to get the median.
Finally the median is calculated.
If you have a table R with a column named A, and you want the median of A, you can do as follows:
SELECT A FROM R R1
WHERE ( SELECT COUNT(A) FROM R R2 WHERE R2.A < R1.A ) = ( SELECT COUNT(A) FROM R R3 WHERE R3.A > R1.A )
Note: This will only work if there are no duplicated values in A. Also, null values are not allowed.
Simplest ways me and my friend have found out... ENJOY!!
SELECT count(*) INTO #c from station;
select ROUND((#c+1)/2) into #final;
SELECT round(lat_n,4) from station a where #final-1=(select count(lat_n) from station b where b.lat_n > a.lat_n);
Here is a solution that is easy to understand. Just replace Your_Column and Your_Table as per your requirement.
SET #r = 0;
SELECT AVG(Your_Column)
FROM (SELECT (#r := #r + 1) AS r, Your_Column FROM Your_Table ORDER BY Your_Column) Temp
WHERE
r = (SELECT CEIL(COUNT(*) / 2) FROM Your_Table) OR
r = (SELECT FLOOR((COUNT(*) / 2) + 1) FROM Your_Table)
Originally adopted from this thread.
SELECT * FROM `your_table` LIMIT 0, 10
->This will display the first 1,2,3,4,5,6,7,8,9,10
SELECT * FROM `your_table` LIMIT 5, 5
->This will show records 6, 7, 8, 9, 10
I want to Show data 2,3,4,5,6,7,8,9,10,1 and
next day 3,4,5,6,7,8,9,10,1,2
day after next day 4,5,6,7,8,9,10,1,2,3
IS IT POSSIBLE with out updating any data of this table ???
You can do this using the UNION syntax:
SELECT * FROM `your_table` LIMIT 5, 5 UNION SELECT * FROM `your_table`
This will first select rows within your limit, and then combine the remainder from the second select. Note that you don't need to set a limit on the second select statement:
The default behavior for UNION is that duplicate rows are removed from the result. The optional DISTINCT keyword has no effect other than the default because it also specifies duplicate-row removal. With the optional ALL keyword, duplicate-row removal does not occur and the result includes all matching rows from all the SELECT statements.
I don't think this might be achieved using a simple Select (I may be wrong). I think you'll need a stored procedure.
You've tagged this as Oracle, though your SQL syntax would be invalid for Oracle because it doesn't support LIMIT
However, here's a solution that will work in Oracle:
select *
from ( select rownum as rn,
user_id
from admin_user
order by user_id
) X
where X.rn > :startRows
and X.rn <= :startRows + :limitRows
order by case when X.rn <= :baseRef
then X.rn + :limitRows
else
X.rn
end ASC
;
where :startRows and :limitRows are the values for your LIMIT, and :baseRef is a value between 0 and :limitRows-1 that should be incremented/cycled on a daily basis (ie on day 1 it should be 0; on day 2, 1; on day 10, 9; on day 11 you should revert to 0). You could actually use the current date, converted to Julian and take the remainder when divided by :limitRows to automate calculating :baseRef
(substitute your own column and table names as appropriate)
Well, it might be a little bit late for the author of the question, but could be useful for people.
Short answer: It is possible to do the "spin" like author asked.
Long answer: [I'm going to explain for MySQL first - where I tested this]
Let's imagine that we have table your_table (INT rn, ...). What you want is to sort in specific way ("spin" with beginning at the rn=N). First condition of ordering is rn >= N desc. The idea (at least how I understand this) is we change the order from asc to desc and split our table in two parts (<N and >=N). Then we order this back by rn but asc order. It will execute sorting for each group independently. So here is our query:
select * from your_table where rn between 1 and 10
order by rn >= N desc, rn asc;
If you don't have rn column - you always can use the trick with parameter
select t.*, #rownum := #rownum + 1 AS rn
from your_table t,
(SELECT #rownum := 0) r
where #rownum < 10 /* here be careful - we already increased by 1 the rownum */
order by #rownum >=N - 1 desc, /* another tricky place (cause we already increased rownum) */
#rownum asc;
I don't know if the last one is efficient, though.
For Oracle, you always can use rownum. And I believe that you will have the same result (I didn't test it!).
Hope it helps!