I have a database with millions of records.
The table is structured as:
Table name : record
Filed1: Name (varchar)(Primary key)
Field2: Record(int/bigint)
example:
Name | Record
Darrin | 256
Aaron | 3
Daryl | 12
...
I need to know what position does the user with the name 'namex' in sorted records.
Currently i implement this solution:
...
$name=namex;
$query= mysqli_query($mysqli,"SELECT Name FROM record ORDER BY Record DESC");
$x=0;
$rank=0;
if ($query->num_rows > 0) {
// output data of each row
while($row = $query->fetch_assoc()) {
if($row["Name"]==$name){
$rank=$x+1;
echo "Rank : $rank<br>";
break;
}
$x++;
}
}
...
With it and 1 million records in the database, the answer comes in about 4 second.
I tried to put a table index on the field Record but have remained the same performance.
how can I reduce the execution times?
Since I don't know what DBMS you are using for (in tags you are using both mysql and sql-server...), you can you create a view (for SQL server have to be a indexed view) or for mysql implement/emulate a kind of materialized view (has a better performance). The view is good to get a better perfomance through some DBMS. For MySQL may have no difference.
In the view show up the rank position as the query below (mysql example):
CREATE VIEW ranked_record AS
SELECT
record.Name,
#curRank := #curRank + 1 AS rank
FROM
record,
(SELECT #curRank := 0) r
ORDER BY Record DESC;
or SQL server:
CREATE VIEW ranked_record AS
SELECT
record.Name,
row_number() over(ORDER BY record)
FROM
record;
And just run your query:
SELECT name , rank FROM ranked_record WHERE name LIKE 'some name'
Update:
After John comments, I've realized about the error from views using variables. It isn't possible due the "feature/bug" of/from MySQL
Due this, you can choose about use this as subquery in FROM clause:
SELECT
name,
rank
FROM (
SELECT
record.Name,
#curRank := #curRank + 1 AS rank
FROM
record,
(SELECT #curRank := 0) r
) AS ranked_record
WHERE
name LIKE 'some name';
OR create a function to count the rank inside the view (like this example):
CREATE FUNCTION `func_inc_var_session`() RETURNS int(11)
begin
SET #var := IFNULL(#var,0) + 1;
return #var;
end;
Then, create your view as before, just using the function instead of the variable:
CREATE VIEW ranked_record AS
SELECT
record.Name,
func_inc_var_session() as rank
FROM
record
ORDER BY Record DESC;
We can write this in sql query instead of fetching the records and looping it in PHP.
select row = row_number() over( order by Record ) , * from record where name like 'namex'
I dont really understand why are you looping every record knowing that the column "Name" its your PK (unique values).
$name="Darrin";
$query= mysqli_query($db_connection, "SELECT COUNT(1) as rank FROM record WHERE Name = '".mysqli_real_escape_string($name)."' ORDER BY Record DESC");
$row = mysqli_fetch_row($query);
if ($query->num_rows > 0) {
echo "Rank : $row["rank"]<br>";
}
Related
This is my query on MySql and I want to write it on Laravel. How to write this MySql query on Laravel Controller? Beacuse i try to write it on laravel and the result is different from MySql
SET #row_num=0;
SELECT #row_num:=#row_num+1,
model.name, date,
sum(a), sum(b),
SUM(SUM(b)) OVER(PARTITION BY model.name ORDER BY date ROWS BETWEEN 2 PRECEDING AND CURRENT ROW),
IF(#row_num>2, SUM(SUM(b)/3) OVER(PARTITION BY model.name ORDER BY date ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) , NULL)
FROM table INNER JOIN model ON table.id_model = model.id
WHERE id_model = 1
GROUP BY date
Step 0: Indent that query so it's easier to work with
I used https://codebeautify.org/sqlformatter for this. The result was the following:
SET
#row_num = 0;
SELECT
#row_num := #row_num + 1,
model.name,
date,
sum(a),
sum(b),
SUM(
SUM(b)
) OVER(
PARTITION BY model.name
ORDER BY
date ROWS BETWEEN 2 PRECEDING
AND CURRENT ROW
),
IF(
#row_num > 2,
SUM(
SUM(b)/ 3
) OVER(
PARTITION BY model.name
ORDER BY
date ROWS BETWEEN 2 PRECEDING
AND CURRENT ROW
),
NULL
)
FROM
table
INNER JOIN model ON table.id_model = model.id
WHERE
id_model = 1
GROUP BY
date
Step 1: Identify different statements.
Your query (or queries) have 2 statements. a SET statement and a SELECT statement.
Step 2: Translate the statements to Query Builder if possible.
Start with the shortest/easier one first
Step 2.1: Translate the SET Statement.
To run a statement where we don't care about the results (just that it runs), DB::statement() can be used. This will mostly be copying the SQL.
DB::statement('SET #row_num = 0');
Step 2.2: Translate the SELECT statement.
For complex SQL queries, the easiest way to translate them is to run them with tinker and check if they are the same when using toSql().
Create a temporary .php file: touch myquery.php.
Write your query work-in-progress in your temp php file, finish with ->toSql(); and dump the results.
Run it through tinker: php artisan tinker myquery.php
Keep working on the query in your temp php file
Repeat steps 3 and 4 until the query is the one you want.
Again, we can start with the easy parts.
SQL
Query Builder
INNER JOIN model ON table.id_model = model.id
join('model', 'table.id_model', 'model.id')
WHERE id_model = 1
where('id_model', 1)
GROUP BY date
groupBy('date')
Now the query should look like this:
$query = DB::table('table')
->join('model', 'table.id_model', 'model.id')
->where('id_model', 1)
->groupBy('date')
->toSql();
As for the columns selected; many of them are aggregate functions so the use of selectRaw() is needed. As the name implies, you pretty much pass raw SQL to it. You can put it all in a single selectRaw() but I prefer to use one per column for readability.
As a result, the final query looks like this:
$query = DB::table('table')
->selectRaw('#row_num := #row_num + 1')
->selectRaw('model.name')
->selectRaw('date')
->selectRaw('sum(a)')
->selectRaw('sum(b)')
->selectRaw('SUM(SUM(b)) OVER (PARTITION BY model.name ORDER BY date ROWS BETWEEN 2 PRECEDING AND CURRENT ROW)')
->selectRaw('IF(row_num > 2, SUM(SUM(b)/3) OVER (PARTITION BY model.name ORDER BY date ROWS BETWEEN 2 PRECEDING AND CURRENT ROW), NULL)')
->join('model', 'table.id_model', 'model.id')
->where('id_model', 1)
->groupBy('date')
->toSql();
And there, it should be done.
Step 3: Cleanup
Step 3.1: Copy the queries to your Controller or wherever you need to use them.
DB::statement('SET #row_num = 0');
$results = DB::table('table')
->selectRaw('#row_num := #row_num + 1')
->selectRaw('model.name')
->selectRaw('date')
->selectRaw('sum(a)')
->selectRaw('sum(b)')
->selectRaw('SUM(SUM(b)) OVER (PARTITION BY model.name ORDER BY date ROWS BETWEEN 2 PRECEDING AND CURRENT ROW)')
->selectRaw('IF(row_num > 2, SUM(SUM(b)/3) OVER (PARTITION BY model.name ORDER BY date ROWS BETWEEN 2 PRECEDING AND CURRENT ROW), NULL)')
->join('model', 'table.id_model', 'model.id')
->where('id_model', 1)
->groupBy('date')
->get();
Step 3.2: Delete temp php file.
You can try DB::statement() or DB::unprepared()
I wrote a CTE to remove non numeric values from a data set, then get a count of numeric values within a range.
WITH dtr
AS ( SELECT resultlevel r
FROM dbo.Result
WHERE DrugID = 'AMP'
AND ISNUMERIC(ResultLevel) = 1
AND AuditStamp > '1/1/2016'
AND DeleteFlag = 0
)
SELECT COUNT(*)
FROM dtr
WHERE CONVERT(INT, r) BETWEEN 50 AND 75
This returns an error in SMS
Msg 245, Level 16, State 1, Line 2
Conversion failed when converting the varchar value 'PND ' to data type int.
This error is completely possible without the 'dtr' query in the CTE.
When I rewrite this, instead of a CTR, but a TEMP table, it works.
SELECT resultlevel r
INTO #d
FROM dbo.Result
WHERE DrugID = 'AMP'
AND ISNUMERIC(ResultLevel) = 1
AND AuditStamp > '1/1/2016'
AND DeleteFlag = 0
SELECT COUNT(*)
FROM #d
WHERE CONVERT(INT, r) BETWEEN 50 AND 75
So my questions is why?? I have always thought a CTE was like creating a TEMP table.
TEST DATA
if object_id('tempdb..#temp') is not null drop table #temp
create table #temp (result char(5))
insert into #temp (result) values
('1'),('A'),('>2'),('PEN ') ,('#3'),('-2'),('-33')
;with isnum AS (
SELECT result
FROM #temp
WHERE ISNUMERIC(result) = 1)
--Selecting from the CTE yields 1, -2, and -33 all of which can be converted to INT
--Running the query with the where clause causes the conversion error
SELECT
result,
ISNUMERIC(result)
FROM isnum
--WHERE CONVERT(INT,result) > 1
In SQL Server there is Logical Processing Order of the SELECT statement, which determines when the objects defined in one step are made available to the clauses in subsequent steps:
FROM
ON
JOIN
WHERE
GROUP BY
WITH CUBE or WITH ROLLUP
HAVING
SELECT
DISTINCT
ORDER BY
TOP
This is how your query is going to be proccesed and your query looks perfectly fine. But sometimes, the SQL Server decides not to follow this order in order to optimize your query.
In your case, the SQL Server might be simplyfing/transforming your query into another and performing the convert function, before applying the where isnumeric filtering.
If we made your query a little more complex (but still giving the same results), the SQL Server is executing the code correctly this time:
;with isnum AS (
SELECT result
FROM #temp
WHERE ISNUMERIC(result) = 1
GROUP BY result
HAVING MAX(result) = result
)
SELECT
result,
ISNUMERIC(result)
FROM isnum
WHERE CONVERT(INT,result) > 1;
In your case (and this is what I am doing in such situations when different types are stored in one column), you can simply use TRY_CONVERT function:
;with isnum AS (
SELECT result
FROM #temp
WHERE ISNUMERIC(result) = 1)
SELECT
result,
ISNUMERIC(result)
FROM isnum
WHERE TRY_CONVERT(INT, result) > 1
My Ruby plugin acts_as_list for sorting generates gaps in the position column when I insert Thing 2.0 at position 2 of a list from 1-3 and I want to get this list.
formatted_position;position;name
1;1;Thing 1
2;2;Thing 2.0;
3;4;Thing 2
4;5;Thing 3
So I tried ...
UPDATE user_ranking_items JOIN (SELECT #rownum := 0) r SET formatted_position = #rownum := #rownum + 1 WHERE ranking_id = 1 ORDER BY position ASC
But I got the MySQL exception Incorrect usage of UPDATE and ORDER BY.
How to handle this?
P.S.: Selecting works and returns the list from above ...
SELECT position, #rownum := #rownum + 1 AS formatted_position FROM user_ranking_items JOIN (SELECT #rownum := 0) r WHERE ranking_id = 1 ORDER BY position ASC;
The "trick" is to wrap your query that's working (with the user variable and the order by) as an inline view (MySQL calls it a derived table). Reference that whole view as a row source in the UPDATE statement.
Since you've already got the query that returns the result you want, you just need to have it return a unique id (e.g. the primary key) so you can do the JOIN between the inline view back to the original row in the target of the UPDATE.
Something like this:
UPDATE user_ranking_items t
JOIN (
-- your query here
) s
ON s.id = t.id
SET t.col = s.new_col_val
(This works because MySQL first materializes that inline view, as a temporary MyISAM table, and then the UPDATE references the temporary MyISAM table.)
For my application I would like to do the following:
SELECT *
FROM `LLS_USERS`
LIMIT 0,111
WHERE
(`USR_LOGIN`=
(CONCAT(
(SELECT LEFT(`USR_FIRST_NAME`, 1) FROM `LLS_USERS`;),
(SELECT `USR_LAST_NAME`FROM `LLS_USERS`;)
);
)
)
Basically I need to select all the rows in the user table where the user's login matches up with the first initial of the user name concatinated with the user's last name. So the SQL query will generate a table for me of all the selected rows where this is true in phpMyAdmin.
I know I have 111 users in my database currently.
Does anyone know what's wrong with my syntax?
UPDATE: SOLUTION is:
SELECT *
FROM `LLS_USERS`
WHERE (
`USR_LOGIN` = ( left( `USR_FIRST_NAME` , 1 ) || `USR_LAST_NAME` )
)
LIMIT 0 , 111;
Take SQL as a (real life) Language here in this basic example:
Give me all rows of a table that match the condition loginname= 1 char of first name following lastname
->
SELECT * FROM LLS_USERS where USR_LOGIN= LEFT(USR_FIRST_NAME, 1)||USR_LAST_NAME
Thats all, thats the beautiness of SQL
I have a table like this (MySQL 5.0.x, MyISAM):
response{id, title, status, ...} (status: 1 new, 3 multi)
I would like to update the status from new (status=1) to multi (status=3) of all the responses if at least 20 have the same title.
I have this one, but it does not work :
UPDATE response SET status = 3 WHERE status = 1 AND title IN (
SELECT title FROM (
SELECT DISTINCT(r.title) FROM response r WHERE EXISTS (
SELECT 1 FROM response spam WHERE spam.title = r.title LIMIT 20, 1)
)
as u)
Please note:
I do the nested select to avoid the famous You can't specify target table 'response' for update in FROM clause
I cannot use GROUP BY for performance reasons. The query cost with a solution using LIMIT is way better (but it is less readable).
EDIT:
It is possible to do SELECT FROM an UPDATE target in MySQL. See solution here
The issue is on the data selected which is totaly wrong.
The only solution I found which works is with a GROUP BY:
UPDATE response SET status = 3
WHERE status = 1 AND title IN (SELECT title
FROM (SELECT title
FROM response
GROUP BY title
HAVING COUNT(1) >= 20)
as derived_response)
Thanks for your help! :)
MySQL doesn't like it when you try to UPDATE and SELECT from the same table in one query. It has to do with locking priorities, etc.
Here's how I would solve this problem:
SELECT CONCAT('UPDATE response SET status = 3 ',
'WHERE status = 1 AND title = ', QUOTE(title), ';') AS sql
FROM response
GROUP BY title
HAVING COUNT(*) >= 20;
This query produces a series of UPDATE statements, with the quoted titles that deserve to be updated embedded. Capture the result and run it as an SQL script.
I understand that GROUP BY in MySQL often incurs a temporary table, and this can be costly. But is that a deal-breaker? How frequently do you need to run this query? Besides, any other solutions are likely to require a temporary table too.
I can think of one way to solve this problem without using GROUP BY:
CREATE TEMPORARY TABLE titlecount (c INTEGER, title VARCHAR(100) PRIMARY KEY);
INSERT INTO titlecount (c, title)
SELECT 1, title FROM response
ON DUPLICATE KEY UPDATE c = c+1;
UPDATE response JOIN titlecount USING (title)
SET response.status = 3
WHERE response.status = 1 AND titlecount.c >= 20;
But this also uses a temporary table, which is why you try to avoid using GROUP BY in the first place.
I would write something straightforward like below
UPDATE `response`, (
SELECT title, count(title) as count from `response`
WHERE status = 1
GROUP BY title
) AS tmp
SET response.status = 3
WHERE status = 1 AND response.title = tmp.title AND count >= 20;
Is using GROUP BY really that slow ? The solution you tried to implement looks like requesting again and again on the same table and should be way slower than using GROUP BY if it worked.
This is a funny peculiarity with MySQL - I can't think of a way to do it in a single statement (GROUP BY or no GROUP BY).
You could select the appropriate response rows into a temporary table first then do the update by selecting from that temp table.
you'll have to use a temporary table:
create temporary table r_update (title varchar(10));
insert r_update
select title
from response
group
by title
having count(*) < 20;
update response r
left outer
join r_update ru
on ru.title = r.title
set status = case when ru.title is null then 3 else 1;