I have a query that I wrote a couple of years ago for a membership website. I'm not extremely well versed in using $wpdb (or MYSQL code in general) to write custom queries, and the site has grown quite a bit. There are about 150k rows in the wp_usermeta table now, and the page where the query runs hangs for a couple of seconds now before loading. I expect that this will get worse as time goes on and the site gains more users.
Any help in figuring out how to speed this query up would be greatly appreciated.
$paged = (get_query_var('paged')) ? get_query_var('paged') : 1;
$limit = 15;
$offset = ($paged - 1) * $limit;
$key = 'first_name';
$sql = "SELECT SQL_CALC_FOUND_ROWS {$wpdb->users}.* FROM {$wpdb->users}
INNER JOIN {$wpdb->usermeta} wp_usermeta ON ({$wpdb->users}.ID = wp_usermeta.user_id)
WHERE 1=1
AND wp_usermeta.meta_key = '$key'
AND wp_usermeta.meta_value <> ''
AND wp_users.user_registered < '2014-01-30'
ORDER BY wp_usermeta.meta_value ASC
LIMIT $offset, $limit";
$members = $wpdb->get_results($sql);
$found_rows = $wpdb->get_var("SELECT FOUND_ROWS();");
foreach ($members as $member) { // display member info here }
*Note: I am paginating...not displaying all results on one page.
A couple of things here.
First, you seem to be ordering by user first name, but not retrieving or displaying it. That's a bit odd. You could order by wp_users.user_nicename, and omit the join to wp_usermeta. That will save some time in joining. That column's value is set by the user in her profile, and reflects the identity by which she wants to be known. So, it's an appropriate ordering column, most likely.
Second, you're retrieving all the columns (*) in wp_users. Can you avoid retrieving them all, and rather enumerate the one you actually need? That will save time in sorting.
Your query would become this.
$sql = "SELECT SQL_CALC_FOUND_ROWS
id, user_login, user_nicename, etc etc
FROM {$wpdb->users}
WHERE user_registered < '2014-01-30'
ORDER BY user_nicename ASC
LIMIT $offset, $limit";
You are stuck, as you are paginating a large number of users, with some inescapable inefficiency. But this should be better than what you have.
Related
I am not a professional programmer, but I assist a school in automating their assessments. I have a list of just over 1000 students with 3 assessment scores for each one every year and I need to create a list with the average of these three scores in descending order, limiting it to the top 30. I can calculate averages and display the results, but I can't sort or limit. In the first part of the code, I select the IDs from all students and store them into the array $alunos[] for the current year ($IDanoatual). In the second part, I use a for loop to calculate the average of these grades for each student and display them. Both codes lookup the same table ( audp_l_notasfinais). I tried using the foreach statement to filter and sort, but I couldn't resolve this issue.
$sela = "select id_aluno from audp_l_notasfinais
where id_ano = '$IDanoatual'
";
$qsela = mysqli_query($conn,$sela);
$contasel = mysqli_num_rows($qsela);
while ($row = mysqli_fetch_assoc($qsela)){
$alunos[] = $row['id_aluno'];
}
for ($i=0; $i<$contasel; $i++){
$selnotasA = "select avg(NULLIF(nts.mef,0)) as NtutA, aln.stdname Naln
from audp_l_notasfinais nts
inner join audp_c_alunos aln on aln.id_alunos = nts.id_aluno
where nts.id_aluno = '$alunos[$i]' and nts.id_ano='$IDanoatual'
";
$qrynmal = mysqli_query($conn,$selnotasA);
while ($row=mysqli_fetch_assoc($qrynmal)){
echo "Name: ".$row['Naln']." - Average: ".$row['NtutA']."<br>";
}
}
You have not included much detail in your question. Adding your CREATE TABLE statements and some sample data in markdown tables would help, and get a better response.
It looks like $IDanoatual could be coming from user input, in which case you really need to read about and understand SQL Injection and how to mitigate the risk with prepared statements.
Best guess -
select aln.id_alunos, aln.stdname Naln, avg(NULLIF(nts.mef,0)) as NtutA
from audp_c_alunos aln
inner join audp_l_notasfinais nts
on aln.id_alunos = nts.id_aluno
and nts.id_ano = '$IDanoatual'
group by aln.id_alunos
order by NtutA desc
limit 30;
Hi I need to get the results and apply the order by only in the limited section. You know, when you apply order by you are ordering all the rows, what I want is to sort only the limited section, here is an example:
// all rows
SELECT * FROM users ORDER BY name
// partial 40 rows ordered "globally"
SELECT * FROM users ORDER BY name LIMIT 200,40
The solution is:
// partial 40 rows ordered "locally"
SELECT * FROM (SELECT * FROM users LIMIT 200,40) AS T ORDER BY name
This solution works well but there is a problem: I'm working with a Listview component that needs the TOTAL rows count in the table (using SQL_CALC_FOUND_ROWS). If I use this solution I cannot get this total count, I will get the limited section count (40).
I hope you will give me solution based on the query, for example something like: "ORDER BY LOCALLY"
Since you're using PHP, might as well make things simple, right? It is possible to do this in MySQL only, but why complicate things? (Also, placing less load on the MySQL server is always a good idea)
$result = db_query_function("SELECT SQL_CALC_FOUND_ROWS * FROM `users` LIMIT 200,40");
$users = array();
while($row = db_fetch_function($result)) $users[] = $row;
usort($users,function($a,$b) {return strnatcasecmp($a['name'],$b['name']);});
$totalcount = db_fetch_function(db_query_function("SELECT FOUND_ROWS() AS `count`"));
$totalcount = $totalcount['count'];
Note that I used made-up function names, to show that this is library-agnostic ;) Sub in your chosen functions.
I have a fairly large dataset and a query that requires two joins, so efficiency of the query is very important to me. I need to retrieve 3 random rows from the database that meet a condition based on the result of a join. Most obvious solution is pointed out as inefficient here, because
[these solutions] need a sequential scan of all the table (because the random value associated with each row needs to be calculated - so that the smallest one can be determined), which can be quite slow for even medium sized tables.
However, the method suggested by the author there (SELECT * FROM table WHERE num_value >= RAND() * (SELECT MAX(num_value) FROM table) LIMIT 1 where num_value is ID) doesn't work for me because some IDs might be missing (because some rows may have been been deleted by users).
So, what would be the most efficient way to retrieve 3 random rows in my situation?
EDIT: the solution does not need to be pure SQL. I also use PHP.
Since you don't want many results, there are a couple of interesting options using LIMIT and OFFSET.
I'm going to assume an id column which is unique and suitable for sorting.
The first step is to execute a COUNT(id), and then select random 3 numbers from 0 to COUNT(id) - 1 in PHP. (How to do that is a separate question, and the best approach depends on the number of rows total and the number you want).
The second step has two options. Suppose the random numbers you selected are 0, 15, 2234. Either have a loop in PHP
// $offsets = array(0, 15, 2234);
foreach ($offsets as $offset) {
$rows[] = execute_sql('SELECT ... ORDER BY id LIMIT 1 OFFSET ?', $offset);
}
or build a UNION. Note: this requires sub-selects because we're using ORDER BY.
// $offsets = array(0, 15, 2234);
$query = '';
foreach ($offsets as $index => $offset) {
if ($query) $query .= ' UNION ';
$query .= 'SELECT * FROM (SELECT ... ORDER BY id LIMIT 1 OFFSET ?) Sub'.$index;
}
$rows = execute_sql($query, $offsets);
Adding your RAND() call into the ORDER BY clause should allow you to ignore the ID. Try this:
SELECT * FROM table WHERE ... ORDER BY RAND() LIMIT 3;
After having performance issues pointed out, your best bet may be something along these lines (utilizing PHP):
$result = PDO:query('SELECT MAX(id) FROM table');
$max = $result->fetchColumn();
$ids = array();
$rows = 5;
for ($i = 0; $i < $rows; $i++) {
$ids[] = rand(1, $max);
}
$ids = implode(', ', $ids);
$query = PDO::prepare('SELECT * FROM table WHERE id IN (:ids)');
$results = $query->execute(array('ids' => $ids));
At this point you should be able to select the first 3 results. The only issue with this approach is dealing with deleted rows and you might have to either bump the $rows var or add some logic to do another query in case you didn't receive at least 3 results back.
I am not a programmer or good at programming, just laying this out. I had a site coded for me, but the coder is not interested in further development. I need to rank images in the database by losses (kinda like votes) to views of an image. Originally, here is what was written for the code:
// get images
$result = $this->db->query("SELECT * FROM `ylyl_images` ORDER BY `loses` DESC LIMIT $start, 12");
That basically orders the images on the site by loses, and to paginate, it has the $start variable which is incremented by 12 for every "next page" click.
$result = $this->db->query("SELECT (`loses`/`views`) AS `ratio` FROM `ylyl_images` WHERE `views` > 150 ORDER BY `ratio` DESC LIMIT $start, 12");
That is what I came up with to order images based on ratio of loses to views, but the query doesn't return anything. I tried replacing the $start variable with just 0, but it still doesn't work.
Are you returning the right fields? In the code above, you Changed:
SELECT * FROM...
To...
SELECT loses/views AS ratio FROM...
Perhaps you need to return more fields. While I do not recommend 'SELECT *', you could do the following:
SELECT *, loses/views AS ratio FROM...
my $sth = $dbh->prepare("SELECT id
FROM user
WHERE group == '1'
ORDER BY id DESC
LIMIT 1");
I was trying to get the id of the last row in a table without reading the whole table.
I am already accessing via:
my $sth = $dbh->prepare("SELECT name,
group
FROM user
WHERE group == '1'
LIMIT $from, $thismany");
$sth->execute();
while(my ($name,$group) = $sth->fetchrow_array()) {
...and setting up a little pagination query as you can see.
But, I am trying to figure out how to detect when I am on the last (<= 500) rows so I can turn off my "next 500" link. Everything else is working fine. I figured out how to turn off the "previous 500" link when on first 500 page all by myself!
I thought I would set up a "switch" in the while loop so if ($id = $last_id) I can set the "switches" var.
Like:
if ($id = $last_id) {
$lastpage = 1; #the switch
}
So I can turn off next 500 link if ($lastpage == 1).
I am really new to this and keep getting stuck on these types of things.
Thanks for any assistance.
Try to grab an extra row and see how many rows you really got. Something like this:
my #results = ( );
my $total = 0;
my $sth = $dbh->prepare(qq{
SELECT name, group
FROM user
WHERE group = ?
LIMIT ?, ?
});
$sth->execute(1, $from, $thismany + 1);
while(my ($name, $group) = $sth->fetchrow_array()) {
push(#results, [$name, $group]); # Or something more interesting.
++$total;
}
$sth->finish();
my $has_next = 0;
if($total == $thismany + 1) {
pop(#results);
$has_next = 1;
}
And BTW, please use placeholders in all of your SQL, interpolation is fraught with danger.
Always asking for one more row than you are going to show, as suggested by mu is too short, is a good way.
But if you want to take the other suggested approach of doing two separate queries, one to get the desired rows, and one to get the total count if there had not been a limit clause, MySQL provides an easy way to do that while combining as much of the work as possible:
SELECT SQL_CALC_FOUND_ROWS name, group FROM user WHERE group = '1' LIMIT ..., ...;
then:
SELECT FOUND_ROWS();
The SQL_CALC_FOUND_ROWS qualifier changes what a following FOUND_ROWS() returns without requiring you to do a whole separate SELECT COUNT(*) from user WHERE group = '1' query.
SELECT COUNT(*) from tablename will give you the number of rows, so if you keep a running count of how many rows you have read so far, you'll know when you're on the last page of results.
You could generate that query with (untested; away from a good workstation at the moment):
my $sth = $dbh->prepare("select COUNT(*) FROM user WHERE group == '1'");
my #data = $sth->fetchrow_array;
my $count = $data->[0];
(PS. you should be aware of SQL injection issues -- see here for why.)
As Ether mentioned in the comments, pagination usually requires two queries. One to return your paged set, the other to return the total number of records (using a COUNT query).
Knowing the total number of records, your current offset and the number of records in each page is enough data to work out how many pages there are in total and how many before and after the current page.
Although your initial suggestion of SELECT id FROM table WHERE ... ORDER BY id DESC LIMIT 1 should work for finding the highest matching ID, the standard way of doing this is SELECT max(id) FROM table WHERE ...