MySQL + PHP pagination with no offset - mysql

Ok, here is the code i have for my pagination:
$SQL = "SELECT
cpc.product_id,
cp.product_internal_ref,
cp.product_name,
cpa.product_sale_price,
cpa.is_product_service,
cpa.product_service_price,
cpi.image_name,
cpi.image_ext
FROM catalog_products_categories cpc
JOIN catalog_products cp ON cp.product_id = cpc.product_id
JOIN catalog_products_attributes cpa ON cpa.product_id = cpc.product_id
LEFT JOIN catalog_products_images cpi ON cpi.product_id = cpc.product_id
WHERE cpc.category_id = ".$catID;
// PAGINATOR SECTION
if($paginatorVARS['paginatorACTION'] == "next") {
$SQL .= " AND cpc.product_id > ".$paginatorVARS['paginatorGOID']." ";
$SQL .= "GROUP BY cpc.product_id ORDER BY cpc.product_id ASC LIMIT ".$paginatorVARS['catalogPaginatorPAGEROWS'];
}
elseif($paginatorVARS['paginatorACTION'] == "prev") {
$SQL .= " AND cpc.product_id < ".$paginatorVARS['paginatorGOID']." ";
$SQL .= "GROUP BY cpc.product_id ORDER BY cpc.product_id DESC LIMIT ".$paginatorVARS['catalogPaginatorPAGEROWS'];
}
// END PAGINATOR SECTION
I used the method described here: http://www.slideshare.net/Eweaver/efficient-pagination-using-mysql but i can't seam to find a way to also sort by other columns like for example by cpa.product_sale_price. If i do : ORDER BY cpc.product_id ASC, cpa.product_sale_price DESC/ASC it will break the paginator next results i dont know what happens... Please help!!

not saying their stuff is perfect, but it is often a place to start, to consider if you really want to reinvent the wheel, or take their concepts and incorporate it into your new rounder wheel. take a look at
http://dev.sencha.com/deploy/ext-4.0.0/examples/grid/paging.html
and use fiddler to examine the http gets that are occuring during pagination, as well as changes in the sort context. when it starts out it brings back the http response header, json data with jsonp callback, and top node value at the end of the json data specifying record count=6679 / 50 per page meaning 134 pages total as calculated by the front-end. the record count can vary from call to call (from page to page) as rows are inserted and deleted.
the sort column gets passed in the query string just like the pages impact the limit start,total that goes into sql. this can be seen in fiddler output showing changes in pages or sort column requests (column name, and asc or desc):
GET /forum/topics-browse-remote.php?_dc=1369401925806&page=1&start=0&limit=50&sort=lastpost&dir=DESC&callback=Ext.data.JsonP.callback1 HTTP/1.1
GET /forum/topics-browse-remote.php?_dc=1369401977137&page=1&start=0&limit=50&sort=replycount&dir=ASC&callback=Ext.data.JsonP.callback2 HTTP/1.1
GET /forum/topics-browse-remote.php?_dc=1369401978355&page=1&start=0&limit=50&sort=replycount&dir=DESC&callback=Ext.data.JsonP.callback3 HTTP/1.1

Related

Can't wrap my head around MySQL statement

I have two tables:
cache and main
In cache there are a lot of fields; in main a little less. A UNION is not going to work because of the unequal number of columns.
cache
client - file - target - many other columns
main
client - file - target - few other columns
From cache I would like all columns for which main.target LIKE '%string%', cache.client = main.client, cache.file = main.file
For these particular records, target, client and file are always the same in main and cache.
I just can't get my head around this, but then again MySQL never was my strongest point.
Thank you very much in advance!
In the end combining the two SELECT statements with a UNION made things very complicated, for the simple reason there were countless other queries, some without UNION, that in the end all had to be processed by the same end routine presenting the results. As this was only a one-time query and time wasn't really an issue, in the end I just used SELECT on the two different tables and then combined the results by checking if a certain field was present. If not, the remaining results had to be fetched from the cache table; if so, the remaining results had to be fetched from the main table.
I actually wonder whether this solution is faster, slower or just as fast.
if (!isset($row['current']))
{
$field = $row['field'];
$sqlcache = "SELECT * FROM " . $dbtable . " WHERE (client = '$sqlclient' AND file = '$sqlfile' AND field = '$field')";
$resultcache = $conn->query($sqlcache);
if (!$resultcache)
{
die($conn->error);
}
$rowcache = $resultcache->fetch_assoc();
$currenttarget = $rowcache['current'];
$context = $rowcache['context'];
$dirtysource = $rowcache['dirtysource'];
$stringid = $rowcache['stringid'];
$limit = $rowcache['maxlength'];
$locked = $rowcache['locked'];
$filei = $rowcache['filei'];
}
else
{
$currenttarget = $row['current'];
$context = $row['context'];
$dirtysource = $row['dirtysource'];
$stringid = $row['stringid'];
$limit = $row['maxlength'];
$locked = $row['locked'];
$filei = $row['filei'];
}

SQL SELECT query excluding an array of id's for infinite scroll [duplicate]

I have followed help located in this topic: Using infinite scroll w/ a MySQL Database
And have gotten close to getting this working properly. I have a page that is displayed in blocks using jquery masonry, in which the blocks are populated by data from a mysql database. When I scroll to the end of the page I successfully get the loading.gif image but immediately after the image it says "No more posts to show." which is what it should say if that were true. I am only calling in 5 posts initially out of about 10-15, so the rest of the posts should load when I reach the bottom of the page but I get the message that is supposed to come up when there really aren't any more posts.
Here is my javascript:
var loading = false;
$(window).scroll(function(){
if($(window).scrollTop() == $(document).height() - $(window).height()) {
var h = $('.blockContainer').height();
var st = $(window).scrollTop();
var trigger = h - 250;
if((st >= 0.2*h) && (!loading) && (h > 500)){
loading = true;
$('div#ajaxLoader').html('<img src="images/loading.gif" name="HireStarts Loading" title="HireStarts Loading" />');
$('div#ajaxLoader').show();
$.ajax({
url: "blocks.php?lastid=" + $(".masonryBlock:last").attr("id"),
success: function(html){
if(html){
$(".blockContainer").append(html);
$('div#ajaxLoader').hide();
}else{
$('div#ajaxLoader').html('<center><b>No more posts to show.</b></center>');
}
}
});
}
}
});
Here is the php on the page the blocks are actually on. This page initially posts 5 items from the database. The javascript grabs the last posted id and sends that via ajax to the blocks.php script, which then uses the last posted id to grab the rest of the items from the database.
$allPosts = $link->query("/*qc=on*/SELECT * FROM all_posts ORDER BY post_id DESC LIMIT 5");
while($allRows = mysqli_fetch_assoc($allPosts)) {
$postID = $link->real_escape_string(intval($allRows['post_id']));
$isBlog = $link->real_escape_string(intval($allRows['blog']));
$isJob = $link->real_escape_string(intval($allRows['job']));
$isVid = $link->real_escape_string(intval($allRows['video']));
$itemID = $link->real_escape_string(intval($allRows['item_id']));
if($isBlog === '1') {
$query = "SELECT * FROM blogs WHERE blog_id = '".$itemID."' ORDER BY blog_id DESC";
$result = $link->query($query);
while($blogRow = mysqli_fetch_assoc($result)) {
$blogID = $link->real_escape_string($blogRow['blog_id']);
$blogTitle = $link->real_escape_string(html_entity_decode($blogRow['blog_title']));
$blogDate = $blogRow['pub_date'];
$blogPhoto = $link->real_escape_string($blogRow['image']);
$blogAuthor = $link->real_escape_string($blowRow['author']);
$blogContent = $link->real_escape_string($blogRow['content']);
//clean up the text
$blogTitle = stripslashes($blogTitle);
$blogContent = html_entity_decode(stripslashes(truncate($blogContent, 150)));
echo "<div class='masonryBlock' id='".$postID."'>";
echo "<a href='post.php?id=".$blogID."'>";
echo "<div class='imgholder'><img src='uploads/blogs/photos/".$blogPhoto."'></div>";
echo "<strong>".$blogTitle."</strong>";
echo "<p>".$blogContent."</p>";
echo "</a>";
echo "</div>";
}
}
Here is the php from the blocks.php script that the AJAX calls:
//if there is a query in the URL
if(isset($_GET['lastid'])) {
//get the starting ID from the URL
$startID = $link->real_escape_string(intval($_GET['lastid']));
//make the query, querying 25 fields per run
$result = $link->query("SELECT * FROM all_posts ORDER BY post_id DESC LIMIT '".$startID."', 25");
$html = '';
//put the table rows into variables
while($allRows = mysqli_fetch_assoc($result)) {
$postID = $link->real_escape_string(intval($allRows['post_id']));
$isBlog = $link->real_escape_string(intval($allRows['blog']));
$isJob = $link->real_escape_string(intval($allRows['job']));
$isVid = $link->real_escape_string(intval($allRows['video']));
$itemID = $link->real_escape_string(intval($allRows['item_id']));
//if the entry is a blog
if($isBlog === '1') {
$query = "SELECT * FROM blogs WHERE blog_id = '".$itemID."' ORDER BY blog_id DESC";
$result = $link->query($query);
while($blogRow = mysqli_fetch_assoc($result)) {
$blogID = $link->real_escape_string($blogRow['blog_id']);
$blogTitle = $link->real_escape_string(html_entity_decode($blogRow['blog_title']));
$blogDate = $blogRow['pub_date'];
$blogPhoto = $link->real_escape_string($blogRow['image']);
$blogAuthor = $link->real_escape_string($blowRow['author']);
$blogContent = $link->real_escape_string($blogRow['content']);
$blogTitle = stripslashes($blogTitle);
$blogContent = html_entity_decode(stripslashes(truncate($blogContent, 150)));
$html .="<div class='masonryBlock' id='".$postID."'>
<a href='post.php?id=".$blogID."'>
<div class='imgholder'><img src='uploads/blogs/photos/".$blogPhoto."'></div>
<strong>".$blogTitle."</strong>
<p>".$blogContent."</p>
</a></div>";
}
}
echo $html;
}
I have tried using the jquery infinite-scroll plugin, but it seemed much more difficult to do it that way. I don't know what the issue is here. I have added alerts and did testing and the javascript script is fully processing, so it must be with blocks.php right?
EDIT: I have made a temporary fix to this issue by changing the sql query to SELECT * FROM all_posts WHERE post_id < '".$startID."' ORDER BY post_id DESC LIMIT 15
The blocks are now loading via ajax, however they are only loading one block at a time. The ajax is sending a request for every single block and they are fading in one after another, is it possible to make them all fade in at once with jquery masonry?
I seen your code in another answer, and I would recommend using the LIMIT functionality in MySql instead of offsetting the values. Example:
SELECT * FROM all_posts ORDER BY post_id DESC LIMIT '".(((int)$page)*5)."',5
This will just take a page number in the AJAX request and get the offset automatically. It's one consistent query, and works independent of the last results on the page. Send something like page=1 or page=2 in your jQuery code. This can be done a couple different ways.
First, count the number of elements constructed on the page and divide by the number on the page. This will yield a page number.
Second, you can use jQuery and bind the current page number to the body:
$(body).data('page', 1)
Increment it by one each page load.
Doing this is really the better way to go, because it uses one query for all of the operations, and doesn't require a whole lot of information about the data already on the page.
Only thing to note is that this logic requires the first page request to be 0, not 1. This is because 1*5 will evaluate to 5, skipping the first 5 rows. If its 0, it will evaluate to 0*5 and skip the first 0 rows (since 0*5 is 0).
Let me know any questions you have!
Have you tried doing any debugging?
If you are not already using, I would recommend getting the firebug plugin.
Does the ajax call return empty? If it does, try echoing the sql and verify that is the correct statement and that all the variables contain the expected information. A lot of things could fail considering there's a lot of communication happening between client, server and db.
In response to your comment, you are adding the html in this piece of code:
if(html){
$(".blockContainer").append(html);
$('div#ajaxLoader').hide();
}
I would do a console.log(html) and console.log($(".blockContainer").length) before the if statement.

Display selected amount of records from database

I need a page that displays records from a database, sorted by their jobs. So the database holds different kind of persons with different jobs. For example, on the page "teacher" I just want to display all the teachers, not the other persons. I want to have 3 persons on a page and a button "previous" and "next" beneath it. If an user clicks the next-button I want the next 3 records to be shown. When the user reaches the last records, I need the next-button to disappear. Same goes for "previous".
What I have so far:
This piece of code creates a value named startrow.
if (!isset($_GET['startrow']) or !is_numeric($_GET['startrow'])) {
//give the value of the starting row 0 because nothing was found in URL
$startrow = 0;
//otherwise take the value from the URL
} else {
$startrow = (int)$_GET['startrow'];
}
The query I have:
$sql = mysql_query("SELECT * FROM $tbl_name WHERE jobs='teacher' LIMIT $startrow, 3")or
die(mysql_error());
$sql2 = "SELECT COUNT(*) AS TotalJobs FROM $tbl_name WHERE jobs='teacher'";
$result_count = mysql_query($sql2);
$count = mysql_fetch_array($result_count);
I created the $count for the if / else function in the next part (the part that doesn't seem to work the way I want it to):
if ($startrow < $count )
echo 'Next';
$prev = $startrow - 3;
//only print a "Previous" link if a "Next" was clicked
if ($prev >= 0)
echo 'Previous';
So the previous-button seems to work the way it should. The next doesn't. I've created the
if ($startrow < $count)
so basicly, if the value of startrow is smaller than the total number of records, it puts a next-button. But if I test this, it displays the next-button anyhow, no matter the value of startrow.
What am I missing here?
mysql_fetch_array returns an array containing the count, not the value itself. Use $count['TotalJobs'] instead.
Also, you shouldn't use the mysql extension anymore, instead use PDO or MySQLi.
Edit
You can change:
$count = mysql_fetch_array($result_count);
to
$countArr = mysql_fetch_array($result_count);
$count = $countArr['TotalJobs'];

MySql: Best way to run high number of search queries on a table

I have two tables, one is static database that i need to search in, the other is dynamic that i will be using to search the first database. Right now i have two separate queries. First on page load, values from second table are passed to first one as search term, and i am "capturing" the search result using cURL. This is very inefficient and probably really wrong way to do it, so i need help in fixing this issue. Currently page (html, front-end) takes 40 seconds to load.
Possible solutions: Turn it into function, but still makes so many calls out. Load table into memory and then run queries and unload cache once done. Use regexp to help speed up query? Possible join? But i am a noob so i can only imagine...
Search script:
require 'mysqlconnect.php';
$id = NULL;
if(isset($_GET['n'])) { $id = mysql_real_escape_string($_GET['n']); }
if(isset($_POST['n'])) { $id = mysql_real_escape_string($_POST['n']); }
if(!empty($id)){
$getdata = "SELECT id, first_name, last_name, published_name,
department, telephone FROM $table WHERE id = '$id' LIMIT 1";
$result = mysql_query($getdata) or die(mysql_error());
$num_rows = mysql_num_rows($result);
while($row = mysql_fetch_array($result, MYSQL_ASSOC))
{
echo <<<PRINTALL
{$row[id]}~~::~~{$row[first_name]}~~::~~{$row[last_name]}~~::~~{$row[p_name]}~~::~~{$row[dept]}~~::~~{$row[ph]}
PRINTALL;
}
}
HTML Page Script:
require 'mysqlconnect.php';
function get_data($url)
{
$ch = curl_init();
$timeout = 5;
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,$timeout);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$getdata = "SELECT * FROM $table WHERE $table.mid != '1'ORDER BY $table.$sortbyme $o LIMIT $offset, $rowsPerPage";
$result = mysql_query($getdata) or die(mysql_error());
while($row = mysql_fetch_array($result, MYSQL_ASSOC))
{
$idurl = 'http://mydomain.com/dir/file.php?n='.$row['id'].'';
$p_arr = explode('~~::~~',get_data($idurl));
$p_str = implode(' ',$p_arr);
//Use p_srt and p_arr if exists, otherwise just output rest of the
//html code with second table values
}
As you can see, second table may or may not have valid id, hence no results but second table is quiet large, and all in all, i am reading and outputting 15k+ table cells. And as you can probably see from the code, i have tried paging but that solution doesn't fit my needs. I have to have all of the data on client side in single html page. So please advice.
Thanks!
EDIT
First table:
id_row id first_name last_name dept telephone
1 aaa12345 joe smith ANS 800 555 5555
2 bbb67890 sarah brown ITL 800 848 8848
Second_table:
id_row type model har status id date
1 ATX Hybrion 88-85-5d-id-ss y aaa12345 2011/08/12
2 BTX Savin none n aaa12345 2010/04/05
3 Full Hp 44-55-sd-qw-54 y ashley a 2011/07/25
4 ATX Delin none _ smith bon 2011/04/05
So the second table is the one that gets read and displayed, first is read and info displayed if ID is positive match. ID is only unique in the first one, second one has multi format input so it could or could not be ID as well as could be duplicate ID. Hope this gives better understanding of what i need. Thanks again!
A few things:
Curl is completely unnecessary here.
Order by will slow down your queries considerably.
I'd throw in an if is_numeric check on the ID.
Why are you using while and mysql_num_rows when you're limiting to 1 in the query?
Where are $table and these other things being set?
There is code missing.
If you give us the data structure for the two tables in question we can help you with the queries, but the way you have this set up now, I'm surprised its even working at all.
What you're doing is, for each row in $table where mid!=1 you're executing a curl call to a 2nd page which takes the ID and queries again. This is really really bad, and much more convoluted than it needs to be. Lets see your table structures.
Basically you can do:
select first_name, last_name, published_name, department, telephone FROM $table1, $table2 WHERE $table1.id = $table2.id and $table2.mid != 1;
Get rid of the curl, get rid of the exploding/imploding.

MySQL - Perl: How to get array of zip codes within submitted "x" miles of submitted "zipcode" in Perl example

I have found many calculations here and some php examples and most are just over my head.
I found this example:
SELECT b.zip_code, b.state,
(3956 * (2 * ASIN(SQRT(
POWER(SIN(((a.lat-b.lat)*0.017453293)/2),2) +
COS(a.lat*0.017453293) *
COS(b.lat*0.017453293) *
POWER(SIN(((a.lng-b.lng)*0.017453293)/2),2))))) AS distance
FROM zips a, zips b
WHERE
a.zip_code = '90210' ## I would use the users submitted value
GROUP BY distance
having distance <= 5; ## I would use the users submitted value
But, I am having trouble understanding how to implement the query with my database.
It looks like that query has all I need.
However, I cannot even find/understand what b.zip_code actually is! (whats the b. and zips a, zips b?)
I also do not need the state in the query.
My mySQL db structure is like this:
ZIP | LAT | LONG
33416 | 26.6654 | -80.0929
I wrote this in attempt to return some kind of results (not based on above query) but, it only kicks out one zip code.
## Just for a test BUT, in reality I desire to SELECT a zip code WHERE ZIP = the users submitted zip code
## not by a submitted lat lon. I left off the $connect var, assume it's there.
my $set1 = (26.6654 - 0.20);
my $set2 = (26.6654 + 0.20);
my $set3 = (-80.0929 - 0.143);
my $set4 = (-80.0929 + 0.143);
my $test123 = $connect->prepare(qq{SELECT `ZIP` FROM `POSTAL`
WHERE `LAT` >= ? AND `LAT` <= ?
AND `LONG` >= ? AND `LONG` <= ?}) or die "$DBI::errstr";
$test123->execute("$set1","$set2","$set3","$set4") or die "$DBI::errstr";
my $cntr;
while(#zip = $test123->fetchrow_array()) {
print qq~$zip[$cntr]~;
push(#zips,$zip[$cntr]);
$cntr++;
}
As you can see, I am quite the novice so, I need some hand holding here with verbose explanation.
So, in Perl, how can I push zip codes into an array from a USER SUBMITTED ZIP CODE and user submitted DISTANCE in miles. Can be a square instead of a circle, not really that critical of a feature. Faster is better.
I'll tackle the small but crucial part of the question:
However, I cannot even find/understand what b.zip_code actually is! (whats the "b." and "zips a, zips b"?)
Basically, the query joins two tables. BUT, both tables being joined are in fact the same table - "zips" (in other words, it joins "zips" table to itself"). Of course, since the rest of the query needs to understand when you are referring to the first copy of the "zips" table and when to the second copy of the "zips" table, you are giving a table alias to each copy - to wit, "a" and "b"'.
So, "b.xxx" means "column xxx from table zips, from the SECOND instance of that table being joined".
I don't see what's wrong with your first query. You have latitude and longitude in your database (if I'm understanding, you're comparing a single entry to all others). You don't need to submit or return the state that's just part of the example. Make the first query work like this:
my $query = "SELECT b.zip_code,
(3956 * (2 * ASIN(SQRT(
POWER(SIN(((a.lat-b.lat)*0.017453293)/2),2) +
COS(a.lat*0.017453293) *
COS(b.lat*0.017453293) *
POWER(SIN(((a.lng-b.lng)*0.017453293)/2),2))))) AS distance
FROM zips a, zips b WHERE
a.zip_code = ?
GROUP BY distance having distance <= ?";
my $sth = $dbh->prepare($query);
$sth->execute( $user_submitted_zip, $user_submitted_distance );
while( my ($zip, $distance) = $sth->fetchrow() ) ) {
# do something
}
This won't be that fast, but if you have a small record set ( less than 30k rows ) it should be fine. If you really want to go faster you should look into a search engine such as Sphinx which will do this for you.
fetchrow_array returns a list of list references, essentially a two-dimensional array, where each row represents a different result from the database query and each column represents a field from the query (in your case, there is only one field, or column, per row).
Calling while ($test123->fetchrow_array()) will cause an infinite loop as your program executes the query over and over again. If the query returns results, then the while condition will be satisfied and the loop will repeat. The usual idiom would be to say something more like for my $row ($test123->fetchrow_array()) { ..., which will only execute the query once and then iterate over the results.
Each result is a list reference, and the zip code you are interested in is in the first (and only) column, so you could accumulate the results in an array like this:
my #zips = (); # for final results
for my $row ($test123->fetchrow_array()) {
push #zips, $row->[0];
}
or even more concisely with Perl's map statement:
my #zips = map { $_->[0] } $test123->fetchrow_array()
which does the same thing.