Ok, so what is the best practice when it comes down to paginating in mysql. Let me make it more clear, let's say that a given time I have 2000 records and there are more being inserted. And I am displaying 25 at a time, I know I have to use limit to paginate through the records. But what am I supposed to do for the total count of my records? Do I count the records every time users click to request the next 25 records. Please, don't tell me the answer straight up but rather point me in the right direction. Thanks!
The simplest solution would be to just continue working with the result set normally as new records are inserted. Presumably, each page you display will use a query looking something like the following:
SELECT *
FROM yourTable
ORDER BY someCol
LIMIT 25
OFFSET 100
As the user pages back and forth, if new data were to come in it is possible that a page could change from what it was previously. From a logical point of view, this isn't so bad. For example, if you had an alphabetical list of products and a new product appeared, then the user would receive this information in a fairly nice way.
As for counting, your code can allow moving to the next page so long as data is there to support a new page being added. Having new records added might mean more pages required to cover the entire table, but it should not affect your logic used to determine when to stop allowing pages.
If your table has a date or timestamp column representing when a record was added, then you might actually be able to restrict the entire result set to a snapshot in time. In this case, you could prevent new data from entering over a given session.
3 sugggestions
1. Only refreshing the data grid, while clicking the next button via ajax (or) storing the count in session for the search parameters opted .
2. Using memcache which is advanced, can be shared across all the users. Generate a unique key based on the filter parameters and keep the count. So you won't hit the data base. When a new record, gets added then you need to clear the existing memcache key. This requires a memache to be running.
3. Create a indexing and if you hit the db for getting the count alone. There won't be much any impact on performance.
Related
I will try to explain as simple as possible.
There is a table which in start gets first 30 rows of n rows and has sorting, search using mat-table. So even if it is sorted only the first 30 rows will be sent and when the user goes to next page it gets the next 30 rows from backend each time making a new request.
Now each row has a button that will take it to another component which will get some detailed data about the specific row.
This component has a previous and next feature that will get the detailed view of the next or previous row data in the same order that is displayed in the table(sorted, search Result, page number).
Currently the table rows are made again in backend(Django) with all the sort, search and other parameters then find the current row and send the next and previous row (will take minimum 5 hits on DB).
Hence it very slow.
In Frontend I can only pass the data of only that page, which will have problem in next or previous page.
How to properly tackle this...
Normal search UIs don't focus on 30 rows at a time. Instead, they first search the entire dataset, then 'paginate' the results. (Or is that what you intended to say?)
There are details that can let the processing work fast, or there may be details that prevent speed. Please go into details about the table structure and the possible search criteria.
I have looked and haven't found a method on here to do this. I am assuming my search is skewed and I just missed it, if this is the case, please let me know.
Anywhooo, I have a large and unwieldy report coming out of SAP every day. Because it will often have some strangeness, we import that into an Access database so we can keep an eye on the stuff we need in our department. I am using a combination of 6 fields to create a primary key in Access. The information in those fields is about the only thing consistent I get out of this SAP report, but the remainder of the data can be considered dynamic and can change from day to day. Usually this is a matter of filling in a few blanks, Occasionally this is changing existing data, and on rare occasions, it may involve deleting data out of a handful of fields.
The SAP report is around 130 columns of data, So I'm looking for an efficient way to roll in the changes without overwriting what folks put in there manually.
EDIT:
Here is the way this is used. SAP (for reasons I'm not going to go into) sometimes will have bad data show up in the daily report. We are using Access to track and put the correct data in to something that we can generate much more accurate summaries. What the users put in is to be considered true and accurate.
The transactions we are tracking can take a long time to complete. Most take around 30 days to complete. That's why I will have blank fields on one day, and several of them to be filled in on the next. We might not get any for the next few days and then a bunch more are filled in later. That is the normal flow.
What I have to account for is the odd occasion where a mistake is made early in the process. At a certain point, an error will break SAP's ability to update anything at all in the report we have to use.
I have 3 fields set up that trigger what my users daily work is going to be. There is a logical flow so that user 1 completes what he needs to do and then that record will show up on User 2's report. These fields will also stop the general update process in an exception report if there is a difference in what is coming in from SAP, and what is already in my database.
What I am looking for is some way to systematically fill in blank fields, on existing records in access. I do not want to overwrite if something is in a field, only the null values. I can do this on one field at a time, but each record has about 130 fields. I'm wondering if there is a way I could do this in just 1 query?
Thanks all! I hope the edit makes more sense now
A simple google for "Access SQL update null values" could have yeilded you what you need. But if all you need to do is fill constant values into empty fields then something like:
UPDATE Table SET Table.field1 = VALUE
WHERE Table.field2 is NULL;
Now if this data is different for each record based on; say data from another field, then you may need to write some VBA to build that value/string for you. But otherwise if you are JUST updating null fields to include data, then a simple UPDATE statement will do
EDIT Based on new info:
So if I'm understanding correctly: you have two tables. One table with the blank fields and another table that contains the values you need.
If this is the case, you can use a similar UPDATE statement, but use an inner join to get the data you need from table B to fill in table A
UPDATE TableA INNER JOIN TableB ON TableA.KeyField = TableB.KeyField
SET TableA.NullField = TableB.NotNullField
WHERE TableA.NullField Is NULL;
I'm hoping this will be a rather simple question to answer, as I'm not looking for any specific code. I have a table on a classic asp page populated from an sql server. I've just set the table up so that each row is clickable and takes you to a page to edit the data in the row. My question is this: Would I be better off trying to use the recordset that populated the table or should I reconnect to the db and pull just the record I want edited.
As always; It Depends. It depends on what you need to edit about the record. It depends on How far apart your DB and site are from each other. It depends on which machine, if the DB and site are on separate machines, is more powerful.
That being said, you should make a new call for that specific record. The reason mainly being because of a specification you made in your question:
...and takes you to a page to edit the data in the row
You should not try to pass a record set between pages. There are a few reasons for this
Only collect what you need
Make sure data is fresh
Consider how your program will scale
On point 1 there are two ways to look at this. One is that you are trying to pass the entire record set across a page when you only need 1 record. There are few situations where another DB call would cost more than this. The other is you are only passing one record which would make me question your design. Why does this record set have every item related to a record. You are selecting way too much for just a result list. Or if the record is that small then Why do you need the new page. Why can you not just reveal an edit template for the item if it is that minimal.
On point 2 consider the following scenario. You are discussing with a coworker how you need to change a customer's record. You pull up this result set in an application but then nature calls and you step away from you desk. The coworker gets called by the customer and asked why the record is not updated yet. To placate the customer your coworker makes the changes. Now you are using an old record set and may overwrite additional changes your coworker made while you were away. This all happens because you never update the record set, you always just pass the old one from page to page.
On point 3 we can look back a point 1 a bit. let us say that you are passing 5 fields now. You decide though that you need a comments field to attach to one of your existing fields. do you intend to pass 2000 characters of that comment field to the next page? How about if each of the 5 need a comment field? Do you intend to pass 10,000 characters for a properly paged record set of 10? do you not do record set paging and need to pass 10,000 characters for a full 126 records.
There are more reasons too. Will you be able to keep your records secure passing them this way? Will this effect your user's experience because they have a crummy computer and cannot build that quick of a post request quickly? Generally it is better to only keep what you need and in most situations your result set should not have everything you need to edit.
For news-ticker applications, like the one in Facebook, we see that as we scroll further, older news appear. Now surely the news are inserted into the table as they occur, so a normal selection would always retrieve older records earlier, whereas here the reverse occurs. I assume the way it is done is that when the user scrolls down to the end, FB sends an Ajax request with the id of the last news-id currently present in the ticker (couldn't identify for sure in Firebug, FB sends loads of data!), the PHP queries the DB, flips the result set according to the time column, then extracts the say next 5 records following the one with the received id. Now such tables are huge, so flipping them frequently surely takes a heavy toll on the DB. So is there any way to achieve this without flipping?
If you don't specify an ORDER BY variable in your query, MySQL is not guaranteed to return the records in order of insertion. See this answer for more detailed information. If you want to be sure you're getting the most recent rows, you need to have an insertion time column and sort on that.
If you're sorting on time desc, then you can use the usual LIMIT clause to request just the first 5 records (i.e., the 5 most recent), or the 5 most recent after a certain ID, etc.
Yes, keeping in mind that we're speaking about this very abstractly. If you wanted to access an indexed collection in reverse order.
$collection = array(); // Filled through request to server
....
for ($i = sizeof($collection)-1; $i >= 0; $i--) {
echo $collection[$i]
.... // execute some action based on access to
}
Which is to say, there's no reason you can't access an array from its last index.
specify order by in your sql statement to get the oldest first. and do where id > last_id to get the rows after the last_id
I am trying to determine the best method of collecting a large list from a database and then displaying and filtering the results on the client side. Let me give a quick example:
Example: I've got a database with customer data and currently it contains around 2000 records. This number is constantly increasing. On my website I have a page that I want to be able to query said database based on information such as name, email, phone number etc. and of course display the results (when a user types in Smith it returns all records containing the name Smith). I am planning on using AJAX so that I can query the database and display the results on the fly similar to how google does it. When a user begins searching, results will start showing up on the page as they are found.
Possible Solutions:
Unfortunately I am stumped on how to go about implementing something like this. I am considering using a ValueList pattern. When the user first loads the page, should I be querying the database and storing every record in a collection and then searching that collection list and displaying the results on my jsp page? Essentially creating a java database. The thing I like about the ValueList pattern is that I take one huge hit on page load and dump the entire database in objects stored in a list. What if the database is larger though, say 2,000,000 records?
Or should I be using a simple DOA pattern without the ValueList and query the database for each individual search? This would result in a LOT of database queries, especially considering that I plan on returning results as the user types in the search box.
Edit: The more I think about this, the more it is an AJAX question. My biggest concern should be how to query my database while the user is typing. Do I set some sort of listener to listen for the user to stop typing and then perform the query?
I would use Solr for this type of task.
Fields, which you are going to use for searching should be indexed with Solr.
Then you do an ajax query to Solr and get the result. You can set the order, number of items per page and show results only for current page.
Solr has a lot of other features that can be useful for you.