I am using Flex 3, with BlazeDS to interact with Java layer and fatch the data. I am getting more then 10000 rows of data at a time to display in my datagrid. I am displaying it 200 at a time and using it paging for the application. (Whenever I call the next button link, a server call happen and fetch the next 200 data)
I was wondering is there any other technique we can use flex side to buffer the 10000 data and display everything without calling server everytime.
Thanks for your any help.
You can put an intermediate layer in your Flex application which can hold all data in it, and then perform the paging on it.
Fetch all data
Store it in an ArrayCollection (or ArrayList, Array or other that
suit your needs)
Create a method to retrieve wanted data from it, for example:
retrieveData(offset:int, limit:int):ArrayCollection
It should retrieve the data starting from offset and end at offset + limit.
Calling retrieveData(500, 20); should return the 20 items starting
from position 500.
Hope that helps.
Related
I'm looking to pull data from a REST API that uses a record offset system. I need to perform basic arithmetic on the returned (current_offset + item_count) data to calculate the next offset.
Unfortunately, this API uses a variable page size, so simple offsets won't work. Nor does it return the next offset to use in the response, just the component values.
However, it appears that there's no way to do even basic arithmetic that I can see. Am I just missing it?
I have phonorgraph object with billions of rows and we are querying it through object set service
for example, I want to get all DriverLicences from certain city.
#Function()
public getDriverLicences(city: string): ObjectSet<DriverLicences> {
let drivers = Objects.search().DriverLicences().filter(row => row.city.exactMatch(city));
return drivers ;
}
I am facing this error when I am trying query it from slate:
ERROR 400: {"errorCode":"INVALID_ARGUMENT","errorName":"ObjectSet:PagingAboveConfiguredLimitNotAllowed","errorInstanceId":"0000-000","parameters":{}}
I understand that I am probably retrieving more than 100 000 results but I need all the results because of the implemented logic in the front is a complex slate dashboard built by another team that we cannot re-factor.
The issue here is that, specifically in the Slate <> Function connector, there is a "translation layer" that serializes the contents of the object set and provides a response data structure that materializes the property:value pairs for each object in the set.
This clearly doesn't work for large object sets where throwing so much data into the browser is likely to overwhelm the resources allocated to the tab.
From context it seems like you might be migrating an existing Slate app over to Functions; in the current version, how is the query limiting the number of results returned? It certainly must not be returning several 100 thousand results for further processing on the front end? (And if so, that might be an anti-pattern to consider addressing).
As for options that you could currently explore, you can sort your object set and then specify a smaller limit to return:
Objects.search().DriverLicences().filter(row => row.city.exactMatch(city)).orderBy(date_of_issue).take(100)
You'll find a few more details in the Functions documentation Reference entry on Ontology API: Object Sets in the section on Ordering and limiting.
You can even make a work around for the (current) lack of paging when return an ObjectSet to Slate by using the last value from the property ordered on (i.e. date_of_issue) as a filter in the subsequent request and return the next N objects.
This can work if you need a Slate table or HTML widget that renders on set of results then, on a user action, gets the next page.
This issue is bugging me for some time now. To test it I just installed a fresh Apigility, set the db (PDO:mysql) and added a DB-Connected service. In the table I have 40+ records. When I make a GET collection request the response looks OK (with the default HAL content negotiation). Then I change the content negotiation to JSON. Now when I make a GET collection request my response contains only 10 elements.
So my question is: where do I set/change this limit?
You can set the page size manually, like so:
$paginator = $this->getAlbumTable()->fetchAll(true);
// set the current page to what has been passed in query string, or to 1 if none set
$paginator->setCurrentPageNumber((int) $this->params()->fromQuery('page', 1));
// set the number of items per page to 10
$paginator->setItemCountPerPage(10);
http://framework.zend.com/manual/current/en/tutorials/tutorial.pagination.html
Could you please send the page_size, total_items part at the end of the json output?
it's like:
"page_count": 140002,
"page_size": 25,
"total_items": 3500035,
"page": 1
This is not an ideal fix, because it requires you to go into the source code rather than using the page size given in the UI.
The collection class that is auto generated for you by the DB-Connected style derives off of Zend/Paginator/Paginator. This class defines the $defaultItemCountPerPage static protected member which is defaulted to 10. That's why you're only getting 10 results. If you open up the auto-generated collection class for your entity and add: protected static $defaultItemCountPerPage = 100; in the otherwise empty class, you will see that you now get up to 100 results in the response. You can look at other Paginator class variables and methods that you could replace in your derived class to get your desired behavior.
This is not an ideal solution. I'd prefer that the generated code automatically used the same configed page size that the HalJson strategy uses. Maybe I'll contribute a PR to change that. Or, maybe I'll just use the HalJson approach. It does seem like the better way to go. You should have some limit to how much data you load in from the DB at a time to not have an overly long running query or an overly large collection of data coming back you have to deal with. And, whatever limit you set, what do you do when you hit that limit? With the simple Json method, you can't ever get "page 2" of data. So, if you are going to work with some sizeable amount of data, it might be better to use HalJson on and then have some logic on the client side to grab pages of data at a time as needed. The returned JSON structure is a little more complicated, but not terribly so.
I'm probably in the same spot you are -- I'm trying to do a simple little api to play with while keeping everything simple and so I didn't want the client to have to deal with the other stuff in HalJson, but probably better to deal with that complexity and have a smooth way to page through data if you're going to use this with some real set of data. At least, that's the pep talk I'm giving myself right now. :-)
I have UITableView representing list of cities (100 cities).
For each city I want to call specific remote(URL) JSON to get city's weather information and populate response data for each city cell in the UITableView.
When I run application, I want to see my table as fast as possible, so I don't need to wait for all json responses. I want that informations got asynchronously (when specific json is loaded, set it's information for corresponding city cell in the UITableView).
Note: It is important for me to call seperate remote JSON files.
Which technic is the best for this task?
I would start with the following approach:
Create a data structure to hold city information, including:
path to your data service,
service call "state" (idle, waiting, completed, error),
weather information (from JSON returned by service call)
When you first show the table, you will want to:
initialize your array (of the aforementioned data structure),
initiate each service call asynchronously,
set each row (city) state to waiting.
You will also probably want to return a custom UITableCellView with the city name (if you already have it) and a spinning activity indicator. This will be your best option to have a fast load time (not waiting for services to complete) and give some visual indication that the data is loading.
Each service call should use the ViewController as its delegate; you will need a key field so that when the services return, they can identify with which row/city they are associated.
As each service completes and calls the delegate, it will send the data to the ViewController, which (in turn) will update the array and initiate a UITableView update.
The UITableView update is, in my opinion, the most difficult part. Typically cells are drawn or updated when they become visible; the table pre-fetches all visible cells' geometry and then queries the actual contents when it's ready to draw each cell; as a result, your strategy for updating cells will depend on how your table is used.
If your cell geometry changes, you will most likely need to redraw your entire table; I shudder to think about what 50 simultaneous UITableView redraws will do for your app, so you might need to set a time-threshold to "chunk" updates and handle drawing more intelligently.
[theTableView reloadData] will cause the entire table to be re-queried and redrawn.
If your cell geometry does not change, you can try to be more surgical of updating only the visible cells (the non-visible ones aren't an issue since their data will be queried when they become visible).
[theTableView visibleCells] returns an array of visible cells; when your service call returns, you could update the data and then search the array to see if the cell in question is visible; if it is, you will probably need to send the specific UITableCellView a setNeedsDisplay message.
There is a good explanation of setNeedsDisplay, setNeedsLayout, and 'reloadData' at http://iosdevelopertips.com/cocoa/understanding-reload-repaint-and-re-layout-for-uitableview.html.
There is a relevant SO question at How to refresh UITableViewCell?
Lastly, you will probably want to implement some updating logic in the service delegate error routine, just so you don't create endlessly spinning activity indicators.
I do this now while searching multiple servers. I use Core Data, but you can use an NSMutableArray to accumulate your JSON responses.
Every time you finish receiving date from one of your servers (for example, when connectionDidFinishLoading executes), take the JSON data object and add it to an NSMutableArray (let's call it weatherResults) (add it using the addObject method). You may want to convert the JSON to an NSDictionary before adding it to the mutable array weatherResults.
Assuming your dataSource delegate methods refer to what is in the weatherResults NSMutableArray (for example, getting the number of rows from the size of the array using [weatherResults count]) you can do the following:
After inserting the object to the array, you can simply call reloadData in the dataSource controller. You will see the table update as each new JSON results arrives. The results should append to the bottom of the table as they come in. If you want to sort the NSMutableArray each time a JSON results arrives, you can do that too.
I do this and the time it takes to resort and reload the table is insignificant on my iPad. If you do not resort, it should be even faster.
By the way, in this explanation, I assume that the JSON response contains all of the information that you need to fill in your table cell. That may not be the case. If it's not, you will have to correlate the response with other information you have, such as a list of cities that your program is presenting.
I'm doing a little research on possible application of EWS in our existing project which is written with heavy use of MAPI and I found out something disturbing about performance of LoadPropertiesForItems() method.
Consider such scenario:
we have 10000 (ten thousands) messages in Inbox folder
we want to get approximately 30 properties of every message to see if they satisfy our conditions for further processing
messages are retrieved from server in packs of 100 messages
So, code looks like this:
ItemView itemsView = new ItemView(100);
PropertySet properties = new PropertySet();
properties.Add(EmailMessageSchema.From);
/*
add all necessary properties...
*/
properties.Add(EmailMessageSchema.Sensitivity);
FindItemsResults<Item> findResults;
List<EmailMessage> list = new List<EmailMessage>();
do
{
findResults = folder.FindItems(itemsView);
_service.LoadPropertiesForItems(findResults, properties);
foreach (Item it in findResults)
{
... do something with every items
}
if (findResults.NextPageOffset.HasValue)
{
itemsView.Offset = findResults.NextPageOffset.Value;
}
}while(findResults.MoreAvailable);
And the problem is that every increment of itemsView.Offset property makes LoadPropertiesForItems method longer to execute. For first couple of iterations it is not very noticeable but around 30th time loop makes that call time increases from under 1 second to 8 or more seconds. And memory allocation hits physical limits causing out of memory exception.
I'm pretty sure that my problems are "offset related" because I changed a code a little to that:
itemsView = new ItemView(100, offset, OffsetBasePoint.Beginning);
...rest of loop
if (findResults.NextPageOffset.HasValue)
{
offset = findResults.NextPageOffset.Value;
}
and I manipulated offset variable (declared outside of loop) in that way that I set its value on 4500 at start and than in debug mode after first iteration I changed its value to 100. And according to my suspicions first call of LoadPropertiesForItems took veeeery long to execute and second call (with offset = 100) was very quick.
Can anyone confirm that and maybe propose some solution for that?
Of course I can do my work without using an offset but why should I? :)
Changing the offset is expensive because the server has to iterate through the items from the beginning -- it isn't possible to have an ordinal index for messages because new messages can be inserted into the view in any order (think of a view over name or subject).
Paging through all the items once is the best approach.