I am trying to return large result set from WCF service. Large result set would probably have approax 500K records and each record will have 150 columns.
I know by configuring WCF bindings we can return large result set. But I am not sure about the limit. I tried this scenario but got error to increase the limit of "maxItemsInObjectGraph" property even I set the value as "2147483647" of this property. I googled the alternate options and found that this can be achieved with messageEncoding and transferMode property of Binding. I tried with "Mtom" and "StreamResponse" but I am not sure how it is working?
I also referred this link but dont want to make pagination as my client wants data in one go.
So Conclusion is:
1. Can we return large result set from WCF? Or Does the use of WCF best to return large result set or I need to move to different way like WebAPI?
2. StreamResponse should work like returning results in chunck I guess but after implementation, I don't think it is working as I am getting result altogether.
Your inputs will be appreciated.
Thanks!!
You can return large object sets from WCF service. WebAPI or other .NET service framework doesn't have advantage considering transferring a lot of data.
StreamResponse is the best choice for you. It transfers message portion by portion, so that you don't have all your 500K records on the wire at one moment. It's a transport layer option, hence for client it seems like he got the whole message at once. You can manage that portion size by editing MaxBufferSize property of your binding.
To minimize the size of data you can also compress it using BinaryMessageEncoding.
<customBinding>
<binding name="primaryBinding" openTimeout="00:01:00" closeTimeout="00:01:00"
sendTimeout="00:30:00" receiveTimeout="00:30:00">
<binaryMessageEncoding compressionFormat="GZip">
<readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647"
maxArrayLength="2147483647" maxBytesPerRead="2147483647"
maxNameTableCharCount="2147483647" />
</binaryMessageEncoding>
<httpsTransport transferMode="Streamed" maxReceivedMessageSize="6000000"
maxBufferSize="6000000" maxBufferPoolSize="12000000" />
</binding>
Related
I have phonorgraph object with billions of rows and we are querying it through object set service
for example, I want to get all DriverLicences from certain city.
#Function()
public getDriverLicences(city: string): ObjectSet<DriverLicences> {
let drivers = Objects.search().DriverLicences().filter(row => row.city.exactMatch(city));
return drivers ;
}
I am facing this error when I am trying query it from slate:
ERROR 400: {"errorCode":"INVALID_ARGUMENT","errorName":"ObjectSet:PagingAboveConfiguredLimitNotAllowed","errorInstanceId":"0000-000","parameters":{}}
I understand that I am probably retrieving more than 100 000 results but I need all the results because of the implemented logic in the front is a complex slate dashboard built by another team that we cannot re-factor.
The issue here is that, specifically in the Slate <> Function connector, there is a "translation layer" that serializes the contents of the object set and provides a response data structure that materializes the property:value pairs for each object in the set.
This clearly doesn't work for large object sets where throwing so much data into the browser is likely to overwhelm the resources allocated to the tab.
From context it seems like you might be migrating an existing Slate app over to Functions; in the current version, how is the query limiting the number of results returned? It certainly must not be returning several 100 thousand results for further processing on the front end? (And if so, that might be an anti-pattern to consider addressing).
As for options that you could currently explore, you can sort your object set and then specify a smaller limit to return:
Objects.search().DriverLicences().filter(row => row.city.exactMatch(city)).orderBy(date_of_issue).take(100)
You'll find a few more details in the Functions documentation Reference entry on Ontology API: Object Sets in the section on Ordering and limiting.
You can even make a work around for the (current) lack of paging when return an ObjectSet to Slate by using the last value from the property ordered on (i.e. date_of_issue) as a filter in the subsequent request and return the next N objects.
This can work if you need a Slate table or HTML widget that renders on set of results then, on a user action, gets the next page.
I am trying to retrieve the properties using this method: GET :urn/metadata/:guid/properties
This is something that we have running and works daily in our workflows, but I think this is an especailly large model.
For this particular model we are getting the following repsonse:
413 Request Entity Too Large
{Diagnostic": "Please use the 'forceget' parameter to force querying the data."}
Can anyone advise me as to how I do apply the forceget parameter to this call as I can't see any mention of it in the api docs.
forceget (string): To force get the large resource even if it exceeded the expected maximum length. Possible values: true, false. The implicit value is false.
The maximum data size 'without' forceget=true, is 2097152 bytes gzip compressed.
If you add 'forceget' then there is no size limit, however, if the generation of the gzip file takes longer than 2 hours, it will time out and try again later and eventually give up and report an error.
More details in this blog post: https://forge.autodesk.com/blog/faster-get-hierarchy-api-and-how-solve-error-413
#augusto-goncalves. Do you know what is the max number of parameters that are possible to get through a request with 'forceget' param?
... and what is the maximum number of properties we might have in the model so that we can retrieve all properties from that endpoint WITHOUT using 'forceget' parameter?
This issue is bugging me for some time now. To test it I just installed a fresh Apigility, set the db (PDO:mysql) and added a DB-Connected service. In the table I have 40+ records. When I make a GET collection request the response looks OK (with the default HAL content negotiation). Then I change the content negotiation to JSON. Now when I make a GET collection request my response contains only 10 elements.
So my question is: where do I set/change this limit?
You can set the page size manually, like so:
$paginator = $this->getAlbumTable()->fetchAll(true);
// set the current page to what has been passed in query string, or to 1 if none set
$paginator->setCurrentPageNumber((int) $this->params()->fromQuery('page', 1));
// set the number of items per page to 10
$paginator->setItemCountPerPage(10);
http://framework.zend.com/manual/current/en/tutorials/tutorial.pagination.html
Could you please send the page_size, total_items part at the end of the json output?
it's like:
"page_count": 140002,
"page_size": 25,
"total_items": 3500035,
"page": 1
This is not an ideal fix, because it requires you to go into the source code rather than using the page size given in the UI.
The collection class that is auto generated for you by the DB-Connected style derives off of Zend/Paginator/Paginator. This class defines the $defaultItemCountPerPage static protected member which is defaulted to 10. That's why you're only getting 10 results. If you open up the auto-generated collection class for your entity and add: protected static $defaultItemCountPerPage = 100; in the otherwise empty class, you will see that you now get up to 100 results in the response. You can look at other Paginator class variables and methods that you could replace in your derived class to get your desired behavior.
This is not an ideal solution. I'd prefer that the generated code automatically used the same configed page size that the HalJson strategy uses. Maybe I'll contribute a PR to change that. Or, maybe I'll just use the HalJson approach. It does seem like the better way to go. You should have some limit to how much data you load in from the DB at a time to not have an overly long running query or an overly large collection of data coming back you have to deal with. And, whatever limit you set, what do you do when you hit that limit? With the simple Json method, you can't ever get "page 2" of data. So, if you are going to work with some sizeable amount of data, it might be better to use HalJson on and then have some logic on the client side to grab pages of data at a time as needed. The returned JSON structure is a little more complicated, but not terribly so.
I'm probably in the same spot you are -- I'm trying to do a simple little api to play with while keeping everything simple and so I didn't want the client to have to deal with the other stuff in HalJson, but probably better to deal with that complexity and have a smooth way to page through data if you're going to use this with some real set of data. At least, that's the pep talk I'm giving myself right now. :-)
I use a MagicalRecord, and am having a little trouble using it.
A server sends me a JSON, and I need it as quickly as possible ro map to the existing NSManagedObject and give it to the block.
NSManagedObjectContext *localContext = [NSManagedObjectContext MR_contextForCurrentThread];
[Review MR_importFromObject:dictionary inContext:localContext];
[localContext MR_saveOnlySelfAndWait];
And after:
[[CacheOperation sharedOperation]saveBestRateProductByDict:reviewDict];
Review *review = [Review MR_findFirstByAttribute:#"id" withValue:[reviewDict objectForKey:#"id"]];
But if I have many objects, it takes a lot of time.
How I can map NSDictionary to NSManagedObject without writing to CoreData?
I guess MR_importFromObject check for the existence of the object to have an insert or update behavior.
That's great for most cases. (and it was made for the 90% http://www.cimgf.com/2012/05/29/importing-data-made-easy/ )
But you are in the 10% (me too, if it can be of any support)
This behavior mean that there is a request to find the object, and a request to update it. Multiple by the number of objects, that can be huge.
You can refer to a good apple doc (the part Implementing Find-or-Create Efficiently ) :
https://developer.apple.com/library/ios/documentation/Cocoa/Conceptual/CoreData/Articles/cdImporting.html
One trick is to make only one request for all the objects you want to update, and one request to update them all. It's worse in memory usage but better in I/O, that should speed you up.
We went another way using TMCache and storing raw JSON for objects changing frequently.
Hope this help.
Forgive me, I'm very new to using REST.
Currently I'm using SP2013 Odata (_api/web/lists/getbytitle('<list_name>')/items?) to get the contents of a list. The list has 199 items in it so I need to call it twice and each time ask for a different set of items. I figured I could do this by calling:
_api/web/lists/getbytitle('<list_name>')/items?$skip=100&$top=100
each time changing however many I need to skip. The problem is this only ever returns the first 100 items. Is there something I'm doing wrong or is $skip broken in the OData service?
Is there a better way to iterate through REST calls, assuming this way doesn't work or isn't practical?
I'm using the JSon protocol with the Accept Header equaling application/json;odata=verbose
I suppose the $top=100 isn't really necessary
Edit: I've looked it up more and, I'm not entirely sure of the terms here, but using $skip works fine if you're using the method introduced with SharePoint 2010, i.e., _vti_bin/ListData.svc/<list_name>?$skip=100
Actually, funny enough, the old way doesn't set a 100 item limit on returns. So skip isn't even necessary. But, if you'd like to only return a certain segment of data, you'd have to do something like:
_vti_bin/ListData.svc/<list_name>?$skip=x&$top=(x+y)
where each time through the loop you would have something like x+=y
You can either use the old method which I described above, or check out my answer below for an explanation of how to do this using SP2013 OData
Alright, I've figured it out. $skip isn't a command which is meant to be used at the items? level. It works only at the lists? level. But, there's a way to do this, actually much easier than what I wanted to do.
If you just want all the data
In the returned data, assuming the list you are calling holds more than 100 items, there will be a __next at d/__next (assuming you are using json). This __next (it is a double underscorce, keep that in mind. I had a few problems at first because I was trying to get d/_next which never returned anything) is the right URL to get the next set of items. __next will only ever be a value if there is another set of items available to get.
I ended up creating a RequestURL variable which was initially set to to original request, but was changed to d/__next at the end of the loop. Then the loop went and checked if the RequestURL was not empty before going inside the loop.
Forgive my lack of code, I'm using SharePoint Designer 2013 to make this, and the syntax isn't horribly descriptive.
If you'd only like a small set of data
There's probably a few situations where you would only want x amount of rows from your list each time you go through the loop and that's real easy to do as well.
if you just add a $top=x parameter to your request, the __next URL that comes back with the response will give you the next x rows from your list. Eventually when there are no rows left to return __next won't be returned with the response.
Don't forget that in order to use __next you need to have a
$skiptoken=Paged=TRUE
in the url as well.