Currently I have a shiny web app that can do some calculations on a 3GB data.frame loaded in memory.
Now instead of implementing this feature on shiny web app, I need to make it a restful service that pipelines its calculation to another app in JSON format, so that people can use it by sending http request with url like http://my-app.com/function
I'm trying opencpu right now, but I don't quite understand how can I load and keep the large data in memory so that I can use api of opencpu to call functions in package just to do calculations, not to load the large data from disk every time I send http request.
One workaround might be to use hbase as in memory database and use rhbase to load the data. But before I invest time in learning it I want to know if it is a reasonable choice for 3GB data.frame since it might add more overhead in serialization and other stuff that offset its speed benefit.
What would be a better way to implement this functionality? Solutions using packages other than opencpu are also welcome, and it's better to be free.
You may want to look at Plumber. You decorate your R functions with comment code (that can include you loading data) and it makes it available via a REST API.
You should put your data in a package and add this package to preload in the server config.
Related
I have been looking everywhere for a solution to this problem.
At my work, we are trying to integrate Maximo with another system via the other systems REST API (which returns JSON responses). I am able to make this integration work on a small scale, however this API is taking upwards of 5 seconds to respond per request. Currently, I have defined this system as a JSON Resource, and I copy daily "snapshots" of the non-persistent data to a persistent attribute using an automation script. The requests all run in a sequence - which works slowly for 5 assets in testing, and will definitely not scale to 1000's of calls a day.
Assume that the API of the external system cannot be modified in any way... Is there a way to query this API in a non-blocking way? I'd imagine that if I could send a request, and send the next, etc. without needing to wait for a reply to proceed, this would solve the problem.
I looked into Invocation and Publishing Channels, and also Enterprise Services, and it seems like Enterprise Services along with JMS Queues might be what I need, however documentation says that these only support queuing incoming data... and I can't see how this solves my problem.
Any help? I am completely stuck on this.
Thank you!
I had to do something that sounds similar, once. I tried JSON Resources, but they didn't work for me. I ended up using the examples in Maximo 7.6 Scripting Features to do it. The first code sample in that document is a library script for making HTTP/S calls using out-of-the-Maximo-box libraries, and other examples in that document use IBM's JSONObject and JSONArray classes (also available out of the Maximo box) to parse responses.
To get things going concurrently / multithreaded, you could configure a cron task to call your automation script, and configure multiple instances on various schedules to call the same one and use the args or some other mechanism to prevent collisions.
I'm trying to build my first Sproutcore App and I struggle to connect it to a MySQL-Database or any datasource other than fixture. I can't seem to find ANY tutorial except this one from 2009 which is marked as deprecated: http://wiki.sproutcore.com/w/page/12413058/Todos%2007-Hooking%20Up%20to%20the%20Backend .
Do people usually not connect SC-Apps to a Database? If they do so, how do they find out how to? Or does the above mentioned tutorial still work? A lot of gem-commands in the introduction seems to already differ from the official Sproutcore getting-started-guide.
SproutCore apps, as client-side "in-browser" apps, cannot connect directly to a MySQL or any other non-browser database. The application itself runs only within the user's browser (it's just HTML, CSS & JavaScript once built and deployed) and typically accesses any external data via XHR requests to an API or APIs. Therefore, you will need to create a service wrapper around your MySQL database in order for your client-side app to be able to load and update data.
There are two things worth mentioning. The first is that since the SproutCore app contains all of your user interface and a great deal of business logic, your API can be quite simple and should only return raw data (such as JSON). The second is that, I should mention that the client-server design, while more tedious to implement, is absolutely necessary in practice, because you can never trust the client side code, which is in the hands of a possibly nefarious user. Therefore, your API should also act as the final gatekeeper to validate all requests from the client.
This tutorial I found helped me a lot. Its very brief and demonstrates how to implement a very simple login-app, how to send post-requests (triggered by the login-button-action) to the backend-server and how to asynchronously process the response inside the Sproutcore-App:
http://hawkins.io/2011/04/sproutcore_login_tutorial/
My application uses GWT-RPC to communicate to the server. Is there anyway to transparentlly serialzie my data using JSON without changing the RPC layer?
IMHO this could be acheived by changing the serializers and using autobean codex in the UI.
Why do I need that?
I want to make cross domain RPC calls
I want to call the server side from a non GWT-app without providing an extra layer in server side.
UPDATE I just stumbled upon http://code.google.com/p/gerrit/source/browse/README?repo=gwtjsonrpc which is actively maintained (as it's part of Gerrit, the code-review tool used by the Android team)
Have a look at http://code.google.com/p/gwt-rpc-plus/ but it's no longer maintained...
If you really need it and don't want to move away from GWT-RPC, then replacing GWT's serializers should be possible: that's exactly what deRPC (com.google.gwt.rpc) does re. standard GWT-RPC (com.google.gwt.user.rpc), but it needs to do a bit more than that (namely: generate the serialization code for the client-side, as there's no reflection at runtime).
This going to be a hard task. I don't think the changing of serializers will work, GWT-RPC serializers are working with input as with a stream (basically data sent from server are in fact in JSON format, but they can be parsed only by GWT-RPC). You will have to create totally new generator, which will create code for parsing and object serialization/deserialization. AutoBean framework might be very helpful in this case. In the end you should be able to to migrate from GWT-RPC serialization to some other protocol without actually changing current code, which is using GWT-RPC services.
The biggest problem is cross-domain messaging. Normally you would use JSONP, but the problem is that JSONP basically allows only GET requests, if you need send a lot of data to the other server, you might not be able to fit everything into single requests. You can solve such problem with cross domain document messaging (e.g. you will open iframe, which will load special communication javascript from remote server, and you will use this iframe as proxy for your service via postMessage) , but this feature is not supported in IE7.
I want to make a django server to refresh the content that you approach the database, if the idea is to first make the user see the current contents of the database and as the valley became the new content, this content comes and is placed above the previous content without reloading the page, in another part of the site is to make you change the current content with the new as it gets to the database?
evserver clearer is my choice, but really do not know how and what would be the most simple and efficient?
I think you should avoid HTTP Polling. Here's why:
if the frequency of the setInterval combined with the number of users on your web app is going to lead to a big resource drain. If you go through slides 9 to 19 in this presentation you'll see some quite dramatic figures for using Push (Note: this example uses a hosted service but hosting your own realtime server and using Push also has similar benefits)
between setInterval calls the data displayed in your app is potentially out of data. Using a Push technology means the instant that new data is available it can be push and displayed in your app. You don't want users looking at an app and thinking they are seeing correct information when they are not.
You should take a the following StackOverflow questions:
Django / Comet (Push): Least of all evils?
Need help understanding Comet in Python (with Django)
For Python/Comet see:
Python Comet Server
The latest recommendation for Comet in Python?
I'd recommend you also start considering "WebSockets" as well as "Comet". Most Comet servers now prefer to use a WebSocket connection when possible.
If you'd prefer to avoid installing and managing your own Comet/WebSocket solution then you could use a realtime hosted service which will allow you Push data through them using a REST API and your clients can receive events by embedding a JavaScript library and writing a small about of code to subscribe and receive the event.
The steps are quite straightforward:
Write a model to store data in DB
Write a view that will generate JSON-serialized data upon POST request.
Write a template that will contain JavaScript with setInterval() that will
proceed AJAX requests to the view and render recieved data. (I'd suggest using JQuery as it's well documented and widespread).
Can anyone explain to me the benefits of using XML-RPC over a straight HTML Form ?
On first glance, they seem to accomplish the same thing.
The XML-RPC is "formatted" using XML, but you can do the same in a form (think textbox).
I have an app that takes data from a script that runs on the client PC. The output from the script is XML. Currently it is submitted to the app (PHP using CodeIgniter) via a form POST. I have been told to look at using XML-RPC, but I am trying to understand "why"...
I primary benefit of XML-RPC is that you don't have to write any (or nearly any) glue code to get remote processes to communicate. There are a wide variety of XML-RPC client libraries available for many languages. In the case that you have a rich API of functions, XML-RPC can be a very easy way connect remote processes to that API. Performing the same task with a plain www-form-encoded POST will require you to convert an API to a form and dispatch requests into api calls. There are a few systems that can help with that, but it is certain to be more difficult than just exposing the API through XML-RPC.
On the other hand, if (as it sounds) you already have a rich API exposed through plain form encoded requests, It's hard to justify the work of porting both client and server to another interface.
If you mean encoding data in the www-form-encoded (or "query string style") format vs POSTing XML-RPC, it's really a matter of taste. Some people prefer to use XML for everything.
XML-RPC is a bit dated, in my opinion. Unless you're talking about a different protocol, this is the one that preceeded SOAP. I wouldn't use it at all. Instead, I'd either use a SOAP-based service, or a REST-based one.
HTML forms are not comparable to XML-RPCs.
You use HTML forms to get some input data through a WebPage, while you use XML-RPC to execute remote procedures using XML.
Usually you will use XML-RPC for a process without a UI trying to execute some procedure remotely.