I have an optimization algorithm deployed as a live deployment. It takes a set of objects and returns a set of objects with potentially different size. This works just fine when I'm using the REST api.
The problem is, I want to let the user of the workshop app query the model with a set of objects. The returned objects need to be written back to the ontology.
I looked into an action backed function but it seems like I can't query the model from a function?!
I looked into webhooks but it seems to not fit the purpose and I also would need to handle the API key and can't write back to the ontology?!
I know how to query the model with scenarios but it is per sample and that does not fit the purpose, plus I cant write back to the ontology.
My questions:
Is there any way to call the model from a function and write the return back to the ontology?
Is there any way to call a model from workshop with a set of objects and write back to the ontology?
Is modeling objectives just the wrong place for this usecase?
Do I need to implement the optimization in Functions itself?
I answered the questions, as well I tried to address some of the earlier points.
Q: "I looked into an action backed function but it seems like I can't query the model from a function?!"
A: That is correct, at this time you can't query a model from a function. However there are javascript based linear optimization libraries which can be used in a function.
Q: "I looked into webhooks but it seems to not fit the purpose and I also would need to handle the API key and can't write back to the ontology?!"
A: Webhooks are for hitting resources on networks where a magritte agent are installed. So if you have like a flask app on your corporate network you could hit that app to conduct the optimization. Then set the webhook as "writeback" on an action and use the webhook outputs as inputs for a ontology edit function.
Q: "I know how to query the model with scenarios but it is per sample and that does not fit the purpose, plus I cant write back to the ontology."
A: When querying a model via workshop you can pass in a single object as well as any objects linked in a 1:1 relationship with that object. This linking is defined in the modeling objective modeling api. You are correct to understand you can't pass in an arbitrary collection of objects. You can write back to the ontology however, you have to set up an action to apply the scenario back to the ontology (https://www.palantir.com/docs/foundry/workshop/scenarios-apply/).
Q: "Is there any way to call the model from a function and write the return back to the ontology?"
A: Not from an ontology edit function
Q: "Is there any way to call a model from workshop with a set of objects and write back to the ontology?"
A: Only object sets where the objects have 1:1 links within the ontology. You can writeback by appyling the scenario (https://www.palantir.com/docs/foundry/workshop/scenarios-apply/).
Q: "Is modeling objectives just the wrong place for this usecase? Do I need to implement the optimization in Functions itself?"
A: If you can write the optimization in an ontology edit function it will be quite a bit more straightforward. The main limitation of this is you have to use Typescript which is not as commonly used for this kind of thing as Python. There are some basic linear optimization libraries available for JS/TS.
I have dash app connected to an AWS RDS. I have a live-updated graph that triggers a callback with a n_interval of 5min to query the database and do some expensive formatting. I store the transformed data (~500 data points) in a dcc.store from which another 6 graphs and a datatable utilize this data (no further processing required. My question is: To further improve the efficiency of the dashboard should I utilize client-side callbacks instead of dcc.store? Since from what I've read the client-side only utilizes the client browser and doesn't need to communicate back to the dash server on callback? Thank you.( I'm secretly hoping it makes little difference as I don't want to learn javascript)
The answer depends a lot on your architecture, how complex are the graphs and how many server side callbacks are currently used.
You have already mentioned the main advantage of clientside callbacks: since the data is on the clientside, it saves some time and network by updating the components directly on the client. The major inconvenience is that those are synchronous calculations and will block the app main thread (dash does not support promises and asynchronous for now), which Can lead to a bad UX if these functions take too much time.
For very simple plots and table, I can guarantee that there is a significant improvement in using clientside callbacks, especially because it avoids the Plotly API that can be much slower than just defining a JSON object with traces and layout.
I'm following official docs guide to write a Scala script for launching EMR cluster using AWS Java SDK. I'm able to identify 3 major steps needed here:
Instantiating an EMR Client
I do this using AmazonElasticMapReduceClientBuilder.defaultClient()
Creating a JobFlowRequest
I create a RunJobFlowRequest object and supply it with JobFlowInstancesConfig (both objects are supplied with appropriate parameters depending on the requirement)
Running JobFlowRequest
This is done by calling emrClient.runJobFlow(runJobFlowRequest) which returns a RunJobFlowResult object
But RunJobFlowResult object doesn't provide any clue as to whether the cluster was launched successfully or not (with all the given configurations)
Now I'm aware that listClusters() method of the emrClient can be used to get cluster id of the newly-launched cluster through which we can query the state of the cluster using describeCluster() call. However since I'm using a Scala script to perform all this stuff, I need the process to be automated (here looking up the cluster id in the result of getClusters() will have to be done manually)
Is there any way this could be achieved?
You have all the pieces there but haven't quite stitched them together.
The cluster's id can be retrieved from RunJobFlowResult.getJobFlowId(). (It is a string starting with "j-".) Then you can pass this jobFlowId to DescribeCluster.
I don't blame you for your confusion though, since it's called "jobFlowId" for some methods (mainly older API methods) and "clusterId" in other methods. They are really the same thing though.
I'm using a dynamic language (Clojure) to create CUDA contexts in a interactive development way using JCuda. Often I will call an initializer that includes the call to jcuda.driver.JCudaDriver/cuInit. Is it safe to call cuInit multiple times? In addition, is there something like a destroy method for cuInit? I ask since its possible for an error code CUDA_ERROR_DEINITIALIZED to be returned.
To answer the question, yes it is probably safe to call cuInit multiple times. I haven't noticed any side effects from doing so.
Note, however, that cuInit only triggers one-time initialisation processes inside the API. It doesn't do anything with devices, or contexts and it definitely can't return CUDA_ERROR_DEINITIALIZED. Doing the steps you would do after calling cuInit in an application (ie. creating a context) would have real implications - doing so creates a new context each time you call it and resource exhaustion will occur if contexts are not actively destroyed. There is no equivalent deinitialisation call for the API. I guess the intention is that once intialised, the runtime API is expected to stay in that state until an application terminates.
For example, what does it mean in this quote?
Integrating with an external API is almost a guarantee in any modern web app. To effectively test such integration, you need to stub it out. A good stub should be easy to create and consistently up-to-date with actual, current API responses. In this post, we’ll outline a testing strategy using stubs for an external API.
A stub is a controllable replacement for an Existing Dependency (or collaborator)
in the system. By using a stub, you can test your code without
dealing with the dependency directly.
External Dependency - Existing Dependency:
It is an object in your system that your code
under test interacts with and over which you have no control. (Common
examples are filesystems, threads, memory, time, and so on.)
Forexample in below code:
public void Analyze(string filename)
{
if(filename.Length>8)
{
try
{
errorService.LogError("long file entered named:" + filename);
}
catch (Exception e)
{
mailService.SendEMail("admin#hotmail.com", "ErrorOnWebService", "someerror");
}
}
}
You want to test mailService.SendEMail() method, but to do that you need to simulate an Exception in your test method, so you just need to create a Fake Stub errorService object to simulate the result you want, then your test code will be able to test mailService.SendEMail() method. As you see you need to simulate a result which is from an another Dependency which is ErrorService class object (Existing Dependency object).
A stub, in this context, means a mock implementation.
That is, a simple, fake implementation that conforms to the interface and is to be used for testing.
Layman's terms, it's dummy data (or fake data, test data...etc.) that you can use to test or develop your code against until you (or the other party) is ready to present/receive real data. It's a programmer's "Lorem Ipsum".
Employee database not ready? Make up a simple one with Jane Doe, John Doe...etc.
API not ready? Make up a fake one by creating a static .json file containing fake data.
In this context, the word "stub" is used in place of "mock", but for the sake of clarity and precision, the author should have used "mock", because "mock" is a sort of stub, but for testing. To avoid further confusion, we need to define what a stub is.
In the general context, a stub is a piece of program (typically a function or an object) that encapsulates the complexity of invoking another program (usually located on another machine, VM, or process - but not always, it can also be a local object). Because the actual program to invoke is usually not located on the same memory space, invoking it requires many operations such as addressing, performing the actual remote invocation, marshalling/serializing the data/arguments to be passed (and same with the potential result), maybe even dealing with authentication/security, and so on. Note that in some contexts, stubs are also called proxies (such as dynamic proxies in Java).
A mock is a very specific and restrictive kind of stub, because a mock is a replacement of another function or object for testing. In practice we often use mocks as local programs (functions or objects) to replace a remote program in the test environment. In any case, the mock may simulate the actual behaviour of the replaced program in a restricted context.
Most famous kinds of stubs are obviously for distributed programming, when needing to invoke remote procedures (RPC) or remote objects (RMI, CORBA). Most distributed programming frameworks/libraries automate the generation of stubs so that you don't have to write them manually. Stubs can be generated from an interface definition, written with IDL for instance (but you can also use any language to define interfaces).
Typically, in RPC, RMI, CORBA, and so on, one distinguishes client-side stubs, which mostly take care of marshaling/serializing the arguments and performing the remote invocation, and server-side stubs, which mostly take care of unmarshaling/deserializing the arguments and actually execute the remote function/method. Obviously, client stubs are located on the client side, while sever stubs (often called skeletons) are located on the server side.
Writing good efficient and generic stubs becomes quite challenging when dealing with object references. Most distributed object frameworks such as RMI and CORBA deal with distributed objects references, but that's something most programmers avoid in REST environments for instance. Typically, in REST environments, JavaScript programmers make simple stub functions to encapsulate the AJAX invocations (object serialization being supported by JSON.parse and JSON.stringify). The Swagger Codegen project provides an extensive support for automatically generating REST stubs in various languages.
Stub is a function definition that has correct function name, the correct number of parameters and produces dummy result of the correct type.
It helps to write the test and serves as a kind of scaffolding to make it possible to run the examples even before the function design is complete
This phrase is almost certainly an analogy with a phase in house construction — "stubbing out" plumbing. During construction, while the walls are still open, the rough plumbing is put in. This is necessary for the construction to continue. Then, when everything around it is ready enough, one comes back and adds faucets and toilets and the actual end-product stuff. (See for example How to Install a Plumbing Stub-Out.)
When you "stub out" a function in programming, you build enough of it to work around (for testing or for writing other code). Then, you come back later and replace it with the full implementation.
You have also a very good testing frameworks to create such a stub.
One of my preferrable is Mockito There is also EasyMock and others... But Mockito is great you should read it - very elegant and powerfull package
RPC Stubs
Basically, a client-side stub is a procedure that looks to the client as if it were a callable server procedure.
A server-side stub looks to the server as if it's a calling client.
The client program thinks it is calling the server; in fact, it's calling the client stub.
The server program thinks it's called by the client; in fact, it's called by the server stub.
The stubs send messages to each other to make the RPC happen.
Source
"Stubbing-out a function means you'll write only enough to show that the function was called, leaving the details for later when you have more time."
From: SAMS Teach yourself C++, Jesse Liberty and Bradley Jones
Stub is a piece of code which converts the parameters during RPC (Remote procedure call).The parameters of RPC have to be converted because both client and server use different address space. Stub performs this conversion so that server perceive the RPC as a local function call.
A stub can be said as a fake substitute of the original function, which gives output, which is not accessible right now because of reasons:
It is not developed right now
It is not callable from the current environment (maybe testing)
A stub has :
Exact number of parameters
Exact output format (not necessarily correct output)
Why a stub is used?
When function is not accessible in an environment such as testing, or when its implementation is not available.
Example:
let's say you want to test a function in which there's a network call. While testing the code, you cannot wait for a network call's result for your test. so you write a mock output of the network call and proceed with your test.
TestFunction(){
// Some things here
// Some things here
var result = networkCall(param)
// something depending on the result
}
This networkCall gives out lets say a string, so you have to create a function with exact same parameters and it should give string output.
String fakeNetworkCall(int param){
if(param == 1) return "OK";
else return "NOT OK";
}
Now you have written a fake function, use this as replacement in your code
TestFunction(){
// Some things here
// Some things here
var result = fakeNetworkCall(param)
// something depending on the result
}
This fakeNetworkCall is a stub.