I am new to completableFuture, what I am trying to do is I have this logic below in spring boot project which I am trying to convert it to the parallel processing approach using completableFuture.
#Transaction
void serviceMethod{
for(Object obj : objList) //objlist can have 10000 objects and its a multi level composite objects
{
//Get corresponding entity obj from the database, if not found throw user exception
//Process i.e. change some fields
}
}
In the above logic sinnce the method is annotated with #Transaction I am not calling JPA save explicitly to save the entity.
Now, I am trying to do parallel processing with the above logic.
#Transaction
void serviceMethod{
for(Object obj : objList) //objlist can have 10000 objects and its a multi level composite objects
{
//Get corresponding entity obj from the database, if not found throw user exception
CompletableFuture<Optional<Obj>> optionalObjFuture = CompletableFuture.supplyAsync( () -> {//get the object from repository})
CompletableFuture<Obj> objFuture = optionalObjFuture.thenApply( optionalObj -> {
if(obj.isPresent()){
return obj.get();
}
else{
throw user exception;
}
})
////Process i.e. change some fields
}
}
Now the question is
what is the approach I have to follow to break the for loop when there is an exception?
How to handle transaction in this scenario. Is there any way to handle transaction without the need to call saveAll on the processed objects stored in datastructure?
For (1), you could split those 10_000 calls into much smaller batches. For example:
public static void main(String[] args) throws Exception {
int innerIterations = 5;
ExecutorService service = Executors.newFixedThreadPool(innerIterations);
for(int i=0;i<10;++i) {
List<CompletableFuture<String>> list = new ArrayList<>();
int j = 0;
for(;j<innerIterations;++j) {
CompletableFuture<Optional<String>> cf = CompletableFuture.supplyAsync(() -> Optional.of("a"), service);
CompletableFuture<String> cf2 = cf.thenApplyAsync(x -> {
if(x.isPresent()) {
return x.get();
} else {
throw new RuntimeException("test");
}
}, Executors.newFixedThreadPool(2));
list.add(cf2);
}
i += j;
CompletableFuture<Void> all5 = CompletableFuture.allOf(list.toArray(new CompletableFuture[0]));
if(all5.isCompletedExceptionally()) {
// log exception, whatever
break;
} else {
List<String> set = list.stream().map(CompletableFuture::join).collect(Collectors.toList());
System.out.println(set);
}
}
}
This does a couple of things:
splits the initial 10 into two batches of 5.
call CompletableFuture::allOf on those 5 futures. Read the documentation to see that if at least one future fails from those 5, then all5 failed also.
if all5 did not fail (that is all5.isCompletedExceptionally() did not pass), call CompletableFuture::join to get all of the results. There will be no "joining" really, since the previous allOff already waited for them to be completed.
For the second question - you can't. You will need to create a transaction manually, but Spring makes it rather easy.
Related
Below is the sample code I'm using to understand exception handling in completablefuture in java8.
If we make use of exceptionally method as per doc,
exceptionally method catches even runtime exception as well and proceeds to last block in the pipeline.
if we don't use exceptionally method then, it justs prints running and exits.
Correct me if my understanding isn't correct.
Question is Lets say if i want to throw runtime exception and want application to stop. Basically if i throw Runtime exception , it shouldn't proceed to next block in pipeline. How should i do that. Any pointers are helpful.
public static void main(String[] args) {
final CompletableFuture<String> retrieveName = CompletableFuture.supplyAsync(() -> {
System.out.println("running");
int i = 0;
if(i == 0) {
throw new RuntimeException("ding");
}
return "test";
}).exceptionally(it -> {
System.out.println(it.getMessage());
return "empty";
}).thenApply(it -> {
System.out.println("last block" + it);
return "dummy";
});
}
Try this:
public static void main(String[] args) {
try {
final CompletableFuture<String> retrieveName = CompletableFuture.supplyAsync(() -> {
System.out.println("running");
int i = 0;
if (i == 0) {
throw new RuntimeException("ding");
}
return "test";
}).exceptionally(it -> {
if (it.getMessage().contains("ding")) {
throw (RuntimeException) it;
}
System.out.println(it.getMessage());
return "empty";
}).thenApply(it -> {
System.out.println("last block" + it);
return "dummy";
});
retrieveName.join();
} catch (Exception e) {
System.out.println("main() exception, cause=" + e.getCause());
}
}
This is the output:
running
main() exception, cause=java.lang.RuntimeException: ding
I made 3 small changes to your code:
Wrapped it all in a try-catch
Threw a RuntimeException in exceptionally() for the "ding" exception.
Added a call to retrieveName.join(). From the Javadoc for CompletableFuture.join():
public T join​()
Returns the result value when complete, or throws an (unchecked) exception if completed exceptionally.
Update based on OP feedback ------->
Lets say if i want to throw runtime exception and want application to
stop. Basically if i throw Runtime exception , it shouldn't proceed to
next block in pipeline. How should i do that.
You can achieve what you want with just 2 changes to your code:
[1] Completely remove the exceptionally() callback so the CompletableFuture (CF) terminates with an exception. In exceptionally() in the OP code the exception was being swallowed rather than rethrown, and returning a CF, so the thenApply() method was still performed.
[2] Add a call to retrieveName.join() at the end of main(). This is a blocking call, but since the thread had terminated with an exception that 's not really relevant for the sample code. The join() method will extract the thrown RunTimeException and re-throw it, wrapped in a CompletionException.
Here's your modified code:
public static void main(String[] args) {
final CompletableFuture<String> retrieveName = CompletableFuture.supplyAsync(() -> {
System.out.println("running");
int i = 0;
if(i == 0) {
throw new RuntimeException("ding");
}
return "test";
}).thenApply(it -> {
System.out.println("last block" + it);
return "dummy";
});
retrieveName.join();
}
Notes:
This is not how to do things in Production. The blocking call from join() was not a problem here, but could be for a long running CF. But you obviously can't extract the exception from the CF until it is complete, so it makes sense that the join() call blocks.
Always bear in mind that main() is not running in the same thread(s) as the CF.
An alternative approach (if viable) might be to handle all the necessary post-exception actions (logging, etc,) within exceptionally() and then terminate normally with a suitable return value (e.g. "Exception handled!") rather than propagating the exception.
You can check whether the CF is still running by calling the non-blocking isDone() method. You can also check whether the CF ended with an exception (isCompletedExceptionally()) or was cancelled(isCancelled​()).
I am trying to use map-reduce on CouchBase Lite. I have documents and they are being channelised. All doucments what I want are coming to Couchbase Lite. But When I Try to run map-reduce on them I am getting the following error
com.couchbase.lite.CouchbaseLiteException: Error when calling map block of view 'calendar', Status: 593 (HTTP 500 Application callback block failed)
Below is my map reduce function
private View createView(Database database){
View calendarView = database.getView("calendar");
calendarView.setMap(new Mapper() {
#Override
public void map(Map<String, Object> document, Emitter emitter) {
emitter.emit((long) document.get("date"),(long) document.get("cost"));
}
},"2");
return calendarView;
}
and Below is the part of main where I am calling the view and querying over it
View calendarView = createView(database);
Query query = database.getView("calendar").createQuery();
query.setStartKey(1472467249448l);
query.setEndKey(1472553649449l);
QueryEnumerator result = null;
try {
result = query.run();
} catch (CouchbaseLiteException e) {
e.printStackTrace();
}
for (Iterator<QueryRow> it = result; it.hasNext(); ) {
QueryRow row = it.next();
Log.d(TAG, row.getValue().toString());
}
As borrrden said in a comment above, Application callback block failed means your map function threw an exception. Use a debugger to find out what it is.
A likely possibility is that one of the documents in your database does not have a date property. In that case your map function would be passing null to the first (key) parameter of emit, which is an invalid argument.
I got this error when I executed the following code:
var insert = new TableBatchOperation();
foreach (var entity in entities)
{
insert.Insert(entity);
}
cloudTable.ExecuteBatch(insert);
Where the entities collection contained 512 elements.
The Azure SDK through a StorageException:
"Unexpected response code for operation : 99"
What does this error mean, and how can I solve this?
This un-descriptive error means that Azure bulk operations (at least in this case) takes up to 100 elements. Limit your batch and you'll be good.
I ended up using something like this:
public void Insert(IEnumerable<T> entities)
{
foreach (var chunk in entities.Chunk(100))
{
InsertMaxLimitElements(chunk);
}
}
private void InsertMaxLimitElements(IEnumerable<T> chunk)
{
var insert = new TableBatchOperation();
foreach (var entity in chunk)
{
insert.Insert(entity);
}
cloudTable.ExecuteBatch(insert);
}
The Chunk extension method was copied from this answer:
public static IEnumerable<IEnumerable<T>> Chunk<T>(this IEnumerable<T> source, int chunksize)
{
while (source.Any())
{
yield return source.Take(chunksize);
source = source.Skip(chunksize);
}
}
If the batch size is < 100, it could also mean that you have duplicate entries (two entries with the same rowkey). This is very likely if it succeeds when you write the entries individually but fails writing in batch.
I have a ASP.NET WebApi project that I am working on. The boss would like the returns to support "partial response", meaning that though the data model might contain 50 fields, the client should be able to request specific fields for the response. The reason being that if they are implementing for example a list they simply don't need the overhead of all 50 fields, they might just want the First Name, Last Name and Id to generate the list. Thus far I have implemented a solution by using a custom Contract Resolver (DynamicContractResolver) such that when a request comes in I am peeking into it through a filter (FieldListFilter) in the OnActionExecuting method and determining if a field named "FieldList" is present and then if it is I am replacing the current ContractResolver with a new instance of my DynamicContractResolver and I pass the fieldlist to the constructor.
Some sample code
DynamicContractResolver.cs
protected override IList<JsonProperty> CreateProperties(Type type, Newtonsoft.Json.MemberSerialization memberSerialization)
{
List<String> fieldList = ConvertFieldStringToList();
IList<JsonProperty> properties = base.CreateProperties(type, memberSerialization);
if (fieldList.Count == 0)
{
return properties;
}
// If we have fields, check that FieldList is one of them.
if (!fieldList.Contains("FieldList"))
// If not then add it, FieldList must ALWAYS be a part of any non null field list.
fieldList.Add("FieldList");
if (!fieldList.Contains("Data"))
fieldList.Add("Data");
if (!fieldList.Contains("FilterText"))
fieldList.Add("FilterText");
if (!fieldList.Contains("PageNumber"))
fieldList.Add("PageNumber");
if (!fieldList.Contains("RecordsReturned"))
fieldList.Add("RecordsReturned");
if (!fieldList.Contains("RecordsFound"))
fieldList.Add("RecordsFound");
for (int ctr = properties.Count-1; ctr >= 0; ctr--)
{
foreach (string field in fieldList)
{
if (field.Trim() == properties[ctr].PropertyName)
{
goto Found;
}
}
System.Diagnostics.Debug.WriteLine("Remove Property at Index " + ctr + " Named: " + properties[ctr].PropertyName);
properties.RemoveAt(ctr);
// Exit point for the inner foreach. Nothing to do here.
Found: { }
}
return properties;
}
FieldListFilter.cs
public override void OnActionExecuting(System.Web.Http.Controllers.HttpActionContext actionContext)
{
if (!actionContext.ModelState.IsValid)
{
throw new HttpResponseException(HttpStatusCode.BadRequest);
}
// We need to determine if there is a FieldList property of the model that is being used.
// First get a reference to the model.
var modelObject = actionContext.ActionArguments.FirstOrDefault().Value;
string fieldList = string.Empty;
try
{
// Using reflection, attempt to get the value of the FieldList property
var fieldListTemp = modelObject.GetType().GetProperty("FieldList").GetValue(modelObject);
// If it is null then use an empty string
if (fieldListTemp != null)
{
fieldList = fieldListTemp.ToString();
}
}
catch (Exception)
{
fieldList = string.Empty;
}
// Update the global ContractResolver with the fieldList value but for efficiency only do it if they are not the same as the current ContractResolver.
if (((DynamicContractResolver)GlobalConfiguration.Configuration.Formatters.JsonFormatter.SerializerSettings.ContractResolver).FieldList != fieldList)
{
GlobalConfiguration.Configuration.Formatters.JsonFormatter.SerializerSettings.ContractResolver = new DynamicContractResolver(fieldList);
}
}
I can then send a request with the json content payload looking as such:
{
"FieldList":"NameFirst,NameLast,Id",
"Data":[
{
"Id":1234
},
{
"Id":1235
}
]
}
and I will receive a response like so:
{
"FieldList":"NameFirst,NameLast,Id",
"Data":[
{
"NameFirst":"Brian",
"NameLast":"Mueller",
"Id":1234
},
{
"NameFirst":"Brian",
"NameLast":"Mueller",
"Id":1235
}
]
}
I believe that using the ContractResolver might run into threading issues. If I change it for one request is it going to be valid for all requests thereafter until someone changes it on another request (seems so through testing) If that is the case, then I don't see the usefulness for my purpose.
In summary, I am looking for a way to have dynamic data models such that the output from a request is configurable by the client on a request by request basis. Google implements this in their web api and they call it "partial response" and it works great. My implementation works, to a point but I fear that it will be broken for multiple simultaneous requests.
Suggestions? Tips?
A simpler solution that may work.
Create a model class with all 50 members with nullable types.
Assign values to the requested members.
Just return the result in the normal way.
In your WebApiConfig.Register() you must set the null value handling.
config.Formatters.JsonFormatter.SerializerSettings =
new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore };
You must not touch the configuration. You need the contract resolver on per-request basis. You can use it in your action method like this.
public class MyController : ApiController
{
public HttpResponseMessage Get()
{
var formatter = new JsonMediaTypeFormatter();
formatter.SerializerSettings.ContractResolver =
new DynamicContractResolver(new List<string>()
{"Id", "LastName"}); // you will get this from your filter
var dto = new MyDto()
{ FirstName = "Captain", LastName = "Cool", Id = 8 };
return new HttpResponseMessage()
{
Content = new ObjectContent<MyDto>(dto, formatter)
};
// What goes out is {"LastName":"Cool","Id":8}
}
}
By doing this, you are locking yourself into JSON content type for response messages but you have already made that decision by using a Json.NET specific feature. Also, note you are creating a new JsonMediaTypeFormatter. So, anything you configure to the one in the configuration such as media type mapping is not going to be available with this approach though.
I know this question is from many years ago, but if you're looking to do this with modern releases of the framework, I'd recommend nowadays to use OData services (http://www.asp.net/web-api/overview/odata-support-in-aspnet-web-api/using-select-expand-and-value).
for example:
class repository {
private DataContext db = new DataContext();
public IQueryable<Blah> someMethod(int id){
return from b in db.Blah ... select b;
}
public IQueryable<Blah> someMethod2(int id){
return from b in db.Blah ... select b;
}
public IQueryable<Blah> someMethod3(int id){
return from b in db.Blah ... select b;
}
}
OR
Should we make a new DataContext WITHIN each of those methods?
I think we are having some errors once user load increases due to us only having ONE DataContext per Repository instance, is this an accurate assumption?
See also the answer to this question.
In short, especially if you are using a repository pattern, you should create and dispose of a datacontext for each unit of work. Typically I use something like:
public someclass someMethod(int id)
{
using (var db = new SomeDataContext())
{
return db.FindMyClass(id);
}
}
What I did personally, is make the repository disposable. You then get constructs like:
void DeleteCustomer(int id)
{
using(var repos = GetRepos())
{
var customer = repos.GetAll<Customer>().Single(x => x.Id == id);
repos.Delete(customer);
repos.SaveChanges();
}
}
This can be implemented by creating the context in the repository ctor, and disposing it in the Dispose() implementation.
You need to make sure you're not adding/changing/delete objects and selecting from the same context. A context is made to 'last' a unit of work.
You need to be careful of stuff like this though:
IQueryable<Customer> GetCustomers()
{
using(var repos = GetRepos())
{
return repos.GetAll<Customer>();
}
}
void Test()
{
// This will throw an exception, because it extends the linq query
// while the context is disposed.
var customers = GetCustomers().Where(x => x.Id == 123);
}
In that case it's better to move the repository outside as much as possible:
IQueryable<Customer> GetCustomers(MyRepository repos)
{
return repos.GetAll<Customer>();
}
void Test()
{
using(var repos = ...)
{
var customers = GetCustomers(repos).Where(x => x.Id == 123);
}
}
I know this is not quite the same but, in a heavily used asp.net site using old fashioned data adapters, I used to open the connection to the database on page init and close it on page prerender. However I found that when the site was under more load the pages began to crawl. I read somewhere that it is always best to open as late as possible and close as early as possible.
Now I open the context in each method, however due to linq 2 sql and its deferred execution I am not sure if this makes much difference.
I would run the SQL profiler during busy moments to see where the bottle neck lies...