Caffeine cache, perform eviction of expired element only when put succeds - caffeine

I use Caffeine cache to cache data coming from the DB that need to served to a rest endpoint.
So the cache is updated ONLY on read operations on the DB.
In the nominal case, I want the cache to take the lead of responding until the data is not older than some point of time ( => this case is ok by defining the correct expiration options)
In case of DB access failure, I want to fallback to the cache even if the data in the cache is expired.
This use case supposes that expired data are not yet removed.
The solution I'm thinking of is to make Eviction of Items from the cache only after a successful PUT (a successful PUT means the DB is working correctly).
Is that possible to do?

This could be accomplished by using a victim cache to capture recently expired entries and resurrect from it on a load if the db is down. The victim cache would need its own bounding, e.g. a longer expiration threshold.
Cache<K, V> victimCache = Caffeine.newBuilder()
.expireAfterWrite(10, TimeUnit.MINUTE)
.build();
LoadingCache<K, V> mainCache = Caffeine.newBuilder()
.expireAfterWrite(1, TimeUnit.MINUTE)
.writer(new CacheWriter<K, V>() {
public void write(K key, V value) { /* ignored */ }
public void delete(K key, V value, RemovalCause cause) {
if (cause == RemovalCause.EXPIRED) {
victimCache.put(key, value);
}
})
.build(key -> {
try {
// load from db
} catch (DatabaseAccessException e) {
return victimCache.asMap().remove(key);
}
});
This would allow a load failure to resurrect the entry from the victim cache, if the entry is present.

Related

Spring Boot+ MySQL: LAZY Loading issue - No operations allowed after statement closed

I implement a Dashboard functionality that checks every time at program start a list of Requirement-Objects for a bunch of different characteristics like progress, missing data and alike and sets for each characteristic a dedicated beacon on the UI.
protected void initializePerformanceIndicator() {
try {
updateA();
updateB();
...
updateF();
updateG();
} catch (Exception e) {
ErrorHandler.showError("Cannot show KPI Performance", e);
}
}
The checks have different compute demands some are faster some slower, therefore each of this checks runs in a dedicated Task to provide some feedback to the user. The skeleton of such a Task is always the same
protected void updateA() throws Exception {
Task<Void> task = new Task<Void>() {
#Override
protected Void call() throws Exception {
embeddedBudgetKPIController.setHyperlink("Budget", null);
embeddedBudgetKPIController.setToolTip("...");
ObservableList<UserRequirement> issues = FXCollections.observableArrayList();
List<UserRequirement> requirements = reqService.getAllUserRequirements(false); // all requirements of the selected product
for(UserRequirement req: requirements) {
if(*some criteria*) {
issues.add(req);
}
}
if(issues.isEmpty()) {
embeddedBudgetKPIController.setBeaconColor(Color.GREEN);
} else {
embeddedBudgetKPIController.setBeaconColor(Color.RED);
}
return null;
};
};
task.setOnSucceeded(e -> {
// Nothing to do
});
Thread tt = new Thread(task);
tt.start();
}
Before initializePerformanceIndicator is called, I retrieved already elsewhere the data from the database querying a number Spring Repositories:
protected final ObservableList<UserRequirement> allUserRequirements = FXCollections.observableArrayList();
public synchronized ObservableList<UserRequirement> getAllUserRequirements(boolean forceUpdate) throws Exception {
logger.debug(""); // show that this method is called
Product selectedProduct = SelectedScope.getSelectedProduct();
if(selectedProduct == null) {
throw new Exception("No selProduct selected");
}
if(forceUpdate || allUserRequirements.isEmpty()) {
allUserRequirements.clear();
allUserRequirements.addAll(epicRepository.findByProductAndRevisionSuccessorIsNull(selectedProduct));
allUserRequirements.addAll(themeRepository.findByProductAndRevisionSuccessorIsNull(selectedProduct));
allUserRequirements.addAll(userStoryRepository.findByProductAndRevisionSuccessorIsNull(selectedProduct));
allUserRequirements.addAll(tangibleRepository.findByProductAndRevisionSuccessorIsNull(selectedProduct));
}
return allUserRequirements;
}
and as you see updateBudgetKPIController calls getallUserRequirements with the parameter false. Therefore it returns the buffered result set and is not re-fetching data from database. So far everything is fine.
I can run each of these Tasks individually without problem. I tried a number combinations with 2 Tasks. Works fine, but the program will never show more than three or four beacons. Which ones are shown differs as well - what is expected as a consequence of the different Tasks. If I exceed three or four Tasks I often get no error at all, but the UI is just not showing more than three to four beacons.
Sometimes I do get an error message, which is
WARN 08:14 o.h.e.j.s.SqlExceptionHelper.logExceptions:137: SQL Error: 0, SQLState: S1009
ERROR 08:14 o.h.e.j.s.SqlExceptionHelper.logExceptions:142: No operations allowed after statement closed.
I debugged it, and realized that I was generating way too many select statements. The UserRequirement entity has almost a dozen OneToMany relations, some where defined with FetchType.LAZY, so I thought it would be better anyway to configure all these relations as
#OneToMany(fetch = FetchType.LAZY, mappedBy="parent", cascade = CascadeType.ALL)
Because of the LAZY loading, every Task tries to load additional data in the if(*some criteria*) part.
The problem did not disappear but I get more information, as the error is now
WARN 11:02 o.h.c.i.AbstractPersistentCollection.withTemporarySessionIfNeeded:278: Unable to close temporary session used to load lazy collection associated to no session
WARN 11:02 o.h.e.j.s.SqlExceptionHelper.logExceptions:137: SQL Error: 0, SQLState: S1009
ERROR 11:02 o.h.e.j.s.SqlExceptionHelper.logExceptions:142: No operations allowed after statement closed.
So I do have a LAZY loading issue.
I am using Spring Boot 2.1.6, MySQL 8.0.15 Community Server, Hibernate Core {5.3.10.Final}, Java 1.8.0_211 and the com.mysql.cj.jdbc.Driver
From a former issue, I have in my properties file the following configuration
# Prevent LazyInitializationException
spring.jpa.properties.hibernate.enable_lazy_load_no_trans=true
Don't know whether this has a side effect?!
Probably changing the LAZY loading to EAGER will fix it - haven't tried yet - but it would delay program start significantly. Therefore I would prefer a solution with LAZY loading.
Any ideas? I also appreciate any ideas regarding how to further isolate the root cause as the error message is not really explicit and I can't see which part of my code triggers it. Plus when I debug it, the behavior changes as I compute all Tasks sequentially rather then in parallel. Thank you in advance.
The issue was caused by different Tasks accessing the same getter of some of the entities. If the first getter call opened a connection, the second call got on it, and then the first call closed the ResultSet, the second call one was in trouble. Synchronizing the getter method solved the problem.

Hystrix/Feign to solely react on HTTP status 429

I'm using Feign from the spring-cloud-starter-feign to send requests to a defined backend. I would like to use Hystrix as a circuit-breaker but for only one type of use-case: If the backend responds with a HTTP 429: Too many requests code, my Feign client should wait exactly one hour until it contacts the real backend again. Until then, a fallback method should be executed.
How would I have to configure my Spring Boot (1.5.10) application in order to accomplish that? I see many configuration possibilities but only few examples which are - in my opinion - unfortunately not resolved around use-cases.
This can be achieved by defining an ErrorDecoder and taking manual control of the Hystrix Circuit Breaker. You can inspect the response codes from the exceptions and provide your own fallback. In addition, if you wish to retry the request, wrap and throw your exception in a RetryException.
To meet your Retry requirement, also register a Retryer bean with the appropriate configuration. Keep in mind that using a Retryer will tie up a thread for the duration. The default implementation of Retryer does use an exponential backoff policy as well.
Here is an example ErrorDecoder taken from the OpenFeign documentation:
public class StashErrorDecoder implements ErrorDecoder {
#Override
public Exception decode(String methodKey, Response response) {
if (response.status() >= 400 && response.status() <= 499) {
return new StashClientException(
response.status(),
response.reason()
);
}
if (response.status() >= 500 && response.status() <= 599) {
return new StashServerException(
response.status(),
response.reason()
);
}
return errorStatus(methodKey, response);
}
}
In your case, you would react to 419 as desired.
You can forcibly open the Circuit Breaker setting this property at runtime
hystrix.command.HystrixCommandKey.circuitBreaker.forceOpen
ConfigurationManager.getConfigInstance()
.setProperty(
"hystrix.command.HystrixCommandKey.circuitBreaker.forceOpen", true);
Replace HystrixCommandKey with your own command. You will need to restore this circuit breaker back to closed after the desired time.
I could solve it with the following adjustments:
Properties in application.yml:
hystrix.command.commandKey:
execution.isolation.thread.timeoutInMilliseconds: 10_000
metrics.rollingStats.timeInMilliseconds: 10_000
circuitBreaker:
errorThresholdPercentage: 1
requestVolumeThreshold: 1
sleepWindowInMilliseconds: 3_600_000
Code in the respective Java class:
#HystrixCommand(fallbackMethod = "fallbackMethod", commandKey = COMMAND_KEY)
public void doCall(String parameter) {
try {
feignClient.doCall(parameter);
} catch (FeignException e) {
if (e.status() == 429) {
throw new TooManyRequestsException(e.getMessage());
}
}
}

How to recover from database errors in Grails within a transaction

In short, what I am trying solve is how to recover from certain database errors in a Grails application using Hibernate and continue on with the transaction skipping over the failed row updates that are part of a batch of changes.
The application uses Grails 2.3.11 but I have also tried with version 1.3.8 with similar failed results.
Basically there is a Grails service class that iterates over a list of imported records and attempts to update associated master records appropriately. In certain situations exceptions might occur during the domain.save(flush:true) call e.g. org.hibernate.exception.DataException thrown due to issues like (Data truncation: Data too long for column ...).
At this point I have tried:
Disabling transactions
Using domainObj.withTransaction() for each individual record
Trying various #Transactional annotations
Calling domain.clearErrors() and domain.discard() after catching the exception
Tried using a nested service with Transactional annotation with noRollbackFor as shown below
A number of other approaches but nothing I've tried has worked
Example code:
#Transactional
class UpdateService {
public updateBatch(Integer batchId) {
...
list.each { record ->
record.value = 123
try {
nestedService.saveDomain()
} catch (e) {
record.clearErrors()
record.discard()
}
}
batch.status = "POSTED"
batch.save()
}
}
#Transactional
class NestedService {
#Transactional(propagation = Propagation.REQUIRED, noRollbackFor = RuntimeException.class)
public void saveDomain(domainObj) throws RuntimeException {
if (domainObj.validate() && domainObj.save(flush:true) {
log.info "domain $domain was saved"
}
}
}
Once an error occurs I can't seem to clear out the hibernate session. On each subsequent record being updated I receive the error:
org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction
where it indicates the original failed domain id.
Revision:
Vahid, Thanks for the suggestions. I have tried that. I realized one issue is that I am passing objects across transactional boundaries. So I experimented with the NestedService class do something along the lines of:
#Transactional(propagation = Propagation.REQUIRE_NEW)
public void saveDomain(domainObj) {
def newObj = new Domain.get(domainObj.id)
newObj.properties = domainObj.properties
if (newObj.validate() && newObj.save(force:true) ) { ... }
I expected that to work but the original domainObj still fails even though I'm not calling the save on it. Very strange...
A simple approach would be to loop and then use validate(). If it does fail, then just store the id of the failed entity and proceed.
if(!domainObject.validate()){
// store Id for trying it again later ?
}else{
// Save
}

How to count all HTTP requests sent, retries in?

Some use cases require being able to count the requests sent by the Apache API. For example, when massively requesting a web API, which API requires an authentication through an API key, and which TOS limits the requests count in time for each key.
Being more specific on the case, I'm requesting https://domain1/fooNeedNoKey, and depending on its response analyzed data, I request https://domain2/fooNeedKeyWithRequestsCountRestrictions. All sends of those 1-to-2-requests sequences, are performed through a single org.apache.http.impl.client.FutureRequestExecutionService.
As of now, depending on org.apache.httpcomponents:httpclient:4.3.3, I'm using those API elements:
org.apache.http.impl.client.FutureRequestExecutionService, to perform multi-threaded HTTP requests. It offers time metrics (how much time did an HTTP thread took until terminated), but no requests counter metrics
final CloseableHttpClient httpClient = HttpClients.custom()
// the auto-retry feature of the Apache API will retry up to 5
// times on failure, being also allowed to send again requests
// that were already sent if necessary (I don't really understand
// the purpose of the second parameter below)
.setRetryHandler(new StandardHttpRequestRetryHandler(5, true))
// for HTTP 503 'Service unavailable' errors, also retrying up to
// 5 times, waiting 500ms between each retry. Guessed is that those
// 5 retries are part of the previous "global" 5 retries setting.
// The below setting, when used alone, would allow to only enable
// retries for HTTP 503, or to get a greater count of retries for
// this specific error
.setServiceUnavailableRetryStrategy(new DefaultServiceUnavailableRetryStrategy(5, 500))
.build();, which customizes the Apache API retry behavior
Getting back to the topic :
A request counter could be created by extending the Apache API retry-related classes quoted before
Alternatively, an Apache API support unrelated ticket tends to indicate this requests-counter metrics could be available and forwarded out of the API, into Java NIO
Edit 1:
Looks like the Apache API won't permit this to be done.
Quote from the inside of the API, RetryExec not beeing extendable in the API code I/Os:
package org.apache.http.impl.execchain;
public class RetryExec implements ClientExecChain {
..
public CloseableHttpResponse execute(
final HttpRoute route,
final HttpRequestWrapper request,
final HttpClientContext context,
final HttpExecutionAware execAware) throws IOException, HttpException {
..
for (int execCount = 1;; execCount++) {
try {
return this.requestExecutor.execute(route, request, context, execAware);
} catch (final IOException ex) {
..
if (retryHandler.retryRequest(ex, execCount, context)) {
..
}
..
}
}
The 'execCount' variable is the needed info, and it can't be accessed since it's only locally used.
As well, one can extend 'retryHandler', and manually count requests in it, but 'retryHandler.retryRequest(ex, execCount, context)' is not provided with the 'request' variable, making it impossible to know on what we're incrementing a counter (one may only want to increment the counter for requests sent to a specific domain).
I'm out of Java ideas for it. A 3rd party alternative: having the Java process polling a file on disk, managed by a shell script counting the desired requests. Sure it will make a lot of disk read-accesses and will be a hardware killer option.
Ok, the work around was easy, the HttpContext class of the API is intended for this:
// optionnally, in case your HttpCLient is configured for retry
class URIAwareHttpRequestRetryHandler extends StandardHttpRequestRetryHandler {
public URIAwareHttpRequestRetryHandler(final int retryCount, final boolean requestSentRetryEnabled)
{
super(retryCount, requestSentRetryEnabled);
}
#Override
public boolean retryRequest(final IOException exception, final int executionCount, final HttpContext context)
{
final boolean ret = super.retryRequest(exception, executionCount, context);
if (ret) {
doForEachRequestSentOnURI((String) context.getAttribute("requestURI"));
}
return ret;
}
}
// optionnally, in addition to the previous one, in case your HttpClient has specific settings for the 'Service unavailable' errors retries
class URIAwareServiceUnavailableRetryStrategy extends DefaultServiceUnavailableRetryStrategy {
public URIAwareServiceUnavailableRetryStrategy(final int maxRetries, final int retryInterval)
{
super(maxRetries, retryInterval);
}
#Override
public boolean retryRequest(final HttpResponse response, final int executionCount, final HttpContext context)
{
final boolean ret = super.retryRequest(response, executionCount, context);
if (ret) {
doForEachRequestSentOnURI((String) context.getAttribute("requestURI"));
}
return ret;
}
}
// main HTTP querying code: retain the URI in the HttpContext to make it available in the custom retry-handlers code
httpContext.setAttribute("requestURI", httpGET.getURI().toString());
try {
httpContext.setAttribute("requestURI", httpGET.getURI().toString());
httpClient.execute(httpGET, getHTTPResponseHandlerLazy(), httpContext);
// if request got successful with no need of retries, of if it succeeded on the last send: in any cases, this is the last query sent to server and it got successful
doForEachRequestSentOnURI(httpGET.getURI().toString());
} catch (final ClientProtocolException e) {
// if request definitively failed after retries: it's the last query sent to server, and it failed
doForEachRequestSentOnURI(httpGET.getURI().toString());
} catch (final IOException e) {
// if request definitively failed after retries: it's the last query sent to server, and it failed
doForEachRequestSentOnURI(httpGET.getURI().toString());
} finally {
// restoring the context as it was initially
httpContext.removeAttribute("requestURI");
}
Solved.

Hibernate Encryption of Database Completely Transparent to Application

I'm working on a Grails 1.0.4 project that has to be released in less than 2 weeks, and the customer just came up with a requirement that all data in the database should be encrypted.
Since encryption of every database access in the application itself could take a lot of time and will be error prone, the solution I seek is some kind of encryption transparent to the application.
Is there a way to setup Hibernate to encrypt all data in all tables (except maybie the id and version columns) or should I seek a MySQL solution (we're using MySQL 5.0) ?
EDIT:
Thanks for all of your posts for alternative solutions, if the customer changes mind it would be great. As for now, the requirement is "No plain text in the Database".
Second thing I'd like to point out is that I'm using Grails, for those not fammiliar with it, It's a convention over configuration, so even small changes to the application that are not by convention should be avoided.
If you end doing the work in the application, you can use Hibernate custom types and it wouldn't add that many changes to your code.
Here's an encrypted string custom type that I've used:
import org.hibernate.usertype.UserType
import org.apache.log4j.Logger
import java.sql.PreparedStatement
import java.sql.ResultSet
import java.sql.SQLException
import java.sql.Types
class EncryptedString implements UserType {
// prefix category name with 'org.hibernate.type' to make logging of all types easier
private final Logger _log = Logger.getLogger('org.hibernate.type.com.yourcompany.EncryptedString')
Object nullSafeGet(ResultSet rs, String[] names, Object owner) throws SQLException {
String value = rs.getString(names[0])
if (!value) {
_log.trace "returning null as column: $names[0]"
return null
}
_log.trace "returning '$value' as column: $names[0]"
return CryptoUtils.decrypt(value)
}
void nullSafeSet(PreparedStatement st, Object value, int index) throws SQLException {
if (value) {
String encrypted = CryptoUtils.encrypt(value.toString())
_log.trace "binding '$encrypted' to parameter: $index"
st.setString index, encrypted
}
else {
_log.trace "binding null to parameter: $index"
st.setNull(index, Types.VARCHAR)
}
}
Class<String> returnedClass() { String }
int[] sqlTypes() { [Types.VARCHAR] as int[] }
Object assemble(Serializable cached, Object owner) { cached.toString() }
Object deepCopy(Object value) { value.toString() }
Serializable disassemble(Object value) { value.toString() }
boolean equals(Object x, Object y) { x == y }
int hashCode(Object x) { x.hashCode() }
boolean isMutable() { true }
Object replace(Object original, Object target, Object owner) { original }
}
and based on this it should be simple to create similar classes for int, long, etc. To use it, add the type to the mapping closure:
class MyDomainClass {
String name
String otherField
static mapping = {
name type: EncryptedString
otherField type: EncryptedString
}
}
I omitted the CryptoUtils.encrypt() and CryptoUtils.decrypt() methods since that's not Grails-specific. We're using AES, e.g. "Cipher cipher = Cipher.getInstance('AES/CBC/PKCS5Padding')". Whatever you end up using, make sure it's a 2-way crypto, i.e. don't use SHA-256.
If the customer is worried about someone physically walking away with the hard drive then using a full disk solution like Truecrypt should work. If there worried about traffic being sniffed then take a look at this part of the mysql documentation on ssl over JDBC. Remember if someone compromises your server all bets are off.
the customer could easily do this without changing a thing in your application.
first, encrypt the communications between the server by turning on SSL in the mysql layer, or use an SSH tunnel.
second, store the mysql database on an encrypted volume.
any attack that can expose the filesystem of the mysql database or the credentials needed to log in to the mysql server is not mitigated by encrypting the data since that same attack can be used to retrieve the encryption key from the application itself.
Well it has been a long time since I've asked the question. In the meantime, thanks for all the answers. They were great when dealing with the original idea of encrypting the entire database, but the requirement changed to only encrypting sensitive user info, like name and address. So the solution was something like the code down below.
We've implemented an Encrypter which reads the encryption method from the record ( so there can be different encryption per record) and use it to connect transient duplicate fields to the ones encrypted in the database. The added bonus/drawbacks are:
The data is also encrypted in memory, so every access to the method getFirstName descrypts the data (I guess there is a way to cache decrypted data, but I dont need it in this case)
Encrypted fields cannot be used with default grails/hibernate methods for search through database, we've made custom methods in services that get data, encrypt it and then use the encrypted data in the where clause of a query. It's easy when using User.withCriteria
class User {
byte[] encryptedFirstName
byte[] encryptedLastName
byte[] encryptedAddress
Date dateCreated // automatically set date/time when created
Date lastUpdated // automatically set date/time when last updated
EncryptionMethod encryptionMethod = ConfigurationHolder.config.encryption.method
def encrypter = Util.encrypter
static transients = [
'firstName',
'lastName',
'address',
'encrypter'
]
static final Integer BLOB_SIZE = 1024
static constraints = {
encryptedFirstName maxSize: BLOB_SIZE, nullable: false
encryptedLastName maxSize: BLOB_SIZE, nullable: false
encryptedAddress maxSize: BLOB_SIZE, nullable: true
encryptionMethod nullable: false
} // constraints
String getFirstName(){
decrypt('encryptedFirstName')
}
void setFirstName(String item){
encrypt('encryptedFirstName',item)
}
String getLastName(){
decrypt('encryptedLastName')
}
void setLastName(String item){
encrypt('encryptedLastName',item)
}
String getAddress(){
decrypt('encryptedAddress')
}
void setAddress(String item){
encrypt('encryptedAddress',item)
}
byte[] encrypt(String name, String value) {
if( null == value ) {
log.debug "null string to encrypt for '$name', returning null"
this.#"$name" = null
return
}
def bytes = value.getBytes(encrypter.ENCODING_CHARSET)
def method = getEncryptionMethod()
byte[] res
try {
res = encrypter.encrypt( bytes, method )
} catch(e) {
log.warn "Problem encrypting '$name' data: '$string'", e
}
log.trace "Encrypting '$name' with '$method' -> '${res?.size()}' bytes"
this.#"$name" = res
}
String decrypt(String name) {
if(null == this.#"$name") {
log.debug "null bytes to decrypt for '$name', returning null"
return null
}
def res
def method = getEncryptionMethod()
try {
res = new String(encrypter.decrypt(this.#"$name", method), encrypter.ENCODING_CHARSET )
} catch(e) {
log.error "Problem decrypting '$name'", e
}
log.trace "Decrypting '$name' with '$method' -> '${res?.size()}' bytes"
return res
}
}
Another option is to use a JDBC driver that encrypts/decrypts data on the fly, two way. Bear in mind that any solution will probably invalidate searches by encrypted fields.
IMHO the best solution is the one proposed by longneck, it will make everything much easier, from administration to development. Besides, bear in mind that any solution with client-side encryption will render all your db data unusable outside of the client, ie, you will not be able to use nice tools like a jdbc client or MySQL query browser, etc.
Jasypt integrates with Hibernate: http://jasypt.org/hibernate3.html. However, queries which use WHERE clauses cannot be used
Generated ids, version, mapped foreign keys - basically everything maintained by Hibernate - are out unless you intend to declare custom CRUD for all of your classes and manually encrypt them in queries.
For everything else you've got a couple of choices:
#PostLoad and #PrePersist entity listeners will take care of all non-query operations.
Implementing custom String / Long / Integer / etc... types to handle encryption will take care of both query and CRUD operations; however the mapping will become rather messy.
You can write a thin wrapper around a JDBC driver (as well as Connection / Statement / PreparedStatement / ResultSet / etc...) to do the encryption for you.
As far as queries go you'll have to handle encryption manually (unless you're going with #2 above) but you should be able to do so via a single entry point. I'm not sure how (or if) Grails deals with this, but using Spring, for example, it would be as easy as extending HibernateTemplate.