Hibernate Encryption of Database Completely Transparent to Application - mysql

I'm working on a Grails 1.0.4 project that has to be released in less than 2 weeks, and the customer just came up with a requirement that all data in the database should be encrypted.
Since encryption of every database access in the application itself could take a lot of time and will be error prone, the solution I seek is some kind of encryption transparent to the application.
Is there a way to setup Hibernate to encrypt all data in all tables (except maybie the id and version columns) or should I seek a MySQL solution (we're using MySQL 5.0) ?
EDIT:
Thanks for all of your posts for alternative solutions, if the customer changes mind it would be great. As for now, the requirement is "No plain text in the Database".
Second thing I'd like to point out is that I'm using Grails, for those not fammiliar with it, It's a convention over configuration, so even small changes to the application that are not by convention should be avoided.

If you end doing the work in the application, you can use Hibernate custom types and it wouldn't add that many changes to your code.
Here's an encrypted string custom type that I've used:
import org.hibernate.usertype.UserType
import org.apache.log4j.Logger
import java.sql.PreparedStatement
import java.sql.ResultSet
import java.sql.SQLException
import java.sql.Types
class EncryptedString implements UserType {
// prefix category name with 'org.hibernate.type' to make logging of all types easier
private final Logger _log = Logger.getLogger('org.hibernate.type.com.yourcompany.EncryptedString')
Object nullSafeGet(ResultSet rs, String[] names, Object owner) throws SQLException {
String value = rs.getString(names[0])
if (!value) {
_log.trace "returning null as column: $names[0]"
return null
}
_log.trace "returning '$value' as column: $names[0]"
return CryptoUtils.decrypt(value)
}
void nullSafeSet(PreparedStatement st, Object value, int index) throws SQLException {
if (value) {
String encrypted = CryptoUtils.encrypt(value.toString())
_log.trace "binding '$encrypted' to parameter: $index"
st.setString index, encrypted
}
else {
_log.trace "binding null to parameter: $index"
st.setNull(index, Types.VARCHAR)
}
}
Class<String> returnedClass() { String }
int[] sqlTypes() { [Types.VARCHAR] as int[] }
Object assemble(Serializable cached, Object owner) { cached.toString() }
Object deepCopy(Object value) { value.toString() }
Serializable disassemble(Object value) { value.toString() }
boolean equals(Object x, Object y) { x == y }
int hashCode(Object x) { x.hashCode() }
boolean isMutable() { true }
Object replace(Object original, Object target, Object owner) { original }
}
and based on this it should be simple to create similar classes for int, long, etc. To use it, add the type to the mapping closure:
class MyDomainClass {
String name
String otherField
static mapping = {
name type: EncryptedString
otherField type: EncryptedString
}
}
I omitted the CryptoUtils.encrypt() and CryptoUtils.decrypt() methods since that's not Grails-specific. We're using AES, e.g. "Cipher cipher = Cipher.getInstance('AES/CBC/PKCS5Padding')". Whatever you end up using, make sure it's a 2-way crypto, i.e. don't use SHA-256.

If the customer is worried about someone physically walking away with the hard drive then using a full disk solution like Truecrypt should work. If there worried about traffic being sniffed then take a look at this part of the mysql documentation on ssl over JDBC. Remember if someone compromises your server all bets are off.

the customer could easily do this without changing a thing in your application.
first, encrypt the communications between the server by turning on SSL in the mysql layer, or use an SSH tunnel.
second, store the mysql database on an encrypted volume.
any attack that can expose the filesystem of the mysql database or the credentials needed to log in to the mysql server is not mitigated by encrypting the data since that same attack can be used to retrieve the encryption key from the application itself.

Well it has been a long time since I've asked the question. In the meantime, thanks for all the answers. They were great when dealing with the original idea of encrypting the entire database, but the requirement changed to only encrypting sensitive user info, like name and address. So the solution was something like the code down below.
We've implemented an Encrypter which reads the encryption method from the record ( so there can be different encryption per record) and use it to connect transient duplicate fields to the ones encrypted in the database. The added bonus/drawbacks are:
The data is also encrypted in memory, so every access to the method getFirstName descrypts the data (I guess there is a way to cache decrypted data, but I dont need it in this case)
Encrypted fields cannot be used with default grails/hibernate methods for search through database, we've made custom methods in services that get data, encrypt it and then use the encrypted data in the where clause of a query. It's easy when using User.withCriteria
class User {
byte[] encryptedFirstName
byte[] encryptedLastName
byte[] encryptedAddress
Date dateCreated // automatically set date/time when created
Date lastUpdated // automatically set date/time when last updated
EncryptionMethod encryptionMethod = ConfigurationHolder.config.encryption.method
def encrypter = Util.encrypter
static transients = [
'firstName',
'lastName',
'address',
'encrypter'
]
static final Integer BLOB_SIZE = 1024
static constraints = {
encryptedFirstName maxSize: BLOB_SIZE, nullable: false
encryptedLastName maxSize: BLOB_SIZE, nullable: false
encryptedAddress maxSize: BLOB_SIZE, nullable: true
encryptionMethod nullable: false
} // constraints
String getFirstName(){
decrypt('encryptedFirstName')
}
void setFirstName(String item){
encrypt('encryptedFirstName',item)
}
String getLastName(){
decrypt('encryptedLastName')
}
void setLastName(String item){
encrypt('encryptedLastName',item)
}
String getAddress(){
decrypt('encryptedAddress')
}
void setAddress(String item){
encrypt('encryptedAddress',item)
}
byte[] encrypt(String name, String value) {
if( null == value ) {
log.debug "null string to encrypt for '$name', returning null"
this.#"$name" = null
return
}
def bytes = value.getBytes(encrypter.ENCODING_CHARSET)
def method = getEncryptionMethod()
byte[] res
try {
res = encrypter.encrypt( bytes, method )
} catch(e) {
log.warn "Problem encrypting '$name' data: '$string'", e
}
log.trace "Encrypting '$name' with '$method' -> '${res?.size()}' bytes"
this.#"$name" = res
}
String decrypt(String name) {
if(null == this.#"$name") {
log.debug "null bytes to decrypt for '$name', returning null"
return null
}
def res
def method = getEncryptionMethod()
try {
res = new String(encrypter.decrypt(this.#"$name", method), encrypter.ENCODING_CHARSET )
} catch(e) {
log.error "Problem decrypting '$name'", e
}
log.trace "Decrypting '$name' with '$method' -> '${res?.size()}' bytes"
return res
}
}

Another option is to use a JDBC driver that encrypts/decrypts data on the fly, two way. Bear in mind that any solution will probably invalidate searches by encrypted fields.
IMHO the best solution is the one proposed by longneck, it will make everything much easier, from administration to development. Besides, bear in mind that any solution with client-side encryption will render all your db data unusable outside of the client, ie, you will not be able to use nice tools like a jdbc client or MySQL query browser, etc.

Jasypt integrates with Hibernate: http://jasypt.org/hibernate3.html. However, queries which use WHERE clauses cannot be used

Generated ids, version, mapped foreign keys - basically everything maintained by Hibernate - are out unless you intend to declare custom CRUD for all of your classes and manually encrypt them in queries.
For everything else you've got a couple of choices:
#PostLoad and #PrePersist entity listeners will take care of all non-query operations.
Implementing custom String / Long / Integer / etc... types to handle encryption will take care of both query and CRUD operations; however the mapping will become rather messy.
You can write a thin wrapper around a JDBC driver (as well as Connection / Statement / PreparedStatement / ResultSet / etc...) to do the encryption for you.
As far as queries go you'll have to handle encryption manually (unless you're going with #2 above) but you should be able to do so via a single entry point. I'm not sure how (or if) Grails deals with this, but using Spring, for example, it would be as easy as extending HibernateTemplate.

Related

Fiware - Cygnus mongoSink metadata persistence

I'm trying to persist with cygnus using a Mongo sink, data from entities with metadata data estructures. So far, I haven't been able to achieve that.
I'm using cygnus version 0.13.0. Seems to be possible to save metadata info using MySQL and CKAN persistence sinks.
¿Is it possible too using Mongo?
¿Is it configuration matter?
Thanks in advance for any help.
Cygnus does not store the attribute metadata in MongoDB. This is because the internal usage we make of Cygnus when persisting in MongoDB, which imposes a strong constraint regarding this issue.
Anyway, modifying the code in a fork of yourself in order to fix this should be relatively easy. Simply have a look on this method:
private Document createDoc(long recvTimeTs, String entityId, String entityType, String attrName, String attrType, String attrValue) {
Passing an additional parameter String attrMd and appending this value to the doc variable should do the trick:
private Document createDoc(long recvTimeTs, String entityId, String entityType, String attrName, String attrType, String attrValue, String attrMd) {
Document doc = new Document("recvTime", new Date(recvTimeTs));
switch (dataModel) {
case DMBYSERVICEPATH:
doc.append("entityId", entityId)
.append("entityType", entityType)
.append("attrName", attrName)
.append("attrType", attrType)
.append("attrValue", attrValue)
.append("attrMd", attrMd);
break;
case DMBYENTITY:
doc.append("attrName", attrName)
.append("attrType", attrType)
.append("attrValue", attrValue)
.append("attrMd", attrMd);
break;
case DMBYATTRIBUTE:
doc.append("attrType", attrType)
.append("attrValue", attrValue)
.append("attrMd", attrMd);
break;
default:
return null; // this will never be reached
} // switch
return doc;
} // createDoc
Starting with version 1.8.0, FIWARE CYGNUS adds metadata support. As you can see in the template of the configuration file, the only thing you have to do is set the property cygnus-ngsi.sinks.mongo-sink.attr_metadata_store to True, which by the way is set to False by default.
Regards!

Adobe AIR SQLResult listener reached, but no data in SQLite

I'm currently working on a project using AIR and Flex that uses a remote data source to persist data locally in a SQLite database. Currently, there's a lot of copy and paste code that I was trying to alleviate, so since we already use a DAO pattern with several common queries that get passed to it and a type that creates SQLStatement values, I figured I would simplify our codebase even more.
I applied the Adapter pattern to allow a wider range of possible database operations to be performed ([saveOrUpdate, find, findAll, remove] => [selectSingle, selectMultiple, insert, updateSingle, updateMultiple, deleteSingle, deleteMultiple]). I also applied the Strategy pattern to two aspects of the statement runner: the first time for what sort of aggregated type to return (either an Array of records or an ArrayCollection of records) for the selectMultiple function; the second time for creating or not creating historical records (ChangeObjects).
After applying these patterns and testing some refactored code, it worked perfectly with an existing SQLite database. I neglected to test its compatibility with the remote data source, since the saving mechanisms are used during that process as well. After refactoring and simplifying our code and nearing the end of the development cycle, I tested the download.
It would read data from the SQLite database, despite the fact that there was actually no data in it according to sqlite3.
I will give the related piece of code for this.
public class BaseDaoAdaptee {
private var returnStrategy: ReturnTypeStrategy;
private var trackingStrategy: TrackingStrategy;
private var creator: StatementCreator;
public function insert(queryTitle: String,
object: DaoAwareDTO,
parameters: Array,
mutator: Function,
handler: Function): void {
var statement: SQLStatement;
mutator = creator.validEmptyFunction(mutator);
handler = creator.validFault(handler);
statement = defaultStatement(queryTitle, parameters, handler);
statement.addEventListener(SQLEvent.RESULT,
trackingStrategy.onInserted(object, mutator), false, 0, true);
statement.execute();
}
}
The code for the TrackingStrategy implemented:
public class TrackedStrategy
implements TrackingStrategy {
public function onInserted(object: DaoAwareDTO,
callback: Function): Function {
return function (event: SQLEvent): void {
var change: Change,
id:Number = event.target.getResult().lastInsertRowID;
creator.logger.debug((event.target as SQLStatement).itemClass + ' (id # ' + id + ') inserted');
(object as Storeable).id = id;
change = new Creation(object);
change.register();
callback();
};
}
}
The logger reads that various database records were inserted, and when stopped on a breakpoint in the above lambda, "object" has all proper values. When running a Select statement in sqlite3, no records ever get returned.
Why would this happen?
Turns out an open transaction on a SQLConnection value was the cause. Got to love team projects. Commit or rollback your SQLConnection transactions!

What is the equivilant of C#'s generic Dictionary in ActionScript 3?

I want to have a collection of objects, which will be of a class I created called Server. A Server has a string property which is it's IP address, as well as many other pieces of data and objects.
I will have methods for adding and removing servers to this collection, and there will be a need to find a server by it's IP address occasionally. If I were doing this in C# I would use a Dictionary< where the IP string would be the key and the Server object would be the value. I could easily check to see if an item exists in the Dictionary before attempting to add it.
So my requirements are:
1. Ability to add items to the collection (I don't care where they go, front, back, middle)
2. Ability to remove items from anywhere in the collection.
3. Ability to determine if a particular IP address already exists in the collection.
4. Ability to get a reference to a Server object by it's IP.
Edit: Oh yes, I would like it to be strongly typed like the Vector... I guess it's not absolutely necesary, but would be nice.
So it seems like an associative arrays will give me what I need, except I'm not sure about how to do #3 or #4.
public var Servers:Object = new Object( );
public function AddServer(server:Server):void
{
//TODO:need to check if it exists first and throw an error if so
//(it's the caller's responsibility to call DoesServerExist first)
Servers[server.IP] = server;
}
public function RemoveServer(IP:string):void
{
//is it OK to attempt to delete an item if it doesn't already exist?
//do I need to check if it exists before doing delete?
delete Servers[IP];
}
public function DoesServerExist(IP:string):bool
{
//Do I loop through all the elements testing it's IP property?
//Or can I just do something like this?
if(Servers[IP] == null)
{
return false;
}
else
{
return true;
}
}
public function GetServer(IP:string):Server
{
return Servers[IP];//what is returned if this IP doesn't exist?
}
Call me goofy, but why not use the Dictionary class? That gets you everything except strong typing.
If you want strong typing then I'd say you need a custom container, which wraps up a Vector of Servers, and a Dictionary or associative array of IP strings that indexes into the Vector. Then you'd need to expose methods for access, test, insert and remove.
You can just use an array. Example:
var dict:Array = [];
var ip = "164.157.012.122"
dict[ip] = "Server name"
if (dict[ip] == "Server name"){
trace("Yay");
}
//membership
if (dict[ip]){
trace(ip + " is a member of dict");
} else {
trace (ip + " is not a member");
}
//removal:
dict[ip] = null;
AS3 does not really have a built in Dictionary class, unfortunately.

Exceptions, return values, and contextual information

I know that this type of question has been asked over and over again, however, I have yet to find a definitive answer for the problem I am looking into.
From all the content on exception handling that I have read it appears that the general concensus is that exceptions should only be used for exceptional circumstances. I've also read in many places that one should use, where possible, return values to indicate problems or failures (such as login failure, validation failure of some sort). My problem is, when using these return values, how does one communicate the contextual information of the problem? With exceptions, one can add the contextual information to the exception and allow that to bubble up. Let me try and use a code example to explain:
Let's say we have a basic abstract class (I've left out some of the details) which represents some kind of format definition for a String. This class essentially dictates how the format of a given string should be.
public abstract class ADataEntryDefinition
{
public boolean isValid(String data);
}
let's say I extend this to perform some security validation on the string:
public class SecureDataEntryDefinition extends ADataEntryDefinition
{
public boolean isValid(String data)
{
//do some security checks on the format of the data
}
}
The validate method will take in a String and return true if the string matches the data definition defined by the class.
Moving along, let's say I have a class which manages several of these data definitions, and this class' responsibility is to validate each entry in a comma separated String against one of the data definitions it maintains.
public class DataSetDefinitions
{
private List<ADataEntryDefinition> dataDefinitions = ...
public boolean isValid(String dataValues)
{
//obtain each string in dataValues delimited by a ',' into String[]
//called dataEntryValues
int i=0;
for (ADataEntryDefinition dataEntry : dataDefinitions)
{
if (!dataEntry.isValid(dataEntryValues[i++])
{
return false;
}
}
return true;
}
}
Now, to me these methods seem way to general to throw exceptions in the event of invalid data (for one, invalid data may be expected in some cases). In this case, I like the approach of returning true/false to indicate validation failure and subsequently allowing the caller to judge how serious it is. So the caller does the following:
boolean success = false;
success = dataSetDefinitions.isValid(someString);
Suppose a specific caller like the above deems the failed validation to be critical, and hence, must subsequently throw an exception to prevent processing from continuing; where should it obtain the contextual information it needs to convey the problem... how should it know that 2 layers (calls) down the validation actually failed due to security problems in the SecureDataEntryDefinition class (or any other subclass for that matter).
I guess I could add a method like so:
public class DataSetDefinitions
{
private List<ADataEntryDefinition> dataDefinitions = ...
public boolean isValid(String dataValues)
{
....
}
public String getValidationErrorMsg() {...}
}
which would return the error message of the last failed validation. Then, the following could be done by the caller upon failed validation:
success = dataSetDefinitions.isValid(someString);
if (!success)
throw new SomeException(dataSetDefinitions.getValidationErrorMsg());
But to me this just seems like having the class (DataSetDefinitions in this case) know or maintain state about the previous validation which it shouldn't. Taking into account that this class may perform validation of several different, independent strings, it seems wrong having it maintain state about the validation of any given one of them.
I guess this question is essentially asking how one designs methods to be general - not taking the law into their own hands by throwing exceptions unnecessarily, but allowing callers to decide on the severity - but still allowing the callers to obtain detailed contextual information in the event that the caller needs to communicate the problem. Is there a better way of doing the above?
Apologies if this was very long-winded :/ Any responses will be appreciated.
Ciao.
Don't return a bool. Return a class that encapsulates the success/failure state, plus the associated information. That way, you can do something like:
DataEntryStatus status = isValid(...);
if (!status.isValid()) {
throw status.generateStatusException();
}
and the status object itself generates the appropriate exception, thus maintaining encapsulation.
You could return a user defined class instead of a simple bool in order to provide more contextual information.
It would be something similar to the strategy used with events. We have a EventArgs class from which other classes derive in order to provide more contextual information for a given type of event.
The way i solve it most of the time is defining several class constants and return these. Then in the business logic of my controllers i would just check against these values statically.
<?php
class Test
{
const SUCCESS = 1000;
const EMAIL_FAIL = 2001;
const SAVE_FAIL = 2002;
...
public function save($value)
{
if (!$this->writetodb($value)
return self::SAVE_FAIL;
elseif(!$this->sendMailToAdmin())
return self::EMAIL_FAIL;
else
return self::SUCCESS;
}
}
$test = new Test();
$result = $test->save('my value');
switch ($result) {
case Test::SUCCESS:
echo 'Yay!';
break;
case Test::SAVE_FAIL:
echo 'Error saving!';
break;
case Test::EMAIL_FAIL:
echo 'Error sending email!';
break;
}

Linq to SQL and concurrency with Rob Conery repository pattern

I have implemented a DAL using Rob Conery's spin on the repository pattern (from the MVC Storefront project) where I map database objects to domain objects using Linq and use Linq to SQL to actually get the data.
This is all working wonderfully giving me the full control over the shape of my domain objects that I want, but I have hit a problem with concurrency that I thought I'd ask about here. I have concurrency working but the solution feels like it might be wrong (just one of those gitchy feelings).
The basic pattern is:
private MyDataContext _datacontext
private Table _tasks;
public Repository(MyDataContext datacontext)
{
_dataContext = datacontext;
}
public void GetTasks()
{
_tasks = from t in _dataContext.Tasks;
return from t in _tasks
select new Domain.Task
{
Name = t.Name,
Id = t.TaskId,
Description = t.Description
};
}
public void SaveTask(Domain.Task task)
{
Task dbTask = null;
// Logic for new tasks omitted...
dbTask = (from t in _tasks
where t.TaskId == task.Id
select t).SingleOrDefault();
dbTask.Description = task.Description,
dbTask.Name = task.Name,
_dataContext.SubmitChanges();
}
So with that implementation I've lost concurrency tracking because of the mapping to the domain task. I get it back by storing the private Table which is my datacontext list of tasks at the time of getting the original task.
I then update the tasks from this stored Table and save what I've updated
This is working - I get change conflict exceptions raised when there are concurrency violations, just as I want.
However, it just screams to me that I've missed a trick.
Is there a better way of doing this?
I've looked at the .Attach method on the datacontext but that appears to require storing the original version in a similar way to what I'm already doing.
I also know that I could avoid all this by doing away with the domain objects and letting the Linq to SQL generated objects all the way up my stack - but I dislike that just as much as I dislike the way I'm handling concurrency.
I worked through this and found the following solution. It works in all the test cases I (and more importantly, my testers!) can think of.
I am using the .Attach() method on the datacontext, and a TimeStamp column. This works fine for the first time that you save a particular primary key back to the database but I found that the datacontext throws a System.Data.Linq.DuplicateKeyException "Cannot add an entity with a key that is already in use."
The work around for this I created was to add a dictionary that stored the item I attach the first time around and then every subsequent time I save I reuse that item.
Example code is below, I do wonder if I've missed any tricks - concurrency is pretty fundamental so the hoops I'm jumping through seem a little excessive.
Hopefully the below proves useful, or someone can point me towards a better implementation!
private Dictionary<int, Payment> _attachedPayments;
public void SavePayments(IList<Domain.Payment> payments)
{
Dictionary<Payment, Domain.Payment> savedPayments =
new Dictionary<Payment, Domain.Payment>();
// Items with a zero id are new
foreach (Domain.Payment p in payments.Where(p => p.PaymentId != 0))
{
// The list of attached payments that works around the linq datacontext
// duplicatekey exception
if (_attachedPayments.ContainsKey(p.PaymentId)) // Already attached
{
Payment dbPayment = _attachedPayments[p.PaymentId];
// Just a method that maps domain to datacontext types
MapDomainPaymentToDBPayment(p, dbPayment, false);
savedPayments.Add(dbPayment, p);
}
else // Attach this payment to the datacontext
{
Payment dbPayment = new Payment();
MapDomainPaymentToDBPayment(p, dbPayment, true);
_dataContext.Payments.Attach(dbPayment, true);
savedPayments.Add(dbPayment, p);
}
}
// There is some code snipped but this is just brand new payments
foreach (var payment in newPayments)
{
Domain.Payment payment1 = payment;
Payment newPayment = new Payment();
MapDomainPaymentToDBPayment(payment1, newPayment, false);
_dataContext.Payments.InsertOnSubmit(newPayment);
savedPayments.Add(newPayment, payment);
}
try
{
_dataContext.SubmitChanges();
// Grab the Timestamp into the domain object
foreach (Payment p in savedPayments.Keys)
{
savedPayments[p].PaymentId = p.PaymentId;
savedPayments[p].Timestamp = p.Timestamp;
_attachedPayments[savedPayments[p].PaymentId] = p;
}
}
catch (ChangeConflictException ex)
{
foreach (ObjectChangeConflict occ in _dataContext.ChangeConflicts)
{
Payment entityInConflict = (Payment) occ.Object;
// Use the datacontext refresh so that I can display the new values
_dataContext.Refresh(RefreshMode.OverwriteCurrentValues, entityInConflict);
_attachedPayments[entityInConflict.PaymentId] = entityInConflict;
}
throw;
}
}
I would look at trying to utilise the .Attach method by passing the 'original' and 'updated' objects thus achieving true optimistic concurrency checking from LINQ2SQL. This IMO would be preferred to using version or datetime stamps either in the DBML objects or your Domain objects. I'm not sure how MVC allows for this idea of persisting the 'original' data however.. i've been trying to investigate the validation scaffolding in the hope that it's storing the 'original' data.. but i suspect that it is as only as good as the most recent post (and/or failed validation). So that idea may not work.
Another crazy idea i had was this: override the GetHashCode() for all of your domain objects where the hash represents the unique set of data for that object (minus the ID of course). Then, either manually or with a helper bury that hash in a hidden field in the HTML POST form and send it back to your service layer with your updated domain object - do the concurrency checking in your service layer or data layer (by comparing the original hash with a newly extracted domain object's hash) but be aware that you need to be checking for and raising concurrency exceptions yourself. It's nice to use the DMBL functions but the idea of abstracting away the data layer is so to not depend on the particular implementation's features etc. So having full control of the optimistic concurrency checking on your domain objects in your service layer (for example) seems like a good approach to me.