I am making an application where there is some PII (personally identifiable information) being stored, such as the Date of Birth along with the name and a few other fields. I want to encrypt the data with AES_ENCRYPT using CBC on MySQL, however there is an administrator who needs to see the information, but if I encrypt the user's information, the administrator won't be able to see. Should I encrypt the PII with the username and have the administrator not be able to see and encrypt the username with the username, or should I leave the username unencrypted but encrypt everything else with the username, leaving some security risks, but allowing the administrator to see the information. What should I do?
Generally speaking, encrypting data at rest in DBMSs with symmetric encryption like AES is both unnecessary and insufficient to protect it from attack.
Why?
For one thing, the weak link in a system like this is typically the web server, not the MySQL server machine. But the web server must contain the necessary key to both encrypt and decrypt the data. If an attacker pwns your web server, he takes possession of your key and your algorithms to generate keys, and therefore he pwns your sensitive data. He also has a really robust way to figure out what data you think is sensitive.
It's much less likely the attacker will be able to compromise your MySQL server directly; in situations involving PII those machines ordinarily live behind firewalls.
For another thing: your administrator. He will need to have possession of the key, and administrators are juicy targets for attackers. (I'm not saying he's a black hat, just that his white hat has a big red target on it.)
Another thing: it may give you a false sense of security.
(Notice that passwords are not encrypted: they are hashed. It's simple to compare a password to a hashed password and say "match" or "no match." But it's much harder to take a hashed password and recover the original one. So the argument that passwords are "encrypted" isn't relevant to this situation.)
I seriously suggest you spend your time and money not on encrypting your PII at rest as you suggest. Instead spend your resources to make sure your web site is secure, and making sure your MySQL server is safely behind a robust firewall and invisible to the public internet.
Look at the OWASP Top Ten vulnerabilities. Spend time dealing with them on your site. Use a penetration test tool like burp scan or zap scan to try to break in.
Securing your web site is like the old joke about the two guys and the bear.
Bear: roar.
Joe: I hope I can outrun the bear.
Mike: I just have to outrun you.
In cybersecurity, you don't want to be Joe.
Related
I'm already using salted hashing to store passwords in my database, which means that I should be immune to rainbow table attacks.
I had a thought, though: what if someone does get hold of my database? It contains the users' email addresses. I can't really hash these, because I'll be using them to send notification emails, etc..
Should I encrypt them?
Bruce Schneier has a good response to this kind of problem.
Cryptography is not the solution to your security problems. It might be part of the solution, or it might be part of the problem. In many situations, cryptography starts out by making the problem worse, and it isn't at all clear that using cryptography is an improvement.
Essentially encrypting your emails in the database 'just in case' is not really making the database more secure. Where are the keys stored for the database? What file permissions are used for these keys? Is the database accesable publically? Why? What kind of account restrictions are in place for these accounts? Where is the machine stored, who has physical access to this box? What about remote login/ssh access etc. etc. etc.
So I guess you can encrypt the emails if you want, but if that is the extent of the security of the system then it really isn't doing much, and would actually make the job of maintaining the database harder.
Of course this could be part of an extensive security policy for your system - if so then great!
I'm not saying that it is a bad idea - But why have a lock on the door from Deadlocks'R'us which cost $5000 when they can cut through the plywood around the door? Or come in through the window which you left open? Or even worse they find the key which was left under the doormat. Security of a system is only as good as the weakest link. If they have root access then they can pretty much do what they want.
Steve Morgan makes a good point that even if they cannot understand the email addresses, they can still do a lot of harm (which could be mitigated if they only had SELECT access)
Its also important to know what your reasons are for storing the email address at all. I might have gone a bit overboard with this answer, but my point is do you really need to store an email address for an account? The most secure data is data that doesn't exist.
I realize this is a dead topic, but I agree with Arjan's logic behind this. There are a few things I'd like to point out:
Someone can retrieve data from your database without retrieving your source code (i.e. SQL injection, third-party db's). With this in mind, it's reasonable to consider using an encryption with a key. Albeit, this is only an added measure of security, not the security...this is for someone who wants to keep the email more private than plaintext,
In the off chance something is overlooked during an update, or an attacker manages to retrieve the emails.
IMO: If you plan on encrypting an email, store a salted hash of it as well. Then you can use the hash for validation, and spare the overhead of constantly using encryption to find a massive string of data. Then have a separate private function to retrieve and decrypt your emails when you need to use one.
In common with most security requirements, you need to understand the level of threat.
What damage can be done if the email addresses are compromised?
What's the chance of it happening?
The damage done if email addresses are REPLACED could be much greater than if they're EXPOSED. Especially if you're, for example, using the email address to verify password resets to a secure system.
The chance of the passwords being either replaced or exposed is much reduced if you hash them, but it depends what other controls you have in place.
I would say it depends on the application of your database.
The biggest problem is, where do you store the encryption key? Because if the hacker has excess to anything more than your DB, all your efforts are probably wasted. (Remember, your application will need that encryption key to decrypt and encrypt so eventually the hacker will find the encryption key and used encryption scheme).
Pro:
A leak of your DB only will not expose the e-mail addresses.
Cons:
Encryption means performance loss.
Allot of database actions will be harder if not impossible.
Don't accidentally conflate encryption with obfuscation. We commonly obfuscate emails to prevent spam. Lots of web sites will have "webmaster _at_ mysite.com" to slow down crawlers from parsing the email address as a potential spam target. That should be done in the HTML templates -- there's no value to doing this in persistent database storage.
We don't encrypt anything unless we need to keep it secret during transmission. When and where will your data being transmitted?
The SQL statements are transmitted from client to server; is that on the same box or over a secure connection?
If your server is compromised, you have an unintentional transmission. If you're worried about this, then you should perhaps be securing your server. You have external threats as well as internal threats. Are ALL users (external and internal) properly authenticated and authorized?
During backups you have an intentional transmission to backup media; is this done using a secure backup strategy that encrypts as it goes?
Both SQL Server and Oracle (and I believe also others DBs) support encryption of data at the database level. If you want to encrypt something why don't simply abstract the access to the data that could be encrypted on the database server side and left the user choose if use the encrypted data (in this case the SQL command will be different) or not. If the user want to user encrypted data then it can configure the database server and all the maintenance work connected with key management is made using standard DBA tool, made from the DB vendor and not from you.
A copy of my answer at What is the best and safest way to store user email addresses in the database?, just for the sake of the search...
In general I agree with others saying it's not worth the effort. However, I disagree that anyone who can access your database can probably also get your keys. That's certainly not true for SQL Injection, and may not be true for backup copies that are somehow lost or forgotten about. And I feel an email address is a personal detail, so I wouldn't care about spam but about the personal consequences when the addresses are revealed.
Of course, when you're afraid of SQL Injection then you should make sure such injection is prohibited. And backup copies should be encrypted themselves.
Still, for some online communities the members might definitely not want others to know that they are a member (like related to mental healthcare, financial help, medical and sexual advice, adult entertainment, politics, ...). In those cases, storing as few personal details as possible and encrypting those that are required (note that database-level encryption does not prevent the details from showing using SQL Injection), might not be such a bad idea. Again: treat an email address as such personal detail.
For many sites the above is probably not the case, and you should focus on prohibiting SELECT * FROM through SQL Injection, and making sure visitors cannot somehow get to someone else's personal profile or order information by changing the URL.
It's worth to encrypt data in Databases, it's not making it a bit more difficult but way more difficult when its encrypted in the right way so stop philosophy and encrypt the sensitive data ;)
#Roo
I somewhat agree to what you are saying but isn't it worth encrypting the data just to make it a little more difficult for someone to get it?
With your reasoning, it would be useless to have locks or alarms in your house, because they can also easily be compromised.
My reply:
I would say that if you have sensitive data that you don't want to fall in the wrong hands, you should probably do it as hard as you can for a hacker to get it, even if it's not 100% fool proof.
I miss one important answer here.
When you have European users, you have to comply with the GDPR rules.
Email addresses are considered personal data meaning Art.5 does apply on email addresses.
processed in a manner that ensures appropriate security of the
personal data, including protection against unauthorised or unlawful
processing and against accidental loss, destruction or damage, using
appropriate technical or organisational measures (‘integrity and
confidentiality’).
Of course this does not say that you must encrypt email addresses. But by encrypting the data you do protect it from snooping employees. And protect yourself as a developer from requests to make a manual change in the database.
You really have to weigh your worst case senario of someone obtaining those email addresses, the likelihood of someone obtaining them, and your extra effort/time needed to impliement the change.
Goal:
I'd like to allow users to create questions and collect information from other users in a custom web app (PHP/MySQL in a shored hosting environment) and protect the data collected.
Background:
The default questions that all users answer are general enough that they can not be construed as personally identifiable information (PII), thus limiting my liability to protect it, but users who create their own questions will likely ask for PII which then becomes a liability.
What I would like to do is protect this information in such a way that if either the hosting account or the database were compromised (or both!), the PII would not be recoverable without a significant amount of work, and even then, only a small portion would theoretically be recoverable.
Proposed solution:
Assuming MySQL's built-in AES_ENCRYPT()/AES_DECRYPT() functions are used to encrypt the PII table, the passphrase would need to be stored in the hosting account, so if the hosting account were compromised, the data could easily be read.
Since the users' passwords are well protected (hashed with salt), I'm thinking of capturing their plaintext password during authentication, encrypting it, and storing it in the PHP session until the user logs out.
A public/private key combo will be created for each user with the private key being password protected with the user's password + salt.
Then, when PII data based on that user's custom questions are added to the DB, the user's public key would be used to encrypt the PII that they collected through the app. When the data is read (only when the user is logged in), the data would be unencrypted with the user's private key (which is unlocked with their password + salt).
The benefits I see are:
in the worst case scenario, where servers completely compromised, app code is read to find encryption keys, PHP session files are decrypted to find user's passwords, then entries in the PII table associated with that user are decrypted, then only the PII collected from questions of currently logged-in users could be recovered. Any users not logged in would be safe.
even the DBA or similar wouldn't be able to read the PII.
The drawbacks I see are:
user passwords are stored in a recoverable form while they are logged in.
users who forget their passwords would loose access to their data.
each relatively small bit of data will take up much more space in the DB due to encryption.
My question: Is there a better way to do this?
I see a number problems with this design from a security perspective. First of all passwords must never be encrypted, this is a vulnerability identified by CWE-257.
Further more MySQL's AES_ENCRYPT() is complete garbage for more than one reason. It uses EBC mode, and here is a good example of why this is crap:
Original Image:
EBC Mode (which is what mysql's AES_ENCRYPT() uses):
But if the database where compromised the attacker is going to defeat AES_ENCRYPT() by enabling the query log.
Using the user's password for encryption should be avoided, you should be using a crypgoraphic nonce. If you do use a password make sure you use a String2Key funciton. You must also use CBC or CMAC mode with a random iv. I don't really see how asymmetric cryptography can help. Asymmetric cryptography is very slow, memory intensive. They data that it protects is made less secure when the attacker controls the message because you can compare cipher text messages. This is why an random IV is important, and in the asymmetric world you don't have this level of protection.
Key Generation should look something like:
$key=string2key($base_nonce.$salt.$user_password)
Make sure the output of your string2key function is the same size as your keyspace. So aes 128 needs a 128bit key. Each password should have its own $salt, and the $base is a cryptographic nonce stored in textfile. (An attacker would have to read this file before he can crack the key, if this value is large like 128 bits then its a moot point.) Each message needs its own $iv and this value must also be a cryptographic nonce (similar to a salt). I would generate the $salt,$iv and $base_nonce from /dev/urandom. The IV can be stored in plain text in a column in your database along with the cipher text.
From a legal standpoint even if you build a secure cryptogrpahic system you still have problems with insider threats and if the server is completely compromised, all of the data will still be compromised. This really isn't an engineering problem.
The best defense against a legal threat is a strong Terms and Conditions written by a skilled lawyer.
One concern I'd have is the following. The "any users not logged in would be safe" part is too optimistic. By protecting the private key with the user's password, you open yourself up to various password brute-forcing attacks. Not just for current sessions, but for all. One effective one is to simply enumerate the top 100 common passwords, say, and just try them out against all users. Attacker is bound to uncover some keys. (I'm assuming that you're either storing per-user random salt with the user record, which the attacker can see, or else that you have a secret salt that the attacker was able to obtain through the compromise.)
I'm creating a service that will gather user data such as username, name, email, login password. How do I securely store this data? One thing I was thinking is store it encrypted in the DB so that if anyone gets access to the DB they won't be able to use the data.
But that arises two issues - #1 - much much slower search of the DB and #2 the encryption key will still be stored in the Perl script that will encrypt the data submitted by the user so if someone gets access to the whole server they'll still be able to decrypt all data.
So I was wondering what do services such as Twitter, Facebook, Hotmail, etc. use to securely store data?
BTW: as I said I'm working with Perl/MySQL.
Thank you all nice people!
You should try and avoid encrypting everything in the database. At the very least you will have to figure out which fields are being used for searching, relations, sorting etc and leave those unencrypted. You have to consider layers of encryption based on who you are hiding the data from. It would also help if you are clear about the differences between one way hashing (MD5, SHA), symmetric encryption and asymmetric encryption.
Encrypting passwords in the database makes sense because it hides the passwords from the dba or anyone with the ability to query the db directly. Actually, passwords are not encrypted, they are hashed. This way you can never recover a password, you can only generate a new one.
Data that needs to be recovered will be encrypted, so that it can be decrypted. There are various algorithms for doing that and they have different attributes of strength and cost (in terms of processing time and memory). You should choose one which suits your purpose.
The thing here is that encryption and decryption need to be done in the application and not in the database. The reason for this is that at some point there might be an open network between the database and the application servers and any data transmitted between the application and the database could be sniffed.
In deciding between symmetric and asymmetric encryption, there are two factors. One is for how long does this information need to be kept secure. If it is a bid on an auction that is going to end in 2 days, you may use symmetric encryption, because it ideally will not matter if people figure out things after that. Credit card numbers typically do not have to be kept safe. If you are storing the expiry date and the ccv numbers of the card then those have to be kept safe, typically for years. In that case you have to use asymmetric encryption.
In encrypting the whole file system it is not the performance degradation that you have worry about mainly. As far as I have seen, it requires a person to key in the correct password when OS is booting, and that requires physical access, and persons who can be trusted to know the password staying as close to the servers as the SLA requires. In this it is like setting a bios password or a grub password. If you indeed encrypt your file system, make sure to verify this or find a way around it.
Others have pointed out that encryption will incur a slowdown. How much will depend on lots of different factors, so you'll need to benchmark your code.
But in reality the most important thing to protect is probably the password since if that is compromised then the users' accounts on other services could be compromised as well since people tend to reuse the same username/email/password combos.
Fortunately passwords are easy to secure. Use a 1-way hash like SHA1 with a salt (to protect against rainbox tables) and never store the actual password in your DB. Store the salted-hash. Then when the user logs in you can check the pw they give you against the hashed one to make sure it matches without ever having to store what their pw really is.
See my answer here. The gist of it is that you can add security, but there is no 100% foolproof way to avoid a compromise of (some of) your user information if your web application is completely broken.
Encrypting the filesystem on which the database is stored minimizes many other types of vulnerabilities. Encrypting data in the database protects you against those and also against a user breaking your DB password. Encrypting data in the database with a hardware token provides the best protection but is inconvenient.
You have a few options:
You can encrypt the data in the middle tier
You can encrypt the database
You will encounter slower searches either way. In both cases, the encryption key is not included in the backup file set.
My university has a portal which students use to register for classes. If you want to get into a full class, you have to keep checking the portal, and sign up when the class has an opening.
I wrote a tool that can check for openings and register automatically, but it needs the students university username and password. These passwords are tied to email accounts, network shares, server logins, and most every other university service.
Is there any way to do this securely?
In security, the most important thing is the "threat model". What kind of attack do you fear?
somebody may steal the computer where this program runs on: put the computer in a locked room.
somebody may hack into the computer and read it from memory: use firewalls and other protection against remote attacks
other users may read the hard disk where the password is stored: only store the password in memory (which would require re-entering it every time you start the program)
the super user may read the password even if it is in memory: only run the program on a computer where you trust the superuser.
etc.
Unfortunately, this is not really possible -- at least not the way you want to do it -- unless the university provides a key-based authentication API. You could always ask them nicely, but they'll probably be too busy to help. If you give your users full disclosure and keep your server secure, it should be enough.
Actually, there is one way to do it through the web without storing passwords -- you could use a Java or Flash app. Unfortunately your users would need to leave the browser open while the app does its work, but this way you wouldn't need to store the information.
You could encrypt the password strings when you store them and then decrypt them when you need to try logging in. Simply generate a symmetric key and use that to encrypt and decrypt the passwords for storage and retrieval (respectively).
You can't store them entirely secure because you'd need to be able to encrypt and decrypt so one-way hash algorithms like MD5, SHA-1, SHA-2 wouldn't suffice. You could look into something like DES or Triple-DES encryption.
I do not think there is. As Martin pointed out one way encryption won't do it for you. Also this will create a maintenance nightmare for you - every time a user changes password you will have to update your data.
I think to make it really work you have to change the design: find a way to do the registration without the user password, i.e. talk to the owners of the app if they would give you an account through which you can do registration on behalf of somebody else
there's this interesting problem i can not solve myself. I will be very glad, if you help me.
Here's it:
there are many client applications that send data records to one MySQL server.
Few data records are not very important, but the whole database is. (You can imagine it is facebook DB :) )
Is there any way to ensure that
data from DB won't be used by anyone but true owner
DB will preserve essential features such as sorting etc.
assuming that attacker can mysteriously gain full access to server?
You can't simply encrypt data client-side and store it encrypted, since client application is wide-spread and attacker can get key from it.
Maybe adding some layers between application and DB, or combining encryption methods client- and server-side (using mysql built-in methods) will help?
As long as the database needs to start up and run unattended you can't hide the keys from a compromised root account (= 'mysterious full access'). Anywhere the database could possibly store the master key(s), the root will also have access. No amount of business layers or combination of client-server encryption will ever circumvent this simple fact. You can obfuscate it till the day after but if the prize is worth then root can get it.
One alternative is to require a manually assisted start up process, ie. a human enters the master key password during the server boot (or hardware module PIN), but this is extremely hard to maintain in real world, it requires a highly trusted employee to be on pager call to log in and start the database whenever there is downtime.
Solutions like TPM offer protection against physical loss of the server, but not against a compromised root.
Your root is as important as the database master key(s), so you must protect your root with the same care as the keys. This means setting up operating procedures, screening who has access to root, rotating the root password and so on and so forth. The moment someone gains 'mysteriously full access' the game is pretty much lost.
I pretty much agree with Remus Rusanu's answer.
Maintaining good security is hard, but you can always pay attention to what you do. When ever you access sensitive information carefully verify your query and make sure it cannot be spoofed or exploited to gain access to information which shouldn't be accessible by given client.
If you can roll out physical access to the box by the attacker then there are several things you can do to harden your security. First of all I'd configure ssh access only to only allow connections from specific IP or IP range (and of course no root access). You can also do that that on your firewall. This would mean that the weakest link is your server (the application which receives data/requests from clients, could be web-server and whatever scripts you use). Now you "just" have to make sure that no one can exploit your server. There are a lot more things you could do to harden your system, but it think it would be more appropriate to ask on ServerFault.
If you're worried about physical access to the PC, there isn't really much you can do and most stuff has already been mentioned in Remus answer.
There's also another option. This is by far the most ineffective method from speed and ease to develop viewpoint, but it would partly protect you from any kind of an attack on your server (including physical). It's actually quite simple, but a bit hard to implement - only store the encrypted data in the database and handle all encryption/decryption client-side using javascript or flash. Only the client will have the key and data will always be transfered over the wire and stored in encrypted format. The biggest drawback is that once client forgets the key there's no way back, the data is inaccessible.
Of course it's all matter of time, money and effort - with enough of these anything can be broken.
I've no idea if such a thing exists in MySql, but row-level-versioning in Oracle enables you to define access rights on row-level IN the database: so that means, regardless of what tool is being used to access the data, the user only ever sees the same selection as determined by his/her credentials.
So if my username/role is only allowed to see data limited by some WHERE clause, that can appended to each and every SELECT that appears in the database, regardless of whether it comes from a web app, a SQL querying tool, or whatever.
I will use a 2nd layer and a firwall between them.
so you have firewall ---- web server --- firewall -- 2nd layer server --- firewll --- db
it will be wise to use different platfroms between layers, it all depends how important is the data.
anyway - the web server should have no access to DB.
about preserving sort - if you use a file encrypotion mechisim - it will only protect you from Hard drive theaft.
if you encrypt the data it self, and if you do it smartly (storing the keys in a separate place) you will not loose sorting as you will look for the encryoted entry and not the real one- but now you have another thing to protect....