I'm creating a service that will gather user data such as username, name, email, login password. How do I securely store this data? One thing I was thinking is store it encrypted in the DB so that if anyone gets access to the DB they won't be able to use the data.
But that arises two issues - #1 - much much slower search of the DB and #2 the encryption key will still be stored in the Perl script that will encrypt the data submitted by the user so if someone gets access to the whole server they'll still be able to decrypt all data.
So I was wondering what do services such as Twitter, Facebook, Hotmail, etc. use to securely store data?
BTW: as I said I'm working with Perl/MySQL.
Thank you all nice people!
You should try and avoid encrypting everything in the database. At the very least you will have to figure out which fields are being used for searching, relations, sorting etc and leave those unencrypted. You have to consider layers of encryption based on who you are hiding the data from. It would also help if you are clear about the differences between one way hashing (MD5, SHA), symmetric encryption and asymmetric encryption.
Encrypting passwords in the database makes sense because it hides the passwords from the dba or anyone with the ability to query the db directly. Actually, passwords are not encrypted, they are hashed. This way you can never recover a password, you can only generate a new one.
Data that needs to be recovered will be encrypted, so that it can be decrypted. There are various algorithms for doing that and they have different attributes of strength and cost (in terms of processing time and memory). You should choose one which suits your purpose.
The thing here is that encryption and decryption need to be done in the application and not in the database. The reason for this is that at some point there might be an open network between the database and the application servers and any data transmitted between the application and the database could be sniffed.
In deciding between symmetric and asymmetric encryption, there are two factors. One is for how long does this information need to be kept secure. If it is a bid on an auction that is going to end in 2 days, you may use symmetric encryption, because it ideally will not matter if people figure out things after that. Credit card numbers typically do not have to be kept safe. If you are storing the expiry date and the ccv numbers of the card then those have to be kept safe, typically for years. In that case you have to use asymmetric encryption.
In encrypting the whole file system it is not the performance degradation that you have worry about mainly. As far as I have seen, it requires a person to key in the correct password when OS is booting, and that requires physical access, and persons who can be trusted to know the password staying as close to the servers as the SLA requires. In this it is like setting a bios password or a grub password. If you indeed encrypt your file system, make sure to verify this or find a way around it.
Others have pointed out that encryption will incur a slowdown. How much will depend on lots of different factors, so you'll need to benchmark your code.
But in reality the most important thing to protect is probably the password since if that is compromised then the users' accounts on other services could be compromised as well since people tend to reuse the same username/email/password combos.
Fortunately passwords are easy to secure. Use a 1-way hash like SHA1 with a salt (to protect against rainbox tables) and never store the actual password in your DB. Store the salted-hash. Then when the user logs in you can check the pw they give you against the hashed one to make sure it matches without ever having to store what their pw really is.
See my answer here. The gist of it is that you can add security, but there is no 100% foolproof way to avoid a compromise of (some of) your user information if your web application is completely broken.
Encrypting the filesystem on which the database is stored minimizes many other types of vulnerabilities. Encrypting data in the database protects you against those and also against a user breaking your DB password. Encrypting data in the database with a hardware token provides the best protection but is inconvenient.
You have a few options:
You can encrypt the data in the middle tier
You can encrypt the database
You will encounter slower searches either way. In both cases, the encryption key is not included in the backup file set.
Related
I am making an application where there is some PII (personally identifiable information) being stored, such as the Date of Birth along with the name and a few other fields. I want to encrypt the data with AES_ENCRYPT using CBC on MySQL, however there is an administrator who needs to see the information, but if I encrypt the user's information, the administrator won't be able to see. Should I encrypt the PII with the username and have the administrator not be able to see and encrypt the username with the username, or should I leave the username unencrypted but encrypt everything else with the username, leaving some security risks, but allowing the administrator to see the information. What should I do?
Generally speaking, encrypting data at rest in DBMSs with symmetric encryption like AES is both unnecessary and insufficient to protect it from attack.
Why?
For one thing, the weak link in a system like this is typically the web server, not the MySQL server machine. But the web server must contain the necessary key to both encrypt and decrypt the data. If an attacker pwns your web server, he takes possession of your key and your algorithms to generate keys, and therefore he pwns your sensitive data. He also has a really robust way to figure out what data you think is sensitive.
It's much less likely the attacker will be able to compromise your MySQL server directly; in situations involving PII those machines ordinarily live behind firewalls.
For another thing: your administrator. He will need to have possession of the key, and administrators are juicy targets for attackers. (I'm not saying he's a black hat, just that his white hat has a big red target on it.)
Another thing: it may give you a false sense of security.
(Notice that passwords are not encrypted: they are hashed. It's simple to compare a password to a hashed password and say "match" or "no match." But it's much harder to take a hashed password and recover the original one. So the argument that passwords are "encrypted" isn't relevant to this situation.)
I seriously suggest you spend your time and money not on encrypting your PII at rest as you suggest. Instead spend your resources to make sure your web site is secure, and making sure your MySQL server is safely behind a robust firewall and invisible to the public internet.
Look at the OWASP Top Ten vulnerabilities. Spend time dealing with them on your site. Use a penetration test tool like burp scan or zap scan to try to break in.
Securing your web site is like the old joke about the two guys and the bear.
Bear: roar.
Joe: I hope I can outrun the bear.
Mike: I just have to outrun you.
In cybersecurity, you don't want to be Joe.
I'm already using salted hashing to store passwords in my database, which means that I should be immune to rainbow table attacks.
I had a thought, though: what if someone does get hold of my database? It contains the users' email addresses. I can't really hash these, because I'll be using them to send notification emails, etc..
Should I encrypt them?
Bruce Schneier has a good response to this kind of problem.
Cryptography is not the solution to your security problems. It might be part of the solution, or it might be part of the problem. In many situations, cryptography starts out by making the problem worse, and it isn't at all clear that using cryptography is an improvement.
Essentially encrypting your emails in the database 'just in case' is not really making the database more secure. Where are the keys stored for the database? What file permissions are used for these keys? Is the database accesable publically? Why? What kind of account restrictions are in place for these accounts? Where is the machine stored, who has physical access to this box? What about remote login/ssh access etc. etc. etc.
So I guess you can encrypt the emails if you want, but if that is the extent of the security of the system then it really isn't doing much, and would actually make the job of maintaining the database harder.
Of course this could be part of an extensive security policy for your system - if so then great!
I'm not saying that it is a bad idea - But why have a lock on the door from Deadlocks'R'us which cost $5000 when they can cut through the plywood around the door? Or come in through the window which you left open? Or even worse they find the key which was left under the doormat. Security of a system is only as good as the weakest link. If they have root access then they can pretty much do what they want.
Steve Morgan makes a good point that even if they cannot understand the email addresses, they can still do a lot of harm (which could be mitigated if they only had SELECT access)
Its also important to know what your reasons are for storing the email address at all. I might have gone a bit overboard with this answer, but my point is do you really need to store an email address for an account? The most secure data is data that doesn't exist.
I realize this is a dead topic, but I agree with Arjan's logic behind this. There are a few things I'd like to point out:
Someone can retrieve data from your database without retrieving your source code (i.e. SQL injection, third-party db's). With this in mind, it's reasonable to consider using an encryption with a key. Albeit, this is only an added measure of security, not the security...this is for someone who wants to keep the email more private than plaintext,
In the off chance something is overlooked during an update, or an attacker manages to retrieve the emails.
IMO: If you plan on encrypting an email, store a salted hash of it as well. Then you can use the hash for validation, and spare the overhead of constantly using encryption to find a massive string of data. Then have a separate private function to retrieve and decrypt your emails when you need to use one.
In common with most security requirements, you need to understand the level of threat.
What damage can be done if the email addresses are compromised?
What's the chance of it happening?
The damage done if email addresses are REPLACED could be much greater than if they're EXPOSED. Especially if you're, for example, using the email address to verify password resets to a secure system.
The chance of the passwords being either replaced or exposed is much reduced if you hash them, but it depends what other controls you have in place.
I would say it depends on the application of your database.
The biggest problem is, where do you store the encryption key? Because if the hacker has excess to anything more than your DB, all your efforts are probably wasted. (Remember, your application will need that encryption key to decrypt and encrypt so eventually the hacker will find the encryption key and used encryption scheme).
Pro:
A leak of your DB only will not expose the e-mail addresses.
Cons:
Encryption means performance loss.
Allot of database actions will be harder if not impossible.
Don't accidentally conflate encryption with obfuscation. We commonly obfuscate emails to prevent spam. Lots of web sites will have "webmaster _at_ mysite.com" to slow down crawlers from parsing the email address as a potential spam target. That should be done in the HTML templates -- there's no value to doing this in persistent database storage.
We don't encrypt anything unless we need to keep it secret during transmission. When and where will your data being transmitted?
The SQL statements are transmitted from client to server; is that on the same box or over a secure connection?
If your server is compromised, you have an unintentional transmission. If you're worried about this, then you should perhaps be securing your server. You have external threats as well as internal threats. Are ALL users (external and internal) properly authenticated and authorized?
During backups you have an intentional transmission to backup media; is this done using a secure backup strategy that encrypts as it goes?
Both SQL Server and Oracle (and I believe also others DBs) support encryption of data at the database level. If you want to encrypt something why don't simply abstract the access to the data that could be encrypted on the database server side and left the user choose if use the encrypted data (in this case the SQL command will be different) or not. If the user want to user encrypted data then it can configure the database server and all the maintenance work connected with key management is made using standard DBA tool, made from the DB vendor and not from you.
A copy of my answer at What is the best and safest way to store user email addresses in the database?, just for the sake of the search...
In general I agree with others saying it's not worth the effort. However, I disagree that anyone who can access your database can probably also get your keys. That's certainly not true for SQL Injection, and may not be true for backup copies that are somehow lost or forgotten about. And I feel an email address is a personal detail, so I wouldn't care about spam but about the personal consequences when the addresses are revealed.
Of course, when you're afraid of SQL Injection then you should make sure such injection is prohibited. And backup copies should be encrypted themselves.
Still, for some online communities the members might definitely not want others to know that they are a member (like related to mental healthcare, financial help, medical and sexual advice, adult entertainment, politics, ...). In those cases, storing as few personal details as possible and encrypting those that are required (note that database-level encryption does not prevent the details from showing using SQL Injection), might not be such a bad idea. Again: treat an email address as such personal detail.
For many sites the above is probably not the case, and you should focus on prohibiting SELECT * FROM through SQL Injection, and making sure visitors cannot somehow get to someone else's personal profile or order information by changing the URL.
It's worth to encrypt data in Databases, it's not making it a bit more difficult but way more difficult when its encrypted in the right way so stop philosophy and encrypt the sensitive data ;)
#Roo
I somewhat agree to what you are saying but isn't it worth encrypting the data just to make it a little more difficult for someone to get it?
With your reasoning, it would be useless to have locks or alarms in your house, because they can also easily be compromised.
My reply:
I would say that if you have sensitive data that you don't want to fall in the wrong hands, you should probably do it as hard as you can for a hacker to get it, even if it's not 100% fool proof.
I miss one important answer here.
When you have European users, you have to comply with the GDPR rules.
Email addresses are considered personal data meaning Art.5 does apply on email addresses.
processed in a manner that ensures appropriate security of the
personal data, including protection against unauthorised or unlawful
processing and against accidental loss, destruction or damage, using
appropriate technical or organisational measures (‘integrity and
confidentiality’).
Of course this does not say that you must encrypt email addresses. But by encrypting the data you do protect it from snooping employees. And protect yourself as a developer from requests to make a manual change in the database.
You really have to weigh your worst case senario of someone obtaining those email addresses, the likelihood of someone obtaining them, and your extra effort/time needed to impliement the change.
I have very sensitive data of mine in the database available in the public domain. Though I have secured it to prevent sql injections, I little bit worried about the data. Can I able to encrypt the entire database using any advance encryption algorithms. Is that a correct way of securing the data? Any suggestions?
Database: MySql
I'm not sure to understand how a database in the public domain (this has some very specific legal meaning) can contain sensitive data.
What you might perhaps do is encrypt some columns of your database, that is have the encryption and decryption happening in your client application, on some columns of each rows.
First of all, only data encryption is not the answer.
You should know and control "Who can access and accessed data".
Although you encrypt data itself, attacker can get the encrypted data.
To prevent it, you should control the access privilege or authority.
Also, you should know who accessed encrypted data. Because in these days, data can be stolen by not only external users, but also internal users.
You should encrypt data, control privilege and audit who accessed.
In those days, those three(3) technology have been developed in each way.
Most companies have bought each solutions.
But for personal user, it is hard to buy it and build those environment itself.
I would you like to review MyDiamo that provides those three(3) functions.
I receive data, and use aes or blowfish to encrypt it before saving it to the database, so the encryption is done at the application level. If someone steals the database, the data will be relatively safe unless they stole the application also (where the key is stored/accessed).
I'm now looking into database encryption with libraries like ezNcrypt for MySQL, Encryption-GENERAL, or SQLCipher.
But I don't quite understand how database encryption works. If the application only passes raw unaltered data to the database, and the database decrypts the data itself somehow, wouldn't that make database-level encryption less secure if the database was stolen since 100% of the encryption component was stolen?
In my current situation, if a database is stolen, the attacker would have to steel the second component (the key which is at the application level) to decrypt the database. But with database encryption, the database itself has full responsibility of the encryption, so by stealing the database, wouldn't the attacker have everything needed to decrypt the database?
Maybe I'm not clear on how database-level decryption works.
The encryption algorithm is applied at different points in your application stack, but at the end of the day the process is exactly the same regardless if you use an application or db-layer solution. The key has to be stored somewhere and if an attacker gets both the encrypted data and the key, then they have access to the data.
So, the real question boils down to how you store your key:
Plaintext - if it's in plaintext somewhere in the filesystem then that's your weak point. Maybe it's in the application code, maybe in a config file somewhere. In any case, someone who gains administrator access to the server (or simply steals the hard drive) has what they need, and obscurity is your only protection.
Manually-entered - If you have a human user enter the key when the application/database/pc is started, then you mostly* alleviate the issue of a plaintext key. In exchange, you require a user to remember the key and you give up the ability to have a completely automated restart (bad news for a server that needs to stay up).
* I say mostly because technically the key is still available in plaintext somewhere in RAM. But that's a lot harder to get at than something stored on disk (memory swapping notwithstanding).
Regarding MySQL specifically, you might find this thread helpful.
What method do you use to authenticate your users? If the authentication method is the same in each case then encrypting in the application is not likely to be any more or less secure than in the database. The most likely vector of attack is still the same. It seems much less likely that an intruder would have an opportunity actually to steal the database rather than just gain access to it - unless you are doing something very silly.
there's this interesting problem i can not solve myself. I will be very glad, if you help me.
Here's it:
there are many client applications that send data records to one MySQL server.
Few data records are not very important, but the whole database is. (You can imagine it is facebook DB :) )
Is there any way to ensure that
data from DB won't be used by anyone but true owner
DB will preserve essential features such as sorting etc.
assuming that attacker can mysteriously gain full access to server?
You can't simply encrypt data client-side and store it encrypted, since client application is wide-spread and attacker can get key from it.
Maybe adding some layers between application and DB, or combining encryption methods client- and server-side (using mysql built-in methods) will help?
As long as the database needs to start up and run unattended you can't hide the keys from a compromised root account (= 'mysterious full access'). Anywhere the database could possibly store the master key(s), the root will also have access. No amount of business layers or combination of client-server encryption will ever circumvent this simple fact. You can obfuscate it till the day after but if the prize is worth then root can get it.
One alternative is to require a manually assisted start up process, ie. a human enters the master key password during the server boot (or hardware module PIN), but this is extremely hard to maintain in real world, it requires a highly trusted employee to be on pager call to log in and start the database whenever there is downtime.
Solutions like TPM offer protection against physical loss of the server, but not against a compromised root.
Your root is as important as the database master key(s), so you must protect your root with the same care as the keys. This means setting up operating procedures, screening who has access to root, rotating the root password and so on and so forth. The moment someone gains 'mysteriously full access' the game is pretty much lost.
I pretty much agree with Remus Rusanu's answer.
Maintaining good security is hard, but you can always pay attention to what you do. When ever you access sensitive information carefully verify your query and make sure it cannot be spoofed or exploited to gain access to information which shouldn't be accessible by given client.
If you can roll out physical access to the box by the attacker then there are several things you can do to harden your security. First of all I'd configure ssh access only to only allow connections from specific IP or IP range (and of course no root access). You can also do that that on your firewall. This would mean that the weakest link is your server (the application which receives data/requests from clients, could be web-server and whatever scripts you use). Now you "just" have to make sure that no one can exploit your server. There are a lot more things you could do to harden your system, but it think it would be more appropriate to ask on ServerFault.
If you're worried about physical access to the PC, there isn't really much you can do and most stuff has already been mentioned in Remus answer.
There's also another option. This is by far the most ineffective method from speed and ease to develop viewpoint, but it would partly protect you from any kind of an attack on your server (including physical). It's actually quite simple, but a bit hard to implement - only store the encrypted data in the database and handle all encryption/decryption client-side using javascript or flash. Only the client will have the key and data will always be transfered over the wire and stored in encrypted format. The biggest drawback is that once client forgets the key there's no way back, the data is inaccessible.
Of course it's all matter of time, money and effort - with enough of these anything can be broken.
I've no idea if such a thing exists in MySql, but row-level-versioning in Oracle enables you to define access rights on row-level IN the database: so that means, regardless of what tool is being used to access the data, the user only ever sees the same selection as determined by his/her credentials.
So if my username/role is only allowed to see data limited by some WHERE clause, that can appended to each and every SELECT that appears in the database, regardless of whether it comes from a web app, a SQL querying tool, or whatever.
I will use a 2nd layer and a firwall between them.
so you have firewall ---- web server --- firewall -- 2nd layer server --- firewll --- db
it will be wise to use different platfroms between layers, it all depends how important is the data.
anyway - the web server should have no access to DB.
about preserving sort - if you use a file encrypotion mechisim - it will only protect you from Hard drive theaft.
if you encrypt the data it self, and if you do it smartly (storing the keys in a separate place) you will not loose sorting as you will look for the encryoted entry and not the real one- but now you have another thing to protect....