I have a web application that uses a symmetric encryption algorithm.
How would you store the secret key and initialization vector? Storing as a literal in the code seems like a bad idea. How about app settings? What is the best practice here?
One standard approach in the webapp world is to split the key and put it in different places. E.g., you might split the key and put part of it in the filesystem (outside of the 'webapps' directory), part of it in the JNDI configuration (or .net equivalent), and part of it in the database. Getting any single piece isn't particularly hard if you're compromised, e.g., examining backup media or SQL injection, but getting all of the pieces will require a lot more work.
You can split a key by XOR-ing it with random numbers of the same size. (Use a cryptographically strong random number generator!) You can repeat this process several times if you want to split the key into multiple pieces. At the end of the process you want, e.g., three partial keys such that p1 ^ p2 ^ p3 = key. You might need to base64-encode some of the partial keys so they can be stored properly, e.g., in a JNDI property.
(There are more sophisticated ways to split a key, e.g., an n-of-m algorithm where you don't require all of the pieces to recreate the key, but that's -far- beyond what you need here.)
If you can require the user to actively enter the password, there are PBE (password-based encryption) algorithms that convert a password to a good symmetric key. You want to find one that requires an external file as well. Again it's a case the tape backups or the password itself isn't enough, you need both. You could also use this to split the password into two pieces with JNDI - you can use a plaintext passphrase in JNDI and an initialization file somewhere in the filesystem.
Finally, whatever you do be sure you can 'rekey' your application fairly easily. One approach is to use the password obtained above to decrypt another file that contains the actual encryption key. This makes it easy to change the password if you think it's been compromised without requiring a massive reencryption of all of the data - just reencrypt your actual key.
Is it possible for you to enter a password interactively whenever the application starts up? That way you don't have to store the key, or at least any keys (whether they are symmetric or private keys) can be encrypted with this "bootstrap" password.
If not, store your secret key in a file by itself and modify its permissions to make it accessible only to the user running the web application.
These approaches are platform-agnostic. For more concrete suggestions, information about your platform would be helpful.
By the way, an initialization vector should be used for only one message. And IVs do not have be kept secret, so you could store it anywhere, but storing it with the one message that uses it is customary.
I have used an approach where my application requires a symmetric key when it starts and looks for it in a certain file. Once the application has started up I remove the file. A copy of the file is kept remotely for any required restarts. Obviously this approach is not viable if your applciation has frequent restarts.
Another alternative would be a certificate manager such as the Windows Certificate Store. It can store certificates and their keys securely and it's also possible to mark private keys as non-exportable so it would require some serious hacking to get the key out. Your application could load its certificate from the Certificate Store and be able to call operations to sign requests or generate new symmetric keys. In addition you can assign permissions to different certifcate stores so that only certain privileged accounts would be able to access the certificate.
stick it in the web.config and encrypt that section
This SO question talks more about web.config encryption
This should help ...
http://msdn.microsoft.com/en-us/library/ms998280.aspx
But, you really should consider going to PKI if you are serious about protecting your data.
We have a slightly different, but related issue. We have keys generated every few days, and when decrypting, we have to try all our keys because we do not know which day the encryption took place. What we did was to encrypt the keys one more time and store them as secretes. This way, we only have one set of keys to manage.
For secure storing of encryption key you can use KMS service of AWS. Please use this service for storing such confidential keys. PFB url for kms service.
documentation : https://aws.amazon.com/kms/
Related
Edited based on comments.
My company is developing a system and is in the early design stages. I am the lead on the project. Some of the information to be stored in the system is highly confidential. The system must protect this information such that only authorized users have the ability to see this information. In fact, it is such that we want to protect it from ourselves even. The database to be used will probably be mySQL but does not have to be. I am aware that the data can be encrypted in the database. I am also aware that the software will have to be able to retrieve this data from the database for authorized users. I do not want any developers, including myself, to be able to decrypt and view this information -- in production -- but they do need to be able to view it during development(we will disable encryption during development).
The software will be a cloud based offering but will have an in house hosting option as well.
We can have a software based encryption and I realize that if software can get at the data then it's accessible - but I want this as tight as possible so that a developer cannot (including myself) view this data, but signed in, authorized users, can view it.
I'm stuck in my own thinking box about how to do this so I'm seeking advice as to how to secure this.
The client side will need a decryption key to decrypt the data. The software will have to allow access to this key for authorized users. The users cannot hold this key as the "client" devices could be virtually anything. Now I know I have no control over what the end users who have access do with the data, and that's not by concern. I want them to be able to store this information and have it secure in such a way that I cannot see it! I also do not want to have the ability to make myself authorized to see it - but to have that ability rest in the hands of a user side trusted administrator.
The data can be binary data. I do not want to encrypt the entire database, nor do I want to ever need to re-encrypt data because it can be huge amounts of data.
I have read posts about using GnuPG to handle this with public key for encryption and decryption via any one of a set of multiple private keys. This might be feasable for this but I'm looking for multiple solutions. My "issue" with this is the pass phrase protecting the private key and issueance of these private keys. I really don't want to user having to key in the passphrase associated with the private key every time that they login to the system or try to view the encrypted/protected data, but I don't want to store this passphrase anywhere a developer can see it either!
I understand about PKI but it doesn't offer protection from developers who can access the master decryption key which is shared using a public/private keypair.
I've edited the question with a proposed solution, please punch holes in it.
Development Language is C#. Microsoft supports asymetric keys which is what we want to use.
An X509 Certificate will be utilzied for the encryption keys. The public key will be stored in the database and used for all encryption. The private key will be stored in the database as well but encrypted using a passphrase entered by a trusted administrator. (This is the weak link in my mind, more on this later).
The administrator will create users who have access to the encrypted data. The users will be issued a copy of the private decryption key via the administrator and this key will be associated with their user account and encrypted in the system using their initial pass phrase.
When the user first logs in and is prompted to change their pass phrase, the private key will be re-encrypted using their new pass phrase. Their pass phrase is not stored in the database, only a hash for validation. Their keyed pass phrase will be encrypted using a timestamp or other data and will be stored on their local computer not on the server or in the servers identifable memory except when it is used for decryption of the private key. A software algorithm will be created to retrieve the decryption key for the encrypted master decryption key associated with this user.
Thus users can be created and revoked at the will of the administrator. Users can have their pas phrases reset - but in that case a new decryption key would need to be issued because the old one would not be decryptable any longer.
The one flaw I see, and I'm open to input, is if the administrator forgets their pass phrase and can no longer issue private keys. We could write code that given a users password, recovers the private key and encrypts it using the administrators new password when they enter it.
Another weak link is user pass phrase. If that becomes compromised then a developer could use that to obtain a copy of the private decription key and decrypt data.
Also - if for some reason an administrator needs to be let go we have the same issue as above where we don't have a copy of the private key accessible to share but we could get one using a users pass phrase.
Any advice on overcoming the perceived weaknesses?
We cannot tolerate exposure of the master decryption private key - if it is exposed then the entire database needs to be re-encrypted using a new keypair.
Thank you for your comments.
If you need to protect the data from yourselves, the only option is end-to-end encryption. User has a private key and it never leaves their computer. Devs/sysadmins never have private keys and can never decrypt.
This would require you make your client software source available to users so they can see that you are not leaking their info.
Use USB smart cards for easier/safer key management. They key is generated on the smart card and never leaves it. The smart card does the decryption on your apps behalf.
This make searching harder, but you can fix some of this by storing hashes along w the encrypted values, and searching on them.
Update, answering your questions in your comment:
Web based is not an option. You need a thick client
There is no "master key"
Please clarify "authorized user". Authorized by whom? Please provide an example.
Another option would be to skip encryption entirely and rely solely on access control. Use a db like postgresql, ms sql, or oracle that does row-level security. Don't give access to any one DBA. Give part of password to one dba, another part to another. DBA's need to be in same room to access the server.
Either way, you should be familiar with https://en.wikipedia.org/wiki/Secret_sharing
If you were to condider SQL Server instead of mysql, then the most secure design I can think of that meets your requirements is to require SQL Server Enterprise Edition an Extensible Key Management system that uses a Hardware Security Module. SQL Server Extensible Key Management enables the encryption keys that protect the database files to be stored in an off-box device such as a smartcard, USB device, or EKM/HSM module. EKMs are only supported in Enterprise edition. An HSM will allow for the delegation and isolation of managing the encryption keys and functions under a group other than both developers and DBAs, if that is your intent. Other features I would suggest in the design include the following options to complete end to end transport\session encryption:
Installing a domain or CA certificate on the SQL Server and enabling Force Protocol Encryption or enabling IPSec on the host of the instance
Installing a domain or CA certificate on all IIS websites or web services and forcing encryption in the connection strings and https
The benefits of this design over SQL Server Always Encrypted are that the encryption is protected by a single system that can be monitored and defended with more focus as opposed to numerous clients that increase exposure to theft of the private key from one of the users key stores. Also, key rotation for EKMs should be easy for those who are managing the system. It is also possible with an EKM to require that more than one person be involved in generating keys.
This is more of a security question regarding using AES_ENCRYPT to generate encryption on data being inserted into a MySQL database.
What is the best location in which to store the key that is used to encrypt the data ? Obviously not in the database ! :)
Well, you don't have many options. Wherever you put that key (database, code, file), it is easily found as long as other people has access to the machine.
What you could do is that you encrypt that key with another key based on some password (which is not stored anywhere, at least not locally) and ask for that password on the startup of the application. This way, you can store encrypted AES_ENCRYPT key into your database, decrypt it after logging in with your password and start using it.
Security by obscurity!
If your webserver is compromised, then the attacker can access the key, no matter where it's stored - since the code must be able to find the key to do the encryption/decryption - and the code explains where it finds the key. The only scenario where this adds real value is in protecting the data outside of the application (e.g. on a backup tape). However since you're compromising the DBMs's ability to optimize queries and creating a much bigger data footprint, for such a purpose as a backup, it makes a lot more sense to encrypt the backup or the filesystem - not individual data items.
Even if you use keys which are not permanently stored within your application (e.g. a HTTP basic authentication password supplied over SSL) there are still a lot of risks that the data will be compromised - and you've got problems with sharing data between different users.
In order to provide a sensible answer we need to know what the threat model is and whether you have external constraints such as PCI-DSS
The issues of securely storing keys and passwords used in your PHP / Python / other application on a server is not only related to hiding the keys from an attacker who has gained root on your sever, although you can make it more difficult for an attacker who has gained root to access them, it can eventually be done.
However, keys / passwords can be lost in many other ways and so must be protected. For example, if your software is being updated from a development environment, i.e. being pushed and pulled through a git server, you do not want the keys to be included in plain-text in the source code. That would give anyone on your development team access to them.
One option to store keys "more securely" is to have them configured as environment variables and then include them in your application by accessing that environment variable instead of having the key in "plain-text" within your application.
However, this requires that you set the environment variable to be persistent so that if you reboot the sever it will automatically be set again, or else you must set it each time you reboot.
If you are using Apache web-server, you may also set Apache environment variables for sensitive keys / passwords in the httpd.conf file, and then access them from your PHP script. You can also restrict the permissions on the httpd.conf file for only root to have read/write.
// Example use of getenv()
$sensitive_key = getenv("VERY_SENSITIVE_KEY");
// Example use of apache_getenv()
$sensitive_key = apache_getenv("VERY_SENSITIVE_KEY");
This means that the key / password is not included in the application source code itself, and will be less likely to escape the server.
I'm creating a service that will gather user data such as username, name, email, login password. How do I securely store this data? One thing I was thinking is store it encrypted in the DB so that if anyone gets access to the DB they won't be able to use the data.
But that arises two issues - #1 - much much slower search of the DB and #2 the encryption key will still be stored in the Perl script that will encrypt the data submitted by the user so if someone gets access to the whole server they'll still be able to decrypt all data.
So I was wondering what do services such as Twitter, Facebook, Hotmail, etc. use to securely store data?
BTW: as I said I'm working with Perl/MySQL.
Thank you all nice people!
You should try and avoid encrypting everything in the database. At the very least you will have to figure out which fields are being used for searching, relations, sorting etc and leave those unencrypted. You have to consider layers of encryption based on who you are hiding the data from. It would also help if you are clear about the differences between one way hashing (MD5, SHA), symmetric encryption and asymmetric encryption.
Encrypting passwords in the database makes sense because it hides the passwords from the dba or anyone with the ability to query the db directly. Actually, passwords are not encrypted, they are hashed. This way you can never recover a password, you can only generate a new one.
Data that needs to be recovered will be encrypted, so that it can be decrypted. There are various algorithms for doing that and they have different attributes of strength and cost (in terms of processing time and memory). You should choose one which suits your purpose.
The thing here is that encryption and decryption need to be done in the application and not in the database. The reason for this is that at some point there might be an open network between the database and the application servers and any data transmitted between the application and the database could be sniffed.
In deciding between symmetric and asymmetric encryption, there are two factors. One is for how long does this information need to be kept secure. If it is a bid on an auction that is going to end in 2 days, you may use symmetric encryption, because it ideally will not matter if people figure out things after that. Credit card numbers typically do not have to be kept safe. If you are storing the expiry date and the ccv numbers of the card then those have to be kept safe, typically for years. In that case you have to use asymmetric encryption.
In encrypting the whole file system it is not the performance degradation that you have worry about mainly. As far as I have seen, it requires a person to key in the correct password when OS is booting, and that requires physical access, and persons who can be trusted to know the password staying as close to the servers as the SLA requires. In this it is like setting a bios password or a grub password. If you indeed encrypt your file system, make sure to verify this or find a way around it.
Others have pointed out that encryption will incur a slowdown. How much will depend on lots of different factors, so you'll need to benchmark your code.
But in reality the most important thing to protect is probably the password since if that is compromised then the users' accounts on other services could be compromised as well since people tend to reuse the same username/email/password combos.
Fortunately passwords are easy to secure. Use a 1-way hash like SHA1 with a salt (to protect against rainbox tables) and never store the actual password in your DB. Store the salted-hash. Then when the user logs in you can check the pw they give you against the hashed one to make sure it matches without ever having to store what their pw really is.
See my answer here. The gist of it is that you can add security, but there is no 100% foolproof way to avoid a compromise of (some of) your user information if your web application is completely broken.
Encrypting the filesystem on which the database is stored minimizes many other types of vulnerabilities. Encrypting data in the database protects you against those and also against a user breaking your DB password. Encrypting data in the database with a hardware token provides the best protection but is inconvenient.
You have a few options:
You can encrypt the data in the middle tier
You can encrypt the database
You will encounter slower searches either way. In both cases, the encryption key is not included in the backup file set.
I receive data, and use aes or blowfish to encrypt it before saving it to the database, so the encryption is done at the application level. If someone steals the database, the data will be relatively safe unless they stole the application also (where the key is stored/accessed).
I'm now looking into database encryption with libraries like ezNcrypt for MySQL, Encryption-GENERAL, or SQLCipher.
But I don't quite understand how database encryption works. If the application only passes raw unaltered data to the database, and the database decrypts the data itself somehow, wouldn't that make database-level encryption less secure if the database was stolen since 100% of the encryption component was stolen?
In my current situation, if a database is stolen, the attacker would have to steel the second component (the key which is at the application level) to decrypt the database. But with database encryption, the database itself has full responsibility of the encryption, so by stealing the database, wouldn't the attacker have everything needed to decrypt the database?
Maybe I'm not clear on how database-level decryption works.
The encryption algorithm is applied at different points in your application stack, but at the end of the day the process is exactly the same regardless if you use an application or db-layer solution. The key has to be stored somewhere and if an attacker gets both the encrypted data and the key, then they have access to the data.
So, the real question boils down to how you store your key:
Plaintext - if it's in plaintext somewhere in the filesystem then that's your weak point. Maybe it's in the application code, maybe in a config file somewhere. In any case, someone who gains administrator access to the server (or simply steals the hard drive) has what they need, and obscurity is your only protection.
Manually-entered - If you have a human user enter the key when the application/database/pc is started, then you mostly* alleviate the issue of a plaintext key. In exchange, you require a user to remember the key and you give up the ability to have a completely automated restart (bad news for a server that needs to stay up).
* I say mostly because technically the key is still available in plaintext somewhere in RAM. But that's a lot harder to get at than something stored on disk (memory swapping notwithstanding).
Regarding MySQL specifically, you might find this thread helpful.
What method do you use to authenticate your users? If the authentication method is the same in each case then encrypting in the application is not likely to be any more or less secure than in the database. The most likely vector of attack is still the same. It seems much less likely that an intruder would have an opportunity actually to steal the database rather than just gain access to it - unless you are doing something very silly.
I have a web based (perl/MySQL) CRM system, and I need a section for HR to add details about disciplinary actions and salary.
All this information that we store in the database needs to be encrypted so that we developers can't see it.
I was thinking about using AES encryption, but what do I use as the key? If I use the HR Manager's password then if she forgets her password, we lose all HR information. If she changes her password, then we have to decrypt all information and re-encrypt with the new password, which seems inefficient, and dangerous, and could go horrifically wrong if there's an error half way through the process.
I had the idea that I could have an encryption key that encrypts all the information, and use the HR manager's password to encrypt the key. Then she can change her password all she likes and we'll only need to re-encrypt the key. (And without the HR Manager's password, the data is secure)
But then there's still the problem of multi-user access to the encrypted data.
I could keep a 'plaintext' copy of the key off site, and encrypt it with each new HR person's password. But then I know the master key, which doesn't seem ideal.
Has anyone tried this before, and succeeded?
GnuPG allows documents to be encrypted using multiple public keys, and decrypted using any one of the corresponding private keys. In this way, you could allow data to be encrypted using the public keys of the everyone in the HR department. Decryption could be performed by any one having one of the private keys. Decryption would require both the private key and the passphrase protecting the key to be known to the system. The private keys could be held within the system, and the passphrase solicited from the user.
The data would probably get quite bloated by GnuPG using lots of keys: it has to create a session key for the payload and then encrypt that key using each of the public keys. The encrypted keys are stored alongside the data.
The weak parts of the system are that the private keys need to be available to the system (ie. not under the control of the user), and the passphrase will have to pass through the system, and so could be compromised (ie. logged, stolen) by dodgy code. Ultimately, the raw data passes through the system too, so dodgy code could compromise that without worrying about the keys. Good code review and release control will be essential to maintain security.
You are best avoiding using MySQL's built in encryption functions: these get logged in the replication, slow, or query logs, and can be visible in the processlist - and so anyone having access to the logs and processlist have access to the data.
Why not just limit access to the database or table in general. That seems much easier. If the developer has access to query the production, there is no way to prevent them from seeing the data b/c at the end of the day, the UI has to decrypt / display the data anwyays.
In the experience I've had, the amount of work it takes to achieve the "developers cannot see production data at all" is immense and nearly imposible. At the end of the day, if the developers have to support the system, it will be difficult to achieve. If you have to debug a production problem, then it's impossible not to give some developers access to production data. The alternative is to create a large number of levels and groups of support, backups, test data, etc..
It can work, but it's not as easy as business owners may think.
Another approach is to use a single system-wide key stored in the database - perhaps with a unique id so that new keys can be added periodically. Using Counter Mode, the standard MySQL AES encryption can be used without directly exposing the cleartext to the database, and the size of the encrypted data will be exactly the same as the size of the cleartext. A sketch of the algorithm:
The application generates a unique initial counter value for the record. This might be based on some unique attribute of the record, or you could generate and store a unique value for this purpose.
The application generates a stream of counter blocks for the record based on the initial counter value. The counter stream must be the same size or up to 1 block larger than the cleartext.
The application determines which key to use. If keys are being periodically rotated, then the most recent one should be used.
The counter stream is sent to the database to be encrypted: something like
select aes_encrypt( 'counter', key ) from hrkeys where key_id = 'id';
The resulting encrypted counter value is trimmed to the length of the cleartext, and XORed with the cleartext to produce the encrypted text.
The encrypted text is stored.
Decryption is exactly the same process applied to the encrypted text.
The advantages are that the cleartext never goes any where near the database, and so the administrators cannot see the sensitive data. However, you are then left with the problem of preventing your adminstrators from accessing the encrypted counter values or the keys. The first can be achieved by using SSL connections between your application and database for the encryption operations. The second can be mitigated with access control, ensuring that the keys never appear in the database dumps, storing the keys in in-memory tables so that access control cannot be subverted by restarting the database with "skip-grants". Ultimately, the only way to eliminate this threat is to use a tamper-proof device (HSM) for performing encryption. The higher the security you require, the less likely you will be able to store the keys in the database.
See Wikipedia - Counter Mode
I am just thinking out loud.
This seems to call for a public/private key mechanism. The information would be stored encrypted with the HR public key and would only be viewable by someone in possession of the associated private key.
This, to me, seems to rule out a web based interface to view these confidential data (entering them via the web interface is certainly feasible).
Given that individuals come and go, tying the keys to a specific person's account seems infeasible. Instead, one must handle key distribution separately and have a mechanism for someone to change the keypair used (and re-encrypt the database — again without the use of a web interface) in case the current HR manager is replaced with someone else. Of course, nothing would prevent the HR manager from dumping all the data before leaving while before the keys are replaced.
I'm not sure how feasible this is currently, or what current stable DB systems have support for this, but alternate authentication mechanisms at the database level may help. For example Drizzle, a refactoring of the MySQL code base, supports (or aims to?) completely pluggable authentication, allowing no auth, server housed auth, or auth through PAM or some other mechanism, meaning you can use LDAP.
If you had different levels of access based on the database connection, and the application login also specified what you could actually access in the database, you could theoretically build a system where it wasn't possible to access the confidential database info unless using an account with specific access rights, regardless of the privilege escalation attempts in the application itself.
As long as the people setting user account access rights can be trusted or themselves are OK to see the confidential information, this should be fairly secure.
P.S. It might be useful to use a generic DB connection for "regular" application information, but when an attempt to access confidential information is made, then the specific DB connection is attempted. This allows for a few DB connections to handle most requests, assuming the majority of users aren't viewing confidential info. Otherwise, a separate DB connection per user may become burdensome to the DB.