Hey there I have a question about the best way to store data encrypted in my database. I use Node.js, a MySQL database and sequelize 6.6.5 as ORM.
Here's what I do:
With beforeCreate and beforeUpdate hooks I'm encrypting my data
before storing it in the database.
With the beforeFind hook I encrypt the condition for querying before
doing so.
And with afterCreate, afterUpdate and afterFind hooks I decrypt the
data to work with it after creating updating or querying for it.
But the the querying itself raises some problems for me which I think come with the way I encrypt my data. I use the Node.js crypto module with the aes-256-cbc algorithm and a random IV for every encryption.
With the random IV every encryption results in a different string. That's why even if I use the beforeFind hook to encrypt my condition the query will never return any result.
myModel.create({myField: "someData"});
// with the beforeCreate hook encrypting this the database will contain something like this
// myField: "1ac4e952cf6207e5fd79630e0e82c901"
myModel.findAll({ where: { myField: "someData" } });
// The beforeFind hook encrypts this condition but as mentioned the result is not the same
// as the encrpyted value in the database
// It will look something like this:
// { where: { myField: "e203a4e22cf654w5fd7390300ef2c2f2" } }
// Because "1ac4e952cf6207e5fd79630e0e82c901" != "e203a4e22cf654w5fd7390300ef2c2f2"
// the query results in null
I obviously could use the same IV to encrypt my data which then would lead to every encryption of the same source resulting in the same encrypted string but I would rather not do that if there is any other way to make it work like this.
So basically my two question are:
Is there a way to make this work with the an encryption using a random IV?
Or is there an even better way to store the data encrypted in the database?
Thank you all in advance!
The purpose of the random (salt) part is exactly to prevent what you are trying to do.
I'm not sure about your use case but sometimes it's ok to encrypt without a salt (same data => same hash), sometimes (think of the user password) absolutely not ok.
From what you have posted I don't know where you are saving the random part, otherwise how do you decrypt the data?
Related
I was wondering which approach is better for switching databases...
The idea is to get the database name from a subdomain, and make the specific route SQL query use that databases, until a request comes from another subdomain.
This switch will happen constantly depending on each API request.
changeUser
This can be a middleware before each API route.
pool.getConnection(function(err, conn) {
if (err) {
// handle/report error
return;
}
conn.changeUser({
database: req.session.dbname
}, function (err) {
if (err) {
// handle/report error
return;
}
// Use the updated connection here, eventually
// release it:
conn.release();
});
});
USE DATABASE
Simply prepend each query with the USE statement. This can also be a middleware.
USE specific_db;
select * from table;
If you just need to switch to a different default database, I'd use USE. This preserves your session, so you can continue a transaction, or use temporary tables or session variables. Also your user privileges remain the same.
changeUser() starts a new session, optionally you can change user, or change the default database. But any session-scoped things such as I listed above are ended.
I don't think we can say either is "better" because they are really different actions, each suited to their own purpose. It's like asking whether if() is better than while().
So it depends what is the purpose for the change in your case. You have clarified in the comments that you are doing this in middleware at the time you handle a new request.
It's important to avoid leaking information between requests, because session variables or temp tables might contain private data for the user of the previous request. So it's preferred to reset all session-scoped information. changeUser() will accomplish that, but USE won't.
My analogy is that changeUser() is like logging into Linux potentially as a different user, but USE is like staying in the same Linux shell session, but simply using cd to change directory.
I found similar replies but nothing really straightforward.
How can AES_DECRYPT be used only for the password field in a query using MySQL extension in NodeJS ?
What I have is as follow:
app.post("/verify",function(req,res){
connection.query('SELECT *, FROM `bosses` where u=? and p=?', [req.body.user,req.body.pass], function (error, results, fields) {
if(results.length){
session.loggedin=1;
res.redirect('/zirkus');
}else{
res.redirect('/soccer');
}
});
I assume that I need to modify the query with something like this:
connection.query('SELECT *, FROM `bosses` where u=? and p=AES_DECRYPT (?, 'ENCRYPTIONKEY')', [req.body.user,req.body.pass], function (error, results, fields) {
but somehow I can't get it to work properly. Should I use a placeholder for the encryption key too ?
EDIT
Thanks for the replies and explanation on why this was generally a bad idea :)
Here is a variation: no decryption password is stored in the code:
connection.query('SELECT *, AES_DECRYPT(p, ?) AS `key` FROM bosses WHERE u = ?', [req.body.pass, req.body.user], function (error, results, fields) {
console.log (req.body.pass + req.body.user )
if(results[0].key){
session.loggedin=1;
res.redirect('/zirkus');
}else{
res.redirect('/soccer');
}
});
});
Here the admin user types the decryption password in the form and if the decryption is successful (the key returns true) it allows the user to log in (without using or saving the password) else access is denied.
I assume that in this solution the only downside are the mysql logs right ?
Answer 1: Don't use encryption for storing user passwords. Use hashing.
There's no reason you need to decrypt user passwords, ever. Instead, when the user logs in, you hash their input with the same hashing function and compare the result to the hash string stored in the database.
Try bcrypt: https://www.npmjs.com/package/bcrypt
Also read https://blog.codinghorror.com/youre-probably-storing-passwords-incorrectly/
Answer 2: I never do encryption or hashing in SQL expressions. The reason is that the if you use the query log, it will contain the plaintext of the sensitive content, as it appears in SQL expressions. It will also be visible in the PROCESSLIST.
Instead, if you need to do encryption or hashing of sensitive content, do it in your application code, and then use the result in SQL statements.
Re your edit:
I assume that in this solution the only downside are the mysql logs right ?
No. The problem is that you're storing the password using reversible encryption. There is no reason to reverse a user password. If I visit a website that offers a "password recovery" feature where they can tell me what my password was (no matter how many other security checks they do), then I know they're storing passwords wrong.
If passwords are stored in a reversible encrypted format, this creates the possibility that someone else other than me can reverse the encryption and read my password. That will never happen with hashing, because you can't reverse hashing to get the original content.
If it is because of the logs ... ?
You could disable the query logs, of course. But there's also other places where the query is visible, such as:
the binary log (if you use statement-based binary logs)
the PROCESSLIST
the performance_schema statement tables
the MySQL network protocol. That is, if you don't use TLS to encrypt the connection between the application and the database, someone could intercept packets on the network and see the plaintext query with the plaintext content.
In your edited example, they could view the user's plaintext decryption key in any of the above contexts.
... why MySQL has this function ...?
There are legitimate uses of encryption other than user passwords. Sometimes you do need to decrypt encrypted content. I'm just talking about user passwords. User passwords can be authenticated without decryption, as I described at the top of this answer. It's covered in the blog I linked to, and also as a chapter in my book SQL Antipatterns Volume 1: Avoiding the Pitfalls of Database Programming.
Another use of encryption and corresponding decryption function in SQL is when you develop code as stored procedures. It would be inconvenient to have to return encrypted data to the client application just to decrypt it, and then send it back to your stored procedures for further processing it.
You have to use doubole quotes for the decryption key or escaping ut
connection.query('SELECT *, FROM `bosses` where u=? and p=AES_DECRYPT (?, "ENCRYPTIONKEY)', [req.body.user,req.body.pass], function (error, results, fields) {
if(results.length){
session.loggedin=1;
res.redirect('/zirkus');
}else{
res.redirect('/soccer');
}
});
But as in every language passwords are usually only stored as hashed values, so that they can't be easily reconstructed, even with the logs. so chelkc for example https://coderrocketfuel.com/article/using-bcrypt-to-hash-and-check-passwords-in-node-js
I'm attempting to use Shiro for Authentication and Authorization for a JSF Web Application I'm building. Unfortunately, I'm still having some difficulty wrapping my head around how it all fits together.
I've been successful (100% using the shiro.ini file) configuring authentication back to a JDBC realm where a test set of credentials are stored. It has worked perfectly for me when credentials are stored in plaintext.
My ultimate goal is to unify an existing credential set in a MySQL database. The passwords are stored as SHA-256 salted hashes. I've spent an entire day reading over the documentation (minus Javadocs) that is available, but I'm still having some difficulty understanding exactly how to set it up.
In an attempt to implement in stages, I've modified my shiro.ini as follows with the intention of simply using SHA-256 hashes:
[main]
dataSource = org.apache.shiro.jndi.JndiObjectFactory
dataSource.resourceName = jdbc/Communicator_dev
jdbcRealm = org.apache.shiro.realm.jdbc.JdbcRealm
jdbcRealm.dataSource = $dataSource
dataSource.resourceRef = true;
jdbcRealm.authenticationQuery = select password from account where site_id = ?
jdbcRealm.userRolesQuery = select user_role from web_roles where site_id = ?
# From https://stackoverflow.com/questions/20742666/shiro-with-jdbc-and-hashed-passwords.
#
passwordService = org.apache.shiro.authc.credential.DefaultPasswordService
#configure the passwordService to use the settings you desire
#...
passwordMatcher = org.apache.shiro.authc.credential.PasswordMatcher
passwordMatcher.passwordService = $passwordService
#...
# Finally, set the matcher on a realm that requires password matching for account authentication:
jdbcRealm.credentialsMatcher = $passwordMatcher
The actual login logic is programmatic in a page backing bean. Here's the simple test source I'm currently using:
// Create auth token
UsernamePasswordToken token = new UsernamePasswordToken(this.siteID, this.password);
// Get the current subject
Subject currentUser = SecurityUtils.getSubject();
// Attempt to login
try {
currentUser.login(token);
} catch (AuthenticationException e) {
System.out.println("Invalid creds.");
return "";
}
return "authenticated.xhtml?faces-redirect=true";
This code works perfectly with plaintext passwords stored in my RDBMS, but now that I've hashed them, It's failing.
From my understanding of the framework, I believe the problem lies with the AuthenticationToken. I know that I need to use a different token to ultimately implement the Salted Hashes stored in my RDBMS, but I'm confused on how to proceed.
(1) I don't want to reinvent the wheel. Does Shiro have something that does this natively? I've checked out Les' links to PasswordMatcher and PasswordService (from link shiro with jdbc and hashed passwords) but this still isn't clear. Do I need to sub-classingPasswordMatcher?
(2) An architecture question: Who actually calls the doCredentialsMatch(..) method? Is it the Realm during the execution of the login(...) method?
(3) The AuthenticationInfo parameter of the doCredentialsMap(...) method .. Is that supplied by the Realm? Since Realms encapsulate the actual security data, is this an object created from, in my case, the SQL queries that return the password from the RDBMS?
Thank you very much for your time! I'm hoping to be able to contribute to the documentation when I get my head wrapped around it all.
Item 1:
I suspect you may be running into this issue involving the "salt style" parameter of the JdbcRealm which defaults to "NO_SALT". This causes hashing to work but if you're adding a salt to your password the realm will be unable to properly match them.
Here are your steps afterwards:
The default query for a COLUMN based salt style is as follows, "select password, password_salt from users where username = ?". If you cannot use that structure you need to provide a new query via your 'shiro.ini' with a similar structure.
jdbcRealm.authenticationQuery=select password, salt_column_here from users where username = ?
Here is a related question.
Item 2: Yes, the realm calls the doCredentialsMatch(..) method.
Item 3: Yes, the realm supplies the AuthenticationInfo to the doCredentialsMatch(..) method.
I've got to add like 25000 records to database at once in Rails.
I have to validate them, too.
Here is what i have for now:
# controller create action
def create
emails = params[:emails][:list].split("\r\n")
#created_count = 0
#rejected_count = 0
inserts = []
emails.each do |email|
#email = Email.new(:email => email)
if #email.valid?
#created_count += 1
inserts.push "('#{email}', '#{Date.today}', '#{Date.today}')"
else
#rejected_count += 1
end
end
return if emails.empty?
sql = "INSERT INTO `emails` (`email`, `updated_at`, `created_at`) VALUES #{inserts.join(", ")}"
Email.connection.execute(sql) unless inserts.empty?
redirect_to new_email_path, :notice => "Successfuly created #{#created_count} emails, rejected #{#rejected_count}"
end
It's VERY slow now, no way to add such number of records 'cause of timeout.
Any ideas? I'm using mysql.
Three things come into mind:
You can help yourself with proper tools like:
zdennis/activerecord-import or jsuchal/activerecord-fast-import. The problem is with, your example, that you will also create 25000 objects. If you tell activerecord-import to not use validations, it will not create new objects (activerecord-import/wiki/Benchmarks)
Importing tens thousands of rows into relational database will never be super fast, it should be done asynchronously via background process. And there are also tools for that, like DelayedJob and more: https://www.ruby-toolbox.com/
Move the code that belongs to model out of controller(TM)
And after that, you need to rethink the flow of this part of application. If you're using background processing inside a controller action like create, you can not just simply return HTTP 201, or HTTP 200. What you need to do is to return "quick" HTTP 202 Accepted, and provide a link to another representation where user could check the status of their request (do we already have success response? how many emails failed?), as it is in now beeing processed in the background.
It can sound a bit complicated, and it is, which is a sign, that you maybe shouldn't do it like that. Why do you have to add like 25000 records in one request? What's the backgorund?
Why don't you create a rake task for the work? The following link explains it pretty well.
http://www.ultrasaurus.com/sarahblog/2009/12/creating-a-custom-rake-task/
In a nutshell, once you write your rake task, you can kick off the work by:
rake member:load_emails
If speed is your concern, I'd attack the problem from a different angle.
Create a table that copies the structure of your emails table; let it be emails_copy. Don't copy indexes and constraints.
Import the 25k records into it using your database's fast import tools. Consult your DB docs or see e.g. this answer for MySQL. You will have to prepare the input file, but it's way faster to do — I suppose you already have the data in some text or tabular form.
Create indexes and constraints for emails_copy to mimic emails table. Constraint violations, if any, will surface; fix them.
Validate the data inside the table. It may take a few raw SQL statements to check for severe errors. You don't have to validate emails for anything but very simple format anyway. Maybe all your validation could be done against the text you'll use for import.
insert into emails select * from emails_copy to put the emails into the production table. Well, you might play a bit with it to get autoincrement IDs right.
Once you're positive that the process succeeded, drop table emails_copy.
Does mysql provide a mechanism for storing and retrieving encrypted data? I don't mean passwords, I mean real strings.
I'd like to encrypt a string, store in mysql and then retrieve the decrypted string at a later date.
So, I know there is the AES_Encrypt and decrypt functions. But they ask for a key. (which is fine) but I wondering if you call those functions and use your user password as the key. Or something else that is super-simple.
Also, is there a simple wrapper for the AES_Encrypt & decrypt functions in Rails? Or do you need to build the query manually?
You can just concat the encrypt functions:
select aes_encrypt('MyData',Password('MyPassword'))
and back again..
select Aes_decrypt( aes_encrypt('MyData',Password('MyPassword'))
, Password('MyPassword'))
If I understand you, then all you need is a method to generate an AES key from your (or other) user password?
Shouldn't you be asking 'Is there an easy method to generate an AES-key from 5-20char string'?
As you point out, the other tools are already in place in mysql: http://dev.mysql.com/doc/refman/5.0/en/encryption-functions.html
Also you might find some ideas in this post here on SO.
$pass = $_POST['pass'];
$sql = "INSERT INTO testperson (name,password,contact) VALUES('$name',md5('$pass'),$cont)";
Just write md5 before the input you want to encrypt, like password.