golang - elegant way to omit a json property from being serialized - json

I've got a user struct, which has sensitive fields like password:
type User struct {
UID string `json:"uid" binding:"required"`
Password string `json:"password" binding:"required"`
EmailAddress string `json:"email" binding:"required"`
}
Now I want to be able to use this struct to register a user and update, delete but also to view. What I don't want is for the password to be serialized for viewing. I can, of course, make a custom marshaller but is that the only way? I tried using the json:"-" option but that causes it to be ignored while unmarshalling as well, which I don't want. Is there a better way?
EDIT:
To put some of you guys at ease, I'm NOT going to be storing the password in plaintext, of course. It's the bcrypt hash of the password, but still. I don't want it to be returned when I search for users.

I'd say implementing json.Marshaler is the elegant solution if you want custom marshaling. It's quite simple in this case:
func (u User) MarshalJSON() ([]byte, error) {
type user User // prevent recursion
x := user(u)
x.Password = ""
return json.Marshal(x)
}
Add "omitempty" in your User type if you don't want the password field at all when marshaling.

I would go with another struct and composition.
Password should never be stored in plain-text, they should be securely hashed (bcrypt, pbkdf2, etc.). That hash is the one that has to be stored and should never be serialized. By using composition, you can do something like this:
type User struct {
UID string `json:"uid" binding:"required"`
HashedPassword string `json:"-"`
EmailAddress string `json:"email" binding:"required"`
}
type UserFormData struct {
User
Password string `json:"password" binding:"required"`
}
This also gives you more flexibility. For instance, if you ask the user to confirm the password, you can simply change the UserFormData struct like this:
type UserFormData struct {
User
Password string `json:"password" binding:"required"`
ConfirmPassword string `json:"confirm_password" binding:"required"`
}
Which also has the advantage to keep that serialization details outside your User object.

A simple solution would be to sanitize the user struct before marshaling it:
type User struct {
UID string `json:"uid" binding:"required"`
Password string `json:"password,omitempty" binding:"required"`
EmailAddress string `json:"email" binding:"required"`
}
func sanitizeUser(u User) User {
return User{u.UID, "", u.EmailAddress}
}
Demo: https://play.golang.org/p/RjKVoFc9o8

Now I want to be able to use this struct to register a user and update, delete but also to view.
Another solution is not to store the password at all in the struct. You don't need it to view, or delete, or update (normally).
You need it to create the user record, at which point you'll store a hash in your data store.
You need it to verify their identity (on login), at which point you verify against the hash in your data store, then usually issue a token they can use to keep accessing the service.
So there are only a few points you need it, and at those points you can simply keep it in memory separately and verify identity, it doesn't need to be exposed or stored in the struct for most operations. This is IMO more elegant than having it in the struct where it can very easily be exposed by mistake in export or logging.

You can query the database with a select statement to get only fields you like.
db.Model(&User{}).Select([]string{"uid","email"}).Find(&users)
Also add omitempty in your User type since empty response is sent for fields not included in select.

Related

Use Validator/Dive to enter an array and validate nested structs in go

type myType struct {
value int `json:"value"`
Name string `json:"name" validate:"required"`
URL string `json:"URL" validate:"required"`
args []otherType `json:"args" validate:"dive", "required"`
}
type otherType struct {
name string `validate:"required"`
origin string `validate:"required"`
}
err := paramsValidator.Validate(someInstantiationOfThisStruct)
Hello there! I'm a tad bit stumped on using validator's dive feature. This specific combination of validation scheme isn't present in the documentation for the validator, and I was unable to get it working with a little bit of tweaking.
I would like to simply enter the args array in the primary struct, and validate each of two sets of otherType. However I don't quite understand how this is supposed to transpire.
I understand dive incorrectly and it's not working of course, as the validator is unable to determine incorrect validations using Validate().
Is there any particular thing I'm doing wrong? In general how should I approach evaluating and validating args that are in an array?
I was able to figure it out. I am so sorry for even posting! I was stumped for thirty minutes, but the solution was not that particularly bad.
type myType struct {
value int `json:"value"`
Name string `json:"name" validate:"required"`
URL string `json:"URL" validate:"required"`
args []otherType `json:"args" validate:"dive", "required"`
}
type otherType struct {
name string `validate:"required"`
origin string `validate:"required"`
}
is the updated code. There was a missing , between "dive" and "required", and I had posted code that read
`validate: "dive, required"
dyslexia sorry! :(
I came searching for the answer here but the solution didn't work for me.
In order to validate nested struct using go-playground/validator add dive.
So add below code to the nested struct at top level
`validate:"required,dive,required"`
Note: add without spaces, also make sure the fields are exposed (use PascalCase) to package incase u importing the struct
type myType struct {
value int `json:"value"`
Name string `json:"name" validate:"required"`
URL string `json:"URL" validate:"required"`
Args []OtherType `json:"args" validate:"required,dive,required"`
}
type OtherType struct {
Name string `validate:"required"`
Origin string `validate:"required"`
}
Note: This validation is as per my use case where i want Args to be required and also want it to be exposed to other packages. Just trying to help other who come searching for the same issue as "Dive" is not documented properly in go/playground documentation

How do I get the values of the properties of a model object in ASP.NET MVC when passed in as an object?

I would like to be able to write to a log the properties of various objects that I might pass to my LogEvent function. I have various objects defined, such as Town, Taxpayer, TaxedProperty, TaxAccount, etc. in my ASP.NET MVC application. I would like to write to some log passing in my object along with a bunch of other information.
I am writing this to a database table and I have a large text field in which I can insert a representation of the object that was being processed when a bad thing happened. Perhaps the user was trying to edit a taxpayer they did not have access to. So, I would write to the log that this user tried to access this taxpayer and I would do this:
kurantLogic.LogEvent(userName, null, null, Request.HttpMethod, "Taxpayers/Edit: Access denied because user does not have access to the taxpayer with ID=" + taxpayer.ID, false, taxpayer);
the last argument is the taxpayer object, but it could be some other object, such as a town. Here is another example:
kurantLogic.LogEvent(userName, null, null, Request.HttpMethod, "Towns/SetUpYourTown: Access denied because user does not have access to the town with ID=" + town.ID, false, town);
or here is an example of logging a good thing:
kurantLogic.LogEvent(userName, null, null, Request.HttpMethod, "Towns/SetUpYourTown: Success, Town edit saved = " + town.Name, false, town);
I would like to list out the fields in the town or taxpayer or whatever object is passed in without having to hard code all the properties of the thing I might pass in .
Here is my code that I started with:
public void LogEvent(string userName, int? userID, int? townID, string httpVerb, string description, bool isError, Object obj)
{
JsonSerializerSettings settings = new JsonSerializerSettings();
settings.MaxDepth = 1;
settings.ReferenceLoopHandling = ReferenceLoopHandling.Ignore;
settings.NullValueHandling = NullValueHandling.Ignore;
string bigText = JsonConvert.SerializeObject(obj, settings);
LogEvent(userName, userID, townID, httpVerb, description, isError, bigText);
}
That kind of worked except that the JSON converter wrote out all the collections of things that the object contained, and I cannot find a way to suppress that. So, if there is a way to make JSON.SerializeObject include only the properties that are strings, ints, decimals, booleans, and not other objects, that would be a good solution. But I thought I could use Reflection to do it, but am having trouble with that.
My Taxpayer class has things like TaxedProperties that are associated with it, and I don't those logged. This is what I am getting now. At the end of this long JSON serialization you can see the simple properties listed.
Well, I could not post the JSON because StackOverflow thinks it looks like spam! I hope you can help.
I just want the "scalar" properties of the object logged without hard-coding. I would think that should be possible with either JSON or Reflection, but I am stumped.
You can tell Json.Net to ignore certain properties with the [JsonIgnore] attribute.

One to One relation mapping using GORM for Golang

I am trying to understand how GORM works for one to one relational mapping with MySQL. I have 2 structs like so:
type User struct {
Id uint `gorm:"AUTO_INCREMENT"`
FirstName string `gorm:"column:first_name"`
LastName string `gorm:"column:last_name"`
EncryptedUserId string `gorm:"size:255"`
Email string `gorm:"not null;unique"`
Password string `gorm:"not null;unique"`
CreatedAt int64 `gorm:"type(timestamp)"`
}
type UserSession struct {
Id uint `gorm:"AUTO_INCREMENT"`
UserId User
SessionToken string `gorm:"column:session_token"`
CreatedAt int64 `gorm:"type(timestamp)"`
}
User and UserSession share one to one relation. But when code above is run, the column UserId for UserSession table is not created. Even after specifying the foreign key constraint gorm:"ForeignKey:Id"
the result is same. Why isn't the above code working? Is anything missing in the struct definition?
I can't comment your questions so I would ask it here:
Do you migrate your schema in any way like:
db.AutoMigrate(&User{}, &UserSession{})
?
If you do, you should get some detailed errors in log, which might be useful for you.
The way I have managed to get something similar to work is this.
type UserSession struct {
Id uint `gorm:"AUTO_INCREMENT"`
UserId uint
User User
SessionToken string `gorm:"column:session_token"`
CreatedAt int64 `gorm:"type(timestamp)"`
}

Golang Gorm error migrating struct

I have a small mistake in my small application and I really do not see where it can come from. So I have 4 structs, one of the 4 struct has several one-to-one relationships.
I connect to my database and use automigrate to migrate my 4 structs and create the necessary tables.
The problem is at this point, it does not create anything in the database and in the terminal I have this message:
(Error 1060: Name of the 'id' field already in use)
My code
main.go
package main
import (
"fmt"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/mysql"
)
var db *gorm.DB
var err error
const (
mysupersecretpassword = "cr9ih_pvr9f9kc75n#bz&y%(#+^&1_#hr0^)-$kv%n3dh84$^w"
)
func main() {
db, err = gorm.Open("mysql", "root:root#/test?charset=utf8&parseTime=True")
if err != nil {
fmt.Println(err)
}
defer db.Close()
db.AutoMigrate(&User{}, &Ads{}, &Type{}, &Category{}, &Location{})
}
models.go
package main
import (
"github.com/jinzhu/gorm"
)
type User struct {
gorm.Model
Username string `json:"username"`
Email string `json:"email" form:"email"`
Password string `json:"password" form:"password"`
active bool `json:"active" gorm:"default:0"`
level bool `json:"level" gorm:"default:0"`
}
type Type struct {
gorm.Model
Name string `json:"name" form:"name"`
}
type Category struct {
gorm.Model
CatID uint `json:"category-parent" form:"category-parent" gorm:"default:0"`
Name string `json:"name" form:"name"`
}
type Location struct {
gorm.Model
Location string `json:"location" form:"location"`
}
type Ads struct {
gorm.Model
User User `json:"user"`
Type Type `json:"type" form:"type"`
Category Category `json:"category" form:"category"`
Title string `json:"title" form:"title"`
Content string `json:"content" form:"content"`
Location Location `json:"location" form:"location"`
}
Waiting for an answer that could put me on the right path :)
AutoMigrate will ONLY create tables, missing columns and missing indexes, and WON'T change existing column's type or delete unused columns to protect your data.
I would guess that one of your tables already exists, and the id column in that table is a different type than what gorm.Model wants to create. I would figure out which table it is by doing:
db.AutoMigrate(&User{})
db.AutoMigrate(&Ads{})
db.AutoMigrate(&Type{})
db.AutoMigrate(&Category{})
db.AutoMigrate(&Location{})
and seeing where it fails. Then, I would backup that table (just in case), and then either just drop the table completely, or rename the id column to tmp_id, see if automigrate fixes it, and if so, drop the tmp_id column.
FYI I've found Gorm will not log proper error or any error if it has permission issues on the user. For example the table existed when logged in as root, but the user I was logging in as didn't see it at all, Gorm just ran through Migration without creating the table or altering it (even though it was different schema) just didn't report any permission issue at all.

Hibernate Encryption of Database Completely Transparent to Application

I'm working on a Grails 1.0.4 project that has to be released in less than 2 weeks, and the customer just came up with a requirement that all data in the database should be encrypted.
Since encryption of every database access in the application itself could take a lot of time and will be error prone, the solution I seek is some kind of encryption transparent to the application.
Is there a way to setup Hibernate to encrypt all data in all tables (except maybie the id and version columns) or should I seek a MySQL solution (we're using MySQL 5.0) ?
EDIT:
Thanks for all of your posts for alternative solutions, if the customer changes mind it would be great. As for now, the requirement is "No plain text in the Database".
Second thing I'd like to point out is that I'm using Grails, for those not fammiliar with it, It's a convention over configuration, so even small changes to the application that are not by convention should be avoided.
If you end doing the work in the application, you can use Hibernate custom types and it wouldn't add that many changes to your code.
Here's an encrypted string custom type that I've used:
import org.hibernate.usertype.UserType
import org.apache.log4j.Logger
import java.sql.PreparedStatement
import java.sql.ResultSet
import java.sql.SQLException
import java.sql.Types
class EncryptedString implements UserType {
// prefix category name with 'org.hibernate.type' to make logging of all types easier
private final Logger _log = Logger.getLogger('org.hibernate.type.com.yourcompany.EncryptedString')
Object nullSafeGet(ResultSet rs, String[] names, Object owner) throws SQLException {
String value = rs.getString(names[0])
if (!value) {
_log.trace "returning null as column: $names[0]"
return null
}
_log.trace "returning '$value' as column: $names[0]"
return CryptoUtils.decrypt(value)
}
void nullSafeSet(PreparedStatement st, Object value, int index) throws SQLException {
if (value) {
String encrypted = CryptoUtils.encrypt(value.toString())
_log.trace "binding '$encrypted' to parameter: $index"
st.setString index, encrypted
}
else {
_log.trace "binding null to parameter: $index"
st.setNull(index, Types.VARCHAR)
}
}
Class<String> returnedClass() { String }
int[] sqlTypes() { [Types.VARCHAR] as int[] }
Object assemble(Serializable cached, Object owner) { cached.toString() }
Object deepCopy(Object value) { value.toString() }
Serializable disassemble(Object value) { value.toString() }
boolean equals(Object x, Object y) { x == y }
int hashCode(Object x) { x.hashCode() }
boolean isMutable() { true }
Object replace(Object original, Object target, Object owner) { original }
}
and based on this it should be simple to create similar classes for int, long, etc. To use it, add the type to the mapping closure:
class MyDomainClass {
String name
String otherField
static mapping = {
name type: EncryptedString
otherField type: EncryptedString
}
}
I omitted the CryptoUtils.encrypt() and CryptoUtils.decrypt() methods since that's not Grails-specific. We're using AES, e.g. "Cipher cipher = Cipher.getInstance('AES/CBC/PKCS5Padding')". Whatever you end up using, make sure it's a 2-way crypto, i.e. don't use SHA-256.
If the customer is worried about someone physically walking away with the hard drive then using a full disk solution like Truecrypt should work. If there worried about traffic being sniffed then take a look at this part of the mysql documentation on ssl over JDBC. Remember if someone compromises your server all bets are off.
the customer could easily do this without changing a thing in your application.
first, encrypt the communications between the server by turning on SSL in the mysql layer, or use an SSH tunnel.
second, store the mysql database on an encrypted volume.
any attack that can expose the filesystem of the mysql database or the credentials needed to log in to the mysql server is not mitigated by encrypting the data since that same attack can be used to retrieve the encryption key from the application itself.
Well it has been a long time since I've asked the question. In the meantime, thanks for all the answers. They were great when dealing with the original idea of encrypting the entire database, but the requirement changed to only encrypting sensitive user info, like name and address. So the solution was something like the code down below.
We've implemented an Encrypter which reads the encryption method from the record ( so there can be different encryption per record) and use it to connect transient duplicate fields to the ones encrypted in the database. The added bonus/drawbacks are:
The data is also encrypted in memory, so every access to the method getFirstName descrypts the data (I guess there is a way to cache decrypted data, but I dont need it in this case)
Encrypted fields cannot be used with default grails/hibernate methods for search through database, we've made custom methods in services that get data, encrypt it and then use the encrypted data in the where clause of a query. It's easy when using User.withCriteria
class User {
byte[] encryptedFirstName
byte[] encryptedLastName
byte[] encryptedAddress
Date dateCreated // automatically set date/time when created
Date lastUpdated // automatically set date/time when last updated
EncryptionMethod encryptionMethod = ConfigurationHolder.config.encryption.method
def encrypter = Util.encrypter
static transients = [
'firstName',
'lastName',
'address',
'encrypter'
]
static final Integer BLOB_SIZE = 1024
static constraints = {
encryptedFirstName maxSize: BLOB_SIZE, nullable: false
encryptedLastName maxSize: BLOB_SIZE, nullable: false
encryptedAddress maxSize: BLOB_SIZE, nullable: true
encryptionMethod nullable: false
} // constraints
String getFirstName(){
decrypt('encryptedFirstName')
}
void setFirstName(String item){
encrypt('encryptedFirstName',item)
}
String getLastName(){
decrypt('encryptedLastName')
}
void setLastName(String item){
encrypt('encryptedLastName',item)
}
String getAddress(){
decrypt('encryptedAddress')
}
void setAddress(String item){
encrypt('encryptedAddress',item)
}
byte[] encrypt(String name, String value) {
if( null == value ) {
log.debug "null string to encrypt for '$name', returning null"
this.#"$name" = null
return
}
def bytes = value.getBytes(encrypter.ENCODING_CHARSET)
def method = getEncryptionMethod()
byte[] res
try {
res = encrypter.encrypt( bytes, method )
} catch(e) {
log.warn "Problem encrypting '$name' data: '$string'", e
}
log.trace "Encrypting '$name' with '$method' -> '${res?.size()}' bytes"
this.#"$name" = res
}
String decrypt(String name) {
if(null == this.#"$name") {
log.debug "null bytes to decrypt for '$name', returning null"
return null
}
def res
def method = getEncryptionMethod()
try {
res = new String(encrypter.decrypt(this.#"$name", method), encrypter.ENCODING_CHARSET )
} catch(e) {
log.error "Problem decrypting '$name'", e
}
log.trace "Decrypting '$name' with '$method' -> '${res?.size()}' bytes"
return res
}
}
Another option is to use a JDBC driver that encrypts/decrypts data on the fly, two way. Bear in mind that any solution will probably invalidate searches by encrypted fields.
IMHO the best solution is the one proposed by longneck, it will make everything much easier, from administration to development. Besides, bear in mind that any solution with client-side encryption will render all your db data unusable outside of the client, ie, you will not be able to use nice tools like a jdbc client or MySQL query browser, etc.
Jasypt integrates with Hibernate: http://jasypt.org/hibernate3.html. However, queries which use WHERE clauses cannot be used
Generated ids, version, mapped foreign keys - basically everything maintained by Hibernate - are out unless you intend to declare custom CRUD for all of your classes and manually encrypt them in queries.
For everything else you've got a couple of choices:
#PostLoad and #PrePersist entity listeners will take care of all non-query operations.
Implementing custom String / Long / Integer / etc... types to handle encryption will take care of both query and CRUD operations; however the mapping will become rather messy.
You can write a thin wrapper around a JDBC driver (as well as Connection / Statement / PreparedStatement / ResultSet / etc...) to do the encryption for you.
As far as queries go you'll have to handle encryption manually (unless you're going with #2 above) but you should be able to do so via a single entry point. I'm not sure how (or if) Grails deals with this, but using Spring, for example, it would be as easy as extending HibernateTemplate.