I am having some trouble with the UnmarshalJSON and MarshalJSON i am not sure i understand them correctly or atleast i have an error that i have not been able to spot and in my 3 hours of googling i haven't been able find, so here goes.
I have the following User struct
type User struct {
ID uuid.UUID `gorm:"primaryKey,type:string,size:36,<-:create" json:"id"`
Username string `gorm:"unique" json:"username"`
Password PasswordHash `gorm:"type:string" json:"password"`
CreatedAt time.Time `gorm:"autoCreateTime:milli" json:"created_at"`
UpdatedAt time.Time `gorm:"autoUpdateTime:milli" json:"updated_at,omitempty"`
DeletedAt gorm.DeletedAt `gorm:"index" json:"deleted_at,omitempty"`
}
which when i try to convert into JSON, with the json.Marshal function from the build in encoding/json library, i get the following:
{"id":"3843298e-74d4-4dd7-8eff-007ab34a4c19","username":"root","password":{},"created_at":"0001-01-01T00:00:00Z","updated_at":"0001-01-01T00:00:00Z","deleted_at":null}
I expected some thing like:
{"id":"3843298e-74d4-4dd7-8eff-007ab34a4c19","username":"root","password":"$argon2id$v=19$m=4194304,t=1,p=64$Z9EFSTk26TQxx+Qv9g58gQ$4At0rvvv9trRcFZmSMXY0nISBuEt+1X8mCRAYbyXqSs","created_at":"0001-01-01T00:00:00Z","updated_at":"0001-01-01T00:00:00Z","deleted_at":null}
i can't seem to get the password field to be a string of the hashed password.
even when i have the following methods on the PasswordHash struct
type Params struct {
memory uint32
iterations uint32
parallelism uint8
saltLength uint32
keyLength uint32
}
type PasswordHash struct {
hash []byte
salt []byte
params Params
}
func (h *PasswordHash) String() string {
b64Salt := base64.RawStdEncoding.EncodeToString(h.salt)
b64Hash := base64.RawStdEncoding.EncodeToString(h.hash)
return fmt.Sprintf(
"$%s$v=%d$m=%d,t=%d,p=%d$%s$%s",
algoName,
argon2.Version,
h.params.memory,
h.params.iterations,
h.params.parallelism,
b64Salt,
b64Hash,
)
}
func (h *PasswordHash) UnmarshalJSON(data []byte) error {
var v string
if err := json.Unmarshal(data, &v); err != nil {
return err
}
params, salt, hash, err := decodeHash(v)
if err != nil {
return err
}
h.params = params
h.salt = salt
h.hash = hash
return nil
}
func (h *PasswordHash) MarshalJSON() ([]byte, error) {
//return []byte(h.String()), nil
return json.Marshal(h.String())
}
So i guess my question is, shouldn't the MarshalJSON be called on the PasswordHash struct when trying to convert the user to JSON? and if so how come i can't seem to get it to be a string value?
You have MarshalJSON defined with a receiver of *PasswordHash, but your value is type PasswordHash. Change the receiver to PasswordHash and it works as expected: https://go.dev/play/p/WukE_5JBEPL
Related
Sorry If i'm posting a question that has already been answered, but I can't seem to find any similar situations on here. I have a websocket client that receives dynamic json data with overlapping fields. The fact that the fields overlap has has made Unmarshalling very difficult for me.
I have structs for the data types I receive, but I need a way to check the json data before I unmarshal it to a specific struct. I was hoping that an interface could act as a temporary holder and I would then be able to match the interface to the specific struct I want to unmarshal to, but that doesn't seem possible, or I just don't know how to go about it. Here are a few examples of the data types I'm receiving and structs to go along with it in case that helps.
response 1: {"connectionID":17973829270596587247,"event":"systemStatus","status":"online","version":"1.9.0"}
response 2: {"channelID":328,"channelName":"ohlc-5","event":"subscriptionStatus","pair":"XBT/USD","status":"subscribed","subscription":{"interval":5,"name":"ohlc"}}
response 3: [328,["1649576721.042916","1649577000.000000","42641.50000","42641.50000","42641.50000","42641.50000","42641.50000","0.00335101",2],"ohlc-5","XBT/USD"]
response 4: {"event":"heartbeat"}
structs below
import (
"time"
"encoding/json"
)
type ConnStatus struct {
ConnectionID uint64 `json:"connectionID"`
Event string `json:"event"`
Status string `json:"status"`
Version string `json:"version"`
}
type HeartBeat struct {
Event string `json:"event"`
}
type OHLCsuccess struct {
ChannelID int `json:"channelID"`
ChannelName string `json:"channelName"`
Event string `json:"event"`
Pair string `json:"pair"`
Status string `json:"status"`
Subscription OHLC `json:"subscription"`
}
type OHLC struct {
Interval int `json:"interval"`
Name string `json:"name"`
}
type OHLCUpdates struct {
ChannelID int
OHLCArray OHLCNewTrade
ChannelName string
Pair string
}
type OHLCNewTrade struct {
StartTime UnixTime
EndTime UnixTime
Open float64
High float64
Low float64
Close float64
VWAP float64
Volume float64
Count int
}
type UnixTime struct {
time.Time
}
func (u *UnixTime) UnmarshalJSON(d []byte) error {
var ts int64
err := json.Unmarshal(d, &ts)
if err != nil {
return err
}
u.Time = time.Unix(ts, 0).UTC()
return nil
}
Any idea(s) on how to go about this? Thanks in advance for the help!
Are you in control of the different responses? If so, wow about adding a "type" field to the top level?
See "How to put everything at the top level" section on https://eagain.net/articles/go-dynamic-json/ for more info.
E.g. (untested):
func UnmarshalJSON(d []byte) error {
var jsonValue map[string]interface{}
err := json.Unmarshal(d, &jsonValue)
if err != nil {
return err
}
switch jsonValue["type"] {
case 1:
// unmarshal into struct type 1
case 2:
// unmarshal into struct type 2
default:
// throw err
}
// or if you don't have access to type:
if jsonValue["connectionID"] != nil {
// unmarshal into struct type 1
}
return nil
}
Alternatively you could try to (strictly) unmarshal into each struct, until you don't get an error, e.g. something like:
func DetermineStruct(d []byte) int {
var connStatus *ConnStatus
reader := bytes.NewReader(d)
decoder := json.NewDecoder(reader)
decoder.DisallowUnknownFields()
err := decoder.Decode(connStatus)
if err == nil {
panic(err)
}
err = json.Unmarshal(d, &connStatus)
if err == nil {
return 1
}
var ohlcSuccess OHLCsuccess
err = json.Unmarshal(d, &ohlcSuccess)
if err == nil {
return 2
}
}
Lets say I have two struct that are related like this:
type SecretUser struct {
UserInfo `json:"userInfo"`
Password string `json:"password"`
}
type UserInfo struct {
FirstName string `json:"firstName"`
LastName string `json:"lastName"`
Email string `json:"email"`
}
And I receive a JSON in this form:
{
"firstName": "nice",
"lastName":"guy",
"email":"nice#guy.co.uk",
"password":"abc123"
}
I want to unmarshall this JSON into a SecretUser. Is there a better way than doing it like this?
func (u *User) UnmarshalJSON(data []byte) error {
var objmap map[string]*json.RawMessage
var password string
var err error
err = json.Unmarshal(data, &objmap)
if err != nil {
return err
}
if err := json.Unmarshal(data, &u.UserInfo); err != nil {
return err
}
err = json.Unmarshal(*objmap["password"], &password)
if err != nil {
return err
}
u.Password = password
return nil
}
Basically, I partially unmarshall the JSON into a UserInfo struct and then read it again to extract the password. I don't want to create another struct for just unmarshalling this JSON cleanly or use an external library (unless it's part of the standard). Is there a more clean/efficient way of doing this, without reading the JSON twice or setting every field manually from a map?
Simply include the UserData into the SecretUser struct and do not specify a json tag for it.
type UserInfo struct {
FirstName string `json:"firstName"`
LastName string `json:"lastName"`
Email string `json:"email"`
}
type SecretUser struct {
UserInfo
Password string `json:"password"`
}
func main() {
data := []byte(`{"firstName": "nice","lastName":"guy","email":"nice#guy.co.uk","password":"abc123"}`)
var u SecretUser
json.Unmarshal(data, &u)
fmt.Println(u)
}
Go Play Space example
I have a question in Go especially with gin-gionic and gorm.
Let's say I have model like this
// Classroom struct.
type Classroom struct {
gorm.Model
Name string `json:"name"`
Code string `json:"code"`
StartedAt time.Time `json:"started_at"`
}
I want to create data of Classroom Model with this JSON
{
"name": "Math",
"code": "math-mr-robie",
"started_at": "2020-10-10 10:00:00"
}
But when I bind the JSON data, I got this following error
parsing time ""2020-10-10 10:00:00"" as ""2006-01-02T15:04:05Z07:00"": cannot parse " 10:00:00"" as "T"
I know that error appear because of the format that I sent was not the exact format of time.Time?
Is it possible to set default format of time.Time?
How to do that?
Because I've try to add .Format in after time.Time but error occurs.
// Classroom struct.
type Classroom struct {
gorm.Model
Name string `json:"name"`
Code string `json:"code"`
StartedAt time.Time.Format("2006-01-02 15:04:05") `json:"started_at"`
}
I resolve this issue by creating new struct JSONData that contain time inside it.
// JSONData struct.
type JSONData struct {
Time time.Time
}
After I red Customize Data Types in Gorm and see some examples here then I add some methods
// Scan JSONDate.
func (j *JSONDate) Scan(value interface{}) (err error) {
nullTime := &sql.NullTime{}
err = nullTime.Scan(value)
*j = JSONDate{nullTime.Time}
return
}
// Value JSONDate.
func (j JSONDate) Value() (driver.Value, error) {
y, m, d := time.Time(j.Time).Date()
return time.Date(y, m, d, 0, 0, 0, 0, time.Time(j.Time).Location()), nil
}
// GormDataType gorm common data type
func (j JSONDate) GormDataType() string {
return "timestamp"
}
For the gin things. Another resource #Eklavya given. So I add another methods.
// UnmarshalJSON JSONDate.
func (j *JSONDate) UnmarshalJSON(b []byte) error {
s := strings.Trim(string(b), "\"")
t, err := time.Parse(helpers.YMDHIS, s)
if err != nil {
return err
}
*j = JSONDate{
Time: t,
}
return nil
}
// MarshalJSON JSONDate.
func (j JSONDate) MarshalJSON() ([]byte, error) {
return []byte("\"" + j.Time.Format(helpers.YMDHIS) + "\""), nil
}
// Format method.
func (j JSONDate) Format(s string) string {
t := time.Time(j.Time)
return t.Format(helpers.YMDHIS)
}
And it's works!
I came across this question with the same issue and found that if you are looking for a specific time format like the ISOStrings sent from the browser you can have something like this;
type Classroom struct {
gorm.Model
Name string `json:"name"`
Code string `json:"code"`
StartedAt time.Time `json:"started_at" time_format:"RFC3339"`
}
With the time_format I did not need to define marshal functions to handle the formatting. However, if you need to do a completely custom format for your date and time then I believe you will need to define these functions.
When I make an HTTP call to a REST API I may get the JSON value count back as a Number or String. I'ld like to marshal it to be an integer in either case. How can I deal with this in Go?.
Use the "string" field tag option to specify that strings should be converted to numbers. The documentation for the option is:
The "string" option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types. This extra level of encoding is sometimes used when communicating with JavaScript programs:
Here's an example use:
type S struct {
Count int `json:"count,string"`
}
playground example
If the JSON value can be number or string, then unmarshal to interface{} and convert to int after unmarshaling:
Count interface{} `json:"count,string"`
Use this function to convert the interface{} value to an int:
func getInt(v interface{}) (int, error) {
switch v := v.(type) {
case float64:
return int(v), nil
case string:
c, err := strconv.Atoi(v)
if err != nil {
return 0, err
}
return c, nil
default:
return 0, fmt.Errorf("conversion to int from %T not supported", v)
}
}
// Format of your expected request
type request struct {
ACTIVE string `json:"active"`
CATEGORY string `json:"category"`
}
// struct to read JSON input
var myReq request
// Decode the received JSON request to struct
decoder := json.NewDecoder(r.Body)
err := decoder.Decode(&myReq)
if err != nil {
log.Println( err)
// Handler for invalid JSON received or if you want to decode the request using another struct with int.
return
}
defer r.Body.Close()
// Convert string to int
numActive, err = strconv.Atoi(myReq.ACTIVE)
if err != nil {
log.Println(err)
// Handler for invalid int received
return
}
// Convert string to int
numCategory, err = strconv.Atoi(myReq.CATEGORY)
if err != nil {
log.Println(err)
// Handler for invalid int received
return
}
I had the same problem with a list of values where the values were string or struct. The solution I'm using is to create a helper struct with fields of expected types and parse value into the correct field.
type Flag struct {
ID string `json:"id"`
Type string `json:"type"`
}
type FlagOrString struct {
Flag *Flag
String *string
}
func (f *FlagOrString) UnmarshalJSON(b []byte) error {
start := []byte("\"")
for idx := range start {
if b[idx] != start[idx] {
return json.Unmarshal(b, &f.Flag)
}
}
return json.Unmarshal(b, &f.String)
}
var MainStruct struct {
Vals []FlagOrString
}
Custom Unmarshaller simplifies a code. Personally I prefer this over interface{} as it explicitly states what a developer expects.
I'm writing some sort of RESTfull API based Object relational mapper in go.
I plan to make it MIT licensed, when i finish it.
The idea is to use some 3rd party REST API as data storage, and the golang client will query it for data needed.
The API responses are JSONs with known structure.
this is my code:
type AClient struct {
Id string `json:"id"`
Uid string `json:"uid"`
FirstName string `json:"firstName"`
LastName string `json:"lastName"`
CreatedAt string `json:"createdAt"`
UpdatedAt string `json:"updatedAt"`
City string `json:"city"`
Address string `json:"address"`
Telefone string `json:"telefone"`
Zip string `json:"zip"`
Telefon string `json:"telefon"`
Comment string `json:"comment"`
}
type AEvents struct {
Id string `json:"id"`
Security bool `json:"security"`
Description string `json:"description"`
Good AGood `json:"good"`
Client AClient `json:"client"`
Author AAuthor `json:"author"`
InFuture bool `json:"inFuture"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
}
type Entry struct {
AEvents //this have to be changed to `AClients` in runtime when needed
}
type ORM struct {
ApiUrl string
ModelName string
ModelInterface Entry
HuntKey string
HuntSid string
Csrf string
}
func (o *ORM) Query(parameters map[string]string) ([]Entry, AMetadata, error) {
responseParsed := struct {
Status string `json:"status"`
Metadata AMetadata `json:"metadata"`
Data []Entry `json:"data"` //todo - use o.ModelInterface
}{}
client := &http.Client{}
var queryString string
for k, v := range parameters {
queryString = queryString + fmt.Sprintf("%v=%v&", url.QueryEscape(k), url.QueryEscape(v))
}
req, err := http.NewRequest("GET", fmt.Sprintf("%v%v?%v", o.ApiUrl, o.ModelName, queryString), nil)
fmt.Println("---------------------------------------------")
fmt.Println(fmt.Sprintf("[GET] %v%v?%v", o.ApiUrl, o.ModelName, queryString))
req.Header.Set("huntKey", o.HuntKey)
if err != nil {
return nil, AMetadata{}, err
}
res, err1 := client.Do(req)
defer res.Body.Close()
if err1 != nil {
return nil, AMetadata{}, err1
}
if res.StatusCode == 200 {
for _, v := range res.Cookies() {
if v.Name == "XSRF-TOKEN" {
o.Csrf = v.Value
}
if v.Name == "hunt.sid" {
o.HuntSid = v.Value
}
}
fmt.Printf("CSRF %v\n", o.Csrf)
fmt.Printf("HuntSid %v\n", o.HuntSid)
fmt.Println("---------------------------------------------")
raw, err2 := ioutil.ReadAll(res.Body)
if err2 != nil {
return nil, AMetadata{}, err2
} else {
err2 = json.Unmarshal(raw, &responseParsed)
return responseParsed.Data, responseParsed.Metadata, nil
}
} else {
return nil, AMetadata{}, errors.New("Unable to fetch data!")
}
}
How can I make this:
When instantiating the ORM object, how can i pass the struct name, that will be used to parse the JSON response. The current code works with the struct of AEvents, but i want it to be easy changeble t AClient and so on.
UPD:
i have reviewed the code of https://github.com/jinzhu/gorm and find out tons of things how can i implement it.
Than, as i have promised, I publish this code as open source -https://github.com/vodolaz095/hrorm
You can use reflection, but be warned that it is non-trivial and relatively slow. I don't know of any other way.
http://golang.org/pkg/reflect/
http://blog.golang.org/laws-of-reflection
The simplest way to do this is to make your parameter of type interface{} and pass in an empty (uninitialized) instance of the struct you wish to unmarshal into. I highly suggest reading the second of the two links I list - it provides a clear introduction to reflection and how to use it for solving problems like this one.