Custom unmarshalling from flat json to nested struct - json

Lets say I have two struct that are related like this:
type SecretUser struct {
UserInfo `json:"userInfo"`
Password string `json:"password"`
}
type UserInfo struct {
FirstName string `json:"firstName"`
LastName string `json:"lastName"`
Email string `json:"email"`
}
And I receive a JSON in this form:
{
"firstName": "nice",
"lastName":"guy",
"email":"nice#guy.co.uk",
"password":"abc123"
}
I want to unmarshall this JSON into a SecretUser. Is there a better way than doing it like this?
func (u *User) UnmarshalJSON(data []byte) error {
var objmap map[string]*json.RawMessage
var password string
var err error
err = json.Unmarshal(data, &objmap)
if err != nil {
return err
}
if err := json.Unmarshal(data, &u.UserInfo); err != nil {
return err
}
err = json.Unmarshal(*objmap["password"], &password)
if err != nil {
return err
}
u.Password = password
return nil
}
Basically, I partially unmarshall the JSON into a UserInfo struct and then read it again to extract the password. I don't want to create another struct for just unmarshalling this JSON cleanly or use an external library (unless it's part of the standard). Is there a more clean/efficient way of doing this, without reading the JSON twice or setting every field manually from a map?

Simply include the UserData into the SecretUser struct and do not specify a json tag for it.
type UserInfo struct {
FirstName string `json:"firstName"`
LastName string `json:"lastName"`
Email string `json:"email"`
}
type SecretUser struct {
UserInfo
Password string `json:"password"`
}
func main() {
data := []byte(`{"firstName": "nice","lastName":"guy","email":"nice#guy.co.uk","password":"abc123"}`)
var u SecretUser
json.Unmarshal(data, &u)
fmt.Println(u)
}
Go Play Space example

Related

Update data from incoming json request with existing json format using golangg

I have some json data coming from Postman and I want to update it in existing json file (update only required field not all fields) using Golang.
I created IP and port and open channel between me and postman I need to understand how update required element not all element(update name and phone inside authenticator) using Golang.
type he
type UserType struct {
User []struct {
CdbID string `json:"cdb_id"`
Firstname string `json:"firstname"`
Lastname string `json:"lastname"`
Phone int64 `json:"phone"`
Email string `json:"email"`
Address []struct {
Street string `json:"street"`
City string `json:"city"`
Zip string `json:"zip"`
Country string `json:"country"`
} `json:"address"`
Authenticators []Authenticator `json:"authenticators"`
VoiceSig string `json:"voice_sig"`
VoicesigCreatedTime string `json:"voicesig_created_time"`
Status string `json:"status"`
} `json:"user"`
}
type Authenticator struct {
Name string `json:"name"`
Phone int64 `json:"phone"`
}
var u UserType
func main() {
//Parse the Json-encoded Data and store the results in u
// update the user
r := mux.NewRouter()
r.HandleFunc("/", serverhome).Methods("POST")
log.Fatal(http.ListenAndServe("192.168.101.168:4000", r))
}
func serverhome(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("<h1>welcome to golang</h1>"))
file, err := os.ReadFile("user.json") //Read File
if err != nil {
panic(err)
}
json.Unmarshal(file, &u)
body, err := ioutil.ReadAll(r.Body) //read incoming request
if err != nil {
panic(err)
}
fmt.Println(string(body))
u.User[0].Authenticators = append(u.User[0].Authenticators, Authenticator{
Name: "",
})
result, e := json.Marshal(u) //Returns the Json encoding of u into the variable result
if e != nil {
panic(e)
}
var n = 0
n, err = os.Stdout.Write(result) //The line of code golang.org uses to print the Json encoding
if err != nil {
panic(err)
}
fmt.Println(n)
_ = os.WriteFile("test.json", result, 0644)
}re
type UserType struct {
User []struct {
CdbID string `json:"cdb_id"`
Firstname string `json:"firstname"`
Lastname string `json:"lastname"`
Phone int64 `json:"phone"`
Email string `json:"email"`
Address []struct {
Street string `json:"street"`
City string `json:"city"`
Zip string `json:"zip"`
Country string `json:"country"`
} `json:"address"`
Authenticators []Authenticator `json:"authenticators"`
VoiceSig string `json:"voice_sig"`
VoicesigCreatedTime string `json:"voicesig_created_time"`
Status string `json:"status"`
} `json:"user"`
}
type Authenticator struct {
Name string `json:"name"`
Phone int64 `json:"phone"`
}
var u UserType
func main() {
//Parse the Json-encoded Data and store the results in u
// update the user
r := mux.NewRouter()
r.HandleFunc("/", serverhome).Methods("POST")
log.Fatal(http.ListenAndServe("192.168.101.168:4000", r))
}
func serverhome(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("<h1>welcome to golang</h1>"))
file, err := os.ReadFile("user.json") //Read File
if err != nil {
panic(err)
}
json.Unmarshal(file, &u)
body, err := ioutil.ReadAll(r.Body) //read incoming request
if err != nil {
panic(err)
}
fmt.Println(string(body))
u.User[0].Authenticators = append(u.User[0].Authenticators, Authenticator{
Name: "",
})
result, e := json.Marshal(u) //Returns the Json encoding of u into the variable result
if e != nil {
panic(e)
}
var n = 0
n, err = os.Stdout.Write(result) //The line of code golang.org uses to print the Json encoding
if err != nil {
panic(err)
}
fmt.Println(n)
_ = os.WriteFile("test.json", result, 0644)
}

Convert sub struct to string and back

I am having some trouble with the UnmarshalJSON and MarshalJSON i am not sure i understand them correctly or atleast i have an error that i have not been able to spot and in my 3 hours of googling i haven't been able find, so here goes.
I have the following User struct
type User struct {
ID uuid.UUID `gorm:"primaryKey,type:string,size:36,<-:create" json:"id"`
Username string `gorm:"unique" json:"username"`
Password PasswordHash `gorm:"type:string" json:"password"`
CreatedAt time.Time `gorm:"autoCreateTime:milli" json:"created_at"`
UpdatedAt time.Time `gorm:"autoUpdateTime:milli" json:"updated_at,omitempty"`
DeletedAt gorm.DeletedAt `gorm:"index" json:"deleted_at,omitempty"`
}
which when i try to convert into JSON, with the json.Marshal function from the build in encoding/json library, i get the following:
{"id":"3843298e-74d4-4dd7-8eff-007ab34a4c19","username":"root","password":{},"created_at":"0001-01-01T00:00:00Z","updated_at":"0001-01-01T00:00:00Z","deleted_at":null}
I expected some thing like:
{"id":"3843298e-74d4-4dd7-8eff-007ab34a4c19","username":"root","password":"$argon2id$v=19$m=4194304,t=1,p=64$Z9EFSTk26TQxx+Qv9g58gQ$4At0rvvv9trRcFZmSMXY0nISBuEt+1X8mCRAYbyXqSs","created_at":"0001-01-01T00:00:00Z","updated_at":"0001-01-01T00:00:00Z","deleted_at":null}
i can't seem to get the password field to be a string of the hashed password.
even when i have the following methods on the PasswordHash struct
type Params struct {
memory uint32
iterations uint32
parallelism uint8
saltLength uint32
keyLength uint32
}
type PasswordHash struct {
hash []byte
salt []byte
params Params
}
func (h *PasswordHash) String() string {
b64Salt := base64.RawStdEncoding.EncodeToString(h.salt)
b64Hash := base64.RawStdEncoding.EncodeToString(h.hash)
return fmt.Sprintf(
"$%s$v=%d$m=%d,t=%d,p=%d$%s$%s",
algoName,
argon2.Version,
h.params.memory,
h.params.iterations,
h.params.parallelism,
b64Salt,
b64Hash,
)
}
func (h *PasswordHash) UnmarshalJSON(data []byte) error {
var v string
if err := json.Unmarshal(data, &v); err != nil {
return err
}
params, salt, hash, err := decodeHash(v)
if err != nil {
return err
}
h.params = params
h.salt = salt
h.hash = hash
return nil
}
func (h *PasswordHash) MarshalJSON() ([]byte, error) {
//return []byte(h.String()), nil
return json.Marshal(h.String())
}
So i guess my question is, shouldn't the MarshalJSON be called on the PasswordHash struct when trying to convert the user to JSON? and if so how come i can't seem to get it to be a string value?
You have MarshalJSON defined with a receiver of *PasswordHash, but your value is type PasswordHash. Change the receiver to PasswordHash and it works as expected: https://go.dev/play/p/WukE_5JBEPL

How to prevent an array from being embedded into another array in my json response

In my api server I return a json object which has string arrays in it. I am finding that my arrays are being embedded into another array like this:
"Items": [
"[\"QQTISGXSJIS4DEV36JCBWQ4X\", \"HCOWEB7NVIQEUAINMM2KUV6J\", \"FCKP7D3H6Q7RQIRKSPVZBRHL\", \"UQLVH65PPBTVK6KMIV5KMGY6\", \"UR2XTXJFVURE5ERBLNW7ZUCR\", \"75N66F4DYGPM57V47N3IBMKD\", \"HQ2CRXQFPQM7TNNDZXZ2MQ2B\", \"3SLGKFR5GPHVZMQM4YM6KI4U\", \"UCQ3J7GYAYPZOCQKWIRGNGNY\", \"6INWDYWUFX6L5JYX2HEVMMHX\", \"ASQBRMKYSK2TINHBYQIWATS5\", \"QPCHVJ4HXYTUJNEZWQCKM5I3\", \"7JPYYH64Y3FQK6YJX5NBXMM6\", \"BI4NIBBOFBYAAS7ZROD6XEMB\", \"RGU3X36VYMXX4N3XPEZKY76K\", \"PLHVIQ7QT6TBWI5BZX6EJI74\", \"YATHGR6W6BIKFYXVZMGVBRB4\", \"ZZ5KZ5ZSBVLQRDKR2SJQ5CXW\", \"TNH56AOIMFSLOX5AW5I6WYP2\", \"VIFSURNJWJ6YYKXIWTWRNY6F\"]"
]
You can a complete JSON object here: https://gist.github.com/yshuman1/31b39333e2cd187707d98817171c3914
I am using gorm and saving my array as pq.StringArray.
Here is my function that returns that json:
func (u *Users) RetrieveItemCatModSort(w http.ResponseWriter, r *http.Request) {
var l model.ItemCatModSort
type data struct {
Email string
Password string
Location string
}
var form data
if err := parseForm(r, &form); err != nil {
log.Infof("error parsing form for logging in Kiosk %v", err)
return
}
user, err := u.us.ByEmail(string(form.Email))
if err != nil {
log.Infof("error looking up user by email while retrieving items %v", err)
return
}
bcrypt.CompareHashAndPassword([]byte(user.KioskPwHash), []byte(form.Password))
if err != nil {
log.Infof("invalid login credentials error: %v submitted data:\n%#v", err, form)
w.Header().Set("Content-Type", "application/json; charset=UTF-8")
w.WriteHeader(422) // unprocessable entity
if err := json.NewEncoder(w).Encode(err); err != nil {
panic(err)
}
}
//lookup oauth token in DB
token, err := u.os.Find(user.ID, "square")
if err != nil {
fmt.Println(err)
}
l.LocationID = form.Location
l.Modifiers, err = pos.GetModifiers(token.AccessToken, l.LocationID)
if err != nil {
log.Infof("Error grabbing modifiers using GetInventory. %#v", err)
}
l.Categories, err = pos.GetCategories(token.AccessToken)
if err != nil {
log.Infof("Error grabbing categories using GetInventory. %#v", err)
}
l.Items, err = pos.GetItems(token.AccessToken, l.LocationID)
if err != nil {
log.Infof("Error grabbing items using GetInventory. %#v", err)
}
l.ItemSort, err = u.is.FindItemSortByLocation(l.LocationID)
if err != nil {
log.Infof("Error grabbing item sort. %#v", err)
}
l.CatSort, err = u.is.FindCatSortByLocation(l.LocationID)
if err != nil {
log.Infof("Error grabbing cat sort. %#v", err)
}
lJSON, err := json.Marshal(l)
if err != nil {
log.Infof("error marshalling data to lJSON inside RetrieveLocationInv %v", err)
return
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
w.Write(lJSON)
}
Here are some of the models that I use:
type Categories struct {
Categories []Category `json:"objects"`
}
type ItemCatModSort struct {
LocationID string
Categories Categories
Items []Item
Modifiers []Modifier
ItemSort []ItemSort
CatSort CatSort
}
type CatSort struct {
gorm.Model
LocationID string
Category pq.StringArray `gorm:"type:varchar(10485760)[]"`
}
type Item struct {
gorm.Model
Type string `json:"type"`
ID string `json:"id"`
UpdatedAt time.Time `json:"updated_at"`
Version int64 `json:"version"`
IsDeleted bool `json:"is_deleted"`
PresentAtAllLocations bool `json:"present_at_all_locations"`
PresentAtLocationIds []string `json:"present_at_location_ids"`
AbsentAtLocationIds []string `json:"absent_at_location_ids"`
ItemData struct {
Name string `json:"name"`
Description string `json:"description"`
Visibility string `json:"visibility"`
CategoryID string `json:"category_id"`
ModifierListInfo []struct {
ModifierListID string `json:"modifier_list_id"`
Visibility string `json:"visibility"`
MinSelectedModifiers int `json:"min_selected_modifiers"`
MaxSelectedModifiers int `json:"max_selected_modifiers"`
Enabled bool `json:"enabled"`
} `json:"modifier_list_info"`
ImageURL string `json:"image_url"`
Variations []struct {
Type string `json:"type"`
ID string `json:"id"`
UpdatedAt time.Time `json:"updated_at"`
Version int64 `json:"version"`
IsDeleted bool `json:"is_deleted"`
PresentAtAllLocations bool `json:"present_at_all_locations"`
PresentAtLocationIds []string `json:"present_at_location_ids"`
AbsentAtLocationIds []string `json:"absent_at_location_ids"`
ItemVariationData struct {
ItemID string `json:"item_id"`
Name string `json:"name"`
Sku string `json:"sku"`
Ordinal int `json:"ordinal"`
PricingType string `json:"pricing_type"`
PriceMoney struct {
Amount int `json:"amount"`
Currency string `json:"currency"`
} `json:"price_money"`
} `json:"item_variation_data"`
} `json:"variations"`
ProductType string `json:"product_type"`
SkipModifierScreen bool `json:"skip_modifier_screen"`
} `json:"item_data"`
}
basically instead of
"Items": [
"[\"QQTISGXSJIS4DEV36JCBWQ4X\", \"HCOWEB7NVIQEUAINMM2KUV6J\",... \"VIFSURNJWJ6YYKXIWTWRNY6F\"]" ]
looking like that, I'd want it to look like this:
"Items": ["QQTISGXSJIS4DEV36JCBWQ4X\", \"HCOWEB7NVIQEUAINMM2KUV6J\",... \"VIFSURNJWJ6YYKXIWTWRNY6F\" ]

Unmarshalling json to structure using json.RawMessage

I need to unmarshal json object which may have the following formats:
Format1:
{
"contactType": 2,
"value": "0123456789"
}
Format2:
{
"contactType": "MobileNumber",
"value": "0123456789"
}
The structure I'm using for unmarshalling is:-
type Contact struct {
ContactType int `json:"contactType"`
Value string `json:"value"`
}
But this works only for format 1. I don't want to change the datatype of ContactType but I want to accommodate the 2nd format as well. I heard about json.RawMarshal and tried using it.
type Contact struct {
ContactType int
Value string `json:"value"`
Type json.RawMessage `json:"contactType"`
}
type StringContact struct {
Type string `json:"contactType"`
}
type IntContact struct {
Type int `json:"contactType"`
}
This gets the unmarshalling done, but I'm unable to set the ContactType variable which depends on the type of json.RawMessage. How do I model my structure so that this problem gets solved?
You will need to do the unmarshalling yourself. There is a very good article that shows how to use the json.RawMessage right and a number of other solutions to this very problem, Like using interfaces, RawMessage, implemention your own unmarshal and decode functions etc.
You will find the article here: JSON decoding in GO by Attila Oláh
Note: Attila has made a few errors on his code examples.
I taken the liberty to put together (using some of the code from Attila) a working example using RawMessage to delay the unmarshaling so we can do it on our own version of the Decode func.
Link to GOLANG Playground
package main
import (
"fmt"
"encoding/json"
"io"
)
type Record struct {
AuthorRaw json.RawMessage `json:"author"`
Title string `json:"title"`
URL string `json:"url"`
Author Author
}
type Author struct {
ID uint64 `json:"id"`
Email string `json:"email"`
}
func Decode(r io.Reader) (x *Record, err error) {
x = new(Record)
if err = json.NewDecoder(r).Decode(x); err != nil {
return
}
if err = json.Unmarshal(x.AuthorRaw, &x.Author); err == nil {
return
}
var s string
if err = json.Unmarshal(x.AuthorRaw, &s); err == nil {
x.Author.Email = s
return
}
var n uint64
if err = json.Unmarshal(x.AuthorRaw, &n); err == nil {
x.Author.ID = n
}
return
}
func main() {
byt_1 := []byte(`{"author": 2,"title": "some things","url": "https://stackoverflow.com"}`)
byt_2 := []byte(`{"author": "Mad Scientist","title": "some things","url": "https://stackoverflow.com"}`)
var dat Record
if err := json.Unmarshal(byt_1, &dat); err != nil {
panic(err)
}
fmt.Printf("%#s\r\n", dat)
if err := json.Unmarshal(byt_2, &dat); err != nil {
panic(err)
}
fmt.Printf("%#s\r\n", dat)
}
Hope this helps.

Pass `struct` as type name to function argument

I'm writing some sort of RESTfull API based Object relational mapper in go.
I plan to make it MIT licensed, when i finish it.
The idea is to use some 3rd party REST API as data storage, and the golang client will query it for data needed.
The API responses are JSONs with known structure.
this is my code:
type AClient struct {
Id string `json:"id"`
Uid string `json:"uid"`
FirstName string `json:"firstName"`
LastName string `json:"lastName"`
CreatedAt string `json:"createdAt"`
UpdatedAt string `json:"updatedAt"`
City string `json:"city"`
Address string `json:"address"`
Telefone string `json:"telefone"`
Zip string `json:"zip"`
Telefon string `json:"telefon"`
Comment string `json:"comment"`
}
type AEvents struct {
Id string `json:"id"`
Security bool `json:"security"`
Description string `json:"description"`
Good AGood `json:"good"`
Client AClient `json:"client"`
Author AAuthor `json:"author"`
InFuture bool `json:"inFuture"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
}
type Entry struct {
AEvents //this have to be changed to `AClients` in runtime when needed
}
type ORM struct {
ApiUrl string
ModelName string
ModelInterface Entry
HuntKey string
HuntSid string
Csrf string
}
func (o *ORM) Query(parameters map[string]string) ([]Entry, AMetadata, error) {
responseParsed := struct {
Status string `json:"status"`
Metadata AMetadata `json:"metadata"`
Data []Entry `json:"data"` //todo - use o.ModelInterface
}{}
client := &http.Client{}
var queryString string
for k, v := range parameters {
queryString = queryString + fmt.Sprintf("%v=%v&", url.QueryEscape(k), url.QueryEscape(v))
}
req, err := http.NewRequest("GET", fmt.Sprintf("%v%v?%v", o.ApiUrl, o.ModelName, queryString), nil)
fmt.Println("---------------------------------------------")
fmt.Println(fmt.Sprintf("[GET] %v%v?%v", o.ApiUrl, o.ModelName, queryString))
req.Header.Set("huntKey", o.HuntKey)
if err != nil {
return nil, AMetadata{}, err
}
res, err1 := client.Do(req)
defer res.Body.Close()
if err1 != nil {
return nil, AMetadata{}, err1
}
if res.StatusCode == 200 {
for _, v := range res.Cookies() {
if v.Name == "XSRF-TOKEN" {
o.Csrf = v.Value
}
if v.Name == "hunt.sid" {
o.HuntSid = v.Value
}
}
fmt.Printf("CSRF %v\n", o.Csrf)
fmt.Printf("HuntSid %v\n", o.HuntSid)
fmt.Println("---------------------------------------------")
raw, err2 := ioutil.ReadAll(res.Body)
if err2 != nil {
return nil, AMetadata{}, err2
} else {
err2 = json.Unmarshal(raw, &responseParsed)
return responseParsed.Data, responseParsed.Metadata, nil
}
} else {
return nil, AMetadata{}, errors.New("Unable to fetch data!")
}
}
How can I make this:
When instantiating the ORM object, how can i pass the struct name, that will be used to parse the JSON response. The current code works with the struct of AEvents, but i want it to be easy changeble t AClient and so on.
UPD:
i have reviewed the code of https://github.com/jinzhu/gorm and find out tons of things how can i implement it.
Than, as i have promised, I publish this code as open source -https://github.com/vodolaz095/hrorm
You can use reflection, but be warned that it is non-trivial and relatively slow. I don't know of any other way.
http://golang.org/pkg/reflect/
http://blog.golang.org/laws-of-reflection
The simplest way to do this is to make your parameter of type interface{} and pass in an empty (uninitialized) instance of the struct you wish to unmarshal into. I highly suggest reading the second of the two links I list - it provides a clear introduction to reflection and how to use it for solving problems like this one.