Golang: Decode json string to struct from mysql db - mysql

I'm trying to get informations from my db, and one of my fields is actually JSON stored as a string and I would like to get it as a struct.
This is my row's struct :
//there is json flag because I use it to get data from redis too
type InfoClip struct {
ClipId string `json:clipId`
StreamUrl string `json:streamUrl`
StartTimeCode int `json:startTimeCode`
EndTimeCode int `json:endTimeCode`
CreatedAt string `json:createdAt`
Metas string `json:metas` // here I get a string instead of a 'metas' struct
SourceId string `json:sourceId`
ProviderName string `json:providerName`
ProviderReference string `json:providerReference`
PublicationStatus string `json:publicationStatus`
UserId string `json:userId`
Name string `json:name`
FacebookPage string `json:facebookPage`
TwitterHandle string `json:twitterHandle`
PermaLinkUrl string `json:permalinkUrl`
Logo string `json:logo`
Link string `json:link`
}
This is my metas struct :
type metas struct {
Title string `json:title`
Tags []string `json:tags`
categories []string `json:permalink`
}
This is how I'm trying to get this field
func decodeJsonSql (met string) (*metas, error) {
m := metas{}
if err := json.Unmarshal([]byte(met), &m); err != nil {
fmt.Printf("Error decode metas: ", err)
return nil, err
} else {
return &m, err
}
}
func CheckIdSql(mediaId string) (error){
datab, err := sql.Open("mysql", "tcp()")
if err != nil {
fmt.Printf("[SQL ERROR] Cannot Open db => ", err)
return err
}
if err := datab.Ping(); err != nil {
fmt.Printf("[SQL ERROR] db connection => ", err)
return err
}
fmt.Printf("[SQL ONLINE] =>", datab)
defer datab.Close()
q := "SELECT c.id AS clipId, c.streamUrl, c.startTimecode, c.endTimecode, c.createdAt, s.metas,... FROM clips WHERE c.id = ?"
rows, err := datab.Query(q, mediaId)
if err != nil || err == sql.ErrNoRows {
fmt.Printf("SQL Err: %s", err)
return err
}
clips := InfoClip{}
for rows.Next() {
rows.Scan(&clips.ClipId, &clips.StreamUrl, &clips.StartTimeCode, &clips.EndTimeCode, &clips.CreatedAt, &clips.Metas, ...)
}
ret, err := decodeJsonSql(clips.Metas)
if err != nil{
return err
}
clips.Metas = ret
fmt.Printf("\n\n[SQL DEBUG RESPONSE]: %v", clips)
return nil
}
But this process is pretty heavy, surely there is an easier way?
Thanks.

You can make your metas struct implement the sql.Scanner interface
It should look something like this:
func (m *metas) Scan(src interface{}) error {
strValue, ok := src.(string)
if !ok {
return fmt.Errorf("metas field must be a string, got %T instead", src)
}
return json.Unmarshal([]byte(strValue), m)
}
After that you can use it as an InfoClip field and pass it directly to Scan and drop the decodeJsonSql:
type InfoClip struct {
// [...]
Metas metas `json:metas`
// [...]
}
and
q := "SELECT c.id AS clipId, c.streamUrl, c.startTimecode, c.endTimecode, c.createdAt, s.metas,... FROM clips WHERE c.id = ?"
row := datab.QueryRow(q, mediaId)
clips := InfoClip{}
err := row.Scan(&clips.ClipId, &clips.StreamUrl, &clips.StartTimeCode, &clips.EndTimeCode, &clips.CreatedAt, &clips.Metas) // [...]
if err != nil {
fmt.Printf("SQL Err: %s", err)
return err
}
(BTW, as you can see, I replaced datab.Query with datab.QueryRow as you are expecting only one result)

Related

How to do a generic decode with golang

I've seen some go code that looks like this :
type userForm struct {
Name string `json:"name" validate:"min=2"`
Surname string `json:"surname" validate:"min=2"`
Phone string `json:"phone" validate:"min=10"`
Email string `json:"email,omitempty" validate:"omitempty,email"`
}
type emailForm struct {
Email string `json:"email" validate:"email"`
}
func decodeUserForm(r *http.Request) (userForm, error) {
var body userForm
d := json.NewDecoder(r.Body)
if err := d.Decode(&body); err != nil {
return body, NewHTTPError(err, 400, "unable to decode user form")
}
return body, validateStruct(body)
}
func decodeEmailForm(r *http.Request) (emailForm, error) {
var body emailForm
d := json.NewDecoder(r.Body)
if err := d.Decode(&body); err != nil {
return body, NewHTTPError(err, 400, "unable to decode email form")
}
return body, validateStruct(body)
}
I find two functions redundant. Is there a simpler way to merge those two into a more generic function? Is it good practice in Go?
func decodeForm(r *http.Request, dst interface{}) error {
if err := json.NewDecoder(r.Body).Decode(dst); err != nil {
return NewHTTPError(err, 400, "unable to decode email form")
}
return validateStruct(dst)
}
Then use it as so:
var body emailForm
if err := decodeForm(r, &body); err != nil {
panic(err)
}

how to unmarshal json object if object is returning as empty string instead of empty struct

I'm receiving some data as JSON, but if a object is empty, it does not return a empty struct but a empty
string instead, and when unmarshaling, it returns an error.
So instead of data being {"key":{}} is {"key":""}} , it does not work even using omitempty field
Example: https://play.golang.org/p/N1iuWBxuo1C
type Store struct {
Title string `json:"title,omitempty"`
Item item `json:"item,omitempty"`
}
type item struct {
Price float32 `json:"price,omitempty"`
Kind string `json:"kind,omitempty"`
}
func main() {
var data1 Store
json1 := []byte(`{"title":"hello world","item":{"price":45.2,"kind":"fruit"}}`)
if err := json.Unmarshal(json1, &data1); err != nil {
log.Println("1, err: ", err)
return
}
log.Printf("data1: %+v\n", data1)
var data2 Store
json2 := []byte(`{"title":"hello world","item":{}}`)
if err := json.Unmarshal(json2, &data2); err != nil {
log.Println("2, err: ", err)
return
}
log.Printf("data2: %+v\n", data2)
var data3 Store
json3 := []byte(`{"title":"hello world","item":""}`)
if err := json.Unmarshal(json3, &data3); err != nil {
log.Println("3, err: ", err)
return
}
log.Printf("data3: %+v\n", data3)
}
You can have your item type implement the json.Unmarshaler interface.
func (i *item) UnmarshalJSON(data []byte) error {
if string(data) == `""` {
return nil
}
type tmp item
return json.Unmarshal(data, (*tmp)(i))
}
https://play.golang.org/p/1TrD57XULo9
This might be a matter of taste, but "" is a string with zero length. Not an empty object. JSON uses null to describe something empty. This works:
json3 := []byte(`{"title":"hello world","item":null}`)
if err := json.Unmarshal(json3, &data3); err != nil {
log.Println("3, err: ", err)
return
}
As far as the documentation goes, omitempty is an encoding option:
The "omitempty" option specifies that the field should be omitted from
the encoding if the field has an empty value, defined as false, 0, a
nil pointer, a nil interface value, and any empty array, slice, map,
or string.
json.Unmarshal does not specify any use of the omitempty tag.
If you don't have control over the input, use an interface type, a type switch and type assertion:
type Store struct {
Title string `json:"title,omitempty"`
Item item `json:"item,omitempty"`
}
type item struct {
Price float32 `json:"price,omitempty"`
Kind string `json:"kind,omitempty"`
}
func unmarshal(js []byte) (*Store, error) {
var data = struct { // Intermediate var for unmarshal
Title string
Item interface{}
}{}
if err := json.Unmarshal(js, &data); err != nil {
return nil, err
}
s := &Store{Title: data.Title}
switch item := data.Item.(type) { // type switch
case string, nil:
return s, nil // Item remains empty
case map[string]interface{}:
p, ok := item["price"].(float64) // assertion
if ok {
s.Item.Price = float32(p)
}
s.Item.Kind, _ = item["kind"].(string) // _ prevents panic
return s, nil
default:
return nil, errors.New("Unknown type")
}
}
func main() {
jsons := [][]byte{
[]byte(`{"title":"hello world","item":{"price":45.2,"kind":"fruit"}}`),
[]byte(`{"title":"hello world","item":{}}`),
[]byte(`{"title":"hello world","item":""}`),
[]byte(`{"title":"hello world","item":null}`),
}
for i, js := range jsons {
data, err := unmarshal(js)
if err != nil {
log.Println("1, err: ", err)
return
}
log.Printf("data %d: %+v\n", i, data)
}
}
https://play.golang.org/p/Dnq1ZVfGPE7
Create a type like type ItemOrEmptyString item
And implement Unmarshal interface for it to handle your custom case.
func(ies *ItemOrEmptyString)UnmarshalJSON(d []byte) error{
var i item
if string(d) == `""` {
return nil
}
err := json.Unmarshal(d, &i)
*ies = ItemOrEmptyString(i)
return err
}
Full code here

Unable to unmarshal golang response

I've been trying to extract some JSON by unmarshalling my json response into structs, but I have no idea why it's not doing it properly. I've also tried gjson but same result. Am I missing something here?
JSON Result:
{"availabilities":[{"pickup":{"status":"OnlineOnly","purchasable":false},"shipping":{"status":"InStockOnlineOnly","purchasable":true},"sku":"12341231","sellerId":"438178","saleChannelExclusivity":"OnlineOnly","scheduledDelivery":false,"isGiftCard":false,"isService":false}]}
Code:
// Inventory ...
type Inventory struct {
Availabilities []Availability `json:"availabilities"`
}
// Availability ...
type Availability struct {
Sku string `json:"sku"`
SellerID string `json:"sellerId"`
SaleChannelExclusivity string `json:"saleChannelExclusivity"`
ScheduledDelivery bool `json:"scheduledDelivery"`
IsGiftCard bool `json:"isGiftCard"`
IsService bool `json:"isService"`
Pickup Statuses `json:"pickup"`
Shipping Statuses `json:"shipping"`
}
// Statuses ..
type Statuses struct {
Status string `json:"status"`
Purchasable bool `json:"purchasable"`
}
func (pr *Program) checkInventory() {
url := fmt.Sprintf("https://www.bestbuy.ca/ecomm-api/availability/products?accept-language=en-CA&skus=%s", pr.Sku)
log.Infof("URL %s", url)
resp, err := http.Get(url)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
bodyBytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
log.Info(string(bodyBytes))
var inv Inventory
json.Unmarshal(bodyBytes, &inv)
log.Infof("%+v", inv)
}
Console:
INFO[2020-04-07T03:01:10-07:00] URL https://www.bestbuy.ca/ecomm-api/availability/products?accept-language=en-CA&skus=12341231
INFO[2020-04-07T03:01:10-07:00] {"availabilities":[{"pickup":{"status":"OnlineOnly","purchasable":false},"shipping":{"status":"InStockOnlineOnly","purchasable":true},"sku":"12341231
,"sellerId":"438178","saleChannelExclusivity":"OnlineOnly","scheduledDelivery":false,"isGiftCard":false,"isService":false}]}
INFO[2020-04-07T03:01:10-07:00] {Availabilities:[]}
The problem is in the json.Unmarshall call. It is returning an error: "invalid character 'ï' looking for beginning of value” from json.Unmarshal
As it is explained here: The server is sending you a UTF-8 text string with a Byte Order Mark (BOM). The BOM identifies that the text is UTF-8 encoded, but it should be removed before decoding.
This can be done with the following line (using package "bytes"):
body = bytes.TrimPrefix(body, []byte("\xef\xbb\xbf"))
So the resulting working code is:
// Inventory ...
type Inventory struct {
Availabilities []Availability `json:"availabilities"`
}
// Availability ...
type Availability struct {
Sku string `json:"sku"`
SellerID string `json:"sellerId"`
SaleChannelExclusivity string `json:"saleChannelExclusivity"`
ScheduledDelivery bool `json:"scheduledDelivery"`
IsGiftCard bool `json:"isGiftCard"`
IsService bool `json:"isService"`
Pickup Statuses `json:"pickup"`
Shipping Statuses `json:"shipping"`
}
// Statuses ..
type Statuses struct {
Status string `json:"status"`
Purchasable bool `json:"purchasable"`
}
func (pr *Program) checkInventory() {
url := fmt.Sprintf("https://www.bestbuy.ca/ecomm-api/availability/products?accept-language=en-CA&skus=%s", pr.Sku)
log.Infof("URL %s", url)
resp, err := http.Get(url)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
bodyBytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
body := bytes.TrimPrefix(bodyBytes, []byte("\xef\xbb\xbf"))
log.Info(string(body))
var inv Inventory
err = json.Unmarshal([]byte(body), &inv)
if err != nil {
log.Fatal(err)
}
log.Infof("%+v", inv)
}

How to prevent an array from being embedded into another array in my json response

In my api server I return a json object which has string arrays in it. I am finding that my arrays are being embedded into another array like this:
"Items": [
"[\"QQTISGXSJIS4DEV36JCBWQ4X\", \"HCOWEB7NVIQEUAINMM2KUV6J\", \"FCKP7D3H6Q7RQIRKSPVZBRHL\", \"UQLVH65PPBTVK6KMIV5KMGY6\", \"UR2XTXJFVURE5ERBLNW7ZUCR\", \"75N66F4DYGPM57V47N3IBMKD\", \"HQ2CRXQFPQM7TNNDZXZ2MQ2B\", \"3SLGKFR5GPHVZMQM4YM6KI4U\", \"UCQ3J7GYAYPZOCQKWIRGNGNY\", \"6INWDYWUFX6L5JYX2HEVMMHX\", \"ASQBRMKYSK2TINHBYQIWATS5\", \"QPCHVJ4HXYTUJNEZWQCKM5I3\", \"7JPYYH64Y3FQK6YJX5NBXMM6\", \"BI4NIBBOFBYAAS7ZROD6XEMB\", \"RGU3X36VYMXX4N3XPEZKY76K\", \"PLHVIQ7QT6TBWI5BZX6EJI74\", \"YATHGR6W6BIKFYXVZMGVBRB4\", \"ZZ5KZ5ZSBVLQRDKR2SJQ5CXW\", \"TNH56AOIMFSLOX5AW5I6WYP2\", \"VIFSURNJWJ6YYKXIWTWRNY6F\"]"
]
You can a complete JSON object here: https://gist.github.com/yshuman1/31b39333e2cd187707d98817171c3914
I am using gorm and saving my array as pq.StringArray.
Here is my function that returns that json:
func (u *Users) RetrieveItemCatModSort(w http.ResponseWriter, r *http.Request) {
var l model.ItemCatModSort
type data struct {
Email string
Password string
Location string
}
var form data
if err := parseForm(r, &form); err != nil {
log.Infof("error parsing form for logging in Kiosk %v", err)
return
}
user, err := u.us.ByEmail(string(form.Email))
if err != nil {
log.Infof("error looking up user by email while retrieving items %v", err)
return
}
bcrypt.CompareHashAndPassword([]byte(user.KioskPwHash), []byte(form.Password))
if err != nil {
log.Infof("invalid login credentials error: %v submitted data:\n%#v", err, form)
w.Header().Set("Content-Type", "application/json; charset=UTF-8")
w.WriteHeader(422) // unprocessable entity
if err := json.NewEncoder(w).Encode(err); err != nil {
panic(err)
}
}
//lookup oauth token in DB
token, err := u.os.Find(user.ID, "square")
if err != nil {
fmt.Println(err)
}
l.LocationID = form.Location
l.Modifiers, err = pos.GetModifiers(token.AccessToken, l.LocationID)
if err != nil {
log.Infof("Error grabbing modifiers using GetInventory. %#v", err)
}
l.Categories, err = pos.GetCategories(token.AccessToken)
if err != nil {
log.Infof("Error grabbing categories using GetInventory. %#v", err)
}
l.Items, err = pos.GetItems(token.AccessToken, l.LocationID)
if err != nil {
log.Infof("Error grabbing items using GetInventory. %#v", err)
}
l.ItemSort, err = u.is.FindItemSortByLocation(l.LocationID)
if err != nil {
log.Infof("Error grabbing item sort. %#v", err)
}
l.CatSort, err = u.is.FindCatSortByLocation(l.LocationID)
if err != nil {
log.Infof("Error grabbing cat sort. %#v", err)
}
lJSON, err := json.Marshal(l)
if err != nil {
log.Infof("error marshalling data to lJSON inside RetrieveLocationInv %v", err)
return
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
w.Write(lJSON)
}
Here are some of the models that I use:
type Categories struct {
Categories []Category `json:"objects"`
}
type ItemCatModSort struct {
LocationID string
Categories Categories
Items []Item
Modifiers []Modifier
ItemSort []ItemSort
CatSort CatSort
}
type CatSort struct {
gorm.Model
LocationID string
Category pq.StringArray `gorm:"type:varchar(10485760)[]"`
}
type Item struct {
gorm.Model
Type string `json:"type"`
ID string `json:"id"`
UpdatedAt time.Time `json:"updated_at"`
Version int64 `json:"version"`
IsDeleted bool `json:"is_deleted"`
PresentAtAllLocations bool `json:"present_at_all_locations"`
PresentAtLocationIds []string `json:"present_at_location_ids"`
AbsentAtLocationIds []string `json:"absent_at_location_ids"`
ItemData struct {
Name string `json:"name"`
Description string `json:"description"`
Visibility string `json:"visibility"`
CategoryID string `json:"category_id"`
ModifierListInfo []struct {
ModifierListID string `json:"modifier_list_id"`
Visibility string `json:"visibility"`
MinSelectedModifiers int `json:"min_selected_modifiers"`
MaxSelectedModifiers int `json:"max_selected_modifiers"`
Enabled bool `json:"enabled"`
} `json:"modifier_list_info"`
ImageURL string `json:"image_url"`
Variations []struct {
Type string `json:"type"`
ID string `json:"id"`
UpdatedAt time.Time `json:"updated_at"`
Version int64 `json:"version"`
IsDeleted bool `json:"is_deleted"`
PresentAtAllLocations bool `json:"present_at_all_locations"`
PresentAtLocationIds []string `json:"present_at_location_ids"`
AbsentAtLocationIds []string `json:"absent_at_location_ids"`
ItemVariationData struct {
ItemID string `json:"item_id"`
Name string `json:"name"`
Sku string `json:"sku"`
Ordinal int `json:"ordinal"`
PricingType string `json:"pricing_type"`
PriceMoney struct {
Amount int `json:"amount"`
Currency string `json:"currency"`
} `json:"price_money"`
} `json:"item_variation_data"`
} `json:"variations"`
ProductType string `json:"product_type"`
SkipModifierScreen bool `json:"skip_modifier_screen"`
} `json:"item_data"`
}
basically instead of
"Items": [
"[\"QQTISGXSJIS4DEV36JCBWQ4X\", \"HCOWEB7NVIQEUAINMM2KUV6J\",... \"VIFSURNJWJ6YYKXIWTWRNY6F\"]" ]
looking like that, I'd want it to look like this:
"Items": ["QQTISGXSJIS4DEV36JCBWQ4X\", \"HCOWEB7NVIQEUAINMM2KUV6J\",... \"VIFSURNJWJ6YYKXIWTWRNY6F\" ]

golang json array unmarshal into type struct

What is causing this to not break apart json? Anyone have an idea on how type is wrong?
With error of:
{"Rates":[{"Symbol":"EURAUD","Bid":"1.45492","Ask":"1.45608","Spread":"11.60","ProductType":"1",},{"Symbol":"Copper","Bid":"2.6068","Ask":"2.6088","Spread":"2.00","ProductType":"3",},{"Symbol":"AUDNZD","Bid":"1.08999","Ask":"1.09177","Spread":"17.80","ProductType":"1",},{"Symbol":"EURSEK","Bid":"9.63786","Ask":"9.65569","Spread":"178.30","ProductType":"1",},{"Symbol":"CADJPY","Bid":"81.629","Ask":"81.708","Spread":"7.90","ProductType":"1",},{"Symbol":"USDCHF","Bid":"0.99463","Ask":"0.99527","Spread":"6.40","ProductType":"1",},{"Symbol":"USDCNH","Bid":"6.8973","Ask":"6.8993","Spread":"20.00","ProductType":"1",},{"Symbol":"US30","Bid":"20950.00","Ask":"20952.00","Spread":"2.00","ProductType":"2",},{"Symbol":"XAGUSD","Bid":"17.202","Ask":"17.25","Spread":"4.80","ProductType":"5",},{"Symbol":"USDSEK","Bid":"8.84794","Ask":"8.85542","Spread":"74.80","ProductType":"1",},{"Symbol":"AUDCHF","Bid":"0.74417","Ask":"0.74588","Spread":"17.10","ProductType":"1",},{"Symbol":"GER30","Bid":"12431.05","Ask":"12433.45","Spread":"2.40","ProductType":"2",},{"Symbol":"USOil","Bid":"49.16","Ask":"49.21","Spread":"5.00","ProductType":"3",},{"Symbol":"GBPNZD","Bid":"1.88546","Ask":"1.88762","Spread":"21.60","ProductType":"1",},{"Symbol":"EURCAD","Bid":"1.48748","Ask":"1.48893","Spread":"14.50","ProductType":"1",},{"Symbol":"EURUSD","Bid":"1.08977","Ask":"1.08997","Spread":"2.00","ProductType":"1",},{"Symbol":"AUS200","Bid":"5922.00","Ask":"5924.00","Spread":"2.00","ProductType":"2",},{"Symbol":"EURJPY","Bid":"121.512","Ask":"121.57","Spread":"5.80","ProductType":"1",},{"Symbol":"EURGBP","Bid":"0.84132","Ask":"0.84208","Spread":"7.60","ProductType":"1",},{"Symbol":"EURNOK","Bid":"9.34136","Ask":"9.36364","Spread":"222.80","ProductType":"1",},{"Symbol":"USDCAD","Bid":"1.36524","Ask":"1.36588","Spread":"6.40","ProductType":"1",},{"Symbol":"GBPCHF","Bid":"1.28753","Ask":"1.28922","Spread":"16.90","ProductType":"1",},{"Symbol":"GBPAUD","Bid":"1.72838","Ask":"1.7303","Spread":"19.20","ProductType":"1",},{"Symbol":"USDJPY","Bid":"111.51","Ask":"111.537","Spread":"2.70","ProductType":"1",},{"Symbol":"USDNOK","Bid":"8.57607","Ask":"8.58684","Spread":"107.70","ProductType":"1",},{"Symbol":"AUDCAD","Bid":"1.02173","Ask":"1.02347","Spread":"17.40","ProductType":"1",},{"Symbol":"FRA40","Bid":"5259.60","Ask":"5267.30","Spread":"7.70","ProductType":"2",},{"Symbol":"AUDUSD","Bid":"0.74858","Ask":"0.74899","Spread":"4.10","ProductType":"1",},{"Symbol":"USDHKD","Bid":"7.77769","Ask":"7.77956","Spread":"18.70","ProductType":"1",},{"Symbol":"NZDCHF","Bid":"0.68192","Ask":"0.68406","Spread":"21.40","ProductType":"1",},{"Symbol":"EURTRY","Bid":"3.86851","Ask":"3.87478","Spread":"62.70","ProductType":"1",},{"Symbol":"AUDJPY","Bid":"83.469","Ask":"83.543","Spread":"7.40","ProductType":"1",},{"Symbol":"USDZAR","Bid":"13.3464","Ask":"13.3941","Spread":"477.00","ProductType":"1",},{"Symbol":"Bund","Bid":"161.78","Ask":"161.81","Spread":"3.00","ProductType":"4",},{"Symbol":"USDMXN","Bid":"18.81249","Ask":"18.83178","Spread":"192.90","ProductType":"1",},{"Symbol":"USDTRY","Bid":"3.54925","Ask":"3.5536","Spread":"43.50","ProductType":"1",},{"Symbol":"USDOLLAR","Bid":"12232.00","Ask":"12237.00","Spread":"5.00","ProductType":"7",},{"Symbol":"JPN225","Bid":"19195.00","Ask":"19205.00","Spread":"10.00","ProductType":"2",},{"Symbol":"UK100","Bid":"7197.80","Ask":"7198.90","Spread":"1.10","ProductType":"2",},{"Symbol":"HKG33","Bid":"24650.00","Ask":"24655.00","Spread":"5.00","ProductType":"2",},{"Symbol":"CADCHF","Bid":"0.72748","Ask":"0.72979","Spread":"23.10","ProductType":"1",},{"Symbol":"NAS100","Bid":"5582.80","Ask":"5583.80","Spread":"1.00","ProductType":"2",},{"Symbol":"NGAS","Bid":"3.2645","Ask":"3.2755","Spread":"11.00","ProductType":"3",},{"Symbol":"ZARJPY","Bid":"8.323","Ask":"8.361","Spread":"3.80","ProductType":"1",},{"Symbol":"GBPCAD","Bid":"1.76724","Ask":"1.76912","Spread":"18.80","ProductType":"1",},{"Symbol":"ESP35","Bid":"10712.00","Ask":"10720.00","Spread":"8.00","ProductType":"2",},{"Symbol":"GBPUSD","Bid":"1.29452","Ask":"1.29527","Spread":"7.50","ProductType":"1",},{"Symbol":"SPX500","Bid":"2384.18","Ask":"2384.68","Spread":"5.00","ProductType":"2",},{"Symbol":"GBPJPY","Bid":"144.336","Ask":"144.448","Spread":"11.20","ProductType":"1",},{"Symbol":"EUSTX50","Bid":"3554.00","Ask":"3555.00","Spread":"1.00","ProductType":"2",},{"Symbol":"TRYJPY","Bid":"31.378","Ask":"31.44","Spread":"6.20","ProductType":"1",},{"Symbol":"NZDCAD","Bid":"0.93642","Ask":"0.93862","Spread":"22.00","ProductType":"1",},{"Symbol":"EURNZD","Bid":"1.58644","Ask":"1.58916","Spread":"27.20","ProductType":"1",},{"Symbol":"XAUUSD","Bid":"1267.79","Ask":"1268.26","Spread":"47.00","ProductType":"5",},{"Symbol":"NZDUSD","Bid":"0.68587","Ask":"0.68692","Spread":"10.50","ProductType":"1",},{"Symbol":"NZDJPY","Bid":"76.489","Ask":"76.607","Spread":"11.80","ProductType":"1",},{"Symbol":"UKOil","Bid":"51.84","Ask":"51.89","Spread":"5.00","ProductType":"3",},{"Symbol":"CHFJPY","Bid":"112.02","Ask":"112.148","Spread":"12.80","ProductType":"1",},{"Symbol":"EURCHF","Bid":"1.08416","Ask":"1.08459","Spread":"4.30","ProductType":"1",}]}
{"Rates":[{"Symbol":"EURAUD","Bid":"1.45492","Ask":"1.45608","Spread":"11.60","ProductType":"1"},{"Symbol":"Copper","Bid":"2.6068","Ask":"2.6088","Spread":"2.00","ProductType":"3"},{"Symbol":"AUDNZD","Bid":"1.08999","Ask":"1.09177","Spread":"17.80","ProductType":"1"},{"Symbol":"EURSEK","Bid":"9.63786","Ask":"9.65569","Spread":"178.30","ProductType":"1"},{"Symbol":"CADJPY","Bid":"81.629","Ask":"81.708","Spread":"7.90","ProductType":"1"},{"Symbol":"USDCHF","Bid":"0.99463","Ask":"0.99527","Spread":"6.40","ProductType":"1"},{"Symbol":"USDCNH","Bid":"6.8973","Ask":"6.8993","Spread":"20.00","ProductType":"1"},{"Symbol":"US30","Bid":"20950.00","Ask":"20952.00","Spread":"2.00","ProductType":"2"},{"Symbol":"XAGUSD","Bid":"17.202","Ask":"17.25","Spread":"4.80","ProductType":"5"},{"Symbol":"USDSEK","Bid":"8.84794","Ask":"8.85542","Spread":"74.80","ProductType":"1"},{"Symbol":"AUDCHF","Bid":"0.74417","Ask":"0.74588","Spread":"17.10","ProductType":"1"},{"Symbol":"GER30","Bid":"12431.05","Ask":"12433.45","Spread":"2.40","ProductType":"2"},{"Symbol":"USOil","Bid":"49.16","Ask":"49.21","Spread":"5.00","ProductType":"3"},{"Symbol":"GBPNZD","Bid":"1.88546","Ask":"1.88762","Spread":"21.60","ProductType":"1"},{"Symbol":"EURCAD","Bid":"1.48748","Ask":"1.48893","Spread":"14.50","ProductType":"1"},{"Symbol":"EURUSD","Bid":"1.08977","Ask":"1.08997","Spread":"2.00","ProductType":"1"},{"Symbol":"AUS200","Bid":"5922.00","Ask":"5924.00","Spread":"2.00","ProductType":"2"},{"Symbol":"EURJPY","Bid":"121.512","Ask":"121.57","Spread":"5.80","ProductType":"1"},{"Symbol":"EURGBP","Bid":"0.84132","Ask":"0.84208","Spread":"7.60","ProductType":"1"},{"Symbol":"EURNOK","Bid":"9.34136","Ask":"9.36364","Spread":"222.80","ProductType":"1"},{"Symbol":"USDCAD","Bid":"1.36524","Ask":"1.36588","Spread":"6.40","ProductType":"1"},{"Symbol":"GBPCHF","Bid":"1.28753","Ask":"1.28922","Spread":"16.90","ProductType":"1"},{"Symbol":"GBPAUD","Bid":"1.72838","Ask":"1.7303","Spread":"19.20","ProductType":"1"},{"Symbol":"USDJPY","Bid":"111.51","Ask":"111.537","Spread":"2.70","ProductType":"1"},{"Symbol":"USDNOK","Bid":"8.57607","Ask":"8.58684","Spread":"107.70","ProductType":"1"},{"Symbol":"AUDCAD","Bid":"1.02173","Ask":"1.02347","Spread":"17.40","ProductType":"1"},{"Symbol":"FRA40","Bid":"5259.60","Ask":"5267.30","Spread":"7.70","ProductType":"2"},{"Symbol":"AUDUSD","Bid":"0.74858","Ask":"0.74899","Spread":"4.10","ProductType":"1"},{"Symbol":"USDHKD","Bid":"7.77769","Ask":"7.77956","Spread":"18.70","ProductType":"1"},{"Symbol":"NZDCHF","Bid":"0.68192","Ask":"0.68406","Spread":"21.40","ProductType":"1"},{"Symbol":"EURTRY","Bid":"3.86851","Ask":"3.87478","Spread":"62.70","ProductType":"1"},{"Symbol":"AUDJPY","Bid":"83.469","Ask":"83.543","Spread":"7.40","ProductType":"1"},{"Symbol":"USDZAR","Bid":"13.3464","Ask":"13.3941","Spread":"477.00","ProductType":"1"},{"Symbol":"Bund","Bid":"161.78","Ask":"161.81","Spread":"3.00","ProductType":"4"},{"Symbol":"USDMXN","Bid":"18.81249","Ask":"18.83178","Spread":"192.90","ProductType":"1"},{"Symbol":"USDTRY","Bid":"3.54925","Ask":"3.5536","Spread":"43.50","ProductType":"1"},{"Symbol":"USDOLLAR","Bid":"12232.00","Ask":"12237.00","Spread":"5.00","ProductType":"7"},{"Symbol":"JPN225","Bid":"19195.00","Ask":"19205.00","Spread":"10.00","ProductType":"2"},{"Symbol":"UK100","Bid":"7197.80","Ask":"7198.90","Spread":"1.10","ProductType":"2"},{"Symbol":"HKG33","Bid":"24650.00","Ask":"24655.00","Spread":"5.00","ProductType":"2"},{"Symbol":"CADCHF","Bid":"0.72748","Ask":"0.72979","Spread":"23.10","ProductType":"1"},{"Symbol":"NAS100","Bid":"5582.80","Ask":"5583.80","Spread":"1.00","ProductType":"2"},{"Symbol":"NGAS","Bid":"3.2645","Ask":"3.2755","Spread":"11.00","ProductType":"3"},{"Symbol":"ZARJPY","Bid":"8.323","Ask":"8.361","Spread":"3.80","ProductType":"1"},{"Symbol":"GBPCAD","Bid":"1.76724","Ask":"1.76912","Spread":"18.80","ProductType":"1"},{"Symbol":"ESP35","Bid":"10712.00","Ask":"10720.00","Spread":"8.00","ProductType":"2"},{"Symbol":"GBPUSD","Bid":"1.29452","Ask":"1.29527","Spread":"7.50","ProductType":"1"},{"Symbol":"SPX500","Bid":"2384.18","Ask":"2384.68","Spread":"5.00","ProductType":"2"},{"Symbol":"GBPJPY","Bid":"144.336","Ask":"144.448","Spread":"11.20","ProductType":"1"},{"Symbol":"EUSTX50","Bid":"3554.00","Ask":"3555.00","Spread":"1.00","ProductType":"2"},{"Symbol":"TRYJPY","Bid":"31.378","Ask":"31.44","Spread":"6.20","ProductType":"1"},{"Symbol":"NZDCAD","Bid":"0.93642","Ask":"0.93862","Spread":"22.00","ProductType":"1"},{"Symbol":"EURNZD","Bid":"1.58644","Ask":"1.58916","Spread":"27.20","ProductType":"1"},{"Symbol":"XAUUSD","Bid":"1267.79","Ask":"1268.26","Spread":"47.00","ProductType":"5"},{"Symbol":"NZDUSD","Bid":"0.68587","Ask":"0.68692","Spread":"10.50","ProductType":"1"},{"Symbol":"NZDJPY","Bid":"76.489","Ask":"76.607","Spread":"11.80","ProductType":"1"},{"Symbol":"UKOil","Bid":"51.84","Ask":"51.89","Spread":"5.00","ProductType":"3"},{"Symbol":"CHFJPY","Bid":"112.02","Ask":"112.148","Spread":"12.80","ProductType":"1"},{"Symbol":"EURCHF","Bid":"1.08416","Ask":"1.08459","Spread":"4.30","ProductType":"1"}]}
panic: json: cannot unmarshal string into Go value of type
main.MsgRatesArray
goroutine 1 [running]:
main.main()
/tmp/test.go:50 +0x52c
With this code:
package main
import (
"log"
"fmt"
"net/http"
"bytes"
"io/ioutil"
"strings"
"github.com/pquerna/ffjson/ffjson"
)
type MsgRatesArray struct {
RateQuote []MsgRateQuoteJson `json:"Rates"`
}
type MsgRateQuoteJson struct {
SymbolName string `json:"Symbol"`
SymbolBid int64 `json:"Bid"`
SymbolAsk int64 `json:"Ask"`
SymbolSpread int64 `json:"Spread"`
SymbolPT string `json:"ProductType"`
}
var respBytes []byte
func main() {
var msg MsgRatesArray
response,err := http.Get("https://ratesjson.fxcm.com/DataDisplayer")
if err != nil {
log.Fatal(err)
}
defer response.Body.Close()
respBytes, err := ioutil.ReadAll(response.Body)
//Get bad JSON into string
jsonBytes := respBytes[bytes.Index(respBytes, []byte("{")):bytes.LastIndex(respBytes, []byte("}"))+1]
jsonString := string(jsonBytes)
fmt.Println(jsonString)
// Use a positive number to indicate max replacement count to fix bad JSON string so we can remove comma in JSON ARRAY.
result := strings.Replace(jsonString, "\",}", "\"}", -1)
fmt.Println(result)
// Turn GOOD JSON string back to JSON BYTES (BIN)
jsonBytes2, err := ffjson.Marshal(result)
if err != nil {
panic(err)
}
// Parse JSON !
err = ffjson.Unmarshal(jsonBytes2, &msg)
if err != nil {
panic(err)
}
}
What is wrong with type for json array?
In the returned json Bid, Ask, and Spread are json strings not integers, so change your type definition to this:
type MsgRateQuoteJson struct {
SymbolName string `json:"Symbol"`
SymbolBid string `json:"Bid"`
SymbolAsk string `json:"Ask"`
SymbolSpread string `json:"Spread"`
SymbolPT string `json:"ProductType"`
}
And marshaling a json string to get json bytes is not the correct way, just convert the string to a byte slice like this:
jsonBytes2 := []byte(result)
... and you're good to go:
// Parse JSON !
err = ffjson.Unmarshal(jsonBytes2, &msg)
if err != nil {
panic(err)
}
Edit:
If you want to convert those strings into specific types during the json unmarshaling, you can do so by defining an UnmarshalJSON method on the *MsgRateQuoteJson type, plus with the help of the strconv package, like this:
type MsgRateQuoteJson struct {
SymbolName string `json:"Symbol"`
SymbolBid float64 `json:"Bid"`
SymbolAsk float64 `json:"Ask"`
SymbolSpread float64 `json:"Spread"`
SymbolPT int64 `json:"ProductType"`
}
func (msg *MsgRateQuoteJson) UnmarshalJSON(data []byte) (err error) {
m := map[string]string{}
if err = ffjson.Unmarshal(data, &m); err != nil {
return err
}
msg.SymbolName = m["Symbol"]
if msg.SymbolBid, err = strconv.ParseFloat(m["Bid"], 64); err != nil {
return err
}
if msg.SymbolAsk, err = strconv.ParseFloat(m["Ask"], 64); err != nil {
return err
}
if msg.SymbolSpread, err = strconv.ParseFloat(m["Spread"], 64); err != nil {
return err
}
if msg.SymbolPT, err = strconv.ParseInt(m["ProductType"], 10, 64); err != nil {
return err
}
return nil
}